14:00:16 <efried> #startmeeting nova
14:00:16 <openstack> Meeting started Thu Sep 19 14:00:16 2019 UTC and is due to finish in 60 minutes.  The chair is efried. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:17 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:00:19 <openstack> The meeting name has been set to 'nova'
14:00:29 <dansmith> o/
14:00:37 <gibi> \o
14:00:41 <mriedem> o/
14:00:41 <takashin> o/
14:01:09 <bauzas> \o
14:02:10 * mordred throws a smoke bomb into the nova room and runs away giggling
14:02:17 <mriedem> twice in one week
14:02:49 <efried> #link agenda https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting
14:03:14 <efried> #topic Last meeting
14:03:14 <efried> #link Minutes from last meeting: http://eavesdrop.openstack.org/meetings/nova/2019/nova.2019-09-12-21.01.html
14:03:14 <efried> Actions from last week:
14:03:14 <efried> #link train blueprints scrubbed https://blueprints.launchpad.net/nova/train
14:03:14 <efried> #link cycle highlights merged https://review.opendev.org/#/c/681943/
14:03:15 <efried> #link and updated https://review.opendev.org/#/c/682509/
14:03:31 <efried> #topic Release News
14:03:31 <efried> #link 1 week until RC1: https://wiki.openstack.org/wiki/Nova/Train_Release_Schedule
14:03:31 <efried> Waiting for two blueprints to merge:
14:03:31 <efried> #link https://review.opendev.org/#/q/topic:bp/virtual-persistent-memory+status:open (3 open changes)
14:03:31 <efried> #link https://review.opendev.org/#/q/topic:bp/cpu-resources+status:open (16 open changes)
14:04:08 <efried> reallyl 1 and 10, respectively. The remainder are cleanups etc.
14:04:28 <dansmith> so,
14:04:40 <dansmith> we really have to merge the reshaper for pcpu or we're toast
14:04:47 <dansmith> is that where you're counting the end of the ten?
14:04:52 <efried> yes
14:05:06 <dansmith> that really should have been earlier in the series I think :/
14:05:33 <efried> Not sure it would have worked if it was.
14:06:01 <dansmith> of course.. it just gets tested end to end later once we can generate the inventory
14:06:29 <mriedem> i never looked back on it, but did the reshaper for pcpu account for in-progress (not yet confirmed) migrations?
14:06:37 <stephenfin> mriedem: yup
14:06:38 <efried> do you feel strongly enough about this that you think we should try to reorder the series at this stage?
14:07:18 <efried> dansmith: than was for you ^
14:07:27 <dansmith> efried: we're jamming so much in way past the deadline that I don't think much matters anymore
14:07:43 <dansmith> to be totally honest.
14:09:07 <efried> here's this too
14:09:07 <efried> #link train to-do https://etherpad.openstack.org/p/nova-train-release-todo
14:09:27 <mriedem> from ^,
14:09:31 <mriedem> we're down to 2 blocking bugs https://bugs.launchpad.net/nova/+bugs?field.tag=train-rc-potential
14:09:34 <mriedem> i'm +2 on both,
14:09:39 <mriedem> 1 has a conflict with the pcpu series
14:09:39 <dansmith> I'm super curious about the migration skip thing
14:09:57 <mriedem> let's get to that in gate status
14:09:58 <mriedem> coming up
14:10:15 <dansmith> well, it's on the todo, but sure
14:10:24 <mriedem> the revert is yeah
14:10:52 <efried> moving on?
14:11:05 <efried> or did you want to go deeper into those bugs?
14:11:13 <mriedem> keep going
14:11:55 <efried> #topic Bugs (stuck/critical)
14:11:55 <efried> No Critical bugs
14:11:55 <efried> #link 75 new untriaged bugs (-1 since the last meeting): https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New
14:11:55 <efried> #link 4 untagged untriaged bugs (+1 since the last meeting): https://bugs.launchpad.net/nova/+bugs?field.tag=-*&field.status%3Alist=NEW
14:12:36 <efried> bug words?
14:12:44 <mriedem> no
14:12:48 <efried> #topic Gate status
14:12:48 <efried> Some zuul restarts yesterday.
14:12:48 <efried> #info Temporarily skipping TestNovaMigrationsMySQL until VPMEM and PCPU changes merge: https://review.opendev.org/#/c/683009/ - will revert the skip before RC1.
14:12:48 <efried> #link check queue gate status http://status.openstack.org/elastic-recheck/index.html
14:12:48 <efried> #link 3rd party CI status (seems to be back in action) http://ciwatch.mmedvede.net/project?project=nova
14:12:58 <mriedem> before getting into that skip,
14:13:11 <mriedem> i noticed the bottom of the pcpu series was rechecked and had a failure in the live migration job https://review.opendev.org/#/c/680107/
14:13:23 <mriedem> someone should investigate that (i've got it open but haven't dug in yet)
14:14:01 <mriedem> as for skipping TestNovaMigrationsMySQL that's temporary to get the vpmem and pcpu series through which continue to fail on that bug,
14:14:11 <mriedem> for those that don't know, that bug only shows up on one node provider,
14:14:14 <mriedem> we do'nt have a clear fix,
14:14:26 <mriedem> i'm not aware of any remaining open changes we plan on merging that change db schema
14:14:27 <efried> Yeah, has nothing to do with VPMEM/PCPU specifically. It's just gate fails frequently enough on this bug that it's just keeping stuff from merging. We haven't figured out how to fix it yet.
14:14:33 <mriedem> and we still have the pg version of that test
14:15:01 <efried> been trying to dig in and get logs and stuff, but so far no enlightenment.
14:15:02 <bauzas> we're not merging a DB migration AFAIK
14:15:05 <dansmith> if it was the middle of the cycle, that would sound totally legit
14:15:05 <mriedem> so it's a temporary skip to unblock the gate since it seems we're pot committed to getting vpmem and pcpu merged regardless of when
14:15:08 <bauzas> so I think it's okay
14:16:48 <efried> #link logstash for bug 1823251 http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22sqlalchemy.exc.InterfaceError:%20(pymysql.err.InterfaceError)%20(0,%20'')%5C%22%20AND%20tags:%5C%22console%5C%22%20AND%20voting:1&from=864000s
14:16:49 <openstack> bug 1823251 in OpenStack Compute (nova) "Spike in TestNovaMigrationsMySQL.test_walk_versions/test_innodb_tables failures since April 1 2019 on limestone-regionone" [High,Confirmed] https://launchpad.net/bugs/1823251
14:17:49 <efried> move on?
14:18:15 <efried> #topic Reminders
14:18:15 <efried> #link Release note prelude: https://etherpad.openstack.org/p/nova-train-prelude
14:18:15 <efried> #link September 20 (tomorrow) is the deadline for Forum topic submissions: https://cfp.openstack.org/
14:18:54 <efried> what's the action on the prelude? I guess that gets parlayed into a reno patch at some point soon?
14:19:00 <mriedem> yes
14:19:02 <mriedem> needs an owner
14:19:14 <mriedem> most if it can be lifted from the highlights
14:19:45 <efried> #help Volunteer to write up the prelude patch
14:19:54 <bauzas> I can
14:19:59 <efried> Merci bien bauzas
14:20:03 <bauzas> I did in the past, I'm a bit rusty tho
14:20:11 <efried> #action bauzas to write reno prelude patch
14:20:30 <efried> #topic Stable branch status
14:20:30 <efried> #link stable/stein: https://review.openstack.org/#/q/status:open+(project:openstack/os-vif+OR+project:openstack/python-novaclient+OR+project:openstack/nova)+branch:stable/stein
14:20:30 <efried> #link stable/rocky: https://review.openstack.org/#/q/status:open+(project:openstack/os-vif+OR+project:openstack/python-novaclient+OR+project:openstack/nova)+branch:stable/rocky
14:20:30 <efried> #link stable/queens: https://review.openstack.org/#/q/status:open+(project:openstack/os-vif+OR+project:openstack/python-novaclient+OR+project:openstack/nova)+branch:stable/queens
14:20:32 <bauzas> but if that's all about filing a patch stuffed by some etherpad, I'm cool
14:20:39 <efried> mriedem: words?
14:20:44 * bauzas bails out for 20 mins (kids)
14:20:50 <mriedem> stable/stein is mostly flushed so we could release that soonish
14:21:08 <mriedem> i've gone through rocky and queens this week, +2ed what i can, but some of those are waiting for another core or stein to be released
14:23:17 <efried> #action stable core to flush rocky/queens
14:23:28 <efried> #topic Sub/related team Highlights
14:23:28 <efried> Placement (cdent)
14:23:48 <efried> #link latest pupdate http://lists.openstack.org/pipermail/openstack-discuss/2019-September/009160.html
14:24:03 <efried> guess that was 2w ago (but we didn't talk about it last week)
14:24:55 <efried> I think the highlights here are
14:24:56 <efried> #link consumer types in progress https://review.opendev.org/#/q/topic:bp/support-consumer-types
14:25:10 <efried> If someone in nova cares about using that ^ they could help review it.
14:25:12 <efried> and
14:25:23 <efried> tetsuro is the ussuri placement PTL \o/
14:25:41 <mriedem> with chris out this week and rc1 next week i'd think consumer types (also being low priority) is going to be deferred
14:26:05 <efried> I would think so as well
14:26:15 <efried> tetsuro: opinion ^ ?
14:26:46 <tetsuro> I'd like you to help, but pleasure to continue working with you
14:27:25 <tetsuro> s/you/openstackers
14:27:43 <efried> API (gmann)
14:28:13 <efried> I didn't see anything on the ML this week. There's probably not much to say at this stage in the release, when API changes should be either done or deferred, yah?
14:28:28 <efried> #topic Stuck Reviews
14:28:36 <efried> Any?
14:28:57 <efried> #topic Review status page
14:28:57 <efried> #link http://status.openstack.org/reviews/#nova
14:28:57 <efried> Count: 459 (-5); Top score: 2514
14:29:32 <efried> #topic Open discussion
14:29:32 <efried> Request you all to review this
14:29:32 <efried> #link Ignore root_gb for BFV in simple tenant usage API https://review.opendev.org/#/c/612626/
14:29:45 <mriedem> i left comments on that - same as in irc the other day
14:29:51 <mriedem> needs to be split up
14:29:56 <efried> k. Is this on the RC1 discussion list?
14:30:09 <mriedem> it's super latent, as in forever, behavior, so no
14:30:46 <mriedem> it adds an extra table join,
14:30:48 <mriedem> when pulling instances,
14:30:54 <mriedem> so the model query impacts there could be severe
14:31:08 <mriedem> so i'd really want someone more in the know on that stuff to review it in isolation - which is why i asked for it to be split up
14:31:34 <mriedem> tl;dr join between instances and bdms to determine if each instance in the result set is volume-backed
14:31:57 <dansmith> that
14:32:01 <dansmith> likely would suck
14:32:07 <mriedem> yeah,
14:32:19 <mriedem> and i've always been told that os-simple-tenant-usage performance already sucks,
14:32:31 <mriedem> and it's the first thing someone hits when logging into horizon
14:32:55 <efried> [Not landing in Train] should be made clear to shilpasd and bhagyashris, if it hasn't already. I get the impression (from their IRC queries and the fact this is on the meeting agenda) that they may still be thinking that's possible.
14:33:14 <mriedem> bhagyashris has been pushing it a long time
14:34:25 <efried> #link Specless blueprint: https://blueprints.launchpad.net/nova/+spec/mox-removal-ussuri
14:34:31 <efried> I'm just going to approve this
14:34:32 <takashin> Would you approve the blueprint?
14:34:44 <takashin> Thank you.
14:35:11 <efried> The remaining patches were +A a week ago, then we deferred them as being too close to RC1, so we should be able to land this and close it out early in ussuri
14:35:22 <efried> deltas since +A are just updating commit message with new bp name.
14:35:34 <takashin> Yes
14:35:51 <mriedem> efried: but you're not going to re-approve those for train right now right?
14:36:09 <efried> correct
14:36:15 <efried> wait until ussuri forks.
14:36:47 <efried> on the topic of ussuri
14:37:05 <efried> This isn't a fully formulated plan yet, but I want to put it in your heads.
14:37:23 <efried> I'd like to dramatically constrain the scope of the release in terms of how many blueprints we "approve"
14:37:52 <efried> on the theory that proposers would rather get a hard "no" right now than have NO IDEA whether their "approved" thing is going to land, and find out late in the release that it doesn't have a chance.
14:38:46 <mriedem> pretty sure we tried a version of that in newton/ocata,
14:38:52 <mriedem> by constraining deadlines
14:39:06 <mriedem> non-priority spec freeze/FF, priority spec freeze/FF
14:39:27 <efried> Counting up the number of approved-but-not-completed plus proposed-but-not-approved-before-spec-freeze blueprints from Train, and subtracting the handful that I don't think actually have any interest behind them, the number is eerily close to the number of blueprints we actually completed in Train. So I'm thinking of constraining roughly along those lines.
14:39:32 <mriedem> the end result is that cores are most busy 2 weeks before a deadline
14:39:46 <efried> well, that's what happened in train too
14:39:54 <mriedem> it's what has *always* happened
14:39:58 <mriedem> regardless of when we shift the goal posts
14:40:07 <efried> So we might as well be busy with less stuff for those two weeks.
14:40:23 <sean-k-mooney> efried: ya that last bit does not happen
14:40:36 <sean-k-mooney> you just end up being busy more often
14:40:53 <sean-k-mooney> we could try it again
14:40:58 <efried> anyway, if the dates-based experiment didn't succeed, perhaps a content-based experiment will have different results.
14:41:21 <mriedem> i guess it depends on what you consider as "having interest",
14:41:41 <mriedem> i know the spot instances stuff from cern has wide interest outside of cern and has been in production at cern for awhile,
14:41:42 <efried> different experiment sean-k-mooney. But agree, just because it didn't help once doesn't mean it couldn't be refined and/or retried and have better results.
14:41:52 <mriedem> but the guy pushing it is obviously busy with his day job
14:42:09 * bauzas is back
14:42:22 <mriedem> we also didn't have runways back when we did the strict granular deadlines thing, but i'm not sure how much runways are helping at this point either
14:42:38 <sean-k-mooney> mriedem: ya it would be a nice feature to have
14:42:53 <bauzas> just an open thought, do we feel we need to stick with cycle-with-rc ?
14:43:00 <bauzas> https://releases.openstack.org/reference/release_models.html
14:43:08 <efried> for the nova deliverable?
14:43:11 <bauzas> yup
14:43:16 <efried> What else would it be?
14:43:32 <sean-k-mooney> i think we need to be careful not to have serise in runways that touch the same code. the numa lm vpmem and pcpu code show that does not work well
14:43:41 <bauzas> there is cycle-with-intermediary, like swift
14:43:59 <sean-k-mooney> bauzas: that still requries a release per milestone apparently
14:44:02 <efried> bauzas: interesting idea, but how would that help?
14:44:06 <bauzas> but not worth engaging it now
14:44:11 <sean-k-mooney> at least that is what i was told for os-vif
14:44:17 <sean-k-mooney> but it coudl be worth trying
14:44:19 <bauzas> that said, I'm not sure we will have quota at the PTG
14:44:23 <bauzas> in terms of attendance
14:44:28 <mriedem> i do'nt think that would change anything
14:44:41 <gibi> my running theory is that we are bad at reviewing half done patch series which means long series are pushed close to the deadline
14:44:41 <mriedem> plus,
14:44:46 <bauzas> less milestone heckness IMHO
14:44:52 <dansmith> not unless the deployment projects support deploying the intermediary releases
14:44:57 <mriedem> what would cycle-with-intermediary mean for grenade testing?
14:45:01 <stephenfin> gibi: Yeah, big series are hard to review
14:45:02 <dansmith> some people install just swift, or separate from everything else
14:45:02 <efried> Yeah, that (PTG sparseness) is another reason constraining to already-approved/proposed will be helpful for ussuri. We ought to be able to resolve any issues via email pretty easily.
14:45:18 <dansmith> but unless OSA, tripleo, etc can deploy those releases, I don't think anyone is going to actually use them
14:45:21 <bauzas> dansmith: you're actually making a great point
14:45:49 <bauzas> I thought we could cope with intermediate releases that our deployment teams would consume
14:46:02 <bauzas> but if they aren't ready, the question is nonsense then
14:46:10 <dansmith> I think we'd still have to maintain big-to-big release compatibility,
14:46:26 <dansmith> which means there's less reason to use the minor releases
14:46:30 <dansmith> so it becomes overhead I think
14:46:50 <dansmith> it's an intermediate deadline, which might be valuable, but otherwise I'm not sure what it accomplishes
14:46:53 <bauzas> anyway, like I said, just an open thought, and not the right time for it
14:46:58 <sean-k-mooney> i think osa and kolla woudl be able to deploy the intermedira failrly simplery
14:47:21 <mriedem> would be able to and giving a shit about doing so are very different things
14:47:21 <dansmith> if we assigned approved blueprints to milestones we have today and just hard deferred things that didn't make it, that would be lighter but the same sort of deal
14:47:21 <bauzas> and I'm all up balancing the pros and cons
14:47:32 <dansmith> however, we push things past even the most important deadlines these days,
14:47:36 <dansmith> so I can't imagine we're going to really do that
14:48:05 <bauzas> dansmith: my thoughts come from the fact I would honestly try to decrease our pressure on those "deadlines" by releasing more often
14:48:06 <mriedem> waterfall, people. have you heard of it? i'm hearing people saying it's the best.
14:48:11 <bauzas> heh
14:48:21 <bauzas> we need to be agile.
14:48:25 <dansmith> bauzas: yeah, I just don't believe it's going to happen.. look at the shambles the current deadline is in :)
14:48:35 <bauzas> you're probably right
14:49:05 <bauzas> anyway, that's a bikeshed at the moment
14:49:14 <efried> anything else before we close?
14:49:20 <mriedem> efried: i'd say if you want to pursue the thought,
14:49:22 <mriedem> it's ML time
14:49:30 <bauzas> ++
14:49:34 <efried> yes, I'm planning to hit that next week some time.
14:49:41 <efried> after RC1
14:50:16 <efried> hopefully that doesn't leave too much time for new proposals to come in
14:50:59 <mriedem> i have some rmd specs...
14:51:16 <efried> I'm going to be on PTO today & tomorrow (family in town). Email if you need anything - I'll be checking frequently.
14:51:22 <efried> mriedem: too soon
14:51:34 <efried> #endmeeting