16:00:38 <bauzas> #startmeeting nova
16:00:38 <opendevmeet> Meeting started Tue Sep 13 16:00:38 2022 UTC and is due to finish in 60 minutes.  The chair is bauzas. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:38 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:38 <opendevmeet> The meeting name has been set to 'nova'
16:00:45 <bauzas> heh
16:00:47 <dansmith> o/
16:00:52 <bauzas> sorry, was finishing to update the agenda :)
16:01:02 <bauzas> you can raise hands if you want tho
16:01:07 <bauzas> or meh
16:01:14 <bauzas> bonjour everyone
16:01:17 <bauzas> #link https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting
16:01:46 <bauzas> ok, let's start, I'm pretty sure people who started earlier are not already gone
16:01:55 <bauzas> #topic Bugs (stuck/critical)
16:02:01 <bauzas> #info One Critical bug
16:02:07 <bauzas> #link https://bugs.launchpad.net/nova/+bug/1988311 Now u-c updated https://review.opendev.org/c/openstack/requirements/+/856044 shall we set to High or close the bug ?
16:02:12 <bauzas> #link https://bugs.launchpad.net/nova/+bug/1988311 Now u-c updated https://review.opendev.org/c/openstack/requirements/+/856044 shall we set to High or close the bug ?
16:02:17 <bauzas> so,
16:02:33 <bauzas> oslo.concurrency now is released with 5.0.1
16:02:39 <bauzas> upper-constraints are now updated
16:02:47 <bauzas> #link https://review.opendev.org/c/openstack/requirements/+/856044
16:03:26 <bauzas> I verified and both stable/yoga and master ask for oslo.concurrency older than 4.5.0
16:03:36 <bauzas> https://review.opendev.org/c/openstack/requirements/+/856044
16:03:41 <sean-k-mooney> if the locks are now fixed with the oslo release we could clsoe but we need to raise our min verion
16:03:52 <bauzas> so technically, I haven't checked but our gate uses the latest version
16:04:00 <sean-k-mooney> otherwise the problem still exstis and we might need to work around it on stable branches in nvoa
16:04:02 <bauzas> sean-k-mooney: correct, that's my point
16:04:26 <sean-k-mooney> gibi i assume we will backprot the eventlet fix in oslo
16:04:26 <bauzas> so we need to bump our mins for both yoga and master
16:04:28 <sean-k-mooney> to stable
16:04:41 <sean-k-mooney> we cant bump mins on stable
16:04:53 <sean-k-mooney> but we need to backport and ensure its allwoed by upper constratis
16:05:01 <sean-k-mooney> or we need to patch the lock in nova
16:05:05 <sean-k-mooney> and backprot that
16:05:13 <gibi> I assume we backport the fix in oslo
16:05:21 <sean-k-mooney> that would be my prefernce too
16:05:22 <gibi> but I did not have time to propose that
16:05:37 <bauzas> ok, at least I propose to set the bug to High
16:05:47 <bauzas> the gate isn't longer impact, our distros are tho
16:05:49 <sean-k-mooney> ack
16:05:54 <bauzas> impacted*
16:05:56 <sean-k-mooney> well its cloased for master
16:06:02 <sean-k-mooney> and high for the older brnaches
16:06:16 <bauzas> not in the Nova project
16:06:32 <bauzas> I can set a branch series for the bug
16:06:34 <sean-k-mooney> you want to keep it up for the min version bump
16:06:35 <bauzas> lemme do it
16:06:39 <bauzas> correct
16:06:39 <gibi> ack
16:06:40 <bauzas> so,
16:06:55 <bauzas> set to High, open a Yoga series to the bug report and set it to high too
16:07:06 <bauzas> and ask for the bump in master and the backport in Yoga
16:07:08 <bauzas> wfy ?
16:07:08 <sean-k-mooney> +1
16:07:23 <bauzas> ok, let me do the paperwork
16:07:33 <sean-k-mooney> well backport in oslo for yoga but yes
16:07:48 <gibi> looks OK
16:08:31 <bauzas> done
16:08:39 <bauzas> moving on
16:08:57 <bauzas> #link https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New 5 new untriaged bugs (-4 since the last meeting)
16:09:05 <bauzas> kudos to Uggla for this excellent work
16:09:10 <bauzas> #link https://storyboard.openstack.org/#!/project/openstack/placement 26 open stories (+0 since the last meeting) in Storyboard for Placement
16:09:16 <bauzas> #info Add yourself in the team bug roster if you want to help https://etherpad.opendev.org/p/nova-bug-triage-roster
16:09:24 <bauzas> Uggla: do you want to discuss any bug ?
16:09:41 <Uggla> bauzas, yes
16:10:18 <Uggla> bauzas, https://bugs.launchpad.net/nova/+bug/1896617 seems valid for me. I have triaged it.
16:10:40 <opendevreview> Balazs Gibizer proposed openstack/nova master: Bump min oslo.concurrencty to >= 5.0.1  https://review.opendev.org/c/openstack/nova/+/857491
16:10:49 <Uggla> But I would like your opinion
16:11:49 <bauzas> shit
16:12:07 <bauzas> yeah, if we set the perms
16:14:16 <sean-k-mooney> hum
16:14:25 <sean-k-mooney> so right now this sound like an rfe to me
16:15:06 <sean-k-mooney> or at least its a new restriction how what nova ia and is not allwoed to assume
16:15:24 <sean-k-mooney> normally we assume that libvirt and nova can share a group
16:16:37 <sean-k-mooney> we do not document that you can harden in this way
16:16:44 <sean-k-mooney> so to me its not explcitly supported
16:17:55 <bauzas> sorry, had a doorbell
16:18:00 <bauzas> I'm back
16:18:28 <bauzas> yeah, so maybe the bug is valid as a Wishlist ?
16:19:15 <bauzas> sean-k-mooney: agreed ?
16:19:57 <sean-k-mooney> i dont think its a bug
16:20:05 <sean-k-mooney> we could have it be whishlist yes
16:20:11 <sean-k-mooney> as a mini feature
16:20:14 <sean-k-mooney> im writign a responce
16:20:20 <sean-k-mooney> but i think the aswer is to
16:20:21 <bauzas> cool, can we move on then ?
16:20:31 <sean-k-mooney> chown nova:$libvirt-group
16:20:44 <sean-k-mooney> and then libvirt gets read via the group
16:20:49 <bauzas> if you really think this is a feature, we can set it to Invalid/Wishlist and ask for a blueprint to be filled
16:20:59 <bauzas> but that's tough for reporters
16:21:36 <bauzas> sean-k-mooney: yeah, ideally the group shall be libvirt but this sounds distro-oriented
16:21:39 <sean-k-mooney> ya i think so to as its is not a documented deployment model we support
16:21:50 <bauzas> problem is, we set perms
16:21:54 <sean-k-mooney> well it the "libvirt group" but that is disto dependet
16:21:56 <bauzas> so we're opiniated
16:22:06 <sean-k-mooney> we can have that be a config option
16:22:13 <bauzas> sean-k-mooney: yeah that's my point, this is distro-specific
16:22:31 <bauzas> I honestly don't like us to be opinionated about the perms we shall set
16:22:43 <bauzas> so, yeah a config knob seems maybe the best
16:22:51 <bauzas> or... I dunno
16:23:22 <bauzas> shall we just assume the tempdir is readable by anyone ?
16:23:28 <sean-k-mooney> we have some requirment around this for vhost-user by the way
16:23:37 <bauzas> not by *anyone*
16:24:09 <bauzas> but at least should we just document this is distro-specific and us having a requirement that the directory you set in nova.conf has right perms accordingly?
16:24:30 <bauzas> and remove this chmod in the code
16:24:45 <sean-k-mooney> well maybe not
16:24:45 <bauzas> this would require an upgrade relnote so a spec tho
16:25:03 <sean-k-mooney> i would have to look at the code but we can take this offline after the meeting
16:25:08 <bauzas> I don't like the idea of nova managing the underlying OS
16:25:15 <bauzas> cool
16:25:18 <Uggla> if I'm not wrong we are setting o+r, I don't really get the diff if we add o+g too.
16:25:48 <bauzas> Uggla: the point is that I don't see why Nova should do it
16:26:20 <sean-k-mooney> bauzas:
16:26:21 <Uggla> but it is already in the driver.py.
16:26:22 <bauzas> shalln't we just assume the directory are correctly readable by the required users ?
16:26:35 <sean-k-mooney> we assume tha tnova is in the libvirt group so that we can read the disk to upload the snapshot
16:26:58 <sean-k-mooney> in there case nova si not in the libvirt-qemu group i think
16:27:26 <sean-k-mooney> althoguh they apprently worked around this by addign libvirt-qemu user ot nova group
16:27:29 <sean-k-mooney> which is backwards
16:28:27 <bauzas> let's put this bug as Opinion then
16:28:42 <bauzas> this is half distro-specific, half a nova problem
16:29:10 <bauzas> and people wanting to discuss the resolution are more than happy to engage later on after the meeting
16:29:27 <bauzas> time flies and we're at the half of the meeting
16:29:39 <bauzas> agreed ?
16:29:54 <sean-k-mooney> ack
16:30:15 <Uggla> ack
16:31:08 <bauzas> moving on
16:31:09 <gibi> works for me
16:31:26 <bauzas> sean-k-mooney: do you have time to do some upstream bug triage this week ?
16:31:32 <bauzas> keeping in mind this is best-effort
16:31:52 <sean-k-mooney> i guess so
16:32:05 <Uggla> bauzas, fyi there are only 4 new bugs remaining now.
16:32:14 <bauzas> \o/
16:32:18 <bauzas> sean-k-mooney: thanks
16:32:27 <bauzas> and again, if you can't, no worries
16:34:29 <bauzas> #info bug baton is being passed to sean-k-mooney
16:34:31 <bauzas> damn
16:34:33 <bauzas> #info bug baton is being passed to sean-k-mooney
16:34:39 <bauzas> #topic Gate status
16:34:43 <bauzas> #link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs
16:34:47 <bauzas> #link https://zuul.openstack.org/builds?project=openstack%2Fplacement&pipeline=periodic-weekly Placement periodic job status
16:34:51 <bauzas> #link https://zuul.openstack.org/builds?job_name=tempest-integrated-compute-centos-9-stream&project=openstack%2Fnova&pipeline=periodic-weekly Centos 9 Stream periodic job status
16:34:56 <bauzas> #link https://zuul.opendev.org/t/openstack/builds?job_name=nova-emulation&pipeline=periodic-weekly&skip=0 Emulation periodic job runs
16:35:00 <bauzas> #info Please look at the gate failures and file a bug report with the gate-failure tag.
16:35:04 <bauzas> #info STOP DOING BLIND RECHECKS aka. 'recheck' https://docs.openstack.org/project-team-guide/testing.html#how-to-handle-test-failures
16:35:14 <bauzas> all runs are green, nothing to say
16:35:20 <bauzas> moving on ?
16:36:16 <bauzas> looks so
16:36:21 <bauzas> #topic Release Planning
16:36:25 <bauzas> #link https://releases.openstack.org/zed/schedule.html
16:36:29 <bauzas> #info RC1 is this Thursday
16:36:35 <bauzas> this is important ^
16:36:51 <sean-k-mooney> yep branches will be cut once rc1 is out
16:36:56 <bauzas> as a reminder, we'll branch stable/zed from this point in time
16:36:57 <gibi> I've just pushed the oslo.concurrency min bump https://review.opendev.org/c/openstack/nova/+/857491 that is important for RC!
16:37:00 <gibi> RC1
16:37:25 <bauzas> gibi: you give me a perfect opportunity for the next item
16:38:06 <bauzas> as a reminder too, after RC1 and before GA, we will only be able to accept stable/zed backports if and only if those are regression bugfixes
16:38:41 <bauzas> I have a couple of bugs waiting for reviews that help our vgpu state of art, but let's defer them to Antelope
16:38:58 <bauzas> s/bugs/bugfixes/
16:39:13 <bauzas> so those 2 days are crucial for any bugfix requring to be merged
16:39:28 <bauzas> accordingly, I created an etherpad for tracking
16:39:33 <bauzas> #link https://etherpad.opendev.org/p/nova-zed-rc-potential Zed RC tracking etherpad
16:39:43 * sean-k-mooney man we are expeded to write bugfixes and bugs so un reasonable :P
16:39:52 <bauzas> this etherpad will be used for tracking RC1 prep, and other RCs until Zed GA
16:40:07 <sean-k-mooney> have you created the bug lable yet
16:40:13 <bauzas> please take a look on it, as I beg for reviewers
16:40:15 <sean-k-mooney> in launchpad
16:40:16 <bauzas> reviews*
16:40:28 <bauzas> sean-k-mooney: you mean the stable tag ?
16:40:30 <bauzas> yes
16:41:17 <bauzas> #info people can tag launchpad bug reports with zed-rc-potential tag if they consider the bugfix important for RC1 or later (if regression fix)
16:41:19 <sean-k-mooney> i mean zed-rc-potential tag yes
16:41:32 <bauzas> #link https://bugs.launchpad.net/nova/+bugs?field.tag=zed-rc-potential
16:41:37 <bauzas> all of this is in the etherpad
16:41:46 <bauzas> and accordingly,
16:41:59 <bauzas> gibi: I just add your https://review.opendev.org/c/openstack/nova/+/857491 to the list of RC1 needed patches
16:42:09 <gibi> I've already added
16:42:20 <gibi> L43
16:42:43 <bauzas> perfect
16:42:46 <sean-k-mooney> i just made it an offical tag too so it will come up when you type
16:43:01 <bauzas> sean-k-mooney: oh good point, I forgot this cycle
16:43:11 <bauzas> thanks
16:43:18 <bauzas> so, we have a couple of important patches
16:43:36 <bauzas> for those which are from the release team, there is a ETD
16:43:42 <bauzas> which is Friday
16:43:59 <bauzas> if we don't say anything, those will be merged as it is
16:44:06 <bauzas> so, reviews.
16:44:29 <bauzas> I'll ping a couple of cores those two days to ensure we're in a good shape before we branch
16:44:37 <gibi> ack
16:45:11 <bauzas> the placement and nova zed branch patches are already there :)
16:45:19 <bauzas> if you don't wanna freak out :)
16:45:38 <bauzas> but I'm not happy with the SHA1 so I'll officially -1 the nova one (cc elodilles ;) )
16:45:51 <bauzas> ok, now, I have one question
16:46:09 <elodilles> bauzas: saw that, thanks!
16:46:14 <bauzas> with https://review.opendev.org/c/openstack/nova/+/855706, sean-k-mooney raised a good point about the min compute version we shall support for Zed
16:46:27 <bauzas> for the moment, we say Xena
16:46:29 <elodilles> (and waiting for a response on the placement patch :))
16:46:39 <bauzas> in https://github.com/openstack/nova/blob/master/nova/objects/service.py#L233
16:46:44 <sean-k-mooney> yep it should by Yoga
16:46:46 <sean-k-mooney> https://github.com/openstack/governance/blob/master/resolutions/20220210-release-cadence-adjustment.rst#proposed-solution
16:47:01 <sean-k-mooney> """Y->A should be a "dress rehearsal" where we have the jobs enabled to help smoke out any issues, but where hard guarantees are not yet made."""
16:47:19 <bauzas> sean-k-mooney: takashi made a patch accordingly
16:47:46 <opendevreview> Merged openstack/os-vif stable/zed: Update .gitreview for stable/zed  https://review.opendev.org/c/openstack/os-vif/+/856780
16:47:46 <bauzas> #link https://review.opendev.org/c/openstack/nova/+/856895
16:48:16 <bauzas> sean-k-mooney: agreed on the resolution so we then need to merge https://review.opendev.org/c/openstack/nova/+/856895 before RC1
16:48:45 <bauzas> sean-k-mooney: I personally feel the rpc alias and the min version bump shall continue to be separate patches
16:48:54 <bauzas> so I'll +2 boht
16:49:06 <sean-k-mooney> ok
16:49:15 <sean-k-mooney> if they are both merged for rc then im ok with that
16:49:37 <sean-k-mooney> i should drop my -1 then on the other patch
16:50:07 <sean-k-mooney> i would prefer to do these together in general but either works
16:50:25 <bauzas> sean-k-mooney: previously we kept them separately
16:50:35 <gibi> I have to review those
16:50:54 <sean-k-mooney> i set review priorty on both
16:51:00 <bauzas> as ideally, rpc aliases should be merged just before RC1 while min version bumps shall be merged at the beginning of the release
16:51:05 <sean-k-mooney> ill look at them agin later and droped my -1
16:51:29 <sean-k-mooney> im not convice thats the case
16:51:42 <sean-k-mooney> or that they need to be seperate
16:52:04 <bauzas> I'll update the PTL guide to reflect this
16:52:13 <bauzas> and sean-k-mooney, you're up to debate in such patch
16:52:23 <sean-k-mooney> we had some discussion about if we shoudl be recording the min or max verison supproted by a release by the way
16:52:25 <bauzas> (in the PTL guide patch, I mean)
16:52:36 <bauzas> moving on
16:52:47 <bauzas> #topic PTG planning
16:52:51 <bauzas> #link https://etherpad.opendev.org/p/nova-antelope-ptg Antelope PTG etherpad
16:52:53 <bauzas> as a reminder ^
16:52:58 <bauzas> #link https://ptg.opendev.org/ptg.html PTG schedule
16:53:03 <bauzas> I made the bookings as agreed ^
16:53:07 <opendevreview> Merged openstack/os-vif stable/zed: Update TOX_CONSTRAINTS_FILE for stable/zed  https://review.opendev.org/c/openstack/os-vif/+/856781
16:53:22 <bauzas> #topic Review priorities
16:53:27 <bauzas> #link https://review.opendev.org/q/status:open+(project:openstack/nova+OR+project:openstack/placement+OR+project:openstack/os-traits+OR+project:openstack/os-resource-classes+OR+project:openstack/os-vif+OR+project:openstack/python-novaclient+OR+project:openstack/osc-placement)+(label:Review-Priority%252B1+OR+label:Review-Priority%252B2)
16:53:45 <bauzas> nothing to say about those
16:54:00 <bauzas> feel free to engage on review-prio patches during the day
16:54:06 <bauzas> #topic Stable Branches
16:54:09 <sean-k-mooney> we should add rc bugs to the review priortes
16:54:15 <bauzas> elodilles: your 5-min turn
16:54:20 <bauzas> sean-k-mooney: I can do it
16:54:25 <elodilles> ack
16:54:32 <elodilles> #info stable/yoga seems to be blocked by openstacksdk-functional-devstack job
16:54:48 <elodilles> the py39 issue was resolved,
16:55:05 <elodilles> but it seems openstacksdk-functional-devstack fails now instead :/
16:55:13 <opendevreview> Merged openstack/python-novaclient stable/zed: Update .gitreview for stable/zed  https://review.opendev.org/c/openstack/python-novaclient/+/856788
16:55:15 <opendevreview> Merged openstack/python-novaclient stable/zed: Update TOX_CONSTRAINTS_FILE for stable/zed  https://review.opendev.org/c/openstack/python-novaclient/+/856789
16:55:22 <elodilles> JayF called our attention yesterday about it
16:55:22 <bauzas> damn
16:55:44 <elodilles> i had only a quick look and it fails on other projects gate as well, so it is not just nova
16:55:53 <elodilles> beyond this
16:55:56 <elodilles> #info stable/stein (and older) are blocked: grenade and other devstack based jobs fail with the same timeout issue as stable/train was previously
16:56:03 <elodilles> #info stable branch status / gate failures tracking etherpad: https://etherpad.opendev.org/p/nova-stable-branch-ci
16:56:12 <elodilles> the usual things ^^^
16:56:16 <elodilles> that's it
16:56:39 <bauzas> elodilles: fancy writing an email to the world explaining the gate blocker ?
16:56:53 <elodilles> bauzas: yepp, i can do that
16:57:03 <bauzas> I just hope someone will just magically wake up and say he'll look
16:57:27 <bauzas> elodilles: all projects being impacted by the sdk, you said ?
16:57:42 <elodilles> not all
16:57:49 <elodilles> but i've seen some others
16:57:51 <sean-k-mooney> is that the storage cleanpu failure
16:57:57 <sean-k-mooney> in the functional job
16:58:03 <sean-k-mooney> where its failing to delete the volume
16:58:09 <sean-k-mooney> becuase the snapshot is not deleted
16:58:29 <sean-k-mooney> ah it is test_block_storage_cleanup
16:58:30 <elodilles> yepp, test_block_storage_cleanup
16:58:42 <sean-k-mooney> ya so they trided to fix it by doing it in a loop twice
16:58:55 <sean-k-mooney> but that obviouly does not work
16:59:08 <sean-k-mooney> so the test is flaky
16:59:14 <elodilles> oh, so it's a known issue :-o
16:59:18 <bauzas> => remove the flakey test then
16:59:27 <sean-k-mooney> ill see if i can get the patch
16:59:32 <bauzas> and unblock the gate
16:59:44 <sean-k-mooney> we cant currently filter those
16:59:48 <sean-k-mooney> via the job config
17:00:25 <bauzas> anyway, we're on time
17:00:36 <sean-k-mooney> efoley and i https://review.opendev.org/c/openstack/openstacksdk/+/856509 tried to fix it by backporting that
17:00:47 <sean-k-mooney> but then we realised it always uses master
17:00:49 <bauzas> let's wrap this meeting and continue the stable/yoga discussion right after if you want
17:00:59 <bauzas> #topic Open discussion
17:01:01 <bauzas> nothing
17:01:02 <sean-k-mooney> so https://review.opendev.org/c/openstack/openstacksdk/+/852992 was ment to fix it but does not
17:01:14 <JayF> Might be worth noting for meeting notes purposes that folks running Ironic back as far as train should update their Nova checkout
17:01:22 <JayF> as all known fixes for the Ironic driver have been backported
17:01:25 <opendevreview> Merged openstack/osc-placement stable/zed: Update .gitreview for stable/zed  https://review.opendev.org/c/openstack/osc-placement/+/856784
17:01:26 <opendevreview> Merged openstack/osc-placement stable/zed: Update TOX_CONSTRAINTS_FILE for stable/zed  https://review.opendev.org/c/openstack/osc-placement/+/856785
17:01:45 <JayF> (and I know for em branches we don't do releases)
17:01:58 <bauzas> JayF: want me to offically set an #info ?
17:01:59 <bauzas> :)
17:02:08 <JayF> bauzas: if you think anyone who cares would read it :D
17:02:29 <sean-k-mooney> elodilles: i just opened a revert https://review.opendev.org/c/openstack/openstacksdk/+/857471
17:02:52 <sean-k-mooney> the https://review.opendev.org/c/openstack/openstacksdk/+/852992 fixes an issue and intoduces the flaky test
17:02:58 <sean-k-mooney> so maybe a partial revert
17:03:11 <sean-k-mooney> keep the fix but revert the new test caes
17:03:15 <bauzas> #info Ironic operators running train or later are more than welcome to upgrade their nova checkout with latest stable releases since bugfixes are released now
17:03:24 <bauzas> JayF: ^ happy ? :)
17:03:28 <JayF> thank you :D
17:03:31 <elodilles> sean-k-mooney: ack, thanks for the info! i'll look at them then
17:03:51 <gibi> (it feels like we have to parallel meeting both overrun its time)
17:03:53 <bauzas> JayF: people reading our notes are beasts I don't know
17:03:57 <gibi> *two
17:04:12 <bauzas> gibi: well, I'll offload you some task
17:04:16 <bauzas> thanks all
17:04:20 <bauzas> #endmeeting