16:00:02 <bauzas> #startmeeting nova
16:00:02 <opendevmeet> Meeting started Tue Apr 25 16:00:02 2023 UTC and is due to finish in 60 minutes.  The chair is bauzas. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:02 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:02 <opendevmeet> The meeting name has been set to 'nova'
16:00:13 <bauzas> hey everyone
16:00:17 <elodilles> o/
16:00:21 <dansmith> o/
16:00:22 <bauzas> we are quite busy today so let's try to be quick
16:00:26 <ykarel> o/
16:00:33 <auniyal> o/
16:00:37 <bauzas> #link https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting
16:00:46 <bauzas> #topic Bugs (stuck/critical)
16:00:51 <bauzas> #info No Critical bug
16:00:54 <bauzas> #link https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New 17 new untriaged bugs (+4 since the last meeting)
16:01:00 <bauzas> #info Add yourself in the team bug roster if you want to help https://etherpad.opendev.org/p/nova-bug-triage-roster
16:01:20 <bauzas> honestly, I think I forgot to tell melwitt to look at the bugs, so it's on me :)
16:01:46 <bauzas> the next person in the roster is artom
16:01:55 <artom> ohhai
16:01:57 <bauzas> artom: fancy triaging some upstream bugs if you like ?
16:02:05 <artom> Sure
16:02:10 <bauzas> thanks a lot
16:02:17 <artom> CLOSED RUSTWILLFIXIT
16:02:19 <bauzas> I'll also try to cherry-pick some
16:02:42 <bauzas> artom: well, we're on Launchpad
16:03:15 <bauzas> so it'd be 'Closed' only with a comment saying you'd think Rust would work
16:03:20 <bauzas> or even 'Wontfix'
16:03:22 <bauzas> anyway
16:03:28 <bauzas> #info bug baton is being passed to artom
16:03:32 <bauzas> #topic Gate status
16:03:41 <bauzas> grab your popcorns
16:03:47 <bauzas> #link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs
16:03:52 <bauzas> #link https://etherpad.opendev.org/p/nova-ci-failures
16:03:54 <bauzas> but alas,
16:04:17 <bauzas> #link https://lists.openstack.org/pipermail/openstack-discuss/2023-April/033454.html Gate is blocked
16:04:46 <bauzas> last status : https://lists.openstack.org/pipermail/openstack-discuss/2023-April/033468.html
16:05:03 <bauzas> dansmith: sean-k-mooney: elodilles: wanting to add anything else on that ?
16:05:18 <dansmith> probably not
16:05:30 <sean-k-mooney> we are mainly waiting on ci
16:05:36 <sean-k-mooney> so no i think that fine
16:05:42 <elodilles> i've just updated the nova patches according to the discussions (if i did not miss anything)
16:05:51 <bauzas> we have two concurrent patches afaics https://review.opendev.org/c/openstack/nova/+/881409 and https://review.opendev.org/c/openstack/nova/+/881339
16:05:53 <elodilles> if those will be the chosen ones :)
16:06:07 <bauzas> we may want to only use one :)
16:06:19 <dansmith> yeah
16:06:51 <sean-k-mooney> elodilles: your missing the depend on agains the devstack patch but those need to be squashed first
16:06:51 <bauzas> anyway, we'll sort that out of the meeting
16:06:53 <sean-k-mooney> so its fine for now
16:07:21 <elodilles> ++
16:07:55 <bauzas> yup and thanks to dansmith and sean-k-mooney for working hardly on the ceph patches
16:08:04 <sean-k-mooney> mainly dansmith
16:08:05 <bauzas> -EOLDGREEK to me
16:08:18 <sean-k-mooney> i have been busy with other things other then checking back every now and then
16:08:27 <bauzas> (I mean, I know what ceph does and all the things, but that is a different story)
16:08:42 <sean-k-mooney> anywya we can move on
16:08:46 <bauzas> cool
16:08:51 <bauzas> https://zuul.openstack.org/builds?project=openstack%2Fnova&project=openstack%2Fplacement&pipeline=periodic-weekly
16:09:01 <bauzas> unsurprisingly they are all green
16:09:15 <bauzas> but by their runs we didn't had the py39 updates :)à
16:09:31 <sean-k-mooney> good timing i guess
16:10:31 <bauzas> well, tooz upgraded to 4.0 the week before
16:10:32 <opendevreview> Merged openstack/nova stable/xena: Fix rescue volume-based instance  https://review.opendev.org/c/openstack/nova/+/875343
16:10:46 <bauzas> so that's fortunate we haven't merged the u-c patch during the weekend
16:10:58 <bauzas> anyway, moving on
16:11:04 <bauzas> #info Please look at the gate failures and file a bug report with the gate-failure tag.
16:11:09 <bauzas> #info STOP DOING BLIND RECHECKS aka. 'recheck' https://docs.openstack.org/project-team-guide/testing.html#how-to-handle-test-failures
16:11:14 <bauzas> #topic Release Planning
16:11:18 <bauzas> #link https://releases.openstack.org/bobcat/schedule.html
16:11:23 <bauzas> #info Nova deadlines are set in the above schedule
16:11:28 <bauzas> #info Bobcat-1 is in 2 weeks
16:11:51 <bauzas> (we'll have a stable branch review day on Bobcat-1 but I'll explain this the next week)
16:12:02 <bauzas> #topic Review priorities
16:12:09 <bauzas> #link https://review.opendev.org/q/status:open+(project:openstack/nova+OR+project:openstack/placement+OR+project:openstack/os-traits+OR+project:openstack/os-resource-classes+OR+project:openstack/os-vif+OR+project:openstack/python-novaclient+OR+project:openstack/osc-placement)+(label:Review-Priority%252B1+OR+label:Review-Priority%252B2)
16:12:14 <bauzas> #info As a reminder, cores eager to review changes can +1 to indicate their interest, +2 for committing to the review
16:12:19 <bauzas> #topic Stable Branches
16:12:28 <bauzas> elodilles: take the mic
16:12:43 <elodilles> yepp
16:12:53 <elodilles> beyond the usual stuff,
16:13:04 <elodilles> (unblocked gate & many rechecks)
16:13:19 <elodilles> auniyal prepared release patches for yoga and zed
16:13:30 <elodilles> they haven't merged yet
16:13:53 <elodilles> and meanwhile 1-1 patches merged to stable/yoga and zed
16:14:07 <auniyal> yes, got 1 +2 yet :), thanks for review elodilles
16:14:12 <elodilles> otherwise they are good as they are
16:14:26 <elodilles> auniyal: thanks for proposing the patches :)
16:14:41 <bauzas> cool
16:15:54 <bauzas> anything else ?
16:16:04 <elodilles> nope, i think that was all
16:16:07 <elodilles> from my side
16:16:15 <elodilles> sorry :)
16:17:10 <bauzas> cool, moving on
16:17:19 <bauzas> #topic Open discussion
16:17:29 <bauzas> (ykarel) Allow to add tb-cache size libvirt option for qemu, context https://bugs.launchpad.net/nova/+bug/1949606
16:17:33 <bauzas> ykarel: go for it
16:19:20 <ykarel> hi
16:19:41 <ykarel> so qemu-5.0.0(included in ubuntu jammy) update default tb cache size to 1 GiB(from 32 MiB) for system-emulated guest vms and with that each guest VM is using much more system memory(1 GB+ memory for guest), resulting into oom-kill issues when creating multiple guest vms concurrently in neutron scenario jobs using ubuntu guest vms
16:20:17 <ykarel> libvirt-8.0.0 added an option to configure it per guest vm
16:20:25 <ykarel> Currently testing WIP nova patch https://review.opendev.org/c/openstack/nova/+/868419 in https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/881391
16:20:46 <ykarel> should this be a specless RFE or can continue as a bug?
16:20:54 <sean-k-mooney> ykarel: did you file the specless pluprint i asked for
16:20:56 <sean-k-mooney> ah
16:20:58 <sean-k-mooney> its not a bug
16:21:19 <ykarel> sean-k-mooney, i just added to meeting agenda to get clearity
16:21:29 <ykarel> but if it's a way to go then i will file it
16:21:30 <sean-k-mooney> we might conider it to be a small enough workaround that we might want to backport it upstream
16:21:33 <sean-k-mooney> to enable ci testing
16:22:27 <ykarel> hmm mainly would be needed for releases running on jammy
16:22:30 <sean-k-mooney> the issue is that i dont think 22.04 has libvirt 8.0.0
16:22:37 <ykarel> it has
16:22:44 <ykarel> that's what we testing
16:22:54 <sean-k-mooney> ah ok
16:23:07 <bauzas> we damn need update of https://docs.openstack.org/nova/latest/reference/libvirt-distro-support-matrix.html
16:23:19 <sean-k-mooney> yes
16:23:30 <sean-k-mooney> so your patch is mising a min libvirt version check
16:23:33 <bauzas> we're still having libvirt 6.0.0 as a min, right?
16:23:35 <sean-k-mooney> so that will need to be added
16:23:40 <ykarel> from job logs libvirt-daemon                        8.0.0-1ubuntu7.4
16:23:55 <sean-k-mooney> bauzas: yes and im hoping to bump that to 7.0.0 this release
16:23:58 <ykarel> sean-k-mooney, yes will include that in next update
16:24:10 <sean-k-mooney> but we will sitll need to check 8.0.0 supprot
16:24:15 <sean-k-mooney> ack
16:24:16 <bauzas> sean-k-mooney: but we would still need to only set it if libvirt >=8
16:24:32 <bauzas> do we want to schedule on it, or is it just a performance fix ?
16:24:34 <sean-k-mooney> ya either only set it or preferable check this in init-host
16:24:43 <sean-k-mooney> and raise an error if you set the config option on an old libvirt
16:24:58 <bauzas> will there be any config knob ?
16:25:07 <sean-k-mooney> yes there shoudl be
16:25:12 <ykarel> yes that's second question
16:25:14 <sean-k-mooney> to set the cache size
16:25:15 <ykarel> Default setting for this new option, unconfigured or set to defaults like 32MB or 128 MB etc?
16:26:07 <bauzas> ykarel: if that's a config option, that indeed needs to be an hard fail if the operators sets this config value
16:26:18 <bauzas> and libvirt isn't recent enough
16:26:23 <sean-k-mooney> its not somethign we should hardcode
16:26:28 <ykarel> bauzas, sure will take care that in the patch
16:26:29 <bauzas> for that reason, I'm not happy with a default value except none
16:26:47 <sean-k-mooney> so it shoudl eb a config option im ok with a low default or leaving it unset
16:27:02 <bauzas> I'd prefer it unset for upgrade reasons
16:27:04 <ykarel> okk Thanks will keep it like that no default and let user configure it
16:27:05 <sean-k-mooney> bauzas: to your poitn it woudl be nice to have a triat for this but not required
16:27:28 <bauzas> ykarel: s/user/operator but I think I get your point
16:27:50 <ykarel> so will propose the blueprint and update the patch with as per all the suggestions, Thanks
16:27:50 <sean-k-mooney> in this case really it will be the zuul job
16:28:01 <ykarel> yes
16:28:03 <sean-k-mooney> this is of interest to peopel usign qemu for emulation
16:28:06 <bauzas> sean-k-mooney: man, we could recommend the operators to provide custom traits for this, exactly like vgpu types
16:28:36 <bauzas> I mean, eventually all the computes will support that, right?
16:28:44 <sean-k-mooney> yes
16:28:53 <bauzas> after a couple of releases, once we cap libvirt to >=8
16:29:10 <sean-k-mooney> so its not goign to break live migration
16:29:13 <bauzas> so I'm not a big fan of adding some scheduling thing for something that will eventually be supported mid-term
16:29:24 <sean-k-mooney> because we do not allwo live migration from a newer to older microversion
16:29:36 <sean-k-mooney> and for cold migration we will regenerate the xml on the approiate host
16:29:41 <sean-k-mooney> based on what it has avaiable
16:29:43 <bauzas> s/microversion/libvirt version but yeah
16:29:55 <sean-k-mooney> so i dont think we need anythign sepcial here
16:30:01 <sean-k-mooney> so ya custom trait woudl eb fine with me
16:30:13 <sean-k-mooney> they can use provider.yaml to set that if they want
16:30:17 <bauzas> cool, so that only seems a config knob to add, a check on init_host to fail if set and some magic in the driver to enable it
16:30:30 <bauzas> amirite ?
16:30:34 <sean-k-mooney> more or less
16:30:48 <bauzas> then, I'm OK for specless
16:30:56 <sean-k-mooney> +docs test ectra but its effectivly self contaied in the libvirt driver + the config tweak
16:30:58 <bauzas> we had precedents
16:31:11 <bauzas> + a relnote obviously
16:31:24 <bauzas> ykarel: do you agree with the direction ?
16:31:30 <ykarel> bauzas, yes
16:31:31 <sean-k-mooney> im ok with specless too assuming all of the above are done
16:31:41 <bauzas> anyone disagreeing ?
16:32:07 <bauzas> looks not
16:32:20 <ykarel> Thanks folks \o/
16:32:41 <sean-k-mooney> the only other thin i would suggest is once this is done a devstack patch shoudl be added to set this by default to say 32mb if qemu is used instead of kvm
16:33:37 <sean-k-mooney> thats out of scope fo this meeting and coud be doen on a per job basis too
16:33:39 <bauzas> #agreed enabling tb cache seems a specless blueprint, provided it only adds a config knob defaulting to unset, init_host failing on an older libvirt and just libvirt config tweak
16:34:01 <bauzas> #action ykarel to ping bauzas once the blueprint is created so that he can approve it
16:34:34 <bauzas> ykarel: and yeah, the scope of this feature can include devstack change and testing, for sure
16:34:53 <bauzas> like we could enable it in nova-next
16:35:41 <sean-k-mooney> bauzas: it will reduce the memory pressue in all our jobs so ocne we know it does not have a negitive impact we will proably want to have it enable din all of them
16:35:42 <bauzas> anything else to add on this item ?
16:35:55 <sean-k-mooney> but we can take it slow like the mariadb reduced memroy
16:36:00 <bauzas> sean-k-mooney: oh yeah, but I'd be in favor of testing it first
16:36:08 <bauzas> yah
16:36:33 <bauzas> ok, fwiw, I have another item
16:36:53 <bauzas> (bauzas) Can https://blueprints.launchpad.net/nova/+spec/cold-migrate-to-host-policy be specless ?
16:37:05 <bauzas> tl;dr: we discussed this at the PTG
16:37:29 <sean-k-mooney> assuming there is no change in the default policy then yes i think so
16:37:32 <bauzas> operators want a better granularity and maybe change the cold-migrate action to be admin_or_owner
16:37:46 <bauzas> but, here, we're just adding a new policy which is admin-only
16:37:55 <bauzas> so no API change, and no policy changes
16:38:11 <bauzas> it will just go check a separate policy if host is set
16:38:32 <bauzas> (literally a one-liner patch besides the policy file)
16:38:55 <bauzas> any objections to have it specless ?
16:38:55 <sean-k-mooney> so basicaly there will be two poicies now for cold migration one for migratoin with a host adn one without but admin only by default
16:39:01 <sean-k-mooney> and then operators can choose
16:39:18 <sean-k-mooney> +1
16:39:37 <bauzas> correct, like we have for os_compute_api:servers:create:forced_host
16:40:10 <bauzas> except I won't change the defaut rule for os_compute_api:os-migrate-server:migrate
16:40:34 <bauzas> there will be os_compute_api:os-migrate-server:migrate and os_compute_api:os-migrate-server:migrate:host
16:40:34 <bauzas> both being admin-only
16:40:48 <bauzas> (and operators can decide to open os_compute_api:os-migrate-server:migrate to endusers)
16:41:10 <bauzas> so I reiterate, any objection to have it specless ?
16:41:40 <bauzas> looks not
16:41:49 <bauzas> if so,
16:42:08 <sean-k-mooney> as long ast there is at least a blueprint im happy. i dislike changing policy without any tracker to works for me
16:42:17 <bauzas> #agreed https://blueprints.launchpad.net/nova/+spec/cold-migrate-to-host-policy accepted as a specless feature for Bobcat
16:42:31 <bauzas> sean-k-mooney: there is a blueprint, and there will be a relnote
16:42:45 <bauzas> and there will be functional tests covering this
16:42:53 <sean-k-mooney> yep all good.
16:43:15 <bauzas> I don't think we need a tempest change, do you think it's a nice to have ?
16:43:29 <sean-k-mooney> i dont think tempst shoudl test non default policy
16:43:41 <bauzas> yeah, that was my question
16:43:48 <bauzas> I'm not a QA expert
16:43:55 <bauzas> tempest is branchless
16:44:05 <bauzas> so that would be a bit hard to test it with tempest
16:44:22 <sean-k-mooney> i suspect you could reuse some fo the exsiting test with the right config if you needed too
16:44:24 <bauzas> anyway, I think we're done on this
16:44:44 <bauzas> about tempest, we could discuss this on the review time
16:44:50 <bauzas> thanks folks
16:44:51 <opendevreview> Artom Lifshitz proposed openstack/nova master: Reproduce bug 1995153  https://review.opendev.org/c/openstack/nova/+/862967
16:44:51 <opendevreview> Artom Lifshitz proposed openstack/nova master: Save cell socket correctly when updating host NUMA topology  https://review.opendev.org/c/openstack/nova/+/862964
16:45:01 <bauzas> any other item to add before we end the meeting ?
16:45:22 <auniyal> small thing o/
16:45:27 <auniyal> CI on yoga: this one keep failing for different reasons, mostly
16:45:27 <auniyal> https://review.opendev.org/c/openstack/nova/+/839922
16:45:28 <bauzas> shot
16:45:42 <auniyal> mostly volume tests
16:46:23 <sean-k-mooney> ya that kind fo a pain im not sure there is anythin we can do beyond recheck
16:46:29 <sean-k-mooney> is it the volume detach tests
16:46:45 <sean-k-mooney> auniyal: gibi found some tests that are not waiting properly
16:46:47 <bauzas> I think this is also tracked on the stable CI failures etherpad
16:46:50 <auniyal> yes, attach and detach , but they are always different
16:47:04 <auniyal> sometime tomeout
16:47:15 <sean-k-mooney> yoga is not EM right so its still using tempest master?
16:47:20 <auniyal> no
16:47:46 <auniyal> tbc no, its not EM
16:47:50 <sean-k-mooney> ok
16:47:58 <sean-k-mooney> so it still can get tempest fixes if we fix those tests
16:48:09 <bauzas> yup
16:48:40 <bauzas> are we done ?
16:48:59 <auniyal> sorry I didn't get, any action on above
16:49:20 <auniyal> we need to fix tempest tests ?
16:49:47 <sean-k-mooney> i think just continue to reheck it. gibi found at least on test that is not waiting for sshable
16:49:53 <bauzas> no, we have some tempest patches up
16:49:57 <sean-k-mooney> and notice other dont appear to eb waiting but i dont have the context
16:50:01 <bauzas> and yoga would benefit from those
16:50:10 <bauzas> since tempest is branchless
16:50:14 <sean-k-mooney> oh do you have a link?
16:50:16 <auniyal> ack thanks
16:51:54 <bauzas> I was referring to gibi's recent discoveries of testing gap for ssh wait
16:52:13 <bauzas> (sorry was looking at the -tc meeting)
16:52:23 <bauzas> -tc chan*
16:52:35 <bauzas> can we close this meeting now ?
16:52:38 <sean-k-mooney> its fine we can wrap this here and chat after
16:53:06 <bauzas> cool
16:53:08 <bauzas> thanks all
16:53:12 <bauzas> #endmeeting