16:00:10 <bauzas> #startmeeting nova
16:00:10 <opendevmeet> Meeting started Tue Nov 16 16:00:10 2021 UTC and is due to finish in 60 minutes.  The chair is bauzas. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:10 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:10 <opendevmeet> The meeting name has been set to 'nova'
16:00:18 <gibi> o/
16:00:21 <elodilles> o/
16:00:28 <bauzas> #link https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting
16:00:48 <bauzas> good 'day, everyone ;)
16:00:52 <whoami-rajat> Hi
16:00:56 <opendevreview> Merged openstack/nova-specs master: Integration With Off-path Network Backends  https://review.opendev.org/c/openstack/nova-specs/+/787458
16:01:11 <gmann> o/
16:01:38 <bauzas> I'll have to hardstop working in 45-ish mins, sooo
16:01:42 <bauzas> #chair gibi
16:01:42 <opendevmeet> Current chairs: bauzas gibi
16:01:44 <bauzas> sorry again
16:01:53 <gibi> so I will take the rest
16:01:57 * bauzas is  a taxi
16:02:13 <bauzas> anyway, let's start
16:02:20 <bauzas> #topic Bugs (stuck/critical)
16:02:26 <bauzas> #info No Critical bug
16:02:33 <bauzas> #link https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New 28 new untriaged bugs (+3 since the last meeting)
16:02:38 <bauzas> #help Nova bug triage help is appreciated https://wiki.openstack.org/wiki/Nova/BugTriage
16:02:43 <bauzas> I'm really a sad panda
16:03:07 <bauzas> in general, I'm triaging bugs on Tuesday, but I forgot about our today's spec review day :)
16:03:17 <bauzas> so I'll look at the bugs tomorrow
16:03:38 <bauzas> in case people want to help us, <3
16:03:52 <bauzas> any bug to discuss ?
16:04:26 <bauzas> #link https://storyboard.openstack.org/#!/project/openstack/placement 33 open stories (+1 since the last meeting) in Storyboard for Placement
16:04:30 <bauzas> about thisq.$
16:04:34 <bauzas> this... *
16:04:49 <bauzas> I tried to find which story was new :)
16:05:15 <bauzas> but the last story was already the one I knew
16:05:20 <bauzas> so, in case people know...
16:05:49 <dansmith> o/
16:06:12 <gibi> bauzas: if at some point I have time I can try to dig but I pretty full at the moment
16:06:12 <bauzas> also, Storyboard is a bit... slow, I'd say
16:06:46 <bauzas> 5 secs at least everytime it takes for looking about a story
16:07:24 <bauzas> I mean, for stories, maybe we should use Facebook then ? :p
16:07:35 <bauzas> (heh, :p )
16:07:53 * bauzas was joking in case people were not knowing
16:08:18 <bauzas> OK, this looks like a bad joke
16:08:20 <bauzas> moving on :p
16:08:30 <bauzas> #topic Gate status
16:08:36 <bauzas> #link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs
16:08:47 <bauzas> nothing new
16:08:58 <bauzas> #link https://zuul.openstack.org/builds?project=openstack%2Fplacement&pipeline=periodic-weekly Placement periodic job status
16:09:24 <bauzas> now placement-nova-tox-functional-py38 job works again :)
16:09:26 <bauzas> thanks !
16:09:55 <bauzas> #topic Release Planning
16:10:04 <bauzas> #info Yoga-1 is due Nova 18th #link https://releases.openstack.org/yoga/schedule.html#y-1
16:10:17 <bauzas> which is in 2 days
16:10:27 <bauzas> nothing really to say about it
16:10:32 <bauzas> #info Spec review day is today
16:10:52 <bauzas> I think I reviewed all the specs but one (but I see this one was merged ;) )
16:11:14 <bauzas> thanks for all who already reviewed specs
16:11:28 <gibi> yeah I think we pushed forward all the open specs
16:12:01 <whoami-rajat> Sorry If I'm interrupting but I had one doubt regarding my spec
16:12:04 <bauzas> we merged 3 specs today
16:12:41 <bauzas> whoami-rajat: no worries, we can discuss this spec if you want during the open discussion topic
16:12:52 <whoami-rajat> ack thanks bauzas
16:12:55 <bauzas> whoami-rajat: but what is your concern ?
16:13:07 <bauzas> a tl;dr if you prefer
16:14:03 <bauzas> for other specs, I'll mark the related blueprints accepted in Launchpad by tomorrow
16:14:21 <whoami-rajat> bauzas, so I'm working on the reimage spec for volume backed instances and we decided to send connector details with the reimage API call and cinder will do the attachment update (this was during PTG), Lee pointed out that we should follow our current mechanism of nova doing attachment update like we do for other operations
16:15:00 <bauzas> ok, if this is a technical question, let's discuss this during the open discussion topic as I said
16:15:26 <whoami-rajat> sure, np
16:15:42 <bauzas> ok, next topic then
16:15:46 <bauzas> #topic Review priorities
16:15:52 <bauzas> #link  https://review.opendev.org/q/status:open+(project:openstack/nova+OR+project:openstack/placement)+label:Review-Priority%252B1
16:16:06 <bauzas> #info https://review.opendev.org/c/openstack/nova/+/816861 bauzas proposing a documentation change for helping contributors to ask for reviews
16:16:16 <bauzas> gibi already provided some comments on it
16:16:58 <bauzas> I guess the concern is how to help contributors to ask for reviews priorities like we did with the etherpad
16:17:29 <bauzas> but if we have a consensus saying that it is not an issue, I'll stop
16:18:02 <bauzas> but my only concern is that I think asking people to come on IRC and ping folks is difficult so we could use gerrit
16:19:09 <gibi> what is more difficult? Finding the reason of a faul in nova code and fixing it or joing IRC to ask for review help?
16:19:46 <sean-k-mooney> well one you "might" be able to do offlien/async
16:20:01 <sean-k-mooney> the other invovles talking to peopel albeit by text
16:20:25 <sean-k-mooney> unfortunetly those are sometime non overlaping skill sets
16:20:32 <bauzas> gibi: I'm just thinking of on and off contributors that just provide bugfixes
16:20:33 <gibi> doing code review is talking to people via text :)
16:21:12 <bauzas> but let's continue discussing this in the proposal, I don't wanna drag the whole attention by now
16:21:19 <sean-k-mooney> bauzas: for one off patches i think the expectaion shoudl still be on use to watch the patchs come in and help them
16:21:28 <sean-k-mooney> rather ten assuemign they will use any tools we provide
16:21:49 <bauzas> sean-k-mooney: yeah but then how to discover them ?
16:22:08 <bauzas> either way, let's discuss this by Gerrit :p
16:22:10 <sean-k-mooney> well if its a similar time zone i watch for teh irc bot commeting for the patches
16:22:24 <sean-k-mooney> if i dont recognise it or the name i open it
16:22:49 <sean-k-mooney> and then one of use can request the reqview priority in gerrirt or publicise the patch to others
16:23:15 <bauzas> that's one direction
16:23:46 <sean-k-mooney> if there is something in gerrit i can set im happy to do that on patches when i think they are ready otherwise ill just ping them to ye as i do now
16:23:55 <bauzas> either way, we have a large number of items for the open discussion topic, so let's move on
16:24:01 <sean-k-mooney> ack
16:24:09 <bauzas> #topic Stable Branches
16:24:23 <bauzas> elodilles: fancy copy/pasting or do you want me to do so ?
16:24:35 <elodilles> either way is OK :)
16:25:42 <bauzas> I can do it
16:25:44 <bauzas> #info stable gates' status look OK, no blocked branch
16:25:50 <bauzas> #info final ussuri nova package release was published (21.2.4)
16:25:55 <bauzas> #info ussuri-em tagging patch is waiting for final python-novaclient release patch to merge
16:26:00 <bauzas> #link https://review.opendev.org/c/openstack/releases/+/817930
16:26:05 <bauzas> #link https://review.opendev.org/c/openstack/releases/+/817606
16:26:09 <bauzas> #info intermittent volume detach issue: afaik Lee has an idea and started to work on how it can be fixed:
16:26:14 <bauzas> #link https://review.opendev.org/c/openstack/tempest/+/817772/
16:26:19 <bauzas> any question ?
16:26:25 <elodilles> thanks :)
16:27:40 <bauzas> looks like none
16:27:47 <bauzas> #topic Sub/related team Highlights
16:27:47 <gibi> the volume detach issue feel more an more like not related to detach
16:27:51 <bauzas> #undo
16:27:51 <opendevmeet> Removing item from minutes: #topic Sub/related team Highlights
16:28:05 <gibi> the kernel panic happens before we issue detach
16:28:07 <elodilles> gibi: true
16:28:21 <gibi> it is either related to the attach or the live migration itself
16:28:50 <gibi> I have trials placing sleep in different places to see where we are too fast https://review.opendev.org/c/openstack/nova/+/817564
16:28:52 <bauzas> which stable branches are impacted ?
16:28:57 <gibi> stable/victoria
16:29:01 <bauzas> ubuntu focal-ish I guess ?
16:29:20 <bauzas> ack thanks
16:29:21 <elodilles> (and other branches as well, but might be different root causes)
16:29:45 <gibi> I only see kernel panic in stable/victoria (a lot) and one single failure in stable/wallaby
16:30:08 <gibi> so if there are detach issues in older stable that is either not causing kernel panic, or we don't see the panic in the logs
16:30:33 <bauzas> I guess kernel versions are different between branches
16:30:41 <bauzas> right?
16:31:06 <bauzas> could we imagine somehow to verify another kernel version for stable/victoria
16:31:06 <bauzas> ?
16:31:06 <gibi> we tested with guest cirros 0.5.1 (victoria default) and 0.5.2 (master default) it is reproducible with both
16:31:23 <bauzas> ack so unrelated
16:31:31 <gibi> there is a summary here https://bugs.launchpad.net/nova/+bug/1950310/comments/8
16:32:19 <bauzas> #link https://bugs.launchpad.net/nova/+bug/1950310/comments/8 explaining the guest kernel panic related to stable/victoria branch
16:32:22 <sean-k-mooney> ya the fiew cases i looked at with you last week were all happing befoer detach
16:32:38 <sean-k-mooney> so its either the attach or live migration
16:32:44 <gibi> sean-k-mooney: I have more logs in the runs of https://review.opendev.org/c/openstack/nova/+/817564 if you are interested
16:32:52 <sean-k-mooney> i looked downstream at our qemu bugs but didnt see anythign relevent
16:33:09 <sean-k-mooney> gibi: sure ill try and take a look proably tomorrow
16:33:15 <sean-k-mooney> but ill open it in a tab
16:33:57 <gibi> sean-k-mooney: thanks, I will retrigger that patch for a couple times to see if the current sleep before the live migration helps
16:34:24 <bauzas> a good sleep always helps
16:34:39 <bauzas> :)
16:34:45 <elodilles> :]
16:34:46 <sean-k-mooney> when sleep does not work we can also try a trusty print statement
16:35:09 <gibi> sleep is not there as a solution but as a troubleshooting to see which step we are too fast :D
16:35:24 * sean-k-mooney is dismayed by how may race condition __dont__ appeare when you use print for debugging
16:35:33 <gibi> and I do have a lot of print(server.console) like statements in the tempest :D
16:36:10 <sean-k-mooney> i think we can move on but its good you were able to confirm we were attaching before the kerenl finished booting
16:36:22 <sean-k-mooney> at least in some cases
16:36:57 <sean-k-mooney> that at least lend weight to the idea we are racing
16:37:05 <bauzas> ok, let's move on
16:37:12 <gibi> ack
16:37:12 <bauzas> again, large agenda todayu
16:37:17 <bauzas> #topic Sub/related team Highlights
16:37:23 <bauzas> damn
16:37:24 <bauzas> #topic Sub/related team Highlights
16:37:40 <bauzas> Libvirt : lyarwood ?
16:38:15 <bauzas> I guess nothing to tell
16:38:19 <bauzas> moving on to the last topic
16:38:31 <bauzas> #topic Open discussion
16:39:02 <bauzas> whoami-rajat: please queue
16:39:09 <whoami-rajat> thanks!
16:39:19 <bauzas> (kashyapc) Blueprint for review: "Switch to 'virtio' as the default display device" -- https://blueprints.launchpad.net/nova/+spec/virtio-as-default-display-device
16:39:27 <bauzas> this is a specless bp ask
16:39:37 <bauzas> kashyap said " The full rationale is in the blueprint; in short: "cirrus"  display device has many limitations and is "considered harmful"[1] by  QEMU graphics maintainers since 2014."
16:40:00 <bauzas> do we need a spec for this bp or are we OK for approving it by now ?
16:40:19 <whoami-rajat> so lyarwood had a concern with my reimage spec, we agreed to pass the connector info to reimage API (cinder) and cinder will do attachment update and return the connection info with events payload
16:40:22 <gibi> I think we don't need a spec this is pretty self contained in the libvirt driver
16:40:23 <bauzas> kashyap was unable to attend the meeting today
16:40:27 <whoami-rajat> (in PTG)
16:40:38 <sean-k-mooney> i think we are ok with approving it the main thing to call out is we will be chaing it for existing isntnace too
16:40:39 <bauzas> whoami-rajat: please hold, sorry
16:40:43 <whoami-rajat> oh ok
16:40:45 <gibi> the only open question we had with sean-k-mooney is how to change the default
16:41:05 <gibi> but kashyap tested it out that changing the default during hard reboot not cause any trouble to guests
16:41:16 <gibi> as the new video dev has a fallback vga mode
16:41:17 <bauzas> gibi: I'm thinking hard of any potential upgrade implication
16:41:32 <sean-k-mooney> right so when we dicussed this before we decied to change it only for new instances to avoid upgrade issue
16:41:50 <bauzas> correct
16:41:57 <sean-k-mooney> our downstream QE tested this with windows guests and linux guest and both seamd to be ok with the change
16:42:01 <bauzas> I'm in favor of not touching the running instances
16:42:11 <bauzas> or asking to rebuild them
16:42:18 <gibi> we are not toching the running instance, we only touch hard rebooting instances
16:42:22 <sean-k-mooney> so kasyap has impletne this for all instnaces
16:42:49 <bauzas> gibi: which happens when you stop/start, right?
16:42:54 <gibi> right
16:42:54 <sean-k-mooney> bauzas: yes as gibi says it will only take effect when the xml is next regenreted
16:42:59 <gibi> it happens while the guest is not running
16:43:21 <gibi> it is not an unplug/plug for a running guest
16:43:29 <bauzas> do we want admins to opt-in instances ?
16:43:41 <bauzas> or do we agree it would be done automatically?
16:43:42 <sean-k-mooney> it will happen on start/stop hard reboot or a non live move operations
16:44:03 <gibi> bauzas: I trust kashyap that it is safe to change this device
16:44:14 <bauzas> do we also want to have a nova-status upgrade check for yoga about this ?
16:44:23 <sean-k-mooney> no
16:44:24 <bauzas> gibi: me too
16:44:29 <sean-k-mooney> why would we need too
16:44:35 <sean-k-mooney> we are not removing support for cirrus
16:44:36 <gibi> we don't remove cirros
16:44:41 <sean-k-mooney> jsut not the default
16:44:46 <gibi> yepp
16:45:00 <sean-k-mooney> gibi: context is downstream it is being remvoed form rhel 9
16:45:04 <bauzas> sean-k-mooney: sure, that just means that long-living instances could continue running cirros
16:45:14 <sean-k-mooney> so wwe need to care about it for our product
16:45:24 <sean-k-mooney> actully cirrus is not beeing remvoed in rhel 9
16:45:35 <sean-k-mooney> but like in rhel 10
16:46:06 <sean-k-mooney> bauzas: yep which i think is ok
16:46:27 <sean-k-mooney> we coudl have a nova status check but it woudl have to run on the compute nodes
16:46:35 <sean-k-mooney> which is kind of not nice
16:46:40 <sean-k-mooney> since it woudl have to check the xmls
16:46:51 <bauzas> I know
16:46:53 <sean-k-mooney> so i woudl not add it personally
16:47:19 <bauzas> I'm just saying that we enter a time that could last long
16:47:20 <gibi> I agree, we don't need upgrade check
16:48:16 <sean-k-mooney> shal we continue this in the patch review
16:48:17 <bauzas> but agreed on the fact this is not a problem until cirros support is removed and this is not an upstream question
16:48:31 <bauzas> sean-k-mooney: you're right, nothing needing a spec
16:49:06 <bauzas> #agreed https://blueprints.launchpad.net/nova/+spec/virtio-as-default-display-device is accepted as specless BP for the Yoga release timeframe
16:49:10 <bauzas> moving on
16:49:12 <gibi> \o/
16:49:17 <bauzas> next item
16:49:30 <bauzas> (kashyapc) Blueprint for review: "Add ability to control the memory used by fully emulated QEMU guests -- https://blueprints.launchpad.net/nova/+spec/control-qemu-tb-cache
16:49:39 <bauzas> again, a specless bp ask
16:49:54 <bauzas> he said " This blueprint allows us to configure how much memory a  plain-emulated (TCG) VM, which is what OpenStack CI uses.  Recently,  QEMU changed the default memory used by TCG VMs to be much higher, thus  reducing the no. of VMs you TCG could run per host.  Note: the libvirt  patch required for this  will be in libvirt-v7.10.0 (December 2021)."
16:49:59 <bauzas> " See this issue for more details: https://gitlab.com/qemu-project/qemu/-/issues/693 (Qemu increased memory usage with TCG)"
16:50:21 <sean-k-mooney> im a little torn on this
16:50:41 <sean-k-mooney> im not sure i like this being a per host config option
16:50:51 <sean-k-mooney> but its also breaking existing deployemnts
16:51:04 <sean-k-mooney> so we cant really adress that with flavor extra specs or iamge properties
16:51:17 <sean-k-mooney> sicne it would be a pain for operators to use
16:51:17 <gibi> but that requires rebuild of existing instances
16:51:21 <sean-k-mooney> yep
16:51:32 <sean-k-mooney> so with that in mind the config option proably is the way to go
16:51:51 <sean-k-mooney> just need to bare in mind it might chagne after a hard reboot if you live migrate
16:51:53 <gibi> yeah, config as a first step, if later more fine grained control is needed we can add an extra_spec
16:52:24 <bauzas> there are libvirt dependencies
16:52:28 <sean-k-mooney> if we capture the (this should really be the same on all host in a region) pice in the docs im ok with this
16:52:37 <sean-k-mooney> bauzas: and qemu deps
16:52:37 <bauzas> you need a recent libvirt in order to be able to use it
16:52:43 <bauzas> right
16:52:45 <sean-k-mooney> its only supproted on qemu 5.0+
16:52:45 <gibi> sean-k-mooney: yeah that make sense to document
16:53:05 <sean-k-mooney> so we will need a libvirt verion and qemu check in the code
16:53:12 <sean-k-mooney> which is fine we know how to do that
16:53:15 <bauzas> so, if this is a configurable, this has to explain which versions you need
16:53:23 <sean-k-mooney> yep
16:53:33 <bauzas> we would expose something unusable for the most
16:53:50 <sean-k-mooney> the only tricky bit will be live migration
16:54:02 <sean-k-mooney> if the dest is not new enough but the host is
16:54:04 <bauzas> correct, the checks ?
16:54:16 <sean-k-mooney> we will need to make sure we validate that
16:54:23 <bauzas> right
16:54:36 <bauzas> but this looks to me an implementation detail
16:54:50 <bauzas> all of this seems not needing a spec, right?
16:54:59 <bauzas> upgrade concerns are N/A
16:55:10 <bauzas> as you explicitely need a recent qemu
16:55:18 <sean-k-mooney> am the live migration check will be a littel complex but other then that i dont see a need for a spec
16:55:41 <sean-k-mooney> im a littel concerned about the livemgration check which is what makes me hesitate to say no spec
16:55:45 <bauzas> we can revisit this decision if the patch goes hairy
16:55:53 <sean-k-mooney> yes
16:55:55 <sean-k-mooney> that works for mee
16:56:25 <gibi> works for me too
16:56:36 <sean-k-mooney> i htink we have the hypervior version avaible in the conductor so i think we can do it without an rpc/object change
16:56:36 <bauzas> #agreed https://blueprints.launchpad.net/nova/+spec/control-qemu-tb-cache can be a specless BP but we need to know more about the live migration checks before we approve
16:56:51 <bauzas> gibi: sean-k-mooney: works for you what I wrote ?
16:57:04 <sean-k-mooney> +1
16:57:08 <bauzas> ok,
16:57:13 <bauzas> next topic is ganso
16:57:20 <bauzas> and eventually, whoami-rajat
16:57:27 <ganso> hi!
16:57:35 <bauzas> ganso: you have one min :)
16:57:50 <ganso> so my question is about adding hw_vif_multiqueue_enabled setting to flavors
16:57:56 <ganso> it was removed from the original spec
16:57:57 <ganso> https://review.opendev.org/c/openstack/nova-specs/+/128825/comment/7ad32947_73515762/#90
16:58:06 <ganso> today it can be only used in image properties
16:58:26 <ganso> does it make at all semantically or is this something that only makes sense as an image property?
16:58:32 <sean-k-mooney> ya this came up semi recently
16:58:38 <sean-k-mooney> i think we can just add this in the flavor
16:58:49 <bauzas> the other way would be a concern to me
16:58:59 <ganso> ok. Would this require a spec?
16:59:00 <bauzas> as users could use a new property
16:59:20 <sean-k-mooney> well image propertise are for exposing thing that affect the virtualised hardware
16:59:24 <bauzas> but given we already accept this for images, I don't see a problem with accepting it as a flavor extraspec
16:59:30 <sean-k-mooney> so in gnerally you want that to be user setable
17:00:00 <ganso> great
17:00:07 <bauzas> sean-k-mooney: right, I was just explaning that image > flavor seems not debatable while flavor > image seems to be discussed
17:00:16 <ganso> to me it sounds simple enough to not require a spec, do you agree?
17:00:37 <bauzas> good question
17:00:45 <bauzas> but we're overtime
17:00:49 <sean-k-mooney> https://blueprints.launchpad.net/nova/+spec/multiqueue-flavor-extra-spec
17:01:04 <sean-k-mooney> this is the implemation https://review.opendev.org/q/topic:bp/multiqueue-flavor-extra-spec
17:01:04 <bauzas> ganso: whoami-rajat: let's continue discussing your concerns after the meeting
17:01:10 <bauzas> #endmeeting