16:00:15 <bauzas> #startmeeting nova
16:00:15 <opendevmeet> Meeting started Tue Nov 23 16:00:15 2021 UTC and is due to finish in 60 minutes.  The chair is bauzas. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:15 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:15 <opendevmeet> The meeting name has been set to 'nova'
16:00:25 <gibi> o/
16:00:29 <elodilles> o/
16:00:32 <bauzas> #link https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting
16:00:37 <bauzas> good day, 'vyone
16:00:37 <lyarwood> \o
16:01:37 <bauzas> ok, let's start
16:01:43 * kashyap waves
16:02:04 <bauzas> a few folks could be off b/c of some holiday they have on Thursday :)
16:02:16 <bauzas> not French , tho
16:02:28 <bauzas> #topic Bugs (stuck/critical)
16:02:33 <bauzas> #info No Critical bug
16:02:39 <bauzas> #link https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New 29 new untriaged bugs (+1 since the last meeting)
16:02:50 <bauzas> I did a bit of triage and will continue on it tomorrow
16:03:17 <bauzas> thanks for the people who did this too
16:03:22 <bauzas> #help Nova bug triage help is appreciated https://wiki.openstack.org/wiki/Nova/BugTriage
16:03:35 <bauzas> #link https://storyboard.openstack.org/#!/project/openstack/placement 33 open stories (+0 since the last meeting) in Storyboard for Placement
16:03:42 <bauzas> any bug to discuss by now ?
16:04:21 <bauzas> #topic Gate status
16:04:29 <bauzas> #link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs
16:04:45 <bauzas> no new gate bug AFAICS
16:05:03 <bauzas> btw. we could triage three of them that are NEW :)
16:05:13 <bauzas> (even 4)
16:05:30 <bauzas> #link https://zuul.openstack.org/builds?project=openstack%2Fplacement&pipeline=periodic-weekly Placement periodic job status
16:05:42 <bauzas> we have an issue
16:05:57 <bauzas> #info tempest-integrated-placement job had a CI issue
16:06:13 <bauzas> #link https://zuul.openstack.org/build/43f380b91bd14579806d0b9be0d702c8 the job itself
16:06:44 <bauzas> looks like an unrelated package issue
16:06:49 <gibi> it is a known issue https://zuul.openstack.org/build/43f380b91bd14579806d0b9be0d702c8/log/job-output.txt#4206
16:07:02 <gibi> the pmlogger service did not start
16:07:09 <kashyap> What does the service even do?
16:07:10 <gibi> we see it time to time in nova jobs too
16:07:20 <gibi> kashyap: never digged into it
16:07:26 <kashyap> Is it this? - https://man7.org/linux/man-pages/man1/pmlogger.1.html
16:07:38 <kashyap> Anyway, don't want to distract from the main point :)
16:08:05 <bauzas> #agreed this is a known issue related to pmlogger, should be automatically fixed next run
16:08:13 <bauzas> agreed on the agreed ? ^
16:08:14 <gibi> pmlogger.service - Performance Metrics Archive Logge
16:08:25 <gibi> so yes
16:08:39 <gibi> bauzas: sure
16:08:48 <bauzas> ok, if this is done, let's move on
16:08:56 <bauzas> #info Please look at the gate failures, file a bug, and add an  elastic-recheck signature in the opendev/elastic-recheck repo (example: https://review.opendev.org/#/c/759967)
16:09:06 <bauzas> any other gate issue to mention ?
16:10:07 <bauzas> looks not
16:10:13 <bauzas> #topic Release Planning
16:10:23 <bauzas> Yoga-1 was Nova 18th#link https://releases.openstack.org/yoga/schedule.html#y-1
16:10:27 <bauzas> dang
16:10:37 <bauzas> link https://releases.openstack.org/yoga/schedule.html#y-1
16:10:43 <bauzas> Yoga-1 was Nova 18th
16:10:50 <bauzas> #info Spec review day helped to merge 3 specs
16:11:06 <bauzas> thanks people who reviewed them
16:12:34 <bauzas> #info at Yoga-1 timeframe, Nova already accepted 7 specs and 4 specless BPs
16:12:44 <bauzas> (sorry, was counting)
16:13:06 <bauzas> anything to mention about the pace we have ?
16:13:50 <bauzas> #info reminder that we plan another spec review day mid-Dec, exact date to be discussed in a next meeting
16:14:03 <bauzas> moving on ?
16:14:40 <bauzas> I guess so
16:14:46 <bauzas> #topic Review priorities
16:14:56 <bauzas> #link https://review.opendev.org/q/status:open+(project:openstack/nova+OR+project:openstack/placement)+label:Review-Priority%252B1
16:15:02 <bauzas> #link https://review.opendev.org/c/openstack/nova/+/816861 bauzas proposing a documentation change for helping contributors to ask for reviews
16:15:11 <bauzas> that reminds me I need to update this change
16:15:52 <bauzas> #action bauzas to update his change to reflect the consensus about non-ownership ask for this label
16:16:42 <bauzas> but then, I'll ask next week what and how people feel about how to asynchronously ask reviewers (cores and non-cores) to look at changes
16:16:53 <bauzas> or the other way, how reviewers can discover changes
16:17:47 <bauzas> we have room for contributors happy to review that we can help them how to do this
16:18:19 <gibi> bauzas: btw, can we remove review priority from https://review.opendev.org/c/openstack/nova/+/810220 it is in -1 since 5th of Nov
16:19:06 <bauzas> gibi: fair enough, you provided good comments on it, and I didn't added my ones since I agree with you
16:19:16 <gibi> and I have the same feeling about https://review.opendev.org/c/openstack/nova/+/810849 too
16:19:27 <bauzas> even if I'd like this bugfix to be merged somehow
16:19:54 <gibi> and this too https://review.opendev.org/c/openstack/nova/+/803713
16:21:22 <bauzas> #info we removed the Review-Priority flag from a few changes as they were lacking updates
16:21:48 <bauzas> I added a comment to each of the changes explaining to ping me again once they revise the change
16:22:07 <bauzas> gibi: thanks for the cleanup
16:22:33 <bauzas> I should somehow think about how to periodically check this
16:22:59 <bauzas> that's it for the RP labelling and usage ?
16:24:32 <bauzas> I guess so
16:24:37 <bauzas> #topic Stable Branches
16:24:44 <bauzas> elodilles: the mic is yours
16:24:52 <elodilles> #info stable gates are not blocked
16:24:58 <bauzas> woow
16:25:00 <elodilles> #info Ussuri transitioned to Extended Maintenance for nova projects \o/
16:25:09 <elodilles> and that's it i think
16:25:25 <elodilles> i have no updates for the intermittent failures on stable gates :(
16:25:47 <bauzas> elodilles: don't wanna release any stable branch as we're around yoga-1 ?
16:26:04 <elodilles> good question
16:26:25 <bauzas> we could do a xena .z release
16:26:33 <elodilles> i haven't checked how much new content we have on stable branches
16:26:35 <bauzas> not sure if people want it tho
16:26:48 <bauzas> I'm pretty sure gibi would like a release, nope?
16:26:53 <bauzas> given of the backports
16:27:14 <bauzas> or do people want to stack a bit more until we release ?
16:27:28 <elodilles> i can prepare some if there is need :)
16:29:04 <gibi> bauzas: no hard push on a release
16:29:07 <bauzas> I have no personal interest beyond curiosity
16:29:20 <bauzas> ok, we can wait them
16:29:23 <bauzas> then*
16:29:41 <gibi> bauzas: the waiting for plug fix needed on victora / pike for my downstream counterpart
16:29:43 <elodilles> (i don't see much merged content)
16:29:54 <opendevreview> Dan Smith proposed openstack/nova master: Revert project-specific APIs for servers  https://review.opendev.org/c/openstack/nova/+/816206
16:30:23 <bauzas> gibi: well, for us, wallaby and train, but whatever
16:30:44 <gibi> yeah probably train too
16:30:49 <bauzas> I guess we're done with this topic, we can rediscuss about the opportunity to release a bit later in the cycle
16:31:44 <elodilles> yes, let's discuss later
16:32:00 <bauzas> ++
16:32:08 <bauzas> #topic Sub/related team Highlights
16:32:14 <bauzas> Libvirt (lyarwood)
16:32:19 <lyarwood> We are tracking a QEMU 6.1.0 regression downstream with [libvirt]num_pcie_ports >= 15 when using the q35 machine type, we aren't hitting it yet upstream in Nova itself as we only use q35 in nova-next but TripleO is seeing it with centos 8 and 9 stream after a QEMU rebase. The workaround is to lower the port count to <=14. Just a heads up for now, I've got a TODO to create a known issue release note in Nova for the time being while we
16:32:19 <lyarwood> wait on a fix in QEMU.
16:32:57 <bauzas> ouch.
16:33:00 <lyarwood> and that's all I have this week
16:33:27 <sean-k-mooney> ack so noting to do in nova other then the known issue
16:33:35 <bauzas> ack, I also saw some bad libvirt modification that creates problems for our vgpu support
16:33:39 <sean-k-mooney> and perhaps change job config if needd
16:33:50 <lyarwood> sean-k-mooney: ack pretty much
16:33:55 <bauzas> https://bugs.launchpad.net/nova/+bug/1951656
16:34:27 <bauzas> the fix is easy but that makes me worried about the fact that the namings we have are not a public API
16:35:06 <sean-k-mooney> its not the first time we have been broken by such name changes
16:35:45 <bauzas> can't imagine what would happen if the drivers can also change their mdev type names
16:36:04 <bauzas> we're a bit fragile
16:36:08 <bauzas> anyway
16:36:12 <bauzas> let's not overdiscuss this
16:36:26 <bauzas> we have a few specless bps requests again this week
16:38:10 <bauzas> #info QEMU 6.1.0 regression downstream with [libvirt]num_pcie_ports >= 15 requires for the moment to work around by lowering the port count to <=14. lyarwood will provide a relnote for it until the QEMU regression is fixed
16:38:27 <bauzas> moving on
16:38:31 <bauzas> #topic Open discussion
16:38:35 <bauzas> #topic Open discussion
16:38:44 <bauzas> (dasp) Blueprint for review: "Make no_compression_image_types configurable" -- https://blueprints.launchpad.net/nova/+spec/configurable-no-compression-image-types
16:38:49 <bauzas> dasp: around ?
16:38:57 <bauzas> Rationale: hardcoded behavior slows down cold migrations with local RAW disk images to 8 mbps and maxes CPU.  It might be a good idea to change the default value as well or disable compression for all types.
16:39:58 <dansmith> this is compression of just the stream?
16:41:04 * bauzas reads the blueprint description
16:41:24 <dansmith> I assume the original thought is that raw disks *might* be mostly zeroes which compress to nothing during transfer,
16:41:38 <dansmith> but that probably falls apart over time and makes it compression for no reason
16:41:41 <bauzas> that's my general assumption
16:42:06 <sean-k-mooney> for ssh
16:42:09 <dansmith> ideally libvirt would just do hole detection and not transfer the holes, I think, but...
16:42:10 <sean-k-mooney> it just adds -C
16:42:19 <sean-k-mooney> for rsync it enable rsync compression
16:42:20 <dansmith> sean-k-mooney: ah, right
16:42:35 <dansmith> well, I see no reason to prevent people from choosing this, so seems okay to me
16:42:40 <bauzas> oh the ssh transfer itself ?
16:42:47 <sean-k-mooney> yes
16:42:56 <bauzas> I guess the proposal is to make it configurable ?
16:43:01 <dansmith> yeah
16:43:10 <bauzas> but that would be for all images of the same type
16:43:18 <dasp> I'm here
16:43:18 <dansmith> default to the same behavior it looks like, so no change unless you want change
16:43:21 <bauzas> (I guess)
16:43:37 <sean-k-mooney> its this https://github.com/openstack/nova/blob/50fdbc752a9ca9c31488140ef2997ed59d861a41/nova/virt/libvirt/volume/remotefs.py#L192-L193
16:43:50 <bauzas> dansmith: if this is opt-in for uncompress, that's OK
16:43:55 <dansmith> bauzas: right
16:44:07 <bauzas> dansmith: if that's changing the default to *not* compress raw images, I'm -1
16:44:14 <sean-k-mooney> so the current patch if they have not reviesed it maintianed the exsting behavior
16:44:19 <dansmith> bauzas: no, it's keeping the same default, just allowing people to control it
16:44:34 <sean-k-mooney> and i was suggesting we would not change the default without more dicussion
16:44:47 <bauzas> then it looks to me a simple configurable ask, which doesn't require a spec provided they don't break existing behaviour
16:44:49 <sean-k-mooney> although both rsync and ssh recommend it only for slow connections
16:45:09 <dasp> yes, changing default is not in scope right now as it comes from me. However, based on my tests, the default behavior is very bad unless I'm doing something unusually wrong.
16:45:37 <dasp> So I'm likely going to propose another future spec to change the default separately.
16:45:44 <sean-k-mooney> dasp: right so for now i think we shoudl add the option and consider changing the default after we get some operator input
16:45:53 <bauzas> what sean-k-mooney said
16:45:56 <dansmith> I dunno if discard will cause qemu to zero sections of a raw disk, but if not, over time your raw disk of zeroes becomes not that, so compression becomes useless after a point
16:45:57 <bauzas> expose the new knob
16:46:02 <bauzas> let operators play with it
16:46:17 <bauzas> and give us figures about why this is helpful to change the default
16:47:54 <bauzas> anyone having concerns about https://blueprints.launchpad.net/nova/+spec/configurable-no-compression-image-types NOT being a specless BP ?
16:48:36 <bauzas> no API modifications, no upgrade concerns, no DB modifications, just a configurable add
16:48:53 <sean-k-mooney> this is ok to me so no objections
16:48:59 <dansmith> yup
16:49:00 <bauzas> looks like nobody is freaking out
16:49:21 <bauzas> #agreed https://blueprints.launchpad.net/nova/+spec/configurable-no-compression-image-types is approved as a specless BP, provided no change in default behaviour
16:49:30 <bauzas> next one
16:49:38 <bauzas> (jhartkopf) Update user data spec (https://review.opendev.org/c/openstack/nova-specs/+/816542)
16:49:43 <bauzas> jhartkopf: around ?
16:49:51 <bauzas> are you just asking for spec review ?
16:50:07 <jhartkopf> around
16:50:35 <bauzas> oh, I remember this one
16:50:51 <bauzas> that's an interesting case
16:50:52 <sean-k-mooney> i think jhartkopf  wanted to expand on there usecase
16:51:01 <bauzas> right
16:51:11 <jhartkopf> We have already discussed this spec extensively, but I’d like to focus on a specific use case now.
16:51:38 <jhartkopf> We think the change would make sense when using Cloudbase-init plugins, which can run at every boot and configures guest settings based on user data
16:51:56 <jhartkopf> We already brought that up, but still think this would be a valid use case
16:52:21 <sean-k-mooney> cloud-init support that too by the way its just not how its typically used
16:52:32 <bauzas> jhartkopf: IIRC, your usecase was for ssh key management, right?
16:52:50 <sean-k-mooney> i thikn that was someone else usecase
16:52:56 <jhartkopf> bauzas: no, that was the other person who wanted the same feature some years ago
16:53:05 <bauzas> hah, apologies then
16:53:06 <sean-k-mooney> for ssh key manament i do think there is vaule in support multiple keys and updating them
16:53:39 <bauzas> jhartkopf: my concern is, which specific cases are useful that require the userdata to be mutable ?
16:53:56 <bauzas> besides the fact that other big Fives do this
16:54:28 <sean-k-mooney> speificly is there a usecase you are trying to enable that is a usecase for the end user/tenant not the operator
16:54:30 <jhartkopf> sean-k-mooney: Cloud-init supports this as well, yes. But why would you want to restrict API functionality and potentially restrict other's workflows?
16:55:09 <bauzas> sean-k-mooney: indeed, cases about end user
16:55:16 <sean-k-mooney> the user-data is "owned" by the enduser so if the reason to make it mutable is so operators can chagne it it would not be consitnet with the usage of that api
16:55:39 <dasp> when rebuilding an instance, you can't redeclare userdata (AFAIK) but you effectively make cloud-init re-run, that's a trap my users run into
16:56:07 <sean-k-mooney> dasp: for rebuild we do allow new user data to be pass in a later microverion i belive
16:56:18 <bauzas> yeah that rings me a bell
16:56:19 <sean-k-mooney> dasp: you are correct that orginally we did not
16:56:19 <bauzas> sec
16:56:21 <dasp> oh, sorry my org is running a bit "behind" on versions
16:57:12 <bauzas> sean-k-mooney: 2.3
16:57:18 <bauzas> so... Kilo
16:57:20 <sean-k-mooney> 2.54 allows the key to be changed on rebuild https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id50 and 2.57 allows user data to be updated
16:57:48 <bauzas> oh, that one
16:57:52 <sean-k-mooney> bauzas: its no 2.3
16:57:59 <bauzas> yeah my bad
16:58:06 <bauzas> this is Queens
16:58:06 <sean-k-mooney> but ya since queens
16:58:32 <bauzas> we only have 2 mins left
16:58:41 <bauzas> (as a timekeeper)
16:59:07 <sean-k-mooney> jhartkopf: we could proably make it mutable if there is a use facing usecase but we would need to support config dirve and note that the change may not fully be ableable untill after a hard reboot
16:59:23 <bauzas> I'll have to call it a wrap
16:59:25 <bauzas> for this week
16:59:30 <bauzas> we can revisite this in the spec
17:00:21 <bauzas> #info https://review.opendev.org/c/openstack/nova-specs/+/816542 requires propre description of enduser cases for mutable user data
17:00:31 <bauzas> that's it, we're over time
17:00:34 <bauzas> thanks all
17:00:36 <bauzas> #endmeeting