16:01:03 <bauzas> #startmeeting nova
16:01:03 <opendevmeet> Meeting started Tue Feb  7 16:01:03 2023 UTC and is due to finish in 60 minutes.  The chair is bauzas. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:01:03 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:01:03 <opendevmeet> The meeting name has been set to 'nova'
16:01:13 <bauzas> sorry folks, forgot to remind you of the meeting
16:01:43 <bauzas> who's around ?
16:01:57 <Uggla> o/
16:02:04 <elodilles> o/
16:02:44 <bauzas> I guess we can make a soft start
16:02:55 <bauzas> #link https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting
16:03:02 <bauzas> #topic Bugs (stuck/critical)
16:03:07 <bauzas> #info No Critical bug
16:03:11 <bauzas> #link https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New 28 new untriaged bugs (+1 since the last meeting)
16:03:15 <bauzas> #info Add yourself in the team bug roster if you want to help https://etherpad.opendev.org/p/nova-bug-triage-roster
16:03:30 <bauzas> Uggla: fancy getting the bug triage baton for this week N
16:03:32 <bauzas> ?
16:04:02 <sean-k-mooney> o/
16:04:09 <Uggla> I will be out next week so I would rather postponed if possible
16:04:49 <bauzas> ack, so artom would you want to continue having the triage baton for an extra week ?
16:04:59 <Uggla> If not I'll try to do my best till the end of the week.
16:05:13 <artom> Ah, I completely dropped the ball, didn't I?
16:05:18 <artom> Yeah, I can keep it
16:05:26 <bauzas> ++
16:05:31 <bauzas> artom: no worries
16:05:37 <bauzas> and thanks
16:05:41 <gibi> o/
16:06:20 <dansmith> o/
16:06:28 <bauzas> ok moving on
16:06:34 <bauzas> #topic Gate status
16:06:39 <bauzas> #link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs
16:06:42 <bauzas> new item
16:06:52 <bauzas> #link https://etherpad.opendev.org/p/nova-ci-failures Etherpad for tracking CI failures
16:07:14 <gibi> there is a fairly long list but please add to it if you see failures
16:07:18 <gibi> that are not on the list
16:07:31 <bauzas> we have some of them are hit us in some place between the chair and the man
16:08:05 <bauzas> I think the hardest one is at the bottom of the document
16:08:05 <dansmith> oof yeah
16:08:23 <bauzas> I had a lovely morning and a half-afternoon spent on that one
16:08:42 <bauzas> in the context of a soonish feature freeze, more hands are more than welcome
16:09:01 <dansmith> I wrote the replace_location test, so I can look into that one.. it's a glance test though. I'm sure it's poking some bug in glance, because until I wrote that we didn't really have any tests for that stuff
16:09:08 <bauzas> because don't expect your patches to be reviewed if most of the cores are having their days spent on fixing CI problems
16:09:08 <dansmith> but maybe it is resolvable
16:10:06 <gibi> dansmith: there is https://bugs.launchpad.net/glance/+bug/1999800 and https://bugs.launchpad.net/glance/+bug/2006473 both location tests
16:10:17 <bauzas> and yeah, I know, debugging a CI failure isn't exactly the best experience you may have of working on an opensource project, but let's be honest and say that's necessary to have an healthy gate
16:10:26 <gibi> bauzas: +1
16:10:35 <dansmith> okay the former is the same as bauzas' one
16:10:41 <gibi> yeah probably
16:11:09 <dansmith> yeah from the logs, the test is clearly doing something legit and glance is rejecting it but shouldn't
16:11:16 <bauzas> gibi: I created https://bugs.launchpad.net/nova/+bug/2004641 but it seems duplicate of https://bugs.launchpad.net/glance/+bug/1999800
16:11:27 <dansmith> might be because it fails to talk to the cirros site occasionally, so maybe we can use an openstack infra url instead
16:11:43 <dansmith> bauzas: indeed
16:11:52 <sean-k-mooney> i tought we tried to pull those form provider proxies in ci
16:11:54 <gibi> bauzas: https://bugs.launchpad.net/tempest/+bug/2004641 and https://bugs.launchpad.net/glance/+bug/2006473 are duplicates but https://bugs.launchpad.net/glance/+bug/1999800 is a separate tc
16:11:55 <bauzas> I can close my one as duplicate
16:12:06 <dansmith> bauzas: ++
16:12:20 <sean-k-mooney> github is more repliable for downlaoding cirrors images by the way then the cirros site
16:12:22 <bauzas> I just ideally would like to track that bug in our project
16:13:01 <dansmith> sean-k-mooney: the cirros page just redirects to the github one
16:13:16 <sean-k-mooney> oh they finally implmetned that
16:13:29 <dansmith> sean-k-mooney: and we're just using CONF.image.http_image in that test
16:14:03 <sean-k-mooney> oh this is not the image pulled by https://github.com/openstack/devstack/blob/master/stackrc#L670-L708
16:14:06 <dansmith> the github URL is crazy long with tons of tokens and other values after the redirects it does
16:14:11 <bauzas> gibi: ack will mark your https://bugs.launchpad.net/nova/+bug/2004641 as duplicate of mine, then
16:14:12 <dansmith> sean-k-mooney: this is a tempest test
16:14:33 <sean-k-mooney> right the one with the larger image
16:14:41 <dansmith> no
16:14:58 <dansmith> gibi: it's the same test case, different behavior, but I'm guessing its the sameish problem
16:15:37 <bauzas> ok, you know what, I'll add mine in the tracking etherpad, and we'll figure out
16:15:49 <bauzas> the three of them are set against Glance either way
16:15:57 <sean-k-mooney> im surpised that the tempest test is not using the one we prestage in the vm but ok
16:16:48 <sean-k-mooney> i was expecting CONF.image.http_image to be file:///opt/devstack/data/cirros...
16:17:05 <gibi> dansmith: yeah probably similar root cause
16:17:06 <dansmith> sean-k-mooney: it can't be because that is specifically for testing fetching an image server-side from http
16:17:22 <sean-k-mooney> ah thanks i was missing that context
16:17:52 <sean-k-mooney> oh that that in https://bugs.launchpad.net/glance/+bug/2006473 i was only familar with https://bugs.launchpad.net/glance/+bug/1999800
16:18:08 <dansmith> they're the same test
16:18:31 <dansmith> sorry, the same test helper
16:18:48 <bauzas> and probably the rootcause
16:18:52 <sean-k-mooney> ya so likely the same cause
16:18:53 <bauzas> same rootcause
16:19:03 <bauzas> which is a flakey httpservice
16:19:48 <bauzas> either way, seems we have a path forward with the github image repo then ?
16:19:53 * bauzas trying to read between the lines
16:21:09 <bauzas> looks like people are gone
16:21:33 <dansmith> bauzas: no, it's already using that via redirect
16:21:36 <bauzas> there is another CI failure I'd like to talk about
16:21:40 <sean-k-mooney> from the name i would not expect either to depned on downloading an image over http but i have not looked at the detail of the test. i was expecting tempest to upload the image form disk.
16:21:41 <dansmith> bauzas: I'll take it and work something out
16:21:54 <bauzas> dansmith: very much appreciated, trust me.
16:22:10 <bauzas> dansmith: fwiw, the hits number seems low compared to other bits
16:22:29 <bauzas> bites*
16:22:41 <bauzas> so, about https://bugs.launchpad.net/nova/+bug/1946339
16:22:42 <dansmith> yeah, but if we have no other obvious ones to work on, at least I can make some progress on this :)
16:22:50 <bauzas> dansmith: heh
16:23:15 <bauzas> so, after a day of co-investigation with my CSI partner gibi on https://bugs.launchpad.net/nova/+bug/1946339
16:23:30 <bauzas> we identified this may come from a non-poisoned libvirt
16:23:56 <bauzas> the funny part is that we hit this in a thread, not in the main test
16:24:04 <bauzas> hence why we missed it before
16:24:21 <bauzas> I have a question
16:24:40 <bauzas> do people agree with merging https://review.opendev.org/c/openstack/nova/+/872975 even if it says it's a dnm ?
16:24:54 <bauzas> (tbc, I can make an update and remove the dnm title)
16:25:04 <dansmith> we should remove the dnm for sure
16:25:18 <sean-k-mooney> bauzas: melwitt had a patch to poison importing libvrt that should catuch this by the way
16:25:29 <opendevreview> Sylvain Bauza proposed openstack/nova master: Add logging for leaking out the non-poisoned libvirt testcase  https://review.opendev.org/c/openstack/nova/+/872975
16:25:42 <dansmith> bauzas: do you know about the thing you can do to add additional test payload report sections?
16:25:46 <bauzas> dansmith: acked ^
16:25:57 <dansmith> depending on what you're trying to do, that can be more useful than logging sometimes
16:26:09 <bauzas> dansmith: nope, hence my sending the bottle to the sea, asking for advices
16:26:15 <sean-k-mooney> bauzas: can you put a sleep in that busy loop too
16:26:29 <dansmith> it's not a busy loop is it?
16:26:33 <bauzas> nope
16:26:44 <gibi> it is walking a tree up
16:26:55 <bauzas> we're trying to find an attribute from an eventlet object and if we can't find it, we walk the ascendance
16:26:57 <sean-k-mooney> it will loop until the test_case_id is not None
16:27:10 <gibi> it walks along the eventlet.parent link
16:27:12 <sean-k-mooney> i guess its proably fine
16:27:31 <gibi> so while it busy it is bounded
16:27:38 <sean-k-mooney> oh sorry your right it is doing that
16:27:41 <sean-k-mooney> ok
16:27:42 <bauzas> dansmith: so, about the payload reporting, you gained my interest
16:27:52 <dansmith> bauzas: https://github.com/openstack/glance/blob/master/glance/tests/functional/__init__.py#L1129-L1130
16:28:13 <dansmith> bauzas: that adds another section of the test failure reporting, like "here's the stdout I captured" and "here are the log lines I captured"
16:28:23 <bauzas> ffff
16:28:37 <bauzas> dansmith: ++
16:28:43 <dansmith> helps to separate nova-logging from something specifically to be reported by the test case
16:28:52 <dansmith> especially if debug logging isn't captured, or is being mocked out, etc
16:29:18 <dansmith> in glance I found it useful because their functional workers run outside the main process, but also in some cases where I needed to debug failures
16:29:28 <dansmith> (failures that happen infrequently)
16:29:39 <dansmith> anyway, just FYI, might be helpful
16:29:48 <bauzas> it could be
16:29:48 <gibi> dansmith: ohh that is good to know :)
16:30:22 <sean-k-mooney> oh addDetail
16:30:31 <bauzas> dansmith: the problem is that we get an exception from a test which is actually not due by this test but rather by a leaked eventlet thread that blows up at that point in time
16:30:40 <sean-k-mooney> i have seen that before but never looked into it ya look useful
16:31:18 <dansmith> gibi: yeah, it's kinda nice :)
16:31:39 <bauzas> ideally I would like to trace the whole parenting stack that triggered the leaky thread
16:32:32 <gibi> bauzas: we will hopefuly get the name of the leaky test case and then we can create a local reproduction
16:33:11 <bauzas> gibi: a stack would have been better but yeah
16:33:20 <gibi> you have a stack
16:33:25 <gibi> but it start when the thread starts
16:33:40 <bauzas> that's the parent stack I want :)
16:33:43 <gibi> yeah
16:33:45 <gibi> that is hard
16:35:09 <bauzas> yup
16:35:25 <bauzas> anyway, reviews appreciated on https://review.opendev.org/c/openstack/nova/+/872975
16:35:40 <bauzas> moving on ?
16:35:52 <sean-k-mooney> sure
16:36:39 <bauzas> #link https://zuul.openstack.org/builds?project=openstack%2Fnova&project=openstack%2Fplacement&pipeline=periodic-weekly Nova&Placement periodic jobs status
16:36:41 <gibi> I'm on it
16:36:51 <bauzas> #info Please look at the gate failures and file a bug report with the gate-failure tag.
16:36:54 <bauzas> #info Please look at the gate failures and file a bug report with the gate-failure tag.
16:37:01 <bauzas> #info STOP DOING BLIND RECHECKS aka. 'recheck' https://docs.openstack.org/project-team-guide/testing.html#how-to-handle-test-failures
16:37:12 <bauzas> done with this topic, phew.
16:37:18 <bauzas> #topic Release Planning
16:37:23 <bauzas> #link https://releases.openstack.org/antelope/schedule.html
16:37:28 <bauzas> #info Antelope-3 is in 1.2 weeks
16:37:39 <bauzas> I said 1.2 because it will be on Thursday next week
16:37:47 <bauzas> #link https://blueprints.launchpad.net/nova/antelope All accepted blueprints for 2023.1
16:37:52 <bauzas> #link https://etherpad.opendev.org/p/nova-antelope-blueprint-status Blueprint status for 2023.1
16:37:59 <bauzas> feel free to comment it as much as you want ^
16:38:28 <bauzas> I was originally planning to do a full reviews set by today, but due to the former topic, I abandoned my promise
16:39:00 <elodilles> a bit related to release: 'Release final os-vif for 2023.1 Antelope' https://review.opendev.org/c/openstack/releases/+/872779
16:39:06 <bauzas> (yet again saying, don't expect the reviews to magically happen, be present and interact with us)
16:39:19 <bauzas> elodilles: good catch I forgot to add it the agenda
16:39:23 <bauzas> Important :
16:40:05 <bauzas> #info Thursday is the non-client libs feature freeze, which means we can only accept features changes for os-vif, os-traits and os-rc up until Thursday
16:40:25 <bauzas> later changes will be on hold until next release
16:40:40 <elodilles> ++
16:41:17 <bauzas> I haven't looked at os-vif, os-traits and os-resourceclasses master branches, but I think we have open changes on them
16:42:16 <bauzas> so, if anyone wants some addition to those libraries, I'd recommend them to ping me or anyone else for reviews
16:43:05 <bauzas> last point
16:43:18 <bauzas> FeatureFreeze is on next Thursday
16:43:26 <bauzas> we'll see how the gate goes by that time
16:43:59 <bauzas> but as for the older releases, the most important for having your series accepted for Antelope is to get a +W before Thursday EOB
16:44:20 <bauzas> we'll manage the rechecks if needed
16:44:41 <bauzas> don't freak out by the gate stability, but please continue to ensure your patches are ready for reviews
16:44:55 <sean-k-mooney> elodilles: i was planning to propose an os-vif release to include rodolfos patches
16:45:06 <sean-k-mooney> so i want to confirm the sha before we move forward with that
16:45:10 <sean-k-mooney> ill do that after the meeting
16:45:16 <elodilles> sean-k-mooney: as i know he updated the release patch already
16:45:18 <bauzas> sean-k-mooney: thanks, appreciated
16:45:47 <elodilles> sean-k-mooney: but please -1 if something is still missing
16:45:53 <sean-k-mooney> ack just looking now ill +1 if its correct
16:46:06 <elodilles> sean-k-mooney: that is even better :)
16:46:46 <bauzas> I think for os-traits I've seen one patch from Uggla
16:47:02 <bauzas> but I don't think we can reasonably merge the nova related series
16:47:15 <Uggla> yep but it can wait.
16:47:34 <bauzas> next topic, if so
16:47:37 <bauzas> #topic vPTG Planning
16:47:43 <bauzas> a bit early butn
16:47:49 <bauzas> #link https://www.eventbrite.com/e/project-teams-gathering-march-2023-tickets-483971570997 Register your free ticket
16:48:26 <bauzas> also
16:48:34 <bauzas> #link https://etherpad.opendev.org/p/nova-bobcat-ptg Draft PTG etherpad
16:48:55 <bauzas> every cycle, we're asked how long we should have sessions
16:49:25 <bauzas> I thought it would be better to somehow have an idea on how many topics we gonna discuss before saying how many slots we need :)
16:49:36 <bauzas> but I know
16:49:47 <bauzas> lots of topics will arrive the week before the PTG :)
16:50:03 <sean-k-mooney> the more virtual ptgs we have the less energy i have for them. that said i would prefer to have more slots over more days then a few short long ones
16:50:04 <bauzas> the thing is, you have the etherpad, feel free to amend it
16:50:14 <sean-k-mooney> s/short//
16:50:41 <bauzas> sean-k-mooney: this cycle, we will also have a "physical PTG" at the middle of bobcat-1
16:50:54 <bauzas> that could alleviate some discussions
16:50:55 <sean-k-mooney> that should really be for C planning
16:51:08 <bauzas> or B implementation phasing :)
16:51:26 <sean-k-mooney> either/both its close to Spec Freeze
16:51:33 <bauzas> I haven't checked whether the proposed B agenda is merged yet
16:51:39 <sean-k-mooney> so proably to late for directional changes on large specs
16:52:01 <bauzas> sean-k-mooney: don't disagree, I'm just saying it could help some small contributors to get audience when they need
16:52:02 <elodilles> B schedule was merged
16:52:08 <bauzas> cool
16:52:10 <bauzas> so
16:52:13 <sean-k-mooney> https://releases.openstack.org/bobcat/schedule.html
16:52:16 <sean-k-mooney> so yes its merged
16:53:02 <bauzas> https://releases.openstack.org/bobcat/schedule.html ays that pPTG will be 3 weeks before specfreeze (if we agree on the PTG at b-2 being spec freeze)
16:53:23 <bauzas> so, that's why I'm saying we could have a shorter but productive vPTG
16:53:31 <bauzas> like 2 hours per day
16:53:45 <bauzas> (and ideally, I'd like to attend some TC discussions this time)
16:54:00 <sean-k-mooney> i would prefer to frontload the plannign to the vPTG and have the physical one be more C focused but ok
16:54:16 <bauzas> don't disagree
16:54:19 <sean-k-mooney> because of its time in the cycle it feels more like a fourm then a ptg
16:54:23 <bauzas> anyway, we're rushing out of time
16:54:30 <bauzas> #topic Review priorities
16:54:41 <bauzas> #link https://review.opendev.org/q/status:open+(project:openstack/nova+OR+project:openstack/placement+OR+project:openstack/os-traits+OR+project:openstack/os-resource-classes+OR+project:openstack/os-vif+OR+project:openstack/python-novaclient+OR+project:openstack/osc-placement)+(label:Review-Priority%252B1+OR+label:Review-Priority%252B2)
16:54:47 <bauzas> #info As a reminder, cores eager to review changes can +1 to indicate their interest, +2 for committing to the review
16:54:51 <bauzas> #topic Stable Branches
16:54:57 <bauzas> elodilles: you have 5 mins (sorry)
16:54:59 <elodilles> #info stable branches seem to be OK back till wallaby
16:55:04 <bauzas> huzzah
16:55:05 <elodilles> stable/wallaby gate is passing (failing openstacksdk-functional-devstack job was removed from wallaby)
16:55:13 <bauzas> thanks gmann for the hard work on stable/wallaby
16:55:18 <elodilles> yepp
16:55:19 <elodilles> ++
16:55:21 <elodilles> #info stable/victoria gate is affected by the same failing openstacksdk-functional-devstack job
16:55:30 <elodilles> #info ussuri and train gates are broken broken ('Could not build wheels for bcrypt, cryptography' errors)
16:55:38 <elodilles> #info stable branch status / gate failures tracking etherpad: https://etherpad.opendev.org/p/nova-stable-branch-ci
16:55:41 <elodilles> EOM
16:55:54 <bauzas> elodilles: I'll pay attention to the ussuri branch with the CVE backport
16:56:01 <elodilles> bauzas: ack
16:56:10 <bauzas> ... when I have time :)
16:56:18 <elodilles> bauzas: i have a proposed potential workaround
16:56:26 <elodilles> for ussuri
16:56:42 <bauzas> elodilles: cool, let's figure that out after the meeting, tomorrow per say
16:56:51 <elodilles> bauzas: ++
16:56:54 <bauzas> fwiw, I'm planning to deliver the cve fix down to ussuri
16:57:07 <bauzas> but not provide any backport to train
16:57:08 <elodilles> why not train? :)
16:57:16 <bauzas> due to the oslo.utils versioning
16:57:26 <bauzas> most of the distros now made the backports
16:57:36 <bauzas> so it's upstream support
16:57:42 <bauzas> and Train is on EM
16:57:50 <elodilles> well, Wallaby is EM
16:57:56 <bauzas> and ussuri too
16:57:57 <elodilles> (and Xena soon, too)
16:58:12 <bauzas> but it was simple to backport the fix down to ussuri
16:58:19 <bauzas> it was cheap, so we proposed it
16:58:28 <bauzas> backporting it to train is a totally different story
16:58:28 <elodilles> ok :) thanks for that!
16:58:54 <bauzas> it requires some oslo.utils backport too (and then a janga puzzle with dependency management)
16:59:06 <elodilles> :S
16:59:24 <bauzas> so, things are said, crystal clear.
16:59:39 <elodilles> thanks, i see
16:59:41 <bauzas> elodilles: thanks elodilles for the stable report
16:59:46 <elodilles> np
16:59:50 <bauzas> last point for the 20 secs left
16:59:54 <bauzas> #topic Open discussion
16:59:59 <bauzas> nothing on the agenda
17:00:04 <bauzas> so I'll close the meeting
17:00:16 <bauzas> feel free to add your items for next week
17:00:19 <bauzas> thnaks all
17:00:21 <bauzas> #endmeeting