16:02:02 <bauzas> #startmeeting nova
16:02:02 <opendevmeet> Meeting started Tue May 17 16:02:02 2022 UTC and is due to finish in 60 minutes.  The chair is bauzas. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:02:02 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:02:02 <opendevmeet> The meeting name has been set to 'nova'
16:02:10 <bauzas> sorry, was late
16:02:11 <gibi> o/
16:02:14 <dansmith> o/
16:02:16 <elodilles> o/
16:02:25 <Uggla> o/
16:02:33 <bauzas> #link https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting
16:02:47 * bauzas was having his brain locked due to some previous meeting
16:03:12 <bauzas> and the heat wave here makes me melting
16:03:25 <bauzas> let's start then
16:03:31 <bauzas> #topic Bugs (stuck/critical)
16:03:36 <bauzas> #info No Critical bug
16:03:40 <bauzas> #link https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New 14 new untriaged bugs (-5 since the last meeting)
16:03:49 <bauzas> kudos to Uggla for this excellent work
16:04:26 <bauzas> he wrote a triage etherpad (yet again, this wasn't mandatory to do it) but in case you wanna know what he triaged https://etherpad.opendev.org/p/nova-bug-triage-20220510
16:04:46 <bauzas> #link https://storyboard.openstack.org/#!/project/openstack/placement 26 open stories (0 since the last meeting) in Storyboard for Placement
16:04:53 <bauzas> #info Add yourself in the team bug roster if you want to help https://etherpad.opendev.org/p/nova-bug-triage-roster
16:05:20 <bauzas> which leads me to the last point
16:05:51 <bauzas> sean-k-mooney: are you okay with holding the bug baton for this week ?
16:06:01 <sean-k-mooney> yes
16:06:06 <bauzas> gracias
16:06:12 <bauzas> #info Next bug baton is passed to sean-k-mooney
16:06:38 <bauzas> any bugs to discuss before we move on ?
16:07:03 <bauzas> #topic Gate status
16:07:10 <bauzas> #link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs
16:07:15 <bauzas> #link https://zuul.openstack.org/builds?project=openstack%2Fplacement&pipeline=periodic-weekly Placement periodic job status
16:07:19 <bauzas> #link https://zuul.opendev.org/t/openstack/builds?job_name=nova-emulation&pipeline=periodic-weekly&skip=0 Emulation periodic job runs
16:07:25 <bauzas> all the above seems fine ^
16:07:37 <bauzas> and not fine like in a fire
16:07:51 <artom> I think Uggla had a bug he wanted to double-check? Dunno if you want to leave that for the open discussion
16:07:59 <artom> triaged
16:07:59 <artom> 1959186 (appears to be valid TBC tomorrow)
16:08:01 <bauzas> artom: we can
16:08:52 <sean-k-mooney> https://bugs.launchpad.net/nova/+bug/1959186
16:08:59 <bauzas> I'll leave it for open discussion then
16:09:06 <bauzas> people can start digesting it meanwhile
16:09:16 * bauzas has zero context yet
16:09:25 <sean-k-mooney> initally this does not seam like a bug
16:09:28 <bauzas> #info Please look at the gate failures and file a bug report with the gate-failure tag.
16:09:29 <sean-k-mooney> but know behavior but ya
16:09:34 <bauzas> #info STOP DOING BLIND RECHECKS aka. 'recheck' https://docs.openstack.org/project-team-guide/testing.html#how-to-handle-test-failures
16:09:35 <sean-k-mooney> lets wait till later
16:09:46 <bauzas> #topic Release Planning
16:09:52 <bauzas> #link https://releases.openstack.org/zed/schedule.html
16:09:56 <bauzas> #info Zed-1 is due in 2 days
16:09:58 <bauzas> \o/
16:10:09 <bauzas> before we discuss about this milestone
16:10:13 <bauzas> #info Spec review day happened last week on May 10th
16:10:18 <bauzas> #info 2 specs were approved but a lot of them were reviewed. Kudos to the team for this hard work.
16:10:26 <gibi> <3
16:10:42 <bauzas> I've seen lot of feedback there
16:10:50 <bauzas> very happy
16:10:54 <bauzas> now,
16:10:57 <bauzas> zed-1
16:11:20 <bauzas> in theory, we only have one thing to take care of for this milestone
16:11:31 <bauzas> libraries releases
16:12:29 <sean-k-mooney> yep the release patches have been proposed
16:12:36 <sean-k-mooney> i have not looked at them yet
16:13:07 <sean-k-mooney> are theree any patches peopel would like to wait for?
16:13:18 <bauzas> that's my humble question
16:13:32 <sean-k-mooney> there is one patch in flight in os-vif but i dotn think we have to wait for that
16:13:32 <bauzas> do people care of something they'd like to see published before end of this cycle for the libs ?
16:13:44 <gibi> no need to wait, we can push out a new lib release any time it is freeee
16:13:45 <bauzas> os-traits seems interesting to look at
16:14:08 <sean-k-mooney> yep we use cycle with intermdiaries
16:14:16 <sean-k-mooney> which requries at least 1 release per cycle
16:14:22 <sean-k-mooney> but we can have as many as we want
16:14:29 <bauzas> https://review.opendev.org/q/project:os-traits+is:open is empty :)
16:14:48 <bauzas> oh shit
16:14:48 <sean-k-mooney> yep i think we close to approving spec that would add some
16:14:52 <bauzas> https://review.opendev.org/q/project:openstack/os-traits+is:open
16:14:56 <bauzas> better now:)
16:15:26 <sean-k-mooney> so the manial share spec is still pending
16:15:30 <bauzas> ok, only the manila-related traits
16:15:34 <sean-k-mooney> so Uggla's patch cant merge yet
16:15:36 <bauzas> and yeah the spec is pending
16:15:38 <bauzas> correct
16:15:46 <bauzas> so, let's release os-traits by now
16:15:52 <sean-k-mooney> we can merge teh runtim patch but that is not urgent
16:16:00 <sean-k-mooney> +1
16:16:04 <bauzas> and we'll publish a newer os-traits release once the spec is approved
16:16:07 <bauzas> Uggla: ^
16:17:16 <elodilles> note: os-traits is in independent release model, so release patch won't be generated by release team
16:17:21 <bauzas> https://review.opendev.org/q/project:openstack/os-resource-classes+is:open is a bit more crap
16:17:32 <bauzas> oh shit you're right
16:17:44 <bauzas> I was looking at https://governance.openstack.org/tc/reference/projects/nova.html#deliverables
16:17:50 <Uggla> bauzas, \o/
16:17:52 <bauzas> but it's not describing the release models
16:17:54 <sean-k-mooney> elodilles: os-traits isnt independint is it
16:18:09 <bauzas> I'm lost with all the references
16:18:20 <sean-k-mooney> oh it is
16:18:22 <bauzas> for release models, this is another website unrelated to governance
16:18:24 <sean-k-mooney> well that does not make sense
16:18:34 <sean-k-mooney> because placment is tightly coupled ot it
16:18:36 <elodilles> sean-k-mooney: it is
16:18:53 <sean-k-mooney> elodilles: placmentn currently only works with exactly one version of os-triats
16:18:55 <bauzas> ok, found the right doc https://releases.openstack.org/teams/nova.html
16:19:03 <sean-k-mooney> https://github.com/openstack/releases/blob/master/deliverables/_independent/os-traits.yaml
16:19:24 <bauzas> this leaves os-r-c and os-traits be independent
16:19:49 <sean-k-mooney> right but placment assert the exact numober of triat and resouce classes in its test
16:19:55 <bauzas> https://releases.openstack.org/teams/nova.html#independent
16:20:13 <bauzas> sean-k-mooney: we discussed this at the PTG right ?
16:20:15 <sean-k-mooney> so it only works form a unit test perspecive with exactly one version of os-traits/os-rescoue classes
16:20:16 <bauzas> and we agreed on a proposal
16:20:17 <sean-k-mooney> yep
16:20:22 <sean-k-mooney> relax the test
16:20:25 <bauzas> yup
16:20:29 <bauzas> so let's do it
16:20:33 <sean-k-mooney> yep
16:20:45 <sean-k-mooney> just pointing out that indpenent did not make sense wiht that context
16:21:00 <sean-k-mooney> but ok we can wait to release it until we have new traits
16:21:08 <sean-k-mooney> and fix the test in the interim
16:21:22 <bauzas> point is, we should only care of os-vif and client releases for placement and nova, that's it
16:21:40 <bauzas> (for zed-1 I mean)
16:21:58 <sean-k-mooney> yep
16:23:06 <bauzas> gmann: btw. this is not going well for https://review.opendev.org/c/openstack/os-vif/+/840020
16:23:15 <bauzas> but let's discuss this on open discussion ^
16:23:26 <bauzas> and let's continue the agenda
16:23:35 <bauzas> #topic Review priorities
16:23:40 <bauzas> #link https://review.opendev.org/q/status:open+(project:openstack/nova+OR+project:openstack/placement+OR+project:openstack/os-traits+OR+project:openstack/os-resource-classes+OR+project:openstack/os-vif+OR+project:openstack/python-novaclient+OR+project:openstack/osc-placement)+label:Review-Priority%252B1
16:23:59 <bauzas> #link https://review.opendev.org/c/openstack/project-config/+/837595 Gerrit policy for Review-prio contributors flag. Naming bikeshed in there.
16:24:08 <bauzas> reviews are welcome on ^
16:24:21 <bauzas> I see gibi having concerns on the naming
16:24:49 <gibi> I have no good suggestion
16:24:53 <gibi> so feel free to ignore me
16:27:03 <gibi> my problem on the current naming is that is sounds like a core approves that something is a prioirty
16:27:19 <gibi> but the aim is instead that the core says "I will review this"
16:27:33 <bauzas> gibi: I can propose something else
16:27:39 <bauzas> to unblock the patch
16:27:52 <bauzas> #link https://docs.openstack.org/nova/latest/contributor/process.html#what-the-review-priority-label-in-gerrit-are-use-for Documentation we already have
16:27:55 <sean-k-mooney> bauzas: sure go for it
16:28:24 <bauzas> next topic,
16:28:34 <bauzas> #topic Stable Branches
16:28:41 <bauzas> elodilles: I haven't seen you updating the section
16:28:45 <bauzas> so far, so good ?
16:29:06 <elodilles> #info ussuri and older branches are still blocked, newer branches should be OK
16:29:32 <bauzas> I think we said last week, newer branches *are* OK :)
16:29:35 <elodilles> i haven't got any further with the investigation of ussuri branch :/
16:30:10 <elodilles> bauzas: yes, unfortunately nothing news :(
16:30:15 <bauzas> damn, looks like we have a netsplit at the time of the meeting
16:30:27 <elodilles> yepp, it seems :S
16:30:35 <sean-k-mooney> people will be back soon proably
16:30:55 <bauzas> hopefully yes
16:31:00 <sean-k-mooney> looks like mainly NA folks
16:31:01 <bauzas> let's continue then
16:31:24 <bauzas> sean-k-mooney: yeah, one OFTC server seems to have killed the conns
16:31:34 <elodilles> nothing more to add for now :X
16:31:55 <bauzas> elodilles: at one time, we'll need to cut the rope on the older branches
16:32:38 <sean-k-mooney> older starting form ?
16:32:38 <elodilles> bauzas: indeed. now ussuri is in bad shape for weeks now...
16:32:50 <sean-k-mooney> hum
16:32:52 <bauzas> sean-k-mooney: ussuri
16:33:00 <sean-k-mooney> so we woudl be droping train and below too
16:33:23 <elodilles> actually it fails due to multiple intermittent failures
16:33:27 <bauzas> we need help or some action in order to mitigate the blocker
16:33:42 <bauzas> elodilles: yup, constant rechecks
16:33:45 <sean-k-mooney> bauzas: what is the blocker exactly
16:34:01 <bauzas> the failure ratio is so high it becomes unpractical to merge
16:34:15 <sean-k-mooney> right but on what issue
16:34:22 <bauzas> a couple of them
16:34:28 * sean-k-mooney has not looked at stable/ussiri
16:34:29 <bauzas> elodilles: we tracked them, right
16:34:46 <sean-k-mooney> ok i was wonderign if there was one we shoudl look at specificly
16:34:57 <elodilles> sean-k-mooney: guest kernel panics, volume timeouts,
16:35:12 <sean-k-mooney> as in volume detach
16:35:19 <sean-k-mooney> is it resize related?
16:35:44 <sean-k-mooney> becasue we noticed today that reize is not suing hte sshable change yet
16:35:56 <sean-k-mooney> gibi proposed a patch to cover that
16:36:08 <elodilles> also this bug comes up often: https://bugs.launchpad.net/nova/+bug/1901739
16:36:39 <sean-k-mooney> im not familar with that
16:37:22 <sean-k-mooney> looks like lee fixed that in V but it has not been backported
16:37:52 <gibi> it is a mix of fixes already due to zuul migration and ubuntu version cahgne
16:37:55 <gibi> chnage
16:37:56 <sean-k-mooney> ok we can proably move on but if you have a list elodilles  please share
16:38:48 <elodilles> sean-k-mooney: well it's on the recheck list on this patch: https://review.opendev.org/c/openstack/nova/+/838033
16:38:51 <elodilles> :/
16:39:28 <sean-k-mooney> can we ask infra to force merge that
16:39:57 <elodilles> sean-k-mooney: it won't help to other patches, they will need the same rechecks
16:40:17 <sean-k-mooney> yes i know
16:40:35 <bauzas> in general, we can try to release the jobs
16:40:50 <bauzas> in order to merge one specific patch we want
16:41:05 <elodilles> yes, one way is to decrease the test coverage
16:41:05 <gmann> +1, better than force merge
16:41:06 <bauzas> we did that in some past
16:41:19 <sean-k-mooney> ok
16:41:26 <gibi> here that would mean to cut out devstack based jobs
16:41:29 <sean-k-mooney> we can make some non voting for now
16:41:35 <gibi> an rely on unit and functional
16:41:43 <gibi> *and
16:41:45 <bauzas> this was ages ago
16:41:47 <gmann> yeah just for temporary  time
16:41:53 <elodilles> gibi: or some volume related tests. though i don't know which is worse :S
16:41:53 <bauzas> but I still remember the magic formula
16:42:22 <gibi> gmann: it would not be temporary if we disable jobs we will never fix tem
16:42:25 <gibi> them
16:42:34 <sean-k-mooney> well i asusme we have  a limited set of patches that we want to merge and we woudl turn these abck on like droping lc jobs
16:42:54 <sean-k-mooney> so i was epecting disable patch then revert
16:43:02 <bauzas> gibi: the idea is to relax the jobs before merging patches we want and then enabling again the jobs
16:43:19 <gibi> bauzas: and will we do it for every patch we want to merge?
16:43:21 <bauzas> and see whether the failure ratio drops again
16:43:23 <gmann> yeah that is what I thought, enabling them again
16:43:40 <gibi> bauzas: it would work iff we would have patches in queue that fixes those breaks but we dont
16:43:48 <bauzas> gibi: no, we need to identify a set of patches we'd like to merge in order to help with CI stability
16:44:01 <gibi> we don't have the fixes :/
16:44:33 <gibi> it is not like merge these 5 patches to stabilize ussuri, we don't have those 5 patches ready
16:44:50 <gibi> we have random patches backported and waiting
16:44:51 <gmann> humm
16:45:25 <bauzas> ok, we won't obviously solve this problem by now
16:45:33 <gmann> one thing to note if we are removing integration tests coverage then it is better to make branch EOL like we did for ocata
16:45:38 <gibi> so something like: 1) collect the issue to be fixed in ussurit to stabilize CI 2) create fixes for them 3) force merge them 4) profit
16:45:39 <bauzas> let's state we all know about this problem and we'll figure out the solutions later
16:45:58 <bauzas> gibi: yeah, sounds 1/ and 2/ are not done yet
16:45:59 <elodilles> maybe if that "SSHABLE" fix helps things in ussuri we have one, but still we probably have other failures
16:46:30 <gibi> elodilles: a buncs of SSHABLE already landed, so we should see them help
16:46:38 <gmann> elodilles: that is again difficult as we are not supposed to use tempest master for ussuri so not all master change will go for ussuri testing
16:46:43 <bauzas> elodilles: ideally an etherpad of doom may help gathering ideas and feedback
16:46:48 <gibi> gmann: ahh
16:46:50 <gibi> that explains
16:46:55 <gmann> I have patch  up to cap tempest for ussuri but that is not merged yet
16:46:58 <sean-k-mooney> gmann: wait we are not?
16:47:08 <sean-k-mooney> gmann: temest is ment to be branchless upstream
16:47:13 <gmann> but we have stopped ussuri support in tempest master
16:47:31 <sean-k-mooney> hum
16:47:37 <sean-k-mooney> because of what reason?
16:47:41 <gmann> sean-k-mooney: yes and master tempest is for SUpported branch and we cannot support EM branches
16:47:42 <sean-k-mooney> python version ?
16:48:04 <sean-k-mooney> shoudl that not be a project choice
16:48:16 <gmann> EM branches - #link https://docs.openstack.org/tempest/latest/stable_branch_support_policy.html
16:48:17 <sean-k-mooney> tempest should not have to support it
16:48:42 <sean-k-mooney> but project that support em branches shoudl still be able to choose which tempest to run
16:48:46 <gmann> sean-k-mooney: we have to pin tempest for EM branch so that using master there can break them anytime
16:49:06 <gmann> sean-k-mooney: yeah, you can but no guarantee if master tempest will work
16:49:19 <gmann> as long as it is passing it is ok to use
16:49:58 <sean-k-mooney> ack
16:50:12 <sean-k-mooney> so i think i would prefer to keep tempest uncapped until ti breaks
16:50:14 <elodilles> (well, i guess changes on tempest master can affect not only EM branches, but they are affected most likely)
16:50:30 <sean-k-mooney> elodilles: well nova uses microversion
16:50:40 <sean-k-mooney> so it should work form our point of view
16:50:53 <gmann> elodilles: for supported branches we make sure tempest does not break them but for EM it is not the case
16:50:57 <sean-k-mooney> chagnes to tempest should not break our stable branches expcti if there ar package dep issues
16:51:14 <gmann> plan is : by default devstack will cap the tempest in ussuri and project can override it in jobs
16:51:15 <bauzas> I need to timestop the conversation
16:51:15 <sean-k-mooney> which i guess is why this was done
16:51:41 <gmann> yeah, we can discuss it in qa or nova channel after meeting
16:51:49 <sean-k-mooney> +1
16:51:51 <elodilles> +1
16:51:52 <bauzas> gmann: sean-k-mooney: gibi: you're all free to continue the conversation after the meeting
16:51:57 <bauzas> cool
16:51:57 <gmann> sure
16:52:01 <bauzas> moving on quickly
16:52:05 * gibi will pass out after the meet :/
16:52:08 <bauzas> #topic Open discussion
16:52:21 <bauzas> Uggla: do you want to discuss https://bugs.launchpad.net/nova/+bug/1959186
16:52:22 <bauzas> ?
16:52:44 <bauzas> Uggla: you left this bug with comments on your triage etherpad
16:52:53 <Uggla> yep
16:53:18 <Uggla> It appears valid to me, but I would like a crosscheck.
16:53:26 <sean-k-mooney> i dont think it is
16:54:02 <sean-k-mooney> there is a long runnign knwon issue that you cant delete in use image form glance if it uses ceph
16:54:14 <sean-k-mooney> that only happens if you use raw images
16:54:27 <sean-k-mooney> at that enabel our copy on write shallow clone code path
16:54:34 <bauzas> sounds at least unrelated to nova
16:54:42 <sean-k-mooney> which i suspect is what is happening here with the snapshot
16:54:45 <bauzas> maybe valid but not on our side
16:54:52 <bauzas> right?
16:55:21 <dansmith_> it's really desired behavior even
16:55:27 <sean-k-mooney> there have been some feature request in this area
16:55:39 <sean-k-mooney> some peopel woudl like too break the shallow copy
16:55:43 <dansmith_> I think it'd be a feature on the glance side, IIRC
16:56:07 <sean-k-mooney> ya so they have not providded enough info in the bug
16:56:12 <sean-k-mooney> we do not know the image type
16:56:21 <dansmith> nova doesn't even know about the linkage after it's created right? so if the link is broken, nova is fine with it
16:56:21 <sean-k-mooney> to confirm wone way or another
16:56:39 <sean-k-mooney> am im not sure
16:57:00 <bauzas> dansmith: hence my point, unrelated to the project
16:57:04 <bauzas> => Opinion
16:57:14 <levy14> have a feature proposal, but not in the agenda yet. may I ask here and gauge interest/feasibility?
16:57:15 <dansmith> bauzas: yeah
16:57:24 <sean-k-mooney> there is some metadata on teh image but i dont know if we read that when we create the new vm
16:57:29 <sean-k-mooney> or tack it on our side
16:57:43 <bauzas> sean-k-mooney: we can add glance as a service project in the bug report
16:57:50 <bauzas> and mark it invalid on our side
16:58:02 <bauzas> we'll see it coming back if that's really a nova bug
16:58:21 <bauzas> Uggla: works for you ?
16:58:26 <Uggla> yes
16:58:57 <bauzas> OK, sold
16:59:12 <bauzas> #agreed https://bugs.launchpad.net/nova/+bug/1959186 to be punted to Glance
16:59:56 <sean-k-mooney> ill do that  and ask for more info in the bug
17:00:02 <bauzas> I wanted to discuss https://review.opendev.org/c/openstack/os-vif/+/840020 but we're at time
17:00:14 <bauzas> thanks all
17:00:16 <bauzas> #endmeeting