16:00:43 <bauzas> #startmeeting nova
16:00:43 <opendevmeet> Meeting started Tue Oct 11 16:00:43 2022 UTC and is due to finish in 60 minutes.  The chair is bauzas. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:43 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:43 <opendevmeet> The meeting name has been set to 'nova'
16:00:45 <bauzas> heyho
16:00:50 <elodilles> o/
16:01:09 <bauzas> who's around ?
16:02:08 <bauzas> we can start if needed
16:02:20 <bauzas> hopefully, should be a quick one, as we have the PTG next week
16:02:43 <elodilles> ++
16:02:50 <bauzas> #topic Bugs (stuck/critical)
16:02:56 <bauzas> #info No Critical bug
16:03:01 <bauzas> #link https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New 4 new untriaged bugs (-1 since the last meeting)
16:03:01 <gibi> o/
16:03:05 <dansmith> o/
16:03:10 <bauzas> #link https://storyboard.openstack.org/#!/project/openstack/placement 26 open stories (+0 since the last meeting) in Storyboard for Placement
16:03:20 <bauzas> elodilles: thanks for having looked at the bugs
16:03:24 <elodilles> np
16:03:30 <bauzas> anything you would want to discuss ?
16:03:36 <bauzas> #info Add yourself in the team bug roster if you want to help https://etherpad.opendev.org/p/nova-bug-triage-roster
16:03:48 <elodilles> maybe one bug
16:03:52 <elodilles> (or two)
16:04:14 <Uggla> */
16:04:19 <elodilles> related to the well-know volume timeout failure
16:04:33 <elodilles> a new general bug was open: https://bugs.launchpad.net/nova/+bug/1992328
16:04:52 <elodilles> i know that there are a ton of similar bugs already
16:05:04 <elodilles> some are more specific and some are general
16:05:18 <bauzas> elodilles: do you want to close it as a duplicate ?
16:05:34 <elodilles> if we have the exact same somewhere, then we could
16:05:44 <elodilles> otherwise we can keep it open...
16:06:11 <elodilles> also saw that maybe somewhat coupled with this 30 days old bug: https://bugs.launchpad.net/nova/+bug/1989232
16:06:30 <elodilles> i mean this is also opened around the same issue
16:07:50 <elodilles> otherwise i didn't have time to go deep into them
16:08:05 <bauzas> well, I dunno
16:08:09 <elodilles> so any idea about what to do with these bugs are welcome
16:08:38 <gibi> these need to be trobule shooted to find the root cause
16:08:47 <gibi> but I don't have time for that
16:09:05 <gibi> timeout is a the visible fault but there is deeper reasons why that happnes
16:09:28 <bauzas> ok, so let's continue to have this bug report then
16:09:43 <bauzas> and if someone wants to fix it, he/she could duplicate it if needed
16:09:57 <elodilles> ack
16:10:59 <bauzas> ok, any other bug report to look at ?
16:11:11 <elodilles> nothing else from me
16:12:52 <bauzas> ok, continuing then
16:13:09 <bauzas> gibi: can you be the bug baton for next week ?
16:13:15 <gibi> lets see
16:13:17 <bauzas> even if next week it will be the PTG ?
16:13:30 <gibi> I will take it
16:13:32 <bauzas> ok
16:13:34 <bauzas> thanks
16:13:41 <bauzas> #info bug baton is being passed to gibi
16:13:58 <bauzas> (sorry, passing it to you, as I was having the baton for 2 weeks :) )
16:14:13 <bauzas> moving on
16:14:15 <bauzas> #topic Gate status
16:14:20 <bauzas> #link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs
16:14:26 <bauzas> #link https://zuul.openstack.org/builds?project=openstack%2Fnova&project=openstack%2Fplacement&pipeline=periodic-weekly Nova&Placement periodic jobs status
16:14:28 <bauzas> heh :)
16:14:44 <bauzas> as you see, we have a timeout with the centos9-fips job
16:15:20 <bauzas> but I dunno for how long it was having this timeout
16:16:20 <elodilles> https://zuul.openstack.org/builds?job_name=tempest-centos9-stream-fips&project=openstack%2Fnova&project=openstack%2Fplacement&pipeline=periodic-weekly&skip=0
16:16:28 <elodilles> it seems always?
16:16:33 <bauzas> yes
16:17:53 <bauzas> I guess the owner of the job was adalee, right?.
16:18:40 <bauzas> this is bizarre, I don't see where the job run is timing out
16:20:01 <sean-k-mooney> well the default timeout i think is 2 hours
16:20:14 <sean-k-mooney> so it might just need a little longer sicne the fips job does a reboot
16:20:24 <bauzas> yeah but the job seems to have done
16:20:30 <bauzas> be* done
16:20:38 <bauzas> anyway, nothing urgent
16:20:53 <bauzas> this is just the fact we can't get the result
16:21:00 <bauzas> even if it looks tempest works fine
16:21:11 <sean-k-mooney> its posible it has addtional test in a post playbook
16:21:24 <bauzas> I don't know who wrote the job
16:21:34 <bauzas> but I'll try to find it
16:21:46 <bauzas> should be easy to find
16:22:24 <clarkb> https://zuul.openstack.org/build/5f6e6a2f65ee4a5e90754d07142fab9f/log/job-output.txt#23945-23946 shows where it is timing out
16:23:24 <bauzas> found who added it https://review.opendev.org/c/openstack/nova/+/831844
16:23:54 <bauzas> thanks clarkb, will look
16:24:31 <sean-k-mooney> ok its tempest full
16:24:47 <sean-k-mooney> so its two tempest runs
16:25:05 <bauzas> | RUN END RESULT_TIMED_OUT: [untrusted : opendev.org/openstack/tempest/playbooks/devstack-tempest.yaml@master]
16:25:09 <sean-k-mooney> the first one completed but the second that runs the slow test and senarior tests timed out
16:25:10 <bauzas> hmpf
16:25:24 <bauzas> yup
16:25:45 <sean-k-mooney> so ya it just need anther say 30 mins added to the  tiemout to be safe
16:26:05 <sean-k-mooney> it also could have just been a slow node
16:26:10 <clarkb> and maybe double check logs to see that something didn't get stuck there due to fips
16:26:19 <clarkb> there is a bit of a time delta between the timeout and tempest reporting anything
16:26:38 <clarkb> oh its only 15 seconds nevermind
16:26:42 <sean-k-mooney> yep
16:26:47 <sean-k-mooney> there isnt really a break in the logs
16:26:53 <clarkb> I mathed wrong the first time
16:27:08 <bauzas> yes, this is just a slow test
16:27:42 <bauzas> but should we modify the timeout for that job to be larger ?
16:27:46 <sean-k-mooney> i dont know how much time the reboot adds for fips but 2 hours is proably borderlien
16:28:01 <bauzas> we can DNM this at least
16:28:16 <bauzas> and verify if adding more timeout time helps
16:28:17 <sean-k-mooney> i would add 30mins and monitor it and see roughly how long it takes over the next few weeks
16:28:23 <bauzas> right
16:28:46 <bauzas> I can propose a patch
16:28:51 <sean-k-mooney> ack
16:29:21 <bauzas> #action bauzas to add a 30min more timeout for the centos9-fips periodic job so we will see whether it fixes the timeout
16:29:27 <bauzas> moving on
16:29:38 <bauzas> #info Please look at the gate failures and file a bug report with the gate-failure tag.
16:29:51 <sean-k-mooney> bauzas: its here https://opendev.org/openstack/tempest/src/branch/master/zuul.d/integrated-gate.yaml#L325 just incase your wondering where to do that
16:31:19 <bauzas> ack, thanks but I found it
16:31:40 <bauzas> hence the DNM patch in nova
16:31:51 <bauzas> to test it
16:32:07 <bauzas> with experimental
16:32:10 <bauzas> anyway
16:32:36 <bauzas> nothing to tell more about blind rechecks, I haven't looked at the numbers
16:32:38 <bauzas> moving on
16:32:48 <bauzas> #info STOP DOING BLIND RECHECKS aka. 'recheck' https://docs.openstack.org/project-team-guide/testing.html#how-to-handle-test-failures
16:32:53 <bauzas> #topic Release Planning
16:32:59 <bauzas> #link https://releases.openstack.org/zed/schedule.html
16:33:04 <bauzas> #info Zed GA was last week, kudos to the team.
16:33:10 <bauzas> we're officially done with Zed :)
16:33:21 <bauzas> so...
16:33:23 <bauzas> #link https://releases.openstack.org/antelope/schedule.html
16:33:28 <bauzas> #info Antelope-1 is planned in 5 weeks
16:33:38 <bauzas> nothing to tell it, the crux is for the next topic
16:33:50 <bauzas> #topic PTG planning
16:33:55 <bauzas> #link https://etherpad.opendev.org/p/nova-antelope-ptg Antelope PTG etherpad
16:34:00 <bauzas> #link https://ptg.opendev.org/ptg.html PTG schedule
16:34:09 <bauzas> #info Attempted schedule in the etherpad
16:34:21 <bauzas> yeah I did a bit of a topics scrambling
16:34:38 <bauzas> but honestly, we don't have a large agendat *yet*
16:35:04 <bauzas> I'll just move the nova-otherproject topics
16:35:29 <bauzas> for the moment, we have nova-neutron (Thur) and nova-ironic (reminder : *Monday*)
16:35:44 <bauzas> do people want to talk with cinder or other projects ?
16:35:52 <bauzas> I haven't got any ask from the cyborg project
16:36:06 <bauzas> or oslo or QA ?
16:36:25 <bauzas> or manila, or glance... :)
16:36:57 <gibi> nothing on my side
16:37:04 <sean-k-mooney> i think the ironic topic will fill any space we give it
16:37:14 <sean-k-mooney> but im not sure if we will be able to make progress remotely
16:37:57 <bauzas> hah
16:38:31 <bauzas> sean-k-mooney: as I already said a couple of times, I don't think we will find a solution
16:38:33 <bauzas> but,
16:38:55 <bauzas> at least I want our community to understand why we can't just modify an instance .host DB value
16:39:25 <sean-k-mooney> we also shoudl not have the hashring in the driver as we do today
16:39:35 <bauzas> if we're able to explain it even after all the comments dansmith and you did, well, it will be nice then
16:40:03 <sean-k-mooney> i look forward to your summary
16:40:16 <bauzas> anyway, I don't want to discuss this spec by now, we'll have time on Monday :)
16:40:21 * sean-k-mooney has flushed most of that out of my brain but will re read the spec
16:40:31 <bauzas> sean-k-mooney: /o\
16:40:46 <bauzas> but yeah, I'll summarize this session after the PTG :)
16:40:55 <bauzas> moving on
16:41:12 <bauzas> looks we can find other projects to discuss if we want during the PTG anyxway
16:41:32 <bauzas> as a reminder too, we could get pain points or feature requests from operators
16:41:51 <bauzas> #link https://etherpad.opendev.org/p/oct2022-ptg-operator-hour-nova Operator specific etherpad for our nova operator hours
16:41:59 <sean-k-mooney> i hope we do. even if they are ones we have gotten before
16:42:23 <bauzas> #topic Review priorities
16:42:29 <bauzas> #link https://review.opendev.org/q/status:open+(project:openstack/nova+OR+project:openstack/placement+OR+project:openstack/os-traits+OR+project:openstack/os-resource-classes+OR+project:openstack/os-vif+OR+project:openstack/python-novaclient+OR+project:openstack/osc-placement)+(label:Review-Priority%252B1+OR+label:Review-Priority%252B2)
16:42:55 <bauzas> I need to add some changes I have as a review-prio
16:43:05 <bauzas> apart from this, nothing to tell
16:43:17 <bauzas> next topic ?
16:43:27 <bauzas> (we'll discuss the review-prio flag at the PTG anyway)
16:43:42 <bauzas> #topic Stable Branches
16:43:51 <bauzas> elodilles: your dog.
16:43:53 <elodilles> o/
16:43:56 <elodilles> #info from stable/zed back till stable/ussuri branches' gates should be OK
16:44:07 <elodilles> #info stable/train seems broken again: devstack-gate's test-matrix playbook fails ("No module named yaml")
16:44:26 <elodilles> and it seems we already have a workaround for this: https://review.opendev.org/c/openstack/devstack-gate/+/860961
16:44:46 <elodilles> and for some information this came with Ansible 6
16:45:03 <elodilles> the rest are the same:
16:45:06 <elodilles> #info stable/stein (and older) are blocked: grenade and other devstack based jobs fail with the same timeout issue as stable/train was previously
16:45:13 <elodilles> #info stable branch status / gate failures tracking etherpad: https://etherpad.opendev.org/p/nova-stable-branch-ci
16:45:18 <elodilles> and that's all
16:45:46 <bauzas> cool thanks
16:46:01 <bauzas> can't wait for next week for discussing about stein :)
16:46:12 <elodilles> stein and older :)
16:46:35 <bauzas> oh you're right btw.
16:46:42 <bauzas> we can now backport again \o/
16:46:46 <bauzas> given Zed is released
16:46:49 <bauzas> huzzah
16:46:56 <elodilles> \o/
16:47:00 <gibi> :)
16:47:15 <bauzas> ok, anything to add ?
16:47:33 <bauzas> looks not
16:47:40 <bauzas> #topic Open discussion
16:47:47 <bauzas> just two paperwork items on my side
16:47:58 <bauzas> (bauzas) Cancel next week's meeting due to PTG
16:48:10 <bauzas> I forgot the question mark, but is this really needed ?
16:48:27 <bauzas> anyone having problems about cancelling ?
16:48:42 <bauzas> like, I can only attend by IRC, zoom is terrible to me
16:49:13 <bauzas> or, I can show my face, I just prefer us to continuously discussing over IRC
16:49:28 <bauzas> looks not
16:49:57 <gibi> no need for a meeting next week
16:50:00 <gibi> on IRC
16:50:01 <bauzas> #action bauzas to communicate (email and wiki) on Nov 18 nova meeting be CANCELLED
16:50:14 <bauzas> last item from me
16:50:23 <bauzas> (bauzas) Nov-1 is an holiday here, anyone wants to run the meeting ?
16:50:31 <bauzas> all said ^
16:51:00 <bauzas> I'm usually on perpetual PTO, but this time I'm asked to be on holiday
16:51:38 <bauzas> I guess most of the Europe is also having holiday at that day, if I trust my company calendar
16:51:52 <bauzas> shall we just cancel it too ?
16:52:11 <bauzas> the fun is, Nov-8 is ALSO an holiday for me the week after
16:52:13 <gibi> I'm of on Nov-1 too
16:52:16 <gibi> off
16:52:23 <elodilles> same for me
16:52:27 <gibi> cancel it
16:52:53 <bauzas> yeah, was just opening the door to anyone fancy running the meeting if he/she wanted
16:53:07 <gibi> I don't see much hands in the air
16:53:08 <gibi> :)
16:53:10 <dansmith> ++ cancel
16:53:12 <bauzas> gibi: on holiday nova-8 as well or not ?
16:53:17 <bauzas> shit
16:53:19 <bauzas> nov-8
16:53:58 <bauzas> oh nevermind
16:54:01 <bauzas> I'm stupid
16:54:09 <bauzas> I'm not good at knowing our holidays
16:54:17 <bauzas> I confused myself with may
16:54:35 <bauzas> nov-8 is a workday, that's nov-11 which is holiday
16:54:42 * bauzas feels ashamed
16:54:52 <bauzas> anyway, sold.
16:54:57 <gibi> I will be in Brno on nov-8
16:55:15 <gibi> so working but not on upstream nova :)
16:55:26 <bauzas> #undo
16:55:26 <opendevmeet> Removing item from minutes: #action bauzas to communicate (email and wiki) on Nov 18 nova meeting be CANCELLED
16:55:33 <gibi> Nova 18?
16:55:36 <bauzas> #action bauzas to communicate (email and wiki) on Oct 18 nova meeting be CANCELLED
16:55:37 <gibi> Nov 18?
16:55:45 <bauzas> #action bauzas to communicate (email and wiki) on Nov 1st nova meeting be CANCELLED
16:55:54 <gibi> getting better :)
16:56:01 <bauzas> I realized
16:56:02 <elodilles> and Oct 18
16:56:13 <gibi> oct 18 and nov 1 ACK
16:56:14 <elodilles> sorry, there it is
16:56:18 <elodilles> :X
16:56:23 <bauzas> after this and my confusion on holiday dates, that means it's time for me to end the meeting
16:56:24 <gibi> calendars are hard :D
16:57:08 <bauzas> thanks all and kudos for the hard work you made on Zed, btw.
16:57:20 <bauzas> #endmeeting