16:00:23 <bauzas> #startmeeting nova
16:00:23 <opendevmeet> Meeting started Tue Jul 25 16:00:23 2023 UTC and is due to finish in 60 minutes.  The chair is bauzas. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:23 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:23 <opendevmeet> The meeting name has been set to 'nova'
16:00:39 <bauzas> hey, let's try to have a 10-min meeting if we can
16:00:43 <dansmith> o/
16:00:51 <bauzas> #link https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting
16:00:52 <gmann> o/
16:01:01 <auniyal> o/
16:01:22 <elodilles> o/
16:01:24 <bauzas> ok, starting
16:01:34 <bauzas> if someone wants to continue discussing, I can pass the chair
16:01:40 <bauzas> #topic Bugs (stuck/critical)
16:01:44 <bauzas> #info No Critical bug
16:01:48 <bauzas> #link https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New 40 new untriaged bugs (+0 since the last meeting)
16:01:58 <bauzas> Uggla made a good effort on triaging this week, thanks
16:02:11 <bauzas> he shared to me https://etherpad.opendev.org/p/nova-bug-triage-20230725
16:02:27 <bauzas> #info Add yourself in the team bug roster if you want to help https://etherpad.opendev.org/p/nova-bug-triage-roster
16:02:39 <bauzas> elodilles: can you be the baton owner for this week ?
16:02:44 <elodilles> yepp
16:02:49 <bauzas> cool thanks !
16:02:53 <bauzas> #info bug baton is being passed to elodilles
16:02:57 <bauzas> any bug to discuss ?
16:02:58 <elodilles> i'll be off on Friday, but otherwise fine!
16:03:09 <bauzas> elodilles: I'll be off next week so :)
16:03:18 <bauzas> #topic Gate status
16:03:18 <elodilles> touche ;)
16:03:22 <bauzas> #link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs
16:03:27 <bauzas> #link https://zuul.openstack.org/builds?project=openstack%2Fnova&project=openstack%2Fplacement&pipeline=periodic-weekly Nova&Placement periodic jobs status
16:03:31 <bauzas> #info nova-emulation back to work
16:03:35 <bauzas> #link https://zuul.openstack.org/build/2eb47b5dcb944b7eb03224e6ee599a57
16:03:41 <dansmith> gmann and I have been working on a bunch of patches to address job timeout issues
16:03:46 <bauzas> now we have a failure on tempest-integrated-placement
16:03:54 <dansmith> still lots in flight, but we're hopeful that it will improve those issues at least
16:03:56 <bauzas> but we'll see next week
16:04:01 <dansmith> but there are still lots of other spurious fails
16:04:10 <bauzas> dansmith: I was about to mention the other pipelines but the periodic-weekly
16:04:21 <bauzas> #info Please look at the gate failures and file a bug report with the gate-failure tag.
16:04:26 <dansmith> really need all hands on deck to improve this situation
16:04:32 <gmann> yeah, timeout now a days is due to multiple reasons and we are trying to improve a few of them
16:04:33 <bauzas> dansmith: so, yeah, there are a bunch of patches in flight
16:05:03 <bauzas> dansmith: gmann: I don't know if you have seen my pings today but yeah, there is another suspect
16:05:22 <bauzas> #link https://bugs.launchpad.net/tempest/+bug/2024859
16:05:26 <dansmith> bauzas: the ping was about the mkfs issue?
16:05:33 <dansmith> that's known and unclear how to resolve, AFAICT
16:05:34 <bauzas> yup
16:05:35 <gmann> bauzas: I saw that, I did not look into that expect melwitt trying with some change
16:05:42 <gmann> yeah mkfs
16:05:50 <dansmith> but I'm highly concerned that we've got a group of workers that is suddenly very very IO constrained
16:06:03 <bauzas> sean-k-mooney has a WIP proposal
16:06:04 <dansmith> because we're seeing 100% slowdown on some nodes that seems IO-related, like that mkfs issue
16:06:18 <dansmith> melwitt had a proposal but it turned out not to help, last I checked
16:06:24 <gmann> yeah
16:06:27 <sean-k-mooney> the nova-lvm job passed with it btu i dont know if it was failing consitently before
16:06:28 <bauzas> about using disk cachinh
16:06:39 <bauzas> but I'm torn enabling it
16:06:49 <sean-k-mooney> https://review.opendev.org/c/openstack/nova/+/889383
16:07:00 <bauzas> anyway, I don't disagree, we need to continue digging into it
16:07:01 <sean-k-mooney> well its partly enabled already
16:07:04 <gmann> sean-k-mooney: might not be so consistent but I see this 2-3 times in a week
16:07:30 <bauzas> dansmith: gmann: from the most remaining issues, is nova just a coal canary or is responsible for some of them ?
16:07:31 <sean-k-mooney> i was suggeting setting  disk_cachemode: "file=none,block=unsafe,network=writeback"
16:07:43 <sean-k-mooney> block could also be writeback
16:08:22 <dansmith> bauzas: need more triage to know really
16:08:30 <dansmith> bauzas: I've been focusing on the timeout stuff for a week
16:08:47 <gmann> bauzas: timeout is more of test runner unbalancing on test worker and slow test etc etc
16:08:51 <bauzas> ok, I'll try to get more hands on it before I leave
16:09:08 <gmann> because timeout hold many of the gate fixes so that should be fixed first :)
16:09:18 <bauzas> ack
16:09:31 <bauzas> moving on
16:09:33 <bauzas> #topic Release Planning
16:09:37 <bauzas> #link https://releases.openstack.org/bobcat/schedule.html
16:09:41 <bauzas> #link https://etherpad.opendev.org/p/nova-bobcat-blueprint-status Etherpad for tracking blueprints status
16:09:45 <bauzas> #info 5 weeks before FeatureFreeze
16:10:00 <bauzas> as I said before, my attention will go to features sets this week
16:10:29 <bauzas> in theory, today was a Feature Review Day
16:10:40 <bauzas> but I feel many people having different priorities
16:10:50 <bauzas> so I haven't really called it
16:11:14 <bauzas> #topic Review priorities
16:11:18 <bauzas> #link https://review.opendev.org/q/status:open+(project:openstack/nova+OR+project:openstack/placement+OR+project:openstack/os-traits+OR+project:openstack/os-resource-classes+OR+project:openstack/os-vif+OR+project:openstack/python-novaclient+OR+project:openstack/osc-placement)+(label:Review-Priority%252B1+OR+label:Review-Priority%252B2)
16:11:22 <bauzas> #info As a reminder, people eager to review changes can +1 to indicate their interest, +2 for asking cores to also review
16:11:26 <bauzas> #topic Stable Branches
16:11:31 <bauzas> elodilles: your time
16:11:35 <elodilles> #info nova-lvm / nova-ceph-multistore jobs are fixed on all branches (2023.1, zed, yoga) \o/
16:11:38 <elodilles> #info stable/victoria gate fix has also landed
16:11:40 <elodilles> #info gates from 2023.1 back till train should be OK
16:11:47 <elodilles> #info stable branch status / gate failures tracking etherpad: https://etherpad.opendev.org/p/nova-stable-branch-ci
16:11:52 <elodilles> EOM
16:11:55 <bauzas> thanks
16:12:02 <bauzas> #info train-eol patch proposed https://review.opendev.org/c/openstack/releases/+/885365
16:12:09 <bauzas> we're missing a second release core vote
16:12:14 <bauzas> but should happen eventually
16:12:20 <elodilles> yepp
16:12:32 <bauzas> auniyal had a point, I'll rephrase it
16:12:44 <bauzas> #info Please review these backport patches of stable 2023.1, zed and yoga for next minor release
16:12:49 <bauzas> #info most of these already have one +2.
16:12:52 <bauzas> #link https://etherpad.opendev.org/p/release-liaison-PatchesToReview
16:13:09 <bauzas> and the stable branches are now back in healthy state
16:13:18 <bauzas> #topic Open discussion
16:13:26 <bauzas> I have one item
16:14:17 <bauzas> as I said previously, I'll be on PTO starting next Tues
16:14:31 <bauzas> gibi volunteered for running the Aug 15 meeting
16:14:38 <bauzas> (and Aug 22)
16:14:49 <bauzas> but we may need someone to chair the Aug 1 and Aug 8 meetings
16:14:57 <bauzas> or we skip them
16:15:40 <gibi> jepp I still OK with the 15 and 22
16:15:51 <bauzas> so, anyone fancy volunteering ? if not, nevermind, I'll send an email to cancel the two weeks
16:16:31 <bauzas> I can try to handle Aug 1 meeting as I won't (yet) be on a plane, but I can't promise
16:16:48 <bauzas> ok, let's consider the meetings cancelled
16:17:04 <bauzas> #info Aug 1 and Aug 8 Nova meetings are CANCELLED
16:17:15 <bauzas> #action bauzas to notify the ML accordingly
16:17:17 <bauzas> that's it
16:17:19 <bauzas> for me
16:17:24 <bauzas> anything anyone ?
16:18:24 <bauzas> ok, thanks then
16:18:41 <bauzas> #endmeeting