16:00:11 <bauzas> #startmeeting nova
16:00:11 <opendevmeet> Meeting started Tue Mar  7 16:00:11 2023 UTC and is due to finish in 60 minutes.  The chair is bauzas. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:11 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:11 <opendevmeet> The meeting name has been set to 'nova'
16:00:18 <gibi> o/
16:00:21 <bauzas> hey folks, welcome in the nova meeting
16:00:43 <bauzas> #link https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting
16:01:25 <bauzas> ok, guess people will join softly, let's start
16:01:26 <elodilles> o/
16:01:32 <bauzas> #topic Bugs (stuck/critical)
16:01:36 <bauzas> #info No Critical bug
16:01:45 <bauzas> (which is always good during a RC period)
16:01:45 <Uggla> o/
16:01:50 <bauzas> #link https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New 14 new untriaged bugs (-3 since the last meeting)
16:02:08 <bauzas> kudos to sean-k-mooney for the triage work, despite all the workload he already had
16:02:12 <bauzas> pretty impressive
16:02:24 <bauzas> sean-k-mooney: anything you wanna share to the team ?
16:03:30 <gibi> (many things happens in parallel real time)
16:03:33 <bauzas> ah, auniyal also helped, I see, we have two (optional, as always) triage etherpads this week
16:03:47 <bauzas> #link https://etherpad.opendev.org/p/nova-bug-triage-20230301
16:03:53 <bauzas> #link https://etherpad.opendev.org/p/nova-bug-triage-20230306
16:04:11 <bauzas> gibi: I wish I would have an internal semaphore more than equals to 2
16:04:44 <bauzas> even 2 is hard for me to handle
16:04:47 <bauzas> anyway
16:05:01 <sean-k-mooney> i think i mentioned the main too downstream
16:05:10 <sean-k-mooney> we broke windows + amd in zed
16:05:12 <bauzas> sean-k-mooney wrote something about https://bugs.launchpad.net/nova/+bug/2009280
16:05:18 <sean-k-mooney> we should fix that and there is a mig issue
16:05:36 <sean-k-mooney> those are the two highlights
16:06:07 <sean-k-mooney> https://bugs.launchpad.net/nova/+bug/2009280 im in two mind about how to fix
16:06:13 <bauzas> #link https://bugs.launchpad.net/nova/+bug/2009280 AMD-based computes may have a problem with Zed
16:06:23 <sean-k-mooney> we can turn that off  entirly or selectivly for amd hosts
16:06:48 <bauzas> sean-k-mooney: given we merged the enlightments in Zed, this is unfortunately not a RC candidate :(
16:06:49 <sean-k-mooney> i think turning off hv-evmc  for everythign it the better option for backporting
16:06:57 <sean-k-mooney> correct
16:07:00 <sean-k-mooney> just a normal bug
16:07:15 <sean-k-mooney> we can adress it and backprot after the offical relase is done
16:07:15 <bauzas> sean-k-mooney: I agree, disabling the whole feature seems to me easier and less errorprone
16:07:43 <bauzas> sean-k-mooney: putting the bug importance as High
16:07:56 <bauzas> it's basically blocking a nova boot on windows images if AMD-based
16:08:03 <sean-k-mooney> ack
16:08:04 <bauzas> that's quite a large impact
16:08:08 <sean-k-mooney> if you set the image property
16:08:25 <sean-k-mooney> but i was debating between medium and high so im ok with high
16:08:32 <bauzas> true, but how many production-level clouds are not setting the property ? :)
16:08:47 <bauzas> note : it was an open question :d
16:09:13 <sean-k-mooney> hehe i dont know but i think in either case we shoudl fix and backport it sonner rather then later
16:09:45 <sean-k-mooney> i might try and write a ptch for it but if other want to go ahead
16:10:01 <sean-k-mooney> i think we can move on for today
16:10:06 <bauzas> cool
16:10:10 <bauzas> and thanks
16:10:32 <bauzas> for the MIG bug, let's discuss this off the meeting, but upstream support on nvidia is very limited
16:11:01 <bauzas> unless the bug is reproducable on another hardware, I would tend to mark it Invalid either way
16:11:09 <sean-k-mooney> ya it borderlien a feature but i think it need some discussion outside the meeting
16:11:22 <bauzas> cool, I don't disagree
16:11:25 <bauzas> moving on
16:11:33 <bauzas> #info Add yourself in the team bug roster if you want to help https://etherpad.opendev.org/p/nova-bug-triage-roster
16:11:46 <bauzas> elodilles: well, this is your turn this week
16:11:56 <elodilles> bauzas: ack
16:11:57 <bauzas> how do you feel with all the releases patches coming up ?
16:12:05 <bauzas> + the reno ugly bug
16:12:22 <elodilles> bauzas: let's not postpone it now, i'll take the baton
16:12:25 <bauzas> (well, technically not a bug, but rather an unexpected lack of support)
16:12:38 <bauzas> cool cool
16:12:40 <bauzas> and thanks
16:12:49 <bauzas> elodilles: don't feel pressured
16:12:54 <bauzas> #info bug baton is being passed to elodilles
16:12:59 <bauzas> next topic
16:13:04 <bauzas> #topic Gate status
16:13:10 <bauzas> #link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs
16:13:15 <bauzas> #link https://etherpad.opendev.org/p/nova-ci-failures
16:13:29 <bauzas> so, we have less oom issues, which is good
16:13:41 <bauzas> mysql is less a noisy neighbor
16:13:55 <bauzas> but we only touched a few jobs
16:15:12 <bauzas> if you see other jobs that oom-kill mysql  but nova-next and nova-ceph-multistore, let us know
16:15:57 <bauzas> apart from that, I've seen other failures, but I hadn't a lot of time of digging
16:16:13 <bauzas> unless anyone wants to address a specific failure, we may need to move on
16:16:37 <bauzas> looks so
16:16:50 <bauzas> #link https://zuul.openstack.org/builds?project=openstack%2Fnova&project=openstack%2Fplacement&pipeline=periodic-weekly Nova&Placement periodic jobs status
16:17:17 <bauzas> so we had periodic tempest-integrated-placement on stable/yoga that timed out
16:18:14 <bauzas> I checked the log and it was just a slow job run
16:18:26 <bauzas> so hopefully things will order back at the next run
16:18:38 <bauzas> (and zed and master runs are OK)
16:18:56 <bauzas> #info Please look at the gate failures and file a bug report with the gate-failure tag.
16:19:00 <bauzas> #info STOP DOING BLIND RECHECKS aka. 'recheck' https://docs.openstack.org/project-team-guide/testing.html#how-to-handle-test-failures
16:19:06 <bauzas> #topic Release Planning
16:19:13 <bauzas> #link https://releases.openstack.org/antelope/schedule.html
16:19:19 <bauzas> #info Antelope-rc1 was this Monday
16:19:40 <bauzas> so, we're now in a duplicity period, where we are both on Bobcat and Antelope timeframes :)
16:20:01 <bauzas> if you've seen Tenet, you got it
16:20:08 <bauzas> (the movie)
16:20:19 <bauzas> now, we have a 2023.1 branch
16:20:43 <elodilles> ~o~
16:20:53 <bauzas> #info as a reminder, no backports are possible on the 2023.1 branch *unless* they are regression bugfixes
16:21:27 <bauzas> and accordingly all stable branches go on a quiet period since they're now dependent on the antelope branch
16:22:23 <bauzas> #link https://etherpad.opendev.org/p/nova-antelope-rc-potential
16:22:46 <bauzas> fwiw, our release notes are currently broken to be shown
16:22:51 <bauzas> for Antelope
16:23:19 <sean-k-mooney> we can still merge exsting backport but noting to 2023.1
16:23:22 <bauzas> context : https://review.opendev.org/c/openstack/nova/+/876553
16:23:24 <sean-k-mooney> that is not for an rc
16:23:56 <bauzas> sean-k-mooney: well, if the original patch is on Antelope, sure
16:24:35 <sean-k-mooney> we have plent of open backport on exsting stable branch that coudl use review
16:24:36 <bauzas> elodilles: iiuc, we'll need to enable the new reno ordering in our releasenotes ?
16:24:49 <bauzas> once the reno patch got landed
16:24:50 <elodilles> bauzas: fyi, reno workaround has landed, but not yet released
16:24:59 <bauzas> oh, missed that
16:25:08 <elodilles> bauzas: and we don't need to do anything, but use the new reno release
16:25:10 <bauzas> sean-k-mooney: yup, don't disagree
16:25:11 <sean-k-mooney> do we need to do any changes once its released
16:25:22 <sean-k-mooney> or will the release notes get rebuilt automatically
16:25:27 <bauzas> elodilles: ah, right, so dhellmann's proposition was accepted
16:25:36 <sean-k-mooney> ok cool
16:25:43 <sean-k-mooney> if we dont need to do anything great
16:25:54 <bauzas> sean-k-mooney: yeah, so there was a patch up for opting-in to the new reno ordering
16:26:05 <sean-k-mooney> by use the new reno does that mean increase the min version in requirements
16:26:09 <bauzas> but doug said he was ok with having it by default
16:26:24 <sean-k-mooney> or just realy on upper-constratis
16:26:28 <bauzas> sean-k-mooney: reno isn't part of our requirements
16:26:36 <bauzas> (unless I'm wrong)
16:26:38 <sean-k-mooney> its in our doc requirements i think
16:26:47 <bauzas> that's a specific target, which isn't docs
16:26:54 <elodilles> actually, it is part of upper-constraints
16:26:59 <sean-k-mooney> https://github.com/openstack/nova/blob/master/doc/requirements.txt#L13
16:27:06 <bauzas> https://github.com/openstack/nova/blob/master/tox.ini#L238
16:27:15 <elodilles> but for master branch (bobcat) I think it will be soon possible to bump
16:27:36 <sean-k-mooney> https://github.com/openstack/nova/blob/master/tox.ini#L238-L242
16:27:43 <sean-k-mooney> the release notes use the docs requirements file
16:27:47 <bauzas> sean-k-mooney: snap
16:28:01 <sean-k-mooney> so ya it will automatically get picked up from uc
16:28:12 <sean-k-mooney> but im wonderign if we want to block the older releases  or not
16:28:19 <sean-k-mooney> we dont have too but we could
16:28:30 <bauzas> it's backwards-compatible
16:28:44 <bauzas> so I don't think we need
16:28:57 <sean-k-mooney> yep i was more thinking of a signal to packages
16:29:06 <sean-k-mooney> but im fine with not bumping the min version
16:29:19 <bauzas> elodilles: the new reno release is a x. version ?
16:29:25 <bauzas> or a .y ?
16:29:35 <sean-k-mooney> all of the offical release notes will be update proeprly
16:29:41 <elodilles> it will be a MAJOR version bump
16:29:46 <bauzas> cool, so x.
16:29:57 <elodilles> yepp
16:30:04 <bauzas> sean-k-mooney: well technically, our master branch will need this
16:30:22 <sean-k-mooney> yep we would merge it in master and backprot to 2023.1
16:30:25 <bauzas> maybe the 2023.1 branch will need it too
16:30:37 <sean-k-mooney> reno>=4.0.0
16:30:48 <bauzas> but for other branches, they shouldn't mention the antelope notes
16:30:54 <sean-k-mooney> correct
16:31:07 <sean-k-mooney> we cant bump the requirement on older banches anyway
16:31:20 <bauzas> sean-k-mooney: the patch in question is https://review.opendev.org/c/openstack/nova/+/876553
16:31:22 <sean-k-mooney> we only have the opertunity to do it on 2023.1 because we have not releassed yet
16:31:47 <bauzas> yes
16:31:52 <bauzas> and we'll need it once we tag
16:32:15 <bauzas> a stable antelope release, I mean
16:32:27 <bauzas> oh wait no
16:32:34 <bauzas> maybe it's unnecessaru
16:32:46 <bauzas> our gits tags will mention the project tags, not the openstack release
16:32:59 <bauzas> the problem with reno is about the branches ordering
16:33:14 <bauzas> anyway, we'll figure that out
16:33:20 <bauzas> no need to spec it up too
16:33:23 <bauzas> here
16:33:49 <bauzas> we also need to merge https://review.opendev.org/c/openstack/nova/+/875621/ but I need to respin it
16:34:13 <sean-k-mooney> reno also supprots metadata in the commit log to denote a release
16:34:20 <sean-k-mooney> so that might also help
16:34:27 <sean-k-mooney> but ya we can proably move on
16:34:32 <sean-k-mooney> we willl figure this out
16:34:40 <bauzas> last but not the least (rc1 is a busy time for a PTL), I did a bit of launchpad scrubbing
16:35:05 <bauzas> #info https://blueprints.launchpad.net/nova/antelope now shows 'deferred' for approved but non-implemented Blueprints
16:35:33 <bauzas> the 2023.2 specs directory is open and I already fast-approved two specs that were accepted in Antelope
16:35:47 * sean-k-mooney clicks
16:36:04 <bauzas> I'll add the launchpad bobcat series after the meeting so technically, we can now start to approve new specs and specless blueprints
16:36:20 <sean-k-mooney> ack
16:36:35 <sean-k-mooney> we should start using the placement repo for placement thigns now too
16:36:38 <bauzas> that's it for this topic (larger than usual)
16:36:45 <sean-k-mooney> were started to do that partly last cycle but not consitently
16:36:58 <bauzas> sean-k-mooney: placement will also have its own launchpad bobcat series
16:37:05 <sean-k-mooney> yes
16:37:16 <bauzas> and yeah, we have to mimic the same than nova
16:37:25 <bauzas> as we agreed on stopping to use Storyboard
16:37:27 <sean-k-mooney> but we used a mix of stories and task instory board and some lanchpad stuff
16:37:39 <sean-k-mooney> so lets try and just use lanuchpad this time
16:37:58 <bauzas> that's a good point, I should somehow signal in Storyboard that we no longer support it for Placezment
16:38:23 <sean-k-mooney> ideally we should see if we can make it readonly and/or clsoe the exiting issues
16:38:38 <bauzas> #action bauzas to clean-up the stories in Storyboard and tell their owners to recreate a Launchpad bug or blueprint if they wish to carry it on
16:39:05 <bauzas> moving on
16:39:17 <bauzas> #info Uggla and auniyal are the new release liaisons https://review.opendev.org/c/openstack/releases/+/876758
16:39:29 <elodilles> \o/
16:39:43 * bauzas just hopes the releases team catches this patch :)
16:39:53 <bauzas> elodilles: don't feel overlooked :p
16:39:58 <elodilles> :]
16:40:05 <elodilles> i won't :)
16:40:09 <auniyal> we need to update the bug tracker link here https://opendev.org/openstack/placement
16:40:39 <bauzas> auniyal: excellent point, fancy creating a change against it ?
16:40:45 <auniyal> yes
16:40:52 <sean-k-mooney> i belvie that is in the governance repo
16:40:53 <bauzas> cool, thanks
16:41:02 <sean-k-mooney> posibly the main config repo
16:41:07 <bauzas> sean-k-mooney: I'm pretty it isn't
16:41:13 <bauzas> sure*
16:41:38 <bauzas> I've recently seen the project yaml files and I don't remember any mention of the bug tracking system
16:41:52 <bauzas> but I could have missed it
16:42:00 <bauzas> anyway, action is on me
16:42:22 <bauzas> next topic, and I'd like to see some quorum on it
16:42:24 <bauzas> #topic vPTG Planning
16:42:34 <bauzas> the usual reminder :
16:42:36 <bauzas> #link https://www.eventbrite.com/e/project-teams-gathering-march-2023-tickets-483971570997 Register your free ticket
16:43:02 <bauzas> even if this is a free event, it helps the foundation folks to correctly identify the attendance
16:43:21 <bauzas> #link https://etherpad.opendev.org/p/nova-bobcat-ptg Draft PTG etherpad
16:43:29 <bauzas> #info we need to agree on the agenda and how many slots we book
16:43:44 <bauzas> the ptg website is up
16:44:04 <bauzas> #link https://ptg.opendev.org/ptg.html
16:44:34 <bauzas> as you see, I've booked four hour slots per day between Tues and Friday
16:44:50 <bauzas> this is quite a large time, but I don't want to consume all of them
16:45:20 <bauzas> anyone having other opinions on the strawman proposal of 4x 4 hours ?
16:45:46 <bauzas> this would be 13:00-17:00 UTC
16:46:22 <bauzas> ideally, I'd also want to attend the TC sessions so I may require some co-chair if it conflicts
16:47:11 <bauzas> again, anyone having concerns with this proposal ?
16:47:57 <sean-k-mooney> no concerns really i assume we will subdevide the 4 hour into 4 50min slots and 10 min breaks
16:48:09 <bauzas> that's heard and agreed
16:48:21 <bauzas> I'm not a fan of long running days
16:48:26 <bauzas> this is exhausting at most.
16:49:05 <bauzas> I was somehow hoping the PTG could turn into some physical event before I stop running as a PTL, but I was wrong and I need to support it
16:49:57 <bauzas> and I think we'll possibly never return to a twice-a-year physical design event, so the ship has sailed and we need to get used to it
16:50:17 <bauzas> zoom FTW, yay.
16:50:36 <bauzas> anyway, I don't hear strong objections and the schedule is flexible
16:50:44 <bauzas> we can modify that later if we wish
16:51:09 <bauzas> this is just a matter of booking or unbooking a timeslot
16:51:20 <bauzas> moving on then
16:51:56 <bauzas> #action bauzas to notify the nova community by a ML thread of the proposed agenda for Nova sessions
16:52:34 <bauzas> again, as a reminder, don't forgot to add your PTG topics in advance
16:52:42 <bauzas> #topic Review priorities
16:52:49 <bauzas> #link https://review.opendev.org/q/status:open+(project:openstack/nova+OR+project:openstack/placement+OR+project:openstack/os-traits+OR+project:openstack/os-resource-classes+OR+project:openstack/os-vif+OR+project:openstack/python-novaclient+OR+project:openstack/osc-placement)+(label:Review-Priority%252B1+OR+label:Review-Priority%252B2)
16:52:54 <bauzas> moving on
16:53:00 <bauzas> #info As a reminder, cores eager to review changes can +1 to indicate their interest, +2 for committing to the review
16:53:04 <bauzas> #topic Stable Branches
16:53:09 <bauzas> elodilles: you have a quick time
16:53:16 <elodilles> ack
16:53:23 <elodilles> start with some repetition
16:53:26 <elodilles> #info stable/2023.1 branches were cut
16:53:34 <elodilles> #info stable gates seem to be OK - though it's hard to merge patches due to intermittent failures
16:53:43 <bauzas> unfortunately true
16:53:51 <elodilles> on older branches especially
16:53:54 <elodilles> #info stable branch status / gate failures tracking etherpad: https://etherpad.opendev.org/p/nova-stable-branch-ci
16:54:02 <elodilles> and that's all
16:54:07 <elodilles> to be quick
16:54:25 <bauzas> thanks a lot
16:54:33 <bauzas> last topic
16:54:40 <bauzas> #topic Open discussion
16:54:46 <bauzas> we have an item
16:54:53 <bauzas> (astupnik) Known Issues section of Release Notes contains only  known issues added during specific release cycle instead of complete  list of known issues (all issues notes from nova/releasenotes/notes). I  am wondering if this is normal or there is some bug in documentation  framework? Example: min-bandwidth-workaround-0533ad03f67592a9.yaml  introduced using I41f42c1a7595d9e6a73d1261bf1ac1d47ddadcdf and still  affecting Nova.
16:55:04 <bauzas> tl;dr: answer is yes, this is expected behaviour
16:55:40 <bauzas> our release notes tooling provides a list of items based on YAML files that exist on a git branch
16:56:00 <bauzas> and accordingly heavily relies on git for the ordering and sorting
16:56:16 <astupnik> ack. I am wondering what's the best approach for tracking known issues?
16:56:33 <bauzas> known issues that aren't fixed ?
16:56:43 <sean-k-mooney> we dont really want to retoactivly delete the old release notes for knwon issues
16:56:56 <bauzas> that's a very good question and I don't have an existing solution on top of my head
16:56:57 <sean-k-mooney> we generally try ot refence the previous know issue in the patch that fixes it
16:57:02 <sean-k-mooney> and update the docs where approprate
16:57:35 <astupnik> the problem is that known issues in release notes only contain new issues, not ones that existed before release...
16:57:50 <sean-k-mooney> correct that is waht we wanted
16:57:55 <astupnik> ack
16:57:56 <bauzas> well, you have a list of known issues per release here https://docs.openstack.org/releasenotes/nova/zed.html
16:57:59 <bauzas> and others
16:58:17 <sean-k-mooney> we also tend to do thins like this https://docs.openstack.org/nova/latest/admin/virtual-gpu.html#caveats
16:58:39 <bauzas> but we don't have a specific page that references a list of known issues over all the releases, despite that being possible with the reno tool, I think
16:59:04 <sean-k-mooney> i dont know how we would mark them as resolved
16:59:18 <bauzas> the best remains to bug the worldwide intelligence that stays on that channel :)
16:59:23 <sean-k-mooney> you dont really want a list of all know issue ever just the ones that are still outstanding
16:59:34 <astupnik> thank you for clarifications. Simplest way I found is to look for "issue" in release notes folder. I was wondering if there is more user-friendly option.
16:59:40 <bauzas> sean-k-mooney: yeah tracking is a problem
16:59:46 <sean-k-mooney> im sure chatgpt can fix this for us :P
16:59:53 <bauzas> ideally a known issue should mention a bug report
16:59:55 <astupnik> sean-k-mooney: I want all unresolved issues
17:00:13 <sean-k-mooney> that is our launchpad open bug list
17:00:19 <astupnik> heh, ok
17:00:21 <bauzas> so people should know the status of that issue by querying the bug tracker
17:00:36 <bauzas> sean-k-mooney: on a side note, I have a PTG topic on it :)
17:00:45 <sean-k-mooney> ack
17:00:45 <bauzas> -ETOOMANY open bugs
17:00:58 <bauzas> let's just axe them ::)
17:01:06 <bauzas> anyway, we're overtime
17:01:08 <bauzas> thanks all
17:01:10 <bauzas> #endmeeting