16:00:32 <bauzas> #startmeeting nova
16:00:32 <opendevmeet> Meeting started Tue Sep 28 16:00:32 2021 UTC and is due to finish in 60 minutes.  The chair is bauzas. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:32 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:32 <opendevmeet> The meeting name has been set to 'nova'
16:00:41 <bauzas> hola !
16:01:17 <bauzas> #link https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting
16:01:40 <bauzas> thanks for being here
16:01:44 <bauzas> who's around ?
16:01:56 * stephenfin is lurking
16:01:57 <elodilles> o/
16:02:24 <gibi> \o
16:02:44 <bauzas> let's start
16:02:52 <bauzas> #topic Bugs (stuck/critical)
16:03:03 <bauzas> No Critical bug
16:03:16 <bauzas> #link 13 new untriaged bugs (+0 since the last meeting): #link https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New
16:03:25 <bauzas> thanks for people who triaged them :)
16:03:41 <bauzas> one open bug marked with xena-rc-potential tag #link https://bugs.launchpad.net/nova/+bugs?field.tag=xena-rc-potential
16:03:42 <gibi> bauzas: was it you ?:)
16:03:51 <bauzas> gibi: shhhhhhhhtttt :p
16:04:16 <bauzas> so, https://bugs.launchpad.net/nova/+bug/1945323 is proposed against RC*
16:04:18 <gibi> :x
16:04:40 <bauzas> tl;dr: last osbrick release is creating problems for NVMe volumes
16:05:01 <bauzas> the fix is provided https://review.opendev.org/c/openstack/nova/+/811413 but not yet merged
16:05:44 <bauzas> we'll discuss about RC releases in the next topic, but here the question is, should we try to merge ^ before Thursday as it's the deadline for RC
16:05:49 <bauzas> ?
16:06:01 <bauzas> lyarwood had concerns and I have an opinion
16:06:14 <gibi> I think if the fix is valid then we should aim to land it
16:06:21 <gibi> but the gate state might not allow that
16:06:37 <bauzas> given the gate is bad at the moment (again, we will discuss this later), we could try to land it but at least we should provide a releasenotes documentation
16:06:52 <bauzas> telling about the bug
16:07:25 <bauzas> so, in case we can't land the fix, operators would still see the documentation and for example packagers wouldn't use the latest osbrick if they want to support NVMe
16:07:30 <bauzas> thoughts ?
16:07:46 <bauzas> remember that we have a "bugs" section in nova relnotes
16:07:55 <bauzas> (this gives a "known bugs" section)
16:08:09 <gibi> I agree to land a reno, regardless of the fix
16:08:39 <bauzas> given I proposed, I need to write it
16:08:48 <bauzas> but someone could write it if they want :)
16:09:01 <bauzas> e0ne: around ?
16:09:20 <bauzas> we're discussing about your open bug https://bugs.launchpad.net/nova/+bug/1945323
16:09:27 <e0ne> bauzas: hi
16:09:36 <bauzas> and we're proposing you to write a reno change documenting the known bug
16:10:12 <bauzas> so in case we can't merge the fix before thursday's deadline for RCs, operators would have a documentation explaining how to workaround
16:10:24 <e0ne> ok
16:10:29 <e0ne> fair enough
16:10:32 <bauzas> e0ne: can you write it ?
16:10:43 <e0ne> I understand that we found it too late
16:10:49 <bauzas> I wish you would be better than me explaining the issue and how to workaround for an ops
16:10:54 <e0ne> bauzas: ok, will do it a bit later today
16:11:14 <bauzas> e0ne: the problem is not really about the time, this is more about the gate block we have atm
16:11:22 <lyarwood> sorry back
16:11:33 <gmann> on gate url issue  - I think we need to go with no trailing '/' in both place but we need to push changes on stable branches first to make grenade job work. basically in reverse order. I will try after an hr or so, currently in another meeting.
16:11:39 <bauzas> lyarwood: you had concerns on this bug, btw. take the mic
16:12:06 <bauzas> gmann: we'll discuss the gate problems litterally after this one :)
16:12:11 <gmann> k
16:12:21 <lyarwood> bauzas: I don't even think it's a valid nova bug at the moment, it doesn't look like os-brick has actually changed it's return value with that fix
16:12:36 <lyarwood> bauzas: and even if it had that's on os-brick not nova
16:12:54 <bauzas> do we have a reproduction test ?
16:12:58 <lyarwood> tbh this just smells like a poorly tested codepath
16:12:58 <bauzas> e0ne: ^
16:13:17 <e0ne> lyarwood: there was an issue with CI. it's fixed now
16:13:50 <bauzas> e0ne: you mean that CI wasn't using the latest osbrick hence not seeing the issue ?
16:13:54 <e0ne> mellanox CI should fail with this issue on os-brick patches now
16:14:00 <bauzas> ack
16:14:51 <bauzas> fwiw, gibi also has another bugfix he wants to merge that's not a regression, so I guess we will provide a Xena stable release like 2 weeks after GA
16:15:05 <e0ne> bauzas: when I merged that patch I was confused by mellanox CI Cinder-tgtadm
16:15:14 <lyarwood> yup either way I don't think this needs to land in rc
16:15:29 <gibi> bauzas: yupp this is a 1/3 xena regression and 2/3 existing bug in wallaby https://bugs.launchpad.net/nova/+bug/1945310
16:15:59 <gibi> bauzas: and I'm fine to include it in 24.0.1 later
16:16:09 <bauzas> ok, then let's do this
16:16:12 <e0ne> lyarwood: ok, I totally understand you. thanks for letting me discuss it
16:16:24 <bauzas> e0ne: ping me when you're done with the reno patch
16:16:34 <e0ne> bauzas: ok, will do
16:16:35 <bauzas> e0ne: use the 'bugs' section for it please
16:16:50 <bauzas> cool
16:17:06 <bauzas> and for the fix itself, let's review it sooner than later in master
16:17:30 <bauzas> so we could backport it to stable/xena as quickly as we can
16:17:46 <bauzas> lyarwood: do you had time to review the fix itself ?
16:18:02 <lyarwood> I already have several times
16:18:12 <bauzas> nice
16:18:25 <bauzas> so we need a second core volunteering for reviewing it :)
16:18:27 <lyarwood> https://review.opendev.org/c/openstack/os-brick/+/800014/7/os_brick/initiator/connectors/nvmeof.py#199 and FWIW I think this is a regression in os-briclk
16:18:42 <bauzas> hah
16:18:52 <bauzas> so, no nova impact ?
16:19:00 <e0ne> lyarwood: I'll discuss it in cinder meeting tomorrow too
16:19:11 <bauzas> ok, then let's clarify the solution later
16:19:29 <bauzas> the reno doc is the priority, the fix itself needs a bit more care I guess
16:19:31 <bauzas> moving on
16:19:41 <bauzas> #topic Gate status
16:19:48 <bauzas> Nova gate bugs #link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure
16:19:55 <bauzas> we triaged a few other ones
16:20:00 <bauzas> but,
16:20:05 <bauzas> CI is blocked due to nova-grenade-multinode issue and apache placement issue
16:20:08 <bauzas> gmann: your turn
16:20:38 <gmann> frickler: ianw_pto figured it out on apache2 security bugs, you might have seen in email
16:21:03 <gmann> we have two option 1. no trailing '/' 2. having trailing '/' in both url and target
16:21:26 <gmann> later one seems causing keystone error so we might need to go with first one
16:21:30 <bauzas> grabbing the link for the sake of understanding
16:21:45 <gibi> that is the current fix proposal #link https://review.opendev.org/c/openstack/devstack/+/811399
16:21:54 <bauzas> #link http://lists.openstack.org/pipermail/openstack-discuss/2021-September/025101.html
16:22:12 <gmann> but we need to do it on stable branches first due to grenade jobs , I will try to push changes in stable in reverse order and see if it all work till master.
16:22:25 <bauzas> tl:dr: apache config is swallowing the placement endpoint URL
16:22:46 <gmann> so in summary it may take time, hope today we get fixes in
16:23:08 <gmann> gibi: yeah and ceph things on victoria is another thing
16:23:08 <gibi> gmann: the stable/victoria fix failing with ceph issue now https://review.opendev.org/c/openstack/devstack/+/811399
16:23:17 <bauzas> gmann: any pointer for the nova-grenade-multinode issue ?
16:23:20 <gibi> gmann: yepp, it is not relataed but it is blocking
16:23:26 <gmann> true
16:23:39 <bauzas> gmann: can't see any bug for either the apache issue or the grenade thing
16:23:51 <gmann> bauzas:  nova-grenade-multinode  you mean for neutron-truck extension right?
16:23:56 <bauzas> this is correct
16:23:57 <gibi> gmann: right
16:24:20 <gmann> yeah, I have lyarwood patch in list. let's get this apache things in then I will take a look into that
16:24:28 <lyarwood> https://review.opendev.org/c/openstack/grenade/+/811117 is the fix for that
16:24:31 <lyarwood> I can create a bug
16:24:32 <gibi> yepp we need to fix the apache first anyhow
16:24:34 <gmann> yeah ^^
16:24:55 <gmann> lyarwood: that will be good to track, please file
16:25:00 <bauzas> #link https://review.opendev.org/c/openstack/grenade/+/811117
16:25:15 <bauzas> gmann: if you can do the same for the devstack bug, it would be appreciated
16:25:36 <gmann> bauzas: sure, we have not logged yet :). will do
16:25:47 <gmann> lyarwood:  is nova-grenade-multinode  breaking in stable too?
16:26:06 <lyarwood> Yeah everything with focal
16:26:10 <lyarwood> oh wait
16:26:11 <bauzas> \o/
16:26:15 <lyarwood> I'm mixing things
16:26:26 <gmann> we did extension cap in devstack stable/ but need to check if we have neutron-truck in the list there or no
16:26:34 <lyarwood> no AFAICT that's just stable/xena to master I think
16:27:04 <gmann> lyarwood: ok, stable/xena is not capped with extension yet
16:27:08 <bauzas> we definitely need bug reports to understand the impact :)
16:27:22 <gibi> I also think wallaby is not affecte by the trunk testing problem in grenade
16:27:43 <bauzas> stable/xena is the top priority I'd say
16:27:49 <lyarwood> gmann: would you have a link to the cap on stable/wallaby?
16:27:56 <gmann> gibi: ok, need to check if that extension is enbaled or not may be we are just skipping it :) but I will check
16:28:02 <bauzas> we can delay stable releases
16:28:10 <bauzas> but we can't delay Xena GA
16:28:27 <gmann> lyarwood: https://github.com/openstack/devstack/blob/stable/wallaby/lib/tempest#L669-L685
16:28:29 <gmann> #link https://github.com/openstack/devstack/blob/stable/wallaby/lib/tempest#L669-L685
16:28:30 <gibi> but we have to fix stable grenade first to fix master to have GA :D
16:28:43 <bauzas> yeah, tangled.
16:28:54 <bauzas> and this is not a Disney.
16:29:06 <bauzas> anyway, I guess we can move on ?
16:29:07 <gibi> with this apache update just now Murphy hit us hard
16:29:56 <bauzas> with this last words from gibi full of hope, I guess we can move on
16:29:57 <bauzas> :D
16:30:07 <gmann> I will try to push extensions cap in xena too but anyways we have to get apache fix in first
16:30:20 <bauzas> Placement periodic job status #link https://zuul.openstack.org/builds?project=openstack%2Fplacement&pipeline=periodic-weekly
16:30:29 <bauzas> Please look at the gate failures, file a bug, and add an  elastic-recheck signature in the opendev/elastic-recheck repo (example:  #link https://review.opendev.org/#/c/759967)
16:30:34 <bauzas> #topic Release Planning
16:30:40 <bauzas> Release tracking etherpad #link https://etherpad.opendev.org/p/nova-xena-rc-potential
16:31:00 <bauzas> RC2 proposals are up for approval #link https://review.opendev.org/c/openstack/releases/+/811092 and https://review.opendev.org/c/openstack/releases/+/811094
16:31:08 <bauzas> Remember to propose regression bugfixes for a new RC with nova-xena-rc-potential
16:31:13 <bauzas> Final RC to be proposed before end of Thursday Oct 1st (only if needed)
16:31:20 <bauzas> (gibi): Potential Xena regression #link https://bugs.launchpad.net/nova/+bug/1945310
16:31:32 <gibi> yepp
16:31:34 <gibi> as I said about
16:31:35 <bauzas> gibi: the floor is yours
16:31:47 <gibi> it is 1/3 xena regression 2/3 bug in wallaby
16:31:55 <gibi> fix is proposed
16:32:01 <gibi> both on master, xena, wallaby
16:32:12 <gibi> but I'm OK to delay this until 24.0.1
16:32:18 <bauzas> for this 1/3, and given the gate, can we assume we'll deliver the fix only as a stable release patch ?
16:32:27 <bauzas> ok, sold
16:32:32 <gibi> yepp
16:32:39 <bauzas> I'll approve the RC2 patches tonight then
16:32:44 <lyarwood> https://bugs.launchpad.net/grenade/+bug/1945346 for the neutron-trunk issue
16:32:45 <gibi> it only affect min bw + live migration + non-admin token
16:32:46 <bauzas> until someone has strong objections
16:33:04 <bauzas> we *can* deliver a RC3 tomorrow or thursday if needed
16:33:11 <gibi> bauzas: I'm OK to land RC2 both for nova and placement now
16:33:13 <bauzas> keeping in mind we're in a bad shape for it tho
16:33:25 <bauzas> but, crossed fingers
16:33:44 <bauzas> gibi: ack, will tell the release folks we're good to go for the current proposals
16:33:58 <bauzas> #topic Review priorities
16:33:59 <elodilles> (bauzas: ack :))
16:34:04 <bauzas> https://review.opendev.org/q/status:open+(project:openstack/nova+OR+project:openstack/placement)+label:Review-Priority%252B1
16:34:18 <bauzas> I've used a bit this rp label
16:34:41 <bauzas> I'll try to do reviews for the ones I've taken before end of this week
16:35:38 <bauzas> if someone btw. has patches to review, please ping me on IRC so I can mark them with the Review-Priority flag
16:35:53 <bauzas> also, avoid CC'ing nova-core in gerrit, this is useless
16:36:35 <artom> bauzas, since you asked https://review.opendev.org/c/openstack/nova/+/808474/3 :)
16:36:38 <bauzas> (until we discuss about what to do with this tag at the PTG)
16:37:27 <bauzas> artom: your wish come true
16:37:42 <bauzas> #topic PTG Planning
16:37:50 <bauzas> every info is in the PTG etherpad #link https://etherpad.opendev.org/p/nova-yoga-ptg
16:37:53 <artom> Wait, we're in the meeting, are we?
16:37:57 <artom> Sorry, completely out of it
16:37:57 <bauzas> If you see a need for a specific cross project section then please let me know
16:38:14 <bauzas> artom: you're a terrible coworker
16:38:35 <artom> I'm a terrible human being in general, but that's besides the point ;)
16:38:47 <bauzas> fwiw, I can start to write up a proposed PTG agenda as a tentative next week
16:39:38 <bauzas> at least we need to sync up with the oslo team about when to have a cross-project discussion for the policy thing (/cc gmann)
16:40:06 <bauzas> #topic Stable Branches
16:40:12 <bauzas> nova's stable/ussuri and stable/train are blocked (due to latest virtualenv uses latest setuptools which removed use_2to3)
16:40:29 <bauzas> (last but not the least large topic)
16:40:35 <bauzas> large discussion occurred on the ML #link http://lists.openstack.org/pipermail/openstack-discuss/2021-September/024987.html
16:40:42 <bauzas> looks a consensus is going forward with pinning virtualenv on stable branches # link https://review.opendev.org/c/openstack/nova/+/810461
16:40:58 <bauzas> tbc, we WON'T pin virtualenv (and setuptools) on master
16:41:21 <bauzas> but we WILL pin virtualenv (and setuptools accordingly) on the required stable branches
16:41:32 <bauzas> anyone disagreeing with this plan ?
16:42:07 <gibi> (as a side note this only pins setuptools if you use tox, if you install the package by other means the this solution doesn't help)
16:42:08 <bauzas> gibi: you're the mastermind of this plan, you could add things if you wish
16:42:20 <bauzas> gibi: excellent detail indeed
16:42:27 <bauzas> the pin is only done on tox.ini
16:42:34 <gibi> currently we have issue with lower-constraints and tox, so it helps us
16:42:47 <gibi> but on the ML there was other reports where devstack install failed on stable
16:42:54 <bauzas> packagers are free to pick any dependency they want, for sure
16:43:12 <gibi> so in the long run we might need additional fix for devstack too
16:43:45 <bauzas> gibi: you mean, we should pin virtualenv in stable devstack releases ?
16:44:04 <bauzas> (which makes sense)
16:44:07 <gibi> bauzas: potentially yes. I haven't see such failure myself yet
16:44:15 <gibi> so I'm not rushing with it
16:44:18 <bauzas> interesting
16:44:25 <bauzas> any distro impacted so far ?
16:44:31 <bauzas> (for devstack)
16:44:46 <gibi> I have no details
16:44:50 <bauzas> awesome
16:44:52 <elodilles> as I understood devstack is not hit with this issue in general (but I might be wrong)
16:44:56 <bauzas> and I guess no bug report
16:45:10 <bauzas> elodilles: what I guess is that it depends on the related OS
16:45:27 <elodilles> bauzas: that sounds possible
16:45:58 <bauzas> then, the one who claims there is a bug should explain which env they use
16:46:07 <bauzas> so we could reproduce it
16:46:31 <bauzas> for the moment, let's not try to boil the ocean
16:46:42 <gibi> bauzas: as I said I saw reports on ML but I did not see it in nova yet, so probably the devstack part is not a nova issue right now
16:47:04 <bauzas> gibi: ok, so let's move on with the tox change
16:47:12 <gibi> ack
16:47:25 <bauzas> but again, we'll probably hit the other gate issues
16:47:27 <bauzas> anyway
16:47:40 <bauzas> this release sounds so fun
16:47:54 <bauzas> moving on
16:47:57 <bauzas> #topic Stable Branches
16:48:03 <bauzas> whoops
16:48:10 <bauzas> placement branches are also impacted
16:48:14 <bauzas> we waited with the wallaby release until xena RC2 gets released,  now we could release from victoria and ussuri together with wallaby as  well
16:48:46 <elodilles> i can prepare the release patches for victoria and ussuri if needed
16:49:04 <bauzas> elodilles: I guess we'll need some kind of smart stitching between all the concurrent patches so we can unblock things one after all
16:49:46 <bauzas> elodilles: sure, let's discuss this tomorrow
16:50:05 <elodilles> bauzas: ack
16:50:11 <bauzas> other points on stable to discuss ?
16:50:21 <elodilles> bauzas: actually, i'll be on pto tomorrow, but be back on thursday
16:50:42 <elodilles> but anyway, nothing else from me :X
16:50:45 <bauzas> elodilles: ok, then ping me on thursday so we can arrange a quick wrap-up about what to release and what's missing
16:50:54 <bauzas> #topic Sub/related team Highlights
16:50:58 <elodilles> bauzas: ++
16:50:59 <bauzas> Libvirt (bauzas)
16:51:06 <bauzas> nothing to tell honestly
16:52:22 <bauzas> btw. I will change the libvirt chairing to lyarwood for next week
16:52:30 <bauzas> apologies, i forgot to modify it
16:52:42 <bauzas> lyarwood: anything worth mentioning about libvirt ?
16:53:00 <lyarwood> not yet :)
16:53:08 <bauzas> hold your patchrds
16:53:10 <bauzas> patches*
16:53:22 <bauzas> #topic Open discussion
16:53:26 <ganso> o/
16:53:28 <bauzas> (bauzas) : Call for Outreachy mentoring extended until Sep 29 #link http://lists.openstack.org/pipermail/openstack-discuss/2021-September/024992.html
16:53:46 <bauzas> we have an opportunity for mentoring here
16:54:20 <bauzas> we haven't brainstormed about the potential projects we could propose to outreachy students
16:54:29 <bauzas> but I'm sure we have room for them
16:54:39 <bauzas> if anyone has any idea, shoot
16:54:50 <bauzas> I can propose to mentor
16:54:54 <gibi> Dear student please fix the CI before GA, thanks. :P
16:55:00 <gmann> :)
16:55:34 <bauzas> lol
16:55:57 <bauzas> I've noticed a bit late tho
16:56:01 <bauzas> anyway
16:56:02 <gibi> do we have ongoing work where easy to join?
16:56:39 <gibi> mypy maybe ;)
16:57:01 <bauzas> gibi: I was about to say functional testing but yeah
16:57:15 <ganso> hi! sorry I wanted to ask this before the meeting ended. 2 weeks ago it was mentioned in this meeting that there would be a new tag release soon for all stable branches. Is there any more specific estimate on when the new tag release will happen?
16:57:16 <gibi> bauzas: do we still have osc novaclient gaps?
16:57:31 <bauzas> gibi: I guess so
16:57:31 <gibi> bauzas: that would be also a good proposal
16:57:52 <gibi> bauzas: or simply filling the osc gaps for placement, we have missed a list of microversions there
16:58:12 <bauzas> interesting idea
16:58:46 <bauzas> one alterative I had in mind was to see how to use some mdev testing framework for validating our GPUs in our CI
16:59:06 <bauzas> but anyway, ganso has a question and we're 1 min
16:59:30 <bauzas> ganso: we just discussed we're considering to tag stable releases end of this week if we can
16:59:48 <ganso> great!
16:59:49 <ganso> thanks!
17:00:06 <bauzas> *if we can* as we have some blockers that need help
17:00:16 <bauzas> but we're overtime
17:00:23 <bauzas> thanks all
17:00:25 <bauzas> #endmeeting