16:00:51 #startmeeting nova 16:00:51 Meeting started Tue Sep 21 16:00:51 2021 UTC and is due to finish in 60 minutes. The chair is bauzas. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:00:51 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:00:51 The meeting name has been set to 'nova' 16:01:14 #link https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting 16:01:28 sorry for interupting you, folks :) 16:01:37 who's around ? 16:01:45 o/ 16:01:46 * stephenfin lurks 16:01:49 \o 16:01:53 o/ 16:02:28 o/ 16:02:55 we have some large discussions today, let's start quickly 16:03:04 #topic Bugs (stuck/critical) 16:03:08 No Critical bug 16:03:14 #link 13 new untriaged bugs (-0 since the last meeting): #link https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New 16:03:18 no open bugs marked with xena-rc-potential tag #link https://bugs.launchpad.net/nova/+bugs?field.tag=xena-rc-potential 16:03:21 please start marking release critical bugs with xena-rc-potential tag 16:03:34 reminder, we are in RC phase 16:03:42 meaning that we should focus on regression bugs 16:04:01 yoga is now the master branch 16:04:10 any bug to discuss ? 16:04:37 #topic Gate status 16:04:43 Nova gate bugs #link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure 16:04:46 we have a long list 16:04:53 Placement periodic job status #link https://zuul.openstack.org/builds?project=openstack%2Fplacement&pipeline=periodic-weekly 16:05:03 we now we failed because of a dependency issue 16:05:08 know* we 16:05:30 but now https://review.opendev.org/c/openstack/placement/+/810001 is merged 16:05:34 so we can recheck 16:05:43 any concern so far ? 16:06:04 moving on 16:06:10 Please look at the gate failures, file a bug, and add an elastic-recheck signature in the opendev/elastic-recheck repo (example: #link https://review.opendev.org/#/c/759967) 16:06:19 #topic Release Planning 16:06:30 (we'll discuss the placement dependency bump later) 16:06:37 Release tracking etherpad #link https://etherpad.opendev.org/p/nova-xena-rc-potential 16:06:51 as you can see, nothing fancy to tell 16:06:57 we need https://review.opendev.org/c/openstack/nova/+/810192 then an RC2 for nova 16:07:01 most of our efforts should go to merging bugfixes 16:07:19 gibi: right, I forgot about it 16:08:16 strangely, LP says Fix Released for Xena https://bugs.launchpad.net/nova/+bug/1944111 16:08:20 for some reason the bug is not showing up in the rc critical query 16:08:47 feels like an LP bug around branching 16:08:57 yup 16:09:06 will add it in the etherpad for tracking 16:09:22 already done :) 16:09:29 hah 16:09:35 naïce ;) 16:09:57 ok, let's wait a bit for releasing RC2 16:10:06 We now have RC1 releases for both Nova and Placement #link https://review.opendev.org/c/openstack/releases/+/808706 and https://review.opendev.org/c/openstack/releases/+/808713 16:10:12 We will need a Placement RC2 release due to #link https://review.opendev.org/c/openstack/placement/+/810001 16:10:36 https://review.opendev.org/c/openstack/placement/+/810193 needs special treatme 16:10:41 treatment 16:10:57 +2d now 16:11:17 we have the last RC deadline at 1st of Oct afaik 16:11:25 yeah 16:11:29 so yes, we can hold up RC2 a bit to see if anything else pops up 16:11:34 hence me saying let's hold a bit for a RC2 proposal 16:11:43 ya 16:11:54 Remember to propose regression bugfixes for a new RC with nova-xena-rc-potential 16:12:06 #topic Review priorities 16:12:14 https://review.opendev.org/q/status:open+(project:openstack/nova+OR+project:openstack/placement)+label:Review-Priority%252B1 16:12:28 this is absolutely empty 16:12:54 cores, please enjoy this new button until we discuss other ways to use it at the PTG 16:13:29 is there anyone wanting to have their changes be a priority for reviews ? 16:13:53 (that's your time, folks) 16:13:59 Wait, only cores can set the priority label? 16:14:15 with the current implementation, yes 16:14:31 I'm about to propose other possibilities at the PTG 16:14:41 (which reminds me I have to write something in the etherpad) 16:14:46 Can a non-core propose the label, or is the whole thing owned entirely by cores? 16:15:08 artom: dude, that's the whole discussion we had in the related doc change :) 16:15:14 artom: this is the currently agreed process https://review.opendev.org/c/openstack/nova/+/792357 but sure we can discuss it on the PTG 16:15:18 surely it will be totally useless if not curated by a smaller group right? 16:15:32 Sorry, not opening pandora's box again, just catching up 16:15:34 I'm happy to see people offering alternatives :) 16:15:35 in bug trackers where everyone can set priority, people set their bugs to be "urgent" all the time 16:15:37 Takashi Natsume proposed openstack/nova master: Update min supported service version for Yoga https://review.opendev.org/c/openstack/nova/+/809932 16:15:42 I can't imagine it will be useful otherwise 16:15:46 Takashi Natsume proposed openstack/nova master: Update min supported service version for Yoga https://review.opendev.org/c/openstack/nova/+/809932 16:16:10 dansmith: I had an alternative proposal I left in the comments but we can discuss this at the PTG (and I'll propose a documentation change) 16:16:21 ack 16:16:21 Essentially, for my own selfish needs, all I can do with the Priority label is query it, and look at those reviews in case my opinion on them is of any use to anyone, right? 16:16:51 artom: for the moment, the idea is that cores make some review vows when setting the label 16:17:04 artom: if you help merging reviews in priority then cores will have more bandwidth to look at other reviews including yours :D 16:17:04 meaning they engage themselves to review this patch in question 16:17:35 but I'm not sure we should continue this conversation now, we have some agenda left 16:17:42 (and a stretched time) 16:18:07 Understood. 16:18:45 moving on 16:18:55 #topic PTG Planning 16:19:01 every info is in the PTG etherpad #link https://etherpad.opendev.org/p/nova-yoga-ptg 16:19:14 If you see a need for a specific cross project section then please let me know (gibi or bauzas) 16:19:18 gibi: I jinxed you :p 16:19:42 at least we can see regular updates from sean-k-mooney :) 16:20:05 I expect more and more topics as we close to the PTG date 16:20:16 I think the current list is normal given we have a month still 16:20:25 absolutely right 16:20:26 (a bit less of a month 16:20:27 ) 16:20:31 I'm not afraid of having extra time 16:20:36 me neither 16:21:15 but I'd say, take this month as an opportunity to start designing your features, so you can raise your questions at the PTG 16:21:36 anyway, moving on 16:21:42 #topic Stable Branches 16:21:58 (that could last a bit) 16:22:00 nova's stable/ussuri and stable/train are blocked (due to latest virtualenv uses latest setuptools which removed use_2to3) 16:22:07 the future proof solution would be to pin virtualenv during tox install for stable branches, otherwise new errors can appear with every new release of setuptools, virtualenv, etc 16:22:12 until we decide about the right solution we can maybe set lower-constraints job as non-voting ( https://review.opendev.org/809955 ) 16:22:16 probably placement branches have the same errors 16:22:21 elodilles: floor is yours 16:22:30 bauzas: yes, this is the same error 16:22:35 Merged openstack/nova stable/xena: Add missing __init__.py in nova/db/api https://review.opendev.org/c/openstack/nova/+/810192 16:22:47 it affects a lot of projects that still uses decorator 3.4 as dep 16:22:56 nova affected from ussuri backwards 16:23:02 placement affected all the way to master 16:23:08 (master fix landed) 16:23:09 ah I missed this review sorry 16:23:10 * lyarwood opens 16:23:11 super awesome 16:23:31 we may want to consider droping that dep at some point 16:23:38 we actually have a longer explanation in the open discussion section 16:23:55 let's just hold this discussion until that point 16:24:10 although we are unlikely to hit the same issue again 16:24:12 (which we will have 30 mins for) 16:24:20 #topic Sub/related team Highlights 16:24:37 Libvirt (bauzas) 16:24:45 bauzas: happy to take over the libvirt part now you're PTL btw 16:24:56 lyarwood: awesome 16:24:58 I was asking for it 16:25:03 bauzas: not that I have anything for today but we have a few things this cycle 16:25:07 any stuff to raise ? 16:25:08 kk 16:25:27 #info lyarwood to chair the libvirt subteam by now 16:25:43 lyarwood: thanks for offering your name 16:25:55 Don't take it in vain now 16:26:03 #topic Open discussion 16:26:10 one last paperwork bit 16:26:14 (gibi): release liaison role 16:26:22 so 16:26:38 bauzas: is now the PTL and the release liaison as well 16:26:42 which is suboptimal 16:26:52 which means I'm also a bottleneck now 16:26:53 do we have a volunteer to take that role over? 16:27:11 it is not hard, you will get cc-d to release proposal patches to check and approve 16:27:19 tbc, the release liaison role is about reviewing patches from the release team 16:27:46 until either the PTL or the release liaison approves the patch, it can't land 16:28:03 i mean i can do it if no one else wants too i really dont mind 16:28:06 example https://review.opendev.org/c/openstack/releases/+/808706 16:28:44 i keep an eye on the os-vif ones anyway but someone on the stable team might make more sense 16:28:52 sean-k-mooney: appreciated 16:29:19 I'll make the changes in the appropriate repo so you'd get automatically CC'd 16:29:29 sean-k-mooney: thank you 16:29:53 no worries 16:29:55 * bauzas just has to figure out which specific file to update in which specific repo :D 16:29:58 well, as I also a release core and also propose nova releases lately it would be weird to propose and approve my own patches and then +W o:) 16:30:36 elodilles: heh, depending whether you're schizophrenic, this could work 16:30:47 :D 16:31:17 ok, moving on 16:31:22 sean-k-mooney: thanks again 16:31:37 now the big discussion 16:31:47 pasting the whole section 16:31:49 (gibi): gathering opinions about the current lower-constraints failure. 16:31:56 bottom line: on stable branches we are installing tox which installs virtualenv which bundles setuptools unconstrained. This now leads to that we cannot install decorator 3.4.0 on stable any more as it depends on "user_2to3" from setuptools but the recent setuptools 58.0 removed support for that. 16:32:02 Options to resolve the situation bump decorator major version from 3.4.0 to 4.0.0 on stable branches. Does it against stable policy? pin virtualenv version on stable during tox install disable lower-constraints testing 16:32:14 shit, I pasted wrong 16:32:20 gibi: your turn 16:32:27 explain the 3 options 16:32:27 let me untangle that 16:32:50 so option 1) bump major version of decorator from 3.4 to 4.0 on stable. Is it allowed on stable? 16:33:01 option 2) pin virtualenv during tox install 16:33:11 option 3) disable lower-constraints job 16:33:28 I personally think that using unconstrained setuptools on stable is dangerous 16:33:36 so I would go with 2) long turn 16:33:39 term 16:33:53 Yeah, seems like 2 is the safest... what's the danger with it? Is there a catch? 16:33:53 Agreed, 2 would be my choice 16:33:57 gibi: let's explain which stable branches again are impacted 16:34:13 you already told but this doesn't harm to tell again 16:34:14 so in nova we are impacted stable/ussuri and older 16:34:16 gibi: technially i dont think we are allowe to bump miniums on stable on the other hand we already did it a while ago for the inital lower constraitnts fix 16:34:26 on placement we are impacted on master but the fix has been landed 16:34:40 on stable/xena we could release RC2 with the bump 16:34:47 I also like option 2, and actually found some (abandoned) trial from the past that was similar https://review.opendev.org/q/topic:constrain-tox-install 16:34:50 but on stable/wallaby and back the same quertion applies 16:35:12 from a disto point of view 2 is not great 16:35:22 since they cannot pin them the same way 16:35:25 option 1 has a question I can't answer 16:35:32 but most distros also dont use lower constratints 16:35:33 does LTS distros use unconstrained setuptools as well? 16:35:54 can we deliver a .y release by bumping the decorator dependency without breaking the semver rules ? 16:36:07 rhel ignored the upper and lower constraitnts 16:36:14 debian used to look at lower 16:36:25 ubunut used to look at upper 16:36:46 we can try 2 but i feel like we are going to end up with 3 16:37:00 sean-k-mooney: sure we could end up with 3 for different reasons :) 16:37:12 So 2 for distros means "maintain this old version of virtualenv in your repos", right? 16:37:22 bauzas: I think we can, but it's probably not fortunate. semver says that req changes needs MINOR / .y version bump 16:37:28 But... don't we already have upper-constraints that does the exact same thing? 16:37:36 artom: yes, as virtualenv bundles setuptools 16:37:56 elodilles: so option 1 doesn't break stable policy per se 16:38:00 artom: this is tricky as the tox install in our CI does not use constraint file 16:38:14 artom: we install tox, then with tox we install deps 16:38:25 artom: but when we install tox it pulls in setuptools 16:38:59 Oh, so this is before any -constraints is applied 16:39:15 the problem is with the venv, not the dependencies 16:39:20 artom: right 16:39:30 because of the bundle 16:39:30 bauzas: I also haven't found that written, but I remember that we tried to avoid req changes 16:39:48 but, there are ways to create venvs without bundling setuptools, right? 16:40:05 bauzas: and the problem is that we will face the same issue whenever something new things comes in 16:40:12 Wait, I though the decorator==3.4.0 install happened *inside* the venv, as a dependency? 16:40:29 elodilles: agreed, option 1 only fixes the decorator issue and doesn't resolve any other issue 16:40:49 I'm also in favor of option 2 16:41:05 artom: to have a venv where you can install deps, you have to have the venv package installed 16:41:08 but I wonder whether we could just tell tox to create venvs without bundling setuptools 16:41:35 just weighing in, 3 isn't an ideal solution either 16:41:35 bauzas: I think if you have a venv without setuptools then you dont have pip in the venv either 16:41:48 bauzas: so you cannot install additional things with pip 16:41:56 gibi: are you sure ? 16:42:02 bauzas: 75% 16:42:05 I think it does install pip 16:42:14 I think pip depends on setuptools 16:42:18 in this case it's the lower-constraint that has broken, however, it's conceivable that the upper constraint for a particularly old release could also become incompatible with setuptools 16:42:20 but I could be wrong 16:42:32 stephenfin: good point 16:42:35 gibi, right, so we install tox on the "host", which pulls in unconstrained setuptools, which then break installing decorator==3.4.0, my question is, aren't we installing decorator==3.4.0 inside the venv? 16:42:47 artom: hm, 16:42:48 So it should be independent of what's on the "host"? 16:42:53 artom: you have a point 16:43:13 artom: for some reason our ci install tox already in a venv but I lost following that track 16:43:40 artom: it installs setuptools in the venv also, right? 16:44:05 Don't know... but then it'd be a separate venv for the actual nova install that pulls in decorator... 16:44:40 I agree that something is missing from our understanding about venvs and tox install 16:45:00 >>> setuptools.__file__ 16:45:06 '/home/stephenfin/Development/openstack/nova/.tox/py36/lib/python3.6/site-packages/setuptools/__init__.py' 16:45:16 https://tox.readthedocs.io/en/latest/config.html#conf-requires 16:45:33 if we need to pin it we can pin the setuptools and pip version with requires 16:45:45 I think it's doable 16:46:03 tox pulls in virtualenv, virtualenv bundles setuptools for the new virtualenvs it makes 16:46:03 sean-k-mooney: good finding 16:46:09 it is but fungi and clarkb advised against doing that 16:46:14 in my abandoned trial in the past i tried to use upper-constraints.txt for tox install, it probably could solve the issue if that works (i don't remember whether it worked) 16:46:16 virtualenv has an escape hatch for that but I'm not sure how tox exposes that (if at all) 16:46:24 https://paste.opendev.org/show/809477/ 16:46:27 you need to pin virtualenv is the tldr if you are going to pin something 16:46:34 but you have to do it when installing tox 16:46:46 you can create a venv without setuptools and pip it later 16:46:58 right but does tox support any of that? 16:47:04 as it is tox making the virtualenv automatically 16:47:15 tox uses the venv module, right?N 16:47:23 not by default I think it can 16:47:32 clarkb, gibi, ah, that's the missing understanding, when virtualenv creates a venv, it automatically stuffs the setuptools version it came with in that venv... right? 16:47:35 basically oyu have to see how much of this is configurable on the tox side to do what you want 16:47:41 artom: yes exactly 16:47:44 Stephen Finucane proposed openstack/nova master: db: Add migration to resolve shadow table discrepancies https://review.opendev.org/c/openstack/nova/+/805738 16:47:45 Stephen Finucane proposed openstack/nova master: tests: Walk database migrations in correct order https://review.opendev.org/c/openstack/nova/+/810291 16:48:14 clarkb: you're right, I need to dig into tox 16:48:20 This looks like good background reading https://tox.readthedocs.io/en/latest/example/package.html 16:48:21 clarkb: thanks 16:48:25 Sounds like we have no choice but to pin virtualenv then 16:48:30 Sounds like the escape hatches we need might be there 16:48:42 Unless there's a way to tell virtualenv which setuptools version to install in the venvs in creates 16:48:53 if you want tox to use venv as the virtualenv backend, these days the trick is to export VIRTUALENV_CREATOR = venv in the inherited setenv in your tox.ini 16:49:21 there used to be a tox-venv plugin, but that was deprecated/abandoned once virtualenv grew the option to call out to the venv module itself 16:49:30 can we use https://tox.readthedocs.io/en/latest/config.html#conf-requires ? 16:49:53 so it's not tox calling venv directly, but rather tox calling virtualenv and then virtualenv being told to use the venv module to create the env rather than its internal implementation 16:50:13 i feel like the complexity of maintianing this pining outways the usefullness of the job 16:50:29 sean-k-mooney: that's the problem with option 2 16:50:45 ideally, I'd like us to maitain setuptools in our reqs 16:50:52 maintain* even 16:50:59 but this doesn't sound easy 16:51:03 So the value of the job is what - guarantee that will still work with the oldest feasible versions of things? 16:51:12 (lower-constraints I mean) 16:51:25 sean-k-mooney: We will always have pinning in the form of upper-constraints though 16:51:26 yes basically to make sure when we backport we dont raise the requiremetns as a result 16:51:41 the correct way to maintain setuptools in your requirements is via the pyproject file 16:51:44 stephenfin: right but for stable we are not allowed to raise miniums 16:51:47 not requirements 16:52:14 and? we can't raise maximums either 16:52:17 the new pyproject file allows you to specify build time deps (possibly even tools other than setuptools) 16:52:28 I don't know what would be involved to make all of that work with pbr though 16:53:50 clarkb: how other projects consider this issue ? do they want to pin setuptools as well ? 16:54:01 or do they just fix the decorator issue ? 16:54:12 (and backport without specific care) 16:54:22 I guess we're not alone in the dark 16:54:31 msot of the fixes I've seen have been to fix the dependencies 16:54:41 and about backports ? 16:54:44 openstack governance had to switch pydot2 to pydot for example 16:55:03 zuul-jobs dropped support for old python3.5 which allowed it to use a newer version of a lib 16:55:27 I don't know how backports or stable issues have been handled 16:56:34 gibi: I guess you have to write something and bubble it to the community for cross-project thinking 16:57:08 bauzas: I can write up a summary from today on the ML 16:57:09 that could become an epic saga, but it's worth starting it 16:57:28 gibi: appreciated, here lemme summarize for the audience 16:57:46 option 2 seems to be the better, but we struggle finding a good way to write it 16:58:22 Stupid question, but could we not just install virtualenv== first, and *then* tox? 16:58:23 option 1 seems to be the alternative, option 3 seems not accepted 16:58:27 Or would tox just upgrade? 16:58:49 artom: that could be also something to try 16:58:56 artom: virtualenv pulls the latest setuptools IIRC 16:59:04 but that's a flag 16:59:15 bauzas: old virtualenv will pull old setuptools I guess 16:59:22 again, the problem is how to trigger this flag thru tox 16:59:42 gibi: I remember having played with it before and you're right 17:00:01 but you can explicitely specific to pull the latest setuptools 17:00:07 either way, we're at time 17:00:24 gibi: thanks for writing the summary and raising it to the community 17:00:34 folks, wrapping up 17:00:36 thanks 17:00:40 #endmeeting