19:01:12 <clarkb> #startmeeting infra
19:01:13 <openstack> Meeting started Tue Sep  1 19:01:12 2020 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:14 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:16 <openstack> The meeting name has been set to 'infra'
19:01:27 <clarkb> #link http://lists.opendev.org/pipermail/service-discuss/2020-August/000080.html Our Agenda
19:01:31 <ianw> o/
19:01:40 <clarkb> #topic Announcements
19:01:44 <clarkb> I didn't have any announcements
19:02:06 <fungi> forum cfp is open
19:02:16 <clarkb> did that just happen today?
19:02:23 <fungi> yep
19:02:28 <clarkb> 4 hours ago says my email, thanks for the notice
19:02:34 <clarkb> I had missed that
19:03:04 <clarkb> looks like the deadline for submissions is september 14 so we have about 2 weeks to add proposals there
19:03:29 <clarkb> #topic Actions from last meeting
19:03:38 <clarkb> #link http://eavesdrop.openstack.org/meetings/infra/2020/infra.2020-08-25-19.01.txt minutes from last meeting
19:03:55 <clarkb> No actions from last meeting. It was pretty informal too since many were out on vacation or otherwise unable to attend
19:03:59 <corvus> o/
19:04:08 <clarkb> #topic Specs approval
19:04:24 <clarkb> I did approve fungi's identity broker service spec some time ago in the past
19:04:38 <clarkb> If folks want to start working on that there is a spec now :)
19:04:41 <fungi> late last week i think
19:04:58 <clarkb> the days all run together anymore
19:05:03 <fungi> i was "on vacation" so my memory of last week is hazy
19:05:31 <clarkb> #topic Priority Efforts
19:05:39 <clarkb> #topic Update Config Management
19:06:20 <clarkb> I didn't have much on this topic other than to call out that I'm investigating nodepool builder arm64 images again now that pyca/cryptography has arm64 wheel published on pypi
19:06:51 <clarkb> It looks like there are additional things we need wheels for (expected) and the openstack wheel mirror currently is sufficient to get the job to run under the hour timeout
19:07:26 <clarkb> all of that to say we haven't regressed on what we already knew and we have made some progress
19:07:41 <ianw> yay progress! :)
19:08:12 <clarkb> did anyone else have config management changes to call out? Any new ansiblification or docker container work?
19:09:18 <fungi> i didn't have any, i don't think
19:09:46 <clarkb> #topic OpenDev
19:09:56 <clarkb> #link https://review.opendev.org/#/c/748263/ Update opendev.org front page content
19:10:11 <clarkb> This is a change that ttx put together to try and capture the "why opendev" better
19:10:28 <clarkb> looks like it has received some reviews and CI is currently failing on a setuptools 50 issue
19:10:43 <clarkb> if ya'll have time to review that it would be great, I think it is a good improvement
19:11:41 <clarkb> On the Gerrit upgrade side of things Luca has confirmed that disk performance is incredibly important for the gerrit upgrade process. That means we do want to update review-test to mimic production with its cinder volumes
19:12:07 <clarkb> There is one or two small things I want to test on my local gerrit upgrade test setup, but then I'll be looking into that next
19:12:28 <clarkb> I think I'll start with trying to add a volume to the existing server
19:12:55 <clarkb> And finally: Non master repo HEAD support for new projects has landed. We can offer this to new projects now.
19:13:25 <clarkb> I believe that gitea, gerrit, jeepy, and gerritlib work is complete to make this possible. That said it won't surprise me if we discover there are unknown unknowns we have to address once this is attempted
19:14:22 <clarkb> Any other opendev related topics to bring up?
19:14:30 <fungi> this is sufficient for creating a new project with a different default branch, but doesn't cover changing the default branch of an existing project, right?
19:14:36 <clarkb> fungi: correct
19:14:57 <clarkb> I am hoping we can work through those unknown unknowns with a new repo before we tackle the additional steps of converting a project
19:15:39 <fungi> makes sense
19:15:45 <clarkb> I think the conversion process for an existing project will look something like: update projects.yaml to set the branch value then run an outof band playbook to update gitea and gerrit values. Fixing CI is an exercise left up to the reader
19:16:18 <clarkb> pabelanger confirms that zuul itself is fine when you do a conversion
19:16:28 <clarkb> but jobs content is likely going to need updates depending on the projects
19:16:44 <clarkb> (we also added a unittest to zuul to ensure we don't regress that functionality)
19:17:24 <clarkb> s/conversion/rename/
19:18:23 <clarkb> #topic General Topics
19:18:36 <clarkb> #topic Setuptools 50 Release
19:19:03 <clarkb> ~Sunday pypa released a new version of setuptools which has had a number of regressions assocaited with it
19:19:44 <clarkb> in particular it seems that older python3 like 3.5 has import issues for setuptools when installing setuptools 50. The other issue we've seen is the change in instalation directory when building packages from source and not installing them from wheels
19:20:13 <clarkb> in some places we're settinga flag to use the distro distutils (its actually setuptools vendoring distutils that causes the problems) and in others we are pinning setuptools
19:20:31 <ianw> clarkb: has the change in directory not been matched with the packages not being found in any instances you're aware of?
19:20:40 <clarkb> it does look like upstream is working on fixing the older python issues though the installation location isn't expected to change again (that was intentionaly for them)
19:21:13 <fungi> it's also worth remembering that setuptools doesn't consider itself to be "vendoring" distutils, the idea is that the setuptools maintainers are taking over distutils maintenance so it can eventually be removed from stdlib
19:21:34 <clarkb> ianw: the ones I've seen were all https://github.com/pypa/setuptools/issues/2357 and the /usr/local/bin vs /usr/bin problems
19:21:51 <ianw> (i.e. what i'm saying is that s,/usr/local/bin,/usr/bin, doesn't work, because the binary can't find it's packages anyway)
19:22:31 <clarkb> ianw: oh I haven't seen that I don't think (probably because we are either using distro distutils or pinning to 49)
19:22:48 <clarkb> its possible that if python3.5 issues are fixed upstream we'd see a lot more of that?
19:22:53 <corvus> this is the error from that ttx change clarkb mentioned earlier: https://zuul.opendev.org/t/openstack/build/b6aaf1af991948ffa3062a4c3be94906   --  what error is this?
19:22:59 <fungi> yeah, debian's patched distutils is necessary to work with the patched module import path in their interpreter
19:23:15 <clarkb> corvus: I believe that to be https://github.com/pypa/setuptools/issues/2357 its a xenial python3.5 job failing to import setuptools
19:23:46 <ianw> yeah -- https://github.com/pypa/setuptools/issues/2350#issuecomment-684082760
19:24:13 <ianw> basically if you're seeing your binaries moved; you've used "pip install -e" or possibly setup.py directly, or otherwise *not* built a wheel
19:24:34 <ianw> because, of course, the paths are totally different and wheels continue to work while installing doesn't
19:24:34 <corvus> clarkb: so iiuc, we're waiting on a setuptools release for that?
19:25:04 <clarkb> corvus: ya or we can pin setuptools there or use the disto distutils. I only just looked at that job a few minutes before the meeting so haven't had time to think about which approach may be best
19:25:20 <clarkb> there is a 50.0.1 from a couple hurs ago
19:25:22 <clarkb> *hours
19:25:27 <clarkb> so maybe we shoul drecheck and see if it owrks now?
19:25:40 <corvus> done
19:25:47 <ianw> (i wouldn't hold much hope for that, i was scrolling through the issues, there's quite a few, and remember someone said it didn't fix it)
19:26:29 <clarkb> but ya the issues have been pretty far ranging and exhibit in different ways. Wanted to call out the change in case anyone missed it or had failures they weren't sure about
19:26:37 <clarkb> I'm happy to help debug any that pop up
19:26:44 <ianw> ++
19:27:12 <ianw> devstack has just used the environment variable ...
19:27:45 <clarkb> the reason I went with pinning on our image builds instead is they build on a ton of different platforms and I didn't want t othink through waht using distutils on each of them may mean
19:28:01 <clarkb> was easier to use setuptools 49 in that case an have the same setup across the board in the virtualenvs
19:29:14 <clarkb> Any other questions or thoughts on this topic? I don't expect we'll solve the problems here and now but wanted to call it out as an ongoing issue
19:29:46 <corvus> clarkb: when do you think we'll lift the pin?
19:29:54 <corvus> ie, what needs to happen for that
19:29:57 * ianw declines to offer furter thoughts :)
19:30:10 <clarkb> corvus: we need the older python3 situation to be fixed
19:30:23 <clarkb> corvus: upstream seems to acknowledge the bugs and is working on them so that should eventually happen
19:30:32 <clarkb> there are like 8 bugs at least though :/
19:30:35 <fungi> probably if 748263 passes, that's a sign we can unroll it?
19:30:51 <ianw> "Indeed, the impact is undesirably huge. I had gone down the route of attempting to maintain compatibility and incorporate the debian-specific changes, but decided against that. However, that was before #2259, which expanded the scope of the hack. Perhaps we should re-open #2232 to address Debian-specific needs.
19:30:51 <ianw> "
19:31:04 <ianw> https://github.com/pypa/setuptools/issues/2352#issuecomment-684989322
19:31:18 <ianw> so i think the issue is in flux (that's from the maintainer)
19:31:48 <corvus> fungi: isn't 263 running on a pinned image?
19:32:07 <clarkb> corvus: we only pinned the venvs we preinstall on the image
19:32:11 <corvus> oh gotcha
19:32:12 <fungi> oh, i guess it failed before we rolled the images back, not after?
19:32:28 <clarkb> if the job content itself is doing setuptools things it may trip over the issue
19:32:46 <fungi> but yeah, if that job is installing a new setuptools on its own, then it's independent of the image pin
19:33:04 <clarkb> I do expect upstream will figur ethings out
19:33:09 <clarkb> they sorted stuff out after the 48 release
19:33:18 <clarkb> it may just need a little time given the number of problems with 50
19:33:29 <fungi> 2020-09-01 14:00 was long after we rolled images back, so i think it must have pulled its own setuptools
19:33:53 <fungi> but i haven't had time to dig into the build log yet
19:35:10 <clarkb> Making ask.openstack.org read only
19:35:17 <clarkb> er that needed a topic prefix
19:35:19 <clarkb> #topic Making ask.openstack.org read only
19:35:36 <clarkb> #link https://review.opendev.org/#/c/746497/ set ask.openstack.org to read only
19:35:49 <clarkb> this has had a fair bit of feedback at this point and no one has objected
19:36:05 <clarkb> I was hoping to give the meeting here one last chance to hold off, otherwise I planned to approve the change
19:36:08 <fungi> seems we can go forth and disable
19:37:26 <clarkb> I didn't expect any objections. Let me know before the end of the meeting I Guess if you did have one and we can wait
19:37:32 <clarkb> #topic PTG Planning
19:37:49 <clarkb> October PTG registration is now open: https://www.openstack.org/ptg/
19:37:59 <clarkb> I don't think I've done that myself yet so this serves as a personal reminder
19:38:10 <clarkb> I also haven't gotten to the planning document yet simply too many other things popping up lately
19:38:28 <clarkb> #link https://etherpad.opendev.org/opendev-ptg-planning-oct-2020 October PTG planning starts here
19:38:33 <diablo_rojo> clarkb, I saw you filled out the ethercalc, but not the survey?
19:38:42 <clarkb> diablo_rojo: I did both
19:38:50 <diablo_rojo> clarkb, oh cool.
19:38:51 <diablo_rojo> Thanks!
19:38:59 <clarkb> I guess double check you received my response?
19:39:12 <clarkb> and any one should feel free to put ideas on that etherpad
19:39:30 <diablo_rojo> clarkb, I don't see it in the spreadsheet, but it could have gotten missed by the integration.
19:40:05 <fungi> i'm also checking to see if anyone thinks the openstack tact sig needs dedicated ptg slots, but my guess is we'll just tag along with opendev
19:40:20 <clarkb> diablo_rojo: let me know if I should fill it out again
19:40:42 <diablo_rojo> clarkb, looks like its not in the formstack data either so, yes please :)
19:40:46 <fungi> (or with the openstack qa team, depending on topics)
19:40:46 <diablo_rojo> Maybe you didn't click submit?
19:41:11 <clarkb> maybe I filled out the previous survey link bceause my email seraching foo was bad
19:41:14 <clarkb> I'll resubmit
19:41:25 <diablo_rojo> clarkb, thanks in advance
19:41:37 <diablo_rojo> Here's the link if you need: https://openstackfoundation.formstack.com/forms/oct2020_vptg_survey
19:41:41 <clarkb> thanks
19:41:48 <clarkb> #topic Bup and Borg Backups
19:41:56 <fungi> the original announcement linked old ethercalc and old survey, fwiw
19:42:16 <clarkb> ianw: anything we can do to help on this? Just making sure we don't forget about it or have it stuck on something we can help with
19:42:24 <clarkb> (I know there has been a lot of other happenings recently)
19:42:56 <ianw> yeah it's just been me not having a chunk to bring it all up
19:43:01 <ianw> a chunk of time
19:43:08 <clarkb> k
19:43:15 <clarkb> #topic Open Discussion
19:43:28 <clarkb> The floor is open
19:45:37 <fungi> ask.openstack.org going once...
19:45:42 <fungi> twice...
19:45:44 <fungi> ;)
19:45:52 <clarkb> fungi: you want to hit the button or should I?
19:45:58 <clarkb> re ask.o.o
19:46:22 <clarkb> Sounds like that may be the meeting. Thank you everyone
19:46:27 <corvus> push the button frank
19:46:27 <diablo_rojo> Do it do it do it.
19:46:31 <clarkb> #endmeeting