19:01:02 <clarkb> #startmeeting infra
19:01:02 <opendevmeet> Meeting started Tue Oct 26 19:01:02 2021 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:02 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:02 <opendevmeet> The meeting name has been set to 'infra'
19:01:04 <clarkb> #link http://lists.opendev.org/pipermail/service-discuss/2021-October/000293.html Our Agenda
19:01:10 <clarkb> #topic Announcements
19:01:23 <clarkb> This wasn't on the agenda but I saw an email going by indicating fungi may be doing gpg stuff?
19:01:33 <clarkb> fungi: ^ do you need us to go and sign that key in the near future?
19:01:46 <fungi> ooh, i'm actually in the midst of completing that now
19:02:13 <fungi> i'm updating the process, so not exactly no
19:02:21 <clarkb> ok, will wait for those updates then :)
19:02:29 <clarkb> #topic Actions from last meeting
19:02:35 <clarkb> #link http://eavesdrop.openstack.org/meetings/infra/2021/infra.2021-10-19-19.01.txt minutes from last meeting
19:02:42 <clarkb> There were no actions recorded last meeting
19:02:42 <fungi> with the sks keyserver network collapse, the next best option is to switch to the gnupg.org keyservers, but they don't publish thord-party signatures
19:02:54 <fungi> er, third-party
19:03:16 <clarkb> #topic Specs
19:03:21 <clarkb> #link https://review.opendev.org/810990 Mailman 3 spec
19:03:34 <clarkb> This is still needing reviews. infra-root if you can find time for this it would be greatly appreciated
19:03:38 <clarkb> #action infra-root review Mailman 3 spec https://review.opendev.org/810990
19:04:56 <clarkb> Not sure there is much more to say on that other than we need reviews.
19:05:03 <clarkb> #topic Topics
19:05:09 <clarkb> #topic Improving OpenDev's CD throughput
19:05:40 <clarkb> ianw: I rebased your first chagne to fix a merge conflict I introduced with some of the gerrit upgrade cleanup work. This change is +1 now
19:05:45 <clarkb> ianw: I guess we should try and land it this week?
19:05:56 <clarkb> #link https://review.opendev.org/c/opendev/system-config/+/807672 is the change I'm talking about
19:06:10 <ianw> i noticed some of the other dependent changes acquired a -1 somehow, i haven't dug into it yet
19:06:38 <ianw> but yep, we can keep at it :)
19:06:39 <clarkb> ah maybe we should ensure the rebase I did didn't introduce any of those problems
19:06:49 <clarkb> I'll put this on my TODO list for this week though
19:07:12 <clarkb> #topic Gerrit account cleanups
19:07:52 <clarkb> I haven't done the bulk work on this that is on my TODO list but did do a one off to allow a user to log in again (they had problems and I cc'd funig and ianw on the last email since they have looked at gerrit account stuff before)
19:08:03 <clarkb> The log of the work I did for that user is on review02 in the typical location
19:08:32 <clarkb> essentially I retired an old account and cleaned up an email address from that account so the user can associate it with a newer account they appear to be using (based on openid external ids)
19:08:52 <clarkb> #topic Fedora 34 Booting Problems
19:09:05 <clarkb> ianw: ^ can you quickly fill us in on the fun around that?
19:09:58 <ianw> at least one problem is that it seems the initramfs has stopped shipping the drivers to mount the root disk on Xen, i.e. RAX
19:10:26 <ianw> there is a dracut fix for that, but it's hard to deploy
19:10:42 <clarkb> #link https://bugzilla.redhat.com/show_bug.cgi?id=2010058 fedora 34 initramfs lacking xen drivers
19:10:46 <ianw> #link https://review.opendev.org/c/openstack/diskimage-builder/+/815409
19:11:00 <ianw> is a generic fix for dracut-regnerate, which ... uses dracut to regenerate the initramfs
19:11:05 <clarkb> ianw: I left a comment on that dib change wondering about centos 7
19:11:19 <ianw> #link https://review.opendev.org/c/openstack/diskimage-builder/+/815385
19:11:29 <clarkb> centos 7 is python2.7 platform and uses dracut? Or maybe dracut is newer than centos 7?
19:12:26 <ianw> even centos7 has python3 now
19:12:46 <clarkb> ianw: will dracut run with it?
19:12:55 <clarkb> I thought that centos 3 was independent of all the normal system tools
19:13:22 <ianw> it's not dracut we need python-yaml there for, it's to run the dracut-regenerate tool from bin/ which is written in python
19:13:46 <clarkb> oh!
19:14:26 <ianw> yeah, a bit confusing
19:14:53 <clarkb> I've updated my review to a +2 thanks
19:15:37 <clarkb> Then we also need to install some special dracut testing package because they haven't landed it upstream yet?
19:16:08 <ianw> yeah; and then figure out how to test it, or deploy it
19:16:50 <ianw> the most expedient method would be to hand-edit in the change to the nodepool-builder container
19:16:58 <ianw> my previous effort at that wasn't a shining example
19:17:43 <clarkb> Probbaly the best thing is to do a small local build without all the git repos and extra stuff. Then manually upload to rax and boot it. The hardest thing about that is the manually upload step since rax image upload process si wird
19:17:52 <clarkb> you're basically forced to use shade/openstacksdk
19:18:03 <ianw> yes ...
19:18:08 <ianw> what i was trying to avoid :)
19:19:45 <ianw> i'm not sure if they're going to regenerate the default initramfs for f34 or what
19:20:18 <clarkb> You'd hope they will since it has anothre 7 months or so of life. Kind of annoying that fixing existing users is less important that addressing an unreleased release
19:20:43 <ianw> if i could get into a rescue mode i could copy in the initramfs, but that doesn't seem to work either
19:21:25 <clarkb> ianw: I think you may have to override the rescue mode host image. By default it uses the image that is being rescued
19:22:25 <ianw> yeah, i thought the idea was that it boots into a completely separate mini-vm that then has your old vm as a block device
19:22:53 <clarkb> ianw: it does, but it has to have an image to boot that separate VM with and by default it uses the same image I think
19:23:09 <clarkb> there is a flag to set a different image for the boot which might solve the problem
19:23:42 <ianw> huh, ok ... well i'll have a poke but i wasn't aware of that flag
19:24:22 <clarkb> Anything else on this topic before we move on?
19:24:41 <ianw> i guess i'll have to try the build method
19:25:02 <ianw> no, it's just annoying and not particularly what any of us want to be doing :)
19:25:11 <clarkb> indeed, thank you for looking at it
19:25:24 <clarkb> #topic Begin planning for Gerrit 3.4 upgrade
19:26:07 <clarkb> We're on 3.3 now but 3.4 exists and we should start looking at an upgrade to that version. As with 3.2 -> 3.3 we can do a downgrade from 3.4 -> 3.3 if we need to
19:26:44 <clarkb> We are already building 3.4 images and have an upgrade job to test the 3.3 -> 3.4 upgrade. There appears to only be an index change between the two and that gets updated online after restarting on 3.4
19:27:16 <clarkb> #link https://www.gerritcodereview.com/3.4.html Release notes
19:27:48 <clarkb> I think it would be good if we can find time to read over release notes to find any scary things or things that we expect will need testing. Then hold a 3.4 test node and manually interact with it a little bit to look for any unexpected changes
19:28:03 <clarkb> We can also use that held node to test a downgrade to 3.3.
19:28:19 <ianw> "HTML plugin support is removed" -- we don't have anything still hanging on to that do we?
19:28:36 <clarkb> ianw: I wonder if our theming is considered an html plugin?
19:28:49 <clarkb> I don't know how that plugs into gerrit
19:30:04 <fungi> it's a polygerrit plugin
19:30:05 <ianw> https://groups.google.com/g/repo-discuss/c/YJY8oQZZo44/m/nF1BtuPQAwAJ lists "zuul-status" as requiring updates to polymer 3
19:31:20 <ianw> (our display is zuul-summary-status confusingly and isn't affected)
19:31:29 <clarkb> zuul-status isn't one that we use either
19:31:49 <clarkb> fungi: I guess we needt o double check that it is polymer 3 safe
19:32:00 <clarkb> we should be able to verify that with a held test instance (either it will work or not)
19:32:28 <clarkb> The other thing we should try to do is improve the automated testing with new stuff as we find it so that we're not needing to do extra checks for the 3.5 upgrade
19:32:28 <ianw> i'm happy to take an action item to get a checklist started, setup a held node and investigate downgrade
19:32:50 <clarkb> #action ianw Start Gerrit 3.4 upgrade checklist, setup a held node, and investigate the downgrade
19:33:07 <ianw> ++ yep anything we find that we can test we should :)
19:33:08 <clarkb> ianw: thanks. And ya at this point I think we just need one or two people to start looking at it a bit more in depth so that we can fix issues and then figure out a plan from there
19:33:47 <fungi> i expect our test screenshots alone will be sufficient for confirming the theme is okay
19:34:02 <fungi> but deeper interactive testing is indeed warranted
19:34:39 <clarkb> fungi: ya there are plenty of unknown unknowns and actually loading up the UI is a good way to discover that stuff
19:34:52 <clarkb> sounds like we've got a plan to make a plan. Anything else we want to talk about on this subject?
19:36:07 <clarkb> #topic Open Discussion
19:36:27 <clarkb> We reenabled the inmotion cloud today. Please keep an eye out for oddities there, but they made a number of changes whcih we hope will make for a happier cloud
19:36:36 <fungi> i'll have changes up shortly for the process updates and new signing key clarkb mentioned earlier
19:36:36 <clarkb> thank you yuriys (not in here but still we appreciate it)
19:37:32 <clarkb> I started looking at mordred's pep 517 change in pbr today. I think it may not work properly based on some early testing. setup.py is being deprecated and we'll need an option going forward one way or another and having PBR support for pep517 is probably a good stepping stone to something richer
19:37:32 <ianw> if anyone was following the wheel discussion in zuul yesterday i wrote up
19:37:36 <ianw> #link https://review.opendev.org/c/zuul/zuul/+/815406
19:38:10 <clarkb> oh yes I need to followup on that /me adds to the todo list :)
19:38:26 <corvus> ianw: it looks like you're proposing opendev wheel generation as the main solution?  (as opposed to a nodepool dependencies image?)
19:39:13 <ianw> corvus: i think it makes the most sense.  then anything that uses the python base images could use the wheels safely
19:39:36 <corvus> as far as implementation details go, that probably makes it more of an opendev spec?
19:40:05 <ianw> true, if that's what we agree on
19:40:19 <corvus> maybe it's sort of like the matrix spec:  "zuul project says we want this; details to be hashed out with opendev"?
19:41:00 <ianw> looking at the wheels actually used, i think pynacl is the big problem
19:41:35 <ianw> i have offered there to help build manylinux arm64 wheels
19:42:26 <corvus> ianw: thanks for writing that up; i'll give it a look (probably mostly through the lens i described above)
19:42:54 <ianw> clarkb: is it possible to summarise the pbr/pep517/setup.py thing?
19:43:27 <clarkb> ianw: ya basically pypa has decided taht the setup.py packaging interface is deprecated and will go away in the future. Setuptools and Wheel are not going anywhere just the setup.py script interface
19:43:49 <fungi> ianw: in short, setuptools has marked any invocations of `setup.py build` as well as setting setup_requires metadata or calling into easy_install as deprecated
19:43:53 <clarkb> ianw: this means that eventually you'll have to use some other tool like pip or build or poetry to manage python packages for your projects
19:44:29 <clarkb> the standard they are coalescing around supporting is the pep517/pep518 specifications where you put information about how to build your projects in pyproject.toml and then pip/build/poetry know what to do with them
19:44:35 <fungi> setup.py isn't exactly deprecated, only calling it directly, as well as some things which they now feel should be done through pyproject.toml configuration
19:45:08 <fungi> basically they would like to eventually be able to get rid of setuptools directly managing build requirements
19:45:38 <corvus> i'd like to share a change that i did not push up for review:  i wrote, but did not push, a change to split out zuul geard serving to a separate container.  i wrote it to prep for running a second scheduler.  but i realized that the way we have everything set up, it's fine for a second scheduler to also run gearman because nothing will use it (even the second scheduler).  so i have that sitting on the shelf if we need it, but don't plan
19:45:39 <corvus> to use it.  i will hopefully soon push up a change to spin up a new scheduler host for zuul.
19:45:41 <fungi> so expect you to call a wrapper like https://pypi.org/project/build instead, which will ask pip to satisfy your build backend's requirements first before it calls into setuptools
19:46:13 <clarkb> corvus: that makes sense
19:46:35 <clarkb> in a PBR context what is important is that we haev some way to run PBR then setuptools without going through setup.py
19:46:52 <clarkb> the pep517 spells out the high level interface for doing that and mordred wrote a change to pbr to integrate with it that way
19:47:21 <clarkb> through testing I'm not entirely sure it is working as expected so trying to dig into that, but I doubt this is going to be a full time thing for me until it is fixed. More of a poke at it as able to learn more about the python packaging changes
19:47:28 <clarkb> if someone else ends up digging into it I won't be sad
19:48:25 <clarkb> fungi: re setup.py not being fully deprecated I think the effect won't be much different
19:48:56 <clarkb> fungi: if pip won't run setup.py to build stuff for you then you're going to need to at the very least add a pyproject.toml that says run setuptools but then you can just setup.cfg
19:49:05 <ianw> ok, thanks for the overview.  i'd been vaguely aware of how it hangs together and even added some toml files, etc. but i'll have to bootstrap myself on the pbr level details :)
19:49:12 <clarkb> I Guess for complicated builds that do compiles against external libs you might still have a setup.py
19:49:16 <fungi> well, they still expect you to be able to use setup.py to instrument setuptools, at least for a while
19:49:27 <clarkb> fungi: ya but for 99% of projects that is a setup.cfg :)
19:49:34 <fungi> lots of non-pure-python projects need it for things like building c extensions
19:49:48 <clarkb> in fact one thing we might consider for a number of projects is dropping pbr entirely in favor of setuptools + setup.cfg + pyproject.tml
19:49:51 <fungi> for pure python projects you're right
19:49:53 <clarkb> but one thing at a time
19:50:20 <fungi> we'd need setuptools-scm to do the versioning, at a minimum
19:50:26 <corvus> is it possible to get git versioning with that trio?
19:50:30 <corvus> answered :)
19:50:41 <fungi> and i don't think it generates authors files or changelogs either
19:51:02 <fungi> it might, i'd need to revisit it
19:51:17 <clarkb> ya I'm not yet sure what all the things we'd lose are. But possibility to simplify exists at least
19:51:26 <fungi> there are going to be lots of corner cases we've solved which likely aren't met by other tools out of the box
19:51:56 <fungi> gutting bits of pbr in favor of other modules/mechanisms which exist bit at a time would probably make the most sense
19:52:05 <ianw> heh, doesn't that imply that the replacement implements "build reasonableness".  that sounds like a big assumption :)
19:52:25 <fungi> and keeping pbr as a wrapper around those things for the transition
19:53:40 <clarkb> Alright anything else or can we go eat $meal now :)
19:55:11 <clarkb> #endmeeting