Tuesday, 2021-10-26

clarkbAnyone else here for the meeting? We will get started soon18:59
ianwo/18:59
clarkbAgenda is relatively light (been a busy few weeks). Hopefully that means I don't take up a bunch of your time18:59
clarkb#startmeeting infra19:01
opendevmeetMeeting started Tue Oct 26 19:01:02 2021 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.19:01
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.19:01
opendevmeetThe meeting name has been set to 'infra'19:01
clarkb#link http://lists.opendev.org/pipermail/service-discuss/2021-October/000293.html Our Agenda19:01
clarkb#topic Announcements19:01
clarkbThis wasn't on the agenda but I saw an email going by indicating fungi may be doing gpg stuff?19:01
clarkbfungi: ^ do you need us to go and sign that key in the near future?19:01
fungiooh, i'm actually in the midst of completing that now19:01
fungii'm updating the process, so not exactly no19:02
clarkbok, will wait for those updates then :)19:02
clarkb#topic Actions from last meeting19:02
clarkb#link http://eavesdrop.openstack.org/meetings/infra/2021/infra.2021-10-19-19.01.txt minutes from last meeting19:02
clarkbThere were no actions recorded last meeting19:02
fungiwith the sks keyserver network collapse, the next best option is to switch to the gnupg.org keyservers, but they don't publish thord-party signatures19:02
fungier, third-party19:02
clarkb#topic Specs19:03
clarkb#link https://review.opendev.org/810990 Mailman 3 spec19:03
clarkbThis is still needing reviews. infra-root if you can find time for this it would be greatly appreciated19:03
clarkb#action infra-root review Mailman 3 spec https://review.opendev.org/81099019:03
clarkbNot sure there is much more to say on that other than we need reviews.19:04
clarkb#topic Topics19:05
clarkb#topic Improving OpenDev's CD throughput19:05
clarkbianw: I rebased your first chagne to fix a merge conflict I introduced with some of the gerrit upgrade cleanup work. This change is +1 now19:05
clarkbianw: I guess we should try and land it this week?19:05
clarkb#link https://review.opendev.org/c/opendev/system-config/+/807672 is the change I'm talking about19:05
ianwi noticed some of the other dependent changes acquired a -1 somehow, i haven't dug into it yet19:06
ianwbut yep, we can keep at it :)19:06
clarkbah maybe we should ensure the rebase I did didn't introduce any of those problems19:06
clarkbI'll put this on my TODO list for this week though19:06
clarkb#topic Gerrit account cleanups19:07
clarkbI haven't done the bulk work on this that is on my TODO list but did do a one off to allow a user to log in again (they had problems and I cc'd funig and ianw on the last email since they have looked at gerrit account stuff before)19:07
clarkbThe log of the work I did for that user is on review02 in the typical location19:08
clarkbessentially I retired an old account and cleaned up an email address from that account so the user can associate it with a newer account they appear to be using (based on openid external ids)19:08
clarkb#topic Fedora 34 Booting Problems19:08
clarkbianw: ^ can you quickly fill us in on the fun around that?19:09
ianwat least one problem is that it seems the initramfs has stopped shipping the drivers to mount the root disk on Xen, i.e. RAX19:09
ianwthere is a dracut fix for that, but it's hard to deploy19:10
clarkb#link https://bugzilla.redhat.com/show_bug.cgi?id=2010058 fedora 34 initramfs lacking xen drivers19:10
ianw#link https://review.opendev.org/c/openstack/diskimage-builder/+/81540919:10
ianwis a generic fix for dracut-regnerate, which ... uses dracut to regenerate the initramfs19:11
clarkbianw: I left a comment on that dib change wondering about centos 719:11
ianw#link https://review.opendev.org/c/openstack/diskimage-builder/+/81538519:11
clarkbcentos 7 is python2.7 platform and uses dracut? Or maybe dracut is newer than centos 7?19:11
ianweven centos7 has python3 now19:12
clarkbianw: will dracut run with it?19:12
clarkbI thought that centos 3 was independent of all the normal system tools19:12
ianwit's not dracut we need python-yaml there for, it's to run the dracut-regenerate tool from bin/ which is written in python19:13
clarkboh!19:13
ianwyeah, a bit confusing19:14
clarkbI've updated my review to a +2 thanks19:14
clarkbThen we also need to install some special dracut testing package because they haven't landed it upstream yet?19:15
ianwyeah; and then figure out how to test it, or deploy it19:16
ianwthe most expedient method would be to hand-edit in the change to the nodepool-builder container19:16
ianwmy previous effort at that wasn't a shining example19:16
clarkbProbbaly the best thing is to do a small local build without all the git repos and extra stuff. Then manually upload to rax and boot it. The hardest thing about that is the manually upload step since rax image upload process si wird19:17
clarkbyou're basically forced to use shade/openstacksdk19:17
ianwyes ...19:18
ianwwhat i was trying to avoid :)19:18
ianwi'm not sure if they're going to regenerate the default initramfs for f34 or what19:19
clarkbYou'd hope they will since it has anothre 7 months or so of life. Kind of annoying that fixing existing users is less important that addressing an unreleased release19:20
ianwif i could get into a rescue mode i could copy in the initramfs, but that doesn't seem to work either19:20
clarkbianw: I think you may have to override the rescue mode host image. By default it uses the image that is being rescued19:21
ianwyeah, i thought the idea was that it boots into a completely separate mini-vm that then has your old vm as a block device19:22
clarkbianw: it does, but it has to have an image to boot that separate VM with and by default it uses the same image I think19:22
clarkbthere is a flag to set a different image for the boot which might solve the problem19:23
ianwhuh, ok ... well i'll have a poke but i wasn't aware of that flag19:23
clarkbAnything else on this topic before we move on?19:24
ianwi guess i'll have to try the build method19:24
ianwno, it's just annoying and not particularly what any of us want to be doing :)19:25
clarkbindeed, thank you for looking at it19:25
clarkb#topic Begin planning for Gerrit 3.4 upgrade19:25
clarkbWe're on 3.3 now but 3.4 exists and we should start looking at an upgrade to that version. As with 3.2 -> 3.3 we can do a downgrade from 3.4 -> 3.3 if we need to19:26
clarkbWe are already building 3.4 images and have an upgrade job to test the 3.3 -> 3.4 upgrade. There appears to only be an index change between the two and that gets updated online after restarting on 3.419:26
clarkb#link https://www.gerritcodereview.com/3.4.html Release notes19:27
clarkbI think it would be good if we can find time to read over release notes to find any scary things or things that we expect will need testing. Then hold a 3.4 test node and manually interact with it a little bit to look for any unexpected changes19:27
clarkbWe can also use that held node to test a downgrade to 3.3.19:28
ianw"HTML plugin support is removed" -- we don't have anything still hanging on to that do we?19:28
clarkbianw: I wonder if our theming is considered an html plugin?19:28
clarkbI don't know how that plugs into gerrit19:28
fungiit's a polygerrit plugin19:30
ianwhttps://groups.google.com/g/repo-discuss/c/YJY8oQZZo44/m/nF1BtuPQAwAJ lists "zuul-status" as requiring updates to polymer 319:30
ianw(our display is zuul-summary-status confusingly and isn't affected)19:31
clarkbzuul-status isn't one that we use either19:31
clarkbfungi: I guess we needt o double check that it is polymer 3 safe19:31
clarkbwe should be able to verify that with a held test instance (either it will work or not)19:32
clarkbThe other thing we should try to do is improve the automated testing with new stuff as we find it so that we're not needing to do extra checks for the 3.5 upgrade19:32
ianwi'm happy to take an action item to get a checklist started, setup a held node and investigate downgrade19:32
clarkb#action ianw Start Gerrit 3.4 upgrade checklist, setup a held node, and investigate the downgrade19:32
ianw++ yep anything we find that we can test we should :)19:33
clarkbianw: thanks. And ya at this point I think we just need one or two people to start looking at it a bit more in depth so that we can fix issues and then figure out a plan from there19:33
fungii expect our test screenshots alone will be sufficient for confirming the theme is okay19:33
fungibut deeper interactive testing is indeed warranted19:34
clarkbfungi: ya there are plenty of unknown unknowns and actually loading up the UI is a good way to discover that stuff19:34
clarkbsounds like we've got a plan to make a plan. Anything else we want to talk about on this subject?19:34
clarkb#topic Open Discussion19:36
clarkbWe reenabled the inmotion cloud today. Please keep an eye out for oddities there, but they made a number of changes whcih we hope will make for a happier cloud19:36
fungii'll have changes up shortly for the process updates and new signing key clarkb mentioned earlier19:36
clarkbthank you yuriys (not in here but still we appreciate it)19:36
clarkbI started looking at mordred's pep 517 change in pbr today. I think it may not work properly based on some early testing. setup.py is being deprecated and we'll need an option going forward one way or another and having PBR support for pep517 is probably a good stepping stone to something richer19:37
ianwif anyone was following the wheel discussion in zuul yesterday i wrote up19:37
ianw#link https://review.opendev.org/c/zuul/zuul/+/81540619:37
clarkboh yes I need to followup on that /me adds to the todo list :)19:38
corvusianw: it looks like you're proposing opendev wheel generation as the main solution?  (as opposed to a nodepool dependencies image?)19:38
ianwcorvus: i think it makes the most sense.  then anything that uses the python base images could use the wheels safely19:39
corvusas far as implementation details go, that probably makes it more of an opendev spec?19:39
ianwtrue, if that's what we agree on19:40
corvusmaybe it's sort of like the matrix spec:  "zuul project says we want this; details to be hashed out with opendev"?19:40
ianwlooking at the wheels actually used, i think pynacl is the big problem19:41
ianwi have offered there to help build manylinux arm64 wheels19:41
corvusianw: thanks for writing that up; i'll give it a look (probably mostly through the lens i described above)19:42
ianwclarkb: is it possible to summarise the pbr/pep517/setup.py thing?19:42
clarkbianw: ya basically pypa has decided taht the setup.py packaging interface is deprecated and will go away in the future. Setuptools and Wheel are not going anywhere just the setup.py script interface19:43
fungiianw: in short, setuptools has marked any invocations of `setup.py build` as well as setting setup_requires metadata or calling into easy_install as deprecated19:43
clarkbianw: this means that eventually you'll have to use some other tool like pip or build or poetry to manage python packages for your projects19:43
clarkbthe standard they are coalescing around supporting is the pep517/pep518 specifications where you put information about how to build your projects in pyproject.toml and then pip/build/poetry know what to do with them19:44
fungisetup.py isn't exactly deprecated, only calling it directly, as well as some things which they now feel should be done through pyproject.toml configuration19:44
fungibasically they would like to eventually be able to get rid of setuptools directly managing build requirements19:45
corvusi'd like to share a change that i did not push up for review:  i wrote, but did not push, a change to split out zuul geard serving to a separate container.  i wrote it to prep for running a second scheduler.  but i realized that the way we have everything set up, it's fine for a second scheduler to also run gearman because nothing will use it (even the second scheduler).  so i have that sitting on the shelf if we need it, but don't plan19:45
corvusto use it.  i will hopefully soon push up a change to spin up a new scheduler host for zuul.19:45
fungiso expect you to call a wrapper like https://pypi.org/project/build instead, which will ask pip to satisfy your build backend's requirements first before it calls into setuptools19:45
clarkbcorvus: that makes sense19:46
clarkbin a PBR context what is important is that we haev some way to run PBR then setuptools without going through setup.py19:46
clarkbthe pep517 spells out the high level interface for doing that and mordred wrote a change to pbr to integrate with it that way19:46
clarkbthrough testing I'm not entirely sure it is working as expected so trying to dig into that, but I doubt this is going to be a full time thing for me until it is fixed. More of a poke at it as able to learn more about the python packaging changes19:47
clarkbif someone else ends up digging into it I won't be sad19:47
clarkbfungi: re setup.py not being fully deprecated I think the effect won't be much different19:48
clarkbfungi: if pip won't run setup.py to build stuff for you then you're going to need to at the very least add a pyproject.toml that says run setuptools but then you can just setup.cfg19:48
ianwok, thanks for the overview.  i'd been vaguely aware of how it hangs together and even added some toml files, etc. but i'll have to bootstrap myself on the pbr level details :)19:49
clarkbI Guess for complicated builds that do compiles against external libs you might still have a setup.py19:49
fungiwell, they still expect you to be able to use setup.py to instrument setuptools, at least for a while19:49
clarkbfungi: ya but for 99% of projects that is a setup.cfg :)19:49
fungilots of non-pure-python projects need it for things like building c extensions19:49
clarkbin fact one thing we might consider for a number of projects is dropping pbr entirely in favor of setuptools + setup.cfg + pyproject.tml19:49
fungifor pure python projects you're right19:49
clarkbbut one thing at a time19:49
fungiwe'd need setuptools-scm to do the versioning, at a minimum19:50
corvusis it possible to get git versioning with that trio?19:50
corvusanswered :)19:50
fungiand i don't think it generates authors files or changelogs either19:50
fungiit might, i'd need to revisit it19:51
clarkbya I'm not yet sure what all the things we'd lose are. But possibility to simplify exists at least19:51
fungithere are going to be lots of corner cases we've solved which likely aren't met by other tools out of the box19:51
fungigutting bits of pbr in favor of other modules/mechanisms which exist bit at a time would probably make the most sense19:51
ianwheh, doesn't that imply that the replacement implements "build reasonableness".  that sounds like a big assumption :)19:52
fungiand keeping pbr as a wrapper around those things for the transition19:52
clarkbAlright anything else or can we go eat $meal now :)19:53
clarkb#endmeeting19:55
opendevmeetMeeting ended Tue Oct 26 19:55:11 2021 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)19:55
opendevmeetMinutes:        https://meetings.opendev.org/meetings/infra/2021/infra.2021-10-26-19.01.html19:55
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/infra/2021/infra.2021-10-26-19.01.txt19:55
opendevmeetLog:            https://meetings.opendev.org/meetings/infra/2021/infra.2021-10-26-19.01.log.html19:55
clarkbthanks everyone!19:55
fungithanks clarkb!19:56

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!