| clarkb | Anyone else here for the meeting? We will get started soon | 18:59 |
|---|---|---|
| ianw | o/ | 18:59 |
| clarkb | Agenda is relatively light (been a busy few weeks). Hopefully that means I don't take up a bunch of your time | 18:59 |
| clarkb | #startmeeting infra | 19:01 |
| opendevmeet | Meeting started Tue Oct 26 19:01:02 2021 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. | 19:01 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 19:01 |
| opendevmeet | The meeting name has been set to 'infra' | 19:01 |
| clarkb | #link http://lists.opendev.org/pipermail/service-discuss/2021-October/000293.html Our Agenda | 19:01 |
| clarkb | #topic Announcements | 19:01 |
| clarkb | This wasn't on the agenda but I saw an email going by indicating fungi may be doing gpg stuff? | 19:01 |
| clarkb | fungi: ^ do you need us to go and sign that key in the near future? | 19:01 |
| fungi | ooh, i'm actually in the midst of completing that now | 19:01 |
| fungi | i'm updating the process, so not exactly no | 19:02 |
| clarkb | ok, will wait for those updates then :) | 19:02 |
| clarkb | #topic Actions from last meeting | 19:02 |
| clarkb | #link http://eavesdrop.openstack.org/meetings/infra/2021/infra.2021-10-19-19.01.txt minutes from last meeting | 19:02 |
| clarkb | There were no actions recorded last meeting | 19:02 |
| fungi | with the sks keyserver network collapse, the next best option is to switch to the gnupg.org keyservers, but they don't publish thord-party signatures | 19:02 |
| fungi | er, third-party | 19:02 |
| clarkb | #topic Specs | 19:03 |
| clarkb | #link https://review.opendev.org/810990 Mailman 3 spec | 19:03 |
| clarkb | This is still needing reviews. infra-root if you can find time for this it would be greatly appreciated | 19:03 |
| clarkb | #action infra-root review Mailman 3 spec https://review.opendev.org/810990 | 19:03 |
| clarkb | Not sure there is much more to say on that other than we need reviews. | 19:04 |
| clarkb | #topic Topics | 19:05 |
| clarkb | #topic Improving OpenDev's CD throughput | 19:05 |
| clarkb | ianw: I rebased your first chagne to fix a merge conflict I introduced with some of the gerrit upgrade cleanup work. This change is +1 now | 19:05 |
| clarkb | ianw: I guess we should try and land it this week? | 19:05 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/807672 is the change I'm talking about | 19:05 |
| ianw | i noticed some of the other dependent changes acquired a -1 somehow, i haven't dug into it yet | 19:06 |
| ianw | but yep, we can keep at it :) | 19:06 |
| clarkb | ah maybe we should ensure the rebase I did didn't introduce any of those problems | 19:06 |
| clarkb | I'll put this on my TODO list for this week though | 19:06 |
| clarkb | #topic Gerrit account cleanups | 19:07 |
| clarkb | I haven't done the bulk work on this that is on my TODO list but did do a one off to allow a user to log in again (they had problems and I cc'd funig and ianw on the last email since they have looked at gerrit account stuff before) | 19:07 |
| clarkb | The log of the work I did for that user is on review02 in the typical location | 19:08 |
| clarkb | essentially I retired an old account and cleaned up an email address from that account so the user can associate it with a newer account they appear to be using (based on openid external ids) | 19:08 |
| clarkb | #topic Fedora 34 Booting Problems | 19:08 |
| clarkb | ianw: ^ can you quickly fill us in on the fun around that? | 19:09 |
| ianw | at least one problem is that it seems the initramfs has stopped shipping the drivers to mount the root disk on Xen, i.e. RAX | 19:09 |
| ianw | there is a dracut fix for that, but it's hard to deploy | 19:10 |
| clarkb | #link https://bugzilla.redhat.com/show_bug.cgi?id=2010058 fedora 34 initramfs lacking xen drivers | 19:10 |
| ianw | #link https://review.opendev.org/c/openstack/diskimage-builder/+/815409 | 19:10 |
| ianw | is a generic fix for dracut-regnerate, which ... uses dracut to regenerate the initramfs | 19:11 |
| clarkb | ianw: I left a comment on that dib change wondering about centos 7 | 19:11 |
| ianw | #link https://review.opendev.org/c/openstack/diskimage-builder/+/815385 | 19:11 |
| clarkb | centos 7 is python2.7 platform and uses dracut? Or maybe dracut is newer than centos 7? | 19:11 |
| ianw | even centos7 has python3 now | 19:12 |
| clarkb | ianw: will dracut run with it? | 19:12 |
| clarkb | I thought that centos 3 was independent of all the normal system tools | 19:12 |
| ianw | it's not dracut we need python-yaml there for, it's to run the dracut-regenerate tool from bin/ which is written in python | 19:13 |
| clarkb | oh! | 19:13 |
| ianw | yeah, a bit confusing | 19:14 |
| clarkb | I've updated my review to a +2 thanks | 19:14 |
| clarkb | Then we also need to install some special dracut testing package because they haven't landed it upstream yet? | 19:15 |
| ianw | yeah; and then figure out how to test it, or deploy it | 19:16 |
| ianw | the most expedient method would be to hand-edit in the change to the nodepool-builder container | 19:16 |
| ianw | my previous effort at that wasn't a shining example | 19:16 |
| clarkb | Probbaly the best thing is to do a small local build without all the git repos and extra stuff. Then manually upload to rax and boot it. The hardest thing about that is the manually upload step since rax image upload process si wird | 19:17 |
| clarkb | you're basically forced to use shade/openstacksdk | 19:17 |
| ianw | yes ... | 19:18 |
| ianw | what i was trying to avoid :) | 19:18 |
| ianw | i'm not sure if they're going to regenerate the default initramfs for f34 or what | 19:19 |
| clarkb | You'd hope they will since it has anothre 7 months or so of life. Kind of annoying that fixing existing users is less important that addressing an unreleased release | 19:20 |
| ianw | if i could get into a rescue mode i could copy in the initramfs, but that doesn't seem to work either | 19:20 |
| clarkb | ianw: I think you may have to override the rescue mode host image. By default it uses the image that is being rescued | 19:21 |
| ianw | yeah, i thought the idea was that it boots into a completely separate mini-vm that then has your old vm as a block device | 19:22 |
| clarkb | ianw: it does, but it has to have an image to boot that separate VM with and by default it uses the same image I think | 19:22 |
| clarkb | there is a flag to set a different image for the boot which might solve the problem | 19:23 |
| ianw | huh, ok ... well i'll have a poke but i wasn't aware of that flag | 19:23 |
| clarkb | Anything else on this topic before we move on? | 19:24 |
| ianw | i guess i'll have to try the build method | 19:24 |
| ianw | no, it's just annoying and not particularly what any of us want to be doing :) | 19:25 |
| clarkb | indeed, thank you for looking at it | 19:25 |
| clarkb | #topic Begin planning for Gerrit 3.4 upgrade | 19:25 |
| clarkb | We're on 3.3 now but 3.4 exists and we should start looking at an upgrade to that version. As with 3.2 -> 3.3 we can do a downgrade from 3.4 -> 3.3 if we need to | 19:26 |
| clarkb | We are already building 3.4 images and have an upgrade job to test the 3.3 -> 3.4 upgrade. There appears to only be an index change between the two and that gets updated online after restarting on 3.4 | 19:26 |
| clarkb | #link https://www.gerritcodereview.com/3.4.html Release notes | 19:27 |
| clarkb | I think it would be good if we can find time to read over release notes to find any scary things or things that we expect will need testing. Then hold a 3.4 test node and manually interact with it a little bit to look for any unexpected changes | 19:27 |
| clarkb | We can also use that held node to test a downgrade to 3.3. | 19:28 |
| ianw | "HTML plugin support is removed" -- we don't have anything still hanging on to that do we? | 19:28 |
| clarkb | ianw: I wonder if our theming is considered an html plugin? | 19:28 |
| clarkb | I don't know how that plugs into gerrit | 19:28 |
| fungi | it's a polygerrit plugin | 19:30 |
| ianw | https://groups.google.com/g/repo-discuss/c/YJY8oQZZo44/m/nF1BtuPQAwAJ lists "zuul-status" as requiring updates to polymer 3 | 19:30 |
| ianw | (our display is zuul-summary-status confusingly and isn't affected) | 19:31 |
| clarkb | zuul-status isn't one that we use either | 19:31 |
| clarkb | fungi: I guess we needt o double check that it is polymer 3 safe | 19:31 |
| clarkb | we should be able to verify that with a held test instance (either it will work or not) | 19:32 |
| clarkb | The other thing we should try to do is improve the automated testing with new stuff as we find it so that we're not needing to do extra checks for the 3.5 upgrade | 19:32 |
| ianw | i'm happy to take an action item to get a checklist started, setup a held node and investigate downgrade | 19:32 |
| clarkb | #action ianw Start Gerrit 3.4 upgrade checklist, setup a held node, and investigate the downgrade | 19:32 |
| ianw | ++ yep anything we find that we can test we should :) | 19:33 |
| clarkb | ianw: thanks. And ya at this point I think we just need one or two people to start looking at it a bit more in depth so that we can fix issues and then figure out a plan from there | 19:33 |
| fungi | i expect our test screenshots alone will be sufficient for confirming the theme is okay | 19:33 |
| fungi | but deeper interactive testing is indeed warranted | 19:34 |
| clarkb | fungi: ya there are plenty of unknown unknowns and actually loading up the UI is a good way to discover that stuff | 19:34 |
| clarkb | sounds like we've got a plan to make a plan. Anything else we want to talk about on this subject? | 19:34 |
| clarkb | #topic Open Discussion | 19:36 |
| clarkb | We reenabled the inmotion cloud today. Please keep an eye out for oddities there, but they made a number of changes whcih we hope will make for a happier cloud | 19:36 |
| fungi | i'll have changes up shortly for the process updates and new signing key clarkb mentioned earlier | 19:36 |
| clarkb | thank you yuriys (not in here but still we appreciate it) | 19:36 |
| clarkb | I started looking at mordred's pep 517 change in pbr today. I think it may not work properly based on some early testing. setup.py is being deprecated and we'll need an option going forward one way or another and having PBR support for pep517 is probably a good stepping stone to something richer | 19:37 |
| ianw | if anyone was following the wheel discussion in zuul yesterday i wrote up | 19:37 |
| ianw | #link https://review.opendev.org/c/zuul/zuul/+/815406 | 19:37 |
| clarkb | oh yes I need to followup on that /me adds to the todo list :) | 19:38 |
| corvus | ianw: it looks like you're proposing opendev wheel generation as the main solution? (as opposed to a nodepool dependencies image?) | 19:38 |
| ianw | corvus: i think it makes the most sense. then anything that uses the python base images could use the wheels safely | 19:39 |
| corvus | as far as implementation details go, that probably makes it more of an opendev spec? | 19:39 |
| ianw | true, if that's what we agree on | 19:40 |
| corvus | maybe it's sort of like the matrix spec: "zuul project says we want this; details to be hashed out with opendev"? | 19:40 |
| ianw | looking at the wheels actually used, i think pynacl is the big problem | 19:41 |
| ianw | i have offered there to help build manylinux arm64 wheels | 19:41 |
| corvus | ianw: thanks for writing that up; i'll give it a look (probably mostly through the lens i described above) | 19:42 |
| ianw | clarkb: is it possible to summarise the pbr/pep517/setup.py thing? | 19:42 |
| clarkb | ianw: ya basically pypa has decided taht the setup.py packaging interface is deprecated and will go away in the future. Setuptools and Wheel are not going anywhere just the setup.py script interface | 19:43 |
| fungi | ianw: in short, setuptools has marked any invocations of `setup.py build` as well as setting setup_requires metadata or calling into easy_install as deprecated | 19:43 |
| clarkb | ianw: this means that eventually you'll have to use some other tool like pip or build or poetry to manage python packages for your projects | 19:43 |
| clarkb | the standard they are coalescing around supporting is the pep517/pep518 specifications where you put information about how to build your projects in pyproject.toml and then pip/build/poetry know what to do with them | 19:44 |
| fungi | setup.py isn't exactly deprecated, only calling it directly, as well as some things which they now feel should be done through pyproject.toml configuration | 19:44 |
| fungi | basically they would like to eventually be able to get rid of setuptools directly managing build requirements | 19:45 |
| corvus | i'd like to share a change that i did not push up for review: i wrote, but did not push, a change to split out zuul geard serving to a separate container. i wrote it to prep for running a second scheduler. but i realized that the way we have everything set up, it's fine for a second scheduler to also run gearman because nothing will use it (even the second scheduler). so i have that sitting on the shelf if we need it, but don't plan | 19:45 |
| corvus | to use it. i will hopefully soon push up a change to spin up a new scheduler host for zuul. | 19:45 |
| fungi | so expect you to call a wrapper like https://pypi.org/project/build instead, which will ask pip to satisfy your build backend's requirements first before it calls into setuptools | 19:45 |
| clarkb | corvus: that makes sense | 19:46 |
| clarkb | in a PBR context what is important is that we haev some way to run PBR then setuptools without going through setup.py | 19:46 |
| clarkb | the pep517 spells out the high level interface for doing that and mordred wrote a change to pbr to integrate with it that way | 19:46 |
| clarkb | through testing I'm not entirely sure it is working as expected so trying to dig into that, but I doubt this is going to be a full time thing for me until it is fixed. More of a poke at it as able to learn more about the python packaging changes | 19:47 |
| clarkb | if someone else ends up digging into it I won't be sad | 19:47 |
| clarkb | fungi: re setup.py not being fully deprecated I think the effect won't be much different | 19:48 |
| clarkb | fungi: if pip won't run setup.py to build stuff for you then you're going to need to at the very least add a pyproject.toml that says run setuptools but then you can just setup.cfg | 19:48 |
| ianw | ok, thanks for the overview. i'd been vaguely aware of how it hangs together and even added some toml files, etc. but i'll have to bootstrap myself on the pbr level details :) | 19:49 |
| clarkb | I Guess for complicated builds that do compiles against external libs you might still have a setup.py | 19:49 |
| fungi | well, they still expect you to be able to use setup.py to instrument setuptools, at least for a while | 19:49 |
| clarkb | fungi: ya but for 99% of projects that is a setup.cfg :) | 19:49 |
| fungi | lots of non-pure-python projects need it for things like building c extensions | 19:49 |
| clarkb | in fact one thing we might consider for a number of projects is dropping pbr entirely in favor of setuptools + setup.cfg + pyproject.tml | 19:49 |
| fungi | for pure python projects you're right | 19:49 |
| clarkb | but one thing at a time | 19:49 |
| fungi | we'd need setuptools-scm to do the versioning, at a minimum | 19:50 |
| corvus | is it possible to get git versioning with that trio? | 19:50 |
| corvus | answered :) | 19:50 |
| fungi | and i don't think it generates authors files or changelogs either | 19:50 |
| fungi | it might, i'd need to revisit it | 19:51 |
| clarkb | ya I'm not yet sure what all the things we'd lose are. But possibility to simplify exists at least | 19:51 |
| fungi | there are going to be lots of corner cases we've solved which likely aren't met by other tools out of the box | 19:51 |
| fungi | gutting bits of pbr in favor of other modules/mechanisms which exist bit at a time would probably make the most sense | 19:51 |
| ianw | heh, doesn't that imply that the replacement implements "build reasonableness". that sounds like a big assumption :) | 19:52 |
| fungi | and keeping pbr as a wrapper around those things for the transition | 19:52 |
| clarkb | Alright anything else or can we go eat $meal now :) | 19:53 |
| clarkb | #endmeeting | 19:55 |
| opendevmeet | Meeting ended Tue Oct 26 19:55:11 2021 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 19:55 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/infra/2021/infra.2021-10-26-19.01.html | 19:55 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/infra/2021/infra.2021-10-26-19.01.txt | 19:55 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/infra/2021/infra.2021-10-26-19.01.log.html | 19:55 |
| clarkb | thanks everyone! | 19:55 |
| fungi | thanks clarkb! | 19:56 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!