| clarkb | my gerrit show-queue isn't going through to see what it is spinning on. I've manually stopped apache to see if that backs off some of the demand in order to get that info. If it doesn't help I may just go ahwad and stop gerrit | 00:04 |
|---|---|---|
| *** dtantsur_ is now known as dtantsur | 00:08 | |
| SamuelAllan[m] | Hi, who should I talk to about requesting a username change on opendev? Thanks. 🙂 | 00:21 |
| clarkb | SamuelAllan[m]: I'm in the middle of trying to get gerrit into a happy state again. But we will need more specifics. What service? | 00:23 |
| clarkb | mnaser: I identified a problematic IP address, blocked them, then restarted gerrit and apache. I think things are coming back now | 00:24 |
| clarkb | mnaser: if you are still around can you confirm (or not) if things are looking better for you too? | 00:25 |
| clarkb | #status log Restarted Gerrit and apache on review02 to unstuck Gerrit in an unhappy state consuming all the CPU | 00:25 |
| opendevstatus | clarkb: finished logging | 00:25 |
| SamuelAllan[m] | clarkb: ah no rush, all the best. This is for https://review.opendev.org/ though. | 00:28 |
| mnaser | clarkb: yeah, things are loading properly now. | 00:30 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update Gerrit 3.8 to 3.8.5 and 3.9 to 3.9.4 https://review.opendev.org/c/opendev/system-config/+/912470 | 00:33 |
| clarkb | mnaser: thanks for confirming ^ I used to check things out a bit more and pushing works | 00:33 |
| clarkb | SamuelAllan[m]: Gerrit doesn't support username changes. The username is fairly opaque though and is really only something that you need to set when talking to Gerrit via ssh or the http api | 00:34 |
| clarkb | SamuelAllan[m]: https://groups.google.com/g/repo-discuss/c/y3Kq0YiKC7Q/m/1HT6w6QA7tQJ there is more info in this thread upstream as to why this is the case | 00:35 |
| SamuelAllan[m] | Ah that's sad, too bad the branch names aren't on some numeric user IDs :/ | 00:36 |
| SamuelAllan[m] | Guess I'm stuck with it then. Thanks for checking. | 00:37 |
| clarkb | SamuelAllan[m]: you can stick the username in your .gitreview file then in theory never think about it again | 00:37 |
| clarkb | er no your .gitconfig | 00:37 |
| clarkb | `git config --global gitreview.username YOURUSERNAMEHERE` | 00:37 |
| SamuelAllan[m] | Yep that's what I have, so it's not a big deal. It's just that I'd changed my surname, so I've been trying to update everywhere that references my old name where possible. | 00:38 |
| clarkb | ya there are definitely reasons it would be nice to do so. Unfortunately they made some explicit choices to not support it (and even removed the functionality...) | 00:39 |
| SamuelAllan[m] | Fair enough :) | 00:39 |
| clarkb | SamuelAllan[m]: you should be able to change your Full Name and add extra email addresses to the gerrit account though | 00:45 |
| clarkb | and typically that is what humans will see day to day | 00:45 |
| clarkb | that can be done under the settings page in gerrit once you login | 00:45 |
| SamuelAllan[m] | Yep I've done that, thanks :) | 00:50 |
| opendevreview | Dr. Jens Harbott proposed opendev/glean master: Add tox jobs on modern python versions https://review.opendev.org/c/opendev/glean/+/915827 | 06:59 |
| frickler | infra-root: glean no longer works on py3.12, sadly this isn't covered by unittests though, see ^^ vs. https://paste.opendev.org/show/bXcyjOeCWzQFvYOY3tXR/ | 09:00 |
| fungi | hopefully we can reuse one of the other importlib replacement patterns we've already applied elsewhere | 11:22 |
| dtantsur | Hi folks! Not exactly the right channel, but people here may know, sooo.. Does anyone still use gertty? Were you able to make it work with Python 3.12 and latest dependencies? | 11:46 |
| dtantsur | I miss it a lot :( | 11:46 |
| fungi | dtantsur: yeah, i'm running it on python 3.12 | 11:54 |
| fungi | have you tried installing the current branch tip? that might help | 11:55 |
| dtantsur | fungi: yep, just did git pull, running of 5b577fd11608b8e62e272f54e239722725b6d940 Merge "Fix urwid > 2.4.2 compatibility" | 12:23 |
| dtantsur | (or rather "not running" since the sync loop crashes with broken pipe, among other problems) | 12:23 |
| dtantsur | mmm, interesting, it works better inside a venv than with `pip install --user` | 12:26 |
| dtantsur | I wonder if some Fedora packages get in the way | 12:27 |
| fungi | dtantsur: oh, yeeah i just totally run it from a dedicated venv, always have | 12:37 |
| dtantsur | lesson learned, thanks for confirming :) | 12:37 |
| fungi | i think the python language ecosystem as a whole has come to the conclusion that pip install --user was almost as bad an idea as sudo pip install | 12:41 |
| fungi | every python-based tool i install that isn't packaged by my distro gets its own venv in ~/lib and then i put symlinks to the relevant entrypoints in ~/bin | 12:42 |
| clarkb | infra-root if we can get reviews on https://review.opendev.org/c/opendev/system-config/+/912470 and get that landed today I should be able to do a gerrit service restart onto the new 3.8.5 image | 15:10 |
| clarkb | I believe the 3.9 updates will also address our existing concerns with the 3.9 upgrade path (and results) so we'll be able to start testing that properly afterwards too | 15:11 |
| fungi | lgtm, i'm around for a restart whenever | 15:14 |
| dtantsur | I haven't enjoyed gerrty for long: sqlalchemy.exc.ArgumentError: Additional keyword arguments are not accepted by this function/method. The presence of **kw is for pep-484 typing purposes | 15:17 |
| dtantsur | This is literally clicking on any project. So it's still completely broken, just in a different way than before :( | 15:17 |
| dtantsur | fungi: do you know if it's actually working with SQLA 2? | 15:18 |
| fungi | current branch state should be, sqla v2 support merged in february | 15:19 |
| fungi | i'll check which version i have in my venv | 15:19 |
| dtantsur | my pip freeze https://paste.opendev.org/show/bh4WWDVMhC9wmok78x7B/ | 15:23 |
| fungi | dtantsur: my workstation's gertty venv is apparently still using SQLAlchemy==1.4.49 | 15:23 |
| dtantsur | hmm, lemme try with that | 15:23 |
| fungi | and indeed if i try with a venv containing SQLAlchemy==2.0.29 it doesn't even start for me before i get a traceback | 15:24 |
| fungi | sqlalchemy.exc.InvalidRequestError: The 'sqlalchemy.orm.mapper()' function is removed as of SQLAlchemy 2.0. Use the 'sqlalchemy.orm.registry.map_imperatively()` method of the ``sqlalchemy.orm.registry`` class to perform classical mapping. | 15:24 |
| dtantsur | yep, 1.4.52 works | 15:24 |
| fungi | prometheanfire: ^ do you happen to know if your patch for sqla v2 was incomplete in some way? | 15:25 |
| fungi | oh, wait, my test was only with released gertty | 15:26 |
| dtantsur | I'm using git master | 15:26 |
| fungi | yeah, i'm rebuilding my venv now | 15:30 |
| fungi | realized the system where i had a newer venv built for it was installing git master older than when the sqla v2 patch merged | 15:31 |
| fungi | (not a machine where i run it often) | 15:32 |
| clarkb | frickler: fungi: this importlib vs pkg_resources thing is actually a bit annoying beacuse the importlib stuff doesn't supprot what we need until python3.12. So we basically have to use pkg_resources until pkg_resources is gone which inverts the import checks from how I would prefer to do it... | 15:41 |
| fungi | that's similar to how we did it in pbr though, yes? | 15:44 |
| fungi | isn't there a specific importlib submodule we need which doesn't exist until 3.12 and we can try to import that first? | 15:45 |
| clarkb | fungi: the submodule exists in 3.7, but then adds methods we need in 3.9 but doesn't support functioanlity we need until 3.12 | 15:45 |
| fungi | or is it just a new feature that's suddenly supported by an pre-existing part of importlib | 15:45 |
| fungi | aha, okay, yeah that's ugly | 15:46 |
| clarkb | its kidna like they said "we're deleting the odl thing" then last minute realized the new thing was nowhere near as useful | 15:46 |
| clarkb | specifically we need to be able to copy directories out of package installs | 15:46 |
| fungi | well, they've also walked back some of the deprecation plans for parts of importlib in 3.13, so still can't seem to make up their minds as to what they want | 15:46 |
| fungi | stuff that's been deprecated is apparently going to get un-deprecated | 15:47 |
| clarkb | fun | 15:49 |
| clarkb | the deprecated stuff can't do directories either though | 15:50 |
| frickler | so ... maybe we can finally switch to cloud-init? | 15:56 |
| fungi | did cloud-init drop all its python dependencies finally? | 15:56 |
| fungi | or i suppose we could try to run it from an isolated venv | 15:57 |
| frickler | well we run devstack in an isolated venv now | 15:57 |
| fungi | i guess it's really distro-packaged cloud-init which was causing the dependency painpoint for us | 15:57 |
| frickler | or should, if it weren't for centos | 15:57 |
| clarkb | I don't have any desire to use cloud init fwiw | 15:58 |
| clarkb | its not just the dependency problem. cloud init changes its behavior under you when your cloud changes what goes into metadata | 15:58 |
| clarkb | and I have what i think is a working patch I'm just doing testing | 15:59 |
| fungi | yeah, we'd be playing whack-a-mole with figuring out how to force disable a bunch of cloud-init features in order to ensure as much cross-cloud and cross-platform behavior consistency as possible | 15:59 |
| clarkb | my comments were moer that python hasn't planned out this deprecation path properly not that glean is a dead end | 16:00 |
| opendevreview | Clark Boylan proposed opendev/glean master: Use importlib when pkg_resources isn't available https://review.opendev.org/c/opendev/glean/+/915907 | 16:05 |
| clarkb | something like that should work I think. The dib image build functional testing should be good coverage of that at least covering that it doesn't regress python 3.11 and odler | 16:06 |
| opendevreview | Clark Boylan proposed opendev/glean master: Use importlib when pkg_resources isn't available https://review.opendev.org/c/opendev/glean/+/915907 | 16:11 |
| clarkb | it helps to write syntactically correct python though | 16:11 |
| fungi | prometheanfire: with new enough gertty to include your sqla v2 support patch, i'm still getting "TypeError: MetaData.__init__() got an unexpected keyword argument 'bind'" from https://opendev.org/ttygroup/gertty/src/branch/master/gertty/dbsupport.py#L36 under latest sqla, any ideas? | 16:13 |
| clarkb | fungi: the gerrit 3.8.5 screenshots lgtm. Maybe we give that another 45 mintues or so for anyone else to review it and if no one does by then we approve it and plan a restart later today? | 16:13 |
| fungi | sgtm, yep | 16:14 |
| fungi | reviewers still have plenty of opportunity to object and call a halt to the restart later, worst case | 16:15 |
| fungi | i'll plan to +3 it at 17:00 utc | 16:15 |
| opendevreview | Clark Boylan proposed opendev/glean master: Use importlib when pkg_resources isn't available https://review.opendev.org/c/opendev/glean/+/915907 | 16:22 |
| clarkb | frickler: where were you testing glean with python3.12? I wonder if I can set a depends on or similar to get that covered or if it was manual? | 16:24 |
| fungi | approved 912470 just now | 17:00 |
| clarkb | thanks | 17:09 |
| clarkb | I've just finished up another round of debugging on my held etherpad node https://github.com/ether/etherpad-lite/issues/6320#issuecomment-2057419506 is the result | 17:09 |
| clarkb | I'll see what they say in response then possibly update the change and hold another node | 17:10 |
| opendevreview | Clark Boylan proposed opendev/glean master: Use importlib when pkg_resources isn't available https://review.opendev.org/c/opendev/glean/+/915907 | 17:11 |
| frickler | clarkb: I tried building noble downstream, I can see if I can test with your patch tomorrow | 17:15 |
| clarkb | frickler: sounds good | 17:15 |
| opendevreview | Dr. Jens Harbott proposed openstack/diskimage-builder master: Add Ubuntu 24.04 (noble) build to testing https://review.opendev.org/c/openstack/diskimage-builder/+/915915 | 17:19 |
| frickler | clarkb: maybe ^^ shows the issue, too. on my older local nodepool I needed to fix the older debootstrap first | 17:20 |
| clarkb | ack we shall see. I'll update with depends on if it looks like that is the next fix | 17:21 |
| opendevreview | Merged opendev/system-config master: Update Gerrit 3.8 to 3.8.5 and 3.9 to 3.9.4 https://review.opendev.org/c/opendev/system-config/+/912470 | 18:07 |
| frickler | seems we need debootstrap from unstable again: E: No such script: /usr/share/debootstrap/scripts/noble https://zuul.opendev.org/t/openstack/build/ef4d15ca9beb4733a6eee91a6d350d05 | 18:08 |
| clarkb | frickler: I guess that should be expected though I'm always surprised that debootstrap isn't more flexible. | 18:12 |
| clarkb | fungi: gerrit image change has landed I've got lunch coming up but after that may be a good time for a restart | 18:12 |
| fungi | clarkb: yeah, it's deployed now | 18:26 |
| fungi | so we can start whenever. after your lunch works for me | 18:26 |
| fungi | my only afk agenda is to cook dinner, but that's probably at least a few hours out still | 18:26 |
| clarkb | fungi: we could also just do it right now? | 18:30 |
| fungi | wfm | 18:30 |
| fungi | status notice The Gerrit service on review.opendev.org will be offline momentarily for a restart in order to apply a patch update | 18:30 |
| fungi | something like that work? | 18:30 |
| fungi | `sudo mv /home/gerrit2/review_site/data/replication/ref-updates/waiting /home/gerrit2/tmp/upgrade-3.8.5/` between the down and the up -d? | 18:31 |
| clarkb | ya that all looks good. We also need to pull the new image | 18:33 |
| clarkb | I don't think the docker-compose pull is automatic | 18:33 |
| fungi | ah, i probably assumed incorrectly that the deploy job would | 18:33 |
| fungi | so this... | 18:33 |
| fungi | sudo docker-compose pull && sudo mkdir -p /home/gerrit2/tmp/upgrade-3.8.5 && sudo docker-compose -f /etc/gerrit-compose/docker-compose.yaml down && sudo mv /home/gerrit2/review_site/data/replication/ref-updates/waiting /home/gerrit2/tmp/upgrade-3.8.5/ && sudo docker-compose -f /etc/gerrit-compose/docker-compose.yaml up -d | 18:33 |
| clarkb | yup that looks right. As a general rule I avoid using docker-compose -f in case I forget that there are env vars that amtter but in this case gerrit's compose file doesn't use them iirc | 18:34 |
| fungi | #status notice The Gerrit service on review.opendev.org will be offline momentarily for a restart in order to apply a patch update | 18:34 |
| opendevstatus | fungi: sending notice | 18:34 |
| -opendevstatus- NOTICE: The Gerrit service on review.opendev.org will be offline momentarily for a restart in order to apply a patch update | 18:35 | |
| fungi | pull completed and `docker image list` mentions "opendevorg/gerrit 3.8 8da8e1247944 58 minutes ago 682MB" | 18:35 |
| fungi | that seems to be the one we expect | 18:35 |
| fungi | restarting on it now with the queue mv in the middle | 18:36 |
| clarkb | thanks | 18:36 |
| clarkb | INFO com.google.gerrit.pgm.Daemon : Gerrit Code Review 3.8.5-2-g7e93b96923-dirty ready | 18:36 |
| fungi | webui is up again and says "Powered by Gerrit Code Review (3.8.5-2-g7e93b96923-dirty)" | 18:37 |
| opendevstatus | fungi: finished sending notice | 18:37 |
| fungi | all done | 18:38 |
| clarkb | diffs load for me as well | 18:39 |
| fungi | yeah, poking around everything seems to be working as usual | 18:39 |
| opendevreview | Clark Boylan proposed opendev/glean master: Use importlib when pkg_resources isn't available https://review.opendev.org/c/opendev/glean/+/915907 | 19:21 |
| mnaser | question: how does opendev get the ability to have multiple python versions to test against? | 20:13 |
| mnaser | is there some ppa you preload or something along those lines? | 20:13 |
| fungi | mnaser: mostly we map abstract job definitions to specific nodesets for platforms where those versions are available already | 20:14 |
| mnaser | aaaaah | 20:15 |
| fungi | mnaser: in some cases we set ensure-python to use its pyenv method instead of distro packages | 20:15 |
| mnaser | so py311 jobs are .. debian i guess? | 20:15 |
| mnaser | oh i see | 20:15 |
| mnaser | https://opendev.org/openstack/openstack-zuul-jobs/src/branch/master/zuul.d/jobs.yaml#L351-L367 | 20:16 |
| mnaser | i see | 20:16 |
| fungi | if you pull up an example build you can easily see what platform it used, though some ubuntu lts versions provide more than one python minor version in their package repositories already | 20:16 |
| mnaser | jammy for 3.10, bookwarm for 3.11 and 3.12 bookworm but using pyenv | 20:16 |
| mnaser | that makes a lot of sense | 20:16 |
| mnaser | ok thats super handy, thank you | 20:17 |
| fungi | those behaviors are mostly descended from openstack's testing needs, where "such and such python version" has really meant "a popular platform which provides this python version" | 20:18 |
| mnaser | yeah, well i'm slowly moving our magnum-cluster-api driver into zuul and one of the challenges is testing across multiple python versions (because it runs with openstack :)) | 20:18 |
| fungi | if we were much more strict about it, we'd probably pre-build stow images for the latest of each minor cpython interpreter on a consistent platform and use ensure-pythons stow method, or something | 20:19 |
| mnaser | so needed to tackle the same thing, but realized that just running the job doesn't do the trick, it tries to apt install and flips its hand | 20:19 |
| mnaser | as much of a big overhead it is, this feels like something that a container would benefit from | 20:19 |
| fungi | the benefit of the current approach is that we're testing on a variety of platforms, so getting different libvirt, qemu, kernel, and so on | 20:20 |
| mnaser | yeah that is an extra layer of coverage | 20:20 |
| mnaser | in my case its pure python unit tests so it doesnt matter (in theory..) but yeah | 20:20 |
| fungi | it's a bigger concern for frameworks like devstack which are calling system-installed tools, c libs, kernel interfaces, and so on | 20:21 |
| clarkb | pyenv has been working for zuul to test against py312 fwiw | 20:24 |
| clarkb | it adds a couple minutes to each job but then the runtime is inline with the packaed 311 jobs | 20:24 |
| fungi | but even beyond that, the distro-packaged cpythons have a variety of distro-specific packages, so cpython as distributed by rocky isn't identical to cpython as distributed by debian, even for similar versions | 20:29 |
| fungi | back in the bad old days, openstack used to have to work around behavior differences due to things red hat was backporting into their cpython builds, for example | 20:30 |
| fungi | er, distro-specific patches i meant | 20:31 |
| opendevreview | Clark Boylan proposed opendev/system-config master: DNM Update etherpad to future v2.0.3 https://review.opendev.org/c/opendev/system-config/+/914119 | 20:39 |
| opendevreview | Clark Boylan proposed opendev/system-config master: DNM force etherpad failure to hold node https://review.opendev.org/c/opendev/system-config/+/840972 | 20:39 |
| clarkb | I'm going to rotate the held node now | 20:39 |
| clarkb | I don't expect that to be working yet but should be close in terms of shape | 20:40 |
| clarkb | fungi: should I pull the proposal bot discussion off the meeting agenda or do you think we've got more to discuss there? | 20:54 |
| fungi | it can come off. we provided direct feedback on the proposed change | 20:54 |
| clarkb | thanks | 20:58 |
| clarkb | ok just made edits to the meeting agenda. Anything else we need to add? | 21:07 |
| prometheanfire | fungi: honestly, not really, I'm starting to think that using it from a venv is the only stable way to use gertty anymore | 22:02 |
| fungi | prometheanfire: well, that was from a venv | 22:22 |
| fungi | just with the latest sqlalchemy, and starting it with no configuration cache, so maybe it's hitting additional code paths? | 22:23 |
| fungi | er, starting it with no configuration *and no* cache | 22:23 |
| opendevreview | Clark Boylan proposed opendev/system-config master: DNM Update etherpad to future v2.0.3 https://review.opendev.org/c/opendev/system-config/+/914119 | 23:04 |
| opendevreview | Clark Boylan proposed opendev/system-config master: DNM force etherpad failure to hold node https://review.opendev.org/c/opendev/system-config/+/840972 | 23:04 |
| clarkb | after manually fixing the settings.json on the held node the process of getting a bearer token then authenticating with ti seems to work | 23:05 |
| clarkb | I suspect that this will pass now and we can start planning an etherpad upgrade as soon as etherpad has a release with this new functioanlity in it | 23:05 |
| clarkb | I'll rotate the hold now | 23:05 |
| clarkb | that passes testing now and meeting agenda is sent | 23:39 |
| clarkb | we'll need to decide how we want to document the easiest path for etherpad admin tasks but I think other than that we're ready to upgrade after some manaul testing of the service itself once 2.0.3 or newer shows up | 23:40 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!