Wednesday, 2025-02-05

opendevreviewJames E. Blair proposed opendev/zuul-providers master: Add more flavors  https://review.opendev.org/c/opendev/zuul-providers/+/94075600:23
corvusclarkb: fungi ^ i'm aiming for two birds with one stone there00:23
clarkb+2 from me. My only other thought is bullseye is showing its age now we may want to work on bookworm and/or noble nodes next to do testing on forward looking stuff00:28
corvusoh heh yep.  zuul runs tests on jammy(!) now..  not sure if they'll run on bullseye, but if not, then that probably increases the priority for either bookworm or noble00:32
corvusoh 3.12 runs on noble00:32
corvusso we'd need at least one of those to pull over the 3.12 job00:32
*** ralonsoh_ is now known as ralonsoh08:10
*** ralonsoh_ is now known as ralonsoh10:31
opendevreviewMerged opendev/zuul-providers master: Add more flavors  https://review.opendev.org/c/opendev/zuul-providers/+/94075613:16
*** ralonsoh_ is now known as ralonsoh15:15
clarkbfungi: I think several mirrors need updating from focal to noble which implies we need openafs packages for noble. Did the new openafs ppa package update get build for noble? I think we may have one noble mirror somewhere too (rax flex?)15:49
clarkbit occurred to me ^ is somethign to consider in prep for a sprint replacement of servers next week15:50
fungiclarkb: https://launchpad.net/~openstack-ci-core/+archive/ubuntu/openafs says we built it for noble on 2024-11-1915:52
clarkbcool I wonder if the version is new neough to be used. I'll try to follow up on that15:53
clarkbthe other thing I'm trying to sort out is keycloak since I want to update its db soon I realized I should login to be able to test it is happy after doing so. Then realized I believe we redeployed it and started over and I need to create a new account15:53
clarkbunfortunately our docs don't seem to indicate how that is done. Any one remember?15:53
clarkbhttps://docs.opendev.org/opendev/system-config/latest/keycloak.html maybe the broken ref in that document is what i need to find15:54
clarkbyes that seems to have what I need. Let me fix the ref before I forget15:55
fungiclarkb: this server is running on noble: https://mirror02.ca-ymq-1.vexxhost.opendev.org/15:56
fungiso seems to work fine15:56
opendevreviewClark Boylan proposed opendev/system-config master: Fix zuul-admins doc ref  https://review.opendev.org/c/opendev/system-config/+/94080415:57
clarkband dpkg reports the version there should be from our ppa cool15:57
clarkbthanks!15:57
fungii'll be heading out to an appointment in about 45 minutes, but don't expect to be gone more than an hour so should be back before 17:00 utc16:00
fungier, before 18:00 utc i mean16:00
clarkbfungi: now that pbr has released should we work on a bindep release of at least the beginning portion of the pyproject.toml stack?16:08
fungiyeah probably, hoping we'll get some people to weigh in with preferences on the individual changes in that series16:08
fungii'm happy to restack them in a different order and work out the rebase conflicts if there's some consensus that one or more of the later changes in the stack is desired but its parents are not16:13
clarkbI think I'm happy with them as is. In particular I don't mind having a small tool do what pypa wants us to do even if it deviates from our own preferences. It seems like a good way to evaluate our ability to be more in line with pypa16:14
opendevreviewMerged opendev/system-config master: Fix zuul-admins doc ref  https://review.opendev.org/c/opendev/system-config/+/94080416:21
clarkbthe docs worked great by the way. So once ^ is deployed should be easier to find16:30
opendevreviewJeremy Stanley proposed opendev/bindep master: Stop overriding SetupTools module scanning  https://review.opendev.org/c/opendev/bindep/+/94081216:36
fungislight alteration to the schedule, i've been told i'm going out for lunch after the appointment, so will more likely return closer to 19:00 utc16:39
clarkback16:40
clarkbnow that I have a working keycloak account I'd like to land https://review.opendev.org/c/opendev/system-config/+/940348 to update the db location for keycloak. I do note that the mariadb image updated recently and we mirrored it (oddly the image age is 3 months according to docker image list but keycloak's image id matches the image id for etherpad and keyclock gets it from docker16:43
clarkbhub and etherpad from our quay mirror so everything is matching)16:43
clarkbis anyone willing to review that change? It doesn't have any reviews yet16:43
clarkband you may want to check the image update situatuion I noted above16:43
fungiyeah, that's sorta odd16:44
fungianyway, lgtm, +216:44
fungii'm headed out now, back in a couple of hours16:44
clarkbhttps://hub.docker.com/layers/library/mariadb/10.11/images/sha256-3595c1017b57939d4b5c5bc79f5d4822e607d6408f220c2e50a488caacf0c40a it did update there yesterday16:45
clarkbso nto sure why the client tooling reports it as older16:45
clarkbclock wrong on the build host maybe?16:45
clarkbor an old image that finally got updated?16:45
opendevreviewMerged opendev/system-config master: Switch keycloak to opendevmirror hosted mariadb image  https://review.opendev.org/c/opendev/system-config/+/94034817:20
clarkbkeycloak db updated then the deploy failed in a handler for restarting services. I haven't logged out and in aggain but the service seems to be up. I'm going to debug the handler first17:26
opendevreviewClark Boylan proposed opendev/system-config master: Fix keycloak container restart handler  https://review.opendev.org/c/opendev/system-config/+/94081917:30
clarkbits a simple issue and I don't think it is a problem in this case because the db did "update". I'll work on testing functiaonlity next17:31
clarkbI am able to login with my account so seems happy17:33
clarkblooking at the bindep stack again I think the "safe" set that I expect to be less contentious are 816741 938520 938568 940219 and 940812 (if this laste one works without the others)18:04
clarkbfungi: ^ maybe we want to restack with the requirements file changes last though I'm not sure if that would require additional shuffling of metadata18:04
opendevreviewClark Boylan proposed opendev/system-config master: Add AI openstack working group  list to lists.openinfra.org  https://review.opendev.org/c/opendev/system-config/+/94082118:14
clarkbinfra-root if you're happy with new grafana server can you let me know on https://review.opendev.org/c/opendev/system-config/+/940752 ? Don't necessarily need a full review on that just an ack that there don't appear to be problems with the new service before I start tearing down the old one18:21
clarkbI have a doctor appointment from ~1630-1830 tomorrow. Just a heads up I'll be out during that time. 18:33
clarkbcorvus: I was wrong we do run manage-projects out of the gerrit container image18:44
clarkbbut we do it via docker run in a single use container instance not exec'ing the actual service container18:45
clarkbI wonder if maybe we should hav a jeepyb image for commands like that. We install to the gerrit container because the gerrit hooks need to be local (or bind mounted I suppose but having that work with content from another container image seems annoying)18:45
clarkbanyway the jeepyb change failed on rate limits18:46
clarkbwe can recheck if we're happy with the status quo or brainstorm furhter on alternative approaches though I'm not coming up with anything great along those lines18:46
fungiclarkb: i was planning to try merging 940812 into one of the earlier changes anyway, just wanted to see if it passed all the current jobs first18:54
fungithe gerrit hooks run scripts from jeepyb, which needs it to be installed to the python search path18:57
fungibecause those scripts import modules from jeepyb18:59
clarkbya we need jeepyb accessible to gerrit hooks and installing them into thei mage is still probably the easiest way to do that19:03
clarkbin this case we're trying to udpate a non hook script so I was trying to determine if we had to rebuild a gerrit image (we do) and whether or not that is always appropriate (its probably simplest for now)19:03
corvuswe don't update this a lot... if we did, then separating it out might be worthwhile.  but for occasional updates, probably not. i  think we can probably just acknowledge that we'll get the new behavior whenever we next docker pull on that machine19:09
opendevreviewClark Boylan proposed opendev/system-config master: Update to gitea 1.23.2  https://review.opendev.org/c/opendev/system-config/+/94082319:13
clarkbcorvus: thats a good point too19:14
clarkbgitea's new release is hot off the presses ^19:14
corvuslooking at packges.debian.org and ubuntu.com -- it looks like python3.11 is in bookworm,sid,jammy.  and 3.12 is in trixie,sid,noble,oracular19:23
corvusso to use debian nodes, zuul would need bookworm and trixie; to use ubunte nodes it would need jammy and noble (what it currently uses)19:23
fungii concur.19:24
funginote that 3.12 will likely disappear from trixie and sid very soon19:24
corvusi wonder if i should add bookworm and trixie jobs/images, or add jammy/noble images.  we've talked about maybe using debian for a while now, but maybe that's too many moving parts....19:24
corvusah19:24
fungitrixie is expected to release with only 3.1319:25
corvusokay i think i should probably just do the ubuntu nodes then19:25
corvus"just use debian" is not easier in this particular circumstance; but might still be interesting later19:25
fungi3.12 in trixie (and hence sid) was only a stop-gap while 3.13 hadn't made it into the distro yet, but work there is focused on porting the remaining packages to 3.13 so that 3.12 can be removed19:26
fungibasically the python package maintainers in debian don't want to be stuck maintaining more than one python version in stable19:27
fungi3.12-3.13 is just taking a bit of time to complete mainly because of the stdlib cleanup in 3.13 ("removing dead batteries")19:27
opendevreviewJeremy Stanley proposed opendev/bindep master: Evacuate most metadata out of setup.cfg  https://review.opendev.org/c/opendev/bindep/+/93852019:30
opendevreviewJeremy Stanley proposed opendev/bindep master: Drop support for Python 3.6  https://review.opendev.org/c/opendev/bindep/+/93856819:30
opendevreviewJeremy Stanley proposed opendev/bindep master: Drop requirements.txt  https://review.opendev.org/c/opendev/bindep/+/93857019:30
opendevreviewJeremy Stanley proposed opendev/bindep master: Drop tox.ini  https://review.opendev.org/c/opendev/bindep/+/94021919:30
opendevreviewJeremy Stanley proposed opendev/bindep master: Drop auxiliary requirements files  https://review.opendev.org/c/opendev/bindep/+/94071119:30
fungiclarkb: i've abandoned 940812 and combined it into 938520 now19:30
Clark[m]Cool I'm sorting out an early lunch then will re review 19:30
opendevreviewJames E. Blair proposed opendev/zuul-providers master: Add ubuntu jammy and noble images  https://review.opendev.org/c/opendev/zuul-providers/+/94082619:33
fungiwell, let's see if it passes first, it's possible that has to shuffle to after the python 3.6 support removal19:34
opendevreviewJames E. Blair proposed opendev/zuul-providers master: Add ubuntu jammy and noble images  https://review.opendev.org/c/opendev/zuul-providers/+/94082619:36
opendevreviewJames E. Blair proposed opendev/zuul-providers master: Run ubuntu jammy and noble build jobs  https://review.opendev.org/c/opendev/zuul-providers/+/94082719:36
corvusClark: fungi when you have a second, can you review 940826 -- that needs to merge before 827 can run the check jobs (which will take a while, so i'd like to get that started)19:38
Clark[m]Will do19:40
fungilgtm, approved19:41
corvusthx!19:41
fungiit was pretty quick to review by just comparing to the already existing entries19:41
corvusyeah pretty mechanical19:41
opendevreviewMerged opendev/zuul-providers master: Add ubuntu jammy and noble images  https://review.opendev.org/c/opendev/zuul-providers/+/94082619:45
opendevreviewMerged opendev/system-config master: Fix keycloak container restart handler  https://review.opendev.org/c/opendev/system-config/+/94081919:45
clarkbI've approved the change to remove grafana01 from the inventory20:04
jrosseri think this means that i cannot use sudo in a tox job https://opendev.org/zuul/zuul-jobs/src/branch/master/playbooks/tox/run.yaml#L320:05
jrosserhave i understood that correctly?20:06
clarkbjrosser: you can't use sudo if using the default tox job playbook. That is correct. This is there bceause we learned very early on that developers will hide root escalations inside of unittests20:06
fungigenerally yes, that's there specifically to keep accidental uses of sudo from creeping into tox envs and leading to devs needing to run things locally with root access20:06
clarkbyou can run tox a different way if that is particularly problematic (just keep in mind if you expect people to run this stuff on a laptop that may be surprising to those people)20:07
fungiwithout it, nova unit tests would have required access to destroy your workstation20:07
jrosseri have ended up very deep down a debugging rabbit hole, where a workaround is some apparmor manipulation20:08
clarkbyou could fix apparmor before invoking tox20:08
fungiput the manipulation in a pre-run phase playbook20:08
clarkb(but then other people invoking tox would also need to know to do that)20:08
fungibut yes, it does imply complexity for anyone running that locally20:08
fungithough at least it means that when they do that, which needs them to do it with root permissions, they'll be aware of it20:09
clarkbthe basic tox jobs run a setup script20:09
jrosseryes i was wanting to keep this least-messy for local running too20:09
clarkbthat occurs before revoking permissions iirc and some projects use it to set up databases for example. That might be a good appraoch then people have the option to run the same script locally20:10
fungisince "apparmor manipulation" means you're making permanent changes to the apparmor configuration of some dev's workstation20:10
clarkbtools/test-setup.sh20:10
clarkbnova has an example of this20:10
fungiagreed, we tried to standardize on sticking things that need root/system permanent (or semi-permanent) additions/changes into a tools/test-setup.sh so it's discoverable20:11
fungiswift for example uses it (or used to use it) to mount a xfs filesystem for performing attribute tests20:12
fungiyeah, looks like they still do: https://opendev.org/openstack/swift/src/branch/master/tools/test-setup.sh#L7-L1420:13
jrossermy setup is with molecule20:18
jrosserso this is tox -> molecule -> ansible (tests/prepare.yml)20:19
jrosserideally i would not want to be doing anything to apparmor at all20:20
jrosserbut at the moment i cannot find the root cause of the error i get20:20
clarkbis it platform specific? could try running elsewhee (like update jammy to noble or use debian or something)20:23
jrosserit is sort of platform specific, when running a centos9 systemd docker container via molecule you can't ssh to it, because of `unix_chkpwd[578]: could not obtain user info (root)` on the host and a corresponding apparmor denied log message20:24
jrosserthe exact same thing switching out the container image for rocky/debian/focal/noble is fine20:25
jrosserbut the root cause of the centos one causing the apparmor error, i can't pin down20:26
clarkbthat implies the problem is in the host side apparmor (containers don't run kernel protections anyway because no kernel? and centos uses selinux). But I'm wondering if a differetn apparmor setup say replacing jammy with noble or vice versa fixes it20:26
fungiyeah, i was about to ask if it could be apparmor fighting with selinux20:26
jrossera very brute force `aa-teardown` on the host seems to let it proceed20:27
opendevreviewMerged opendev/jeepyb master: Create a stub .zuul.yaml  https://review.opendev.org/c/opendev/jeepyb/+/94054720:28
jrosseri could try switching the node type from noble to jammy20:28
clarkbre 940547 we should plan to restart gerrit and udpate the gerrit mariadb. Maybe tomorrow after my doctor appointment?20:30
clarkbinfra-root ^ fyi that way we aren't surprised by gerrit image updates later20:30
fungii'm on a 15:30-16:30 utc call tomorrow, but available to help with a gerrit restart outside that time20:32
clarkbya I need to leave for the appointment around 16:30 and be back around 18:30 as an estimate20:32
opendevreviewMerged opendev/system-config master: Add AI openstack working group  list to lists.openinfra.org  https://review.opendev.org/c/opendev/system-config/+/94082120:37
clarkbfungi: the bindep stack lgtm after reorganizing it20:46
clarkband I think we can probably proceed with the first three if there isn't any other input. THose seem pretty safe20:48
clarkbthe change to add that mailing list is going to run manage-projects (I dno't know where the overlap is for thsoe file matches) which should use the new gerrit image. Just fyi. I don't expect any problems as we aren't creating a new repo (and we test creating repos in ci for jeepyb too)20:54
clarkbya the job completed and reported success21:01
clarkbhttps://zuul.opendev.org/t/openstack/build/a04032a1d7c34fa7a73ff4beac74251f21:01
fungigood deal21:01
fungiclarkb: regarding the bindep stack, should 940219 get shuffled higher up? i couldn't tell if we were keeping tox.ini around for any particular reason or whether it was merely missed cleanup from the switch to nox21:05
clarkbfungi: I think it was "missing" cleanup. Initially tox.ini files were kept to make the transition easier as well as rollback but I think we're past that. I would move it up myself since I think it is safe to proceed with at this point21:06
fungiwill do, thanks21:06
corvusthe two image build jobs are running on ovh; the jammy one had a 20-minute pause while fetching .deb archives from the mirror; the noble one similarly had a 30-minute pause doing the same thing.  the times overlapped, but did not start or end exactly at the same time.  (20:28-20:47 in one instance, and 20:31-21:06 in the other)21:09
corvusi don't know if that's normal, or exceptional.  but probably worth keeping an eye on and looking into if it happens again?21:10
clarkbthat does seem excessive. I wonder if that implies our afs caches aren't being kept warm21:10
corvusgood thought; currently we use a smaller set of mirrors for our image builds; but this will spread it out a bit more21:11
clarkbits possible that if we start running these regularly the caches would stay warmer and runtime would improve. Definitely worth watching21:12
corvus++21:12
opendevreviewJeremy Stanley proposed opendev/bindep master: Drop tox.ini  https://review.opendev.org/c/opendev/bindep/+/94021921:13
opendevreviewJeremy Stanley proposed opendev/bindep master: Use PBR's pyproject.toml build-backend support  https://review.opendev.org/c/opendev/bindep/+/81674121:13
opendevreviewJeremy Stanley proposed opendev/bindep master: Evacuate most metadata out of setup.cfg  https://review.opendev.org/c/opendev/bindep/+/93852021:13
opendevreviewJeremy Stanley proposed opendev/bindep master: Drop support for Python 3.6  https://review.opendev.org/c/opendev/bindep/+/93856821:13
opendevreviewJeremy Stanley proposed opendev/bindep master: Drop requirements.txt  https://review.opendev.org/c/opendev/bindep/+/93857021:13
opendevreviewJeremy Stanley proposed opendev/bindep master: Drop auxiliary requirements files  https://review.opendev.org/c/opendev/bindep/+/94071121:13
clarkbidea: because those are pacakges we bake into our normal images the test jobs never need to fetch them so they are always cold21:13
clarkbbasically this set of packages is disjoint from the set of packages that test nodes would normally fetch21:13
corvusyeah. though i would normally expect that to manifest more like "runs slow for a while" rather than "stops for 30m" -- unless we're getting buffered output or something21:14
corvus(if we are seeing buffered output, then "runs slow for a while" would look exactly like "stops for 30m"21:14
clarkboh ya I was assuming buffered output or at least "all the apt stuff took 30 minutes"21:15
clarkbif its hard stopping that seems extra weird21:15
corvusi guess the question is whether dib buffers output from the apt command in whatever element is doing that... there's a couple of layers involved21:16
clarkbya I think dib is a python command so dib could be buffering in python. There is an env var to have it not do that we could try setting if it persists21:17
corvusbut even if we hypothesize it's buffered -- there are some other timestamps involved that look like we got buffer flushes every few minutes; so if that's the case, then we're looking at maybe 2m between buffer flushes then 30m, so still weird.21:17
opendevreviewMerged opendev/system-config master: Remove grafana01 from config management  https://review.opendev.org/c/opendev/system-config/+/94075221:38
corvusclarkb: fungi https://review.opendev.org/940827 passed if you want to approve it; we'll get new images then21:39
clarkbcorvus: +2 form me22:00
clarkbnot sure if fungi is still around may be worth approving as is?22:01
clarkbdoes anyone want to remove the grafana01 dns cleanups now that grafana01 is no longer in inventory: https://review.opendev.org/c/opendev/zone-opendev.org/+/94075322:01
fungiclarkb: if you mean 940827 i approved it 5-6 minutes ago22:01
clarkbfungi: ya sorry I should've refereshed22:03
opendevreviewMerged opendev/zone-opendev.org master: Cleanup grafana01 DNS records  https://review.opendev.org/c/opendev/zone-opendev.org/+/94075322:04
clarkbwith ^ done I cleaned up the emergency file. paste01 was still in there too22:26
clarkbI'll plan to deelte grafana01 probably Friday /me makes a note on the todo list for that22:26
clarkbthe gitea 1.23.2 screenshots look good to me: http://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_234/940823/1/check/system-config-run-gitea/234284d/bridge99.opendev.org/screenshots/22:28
clarkbI went ahead and approved the tox.ini removal change for bindep. That one shouldn't be controversial since we switched voer to nox some time ago22:50
clarkbI'll leave the others in plce for more feedback for now22:50
clarkbfungi: looking at your TODO on https://review.opendev.org/c/opendev/bindep/+/816741/24/setup.py PBR is acting as a build hook for setuptools already. I think the piece that is missing is pbr needs to appent the pbr=True flag to how it invokes setuptools?22:51
clarkbright now its passing through to setuptools which eventually calls setup.py which is why setting pbr=True there works22:52
fungisomething like that maybe? i'm unclear on what the missing piece is to be able to drop setup.py entirely22:52
clarkbI wonder if we can pass arbitrary setup() flags in setup.cfg or pyproject.toml to work around that22:52
clarkbfungi: its basically needing to get pbr=True into setup() or have pbr detect it is being used some other way I think22:52
clarkbwith pyproject.toml the flow now is pbr hook -> setuptools hook -> setup.py -> setup()22:53
clarkbI don't think we can easily get rid of pbr hook -> setuptools hook as pbr itself assumes it is running within/alongside setuptools. But if we can influence the config somehow to enable pbr without the argument to setup() then I think we can do it22:53
clarkbhttps://opendev.org/openstack/pbr/src/branch/master/pbr/core.py#L64-L154 that is why we set pbr=True in setup()22:55
clarkbwhich is tied into setuptools/distutils via https://opendev.org/openstack/pbr/src/branch/master/setup.cfg#L41-L4222:55
clarkbso basically we need to invoke that method somehow I think22:56
clarkbfungi: maybe you can do [options]\npbr=True in setup.cfg?22:58
opendevreviewMerged opendev/bindep master: Drop tox.ini  https://review.opendev.org/c/opendev/bindep/+/94021922:59
clarkbI'll push that up and see what happens23:00
fungiyeah, if we can figure out how to inject it from pyproject.toml, bonus23:00
fungialso if we can figure out how to drop the name field from setup.cfg (so that we can drop setup.cfg), extra bonus23:01
fungifeels a little unfinished that we still need both stub setup.py and setup.cfg files23:01
opendevreviewClark Boylan proposed opendev/bindep master: Test pbr=True set via setup.cfg  https://review.opendev.org/c/opendev/bindep/+/94084123:02
clarkbthe pyproject.toml stuff that is documented seems a lot more rigid23:02
clarkbhttps://setuptools.pypa.io/en/latest/userguide/pyproject_config.html and options doesn't seem to be an option23:02
opendevreviewClark Boylan proposed opendev/bindep master: Test pbr=True set via setup.cfg  https://review.opendev.org/c/opendev/bindep/+/94084123:06
clarkbfungi: ^ you'll like that. The errors I got seem to imply I need py_modules back23:06
clarkbmaybe if you define any options then that one becomes required?23:06
clarkbso that seems to install but we don't actually install bindep23:10
clarkbso ya that doesn't seem to work23:10
clarkboh well was worth a shot. I wonder if the problem is we're using a distutils hook and not setuptools and options is setuptools only23:11
clarkbwhen we go through setup() I mean it uses a distutils hook apparently23:11
opendevreviewMerged opendev/zuul-providers master: Run ubuntu jammy and noble build jobs  https://review.opendev.org/c/opendev/zuul-providers/+/94082723:38
corvushttps://imgur.com/a/AqLgCHL23:56
corvusthat's two images being built in the same buildset23:56
corvusfirst time we've seen that live; eventually the nightly builds will be one buildset with a bunch of image jobs.23:57

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!