| corvus | clarkb: do you grok https://zuul.opendev.org/t/openstack/build/e139bc90e10b4fa6aa571608bceb2396 | 00:13 |
|---|---|---|
| corvus | i'm not sure why we put xenial in there on our current focal hosts | 00:15 |
| corvus | oh i see it, nm | 00:16 |
| opendevreview | James E. Blair proposed opendev/system-config master: WIP: Test zuul on jammy https://review.opendev.org/c/opendev/system-config/+/883986 | 00:17 |
| *** amoralej_ is now known as amoralej | 09:16 | |
| frickler | infra-root: I'm seeing a lot of (zombie?) "git cat-file --batch-check" processes on our executors, is that a known issue? (first discovered this on our local zuul, which has more seldom restarts) | 10:29 |
| *** amoralej is now known as amoralej|lunch | 10:55 | |
| fungi | i've definitely seen them for... years | 11:39 |
| frickler | I'm not sure if that's meant to comfort me or rather not ... ;) | 11:40 |
| fungi | i don't think they're zombies, they're still parented to the executor process and seem to only be as old as the last exectutor container restart | 11:41 |
| frickler | yes, not zombies in the unix meaning of the word. can still be quite a lot for deployments that don't do automated regular restarts | 11:42 |
| fungi | maybe due to a bug in GitPython not cleaning up subprocesses it spawns? | 11:59 |
| fungi | https://github.com/gitpython-developers/GitPython/issues/1333 | 12:00 |
| fungi | seems similar | 12:00 |
| fungi | here's a workaround in another project using gitpython: https://github.com/SwissDataScienceCenter/renku-python/pull/2928 | 12:02 |
| SvenKieske | mhm, you can easily dos yourself with such behaviour depending how fast this process list grows - apparently not fast enough, yet. | 12:07 |
| fungi | probably part of why we don't end up with lots is that we perform rolling restarts of all the zuul services to update their containers at least once a week | 12:09 |
| fungi | we're averaging 170 git cat-file processes per executor at the moment. i'll check again in a bit to see if it increases | 12:16 |
| fungi | we're about 4 days since the last restarts | 12:16 |
| SvenKieske | I think it would be worthwhile nevertheless to fix these, how much ram do these consume? | 12:22 |
| *** amoralej|lunch is now known as amoralej | 12:33 | |
| fungi | hard to say in aggregate because they might share pages, but we track memory utilization on the executors and throttle builds automatically when it gets too high: https://grafana.opendev.org/d/21a6e53ea4/zuul-status?orgId=1&viewPanel=27 | 12:36 |
| fungi | the memory used by the git cat-file processes is a tiny fraction of what python/ansible consumes | 12:36 |
| fungi | and yes, it would be worthwhile to fix. probably the best way to do that is upstream in gitpython | 12:37 |
| fungi | they've even marked the issue about it as "help wanted" | 12:37 |
| fungi | if you look at the executory memory utilization graphs, they don't seem to get worse as the week goes on and approaches the next scheduled restart, so it doesn't look like those leftover processes are contributing significantly to memory utilization | 12:42 |
| fungi | also the average of 170 git cat-file processes i calculated across the executors was not to imply that all of those are left over, i'd probably need to filter them on start time or some similar heuristic to better measure that | 12:47 |
| SvenKieske | looking at the upstream bug I don't think they are aware that this happens on many more systems than just the ones reported | 12:52 |
| SvenKieske | I don't know really how GitPyhton interacts with zuul here, but the suggestion seems clear: use libgit2 via it's python bindings instead | 12:53 |
| SvenKieske | that's at least my take when reading https://github.com/gitpython-developers/GitPython/issues/1333#issuecomment-913078611 | 12:54 |
| fungi | zuul relies on gitpython in this particular case to attempt git merges and then extract (possibly speculative) configuration files from all the branches in them | 12:54 |
| fungi | and to construct the speculative repository states which are exposed to the build workspaces and replicated to the job nodes assigned to a given build | 12:55 |
| fungi | anyway, this is not the zuul project's discussion channel, we just happen to be running a deployment of zuul. the #zuul:matrix.org channel or zuul-discuss@lists.zuul-ci.org mailing list are where potential redesigns like should be brought up | 12:57 |
| SvenKieske | well, Byron is the primary author of GitPython and afaik is currently full time working on gitoxide, the rust reimplementation of git, so I wouldn't bet on many upstream work. he also suggested to use libgit2 instead | 12:57 |
| fungi | er, #zuul:opendev.org matrix channel i meant | 12:58 |
| SvenKieske | re: zuul dev; sure | 12:58 |
| SvenKieske | didn't know they are on matrix, good to know | 12:58 |
| SvenKieske | to end this discussion: is anyone aware of a bug tracking this on the zuul side? | 12:59 |
| fungi | yeah, opendev has a dedicated matrix homeserver and zuul's discussion channel is hosted there | 12:59 |
| fungi | SvenKieske: i am not aware of one, no, but since it's a bug in a library used by zuul it's sort of a grey area as to whether it could be considered a zuul bug | 13:00 |
| fungi | but if the situation really is "gitpython is abandoned by its author who suggests everyone should switch to libgit2" then maybe that's worth bringing up | 13:01 |
| SvenKieske | yeah, and given that a fix doesn't seem to materialize, maybe they/someone can implement a workaround in zuul, besides restarting :) | 13:03 |
| fungi | huh, i can't find libgit2 on pypi | 13:04 |
| fungi | aha, it's pygit2 | 13:04 |
| fungi | i assume most distros have libgit2 by now, but it's relatively recent in zuul years (seems it only landed in debian as far back as bullseye) | 13:08 |
| fungi | that would probably explain why it was designed around gitpython instead of pygit2 bindings | 13:09 |
| clarkb | I think licensing was an issue in the way back times too | 15:16 |
| fungi | looks like it's currently "GPLv2 with a Linking Exception" | 15:22 |
| fungi | while gitpython is bsd | 15:22 |
| clarkb | ya it was just GPL before iirc and the last time it came up zuul was part of openstack and that was something to be avoided | 15:23 |
| clarkb | I don't thnik it would be an issue today | 15:23 |
| fungi | it's possible https://github.com/libgit2/pygit2/blob/master/COPYING#L10-L19 also addresses apache/gpl2 compatibility issues, but i'd want a lawyer to confirm that | 15:26 |
| clarkb | fungi: well the linking exception is what fixes it now | 15:28 |
| clarkb | I want to say that exception didn't exist before | 15:28 |
| clarkb | the alternative in the past was to relicense zuul to something gplv2 compat | 15:29 |
| clarkb | corvus: fungi frickler https://etherpad.opendev.org/p/opendev-quay-migration-revert I'm starting to put a plan together there. Once I've got the list of steps outlined I'll start filling in specifics and we can move forward | 15:29 |
| fungi | right, i mean i'm not 100% confident in my reading of the exception clause | 15:29 |
| clarkb | ah | 15:29 |
| fungi | and would prefer openinfra's legal counsel weigh in on it before we said sure it's compatible with zuul's license | 15:30 |
| fungi | but, again, the zuul matrix channel or zuul mailing list is the place for the discussion | 15:31 |
| clarkb | ++ | 15:31 |
| fungi | openssl security advisory coming on tuesday. highest severity ranked "moderate" by their security folks though | 15:33 |
| clarkb | if any infra-root have time to skim over https://etherpad.opendev.org/p/opendev-quay-migration-revert to make sure that looks sane then give the sync back to docker.io script paste a close look for accuracy that would be appreciated. I'm at a point where I'd like to get that done as well as the announcement before taking further steps (just to avoid getting ahead of myself) | 15:44 |
| clarkb | and while I wait for that I will do local software updates | 15:44 |
| corvus | fwiw, we evaluated using libgit2 in zuul, and there are significant performance penalties. we can use it for some things, but for others, we would just need to shell out to a git process. | 16:03 |
| dmendiza[m] | Hi friends! I need some help debugging a Centos 9 devstack gate. Looks like things are failing because liberasurecode-devel can't be found: https://zuul.opendev.org/t/openstack/build/79df356099344440b83e2e13d183a073/log/job-output.txt#3238-3239 | 16:09 |
| dmendiza[m] | Seems to be a swift requirement: https://opendev.org/openstack/devstack/src/commit/34afa91fc9f830fc8e1fdc4d76e7aa6d4248eaaa/files/rpms/swift#L2 | 16:09 |
| fungi | dmendiza[m]: yep, any idea where it comes from in centos stream 9? | 16:10 |
| fungi | epel? rdo? | 16:11 |
| fungi | there's a proposed change to switch the rdo version used: https://review.opendev.org/883790 | 16:12 |
| fungi | calls out ovs versions as the reason, but i wonder if ec is in a similar situation | 16:12 |
| dmendiza[m] | looks like the repo is delorean-master-testing ... at least in a local devstack vm I spun up a couple of days ago | 16:13 |
| fungi | er, i guess that change i linked is for /SIGs | 16:14 |
| fungi | there's a code comment in devstack that says "Some necessary packages are in openstack repo, for example liberasurecode-devel" | 16:15 |
| fungi | right before it does `install_package hostname openstack-release-wallaby` | 16:16 |
| fungi | oh, that's for openEuler-22.03 though not centos | 16:16 |
| fungi | so i'm guessing it normally gets handled by the _install_rdo function | 16:17 |
| fungi | and for centos stream 9 it grabs http://trunk.rdoproject.org/centos9-master/delorean-deps.repo and updates from that | 16:18 |
| fungi | dmendiza[m]: are you sure older runs didn't have a similar error? it looks like that was nonfatal and the job died a few seconds later because it couldn't find pip: https://zuul.opendev.org/t/openstack/build/79df356099344440b83e2e13d183a073/log/controller/logs/devstacklog.txt#928 | 16:24 |
| fungi | 2023-05-23 12:49:14.139 | /usr/bin/python3.9: No module named pip | 16:25 |
| dmendiza[m] | fungi: so, what I think is happening, is that the dnf install dies because it can't find the one package, which results with python3-pip not being installed | 16:25 |
| fungi | aha, okay that would make sense | 16:26 |
| fungi | so in that case the main question i have is whether it's trying to download that package from our centos stream 9 mirror and not finding it, or trying to install it from elsewhere | 16:27 |
| fungi | dmendiza[m]: we have it here in our mirrors: https://mirror.bhs1.ovh.opendev.org/centos-stream/SIGs/9-stream/cloud/x86_64/openstack-zed/Packages/l/ | 16:30 |
| dmendiza[m] | hmm... 🤔 ... only thing I can think of is maybe a signature check failure because it's only happening on our FIPS enabled gate | 16:31 |
| * dmendiza[m] tries to test that theory | 16:31 | |
| fungi | dmendiza[m]: also make sure it's referencing an rdo package collection which actually has packages. looks like yoga, zed and antelope have liberasurecode-devel | 16:33 |
| opendevreview | Eduardo Alberti proposed openstack/project-config master: Add Kubernetes Power Manager app to StarlingX https://review.opendev.org/c/openstack/project-config/+/884250 | 16:33 |
| fungi | dmendiza[m]: wallaby, xena and bobcat have no packages directory at all | 16:33 |
| dmendiza[m] | yeah, this is a patch to bobcat | 16:33 |
| fungi | presumably you'll want to use rdo antelope packages for now until rdo bobcat exists | 16:34 |
| fungi | i mean, it exists it's just empty (no packages directory yet) https://mirror.bhs1.ovh.opendev.org/centos-stream/SIGs/9-stream/cloud/x86_64/openstack-bobcat/ | 16:34 |
| fungi | or you might need to grab them from https://trunk.rdoproject.org/centos9-master/deps/latest/x86_64/ (i don't think we mirror that anywhere?) | 16:36 |
| clarkb | infra-root I did a test run of my sync script to cboylan/python-builder: https://hub.docker.com/r/cboylan/python-builder/tags spot checking against https://quay.io/repository/opendevorg/python-builder?tab=tags&tag=latest I think this looks good? both arm and x86 images made it across and hashes seem to match spot checking 3.10-bullseye and 3.11-bullseye | 16:36 |
| clarkb | please check that for sanity, but I'll plan to proceed with syncing from quay.io to docker.io in opendevorg soon | 16:37 |
| dmendiza[m] | fungi: yeah, looks like the package installs just fine without fips, but fails like so when fips is enabled: https://paste.opendev.org/show/bqIz1E7EmsmkiBR89EtG/ | 16:43 |
| fungi | neat | 16:45 |
| clarkb | https://bugs.launchpad.net/octavia/+bug/2020434 | 16:46 |
| clarkb | I think this is a distro bug? curl not working under fips isn't something we can fix for you | 16:47 |
| johnsom | clarkb I just saw this patch too: https://review.opendev.org/c/openstack/devstack/+/884247 | 16:48 |
| fungi | yep, having arrived at that, it's probably something to discuss in #openstack-qa or #rdo | 16:48 |
| johnsom | It seems like that RDO host has old ciphers/protocols maybe? | 16:48 |
| fungi | if any of you know someone at red hat, that's where i'd start | 16:49 |
| johnsom | fungi lol | 16:49 |
| dmendiza[m] | heh ... yeah looks like the TLS handshake fails under fips: https://paste.opendev.org/show/bFHRdcEXY5WSFx2ad1wG/ | 16:55 |
| clarkb | infra-root I manually checked `sudo docker run -it --rm docker.io/cboylan/python-builder:3.11-bullseye bash` on my local machine and on nb04 to double check that the arm and amd manifest stuff seems to be working and it appears to be fine | 16:56 |
| * dmendiza[m] takes his broken toy to #_oftc_#rdo:matrix.org | 16:58 | |
| fungi | dmendiza[m]: johnsom: back on topic for this channel though, if the delorean/rdo packages can be mirrored the same way we rsync normal centos packages, that would mask the problem (not solve it because users doing their own fips compliance testing with devstack would still hit it) but would cut down on devstack centos jobs connecting out across the wilds of the internet to install stuff | 16:59 |
| clarkb | fungi: I don't think that is true | 16:59 |
| clarkb | rdo/delorean are in a ton of flux at all times | 16:59 |
| clarkb | the delorean content in particular is basically development packages that get updated all the time randomly | 17:00 |
| fungi | yeah, if the way their package repositories are laid out isn't stable for mirroring, then it would probably be unworkable | 17:00 |
| fungi | or if they don't update packages and indices in the right order (avoiding removing old packages for a while after new versions are added) | 17:00 |
| fungi | granted, that would be a problem for direct consumers of those packages too, even without us mirroring them | 17:01 |
| clarkb | fungi: did you want to look at the container image snc stuff before I proceed? | 17:08 |
| fungi | i'm looking through it now, yep | 17:08 |
| fungi | i think it makes sense, i'm just going back over it to be sure | 17:09 |
| clarkb | thanks! I'll wait then | 17:09 |
| fungi | i haven't ever uploaded images to dockerhub or made tags there, so i'm taking for granted that part is correct | 17:10 |
| dmendiza[m] | I'm thinking this might be an OpenSSL issue and not the cert that RDO is using https://github.com/openssl/openssl/issues/19989 | 17:11 |
| clarkb | ack. I think if you cross check the content between quay.io/opendevorg/python-builder and docker.io/cboylan/python-builder that is probably good enough | 17:11 |
| clarkb | fungi: I used the script with a small modification of the new side namespace to cboylan in order to do that | 17:12 |
| fungi | clarkb: corrected a minor typo on the announcement | 17:12 |
| fungi | dmendiza[m]: so the server is using tls 1.2 and expecting ems negotiation but openssl hasn't enabled it in fips mode? | 17:15 |
| dmendiza[m] | I think turning on FIPS in the client requires EMS, but for some reason it is not enabled ... at least that's what I think this error is trying to tell us: | 17:17 |
| dmendiza[m] | error:1C8000E9:Provider routines:kdf_tls1_prf_derive:ems not enabled | 17:17 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Revert "Move python builder/base images to quay.io" https://review.opendev.org/c/opendev/system-config/+/884256 | 17:18 |
| fungi | clarkb: the todo list makes references to "step 0" but the list is unnumbered. is that referring to the first bullet? | 17:18 |
| clarkb | fungi: yes | 17:18 |
| fungi | k, thx | 17:19 |
| fungi | plan lgtm, i'll review that first change | 17:19 |
| fungi | i take it that's a straight up revert of the original, hence going back to relative image names | 17:21 |
| clarkb | fungi: I changed the rebuild string in the Dockerfiles but otherwise it is a straight revert | 17:22 |
| clarkb | (I didn't want confusing timetsamps that go backwards in those files) | 17:22 |
| clarkb | fungi: ok I'm going to push one more change then switch over to doing the sync step. We shouldn't land any of these until after that sync step completes | 17:23 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Revert "Move assets image to quay.io" https://review.opendev.org/c/opendev/system-config/+/884257 | 17:23 |
| clarkb | alright now to get logged in for the sync step and sort through that. | 17:24 |
| clarkb | I'm going to do just a few at a time. I started with assets because its content hasn't actually chagned so is safe. https://hub.docker.com/r/opendevorg/assets/tags and https://quay.io/repository/opendevorg/assets?tab=tags seem to be in aggrement on that 92f... value | 17:29 |
| clarkb | I'm also only logged into the docker side so I can't accidentally push the other direction | 17:31 |
| clarkb | ok all 14 images have been synced back. I've checked all the images (only spot checked tags on each image though) and they all seem to line up between docker and quay | 17:47 |
| clarkb | I'll wait for a +1 from zuul on the two changes I've pushed so far then I'll send the announcement email | 17:48 |
| clarkb | then we can hopefully approve those two and continue on to the others | 17:49 |
| fungi | the assets revert already came back +1 | 17:49 |
| fungi | the first change hasn't reported yet | 17:49 |
| clarkb | ya the assets build is pretty trivial it just copies a few files into a container image | 17:49 |
| clarkb | I'll wait for a +1 on the other one too | 17:49 |
| clarkb | corvus: do you want to review these change sbefore the get approved? | 17:50 |
| corvus | can do | 17:50 |
| corvus | lgtm; i can't think of a reason not to do it that way | 17:51 |
| corvus | on another subject, i was doing some digging into performance metrics for the nodepool zk persistent watches change, and i'm pretty sure i mentioned here already that you can clearly see the improvement in zk latency when we made the change (on april 11): https://grafana.opendev.org/d/21a6e53ea4/zuul-status?orgId=1&from=1680542750000&to=1681493150000 | 17:52 |
| corvus | but what i just noticed is that i think we can actually see an improvement in zuul performance too: https://grafana.opendev.org/d/2e89fb78e5/zuul-performance-metrics?orgId=1&var-tenant=openstack&from=1680541200000&to=1681578000000 | 17:52 |
| corvus | the pipeline refresh time for the periodic pipeline drops from 5.0 seconds to 3.6 seconds after that change; likely because zk has more bandwidth to handle the zuul requests. | 17:53 |
| clarkb | looking at the jobs for the base python images change I will need to resync accessbot, haproxy-statsd and zookeeper-statsd after this lands. Not a big deal and that is why I've got a note to check if anything needs to be resynced again | 17:55 |
| clarkb | corvus: is the periodic pipeline the most expensive in terms of time due to the thundering herd? | 17:59 |
| corvus | clarkb: yeah, i think that's what that spike is | 18:02 |
| clarkb | corvus: fungi: I take it your are both good with me sending the email and approving those two changes? | 18:07 |
| clarkb | zuul just reported a +1 back | 18:07 |
| fungi | affirmative | 18:07 |
| corvus | clarkb: yes | 18:07 |
| opendevreview | Eduardo Alberti proposed openstack/project-config master: Add Kubernetes Power Manager app to StarlingX https://review.opendev.org/c/openstack/project-config/+/884250 | 18:09 |
| clarkb | email sent. Approving the changes next. | 18:09 |
| corvus | clarkbfungi the "test zuul/nodepool" on jammy changes are +1 from zuul; what's next for upgrading those servers? land those changes, then delete, relaunch, and update dns one at a time? | 18:32 |
| fungi | that's probably safest. we could launch multiple replacements in parallel if we have the quota for it instead of deleting and replacing one by one | 18:34 |
| clarkb | in the past I've updated the testing at the same time that we update the inventory with the new nodes. But I don't think that is super important. Otherwise up basically that | 18:34 |
| corvus | in this case there's a 1:N where N is like 25 relationship with testing and real nodes, so exactly when to update is a grey area :) | 18:35 |
| corvus | so maybe i'll start by spinning up as many new mergers as possible, then swapping them out; then executors, then schedulers, then nodepool | 18:36 |
| fungi | sgtm, thanks! | 18:38 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Revert "Migrate statsd sidecar container images to quay.io" https://review.opendev.org/c/opendev/system-config/+/884274 | 18:40 |
| corvus | i'm launching 6 merger nodes | 18:53 |
| opendevreview | Merged opendev/system-config master: Revert "Move python builder/base images to quay.io" https://review.opendev.org/c/opendev/system-config/+/884256 | 18:55 |
| opendevreview | Merged opendev/system-config master: Revert "Move assets image to quay.io" https://review.opendev.org/c/opendev/system-config/+/884257 | 18:56 |
| clarkb | the promotion jobs for the base images have run in deploy successfully. I now need to resync haproxy-statsd, zookeeper-statsd, and accessbot. I'm going to do that now then I'm going to pop out for lunch | 18:59 |
| clarkb | the pushes to docker seem to have worked for base images too | 19:02 |
| clarkb | Resync for haproxy-statsd, zookeeper-statsd and accessbot is done | 19:08 |
| clarkb | I think that https://review.opendev.org/c/opendev/system-config/+/884274 is ready when we are assuming it passes CI and reviewers are happy with it. Definitely look closely at that one as I had to do a partial revert of the second change. cc fungi and corvus | 19:09 |
| clarkb | I'm going to pause here and find lunch. I'll continue to push up changes when I return | 19:09 |
| corvus | i'm going to shut down all of the trash compactors on the detention level | 20:05 |
| corvus | er, i mean, all of the zuul mergers | 20:05 |
| clarkb | 884274 is about to report success. I think it is good to merge if yall are happy with it | 20:14 |
| corvus | clarkb: lgtm | 20:15 |
| opendevreview | James E. Blair proposed opendev/zone-opendev.org master: Replace all zuul mergers https://review.opendev.org/c/opendev/zone-opendev.org/+/884283 | 20:17 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Revert "Move system-config irc bots into quay.io" https://review.opendev.org/c/opendev/system-config/+/884284 | 20:18 |
| opendevreview | James E. Blair proposed opendev/system-config master: Upgrade zuul mergers to jammy https://review.opendev.org/c/opendev/system-config/+/884285 | 20:18 |
| corvus | clarkb: fungi ^ those 2 changes should accomplish the jammy upgrade for zm*. i have shut all current mergers down (relying on executors for now), so should be safe to apply at will. | 20:18 |
| corvus | clarkb: and i included the jammy test upgrade for only the merger in that change :) | 20:19 |
| clarkb | corvus: replacing them in place like that may require you to clear out ansible caches. I can't remember if that was a problem the last time around | 20:20 |
| clarkb | since ansible will cache the wrong host facts | 20:20 |
| corvus | hrm, something weird happened with the tabs in the dns change... want me to fix that? | 20:20 |
| corvus | clarkb: ack. when do you think is the best time for that? | 20:20 |
| clarkb | corvus: we run zuul hourly so you'd probably have to put hosts in the emergency file to avoid that. Maybe better to see if it breaks when the change lands and if so then clear it out and let the hourly job correct things | 20:21 |
| clarkb | re tabs I haven't been too picky about it | 20:21 |
| corvus | wfm. just rm -fr /var/cache/ansible/facts/zm* | 20:21 |
| clarkb | I try to make things consistent when I make changes but otherwise its meh | 20:21 |
| clarkb | and ya I think that should do it | 20:21 |
| corvus | k. happy to fix the tabs if desired; just don't want to churn under you if no one cares. | 20:22 |
| clarkb | +2 on both changes. I didn't go and veryify every ip address and host key :) | 20:22 |
| clarkb | but the changes look like what I would expect them to look | 20:22 |
| opendevreview | Maksim Malchuk proposed openstack/diskimage-builder master: Extend the checksum files generation procedure https://review.opendev.org/c/openstack/diskimage-builder/+/881298 | 20:34 |
| opendevreview | Clark Boylan proposed opendev/gerritbot master: Revert "Move gerritbot to quay.io" https://review.opendev.org/c/opendev/gerritbot/+/884286 | 20:36 |
| opendevreview | Clark Boylan proposed opendev/statusbot master: Revert "Move statusbot to quay.io" https://review.opendev.org/c/opendev/statusbot/+/884287 | 20:39 |
| opendevreview | Maksim Malchuk proposed openstack/diskimage-builder master: Extend the checksum files generation procedure https://review.opendev.org/c/openstack/diskimage-builder/+/881298 | 20:43 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Revert "Move pull external IRC bot images from quay.io" https://review.opendev.org/c/opendev/system-config/+/884289 | 20:43 |
| opendevreview | Clark Boylan proposed opendev/grafyaml master: Revert "Migrate grafyaml container images to quay.io" https://review.opendev.org/c/opendev/grafyaml/+/884291 | 20:50 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Revert "Pull grafyaml from quay.io" https://review.opendev.org/c/opendev/system-config/+/884292 | 20:53 |
| dmendiza[m] | fungi re: FIPS jobs failing, turning on FIPS mode in the client requires that TLS 1.2 use EMS, but some repos have EMS disabled in openssl, so the negotiation fails. Tracking a workaround here: https://review.opendev.org/c/openstack/devstack/+/884277 | 20:53 |
| opendevreview | Clark Boylan proposed openstack/project-config master: Revert "Pull grafyaml from quay.io" https://review.opendev.org/c/openstack/project-config/+/884293 | 20:54 |
| opendevreview | Clark Boylan proposed opendev/lodgeit master: Revert "Move lodgeit image publication to quay.io" https://review.opendev.org/c/opendev/lodgeit/+/884296 | 21:00 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Revert "Pull lodgeit from quay.io" https://review.opendev.org/c/opendev/system-config/+/884298 | 21:02 |
| clarkb | ok all of the required changes for this rollback should be pushed now and recorded in https://etherpad.opendev.org/p/opendev-quay-migration-revert I think I managed to set the proper tag on all of them too | 21:03 |
| clarkb | I would like to approve them one by one because sometimes there are unexpected side effects that cause an image you don't expect to update to update | 21:03 |
| clarkb | and doing them one by one allows us to check and resync to docker hub if necessary | 21:03 |
| opendevreview | Merged opendev/zone-opendev.org master: Replace all zuul mergers https://review.opendev.org/c/opendev/zone-opendev.org/+/884283 | 21:04 |
| clarkb | https://review.opendev.org/c/opendev/system-config/+/884247 is the next change to land and should be safe to do so. fungi I'll give you a bit of time to review that one since I ended up doing a partial revert and would like eyeballs on it if possible | 21:04 |
| clarkb | and then its just a matter of landing the changes, verifying results and moving on to the next one | 21:05 |
| fungi | stepped away to make/eat dinner but checking merger changes now | 21:05 |
| fungi | already merged! good deal | 21:05 |
| fungi | oh, dns changed already merged, inventory change already gating but lgtm | 21:06 |
| fungi | i'll review the rest of the quay migration reverts in that case | 21:07 |
| clarkb | oops I swapped two numbers in the next up change number it should be https://review.opendev.org/c/opendev/system-config/+/884274 | 21:18 |
| clarkb | and fungi has +2'd it so I will approve it now | 21:18 |
| clarkb | corvus: if you have time for https://review.opendev.org/c/opendev/system-config/+/884284 that is the next one up. I don't expect I'll get much beyond these two today just with the time involved to test, merge, deploy | 21:23 |
| corvus | lgtm | 21:31 |
| opendevreview | Merged opendev/system-config master: Upgrade zuul mergers to jammy https://review.opendev.org/c/opendev/system-config/+/884285 | 22:10 |
| corvus | things have landed, dns is updated, and we're between zuul service playbook runs, so i'm going to go ahead and clear the fact cache for zm* | 22:40 |
| corvus | hopefully next run at approx 23:11 should deploy | 22:41 |
| opendevreview | Merged opendev/system-config master: Revert "Migrate statsd sidecar container images to quay.io" https://review.opendev.org/c/opendev/system-config/+/884274 | 22:42 |
| clarkb | corvus: its is running now | 23:42 |
| corvus | looks like it finished | 23:50 |
| clarkb | corvus: I don't think it auto started the services though but ya seems to hvae deployed successfully | 23:50 |
| corvus | i'll start up zm01 and watch it | 23:50 |
| corvus | it's up...and idle :) | 23:51 |
| clarkb | the statsd containers promoted successfully. They should roll out soon enough | 23:51 |
| clarkb | I'll continue the revert work tomorrow | 23:51 |
| corvus | clarkb: before you head out; do you reckon i should delete the old mergers now? | 23:52 |
| corvus | is the best way to do that to open an openstack client shell on bridge and delete from there? | 23:52 |
| corvus | (zm01 merged something, i will start the others) | 23:53 |
| clarkb | corvus: in the past I've left things disabled on the old host for a short time just in case a revert is necessary. That seems less likely for mergers though | 23:54 |
| corvus | yeah, we can live without them, so i'll just remove them now | 23:54 |
| clarkb | corvus: and ya `openstack server list | grep zm01.opendev.org` then find the uuid for the old one using `openstack server show $uuid` then `openstack server delete $uuid` | 23:54 |
| clarkb | you can't delete by name since there are duplicates | 23:55 |
| corvus | good point; maybe we should put the uuid in inventory files to make this easier | 23:56 |
| clarkb | I tend to do a server show on whatever I'm going to delete first anyway like doing a select before a delete. Just makes me feel better :) | 23:58 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!