| opendevreview | Merged opendev/system-config master: Update to latest tag for Limnoria https://review.opendev.org/c/opendev/system-config/+/888110 | 00:58 |
|---|---|---|
| fungi | keeping an eye on that ^ deploy to make sure the bot reconnects | 01:00 |
| fungi | welcome back, opendevmeet! | 01:02 |
| frickler | fungi: ianw: I renamed the two .conf files on meetpad01, iiuc it will take until tomorrow before the next run happens | 09:38 |
| *** amoralej is now known as amoralej|lunch | 11:01 | |
| *** amoralej|lunch is now known as amoralej | 12:07 | |
| fungi | thanks frickler! i guess we can check in on it again tomorrow and see if the problem has recurred or it successfully updated | 12:21 |
| amorin | hey all, is the location for openinfra summit 2024 already known? Did I miss the notification for it? | 14:33 |
| *** d34dh0r5- is now known as d34dh0r53 | 15:01 | |
| fungi | amorin: not to my knowledge. i think the plan is to discuss it at the board of directors meeting on tuesday: http://board.openinfra.dev/meetings/2023-08-01 | 15:18 |
| amorin | ack, will wait then :) | 15:19 |
| amorin | thanks | 15:19 |
| fungi | any time! | 15:20 |
| ildikov | Hi, I have a quick question. I know that there were discussions around moving OpenDev container images over from DockerHub to Quay. Have you all looked into other options as well before more seriously considering Quay? Did you have a list of criteria that you used for evaluation? | 15:45 |
| ildikov | Background: the StarlingX community has been having issues with DockerHub recently, so the idea to switch came up. Nexus Repository and Quay came up as options, but people don't have a lot of experience, especially with the latter. | 15:47 |
| fungi | ildikov: i don't think we looked very hard at other options besides quay or hosting our own registry. the main reason we rolled back off quay was because the docker client has baked-in fallback policies which favor dockerhub in ways which make our current test model (relying on jobs that can depend on other proposed changes to container images) is impossible with the docker client, and | 16:15 |
| fungi | switching to other clients is something we're still working on | 16:16 |
| fungi | another community worth looking at is scs, they started hosting their own image registry for their distribution | 16:16 |
| fungi | in unrelated news, just saw this on login.ubuntu.com: "We are currently in read-only mode. That means that at the moment you can't create new accounts, update your personal information, or request password reminders." | 16:16 |
| fungi | heads up, in case anyone runs into problems | 16:17 |
| ildikov | thanks fungi, it's great info! | 16:35 |
| fungi | ildikov: maybe also worth noting, zuul moved their container image hosting to quay, but they don't need the same degree of dependent testing (for opendev's case it's more related to replicating production deployments of preview images to mock servers and interacting with them in subsequent jobs/changes) | 16:37 |
| ildikov | that makes sense | 16:37 |
| ildikov | fungi: Zuul is using Quay as a service and it's not a self-hosted platform, right? | 16:38 |
| fungi | correct, quay.io specificially | 16:39 |
| fungi | https://quay.io/organization/zuul-ci | 16:39 |
| fungi | i think what scs is using to host their own registry is software called "harbor" according to ttps://sovereigncloudstack.github.io/website/tech/2023/05/30/registry-migration-upgrade/ | 16:41 |
| fungi | unfortunately, https://registry.scs.community/ goes straight to a login so it's hard to tell for sure | 16:42 |
| ildikov | Oh, cool, I’ll try to dig into that | 17:25 |
| ildikov | Thanks! | 17:25 |
| fungi | we're about 90 minutes out from around when we planned to do the gerrit restart. the opendevorg/gerrit:3.7 tag seems to indicate a successful upload from the change merged yesterday: https://hub.docker.com/layers/opendevorg/gerrit/3.7/images/sha256-b413945441f6baf2fa7ffaed22b5926c615363a18e85f3cf795463c4273ce003?context=explore | 19:37 |
| fungi | i've done a docker-compose pull and image list shows the updated images, so i think we're all set. down the service, recursively delete /home/gerrit2/review_site/data/replication/ref-updates, then up everything again | 19:41 |
| Clark[m] | I think we may just need to delete the waiting/ dir under that path? If you ls | wc -l you can find the big dir | 19:44 |
| fungi | ah, okay | 19:44 |
| Clark[m] | And moving it aside and deleting later may be much quicker? | 19:44 |
| fungi | also i'll do a status notice about an hour ahead (15 minutes form now) | 19:44 |
| fungi | can do | 19:44 |
| fungi | presumably gerrit will recreate that directory | 19:45 |
| Clark[m] | Yes it should since before the bind mount it was starting without the dir every time | 19:45 |
| Clark[m] | clarification: the delete won't be any faster but we can perform it after Gerrit is restarted if we mv rather than rm while Gerrit is down | 19:46 |
| clarkb | ildikov: the alternative registry options we considered were github's registry or quay for hosted versions and for self hosted gitea and zuul registry were considered. We prefered hosted for simplicity reasons (we don't have to deal with storage and bw and pruning) | 19:57 |
| clarkb | quay is open source and the hosted version is free for open source projects so that was a straightforward choice (with the acknowledgement that we could end up where docker had initially put us in the future) | 19:58 |
| fungi | #status notice The Gerrit service on review.opendev.org will be offline briefly for a minor upgrade at 21:00 utc, approximately an hour from now | 20:01 |
| opendevstatus | fungi: sending notice | 20:01 |
| -opendevstatus- NOTICE: The Gerrit service on review.opendev.org will be offline briefly for a minor upgrade at 21:00 utc, approximately an hour from now | 20:01 | |
| opendevstatus | fungi: finished sending notice | 20:04 |
| ildikov | clarkb: +1, thank you for sharing some more details! | 20:05 |
| clarkb | fungi: there are "only" 29k entries in the waiting/ dir and ya that is the dir | 20:06 |
| clarkb | I expected more | 20:06 |
| clarkb | fungi: and if we mv waiting to /home/gerrit2/tmp/$dir/waiting we'll stay on the same fs and a mv should be almost free | 20:06 |
| fungi | right, will definitely save some time | 20:08 |
| clarkb | 90cc4cec43ba is the current gerrit image | 20:37 |
| fungi | opendevorg/gerrit <none> 90cc4cec43ba 2 months ago 790MB | 20:44 |
| fungi | and we're restarting on | 20:45 |
| fungi | opendevorg/gerrit 3.7 3a2e576abc94 23 hours ago 793MB | 20:45 |
| fungi | #status notice The Gerrit service on review.opendev.org will be offline briefly for a minor upgrade, but should return shortly | 21:00 |
| opendevstatus | fungi: sending notice | 21:00 |
| -opendevstatus- NOTICE: The Gerrit service on review.opendev.org will be offline briefly for a minor upgrade, but should return shortly | 21:00 | |
| fungi | i'll start momentarily | 21:00 |
| clarkb | cool I can follow along | 21:00 |
| opendevstatus | fungi: finished sending notice | 21:03 |
| fungi | i'll `mv /home/gerrit2/review_site/data/replication/ref-updates/waiting /home/gerrit2/tmp/upgrade-3.7.4/` once the containers are down | 21:03 |
| fungi | taking the containers down now | 21:03 |
| fungi | moving the replication queue | 21:03 |
| fungi | upping containers now | 21:04 |
| fungi | webui is loading for me now and says "Powered by Gerrit Code Review (3.7.4-10-g56c0b3d9ce-dirty)" | 21:05 |
| clarkb | I see that too | 21:05 |
| clarkb | and the waiting dir was recreated as expected | 21:05 |
| fungi | current /home/gerrit2/review_site/data/replication/ref-updates/waiting is there and empty, yep | 21:05 |
| clarkb | pushing a change/patchset if anyone has one ready would be a good adidtional test but this lgtm | 21:07 |
| fungi | i've got some changes i need to fix, i'll try to do one real quick | 21:07 |
| fungi | i've self-approved some trivial changes instead, that'll still get us replication events once they merge | 21:12 |
| fungi | also looks like this is a good time to approve 888111 | 21:13 |
| clarkb | ++ | 21:14 |
| fungi | 886641 in the gate is just waiting for a node to run a docs build | 21:15 |
| fungi | though it's been waiting 6 minutes for that node already | 21:15 |
| fungi | i probably need to update the nodeset for the ossa repo's jobs | 21:16 |
| fungi | lots of stuff that got approved after it is already running jobs | 21:16 |
| clarkb | it is running now | 21:19 |
| fungi | yeah, and that's from a standard openstack job template, so using ubuntu-jammy | 21:19 |
| fungi | just got unlucky with boot failures i guess | 21:19 |
| fungi | at least it's wrapping up and uploading logs already | 21:20 |
| clarkb | https://review.opendev.org/c/starlingx/monitoring/+/887282 this one just merged | 21:20 |
| fungi | https://opendev.org/openstack/ossa/commits/branch/master reflects it | 21:21 |
| fungi | so replication is working | 21:21 |
| clarkb | the starlingx/monitoring commit replicated as well | 21:22 |
| fungi | perfect, i'm going to go do some very exciting yardwork while i wait for the latest meetbot image update to go | 21:22 |
| clarkb | sound sgood. Thank you for pushing this along | 21:23 |
| clarkb | I too will resume morning activities | 21:23 |
| opendevreview | Merged openstack/project-config master: Switch from deprecated require-approval to require https://review.opendev.org/c/openstack/project-config/+/883431 | 21:23 |
| opendevreview | Merged opendev/system-config master: Update ircbot and matrix-eavesdrop to python-3.11 https://review.opendev.org/c/opendev/system-config/+/888111 | 22:00 |
| opendevreview | Merged opendev/system-config master: Add $self to the list of files for python images. https://review.opendev.org/c/opendev/system-config/+/888663 | 22:04 |
| opendevreview | Merged opendev/system-config master: Update docker-images/python* to remove '.*' https://review.opendev.org/c/opendev/system-config/+/888664 | 22:04 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!