Friday, 2023-06-23

Unit193I guess if that's linked, might be useful too?00:45
NeilHanlonyep, thanks Unit193!01:00
fricklermnasiadka: ubuntu-ports should have been up2date again since about 0UTC06:19
fricklerstarting afs volume additions now06:19
fricklerhumm, seems our OSC version doesn't like to create volumes on rackspace: VolumeManager.create() got an unexpected keyword argument 'backup_id'06:23
fricklergotta check some old venvs06:23
fricklero.k., what I have in ~/venv does a better job06:24
fricklerthere's some inconsistency in our setup, the volumes are called afs0? but the servers are still afs0?.dfw.openstack.org06:30
fricklernot sure if renaming the volumes would be a good idea. I did rename one of the new volumes which I incidentally created with a wrong name first, so it should work fine technically06:31
frickler"pvs" on afs02 says for all pv: WARNING: PV /dev/xvdh1 in VG main is using an old PV header, modify the VG to update.06:33
fricklerguess that should clean up once I actually update the vg, we'll see. I'll still start with afs0106:34
mnasiadkafrickler: thanks06:38
frickler/dev/mapper/main-vicepa  5.0T  3.6T  1.4T  72% /vicepa06:47
fricklerI'll wait a bit just in case before I move to afs0206:48
fricklerafter doing the vgextend on afs02, the warning about old headers is gone as expected07:12
fricklero.k., done, everything lgtm, just grafana seem to need some time to pick up the new used% data07:19
frickleris anyone else seeing gerrit slowness? might be my local connection only07:25
frickler#status log added another 1T volume for /vicepa on afs01/02.dfw to increase AFS capacity07:38
opendevstatusfrickler: finished logging07:38
fricklerI've prepared a list of quota adjustments at, those would in preparation of mirroring bookworm use up 590 of the 1024 newly added Gs08:13
opendevreviewCaio Cesar Ferreira proposed openstack/project-config master: Add Power Metrics app to StarlingX
fricklerclarkb: fungi: please have a look at the quota adjustments and then the bookwork mirror patch if we merge that soonish I'd start the initial run in a screen with log held and watch it over the weekend14:08
fricklerbookworm even14:08
frickler would also be nice, although likely we have a couple of months now again14:12
frickleralso it seems we've lost track of, was there any reason not to merge it? guess I'll just approve it ...14:16
opendevreviewMerged openstack/project-config master: wheel builds : move to individual releases
fungifrickler: proposed adjustments lgtm, though any idea why the centos volume needs so much more space than centos-stream? makes me suspect we missed cleaning something up14:37
fungiaha, i guess we still have 7 mirrored14:38
fungiand we still have centos-7 nodes14:39
fricklerso more cleanup potential ahead, yay14:39
fricklerI'll go ahead with the quota adjustsments, then14:40
fungithanks for working on all that!14:41
fungiwe can also reduce quotas in some of those volumes later if we clean up enough content14:44
fungithe quotas are mainly serving as a backstop so that something going haywire on some particular mirror can't fill up the disk and impact other unrelated volumes14:44
frickleryes, will recheck in particular once we know how much space the bookwork mirrors really need15:05
frickler#status log updated quota for multiple AFS mirror.* volumes to adapt to current demand15:06
opendevstatusfrickler: finished logging15:06
clarkbfrickler: thank you for taking care of that16:22
clarkbtonyb: I approved the insecure ci registry test fix. Thank you for digging into that and apologies it took me so long to review it16:31
clarkbfungi: frickler: next up for insecure ci registry is launching a new jammy node and updating to add it into the inventory then also pushing dns updates. Any interest in that? I think corvus, ianw and I have launched nodes recently and want to make sure this opportunity doesn't slip by if you want ot make sure you're still16:33
clarkbfamiliar with it16:33
opendevreviewMerged opendev/system-config master: Update testing for insecure-ci-registry
fungii have to run some pre-travel errands in about 30 minutes, but should be able to knock that out when i get back17:00
fungifrom the errand-running17:00
frickleryeah, pretty sure there'll be other opportunities for node launches. also still waiting for reviews on after I almost decided to self-approve this morning17:18
fungii approved it just now after double checking the official mirrors, but we'll want to check the reprepro logs from the next pass after that deploys17:28
fungiactually, we'll probably want to hold the lock for the debian cronjob on mirror-update.o.o and run the first pass manually in screen with timeouts turned off17:34
fungii'll go ahead and grab the lock in a screen session17:35
fungiinfra-root: i did `flock -n /var/run/reprepro/debian.lock bash` in a root screen session on mirror-update.o.o just now17:36
clarkbdid you make sure you grabbed it? I think it doesn't give much indication if it fails since you are already typicalyl running bash17:36
fungiyeah, there was no session running at the time17:36
fungino cronjob running for it, i mean17:37
fungianyway, heading out to run errands now, but can help with the initial mirror run for that in an hour or so17:37
opendevreviewMerged opendev/system-config master: reprepro: add Debian Bookworm config
fungii'm back, starting the mirror run since the deploy succeeded a few minutes ago19:11
fungiand it's going now19:12
fungiit's on so going to be quite a while, i think19:48
clarkbnot very zoom zoom20:03
clarkbI'm launching a new ci registry momentarily20:07
opendevreviewClark Boylan proposed opendev/system-config master: Deploy on jammy
opendevreviewClark Boylan proposed opendev/ master: Add DNS records for insecure-ci-registry02
opendevreviewClark Boylan proposed opendev/system-config master: Deploy on jammy
opendevreviewClark Boylan proposed opendev/ master: Remove insecure-ci-registry01 and update CNAME to 02
clarkbtonyb: ^ fyi something like that to finish up this process20:23
opendevreviewMerged opendev/ master: Add DNS records for insecure-ci-registry02
fungidebian main mirroring is finally past l/lib...22:07
corvusi've been seeing multiple post failures on nodepool jobs during the cleanup phase22:09
corvuser, sorry, zuul and nodepool upload-image jobs22:10
corvuslooks like every "-image" job is hitting it22:11
Clark[m]In cleanup? I didn't think that was supposed to be able to break jobs22:12
corvusoh yep that's a red herring22:14
corvusit's a timeout on push to intermediate registry22:14
Clark[m]Looking at the log it ya that22:14
corvusi saw some intermediate registry chat earlier -- anything happening with thatL22:15
Clark[m]I want to say when we've seen this before we have had to restart the daemon for the registry because cherry py gets sad22:15
Clark[m]No just DNS for a new server but the new server hasn't been added to inventory yet22:15
Clark[m]It shouldn't affect anything yet22:15
fungiwe haven't replaced the registry server yet, no22:15
funginew dns records were added, the cname is still untouched22:15
corvuscool, i'll check logs and (probably) restart22:16
fungithe inventory addition is in the gate22:16
fungicname change will be ready to approve once that deploys22:16
corvusssl.SSLError: [SSL: NO_SUITABLE_KEY_SHARE] no suitable key share (_ssl.c:1007)22:16
corvus2023-06-23 17:15:06,189 ERROR cherrypy.error: [23/Jun/2023:17:15:06] ENGINE Error in HTTPServer.serve22:16
corvusyeah i think that's one of the "the ssl subsystem in cherrypy is hung" errors22:17
corvusconnection attempts were hanging, so i kicked it. better now22:18
opendevreviewMerged opendev/system-config master: Deploy on jammy

Generated by 2.17.3 by Marius Gedminas - find it at!