Friday, 2022-05-13

*** rlandy|bbl is now known as rlandy|out00:18
*** Guest0 is now known as prometheanfire00:29
ianwnot sure if we discussed it here wrt. the fedora mirror; the admins did get back to us and told us for rsync not to use the "" and that is always in the US anyway00:48
ianwanyway the fb mirror seems stable enough, so i guess we'll just stay with that, but good info just in case00:50
*** dviroel|afk is now known as dviroel|out01:26
opendevreviewMerged opendev/system-config master: gitea-set-org-logos: openstack logo centered
ianwi think all the 9-stream images should have unbound log fix no03:20
ianw[root@centos-9-stream-ovh-bhs1-0029635673 log]# ls -lh unbound.log 03:28
ianw-rw-r--r--. 1 unbound root 541K May 13 03:28 unbound.log03:28
johnsomack, I will fire off my test03:38
johnsomianw wahoo. It seems to work:
*** ysandeep|rover|out is now known as ysandeep|rover04:50
opendevreviewDavid Ostrovsky proposed zuul/zuul-jobs master: Bump bazelisk version to 1.11.0
opendevreviewDavid Ostrovsky proposed zuul/zuul-jobs master: Bump bazelisk version to 1.11.0
opendevreviewSandeep Yadav proposed openstack/diskimage-builder master: [DNM] test patch
opendevreviewMerged openstack/diskimage-builder master: Adopted dkms element to work on Ubuntu Jammy and nvidia drivers
opendevreviewMerged openstack/diskimage-builder master: Add Jammy functesting to dib
*** jpena|off is now known as jpena07:02
opendevreviewAlbin Vass proposed zuul/zuul-jobs master: Re-enable tests for prepare-workspace-git
opendevreviewAlbin Vass proposed zuul/zuul-jobs master: mirror-workspace-git: urlencode src_dir
opendevreviewAlbin Vass proposed zuul/zuul-jobs master: Add jobs to test base roles with unique workspace scheme
*** jpodivin_ is now known as jpodivin07:15
*** ysandeep|rover is now known as ysandeep|rover|lunch07:40
*** lajoskatona_ is now known as lajoskatona07:42
*** ysandeep|rover|lunch is now known as ysandeep|rover08:49
*** rlandy|out is now known as rlandy10:09
*** marios is now known as marios|afk10:10
*** marios|afk is now known as marios10:45
*** dviroel|out is now known as dviroel11:00
opendevreviewAlbin Vass proposed zuul/zuul-jobs master: mirror-workspace-git: urlencode src_dir
opendevreviewAlbin Vass proposed zuul/zuul-jobs master: mirror-workspace-git: urlencode src_dir
opendevreviewAlbin Vass proposed zuul/zuul-jobs master: mirror-workspace-git: urlencode src_dir
opendevreviewJeremy Stanley proposed opendev/system-config master: DNM: Testing image builds for bazelisk 1.11.0
opendevreviewJeremy Stanley proposed opendev/system-config master: DNM: Testing image builds for bazelisk 1.11.0
dtantsurHey folks! If a patch modifies the set of post jobs, will the new or the old jobs be triggered by its merge?13:32
dtantsurIt seems that "the old" is the answer, which I find a bit surprising.13:33
fungidtantsur: it should be using the just-merged behavior14:27
fungihave an example where it didn't?14:28
fungicould be a bug, or maybe there's some other cause14:28
dtantsurI had an example, but I guess its gone already14:39
fungiwas it from more than a month ago?14:39
dtantsurfungi: now, today. when was merged.14:40
fungithen we should have logs from the post pipeline jobs which that triggered14:40
fungii can look closer once the openstack release meeting is over14:41
dtantsurthanks! it's not urgent, I just was surprised and wanted to check if it's a bug14:42
fricklerI can confirm that the centos-8 was still triggered instead of centos-9. will check zuul logs now
*** ykarel_ is now known as ykarel15:05
fricklerfungi: corvus: not sure I've collected all the relevant logs, but it looks like a race between the reconfiguration handling on zuul02 and the freezing of the post jobs on zuul01
fungioh! so the reconfig was handled by one scheduler but the post pipeline was being managed by the other?15:11
*** dviroel is now known as dviroel|lunch15:12
clarkbThe pipeline is supposed to know if the cached config is stale based on the zk monotonically increasing version numbers15:25
clarkbit checks that number to see if it is old/stale and updates if not. But maybe we're processing things out of order so that check ends up valid?15:26
clarkbfungi: I'm doing local updates and will reboot then I'm going to go grab the lock for debian reprepro so we can land
clarkbthe number of packages I have to pull and install is quite large. This might take longer than initially expected15:40
clarkbits like they updated gcc and recompiled everything15:40
fungisounds like my experience yesterday. i'd put off updates for far too long because i didn't want to have to find time to rebuild my terminal emulator presets since there was a major update to crt which switched around the font handling and geometry15:43
fungibecause i'd procrastinated, apt wanted to download >1.5gb of packages and needed a further ~1gb free on top of that to unpack them all15:44
fungimoral of this story: don't put off package updates, but especially not if you're using debian/sid (because the package churn for it is insane)15:45
*** ysandeep|rover is now known as ysandeep|out15:55
*** dviroel|lunch is now known as dviroel16:00
*** marios is now known as marios|out16:06
clarkbI've grabbed the debian repo mirror lock on mirror-update in a screen. Tired of waiting on local package updates16:32
clarkbfungi: any reason to not approve that change now?16:32
funginope, approving now16:32
*** gibi is now known as gibi_pto16:39
clarkbOne thing we should do when these cleanups are complete (and debian is the last one?) is reset our afs volume quotas to be more inline with current utilization16:44
fungialso i tested gerrit image builds with a depends-on to davido's ensure-bazelisk update for zuul-jobs, and i see build failures for 3.5 and master (but 3.4 works with it, looks like)16:45
fungii need to dig into the actual errors though, as they may be unrelated16:45
clarkbfungi: looks like the jvm crashed16:49
clarkbthats exciting16:49
clarkb PerfMemory::alloc(unsigned long)+0x59 I wonder if it ran out of memory16:53
*** jpena is now known as jpena|off16:59
fungithe log for the 3.4 build does indicate it installed and used the new bazelisk, so it's a valid test at least17:03
fungiBazelisk version: v1.11.017:03
fungiin both the working 3.4 and failing 3.5 builds17:03
fungii'm going to recheck it to see if the 3.5 failure is persistent17:03
opendevreviewMerged opendev/system-config master: Stop mirroring source packages for debian
mordredwhen I was doing those builds originally - weird memory failure errors were the biggest struggle I had17:05
fungiyeah, that's why i'm going to recheck a few times17:05
mordredit was one of the reasons I eventually gave up on having a multi-stage dockerfile and just built it on the vm and copied the war into the container17:06
fungientirely possible we'll see 3.5 succeed and 3.4 crash next17:06
mordredwon't that be exciting?17:06
fungiit'll certainly be... something17:06
fungiit might be a clue that i should knock off early for some friday beverages17:07
fungionly time will tell17:07
clarkbI'm going to pause midday and go on a bike ride before the big rain storms arrive17:21
clarkbthe weather here continues to think it is march and not may17:24
clarkbit is getting old, but have to take advantage of the lack of rain as much as possible17:24
clarkbI'm going to run clearvanished, deleteunreferenced, then a normal reprepro run against hte debian mirrors now (the deploy is just about done)17:27
clarkbfungi: hrm we may still be setting the deb-src on debian nodes. I don't know why17:44
clarkbfungi: should I ^C the reprepro update that would likely break all the debian jobs as a result?17:44
clarkbbecause I'm a derp17:48
clarkbI think we can just quickly alnd a fix for this sintead17:48
fungiwhere are we still setting it? dib?17:49
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Fix the disabling of deb src repos in configure-mirrors
clarkbfungi: ^ no I hopped on a running job and checked and it was there I think that is the fix I was a derop17:50
clarkbanyway I did a ^C whcih stopped the script after it checked reprepro mirror integrity so no publish has been made yet17:51
clarkbwe shouldn't be broken yet, but landing ^ and then checking another host doesn't add deb-src then running reprepro would be good17:51
clarkbonce that change lands we can trigger a job that runs against debian then hop on the server and check what /etc/apt/sources.list.d/defaults.list looks like17:53
clarkband then we can rerun the reprepro command in the screen. Good to know that a single ^C seems to be safe. Reprepro even reorts that giving it a second ^C will kill it but at the risk of db corruption. Otherwise it seems to wait until it can exit cleanly (but with non zero exit code)17:57
fungiyeah, love robust programs which actually trap sigint with useful behaviors17:58
fungitrue to form, random jvm allocation crash did not occur on the 3.5 build after rechecking (nor on 3.4)18:02
fungimaster build still failed, but i haven't looked into why18:02
fungithe master builds seem to be unrelated persistent breakage18:07
opendevreviewMerged zuul/zuul-jobs master: Fix the disabling of deb src repos in configure-mirrors
fungiplugins/its-base/src/main/java/com/googlesource/gerrit/plugins/its/base/util/ error: package org.apache.commons.lang does not exist18:07
fungii'll recheck again to see if the jvm crash is frequent18:08
clarkbfungi: I rechecked which should get us bullseye and buster nodes to check live for deb-src18:08
clarkbfungi: ya looks good on bullseye18:09
clarkbonly deb bullseye main is in there and the mlt01 mirror indicates this isn't our default from the image build18:10
fungihowever, we should probably wait a bit before proceeding with the mirror cleanup in case there are jobs started from an earlier config state which might try to pull indices18:10
clarkbgood point18:10
fungithough the mirror cleanup also won't take effect globally until we vos release the volume18:10
clarkbya though thats what the next step is on the mirror (running our normal reprepro update script which does a vos release)18:11
clarkbI think we can wait on that a bit if we want. THough I was planning on stepping out in about an hour for that bike ride. I can update the mirror afterwards18:11
clarkbbuster looks good too18:12
clarkbI've run a nodepool list and filtered for debian nodes so that I can run it again later and see if any of the old nodes still exist18:13
clarkbif they do we can wait, if they don't we can proceed18:13
clarkbok all of the debian nodes from before the fix appear to have rotated out. I'm going to run reprepro now20:58
clarkbit has completed and I have released the flock21:16
clarkbthat saved us another 121GB of disk21:18
fungipretty significant!21:45
fungithat was including the security mirror?21:46
*** dviroel is now known as dviroel|out21:57
clarkbno I think security is separate and never had source22:14
clarkbalso sorry I got totally nerd sniped by my xmonad config. Turns out I had a bunch of deprecated stuff that I could replace with some simpler config22:14
clarkbturns out I can still read just enough haskell to be dangerous :)22:21
clarkbalso confirmed that tumbleweed updated its compiler to gcc 12.1 and recompiled a bunch of stuff whcih explains my giant update today22:30
opendevreviewMerged zuul/zuul-jobs master: Bump bazelisk version to 1.11.0

Generated by 2.17.3 by Marius Gedminas - find it at!