Wednesday, 2022-05-11

ianwfacebook is a lot of things, but "under-resourced infrastructure" probably isn't one of those things :)00:02
*** rlandy|bbl is now known as rlandy|out00:03
clarkbsometimes they break bgp and can't open the datacenter doors though00:04
fungior they break the data center doors and then can't fix bgp00:10
fungiand when it's both... that's called a thursday00:11
*** artom_ is now known as artom00:47
Clark[m] "The log was not found. It may have been deleted based on retention settings." Guess I'm not debugging it then. Also every other job was cancelled so I don't get any logs from them either?01:10
Clark[m]It's like we need to help GitHub with their CI stuff01:11
Clark[m]ianw: the change was merged anyway. If you have time can you check the run we get in a few hours? Not sure if that is past the end of your day01:12
ianwClark[m]: oh, excellent, yes i'll check on it01:19
ianwi'm pretty sure we've got better ci of many upstreams than they do of themselves :)01:19
*** ysandeep|out is now known as ysandeep|rover01:21
ianwoh although i'm not sure we run dev01:28
ianwoh, we do ...01:30
opendevreviewMerged opendev/system-config master: mirror-update: switch Fedora mirror
*** ysandeep|rover is now known as ysandeep|rover|bbl02:08
ianwwe could probably cull i686 packages too, fedora seems to pull them in03:09
*** bhagyashris|out is now known as bhagyashris03:58
*** ysandeep|rover|bbl is now known as ysandeep|rover04:23
ianwi can see that ~ 2022-05-11T03:22:24Z updated everywhere to bee5cb55a133905c49794a4962fdb1b16b9c92f9 ; which is clarkb's change + something trivial04:43
ianwwe then reissued codesearch, vexxhost mirror, limestone mirror, zuul01, zuul0204:47
ianw$ openssl s_client -servername -connect 2>/dev/null | openssl x509 -noout -dates04:50
ianwnotBefore=May 11 02:31:33 2022 GMT04:50
ianwnotAfter=Aug  9 02:31:32 2022 GMT04:50
ianwwhich seemed weird, but then04:50
*** soniya29 is now known as soniya29|ruck04:51
ianwle certs are backdated by an hour to avoid clock skew issues.  which is logical, but TIL.  so the timestamps line up04:51
ianwergo, clarkb fixed it :)04:51
fricklerI tested running reprepro to mirror 22.04 on a 22.04 host and I get the same errors as on 20.04, so currently that won't help us. updated accordingly05:26
fricklerI also checked that our mirrors are still stuck at 10% phased updates, while upstream has long finished all phasing for current 22.04 updates05:27
opendevreviewMerged opendev/system-config master: mirror-update: clean up Fedora ignores
ianwfrickler: thanks, that's interesting and a bit annoying.  is that error different to the compresssion issue though?06:33
*** ysandeep|rover is now known as ysandeep|rover|brb07:29
*** soniya is now known as soniya|ruck07:40
*** ysandeep|rover|brb is now known as ysandeep|rover07:40
fricklerianw: the error on 22.04 looks exactly the same as on 20.04: zstd: error 70 : Write error : cannot write decoded block : Broken pipe07:49
*** soniya|ruck is now known as soniya|ruck|dinner07:51
*** soniya|ruck|dinner is now known as soniya|ruck|lunch07:53
opendevreviewAlbin Vass proposed zuul/zuul-jobs master: mirror-workspace-git: urlencode src_dir
opendevreviewAlbin Vass proposed zuul/zuul-jobs master: Re-enable tests for prepare-workspace-git
*** ysandeep|rover is now known as ysandeep|rover|lunch08:01
*** soniya|ruck|lunch is now known as soniya|ruck08:56
*** ysandeep|rover|lunch is now known as ysandeep|rover09:02
*** jpena|off is now known as jpena09:39
*** bhagyashris_ is now known as bhagyashris11:03
*** rlandy|out is now known as rlandy11:21
*** dviroel|out is now known as dviroel11:31
*** ysandeep|rover is now known as ysandeep|rover|brb11:32
fungiianw: frickler: seems to indicate that the error is benign, but also suggests that the solution may be as simple as installing zstd?11:32
fungithough i think people discussing there may be rerunning reprepro, not seeing the error (because no new affected packages were synced that time) and assuming it was fixed11:34
fungihard to know11:34
*** ysandeep|rover|brb is now known as ysandeep|rover11:37
*** artom_ is now known as artom11:42
*** ysandeep|rover is now known as ysandeep|rover|mtg11:55
fungior that they're conflating multiple issues11:55
fricklerzstd is installed on my jammy vm, seems it is preinstalled in the ubuntu cloud image, also reprepro depends on it. but interesting to see that ionos seems to have done their own fork. but understandable seeing the timeline11:55
* frickler fights the urge to dig into reprepro sources11:57
*** ysandeep|rover|mtg is now known as ysandeep|rover12:37
*** ysandeep|rover is now known as ysandeep|rover|brb13:21
*** ysandeep|rover|brb is now known as ysandeep|rover13:31
*** bhagyashris_ is now known as bhagyashris13:33
clarkbyay for having certs refreshed14:53
clarkbI have a dentist visit in a couple of hours so will be popping out after my morning meeting14:53
clarkbfungi: frickler: ya I agree zstd seemed to be installed14:53
*** soniya|ruck is now known as soniya|out15:05
*** dviroel is now known as dviroel|lunch15:09
johnsomHi neighbors, I am seeing "Could not resolve host:" on at least one job. I think I saw some recent chatter on this issue. Does it still need research or is it a known issue? The unbound log is empty.15:21
*** ysandeep|rover is now known as ysandeep|rover|out15:26
fungijohnsom: i saw someone point out a similar name resolution error on a build earlier, though it was in a different provider so i guess it's not provider-specific networking issues (or it's impacting multiple providers). normally the nodes should have unbound installed and running as recursive resolvers pointing at a couple of different dns resolver ip addresses on the internet (opendns and15:30
fungigoogle dns maybe?), and then resolv.conf configured to use the loopback address for lookups15:30
fungithis is in order to perform lookup caching locally and reduce the number of dns queries each node sends15:31
johnsomYeah, unbound is there, but the log is empty and I don't see the conf for it. Basically I will dig a bit deeper on this if it's not a "known" issue. I just didn't want to duplicate effort.15:31
fungiright. i also compared to a working build earlier and the unbound log was empty then too15:32
johnsomOk, let me dig into this and see if I can find why the log is empty in these runs. That at least is a starting point.15:33
fungiagreed, i don't know if that means we've broken unbound logging/log collection, or that we're not going through unbound for queries so it has nothing to log15:35
johnsomIt normally has some startup noise. It looks like it's just centos jobs that the log is empty (checked centos 9 stream) so far. The ubuntu jobs are populated.15:36
clarkbyou should always always link to specific job logs15:36
clarkbthere are so many things that can affect dns resolution like running in a conatiner15:37
clarkbthat seems to fail because dnf didn't find a rabbitmq package? Where is the dns lookup failure?15:39
clarkbok so that is happening on centos-9 on our base image install without cnotainers being involved15:41
clarkbinteresting that that error wasn't fatal15:42
clarkbboth ns1 and ns2 currently resolve that name so we aren't stuck in some sort of bad authoritative server problem15:42
clarkbthe other hunch is that we should check if centos-9 is honoring our unbound resolver or if it is doing its own thing because centos 9 is new15:43
clarkbthe empty unbound log that you point out is suspicous15:44
*** marios is now known as marios|out15:44
fungisystemd has its own bus-based dns resolver implementation, right? maybe it's going through that instead and ignoring resolv.conf completely15:44
johnsomSo far, ubuntu and debian are all collecting the logs, but c9s and the fedora jobs have empty logs. I am poking around for a c8s job that still has logs to see if it's empty too. I suspect there is just a collection issue on the RH tribe 15:45
clarkbfungi: it is supposed to figure that out I thought15:46
clarkbbut ya its been a semi frequent fight to convince various distros to actually use our resolver and not the one the cloud attempts to use15:46
fricklerthese are fips jobs, maybe the reboot interferes with our unbound setup15:47
clarkb shows unbound is running at least15:48
johnsomYeah, c8s non-fips also has an empty unbound log. So, we have a log collection issue on that side.15:49
johnsomI don't mean to distract you all, I am fine digging into this15:50
clarkbits fine I'm going to disappear to the dentist soon :P15:50
clarkbthen I'll be very distracted15:50
johnsomlol, I will certainly take unbound over that15:51
fungi"better than a trip to the dentist"15:54
*** dviroel|lunch is now known as dviroel16:03
*** jpena is now known as jpena|off17:01
johnsomThis is probably why:
johnsomGiving this a go:
fungioh, makes sense18:26
fungigood catch!18:26
opendevreviewDenys Mishchenko proposed openstack/diskimage-builder master: Adopted dkms element to work on Ubuntu Jammy and nvidia drivers
opendevreviewDenys Mishchenko proposed openstack/diskimage-builder master: Adopted dkms element to work on Ubuntu Jammy and nvidia drivers
clarkbon zuul02 I see Le_Keylength='2048' which confirms my expectation that it would write that back out again and this is only a problem at the migration point19:53
clarkbnow that we are close to being able to remove debian src packages from our mirrors I'm realizing I should've written down the repo cleanup steps someplace other than eavesdrop's logs20:10
fungii thought ianw added them to the system-config docs... double-checking, i might have imagined reviewing that change20:11
fungiis that what you were looking for?20:13
clarkbah yup there they are20:14
clarkbianw: if you get a chance today is a straightforward system-config cleanup but good to have eyebalsl on that in case we end up unconfiguring stuff that is still alive20:52
*** dviroel is now known as dviroel|afk20:53
*** timburke__ is now known as timburke21:00
fungipip 22.1 was just tagged.
fungi(most of the changes appear under the 22.1b1 entry though because apparently towncrier isn't as smart as reno)21:22
*** rlandy is now known as rlandy|bbl22:08
ianwclarkb: will do22:13
clarkbfungi: I'm not seeing anything problematic in there but "Explains why specified version cannot be retrieved when Requires-Python is not satisfied." should be a nice addition22:15
*** mazzy5098812929580859494 is now known as mazzy50988129295808594922:22
ianw is another one of these in use ppa's i guess23:19
ianwiirc we use it for trusty ensure-pip?23:19
clarkboh xenial? Its beacuse the distro pip is pip 8 and and doesn't understand python-requires so we do a pip 9 so that it can upgrade to the latest version that undersatnds python 3.5 iirc23:20
ianwi'm guessing we can remove ?23:20
clarkbianw: I believe that is correct. We hit that with zuul iirc23:20
clarkband it did end up getting an SRU eventually23:21
ianwand afaics we don't use any more?23:24
ianw we dropped from the nodepool image, but now use the unstable version of debootstrap23:25
ianwthat was fairly recent though so i think leave that for the time being23:25
opendevreviewMerged opendev/system-config master: Remove group vars for removed services
clarkbya bwrap we switched to the distro version as soon as it supported user namespacing in the distro version iirc and we've been on distro versions for quite some time23:28
ianwi'll get rid of that to avoid confusion too23:28
opendevreviewIan Wienand proposed opendev/infra-vhd-util-deb master: Add README.rst about empty master branch
opendevreviewMerged opendev/infra-vhd-util-deb master: Add README.rst about empty master branch
ianwopenstack-zuul-jobs 840572 merged; so i'm going to push some changes to the distro branches now that will build and push openafs/vhd-util packages to the production repos23:33
opendevreviewIan Wienand proposed opendev/infra-vhd-util-deb focal: Trigger upstream build
clarkbcorvus: this isn't urgent but I'm selfishly wanting to update zuul(-web) to the version that will scroll to the linked log line. Is that something that would be reaosnable to do tomorrow?23:37
opendevreviewIan Wienand proposed opendev/infra-openafs-deb trusty: Reset changelog version
opendevreviewIan Wienand proposed opendev/infra-openafs-deb xenial: Add build stamp to push to production PPA
opendevreviewIan Wienand proposed opendev/infra-openafs-deb bionic: Add build stamp to push to production PPA
opendevreviewIan Wienand proposed opendev/infra-openafs-deb focal: Add build stamp to push to production PPA
opendevreviewIan Wienand proposed opendev/infra-openafs-deb jammy: Add build stamp to push to production PPA
corvusclarkb: i see no reason not too; should be possible to do standalone23:45
clarkbgreat, I'll give that a go tomorrow morning then23:45
corvusfyi, i'm giving a talk to the gerrit hackathon tomorrow morning, so won't be available until later.  but don't wait for me.  :)23:47
opendevreviewMerged opendev/infra-openafs-deb trusty: Reset changelog version
opendevreviewMerged opendev/infra-vhd-util-deb focal: Trigger upstream build

Generated by 2.17.3 by Marius Gedminas - find it at!