Monday, 2022-04-11

ianw# top01:01
ianwterminals database is inaccessible01:01
ianwthat's an interesting one on our 9-stream nodes01:01
fungii guess we're missing whatever package provides the termcap file?01:01
fungier, terminfo these daus01:02
ianwyeah, you'd think it would be pulled in as a dependency01:02
fungiprobably /lib/terminfo or /usr/share/terminfo directories01:02
ianwncurses-base not installed01:03
fungiyeah, that's ~ what provides it on debian01:03
fungisame package name01:04
fungiwell, that and ncurses-term01:04
fungincurses-base populates /lib/terminfo and then ncurses-term provides the contents under /usr/share/terminfo01:05
ianwprocps-ng -> libncurses -> ncurses-base ... so how is it not installed?01:05
ianw2022-04-09 14:06:37.022 | >   Installing       : ncurses-6.2-8.20210508.el9.x86_64                   66/22701:06
ianw didn't really record why we're installing ncurses-libs there01:09
ianwit looks like things are deliberately setup so that you don't have to pull in ncurses-base for non-interactive container images, etc.01:11
fungifunky funky01:12
opendevreviewIan Wienand proposed openstack/diskimage-builder master: yum-minimal: clean up release package installs
opendevreviewSteve Baker proposed openstack/diskimage-builder master: Set machine-id to uninitialized to trigger first boot
opendevreviewMerged openstack/diskimage-builder master: Update fedora element testing to F35
opendevreviewMerged opendev/git-review master: Treat ^C as "no" at confirmation prompt
opendevreviewMerged opendev/git-review master: Force use of scp rather than sftp when possible
*** bhagyashris is now known as bhagyashris|out06:26
hrw - if we have AFS volumes for CentOS 9 wheel mirrors then can it be merged?10:19
*** rlandy_ is now known as rlandy10:50
*** dviroel|out is now known as dviroel11:15
fungihrw: looks like we have rw and ro volumes for mirror.wheel.cent9s{a,x}64 so the volume releases themselves should at least work, and otherwise that only seems to add the publish jobs to openstack/requirements periodic pipeline, so worst case it'll result in periodic failures if the wheel builds aren't going yet12:06
hrwwill look at requirements12:07
fungii've approved 836829 (its dependency)12:07
fungiand that merged, so i'll go ahead and single-core approve 836793 as it shouldn't be especially problematic if it runs early12:21
opendevreviewMerged openstack/project-config master: Build and publish wheel mirror for CentOS Stream 9
*** ralonsoh_ is now known as ralonsoh12:34
fungihrw: ^ keep an eye on the periodic pipeline results for the openstack/requirements repo12:37
fungisince that's probably where you'll be able to tell whether it's working as desired12:37
hrwand have to send one for requirements13:23
hrwor w8... I looked at o/req jobs and got lost in how the hell it works at all d:13:24
hrwthere is no build-wheel-cache-debian-bullseye in project.yaml but bullseye cache exists13:25
fungidoes it have contents?13:26
fungiyeah, seems like it does anyway:
hrwI am in too bad shape to help today ;(13:29
fungihrw: build-wheel-cache-debian-bullseye is a parent of publish-wheel-cache-debian-bullseye which is applied to the requirements project here: (that may take a while to load, it sure chews up my browser anyway)13:30
fungiso build-wheel-cache-debian-bullseye isn't used directly, we could mark it as an abstract job in order to make that more obvious, i suppose13:32
hrwah. that way13:35
*** artom_ is now known as artom14:27
*** dviroel is now known as dviroel|lunch|afk15:32
Clark[m]fungi: anything I should look at with urgency? I've had a morning already and need to find food etc so going slow15:55
clarkbok I've managed to get a real irc client but still no breakfast16:06
*** marios is now known as marios|out16:08
fungiclarkb: nothing is urgent, no16:09
fungiwe're troubleshooting a spellchecking sphinx extension in #openstack-doc if you're interested, but not much else goin on16:10
fungicentos 9 wheel building jobs merged too16:10
clarkboh exciting. My plan was to look at deleting the old subunit stff and health api server but that was contingnent on getting my office back and I don't have it for another day after parents missed their flight16:12
clarkbSeems like there are some zuul reviews to do and the meeting agenda and something else that I will probably find when I open my notes16:13
fungiadded to the agenda, but we're probably due for another git-review release. a few fixes have just merged, but there are a few outstanding changes for it which may merit further discussion16:17
clarkbre python3.5 I think the reason we've tried to keep it around is that people use git-reveiew on all sorts of platforms16:33
clarkbbut I guess we can tell them to install old git review now16:34
fungiyeah, the only reason it's really being proposed to drop is to avoid extra fallback logic for the signed commits workaround16:34
fungiwe could decide the additional complexity is worth keeping 3.5 support around for longer, or delay adding the signed commits workaround to a future release16:35
clarkbI think we can pin it. I just need toget situated enough to +2 it16:36
fungiarewe actually still using it in any xenial-based jobs?16:37
clarkbI don't think we are16:38
fungijust curious what we actually need to pin16:38
clarkbI've approved and the python3.5 and git fix will need to be rebased on that16:38
clarkbits more for users on mac os iirc16:38
fungiyep, i expected so16:38
fungiwhat's for users on mac again?16:39
clarkbtheres a whole thing about how mac osx doesn't update python iirc16:39
clarkbfungi: do you understand ?16:40
clarkbI wish we had a full traceback16:40
clarkboh you've got comments I need to read them16:40
fungiclarkb: yeah, the underlying problem is that we reference __doc__ without defining it, and it doesn't get inherited16:40
fungiException has __doc__ but inheriting from a class which has a __doc__ doesn't really work because for some magic reason i don't quite fathom it gets initialized to None16:42
clarkbI guess that makes sense since __doc__ is specific to the class. I think we should use another attribute assigned to self.__doc__ in the parent?16:42
clarkbin the parent basedoc = self.__doc__ then we can use self.basedoc instead of self.__doc__ in the children?16:42
fungii think we should probably just remove __doc__ everywhere since nothing defines it anyway16:42
fungiand it can't be inherited )and what it would inherit from the Exception class is useless if it even were possible)16:43
clarkbok just had my brother check a macbook and maybe 3.5 isn't a problem there anymore16:43
clarkboh if nothing is defining it (isin't it the docstring?) then ya dropping it seems fine16:43
clarkbok dlayed school run now due to snow. Back in a bit16:45
fungiohhh, so that's what's magic about __doc__16:51
fungithe docstring is stuffed in there, and if there is no docstring then you end up with None instead of anything getting inherited16:51
opendevreviewJeremy Stanley proposed opendev/git-review master: Fix git_review.cmd.CommandFailed: <exception str() failed>
fungiclarkb: ^16:57
opendevreviewMerged opendev/git-review master: Clean up package metadata
opendevreviewJeremy Stanley proposed opendev/git-review master: Drop support for Python 3.5
opendevreviewJeremy Stanley proposed opendev/git-review master: Fix submitting signed patches
clarkblgtm +2s all around17:04
fungionce those merge, we'll be back to 0 open changes for git-review and in good shape to tag 2.3.0 (i assume dropping py35 support doesn't warrant a major version bump anyway)17:06
fungiwe can discuss further during the meeting17:06
clarkbya  Ithink thats fine and discussing at the meeting sounds great17:06
fungisince we're increasing the minimum required python and git, it's worth some additional eyeballs17:07
clarkbapparently 100k have no power today now. I don't expect problems myself but heads up that is a possibility I suppose17:20
clarkbbasically the worst of it is over so if it hasn't happened yet I don't expect it too17:21
fungii'd take it as an excuse to go sledding17:22
clarkbthats the thing it smotsly melted here. THe issue is about 3 miles east of me which is between here and the iarport17:22
clarkband if you go uphill a bit it turns into a massive mess17:22
opendevreviewMerged opendev/git-review master: Fix git_review.cmd.CommandFailed: <exception str() failed>
opendevreviewEbbex proposed opendev/bindep master: Add AlmaLinux support
opendevreviewEbbex proposed opendev/bindep master: Pin distro to fix tox tests
clarkbthe failure related to ^ are really weird18:30
clarkbit works when you run a single test but not when you run tests together in parallel or serially18:30
clarkbI suspect our fixturing that was originally boorrwed from distro but maybe they changed it18:31
*** pojadhav is now known as pojadhav|out18:40
clarkbI've figured it out18:54
opendevreviewClark Boylan proposed opendev/bindep master: Fix bindep testing with distro 1.7.0 and greater
opendevreviewClark Boylan proposed opendev/bindep master: Add AlmaLinux support
opendevreviewClark Boylan proposed opendev/bindep master: Fix bindep testing with distro 1.7.0 and greater
opendevreviewClark Boylan proposed opendev/bindep master: Add AlmaLinux support
clarkbok that should pass testing now19:01
opendevreviewClark Boylan proposed opendev/bindep master: Fix bindep testing with distro 1.7.0 and greater
opendevreviewClark Boylan proposed opendev/bindep master: Add AlmaLinux support
opendevreviewMohammed Naser proposed zuul/zuul-jobs master: build-docker-image: Fix buildset registry pushes
*** rlandy_ is now known as rlandy19:28
opendevreviewMerged zuul/zuul-jobs master: Fix a bug in s3 log uploader with .gz files
mnaser is trivial and would be nice to have if anyoen is around +)20:10
opendevreviewMerged opendev/system-config master: docker: add arm64 mirroring
opendevreviewJames E. Blair proposed zuul/zuul-jobs master: Add upload-logs-ibm role
opendevreviewJames E. Blair proposed zuul/zuul-jobs master: Add per-build WinRM cert generation
*** dviroel is now known as dviroel|out22:02
clarkbfungi: ianw should be super easy if you havea chance22:05
ianwtoo slow :)22:12
fungisorry! ;)22:12
clarkb and are another couple of easier ones off of that cleanup topic. The others involve a bit more in depth review I think but also appreciate reviews on those too :)22:19
clarkband fixes bindep testing with latest distro release and the child adds alma linux support to bindep22:19
clarkbI think those are straightforward and we should probably plan a bindep release as well as git review in the near future22:20
clarkbI'm going to take the gerrit fork retirement off the meeting agenda as I think that is complete (I saw the zuul side change land yesterday)22:21
ianwi would have thought pip could sort out which version of distro to isntall, but lgtm22:24
clarkbianw: I think it can but I wanted to force new python onto the new version to ensure we test the other half of that condition22:24
clarkbit would work 99% of the time without the explicit version stuff. This just makes it extra sure22:25
opendevreviewMerged opendev/system-config master: Clean up more puppetmaster remnants
opendevreviewMerged zuul/zuul-jobs master: Add upload-logs-ibm role
clarkbmeeting agenda has been updated. I'll send that out before my EOD get any updates in there now :)22:30
mnaseri'm seeing openstack/loci jobs failing with "File has unexpected size (1220414 != 1220413). Mirror sync in progress? [IP: 80]"22:43
mnaserthat's not technically possible, is it?  with afs and volume releases and all?22:44
opendevreviewMerged opendev/system-config master: Remove openstackwatch docs
ianwmnaser: it is possible if we sync from an upstream who isn't as picky about being atomic as us :)22:45
ianwis it fedora?22:46
mnaserubuntu focal22:46
mnaseri haven't rechecked yet, but also trying to help stabilize the jobs too --
clarkbI think if it happens on ubuntu it implies that you may not be apt-get updating close enough to the installation? but maybe it is something else22:46
clarkbsince reprepro is supposed to verify things22:46
mnaseryeah i'm surprised a lot more things werent on fire if this happened..22:48
mnaser"File has unexpected size (1220414 != 1220413)" .. only 1... byte? and this apt-get stuff happens inside a docker container, so who knows if maybe that's the cause22:49
clarkbalso particularly when talking about containers is it even using our mirrors? I know loci refused to for a long time as we don't sign them22:50
clarkbit could be an internet thing if off by one byte and not using our mirrors22:50
ianwheh "an internet thing" :)22:50
mnaserthey must have caved in :) 22:50
mnaserthe logs does say: Failed to fetch  File has unexpected size (1220414 != 1220413)22:50
ianwspeaking of internet things ... how does the "openstack-mirroring" bot on github work?22:50
clarkbmnaser: ok thats just good to confirm22:51
ianwi see that as 1220414 bytes22:51
ianwso the packages.gz must be wrong22:51
clarkbmnaser: if you drop the -int from the hostname you will be ble to access the same data at the same apache22:51
clarkb reports 977ca51f5f804011913a1cf8dd2d15ac 1220414 universe/binary-amd64/Packages.gz22:53
clarkber thats the md5sum22:54
mnaserso it expected 1220413 (md5sum a3c7751988338a7a27df2a9e75dfae88)22:55
ianwbut it says 1220414, which is the size that i see22:55
clarkbianw: yup if I grab it and hash it and check filesizes everything lines up22:55
clarkbI suspect that the container iddn't apt-get update close enough to an install? something like that22:55
clarkbor maybe the Release file was grabbed, then the afs releas ehappened, then it grabbed the Packages.gz22:55
clarkbor the container didn't update at all and is using its cached version22:56
mnaserit looks like the `apt-get update` is actually what failed22:56
clarkbah ok so then the release file and packages.gz download racing a release seems most likely22:56
ianwhowever the chance that is out by one byte?22:57
mnaserwhat is interesitng is this wanst eh only job that failed but a bunch of other ones22:57
clarkbianw: I don't think its that crazy considering the packages likely don't change much and it is compressed22:58
mnaserone failed at "2022-04-11 20:42:04.415792" and another failed at "2022-04-11 20:42:51.408161"22:58
clarkbso we're probably looking at a fairly small window that gz can compressing things to due to minimal updates over time?22:58
mnaseri wonder if the window got bigger because of the arm64 docker image syncs.. maybe?22:59
clarkbanyway we can verify the content the mirrors serves matches what it has in the release file22:59
clarkbso I odn't think the mirror is to blame22:59
mnaseryeah i figured if it was the mirror, we'd see *a lot* more noise here :)22:59
clarkbunless it was broken and then fixed itself in the interim period22:59
mnasermaybe im just REALLY lucky :)22:59
ianwEnd timer for mirror_ubuntu23:01
ianw2022-04-11 22:41:52  | Done.23:01
ianwsuspiciously close23:01
ianwto answer my own question; openstack-mirroring user is actually driven from the secret "openstack-github-mirroring"
clarkboh ya sorry its all done via zuul jobs23:11
fungialso does the pull request closing for openstack's github repos these days23:14
opendevreviewIan Wienand proposed opendev/system-config master: doc/jeepyb: further cleanups
ianwclarkb/fungi: ^ noted :)23:16
opendevreviewMerged zuul/zuul-jobs master: build-docker-image: Fix buildset registry pushes
clarkbianw: that failed doc builds23:25
opendevreviewIan Wienand proposed opendev/system-config master: doc/jeepyb: further cleanups
opendevreviewMerged opendev/system-config master: doc/jeepyb: further cleanups
clarkbok meeting agenda sent23:59

Generated by 2.17.3 by Marius Gedminas - find it at!