| ianw | # top | 01:01 |
|---|---|---|
| ianw | terminals database is inaccessible | 01:01 |
| ianw | that's an interesting one on our 9-stream nodes | 01:01 |
| fungi | i guess we're missing whatever package provides the termcap file? | 01:01 |
| fungi | er, terminfo these daus | 01:02 |
| ianw | yeah, you'd think it would be pulled in as a dependency | 01:02 |
| fungi | probably /lib/terminfo or /usr/share/terminfo directories | 01:02 |
| ianw | ncurses-base not installed | 01:03 |
| fungi | yeah, that's ~ what provides it on debian | 01:03 |
| fungi | same package name | 01:04 |
| fungi | well, that and ncurses-term | 01:04 |
| fungi | ncurses-base populates /lib/terminfo and then ncurses-term provides the contents under /usr/share/terminfo | 01:05 |
| ianw | procps-ng -> libncurses -> ncurses-base ... so how is it not installed? | 01:05 |
| ianw | 2022-04-09 14:06:37.022 | > Installing : ncurses-6.2-8.20210508.el9.x86_64 66/227 | 01:06 |
| ianw | https://opendev.org/openstack/diskimage-builder/src/branch/master/diskimage_builder/elements/yum-minimal/root.d/08-yum-chroot#L91 | 01:07 |
| ianw | https://review.opendev.org/c/openstack/diskimage-builder/+/811392 didn't really record why we're installing ncurses-libs there | 01:09 |
| ianw | it looks like things are deliberately setup so that you don't have to pull in ncurses-base for non-interactive container images, etc. | 01:11 |
| fungi | funky funky | 01:12 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: yum-minimal: clean up release package installs https://review.opendev.org/c/openstack/diskimage-builder/+/837248 | 04:16 |
| opendevreview | Steve Baker proposed openstack/diskimage-builder master: Set machine-id to uninitialized to trigger first boot https://review.opendev.org/c/openstack/diskimage-builder/+/837251 | 05:12 |
| opendevreview | Merged openstack/diskimage-builder master: Update fedora element testing to F35 https://review.opendev.org/c/openstack/diskimage-builder/+/830113 | 05:15 |
| opendevreview | Merged opendev/git-review master: Treat ^C as "no" at confirmation prompt https://review.opendev.org/c/opendev/git-review/+/837044 | 05:18 |
| opendevreview | Merged opendev/git-review master: Force use of scp rather than sftp when possible https://review.opendev.org/c/opendev/git-review/+/823413 | 05:43 |
| *** bhagyashris is now known as bhagyashris|out | 06:26 | |
| hrw | https://review.opendev.org/c/openstack/project-config/+/836793 - if we have AFS volumes for CentOS 9 wheel mirrors then can it be merged? | 10:19 |
| *** rlandy_ is now known as rlandy | 10:50 | |
| *** dviroel|out is now known as dviroel | 11:15 | |
| fungi | hrw: looks like we have rw and ro volumes for mirror.wheel.cent9s{a,x}64 so the volume releases themselves should at least work, and otherwise that only seems to add the publish jobs to openstack/requirements periodic pipeline, so worst case it'll result in periodic failures if the wheel builds aren't going yet | 12:06 |
| hrw | ok | 12:07 |
| hrw | will look at requirements | 12:07 |
| fungi | i've approved 836829 (its dependency) | 12:07 |
| hrw | thx | 12:09 |
| fungi | and that merged, so i'll go ahead and single-core approve 836793 as it shouldn't be especially problematic if it runs early | 12:21 |
| opendevreview | Merged openstack/project-config master: Build and publish wheel mirror for CentOS Stream 9 https://review.opendev.org/c/openstack/project-config/+/836793 | 12:34 |
| *** ralonsoh_ is now known as ralonsoh | 12:34 | |
| fungi | hrw: ^ keep an eye on the periodic pipeline results for the openstack/requirements repo | 12:37 |
| fungi | since that's probably where you'll be able to tell whether it's working as desired | 12:37 |
| hrw | and have to send one for requirements | 13:23 |
| hrw | or w8... I looked at o/req jobs and got lost in how the hell it works at all d: | 13:24 |
| hrw | there is no build-wheel-cache-debian-bullseye in project.yaml but bullseye cache exists | 13:25 |
| fungi | does it have contents? | 13:26 |
| fungi | yeah, seems like it does anyway: https://static.opendev.org/mirror/wheel/debian-11-x86_64/d/django/ | 13:27 |
| hrw | I am in too bad shape to help today ;( | 13:29 |
| fungi | hrw: build-wheel-cache-debian-bullseye is a parent of publish-wheel-cache-debian-bullseye which is applied to the requirements project here: https://opendev.org/openstack/project-config/src/branch/master/zuul.d/projects.yaml#L4826 (that may take a while to load, it sure chews up my browser anyway) | 13:30 |
| fungi | so build-wheel-cache-debian-bullseye isn't used directly, we could mark it as an abstract job in order to make that more obvious, i suppose | 13:32 |
| hrw | ah. that way | 13:35 |
| *** artom_ is now known as artom | 14:27 | |
| *** dviroel is now known as dviroel|lunch|afk | 15:32 | |
| Clark[m] | fungi: anything I should look at with urgency? I've had a morning already and need to find food etc so going slow | 15:55 |
| clarkb | ok I've managed to get a real irc client but still no breakfast | 16:06 |
| *** marios is now known as marios|out | 16:08 | |
| fungi | clarkb: nothing is urgent, no | 16:09 |
| fungi | we're troubleshooting a spellchecking sphinx extension in #openstack-doc if you're interested, but not much else goin on | 16:10 |
| fungi | centos 9 wheel building jobs merged too | 16:10 |
| clarkb | oh exciting. My plan was to look at deleting the old subunit stff and health api server but that was contingnent on getting my office back and I don't have it for another day after parents missed their flight | 16:12 |
| clarkb | Seems like there are some zuul reviews to do and the meeting agenda and something else that I will probably find when I open my notes | 16:13 |
| fungi | added to the agenda, but we're probably due for another git-review release. a few fixes have just merged, but there are a few outstanding changes for it which may merit further discussion | 16:17 |
| clarkb | re python3.5 I think the reason we've tried to keep it around is that people use git-reveiew on all sorts of platforms | 16:33 |
| clarkb | but I guess we can tell them to install old git review now | 16:34 |
| fungi | yeah, the only reason it's really being proposed to drop is to avoid extra fallback logic for the signed commits workaround | 16:34 |
| fungi | we could decide the additional complexity is worth keeping 3.5 support around for longer, or delay adding the signed commits workaround to a future release | 16:35 |
| clarkb | I think we can pin it. I just need toget situated enough to +2 it | 16:36 |
| fungi | arewe actually still using it in any xenial-based jobs? | 16:37 |
| clarkb | I don't think we are | 16:38 |
| fungi | just curious what we actually need to pin | 16:38 |
| clarkb | I've approved https://review.opendev.org/c/opendev/git-review/+/837228 and the python3.5 and git fix will need to be rebased on that | 16:38 |
| clarkb | its more for users on mac os iirc | 16:38 |
| fungi | yep, i expected so | 16:38 |
| fungi | what's for users on mac again? | 16:39 |
| clarkb | python3.5 | 16:39 |
| clarkb | theres a whole thing about how mac osx doesn't update python iirc | 16:39 |
| clarkb | fungi: do you understand https://review.opendev.org/c/opendev/git-review/+/823130/3/git_review/cmd.py ? | 16:40 |
| clarkb | I wish we had a full traceback | 16:40 |
| clarkb | oh you've got comments I need to read them | 16:40 |
| fungi | clarkb: yeah, the underlying problem is that we reference __doc__ without defining it, and it doesn't get inherited | 16:40 |
| fungi | Exception has __doc__ but inheriting from a class which has a __doc__ doesn't really work because for some magic reason i don't quite fathom it gets initialized to None | 16:42 |
| clarkb | I guess that makes sense since __doc__ is specific to the class. I think we should use another attribute assigned to self.__doc__ in the parent? | 16:42 |
| clarkb | in the parent basedoc = self.__doc__ then we can use self.basedoc instead of self.__doc__ in the children? | 16:42 |
| fungi | i think we should probably just remove __doc__ everywhere since nothing defines it anyway | 16:42 |
| fungi | and it can't be inherited )and what it would inherit from the Exception class is useless if it even were possible) | 16:43 |
| clarkb | ok just had my brother check a macbook and maybe 3.5 isn't a problem there anymore | 16:43 |
| clarkb | oh if nothing is defining it (isin't it the docstring?) then ya dropping it seems fine | 16:43 |
| clarkb | ok dlayed school run now due to snow. Back in a bit | 16:45 |
| fungi | ohhh, so that's what's magic about __doc__ | 16:51 |
| fungi | the docstring is stuffed in there, and if there is no docstring then you end up with None instead of anything getting inherited | 16:51 |
| opendevreview | Jeremy Stanley proposed opendev/git-review master: Fix git_review.cmd.CommandFailed: <exception str() failed> https://review.opendev.org/c/opendev/git-review/+/823130 | 16:57 |
| fungi | clarkb: ^ | 16:57 |
| opendevreview | Merged opendev/git-review master: Clean up package metadata https://review.opendev.org/c/opendev/git-review/+/837228 | 16:58 |
| opendevreview | Jeremy Stanley proposed opendev/git-review master: Drop support for Python 3.5 https://review.opendev.org/c/opendev/git-review/+/837222 | 17:00 |
| opendevreview | Jeremy Stanley proposed opendev/git-review master: Fix submitting signed patches https://review.opendev.org/c/opendev/git-review/+/823318 | 17:00 |
| clarkb | lgtm +2s all around | 17:04 |
| fungi | once those merge, we'll be back to 0 open changes for git-review and in good shape to tag 2.3.0 (i assume dropping py35 support doesn't warrant a major version bump anyway) | 17:06 |
| fungi | we can discuss further during the meeting | 17:06 |
| clarkb | ya Ithink thats fine and discussing at the meeting sounds great | 17:06 |
| fungi | since we're increasing the minimum required python and git, it's worth some additional eyeballs | 17:07 |
| clarkb | apparently 100k have no power today now. I don't expect problems myself but heads up that is a possibility I suppose | 17:20 |
| clarkb | basically the worst of it is over so if it hasn't happened yet I don't expect it too | 17:21 |
| fungi | i'd take it as an excuse to go sledding | 17:22 |
| clarkb | thats the thing it smotsly melted here. THe issue is about 3 miles east of me which is between here and the iarport | 17:22 |
| fungi | aww | 17:22 |
| clarkb | and if you go uphill a bit it turns into a massive mess | 17:22 |
| opendevreview | Merged opendev/git-review master: Fix git_review.cmd.CommandFailed: <exception str() failed> https://review.opendev.org/c/opendev/git-review/+/823130 | 17:48 |
| opendevreview | Ebbex proposed opendev/bindep master: Add AlmaLinux support https://review.opendev.org/c/opendev/bindep/+/837118 | 18:12 |
| opendevreview | Ebbex proposed opendev/bindep master: Pin distro to fix tox tests https://review.opendev.org/c/opendev/bindep/+/837394 | 18:12 |
| clarkb | the failure related to ^ are really weird | 18:30 |
| clarkb | it works when you run a single test but not when you run tests together in parallel or serially | 18:30 |
| clarkb | I suspect our fixturing that was originally boorrwed from distro but maybe they changed it | 18:31 |
| *** pojadhav is now known as pojadhav|out | 18:40 | |
| clarkb | I've figured it out | 18:54 |
| opendevreview | Clark Boylan proposed opendev/bindep master: Fix bindep testing with distro 1.7.0 and greater https://review.opendev.org/c/opendev/bindep/+/837394 | 18:59 |
| opendevreview | Clark Boylan proposed opendev/bindep master: Add AlmaLinux support https://review.opendev.org/c/opendev/bindep/+/837118 | 18:59 |
| opendevreview | Clark Boylan proposed opendev/bindep master: Fix bindep testing with distro 1.7.0 and greater https://review.opendev.org/c/opendev/bindep/+/837394 | 19:01 |
| opendevreview | Clark Boylan proposed opendev/bindep master: Add AlmaLinux support https://review.opendev.org/c/opendev/bindep/+/837118 | 19:01 |
| clarkb | ok that should pass testing now | 19:01 |
| opendevreview | Clark Boylan proposed opendev/bindep master: Fix bindep testing with distro 1.7.0 and greater https://review.opendev.org/c/opendev/bindep/+/837394 | 19:21 |
| opendevreview | Clark Boylan proposed opendev/bindep master: Add AlmaLinux support https://review.opendev.org/c/opendev/bindep/+/837118 | 19:21 |
| opendevreview | Mohammed Naser proposed zuul/zuul-jobs master: build-docker-image: Fix buildset registry pushes https://review.opendev.org/c/zuul/zuul-jobs/+/837405 | 19:23 |
| *** rlandy_ is now known as rlandy | 19:28 | |
| opendevreview | Merged zuul/zuul-jobs master: Fix a bug in s3 log uploader with .gz files https://review.opendev.org/c/zuul/zuul-jobs/+/798802 | 20:04 |
| mnaser | https://review.opendev.org/c/opendev/system-config/+/837232 is trivial and would be nice to have if anyoen is around +) | 20:10 |
| mnaser | =) | 20:10 |
| opendevreview | Merged opendev/system-config master: docker: add arm64 mirroring https://review.opendev.org/c/opendev/system-config/+/837232 | 21:11 |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: Add upload-logs-ibm role https://review.opendev.org/c/zuul/zuul-jobs/+/826158 | 21:20 |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: Add per-build WinRM cert generation https://review.opendev.org/c/zuul/zuul-jobs/+/837416 | 21:23 |
| *** dviroel is now known as dviroel|out | 22:02 | |
| clarkb | fungi: ianw https://review.opendev.org/c/opendev/system-config/+/249815 should be super easy if you havea chance | 22:05 |
| ianw | too slow :) | 22:12 |
| fungi | sorry! ;) | 22:12 |
| clarkb | https://review.opendev.org/c/opendev/system-config/+/505342 and https://review.opendev.org/c/opendev/system-config/+/529426 are another couple of easier ones off of that cleanup topic. The others involve a bit more in depth review I think but also appreciate reviews on those too :) | 22:19 |
| clarkb | and https://review.opendev.org/c/opendev/bindep/+/837394 fixes bindep testing with latest distro release and the child adds alma linux support to bindep | 22:19 |
| clarkb | I think those are straightforward and we should probably plan a bindep release as well as git review in the near future | 22:20 |
| clarkb | I'm going to take the gerrit fork retirement off the meeting agenda as I think that is complete (I saw the zuul side change land yesterday) | 22:21 |
| ianw | i would have thought pip could sort out which version of distro to isntall, but lgtm | 22:24 |
| clarkb | ianw: I think it can but I wanted to force new python onto the new version to ensure we test the other half of that condition | 22:24 |
| clarkb | it would work 99% of the time without the explicit version stuff. This just makes it extra sure | 22:25 |
| opendevreview | Merged opendev/system-config master: Clean up more puppetmaster remnants https://review.opendev.org/c/opendev/system-config/+/249815 | 22:25 |
| opendevreview | Merged zuul/zuul-jobs master: Add upload-logs-ibm role https://review.opendev.org/c/zuul/zuul-jobs/+/826158 | 22:30 |
| clarkb | meeting agenda has been updated. I'll send that out before my EOD get any updates in there now :) | 22:30 |
| mnaser | hmmmm | 22:43 |
| mnaser | i'm seeing openstack/loci jobs failing with "File has unexpected size (1220414 != 1220413). Mirror sync in progress? [IP: 10.209.128.57 80]" | 22:43 |
| mnaser | that's not technically possible, is it? with afs and volume releases and all? | 22:44 |
| opendevreview | Merged opendev/system-config master: Remove openstackwatch docs https://review.opendev.org/c/opendev/system-config/+/505342 | 22:44 |
| ianw | mnaser: it is possible if we sync from an upstream who isn't as picky about being atomic as us :) | 22:45 |
| ianw | is it fedora? | 22:46 |
| mnaser | ubuntu focal | 22:46 |
| mnaser | i haven't rechecked yet, but also trying to help stabilize the jobs too -- https://zuul.opendev.org/t/openstack/build/9bfffb0b39504381ad802155eaff00f1 | 22:46 |
| clarkb | I think if it happens on ubuntu it implies that you may not be apt-get updating close enough to the installation? but maybe it is something else | 22:46 |
| clarkb | since reprepro is supposed to verify things | 22:46 |
| mnaser | yeah i'm surprised a lot more things werent on fire if this happened.. | 22:48 |
| mnaser | "File has unexpected size (1220414 != 1220413)" .. only 1... byte? and this apt-get stuff happens inside a docker container, so who knows if maybe that's the cause | 22:49 |
| clarkb | also particularly when talking about containers is it even using our mirrors? I know loci refused to for a long time as we don't sign them | 22:50 |
| clarkb | it could be an internet thing if off by one byte and not using our mirrors | 22:50 |
| ianw | heh "an internet thing" :) | 22:50 |
| mnaser | they must have caved in :) | 22:50 |
| mnaser | the logs does say: Failed to fetch http://mirror-int.ord.rax.opendev.org/ubuntu/dists/focal-updates/universe/binary-amd64/Packages.gz File has unexpected size (1220414 != 1220413) | 22:50 |
| ianw | speaking of internet things ... how does the "openstack-mirroring" bot on github work? | 22:50 |
| clarkb | mnaser: ok thats just good to confirm | 22:51 |
| ianw | i see that as 1220414 bytes | 22:51 |
| ianw | so the packages.gz must be wrong | 22:51 |
| clarkb | mnaser: if you drop the -int from the hostname you will be ble to access the same data at the same apache | 22:51 |
| clarkb | http://mirror.ord.rax.opendev.org/ubuntu/dists/focal-updates/Release reports 977ca51f5f804011913a1cf8dd2d15ac 1220414 universe/binary-amd64/Packages.gz | 22:53 |
| clarkb | er thats the md5sum | 22:54 |
| mnaser | so it expected 1220413 (md5sum a3c7751988338a7a27df2a9e75dfae88) | 22:55 |
| ianw | but it says 1220414, which is the size that i see | 22:55 |
| clarkb | ianw: yup if I grab it and hash it and check filesizes everything lines up | 22:55 |
| clarkb | I suspect that the container iddn't apt-get update close enough to an install? something like that | 22:55 |
| clarkb | or maybe the Release file was grabbed, then the afs releas ehappened, then it grabbed the Packages.gz | 22:55 |
| clarkb | or the container didn't update at all and is using its cached version | 22:56 |
| mnaser | it looks like the `apt-get update` is actually what failed | 22:56 |
| clarkb | ah ok so then the release file and packages.gz download racing a release seems most likely | 22:56 |
| ianw | however the chance that is out by one byte? | 22:57 |
| mnaser | what is interesitng is this wanst eh only job that failed but a bunch of other ones | 22:57 |
| clarkb | ianw: I don't think its that crazy considering the packages likely don't change much and it is compressed | 22:58 |
| mnaser | one failed at "2022-04-11 20:42:04.415792" and another failed at "2022-04-11 20:42:51.408161" | 22:58 |
| clarkb | so we're probably looking at a fairly small window that gz can compressing things to due to minimal updates over time? | 22:58 |
| mnaser | i wonder if the window got bigger because of the arm64 docker image syncs.. maybe? | 22:59 |
| clarkb | anyway we can verify the content the mirrors serves matches what it has in the release file | 22:59 |
| clarkb | so I odn't think the mirror is to blame | 22:59 |
| mnaser | yeah i figured if it was the mirror, we'd see *a lot* more noise here :) | 22:59 |
| clarkb | unless it was broken and then fixed itself in the interim period | 22:59 |
| mnaser | maybe im just REALLY lucky :) | 22:59 |
| ianw | End timer for mirror_ubuntu | 23:01 |
| ianw | 2022-04-11 22:41:52 | Done. | 23:01 |
| ianw | suspiciously close | 23:01 |
| ianw | to answer my own question; openstack-mirroring user is actually driven from the secret "openstack-github-mirroring" https://opendev.org/openstack/project-config/src/branch/master/zuul.d/secrets.yaml#L662 | 23:08 |
| clarkb | oh ya sorry its all done via zuul jobs | 23:11 |
| fungi | also does the pull request closing for openstack's github repos these days | 23:14 |
| opendevreview | Ian Wienand proposed opendev/system-config master: doc/jeepyb: further cleanups https://review.opendev.org/c/opendev/system-config/+/837433 | 23:16 |
| ianw | clarkb/fungi: ^ noted :) | 23:16 |
| fungi | awesome | 23:17 |
| opendevreview | Merged zuul/zuul-jobs master: build-docker-image: Fix buildset registry pushes https://review.opendev.org/c/zuul/zuul-jobs/+/837405 | 23:23 |
| clarkb | ianw: that failed doc builds | 23:25 |
| opendevreview | Ian Wienand proposed opendev/system-config master: doc/jeepyb: further cleanups https://review.opendev.org/c/opendev/system-config/+/837433 | 23:29 |
| opendevreview | Merged opendev/system-config master: doc/jeepyb: further cleanups https://review.opendev.org/c/opendev/system-config/+/837433 | 23:58 |
| clarkb | ok meeting agenda sent | 23:59 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!