| clarkb | and the job completed successfully all looks good to me | 00:00 |
|---|---|---|
| opendevreview | Merged opendev/system-config master: Clean up defunct OpenStack mailing lists https://review.opendev.org/c/opendev/system-config/+/839990 | 00:01 |
| opendevreview | Merged opendev/system-config master: third-party CI: reminder to keep address current https://review.opendev.org/c/opendev/system-config/+/840251 | 00:01 |
| clarkb | I need to go prep dinner now. Thanks for the help getting this stuff in | 00:01 |
| opendevreview | Ian Wienand proposed opendev/base-jobs master: Add centos-9-stream-arm64 nodeset https://review.opendev.org/c/opendev/base-jobs/+/840255 | 00:11 |
| ianw | clarkb: ^ i think that was just what was missing | 00:12 |
| opendevreview | Merged openstack/diskimage-builder master: Revert "Fallback to persistent netifs names with systemd" https://review.opendev.org/c/openstack/diskimage-builder/+/838863 | 00:26 |
| opendevreview | Merged opendev/base-jobs master: Add centos-9-stream-arm64 nodeset https://review.opendev.org/c/opendev/base-jobs/+/840255 | 00:31 |
| *** rlandy|rover|bbl is now known as rlandy|rover | 00:36 | |
| *** rlandy|rover is now known as rlandy|out | 00:39 | |
| opendevreview | Balendu Mouli Burla proposed openstack/project-config master: Add SRIOV FEC Operator app to StarlingX https://review.opendev.org/c/openstack/project-config/+/840263 | 03:58 |
| ianw | i'll do that gerrit restart in a minute | 04:18 |
| ianw | RepoDigests": [ | 04:44 |
| ianw | "opendevorg/gerrit@sha256:ee41b17e44b83a3b22632aa7d688b6c6a6f80d309a430bc9ccad60f6a6b4e6e9" | 04:44 |
| ianw | ], | 04:44 |
| ianw | which matches https://hub.docker.com/layers/opendevorg/gerrit/3.4/images/sha256-ee41b17e44b83a3b22632aa7d688b6c6a6f80d309a430bc9ccad60f6a6b4e6e9?context=explore | 04:44 |
| ianw | that matches @ https://zuul.opendev.org/t/openstack/build/82bea43f57f0498a91ee98bc7e57a111/log/job-output.txt#7663 | 04:52 |
| ianw | the other two changes are config changes, which have applied | 04:52 |
| ianw | ok, we're back | 04:56 |
| ianw | #status log restarted gerrit to pickup changes deployed with stack @ https://review.opendev.org/c/opendev/system-config/+/839251/ | 04:57 |
| opendevstatus | ianw: finished logging | 04:57 |
| *** marios is now known as marios|ruck | 05:08 | |
| ianw | infra-root: if we can get https://review.opendev.org/c/opendev/system-config/+/839841 in before the wheel run, i think we'll finally be back to it | 06:34 |
| ianw | that will fix up the publishing steps on 9-stream. | 06:34 |
| frickler | ianw: +2, not sure when the wheel builds run, are those in the daily periodic pipeline? then we'd have time to wait for another review | 07:28 |
| frickler | does anyone intend to look at the centos-7 builds or should we just stop them for now? | 07:28 |
| *** jpena|off is now known as jpena | 07:34 | |
| *** rlandy|out is now known as rlandy | 10:24 | |
| *** dviroel|out is now known as dviroel | 11:18 | |
| fungi | did someone start looking into it yet? pip list in the builder container on nb01 says dib 3.20.3 is installed, but the line numbers in the failing logs don't seem to correspond to the commands on those lines | 11:47 |
| fungi | in particular, this call: | 11:48 |
| fungi | + /opt/dib_tmp/dib_build.qOsibSQN/hooks/root.d/08-yum-chroot:main:348 : _run_chroot yum -y --releasever=7 '' | 11:48 |
| fungi | in 3.20.3 that line is a code comment: | 11:49 |
| fungi | https://opendev.org/openstack/diskimage-builder/src/tag/3.20.3/diskimage_builder/elements/yum-minimal/root.d/08-yum-chroot#L348 | 11:49 |
| fungi | it's almost as if the element has been hand-patched in place on the builder | 11:50 |
| fungi | or something went wrong with the dib install when the container was built | 11:50 |
| opendevreview | Merged opendev/system-config master: Test openafs roles on CentOS 9-stream https://review.opendev.org/c/opendev/system-config/+/839841 | 11:59 |
| fungi | yeah, i've grabbed a copy of the 08-yum-chroot being used on nb01 and it doesn't seem to correspond to the state of 3.20.3 or master | 12:05 |
| fungi | the only change to merge altering that file since 3.20.3 was https://review.opendev.org/837248 and the file being used on the builder looks similar to that but with two lines changed in the vicinity of the failure we're logging | 12:07 |
| fungi | it adds a --releasever=${DIB_RELEASE/-*/} to the _run_chroot invocation, which can be seen in the logged command | 12:11 |
| fungi | adds it to the yum update call on the previous line as well, but that seems to be working | 12:11 |
| fungi | frickler: per discussion in #openstack-dib, it looks like https://review.opendev.org/839840 is a variant of that and ianw seems to think it might fix the issue | 12:13 |
| fungi | i guess ianw may have altered the 08-yum-chroot on nb01 in order to test some theories | 12:14 |
| fungi | which explains the confusing state with it now | 12:14 |
| *** rlandy is now known as rlandy|mtg | 13:05 | |
| amorin | I received an highligh from yesterday, anything wrong on OVH cloud? | 13:06 |
| fungi | amorin: looks like we were getting some nova.exception.OrphanedObjectError errors like this in gra1 "Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible." | 13:11 |
| fungi | that's when calling the /servers/detail api method from our cleanup loop | 13:14 |
| fungi | seems to still be going on | 13:14 |
| fungi | amorin: i don't think it's impacting much if anything on our side, more that we wanted to let you know | 13:14 |
| *** artom__ is now known as artom | 13:21 | |
| amorin | ack, ok | 13:41 |
| amorin | do you have any nova instance ID maybe? | 13:41 |
| amorin | fungi | 13:41 |
| clarkb | amorin: no this appears to happen when doing a detailed servers listing when we looked for leaked instances | 13:53 |
| clarkb | amorin: we aren't querying a specific instance but instead GETing /servers/detail which is meant to give us a list of instanes iirc | 13:53 |
| amorin | on GRA1? | 13:53 |
| clarkb | amorin: correct | 13:53 |
| amorin | checking | 13:53 |
| clarkb | fungi: yes ianw hand patched it then pushed that fix you found. The plan currently aiui is to do a release ~today to fix it properly | 14:00 |
| fungi | well, the hand patched version is still failing according to the centos-7 image build logs fromnb01 | 14:01 |
| *** rlandy|mtg is now known as rlandy | 14:02 | |
| fungi | and is also implemented a bit differently from the change we reviewed | 14:34 |
| clarkb | looking at the change we reviewed I think it avoids this problem | 14:37 |
| clarkb | but I'm trying to find the content in the container now to compare properly | 14:37 |
| fungi | clarkb: i copied it to my homedir, snagged from an ephemeral build dir | 14:41 |
| fungi | clarkb: nb01:~fungi/08-yum-chroot | 14:41 |
| fungi | that's what it's using now | 14:42 |
| clarkb | fungi: it has to do with the multiline command on line ~348 | 14:43 |
| clarkb | fungi: the upstream fix doesn't modify that and instead updates the YUM var I Think the upstream change is likely to work assuming this is the actual fix | 14:44 |
| clarkb | even the view syntax highlighting on nb01 marks that \ as not doing what we think it does but I don't understand why yet | 14:44 |
| fungi | yeah, i was trying to figure out why that line continuation was breaking things | 14:45 |
| fungi | but i agree that's the simplest explanation | 14:45 |
| fungi | either that or an embedded newline creeping into the version substitution there, which would be even more strange | 14:46 |
| clarkb | fungi: its trailing whitespace so the \ affects the space not the newline | 14:46 |
| fungi | aha! | 14:47 |
| fungi | wow that was subtle, i didn't even notice the space at the end | 14:47 |
| * fungi sighs | 14:47 | |
| clarkb | there is a tab then some spaces? I can only imagine the issue was in trying to edit that file in the container without and editor | 14:47 |
| clarkb | fungi: I copied it into my local editor which shows me that info | 14:47 |
| clarkb | /usr/local/lib/python3.9/site-packages/diskimage_builder/elements/yum-minimal/root.d/08-yum-chroot is the path in the container if we want to edit it | 14:48 |
| fungi | yeah, looks like it's a mix of hard tab and spaces after the \ | 14:48 |
| fungi | so just a dirty edit | 14:48 |
| clarkb | anyway I think at as long as setting --releasever is the fix the upstream change is fine | 14:50 |
| fungi | seems like there was also strong interest in a dib release to include the ifname revert which just merged, so we should probably aim for having one asap | 14:55 |
| fungi | though the centos-7 fix just failed two jobs in the gate pipeline | 14:57 |
| clarkb | yes ianw indicated a desire to release dib today | 14:58 |
| fungi | oh, only one of the failures is from our zuul though | 14:59 |
| fungi | looks like openSUSE-Leap-15.3.x86_64-NoCloud.qcow2 had an unexpected checksum | 15:01 |
| fungi | i'l retrieving the image and checksum myself to make sure there's no persistent issue with it | 15:12 |
| fungi | also it's a little silly that dib greps the checksum file rather than using sha256sum -c | 15:20 |
| fungi | but maybe there's a reason i'm not aware of | 15:20 |
| fungi | anyway, i can reproduce the problem. the openSUSE-Leap-15.3.x86_64-NoCloud.qcow2 i downloaded has sha256sum a15e62e9c936d47675b389614723cd05d29d1531469e03c53fed50e779ee74ab but the .sha256 file i downloaded next to it says it should be 62d0892f9a6ddb463ddd92674cc793190dfafffc50784b7f565e7eb28df10191 | 15:21 |
| clarkb | is that from our mirrors? | 15:22 |
| fungi | no, the dib jobs pull directly from https://download.opensuse.org/repositories/Cloud:/Images:/Leap_15.3/images/ | 15:22 |
| clarkb | fun | 15:22 |
| fungi | looks like the image was last updated 03-May-2022 12:08 and the .sha256 file for it 03-May-2022 12:47 so not a totally different day or anything | 15:24 |
| fungi | the metadata linked for those is also only about 22 minutes apart | 15:26 |
| clarkb | fungi: we can probably make the opensuse job non voting in dib | 15:32 |
| clarkb | the changes we're trying to alnd shouldn't affect opensuse | 15:32 |
| fungi | yeah, i'll push a change | 15:32 |
| *** Guest1122 is now known as diablo_rojo_phone | 15:37 | |
| fungi | clarkb: i think we need to disable opensuse testing in the functest job, as that's what's failing | 15:44 |
| clarkb | fungi: ah, I think you can just comment that out? the functest runs some scripts stuff iirc | 15:45 |
| clarkb | fungi: tests/run_functests.sh that seems to indicate that some of the opensuse tests may already be skipped? | 15:47 |
| clarkb | I wonder if we just need to add this particular test to that list near the top of the file | 15:47 |
| opendevreview | Jeremy Stanley proposed openstack/diskimage-builder master: yum-minimal: workaround missing $releasedir variable https://review.opendev.org/c/openstack/diskimage-builder/+/839840 | 15:48 |
| opendevreview | Jeremy Stanley proposed openstack/diskimage-builder master: Temporarily stop running OpenSUSE functtests https://review.opendev.org/c/openstack/diskimage-builder/+/840328 | 15:48 |
| opendevreview | Jeremy Stanley proposed openstack/diskimage-builder master: Revert "Temporarily stop running OpenSUSE functtests" https://review.opendev.org/c/openstack/diskimage-builder/+/840329 | 15:48 |
| *** dviroel is now known as dviroel|lunch | 15:53 | |
| clarkb | fungi: https://review.opendev.org/c/openstack/diskimage-builder/+/839307 is another change ianw called out as being good to land. I guess if we approve it now it will fail on the opensuse thing too. Maybe we should just approve the opensuse test update and then we can land these toehr changes? | 15:55 |
| *** marios|ruck is now known as marios|out | 15:57 | |
| clarkb | infra-root checking on gerrit with its new httpd log and the file is there and the content looks as I would expect it. One minor concern is that the filesize is almost 3x the sshd log file size. We have plenty of room to spare on the device and it will get compressed when rotated | 16:00 |
| clarkb | but we should keep an eye on it just to be sure it doesn't get out of control | 16:00 |
| opendevreview | Neil Hanlon proposed openstack/diskimage-builder master: Ensure passwd is installed on RH and derivatives https://review.opendev.org/c/openstack/diskimage-builder/+/840352 | 16:06 |
| opendevreview | Clark Boylan proposed opendev/base-jobs master: Add ubuntu jammy nodesets https://review.opendev.org/c/opendev/base-jobs/+/840355 | 16:31 |
| *** dviroel|lunch is now known as dviroel | 16:56 | |
| *** jpena is now known as jpena|off | 17:00 | |
| clarkb | couple of things I've noticed when trying to get zuul updated to running on jammy. PostgreSQL attempts to write a history file in your homedir which isn't a fatail failure if you get sudo wrong. Annoying but we can deal with that. The python_version var for ensure-python really should be a string because 3.1 and 3.10 are equivalent when interpreted as floats. So far I've only needed | 18:03 |
| clarkb | changes in the zuul repo's .zuul.yaml and test-setup.sh script to address things (oh and bindep updates) | 18:04 |
| clarkb | overall there aren't any major issues I've hit yet | 18:04 |
| fungi | testing my personal projects with latest prereleases of python always turns up fun incompatibilities in dependencies | 18:11 |
| fungi | though part of that is running tests with deprecation warnings as errors | 18:12 |
| clarkb | as projects add py3.10 testing they should convert all python_version values to strings | 18:13 |
| fungi | well, since we don't have 3.1 available, they're not going to be able to merge the added testing anyway (or aren't going to be able to merge anything after it's on, if that addition isn't gated by the project using it) | 18:18 |
| clarkb | yup it errors | 18:27 |
| fungi | i expect word will get around quickly that they need to add quotes | 18:31 |
| clarkb | the zuul py10 unittest job does pass as does the quickstart using our bullseye python3.10 images so thats all looking promising. I do have a new patchset to push once this one reports that should fix the postgres thing | 18:35 |
| opendevreview | Merged openstack/diskimage-builder master: Temporarily stop running OpenSUSE functtests https://review.opendev.org/c/openstack/diskimage-builder/+/840328 | 18:57 |
| ianw | fungi: sorry, yeah i logged some updates @ https://meetings.opendev.org/irclogs/%23opendev/%23opendev.2022-04-29.log.html#t2022-04-29T03:09:30 | 19:04 |
| fungi | thanks, no worries i think we got it figured out and your fix should (finally) merge at any moment | 19:04 |
| ianw | because i could only replicate in the prod environment, i hand edited it. i put a change in that worked with "vm centos-minimal" that i was trying by hand, then i went and got some lunch | 19:04 |
| ianw | this is how it "accidentally" got a working build out and uploaded | 19:05 |
| clarkb | ianw: it seems to have broken after that due to extra whitespace on nb01 | 19:05 |
| clarkb | but I think the change pushed to dib is fine | 19:05 |
| ianw | i probably introduced that error in the container after, but didn't test the builds any further | 19:05 |
| opendevreview | Merged openstack/diskimage-builder master: yum-minimal: workaround missing $releasedir variable https://review.opendev.org/c/openstack/diskimage-builder/+/839840 | 19:52 |
| fungi | yay! | 19:57 |
| ianw | so i think a dib release and nodepool bump will be good today to get centos7 at least back | 20:26 |
| clarkb | ++ | 20:29 |
| fungi | carrying over from #openstack-dib, if we're talking about having an automated process sign source uploads to the ppa builder for consumption by opendev infrastructure and jobs generally, then we probably want an opendev-specific signing key, though we could use the existing openstack signing process as a template to work from | 20:30 |
| fungi | part of what's kept me from making one yet is not being able to decide what e-mail address to associate with it | 20:31 |
| fungi | we don't have a general opendev.org mta | 20:31 |
| fungi | though maybe we could use the opendev-incident address, i suppose | 20:32 |
| ianw | that seems reasonable | 20:32 |
| fungi | er, service-incident i mean | 20:33 |
| ianw | in the changelog, it would have the email of the person who updated it anyway | 20:33 |
| clarkb | corvus: can https://review.opendev.org/c/opendev/system-config/+/726490 and https://review.opendev.org/c/opendev/system-config/+/726248/ be abandoned because we went with zuuld for the processes instead of 'zuul' ? | 20:38 |
| corvus | clarkb: i'm like 85% sure that's the case. | 20:40 |
| clarkb | corvus: thanks I'm pretty sure that is what happened. We basically renamed things for the service side so that job side zuul could continue to be zuul as 1000 | 20:44 |
| clarkb | fungi: for https://review.opendev.org/c/opendev/system-config/+/760770 is that one of the mailing lists that you are cleaning up? if so can you abandon it? | 20:44 |
| fungi | yep, done thanks! | 20:46 |
| fungi | https://peps.python.org/pep-0690/ is going to be interesting | 20:46 |
| fungi | meta says it improved startup time 40-70% depending on the nature of the script | 20:50 |
| fungi | main downside is that it may defer some import side-effects until the first object from that module is referenced by the caller | 20:51 |
| opendevreview | Merged openstack/diskimage-builder master: centos: avoid head pipe failure https://review.opendev.org/c/openstack/diskimage-builder/+/839307 | 21:04 |
| *** dviroel is now known as dviroel|out | 21:30 | |
| opendevreview | Clark Boylan proposed openstack/diskimage-builder master: Add Jammy functesting to dib https://review.opendev.org/c/openstack/diskimage-builder/+/840391 | 22:55 |
| opendevreview | Merged openstack/diskimage-builder master: containerfile: update test to jammy https://review.opendev.org/c/openstack/diskimage-builder/+/838981 | 22:56 |
| opendevreview | Clark Boylan proposed openstack/diskimage-builder master: Add Jammy functesting to dib https://review.opendev.org/c/openstack/diskimage-builder/+/840391 | 22:58 |
| ianw | error: remote unpack failed: error Missing tree 763aed130dae87d86a3cdbea3e0e82e6fd3f3746 | 23:05 |
| ianw | To ssh://review.opendev.org:29418/openstack/diskimage-builder.git | 23:05 |
| ianw | ! [remote rejected] 3.21.0 -> 3.21.0 (n/a (unpacker error)) | 23:05 |
| ianw | that's a new one ... | 23:06 |
| clarkb | ianw: thankfully a known one | 23:11 |
| clarkb | git review has a flag to address that that you can pass when pushing the tag (git review only does it for pushing changes) | 23:12 |
| * clarkb looks | 23:12 | |
| clarkb | ianw: `git push --no-thin` | 23:12 |
| clarkb | infra-root and diablo_rojo_phone how does https://etherpad.opendev.org/p/rdrjMFJuUe-uCjRgEjkr look for ethercalc shutdown notice? | 23:13 |
| ianw | oh, right, yeah, that | 23:13 |
| ianw | clarkb: thanks :) | 23:13 |
| ianw | i've pushed 3.21.0 now | 23:13 |
| opendevreview | Clark Boylan proposed openstack/project-config master: Use https with apt in our ubuntu image builds https://review.opendev.org/c/openstack/project-config/+/840392 | 23:32 |
| opendevreview | Clark Boylan proposed openstack/project-config master: Add Jammy arm64 images https://review.opendev.org/c/openstack/project-config/+/840393 | 23:32 |
| opendevreview | Clark Boylan proposed openstack/project-config master: Add Jammy arm64 test nodes https://review.opendev.org/c/openstack/project-config/+/840394 | 23:32 |
| clarkb | infra-root anyone know why we wouldn't use https with ubuntu image builds? I put that change first as it seemed like a good idea. Anyway that stack should get us jammy arm64 images and nodes | 23:32 |
| ianw | yeah i think that's just a hang-over from pre-LE days | 23:33 |
| clarkb | https://review.opendev.org/c/opendev/system-config/+/838948 is a not urgent clean up change from when we removed service (it removes the groups from system-config) | 23:42 |
| clarkb | fungi: I'm not sure your comment about the pacakges being signed is accurate since this is our mirrors | 23:58 |
| clarkb | I mean they are signed but we don't check the signatures? | 23:58 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!