Tuesday, 2022-05-03

clarkband the job completed successfully all looks good to me00:00
opendevreviewMerged opendev/system-config master: Clean up defunct OpenStack mailing lists  https://review.opendev.org/c/opendev/system-config/+/83999000:01
opendevreviewMerged opendev/system-config master: third-party CI: reminder to keep address current  https://review.opendev.org/c/opendev/system-config/+/84025100:01
clarkbI need to go prep dinner now. Thanks for the help getting this stuff in00:01
opendevreviewIan Wienand proposed opendev/base-jobs master: Add centos-9-stream-arm64 nodeset  https://review.opendev.org/c/opendev/base-jobs/+/84025500:11
ianwclarkb: ^ i think that was just what was missing00:12
opendevreviewMerged openstack/diskimage-builder master: Revert "Fallback to persistent netifs names with systemd"  https://review.opendev.org/c/openstack/diskimage-builder/+/83886300:26
opendevreviewMerged opendev/base-jobs master: Add centos-9-stream-arm64 nodeset  https://review.opendev.org/c/opendev/base-jobs/+/84025500:31
*** rlandy|rover|bbl is now known as rlandy|rover00:36
*** rlandy|rover is now known as rlandy|out00:39
opendevreviewBalendu Mouli Burla proposed openstack/project-config master: Add SRIOV FEC Operator app to StarlingX  https://review.opendev.org/c/openstack/project-config/+/84026303:58
ianwi'll do that gerrit restart in  a minute04:18
ianwRepoDigests": [04:44
ianw            "opendevorg/gerrit@sha256:ee41b17e44b83a3b22632aa7d688b6c6a6f80d309a430bc9ccad60f6a6b4e6e9"04:44
ianw        ],04:44
ianwwhich matches https://hub.docker.com/layers/opendevorg/gerrit/3.4/images/sha256-ee41b17e44b83a3b22632aa7d688b6c6a6f80d309a430bc9ccad60f6a6b4e6e9?context=explore04:44
ianwthat matches @ https://zuul.opendev.org/t/openstack/build/82bea43f57f0498a91ee98bc7e57a111/log/job-output.txt#766304:52
ianwthe other two changes are config changes, which have applied04:52
ianwok, we're back04:56
ianw#status log restarted gerrit to pickup changes deployed with stack @ https://review.opendev.org/c/opendev/system-config/+/839251/04:57
opendevstatusianw: finished logging04:57
*** marios is now known as marios|ruck05:08
ianwinfra-root: if we can get https://review.opendev.org/c/opendev/system-config/+/839841 in before the wheel run, i think we'll finally be back to it06:34
ianwthat will fix up the publishing steps on 9-stream.  06:34
fricklerianw: +2, not sure when the wheel builds run, are those in the daily periodic pipeline? then we'd have time to wait for another review07:28
fricklerdoes anyone intend to look at the centos-7 builds or should we just stop them for now?07:28
*** jpena|off is now known as jpena07:34
*** rlandy|out is now known as rlandy10:24
*** dviroel|out is now known as dviroel11:18
fungidid someone start looking into it yet? pip list in the builder container on nb01 says dib 3.20.3 is installed, but the line numbers in the failing logs don't seem to correspond to the commands on those lines11:47
fungiin particular, this call:11:48
fungi+ /opt/dib_tmp/dib_build.qOsibSQN/hooks/root.d/08-yum-chroot:main:348              :   _run_chroot yum -y --releasever=7 ''11:48
fungiin 3.20.3 that line is a code comment:11:49
fungiit's almost as if the element has been hand-patched in place on the builder11:50
fungior something went wrong with the dib install when the container was built11:50
opendevreviewMerged opendev/system-config master: Test openafs roles on CentOS 9-stream  https://review.opendev.org/c/opendev/system-config/+/83984111:59
fungiyeah, i've grabbed a copy of the 08-yum-chroot being used on nb01 and it doesn't seem to correspond to the state of 3.20.3 or master12:05
fungithe only change to merge altering that file since 3.20.3 was https://review.opendev.org/837248 and the file being used on the builder looks similar to that but with two lines changed in the vicinity of the failure we're logging12:07
fungiit adds a --releasever=${DIB_RELEASE/-*/} to the _run_chroot invocation, which can be seen in the logged command12:11
fungiadds it to the yum update call on the previous line as well, but that seems to be working12:11
fungifrickler: per discussion in #openstack-dib, it looks like https://review.opendev.org/839840 is a variant of that and ianw seems to think it might fix the issue12:13
fungii guess ianw may have altered the 08-yum-chroot on nb01 in order to test some theories12:14
fungiwhich explains the confusing state with it now12:14
*** rlandy is now known as rlandy|mtg13:05
amorinI received an highligh from yesterday, anything wrong on OVH cloud?13:06
fungiamorin: looks like we were getting some nova.exception.OrphanedObjectError errors like this in gra1 "Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible."13:11
fungithat's when calling the /servers/detail api method from our cleanup loop13:14
fungiseems to still be going on13:14
fungiamorin: i don't think it's impacting much if anything on our side, more that we wanted to let you know13:14
*** artom__ is now known as artom13:21
amorinack, ok13:41
amorindo you have any nova instance ID maybe?13:41
clarkbamorin: no this appears to happen when doing a detailed servers listing when we looked for leaked instances13:53
clarkbamorin: we aren't querying a specific instance but instead GETing /servers/detail which is meant to give us a list of instanes iirc13:53
amorinon GRA1?13:53
clarkbamorin: correct13:53
clarkbfungi: yes ianw hand patched it then pushed that fix you found. The plan currently aiui is to do a release ~today to fix it properly14:00
fungiwell, the hand patched version is still failing according to the centos-7 image build logs fromnb0114:01
*** rlandy|mtg is now known as rlandy14:02
fungiand is also implemented a bit differently from the change we reviewed14:34
clarkblooking at the change we reviewed I think it avoids this problem14:37
clarkbbut I'm trying to find the content in the container now to compare properly14:37
fungiclarkb: i copied it to my homedir, snagged from an ephemeral build dir14:41
fungiclarkb: nb01:~fungi/08-yum-chroot14:41
fungithat's what it's using now14:42
clarkbfungi: it has to do with the multiline command on line ~34814:43
clarkbfungi: the upstream fix doesn't modify that and instead updates the YUM var I Think the upstream change is likely to work assuming this is the actual fix14:44
clarkbeven the view syntax highlighting on nb01 marks that \ as not doing what we think it does but I don't understand why yet14:44
fungiyeah, i was trying to figure out why that line continuation was breaking things14:45
fungibut i agree that's the simplest explanation14:45
fungieither that or an embedded newline creeping into the version substitution there, which would be even more strange14:46
clarkbfungi: its trailing whitespace so the \ affects the space not the newline14:46
fungiwow that was subtle, i didn't even notice the space at the end14:47
* fungi sighs14:47
clarkbthere is a tab then some spaces? I can only imagine the issue was in trying to edit that file in the container without and editor14:47
clarkbfungi: I copied it into my local editor which shows me that info14:47
clarkb/usr/local/lib/python3.9/site-packages/diskimage_builder/elements/yum-minimal/root.d/08-yum-chroot is the path in the container if we want to edit it14:48
fungiyeah, looks like it's a mix of hard tab and spaces after the \14:48
fungiso just a dirty edit14:48
clarkbanyway I think at as long as setting --releasever is the fix the upstream change is fine14:50
fungiseems like there was also strong interest in a dib release to include the ifname revert which just merged, so we should probably aim for having one asap14:55
fungithough the centos-7 fix just failed two jobs in the gate pipeline14:57
clarkbyes ianw indicated a desire to release dib today14:58
fungioh, only one of the failures is from our zuul though14:59
fungilooks like openSUSE-Leap-15.3.x86_64-NoCloud.qcow2 had an unexpected checksum15:01
fungii'l retrieving the image and checksum myself to make sure there's no persistent issue with it15:12
fungialso it's a little silly that dib greps the checksum file rather than using sha256sum -c15:20
fungibut maybe there's a reason i'm not aware of15:20
fungianyway, i can reproduce the problem. the openSUSE-Leap-15.3.x86_64-NoCloud.qcow2 i downloaded has sha256sum a15e62e9c936d47675b389614723cd05d29d1531469e03c53fed50e779ee74ab but the .sha256 file i downloaded next to it says it should be 62d0892f9a6ddb463ddd92674cc793190dfafffc50784b7f565e7eb28df1019115:21
clarkbis that from our mirrors?15:22
fungino, the dib jobs pull directly from https://download.opensuse.org/repositories/Cloud:/Images:/Leap_15.3/images/15:22
fungilooks like the image was last updated 03-May-2022 12:08 and the .sha256 file for it 03-May-2022 12:47 so not a totally different day or anything15:24
fungithe metadata linked for those is also only about 22 minutes apart15:26
clarkbfungi: we can probably make the opensuse job non voting in dib15:32
clarkbthe changes we're trying to alnd shouldn't affect opensuse15:32
fungiyeah, i'll push a change15:32
*** Guest1122 is now known as diablo_rojo_phone15:37
fungiclarkb: i think we need to disable opensuse testing in the functest job, as that's what's failing15:44
clarkbfungi: ah, I think you can just comment that out? the functest runs some scripts stuff iirc15:45
clarkbfungi: tests/run_functests.sh that seems to indicate that some of the opensuse tests may already be skipped?15:47
clarkbI wonder if we just need to add this particular test to that list near the top of the file15:47
opendevreviewJeremy Stanley proposed openstack/diskimage-builder master: yum-minimal: workaround missing $releasedir variable  https://review.opendev.org/c/openstack/diskimage-builder/+/83984015:48
opendevreviewJeremy Stanley proposed openstack/diskimage-builder master: Temporarily stop running OpenSUSE functtests  https://review.opendev.org/c/openstack/diskimage-builder/+/84032815:48
opendevreviewJeremy Stanley proposed openstack/diskimage-builder master: Revert "Temporarily stop running OpenSUSE functtests"  https://review.opendev.org/c/openstack/diskimage-builder/+/84032915:48
*** dviroel is now known as dviroel|lunch15:53
clarkbfungi: https://review.opendev.org/c/openstack/diskimage-builder/+/839307 is another change ianw called out as being good to land. I guess if we approve it now it will fail on the opensuse thing too. Maybe we should just approve the opensuse test update and then we can land these toehr changes?15:55
*** marios|ruck is now known as marios|out15:57
clarkbinfra-root checking on gerrit with its new httpd log and the file is there and the content looks as I would expect it. One minor concern is that the filesize is almost 3x the sshd log file size. We have plenty of room to spare on the device and it will get compressed when rotated16:00
clarkbbut we should keep an eye on it just to be sure it doesn't get out of control16:00
opendevreviewNeil Hanlon proposed openstack/diskimage-builder master: Ensure passwd is installed on RH and derivatives  https://review.opendev.org/c/openstack/diskimage-builder/+/84035216:06
opendevreviewClark Boylan proposed opendev/base-jobs master: Add ubuntu jammy nodesets  https://review.opendev.org/c/opendev/base-jobs/+/84035516:31
*** dviroel|lunch is now known as dviroel16:56
*** jpena is now known as jpena|off17:00
clarkbcouple of things I've noticed when trying to get zuul updated to running on jammy. PostgreSQL attempts to write a history file in your homedir which isn't a fatail failure if you get sudo wrong. Annoying but we can deal with that. The python_version var for ensure-python really should be a string because 3.1 and 3.10 are equivalent when interpreted as floats. So far I've only needed18:03
clarkbchanges in the zuul repo's .zuul.yaml and test-setup.sh script to address things (oh and bindep updates)18:04
clarkboverall there aren't any major issues I've hit yet18:04
fungitesting my personal projects with latest prereleases of python always turns up fun incompatibilities in dependencies18:11
fungithough part of that is running tests with deprecation warnings as errors18:12
clarkbas projects add py3.10 testing they should convert all python_version values to strings18:13
fungiwell, since we don't have 3.1 available, they're not going to be able to merge the added testing anyway (or aren't going to be able to merge anything after it's on, if that addition isn't gated by the project using it)18:18
clarkbyup it errors18:27
fungii expect word will get around quickly that they need to add quotes18:31
clarkbthe zuul py10 unittest job does pass as does the quickstart using our bullseye python3.10 images so thats all looking promising. I do have a new patchset to push once this one reports that should fix the postgres thing18:35
opendevreviewMerged openstack/diskimage-builder master: Temporarily stop running OpenSUSE functtests  https://review.opendev.org/c/openstack/diskimage-builder/+/84032818:57
ianwfungi: sorry, yeah i logged some updates @ https://meetings.opendev.org/irclogs/%23opendev/%23opendev.2022-04-29.log.html#t2022-04-29T03:09:3019:04
fungithanks, no worries i think we got it figured out and your fix should (finally) merge at any moment19:04
ianwbecause i could only replicate in the prod environment, i hand edited it.  i put a change in that worked with "vm centos-minimal" that i was trying by hand, then i went and got some lunch19:04
ianwthis is how it "accidentally" got a working build out and uploaded19:05
clarkbianw: it seems to have broken after that due to extra whitespace on nb0119:05
clarkbbut I think the change pushed to dib is fine19:05
ianwi probably introduced that error in the container after, but didn't test the builds any further19:05
opendevreviewMerged openstack/diskimage-builder master: yum-minimal: workaround missing $releasedir variable  https://review.opendev.org/c/openstack/diskimage-builder/+/83984019:52
ianwso i think a dib release and nodepool bump will be good today to get centos7 at least back20:26
fungicarrying over from #openstack-dib, if we're talking about having an automated process sign source uploads to the ppa builder for consumption by opendev infrastructure and jobs generally, then we probably want an opendev-specific signing key, though we could use the existing openstack signing process as a template to work from20:30
fungipart of what's kept me from making one yet is not being able to decide what e-mail address to associate with it20:31
fungiwe don't have a general opendev.org mta20:31
fungithough maybe we could use the opendev-incident address, i suppose20:32
ianwthat seems reasonable20:32
fungier, service-incident i mean20:33
ianwin the changelog, it would have the email of the person who updated it anyway20:33
clarkbcorvus: can https://review.opendev.org/c/opendev/system-config/+/726490 and https://review.opendev.org/c/opendev/system-config/+/726248/ be abandoned because we went with zuuld for the processes instead of 'zuul' ?20:38
corvusclarkb: i'm like 85% sure that's the case.20:40
clarkbcorvus: thanks I'm pretty sure that is what happened. We basically renamed things for the service side so that job side zuul could continue to be zuul as 100020:44
clarkbfungi: for https://review.opendev.org/c/opendev/system-config/+/760770 is that one of the mailing lists that you are cleaning up? if so can you abandon it?20:44
fungiyep, done thanks!20:46
fungihttps://peps.python.org/pep-0690/ is going to be interesting20:46
fungimeta says it improved startup time 40-70% depending on the nature of the script20:50
fungimain downside is that it may defer some import side-effects until the first object from that module is referenced by the caller20:51
opendevreviewMerged openstack/diskimage-builder master: centos: avoid head pipe failure  https://review.opendev.org/c/openstack/diskimage-builder/+/83930721:04
*** dviroel is now known as dviroel|out21:30
opendevreviewClark Boylan proposed openstack/diskimage-builder master: Add Jammy functesting to dib  https://review.opendev.org/c/openstack/diskimage-builder/+/84039122:55
opendevreviewMerged openstack/diskimage-builder master: containerfile: update test to jammy  https://review.opendev.org/c/openstack/diskimage-builder/+/83898122:56
opendevreviewClark Boylan proposed openstack/diskimage-builder master: Add Jammy functesting to dib  https://review.opendev.org/c/openstack/diskimage-builder/+/84039122:58
ianwerror: remote unpack failed: error Missing tree 763aed130dae87d86a3cdbea3e0e82e6fd3f374623:05
ianwTo ssh://review.opendev.org:29418/openstack/diskimage-builder.git23:05
ianw ! [remote rejected]   3.21.0 -> 3.21.0 (n/a (unpacker error))23:05
ianwthat's a new one ...23:06
clarkbianw: thankfully a known one23:11
clarkbgit review has a flag to address that that you can pass when pushing the tag (git review only does it for pushing changes)23:12
* clarkb looks23:12
clarkbianw: `git push --no-thin`23:12
clarkbinfra-root and diablo_rojo_phone how does https://etherpad.opendev.org/p/rdrjMFJuUe-uCjRgEjkr look for ethercalc shutdown notice?23:13
ianwoh, right, yeah, that23:13
ianwclarkb: thanks :)23:13
ianwi've pushed 3.21.0 now23:13
opendevreviewClark Boylan proposed openstack/project-config master: Use https with apt in our ubuntu image builds  https://review.opendev.org/c/openstack/project-config/+/84039223:32
opendevreviewClark Boylan proposed openstack/project-config master: Add Jammy arm64 images  https://review.opendev.org/c/openstack/project-config/+/84039323:32
opendevreviewClark Boylan proposed openstack/project-config master: Add Jammy arm64 test nodes  https://review.opendev.org/c/openstack/project-config/+/84039423:32
clarkbinfra-root anyone know why we wouldn't use https with ubuntu image builds? I put that change first as it seemed like a good idea. Anyway that stack should get us jammy arm64 images and nodes23:32
ianwyeah i think that's just a hang-over from pre-LE days23:33
clarkbhttps://review.opendev.org/c/opendev/system-config/+/838948 is a not urgent clean up change from when we removed service (it removes the groups from system-config)23:42
clarkbfungi: I'm not sure your comment about the pacakges being signed is accurate since this is our mirrors23:58
clarkbI mean they are signed but we don't check the signatures?23:58

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!