Tuesday, 2023-01-10

opendevreviewIan Wienand proposed opendev/system-config master: launch: permit ssh root login on base image  https://review.opendev.org/c/opendev/system-config/+/86961500:27
ianwUnable to enable service systemd-timesyncd: Failed to enable unit: Unit file /etc/systemd/system/systemd-timesyncd.service is masked.00:32
ianwmaybe this image is too opinionated ... geez this feels like the old days of snapshot images00:34
*** rlandy is now known as rlandy|out00:38
opendevreviewMerged openstack/diskimage-builder master: Clean up tox.ini for tox v4  https://review.opendev.org/c/openstack/diskimage-builder/+/86957900:47
*** JasonF is now known as JayF02:17
ianwok, i went back to making our own image.  converting it to raw and copying some properties seeme to work it out02:55
ianwthe host keys still don't display correct in the launch node output02:55
opendevreviewIan Wienand proposed opendev/system-config master: Add nb04.opendev.org  https://review.opendev.org/c/opendev/system-config/+/86962204:08
*** yadnesh|away is now known as yadnesh04:11
opendevreviewIan Wienand proposed opendev/system-config master: doc/nodepool: update vhd-util docs  https://review.opendev.org/c/opendev/system-config/+/86962304:13
opendevreviewIan Wienand proposed opendev/zone-opendev.org master: Add nb04.opendev.org  https://review.opendev.org/c/opendev/zone-opendev.org/+/86962604:23
*** ysandeep is now known as ysandeep|ruck04:39
*** ysandeep|ruck is now known as ysandeep|ruck|brb05:41
*** pojadhav- is now known as pojadhav05:51
ianwok, nb04 should be ready.  it's up, and got an 800gb volume attached, which is the same as in linaro05:53
*** marios is now known as marios|rover06:04
*** ysandeep|ruck|brb is now known as ysandeep|ruck06:04
*** ysandeep|ruck is now known as ysandeep|ruck|brb06:49
*** ysandeep|ruck|brb is now known as ysandeep|ruck06:59
*** jpena|off is now known as jpena08:20
opendevreviewMerged opendev/base-jobs master: Fix tox.ini for tox v4  https://review.opendev.org/c/opendev/base-jobs/+/86958209:39
*** ysandeep|ruck is now known as ysandeep|afk10:11
*** rlandy|out is now known as rlandy11:13
*** ysandeep|afk is now known as ysandeep|ruck11:31
*** bhagyashris is now known as bhagyashris|brb13:36
opendevreviewLuke Odom proposed openstack/diskimage-builder master: Map curl to curl-minimal for rocky 9  https://review.opendev.org/c/openstack/diskimage-builder/+/86942413:59
opendevreviewLuke Odom proposed openstack/diskimage-builder master: Add swap support  https://review.opendev.org/c/openstack/diskimage-builder/+/86927013:59
*** dasm|off is now known as dasm14:00
dpawlikClark[m], fungi: hey, I would like to make a version e.g. 1.0.0 or 0.1.0 of logscraper. I check few docs related to Openstack release, but can not find proper information about versioning such projects. Should I create a PS similar to https://review.opendev.org/c/openstack/releases/+/607010 or just make a tag, push it to openstack/ci-log-processing14:02
dpawlikproject and then add the project there https://github.com/openstack/releases/blob/master/tools/build_tag_history.sh#L21-L57 or do it earlier?14:02
fungidpawlik: sig repositories and independent deliverables don't use the coordinated openstack release process, their maintainers just push signed git tags directly to gerrit14:04
dpawlikfungi: ack. Should I also add entry to build_tag_history.sh or not needed?14:07
funginot needed14:08
fungiwe do need to check the acl to make sure you have signed tag push access to the repo, but i'm in a meeting at the moment. can check after14:09
dpawlikfungi: ok. No rush14:12
fungidpawlik: is it the openstack/ci-log-processing repository?14:17
fungidpawlik: propose a change to https://opendev.org/openstack/project-config/src/branch/master/gerrit/acls/openstack/ci-log-processing.config which adds a section like the one described at https://docs.opendev.org/opendev/infra-manual/latest/creators.html#creation-of-tags and use ci-log-processing-release as the group name14:20
fungii'll approve it and add you to the group once it deploys14:21
fungidpawlik: also you'll probably want to at least look over https://docs.opendev.org/opendev/infra-manual/latest/creators.html#prepare-an-initial-release and https://docs.opendev.org/opendev/infra-manual/latest/drivers.html#tagging-a-release14:22
opendevreviewdaniel.pawlik proposed openstack/project-config master: Add createSignedTag permissions for openstack/ci-log-processing project  https://review.opendev.org/c/openstack/project-config/+/86972714:29
dpawlikack fungi. Thanks for help14:29
opendevreviewMerged openstack/project-config master: Add createSignedTag permissions for openstack/ci-log-processing project  https://review.opendev.org/c/openstack/project-config/+/86972714:52
fungidpawlik: i've added your gerrit account as the initial member of ci-log-processing-release15:13
dpawlikcool, thank you fungi15:19
dpawlikwe will merge few changes, apply on the production and if all is fine for a week/2 weeks I will do a release15:20
fungisounds good15:20
dpawlikI see that cluster is not so full now so maybe after upgrading rdo/sf infra I will send an email to opendev mailing list and schedule Opensearch upgrade to newer version 15:22
dpawlikthey have automated process on AWS, but the cluster will be not reachable during that time15:22
*** ysandeep|ruck is now known as ysandeep|out15:24
clarkbfungi: frickler  have time for https://review.opendev.org/c/opendev/base-jobs/+/869580 ?15:57
fricklerclarkb: ack, I had that open earlier today but then got distracted16:06
opendevreviewMerged openstack/project-config master: Add the "api-ref-jobs" template to CloudKitty  https://review.opendev.org/c/openstack/project-config/+/86765116:16
opendevreviewMerged opendev/base-jobs master: Add opendev nox docs promotion  https://review.opendev.org/c/opendev/base-jobs/+/86958016:18
*** marios|rover is now known as marios|out16:46
opendevreviewClark Boylan proposed opendev/bindep master: Use nox  https://review.opendev.org/c/opendev/bindep/+/86800417:05
*** jpena is now known as jpena|off17:24
fungiclarkb: was https://goharbor.io/docs/2.7.0/administration/configure-proxy-cache/ one of the options we looked at for container image caching?17:48
Clark[m]I don't think so. We primary looked at using the docker registry as it documents the use case but then has no method of pruning.18:01
Clark[m]Lack of pruning appears to be a fatal flaw in basically every container images registry 18:01
fungistorage is free and infinitely available, right?18:02
Clark[m]That one says "log in to the web UI to start garbage collection"... And garbage collection isn't quite right either for a cache18:02
fungimaybe the web ui is backed by a rest api at least18:04
Clark[m]It also has quotas but they are project level not registry level. Really as far as I can tell no one has really implemented a lru cache for docker images on finite disk space18:05
fungiapparently it's the registry that the sovereign cloud stack distribution ships/uses, just couldn't remember discussing that one previously18:05
Clark[m]Anymore it seems like caches like that are not considered worth supporting. Even pypi is going to kill our ability to cache properly18:07
fungiwe just need them to drop a cdn endpoint in each of our donor cloud regions ;)18:08
clarkbfungi the parent change for the bindep nox change needs review too https://review.opendev.org/c/opendev/bindep/+/868003/ that change fixes an issue with updated deps on rolling release distros18:47
fungioh, yep18:47
fungithat one lgtm too18:48
opendevreviewGustavo Sanchez proposed openstack/project-config master: Add the woodpecker charm to Openstack charms  https://review.opendev.org/c/openstack/project-config/+/86975118:58
fungi#status log Restarted services on lists.openstack.org since some mailman processes were terminated earlier today by out-of-memory events19:24
opendevstatusfungi: finished logging19:24
*** mtomaska__ is now known as mtomaska19:27
*** rlandy is now known as rlandy|brb19:37
clarkbianw: did catch a small issue on https://review.opendev.org/c/openstack/project-config/+/868443/219:47
ianwthanks, yep that's a typo.  will fix19:48
*** rlandy|brb is now known as rlandy20:01
opendevreviewMerged opendev/zone-opendev.org master: Add nb04.opendev.org  https://review.opendev.org/c/opendev/zone-opendev.org/+/86962620:09
opendevreviewJoshua Watt proposed zuul/zuul-jobs master: use-buildset-registry: Respect docker_mirror  https://review.opendev.org/c/zuul/zuul-jobs/+/86976020:15
opendevreviewJoshua Watt proposed zuul/zuul-jobs master: use-docker-mirror: Also run task if docker_mirror is defined  https://review.opendev.org/c/zuul/zuul-jobs/+/86976120:16
opendevreviewMerged opendev/system-config master: Drop openEuler 20.03 LTS SP2 repo mirror  https://review.opendev.org/c/opendev/system-config/+/84879620:34
opendevreviewJoshua Watt proposed zuul/zuul-jobs master: use-buildset-registry: Prepend buildset registry to mirrors  https://review.opendev.org/c/zuul/zuul-jobs/+/86976020:36
opendevreviewClark Boylan proposed opendev/system-config master: Remove Gerrit 3.5 images  https://review.opendev.org/c/opendev/system-config/+/86976321:02
opendevreviewClark Boylan proposed opendev/system-config master: Convert Gerrit images to python3.10  https://review.opendev.org/c/opendev/system-config/+/86976421:02
opendevreviewClark Boylan proposed opendev/system-config master: Add Gerrit 3.7 images  https://review.opendev.org/c/opendev/system-config/+/86976521:02
opendevreviewClark Boylan proposed openstack/project-config master: Update jeepyb gerrit image build deps  https://review.opendev.org/c/openstack/project-config/+/86976621:03
clarkbI split things up a bit as I'm unsure how painful getting 3.7 going will be but the other bits can happen before that pretty easily21:03
ianwi guess we should add a nb04 nodepool config before adding the host to the inventory, not sure if the deployment will work without a config file21:08
opendevreviewIan Wienand proposed openstack/project-config master: Add nb04 config  https://review.opendev.org/c/openstack/project-config/+/86976921:26
ianwdid something change in openstack-zuul-jobs-linters?21:53
opendevreviewIan Wienand proposed openstack/project-config master: Add nb04 config  https://review.opendev.org/c/openstack/project-config/+/86976921:57
opendevreviewIan Wienand proposed openstack/project-config master: openstack-afs.yaml : correct indentation  https://review.opendev.org/c/openstack/project-config/+/86977221:57
clarkbianw: nothing has changed with that job that i know of. However tox releases happen regularly now and often break things21:58
ianwi think it appears to have found an indentation that doesn't seem standard in project-config.  but wondering why it just found it21:59
ianwlast timeit passed it ran with ansible-lint 6.4.022:04
ianwyamllint 1.29.0 released about 4 hours ago ... so that would be it22:05
ianwopendev-buildset-registry -- post_failure22:08
ianwso one problem fixed, now another :)22:08
ianwhttps://zuul.opendev.org/t/openstack/build/c3af3434793649e69027628277f9b5fc doesn't help22:09
ianwhttps://zuul.opendev.org/t/openstack/builds?job_name=opendev-buildset-registry&project=openstack/project-config ... ran ok half hour ago ...22:09
clarkbI just rechecked a change that didn't collect logs due to the host becoming unreachable after a failre22:10
clarkbI'm going to try and watch the console live just in case22:10
clarkbmaybe something more systemic?22:10
clarkboh wait the unreachable is for the cleanup phase trying to run df22:11
clarkbit says there were no upload failures22:11
clarkber when I grep the uuid on hte executor that ran the job22:12
clarkbsuccessful jobs don't have logs either22:12
clarkbcorvus: ^22:12
clarkboh its an ovh outage22:12
clarkbI figured that out by trying to click on a site preview link and got a 503 from ovh22:12
clarkbI'm guessing if I pull up the console log for my browser when trying to fetch the manifest file I'll get similar22:13
clarkbthe browser says cors doesn't allow this22:13
fungimakes sense22:14
clarkbexcept we explicitly set cors options on the swift uploads to allow this...22:14
clarkbso ya I'm not sure what the next step for debugging is22:15
clarkbunless maybe openstacksdk just updated on the executors somehow (we last updated them friday/saturdayish and I think logs were working yesterday just fine)22:15
timburkeif swift's sending back 503s, i wouldn't expect any CORS info (whether stored on containers or objects) to make it back to the proxy, much less out to the client22:16
fungiugh, the exim queue on lists.openstack.org is currently around 220k messages (i think, i had to check on the filesystem because mailq just hangs)22:17
fungiand it's growing, not shrinking. that's not good22:17
clarkbtimburke: oh i see the 503 leads to no cors headers so the issue is still liekly whatever causes the 503. that makes sense22:18
clarkbin that case I think we can observe for a bit since in theorythese objects are getting uploaded successfully it is just the retrieval that fails22:18
ianwclarkb: did the change you mention about fail POST_FAILURE?22:19
clarkbianw: no it failed properly. Its possible yours is different22:19
clarkbfungi: is that for mail coming in or out?22:20
clarkb(I'm assuming out)22:20
ianwwhat's the best way to turn a job id/event id -> the executor where it ran?22:20
clarkbianw: grep the build uuid in /var/log/zuul/executor-debug.log across all twelve executors22:20
clarkbianw: if you get the logs back it tells you what executor it is, unfortunately in this case we don't have the logs so have to brute force it22:20
ianwclarkb: ok, cool, so "what i was doing" is still the best way :)22:21
clarkbERROR: Could not find a version that satisfies the requirement PyYAML>=3.1.0 (from versions: none) <- the legit error in my build22:22
ianwaction: zuul_swift_upload22:24
ianwfailed: true22:24
ianwbut it's no_log so ... all i know is it didn't work22:24
clarkbianw: it should log where it was trying to upload too22:24
clarkbpossible the errors are not just in retrieving objects but also uploding them? was it ovh (gra is where i saw the issues specifically)22:24
ianw"_swift_provider_name": "ovh_gra"22:25
clarkbok we might need to pull that region out of the list then now that we have evidence of it breaking uploads22:26
ianwit's hard to pinpoint but https://zuul.opendev.org/t/openstack/builds?result=POST_FAILURE&skip=0 shows an uptick ~20:00UTC?22:29
ianw(this is probably accessible via statsd)22:29
ianwif https://public-cloud.status-ovhcloud.com/ is the right thing, afaics it's not showing any failures22:31
clarkbspot checking I haven't found any evidence of bhs1 errors yet. I'll push a change just to disbale gra22:32
opendevreviewClark Boylan proposed opendev/base-jobs master: Disable OVH GRA1 log uploads  https://review.opendev.org/c/opendev/base-jobs/+/86977522:32
clarkbinfra-root ^22:33
ianwif it fails i guess force merge?22:34
corvushopefully we don't have a linter that disallows "#-"  :)22:35
clarkbit did fail on a post failure. I've rechecked it and if tha doesn't work I'll see about force merging it22:38
clarkbhows this look #status notice One of our CI job log storage providers appears to be having trouble with log uploads and retrievals. We are in the process of removing that provider from the pool.22:40
fungilgtm, thanks!22:41
clarkb#status notice One of our CI job log storage providers appears to be having trouble with log uploads and retrievals. We are in the process of removing that provider from the pool.22:43
opendevstatusclarkb: sending notice22:43
-opendevstatus- NOTICE: One of our CI job log storage providers appears to be having trouble with log uploads and retrievals. We are in the process of removing that provider from the pool.22:43
clarkbok I understand the pyyaml thing. The issue is pyyaml makes python3.9 and python3.10 specific wheels. It doesn't do python3 abi. Bullseye python is 3.9 which is why that all worked before but now trying to chnge the base image to 3.10 isn't working because the java side image that is 3.10 doesn't see the wheel as valid?22:43
clarkbya that is exactly it. We use the openjdk 11 upstream image22:45
clarkbok no python3.10 for now22:45
opendevstatusclarkb: finished sending notice22:46
clarkbone of the jobs for the base-jobs gating is trying to uploda to bhs1 and taking longer than I expect22:46
clarkbso I might remove bhs1 too then force merge22:46
clarkbyup it just failed against bhs1. New ps22:46
JayFhttps://review.opendev.org/c/openstack/ironic-python-agent/+/867915 any suggestions for why this might not be cooperating and merging? The patch it depends-on has landed... I did W-1/W+1 to try and knock it loose, unsuccessfully22:47
opendevreviewClark Boylan proposed opendev/base-jobs master: Disable OVH BHS1 and GRA1 log uploads  https://review.opendev.org/c/opendev/base-jobs/+/86977522:47
JayFwould strongly prefer not having to modify the patch but I can do that if it's the only option :/ 22:47
opendevreviewMerged opendev/base-jobs master: Disable OVH BHS1 and GRA1 log uploads  https://review.opendev.org/c/opendev/base-jobs/+/86977522:50
clarkbok force merge successful22:50
clarkbJayF: give us a few to dig ourselves out of the swift provider problem and I can probably take a look22:51
JayFOn an unrelated item; if someone is available I need https://review.opendev.org/admin/groups/835647ed8ebcc92ae0bdcfcb1b25adba02d972b1 to be seeded with me so I can populate the group membership 22:51
JayFclarkb: no rush at all on either of these22:51
ianwclarkb: we probably need to debug it incase its an expired token or something that won't resolve itself?22:51
clarkbianw: ya maybe. Its never been that in the past but it is theoretically possible22:52
ianwJayF: i think https://review.opendev.org/c/openstack/ironic-python-agent/+/868065/1 needs to merge?22:53
fungiJayF: i think the change you want to merge has a parent which isn't approved yet? https://review.opendev.org/c/openstack/ironic-python-agent/+/86806522:53
JayFthis is a triple jinx, because I think I figured that out right as you were telling me 22:53
clarkbJayF: I went ahead and added you but note that TheJulia or dtantsur could have added you too22:53
JayFclarkb: oh, I didn't realise they were already members or I wouldn't have bugged you, that's my bad22:54
fungiyeah, also iurygregory and rpittau are in it22:55
clarkbfungi: I don't see them22:55
clarkboh they are there now wtf22:55
clarkbmaybe jayf just added them22:55
fungiyeah, possible i pulled it up after he started editing22:56
JayFI just added them, absolutely22:56
JayFwe had an agreement at the meeting about who goes into this group :D 22:56
clarkbianw: so the thing that makes me think we aren't at fault is the 503s tring to get files back that were supposedly uploaded successfully22:57
clarkbianw: if it was purely upload failures then I would be more worried about something on our end being at fault. But we have at least one job where it uplaoded just fine according to apis then refused to return the results22:57
opendevreviewClark Boylan proposed opendev/system-config master: Add Gerrit 3.7 images  https://review.opendev.org/c/opendev/system-config/+/86976522:58
ianwtrue ... there's probably not a quick way to test22:59
clarkbwe can modify the base-test job to only talk to ovh22:59
ianwoh, of course, yeah so we should do that to confirm operation before re-enabling23:03
clarkbalso amorin may have thoughts when the EU day starts again23:07
*** rlandy is now known as rlandy|out23:25
opendevreviewIan Wienand proposed opendev/system-config master: [wip] add variable to block UA's for mailman  https://review.opendev.org/c/opendev/system-config/+/86977923:38

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!