| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Add Rocky 8/9 builds https://review.opendev.org/c/opendev/zuul-providers/+/949696 | 04:45 |
|---|---|---|
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Add Rocky 8/9 builds https://review.opendev.org/c/opendev/zuul-providers/+/949696 | 04:48 |
| opendevreview | Michal Nasiadka proposed zuul/zuul-jobs master: ensure-dib: Add podman https://review.opendev.org/c/zuul/zuul-jobs/+/949697 | 04:50 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Add Rocky 8/9 builds https://review.opendev.org/c/opendev/zuul-providers/+/949696 | 04:50 |
| opendevreview | Michal Nasiadka proposed zuul/zuul-jobs master: ensure-dib: Add newuidmap and podman https://review.opendev.org/c/zuul/zuul-jobs/+/949697 | 04:58 |
| opendevreview | Michal Nasiadka proposed zuul/zuul-jobs master: ensure-dib: Add newuidmap and podman https://review.opendev.org/c/zuul/zuul-jobs/+/949697 | 05:00 |
| opendevreview | Michal Nasiadka proposed zuul/zuul-jobs master: ensure-dib: Add podman and rootlesskit https://review.opendev.org/c/zuul/zuul-jobs/+/949697 | 05:01 |
| *** ralonsoh_out is now known as ralonsoh\ | 05:46 | |
| *** ralonsoh\ is now known as ralonsoh | 05:47 | |
| opendevreview | Benjamin Schanzel proposed zuul/zuul-jobs master: Add a meta log upload role with a failover mechanism https://review.opendev.org/c/zuul/zuul-jobs/+/795336 | 07:16 |
| mnasiadka | clarkb: we need a new DIB release to switch rocky-container to quay.io (https://opendev.org/openstack/diskimage-builder/commit/cdaf45b9e00af4f4f29f80439abe11e55f18306f) - how do I ,,orchestrate'' this? | 07:34 |
| mnasiadka | IIRC release team does not do DIB releases | 07:34 |
| frickler | mnasiadka: there's a dib IRC channel, not sure who still hangs out there | 07:39 |
| mnasiadka | Well, usually ianw did the releases - I'm happy to help with that, but I guess I would need some additional rights to push tags to DIB repo :) | 07:39 |
| mnasiadka | Or we can ,,onboard'' DIB to openstack/releases repo under cycle independent | 07:40 |
| mnasiadka | but yes, let's move the discussion to DIB channel | 07:41 |
| frickler | I'm not there fwiw and it also doesn't seem to get logged. ping here again if you can make no progress | 07:41 |
| opendevreview | ribaudr proposed openstack/project-config master: Add team IRC ops for #openstack-nova https://review.opendev.org/c/openstack/project-config/+/949707 | 08:36 |
| *** ralonsoh_ is now known as ralonsoh | 09:21 | |
| opendevreview | Merged openstack/diskimage-builder master: Remove qemu-debootstrap from debootstrap element https://review.opendev.org/c/openstack/diskimage-builder/+/946550 | 10:38 |
| opendevreview | Merged openstack/diskimage-builder master: Remove the usage of pkg_resource https://review.opendev.org/c/openstack/diskimage-builder/+/933324 | 12:28 |
| opendevreview | Merged openstack/project-config master: Add team IRC ops for #openstack-nova https://review.opendev.org/c/openstack/project-config/+/949707 | 13:05 |
| opendevreview | Merged openstack/project-config master: to create a new repo for a cfn new launched sub-group heterogeneous distributed training framework https://review.opendev.org/c/openstack/project-config/+/949555 | 13:29 |
| fungi | https://superuser.openinfra.org/articles/opendev-and-rackspace-building-stronger-open-infrastructure-together/ | 13:31 |
| frickler | does ^^ mean that IPv6 is ready now? *scnr* | 13:37 |
| fungi | i read it as "real soon now" ;) | 13:39 |
| fungi | dan_with might know how close they are to dual-stack global networking | 13:39 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Add Rocky 8/9 builds, labels and provider config https://review.opendev.org/c/opendev/zuul-providers/+/949696 | 13:52 |
| Clark[m] | mnasiadka: we don't run dib releases with Openstack releases because of chicken and egg problems/concerns. But ya someone in the release group needs to push a tag. I can do it if I ever dig my gpg key out of cold storage. Fungi did one once recently too iirc. | 13:52 |
| opendevreview | Michal Nasiadka proposed zuul/zuul-jobs master: ensure-dib: Add podman and rootlesskit https://review.opendev.org/c/zuul/zuul-jobs/+/949697 | 13:52 |
| mnasiadka | Clark: ianw sorted it out on #openstack-dib - thanks :) | 13:53 |
| Clark[m] | Oh cool | 13:53 |
| fungi | yeah, there was a fair bit of discussion over there about the release process | 13:53 |
| mnasiadka | Well I think there was a couple of important patches in DIB since Dec 2024 (previous release) - so user-experience wise it would be good to release things more often :) | 13:54 |
| mnasiadka | at least rocky builds now use quay.io instead of docker hub | 13:54 |
| fungi | sure, also our nodepool/zuul deployments use released dib, so we can't take advantage of any changes until there's a release (which is why infra-core is included in diskimage-builder-release) | 14:00 |
| opendevreview | Michal Nasiadka proposed zuul/zuul-jobs master: ensure-dib: Add podman and rootlesskit https://review.opendev.org/c/zuul/zuul-jobs/+/949697 | 14:02 |
| opendevreview | Michal Nasiadka proposed zuul/zuul-jobs master: ensure-dib: Add podman and rootlesskit https://review.opendev.org/c/zuul/zuul-jobs/+/949697 | 14:02 |
| clarkb | I'll approve teh gitea 1.23.8 update in a few minutes if there aer no objections | 14:42 |
| clarkb | I don't see anything in scrollback or email that would indicate I should not do this but let me know if I missed something | 14:43 |
| fungi | please do, i'm around all day | 14:47 |
| clarkb | done it is on its way in (which will probably take about an hour or maybe even a little more | 14:48 |
| mnasiadka | Ok then, the niz-rocky builds are on image conversion level so they will be good to go when it finishes - I'm off the hook :) | 14:49 |
| clarkb | mnasiadka: thanks again for getting those images sorted out | 14:56 |
| mnasiadka | no problem, happy to help - nice difference from chasing breakages in Kolla world ;-) | 15:05 |
| mnasiadka | Question to some more Gerrit-knowledgeable people - in Kolla we have Review-Priority and Backport-Candidate labels - is there a way that a vote on this label would override other votes? As in person A votes RP+1, another person wants to change that to -1 - and we end up with one +1 and one -1 - I'd like that to be more of a ,,label'' than a vote with only one value... | 15:18 |
| mnasiadka | well, than a vote with multiple values from multiple people | 15:19 |
| clarkb | I think hashtags are better suited to this problem | 15:19 |
| clarkb | I forget why others have said taht won't work for review priority though. Maybe it is because anyone can set hashtags if we open them up globally (we haevn't yet but that is the idea) | 15:20 |
| clarkb | but no in general individual own their votes. The actions you take on those votes can be scoped to specific groups or users, But I don't think we can generically say this vote goes away if someone else votes something different | 15:20 |
| mnasiadka | I think we have an ACL in Kolla that allows them only for core-reviewers - let me check | 15:20 |
| mnasiadka | well, hashtags sound like a better suited solution for that instead of standard voting mechanism | 15:22 |
| corvus | only other thing i'd say about votes is there are different options for calculating the winner (max with and without blocking, for example). not sure if that's flexible enough to accommodate what you want. but also, hashtags ftw. | 15:50 |
| fungi | right... it's in that category of technical solutions to social challenges: document the hashtags your project intends to use for specific situations, if someone misuses them then have a chat with that person about it, and if they're unreasonable then escalate it to project leadership/platform admins | 15:54 |
| fungi | micro-managing per-project access to set and remove hashtags is almost certainly an overoptimization | 15:55 |
| fungi | if a user starts abusing access by going around randomly setting or removing hashtags on projects, i have no qualms about disabling their account immediately | 15:56 |
| opendevreview | Clark Boylan proposed opendev/system-config master: DNM Forced fail on Gerrit to test 3.11 upgrade and downgrade https://review.opendev.org/c/opendev/system-config/+/893571 | 16:04 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update Gerrit images to 3.10.6 and 3.11.3 https://review.opendev.org/c/opendev/system-config/+/949778 | 16:04 |
| clarkb | there is a newer gerrit release for 3.10 and 3.11. I figure getting those updated is a godo step 0 before we start testing upgrade stuff. Then the second change there has a couple of holds in place to make testing of the upgrade easy | 16:06 |
| fungi | agreed | 16:07 |
| clarkb | gitea change should merge in a minute or two. Its uploading logs | 16:38 |
| opendevreview | Merged opendev/system-config master: Update to gitea 1.23.8 https://review.opendev.org/c/opendev/system-config/+/949544 | 16:39 |
| clarkb | and it is deploying now | 16:41 |
| clarkb | https://gitea09.opendev.org:3081/opendev/system-config/ is up and reports the expected version. The page rendered how I expect it. I'll do a clone test next | 16:44 |
| fungi | lgtm! | 16:44 |
| clarkb | clone works | 16:45 |
| fungi | 09 seems to be working | 16:45 |
| clarkb | ya so far it looks happy. We're through at least gitea11 at this point | 16:48 |
| clarkb | https://zuul.opendev.org/t/openstack/build/5585152355814cc089d9c1fdde0e2138 success and my checks of individual backeds look good | 16:54 |
| clarkb | infra-root checking the giteas I notice that gitea10-gitea14 report no space left on device for /var/log/syslog and the journals on May 10 (from dmesg -T). df -h reports plenty of disk now and we do seem to be writing to syslog and maybe the journal as well. Gitea seems to have up to date content too and / is not mounted ro. Also gitea09 doesn't seem to have been hit by this | 17:02 |
| clarkb | not sure what is going on there but it is weird enough that it may be worth someone else doing a quick check that there isn't anything terribly wrong we need to intervene for | 17:02 |
| clarkb | I'm beginning to suspect some temporary blip in storage for those servers and when storage resumed normal operations so did our servers | 17:03 |
| clarkb | I think if things were persistently sad the upgrade we just did would have failed (due to being unable to fetch and store new docker images) | 17:04 |
| clarkb | infra-root https://104.130.253.194/q/status:open+-is:wip is a held Gerrit 3.11 which we can use to interact with it and see that it works as expected. I also held a 3.10 node and that is the node I'll use to test the upgrade and downgrade process | 17:30 |
| clarkb | basically this 3.11 node should be safe to use at any time as a "what does 3.11 look like" check then the other node iswhere things will go up and down and change versions | 17:30 |
| clarkb | visually this doesn't seem all that different | 17:31 |
| fungi | the free space is enough that i doubt it dipped into root-only overhead during rotation | 17:44 |
| fungi | also 09 and 10 have basically identical utilization at the moment | 17:45 |
| clarkb | ya and it should rotate more regularly than only on the 10th and not since | 17:45 |
| clarkb | this is why I suspect something on the underlying cloud | 17:45 |
| fungi | agreed | 17:45 |
| corvus | clarkb: could check cacti graphs | 17:46 |
| clarkb | ah yup | 17:46 |
| clarkb | corvus: that was a great idea. Gitea10 does seem to have run out of disk on the 10th | 17:47 |
| clarkb | it was very sawtooth | 17:47 |
| clarkb | suddenly I'm reminded of the tarball generation problem and i think that must be what happened hwere | 17:47 |
| fungi | okay, so maybe rotation related after all | 17:47 |
| clarkb | we run a cron to prune those daily and that wasn't keeping up | 17:47 |
| fungi | oh! yes, tarballs | 17:48 |
| clarkb | fungi: ya not log rotation but rotation of the tarball artifacts | 17:48 |
| fungi | agreed, that would definitely explain it | 17:48 |
| fungi | and 09 just got lucky i guess | 17:48 |
| corvus | while /bin/true; rm tarballs; end | 17:48 |
| fungi | you forgot the "do" ;) | 17:48 |
| clarkb | ya I think we can probably just update the cron to run hourly or twice a day or similar | 17:48 |
| clarkb | I'll prep a change for that so we have it if it becomes useful again | 17:48 |
| clarkb | (right now things seems stable for the last few days) | 17:49 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Run gitea tarball cleanup every 6 hours https://review.opendev.org/c/opendev/system-config/+/949790 | 17:56 |
| clarkb | in related news github announced anonymous request rate limit changes as even they are being crushed by the bots on the internet vying for AI supremacy | 17:57 |
| clarkb | looks like prior to the 10th it would get close to the limit but not exceed it | 18:01 |
| clarkb | then on the 10th we got "lucky" | 18:01 |
| clarkb | so ya I think 949790 should help mitigate for the future if we don't have better ideas (pretty sure I checked and we cannot disable this feature entirely otherwise I would) | 18:02 |
| opendevreview | Merged zuul/zuul-jobs master: Add a meta log upload role with a failover mechanism https://review.opendev.org/c/zuul/zuul-jobs/+/795336 | 18:16 |
| corvus | i think we could adapt ^ for use in our environment... but then we might not notice cloud storage failures.... | 18:18 |
| corvus | something to think about | 18:18 |
| clarkb | I guess we could generate a random order for the swift backends then pass that entire list to this new role? | 18:20 |
| clarkb | then as long as any one of them succeeds we would avoid job failures. With the current backends we use its likely that 3 fail or 2 fail given that 3 belong to one cloud and 2 to another. So we probably need at least 4 options and at that point you may as well use all 5 | 18:21 |
| fungi | reminiscent of (mike pondsmith/r. talsorian games) cyberpunk lore where where the old net was overrun by rogue ai systems so they had to build the blackwall to keep them from leaking into the new reconstructed net after the datakrash of 2022. even the timeline isn't too far off | 18:22 |
| mnasiadka | fungi: if hashtags are limited to core reviewers we should be fine ;) | 18:28 |
| mnasiadka | Ok - both niz-rocky patches have passed Zuul and are good to go https://review.opendev.org/q/topic:%22niz-rocky%22 | 18:29 |
| corvus | hashtags are very useful for non-core reviewers, i would encourage not limiting them. if it's important enough to restrict, then it should probably be a label (and then look into the submit rules) | 18:34 |
| fungi | mnasiadka: yeah, maybe you misread me. i said limiting hashtags to core reviewers is a wasteful overoptimization at best. if people misuse them (core or otherwise) then talk to them. if they won't listen, talk to us | 18:38 |
| fungi | people problems require people solutions | 18:38 |
| ildikov | Hi y'all, I'm reaching out about an error I got on my patch to update a StarlingX doc page: https://review.opendev.org/c/starlingx/governance/+/949746 | 19:02 |
| ildikov | If I read the logs correctly, it seems like a library issue and not an error in my patch. But I might've missed smth. Can someone please help me confirm one way or the other? | 19:03 |
| fungi | looking | 19:04 |
| Clark[m] | I think that is the problem that occurs when you run old flake8 on modern python (likely due to the node set being updated to noble) | 19:06 |
| Clark[m] | If you update flake8 it should fix the problem | 19:07 |
| ildikov | @Clark[m] great, that confirms my suspicion | 19:12 |
| ildikov | @Clark[m] sooo, how can I update flake8? :) | 19:13 |
| fungi | yeah, current flake8 per https://pypi.org/project/flake8/ is 7.2.0, that job seems to have installed 3.8.4 (from 2020) according to the log, it's probably using an old constraints file but i'm digging into that now | 19:15 |
| fungi | the log says it ran `pip install -U -r /home/zuul/src/opendev.org/starlingx/governance/test-requirements.txt` | 19:17 |
| fungi | so it's coming in as an indirect dependency via https://opendev.org/starlingx/governance/src/branch/master/test-requirements.txt#L1 | 19:17 |
| ildikov | ah, ok | 19:17 |
| ildikov | I checked that file, among other possible ones, and then realized I might need help figuring the dependency out | 19:18 |
| fungi | the current hacking version is 7.0.0 | 19:18 |
| fungi | just for reference | 19:18 |
| fungi | but i'd probably try to match to whatever version of hacking other starlingx projects are using | 19:19 |
| ildikov | ok, cool, I'll check | 19:19 |
| slampton | Hi, I'm an e-mail admin, troubleshooting the issue of not being able to receive e-mail from Gerrit (162.253.55.233). The thing is, we're receiving e-mail from openstack.org from that IP, but not from opendev.org. That suggests to me that you have a difference in either reverse DNS, SPF. DKIM, or DMARC records between those domains. | 19:28 |
| slampton | Proofpoint says they have removed the IP block, even though the lookup tool know gives an error. Can you provide smtp logs from your end, when sending from opendev.org to windriver.com? | 19:28 |
| fungi | slampton: i'll get those now, just a sec | 19:30 |
| slampton | thx | 19:30 |
| fungi | for the record, i requested removal of that address from proofpoint weeks ago and received no response, they seem to be completely unstaffed or otherwise defunct | 19:30 |
| slampton | that's ridiculous, they seem to be staffed just fine, if you are a paying customer | 19:33 |
| fungi | yeah, maybe they just ignore anything put into their removal request form. anyway i do see messages getting accepted right now in the log... (redacted) 2025-05-14 19:29:51 1uFHna-00000006wlT-0tv7 -> REDACTED@windriver.com R=dnslookup T=remote_smtp H=mxb-0064b401.gslb.pphosted.com [205.220.178.238] X=TLS1.2:ECDHE_SECP256R1__RSA_SHA256__AES_256_GCM:256 CV=yes C="250 2.0.0 46mbc8sm08-1 | 19:36 |
| fungi | Message accepted for delivery" | 19:36 |
| fungi | i'm looking to see when the last refusal was | 19:37 |
| mnasiadka | fungi: makes sense, after giving some thoughts we’re probably going to switch backport-candidate from review labels to use hashtags, but for review priority I would probably still stick with review labels (but think about giving the PTL/cores option to delete RP votes - because we sometimes end up with that semi-active person setting review priority - which needs to be lowered) | 19:37 |
| fungi | slampton: most recent rejection i see from pphosted.com in our logs was 2025-05-08 08:22:35 utc, almost a week ago now | 19:39 |
| fungi | there was an incident around 2025-05-12 02:38-04:00 with problems looking up the domain in dns, but that only resulted in messages being temporarily deferred | 19:41 |
| fungi | slampton: if messages aren't getting through for the past week, then my only guess is proofpoint has started 250/2.0.0 accepting the messages but then dumping them in the bitbucket | 19:42 |
| ildikov | I was finally able to find a hacking version that worked, thanks fungi for the help in figuring that out | 19:52 |
| fungi | ildikov: Clark[m] too, but you're welcome! | 19:59 |
| ildikov | yes, thanks Clark[m] too! | 19:59 |
| slampton | fungi: thanks, that correlates with the removal of the IP block. They still seem to be silently dropped, because I don't see anything in our logs from @opendev.org. I've asked Proofpoint to investigate further | 19:59 |
| clarkb | fungi: slampton: looking at my personal inbox I think the deliveries may be coming from review@openstack.org | 20:02 |
| fungi | yes, we don't have e-mail set up for the opendev.org domain for $reasons, so still use @openstack.org for the addresses | 20:02 |
| fungi | we want messages to be deliverable, therefore we only send from addresses at a domain that actually accepts smtp deliveries itself | 20:03 |
| fungi | confirmed, the sender (smtp "mail from") on those messages should be review@openstack.org | 20:06 |
| fungi | the rfc 822 "from:" header will also be review@openstack.org | 20:07 |
| fungi | though with varying names | 20:08 |
| mnaser | OSError: [Errno 39] Directory not empty: '/opt/stack/requirements/.eggs/pbr-6.1.1-py3.10.egg/pbr-6.1.1.dist-info' -> '/opt/stack/requirements/.eggs/pbr-6.1.1-py3.10.egg/EGG-INFO' | 20:28 |
| mnaser | Running setup.py install for openstack_requirements: finished with status 'error' | 20:28 |
| mnaser | waaah, do we just keep catching issues lol | 20:28 |
| mnaser | https://github.com/vexxhost/magnum-cluster-api/actions/runs/15030117460/job/42240444170#step:8:6705 | 20:30 |
| mnaser | https://github.com/pypa/setuptools/releases | 20:31 |
| mnaser | an hour ago a job passed | 20:31 |
| mnaser | trying to look at https://github.com/pypa/setuptools/compare/v80.5.0...v80.7.0 | 20:32 |
| slampton | fungi: aha, so we may be good then | 20:33 |
| mnaser | job passed with setuptools 80.4.0 | 20:34 |
| mnaser | so a bunch happened here https://github.com/pypa/setuptools/compare/v80.4.0...v80.7.0 | 20:35 |
| clarkb | mnaser: the code that raised the exception is 7 years old | 20:35 |
| fungi | but maybe not a code path we used to hit | 20:35 |
| clarkb | fungi: ya or something was previously ensuring the directory on the dst side was empty and now it isn't | 20:36 |
| clarkb | os.rename says you can't rename foo/ into bar/ if bar/ is not empty | 20:36 |
| fungi | also possible | 20:36 |
| mnaser | https://github.com/vexxhost/magnum-cluster-api/actions/runs/15029256236/job/42237574746#step:8:6576 installed just fine an hour ago | 20:37 |
| mnaser | where setuptools was 80.4.0 | 20:37 |
| clarkb | which is what I think happened here. Something had already written to bar/ (/opt/stack/requirements/.eggs/pbr-6.1.1-py3.10.egg/) and kaboom because it already had an EGG-INFO in it | 20:37 |
| fungi | i wonder if that's pbr's metadata writer for the pbr.json file | 20:38 |
| mnaser | is setuptools pinned in opendev? | 20:39 |
| fungi | depends on the project (whether it uss a pyproject.toml), phase (build vs install) and invoking tool (e.g. tox may install specific setuptools versions) | 20:40 |
| fungi | so the answer is "sometimes"? | 20:41 |
| mnaser | oh.. | 20:41 |
| mnaser | i think the flood gates of failure are about to come | 20:41 |
| mnaser | https://zuul.opendev.org/t/openstack/status see 949800,1 -- failed with same reason | 20:41 |
| mnaser | https://zuul.opendev.org/t/openstack/build/13528f8c1f99421391600a49673e66df | 20:41 |
| clarkb | fungi: I think that goes into a different file not EGG-INFO looks like it goes to pbr.json. However, it is defined as a egg_info.writers entrypoint | 20:42 |
| mnaser | im running a quick pip install in a docker container here to see whats in the folder | 20:44 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!