Wednesday, 2025-05-14

opendevreviewMichal Nasiadka proposed opendev/zuul-providers master: Add Rocky 8/9 builds  https://review.opendev.org/c/opendev/zuul-providers/+/94969604:45
opendevreviewMichal Nasiadka proposed opendev/zuul-providers master: Add Rocky 8/9 builds  https://review.opendev.org/c/opendev/zuul-providers/+/94969604:48
opendevreviewMichal Nasiadka proposed zuul/zuul-jobs master: ensure-dib: Add podman  https://review.opendev.org/c/zuul/zuul-jobs/+/94969704:50
opendevreviewMichal Nasiadka proposed opendev/zuul-providers master: Add Rocky 8/9 builds  https://review.opendev.org/c/opendev/zuul-providers/+/94969604:50
opendevreviewMichal Nasiadka proposed zuul/zuul-jobs master: ensure-dib: Add newuidmap and podman  https://review.opendev.org/c/zuul/zuul-jobs/+/94969704:58
opendevreviewMichal Nasiadka proposed zuul/zuul-jobs master: ensure-dib: Add newuidmap and podman  https://review.opendev.org/c/zuul/zuul-jobs/+/94969705:00
opendevreviewMichal Nasiadka proposed zuul/zuul-jobs master: ensure-dib: Add podman and rootlesskit  https://review.opendev.org/c/zuul/zuul-jobs/+/94969705:01
*** ralonsoh_out is now known as ralonsoh\05:46
*** ralonsoh\ is now known as ralonsoh05:47
opendevreviewBenjamin Schanzel proposed zuul/zuul-jobs master: Add a meta log upload role with a failover mechanism  https://review.opendev.org/c/zuul/zuul-jobs/+/79533607:16
mnasiadkaclarkb: we need a new DIB release to switch rocky-container to quay.io (https://opendev.org/openstack/diskimage-builder/commit/cdaf45b9e00af4f4f29f80439abe11e55f18306f) - how do I ,,orchestrate'' this?07:34
mnasiadkaIIRC release team does not do DIB releases07:34
fricklermnasiadka: there's a dib IRC channel, not sure who still hangs out there07:39
mnasiadkaWell, usually ianw did the releases - I'm happy to help with that, but I guess I would need some additional rights to push tags to DIB repo :)07:39
mnasiadkaOr we can ,,onboard'' DIB to openstack/releases repo under cycle independent07:40
mnasiadkabut yes, let's move the discussion to DIB channel07:41
fricklerI'm not there fwiw and it also doesn't seem to get logged. ping here again if you can make no progress07:41
opendevreviewribaudr proposed openstack/project-config master: Add team IRC ops for #openstack-nova  https://review.opendev.org/c/openstack/project-config/+/94970708:36
*** ralonsoh_ is now known as ralonsoh09:21
opendevreviewMerged openstack/diskimage-builder master: Remove qemu-debootstrap from debootstrap element  https://review.opendev.org/c/openstack/diskimage-builder/+/94655010:38
opendevreviewMerged openstack/diskimage-builder master: Remove the usage of pkg_resource  https://review.opendev.org/c/openstack/diskimage-builder/+/93332412:28
opendevreviewMerged openstack/project-config master: Add team IRC ops for #openstack-nova  https://review.opendev.org/c/openstack/project-config/+/94970713:05
opendevreviewMerged openstack/project-config master: to create a new repo for a cfn new launched sub-group heterogeneous distributed training framework  https://review.opendev.org/c/openstack/project-config/+/94955513:29
fungihttps://superuser.openinfra.org/articles/opendev-and-rackspace-building-stronger-open-infrastructure-together/13:31
fricklerdoes ^^ mean that IPv6 is ready now? *scnr*13:37
fungii read it as "real soon now" ;)13:39
fungidan_with might know how close they are to dual-stack global networking13:39
opendevreviewMichal Nasiadka proposed opendev/zuul-providers master: Add Rocky 8/9 builds, labels and provider config  https://review.opendev.org/c/opendev/zuul-providers/+/94969613:52
Clark[m]mnasiadka: we don't run dib releases with Openstack releases because of chicken and egg problems/concerns. But ya someone in the release group needs to push a tag. I can do it if I ever dig my gpg key out of cold storage. Fungi did one once recently too iirc.13:52
opendevreviewMichal Nasiadka proposed zuul/zuul-jobs master: ensure-dib: Add podman and rootlesskit  https://review.opendev.org/c/zuul/zuul-jobs/+/94969713:52
mnasiadkaClark: ianw sorted it out on #openstack-dib - thanks :)13:53
Clark[m]Oh cool 13:53
fungiyeah, there was a fair bit of discussion over there about the release process13:53
mnasiadkaWell I think there was a couple of important patches in DIB since Dec 2024 (previous release) - so user-experience wise it would be good to release things more often :)13:54
mnasiadkaat least rocky builds now use quay.io instead of docker hub13:54
fungisure, also our nodepool/zuul deployments use released dib, so we can't take advantage of any changes until there's a release (which is why infra-core is included in diskimage-builder-release)14:00
opendevreviewMichal Nasiadka proposed zuul/zuul-jobs master: ensure-dib: Add podman and rootlesskit  https://review.opendev.org/c/zuul/zuul-jobs/+/94969714:02
opendevreviewMichal Nasiadka proposed zuul/zuul-jobs master: ensure-dib: Add podman and rootlesskit  https://review.opendev.org/c/zuul/zuul-jobs/+/94969714:02
clarkbI'll approve teh gitea 1.23.8 update in a few minutes if there aer no objections14:42
clarkbI don't see anything in scrollback or email that would indicate I should not do this but let me know if I missed something14:43
fungiplease do, i'm around all day14:47
clarkbdone it is on its way in (which will probably take about an hour or maybe even a little more14:48
mnasiadkaOk then, the niz-rocky builds are on image conversion level so they will be good to go when it finishes - I'm off the hook :)14:49
clarkbmnasiadka: thanks again for getting those images sorted out14:56
mnasiadkano problem, happy to help - nice difference from chasing breakages in Kolla world ;-)15:05
mnasiadkaQuestion to some more Gerrit-knowledgeable people - in Kolla we have Review-Priority and Backport-Candidate labels - is there a way that a vote on this label would override other votes? As in person A votes RP+1, another person wants to change that to -1 - and we end up with one +1 and one -1 - I'd like that to be more of a ,,label'' than a vote with only one value...15:18
mnasiadkawell, than a vote with multiple values from multiple people15:19
clarkbI think hashtags are better suited to this problem15:19
clarkbI forget why others have said taht won't work for review priority though. Maybe it is because anyone can set hashtags if we open them up globally (we haevn't yet but that is the idea)15:20
clarkbbut no in general individual own their votes. The actions you take on those votes can be scoped to specific groups or users, But I don't think we can generically say this vote goes away if someone else votes something different15:20
mnasiadkaI think we have an ACL in Kolla that allows them only for core-reviewers - let me check15:20
mnasiadkawell, hashtags sound like a better suited solution for that instead of standard voting mechanism15:22
corvusonly other thing i'd say about votes is there are different options for calculating the winner (max with and without blocking, for example).  not sure if that's flexible enough to accommodate what you want.  but also, hashtags ftw.15:50
fungiright... it's in that category of technical solutions to social challenges: document the hashtags your project intends to use for specific situations, if someone misuses them then have a chat with that person about it, and if they're unreasonable then escalate it to project leadership/platform admins15:54
fungimicro-managing per-project access to set and remove hashtags is almost certainly an overoptimization15:55
fungiif a user starts abusing access by going around randomly setting or removing hashtags on projects, i have no qualms about disabling their account immediately15:56
opendevreviewClark Boylan proposed opendev/system-config master: DNM Forced fail on Gerrit to test 3.11 upgrade and downgrade  https://review.opendev.org/c/opendev/system-config/+/89357116:04
opendevreviewClark Boylan proposed opendev/system-config master: Update Gerrit images to 3.10.6 and 3.11.3  https://review.opendev.org/c/opendev/system-config/+/94977816:04
clarkbthere is a newer gerrit release for 3.10 and 3.11. I figure getting those updated is a godo step 0 before we start testing upgrade stuff. Then the second change there has a couple of holds in place to make testing of the upgrade easy16:06
fungiagreed16:07
clarkbgitea change should merge in a minute or two. Its uploading logs16:38
opendevreviewMerged opendev/system-config master: Update to gitea 1.23.8  https://review.opendev.org/c/opendev/system-config/+/94954416:39
clarkband it is deploying now16:41
clarkbhttps://gitea09.opendev.org:3081/opendev/system-config/ is up and reports the expected version. The page rendered how I expect it. I'll do a clone test next16:44
fungilgtm!16:44
clarkbclone works16:45
fungi09 seems to be working16:45
clarkbya so far it looks happy. We're through at least gitea11 at this point16:48
clarkbhttps://zuul.opendev.org/t/openstack/build/5585152355814cc089d9c1fdde0e2138 success and my checks of individual backeds look good16:54
clarkbinfra-root checking the giteas I notice that gitea10-gitea14 report no space left on device for /var/log/syslog and the journals on May 10 (from dmesg -T). df -h reports plenty of disk now and we do seem to be writing to syslog and maybe the journal as well. Gitea seems to have up to date content too and / is not mounted ro. Also gitea09 doesn't seem to have been hit by this17:02
clarkbnot sure what is going on there but it is weird enough that it may be worth someone else doing a quick check that there isn't anything terribly wrong we need to intervene for17:02
clarkbI'm beginning to suspect some temporary blip in storage for those servers and when storage resumed normal operations so did our servers17:03
clarkbI think if things were persistently sad the upgrade we just did would have failed (due to being unable to fetch and store new docker images)17:04
clarkbinfra-root https://104.130.253.194/q/status:open+-is:wip is a held Gerrit 3.11 which we can use to interact with it and see that it works as expected. I also held a 3.10 node and that is the node I'll use to test the upgrade and downgrade process17:30
clarkbbasically this 3.11 node should be safe to use at any time as a "what does 3.11 look like" check then the other node iswhere things will go up and down and change versions17:30
clarkbvisually this doesn't seem all that different17:31
fungithe free space is enough that i doubt it dipped into root-only overhead during rotation17:44
fungialso 09 and 10 have basically identical utilization at the moment17:45
clarkbya and it should rotate more regularly than only on the 10th and not since17:45
clarkbthis is why I suspect something on the underlying cloud17:45
fungiagreed17:45
corvusclarkb: could check cacti graphs17:46
clarkbah yup17:46
clarkbcorvus: that was a great idea. Gitea10 does seem to have run out of disk on the 10th17:47
clarkbit was very sawtooth17:47
clarkbsuddenly I'm reminded of the tarball generation problem and i think that must be what happened hwere17:47
fungiokay, so maybe rotation related after all17:47
clarkbwe run a cron to prune those daily and that wasn't keeping up17:47
fungioh! yes, tarballs17:48
clarkbfungi: ya not log rotation but rotation of the tarball artifacts17:48
fungiagreed, that would definitely explain it17:48
fungiand 09 just got lucky i guess17:48
corvuswhile /bin/true; rm tarballs; end17:48
fungiyou forgot the "do" ;)17:48
clarkbya I think we can probably just update the cron to run hourly or twice a day or similar17:48
clarkbI'll prep a change for that so we have it if it becomes useful again17:48
clarkb(right now things seems stable for the last few days)17:49
opendevreviewClark Boylan proposed opendev/system-config master: Run gitea tarball cleanup every 6 hours  https://review.opendev.org/c/opendev/system-config/+/94979017:56
clarkbin related news github announced anonymous request rate limit changes as even they are being crushed by the bots on the internet vying for AI supremacy17:57
clarkblooks like prior to the 10th it would get close to the limit but not exceed it18:01
clarkbthen on the 10th we got "lucky"18:01
clarkbso ya I think 949790 should help mitigate for the future if we don't have better ideas (pretty sure I checked and we cannot disable this feature entirely otherwise I would)18:02
opendevreviewMerged zuul/zuul-jobs master: Add a meta log upload role with a failover mechanism  https://review.opendev.org/c/zuul/zuul-jobs/+/79533618:16
corvusi think we could adapt ^ for use in our environment... but then we might not notice cloud storage failures....18:18
corvussomething to think about18:18
clarkbI guess we could generate a random order for the swift backends then pass that entire list to this new role?18:20
clarkbthen as long as any one of them succeeds we would avoid job failures. With the current backends we use its likely that 3 fail or 2 fail given that 3 belong to one cloud and 2 to another. So we probably need at least 4 options and at that point you may as well use all 518:21
fungireminiscent of (mike pondsmith/r. talsorian games) cyberpunk lore where where the old net was overrun by rogue ai systems so they had to build the blackwall to keep them from leaking into the new reconstructed net after the datakrash of 2022. even the timeline isn't too far off18:22
mnasiadkafungi: if hashtags are limited to core reviewers we should be fine ;)18:28
mnasiadkaOk - both niz-rocky patches have passed Zuul and are good to go https://review.opendev.org/q/topic:%22niz-rocky%2218:29
corvushashtags are very useful for non-core reviewers, i would encourage not limiting them.  if it's important enough to restrict, then it should probably be a label (and then look into the submit rules)18:34
fungimnasiadka: yeah, maybe you misread me. i said limiting hashtags to core reviewers is a wasteful overoptimization at best. if people misuse them (core or otherwise) then talk to them. if they won't listen, talk to us18:38
fungipeople problems require people solutions18:38

Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!