| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Add Rocky 8/9 builds https://review.opendev.org/c/opendev/zuul-providers/+/949696 | 04:45 |
|---|---|---|
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Add Rocky 8/9 builds https://review.opendev.org/c/opendev/zuul-providers/+/949696 | 04:48 |
| opendevreview | Michal Nasiadka proposed zuul/zuul-jobs master: ensure-dib: Add podman https://review.opendev.org/c/zuul/zuul-jobs/+/949697 | 04:50 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Add Rocky 8/9 builds https://review.opendev.org/c/opendev/zuul-providers/+/949696 | 04:50 |
| opendevreview | Michal Nasiadka proposed zuul/zuul-jobs master: ensure-dib: Add newuidmap and podman https://review.opendev.org/c/zuul/zuul-jobs/+/949697 | 04:58 |
| opendevreview | Michal Nasiadka proposed zuul/zuul-jobs master: ensure-dib: Add newuidmap and podman https://review.opendev.org/c/zuul/zuul-jobs/+/949697 | 05:00 |
| opendevreview | Michal Nasiadka proposed zuul/zuul-jobs master: ensure-dib: Add podman and rootlesskit https://review.opendev.org/c/zuul/zuul-jobs/+/949697 | 05:01 |
| *** ralonsoh_out is now known as ralonsoh\ | 05:46 | |
| *** ralonsoh\ is now known as ralonsoh | 05:47 | |
| opendevreview | Benjamin Schanzel proposed zuul/zuul-jobs master: Add a meta log upload role with a failover mechanism https://review.opendev.org/c/zuul/zuul-jobs/+/795336 | 07:16 |
| mnasiadka | clarkb: we need a new DIB release to switch rocky-container to quay.io (https://opendev.org/openstack/diskimage-builder/commit/cdaf45b9e00af4f4f29f80439abe11e55f18306f) - how do I ,,orchestrate'' this? | 07:34 |
| mnasiadka | IIRC release team does not do DIB releases | 07:34 |
| frickler | mnasiadka: there's a dib IRC channel, not sure who still hangs out there | 07:39 |
| mnasiadka | Well, usually ianw did the releases - I'm happy to help with that, but I guess I would need some additional rights to push tags to DIB repo :) | 07:39 |
| mnasiadka | Or we can ,,onboard'' DIB to openstack/releases repo under cycle independent | 07:40 |
| mnasiadka | but yes, let's move the discussion to DIB channel | 07:41 |
| frickler | I'm not there fwiw and it also doesn't seem to get logged. ping here again if you can make no progress | 07:41 |
| opendevreview | ribaudr proposed openstack/project-config master: Add team IRC ops for #openstack-nova https://review.opendev.org/c/openstack/project-config/+/949707 | 08:36 |
| *** ralonsoh_ is now known as ralonsoh | 09:21 | |
| opendevreview | Merged openstack/diskimage-builder master: Remove qemu-debootstrap from debootstrap element https://review.opendev.org/c/openstack/diskimage-builder/+/946550 | 10:38 |
| opendevreview | Merged openstack/diskimage-builder master: Remove the usage of pkg_resource https://review.opendev.org/c/openstack/diskimage-builder/+/933324 | 12:28 |
| opendevreview | Merged openstack/project-config master: Add team IRC ops for #openstack-nova https://review.opendev.org/c/openstack/project-config/+/949707 | 13:05 |
| opendevreview | Merged openstack/project-config master: to create a new repo for a cfn new launched sub-group heterogeneous distributed training framework https://review.opendev.org/c/openstack/project-config/+/949555 | 13:29 |
| fungi | https://superuser.openinfra.org/articles/opendev-and-rackspace-building-stronger-open-infrastructure-together/ | 13:31 |
| frickler | does ^^ mean that IPv6 is ready now? *scnr* | 13:37 |
| fungi | i read it as "real soon now" ;) | 13:39 |
| fungi | dan_with might know how close they are to dual-stack global networking | 13:39 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Add Rocky 8/9 builds, labels and provider config https://review.opendev.org/c/opendev/zuul-providers/+/949696 | 13:52 |
| Clark[m] | mnasiadka: we don't run dib releases with Openstack releases because of chicken and egg problems/concerns. But ya someone in the release group needs to push a tag. I can do it if I ever dig my gpg key out of cold storage. Fungi did one once recently too iirc. | 13:52 |
| opendevreview | Michal Nasiadka proposed zuul/zuul-jobs master: ensure-dib: Add podman and rootlesskit https://review.opendev.org/c/zuul/zuul-jobs/+/949697 | 13:52 |
| mnasiadka | Clark: ianw sorted it out on #openstack-dib - thanks :) | 13:53 |
| Clark[m] | Oh cool | 13:53 |
| fungi | yeah, there was a fair bit of discussion over there about the release process | 13:53 |
| mnasiadka | Well I think there was a couple of important patches in DIB since Dec 2024 (previous release) - so user-experience wise it would be good to release things more often :) | 13:54 |
| mnasiadka | at least rocky builds now use quay.io instead of docker hub | 13:54 |
| fungi | sure, also our nodepool/zuul deployments use released dib, so we can't take advantage of any changes until there's a release (which is why infra-core is included in diskimage-builder-release) | 14:00 |
| opendevreview | Michal Nasiadka proposed zuul/zuul-jobs master: ensure-dib: Add podman and rootlesskit https://review.opendev.org/c/zuul/zuul-jobs/+/949697 | 14:02 |
| opendevreview | Michal Nasiadka proposed zuul/zuul-jobs master: ensure-dib: Add podman and rootlesskit https://review.opendev.org/c/zuul/zuul-jobs/+/949697 | 14:02 |
| clarkb | I'll approve teh gitea 1.23.8 update in a few minutes if there aer no objections | 14:42 |
| clarkb | I don't see anything in scrollback or email that would indicate I should not do this but let me know if I missed something | 14:43 |
| fungi | please do, i'm around all day | 14:47 |
| clarkb | done it is on its way in (which will probably take about an hour or maybe even a little more | 14:48 |
| mnasiadka | Ok then, the niz-rocky builds are on image conversion level so they will be good to go when it finishes - I'm off the hook :) | 14:49 |
| clarkb | mnasiadka: thanks again for getting those images sorted out | 14:56 |
| mnasiadka | no problem, happy to help - nice difference from chasing breakages in Kolla world ;-) | 15:05 |
| mnasiadka | Question to some more Gerrit-knowledgeable people - in Kolla we have Review-Priority and Backport-Candidate labels - is there a way that a vote on this label would override other votes? As in person A votes RP+1, another person wants to change that to -1 - and we end up with one +1 and one -1 - I'd like that to be more of a ,,label'' than a vote with only one value... | 15:18 |
| mnasiadka | well, than a vote with multiple values from multiple people | 15:19 |
| clarkb | I think hashtags are better suited to this problem | 15:19 |
| clarkb | I forget why others have said taht won't work for review priority though. Maybe it is because anyone can set hashtags if we open them up globally (we haevn't yet but that is the idea) | 15:20 |
| clarkb | but no in general individual own their votes. The actions you take on those votes can be scoped to specific groups or users, But I don't think we can generically say this vote goes away if someone else votes something different | 15:20 |
| mnasiadka | I think we have an ACL in Kolla that allows them only for core-reviewers - let me check | 15:20 |
| mnasiadka | well, hashtags sound like a better suited solution for that instead of standard voting mechanism | 15:22 |
| corvus | only other thing i'd say about votes is there are different options for calculating the winner (max with and without blocking, for example). not sure if that's flexible enough to accommodate what you want. but also, hashtags ftw. | 15:50 |
| fungi | right... it's in that category of technical solutions to social challenges: document the hashtags your project intends to use for specific situations, if someone misuses them then have a chat with that person about it, and if they're unreasonable then escalate it to project leadership/platform admins | 15:54 |
| fungi | micro-managing per-project access to set and remove hashtags is almost certainly an overoptimization | 15:55 |
| fungi | if a user starts abusing access by going around randomly setting or removing hashtags on projects, i have no qualms about disabling their account immediately | 15:56 |
| opendevreview | Clark Boylan proposed opendev/system-config master: DNM Forced fail on Gerrit to test 3.11 upgrade and downgrade https://review.opendev.org/c/opendev/system-config/+/893571 | 16:04 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update Gerrit images to 3.10.6 and 3.11.3 https://review.opendev.org/c/opendev/system-config/+/949778 | 16:04 |
| clarkb | there is a newer gerrit release for 3.10 and 3.11. I figure getting those updated is a godo step 0 before we start testing upgrade stuff. Then the second change there has a couple of holds in place to make testing of the upgrade easy | 16:06 |
| fungi | agreed | 16:07 |
| clarkb | gitea change should merge in a minute or two. Its uploading logs | 16:38 |
| opendevreview | Merged opendev/system-config master: Update to gitea 1.23.8 https://review.opendev.org/c/opendev/system-config/+/949544 | 16:39 |
| clarkb | and it is deploying now | 16:41 |
| clarkb | https://gitea09.opendev.org:3081/opendev/system-config/ is up and reports the expected version. The page rendered how I expect it. I'll do a clone test next | 16:44 |
| fungi | lgtm! | 16:44 |
| clarkb | clone works | 16:45 |
| fungi | 09 seems to be working | 16:45 |
| clarkb | ya so far it looks happy. We're through at least gitea11 at this point | 16:48 |
| clarkb | https://zuul.opendev.org/t/openstack/build/5585152355814cc089d9c1fdde0e2138 success and my checks of individual backeds look good | 16:54 |
| clarkb | infra-root checking the giteas I notice that gitea10-gitea14 report no space left on device for /var/log/syslog and the journals on May 10 (from dmesg -T). df -h reports plenty of disk now and we do seem to be writing to syslog and maybe the journal as well. Gitea seems to have up to date content too and / is not mounted ro. Also gitea09 doesn't seem to have been hit by this | 17:02 |
| clarkb | not sure what is going on there but it is weird enough that it may be worth someone else doing a quick check that there isn't anything terribly wrong we need to intervene for | 17:02 |
| clarkb | I'm beginning to suspect some temporary blip in storage for those servers and when storage resumed normal operations so did our servers | 17:03 |
| clarkb | I think if things were persistently sad the upgrade we just did would have failed (due to being unable to fetch and store new docker images) | 17:04 |
| clarkb | infra-root https://104.130.253.194/q/status:open+-is:wip is a held Gerrit 3.11 which we can use to interact with it and see that it works as expected. I also held a 3.10 node and that is the node I'll use to test the upgrade and downgrade process | 17:30 |
| clarkb | basically this 3.11 node should be safe to use at any time as a "what does 3.11 look like" check then the other node iswhere things will go up and down and change versions | 17:30 |
| clarkb | visually this doesn't seem all that different | 17:31 |
| fungi | the free space is enough that i doubt it dipped into root-only overhead during rotation | 17:44 |
| fungi | also 09 and 10 have basically identical utilization at the moment | 17:45 |
| clarkb | ya and it should rotate more regularly than only on the 10th and not since | 17:45 |
| clarkb | this is why I suspect something on the underlying cloud | 17:45 |
| fungi | agreed | 17:45 |
| corvus | clarkb: could check cacti graphs | 17:46 |
| clarkb | ah yup | 17:46 |
| clarkb | corvus: that was a great idea. Gitea10 does seem to have run out of disk on the 10th | 17:47 |
| clarkb | it was very sawtooth | 17:47 |
| clarkb | suddenly I'm reminded of the tarball generation problem and i think that must be what happened hwere | 17:47 |
| fungi | okay, so maybe rotation related after all | 17:47 |
| clarkb | we run a cron to prune those daily and that wasn't keeping up | 17:47 |
| fungi | oh! yes, tarballs | 17:48 |
| clarkb | fungi: ya not log rotation but rotation of the tarball artifacts | 17:48 |
| fungi | agreed, that would definitely explain it | 17:48 |
| fungi | and 09 just got lucky i guess | 17:48 |
| corvus | while /bin/true; rm tarballs; end | 17:48 |
| fungi | you forgot the "do" ;) | 17:48 |
| clarkb | ya I think we can probably just update the cron to run hourly or twice a day or similar | 17:48 |
| clarkb | I'll prep a change for that so we have it if it becomes useful again | 17:48 |
| clarkb | (right now things seems stable for the last few days) | 17:49 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Run gitea tarball cleanup every 6 hours https://review.opendev.org/c/opendev/system-config/+/949790 | 17:56 |
| clarkb | in related news github announced anonymous request rate limit changes as even they are being crushed by the bots on the internet vying for AI supremacy | 17:57 |
| clarkb | looks like prior to the 10th it would get close to the limit but not exceed it | 18:01 |
| clarkb | then on the 10th we got "lucky" | 18:01 |
| clarkb | so ya I think 949790 should help mitigate for the future if we don't have better ideas (pretty sure I checked and we cannot disable this feature entirely otherwise I would) | 18:02 |
| opendevreview | Merged zuul/zuul-jobs master: Add a meta log upload role with a failover mechanism https://review.opendev.org/c/zuul/zuul-jobs/+/795336 | 18:16 |
| corvus | i think we could adapt ^ for use in our environment... but then we might not notice cloud storage failures.... | 18:18 |
| corvus | something to think about | 18:18 |
| clarkb | I guess we could generate a random order for the swift backends then pass that entire list to this new role? | 18:20 |
| clarkb | then as long as any one of them succeeds we would avoid job failures. With the current backends we use its likely that 3 fail or 2 fail given that 3 belong to one cloud and 2 to another. So we probably need at least 4 options and at that point you may as well use all 5 | 18:21 |
| fungi | reminiscent of (mike pondsmith/r. talsorian games) cyberpunk lore where where the old net was overrun by rogue ai systems so they had to build the blackwall to keep them from leaking into the new reconstructed net after the datakrash of 2022. even the timeline isn't too far off | 18:22 |
| mnasiadka | fungi: if hashtags are limited to core reviewers we should be fine ;) | 18:28 |
| mnasiadka | Ok - both niz-rocky patches have passed Zuul and are good to go https://review.opendev.org/q/topic:%22niz-rocky%22 | 18:29 |
| corvus | hashtags are very useful for non-core reviewers, i would encourage not limiting them. if it's important enough to restrict, then it should probably be a label (and then look into the submit rules) | 18:34 |
| fungi | mnasiadka: yeah, maybe you misread me. i said limiting hashtags to core reviewers is a wasteful overoptimization at best. if people misuse them (core or otherwise) then talk to them. if they won't listen, talk to us | 18:38 |
| fungi | people problems require people solutions | 18:38 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!