| -@gerrit:opendev.org- Michal Nasiadka proposed: [openstack/diskimage-builder] 982231: Add support for building Ubuntu Resolute (26.04) https://review.opendev.org/c/openstack/diskimage-builder/+/982231 | 09:19 | |
| -@gerrit:opendev.org- Michal Nasiadka proposed: [openstack/diskimage-builder] 982231: Add support for building Ubuntu Resolute (26.04) https://review.opendev.org/c/openstack/diskimage-builder/+/982231 | 09:25 | |
| -@gerrit:opendev.org- Michal Nasiadka proposed: [openstack/diskimage-builder] 982231: Add support for building Ubuntu Resolute (26.04) https://review.opendev.org/c/openstack/diskimage-builder/+/982231 | 10:49 | |
| -@gerrit:opendev.org- Jason Paroly proposed: [openstack/diskimage-builder] 982639: Remove unnecessary return statement https://review.opendev.org/c/openstack/diskimage-builder/+/982639 | 12:29 | |
| -@gerrit:opendev.org- Thales Elero Cervi proposed wip: [openstack/project-config] 982495: Add repo app-prometheus to StarlingX https://review.opendev.org/c/openstack/project-config/+/982495 | 13:11 | |
| -@gerrit:opendev.org- Thales Elero Cervi proposed wip: [openstack/project-config] 982495: Add repo app-prometheus to StarlingX https://review.opendev.org/c/openstack/project-config/+/982495 | 13:22 | |
| -@gerrit:opendev.org- Thales Elero Cervi marked as active: [openstack/project-config] 982495: Add repo app-prometheus to StarlingX https://review.opendev.org/c/openstack/project-config/+/982495 | 13:46 | |
| -@gerrit:opendev.org- Michal Nasiadka proposed: [openstack/diskimage-builder] 982231: Add support for building Ubuntu Resolute (26.04) https://review.opendev.org/c/openstack/diskimage-builder/+/982231 | 14:38 | |
| @mnasiadka:matrix.org | fungi: Sorry to call you to the whiteboard, but can you have a look in https://review.opendev.org/c/opendev/system-config/+/982473 ? (at least the dns change before serial becomes stale because of some other urgent dns entry) :) | 14:58 |
|---|---|---|
| @fungicide:matrix.org | is there not a corresponding aaaa (ipv6) record for that server? mirror02 has one | 15:15 |
| @clarkb:matrix.org | fungi: I don't think openmetal has ipv6 | 15:16 |
| @clarkb:matrix.org | oh heh its too early thats the ovh server | 15:16 |
| @clarkb:matrix.org | the server does report a /128 global address from `ip addr` | 15:17 |
| @fungicide:matrix.org | yeah, i'm assuming our launch script didn't spit out a v6 address in the cut-n-paste output | 15:17 |
| @clarkb:matrix.org | I wonder if there is a bug in our launch script since ovh ipv6 is weird and may not be properly recorded by the neutron output? | 15:17 |
| @clarkb:matrix.org | https://etherpad.opendev.org/p/zev9kHUEZ3fe29YweKMa how does this look for announcing the gerrit upgrade. In particular I'm not super tied to the 2000 UTC time. Last time we did this I think that time was chosen to make it easier for tony.breeds to particiapte. I'm happy to use that time block again if tony.breeds is interested. But if not I'm happy to go earlier | 15:24 |
| -@gerrit:opendev.org- Zuul merged on behalf of Michal Nasiadka: [opendev/zone-opendev.org] 982472: Add mirror03.bhs1.ovh.opendev.org https://review.opendev.org/c/opendev/zone-opendev.org/+/982472 | 15:25 | |
| @fungicide:matrix.org | lgtm | 15:25 |
| @fungicide:matrix.org | once that ^ deploys i'll approve the inventory addition | 15:25 |
| @clarkb:matrix.org | fungi: mnasiadka I guess we can add aaaa records later without much trouble | 15:27 |
| @fungicide:matrix.org | yes, and public_v6 in the inventory | 15:27 |
| @clarkb:matrix.org | fungi: does that date and time work? its a Sunday afternoon for us | 15:33 |
| @fungicide:matrix.org | i'm fine with it | 15:33 |
| @clarkb:matrix.org | happy to adjust to accomodate those who are interested in helping if there is interest and that time does not work | 15:33 |
| @clarkb:matrix.org | cool | 15:33 |
| @clarkb:matrix.org | I added one more short paragraph to the etherpad draft email to notify of expected shorter restarts after the openstack release on april 1 as part of our prep work | 15:39 |
| @fungicide:matrix.org | also lgtm, thanks! | 15:47 |
| @clarkb:matrix.org | fungi: since you were following along last week: I'm not sure if you saw but I got lodgeit working with python3.14. Well enough to accept a paste and return it back to me on a held test node: https://217.182.140.117/show/bQOIcjv579pgclVa6Tvl/ | 15:56 |
| @clarkb:matrix.org | I feel like that was sufficient prove things work to feel comfortable moving forward with https://review.opendev.org/c/opendev/system-config/+/982180 as long as we don't think python 3.13 is a hard requirement that we should step through instead | 15:57 |
| @fungicide:matrix.org | sounds good. i'm trying to free up a moment to look at mounting a custom image (even just a blank one) over top of the logo in the anubis container on your held lists99 node | 16:03 |
| @clarkb:matrix.org | I got email from github notifying of new jitsi meet docker images today. Calling that out since the PTG is happening in a few weeks. We should remember to stick the servers in the emergency file to prevent them automatically updating on us during that event | 16:14 |
| @mnasiadka:matrix.org | Ok, let me add AAAA and v6 inventory entry for mirror3.bhs1.ovh | 16:18 |
| -@gerrit:opendev.org- Michal Nasiadka proposed: [opendev/system-config] 982682: Add mirror03.bhs1.ovh v6 address https://review.opendev.org/c/opendev/system-config/+/982682 | 16:18 | |
| @mnasiadka:matrix.org | Will keep an eye for that for next mirrors | 16:18 |
| -@gerrit:opendev.org- Michal Nasiadka proposed: [opendev/zone-opendev.org] 982683: Add AAAA entry for mirror03.bhs1.ovh https://review.opendev.org/c/opendev/zone-opendev.org/+/982683 | 16:20 | |
| -@gerrit:opendev.org- Michal Nasiadka proposed: [opendev/zone-opendev.org] 982683: Add AAAA entry for mirror03.bhs1.ovh https://review.opendev.org/c/opendev/zone-opendev.org/+/982683 | 16:21 | |
| -@gerrit:opendev.org- Michal Nasiadka proposed: [opendev/system-config] 982682: Add mirror03.bhs1.ovh v6 address https://review.opendev.org/c/opendev/system-config/+/982682 | 16:21 | |
| @clarkb:matrix.org | I ssh'd in and grabbed the ip address reported by the host and confirm that it matches what is in those two changes | 16:24 |
| @scott.little:matrix.org | https://review.opendev.org/c/openstack/project-config/+/982495 +1 | 16:25 |
| @mnasiadka:matrix.org | fungi: Is missing +w on https://review.opendev.org/c/openstack/project-config/+/951395 by purpose, or can I push it through the gates? | 16:28 |
| -@gerrit:opendev.org- Zuul merged on behalf of Michal Nasiadka: [opendev/system-config] 982473: Add new mirror in bhs1.ovh - mirror03 https://review.opendev.org/c/opendev/system-config/+/982473 | 16:29 | |
| @fungicide:matrix.org | mnasiadka: it was in case sean-k-mooney wanted to drop the jobs from unmaintained branches entirely | 16:39 |
| -@gerrit:opendev.org- Zuul merged on behalf of Michal Nasiadka: [opendev/zone-opendev.org] 982683: Add AAAA entry for mirror03.bhs1.ovh https://review.opendev.org/c/opendev/zone-opendev.org/+/982683 | 16:39 | |
| @fungicide:matrix.org | or in case other reviewers felt more strongly about it | 16:41 |
| @fungicide:matrix.org | though i guess stephen was the one who uploaded the latest patchset | 16:42 |
| -@gerrit:opendev.org- Zuul merged on behalf of Michal Nasiadka: [opendev/system-config] 982682: Add mirror03.bhs1.ovh v6 address https://review.opendev.org/c/opendev/system-config/+/982682 | 16:47 | |
| @clarkb:matrix.org | I've just done a first pass update on the team meeting agenda. Feel free to edit that wiki page today and I'll send that out later. Or let me know what is missing and I can add it | 16:50 |
| @mnasiadka:matrix.org | fungi: well, it only affects 2026.1 and later, the rest stays on focal/jammy - which I guess is reasonable since resolute is coming soon with python 3.14 | 16:51 |
| @clarkb:matrix.org | I'll also send that gerrit upgrade announcement soon. If anyone other than fungi is interested and has feedback on the timing please let me know | 16:52 |
| -@gerrit:opendev.org- Dmitriy Rabotyagov proposed: [zuul/zuul-jobs] 899230: Add role for uploading Ansible collections to Galaxy https://review.opendev.org/c/zuul/zuul-jobs/+/899230 | 17:04 | |
| @mnasiadka:matrix.org | Clark: should I start tomorrow with next OVH mirror - or do you prefer to wait until bhs1 one is switched over and deleted? | 17:04 |
| @clarkb:matrix.org | mnasiadka: I think we can do them in parallel. Just don't want to have so many in flgiht it is hard to track (2 should be fine) | 17:06 |
| @clarkb:matrix.org | infra-root I just rechecked 680178 to test ovh swift log uploads. If that continues to come back happy I think we should proceed with https://review.opendev.org/c/opendev/base-jobs/+/982477 and reenable ovh log uploads | 17:15 |
| @mnasiadka:matrix.org | Clark: Did not think about doing more than 2, my parallelism is not very good :) | 17:18 |
| @mnasiadka:matrix.org | Ok, mirror03.bhs1.ovh deployed, did some browsing of afs artifacts - looks good | 17:22 |
| -@gerrit:opendev.org- Michal Nasiadka proposed: [opendev/zone-opendev.org] 982700: Promote mirror.bhs1.ovh to new mirror03 https://review.opendev.org/c/opendev/zone-opendev.org/+/982700 | 17:24 | |
| @mnasiadka:matrix.org | Resolute 26.04 patch for DIB needs a second +2 (otherwise that it's not really adding support, just adding testing) - https://review.opendev.org/c/openstack/diskimage-builder/+/982231 | 17:26 |
| @clarkb:matrix.org | https://review.opendev.org/c/zuul/zuul-jobs/+/680178 tests are happy so I think it is safe to proceed with https://review.opendev.org/c/opendev/base-jobs/+/982477 to reenable ovh swift log uploads. If someone else wants to dobule check that before i potentially break half the zuul jobs that would be great :) | 17:45 |
| @fungicide:matrix.org | looks like replacing the mascot images for anubis is nontrivial. from what i can tell, the're the webp files at https://github.com/TecharoHQ/anubis/tree/main/web/static/img but they get encoded and embedded into a js blob that's delivered to the client and rendered client-side? browsers never seem to request those files from the server | 17:54 |
| @fungicide:matrix.org | further, the whole thing is compiled into a monolithic elf executable that runs as the proxy process, even the html/js content is part of that executable | 17:55 |
| @fungicide:matrix.org | i think we'd have to fork the project and build our own container images from source | 17:57 |
| @clarkb:matrix.org | yes the business version docs hinted that they make it difficult | 18:07 |
| @clarkb:matrix.org | I wasn't sure how that would manifest, but embedding things straight into the js that is then embedded into an elf would do it | 18:07 |
| @clarkb:matrix.org | I'm going to work on lunch shortly. When I get back I'll send the gerrit upgrade announcement email and then still happy to reenable ovh log uploads if they look good to at least one other person | 18:44 |
| @fungicide:matrix.org | looking at comments in a lwn article about anubis, i came across yet another bot blocker i hadn't seen yet: https://git.gammaspectra.live/git/go-away | 18:51 |
| @fungicide:matrix.org | (was developed as a more configurable replacement for anubis, sourcehut switched to it in july) | 18:53 |
| -@gerrit:opendev.org- Zuul merged on behalf of Thales Elero Cervi: [openstack/project-config] 982495: Add repo app-prometheus to StarlingX https://review.opendev.org/c/openstack/project-config/+/982495 | 19:12 | |
| @clarkb:matrix.org | Gerrit upgrade timing is official: https://lists.opendev.org/archives/list/service-announce@lists.opendev.org/thread/WXKGKK4L2E57K3OBCAEYLBDERA2POFGV/ | 20:04 |
| @fungicide:matrix.org | `docker: Error response from daemon: unknown: failed to resolve reference "quay.io/opendevmirror/registry:2": unexpected status from HEAD request to https://quay.io/v2/opendevmirror/registry/manifests/2: 502 Bad Gateway` https://zuul.opendev.org/t/openstack/build/a086146143ec4236ae5c5c7460f752c2 | 20:23 |
| -@gerrit:opendev.org- Zuul merged on behalf of Jason Paroly: [openstack/diskimage-builder] 982639: Remove unnecessary return statement https://review.opendev.org/c/openstack/diskimage-builder/+/982639 | 20:24 | |
| @fungicide:matrix.org | i too get a `502 Bad Gateway` response from the url | 20:24 |
| @fungicide:matrix.org | looks like even top-level https://quay.io/ is erroring for me, `504 Gateway Time-out` | 20:26 |
| @fungicide:matrix.org | i guess they're having a bad day | 20:26 |
| @fungicide:matrix.org | anyway, https://review.opendev.org/c/opendev/system-config/+/982180 isn't going to merge for now, i guess | 20:27 |
| @fungicide:matrix.org | i'll recheck it once things seem to have cleared up there | 20:28 |
| @clarkb:matrix.org | how exciting (not really and I hope they figure it out soon) | 20:30 |
| @clarkb:matrix.org | fungi: I don't think quay impacts https://review.opendev.org/c/opendev/base-jobs/+/982477 any concern with approving that change now? | 20:31 |
| @fungicide:matrix.org | none, i went ahead and approved now | 20:32 |
| @clarkb:matrix.org | thanks! I'll keep an eye out for trouble | 20:32 |
| @clarkb:matrix.org | I'm likely going to pop out for a bike ride in a couple of hours before the rain comes back tomorrowish. So now is your last reminder to check the meeting agenda if there was something important you wanted to get added to that. I think I managed to update it to the current relevant stuff, but it wouldn't surprise me to have overlooked something | 20:33 |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/base-jobs] 982477: Revert "Disable job log uploads to ovh swift" https://review.opendev.org/c/opendev/base-jobs/+/982477 | 20:37 | |
| @clarkb:matrix.org | https://zuul.opendev.org/t/openstack/build/b69d045f03ad446194898f0f13c33ccd/logs this build appears to have uploaded to ovh successfully | 20:47 |
| @fungicide:matrix.org | i was able to get the main quay page to load for me, but it was *very* slow | 20:48 |
| @fungicide:matrix.org | https://status.redhat.com/incidents/04vm68m00t61 "quay.io API failures: We are seeing 502s with quay.io pushes & pulls [Posted 37 minutes ago. Mar 30, 2026 - 20:11 UTC]" | 20:49 |
| @fungicide:matrix.org | i guess that's the incident to watch | 20:51 |
| @mordred:waterwanders.com | fungi: so ... if I wanted to, say, unsubscribe from openstack-dev@ ... how would I do that? | 20:55 |
| @mordred:waterwanders.com | I tried signing up for an account on lists.o.o but it only shows me subscribed to Edge-computing and service-discuss | 20:56 |
| @jim:acmegating.com | mordred: i wonder if it's just under a different address; maybe look at the headers and see what the recipient address is? | 20:57 |
| @fungicide:matrix.org | mordred: openstack-dev hasn't existed for years, i guess you're thinking about openstack-discuss. mailman v3 is set up in a multi-domain configuration so check lists.openstack.org instead of lists.opendev.org | 20:57 |
| @fungicide:matrix.org | edge-computing and service-discuss are lists on lists.opendev.org which is distinct from lists.openstack.org | 20:58 |
| @mordred:waterwanders.com | AHHHH | 20:58 |
| @fungicide:matrix.org | back to quay news: "Update: The team has identified the issue and is investigating. We've shifted Quay to be read-only now so pulls should gradually begin restoring but pushes will still fail. [Posted 1 minute ago. Mar 30, 2026 - 20:58 UTC]" | 20:59 |
| @fungicide:matrix.org | so i guess let's not recheck anything still | 20:59 |
| @mordred:waterwanders.com | that was it, btw, thanks! | 21:00 |
| @fungicide:matrix.org | of course! | 21:01 |
| @fungicide:matrix.org | any time | 21:01 |
| @clarkb:matrix.org | I think you can still send the unsubscribe email to the list address too | 21:07 |
| @clarkb:matrix.org | Google says there is a special $listname-leave@domain address to use | 21:08 |
| @mordred:waterwanders.com | totally worked when I logged in to the right place :) | 21:08 |
| @fungicide:matrix.org | still no new updates on the quay outage | 22:07 |
| @fungicide:matrix.org | #status log Pruned /opt/backups-202010 on backup02.ca-ymq-1.vexxhost reducing utilization from 95% to 68% | 22:40 |
| @status:opendev.org | @fungicide:matrix.org: finished logging | 22:40 |
| @fungicide:matrix.org | i'll stepping away for the evening, but will check back in on the quay situation tomorrow and recheck the base-python image change if things look better | 22:41 |
| @clarkb:matrix.org | thanks! I too am going to pop out for the bike ride now. Meeting agenda should be in inboxes | 22:41 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!