| opendevreview | OpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml https://review.opendev.org/c/openstack/project-config/+/957995 | 02:26 |
|---|---|---|
| *** elodilles_pto is now known as elodilles | 08:19 | |
| opendevreview | Benjamin Schanzel proposed zuul/zuul-jobs master: ensure-dib: Don't report task failure when dib is not installed https://review.opendev.org/c/zuul/zuul-jobs/+/959112 | 09:14 |
| *** dhill is now known as Guest25648 | 13:44 | |
| opendevreview | Clark Boylan proposed opendev/zuul-providers master: Disable raxflex sjc3 for cloud maintenance window https://review.opendev.org/c/opendev/zuul-providers/+/959200 | 14:53 |
| clarkb | infra-root ^ I'll plan to land that after my morning of meetings today then shutdown the mirror in SJC3 so that we're ready for the maintenance window tomorrow | 14:53 |
| clarkb | I'm going to send out our meeting agenda momentarily. Let me know if you want things edited in the next few minutes. I just hit save changes on the wiki with my edits | 15:43 |
| opendevreview | Clark Boylan proposed openstack/diskimage-builder master: Drop unnecessary debian security location configuration switch https://review.opendev.org/c/openstack/diskimage-builder/+/958572 | 17:06 |
| clarkb | my matrix client says connectivity to the server has been lost. Seems to affect all rooms (on different homeservers) and private messages. I feel like that means it islikely an issue on my end rather than the serverside? | 17:45 |
| clarkb | curious if anyone else is suffering through similar | 17:45 |
| clarkb | looking in my browser's network debugging window it looks like matrix-client.matrix.org is returning 429 messages which is likely the source of my matrix connectivity errors | 18:37 |
| clarkb | looks like this is hosted by cloudflare if the server: cloudflare header can be believed. Maybe they changed rate limit settings and boom exploded clients | 18:38 |
| clarkb | https://status.matrix.org/ ok this shows there is a problem. Maybe the 429 is a symptom of them trying to get clients to stay away while they fix it | 18:39 |
| * clarkb will wait patiently | 18:39 | |
| frickler | fwiw there seems to be no issue between my homeserver and opendev | 18:41 |
| clarkb | ya I think this will affect anyone with matrix.org accounts (like me) | 18:41 |
| clarkb | but clients using accounts on other homeservers should be fine as long as they aren't also trying ot reach rooms hosted by matrix.org (which includes the ircbridges) | 18:42 |
| mnasiadka | Does that mean it’s better to run your own server? :-) | 18:55 |
| clarkb | only if you can maintain better uptime than matrix.org | 18:59 |
| opendevreview | Merged opendev/zuul-providers master: Disable raxflex sjc3 for cloud maintenance window https://review.opendev.org/c/opendev/zuul-providers/+/959200 | 19:38 |
| clarkb | "We are in the process of restoring the matrix.org database from a backup. The matrix.org homeserver will be offline until this has been completed" | 20:02 |
| corvus_ | that doesn't sound like a fun day | 20:03 |
| fungi | wow | 20:04 |
| fungi | yes, someone's day is not going to plan there | 20:04 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Exclude django caches from lists.o.o backups https://review.opendev.org/c/opendev/system-config/+/959236 | 20:27 |
| clarkb | I've made some assertions about mailman3 and django and whether or not these caches are worth preserving based on about 5 minutes of googling | 20:27 |
| clarkb | reviewers may want to see if they can find evidence that my assertions are faulty when reviewing that change | 20:27 |
| opendevreview | Clark Boylan proposed opendev/glean master: Switch to new devstack based testing https://review.opendev.org/c/opendev/glean/+/953163 | 20:49 |
| clarkb | infra-root ^ I think we should go ahead and land that change when we land the bionic cleanup fix for glean (I had to rebase and clean some stuff up due to the bionic jobs not working by default anymore) | 20:49 |
| clarkb | looks like rax flex sjc3 dropped to 0 nodes according to grafana about 20 minutes ago | 20:55 |
| clarkb | I'll wait a bit longer then shutdown the mirror node | 20:56 |
| clarkb | last call on https://review.opendev.org/c/openstack/diskimage-builder/+/958572 I'll approve it shortly if there are no objections so that we can work on fixingthe trixie image builds | 21:01 |
| clarkb | corvus_: I think your zuul fix broke test cases. I'm guessing due to some internal fakes/mocks no longer aligning | 21:06 |
| corvus_ | clarkb: yep, it' a test-only thing. i'm running the test suite locally right now to verify the fix so we don't waste any more time | 21:07 |
| clarkb | ack thanks. I'll rereview when pushed | 21:07 |
| corvus_ | clarkb: https://review.opendev.org/c/zuul/zuul/+/959228 updated and is clean locally | 21:21 |
| clarkb | corvus_: +2 thanks | 21:23 |
| clarkb | oh I also forgot about ze11. Updating the emergency file to include the sjc3 mirror reminded me | 21:27 |
| clarkb | the system had pending updates so I'm rebooting it first just to be sure it comes up cleanly then I'll shut it back down again in prep for tomorrow | 21:29 |
| clarkb | and it has been added to the emergency file so ansible should ignore it | 21:29 |
| clarkb | reboot looked good so I've shut it down now. Nova reports status is SHUTOFF. We should be ready for tomorrows maintenance now | 21:33 |
| opendevreview | Merged openstack/diskimage-builder master: Drop unnecessary debian security location configuration switch https://review.opendev.org/c/openstack/diskimage-builder/+/958572 | 21:56 |
| clarkb | ok I'm going to approve the trixie fix now that ^ has merged | 21:57 |
| clarkb | https://review.opendev.org/c/opendev/zuul-providers/+/958561 this change for the record | 21:57 |
| clarkb | looks like ppa access may be sad right now for the trixie image builds :/ | 22:11 |
| corvus_ | this is clearly not a good day for sysadmins | 22:27 |
| clarkb | corvus_: I think the zuul fixup change hit some network blip too | 22:39 |
| gouthamr | https://ptg.opendev.org/ is down | 22:42 |
| gouthamr | ^ totally unrelated i'd guess, was curious if it was planned | 22:43 |
| clarkb | not planned | 22:44 |
| gouthamr | ack, was just looking for a link to the team signups.. i don't think we'd worry for a few more weeks | 22:45 |
| clarkb | I see the problem. I should be able to fix it pretty quickly | 22:46 |
| clarkb | just need to test it first | 22:46 |
| clarkb | gouthamr: I don't think ptg.opendev.org tells you has has signed up until diablo_rojo configures the bot | 22:48 |
| clarkb | gouthamr: but this is a bug in the hosting after the irc services moved to a new service. I'll fix that aspect | 22:48 |
| gouthamr | ah, i was looking for this form clarkb - thought i'd find it on the website: https://openinfrafoundation.formstack.com/forms/oct2025_ptg_survey | 22:49 |
| opendevreview | Clark Boylan proposed opendev/zone-opendev.org master: Point ptg.opendev.org to the new eavesdrop02 server https://review.opendev.org/c/opendev/zone-opendev.org/+/959253 | 22:50 |
| clarkb | gouthamr: ^ that is the fix for the web service not being accessible | 22:51 |
| clarkb | I'll probably just go ahead and self approev that if ci passes | 22:51 |
| gouthamr | ah nice! ty clarkb | 22:51 |
| opendevreview | Merged opendev/zone-opendev.org master: Point ptg.opendev.org to the new eavesdrop02 server https://review.opendev.org/c/opendev/zone-opendev.org/+/959253 | 22:56 |
| clarkb | gouthamr: once your dns cache evicts the old record you should see that ptg.opendev.org works again. The TTL was 1 hour so approximately an hour from now | 22:59 |
| gouthamr | yes! Works | 22:59 |
| clarkb | I've got to wait ~2600 seconds locally | 22:59 |
| corvus_ | clarkb: fetching the zuul-scheduler image from quay.io took 10m which exceeded the startup timeout for one of the bootstrapping scripts | 23:15 |
| corvus_ | so, continuing today's theme i guess | 23:16 |
| clarkb | corvus_: the third retry for the trixie build does seem to have run further than the first two attempts at least | 23:18 |
| clarkb | and I'm under 1500 seconds on my dns cache timeout to confirm I fixed ptg.opendev.org (I could dig out my router credentials and restart unbound but meh) | 23:19 |
| opendevreview | Merged opendev/zuul-providers master: Build actual trixie now that it is released https://review.opendev.org/c/opendev/zuul-providers/+/958561 | 23:34 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!