| -@gerrit:opendev.org- chandan kumar proposed: [openstack/project-config] 996426: Add stable branch IRC notifications for Cyborg https://review.opendev.org/c/openstack/project-config/+/996426 | 08:16 | |
| -@gerrit:opendev.org- Anil Shashikumar Belur proposed: | 08:41 | |
| - [opendev/system-config] 995755: Vendor the MeetBot rendering engine into matrix-eavesdrop https://review.opendev.org/c/opendev/system-config/+/995755 | ||
| - [opendev/system-config] 995756: Add meeting engine integration to matrix-eavesdrop https://review.opendev.org/c/opendev/system-config/+/995756 | ||
| - [opendev/system-config] 995757: Wire meeting support into the matrix-eavesdrop bot https://review.opendev.org/c/opendev/system-config/+/995757 | ||
| - [opendev/system-config] 995758: Deploy matrix-eavesdrop meeting support https://review.opendev.org/c/opendev/system-config/+/995758 | ||
| - [opendev/system-config] 995759: Add a functional test for matrix-eavesdrop meetings https://review.opendev.org/c/opendev/system-config/+/995759 | ||
| -@gerrit:opendev.org- yatin proposed: [zuul/zuul-jobs] 995915: [DNM][multi node bridge] use vxlan tunnel https://review.opendev.org/c/zuul/zuul-jobs/+/995915 | 09:49 | |
| @harbott.osism.tech:regio.chat | infra-root: looks like mirror.gra1.ovh.opendev.org is being sad, I cannot even connect via SSH | 12:17 |
|---|---|---|
| @sean-k-mooney:matrix.org | it looks like its a new issue in the last 24 hours https://tinyurl.com/2chcxd6v | 12:18 |
| @sean-k-mooney:matrix.org | i suspect it not a complete failure and the retireis we are doing with download are masking some of it | 12:19 |
| @sean-k-mooney:matrix.org | i was speculationg on #openstack-infa that disablity the mirror for that region may be a workaround but it woudl be good to get input form the ovh folks if there is a better approch | 12:23 |
| @harbott.osism.tech:regio.chat | trying a reboot via the vnc console now | 12:23 |
| @sean-k-mooney:matrix.org | :) alway a good first step | 12:24 |
| @harbott.osism.tech:regio.chat | it looks like connectivity via IPv6 is still and completely broken. ssh via v4 works (didn't think of explicitly testing that before) | 12:25 |
| @harbott.osism.tech:regio.chat | so maybe we should disable that region for now and then check with OVN? or maybe arnaudm is around and can help? | 12:26 |
| @sean-k-mooney:matrix.org | you could tweak the system to prefer ipv4 | 12:26 |
| @harbott.osism.tech:regio.chat | ``` | 12:29 |
| frickler@mirror04:~$ tracepath -n opendev.org | ||
| 1?: [LOCALHOST] 0.019ms pmtu 1500 | ||
| 1: 2001:41d0:302:1000::1 0.777ms | ||
| 1: 2001:41d0:302:1000::1 0.183ms | ||
| 2: fd00::ffe 0.270ms | ||
| 3: 2001:41d0:302:b10:ffff:ffff:ffff:ffff 0.918ms | ||
| 4: 2001:41d0:0:1:1:b10:2:1 0.413ms | ||
| 5: no reply | ||
| ... | ||
| ``` | ||
| @harbott.osism.tech:regio.chat | ovh status page only shows some maintenances being in progress | 12:29 |
| @fungicide:matrix.org | oh, i hadn't read scrollback in this channel yet, ignore my comments in irc | 12:38 |
| @sean-k-mooney:matrix.org | ya we can continue the conversation here. | 12:40 |
| @fungicide:matrix.org | my isp claims they expect to have the outage for my home broadband resolved by 14:00 utc, so hopefully i can be more help within the hour | 13:05 |
| -@gerrit:opendev.org- Dr. Jens Harbott proposed: [opendev/zuul-providers] 996449: Disable ovh-gra1 region temporarily https://review.opendev.org/c/opendev/zuul-providers/+/996449 | 13:29 | |
| @fungicide:matrix.org | logging into gerrit from my phone is not going to work out, but you have my +2 in spirit | 13:37 |
| @fungicide:matrix.org | okay, my internet is back now | 14:16 |
| @fungicide:matrix.org | approved 996449 | 14:17 |
| -@gerrit:opendev.org- Zuul merged on behalf of Dr. Jens Harbott: [opendev/zuul-providers] 996449: Disable ovh-gra1 region temporarily https://review.opendev.org/c/opendev/zuul-providers/+/996449 | 14:19 | |
| @dpanech:matrix.org | Hi, it seems like one of the pypi mirrors is out of whack in Zuul (??) . This failed: https://zuul.opendev.org/t/openstack/build/eec8a8ba509b4a5ab29963e53fe209dc . But a slightly earlier run succeeded: https://zuul.opendev.org/t/openstack/build/e40ba86e874c42d29c9a0689e7a5bde1 . | 15:43 |
| My AI helper says this: https://paste.opendev.org/show/b4BfcMfPPVSQD6LYYOdO/ | ||
| @fungicide:matrix.org | davlet: is it still happening? we turned down the region where we saw a problem (ovh gra1) | 15:45 |
| @fungicide:matrix.org | we think the cloud provider's staff are going to have to look into a networking problem impacting our mirror server vm in that region | 15:46 |
| @dpanech:matrix.org | fungi: understood, waiting for "recheck" to kick in | 15:59 |
| @fungicide:matrix.org | davlet: to be precise, a little over 1.5 hours ago we merged a change to tell zuul to stop booting new test nodes in the affected region, but jobs which got node assignments filled and had already started running by that time may have still errored out | 16:00 |
| @fungicide:matrix.org | at the moment we're not certain what time the problem started | 16:01 |
| @dpanech:matrix.org | fungi: ok its working now, thanks | 16:24 |
| @fungicide:matrix.org | of course! thanks for checking | 16:28 |
| @fungicide:matrix.org | mnasiadka: apologies for the delay, today was sort of hectic, but i've added your ssh key to the authorized list for the root account on the 7 servers mentioned for openmetal in our passwords list | 17:17 |
| @fungicide:matrix.org | you should be able to directly ssh into them. obviously ignore the vip address included there since that's mapped to one of the other servers | 17:18 |
| -@gerrit:opendev.org- Dr. Jens Harbott proposed: [opendev/zuul-providers] 996510: Revert "Disable ovh-gra1 region temporarily" https://review.opendev.org/c/opendev/zuul-providers/+/996510 | 17:57 | |
| @harbott.osism.tech:regio.chat | infra-root: testing from the mirror node things seem to be working again ^^ but we can also wait until my morning with the revert if you prefer | 17:58 |
| @jim:acmegating.com | Jens Harbott: thanks, i've approved. i'll be around for a while if someone reports a problem. | 18:31 |
| -@gerrit:opendev.org- Zuul merged on behalf of Dr. Jens Harbott: [opendev/zuul-providers] 996510: Revert "Disable ovh-gra1 region temporarily" https://review.opendev.org/c/opendev/zuul-providers/+/996510 | 18:32 | |
| @fungicide:matrix.org | thanks Jens Harbott and corvus! i needed to hurry and do post-vacation yardwork between storms but will be back around the keyboard again in a few | 18:54 |
| @mnasiadka:matrix.org | fungi: frickler added my access earlier, I did have a brief look, restored some vip sanity, but the host with .3 had some weird networking issues with half of the LACP bond in churn actor state (and sort of loosing connectivity when getting out of that state, and then back to churned). I rebooted it in hope of it getting better but no luck. I’m planning to try removing the faulty NIC from the bond tomorrow - do we have any support email or some ticketing system we should use for OpenMetal support? | 19:36 |
| @fungicide:matrix.org | yes, contact info is in the same file as the ip addresses | 19:37 |
| @mnasiadka:matrix.org | Ok, once I get to the bottom of the problem I’ll reach out to them if necessary | 19:43 |
| @fungicide:matrix.org | out of curiosity, did i add redundant copies of your ssh key to the servers, or did you get access set up a different way? | 19:53 |
| @mnasiadka:matrix.org | I think I had access to the first server only directly, but then I could get to others from that server | 19:55 |
| @mnasiadka:matrix.org | So if there’s a redudant entry, it’s only on one - I’ll clean it up in the morning | 19:56 |
| @fungicide:matrix.org | okay, cool, no worries i was just wondering | 19:56 |
| @fungicide:matrix.org | i did check one of the servers and didn't see your key so wrongly assumed you didn't have it on any of them and blindly appended it to all of them | 19:57 |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/system-config] 994565: Make public ip address public not private https://review.opendev.org/c/opendev/system-config/+/994565 | 20:38 | |
| @fungicide:matrix.org | woohoo! | 20:42 |
| @fungicide:matrix.org | now here's hoping we don't need to emergency revert and clean up behind that if something goes wrong, keeping an eye on the deployment now | 20:42 |
| @fungicide:matrix.org | which already succeeded | 20:43 |
| @clarkb:matrix.org | I think that should only affect testing fwiw | 20:43 |
| @fungicide:matrix.org | yes, deployment succeeded quickly, running only infra-prod-bootstrap-bridge | 20:44 |
| @fungicide:matrix.org | yeah, playbooks/zuul/run-base.yaml doesn't seem to be used in deploy jobs | 20:48 |
| @fungicide:matrix.org | https://zuul.opendev.org/t/openstack/build/fa744ffd9f7448898e103f587a59c98e/console#2/3/6/bridge99.opendev.org is where it was exercised in the gate | 20:49 |
| @fungicide:matrix.org | and we only flip the var on the zuul-jobs role in the system-config-run abstract job which isn't used in deployment | 20:51 |
| @fungicide:matrix.org | so this wouldn't have altered production /etc/hosts files anyway | 20:51 |
| @fungicide:matrix.org | but that's now cleared the runway for the prometheus spec changes | 20:52 |
| -@gerrit:opendev.org- Eric Ball proposed: [opendev/system-config] 996553: Update docker images to Python 3.14 https://review.opendev.org/c/opendev/system-config/+/996553 | 21:48 | |
| -@gerrit:opendev.org- Monty Taylor https://matrix.to/#/@mordred:inaugust.com proposed: [opendev/system-config] 996564: Trigger updated python-base image builds https://review.opendev.org/c/opendev/system-config/+/996564 | 23:10 | |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!