| SvenKieske | Hey, anyone any idea why a CI log collection run would fail with ssh host key change error? https://zuul.opendev.org/t/openstack/build/eb93d47119144046ac06b05bdab3bcfa | 12:48 |
|---|---|---|
| frickler | SvenKieske: that is a known, but rather seldom issue, we mostly attribute it to networking being a bit messed up in a cloud, causing a public IP getting handed out to two different instances | 12:57 |
| SvenKieske | ok, then it's now also known to me :) | 13:00 |
| fungi | yeah, sometimes server instances get "lost" and left behind, still responding to arp requests for their assigned address even after openstack thinks the address has been freed up and reassigns it to another instance | 13:05 |
| fungi | i've heard some providers just run periodic scans for them across all hypervisor hosts, checking with a tool like virsh to find leftover virtual machines and terminate them | 13:06 |
| *** ykarel_ is now known as ykarel | 14:01 | |
| ykarel | Hi we seeing issue with meetpad https://meetpad.opendev.org/neutron-ci-meetings | 14:04 |
| ykarel | TypeError: Cannot read properties of undefined (reading 'skinName') | 14:04 |
| ykarel | at unknown at line -1 | 14:04 |
| ykarel | ErrorId: TwpFpi6TkyNRMdLNGJU7 | 14:04 |
| ykarel | Unhandled Promise rejection | 14:04 |
| ykarel | URL: https://meetpad.opendev.org/etherpad/p/neutron-ci-meetings?showControls=true&showChat=false&showLineNumbers=true&useMonospaceFont=false&userName=yatin | 14:04 |
| ykarel | UserAgent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36 | 14:04 |
| fungi | ykarel: yes, there is temporarily an issue due to a regression in a recent etherpad upgrade, we're addressing it with https://review.opendev.org/c/opendev/system-config/+/929305 | 14:11 |
| fungi | hopefully we'll have it back in shape later today | 14:11 |
| fungi | videoconference functionality should still work, you'll just need to open the corresponding etherpad in a second tab/window to access it in the meantime | 14:12 |
| ykarel | fungi, thx for the details, yeap we worked around like ^ | 14:19 |
| fungi | infra-root: the held etherpad bugfix sample is at 158.69.73.113 and https://etherpad.opendev.org/p/isitbroken seems fine based on my testing | 14:26 |
| clarkb | fungi: cool, deciding whether or not we want to upgrade to the random commit was on my todo list for this morning. There has been no response to my question on the related issue and no 2.2.5 release | 14:47 |
| clarkb | also I may haev used .dev wrong in the tag version but I think thats fine as long as we have a distinct value | 14:47 |
| fungi | yeah, i think going ahead makes the most sense, our users are clearly noticing | 14:50 |
| clarkb | once I've had some tea and breakfast I can check the held node for any problems on my end and if it still looks good we can proceed then | 14:51 |
| fungi | sounds great! | 14:52 |
| clarkb | fungi: https://review.opendev.org/c/openstack/project-config/+/929169/ and child add nested virt to our additional clouds that can support it (theoretically). https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/926970 and https://review.opendev.org/c/openstack/project-config/+/926848 are the ubuntu noble ansible-lint updates for those repos | 14:53 |
| clarkb | they are the main things in my backlog other than etherpad if you ahve time to review them that would be great | 14:53 |
| fungi | you bet | 14:53 |
| opendevreview | Merged openstack/project-config master: Add rockylinux nodes to openmetal provider https://review.opendev.org/c/openstack/project-config/+/929169 | 15:00 |
| opendevreview | Merged openstack/project-config master: Add nested virt labels to raxflex and openmetal providers https://review.opendev.org/c/openstack/project-config/+/929170 | 15:05 |
| clarkb | johnsom: ^ fyi | 15:10 |
| corvus | do we really want to fully qualify all the module names? that makes ansible so much less fun and easy to use | 15:17 |
| clarkb | corvus: I'm indifferent. It does seem to be the way that ansible wants you to write modules | 15:21 |
| clarkb | eg that isn't just the linter's opinion but the official ansible stance? | 15:21 |
| corvus | write modules or write plays? | 15:22 |
| clarkb | sorry I meant that is how ansible wants you to write out the module names/refer to them where they are used | 15:23 |
| corvus | yeah, it just seems like such a waste for the builtins which is most of what we do | 15:26 |
| clarkb | I'm happy to disable that rule. Do we want to do that as a followup and leave those changes as is or disable it and undo the updates in those changes (so that we don't end up with a mix) | 15:29 |
| corvus | i don't feel strongly about it (i realize i'm saying "let's use the old not-recommended syntax") and i also don't do much work in those repos. but to the extent that this conversation may be relevant for opendev-related repos, that's how i feel. :) | 15:32 |
| clarkb | fungi: I have checked the held node and agree it seems to be functional | 15:37 |
| fungi | cool, i suppose i can go ahead and approve the change | 15:57 |
| JayF | FYI; update worked on a wiki page just now, but it did have a SQL error on save. Clever me refreshed the page immediately to see if the change took effect ... before copying out the error. I think it's probably/a blip but wanted to share. | 15:57 |
| fungi | JayF: i've noticed that it's been really slow today too. my guess is crawlers hitting it again | 15:59 |
| fungi | hopefully we can up the priority of swapping the new containerized deployment into production rsn | 16:00 |
| fungi | the error was probably an insert/update query timeout | 16:00 |
| opendevreview | Merged opendev/system-config master: Update etherpad to current develop branch state https://review.opendev.org/c/opendev/system-config/+/929305 | 16:33 |
| clarkb | the image is promoting now then we should automatically deploy ^ | 16:36 |
| clarkb | it has been up for almost a minute and reports being healthy | 16:39 |
| clarkb | a random pad loads for me. Now to load meetpad | 16:39 |
| clarkb | pad loads in meetpad now | 16:39 |
| fungi | awesome. thanks for the quick fix! | 16:39 |
| clarkb | looks like we got alerts for two ssl certs expiring a couple days ago but not again last night. I guess that implies the latest run was successful? | 17:23 |
| clarkb | something to keep an eye on | 17:23 |
| fungi | yeah, since i didn't see any repeats i didn't dig into it | 17:23 |
| opendevreview | Julia Kreger proposed openstack/diskimage-builder master: docs: add two contextual warnings to the replace-partition element https://review.opendev.org/c/openstack/diskimage-builder/+/928819 | 18:09 |
| opendevreview | Clark Boylan proposed openstack/diskimage-builder master: Update functests https://review.opendev.org/c/openstack/diskimage-builder/+/923896 | 20:47 |
| clarkb | I've made some edits to the meeting agenda for tomorrow. Anything else to add/change/remove before I send that out in a bit? | 22:12 |
| fungi | i've got nothing | 22:14 |
| clarkb | as a heads up it looks like pypi package index requests may be unreliable in at least some cloud regions | 22:53 |
| clarkb | I don't think this is anything to be terribly concerned about unless it persists. I want to say we've seen this in the past and had to force access to pypi over ipv4 instead of ipv6 or similar | 22:53 |
| JayF | https://github.com/pypi/warehouse/issues/16725 they fixed it | 22:54 |
| JayF | or at least, they claim to have fixed it | 22:54 |
| clarkb | cool the last occurence I saw of this was before they said they fixed it | 22:57 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: Drop logic for old python versions https://review.opendev.org/c/openstack/diskimage-builder/+/927781 | 23:27 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!