| @jim:acmegating.com | we *were* having problems because we weren't timing out.... | 01:33 |
|---|---|---|
| @jim:acmegating.com | we are not running the new timeout code yet, that merged tuesday | 01:36 |
| @jim:acmegating.com | also i think that was only on downloads, not uploads | 01:36 |
| @jim:acmegating.com | yeah i don't think this is on us | 01:41 |
| -@gerrit:opendev.org- Ivan Anfimov proposed wip: [openstack/diskimage-builder] 1002749: Replaced usage outdate egrep to grep https://review.opendev.org/c/openstack/diskimage-builder/+/1002749 | 13:05 | |
| -@gerrit:opendev.org- Ivan Anfimov proposed wip: [openstack/diskimage-builder] 1002749: Replaced usage outdate egrep to grep https://review.opendev.org/c/openstack/diskimage-builder/+/1002749 | 13:10 | |
| -@gerrit:opendev.org- Ivan Anfimov marked as active: [openstack/diskimage-builder] 1002749: Replaced usage outdate egrep to grep https://review.opendev.org/c/openstack/diskimage-builder/+/1002749 | 13:10 | |
| @clarkb:matrix.org | corvus: interesting. I wonder if we need keepalives in the transfer layer to keep a firewall or proxy happy. | 15:12 |
| @jim:acmegating.com | this part of the process is handled by openstacksdk | 15:13 |
| @jim:acmegating.com | (so it is possible if there was a behavior change, it came from there) | 15:13 |
| @clarkb:matrix.org | it is library release time for openstack. But releases June 24, August 3, August 26, and August 27 don't really line up with when I think this started | 15:14 |
| @clarkb:matrix.org | It would be the ~saturday after each release when we rollout the new containers, but even then late july doesn't line up | 15:15 |
| @clarkb:matrix.org | with two recent releases I do wonder if we should just check back in on Monday and see if anything changes over the weekend with the new rollout | 15:18 |
| @jim:acmegating.com | i can't say how this info would affect that analysis (maybe not at all), but keep in mind that we *also* have the issue with threads getting stuck and jamming up the whole sequence (should be fixed by the upcoming download timeout change in the next restart), and that is also interacting with the ansible playbook getting stuck on apt upgrades -- so our "regular weekly updates/restarts" have been very irregular for the past several weeks | 15:20 |
| @jim:acmegating.com | i agree it makes sense to start debugging this more once we're sure we're on the current version | 15:23 |
| @clarkb:matrix.org | ya sounds like the next update should give us a good baseline to work with. | 15:24 |
| @jim:acmegating.com | In addition to the container/python upgrades, there is a significant change in the log streaming pipeline in this weekend's zuul updates. just in case there's something wrong with that, i have prepared a revert change. the idea is that if we need to do an emergency revert, we can use the speculative image from https://review.opendev.org/1002784 to get working executors without that change (but still with all the other updates); once it's restored, we can then update the commit message for that change and merge it. | 15:30 |
| hopefully we don't need any of that, but i'll feel better having the image build already there if we do | ||
| @clarkb:matrix.org | Any chance we can land https://review.opendev.org/c/opendev/irc-meetings/+/1002668 before next week to make that time more official? (Trying this separate office hours thing in a more APAC timezone friendly tiem slot) | 15:30 |
| @clarkb:matrix.org | thank you for the heads up and the easy rollback prep | 15:31 |
| @clarkb:matrix.org | I am going to approve https://review.opendev.org/c/opendev/system-config/+/1002423 nowish to update our irc/matrix bots to python3.14. I will work on a limnoria restart once that is all done (all Friday meetings should be complete by now) | 15:35 |
| @fungicide:matrix.org | yes, at least the openstack release meeting is wrapped up | 15:36 |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/irc-meetings] 1002668: Add OpenDev Office Hour Meeting https://review.opendev.org/c/opendev/irc-meetings/+/1002668 | 15:39 | |
| @clarkb:matrix.org | fungi: https://review.opendev.org/c/opendev/system-config/+/1000379 improves selenium screenshots against lists with anubis and the parent does the same for gitea (I noticed the problem with gitea first). Should be test only for both services so pretty safe for a Friday | 15:40 |
| @clarkb:matrix.org | https://review.opendev.org/c/opendev/system-config/+/1002179 and parent are slightly more impactful changes to gitea. 1002179 adds optional support for a shared anubis key and the parent ensures we restart gitea when config files for anubis or docker compose change | 15:40 |
| @fungicide:matrix.org | i'll try to get to them here in a bit, but agree today is a good time for the anubis key thing | 15:53 |
| @clarkb:matrix.org | fungi: note that I haven't put a secret vars key in place yet | 16:04 |
| @clarkb:matrix.org | But that shouldn't take long if we decide to proceed | 16:05 |
| -@gerrit:opendev.org- Zuul merged on behalf of Anil Shashikumar Belur: [opendev/system-config] 1002423: Build irc and matrix bots on python3.14 https://review.opendev.org/c/opendev/system-config/+/1002423 | 16:06 | |
| @clarkb:matrix.org | that restarted both matrix-eavesdrop and the limnoria irc bot | 16:12 |
| @clarkb:matrix.org | looks like the irc bot has rejoined at least one channel. I will double check logs are recording as expected too | 16:12 |
| @clarkb:matrix.org | https://meetings.opendev.org/irclogs/%23opendev/%23opendev.2026-08-28.log that the last couple of messages in it so matrix lgtm | 16:13 |
| @clarkb:matrix.org | https://meetings.opendev.org/irclogs/%23openstack-tc/%23openstack-tc.2026-08-28.log has fungi's latest message there so I think this is working | 16:16 |
| @fungicide:matrix.org | great | 16:16 |
| @clarkb:matrix.org | accessbot is still running | 16:16 |
| @clarkb:matrix.org | and accessbot was successful. So ya that all looks good to me. Gerrit is the last one in the stack and I was hoping that they would have a bugfix release we could combine restarts for. Seems we're due for one of those releases | 16:22 |
| @fungicide:matrix.org | Clark: not urgent, but 1002179 lgtm so if you want to add a random `gitea_anubis_shared_key` to the private gitea group vars or let me know once you've done so, i'll approve the change and help track deployment progress | 18:54 |
| @fungicide:matrix.org | er, if you want **me** to add it or want to do it yourself | 18:54 |
| @fungicide:matrix.org | i just don't want to duplicate effort | 18:55 |
| @clarkb:matrix.org | fungi: I will add it now then I'll eat lunch if you want to approve it | 19:09 |
| @fungicide:matrix.org | on it | 19:09 |
| @clarkb:matrix.org | fungi: don't forget you have to review and approve the parent too | 19:10 |
| @fungicide:matrix.org | d'oh! | 19:12 |
| @fungicide:matrix.org | i missed it had a parent, but yeah that lgtm too | 19:12 |
| @fungicide:matrix.org | thanks for pointing it out. friday afternoon and all that | 19:13 |
| @clarkb:matrix.org | the secret has been created so we should be set now | 19:13 |
| @fungicide:matrix.org | awesome, i'm around and can keep an eye on it | 19:13 |
| @clarkb:matrix.org | feel free to double check it. I Think its supposed to be 64 chars for 32 bytes of openssl generated info | 19:13 |
| @fungicide:matrix.org | one of my many eyes | 19:13 |
| @fungicide:matrix.org | the gitea anubis changes should be merging any moment | 20:32 |
| -@gerrit:opendev.org- Zuul merged on behalf of Anil Shashikumar Belur: [opendev/system-config] 1002426: Restart gitea when its config files change https://review.opendev.org/c/opendev/system-config/+/1002426 | 20:34 | |
| @fungicide:matrix.org | the other one is right behind | 20:35 |
| @clarkb:matrix.org | and that first one should be a noop in prod | 20:35 |
| @fungicide:matrix.org | yeah, the second is at the point of taking screenshots | 20:35 |
| @fungicide:matrix.org | okay, it's wrapping up now | 20:40 |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/system-config] 1002179: Add support for shared anubis secret keys in gitea https://review.opendev.org/c/opendev/system-config/+/1002179 | 20:42 | |
| @fungicide:matrix.org | the first is nearly done in deploy so that should start going momentarily | 20:46 |
| @fungicide:matrix.org | and there we are, infra-prod-service-gitea is now running for it | 20:48 |
| @fungicide:matrix.org | gitea09 is already done so impact should be seen if there is any | 20:49 |
| @fungicide:matrix.org | i'll run manual tests once all of them are deployed | 20:50 |
| @fungicide:matrix.org | nearly there | 20:54 |
| @clarkb:matrix.org | web still seems to work for me | 20:58 |
| @clarkb:matrix.org | if we really want to test the key thing we could figure out which backend one of us is talking to and then remove it from the proxy to force us onto a different one then check that we don't do the cahllenge again | 20:59 |
| @clarkb:matrix.org | but that is probably overkill. I think it is sufficient to know that things generally work | 21:00 |
| @fungicide:matrix.org | yeah, everything's still working normally for me as well, so i think if we get complaints we'll look into them but it seems fine | 21:04 |
| -@gerrit:opendev.org- Monty Taylor https://matrix.to/#/@mordred:inaugust.com proposed: [openstack/project-config] 1002832: Add a few more wandertracks and inaugust repos https://review.opendev.org/c/openstack/project-config/+/1002832 | 22:04 | |
| -@gerrit:opendev.org- Monty Taylor https://matrix.to/#/@mordred:inaugust.com proposed: [zuul/zuul-jobs] 1001597: Get artifacts from builds via buildset https://review.opendev.org/c/zuul/zuul-jobs/+/1001597 | 23:01 | |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!