| @bbezak:matrix.org | hi - it looks like zuul has some issues. status of changes are not refreshing, some jobs are showing "Build ID xxxx not found" - for example - https://zuul.opendev.org/t/openstack/status?change=977959 | 13:12 |
|---|---|---|
| @bbezak:matrix.org | interesting spike here - https://grafana.opendev.org/d/21a6e53ea4/zuul-status?orgId=1&from=2026-03-02T13:00:00.000Z&timezone=utc&viewPanel=panel-36&to=2026-03-02T14:00:00.000Z | 14:07 |
| @harbott.osism.tech:regio.chat | do you still see issues? seems to have somehow repaired itself by now | 14:23 |
| @fungicide:matrix.org | #status log Rebooted wiki.openstack.org restoring it to working order again after further database disconnects | 14:41 |
| @status:opendev.org | @fungicide:matrix.org: finished logging | 14:41 |
| @bbezak:matrix.org | Jens Harbott: it is better now. however still status refresh in zuul is a bit sluggish - for instance after pushing new patchset - it is being refreshed after several minutes - before it was almost instant | 15:19 |
| @fungicide:matrix.org | Bartosz Bezak: that will vary since zuul listens to events from gerrit about uploads and those go into an event queue. zuul pauses processing that queue during some other kinds of activities, such as reconfigurations | 15:30 |
| @fungicide:matrix.org | right now zuul has a backlog of 221 trigger events and 1 management event, according to https://zuul.opendev.org/t/openstack/status | 15:30 |
| @fungicide:matrix.org | and now you can see the events queues are moving again, and the reconfiguration time updated, so it was paused for another tenant reconfiguration that time. according to https://zuul.opendev.org/t/openstack/system-events is was for the WRCP_26.03 branch of the starlingx/root repository | 15:37 |
| @fungicide:matrix.org | and the event queues seem to be paused again, likely another tenant reconfiguration | 15:39 |
| @bbezak:matrix.org | got it, thx for the info | 15:39 |
| @mnasiadka:matrix.org | fungi: That tenant reconfig caused a bit of stall on everything for some time, I'd say around 15/20 minutes, which is not that great | 15:44 |
| @clarkb:matrix.org | it is likely due to the googlesource gerrit | 15:52 |
| @clarkb:matrix.org | corvus: has a change up to drop it. Doing so will break our gerrit container image build jobs as they rely on the repos on that gerrit. I think this is the only place we rely on repos from that gerrit and we can update the jobs to clone them within the job rather than relying on the zuul git repo management | 15:53 |
| @clarkb:matrix.org | corvus: it looks like my local access to gerrit.googlesource.com may also be slow? And the web ui isn't loading properly. I wonder if this is more of a global problem for that service | 15:55 |
| @tkajinam:matrix.org | yeah it doesn't load for me now, either | 15:56 |
| @tkajinam:matrix.org | hmm it seems the access causes randome result. sometimes it succeeds. sometimes it fails with 500/502/503 error... | 15:57 |
| @mnasiadka:matrix.org | https works for me | 15:58 |
| @mnasiadka:matrix.org | quite fast | 15:58 |
| @clarkb:matrix.org | I've just asked in their discord server. They noted that the gerrit code review server was slow and deployed a rollback which corrected things. I've asked if the source code hosting portion is related or similarly impacted | 15:58 |
| @mnasiadka:matrix.org | (although it required a google id login) | 15:58 |
| @bbezak:matrix.org | I've seen 502 couple of times | 15:58 |
| @clarkb:matrix.org | mnasiadka: ah maybe they are throttling anonymous access heavily. Hopefully I get an answer from their discord server | 15:59 |
| @clarkb:matrix.org | and if not we can drop the source and update our jobs | 15:59 |
| @tkajinam:matrix.org | mnasiadkado you see consistent result in several reloads ? | 15:59 |
| @mnasiadka:matrix.org | yup, 20 retries and no failure | 16:00 |
| @mnasiadka:matrix.org | but it seems I'm going over ipv6 :) | 16:00 |
| @clarkb:matrix.org | I have joined the opendevent meetpad room and we can discuss all the gerrit things there if we wish | 16:01 |
| @mnasiadka:matrix.org | Clark: can't join today, but will be there tomorrow | 16:01 |
| @clarkb:matrix.org | mnasiadka: ack thanks | 16:02 |
| @clarkb:matrix.org | It feels more ad hoc this time around due to my lack of consistent availability recently. But I'm happy to make the most of it and will see you tomorrow | 16:02 |
| @fungicide:matrix.org | oh, sorry, omw! | 16:13 |
| -@gerrit:opendev.org- Zuul merged on behalf of Dmitriy Rabotyagov: [openstack/project-config] 978084: Stop publishing collection to PyPi https://review.opendev.org/c/openstack/project-config/+/978084 | 16:35 | |
| -@gerrit:opendev.org- Michal Nasiadka proposed: [openstack/project-config] 978566: propose-updates: Add test-requirements target https://review.opendev.org/c/openstack/project-config/+/978566 | 16:52 | |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/system-config] 978575: Build Gerrit using direct clones of repos not Zuul managed repos https://review.opendev.org/c/opendev/system-config/+/978575 | 19:06 | |
| @clarkb:matrix.org | corvus: ^ I went with your suggestion of a new pre-run playbook that simply stages the repos in teh same locations that zuul would. I won't be surprised if there ends up being some bugs in my implementation but I think this approach will work just fine once I track those down | 19:07 |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/system-config] 978575: Build Gerrit using direct clones of repos not Zuul managed repos https://review.opendev.org/c/opendev/system-config/+/978575 | 19:33 | |
| @clarkb:matrix.org | I think cloning the git repos is also super slow and I'm hitting timeouts in pre-run so it retries. I'll have to look closer after lunch | 20:28 |
| @jim:acmegating.com | Clark: random anecdata: my test curl command running on my workstation is now fast | 20:57 |
| @jim:acmegating.com | 0.5s | 20:57 |
| @clarkb:matrix.org | corvus: oh looks like the later runs for my change were fast then hit another error | 21:03 |
| @clarkb:matrix.org | the first one I checked (the first attempt) had some slow clones in it) | 21:03 |
| @clarkb:matrix.org | so maybe this is happier all around now? | 21:03 |
| @clarkb:matrix.org | browsing the web ui seems to work well too | 21:04 |
| @clarkb:matrix.org | I'm going to make this change work for completeness in case the problem comes back. Upstream hasn't made any indication that they have fixed anything yet so this seems possible | 21:04 |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/system-config] 978575: Build Gerrit using direct clones of repos not Zuul managed repos https://review.opendev.org/c/opendev/system-config/+/978575 | 21:06 | |
| @clarkb:matrix.org | I wasn't planning on sending an infra team meeting agenda for tomorrow since we're doing opendevent. I guess I can send an email notifying of the change of venue instead | 21:07 |
| @clarkb:matrix.org | looks like the last reconfiguration for the openstack tenant was ~2 hours ago so not sure if we'd see better results within zuul yet. | 21:11 |
| @clarkb:matrix.org | quick someone land a change to update configs :) | 21:11 |
| -@gerrit:opendev.org- David Shrewsbury proposed: [opendev/bindep] 978585: New exit code of 2 for input file parse errors. https://review.opendev.org/c/opendev/bindep/+/978585 | 21:40 | |
| @shrews:matrix.org | fungi: I feel like there was a magic incantation for generating a new release note. Is that documented somewhere so I can refresh my brain? | 21:59 |
| @shrews:matrix.org | or am i misremembering how that worked? | 21:59 |
| @fungicide:matrix.org | there is. it's the https://pypi.org/p/reno tool but you can run it under tox | 22:00 |
| @shrews:matrix.org | ah, reno, yes. thx! | 22:01 |
| @fungicide:matrix.org | er, under nox now, we switched from tox to nox | 22:01 |
| @mordred:waterwanders.com | zomg it's shrews | 22:02 |
| @shrews:matrix.org | zomg, it's the old gang is together again or something. heya mordred ! | 22:03 |
| @shrews:matrix.org | * zomg, it's like the old gang is together again or something. heya mordred ! | 22:03 |
| @fungicide:matrix.org | mmm, actually running it under nox is a little complicated, so probably easier to just install reno into a temporary venv (or from a distro package) and then `path/to/reno new some-summary-name` in the top-level directory of the bindep repo | 22:05 |
| @shrews:matrix.org | mordred: come visit me and olaph again. he has a new farm with plenty of space for cooking out and beer drinking | 22:06 |
| @shrews:matrix.org | fungi: gotcha. i'll figure it out | 22:06 |
| @mordred:waterwanders.com | shrews: heck yes. it's been way too long | 22:07 |
| @clarkb:matrix.org | corvus: according to discord things may be better now which seems to reflect what we have observed since noonish pacific. That said they also said they aren't aware of any specific changes that were made to fix anything | 22:46 |
| @clarkb:matrix.org | so I'm thinking maybe we hold off on my gerrit image build update change and the gerrit.googlesource.com connection removal from zuul for now and monitor? My change does seem to work at least so we can fall back to the connection removal and that update if we need to | 22:47 |
| @jim:acmegating.com | Clark: sounds like a good plan | 22:47 |
| @clarkb:matrix.org | screenshots show the expected gerrit versions and the theming is in place so I think we managed to build what we expected | 22:48 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!