| corvus | there is a problem with the sql optimization; i have a fix in 924116 which i have self-approved; when it merges we should restart all zuul-schedulers and web | 00:05 |
|---|---|---|
| fungi | lgtm, thanks! | 00:10 |
| fungi | i guess doing both makes the query far slower? | 00:11 |
| corvus | yeah; the straight join only works as long as we don't join to provides; but the good news is that for queries where we specify provides, there's a lot of columns we filter on so the query planner makes good decisions in that case | 00:16 |
| fungi | oh awesome | 00:16 |
| corvus | i would describe the runtime of the straight join with provides as "indefinite"; it's caused the zuul tenant to stop processing for a few hours | 00:16 |
| corvus | (i've dequeued everything there and killed the sql queries manually to get the change through) | 00:17 |
| corvus | (you need stacked changes with provides/requires to see the problem, which is what i've been doing today, so that's how i noticed) | 00:17 |
| corvus | (and why only the zuul tenant is currently affected) | 00:18 |
| fungi | oh wow, okay so definitely non-performant ;) | 00:18 |
| corvus | #status log restarted zuul-scheduler and -web to pick up 924116 | 01:42 |
| opendevstatus | corvus: finished logging | 01:42 |
| corvus | much better now | 01:43 |
| frickler | corvus: it seems like the restart may have left a large number of stuck buildsets in the periodic{,-weekly} pipelines. if I try to look at the console log of some of the jobs that are meant to be in progress, it shows "build xyz not found" | 09:32 |
| frickler | I'll leave this in place for you to have a look at, but possible some manual cleanup might be needed | 09:33 |
| frickler | sorry, not --weekly, but periodic-stable. the jobs in periodic actually seem to be progressing. for later comparison, there are 83 buildsets in that pipeline currently | 09:36 |
| frickler | oh, maybe the event queue processing is failing. "Events: 0 trigger events, 0 management events, 498 results." | 09:36 |
| frickler | actually it looks like the events are getting processed, just slowly. down to 470 now. guess I'll just check back later | 09:57 |
| opendevreview | Jay Faulkner proposed openstack/diskimage-builder master: [gentoo] Fix+Update CI for 23.0 profile https://review.opendev.org/c/openstack/diskimage-builder/+/923985 | 20:27 |
| opendevreview | Jay Faulkner proposed openstack/diskimage-builder master: [gentoo] Fix+Update CI for 23.0 profile https://review.opendev.org/c/openstack/diskimage-builder/+/923985 | 20:48 |
| opendevreview | Harry Kominos proposed openstack/diskimage-builder master: feat: Add new autoupdates element https://review.opendev.org/c/openstack/diskimage-builder/+/924069 | 21:37 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!