| @jim:acmegating.com | i looked at the database query times, and did not see the improvement i expected. the time queries are not using the new index i added. i tracked the problem down to this: i ran mariadb 10.11 in my local testing (the same as opendev prod), but apparently the import process did not trigger the stats updates which normally happen. when i ran 'analyze table' locally, my local queries stopped using the index as well. so it looks like 10.11 with up-to-date statistics has the query planner making a bad choice. | 14:10 |
|---|---|---|
| the good news is that if i upgrade my local instance to mariadb 13, it does use the new index. so i'm guessing there are some query planner improvements there. i think we should upgrade mariadb. | ||
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [opendev/system-config] 1010020: Upgrade zuul mariadb https://review.opendev.org/c/opendev/system-config/+/1010020 | 14:15 | |
| @fungicide:matrix.org | https://zuul.opendev.org/t/openstack/project/opendev.org/openstack/python-ironicclient is missing a stable/2026.2 branch, i suspect zuul missed an event back when we were having ipv6 issues in vexxhost. should i initiate a tenant reconfiguration? | 14:37 |
| -@gerrit:opendev.org- Stephen Finucane proposed: [openstack/project-config] 1003840: Remove references to ironic-core https://review.opendev.org/c/openstack/project-config/+/1003840 | 14:41 | |
| @fungicide:matrix.org | proposing to execute `zuul-scheduler tenant-reconfigure openstack` on zuul01 | 14:46 |
| @jim:acmegating.com | fungi: that'll pause things for a while. as long as you're okay with that, sgtm. | 14:47 |
| @fungicide:matrix.org | what's "a while" in this case (order of magnitude)? minutes? hours? | 14:48 |
| @fungicide:matrix.org | i can arrange to run it later on when activity slows down for the day | 14:48 |
| @fungicide:matrix.org | i suppose we're still at risk of github api rate limits stalling the reconfiguration too? | 14:55 |
| @jim:acmegating.com | fungi: it's hard to say since it's been some time since we've done it, but i think more than 5 minutes... less than 30? | 14:58 |
| @jim:acmegating.com | i doubt github api limits will be an issue; that tended only to happen when we had multiple full reconfigs in sequence | 14:59 |
| @fungicide:matrix.org | thanks, yeah i'll plan to do it later today and try to track the duration it records for future reference | 15:00 |
| @mnasiadka:matrix.org | corvus: Wonder if those improvements are in mariadb 12.3 (which is an LTS) - 13.0 community support ends in 2 months ;-) | 15:01 |
| @jim:acmegating.com | mnasiadka: good question, i'll check! | 15:09 |
| @jim:acmegating.com | mnasiadka: looks good, i'll update my patch | 15:11 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [opendev/system-config] 1010020: Upgrade zuul mariadb https://review.opendev.org/c/opendev/system-config/+/1010020 | 15:17 | |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: | 15:20 | |
| - [opendev/system-config] 1010020: Upgrade zuul mariadb https://review.opendev.org/c/opendev/system-config/+/1010020 | ||
| - [opendev/system-config] 1010030: Mirror mariadb 12.3 https://review.opendev.org/c/opendev/system-config/+/1010030 | ||
| -@gerrit:opendev.org- Jeremy Stanley https://matrix.to/#/@fungicide:matrix.org proposed: [opendev/system-config] 1010032: Add OSUOSL to the Cloud Donors list on our site https://review.opendev.org/c/opendev/system-config/+/1010032 | 15:41 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Stephen Finucane: [openstack/project-config] 1005094: Add approvers to the "special" groups https://review.opendev.org/c/openstack/project-config/+/1005094 | 15:44 | |
| @mordred:waterwanders.com | fungi, sean-k-mooney just was reminded of something that didn't come up in the PTG session where review-style agents are helpful (great example sean about the bug-at-a-distance find) ... agents watching, noticing and paying attention to periodic and post jobs where humans almost never do. "This periodic job failed, I've looked at it and found there is this bug" - which can either than create a bug which people would see or a patch which people would see | 16:22 |
| @fungicide:matrix.org | i need to disappear to run some errands, should be back in about 90 minutes if anything comes up | 16:24 |
| @mnasiadka:matrix.org | > <@mordred:waterwanders.com> fungi, sean-k-mooney just was reminded of something that didn't come up in the PTG session where review-style agents are helpful (great example sean about the bug-at-a-distance find) ... agents watching, noticing and paying attention to periodic and post jobs where humans almost never do. "This periodic job failed, I've looked at it and found there is this bug" - which can either than create a bug which people would see or a patch which people would see | 17:09 |
| That’s a very interesting approach, I hate traversing Kolla logs to find what went wrong in the publish job | ||
| -@gerrit:opendev.org- Jeremy Stanley https://matrix.to/#/@fungicide:matrix.org proposed: [opendev/system-config] 1010032: Add OSUOSL to the Cloud Donors list on our site https://review.opendev.org/c/opendev/system-config/+/1010032 | 19:09 | |
| -@gerrit:opendev.org- Abhishek Annigeri proposed: [openstack/project-config] 1010049: ci: build Skyline Console assets for releases https://review.opendev.org/c/openstack/project-config/+/1010049 | 20:07 | |
| @fungicide:matrix.org | doing tenant-reconfig for openstack now | 22:06 |
| @fungicide:matrix.org | for the record, that ended up being `sudo docker-compose -f /etc/zuul-scheduler/docker-compose.yaml exec scheduler zuul-scheduler tenant-reconfigure openstack` | 22:09 |
| @jim:acmegating.com | 2026-10-12 22:24:25,400 INFO zuul.Scheduler: Reconfiguration complete (smart: False, tenants: ['openstack'], duration: 928.348 seconds) | 22:26 |
| @jim:acmegating.com | between 5 and 30 mins was a reasonable guess | 22:27 |
| @clarkb:matrix.org | Did the branch show up afterwards? | 22:28 |
| @jim:acmegating.com | looks like it | 22:30 |
| @fungicide:matrix.org | yes, https://zuul.opendev.org/t/openstack/project/opendev.org/openstack/python-ironicclient shows a stable/2026.2 now | 22:31 |
| @clarkb:matrix.org | Excellent | 22:31 |
| @fungicide:matrix.org | i'll recheck the affected changes for that particular project+branch, but be aware there may be others nobody's noticed yet | 22:32 |
| @jim:acmegating.com | Clark: if you have 5s for a perfunctory review, merging https://review.opendev.org/1010030 now will get started on a container mirror that could be useful (for the child change) | 22:42 |
| @clarkb:matrix.org | I have reviewed both. Thank you for continuing to track that down | 22:48 |
| @jim:acmegating.com | thanks, and you're welcome! | 22:52 |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [opendev/system-config] 1010030: Mirror mariadb 12.3 https://review.opendev.org/c/opendev/system-config/+/1010030 | 22:57 | |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!