| opendevreview | James E. Blair proposed zuul/zuul-jobs master: Add zuul-tenant-conf-check role/job https://review.opendev.org/c/zuul/zuul-jobs/+/907363 | 00:13 |
|---|---|---|
| tonyb | clarkb: I'm still working through it but basically grab a UUID from `openstack server list` if all of the following are true; 1) there isn't a matching entry in nova.instances, 2) there is an entry in nova_cell0.instances in state deleted then you can delete the entry from nova_api.build_requests | 00:28 |
| tonyb | I've cleared out all of those, but there are 12 records (from dating back to June 2023) that were scheduled but never built than need to be cleaned out so it's off to openstack-nova for me. | 00:29 |
| opendevreview | Tim Burke proposed opendev/bindep master: Add classifiers for py310, py311 https://review.opendev.org/c/opendev/bindep/+/907366 | 00:36 |
| Clark[m] | Those are db edits? Those look like schemas and tables | 00:36 |
| tonyb | Okay AFAICT InMotion is "fixed". I need food but I'll write up what I did for review | 01:21 |
| tonyb | Clark[m]: Yup Had to get into the DB to correct the state. | 01:21 |
| *** elodilles is now known as elodilles_afk | 10:33 | |
| *** liuxie is now known as liushy | 11:43 | |
| *** elodilles_afk is now known as elodilles | 15:21 | |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: Add zuul-tenant-conf-check role/job https://review.opendev.org/c/zuul/zuul-jobs/+/907363 | 15:40 |
| clarkb | infra-root I'm done with doctor appointments now. Should we proceed with https://review.opendev.org/c/opendev/system-config/+/907349 ? | 18:09 |
| clarkb | There was a new gitea release too I'll get a change up for that soon | 18:09 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Upgrade gitea to 1.21.5 https://review.opendev.org/c/opendev/system-config/+/907472 | 18:16 |
| fungi | clarkb: yep, i can approve it | 18:22 |
| clarkb | looks like gerrit 3.9.x has a problem preventing offline reindexing? Not sure where that endedup. Also someone is struggling with the lucene version compatibility issue that should have been fixed by 3.9.1 and they indicate they are using 3.9.1 | 18:23 |
| clarkb | half wondering if I should retrigger our CI for gerrit to see what happens | 18:23 |
| fungi | yeah, maybe they regressed it | 18:23 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Rebuild our Gerrit images https://review.opendev.org/c/opendev/system-config/+/907478 | 18:27 |
| clarkb | ok that is a not completely useless change that will do this | 18:27 |
| opendevreview | Merged opendev/system-config master: Update to etherpad 1.9.6 https://review.opendev.org/c/opendev/system-config/+/907349 | 19:15 |
| clarkb | that should deploy in a few minutes | 19:18 |
| clarkb | its done. The etehrpads I've been looking at recently load | 19:25 |
| fungi | yep, https://etherpad.opendev.org/p/keycloak-refresh-2024 is still fine for me and even kept my author color/name | 19:26 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: DNM: Try future Keycloak 24.0 https://review.opendev.org/c/opendev/system-config/+/907253 | 19:28 |
| fungi | just making sure the latest edits for the 23.0 change don't break with the nightly image | 19:28 |
| fungi | since i didn't actually have it using a working rdbms the last time i tried it | 19:29 |
| clarkb | our gerrit images look fine as well as the upgrade. I don't know what that person ran into | 19:51 |
| frickler | this is the second job I see failing due to 500 errors from opendev.org. not sure what went wrong there, but likely something to keep an eye on https://zuul.opendev.org/t/openstack/build/c52483d2b199439880c347ca5be97dd4 | 19:52 |
| clarkb | the 1.21.5 upgrade does fix some sort of resource leak | 19:52 |
| clarkb | and I think there is some database improvements too | 19:53 |
| clarkb | I can clone requirements from all the gite backends currently just fine | 19:55 |
| clarkb | side note jobs should not clone repos that way | 19:55 |
| clarkb | propose-update-constraints should probably have oepnstack/requirements as a required project then it can use the repo in /home | 19:55 |
| clarkb | looks lik ethis job is expecting the old cache locations maybe that is the problem | 19:56 |
| frickler | the script would still pull the upstream remote https://opendev.org/openstack/releases/src/branch/master/tools/clone_repo.sh#L162-L168 | 20:01 |
| frickler | but I also wonder where that old script is coming from. https://opendev.org/openstack/releases/src/branch/master/tools/clone_repo.sh#L148 does have the correct path | 20:06 |
| frickler | oh, we have an old version in project-config https://opendev.org/openstack/project-config/src/branch/master/roles/copy-release-tools-scripts/files/release-tools/clone_repo.sh#L158 | 20:07 |
| opendevreview | Dr. Jens Harbott proposed openstack/project-config master: Use updated git cache dir default path https://review.opendev.org/c/openstack/project-config/+/907492 | 20:12 |
| frickler | clarkb: ^^ weird how this has gone unnoticed for 4 years | 20:13 |
| frickler | but I don't think it would have helped with the earlier failure. we can still discuss whether that update is actually needed or the job can just rely on what zuul provides | 20:14 |
| fungi | i also wonder why we have two copies of the script | 20:30 |
| fungi | as far as why it exists, while i agree that the requirements repo cloning for the job updating constraints doesn't really need to use it, there are other release jobs that need to dynamically decide which projects they're going to act on, and for those i don't know of any good way to use required-projects | 20:33 |
| fungi | take the most common case, the tag-releases job: someone proposes a change to the requirements repo that adds one or more tags to one or more git repositories out of many hundreds of possible repositories. sure the job could be constantly updated to include every release-managed project as a required-project, but then zuul will be tasked with checking out refs for many hundreds of repositories | 20:36 |
| fungi | when the job only needed one or at most a handful. occasionally, at coordinated release time, we do merge a change which creates a few hundred tags, but that's a rarity (happens twice annually) | 20:36 |
| fungi | if there were a way to dynamically adjust the required-projects list for a particular build based on the content of the diff for the corresponding commit, then that could solve it, but that would also cross between some distinct layers of zuul's design in ways i don't think would be positive | 20:38 |
| clarkb | maybe we have two copies because one needs to work from a trusted context? but eventhen it could use the copy in the other repo | 21:00 |
| clarkb | fungi: ya I think you'd just need to set all the repos as requiredon jobs like that which would oprobably cause more problems than only fetching what you need when you need it | 21:01 |
| clarkb | frickler: https://paste.opendev.org/show/bQXY3JB8R0nfngXagDAh/ this is the error | 21:11 |
| clarkb | looking at the code it looks like gitea checks if the user making the request owns the repo and to do that it looks up the user in the db (you'd think it could short circuit for anyonymous requests). And the error is saying there are too many db connections | 21:15 |
| clarkb | currently I see gitea using one db connection that is established and a number are in time wait | 21:18 |
| clarkb | I think the default for mariadb is 150 (I'll check this). Maybe all those time wait connection count against this limit and we should bump it? | 21:18 |
| clarkb | the value on our server appaers to be 100 | 21:19 |
| clarkb | I think we can write a change similar to fungi's for keycloak to mix in extra mariadb server settings and bump that up to say 300? | 21:20 |
| fungi | we also do something similar for transfer size in the mailman3 mariadb container | 21:21 |
| clarkb | it is interesting that the docker images must override this as all the docs I can find say the default should be 150 | 21:23 |
| clarkb | and if so it may not be as simple as setting that option | 21:23 |
| opendevreview | Merged openstack/project-config master: Use updated git cache dir default path https://review.opendev.org/c/openstack/project-config/+/907492 | 21:23 |
| clarkb | ok I found it in the container image. /etc/mysqld/my.cnf sets that limit before it loads those extra config files so we're fine to do that | 21:25 |
| clarkb | unfortunately no comment about why they override the default. Just a comment saying # Fine Tuning | 21:25 |
| clarkb | doubleing to 200 is probably enough then. I was doubling 150 to get 300 previously | 21:26 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Increase gitea db connection limit https://review.opendev.org/c/opendev/system-config/+/907500 | 21:37 |
| fungi | heck, the original mariadb default might have been enough | 22:12 |
| fungi | but doubling seems fine to me | 22:12 |
| fungi | i guess the mariadb container maintainers think it needs to be lower when in a container? | 22:14 |
| fungi | maybe smaller deployments are more common with containers or something | 22:15 |
| clarkb | ya that could be the motivation | 22:15 |
| tonyb | Okay I think I'm done with inmotion, I'll continue to monitor today. I did a bit of a write-up: https://etherpad.opendev.org/p/opendev-inmotion_debugging | 22:56 |
| *** tosky_ is now known as tosky | 23:14 | |
| fungi | thanks tonyb! | 23:48 |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: Add zuul-tenant-conf-check role/job https://review.opendev.org/c/zuul/zuul-jobs/+/907363 | 23:56 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!