Thursday, 2024-02-01

opendevreviewJames E. Blair proposed zuul/zuul-jobs master: Add zuul-tenant-conf-check role/job
tonybclarkb: I'm still working through it but basically grab a UUID from `openstack server list` if all of the following are true; 1) there isn't a matching entry in nova.instances, 2) there is an entry in nova_cell0.instances in state deleted then you can delete the entry from nova_api.build_requests00:28
tonybI've cleared out all of those, but there are 12 records (from dating back to June 2023) that were scheduled but never built than need to be cleaned out so it's off to openstack-nova for me.00:29
opendevreviewTim Burke proposed opendev/bindep master: Add classifiers for py310, py311
Clark[m]Those are db edits? Those look like schemas and tables00:36
tonybOkay AFAICT InMotion is "fixed".  I need food but I'll write up what I did for review01:21
tonybClark[m]: Yup Had to get into the DB to correct the state.01:21
*** elodilles is now known as elodilles_afk10:33
*** liuxie is now known as liushy11:43
*** elodilles_afk is now known as elodilles15:21
opendevreviewJames E. Blair proposed zuul/zuul-jobs master: Add zuul-tenant-conf-check role/job
clarkbinfra-root I'm done with doctor appointments now. Should we proceed with ?18:09
clarkbThere was a new gitea release too I'll get a change up for that soon18:09
opendevreviewClark Boylan proposed opendev/system-config master: Upgrade gitea to 1.21.5
fungiclarkb: yep, i can approve it18:22
clarkblooks like gerrit 3.9.x has a problem preventing offline reindexing? Not sure where that endedup. Also someone is struggling with the lucene version compatibility issue that should have been fixed by 3.9.1 and they indicate they are using 3.9.118:23
clarkbhalf wondering if I should retrigger our CI for gerrit to see what happens18:23
fungiyeah, maybe they regressed it18:23
opendevreviewClark Boylan proposed opendev/system-config master: Rebuild our Gerrit images
clarkbok that is a not completely useless change that will do this18:27
opendevreviewMerged opendev/system-config master: Update to etherpad 1.9.6
clarkbthat should deploy in a few minutes19:18
clarkbits done. The etehrpads I've been looking at recently load19:25
fungiyep, is still fine for me and even kept my author color/name19:26
opendevreviewJeremy Stanley proposed opendev/system-config master: DNM: Try future Keycloak 24.0
fungijust making sure the latest edits for the 23.0 change don't break with the nightly image19:28
fungisince i didn't actually have it using a working rdbms the last time i tried it19:29
clarkbour gerrit images look fine as well as the upgrade. I don't know what that person ran into19:51
fricklerthis is the second job I see failing due to 500 errors from not sure what went wrong there, but likely something to keep an eye on
clarkbthe 1.21.5 upgrade does fix some sort of resource leak19:52
clarkband I think there is some database improvements too19:53
clarkbI can clone requirements from all the gite backends currently just fine19:55
clarkbside note jobs should not clone repos that way19:55
clarkbpropose-update-constraints should probably have oepnstack/requirements as a required project then it can use the repo in /home19:55
clarkblooks lik ethis job is expecting the old cache locations maybe that is the problem19:56
fricklerthe script would still pull the upstream remote
fricklerbut I also wonder where that old script is coming from. does have the correct path20:06
frickleroh, we have an old version in project-config
opendevreviewDr. Jens Harbott proposed openstack/project-config master: Use updated git cache dir default path
fricklerclarkb: ^^ weird how this has gone unnoticed for 4 years20:13
fricklerbut I don't think it would have helped with the earlier failure. we can still discuss whether that update is actually needed or the job can just rely on what zuul provides20:14
fungii also wonder why we have two copies of the script20:30
fungias far as why it exists, while i agree that the requirements repo cloning for the job updating constraints doesn't really need to use it, there are other release jobs that need to dynamically decide which projects they're going to act on, and for those i don't know of any good way to use required-projects20:33
fungitake the most common case, the tag-releases job: someone proposes a change to the requirements repo that adds one or more tags to one or more git repositories out of many hundreds of possible repositories. sure the job could be constantly updated to include every release-managed project as a required-project, but then zuul will be tasked with checking out refs for many hundreds of repositories20:36
fungiwhen the job only needed one or at most a handful. occasionally, at coordinated release time, we do merge a change which creates a few hundred tags, but that's a rarity (happens twice annually)20:36
fungiif there were a way to dynamically adjust the required-projects list for a particular build based on the content of the diff for the corresponding commit, then that could solve it, but that would also cross between some distinct layers of zuul's design in ways i don't think would be positive20:38
clarkbmaybe we have two copies because one needs to work from a trusted context? but eventhen it could use the copy in the other repo21:00
clarkbfungi: ya I think you'd just need to set all the repos as requiredon jobs like that which would oprobably cause more problems than only fetching what you need when you need it21:01
clarkbfrickler: this is the error21:11
clarkblooking at the code it looks like gitea checks if the user making the request owns the repo and to do that it looks up the user in the db (you'd think it could short circuit for anyonymous requests). And the error is saying there are too many db connections21:15
clarkbcurrently I see gitea using one db connection that is established and a number are in time wait21:18
clarkbI think the default for mariadb is 150 (I'll check this). Maybe all those time wait connection count against this limit and we should bump it?21:18
clarkbthe value on our server appaers to be 10021:19
clarkbI think we can write a change similar to fungi's for keycloak to mix in extra mariadb server settings and bump that up to say 300?21:20
fungiwe also do something similar for transfer size in the mailman3 mariadb container21:21
clarkbit is interesting that the docker images must override this as all the docs I can find say the default should be 15021:23
clarkband if so it may not be as simple as setting that option21:23
opendevreviewMerged openstack/project-config master: Use updated git cache dir default path
clarkbok I found it in the container image. /etc/mysqld/my.cnf sets that limit before it loads those extra config files so we're fine to do that21:25
clarkbunfortunately no comment about why they override the default. Just a comment saying # Fine Tuning21:25
clarkbdoubleing to 200 is probably enough then. I was doubling 150 to get 300 previously21:26
opendevreviewClark Boylan proposed opendev/system-config master: Increase gitea db connection limit
fungiheck, the original mariadb default might have been enough22:12
fungibut doubling seems fine to me22:12
fungii guess the mariadb container maintainers think it needs to be lower when in a container?22:14
fungimaybe smaller deployments are more common with containers or something22:15
clarkbya that could be the motivation22:15
tonybOkay I think I'm done with inmotion,   I'll continue to monitor today.  I did a bit of a write-up:
*** tosky_ is now known as tosky23:14
fungithanks tonyb!23:48
opendevreviewJames E. Blair proposed zuul/zuul-jobs master: Add zuul-tenant-conf-check role/job

Generated by 2.17.3 by Marius Gedminas - find it at!