19:00:10 <clarkb> #startmeeting infra
19:00:10 <opendevmeet> Meeting started Tue Jan 30 19:00:10 2024 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:00:10 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:00:10 <opendevmeet> The meeting name has been set to 'infra'
19:01:18 <clarkb> #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/L6CVNQQLCEZCK3I7I7FQUC7VV2UWSBHM/ Our Agenda
19:01:24 <clarkb> #topic Announcements
19:01:31 <clarkb> Service coordinator nominations open February 6, 2024 - February 20, 2024
19:01:36 <clarkb> #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/TB2OFBIGWZEYC7L4MCYA46EXIX5T47TY/
19:01:38 <clarkb> That is one week away
19:01:41 <clarkb> rouhgly
19:02:00 <clarkb> OpenDev PrePTG February 14+15 at 14:00-16:00UTC and 22:00-00:00UTC both days on our meetpad service.
19:02:06 <clarkb> #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/KUZG7DPU77GC3M3MKKTSYY655JLIYV4Z/
19:02:21 <clarkb> and the Pre PTG is two weeks away
19:02:50 <clarkb> #topic Server Upgrades
19:03:03 <clarkb> #link https://review.opendev.org/c/opendev/system-config/+/905510 Upgrading meetpad service to jammy
19:03:25 <clarkb> fungi: frickler maybe you have time to review those changes? Otherwise I think I may work with tonyb and proceed with things with just my reviews
19:03:49 <clarkb> (which coincidentally is one of the agenda items for the pre ptg, whether or not we want to do more singe reviewer approvals)
19:04:56 <clarkb> #link https://etherpad.opendev.org/p/opendev-bionic-server-upgrades
19:05:13 <clarkb> this etherpad also has notes on a wiki replacement plan. If you haven't read them that may be a good idea.
19:05:43 * frickler makes notes to take a look, also at those reviews
19:06:43 <clarkb> thanks!
19:06:50 <clarkb> #topic Python container updates
19:07:34 <clarkb> Nothing new to mention here. I think we're pretty much stalled out on the zuul oeprator stuff. But it isn't urgent. I may just remove this from the next meeting's agenda
19:08:40 <clarkb> #topic AFS quota issues
19:09:01 <clarkb> somewhere on my todo list that keeps getting overlooked is ubuntu ports cleanup. /me adds it again and maybe it will get looked at
19:10:35 <clarkb> #topic Broken wheel build issues
19:10:48 <clarkb> the wheels are building and being written to afs now
19:10:58 <clarkb> The remaining problem has to do with volume selection for vos release
19:11:04 <clarkb> #link https://review.opendev.org/c/openstack/project-config/+/907150 Will fix centos 8 and 9 stream
19:11:29 <clarkb> In theory that change will have fixed these problems and everything centos 8/9 stream wheel related will be happy
19:11:43 <clarkb> I did notice thatwhile centos 7 was working when this first came up it is not working now
19:11:47 <clarkb> I haven't had a chance to look at it
19:12:57 <clarkb> one thing I noticed is that I'm not sure these wheels were ever really working properly for stream
19:13:15 <clarkb> that may put more weight into the "maybe we should just shut it down side of things"
19:13:49 <clarkb> however if things work now then letting it continue to be happy is least effort
19:14:23 <frickler> til next time it breaks ;)
19:14:24 <clarkb> #topic Keycloak upgrade struggles
19:14:28 <clarkb> ya exactly :)
19:15:23 <clarkb> recently fungi and I started poking at keycloak upgrades. A little while back keycloak moved from wildfly as its runtime to quarkus. When they did this they didn't really make the container image migration a simple direct one
19:15:40 <clarkb> this resulted in us getting stuck behind on a legacy set of images and now we are trying to figure out how to get onto the new quarkus stuff
19:15:48 <fungi> but also some useful discoveries
19:16:06 <clarkb> The biggest issue we've run into is that the existing service uses an h2 databse but keycloak considers h2 to not be production worthy so there is no migration path for that
19:16:14 <clarkb> file paths and names and db credentials have all changed
19:16:35 <clarkb> this seems to be a clear sign we should move to a real db so that is what fungi has been doing most recently. Getting mariadb up alongside keyloack
19:16:40 <fungi> i did try some basic migration which made it into production briefly over the weekend before i rolled it back due to breakage
19:16:58 <clarkb> ya including renaming and moving files
19:16:58 <fungi> it would be good for someone who has successfully used keycloak.opendev.org recently to test that it's working as expected again
19:17:45 <corvus> yep wfm
19:18:03 <fungi> anyway, for a number of reasons, it appears that trying to "migrate" our existing data will be extremely involved (switching database format, upgrading through several keycloak releases, also we should move to latest ubuntu lts while we're at it)
19:18:09 <tonyb> and me
19:18:52 <fungi> given that the service is used by at most half a dozen people in a non-production-critical path right now, it will probably be less work to set it up from scratch on the latest keycloak images on new ubuntu
19:18:56 <clarkb> ya the end result of all this wheel spinning seems to be we should do a new deployment with that we have laerned then rebuild the currently small set of integration we have using it
19:19:09 <clarkb> then we should be in a much better position for the future and being able to upgrade things
19:19:17 <fungi> i've got a change underway (close to complete i think) to do that
19:19:28 <fungi> just adding some more hygiene to the testing now
19:19:46 <fungi> like switching to group_vars and using keycloak99 for the test node, etc
19:20:39 <fungi> tests so far are passing, i've been adding some safety like making sure keycloak and mariadb only listen on the loopback address
19:21:23 <fungi> anyway, hopefully we can have a new keycloak02.opendev.org up by the time for our pre-ptg and start talking about moving forward some more on the sso spec
19:21:30 <clarkb> sounds good
19:21:37 <tonyb> +1
19:22:25 <fungi> i didn't have anything else for this topic, but maybe folks have questions
19:22:39 <corvus> plan sgtm, thanks!
19:23:07 <clarkb> #topic InMotion/OpenMetal Cloud RabbitMQ Problems
19:23:22 <clarkb> wanted to follow up on this. frickler's hunch was correct. rabbitmq had a sad
19:23:49 <clarkb> tonyb: can speak more to this than I can but initially a simple docker restart of rabbitmq containers was done to try and get rabbit to cooperate and that didn't help at all
19:24:08 <clarkb> then later a restart using kolla tooling was done and that got things working again
19:24:24 <clarkb> however there are still a number of leaked nodes we need to clean up which likely involve deeper admin work
19:24:50 <tonyb> Yup that one.
19:24:51 <clarkb> one thing I have had to do in the past to make nodes deletable is to reset their task state if they are already in a deleting task state because nova won't retry if task state is already deleting
19:25:10 <clarkb> then there is another issue we've encountered which had to do with a mismatch in cell accounting
19:25:28 <clarkb> melwitt helped the cells issue if we need help again
19:26:08 <tonyb> I'll get more details and then reach out on nova/melwitt
19:26:39 <tonyb> in the next 24-48 hours e should have things "sorted"
19:27:20 <clarkb> thanks!
19:27:29 <clarkb> #topic Open Discussion
19:27:44 <clarkb> That was all I had on the agenda. A reminder to add your content and interest to the pre ptg etherpad
19:29:00 <corvus> #link https://etherpad.opendev.org/p/opendev-preptg-202402
19:31:18 <clarkb> sounds like that may be everything
19:31:26 <clarkb> a rare short meeting. THank you for your time everyone!
19:31:38 <clarkb> I'll close this out in a few more minutes if nothing else pops up
19:31:41 <fungi> thanks clarkb!
19:31:59 <tonyb> Thanks all!
19:33:07 <corvus> thanks clarkb
19:35:21 <clarkb> #endmeeting