19:00:12 #startmeeting infra 19:00:12 Meeting started Tue Mar 19 19:00:12 2024 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:00:12 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:00:12 The meeting name has been set to 'infra' 19:00:25 #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/2KGXT47R66KRZPHD6I2HXB23SLIKCYOZ/ Our Agenda 19:00:35 #topic Announcements 19:00:50 fungi: made a git-review 2.4.0 release yesterday. Now is a good time to update if you haven't already 19:01:02 having us use the latest version helps find any issues that might be present through our day to day use 19:01:29 And the PTG is fast approaching 19:01:31 #link https://etherpad.opendev.org/p/apr2024-ptg-opendev Get your PTG agenda items on the agenda doc 19:01:32 though also installing the branch tip should always be safe, since we do test it 19:01:51 and helps us find problems prior to tagging a new release 19:02:00 ++ 19:02:25 for the PTG I've gone ahead and added soem stuff. But as previously mentioend I expect ti to be fairly low key on our side and more of an outreach to our users sort of thing 19:02:34 help them with their items rather than push any specific thigns from our side 19:02:43 that said feel free to add content to that etherpad 19:03:43 #topic Server Upgrades 19:04:05 I haven't seen any new movement on this and I've been plenty distracted by old distro cleanups and rax auth and zuul reviews 19:04:30 I suspect that the rest of us may similarly be distracted and not have anything new here. I'll give it a few minuets for others to chime in if I've missed anything before we move on 19:07:28 #topic MariaDB Upgrades 19:07:36 #link https://review.opendev.org/c/opendev/system-config/+/910999 Upgrade refstack mariadb to 10.11 19:07:40 #link https://review.opendev.org/c/opendev/system-config/+/911000 Upgrade etherpad mariadb to 10.11 19:07:54 if I can get reviews on these changes I'm happy to approve and watch the rollout when it is convenient 19:08:21 I think thats mostlywhere this is at. Just sanity checks that we're ok with the process applied to these services and then finding time to do it 19:09:31 #topic AFS Mirror cleanups 19:09:36 CentOS 7 is gone 19:09:43 and there was much rejoicing 19:09:43 thank you everyone for the help getting things into shape to make that possible 19:10:20 if that experience has taught me anything, it's that i'm not looking forward to untangling ubuntu-xenial removal 19:10:23 I did notice that I made some mistakes in cleaning up the opensuse obs centos 7 stuff whcih led to even more cleanups to the opensuse volume 19:10:29 #link https://review.opendev.org/c/opendev/system-config/+/913608 Final OpenSUSE mirror cleanup 19:10:48 this is opensuse cleanup but it is related to centos 7 :) and I think is the last step in finally clearing up both opensuse and centos 7 19:11:12 and ya after that is done it is time to start looking at ubuntu xenial removal which is going to be "fun" 19:11:30 I've already got some changes up under topic:drop-ubuntu-xenial but that is going to be just the tip of the iceberg I expect 19:11:55 more like a continental ice sheet 19:12:12 or at least a modest glacier 19:12:14 The upside to all of this is our afs volume capacity is looking far moer healthy. Which means we should have plenty of space for things like ubuntu 24.04 and maybe others 19:13:12 so ya reviews welcome on that last cleanup change then be on the lookout for xenial removal stuff probably more serisouly after the PTG? I think between now and then I may have too much other stuff to work on 19:13:23 #topic Rebuilding Gerrit Images 19:13:27 Of which this is one :) 19:13:59 I'd like to get our gerrit 3.9 image updated to 3.9.2. One big reason for this is it will make it easier to test 3.9's broken change reindexing to better understand if and how that affects us before we upgrade 19:14:19 sounds like a great idea 19:14:24 but that will also upate our gerrit 3.8 image which means we should couple landing that change with a production restart of gerrit. Might be a good thing for late this week 19:15:06 similar to the db upgrade changes I'm happy if others review the changes then I can approve and babysit as necessary (in this case do a docker-compose pull and down and up -d) 19:15:32 re the 3.9 reindexing issue I also put it on the April 4th gerrit community meeting agenda to discuss the impact and better understand the issue 19:15:47 it seems like a fairly serious problem but they aren't rushing to fix it so I may just lack understanding 19:16:47 #link https://review.opendev.org/c/opendev/system-config/+/912470 Update our 3.9 image to 3.9.2 19:16:56 I forgot to link the change. That chagne is the one that needs reviews :) 19:18:22 #topic Rackspace MFA 19:18:58 We will be required to setup MFA in rax March 26th but last week we decided to opt into it early to find any expected problems in a more controlled manner 19:19:12 as noted yesterday fungi and I have meetings basically all day today so we said we'd do this tomorrow 19:19:22 fungi: I'll just catch up with you tomorrow moring on that and we can work through it? 19:19:41 yep, sounds good to me 19:20:36 #topic Project Renames 19:20:53 preparing for this also in the list of things I need to look at before worrying too much about ubuntu xenial removal 19:21:08 #link https://review.opendev.org/c/opendev/system-config/+/911622 Move gerrit replication queue aside during project renames. 19:21:15 this is a prep change for the playbook itself 19:23:10 Once we're a bit closer we'll need to records changes too but don't need to do that too early 19:24:32 #topic Linaro Cloud Cert Renewal 19:24:47 Our helpful certchecker tool is warning us that this cert will expire on April 15 19:25:07 I've written down the process for renewing the cert on the node itself which si reachable from bridge 19:25:28 its fairly straightforward we run acme.sh to get a new LE cert. Then copy its contents where kolla can find it and run kolla 19:26:21 where is that writeup? I could take a look at this 19:26:31 frickler: its in the root homedir on the server itself 19:27:12 should we port that process document into system-config's docs for posterity, or is it too full of sensitive info? 19:27:22 fungi: we probably could 19:27:35 I don't think it has any sensitive info in the process. 19:28:15 the server is access via bridge (and the server is the last one in our ansible inventory). You ssh in via bridge and then the doc is there. Happy to help port to our proper docs now that we've used thep rocess more than once and didn't need to make big changes ot it 19:28:24 the first time I wrote it down I was distilling from a couple of emails. 19:28:37 but I think tonyb did the last one without any issue off the local doc so ya proper hosting of that is fine 19:28:41 if discoverability is a concern, replacing it with a one-line file linking to the appropriate section of our docs would likely work 19:29:06 ++ 19:29:09 I'll take a look at it later this week, seems it isn't urgent yet 19:29:17 frickler: ya not urgent. Let me know if you have any questions 19:29:30 except for the daily warning mail, which is a good reminder ;) 19:29:45 #topic Nodepool Image Delete After Upload 19:30:01 corvus wanted to point out that this new feature has landed in nodepool and we can make use of it if we like 19:30:30 basically you can update the configs to opt into having nodepool builder clean up on disk image files after they have been uploaded to clouds. Additionally you can opt to keep certain image formats and clear out others. 19:30:57 In our case I think we could choose to delete all but the qcow2 format of the image since that file is the smallest and we can generate the other two (raw and vhd) from the qcow2 if necessary 19:31:03 might be worth asking what we want out of local images; recovery in case of cloud issues? user access for reproducibility? 19:31:32 corvus: currently we do host files for users to do local reproduction but we only serve the qcow2 files to force people into the lower bandwidth option 19:31:52 for this reason I don't think we want to delete all of the image files after upload, but I do think we can get away with keeping only the qcow2 file 19:31:52 so qcow2 for user access would be great then; and i think we're probably not too worried about cloud issues? 19:31:57 we did make them downloadable for user reproducibility reasons, i point people to those with some regularity 19:32:23 corvus: ya I think for cloud issues we have enough clouds that we would fetch from one of them to push into another if absolutely necessary 19:32:33 or just rebuild and let other clouds deal with that particular image type in the interim 19:32:36 like, if we lose a cloud we could probably live with either "let the next rebuild fix it" or "convert it ourselves" if we're in a dire situation. 19:32:43 exactly 19:33:11 or remove that cloud from out configuration temporarily 19:33:24 s/out/our/ 19:34:44 maybe go ahead and push up a change to keep only the qcow2s post upload and if there are any further concerns they can be posted in review? 19:35:17 with a similar reasoning we might be ok with only keeping the latest local image and not multiple iterations? 19:35:44 yes, though i'm not sure if nodepool can express that yet 19:36:13 yes, this was more of like another feature idea 19:37:09 #topic Open Discussion 19:37:27 That took us to the end of the listed agenda but I forgot to add a couple things before sending it 19:37:30 #link https://review.opendev.org/c/opendev/system-config/+/913686 Update Gitea to 1.21.8 19:38:00 This fixes a number of bugs in gitea supposedly. Would be good to get that in to keep up to date 19:38:13 And then separately etherpad released a 2.0 (and then very quickly a 2.0.1) 19:38:37 The changes to etehrpad appear to largely be in how etherpad is installed and managed (they use `pnpm` now I don't know what that means really yet) 19:39:01 It appears that this means we'll have to completely redo our docker image builds but I think the upgrade itself is straightforward as the db doesn't change 19:39:24 pnpm seems to be a package manager for npm 19:39:32 if anyone else wants to look into that please feel free. Our current image is basically a fork of their image because they weren't reliably rebuilding their images for nodejs security issues 19:39:47 So we need to port our fork over to the new stuff and then test the result 19:39:58 https://pnpm.io/ 19:40:15 you'll note that npm is a package manager itsefl 19:40:19 so now we have package manager managers 19:40:33 looks like a meta package manager that tries to use some sort of caching file tree 19:40:52 throws around terms like "content-addressable storage" 19:41:27 if I end up looking at this it will almost certainly be a post PTG task for me 19:41:38 and post PTG is probably fine since big update to etherpad pre PTG are scary 19:41:41 also looks like pnpm positions itself as a more efficient replacement for npm 19:42:49 Anything else? 19:43:17 I feel like there is a lot but we're moving slowly due to the openstack release. I bring that up to reassure people its ok 19:43:33 priorities and all that 19:43:57 pnpm seems to also try t replace yarn 19:44:16 fungi: the js community never found a tool that was too good to replace 19:44:42 i secretly replaced the js community with folgers crystals 19:46:04 sounds like that may be it 19:46:08 Thanks everyone! 19:46:15 #endmeeting