19:00:33 <clarkb> #startmeeting infra
19:00:33 <opendevmeet> Meeting started Tue May  7 19:00:33 2024 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:00:33 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:00:33 <opendevmeet> The meeting name has been set to 'infra'
19:00:45 <clarkb> #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/HDXZPH7XJI5O5SDOCC35RGUALV7SUDZS/ Our Agenda
19:00:59 <clarkb> We have an agenda but I half expect this to go quickly as I believe attendance may be light
19:02:28 <clarkb> #topic Announcements
19:02:48 <clarkb> I don't have anythign to announce other than that I expect this to go quickly and attendance to be light
19:02:57 <clarkb> #topic Upgrading old servers
19:03:13 <clarkb> With the release of Noble the year long countdown to getting off of focal has started
19:03:17 <clarkb> #link https://etherpad.opendev.org/p/opendev-focal-server-upgrades
19:03:38 <clarkb> tonyb has started putting notes together for the needs there. I added some annotations and flavor to try and categorize things
19:03:51 <clarkb> tonyb: would it be preferable for me to just go ahead and rearrange or do you like having the notes?
19:04:07 <tonyb> Whatever works for you.
19:04:22 <tonyb> I certainly have no objection to you moving things around
19:04:44 <tonyb> I just did the bit I knew and figured I'd slowly figure the rest out
19:05:58 <clarkb> sounds good
19:06:11 <clarkb> The other idea I had here was to maybe try and consolidate the various server upgrade todo lists into one big list?
19:06:21 <clarkb> then we don't need new documents over time as we have different upgrades to do
19:07:27 <clarkb> I guess one reason not to do that is things might get too cluttered. I'm open to feedback on that
19:07:47 <tonyb> Yup we can do that.   Should be easy enough.   I'm happy to give it a try and see what it looks like.
19:07:52 <clarkb> thanks!
19:08:11 <clarkb> I did also want to note that your changes to prep meetpad upgrades to jammy have landed and the changes to add the new servers are up
19:08:28 <tonyb> Yup.
19:08:46 <clarkb> as mentioned in review on the inventory update I think we want to do that with the old servers in the emergency file so that we don't end up with a weird cluster state but otherwise I think this should be straightforward
19:08:47 <tonyb> As you pointed out we probably need a little care before takign the next step.
19:09:33 <tonyb> I'll double check, but I think we're good to go. perhaps Thursday sometime?
19:10:34 <clarkb> that should work for me
19:10:47 <tonyb> cool beans
19:10:48 <clarkb> mornings are generally better simply because then I feel more confident we'll have time to work through any issues over the day
19:11:37 <clarkb> #topic MariaDB Upgrades
19:11:51 <tonyb> Yup.   We can figure out a time that works with TZ, and real world commitments in #opendev
19:12:00 <clarkb> ++
19:12:07 <clarkb> #link https://review.opendev.org/c/opendev/system-config/+/916848 Update Gerrit's Mariadb
19:12:23 <clarkb> this is still lacking reviews. I'm happy to approve this change if others review it and give it +2s
19:12:41 <tonyb> I thought I had.
19:12:43 <tonyb> I have now
19:12:53 <clarkb> specifically I think we should plan to do a short gerrit downtime after the change merges to ensure we've got the upgrade in place after merging it as this won't happen automatically
19:12:55 <clarkb> thanks!
19:13:22 <clarkb> but other than that change and the manual tasks to deploy it all mariadb upgrades are complete
19:13:33 <clarkb> #topic AFS Mirror Cleanups
19:13:40 <clarkb> topic:drop-ubuntu-xenial has changes up for rewiew.
19:14:02 <clarkb> I think this is largely where we're at now. I'm hoping to avoid a large backlog of changes like this sitting in my open queue. Ideally I write a few and we merge a few and slowly chip away
19:14:35 <clarkb> Looks like a number of them have reviews and I just need to approve them when I can monitor. I think I'll try to do some of that today/tomorrow. We'll see how the days go
19:14:56 <tonyb> Sounds good to me.
19:15:25 <clarkb> #topic Building Ubuntu Noble Images and Test Nodes
19:16:00 <clarkb> As mentioned last week the next step here is adding Noble to our mirror content and babysitting what is likely to be a large set of data that needs to be downloaded and released in afs
19:16:11 <clarkb> in particular i Know we'll need a quota bump on the volume at least.
19:16:38 <clarkb> I don't thik anyone has volunteered for that yet and I'm trying to focus on the cleanups until that is more fully done before I take on adding stuff. But if anyone else has time for this I think we can add content
19:17:18 <tonyb> I doubt I'll have time before this meeting next week but I can probably do it after that.
19:17:50 <tonyb> I have spent some time reading and learning more about AFS so I can try to pick up work items in that area.
19:19:01 <clarkb> cool we can resync at the next meeting then
19:19:11 <clarkb> #topic Gerrit 3.9 Upgrade Planning
19:19:27 <clarkb> #link https://etherpad.opendev.org/p/gerrit-upgrade-3.9 Upgrade prep and process notes
19:19:36 <clarkb> I spent a bit of yesterday doing some testing of this and I have learned new things
19:19:47 <clarkb> #link https://review.opendev.org/c/opendev/system-config/+/918333 Need updated and promoted 3.9 image before doing more testing.
19:20:19 <clarkb> First up is that when I thought I had updated the 3.9 image previously I didn ot because the promote job didn't run. Pretty sure this happend for the same reason it didn't run against an etherpad change recently: I only modified the job definitions and not the dockerfile
19:20:51 <clarkb> this will also update our 3.8 image so we should plan a restart for that. Maybe combine the 3.8 update and the mariadb upgrade after merging both of those changes?
19:21:11 <clarkb> Once that is done I can resume testing particularly of the downgrade using images that are likely to end up in production rather than old buggy images
19:21:36 <clarkb> One of the questions from last week is whether or not hte change topic limit limitation of 5k changes per topic by default would be a problem for us
19:21:57 <tonyb> Yeah I agree WRT combining the timing for MariaDB and the new 3.8 container restart
19:22:07 <clarkb> through testing I am confident that the limit only applies to open changes and not the total of open and closed changes. This means we're fine for the new-release topic which is known to have more than 5k total but never near that many open
19:22:33 <clarkb> I also tested an upgrade from 3.8 to 3.9 with the limit being exceeded and that did not seem to impact the upgrade process at all
19:23:02 <tonyb> which is all goodness
19:23:06 <clarkb> The only thing the limit seems to affect is when at or over the limit and you try to add a chagne to the topic you get an error. But even that is pretty forgiving. If you restore an abandoned change with a topic that would be over the limit the restore works just fine
19:24:01 <clarkb> yup I'm fairly confident now that we can do the upgrade and not worry about the topic limit. Then if we ever find a reason to have more than 5k open changes in a topic we can revisit and adjust the config. But for now we're good as is
19:24:33 <clarkb> The other thing I've tested and tried to improve upstream is building the Gerrit docs without webfonts
19:24:36 <tonyb> Thanks Clark
19:24:56 <clarkb> Testing shows some slight rendering differences but they are all minor and I think I'd rather have small rendering differences than load a bunch of fonts from google
19:25:09 <tonyb> on gerrit tangent did our without-webfonts patch get any feedback?
19:25:14 <clarkb> By default the gerrit release war does not know how to remove the webfont though so I wrote a change that I pushed upstream to do that
19:25:23 <clarkb> #link https://gerrit-review.googlesource.com/c/gerrit/+/424404 Upstream change to have release war build target without webfonts
19:25:29 <clarkb> tonyb: not yet the last time I checked
19:25:30 <corvus> i wonder if we should start considering discouraging topics for things like that and encourage tags instead
19:26:31 <corvus> since topics do have a use in gerrit for submitwholetopic.  i don't have any expectation we would ever turn that on for openstack, however, it's starting to feel like the wrong tool for the job now that it has that additional functionality in gerrit
19:26:48 <corvus> ("things like that" == release process)
19:27:01 <clarkb> corvus: ya I think that gerrit diverged on how topics are used and never really communicated that explicitly
19:27:09 <clarkb> it just happened naturally due to the same topic merge functionality I think
19:27:13 <corvus> exactly
19:27:45 <corvus> anyway, no need to discuss further or decide now; just throwing out a brainstorm :)
19:27:46 <tonyb> corvus: We could probably do that.  We'd just need to ensure that the tags stuff is as easy to use as the topics stuff ATM
19:28:03 <tonyb> I've never really looked a tags
19:28:05 <clarkb> tonyb: ya thats an acl update. But also I think we can address that separately to the upgrade
19:28:13 <tonyb> Yup
19:28:20 <corvus> tonyb: yeah, and i think that means really doing that acl update.  which would be a good side effect of the process. because tags rock.
19:28:22 <clarkb> basically we just need to allow registered users or whatever to globally set hashtags then let projects limit that if they wish
19:29:01 <clarkb> That was the end of what I learned about the upgrade process. The good news is everything I was concerned about so far has been a non issue
19:29:13 <clarkb> I'll resume testing once we have new images and make sure the revert process is viable for us
19:30:11 <clarkb> #topic Wiki ssl cert renewal
19:30:23 <clarkb> This is on my todo list to do friday ish. Just a few more days then the notifications should go away
19:30:41 <clarkb> as before noting it in the meeting to make sure it isn't overlooked and everyone knows that it will be dealt with before it expires
19:30:49 <clarkb> #topic Open Discussion
19:31:12 <clarkb> I maybe should've considered this an announcement but I've been asked to let people know the CFP for the openinfra summit in korea is open until may29
19:31:21 <tonyb> Oh and git-review does hashtags already
19:31:47 <clarkb> #link https://openinfrafoundation.formstack.com/forms/openinfra_asia_summit_2024 OpenInfra Summit CFP is open
19:32:03 <tonyb> Thanks.
19:32:32 <clarkb> The event happens at the beginning of September in a city outside of Seoul (I think within metro service distance)
19:33:14 <tonyb> I've been asked by dpawlik to see if there are objecstions to enbing the elasticsearch reporter in the opendev zuul?
19:33:41 <tonyb> I'm guessing it's report to the existing opensearch site
19:33:59 * tonyb would really like to go to Korea for the summit
19:34:01 <clarkb> it would probably be good to describe what the use case is. My initial reponse having run a large elasticsearch cluster for years is that I want nothing to do with that
19:34:34 <tonyb> LOL
19:35:12 <clarkb> I'm also not sure we should have reporters to sinks that we don't run (smtp is a weird one we report to the local smtp server iirc but then normal smtp happens to deliver elsewhere)
19:35:23 <clarkb> but understanding the use case would be helpful before saying anythign definitive
19:35:26 <tonyb> I think it is about consistency with other elasticserach site used by some team members
19:36:01 <tonyb> so the same data is available for opendev as is for rdoproject
19:36:21 <tonyb> but yes I'll get more details
19:36:34 <clarkb> I think the answer to reporting to a third party elasticsearch is no. Reporting to an elasticsearhc/opensearch we run is a maybe with the side note that in theory the data reported is far less of a deluge than the log data was and it might be easier to maintain
19:37:28 <tonyb> Okay.
19:37:41 <tonyb> and opensearch.logs.openstack.org would be considered 3rd party?
19:37:54 <clarkb> yes, opendev doesn't run it
19:38:05 <tonyb> Cool.  Just checking.
19:38:51 <clarkb> and to justify running something like that ourslves we'd need to understand what it enabels that existing zuul apis do not
19:39:02 <clarkb> which goes back to use cases
19:39:25 <tonyb> Makes sense
19:40:49 <tonyb> I think that's all I had
19:41:49 <clarkb> for me as well
19:41:56 <clarkb> I'll give it another minute before calling it a meeting
19:42:46 * tonyb twiddles thumbs
19:43:37 <clarkb> and time!
19:43:39 <clarkb> #endmeeting