| clarkb | meeting time | 19:00 |
|---|---|---|
| clarkb | haven't had one of these in a while | 19:00 |
| clarkb | #startmeeting infra | 19:01 |
| opendevmeet | Meeting started Tue Jan 10 19:01:09 2023 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. | 19:01 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 19:01 |
| opendevmeet | The meeting name has been set to 'infra' | 19:01 |
| clarkb | #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/5WW6CBUKQUWVNW7C5RRLXW7PWUXDPY75/ Our Agenda | 19:01 |
| clarkb | #topic Announcements | 19:01 |
| clarkb | The Open Infra Foundation individual board member election is happening right now | 19:02 |
| clarkb | if you are a member of the foundation you should've received a ballot via email. | 19:02 |
| clarkb | Please vote if you are able to. | 19:03 |
| clarkb | #topic Bastion Host Updates | 19:03 |
| clarkb | ianw: I know prior to holidays you were querying people to see if anything else needed migrating from the old bastion before it was shutdown. | 19:03 |
| clarkb | Looks like the old bastion is still up, but maybe we can shut it down soon? I didn't see anyone indicate anything needed to be moved at last poll | 19:04 |
| ianw | yeah i think i got a response from everyone active | 19:04 |
| ianw | i can shut it down, and we can consider removal after a little | 19:04 |
| clarkb | sounds good | 19:05 |
| ianw | more time | 19:05 |
| clarkb | Anything else bastion related? | 19:05 |
| ianw | i don't think so ... the backup bits i think might need a rebase now | 19:05 |
| clarkb | #topic Mailman 3 | 19:06 |
| clarkb | I think we may still need to restart services to pick up the site owner update. ALso frickler discovered that local root alias is not working anyway. | 19:06 |
| clarkb | do we want to use a different site owner value or fix the root@lists01 alias in exim? I assume we prefer fixing the alias? | 19:06 |
| fungi | and i have some more related changes up for review | 19:07 |
| fungi | including an upgrade to the latest mm3 versions from last week | 19:07 |
| clarkb | these changes are related to fixing the vhosting of lists themselves? | 19:07 |
| fungi | yes | 19:07 |
| clarkb | https://review.opendev.org/c/opendev/system-config/+/867986/ and children? Looks like they are all workinprogress right now | 19:08 |
| clarkb | https://review.opendev.org/q/topic:mailman3+status:open is a better link. Finds a couple extra changes | 19:08 |
| clarkb | fungi: what do you think the plan there should be? | 19:09 |
| fungi | just a sec, need to page a lot of this back into my head ;) | 19:09 |
| fungi | i think i still need to review the failures on 867987 | 19:09 |
| fungi | i have a feeling it's probably just going to need test adjustments, but don't think i've confirmed that yet | 19:10 |
| fungi | the upgrade is up for discussion. the change passes at least | 19:11 |
| clarkb | fungi: was that generated to be in sync with how the upstream docker images are handling the version upgrade too? | 19:11 |
| fungi | and the upgrade notes for the new versions don't indicate any extra steps needed | 19:11 |
| clarkb | my main concern is with the dep pinning as that seems to be a major headache for everyone doing mm3 | 19:11 |
| clarkb | but once you have a good install it sounds like they handle upgrades pretty transparently | 19:11 |
| fungi | yes, much of that was ported from the docker image repo | 19:11 |
| clarkb | awesome. In that case I think we can land that either before or after fixing the other issues, Probably your preference really | 19:12 |
| fungi | i think i'd like to dig into fixing the domain separation first | 19:12 |
| fungi | after we get that ironed out though, and upgrade mailman to the latest releases, i should be ready to start planning for the next site migrations | 19:13 |
| clarkb | works for me. Maybe ping us when you're in a state where extra eyeballs could be helpful? Whether just code review or helping with debugging on held nodes | 19:13 |
| fungi | yep | 19:13 |
| fungi | migrating lists.katacontainers.io should probably be prioritized since it frees up an entire server | 19:13 |
| clarkb | makes sense | 19:14 |
| fungi | but also i've been fighting oom kills of random processes on the old lists.openstack.org server so the more we move off of it the better | 19:14 |
| fungi | (i just moments ago killed and restarted all relevant processes on that server yet again) | 19:14 |
| clarkb | Anything else on this topic? | 19:15 |
| corvus | fungi: i'm assuming you shut down/disabled the migrated processes already? | 19:15 |
| fungi | yes | 19:15 |
| corvus | figured :) | 19:15 |
| fungi | yeah, if i hadn't, that would be an easy way to relieve some pressure on it | 19:15 |
| fungi | too bad | 19:15 |
| clarkb | #topic Quo vadis Storyboard | 19:17 |
| clarkb | I swear I sent that followup email I said I would send but I'm not finding it now... | 19:17 |
| clarkb | In any case there hasn't been much movement on this topic. As mentioned before I think we need to make a proposal (probably towards turning it off in a gracefaul manner if possible) based on the feedback we've gotten | 19:19 |
| clarkb | I feel like I've got enough distractions this week that I won't get to that this week whcih means we can let it simmer for a bit longer :/ | 19:20 |
| clarkb | Still happy to receive any and all additional feedback others may have though | 19:20 |
| clarkb | #topic Gerrit 3.6 | 19:21 |
| clarkb | The upgrade happened and we haven't reverted | 19:21 |
| clarkb | ianw sent email to the repo discuss list to bring up the label listing behavior that is new in 3.6 (I have't seent an responses yet) | 19:21 |
| clarkb | I think we are at a point where we should remove 3.5 image builds, add 3.7 image builds, and update our upgrade testing to perform a 3.6 to 3.7 upgrade | 19:22 |
| ianw | ++ i don't see us going back to 3.5 | 19:22 |
| fungi | sgtm | 19:22 |
| clarkb | ianw: I'm happy to help with the wrangling of those changes if you'd like. | 19:22 |
| ianw | sure :) | 19:23 |
| ianw | i also started on | 19:23 |
| ianw | #link https://review.opendev.org/c/openstack/project-config/+/867931 | 19:23 |
| ianw | to update copyConditions in our configs | 19:24 |
| clarkb | this is one of potentially several similar types of changes we'll need to make prior to upgrading to 3.7 | 19:24 |
| ianw | we have some in our All-Projects which I think will need manual intervention, but i'd like to put some of it into the gate testing first | 19:24 |
| clarkb | ++ there is less blast radius outside of all-projects | 19:25 |
| ianw | some work started on that @ | 19:25 |
| ianw | #link https://review.opendev.org/c/opendev/system-config/+/868054 | 19:25 |
| clarkb | anything else gerrit upgrade related before we jump to the next thing? | 19:26 |
| ianw | nope, more to discuss in the future as we work on submit requirements etc :) | 19:26 |
| clarkb | #topic Nox | 19:27 |
| clarkb | This is the evolution of the Tox v4 topic we had previously | 19:27 |
| clarkb | The way the tox v4 situation has panned out has led to us needing to deal with new issues with every tox v4 release | 19:27 |
| clarkb | Eventually some of us got tired of that and started looking at alternatives: in particular nox. | 19:27 |
| clarkb | Nox is a tox alternative that uses a simple python file to specify its targets (it calls them sessions) and relies on standard tools and processes for installing things | 19:28 |
| clarkb | This to me is the main feature with nox that has me wanting to switch. Tox has reimplemented libraries for certain peps and it led to weird behavior | 19:28 |
| clarkb | Sticking to standard tools ensures we don't need todebug oddness like that | 19:29 |
| clarkb | Long story short Zuul is already largely moved to nox and I think OpenDev should do the same | 19:29 |
| clarkb | #link https://review.opendev.org/c/opendev/bindep/+/868004 An example conversion | 19:29 |
| clarkb | I've got this change up for bindep that serves as an example too | 19:29 |
| clarkb | All of the zuul-jobs tooling for artifact retrieval and siblings support should be there now too | 19:29 |
| fungi | the only counterargument against nox i've heard so far is that some folks prefer the declarative nature of tox.ini files over turing-complete python scripts for configuration, but tox.ini is pretty far down the road to being a programming language of its own at this point too | 19:30 |
| clarkb | And as a user I've been relly happy using nox. I don't think I would suggest the move if I wasn't. The syntax for command is a little different but its not so different and it is quick to pick up | 19:30 |
| clarkb | fungi: ya and I would argue that this is tox's greatest flw right now because they keep changing the semantic meaning of that declarativeconfig | 19:30 |
| clarkb | making it effectively useless | 19:31 |
| fungi | i'm good with switching bindep over to see how it goes (and zuul/nodepool are already basically there) | 19:32 |
| clarkb | Anyway, I intend on pushing changes to move active opendev repos to nox. But I don't want to get too far down that path until we've got general consensus this isn't a problem | 19:32 |
| clarkb | So please do take a look at the bindep change, fetch that change locally and run nox yourselves and see what you think. THen let me know :) | 19:33 |
| fungi | convincing large and complex projects like openstack to switch away from tox is probably not a windmill i feel like tilting at (especially since they're stuck using tox on older branches for years to come), but i think for opendev's projects it makes sense and shouldn't be too big of a lift | 19:33 |
| corvus | if it were me, i'd pin those older branches to tox v3 then nox on master | 19:34 |
| fungi | yeah, they're pinning the older branches anyway, as it turns out | 19:34 |
| fungi | but stuff like release tooling will have to support both | 19:34 |
| clarkb | depending on where I get with zuul and opedev I might take a look at the release tooling, but I agree its a much stronger preexisting force you run up against there | 19:35 |
| corvus | i'm very happy with nox and i like the idea of us using it for opendev projects | 19:35 |
| clarkb | woot my first couple of bits of feedback :) | 19:36 |
| clarkb | I think we can move on to the next thing | 19:36 |
| clarkb | #topic Service Coordinator Election | 19:36 |
| clarkb | It has been about 5 months since we last elected our service coordinator | 19:36 |
| clarkb | 6 months after our previous nomination period is January 31, 2023 - February 14, 2023 | 19:37 |
| clarkb | I'd like to propose we open that two week period for nominations for the next service coordinator and take it from there | 19:37 |
| clarkb | If there are no objections to this timeframe I'll send email to service-discuss making it official soon | 19:38 |
| clarkb | as I've said before I'm happy for someone else to do the work too :) | 19:38 |
| clarkb | #topic Linaro Cloud Move and Cleanup | 19:39 |
| clarkb | ianw: want to fill us in on the recent updates for the linaro cloud move | 19:39 |
| ianw | yep sure, we have credentials | 19:40 |
| ianw | there are changes to enable the cloud for CI @ | 19:41 |
| ianw | #link https://review.opendev.org/q/topic:linaro-2022 | 19:41 |
| ianw | one issue is that it doesn't have much disk. so much so that our 800gb volume for the builder /opt would leave nothing else | 19:42 |
| ianw | so i asked osuosl if they could give us compute quota to run a nodepool builder there, which they kindly provided | 19:42 |
| ianw | i have the host up, storage attached, etc, and some changes out there to add it to system-config & dns | 19:43 |
| ianw | so i think i'd like to add that, and get it working and building things | 19:43 |
| ianw | then disable nb03 | 19:43 |
| ianw | then cut testing over to new cloud | 19:44 |
| fungi | sounds great | 19:44 |
| ianw | then disable old cloud | 19:44 |
| clarkb | oh I just approved the first two chagnes that shift over to the new cloud | 19:44 |
| clarkb | should I WIP them? | 19:44 |
| clarkb | I guess its only the second that is a problem. I'll WIP it | 19:44 |
| ianw | it's ok, just we will want to monitor to make sure it is actually working | 19:44 |
| clarkb | heh ok switched back to +A | 19:45 |
| ianw | it can all really happen in any order, depending on our attention span to monitor the different bits :) | 19:45 |
| clarkb | thank you for picking this up. I think it will make ed happy and in theory these newer test nodes should be very quick | 19:45 |
| clarkb | Anything else related to the cloud moves? | 19:46 |
| ianw | yep, good to try and work with the providers to keep everyone happy :) | 19:46 |
| ianw | no more on that, will keep working at it | 19:47 |
| clarkb | #topic Upgrading Bionic Servers | 19:47 |
| clarkb | #link https://etherpad.opendev.org/p/opendev-bionic-server-upgrades | 19:47 |
| clarkb | This is going to take on a bit more urgency due to bionic EOL in april | 19:48 |
| clarkb | I'm personally hoping to be able to really start picking things off of that list starting next week. | 19:48 |
| clarkb | If anyone else ends up helping out please stick a note in the etherpad so that we don't step on each other. Also, help is appreciated :) | 19:48 |
| clarkb | #topic Open Discussion | 19:50 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/866781 Retire Mordred as infra root | 19:50 |
| ianw | one thing frickler pointed out was the increasing storage numbers | 19:50 |
| clarkb | Can other infra-root review this change? mordred pushed it himself so no concerns there. Unfortunately I think the time has come for this little bit of cleanup | 19:50 |
| ianw | #link https://grafana.opendev.org/d/9871b26303/afs?orgId=1&from=now-6M&to=now | 19:50 |
| frickler | yes, not critical yet, but worth watching | 19:51 |
| clarkb | looks like ubuntu and ubuntu.ports have grown a bit | 19:51 |
| clarkb | I wonder why | 19:51 |
| clarkb | since they should be pretty fixed after the jammy release last april | 19:51 |
| ianw | which, apropos the prior topic, one win we could have there is purging everything xenial related | 19:51 |
| clarkb | the open euler mirror also doubled in size | 19:52 |
| clarkb | so ya we probably needto make another pruning pass | 19:52 |
| corvus | i'd like to land the nodepool openstack statemachine change soon. there is a non-zero chance of real-world performance problems. | 19:52 |
| corvus | once we get all the nodepool tox/nox/k8s changes merged, how about i tag a 8.0.1 release, then if there is a problem we can revert opendev to it quickly? | 19:52 |
| clarkb | corvus: ++ to tagging first | 19:52 |
| ianw | clarkb: yeah, i thought we got a new openeuler release but purged the old one | 19:53 |
| clarkb | debian security almost doubled too | 19:53 |
| clarkb | definitely a good idea to dig into that. Thank you for calling it out ianw and frickler | 19:54 |
| corvus | cool, i'll make sure the tag happens and will keep folks updated. let me know if there's any other concerns, preparation, or safeguards we'd like for opendev. i figured i would manually upgrade launchers slowly to keep an eye on things. | 19:54 |
| clarkb | corvus: you'll need to put them in the emergency file to do them manually as our hourly jobs are really good at updating them otherwise | 19:55 |
| clarkb | (remember this is on bridge01.opendev.org now) | 19:55 |
| corvus | clarkb: ack. and i've already retrained to use bridge.opendev.org :) | 19:55 |
| corvus | and thanks, i forgot that could happen hourly and not nightly | 19:56 |
| corvus | so: tag, emergency file, manual upgrade. | 19:56 |
| ianw | #link https://review.opendev.org/c/opendev/system-config/+/848796/3 | 19:56 |
| ianw | ^ is the openeuler mirror drop -- i knew i'd seen something about it. we should probably do that at least | 19:56 |
| clarkb | ++ I'll review it now | 19:57 |
| corvus | (incidentally, tagging 8.0.1 will be a nice double check of any release job nox updates) | 19:58 |
| corvus | if we have to do an 8.0.2 it's no big deal | 19:58 |
| fungi | numbers are free | 19:58 |
| fungi | we won't run out | 19:58 |
| clarkb | And we are just about at time | 19:59 |
| clarkb | thank you everyone. Apologies for not communicating better last week. I ened up sick and wasn't in a place to computer. I'm still sort of on the tail end of that too but much more with it now :) | 19:59 |
| clarkb | We'll be back next week. Happy new year! | 19:59 |
| fungi | we didn't have anything worth covering last week anyway | 19:59 |
| fungi | thanks clarkb! | 20:00 |
| clarkb | #endmeeting | 20:00 |
| opendevmeet | Meeting ended Tue Jan 10 20:00:01 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 20:00 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/infra/2023/infra.2023-01-10-19.01.html | 20:00 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/infra/2023/infra.2023-01-10-19.01.txt | 20:00 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/infra/2023/infra.2023-01-10-19.01.log.html | 20:00 |
| -opendevstatus- NOTICE: One of our CI job log storage providers appears to be having trouble with log uploads and retrievals. We are in the process of removing that provider from the pool. | 22:43 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!