| * tonyb plays typical gameshow countdown music | 18:58 | |
| fungi | i hope it's the music from the gameshow "countdown" | 18:58 |
|---|---|---|
| clarkb | I'll take OpenStack for $400 | 18:58 |
| fungi | sounds expensive | 18:59 |
| tonyb | clarkb: What will the name of the next OpenStack release be? | 18:59 |
| fungi | too long? | 19:00 |
| clarkb | what is caracal | 19:00 |
| fungi | no, wait, unpronouncible! | 19:00 |
| clarkb | #startmeeting infra | 19:00 |
| opendevmeet | Meeting started Tue Oct 31 19:00:32 2023 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. | 19:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 19:00 |
| opendevmeet | The meeting name has been set to 'infra' | 19:00 |
| clarkb | #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/IILSVTDAEDTRTCRSZZ3P2UKY4CIOKUEY/ Our Agenda | 19:00 |
| clarkb | #topic Announcements | 19:00 |
| clarkb | I announced the gerrit 3.8 upgrade for November 17 and fungi announced a mm3 upgrade for Thursday | 19:01 |
| fungi | short notice, i mainly just didn't want ui changes and logouts to surprise anyone | 19:01 |
| clarkb | Also worth noting that November 23 is a big US holiday which may mean people have varying numbers of days off around then | 19:02 |
| clarkb | other than that I didn't have any announcements | 19:02 |
| clarkb | #topic Mailman 3 | 19:02 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/899300 Upgrade to latest mm3 suite of tools | 19:02 |
| clarkb | this is the change that corresponds to the mailman3 upgrade fungi announced | 19:03 |
| fungi | as of moments ago the remaining cleanup changes merged | 19:03 |
| clarkb | there were two other cleanup changes but ya they have both merged now | 19:03 |
| fungi | so, yes, and yes, 899300 is the upgrade for thursday | 19:03 |
| clarkb | outside of those changes we still need to snapshot and cleanup the old server at some point | 19:04 |
| clarkb | and I've left the question of whether or not we add MX records on the agenda | 19:04 |
| clarkb | I'm somewhat inclined to leave things alone in DNS since this seems to be working and is simpler to manage | 19:04 |
| tonyb | yeah they shouldn't be needed. | 19:04 |
| fungi | down the road we can revisit things like spf and dkim signing if they become necessary, but i'd rather avoid them as long as we can get away with | 19:05 |
| tonyb | ++ | 19:05 |
| clarkb | fungi: both google and yahoo have been making statements about requiring that stuff early next year... | 19:06 |
| clarkb | but ya lets worry about it when we get more concrete details | 19:06 |
| clarkb | meanwhile the vast majority of spam I receive comes from gmail addresses | 19:06 |
| clarkb | Anything else mailing list related? | 19:06 |
| fungi | nope | 19:07 |
| clarkb | #topic Server Upgrades | 19:07 |
| clarkb | tonyb started looking at jammy mirror testing | 19:07 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/899710 | 19:07 |
| tonyb | it's currently failing due to a change in behaviour in curl | 19:07 |
| clarkb | this appears to be failing but I think for a test framwork reason not necessarily because the jammy deployment is failing. I don't fully understand the failure though | 19:07 |
| clarkb | oh is it curl that changed? fun | 19:07 |
| tonyb | yeah | 19:07 |
| tonyb | a command line that works on focal fails on newer curls | 19:08 |
| tonyb | I'm looking at what the right fix is. | 19:08 |
| clarkb | tonyb: I think what the goal is there is to set SNI stuff properly so that we get the correct responses? | 19:08 |
| tonyb | I'll propose a testing fix first and then add the jammy server | 19:09 |
| clarkb | otherwise using localhost we don't align with the apache vhost matching | 19:09 |
| fungi | i suppose an alternative could be to fiddle with /etc/hosts on the node, but that feels... dirty | 19:09 |
| tonyb | yeah. that makes sense. we could probably do something in python itself | 19:10 |
| tonyb | it should be a quick fix once I get back to my laptop | 19:10 |
| clarkb | looking forward to it | 19:10 |
| clarkb | I am not aware of any other updates but wanted to mention we are also in a good period of time to look at meetpad server replacements since the PTG just ended | 19:11 |
| tonyb | at some point we should decide if we need testing for 3 Ubuntu releases | 19:11 |
| tonyb | meetpad is next | 19:11 |
| clarkb | tonyb: in general I think we're trying to align with all the things we're deploying. As we replace servers and reduce the total list of mirrors we can reduce the ubuntu flavors | 19:11 |
| clarkb | tonyb: the main concern there is apache version differences (which haven't been a problem in more recent years) and openafs functionality | 19:12 |
| tonyb | okay that's pretty much what I thought | 19:12 |
| fungi | yeah, basically we want to test that the changes we make to stuff continues to work on the platforms we're currently running on, and then once we're not running on those platforms we can stop testing them | 19:12 |
| tonyb | ++ | 19:12 |
| fungi | on a service-by-service basis | 19:13 |
| clarkb | once upon a time we tried to be more generally compatible with people doing similar to us outside of our env but realized it was too much effort and should focus on our set of things | 19:13 |
| fungi | so basically, if we upgrade the meetpad servers to jammy, we can then switch to only testing that meetpad deployments work on jammy | 19:13 |
| clarkb | #topic Python Container Updates | 19:14 |
| clarkb | #link https://review.opendev.org/q/(+topic:bookworm-python3.11+OR+hashtag:bookworm+)status:open | 19:14 |
| clarkb | this is very close to the finish line (as much as there is one) | 19:14 |
| clarkb | python 3.9 is gone and the current TODOs are to update zuul-operator and OSC to python3.11 | 19:14 |
| clarkb | OSC should merge its change soon I expect as openstack is voting on python3.11 jobs now which makes switching the image to python3.11 safe | 19:15 |
| clarkb | on the zuul-operator side of things the CI jobs there are all unhappy and I'm not quite sure the scope of the necessary fixes yet | 19:15 |
| clarkb | I was hoping zuul-operator users would get it sorted soon enough but I may need to help out | 19:15 |
| clarkb | once that is done we can drop python3.10 image builds | 19:15 |
| tonyb | yay | 19:15 |
| clarkb | I've also got a change up to add python 3.12 images but that is failing because uwsgi doesn't support python3.12 yet. | 19:16 |
| clarkb | I think we can wait for them to make a release that works (there is upstream effort to support newer python but not yet in a release) | 19:16 |
| tonyb | a quick tangent, I think it'd be good to remove old images/tags from the public registry | 19:16 |
| tonyb | leaving buster based 3.7 images feels dangerous? | 19:17 |
| clarkb | maybe? openshift recently broke zuul's openshift functional testing because they deleted old images | 19:17 |
| fungi | sounds like a refcounting challenge | 19:17 |
| tonyb | I could generate a list of things we could tag as deprecated and pull later | 19:17 |
| clarkb | there is definitely a tradeoff. I think if someone is using an image for testing its fine, but you're right you wouldn't want it in production | 19:17 |
| clarkb | maybe retag as foo-deprecated | 19:18 |
| tonyb | fair enough. | 19:18 |
| fungi | foo-dangerous | 19:18 |
| clarkb | then people have an out but it also makes it more apparent if something should not be used | 19:18 |
| tonyb | yeah that's sort of what I was thinking | 19:18 |
| clarkb | I think that would be my preference over proper deletion | 19:18 |
| fungi | foo-if-it-breaks-you-get-to-keep-the-pieces | 19:18 |
| tonyb | that's all I had | 19:19 |
| clarkb | #topic Gitea 1.21 | 19:19 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/897679 | 19:19 |
| clarkb | still no proper release and no changelog | 19:19 |
| clarkb | I have tried to keep up with their updates though and it generally works for us other than the ssh key size check thing that I disabled in that change | 19:20 |
| clarkb | I've left this on the agenda under the assumption we'll have to make decissions soon but upstream hasn't made that the case yet | 19:20 |
| clarkb | #topic Gerrit 3.8 Upgrade | 19:21 |
| clarkb | This is one with a bit more detail | 19:21 |
| clarkb | #link https://etherpad.opendev.org/p/gerrit-upgrade-3.8 | 19:21 |
| clarkb | #link https://lists.opendev.org/archives/list/service-announce@lists.opendev.org/thread/XT26HFG2FOZL3UHZVLXCCANDZ3TJZM7Q/ | 19:21 |
| clarkb | I have announced the plan to upgrade Gerrit to 3.8 on November 17 at 15:30-16:30 UTC | 19:21 |
| clarkb | I've tested the downgrade path on a held CI node and then re upgraded it for the experience of it | 19:21 |
| clarkb | Yesterday we merged a config update necessary for 3.8 that we'll want to have in place under 3.7 to ensure it is working there as well. My plan is to restart Gerrit later today | 19:22 |
| clarkb | this config update shouldn't result in any behavioral differences. It is entirely about maintaining compatibility of acceptable config in gerrit 3.8 | 19:22 |
| clarkb | fungi: if I want to do that restart around say 22:00 UTC is that a bad time for you? | 19:23 |
| clarkb | fungi: maybe better to ask if ther eis a good time for you later today? | 19:23 |
| fungi | sure, i can help at 22:00 utc | 19:24 |
| clarkb | cool I think that time should work for me | 19:24 |
| fungi | great | 19:24 |
| clarkb | the last thing I've noticed is a traceback starting up the plugin manager plugin. Upstream thought they had already fixed it which made me concerned this waws a problem with our builds but on closer inspection it seems to be a different problem (tracebacks differ) | 19:25 |
| clarkb | also we hit it on 3.7 too so shouldn't impact 3.8 | 19:25 |
| fungi | so basically not a regression | 19:25 |
| clarkb | more of a thing to be aware of as an expected startup tracebacks that looks scary but is believed to be fine | 19:26 |
| fungi | just some continued broken for a feature we're not using | 19:26 |
| clarkb | yup | 19:26 |
| clarkb | and that was all I had. The etherpad has pointers to the held node if anyone wants to take a look at it | 19:26 |
| clarkb | #topic Etherpad 1.9.4 | 19:27 |
| fungi | in progress | 19:27 |
| clarkb | in the time it took us to be ready to upgrade to 1.9.3 they released 1.9.4. Fun fact: 1.9.4 fixes the mysql isn't utf8mb4 encoded bug I filed with them years ago | 19:27 |
| fungi | i need to finish diffing the upstream container configs | 19:27 |
| clarkb | we worked around that by manually setting the encoding on the db but before that etherpad hard crashed because it couldn't log in this instance | 19:27 |
| tonyb | was that the poo emoji crash from like Vancouver? | 19:28 |
| fungi | snowman | 19:28 |
| fungi | but yes | 19:28 |
| tonyb | okay | 19:28 |
| clarkb | tonyb: no, this was on the db level not the table level. They had fixed the table level thing prior | 19:28 |
| clarkb | its all related though. | 19:29 |
| clarkb | In this case they wanted to log "warning this is probably a problem" but their loggign was broken so the whole thing crashed | 19:29 |
| clarkb | rather than bad bytes causing the crash later | 19:29 |
| tonyb | lol | 19:29 |
| fungi | ah, right, that problem | 19:29 |
| clarkb | fixing the db level encoding meant it never tried to log and things proceeded :) | 19:30 |
| fungi | also related, update to log4js which invalidates some of the config we're carrying, preventing the service from starting, which is why i need to more deeply diff the configs | 19:30 |
| clarkb | fungi: I guess once we have an updated change we'll hold a node and do another round of testing | 19:30 |
| fungi | correct | 19:30 |
| tonyb | sounds good. | 19:30 |
| clarkb | #topic Open Discussion | 19:31 |
| clarkb | that was it for the emailed agenda | 19:31 |
| clarkb | worth noting we just updated nodepool to exclude openstacksdk 2.0.0 as it isn't compatible with rax cinder v2 apis | 19:31 |
| clarkb | a fix is in progress in openstacksdk which frickler and I mentioned we could help test | 19:31 |
| clarkb | this effectively took rax offline in nodepool for a few days. It also causes nodepool to not mark nodes as node failures when a cloud is failing like that | 19:32 |
| clarkb | I kinda want to make nodepool fail the request in that cloud when the cloud is throwing errors rather than try forever | 19:32 |
| tonyb | that seems like it'd be move visible | 19:33 |
| fungi | the openstack vmt would like a private room on the opendev matrix homeserver to use instead of its current restricted irc channel, since some members are joining from the oftc matrix bridge which doesn't handle nickserv identification very well. i doubt there will be any objections, but... objections? otherwise i'll work on adding it | 19:33 |
| frickler | it failed very early, kind of a similar scenario to the expired cert issue, which also could use better handling | 19:33 |
| clarkb | frickler: ya I think they have the same underlying failure method for request handling internally with nodepool which is to basically move on and then the request is never completed | 19:33 |
| clarkb | fungi: no objections from me. Worth noting private and encrypted are distinct in matrix so you'll have to decide on those two things separately iirc | 19:34 |
| fungi | yeah, it'll be both in this case | 19:34 |
| clarkb | private is basically invite only and then encrypted is whether or not everyone is doing e2e amongst themselves | 19:35 |
| frickler | afaict even then some things are not encrypted like emojis | 19:35 |
| frickler | but not worse than IRC likely, so no objection either | 19:35 |
| tonyb | thanks fungi | 19:36 |
| fungi | the vmt uses its private communication channel only for coordinating things which can't be mentioned in public (and even then it's just things like "i triaged this private bug, please take a look: <url>" so emojis rarely come into it ;) | 19:36 |
| clarkb | re Holidays the 10th is also a holiday here and I'm taking advantage for a long weekend. I won't be around on the 10th and 13th | 19:37 |
| fungi | i'll try to be around | 19:38 |
| frickler | one question regarding the branch deletion I did for kayobe earlier: the github mirror should sync this on the next merged change, right? | 19:38 |
| fungi | correct | 19:38 |
| fungi | that job only gets triggered by changes merging, so addition/deletion of branches or pushing tags doesn't replicate immediately | 19:38 |
| frickler | ok, so we'll wait for that to happen and then can check again | 19:39 |
| fungi | it could probably be added to additional pipelines, if that becomes a bigger problem | 19:39 |
| frickler | I don't think it is urgent in this case, just wanted to cross that check off my list | 19:40 |
| clarkb | last call for anything else. Otherwise we can all have a few minutes back for $meal or sleep | 19:41 |
| clarkb | thank you for your time and help everyone! We'll be back here same time and place next week. | 19:43 |
| clarkb | #endmeeting | 19:43 |
| opendevmeet | Meeting ended Tue Oct 31 19:43:17 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 19:43 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/infra/2023/infra.2023-10-31-19.00.html | 19:43 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/infra/2023/infra.2023-10-31-19.00.txt | 19:43 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/infra/2023/infra.2023-10-31-19.00.log.html | 19:43 |
| fungi | thanks clarkb! | 19:43 |
| tonyb | thanks all | 19:43 |
| frickler | o/ | 19:43 |
| -opendevstatus- NOTICE: Gerrit on review.opendev.org will be restarted to pick up a configuration change required as part of Gerrit 3.8 upgrade preparations. | 22:00 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!