| JonathanWright[m] | Me? Alabama | 11:11 |
|---|---|---|
| fungi | oh neat, i have family around mobile al | 13:22 |
| JayF | I know some folks in that area too. | 15:28 |
| JonathanWright[m] | I'm always down for a meetup with fellow nerds for lunch or something | 15:50 |
| JonathanWright[m] | I'm around the Birmingham area. Mobile is about 4 hours south of me. | 15:50 |
| JayF | That's mainly the source of my question; but I'm probably a good 24 hour drive from you I'd suspect lol | 15:51 |
| JayF | nearly the maximum distance from Alabama possible here in Tacoma, WA :D | 15:51 |
| JonathanWright[m] | Who says I can only drive? ;) | 15:51 |
| JonathanWright[m] | I'll be in Seattle in November | 15:51 |
| JonathanWright[m] | * in November for Seagl | 15:51 |
| fungi | i'll be at all things open in raleigh in about a month, happy to get together with anybody else planning to be there | 15:53 |
| JonathanWright[m] | I'll be there! | 15:55 |
| fungi | oh right on. i'll be there all three days (including the pre-conference sessions on sunday), have a talk to give on tuesday afternoon and then immediately after that i hop a plane to paris for the openinfra summit | 15:56 |
| fungi | ato is fairly easy for me since it's only 3-4 hours by car, though i sadly end up skipping a lot of years due to schedule conflicts | 15:58 |
| JonathanWright[m] | Where do you live? | 15:58 |
| fungi | off the coast of nc, on that chain of barrier islands | 15:58 |
| JonathanWright[m] | Ah neat, was going to offer to swing by and pick you up but that's not on my way :p | 15:59 |
| fungi | it's... not on anyone's way. part of why i chose to live here ;) | 15:59 |
| JonathanWright[m] | Sounds amazing honestly | 15:59 |
| fungi | most of the time yes, so long as it's not underwater | 16:00 |
| clarkb | "not on anyone's way" except for the occasional hurricane. Here's hoping for a calm season | 16:00 |
| JonathanWright[m] | How's internet availability there? | 16:00 |
| fungi | ++ | 16:00 |
| fungi | JonathanWright[m]: not-too-terrible broadband | 16:01 |
| JonathanWright[m] | nice | 16:01 |
| fungi | cable company that supplies it does ipv6 prefix delegation properly too, which has been nice | 16:01 |
| fungi | and supports dhcp6 requests for multiple prefix assignments | 16:02 |
| JonathanWright[m] | That's very nice. I finally got symmetrical fiber a few years ago but....no ipv6. | 16:03 |
| fungi | there's a new provider burying fiber through the neighborhoods here right now, and that'll probably be the first thing i check when i sign up for trial service | 16:04 |
| * gouthamr a brief public service announcement interruption | 16:05 | |
| gouthamr | tc-members: a gentle reminder that our weekly IRC meeting will occur here in ~55 minutes | 16:05 |
| gouthamr | Agenda is here: https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee | 16:05 |
| * gouthamr continue :) | 16:05 | |
| gtema | Gouthamr: I would not be able to participate today | 16:36 |
| gouthamr | gtema: ack, ty for letting me know | 16:48 |
| bauzas | gouthamr: as said earlier, I won't be able to attend today's meeting, alas. | 16:52 |
| gouthamr | bauzas: ack, I’ll poll for a new meeting time once we have the poll wrapped up | 16:53 |
| cardoe | early o/ | 17:00 |
| gouthamr | doesn't count :D | 17:00 |
| * gouthamr kidding | 17:00 | |
| gouthamr | #startmeeting tc | 17:00 |
| opendevmeet | Meeting started Tue Sep 9 17:00:46 2025 UTC and is due to finish in 60 minutes. The chair is gouthamr. Information about MeetBot at http://wiki.debian.org/MeetBot. | 17:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 17:00 |
| opendevmeet | The meeting name has been set to 'tc' | 17:00 |
| gouthamr | Welcome to the weekly meeting of the OpenStack Technical Committee. A reminder that this meeting is held under the OpenInfra Code of Conduct available at https://openinfra.dev/legal/code-of-conduct. | 17:01 |
| gouthamr | Today's meeting agenda can be found at https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee | 17:01 |
| gouthamr | #topic Roll Call | 17:01 |
| mnasiadka | o/ | 17:01 |
| seunghunlee | o/ | 17:01 |
| frickler | \o | 17:01 |
| spotz[m] | o/ | 17:01 |
| cardoe | o/ | 17:01 |
| noonedeadpunk | o/ | 17:01 |
| spotz[m] | jonathanspw meeting! | 17:02 |
| JonathanWright[m] | o/ | 17:02 |
| spotz[m] | :) | 17:02 |
| gouthamr | noted absence: g t e m a, b a u z a s | 17:03 |
| gouthamr | courtesy ping: gmaan, jbernard | 17:03 |
| frickler | gmaan is on pto afaict | 17:04 |
| spotz[m] | Yeah he is | 17:05 |
| gouthamr | ah true, i was wondering if he meant to join | 17:05 |
| gouthamr | lets get started | 17:05 |
| gouthamr | #topic Last week's AIs | 17:05 |
| gouthamr | update on leaderless projects (Vitrage, OpenStack Charms, Venus, Monasca) | 17:06 |
| gouthamr | #link https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/thread/WR277VO5VNJP2Z5WPZ7RPWRSW2Y7F7CT/ (charms) | 17:06 |
| gouthamr | #link https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/thread/22GL7TB54WY4T2JDEVKFQCEKQY4AAVAN/ (vitrage) | 17:06 |
| gouthamr | #link https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/thread/G4FSLQDGJBRY6G4JQ3SSIPN2UVX7BC4K/ (venus) | 17:06 |
| gouthamr | ^ i sent these out under 24 hours ago, so we'll give folk a week to respond | 17:07 |
| gouthamr | #link https://etherpad.opendev.org/p/2026.1-leaderless | 17:07 |
| gouthamr | the next AI we took was around - reminding people to opt-in to CIVS emails | 17:08 |
| gouthamr | ianychoi did this a few days ago, you can see the chat in #openstack-election for details/brainstorming that went on.. | 17:08 |
| gouthamr | hopefully this translates to more people exercising their vote | 17:09 |
| gouthamr | a related AI was around the reduced electorate - which we suspected to be due to OIF membership renewal requirements | 17:11 |
| ianychoi | Note: the voters actually cast so far are: 66 out of 71 for TC and 9 out of 21 for Horizon election | 17:11 |
| gouthamr | i think this is worth surfacing to the foundation-board email list post the elections | 17:11 |
| ianychoi | s/71/171 | 17:11 |
| gouthamr | woah, i celebrated for a second! thanks for the update ianychoi | 17:11 |
| spotz[m] | I was going to say great turn out | 17:11 |
| gouthamr | the number of people that voted has exceeded 2025.1, which is good! https://civs1.civs.us/cgi-bin/results.pl?id=E_079cc0f0dda5010c | 17:12 |
| noonedeadpunk | 71 for TC vote is not a thing to celebrate... | 17:12 |
| fungi | still a week to go, i haven't cast my ballot yet ;) | 17:13 |
| ianychoi | Note: the voters actually cast so far are: 66 out of 171 for TC and 9 out of 21 for Horizon election | 17:13 |
| spotz[m] | Weel 66 out of 71 is, just not 66 out of 171 | 17:13 |
| ianychoi | noonedeadpunk: it was my typo - 171, not 71 | 17:13 |
| noonedeadpunk | yeah, I got that :) | 17:13 |
| gouthamr | sorry, i'd like to be positive around this.. i've seen this slowly/steadily improve over the past few cycles | 17:13 |
| fungi | one thing i don't think anyone has analyzed yet is what percentage of contributors were openinfr foundation individual members for the last election vs what percentage of contributors are oif individual members for this election | 17:15 |
| gouthamr | yes, i'd like to pursue that and related analyses after the polls have closed | 17:16 |
| fungi | a drop in electorate size can be attributed to fewer contributors people reestablishing their membership, or can be attributed to a drop in contributor count | 17:16 |
| ianychoi | I agree with gouthamr. And I believe OIF renew membership impacted on this election - from data: There are 171 items using "yq 'to_entries | map(select(.value.member)) | length' _all_owners.yaml", and 486 items using "yq 'to_entries | map(select(.value.member == null)) | length' _all_owners.yaml" | 17:16 |
| spotz[m] | Very valid point, I would also go back to the last election around this time to be more accurate vs the other cycle of the year | 17:17 |
| fungi | my guess is it's a mix of the two, hard to know without closer review which is predominately responsible | 17:17 |
| gouthamr | and loop folks from the foundation as we discussed earlier to boost renewals, and actually educating folks what it means to be a foundation individual member vs a community member, etc | 17:17 |
| ianychoi | Also, not sure at moment to dig into now, but might it worth if we go for election although there is just 1 candidate? One Horizon electorate reached me via e-mail and I was personally surprised that she/he did not recognize the existence of election, since I believe there were no elections for Horizon for recent several cycles | 17:18 |
| fungi | ianychoi: keep in mind that in recent years less than 50% of contributors signed up to be individual members anyway | 17:18 |
| ianychoi | (Actually, her/his electorate was invalid due to late renew to OIF) | 17:19 |
| gouthamr | ianychoi: ah! these two concerns were from the same contributor | 17:20 |
| gouthamr | apologies for not having responded there so far, but i wanted to consult with folks that have been here longer than i have | 17:20 |
| gouthamr | the problem they pointed out was that the electoral rolls were compiled at a time when they were a "community" member, but, they converted to a "foundation individual" member soon after | 17:21 |
| gouthamr | they were suggesting that our deadlines weren't clear as to when someone has to be a foundation individual member to qualify for a ballot | 17:21 |
| ianychoi | Thank you for detailed explanation on the situation gouthamr | 17:21 |
| fungi | i guess the suggestion is to run 40+ polls for a single ptl in each project regardless of whether they have multiple candidates? | 17:21 |
| fungi | in order to make it clear there are the possibility for ptl elections? | 17:22 |
| spotz[m] | Back when I joined everyone had to be a Foundation member, when Community only became a thing we were very specific that voting was the one thing they couldn't do in OUI presentations | 17:22 |
| gouthamr | there's a contribution period that's well defined, but, we haven't stated if we will check for foundation individual member at the same time | 17:22 |
| fungi | the check for foundation membership happens at the time when the electorate is built, essentially, so after the nomination period but before the end of campaigning | 17:23 |
| gouthamr | ack, its implied in the same deadline - but they contended that folks could change their membership with a click of a button | 17:24 |
| ianychoi | I think anyone can change anytime between Foundation and Community membership | 17:24 |
| fungi | or in the case of nominations, membership for each nominee is checked at the time their candidacy is proposed | 17:24 |
| spotz[m] | But you can then fix and re-propose yourself, I think I've seen that in the past | 17:25 |
| gouthamr | (ianychoi completed the rest of the concern) | 17:25 |
| gouthamr | spotz[m]: yes, this happened quite a bit during the nomination period.. | 17:25 |
| clarkb | in my local government elections when casting the ballot I have to swear under penalty of perjury that I am still a valid elector | 17:26 |
| gouthamr | but, electors usually react late and can't retroactively fix things | 17:26 |
| clarkb | you're always going to have a race condition here. | 17:26 |
| gouthamr | yes, some of it has to be based on trust/honor.. i'd want to commend the person for pursuing this.. i think they want clarity in the deadlines/docs so they/others don't get caught up on this | 17:27 |
| gouthamr | its different for candidates - they get feedback on their nomination proposals and can make membership changes to get their nominations accepted | 17:27 |
| gouthamr | electors cannot vote in elections if they fat-fingered something during the OIF membership renewal/making routine profile updates | 17:29 |
| frickler | is there a legal reason to require OIF membership for voting? | 17:29 |
| spotz[m] | I believe so | 17:29 |
| fungi | when we first dropped oif individual membership checks from the cla signing in gerrit, i recall including reminders in election communications for how potential voters can check their member status | 17:29 |
| gouthamr | yes, our charter explicitly states it - and that language came from the erstwhile foundation bylaws i think | 17:29 |
| fungi | while i don't know for sure, i expect the reason is that it provides a legal assertion that the voter is represented by a specific, non-duplicated individual | 17:31 |
| frickler | well nobody really can verify that for OIF memberships, either? | 17:31 |
| fungi | so that there is ostensible legal recourse if someone merges changes under a bunch of different gerrit accounts and uses them to multi-vote | 17:31 |
| fungi | the foundation has (or at least had when not part of the lf) a legal responsibility for the integrity of its own board elections and ballot measures, which rested on at least some minimal legal strength for its individual member agreement | 17:33 |
| fungi | tc elections were originally described by the foundation's own bylaws, but even once separated were able to piggy-back on some of that assurance | 17:34 |
| gouthamr | +1 | 17:35 |
| fungi | if the tc wants to pursue removing the requirement for electors and/or candidates to be oif individual members, i can ask about having the foundation consult their counsel on our behalf | 17:35 |
| gouthamr | this is an area to explore if we _need to_ - not the time for that discussion atm imho; lets take this to the PTG perhaps | 17:36 |
| spotz[m] | I think it has value but then I've always been a member:) | 17:36 |
| spotz[m] | Good call on PTG | 17:37 |
| gouthamr | lets do the election retro, and start a ML thread with foundation folks, and discuss this further | 17:37 |
| ianychoi | +1 :) | 17:37 |
| gouthamr | does any one else have any other AI to highlight from the past week/s | 17:37 |
| spotz[m] | I brought someone from Alma as requested:) | 17:38 |
| gouthamr | ++ thanks, its the topic right after the mysql changes one | 17:38 |
| * gouthamr notes that he pinged jbernard regarding the TC resolution for phone-home from OpenStack | 17:39 | |
| JonathanWright[m] | spotz[m]: Hiiii | 17:39 |
| gouthamr | he plans to follow up post RC | 17:39 |
| gouthamr | #topic Default MySQL/MariaDB charset/collations changes | 17:39 |
| gouthamr | #link https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/message/VR7XUI7V7QGFNHKFOXLFJJP6P4GGKOM3/ | 17:39 |
| gouthamr | seunghunlee: thanks for raising this topic on the ML | 17:40 |
| seunghunlee | Thanks for bring this to today's topic | 17:40 |
| noonedeadpunk | ++ | 17:41 |
| noonedeadpunk | yeah, sorry, I promised to write ML but -ENOTIME | 17:41 |
| noonedeadpunk | this was a really good summary | 17:41 |
| seunghunlee | thanks | 17:42 |
| seunghunlee | So, what does TC think on this matter? how should we approach? | 17:43 |
| gouthamr | i think some more discussion happened here wrt Ironic specifically | 17:43 |
| noonedeadpunk | frankly, I'd love to have a community goal regaridg change of default charset to mb4 | 17:43 |
| noonedeadpunk | that feels like most of folks are in alignment with atm | 17:43 |
| clarkb | yes basically the assertion is that ironic cannot do mb3 with some hand wavyness about that. I'm fairly confident that the problem has to do with index sizes and ironic's indexes | 17:44 |
| clarkb | innodb limits the size of an index to either 767 or 3072 byes depending on the row format | 17:44 |
| noonedeadpunk | I'm not sure about collations though. As it seems that there're really different defaults and available set of collactions, except the generic one, depending on the engine | 17:44 |
| noonedeadpunk | like mariadb default != mysql | 17:44 |
| noonedeadpunk | and I have not checked percona | 17:44 |
| clarkb | and mb4 can easily push an index over 767 byes (255 * 3 = 765 but 255 * 4 > 767) | 17:45 |
| JayF | I know that TheJulia did a lot of work in this area around the bug, and our fixes were to fix an actual operator issue. We should be very, very careful about what actions we do -- especially what actions TC dictates -- here. | 17:45 |
| noonedeadpunk | I can recall somebody said it was already solved? | 17:45 |
| noonedeadpunk | but not really sure | 17:46 |
| clarkb | noonedeadpunk: the solution was to not allow mb4 | 17:46 |
| noonedeadpunk | ah | 17:46 |
| JayF | This is the bug Ironic fixed, that from the current perspective, makes Ironic seemingly more broken (although I'm not sure if either perspective is universal) | 17:46 |
| JayF | (explicitly using _mb3 so utf8 wouldn't be _mb4) | 17:46 |
| frickler | iiuc openstack cannot enforce this, since we do not deploy the DB and things are different for maria/psql/others anyway | 17:47 |
| noonedeadpunk | yeah | 17:47 |
| noonedeadpunk | #link https://opendev.org/openstack/ironic/src/commit/5a33e8dbe7faaeb751e4e4cddb8f9c06c67a072c/ironic/db/sqlalchemy/alembic/versions/2581ebaf0cb2_initial_migration.py#L41 | 17:47 |
| frickler | so not sure what that community should would look like | 17:47 |
| JayF | frickler: at table create time you can specify charset | 17:47 |
| clarkb | and yes care should be taken to not make things worse. But in theory switching innodb rowformats (I don't know what the implications of this are it probably impacts performance) so that indexes can be 3072 byes, then switching to utf8mb4, then switching collation is probably the path to take if wanting to address both type and collation | 17:47 |
| noonedeadpunk | so eventually, MariaDB has announced a plan to remove `UTF8` alias to MB3 soonish | 17:47 |
| frickler | *community goal | 17:48 |
| JayF | frickler: the initial ironic bug was we specified utf8, a default changed the pointer of that to utf8_mb4 and $something broke (I think the identity of the $something is still being archeologied) | 17:48 |
| noonedeadpunk | so we need to anyway ensure projects are explicit with using charset in their migrations | 17:48 |
| noonedeadpunk | which might be a good opprotunity for some project to use different charsets | 17:48 |
| noonedeadpunk | for instance, placement not defining charset at all | 17:49 |
| noonedeadpunk | #link https://opendev.org/openstack/placement/src/branch/master/placement/db/sqlalchemy/alembic/versions/b4ed3a175331_initial.py | 17:49 |
| noonedeadpunk | nova uses `utf8` which is mb3, but also is about to be removed | 17:51 |
| noonedeadpunk | #link https://opendev.org/openstack/nova/src/commit/2952c109489f9cbe082f9a9e79d4ac5b15dba290/nova/db/api/migrations/versions/d67eeaabee36_initial_version.py#L59 | 17:51 |
| noonedeadpunk | while some are using latin1 | 17:51 |
| noonedeadpunk | so I guess even if we're talking about some community goal - it should be about being explicit about mb3 vs mb4 and stop using aliases | 17:52 |
| clarkb | utf8mb3 will be removed in future releases of mysql too | 17:52 |
| clarkb | which is why I think it is important to consider changing the type as well as the collation | 17:52 |
| noonedeadpunk | right.... | 17:53 |
| noonedeadpunk | I wonder if latin1 should be used for places where mb4 is impossible? | 17:53 |
| noonedeadpunk | like Ironic usecase? | 17:53 |
| fungi | in other software, we've seen some obvious and severe bugs related to users inputting 4-byte codepoints and the database backend not being configured to cope with storing them | 17:54 |
| clarkb | I don't think ironic's case is impossible | 17:54 |
| clarkb | but I think that is probably the first thing someone needs to sort out | 17:54 |
| noonedeadpunk | but if we can't get common denominator for charsets, we probably should just leave collations alone | 17:55 |
| clarkb | know that with confidence one way or another then use that to decide whether or not both type and collation should change or just collation | 17:55 |
| noonedeadpunk | and do recommendation of some generic_ci or smth | 17:55 |
| gouthamr | noob question: if no code changes were made to Flamingo (and earlier supported releases), would deployment tooling be able to handle this like the way you've proposed in openstack-ansible? | 17:56 |
| noonedeadpunk | yes | 17:56 |
| noonedeadpunk | but | 17:56 |
| clarkb | for the charset/collation is there an accounting of what each project is already doing by default? | 17:57 |
| clarkb | that way we can understand if case sensitive or insensitive is more common etc? | 17:57 |
| noonedeadpunk | this was a specific thing about upgrade of MariaDB version default version | 17:57 |
| gouthamr | (the code changes i am understanding will help clean things up for the future, and we could use the TC goals framework to define/recommend stuff to project teams and have this in Gazpacho if possible) | 17:57 |
| noonedeadpunk | so we were able to handle this case in context of this | 17:57 |
| clarkb | its possible that you'll have a large cohort with basically the same roles for charset/collation and they can all standardize. Then deal with the odd ones separately? | 17:57 |
| fungi | clarkb: i think seunghunlee's post to openstack-discuss had some analysis along those lines | 17:57 |
| noonedeadpunk | but it also opened can of worms | 17:57 |
| noonedeadpunk | and potentially with the next release - most of existing migrations will just fail at the very begining | 17:58 |
| clarkb | fungi: the email distinguishes those taht set the value but not what they set it to | 17:58 |
| noonedeadpunk | if alias for `utf8` is removed | 17:59 |
| clarkb | I guess I'm suggesting that someone draw up a big table with those concrete details in order to determine if there is a rough standard we're already adhering to | 17:59 |
| gouthamr | JonathanWright[m]: spotz[m]: apologies, but it looks like our agenda was loaded; do you mind the Alma discussion occurring right after we wrap up here? I can still capture the discussion and follow up in further weeks | 17:59 |
| spotz[m] | It's up to him:) | 17:59 |
| noonedeadpunk | when I looked through project, most are using utf8mb3 atm | 17:59 |
| noonedeadpunk | nobody defines collations explicitly | 17:59 |
| clarkb | noonedeadpunk: the email says ` services such as Nova, Keystone, Cinder, designate provides charset when creating tables` | 18:00 |
| JonathanWright[m] | Sure | 18:00 |
| JonathanWright[m] | I don't have any other committments right now | 18:00 |
| noonedeadpunk | yes, charset is provided by most. I think that placement is close to only exception | 18:00 |
| gouthamr | ty JonathanWright[m] spotz[m] | 18:00 |
| noonedeadpunk | I can build some rought table for the next time | 18:01 |
| clarkb | noonedeadpunk: cool I think that would be helpful to understand the scope of the problem and whether or not everythign is already super disjoint or has lots of informal consistency | 18:01 |
| mnasiadka | Before we get to that Gazpacho cycle goal (or later) - should we set a recommendation to users to avoid MariaDB 10.5 and higher in the install guide? (https://docs.openstack.org/install-guide/environment-sql-database.html) | 18:01 |
| gouthamr | we'd begin with that ^ at the moment, i don't know if noonedeadpunk's solution works with kolla-*, or any other installer atm | 18:02 |
| clarkb | then separately for mb3 vs mb4 it would probably be helpful to identify which projects have conflicts wit hthe larger type (we already know ironic does; anyone else?) and then start determining if there is a path forward to address that (different rowformat, different indexes, whatever) | 18:02 |
| noonedeadpunk | mnasiadka: 11.5 :) | 18:02 |
| mnasiadka | noonedeadpunk: right, I have a bad day today ;-) | 18:03 |
| noonedeadpunk | fwiw, current LTS release for MariaDB is 11.8 | 18:03 |
| * gouthamr we're a few mins over already.. | 18:03 | |
| mnasiadka | noonedeadpunk: although devstack is testing 10.11 | 18:03 |
| mnasiadka | at least in centos jobs | 18:03 |
| noonedeadpunk | but 11.4 is supported for quite a while anyway | 18:03 |
| noonedeadpunk | but frankly - I'd won't say to avoid >11.5 | 18:04 |
| gouthamr | noonedeadpunk seunghunlee: if you would, can you please add your findings to an etherpad or a change on gerrit - you could just put it up as a proposed goal and let the discussion occur - i think it'd be helpful to brainstorm all of this | 18:04 |
| noonedeadpunk | as in OSA we're planning to make 11.l8 as default, and with the patch in mind - it works nicely | 18:04 |
| mnasiadka | We decided to stick to 11.4 in Kolla for now | 18:05 |
| seunghunlee | Sure which etherpad should I use? | 18:05 |
| gouthamr | we can make one up | 18:06 |
| gouthamr | http://etherpad.opendev.org/p/gazpacho-collations-charsets | 18:06 |
| seunghunlee | thanks | 18:06 |
| noonedeadpunk | I think they closed some compatability gap with MySQL 8 in 11.8? | 18:07 |
| gouthamr | there's nothing there, but, we can use this as reference to drive further brainstorm | 18:07 |
| gouthamr | i'll wrap up the meeting now.. any thing else to add to the minutes today? | 18:07 |
| noonedeadpunk | not sure though | 18:07 |
| gouthamr | we'll discuss Alma Linux here in a few | 18:08 |
| gouthamr | thank you all for attending | 18:08 |
| gouthamr | any other discussion topics will be (re)-visited next week! | 18:08 |
| gouthamr | #endmeeting | 18:08 |
| opendevmeet | Meeting ended Tue Sep 9 18:08:51 2025 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 18:08 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/tc/2025/tc.2025-09-09-17.00.html | 18:08 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/tc/2025/tc.2025-09-09-17.00.txt | 18:08 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/tc/2025/tc.2025-09-09-17.00.log.html | 18:08 |
| noonedeadpunk | but indeed, might be there's no reason to push for 11.8 looking at the support timeline | 18:09 |
| noonedeadpunk | https://mariadb.org/about/#maintenance-policy | 18:09 |
| noonedeadpunk | like - 11.4 is supported longer, then 11.8 | 18:09 |
| mnasiadka | noonedeadpunk: that's what we decided, even 10.11 has 2 more years of support | 18:10 |
| mnasiadka | even more than two | 18:10 |
| noonedeadpunk | you still won't be able to jump to 12.3 from 11.4 though | 18:10 |
| gouthamr | JonathanWright[m]: hello there, thank you for joining us here.. you may have already gotten a fair amount of context here, but, we develop OpenStack fairly distro-agnostic, and we test how things work on a number of Linux distributions via CI jobs.. we've had Ubuntu, CentOS and Rocky Linux in the "runtime" that the TC defines for OpenStack services/deployment projects etc.. | 18:11 |
| gouthamr | the latest runtime was published here: https://governance.openstack.org/tc/reference/runtimes/2026.1 | 18:11 |
| noonedeadpunk | 10.11 is exactly same as 11.8? | 18:11 |
| JonathanWright[m] | I have 0 context :) | 18:11 |
| noonedeadpunk | like couple of month difference | 18:11 |
| fungi | context is overrated | 18:11 |
| gouthamr | haha | 18:12 |
| noonedeadpunk | +1 | 18:12 |
| JonathanWright[m] | Yep, I showed up planning on winging it | 18:12 |
| gouthamr | that's admirable then! | 18:12 |
| fungi | that's every day for me | 18:12 |
| mnasiadka | noonedeadpunk: why? you can jump from 10.11 to 11.8 - in theory they are not limiting versions | 18:12 |
| noonedeadpunk | but they do support only upgrades between closest LTS? | 18:12 |
| mnasiadka | noonedeadpunk: where is it written? ;-) https://mariadb.com/docs/server/server-management/install-and-upgrade-mariadb/upgrading/upgrading-between-major-mariadb-versions | 18:13 |
| gouthamr | JonathanWright[m]: when posting this runtime, sean-k-mooney asked why we can't have AlmaLinux as one of these.. | 18:13 |
| gouthamr | mnasiadka wondered separately why we'd need to, since all these RPM based distros could look/feel/react similarly.. | 18:13 |
| JonathanWright[m] | gouthamr: It's kinda weird seeing openstack so debian/ubuntu centric. Could've sworn a few years back it was pretty RHEL/CentOS centric? When I was using it say ~2018ish I could've sworn that was the case. | 18:13 |
| JonathanWright[m] | It'd be great to see AlmaLinux in more places around OpenStack and if anything is needed on our side to enable that we're happy to help. | 18:14 |
| fungi | a bunch of the early openstack folks came out of the ubuntu community and we basically started with most of our testing on ubuntu. red hat didn't really get heavily involved until a few years later | 18:14 |
| noonedeadpunk | mnasiadka: I can recall that from some talk with monty one day.... | 18:14 |
| gouthamr | JonathanWright[m]: stability of things mostly; we certainly don't want to project any perception that Ubuntu is the only "supported" distro.. but, centos/fedora/etc broke on the CI frequently | 18:15 |
| frickler | centos' move to stream and the resulting instabilities certainly contributed to the decline in testing | 18:15 |
| fungi | rh-based platforms have mostly played second fiddle in testing we added them | 18:15 |
| JonathanWright[m] | gouthamr: So there's a question of why add Alma when it's similar? | 18:15 |
| noonedeadpunk | they neither don't test it, nor support for enterprise, nor aim it to work | 18:15 |
| fungi | er, since we added them | 18:15 |
| JayF | JonathanWright[m]: I remember reading something on LWN (I wasn't able to find the citation) about some collaboration around coming up with an "enterprise linux" term which would imply compat with Alma/Rocky/Oracle/etc | 18:15 |
| JayF | JonathanWright[m]: has there been any progress on this front? Where OpenStack could say the work with "Distros following standard ZYX" | 18:16 |
| gouthamr | sean-k-mooney's reasoning was interesting; AlmaLinux 10 continues to support older CPUs (x86-64-v2), which CentOS Stream 10 will not | 18:16 |
| JonathanWright[m] | JayF: This doesn't sound familiar to me, unless you're talking about the OpenELA stuff (which we're not involved in) | 18:16 |
| JayF | JonathanWright[m]: that sounds familiar, and noted | 18:16 |
| spotz[m] | Sounds like that yeah | 18:16 |
| * noonedeadpunk having quite some general issues with EL10 as a whole | 18:16 | |
| gouthamr | which, would really help with our aging CI providers to some extent while alleviating OpenStack service maintainers from having to worry about getting test coverage across distros | 18:16 |
| fungi | from my viewpoint, alma is most similar for us to rocky (and maybe openeuler though we dropped it as a platform recently due to lack of engagement), centos has become fairly un-rhel-like hence the search for alternatives that are more like rhel | 18:17 |
| JonathanWright[m] | gouthamr: Correct, among some other reasons as well - frame pointer enablement, btrfs in the kernel, older hardware still natively carried in the kernel (lots of RAID cards/NICs that RHEL disables for example) just to name a few. | 18:17 |
| gouthamr | ++ | 18:17 |
| fungi | and rhel is something we can't actually test upstream because rh folks have failed to ever be able to work out the licensing constraints to allow us to do it openly | 18:17 |
| spotz[m] | Hyperscale SIG output has btrfs | 18:17 |
| noonedeadpunk | so if we're looking for most close alternative to RHEL, that would be Rocky I guess? | 18:18 |
| JonathanWright[m] | Alma would be a great target. We do some extra stuff...but we don't break RHEL compatibility. Core mission is still RHEL compatible even though we're getting to that target a bit differently than in the past. | 18:18 |
| mnasiadka | Skimming the devstack repo it seems there are centos stream 9/10 jobs and rocky 9 jobs | 18:18 |
| noonedeadpunk | as sounds like Alma is quite different for better (but potentially for worse for us?) | 18:18 |
| noonedeadpunk | as from what I hear, if we ensure that things work for RHEL - they will for Alma | 18:19 |
| JonathanWright[m] | We've never had a reported breakage with RHEL compatibility from anything "extra" we've done. We're very careful about that. Another point is we do work upstream in Fedora/CentOS and can actually fix things. | 18:19 |
| JayF | noonedeadpunk: I kinda see it as the flip side: saying we support alma increases our user base. Especially since it's unlikely the listed changes would directly impact OpenStack. | 18:19 |
| mnasiadka | So maybe if somebody wants to introduce Alma - we could have both Rocky and Alma and skip testing on CentOS Stream (which I see nobody here cares about) | 18:19 |
| noonedeadpunk | thus we should be doing as close to RHEL as possible | 18:19 |
| gouthamr | JayF: +1 | 18:19 |
| noonedeadpunk | mnasiadka: except that among all them we can use only centos stream 10 now? | 18:20 |
| noonedeadpunk | as all rest just not having very basic packages? | 18:20 |
| spotz[m] | CentOS stream is as close to RHEL you're going to get. | 18:20 |
| fungi | the constant churn and regular breakages on centos stream, with no apparent urgency from them to fix those problems when they crop up, does make it a worse test platform than pre-stream centos was | 18:20 |
| noonedeadpunk | spotz[m]: is'nt that rocky with all respect? | 18:20 |
| JonathanWright[m] | I have no clue what the user spread looks like in terms of RHEL vs Rocky vs Alma when it comes to openstack users, but there'd be a big benefit to users (and openstack) in using Alma over Rocky specifically because we can fix things potentially sooner than RHEL and have a good track record in doing so. | 18:20 |
| spotz[m] | No | 18:20 |
| fungi | centos stream is as close to future rhel as you're going to get, probably | 18:20 |
| noonedeadpunk | as trons of things which are built against centos stream are not working on rhel? | 18:21 |
| fungi | modulo bugs that land in centos stream and get ironed out there before they make it into rhel | 18:21 |
| noonedeadpunk | like ceph | 18:21 |
| noonedeadpunk | or systemd | 18:21 |
| JonathanWright[m] | CentOS Stream tracks the next upcoming minor of RHEL. To call it explicitly "not RHEL" isn't really accurate. | 18:21 |
| spotz[m] | If we need to have stream folks come and meet with us again I can grab some | 18:21 |
| clarkb | its worth noting that the reason we have rocky test nodes is that rocky community members did the work to support them | 18:21 |
| JayF | fungi: yeah, which is why I think we should be motivated to replace centos stream use with alma or rocky -- we want to QA OpenStack, not CentOS/RHEL | 18:21 |
| mnasiadka | noonedeadpunk: for Kolla CentOS Stream is just early adopting EL10 (and then having a place where we see what is going to change in next minor EL10 releases) - nobody really uses that outside of the ,,dev cycle'' | 18:21 |
| fungi | i think the question for openstack is whether we want to test what rhel users will probably be running in the future maybe kinda sorta, or what rhel users are currently using | 18:21 |
| fungi | for ubuntu and debian we don't currently test on their development versions either, we test on their lts server releases | 18:22 |
| noonedeadpunk | imo, I think we should try to cover the most EL-like distros with least amount of CI time | 18:22 |
| fungi | for me, centos stream is more akin to debian/testing while rhel is more like debian/stable | 18:23 |
| JonathanWright[m] | clarkb: yeah I talked to mnasiadka about this a while back and just haven't found the dev cycles myself to do the work yet, but I also thought it would be an uphill battle getting buy-in from the openstack side. If there's interest I can definitely prioritize the work on my side. | 18:23 |
| JonathanWright[m] | Lance from OSUOSL was going to help us with it as well as he has an interest in it since OSU runs their openstack stuff all on alma | 18:24 |
| JayF | JonathanWright[m]: doing the work has like a 49% voting share in openstack in practice :) | 18:24 |
| JayF | JonathanWright[m]: especially if folks have confidence people will keep doing the work to keep things in good shape | 18:24 |
| noonedeadpunk | hehe, right | 18:24 |
| * gouthamr didn't know that OpenStack at OSUOSL runs on Alma - pretty cool! | 18:24 | |
| JonathanWright[m] | Lance is a pretty outspoken Alma supporter :) | 18:24 |
| gouthamr | ++ nice | 18:24 |
| fungi | lance is Ramereth[m] here btw | 18:24 |
| JonathanWright[m] | And he's been the driver behind things like https://github.com/AlmaLinux/ALESCo/pull/3 which directly relate to openstack | 18:25 |
| noonedeadpunk | so I really wonder if we are expecting to have/see any difference for Alma/Rocky in CI | 18:25 |
| clarkb | from the infrastructure side supporting debian and ubuntu are almost trivial compared to the RHEL family. Fewer major things change release to release and we can often simply change the release name to build and we're done | 18:25 |
| noonedeadpunk | and it makes sense to run both in projects on regular basis | 18:26 |
| clarkb | which is why I think it is important to have someone who isn't us (the opendev team) have ownership of anything in the rhel family space | 18:26 |
| clarkb | because it tends to be a lot of work and not just release to release with stream | 18:26 |
| noonedeadpunk | can't agree more on trivialism of ubuntu/debian support | 18:26 |
| JonathanWright[m] | Ramereth | 18:26 |
| JonathanWright[m] | Are you talking about in building the ifnra or building packages for the OSs? | 18:27 |
| noonedeadpunk | changes you need to make for new major release | 18:27 |
| clarkb | JonathanWright[m]: building the infrastructure within our CI system. Things like image building, network bootstrapping, etc | 18:27 |
| gouthamr | noonedeadpunk: wouldn't we let project maintainers decide through test job configuration? | 18:27 |
| fungi | we don't really do any distro packaging upstream, we leave that to package maintainers in each distro's community | 18:27 |
| clarkb | literally with debian and ubuntu most releases are chagne the name, build new image and you're probably done. | 18:27 |
| clarkb | with fedora, centos, etc this process usually involves fixing about 20 things. | 18:28 |
| noonedeadpunk | gouthamr: well... I guess depends on how much infra resources we have right now? | 18:28 |
| clarkb | the v3 requirement was a huge headache along those lines | 18:28 |
| JonathanWright[m] | Do you not just use images from upstream projects? I guess I don't understand why you're building images | 18:28 |
| noonedeadpunk | gouthamr: I am personally very concerned about amount of jobs we have now in OSA due to various scenarios and distros support | 18:28 |
| gouthamr | noonedeadpunk: with Alma, i think there'll be more test resources than Rocky or CS10 | 18:28 |
| JonathanWright[m] | Am I correct in thinking we're talking about VM/container images that the CI runs within, right? | 18:28 |
| noonedeadpunk | and I'd really love to add Alma as well to supported ones..... | 18:28 |
| clarkb | JonathanWright[m]: yes | 18:28 |
| fungi | JonathanWright[m]: there's a laundry list of reasons not to use full "cloud" images from the distros | 18:28 |
| noonedeadpunk | But well.... | 18:29 |
| noonedeadpunk | due to old CPU support, right... | 18:29 |
| clarkb | JonathanWright[m]: we do not use the images from the upstream distros. They tend to be bloated with stuff we don't need, do bad cloud-init configuration by default (why does every distro have a special username?), but importantly we're caching our git repos in the images to speed up CI test runs | 18:29 |
| fungi | when it comes to the images we boot our ci/cd test nodes on at least | 18:29 |
| JonathanWright[m] | Would us maintaining a cloud image tailored to your CI help? That's easy for us... | 18:29 |
| clarkb | JonathanWright[m]: we'd want you to help us build those images within our CI system | 18:29 |
| noonedeadpunk | I think for Rocky we're using docker image as a base? | 18:30 |
| clarkb | noonedeadpunk: correct | 18:30 |
| fungi | part of it for us is papering over the per-distro inconsistent bits our ci systems rely on | 18:30 |
| mnasiadka | DIB also has almalinux-container element | 18:30 |
| JonathanWright[m] | Rocky's docker image never gets updates...not sure if y'all have noticed that | 18:30 |
| mnasiadka | it just needs probably an update to build 10 in addition to 9 | 18:30 |
| noonedeadpunk | JonathanWright[m]: I think they manage it on quay? | 18:30 |
| clarkb | JonathanWright[m]: that isn't an issue if the platform can update itself | 18:30 |
| fungi | "as a base" means we chroot into it and then start updating and installing packages | 18:30 |
| noonedeadpunk | https://quay.io/repository/rockylinux/rockylinux?tab=tags | 18:30 |
| fungi | so just something to bootstrap the rootfs from | 18:31 |
| JonathanWright[m] | oh that one might be updated. dockerhub has been trying to get them to fix their crap for like over a year | 18:31 |
| mnasiadka | I don't think I have the patience for DockerHub anymore | 18:31 |
| mnasiadka | :-) | 18:31 |
| noonedeadpunk | and actually, quay would be even preferable for us due to dockerhub ratelimits.... | 18:31 |
| JonathanWright[m] | We have quay and dockerhub among others :) | 18:31 |
| fungi | i don't think anyone uses dockerhub any more. they locked out most of their users and publishers | 18:31 |
| noonedeadpunk | ++ | 18:31 |
| JonathanWright[m] | No disagreement at all about dockerhub and its....issues | 18:31 |
| clarkb | and fwiw typically with rocky we don't need much effort except when there is a new rleease | 18:32 |
| clarkb | this is distinct to centos stream and I suspect alma is more like rocky here than stream | 18:32 |
| JonathanWright[m] | There's no reason to expect Alma would be any different ;) | 18:32 |
| fungi | pretty sure moby/docker are just trying to orchestrate a bankruptcy filing as their exit plan | 18:32 |
| JonathanWright[m] | alma matches rhel, not stream | 18:32 |
| clarkb | so absically set things up to build the image for a release then things are usually happy until the next release | 18:32 |
| noonedeadpunk | JonathanWright[m]: btw, do you have docker images with `init` out of the box? | 18:33 |
| clarkb | but the lift release to release has sometimes still been large (like we see with v3 whcih doesn't affect alma as much) | 18:33 |
| JonathanWright[m] | noonedeadpunk: somewhere, yes | 18:33 |
| noonedeadpunk | Just thinking about things like molecule tests, which need some noop init and preferably systemd.... | 18:33 |
| clarkb | 9 -> also updated all the networking stuff to delete compatbility and force you to use network manager | 18:33 |
| clarkb | er 9 -> 10 | 18:33 |
| noonedeadpunk | `ubi-init` for rocky works nicely for this purpose.... | 18:34 |
| fungi | yeah, the x86_v3 requirement has essentially blocked us from testing centos stream 10 much | 18:34 |
| JonathanWright[m] | https://hub.docker.com/r/almalinux/9-init | 18:34 |
| JonathanWright[m] | should be on quay too | 18:34 |
| clarkb | so as examples the two big things we had to solve for stream 10 and rocky 10 were figuring out how to deal with the hardware requirements within our environment to both build images and then boot them. But also update all the networking managment stuff to support network manager | 18:34 |
| noonedeadpunk | And it's over a year now that systemd-networkd can't be built for EPEL10 | 18:34 |
| noonedeadpunk | https://bugzilla.redhat.com/show_bug.cgi?id=2303892 | 18:34 |
| JonathanWright[m] | I'm already in the comments on that issue :) I maintain netplan in EPEL which needs systemd-networkd | 18:35 |
| JonathanWright[m] | * netplan in Fedora/EPEL which | 18:35 |
| noonedeadpunk | and the one built in CentOS SIG just crashes instantly outside of CentOS | 18:35 |
| noonedeadpunk | (from hyperscale sig) | 18:36 |
| clarkb | then there is the whole question about mirroring packages. We are not mirroring packages for rocky (I'm trying to avoid mirroring stuff that is less commonly used to keep the overhead on the team's side down). With centos stream one of the struggles there is they seem to rarely delete old packages and some packages are huge | 18:37 |
| JonathanWright[m] | Because hyperscale has a different version of systemd entirely | 18:37 |
| clarkb | so the mirror grows indefinitely at a noticeable rate | 18:37 |
| noonedeadpunk | well. it doesn;t mean it should crash on RHEL... | 18:37 |
| JonathanWright[m] | Hey now we're in my arena. Mirroring and our mirror-related infra is my pet project :D | 18:37 |
| noonedeadpunk | and work on CentOS Stream... | 18:37 |
| noonedeadpunk | anyway | 18:37 |
| clarkb | JonathanWright[m]: I think to start if we add test nodes for alma we'd see how we do without dedicated mirrors similar to rocky. I just want to call out some of teh struggles we've head | 18:37 |
| JonathanWright[m] | noonedeadpunk: yes it does, because it depends on the version of systemd | 18:38 |
| JonathanWright[m] | hyperscale ships newer systemd than RHEL/stream, and systemd-networkd within hyperscale relies on it. | 18:38 |
| clarkb | the other issue with the rpm based distros (this affected opensuse too when we mirroed them) is there doesn't seem to be a "check the mirror is valid" tool | 18:38 |
| mnasiadka | clarkb: Kolla is running a lot of Rocky jobs, we haven't really had a lot of failures due to upstream mirrors unavailability - so I don't think that's a very big problem | 18:38 |
| clarkb | mnasiadka: ack that is good to know | 18:38 |
| JonathanWright[m] | Where is the testing infra location geographically, or is it all over? | 18:38 |
| noonedeadpunk | clarkb: we actually have | 18:38 |
| fungi | JonathanWright[m]: all over the globe | 18:38 |
| clarkb | we use openafs to store the data and can "release" the content when it is known to be good. We do this with the deb mirrors. We cannot do this with rpm based mirrors because the tools aren't there | 18:39 |
| fungi | though these days na and eu | 18:39 |
| noonedeadpunk | and we had to replace rocky mirrorlist with a baseurl, as mirrors are really unreliable except the official one | 18:39 |
| fungi | we've had test infrastructure in asia but don't currently | 18:39 |
| JonathanWright[m] | You wouldn't need to run your own mirrors. We have a very health mirror network, about 4x the size of rocky or centos, and we monitor mirrors and pull out bad ones, serve good, close and fast ones, etc. | 18:39 |
| noonedeadpunk | https://opendev.org/openstack/openstack-ansible/src/commit/b325716e9155e2242200b2dd4054731db7606571/zuul.d/playbooks/pre-gate-cleanup.yml#L32-L41 | 18:40 |
| JonathanWright[m] | https://i.imgur.com/3i0ntdJ.png | 18:40 |
| clarkb | JonathanWright[m]: the main issue we've seen with rpm distros is that they very often update the indexes and either the packages listed don't exist yet or the hashes mismatch | 18:41 |
| noonedeadpunk | they were constantly not catching up with syncs, especially around releases for centos/fedora | 18:41 |
| noonedeadpunk | exactly | 18:41 |
| fungi | the package mirroring we do is to provide local caches within each cloud region where we're testing so that test nodes don't connect over the wider internet when they don't have to | 18:41 |
| JonathanWright[m] | clarkb: That's not an issue with our mirror system either. Fedora/CentOS/Rocky all use Fedora's "mirrormanager" software which has some quirks. We wrote our own and addressed that issue years back. | 18:41 |
| clarkb | JonathanWright[m]: neat. You must be copying files in order to avoid those problems then? Its something I wish was more common even with the deb side of things (but at least there we can delay publication until things are coherent) | 18:42 |
| clarkb | anyway those are the big considerations from a make this happen in the CI system standpoint | 18:43 |
| JonathanWright[m] | If you serve a different mirror per repo (for example baseos, appstream, crb, etc.) then you can end up with a split-brain situation where dependent packages are missing from one mirror to the next. That's the underlying problem. | 18:43 |
| fungi | well, for our debian/ubuntu package mirrors we directly generate our own indices of the retrieved packages rather than reusing the distros' indices, which sidesteps the synchronization problem | 18:43 |
| fungi | but in so doing, we can't rely on their index signing either | 18:44 |
| JonathanWright[m] | Sounds like we have a leg up on ubuntu then ;) | 18:44 |
| * gouthamr stepped away for a bit and came back to see this great discussion.. | 18:45 | |
| clarkb | so probably the main thing to figure out is what are openstack's goals with test platforms and then ensuring the CI system is in alignment with that? | 18:45 |
| fungi | part of that is as a workaround for mismatched indices, part is because we do some selective mirroring and don't e.g. pull all supported processor architectures | 18:45 |
| clarkb | and that may include adding alma | 18:45 |
| noonedeadpunk | we actually had also couple of requyests last year for adding alma specifically | 18:45 |
| gouthamr | i was going to wait to see if we can get some tests before recommending its addition to the runtime | 18:45 |
| gouthamr | (for OpenStack i mean) | 18:45 |
| clarkb | gouthamr: I think that is fine, but having more clarity around the current goals of the testing would probably help determine if starting that at all is worthwhile | 18:46 |
| fungi | getting some dib alma-minimal images added to opendev/zuul-providers would make that possible | 18:46 |
| JonathanWright[m] | Sounds like you probalby don't even need to do any internal mirroring/caching for alma, and if you do, you can just pull it down natively without having to rebuild the indexes. I can't imagine just grabbing the packages from the internet would put a strain on CI resources? | 18:46 |
| noonedeadpunk | if we have images - I could try that on Epoxy in OSA with Alma 9 | 18:46 |
| noonedeadpunk | (10 is just generally blocked by networkd for us) | 18:47 |
| fungi | JonathanWright[m]: grabbing packages from the internet doesn't put a strain on resources (other than bandwith utilization), but at the scale of testing we perform we feel acutely every single momentary network disruption | 18:47 |
| gouthamr | JonathanWright[m]: tangentially wanted to note that #opendev (here on OFTC, #_oftc_#opendev:matrix.org) is a great place to keep in touch with opendev administrators and discuss infra stuff :) (ofcourse continue to discuss here as much as you need) | 18:47 |
| JonathanWright[m] | noonedeadpunk: systemd-networkd? Let me work on pushing that | 18:48 |
| clarkb | my personal opinion here is that we actually try to boil the ocean way too much on this stuff. I think the differences between distros are fairly minor for most openstack services once you get beyond the difference in python versions | 18:48 |
| noonedeadpunk | JonathanWright[m]: yup.... | 18:48 |
| clarkb | but that doesn't stop us from running huge sets of jobs that are all just copies of one another trying to cover eveything on all the platforms | 18:48 |
| noonedeadpunk | we rely heavily on systemd-networkd for organizing networking | 18:49 |
| fungi | clarkb: i think where it comes up most is the deployment projects, since they're more closely tied to per-distro implementation details | 18:49 |
| gouthamr | clarkb: that and we've concluded they're not the same given that with Alma10, we'll be able to use more of our existing CI providers | 18:49 |
| clarkb | fungi: thats a good point. But even then many of those projects are building like 5 different versions of the same containers that ultimately deploy openstack | 18:49 |
| clarkb | fungi: they could just build one and be done and test that works and everyone can use those container images | 18:49 |
| fungi | oh, for container-based deployment projects, yes there's less need for variant testing probably | 18:50 |
| JonathanWright[m] | Users really do care about having their OS explicitly listed as supported. That's a battle we've been fighting with a vendor picks up and lists rocky but not alma | 18:50 |
| clarkb | I guess in my mind it isn't don't test rocky and alma and stream. Its determine where you get value in actually doing so don't just copy paste | 18:50 |
| mnasiadka | clarkb: that's usually easier said than done when you decide to run libvirt/kvm in a container and other similar things ;-) | 18:51 |
| fungi | fair, there's some things openstack needs to do for which containers are at best a leaky abstraction | 18:51 |
| clarkb | mnasiadka: yes and occasionally iscsi stuff for cinder | 18:51 |
| JonathanWright[m] | Is actual infra a limiting factor as well? We might can help on that front if so. | 18:52 |
| fungi | opendev doesn't turn down additional test capacity, though basically we rely on openstack cloud providers | 18:52 |
| clarkb | JonathanWright[m]: I don't think so. We're always happy to have more if it is available (particularly arm64 as only osuosl is providing that today), but its rare for us to hit capacity consistently right now | 18:52 |
| mnasiadka | I think another aarch64 provider surely would not hurt | 18:53 |
| fungi | agreed | 18:53 |
| JonathanWright[m] | We do have some arm bare metal and could probably spare some. | 18:53 |
| JonathanWright[m] | We recently got a (lot) of power9 ppc64le as well. It's not in prod quite yet...still in my basement, but we could share that too if it'd help. | 18:53 |
| fungi | bare metal is hard for opendev to utilize, since as i said we basically rely on cloud providers and have optimized for that case | 18:53 |
| JonathanWright[m] | makes sense | 18:53 |
| fungi | but if you wanted to run an arm-based openstack cloud like osuosl does, we could use that probably | 18:54 |
| mnasiadka | We dropped ppc64le support in Kolla in like 2021 | 18:54 |
| fungi | there are people out there (mainly deep in the bowels of blue-painted buildings) who do still support ppc and system-z openstack deployments, but they don't really help upstream with testing | 18:55 |
| mnasiadka | I don't know if there's appetite for third architecture, given aarch64 is not that popular in OpenStack | 18:55 |
| JonathanWright[m] | OSU does ppc64le with openstack still haha | 18:55 |
| JayF | mnasiadka: We gotta think about that in context: for instance, Ironic uses aarch64 images for IPA even though we don't have a lot of server usage on aarch64 | 18:56 |
| fungi | oh i didn't realize they had it running | 18:56 |
| fungi | (ppc in osuosl i mean) | 18:56 |
| JonathanWright[m] | aarch64 is not that popular in anything really, at least relative to x86_64. I think on our usage charts it's somewhere like 1% while x86_64 is 99%. s390x and ppc64le combined are a fraction of a tenth of a percent. | 18:56 |
| JayF | mnasiadka: I imagine we'll have risc-v soon too, and those will be architectures that IPA will need to run on :) | 18:56 |
| JonathanWright[m] | We're close on riscv64 builds. Have been working on it for about a year and finally have some fully functional test builds. | 18:57 |
| mnasiadka | JayF: aarch64 is getting more traction now even in HPC environments, than it got in the recent years - but I'm not that convinced it's going to persist ;-) | 18:57 |
| JayF | mnasiadka: I am incredibly convinced it will; the integration opportunities between NVIDIA and ARM are too great | 18:57 |
| JayF | it might be niche but it's going to be in extended use in many use cases | 18:58 |
| JayF | even if it doesn't become the default "give me a server" answer :) | 18:58 |
| fungi | between "giving the finger to intel" and some reduction in power draw for equivalent processing loads, arm seems to have some promise | 18:59 |
| JonathanWright[m] | So I guess my question is...how can I help? Trying to figure out exactly how all the opendev CI and such works wouldn't be the best use of my time, but perhaps someone already involved heavily in it could help bootstrap the work and I could work alongside answer questions and doing enablement that's needed within alma, like perhaps the images that were mentioned, looking at mirroring-related stuff, etc.? | 19:00 |
| JonathanWright[m] | s/best use of my time/efficient for me to figure out when someone could probably do it way faster/ | 19:00 |
| mnasiadka | So - ultimately I'm with clarkb - that there should be some testing strategy defined, and surely we need contributors to make sure EL is a better OpenStack testing component than it is right now | 19:00 |
| clarkb | step 0 is probably someone proposed a change to build alma images (can probably copy paste the rocky configs for that and do replacement of the text as necessary) | 19:01 |
| clarkb | then if something breaks whoever is doing that can reach out to JonathanWright[m] for help debugging | 19:01 |
| spotz[m] | Forming a SIG or other group to work on this wouldn't be a bad idea | 19:01 |
| mnasiadka | Because currently it's probably both Kolla and OSA teams trying to make Rocky working for our users | 19:01 |
| mnasiadka | And not a lot of other things happening | 19:01 |
| mnasiadka | (I might be wrong) | 19:01 |
| fungi | we don't really have per-platform sigs in openstack now, not that we couldn't | 19:03 |
| mnasiadka | clarkb: step 0 is adding support for '10' in almalinux-container most probably, then using that in zuul-launcher - unless we want to build 9 - but then it's python doesn't fit current PTI | 19:04 |
| fungi | for an ongoing effort, adding a sig could work. for a more immediate effort with a defined end-state, a popup team would be more appropriate | 19:04 |
| spotz[m] | I was thinking more of a general operating system one | 19:04 |
| clarkb | mnasiadka: ya and possibly any glean updates to detect the platform correctly | 19:04 |
| mnasiadka | clarkb: I have sort of a deja vu ;-) | 19:04 |
| fungi | spotz[m]: oh, like an "operating systems sig" that covers the various operating systems openstack runs and tests on? | 19:05 |
| clarkb | but if someone can drive that from the openstack side then any questions of why doesn't this work like the others can be forwarded to JonathanWright[m] | 19:05 |
| JonathanWright[m] | For sure. I'm happy to be a part of things, etc. but having someone already within openstack working towards it as well makes the most sense I think. | 19:05 |
| fungi | we have a "packaging sig" but i think it's fairly defunct (at least the people listed as the chairs for it haven't been around much lately) | 19:05 |
| spotz[m] | Yeah cause from reading through this discussion it doesn't seem like there's much communication going back and forth | 19:06 |
| * Ramereth[m] waves | 19:15 | |
| Ramereth[m] | Any questions you have for me with regard to Alma+OpenStack? I haven't had any problem with compatibility FWIW. | 19:15 |
| fungi | Ramereth[m]: i was excited to hear that you also have a ppc64le openstack cloud | 19:21 |
| fungi | not that we would necessarily have a use for that, but definitely cool! | 19:21 |
| Ramereth[m] | fungi: yeah, that was technically our first openstack cluster :) We've been running that for almost 15 years I think | 19:24 |
| fungi | oh wow! | 19:25 |
| fungi | no s390x though ;) | 19:25 |
| Ramereth[m] | We have it running on POWER8, POWER9 and now POWER10 (although that one is complicated due to the removal of OPAL firmware) | 19:25 |
| Ramereth[m] | we have VMs from Marist College. Let them handle that ;) | 19:26 |
| fungi | i know some folks running it on z but they're deep inside ibm | 19:26 |
| fungi | doesn't come up much | 19:26 |
| Ramereth[m] | yeah, they basically internalized all their PowerVM codebase for openstack | 19:26 |
| Ramereth[m] | What's amusing is that IBM devs prefer our cluster over getting access on their clouds | 19:27 |
| fungi | good job! | 19:27 |
| Ramereth[m] | But I'm 1+++ on getting Alma tested in OpenStack. Let me know if there's anyway I can help with that | 19:28 |
| gouthamr | should we revive the packaging sig for this? we'll need to find a new set of volunteers interested in the space.. packaging-sig specifically maintained the "rpm-packaging" repo.. Dirk's still active there - but seems like a one-person team to me, and this effort of introducing and maintaining Alma Linux to our test infra doesn't seem to fit into the mission 100% | 20:07 |
| gouthamr | alternatively, a pop-up-team can be made with JonathanWright[m], Ramereth[m] and anyone else interested where we'd define the steps to get Alma Linux into the runtime, and after that goal is met, the team can disband and support long term maintenance efforts in a less formal way | 20:07 |
| gouthamr | noonedeadpunk: seunghunlee, i added some structure to https://etherpad.opendev.org/p/gazpacho-collations-charsets .. based on your inputs there, we can bring this topic back to discussion to the next weekly meeting | 20:08 |
| fungi | gouthamr: yeah, i was using the packaging sig more as precedent for something like a test platforms sig, i agree that the missions don't overlap much | 20:11 |
| fungi | (though the interested parties might overlap, maybe that is more important?) | 20:11 |
| gouthamr | true, we could seek more participation on the ML | 20:12 |
| fungi | though also i could see an argument for just organizing it as an effort within the testing and collaboration tools sig if that's what folks prefer | 20:12 |
| gouthamr | that’d be neat too | 20:13 |
| fungi | granted, if tact sig takes it on, i'd be side-eyeing some folks to volunteer as co-chairs | 20:13 |
| fungi | only so many hours in my day | 20:13 |
| JonathanWright[m] | I'll show up wherever the work is going to happen :) | 20:15 |
| fungi | for context (overrated, i know), tact sig is essentially the interface between the openstack community and the opendev collaboratory | 20:16 |
| fungi | so things like this at the intersection of openstack and opendev are well within tact sig remit | 20:17 |
| fungi | like getting platforms into opendev that openstack wants to use for testing, in this particular case | 20:18 |
| fungi | but also, i don't know that "where" the work happens governance-wise matters all that much. interested parties just need to communicate and get the necessary bits done, and if there is a particular need for some sort of governance then sig or whatever is a discussion to have | 20:20 |
| fungi | on the whole we're not especially territorial here | 20:21 |
| spotz[m] | Sorry had to step away but I would like to be involved in this | 20:22 |
| JonathanWright[m] | Can we get a volunteer from the openstack side that I can work with? | 20:22 |
| fungi | i'm going to intentionally not volunteer to coordinate, just because my dance card is overfull already, but am happy to pitch in where needed | 20:24 |
| spotz[m] | Ok let's get tact SIG going again. | 20:30 |
| fungi | well, tact sig is always going, just hit me up with specific governance needs | 20:48 |
| fungi | but co-chairs are most welcome | 20:48 |
| sean-k-mooney | gouthamr: the 2 main reasons alma was interesting to me was 1 yes x86-64-v2 support so it could run on the old rax xen ci nodes, the ohter was like debian alma does not compile out spice supprot form qemu | 21:18 |
| gouthamr | ah | 21:19 |
| sean-k-mooney | at least in those two instnaces alma seam to package things more inline with what is supproted in the upstream projects | 21:19 |
| sean-k-mooney | its valid for each distro to choose how they want to test and build the pacakges they ship | 21:20 |
| sean-k-mooney | but rhel at least is opionated in that regard for better or worse when it comes to libvirt but ya the fact i cant boot c10s on my home hardware unless i use my gaming pc or work laptop makes it harder to test | 21:22 |
| JonathanWright[m] | if principles matter we're a registered non-profit foundation (501c6) as well which more closely aligns with openstack as a foundation | 21:22 |
| fungi | the governing org's tax status doesn't matter all that much in this case, we also test on ubuntu which is essentially owned by a for-profit company, but being not-for-profit is cool never the less | 21:26 |
| JonathanWright[m] | fair, was thinking about debian, not ubuntu | 21:26 |
| JayF | JonathanWright[m]: I suspect the reality is more: most folks here have a specific company or product they work for, and it's tough to justify time that doesn't intersect with our usages. For instance, I'm happy to provide reviews and moral support, but I can't justify committing to Alma support of anything b/c my employer doesn't use it at all. | 21:26 |
| fungi | well, debian is represented by software in the public interest (i'm a director and officer of spi these days for unrelated reasons), and it's a usa irs 501(c)(3) charity instead of (c)(6) | 21:27 |
| sean-k-mooney | well the use fo debian upstream is pretty limited. i was more commenting on that becase when we watnted to test spice last year debian was the only os that supproted it out of the set we had in ci | 21:27 |
| JayF | (this is also why my gentoo contribution, except for bumping claude-code packages that I use daily, are on my own time) | 21:27 |
| JayF | fungi: soon you'll get Gentoo as well :) all the fun distributions will be SPI :P | 21:28 |
| fungi | (spi is also the fiscal sponsor for gentoo for that matter) | 21:28 |
| fungi | and arch linux, if we're making a list... | 21:28 |
| JayF | yeah, I missed being a foundation member by like a day to vote in the anticipated-final gentoo foundation board election | 21:28 |
| JayF | debian/arch/gentoo is basically... all non-corporate distros with significant popularity except maybe nixos? | 21:29 |
| JayF | that's impressive | 21:29 |
| fungi | impressive to me as it happened more or less by accident | 21:29 |
| fungi | i think arch and gentoo sought out spi because it hosts debian, so maybe not entirely an accident, but spi didn't set out to be that exactly | 21:30 |
| JayF | You're basically somehow trying to ignore SPI's own positive reputation being an influential factor :) | 21:31 |
| fungi | and spi has tons of other associated projects that aren't distros, like systemd and openssl | 21:31 |
| fungi | well, i'll take some of the blame for gentoo and arch since i acted as the board sponsor for their applications | 21:32 |
| JayF | Larry moos a moo of appreciation in your way lol | 21:33 |
| fungi | thanks larry | 21:35 |
| sean-k-mooney | JayF: speaking of claude-code and gemini to a much lesser degree, as someone with dyslxia i have found both help me a lot when drafting pros like the watcher cycle highlights https://review.opendev.org/c/openstack/releases/+/960325 | 22:05 |
| fungi | i'm also dyslexic, but can't shake the feeling that i'm giving up a significant amount of personal agency by leaning on those kinds of tools (feel the same about auto-correct, grammar checkers and typeahead suggestions too though) | 22:07 |
| sean-k-mooney | in the past i have relied on tools like gramerly or some opensource alternitives but its quite nice ot be able to have better assistence in your normal editor rather then always having to use a janky web application and then havign to fix all the formatting after the fact | 22:07 |
| fungi | granted, i type painstakingly slowly and re-read what i've written several times over, which is maybe not the best use of my time either | 22:08 |
| sean-k-mooney | fungi: it depend on how you use them. i generally flesh out the bones of the content first and then go "can you fix all the spelling and grammer issues" aka translate form sean speak to english please | 22:09 |
| JayF | fungi: I kinda agree with you; it becomes an audience thing for me. If I'm working on a blogpost that won't have a byline, that's trying to get a message across; sure, I'll put it through the soulless machine :D When doing blogposts or content in my own voice, I avoid AI tools except as a brainstorming tool or as a secondary tool (e.g. write a bunch of notes; ask AI to suggest a structure) | 22:09 |
| sean-k-mooney | ya tone and voice are things i then to iterate on and manuall tweak | 22:10 |
| JayF | One of the copy editors at GR-OSS who works with us gave me a really good piece of advice once: feed the end result of your AI-assisted writing through an AI detection tool and it'll help highlight the portions that are trope-y | 22:10 |
| sean-k-mooney | its funny some of the ai tools now come with a "humanize" option | 22:13 |
| sean-k-mooney | im like 1 why is that not the default | 22:13 |
| sean-k-mooney | and 2 how is it really going to be differnt. like are you going to readd typos and ignore style guidance the way humans typicaly do based on ther backgroudn and what "feel right" to them | 22:14 |
| Ramereth[m] | <gouthamr> "should we revive the packaging..." <- What exactly is/was the packaging sig? Is that something separate than say something like RDO on RHEL? | 22:14 |
| sean-k-mooney | maybe related to https://opendev.org/openstack/rpm-packaging | 22:15 |
| sean-k-mooney | three used to be a sepeate effort in the openstack/openinfra foundation governace to package openstack on rpm based distos | 22:15 |
| fungi | Ramereth[m]: rdo is a downstream activity, the packaging sig was basically an excuse for package maintainers from multiple distributions to collaborate upstream on shared challenges that could be solved within the upstream community | 22:15 |
| sean-k-mooney | ah the sig, https://opendev.org/openstack/rpm-packaging was one of the conreate outputs of tha tsig correct | 22:16 |
| sean-k-mooney | were there also deb-packaging or similar in the past | 22:17 |
| fungi | but yeah the entry for it on https://governance.openstack.org/sigs/ links to https://wiki.openstack.org/wiki/Rpm-packaging implying it was mostly an rpm-oriented effort | 22:17 |
| fungi | "started with the goal of unifying the independent packaging efforts of RDO and SUSE, but is really open for anyone who wants to contribute RPM packaging spec files" | 22:18 |
| fungi | "We intend to also maintain the packaging of the stable/ branch lifecycle of the OpenStack Mitaka release." ...that kinda dates it | 22:19 |
| sean-k-mooney | i think in the long past rdo only supported centos and later it expanded to other rpm distos | 22:19 |
| sean-k-mooney | as in in the centos 7 days | 22:19 |
| Ramereth[m] | ah okay, thanks for the clarification | 22:23 |
| sean-k-mooney | fungi: at least in the wiki there used to be a deb packaging effort as well | 22:24 |
| sean-k-mooney | https://wiki.openstack.org/wiki/DEB-packaging | 22:24 |
| fungi | yeah | 22:24 |
| fungi | the two never really got together sadly | 22:25 |
| fungi | i think there was too much disagreement on... well... just about everything | 22:25 |
| sean-k-mooney | you mean debian and rpm packageing or rdo and upstream rpm packaging | 22:26 |
| fungi | the deb-packaging and rpm-packaging groups | 22:26 |
| sean-k-mooney | the upstream packaging repos just had the spec files right. they never actully delivered/host build artifact if im not mistaken | 22:26 |
| fungi | right | 22:27 |
| fungi | it was an attempt to share work so the distros didn't basically end up developing slightly different rpm spec files completely independent of one another | 22:27 |
| sean-k-mooney | ya, im not that famialr with rdo in genral but i belive alot of there methodolgies and process were partly influcnace by fedroa's packaging policies coupled with redhats | 22:29 |
| sean-k-mooney | you woudl think the two are the same but there is quite a time lag between the two | 22:29 |
| sean-k-mooney | rdo was kind of stuck in the midel since the way the rpms would be built downstream is partly defiend in the rdo spec files | 22:30 |
| sean-k-mooney | the upstream rpm specs files int the rpm-packaging repos are more "pure" in that sense | 22:31 |
| JayF | I will state here what I say to Gentoo python project anytime it comes up: I think (community supported) distro packages for openstack repositories are a bad idea | 22:31 |
| JayF | you either need a vendor providing it, or just consume from pypi/git | 22:31 |
| JayF | I know some folks are gonna do it, and it's good to have them cooperate, I just personally don't get the value | 22:32 |
| sean-k-mooney | i dont really disagreee with that statement | 22:32 |
| sean-k-mooney | if you build generic packages then it wont fit with the policy fo any disto | 22:33 |
| JayF | community supported extends to distro-community too, fwiw | 22:33 |
| JayF | I mean unless you are paying for goods and services, use git or pypi :) | 22:34 |
| sean-k-mooney | and as soon as you pick a convention then you make it hard for other to contibute | 22:34 |
| sean-k-mooney | as an upstream comunity we do kind of have 2 binary artifacts beyond our tarballs and the wheels we push to pypi.. the kolla images, and the loki ones, both in oci format | 22:36 |
| sean-k-mooney | so we support tarballs for packagers to consume. pythons native packaging format via pypi and contianer images for those that want that | 22:38 |
| sean-k-mooney | hum its ode that the loking images are pushed to https://quay.io/organization/airshipit | 22:40 |
| sean-k-mooney | although i guess that implies airship is the pirmary user of those today | 22:40 |
| cardoe | Yeah I wish that wasn’t the name. | 22:47 |
| cardoe | That’s what OpenStack helm uses. | 22:47 |
| clarkb | re claude the terms of service limit what you can use the code for (specifically nothing illegal and no competition with claude iirc). It isn't clear to me if it is safe to use claude generated output in open source projects as a result (I think it is possible that only you the claude user are at risk there fo having your account banned, but I'm not a lawyer and that sort of | 23:04 |
| clarkb | thing is fuzzy to me) | 23:04 |
| clarkb | I don't think anything prevents the loki team from publishing elsewhere. Just a matter ofconfiguring an alternative account and updating job configs | 23:07 |
| sean-k-mooney | i suspect the enforceablity of such terms veries widly across juristictions but ya, im sure the bad guys doign illegal things will read that and comply | 23:08 |
| sean-k-mooney | clarkb: didnt zuul or the infra team have a oci image builtign thing that uses requirementa dn bindeps at some point. is that what became loci, i remember sam yaple was involved in the early days fo loci and also in kolla for a time but i dont recall if loci and the zuul built continaer shared heritage | 23:11 |
| clarkb | sean-k-mooney: yes we have that no its completely inrelated to loci | 23:12 |
| clarkb | sean-k-mooney: opendev/system-config/docker/python-base/Dockerfile and docker/python-builder/Dockerfile. Then you can see how it is used in zuul's Dockerfile | 23:12 |
| clarkb | that system is one of the reasons I feel like openstack tries too hard to boil the ocean | 23:12 |
| sean-k-mooney | ah ok. i think loci tries to do somethign similar in that it tried to use wheels for all the shared deps between contaienr and bindeps fo non python deps | 23:13 |
| clarkb | we don't really care what the base os is (it happens to be debian by default from the python images), we took the upstream python images and made them understand requirements and bindep and zuul speculative states and not you can crank out images without much thought | 23:13 |
| sean-k-mooney | cool | 23:14 |
| sean-k-mooney | in terms of not boiling the ocean ya that partly why most openstack project declared maintianing pakcaging out os scope | 23:18 |
| sean-k-mooney | i.e. https://docs.openstack.org/nova/latest/contributor/project-scope.html#deployment-and-packaging | 23:19 |
| clarkb | I guess it is worth noting that someone did experiment with alpine and musl based images and found that musl's tradeoffs for fitting into smaller spaces impact python performance in ways that are not great | 23:19 |
| clarkb | essentially glibc will sacrifice memory for performance. musl avoids doing that much more and python being slow gets hit hard | 23:19 |
| clarkb | but thats more of a musl vs glibc thing than alpine vs debian | 23:20 |
| sean-k-mooney | i have plaid with that a little (alpine images) but i never got as far as that | 23:20 |
| sean-k-mooney | if your going for size distroless is proably a better middlegorund | 23:20 |
| sean-k-mooney | i.e. https://github.com/GoogleContainerTools/distroless | 23:21 |
| sean-k-mooney | still mostly debian based but smaller then the standard debign pythohn images i belive | 23:21 |
| sean-k-mooney | i belive they stil use glibc ectra | 23:22 |
| sean-k-mooney | althoguh when you get pass the base image to gcr.io/distroless/python3-debian12 for example i dont knwo how much fo the size reduction is lost to get a practical python runtime | 23:23 |
| clarkb | ya for a lot of our use cases having a proper distro available to install git and so on is often useful | 23:25 |
| clarkb | its a balancing act. I think one that you can sway towards larger images if you aren't building an image for N different platforms | 23:25 |
| sean-k-mooney | gcr.io/distroless/python3-debian12 latest 51be3dc3e33e 27 seconds ago 55.4 MB | 23:26 |
| sean-k-mooney | not bad actully i think that about half of the python debisna image but again once you wanted to add the bindeps it coudl be problmeatic | 23:26 |
| sean-k-mooney | its basiclaly the same size as the offical python3.12-alpine image | 23:27 |
| sean-k-mooney | docker.io/library/python 3.12-alpine 2d6c23e3d401 3 months ago 51.1 MB | 23:27 |
| sean-k-mooney | but the python one is likely more debugable if you need it | 23:28 |
| sean-k-mooney | anyway night all o/ | 23:29 |
| clarkb | o/ | 23:33 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
