| opendevreview | Jeremy Stanley proposed openstack/project-team-guide master: Link to instructions for managing Gerrit groups https://review.opendev.org/c/openstack/project-team-guide/+/962347 | 13:11 |
|---|---|---|
| fungi | just a heads up, this ^ came up in trying to help educate the skyline maintainers | 13:12 |
| opendevreview | Merged openstack/project-team-guide master: Link to instructions for managing Gerrit groups https://review.opendev.org/c/openstack/project-team-guide/+/962347 | 13:30 |
| clarkb | https://issues.redhat.com/browse/RHEL-93891 is now https://issues.redhat.com/browse/CS-3015 and the response seems to basically be that yes extended boot loader partition type is probably wrong, but regardless centos 10 stream will do whatever rhel does (which is also wrong identifying the root partition as a data partition instead) so any changes will be made to match rhel and | 14:54 |
| clarkb | if we want something different we need to get rhel to change | 14:54 |
| clarkb | since we are not rhel users/customers I don't think that is a path we can really entertain. This is starting to feel very similar to the broken ping issue in stream 8. The problem is right there and we can articulate why the current state is wrong but there isn't any desire to fix it unless rhel does so first | 14:55 |
| clarkb | and since we don't have any input/sway/pull with rhel we're basically without a voice when trying to use centos | 14:55 |
| clarkb | mnasiadka TheJulia spotz[m] ^ | 14:56 |
| mnasiadka | Yes, I’ve seen that comment, I’m starting to think CentOS Stream is exactly the opposite of what was marketed - it seems there’s a lot of decisions happening in RHEL (and even packages that hit RHEL first, and then are imported to CentOS Stream) - but that’s only my observation. | 15:10 |
| mnasiadka | I don’t care enough to pursue that discussion any more ;-) | 15:10 |
| TheJulia | clarkb: I cannot argue with your logic. That being said, given it is a valid concern, perhaps there should be a discussion in the community as it relates to CentOS usage. Specifically if a community is willing to say "we don't want to deal with this because of these reasons", then it starts to articulate business and resulting engagement scenarios. | 15:32 |
| clarkb | I guess from my perspective I'm not sure I understand why its on the projcet to articulate business engagement scenarios | 15:59 |
| clarkb | we've been pretty candid about when and how centos has made thigns difficult. If that feedback turns into "but rhel" then I feel like we've done more than what should be expected of us | 16:00 |
| clarkb | fundamentally this goes back to what does openstack want to test, what does openstack think it is capable of testing with the time and resources it has, and how do we map that into "real world" platforms and scenarios | 16:02 |
| clarkb | as mentioned before we can muddle through if one of the answers to those questions is centos, but it is worth noting the struggles we've had as that mapping is created/updated/maintained | 16:02 |
| TheJulia | no, I mean, its up to the businesses to do so in response to a project making a decision. | 16:05 |
| TheJulia | Decisions/actions can then drive decisions of what to do, which can include changing patterns of behavior, for better or worse. | 16:06 |
| TheJulia | So if the community decides as a whole, centos just doesn't align, then that itself is a data point. Even the discussion is a solid data point which can be consumed and understood by a business as well. Its not up to the project to force that discussion, the project should do what is right for the project. Does that make sense? | 16:07 |
| clarkb | yes I think that is aligned with what I'm saying too. We may have feedback, but it isn't up to us to somehow force the upstream to engage with that feedback. We'll just do the best we can to use the platform if it makes sense for us and do our best to communicate the issues that arise | 16:08 |
| TheJulia | exactly | 16:09 |
| fungi | put another way, we already have solutions to openstack's needs. if centos wants to be another solution then let's talk about what that would entail, but there's no incentive for openstack to spend a lot of time worrying about what centos does any more | 16:10 |
| TheJulia | Its a two way street as well.. and saying they are only going to do what rhel does means it is just another variation derivative | 16:10 |
| TheJulia | fungi: exactly | 16:10 |
| TheJulia | That being said, a resolution or something setting a broad consensus would be a powerful statement | 16:10 |
| TheJulia | and carries way more weight, which then someone say up in one part of Red Hat's org structure can then go to the other parts and go "hey, knock this off" | 16:11 |
| TheJulia | or "fix this", or "find a way to do better" | 16:11 |
| fungi | old centos served the need openstack had, red hat decided (as is their choice) to not supply that exact solution any longer, other groups showed up to make something roughly equivalent to what old centos was, people started using those alternatives, and openstack can too | 16:12 |
| TheJulia | yup | 16:12 |
| TheJulia | To be clear, I don't like the topic, I don't like the discussion, but I do agree and think it does a disservice not to have the discussion. It is about what is right for the community first and foremost. | 16:14 |
| gouthamr | agree with TheJulia entirely; i don't know if there's a need/interest to test for Red Hat that could sustain all the work we need to do to keep up with CS - that's perfectly fine, and it might come around someday when we have folks that care... are our "tested runtimes" declarations lacking this sort of clarification? | 16:24 |
| gouthamr | we call out CS is unstable and we'd prefer it only in non-voting jobs: https://governance.openstack.org/tc/reference/runtimes/2026.1 | 16:25 |
| fungi | to be clear, i think the status quo now is viable, and if centos maintainers want to understand why things are this way then i'm happy to be in that conversation | 16:25 |
| TheJulia | based upon RHEL being called a tip, I wouldn't consider it unstable, I'd consider it separate derivitive and almost, "unworthy" | 16:26 |
| TheJulia | But I am biased because the original motivation was to build trust and help have a feedback loop. If that is not a thing, then why use it? | 16:27 |
| fungi | what i *don't* want is for there to be some sort of rivalry-driven competition, or a mistaken impression that openstack testing on rocky (or alma, or openeuler, or oracle linux, or whatever) is an insult to the work the centos community are doing | 16:28 |
| fungi | it's all open source, and we're all in it together | 16:29 |
| TheJulia | That is always a risk of building perception wise, but I think if there is consensus to shift/change, then $something detaied backing context would really be needed to help set that stage. The worst thing which could happen is everything is just an IRC discussion and everyone says "nah, I'll just quietly change jobs/artifacts/testing" | 16:29 |
| TheJulia | err, detailed | 16:30 |
| fungi | it's probably part of some fundamental law of human nature... testing platform choices seem to be following a path of least resistance, so when enough projects run into problems testing on one platform that they're not seeing on another then they will have a tendency to shift preference over time without necessarily quantifying the reasons or stating it as an explicit decision | 16:31 |
| fungi | point | 16:31 |
| gouthamr | we could add some language in our testing guideline to suggest that we're testing with a subset of platforms that our users rely on, and where we have some expertise.. we aren't against adding any distribution to this list, we'd like help to do so | 16:32 |
| TheJulia | absolutely, but the quietly changing without expressing why creates a context vacuum where people can start jumping to the wrong conclusoins. | 16:32 |
| clarkb | and really it would be great for openstack to articulate what it wants to test and what it is able to test as that feeds into ^ | 16:37 |
| gouthamr | sounds reasonable - this came up in other contexts too (most recently, database versions) | 16:39 |
| * gouthamr proceeds to add it to PTG, and will need inputs from QA/gmaan_pto (when he's back), and the TaCT SIG | 16:40 | |
| TheJulia | To phrase another way, to have a discussion and to reach a consensus in some shape or form helps set/reset expectations. | 16:40 |
| fungi | in the past we had identified rhel and ubuntu lts as the most typical platforms for openstack deployments (initially assumed, later backed up by user surveys), but challenges testing on actual rhel caused us to need to identify a proxy distro | 16:41 |
| fungi | and so our testing regimen was ostensibly focused on testing the next openstack release on the platforms we expected users to be most likely to want to install it on | 16:41 |
| * gouthamr taking some of this to https://etherpad.opendev.org/p/oct2025-ptg-os-tc | 16:42 | |
| fungi | one thing that is probably warranted in revisiting that design is to review more recent user survey responses and see if the platforms we're choosing still reasonably represent what users deploy on | 16:43 |
| gouthamr | speaking of user survey, i need to check about the data dump from the last two ! | 16:43 |
| fungi | yeah, to be clear i absolutely don't know the answer, and am as curious as anyone | 16:44 |
| clarkb | from an ideals perspective I think something like "OpenStack needs to test with modern python versions on platforms with at least several years of support" reflects the current reality if we want to try and boil it down into one generic statement | 16:44 |
| clarkb | and today that basically means Ubuntu or a RHEL clone (as all other distros quickly go out of support) | 16:45 |
| clarkb | then from there you can build on wants (think database systems, virtualization systems/tooling, storage, etc as well as maybe forward looking testing to catch problems early and so on) | 16:46 |
| gouthamr | yeah.. that sounds like a good approach, and ty for sharing context on why the prior testing guidelines looked/sounded like this.. | 16:48 |
| fungi | another aspect of this that comes up from time to time is that the way we deploy openstack software to test it is increasingly farther removed from how users would actually install it in production (in great part because in the early days we didn't know how users would end up installing it) | 16:51 |
| clarkb | I should note that if we're (re)defining things nothing says we have to stick with what we've been doing either. But I suspect that if we look at the bare minimum that is reasoanble I think we end up with something similar in terms of platform with newish python and also support measured in years not months | 16:51 |
| fungi | devstack came about in an era where we had no deployment projects, our only install documentation was for from-source installs on commodity platforms, and commercial distributions all relied on manual work by downstream package maintainers in those distros | 16:52 |
| fungi | those aspects aren't necessarily all true today, more than a decade down the line, so worth revisiting too | 16:53 |
| gouthamr | true, but, wouldn't throw the baby out with the bathwater.. we could evolve devstack given that developers truly understand it, and are able to still reproduce 'most anything with devstack that we'd hit with a production install, sometimes with creativity.. we have several other deployment projects in the community, it'd be tough for project teams to gain expertise to maintain testing with these... | 16:56 |
| gouthamr | otoh, if deployment projects could run test jobs against services, it'd be nice.. | 16:56 |
| clarkb | yup another goal behind devstack is to be explanatory | 16:56 |
| clarkb | which is useful for people new to the system to pick up and understand whati sgoing on. | 16:57 |
| gouthamr | ++, its a bit of heavy-lifting at first, but easily understood after | 16:57 |
| fungi | it was originally meant for developers to use for local testing in their development environments, but worked well enough that it could be used in automated testing and that brought a synergy between ci and what devs could reasonably reproduce locally | 17:00 |
| JayF | I read all the backlog and just wanna note: *not* using Rocky/Alma in favor of CentOS to try to avoid being perceptions of being political/playing favorites *is* a form of being political and playing favorites. | 17:01 |
| JayF | We should use the best purpose-fit tool for the job, and CentOS stream doesn't seem like it fits the use case of "proxy distro for RHEL" anymore (and IMO it hasn't for a while; but this is as clear as it's been) :/ | 17:02 |
| fungi | i think what's being requested of openstack by the centos community at this point is to articulate why that is | 17:03 |
| fungi | and i doubt it's because they don't know the answers, but rather because they feel like the people in charge of all the money want to hear it from someone else | 17:04 |
| TheJulia | fungi: the people who control budgets will need to plan/account and need $something more that they can point to regardless of which side of the discussion that is on :) | 17:05 |
| fungi | yep | 17:06 |
| TheJulia | But yeah, $something consensus that can be agreed upon which is in the form of challenges or a problem statement is critical. | 17:06 |
| * gouthamr thinks this discussion is going in a different but useful direction, but isn't the intent of what spotz[m] was mentioning at all :D | 17:06 | |
| clarkb | I guess from where I'm sitting if what you want is a stable distro with many years of support it is trivially obseravable that centos stream is not that | 17:07 |
| clarkb | so its weird to me that everyone needs/wants to formalize this | 17:07 |
| fungi | my point is that i strongly suspect the centos folks know the reasons, not just from us but probably several years of redditsphere and whatever else bemoaning the changes made to centos, but they don't feel sufficiently enfranchised to be able to convince their management of the business impact | 17:07 |
| clarkb | by definition centos is not that. Its a forward looking distro release for the thing that is stable with many years of support but by definition is not that itself aiui | 17:07 |
| fungi | *exccept* in cases where a change was made to rhel and centos is now stuck with that choice too | 17:09 |
| clarkb | ya and I think that some of that is rhel is setting the forward looking goal posts | 17:09 |
| JayF | Honestly it is hard to imagine the original CentOS stream changes not having this as a natural endpoint. Trying to test a stable-obsessed tool against a forward-looking distro is exceedingly difficult. | 17:21 |
| fungi | yes, but trying to explain that to business managers who decided to that if they stopped providing a free competitor to their paid product is hard, even when they have clear evidence that many users who depended on the old thing teamed up to make a replacement for it instead of using the new thing | 17:27 |
| fungi | removing a product from the market doesn't necessarily force consumers to use your alternative, it's just as likely to create a vacuum that others will fill to satisfy your users' needs | 17:28 |
| JayF | I think most folks in this chat have had a situation where we've had to deliver messages to deaf ears :) | 17:35 |
| fungi | absolutely, though i don't know that using the openstack community to attempt redelivery of the message will necessarily work any better | 17:36 |
| fungi | worth a try i guess | 17:36 |
| TheJulia | Perhaps ears which may not hear as we do, but can still read a direct and to the point problem statement ;) | 17:36 |
| JayF | fungi: I'm going to be 100% honest, it's worked for me in the past. | 17:36 |
| JayF | fungi: so I'm totally willing to help review whatever message we wanna send | 17:37 |
| fungi | same! | 17:37 |
| frickler | a second review on https://review.opendev.org/c/openstack/openstack-manuals/+/961925/4 in preparation of the release would be nice | 18:07 |
| gouthamr | on it frickler | 18:08 |
| gouthamr | frickler: has the process been that we'd merge this and then follow up later to add links to projects that are late to merge the branch creation changes? | 18:26 |
| gouthamr | i think so, because foundation folks are starting to do some release marketing | 18:33 |
| gouthamr | fungi: wdyt? we'll merge the docs for 2025.2 with all projects that have merged the bot generated branching changes.. a few are missing: https://review.opendev.org/c/openstack/openstack-manuals/+/961925/4/www/project-data/2025.2.yaml | 18:34 |
| fungi | "It still keeps 2026.2 Flamingo in development phase." (from the commit message) | 18:39 |
| fungi | so i think that's fine if there are still gaps | 18:39 |
| gouthamr | yes, the final change will merge on Oct 1st | 18:39 |
| gouthamr | but, don't know if you want the URLs for marketing | 18:39 |
| fungi | exactly, we do that as one of the coorinated release steps | 18:40 |
| fungi | marketing is going to use a page similar to https://www.openstack.org/software/openstack-epoxy anyway, which the foundation's web content folks prepare ahead of time | 18:41 |
| fungi | usually they get it set up and soft published the day prior | 18:42 |
| gouthamr | ah, i see | 18:42 |
| fungi | but also the press release embargo is set for after the final docs change will merge | 18:43 |
| gouthamr | ack | 18:50 |
| opendevreview | Merged openstack/openstack-manuals master: [www] Setup 2026.1 Gazpacho and add project data to Flamingo https://review.opendev.org/c/openstack/openstack-manuals/+/961925 | 18:53 |
| fungi | usually we start the release process around 10:00 utc and wrap up around 14:00 utc with the press releases going out at 15:00 utc | 18:54 |
| gouthamr | that's quite early in the morning for Texas :D | 18:58 |
| gouthamr | tc-members: I | 18:59 |
| gouthamr | (gah) sorry, hit enter too soon | 18:59 |
| spotz[m] | fungi: clarkb JayF - I would just like to get folks together to make sure we (CentOS hat on) are doing what we can to be useful for others. This conversation has already been helpful for finding a ticket that was in an unintentional black hole and getting it worked on. Maybe even the answer here is to use the output of the new Proposed Updated SIG vs Stream itself. But without folks talking and working together we won't know the answer. | 18:59 |
| spotz[m] | And with normal hat on you know I like everyone to work together | 18:59 |
| gouthamr | tc-members: I'm finding these slots from our first poll: | 19:04 |
| gouthamr | - 1600 UTC - 1700 UTC (Wednesday) - frickler and gtema can't make it | 19:04 |
| gouthamr | - 1700 UTC - 1800 UTC (Thursday) - tonyb and frickler can't make it | 19:04 |
| gouthamr | - 1500 UTC - 1600 UTC (Monday) - tonyb can't make it | 19:04 |
| gouthamr | - 1400 UTC - 1500 UTC (Wednesday) - tonyb can't make it | 19:04 |
| gouthamr | - 1700 UTC - 1800 UTC (Tuesday) - tonyb and bauzas can't make it | 19:04 |
| gouthamr | With this, i don't feel comfortable changing our existing time slot; in a month, it'll get better for bauzas as he stated earlier.. so i propose keeping the next meeting for 1700 UTC on Tuesday. | 19:04 |
| gouthamr | but, i'd like us to consider 1600 UTC - 1700 UTC (Wednesday) as an alternate time slot unless there are other suggestions. We can discuss it further here, or at the meeting on Tuesday (sorry tonyb) | 19:04 |
| spotz[m] | Currently I have 2 other meetings in that Wednesday time slot but those aren't on UTC so I don't think will shift like we will. I think it might work as we'll be falling back so an hour later? OR am I timezone challenged again? | 19:08 |
| gouthamr | haha, not just you :) :/ :( | 19:09 |
| JayF | spotz[m]: I'm happy to help where I can, but TBH I'm not really invested in RHEL/CentOS ecosystem beyond whatever we need to test OpenStack. So I'm happy to help folks from TaCT SIG (or Ironic) articulate the issues, but it's not something I can spend much time on. | 19:11 |
| spotz[m] | We should also figure out if we want to keep the meeting during Summit or not, I know I'll miss the one the week or and after. But if it's just me we should have quorum and be GTG | 19:11 |
| fungi | also fair warning, the ptg is happening in the fright zone between usa/eu/au dst changes | 19:12 |
| spotz[m] | Geez | 19:12 |
| fungi | people will need to pay extra close attention to ptg time slots | 19:12 |
| spotz[m] | Why do I think eyes are on me:) | 19:12 |
| fungi | i recommend keeping a clock set to utc (all mine are anyway), though maybe that sounds silly to some | 19:12 |
| spotz[m] | I have a column in my calendar for UTC which is what I just checked to see conflicts | 19:14 |
| clarkb | spotz[m]: I think my other piece of feedback based on the recent issue as well as the old ping problem is that preventing CentOS from meeting the needs of its users in deference to the needs of RHEL makes it hard to justify using as a user | 20:21 |
| clarkb | spotz[m]: it basically feels like I need to be a RHEL subscriber to convince anyone to fix thing in CentOS which feels backward to me. Instead, if we look at the ping example CentOS could've reverted quickly and gone back to the drawing board with RHEL to find a path forward that doesn't break users after unbreaking them via a revert. In the case of the image partition issue | 20:23 |
| clarkb | CentOS could take this feedback to RHEl directly and say "hey our images have inappropriate partition type labels lets fix that" | 20:23 |
| clarkb | but in both cases it is/was my understanding that things had to be fixed in RHEL first and to do that you need to c onvince RHEL to change and I don't have any ability to do that | 20:23 |
| spotz[m] | @clarkb Working on getting that fixed | 20:23 |
| clarkb | the forward looking distro should be able to say "oops this was bad" fix it, then make a plan to avoid the badness by the time it gets to RHEL | 20:24 |
| clarkb | that imo is the entire point of having a forward looking distro. Find where people will trip on stuff and smooth it over | 20:24 |
| clarkb | but my experience is taht what happens is we find the tripping hazards and they don't get smoothed over unless RHEL does so first | 20:25 |
| spotz[m] | Again working on it, but if I/we don't know about it we can't | 20:39 |
| clarkb | I understand that, but I also think it is unfair to say this is an unknown. It happens every time I have been involved in trying to fix somethign with stream | 20:40 |
| clarkb | and its defeating to put time and effort into things like https://issues.redhat.com/browse/CS-3015 to accurately and clearly convey and issue to be met with "fix rhel" | 20:41 |
| spotz[m] | Working on it | 20:42 |
| clarkb | I feel like the ball is out of my court now. I've shown why something si wrong. And if that isn't good enough for centos to consider a change then I'm wasting my time. I don't even expcet a change ot necessarily happen I just want it to be an option but it isnt | 20:42 |
| clarkb | answers like "this is too risky" or "you've misintepretted the partition labeling scheme" or "this altnernative is more correct because X Y Z" are all acceptable outcomes too | 20:43 |
| clarkb | but deferring to a third party that is unchangeable as a matter of course is not imo | 20:43 |
| clarkb | or even "yes this should be changed, we don't want to be out of sync with rhel so we will take this to rhel and see what we can do" would be great | 20:45 |
| clarkb | unrelated: but I just saw https://openssf.org/blog/2025/09/23/open-infrastructure-is-not-free-a-joint-statement-on-sustainable-stewardship/ | 21:21 |
| JayF | I agree with some of that, but it's only a short step away from "big oss projects need to pay rent, too" which has been how those kinda things have gone historically IME | 21:24 |
| clarkb | I think the popularity of AI toolchains being written in python hit pypi pretty hard. At one time you had a number of pcakges pushing daily updates with builds for different versions of cuda and python totalling upwards of 1GB per day per package. I think pypi has a 500MB quota per package now | 21:27 |
| JayF | I have an npm-based example: claude-code is on npm, and literally is pushing a new release on average about 2 every 3 days | 21:28 |
| clarkb | nope its 10GB total and 100MB for individual files | 21:28 |
| clarkb | a lot bigger than I thought, but I'm pretty sure those rules came about due to the flood of package publications that were hitting them | 21:28 |
| JayF | not a huge file, of course, but their autoupdate is hitting npm every single time | 21:28 |
| JayF | (and, as that article nods to, is taking up mirror space on NPM -- and gentoo -- mirrors) | 21:29 |
| clarkb | https://pypi.org/stats/ this has a neat breakdown of package by total size | 21:32 |
| clarkb | tensorflow alone is like more than 1TB | 21:32 |
| clarkb | which is about 1/30th all of pypi | 21:34 |
| clarkb | that is impressive | 21:34 |
| fungi | impressively galling | 21:44 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!