| *** bauzas6 is now known as bauzas | 00:43 | |
| opendevreview | Merged openstack/governance master: Add TC/PTL results from 2026.1 election https://review.opendev.org/c/openstack/governance/+/961598 | 00:46 |
|---|---|---|
| gouthamr | tonyb: was waiting on everyone at this point - it currently looks like we don't have a good overlapping time.. the best we could do is: 1600 UTC - 1700 UTC on Wednesdays or, between 1600 UTC and 1800 UTC on Thursdays | 05:36 |
| gouthamr | I don't know if folks considered the DST changes in Oct/Nov.. US/Canada will observe a change on Nov 2, EU at the end of Oct and Australia at the beginning of Oct... | 05:36 |
| gouthamr | its still god awful early for you in any case... so, i think lets float the idea of alternate weeks, or have a monthly APAC-friendly meeting slot | 05:39 |
| tonyb | That's rough. Realistically (post DST) 1800UTC is the earliest I can be up. So yeah I think alternate weeks would be good | 05:39 |
| tonyb | I don't recall and options that were late (very small UTC times) for me in the poll | 05:40 |
| frickler | so if I count correctly, we have 5 x EU, 3 x US and 1 x APAC based TC members. IMO any schedule we come up with should take these weights into consideration. tonyb: what would be your latest reasonable meeting time? like something between 06 and 12 UTC-ish I'd assume? | 05:51 |
| tonyb | For a once a week meeting can more or less do anything other than UTC 1300-1859 | 05:54 |
| tonyb | so yeah for a one hour meeting starting by UTC 1200 will mean we usually finish by UTC1300 | 05:54 |
| tonyb | If it's going to be a late one I'd appreciate avoiding UTC Tue/Wed as I already have a Tue UTC1900 meeting (opendev) and late nights+early starts don't mix | 05:56 |
| frickler | well I assume anything between 08-13 UTC will be equally bad for US folks, so we likely could choose something in the early range from that, which might then even be feasible for some stay-up-late west-coaster ;) likely we should do another poll for that range | 06:05 |
| tonyb | that works for me | 07:41 |
| *** jbernard_ is now known as jbernard | 14:05 | |
| *** diablo_rojo_phone_ is now known as diablo_rojo_phone | 14:19 | |
| *** diablo_rojo_phone is now known as Guest27275 | 14:20 | |
| *** mnasiadka_ is now known as mnasiadka | 14:26 | |
| *** johnsom_ is now known as johnsom | 14:26 | |
| *** TheJulia_ is now known as TheJulia | 14:26 | |
| *** prometheanfire is now known as Guest27314 | 14:28 | |
| *** johnsom_ is now known as johnsom | 14:39 | |
| *** mnaser_ is now known as mnaser | 14:39 | |
| *** tmazur_ is now known as tmazur | 14:39 | |
| *** cardoe_ is now known as cardoe | 15:32 | |
| opendevreview | Stephen Finucane proposed openstack/governance master: Retire shade https://review.opendev.org/c/openstack/governance/+/962197 | 15:56 |
| *** gthiemon1e is now known as gthiemonge | 16:01 | |
| spotz[m] | clarkb: fungi - I have an ex-OpenStacker current CentOS Stream person coming to Summit so we can hopefully figure out the issues you're finding and make a plan. Not just open to you both but wasn't sure who all to tag | 17:29 |
| clarkb | spotz[m]: I'm not aware of any major issues at the moment I think they were all largely worked through. We haev noticed the mirror grows perpetually as centos stream doesn't seem to clear out old packages and some packages are quite large (liek those for firefox or java) | 17:31 |
| clarkb | but tonyb has a tool to verify repo state that he thinks can be used to prune the extra packages | 17:31 |
| fungi | afaik the only significant issue we have with it is that we can't boot it on a significant percentage of our systems due to the x64v3 requirement, and yeah that their mirrors just grow boundlessly and are never pruned of old package versions | 17:31 |
| clarkb | I think the discussion before was more frustration that adding a new debuntu release is trivial but adding centos/rhel/fedora/etc requires massive work each release just to get booting | 17:32 |
| clarkb | like we literally just change the release name and build a new image for debuntu most releases. | 17:33 |
| fungi | that's "we as in opendev" though, "we as in openstack" have some additional challenges like centos stream not accurately reflecting the *current* state of rhel but rather a speculative future state where bugs get tested on the users and not reverted with any urgency | 17:33 |
| fungi | (like that one time ping was broken for... a month?) | 17:34 |
| clarkb | ya I guess the most recent issue along those ^ lines was centos stream broke their cloud images which some openstack projects (like ironic and dib) rely on for their own testing | 17:35 |
| clarkb | and there was an issue for it that calimed to fix it but it was broken for about 2.5 - 3 weeks (the assertion it was fixed was wrong) | 17:35 |
| clarkb | the issue was short writes of the images on the storage side so the file was incomplete/corrupt and the hash files were empty | 17:36 |
| fungi | more generally, openstack wants to test on lts server distributions, while centos stream is more of a test-type distribution with different stability expectations | 17:36 |
| fungi | we were trying to test on centos as an analogue for rhel, because licensing challenges prevented testing on actual rhel, but once traditional centos was abandoned we ended up testing on stream which really isn't the same thing | 17:37 |
| clarkb | ya what would probably be useful is for openstack to decide what sort of testing is useful and then if there are conflicts with stream in making that happen we can take that discussion to the summit | 17:40 |
| clarkb | opendev will struggle through adding platforms if there is a desire to add it (and if there are volunteers to add thigns) | 17:41 |
| spotz[m] | Well I thought we had gotten close to resolving the issue of the licenses, but Stream is the next point release of RHEL so yeah there could be some issues but hopefully there shouldn't be. But I think getting everyone together to talk would benefit both projects so we can figure out a path forward to reduce any issues. For me it kind of sucks walking into bashing when I'm involved in both projects and nothing has been raised to the other | 17:45 |
| spotz[m] | project. Will it get something fixed faster but if they don't know they can't do anything | 17:45 |
| spotz[m] | * fixed faster maybe not but if | 17:46 |
| fungi | i don't really see it as a bashing, just a mismatch. centos stream isn't designed to meet these particular needs, and i don't think it's intended to | 17:46 |
| clarkb | I think there is also a communication flaw in that half of the things we run into are private and we're not allowed to see issues for | 17:47 |
| clarkb | and I believe in some cases even when we file the issues from the public side those images are not visible to others on the public side | 17:47 |
| fungi | well, yes. not using secret red hat bug trackers would be a huge improvement in transparency | 17:47 |
| clarkb | I want to say this happened recently with an issue that mnasiadka filed | 17:47 |
| clarkb | so yes there aer communication issues, but I feel like the problem is not on our side. The tools we have available to us are hostile to its users | 17:48 |
| spotz[m] | We are working on that issue, some policies auto convert issues. We're trying to get the main one to stay open but open another if needed | 17:48 |
| clarkb | *those issues. Heh too much discussion about images | 17:48 |
| fungi | but i don't want to imply that if centos stream was more transparent with bugs it would suddenly fit the lts server use case | 17:48 |
| fungi | i'm happy to tell them that bringing back traditional post-rhel centos would fit what openstack needs, though i doubt that feedback is especially helpful | 17:50 |
| mnasiadka | I don’t think Stream is ever going to fit the LTS use case, and even in Kolla-land we often find centos stream jobs fail because some new packages introduced breakages, while Rocky 10 remains stable | 17:50 |
| mnasiadka | And those breakages never propagate to RHEL/Rocky next minor release | 17:51 |
| clarkb | I personally think there is value in testing against forward looking distros. I really wish the tumbleweed stuff had been useable enough. The problem with these things is that if you don't have a group dedicated to debugging issues that do arise then its just noise/problems | 17:51 |
| clarkb | and unfortunately, I don't think openstack has the bandwidth right now for that more forward looking stuff whcih is part of the issue here | 17:51 |
| mnasiadka | Kolla runs Stream jobs just to see if something in next RHEL minor release is going to break us, but it’s rather not proven to be useful | 17:52 |
| clarkb | but that is why I think openstack should probably try to define what it cares about from a testing standpoint. If oepnstack sees value there and wants to try and put resources behind it then we should do our best to make it happen. But if openstack doesn't want to then we shouldn't | 17:52 |
| fungi | and it seemed like the abandonment of traditional centos was a political/product decision that red hat didn't want a "free" equivalent of rhel that competed with their license revenue, which i respect after all it's their money to spend and their business model to manage, but as such i strongly doubt a post-rhel centos will ever come back | 17:53 |
| mnasiadka | So if OpenStack as a whole wants to invest cycles to test also on EL (in addition to Debuntu) - I think Rocky/Alma is the way to go, unless we can use RHEL (which we can’t) | 17:53 |
| mnasiadka | (And I don’t think there’s a lot of people using RHEL and not using RHOSP - but I might be mistaken) | 17:54 |
| fungi | as for the "rhel licensing being worked out" as of a couple years ago red hat said they were going to assign some contributors to work through the logistics and implementation needed to handle the license management, as well as a plan for how devs and users could locally reproduce runs without needing to jump through licensing hoops of their own, but that never actually | 17:55 |
| fungi | materialized | 17:55 |
| TheJulia | There is a definite contingent who just accepts the movement drift and rolls with it with Centos, but its all going to boil down to what folks are comfortable with in the end | 17:56 |
| clarkb | but yes I agree with fungi. I think a huge part of the problem here is a mismatch in expectations. CentOS communication was that stream should work for you if you were using centos before. Our experience is that isn't the case for openstack testing/gating | 17:56 |
| clarkb | and the reasons it doesn't work include problems with communication tools breaking down the ability to report and track issues taht are discovered, udpate flows to packages not making sense to outsiders (the ping issue for example), and generally just unexpected updates (I wonder if there is a schedule we could see for general things like networkmanager, libvirt/qemu, etc) | 17:57 |
| fungi | but also fundamental design choices, "just try harder and have fewer bugs" isn't going to solve that, and i don't want to give the impression it would | 17:59 |
| spotz[m] | Let's get everyone together and see what we can do and what we need to bring back to see what we can do on the other end | 18:00 |
| fungi | traditional centos, while problematic, was closer to what openstack needed and still needs. when that disappeared, other distros like rocky and alma showed up to fill that same space and i agree they make for a more sensible choice in this specific use case. that's not meant as a slight against centos stream | 18:00 |
| TheJulia | I feel like the discussion then goes in circles | 18:03 |
| fungi | but yeah, if there's going to be a useful discussion it should really be more about openstack's challenges rather than opendev's challenges. solving the needs of openstack is the hard part, opendev will go along with whatever openstack wants in the end | 18:03 |
| clarkb | the other concern that came up recently was that rdo is going away maybe? so we may need new packages for a few dependencies? | 18:05 |
| clarkb | I want to say it was rabbit, ovs, and smoething else. | 18:05 |
| clarkb | maybe it was the erlang runtime for rabbitmq | 18:05 |
| fungi | even if that discussion is really just explaining politely to the centos folks that openstack has a use case that stream just fundamentally isn't suited for | 18:05 |
| TheJulia | I think a huge part of a problem is there is a tendency to let perceptions be the leading driver. OpenStack as a whole is always going to be a complex matrix and overall things need to be data driven. | 18:06 |
| clarkb | TheJulia: conversely we could say that we need data on what value centos stream would provide over the alternatives (I do think there is avlue as I mentioned before with forward looking testing, but it requires dedicated effort to work with and we don't have that today) | 18:07 |
| fungi | yes, and openstack's testing is great at finding bugs in platforms, but if we test openstack on centos stream we're essentially doing upstream testing for rhel to keep bugs out of rhel at the expense of our own limited contributor and systems bandwidth | 18:07 |
| fungi | this is why we also don't test upstream on fedora, ubuntu non-lts versions, debian's testing release, et cetera | 18:08 |
| fungi | (or tumbleweed as clarkb mentioned) | 18:09 |
| TheJulia | So, going back to perceptions, the perception at hand is just testing and finding bugs. I'd argue the same exists with every distro on some level. Nothing is perfect. We pay the cost regardless, we just save it up and have to deal with it for major version changes or we pay it as we go. | 18:12 |
| clarkb | right, a big example of that is the eventlet not working with modern python problem that kicked off the whole maybe we should use eventlet less process | 18:14 |
| TheJulia | frankly, as an overloaded engineer, I prefer smaller hits as I go. Major version changes is pretty much a huge *surprise* moment where I potentially loose weeks at a time. | 18:14 |
| clarkb | this is why I think an idea testing setup probably does have sanity checks against forward looking systems, but relies on things that are expected to be stable for gating | 18:14 |
| clarkb | because our system essentialyl requires gating to pass this is the default for most (probably all) of the opesntack projects and makes it easy to not worry about the forward looking stuff if you can't anymore | 18:15 |
| clarkb | but imo that means you probably don't want to gate on centos stream. You need to test it separately. but if people only care about the gates (basically the situation we are in now) how do you incentivize the other thing? I don't know | 18:15 |
| clarkb | we've failed at it multiple times | 18:15 |
| clarkb | non ubuntu lts releases, opensuse tumbleweed, fedora, and centos stream have all run into this problem | 18:16 |
| TheJulia | ... even ubuntu releases have | 18:17 |
| TheJulia | even LTS | 18:17 |
| clarkb | well no we gate on ubuntu LTS (and previousl centos before stream) because it is stable enough for that purpose | 18:17 |
| mnasiadka | clarkb: Kolla has ditched RDO for rpm distros and replaced that with EPEL and some CentOS SIG repos (Ceph, OVS/OVN) plus use RMQ and MariaDB directly from their respective upstream repos - I don’t see a lot of problems in doing the same in devstack | 18:17 |
| clarkb | I'm saying for the forward looking stuff we've failed in basically every attempt | 18:17 |
| TheJulia | example, they incremented the dnsmasq version in response in response to a CVE. That too had fallout and means the gates can't be perfect. Besides, are we focused on perfection or the range of "good to great!" | 18:18 |
| clarkb | because people run out of steam and things break enough that if you are not consitently on top of it then you just get 100% failure over time | 18:18 |
| clarkb | TheJulia: I think its about balance | 18:18 |
| clarkb | Ubuntu LTS and CentOS pre stream broke infrequently | 18:18 |
| clarkb | at a rate low enough that people were able to address it because 1) rate is low and 2) we basically have no other choice this is the baseline | 18:19 |
| clarkb | with the more regularly updating distro releases things break far more often and we have just not had the ability to keep up and also do the other parts of our jobs | 18:19 |
| TheJulia | I think the last centos stream break was the base images the pushed to the mirrors? That impacted tons of communities | 18:19 |
| clarkb | TheJulia: yes the corrupted cloud images are the last breakage I am aware of. Prior to that was the issue mnasiadka filed that I can't even remember what that was now (but also never hda access to the issue because ti was auto privated) | 18:20 |
| TheJulia | clarkb: but then we all had to pay a huge cost whenever we went to the next version | 18:20 |
| clarkb | TheJulia: that hasn't been the case with debuntu in my experience | 18:20 |
| clarkb | it was an issue with centos | 18:20 |
| clarkb | since the systemd switch things have been very stable in debuntu with things like network configuration and daemon management | 18:21 |
| clarkb | I think the upstart -> systemd move was the last big lift with debuntu | 18:21 |
| fungi | i expect because rhel/centos type distros traditionally haven't suported in-place upgrade workflows between major versions, there's a different approach to backward-compatibility between releases | 18:21 |
| TheJulia | Oh yeah, centos 7->8, 8->9-stream was painful. But then the other question is how much technical debt is a distro maintaining? | 18:21 |
| clarkb | I remember the issue mnasiadka filed was about the centos images not labeling their partitions correctly | 18:23 |
| clarkb | I believe that they probably still aren't labeled correctly but we gave up and just work around it now | 18:23 |
| mnasiadka | https://issues.redhat.com/browse/RHEL-93891 | 18:23 |
| TheJulia | mnasiadka: thanks, I was about to ask for that | 18:23 |
| mnasiadka | Found it ;) | 18:23 |
| clarkb | that issue is still not public | 18:23 |
| clarkb | so ya I was neverable to even engage with it | 18:23 |
| TheJulia | wow, that stinks | 18:24 |
| TheJulia | yeah, restricted to RH Engineering | 18:24 |
| mnasiadka | clarkb: and still unresolved and at least on my end looks like nothing happened | 18:24 |
| TheJulia | I suspect its not correctly labled/filed for them to pick it up | 18:25 |
| TheJulia | FWIW | 18:25 |
| clarkb | and I can understand that changing that might be too risky vs the impact, but if I'm not even allowed to be told that via a properly filed issue with the upstream well now you can probably understand why some of us don't like dealing with this | 18:25 |
| mnasiadka | Anyway, I’m not that interested in Stream, I did the DIB/Glean stuff to get Rocky 10 in OpenDev | 18:25 |
| clarkb | and to be clear ubuntu typically doesn't fix a lot of issues like that either. But you usually get a response telling you that you have to wait for the next release or it wont be fixed at all and you can have that conversation | 18:26 |
| TheJulia | mnasiadka: are you able to view/edit the content on that item you linked? | 18:29 |
| mnasiadka | TheJulia: yes, no movement in that ticket since 12th June, assigned to pm-rhel | 18:31 |
| mnasiadka | No comments, no nothing really | 18:31 |
| TheJulia | Your going to need to point them to the fact that this is the images they are publishing, that they don't conform to BLS spec. They may be okay with that | 18:31 |
| TheJulia | furthermore, I think you'll get more traction with the component set to centos-stream-release | 18:32 |
| TheJulia | you may be able to change Security Level, if not I can change it, although the bot changed it outright. No idea why. | 18:33 |
| spotz[m] | There's a lot of bot stuff going on. If you give Julia and I the ticket we can poke folks | 18:36 |
| TheJulia | RHEL-93891 | 18:36 |
| clarkb | prior to that issue I think the main issues were the x86-64-v3 requirement and the removal of backward compatible network config from NetworkManager (so network config has to be native NetworkManager config) | 18:37 |
| TheJulia | Yeah, a lot of stuff like that is because we can't actually buy any hardware which is not v3 | 18:37 |
| clarkb | support for NetworkManager direct configuration was added to glean and we were lucky that all of our nested virt capable providers are also x86-64-v3 capable instances so we just run centos 10 stream on the nested virt labels | 18:37 |
| TheJulia | Similarly, we can't buy native legacy bios booting hardware anymore | 18:38 |
| clarkb | huh I still own plenty :) | 18:38 |
| TheJulia | so why keep bios boot around when it is also a huge PITA | 18:38 |
| clarkb | granted most of them are machiens in my closet collecting dust but my fileserver is v2 and not v3 | 18:38 |
| TheJulia | And you restrict the memory range for PCI interactions | 18:38 |
| clarkb | one reason to keep bios around: no expiring keys that will break your boots on september 11, 2025 | 18:39 |
| clarkb | (though disabling secure boot has the same effect) | 18:39 |
| TheJulia | ... I do love how they shipped updates and people just didn't want to upgrade to a version with the key updates | 18:39 |
| TheJulia | so similar effect in the end | 18:40 |
| clarkb | I suspect the v2 vs v3 issue is bigger than bios vs uefi. My fileserver is uefi but not v3 | 18:41 |
| clarkb | however when it comes to the issues opendev ran into I suspect they are equiavlent | 18:41 |
| TheJulia | Yeah, bios gear has basically disappeared | 18:42 |
| TheJulia | ... although some vendors have special UEFI firmware as well that... yeah | 18:42 |
| TheJulia | just... ugh | 18:42 |
| clarkb | tumbleweed has mixin packages for v3 support. I'm not sure if opensuse leap does this too or if tumbleweed is the beta test. But I erally like the appraoch continues to support old hardware while speeding up processes on hardware that can take advantage of it | 18:45 |
| clarkb | `zypper search -- -v3` says libboost, libz, python, openssl, some media stuff, and more all have v3 mix in packages | 18:46 |
| clarkb | a downside to this approach is you can't do this for the kernel. You'd need to build and ship completely separate kernels I think | 18:48 |
| clarkb | then have grub updates select the correct kernel based on the current hardware. But if you install the old hardware kernel too then you can potentially move root disks around and override the kernel via grub at boot | 18:48 |
| TheJulia | At the same time, I can't in good conscious have a customer rolling out a huge bunch of infra on 12+ year old gear. There is a balance to be had. | 18:52 |
| clarkb | TheJulia: I think the real range is like ~4 years | 18:54 |
| clarkb | (atom didn't get v3 until 2021 and there are many valid reasons to run atom devices in professional settings like embedded devices, factories, etc) | 18:54 |
| TheJulia | V4 is more like in the last 4 years, fwiw. Started with Intel Skylake and AMD Zen4s. V3 is Haswell/Excavator (okay, 10 years for Excavators) | 18:56 |
| clarkb | atom does not have v4 yet | 18:56 |
| clarkb | the first atom release with v3 according to wikipedia was gracemont released november 2021. No releases yet with v4 | 18:57 |
| TheJulia | wow, 3 years for atom | 18:57 |
| TheJulia | that is... a sign to migrate away from Atom microarch | 18:57 |
| clarkb | I would disagree. Atom fits a particular use case but one that is very useful if you need it | 18:57 |
| clarkb | low power soc's for hardware dedicated to accomplishing some task (which might be in cars, factory machinery, etc) | 18:58 |
| TheJulia | At this point, I'd still ask why not Arm :) | 18:59 |
| TheJulia | anyway, its academic at that point | 18:59 |
| TheJulia | mnasiadka: following up, its now assigned to a human and a team. I've also removed the security. We'll try to get some clarity on fixing process/bots since the original component set was apparently wrong and a bot made a hidden comment but that doesn't help someone who doesn't have elevated rights. We'll try to get it sorted out. | 19:15 |
| mnasiadka | TheJulia: thanks, we did a workaround in DIB, so that’s not that important - but if all tickets raised by people like me get to purgatory, then it’s not fun :-) | 19:22 |
| spotz[m] | We're working on that issue | 19:53 |
| TheJulia | mnasiadka: more information requested on the bug. | 20:05 |
| fungi | i have a bunch of new machines that give me the option to switch from uefi to bios boot, so i don't agree that it's entirely gone away (though i do agree that there's basically nothing new being sold that is incapable of uefi, at least for x86-based systems) | 21:41 |
| clarkb | doesn't look liek you can previwe the formating of comments in the red hat bug tracker? In any case I posted a response there to add more details | 23:36 |
| *** iurygregory_ is now known as iurygregory | 23:41 | |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!