| *** clarkb is now known as Guest24718 | 07:46 | |
| *** Guest24718 is now known as clarkb | 13:47 | |
| opendevreview | Ghanshyam proposed openstack/governance master: Show inactive project status in project.yaml https://review.opendev.org/c/openstack/governance/+/958229 | 17:01 |
|---|---|---|
| JayF | I just read the eventlet deadline move change that landed during my sabbatical; I'm going to note specifically: GR-OSS will not be dedicating resources to keeping the lights on for eventlet that long. It's actively harmful to the ecosystem to wait until 2027.2 to completely remove it. | 18:27 |
| opendevreview | Jay Faulkner proposed openstack/governance master: Retire metalsmith project from bare metal program https://review.opendev.org/c/openstack/governance/+/958297 | 18:33 |
| opendevreview | Jay Faulkner proposed openstack/governance master: Retire metalsmith project from bare metal program https://review.opendev.org/c/openstack/governance/+/958297 | 18:47 |
| sean-k-mooney | JayF: you realise we actuly have not even deprecated supprot for eventlet in most of the project right. we are racying ot see if we can get to the point of having expermintal support for runnign in threaded mode in 2026.1 so that we can deprecated it there and be able to remove it in 2027.1 the over all time frame has not really changed the doc was just updated to refelct the | 19:35 |
| sean-k-mooney | fact that we are treatign 2026.1 as streach goal to be able to run without it and 2027.1 is the last release we expect to supprot using eventlet | 19:35 |
| JayF | sean-k-mooney: I think those arguments ignore an impending technical train coming down the tracks with every new python version. We can't just say things will work for a period of time when it's extremely likely we don't control all the pieces to make that true. | 19:36 |
| JayF | sean-k-mooney: my experts have been telling me that already python 3.13+eventlet is dubious | 19:36 |
| sean-k-mooney | we have it fuly working on 3.13 now | 19:36 |
| JayF | the subset of eventlet+openstack we test in CI works, yes | 19:37 |
| JayF | I do not think that can be implied to say "eventlet always works in python 3.13" | 19:37 |
| sean-k-mooney | well it works enouch for debain right now | 19:37 |
| sean-k-mooney | to be clear the document is seting the hard deadlien for when we need to be done | 19:38 |
| sean-k-mooney | its not the dates we are targeting | 19:38 |
| sean-k-mooney | nova might and i stress might be able to run wihtout devstack next cycle | 19:38 |
| sean-k-mooney | water very likely will | 19:39 |
| JayF | I've been in the community long enough to not trust us to push that deadline out yet again, consequences be damned. I'm trying to do the best I can to ensure that doesn't happen again. | 19:39 |
| sean-k-mooney | well we cannot deprecate supprot for it until we ahve a working repalcement so i dont know what documenting a timelien that does not match realtiy would have done | 19:40 |
| JayF | if we have eventlet deliverables in 2027.2, that means until end of of 2028, more or less, we'll have eventlet software to maintain | 19:40 |
| sean-k-mooney | other then sete bad expectation for everyone invovled | 19:40 |
| JayF | which is absurd from my pov | 19:40 |
| sean-k-mooney | JayF: that not wha tthat says | 19:40 |
| sean-k-mooney | 2027.2 is the first release with no eventlet | 19:41 |
| JayF | ah, so I'm 6 months off | 19:41 |
| JayF | the point still is valid | 19:41 |
| sean-k-mooney | that the release where we are deleteing any remaining code | 19:41 |
| JayF | essentially two more years of supporting a broken model | 19:41 |
| JayF | because we have to support what we release for 18 months | 19:41 |
| JayF | that's the real "eol" for eventlet openstack | 19:41 |
| sean-k-mooney | effectivly but keep in mind that supproting it for 18 months does not mean on all python version | 19:42 |
| sean-k-mooney | JayF: one of the thigns wew have been talking about as we have been doign this si that we may only supprot the non eventlet mode in 3.14 | 19:42 |
| sean-k-mooney | and to keep running in the legacy mode you will need to use an older python if eventelt is not fucntional on the new one | 19:43 |
| JayF | That would reduce my concern a nontrivial amount, but I still am not "ok" with the plan we've created. | 19:43 |
| JayF | I'm clearly in the minority there, and that's fine, but I didn't want to let the topic lie because, as I said above, we have a tendency as a community to let the tail of these kinds of migrations last indefinitely long. | 19:43 |
| sean-k-mooney | well nova was ask by operator to have at least 1 slurp release that supprot both | 19:43 |
| sean-k-mooney | we are trying to make that 2026.1 | 19:44 |
| JayF | I don't think operators are informed well enough to make that decision tbh | 19:44 |
| sean-k-mooney | if we dont get ther we either need to say no to the operator that wnated it or make that 2027.1 | 19:44 |
| JayF | I'm *thrilled* Ironic is not doing that path. | 19:44 |
| sean-k-mooney | we want our default to swpa before that | 19:44 |
| JayF | well, I guess "doing" is probably the wrong verb; we're done. | 19:44 |
| sean-k-mooney | im not sure either. we are about 50% maybe more done with nova this cycle | 19:45 |
| sean-k-mooney | we might also get the conductor workign without eventlet | 19:46 |
| sean-k-mooney | that will just leave nova-compute and the console proxies | 19:46 |
| JayF | We have some already-written stuff up on ironicbaremetal.org in the blog about performance changes | 19:46 |
| JayF | which could be gleaned for other projects, I suspect, to help with transition | 19:46 |
| JayF | I suspect the song may be different but it'll all be the same key | 19:46 |
| JayF | https://ironicbaremetal.org/blog/coming-soon-threading/ | 19:47 |
| sean-k-mooney | so far we are not seeign any performnce concrnes from the move but we are hoping to figure out how to do some larger testing | 19:47 |
| JayF | we have no *concerns*, but the shape of the performance is changing from a "what shows up in your graphs" and/or "how do I tune my containers" standpoint | 19:47 |
| JayF | although I guess Ironic may just have more threaded workloads because of the nature of what we do | 19:48 |
| sean-k-mooney | so in nova we used eventle tmainly as a threading lib not as a networkign layer | 19:54 |
| sean-k-mooney | so moving from greenthreads to a real thread pool is a very good fit because we alrady used it as one | 19:54 |
| JayF | yeah | 19:55 |
| JayF | we just need to be sure folks like kolla-ansible or similar with container-based deployments | 19:55 |
| JayF | loosen up the restrictions | 19:55 |
| JayF | since we'll likely be using more ram to get more performance | 19:55 |
| JayF | a good tradeoff, but one that changes the shape of performance | 19:55 |
| sean-k-mooney | even then while we will have a litte more usage we didnt see much sofar | 19:55 |
| sean-k-mooney | but we have a very small data set | 19:56 |
| JayF | Ironic gets a lot of free bonus data | 19:56 |
| sean-k-mooney | at least in the ci the memory usage for the nova schduler has not really increased much | 19:56 |
| JayF | from metal3.io CI and ironic-image CI | 19:56 |
| JayF | over in k8s land | 19:56 |
| JayF | they run in configurations that sometimes are unique to our ci configs which is nice for getting new data | 19:57 |
| JayF | that's how the performance regression surfaced | 19:57 |
| sean-k-mooney | so on the watcher side im really not worred about the meory usage because it used greenthread with a default limit of 4 https://docs.openstack.org/watcher/latest/configuration/watcher.html#watcher_decision_engine.max_general_workers | 19:58 |
| JayF | I think our threading cap is 300 by default, pre and post eventlet | 19:58 |
| * JayF -> AFK for a while | 19:58 | |
| sean-k-mooney | what i woudl be reallin interested in is if cern or a similar operator that updates to flamingo could try the treaded schduler for a day, a week, perhaps even on only 1 of n scchulder | 20:00 |
| sean-k-mooney | and they could let us know how it works at scale. if they notice no real delta that would be awsome. | 20:01 |
| sean-k-mooney | the nova api already was scale by process not thread and also had very littel eventlet usage. im considering proposing making the threaded mode the default for it next cycle but that a ptg dicussion | 20:02 |
| JayF | kubajj in #openstack-ironic is who I'd ask, or email Arne, if you wanted some info. I know they run a couple of patches but nothing that'd be that impactful (they needed a way to pass a kickstart file thru openstack server create into the ironic anaconda driver) | 20:04 |
| clarkb | fwiw I don't think we need to deprecate eventlet its an implementation detail and not user facing | 20:04 |
| clarkb | if the software gets refactored but continues to function the same way as observed by the humans using it you can just change things | 20:04 |
| JayF | clarkb: Ironic didn't. | 20:04 |
| clarkb | some of the library code might have to as those implementation details matter there. But they do not matter to someone deploying nova | 20:05 |
| clarkb | what matters to someone deploying nova is that the service continues to ingest the same configuration and behave consistently with that configuration | 20:05 |
| sean-k-mooney | well wether we deprecate or not we do have to deal with the fact that its likely that some functionaltiy will be lost | 20:05 |
| sean-k-mooney | we are groing to try and prot the in tree virt driver liek vmware but we really dont have a way to test it or zvm | 20:06 |
| clarkb | ya so you'd deprecate the vmware driver in that case | 20:06 |
| sean-k-mooney | im not sure if every cinder voluem driver "will just work" | 20:06 |
| clarkb | not eventlet | 20:06 |
| sean-k-mooney | or if they will also have similar problems | 20:06 |
| sean-k-mooney | ya perhaps | 20:07 |
| sean-k-mooney | clarkb: to be entrilly transparent i didnt wnat to offer the option of swtichign between backend | 20:07 |
| sean-k-mooney | when i was orginally POCing the removal in nova i epxlcitly didnt | 20:07 |
| JayF | Ironic'ers are used enough to hardware vendors breaking us and having to do quick fixes, so we might be a little less afraid of us breaking us :D | 20:08 |
| sean-k-mooney | so the nova core team is really only doing because we were directly asked too | 20:08 |
| clarkb | yes I think that should never have been a toggle too. Once you believed it works without eventlet make the switch and move on. But as you say people asked for it and the decision was made | 20:09 |
| sean-k-mooney | for watcher we have one compoent called the applier that we need to port everything else was started and complete this cycle | 20:09 |
| clarkb | a good lesson for the future if making changes that are largely transparent to operators or users. | 20:09 |
| sean-k-mooney | so next cycle we will proably defatl to treading and in 2026.2 i expect to start deleteing the eventl code | 20:09 |
| clarkb | kinda like not asking end users about type annotations or which sql libraries are supported. We do what makes sense for the project and users roll with them | 20:10 |
| gmaan | well, deadlines are modified considering the community bandwidth and current progress. we cannot always force things on upstream to do with hard deadlines. If it is not working on future python (at least python 3.13 is not issue now), then I am all ok to stay us on pythin version where things work fine | 20:42 |
| gmaan | there are lot of best we can/should do but that is not reality always | 20:42 |
| gmaan | the best will be if we can complete things even as per new deadlines but I will not be surprise if we do not | 20:44 |
| gmaan | and I am talking about all the OpenStack projects (~30 where it is applicable) and not just a few one | 20:45 |
| JayF | Yeah, you are sorta pointing at my real concern: if we already pushed the deadline back once, I'm very concerned we'd push it back again. Our community tends to be last minute at times with deadlines. | 20:45 |
| clarkb | right its less about trying to stick to a strict deadline and more that if we give ourselves 2 years then that is the fastest it will happen | 20:46 |
| clarkb | if we aim for 1 then maybe it gets done sooner | 20:46 |
| gmaan | more matters here is to find more volunteer to finish the work. if we leave on projects then it will not be that fast | 20:48 |
| gmaan | I can bet ~40% projects even not aware of old or new deadlines | 20:49 |
| JayF | To put it plainly: if you're a PTL or DPL liason for an OpenStack project, and you're not aware there is an urgent need to migrate from eventlet, are you doing your job? | 20:50 |
| gmaan | true, that is the challenges we are facing since many years, project activeness and the maintainers bandwidth. we can easily say and retire them but i do not think that is solution to any of the existing problems. | 20:53 |
| gmaan | one day, oslo, keystone might be in same plate. | 20:54 |
| fungi | i clearly picked the wrong time to go find food | 20:54 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!