| gouthamr | tc-members: mnasiadka will be chairing tomorrow's meeting as I'm away.. i started a meeting agenda etherpad: https://etherpad.opendev.org/p/openstack-technical-committee-meeting - it has standing items from our old wiki page. i don't remember if we had any new topics for tomorrow's meeting.. if you had added any to the wiki before it died, could you please do add them to this etherpad? | 00:10 |
|---|---|---|
| gouthamr | ty for chairing mnasiadka! :) | 00:11 |
| mnasiadka | Reminder to myself and other folks from Europe - the meeting is one hour earlier due to time change ;) | 05:24 |
| fungi | yes, this is the "week of confusion" between eu dst end and usa dst end | 14:31 |
| dansmith | not to be confused with the "land of confusion" | 15:06 |
| dansmith | but they really should make a song about this week too | 15:06 |
| dansmith | it should probably use some sort of off-by-one time signature for that "makes you uncomfortable" quality | 15:07 |
| fungi | we can still tap genesis for that, i'm sure they're up to the task | 15:07 |
| * dansmith nods | 15:07 | |
| mnasiadka | tc-members: a gentle reminder that our weekly meeting will be hosted here in around ~43 minutes | 16:17 |
| mnasiadka | #startmeeting tc | 17:01 |
| opendevmeet | Meeting started Tue Oct 27 17:01:19 2026 UTC and is due to finish in 60 minutes. The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot. | 17:01 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 17:01 |
| opendevmeet | The meeting name has been set to 'tc' | 17:01 |
| mnasiadka | #topic rollcall | 17:01 |
| gtema | o/ | 17:01 |
| ChristianBerendt[m] | o/ | 17:02 |
| frickler | \o | 17:04 |
| mnasiadka | courtesy-ping: noonedeadpunk, spotz[m], dansmith, cardoe | 17:04 |
| dansmith | o/ | 17:04 |
| cardoe | o/ | 17:07 |
| mnasiadka | #topic Last Week's Action Items | 17:07 |
| spotz[m] | o/ | 17:08 |
| noonedeadpunk | o/ | 17:08 |
| mnasiadka | Looking at last weeks meting logs - I can see only one - a small reminder for anybody that did not yet comment on the Rust use-case analysis for Keystone - to post their thoughts and up to date votes - which could probably help gtema to work further on the document | 17:09 |
| * noonedeadpunk messed with with time-saving | 17:09 | |
| mnasiadka | #link https://review.opendev.org/c/openstack/governance/+/998652 | 17:09 |
| gtema | right, 2 votes still missing, and honestly the response from goutham is more confusing rather than helpful, would need to wait him to put the reasonable comments | 17:10 |
| mnasiadka | High level responses are usually confusing, let’s get a week more for some comments and a low-level one from gouthamr | 17:11 |
| gtema | right, in the meanwhile I would update the text addressing reasonable comments | 17:12 |
| mnasiadka | I have skimmed the last week meeting for any other action items, but I fail to find any | 17:12 |
| mnasiadka | But if anyone wants to point out I forgot something - please do! | 17:12 |
| cardoe | I'll comment today. | 17:13 |
| dansmith | gtema: are you going to reply with what you want clarified? | 17:13 |
| gtema | dansmith - what do you mean exactly? | 17:13 |
| dansmith | TBH, I don't know what is confusing about gouthamr's review so it'd help me too if you could point out specific things | 17:13 |
| gtema | I read the comment from goutham as: would be good to experiment, I was suggesting previously to focus on sdk/cli instead, but still -1 with no sufficient reasoning | 17:15 |
| gtema | I would update the doc addressing eventlet comments, adding performance measures, maybe rephrasing bit more the auditing benefits | 17:16 |
| noonedeadpunk | we have a different understanding of sufficiency I guess | 17:16 |
| gtema | we do | 17:16 |
| mnasiadka | Ok, let’s try to work out a common understanding in the Gerrit patch - if there are questions for clarification from gtema - please raise them in Gerrit | 17:17 |
| noonedeadpunk | as critics seems to be treated either as non-constructive or non-sufficient | 17:17 |
| mnasiadka | Criticism of criticism can be also viewed as non-constructive or non-sufficient :) | 17:18 |
| gtema | what I am however not going to add is something contradicting the current official process - I am not going to interpret it in the reverse order. | 17:18 |
| spotz[m] | I think if we can get a consensus on what items need more info or clarification it would give us a next step | 17:19 |
| gtema | yes spotz, but this goes into both directions | 17:20 |
| noonedeadpunk | well, current proposal is packed with things that are not even true | 17:20 |
| noonedeadpunk | which makes it very hard to distinguish and validate on the face value | 17:20 |
| gtema | what you do not see as true is not necessarily untrue for others | 17:20 |
| dansmith | okay but when multiple people agree on the untruths, it's a problem | 17:21 |
| noonedeadpunk | sure, can be that as well. would be good if you could elaborate more then, rather them be dissmisive | 17:21 |
| spotz[m] | I think that’s part of the problem Dan | 17:22 |
| mnasiadka | Ok, as I said - let’s try to work out things that need clarification in Gerrit | 17:24 |
| noonedeadpunk | ++ | 17:24 |
| mnasiadka | And I just want to remind, that we’re on step 1 - which is use case analysis - so let’s focus on that ;-) | 17:24 |
| mnasiadka | #link https://governance.openstack.org/tc/reference/new-language-requirements.html#step-1-use-case-analysis | 17:24 |
| mnasiadka | Step 2 lists a lot of things that need definition, which in my mind is a lot of work, which I have already emphasised in my comments | 17:25 |
| gtema | mnasiadka - but you explicitly said you want step 2 to be addressed before you can go for step 1 | 17:25 |
| mnasiadka | I did not explicitly said that, I’d like to see that we have people that will address step 2 | 17:26 |
| mnasiadka | Beyond the 3 core reviewer Keystone team | 17:26 |
| gtema | from where do you want such people to appear? | 17:26 |
| noonedeadpunk | (are all three are signing up for that though? as I haven't seen them reviewing the proposal) | 17:27 |
| gtema | we do not have a clone machine | 17:27 |
| mnasiadka | Well, step 2 needs people doing things, currently I only see gtema trying to address questions from TC members - which just makes me worried | 17:28 |
| mnasiadka | I understand we don’t have a clone machine - and a lot of projects are in the same state as keystone is | 17:28 |
| noonedeadpunk | and again I think we're having consensus that this should be applicable only to new projects so far | 17:28 |
| fungi | the way i'd expect that to transpire is that first keystone needs to end up severely broken and have that publicly known, then people whose businesses depend on it being in working condition begrudgingly agree to assign employees to work on fixing it back up and maintaining it. while it's still being minimally maintained nobody will see the urgency | 17:29 |
| gtema | and at the same time I have a very strong perception, that many TC members are trying to put different expectations to the Keystone team compared to other projects | 17:29 |
| fungi | basically unless the current maintainers step away from it, i have doubts the situation will change | 17:30 |
| noonedeadpunk | fungi: well, it's hjard to do when keystone team is quite explicit that no new features or refactoring or beyond bug fixing will be accepted | 17:30 |
| gtema | I am now doing everything feasible trying to improve the situation around Keystone, but I feel I get only blockers, rather then help | 17:30 |
| dansmith | it's also not good signaling to community members to have a rewrite already in place | 17:31 |
| gtema | noonedeadpunk - now what you are saying is very untrue | 17:31 |
| spotz[m] | Lettstep back for a minutes. There is keystone leave it alone. Then there is a new identity project that would like to use rust | 17:31 |
| noonedeadpunk | gtema: oh, if smth has changed since last discussion - I am very sorry then | 17:31 |
| noonedeadpunk | I could mis-understood you | 17:32 |
| mnasiadka | spotz[m]: the current patch in Gerrit does not imply there is a new identity project, does it? | 17:32 |
| spotz[m] | If you think of it like that it’s not different from Horizon and Akyline | 17:32 |
| noonedeadpunk | But I was under impression that all new features or improvements are expected only be in keystone-rs | 17:32 |
| spotz[m] | It doesn’t and I think that’s where a lot of confusion is because this is being addressed as keystone | 17:33 |
| gtema | noonedeadpunk - no, and we explicitly underlined that in 2 last PTGs | 17:33 |
| noonedeadpunk | if that's not a case - really sorry, I didn't mean to | 17:33 |
| noonedeadpunk | (yeah, I had to miss this ptg completely due to internal /o\) | 17:33 |
| noonedeadpunk | but my memory failed me then - sorry, I am retracting my statement | 17:34 |
| gtema | noonedeadpunk - there is, however, a hack about that - we can't implement certain features in the current code base and it would not be possible to have them in the "current" keystone | 17:34 |
| mnasiadka | I think there are two different topics - one is the rationale why we need a new language for a new identity project - second one is that it seems to me (and probably others) that gtema is driving that alone (and maybe that’s fine for the time being) | 17:35 |
| gtema | for everybody: as I mentioned few times, we can explicitly mention it as a new project, but there are challenges to solve: different services under same service type make it not trivial to co-exist | 17:35 |
| gtema | also keeping compatiblity in sync is much more hard this way | 17:36 |
| gtema | mnasiadka - I believe you are right | 17:36 |
| gtema | and that's why I don't want to mix those into step 1 | 17:36 |
| noonedeadpunk | it would be really great if you could elaborate why these some things can't be implemented in /v4 in existing language in rationale for the new language | 17:37 |
| gtema | I tried, but those comments are usually disrespected as not true. e.g., we need to start doing mTLS with terminating tls connection on the keystone. Yes, python can do this, but not under the load a public cloud is having | 17:38 |
| dansmith | and "rust makes it easier" is useful data, but not the same as "can't be implemented" | 17:38 |
| gtema | right - this is what is very hard for me | 17:38 |
| gtema | I can prove with numbers: 10x more throughput with 1/10th of the RAM usage. How to turn it into the argument for allowing Rust? | 17:39 |
| noonedeadpunk | and, um, terminating connection by rust-based granian is not sufficient for mtls? | 17:39 |
| dansmith | gtema: that's basically the justification by which we approved go... | 17:39 |
| gtema | noonedeadpunk - connections MUST terminate inside the app, otherwise it violates the zero-trust what some customers are explicitly requesting. Similar is valid for oauth2/oidc federation - Keystone itself MUST do it and not rely on proxy in front | 17:40 |
| dansmith | and those are things that come with tradeoffs that are on the other side of the argument that a lot of us have about whether or not the delta with the rest of the projects and processes are worth that | 17:40 |
| noonedeadpunk | do we have go under list of approved ones today? and we have all the infra for go in and ready? | 17:41 |
| gtema | if you point towards Keystone using go I would frame it the following way: Rust was chosen as offering much higher implementation safety guarantees, there is a implementation created in last 2 years. I am not going to rewrite it in go just because go is accepted, I do not have time for that and I do not believe in it being superior. | 17:43 |
| noonedeadpunk | gtema: hm, I kinda get that, I think I was more confused about wsgi-server, as it's the thing that loads application by module load. So in my mind it's not breaking the trust. | 17:44 |
| noonedeadpunk | but I can admit I may be wrong in thinking this way | 17:44 |
| gtema | and for go therere are anyway no step 2 deliverables from what I see | 17:44 |
| noonedeadpunk | I would pick rust over go as well tbh... | 17:44 |
| noonedeadpunk | if these are only 2 things to select from | 17:44 |
| gtema | hehe, so do many in the security sensitive apps | 17:45 |
| fungi | #link https://review.opendev.org/c/openstack/governance/+/988440 pti: Add out-of-date warning to go guide | 17:45 |
| fungi | mnasiadka: ^ | 17:45 |
| gtema | and nobody writes IAM in python | 17:45 |
| fungi | it's there, just marked as outdated as of 2026-05-25 when change 988440 merged | 17:45 |
| noonedeadpunk | I wonder if we should even offboard it, given we get agreement on rust | 17:46 |
| gtema | what do you mean offload? One of the biggest RAM eaters is wsgi | 17:47 |
| noonedeadpunk | gtema: but then should we even have own iam or just adopt keycloack good enough | 17:47 |
| mnasiadka | fungi: and I think the worry of all of us - that Rust can end in the same way, given we only see gtema pushing for this - but the resolution for adding new languages sort of doesn’t indicate that we should ensure that there are enough people to move that thing forward | 17:47 |
| gtema | that's also one of the reasons I am saying we should not simply try to rewrite what's inside python keystone but need an architecturally different approach | 17:47 |
| gtema | noonedeadpunk - we MUST have iam, keycloak is IdP and not IAM | 17:48 |
| fungi | mnasiadka: no, my bad, it did get removed 2026-06-16 | 17:48 |
| fungi | #link https://review.opendev.org/c/openstack/governance/+/989189 pti: Remove Go doc | 17:48 |
| noonedeadpunk | gtema: ok, yes, right | 17:48 |
| gtema | I was thinking about that around a year and all experiments failed | 17:48 |
| mnasiadka | fungi: right, still it serves as history reminding us how this can end :) | 17:48 |
| mnasiadka | Ok - it’s nearly 45 minutes discussing this | 17:49 |
| gtema | we have keystonemiddleware patches offering oauth2 support, but they are pretty much useless since keycloak (and those modules were created under this assumption) simply cannot offer the rest of what we need | 17:49 |
| spotz[m] | Can we stop for a moment please? | 17:49 |
| mnasiadka | Yes, let’s stop - it’s been over 45 minutes discussing this and we’re going into different directions. | 17:50 |
| mnasiadka | Let’s focus on the use case analysis why rust (and why a rewrite instead of a new project) | 17:50 |
| spotz[m] | Let us forget about keystone. Anyone who types keystone for the duration of this discussion owes me $5. Please just discuss Rust and what it would bring to the project | 17:50 |
| spotz[m] | And that's $5/mention! | 17:50 |
| dansmith | I am skeptical of your authority to collect said debts, so: keystone keystone keystone | 17:51 |
| * TheJulia uses sed to make that $15 | 17:51 | |
| gtema | :) perfect reminder of what I was explicitly mentioning few times already | 17:51 |
| spotz[m] | I will see you somewhere Dan! | 17:52 |
| noonedeadpunk | I can't stop it as it's being mentioned multiple times in the proposal :p | 17:52 |
| spotz[m] | Then that is the first thing about the proposal that needs to be changed | 17:52 |
| gtema | because the process requires so - give a detailed use case | 17:52 |
| noonedeadpunk | it;s literally commit msg topic | 17:53 |
| noonedeadpunk | `Add Rust use-case analysis for Keystone` | 17:53 |
| * noonedeadpunk guess owes $5now | 17:53 | |
| spotz[m] | So let's change that to Add Rust use-case for possible future IDM | 17:53 |
| mnasiadka | So, unless it’s technically impossible to not rewrite Keystone, I think a lot of us agrees that it should be a new project with a new name | 17:54 |
| dansmith | that still implies that a whole project could/would be rust-only or primarily rust and that's my main concern | 17:54 |
| spotz[m] | Even if it is possible, it's not the best use of resources to totally rewrite | 17:54 |
| spotz[m] | But something new written in a new language is not a bad think for the project | 17:55 |
| noonedeadpunk | s/resources/tokens/ | 17:55 |
| fungi | for the discussion which transpired between 2016-12-13 and 2017-03-21 about adding golang, in case people want to see what justifications were used at the time: | 17:55 |
| spotz[m] | think=thing | 17:55 |
| fungi | #link https://review.opendev.org/c/openstack/governance/+/410355 Add Golang CTI | 17:55 |
| dansmith | spotz[m]: I disagree | 17:55 |
| mnasiadka | fungi: erm, we didn’t require use case analysis at that time? | 17:56 |
| spotz[m] | dansmith: I'm open why? If it brought new contributors who want to work in that language. Because it won't get approval status as a solo project | 17:56 |
| gtema | for the sake of clarification I would ask people to list explicitly what they want to see changed in the resolution if they are ready to swap their vote. I feel some are not ready for that no matter what it describes | 17:56 |
| noonedeadpunk | fungi: unfortunatelly I don't see reasonining in this one | 17:57 |
| noonedeadpunk | it would be interesting though | 17:57 |
| mnasiadka | Ok, let’s stop again, it’s 3 minutes before end of the meeting. | 17:57 |
| dansmith | spotz[m]: because we have release, dependency management and community-wide familiarity with python.. creating a new component that is a critical requirement of all the other ones that requires different skills and processes is not a net positive to me | 17:57 |
| fungi | noonedeadpunk: we'd probably need to dig it out of tc meeting logs, looks like | 17:58 |
| spotz[m] | I can understand that take completely. And if those resources could be met? | 17:58 |
| dansmith | during the PTG an example was given of another developer specifically finding a regression in keystone (sorry) that would have been nearly impossible to spot without familiarity with oslo libraries in their own project | 17:58 |
| spotz[m] | valid use of keystone no penalty | 17:58 |
| mnasiadka | Anybody that wants to give gtema guidance on what would make them to be convinced - please comment on the patch (and please stick to the Rust use case analysis, we’ll need to find another venue for all the other issues people see with the implementation afterwards) | 17:58 |
| gtema | thanks | 17:59 |
| dansmith | spotz[m]: it's less about meeting those requirements in the moment as long-term.. like skyline showed up with their full team and shortly after, are now hanging on a thread without the rest of the community really being able to maintain it (also discussed in the PTG) | 17:59 |
| noonedeadpunk | fungi: yeah, well, next time then :D | 17:59 |
| noonedeadpunk | I see how it may end up as skyline, surely | 18:00 |
| fungi | noonedeadpunk: https://review.opendev.org/c/openstack/governance/+/398875 had earlier discussion, the addition of the reference doc for new language requirements | 18:00 |
| mnasiadka | Unfortunately we used up all the time, so no other topics today - but feel free to discuss skyline (which was not the topic) after the meeting ;-) | 18:00 |
| mnasiadka | Thank you all for coming and for the lively discussion :) | 18:00 |
| mnasiadka | #endmeeting | 18:01 |
| opendevmeet | Meeting ended Tue Oct 27 18:01:03 2026 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 18:01 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/tc/2026/tc.2026-10-27-17.01.html | 18:01 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/tc/2026/tc.2026-10-27-17.01.txt | 18:01 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/tc/2026/tc.2026-10-27-17.01.log.html | 18:01 |
| fungi | noonedeadpunk: looks like that change started circa 2016-11-17 so probably the tc meetings immediately prior to it | 18:01 |
| noonedeadpunk | fungi: net positive of go was creation of that document | 18:01 |
| spotz[m] | dansmith: But no one was forced to move to Skyline from Horizon. This is why I think it's a new separate thing so no one is forced to use it without taking the risk | 18:01 |
| noonedeadpunk | which I am really glad we have | 18:01 |
| gtema | and than it dies, that's life. But you will not know if it is doesn't have a chance to be born. Progress cannot be reached without try and repeat until you succeed | 18:01 |
| dansmith | spotz[m]: yep, fair, but skyline/horizon are sort of at the other end of the pipeline from keystone | 18:02 |
| noonedeadpunk | progress also does not mean to constantly scrapping one thing for another thing, and then for yet another one. Evolution is also fine, not always revolution is required. | 18:02 |
| dansmith | noonedeadpunk: +1000 | 18:03 |
| spotz[m] | I understand it, and as an ex-admin I would be very wary of switching unless there was something I couldn't live without and the 2 being close enough to be switched in and out for core features | 18:03 |
| fungi | noonedeadpunk: https://meetings.opendev.org/meetings/tc/2016/tc.2016-11-29-20.01.log.html#l-67 is the first mention i find as a topic in a tc meeting, fwiw | 18:06 |
| dansmith | fungi: be honest, you're going as a librarian for halloween right? | 18:08 |
| noonedeadpunk | ` a lot of people complaining that "the TC doesn't want go"` hehe | 18:08 |
| fungi | dansmith: i prefer "openstack historian" | 18:08 |
| noonedeadpunk | 10 years after and discussion is still the same | 18:08 |
| dansmith | fungi: "Curator of the openstack collection" ? | 18:08 |
| fungi | indeed | 18:08 |
| dansmith | what's the actual outfit then? openstack beach towel of yesteryear around your neck like a cape and a tablet with the IRC meeting archives displayed? | 18:09 |
| spotz[m] | I have one of those Sydney towels somewhere | 18:10 |
| dansmith | me too, but I hope the curator of the openstack collection still has his too | 18:10 |
| fungi | oh i do, it's in my archive though | 18:10 |
| spotz[m] | hehe | 18:11 |
| dansmith | heh, nice | 18:11 |
| fungi | "relics of summits past" | 18:11 |
| fungi | please don't tap on the glass | 18:11 |
| fungi | anyway, based on the timeline, i suspect it was the barcelona summit in 2016h2 where the swift folks at rackspace remotely presented the hummingbird poc/demo | 18:12 |
| fungi | i'll see if i can find that forum session | 18:12 |
| fungi | i have a pretty vivid memory of the session (they joined via videoconference as a group), just not which exact summit it was | 18:13 |
| fungi | wow, the feature branch is still there! https://opendev.org/openstack/swift/src/branch/feature/hummingbird | 18:15 |
| mnasiadka | fungi: that doesn’t look like a golang project at a first sight | 18:17 |
| fungi | mnasiadka: it was a swift feature branch, they weren't rewriting *all* of swift, just one component that had some performance benefits from better async request handling | 18:18 |
| clarkb | I think it was specifically the fileio handling backend | 18:18 |
| clarkb | since file io is generally not async having a system that can run more proper threads was valuable | 18:19 |
| clarkb | I guess python threads are proper threads but they have to deal with the gil | 18:19 |
| dansmith | right and that's a much better use-case for brining in a language.. some small component that is perf (or even security) sensitive, IMHO | 18:19 |
| dansmith | bringing .. no need to soak any of these things in salt water | 18:20 |
| mnasiadka | Maybe salt water fixes a lot of problems (and then adds some) | 18:20 |
| fungi | there were a couple of lengthy openstack-dev mailing list threads earlier in 2016 about the topic: https://lists.openstack.org/pipermail/openstack-dev/2016-May/093795.html https://lists.openstack.org/pipermail/openstack-dev/2016-June/096790.html | 18:20 |
| dansmith | mnasiadka: certainly makes them taste better | 18:20 |
| sean-k-mooney | for what its worth greenlets and asyncio both will give you much fo the benifit of goroutines as both can use high performance eventloops like libuv | 18:21 |
| clarkb | sean-k-mooney: yup and now you can go gil free and your real threads act like pthreads | 18:22 |
| sean-k-mooney | it was never the greenthread core fo eventlet that was problematic it was the monkeypatching | 18:22 |
| clarkb | so you don't even need hacky magic to make file io look ansync when it isn't. | 18:22 |
| sean-k-mooney | clarkb: they are or were also evaualting adding native virutal threads | 18:22 |
| fungi | sean-k-mooney: however, in 2016 the options weren't as numerous | 18:22 |
| sean-k-mooney | well at least libuv did exist | 18:23 |
| sean-k-mooney | my expiernce of writign go is making it easey to write concurnt code is only a good thing if you make it easey to write correct concurrent code | 18:24 |
| sean-k-mooney | and go does not do the latter | 18:24 |
| fungi | another historical resolution i mentioned last week which ties into the golang situation is https://review.opendev.org/c/openstack/governance/+/469265 "guidelines for managing releases of binary artifacts" which was reviewed between 2017-05-30 (right on the heels of golang approval) and 2017-06-23 when it was approved | 18:26 |
| fungi | skimming those mailing list threads, i'd completely forgotten that the designate team wanted to reimplement a component of their service in go as well | 18:37 |
| fungi | aha, it was the first vancouver ("liberty" in 2015h1) summit i was recalling the hummingbird presentation at | 18:44 |
| fungi | mnasiadka: oh, and https://opendev.org/openstack/swift/src/branch/feature/hummingbird/go was where the golang files ended up in that feature branch, ftr | 18:48 |
| fungi | "...object-server, object-replicator and object-auditor..." | 18:49 |
| fungi | those were the bits they finished, so more than one daemon at least | 18:49 |
| fungi | they were also working on doing the proxy server, but looks like they didn't finish it before the branch was abandoned | 18:49 |
| sean-k-mooney | fungi: reimplemnint nova schduler came up breifly but we built placment and shoved the hevery lifting off to the db with sql instead | 19:15 |
| fungi | yes, sometimes "just" changing architecture or approach in significant ways can yield significant performance benefits. even rewriting a project in the same language often does | 19:16 |
| fungi | continuing through those mailing list threads turned up https://etherpad.opendev.org/p/r.c0ae49225bc38c95b36ba6eb56c4da49 getting into the weeds on infra(opendev)-specific requirements for developing projects in yet another language | 20:11 |
| fungi | i quite liked ttx's reminder in https://lists.openstack.org/pipermail/openstack-dev/2016-May/094803.html as to why things seem like benefits to some while appearing to others as drawbacks | 20:14 |
| fungi | and lifeless's direct follow-up pointing out that the costs of introducing a new language need to be borne by the people proposing it, i.e. existing teams can't be strong-armed into doing more work just to support the addition | 20:17 |
| fungi | in the current case, that means teams interested in using rust for an openstack component getting more involved in the release team, qa team, vulnerability management team, tact sig (or opendev collaboratory sysadmins), et cetera | 20:18 |
| sean-k-mooney | for rust, for the most part at least there is a local first mindset | 20:18 |
| sean-k-mooney | i.e. all testing build ectra should be runable locally | 20:19 |
| sean-k-mooney | building out ci still needs to be done | 20:19 |
| sean-k-mooney | but its better then some ecosystems like c/java where just geting a built to work can be a massive lift | 20:19 |
| fungi | though this prediction by notmyname did not come to pass: "I expect that a year from now (or sooner) Swift deployers will be running both Python code with PyPy and compiled Golang code" | 20:27 |
| enricokern | is this a thing going rust? | 21:59 |
| enricokern | just curious | 22:00 |
| enricokern | i mean for openstack components | 22:00 |
| mikal | fungi: but on the flip side, if the procedural barriers to innovation within OpenStack are too high, then the innovation will just happen outside OpenStack and OpenStack will become less relevant over time. I think you could argue this has already happened. Why does Proxmox exist for example? What did deployers see in that which they didn't see in | 22:03 |
| mikal | OpenStack? | 22:03 |
| enricokern | you cant really compare proxmox with openstack | 22:03 |
| enricokern | try to scale proxmox to 100 computes... | 22:04 |
| enricokern | not saying they wont evolve, but stil | 22:04 |
| mikal | Why not though? oVirt, OpenStack, Proxmox all solve approximately the same problem. Sure the scaling might be different, but the 100 hypervisor clouds are also the outliers compared to a simple count of deployments. | 22:04 |
| enricokern | multi tenancy, policies, stuff like barbican, lbaas | 22:05 |
| enricokern | all missing | 22:05 |
| enricokern | not saying they wont evolve, but its like more a software for people replacing a vcenter essentials | 22:05 |
| mikal | I think what OpenStack did better than the other two was having a diverse contributor community, but honestly that too has stagnated over the last decade. | 22:05 |
| enricokern | not datacaneter scale | 22:05 |
| mikal | But not everyone wants data centre scale. Does the OpenStack mission statement specifically call out that it only targets large clouds? | 22:05 |
| enricokern | i think atm openstack is on a track back, same as with proxmox. the vmware disaster pushed both projects. but i rather see openstack in my datacenter with 100+ computes then proxmox | 22:06 |
| enricokern | yes not everyone. thats why we have proxmox | 22:06 |
| enricokern | who does not need dc scale can use that | 22:06 |
| mikal | I think we also have proxmox because it was too hard to do proxmox in openstack. | 22:06 |
| enricokern | you can spin this further, not proxmox, what about libvirt plain on k8s? | 22:07 |
| mikal | (He says having not talked to any proxmox people, but my point is openstack already existed and there are _reasons_ that people chose a different path. One of them will have been ease of innovation versus process barriers). | 22:07 |
| enricokern | then openstack running on top of k8s exists too. etc. | 22:07 |
| mikal | I don't think its bad for there to be competition. I do think you could argue its inefficient though. | 22:08 |
| mikal | I just think OpenStack has this "It was a good idea in 2011 and therefore we can't ever reconsider" thing happening. | 22:08 |
| enricokern | i dont see proxmox as competition, as example we use proxmox to run our openstack control planes | 22:09 |
| enricokern | in a standard setup that we do, our controllers run on proxmox | 22:09 |
| enricokern | including sub services for pxe etc. such as maas | 22:09 |
| enricokern | or firewalls like opnsense | 22:09 |
| enricokern | btw. VEEAM is offering a openstack backup solution too soon | 22:10 |
| mikal | I think you have missed my point though. What I am trying to say is that by erecting process barriers for people who want to "fix" Keystone by trying something new, you don't actually improve the current Keystone, but you do close the door to that new thing being tried at all in the OpenStack ecosystem. | 22:10 |
| enricokern | that alone will be a change for vmware migrations | 22:10 |
| mikal | I think a lot of VMWare shops like VDI, and the OpenStack story is very weak there. | 22:10 |
| enricokern | VDI is no issue, you can use omnissa horizon as example just fine with openstack | 22:11 |
| enricokern | we run 1000 omnissa vms with openstack | 22:11 |
| enricokern | as example | 22:11 |
| enricokern | its just bit pain in the ass doing the provisioning for horizon (omnissa one) | 22:12 |
| enricokern | the problem i see with openstack is the clusterfuck of components | 22:13 |
| enricokern | that get discarded etc. | 22:13 |
| enricokern | openstack should get rid finally of crap nobody uses anymore or is badly maintained sooner | 22:13 |
| enricokern | then rabbitmq. rabbitmq itself is fine, and while we had tons of improvements the past years to that part, it stil is shit | 22:14 |
| enricokern | its not rabbitmq, its the libraries all this python garbage is using | 22:14 |
| enricokern | but it improved alot compared to running openstack now on a current version, vs running openstack 4 years ago | 22:15 |
| mikal | Are you really arguing that because a commercial solution for VDI exists the OpenStack community should care about improving free options? | 22:15 |
| enricokern | the people using VDI do not really care about OSS solutions. VDI is more then just providing desktop | 22:15 |
| enricokern | we basically have no "real" VDI oss solution | 22:16 |
| mikal | What is the incentive to clean up dependencies etc though? The OpenStack development process means that even a relatively trivial set of patches like that would take _months_ to merge. | 22:16 |
| enricokern | cant say, i just load out my thoughts. sorry if i offend you, was not my intention | 22:17 |
| enricokern | i mean thats why we have the tc here, they are to decide :) | 22:17 |
| enricokern | but lately i think we are getting better along everything. I do openstack since 15 years now. and atm we are really on a good way | 22:18 |
| enricokern | thank you again broadcom :P | 22:18 |
| mnaser | mikal: +1000 to this. We are sitting in our own bubble and refuse to move with the time. | 22:21 |
| enricokern | whats your proposal to change that ? | 22:22 |
| enricokern | yourself did alot especially with the vmware topic to drive into openstack @mnaser | 22:22 |
| enricokern | with migratekit etc. | 22:22 |
| mnaser | Reduce the walls of policies that let someone drive innovation | 22:23 |
| enricokern | yeah i agree, especially new features and AI opening should be a topic the tc should adress | 22:23 |
| mnaser | gtema has been driving this rust topic for so long and just keeps getting dragged through and through | 22:23 |
| enricokern | but in the end we need to maintain a product thousand of entities use world wirde | 22:24 |
| enricokern | use | 22:24 |
| enricokern | it is also responsibility | 22:24 |
| enricokern | moving slowly isnt bad per se. maybe not this slow, but stil caution is ok | 22:24 |
| mnaser | except he's having to get other people to agree for something he's driving.. for ages. | 22:24 |
| enricokern | who is "he" ? | 22:25 |
| mnaser | [@gtema:matrix.org](https://matrix.to/#/@gtema:matrix.org) in this case for the rust topic. | 22:25 |
| enricokern | ah ok | 22:25 |
| enricokern | i think in the end it is on us pushing who use it, contribute and do stuff | 22:26 |
| enricokern | but you are much longer on this afaik | 22:26 |
| mnaser | We have far more contributors than I've seen to our cluster API driver and migrate kit and I'm pretty sure it's because it pushes new ideas and lives in accessible places i.e. GitHub than I saw when I saw Ptl of OpenStack ansible or people pushing changes to magnum | 22:26 |
| enricokern | your capi driver honestly was major ímportant change. not this helm shit | 22:27 |
| enricokern | i was using this from one of the first versions, we run hundreds of clusters with it | 22:27 |
| mnaser | nice :) we had kinda kept it on GitHub to attract eyes, different set of contributors.. which we've managed to attract a lot from many places | 22:28 |
| enricokern | we also contributed some changes to migratekit. but capi was a game changer | 22:28 |
| mikal | I don't think there has to be a proposal to change things. If the OpenStack community is entrenched in its ways then that's ok, new communities will form in the gaps. That's how we got onto Proxmox. It just depends on if OpenStack _wants_ to be the relevant community or not. | 22:29 |
| mikal | Another example -- I am sure the Perl community has a vibrant group of people living in their time machine and that's lovely, but it doesn't mean the rest of the world is going to stop changing just for them. It just means they become less relevant over time. | 22:30 |
| enricokern | i think thats what the whole purpose of the TC is. to make this kind of decisions, isnt it? | 22:30 |
| mikal | enricokern: why is it gtema's responsibility to maintain a project they think is inherently not fit for purpose any more when we wont consider the alternative for religious reasons? | 22:31 |
| enricokern | but i agree with you on the time machine topic, if this is a concern people raiase it should get adressed | 22:31 |
| mikal | enricokern: that's why there is a drop off in core reviewers in the important projects, its a hugely unrewarding experience. | 22:31 |
| enricokern | yeah sad | 22:31 |
| mikal | mnaser: also, review my PRs kthxbye | 22:32 |
| enricokern | ;) | 22:32 |
| enricokern | im new to this all here, but first real irc session feels like fcking shitload of tension exists | 22:32 |
| mikal | mnaser: oh you did! | 22:33 |
| enricokern | i must say i contributed some stuff to horzion and even talked to them on the last PTG etc. in the video chat etc. chill people, and all works cool | 22:33 |
| spotz[m] | enricokern: It was a heated subject with folks feeling strongly on both sides | 22:36 |
| mnaser | mikal: I merged one and the other is under test thanks :) | 22:36 |
| mikal | mnaser: yeah, I only found that stuff while chasing a magnum flake in kolla-ansible CI which was -1'ing my unrelated patches. So I am totally fine with you tweaking them if needed, I just wanted kolla-ansible to stop being sad about my things. | 22:37 |
| enricokern | i feel kind of sad if people having the same goal cant get along, but well thats how it is | 22:37 |
| clarkb | as a person who ends up holding the bag on a lot of the tech debt around here (whether thats random devstack things or cleanup related to older branches etc) I think it is worth considering what the imapcts are to the community at large before making big decisions like deciding to maintain software in an entirely new language. That said I have also tried to encourage the | 22:38 |
| clarkb | openstack community to practice making hard decisions more often. I think at times the bigger underlying issue isn't the actual decision its that we've forgotten how to make decisions | 22:38 |
| clarkb | how meta | 22:38 |
| clarkb | once upon a time we piled into rooms every 6 months and did this. There were heated discussion but we had practice making decisions | 22:39 |
| clarkb | I'm not saying we need to fly around the world and pile into rooms, but maybe the actual practice of making decisions was more important than we realized | 22:39 |
| mikal | clarkb: I think its true that some people (including you) are carrying more of the tech debt than others and I think that's unfair. On the other hand, how much of that tech debt is because OpenStack is simply too understaffed to maintain the size of thing it grew in the happy times? It seems like one way to improve the tech debt situation is simply | 22:41 |
| mikal | to declare more things to be dead, archive them, and move on. | 22:41 |
| mikal | clarkb: isn't that really what the Keystone team is trying to say here? That they think the best way for them to pay down the Keystone technical debt is to pivot to something else? | 22:41 |
| clarkb | mikal: I think all of that is related for sure. And with the infrastructure stuff where I can make decisions we definitely declare things dead and move on. My point was that its still worth having a discussion now to think about what impacts the decisions we make today have for tomorrow | 22:42 |
| mikal | Oh, I am definitely not saying we should add languages willy nilly. I do think the fact that Keystone's entire core team (as best as I can tell) think this is the right path is a really strong signal. Frankly the state of the art for programming languages has definitely moved in the last 15 years. | 22:43 |
| clarkb | so its unfair to say that having a dicsussion is a problem. Instead I would characterize it as we don't know how to have the discussion effectively anymore and that is the problem | 22:43 |
| enricokern | i must say in my experience as an "outsider" who before never was involved into anything and just being a "consumer" of openstack i never figured issues out. the opposite is the case, ive seen openstack pushing out failed components. The only take is, that it often did take to long todo that | 22:43 |
| mikal | I think that's fair. As an outsider it seems to me this discussion has been going for a very long time and made very little progress. That is, it is unlikely to ever actually result in a decision. | 22:43 |
| mikal | enricokern: its hard. There's always a small set of users who depend on something which turned out to be a mistake, and OpenStack is very hesitant to cut them loose. But on the flip side, those users are also not contributing to support their edge case and there's only so far that charity can go. | 22:44 |
| enricokern | true | 22:45 |
| mikal | I personally think Nova should support exactly one hypervisor stack for example, but that is not an opinion held by others. | 22:45 |
| enricokern | nova can stil do more then kvm? | 22:47 |
| enricokern | who is using that | 22:47 |
| enricokern | that would be a prime example | 22:47 |
| mnaser | VMware driver for awhile and still now I think | 22:49 |
| enricokern | guess as long people maintain it is fine? | 22:50 |
| mikal | I see ironic, libvirt (multiple hypervisor options), vmwareapi, and zvm. I'd probably keep ironic because its not a "hypervisor" and then remove everything else that wasn't libvirt / kvm / qemu. That said, my opinion isn't super relevant given I've been idle in that project for ages. | 22:50 |
| mikal | enricokern: that's the point though -- the people who landed a lot of these options are long gone. | 22:50 |
| mikal | enricokern: although neutron / cinder are probably better examples of the land-and-abandon strategy than nova. | 22:51 |
| enricokern | ironic never worked for me. we use MaaS for as reason, not being bound to canonical, but its all shit, we made best experience using this as example | 22:51 |
| enricokern | but thats also , everyone doing its own thing with it and thats where openstack shines | 22:52 |
| enricokern | you can combine it. like use, we use MaaS for provisioning, osism/kolla for deployment and network with our own netbox based implementation | 22:53 |
| enricokern | so its liek kind of LEGO pieces | 22:53 |
| enricokern | others use ironic for bare metal, sonic for switches etc. | 22:53 |
| enricokern | and really, quality the past years improved immense | 22:54 |
| enricokern | 2-3 years ago no deployment was hassle free, no we are on a way with kolla where new installs and upgrades are "almost" issue free | 22:55 |
| enricokern | i remember when i stil hacked together own chef based openstack installs... | 22:56 |
| sean-k-mooney | enricokern: on the proxmox thing im retryyireally hard to make it work so i dont have to go fix the bug if i find them in my home lab but in comarison to openstack there are so many "cloud" feature that its missing. and it does nto give a log over just plain libvirt as a virt stack. but it has the advantagge that if i hit a but in i i dont feel like i have to fix it | 23:03 |
| sean-k-mooney | s/retryyireally/really really trying/ | 23:04 |
| enricokern | yeah, but it has its merits. i think people like small companies who did run vcenter essentials with 3 hvs go to proxmox | 23:04 |
| enricokern | but if you run a freaking datacenter with 20+ computes you rather pick OS | 23:05 |
| sean-k-mooney | it does have metrics and a somewhat nice ui | 23:05 |
| sean-k-mooney | i mean nothing is where i expect ti to be | 23:05 |
| sean-k-mooney | but it give you nice graphs | 23:05 |
| enricokern | the ui is full shit no question | 23:05 |
| enricokern | but it does what it needs todo | 23:06 |
| mikal | sean-k-mooney: do you think OpenStack is ready to reconsider the decision to not publish prometheus style metrics in services? I see the lack of observability in OpenStack as a really big gap. | 23:06 |
| sean-k-mooney | also copy paste works in tehre web terminal | 23:06 |
| enricokern | hu? we have prometheus since ages rolling out stuff with kolla as example ? | 23:06 |
| mikal | sean-k-mooney: copy and paste works in SPICE too... Just probably not HTML5 transcoded SPICE. | 23:06 |
| enricokern | or do you mean having it direct in the api endpoints? | 23:07 |
| sean-k-mooney | enricokern: we have a policy that service like nova does not provide endpoint to monitor the workloads | 23:07 |
| sean-k-mooney | mikal: i still dont think we should do that | 23:07 |
| enricokern | it doesnt scale anyway | 23:07 |
| mikal | enricokern: you probably mean the openstack-exporter? Its... not good. Polling the REST API for metrics is not a good plan. I mean I'd like say nova-compute to directly expose metrics like every other service on the planet does. | 23:07 |
| sean-k-mooney | mainly because you can deploy exportors pretty simply | 23:07 |
| enricokern | yes this one | 23:07 |
| mikal | sean-k-mooney: but an exporter cannot inspect the internal state of the service? | 23:08 |
| sean-k-mooney | and i dont think we need to write does just deploy prometos and the libvirt/ovs/node exporters on all the nodes | 23:08 |
| sean-k-mooney | mikal: that is true but we dont collect any internel metics | 23:08 |
| enricokern | we run a own monitoring suite scraping from libvirt, but yes the exporter is shit | 23:08 |
| sean-k-mooney | we woudl have to add code to actully create metrics for it to export | 23:08 |
| mikal | A non-nova example because its the one in my head right now. The neutron l3 agent and dhcp agent _know_ how many queued work items they are sitting on. They simply don't expose it anywhere so an exporter cannot ship it. | 23:09 |
| mikal | So as a deployer I have no way to know if my agents are keeping up with their churn rate or not. | 23:09 |
| sean-k-mooney | do they? | 23:09 |
| sean-k-mooney | for what its worth i did start in that direction slightly | 23:09 |
| mikal | They literally have a thread pool consuming a work queue. They track the size of that queue internally, they just don't expose it anywhere. | 23:10 |
| enricokern | i dont think this is something we should maintain as openstack project | 23:10 |
| sean-k-mooney | i.e. addign per process health checks | 23:10 |
| enricokern | it will add complexity to our apis | 23:10 |
| sean-k-mooney | but like many things we never landed it in nova but manilla has them now | 23:10 |
| enricokern | there a dozenz of ways for operators to scrape | 23:10 |
| sean-k-mooney | the idea was to extend that endpoitn eventully wiht some cheap metrics | 23:10 |
| mikal | enricokern: it shouldn't be an API. It should just be a bunch of counters / guages / whatever that is published on a well known port for things to scrape. Its a well paved path. | 23:11 |
| sean-k-mooney | yep | 23:11 |
| sean-k-mooney | https://specs.openstack.org/openstack/nova-specs/specs/yoga/approved/per-process-healthchecks.html | 23:11 |
| mikal | Then deployers can pick whatever scraper they want. | 23:11 |
| enricokern | stil we would need to implement this for all components, and the only way to expose this properly is via the endpoints | 23:12 |
| sean-k-mooney | thats kidn of whot that was inteded tow work | 23:12 |
| mikal | enricokern: its true. Because its hard we should never attempt it. Deployers should just keep scratching their heads when things aren't working right. | 23:12 |
| sean-k-mooney | we were goign to keep som counters in memorbyt and jsut dump the to a json responce if you asked for them | 23:12 |
| enricokern | e.g we would require this as standard for all components | 23:12 |
| enricokern | as an operator myself this was never a painpoint for me as example | 23:13 |
| enricokern | i dont expect this as an openstack part because i can do it myself | 23:13 |
| sean-k-mooney | enricokern: anythign that requries ues to store it in the db and expose it over the rest api is basicly not an option | 23:13 |
| sean-k-mooney | enricokern: but we did talk about have each process have a tiny http endpoint | 23:13 |
| sean-k-mooney | woth otel metrics | 23:13 |
| mikal | enricokern: no seriously, how can you do it yourself if the openstack process doesn't expose it? | 23:13 |
| mikal | enricokern: the only place it exists is a variable in the memory of the neutron agent in my example. | 23:14 |
| enricokern | we deploy own agents on each compute who talk to libvirt | 23:14 |
| enricokern | as example | 23:14 |
| enricokern | and usual monitoring solutions do that too for you already | 23:14 |
| mikal | enricokern: sure, but that's only one example. It doesn't disprove the existence of counter examples. | 23:14 |
| enricokern | the question is, do we need it? is it a problem we need to solve? | 23:15 |
| mikal | I'd be fine with a policy that said "don't publish metrics which overlap with metrics from underlying systems", but there are definitely metrics at the openstack layer than the underlying layers don't know about. | 23:15 |
| enricokern | if its not a problem for operators atm it needs no feature | 23:15 |
| mikal | enricokern: every time my network nodes can't keep up with their churn rate it certainly seems to me like a thing I might want. | 23:15 |
| mikal | enricokern: so as an operator aren't my pain points just as valid as yours? | 23:16 |
| enricokern | its something that could be collected as feedback, kind of whish list. btw. i didnt see this | 23:16 |
| enricokern | yes they are | 23:16 |
| enricokern | i dont say what i do appplies to all others | 23:16 |
| enricokern | its a good point | 23:17 |
| enricokern | as example also logging. we removed our opensearch, fluentd in kolla with loki stack | 23:17 |
| enricokern | opensearch is the same pain shit as rabbitmq | 23:17 |
| sean-k-mooney | mikal: for what its worth not every project is entrily agsint the idea although i suspect most would not want to have to annotat a large protion of our juctions with otel spans ectra | 23:17 |
| sean-k-mooney | but where we have the metrics internally it may be soemthing that neutron may be open to expose | 23:18 |
| sean-k-mooney | mikal: as part of the graceful shutdown work not is gettign a bit more awarness fo the number of inflight operations | 23:18 |
| enricokern | i go to bed guys. but really appreciated the discussion :) | 23:18 |
| sean-k-mooney | and with the eventlet removal we have now more awareness fo the treadpool counts ectra | 23:19 |
| enricokern | n8 | 23:19 |
| sean-k-mooney | but the comptue agetn for exampel still does knot know how may vm boots are queued for it. it just knwo there is a semephor and it has or has not hit it yet | 23:20 |
| mikal | sean-k-mooney: the main way in which my neutron agent examples are bad examples is because I fixed them: https://review.opendev.org/c/openstack/neutron/+/1009917; https://review.opendev.org/c/openstack/neutron/+/1009918. | 23:20 |
| mikal | sean-k-mooney: but I had to sneak up on getting my hands on that metric because we're unwilling to just publish it as an otel metric like everyone else. | 23:21 |
| sean-k-mooney | well we are more unwillign to pus it over rabbit | 23:22 |
| sean-k-mooney | otel on a local http endpoint is more palletable | 23:22 |
| sean-k-mooney | espcially if we never have to store it in the db | 23:22 |
| sean-k-mooney | the perosic state report that your addign them too is exactly what we didnt want to do because it didnt scale well in nova | 23:23 |
| sean-k-mooney | that why after the cpu_monitor we didnt add any more | 23:24 |
| sean-k-mooney | anyway i shoudl proably get some rest. o/ | 23:25 |
| * TheJulia blinks at the chat log | 23:57 | |
| mikal | sean-k-mooney: *shrug* -- doing nothing for a decade is also a pretty terrible solution. | 23:59 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!