14:01:49 <haleyb> #startmeeting neutron_drivers
14:01:49 <opendevmeet> Meeting started Fri Apr 26 14:01:49 2024 UTC and is due to finish in 60 minutes.  The chair is haleyb. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:01:49 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:01:49 <opendevmeet> The meeting name has been set to 'neutron_drivers'
14:02:06 <haleyb> ping list: ykarel, mlavalle, mtomaska, slaweq, obondarev, tobias-urdin, lajoskatona, amotoki, haleyb, ralonsoh, JayF
14:02:13 <TheJulia> o/
14:02:18 <obondarev> o/
14:02:20 <JayF> o/
14:02:25 <ykarel> o/
14:02:28 <slaweq> o/
14:02:39 <jcosmao> o/
14:02:40 <lajoskatona> o/
14:02:56 <mlavalle> \o
14:03:43 <haleyb> ok, looks like we have quorom, two items in the agenda today
14:04:12 <haleyb> first is regarding ironic
14:04:16 <haleyb> #link https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/thread/5SX35AQZ65G6HWQ7WARLJTJK3ASAOWNU/
14:04:50 <haleyb> does someone want to give the quick overview?
14:05:18 <lajoskatona> I suppose the best would be JayF or TheJulia :-)
14:05:38 <TheJulia> The tl;dr is Ironic is looking for a way for standalone operators to toggle network device config remotely in a secure way which is compatible with separation of duties requirements which we can then extend to better support DPU/IPU network configuration at a later point in time
14:06:34 <TheJulia> We have a generalized idea which involves leveraging the basic design of the ML2 interface for part of the mechanism, but disjointed from Neutron entirely. With that, we see a path to also be compatible and address some of the more classical integrated challenges as well, but one step at a time.
14:07:20 * mlavalle met with TheJulia on line last week so very well aware of the proposal
14:08:49 <TheJulia> We're definitely not trying to replace Neutron with this, just a focused effort on a use path which needs attention and capabilities since approximately half of Ironic's use base doesn't have other openstack services
14:09:42 <TheJulia> Questions? Concerns?
14:10:17 <haleyb> TheJulia: so in some cases there isn't really an openstack cloud per-se, just running services like neutron?
14:10:33 <lajoskatona> if I understand well big chunk of these usecases are already from k8s
14:10:57 <obondarev> so ideally you want smth like restricted minimalistic Neutron/ML2 version that you could use and that could also work without rest of OpenStack?
14:10:58 <TheJulia> haleyb: ironic running as either a hidden detail or a user facing service in standalone mode.
14:11:03 <mlavalle> I wasn'r aware of that statistic. very interesting and higher than I would have guessed
14:11:13 <JayF> metal3.io, for instance, is powered by Ironic
14:11:15 <TheJulia> haleyb: so no actual "cloud"
14:11:20 <JayF> many of their users aren't aware we are involved there, either :D
14:12:17 <TheJulia> obondarev: Well, I think we'll build it but sort of accessible as a single entity service which holds no state, so it can be owned by a "network" team
14:13:10 <JayF> One thing that's useful context, I suspect many of you are aware, Ironic already holds the repos for generic physical switch config (networking-generic-switch) and the MVP is more or less connecting Ironic to this service for basic servcing
14:13:20 <TheJulia> mlavalle: realistically, we see our standalone usage growing because people need the tooling to manage the lower levels of infrastucture.
14:13:54 <mlavalle> yeap, it makes sense
14:14:39 <haleyb> so looking at your spec Proposed Change section
14:14:43 <haleyb> #link https://review.opendev.org/c/openstack/ironic-specs/+/916126/4/specs/approved/mercury.rst#91
14:14:50 <lajoskatona> this service should be part of ironic? or a plugin like thing?
14:15:04 <haleyb> what things are going to be required of neutron?
14:15:12 <TheJulia> lajoskatona: Many of them are from k8s, but at the same time the separation of duties/configuration issues we're sort of circling around seem like they could also help the overall model of use for Neutron as well.
14:15:26 <TheJulia> haleyb: I think it might be as short as "please don't break the ml2 interface" :)
14:15:44 <JayF> haleyb: I think a nonzero part of this consultation is also for "if we're doing network stuff, we want the blessing of the networking cloud folks" as well
14:15:55 <TheJulia> haleyb: and no hate mail as well, please :)
14:15:56 <obondarev> so maybe moving some parts of Neutron to a package/lib could be useful for you, right?
14:16:09 <JayF> neutron-lib is already used by NGS
14:16:39 <lajoskatona> +1 for neutron-lib
14:16:49 <obondarev> neutron-lib is good but getting too large maybe..
14:16:54 <lajoskatona> that can be a good boilerplate lib
14:16:54 <TheJulia> lajoskatona: I've kind of proposed this be modeled as an rpc style service, so a node may be configured for it, or realistically neutron directly. It is up to the infrastructure operator
14:17:17 <lajoskatona> TheJulia: thanks
14:17:23 <JayF> The exciting thing is also this provides some interesting bootstrap capabilitites
14:17:33 <TheJulia> obondarev: I'm not sure, it looks like the ML2 interface is cleanly enough defined that I'm not sure we'll need other things from Neutron to directly pull this off
14:17:45 <JayF> since some Ironic nodes could use mercury to get network control initially to bootstrap a neutron/integrated ironic installation
14:18:41 <mlavalle> JayF: like I said to TheJulia during our conversation, if it makes sense for use cases, you have my humble blessing and no hate email from me
14:18:56 <TheJulia> Yes, DPU/IPU devices are definitely "weird" to be mentally modeled since they are basically computer systems inside of computer systems which networking can pass through.
14:19:03 <obondarev> separating ML2 interface + models might make sense, I'm thinking if anyone else could benefit for it..
14:19:30 <haleyb> on obondarev's point, we routinely move things to neutron-lib, are there other logical things we can move to facilitate things? just curious
14:19:34 <TheJulia> obondarev: I think just the ml2 interface unless there are objects getting passed in that I'm not aware of
14:19:52 <TheJulia> obondarev: this is sort of where my neutron context starts to disappear
14:20:22 <obondarev> TheJulia: those are technical details :)
14:20:56 <TheJulia> obondarev: indeed :)
14:22:04 <lajoskatona> From my side the spec can be used to add such high level details if this can be done in n-lib and that can be used ironic
14:22:22 <TheJulia> I *suspect* splitting ml2 stuffs out into a separate library would be super clean for ML2 drivers as well and reduce the neutron-lib dependency
14:22:24 <lajoskatona> by ironic ----^
14:22:51 <mlavalle> beyond addressing the political side, are you recruting help today?
14:23:29 <TheJulia> mlavalle: I don't know, if folks are interested in helping move along the idea that would be awesome, but we're still very early in ironic's discussions
14:23:32 <lajoskatona> if we have to move code to n-lib that is help :-)
14:23:52 <lajoskatona> or to some new ml2-lib
14:23:53 <obondarev> not sure is ML2 interface is changed often, probably not. With that I believe making it a standalone lib/package that could be used by Neutron (of course) and something else is a sane idea
14:23:55 <mlavalle> TheJulia: ok, I
14:24:01 <mlavalle> I
14:24:23 <haleyb> TheJulia: creating and maintaining a new ML2 library would fall on this team of course
14:24:58 <mlavalle> i'll take a look and see if i can help. I'll start by reviewing the spec
14:25:09 <TheJulia> I suspect that would be awesome, it might make sense to double-check networking-generic-switch's use of the neutron-lib to make sure nothing else gets used. I'm not sure about networking-baremetal though
14:25:20 <TheJulia> networking-baremetal has a different reason for existing
14:25:29 <TheJulia> mlavalle: much appreciated
14:25:39 <TheJulia> haleyb: that would be awesome
14:26:42 <lajoskatona> do we need to track the study of this cut to new lib in Neutron RFE for example?
14:26:52 <haleyb> well, i didn't say "yes" :) i see obondarev mentioned it as well but don't know how much work that would be
14:27:06 <obondarev> haleyb: fair point
14:27:13 <TheJulia> haleyb: well, yeah. I think a starting point is to wrap a brain or two around it :)
14:27:27 <haleyb> lajoskatona: if some end goal is to have another library, yes that would be its own RFE, etc
14:28:38 <lajoskatona> perhaps we can create an RFE for this and track it and write the conclusion first to that as an input for ironic spec and for Neutron team also to see the amount of work predicted
14:28:51 <mlavalle> +1
14:28:53 <TheJulia> lajoskatona: that sounds like an excellent path
14:30:28 <haleyb> right. i'm guessing any short-term work in neutron is fixing of bugs as you find them, in those grey areas, as happens already today
14:31:00 <TheJulia> Yeah, what lajoskatona suggests is sort of an ideal path at the moment
14:32:25 <TheJulia> Anything else?
14:32:36 <haleyb> ack. can someone be the point person for neutron for this work? i.e. read specs, attend discussions?
14:32:39 <slaweq> do I understand correctly that the plan is to create new, kind of "neutron-lib" library under neutron team and move parts of the code from neutron there?
14:33:17 <lajoskatona> slaweq: not a plan at themoment, just a possibility
14:33:29 <slaweq> ok
14:33:30 <obondarev> slaweq: I'd say not at this point
14:33:59 <slaweq> but that is the possiblitity discussed here so far, right?
14:34:05 <obondarev> yes
14:34:09 <slaweq> ok
14:34:14 <lajoskatona> the best would be to use n-lib for this also, as that is used by many projects already
14:34:14 <slaweq> thx for clarification
14:34:19 <mlavalle> haleyb: I can do it, bandwidth permitting of course
14:34:36 <slaweq> lajoskatona ++
14:35:41 <haleyb> mlavalle: great, thanks, and feel free to pull me in when necessary
14:36:26 <haleyb> ok, so the outcome is we won't break ML2, and won't send hate mail? :)
14:36:41 <mlavalle> i'll start by filing the rfe lajoskatona suggested
14:36:46 <lajoskatona> I also try to allocate some time for this to have an  idea at least
14:36:47 <TheJulia> :)
14:36:50 <obondarev> haleyb: +1!
14:36:56 <lajoskatona> mlavalle: thanks
14:37:17 <lajoskatona> haleyb: :D
14:37:20 <haleyb> let's just keep the discussion open here (that's the real goal)
14:38:27 <TheJulia> ++
14:39:28 <haleyb> alright, thanks everyone for the great discussion, and good luck!
14:39:35 <TheJulia> Thanks!
14:39:39 <mlavalle> TheJulia JayF thanks for the consideration of bringing us to speed and requesting our opinion. we certainly don't need civil wars in  this community
14:40:25 <TheJulia> Yeah, we definitely want a model of mutually beneficial should this move forward.
14:41:17 <lajoskatona> Thanks for the time and preparation of this topic
14:42:09 <haleyb> we did have one other item, non-ironic related
14:42:18 <haleyb> #link https://bugs.launchpad.net/neutron/+bug/2007674
14:42:51 <haleyb> jcosmao: i don't think we ever triaged this when it initially was filed
14:43:39 <haleyb> hopefully julien is still here
14:43:45 <lajoskatona> yes as I remember it was opened as general bug
14:44:00 <jcosmao> actually, i started working on this topic before finding this rfe
14:44:05 <haleyb> #link https://etherpad.opendev.org/p/neutron-rfe-2007674
14:44:43 <jcosmao> yes, i added some details about changes i want  to propose in this etherpad
14:45:16 <jcosmao> for the full context, same topic than what was discussed at large scale meeting by amorin: https://meetings.opendev.org/meetings/large_scale_sig/2024/large_scale_sig.2024-04-24-09.01.log.html#l-58
14:46:49 <slaweq> so IIUC this really needs first changes in oslo.messaging,right?
14:47:09 <obondarev> jcosmao: did you have a chance to try your changes at scale? It sound like right thing to do, just not sure what other limitations of rabbit it could face..
14:47:50 <jcosmao> yes exactly, most changes are in oslo.messaging
14:48:00 <slaweq> and another question: should we maybe ask for opinion oslo cores?
14:48:07 <mlavalle> obondarev: good point
14:48:14 <jcosmao> obondarev  yes, it's in production since few weeks
14:48:35 <obondarev> jcosmao: oh that's very convincing point :)
14:49:06 <mlavalle> jcosmao: and are you seeig measurable improvements?
14:49:33 <jcosmao> we reduced a lot nb of queues + connection. and yes rolling update of neutron agent are smooth now
14:50:19 <mlavalle> wow, so what is not to like? let us see the code, please
14:50:30 <lajoskatona> +1
14:50:49 <jcosmao> we also worked previously to move on rabbitmq streams for 'fanout' queues.  this one was also a great improvment (queues are shared instead of 1 queue per agent)
14:50:57 <obondarev> to be on the safe side I think we need to know the "capacity" of a connection: how many load it could handle? When is the right time to open new connection
14:52:07 <obondarev> *rather than continue loading existing one
14:52:38 <jcosmao> i don't think tcp connection is a bottleneck, even with 1 connexion it should be ok
14:53:40 <haleyb> jcosmao: have you approached the oslo team or have patches proposed there?
14:53:55 <obondarev> if also rabbit/oslo folks say this as well I think we are safe :)
14:54:37 <lajoskatona> +1 perhaps only neutron doc and cfg should be changed?
14:55:36 <jcosmao> we already submitted few patch to move on quorum queues / stream queues, it's not yet default config.  But we did not yet submitted for this specific rfe
14:56:10 <lajoskatona> ack
14:56:11 <jcosmao> so, i can work on that to submit change on solo, and maybe link to this rfe ?
14:56:21 <obondarev> so +1 from me with confirmation from oslo or rabbit cores (nova is a good proof but still it uses queue not that much as neutron)
14:57:39 <haleyb> jcosmao: do you just see the neutron changes being small to facilitate new arguments/flags to the messaging code?
14:58:03 <haleyb> it might be good to have a spec on what this entails
15:00:31 <mlavalle> and maybe in that spec share some of the measured benefirs already apparent in jcosmao's deployment
15:00:33 <jcosmao> there is not much changes in neutron, most of them, is ensure to pass proper parameter to oslo.messaging Target.
15:01:20 <jcosmao> ok, is there a process to follow to create a spec ?
15:01:21 <slaweq> for me it sounds reasonable from neutron PoV as long as oslo.messaging cores will accept those changes and will not have anything against
15:01:53 <slaweq> and I don't think that RFE is really needed in this case, at least not for neutron
15:03:27 <obondarev> agree with slaweq
15:03:50 <haleyb> jcosmao: we have a neutron-specs repo, but as slaweq mentioned, if the changes are very small to neutron we probably don't need one, they can be described in the bug and/or change
15:04:29 <jcosmao> slaweq  agree also
15:05:06 <lajoskatona> +1, don't make to heavy in Neutron if most of the changes are in oslo
15:05:08 <haleyb> we should vote as we're over time
15:05:17 <jcosmao> so if it's ok for you, i can work on submitting patch to oslo team, with neutron context
15:05:37 <lajoskatona> jcosmao: thanks
15:05:42 <jcosmao> and if ok, i'll continue on neutron side
15:05:44 <mlavalle> +1 from me
15:05:45 <lajoskatona> +1 from me
15:05:53 <haleyb> jcosmao: yes, that works
15:06:05 <haleyb> +1 from me
15:06:54 <obondarev> +1
15:07:13 <mlavalle> jcosmao: we like this minimal effort improvements, thanks!
15:07:34 <haleyb> yes, less rabbit queues is good
15:07:43 <jcosmao> :) thx, i'll keep you updated on that
15:07:54 <obondarev> more rabbit holes! :D
15:07:54 <haleyb> slaweq: were you a +1 ?
15:08:01 <haleyb> rabbit stew?
15:08:37 <obondarev> haleyb: eventually yes
15:09:31 <haleyb> jcosmao: i will approve it, ping people here when you get to the neutron work, thanks for taking this on
15:10:38 <jcosmao> haleyb  ok great
15:10:49 <haleyb> ok, thanks for the time everyone, enjoy your weekend
15:10:53 <haleyb> #endmeeting