15:00:29 <ttx> #startmeeting large_scale_sig
15:00:29 <opendevmeet> Meeting started Wed Sep 15 15:00:29 2021 UTC and is due to finish in 60 minutes.  The chair is ttx. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:29 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:29 <opendevmeet> The meeting name has been set to 'large_scale_sig'
15:00:30 <oneswig_> hi
15:00:37 <ttx> #topic Rollcall
15:00:39 <belmoreira> o/
15:00:47 <ttx> full house today! amorin ?
15:00:47 <oneswig_> o/
15:00:48 <amorin> hello!
15:01:14 <ttx> alright, let's get this started
15:01:19 <ttx> Our agenda for today is at:
15:01:24 <ttx> #link https://etherpad.openstack.org/p/large-scale-sig-meeting
15:01:38 <ttx> #topic Next OpenInfraLive session
15:01:47 <ttx> So we'll have another "Large Scale OpenStack" episode on OpenInfra.Live on Oct 24
15:01:55 <ttx> We had a TODO for belmoreira to draft the episode description
15:02:01 <ttx> But that raised questions whether "Architecture choices" was the best topic
15:02:15 <ttx> belmoreira was considering "ops pain points" instead, but I'm not sure the format would work well
15:02:27 <ttx> Since we can have only a few selected guests, it will end up being their pain points
15:02:46 <ttx> with no way to simply validate that those are shared more widely
15:03:00 <ttx> After meeting a couple of users lately there was a recurrent theme I would like to propose instead
15:03:13 <ttx> "Neutron scaling best practices"
15:03:22 <belmoreira> yeah... sorry for bringing the live episode discussion again
15:03:23 <ttx> Because I got LOTS of questions on the "proper" way to do Neutron at scale in 2021.
15:03:38 <ttx> Like... am I the only one using RBAC and encountering scaling issues?
15:03:49 <ttx> Is OVS or OVN the way of the future? Are provider networks a good idea?
15:04:09 <ttx> We could invite the Neutron PTL and a few others to get their view on those best practices
15:04:12 <amorin> if we do so, we should invite someone from neutron team to discuss
15:04:17 <ttx> Maybe Kolla-Ansible people on their choices of defaults
15:04:21 <ttx> amorin: you read my mindn
15:04:21 <oneswig_> We are about to start an investigation on routed networks and OVN.  I think this is a good topic :-)
15:04:22 <amorin> (you were quicker than me)
15:04:36 <ttx> + a few new users at that moment in their scaling journey that they are wondering if they are betting on the right horse
15:04:45 <ttx> What do you think?
15:05:00 <amorin> great topic in my opinion
15:05:08 <genekuo> Sounds good
15:05:19 <ttx> happy to include one of you if you think you have a great model story on that front
15:05:22 <belmoreira> sounds good
15:06:03 <oneswig_> I can ask the kolla-ansible ptl
15:06:07 <ttx> but the more I look the more I find people who are not 100% sure of their setup
15:06:10 <amorin> we are using custom model, not upstream, so I dont think we are good speakers for this
15:06:18 <ttx> amorin: ack
15:06:33 <ttx> genekuo: do y'all at LINE have a good setup to share? Or struggling too?
15:07:00 <ttx> belmoreira: IIRC your setup is "keep it simple and avoid issues"?
15:07:01 <genekuo> We also have written our own plug-ins, so probably not the best to be one of the speaker
15:07:19 <ttx> heh maybe writing up your own thing is the best practice lol
15:07:53 <amorin> :)
15:08:11 <ttx> Good speakers are people with an horizontal view of what works and what doesn't
15:08:23 <ttx> + a few users with candid questions
15:08:28 <belmoreira> cern uses linux bridge (mostly to have an easier migration from nova-network and it's simple!). If we go for pain points it would be mostly rabbit. However we are investigating Tungsten fabric.
15:08:50 <amorin> maybe zigo?
15:08:53 <belmoreira> the best would be to check with my colleague that manages Neutron
15:08:56 <ttx> One of the users I talked to was struggling with Neutron RBAC generating a lot of load
15:08:58 <amorin> dont remember his company
15:09:08 <ttx> Infomaniak
15:09:21 <amorin> I know he was active on some neutron topics recently
15:09:29 <amorin> but I dont know the scale
15:09:31 <ttx> Like does anyone else use Neutron RBAC? Is that even a good idea?
15:10:29 <ttx> imtiazc: we are discussing doing "Neutron scaling best practices" as the next OILive episode
15:10:49 <ttx> belmoreira: ++
15:11:15 <imtiazc> Thanks! @ttx
15:11:22 <belmoreira> at scale we need to be gentle with Neutron. Any new addition (RBAC) can disturb the force
15:11:34 <ttx> imtiazc: I talked to a few users and they all cited Neutron architectural choices as being one of the most nebulous choice they need to make
15:11:47 <ttx> belmoreira: you sound like Yoda
15:12:17 <ttx> But yeah, Neutron is clearly the new RabbitMQ in terms of ops top-of-mind
15:12:41 <ttx> you can get it wrong and be stuck with it
15:13:26 <imtiazc> There are lots of provider options to choose from. I believe operators also have their different use cases.
15:13:36 <ttx> but even answering " what would you recommend as a driver those days" is a very good question to candidly answer
15:13:37 <oneswig_> ttx: can you confirm the date?
15:13:49 <ttx> Oct 24 14utc
15:13:57 <ttx> week after PTG
15:14:23 <oneswig_> Could it be Oct 28th?  24th is a Sunday
15:14:29 <ttx> haha. hmm
15:14:44 * ttx triplechecks
15:15:09 <ttx> probably 28 yes, checking
15:16:15 <imtiazc> We can go over how we made our decision lately after considering all the viable options.
15:16:21 <ttx> October 14!
15:17:10 <ttx> https://meetings.opendev.org/meetings/large_scale_sig/2021/large_scale_sig.2021-09-01-15.01.log.html#l-30
15:17:14 <ttx> The logs do not lie
15:17:19 <ttx> Sorry for the confusion
15:17:27 <ttx> So... week BEFORE Ptg
15:17:27 <oneswig_> thanks ttx!
15:18:19 <ttx> oneswig_: are you gong to approach the Kolla-Ansible PTL about this? If he can bring the StackHPC experience trying multiple solutions there, taht would be awesome
15:18:23 <ttx> going*
15:18:38 <oneswig_> I can do that, yes.
15:19:02 <oneswig_> Coincidentally he's done quite a lot with OVN upstream as well.
15:19:15 <ttx> #info Oct 14 tentative episode topic: "Neutron scaling best practices"
15:19:31 <ttx> #action oneswig_ to contact Kolla-Ansible PTL about participating
15:19:40 <ttx> #action ttx to contact Neutron PTL about participating
15:20:03 <ttx> #action belmoreira to connect with internal CERN neutron people to see if they have a good angle to bring
15:20:38 <ttx> could be interesting to have Rd Hat or Canonical people on their own choices of drivers / configs that work well in their experience
15:21:11 <imtiazc> We worked closely with RH last year. Their recommendation is OVN.
15:21:34 <ttx> Cool, great to know
15:22:03 <ttx> If we manage to get good answers we could document them in our large Scale Journey wiki
15:22:25 <ttx> but worst case scenario it's a great topic to openly discuss
15:22:52 <ttx> any other idea of who could bring interesting insights to the table?
15:23:26 <ttx> I don;t really want to turn it into a drivers fight with each brand fighting. I prefer to have a bunch of curators discuss what works well for them
15:23:32 <genekuo> It will be great if we can invite someone using Calico
15:23:58 <genekuo> IMO, it's a very different approach compare to other common plug-ins
15:24:11 <amorin> AFAIK it has limitations
15:24:12 <imtiazc> Bloomberg has been using Calico. We just started building with Calico. Salesforce has been using Calico in production for years.
15:24:15 <ttx> hmm yes or at least make sure that angle is covered
15:24:23 <oneswig_> +1
15:24:40 <ttx> OK, other thoughts?
15:24:42 <genekuo> Yeah, it have limitations, but it works very well in some cases
15:24:49 <amorin> ack
15:25:15 <ttx> if you accept limitations it works well" is good insight
15:26:11 <ttx> alright if no other suggestions/actions, we'll move on to next topic
15:26:29 <ttx> #topic Large Scale Journey updates
15:26:29 <imtiazc> Every choice of driver has trade-offs :) Software architecture is all about making the decision that best suits one's needs.
15:26:40 <ttx> imtiazc: no we want absolute answers!
15:26:49 <amorin> lol
15:27:06 <ttx> OVN good! OVS bad! (or is it the other way around??)
15:27:12 <reedip> multi-driver war...
15:27:21 <ttx> #link https://wiki.openstack.org/wiki/Large_Scale_SIG
15:27:27 <ttx> Does anyone has updates/blockers on actions related to documenting the Journey?
15:27:48 <genekuo> I have some updates on the docs
15:27:54 <genekuo> thanks to reedip
15:28:41 <genekuo> Still have to adjust the layout though
15:29:12 <reedip> thanks for the reviews @genekuo...
15:30:05 <ttx> any specific part updated?
15:30:58 <mdelavergne> maybe after the next live ? :P
15:31:01 <genekuo> Yeah, mostly on how to track error rates and saturation/performance issues
15:31:12 <ttx> nice!
15:31:19 <ttx> alright, anyone else?
15:31:45 <ttx> if not...
15:31:49 <ttx> #topic Next meeting
15:31:55 <ttx> #info next IRC meeting on Sept 29
15:32:19 <ttx> My goal is to have most of the guests for the episode confirmed by then
15:32:29 <ttx> as well as a clear episode description
15:33:04 <ttx> so tat we can discuss last details
15:33:06 <ttx> #topic Open discussion
15:33:19 <ttx> Alright, does anyone have a question to discuss with the group?
15:33:35 <genekuo> Nothing from me
15:34:11 <ttx> FWIW I've been inviting to this group those users that are on the verge of scaling up dramatically
15:34:46 <ttx> Because they have lots of questions, which could be a good way of rapidly improving our doc
15:35:06 <ttx> we'll see if they take the bait
15:35:06 <genekuo> It's always good to have more people joining the discussion :)
15:35:14 <amorin> agree
15:35:36 <ttx> yeah, and they usually have experience of newer releases so this can be a two-way street
15:36:23 <ttx> Hopefully that open discussion section of the meeting will be more lively as a result!
15:36:38 <ttx> But for today... if nobody has questions (yet)...
15:36:39 <belmoreira> +1
15:37:12 <ttx> Let's close early again! Have a great day/week, talk to you all soon
15:37:17 <ttx> #endmeeting