15:00:29 #startmeeting large_scale_sig 15:00:29 Meeting started Wed Sep 15 15:00:29 2021 UTC and is due to finish in 60 minutes. The chair is ttx. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:00:29 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:00:29 The meeting name has been set to 'large_scale_sig' 15:00:30 hi 15:00:37 #topic Rollcall 15:00:39 o/ 15:00:47 full house today! amorin ? 15:00:47 o/ 15:00:48 hello! 15:01:14 alright, let's get this started 15:01:19 Our agenda for today is at: 15:01:24 #link https://etherpad.openstack.org/p/large-scale-sig-meeting 15:01:38 #topic Next OpenInfraLive session 15:01:47 So we'll have another "Large Scale OpenStack" episode on OpenInfra.Live on Oct 24 15:01:55 We had a TODO for belmoreira to draft the episode description 15:02:01 But that raised questions whether "Architecture choices" was the best topic 15:02:15 belmoreira was considering "ops pain points" instead, but I'm not sure the format would work well 15:02:27 Since we can have only a few selected guests, it will end up being their pain points 15:02:46 with no way to simply validate that those are shared more widely 15:03:00 After meeting a couple of users lately there was a recurrent theme I would like to propose instead 15:03:13 "Neutron scaling best practices" 15:03:22 yeah... sorry for bringing the live episode discussion again 15:03:23 Because I got LOTS of questions on the "proper" way to do Neutron at scale in 2021. 15:03:38 Like... am I the only one using RBAC and encountering scaling issues? 15:03:49 Is OVS or OVN the way of the future? Are provider networks a good idea? 15:04:09 We could invite the Neutron PTL and a few others to get their view on those best practices 15:04:12 if we do so, we should invite someone from neutron team to discuss 15:04:17 Maybe Kolla-Ansible people on their choices of defaults 15:04:21 amorin: you read my mindn 15:04:21 We are about to start an investigation on routed networks and OVN. I think this is a good topic :-) 15:04:22 (you were quicker than me) 15:04:36 + a few new users at that moment in their scaling journey that they are wondering if they are betting on the right horse 15:04:45 What do you think? 15:05:00 great topic in my opinion 15:05:08 Sounds good 15:05:19 happy to include one of you if you think you have a great model story on that front 15:05:22 sounds good 15:06:03 I can ask the kolla-ansible ptl 15:06:07 but the more I look the more I find people who are not 100% sure of their setup 15:06:10 we are using custom model, not upstream, so I dont think we are good speakers for this 15:06:18 amorin: ack 15:06:33 genekuo: do y'all at LINE have a good setup to share? Or struggling too? 15:07:00 belmoreira: IIRC your setup is "keep it simple and avoid issues"? 15:07:01 We also have written our own plug-ins, so probably not the best to be one of the speaker 15:07:19 heh maybe writing up your own thing is the best practice lol 15:07:53 :) 15:08:11 Good speakers are people with an horizontal view of what works and what doesn't 15:08:23 + a few users with candid questions 15:08:28 cern uses linux bridge (mostly to have an easier migration from nova-network and it's simple!). If we go for pain points it would be mostly rabbit. However we are investigating Tungsten fabric. 15:08:50 maybe zigo? 15:08:53 the best would be to check with my colleague that manages Neutron 15:08:56 One of the users I talked to was struggling with Neutron RBAC generating a lot of load 15:08:58 dont remember his company 15:09:08 Infomaniak 15:09:21 I know he was active on some neutron topics recently 15:09:29 but I dont know the scale 15:09:31 Like does anyone else use Neutron RBAC? Is that even a good idea? 15:10:29 imtiazc: we are discussing doing "Neutron scaling best practices" as the next OILive episode 15:10:49 belmoreira: ++ 15:11:15 Thanks! @ttx 15:11:22 at scale we need to be gentle with Neutron. Any new addition (RBAC) can disturb the force 15:11:34 imtiazc: I talked to a few users and they all cited Neutron architectural choices as being one of the most nebulous choice they need to make 15:11:47 belmoreira: you sound like Yoda 15:12:17 But yeah, Neutron is clearly the new RabbitMQ in terms of ops top-of-mind 15:12:41 you can get it wrong and be stuck with it 15:13:26 There are lots of provider options to choose from. I believe operators also have their different use cases. 15:13:36 but even answering " what would you recommend as a driver those days" is a very good question to candidly answer 15:13:37 ttx: can you confirm the date? 15:13:49 Oct 24 14utc 15:13:57 week after PTG 15:14:23 Could it be Oct 28th? 24th is a Sunday 15:14:29 haha. hmm 15:14:44 * ttx triplechecks 15:15:09 probably 28 yes, checking 15:16:15 We can go over how we made our decision lately after considering all the viable options. 15:16:21 October 14! 15:17:10 https://meetings.opendev.org/meetings/large_scale_sig/2021/large_scale_sig.2021-09-01-15.01.log.html#l-30 15:17:14 The logs do not lie 15:17:19 Sorry for the confusion 15:17:27 So... week BEFORE Ptg 15:17:27 thanks ttx! 15:18:19 oneswig_: are you gong to approach the Kolla-Ansible PTL about this? If he can bring the StackHPC experience trying multiple solutions there, taht would be awesome 15:18:23 going* 15:18:38 I can do that, yes. 15:19:02 Coincidentally he's done quite a lot with OVN upstream as well. 15:19:15 #info Oct 14 tentative episode topic: "Neutron scaling best practices" 15:19:31 #action oneswig_ to contact Kolla-Ansible PTL about participating 15:19:40 #action ttx to contact Neutron PTL about participating 15:20:03 #action belmoreira to connect with internal CERN neutron people to see if they have a good angle to bring 15:20:38 could be interesting to have Rd Hat or Canonical people on their own choices of drivers / configs that work well in their experience 15:21:11 We worked closely with RH last year. Their recommendation is OVN. 15:21:34 Cool, great to know 15:22:03 If we manage to get good answers we could document them in our large Scale Journey wiki 15:22:25 but worst case scenario it's a great topic to openly discuss 15:22:52 any other idea of who could bring interesting insights to the table? 15:23:26 I don;t really want to turn it into a drivers fight with each brand fighting. I prefer to have a bunch of curators discuss what works well for them 15:23:32 It will be great if we can invite someone using Calico 15:23:58 IMO, it's a very different approach compare to other common plug-ins 15:24:11 AFAIK it has limitations 15:24:12 Bloomberg has been using Calico. We just started building with Calico. Salesforce has been using Calico in production for years. 15:24:15 hmm yes or at least make sure that angle is covered 15:24:23 +1 15:24:40 OK, other thoughts? 15:24:42 Yeah, it have limitations, but it works very well in some cases 15:24:49 ack 15:25:15 if you accept limitations it works well" is good insight 15:26:11 alright if no other suggestions/actions, we'll move on to next topic 15:26:29 #topic Large Scale Journey updates 15:26:29 Every choice of driver has trade-offs :) Software architecture is all about making the decision that best suits one's needs. 15:26:40 imtiazc: no we want absolute answers! 15:26:49 lol 15:27:06 OVN good! OVS bad! (or is it the other way around??) 15:27:12 multi-driver war... 15:27:21 #link https://wiki.openstack.org/wiki/Large_Scale_SIG 15:27:27 Does anyone has updates/blockers on actions related to documenting the Journey? 15:27:48 I have some updates on the docs 15:27:54 thanks to reedip 15:28:41 Still have to adjust the layout though 15:29:12 thanks for the reviews @genekuo... 15:30:05 any specific part updated? 15:30:58 maybe after the next live ? :P 15:31:01 Yeah, mostly on how to track error rates and saturation/performance issues 15:31:12 nice! 15:31:19 alright, anyone else? 15:31:45 if not... 15:31:49 #topic Next meeting 15:31:55 #info next IRC meeting on Sept 29 15:32:19 My goal is to have most of the guests for the episode confirmed by then 15:32:29 as well as a clear episode description 15:33:04 so tat we can discuss last details 15:33:06 #topic Open discussion 15:33:19 Alright, does anyone have a question to discuss with the group? 15:33:35 Nothing from me 15:34:11 FWIW I've been inviting to this group those users that are on the verge of scaling up dramatically 15:34:46 Because they have lots of questions, which could be a good way of rapidly improving our doc 15:35:06 we'll see if they take the bait 15:35:06 It's always good to have more people joining the discussion :) 15:35:14 agree 15:35:36 yeah, and they usually have experience of newer releases so this can be a two-way street 15:36:23 Hopefully that open discussion section of the meeting will be more lively as a result! 15:36:38 But for today... if nobody has questions (yet)... 15:36:39 +1 15:37:12 Let's close early again! Have a great day/week, talk to you all soon 15:37:17 #endmeeting