09:00:20 <belmoreira> #startmeeting large_scale_sig 09:00:21 <openstack> Meeting started Wed Jan 15 09:00:20 2020 UTC and is due to finish in 60 minutes. The chair is belmoreira. Information about MeetBot at http://wiki.debian.org/MeetBot. 09:00:22 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 09:00:25 <openstack> The meeting name has been set to 'large_scale_sig' 09:00:34 <amorin> hello! 09:00:36 <belmoreira> Hello everyone. Welcome to the Large Scale SIG meeting. 09:00:47 <oneswig> hello! 09:00:47 <belmoreira> #topic Rollcall 09:00:52 <oneswig> o/ 09:01:01 <belmoreira> o/ 09:01:03 <etp> o/ 09:01:30 <amorin> o/ 09:02:23 <belmoreira> great. let's start with our first topic 09:02:30 <belmoreira> #topic "Scaling within one cluster" goal 09:02:38 <belmoreira> #link https://etherpad.openstack.org/p/large-scale-sig-cluster-scaling 09:03:30 <belmoreira> There's no much progress in the etherpad 09:03:59 <oneswig> Afraid not. Over here we are making slow progress towards improving instrumentation. 09:04:43 <belmoreira> oslo.metrics blueprint is a great idea 09:05:36 <belmoreira> let's wait for masahito feedback in the draft blueprint 09:05:38 <oneswig> I think it ties in well with a fishbowl session from Shanghai, advocating exposing metrics from each service. 09:06:49 <belmoreira> But this only focus in collecting metrics 09:06:58 <belmoreira> I'm sure we can expand this more 09:07:41 <belmoreira> I'm thinking about interaction with other services. Like Placement and how it's deployed 09:07:55 <belmoreira> does it make sense? 09:08:28 <oneswig> belmoreira: can you elaborate? 09:10:05 <belmoreira> in my opinion scaling one nova cell is more than just rabbit and DB. For example, Nova interacts with placement and neutron and in my view we should focus in those scaling limits as well. 09:11:27 <belmoreira> also, being only one cell, how the different components are deployed (shared rabbit or not - for example) will affect the scalability of the deployment 09:11:39 <masahito> Sorry to be late. My IRC client doesn't work well :-( 09:11:48 <oneswig> hi masahito 09:11:57 <belmoreira> welcome masahito 09:12:03 <masahito> hello o/ 09:12:26 <belmoreira> oneswig does it make sense? 09:12:41 <oneswig> belmoreira: is there any study on the growth rate of placement as a function of number of hypervisors? It could be polynomial (ie really bad)... 09:13:01 <oneswig> belmoreira: makes sense to me. 09:13:11 <mdelavergne> (hi, sorry for being late) I think it is definitely relevant to consider the placement and network 09:14:15 <belmoreira> for example: placement is linear, but the number of requests that it can handle will change considering how it's deployed and options 09:14:38 <oneswig> #link discussion from Shanghai on common monitoring via oslo https://etherpad.openstack.org/p/PVG-bring-your-crazy-idea 09:15:21 <oneswig> unfortunately there are no actions or follow-up in that etherpad 09:16:13 <belmoreira> it's not that crazy to me. It would be very cool. 09:17:08 <belmoreira> masahito do you have any update regarding the oslo.metric blueprint 09:17:34 <masahito> Not much update. 09:17:47 <masahito> because of some outage in our cluster :-( 09:18:27 <masahito> But I want to finish to write the first draft by end of January. 09:18:41 <oneswig> was thinking of your talk about LINE yesterday - had a rabbitmq issue on a system here 09:19:23 <belmoreira> Thanks masahito. Let us know. 09:19:49 <belmoreira> oneswig related with scalability? 09:20:11 <oneswig> not this time, only moderate scale 09:20:21 <oneswig> would have been timely otherwise :-) 09:20:38 <belmoreira> anyone as something else related to this topic? 09:21:53 <belmoreira> Let's continue to update the etherpad with our experiences in scaling one nova cell. 09:22:06 <belmoreira> moving to the next topic in the agenda 09:22:16 <belmoreira> #topic Progress on "Documenting large scale operations" goal 09:22:31 <belmoreira> amorin started a thread in the mailing list to document the particular configuration that makes important for large scale deployments 09:22:34 <belmoreira> thanks 09:22:40 <amorin> yes 09:22:40 <belmoreira> #link http://lists.openstack.org/pipermail/openstack-discuss/2020-January/011820.html 09:22:44 <belmoreira> #link https://etherpad.openstack.org/p/large-scale-sig-documentation 09:23:02 <amorin> I got an answer from slawek (neutron PTL) 09:23:19 <amorin> here created a neutron bug: https://bugs.launchpad.net/neutron/+bug/1858419 09:23:19 <openstack> Launchpad bug 1858419 in neutron "Docs needed for tunables at large scale" [Medium,Confirmed] - Assigned to Slawek Kaplonski (slaweq) 09:23:29 <amorin> just like mriedman did for nova 09:23:42 <amorin> so we can collect the tunings over there 09:23:56 <amorin> they also notify us about rabbit and DB params 09:24:04 <amorin> which I believe are important also 09:24:19 <slaweq> amorin: hi, yes and we have some input from liuyulong who is neutron core and has a lot of large scale deployment experience also 09:24:36 <slaweq> so we need to start working on some patches based on those comments 09:24:55 <amorin> yup 09:24:58 <slaweq> and also if You have anything to add there, feel free to write comments or send patches related to this bug :) 09:25:56 <oneswig> sounds great! 09:26:00 <amorin> I am pretty sure I can find some tunings based on OVH experience, I will try to do that 09:26:25 <belmoreira> This is great. Thank you amorin slaweq 09:26:30 <etp> sounds good 09:26:42 <amorin> Also, I proposed to do some documentation change to identify parameters which could affect large scale 09:26:49 <slaweq> amorin: I know You will :) 09:27:01 <amorin> what do you think about this? 09:27:24 <etp> btw has anyone compared different generally available neutron plugins ovs/ovn/...? 09:28:07 <amorin> I havnt, on our side we use custom plugins based on OVS 09:28:30 <amorin> maybe slaweq know if OVN is able to scale? 09:28:32 <oneswig> Only for performance. 09:28:55 <slaweq> amorin: I know there were some tests of ovn on scale done by networking-ovn team 09:28:56 <belmoreira> etp definitely that is interesting. It would be great to know more 09:29:07 <slaweq> I can try to search for them if You want to check them 09:29:15 <etp> i noted that there's spec in Ussuri to move reference from ovs to ovn 09:29:44 <slaweq> etp: no, for Ussuri are only moving networking-ovn code to be in-tree neutron driver 09:30:10 <etp> ah, maybe I misread it :) 09:30:19 <slaweq> but in the future we will probably want to switch our "default" backend to be ovn instead of ovs-agent 09:30:25 <slaweq> but not in Ussuri for sure 09:31:01 <belmoreira> I just added in the etherpad a presentation about CERN network deployment and the configuration options that we use. Will cross check to what is proposed. In our case we use linux bridge 09:31:21 <amorin> ack 09:31:33 <etp> slaweq: we are also looking in to same direction, when it happens remains to be seen 09:32:18 <slaweq> etp: yes 09:32:35 <etp> belmoreira: tnx 09:32:51 <slaweq> If I will have link to any comparison between ovn and other backends I will place it in etherpad https://etherpad.openstack.org/p/large-scale-sig-cluster-scaling 09:32:53 <slaweq> ok? 09:33:14 <belmoreira> please do 09:33:32 <amorin> yes, thanks 09:34:00 <belmoreira> should we have the same approach with other projects? like keystone, placement, glance? 09:34:16 <belmoreira> have a bug open for relevant options for large deployments? 09:35:09 <amorin> I think it could be nice, but it should come from them, we can force them 09:35:49 <amorin> we cant* 09:35:50 <amorin> sorry 09:36:01 <oneswig> I had an action item to poll the Scientific SIG for data points. I'll add it to today's meeting (at 1100 UTC) 09:37:03 <belmoreira> amorin: ok, let's try to signal issues first in the etherpad 09:37:14 <belmoreira> oneswig thanks 09:37:34 <amorin> yes 09:37:38 <belmoreira> anything else related to this topic? 09:38:48 <belmoreira> moving on 09:38:49 <belmoreira> #topic AOB 09:39:16 <belmoreira> Is there something else that you would like to discuss? 09:40:51 <amorin> not on my side 09:40:53 <oneswig> not for me 09:40:57 <belmoreira> #topic Next Meeting 09:41:03 <masahito> not from my side 09:41:18 <belmoreira> If we follow the 2 weeks rule the next meeting will be on January 29. Is this OK? 09:41:34 <oneswig> +1 09:41:47 <masahito> +1 09:41:59 <amorin> +1 09:41:59 <masahito> ls 09:42:09 <mdelavergne> +1 09:42:18 <belmoreira> #agreed next meeting: January 29, 9utc #openstack-meeting 09:42:28 <belmoreira> Anything else before we close the meeting? 09:42:40 <oneswig> have to go, thanks everyone 09:42:57 <belmoreira> Thanks everyone 09:43:05 <oneswig> ps it's on the agenda now https://wiki.openstack.org/wiki/Scientific_SIG#IRC_Meeting_January_15th_2020 09:43:06 <mdelavergne> thanks 09:43:06 <masahito> thanks, all 09:43:20 <etp> thanks all 09:43:21 <belmoreira> #endmeeting