09:00:20 <belmoreira> #startmeeting large_scale_sig
09:00:21 <openstack> Meeting started Wed Jan 15 09:00:20 2020 UTC and is due to finish in 60 minutes.  The chair is belmoreira. Information about MeetBot at http://wiki.debian.org/MeetBot.
09:00:22 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
09:00:25 <openstack> The meeting name has been set to 'large_scale_sig'
09:00:34 <amorin> hello!
09:00:36 <belmoreira> Hello everyone. Welcome to the Large Scale SIG meeting.
09:00:47 <oneswig> hello!
09:00:47 <belmoreira> #topic Rollcall
09:00:52 <oneswig> o/
09:01:01 <belmoreira> o/
09:01:03 <etp> o/
09:01:30 <amorin> o/
09:02:23 <belmoreira> great. let's start with our first topic
09:02:30 <belmoreira> #topic "Scaling within one cluster" goal
09:02:38 <belmoreira> #link https://etherpad.openstack.org/p/large-scale-sig-cluster-scaling
09:03:30 <belmoreira> There's no much progress in the etherpad
09:03:59 <oneswig> Afraid not.  Over here we are making slow progress towards improving instrumentation.
09:04:43 <belmoreira> oslo.metrics blueprint is a great idea
09:05:36 <belmoreira> let's wait for masahito feedback in the draft blueprint
09:05:38 <oneswig> I think it ties in well with a fishbowl session from Shanghai, advocating exposing metrics from each service.
09:06:49 <belmoreira> But this only focus in collecting metrics
09:06:58 <belmoreira> I'm sure we can expand this more
09:07:41 <belmoreira> I'm thinking about interaction with other services. Like Placement and how it's deployed
09:07:55 <belmoreira> does it make sense?
09:08:28 <oneswig> belmoreira: can you elaborate?
09:10:05 <belmoreira> in my opinion scaling one nova cell is more than just rabbit and DB. For example, Nova interacts with placement and neutron and in my view we should focus in those scaling limits as well.
09:11:27 <belmoreira> also, being only one cell, how the different components are deployed (shared rabbit or not - for example) will affect the scalability of the deployment
09:11:39 <masahito> Sorry to be late. My IRC client doesn't work well :-(
09:11:48 <oneswig> hi masahito
09:11:57 <belmoreira> welcome masahito
09:12:03 <masahito> hello o/
09:12:26 <belmoreira> oneswig does it make sense?
09:12:41 <oneswig> belmoreira: is there any study on the growth rate of placement as a function of number of hypervisors?  It could be polynomial (ie really bad)...
09:13:01 <oneswig> belmoreira: makes sense to me.
09:13:11 <mdelavergne> (hi, sorry for being late) I think it is definitely relevant to consider the placement and network
09:14:15 <belmoreira> for example: placement is linear, but the number of requests that it can handle will change considering how it's deployed and options
09:14:38 <oneswig> #link discussion from Shanghai on common monitoring via oslo https://etherpad.openstack.org/p/PVG-bring-your-crazy-idea
09:15:21 <oneswig> unfortunately there are no actions or follow-up in that etherpad
09:16:13 <belmoreira> it's not that crazy to me. It would be very cool.
09:17:08 <belmoreira> masahito do you have any update regarding the oslo.metric blueprint
09:17:34 <masahito> Not much update.
09:17:47 <masahito> because of some outage in our cluster :-(
09:18:27 <masahito> But I want to finish to write the first draft by end of January.
09:18:41 <oneswig> was thinking of your talk about LINE yesterday - had a rabbitmq issue on a system here
09:19:23 <belmoreira> Thanks masahito. Let us know.
09:19:49 <belmoreira> oneswig related with scalability?
09:20:11 <oneswig> not this time, only moderate scale
09:20:21 <oneswig> would have been timely otherwise :-)
09:20:38 <belmoreira> anyone as something else related to this topic?
09:21:53 <belmoreira> Let's continue to update the etherpad with our experiences in scaling one nova cell.
09:22:06 <belmoreira> moving to the next topic in the agenda
09:22:16 <belmoreira> #topic Progress on "Documenting large scale operations" goal
09:22:31 <belmoreira> amorin started a thread in the mailing list to document the particular configuration that makes important for large scale deployments
09:22:34 <belmoreira> thanks
09:22:40 <amorin> yes
09:22:40 <belmoreira> #link http://lists.openstack.org/pipermail/openstack-discuss/2020-January/011820.html
09:22:44 <belmoreira> #link https://etherpad.openstack.org/p/large-scale-sig-documentation
09:23:02 <amorin> I got an answer from slawek (neutron PTL)
09:23:19 <amorin> here created a neutron bug: https://bugs.launchpad.net/neutron/+bug/1858419
09:23:19 <openstack> Launchpad bug 1858419 in neutron "Docs needed for tunables at large scale" [Medium,Confirmed] - Assigned to Slawek Kaplonski (slaweq)
09:23:29 <amorin> just like mriedman did for nova
09:23:42 <amorin> so we can collect the tunings over there
09:23:56 <amorin> they also notify us about rabbit and DB params
09:24:04 <amorin> which I believe are important also
09:24:19 <slaweq> amorin: hi, yes and we have some input from liuyulong who is neutron core and has a lot of large scale deployment experience also
09:24:36 <slaweq> so we need to start working on some patches based on those comments
09:24:55 <amorin> yup
09:24:58 <slaweq> and also if You have anything to add there, feel free to write comments or send patches related to this bug :)
09:25:56 <oneswig> sounds great!
09:26:00 <amorin> I am pretty sure I can find some tunings based on OVH experience, I will try to do that
09:26:25 <belmoreira> This is great. Thank you amorin slaweq
09:26:30 <etp> sounds good
09:26:42 <amorin> Also, I proposed to do some documentation change to identify parameters which could affect large scale
09:26:49 <slaweq> amorin: I know You will :)
09:27:01 <amorin> what do you think about this?
09:27:24 <etp> btw has anyone compared different generally available neutron plugins ovs/ovn/...?
09:28:07 <amorin> I havnt, on our side we use custom plugins based on OVS
09:28:30 <amorin> maybe slaweq know if OVN is able to scale?
09:28:32 <oneswig> Only for performance.
09:28:55 <slaweq> amorin: I know there were some tests of ovn on scale done by networking-ovn team
09:28:56 <belmoreira> etp definitely that is interesting. It would be great to know more
09:29:07 <slaweq> I can try to search for them if You want to check them
09:29:15 <etp> i noted that there's spec in Ussuri to move reference from ovs to ovn
09:29:44 <slaweq> etp: no, for Ussuri are only moving networking-ovn code to be in-tree neutron driver
09:30:10 <etp> ah, maybe I misread it :)
09:30:19 <slaweq> but in the future we will probably want to switch our "default" backend to be ovn instead of ovs-agent
09:30:25 <slaweq> but not in Ussuri for sure
09:31:01 <belmoreira> I just added in the etherpad a presentation about CERN network deployment and the configuration options that we use. Will cross check to what is proposed. In our case we use linux bridge
09:31:21 <amorin> ack
09:31:33 <etp> slaweq: we are also looking in to same direction, when it happens remains to be seen
09:32:18 <slaweq> etp: yes
09:32:35 <etp> belmoreira: tnx
09:32:51 <slaweq> If I will have link to any comparison between ovn and other backends I will place it in etherpad https://etherpad.openstack.org/p/large-scale-sig-cluster-scaling
09:32:53 <slaweq> ok?
09:33:14 <belmoreira> please do
09:33:32 <amorin> yes, thanks
09:34:00 <belmoreira> should we have the same approach with other projects? like keystone, placement, glance?
09:34:16 <belmoreira> have a bug open for relevant options for large deployments?
09:35:09 <amorin> I think it could be nice, but it should come from them, we can force them
09:35:49 <amorin> we cant*
09:35:50 <amorin> sorry
09:36:01 <oneswig> I had an action item to poll the Scientific SIG for data points.  I'll add it to today's meeting (at 1100 UTC)
09:37:03 <belmoreira> amorin: ok, let's try to signal issues first in the etherpad
09:37:14 <belmoreira> oneswig thanks
09:37:34 <amorin> yes
09:37:38 <belmoreira> anything else related to this topic?
09:38:48 <belmoreira> moving on
09:38:49 <belmoreira> #topic AOB
09:39:16 <belmoreira> Is there something else that you would like to discuss?
09:40:51 <amorin> not on my side
09:40:53 <oneswig> not for me
09:40:57 <belmoreira> #topic Next Meeting
09:41:03 <masahito> not from my side
09:41:18 <belmoreira> If we follow the 2 weeks rule the next meeting will be on January 29. Is this OK?
09:41:34 <oneswig> +1
09:41:47 <masahito> +1
09:41:59 <amorin> +1
09:41:59 <masahito> ls
09:42:09 <mdelavergne> +1
09:42:18 <belmoreira> #agreed next meeting: January 29, 9utc #openstack-meeting
09:42:28 <belmoreira> Anything else before we close the meeting?
09:42:40 <oneswig> have to go, thanks everyone
09:42:57 <belmoreira> Thanks everyone
09:43:05 <oneswig> ps it's on the agenda now https://wiki.openstack.org/wiki/Scientific_SIG#IRC_Meeting_January_15th_2020
09:43:06 <mdelavergne> thanks
09:43:06 <masahito> thanks, all
09:43:20 <etp> thanks all
09:43:21 <belmoreira> #endmeeting