08:00:19 #startmeeting large_scale_sig 08:00:19 Meeting started Wed Sep 23 08:00:19 2020 UTC and is due to finish in 60 minutes. The chair is ttx. Information about MeetBot at http://wiki.debian.org/MeetBot. 08:00:20 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 08:00:22 The meeting name has been set to 'large_scale_sig' 08:00:23 #topic Rollcall 08:00:29 Who is here for the Large Scale SIG meeting ? 08:00:34 Hi 08:00:56 mdelavergne: hi! 08:01:29 pinging amorin 08:01:46 hello! 08:01:58 thank for the ping, completely forgot :( 08:02:20 belmoreira seems offline 08:03:02 masahito might be around 08:04:41 While we wait and are between French people -- I had a nice chat with the ops at Societe generale yesterday. They are in the first stages of scaling (think 100->1000 and are a bit in the unknown, which reinforces the need for documenting what to expect when scaling up/out 08:05:16 I told then as they transition to 100->1000 they will likely hit the need to scale out at some point 08:05:22 them* 08:05:40 Nice! 08:05:45 But it's unclear what to watch for, and what will fail first 08:06:00 They might have some surprises :( 08:06:17 (they are already sharding RabbitMQ a bit so I expect that won't be the first thing) 08:06:27 masahito: hi! 08:06:50 alright, let's get started 08:06:52 Our agenda for today is at: 08:06:56 #link https://etherpad.openstack.org/p/large-scale-sig-meeting 08:07:10 #topic Meaningful monitoring 08:07:19 hi! sorry at late. We're hitting a rabbit mq down right now :-( 08:07:30 Two weeks ago we had our second EU-US meeting, and we actually had new participants joining! 08:07:58 masahito: we can catch up later, you should focus on the urgent things :) 08:08:13 One of the things the newcomers were interested in pushing was the idea of "meaningful monitoring" 08:08:24 which I think could be a third workstream for the group 08:08:32 as it's a pretty common challenge across large scale deployments 08:08:56 and also matches the "how do you do metrics/billing in your deployment" question 08:09:08 Thoughts on that? 08:10:46 Is that something interesting to work on, or is it basically a solved problem? 08:11:40 It's definitely a good point, but where is it possible to begin on tackling this problem? 08:12:25 I'd start by defining the need, and discuss whether the current tooling available is covering it or not 08:12:43 sorry for joining late 08:12:47 hi! 08:12:50 Also collecting stories 08:12:54 belmoreira: hi! 08:13:08 We were discussing "meaningful monitoring" as a potential new workstream for the group 08:13:30 The new US participants (James Penick and Erik from Blizzard) were interested in that 08:14:05 belmoreira: does that sound like a valuable thing for the group to explore? 08:14:18 +1 from me. With all the work with oslo metrics looks a good next step 08:14:44 OK I'll start documenting that and come up with a plan we can bootstrap from the US+EU side of the meeting 08:15:08 there was any discussion regarding notifications? 08:15:14 #action ttx to draft a plan to tackle "meaningful monitoring" as a new SIG workstream 08:15:27 belmoreira: no, but I'd say that's part of "meaningful" 08:15:35 like... "actionable monitoring" 08:16:15 there are projects and produce a lot of notifications but no easy/interesting way to get "meaningful" from them 08:16:28 s/and/that 08:16:58 ack 08:17:40 I think that's what they meant. There is data being produced, but it's difficult to derive operational status from it 08:18:26 moving on to our other workstreams... 08:18:29 #topic Progress on "Documenting large scale operations" goal 08:18:31 #link https://etherpad.openstack.org/p/large-scale-sig-documentation 08:18:38 On the OSops side, the repository was just created: 08:18:40 #link https://opendev.org/openstack/osops 08:18:50 It's managed under the "Ops Docs and Tooling" SIG, which is obviously closely related to ours 08:19:12 but probably a better host for tools that are not specific for large scale 08:19:20 amorin: you should reach out to smcginnis (or the SIG) to see where you could propose OSarchiver 08:19:28 Personally I'd say it should have its own directory under tools/ as it is something you intend to maintain... 08:20:52 I think this is a great initiative and for me this should be the "transition" path for the projects adopt/integrate this tools in the projects 08:21:21 this shows that ops still need something that the projects don't offer 08:21:24 ok, I think we lost amorin, but he will pick up the message later I expect 08:21:30 The other open task on that workstream is to collect more metrics/billing stories. 08:21:45 A task we could just move to the new "Meaningful monitoring" workstream, as part of the "current status" collection 08:22:05 Nothing new has been contributed there, so I will push back the action there 08:22:41 #action all to describe briefly how you solved metrics/billing in your deployment in https://etherpad.openstack.org/p/large-scale-sig-documentation 08:22:50 Anything else on that topic? 08:23:26 #topic Progress on "Scaling within one cluster" goal 08:23:30 #link https://etherpad.openstack.org/p/large-scale-sig-cluster-scaling 08:23:41 The work on middleware ping has landed, so we are left with oslo.metrics work... 08:23:50 I did push a basic test framework for oslo.metrics at 08:23:53 #link https://review.opendev.org/#/c/752262/ 08:24:02 I'll continue by trying to add meaningful tests 08:24:12 (but need to dive deeper into the code first) 08:24:17 masahito: any progress on posting your latest changes ? 08:24:46 amorin: did you discuss testing it internally at OVH? 08:24:49 sorry.... nothing now. 08:25:06 That's ok, I'll just push back the action item 08:25:07 ttx yes 08:25:10 #action masahito to push latest patches to oslo.metrics 08:25:11 I talked about it 08:25:22 we did not yet tested it 08:25:30 but we are considering doing it 08:25:30 ok, great! Keep us posted 08:25:34 yup 08:25:53 Our second subtopic is around scaling stories on: 08:25:57 #link https://etherpad.openstack.org/p/scaling-stories 08:26:04 Nothing new posted there... our next action is the forum session at the summit in a month. 08:26:18 Anything else on this topic? 08:26:39 I can contribute for the forum session 08:26:45 if there is space 08:27:14 belmoreira: theer is! The goal will be to create a lively conversation space to encourage others to share their own journey scaling up/out 08:27:37 So it's good to have 2-3 people signed up to bootstrap the discussion 08:27:55 great, a good topic would be moving from cells to regions because scalability issues 08:27:56 We have James Penick (or someone from his team) + Arnaud that have volunteered to help too 08:28:09 belmoreira: nice topic indeed! 08:28:26 very curious about that 08:28:34 is it because of neutron? 08:28:58 amorin of course :) 08:29:24 Maybe we can organize the discussion along different stages. Like "first thing that fails as you add more nodes" (scaling up woes), then moving from cells to regions (scaling out woes) 08:30:10 ttx yes, that makes sense 08:30:44 I'm personally very interested in gettign an order of magnitude of number of nodes/activity for a single cluster, as it's the top question I'm being asked by new companies starting to scale up their deployments 08:31:10 like "what should we expect? What should we watch?" 08:31:26 we have input at OVH about that 08:31:43 And I'm always like... "well, somewhere between 100 and 1000 physical servers, something will fail and force you to scale out" 08:31:48 which is not super-helpful :) 08:32:12 also I'm not even sure that 100-1000 range is current in Ussuri 08:32:16 there are plenty of differents things that could break 08:32:32 and it depends also on what kind of usage you are doing with openstack 08:32:45 that's why it's hard to answer I think 08:33:02 amorin: right, type of activity will change the numbers completely 08:33:02 and the risk that you are willing to take 08:33:34 Also it's normal that "plenty of things can fail", since if there was only one thing, we would have handled that low hanging fruit already 08:34:03 yup 08:34:32 but still having guidance would be super helpful for the ops at that stage of their openstack life 08:34:39 it can be fuzzy 08:34:58 but just knowing that others have goine through that and survived... is good 08:35:44 #topic PTG/Summit plans update 08:35:58 So I did file a Forum session on scaling stories, which was just accepted/scheduled 08:36:05 * ttx checks when 08:36:37 https://www.openstack.org/summit/2020/summit-schedule/events/24746/share-your-openstack-scaling-story 08:36:52 Tuesday, October 20, 7:30am-8:15am CT (2:30pm - 3:15pm UTC) 08:36:53 great 08:37:12 hmm not CT, PT 08:37:39 friendly time slot 08:37:51 #info Forum session Tuesday, October 20, 7:30am-8:15am PT (2:30pm - 3:15pm UTC) 08:37:57 yes, it's good for us 08:38:50 Let me start an etherpad about it 08:38:58 #link https://etherpad.opendev.org/p/w-forum-scaling-stories 08:41:25 I dumped the base themes, feel free to add to that 08:41:54 we only have 45 minutes so I would keep it to 4-5 questions max 08:42:50 ack 08:43:31 +1 08:43:44 Alright let's refine that doc between now and the event 08:43:54 Otherwise for PTG week we will have two one-hour sessions: 08:44:00 #info PTG meeting Wednesday Oct 28 7UTC-8UTC and 16UTC-17UTC 08:44:08 I suspect those will be approved but will let you know when I know more 08:44:31 The idea will be to encourage people from the Forum session to follow up at the PTG hours 08:44:46 (and then to sign them up to join the SIG meeting) 08:45:27 We'll likely use both to introduce the various SIG workstreams and ask for input/help 08:45:42 (both PTG hours) 08:45:57 Any question on that 08:45:59 ? 08:46:33 ok then 08:46:34 #topic Next meeting 08:46:43 Next meeting will be US-EU-friendly on Oct 7, 16utc. 08:46:56 Then we'll have summit and PTG weeks, and then go back to our regular cadence in November. 08:47:06 #info next meeting: Oct 7, 16:00UTC 08:47:14 Anything else before we close? 08:48:09 Alright! Thanks everyone! 08:48:16 #endmeeting