08:00:46 #startmeeting large_scale_sig 08:00:46 Meeting started Wed Jul 8 08:00:46 2020 UTC and is due to finish in 60 minutes. The chair is ttx. Information about MeetBot at http://wiki.debian.org/MeetBot. 08:00:47 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 08:00:49 The meeting name has been set to 'large_scale_sig' 08:00:50 #topic Rollcall 08:00:56 Who is here for the Large Scale SIG meeting ? 08:00:59 Hi! 08:01:03 Hello 08:01:20 France is up and awake 08:01:51 Yep! 08:02:11 :p 08:02:21 I was hoping we'd have a few recruits from opendev week 08:03:00 Let's wait a few minutes and see if anyone else joins, otherwise it will be a short meeting 08:03:34 did you meet some new people during the summit? 08:04:41 masahito is on IRC, just pinged him 08:05:05 hi 08:05:11 o/ 08:05:30 amorin: there were a couple of people who said in the chat they would join 08:06:03 I wonder how much our US-unfriendly time has been harming that will to join 08:06:23 I'll try to identify and contact those who expressed interest in the chat 08:06:24 ack 08:06:38 see if we need to rotate meeting time or anything 08:07:11 #action ttx to identify from the chat interested candidates from Opendev event and invite them to next meeting 08:07:33 masahito: did you have a good time with Opendev last week? 08:07:46 yeah. 08:07:53 Did anyone else have any specific feedback about it? 08:08:09 things to change before teh next one(s)? 08:08:51 I couldn't attend personnally 08:09:42 ok, moving on then 08:09:50 #topic Progress on "Documenting large scale operations" goal 08:09:54 #link https://etherpad.openstack.org/p/large-scale-sig-documentation 08:10:17 my action is still in todo, wasnt able to move that forward 08:10:38 ok, will add it again for the sake of meeting notes 08:10:43 #action amorin to add some meat to the wiki page before we push the Nova doc patch further 08:11:11 amorin: you did, however, post your metrics/billing story on https://etherpad.openstack.org/p/large-scale-sig-documentation 08:11:15 thanks for that! 08:12:09 Good example of fine-tuning ceilometer down to reasonable amounts of data 08:12:17 ah yes :p 08:12:33 Can you tell what you are using for metrics storage? 08:12:51 yes, we are using a mongodb 08:12:59 I can add some details on the etherpad 08:13:23 ok, and are those metrics directly use for billing, or do you have separate data collection for that? 08:13:44 so, the metrics, after beeing sent to the rabbit cluster 08:14:01 are consumed by a ceileter service (I dont remember the name of this one) 08:14:13 but we patched this service a little bit 08:14:29 to do some computation and push the results in a mongodb 08:14:50 then the date stored in mongo are used by some OVH internal robots to do the billing 08:15:06 I dont know if it's answering your question? 08:15:39 it does! thanks 08:16:21 A reminder for others to push their story so that we can compare approaches! 08:16:31 #action all to describe briefly how you solved metrics/billing in your deployment in https://etherpad.openstack.org/p/large-scale-sig-documentation 08:16:47 Anything else on the "Documenting large scale operations" topic ? 08:17:20 #topic Progress on "Scaling within one cluster" goal 08:17:22 #link https://etherpad.openstack.org/p/large-scale-sig-cluster-scaling 08:17:29 masahito: last week we discussed review teams for oslo.metrics. 08:17:39 last meeting I mean 08:17:42 I think it would make sense to add one of the original authors to the oslo.metrics core review team 08:17:49 should we add you? Or someone else from LINE? 08:18:11 Me is fine. 08:18:27 OK, adding you now 08:18:41 If needed, I'll add other folks. 08:18:57 ah! Actually you're already in 08:19:08 https://review.opendev.org/#/admin/groups/2096,members 08:19:17 I must have had that discussion with myself already 08:19:53 So for the moment that's us two + the other oslo-core 08:20:15 Next step is to add basic tests, so that we are reasonably confident we do not introduce regressions 08:20:37 Let us know if someone is interested in looking into that. I have it on my TODO list but it can take me a while before i get there 08:21:06 Anything else to report on the "Scaling within one cluster" topic? 08:21:53 #topic Large Scale SIG slide for OpenStack 10th birthday campaign 08:22:15 We were offered a slide in the 10th birthday slide deck to promote the Large Scale SIG 08:22:19 I proposed the following content: 08:22:21 #link https://etherpad.opendev.org/p/JcyxFIi-F9I_cIi2QOyX 08:22:27 and it was accepted for the 10th anniversary deck 08:22:37 Let me know if you have comments, I can still get them to update it 08:23:03 sounds good to me 08:23:30 #topic Next meeting 08:24:13 Let's plan to have it in two weeks... If as I contact potential newcomers the time is not working out for them, I'll start an email thread to discuss a better time 08:24:31 Does that work for everyone? 08:24:35 yes 08:24:53 We've been struggling to reach critical mass lately, so let's not lose the existing members :) 08:25:15 (+ I think Belmiro is in holidays right now) 08:25:27 #info next meeting: Jul 22, 8:00UTC 08:25:28 I will be off in 2 weeks btw 08:26:34 amorin: ok... I was considering skipping it to be honest... but I was also hoping the new people would be here today. In two weeks I can still get some of teh event momentum to get them to join 08:26:52 in 4 weeks they will probably have forgotten about it 08:27:16 So let's still plan in two weeks, even if we make it a very short meeting 08:27:33 yes, ok 08:27:42 I'm okay. 08:27:42 amorin: have a good time off! 08:27:47 Thanks everyone 08:27:48 thanks :p 08:27:54 before closing up 08:28:06 I'd like to share some stuff we are working on at OVH 08:28:13 please do! 08:28:15 or maybe I should do it on mailing list? 08:28:31 probably larger reach on the ML 08:28:34 but now the suspense is killing me 08:28:42 that will be quick :p 08:29:02 I am pretty sure other large scale groups already hit some DB size issue 08:29:12 so we built a tool, that we call osarchiver 08:29:14 https://github.com/ovh/osarchiver/ 08:29:35 I remember it being around for a while already 08:29:36 basically it's removing (or moving to other DB) some data from a running mysql cluster 08:29:40 yes 08:30:10 I dont know if it make sense, but maybe we can move it under opendev gerrit? 08:30:29 I think it makes sense, if only to make it more discoverable 08:30:46 ok, 08:30:50 There were talks about restarting osops during the opendev event last week 08:31:09 I'll give it some thought on where we could land it for maximum impact 08:31:27 great :p 08:31:44 amorin: could you post a ML thread about the tool and ask if it would make sense to land it somewhere in openstack space 08:31:44 this seems nice 08:31:53 I'll take it from there 08:31:55 yes, will do 08:32:29 the second topic I'd like to talk about is a small patch we did on oslo.messaging: 08:32:32 https://review.opendev.org/#/c/735385/ 08:32:41 to add a ping endpoint on RPC 08:32:51 #action amorin to start a thread on osarchiver proposing to land it somewhere in openstack 08:33:53 behind this patch, we want to monitor our neutron agent (and nova compute) a little bit closer 08:34:19 OK, I'll make sure Sean sees it. I know he was off lately 08:34:39 I dont know if people from LINE or other companies are using something similar 08:35:05 anyway, I just wanted to let the group know about that :p 08:35:14 and that's all for me :p 08:35:59 i think Sean is trying to avoid having two features to do ping/healthcheck 08:36:14 btw, we directly monitor rabbit cluster for the healthcheck 08:36:37 masahito: what do you mean by that 08:36:38 ? 08:36:42 So it could be a good idea to discuss that in a ML thread too, and get operators to +1 your approach 08:36:58 ok, so I will do a mail about that also 08:37:24 masahito: I am really interested in your approach on monitoring this :p 08:37:42 amorin: oops, sorry I misundersood the patch. It's about endpoint for dispatcher. 08:37:47 you can add [largescale-sig] to the subject to encourage large deployers to chime in 08:38:14 yes ok 08:38:47 I'll add those two work items (osarchiver and the ping) to the etherpads so that we remember to track them 08:38:47 masahito: ok, please, check the patch, let me know, maybe I am completely wrong in my approach 08:38:55 thanks 08:39:59 #action amorin to start a [largescale-sig] thread about his middleware ping approach, SIG members can comment if that makes sense for them 08:40:15 #link https://review.opendev.org/#/c/735385/ 08:40:21 Ok I think I captured everything 08:40:27 Anything else before we close? 08:41:18 I'll take that as a no 08:41:22 Have a great week! 08:41:27 that's good for me 08:41:32 thanks, great week also 08:41:34 #endmeeting