08:00:46 <ttx> #startmeeting large_scale_sig
08:00:46 <openstack> Meeting started Wed Jul  8 08:00:46 2020 UTC and is due to finish in 60 minutes.  The chair is ttx. Information about MeetBot at http://wiki.debian.org/MeetBot.
08:00:47 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
08:00:49 <openstack> The meeting name has been set to 'large_scale_sig'
08:00:50 <ttx> #topic Rollcall
08:00:56 <ttx> Who is here for the Large Scale SIG meeting ?
08:00:59 <mdelavergne> Hi!
08:01:03 <amorin> Hello
08:01:20 <ttx> France is up and awake
08:01:51 <mdelavergne> Yep!
08:02:11 <amorin> :p
08:02:21 <ttx> I was hoping we'd have a few recruits from opendev week
08:03:00 <ttx> Let's wait a few minutes and see if anyone else joins, otherwise it will be a short meeting
08:03:34 <amorin> did you meet some new people during the summit?
08:04:41 <ttx> masahito is on IRC, just pinged him
08:05:05 <masahito> hi
08:05:11 <ttx> o/
08:05:30 <ttx> amorin: there were a couple of people who said in the chat they would join
08:06:03 <ttx> I wonder how much our US-unfriendly time has been harming that will to join
08:06:23 <ttx> I'll try to identify and contact those who expressed interest in the chat
08:06:24 <amorin> ack
08:06:38 <ttx> see if we need to rotate meeting time or anything
08:07:11 <ttx> #action ttx to identify from the chat interested candidates from Opendev event and invite them to next meeting
08:07:33 <ttx> masahito: did you have a good time with Opendev last week?
08:07:46 <masahito> yeah.
08:07:53 <ttx> Did anyone else have any specific feedback about it?
08:08:09 <ttx> things to change before teh next one(s)?
08:08:51 <mdelavergne> I couldn't attend personnally
08:09:42 <ttx> ok, moving on then
08:09:50 <ttx> #topic Progress on "Documenting large scale operations" goal
08:09:54 <ttx> #link https://etherpad.openstack.org/p/large-scale-sig-documentation
08:10:17 <amorin> my action is still in todo, wasnt able to move that forward
08:10:38 <ttx> ok, will add it again for the sake of meeting notes
08:10:43 <ttx> #action amorin to add some meat to the wiki page before we push the Nova doc patch further
08:11:11 <ttx> amorin: you did, however, post your metrics/billing story on https://etherpad.openstack.org/p/large-scale-sig-documentation
08:11:15 <ttx> thanks for that!
08:12:09 <ttx> Good example of fine-tuning ceilometer down to reasonable amounts of data
08:12:17 <amorin> ah yes :p
08:12:33 <ttx> Can you tell what you are using for metrics storage?
08:12:51 <amorin> yes, we are using a mongodb
08:12:59 <amorin> I can add some details on the etherpad
08:13:23 <ttx> ok, and are those metrics directly use for billing, or do you have separate data collection for that?
08:13:44 <amorin> so, the metrics, after beeing sent to the rabbit cluster
08:14:01 <amorin> are consumed by a ceileter service (I dont remember the name of this one)
08:14:13 <amorin> but we patched this service a little bit
08:14:29 <amorin> to do some computation and push the results in a mongodb
08:14:50 <amorin> then the date stored in mongo are used by some OVH internal robots to do the billing
08:15:06 <amorin> I dont know if it's answering your question?
08:15:39 <ttx> it does! thanks
08:16:21 <ttx> A reminder for others to push their story so that we can compare approaches!
08:16:31 <ttx> #action all to describe briefly how you solved metrics/billing in your deployment in https://etherpad.openstack.org/p/large-scale-sig-documentation
08:16:47 <ttx> Anything else on the "Documenting large scale operations" topic ?
08:17:20 <ttx> #topic Progress on "Scaling within one cluster" goal
08:17:22 <ttx> #link https://etherpad.openstack.org/p/large-scale-sig-cluster-scaling
08:17:29 <ttx> masahito: last week we discussed review teams for oslo.metrics.
08:17:39 <ttx> last meeting I mean
08:17:42 <ttx> I think it would make sense to add one of the original authors to the oslo.metrics core review team
08:17:49 <ttx> should we add you? Or someone else from LINE?
08:18:11 <masahito> Me is fine.
08:18:27 <ttx> OK, adding you now
08:18:41 <masahito> If needed, I'll add other folks.
08:18:57 <ttx> ah! Actually you're already in
08:19:08 <ttx> https://review.opendev.org/#/admin/groups/2096,members
08:19:17 <ttx> I must have had that discussion with myself already
08:19:53 <ttx> So for the moment that's us two + the other oslo-core
08:20:15 <ttx> Next step is to add basic tests, so that we are reasonably confident we do not introduce regressions
08:20:37 <ttx> Let us know if someone is interested in looking into that. I have it on my TODO list but it can take me a while before i get there
08:21:06 <ttx> Anything else to report on the "Scaling within one cluster" topic?
08:21:53 <ttx> #topic Large Scale SIG slide for OpenStack 10th birthday campaign
08:22:15 <ttx> We were offered a slide in the 10th birthday slide deck to promote the Large Scale SIG
08:22:19 <ttx> I proposed the following content:
08:22:21 <ttx> #link https://etherpad.opendev.org/p/JcyxFIi-F9I_cIi2QOyX
08:22:27 <ttx> and it was accepted for the 10th anniversary deck
08:22:37 <ttx> Let me know if you have comments, I can still get them to update it
08:23:03 <amorin> sounds good to me
08:23:30 <ttx> #topic Next meeting
08:24:13 <ttx> Let's plan to have it in two weeks... If as I contact potential newcomers the time is not working out for them, I'll start an email thread to discuss a better time
08:24:31 <ttx> Does that work for everyone?
08:24:35 <amorin> yes
08:24:53 <ttx> We've been struggling to reach critical mass lately, so let's not lose the existing members :)
08:25:15 <ttx> (+ I think Belmiro is in holidays right now)
08:25:27 <ttx> #info next meeting: Jul 22, 8:00UTC
08:25:28 <amorin> I will be off in 2 weeks btw
08:26:34 <ttx> amorin: ok... I was considering skipping it to be honest... but I was also hoping the new people would be here today. In two weeks I can still get some of teh event momentum to get them to join
08:26:52 <ttx> in 4 weeks they will probably have forgotten about it
08:27:16 <ttx> So let's still plan in two weeks, even if we make it a very short meeting
08:27:33 <amorin> yes, ok
08:27:42 <masahito> I'm okay.
08:27:42 <ttx> amorin: have a good time off!
08:27:47 <ttx> Thanks everyone
08:27:48 <amorin> thanks :p
08:27:54 <amorin> before closing up
08:28:06 <amorin> I'd like to share some stuff we are working on at OVH
08:28:13 <ttx> please do!
08:28:15 <amorin> or maybe I should do it on mailing list?
08:28:31 <ttx> probably larger reach on the ML
08:28:34 <ttx> but now the suspense is killing me
08:28:42 <amorin> that will be quick :p
08:29:02 <amorin> I am pretty sure other large scale groups already hit some DB size issue
08:29:12 <amorin> so we built a tool, that we call osarchiver
08:29:14 <amorin> https://github.com/ovh/osarchiver/
08:29:35 <ttx> I remember it being around for a while already
08:29:36 <amorin> basically it's removing (or moving to other DB) some data from a running mysql cluster
08:29:40 <amorin> yes
08:30:10 <amorin> I dont know if it make sense, but maybe we can move it under opendev gerrit?
08:30:29 <ttx> I think it makes sense, if only to make it more discoverable
08:30:46 <amorin> ok,
08:30:50 <ttx> There were talks about restarting osops during the opendev event last week
08:31:09 <ttx> I'll give it some thought on where we could land it for maximum impact
08:31:27 <amorin> great :p
08:31:44 <ttx> amorin: could you post a ML thread about the tool and ask if it would make sense to land it somewhere in openstack space
08:31:44 <mdelavergne31> this seems nice
08:31:53 <ttx> I'll take it from there
08:31:55 <amorin> yes, will do
08:32:29 <amorin> the second topic I'd like to talk about is a small patch we did on oslo.messaging:
08:32:32 <amorin> https://review.opendev.org/#/c/735385/
08:32:41 <amorin> to add a ping endpoint on RPC
08:32:51 <ttx> #action amorin to start a thread on osarchiver proposing to land it somewhere in openstack
08:33:53 <amorin> behind this patch, we want to monitor our neutron agent (and nova compute) a little bit closer
08:34:19 <ttx> OK, I'll make sure Sean sees it. I know he was off lately
08:34:39 <amorin> I dont know if people from LINE or other companies are using something similar
08:35:05 <amorin> anyway, I just wanted to let the group know about that :p
08:35:14 <amorin> and that's all for me :p
08:35:59 <ttx> i think Sean is trying to avoid having two features to do ping/healthcheck
08:36:14 <masahito> btw, we directly monitor rabbit cluster for the healthcheck
08:36:37 <amorin> masahito: what do you mean by that
08:36:38 <amorin> ?
08:36:42 <ttx> So it could be a good idea to discuss that in a ML thread too, and get operators to +1 your approach
08:36:58 <amorin> ok, so I will do a mail about that also
08:37:24 <amorin> masahito: I am really interested in your approach on monitoring this :p
08:37:42 <masahito> amorin: oops, sorry I misundersood the patch. It's about endpoint for dispatcher.
08:37:47 <ttx> you can add [largescale-sig] to the subject to encourage large deployers to chime in
08:38:14 <amorin> yes ok
08:38:47 <ttx> I'll add those two work items (osarchiver and the ping) to the etherpads so that we remember to track them
08:38:47 <amorin> masahito: ok, please, check the patch, let me know, maybe I am completely wrong in my approach
08:38:55 <amorin> thanks
08:39:59 <ttx> #action amorin to start a [largescale-sig] thread about his middleware ping approach, SIG members can comment if that makes sense for them
08:40:15 <ttx> #link https://review.opendev.org/#/c/735385/
08:40:21 <ttx> Ok I think I captured everything
08:40:27 <ttx> Anything else before we close?
08:41:18 <ttx> I'll take that as a no
08:41:22 <ttx> Have a great week!
08:41:27 <amorin> that's good for me
08:41:32 <amorin> thanks, great week also
08:41:34 <ttx> #endmeeting