16:01:37 <ttx> #startmeeting large_scale_sig
16:01:38 <openstack> Meeting started Wed Oct  7 16:01:37 2020 UTC and is due to finish in 60 minutes.  The chair is ttx. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:01:39 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:01:41 <openstack> The meeting name has been set to 'large_scale_sig'
16:01:45 <ttx> #topic Rollcall
16:01:49 <mdelavergne> Hi!
16:01:51 <ttx> Who is here for the Large Scale SIG meeting ?
16:01:54 <ttx> mdelavergne: hi!
16:01:55 <genekuo_> Hi
16:02:13 <ttx> genekuo_: Hi!
16:02:20 <genekuo_> I'm first time here
16:02:34 <genekuo_> I'm a Infrastructure Engineer at LINE
16:02:40 <genekuo_> masahito is my colleague
16:03:20 <mdelavergne> Welcome!
16:03:28 <ttx> amorin: around?
16:04:07 <ttx> It might be just us 3 today
16:04:13 <ttx> Our agenda for today is at:
16:04:16 <ttx> #link https://etherpad.openstack.org/p/large-scale-sig-meeting
16:04:24 <ttx> #topic PTG/Summit plans update
16:04:32 <ttx> A reminder on the Large Scale SIG activities around Summit and PTG
16:04:43 <ttx> Our Forum session is Tuesday, October 20, 7:30am-8:15am CT
16:04:49 <ttx> #link https://www.openstack.org/summit/2020/summit-schedule/events/24746/share-your-openstack-scaling-story
16:05:02 <ttx> That makes it super early for our US friends and a bit late for our APAC friends
16:05:24 <ttx> genekuo_: it must be super-late for you now
16:05:54 <genekuo_> I'm usually sleep late
16:05:58 <genekuo_> So it's fine
16:06:13 <ttx> I'll moderate the discussion, but we'll also have active participants to help seed the discussion and encourage others to share
16:06:26 <ttx> amorin and belmoreira said they would help
16:06:37 <ttx> In preparation for this session, please add to the etherpad at:
16:06:40 <ttx> #link https://etherpad.opendev.org/p/w-forum-scaling-stories
16:07:10 <ttx> especially if you have things you'd like to see covered
16:07:20 <ttx> The week after that during PTG week we will have two one-hour sessions:
16:07:24 <ttx> #info PTG meeting Wednesday Oct 28 7UTC-8UTC and 16UTC-17UTC
16:07:36 <ttx> Those will be more traditional meetings, the idea being to onboard any new recruit from that forum session
16:07:44 <ttx> Questions on that topic?
16:08:28 <mdelavergne> Not from myself
16:08:54 <ttx> alright, moving on
16:08:58 <ttx> #topic Meaningful monitoring
16:09:08 <ttx> Last month we discussed forming a new workstream around "meaningful monitoring"
16:09:16 <ttx> I tried to bootstrap it in the following etherpad:
16:09:20 <ttx> #link https://etherpad.opendev.org/p/large-scale-sig-meaningful-monitoring
16:09:30 <ttx> genekuo_: is that something that is of interest for you?
16:10:17 <genekuo_> I'll probably will be upstreaming oslo.metrics code that we current have
16:10:36 <ttx> genekuo_: ok, we will cover that in a minute
16:10:44 <ttx> Obviously we need to discuss what we mean by "meaningful monitoring"
16:10:49 <mdelavergne> It would be nice to have some feedback from those who launched this topic :(
16:10:53 <ttx> Is it actionable monitoring, like opinionated/focused monitoring...
16:11:04 <ttx> mdelavergne: yeah, tI was hoping they would be here today
16:11:08 <ttx> since it's "their" time
16:11:13 <genekuo_> This topic is interesting as we have a lot of notifications
16:11:21 <genekuo_> Most of them are not that useful
16:11:47 <ttx> right, so I could see a need for a more targeted monitoring that instead of showing everything, tracks golden signals
16:11:59 <ttx> (as described in that etherpad)
16:12:21 <ttx> But yes I agree with mdelavergne it would be good to hear from those who raised that topic first and hear of their definition
16:12:49 <ttx> moving on to the next workstream
16:12:53 <ttx> #topic Progress on "Scaling within one cluster" goal
16:12:56 <ttx> #link https://etherpad.openstack.org/p/large-scale-sig-cluster-scaling
16:13:04 <ttx> Regarding oslo.metrics, I did push a basic functional test so that we are reasonably sure that it actually works:
16:13:08 <ttx> #link https://review.opendev.org/#/c/755069/
16:13:22 <ttx> genekuo_: would be good to get your review on it (or masahito's)
16:13:28 <genekuo_> Got it
16:13:42 <ttx> Do you know when you'll be able to push the latest version?
16:13:45 <genekuo_> I'll also start writing test once I upstream most of our codes
16:13:52 <genekuo_> There not much left
16:13:59 <genekuo_> I can probably finish it by next week
16:14:03 <ttx> great!
16:14:14 <ttx> Note that according to my testing it seems to be  missing the other side of the code -- the change in oslo.messaging to actually emit those metrics
16:14:33 <ttx> genekuo_: do you have the code for that too?
16:14:56 <genekuo_> We currently haven't have any test yet I think
16:15:02 <genekuo_> Have to double check
16:15:57 <ttx> ok, because as far as I can tell, the oslo.metric code only handles the reception of the message on the socket and it's storage in a Prometheus metric
16:16:16 <ttx> The other side of this workstream is the collection of scaling stories
16:16:19 <ttx> #link https://etherpad.openstack.org/p/scaling-stories
16:16:19 <genekuo_> Yes
16:16:28 <ttx> Nothing new posted there... our next action is the forum session in two weeks
16:16:55 <ttx> Anything else on this "Scaling within one cluster" goal? Questions? Comments?
16:17:09 <genekuo_> I think I can add something to the scaling stories part
16:17:19 <mdelavergne> nice
16:17:34 <genekuo_> We did hit some issue scaling, I'll discuss with masahito tomorrow
16:17:34 <ttx> genekuo_: perfect! Any story, even short, helps!
16:17:58 <ttx> It's basically about "what happens when we add nodes to a cluster, what failed first"
16:18:14 <ttx> (and bonus points for telling how you solved it)
16:18:17 <genekuo_> Got it
16:18:24 <ttx> Moving on to next goal
16:18:27 <ttx> #topic Progress on "Documenting large scale operations" goal
16:18:30 <ttx> #link https://etherpad.openstack.org/p/large-scale-sig-documentation
16:18:51 <ttx> amorin was working on pushing OSarchiver to the OSops repository
16:19:02 <ttx> I guess we'll have to wait for an update on that
16:19:51 <ttx> So for now, just let me know if you have questions on that goal, and if you can help with anything in it
16:20:11 <ttx> #topic Next meeting
16:20:12 <genekuo_> Sounds clear to me for now
16:20:21 <ttx> In two weeks we'll have the Forum session and the week after the live meetings
16:20:29 <ttx> So I propose we get back to our regular rotation two weeks after that
16:20:39 <ttx> Next IRC meeting will be EU+APAC Nov 10, 8utc, then US+EU Nov 24, 16utc.
16:20:44 <ttx> Does that work?
16:20:45 <mdelavergne> ok
16:20:47 <mdelavergne> yep
16:20:49 <genekuo_> ok
16:21:02 <ttx> I'll probably have to send the personal reminder to jpenick and Erik next time
16:21:20 <ttx> since they seem to miss the one I send to teh ML
16:21:26 <ttx> #info next meetings: Nov 10, 8utc; Nov 24, 16utc
16:21:33 <mdelavergne> probably, yes!
16:21:36 <ttx> #topic Open discussion
16:21:45 <ttx> Anything else you'd like to discuss?
16:21:59 <ttx> genekuo_: anything you think this group should do, that is not covered in those 3 goals?
16:22:13 <genekuo_> Haven't think about it yet
16:22:19 <genekuo_> Looks good for me for now
16:22:35 <genekuo_> I'll think about it and provide more feedback if there is
16:23:00 <ttx> feel free to think about it and let us know next time! This is really about what the participants want to do, and try to use the group to help them achieve those objectives
16:23:22 <ttx> Like amorin is leading the doc effort, and you're leading the oslo.metric effort
16:23:32 <ttx> and the rest of the group facilitates
16:24:03 <genekuo_> Sounds good
16:24:06 <ttx> Alright, if you have nothing else... I propose we close early and let genekuo_ go to bed :)
16:24:15 <genekuo_> Thanks!
16:24:16 <mdelavergne> ahah
16:24:19 <ttx> Thanks everyone!
16:24:27 <mdelavergne> thanks to you!
16:24:40 <ttx> Hopefully will see you at the PTG meeting in 3 weeks!
16:24:57 <ttx> (and maybe at the Forum session in two weeks if you can make it!)
16:25:03 <genekuo_> I will join if possible
16:25:06 <ttx> #endmeeting