08:00:02 <ttx> #startmeeting large_scale_sig
08:00:03 <openstack> Meeting started Wed Jun 24 08:00:02 2020 UTC and is due to finish in 60 minutes.  The chair is ttx. Information about MeetBot at http://wiki.debian.org/MeetBot.
08:00:04 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
08:00:06 <ttx> #topic Rollcall
08:00:07 <openstack> The meeting name has been set to 'large_scale_sig'
08:00:14 <ttx> Who is here for the Large Scale SIG meeting ?
08:00:23 <amorin> hello!
08:00:42 <ttx> belmoreira: ping
08:00:53 <belmoreira> o/
08:01:06 <belmoreira> ttx: thanks for the ping
08:01:09 <ttx> huyupeng: hi!
08:01:27 <huyupeng> hello
08:01:46 <ttx> huyupeng: first time joining?
08:01:51 <huyupeng> yes
08:02:01 <brinzhang_> \o
08:02:20 <ttx> huyupeng: care to introduce yourself, and let us know what you're interested in?
08:03:40 <ttx> Or we can jump right in today's meeting agenda:
08:03:43 <ttx> #link https://etherpad.openstack.org/p/large-scale-sig-meeting
08:04:07 <huyupeng> ok, i am intersted in concurrency, resource utilize , data plane performance when we deploy large scale openstack cloud
08:04:41 <ttx> huyupeng: sounds like you found the right group to discuss it. I think those are concerns that the group shares
08:04:59 <ttx> #topic 'Large scale operations' OpenDev event next week
08:05:04 <ttx> #link https://www.openstack.org/events/opendev-2020/
08:05:09 <ttx> Are y'all ready for the event next week?
08:05:34 <ttx> Remember that our goal is to start new discussions, and motivate new organizations to engage publicly
08:05:48 <ttx> The Large Scale SIG should be used to continue those discussions, we don't want this to be a one-off thing without follow-up
08:06:00 <ttx> So the idea will be to heavily promote the SIG during all those discussions
08:06:08 <ttx> We should point to our wiki page
08:06:14 <ttx> #link https://wiki.openstack.org/wiki/Large_Scale_SIG
08:06:24 <ttx> We should also point to our next meeting, July 8, 8utc on #openstack-meeting-3
08:06:34 <ttx> And also to the meeting etherpad for people to add items to the agenda.
08:06:47 <ttx> belmoreira: Anything else we need to prepare before next week event?
08:07:23 <belmoreira> we have the "Upgrades" session on Monday. It would be great to have all your experience there for discussion
08:09:07 <ttx> Also don;t forget to register, it's free
08:09:15 <huyupeng> i care about how rabbitmq behavie when there are a lot client publish message and receive message. how to tunning the rabbitmq
08:09:41 <belmoreira> sorry got disconnected... not sure if you got my msg
08:09:45 <belmoreira> we have the "Upgrades" session on Monday. It would be great to have all your experience there for discussion
08:09:56 <amorin> belmoreira: ack
08:10:11 <amorin> huyupeng: we all struggle on rabbitmq tuning :p
08:10:16 <ttx> huyupeng: we identified this as a scaling bottleneck too. We are working to instrument oslo.messaging to get better data
08:10:30 <ttx> in a new library called oslo.metrics
08:10:35 <ttx> we'll talk about it in a minute
08:10:42 <ttx> #topic Progress on "Documenting large scale operations" goal
08:10:46 <ttx> #link https://etherpad.openstack.org/p/large-scale-sig-documentation
08:10:56 <ttx> For the newcomers, this is one of our current goals for the SIG - produce better documentation to help operators setting up large deployments.
08:11:09 <ttx> In particular, documenting better configuration values when you start hitting issues with default values at a certain scale
08:11:19 <ttx> amorin has been leading that effort
08:11:24 <amorin> I havn't be able to move forward on that topic in the past 2 weeks
08:11:39 <ttx> OK I'll push the action item back
08:11:43 <amorin> so if we can copy back my action,
08:11:43 <ttx> #action amorin to add some meat to the wiki page before we push the Nova doc patch further
08:11:44 <amorin> thanks
08:11:54 <ttx> We also had a TODO for all to describe briefly how you solved metrics/billing in your deployment
08:11:58 <ttx> But not much contribution to that yet
08:12:16 <amorin> completely forgot about that
08:12:18 <ttx> huyupeng: one objective of the SIG is to share experience
08:12:31 <amorin> do we have a etherpad to fill?
08:12:33 <ttx> like how various groups solved common problems
08:12:46 <ttx> I added it into https://etherpad.openstack.org/p/large-scale-sig-documentation
08:12:53 <amorin> ack
08:12:56 <ttx> feel free to drop a line there
08:13:10 <ttx> I'll keep that work item open too
08:13:17 <ttx> #action all to describe briefly how you solved metrics/billing in your deployment in https://etherpad.openstack.org/p/large-scale-sig-documentation
08:13:56 <ttx> There seems to be a lot of custom solution in that space, which points to shortcomings in the openstack services offering
08:14:02 <ttx> Anything else on this topic?
08:15:08 <ttx> #topic Progress on "Scaling within one cluster" goal
08:15:11 <ttx> #link https://etherpad.openstack.org/p/large-scale-sig-cluster-scaling
08:15:21 <ttx> For the newcomers, this is the other current goal of the SIG - identify, measure and push back common scaling limitations within one cluster
08:15:31 <ttx> Like RabbitMQ starting to fall off first
08:16:14 <ttx> One key effort within that goal is to work on bringing LINE's work on instrumenting oslo.messaging into upstream OpenStack
08:16:22 <ttx> I had a TODO on solving merge conflict and push initial oslo.metrics code in
08:16:28 <ttx> It's now in at: https://opendev.org/openstack/oslo.metrics/
08:17:01 <ttx> It's mostly at POC stage, but would be great if this group could look it up and tell us if that sounds like a good idea, and how to improve it
08:17:06 <ttx> We also had a bunch of patches in to make the code more OpenStacky, thanks to hberaud
08:17:13 <ttx> smcginnis and I did the reviews.
08:17:26 <ttx> I think for the more feature-oriented reviews coming in the future, it would be good to have one of the original authors from LINE in the core review team for oslo-metrics
08:17:42 <ttx> I'll try to see with masahito if he would be interested in being added
08:18:02 <ttx> The next step will be to add basic tests so that we are reasonably confident we do not introduce regressions
08:18:18 <ttx> I suspect masahito will mention it next week during the Opendev event
08:19:44 <ttx> Anything else on that topic?
08:19:59 <huyupeng> does oslo.metrics has something to do with rabbitmq tunning ?
08:20:30 <ttx> yes, let me point to teh presentation LINE did at the Summit in Shanghai
08:21:14 <ttx> https://www.openstack.org/videos/shanghai-2019/how-we-used-rabbitmq-in-wrong-way-at-a-scale
08:21:30 <ttx> That will give you the background
08:21:46 <ttx> #topic Large Scale SIG slide for OpenStack 10th birthday campaign
08:21:52 <ttx> Helena on the OSF staff reached out to SIG with the following information:
08:21:53 <huyupeng> thanks, i will see later
08:21:58 <ttx> "OpenStack will be turning 10 years old in less than a month! We are sending out a slide deck to the event organizers who are organizing the 10 years of OpenStack celebration. For this slide deck, is there anything about the Large Scale SIG you would like to include?"
08:22:06 <ttx> I was thinking we could make a slide with high-level goals for the SIG and pointers to our meeting time
08:22:14 <ttx> Anyone interested in working on that? Or I can make a draft and propose it next meeting
08:22:59 <ttx> #action ttx to produce a draft of the 10th birthday slide for the SIG
08:23:52 <ttx> huyupeng: in the etherpad links I provided above, you will find other ways to get involved in those SIG goals, like providing scaling stories
08:24:32 <ttx> Scaling stories are accounts of scaling experiences in your deployment -- what broke first, how did it break, and how you moved past it
08:24:36 <huyupeng> ok,copy that
08:24:51 <ttx> We hope to find common pain points this way (RabbitMQ being an obvious one)
08:25:13 <ttx> We document those in https://etherpad.openstack.org/p/scaling-stories
08:25:28 <ttx> and then copy them to https://wiki.openstack.org/wiki/Large_Scale_Scaling_Stories for long-term storage
08:25:54 <ttx> We hope to have more starting next week after the Opendev event!
08:26:00 <ttx> #topic Next meeting
08:26:08 <ttx> Next meeting in two weeks... That's the week just after the Opendev event, so if we get contacts of people excited to join the SIG, we can strike that iron while it's hot.
08:26:18 <ttx> #info next meeting: Jul 8, 8:00UTC
08:26:23 <amorin> ok
08:26:32 <ttx> Anything else before we close ?
08:27:04 <ttx> If there is a topic you want to discuss next time, make sure to add it in advance to the meeting agenda on https://etherpad.opendev.org/p/large-scale-sig-meeting !
08:28:22 <ttx> Alright, if nothing else for today, have a great day, and hope to see you all virtually at Opendev next week
08:28:33 <ttx> Thanks everyone!
08:28:55 <ttx> #endmeeting