15:00:32 <witek> #startmeeting monasca
15:00:32 <openstack> Meeting started Wed Jun  6 15:00:32 2018 UTC and is due to finish in 60 minutes.  The chair is witek. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:34 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:36 <openstack> The meeting name has been set to 'monasca'
15:00:42 <dougsz> hey all
15:00:45 <jgrassler> Hello
15:00:46 <witek> hello
15:00:49 <koji> hi
15:00:53 <joadavis> Good day
15:01:10 <witek> congratulations joadavis ! :)
15:01:38 <joadavis> ah, thanks, you must have heard about my new family addition. :)
15:01:57 <witek> yes
15:02:18 <witek> we have few items in agenda today
15:02:23 <witek> https://etherpad.openstack.org/p/monasca-team-meeting-agenda
15:02:27 <witek> let's start
15:02:35 <witek> #topic reviews
15:02:44 <witek> https://review.openstack.org/#/c/562393/
15:03:31 <witek> I quickly went through it
15:03:43 <witek> have just given +1
15:04:21 <witek> more reviewers very welcome
15:04:47 <witek> and if not, joadavis feel free to merge if you think it's ready
15:05:08 <joadavis> I've been through it with Ashwin and think it is ready to merge, but it is big enough that any additional eyes are welcome. Thanks for looking at it
15:06:07 <witek> https://review.openstack.org/#/c/563957/
15:06:39 <witek> that's pretty straight forward
15:07:33 <joadavis> yes, I'd meant to throw https://review.openstack.org/#/c/565162/ and https://review.openstack.org/#/c/565165/ on the list as well.  All seemed reasonable to me
15:08:11 <witek> thanks joadavis
15:08:46 <witek> in general we have some backlog of changes in review
15:08:50 <joadavis> I attempted to tox test 565165 but it failed to set up correctly on my devstack, and didn't have time to debug why
15:09:00 <pandy_> Hi All
15:09:12 <mayankkapoor> Hi everyone
15:09:14 <witek> it would be great, if we could invest some more time in reviews and try to get them merged
15:09:30 <witek> pandy_: mayankkapoor: hello
15:09:54 <dougsz> hmm, i also did the joadavis, and it ran all the expected tests
15:10:16 <joadavis> I suspected it was my system, so didn't see it as a blocker
15:10:17 <dougsz> *same
15:10:51 <joadavis> if it had passed, I would have +2
15:10:55 <joadavis> 'd :)
15:11:44 <pandy_> would like to know suggestions for deploying monasca on large scale production setup, mainly looking forward handling kafka
15:12:10 <joadavis> greetings, pandy_.  Yeah, that's on the agenda
15:12:11 <witek> pandy_: please hold on, we're on the first agenda item
15:12:42 <witek> to end up with review topic: I've sent a nomination for dougsz to join the core team
15:12:50 <pandy_> witek, sure, waiting for my turn
15:13:02 <joadavis> +1 for adding dougsz :)
15:13:10 <witek> he actively contributes to log-api and kolla
15:13:24 <witek> and reviews actively
15:13:39 <jgrassler> +1 :-)
15:14:36 <witek> I see no objections; congratulations dougsz! I'll add you to the list after the meeting
15:14:56 <witek> #topic monasca-ceilometer publisher
15:15:10 <dougsz> thank you all :) it's privilege to join. I look forward to helping to keep things moving along
15:15:24 <joadavis> Just a quick update on that.  The publisher contribution has stalled in ceilometer.
15:15:58 <witek> Yes, I've noticed
15:16:11 <joadavis> I chatted with two of the Telemetry guys, but was told effectively that ceilometer is in maintenance mode and I could take it up with the tc or take over ceilometer myself
15:16:39 <witek> with whom have you talked?
15:16:41 <joadavis> chat history in #openstack-telemetry last week if anyone is curious
15:16:46 <joadavis> julien and gordon
15:16:59 <joadavis> So I wanted to see if you witek had any sway. :)
15:17:20 <witek> does TC know that?
15:17:45 <witek> I mean they're adding new publishers theirselves
15:17:53 <joadavis> I don't know if tc or openstack at large is aware of the lack of developers for telemetry projects
15:18:04 <witek> prometheus being the recent example
15:18:15 <joadavis> (and yes, the fact that a new publisher was added two months ago did invalidate that argument)
15:19:08 <joadavis> I'm not sure if i just asked the wrong question, or if having the name "monasca" on it automatically gets a negative reaction from the telemetry team. :P
15:19:38 <witek> I don't know
15:19:43 <witek> I'll go through the logs
15:19:54 <witek> could you point me to them?
15:19:56 <joadavis> MAy 28 and 29th
15:19:59 <witek> thanks
15:20:17 <witek> for now, I have left you some cosmetic comments in review
15:20:34 <witek> I though about writing to them on openstack-dev mailing list
15:20:42 <witek> but want to read the logs first
15:21:03 <witek> I think we can sync on this offline joadavis
15:21:03 <joadavis> thanks.  I have been pondering other long term options, but all would take some work involved that I don't think we have resources to sign up to do
15:21:38 <witek> any other votes on this?
15:22:27 <witek> #topic log-api
15:22:45 <witek> dougsz: your turn
15:22:54 <dougsz> So this is pretty close. I've made a monster patch here: https://review.openstack.org/#/c/570599/
15:23:19 <dougsz> I can split that into two patches if people would prefer - one to add the ElasticSearch repo, and the other for the API
15:23:33 <witek> #link https://review.openstack.org/#/c/570599/
15:24:02 <dougsz> My major question is around common code. I started off by trying to factor of common methods from the monasca API into monasca common so that I could use them in the log-API
15:24:27 <dougsz> But then, after adding a voluptuous schema, I had to change these methods a bit
15:24:54 <dougsz> So I wonder if I should confine the log-api change to log-api repo, and address the common code in later commits
15:25:09 <dougsz> To keep things simpler
15:25:19 <jgrassler> I'd vote for that (take care of the monasca-common adjustments later).
15:25:46 <dougsz> Going one step further, I seriously wonder if combining the monasca-api and monasca-log-api might make it easier to maintain
15:26:06 <jgrassler> I'm not a huge fan of "let's add this and let's also include that" - it's got a way of never ending :-)
15:26:29 <dougsz> Yeah - the change is already pretty big.
15:26:41 <witek> I understand dougsz meant it for longer term
15:28:24 <witek> I basically like the idea of joining these two repos
15:28:44 <witek> which would also be valid for events-api
15:28:59 <witek> but would prefer to plan it for the next release
15:29:17 <dougsz> sounds good
15:29:20 <witek> I think it could reduce the maintenance effort
15:29:40 <witek> other thoughts?
15:29:41 <jgrassler> Yes. From a packaging point of view, I'm all for it...
15:30:04 <joadavis> sounds like a good plan
15:30:09 <dougsz> ok, cool
15:30:27 <dougsz> So i will make the log-query-api all in one place, and leave the refactoring for later
15:31:22 <witek> we should take a closer look though what issues could occur
15:31:33 <witek> I mean, when joining
15:31:57 <jgrassler> Yes. This definitely warrants a spec.
15:32:13 <witek> agree
15:32:22 <dougsz> Maybe I can start by throwing some observations into one
15:32:30 <witek> good idea
15:32:58 <dougsz> ok, thanks all, that's it from me
15:33:10 <witek> folks, please review :)
15:33:26 <jgrassler> For one, operators will no longer be able to separate log and metrics APIs (and given the amount of traffic that goes through them that might be desirable :-)
15:35:01 <dougsz> it'd certainly be nice to collect use cases. From our point of view, we always deploy them together.
15:35:31 <jgrassler> I guess that could be taken care of through the service catalog.
15:36:05 <jgrassler> You just run multiple instances of the unified API services and designate one as "this is for logs" and the other as "this is for metrics"
15:36:29 <dougsz> yeah
15:36:45 <dougsz> that makes sense to me
15:37:19 <witek> let's write up a spec for this, so we can go through all this in review
15:37:41 <dougsz> sounds good, i'll start one
15:37:44 <witek> thanks dougsz
15:37:56 <witek> #topic large deployments
15:38:11 <witek> pandy_: sorry, that you had to wait
15:39:14 <witek> pandy_: so you wanted to have some advice on Kafka configuration, right?
15:39:44 <pandy_> witek, thanks for adding our topic in agenda, mayankkapoor_ will explain
15:39:54 <mayankkapoor> Some brief context. pandy_ and I work together in Reliance Jio. Jio is a telecom startup, and we have our own Openstack private cloud.
15:39:59 <mayankkapoor> In Inida
15:40:03 <mayankkapoor> *India
15:40:24 <witek> nice, have we met in Vancouver?
15:40:49 <mayankkapoor> Unfortunately both pandy_ and I couldn't attend. Harshit from our team was there.
15:41:05 <mayankkapoor> He has attended the Monasca update session I think
15:41:32 <mayankkapoor> So long story short, we have about 2000 bare-metals running Openstack, and about 200 running Kubernetes now
15:42:03 <mayankkapoor> Our openstack deployments are 500 nodes each (400 computes, 100 odd storage nodes)
15:42:29 <mayankkapoor> Till now we haven't enabled ceilometer for VM level monitoring due to scaling issues. We currently run Ocata.
15:43:18 <mayankkapoor> We reviewed Monasca architecture and liked it a lot. So the question is: Would you help point us to configuration and deployment settings meant for large scale clouds?
15:44:04 <witek> how do you want to deploy?
15:44:20 <mayankkapoor> Our current plan is to deploy on bare-metals, running docker swarm
15:44:47 <mayankkapoor> Some issues with Kafka running on docker swarm have led us to separate out the Kafka cluster and run in on bare-metals directly.
15:45:00 <mayankkapoor> We have earmarked 3 bare-metal nodes in our control plane for Monasca
15:45:34 <witek> monasca-helm deploys Kafka together with other components in Kubernetes cluster
15:45:50 <witek> HPE and OP5 use it
15:46:24 <dougsz> mayankkapoor: From my side (StackHPC) we haven't scaled up to anything that size yet, but we're working on a 600 node deployment on the Darwin cluster at Cambridge uni, so we should have some experience in a few months.
15:47:00 <witek> dougsz: how do you deploy?
15:47:02 <mayankkapoor> Yeah saw the helm chart. Ok we will defn try the helm install and report back.
15:47:33 <dougsz> At the moment we have a hybrid deployment of Kolla Ansible + some legacy LXC containers
15:47:59 <dougsz> The Kolla-Ansible deployment should be finished in a few weeks at which point we'll switch over to that entirely.
15:48:15 <joadavis> At SUSE we are still deploying to VMs using ansible or crowbar (containers will come soon :) )
15:48:28 <witek> dougsz: with Monasca included in Kolla?
15:48:31 <jgrassler> Actually Crowbar uses Ansible, too :-)
15:48:43 <dougsz> Yeah that's right
15:50:39 <mayankkapoor> We'll go ahead first with our bare-metal install, and then try the helm install. Thanks for the advice.
15:50:45 <witek> mayankkapoor: it could make sense to dedicate own node(s) for TSDB
15:51:34 <mayankkapoor> @witek: ok we can try that. We'll test first and see if there are any bottlenecks.
15:51:58 <witek> how many measurements do you plan to collect?
15:52:49 <mayankkapoor> Very few: CPU, RAM, Storage are the critical ones. Currently we're using the default ones enabled, but will steadily turn unnece metrics off
15:52:56 <mayankkapoor> *unnecessary metrics off
15:54:24 <mayankkapoor> No further questions for now, thanks for listening
15:55:00 <pandy_> as mayankkapoor will try helm, also looking for advise to handle kafka with better performance
15:55:08 <dougsz> mayankkapoor: It'll be interesting to hear how you get on
15:55:21 <mayankkapoor> @dougz: sure
15:55:50 <witek> I'll ping OP5 folks, if they can give some advice from their deployment
15:56:00 <mayankkapoor> thanks
15:56:31 <witek> I'm also very interested in your progress
15:57:12 <pandy_> witek, thanks, looking forward op5 docs for deployment
15:57:25 <witek> pandy_: mayankkapoor: thanks
15:57:41 <witek> #topic READMEs convertion
15:57:59 <witek> it seems we have to convert our main READMEs to rst
15:58:17 <witek> otherwise we won't be able to create new releases
15:58:50 <witek> I just noticed I have linked wrong story
15:59:03 <witek> the right one:
15:59:06 <witek> https://storyboard.openstack.org/#!/story/2002163
15:59:35 <witek> https://review.openstack.org/572764
15:59:51 <witek> here attempt to create one
16:00:26 <witek> if you have some time please pick up a repo, add a task and convert
16:00:43 <witek> I have to wrap up now
16:00:46 <witek> thanks everyone
16:00:51 <witek> see you next week
16:00:57 <koji> thanks
16:00:57 <joadavis> thanks!
16:01:01 <witek> #endmeeting