15:00:56 <witek> #startmeeting monasca
15:00:57 <openstack> Meeting started Wed Jan  9 15:00:56 2019 UTC and is due to finish in 60 minutes.  The chair is witek. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:58 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:01:00 <openstack> The meeting name has been set to 'monasca'
15:01:08 <witek> hello everyone
15:01:15 <kaiokmo> o/
15:01:27 <witek> hi kaiokmo
15:01:35 <kaiokmo> hello witek
15:01:41 <kaiokmo> hope you enjoyed holidays
15:01:44 <dougsz> hey all
15:01:49 <witek> hi dougsz
15:02:06 <witek> yes, thanks :)
15:02:33 <pandy> hi witek & all, after long time, nice to join weekly meeting
15:02:40 <witek> agenda for today https://etherpad.openstack.org/p/monasca-team-meeting-agenda
15:03:01 <chaconpiza> Hi
15:03:19 <witek> #topic CFP for OpenStack Summit
15:03:30 <witek> just the reminder
15:03:40 <witek> the deadline is in two weeks
15:04:51 <pandy> witek, Sure this time am looking forward to submit presentation with challenges in deploying in swarm mode and handling in scale setup. Will send abstract to you in personal before submitting for review
15:05:18 <witek> sounds good
15:05:27 <witek> I encourage everyone to submit
15:05:35 <witek> presentation or workshop
15:07:01 <witek> as I said, deadline is January 23
15:07:11 <pandy> okay :)
15:07:15 <witek> #topic Operation Query
15:07:45 <pandy> Hi witek, seems it mine
15:08:48 <pandy> yes, I would like to convey in short about our setup again to all here, we have deployed docker-monasca in docker swarm in dedicated baremetals
15:09:27 <pandy> where monasca-agent were installed on 500+ compute nodes in virtualenv
15:09:43 <pandy> Now we would like to collect metrics and group them by tenant name
15:09:57 <pandy> how to achieve that ?
15:10:48 <witek> we don't have API call for that
15:11:03 <witek> probably it would be easiest to query the TSDB directly
15:11:29 <pandy> I request you to explain in detail for my easy understanding
15:12:18 <witek> alternatively you could write a script that would iterate through projects and list metrics for each
15:13:43 <pandy> okay
15:14:09 <pandy> I was looking using API call, fine i try with scripting.
15:15:00 <pandy> Also felt discrepancy while monitoring http_checks of monasca-components like kafka, monasca-api health status
15:15:39 <pandy> Example: If kafka died or some error. Monasca-agent getting status after sometime like 5 minutes in average
15:16:09 <pandy> If status comes up, that also getting delayed to update with monasca-agent components.
15:16:42 <witek> what is you collection period?
15:16:48 <pandy> 30 sec
15:17:16 <pandy> I feel the response time is aggregrated value.
15:18:05 <pandy> Quick workaround, if i restart monasca-agent can reduce response time
15:18:14 <dougsz> Hmm, well, if a node in a Kafka cluster dies, won't that affect throughput?
15:19:32 <pandy> Yes it affect, assume kafka not died and failing with kafka-consumer and producer or async
15:20:32 <pandy> I am running kafka in docker swarm, each worker node having one kafka & zookeeper
15:21:36 <pandy> Based on heavy hit some interval time need to increase kafka topics values and some other values. But looking short time response by monasca-agent
15:22:56 <witek> do you query the http_check metric values, or did you set an alarm on that?
15:23:09 <dougsz> Do you have enough partitions per topic to use all Kafka nodes in parallel?
15:24:39 <pandy> witek, Here is my docker-file for monasca-agent collector and forwarder along with caadvisor http://paste.openstack.org/show/740743/
15:25:36 <witek> how do you measure the delay?
15:26:48 <pandy> Through Grafana dashboard kafka.json
15:28:37 <pandy> witek, whether my docker file environment variables are correct ?
15:28:54 <witek> didn't see anything suspect
15:29:33 <pandy> okay
15:30:03 <witek> don't have an explanation
15:30:19 <witek> you could examine API logs to see
15:30:34 <witek> where the delay is created
15:31:16 <dougsz> +1 for analysing delays across the pipeline
15:31:17 <pandy> You mean monasca-api
15:31:25 <witek> yes
15:31:38 <witek> you can start in agent, then api and persister
15:32:32 <pandy> I can query instance metrics with instant response, hope it was done through mon-api and can make sure that responce of mon-api (depend kafka) is fine.
15:34:03 <pandy> if i query metrics for instance for last 10 seconds working fine, it mean kafka and other mon-api dependent service are fine. But monasca-agent forwarder logs says no response from Mon-api
15:34:25 <pandy> after that 5 min interval logs are getting response.
15:34:49 <pandy> Even i am feeling its not expected one. I will try.
15:36:42 <dougsz> Network saturated between the forwarder and the API?
15:38:43 <pandy> dougsz, i feel your answer right
15:39:10 <pandy> somewhere read that while running application on docker swarm mode network issues happens
15:39:48 <witek> do we have anything else?
15:40:25 <dougsz> toabctl pushed some patches for changing the monasca api config file name to align with other openstack projects
15:40:44 <dougsz> https://review.openstack.org/#/c/628931/
15:40:46 <witek> dougsz: did you have chance to test it?
15:41:00 <dougsz> Not yet, hopefully in the next day or two.
15:41:09 <witek> thanks
15:41:27 <witek> I'm still configuring things up in my new dev env
15:41:39 <dougsz> :)
15:41:57 <witek> I rechecked https://review.openstack.org/435136
15:42:19 <witek> and noticed that it's not tested in devstack
15:42:40 <witek> the Zuul job name changed and hasn't been updated
15:43:01 <witek> will update, and we should merge it
15:43:17 <dougsz> thanks witek - will be nice to make some progress on that patch chain
15:43:33 <witek> yes, it hangs too long
15:44:44 <witek> if there's nothing else, I'll be wrapping up
15:44:47 <pandy> witek, dougsz i have addition question
15:44:52 <witek> go ahead
15:45:59 <pandy> i want to store metrics in influxdb for 30 days or some days. But i want to forecast capacity with InfluxDB with some calculation
15:47:07 <pandy> How to forecast and measure metrics data storing on influxdb on certain interval of time ?
15:47:31 <dougsz> Can you not just look at the disk usage on the influxdb container volume?
15:48:19 <dougsz> We monitor it with Monasca, so you can view historical usage in Grafana, and the see a trend on usage.
15:48:28 <pandy> can i weigh size of metric ?
15:49:26 <dougsz> pass
15:50:20 <pandy> In grafana its collecting IO count & IO read write, don't think it helps to weigh, if mean am sorry and please explain
15:50:55 <dougsz> You should be able to get remaining disk space as well?
15:52:27 <pandy> Through metrics can we find disk space util of influxdb process ?
15:53:04 <pandy> I think we can measure the container disk util from host machine on period interval
15:53:18 <dougsz> indeed, cadvisor, or monasca agent should be able to hel
15:53:20 <dougsz> *help
15:53:57 <dougsz> It depends how you have configured storage for the influxdb docker container volume...
15:55:02 <pandy> Currently I am storing all influxdb container data on host machine and monitoring.
15:55:37 <witek> I agree with dougsz that measuring the actual size will be more accurate than calculation the timeseries size
15:56:38 <pandy> okay
15:56:59 <pandy> Thanks witek, dougsz :)
15:57:13 <witek> thank you guys
15:57:21 <witek> see you next time
15:57:26 <witek> #endmeeting