| openstackgerrit | inspurericzhang proposed openstack/monasca-tempest-plugin master: Update hacking version to latest https://review.openstack.org/629353 | 02:15 |
|---|---|---|
| *** bobh has joined #openstack-monasca | 02:41 | |
| *** bobh has quit IRC | 02:47 | |
| *** spsurya has joined #openstack-monasca | 03:46 | |
| *** jamesgu__ has joined #openstack-monasca | 04:52 | |
| *** jamesgu has quit IRC | 05:02 | |
| *** chaconpiza has quit IRC | 05:02 | |
| *** pandy has joined #openstack-monasca | 05:21 | |
| pandy | Hi All, I would like to monitor health status of monasca components like monasca-api, kafka, zookeeper,mysql,influxdb,notification.. Currently I have installed monasca-agent on host machine, where my monasca components are running in docker container on same host. May i know how to monitor from host monasca-agent ? | 05:24 |
| *** pcaruana has joined #openstack-monasca | 07:06 | |
| *** bobh has joined #openstack-monasca | 07:39 | |
| *** bobh has quit IRC | 07:43 | |
| *** chaconpiza has joined #openstack-monasca | 08:04 | |
| *** witek has joined #openstack-monasca | 08:24 | |
| dirk | witek: ^^ | 08:38 |
| openstackgerrit | Adrian Czarnecki proposed openstack/monasca-api master: Use PYTHON2.7 for npm https://review.openstack.org/628194 | 08:52 |
| openstackgerrit | Adrian Czarnecki proposed openstack/monasca-api master: Use PYTHON2.7 for npm https://review.openstack.org/628194 | 09:04 |
| *** dougsz has joined #openstack-monasca | 09:13 | |
| *** bobh has joined #openstack-monasca | 09:25 | |
| pandy | witek, I would like to know to conver tenant_id to tenant name in monasca agent toread user friendly for end user to understand. Currently can do group by "tenant_id" may i know the file changes need to do for collecting with "tenant" name | 09:44 |
| pandy | https://storyboard.openstack.org/#!/story/2004739 | 09:59 |
| *** bobh has quit IRC | 10:19 | |
| dougsz | pandy: Have you seen the Monasca Agent optional dimensions? https://github.com/openstack/monasca-agent/blob/master/docs/Plugins.md#dimensions | 10:20 |
| dougsz | Could you add a tenant name as an optional dimension? The tenant name would then be added to all metrics harvested by that instance of the Monasca Agent. | 10:21 |
| dougsz | Maybe that would work for your use case? | 10:21 |
| toabctl | dougsz, question about monasca-setup - the use-case for monasca-setup is, to setup the different monasca-agent plugins and write the configurations into /etc/monasca/agent/conf.d . Is that correct? | 10:29 |
| toabctl | I'm asking because I try to monasca-setup with the process detection. but monasca-setup needs the process to be *running* when I run it. Otherwise, the process will not be added to the conf | 10:30 |
| toabctl | this sounds really wrong to me because I want to *monitor* processes (also the procs that are currently not available) | 10:31 |
| dougsz | toabctl: yeah, that's my understanding of it. Personally I statically configure it by just placing config files for the plugins I want in the config directory. | 10:32 |
| dougsz | (although there are some cases when I still use the detection mechanism, but it doesn't always work when things are running in Docker containers) | 10:33 |
| dougsz | for example.. | 10:33 |
| pandy | toabctl, First you need config files ( similar like having new plugin), where monasca-setup will detect plugins ( inside conf.d/) and do changes to configuration. when there is no plugin definitely monasca-setup won't do needful | 10:34 |
| pandy | By default monasca-agent will provide some plugins (inside conf.d/) those can be setup using monasca-setup, else add required plugins (https://github.com/openstack/monasca-agent/blob/master/docs/Plugins.md) and use monasca-setup | 10:35 |
| *** bobh has joined #openstack-monasca | 12:18 | |
| *** bobh has quit IRC | 12:24 | |
| openstackgerrit | Michal Arbet proposed openstack/monasca-common master: Fix python3.7 compatibility https://review.openstack.org/629453 | 12:32 |
| openstackgerrit | Merged openstack/monasca-api master: Use PYTHON2.7 for npm https://review.openstack.org/628194 | 12:49 |
| openstackgerrit | Merged openstack/monasca-log-api master: Add python3 tempest tests https://review.openstack.org/624311 | 12:49 |
| toabctl | dougsz, do you want me to change anything else for https://review.openstack.org/#/c/628931/ ? (just asking because there is no +2 from you) | 13:25 |
| dougsz | toabctl: I'm happy with it, just wanted to do a quick test before +2. Thanks for the patches. | 13:27 |
| witek | Courtesy Monasca meeting reminder in #openstack-monasca: witek, jayahn,iurygregory,ezpz,igorn,haad,sc,joadavis, akiraY,tobiajo,dougsz_,fouadben, amofakhar, aagate, haruki,kaiokmo,pandiyan,charana,guilhermesp,chaconpiza | 14:59 |
| witek | #startmeeting monasca | 15:00 |
| openstack | Meeting started Wed Jan 9 15:00:56 2019 UTC and is due to finish in 60 minutes. The chair is witek. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| *** openstack changes topic to " (Meeting topic: monasca)" | 15:00 | |
| openstack | The meeting name has been set to 'monasca' | 15:01 |
| witek | hello everyone | 15:01 |
| kaiokmo | o/ | 15:01 |
| witek | hi kaiokmo | 15:01 |
| kaiokmo | hello witek | 15:01 |
| kaiokmo | hope you enjoyed holidays | 15:01 |
| dougsz | hey all | 15:01 |
| witek | hi dougsz | 15:01 |
| witek | yes, thanks :) | 15:02 |
| pandy | hi witek & all, after long time, nice to join weekly meeting | 15:02 |
| witek | agenda for today https://etherpad.openstack.org/p/monasca-team-meeting-agenda | 15:02 |
| chaconpiza | Hi | 15:03 |
| witek | #topic CFP for OpenStack Summit | 15:03 |
| *** openstack changes topic to "CFP for OpenStack Summit (Meeting topic: monasca)" | 15:03 | |
| witek | just the reminder | 15:03 |
| witek | the deadline is in two weeks | 15:03 |
| pandy | witek, Sure this time am looking forward to submit presentation with challenges in deploying in swarm mode and handling in scale setup. Will send abstract to you in personal before submitting for review | 15:04 |
| witek | sounds good | 15:05 |
| witek | I encourage everyone to submit | 15:05 |
| witek | presentation or workshop | 15:05 |
| witek | as I said, deadline is January 23 | 15:07 |
| pandy | okay :) | 15:07 |
| witek | #topic Operation Query | 15:07 |
| *** openstack changes topic to "Operation Query (Meeting topic: monasca)" | 15:07 | |
| pandy | Hi witek, seems it mine | 15:07 |
| pandy | yes, I would like to convey in short about our setup again to all here, we have deployed docker-monasca in docker swarm in dedicated baremetals | 15:08 |
| pandy | where monasca-agent were installed on 500+ compute nodes in virtualenv | 15:09 |
| pandy | Now we would like to collect metrics and group them by tenant name | 15:09 |
| pandy | how to achieve that ? | 15:09 |
| witek | we don't have API call for that | 15:10 |
| witek | probably it would be easiest to query the TSDB directly | 15:11 |
| pandy | I request you to explain in detail for my easy understanding | 15:11 |
| witek | alternatively you could write a script that would iterate through projects and list metrics for each | 15:12 |
| pandy | okay | 15:13 |
| pandy | I was looking using API call, fine i try with scripting. | 15:14 |
| pandy | Also felt discrepancy while monitoring http_checks of monasca-components like kafka, monasca-api health status | 15:15 |
| pandy | Example: If kafka died or some error. Monasca-agent getting status after sometime like 5 minutes in average | 15:15 |
| pandy | If status comes up, that also getting delayed to update with monasca-agent components. | 15:16 |
| witek | what is you collection period? | 15:16 |
| pandy | 30 sec | 15:16 |
| pandy | I feel the response time is aggregrated value. | 15:17 |
| pandy | Quick workaround, if i restart monasca-agent can reduce response time | 15:18 |
| dougsz | Hmm, well, if a node in a Kafka cluster dies, won't that affect throughput? | 15:18 |
| pandy | Yes it affect, assume kafka not died and failing with kafka-consumer and producer or async | 15:19 |
| pandy | I am running kafka in docker swarm, each worker node having one kafka & zookeeper | 15:20 |
| pandy | Based on heavy hit some interval time need to increase kafka topics values and some other values. But looking short time response by monasca-agent | 15:21 |
| witek | do you query the http_check metric values, or did you set an alarm on that? | 15:22 |
| dougsz | Do you have enough partitions per topic to use all Kafka nodes in parallel? | 15:23 |
| pandy | witek, Here is my docker-file for monasca-agent collector and forwarder along with caadvisor http://paste.openstack.org/show/740743/ | 15:24 |
| witek | how do you measure the delay? | 15:25 |
| pandy | Through Grafana dashboard kafka.json | 15:26 |
| pandy | witek, whether my docker file environment variables are correct ? | 15:28 |
| witek | didn't see anything suspect | 15:28 |
| pandy | okay | 15:29 |
| witek | don't have an explanation | 15:30 |
| witek | you could examine API logs to see | 15:30 |
| witek | where the delay is created | 15:30 |
| dougsz | +1 for analysing delays across the pipeline | 15:31 |
| pandy | You mean monasca-api | 15:31 |
| witek | yes | 15:31 |
| witek | you can start in agent, then api and persister | 15:31 |
| pandy | I can query instance metrics with instant response, hope it was done through mon-api and can make sure that responce of mon-api (depend kafka) is fine. | 15:32 |
| pandy | if i query metrics for instance for last 10 seconds working fine, it mean kafka and other mon-api dependent service are fine. But monasca-agent forwarder logs says no response from Mon-api | 15:34 |
| pandy | after that 5 min interval logs are getting response. | 15:34 |
| pandy | Even i am feeling its not expected one. I will try. | 15:34 |
| dougsz | Network saturated between the forwarder and the API? | 15:36 |
| pandy | dougsz, i feel your answer right | 15:38 |
| pandy | somewhere read that while running application on docker swarm mode network issues happens | 15:39 |
| witek | do we have anything else? | 15:39 |
| dougsz | toabctl pushed some patches for changing the monasca api config file name to align with other openstack projects | 15:40 |
| dougsz | https://review.openstack.org/#/c/628931/ | 15:40 |
| witek | dougsz: did you have chance to test it? | 15:40 |
| dougsz | Not yet, hopefully in the next day or two. | 15:41 |
| witek | thanks | 15:41 |
| witek | I'm still configuring things up in my new dev env | 15:41 |
| dougsz | :) | 15:41 |
| witek | I rechecked https://review.openstack.org/435136 | 15:41 |
| witek | and noticed that it's not tested in devstack | 15:42 |
| witek | the Zuul job name changed and hasn't been updated | 15:42 |
| witek | will update, and we should merge it | 15:43 |
| dougsz | thanks witek - will be nice to make some progress on that patch chain | 15:43 |
| witek | yes, it hangs too long | 15:43 |
| witek | if there's nothing else, I'll be wrapping up | 15:44 |
| pandy | witek, dougsz i have addition question | 15:44 |
| witek | go ahead | 15:44 |
| pandy | i want to store metrics in influxdb for 30 days or some days. But i want to forecast capacity with InfluxDB with some calculation | 15:45 |
| pandy | How to forecast and measure metrics data storing on influxdb on certain interval of time ? | 15:47 |
| dougsz | Can you not just look at the disk usage on the influxdb container volume? | 15:47 |
| dougsz | We monitor it with Monasca, so you can view historical usage in Grafana, and the see a trend on usage. | 15:48 |
| pandy | can i weigh size of metric ? | 15:48 |
| dougsz | pass | 15:49 |
| pandy | In grafana its collecting IO count & IO read write, don't think it helps to weigh, if mean am sorry and please explain | 15:50 |
| dougsz | You should be able to get remaining disk space as well? | 15:50 |
| pandy | Through metrics can we find disk space util of influxdb process ? | 15:52 |
| pandy | I think we can measure the container disk util from host machine on period interval | 15:53 |
| dougsz | indeed, cadvisor, or monasca agent should be able to hel | 15:53 |
| dougsz | *help | 15:53 |
| dougsz | It depends how you have configured storage for the influxdb docker container volume... | 15:53 |
| pandy | Currently I am storing all influxdb container data on host machine and monitoring. | 15:55 |
| witek | I agree with dougsz that measuring the actual size will be more accurate than calculation the timeseries size | 15:55 |
| pandy | okay | 15:56 |
| pandy | Thanks witek, dougsz :) | 15:56 |
| witek | thank you guys | 15:57 |
| witek | see you next time | 15:57 |
| witek | #endmeeting | 15:57 |
| *** openstack changes topic to "OpenStack Monitoring as a Service | https://wiki.openstack.org/wiki/Monasca" | 15:57 | |
| openstack | Meeting ended Wed Jan 9 15:57:26 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:57 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/monasca/2019/monasca.2019-01-09-15.00.html | 15:57 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/monasca/2019/monasca.2019-01-09-15.00.txt | 15:57 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/monasca/2019/monasca.2019-01-09-15.00.log.html | 15:57 |
| *** chaconpiza has quit IRC | 15:58 | |
| *** pcaruana has quit IRC | 16:03 | |
| *** pandy has quit IRC | 16:29 | |
| openstackgerrit | Witold Bedyk proposed openstack/monasca-notification master: Update Zuul tempest test job names https://review.openstack.org/629564 | 16:38 |
| witek | dougsz: can we rebase https://review.openstack.org/435136 on ^^^ | 16:40 |
| witek | ? | 16:40 |
| dougsz | sure, will do | 16:41 |
| openstackgerrit | Witold Bedyk proposed openstack/monasca-notification master: Make default notifiers pluggable https://review.openstack.org/435136 | 16:42 |
| witek | done | 16:42 |
| dougsz | thanks, you beat me to it :) | 16:43 |
| *** pandy has joined #openstack-monasca | 16:59 | |
| *** Dobroslaw has quit IRC | 17:22 | |
| *** witek has quit IRC | 17:33 | |
| openstackgerrit | 98k proposed openstack/python-monascaclient master: Add doc/requirements.txt to releasenotes tox environment https://review.openstack.org/629592 | 17:35 |
| *** witek has joined #openstack-monasca | 17:45 | |
| *** dougsz has quit IRC | 17:54 | |
| *** witek has quit IRC | 18:03 | |
| *** witek has joined #openstack-monasca | 18:06 | |
| *** witek has quit IRC | 18:33 | |
| *** pandy has quit IRC | 19:29 | |
| *** witek has joined #openstack-monasca | 21:13 | |
| *** witek has quit IRC | 21:24 | |
| openstackgerrit | Merged openstack/monasca-api master: Remove built-in plugins from database schema https://review.openstack.org/603648 | 22:12 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!