| *** chaconpiza has quit IRC | 01:02 | |
| *** mbindlish has joined #openstack-monasca | 03:50 | |
| *** zijlboot has quit IRC | 06:17 | |
| *** nikparasyr has joined #openstack-monasca | 06:54 | |
| *** vishalmanchanda has joined #openstack-monasca | 07:33 | |
| *** zijlboot has joined #openstack-monasca | 07:45 | |
| *** zijlboot has quit IRC | 07:46 | |
| *** zijlboot_ has joined #openstack-monasca | 07:46 | |
| *** zijlboot has joined #openstack-monasca | 07:47 | |
| *** zijlboot_ has quit IRC | 07:51 | |
| *** witek has joined #openstack-monasca | 08:27 | |
| *** k_mouza has joined #openstack-monasca | 08:30 | |
| *** dougsz has joined #openstack-monasca | 08:38 | |
| nikparasyr | good morning. question: I have a monasca HA setup on 3 nodes. HAproxy runs on top of most services. influxdb is not enteprise and therefore replication is out of the question. I want to achieve the replication basically via kafka. I want to config the monasca-persister in a way so that every instance picks all metrics from the metrics topic on kafka and pushes it to its own influxdb instance. To achieve this I configured each | 09:12 |
|---|---|---|
| nikparasyr | monasca_persister config to: | 09:12 |
| nikparasyr | - to connect to its own influxdb instance, and not use the haproxy vip | 09:12 |
| nikparasyr | - to use a different group_id on the metrics topic | 09:12 |
| nikparasyr | I made these changes, flushed all influxdb databases and redeployed. Problem is I still see different metrics landing on each of the instances. Am I missing something? some configuration or something else. :S | 09:12 |
| dougsz | nikparasyr: Nice work - are you using Kolla by any chance? I believe you have the correct approach. Which release of Monasca Persister are you using? I am wondering about how the group id is passed to the python Kafka client (which was upgraded recently) | 09:36 |
| nikparasyr | yes im using kolla. stein. the changes i needed to do were this: https://pastebin.com/NMTCJ2Yw | 09:39 |
| nikparasyr | let me get the version | 09:39 |
| nikparasyr | monasca-persister --version | 09:40 |
| nikparasyr | Found deprecated old location "['/etc/monasca/persister.conf']" of main configuration file | 09:40 |
| nikparasyr | 1.14.0 | 09:40 |
| nikparasyr | I can also see on the monasca persister conf that it has quite some extra options under the [kafka] and [kafka_metrics] segments so maybe im missing something there. also i see that num_processors is set to 2 but unsure if this affects it. | 09:48 |
| witek | I think new Kafka client support was added in 1.15 | 09:51 |
| witek | but even the old one should support it | 09:51 |
| dougsz | Your config looks sensible nikparasyr assuming you have InfluxDB + persister on each node. | 09:52 |
| dougsz | I think Inspur has this working with the Monasca Docker containers, and their approach is very similar: | 09:52 |
| dougsz | https://review.opendev.org/#/c/665337 | 09:52 |
| dougsz | and https://review.opendev.org/#/c/664836 | 09:52 |
| nikparasyr | i do have persister and influxdb on each node. i'll upgrade to 1.15 just in case. Right now though i can get from kafka the group_ids that my config is passing | 09:55 |
| dougsz | that is a good sign | 09:55 |
| dougsz | A long shot, but if it is a dev env have you have tried wiping the kafka and zookeeper docker volumes and redeploying them? | 09:59 |
| nikparasyr | nope, i just wiped the influxdb monasca db, not kafka and zookeeper. i'll do that as well as for example i can still see in kafka the default group_id on the metrics topic before i made the changes. so flushing them is a good idea | 10:01 |
| dougsz | When you restart Monasca persister, in the logs do you see it connect to all partitions (assuming you have one persister in each consumer group)? | 10:01 |
| dougsz | Or does it split the partitions between other persisters in different consumer groups (it shouldn't) | 10:01 |
| nikparasyr | i'll check later and ill come back. having a meeting now | 10:06 |
| dougsz | thanks, very interested to hear if you get it working | 10:07 |
| *** k_mouza has quit IRC | 11:07 | |
| *** k_mouza has joined #openstack-monasca | 11:07 | |
| *** k_mouza has quit IRC | 12:58 | |
| *** jawad_axd has quit IRC | 13:18 | |
| *** jawad_axd has joined #openstack-monasca | 13:18 | |
| *** k_mouza has joined #openstack-monasca | 13:51 | |
| *** dougsz has quit IRC | 13:54 | |
| *** dougsz has joined #openstack-monasca | 14:09 | |
| nikparasyr | small update: monasca-persister logs showed that the partitions are split between the instances. I wiped kafka, zookeeper,influxdb docker volumes and redeployed, logs still show the same issue. Also another issue that popped up (which was also the first time but forgot to mention) is that the monasca db is created only on one of the 3 influxdb instances. This is due to: https://opendev.org/openstack/kolla-ansible/src/branch/st | 14:45 |
| nikparasyr | able/stein/ansible/roles/monasca/tasks/bootstrap.yml#L58. | 14:45 |
| nikparasyr | Another thing. checked the kolla containers for the monasca-persister. for stein the latest version of monasca-persister is 1.14. So, i haven' t upgraded to 1.15 yet. calling it a day. i check/try more stuff tomorrow. | 14:45 |
| *** nikparasyr has left #openstack-monasca | 14:46 | |
| *** mbindlish has quit IRC | 15:49 | |
| *** zijlboot has quit IRC | 16:05 | |
| *** witek has quit IRC | 16:12 | |
| *** dougsz has quit IRC | 16:31 | |
| *** k_mouza has quit IRC | 17:04 | |
| *** k_mouza has joined #openstack-monasca | 17:05 | |
| *** k_mouza has quit IRC | 17:10 | |
| *** vishalmanchanda has quit IRC | 18:17 | |
| *** irclogbot_2 has quit IRC | 18:34 | |
| *** zijlboot has joined #openstack-monasca | 18:34 | |
| *** irclogbot_0 has joined #openstack-monasca | 18:38 | |
| *** k_mouza has joined #openstack-monasca | 19:21 | |
| *** k_mouza has quit IRC | 19:26 | |
| *** k_mouza has joined #openstack-monasca | 21:48 | |
| *** k_mouza has quit IRC | 21:52 | |
| *** k_mouza has joined #openstack-monasca | 22:16 | |
| *** k_mouza has quit IRC | 22:21 | |
| *** k_mouza has joined #openstack-monasca | 22:36 | |
| *** k_mouza has quit IRC | 22:40 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!