| *** witek has joined #openstack-monasca | 08:02 | |
| *** dougsz has joined #openstack-monasca | 09:17 | |
| *** Chiara1234 has joined #openstack-monasca | 09:20 | |
| openstackgerrit | Witold Bedyk proposed openstack/monasca-persister master: Add configuration option influxdb.batch_size https://review.opendev.org/703563 | 09:21 |
|---|---|---|
| Chiara1234 | Hello, I'm trying to see on grafana dashboards the metrics' value, however when adding Monasca as a Datasource I'm not sure how to correctly configure it. I used Horizon as Authenticathion field (since I'm using devstack deploy with monasca-ui) but I receive an error message "Unautorized" | 09:38 |
| Chiara1234 | Also, in the http setting, in the url field, should I use the metrics url? | 09:39 |
| witek | Chiara1234: in Devstack we use the forked version of Grafana | 09:53 |
| witek | it syncs it's user database with Keystone | 09:53 |
| witek | so you can log in as mini-mon:password and the datasource is already pre-configured | 09:54 |
| Chiara1234 | I see! I was using the wrong credentials then! Thanks again :D | 09:55 |
| *** ivve has joined #openstack-monasca | 10:07 | |
| openstackgerrit | Witold Bedyk proposed openstack/monasca-persister master: Add configuration option influxdb.batch_size https://review.opendev.org/703563 | 10:08 |
| *** Chiara1234 has quit IRC | 10:12 | |
| *** openstackgerrit has quit IRC | 10:12 | |
| *** witek has quit IRC | 10:25 | |
| *** witek has joined #openstack-monasca | 10:43 | |
| *** openstackgerrit has joined #openstack-monasca | 10:49 | |
| openstackgerrit | Witold Bedyk proposed openstack/monasca-api master: Allow users to set periodic notifications on all notification types https://review.opendev.org/693863 | 10:49 |
| *** Wasaac has quit IRC | 11:32 | |
| *** Wasaac has joined #openstack-monasca | 11:34 | |
| openstackgerrit | Witold Bedyk proposed openstack/monasca-tempest-plugin master: Set tempest tests for logs as voting again https://review.opendev.org/703593 | 11:58 |
| *** Wasaac has quit IRC | 12:12 | |
| *** Wasaac has joined #openstack-monasca | 12:12 | |
| *** witek has quit IRC | 12:25 | |
| *** witek has joined #openstack-monasca | 12:40 | |
| witek | Courtesy Monasca meeting reminder in 4 minutes in #openstack-monasca: witek, jayahn,iurygregory,ezpz,igorn,haad,sc,joadavis, akiraY,tobiajo,dougsz,fouadben, amofakhar, haru5ny (haruki),kaiokmo,pandiyan,guilhermesp,chaconpiza,hosanai,Wasaac,brtknr | 12:56 |
| witek | #startmeeting monasca | 13:00 |
| openstack | Meeting started Tue Jan 21 13:00:26 2020 UTC and is due to finish in 60 minutes. The chair is witek. Information about MeetBot at http://wiki.debian.org/MeetBot. | 13:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 13:00 |
| *** openstack changes topic to " (Meeting topic: monasca)" | 13:00 | |
| openstack | The meeting name has been set to 'monasca' | 13:00 |
| witek | hello everyone | 13:00 |
| chaconpiza | hi | 13:00 |
| witek | hi Martin | 13:00 |
| witek | the agenda for today consists of two points as of now: | 13:01 |
| witek | https://etherpad.openstack.org/p/monasca-team-meeting-agenda | 13:01 |
| witek | feel free to add new items | 13:01 |
| witek | #topic persister performance optimization | 13:02 |
| *** openstack changes topic to "persister performance optimization (Meeting topic: monasca)" | 13:02 | |
| dougsz | hi all | 13:02 |
| witek | hi Doug | 13:02 |
| witek | I've stumbled upon a blog article about InfluxDB Python client | 13:02 |
| witek | https://www.influxdata.com/blog/writing-data-to-influxdb-with-python/ | 13:03 |
| witek | one interesting thing they mention, is the recommended batch size for sending datapoints to InfluxDB | 13:03 |
| witek | the recommended value is 5000 to 10000 data points per request | 13:04 |
| *** bandorf has joined #openstack-monasca | 13:05 | |
| witek | the client offers an argument to split larger requests to batches of a given size | 13:05 |
| witek | and so I proposed a patch to allow configuring this value in monasca-persister as well | 13:06 |
| witek | https://review.opendev.org/703563 | 13:06 |
| *** adriancz has joined #openstack-monasca | 13:07 | |
| witek | I have to admit I haven't measured performance impact | 13:08 |
| witek | any questions or comments on that? | 13:09 |
| witek | if not, we can move on | 13:09 |
| chaconpiza | We have in devstack: | 13:10 |
| chaconpiza | [kafka_metrics] | 13:10 |
| chaconpiza | batch_size = 30 | 13:10 |
| bandorf | We can execute some system tests, but it will take some time | 13:10 |
| chaconpiza | then it would change to 10000? | 13:10 |
| * witek checking Kafka settings | 13:10 | |
| chaconpiza | I found it in /etc/monasca-persister/monasca-persister.conf | 13:11 |
| bandorf | I'm a bit confused: | 13:11 |
| bandorf | According to Martin, this value has been configured already now. witek: You wrote that you did a change to allow config. of this vale | 13:12 |
| witek | right, the value which Martin mentions configures the batch size consumed from Kafka | 13:13 |
| witek | so, effectively, my change won't have any impact | 13:14 |
| dougsz | `# Maximum number of metrics to buffer before writing to database (integer value) | 13:14 |
| dougsz | #batch_size = 20000 | 13:14 |
| dougsz | ` | 13:14 |
| dougsz | That makes sense - I recall seeing that limit applied in the logs | 13:14 |
| witek | I think adding InfluxDB option doesn't hurt though | 13:16 |
| witek | I should probably document that also Kafka config affects the actual batch size | 13:16 |
| witek | thanks for spotting this chaconpiza | 13:17 |
| witek | batch_size = 30 seems somewhat low though, in terms of InfluxDB performance | 13:19 |
| chaconpiza | and for other topics the batch_size is = 1 | 13:19 |
| witek | but only metrics topic gets persisted to InfluxDB | 13:20 |
| chaconpiza | I am wondering if [kafka_alarm_history] goes to influxDB as well | 13:21 |
| witek | it does, but is more or less negligible in terms of performance | 13:22 |
| chaconpiza | with topic = alarm-state-transitions | 13:22 |
| chaconpiza | ok | 13:22 |
| bandorf | Regarding the batch size for metrics: We can execute some system tests, with different values for the batch size | 13:23 |
| witek | that would be nice in any case | 13:24 |
| witek | I think the reason for choosing low Kafka batch size was to reduce delays when system is under low load | 13:25 |
| dougsz | Doesn't it do a periodic flush as well? | 13:26 |
| dougsz | At least it does for metrics | 13:26 |
| bandorf | Yes, that's what I have in mind as well. | 13:26 |
| witek | you're right again, we have `max_wait_time_seconds` | 13:28 |
| dougsz | I'm just thinking about the Kafka batch_size setting for metrics - I suppose that controls how much data is pulled from typically a spinning disk into RAM from Kafka, and generally you might want that to be a large chunk | 13:28 |
| witek | yes, and having an option to control this independently to what is written to InfluxDB offers more flexibility for performance optimization | 13:29 |
| dougsz | Yeah - perhaps 20000 is on the small side for that. Benchmarking will be interesting | 13:30 |
| bandorf | I assume, currently the batch size "read from kafka" is the batch size used for sending to influxdb? | 13:31 |
| dougsz | yeah, I believe so | 13:31 |
| witek | bandorf: correct | 13:31 |
| witek | playing with these two parameters would be really interesting | 13:32 |
| bandorf | Then, we can relatively easy perform a test with different values for that batch size. | 13:32 |
| witek | bandorf: that would be great, thanks | 13:33 |
| bandorf | I need to check with our system test team when we can execute such tests. | 13:33 |
| dougsz | yeah - I've found influxdb to very much not be the bottle neck until I have 10s of cores dedicated to persister processes | 13:33 |
| dougsz | thanks bandorf | 13:33 |
| witek | nice, thanks for this interesting discussion | 13:34 |
| witek | I think we can move on | 13:34 |
| witek | #topic monasca-log-api repo maintanance | 13:34 |
| *** openstack changes topic to "monasca-log-api repo maintanance (Meeting topic: monasca)" | 13:34 | |
| witek | I'm wondering how we should proceed with maintaining the monasca-log-api repo after we've merged all the code to monasca-api | 13:35 |
| witek | theoretically we could still have cases when we want to provide bugfixes | 13:36 |
| witek | should we drop DevStack as currently proposed and propose bugfixes to stable branches if needed? | 13:37 |
| dougsz | When do we stop supporting it? This cycle? | 13:37 |
| witek | might be to quick, right? I think we can mark it as deprecated | 13:38 |
| dougsz | yeah, and remove in Victoria perhaps? | 13:38 |
| dougsz | So this cycle we still need to target bug fix patches at master | 13:39 |
| dougsz | If we drop devstack support, will that mean we can no longer run tempest against it? | 13:40 |
| witek | the new job is configured with monasca-api DevStack plugin, so it doesn't test monasca-log-api repo | 13:41 |
| witek | yes, I think it's the right way to keep all the old content and CI setup and remove in Victoria or following cycle | 13:43 |
| witek | for now, just mark the repo as deprecated | 13:43 |
| witek | chaconpiza: adriancz: what do you think? | 13:43 |
| adriancz | yes, i think we should mark this repo as deprecated | 13:44 |
| chaconpiza | I agree | 13:44 |
| adriancz | and remove this in next cycle | 13:45 |
| witek | agreed, thanks | 13:45 |
| dougsz | yeah + 1 to remove asap, we don't want the burden | 13:45 |
| bandorf | I agree as well | 13:45 |
| dougsz | I will switch Kolla this cycle | 13:45 |
| witek | #topic AOB | 13:45 |
| *** openstack changes topic to "AOB (Meeting topic: monasca)" | 13:45 | |
| witek | anything else for today? | 13:46 |
| witek | just a notice about self-healing and auto-scaling SIGs joint meeting in 12 minutes | 13:47 |
| witek | in #openstack-meeting | 13:47 |
| dougsz | thanks - I will lurk | 13:48 |
| witek | thanks for joining today and for interesting discussions | 13:48 |
| dougsz | +1 | 13:48 |
| witek | see you next week | 13:48 |
| witek | bye bye | 13:48 |
| chaconpiza | thanks bye | 13:48 |
| bandorf | Thanks and good-bye | 13:48 |
| dougsz | bye | 13:49 |
| witek | #endmeeting | 13:49 |
| *** openstack changes topic to "OpenStack Monitoring as a Service | https://wiki.openstack.org/wiki/Monasca" | 13:49 | |
| openstack | Meeting ended Tue Jan 21 13:49:03 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 13:49 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/monasca/2020/monasca.2020-01-21-13.00.html | 13:49 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/monasca/2020/monasca.2020-01-21-13.00.txt | 13:49 |
| adriancz | thanks bye | 13:49 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/monasca/2020/monasca.2020-01-21-13.00.log.html | 13:49 |
| *** chaconpiza has quit IRC | 15:06 | |
| openstackgerrit | Arseni Lipinski proposed openstack/monasca-kibana-plugin master: [WIP] Upgrade monasca-kibana-plugin for Elkstack update https://review.opendev.org/680075 | 15:19 |
| *** bandorf has quit IRC | 16:12 | |
| *** openstackgerrit has quit IRC | 16:13 | |
| *** arseni-lipinski- has joined #openstack-monasca | 16:36 | |
| *** dougsz has quit IRC | 17:57 | |
| *** openstackgerrit has joined #openstack-monasca | 17:59 | |
| openstackgerrit | Arseni Lipinski proposed openstack/monasca-api master: [WIP] Added build and installation for monasca-kibana-plugin in new API https://review.opendev.org/703661 | 17:59 |
| openstackgerrit | Arseni Lipinski proposed openstack/monasca-api master: [WIP] Add build and installation for monasca-kibana-plugin in new API https://review.opendev.org/703661 | 18:05 |
| *** witek has quit IRC | 18:07 | |
| openstackgerrit | Arseni Lipinski proposed openstack/monasca-kibana-plugin master: [WIP] Upgrade monasca-kibana-plugin for Elkstack update https://review.opendev.org/680075 | 18:10 |
| *** arseni-lipinski- has quit IRC | 18:56 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!