13:00:26 <witek> #startmeeting monasca
13:00:27 <openstack> Meeting started Tue Jan 21 13:00:26 2020 UTC and is due to finish in 60 minutes.  The chair is witek. Information about MeetBot at http://wiki.debian.org/MeetBot.
13:00:28 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
13:00:30 <openstack> The meeting name has been set to 'monasca'
13:00:43 <witek> hello everyone
13:00:47 <chaconpiza> hi
13:00:54 <witek> hi Martin
13:01:28 <witek> the agenda for today consists of two points as of now:
13:01:33 <witek> https://etherpad.openstack.org/p/monasca-team-meeting-agenda
13:01:47 <witek> feel free to add new items
13:02:07 <witek> #topic persister performance optimization
13:02:31 <dougsz> hi all
13:02:50 <witek> hi Doug
13:02:54 <witek> I've stumbled upon a blog article about InfluxDB Python client
13:03:03 <witek> https://www.influxdata.com/blog/writing-data-to-influxdb-with-python/
13:03:48 <witek> one interesting thing they mention, is the recommended batch size for sending datapoints to InfluxDB
13:04:36 <witek> the recommended value is 5000 to 10000 data points per request
13:05:36 <witek> the client offers an argument to split larger requests to batches of a given size
13:06:23 <witek> and so I proposed a patch to allow configuring this value in monasca-persister as well
13:06:30 <witek> https://review.opendev.org/703563
13:08:05 <witek> I have to admit I haven't measured performance impact
13:09:14 <witek> any questions or comments on that?
13:09:59 <witek> if not, we can move on
13:10:07 <chaconpiza> We have in devstack:
13:10:09 <chaconpiza> [kafka_metrics]
13:10:09 <chaconpiza> batch_size = 30
13:10:14 <bandorf> We can execute some system tests, but it will take some time
13:10:26 <chaconpiza> then it would change to 10000?
13:10:46 * witek checking Kafka settings
13:11:16 <chaconpiza> I found it in /etc/monasca-persister/monasca-persister.conf
13:11:55 <bandorf> I'm a bit confused:
13:12:35 <bandorf> According to Martin, this value has been configured already now. witek: You wrote that you did a change to allow config. of this vale
13:13:45 <witek> right, the value which Martin mentions configures the batch size consumed from Kafka
13:14:04 <witek> so, effectively, my change won't have any impact
13:14:13 <dougsz> `# Maximum number of metrics to buffer before writing to database (integer value)
13:14:13 <dougsz> #batch_size = 20000
13:14:13 <dougsz> `
13:14:32 <dougsz> That makes sense - I recall seeing that limit applied in the logs
13:16:18 <witek> I think adding InfluxDB option doesn't hurt though
13:16:44 <witek> I should probably document that also Kafka config affects the actual batch size
13:17:29 <witek> thanks for spotting this chaconpiza
13:19:21 <witek> batch_size = 30 seems somewhat low though, in terms of InfluxDB performance
13:19:53 <chaconpiza> and for other topics the batch_size is = 1
13:20:25 <witek> but only metrics topic gets persisted to InfluxDB
13:21:51 <chaconpiza> I am wondering if [kafka_alarm_history] goes to influxDB as well
13:22:32 <witek> it does, but is more or less negligible in terms of performance
13:22:32 <chaconpiza> with topic = alarm-state-transitions
13:22:39 <chaconpiza> ok
13:23:30 <bandorf> Regarding the batch size for metrics: We can execute some system tests, with different values for the batch size
13:24:03 <witek> that would be nice in any case
13:25:09 <witek> I think the reason for choosing low Kafka batch size was to reduce delays when system is under low load
13:26:04 <dougsz> Doesn't it do a periodic flush as well?
13:26:13 <dougsz> At least it does for metrics
13:26:27 <bandorf> Yes, that's what I have in mind as well.
13:28:29 <witek> you're right again, we have `max_wait_time_seconds`
13:28:36 <dougsz> I'm just thinking about the Kafka batch_size setting for metrics -  I suppose that controls how much data is pulled from typically a spinning disk into RAM from Kafka, and generally you might want that to be a large chunk
13:29:51 <witek> yes, and having an option to control this independently to what is written to InfluxDB offers more flexibility for performance optimization
13:30:14 <dougsz> Yeah - perhaps 20000 is on the small side for that. Benchmarking will be interesting
13:31:18 <bandorf> I assume, currently the batch size "read from kafka" is the batch size used for sending to influxdb?
13:31:32 <dougsz> yeah, I believe so
13:31:36 <witek> bandorf: correct
13:32:10 <witek> playing with these two parameters would be really interesting
13:32:45 <bandorf> Then, we can relatively easy perform a test with different values for that batch size.
13:33:04 <witek> bandorf: that would be great, thanks
13:33:05 <bandorf> I need to check with our system test team when we can execute such tests.
13:33:20 <dougsz> yeah - I've found influxdb to very much not be the bottle neck until I have 10s of cores dedicated to persister processes
13:33:49 <dougsz> thanks bandorf
13:34:15 <witek> nice, thanks for this interesting discussion
13:34:21 <witek> I think we can move on
13:34:45 <witek> #topic monasca-log-api repo maintanance
13:35:35 <witek> I'm wondering how we should proceed with maintaining the monasca-log-api repo after we've merged all the code to monasca-api
13:36:03 <witek> theoretically we could still have cases when we want to provide bugfixes
13:37:03 <witek> should we drop DevStack as currently proposed and propose bugfixes to stable branches if needed?
13:37:17 <dougsz> When do we stop supporting it? This cycle?
13:38:01 <witek> might be to quick, right? I think we can mark it as deprecated
13:38:37 <dougsz> yeah, and remove in Victoria perhaps?
13:39:26 <dougsz> So this cycle we still need to target bug fix patches at master
13:40:01 <dougsz> If we drop devstack support, will that mean we can no longer run tempest against it?
13:41:05 <witek> the new job is configured with monasca-api DevStack plugin, so it doesn't test monasca-log-api repo
13:43:19 <witek> yes, I think it's the right way to keep all the old content and CI setup and remove in Victoria or following cycle
13:43:35 <witek> for now, just mark the repo as deprecated
13:43:56 <witek> chaconpiza: adriancz: what do you think?
13:44:38 <adriancz> yes, i think we should mark this repo as deprecated
13:44:58 <chaconpiza> I agree
13:45:01 <adriancz> and remove this in next cycle
13:45:11 <witek> agreed, thanks
13:45:31 <dougsz> yeah + 1 to remove asap, we don't want the burden
13:45:43 <bandorf> I agree as well
13:45:44 <dougsz> I will switch Kolla this cycle
13:45:57 <witek> #topic AOB
13:46:17 <witek> anything else for today?
13:47:31 <witek> just a notice about self-healing and auto-scaling SIGs joint meeting in 12 minutes
13:47:48 <witek> in #openstack-meeting
13:48:39 <dougsz> thanks - I will lurk
13:48:41 <witek> thanks for joining today and for interesting discussions
13:48:46 <dougsz> +1
13:48:50 <witek> see you next week
13:48:53 <witek> bye bye
13:48:56 <chaconpiza> thanks bye
13:48:56 <bandorf> Thanks and good-bye
13:49:01 <dougsz> bye
13:49:03 <witek> #endmeeting