13:00:18 <witek> #startmeeting monasca
13:00:19 <openstack> Meeting started Tue Dec  3 13:00:18 2019 UTC and is due to finish in 60 minutes.  The chair is witek. Information about MeetBot at http://wiki.debian.org/MeetBot.
13:00:20 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
13:00:22 <openstack> The meeting name has been set to 'monasca'
13:00:31 <witek> hello everyone
13:00:56 <dougsz> hey all
13:01:03 <witek> hi Doug
13:02:12 <witek> the agenda is very light
13:02:15 <witek> https://etherpad.openstack.org/p/monasca-team-meeting-agenda
13:02:29 <witek> dougsz: do you have anything else to discuss today?
13:03:15 <dougsz> I started revamping the Prometheus plugin for Monasca Agent
13:04:02 <witek> please add to the agenda if applicable
13:04:10 <witek> let's start
13:04:20 <dougsz> yeah - probably not worth discussing in detail this week - will wait until more testing has been done
13:04:21 <witek> #topic logs CI job
13:04:42 <witek> small progress on logs CI job
13:04:54 <witek> I've added DEBUG logging level for ES
13:05:01 <witek> https://review.opendev.org/696889
13:05:23 <witek> and from what I understand the logs look OK
13:05:30 <dougsz> Good idea
13:06:04 <witek> it's just that the logs don't get posted to ES
13:07:01 <witek> so we have to investigate further which component earlier in the pipeline stops working correctly
13:07:39 <witek> I've confirmed already earlier that the logs get published to Kafka
13:08:01 <witek> so the next candidate would be logs-persister (Logstash) or Kafka itself
13:09:16 <witek> #topic monasca-agent Kafka plugin bug
13:09:27 <dougsz> We saw one issue with logstash output plugin not posting logs recently, but it turned out to be related to /etc/hosts config
13:09:39 <dougsz> so it's unlikely to be that, given it sometimes passes
13:10:11 <witek> you mean logstash-monasca output plugin?
13:10:50 <witek> logs-persister uses ES output plugin
13:10:51 <dougsz> sorry, logstash kafka output plugin - which I am not sure applied here
13:11:23 <witek> no
13:12:19 <witek> when looking at Kafka I've noticed it's not monitored correctly on DevStack
13:12:48 <witek> the agent's plugin need a refresh
13:12:55 <witek> Kafka plugin
13:13:12 <witek> I've put some notes together:
13:13:14 <witek> https://storyboard.openstack.org/#!/story/2006977
13:14:06 <dougsz> Thanks, I've added myself as a watcher
13:14:30 <dougsz> I figure there are going to be a lot of py3 related issues this cycle
13:14:56 <witek> yes, on the agent side most of all I think
13:15:47 <witek> #topic AOB
13:16:55 <witek> for Prometheus plugin, have you looked at DataDog plugin?
13:17:03 <witek> https://github.com/DataDog/integrations-core/tree/master/prometheus
13:17:43 <dougsz> Hmm, no, thanks for that
13:18:15 <witek> not sure if it helps, but may be worth checking
13:18:28 <dougsz> Yeah, looks like it's missing some of the bits I need
13:18:41 <dougsz> 1. Division of metrics for calculating percentages
13:19:06 <dougsz> 2. Converting counters to rates - looks like it can send deltas which isn't quite what I need
13:19:59 <witek> it would be nice to have these, indeed
13:20:04 <dougsz> I see there is some control for not scraping everything, that is 3) for me
13:20:27 <dougsz> I commonly see 1MB + scrapes on prometheus endpoints which is crazy
13:22:03 <brtknr> i have a question if i may after you dougsz
13:22:19 <dougsz> go for it brtknr
13:22:45 <brtknr> I was wondering if monasca-api supports metrics aggrgation at source
13:23:00 <brtknr> e.g. 30 min average
13:23:07 <brtknr> or daily average
13:23:20 <witek> no
13:23:40 <witek> we could extend the API, and send queries to InfluxDB
13:23:43 <brtknr> i know InfluxDB supports this but wasnt sure if monasca provides a route to this functionality
13:24:12 <witek> right, that would be easy to implement and very useful
13:24:22 <brtknr> is there demand for this feature?
13:24:24 <witek> the downside, we cannot use it for alerting
13:25:48 <witek> I think we should add this if we have developers who could and want to work on it
13:27:14 <dougsz> +1
13:27:51 <brtknr> dougsz: we should probably ask our customers if they'd like us to develop this feature
13:28:40 <dougsz> it won't hurt, but i'm not aware of any demand
13:29:42 <dougsz> I suppose I am slightly worried about tighter integration with InfluxDB
13:31:06 <brtknr> dougsz: are there no devs who want to implement cassandra backend?
13:31:40 <brtknr> dougsz: there are a few loose ends with timerange scoping too
13:32:14 <brtknr> which simply emit warning in the logs and carry on with previous behaviour
13:34:04 <witek> I wouldn't be worried about tighter integration with InfluxDB
13:34:39 <witek> some functionality (like avg, or rate) could be available on InfluxDB only
13:35:10 <dougsz> That's true. I am tempted to try TimescaleDB one day
13:35:50 <dougsz> If would at least be nice to know if it is any good before putting more effort into InfluxDB
13:37:15 <brtknr> Its significantly faster to query summary metrics vs every single data point
13:38:03 <brtknr> A quick test for this is to use the influxdb backend directly and use set intervals to calculate averages
13:38:12 <witek> brtknr: for averages you can use `metric-statistics`
13:38:15 <brtknr> from gravana
13:38:16 <witek> https://github.com/openstack/monasca-api/blob/master/docs/monasca-api-spec.md#get-v20metricsstatistics
13:38:30 <brtknr> from grafana
13:39:06 <brtknr> witek: cool
13:40:35 <witek> not sure how performant it is for longer time ranges
13:42:30 <witek> on another topic, we still have a number of open reviews
13:44:24 <witek> I just copied the links over from last week
13:44:52 <witek> anything else for today?
13:45:19 <witek> thanks for joining
13:45:23 <witek> and see you next week
13:45:32 <witek> bye
13:45:37 <witek> #endmeeting