15:00:33 <rhochmuth> #startmeeting monasca
15:00:34 <zhipengh> c y'all
15:00:36 <openstack> Meeting started Wed Feb  1 15:00:33 2017 UTC and is due to finish in 60 minutes.  The chair is rhochmuth. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:37 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:40 <openstack> The meeting name has been set to 'monasca'
15:00:44 <rhochmuth> https://etherpad.openstack.org/p/monasca-team-meeting-agenda
15:00:54 <rhochmuth> Agenda for Wednesday February 1 2017 (15:00 UTC)
15:00:54 <rhochmuth> 1.	What were the most important features added in the Ocata release? In response to Nick Chase, Editor in Chief, OpenStack:Unlocked.
15:00:54 <rhochmuth> 2.	Review new blueprints: Lot's of exciting new developments in progress.
15:00:54 <rhochmuth> 1.	Templated alarms: https://blueprints.launchpad.net/monasca/+spec/templated-alarms
15:00:54 <rhochmuth> 2.	Monasca sidecar: https://blueprints.launchpad.net/monasca/+spec/monasca-sidecar
15:00:54 <rhochmuth> 3.	Prometheus instrumentation: https://blueprints.launchpad.net/monasca/+spec/prometheus-instrumentation
15:00:55 <rhochmuth> 4.	Mapper: https://review.openstack.org/#/c/420774/
15:00:55 <rhochmuth> 5.	Log Query API: https://blueprints.launchpad.net/monasca/+spec/log-query-api
15:00:56 <rhochmuth> 6.	Alarm inhibition/silencing
15:00:56 <rhochmuth> 7.	Alarm grouping
15:00:57 <rhochmuth> 8.	Monasca Query Language (MQL)
15:00:57 <rhochmuth> 3.	Reviews
15:00:58 <rhochmuth> 1.	https://review.openstack.org/#/c/427571/
15:00:58 <rhochmuth> 1.	FYI: https://review.openstack.org/#/c/425152/
15:01:22 <bklei> o\
15:01:24 <rbak__> o/
15:01:29 <koji> o/
15:01:31 <hosanai> o/
15:01:38 <rhochmuth> hi everyone
15:01:40 <shinya_kwbt> o/
15:01:43 <witek> hi
15:01:47 <rhochmuth> looks like a good agenda to cover this week
15:01:55 <rhochmuth> exciting times are ahead
15:02:17 <notq> hi
15:02:18 <rhochmuth> witek: are there any items you need to cover on the ocata release?
15:02:33 <witek> stable branch comes soon
15:03:02 <rhochmuth> it sounds like we are in good shape
15:03:02 <witek> but we can wait till we have everything what we need in
15:03:29 <dhague> o/
15:03:45 <rhochmuth> so, let's get started then
15:03:59 <rhochmuth> #topic What were the most important features added in the Ocata release?
15:04:25 <rhochmuth> i had an email form Nick Chase the other day asking about the most important features that we've done for the Ocata release
15:04:32 <rhochmuth> i owe him a response
15:04:46 <rhochmuth> i realize this release we didn't add as many features
15:04:56 <rhochmuth> but, the future is looking really good
15:05:27 <rhochmuth> anyway, if anyone has an important feature that i should response to nick chase about, please let me know
15:06:03 <witek> Kibana plugin for logs has been added as OpenStack repo
15:06:40 <rhochmuth> and what does that include specifically?
15:06:58 <witek> authorisation and multi-tenancy
15:07:24 <rhochmuth> Are all the queries now scoped to the tenand ID?
15:07:28 <witek> the ES indeces are scoped per project
15:07:40 <witek> yes
15:08:07 <rhochmuth> So, basically parity with Grafana, in a snese
15:08:11 <rhochmuth> sense
15:08:15 <rhochmuth> ?
15:08:45 <witek> with the difference, that it does not use monasca-datasource
15:09:08 <witek> but basicaly yes
15:09:30 <rhochmuth> and, it still requires your forked version of Kibana?
15:09:57 <witek> no, no forked Kibana
15:10:15 <rhochmuth> so, it basically integrates with the Kibana plugin system
15:10:22 <witek> correct
15:10:23 <rhochmuth> without any changes to their source
15:10:51 <rhochmuth> note to future self, pay more attention
15:10:54 <rhochmuth> :-)
15:11:17 <rhochmuth> witek: that is really great news!
15:11:35 <rhochmuth> your team has made great progress
15:11:45 <rhochmuth> did ES ever get back to you?
15:11:56 <witek> thank you, I will forward to the team :)
15:12:10 <witek> no
15:12:30 <rhochmuth> so, basically, this is all working without any help or involvement from ES?
15:12:43 <witek> right
15:12:54 <rhochmuth> fantastic!
15:13:17 <rhochmuth> so, do folks have any other favorite features worth mentioning
15:13:26 <rhochmuth> i know Tomasz has done a lot of work
15:13:46 <rhochmuth> adding support for the request ID to the monasca APIs was done
15:14:01 <rhochmuth> there was a lot of infra work
15:14:11 <rhochmuth> there are some great things in progress
15:14:26 <rhochmuth> support for InfluxDB 1.1 was added
15:14:30 <rhochmuth> by me
15:14:33 <rhochmuth> can't forget that
15:14:37 <rhochmuth> :-)
15:14:42 <witek> thank you rhochmuth
15:14:56 <rhochmuth> several interesting additions to the agent
15:15:20 <Guest20448> @Roland support for influxdb 1.1. -- is that for Java implementaiton?
15:15:24 <rhochmuth> support for docker, kubernetes and prometheus
15:15:50 <rhochmuth> Guest20448: That was only completed for the Python code
15:16:20 <rhochmuth> So, I think I have enough for Nick
15:16:31 <rhochmuth> I'll cc several folks, on my response
15:16:58 <hoppalm> the docker plugin is in the agent.
15:17:06 <rhochmuth> ahh yes, thanks
15:17:08 <hoppalm> the kubernetes and prometheus monitoring is still up for review
15:17:10 <hoppalm> :(
15:17:40 <rhochmuth> that's ok, i think i'll fdiscuss a lot of the great work that is in progress
15:17:45 <rhochmuth> too
15:17:55 <rhochmuth> but, the kibana work will be the highlight
15:18:02 <rhochmuth> of this release
15:18:26 <rhochmuth> thanks everyone
15:18:38 <rhochmuth> #topic blueprints and upcoming work
15:19:00 <rhochmuth> so, i wanted to point out that there is a lot of work in progress
15:19:09 <rhochmuth> several blueprints have been created
15:19:19 <rhochmuth> and a few more are about to be created
15:19:42 <rhochmuth> Templated alarm descriptions for human readable alerts
15:19:43 <rhochmuth> https://blueprints.launchpad.net/monasca/+spec/templated-alarms
15:20:45 <rhochmuth> There is an implementation that is available on sap's own branch
15:20:55 <rhochmuth> that will get moved over at some point
15:21:16 <rhochmuth> but, want to make folks aware of what is coming and to start to provide feedback
15:21:23 <jobrs> ... in the near future
15:21:35 <rhochmuth> jobrs: thx
15:22:19 <rhochmuth> last week we discussed, https://blueprints.launchpad.net/monasca/+spec/monasca-sidecar
15:22:35 <rhochmuth> wanted to check-in on whether anyone has had sometime to look at this and supply feedback
15:23:21 <timothyb89> it sounds like some of the issues seen with monasca-statsd that led to the sidecar are being resolved
15:23:54 <rhochmuth> timothyv89: how so?
15:24:05 <rhochmuth> timothyb89: ^
15:24:16 <timothyb89> now that we're supporting dogstatsd: https://review.openstack.org/#/c/426706/
15:24:54 <jobrs> we created statsd-coverage for notification, persister and api without running into issues
15:24:55 <timothyb89> one of the issues we saw was language support, dogstatsd has bindings for many languages
15:25:24 <jobrs> IMHO we should just add compatibility to the agent.
15:25:34 <rhochmuth> agress
15:25:37 <rhochmuth> agree
15:25:58 <jobrs> submitted the first part yesterday. What is missing is (restored) support for histograms
15:26:10 <rhochmuth> i thought the sidecar was also desirable for other reasons in a kubernetes env
15:26:16 <hoppalm> yes
15:26:24 <timothyb89> it still does solve some problems, yes
15:26:43 <jobrs> +1, for blackbox monitoring
15:27:05 <hoppalm> running the statsd as side container still carries the need then for authenticating with the api unless you run the statsd as a service in kubernetes which causes high udp traffic
15:27:31 <jobrs> we use prom-statsd sidecar
15:27:40 <hoppalm> yes i think that is the right answer
15:27:43 <jobrs> which is compatible to DogStatsd
15:28:31 <rhochmuth> so, is the statsd work jobrs is adding as replacing the sidecar
15:28:36 <rhochmuth> or do we want both
15:28:41 <rhochmuth> i thought both
15:28:50 <jobrs> yep
15:29:03 <rhochmuth> ok, so i think we are good then
15:29:12 <rhochmuth> i'm actually going to try to start reviewing code again
15:29:26 <rhochmuth> sorry, i've been really busy and got behind on any significant reviews
15:29:49 <rhochmuth> and relying on others to carry this work forward and keep it moving along
15:30:35 <rhochmuth> So, on a related topic, there is the "mapper",
15:30:42 <rhochmuth> https://review.openstack.org/#/c/420774/
15:31:02 <jobrs> let me add one thing to statsd: I believe monasca-sidecar is great, but it would be perfect if only it would process DogStatsd
15:31:25 <hoppalm> +1 to ^
15:31:41 <hoppalm> and in regards to the review I will be looking at it again today or tomorrow
15:31:45 <timothyb89> that is an option as well, sure
15:32:02 <rhochmuth> thx jobrs and timothyb89
15:32:36 <rhochmuth> there is a new blueprint for https://blueprints.launchpad.net/monasca/+spec/log-query-api
15:32:45 <rhochmuth> adding a log query api
15:33:29 <rhochmuth> this is being worked on by witek and steve simpson
15:33:44 <witek> oposite order
15:33:45 <rhochmuth> Also see, https://wiki.openstack.org/wiki/Monasca/Logging/Query_API_Design
15:34:09 <rhochmuth> OK, Steve is leading the charge
15:34:20 <rhochmuth> witek is contributing to it
15:35:24 <witek> Steve, are you there?
15:35:24 <rhochmuth> And, we have a number of engineers at HPE working at alarm inhibition, alarm silencing, alarm grouping, and a new MOnasca QUery Language (MQL)
15:35:47 <rhochmuth> blueprints will be showing up soon for those other areas that i mentioned
15:36:02 <bklei> that's great -- charter has talked about alarm silencing too
15:36:06 <rhochmuth> where's waldo?
15:36:58 <rhochmuth> also, there is a lot of work in creating helm charts for monasca between hpe and sap
15:37:03 <rhochmuth> or sap and hpe
15:37:19 <rhochmuth> no blueprint for that
15:37:34 <rhochmuth> i'm wondering with all this work if we should plan on a mid-cycle
15:38:10 <witek> sure
15:38:13 <rhochmuth> seems like we have enough that a couple of days of focused planning and discussions would be a good idea
15:38:19 <bklei> +1
15:38:36 <rhochmuth> do we want to do this in late february
15:38:48 <rhochmuth> slightly before or after the PTG
15:38:55 <rhochmuth> just in case someone is going to the PTG?
15:39:26 <bklei> no conflict from charter guys
15:40:27 <rhochmuth> we could hold it then on wednesday adn thursday during the same week of ptg
15:40:52 <rhochmuth> if there aren't any conflicts
15:41:07 <witek> is SAP going to PTG?
15:41:34 <rhochmuth> jobrs: ^^^
15:41:42 <rhochmuth> dhague: ^^^
15:42:21 <jobrs> currently not planned
15:42:33 <jobrs> our impression was that participation would be very low on Monasca
15:42:41 <rhochmuth> correct
15:42:52 <rhochmuth> we woudl like to do our own monasca mid-cycle remotely
15:43:05 <rhochmuth> via videoconferenceing the same week as ptg
15:43:11 <rhochmuth> probably wednesday and thursday
15:43:24 <rhochmuth> if that works for you and your team?
15:43:41 <rhochmuth> basically, in three weeks
15:43:54 <jobrs> should be fine
15:44:06 <rhochmuth> cool, sounds good to me then
15:44:56 <rhochmuth> let's tentatively mark that wed and thursday down and unless any conficts occur we'll meet remotely that week
15:45:13 <witek> +!
15:45:27 <rhochmuth> #topic reviews
15:45:35 <rhochmuth> https://review.openstack.org/#/c/427571/
15:46:56 <rhochmuth> i just added a +1
15:47:13 <rhochmuth> https://review.openstack.org/#/c/425152/
15:47:19 <rhochmuth> this looks good to me
15:48:04 <rhochmuth> tomasz has been busy
15:48:06 <rhochmuth> https://review.openstack.org/#/c/425559/
15:49:24 <rhochmuth> https://review.openstack.org/#/c/356403/
15:49:32 <rhochmuth> I'll take a look at both of the above
15:49:40 <rhochmuth> but, it looks good to me
15:50:19 <rhochmuth> i think that is the end of today's agenda
15:50:24 <rhochmuth> #topic open floor
15:50:52 <jobrs> I have a very technical question about the thresholder
15:51:03 <rhochmuth> jobrs: sure
15:51:40 <jobrs> we observe that deleted alarm-definitions resp. their alarms are still processed, causing alarm-state-transition messages
15:51:54 <jobrs> until we restart/kill the thresholder.
15:52:22 <jobrs> we recognised this after we introduced a canary test which creates temporary alarm-definitions every minute
15:52:24 <rhochmuth> if you delete the alarm definition, all alarms created by the definition should have been deleted too
15:52:32 <jobrs> no errors or suspicious entries in the logs.
15:53:19 <jobrs> config DB is also clean; code-review of thesholder did also not reveal anything suspicious
15:53:29 <rbrndt> jobrs: thats a lot of alarm definitions to be pushing through the system on a constant basis
15:53:43 <rhochmuth> hmmm, we can take a look and verify
15:54:08 <rbrndt> I can tell you something that may help in debugging
15:54:22 <rhochmuth> so, if i understand, you create an alarm defiition, which results in new alarms
15:54:48 <rbrndt> If i'm not mistaken it is still the API which handles the alarm definition deletion, while the threshold engine waits for a message on kafka before clearing it's internal memory
15:55:01 <rhochmuth> then, you delete the alarm definition, and the alarms are still there?
15:55:53 <jobrs> I delete the alarm-definition, the API updates the deleted_at field and the alarms are cleared from the database by the thesholder event handler.
15:56:12 <jobrs> our cleanup job takes care of the rest: removing alarm-definitions with deleted_at != null from the tables
15:56:58 <jobrs> the database is clean. but if I watch the alarm-state-transitions topic (using kafkacat) I observe new messages about very old alarms/alarmdefs
15:57:52 <rhochmuth> hmmm, so the threshold engine still has the alarm definition or alarm
15:57:58 <jobrs> rbrndt, rhochmuth: continue on IRC?
15:58:00 <jobrs> yes
15:58:04 <rbrndt> has a bug been filed about this?
15:58:15 <jobrs> no
15:58:29 <rbrndt> it sounds like something that is probably happening everywhere, perhaps we should continue in a bug report
15:58:34 <jobrs> and I could not believe that this is a bug in the code. So we are looking for other explanations
15:59:21 <rhochmuth> on the surface sounds like a bug
15:59:30 <jobrs> thanks, that is a good proposal. I will file a bug and let you know, ok?
15:59:31 <rhochmuth> if you can create a report
15:59:38 <rhochmuth> that woudl be great
15:59:44 <rhochmuth> we'll sdtart looking into ti
15:59:45 <tomasztrebski> we started alrady ?
15:59:48 <rbrndt> lots of detail will help us reproduce this faster
16:00:01 <jobrs> sure
16:00:08 <rhochmuth> tomasztrebski: we started an hour ago
16:00:12 <jobrs> I just wanted to check whether you ran into something similar
16:00:12 <rhochmuth> we are wrapping up
16:00:17 <rhochmuth> thanks jobrs
16:00:23 <rhochmuth> i need to end the meeting
16:00:27 <rhochmuth> for the next group
16:00:28 <tomasztrebski> damn it...I messed up hours
16:00:32 <rhochmuth> thanks everyone
16:00:45 <rhochmuth> tomasztrebski: i saw your reviews in the etherpad agenda
16:00:48 <rhochmuth> i'l; review
16:00:59 <tomasztrebski> rhochmuth: thx
16:01:00 <rhochmuth> see you all next week
16:01:02 <notq> cheers everyone
16:01:04 <tomasztrebski> bey
16:01:05 <rhochmuth> or sooner
16:01:08 <tomasztrebski> *bye
16:01:11 <rhochmuth> #endmeeting