15:00:34 <rhochmuth> #startmeeting monasca
15:00:38 <openstack> Meeting started Wed Jul  6 15:00:34 2016 UTC and is due to finish in 60 minutes.  The chair is rhochmuth. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:39 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:42 <openstack> The meeting name has been set to 'monasca'
15:01:27 <rbrndt> o/
15:01:32 <rbak> o/
15:01:33 <bklei> o/
15:01:34 <Kamil> o/
15:01:36 <witek> hello
15:01:39 <ezpz> o/
15:01:39 <tsv> o/
15:01:42 <shinya_kwbt> o/
15:01:45 <rhochmuth> o/
15:01:46 <Fdaisuke_> o/
15:01:48 <hosanai> o/
15:01:58 <rhochmuth> agenda at: https://etherpad.openstack.org/p/monasca-team-meeting-agenda
15:02:04 <rhochmuth> Agenda for Wednesday July 6, 2016 (15:00 UTC)
15:02:05 <rhochmuth> 1. https://blueprints.launchpad.net/monasca/+spec/delete-metrics
15:02:05 <rhochmuth> 2. https://blueprints.launchpad.net/monasca/+spec/read-only-api-user
15:02:05 <rhochmuth> 3. Beaver log agent, status?
15:02:15 <rhochmuth> Fairly light agenda today
15:02:35 <rhochmuth> so we can leave time at the end for general discussion
15:02:55 <rhochmuth> #topic delete metrics
15:02:56 <rhochmuth> https://blueprints.launchpad.net/monasca/+spec/delete-metrics
15:02:59 <bklei> that's me
15:03:42 <bklei> have we discussed this before?  it's use-case we really need here at charter.  the ability for our MaaS customers to delete their own metrics
15:03:59 <bklei> we've found it's an iterative process when pushing metrics in -- don't get it right the first tiem
15:04:00 <bklei> time
15:04:22 <bklei> we'd like our customers to be able to do their own cleanup
15:04:40 <rhochmuth> DELETE /v2.0/metrics
15:04:44 <bklei> right
15:04:46 <rhochmuth> just use that
15:04:58 <bklei> wait -- that's there? :)
15:05:34 <rhochmuth> DELETE /v2.0/metrics/measurements
15:05:47 <rhochmuth> so are those the proposals
15:05:55 <ddieterly> o/
15:05:58 <rhochmuth> the two proposals
15:06:16 <rhochmuth> the first one would delete the entire metric and all measurements
15:07:19 <bklei> would like both i guess -- but if i had to pick one, i'd prefer DELETE /v2.0/metrics if it cleans up measurements
15:07:37 <bklei> as long as DELETE /v2.0/metrics supported specifying dimensions
15:08:08 <rhochmuth> the first one, with support for metric name, dimensions, as filter options would seems to make sense to me
15:08:32 <rbrndt> This is just for deleting historical data?
15:08:45 <rhochmuth> metrics measurements
15:08:53 <bklei> yeah -- i agree.  our current prune script supports more granularity of measurements by timestamp range, but i think we wouldn't need that for our customers
15:09:10 <bklei> yes -- both bogus metrics/dimensions, and corresponding historical measurements
15:09:15 <rhochmuth> have you looked at performance
15:09:25 <rhochmuth> this used to run very slow in vertica for example
15:09:36 <rhochmuth> but i think you mentioned they improved perforamance
15:09:37 <bklei> yes -- on the vertica side.  we run a prune job nightly
15:09:41 <rhochmuth> i'm not sure about influxdb
15:09:46 <rhochmuth> or cassandra
15:10:11 <bklei> i'm not sure about either of those either
15:10:24 <rhochmuth> so, you don't see performance hit in vertica when deleting metrics
15:10:40 <rhochmuth> in vertica deletes were slow and involved a full table lock as row locks weren't supported
15:10:53 <rhochmuth> so, the db would be locked out while vertica did it
15:10:59 <bklei> vertica 7.2 works fine -- the performance depends on what % of the measurements table you are deleting
15:11:00 <rhochmuth> it's thing
15:11:32 <rhochmuth> does it lock out the db when deleting
15:11:47 <bklei> we do see brief kafka lag during the prune job -- but it's only a couple of minutes, and we're deleting a large amount of data across all projects nightly
15:11:52 <bklei> yes
15:12:08 <rhochmuth> how would you prevent potential customer abuse
15:12:22 <rhochmuth> inadvertent dos attack for example, non-malicious
15:12:37 <rhochmuth> customer says, delete all metrics,
15:12:47 <rhochmuth> then the db is taken out for a while
15:12:52 <rhochmuth> or locked out
15:13:07 <bklei> it's a potential issue
15:13:48 <rhochmuth> i guess i was thinking about implementation
15:14:12 <rhochmuth> the simple implementatino is a direct invocation of truncate or whatever in the db, directly from the API
15:14:55 <rhochmuth> but, i'm wondering if a separate micro-service would be a better approach, so that deletes could be schedule and occur async over some longer time-period
15:15:19 <bklei> yes -- in terms of vertica implementation -- it's a DELETE from.. https://github.com/openstack/puppet-monasca/blob/master/templates/vertica/prune_vertica.py.erb
15:16:03 <bklei> hmm.  a separate service that batches things up.  that could alleviate the dos concern
15:16:03 <rhochmuth> I see
15:16:15 <rhochmuth> monasca-delete
15:17:00 <rhochmuth> anyway, i think there needs to be some discussion on implementation
15:17:10 <bklei> i like that idea -- i can expand the blueprint and discuss implementation alternatives
15:17:10 <rhochmuth> is this one we should cover at the mid-cycle
15:17:19 <rhochmuth> thx
15:17:26 <bklei> yeah -- i think that's appropriate, i'll beef up the blueprint
15:17:37 <bklei> prior to that
15:18:00 <rhochmuth> probably should add as an agenda item to the mid-cycle
15:18:14 <rhochmuth> should we move onto the next topic then?
15:18:19 <bklei> sure -- will add it when i find the link :)
15:18:39 <bklei> got it
15:19:28 <rhochmuth> #topic https://blueprints.launchpad.net/monasca/+spec/read-only-api-user
15:19:33 <rhochmuth> Read only API
15:19:35 <bklei> that' me too
15:20:02 <bklei> right now -- we basically have two API roles, one can POST only, and one can do everything else
15:20:21 <bklei> but -- we'd like to have dashboard/operator users who can see data, but not POST (or DELETE :))
15:20:59 <bklei> grafana supports read-only, we'd like to map those users to back-end readonly
15:21:33 <bklei> does this seem like a use-case others would benefit from?
15:22:30 <bklei> or -- do folks object :)
15:22:35 <rhochmuth> i think it is useful to be able to restrict users to read only access
15:22:42 <bklei> i don't think it would be a huge change
15:23:21 <rhochmuth> i was hoping in the cas of python that we would implement openstack RBAC
15:23:25 <bklei> for sure
15:23:44 <bklei> if folks agree the idea is OK, i'll work the change, both python/java
15:23:52 <rhochmuth> unfortunately, the Java code doesnt' have a similar library
15:24:08 <bklei> right -- but the functionality is there, i'll just extend it
15:24:29 <witek> sounds good to me
15:24:48 <bklei> thx witek
15:25:01 <rhochmuth> sounds fine to me
15:25:16 <bklei> anyone else object? otherwise i'll press ahead with a patch
15:25:23 <witek> :)
15:25:27 <rhochmuth> there are probably a few details to discuss
15:25:55 <bklei> like role name?  i was going to follow suit and make configurable
15:25:56 <rhochmuth> what is your time-frame for implementation
15:26:14 <bklei> soon -- probably start in a week or so
15:27:34 <rhochmuth> sounds like new topic time
15:27:38 <rhochmuth> #topic Beaver log agent, status?
15:27:42 <bklei> thx
15:28:01 <witek> I wanted to ask about the status of the Beaver log agent
15:28:15 <witek> would it be possible to contribute to that?
15:28:16 <rhochmuth> interesting, i just got an email from tsv this morning
15:28:39 <rhochmuth> i thought i saw him join the meeting
15:28:40 <tsv> witek: I am refactoring the existing pull request and added some unit tests. will push an update today or tomorrow max
15:28:56 <witek> where is the repo?
15:29:21 <tsv> https://github.com/python-beaver/python-beaver/pull/406
15:29:31 <witek> tsv: thanks
15:29:47 <rhochmuth> witek: are you interested in the beaver agent?
15:29:50 <tsv> witek: will send you the link once my updates are in
15:29:55 <witek> yes
15:30:14 <rhochmuth> are you thinking about moving for logstash to beaver?
15:30:27 <witek> we have an internal POC where we don't want to install logstash
15:30:42 <witek> more like an alternative
15:30:59 <rhochmuth> ok
15:31:20 <tsv> rhochmuth: sorry about the short notice. fell sick over the weekend
15:31:31 <rhochmuth> np
15:32:28 <witek> so, python-beaver sends logs to logstash?
15:33:05 <tsv> witek: it is highly configurable. the pull request i am working on adds a monascalog transport - which would send the logs to monasca log api
15:33:26 <witek> i see
15:34:50 <witek> we have added the pull request with our logstash output plugin to logstash-plugins repository
15:34:55 <witek> https://github.com/logstash-plugins/logstash-output-monasca_log_api/pull/1
15:35:08 <witek> but it does not get much attention :(
15:35:34 <tsv> witek: i see, will check it out
15:35:34 <rhochmuth> 9 days no comments
15:36:07 <rhochmuth> do we need to vote for it
15:36:39 <witek> for choosing the agent?
15:36:50 <rhochmuth> for merging your pr
15:37:21 <rhochmuth> i thought there was a way to thumbsup the pr
15:37:30 <witek> oh, I think one comment from you or tsv would be enough for now
15:37:41 <bklei> pr == parole officer?
15:37:53 <rhochmuth> pull request
15:39:11 <rhochmuth> bklei: are you planning on a bp for dimensions
15:39:19 <tsv> witek: just left a comment
15:39:27 <witek> thanks a lot!
15:39:37 <bklei> can if you want rhochmuth
15:39:50 <rhochmuth> i think that would be good
15:40:01 <bklei> will add it
15:40:07 <rhochmuth> what is the response that you are thinking about providing
15:40:18 <rhochmuth> also, we have dimenion names and dimension values
15:40:45 <rhochmuth> just wondering if you are thinking about two resources
15:40:49 <rhochmuth> or just one called dimensions
15:41:14 <rhochmuth> i then to think of this as two resources
15:41:28 <rhochmuth> dimension-names
15:41:32 <rhochmuth> dimension-values
15:42:01 <bklei> the use-case we're working for grafana is basically dimension-values
15:42:15 <bklei> for a given dimension name -- what are the possible values
15:42:32 <bklei> example output
15:42:35 <bklei> {
15:42:36 <bklei> "links":[
15:42:36 <bklei> {
15:42:38 <bklei> "rel":"self",
15:42:40 <bklei> "href":"http://dev02-monasca-001.os.cloud.twc.net:8070/v2.0/metrics/dimensions?name=hostname"
15:42:42 <bklei> }
15:42:44 <bklei> ],
15:42:46 <bklei> "elements":[
15:42:48 <bklei> {
15:42:50 <bklei> "id":null,
15:42:52 <bklei> "name":"hostname",
15:42:54 <bklei> "values":[
15:42:56 <bklei> "dev02-compute-001",
15:42:58 <bklei> 
15:43:00 <bklei> not done, but u get the idea
15:43:33 <bklei> i wasn't planning an api to get all the dimension names -- is there a need for that?
15:43:34 <bklei> rbak?
15:43:57 <rbak> I didn't really see a need for that, at least not in grafana.
15:44:24 <rhochmuth> i thought it would be seful to get just the dimension names, given a metric name
15:44:43 <rbak> But you can already do that using metric-list with a metric name
15:44:45 <rhochmuth> the problem with returning all the dimension values, is that could be a huge list
15:45:16 <rhochmuth> metric-list also returns a huge list
15:45:17 <bklei> will need pagination/limit for sure
15:45:46 <bklei> even though potentially huge, still faster and smaller than metric-list
15:45:48 <rhochmuth> so, let's say the user selects "vm.cpu.user_perc"
15:45:55 <rhochmuth> as the metric name
15:46:14 <bklei> this is for templating -- metric name not part of the request
15:46:48 <rhochmuth> there are potentially hundreds of thousands if not millions of VMs
15:47:07 <rhochmuth> there are maybe around 5-50 dimensions for the metric
15:47:10 <rhochmuth> 5-10
15:47:15 <rbak> Sure, but the getting just the dimension values is far better than getting a full metric-list
15:48:09 <rhochmuth> agree
15:48:28 <bklei> our current issue is metric-list kills the browser when trying to build a template
15:48:48 <bklei> paginates until it's out of memory
15:48:49 <rhochmuth> but, i want to leave a api spot open for just getting dimension names
15:49:06 <rbak> Why is that necessary though?
15:49:10 <rhochmuth> so, you can got from name, to dimension name to dimension value
15:49:52 <rhochmuth> i think it is more of a question around optimizing the result set
15:50:02 <bklei> we pretty much have that?  you can --name with metric list
15:50:13 <rbak> You can already accomplish that with a metric-list though.  We already do this in grafana and it's pretty instant.
15:50:52 <rhochmuth> but, you are going to get all the unique metrics for a specific metric name
15:50:53 <rbrndt> metric-list gives a set of dimension sets, lots of duplicates. The dimensions-name would give just a list of the unique dimension names
15:51:06 <rhochmuth> thx rbrndt
15:51:26 <rbak> I suppose.  That's just never been a problem for us.
15:51:32 <bklei> ok -- maybe there's a middle ground, maybe i implement dimension-values for now, can add dimension-names later?
15:52:23 <bklei> and for dimension-values, i can add an optional metric name?
15:52:37 <bklei> but templating in grafana wouldn't use it
15:52:44 <rbrndt> The problem the dimension names resource is more useful for is the case of a single metric names with millions of dimensions sets associated
15:53:07 <rhochmuth> that was the case i was thinking of
15:53:14 <rbrndt> which occurs with lots of vas and a high churn rate
15:53:21 <rbrndt> s/vas/vms
15:53:26 <rhochmuth> for the operator/admin account
15:53:40 <rhochmuth> i don't think this would be required for maas customers/tenants
15:53:59 <rhochmuth> but in the case of the operator/tenant account where every vm in the system is accessible, it would be useful
15:54:23 <bklei> don't disagree, we just haven't run across the use-case asking for that yet
15:55:03 <rhochmuth> we've been hitting it
15:55:16 <rhochmuth> since we've been more focused on the operational monitoring problem
15:55:31 <rhochmuth> and not maas as much yet
15:55:53 <bklei> understood
15:56:17 <bklei> i'll add a BP and we can hash this out there?
15:56:26 <rhochmuth> thx
15:56:29 <bklei> hopefully i can press fwd with dimension-valuse
15:56:32 <bklei> values
15:56:57 <rhochmuth> yes, please proceed
15:57:03 <bklei> gracias
15:57:35 <rhochmuth> so, any other quick topics in closing
15:57:39 <rhochmuth> just a few minutes left
15:57:42 <rbak> rhochmuth: Did you ever get that new monasca-grafana-datasource repo we talked about a couple weeks ago?
15:58:03 <rhochmuth> thanks for reminding me
15:58:10 <rhochmuth> i'll get it submitted
15:58:12 <rhochmuth> sorry
15:58:19 <rbak> np.  thanks for doing that.
15:58:55 <rhochmuth> how are your disucssions going with grafana btw
15:58:58 <Rockyg> o/
15:59:16 <rbak> Not a lot of progress.  We'll keep you updated as we hear more.
15:59:23 <rhochmuth> ok, thx
15:59:31 <rhochmuth> looks like ending the meeting
15:59:59 <rhochmuth> thanks everyone
16:00:06 <witek> thank you
16:00:39 <rhochmuth> #endmeeting