15:00:17 <rhochmuth> #startmeeting monasca
15:00:17 <openstack> Meeting started Wed Oct 19 15:00:17 2016 UTC and is due to finish in 60 minutes.  The chair is rhochmuth. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:18 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:19 <rhochmuth> o/
15:00:20 <rhochmuth> https://etherpad.openstack.org/p/monasca-team-meeting-agenda
15:00:21 <openstack> The meeting name has been set to 'monasca'
15:00:35 <rhochmuth> Agenda for Wednesday October 19, 2016 (15:00 UTC)
15:00:35 <rhochmuth> 1.	Summit Prep
15:00:35 <tomasztrebski> o/
15:00:35 <rhochmuth> 2.	Reviews
15:00:35 <rhochmuth> 1.	https://review.openstack.org/#/c/388183/
15:00:35 <rhochmuth> 2.	https://review.openstack.org/#/c/375717/
15:00:35 <rhochmuth> 3.	Ask Charter Communications about the size of their deployment
15:00:35 <rhochmuth> 4.	 Lost & Forgotten [ help with testing, Tomasz :( ]
15:00:36 <rhochmuth> 1.	https://review.openstack.org/#/c/361318/
15:00:36 <rhochmuth> 2.	https://review.openstack.org/#/c/315742/
15:00:37 <rhochmuth> 5.	Devstack [Openstack] integration
15:00:37 <rhochmuth> 1.	https://review.openstack.org/#/c/387195/
15:00:38 <rhochmuth> 2.	https://review.openstack.org/#/c/385149/
15:00:38 <rhochmuth> 6.	Local deployment fix
15:00:39 <rhochmuth> 1.	https://review.openstack.org/#/c/388017/
15:00:46 <shinya_kwbt> o/
15:00:47 <rbak> o/
15:00:51 <hosanai> o/
15:00:53 <cbellucci> o/
15:00:57 <Kamil_> o/
15:00:58 <rhochmuth> a bit of a higger agenda this week
15:01:07 <tomasztrebski> sorry :(
15:01:09 <rhochmuth> the first topic i woudl like to get to is summit prep
15:01:11 <koji> o/
15:01:20 <rhochmuth> https://etherpad.openstack.org/p/monasca-barcelona-design-summit
15:01:25 <rhochmuth> please see the above link
15:01:26 <bklei> o/
15:01:36 <rhochmuth> are there other items that we should add or expand on
15:01:47 <rhochmuth> i need to update the summit calendar/schedule
15:01:52 <rhochmuth> i woudl like to do that today
15:01:58 <rhochmuth> i'm pretty much already late on this
15:02:13 <rhochmuth> but, maybe unlike last time, only the people involved wwith monasca will show up
15:02:19 <rhochmuth> that could be a good thing
15:02:38 <rhochmuth> because last time in austin, a lot of folks showed up that weren't involved with monasca
15:02:44 <rhochmuth> and that diluted the meeting significantly
15:02:51 <rhochmuth> so, anyway
15:03:01 <rhochmuth> Here is what i have so far
15:03:02 <rhochmuth> 1.	Monasca New Database discussion
15:03:02 <rhochmuth> 1.	Cassandra
15:03:03 <rhochmuth> 1.	https://blueprints.launchpad.net/monasca/+spec/cassandra-tasks
15:03:03 <rhochmuth> 2.	https://etherpad.openstack.org/p/monasca_cassandra
15:03:03 <rhochmuth> 2.	Locked alarms
15:03:03 <rhochmuth> 1.	https://blueprints.launchpad.net/monasca/+spec/locked-alarms
15:03:03 <rhochmuth> 3.	Logging
15:03:03 <rhochmuth> 1.	Multi-tenancy support
15:03:04 <rhochmuth> 4.	Properties/attributes for metrics/logs and sporadic metrics
15:03:04 <rhochmuth> 1.	https://blueprints.launchpad.net/monasca/+spec/publish-logs-to-topic-selectively
15:03:05 <rhochmuth> 5.	Monasca Analytics
15:03:05 <rhochmuth> 1.	Anomaly Detection
15:03:06 <rhochmuth> 2.	Alarm correlation/clustering
15:03:23 <rhochmuth> Obviousely, on the topic of Cassandra, based on the update from Monday, that disucssion needs to expand
15:03:46 <rhochmuth> because, Cassandra has got some difficult areas that we haven't been able to resolve
15:04:00 <rhochmuth> Item 2 is locked alarms
15:04:08 <rhochmuth> there is a blueprint for that one
15:04:31 <rhochmuth> but i woudl like to discuss that one in more detail and close on the design if possible
15:04:43 <rhochmuth> next item is multi-tenenacy
15:04:47 <rhochmuth> for logging
15:04:59 <rhochmuth> looking for fujitsu to lead that discussion
15:05:30 <rhochmuth> the nest item is properties/attribues for metrics/logs, which is related to support for sporadic alarms
15:05:57 <rhochmuth> i'll let you look at the remainder
15:06:30 <rhochmuth> if there are other topics, now is the change to add them
15:06:41 <rhochmuth> a blueprint would also be nice to accompany the discussion if that makes sense
15:07:04 <Kamil_> i think tomasz is in the meeting
15:07:08 <tomasztrebski> as far as I understood this is about discussing the design
15:07:13 <rhochmuth> correct
15:07:27 <tomasztrebski> I added one topic that I think is one thing that monasca would benefit from
15:07:30 <tomasztrebski> last point
15:07:40 <tomasztrebski> monasca maintenance
15:07:49 <rhochmuth> ok, i'll let you lead that discussion then
15:07:58 <tomasztrebski> cool
15:08:28 <tomasztrebski> so IMHO I think that what monasca does not have that could be done semi-automatic is muting certain alarm definitions
15:08:49 <tomasztrebski> based on defined period of time, whether this should be repeatable operations
15:09:19 <rhochmuth> ahhh, is that related to "flood detection/mitigation"
15:09:24 <tomasztrebski> like for example you got server maintenance over the night/weekend and you know certain alarm definitions will lit because of higher resource usage
15:09:33 <tomasztrebski> and you want to mute them
15:09:36 <tomasztrebski> maybe :D
15:10:09 <rhochmuth> so, you want to adjust the thresholds based on patterns
15:10:12 <tomasztrebski> the overall idea would be to expand data model to define maintenance_definition and have an engine to modify alarm_definitions a.k.a. change notifications_enabled flag status
15:10:13 <bklei> agree with the concept tomasztrebski -- that's lacking if you compare monasca alarming vs other tools like icinga/nagios
15:10:28 <bklei> we'd support a change like that for sure
15:10:56 <tomasztrebski> this ain't rocket science but there would be some work to be done in api / ui / client
15:11:01 <tomasztrebski> not sure about the threshold
15:11:16 <rhochmuth> i see, it is simpler than that
15:11:44 <tomasztrebski> and one thing that would need to be added (IMHO) would be extra component to change status because I think threshold is responsible for transitioning basicly
15:11:47 <rhochmuth> we've been interested in this area too
15:12:08 <tomasztrebski> rhochmuth: what do you mean by simpler ?
15:12:10 <rhochmuth> would it be more like alarm supression then
15:12:20 <tomasztrebski> btw: that would be all I had in my head about that topic
15:12:26 <bklei> even just a maintenance flag will go a long way
15:12:27 <rhochmuth> rhochmuth: simpler than adjusting the threshold dynamically
15:12:58 <rhochmuth> so going back to my question, is it related to alarm supression?
15:13:34 <tomasztrebski> if you mean to suppress transition - no, I would keep it the way it is, just disable any notifications there are
15:13:47 <tomasztrebski> but maybe suppressing transition could be an option to discuss - sure
15:13:59 <rhochmuth> ok, sounds like a good area to discuss
15:14:15 <rhochmuth> i also put a topic up for alarm flood detection/mitigation
15:14:32 <rhochmuth> it isn't exactly related, but I would like to cover that
15:14:48 <rhochmuth> as prometheus does that
15:14:50 <tomasztrebski> however I think that without notifications alarm is kind of meaningless, you could also add some sort of extra info to alarm like -> maintenance on or something similar to just hint the user
15:15:17 <rhochmuth> ok, so let's move on with agenda
15:15:22 <rhochmuth> thanks tomasz
15:15:29 <tomasztrebski> np
15:15:38 <rhochmuth> you are up for leading that session in barcelona
15:15:53 <tomasztrebski> :scared_cat_face
15:16:25 <rhochmuth> you will be scared
15:16:34 <rhochmuth> that was yoda
15:16:39 <tomasztrebski> :scared_like_hell_crying_cat_face :P
15:16:54 <tomasztrebski> heh, yoda rlx
15:16:55 <rhochmuth> #topic reviews
15:17:03 <rhochmuth> https://review.openstack.org/#/c/388183/
15:17:16 <bklei> that's me
15:17:42 <bklei> i think we're making headway on that one -- very excited to pull it in.  huge perf diff for vertica
15:17:53 <rhochmuth> we are excited about it too
15:18:14 <bklei> sounds like you saw similar 'minutes to seconds' improvement
15:18:15 <rhochmuth> rbrandt abadoned the other review we posted btw
15:18:27 <rhochmuth> or i thought he was going to abandon
15:18:28 <bklei> oh the projection hint?
15:18:28 <tomasztrebski> pity we (Fujitsu) cannot be excited - vertica does not like Fujitsu ;-)
15:18:44 <rhochmuth> correct bklei
15:18:59 <rbrndt> I'm running tests on the review in question right now. Functionally it looks fine, just testing performance on our loaded cluster
15:19:09 <bklei> you think the refactored query allows the optimizer to choose the right projection?
15:19:13 <bklei> or it doesn't matter
15:20:02 <rhochmuth> rbrndt?
15:20:28 <rbrndt> For the query we put the hint on?
15:20:52 <rbrndt> I think we can wait for the vertica fix, if it's not a prevalent problem
15:20:52 <bklei> right -- that query would be time qualified
15:21:29 <rhochmuth> we good?
15:21:42 <bklei> si
15:21:44 <rhochmuth> https://review.openstack.org/#/c/375717/
15:22:16 <rhochmuth> yeah, so my only issue/concern is the number of dimensinos that are being added and potential performance hit
15:22:21 <bklei> me also -- i'll roll craig's feedback in today -- just an advertisement for publishing vm names and tenant names -- our infra engineers really want this one
15:22:33 <rhochmuth> those bastards
15:22:36 <bklei> valid concern.
15:22:58 <rhochmuth> can't they use a uuid
15:22:59 <bklei> should only change behavior if you add 'tenant_name' to metadata for either libvirt or ovs
15:23:00 <rhochmuth> by now
15:23:18 <rhochmuth> so, have you looked at performance at all
15:23:20 <bklei> and only perf hit for the keystone tenant list when cache gets updated
15:23:30 <bklei> tenant list is fast
15:23:31 <rhochmuth> or is it in the minutia
15:23:47 <rhochmuth> i see
15:23:58 <rhochmuth> well, i'll take a close look
15:24:04 <rhochmuth> i can't argue with it being useful
15:24:04 <bklei> we have something like 500 projects -- it returns fast
15:24:18 <bklei> one less translation step at 2:00 am :)
15:24:36 <rhochmuth> ok, i'll look it over
15:24:46 <rhochmuth> and get back with you if i have any other concerns
15:24:46 <bklei> many thanks -- i'll address craig's feedback today
15:24:51 <bklei> cool
15:25:12 <rhochmuth> #topic Ask Charter Communications about the size of their deployment
15:25:22 <bklei> size meaning?
15:25:25 <rhochmuth> i wanted to ask bklei about size of deployment
15:25:35 <rhochmuth> number of nodes, number tenants
15:25:40 <rhochmuth> if that could be shared
15:25:48 <bklei> generally yes
15:25:48 <rhochmuth> i'm just updating a slide for next week
15:25:54 <rhochmuth> metrics/sec
15:26:18 <bklei> 12K metrics/sec
15:26:30 <rhochmuth> wow, cranking it up since last time
15:26:32 <bklei> 105 Billion measurements in vertica
15:26:47 <rhochmuth> you'll pass mcdonald's soon
15:27:03 <bklei> :) according to vertica prof services  -- we're small
15:27:04 <rhochmuth> how many compute nodes and vms?
15:27:08 <bklei> looking
15:27:57 <bklei> somewhere around 600 to 700 physical nodes, which run the agent
15:28:02 <bklei> that's two datacenters
15:28:21 <bklei> looking up vms/projects
15:28:24 <rhochmuth> 600-700 per two datacenters or aggregate
15:29:30 <bklei> aggregate
15:29:36 <rhochmuth> thx
15:29:42 <bklei> about 1000K vms per region/datacenter
15:29:54 <rhochmuth> awesome, thanks
15:29:59 <bklei> and about 250 customer projects per region/data center
15:30:09 <bklei> and another 50 per datacenter infra related
15:30:11 <rhochmuth> that is kinda in alignment with what we've been targeting with helion
15:30:16 <rhochmuth> you are a little higher
15:30:27 <bklei> colorado joke
15:30:40 <rhochmuth> but, we end up collecting a little more metrics per vm
15:30:50 <rhochmuth> our motto, you are a little higher
15:30:52 <haad1> hi
15:30:52 <tomasztrebski> that's the point where foreign key don't get a joke -_-
15:31:09 <rhochmuth> colorado is high in altitude
15:31:11 <bklei> recreational pot legal in colorado
15:31:18 <rhochmuth> that too
15:31:44 <rhochmuth> thanks bklei.
15:31:52 <bklei> yup
15:32:00 <rhochmuth> i'll add some details to one of the slides being presented in barcelona
15:32:11 <bklei> cool
15:32:20 <rhochmuth> #topic Lost & Forgotten [ help with testing, Tomasz :( ]
15:32:22 <bklei> i'll watch on the youtube
15:32:33 <rhochmuth> :-)
15:32:36 <rhochmuth> https://review.openstack.org/#/c/361318/
15:32:51 <rhochmuth> should we +2 that one
15:32:55 <rhochmuth> seems like it is ready to go
15:33:34 <rhochmuth> rbrandt: is this one ready?
15:33:43 <rbrndt> Yeah, I think so
15:34:11 <rhochmuth> tomasz: are you ok with that?
15:34:17 <tomasztrebski> I was trying to test that, but seems like my monasca knowledge ..ehmm..sucks :D
15:34:34 <rbrndt> This one is a little deep into the testing/statistics call
15:34:45 <rbrndt> some weird behavior that needed to be cleaned up
15:34:53 <rhochmuth> if you have questions rbrndt is the expert
15:34:54 <tomasztrebski> well I brough it up for you (community) to look at, because I could not verify if this actually works or not
15:35:06 <tomasztrebski> no point for it to hang
15:35:39 <tomasztrebski> someone with better idea how this should work will do better job testing it I guess while I get more knowledge about it
15:36:03 <rhochmuth> i will test it later today, assuming i can get a devstack build to work again
15:37:51 <rhochmuth> https://review.openstack.org/#/c/315742/
15:38:24 <tomasztrebski> same story from my side
15:38:29 <rhochmuth> rbrndt: so this one is ready too, i'm assuming
15:38:37 <rbrndt> Yeah, this one seems complete to me
15:38:48 <rhochmuth> just looking for another last sanity check
15:39:00 <rhochmuth> i'll try and get that done sometime today/tomorrow too
15:39:08 <rbrndt> tomasztrebski if we have specific queries that didn't work for you, I can try them here
15:40:47 <tomasztrebski> rbrndt I think that I did not spot the difference using any call using group_by with monascaclient
15:40:55 <tomasztrebski> though maybe client lacks something
15:41:00 <tomasztrebski> not really sure
15:41:32 <tomasztrebski> or maybe...again...I don't really understand how this should work
15:41:42 <rbrndt> Client should be up to date with that review, but I can double check that
15:41:52 <tomasztrebski> damn, I don't look very good here in public..telling all that :(
15:42:23 <tomasztrebski> rbrndt thx
15:42:58 <rhochmuth> rbrndt: so is the python-monascaclient all updated upstream and merged?
15:43:00 <haad1> guys does it make sense to put this patch into review process ?
15:43:08 <haad1> https://gist.github.com/haad/e31a94a3a36592952f64f6c2e47e3ab0
15:43:22 <haad1> monasca-persister will not start without it in mitaka branch
15:43:24 <rbrndt> to my knowledge the monascaclient is updated and merged for the group-by changes
15:44:10 <rhochmuth> haad1: if the persister is not starting in mitaka, we would glady accept the review
15:44:29 <haad1> ok then I will do it then
15:44:34 <rhochmuth> thanks
15:45:54 <rhochmuth> #topic Devstack [Openstack] integration
15:46:09 <rhochmuth> https://review.openstack.org/#/c/387195/
15:46:23 <rhochmuth> how much faster is it?
15:46:34 <rhochmuth> but, it seems like a small change
15:46:41 <rhochmuth> i would have merged already if my devstack worked
15:46:49 <tomasztrebski> thx to Artur
15:46:54 <tomasztrebski> without flag: real	54m50.045s user	3m48.016s sys	0m17.780s
15:47:08 <tomasztrebski> with flag: real	38m50.415s user	3m53.688s sys	0m18.136s
15:47:22 <rhochmuth> wow, a lot faster
15:47:27 <tomasztrebski> once using git_clone enters mainstream branch it should speed up more
15:47:53 <rhochmuth> thx artur
15:47:54 <tomasztrebski> right now it only applies to all those projects that are cloned with git_clone wrapper, so all core openstack services
15:48:04 <haad1> https://review.openstack.org/388775
15:48:27 <rhochmuth> thx haad1
15:48:39 <rhochmuth> will review and merge asap
15:50:12 <rhochmuth> tomasz: i would merge https://review.openstack.org/#/c/387195/, but right now my devstack doesnt' come up do to the mysql issue i mentioned in one of the reveiws
15:50:21 <rhochmuth> do you know if that is resolved?
15:50:44 <tomasztrebski> https://review.openstack.org/#/c/388017/
15:50:49 <tomasztrebski> answer lies here
15:51:05 <rhochmuth> i see, so if i test and have success with that review, i can merge it
15:51:12 <rhochmuth> then merge the other two
15:51:24 <rhochmuth> does that seem correct?
15:51:45 <tomasztrebski> yes, I just hope there are no merge conflicts
15:51:52 <tomasztrebski> if they are I will try to fix them ASAP
15:51:53 <rhochmuth> :-)
15:51:55 <rhochmuth> me too
15:52:06 <tomasztrebski> thx
15:52:07 <rhochmuth> ok, np
15:52:26 <rhochmuth> will do another devstack build and see what happens later today
15:52:40 <tomasztrebski> btw: I think that in overall status of monasca devstack has other areas to improve to match kind of standard way openstack proposes
15:53:20 <tomasztrebski> so if anyone has ideas, maybe we can follow up this topic to make monasca deployment there kind of more predictable for guys who now devstack approach and might expect certain things
15:53:27 <tomasztrebski> just an idea
15:53:36 <rhochmuth> i think it is a great area to cover
15:53:47 <rhochmuth> obviousely we weren't experts when we created that deploy
15:53:54 <rhochmuth> devstack deploy that is
15:54:03 <rhochmuth> and it sounds like there are more improvements to be made
15:54:29 <rhochmuth> is that work a separate discussion in the future
15:54:40 <rhochmuth> at a weekly meeting or in barcelona
15:54:43 <tomasztrebski> but now...tomasz arrived and he's breaking stuff up (yeah I know I am modest :P )
15:55:25 <tomasztrebski> it might be, I have one bigger topic in terms of integration (however it is quite big and not really sure if I could work at this)
15:55:51 <tomasztrebski> I can add it to the design session if you like
15:56:08 <rhochmuth> sure, we can possibly cover that too
15:56:23 <rhochmuth> if we don't have time in a specific session we can cover while we are there
15:56:50 <rhochmuth> #topic Where to put examplary formatting files for notification=> https://review.openstack.org/#/c/349097/
15:56:58 <rhochmuth> last one
15:57:01 <tomasztrebski> ok, just added it to the agend (not really sure if that fits there) but you can check out later
15:57:26 <tomasztrebski> yeah, I brought up a question where to put those examples for notifications methods
15:57:43 <haad1> I have one question
15:57:49 <haad1> can anyone tag new version on stable/newton and merge https://github.com/openstack/monasca-api/commit/e92a9524823863290065e781ad64b39086590647 there ?
15:57:49 <haad1> right now stable/newton is not running at all
15:58:42 <tomasztrebski> there are changes open with cherry pick on stable/newton
15:58:54 <tomasztrebski> might be the reason
15:58:54 <rhochmuth> haad1: i can do that
15:59:19 <haad1> rhochmuth: thanks
15:59:28 <rhochmuth> i'll need to look at it closer though after meeting
16:00:22 <rhochmuth> oops i need to end the meeting
16:00:27 <rhochmuth> just saw the clock
16:00:32 <rhochmuth> thanks everyone
16:00:35 <pratid> bye
16:00:35 <tomasztrebski> thx
16:00:36 <Kamil_> bye
16:00:38 <tomasztrebski> cheers and bye
16:00:39 <shinya_kwbt> bye
16:00:42 <bklei> cya
16:00:49 <rhochmuth> #endmeeting