14:59:03 <ddieterly> #startmeeting monasca
14:59:05 <openstack> Meeting started Wed Sep 30 14:59:03 2015 UTC and is due to finish in 60 minutes.  The chair is ddieterly. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:59:07 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:59:09 <openstack> The meeting name has been set to 'monasca'
15:00:28 <ddieterly> #topic rollcall
15:00:33 <ddieterly> o/
15:00:38 <bklei> o/
15:00:39 <witek> o/
15:00:40 <fabiog> o/
15:00:52 <ddieterly> #topic agenda
15:01:15 <ddieterly> roland is out today, so, no particular agenda in mind
15:01:28 <ddieterly> anyone have something for the agenda?
15:01:34 <bklei> i can give twc perf/vertica update
15:01:42 <fabiog> I have an update on Congress
15:01:58 <ddieterly> ok, cool. let's start with bklei
15:02:13 <ddieterly> #topic perf/vertica update
15:02:26 <witek> i have one organisational topic
15:02:33 <ddieterly> witek: ok
15:02:48 <witek> i think we should use more IRC command from meetbot
15:02:58 <ChristianB> there is an agenda-> https://etherpad.openstack.org/p/monasca-team-meeting-agenda
15:03:02 <witek> to generate minutes automatically
15:03:06 <ddieterly> bklei: take it away please
15:03:20 <bklei> k -- vertica came back and said our perf issue was more an enhancement request, we've escalated and involved our SE, so we have a meeting in the next hour to discuss
15:03:39 <bklei> so no progress yet, but we're rattling cages
15:04:11 <ddieterly> any updates on the errors that you were seeing?
15:04:36 <bklei> no, just saw that one occurrence of bad data, it looked ok from what i could see in the logs
15:04:56 <bklei> i've scraped and if we see it again, will dig further
15:05:05 <bklei> i'm not sure it couldn't have been from a stale agent
15:05:18 <bklei> because i was updating all agents to 1.1.6
15:05:24 <bklei> 1.1.16 that is
15:05:36 <ddieterly> bklei: ok
15:05:40 <bklei> and the timestamp looked earlier than my exercise
15:05:55 <bklei> so will keep an eye out for it
15:06:05 <ddieterly> how could the agent cause that error?
15:06:49 <bklei> we had some nodes that weren't controlled by puppet that may have been really old, like pre millis timestamp old
15:07:10 <bklei> not sure, but now that i know we have all new agents, will see if it happens again
15:07:13 <ddieterly> so, lack of millis might be the problem?
15:07:51 <bklei> just reaching for straws, but we have seen ancient processes running even after an agent update where the restart doensn't really restart
15:08:05 <ddieterly> ok
15:08:12 <bklei> but -- would be nice to know how to 'properly' recover
15:08:21 <bklei> and not kill a whole batch of metrics
15:08:25 <ddieterly> any more info on perf/vertica or should we move on to fabiog?
15:08:41 <bklei> ok to move on, will update next week after meeting today
15:08:56 <ddieterly> yea, not sure how to delete selectively from kafka
15:09:10 <bklei> i'd love to know how to do that :)
15:09:19 <bklei> anyone that knows, please educate me
15:09:34 <ddieterly> have you done any research n that?
15:09:50 <bklei> rm -rf :)
15:09:57 <witek> :)
15:10:01 <fabiog> yeah
15:10:09 <fabiog> but with sudo ;-)
15:10:14 <bklei> but that's no bueno in prod
15:10:21 <tomasztrebski> and for specific or all topics :D
15:10:36 <bklei> all topics is what i did
15:11:03 <bklei> #ddieterly i'll open a bug with the details from yesterday
15:11:06 <bklei> and we can watch it
15:11:16 <ddieterly> we could write a filter that sits in between 2 kafka instances and filters out stuff
15:11:52 <bklei> i guess, would have to think about that
15:12:14 <ddieterly> or dump, filter, and reload
15:12:34 <bklei> yeah -- removing the bad msg from the batch, retry
15:12:37 <bklei> that would be good
15:13:04 <ddieterly> there's probably a solution already out there, just need to find it
15:13:24 <bklei> agreed, lots of kafka experts out there
15:13:32 <bklei> i'm sure there's a way to clean a bad msg
15:13:59 <ddieterly> i hear that kafka is attracting the interest of other openstack projects, so the more we can help with expertise, the better
15:14:11 <bklei> i'm afraid to directly do much of anything with kafka, i break things when i start playing
15:14:17 <bklei> agree
15:14:38 <ddieterly> ok, should we cut over to fabiog?
15:14:42 <bklei> sure
15:14:50 <ddieterly> take it away fabiog
15:15:00 <fabiog> ok, I jus followed up to an AI I had
15:15:31 <fabiog> I contacted the Congress PTL and team for a meeting during Summit
15:16:02 <fabiog> they are going yo see if they can dedicate some time out of the 3 sessions they have
15:16:19 <fabiog> otherwise we need to find another space
15:16:52 <ddieterly> do you have space now?
15:17:02 <fabiog> I will check if Cisco has a dedicated room, but I would like you folks to investigate that too
15:17:51 <fabiog> also I think it would be good to spend some f2f time at summit for people that are going ...
15:17:57 <ddieterly> ok, i would assume roland will be following up with that
15:18:10 <ddieterly> who is going? i am not
15:18:24 <fabiog> anyway I will report on that next week or so and provide a prelim agenda
15:18:26 <bklei> i'm not
15:18:39 <witek> Martin is going and myself
15:18:47 <ddieterly> cool
15:18:51 <fabiog> cool
15:18:59 <ddieterly> fabiog: anything else?
15:19:00 <fabiog> I am going
15:19:07 <fabiog> nope, that's iy
15:19:10 <fabiog> it
15:19:26 <ddieterly> ok, witek take it away
15:20:13 <witek> well, our weekly minutes don't look well :(
15:20:30 <ddieterly> were are the minutes?
15:20:40 <witek> meeting minutes
15:20:48 <witek> http://eavesdrop.openstack.org/meetings/monasca/2015/monasca.2015-09-23-15.00.html
15:21:07 <witek> there are a couple of commands which generate entries there
15:21:14 <witek> https://wiki.debian.org/MeetBot
15:21:27 <witek> we should try to use them, when possible
15:22:08 <ddieterly> ok, roland is usually all over this
15:22:17 <bklei> the log shows the dialog -- is that not enough?
15:22:21 <ddieterly> looks like anyone can use the meetbot commands
15:22:49 <witek> yes, at least part of them
15:23:11 <ddieterly> so what topic are we on now?
15:23:46 <ddieterly> looks like topic, info, and action are the most used
15:24:00 <ddieterly> so, anyone can do info and action
15:24:06 <ddieterly> i can do topic
15:24:51 <ddieterly> witek: did you have a topic at this point?
15:25:04 <witek> yes
15:25:40 <ddieterly> may we know it please ;-)
15:26:04 <witek> oh, i misunderstood you
15:26:21 <witek> well, this week we have product delivery
15:26:32 <witek> so most of our resources are blocked
15:26:40 <ddieterly> at fujitsu?
15:26:59 <witek> for test division
15:27:10 <witek> yes at Fujitsu
15:27:19 <ddieterly> #topic product block at fujitsu test division
15:27:22 <ddieterly> viola
15:27:38 <witek> :)
15:27:50 <ddieterly> anything else?
15:28:12 <ddieterly> fabiog: do we need an action item for your needs?
15:28:22 <tomasztrebski> nice, anyway I think what witek is trying to say is that we can't find a nice amount of time to bring monasca-log-api to the public
15:28:22 <fabiog> no
15:28:28 <tomasztrebski> due to limited resources
15:28:34 <fabiog> you can record it
15:28:40 <fabiog> I will follow up
15:28:56 <ddieterly> fabiog: what is the action item, please?
15:29:01 <witek> that's true
15:29:18 <bklei> #fabiog you need any more data from me on twc prod metrics for ceilosca profiling?
15:29:36 <ddieterly> tomasztrebski: i see
15:30:27 <tomasztrebski> anyway do you think that would be ok to make a change at gerrit as a initial or a draft ?
15:30:43 <tomasztrebski> just to let you guys look at it, maybe even test it ?
15:30:48 <ddieterly> tomasztrebski: yes, that would be fine
15:31:23 <ChristianB> can you please distribute the info about the "amount of metrics" in production?
15:32:01 <ddieterly> ChristianB: whom are you asking?
15:32:04 <witek> #action tomasztrebski pushes python monasca-log-api port to review
15:32:18 <ddieterly> witek: nice
15:32:39 <ChristianB> probably the guys from TWC?
15:32:41 <bklei> christianb: here's what we have after ~5 months of monasca running in prod
15:32:48 <bklei> 977,709 unique metrics
15:32:48 <bklei> 1,523,348 max # of measurements per unique metric
15:32:48 <bklei> 31,132,516,526 total rows in our vertica measurements table
15:33:05 <bklei> could get more detail if needed
15:33:28 <tomasztrebski> ChristianB: that was a question for the community in general
15:33:36 <bmotz> out of curiosity, what sort of sample rate are you using on those metrics?
15:34:07 <bklei> mostly default monasca-agent setting, collecting every 15 seconds
15:34:20 <bklei> which ends up being ~3K metrics/sec
15:34:24 <bklei> last i checked
15:34:34 <bmotz> thanks
15:34:49 <ChristianB> thanks!
15:34:56 <ddieterly> i think that twc is the only production instance at this point
15:34:57 <bklei> np, let me know if u need more detail
15:35:15 <ddieterly> tomasztrebski: you guys in production with monasca?
15:35:41 <bklei> fwiw -- twc loves monasca, we did a demo to our internal teams at twc, showing them what it'll look like as MaaS last week at a conf, they really want it
15:35:43 <witek> ddieterly: no
15:36:10 <bmotz> bklei - is that with Vertica for the back end?
15:36:19 <bklei> yes bmotz
15:36:43 <ddieterly> ok, next topic?
15:37:00 <ddieterly> #topic monasca devstack integration
15:37:15 <ddieterly> so, i've been writing the devstack integration plugin
15:37:29 <ddieterly> its a bash script that installs monasca with the rest of devstack in a vm
15:37:46 <ddieterly> i'm making good progress
15:38:09 <ddieterly> worried that a single vm will  not be sufficient for monasca + devstack
15:38:38 <bklei> not enough horsepower?
15:38:49 <ddieterly> i put up some reviews, but i'm going to hold off with more until i have a complete working solution
15:38:54 <ddieterly> bklei: yea
15:39:24 <ddieterly> if anyone is a bash expert and wants to comment on the reviews, that would be great
15:39:42 <bklei> not an expert, but would like to review
15:40:01 <ddieterly> ok, maybe all start putting up something as wip
15:40:12 <bklei> cool
15:40:20 <ddieterly> i don't like doing that because all the commits have to be squashed into 1
15:40:37 <bklei> don't rush it on my account
15:41:15 <ddieterly> #action ddieterly will push up review as wip for monasca devstack integration
15:41:28 <ddieterly> witek: cool, heh?
15:41:36 <witek> i like ity
15:41:49 <ddieterly> ok, next topic
15:41:56 <tomasztrebski> #topic metrics in monasca-agent, control
15:42:11 <ddieterly> tomasztrebski: you have the floor
15:42:32 <tomasztrebski> can you tell us how granular or tight control can we get in context of metrics that are being loaded in monasca-agent ?
15:42:55 <tomasztrebski> we know for sure that we can disable a set of them, but would it be somehow possible to disable some of them from the set ?
15:43:15 <tomasztrebski> perhaps we've missed something in there, because we couldn't locate anything helpful ?
15:43:23 <ddieterly> #topic metrics in monasca-agent, control
15:44:15 <ddieterly> the agent has a default set of metrics and then the rest are controlled by plugins
15:44:50 <tomasztrebski> yeah, but those plugins are loaded with set of metrics, so would it be only way to modify plugins to somehow disable some of them ?
15:45:11 <tomasztrebski> aren't there any solution to support that
15:45:12 <tomasztrebski> ?
15:45:29 <ddieterly> you can edit the config files and delete the metrics that you don't want
15:46:08 <ddieterly> it is controlled thru config files
15:46:39 <tomasztrebski> I see, well I think we will go from it and see where it leads, I hope that somewhere nice , it was good to ask to make sure anyway, thank you
15:46:50 <ddieterly> np
15:46:58 <ddieterly> if you need more help, just email us
15:47:17 <ddieterly> can we move to next topic?
15:47:24 <tomasztrebski> yes, tgx
15:47:26 <tomasztrebski> *thx
15:47:28 <ddieterly> (only chair can change topic btw)
15:47:38 <ddieterly> #topic code reviews
15:47:46 <tomasztrebski> whoops :)
15:48:06 <ddieterly> so, we'd like to have more code reviews from others other than core
15:48:20 <ddieterly> is there something we can do to help get more reviewers?
15:48:57 <witek> #help support by code review needed
15:49:14 <ddieterly> witek: lol, thanks
15:49:15 <tomasztrebski> well, I am sometimes taking a peek at some of you changes and if there's something meaningful to add, I do so... at least in the context of the code itself
15:49:47 <tomasztrebski> witek: you rock :)
15:49:52 <ddieterly> +1
15:50:36 <ddieterly> any ideas for helping make code reviews more accessible?
15:50:56 <ddieterly> is there something that the core team can do to improve participation?
15:51:41 <tomasztrebski> I think one good idea would be to invite people we'd remember at the time of adding reviews ? I mean, let's say one was involved in monasca-common (python, java) somehow so let's add him to take a look
15:51:57 <tomasztrebski> IMHO, it wont hurt to have more guys there
15:52:19 <fabiog> ddieterly: maybe we should discuss in the meeting some of the important patches that need to be reviewed. We were doing this in Keystone
15:52:24 <ddieterly> #idea add previous code reviewers to reviewers list on code review submission
15:52:43 <ddieterly> fabiog: ok
15:52:47 <bklei> +1 fabiog, advertising here is a start
15:53:01 <tomasztrebski> fabiog: +1
15:53:12 <bklei> and in the monasca irc channel during the week
15:53:14 <ddieterly> fabiog: ok
15:53:27 <fabiog> bklei: is more than advertising you can explain what the patch is about and how it works, it would reduce the time to digest the change
15:53:55 <bklei> sure, good comment in the review header helps
15:54:06 <bmotz> also routinely having tests in patches may help
15:54:21 <ddieterly> #idea discuss patches in monasca weekly meeting
15:55:20 <ddieterly> bmotz: agreed
15:55:39 <ddieterly> 5 mins left
15:55:47 <ddieterly> any other topics need discussing?
15:55:51 <tomasztrebski> just one
15:56:01 <tomasztrebski> https://review.openstack.org/#/c/227709/
15:56:18 <tomasztrebski> can someone help with this ? it is pure jave here but pep8 is failing
15:56:45 <ddieterly> what is the pep8 error?
15:56:50 <tomasztrebski> what surprises me the most is the fact that this change depends on mine (already merged) (also java) that had no issue with pep8
15:57:17 <bmotz> Error is "git commit title ('Fix for decimal separator during type conversion.') should not end with period"
15:57:30 <ddieterly> your commit msg line is too long
15:57:46 <ddieterly> and the title ends in a period
15:57:54 <tomasztrebski> so that might a reason for failure ? ok, that's good to know, I'll leave Łukasz a comment to fix that
15:57:54 <bmotz> you can click through from the gerrit review to the output of the PEP8 job on the Jenkins instance
15:57:57 <bmotz> http://logs.openstack.org/09/227709/2/check/gate-monasca-common-pep8/2a40f34/console.html
15:58:26 <tomasztrebski> omg, I dont know how I missed that....my bad
15:58:36 <bklei> :)
15:59:28 <tomasztrebski> and could anyone give a final vote for https://review.openstack.org/#/c/220914/, it's been already aproved by some of guys ( nice :) )
15:59:31 <tomasztrebski> ?
15:59:41 <tomasztrebski> and that's it...really...no more topics for today :D
16:00:11 <ddieterly> tomasztrebski: does that help?
16:00:29 <ddieterly> ok, time to end the meeting
16:00:45 <witek> ddieterly: thank you
16:00:50 <tomasztrebski> yes, thank you
16:00:59 <ddieterly> your welcome :-)
16:01:08 <ddieterly> #endmeeting