15:00:14 <rhochmuth> #startmeeting monasca
15:00:15 <openstack> Meeting started Wed Apr 20 15:00:14 2016 UTC and is due to finish in 60 minutes.  The chair is rhochmuth. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:17 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:18 <rhochmuth> o/
15:00:19 <openstack> The meeting name has been set to 'monasca'
15:00:19 <vhoward> see you in austin ;)
15:00:25 <witek> hello
15:00:27 <rbak> o/
15:00:31 <hosanai> o/
15:00:39 <slogan> morning
15:00:45 <shinya_kwbt> o/
15:00:45 <cbellucci> o/
15:00:49 <Kamil> o/
15:00:58 <bklei> o.
15:01:00 <bklei> o/
15:01:12 <rhochmuth> agenda has been posted
15:01:14 <rhochmuth> Agenda for Wednesday April 20, 2016 (15:00 UTC)
15:01:14 <rhochmuth> 1. Patches
15:01:14 <rhochmuth> 1.1 https://review.openstack.org/#/c/298863/
15:01:14 <rhochmuth> 1.2 https://review.openstack.org/#/c/301355/
15:01:14 <rhochmuth> 1.3 https://review.openstack.org/#/c/307963/ (brad curious about status)
15:01:14 <rhochmuth> 1.4 https://review.openstack.org/#/c/289675 (same)
15:01:14 <rhochmuth> 2. Heads up on ovs plugin (router bandwidth metrics)
15:01:15 <rhochmuth> 2.1 https://review.openstack.org/#/c/306621 (not fully baked)
15:01:15 <rhochmuth> 3. https://review.openstack.org/#/c/302660/
15:01:20 <rhochmuth> https://etherpad.openstack.org/p/monasca-team-meeting-agenda
15:01:35 <rhochmuth> hi everyone
15:02:04 <rhochmuth> so we have a few reviews that have been asked about
15:02:04 <bklei> good morning!
15:02:21 <rhochmuth> so, let's go through that list and then discuss other items
15:02:28 <rhochmuth> https://review.openstack.org/#/c/298863/
15:02:31 <rbak> Those first couple reviews are mine.
15:02:44 <rbak> That one just needs to get merged
15:02:56 <rbak> It's been sitting around for two weeks with several +1
15:03:05 <rhochmuth> i see, looks like there are a bunch of reviews and +1s
15:03:16 <rhochmuth> i reviewed this morning, so it looks good to me
15:03:20 <rhochmuth> i can merge now
15:03:25 <rbak> Great, thanks
15:03:51 <rhochmuth> then there is https://review.openstack.org/#/c/301355/
15:03:52 <rbak> The next one is the more complicated patch
15:04:11 <rbak> This is the thread-pool replacement patch
15:04:11 <rhochmuth> i looked at this one too
15:04:28 <rhochmuth> there were some quesitons, but i'm not sure it has been resolved
15:04:38 <rbak> I just wanted to talk about this one since it seems like there were some concerns
15:04:49 <rhochmuth> yes
15:04:58 <rhochmuth> is rbrabdt here?
15:05:03 <rhochmuth> rbrandt?
15:05:24 <rhochmuth> also michael hoppal and joe keen would be good
15:05:43 <rhochmuth> anyway, please ask questions
15:05:48 <rbak> I'm open to making changes, I just want to keep this moving forward
15:05:52 <rhochmuth> not sure how much help i'll be
15:06:14 <rhochmuth> so, i'll need to ping all those names i mentioned
15:06:16 <rbak> I just need clarification on what the concerns are so I can fix them.
15:06:49 <rhochmuth> i'll ping them, we were super busy last week on our own releases
15:07:02 <rhochmuth> so, hopefully they can take another look today
15:07:24 <rbak> That's fine.  I'll ping rbrandt later since it doesn't seem like he's on
15:07:48 <rbak> We can move on for now.
15:08:07 <bklei> the next two patches are rbrandt's  -- and i was curious on status
15:08:11 <rhochmuth> actually i did this really cool thing
15:08:27 <rhochmuth> i walked down the aisle and talked to them using my voice
15:08:35 <bklei> strange -- that works?
15:08:43 <rhochmuth> they are connecting in now
15:08:47 <mhoppal> only sometimes with him
15:08:53 <rhochmuth> rbrandt u there
15:08:53 <bklei> :)
15:08:59 <rhochmuth> hoppal u there
15:09:00 <14WAATBIX> I'm apparently under this name
15:09:04 <mhoppal> im here
15:09:06 <ddieterly> o/
15:09:08 <bklei> nice name
15:09:09 <14WAATBIX> no idea...
15:09:15 <rhochmuth> is that you rbrandt
15:09:19 <14WAATBIX> yeah
15:09:22 <rhochmuth> ok
15:09:32 <rhochmuth> rbak is asking about https://review.openstack.org/#/c/301355/
15:09:41 <rhochmuth> this is the muleiprocessing review
15:10:01 <rhochmuth> he's trying to understand next steps and the feedback he;'s received so far
15:10:21 <14WAATBIX> Ah, yes. So the issue at the moment is the lost data if a check fails.
15:10:58 <14WAATBIX> Right now, as far as I am aware, we don't lose data except for the failed check itself
15:11:06 <rbak> That's not correct
15:11:18 <14WAATBIX> jkeen actually may know more of the specifics
15:11:24 <rbak> Right now, worst case, the agent hangs and everything is lost.
15:11:35 <rbak> Until the agent is restarted by hand.
15:11:59 <rbak> This patch doesn't actually change any behavior other than to allow the pool to restart as intended.
15:12:00 <bklei> (and we've hit that a bunch and lost alot of data)
15:12:23 <rbak> The triggering condition and the result of the restart are still the same.
15:12:30 <rhochmuth> so, with this patch, you fixed one big problem
15:12:39 <14WAATBIX> jkeen was exploring the code to see what we used to do and what this new code does
15:12:51 <rhochmuth> are there any other downsides
15:13:22 <rhochmuth> sorry, meant downsides or side effects
15:13:36 <rbak> The only new downside I'm aware of is that it's noisy.  Restarting the pool can throw a lot of errors.
15:13:51 <rhochmuth> you mean the log file
15:14:04 <rbak> Yes.
15:14:12 <14WAATBIX> I know when I was testing it we regularly saw failures in the checks
15:14:23 <rbak> Actually though, I think I finally realized what the miscommunication here is.
15:14:29 <14WAATBIX> which is what prompted the comments
15:14:45 <rbak> There's two different ways this can be triggered, and we're talking about different conditions.
15:15:31 <rbak> Previously, the pool restarted cleanly and no data was lost, unless a check was truly stuck in which case it never restarted and the entire agent hung.
15:16:23 <rbak> In that worst case this version is better.  If a check is just taking too long and the pool restarts, I suppose this does lose more data than before.
15:16:51 <rbak> 14WAATBIX: Why don't I talk to you after the meeting so we don't eat up too much time in the meeting.
15:17:07 <rhochmuth> should also get joe and michael
15:17:15 <14WAATBIX> Sure, I think we're getting synced now
15:17:24 <rhochmuth> please schedule a meeting with all three and anyone else that wants to attend
15:17:34 <rbak> Sure, I can do that.
15:17:40 <rhochmuth> thx rbak
15:18:07 <rhochmuth> next review
15:18:09 <rhochmuth> https://review.openstack.org/#/c/307963/
15:18:33 <bklei> that's 14WAATBIX's -- just wondering status
15:18:44 <bklei> i'm ready to test it locally when told ot
15:18:47 <bklei> to
15:19:13 <14WAATBIX> It should be ready for testing at the moment, I'm trying to get it set up in a large scale environment
15:19:27 <bklei> perfect -- will pull in this am and kick the tires
15:19:32 <rhochmuth> it didn't pass gates
15:19:39 <rhochmuth> is it really ready?
15:19:51 <14WAATBIX> its vertica code changes so the gate shouldn't be affected
15:20:03 <14WAATBIX> the python api tests failed in an alarm test
15:20:08 <rhochmuth> maybe just needs to be rechecked
15:20:09 <14WAATBIX> not sure why, but investigating
15:20:14 <bklei> that seems like a flaky test
15:20:27 <14WAATBIX> it does, hopefully I can rewrite it
15:20:47 <bklei> thx -- will give you feedback after i test your patch
15:20:58 <rhochmuth> thx
15:21:09 <14WAATBIX> thnx
15:21:20 <rhochmuth> https://review.openstack.org/#/c/289675
15:21:34 <bklei> just wondering if that dude is close
15:21:38 <bklei> (multiple metrics)
15:21:45 <rhochmuth> yup
15:22:02 <14WAATBIX> It might need to be rebased on the metric list changes
15:22:05 <rhochmuth> 14WAATBIX
15:22:07 <rhochmuth> ?
15:22:28 <rhochmuth> so when metrics list merges you'll need to rebase
15:22:37 <bklei> i'd guess -- poor 14WAATBIX
15:22:39 <14WAATBIX> I think so, but I'm not 100% sure
15:22:42 <rhochmuth> or which ever goes first
15:23:08 <rhochmuth> besides the rebase, are both reviews ready to go independently?
15:23:16 <14WAATBIX> I think so
15:23:37 <rhochmuth> bklei we are really interested in getting feedback on this one, namely performance analysis
15:23:55 <rhochmuth> hopefully, we've significantly improved things
15:24:03 <rhochmuth> this one will also impact any queries that you are doing
15:24:18 <bklei> we can provide feedback on perf -- but not so much pre-merge :(
15:24:21 <rhochmuth> for example, grafana will need to be enhanced to use the group_by query parameter
15:24:28 <bklei> our best perf tests are in prod
15:24:35 <bklei> oh for sure
15:24:49 <bklei> that's gonna make some graphs scream
15:24:50 <rhochmuth> well, you might want to test this one before production
15:25:07 <rhochmuth> a few more moving parts
15:25:15 <rhochmuth> the level of complexity is much higher
15:25:17 <bklei> we can do a limited amount in the lab, but our best historic results are from prod, but we can get an indication
15:25:41 <bklei> (we're talking multiple metrics -- not limit, yes?)
15:25:55 <rhochmuth> we did test this change against the ceilosca code
15:26:04 <rhochmuth> and saw a huge perf improvement
15:26:16 <rhochmuth> but, unfortuately no real numbers yet
15:26:26 <bklei> i'm excited about this one
15:26:46 <rhochmuth> so, after this change, i'm not sure about the "caching" solution
15:27:03 <rhochmuth> multiple metrics might actually work better than pursuing caching
15:27:25 <bklei> perhaps, things like metric-list would benefit, and that's a pig for us
15:27:48 <rhochmuth> ok, sounds like time to move on
15:27:48 <bklei> metric-list is 1GB of data in our 'infra' project
15:28:14 <rhochmuth> but, with start/end time you limit that down to something managable
15:28:15 <rhochmuth> right
15:28:35 <bklei> not as much as it did before -- with grafana 2
15:28:40 <rhochmuth> also, do you have automated cron josb to remove old data
15:28:40 <bklei> and even 'current' is huge
15:28:44 <bklei> yes
15:28:56 <bklei> but infra keeps 13mos
15:29:04 <bklei> so prune doesn't help that project much
15:29:09 <rhochmuth> i see
15:29:37 <rhochmuth> is there a way to control the range of data returned in grafana 2
15:29:43 <rhochmuth> i didn't see any
15:29:56 <bklei> rbak can comment -- but i don't think so
15:29:58 <rhochmuth> it should just be the current range
15:30:00 <rhochmuth> right?
15:30:05 <rhochmuth> last hour
15:30:08 <rhochmuth> last day
15:30:19 <rbak> I looked away for a moment.  I'm not sure I followed the conversation
15:30:22 <bklei> sounds like the data source doesn't have visibility to graph range
15:30:47 <bklei> (providing timestamp to metric-list)
15:30:58 <rbak> Templating doesn't have access to the time-range
15:31:04 <rhochmuth> so, when you are displaying metric names for example, can you limit the names that are returned to the current time range set in grafana 2
15:31:04 <rbak> The graphs do.
15:31:21 <rbak> It depends on how you do it.
15:31:30 <rhochmuth> and templating is the part that dispays the names and dimensions?
15:31:39 <rbak> If you use templating, no
15:32:13 <rbak> Templating is the feature that queries for dimensions and lets use insert them into the graphs dynamically.
15:32:28 <rbak> I'm not sure if that answers your question
15:33:08 <rhochmuth> not sure, but we can move one
15:33:12 <rhochmuth> how about, Heads up on ovs plugin (router bandwidth metrics)
15:33:21 <rhochmuth> what is this about
15:33:24 <rhochmuth> i saw the review
15:33:28 <bklei> yeah -- let's call this an advertisement
15:33:37 <bklei> it's not ready, just want to gain visibility/support
15:33:45 <bklei> our customers really want BW for virtual routers
15:33:58 <bklei> we could do this locally, but thought others may want it too
15:34:08 <bklei> people that use OVS/neutron
15:34:08 <rhochmuth> yes, this could be very interesting
15:34:28 <bklei> it's pretty much working needs some cleanup though
15:34:33 <rhochmuth> so, syd will be interested and we're going to have a session next week with the neytron team too
15:34:49 <rhochmuth> so, is this a discussion that we want to have next week in austin?
15:35:06 <bklei> ok -- i'll add syd to the review -- and i'll join that discussion if you let me know when/where
15:35:35 <rhochmuth> the fourth session on wednesday morning is the combined monasca/neutron/network discussion
15:35:40 <bklei> perfect
15:35:49 <rhochmuth> it will be only 40 minutes long
15:36:00 <rhochmuth> but, we can also discuss more while we are there
15:36:17 <bklei> sure, sounds good
15:36:18 <rhochmuth> but, i'm very interested in this development
15:36:25 <bklei> oh -- one question on my review
15:36:30 <rhochmuth> so, i'll start taking a look at it
15:36:34 <rhochmuth> sure go ahead
15:36:36 <bklei> is it required to add a 'detector'?
15:36:41 <bklei> we don't use that process here
15:36:44 <rhochmuth> what do you mean?
15:36:56 <bklei> for all plugins -- add a detector
15:37:04 <rhochmuth> detection plugin
15:37:09 <bklei> right -- that
15:37:30 <rhochmuth> would be nice if it makes sense
15:37:32 <bklei> is that required for all plugins?
15:37:38 <rhochmuth> not all plugins have deteciton plugins though
15:37:49 <rhochmuth> so not required
15:37:56 <rhochmuth> the detection plugins configure the agent
15:37:57 <bklei> ok -- i'll add it if folks request
15:38:09 <rhochmuth> and are run seperately, usually at install/deploy time
15:38:22 <rhochmuth> sounds good
15:38:28 <slogan> I stepped away for a bit, yes, plenty to talk about Wednesday
15:38:33 <rhochmuth> i'm not sure how detection woudl work in this case
15:38:33 <bklei> yeah, and we don't use that process, but can add it if folks want it
15:38:45 <bklei> yeah -- i'm not either
15:38:59 <rhochmuth> hi slogan, yes
15:39:22 <rhochmuth> you might want to take a look t that review if you haven't already
15:39:27 <bklei> https://review.openstack.org/#/c/306621
15:39:30 <slogan> will do
15:39:42 <rhochmuth> hre is the link, https://review.openstack.org/#/c/306621/
15:40:01 <bklei> (it's not fully baked -- but close)
15:40:33 <bklei> but early comments welcome
15:40:58 <bklei> ok -- i think that's it for me
15:41:14 <rhochmuth> https://review.openstack.org/#/c/302660/
15:41:22 <rhochmuth> migrate from mysqldb to pymysql
15:41:27 <rhochmuth> it looks good to me
15:41:38 <rhochmuth> just need to run it once
15:41:49 <witek> there seems to be a problem with requirements
15:42:11 <rhochmuth> witek: so that sounds like it isn't ready
15:42:15 <witek> but I did not investigate
15:42:36 <witek> the gate job was successful previously
15:42:41 <Kamil> we are not so sure from where this error is comming
15:42:43 <witek> I just changed the commit message
15:42:51 <Kamil> ContextualVersionConflict: (Babel 2.3.3 (/home/jenkins/workspace/gate-monasca-agent-python27/.tox/py27/lib/python2.7/site-packages), Requirement.parse('Babel!=2.3.0,!=2.3.1,!=2.3.2,!=2.3.3,>=1.3'), set(['oslo.log']))
15:43:50 <rhochmuth> it could be that something changes upstream
15:43:56 <rhochmuth> that is impacting the agent
15:43:57 <rhochmuth> not sure
15:44:11 <hoppalm> i think it is, all of our agent reviews are having that issue
15:44:13 <witek> that is also my first assumption
15:44:37 <hoppalm> i was going to try to look into it sometime today if i get a chance
15:44:47 <rhochmuth> that hoppalm
15:44:53 <rhochmuth> thx
15:44:59 <rhochmuth> not that
15:45:00 <witek> would be great, thanks michael
15:45:12 <rhochmuth> that hoppalm is such a great guy
15:45:17 <witek> :)
15:45:21 <bklei> hotpalm :)
15:45:35 <hoppalm> meanwhile roland is laughing as he types that
15:45:40 <bklei> :)
15:45:54 <witek> we know him too much :)
15:46:10 <rhochmuth> witek: craig is reviewed tomasz's review
15:46:17 <witek> great!
15:46:27 <rhochmuth> he sent me an email this mroning with some general questions
15:46:36 <rhochmuth> but he is diving in on this
15:46:52 <witek> thanks a lot for that!
15:47:16 <rhochmuth> i'm not sure i'll run into him here today, but i'm expecting he'll post some comments to the review soon
15:47:45 <witek> that would be best, or if longer, per email to me and Tomasz
15:48:13 <rhochmuth> ok
15:48:41 <rhochmuth> so, are there any other reviews or questions?
15:49:20 <rhochmuth> for those in austin, i'm wondering if we should have dinner together one night
15:49:42 <witek> good idea
15:49:45 <rhochmuth> thursday is probably the best, but i'm open to suggestions
15:49:45 <bklei> that would be great
15:49:54 <rhochmuth> just as long as bogdan doesn't pick the place
15:50:03 <witek> haha
15:50:19 <rhochmuth> actually, that was fun
15:50:20 <Kamil> Oo
15:51:17 <rhochmuth> i'll send an email out and try and coordinate
15:51:33 <rhochmuth> shinya, you are going to be there
15:51:54 <shinya_kwbt> Yes I will go to austin!
15:51:59 <rhochmuth> hosanai, you are going to be there too?
15:52:01 <rhochmuth> awesome
15:52:14 <shinya_kwbt> I'm looking forward to seeing monasca folks.
15:52:15 <rhochmuth> have you been looking at cassandra, btw
15:52:24 <hosanai> yep
15:52:24 <rhochmuth> me too
15:52:49 <rhochmuth> witek, rbak, bklei, slogan
15:52:50 <shinya_kwbt> Sorry not yet, but I will
15:52:59 <rhochmuth> no problem, was just checking
15:53:19 <rhochmuth> also, as i mentioned, we might want to consider ES too,
15:53:26 <rhochmuth> as that is a part of the logging solution
15:53:34 <rhochmuth> it would help with deploys
15:53:52 <rhochmuth> fabiog will be there too
15:54:09 <rhochmuth> if there is anyone else that is going to austin that i've missed please let me know
15:54:18 <witek> Kouji
15:54:36 <rhochmuth> great
15:54:45 <hosanai> my colleague, daisuke
15:55:09 <rhochmuth> thanks hosanai
15:55:16 <rhochmuth> i'll get him on the email list too
15:55:29 <rhochmuth> we'll coordinate while we are there, but just in case
15:55:30 <hosanai> thanks roland!
15:55:37 <rhochmuth> i wanted to have the names
15:56:06 <rhochmuth> the MOnasca bootcamp is on MOnday
15:56:13 <rhochmuth> if folks are available to help that would be great
15:56:24 <rhochmuth> my plan is to go thorugh an iPython notebook
15:56:32 <rhochmuth> which will have lot's of example of using the API
15:56:37 <rhochmuth> and cover other topics
15:56:38 <slogan> I'll be there
15:56:44 <bklei> ditto
15:56:49 <rhochmuth> it is supposed to he hands-on
15:57:02 <slogan> what kind of help do you anticipate needing?
15:57:03 <witek> I'll be there
15:57:14 <rhochmuth> but given it is only 1 1/2 hour not sure that we can spend 1/2 hour just setting up VMs
15:57:23 <rhochmuth> thanks everyone
15:57:30 <rhochmuth> i'll introduce everyone that is there
15:57:52 <rhochmuth> also, please look at the other sessions invovling monasca
15:58:19 <rhochmuth> twc, fujitsu, cisco all have sessions
15:58:30 <rhochmuth> and don't forget wednesday morning
15:58:38 <rhochmuth> those are all the monasca working sessions
15:59:04 <rhochmuth> there are also some other sessions on monasca or involving monasca in some capacity that are showing up
15:59:08 <slogan> I missed the ones by fujitsu and cisco, I'll go back and look for them
15:59:18 <slogan> awesome
15:59:21 <rhochmuth> fujitsu is on the logging api/service
15:59:31 <slogan> if you have a must see list you are keeping, maybe share that?
15:59:31 <rhochmuth> ciscso one is related to congress
15:59:35 <rhochmuth> and policy management
15:59:55 <rhochmuth> there are also some session on logging that look interesting
16:00:09 <rhochmuth> ok, i think that is done
16:00:14 <rhochmuth> bye-bye everyone
16:00:20 <slogan> see you next week
16:00:21 <rhochmuth> i always cram until the end
16:00:37 <rhochmuth> #endmeeting