15:00:06 <rhochmuth> #startmeeting monasca 15:00:07 <openstack> Meeting started Wed Sep 23 15:00:06 2015 UTC and is due to finish in 60 minutes. The chair is rhochmuth. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:00:08 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:00:10 <openstack> The meeting name has been set to 'monasca' 15:00:15 <rhochmuth> roll call 15:00:25 <fabiog> o/ 15:00:26 <jimbaker> o/ 15:00:27 <rhochmuth> o/ 15:00:31 <bklei> o/ 15:00:34 <qwebirc77657> o/ 15:01:04 <rhochmuth> Agenda looks a little light today at, https://etherpad.openstack.org/p/monasca-team-meeting-agenda 15:01:17 <rhochmuth> 1. Review Gerrit bugs 15:01:17 <rhochmuth> 1. Need more folks contributing to reviews 15:01:17 <rhochmuth> 2. Update on DevStack 15:01:17 <rhochmuth> 3. Update on Tempest Tests 15:01:37 <rhochmuth> Are there other items to discuss 15:01:48 <rhochmuth> How about an update on performance from TWC 15:01:53 <ddieterly> o/ 15:01:53 <rhochmuth> as well as Grafana 15:01:58 <bklei> sure we can do that 15:02:27 <fabiog> I also have an update on Ceilosca 15:02:31 <rhochmuth> is anyone attending from fujitsu? 15:02:40 <rhochmuth> ok, let's talk about ceilosca too 15:02:48 <bklei> can we add timing of next tagging of monasca-agent and pub to pypi? 15:02:49 <witek> yes, but I have to leave earlier 15:04:15 <rhochmuth> ok, we'll discuss agent tags too 15:04:15 <rhochmuth> #topic reviews 15:04:16 <rhochmuth> One area I want to get help on is more reviews 15:04:47 <bklei> are people getting added to reviews, but not reviewing? 15:04:49 <witek> i think, i could help a bit 15:05:00 <rhochmuth> there are a bunch of reviews that I've +1'd, but have been just sitting for a couple of days 15:05:19 <rhochmuth> i would like to see the code flowing 15:05:22 <bklei> twc can be better too 15:06:04 <rhochmuth> in the old days, i would just +2 things, but it align with the overall process better we need multiple companies one each review commit 15:06:24 <bklei> agree 15:06:29 <ddieterly> i think we also needed automated tests 15:06:43 <rhochmuth> ideally, a company shouldn't be +2 their own work, but we're not quite there yet 15:06:46 <ddieterly> we just don't have enough tests to feel confident in submittals 15:06:50 <rhochmuth> but we have to move in that direction 15:06:51 <fabiog> ddieterly: +1 Every patch should have unit test 15:07:19 <ddieterly> i'd also like to see integration tests 15:07:50 <rhochmuth> ddieterly: so the integratino tests is what i'm working on 15:07:54 <rhochmuth> right? 15:08:10 <rhochmuth> that is what tempest tests are for 15:08:10 <ddieterly> yea, not sure how much coverage we have on those 15:08:51 <rhochmuth> well, i think coverage will be good at the API syntactic level 15:09:04 <rhochmuth> and basic functionality 15:09:14 <ddieterly> do the tempest tests use a 'real' db? 15:09:21 <rhochmuth> yes, they do 15:09:26 <rhochmuth> tempest uses the API 15:09:34 <rhochmuth> they are written against the Monasca API 15:09:44 <ddieterly> cool 15:10:02 <rhochmuth> so, getting back to reviews, just need more help in that area 15:10:28 <rhochmuth> witek, are all of tomasz's reviews then ready to get committed? 15:10:59 <witek> i will take a look at them again 15:11:04 <rhochmuth> he has about 5 or 6 reviews sittign in the queue, 15:11:05 <rhochmuth> thanks 15:11:25 <rhochmuth> sorry about the speed, we actually test each commit prior to +2 15:11:29 <rhochmuth> that is the theory 15:11:39 <rhochmuth> so the turn-aroudn time can be gated by testing 15:11:53 <rhochmuth> hopefully, when we have tempest tests up and running this will be automated 15:12:08 <rhochmuth> so, we'll have a little more confidence 15:12:15 <bklei> +1 on that 15:12:25 <fabiog> rhochmuth: in the meantime can you please post a comment when the tests have been done and were successful? 15:12:29 <ddieterly> are we going to require tempest tests for each patch? 15:12:30 <rhochmuth> #topic devstack 15:12:42 <fabiog> rhochmuth: I think this will give more confidence to the other reviewers that the patch works 15:12:42 <rhochmuth> sorry, i changed topics 15:12:54 <fabiog> np 15:12:58 <witek> #help Help is needed on reviews 15:13:20 <rhochmuth> ddieterly i think i need to get the first round of tempest tests done and integrated 15:13:40 <rhochmuth> then we can hopefully have a process where each feature has tests written againt it 15:13:49 <rhochmuth> so, i think the answer is yes 15:13:55 <ddieterly> +1 15:14:03 <jimbaker> +1 15:14:07 <ddieterly> what is the eta on the tempest stuff? 15:14:31 <rhochmuth> i'm going to try and have some code up for review this week 15:14:42 <rhochmuth> i have tests for metrics and alarm definitions complete 15:14:49 <rhochmuth> will work through the rests of the API 15:14:59 <rhochmuth> actually, i don't think everyhting will be done this week 15:15:15 <rhochmuth> but it will be a good start and time to merge and get feedback 15:15:35 <rhochmuth> #topic devstack 15:15:57 <rhochmuth> how is devstack going? 15:15:59 <ddieterly> ok, i'm in the process of converting all the ansible to bash in monasca-api/devstack 15:16:23 <ddieterly> hope to have a complete monasca deployment in devstack in 2-3 weeks 15:16:27 <rhochmuth> sounds exciting 15:16:33 <ddieterly> very 15:16:39 <rhochmuth> :-) 15:16:59 <rhochmuth> so, you've put some reviews up 15:17:10 <ddieterly> yea, already have 15:17:31 <fabiog> can you post it please? 15:17:40 <rhochmuth> #fabiog #ddieterly was asking about using ansible 15:17:47 <rhochmuth> i said can't do that 15:18:00 <Matthias_Christi> what is the reason to convert ansible to bash ? 15:18:01 <rhochmuth> has to be independent of any deployment tool 15:18:14 <ddieterly> https://review.openstack.org/#/q/owner:%22Deklan+Dieterly%22+status:merged,n,z 15:18:45 <rhochmuth> #Matthias_Christi i think the problem is they don't want Ansible in Devstack 15:18:51 <ddieterly> Matthias_Christi: it is the devstack way 15:19:09 <rhochmuth> #fabiog is that your understanding 15:19:17 <ddieterly> bash is the linqua franca of devstack 15:19:41 <jimbaker> for better or for worse... 15:20:25 <ddieterly> yea, c'est la geurre 15:20:25 <fabiog> rhochmuth: yep 15:20:46 <rhochmuth> so, basically 2-3 weeks on devstack, then we'll move onto ci/cd integration 15:20:58 <rhochmuth> ok, next topic? 15:21:08 <ddieterly> hopefully 15:21:15 <rhochmuth> we discussed the tempest tests already 15:21:22 <rhochmuth> so i'll move to performance update 15:21:26 <rhochmuth> #performance 15:21:31 <rhochmuth> #topic performance 15:21:47 <bklei> that's me 15:21:54 <rhochmuth> yes 15:22:10 <bklei> was out last week, in a conf this week, so not much movement, but 15:22:20 <bklei> vertica has agreed current perf is a bug 15:22:28 <rhochmuth> awesome! 15:22:35 <bklei> no fix yet 15:22:44 <rhochmuth> do they have an eta 15:23:04 <bklei> the bug is when 10 or so queries are in parallel 15:23:04 <bklei> no eta yet, pushing them on that 15:23:23 <rhochmuth> what do you mean by 10 queries in parallel 15:23:35 <bklei> asking them to back port to the current version (hot fix), instead of waiting for Q4 major release 15:23:46 <bklei> yes, 10 queries in parellel goes into the toilet 15:24:08 <ddieterly> that explains our puzzlement over the performance at least 15:24:08 <rhochmuth> omg 15:24:15 <bklei> so, i'm waiting on other projection work, until that's fixed 15:24:27 <bklei> (pre-join queries, schema change...) 15:24:32 <rhochmuth> your review is sitting up there, should we wait on that 15:24:40 <fabiog> bklei: wow! and no customers found that before ? 15:24:44 <bklei> exactly ddieterly 15:24:56 <rhochmuth> that switches from inner joins to ... 15:25:02 <bklei> apparently there is one other big customer. but boo on their testing 15:25:04 <rhochmuth> should we wait on that 15:25:05 <ddieterly> i'm surprised that vertica had this bug 15:25:10 <ddieterly> kinda serious 15:25:48 <ddieterly> the vertica luster has been dulled a bit 15:25:58 <rhochmuth> also, would limiting the concurrency to <10 be a work-around 15:26:24 <rhochmuth> doesnt' sound like a good idea, but maybe it is faster 15:26:31 <bklei> the bug# is VER-40005 15:26:45 <bklei> "Time in plan is high when 10+ queries are run concurrently on 3 node cluster" 15:27:01 <bklei> but the engineer thinks it's beyond just a 3 node cluster 15:27:23 <bklei> anyhoo, not much more to say on that till i get an eta 15:27:32 <bklei> will pressure them 15:27:51 <rhochmuth> do you want your review for inner join conversion to just wait then 15:28:09 <bklei> let's wait, not sure it's necessary, may be working around their bug 15:28:25 <rhochmuth> ok 15:28:29 <bklei> thx though 15:28:45 <rhochmuth> #bklei thanks for working through this 15:28:46 <bklei> it wasn't a dramatically better improvement 15:28:49 <bklei> sure 15:29:15 <ddieterly> i think we should still make the change because it is much simpler to implement and reaon about 15:29:17 <rhochmuth> nice to know we know how to code and that the bug was in Vertica 15:29:39 <ddieterly> we just need to make the change globally 15:30:37 <bklei> i'm not opposed to the change #ddieterly, just want a bug fix in vertica before spending much time on tweaking stuff above it 15:30:48 <ddieterly> yea, understood 15:31:01 <rhochmuth> so let's see how fast vertica can respond 15:31:14 <rhochmuth> and then we can measure again with fixes hopeflly 15:31:20 <rhochmuth> and then switch to new code 15:31:20 <bklei> agreed, hope to have better info next week 15:31:31 <rhochmuth> ok, cool 15:31:32 <bklei> yup 15:31:46 <rhochmuth> #topic grafana 15:32:02 <rhochmuth> rbak was adding support to Grafana 2.0 15:32:17 <rhochmuth> #bklei do you know how that is progressing? 15:32:19 <bklei> rbak is sort of here, but in a conf with me -- he's made some progress 15:32:31 <rhochmuth> ok 15:32:43 <bklei> more investigation, no eta or idea of the quantity of work yet. 15:32:54 <rhochmuth> ok 15:32:56 <bklei> but it's sexy, and i'm confident he'll get it done 15:33:12 <rhochmuth> sexy 15:33:14 <bklei> templates are awesome 15:33:15 <ddieterly> bklei: really? 15:33:35 <rhochmuth> ok, we'll check back next week 15:33:39 <bklei> very cool, and it doesn't sound too difficult to add us as a proper 'data source' 15:33:42 <bklei> yup 15:34:01 <rhochmuth> #topic ceilosca 15:34:32 <rhochmuth> #fabiog you are up 15:35:01 <ddieterly> fabiog: you there? 15:35:12 <ddieterly> crickets 15:35:14 <fabiog> rhochmuth: sorry 15:35:36 <fabiog> ok, so we have started to do tests with Ceilosca 15:35:54 <fabiog> and what we found is that Influlx 0.9.1 suffers all sort of issues 15:36:06 <fabiog> it times out with the Persister 15:36:15 <bklei> :( 15:36:17 <fabiog> and it prevents to store large amounts of data 15:36:29 <fabiog> but when we updated to 0.9.4.1 it worked really well 15:36:43 <fabiog> and we didn't have to change a single line of code or configuration 15:36:44 <rhochmuth> problem solved :-) 15:36:49 <rhochmuth> are we done 15:36:50 <fabiog> just switched the package 15:36:56 <ddieterly> fabiog: did you have a chance to test the api end points with 9.4? 15:36:56 <fabiog> rhochmuth: kind of 15:37:15 <ddieterly> i'm concerned that the api might be broken with 9.4 15:37:21 <fabiog> ddieterly: yes, because the ceilometer call in returns call the monasca API 15:37:25 <rhochmuth> so, there were changes i thoguht in .9.3 that were incompatible with .9.1 15:37:27 <fabiog> and we get data back 15:37:47 <rhochmuth> #bmotz was looking at this 15:37:48 <ddieterly> we need to check all the functionality of the end points 15:38:01 <fabiog> so I would like to see the mini-mon installation to be upgraded to use 0.9.4.1 15:38:07 <fabiog> would that be possible? 15:38:15 <ddieterly> fabiog: go for it 15:38:36 <rhochmuth> i don't have a problem updating 15:38:43 <fabiog> ddieterly: ok, so I will try to figure out what and where to change and I will submit a patch 15:38:44 <rhochmuth> just have to address any issues that occur 15:38:58 <rhochmuth> influxdb keeps changing their api, even though they are supposed to be frozen 15:39:00 <ddieterly> it is a very simple change in the ansible vars 15:39:13 <rhochmuth> yeah, so the change in Ansible is trivial 15:39:20 <ddieterly> the problem is finding all the problems 15:39:34 <ddieterly> goes back to comprehensive integration tests 15:39:51 <rhochmuth> #bmotz isn't here today, but you might contact directly to see what he might have uncovered 15:40:02 <fabiog> ddieterly: right. In the ceilosca case seems to work fine, so if there are bugs are not major 15:40:08 <rhochmuth> he was also looking at swithching from http to binary protocol 15:40:12 <fabiog> ddieterly: meaning stop working 15:40:51 <fabiog> rhochmuth: ok, I will get in touch with bmotz 15:40:57 <rhochmuth> thx 15:41:09 <ddieterly> does cceilosca exercise a good portion of the api? 15:41:11 <rhochmuth> if it is all working, then let's switch 15:41:24 <rhochmuth> we can also help to get some testing here 15:41:28 <rhochmuth> possibly 15:41:32 <fabiog> ddieterly: pretty much queries with dimensions 15:41:36 <rhochmuth> i'll need to coordinate with our test team 15:41:40 <fabiog> ddieterly: and post of new measurements 15:41:41 <ddieterly> alarm history, metrics, measurements, statistics 15:41:59 <fabiog> ddieterly: no alarm history or anything related to alarms 15:42:03 <fabiog> the rest yes 15:42:13 <ddieterly> merge-metrics flag, all parameters, etc 15:42:20 <rhochmuth> pagination 15:42:24 <fabiog> ddieterly: probably not all 15:42:29 <ddieterly> oh, yea! 15:42:30 <fabiog> and definitely not pagination 15:42:39 <ddieterly> how in the world did i forget that one 15:42:43 <rhochmuth> what, how can you forget paginatino 15:42:54 <rhochmuth> sorry, inside joke 15:43:57 <rhochmuth> #fabiog so i think plan to to upgrade 15:44:07 <rhochmuth> check with ben motz at cray 15:44:22 <rhochmuth> we'll need to get some testing done 15:44:33 <rhochmuth> to validate completely 15:44:35 <fabiog> rhochmuth: I will send an email in the mailing list asking him for issues 15:44:42 <rhochmuth> thx 15:44:52 <fabiog> but 0.9.4.1 is a huge step forward in performance 15:45:00 <rhochmuth> awesome! 15:45:04 <bklei> +1 15:45:12 <ddieterly> you might want to ask the influxdb guys if there is any compatibility issues that they know of 15:45:14 <rhochmuth> sounds like they are finally getting the streamign raft stuff done 15:45:34 <rhochmuth> #ddieterly can you send an email and cc folks 15:45:46 <ddieterly> sure 15:45:49 <rhochmuth> thx 15:46:01 <rhochmuth> #fabiog wnat are you seeing in performance 15:46:08 <rhochmuth> with the latests influxdb 15:46:42 <fabiog> rhochmuth: I see that Ceilosca is around 30/40% faster 15:46:57 <fabiog> rhochmuth: and it seems that is getting better when the data set is bigger 15:47:03 <fabiog> which is what I was expecting 15:47:10 <rhochmuth> cool 15:47:18 <fabiog> right now we are doing a sort of worst case scenario test 15:47:42 <fabiog> like we are doing open ended query of get samples with 1M measurements 15:47:53 <fabiog> so we need to switch to CURL instead of cello client 15:48:00 <fabiog> because the client times out 15:48:14 <fabiog> but up to 750K the client worked 15:48:20 <fabiog> in both cases 15:48:30 <fabiog> and in both cases failed at 1M 15:48:37 <rhochmuth> how is ceilosca comparing to native ceilometer 15:48:48 <rhochmuth> with the latests influxdb 15:48:58 <fabiog> my preliminary estimates is that is between 30 to 40% faster 15:49:08 <rhochmuth> nice 15:49:23 <fabiog> but I think the real advantage will be in the amount of data we can store in Ceilosca compared to Ceilometer 15:49:34 <rhochmuth> i would like to understand the tests and how that maps onto the overall system too 15:49:38 <fabiog> I want to test with 10M measurements and see what happens 15:49:46 <fabiog> I think Ceilometer will fail 15:49:49 <rhochmuth> 10M is small 15:50:16 <fabiog> right but we could not get Ceilo to work with more than 6M in the past 15:50:23 <rhochmuth> ohhh 15:50:44 <fabiog> let's see 15:50:59 <fabiog> the other thing I am not so sure is that Rally is really representative of the tests 15:51:07 <fabiog> I had strange results with it 15:51:22 <fabiog> so maybe we will use 10 runs using curl instead 15:51:36 <fabiog> it will be definitely more work but the results are more realistic 15:51:49 <fabiog> it will be undoubtedly the time a client will take to receive the data 15:52:26 <fabiog> bklei: how much data you have in Monasca in prod? 15:52:40 <fabiog> bklei: just to have an idea of a real prod env 15:53:06 <fabiog> so I can try to emulate that 15:53:12 <bklei> we've got around maybe 5 months of data, in vertica terms, getting close to 1TB 15:53:33 <fabiog> bklei: but in terms of measurements per metric 15:53:41 <fabiog> is 1M, 10M, 100K ... 15:54:04 <bklei> i can get that to you, don't know off the top of my head 15:54:20 <fabiog> bklei: I would appreciate that 15:54:28 <fabiog> thanks 15:54:35 <bklei> sure, will get u some #'s later today 15:54:39 <bklei> np 15:54:42 <rhochmuth> ok, 5 minutes left 15:54:54 <rhochmuth> should we continue with ceilosca or any other topics, or close 15:55:02 <fabiog> I am done unless there are questions 15:55:23 <bklei> i just want to plug a couple of libvirt plugin patches 15:55:26 <rhochmuth> no further questions your honor 15:55:32 <fabiog> rhochmuth: we need to start talking about summit topics and sessions 15:55:56 <fabiog> agenda for next week, I guess 15:55:58 <rhochmuth> we don't have any summit session 15:56:13 <rhochmuth> i think it will be hard to get, as they are over-booked and we aren't a openstack project yet 15:56:24 <fabiog> ok 15:56:32 <rhochmuth> but, we could reach out to other teams on various topics 15:56:56 <fabiog> yes, I want to talk to the Congress guys 15:57:02 <rhochmuth> there was some areas with the ceilometer team, such as alarming and componentization 15:57:12 <rhochmuth> Yes, Congress was the other one that comes to mind 15:57:27 <rhochmuth> There has been work in adding autoc-scaling to heat for MOnasca 15:57:37 <rhochmuth> I'll be getting an update on that tomorrow 15:57:55 <rhochmuth> Do you want me to contact Congress? 15:58:20 <fabiog> rhochmuth: sure I'll do that 15:58:28 <bklei> MaaS is our big priority at TWC, and so we're very focused on libvirt plugin -- would like to see https://review.openstack.org/#/c/225337/ and https://review.openstack.org/#/c/226405/ in soon and the agent tagged... 15:58:55 <rhochmuth> #bklei ok, i've given a +1 to one of those i believe 15:59:01 <rhochmuth> if others can review that would be great 15:59:05 <bklei> por favor 15:59:07 <rhochmuth> as soon as merged i can tag 15:59:16 <bklei> +1!! 15:59:17 <bklei> thx 15:59:27 <rhochmuth> i was hoping david schroeder at hp could take a look 15:59:32 <rhochmuth> not sure he was around this week 15:59:35 <bklei> me too, it's his baby 15:59:36 <rhochmuth> but will try and ping him 15:59:53 <bklei> thx 16:00:02 <rhochmuth> ok, coming to an end 16:00:05 <rhochmuth> thanks everyone 16:00:17 <ddieterly> ciao! 16:00:18 <bklei> thx #rhochmuth 16:00:26 <rhochmuth> #endmeeting