21:00:48 <flaper87> #startmeeting Zaqar
21:00:48 <openstack> Meeting started Mon Sep 15 21:00:48 2014 UTC and is due to finish in 60 minutes.  The chair is flaper87. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:49 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:52 <openstack> The meeting name has been set to 'zaqar'
21:00:55 <flaper87> #topic Roll Call
21:01:14 <flaper87> kgriffs: vkmc malini
21:02:05 <flaper87> mmh, anyone ?
21:02:09 <vkmc> o/
21:02:16 <flaper87> vkmc: hey :D
21:02:27 <flaper87> looks like it'll be a chat between you and me today
21:02:28 <kgriffs> o/
21:02:34 <vkmc> flaper87, and kgriffs! :D
21:02:38 <flaper87> there he is
21:02:40 <flaper87> :D
21:02:52 * kgriffs drinks teh powerade
21:02:56 * kgriffs is ready to rock
21:03:04 <flaper87> ok, lets do it
21:03:10 <flaper87> #link https://wiki.openstack.org/wiki/Meetings/Zaqar#Agenda
21:03:19 <flaper87> #topic Smoke testing planning for RC (kgriffs)
21:03:27 <flaper87> actually, wait
21:03:30 <flaper87> actions
21:03:38 <flaper87> #topic review actions from last meeting
21:03:47 <flaper87> #link http://eavesdrop.openstack.org/meetings/zaqar/2014/zaqar.2014-09-08-15.00.html
21:03:54 <flaper87> vkmc To document where API checks should go (API or Storage layer)
21:04:04 <vkmc> flaper87, already in the review queue
21:04:10 <flaper87> vkmc: sweet
21:04:15 <flaper87> do you have a link?
21:04:24 <vkmc> https://review.openstack.org/#/c/121424/
21:04:26 <vkmc> sure thing
21:04:36 <flaper87> #link https://review.openstack.org/#/c/121424/
21:04:43 <vkmc> let me know if the location I chose is ok
21:04:50 <vkmc> or if it fits better somewhere else
21:04:57 <flaper87> sure thing, thanks vkmc
21:05:02 <flaper87> next action
21:05:02 <vkmc> np
21:05:03 <flaper87> flaper87 to start adding v1.1 to the client
21:05:09 <flaper87> #link https://review.openstack.org/#/q/status:open+project:openstack/python-zaqarclient+branch:master+topic:bp/api-v1,n,z
21:05:17 <flaper87> it's all up for review
21:05:36 <flaper87> ok, lets move on
21:05:39 <vkmc> yup, I'll review the ones you added earlier tonight
21:05:46 <flaper87> vkmc: +1
21:05:50 <flaper87> #topic Smoke testing planning for RC (kgriffs)
21:05:53 <flaper87> kgriffs: floor is yours
21:06:09 <kgriffs> ok, so I had put that up there because I wanted to make sure the RC was rock-solid
21:06:19 <kgriffs> one thing that doesn't get a lot of testing in pooling
21:06:52 <flaper87> kgriffs: +1
21:06:56 <kgriffs> and also keystone middleware, although there shouldn't be a big issue there
21:07:04 <vkmc> kgriffs, +1
21:07:05 <flwang1> o/ sorry for the late
21:07:14 <flaper87> I've been meaning to setup pooling in devstack by default
21:07:20 <vkmc> hi flwang1 \o
21:07:31 <flwang1> hey guys
21:07:36 <flaper87> I'll try to do that next week
21:07:38 <flaper87> flwang1: 'sup ?
21:07:40 <kgriffs> hi flwang1
21:07:53 <flaper87> #action flaper87 to make devstack use pools by default
21:08:00 <vkmc> I have been inspecting the errors in functional testing for pools today
21:08:05 <vkmc> currently we are skipping those tests
21:08:16 <kgriffs> oh
21:08:19 <vkmc> I also had to add the tests for v1.0, will do that for next week
21:08:20 <flaper87> vkmc: +1 we need to stop skiping them
21:08:24 * flwang1 is frustrating with nova evacuate + ceph recently
21:08:45 <kgriffs> so, let me tell you what I've been up to and we can see if there is any other RC testing we need to get done
21:08:59 * flaper87 listens
21:09:08 <kgriffs> saturday I started setting up a new testing cluster
21:09:15 <kgriffs> ubuntu 14.04
21:09:19 <kgriffs> instead of deb 7
21:09:25 <kgriffs> (since I can now get that at Rackspace)
21:09:58 <kgriffs> so, I have one load generator, one load balancer running nginx (I didn't want to mess with haproxy right now)
21:10:13 <kgriffs> 2 web heads, although I will probably add 1-2 more
21:10:22 <kgriffs> 1 db box currently running a single redis instance
21:10:35 <kgriffs> and then I kept the 3-member replica set for mongo because...
21:10:48 <kgriffs> I needed a pool catalog and redis driver doesn't support that
21:10:56 <kgriffs> so what I have now
21:11:04 <kgriffs> is zaqar set up in a pooled configuration
21:11:17 <kgriffs> I have been hammering it all day with lots of tests with varying load, each test run 5 minutes
21:11:44 <kgriffs> the pool only has a single redis node in it right now
21:11:50 <kgriffs> and I only have one web head active
21:11:58 <kgriffs> my plan is to do another round after adding a web head
21:12:16 <kgriffs> (up to 4)
21:12:21 <flaper87> all that sounds really good
21:12:29 <flaper87> Any numbers you can share?
21:12:38 <kgriffs> at that point I'm thinking redis will saturate and I will add 1-3 more procs to the pool
21:13:14 <vkmc> sweeet
21:13:23 <kgriffs> re smoke testing, everything looks great from the server side. Very reliable, although stdev goes high towards the end, no failed requests
21:13:33 <kgriffs> but
21:13:59 <kgriffs> I did have some trouble getting zaqar-bench working since the patch to remove the v1 from the import statement
21:14:13 <flaper87> mmmh
21:14:27 <flaper87> sorry about that. Any clue of what the error was about ?
21:14:27 <kgriffs> can't remember for sure
21:14:54 <flaper87> It'd be weird if it's related to that but you know... software
21:14:58 <kgriffs> but somebody should go run that with everything latest and see if it is OK. I might have reverted and put the v1 back in, and then I had to hack the claim_id param in python-zaqarclient again
21:15:13 <kgriffs> let me see
21:15:36 <flaper87> #link https://git.openstack.org/cgit/openstack/python-zaqarclient/tree/zaqarclient/queues/client.py
21:15:37 <kgriffs> wait
21:15:39 <kgriffs> I lied
21:15:42 <flaper87> kgriffs: ^ that's what the client do
21:15:44 <flaper87> does*
21:15:45 <kgriffs> I think that might have been a red herring
21:15:50 <flaper87> oh ok
21:15:52 <kgriffs> and I did end up removing v1 module
21:16:02 <kgriffs> but the claim_id error was still preventing me from deleting claimed messages
21:16:17 <flaper87> right, that has been taken care of, IIRC
21:16:19 <flaper87> vkmc: ^
21:16:20 <kgriffs> until I hacked messages.py in the client lib to accept a claim_id param in __init__
21:16:26 <flaper87> I think I reviewed/approed your patch
21:16:30 <flaper87> vkmc: ^
21:16:30 <kgriffs> at least, as of saturday
21:16:40 <flaper87> kgriffs: yeah, I +2 on sunday
21:16:42 <flaper87> IIRC
21:16:43 <kgriffs> oic
21:16:46 <kgriffs> gtk
21:17:02 <flaper87> kgriffs: thanks a lot for working on all that
21:17:04 <flaper87> really
21:17:12 <kgriffs> thanks
21:17:13 <flaper87> it's been really helpful and encouraging
21:17:46 <kgriffs> oh, I should also mention
21:17:47 <flaper87> Do you think we can make that server you're using a third-party CI for benches ?
21:17:53 <flaper87> until we get rally going
21:17:58 <kgriffs> I added stdev, mean, max, and 99 percentile to zaqar-bench
21:17:59 <vkmc> flaper87, that one depends in kgriffs patch
21:18:08 <vkmc> flaper87, I added the comment on Gerrit
21:18:08 <flaper87> vkmc: oh, mmh.
21:18:12 <flaper87> vkmc: ok
21:18:19 <kgriffs> I was going to use Tsung, but it was going to be a real pain to model our scenarious so I thought it would be quicker to keep using zaqar-bench
21:18:19 <flaper87> I'll take a look again in a bit
21:18:20 <kgriffs> (for now)
21:18:24 <vkmc> kgriffs -2 it because we didn't know the root cause
21:18:35 <kgriffs> vkmc: yeah, TBH I haven't had time to dig into that
21:18:40 <flaper87> vkmc: wait, are we talking about the same patch?
21:18:45 <flaper87> vkmc: link? :D
21:18:46 <kgriffs> if someone could investigate, that would be cool
21:18:48 <vkmc> flaper87, the claims_id one
21:19:13 <vkmc> #link https://review.openstack.org/#/c/120867/
21:19:28 <kgriffs> flaper87: vkmc's patch depends on my patch that moves deleting queues to the end of the test instead of beginning,
21:19:38 <vkmc> depends on
21:19:43 <vkmc> https://review.openstack.org/#/c/119900/1
21:19:49 <flaper87> ah ok, but I guess we can make it not depend on yours
21:20:03 <kgriffs> yeah, it's just that without my patch zaqar-bench hangs everytime
21:20:05 <kgriffs> super strange
21:20:05 <vkmc> sure thing
21:20:10 <vkmc> yeap ^
21:20:12 <flaper87> at least we can move that one forward
21:20:20 <flaper87> yeah but the gate is not blocked on that, right?
21:20:34 <kgriffs> no, we don't have a job that runs zaqar-bench do we?
21:20:34 <flaper87> at least you can pull that one from master and the bench env should work
21:20:40 <flaper87> without hacking the client
21:20:46 <kgriffs> right
21:20:55 <flaper87> kgriffs: not yet, I'd like to add one but I don't think infra will be happy if we load the server :P
21:21:02 <flaper87> not sure, I should probably ask
21:21:10 <vkmc> IIRC that bug appeared after kgriffs fix
21:21:13 <kgriffs> regarding numbers, I will have some graphs soon on the wiki
21:21:20 <vkmc> without kgriffs fix, it just enters a loop
21:21:20 <kgriffs> but for now
21:21:28 <flaper87> vkmc: if you can make your patch not depend on kgriffs's we can approve it today
21:21:33 <vkmc> flaper87, will do
21:21:34 <flaper87> we'll investigate the other issue
21:21:36 <kgriffs> keeping observers steady at 50, but increasing producers
21:21:44 <kgriffs> we peak at about 5000 messages posted/sec
21:22:01 <kgriffs> that is for 1 web head and the CPUs end up at about 80%
21:22:01 <flaper87> sweet
21:22:09 <flaper87> sweet sweet sweet
21:22:11 <kgriffs> redis box is doing like 50%
21:22:19 <kgriffs> (for one CPU)
21:22:24 <kgriffs> web heads are 20 CPUs
21:22:34 <flaper87> and no fancy configs, right?
21:22:48 <kgriffs> I don't recall what nginx was (4 workers) but probably around 50% each proc
21:23:20 <kgriffs> flaper87: I didn't spend a ton of time tuning configs. mostly defaults I think.
21:23:29 <kgriffs> but I will post them and you can see
21:23:29 <flaper87> that's even better
21:23:53 <flaper87> if we're doing good with default, I guess tuning them will help increasing those numbers
21:23:55 <kgriffs> nah, we need to suck more
21:24:06 <kgriffs> ;)
21:24:08 <flaper87> :P
21:24:12 <flaper87> ROFL
21:24:28 <flaper87> ok cool
21:24:37 <flaper87> again, kgriffs, thanks for all that work.
21:24:41 <kgriffs> sure
21:24:44 <flaper87> if there are no more questions, lets move on
21:24:54 <flaper87> #topic Zaqar client
21:25:00 <kgriffs> btw, it takes forever to run these, but I'll try to trickle out the results as much as possible
21:25:02 <vkmc> thanks kgriffs :D
21:25:12 <kgriffs> yw
21:25:14 <kgriffs> :D
21:25:19 * vkmc not so sad panda
21:25:30 <flaper87> so, very quick. I don't mean to put any preassure on anyone but we need to release the client before the 18th
21:25:39 <flaper87> so...
21:25:41 <flaper87> #link https://review.openstack.org/#/q/status:open+project:openstack/python-zaqarclient,n,z
21:25:52 <flaper87> also, I'm working on this:
21:25:54 <flaper87> #link https://review.openstack.org/#/c/120437/3
21:26:05 <flaper87> if you guys want to take a look it'd be cool
21:26:11 <flaper87> (especially malini)
21:26:14 <flwang1> I will review them
21:26:15 <flaper87> I'll let her know
21:26:36 <flaper87> I think all v1.1 patches are up
21:26:42 <flaper87> I also fixed the functional tests
21:26:50 <flaper87> and other things that I found on my way
21:27:05 <flaper87> if there's anything missing please, by all means, don't tell me
21:27:11 <flaper87> :P
21:27:16 <flaper87> jokes apart, fix it
21:27:23 <flaper87> now seriously, let me know :)
21:27:29 <flaper87> I'll happily work on that
21:27:43 <flaper87> any questions? otherwise let's move on
21:28:00 <flaper87> cool
21:28:05 <flaper87> #topic What had to be said has been said, I hope
21:28:40 <flaper87> well, you know. 2 weeks, 1 thread and a half, lots of emails, tons of comments. 2 meetings, lot of noise and mixed emotions
21:29:02 <flaper87> if something hasn't been said, I believe it's already becoming quite late. that said, anything could happen.
21:29:16 <flaper87> Our third, and last, meeting is tomorrow. 20 UTC
21:29:34 <flwang1> will it be the final decision?
21:29:35 <flaper87> Our time during the meeting will be, AFAIK, dedicated to vote
21:29:39 <flaper87> flwang1: yes
21:29:46 <flaper87> no more 3h discussions
21:29:59 <flaper87> few comments and then vote
21:30:20 <flaper87> whatever happens, I'm very proud of what we've accomplished and more importantly we'll keep doing it
21:30:32 <flaper87> so, thank you all and lets keep up the amazing work
21:30:44 <flwang1> my feeling is most like awaiting trial
21:30:52 <flaper87> LOOOOOL
21:30:55 <vkmc> +1 flwang1
21:30:56 <flaper87> yeah, pretty much that
21:31:00 <vkmc> I feel the same way
21:31:01 <flwang1> I mean I can't control my fate
21:31:10 <flaper87> you can add to that a few head bangs on the desk
21:31:25 <flaper87> flwang1: that sounds, you can't trust the system
21:31:27 <flaper87> :P
21:31:45 <flaper87> anyway... questions? thoughts? etc? move on?
21:31:47 <flwang1> flaper87: you know what I mean, man
21:31:55 <flaper87> flwang1: I do know ;)
21:31:58 <vkmc> but hey, questions regarding the design or the tools used in Zaqar has been replied
21:32:15 <kgriffs> idk if it will make any difference, but I'll try to publish another round of perf test results before tomorrow
21:32:16 <flwang1> i assume there is no appeal :)
21:32:19 <vkmc> and the team did the best to clear things up
21:32:29 <vkmc> +1 kgriffs
21:32:34 <flaper87> kgriffs: I think it's important, yes.
21:32:40 <kgriffs> kk
21:32:56 <flwang1> kgriffs: thanks for your effort, it's awesome
21:33:19 <kgriffs> thanks man
21:33:25 * flaper87 gives kgriffs a truck full of pop-tarts
21:33:31 * flaper87 just burried kgriffs
21:33:33 <flaper87> muahahahaha
21:33:46 <vkmc> :)
21:33:48 * kgriffs puts in pantry for the zombie apocalypse
21:33:55 <flaper87> LOL
21:33:57 * kgriffs everyone's invited to his house
21:34:05 <flaper87> the pop-tart-zombie apocalypse
21:34:17 <kgriffs> it will be scary. and delicious
21:34:17 <flaper87> ok, anything else?
21:34:24 <flaper87> ROFL
21:34:29 <flaper87> #topic Summit sessions submission
21:34:38 <flaper87> ok, heads up... again :)
21:34:45 <flaper87> #link https://etherpad.openstack.org/p/kilo-zaqar-summit-topics
21:35:00 <flaper87> put whatever you want to talk about there. There won't be a design session tool this time
21:35:12 <flaper87> we'll help organizing and prioritizing things ourselves using our own tools
21:35:30 <flaper87> I figured it'd be better to stick with the already existing etherpad so put things in that link
21:35:50 <flaper87> We'll start discussing those sessions in our meetings starting next week
21:36:08 <kgriffs> ok
21:36:08 <flaper87> one of those proposals will be used for the next OPW mentee
21:36:11 <flaper87> #link https://wiki.openstack.org/wiki/OutreachProgramForWomen/Ideas#Coding
21:36:35 <flaper87> Zaqar has been participating in OPW for the last 2 cycles
21:36:46 <flaper87> I'm very proud of this and I'm super proud of our mentees
21:36:54 <vkmc> :D
21:37:10 <flaper87> I think we should keep giving back to the community and OPW is a good way to do that
21:37:30 <flaper87> I've put myself as a mentor but please, if you've an idea and want to mentor do not hesitate to add it
21:37:53 <flaper87> WE've got enough good, exiting and hard enough tasks that new mentees can work on
21:38:01 <flaper87> that's it from me
21:38:04 <flaper87> questions?
21:38:23 * kgriffs is cool
21:38:28 <flaper87> #topic Rally
21:39:02 <flaper87> we talked with boris-42 last week and he volunteered to host a hangout where he would explain us how Rally works
21:39:07 <flaper87> internals and whatnot
21:39:22 <flwang1> I contacted with him as well
21:39:25 <flaper87> it'd be cool if we all can participate. What time would work for you ?
21:39:41 <flaper87> is 21 UTC good ? (that would be this exact time)
21:39:45 <flwang1> i was told we just need 20 mins to get a plugin in Rally :)
21:39:56 <flaper87> flwang1: yeah, it looks easy
21:40:15 <flaper87> kgriffs: vkmc flwang1 is 21 UTC ok?
21:40:20 <flwang1> i'm OK
21:40:23 <vkmc> I'm ok too
21:40:23 <flaper87> I was thinking on proposing Thursday
21:40:29 <vkmc> when is it?
21:40:30 <flwang1> but TC hate simple/easy stuff
21:40:38 <kgriffs> let me see
21:41:01 <kgriffs> thur at 2100 should work
21:41:09 <flaper87> sweet, ok.
21:41:16 <vkmc> cool, it works for me too
21:41:33 <flaper87> I'll let boris-42 know (unless we woke him up with all this pings)
21:41:42 <flaper87> Thur, 21 UTC it is
21:42:05 <flaper87> he said he'd be in PST this week and the next week so, I'm sure he won't be asleep
21:42:10 * kgriffs is going to build an irc-pinger and hide it under flaper87's pillow
21:42:11 <flaper87> any questions?
21:42:28 * flaper87 doesn't use a pillow
21:42:32 <flaper87> ok, I lied
21:42:35 * kgriffs curses! foiled again.
21:42:36 <flaper87> I just thought that was fun
21:42:38 <flaper87> :P
21:43:07 * flaper87 has the weirdest sleeping pattern ever
21:43:13 <flaper87> anyway
21:43:17 <flaper87> #topic open discussion
21:43:54 <flaper87> please, review the client patches
21:43:56 <flaper87> :P
21:44:00 * flaper87 breaks the ice
21:44:04 <vkmc> please review the CLI patches as well
21:44:05 <vkmc> :p
21:44:08 <flaper87> ... or frozens it again
21:44:10 <flaper87> :P
21:44:13 <flaper87> vkmc: +1
21:44:23 <vkmc> so we have client v1.1 and CLI v1.0
21:44:37 <flaper87> vkmc: well, to be fair that cli works for v1.1 too
21:44:43 <flaper87> at least it should
21:44:46 <flaper87> most of it
21:44:49 * flaper87 hopes
21:44:51 <flaper87> :)
21:44:53 <vkmc> yeah :)
21:45:12 <flaper87> cool, that's it, folks
21:45:23 <flaper87> have a great rest of the day you all!
21:45:23 <kgriffs> groovy
21:45:30 <kgriffs> thanks Mr. Percoco
21:45:46 * flaper87 bows and says some words in a weird language
21:45:49 <vkmc> thanks Fla, you too!
21:45:57 <flaper87> #endmeeting