16:04:39 <jgriffith> #startmeeting cinder
16:04:40 <openstack> Meeting started Wed Apr  9 16:04:39 2014 UTC and is due to finish in 60 minutes.  The chair is jgriffith. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:04:41 <jgriffith> vishy: LOL
16:04:42 <winston-d> :)
16:04:42 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:04:42 <DuncanT-> bswartz: yes
16:04:45 <openstack> The meeting name has been set to 'cinder'
16:04:55 <ayoung> vishy, 3 days?  You mean it is on Saturday?  Or it was last sunday?
16:04:55 * DuncanT- waves
16:04:59 <bswartz> hi
16:05:10 <avishay> hello
16:05:10 <ameade> o/
16:05:18 <xyang> hi
16:05:19 <glenng> Hey Folks
16:05:21 <akerr> o/
16:05:21 <coolsvap> hello
16:05:23 <winston-d> o/
16:05:24 <jgriffith> Ok... short meeting
16:05:31 <mtanino> Hi
16:05:32 <jgriffith> I didn't put anything on the agenda
16:05:38 <jgriffith> but I wanted to quickly go over RC status
16:05:43 <jgriffith> and some other thigns
16:05:44 * jungleboyj will remind you of that in an hour.  ;-)
16:05:44 <jgriffith> things
16:05:56 <jgriffith> #topic Cinder-Specs
16:06:12 <avishay> hah
16:06:15 <jgriffith> So if you haven't been in the loop we're moving toward using gerrit reviews for BP's
16:06:35 <DuncanT-> Awesome
16:06:48 <jgriffith> Ihttps://review.openstack.org/#/q/nova-specs,n,z
16:06:56 * jungleboyj wondered about that.
16:06:56 <jgriffith> remove the 'I'
16:07:09 <jgriffith> I wanted to make sure I communicated it here
16:07:20 <jgriffith> so you all weren't surprised when it poped up
16:07:21 <jgriffith> popped
16:07:29 <jgriffith> also make sure there are no objections?
16:07:43 <jgriffith> I'll get it created and update docs on the wiki when I get aroudn to it
16:07:52 <jgriffith> cool with everyone?
16:07:54 <xyang> what about blueprint already submitted in Icehouse?
16:08:07 <jungleboyj> Good question.
16:08:11 <avishay> jgriffith: what's the process?  submit the review, and once it gets approved, copy-paste it into a BP?
16:08:39 <jgriffith> xyang: if they're not approved yet submit via gerrit
16:09:28 <xyang> does the ViPR driver blueprint need to go through gerrit?
16:09:49 <jgriffith> avishay: yeah, that's kinda the idea
16:09:53 <xyang> code is already there in reviewable state
16:10:03 <avishay> jgriffith: ok cool, nice idea
16:10:03 <jgriffith> xyang: but the BP was never approved IIRC
16:10:25 <jgriffith> xyang: so the transition for things in flight isn't that great
16:10:29 <eharney> i assume the wiki page will fill in some details about how this works... i.e. is the cinder-specs repo the readable "destination" or these end up somewhere else?
16:10:35 <jgriffith> xyang: but the idea is going forward to have a way to track this better
16:10:46 <jgriffith> if you try and do this via LP it's kind of a pain
16:10:54 <jgriffith> eharney: yes
16:11:06 <jgriffith> eharney: I just wanted to introduce it to those who haven't seen it
16:11:13 <xyang> jgriffith: ok, I'll take a look of this new process
16:11:14 <jgriffith> so it wasn't a huge surprise or shock
16:11:24 <DuncanT-> Given how many blueprints we have that are lacking in detail, and how easy it is to loose the discussion, looks like a great plan to me
16:11:33 <jgriffith> DuncanT-: +1
16:11:37 <jungleboyj> So, we want to hash out details and concerns via gerrit before making it, officially a BluePrint.
16:11:41 <avishay> +4
16:11:44 <jgriffith> jungleboyj: yes
16:12:00 <jungleboyj> jgriffith: Ok, cool.  That makes sense now that I get it.  :-)
16:12:04 <jgriffith> jungleboyj: and it saves the problem I have of people "adding approved" to their own bp's and targetting them
16:12:18 <jungleboyj> jgriffith: Gotcha.
16:12:19 <avishay> nobody would ever do that...
16:12:24 <jgriffith> avishay: hehe
16:12:31 <jgriffith> ok... next topic
16:12:36 <jgriffith> #topic RC status
16:12:43 <jgriffith> So we cut our RC2....
16:12:54 <jgriffith> and I promptly found and was informed of a number of issues :(
16:13:50 <jgriffith> vishy: pointed out this one: lp bug#1304719
16:13:53 <jgriffith> that's easy
16:13:59 <jgriffith> but.....
16:14:02 <jgriffith> The trouble is:
16:14:28 <jgriffith> https://bugs.launchpad.net/cinder/+bug/1304122
16:14:31 <uvirtbot> Launchpad bug 1304122 in cinder "tempest.api.volume.test_volumes_get.VolumesGetTest.test_volume_create_get_update_delete fails in gate" [High,Confirmed]
16:14:57 <jgriffith> If you search logstash for this we've been hosed for quite a while
16:15:07 <xyang> jgriffith: can you also give a timeline on this?  by what date a blueprint needs to be submitted, approved, for a patch to have a chance to target a milestone, etc?
16:15:22 <jgriffith> It's likely either the messaging work, or it's the deactivate addd
16:15:29 <jgriffith> I'm working through it today
16:15:44 <jgriffith> but I think it has to do with activate/deativate
16:15:52 <jgriffith> the code has become rather a mess
16:15:55 <jgriffith> so more to come on that
16:16:10 <jgriffith> if anybody sees anything here that jumps out at them let me know
16:16:32 <jgriffith> but I think the best thing is to clean up the lvm code again as it's become quite a mess
16:16:44 <thingee> hey all
16:17:15 <avishay> thingee: yo
16:17:50 <jgriffith> no thoughts?
16:18:28 <winston-d> jgriffith: though you were the one has the most understanding of lvm
16:18:34 <DuncanT-> Nothing obvious
16:18:37 <jgriffith> :)
16:18:42 <jgriffith> ok
16:18:57 <jgriffith> moving on ;)
16:19:01 <jgriffith> # testing RC code
16:19:14 <jgriffith> not sure how much testing others have been doing
16:19:23 <jgriffith> but if you can load up some code and beat on it!
16:19:44 <jgriffith> I've been hammering things a bit and while I'm not seeing flat out failures
16:19:47 <jgriffith> I am concerned
16:19:58 <jgriffith> Performance seems to be bad
16:20:14 <jgriffith> as in simple API calls can take seconds to return
16:20:42 <jgriffith> I ran a "cinder create 1" and it sat there for about 8 seconds
16:20:44 <jgriffith> not good
16:20:50 <jgriffith> not consistent either
16:20:51 <avishay> jgriffith: yeesh... just to get back from API?
16:20:56 <DuncanT-> Yuck. Not noticed that, but I'm a bit behind RC2 at the moment
16:21:06 <kmartin> jgriffith: must of been the backend :)
16:21:14 <hemna> whoa
16:21:15 <jgriffith> avishay: it "LOOKS" like it's a lag before even sending the RPC call
16:21:24 <hemna> I've not seen that here
16:21:26 <jgriffith> avishay: DuncanT- and I see the same thing on nova
16:21:32 <hemna> and I've been hacking on the api a bit for the multi-attach patch
16:21:36 <jgriffith> I'm rebuilding a system now to see if maybe it's my env
16:21:39 <DuncanT-> Is this being seen on the gate?
16:21:43 <jgriffith> but I'd love for other people to look
16:21:45 <avishay> jgriffith: you mean the REST call to API?
16:21:53 <jgriffith> DuncanT-: we don't "time" things in the gate
16:21:56 <jgriffith> it would never fails
16:21:58 <jgriffith> fail
16:22:02 <hemna> oh wait
16:22:03 <jgriffith> just take a very long time
16:22:06 <hemna> I think I did see that once
16:22:18 <winston-d> jgriffith: just for create call?
16:22:19 <DuncanT-> jgriffith: The tempest logs have some timing info in though
16:22:22 <hemna> but thought it was my fault with my hacks...restacked and then it went away
16:22:32 <hemna> I was seeing it on create as well
16:22:41 <jgriffith> winston-d: not sure, but i've seen it on "cinder create" and "nova boot"
16:22:44 <thingee> jgriffith: I'm not aware of this behavior with rc2
16:22:46 <vbala> i do observe some lag during create, but it is intermittent
16:23:05 <DuncanT-> I've just spun up a new devstack... no lag apaprent
16:23:09 <thingee> rather haven't seen it myself
16:23:15 <jgriffith> DuncanT-: keep running it :)
16:23:21 <hemna> If I see it again, I'll try tracking it down
16:23:23 <jgriffith> I suspect you may encounter it
16:23:38 <jgriffith> and look in your logs for RPC Warnings for dropped items etc
16:23:41 <hemna> from what I saw, the create actually went through, but the API call hung on the return
16:23:47 <glenng> jgriffith: So you didn't see it right away?
16:23:52 <jgriffith> hemna: oh... interesting
16:24:00 <jgriffith> glenng: it's random/intermittent
16:24:12 <glenng> jgriffith: Great...
16:24:13 <winston-d> i heard someone complained about 'cinder list' become very slow if there were ~1,000 volumes to display
16:24:20 <jgriffith> glenng: yeah... makes it easier
16:24:25 <winston-d> it could take > 30sec to return.
16:24:28 <jgriffith> winston-d: I think that was me :)
16:24:44 <avishay> i think sqlalchemy is hiding a lot of nasty SQL stuff
16:24:46 <jgriffith> winston-d: but I realized that it was Horizon that seemed to be so slow
16:24:51 <jgriffith> anyway
16:24:55 <jgriffith> I just wanted to point it out
16:24:59 <hemna> I was using the cmdln
16:25:00 <jgriffith> ask people to do some testing
16:25:02 <hemna> fwiw
16:25:10 <jgriffith> hemna: that's a different topic
16:25:15 <avishay> i think we're doing a lot of unecessary DB calls
16:25:31 <jungleboyj> I was doing some creates on the latest code yesterday and didn't see anything.  Will keep an eye out though.
16:25:32 <jgriffith> winston-d: was talking about my complaint the other day of "listing 1000 volumes/instances"
16:25:44 <jgriffith> everybody sloowwww down a sec
16:26:05 <jgriffith> I'm just pointing out that we should all be testing and looking at this sort of thing
16:26:08 <glenng> jgriffith: Any more parameters for us to use in reproducing (VM was up for days, hours, lots of volume, etc.)?
16:26:12 <jgriffith> not looking for theories yet :)
16:26:15 <DuncanT-> Ugh, ok, variance on 'cinder list' with 100 volumes is 1.2 - 15.7 seconds
16:26:20 <jgriffith> glenng: fresh devstack
16:26:26 <hemna> glenng, I had it happen on brand new vms on a fresh stack
16:26:26 <guitarzan> DuncanT-: impressive
16:26:34 <winston-d> DuncanT-: try 1000
16:26:34 <jgriffith> that sucks
16:26:41 <DuncanT-> That does suck
16:26:56 <jgriffith> alright... now that I've ruined everyone's day :)
16:27:00 <bswartz> I blame mysql
16:27:01 <glenng> hehe
16:27:03 <avishay> 1.2 seconds is almost tolerable
16:27:05 <jgriffith> #open-discussion
16:27:06 <avishay> bswartz: +1
16:27:15 <DuncanT-> I blame bswartz
16:27:18 <jgriffith> #topic open-discussion
16:27:20 <avishay> hah
16:27:24 <jgriffith> DuncanT-: I blame bswartz as well
16:28:00 <jgriffith> anybody have anything?
16:28:13 <jgriffith> and yes, I know I need to work on session proposals
16:28:16 <jungleboyj> Wasn't there a fix a couple of months ago for the slow cinder list output?
16:28:17 <jgriffith> I'll get to it :)
16:28:37 <akerr> don't forget to vote on PTL
16:28:40 <jungleboyj> I know we had seen that problem internally and I thought we pushed code to the community for it.
16:28:43 <jgriffith> akerr: OHHHHH!!!!
16:28:46 <jgriffith> YESSSSSSSS
16:28:53 <jgriffith> thingee:
16:29:16 * jgriffith and thingee are running for PTL
16:29:23 <jgriffith> make sure you vote for one of us
16:29:25 <bswartz> seriously though, what does cinder list do other than schlep data out of the database? there's no network or driver interaction there
16:29:45 <hemna> bswartz, well, except we are using sqlalchemy to do the work.
16:30:00 <hemna> mysql can handle a massive load from my past experience with direct bind query calls.
16:30:08 <winston-d> someone (not here) wanted to blame admin-metadata
16:30:13 <DuncanT-> bswartz: Given it is intermittent, could be the API server getting blocking doing something else... periodic tasks have caused this issue in nova
16:30:13 <hemna> I don't know how sqlalchemy actually does the queries
16:30:15 * jungleboyj recommends you don't lose your voting link.  It was quite a process to get it replaced.
16:30:45 * coolsvap voted
16:30:49 <jgriffith> self.db.volume_get_all(context, marker, limit, sort_key, sort_dir, filters=filters)
16:30:52 * hemna voted already
16:31:02 <glenng> *ditto*
16:31:26 * avishay voted for David Wang
16:31:32 <hemna> lol
16:31:47 <avishay> we need to take a hard look at DB accesses for Juno IMO
16:31:51 <jungleboyj> avishay: :-)
16:31:57 <hemna> avishay, +1
16:32:06 * jungleboyj wonders who is David Wang.
16:32:26 <ameade> that would be fun to work on, glance did a lot of DB optimization a few cycles ago
16:32:48 <winston-d> ameade: hwo did that go?
16:32:49 <jgriffith> avishay: perhaps
16:32:50 <hemna> and the multi-attach patch I put up adds another foreign key join to the volumes get db call.
16:33:13 * coolsvap thinks he's the one you get for replaced voter link ;)
16:34:21 <ameade> winston-d: i would say good enough, lol
16:35:07 <mtanino> Will new development process be shown in this wiki page?  https://wiki.openstack.org/wiki/Cinder/how-to-contribute-a-driver
16:35:11 <winston-d> ameade: good to know
16:35:20 <jgriffith> BAH
16:35:40 <jgriffith> mtanino: what specifically?
16:35:48 <jgriffith> mtanino: you mean using gerrit for BP submission?
16:36:02 <mtanino> jgriffith: Yes.
16:36:18 <jgriffith> mtanino: yes, once I set it up I'll make sure docs are updated
16:36:35 <mtanino> jgriffith: I see. Thank you.
16:37:07 <avishay> jgriffith: please send email to the ML when the wiki is updated
16:37:17 <jgriffith> alright... think we can adjourn to openstack-cinder if there are no objections?
16:37:23 <jgriffith> avishay: will do
16:37:28 <jungleboyj> avishay: +1
16:37:37 <jgriffith> #action jgriffith setup cinder-specs and announce on ML
16:38:01 <jgriffith> okie-dokie
16:38:05 <jgriffith> let's get out of here
16:38:12 <jgriffith> thanks everyone
16:38:19 <jgriffith> #endmeeting cinder