16:04:39 #startmeeting cinder 16:04:40 Meeting started Wed Apr 9 16:04:39 2014 UTC and is due to finish in 60 minutes. The chair is jgriffith. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:04:41 vishy: LOL 16:04:42 :) 16:04:42 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:04:42 bswartz: yes 16:04:45 The meeting name has been set to 'cinder' 16:04:55 vishy, 3 days? You mean it is on Saturday? Or it was last sunday? 16:04:55 * DuncanT- waves 16:04:59 hi 16:05:10 hello 16:05:10 o/ 16:05:18 hi 16:05:19 Hey Folks 16:05:21 o/ 16:05:21 hello 16:05:23 o/ 16:05:24 Ok... short meeting 16:05:31 Hi 16:05:32 I didn't put anything on the agenda 16:05:38 but I wanted to quickly go over RC status 16:05:43 and some other thigns 16:05:44 * jungleboyj will remind you of that in an hour. ;-) 16:05:44 things 16:05:56 #topic Cinder-Specs 16:06:12 hah 16:06:15 So if you haven't been in the loop we're moving toward using gerrit reviews for BP's 16:06:35 Awesome 16:06:48 Ihttps://review.openstack.org/#/q/nova-specs,n,z 16:06:56 * jungleboyj wondered about that. 16:06:56 remove the 'I' 16:07:09 I wanted to make sure I communicated it here 16:07:20 so you all weren't surprised when it poped up 16:07:21 popped 16:07:29 also make sure there are no objections? 16:07:43 I'll get it created and update docs on the wiki when I get aroudn to it 16:07:52 cool with everyone? 16:07:54 what about blueprint already submitted in Icehouse? 16:08:07 Good question. 16:08:11 jgriffith: what's the process? submit the review, and once it gets approved, copy-paste it into a BP? 16:08:39 xyang: if they're not approved yet submit via gerrit 16:09:28 does the ViPR driver blueprint need to go through gerrit? 16:09:49 avishay: yeah, that's kinda the idea 16:09:53 code is already there in reviewable state 16:10:03 jgriffith: ok cool, nice idea 16:10:03 xyang: but the BP was never approved IIRC 16:10:25 xyang: so the transition for things in flight isn't that great 16:10:29 i assume the wiki page will fill in some details about how this works... i.e. is the cinder-specs repo the readable "destination" or these end up somewhere else? 16:10:35 xyang: but the idea is going forward to have a way to track this better 16:10:46 if you try and do this via LP it's kind of a pain 16:10:54 eharney: yes 16:11:06 eharney: I just wanted to introduce it to those who haven't seen it 16:11:13 jgriffith: ok, I'll take a look of this new process 16:11:14 so it wasn't a huge surprise or shock 16:11:24 Given how many blueprints we have that are lacking in detail, and how easy it is to loose the discussion, looks like a great plan to me 16:11:33 DuncanT-: +1 16:11:37 So, we want to hash out details and concerns via gerrit before making it, officially a BluePrint. 16:11:41 +4 16:11:44 jungleboyj: yes 16:12:00 jgriffith: Ok, cool. That makes sense now that I get it. :-) 16:12:04 jungleboyj: and it saves the problem I have of people "adding approved" to their own bp's and targetting them 16:12:18 jgriffith: Gotcha. 16:12:19 nobody would ever do that... 16:12:24 avishay: hehe 16:12:31 ok... next topic 16:12:36 #topic RC status 16:12:43 So we cut our RC2.... 16:12:54 and I promptly found and was informed of a number of issues :( 16:13:50 vishy: pointed out this one: lp bug#1304719 16:13:53 that's easy 16:13:59 but..... 16:14:02 The trouble is: 16:14:28 https://bugs.launchpad.net/cinder/+bug/1304122 16:14:31 Launchpad bug 1304122 in cinder "tempest.api.volume.test_volumes_get.VolumesGetTest.test_volume_create_get_update_delete fails in gate" [High,Confirmed] 16:14:57 If you search logstash for this we've been hosed for quite a while 16:15:07 jgriffith: can you also give a timeline on this? by what date a blueprint needs to be submitted, approved, for a patch to have a chance to target a milestone, etc? 16:15:22 It's likely either the messaging work, or it's the deactivate addd 16:15:29 I'm working through it today 16:15:44 but I think it has to do with activate/deativate 16:15:52 the code has become rather a mess 16:15:55 so more to come on that 16:16:10 if anybody sees anything here that jumps out at them let me know 16:16:32 but I think the best thing is to clean up the lvm code again as it's become quite a mess 16:16:44 hey all 16:17:15 thingee: yo 16:17:50 no thoughts? 16:18:28 jgriffith: though you were the one has the most understanding of lvm 16:18:34 Nothing obvious 16:18:37 :) 16:18:42 ok 16:18:57 moving on ;) 16:19:01 # testing RC code 16:19:14 not sure how much testing others have been doing 16:19:23 but if you can load up some code and beat on it! 16:19:44 I've been hammering things a bit and while I'm not seeing flat out failures 16:19:47 I am concerned 16:19:58 Performance seems to be bad 16:20:14 as in simple API calls can take seconds to return 16:20:42 I ran a "cinder create 1" and it sat there for about 8 seconds 16:20:44 not good 16:20:50 not consistent either 16:20:51 jgriffith: yeesh... just to get back from API? 16:20:56 Yuck. Not noticed that, but I'm a bit behind RC2 at the moment 16:21:06 jgriffith: must of been the backend :) 16:21:14 whoa 16:21:15 avishay: it "LOOKS" like it's a lag before even sending the RPC call 16:21:24 I've not seen that here 16:21:26 avishay: DuncanT- and I see the same thing on nova 16:21:32 and I've been hacking on the api a bit for the multi-attach patch 16:21:36 I'm rebuilding a system now to see if maybe it's my env 16:21:39 Is this being seen on the gate? 16:21:43 but I'd love for other people to look 16:21:45 jgriffith: you mean the REST call to API? 16:21:53 DuncanT-: we don't "time" things in the gate 16:21:56 it would never fails 16:21:58 fail 16:22:02 oh wait 16:22:03 just take a very long time 16:22:06 I think I did see that once 16:22:18 jgriffith: just for create call? 16:22:19 jgriffith: The tempest logs have some timing info in though 16:22:22 but thought it was my fault with my hacks...restacked and then it went away 16:22:32 I was seeing it on create as well 16:22:41 winston-d: not sure, but i've seen it on "cinder create" and "nova boot" 16:22:44 jgriffith: I'm not aware of this behavior with rc2 16:22:46 i do observe some lag during create, but it is intermittent 16:23:05 I've just spun up a new devstack... no lag apaprent 16:23:09 rather haven't seen it myself 16:23:15 DuncanT-: keep running it :) 16:23:21 If I see it again, I'll try tracking it down 16:23:23 I suspect you may encounter it 16:23:38 and look in your logs for RPC Warnings for dropped items etc 16:23:41 from what I saw, the create actually went through, but the API call hung on the return 16:23:47 jgriffith: So you didn't see it right away? 16:23:52 hemna: oh... interesting 16:24:00 glenng: it's random/intermittent 16:24:12 jgriffith: Great... 16:24:13 i heard someone complained about 'cinder list' become very slow if there were ~1,000 volumes to display 16:24:20 glenng: yeah... makes it easier 16:24:25 it could take > 30sec to return. 16:24:28 winston-d: I think that was me :) 16:24:44 i think sqlalchemy is hiding a lot of nasty SQL stuff 16:24:46 winston-d: but I realized that it was Horizon that seemed to be so slow 16:24:51 anyway 16:24:55 I just wanted to point it out 16:24:59 I was using the cmdln 16:25:00 ask people to do some testing 16:25:02 fwiw 16:25:10 hemna: that's a different topic 16:25:15 i think we're doing a lot of unecessary DB calls 16:25:31 I was doing some creates on the latest code yesterday and didn't see anything. Will keep an eye out though. 16:25:32 winston-d: was talking about my complaint the other day of "listing 1000 volumes/instances" 16:25:44 everybody sloowwww down a sec 16:26:05 I'm just pointing out that we should all be testing and looking at this sort of thing 16:26:08 jgriffith: Any more parameters for us to use in reproducing (VM was up for days, hours, lots of volume, etc.)? 16:26:12 not looking for theories yet :) 16:26:15 Ugh, ok, variance on 'cinder list' with 100 volumes is 1.2 - 15.7 seconds 16:26:20 glenng: fresh devstack 16:26:26 glenng, I had it happen on brand new vms on a fresh stack 16:26:26 DuncanT-: impressive 16:26:34 DuncanT-: try 1000 16:26:34 that sucks 16:26:41 That does suck 16:26:56 alright... now that I've ruined everyone's day :) 16:27:00 I blame mysql 16:27:01 hehe 16:27:03 1.2 seconds is almost tolerable 16:27:05 #open-discussion 16:27:06 bswartz: +1 16:27:15 I blame bswartz 16:27:18 #topic open-discussion 16:27:20 hah 16:27:24 DuncanT-: I blame bswartz as well 16:28:00 anybody have anything? 16:28:13 and yes, I know I need to work on session proposals 16:28:16 Wasn't there a fix a couple of months ago for the slow cinder list output? 16:28:17 I'll get to it :) 16:28:37 don't forget to vote on PTL 16:28:40 I know we had seen that problem internally and I thought we pushed code to the community for it. 16:28:43 akerr: OHHHHH!!!! 16:28:46 YESSSSSSSS 16:28:53 thingee: 16:29:16 * jgriffith and thingee are running for PTL 16:29:23 make sure you vote for one of us 16:29:25 seriously though, what does cinder list do other than schlep data out of the database? there's no network or driver interaction there 16:29:45 bswartz, well, except we are using sqlalchemy to do the work. 16:30:00 mysql can handle a massive load from my past experience with direct bind query calls. 16:30:08 someone (not here) wanted to blame admin-metadata 16:30:13 bswartz: Given it is intermittent, could be the API server getting blocking doing something else... periodic tasks have caused this issue in nova 16:30:13 I don't know how sqlalchemy actually does the queries 16:30:15 * jungleboyj recommends you don't lose your voting link. It was quite a process to get it replaced. 16:30:45 * coolsvap voted 16:30:49 self.db.volume_get_all(context, marker, limit, sort_key, sort_dir, filters=filters) 16:30:52 * hemna voted already 16:31:02 *ditto* 16:31:26 * avishay voted for David Wang 16:31:32 lol 16:31:47 we need to take a hard look at DB accesses for Juno IMO 16:31:51 avishay: :-) 16:31:57 avishay, +1 16:32:06 * jungleboyj wonders who is David Wang. 16:32:26 that would be fun to work on, glance did a lot of DB optimization a few cycles ago 16:32:48 ameade: hwo did that go? 16:32:49 avishay: perhaps 16:32:50 and the multi-attach patch I put up adds another foreign key join to the volumes get db call. 16:33:13 * coolsvap thinks he's the one you get for replaced voter link ;) 16:34:21 winston-d: i would say good enough, lol 16:35:07 Will new development process be shown in this wiki page? https://wiki.openstack.org/wiki/Cinder/how-to-contribute-a-driver 16:35:11 ameade: good to know 16:35:20 BAH 16:35:40 mtanino: what specifically? 16:35:48 mtanino: you mean using gerrit for BP submission? 16:36:02 jgriffith: Yes. 16:36:18 mtanino: yes, once I set it up I'll make sure docs are updated 16:36:35 jgriffith: I see. Thank you. 16:37:07 jgriffith: please send email to the ML when the wiki is updated 16:37:17 alright... think we can adjourn to openstack-cinder if there are no objections? 16:37:23 avishay: will do 16:37:28 avishay: +1 16:37:37 #action jgriffith setup cinder-specs and announce on ML 16:38:01 okie-dokie 16:38:05 let's get out of here 16:38:12 thanks everyone 16:38:19 #endmeeting cinder