16:01:16 #startmeeting cinder 16:01:17 Meeting started Wed Aug 7 16:01:16 2013 UTC and is due to finish in 60 minutes. The chair is jgriffith. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:01:18 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:01:21 The meeting name has been set to 'cinder' 16:01:24 hey cinder folks 16:01:32 hi 16:01:55 thingee: DuncanT- 16:01:57 hey xyang_ 16:02:03 hello 16:02:03 o/ 16:02:08 howdi 16:02:09 zhiyan: howdy 16:02:19 hi jgriffith 16:02:22 avishay???? 16:02:23 Hey 16:02:31 hey jgriffith 16:02:36 hello 16:02:38 alright.. let's get stareted 16:02:41 and hi DuncanT 16:02:41 started even 16:02:46 dosaboy 16:02:55 #topic make all rbd clones cow 16:03:06 dosaboy: ^^ 16:03:15 so i'm looking for comments here 16:03:29 xiaoi has raised concern about performance issue in ceph 16:03:34 dosaboy: my first question when I looked was: 16:03:38 if we have too many clones from same snapshot 16:03:50 jgriffith: shoot 16:03:52 What if you want to delete something 16:04:09 ie isn't this the issue with everything being dependent through the chain? 16:04:13 or is that incorrect? 16:04:17 ah ok, so we would have to have some logic to clean up the discrete snapshot 16:04:38 jgriffith: so, hoping jdurgin can pitch in here but, 16:05:07 afaik the issue is that having too many cow clones can degredate performance of the original volume 16:05:14 i am not clear on this though 16:05:16 but 16:05:22 dosaboy: yeah, that would be a second concern :) 16:05:35 that performance concern also applies to the caseof clones from snaposhots 16:05:41 which is already supported 16:05:42 so 16:05:48 with this bug/feature 16:05:57 i am not trying to solve that sort of issue 16:06:08 but merely extend the ability to have all clones as cow 16:06:29 the inherent potential problems will be no different to what we already have 16:06:32 and imo 16:06:43 will be dealt with seperately 16:06:48 thoughts? 16:07:39 dosaboy: so I'm no ceph expert :) 16:07:41 dosaboy: but... 16:07:59 dosaboy: I've talked with jdurgin about this sort of thing in the past on a different topic IIRC 16:08:18 dosaboy: and it seems there was some issue with having the relationship back to the volumes/snaps 16:08:23 dosaboy: but I'm really not sure 16:08:42 jgriffith: yeah that is what I have heard too, but 16:08:51 dosaboy: the only concerns I would have would be that you can actually somehow have independent volume entities 16:09:00 that applies to what we currently allow 16:09:05 dosaboy: exactly 16:09:24 dosaboy: now I don't even mind if there's some *hidden* special snapshot/volume on the backend 16:09:26 if we go with this change/addition, it will clearly need testing 16:09:39 dosaboy: the perf issue as you mentioned is a whole different subject 16:09:44 just as what we already have i.e. cloen from snapshot will need testing 16:09:48 ah ok 16:09:55 yeah 2 seperate issues 16:10:20 I don't see it as too much of a problem though to have a discrete snap that is silently managed 16:10:39 failry simple logic to do that 16:10:47 dosaboy: I don't either, but I think others raised some concerns when I suggested that in the past 16:10:51 but regardless... 16:11:06 doesn't sound like anybody has any objections here today? 16:11:11 ok well i don't wanna hog the stage 16:11:16 haha 16:11:32 i'll try to get jdurgin's opinion when he;'s about 16:11:35 ok... well that was easy enough :) 16:11:41 :) 16:11:49 #topic API V1 removal 16:11:54 DuncanT-: ^^ 16:11:57 Hey 16:12:00 dosaboy: thanks by the way! 16:12:03 np 16:12:24 DuncanT-: we may have solved your concern here (s/we/thingee/) 16:12:33 DuncanT-: but go for it 16:12:36 Ok, so we've been looking at API migration. AFAICT Rackspace are in the same situation as us: They have the V2 API turned on but it isn't in the catalogue 16:12:57 There doesn't seem to be a way currently to advertise it as a none-default opetion in the catalogue 16:13:01 np 16:13:06 oops :) 16:13:15 It is also not the default in devstack 16:13:29 we should set it to the default first 16:13:38 before we remove v1 IMO 16:13:46 DuncanT-: hemna yes we're workign on that 16:13:53 This limits the testing exposure for V2, and causes third party apps to tend to be targetted for V1 16:14:28 DuncanT-: So I feel v2 is ready with the testing that has been around it/documenting it required whitebox testing, etc. However I worked on the client changes last night to get us http://grab.objects.dreamhost.com/08-06-2013-21-20-59.png 16:14:46 I have two cinder endpoints in the catalog with different service_types like nova did 16:14:51 With this in mind, I'd like to put some caution in the wind as regards to agressively killing V1 16:14:54 thingee: cool! 16:15:02 thingee: Great stuff! 16:15:06 hi all, sorry i'm late 16:15:07 DuncanT-: so between the additon to the catalog and the type=volumeV2 I think we're good 16:15:09 thingee: I'm pretty interested in that patch :) 16:15:14 guitarzan: :) 16:15:24 DuncanT-: I'm also much more confident that there isn't going to be compat issues anyway :) 16:15:26 We need a migration plan and plenty of time to implement it 16:15:30 wait, you named the service type differently rather than a version entry/ 16:15:57 jgriffith: Just flipping the default in devstack broke tempest tests... so there are at least some issues. I've not looked into the details yet 16:15:58 thingee: not that I know which is better... 16:16:04 DuncanT-, guitarzan: so the reason why i feel confident in v2 is because the code isn't far from v1, because people keep backporting features (when they shouldn't ;) ) 16:16:16 DuncanT-: but if you look at *why* it makes sense 16:16:22 thingee: lol 16:16:33 DuncanT-: anyway... not arguing 16:16:48 DuncanT-, guitarzan, jgriffith: I'll take a look at tempest next once we got this patch up for review 16:17:10 thingee: IIRC the failures were the nova/volume/cinder.py issues 16:17:11 thingee: ya, I'm not much worried about the v2 code 16:17:19 jgriffith: I intend to look at / be told 'why', I'm just trying to pave the way for a less than aggressive removal date, based on the low testing so far 16:17:21 it can be fixed if it needs it 16:17:34 DuncanT-: sure, fair 16:17:45 DuncanT-: but were we talking about actually removal yet anyway? 16:17:56 * guitarzan looks at the topic :) 16:17:56 jgriffith: And of course getting people nervious about it is the best way to see more testing / fix-ups :-) 16:18:02 * hemna reads topic 16:18:03 DuncanT-: haha 16:18:08 guitarzan, DuncanT-: can we agree to reevaluate v2 being default if tempest is fine? 16:18:14 DuncanT- +1 16:18:16 v1 is not going anyway in I 16:18:26 thingee: For devstack? Do it as soon as it works IMO 16:18:36 thingee: sure, the default doesn't much matter to me 16:18:43 thingee: That is the kind of 'not agressive' I like :-) 16:18:49 thingee: we just need config options to choose which one 16:18:49 thingee: to be clear, we need to make V2 default ASAP as far as Im' concerned 16:18:55 thingee: there's no reason not to IMO 16:18:56 thingee: Consider me far less worried :-) 16:19:01 whoa 16:19:03 jgriffith: +1 16:19:06 thingee: we have cinder_endpoint_template, we just need cinder_api_version maybe 16:19:13 guitarzan: config opt isn't going away for v1 in I. 16:19:22 thingee: I meant in nova's code 16:19:30 your nova patch just uses the v2 client 16:19:55 alright so let me make sure I understand 16:20:06 guitarzan: is there any good reason to have the internal API's conigurable? 16:20:19 jgriffith: because the internal api is the same as the external one 16:20:35 guitarzan: but that's the point, it doesn't have to be 16:20:36 default v2 in devstack, leave nova code to using v1 (which would require having both v1 and v2 enabled). 16:20:54 thingee: sorry... I'll let you talk I promise ;) 16:21:14 thingee: I like that idea just because it makes us solve the ambiguous endpoints issue 16:21:17 :) 16:21:38 guitarzan: I agreee with that, but I believe that's being fixed regardless 16:21:47 guitarzan: and needs to be 16:21:54 jgriffith: if that's fixed, then I don't care which is defaulted really 16:22:01 guitarzan: :) 16:22:08 DuncanT-: and what about you? 16:22:09 that lets us add v2 to our catalog to make thingee happy 16:22:14 and our customers can use whichever they want 16:22:27 guitarzan: exactly, that's what I'm thinking 16:22:39 guitarzan: leaves the implementation up to the SP 16:22:51 guitarzan: err... configuration ? 16:22:52 +1 16:23:03 Summary from my PoV: With V1 not going away in Icehouse (no new features is fine and indeed sensible) and Thingee on the case for compatibility, I'm quite happy my concerns are addressed. 16:23:07 although customers will probably have to look at their code if/when we add another endpoint 16:24:08 Sounds like that will fix our immediate problems too, so all good 16:24:20 and life is good again :) 16:24:26 whew 16:24:31 next topic, quick! 16:24:37 #topic open discussion 16:24:38 there are none. everyone go! 16:24:39 That was pleasantly pain free :-) Thanks 16:24:44 anybody.... 16:24:58 DuncanT-: I worked late on the patch just for this discussion :) 16:25:14 * med_ tries to recall the project review status from ttx's meeting yesterday.... 16:25:29 thingee: I had faith you'd fix everything once I made any sort of fuss ;-) 16:25:42 med_: it was "looks ok, going to have a trafffic jam in review queue" 16:25:47 jgriffith: update - i successfully did a live migration today. still have to write tests and add a way to get status. 16:25:57 avishay: nice! 16:26:12 avishay: what did you migrate from->to 16:26:13 very cool 16:26:31 jgriffith: both storwize and LVM, via Vish's Nova code 16:26:34 avishay: about that, there was a question this AM about migrating across Cinder nodes? 16:26:39 avishay: nice! 16:26:43 jgriffith: he went from IBM to 3PAR :) 16:26:47 what does that mean? 16:26:54 kmartin: haha! 16:26:56 avishay: block-migrate? 16:27:02 kmartin: 3par to IBM is the more common use case :P 16:27:04 avishay, do you have a WIP up for us to play with ? 16:27:15 avishay: sorry... so say you weren't using multi-backend on a single cinder node 16:27:25 avishay: you actually had cinder node-a, node-b etc 16:27:35 avishay: I believe it shouldn't matter... but 16:27:42 jgriffith: oh, it was actually on a one node devstack, but should work regardless. need more servers to test. 16:27:50 avishay: :) 16:27:51 dosaboy: block-migrate? 16:27:59 hemna: will put something up as soon as it's sane 16:28:05 famous last words! 16:28:09 it should work 16:28:14 avishay, sweet, looking forward to it. 16:28:16 guitarzan: :) 16:28:17 avishay: what type of migration you talking about? 16:28:22 avishay: yeah, I think it'll work the same untl you actually want to migrate to another cinder setup 16:28:25 guitarzan: haha! 16:28:30 dosaboy: moving a volume from its current backend to another 16:28:36 ah ok 16:28:43 avishay: this is for migrating unattached volume, right? 16:28:56 xyang_: unattached is already merged, attached is coming soon 16:29:04 he said "live" so I assumed attached 16:29:05 winston-1: how are things with the QoS patch? 16:29:09 avishay: cool! 16:29:20 and this is the first time cinder is calling nova - we're no longer just a slave :) 16:29:23 med_: xyang_ the unattached code is already in 16:29:34 ack/nod. 16:29:41 avishay: that API is going to be EXTREMELY useful BTW 16:29:42 jgriffith: ok, will take a look 16:29:44 yea...that QoS patch sitting in the queue is bothering me 16:30:04 looks like it's just waiting for reviews 16:30:05 there's a dependent nova patch for QoS too 16:30:15 https://review.openstack.org/#/c/29737/ 16:30:19 winston-1: oops.. didn't notice you updated 16:30:28 i think jgriffith and DuncanT- had concerns, i deferred review to them 16:30:30 Does this mean we've going to want a nova internal endpoint for cinder->nova calls, just like having a specific internal endpoint for nova->cinder calls like attach and reserve is a good idea? 16:30:31 winston-1: sorry... I'll look at that again today, would like to make sure others do as well 16:30:50 Oh... it's not updated ;( 16:30:52 jgriffith, I'll look at it this morning. 16:31:26 jgriffith, what are you waiting for? I don't see a -1 on it? 16:31:26 DuncanT-: we need that for a number of things, se ML last night between Russell and I on the instance assisted snaps 16:31:28 jgriffith: i'm working on it. since it's totally new patch 16:31:35 DuncanT-: I'm not familiar with the internal endpoint - let's discuss offline? 16:31:36 jgriffith: 70% ready 16:31:45 avishay: Sure 16:31:47 hemna: no need, winston-1 and I talked through it last week 16:31:50 DuncanT-: thanks! 16:31:51 ok 16:32:10 can you -1 the review so we know it's in a wait state for changes ? 16:32:11 jgriffith: I'm days behind on the mailing list, will catch up. Glad to see it being discussed 16:32:13 avishay: that's the "cinder talking to nova" piece 16:32:41 avishay: I think we took that as implying that you had an endpoint setup in cinder (ie cinder --> novaclient) 16:32:44 hemna: Done 16:32:57 jgriffith: is this endpoint a piece of code or an actual endpoint (separate port) 16:32:59 ok thanks. :) 16:33:14 avishay: easiest way to describe it is nova/volume/cinder.py 16:33:23 jgriffith: DuncanT-: I have that written already 16:33:26 avishay: but you said we were talking to nova now, so how did you do that? 16:33:35 avishay: yeah.. that's what I thought :) 16:33:38 jgriffith: cinder/compute/nova.py :) 16:33:51 avishay: most EXCELLENT! 16:34:04 avishay: looking forward to that patch :) 16:34:09 DuncanT-: is that what you meant? 16:35:05 Ok... couple of other things real quick and everybody can get back to reviews (hint hint) 16:35:07 avishay: I meant that you probably don't want these cinder->nova APIs enabled on a customer facing endpoint, in teh same way you don't really want cinder reserve API available on a customer facing endpoint 16:35:27 avishay: Can discuss on channel after if you want 16:35:39 DuncanT-: OK, so that something else. Yes, let's talk after. 16:36:23 So the only other thing I wanted to mention 16:36:32 we've got some monster reviews in the queue 16:36:37 task-flow 16:36:42 is a big one 16:36:56 I'm pretty comfortable with it and would like to get it turned on ASAP 16:37:01 the earlier the more testing 16:37:09 I'll try and do more reviews today, was doing a bit last night 16:37:14 I've run a bunch of create cases with forced failures and so far so good 16:37:15 bye 16:37:31 there are 3 huawei monsters at the bottom of the queue... 16:37:48 avishay: yeah, huawei and coraid and nexenta 16:37:51 BUT 16:37:53 I'll take another look at task-flow. The last things I spotted were all minor 16:38:07 I'd like to sort through things like public volumes R/O volumes first 16:38:18 DuncanT-: great.. thanks :) 16:38:18 jgriffith: i've been working on coraid and nexenta recently 16:38:24 avishay: excellent 16:38:29 OHHHH 16:38:30 jgriffith: thanks, it's ready to get your +2 IMO :) 16:38:32 That reminds me 16:38:53 some minor changes which DuncanT- spotted has been addressed. 16:38:56 jgriffith: yup you're right, read-only just got a new update, we should get these features in 16:39:01 I put together a quick driver-cert last week-end 16:39:10 jgriffith: oooooh shiiiiiny 16:39:10 how is that going? 16:39:18 avishay: I'll get it up on github 16:39:26 sweet 16:39:31 ^^ everyone 16:39:31 jgriffith: Error: "^" is not a valid command. 16:39:39 So here's my dilema right now.... 16:39:42 * hemna readies his git clone command line...... 16:39:57 1. I wrote it in python, it checks devstack and cinder and runs the tempest api/volume tests 16:40:08 avishay: yes, thanks. and jgriffith i'm working on mutliple-attaching now 16:40:17 2. It's extensible so you can add things like compute, networking etc 16:40:21 zhiyan: awesome 16:40:47 So I'm thinking it doesn't belong in "cinder" should be outside somewhere for others to make use of and contribute to 16:40:58 why not in tempest itself ? 16:41:10 tempest has a subproject for stress tests 16:41:12 the question is... is that place, devstack, tempest or maybe a project in packstack? 16:41:15 errr 16:41:28 ^^ stackforge 16:41:29 jgriffith: Error: "^" is not a valid command. 16:41:37 why not cinder/tests/functional or something? just for the run script 16:41:38 uvirtbot: welcome back! 16:41:39 jgriffith: Error: "welcome" is not a valid command. 16:41:49 haha 16:41:57 uvirtbot is not very sociable 16:41:58 avishay: Error: "is" is not a valid command. 16:41:59 avishay: well the only reason is like I said others could use this as well 16:42:33 other projects? 16:42:36 tempest seems like a logical place to me, especially if you think others can use it. 16:42:59 oh, the packaging of results, yes definitely 16:43:35 K... I'll fiddle with it some more and get input from everybody later this week 16:43:39 yea tempest could be the right place 16:44:15 jgriffith: there's some hook to configure the backend, or just supply a cinder.conf file, or flags? 16:44:15 Ok... anything else from folks? 16:44:27 med_: did you want more details from yesterdays project udpate? 16:44:36 noe 16:44:38 nope 16:44:43 avishay: nope, it let's you do that. Assumes a devstack env 16:44:58 avishay: then collects your cinder.conf and git stamps of the repos of interest 16:45:16 med_: kk 16:45:29 jgriffith: OK cool 16:45:43 alrighty then... if nobody has anything let's finish early for a change :) 16:45:48 thanks everyone! 16:45:56 #endmeeting