16:00:31 #startmeeting cinder 16:00:32 Meeting started Wed Nov 7 16:00:31 2012 UTC. The chair is jgriffith. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:00:33 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:00:34 The meeting name has been set to 'cinder' 16:00:53 winston-d: rongze_ around? 16:01:10 rnirmal: creiht 16:01:11 jgriffith, should be, let me check 16:01:15 hi 16:01:22 no need check 16:01:26 rongze_: :) 16:01:58 howdy 16:02:04 creiht: hey there! 16:02:19 hello 16:02:27 j_king: howdy 16:02:35 alright, we're sure to have some stragglers but... 16:02:38 let's get started 16:02:48 #topic gate tests 16:03:08 For those that didn't notice the past few days 16:03:25 We had a non-deterministic failure in Cinder 16:03:33 401 error when talking to the client 16:04:09 Good news https://review.openstack.org/#/c/15541/ 16:04:29 I went down few rat hold with our old friend the zero out on delete :) 16:04:37 Which brings me to the point 16:04:53 Currently what's in the code regarding the secure erase: 16:05:09 1. A secure_delete FLAG was added defaulting to True 16:05:26 2. Gate configs were changed to set the flag to False 16:05:51 3. I dropped the dm_mapper(remove) call (this was the cause of the IO errors in kern.log) 16:06:13 I'm not crazy about leaving the devstack gate configured the way it is right now 16:06:18 but wanted thoughts from others? 16:06:26 am I being paranoid? 16:06:28 i'm fine 16:07:07 The other side of it is I think I'm going to implement that differently anyway 16:07:34 Either a simple "cp /dev/zero /dev/mapper/d-n" or maybe something more sophisticated 16:07:45 Like snapshot /dev/mapper/zero 16:07:51 (on volume create) 16:08:11 If anybody has any strong opinions/concerns lemme know 16:08:18 Or if you have a *better* idea 16:08:51 #topic blueprint targetting 16:09:13 I've been going through the bp's: https://blueprints.launchpad.net/cinder 16:09:23 talked with rongze_ and winston-d last night on a couple 16:09:40 but wanted to get some more feedback on the Grizzly ones 16:10:12 I'm proposing dropping shared-volume https://blueprints.launchpad.net/cinder/+spec/shared-volume 16:10:35 agree 16:10:41 indeed 16:10:43 It would be a nice feature but spanning it across multiple compute nodes would be a problem 16:11:01 particularly since alot of targets don't support multi-initiator 16:11:12 jgriffith: it can actually be very useful for sharing a large data set among many nodes (read-only) 16:11:12 Ok... so if no objections, I'm going to drop that one 16:11:24 jdurgin1: agreed, completely 16:11:39 jdurgin1: but for the iscsi case it's a bit of an issue to implement 16:12:27 I don't mind dropping it for grizzly if no one's interested in it 16:12:48 jdurgin1: I'd like to see it, just priority wise, not sure 16:12:58 yeah, that makes sense 16:13:05 jdurgin1: So I'll drop it from Grizz, and if we can get it great 16:13:08 jgriffith: yeah I'm fine with that 16:13:15 cool 16:13:38 I agree 16:13:42 The hard one: https://blueprints.launchpad.net/cinder/+spec/efficient-vm-boot-from-volume 16:14:25 I'd really like to get something concrete on this one 16:14:47 the image read/write features get us part way there 16:15:09 but we need to put some thought into how we can deal with the whole image transfer/caching etc 16:15:20 I'm thinking Island may help with this? 16:15:51 It need tack a snapshot? 16:15:53 we did some stuff on this for QCOW bootable volumes, but it's still not particularly fast. 16:16:06 jgriffith: how much of that is backend dependent? 16:16:25 creiht: That's kind of a problem actually 16:16:34 I'm still confused about what this blueprint is proposing to implement 16:16:39 it depends on glance as well 16:16:52 jdurgin1: yeah, I think the first step is to clean that up 16:16:56 jdurgin1, me too 16:17:02 jdurgin1: The way I've interpretted it is: 16:17:28 Allows you to use the same back-end for glance as for your instance_storage 16:17:28 Vincent is not here 16:17:54 so if a back-end can do things like fast cloning etc etc you can take advantage of it 16:18:21 but there's some other things that folks brought up at the summit that they'd like to see 16:18:41 Unfortunately NONE of them really articulated the problem they were trying to solve 16:18:56 jgriffith: also, aren't there similar issues for local storage? 16:19:06 perhaps someone could write up a specification for that blueprint? 16:19:15 creiht, i think so 16:19:17 creiht: which issues? 16:19:29 You mean the cross node problem? 16:19:45 Or you mean solve it for local storage as well? 16:19:46 the having to load images slowly to boot an instance 16:19:47 no, compute node have to copy image from glance to local disk 16:19:53 creiht: yes 16:20:05 creiht: sorry, was just using the back-ends as an example 16:20:38 creiht: That's actually the case that received the most attention at the summit 16:20:48 yeah 16:21:02 creiht: and that's where i wonder if Island might have some ideas we can use 16:21:08 cool 16:21:27 sounds like we're all in the same boat on this blueprint.... needs some detail/direction 16:21:36 how did you do it in the solidfire presentation? i remember you were using solidfire for instance storage/glance as well as cinder back-end 16:21:48 I'll see what I can do for that, and I think I'll end up targetting G3 16:22:04 cool 16:22:12 winston-d: a little bit of hacking :) 16:22:22 winston-d: It wouldn't fit the *general* case though 16:22:33 winston-d: It was specific for us 16:22:37 jgriffith, ok 16:23:03 winston-d: Really we had the same problem with sucking images out of glance etc 16:23:25 winston-d: But we use an SF volume for instance_path 16:23:49 winston-d: our internals then do some cooll stuff to optimize... but anyway 16:24:04 as long as nova/cinder has to talk to glance via api to get image, there should be a new API sort of stuff to allow nova/cinder to figure out what storage glance is using. 16:24:11 Ok, I'll work on flushinng that bp out a bit 16:24:35 winston-d: that was added in Folsom, it's how the rbd cloning to volume works in Folsom 16:24:51 jdurgin1: +1 16:25:10 so in my view that was a first step 16:25:18 jdurgin1, sorry i missed that. mind give me pointer? 16:25:28 I think we can exploit that going forward 16:25:39 jgriffith, yes, please 16:26:44 The only other two that we should probably talk about 16:26:53 #link https://blueprints.launchpad.net/glance/+spec/api-v2-store-access 16:26:55 iscsi-chap and multi-backend 16:27:05 #link https://blueprints.launchpad.net/cinder/+spec/effecient-volumes-from-images 16:27:44 I'd also like to mention the Glance meta one. 16:27:45 https://blueprints.launchpad.net/cinder/+spec/retain-glance-metadata-for-billing 16:28:12 dtynan: ahh yes 16:28:16 we've implemented that in our version of diablo and would like to get it upstream 16:28:25 dtynan: that's great with me 16:28:36 we're targetting G1... ;) 16:28:42 dtynan: if you can do it that's cool 16:28:51 dtynan: alright, I'll target it and assign to you 16:29:01 jdurgin1, thx! 16:29:02 dtynan: G1 is like two weeks away, sure you can hit that? 16:29:18 well.... 16:29:37 dtynan: hmmmmm 16:29:44 we've been busy getting it into production here @ HP 16:29:54 dtynan: Yup, we're all busy :) 16:29:58 so we know what the issues are... 16:30:07 dtynan: This is why I left it sitting 16:30:14 dtynan: So it's your call, and what you can commit to 16:30:20 but from an HP-diablo pov, it'll be in production imminently. 16:30:28 yeah, nothing like a challenge, eh? 16:30:30 dtynan: My concernn is that it just sits there forever 16:30:44 diablo haha 16:30:47 :) 16:31:07 dtynan: so tell me what you want to commit to? 16:31:13 yes. 16:31:35 dtynan: dtynan yes? 16:31:48 dtynan: Is that yes for G1, yes for sitting forever? 16:31:49 :) 16:32:04 yes for G1. 16:32:17 Okie Dokie... you're officially on thehook 16:32:49 On to iscsi-chap 16:32:57 I propose .... 16:33:03 wait, did I already bring that up 16:33:05 think I did 16:33:32 No... I didn't 16:33:40 I propose that one is dropped 16:33:55 It's been sitting for a long time and the reality is the back-ends immplement it 16:34:04 Haven't heard too much yelling forit 16:34:30 if vincent comes back and works on it fine, but we should remove the targetting 16:34:33 agree? 16:34:45 sure 16:35:09 fine with me 16:35:13 agree 16:35:17 The other big items are: 16:35:50 FibreChannel, multi-back-end and the API general bucket 16:36:12 Anyy of the folks from HP, IBM or Brocade here today? 16:36:29 Ok... multi-backend 16:36:39 we need to decide on how we want this 16:36:50 rnirmal: provided a working implementation 16:37:09 rnirmal: do we go that route or do we do multiple services/proceses on a single box 16:37:18 HP. 16:37:30 dtynan: yeah, wrong HP group :) 16:37:38 :) 16:38:06 as a refresher: https://review.openstack.org/#/c/11192/ 16:38:13 That's the proposal from rnirmal 16:38:14 jgriffith, i prefer multiple services/processes 16:38:52 winston-d: That's the direction I'm leaning too 16:39:05 that seems a lot easier to configure 16:39:06 winston-d: It solves some concerns that were raised by others 16:39:17 and let's us use the volume_type scheduler :) 16:39:24 as long as there isn't a high level of co-ordination required, I'd be up for multiple processes 16:39:55 j_king: My initial thought is something like multiple managers running in their own process based on back-end 16:40:43 use all the same concepts we use today for multiple volume node configs 16:41:02 creiht: thoughts? 16:41:37 sounds good. I generally prefer the more simple implementation. 16:42:20 jgriffith: I don't have a strong opinion here 16:42:22 my keyboard is going nuts here 16:42:30 creiht: fair enough 16:42:45 Ok, let's proceed with multi-proces 16:42:55 jgriffith: sorry came in late 16:42:55 but your proposal seems reasonable 16:43:07 rnirmal: No prob 16:43:25 rnirmal: We were talking about the multi-back-end immplementation 16:43:45 so multi process within the same manager ? 16:44:02 rnirmal: I was actually thinking multiple managers 16:44:02 or multiple managers ? 16:44:09 multi processes in multi managers 16:44:19 jgriffith: well we don't have to do anything for it 16:44:22 each manager has its own process 16:44:25 :) 16:44:38 and the whole reason for the multi-backend was to get away from having to run multiple managers 16:44:45 winston-d: that's how it's right now 16:45:01 rnirmal: multiple managers on the same node 16:45:17 rnirmal, i thought your concern is too many volume service node to manage 16:46:11 winston-d: also services right... 20 or so init scripts for each one ? 16:46:33 I'd prefer a single configuration to load all the backends... if we do multiple processes within the same manager... that would be fine too 16:46:45 rnirmal, that can be changed, adding one new binary in bin/ can solve that problem. 16:46:56 manager could just load a backend in each process 16:47:00 TBH if you're looking at 20 back-ends I'd rather the init scripts that trying to get the config files correct 16:47:06 just have to config the manager 16:47:57 jgriffith: but why is a single config more harder than 20 configs ? 16:48:17 rnirmal: Why is having an init for each manager harder than 20 configs? 16:48:43 jgriffith: :) 20 log files to parse... 20 everything 16:48:43 rnirmal: just an example, maybe a poor one 16:48:58 rnirmal: hmmm... ok, you've got me there :) 16:49:00 separate config files is generally easier for config management to deal with 16:49:27 the complexity comes with a new layer 'backend' in single manager design. 16:49:44 i'd rather to have separate log files, IMHO 16:50:01 winston-d: +1 16:50:11 winston-d: with what I proposed you can still do that 16:50:18 it gives the option to do both 16:50:21 winston-d: +1 16:50:28 winston-d: you still could with a single manager that manages several backend processes 16:50:32 you are not tied to any particular way 16:50:35 the process just logs 16:50:52 rnirmal, it's true and i think multi manager design can also combine multi configuration files into one 16:51:08 without adding complexity to introduce a new layer 'back-end'. 16:51:44 winston-d: how would you distinguish 20 backends in the current design -> 'host' ? 16:51:49 that's bad as is 16:52:03 irrespective we still need the concept of a 'back-end' 16:52:13 rnirmal, via volume-topic 16:52:14 merely more than just a driver... 16:52:22 winston-d: how? 16:52:43 don't tell me run on multiple hosts 16:52:46 that's how it's right now 16:52:50 as long as different back-end instance has it unique volume-topic, scheduler is able to find it. 16:53:29 how is that more different than with a 'back-end' 16:53:43 Ok, seems we have a bit to work through on this :) 16:53:46 the scheduler either needs to know which 'volume-topic' or which 'backend' 16:54:02 which 'backend' seems more clearer than which 'volume-topic' 16:54:19 rnirmal: why? 16:54:34 just canonically 16:54:46 meh... maybe 16:54:56 So here's the thing IMO 16:55:06 There are advantages and disadvantages to both 16:55:22 What I'm trying to figure out is which is going to be more robust and supportable 16:55:34 with an emphasis on robust 16:56:16 and able-to-scale 16:56:51 winston-d: true, but I think both can scale, just not as clean maybe 16:57:12 right 16:57:25 jgriffith: I'm all for a better solution if we have one :) 16:57:47 rnirmal: well the problem is I think your solution is great 16:58:11 rnirmal: that being said I want to look at all possibilities 16:58:28 So I think what needs to happen is we need to have some code to compare 16:59:01 so maybe an implementation using multiple managers that at least *functions* so we can look at the two together 16:59:07 see what comes out of it? 16:59:20 seem reasonable? 16:59:24 jgriffith: sorry, I'm here a hour late due to Daylight saving apparently. 16:59:28 jgriffith: sure. 16:59:32 kmartin: you and a few others :) 16:59:39 ;) 16:59:53 kmartin: got caught in daylight savings as well :) 17:00:23 ditto 17:00:39 anyway, I think there's enough interest and merit in the multi-manager approach that we should at least pursue it a bit 17:00:57 jgriffith: agreed 17:01:22 ok, and we always have the first implementation that we can go with when it comes down to it 17:01:31 alright... pheww 17:01:57 Ummm... I wanted to save some time to see if folks had bp's they haven't gotten around to filing yet 17:02:14 * jgriffith is looking at creiht 17:02:46 A far as the FC blueprint with details has made it's way through HP legal is now getting reveiwed by the entire team as we meet again tomorrow morning. 17:02:56 thingee: btw, wanted to see what your blockers are 17:03:18 kmartin: wow.. beuarocracy at it's finest :) 17:03:33 jgriffith: yep 17:03:47 jgriffith: my two bps list bootable vols and clearer error messages are blocked by my apiv2 bp 17:03:56 should make that more clear on the bps themselves 17:04:01 thingee: ahhh 17:04:22 thingee: yeah, we should mark them as such 17:04:36 for apiv2 I've posted my github branch which basically has middleware separated out and other api common code. tests pass. 17:04:44 thingee: and maybe make dependencies instead of listing as blocked 17:04:46 it's all in the bp 17:04:56 jgriffith: sounds good 17:05:07 thingee: yeah, was looking at that last night, looks good so far 17:05:14 thingee: Nice work on that for a week-end :) 17:05:37 thingee: unless you were just hiding it prior to that :) 17:06:06 jgriffith: got about 26 tests failing atm for routing stuff. This is a bigger change that I was hoping and a bit of learning curve for me, but I've learned a lot about paste over the weekend and just gotta get things organized to model glance correctly 17:06:27 thingee: I hear ya 17:06:28 @jgriffith I believe there are some folks here at HP planning to submit a bp around volume backup, to swift object store 17:06:52 thingee: make sure you get the fixes for the monkey patching before you try and run with real endpoints :) 17:07:19 meant to mention that volume-backup BP... ;) 17:07:27 jgriffith: I hear that 17:07:36 dtynan: ollie1 noted 17:08:01 keep in mind, there's some code in island that might be leveraged for this as well 17:08:24 and the Lunar folks have some interest so they should be updated as well 17:08:35 thingee we did some stuff for list-bootable-volumes but it was through an hp-specific API hook (for now). I'd be interested in reading your BP. (and will do..) 17:09:18 dtynan: I grabbed it because it seemed doable for me, but I'll be honest, I haven't started yet with some of the other stuff on my plate. 17:09:23 @jgriffith I'll pass that on 17:10:24 I'll be focusing on apiv2 this week. would love to get to it next week if all goes according to plan (heh). 17:10:28 thingee: cool. I'll take a look. we figured an hp-specific API was the best (short term) way of getting the functionality before it's in the mainstream. 17:10:49 thingee: If G1 is possible that would be ideal 17:11:01 thingee: we can of course add to it after that, but 17:11:32 thingee: we should have that in for things to build off of as you pointed out with your other BP's 17:12:00 kmartin: how are you guys feeling about the FC work? 17:12:19 kmartin: I know you're meeting tomororw,but what's the general feel so far? 17:12:24 jgriffith: definitely! I'll be focusing hard on making that happen. 17:12:34 thingee: great, thanks! 17:12:42 jgriffith: the BP looks good and we're shooting for G2 17:12:49 kmartin: G2? Really? 17:12:53 kmartin: Awesome! 17:13:05 that the plan may slip into G3 17:13:12 Ok... we're over, anything else? 17:13:17 kmartin: haha 17:13:24 I'll target G3 :) 17:13:42 fyi we're also working the bp to extend volume usage stats: https://blueprints.launchpad.net/cinder/+spec/volume-usage-metering 17:13:46 sounds safer :) 17:14:00 if you guys have time, please take a look at here: https://etherpad.openstack.org/cinder-backend-capability-report 17:14:49 ollie1: can you update the bp? 17:14:51 that's for back-end to report capability/capacity for scheduling purpose. 17:14:59 ollie1: and what is your timeline? 17:15:18 winston-d: thanks for reminding me :) 17:15:23 I'll check that out with the person on it and update the bp 17:15:31 winston-d: looks like a good start 17:15:46 the guy who did the volume-usage-metering code is on vacation this week. 17:16:05 ollie1: ok, just FYI I'm going to hack up blueprints tonight with prejudice :) 17:16:30 dtynan: ahhh 17:16:44 that's just some examples, we should have mandatory ones and optional ones to report. 17:16:44 alright, well no big deal. We never stop accepting blue-prints 17:16:47 we'll round him up next week, and get him busy on the upstream work :) 17:16:53 it's implemented internally. 17:16:55 winston-d: yes, I'm with ya 17:17:13 winston-d: it's a good model to start and see what we're doing though 17:17:32 alright.... i should probably wrap this up 17:17:41 winston-d: just a heads up, I expect most of those to not make sense (and thus be optional) from a ceph perspective 17:18:07 jdurgin1: I think we'll tweak this a bit to make more sense, but... 17:18:29 jdurgin1: I was thiking that for any we say are mandatory but they don't apply just set None? 17:18:45 jdurgin1: so *kinda* mandatory 17:18:54 jgriffith, jdurgin1, make sense 17:19:11 jgriffith: might as well make them option imo, but we can discuss later 17:19:13 jdurgin1: The mandatory part is just that you can be queiried and return the field 17:19:50 jdurgin1: so maybe better language is that you have to implement the report_capabilities method 17:20:02 mandatory is minimum set of capability that allow built-in filters to work, i think 17:20:04 jdurgin1: what you put in there has guidelines if yousupport 17:20:23 we can hash it out 17:21:07 jdurgin1: but I think theres value in the function, and eve the basics like driver version etc 17:21:17 that makes more sense, but we're over a bit already 17:21:23 we're way over 17:21:25 alright 17:21:30 thanks everyone 17:21:38 don't forget DST next week :) 17:21:50 #endmeeting