16:04:11 #startmeeting cinder 16:04:12 Meeting started Wed Nov 28 16:04:11 2012 UTC. The chair is jgriffith. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:04:13 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:04:15 The meeting name has been set to 'cinder' 16:04:17 o/ 16:04:31 hello 16:04:31 hi 16:04:34 hi 16:04:34 hello 16:05:00 bswartz: The official cut is hopefully tomorrow 16:05:16 bswartz: the mail list has quite a bit of info/discussion/updates 16:05:37 bswartz: The cut date however has passed and we got everything for cinder in that we wanted 16:06:01 alrighty... I didn't update the meetings wiki this week.... sorry about that 16:06:15 I just had a couple of things then I'll open it up for everybody else 16:06:25 #topic FC update 16:06:33 kmartin: Have any updates for us? 16:07:11 working on the code will add the details on Friday to the spec 16:07:29 kmartin: Ok... is the code up on github or somewhere that we can get a preview? 16:07:42 did you see hemna question yesterday regarding the the need to enter a nova bp or not for the linvirt changes? 16:07:54 libvirt 16:07:56 kmartin: I did not see that I don't think 16:08:10 kmartin: But yes, there would need to be BP's for the nova side as well if that was the question 16:08:54 * thingee gives bad advice, sorry guys! 16:08:57 kmartin: Was that the question? 16:08:59 basically, do we need to enter a bp for the libvirt changes on the nova side or just reference the cinder bp in the commit 16:09:22 kmartin: I'd do a BP for the Nova side and reference it as required for the Cinder BP 16:09:36 kmartin: make sense? 16:09:44 ok, will do that 16:09:47 cool 16:09:56 alright... anything else on the subject? 16:09:59 I hope we don't have to go through legal again 16:10:06 kmartin: me too :) 16:10:11 :) 16:10:16 kmartin: If that's a risk we can talk offline 16:10:31 jgriffith: ok thanks 16:10:34 kmartin: Stupid to tie things up for another period of weeks for that silly crap 16:10:43 I agree 16:10:54 alright... well, let me know how that shapes up 16:11:03 sure, will do 16:11:09 and if you can get the code public sooner rather than later that would be good too 16:11:15 kmartin: thanks 16:11:20 #topic LIO target 16:11:26 eharney: :) 16:11:35 hi there 16:11:38 eharney: So first sorry if I misundersood your initial intent there 16:12:00 eharney: I think that doing an LIO version of iSCSI is the right way to start 16:12:07 it's ok... so, it sounds like the current leading idea is to do a targetcli-based one first 16:12:26 eharney: I think that's the right way to go 16:12:50 eharney: I'd like to start with that as the iscsi subclass like we talked about 16:12:58 i don't see any problem with that from my end 16:12:59 right 16:13:15 eharney: Then I'd like to see how things might shape up for a full remote driver for Grizz 16:13:32 eharney: That use case is particularly interesting IMO 16:13:42 eharney: Remote LVM boxes :) 16:14:03 eharney: But I think phasing it and learning as we go might be good 16:14:24 this makes sense to me 16:14:31 eharney: great 16:14:39 eharney: anything else you want to bring up with folks? 16:14:40 my only real issue i've found was that targetcli may need some work, but it shouldn't be anything too bad 16:14:55 i don't think so 16:15:05 eharney: what kind of *work*? 16:15:12 eharney: error reporting etc? 16:15:14 eharney: are you doing anything on the iscsi initiator side, or just target stuff? 16:15:32 yes, error reporting -- it doesn't really do it in a useful fashion for us 16:15:48 i.e. never returns codes, only prints colorful messages 16:15:59 eharney: hmm... yeah that was one of the concerns that somebody mentioned on the ML 16:16:09 bswartz: just the target side 16:16:32 eharney: well, depending it might be a good opportunity to improve tgt-cli as well :) 16:16:53 the alternative was to interface with python-rtslib rather than targetcli, but it has a different set of issues 16:17:03 since it's library code that expects to run as root, and doesn't really fit in the rootwrap scheme well 16:17:11 eharney: yeah, that didn't seem *ready* IMO 16:17:25 well, targetcli uses that library 16:17:38 but yes, i think i have a good plan to go forward for now 16:17:49 eharney: Ok, keep us posted 16:17:58 ok 16:18:03 eharney: thanks! 16:18:15 #topic capabilities reporting 16:18:27 So we started talking about this a bit 16:18:50 I'd like to propose we start moving forward with what is on the etherpad 16:18:53 https://etherpad.openstack.org/cinder-backend-capability-report 16:19:12 It's been idle with the exception of a few things I added this morning 16:19:35 I'd like to start implementing this and we can always tear it apart in review if folks aren't happy with it 16:19:39 jgriffith: is this going to be a new driver entrypoint to be consumed by the scheduler? 16:19:53 bswartz: the capabilities? 16:19:57 yes 16:20:24 bswartz: Could be I suppose 16:20:36 what else is it intended for? 16:20:49 bswartz: The goal for me right now was to be able to report back to a user what a back-end can do 16:20:58 bswartz: Well... a user but more importantly the API 16:21:13 bswartz: One example is the 'online_clone' parameter 16:21:31 bswartz: unless someobdy has a better idea for how to store/report that sort of thing? 16:21:41 by "user" do you mean "end user" or "administrator"? 16:21:59 bswartz: might be either... 16:22:06 jgriffith: with something like "online_clone", what would a user do with that knowledge? 16:22:13 bswartz: There are cases for both 16:22:25 avishay: That's not a good example :) 16:22:34 :) 16:22:35 avishay: That's more for use by the API and Manager IMO 16:23:00 avishay: The user would just get an error message if they tried to do a clone of an attached volume if it was False 16:23:04 So what's a good example of something the user should be aware of? I think the manager/scheduler should be making most/all of these decisions 16:23:21 avishay: actually.. I thnk you're correct 16:23:21 it seems like users should be kept in the dark about driver capabilities (because there may be multiple drivers, and the administrator doesn't want them to know) and the administrators could get everything they need from the driver documentation 16:23:33 bswartz: agreed 16:23:49 so correct my wording above.... 16:24:09 the scheduler seems to be the main consumer of the this API as far as I can see 16:24:12 The goal for me anyway was the ability to do things in the manager and API 16:24:23 bswartz: sure, ok 16:24:37 bswartz: so just to clarify... 16:25:00 okay, I like the idea of a capabilities API so that schedulers can make smarter decisions 16:25:00 bswartz: The intent was to have a methods in the drivers to query and return this info 16:25:57 bswartz: Ok, sounds like we all agree at least at a high level 16:26:10 bswartz: avishay think we need to get some code up to get in to the details 16:26:33 greetings 16:26:34 Any other thoughts on this? Anything horrible on the etherpad, or anything obviously missing? 16:27:14 jgriffith: this is very closely tied to the filter scheduler that's under review, right? 16:27:15 well unless we can see other users for the API, I think we should design the API around things that we know the scheduler will be able to make use of 16:27:55 and it probably ties in with volume_type_extra_specs 16:27:58 avishay: yes, there's potential for that 16:28:06 bswartz: yes 16:28:21 So I think there's a number of possibilities this opens up 16:28:46 anyway... just wanted to make sure we had at leat general agreement to move forward on it 16:29:02 s/leat/least/ 16:29:33 bswartz: So to be clear I had not intended a *new* API 16:29:58 what about support for modifying a volume's type? (obviously not for the first pass) 16:30:20 avishay: I dunno... interesting, but what's the use case for that? 16:30:36 jgriffith: tiering, for example 16:30:49 move a mostly idle volume to slower, cheaper disks 16:31:01 avishay: would modifying the volume's type just change the metadata, or actually move the volume to new storage? 16:31:02 avishay: ahh... a migration 16:31:07 maybe within a back-end, maybe across back-ends 16:31:21 possibly migration 16:31:31 migration could get complicated 16:31:35 avishay: yeah... so that's an interesting one that I thought through and *thought* I had a solution for but maybe not 16:31:40 especially cross-backend 16:31:42 if the same back-end supports the new capabilities, it could do some transformation 16:31:59 bswartz: nothing worth doing is easy :) 16:31:59 avishay: My thought was that for same back-end you leave the type (the type can define the back-end actually) 16:32:11 avishay: extra specs could be used for the different levels 16:32:31 avishay: this works really well for the tiering/moving that you described IMO 16:33:01 jgriffith: the way volume_types are today, it's up to the administrator 16:33:08 avishay: The question I was still dealing with is whether it's an active admin operation, or part of the audit to check for changes in the extra_specs 16:33:21 bswartz: Yes, and I think it should stay that way 16:33:24 jgriffith: a backend can have multiple volume_types and a volume_type can map to multiple backends 16:33:53 bswartz: I think if that's how someobdy wants to implement it that's fine 16:33:55 like i said, we don't need to figure this out right now, but it could be nice for the future 16:34:00 bswartz: Or I should say *utilize* i8t 16:34:01 it 16:34:28 avishay: Actually I have plans for this very thing in G3 so we'll talk about it in the new year :) 16:34:41 jgriffith: OK cool - anything written? 16:34:54 avishay: somewher... 16:35:15 migration is certainly a good idea -- but I don't think we'll be able to do anything so simple as changing the volume_type and then getting the right thing to happen 16:35:17 avishay: I'll throw together a google-doc or etherpad in the next couple weeks 16:35:20 OK, if you come across it I'd be interested to see 16:35:34 bswartz: Yeah, I'm not ready to tackle the migration thing yet 16:35:47 bswartz: unless it's migration within a single back-end 16:36:12 avishay: I'll get info out to everyone on it, but it's not time criticial right now 16:36:13 bswartz: at first we'll probably need to mount both volumes on a server and dd from old to new 16:36:21 avishay: ewwwwwww 16:36:28 avishay: :) 16:36:30 jgriffith: i said "at first" :) 16:36:35 avishay: that would be the lowest common denomiator case 16:36:37 LOL 16:36:39 yes 16:36:55 ok... I think we've covered that sufficiently for now. Let's move on 16:36:57 OK, this isn't urgent - we can move on 16:37:02 #topic G2 workload 16:37:25 #link https://launchpad.net/cinder/+milestone/grizzly-2 16:37:36 so check out the BP list 16:37:45 note the assignments... 16:37:53 It's a bit lop-sided :( 16:38:18 I'd really like to see folks step up and help with some of these. 16:38:35 particularly the API enhancements, we've assigned almost all of them to thingee 16:38:52 While he can do it and won't complain we don't want to take advantage of him either ;) 16:39:08 So if you can.. work with thingee and find ways to help him out on these 16:39:28 there's a ton of work here and I also suspect we'll come up with more to add in the next couple weeks 16:39:38 or buy me a beer at the next summit :) 16:39:43 :) 16:39:52 Another thing to keep in mind on G2... 16:40:15 So I made the mistake on G1 of scheduling everything up until the schedule date 16:40:36 Remember that a few days before we go in to lock-down 16:40:42 and the candidate gets cut 16:40:48 so we loose a couple days there 16:40:54 Then... add in the holidays 16:41:19 So realisticly we're almost looking at Christmas we need to have a pretty good handle on these 16:41:28 jgriffith: I can speak for hemna, still good progress... going through the final review process(yep... with HP legal) for the 3PAR array driver. 16:41:45 kmartin: ok... sounds good 16:42:16 So if somebody is assigned something here and they don't think they're going to work on it let me know earlier rather than later 16:42:35 and again, if you wanna help out, sync up with whoever is assigned and see what you can do to help 16:43:11 anybody have any questions/concerns on the G2 proposals thus far? 16:44:04 cool... 16:44:09 #topic NAS support 16:44:32 So there wasn't near as much interest on the ML as I had anticipated 16:44:56 what were you anticipating? 16:44:59 but regardless, I think if bswartz and folks want to work on this it's useful 16:45:18 bswartz: response from more than the three people that I'd already heard from on the subject :) 16:45:41 yeah it seems like a small number are really excited about it 16:46:00 bswartz: Yeah, and I think it's useful/good soo... 16:46:01 others either don't care or aren't willing to speak up 16:46:10 this is support for file in addition to block? 16:46:20 avishay: yup 16:46:35 bswartz: So I'd like to re-iterate my proposal for an iterative path on this 16:46:53 i missed it on the ML i guess. 16:47:02 bswartz: Start with enhancing/improving the NFS driver that you guys already worked on 16:47:23 bswartz: we don't *have* to export it as iscsi if there's time to change it 16:47:35 so the NFS driver is unrelated to the NAS stuff 16:47:48 bswartz: how so? 16:48:15 well -- I guess it's a matter of perspective again 16:48:21 ;) 16:48:29 the 2 features don't overlap code-wise whatsoever 16:48:42 bswartz: I guess what I'm suggesting is that maybe they should 16:49:06 bswartz: In other words rather than start with everything under the sun, why not build up a usable NAS driver in Cinder 16:49:25 bswartz: We can work on that to use connection info other than iscsi as well 16:49:27 the NFS-based drivers can do everything they need to do inside of a driver, with some modifications on the nova side that have already been made 16:49:35 bswartz: CEPH is a good example here 16:50:03 the NAS support that we're proposing is really about delivering CIFS/NFS directly to the guests 16:50:27 bswartz: Yup, and that's what I was eluding to in terms of the provider_location and connection info changes 16:50:30 or if cinder is being used outside of OpenStack (as some are doing) then delivering CIFS/NFS to whatever 16:51:07 bswartz: So I had proposed one of two directions... 16:51:24 bswartz: Either a Driver model (which I still think can be done pretty well) 16:51:27 bswartz: or 16:51:37 bswartz: A seperate service in Cinder 16:51:57 bswartz: the seperate service is in everybodys best interest for a number of reasons 16:52:04 what would a seperate service in cinder mean? 16:52:20 bswartz: So use Nova as an example 16:52:36 bswartz: Inside nova we had: compute, networking, volumes 16:52:50 so I'm saying in Cinder we have: block, NAS 16:53:05 okay, I think that's a good model conceptually 16:53:09 bswartz: it keeps the seperation of topics that I was concerned about 16:53:18 i think that sounds good 16:53:29 +1 16:53:41 bswartz: It also makes things gateable/testable, and if done right it makes life easier for a break out later 16:53:48 jgriffith: +1 16:54:07 this is weird, we're all agreeing on something NAS related :) 16:54:25 or.. I should say, *I'm* agreeing :) 16:54:43 bswartz: it's a bit of work, but I think it's worth the effort 16:55:05 bswartz: if we're going to do this, we should do it as well as possible 16:55:13 jgriffith: we still need to get together to talk, and I'd like to understand what kind of structural changes to the code would allow it to be "a seperate service in cinder" with the reduced risks you mention 16:55:30 bswartz: sure 16:55:41 bswartz: maybe later today we can skype or something 16:55:50 it's a question of at what level we seperate blocks and NAS 16:56:26 this meeting may be scheduled too early in the west coast and too late in europe/ME/asia - people are too tired to be confrontational :) 16:56:41 avishay: HaHa!! 16:56:50 * jgriffith realizes they're on to him 16:56:55 :) 16:57:12 we're all programmers -- we have caffeine to counter tiredness 16:57:22 bswartz: hehe... so true 16:57:32 Ok.. anything else on this topic we have to address right now? 16:57:52 #topic LVM 16:58:01 One other thing I wanted to touch on real quick 16:58:19 I started looking at some of our LVM/dd issues again and it sucks 16:58:45 particularly throughput to an LVM volume that has a snapshot is horrid 16:59:07 if anybody has any tricks up their sleeve or ideas they've been kicking around on this please let me know 16:59:33 Is it the LVM code that's slow or the way the I/O goes to disk? 16:59:49 put another way: would it still suck if an SSD was used intead of spinning rust? 16:59:53 bswartz: it's more of the way LVM snapshots work than anything else 17:00:02 bswartz: Yeah, pretty much 17:00:24 bswartz: because when there's an LVM snap you're going through an exception table for every write and mapping it to the correct device etc 17:00:29 bunch of overhead there 17:00:34 perhaps LVM is not the right base for the generic driver then? 17:00:44 bswartz: correct 17:00:55 what's the status of the re-write going on for the kernel component of LVM? is that done? it was supposed to support thin provisioning and have better snapshot support 17:01:03 i stopped tracking it a while ago 17:01:18 avishay: I've been looking at that a bit, but the results aren't that much better TBH 17:01:30 avishay: the thin-provisioning itself is still compelling though 17:01:35 jgriffith: OK, disappointing 17:01:48 what other options are viable? 17:01:55 Really I could live with all of this EXCEPT for the kernel bug 17:02:21 remember the dd on delete kernel hang.... 17:02:35 Anyway, I wanted to just sort of throw this out again for folks to think about 17:02:57 #topic free for all 17:03:09 speak up if you have something :) 17:03:24 can LIO help with the above problem? 17:03:33 bswartz: funny you should ask that :) 17:03:35 if we had a LIO driver could we wire it up to some less-sucky backend? 17:03:48 bswartz: I had this crazy idea of using LIO with a file back instead of LVM 17:04:01 bswartz: but I don't thnk that would fly in a production env 17:04:10 you still need to get snapshot support somehow 17:04:20 jgriffith: that's what i thought too 17:04:25 bswartz: So with LIO you can literally use a file as your storage, but it's recommended to use LVM 17:04:45 btrfs? too immature? 17:04:50 avishay: :) 17:04:52 what if you used LIO with a file-based storage, but you used a more powerful filesystem like BTRFS? 17:04:54 i was just thinking about btrfs too, as a longer goal... 17:05:19 avishay: eharney BTRFS has been my latest, but I've read mixed results on how ready it is 17:05:33 avishay: eharney I'm still willing to investigate it a bit 17:05:44 it appears to be pretty rough around the edges still from what i understand 17:05:52 is it still being actively worked on? unfortunately i lost touch with my kernel roots a bit 17:06:17 avishay: it is, but seems to have lost some of it's appeal from what I've been seeing 17:06:18 avishay: i think there is active development ongoing, yes 17:06:51 NetApp actually implements LUNs as files inside our storage controllers, and all of the power comes from the filesystem used 17:07:05 if BTRFS was more production ready I think it would make a great solution 17:07:17 bswartz: so everyone should just buy netapps ;P 17:07:22 avishay: :) 17:07:26 YES~ 17:07:29 YES! 17:07:33 I was thinking NetApps should be free! 17:07:54 i would totally test it out if you wanted to send me a free one. 17:08:04 So I have considered making RBD the ref implementation but I dunno... that's a big switch 17:08:16 that applies to all storage vendors in the room :-p 17:08:25 russellb: :) 17:08:29 :) 17:09:02 alright... we're over, and nobody bit on the RBD thing so let's wrap up for now 17:09:14 have the kernel issues been reported? any response? maybe moving to something else is a bit hasty? i don't know the history 17:09:14 * jgriffith fishes, fishes, fishes 17:09:34 avishay: Oh yes, it's reported and there is some work on it but very slow progress 17:09:45 avishay: You're correct, I may be being very hasty 17:09:49 is RBD more mature than btrfs? 17:09:50 jgriffith: if ceph was packaged by the distros and stable then I'd say it was a reasonable idea 17:10:18 any FC news ? 17:10:26 avishay: well, I believe so... but the real tie in there is it's already an OpenStack option 17:10:33 zykes-: You missed it, check the logs 17:10:42 OK, I need to go - 10 minutes over - see you all next week! 17:10:48 alright folks, I've kept you 10 over already 17:10:52 thanks everybody 17:11:01 catch me on IRC if you need anything or have further questions 17:11:05 #endmeeting