16:02:15 #startmeeting cinder 16:02:15 Meeting started Wed Jan 16 16:02:15 2013 UTC. The chair is jgriffith. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:02:16 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:02:19 The meeting name has been set to 'cinder' 16:02:26 o/ 16:02:27 indeed! 16:02:35 o/ 16:02:35 hello all 16:02:40 How's the new swimming pool John? 16:02:46 hi 16:02:51 hello 16:03:04 DuncanT: at least it will be a fancy below ground pool.... 16:03:14 DuncanT: If I can find the darn thing 16:03:15 hi 16:03:24 jgriffith: good luck! 16:03:25 Wow.. good turn out this week 16:03:31 avishay: Yeah, thanks! 16:03:33 hi 16:03:44 hi 16:03:46 Ok.... let's start with the scality driver 16:03:56 JM1: has joined us to fill in some details 16:04:04 hello 16:04:07 hi everyone 16:04:13 https://review.openstack.org/#/c/19675/ 16:04:41 #topic scality driver 16:04:45 I see that DuncanT just added his own comment 16:05:15 JM1: Yeah, and after our conversation yesterday my only remaining concern is missing functionality 16:05:33 I didn't expect the lack of snapshots to be such an issue 16:06:09 From my POV, I think people expect the things that have always worked in cinder to continue to always work, regardless of backend 16:06:26 hmmm 16:06:42 JM1: Keep in mind that the first thing a tester/user will do is try and run through the existing client commands 16:06:45 I saw that the NFS driver doesn't support snapshot either 16:06:53 but I suppose some people use it? 16:06:56 JM1: Yeah, and it's been logged as a bug 16:07:04 :) 16:07:06 ok 16:07:18 can it be actually implemented with regular NFS? 16:07:20 So why don't you share some info on plans 16:07:35 You had mentioned that you do have plans for this in the future correct? 16:07:36 well, I notice that zadara driver doesn't support snapshot either. 16:07:40 qcow files rather than raw would allow it 16:07:40 Hi all. 16:07:45 regarding snapshots, we have no formal plans so far 16:07:58 oh, I misunderstood 16:07:59 of course we're thinking about implementing it 16:08:03 We added a XenAPINFS , and snapshot support is waiting for review. Although it is a bit generous. 16:08:34 but right now there is no such thing as a release date for this feature 16:08:34 Making deep copies instead of real snapshots ... 16:08:41 hi! 16:09:14 matelakat: interesting 16:09:18 JM1: Is there any sort of hack that comes to mind to make it at least appear to have this? 16:09:30 Similar to what matelakat has done? 16:09:37 jgriffith: well, just as was said, we can do a full copy 16:09:53 JM1: That would alleviate my issue 16:10:06 ditto 16:10:08 +1 16:10:08 For me it's more of continuity 16:10:10 I just thought that this could be more disappointing to users than knowing that they don't have real snapshots 16:10:21 I think we have to keep in mind that snapshot implementations will also need to support future functionality, like restore? 16:10:31 JM1: So I'd rather document to users what you're doing and why it's not a good idea 16:10:36 but give them the opportunity 16:10:49 Remember, most users are doing things automated 16:11:00 jgriffith: +1 16:11:09 They don't want/need to check "if this backend taht" 16:11:11 that 16:11:12 jgriffith: the ability to do fast/efficient snapshots seems like the kind of thing a driver should be able to advertise in it capabilities 16:11:12 etc etc 16:11:36 bswartz: +1 16:11:38 bswartz +1 16:11:38 bswartz: good idea. :) 16:11:40 The continuity and adherence to the API is what matters to me, not the implementation 16:11:51 bswartz: +1 16:12:07 so every backend has to implement every feature in the API? 16:12:11 jgriffith: well at the moment, they will need to also setup our SOFS before using it in cinder 16:12:13 but I agree that some sort of dumb/slow snapshot implementation is better than none 16:12:28 and AFAIK, there can be only one cinder driver at a time 16:12:29 Maybe there should be a generic full copy implementation and those who have efficient snapshots will advertise the capability? 16:12:30 guitarzan: well, I hate to use absolute terms 16:12:44 well, there is obviously some line being drawn here and it is unclear what it is 16:12:47 JM1: Multiple driver support is a hot new feature 16:13:02 DuncanT: do you mean it's being implemented? 16:13:05 JM1: But I consier things like create/snapshot/delete/create-from-snap core 16:13:37 guitarzan: ^^ that was meant for you 16:13:40 JM1: there can be multiple cinder driver for a cinder cluster. 16:13:42 JM1: I believe it works, within limitations of works (requires both drivers to implement the get_stats function) 16:13:54 winston-d: ah good to know! 16:14:00 jgriffith: yeah, I got that :) 16:14:08 guitarzan: you object? 16:14:17 I just think it's an interesting stance to take 16:14:31 people wanting cinder support for a particular backend probably already know about that backend 16:14:36 winston-d: so I suppose there is a system of rules to determine what driver will host what volume? 16:14:50 JM1: yeah, the scheduler 16:14:52 JM1: volume_types 16:14:53 guitarzan: that's fair 16:15:04 guitarzan: let me rephrase it a bit 16:15:07 is there a way for the user to select which backend to use? volume types? 16:15:32 JM1: scheduler decides which back-end to serve the request based on volume type. 16:15:34 The initial review is sure to get a -1 from me, but if the vendor simply can't offer the capability then exceptions can and likely will be made 16:15:39 Can we move multi-driver talk to later in the meetingm, or this is going to get confusing 16:15:49 DuncanT: +1 16:15:51 hemna__: a user shouldn't need to choose a backend, as long as the backend has the capabilities they need 16:16:10 avishay: +1 16:16:17 jgriffith: that sounds reasonable 16:16:22 As DuncanT let's table the back-end discussion for the moment 16:16:56 guitarzan: that's more along the lines of what I had in mind and is why JM1 is here talking to us this morning :) 16:17:02 :) 16:17:13 so JM1.... 16:17:18 I'd also be tempted to start being more harsh on new features being added without a reasonable number of drivers having the functionality added at the same time 16:17:20 so from this I understand that even copying would be an acceptable fallback to support snapshotting 16:17:37 that's something we can do 16:17:43 DuncanT: I would agree, but I don't know that we haven't adhered to that already 16:17:51 JM1: on the manager level? 16:17:55 I've avoided putting SolidFire stuff in core for that very reason 16:18:04 jgriffith: As the project matures, we can bring in tighter rules 16:18:08 matelakat: manager? 16:18:24 the one that calls the driver. 16:18:32 I would guess he means at the driver level 16:18:51 ah, you mean as a fallback for drivers like ours that don't have the feature? 16:18:54 the driver can certainly just do a copy 16:19:02 ahh, I misunderstood if that's the case :) 16:19:08 yes, so on the driver level, you only need a copy 16:19:09 yes I was thinking inside our driver 16:19:55 inside the driver it's easy to copy files, rename, whatever 16:20:00 If it can be made generic enough that NFS can use it to, even better 16:20:09 indeed 16:20:21 DuncanT: +1 16:20:59 the performance cost is high, so that won't be useful with all workloads 16:21:14 but I gather that for some it will be still better than nothing 16:21:23 I think that is true, yes 16:21:56 JM1: well snapshots inparticular have been notriously poor performance items in OpenStack 16:22:01 Maybe we could come up with a name for those snapshots, so it reflects that they are not real snapshots. 16:22:02 LVM snaps suck! 16:22:16 matelakat: we have clones now 16:22:40 LVM snaps are better than full copies 16:22:44 jgriffith: thanks. 16:23:08 ok I gotta jam to work... 16:23:17 bswartz: I didn't ask for what's worse, I just said they suck 16:23:19 matelakat: the snapshot vs backup discussion is another one :) 16:23:19 I don't see how LVM snaps can be worse than a full copy of a 1TB volume 16:23:28 jgriffith: fair enough 16:23:35 They're only worse when you try to use them for something 16:23:42 and they kill perf on your original LVM 16:23:54 it may take longer to make a full copy, but the full copy will perform better 16:23:58 I'm not talking perf of the action itself, but we're really loosing focus here me thinks :) 16:24:00 I think we're drifting again... 16:24:12 guitarzan: +1 16:24:18 Ok... back in here 16:24:34 JM1: Do you have a strong objection to faking snapshot support via a deep copy? 16:24:44 jgriffith: not at all 16:24:45 Or does anybody else on the team have a strong objection? 16:24:51 but I will have to think about details 16:24:55 the api doesn't care if a snapshot is a snapshot or a copy 16:25:03 guitarzan: correct 16:25:12 and come back to you folks to ask implementation questions 16:25:15 So TBH this is exactly what I did in the SF driver anyway 16:25:17 so until the "backup" discussion starts again, we shouldn't worry about implementation 16:25:27 we didn't have the concepts of snapshots so I just clone 16:25:39 eg. can we expect the VM to pause I/O during the snaphost? 16:25:44 that's pretty much what we do in the 3PAR driver as well 16:25:59 JM1: typically no, you can't assume that 16:26:14 JM1: we don't do anything to enforce that 16:26:20 that could be a problem eh... 16:26:29 --force option allows snashot of an attached volume, but it is a 'here be dragons' option 16:26:29 jgriffith: ok so for a copy we need a mechanism to force a pause 16:26:39 you have to be disconnected to snap unless you --force right? 16:26:50 DuncanT: +1 16:26:54 DuncanT: oh, so that means usually we snapshot only unattached volumes? 16:27:00 normal snapshot without force requires source volume to be unattached 16:27:08 JM1: That is my belief 16:27:13 ok, so cp should work 16:27:22 yes 16:27:28 JM1: We don't currently support snap of live volumes, though that will be fixed some time 16:27:42 ah ok 16:27:53 yall speak for yourselves :) 16:27:58 I thought it already worked like that on more capable drivers 16:28:03 to be fair, the api also doesn't prevent you from immediately reattaching :) 16:28:38 again, "here be dragons" 16:28:40 Yeah, I'm thinking of proposing a state machine with an explicit 'snapshotting' state to cover that, but that is a state machine discussion 16:28:50 DuncanT: +1 16:28:59 DuncanT: +1 for states in Cinder!! 16:29:17 Back to JM1 are we good here or is there more we need to work out? 16:29:17 DuncanT: +1 16:29:20 so we've given JM1 a lot of work or stuff to think about 16:29:25 jgriffith: I think we're good 16:29:37 Excellent... anybody else? 16:29:44 I will see how to do a simple implementation with simlpe file copies 16:29:49 DuncanT: +1 for state machine 16:29:51 and resubmit patches 16:30:30 and thank you all for your input 16:30:54 Ok... so what else have we got 16:31:00 being primarily a dev I'm not as familiar with real use cases as I'd like to 16:31:08 Since we started the multi-backend topic shall we go there? 16:31:13 I'm going to time limit it though :) 16:31:27 * jgriffith has learned that topic can take up an entire meeting easily 16:31:56 what is left to implement to support it? 16:32:30 BRB 16:32:39 #topic multi-backend support 16:32:49 So there are option here 16:33:12 hub_cap: is possibly looking at picking up the work rnirmal was doing 16:33:44 So we get back to the question of leaving it to a filter sched option or moving to a more effecient model :) 16:34:10 which option has the best chance of making it in G3 ? 16:34:22 hemna__: They've all got potential IMO 16:34:28 So let me put it this way 16:34:38 The filter schedule is a feature that's in, done 16:34:52 So what we're talking about is an additional option 16:35:03 The ability to have multiple back-ends on a single Cinder node 16:35:18 There are two ways to go about that right now (IMO) 16:35:33 jgriffith: do you mean mutiple processes on one host? or 1 process? 16:35:37 1. The patch that nirmal proposed that provides some intelligence in icking bakc-ends 16:35:53 bswartz: that's two different approaches 16:35:55 2. Running multiple c-vol services on a single Cinder node 16:36:02 oh 16:36:33 (2) seems like it would create some new problems 16:36:33 I favour option 2 from a keeping-the-code-simple POV 16:36:44 DuncanT: +100 16:36:48 lol 16:37:02 bswartz: which are? 16:37:11 well what would go in the cinder.conf file? 16:37:17 the options for every backend? 16:37:25 would different backends get different conf files? 16:37:34 I think you'd have the same problem with 1 or n managers 16:37:35 bswartz: not sure why that's unique between the options? 16:37:53 with n managers you could build a conf for each 16:37:58 BTW: https://review.openstack.org/#/c/11192/ 16:38:03 Different conf files or named sections in a single file... neither is overly complicated 16:38:05 well option 1 forces us to solve that problem explicitly 16:38:07 For a point of reference on what Option 1 looks like 16:38:19 This also illustrates the conf files, not so bad 16:38:25 ty 16:38:33 opt 2 multiple c-vol services on one single cinder node seems good. you can use filter scheduler to choose node 16:38:50 I mean choose c-vol service 16:38:56 hey guys sorry im in IRL meeting. just saw my name 16:39:08 im in favor of yall telling me what to do :D 16:39:09 back 16:40:13 okay I'm in favor of (2) as well 16:40:29 bring on the extra PIDs 16:40:32 could you attach volumes from 2 different cinder services in the same VM? 16:40:40 JM1: Yes 16:40:42 sure 16:40:52 so multiple c-vol services, would that be like nova-api, spawning multiple listeners in teh same pid? 16:40:52 JM1: of course 16:41:16 on different ports 16:41:24 No need 16:41:35 hub_cap: no, just managers 16:41:39 They don't listen on a port, only on rabbit 16:41:42 hub_cap: that's a big different, nova-api workers are listening on _SAME_ port 16:42:02 winston-d: im talking about osapi, metadata and ec2 api in the same launcher 16:42:10 +1 for option #2 16:42:12 hub_cap: while c-vol services listen on AMQP 16:42:27 sure... amqp vs port... not much diff... a unique _thing_ 16:42:37 but ure right it was my bad for saying port :D 16:42:43 hub_cap: I think you're on the same page 16:42:52 i dont think operators will be happy w/ us having 10 pids for 10 backends :) 16:43:03 hub_cap: ah well. they should have their own pid if my memory serves me right. 16:43:07 I think in previous converstations we likened it to swift in a box or something along those lines 16:43:13 i have a vm running let me c 16:43:19 hub_cap: why not? 16:43:24 ure right winston-d 16:43:31 10 pids for ten backends is better than one fault taking out all your backends! 16:43:34 hub_cap: I think 10 would be uncommon -- I see more like 2 or 3 in reality 16:43:44 as long as they are started/stopped by a single binscript, like nova does im down for it 16:43:45 bswartz: hehe 16:43:56 bswartz: ya i know :D 16:44:02 hub_cap: that's what I'm thinking 16:44:11 or maybe 10 instances of 2-3 drivers? 16:44:21 We do introduce a point of failure issue here which sucks though 16:44:24 I expect people to do such crazy things for eg. performance 16:44:35 so yall ok w/ me modeling it like the present nova-api, but w/ the subsitiution of amqp to pids, they create different pids but use a single binscript to start/sotp 16:44:52 ampq to ports... sry 16:44:55 trying to listen IRL too 16:45:01 JM1: 10 instances of 1 driver, on a single node, gives almost no performance improvement 16:45:27 DuncanT: I meant several instances of a driver but each with different tuning 16:45:46 JM1: we're not quite that sophisticated (yet) :) 16:45:46 10 instances of 1 driver on a single node also increases your failure domain if that one node dies 16:45:55 Ok... 16:46:25 bswartz: so that's my concern, however that's the price of admission for this whole concept no matter how you implement it 16:46:41 If HA is a concern, do an HA cinder install or use mltiple back-ends 16:46:50 That being said... back to the topic at hand 16:47:03 or use a redundant storage cluster (hint hint) 16:47:18 JM1: The problem is if the cinder node goes your toast 16:47:21 yeah I'm just saying that in practice I don't expect a huge of PIDs on the same host -- people will spread them out to a reasonable degree 16:47:30 There's now way to get to that redundant storage cluster :) 16:47:50 jgriffith: this will only affect creation of new instances, no? 16:48:03 (but I'm getting off topic) 16:48:03 JM1: creation of new volumes, not instances, but yes 16:48:15 JM1: volumes, create, attach, delete andy api call 16:48:50 back on track here... Sounds like concensus for the multiple processes on a single Cinder node? 16:48:51 You can put multiple instances of API behind a load ballancer and most things work, there are some gaps still 16:48:52 ok so it sounds like we have consensus? 16:49:01 jgriffith: +1 16:49:02 the only thing unaffected by cinder going down is your data access 16:49:04 jgriffith: +2 16:49:04 single config file iirc? right? 16:49:05 hub_cap: hehe 16:49:20 hub_cap: yep, that's what I'm thinking 16:49:23 * hub_cap hopes i didnt open a new can of worms 16:49:26 cool 16:49:31 jgriffith: +3 16:49:36 +4 16:49:39 w/ specific [GROUP NAMES] 16:49:40 I just want to make sure up front... is anybody going to come back and say "why are we doing this?" 16:49:44 speak now 16:50:10 jgriffith: so each process has its own conf file? 16:50:13 hub_cap: GROUP NAMES seems like an interesting approach 16:50:37 jgriffith: I think the HA issue is going to come up. 16:50:56 itll be nice to do CONFIG.backend_one.ip 16:51:19 HA needs to be solved regardless IMO 16:51:35 Orthogonal issues, no? 16:51:52 HA I think is a summit topic, though we might be able to solve some of it before hand 16:52:03 avishay: I would agree, but thingee I think is pointing out something folks will definitely bring up 16:52:35 Agreed 16:52:37 But my answer there is then don't do it.. use multi-nodes, types and the fitlter scheduler 16:53:00 DuncanT: avishay and yes, HA is something we really need but it's a seperate issue IMO 16:53:07 types + filter sched will work here too, right? 16:53:31 avishay: solves the more general problem and use case yes 16:53:56 avishay: but some don't want an independent cinder node for every back-end in their DC 16:54:08 jgriffith: well, i thought the only benefit of multiple c-vol on single node is to save physical machines since a lot of c-vols are just proxy, very lightweight workloads 16:54:49 winston-d: isn't that what I said :) 16:54:56 jgriffith: i agree, but now the scheduler won't choose the backend? 16:54:57 from scheduler point of view, it doesn't even have to know those c-vols are on the same physical server or not. 16:54:58 winston-d: or are we typing in unison :) 16:55:13 winston-d: Ahh.. good point 16:55:15 jgriffith: yup 16:55:32 Ok... off topic again 16:55:34 nevermind my comment 16:55:52 So the question is, everybody on board with hub_cap going down this path? 16:56:02 Seems good to me 16:56:06 me 2 16:56:10 avishay: it still does 16:56:21 me 2 16:56:24 s/path/rabbit hole/ 16:56:25 * jgriffith +1 has wanted it since Essex :) 16:56:35 +1 16:56:51 hub_cap: synonyms :) 16:56:52 it's fine with me 16:56:57 nirmal will be so sad :) 16:56:58 hub_cap: i'd love to help you test/review the patch 16:57:00 * guitarzan runs 16:57:05 sounds good 16:57:05 guitarzan: haha!! 16:57:09 guitarzan: speak up 16:57:13 guitarzan: :) 16:57:18 no, I think multiple managers is good 16:57:32 alright, awesome 16:57:37 Let's move forward then 16:57:42 hub_cap: You da man 16:57:46 winston-d: thank u sir 16:57:50 and jgriffith <3 16:57:50 hub_cap: ping any of us for help though of course 16:57:54 roger 16:58:07 Ok... almost out of time 16:58:14 #topic open discussion 16:58:53 I'm back after being sick since last week wednesday...reviews and what not coming...sorry guys 16:59:23 thingee: glad you're up and about, hope you're feeling better 16:59:29 submitted the generic copy volume<->image patch 16:59:42 we agreed on CONFIG.backend_one.ip type conf files, am i correct? 16:59:48 and DuncanT submitted the backup to swift patch which i hope to review soon 16:59:51 jgriffith: any updates on the get_volume_stats() regarding the drivers providing "None" for values that they can't obtain 17:00:01 Snapshot/volume deletion is the other thing I wanted to bring up 17:00:20 kmartin: Oh... I haven't talked to anyone about that yet I don't think 17:00:29 kmartin: could you elaborate ? 17:00:46 Specifically the fact that currently, if you snapshot a volume then delete the volume, the snapshot is unusable if you need the provider loc/auth of the original volume 17:01:14 DuncanT: sorry... see I told you I'd forget :( 17:01:25 We're out of time :( 17:01:32 BuT 17:01:35 EVERYBODY 17:01:43 winston-d: on 3par we are not able to provide a few of the values that are required 17:01:49 Please take a look at the backup patch DuncanT submitted 17:01:59 :-) 17:02:01 and jump over to #openstack-cinder to finish this conversation 17:02:09 We need to give the Xen folks the channel now 17:02:16 #endmeeting