20:01:51 #startmeeting transfer-service 20:01:52 Meeting started Mon May 6 20:01:51 2013 UTC. The chair is jbresnah. Information about MeetBot at http://wiki.debian.org/MeetBot. 20:01:53 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 20:01:55 The meeting name has been set to 'transfer_service' 20:02:14 ameade is here, anyone else ready? 20:02:20 o/ 20:02:45 iccha, nikhil, rosmaita? 20:02:54 ack 20:02:55 hey jbresnah 20:03:01 excelent 20:03:28 ok 20:03:34 i emailed out the agenda 20:03:53 The first thing is a quick introduction to the idea 20:04:07 it is largely documented elsewhere 20:04:19 https://blueprints.launchpad.net/glance/+spec/image-transfer-service 20:04:38 and there are many links there 20:05:02 The main idea is to have a service that handles data transfer as a first class problem 20:05:23 in the current state of OS trasnfers is a sort of second class operation of storage 20:05:39 there is not end to end management 20:05:43 this would look to solve that 20:05:53 http://tropicaldevel.files.wordpress.com/2013/04/the_wild_west.png 20:06:01 versus 20:06:06 http://tropicaldevel.files.wordpress.com/2013/04/tamed_west.png 20:06:13 sort of show the idea 20:06:29 are people generally familiar with the links in the blue print? 20:07:02 i've read through them 20:07:08 cool 20:07:17 and we had a meeting at the summit on the topic 20:07:35 iccha, ameade, nikhil, and rosmaita were all there 20:07:51 jbresnah: i like your diagram 20:08:00 on use case i want to state really quick came up from the earlier meeting about cloning 20:08:06 read through most of your use cases, looks pretty solid 20:08:19 hoping everyone else adds stuff too 20:08:25 in that we outlined the need for the a glance to glance transfer of an image with its metadata 20:08:26 jbresnah: +1 20:08:28 to close to other regions 20:08:42 i see this servicing fitting in there like this: 20:08:58 an async worker would call out to it to handle the transfer from storage system to storage system 20:09:10 instead of having glance try to handle that work load itself 20:09:33 once that worker was notified by the transfer service of completion it would continue to its next job of updating metadata etc 20:09:47 does that make sense as a use case? 20:10:36 yeah makes sense 20:10:45 how would the notification happen? 20:11:04 right, i am not sure about that just yet 20:11:12 what i have outlined right now is poll based 20:11:13 jbresnah: ameade guess in that particular case notification is handed over to glance? 20:11:20 something like that 20:11:26 ah ok 20:11:33 which goes into the next agenda point 20:11:39 feedback 20:11:42 https://blueprints.launchpad.net/glance/+spec/async-glance-workers ? 20:11:44 any thoughts, specific or general? 20:11:55 jbresnah: do you feel the need for managing transfers 20:12:05 just an extension to what ameade was saying 20:12:17 rosmaita: and I were discussing a bit on the async workers 20:12:23 what do you mean by manage? 20:12:39 and it kinda makes sense to have a queue based system for that 20:12:52 i think the trasnfer should be managed by a service that makes sure it properly completes and does not over/under consume its provisioned resources 20:12:59 if you'r polling then we need a managed state of the transfer 20:13:08 ah, ok 20:13:12 so from glance 20:13:14 something like qonos 20:13:19 glance would be a client to this service 20:13:24 https://github.com/rackspace-titan/qonos 20:13:32 an async worker could poll 20:13:50 or there could be some contact point back (like a tempurl) for completion notification 20:14:28 callbacks would be nice but nothing really has that tech in place 20:14:34 i would prefer that there is an PI boundry between this service and glance 20:14:51 *A*PI 20:15:05 i would not like them to share a messaging queue, for example 20:15:08 ah 20:15:22 tho i am open to discuss that 20:15:24 yes we want it to be indepedent from glance 20:15:39 iccha: I agree 20:15:48 jbresnah: i think that would be good to keep this as independent as possible 20:15:55 ok cool 20:16:04 like APIs 20:16:30 so on that point... should this effort be part of the glance project? in the summit unconference we decided that it should be its own new project 20:16:35 and hope for incubation 20:16:39 any new thoughts there? 20:16:42 so, how would the transfer service affect how clients work with openstack 20:16:42 markwash: any thoughts? 20:16:45 +1 for new 20:16:58 like, clients that want to upload data to glance or swift 20:17:02 markwash: which client? 20:17:13 markwash: well, i think backward compat will be there 20:17:31 sure, but with the new approach, how would it work? 20:17:32 markwash: so the traditional means would still happen 20:17:38 right, so i have a use case for that 20:17:44 getting it... 20:18:08 https://etherpad.openstack.org/transfer_workflow 20:18:17 basic download sort of covers that 20:18:24 the idea would be that a user would form a fileurl 20:18:34 and find a transfer service that can deal with such a url 20:18:47 (don't want to interrupt, i have a thought about where this project should live, will share later) 20:18:58 and then contact the store that galnce is using to get a url, and find a transfer service for it 20:19:03 so a 4 tuple is needed 20:19:13 source url, source service, dest url, dest service 20:19:25 source service can == dest service 20:19:49 in which case it ultimately looks like the service is doing a managed 2 party up/download 20:19:56 does that make sense? 20:20:10 rosmaita: please tell! 20:20:42 markwash: does that sound too heavy weight? 20:20:50 well, this is from our qonos experience ... 20:21:11 oslo is not appropriate because you want a service not a lib 20:21:18 it sounds kinda heavy, I'm wondering about "killer" use cases 20:21:26 but an incubator project is going to be a pain to manage 20:21:34 jbresnah: we prolly won't need source url for upload and dst for download? (wanna make sure we'r on the same page) 20:21:44 if you could find a nice PTL, you could incubate the project in, say, i don't know, glance 20:21:58 do the engineering so it can easily be detached 20:22:03 but build it in glance 20:22:10 I'd have to go to the TC about that probably 20:22:11 that way it will get into the community earlier 20:22:17 I don't think I have the power to redefine the scope of glance 20:22:39 rosmaita: so devel under glance with the intent to break away? 20:22:55 rosmaita: that was what was discuss for keystone and quotas IIRC 20:23:04 markwash: that would be ok, there needs to be a place for small projects to incubate 20:23:15 markwash, nikhil: I am not forgetting about the killer use case thread btw 20:23:18 so maybe a new "project" project, i don't know 20:23:38 rosmaita: here is my take on that: pragmatically I am completely fine with that 20:23:47 but if john does glance-useful stuff to start with, it would make sense to start dev in glance 20:23:48 however, to me it shows that the incubation process might be broken 20:24:16 jbreshah: yes, depends where you want to put your efforts now, fix incubation process or get started on xfer service 20:24:27 rosmaita: I guess that is a good point. at first it is all glance useful stuff 20:24:34 heh 20:24:42 i think so, this is a markwash call, though 20:24:51 * jbresnah would prefer to not fix incubation ;-) 20:25:03 markwash: ok lets get back to killer use case 20:25:03 jbresnah: to me it sounds like whether we want to tie this closely to glance xfer or find storage xfer use cases in general 20:25:07 markwash: or first use case 20:25:14 what about that cloning issue? 20:25:21 it is sort of the perfect starting point 20:25:31 because it is by nature a 3rd party transfer 20:25:54 the client says to glance 'trasnfer this image from you, to another glance' 20:25:56 and then it leaves 20:26:20 for glance to do that right it needs to monitor the trasnfer, restart where needed, verify success etc etc 20:26:33 all these things are what this transfer service wants to do 20:26:35 *and* 20:26:40 in the case of the science community 20:26:52 that wants 1 upload and then a broadcast of the image to many clouds 20:27:06 you may even want to negotation protocol (bit torrent, some wide area, etc) 20:27:16 so that cloning case is really a great starting point for this 20:27:19 thoughts? 20:28:09 i know it is difficult to find places for smaller projects, but i am not a 100 20:28:18 % convinced that it should belong to glance 20:28:21 i'm not sure where glance is heading, though from a HPC perpective and the use case that you just linked it makes sense to abstract out the transfer based on the deployer needs 20:28:23 cloning case makes sense to me, esp if we are starting in glance 20:28:41 nikhil: well hpc is one example 20:28:46 jbresnah: esp when doing multi day longing transfers 20:28:47 in every cloning case there is a trasnfer 20:28:51 that should be managed properly 20:28:55 the generic service doesnt belong in glance but if we are only doing glance use cases at first then it makes sense 20:29:10 ameade: that is what i am pitching 20:29:14 how heavy would it end up being? 20:29:16 ameade: Well put 20:29:25 iccha: I am hoping to make it modular 20:29:37 so a first cut would have single set of plug ins 20:29:40 to address that use case 20:29:43 I'm just seeing a lot of alternatives to the transfer service for each "killer" use case I can come up with 20:29:56 markwash: ex? 20:30:26 so for example, efficiency by having the two endpoints talking directly to each other 20:30:34 that's what the expose-locations thing is about 20:30:55 sure... but i would argue that is the trasnfer service case where source service == dest service 20:31:07 in tat case it would effectively be a managed wget 20:31:09 or some such 20:31:27 where the client can walk away and check back later 20:31:36 ...doesnt have to have full privaledegs 20:31:44 ...doesnt have to baby sit/restart on failure 20:31:45 etc 20:32:16 effectively, the way to think of this transfer service is something that wraps a client to make sure it does the trasnfer right 20:32:21 I don't completely follow src service == dest service 20:32:25 the rest is an implementaion detail 20:32:48 i could say to a single transfer service 'download http://xxxx to file://yyyy' 20:33:03 and it would manage that transfer 20:33:26 does that make sense? 20:33:35 yeah 20:33:44 i see those as the first use cases 20:34:02 take the arch where nova-compute nodes mount a SAN for instance storage 20:34:12 and 1 machine is dedicated for transfering images to it 20:34:20 that machine could do what i said above 20:34:34 only it has a global pitcure so it can schedule them nicely 20:34:39 and make sure the restart on error 20:34:41 etc 20:34:42 so suppose I'm doing "transfer swift://region1/foo/bar swift://region2/foo/bar" 20:34:53 jbresnah: i think you could extend it a bit here and say 20:35:15 can we talk more about how this would be implemented within glance for the cloning use case? so that it's clear and we could maybe agree on it 20:35:17 I'm guessing in the ideal case I'd want the transfer to be happening directly between nodes on the rings of each swift deployment 20:35:22 if you want to transfer http:xxx to list[url1, url2, url3 ...] 20:35:28 ameade: i think that is a good idea 20:35:36 pushing on that use case might make the most sense for clarity 20:35:42 just think that if we do that we can get started too 20:35:51 markwash: probably yes 20:36:07 ameade: guess the dialogue is whether it should be in glance or not 20:36:08 ameade: good point 20:36:19 from what i see in markwash's concern 20:36:23 nikhil: true, and that is a practical use case at hand 20:36:33 markwash: so, if we talk about that use case 20:36:42 markwash: and how we would acco,mplish it within glance 20:36:56 i agrue that the glance code would look a lot like an image trasnfer service 20:37:18 (especially if you let me write it hehehe) 20:37:24 * markwash is unfortunately a bit distracted at the moment 20:37:37 yeah we have to implement it somehow, we would just do it with the fact it could eventually break out into the transfer service in the future 20:37:45 can we start this out as an apache-licensed (2.0) open source project on its own? 20:38:07 markwash: i am open to anything 20:38:10 and then look at optionally using it in glance? 20:38:22 markwash: but naother apporach is to start working on the cloning case, with this in mind 20:38:30 and see how much sense it makes to break it out 20:38:39 +1 cloning 20:38:46 cause really... if we work on this and the cloning case at the same time, there will be work overlap 20:38:47 +1 that work has to be done anyways 20:38:52 right 20:39:02 i think that is the most efficient way to spend time 20:39:18 it has the downside that ineviitably it will be more tightly married to glance then is idea 20:39:21 jbresnah: +1 to cloning 20:39:23 iccha: do you have thoughts there? 20:39:35 isn't this just an optimization to the cloning work? 20:39:48 i would say a generalization 20:39:50 its just a different way of saying "download image X" on the target service 20:39:57 with the potential for a lot of optimization 20:40:07 this thought guides the impl 20:40:09 we need a cloning data transfer management anyhu 20:40:11 markwash: i can see looking at it that way sure 20:40:12 if we are going the glance route, i would like to be as modular as possible.. a separate node, etc because currently glance nodes designed to be more lightweight etc.. 20:40:23 especially when coupled with the direct_url business 20:40:42 iccha: agreed 20:41:31 markwash: what you are sayibn gis true. any effective use case we could come up with could also be solved with a combination of cloning and direct_url exposure 20:41:33 i think anyway 20:41:59 perhaps I'm just being a bit dense (and I'm happy if people want to try to enlighten me as forcefully as necessary) 20:42:02 but... that would be a way to work around functionality instead of supporting it in a first class way, IMHO 20:42:09 but I feel a bit like this is trading a bird in the hand for a bird in the bush 20:42:16 wrt the cloning work 20:42:27 markwash: can you explain a bit more? 20:42:34 well lets say we did the cloning work seperate 20:42:47 then the first step for this service could be transforming that work into the service 20:42:53 bird in hand -> the blueprint and design approach we discussed in the previous meeting 20:42:59 for cloning 20:43:35 we have a pretty low level view the design for that, which means there's more certainty and more momentum 20:43:42 markwash: we need a service to handle data transfer anyway 20:43:56 nikhil: the current plan involves glance as that service 20:44:03 for the cloning use case 20:44:04 markwash: async workers too? 20:44:35 so maybe I need to think more about the "first class" approach to using this 20:44:47 markwash: so it is fair point to say that the cloning effort could be done faster and with less architectural high mindedness if we just do it 20:44:51 the transfer service is a bit different from cloning blueprint, it is solving a different problem 20:44:59 markwash: but it is also fair to say that there will be a ton of overlap in code 20:45:00 would you just say "transfer glance://region1/abcd glance://region2/" ? 20:45:15 iccha__: just abstraction in teh initial phase, no? 20:45:26 iccha: agreed. but the cloning work shares overlap of a subset of what it is solving 20:45:42 but I can also support the current cloning plan with just eventlet threads running on existing glance api nodes 20:45:54 iccha: IOW if the xfer service existed, the cloning work could be more easily done by just using it. 20:46:11 what was decided in cloning meeting? the stores transfer content to each other? 20:46:18 markwash: ummm 20:46:21 markwash: you can 20:46:33 markwash: but you will need to deal with failures 20:46:35 iccha__: that would just be an optimization, normal case is glance2glance 20:46:41 markwash: overlaoding the NIC of those api nodes 20:46:44 etc 20:47:12 markwash: you sure you wanna run multiple threads on glance api nodes for public API glance nodes? 20:47:13 jbresnah: I agree its not ideal, but if you're just testing or at a small scale, it works fine 20:47:24 nikhil: I definitely do *not* want to do that 20:47:30 markwash: if this were done inside of the cloning work, i think the results would be the same, only there would be more points of abstraction etc that would seem not needed to just the cloning work 20:47:31 but some folks with small scale might 20:47:37 and would enjoy not having to deploy new components 20:47:47 hmm 20:47:50 my concern is that this logic gets all over the inside of glance and is a nightmare to move towards this service 20:48:10 the service could share process space with api and still have good points of abstraction at first 20:48:12 ameade: its really easy though 20:48:28 markwash: until we add a million patches for error cases 20:48:29 markwash: guess the basic question is which is more flexibly for scale? scale up and scale down? 20:48:32 ameade: i would worry about that too 20:48:34 if you want the xfer service to do the data transfer, you just use it instead of a manual download and upload 20:49:10 if you want the xfer service to do both data and metadata, then you just work completely outside the clone code as we discussed it earlier 20:49:39 markwash: agree 20:49:45 markwash: i want to get a consise feel for your concern 20:49:57 nikhil: its not about scale up or down here. . the lightweight eventlet one would never scale up 20:50:09 nikhil: when you want to scale up, you would select a different async processing driver 20:50:13 markwash: you are worried that the code does not belong in glance as starting point. or you think cloning should not use it at all? 20:50:35 I think if this existed, it would make sense to do cloning using it 20:50:46 markwash: Ideally i would like to get consensus on having cloning be a good first use case 20:50:56 be it where the code initially lives or not 20:51:10 | 3) Consensus on a basic use case (fully documented and approved later) 20:51:19 from my optimistic set of goals 20:51:27 4) Consensus on part of Glance or a new project 20:51:39 +1, we have 10 min to know our next steps 20:51:43 can we get a vote on #3? 20:51:56 again, maybe I'm just being dense, but I want to understand the advantages more before we go all in on the xfer service as a blocker for cloning 20:52:14 oh, not a blocker 20:52:18 the other way around 20:52:44 i would like to use the cloning use case as a way to show the benefits of xfer 20:52:47 not block it 20:53:03 so then cloning has to come first, because I don't want to say "you can't have cloning unless you deploy a transfer service component" 20:53:08 so, perhaps a quick impl with less features could be done 20:53:26 markwash: i understand that concern 20:53:28 I guess I got here late 20:53:29 so we'r doing both then? 20:53:30 :( 20:53:42 sorry 20:53:45 small impl for cloning and xfer service 20:54:01 i see markwash's concerns about overloading glance with responsibility of data transfer. it is agreed that cloning can be enhanced by data transfer service, but i guess markwash would not like it to be dependent on it 20:54:02 because, i really don't want something to block cloning, please 20:54:03 markwash: so the worry is that if cloning is the main use case, then you worry its effort will be delayed? 20:54:23 * jbresnah nod iccha 20:54:30 jbresnah: what was the other potential starting point? 20:54:46 jbresnah: yes, because I don't know what saying "cloning == xfer service" does to our previous cloning plans 20:54:49 the nova compute case where nodes mount a SANs 20:55:01 i find it less compeling because cloning seems so perfect for me 20:55:04 but it still has legs 20:55:15 what about region-to-region swift transfers. . is that a possible first use case? 20:55:18 especialy when talking about a boot of 1000s of instances 20:55:23 markwash: +1 20:55:30 markwash: isn't that cloning? 20:55:33 jbresnah: i like markwash's use case 20:55:44 yes but store level 20:55:46 jbresnah: I don't think so? 20:55:51 and not blocking glance cloning 20:56:02 jbresnah: suppose I am running something other than swift as my store 20:56:12 swift to swift + import/export == cloning? 20:57:00 markwash: ok, i can see that 20:57:12 3 minutes 20:57:13 the way we talked earlier, I thought cloning was a glance to glance kind of thing 20:57:27 markwash: we may have to table this... 20:57:32 #3 and #4 20:57:36 * markwash is remembering quotes from requiem for a dream 20:57:43 ouch 20:57:47 i hate remebering that movie 20:57:54 i curl up into a fetal position and cry 20:57:57 me too 20:57:59 great movie, but ... wow 20:58:00 anyway 20:58:02 | 1) A list of concerns to be addressed (either at the end or in the future). 20:58:10 i think i can pull a bunch of those from the transcript 20:58:18 | 2) A list of parties interested in working on this (not a time commitment) 20:58:21 how about that part 20:58:28 what is the general interest level 20:58:33 jbresnah: what if the transfer service started off seperate as a way to replace the image upload handler code in nova? 20:58:42 coding, designing, reviewing, etc? 20:58:54 ameade: hmmm 20:59:03 ameade: that makes sense 20:59:08 jbresnah: markwash have we decided to keep it separate from glance or in it? 20:59:12 ameade: and download 20:59:17 something specific like that is good 20:59:21 jbresnah: yeah very easily 20:59:23 i recall we talked about that briefly once 20:59:33 nikhil: I'd like for it to start seperately from glance 20:59:41 jbresnah: yeah really early on 20:59:43 nod 20:59:58 ok 21:00:06 so it looks like, separate from glance 21:00:13 votes? 21:00:13 works for me 21:00:15 i like the idea of store to store transfer going first so it be used in image cloning if needed sooner 21:00:19 +1 21:00:23 * flaper87 will comment later 21:00:30 iccha, i like that 21:00:34 can we do #vote or something? 21:00:41 oh right, sorry 21:00:46 ummm how do i do that? 21:00:54 not sure 21:00:59 * markwash doesn't know either. . . 21:01:09 jbresnah: #startvote 21:01:21 #startvote outside of glance 21:01:22 Unable to parse vote topic and options. 21:01:31 #startvote own_project 21:01:32 Unable to parse vote topic and options. 21:01:34 jbresnah: #startvote Should we vote now? Yes, No, Maybe 21:01:42 #link http://ci.openstack.org/meetbot.html 21:01:47 :) 21:02:03 #vote Yes 21:02:11 #vote maybe 21:02:25 * markwash giggles 21:02:30 jbresnah: has to start the vote :D 21:02:30 heh 21:02:32 #vote yes 21:03:02 bold move 21:03:06 #startvote Should the transfer service be its own project? Yes, No, Maybe 21:03:07 Begin voting on: Should the transfer service be its own project? Valid vote options are Yes, No, Maybe. 21:03:08 Vote using '#vote OPTION'. Only your last vote counts. 21:03:19 #vote yes 21:03:27 #vote apathetic 21:03:28 ameade: apathetic is not a valid option. Valid options are Yes, No, Maybe. 21:03:36 lol 21:03:40 #vote yes 21:03:40 #vote apathetic 21:03:41 ameade: apathetic is not a valid option. Valid options are Yes, No, Maybe. 21:03:56 ameade maybe will be apathetic for now 21:04:01 what was the vote again? sorry, user error 21:04:02 #vote maybe 21:04:02 #vote ameade 21:04:02 nikhil: ameade is not a valid option. Valid options are Yes, No, Maybe. 21:04:03 #vote maybe 21:04:23 markwash: ? I am pretty sure you are a yes.... 21:04:46 we'r voting on Should the transfer service be its own project? 21:04:46 I accidentally left the room right when you said "startvote" 21:04:47 btw 21:04:51 #vote yes 21:04:51 markwash: ^^ 21:05:00 For next steps i will type up the two potential use cases and et a vote on them 21:05:17 jbresnah: you have to call #endvote now 21:05:22 #action jbresnah doc up cloning and simple nova-compute up/down use cases 21:05:26 #endvote 21:05:27 Voted on "Should the transfer service be its own project?" Results are 21:05:28 Maybe (2): ameade, jbresnah 21:05:29 Yes (3): nikhil, markwash, flaper87 21:05:39 where's iccha__ ? 21:05:55 I want to say, we can look at bringing it into glance after its been on its own for a while too 21:06:02 cool 21:06:07 as a way to unbreak the incubation process (if necessary and expedient) 21:06:08