20:03:01 <markwash> #startmeeting glance
20:03:01 <Guest25612> +1
20:03:01 <openstack> Meeting started Thu May 16 20:03:01 2013 UTC.  The chair is markwash. Information about MeetBot at http://wiki.debian.org/MeetBot.
20:03:02 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
20:03:03 <flaper87> yep
20:03:05 <flaper87> \o/
20:03:05 <openstack> The meeting name has been set to 'glance'
20:03:06 <iccha_> hey
20:03:07 <flaper87> \o/
20:03:08 <markwash> hi errbody
20:03:09 <flaper87> \o/
20:03:20 <flaper87> that's a body execption
20:03:28 <flaper87> exception*
20:03:29 <jbresnah> wave
20:03:36 <markwash> I was out for most of last week
20:03:53 <markwash> today, i don't have a really set agenda (not one that I recall, at least!)
20:04:17 <jbresnah> markwash: async workers is an outstanding topic
20:04:22 * flaper87 raises both hands \o/
20:04:29 <markwash> my main item is to target as many blueprints in https://blueprints.launchpad.net/glance/havana
20:04:30 <jbresnah> if there is time for a discussion that may be a good topic
20:04:33 <nikhil__> I may have a dependency level
20:04:33 <esheffield> that's one I'm particularly interested in
20:04:40 <nikhil__> and def asyn workers is most imp
20:04:51 <markwash> okay, cool
20:04:52 <esheffield> async workers that is
20:04:57 <markwash> first, does anybody have announcements?
20:05:16 <nikhil__> who's getting married?
20:05:23 <flaper87> any new kids ?
20:05:35 <markwash> I noticed the new xfer service work was started
20:05:37 <jbresnah> markwash: should better test suite bp be targeted for havana?
20:05:53 <jbresnah> flaper87: actually....
20:06:05 <jbresnah> flaper87: my wife is due with a new kid in < 4 weeks
20:06:12 <iccha_> thats awesome jbresnah
20:06:16 <jbresnah> so if i seem panicky lately....
20:06:31 <jbresnah> also i may disaapear for a few weeks at that point
20:06:34 <jbresnah> just fyi
20:06:44 <markwash> cool
20:06:56 <iccha_> so far here are the items i see mentioned 1. async workers - updates 2. transfer service 3. test suite
20:07:07 <flaper87> jbresnah: congrats ;)
20:07:11 <markwash> I guess lets quickly see if we can make any progress on blueprint targeting, and then discuss async workers?
20:07:17 <nikhil__> jbresnah: gud luk
20:07:21 <ameade> markwash: +1
20:07:24 <flaper87> markwash: +1
20:07:24 <iccha_> +1
20:07:26 <markwash> #topic blueprint targeting
20:07:28 <nikhil__> +1
20:07:35 <markwash> #link https://blueprints.launchpad.net/glance/havana
20:07:51 <markwash> everything there that is not targeted should be
20:08:13 <markwash> and some of the ones with targets might be wrong, considering h1 is not very far away
20:08:42 <nikhil__> markwash: cloning u mean?
20:08:56 <nikhil__> I heard a rumour that it's not in havana?
20:09:08 <iccha_> when is h1?
20:09:31 <markwash> May 30th
20:09:35 <markwash> 2 weeks from today
20:09:43 <rosmaita_> that is pretty soon
20:09:48 <jbresnah> yeah that is soon
20:09:57 <nikhil__> soon-=1
20:10:19 <markwash> nikhil__: I don't know what you mean about cloning
20:10:25 <markwash> its in the havana list, that's all I know
20:10:35 <nikhil__> markwash: ohk, just wanted to make sure
20:10:43 <iccha_> markwash: is bcwaldon working on multiple locations?
20:10:44 <markwash> looking over the list, it seems to me that we still need to sort out upload-download workflow
20:10:55 <markwash> but maybe that is blocked on our async workers discussion?
20:11:01 <markwash> iccha_: not that I've noticed
20:11:08 <jbresnah> markwash: i think that is right
20:11:20 <nikhil__> markwash: yes
20:11:21 <jbresnah> markwash: i think up/down is largely sorted out based on the fallout of async
20:11:27 <iccha_> markwash: it is a dependency but we can figure out the workflow wihtout it assuming async workers will work
20:12:13 <iccha_> do we want to schedule a time to talk about async workers in openstack-glance sometime? is there an intial layout?
20:12:30 <flaper87> I wrote down some ideas and considerations
20:12:38 <iccha_> i am sorry i missed prev 2 meetings not sure if it was already done
20:12:39 <flaper87> and I wanted to share that today if possible
20:12:46 <markwash> sounds good
20:12:49 <rosmaita_> please share!
20:12:53 <flaper87> but I'm ok with doing it in a separate meeting if time is not enough
20:13:09 <markwash> I guess we're not going to make a ton of progress with bp targeting without sorting out the async stuff
20:13:19 <markwash> so let's move on to that soon, but first
20:13:22 <flaper87> rosmaita_: that will cost a couple fo pop-tarts
20:13:35 <markwash> is anyone else interested in tackling multiple image locations?
20:13:45 <rosmaita_> dafuq?
20:13:49 <flaper87> markwash: if no one is going to work on that
20:13:51 <flaper87> I can take it
20:13:53 <jbresnah> markwash: i could do it
20:14:02 <flaper87> or he ^
20:14:04 <flaper87> :D
20:14:08 <jbresnah> flaper87: or we could tag team it...
20:14:14 <flaper87> jbresnah: +1
20:14:18 <flaper87> sounds like a plan
20:14:28 <jbresnah> markwash: i am pretty interested in seeing it get done, so if it is stalled out i could get on it
20:14:32 <markwash> cool, I'll assign jbresnah then
20:14:36 <rosmaita_> with your timezone separation, i think you can work on it continuously
20:14:39 <markwash> haha
20:14:45 <jbresnah> heh
20:14:48 <flaper87> LOL
20:14:49 <flaper87> 24h
20:14:51 <flaper87> :P
20:15:18 <jbresnah> markwash: should we make sure that is cool with bcwaldon first?
20:15:23 <markwash> meh
20:15:32 * jbresnah doesn't like to step on toes
20:15:34 <markwash> I'll talk to him about it tomorrow at some point
20:15:37 <jbresnah> ok
20:15:41 <markwash> don't worry in this case
20:15:56 <markwash> okay, anything else before we launch into async workers?
20:16:11 * flaper87 is worried about https://blueprints.launchpad.net/python-glanceclient/+spec/glance-client-v2
20:16:26 <iccha_> 11 am EST monday in #openstack-glance we will be discussing protected properties in anyone is interested in joining in
20:16:40 <flaper87> iccha_: awesome
20:16:45 <markwash> flaper87: ah good point
20:16:47 <iccha_> https://etherpad.openstack.org/public-glance-protected-props is link iwth some notes
20:16:57 <markwash> flaper87: once we get that, we can look at a major version bump in glanceclient as well
20:17:02 <markwash> which is good for dropping legacy
20:17:08 <flaper87> markwash: exactly
20:17:24 <flaper87> so, I see there hasn't been any movement there
20:17:35 <jbresnah> iccha: i will try to make that, a little bit early for me but possible
20:17:43 <markwash> honestly, I'm kind of out of touch with glance client, and I'd love if there could be a sort of glanceclient lieutenant to help usher us through the process
20:17:45 <markwash> and manage releases
20:17:58 <flaper87> I can help with that
20:18:02 <markwash> well, I'd like to be consulted about managing releases, of course
20:18:04 <markwash> :-)
20:18:38 <flaper87> sure, as for now, I'm worried about the API v2 not being fully supported
20:18:43 <flaper87> and people asking for it T_T
20:18:45 <flaper87> :P
20:18:52 <markwash> cool, flaper87 let's touch base offline to see how we can move forward
20:18:57 <flaper87> markwash: +1
20:19:01 <rosmaita_> i do know that v2 image sharing is in glance client
20:19:22 <iccha_> finally got merged after 2+ months :p
20:19:40 <flaper87> iccha_: :D
20:19:42 * markwash is sad about dropping the ball on that merge
20:20:01 <markwash> sorry to drag you through all those "restores"
20:20:36 <nikhil__> async workers pls?
20:20:40 <markwash> #topic async workers
20:20:50 * markwash passes the mic to flaper87
20:21:00 * flaper87 clears his voice
20:21:14 <flaper87> so, these are some of the thoughts I had related to async workers https://etherpad.openstack.org/glance-async-worker
20:21:40 <flaper87> I took more time to think about what could happen / should be changed if that happens
20:21:45 <flaper87> and how can that be implemented
20:22:27 <nikhil__> one comment
20:22:30 <nikhil__> inline
20:22:40 <jbresnah> i am not sure there should be 1 way to implement the async worker...
20:22:43 <markwash> looking at the two ideas for implementation (1: green threads in the api, 2: async workers across rpc), could we make 1 vs. 2 a configuration setting?
20:22:50 <flaper87> as shown in the pad, the 2 basic ideas are: 1) KISS and do it within glance-api 2) Use something more "complex" and keep it in a separate service (with scheduler and that kind of things)
20:23:00 <jbresnah> i would like to see a plugin interface
20:23:27 <jbresnah> to glance-api proper it may always look like #1
20:23:39 <jbresnah> but the plugin may implement with #2
20:23:41 <esheffield> to me it feels like this may be a wheel that's getting reinvented in several places in openstack
20:23:45 <markwash> jbresnah: +1 that makes sense to me a lot
20:23:48 <flaper87> jbresnah: markwash indeed, that could be addressed with the new oslo's messaging api
20:24:03 <flaper87> " For both ideas it would be nice to use the new messaging "
20:24:26 <flaper87> because it supports blocking executors and eventlet based executors that could be used through an in-memory queue
20:24:27 <jbresnah> i can see cases where messaging may not be needed
20:24:47 <flaper87> jbresnah: agreed
20:25:02 <nikhil__> can we discuss based on use cases
20:25:12 <nikhil__> this looks like a wild goose chase
20:25:12 <jbresnah> nikhil: good plan
20:25:14 <markwash> well, one big use case is devstack
20:26:09 <markwash> to me that's the driving use case for in-api green threads / eventlet
20:26:38 <iccha_> could u elaborate on the devstack use please markwash ?
20:26:49 <flaper87> markwash: ^
20:27:08 <markwash> well, whatever we want to do, we want it to work in super simple deployments like devstack
20:27:28 <markwash> and it would be nice to have those working without adding a generally unnecessary extra process to track
20:27:42 <iccha_> which makes the case for making it configurable stronger
20:27:47 <jbresnah> what is something that devstack would use async workers for?
20:28:00 <nikhil__> QE
20:28:06 <nikhil__> is what comes to mind
20:28:08 <markwash> import and export I guess
20:28:14 <markwash> QE?
20:28:15 <flaper87> markwash: devstack is already capable of configuring the oslo's rpc implementation
20:28:19 <jbresnah> umm, sorry
20:28:28 <flaper87> I think import / export is one of the main points
20:28:32 <nikhil__> do we really need a export?
20:28:48 <flaper87> followed by image introspection
20:28:51 <nikhil__> I feel like totally out of sync then
20:28:53 <jbresnah> yeah, i meant how would async be used as a use case
20:29:05 <nikhil__> jbresnah: yeah
20:29:19 <nikhil__> i mean : me too
20:29:20 <jbresnah> not what glance use cases do we want to not disrupt
20:29:28 <markwash> we're talking about different api calls that process something in the background
20:29:51 <flaper87> exactly
20:30:04 <markwash> so devstack would have the same use cases
20:30:09 <markwash> as any other deployment
20:30:19 <flaper87> I see that being useful in different scenarios (like those in the pad)
20:30:23 <jbresnah> nod
20:30:24 <markwash> just with the additional constraint of it being all in one box
20:30:35 <jbresnah> i just think it may be helpful to walk through how an example async worker would work
20:30:37 <nikhil__> I interpreted jbresnah's question as "is there a need to have it in devstack"?
20:30:43 <markwash> there is also the usecase where an existing deployment upgrades and doesn't want to change their deployment layout at first
20:30:59 <jbresnah> for example, an async worker that handles the upload
20:31:06 <jbresnah> or an async worker that handles image conversion
20:31:17 <iccha_> markwash: are u driving towards why we need both types of implementation and configurable?
20:31:26 <markwash> iccha_: yes I think so
20:31:48 <markwash> that probably wasn't a clear as I should have made it :-)
20:31:56 <markwash> s/a clear/as clear/
20:32:19 <nikhil__> agree
20:32:33 <markwash> devstack and existing deployments might want the new asynchronous operation features without changing their deployment layout (ie, not adding another process / moving part)
20:32:39 <nikhil__> however, for even a small production level deployment
20:32:46 <flaper87> I agree on the fact that both methods should be supported
20:32:59 <flaper87> we can vote :)
20:33:00 <nikhil__> do we think that threads on api nodes could handle just fine?
20:33:20 <jbresnah> i would think that both methods would be supported by way of what ever plugin was in use
20:33:34 <nikhil__> my concern is
20:33:43 <nikhil__> if we are waiting for both support
20:33:43 <jbresnah> i think glance would dictate a plugin interface
20:33:48 <flaper87> nikhil__: I bet they would, my point is, there's already some work going on on similar code in oslo, so, why should we write it again?
20:33:55 <jbresnah> but allow the plugins to implement that iface in wahtever way it felt best
20:34:00 <markwash> flaper87: ohh
20:34:04 <nikhil__> and if just async workers on api nodes do not work, then effort is futile to a great extent
20:34:22 <nikhil__> flaper87: gotcha
20:34:26 <iccha_> does doing a async worker separately and not as a thread in api , does that necessarily mean that it becomes heavier and complex?
20:34:44 <markwash> flaper87: from glance's perspective, the async processing should just be an object or group of objects
20:34:46 <flaper87> so, this is the current state in oslo's new messaging (RPC) api https://github.com/markmc/oslo-incubator/tree/messaging/openstack/common/messaging
20:35:01 <iccha_> could it be configurable to make the worker light weight with say one thread vs more complex with many threads or something like that
20:35:06 <markwash> if messaging supports making the choice of local or remote really easy, we should use that
20:35:25 <flaper87> markwash: I think it does, I'll work on a POC for that
20:35:37 <flaper87> #action flaper87 work on a POC for using oslo's messaging
20:35:47 <jbresnah> iccha: i would think that would be up to the worker
20:36:01 <markwash> what I'm saying is that it doesn't make sense for async processing to have a "messaging" interface, rather messaging should be an implementation detail of what's underneath the interface
20:36:11 <nikhil__> +1
20:36:14 <flaper87> as explained in the pad, I see some extra benefits moving towards that (aling glance with oslo and updating the notification stuff)
20:36:23 <jbresnah> iccha: i would like to see glance proper have a lightweight thread to manage/start/check status/etc of all it ... what mark said
20:36:27 <markwash> I think it makes sense that oslo-common messaging gets us where we want to be
20:37:05 <flaper87> markwash: In my head I see some higher-level API in glance that wraps oslo's calls
20:37:21 <flaper87> like, Async.process_this_stuff(...)
20:37:26 <markwash> I guess the POC is the right first step in any case, though
20:37:36 <flaper87> markwash: yep
20:37:43 <markwash> we can take a look at it, and offer an alternative if one seems necessary
20:37:52 <flaper87> sounds like a plan
20:37:58 <flaper87> I'm not saying that will work :P
20:38:03 <flaper87> if you don't see me on-line
20:38:09 <flaper87> I'm ok, I just ran away
20:38:11 <flaper87> :D
20:39:02 <markwash> do we need task tracking with this approach?
20:39:07 <flaper87> as a sidenote, I didn't work on any POC because I first wanted to agree with everyone on some usecases and perhaps a flow
20:39:21 * markwash thinks we should ignore his task tracking comment for now
20:39:30 <jbresnah> i am not sure what was decided
20:40:01 <flaper87> jbresnah: flaper87 work on a POC for using oslo's messaging
20:41:45 <nikhil__> does marconi have influence on that?
20:41:50 <nikhil__> flaper87: ^^
20:42:00 <flaper87> nikhil__: not sure I understand
20:42:10 <jbresnah> i dont understand why messaging is a first class design choice yet
20:42:14 <nikhil__> https://wiki.openstack.org/wiki/Marconi
20:42:17 <nikhil__> flaper87: ^^
20:42:28 <jbresnah> i would like to be free to have a custom plugin that does its work via fork
20:42:30 <jbresnah> or via a thread
20:42:32 <jbresnah> etc
20:42:36 <jbresnah> i dont like forcing messaging
20:42:45 <nikhil__> +1
20:42:50 <flaper87> jbresnah: well, that's basically what oslo's messaging does
20:42:55 <jbresnah> it seems like the wrong angle of attack
20:42:59 <iccha_> could we have like a pro cons list in the etherpad
20:43:06 * markwash agrees
20:43:06 <flaper87> but with a simple queue that can be either in memory or through MB
20:43:22 <jbresnah> flapper: that is about messaging tho
20:43:28 <flaper87> etherpad stopped working here
20:43:38 <markwash> flaper87: but, there are some cases where you wouldn't use messaging at all, like if you say used fabric to ssh to another node to initiate processing
20:43:42 <jbresnah> seems like the topic has the wrong focus
20:43:43 <nikhil__> last I saw this, we'r trying to do store to store trnasfers using a asycn worker
20:44:05 <markwash> flaper87: or if you wanted to kick off some processing on a node that is really "close" to the data, like a zerovm or docker worker
20:44:13 <jbresnah> right
20:44:32 <flaper87> mmhh
20:44:42 <jbresnah> i think the first bit of work is: define a async worker worlflow
20:44:58 <jbresnah> define the state machine from glance api perpective and client persoective
20:45:14 <jbresnah> then define an interface that async workers will implement
20:45:21 <flaper87> yeap, that's what I was expecting from this meeting
20:45:24 <jbresnah> after that implementation details will make more sense
20:45:45 <flaper87> so we could start working on some POC after that
20:46:14 <jbresnah> oh, and i think task #0 should be a requirements definition
20:46:23 <jbresnah> so we know what we are solving exactly
20:46:30 <jbresnah> i think parts of those things exist
20:46:35 <nikhil__> jbresnah: perfect!
20:46:36 <iccha_> +1
20:46:50 <iccha_> because i was starting to get a little lost here with all the different ideas
20:46:56 <nikhil__> :)
20:47:04 <iccha_> requirements, with a pro con list of basic approaches
20:47:11 <markwash> I'd like to see what "import" would look like in the v2 code, using async processing
20:47:12 <jbresnah> well...
20:47:21 <jbresnah> i am not sure approaches whould be married with requirements
20:47:27 <nikhil__> 4 mins till open discussion? (if someone has one - just a friendly reminder)
20:47:30 <jbresnah> requirements with a list of pros and cons for the requirement
20:47:49 <jbresnah> but it would be good to agree on the requirements up front without mixing in impl discussion
20:48:07 <jbresnah> markwash: i think that is the perfect use case to define around
20:48:23 <flaper87> +1
20:48:32 <iccha_> +1
20:48:34 <markwash> I might not be good at "requirements" though exactly, more like "here's what I want the interface to look like"
20:48:36 <rosmaita_> +1
20:49:00 <rosmaita_> (that was to jbreshah, not markwash!)
20:49:13 <markwash> #action markwash try to code up a stubbed version of import in v2, maybe it will help motivate requirements
20:49:32 <nikhil__> :D
20:49:51 <nikhil__> agree on the "import"
20:50:02 <nikhil__> that's the most imp requirement in my mind
20:50:08 * markwash is trying to "first do no harm" to this discussion, not sure its working
20:50:26 <jbresnah> markwash: i can help with requirements
20:50:41 <jbresnah> tho... i am not sure i am in the position to know what people want exactly...
20:51:50 <esheffield> I'd like to help with that as well
20:51:52 <nikhil__> someone pls create an action item for this
20:51:58 <nikhil__> requirements gathering
20:52:06 <nikhil__> that's software engineering 101
20:52:07 <flaper87> does etherpad work for you?
20:52:13 <iccha_> in terms of import export there are a few etherpads rosmaita_ created
20:52:14 <flaper87> it stopped working here
20:52:16 <jbresnah> etherpad works
20:52:23 <flaper87> mmh
20:52:27 * nikhil__ feels like sync failed
20:52:30 <rosmaita_> not here, stopped working long time ago
20:52:31 <esheffield> not worker for me either
20:52:31 <jbresnah> oh, sorry i thought you meant as a forum for requirement gathering
20:52:41 <iccha_> that is separate jbresnah :)
20:52:42 <jbresnah> etherpad has been falky for me this entire houtr
20:52:48 <flaper87> ok
20:53:01 <iccha_> the import export stuff is just for us to look at import export work flow and req gathering is for async worker
20:53:49 <jbresnah> agreed
20:54:19 <flaper87> so, what are the next steps on this topic?
20:54:36 <flaper87> markwash: will work on that stubbed version of import in v2
20:54:39 <flaper87> right?
20:54:41 <jbresnah> someone should take point on requirements gathering
20:54:44 <nikhil__> pros and cons list for me pls
20:55:01 <nikhil__> i can take requirements gathering
20:55:04 <markwash> flaper87: right
20:55:13 <flaper87> #action nikhil__ to work on requirements gathering
20:55:19 <nikhil__> cool
20:55:24 <jbresnah> nikhil: cool, keep in the loop please
20:55:33 <markwash> nikhil__: ditto what jbresnah said for me
20:55:42 <iccha_> flaper87: have u added ur etherpad to async workers bp?
20:55:47 <flaper87> nikhil__: and me :D
20:56:08 <flaper87> iccha_: nope, I created it today because I took those notes offline
20:56:10 <flaper87> I'll do
20:56:12 <nikhil__> think I'll just keep all posted on openstack-glance
20:56:28 <flaper87> nikhil__: please, have mercy with europeans :D
20:56:40 <flaper87> well, I should definitely sleep more but that's another story
20:56:45 <nikhil__> you mean the timezone rt?
20:56:49 <nikhil__> kk
20:56:50 <flaper87> nikhil__: yep :D
20:56:56 <markwash> any time left for open discussion?
20:57:24 <iccha_> 3 mins
20:57:26 <flaper87> 3 mins
20:57:31 <nikhil__> I dont have any agenda and need to go in 3 mins
20:57:34 <flaper87> markwash: what iccha_ said :D
20:57:38 <markwash> #topic open discussion
20:58:03 <markwash> I'd like to have a meeting time that is more friendly to the other hemisphere
20:58:08 <markwash> at least 1/month
20:58:16 <markwash> not sure if I already said that here, or in #openstack-glance
20:58:16 <rosmaita_> +1
20:58:34 <flaper87> this hemisphere was sleeping when you said that :P
20:58:44 <nikhil__> 15UTC works for me
20:58:53 <markwash> there are people who totally can't make it b/c of the time
20:59:07 <flaper87> markwash: agreed
20:59:13 <markwash> does 1/month sound okay?
20:59:25 <flaper87> +1
20:59:31 <markwash> we can do a trial for the first meeting in june, maybe
20:59:36 <markwash> or next week if the need is more urgent
20:59:46 <jbresnah> i am meeting time flexible
21:00:02 <markwash> jbresnah is on a beautiful beach right now anyway
21:00:07 <jbresnah> heh
21:00:09 <flaper87> haha
21:00:20 <jbresnah> its so hard to type with all this sand in my keyboard
21:00:23 <markwash> haha
21:01:17 <nikhil__> is the meeting over?
21:01:25 <markwash> I guess so
21:01:27 <nikhil__> i mean in terms of discission?
21:01:30 <markwash> anything else?
21:01:40 * markwash is a little distracted today, sorry
21:01:49 <rosmaita_> markwash: did not split the upload/download BP yet
21:01:58 <rosmaita_> started 2 discussion etherpads first
21:02:07 <markwash> links?
21:02:07 <rosmaita_> but, they appear to be unavailable at the moment
21:02:11 <markwash> ah
21:02:12 <rosmaita_> links in  BP
21:02:25 <markwash> good, I'll have a look
21:02:36 * nikhil__ says bye
21:02:52 <markwash> I guess that's all for us today, folks
21:02:55 <markwash> thanks!
21:02:55 <flaper87> cool
21:02:58 <flaper87> thanks
21:03:03 <markwash> #endmeeting