14:03:09 <rosmaita> #startmeeting image import sync
14:03:10 <openstack> Meeting started Tue Oct 11 14:03:09 2016 UTC and is due to finish in 60 minutes.  The chair is rosmaita. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:03:11 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:03:14 <openstack> The meeting name has been set to 'image_import_sync'
14:03:27 <rosmaita> #link https://etherpad.openstack.org/p/glance-ocata-image-import-sync
14:04:23 <rosmaita> ok, so we were going to refresh ourselves about taskflow
14:04:32 <rosmaita> (which i personally did not have time to do)
14:05:09 <Jokke_> So the base_imprt flow is good starting point
14:05:21 <rosmaita> ok
14:05:41 <Jokke_> due to it's architecture we cannot use it as is, but it's pretty good skeleton for what we want to do
14:06:02 <croelandt> sound optimistic
14:06:06 <rosmaita> for the people following along:
14:06:10 <rosmaita> #link http://git.openstack.org/cgit/openstack/glance/tree/glance/async/flows/base_import.py
14:07:01 <rosmaita> i'm looking at all of flavio's notes about glance_store
14:07:26 <rosmaita> that's making me think a design session on glance_store refactoring is a good idea for barcelona
14:07:29 <Jokke_> I was just gonna say that most of the lines are Flavio ranting, but we just ignore that part for now ;)
14:07:39 <Jokke_> ;)
14:07:40 * croelandt loves the "def revert(self): pass"
14:07:58 <rosmaita> yea, flavio does not write boring code
14:08:25 <Jokke_> rosmaita: that's very nice and polite of you :P
14:08:37 <rosmaita> i try
14:09:49 <Jokke_> so my plan for this week is to steal the bits from the base_import we can utilize and make api_image_import flow there
14:10:20 <croelandt> I see
14:10:27 <croelandt> what endpoints would that cover exactly?
14:10:31 <Jokke_> and get the configuration option to use for the scratchpad ... hopefully I'll be able to import image from it to the back end store
14:10:44 <Jokke_> croelandt: none
14:11:14 <Jokke_> that would be just the taskflow to be used under the hood when the image import is done
14:11:34 <Jokke_> with the new workflow
14:12:09 <croelandt> yeah
14:12:24 <croelandt> So the only issue with the inside-out approach is that I have no idea what we're trying to code
14:12:39 <rosmaita> so we'll need: user -> scratchpad, (validation while in scratchpad), scratchpad -> backend
14:12:45 <Jokke_> croelandt: so the execution of that taskflow would be triggered by the last call of the image import process
14:13:15 <rosmaita> Jokke_ is talkind about scratchpad -> backend
14:13:21 <croelandt> ok
14:14:14 <rosmaita> croelandt: it might be worth thinking about how the validation should happen
14:14:41 <Jokke_> and that taskflow would be then pluggable to pull the image data either from the FS (code ready in the base_import) or swift (needs to be done) or external uri (already implemented in the base_import) and push it to the glance_store (already implemented in the base_import)
14:14:59 <croelandt> rosmaita: isn't that part of the spec?
14:15:20 <rosmaita> croelandt: yes, but i think the spec says what more than how
14:17:06 <rosmaita> so the swift stuff doesn't need to happen in the MVP
14:17:28 <Jokke_> rosmaita: but the design needs to be able to facilitate it
14:17:32 <croelandt> rosmaita: can't see the word "scratchpad" in either the spec nor the current code :p
14:17:37 <rosmaita> Jokke_: very true
14:17:42 <rosmaita> croelandt: sorry, "staging area"
14:17:47 <Jokke_> just like the external uri (that just happens to exist already)
14:17:49 <rosmaita> (formerly, "the bikeshed")
14:17:52 <croelandt> oh right
14:18:07 <croelandt> not a native speaker, so I sometimes have trouble jumping between the terms :)
14:18:20 <Jokke_> sorry, yeah, I'll keep using bikeshed again so we are on the same page ;)
14:18:20 <rosmaita> croelandt: sorry about that
14:18:43 <rosmaita> croelandt: in the first few drafts, we called it "the bikeshed" because we couldn't agree on what to call it
14:18:50 <Jokke_> croelandt: same here, so I constantly use wrong ones. Sorry
14:19:22 <Jokke_> rosmaita: actually we called it bikeshed from the very beginning as we _knew_ we wouldn't be able to agree what to call it :P
14:19:43 <rosmaita> well, i think i got overruled on having PUT v2/bikeshed in the api
14:20:00 <rosmaita> (that's why we call it the "staging area" -- PUT v2/stage)
14:20:02 <Jokke_> unfortunately yes. Perhaps we should revive that
14:20:30 <rosmaita> it woudl be good for openstack to have a bikeshedding api
14:20:36 <rosmaita> would keep things more organized
14:23:28 <Jokke_> so yes, when we get that taskflow working, we can then implement the staging and the import endpoints and we effectively have MVP ... then the swift part and the discovery and tons of testing/hardening
14:23:32 <Jokke_> == profit
14:23:59 <rosmaita> so, if Jokke_ is working on the bikeshed -> backend bit, is it too early for croelandt to start on the user -> bikeshed part?
14:24:49 <Jokke_> I think that'd be fine if you croelandt want to take that on
14:25:10 <rosmaita> croelandt: what parts interest you most?
14:25:12 <croelandt> not sure exactly where to start, but why not
14:25:44 <Jokke_> croelandt: we can sync on that offline
14:25:49 <croelandt> rosmaita: I honestly do not really see where we are going, especially because of the inside/out methodology (which is definitely right one in terms of reviews, though)
14:26:15 <croelandt> I could give the user -> bikeshed a try
14:26:18 <rosmaita> Jokke_: can you use help on the bikeshed -> backend part?
14:26:29 <croelandt> if Jokke_ does the bikeshed -> store part, then we have the full stack \o/
14:26:51 <rosmaita> yeah, we would basically have image upload re-implemented!
14:26:53 <Jokke_> ok, I revisit my work plan that much that I try to make flowchart of the macro level
14:27:12 <Jokke_> maybe that helps out to understand what we need there and what bits we are working
14:28:18 <rosmaita> i think Jokke_ 's idea right now is that we start with the user's bits already in the bikeshed, and not worry about how they got there, and write the flow to get them into the backend
14:28:32 <Jokke_> correct
14:28:40 <rosmaita> so for collaboration,
14:28:56 <croelandt> yep
14:29:03 <rosmaita> croelandt, you could either work on the same with Erno (if that would make sense)
14:29:19 <rosmaita> or work on another part that already assumes the bits are in the bikeshed
14:29:28 <rosmaita> (that woudl be the validation stuff)
14:29:37 <Jokke_> so as long as I know the path to the image file on the local node I can just make the workflow, drop the image to that path, use the old _deprecated_ tasks api *sigh* to trigger the flow and see profit
14:29:39 <rosmaita> not sure if it's too early for that, though
14:29:57 <croelandt> validation might be fun
14:30:04 <rosmaita> (i meant about validation stuff, not Jokke_ 's workflow)
14:30:09 <croelandt> rosmaita: is there anything in particular you'd like to work on?
14:30:14 * abashmak pipes up
14:30:20 <rosmaita> hello alex
14:30:30 <abashmak> I have bandwidth to help
14:30:46 <Jokke_> there's actually good question about the validation
14:30:50 <rosmaita> croelandt: i want to work on the discovery part, i need to revise the schema
14:31:13 <Jokke_> do we want that just being tasks on the import flow or separate taskflow that then triggers the import flow
14:31:41 <rosmaita> Jokke_: not sure i understand your question
14:32:10 <rosmaita> abashmak: have you had a chance to work with taskflow yet?
14:32:27 <abashmak> sadly no
14:32:33 <rosmaita> that's ok
14:32:59 <rosmaita> my suggestion would be to look at taskflow and the stuff in the glance/async directory to see how it's being used
14:32:59 <croelandt> rosmaita: oh I have a patch for that
14:33:09 <rosmaita> croelandt: yes, i know
14:33:38 <rosmaita> croelandt: i don't want to take over your patch, but i need to revise the spec to make sure we've captured all the stuff that came up in the defcore discussions
14:33:46 <croelandt> rosmaita: feel free to ping me about that
14:34:24 <rosmaita> croelandt: i do have a question about whether our current json-schema code will be able to handle the proposed schema
14:34:26 <Jokke_> sorry, had to dial in to the next meeting
14:34:33 <rosmaita> Jokke_: ok
14:34:48 <Jokke_> so as the validation is precondition to the import
14:36:09 <Jokke_> we have two options 1) we have validation taskflow that gets triggered by the api call and when it finishes it triggers the import taskflow or 2) the validation is bunch of tasks as part of the import taskflow and will be one integrated taskflow triggered by the api call
14:37:20 <rosmaita> the key thing is that we'll want operators to be able to insert custom validation code
14:37:37 <rosmaita> i'm not sure if either of those is more conducive to that?
14:37:52 <Jokke_> yes and that's why I'd prefer having separate taskflow for it
14:37:53 <rosmaita> maybe (1) is because the validation part is more isolated?
14:38:09 <rosmaita> ok, so we are thinking the same thing
14:38:16 <Jokke_> it would keep the import code away from the people who want to implement their own validations
14:38:54 <rosmaita> yeah, that would be good
14:39:27 <Jokke_> so basically we could document "Write taskflow, set it to config being the used validation flow, make sure it calls api_image_import flow when it's finished
14:39:41 <rosmaita> i like that
14:40:01 <rosmaita> it keeps the database interaction stuff away from the operators
14:40:24 <rosmaita> we handle all the stuff that modifies the image record
14:40:38 * croelandt has a headache now
14:40:42 <rosmaita> we are way over time
14:40:52 <Jokke_> no worries
14:40:52 <abashmak> another possibility is for the custom code to plug into a wrapper, that way it doesn't have to call the api_image_import flow, the wrapper would do that
14:41:06 <Jokke_> croelandt: I'll do the flowchart and walk you through it
14:41:33 <rosmaita> abashmak: that's a good idea, the less we leave to operators, the more friendly the import will be
14:42:34 <Jokke_> abashmak: fell free to look into how to do that wrapper with the glance async/ taskflow code :P
14:42:36 <croelandt> ok
14:42:41 <croelandt> might be nice to sum this up in the etherpad
14:45:10 <rosmaita> ok, got some notes in the etherpad
14:46:01 <rosmaita> ok what's next?
14:46:13 <rosmaita> i guess we have enough for a week?
14:46:36 <Jokke_> yeah, maybe more
14:47:25 <rosmaita> ok, let's plan to meet again next week
14:47:46 <rosmaita> and we can find each other here if we need to
14:48:03 <rosmaita> thanks, and sorry about this going way over 20 min
14:48:11 <rosmaita> #endmeeting image import sync