14:00:34 <markwash> #startmeeting glance
14:00:35 <openstack> Meeting started Thu May  8 14:00:34 2014 UTC and is due to finish in 60 minutes.  The chair is markwash. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:36 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:00:38 <openstack> The meeting name has been set to 'glance'
14:00:52 <markwash> o/ all
14:01:00 <ameade> 0/
14:01:05 <mclaren> howdy
14:02:32 <arnaud> o/
14:03:16 <markwash> I've posted my proposed agenda at the top of https://etherpad.openstack.org/p/glance-team-meeting-agenda
14:03:49 <markwash> I'll go ahead and roll through my few announcements as people continue to come in
14:03:52 <markwash> #topic announcements
14:04:04 <rosmaita> o/
14:04:27 <markwash> Fei Long won't be making it to the summit unfortunately. So I pulled his sessions. Flavio has volunteered to lead his signed-images session.
14:05:33 <ativelkov> o/
14:05:33 <markwash> hmm, I guess that's my only announcement
14:05:49 <markwash> Anybody else have notices or announcements for the rest of us?
14:05:56 <mclaren> o/
14:06:19 <markwash> everybody shout at once, if so
14:06:29 <mclaren> I can do a shameless plug for a late summit slot
14:06:29 <zhiyan> markwash: so for the last session, i will have more time to discuss my ideas, am i right?
14:07:03 <markwash> zhiyan: yeah, that's my thought, just give you the full session slot
14:07:24 <markwash> zhiyan: is that okay with you?
14:07:31 <zhiyan> markwash: thanks, tbh, i'd like to try to get a basic agreement on "indexing" feature
14:07:58 <markwash> mclaren: yeah tell us about the cinder session
14:08:03 <mclaren> ok
14:08:08 <mclaren> this session http://junodesignsummit.sched.org/event/f9073c8f8e10731c299fab98eb67ba59#.U2tGv-Z0y1s
14:08:10 <nikhil___> o/
14:08:13 <zhiyan> 1. common functional api 2. poke fs, 3. indexing implementation
14:08:23 <mclaren> is down as a cinder slot but it should be of interest to glance folks
14:08:48 <mclaren> Its the last slot on the Friday
14:09:10 <markwash> I am sad I will miss it
14:09:20 <mclaren> (and of interest to keystone/swift folks too I think)
14:09:46 <mclaren> we're sad too markwash
14:09:52 <markwash> okay all of that sounds good
14:09:56 <markwash> #topic checkin on summit
14:10:02 <markwash> I guess we already started this checkin
14:10:18 <markwash> does anybody have any last minute questions or issues, or is there something you need me to do for you?
14:10:47 <ativelkov> I've updated the artifacts' session etherpad, just wanted to check if it is ok
14:11:08 <ativelkov> #link https://etherpad.openstack.org/p/juno-hot-artifacts-repository-finalize-design
14:11:47 <arnaud> markwash: did you get a chance to talk to randall?
14:11:51 <zhiyan> oh, seems one session slot is a little short ...
14:12:02 <markwash> arnaud: eek no I did not
14:12:25 <markwash> I've been a release blocker locally for the past week. so I've been very heads-down unfortunately
14:12:35 <arnaud> ativelkov: we might want to double check that he is ok sharing the session before the session
14:12:36 <arnaud> :)
14:12:46 <markwash> haha no let's surprise him!
14:13:00 <ativelkov> :)
14:13:12 <markwash> I'll send an email now or immediately after this meeting
14:13:37 <arnaud> ok sounds good thanks
14:15:16 <markwash> okay
14:15:22 <markwash> let's talk artifacts
14:15:26 <markwash> #topic artifacts design
14:15:36 <markwash> I hope folks have had more of a chance to review the docs (I did!)
14:16:29 <markwash> ativelkov: any notes for us before we start jumping into feedback at random?
14:16:52 <ativelkov> Not now, I think ) Waiting for feedback
14:17:18 <zhiyan> i did but tbh, besides my comments, i still have some question around the details.
14:18:32 <markwash> my feedback is centered around the pain points I feel we suffer with images
14:18:49 <zhiyan> ativelkov: btw, may i know does the api design part ready now?
14:19:13 <markwash> mostly I have seen "status" being more of a hindrance (UX and bugs) than a helpful feedback mechanism
14:19:34 <ativelkov> zhiyan: still in progress. Will try to have something at the summit
14:19:53 <zhiyan> ok, got it. ativelkov thanks
14:19:59 <markwash> so personally I would prefer we drop status altogether
14:20:09 <markwash> also, membership for sharing was never very good
14:20:16 <markwash> I prefer the swift / s3 model of sharing
14:20:22 <markwash> i.e. you share a bucket of things
14:20:33 <ativelkov> markwash: but how will we be able to differentiate the lifecycle state then?
14:20:50 <markwash> ativelkov: part of getting rid of status is making the lifecycle basically a binary state
14:20:57 <markwash> i.e. {exists, does-not-exist}
14:21:48 <ativelkov> This is fine when you import the artifact, but what about uploading the blobs one-by-one, creating dependencies etc?
14:22:23 <markwash> ativelkov: its a good question. is it possible to upload them in a dependencies-first order?
14:23:10 <ativelkov> Yes, it should be
14:23:42 <ativelkov> So you want the artifact creation to be an atomic operation?
14:24:14 <markwash> I think so
14:24:29 <rosmaita> markwash: what about inactivating artifacts?
14:24:38 <markwash> rosmaita: okay there is that one use case
14:24:58 <rosmaita> markwash: also pending delete
14:25:03 <markwash> rosmaita: but if we didn't have status, could we just have a system property "deactivated: <bool>" ?
14:25:25 <markwash> rosmaita: -1 pending delete
14:25:44 * markwash tries to be less heavy handed
14:25:51 <ativelkov> I don't know how can we make uploading of multiple blobs to be atomic unless we do an archive upload (i.e. the import operation)
14:25:52 <rosmaita> markwash: will have to think, don't want a proliferation of boolean properties for all cases
14:26:02 <markwash> let's talk about other ways to accomplish the pending_delete use case
14:26:40 <rosmaita> i'm just wondering whether if we ditch status, we'll wind up recreating it by a bunch of disparate properties
14:26:51 <ativelkov> Well, we can delete the artifact record from the database at once, but create a rabbit message (or whatever it is in Oslo messaging) to schedule a background deletion
14:27:14 <markwash> rosmaita: yeah that would be worse
14:27:15 <ativelkov> so, while the artifact is being deleted there is no record in DB already
14:27:29 <zhiyan> can we leverage existing scurbber to do pending-delete?
14:27:42 <markwash> so pending-delete is about recovery right?
14:27:54 <rosmaita> i was just thinking of it as a status
14:27:58 <rosmaita> off the top of my head
14:28:03 <zhiyan> one reason for recovery, but more for performance i think
14:28:20 <rosmaita> but you have the other direction, like ativelkov says, uploading stuff
14:28:31 <zhiyan> rosmaita: +1
14:28:35 <rosmaita> you might want the glance records first in queued status and then upload the data
14:28:39 <ativelkov> pending-delete is just about a background worker doing the deletion
14:28:39 <rosmaita> just an idea
14:29:03 <markwash> if its just about doing deletion asynchronously then I think it doesn't need to touch the api
14:29:29 <markwash> I think if we want it to touch the api, we need to take a more shadow-table-like approach
14:29:49 <markwash> b/c IMO deleted objects that are waiting to be expunged have a different schema than regular objects
14:29:53 <rosmaita> i am confusing things, i was thinking of pending delete as a user-exposed state indicating that you could still undelete
14:30:14 <markwash> rosmaita: I think that's another use case that we should address
14:30:27 <markwash> rosmaita: I just want to see it separate from the main artifact listing
14:30:42 <rosmaita> ok
14:31:25 <markwash> what I value very highly is this: all artifacts you normally list are *usable* right in that moment
14:31:36 <ativelkov> What is the problem with having the state, btw?
14:31:59 <markwash> if you have another use case (like: I want to recover artifacts I deleted by mistake) you don't find them by doing the regular list operation
14:32:02 <rosmaita> markwash: +1, but we could address that by default image-list filtering
14:32:21 <ativelkov> I just expected the regular list operation to return only READY artifacts
14:32:26 <ativelkov> by default, I mean
14:32:37 <rosmaita> ativelkov: +1
14:32:44 <zhiyan> rosmaita: agreed. and we should only allow end user recover "pending_delete" artifacts
14:33:03 <markwash> I have a bad feeling about default filter values
14:33:25 <ativelkov> what's the problem with them?
14:33:25 <markwash> I can come up with concrete cases where they fall down
14:33:44 <markwash> at the moment I can only speak about it a bit abstractly
14:33:55 <markwash> the problem with them is the overlap of defaults
14:34:10 <rosmaita> i think changing status to {exist, !exist} needs in-person discussion at the summit
14:34:22 <markwash> namely, when you have one filter flag that is set explicitly, you start to want to have different defaults for other  fields as well
14:34:25 <ativelkov> I've put it to the agenda
14:34:34 <markwash> take is_public for example
14:34:54 <markwash> the default is actually "is_public=True || owner == me || is_shared_with_me"
14:35:03 <markwash> this is an incredibly complex default to express in http filter parameters
14:35:21 <markwash> and now suppose we add in "status == pending_delete"
14:35:31 <markwash> do you want to see shared images that are pending delete? public ones?
14:36:04 <ativelkov> (just an idea) may be we have a different list calls for different use cases?
14:36:15 <jokke_> +1
14:36:37 <markwash> ativelkov: I like that idea. actually I think its very similar to what I'm advocating
14:36:41 <ativelkov> say "list my artifacts drafts" (some default set), "list artifacts I may use" (another set), "list artifacts I own" (third set)
14:36:54 <zhiyan> maybe we can just def a default filter value for each field ?  humm
14:37:23 <rosmaita> i don't understand the problem, is it just an http query string complexity issue?
14:37:32 <rosmaita> i mean, you have to decide what the default will be anyway
14:37:45 <rosmaita> e.g., we ignore architecture
14:37:47 <markwash> rosmaita: filters work great when the default is "not filtered on this parameter"
14:38:03 <rosmaita> gotcha
14:38:32 <zhiyan> maybe glanceclient can help this
14:38:48 <markwash> the principle we're advocating when we say "use filters + defaults" is that we have some field of use cases, and we will cover it by providing some eigenvectors the user can leverage
14:38:49 <rosmaita> glanceclient itself needs help!
14:38:57 <arnaud> markwash: don't we want to try to be as much as possible consistent with the way other openstack projects deal with listing things? from my pov, having a different list operation per filter is a nogo
14:39:06 <zhiyan> hehe, it's true
14:39:16 <rosmaita> arnaud: excellent point
14:39:50 <markwash> arnaud: I think we can frame this consistently with other projects
14:40:06 <rosmaita> zhiyan: but seriously, i worry about hiding too much in the client
14:40:32 <markwash> in rest the way you have a different list operation is by having a different endpoint right?
14:40:40 <ativelkov> well, the client may just provide aliases for different defaults
14:41:08 <markwash> so for pending delete, we have a different endpoint that represents the list of artifacts that are going to be deleted
14:41:49 <ativelkov> well, in rest different endpoints usually mean different resource types
14:42:21 <jokke_> I also don't like the idea that we need to make really complex design decisions just to stay consistent with rest of the projects ... it's almost like saying that there needs to be only one api for all projects and the functionality needs to be hidden within it no matter what.
14:42:32 <markwash> yes. but again, shadow tables have different schemas. . type(deleted_X) is different from type(X)
14:43:29 <arnaud> jokke_: my point is not about the internal design but more about the api (what the user will see/use/is used to)
14:43:53 <markwash> arnaud: I guess I'd just argue that we shoudl be consistent with the other most similar resources
14:44:00 <markwash> I don't think images/artifacts and servers are very similar
14:44:07 <markwash> images/artifacts are more like swift objects in a way
14:44:20 <arnaud> but
14:44:28 <markwash> also, re consistency, does any other project really address pending deletion / recovery?
14:44:34 <arnaud> a filter is the same concept for a server and an image
14:44:52 <arnaud> I mean I understand your point
14:45:02 <ativelkov> Another possible approach is not to have the widest possible defaults in the API
14:45:03 <markwash> I want us to still have filters I guess
14:45:06 <jokke_> arnaud: I know, but if it makes the system complex/inconvenient to develop _and_ provides crappy user experience (FE. need to deal with over complex filters, flooding list results etc.) is it worth of it?
14:45:15 <markwash> I just mean there are some use cases where filters are not the right approach
14:45:35 <markwash> and I think pending deletion is one where it is not the right approach
14:46:05 <jokke_> markwash: I second you on that
14:46:16 <markwash> ativelkov: yeah I think widest-possible defaults is the right approach for any filter
14:46:36 <markwash> i.e. if I do GET /collection it should always return everything (modulo pagination)
14:46:50 <markwash> if I do GET /collection?filter=foo it is only a restriction of that set
14:47:30 <markwash> if we constrain the concept of filters in this way, I foresee no problems
14:47:53 <markwash> but it does not solve the pending deletion use case. so we need a different mechanism for that
14:48:14 * markwash is being too loud
14:48:19 <rosmaita> or we eliminate pending delete
14:48:37 <arnaud> +1 markwash I think having a different approach for pending delete is fine/good
14:48:40 <ativelkov> hm
14:48:47 <jokke_> I don't think people will be too happy about that
14:48:58 <ativelkov> let's say /artifacts return all of them, including the deleted ones
14:48:59 <zhiyan> recover is a worth thing tbh
14:49:18 <jokke_> (eliminating pending delete I mean ^^^)
14:49:28 <zhiyan> and on the same time, pending_delete is a good thing for performance
14:49:31 <markwash> it seems like user driven recovery is useful enough to find a spot in our api
14:49:56 <ativelkov> /artifacts?state=pending_delete will return only the pending ones
14:50:29 <rosmaita> i think there  is also a use case for displaying deleted image/artifact records
14:50:36 <ativelkov> and the client will have a method list_active_artifacts() which will eventually call /artifacts?state=ready&owner=...&
14:50:44 <markwash> rosmaita: +1, you can't recover from them always if you can't discover them
14:50:55 <zhiyan> agreed
14:51:06 <rosmaita> but the deleted items could overwhelm the existing ones
14:51:13 <rosmaita> if /GET includes deleted
14:51:17 <markwash> ativelkov: this is a consistent proposal. may I offer an alternative?
14:51:26 <ativelkov> markwash: sure
14:51:43 <markwash> GET /artifacts returns all immediately useable artifacts (i.e. if its an image, you can boot from it)
14:51:55 <markwash> GET /deleted_artifacts shows the artifacts that are deleted but not yet expunged
14:52:23 <markwash> deleted_artifacts have additional system properties such as "deleted_at" and "expires_at"
14:52:51 <markwash> and there is an operation " POST /deleted_artifacts?recover" or somesuch that allows user-driven recovery of deleted artifacts
14:53:14 <rosmaita> markwash: i am thinking of a case where you don't want the image data any more, but you want to  be able to retrieve the image record because of valueable metadata that could be useful to know about
14:53:21 <ativelkov> "immediately useable" means "valid for the current tenant"?
14:53:54 <markwash> ativelkov: hmm, that's part of it
14:54:04 <markwash> there might be other restrictions implied by "useable"
14:54:06 <ativelkov> Yes, that's what I mean.
14:54:43 <ativelkov> And then we may have GET /artifact_drafs to return the artifacts which are not completed (published) yet
14:54:59 <ativelkov> For multi-blob upload etc
14:55:00 <arnaud> rosmaita: have you ever seen that? really? I would be interested to have a concrete example
14:55:51 <markwash> ativelkov: in tasks / import we accomplish that by GET /tasks
14:55:52 <jokke_> how about making /deleted/ instead, FE. GET /deleted/{artifacts, images} returning the deleted ones and you can filter that with peding or what so ever?
14:55:59 <rosmaita> arnaud: if have packages listed in the metadata, may need to go back and seee what was working at that time
14:56:11 <markwash> however, that assumes that the publication process is not interactive
14:56:26 * markwash reads scrollback
14:56:53 <markwash> jokke_: that seems fine
14:56:54 <jokke_> same with the draft ... having /draft/artifacts path rather than /draft_artifacts would keep consistent no matter what we call the actual item (it being artifact, image, template, etc.)
14:57:22 <rosmaita> arnaud: also, for UI uses, may have an instance, you know the image it was created from, may be pulling data from the image record for display purposes
14:57:24 <zhiyan> hmm..i *think* this could be considered with "functional api", maybe we can do this: GET /artifact just return "useable" ones (def usable later), and use POST /artifact/action to get artifact items with different status, and giving the filters in the body with json format
14:57:26 <mclaren> or /pending_deleted?
14:57:48 <markwash> mclaren: I think with /deleted we basically don't need pending deleted anymore
14:58:03 <arnaud> ok I see rosmaita thanks
14:58:09 <mclaren> to me deleted means gone, or are we introducing a 'really deleted'/expunged?
14:58:09 <rosmaita> zhiyan: is there some general info about "functional api" i could look at prior to summit?
14:58:38 <markwash> mclaren: that's what I was thinking. . I suppose it is a bit smelly
14:59:12 <mclaren> or /delete_queue
14:59:18 <markwash> mclaren: but we don't really introduce it, it's just what happens when an artifact leaves the /deleted/artifacts list
14:59:25 <zhiyan> rosmaita: yes, i did. https://etherpad.openstack.org/p/glance-adding-functional-operations-to-api
14:59:30 <markwash> s/it/expunged/
14:59:43 <markwash> eek we're almost out of time!
15:00:09 <jokke_> mclaren & markwash: I think rosmaita made good point with the metadata of deleted which actual payload is already gone
15:01:08 <markwash> jokke_: rosmaita: the use case does seem reasonable. its possible that nothing ever leaves deleted on its own, but that the "binary blobs" do disappear such that recovery becomes impossible
15:01:11 <markwash> okay we're over
15:01:18 <markwash> thanks everybody
15:01:24 <jokke_> thnx
15:01:27 <arnaud> thks
15:01:28 <markwash> I look forward to seeing as many of you as possible at the summit
15:01:32 <ativelkov> We'll continue on the summit, thanks
15:01:36 <markwash> thanks for humoring me!
15:01:39 <markwash> #endmeeting