14:00:34 <markwash> #startmeeting glance 14:00:35 <openstack> Meeting started Thu May 8 14:00:34 2014 UTC and is due to finish in 60 minutes. The chair is markwash. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:00:36 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:00:38 <openstack> The meeting name has been set to 'glance' 14:00:52 <markwash> o/ all 14:01:00 <ameade> 0/ 14:01:05 <mclaren> howdy 14:02:32 <arnaud> o/ 14:03:16 <markwash> I've posted my proposed agenda at the top of https://etherpad.openstack.org/p/glance-team-meeting-agenda 14:03:49 <markwash> I'll go ahead and roll through my few announcements as people continue to come in 14:03:52 <markwash> #topic announcements 14:04:04 <rosmaita> o/ 14:04:27 <markwash> Fei Long won't be making it to the summit unfortunately. So I pulled his sessions. Flavio has volunteered to lead his signed-images session. 14:05:33 <ativelkov> o/ 14:05:33 <markwash> hmm, I guess that's my only announcement 14:05:49 <markwash> Anybody else have notices or announcements for the rest of us? 14:05:56 <mclaren> o/ 14:06:19 <markwash> everybody shout at once, if so 14:06:29 <mclaren> I can do a shameless plug for a late summit slot 14:06:29 <zhiyan> markwash: so for the last session, i will have more time to discuss my ideas, am i right? 14:07:03 <markwash> zhiyan: yeah, that's my thought, just give you the full session slot 14:07:24 <markwash> zhiyan: is that okay with you? 14:07:31 <zhiyan> markwash: thanks, tbh, i'd like to try to get a basic agreement on "indexing" feature 14:07:58 <markwash> mclaren: yeah tell us about the cinder session 14:08:03 <mclaren> ok 14:08:08 <mclaren> this session http://junodesignsummit.sched.org/event/f9073c8f8e10731c299fab98eb67ba59#.U2tGv-Z0y1s 14:08:10 <nikhil___> o/ 14:08:13 <zhiyan> 1. common functional api 2. poke fs, 3. indexing implementation 14:08:23 <mclaren> is down as a cinder slot but it should be of interest to glance folks 14:08:48 <mclaren> Its the last slot on the Friday 14:09:10 <markwash> I am sad I will miss it 14:09:20 <mclaren> (and of interest to keystone/swift folks too I think) 14:09:46 <mclaren> we're sad too markwash 14:09:52 <markwash> okay all of that sounds good 14:09:56 <markwash> #topic checkin on summit 14:10:02 <markwash> I guess we already started this checkin 14:10:18 <markwash> does anybody have any last minute questions or issues, or is there something you need me to do for you? 14:10:47 <ativelkov> I've updated the artifacts' session etherpad, just wanted to check if it is ok 14:11:08 <ativelkov> #link https://etherpad.openstack.org/p/juno-hot-artifacts-repository-finalize-design 14:11:47 <arnaud> markwash: did you get a chance to talk to randall? 14:11:51 <zhiyan> oh, seems one session slot is a little short ... 14:12:02 <markwash> arnaud: eek no I did not 14:12:25 <markwash> I've been a release blocker locally for the past week. so I've been very heads-down unfortunately 14:12:35 <arnaud> ativelkov: we might want to double check that he is ok sharing the session before the session 14:12:36 <arnaud> :) 14:12:46 <markwash> haha no let's surprise him! 14:13:00 <ativelkov> :) 14:13:12 <markwash> I'll send an email now or immediately after this meeting 14:13:37 <arnaud> ok sounds good thanks 14:15:16 <markwash> okay 14:15:22 <markwash> let's talk artifacts 14:15:26 <markwash> #topic artifacts design 14:15:36 <markwash> I hope folks have had more of a chance to review the docs (I did!) 14:16:29 <markwash> ativelkov: any notes for us before we start jumping into feedback at random? 14:16:52 <ativelkov> Not now, I think ) Waiting for feedback 14:17:18 <zhiyan> i did but tbh, besides my comments, i still have some question around the details. 14:18:32 <markwash> my feedback is centered around the pain points I feel we suffer with images 14:18:49 <zhiyan> ativelkov: btw, may i know does the api design part ready now? 14:19:13 <markwash> mostly I have seen "status" being more of a hindrance (UX and bugs) than a helpful feedback mechanism 14:19:34 <ativelkov> zhiyan: still in progress. Will try to have something at the summit 14:19:53 <zhiyan> ok, got it. ativelkov thanks 14:19:59 <markwash> so personally I would prefer we drop status altogether 14:20:09 <markwash> also, membership for sharing was never very good 14:20:16 <markwash> I prefer the swift / s3 model of sharing 14:20:22 <markwash> i.e. you share a bucket of things 14:20:33 <ativelkov> markwash: but how will we be able to differentiate the lifecycle state then? 14:20:50 <markwash> ativelkov: part of getting rid of status is making the lifecycle basically a binary state 14:20:57 <markwash> i.e. {exists, does-not-exist} 14:21:48 <ativelkov> This is fine when you import the artifact, but what about uploading the blobs one-by-one, creating dependencies etc? 14:22:23 <markwash> ativelkov: its a good question. is it possible to upload them in a dependencies-first order? 14:23:10 <ativelkov> Yes, it should be 14:23:42 <ativelkov> So you want the artifact creation to be an atomic operation? 14:24:14 <markwash> I think so 14:24:29 <rosmaita> markwash: what about inactivating artifacts? 14:24:38 <markwash> rosmaita: okay there is that one use case 14:24:58 <rosmaita> markwash: also pending delete 14:25:03 <markwash> rosmaita: but if we didn't have status, could we just have a system property "deactivated: <bool>" ? 14:25:25 <markwash> rosmaita: -1 pending delete 14:25:44 * markwash tries to be less heavy handed 14:25:51 <ativelkov> I don't know how can we make uploading of multiple blobs to be atomic unless we do an archive upload (i.e. the import operation) 14:25:52 <rosmaita> markwash: will have to think, don't want a proliferation of boolean properties for all cases 14:26:02 <markwash> let's talk about other ways to accomplish the pending_delete use case 14:26:40 <rosmaita> i'm just wondering whether if we ditch status, we'll wind up recreating it by a bunch of disparate properties 14:26:51 <ativelkov> Well, we can delete the artifact record from the database at once, but create a rabbit message (or whatever it is in Oslo messaging) to schedule a background deletion 14:27:14 <markwash> rosmaita: yeah that would be worse 14:27:15 <ativelkov> so, while the artifact is being deleted there is no record in DB already 14:27:29 <zhiyan> can we leverage existing scurbber to do pending-delete? 14:27:42 <markwash> so pending-delete is about recovery right? 14:27:54 <rosmaita> i was just thinking of it as a status 14:27:58 <rosmaita> off the top of my head 14:28:03 <zhiyan> one reason for recovery, but more for performance i think 14:28:20 <rosmaita> but you have the other direction, like ativelkov says, uploading stuff 14:28:31 <zhiyan> rosmaita: +1 14:28:35 <rosmaita> you might want the glance records first in queued status and then upload the data 14:28:39 <ativelkov> pending-delete is just about a background worker doing the deletion 14:28:39 <rosmaita> just an idea 14:29:03 <markwash> if its just about doing deletion asynchronously then I think it doesn't need to touch the api 14:29:29 <markwash> I think if we want it to touch the api, we need to take a more shadow-table-like approach 14:29:49 <markwash> b/c IMO deleted objects that are waiting to be expunged have a different schema than regular objects 14:29:53 <rosmaita> i am confusing things, i was thinking of pending delete as a user-exposed state indicating that you could still undelete 14:30:14 <markwash> rosmaita: I think that's another use case that we should address 14:30:27 <markwash> rosmaita: I just want to see it separate from the main artifact listing 14:30:42 <rosmaita> ok 14:31:25 <markwash> what I value very highly is this: all artifacts you normally list are *usable* right in that moment 14:31:36 <ativelkov> What is the problem with having the state, btw? 14:31:59 <markwash> if you have another use case (like: I want to recover artifacts I deleted by mistake) you don't find them by doing the regular list operation 14:32:02 <rosmaita> markwash: +1, but we could address that by default image-list filtering 14:32:21 <ativelkov> I just expected the regular list operation to return only READY artifacts 14:32:26 <ativelkov> by default, I mean 14:32:37 <rosmaita> ativelkov: +1 14:32:44 <zhiyan> rosmaita: agreed. and we should only allow end user recover "pending_delete" artifacts 14:33:03 <markwash> I have a bad feeling about default filter values 14:33:25 <ativelkov> what's the problem with them? 14:33:25 <markwash> I can come up with concrete cases where they fall down 14:33:44 <markwash> at the moment I can only speak about it a bit abstractly 14:33:55 <markwash> the problem with them is the overlap of defaults 14:34:10 <rosmaita> i think changing status to {exist, !exist} needs in-person discussion at the summit 14:34:22 <markwash> namely, when you have one filter flag that is set explicitly, you start to want to have different defaults for other fields as well 14:34:25 <ativelkov> I've put it to the agenda 14:34:34 <markwash> take is_public for example 14:34:54 <markwash> the default is actually "is_public=True || owner == me || is_shared_with_me" 14:35:03 <markwash> this is an incredibly complex default to express in http filter parameters 14:35:21 <markwash> and now suppose we add in "status == pending_delete" 14:35:31 <markwash> do you want to see shared images that are pending delete? public ones? 14:36:04 <ativelkov> (just an idea) may be we have a different list calls for different use cases? 14:36:15 <jokke_> +1 14:36:37 <markwash> ativelkov: I like that idea. actually I think its very similar to what I'm advocating 14:36:41 <ativelkov> say "list my artifacts drafts" (some default set), "list artifacts I may use" (another set), "list artifacts I own" (third set) 14:36:54 <zhiyan> maybe we can just def a default filter value for each field ? humm 14:37:23 <rosmaita> i don't understand the problem, is it just an http query string complexity issue? 14:37:32 <rosmaita> i mean, you have to decide what the default will be anyway 14:37:45 <rosmaita> e.g., we ignore architecture 14:37:47 <markwash> rosmaita: filters work great when the default is "not filtered on this parameter" 14:38:03 <rosmaita> gotcha 14:38:32 <zhiyan> maybe glanceclient can help this 14:38:48 <markwash> the principle we're advocating when we say "use filters + defaults" is that we have some field of use cases, and we will cover it by providing some eigenvectors the user can leverage 14:38:49 <rosmaita> glanceclient itself needs help! 14:38:57 <arnaud> markwash: don't we want to try to be as much as possible consistent with the way other openstack projects deal with listing things? from my pov, having a different list operation per filter is a nogo 14:39:06 <zhiyan> hehe, it's true 14:39:16 <rosmaita> arnaud: excellent point 14:39:50 <markwash> arnaud: I think we can frame this consistently with other projects 14:40:06 <rosmaita> zhiyan: but seriously, i worry about hiding too much in the client 14:40:32 <markwash> in rest the way you have a different list operation is by having a different endpoint right? 14:40:40 <ativelkov> well, the client may just provide aliases for different defaults 14:41:08 <markwash> so for pending delete, we have a different endpoint that represents the list of artifacts that are going to be deleted 14:41:49 <ativelkov> well, in rest different endpoints usually mean different resource types 14:42:21 <jokke_> I also don't like the idea that we need to make really complex design decisions just to stay consistent with rest of the projects ... it's almost like saying that there needs to be only one api for all projects and the functionality needs to be hidden within it no matter what. 14:42:32 <markwash> yes. but again, shadow tables have different schemas. . type(deleted_X) is different from type(X) 14:43:29 <arnaud> jokke_: my point is not about the internal design but more about the api (what the user will see/use/is used to) 14:43:53 <markwash> arnaud: I guess I'd just argue that we shoudl be consistent with the other most similar resources 14:44:00 <markwash> I don't think images/artifacts and servers are very similar 14:44:07 <markwash> images/artifacts are more like swift objects in a way 14:44:20 <arnaud> but 14:44:28 <markwash> also, re consistency, does any other project really address pending deletion / recovery? 14:44:34 <arnaud> a filter is the same concept for a server and an image 14:44:52 <arnaud> I mean I understand your point 14:45:02 <ativelkov> Another possible approach is not to have the widest possible defaults in the API 14:45:03 <markwash> I want us to still have filters I guess 14:45:06 <jokke_> arnaud: I know, but if it makes the system complex/inconvenient to develop _and_ provides crappy user experience (FE. need to deal with over complex filters, flooding list results etc.) is it worth of it? 14:45:15 <markwash> I just mean there are some use cases where filters are not the right approach 14:45:35 <markwash> and I think pending deletion is one where it is not the right approach 14:46:05 <jokke_> markwash: I second you on that 14:46:16 <markwash> ativelkov: yeah I think widest-possible defaults is the right approach for any filter 14:46:36 <markwash> i.e. if I do GET /collection it should always return everything (modulo pagination) 14:46:50 <markwash> if I do GET /collection?filter=foo it is only a restriction of that set 14:47:30 <markwash> if we constrain the concept of filters in this way, I foresee no problems 14:47:53 <markwash> but it does not solve the pending deletion use case. so we need a different mechanism for that 14:48:14 * markwash is being too loud 14:48:19 <rosmaita> or we eliminate pending delete 14:48:37 <arnaud> +1 markwash I think having a different approach for pending delete is fine/good 14:48:40 <ativelkov> hm 14:48:47 <jokke_> I don't think people will be too happy about that 14:48:58 <ativelkov> let's say /artifacts return all of them, including the deleted ones 14:48:59 <zhiyan> recover is a worth thing tbh 14:49:18 <jokke_> (eliminating pending delete I mean ^^^) 14:49:28 <zhiyan> and on the same time, pending_delete is a good thing for performance 14:49:31 <markwash> it seems like user driven recovery is useful enough to find a spot in our api 14:49:56 <ativelkov> /artifacts?state=pending_delete will return only the pending ones 14:50:29 <rosmaita> i think there is also a use case for displaying deleted image/artifact records 14:50:36 <ativelkov> and the client will have a method list_active_artifacts() which will eventually call /artifacts?state=ready&owner=...& 14:50:44 <markwash> rosmaita: +1, you can't recover from them always if you can't discover them 14:50:55 <zhiyan> agreed 14:51:06 <rosmaita> but the deleted items could overwhelm the existing ones 14:51:13 <rosmaita> if /GET includes deleted 14:51:17 <markwash> ativelkov: this is a consistent proposal. may I offer an alternative? 14:51:26 <ativelkov> markwash: sure 14:51:43 <markwash> GET /artifacts returns all immediately useable artifacts (i.e. if its an image, you can boot from it) 14:51:55 <markwash> GET /deleted_artifacts shows the artifacts that are deleted but not yet expunged 14:52:23 <markwash> deleted_artifacts have additional system properties such as "deleted_at" and "expires_at" 14:52:51 <markwash> and there is an operation " POST /deleted_artifacts?recover" or somesuch that allows user-driven recovery of deleted artifacts 14:53:14 <rosmaita> markwash: i am thinking of a case where you don't want the image data any more, but you want to be able to retrieve the image record because of valueable metadata that could be useful to know about 14:53:21 <ativelkov> "immediately useable" means "valid for the current tenant"? 14:53:54 <markwash> ativelkov: hmm, that's part of it 14:54:04 <markwash> there might be other restrictions implied by "useable" 14:54:06 <ativelkov> Yes, that's what I mean. 14:54:43 <ativelkov> And then we may have GET /artifact_drafs to return the artifacts which are not completed (published) yet 14:54:59 <ativelkov> For multi-blob upload etc 14:55:00 <arnaud> rosmaita: have you ever seen that? really? I would be interested to have a concrete example 14:55:51 <markwash> ativelkov: in tasks / import we accomplish that by GET /tasks 14:55:52 <jokke_> how about making /deleted/ instead, FE. GET /deleted/{artifacts, images} returning the deleted ones and you can filter that with peding or what so ever? 14:55:59 <rosmaita> arnaud: if have packages listed in the metadata, may need to go back and seee what was working at that time 14:56:11 <markwash> however, that assumes that the publication process is not interactive 14:56:26 * markwash reads scrollback 14:56:53 <markwash> jokke_: that seems fine 14:56:54 <jokke_> same with the draft ... having /draft/artifacts path rather than /draft_artifacts would keep consistent no matter what we call the actual item (it being artifact, image, template, etc.) 14:57:22 <rosmaita> arnaud: also, for UI uses, may have an instance, you know the image it was created from, may be pulling data from the image record for display purposes 14:57:24 <zhiyan> hmm..i *think* this could be considered with "functional api", maybe we can do this: GET /artifact just return "useable" ones (def usable later), and use POST /artifact/action to get artifact items with different status, and giving the filters in the body with json format 14:57:26 <mclaren> or /pending_deleted? 14:57:48 <markwash> mclaren: I think with /deleted we basically don't need pending deleted anymore 14:58:03 <arnaud> ok I see rosmaita thanks 14:58:09 <mclaren> to me deleted means gone, or are we introducing a 'really deleted'/expunged? 14:58:09 <rosmaita> zhiyan: is there some general info about "functional api" i could look at prior to summit? 14:58:38 <markwash> mclaren: that's what I was thinking. . I suppose it is a bit smelly 14:59:12 <mclaren> or /delete_queue 14:59:18 <markwash> mclaren: but we don't really introduce it, it's just what happens when an artifact leaves the /deleted/artifacts list 14:59:25 <zhiyan> rosmaita: yes, i did. https://etherpad.openstack.org/p/glance-adding-functional-operations-to-api 14:59:30 <markwash> s/it/expunged/ 14:59:43 <markwash> eek we're almost out of time! 15:00:09 <jokke_> mclaren & markwash: I think rosmaita made good point with the metadata of deleted which actual payload is already gone 15:01:08 <markwash> jokke_: rosmaita: the use case does seem reasonable. its possible that nothing ever leaves deleted on its own, but that the "binary blobs" do disappear such that recovery becomes impossible 15:01:11 <markwash> okay we're over 15:01:18 <markwash> thanks everybody 15:01:24 <jokke_> thnx 15:01:27 <arnaud> thks 15:01:28 <markwash> I look forward to seeing as many of you as possible at the summit 15:01:32 <ativelkov> We'll continue on the summit, thanks 15:01:36 <markwash> thanks for humoring me! 15:01:39 <markwash> #endmeeting