14:00:23 <rosmaita> #startmeeting glance
14:00:24 <openstack> Meeting started Thu Jul 31 14:00:23 2014 UTC and is due to finish in 60 minutes.  The chair is rosmaita. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:25 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:00:27 <openstack> The meeting name has been set to 'glance'
14:00:35 <flaper87> o/
14:00:41 <arnaud__> o/
14:00:48 * flaper87 asks permission to request something
14:01:01 <TravT> o/
14:01:03 <rosmaita> the chair recognizes flaper87
14:01:05 <hemanth_> 0/
14:01:08 <jokke_> o/
14:01:08 <flaper87> wow
14:01:11 <flaper87> :P
14:01:25 <rosmaita> just kidding, markwash is chair
14:01:34 <flaper87> so, AFAIK, glance.store is in the agenda. I wanted to ask if we could do it before everything else :P
14:01:39 * flaper87 is running out of battery
14:01:51 <markwash> flaper87: its first
14:01:52 <rosmaita> it's the first item on the agenda
14:01:53 <arnaud__> it is the first item in the agenda
14:01:56 <flaper87> awesome
14:01:57 <markwash> bam
14:02:02 * flaper87 knew he had to read the agenda first
14:02:03 <flaper87> :P
14:02:16 <markwash> okay
14:02:41 <markwash> so what we felt like was a reminder / reweighing of the benefits and costs of the glance.store separation
14:02:45 <markwash> at the mini summit
14:02:54 <flaper87> I've written down this: https://etherpad.openstack.org/p/glance-store
14:02:59 <flaper87> To help brainstorming
14:03:15 <flaper87> I've also talked to zhiyan and he has added some notes there
14:03:45 <markwash> flaper87: thanks, that's a good read
14:03:57 <flaper87> I'm planning to move that to a wiki page
14:04:07 <nikhil___> o/
14:04:09 <markwash> so I think it would probably first make sense if folks have want to add other costs or benefits to the list
14:04:26 <flaper87> yes, please
14:04:39 <flaper87> I didn't put just 1 because I was trying to hide the rest :P
14:04:44 <pkoniszewski> o/
14:04:45 <flaper87> I really couldn't think of other cons
14:04:49 <flaper87> so, please, add
14:06:18 <markwash> okay
14:06:37 <flaper87> (please, put names to the etherpad lables)
14:07:29 <markwash> so I think the one extra con that has been brought up sheds a lot of light on the problem we saw with a lot of the benefits
14:07:49 <markwash> basically, I can't really see how using glance.store for client direct access is really going to work out in the end
14:08:15 <markwash> I wonder if we can talk a bit about that
14:09:33 <flaper87> markwash: sure
14:09:33 <markwash> so let's just try to list items in the etherpad I guess for now
14:09:37 <markwash> we can respond inline in the meeting
14:12:00 <markwash> so. . let's drift back folks
14:12:04 <markwash> :-)
14:12:21 <markwash> I accidentally dispersed us
14:12:27 <nikhil___> :)
14:13:20 <nikhil___> seems like a debate round of interview a for Management position
14:14:07 <markwash> flaper87: so the advantages I see that are relatively unassailable: it would allow other projects to reuse our generic blob store code, though not in a way that would necessarily share the data with a glance backend
14:14:38 <markwash> flaper87: and it promotes a better separation of concerns by making it harder to make calls to the db in the middle of store code
14:15:12 <flaper87> markwash: also, the zero-copy point is important. There's some code that has yet to be written for this that would make sense to have in glance.store
14:15:14 <markwash> flaper87: and it makes it feel socially safer to work with stores as plugins
14:15:22 <flaper87> +1
14:15:29 <markwash> since the main stores would effectively be plugins
14:15:42 <flaper87> also, the code in glance/store is re-usable, it's a shame that we have it there, locked down.
14:15:48 <jokke_> :)
14:15:49 <flaper87> but this is less of a pro
14:15:56 <flaper87> it's more like a social, moral duty
14:16:09 <markwash> flaper87: can you explain how it helps with zero-copy? I kind of am confused on that point
14:16:16 <markwash> it seems like the same thing as client direct access
14:16:24 <markwash> which I'm not very optimistic about
14:17:06 <arnaud__> flaper87, I have a few questions
14:17:09 <arnaud__> about the location
14:17:21 <flaper87> markwash: FWIW, I'm not really optimistic about the client, it's just one more thing that it would allow us to do but it does not need to be done
14:17:31 <arnaud__> because one of the main + is to say we can do the zero copy logic in glance.store
14:17:33 <jokke_> same here ... I'd also like to get bit of a light to the statement in the etherpad regarding glance not oning image data
14:17:57 <arnaud__> but the location is/will be still stored in the glance database and "owned" by glance
14:18:18 <arnaud__> am I missing something?
14:18:29 <flaper87> jokke_: glance has never owned the data. THe fact that glance knows where to put/get it doesn't mean it owns it. A clear example is the fact that glance is capable oof working with images that are in a remote location
14:18:31 <flaper87> arnaud__: reading
14:18:47 <markwash> so I think we need to clean up the cases where glance doesn't clearly own the data
14:18:54 <markwash> rather than further entrenching the idea that it doesn't own it
14:18:58 <jokke_> +1
14:19:01 <markwash> I don't really think glance has much value if it doesn't own the data
14:19:07 <nikhil___> markwash: +1
14:19:12 <mclaren> I may be missing some context, but you can deploy glance so that the database and the data are guaranteed to be in sync
14:19:12 <markwash> since inconsistent data == net loss of value
14:19:20 <jokke_> As if glance does not on the images, why we are even trying to do any level of access control in glance
14:19:28 <nikhil___> markwash: think that we need to state it explicitly what "ownership" means for image data!
14:19:35 <arnaud__> markwash, would you mind clarify what you mean by "own"?
14:19:39 <flaper87> arnaud__: you're not missing anything. The location will still be in glance, the logic needed in glance.sotre is related to the store itself and the things that can be done in it
14:19:43 <nikhil___> may be checksum, size, vsize, etc.
14:20:30 <zhiyan> actually, i think glance.store gives us a chance to add more apis easily to help implementation zero-copy in nova/cinder, at least.
14:20:32 <flaper87> arnaud__: +1, I think there are different interpretations of owning
14:20:43 <jokke_> Also I think if Glance does not own the images, e need to remove the statement that they are immutable
14:20:43 <flaper87> zhiyan: +1
14:20:49 <jokke_> we
14:21:01 <nikhil___> flaper87: +1M about the role of glance.store
14:21:07 <markwash> when I say "glance owns the data" I mean it can completely prevent users and other services from making that data inconsistent with the glance db
14:21:36 <rosmaita> markwash: +1
14:21:41 <nikhil___> yeah, +1
14:21:52 <flaper87> +1
14:22:05 <zhiyan> markwash: i think when we allow glance expose direct-location to client, we have already broken this rule.
14:22:13 <markwash> zhiyan: so let's deprecate that
14:22:16 <markwash> I'm serious
14:22:19 <flaper87> Why?
14:22:25 <flaper87> that's an important feature
14:22:28 <zhiyan> ?
14:22:31 <jokke_> markwash: +1
14:22:33 <markwash> because inconsistent data -> net loss of value
14:22:43 <flaper87> if we deprecate that you're basically forcing people to go through glance which is slow in some cases
14:22:49 <mclaren> rather than deprecate make it admin policy by default?
14:22:52 <zhiyan> markwash: we can do the necessary check on glance.store api level
14:22:53 <arnaud__> +1 zhiyan, as sson as you expose the location, the client can do whatever he wants with the file
14:23:13 <markwash> mclaren: its already only by config, so I think we're actually safe on that front
14:23:19 <flaper87> direct access *needs* to be enabled in the configs
14:23:21 <flaper87> it's not the default
14:23:28 <flaper87> we can put some policies on it
14:23:53 <zhiyan> +1 flaper87
14:24:15 <markwash> so at the summit we had some counter evidence offered to the "glance is the bottleneck" side of things, and felt like we at least need more clarity about the deployment scenarios that we want to optimize glance transfers for
14:24:19 <mclaren> in policy.json I see '"set_image_location": ""'
14:24:25 <arnaud__> agreed flaper87, without the location there is no way at the point to remove glance from the datapath
14:24:36 <jokke_> flaper87: can you provide actual data glance making things slow? as so far there has been claims for such, but no actual data to back those claims up
14:24:50 <flaper87> jokke_: seriously?
14:24:55 <jokke_> flaper87: yes
14:25:04 <jokke_> seriously
14:25:10 <zhiyan> imho, as i  mentioned in etherpad, glance would focus on resource repo, and store focus on content access
14:25:22 <flaper87> I don't have data but there's no need to provide it. If you put glance in the middle it'll add another step in the data path
14:25:28 <zhiyan> and we can implement transferring on glance.store as well
14:25:33 <mclaren> flaper87: we've looked at production data, and for us glance improves performance
14:25:36 <nikhil___> zhiyan: +1 :D
14:25:47 <nikhil___> exactly what was said in the mini-summit
14:25:50 <flaper87> if the image is big and it has to go from the store to glance-api and then nova or whatever
14:25:54 <zhiyan> nikhil___: yes, at least, that's a easier way we can go/do
14:25:56 <flaper87> mclaren: as in, direct access improves ?
14:26:02 <mclaren> another step in the datapath doesn't have to slow you down if it's not the slowest leg
14:26:02 <flaper87> mclaren: got lost, sorry
14:26:07 <jcook> glance can not own the metadata and still be consistent if it would consume notifications from the store when it changes
14:26:15 <jcook> s/metadata/data/
14:26:31 <nikhil___> for us glance is big bottleneck
14:26:32 <flaper87> mclaren: agreed but it's still another step
14:26:35 <zhiyan> use glance.store, we can implement zero-copy, smarter full-copy
14:26:36 <mclaren> flaper87: for example with stunnel/stud folks are actively putting more things in the datapath to improve performance
14:26:55 <jokke_> flaper87: for example getting image from glnce cache seems to be faster than pulling it from swift
14:27:09 <flaper87> right but lets not hide that there are things that can be improved in glance's performance, datapath and the way we treat data
14:27:13 <mclaren> nikhil___: is the glance code a bottleneck or is your glance deployment a bottleneck?
14:27:30 <flaper87> jokke_: the cache code hasn't been touched in years, AFAICT. I don't even think it supports v2
14:27:45 <markwash> zhiyan: no one seems to be saying *how* we will do zero copy, and until I hear that zero copy doesn't use client direct access, we're blocked before we can even consider zero copy
14:27:45 <mclaren> I would like someone to produce a well defined case and supporting data for doing this
14:27:52 <jokke_> +1
14:27:53 <zhiyan> mclaren: it is , in our deployment. (i know it's smaller then rax)
14:28:12 <mclaren> zhiyan: what is, sorry?
14:28:15 <flaper87> it's not about the "data" of how good accessing images directly is. The use-cases of glance.store go beyond that
14:28:22 <jcook> glance seems to act like a proxy for the object store which is inefficient. Shouldn't it instead just worry about the metadata and consume notifications
14:28:34 <arnaud__> I have been using the location in the nova vmware driver, and tbh, that improves a LOT the time to boot and snapshot a vm...
14:28:37 <flaper87> it's about re-using code, improving the way we access data, providing a better support for the on-going work on zero copy
14:28:48 <markwash> flaper87: but in terms of curating the list of pros vs cons we need to know if direct access and its kind are actually on the list
14:28:49 <flaper87> cleaning up the glance code
14:28:54 <mclaren> jcook: again, that is proof by assertion ...
14:29:04 <zhiyan> markwash: imo, when we implement zero-copy, we need some store driver's capability, and i think this stuff can be done in store i mean.
14:29:51 <jcook> mclaren: maybe I missed the contradiction
14:30:02 <zhiyan> and folks, don't forget 3rd ci/cd stuff pls
14:30:21 <markwash> jcook: I don't think there is any service which can send us those notifications
14:30:32 <zhiyan> arnaud__: agreed
14:30:36 <markwash> and really, the data is supposed to be immutable
14:30:43 <nikhil___> mclaren: hard to describe if glance is exactly the bottleneck or the deployment of it is (without any real data -> that may mean nikhil get to work on that) However, one thing is evident that when glance is not used it's much faster, lesser bugs etc.
14:30:48 <flaper87> it'd also help ppl to implement their own stores outside glance and the library (yes, if we refactor the code *in* glance, it can still be done)
14:30:48 <markwash> so we don't really need notifications, we just need it to never be accessed
14:31:02 <jokke_> markwash: that's the biggest concern for me we are giving out by this change
14:31:04 <mclaren> jcook: when you say inefficient, in what sense do you mean? For example swift have proxy nodes, would it be more efficient if swift removed them and swift API calls went straight to the object stores?
14:31:36 <markwash> okay so I need to call time on this discussion and see if we can reframe our effort
14:31:47 <jcook> markwash: the proxy pattern exists for a reason, but why is glance a proxy for object storage. What's the true value?
14:31:59 <markwash> jcook: consistency
14:32:14 <markwash> the only value add glance has at this point is "people can't shoot themselves in the foot with it as easily as without it"
14:32:34 <flaper87> we're talking about openstack services here not random folks
14:32:35 <markwash> glance lite == /dev/null
14:33:00 <markwash> okay, back to refocusing
14:33:12 <mclaren> jcook: also caching which swift doesn't have, and the potential to protect overloading swift object nodes (an image will be stored on at most 3 machines -- which can all get hit simultaneously)
14:33:16 <nikhil___> markwash: true that, however it would be even better for someone to deploy glance without needing that extra checks
14:33:32 <nikhil___> sometimes we do visit websites which do not have valid certs!!
14:33:57 <markwash> to me, the way to proceed here
14:34:10 <markwash> is to basically look at the pros / cons without any of the client-direct-access pros
14:34:32 <markwash> I still feel like there is value in having regular stores live as external plugins
14:34:34 <flaper87> I don't think inconsistency should be used as a con for glance.store, if anything it should be used against direct-access. If we'll keep direct-access then glance.store still makes sense as per the pros mentioned in the etherpad
14:34:48 <flaper87> markwash: +1
14:34:49 <markwash> it makes integration harder, but it makes us do a better job of integrating with completely out-of-tree plugins
14:35:01 <markwash> which allows us to say "please don't merge your random store to our code"
14:35:04 <jcook> mclaren: Is glance acting like a cache a bandaid for inefficient caching on the hypervisor?
14:35:19 <flaper87> markwash: +1
14:35:37 <flaper87> I don't want to rush the discussion but: Can we vote or make a final decision?
14:35:54 <flaper87> We need to figure this out now, the feature freeze is around the corner
14:35:56 <zhiyan> jcook: i think it should be a function for another service ,like transferring stuff
14:36:03 <arnaud__> flaper87,
14:36:05 <jokke_> markwash: on the other side how long it takes us to loose the focus to maintain them "It's just a store plugin, it really does not belong into glance so why should we spend cycles to maintain it"?
14:36:08 <arnaud__> a few more questions
14:36:11 <flaper87> some folks are going in vacation and if we're going to go down this road we need to do it asap
14:36:26 <arnaud__> could you share you plan to integrate glance with glance.store
14:37:06 <flaper87> #link https://review.openstack.org/#/c/100636/
14:37:08 <flaper87> arnaud__: ^
14:37:22 <flaper87> The work is done, it passes locally. I need to release the first alpha today/tomorrow
14:37:30 <flaper87> so we can consume the library in the gate
14:37:30 <markwash> zhiyan: I can't think of a way to make the transfer service make sense wrt security / access-control
14:37:36 <flaper87> (once it's added to the global/requirements)
14:37:43 <mclaren> jcook: the glance cache just stores the images on the glance node's filesystem, making the image a little 'hotter' for nova nodes.
14:37:48 <arnaud__> ok thanks flaper87
14:38:14 <flaper87> np :)
14:38:28 <mclaren> fwiw flaper87's patch for random access of image data opens the door for parallel downloads I think
14:38:58 <flaper87> mclaren: +1
14:39:04 <nikhil___> yes! mclaren
14:39:28 <markwash> yes, random access is fantastic
14:39:32 <nikhil___> there is some authN issue with SingleTenant swift store too
14:39:45 <jokke_> flaper87: what's the consistency between glance.store and current store code? I was looking the logging during the flight and it did not look to be quite up to date last week
14:39:56 <nikhil___> and this code would enble implementing tasks for different stores easily
14:40:17 <jcook> mclaren: If nova is efficient at caching, what value does glance provide for caching it for nova?
14:40:25 <flaper87> jokke_: pls, if there's something missing in glance.store send it my way, I'll update glance.store
14:40:32 <flaper87> it should be pretty much up to date
14:40:40 <flaper87> the swift port landed today
14:40:50 <flaper87> actually, it hasn't landed, I think it's still in the gate
14:40:50 <jokke_> flaper87: thnx, I'll look gin
14:40:57 <nikhil___> jcook: we  may or may not able to cache any size images on nova however, glance is bit more configurable
14:41:12 <jokke_> again ... sorry keyboard acting a bit
14:41:19 <nikhil___> imho, caching == L1 + L2 + L3 cache
14:41:29 <nikhil___> not just L1
14:41:40 <flaper87> ok, can we go back to the original discussion?
14:41:47 <flaper87> Are we going to do this or not?
14:42:09 <flaper87> Should we vote? (based on the current pros/cons and discussion)
14:42:16 <jcook> nikhil___: but is retrieval from glance really faster? it's still limited by network throughput
14:42:16 <zhiyan> the cache of glance is a front catch for backand storage, which saved in glance-api node but nova-cpu node
14:42:27 <flaper87> My rush is that there's still some work left, there's no much time to do it
14:42:36 <flaper87> and I do think we need to do this asap if we want to do it
14:42:59 <markwash> I would be curious to see where folks stand on this after this discussion
14:43:34 <flaper87> markwash: me too, I guess voting will solve it
14:43:43 <flaper87> and I hope I won't regret saying that :P
14:43:48 <arnaud__> lol
14:43:59 <nikhil___> markwash: flaper87 : added another pro point in the etherpad (line 28)
14:44:00 <markwash> let's get a quick +1 or -1 from folks on the following proposition at the moment: we should adopt glance.store
14:44:03 <zhiyan> frankly, i think we need think about how we can use it, with a clear useful use case
14:44:24 <mclaren> fwiw cinder don't have a separate repo for their backends, and have explicitly avoided it
14:44:24 <flaper87> nikhil___: thanks
14:44:31 <markwash> "we should adopt glance.store as an external library"
14:44:49 <flaper87> markwash: they have bricks  and I wouldn't be surprised they will in the future
14:44:51 <jokke_> -1 as the arguments stands now
14:44:53 <flaper87> mclaren: ^
14:45:04 <arnaud__> mclaren, yeah, that's what I mentioned in the ether pad
14:45:09 <flaper87> mclaren: also, their code doesn't need to be reused throughout openstack, AFAICT
14:45:15 <markwash> so far only jokke_ has voted
14:45:22 <flaper87> +1
14:45:32 <hemanth_> +1
14:45:54 <nikhil___> +1
14:45:59 <rosmaita> +1
14:46:03 <zhiyan> +1
14:46:05 <nikhil___> markwash: do we do # startvote ?
14:46:10 <markwash> no
14:46:16 <markwash> this is just a show of hands
14:46:19 <nikhil___> ok, sorry..
14:46:20 * nikhil___ hides
14:46:32 <arnaud__> flaper87, is happy so far :)
14:46:39 <flaper87> arnaud__: fuck yeah
14:46:40 <flaper87> :P
14:46:41 <mclaren> -1 on balance right now
14:46:47 <markwash> -1 (we must strip out the direct-access use case)
14:47:26 <markwash> I have to turn it over to arnaud
14:47:29 <arnaud__> :)
14:47:35 <markwash> since if I don't leave right now my wife will divorce me
14:47:43 <markwash> (because I won't get my visa to go visit her)
14:47:50 <arnaud__> my -1 won't change the result apparently
14:47:51 <arnaud__> :)
14:48:19 <nikhil___> markwash: :D .. should not be that difficult
14:48:20 <markwash> okay, so for now it seems like we're leaning towards adoption, but there are still some questions about how it is going to be used
14:48:27 <markwash> *should* is operative
14:48:33 <markwash> bye!
14:48:33 <zhiyan> arnaud__: but i'm interested in why -1 :)
14:48:35 <flaper87> (fwiw, even if we remove direct access, I think this library is a great contribution to the community in general)
14:48:53 <flaper87> the plan is to use it *just* in glance for now
14:49:00 <flaper87> until J is released
14:49:08 <flaper87> and we have a better plan/API for it
14:49:50 <flaper87> ok, I'll get that as a: "dedicating time to this is still safe and we won't tell you it won't be merged in 2 weeks :P"
14:50:39 <flaper87> arnaud__: I'm curious about your -1
14:50:42 <arnaud__> zhiyan, 1. I think we have many challenges that are much more important for this release J. 2. I think Nova is far to be ready to use this code. 3. I see the Glance stores as drivers (like the cinder/nova ones) where them are kept in the tree. 4. I think we can enrich the API of the stores even without glance.stores
14:50:48 <flaper87> oh thanks :P
14:51:24 <flaper87> arnaud__: but 1 and 2 are not really against the library but the timing
14:51:31 <arnaud__> yes flaper87
14:52:03 <flaper87> arnaud__: re 3. the difference I see is that cinder/nova drivers don't need to be used elsewhere, whereas glance's do
14:52:21 <zhiyan> 1. ok. 2. this is a worth point, but i think it's a point what we need to pay more efforts, like v2 api.
14:52:24 <zhiyan> arnaud__: ^
14:52:35 <arnaud__> :)
14:53:12 <zhiyan> arnaud__: 3, cinder has brick lib (even it on the way out still) 4. yes, but it's hard for other drivers from 3rd, which will not in tree.
14:54:46 <arnaud__> but I also see all the advantages that flaper87 mentions
14:55:48 <arnaud__> anyway! I think you can still be happy flaper87  :)
14:56:33 <zhiyan> flaper87: arnaud__ do you think 3rd ci/cd is a point for glance.store? imho, it's easier for other storage vendor test it, when more projects use it to access image/artfact content, and it's api can be turned to complicated in future.
14:57:49 <jokke_> zhiyan: what makes it easier depending if it's under project a or b?
14:57:51 <arnaud__> zhiyan, I like to see the +1 of Minesweeper (for example), in the glance reviews which make me confident I do not break glance, rather than I do not break glance.store
14:58:40 <zhiyan> arnaud__: but currently most store test cases are disabled in gate :(
14:59:15 <arnaud__> they won't be enabled in glance.store afaik
14:59:17 <jcook> I think you could maintain consistency and still have direct access if you were given a unique "location" from glance to pickup your object (say in the form of a direct download URL, token, or w/e). Just like when you order a burger from Five Guys. Glance is the menu / register. Glance issues you a ticket. You pickup your order at the counter with your ticket.
14:59:18 <zhiyan> i'm not sure how minesweeper configured, even.
14:59:35 <jcook> just sayin
14:59:38 <zhiyan> arnaud__: so, i think we can cover them in 3rd testing
14:59:43 <arnaud__> it runs glance tests with vmware store configured (so it runs functional tests of vmware store)
14:59:51 <zhiyan> yes
14:59:57 <arnaud__> zhiyan, which we can do today no?
15:00:13 <mclaren> jcook: yes, that's something that's been considered. A Swift tempurl or somesuch
15:00:21 <zhiyan> let's move to glance room to talk a little on it.
15:00:24 <zhiyan> arnaud__: ^
15:00:26 <arnaud__> :)
15:00:31 <arnaud__> #endmeeting