14:00:23 #startmeeting glance 14:00:24 Meeting started Thu Jul 31 14:00:23 2014 UTC and is due to finish in 60 minutes. The chair is rosmaita. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:00:25 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:00:27 The meeting name has been set to 'glance' 14:00:35 o/ 14:00:41 o/ 14:00:48 * flaper87 asks permission to request something 14:01:01 o/ 14:01:03 the chair recognizes flaper87 14:01:05 0/ 14:01:08 o/ 14:01:08 wow 14:01:11 :P 14:01:25 just kidding, markwash is chair 14:01:34 so, AFAIK, glance.store is in the agenda. I wanted to ask if we could do it before everything else :P 14:01:39 * flaper87 is running out of battery 14:01:51 flaper87: its first 14:01:52 it's the first item on the agenda 14:01:53 it is the first item in the agenda 14:01:56 awesome 14:01:57 bam 14:02:02 * flaper87 knew he had to read the agenda first 14:02:03 :P 14:02:16 okay 14:02:41 so what we felt like was a reminder / reweighing of the benefits and costs of the glance.store separation 14:02:45 at the mini summit 14:02:54 I've written down this: https://etherpad.openstack.org/p/glance-store 14:02:59 To help brainstorming 14:03:15 I've also talked to zhiyan and he has added some notes there 14:03:45 flaper87: thanks, that's a good read 14:03:57 I'm planning to move that to a wiki page 14:04:07 o/ 14:04:09 so I think it would probably first make sense if folks have want to add other costs or benefits to the list 14:04:26 yes, please 14:04:39 I didn't put just 1 because I was trying to hide the rest :P 14:04:44 o/ 14:04:45 I really couldn't think of other cons 14:04:49 so, please, add 14:06:18 okay 14:06:37 (please, put names to the etherpad lables) 14:07:29 so I think the one extra con that has been brought up sheds a lot of light on the problem we saw with a lot of the benefits 14:07:49 basically, I can't really see how using glance.store for client direct access is really going to work out in the end 14:08:15 I wonder if we can talk a bit about that 14:09:33 markwash: sure 14:09:33 so let's just try to list items in the etherpad I guess for now 14:09:37 we can respond inline in the meeting 14:12:00 so. . let's drift back folks 14:12:04 :-) 14:12:21 I accidentally dispersed us 14:12:27 :) 14:13:20 seems like a debate round of interview a for Management position 14:14:07 flaper87: so the advantages I see that are relatively unassailable: it would allow other projects to reuse our generic blob store code, though not in a way that would necessarily share the data with a glance backend 14:14:38 flaper87: and it promotes a better separation of concerns by making it harder to make calls to the db in the middle of store code 14:15:12 markwash: also, the zero-copy point is important. There's some code that has yet to be written for this that would make sense to have in glance.store 14:15:14 flaper87: and it makes it feel socially safer to work with stores as plugins 14:15:22 +1 14:15:29 since the main stores would effectively be plugins 14:15:42 also, the code in glance/store is re-usable, it's a shame that we have it there, locked down. 14:15:48 :) 14:15:49 but this is less of a pro 14:15:56 it's more like a social, moral duty 14:16:09 flaper87: can you explain how it helps with zero-copy? I kind of am confused on that point 14:16:16 it seems like the same thing as client direct access 14:16:24 which I'm not very optimistic about 14:17:06 flaper87, I have a few questions 14:17:09 about the location 14:17:21 markwash: FWIW, I'm not really optimistic about the client, it's just one more thing that it would allow us to do but it does not need to be done 14:17:31 because one of the main + is to say we can do the zero copy logic in glance.store 14:17:33 same here ... I'd also like to get bit of a light to the statement in the etherpad regarding glance not oning image data 14:17:57 but the location is/will be still stored in the glance database and "owned" by glance 14:18:18 am I missing something? 14:18:29 jokke_: glance has never owned the data. THe fact that glance knows where to put/get it doesn't mean it owns it. A clear example is the fact that glance is capable oof working with images that are in a remote location 14:18:31 arnaud__: reading 14:18:47 so I think we need to clean up the cases where glance doesn't clearly own the data 14:18:54 rather than further entrenching the idea that it doesn't own it 14:18:58 +1 14:19:01 I don't really think glance has much value if it doesn't own the data 14:19:07 markwash: +1 14:19:12 I may be missing some context, but you can deploy glance so that the database and the data are guaranteed to be in sync 14:19:12 since inconsistent data == net loss of value 14:19:20 As if glance does not on the images, why we are even trying to do any level of access control in glance 14:19:28 markwash: think that we need to state it explicitly what "ownership" means for image data! 14:19:35 markwash, would you mind clarify what you mean by "own"? 14:19:39 arnaud__: you're not missing anything. The location will still be in glance, the logic needed in glance.sotre is related to the store itself and the things that can be done in it 14:19:43 may be checksum, size, vsize, etc. 14:20:30 actually, i think glance.store gives us a chance to add more apis easily to help implementation zero-copy in nova/cinder, at least. 14:20:32 arnaud__: +1, I think there are different interpretations of owning 14:20:43 Also I think if Glance does not own the images, e need to remove the statement that they are immutable 14:20:43 zhiyan: +1 14:20:49 we 14:21:01 flaper87: +1M about the role of glance.store 14:21:07 when I say "glance owns the data" I mean it can completely prevent users and other services from making that data inconsistent with the glance db 14:21:36 markwash: +1 14:21:41 yeah, +1 14:21:52 +1 14:22:05 markwash: i think when we allow glance expose direct-location to client, we have already broken this rule. 14:22:13 zhiyan: so let's deprecate that 14:22:16 I'm serious 14:22:19 Why? 14:22:25 that's an important feature 14:22:28 ? 14:22:31 markwash: +1 14:22:33 because inconsistent data -> net loss of value 14:22:43 if we deprecate that you're basically forcing people to go through glance which is slow in some cases 14:22:49 rather than deprecate make it admin policy by default? 14:22:52 markwash: we can do the necessary check on glance.store api level 14:22:53 +1 zhiyan, as sson as you expose the location, the client can do whatever he wants with the file 14:23:13 mclaren: its already only by config, so I think we're actually safe on that front 14:23:19 direct access *needs* to be enabled in the configs 14:23:21 it's not the default 14:23:28 we can put some policies on it 14:23:53 +1 flaper87 14:24:15 so at the summit we had some counter evidence offered to the "glance is the bottleneck" side of things, and felt like we at least need more clarity about the deployment scenarios that we want to optimize glance transfers for 14:24:19 in policy.json I see '"set_image_location": ""' 14:24:25 agreed flaper87, without the location there is no way at the point to remove glance from the datapath 14:24:36 flaper87: can you provide actual data glance making things slow? as so far there has been claims for such, but no actual data to back those claims up 14:24:50 jokke_: seriously? 14:24:55 flaper87: yes 14:25:04 seriously 14:25:10 imho, as i mentioned in etherpad, glance would focus on resource repo, and store focus on content access 14:25:22 I don't have data but there's no need to provide it. If you put glance in the middle it'll add another step in the data path 14:25:28 and we can implement transferring on glance.store as well 14:25:33 flaper87: we've looked at production data, and for us glance improves performance 14:25:36 zhiyan: +1 :D 14:25:47 exactly what was said in the mini-summit 14:25:50 if the image is big and it has to go from the store to glance-api and then nova or whatever 14:25:54 nikhil___: yes, at least, that's a easier way we can go/do 14:25:56 mclaren: as in, direct access improves ? 14:26:02 another step in the datapath doesn't have to slow you down if it's not the slowest leg 14:26:02 mclaren: got lost, sorry 14:26:07 glance can not own the metadata and still be consistent if it would consume notifications from the store when it changes 14:26:15 s/metadata/data/ 14:26:31 for us glance is big bottleneck 14:26:32 mclaren: agreed but it's still another step 14:26:35 use glance.store, we can implement zero-copy, smarter full-copy 14:26:36 flaper87: for example with stunnel/stud folks are actively putting more things in the datapath to improve performance 14:26:55 flaper87: for example getting image from glnce cache seems to be faster than pulling it from swift 14:27:09 right but lets not hide that there are things that can be improved in glance's performance, datapath and the way we treat data 14:27:13 nikhil___: is the glance code a bottleneck or is your glance deployment a bottleneck? 14:27:30 jokke_: the cache code hasn't been touched in years, AFAICT. I don't even think it supports v2 14:27:45 zhiyan: no one seems to be saying *how* we will do zero copy, and until I hear that zero copy doesn't use client direct access, we're blocked before we can even consider zero copy 14:27:45 I would like someone to produce a well defined case and supporting data for doing this 14:27:52 +1 14:27:53 mclaren: it is , in our deployment. (i know it's smaller then rax) 14:28:12 zhiyan: what is, sorry? 14:28:15 it's not about the "data" of how good accessing images directly is. The use-cases of glance.store go beyond that 14:28:22 glance seems to act like a proxy for the object store which is inefficient. Shouldn't it instead just worry about the metadata and consume notifications 14:28:34 I have been using the location in the nova vmware driver, and tbh, that improves a LOT the time to boot and snapshot a vm... 14:28:37 it's about re-using code, improving the way we access data, providing a better support for the on-going work on zero copy 14:28:48 flaper87: but in terms of curating the list of pros vs cons we need to know if direct access and its kind are actually on the list 14:28:49 cleaning up the glance code 14:28:54 jcook: again, that is proof by assertion ... 14:29:04 markwash: imo, when we implement zero-copy, we need some store driver's capability, and i think this stuff can be done in store i mean. 14:29:51 mclaren: maybe I missed the contradiction 14:30:02 and folks, don't forget 3rd ci/cd stuff pls 14:30:21 jcook: I don't think there is any service which can send us those notifications 14:30:32 arnaud__: agreed 14:30:36 and really, the data is supposed to be immutable 14:30:43 mclaren: hard to describe if glance is exactly the bottleneck or the deployment of it is (without any real data -> that may mean nikhil get to work on that) However, one thing is evident that when glance is not used it's much faster, lesser bugs etc. 14:30:48 it'd also help ppl to implement their own stores outside glance and the library (yes, if we refactor the code *in* glance, it can still be done) 14:30:48 so we don't really need notifications, we just need it to never be accessed 14:31:02 markwash: that's the biggest concern for me we are giving out by this change 14:31:04 jcook: when you say inefficient, in what sense do you mean? For example swift have proxy nodes, would it be more efficient if swift removed them and swift API calls went straight to the object stores? 14:31:36 okay so I need to call time on this discussion and see if we can reframe our effort 14:31:47 markwash: the proxy pattern exists for a reason, but why is glance a proxy for object storage. What's the true value? 14:31:59 jcook: consistency 14:32:14 the only value add glance has at this point is "people can't shoot themselves in the foot with it as easily as without it" 14:32:34 we're talking about openstack services here not random folks 14:32:35 glance lite == /dev/null 14:33:00 okay, back to refocusing 14:33:12 jcook: also caching which swift doesn't have, and the potential to protect overloading swift object nodes (an image will be stored on at most 3 machines -- which can all get hit simultaneously) 14:33:16 markwash: true that, however it would be even better for someone to deploy glance without needing that extra checks 14:33:32 sometimes we do visit websites which do not have valid certs!! 14:33:57 to me, the way to proceed here 14:34:10 is to basically look at the pros / cons without any of the client-direct-access pros 14:34:32 I still feel like there is value in having regular stores live as external plugins 14:34:34 I don't think inconsistency should be used as a con for glance.store, if anything it should be used against direct-access. If we'll keep direct-access then glance.store still makes sense as per the pros mentioned in the etherpad 14:34:48 markwash: +1 14:34:49 it makes integration harder, but it makes us do a better job of integrating with completely out-of-tree plugins 14:35:01 which allows us to say "please don't merge your random store to our code" 14:35:04 mclaren: Is glance acting like a cache a bandaid for inefficient caching on the hypervisor? 14:35:19 markwash: +1 14:35:37 I don't want to rush the discussion but: Can we vote or make a final decision? 14:35:54 We need to figure this out now, the feature freeze is around the corner 14:35:56 jcook: i think it should be a function for another service ,like transferring stuff 14:36:03 flaper87, 14:36:05 markwash: on the other side how long it takes us to loose the focus to maintain them "It's just a store plugin, it really does not belong into glance so why should we spend cycles to maintain it"? 14:36:08 a few more questions 14:36:11 some folks are going in vacation and if we're going to go down this road we need to do it asap 14:36:26 could you share you plan to integrate glance with glance.store 14:37:06 #link https://review.openstack.org/#/c/100636/ 14:37:08 arnaud__: ^ 14:37:22 The work is done, it passes locally. I need to release the first alpha today/tomorrow 14:37:30 so we can consume the library in the gate 14:37:30 zhiyan: I can't think of a way to make the transfer service make sense wrt security / access-control 14:37:36 (once it's added to the global/requirements) 14:37:43 jcook: the glance cache just stores the images on the glance node's filesystem, making the image a little 'hotter' for nova nodes. 14:37:48 ok thanks flaper87 14:38:14 np :) 14:38:28 fwiw flaper87's patch for random access of image data opens the door for parallel downloads I think 14:38:58 mclaren: +1 14:39:04 yes! mclaren 14:39:28 yes, random access is fantastic 14:39:32 there is some authN issue with SingleTenant swift store too 14:39:45 flaper87: what's the consistency between glance.store and current store code? I was looking the logging during the flight and it did not look to be quite up to date last week 14:39:56 and this code would enble implementing tasks for different stores easily 14:40:17 mclaren: If nova is efficient at caching, what value does glance provide for caching it for nova? 14:40:25 jokke_: pls, if there's something missing in glance.store send it my way, I'll update glance.store 14:40:32 it should be pretty much up to date 14:40:40 the swift port landed today 14:40:50 actually, it hasn't landed, I think it's still in the gate 14:40:50 flaper87: thnx, I'll look gin 14:40:57 jcook: we may or may not able to cache any size images on nova however, glance is bit more configurable 14:41:12 again ... sorry keyboard acting a bit 14:41:19 imho, caching == L1 + L2 + L3 cache 14:41:29 not just L1 14:41:40 ok, can we go back to the original discussion? 14:41:47 Are we going to do this or not? 14:42:09 Should we vote? (based on the current pros/cons and discussion) 14:42:16 nikhil___: but is retrieval from glance really faster? it's still limited by network throughput 14:42:16 the cache of glance is a front catch for backand storage, which saved in glance-api node but nova-cpu node 14:42:27 My rush is that there's still some work left, there's no much time to do it 14:42:36 and I do think we need to do this asap if we want to do it 14:42:59 I would be curious to see where folks stand on this after this discussion 14:43:34 markwash: me too, I guess voting will solve it 14:43:43 and I hope I won't regret saying that :P 14:43:48 lol 14:43:59 markwash: flaper87 : added another pro point in the etherpad (line 28) 14:44:00 let's get a quick +1 or -1 from folks on the following proposition at the moment: we should adopt glance.store 14:44:03 frankly, i think we need think about how we can use it, with a clear useful use case 14:44:24 fwiw cinder don't have a separate repo for their backends, and have explicitly avoided it 14:44:24 nikhil___: thanks 14:44:31 "we should adopt glance.store as an external library" 14:44:49 markwash: they have bricks and I wouldn't be surprised they will in the future 14:44:51 -1 as the arguments stands now 14:44:53 mclaren: ^ 14:45:04 mclaren, yeah, that's what I mentioned in the ether pad 14:45:09 mclaren: also, their code doesn't need to be reused throughout openstack, AFAICT 14:45:15 so far only jokke_ has voted 14:45:22 +1 14:45:32 +1 14:45:54 +1 14:45:59 +1 14:46:03 +1 14:46:05 markwash: do we do # startvote ? 14:46:10 no 14:46:16 this is just a show of hands 14:46:19 ok, sorry.. 14:46:20 * nikhil___ hides 14:46:32 flaper87, is happy so far :) 14:46:39 arnaud__: fuck yeah 14:46:40 :P 14:46:41 -1 on balance right now 14:46:47 -1 (we must strip out the direct-access use case) 14:47:26 I have to turn it over to arnaud 14:47:29 :) 14:47:35 since if I don't leave right now my wife will divorce me 14:47:43 (because I won't get my visa to go visit her) 14:47:50 my -1 won't change the result apparently 14:47:51 :) 14:48:19 markwash: :D .. should not be that difficult 14:48:20 okay, so for now it seems like we're leaning towards adoption, but there are still some questions about how it is going to be used 14:48:27 *should* is operative 14:48:33 bye! 14:48:33 arnaud__: but i'm interested in why -1 :) 14:48:35 (fwiw, even if we remove direct access, I think this library is a great contribution to the community in general) 14:48:53 the plan is to use it *just* in glance for now 14:49:00 until J is released 14:49:08 and we have a better plan/API for it 14:49:50 ok, I'll get that as a: "dedicating time to this is still safe and we won't tell you it won't be merged in 2 weeks :P" 14:50:39 arnaud__: I'm curious about your -1 14:50:42 zhiyan, 1. I think we have many challenges that are much more important for this release J. 2. I think Nova is far to be ready to use this code. 3. I see the Glance stores as drivers (like the cinder/nova ones) where them are kept in the tree. 4. I think we can enrich the API of the stores even without glance.stores 14:50:48 oh thanks :P 14:51:24 arnaud__: but 1 and 2 are not really against the library but the timing 14:51:31 yes flaper87 14:52:03 arnaud__: re 3. the difference I see is that cinder/nova drivers don't need to be used elsewhere, whereas glance's do 14:52:21 1. ok. 2. this is a worth point, but i think it's a point what we need to pay more efforts, like v2 api. 14:52:24 arnaud__: ^ 14:52:35 :) 14:53:12 arnaud__: 3, cinder has brick lib (even it on the way out still) 4. yes, but it's hard for other drivers from 3rd, which will not in tree. 14:54:46 but I also see all the advantages that flaper87 mentions 14:55:48 anyway! I think you can still be happy flaper87 :) 14:56:33 flaper87: arnaud__ do you think 3rd ci/cd is a point for glance.store? imho, it's easier for other storage vendor test it, when more projects use it to access image/artfact content, and it's api can be turned to complicated in future. 14:57:49 zhiyan: what makes it easier depending if it's under project a or b? 14:57:51 zhiyan, I like to see the +1 of Minesweeper (for example), in the glance reviews which make me confident I do not break glance, rather than I do not break glance.store 14:58:40 arnaud__: but currently most store test cases are disabled in gate :( 14:59:15 they won't be enabled in glance.store afaik 14:59:17 I think you could maintain consistency and still have direct access if you were given a unique "location" from glance to pickup your object (say in the form of a direct download URL, token, or w/e). Just like when you order a burger from Five Guys. Glance is the menu / register. Glance issues you a ticket. You pickup your order at the counter with your ticket. 14:59:18 i'm not sure how minesweeper configured, even. 14:59:35 just sayin 14:59:38 arnaud__: so, i think we can cover them in 3rd testing 14:59:43 it runs glance tests with vmware store configured (so it runs functional tests of vmware store) 14:59:51 yes 14:59:57 zhiyan, which we can do today no? 15:00:13 jcook: yes, that's something that's been considered. A Swift tempurl or somesuch 15:00:21 let's move to glance room to talk a little on it. 15:00:24 arnaud__: ^ 15:00:26 :) 15:00:31 #endmeeting