15:04:39 <markwash> #startmeeting glance
15:04:40 <openstack> Meeting started Fri Nov 15 15:04:39 2013 UTC and is due to finish in 60 minutes.  The chair is markwash. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:04:41 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:04:43 <openstack> The meeting name has been set to 'glance'
15:04:50 <iccha> hey
15:04:55 <rosmaita> howdy
15:05:17 <flaper87> o/
15:05:45 <mclaren> hey all, not sure if everyone got a chance to see the latest draft of the doc I put together?
15:06:02 * rosmaita is reading now
15:06:07 <iccha> same here
15:06:14 <iccha> thanks for putting together the document mclaren
15:06:16 <mclaren> https://region-a.geo-1.objects.hpcloudsvc.com/v1/61624292678963/public_referenced/consistent-multi-tenant-swift-backend.pdf
15:06:26 <markwash> #topic swift backend and security
15:06:42 <markwash> I'm still catching up to the latest revision
15:06:54 <mclaren> np, its all selfishly motivated, I'll like to sleep more peacefully ;-)
15:07:33 <markwash> I was going over this with a security guy recently, and realized again how desperately we need a fix for this
15:07:40 <mclaren> So I still don't have a perfect proposal...
15:08:03 <mclaren> Closest right now is a 2nd swift (but expensive, admin headache etc)
15:08:28 <mclaren> Anyone know any Keystone gurus? I hit the limit of knowlege on how they might be able to help really fast
15:08:48 <rosmaita> mclaren: quick question
15:08:57 <mclaren> sure
15:09:11 <rosmaita> for multitenant, are images really stored one-per-container (bottom of p 1)?
15:09:24 <rosmaita> (i am completely ignorant of multitenant)
15:09:32 <markwash> rosmaita: yes
15:09:36 <mclaren> I believe so, I think that must be the case because Swift ACLs are at the container level
15:09:46 <iccha> thats how we can share images
15:09:52 <iccha> with per containter image acls
15:10:03 <iccha> by granting other users read priviledge
15:10:17 <mclaren> yup
15:10:40 <mclaren> I'm interested to know how people feel about
15:11:10 <mclaren> multi-tenant being less fragile than single tenant
15:11:15 <iccha> like you mentioned mclaren the biggest concern is user being able to access them directly outside of glance and manipulatun them
15:11:23 <iccha> i definitely see the benefits of multitenant
15:11:47 <iccha> also ho easy/painful would it be to migrate from single tenant to multitenant?
15:11:58 <markwash> I do personally wonder if multitenant even really works at a basic level
15:12:23 <markwash> for a long time the basic sharing aspect did not work with swift acls
15:12:24 <mclaren> markwash: ok, can you expand? do you mean the current implementation?
15:12:27 <markwash> yeah
15:12:28 <rosmaita> well, as mclaren points out, it's a pretty crappy user experience when you list your containers
15:13:21 <markwash> I also wonder if we could accomplish some of the general robustness to deletion of X by allowing for a more complex default structure among swift containers in the single tenant store
15:13:45 <iccha> could you elaborate markwash
15:13:48 <markwash> like, instead of having one default set of credentials for all writes
15:13:53 <markwash> we could have some sort of round-robin
15:14:06 <markwash> so that folks generally ended up with multiple equally balanced containers
15:14:44 <mclaren> so password change issues with that! :-P
15:14:51 <mclaren> so/no
15:15:08 <markwash> let's talk about password changes for a sec
15:15:13 <markwash> I have a naive hope
15:15:15 <iccha> have we eliminated all possible work arounds for password issue?
15:15:29 <iccha> markwash: types faster than me :p
15:15:39 <markwash> which is that it is possible to have multiple passwords simultaneously valid on a given account
15:15:42 <mclaren> I think that the user access key keystone blueprint gives hope in that area
15:16:01 <iccha> mclaren: can you link to the bp?
15:16:17 <mclaren> it would allow rolling upload of credentials (key now rather than p/w)
15:16:18 <markwash> so that way you can do a simple "add second creds", "roll update to creds", "delete old creds"
15:16:30 <markwash> ah good
15:16:55 <markwash> mclaren: how much of the issue with password updates would that resolve?
15:16:56 <mclaren> https://blueprints.launchpad.net/keystone/+spec/access-key-authentication
15:17:12 <mclaren> a lot
15:17:15 <iccha> and we would abstract away the user credentials from locatuion
15:17:31 <iccha> and store it elsewhere like config?
15:17:39 <iccha> is that what you re thinking about mark?
15:17:48 <mclaren> but personally I feel multi-tenant has fewer paths to disaster
15:18:20 <mclaren> if we switched to user access keys we're still storing them on disk and can still leak them
15:18:45 <markwash> mclaren: I definitely get that. . I'm just worried about the changes to swift we might need for multi-tenant to get someplace good
15:18:57 <mclaren> the Zen of no-password in the multi-tenant approach I find hugely appealing, and the distributed container nature
15:18:59 <markwash> multi-tenant on an internal-only swift is pretty much the "best" solution
15:19:25 <markwash> but I don't think its viable for any other deployers than the largest
15:19:39 <iccha> but unfortunately hardest one to actually do
15:19:40 <mclaren> sure, I do wonder if there's some keystone 'magic' that could help (waves hands)
15:20:02 <iccha> rosmaita: and me were having the same discussion if we could blacklist/white ips to contains
15:20:06 <iccha> *containers in swift
15:20:35 <markwash> talking about passwords again for a moment
15:20:57 <markwash> I think the highest priority concern on the list is the fact that passwords are being stored in a database (even encrypted is pretty bad)
15:21:44 <rosmaita> yes, makes it impossible to change pw without big DB operations (very slow)
15:22:12 <markwash> rosmaita: well, we could fix the speed of that DB update just by normalizing out the credentials to their own table
15:22:20 <mclaren> yes and no, if the API process has an all powerful password in plaintext it can leak it -- this has happened before and is *bad*
15:22:48 <markwash> but again, that means anyone who owns the db anywhere can try to attack those passwords
15:23:25 <markwash> mclaren: hmm, good point. . but the only way to get around that is to figure out going completely passwordless, right?
15:23:41 <mclaren> I think the single tenant password change is solvable, but I want to go futher!
15:24:03 <mclaren> right
15:24:15 <notmyname> would temporary URLs in swift be useful here?
15:24:28 <notmyname> ie a signed URL that doesn't require extra creds?
15:24:31 <markwash> okiedokie. . I'll back off for a bit. . I'm still really interested in your "best" approach :-)
15:25:04 <markwash> notmyname: I think that does help us with the "client downloads image directly from swift" story
15:25:05 <iccha> notmyname: we would still need the store creentials in a single tenant approach right?
15:25:33 <notmyname> iccha: you would need to know the shared secret, which is set on a per-account basis
15:25:38 <markwash> notmyname: but glance still needs credentials in some form to have permissions to generate that temp url
15:25:42 <markwash> yeah
15:26:36 <notmyname> markwash: if the shared secret is known, the tempurl can be generated with no other access required
15:26:53 <iccha> notmyname: but we would need to store the shared secret
15:26:56 <notmyname> markwash: and you can fetcht he shared secret if you have the creds for the account (ie it's set in account metadata)
15:27:04 <nikhil__> o/
15:27:08 <markwash> interesting
15:27:11 <notmyname> and I'm guessing you already have the user creds
15:27:27 <mclaren> I did think about a per container shared secret with signing as part of a dual auth approach, is that roughtly what we're talking about?
15:27:46 <nikhil__> wonder if you all thinking about shared secrets, we can try to think about a 3 pass auth mechnism
15:27:51 <notmyname> http://docs.openstack.org/developer/swift/misc.html#module-swift.common.middleware.tempurl
15:28:07 <mclaren> are we trying to suggest a way to prevent a user modifying objects in their own container?
15:29:27 * markwash is not sure
15:29:32 <iccha> so glance would have shared secret+user token from context. but users would not have shared secret so cannot access it ? is this a swft setting to ensure dual auth?
15:29:55 <markwash> it seems kind of like we're all focusing on different things right now
15:30:09 <markwash> which is good for parallelism, but really hard to construct a coherent conversation out of it
15:30:20 <iccha> +1
15:30:35 <notmyname> seems like you were looking for a way to give people access to an object without having to create many different containers. tempurl is a way to do it
15:31:06 <markwash> so the focus right now I think is still, how could we make multitenant viable so that we could go passwordless
15:31:20 <markwash> mclaren: do you agree with that focus for the moment?
15:31:23 <mclaren> ok, we can investigate if tempurl potentially helps with that
15:31:28 <mclaren> markwash: yes
15:31:59 <mclaren> I think extra swift hardware is one way, and the holy grail is something software based
15:32:19 <markwash> software based -> something like an internal only endpoint
15:32:29 <mclaren> exactly
15:32:32 <markwash> or a way to mark a container ABAC as "internal only" ?
15:32:42 <mclaren> like a virtual region (waves handes)
15:32:47 <mclaren> yeah
15:32:54 <iccha> you read my mind mclaren
15:33:01 <markwash> ABAC = attribute based access control, where something like your ip, or the interface you came in on, is an attribute
15:33:23 <iccha> yeah like the whitelisting concept
15:33:30 <markwash> one potential problem with this approach
15:33:38 <markwash> is that it would cause headaches for existing billing folks
15:33:59 <markwash> because I think the normal right now is to charge folks for their swift storage & their glance image storage
15:34:17 <iccha> is the charge different?
15:34:21 <mclaren> that's on the swift side, there may be keystone possibilities (eg https://github.com/openstack/keystone/commit/601eeb50b60a2e99041690fe19238202bc203503) but my knowlege on keystone is poor
15:34:45 <markwash> iccha: well, in such a system, if we rolled this out it would suddenly be charging double, even if they're the same
15:35:08 <mclaren> personally I think billing is a secondary issue; design a robust system then figure out how to charge the $
15:35:20 <markwash> mclaren: that makes sense
15:35:32 <rosmaita> it would require coordination before rollout
15:35:40 <rosmaita> but not insurmountable
15:35:42 <markwash> mclaren: would users be able to see such a container in their account?
15:35:56 <mclaren> so, anyone know any keystone gurus? :-)
15:36:23 <nikhil__> markwash: if we keep pass in db and send it as a "digest" we can aboid whitelisting
15:36:27 <mclaren> markwash: ideally no, ideally users wouldn't even know swift is the backend
15:37:09 <nikhil__> (single tenant)
15:37:16 <markwash> well, one thing to keep in mind is that this effort would be concurrent to the general effort of always publishing the location details
15:37:16 <dolphm> o/
15:37:29 <markwash> oh, hey, I know a keystone guru :-)
15:37:31 <iccha> and keystone guu appears
15:37:32 <markwash> hi dolphm
15:37:36 <dolphm> \o/
15:38:05 <markwash> A keystone guru uses knowledge! Its super effective.
15:38:23 <mclaren> hey dolphm general background doc for some night time reading https://region-a.geo-1.objects.hpcloudsvc.com/v1/61624292678963/public_referenced/consistent-multi-tenant-swift-backend.pdf
15:38:49 <dolphm> mclaren: thanks
15:39:13 <mclaren> np, its basically a big shopping list with no solid proposals :-P
15:39:32 <mclaren> although I do think that a multi-tenant approach feels right
15:39:46 <iccha> maybe we could start by asking dolphm the questions we had about what keystone could do or might be planning to do
15:40:04 <dolphm> (i'm reading this doc, but fire away)
15:40:31 <markwash> mclaren: i think you could probably state the questions best
15:40:42 <mclaren> ok
15:41:03 <mclaren> I think we're wondering if there is something which exists in keystone
15:41:12 <mclaren> or could be reasonably implemented
15:41:37 <mclaren> which would give us a 'consistent multi tenant' solution
15:42:02 <dolphm> in the multi-tenant case, shouldn't glance be querying the container for the images, rather than relying on it's own stale db?
15:42:24 <mclaren> swift objects would be stored on behalf of a user but would not be directly accessible by that user
15:42:58 <dolphm> except for a special group of users, everyone else only needs read access to the container for the multi-tenant solution?
15:43:19 <mclaren> hmm
15:43:24 <markwash> its like we're looking for another swift endpoint that in general is guaranteed to have some sort of internal-only ABAC restriction on it
15:43:25 <dolphm> it seems like you want some images shared, and some images managed privately per-tenant?
15:43:36 <dolphm> markwash: abac or rbac?
15:43:40 <markwash> abac
15:43:49 <dolphm> markwash: based on what attributes?
15:43:54 <mclaren> dolphm: shared images can be handled easily enough with swift ACLs I think
15:44:19 <markwash> network location, as a straw man
15:44:29 <mclaren> so I think considering just the non-shared case at first keeps it a little simpler
15:44:48 <dolphm> is it one container per image in the multi-tenant case?
15:44:55 <mclaren> yes
15:45:03 <dolphm> i guess the other way around (one image per container)
15:45:05 <markwash> that's just there for sharing, we could change that
15:45:36 <iccha> markwash: is there object level acl in swift?
15:45:37 <mclaren> swift ACLs are enforced on a container so it may be practical to keep that way of working
15:45:43 <notmyname> sharing can be done with tempurls. which would then allow you to do one glance account + one container per tenant
15:45:43 <dolphm> iccha: i thought there was?
15:45:57 <dolphm> chmouel: o/
15:46:03 <mclaren> no object level ACLs in swift I believe
15:46:13 <notmyname> no. ACLs are container-based
15:46:23 <dolphm> notmyname: thanks!
15:46:33 <markwash> iccha: let's just forget about sharing for now. if we had this swift solution we could make it so only glance has access to the containers and still manage sharing
15:46:56 <iccha> makes sense
15:47:22 <markwash> dolphm: on a slightly different thread, can I use trusts/delegation to give someone a token that only allows them to download a single object from swift?
15:48:06 <chmouel> mclaren: no object acl indeed
15:48:14 <chmouel> dolphm: ^
15:48:32 <dolphm> markwash: unfortunately, no
15:48:38 <dolphm> markwash: roles wouldn't be granular enough
15:48:46 <markwash> k, I guess tempurl is enough for that anyway
15:49:53 <markwash> mclaren: is it fair to say that the big need in this approach is for swift containers that are only accessible over the internal endpoint?
15:50:11 <dolphm> i'd argue with the 'accidental deletion of X' "characteristics" -- those are natural consequences and shouldn't be avoided
15:50:36 <notmyname> FWIW, tempurl is essentially an auth system itself. if someone in the keystone world wanted something to work on, I supposed keystone could implement it too
15:50:51 <mclaren> dolphm: not sure I understand?
15:51:21 <dolphm> mclaren: notmyname: it's a "bearer" url, the "token" is just encoded in the URL
15:53:13 <markwash> notmyname: is there anyway to create a container that the "owner" is not allowed to delete or list, but that the glance service user would be allowed to manipulate?
15:53:15 <mclaren> markwash: really its a way to store the data out of the direct reach of users
15:53:31 <dolphm> “Users should not be able to perform operations to manipulate image data directly via the Swift Api." why not?
15:53:33 <notmyname> markwash: that would be possible to do in auth middleware in swift (ie without changing "core" swift)
15:53:59 <markwash> notmyname: gotcha, so it would require new work, but that work would have a pretty clear place to live?
15:53:59 <iccha> dolphm: we want glance to reflect the current state of images
15:54:00 <mclaren> dolphm: image data is immutable
15:54:02 <notmyname> markwash: but I'm not sure I understand the use case of storing something on behalf of someone where they can't do anything to it
15:54:16 <notmyname> markwash: correct
15:54:35 <markwash> notmyname: store images in the users swift account, but don't let them muck with it and inevitably generate support calls
15:54:37 <mclaren> notmyname: they can access it, but only in ways defined by Glance
15:54:42 <rosmaita> the reason is that glance stores all metadata about the image, need to keep metadata consistent with the actual image
15:54:52 <markwash> mclaren's way of saying it was better :-)
15:54:56 <dolphm> iccha: why is that mutually exclusive with users deleting images in swift, etc? why can't glance pull information from swift?
15:55:33 <notmyname> markwash: hmm...actually I was typing that answer thinking about the ABAC, not the "hidden" containers
15:55:34 <mclaren> markwash's had real world cred :-)
15:55:49 <dolphm> notmyname: are hidden containers a thing?
15:55:53 <notmyname> dolphm: no
15:56:19 <markwash> notmyname: well, blocking "list objects" for that container may be sufficient
15:56:36 <notmyname> markwash: isn't that essentially about tying stuff together at the identity/billing level? what I'm not clear on is why that requires special features in swift
15:56:51 <markwash> notmyname: it is, and that's how our other approach works
15:57:01 <iccha> markwash: this sounds like policies at container level :)
15:57:03 <markwash> however, we really like just reusing the users credentials when they contact us
15:57:04 <notmyname> ie why not have a separate glance account?
15:57:22 <mclaren> notmyname: right, 'something' in keystone may help us...
15:57:33 <dolphm> mclaren: i'm not sure what that would be :(
15:57:36 <markwash> notmyname: essentially, we don't want to store a set of credentials that has super-ultra-mega access to all the images in the cloud
15:57:46 <dolphm> it seems like the blocker here is glance being unwilling/unable to read from swift
15:57:50 <mclaren> dolphm: guru status revoked :-P
15:58:32 <notmyname> markwash: ok (unlike an admin token to keystone? /troll)
15:58:39 <dolphm> markwash: ++
15:58:43 <kgriffs> fwiw, we ran into a similar issue at Rackspace when creating Cloud Backup
15:59:00 <kgriffs> it is a big security risk to dump everyone's data together under one ueber account
15:59:03 <markwash> notmyname: troll point taken :-)
15:59:04 <iccha> dolphm: here is a usecase. there is no way to maintain the integrity of snapshots if we allow users to directly access and manipulate their images in swift
15:59:20 <dolphm> notmyname: <troll acknowledged>
15:59:28 <rosmaita> kgriffs: tell us more
15:59:41 <kgriffs> but, on the other hand, we don't want users messing around with their backup sets directly
15:59:53 <markwash> dolphm: hmm, I'm not sure I understand what you mean about "not being willing to read from swift" ?
15:59:59 <dolphm> kgriffs: what's was the solution?!
16:00:03 <kgriffs> and it adds a lot of noise when they are looking at their account in the control panel or whatever
16:00:46 <dolphm> markwash: a bunch of the points in that pdf would be resolved by understanding that glance's knowledge of image states is stale, and pulling the latest data from swift as necessary
16:00:50 <kgriffs> dolphm: we don't have a great one; just naming the backup containers something that makes it obvious what they are
16:00:56 <markwash> dolphm: ah okay I see
16:01:03 <dolphm> kgriffs: that's kind of what i was thinking here
16:01:08 <kgriffs> and putting a 0-byte file in there that says "do not delete this container!"
16:01:15 <dolphm> glance-image-{uuid} or something
16:01:21 <dolphm> as the container name
16:01:51 <iccha> hmmm but that way we cannot control downloads directly
16:01:53 <markwash> mclaren: I'd like to put my ptl hat on for the last 5 minutes of this meeting, is it that time?
16:01:59 <mclaren> sure
16:02:03 <kgriffs> tbh, if there just a way to not make those containers show up when listing their account, that would be great
16:02:12 <kgriffs> sorry, make them *not* show up
16:02:30 <markwash> <ptl> I think the passwordless approach shows a lot of promise, but still needs a lot more thought
16:02:57 <markwash> if we can't quite figure it out, I still think we should make progress by moving the swift "single-tenant" creds into config from the db
16:03:20 <markwash> in that case, we're cutting out a significant opportunity to attack those creds
16:03:38 <markwash> in my view, the most significant opportunity
16:03:45 <markwash> </ptl>
16:03:46 <markwash> phew
16:03:49 <iccha> markwash: are you concerned about storing password(encrypted) on disk as well?
16:04:02 <markwash> iccha: somewhat, but not nearly as much as storing it in DB
16:04:24 <mclaren> how about a new location format to denote read creds from file?
16:04:42 <mclaren> something like https+swift://:@xxx.com
16:04:56 <mclaren> I suspect that may be illegal right now
16:05:00 <markwash> mclaren: yeah, then I think we could make an online migration tool for it that would update a row at a time
16:05:16 <mclaren> we'd have to be careful with set image location -- to forbid that
16:05:55 <notmyname> I gotta run, but if you have more stuff you need to discuss about swift, please ask in #openstack-swift
16:05:56 <markwash> mclaren: I like that way of thinking about it actually
16:05:59 <markwash> "read from file"
16:06:06 <mclaren> notmyname: thanks
16:06:15 <markwash> that puts a logical cap around how long those creds exist in the code
16:06:27 <markwash> so opportunities to accidentally log them are much less
16:06:38 <markwash> though moving creds out of the location is the biggest win there too
16:06:47 <markwash> imnsho
16:07:24 <iccha> there  is patch with exists to substute it with hiddenuser or some other value right now
16:07:32 <mclaren> ok, thanks to everyone for their time. Let us know if you come up with some genious way to add consistency to the multi-tenant backend
16:08:35 <markwash> should I bang the gavel?
16:08:46 <mclaren> yup, thanks markwash
16:08:50 <markwash> #endmeeting