14:00:04 #startmeeting neutron lbaas 14:00:05 Meeting started Thu Sep 11 14:00:04 2014 UTC and is due to finish in 60 minutes. The chair is dougwig. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:00:06 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:00:09 The meeting name has been set to 'neutron_lbaas' 14:00:09 agenda: 14:00:11 #link https://wiki.openstack.org/wiki/Network/LBaaS#Meeting_11.09.2014 14:00:21 morning 14:00:24 #topic Announcements 14:00:26 morning 14:00:30 morning 14:00:36 o/ 14:00:46 \o/ 14:00:49 Morning! 14:00:49 i think we're in the middle of rc1. i don't think that has any lbaas stuff 14:00:49 morning team 14:00:51 #link #link https://launchpad.net/neutron/+milestone/juno-rc1 14:00:54 link link 14:00:55 #link https://launchpad.net/neutron/+milestone/juno-rc1 14:01:07 kilo specs are around the corner. 14:01:20 any other announcements? 14:01:45 moving 14:01:46 on 14:01:56 #topic Barbican integration -- Keystone Trusts, comments and concerns (rm_work) 14:01:58 rm_work? 14:02:02 yeah 14:02:08 the floor is yours 14:02:26 so, this was my action item from last week (actually to send this to the mailing list) but there were some … complications 14:02:42 so, this is the workflow I had hoped we could use: http://i.imgur.com/74BUbMf.png 14:03:05 I object to the word "hijacks" 14:03:11 Heh! 14:03:13 yes 14:03:15 well 14:03:23 i'm getting to WHY i use that term specifically 14:03:44 so, more granularly, it looks like this: http://goo.gl/X3kKIX 14:04:34 http://www.websequencediagrams.com/cgi-bin/cdraw?lz=dGl0bGUgTEJhYVMtS2V5c3RvbmUtQmFyYmljYW4gSW50ZXJhY3Rpb24gKGFzc3VtaW5nIGFuIEFzeW5jIEFwcGxpYW5jZSkKClVzZXItPgA7CDogR2V0IGEgVXNlciBUb2tlbgpub3RlIGxlZnQgb2YAEwU6IHVzaW5nAB8FbmFtZS9QYXNzd29yZAoAgQAJPgAiBlJldHVybgBBDgBqBgCBIgg6IENyZWF0ZSBhIFNlY3JldCAiQ2VydGlmaWNhdGUiAF8eAIEaBwCBbggAZw8ASgZJRDEAUSJQcml2YXRlIEtleQAuRTIAgUwaQ29udGFpbmVyICJNeSAAgSRIAE0JSUQAg0oHAIQVBQCCUwtUTFMgVGVybWluYXRlZCB 14:04:40 That'll take a minute to digest. 14:04:44 that doesn't have ad stuff 14:04:46 jorgem: the full link is unworkable for IRC <_< 14:04:52 i think my day has been settled 14:05:00 you actually only posted like 1/2 of it :P 14:05:06 wtg jorgem 14:05:16 somehow jorgem's link worked for me 14:05:16 err maybe 1/4 of it 14:05:18 i see the whole link…oh well 14:05:23 look at the diagram 14:05:27 it goes down to like 14:05:38 #link http://i.imgur.com/74BUbMf.png 14:05:40 1/8 of the length of the full one 14:05:44 anyway 14:05:48 #link http://goo.gl/X3kKIX 14:06:07 #link 14:06:07 http://www.websequencediagrams.com/cgi-bin/cdraw?lz=dGl0bGUgTEJhYVMtS2V5c3RvbmUtQmFyYmljYW4gSW50ZXJhY3Rpb24gKGFzc3VtaW5nIGFuIEFzeW5jIEFwcGxpYW5jZSkKClVzZXItPgA7CDogR2V0IGEgVXNlciBUb2tlbgpub3RlIGxlZnQgb2YAEwU6IHVzaW5nAB8FbmFtZS9QYXNzd29yZAoAgQAJPgAiBlJldHVybgBBDgBqBgCBIgg6IENyZWF0ZSBhIFNlY3JldCAiQ2VydGlmaWNhdGUiAF8eAIEaBwCBbggAZw8ASgZJRDEAUSJQcml2YXRlIEtleQAu 14:06:07 RTIAgUwaQ29udGFpbmVyICJNeSAAgSRIAE0JSUQAg0oHAIQVBQCCUwtUTFMgVGVybWluYXRlZCB 14:06:09 yeah is it ok if I give people a minute to digest that before I move on? 14:06:12 heh 14:06:14 yeah seriously 14:06:17 the link doesn't work in IRC :P 14:06:26 i object to the word "appliance" 14:06:35 heh 14:06:36 I did a copy link and was able to get that link up in my browser 14:06:38 blogan: I object to your face, but that's equally irrelevant 14:06:56 balles: it's not the WHOLE diagram 14:07:00 it cuts off 14:07:03 trust me :P 14:07:04 Haha 14:07:05 oh I see 14:07:11 lol 14:07:18 i blame jorgem for this 14:07:33 #link the last link is not useful 14:07:37 anyway 14:08:04 maybe people have managed to kind of look through that 14:08:49 so, the issue is with the kind of Keystone Tokens that will be coming in 14:08:51 I don't see anything wrong with this sequence at first glance. Note that it is early, and I'm not sure my usual snarkiness is awake. 14:09:10 you say that likes a bad thing 14:09:15 So, this is great from a usability standpoint, because it is completely transparent to the user 14:09:25 yep, I oike that 14:09:55 but (and this is the reason I used "hijack" in the first chart, on purpose) it really is a bit scary from a security perspective 14:10:16 and besides that, apparently Trusts cannot be created using a Token that is already a Trust Token 14:10:28 which is what HEAT and possibly Horizon will be using 14:10:42 so in this case instead of having the "user cred" we store the "Trust ID"? 14:10:46 or a Token which the user has specifically made lower privilege, which apparently is a thing people do? 14:10:51 samuelbercovici: yes 14:11:13 which then allows us to "Execute the Trust" with Keystone, which is very similar to getting a token to begin with 14:11:27 is storing "Trust ID" equally dangerous as storing password ? 14:11:32 rm_work, I know Trove is doing something similar 14:11:34 we use our own token to talk to keystone and pass the trust ID, and we get a Trust Token back 14:11:45 can "Trust ID" be hijacked? in other words if someone elase has the "trus id" can he then read the secured information? 14:12:02 vivek-ebay: well, the trust is only useful for our account, so they'd need the TrustID *and* a valid token for our service account 14:12:08 samuelbercovici: ^^ 14:12:25 the TrustID is linked specifically between two accounts 14:12:36 So it's slightly less dangerous than storing the user's password. But only slightly. 14:12:36 but Keystone *does* consider them "sensitive data" 14:12:55 well, what's the alternative? Storing the certs in our DB? 14:12:58 Is there any way around that? 14:13:01 jinx! 14:13:10 ic. so the hijacker needs to impersonat the LBaaS service user whic is supposed to be non-interactive? 14:13:12 if someone were to compromise our API box, yeah, they could get the password to our service account from config, and also access to our DB and then to the TrustIDs 14:13:25 is this the new way of doing cross-service stuff? i note that for using nova, there is just a user/pass in the clear in neutron.conf 14:13:31 I assume this is not specific to LBaaS and should be a common problem. What does Barbican guys suggest ? 14:13:47 well, I have seem some stuff about Postern 14:13:49 ahh, you answered my question one line before i asked it. 14:13:53 but it's kind of a chicken/egg problem 14:14:03 well, we can use encrypted config files 14:14:13 we can force users to enter pwd at startup... 14:14:23 that's am old /solved problem in CS 14:14:31 Eew and eew. 14:14:33 ;) 14:14:45 #link https://github.com/cloudkeep/postern 14:14:58 I think that's the right repo 14:15:06 but obviously… not ready 14:15:49 so anyway, we've got two real issues here 14:16:12 1) This whole thing is a security issue (almost just kicking the rock down the road) 14:16:36 2) We may not actually be able to do Trust creation on our side -- it may be NECESSARY to have the user do it themselves (though we can provide them a template) 14:16:45 * TrevorV late but present :( 14:16:57 2) Worries me more than 1) 14:17:04 2 is a terrible user experience 14:17:06 I don't know if #1 is solvable at the moment, and this is still better than just plaintext certs in the DB 14:17:17 rm_work +1 14:17:19 so yes, #2 here is my real discussion topic 14:17:45 can horizon do #2? 14:17:53 so in other words, the user authorizes lbaas to be allowed access to his secrets, right? 14:18:07 Well, I need to get in touch with someone from Horizon -- I asked in their IRC channel yesterday but never got a response 14:18:09 It looks like the use has to do something with rehards to trusts. The doc says: In both cases, the users authorizes it at setup time to perform this action any time in the future, long after the token is expired. 14:18:12 samuelbercovici: yes 14:18:31 rm_work, ^^^ but I am sure you know that already 14:18:36 balles: right, it allows us to essentially get a new token 14:18:38 on our own 14:18:52 but the issue is that Creating Trusts requires a "full privilege" user token 14:19:12 and many of the use-cases will have some other service already using a lower privilege Trust Token 14:19:19 so let's split that problem 14:19:24 so we'll be given a token that doesn't have the ability to create a Trust 14:19:48 even if horizon exposes it, if horizon uses a trust then horizon would not be able to create another trust for lbaas 14:20:05 blogan: yeah, and i am not 100% clear on that 14:20:07 I am ok with the command line being cumbersome 14:20:10 but i don't know how else horizon would work? 14:20:26 if we think 99% of our users will come from the API, then … whatever? we just ignore the issue and tell them to use a real token 14:20:28 but Horizon should be more like press an Accept button 14:20:48 if we think a ton of people will come in through Horizon (which is my theory) then this may still be a problem 14:20:55 is this a case of openstack deciding that something "should" be a certain way, telling people to do it that way, even though it's really used everywhere in an entirely different fashion, and is nowhere near ready for the "new and correct" way? 14:21:05 rm_work: is there a way that the uses will just authorize lbaas service user access to the private info? 14:21:07 rm_work, ok make sense. 14:21:08 dougwig: I thought so at first 14:21:27 dougwig: but the Keystone guys actually showed me that in a lot of cases my plan will actually fail 14:21:32 actualy since lbaas runs inside neutron, it may be the neutron service user 14:21:33 dougwig: HEAT was the certain example 14:21:33 ? 14:21:40 samuelbercovici: correct 14:21:45 for now it is the Neutron service user 14:22:21 samuelbercovici: so, the way to just "authorize our user" on their side is for them to hit Keystone and create the Trust themselves (using a template we provide) which is the officially recommended approach 14:22:29 but is what everyone is saying is an unacceptable user experience 14:22:40 (I don't disagree that it is quite cumbersome for the user) 14:22:53 so this sounds like it'll be something that we need horizon solve (and which a lot of projects will need). 14:22:57 I am ok for it to becumbersome when using the CLI 14:22:57 yes 14:23:02 say horizon uses a full privelaged token, and it makes setting up trusts simple for a user, wouldn't horizon need some way to know which service account to set a trust up with? 14:23:18 So, I don't know that this is something we have the knowledge to really answer right now, but I want people to be aware of what's going on 14:23:37 blogan: also yes, but we could code that in as part of our plugin maybe? 14:23:40 ok. so the user creates the cer, key and container and afther that, runs the "template...." whicg grants access to neutron services. did you talk with VPNaaS guys on this? 14:23:55 rm_work: actually sounds like an admin api type of operation 14:24:02 samuelbercovici: not yet, I only got this far in the process of understanding this yesterday afternoon 14:24:27 blogan: erg. yeah 14:24:52 so, the *original* original plan was to just straight up have the Neutron service account be a "god account" 14:25:01 and we could just read data willy-nilly 14:25:13 but that's actually scary enough that i'd never want to see it in production 14:25:13 and why did that fail? 14:25:16 rm_work: this is probably a bad idea, but what is stopping us from just reusing the token the user passes to lbaas and talking to barbican with that? other than the token expiring and we have no way to talk to barbican later 14:25:26 theoreticaly speaking, the LBaaS horizon module, may have a section used to upload certificates as part of the lb creation, in this case, it may be easier to implement this "authorization" part in there.... 14:25:31 blogan: um… the thing you just said is the problem 14:25:42 rm_work, how is it today? Is neutron admin only admin for Neutron? 14:25:51 balles: yes, I believe so 14:25:58 okay so thats the only problem 14:25:59 and possibly on specific other services 14:26:13 blogan: yes. 14:26:57 so the OTHER option is to go all the way back to one of the original alternatives, which is: hijack user token, read data from barbican, store data on the neutron-service-account 14:27:02 (back in Barbican) 14:27:05 the shadow-copy thing 14:27:17 that's a lot to digest at 8am. :) i assume you will be following up with the horizon guys and coming up with a plan? 14:27:18 that would stop us from needed to ever retain access to the user's data 14:27:23 dougwig: that's the idea 14:27:34 rm_work: this is worse. 14:27:40 samuelbercovici: i agree 14:27:58 if we wanted to store such data in neutron, we would not be jumping through such hoops 14:28:09 samuelbercovici: i mean, we'd be storing it in Barbican again 14:28:16 samuelbercovici: just, using our service account 14:28:31 but yeah, it's not ideal either 14:28:32 Having a PKI is exactly the reason we don't ewant copies 14:28:35 I don't like the idea of duplicating the certs 14:28:43 +1 14:28:49 I hope people realize I'm still not advocating this 14:28:51 :P 14:28:55 anyway 14:28:58 yep, i do not like this 14:29:01 quit advocating it 14:29:07 I just wanted to get people on the same page 14:29:11 ok, let's circle back to this next week (or the ML) when you get more info on the user experience 14:29:17 I don't think we can solve this with discussion right now 14:29:28 nope 14:29:42 but if anyone thinks of anything… let me know :) 14:30:01 keep this tumbling in your subconscious :) 14:30:03 and to me it has the same problem as geting authorized to acces the data. if the attack vectos is by jeoperdizing the neutron service account, than both solution will have similar cosequences 14:30:09 rm_work, I appreciate you bringing this up for discussions. Has this been on the ML already? I am behind on my emails 14:30:29 balles: no, I did't get it "finished" so I never sent it out 14:30:34 I may send our a brief summary of this 14:30:36 today 14:30:46 even though it's still not a finished proposal 14:31:04 >> I assume this is not specific to LBaaS and should be a common problem. What does Barbican guys suggest ? 14:31:07 Ok I am going to ask Guang from our keystone teamto chime in on this thread. He had some ideas around Truts 14:31:23 vivek-ebay: the only idea they had so far was mentioning the Postern project 14:31:30 balles: ok, thanks 14:31:37 let's move on, and continue this next week or on the ML. good stuff, rm_work 14:31:44 vivek-ebay, I agree with you especially the other advanced services 14:31:48 dougwig: yeah, didn't mean to take *half the meeting* :P 14:31:50 does babican has similar notion of also including the node from which the information is requested (same as ssh)? in this case, you can only get the information using neutron user but also only if it is from the neutron(s) api nodes 14:32:16 rm_work: it's good, there wasn't anything else of substance on the agenda. :) 14:32:20 samuelbercovici: it might be possible to do this from their RBAC policy? MAYBE? But it is not a native concept 14:32:54 ok, and now for a less contentious topic... 14:32:55 #topic Incubator update 14:33:06 samuelbercovici: there is something coming down the pipe for more granularly scoped keystone trusts, but it isn't here yet 14:33:09 mestery, anything new to report? 14:33:10 samuelbercovici, In our deployment we have certs all over the place. I believe a lot of our security is added when we deploy the services as part of our cloud 14:33:37 the same with username/password It is all encrypted 14:33:56 dougwig: Only that markmcclain and I hope to get it rolling this week again. We were focusing on FFE BPs, but the gate queue is now at 30+ hours so ... :( 14:34:13 dougwig, I had the same question :-) I am assuming you mean around the incubator thingy 14:34:18 mestery: do you guys operate on Valve Time? :P 14:34:30 rm_work: Something like that I think ;) 14:34:42 (for reference: https://developer.valvesoftware.com/wiki/Valve_Time ) 14:34:46 mestery: :P 14:34:46 its a combination of valve and blizzard time 14:34:55 lols at rm_work 14:34:58 i guess it was "Destiny" ;-) 14:35:12 heh 14:35:12 Honestly, I apologize for how long this has all taken, we're doing our best here. 14:35:22 mestery: yeah, all in good humor :) 14:35:23 mestery, What's FFE? 14:35:25 Yesterday we spent a few hours looking at a huge spike in neutron full job failures 14:35:29 Which ended up being a nova patch 14:35:33 But the finger was pointed at neutron first :) 14:35:38 FFE == Feature Freeze Exception 14:35:41 that's always fun <_< 14:35:42 BPs which got on are due by tomorrow 14:35:44 rm_work: ++ 14:36:08 look at the link that opened the meeting for a list of rc1/ffe blueprints that need reviewer eyeballs. 14:36:34 dougwig: ++ and thanks! 14:37:11 #topic Open Discussion 14:37:33 mestery, it is clear to me that you guys need to delegate "dev-core" power to the incubated Neutron projects 14:37:34 otherwise you will continue to be bottlenecks 14:37:35 anything else to discuss today? 14:37:48 balles: :) 14:38:23 I think he agreed with you ;) 14:38:59 there's been a long discussion from the nova team on the ML about that, and kyle has made a few appearances. it's worth reading. 14:39:25 yep 14:39:30 dougwig, thanks for the pointer. I will read it after the IRC is done 14:39:58 #link http://lists.openstack.org/pipermail/openstack-dev/2014-September/044872.html 14:40:05 on another matter: summit Paris talks. Are all the people patricipating in LBaaS v2 talk planning to attend? 14:40:24 are voting results out ? 14:40:28 yes 14:40:33 samuelbercovici: as of now, yes, but could be subject to change 14:40:41 samuelbercovici: yes 14:40:47 vivek-ebay: emails were sent out with results 14:40:51 and its totally out of my control if it does cahnge 14:41:04 can we please summarize lbaas selections ? 14:41:12 blogan: if you're not there, we will blame all warts on you and rackspace. 14:41:16 which topics got selected 14:41:18 samuelbercovici: I'll definitely be there. 14:41:22 blogan, This is great news! you guys will be at the summit 14:41:53 balles: hopefully, yes, but we will see what the next month or so brings 14:41:58 #link https://openstacksummitnovember2014paris.sched.org/ 14:41:59 The Octavia talk didn't get selected :-( 14:42:13 We at ebay missed submitting topics for talks due to internal firefight. We had some great stuff to showcase. 14:42:19 balles: not all of us but some of us, though some things have come up that lead me to believe all or most that were going may not be now, but I won't know until tomorrow probably 14:42:26 vivek-ebay: :( 14:42:42 blogan, Crossing my fingers for you guys 14:42:45 blogan: you think tomorrow? :P 14:42:46 balles: i would hav ebeen surprised if it did, no one knows what it is 14:43:10 we got miss by few days. Requested exceptions from core...didn't got :( 14:43:13 rm_work: yes, through the grapevine 14:43:15 blogan, :) The HP reviewers commented that ti was more a design session topic 14:43:22 blogan / balles: yeah, it's sad no one knows what it is so it doesn't get voted on, so no talk, so no one knows what it is :P 14:43:31 vivek-ebay: That's too bad, eh. 14:43:54 time to put that HP marketing power behind Octavia, super bowl commercials 14:44:02 vivek-ebay: is there still the notion of "flash talks"? 14:44:04 Did we get more than one talk? Sam & co. 's LBaaS V2. Anything else 14:44:13 yes, 10 mins 14:44:17 HP Hellion: now with the power of Octavia! *queue roman gods B footage* 14:44:19 not enough... 14:44:20 balles: i dont believe so 14:44:38 I also do: https://openstacksummitnovember2014paris.sched.org/event/2b85b682ccaf3a5961e463b61e2403f8#.VBG1TvmSzY8 14:45:15 ah interesting, ill attend that one if I do go 14:46:04 If I go, I'll be doing a lightning talk on Barbican service integration… assuming it eventually works out :P 14:46:17 rm_work That is funny 14:46:30 rm_work: I have a feeling that talk would need more than 10 mins 14:46:32 rm_work: eventualy might weight a kilo 14:46:32 anything else anyone wants to discuss? 14:46:36 lol 14:47:04 we need Octavia stickers and T-Shirts :-) 14:47:10 xgerman: ah, yes 14:47:19 i would like to discuss...bears 14:47:22 we can have a little Amphora as part of our logo 14:47:32 blogan: black or grizzly? 14:47:36 * blogan vomits into an amphora 14:47:40 samuelbercovici: lol 14:47:43 maybe an Amphora on a scale? :P 14:47:52 with bears to make blogan happy 14:48:08 if bears came along and smashed a room full of amphorae, that'd be great 14:48:08 alright, discussion has sufficiently devolved ^_^ 14:48:12 a great marketing ploy as well 14:48:23 i'm sensing an imminent endmeeting.... :) 14:48:27 xgerman: Totally! 14:48:32 this is highly relevant 14:48:46 * rm_work gets out his Photoshop hat 14:48:48 end it dougwig 14:48:55 #endmeeting