16:00:59 #startmeeting oslo 16:01:00 Meeting started Mon Nov 7 16:00:59 2016 UTC and is due to finish in 60 minutes. The chair is harlowja_at_home. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:01:00 thanks 16:01:02 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:01:03 Thanks. 16:01:04 thx ihrachys :) 16:01:04 The meeting name has been set to 'oslo' 16:01:10 o/ 16:01:14 o/ 16:01:18 courtesy ping for amotoki, amrith, bknudson, bnemec, dansmith, dhellmann, dims 16:01:20 thanks 16:01:23 o/ 16:01:30 ./ 16:01:30 courtesy ping for dougwig, e0ne, flaper87, garyk, gcb, GheRivero, haypo 16:01:30 courtesy ping for HenryG, jd__, jecarey, johnsom, jungleboyj, kgiusti, kragniz 16:01:30 courtesy ping for lifeless, lintan, lxsli, Nakato, ozamiatin, rbradfor, redrobot 16:01:30 courtesy ping for rpodolyaka, sergmelikyan, sileht, spamaps, sreshetnyak, sreshetnyak, stevemar 16:01:30 courtesy ping for therve, thinrichs, toabctl, viktors, zhiyan, zzzeek 16:01:30 hola! 16:01:37 o/ 16:01:40 hello harlowja_at_home 16:01:45 (maybe we should do the whole meeting in spanish?) 16:01:55 hi 16:01:57 Hola 16:02:01 ha 16:02:12 lol, I can't read spanish 16:02:19 buenos dias 16:02:28 como estas 16:02:34 Yo no hablo espaƱol 16:02:41 gcb, me either :-P 16:02:55 hey y'all 16:03:10 Pero puedo usar google translate 16:03:10 #topic Red flags for/from liaisons 16:03:23 nothing from trove 16:03:26 o/ 16:03:27 so its been a little while, any new red flags from folks :) 16:03:28 o? 16:03:30 o/ 16:03:39 johnsom, do u still have any red flag? 16:03:40 nova have one http://logs.openstack.org/periodic/periodic-nova-py27-with-oslo-master/6235dad/testr_results.html.gz 16:03:44 Nothing to report here 16:03:46 no red flags from keystone tha tI know of. 16:03:58 Nope, the global change cleared up our issue 16:03:58 I didn't have time to dig today 16:04:15 gcb, thanks, i'll see if i can look through what might of caused that 16:04:39 harlowja_at_home, thanks 16:05:45 periodic jobs getting stuff done! 16:05:51 http://status.openstack.org/openstack-health/#/?groupKey=build_name&resolutionKey=hour&searchProject=-with-oslo doesn't look to super either, if anyone wants to help investigate why some of those are acting up 16:06:15 i think all jobs are showing up there again, they weren't showing up some time ago, lol 16:06:42 though seems to be similar to '"testtools.matchers._impl.MismatchError: '200 OK' != '401 Unauthorized'"' 16:06:54 which might be something oslo.middleware 16:07:09 so cinder might be affected by the same gcb 16:07:18 Ours have cleared up after the global change, fyi 16:07:54 i'll have to ask the glance folks whats up with http://logs.openstack.org/periodic/periodic-glance-py27-with-oslo-master/b4a278f/console.html#_2016-11-07_06_19_23_225309 (looks unrelated) 16:08:07 ir http://logs.openstack.org/periodic/periodic-glance-py27-with-oslo-master/b4a278f/console.html#_2016-11-07_06_19_21_594028 16:08:27 johnsom, great :) 16:08:27 glance seems to be acting up, ha 16:09:05 anyone interested in looking into those, feel free, otherwise I'll try to also :) 16:09:26 #topic Releases of the week 16:09:41 so just wanted to see what people would like released (after the above issues are looked into) 16:09:48 anything specific from folks? 16:11:06 going once 16:11:43 ok, i'll just get our a batch release then after those current periodic issues get looked into :) 16:12:13 #topic Summit recaps 16:12:43 so thanks all of you who showed up in BCN :) 16:12:58 for those that didn't i've been slowly writing some of the things to do to the ML 16:13:02 #link https://wiki.openstack.org/wiki/Design_Summit/Ocata/Etherpads#Oslo 16:13:10 ^ sessions and such we held 16:13:48 http://lists.openstack.org/pipermail/openstack-dev/2016-November/106609.html (one of the ML recaps) 16:14:02 and http://lists.openstack.org/pipermail/openstack-dev/2016-November/106680.html 16:14:39 i'll send out a couple more i think (of things to do for oslo) but for now I think amrith has one such topic (related to summit also) 16:14:45 amrith, you ready??!? 16:14:54 yes, was just warming up my voice 16:15:13 #topic Oslo.messaging anddddd amrith 16:15:18 ok all yours :) 16:15:24 the topic I have relates to oslo.messaging and adding a level of message security. 16:15:38 I summarized some thoughts in an etherpad 16:15:40 #link https://etherpad.openstack.org/p/secure-oslo-messaging 16:15:51 here is the TLDR 16:15:53 for this meeting 16:15:54 what am I looking for (at this stage) 16:15:54 - do we believe that it is still interesting to oslo to support message signing and verification for oslo_messaging/rpc 16:15:54 - are we ok with an approach where oslo_messaging/rpc provides a framework for consumers to define the signer and verifier, and not directly perform the signing and verification 16:15:56 If the answers to the above two questions are both YES, then I think we have a set of things that we can discuss. If the answer to either is NO, then I think we have to address those issues first. 16:16:08 * amrith shuts up 16:16:30 (for now, for those of you who thought this was permanent; sorry to dissapoint you :)) 16:16:31 wasn't some work already done on this topic? 16:16:38 dhellmann, yes 16:16:41 a lot of work was done 16:16:49 there was a whole kite project done by redhat. 16:16:51 the primary difference was that in past iterations (referenced in the etherpad) 16:17:01 which was for the distribution of the keys 16:17:04 the burden was placed on oslo.messaging to do the signing 16:17:06 and key management 16:17:29 a fundamental difference with my proposal is that I'm looking at oslo.messaging to be merely a framework that allos the consumer to do the signing and verification 16:17:43 allows the consumer to register a callback that will be invoked when the full message is ready 16:17:50 and will invoke a callback on each message received 16:17:54 and what's this ultimately going to be used for? 16:17:59 this makes the implementation (in oslo) much simpler. 16:18:08 dhellmann, I summarized it in the therpad 16:18:09 etherpad 16:18:12 but the basic idea is this 16:18:17 for projects with service vm's 16:18:24 there are transport credentials on the service vm 16:18:30 if the service vm is compromised 16:18:39 then the transport credentials are compromised 16:18:48 johnsom, may also be useful for octavia (if this converges) - fyi 16:18:51 ok, I think my position is unchanged that the service vms shouldn't be connecting to the bus anyway 16:18:57 and one could conceivably have a person generate bogus messages 16:19:21 dhellmann, I agree 16:19:32 dhellmann, they do today. several projects have that construct. in any event, hypervisor escapes are also known to occur 16:19:44 and that would mean that a hypervisor escape would put a nova compute at risk 16:19:46 harlowja_at_home Hmm, not sure. As I mentioned in the session, we use two-way SSL with our service VM agents. 16:19:51 the service vm should have a rest api and use keystone to protect it 16:20:13 dhellmann, where would the service vm get keystone credentials? 16:20:19 johnsom, oh ya, nearly forgot :) 16:20:28 at the end of the day, someone has to store credentials on the service vm 16:20:32 credentials of some kind 16:20:34 amrith : it shouldn't need them if all of the calls are incoming 16:20:46 but they aren't 16:20:49 Basically what dhellmann said, but without the keystone part. 16:20:51 some are merely status reporting 16:21:08 I thought we had this discussion a long time ago and agreed that Zaqar (then Marconi, I think?) was the right way for service vms to do messaging. 16:21:09 even incoming calls need the some service user credential to validate tokens 16:21:33 bnemec, the problem is the same with any mechanism; so long as a transport has credentials, someone needs to generate them 16:21:33 bknudson: ah, good point 16:21:42 or store them 16:21:52 but that's where PKI makes things better 16:22:07 bknudson : though if the only permission that user has is to check the validity of other credentials, it's low risk 16:22:10 it allows the client to at least identify itself in a manner that can be verified by the control plane 16:22:19 But the servicevm can have it's own credential and use trust to authentic with the user one 16:22:19 and the blast crater can be limited to that particular service vm 16:22:24 yes, we could give that user only access to validate tokens 16:22:26 (like heat does) 16:22:46 also, given a token it should be able to validate itself. 16:23:03 so, what y'all are proposing is a solution where service vm's don't talk to the messaging infrastructure 16:23:04 so maybe no service user would be necessary. It just is now because that's how auth_token works. 16:23:13 first let me point out that the control plane is NOT the openstack control plane 16:23:17 it is the trove cntrol plane 16:23:21 amrith : right. instructions should be incoming. "tell me your status" is an instruction. 16:23:34 which is by choice not the same as the openstack control plane 16:24:08 therefore all we're looking for is a mechanism to make that control plane safer. 16:24:20 sorry dhellmann that is polling 16:24:25 yes, it is 16:24:35 but when a db fails, we want the guest to tell the control plane 16:24:44 how frequently would you like to poll? 16:25:19 we already register for notifications from db's to get informed of state changes 16:25:29 how about an unprivileged web hook with no data that just means "you need to ask me my status" 16:25:30 having 'tell me your status' be polling defeats the whole purpose. 16:25:39 * harlowja_at_home starts to wish we had something like etcd (so that it could be used to avoid said polling) 16:26:19 harlowja_at_home : you still have to secure that. the whole point of saying connections are only incoming is to have the rest of the infrastructure drive the service vm, and to treat it as hostile and insecure until proven otherwise 16:26:21 https://coreos.com/etcd/docs/latest/api.html#waiting-for-a-change (such a thing could be used for this) 16:26:31 amrith : you may need to have this discussion with SpamapS' new arch wg 16:26:37 dhellmann +1 16:26:41 dhellmann, all of these are approaches, and by no means the only ones, to avoid the problem. is the feeling that (a) projects shouldn't be using service vm's that connect to the control plane, and (b) oslo isn't interested in message signing for RPC? 16:27:09 my memory is that adding signing to rpc caused serious performance issues, so it was turned off 16:27:15 (c) undecided 16:27:37 dhellmann, it would cause performance issues if we turned it on for everything and everybody 16:27:43 if it's only for some messages, that may mitigate it, but I don't think we ever envisioned the messaging lib being used for talking to untrusted nodes 16:27:54 but we are able to turn it on for those who care about it. 16:28:24 so, if I understand correctly, octavia has full blown client authenticaed ssl 16:28:32 other projects have their own solutions 16:28:42 johnsom, ^ 16:29:13 * harlowja_at_home starting to feel we may need to go talk with the arch-wg, then get some consensus on the ML (or at a future meeting)? 16:29:14 Correct, we issue unique certs to each service VM at boot and manage renewal 16:29:36 we would be creating keypairs, not going full blown ssl 16:31:52 so, what does the team feel? to my two questions ... 16:31:52 - do we believe that it is still interesting to oslo to support message signing and verification for oslo_messaging/rpc 16:31:52 - are we ok with an approach where oslo_messaging/rpc provides a framework for consumers to define the signer and verifier, and not directly perform the signing and verification 16:32:03 if the answers are no and no, that's fine. just let me know 16:33:03 i'm personally ok with it, i don't think the 2 things above need to be tied into whether the choice is right for trove to be doing this (or if it should do this or...) 16:33:19 but my guess is others believe differently :) 16:34:44 I'd be afraid that adding the feature would encourage more anti-patterns like this, but I don't object strongly enough to -2 it 16:34:48 So, in the answer of an answer of 'yes' I'm going to assume that the answer is 'no' and look for other alternatives. if that's not the case, please let me know. 16:34:58 personaly I don't see the point to add hook when we can do the same without hook 16:35:08 that's also a good point 16:35:14 sileht, that's not true; it is not the same with a decorator 16:35:28 upgrade and cross release issues are MUCH easier if this is handled at a level below the project. 16:35:52 sileht, do you mean by not using oslo.messaging (and switching to rest?) 16:35:52 if it is up to the project, with decorators, then the case of a client without the code to handle a signature will fail 16:35:57 because of the extra parameter. 16:36:24 no harlowja_at_home he's talking about using a decorator as I had illustrated in the ML 16:36:40 harlowja_at_home, about https://review.openstack.org/#/c/394389/ 16:36:43 the issue with that is that to handle it in the consumer (trove) signature will have to be a parameter in kwargs 16:37:01 and that means that a client without the code to handle it would barf at the extra parameter 16:37:06 ah, haven't seen that one yet, created this morning, ha 16:37:23 if you aren't doing versioning of that api, you already have a problem trying to change it in any way 16:37:35 dhellmann, this isn't about versioning an API 16:37:45 this is about allowing coexistance 16:37:57 and allowing a version that has no support understand a message that has an extra parameter 16:38:14 amrith, that what oslo.messaging version is for 16:38:17 and this isn't an API, this is every RPC call. 16:38:54 the decorator version is an API change while the hooks version is not 16:39:53 sileht, your @verifier is the thing that pops the signature 16:40:08 what happens to a server with no @verifier? 16:40:30 are you suggesting that we must use version caps on a per guest basis? 16:40:36 amrith: you have to use version 16:40:38 in the service vm context, that is infeasible 16:40:56 you cannot require an operator to upgrade all guest database instances at the same time in order to upgrade the control plane 16:41:21 are you not going to be signing the messages sent from the control plane to the service vm? 16:41:29 yes 16:41:31 absolutely 16:41:36 that's the prime attack vector 16:41:51 one compromised service vm telling another one to go shut itself down, for example. 16:41:55 so how are you going to handle that upgrade case? 16:42:19 if the code is in oslo.messaging, then a guest who doesn't have the code will receive a signature and the old oslo.messaging will just ignore it. 16:42:48 that is a luxury of having the verification (or at least the dispatch of verification) being in dispatch 16:42:55 and where the signature is not an argument 16:43:12 if the code is in the consumer (trove) then all I can do is insert a signature as an argument to the API call 16:43:13 right, so that's why I brought up API versioning. Because you have this one case where you can put the thing into the library and "hide" the change, but you won't be able to do that for all cases. 16:44:20 so we can handle this in the consumer by adding a signature parameter to all API calls 16:44:25 literally all API calls 16:44:26 amrith dhellmann sileht do the three (and more?) want to continue on this perhaps in #openstack-oslo seems like we may not reach consensus just yet :) 16:44:35 harlowja_at_home : good idea 16:44:39 and in the version that supports signatures, remove it. 16:44:42 seems a bit much 16:45:35 harlowja_at_home, it seems like we are at a decision 16:45:49 and the answer seems to be that consumer projects like trove should handle this with api versioning. 16:45:52 ok, whats the ruling :-P 16:45:55 and hooks/decorators. 16:46:05 I'm happy with that decision, just let me know. 16:46:13 or not using rpc at all :-) 16:46:21 dhellmann: ++ 16:46:24 or not using rpc calls for this. 16:46:45 which is a much bigger project, but sure, it is a feasible solution. 16:47:14 fyi dhellmann this is not restricted to projects with service vm's. 16:47:20 okie dokie, amrith do u mind sending out a conclusion to the ML, to at least distribute this decision? 16:47:28 with a hypervisor escape, one can compromise the openstack control plane 16:47:51 amrith : sure 16:48:18 harlowja_at_home, sure thing 16:48:20 will do that. 16:48:23 amrith, thanks :) 16:48:30 alright next topic 16:48:41 #topic OpenStack goals 16:48:47 dhellmann, the mic is yours :) 16:48:51 ah, thanks 16:49:07 as you all know, the goal this cycle is to move off of incubated code to the libraries 16:49:14 #link https://governance.openstack.org/goals/ocata/remove-incubated-oslo-code.html 16:49:17 we have a relatively small number of projects with work to do on that regard 16:49:36 projects have already started, and the patches should all be visible on this report 16:49:41 #link https://review.openstack.org/#/q/topic:goal-remove-incubated-oslo-code+is:open 16:49:47 dhellmann, do these eventually show up in https://governance.openstack.org/goals/ocata/remove-incubated-oslo-code.html#project-teams ? 16:49:56 under 'Completion Artifacts'? 16:50:02 I, and I'm sure the other project folks, would appreciate help with reviews of those changes 16:50:22 harlowja_at_home : usually the artifacts will be links to bugs or blueprints, though some projects have added links to the patches 16:50:32 k 16:51:06 so, when you have time, it would help the community a bit to do some of those reviews in case folks get stuck on transitions in any way 16:51:29 harlowja : that's all I had, thanks for giving me the slot on the agenda 16:51:32 sounds good to me, the last of the last of the last of that cleanup :) 16:51:54 cool, thanks dhellmann 16:52:08 #topic Open discussion 16:52:20 so just a few other tiny links to share 16:52:36 #link https://www.openstack.org/ptg/ 16:52:43 and #link https://releases.openstack.org/ocata/schedule.html 16:52:53 keep those in mind (for folks reviewing, doing changes...) 16:53:14 Jan 16-20 isn't that far away :) 16:53:25 (Final release for non-client libraries) 16:53:41 especially if people go on vacations and ... 16:53:51 in fact, damn that's really not that far away, lol 16:54:39 * harlowja_at_home that's all i had 16:54:40 this release will consist mostly of requirements changes and switching asserts 16:54:46 :) 16:54:48 lol 16:55:17 and cleanups of some sorta 16:55:20 *sort 16:56:18 alright, guess we can end 4 minutes early, since doesn't seem like anyone else has any open discussions :) 16:56:39 thanks amrith dhellmann sileht and others for the insightful discussion around messaging :) 16:57:03 as always #openstack-oslo for more fun! :) 16:57:19 thx harlowja_at_home 16:57:30 oh, one question if we are in open discussion 16:57:44 3 minutes :-P 16:57:46 if the world comes to an end tomorrow, does oslo have any special plans :) 16:57:48 ha 16:58:07 move everyone to canada? 16:58:13 lol 16:58:20 i'm thinking i might like vancouver more and more 16:58:25 unless canada builds a wall... 16:58:26 ha 16:58:28 sure, just follow me y'all ;) 16:59:13 oh, ya, and get out and vote (plllease, especially those in battleground states) :) 16:59:22 (for those in US) 16:59:29 everyone should ... 16:59:41 "vote early, vote often" 17:00:08 :) 17:00:10 #endmeeting