16:00:59 <harlowja_at_home> #startmeeting oslo
16:01:00 <openstack> Meeting started Mon Nov  7 16:00:59 2016 UTC and is due to finish in 60 minutes.  The chair is harlowja_at_home. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:01:00 <electrocucaracha> thanks
16:01:02 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:01:03 <sshank> Thanks.
16:01:04 <harlowja_at_home> thx ihrachys  :)
16:01:04 <openstack> The meeting name has been set to 'oslo'
16:01:10 <gcb> o/
16:01:14 <rpodolyaka> o/
16:01:18 <harlowja_at_home> courtesy ping for amotoki, amrith, bknudson, bnemec, dansmith, dhellmann, dims
16:01:20 <dasanind_> thanks
16:01:23 <johnsom> o/
16:01:30 <amrith> ./
16:01:30 <harlowja_at_home> courtesy ping for dougwig, e0ne, flaper87, garyk, gcb, GheRivero, haypo
16:01:30 <harlowja_at_home> courtesy ping for HenryG, jd__, jecarey, johnsom, jungleboyj, kgiusti, kragniz
16:01:30 <harlowja_at_home> courtesy ping for lifeless, lintan, lxsli, Nakato, ozamiatin, rbradfor, redrobot
16:01:30 <harlowja_at_home> courtesy ping for rpodolyaka, sergmelikyan, sileht, spamaps, sreshetnyak, sreshetnyak, stevemar
16:01:30 <harlowja_at_home> courtesy ping for therve, thinrichs, toabctl, viktors, zhiyan, zzzeek
16:01:30 <harlowja_at_home> hola!
16:01:37 <HenryG> o/
16:01:40 <amrith> hello harlowja_at_home
16:01:45 <harlowja_at_home> (maybe we should do the whole meeting in spanish?)
16:01:55 <bknudson> hi
16:01:57 <johnsom> Hola
16:02:01 <harlowja_at_home> ha
16:02:12 <gcb> lol, I can't read spanish
16:02:19 <bknudson> buenos dias
16:02:28 <harlowja_at_home> como estas
16:02:34 <amrith> Yo no hablo español
16:02:41 <harlowja_at_home> gcb, me either :-P
16:02:55 <dims> hey y'all
16:03:10 <amrith> Pero puedo usar google translate
16:03:10 <harlowja_at_home> #topic Red flags for/from liaisons
16:03:23 <amrith> nothing from trove
16:03:26 <dhellmann> o/
16:03:27 <harlowja_at_home> so its been a little while, any new red flags from folks :)
16:03:28 <blogan> o?
16:03:30 <blogan> o/
16:03:39 <harlowja_at_home> johnsom, do u still have any red flag?
16:03:40 <gcb> nova have one http://logs.openstack.org/periodic/periodic-nova-py27-with-oslo-master/6235dad/testr_results.html.gz
16:03:44 <johnsom> Nothing to report here
16:03:46 <bknudson> no red flags from keystone tha tI know of.
16:03:58 <johnsom> Nope, the global change cleared up our issue
16:03:58 <gcb> I didn't have time to dig today
16:04:15 <harlowja_at_home> gcb, thanks, i'll see if i can look through what might of caused that
16:04:39 <gcb> harlowja_at_home,  thanks
16:05:45 <bknudson> periodic jobs getting stuff done!
16:05:51 <harlowja_at_home> http://status.openstack.org/openstack-health/#/?groupKey=build_name&resolutionKey=hour&searchProject=-with-oslo doesn't look to super either, if anyone wants to help investigate why some of those are acting up
16:06:15 <harlowja_at_home> i think all jobs are showing up there again, they weren't showing up some time ago, lol
16:06:42 <harlowja_at_home> though seems to be similar to '"testtools.matchers._impl.MismatchError: '200 OK' != '401 Unauthorized'"'
16:06:54 <harlowja_at_home> which might be something oslo.middleware
16:07:09 <harlowja_at_home> so cinder might be affected by the same gcb
16:07:18 <johnsom> Ours have cleared up after the global change, fyi
16:07:54 <harlowja_at_home> i'll have to ask the glance folks whats up with http://logs.openstack.org/periodic/periodic-glance-py27-with-oslo-master/b4a278f/console.html#_2016-11-07_06_19_23_225309 (looks unrelated)
16:08:07 <harlowja_at_home> ir http://logs.openstack.org/periodic/periodic-glance-py27-with-oslo-master/b4a278f/console.html#_2016-11-07_06_19_21_594028
16:08:27 <harlowja_at_home> johnsom, great :)
16:08:27 <harlowja_at_home> glance seems to be acting up, ha
16:09:05 <harlowja_at_home> anyone interested in looking into those, feel free, otherwise I'll try to also :)
16:09:26 <harlowja_at_home> #topic Releases of the week
16:09:41 <harlowja_at_home> so just wanted to see what people would like released (after the above issues are looked into)
16:09:48 <harlowja_at_home> anything specific from folks?
16:11:06 <harlowja_at_home> going once
16:11:43 <harlowja_at_home> ok, i'll just get our a batch release then after those current periodic issues get looked into :)
16:12:13 <harlowja_at_home> #topic Summit recaps
16:12:43 <harlowja_at_home> so thanks all of you who showed up in BCN :)
16:12:58 <harlowja_at_home> for those that didn't i've been slowly writing some of the things to do to the ML
16:13:02 <harlowja_at_home> #link https://wiki.openstack.org/wiki/Design_Summit/Ocata/Etherpads#Oslo
16:13:10 <harlowja_at_home> ^ sessions and such we held
16:13:48 <harlowja_at_home> http://lists.openstack.org/pipermail/openstack-dev/2016-November/106609.html (one of the ML recaps)
16:14:02 <harlowja_at_home> and http://lists.openstack.org/pipermail/openstack-dev/2016-November/106680.html
16:14:39 <harlowja_at_home> i'll send out a couple more i think (of things to do for oslo) but for now I think amrith has one such topic (related to summit also)
16:14:45 <harlowja_at_home> amrith, you ready??!?
16:14:54 <amrith> yes, was just warming up my voice
16:15:13 <harlowja_at_home> #topic Oslo.messaging anddddd amrith
16:15:18 <harlowja_at_home> ok all yours :)
16:15:24 <amrith> the topic I have relates to oslo.messaging and adding a level of message security.
16:15:38 <amrith> I summarized some thoughts in an etherpad
16:15:40 <amrith> #link https://etherpad.openstack.org/p/secure-oslo-messaging
16:15:51 <amrith> here is the TLDR
16:15:53 <amrith> for this meeting
16:15:54 <amrith> what am I looking for (at this stage)
16:15:54 <amrith> - do we believe that it is still interesting to oslo to support message signing and verification for oslo_messaging/rpc
16:15:54 <amrith> - are we ok with an approach where oslo_messaging/rpc provides a framework for consumers to define the signer and verifier, and not directly perform the signing and verification
16:15:56 <amrith> If the answers to the above two questions are both YES, then I think we have a set of things that we can discuss. If the answer to either is NO, then I think we have to address those issues first.
16:16:08 * amrith shuts up
16:16:30 <amrith> (for now, for those of you who thought this was permanent; sorry to dissapoint you :))
16:16:31 <dhellmann> wasn't some work already done on this topic?
16:16:38 <amrith> dhellmann, yes
16:16:41 <amrith> a lot of work was done
16:16:49 <bknudson> there was a whole kite project done by redhat.
16:16:51 <amrith> the primary difference was that in past iterations (referenced in the etherpad)
16:17:01 <bknudson> which was for the distribution of the keys
16:17:04 <amrith> the burden was placed on oslo.messaging to do the signing
16:17:06 <amrith> and key management
16:17:29 <amrith> a fundamental difference with my proposal is that I'm looking at oslo.messaging to be merely a framework that allos the consumer to do the signing and verification
16:17:43 <amrith> allows the consumer to register a callback that will be invoked when the full message is ready
16:17:50 <amrith> and will invoke a callback on each message received
16:17:54 <dhellmann> and what's this ultimately going to be used for?
16:17:59 <amrith> this makes the implementation (in oslo) much simpler.
16:18:08 <amrith> dhellmann, I summarized it in the therpad
16:18:09 <amrith> etherpad
16:18:12 <amrith> but the basic idea is this
16:18:17 <amrith> for projects with service vm's
16:18:24 <amrith> there are transport credentials on the service vm
16:18:30 <amrith> if the service vm is compromised
16:18:39 <amrith> then the transport credentials are compromised
16:18:48 <harlowja_at_home> johnsom, may also be useful for octavia (if this converges) - fyi
16:18:51 <dhellmann> ok, I think my position is unchanged that the service vms shouldn't be connecting to the bus anyway
16:18:57 <amrith> and one could conceivably have a person generate bogus messages
16:19:21 <sileht> dhellmann, I agree
16:19:32 <amrith> dhellmann, they do today. several projects have that construct. in any event, hypervisor escapes are also known to occur
16:19:44 <amrith> and that would mean that a hypervisor escape would put a nova compute at risk
16:19:46 <johnsom> harlowja_at_home Hmm, not sure.  As I mentioned in the session, we use two-way SSL with our service VM agents.
16:19:51 <dhellmann> the service vm should have a rest api and use keystone to protect it
16:20:13 <amrith> dhellmann, where would the service vm get keystone credentials?
16:20:19 <harlowja_at_home> johnsom, oh ya, nearly forgot :)
16:20:28 <amrith> at the end of the day, someone has to store credentials on the service vm
16:20:32 <amrith> credentials of some kind
16:20:34 <dhellmann> amrith : it shouldn't need them if all of the calls are incoming
16:20:46 <amrith> but they aren't
16:20:49 <johnsom> Basically what dhellmann said, but without the keystone part.
16:20:51 <amrith> some are merely status reporting
16:21:08 <bnemec> I thought we had this discussion a long time ago and agreed that Zaqar (then Marconi, I think?) was the right way for service vms to do messaging.
16:21:09 <bknudson> even incoming calls need the some service user credential to validate tokens
16:21:33 <amrith> bnemec, the problem is the same with any mechanism; so long as a transport has credentials, someone needs to generate them
16:21:33 <dhellmann> bknudson: ah, good point
16:21:42 <amrith> or store them
16:21:52 <amrith> but that's where PKI makes things better
16:22:07 <dhellmann> bknudson : though if the only permission that user has is to check the validity of other credentials, it's low risk
16:22:10 <amrith> it allows the client to at least identify itself in a manner that can be verified by the control plane
16:22:19 <sileht> But the servicevm can have it's own credential and use trust to authentic with the user one
16:22:19 <amrith> and the blast crater can be limited to that particular service vm
16:22:24 <bknudson> yes, we could give that user only access to validate tokens
16:22:26 <sileht> (like heat does)
16:22:46 <bknudson> also, given a token it should be able to validate itself.
16:23:03 <amrith> so, what y'all are proposing is a solution where service vm's don't talk to the messaging infrastructure
16:23:04 <bknudson> so maybe no service user would be necessary. It just is now because that's how auth_token works.
16:23:13 <amrith> first let me point out that the control plane is NOT the openstack control plane
16:23:17 <amrith> it is the trove cntrol plane
16:23:21 <dhellmann> amrith : right. instructions should be incoming. "tell me your status" is an instruction.
16:23:34 <amrith> which is by choice not the same as the openstack control plane
16:24:08 <amrith> therefore all we're looking for is a mechanism to make that control plane safer.
16:24:20 <amrith> sorry dhellmann that is polling
16:24:25 <dhellmann> yes, it is
16:24:35 <amrith> but when a db fails, we want the guest to tell the control plane
16:24:44 <amrith> how frequently would you like to poll?
16:25:19 <amrith> we already register for notifications from db's to get informed of state changes
16:25:29 <dhellmann> how about an unprivileged web hook with no data that just means "you need to ask me my status"
16:25:30 <amrith> having 'tell me your status' be polling defeats the whole purpose.
16:25:39 * harlowja_at_home starts to wish we had something like etcd (so that it could be used to avoid said polling)
16:26:19 <dhellmann> harlowja_at_home : you still have to secure that. the whole point of saying connections are only incoming is to have the rest of the infrastructure drive the service vm, and to treat it as hostile and insecure until proven otherwise
16:26:21 <harlowja_at_home> https://coreos.com/etcd/docs/latest/api.html#waiting-for-a-change (such a thing could be used for this)
16:26:31 <dhellmann> amrith : you may need to have this discussion with SpamapS' new arch wg
16:26:37 <johnsom> dhellmann +1
16:26:41 <amrith> dhellmann, all of these are approaches, and by no means the only ones, to avoid the problem. is the feeling that (a) projects shouldn't be using service vm's that connect to the control plane, and (b) oslo isn't interested in message signing for RPC?
16:27:09 <dhellmann> my memory is that adding signing to rpc caused serious performance issues, so it was turned off
16:27:15 <harlowja_at_home> (c) undecided
16:27:37 <amrith> dhellmann, it would cause performance issues if we turned it on for everything and everybody
16:27:43 <dhellmann> if it's only for some messages, that may mitigate it, but I don't think we ever envisioned the messaging lib being used for talking to untrusted nodes
16:27:54 <amrith> but we are able to turn it on for those who care about it.
16:28:24 <amrith> so, if I understand correctly, octavia has full blown client authenticaed ssl
16:28:32 <amrith> other projects have their own solutions
16:28:42 <harlowja_at_home> johnsom, ^
16:29:13 * harlowja_at_home starting to feel we may need to go talk with the arch-wg, then get some consensus on the ML (or at a future meeting)?
16:29:14 <johnsom> Correct, we issue unique certs to each service VM at boot and manage renewal
16:29:36 <amrith> we would be creating keypairs, not going full blown ssl
16:31:52 <amrith> so, what does the team feel? to my two questions ...
16:31:52 <amrith> - do we believe that it is still interesting to oslo to support message signing and verification for oslo_messaging/rpc
16:31:52 <amrith> - are we ok with an approach where oslo_messaging/rpc provides a framework for consumers to define the signer and verifier, and not directly perform the signing and verification
16:32:03 <amrith> if the answers are no and no, that's fine. just let me know
16:33:03 <harlowja_at_home> i'm personally ok with it, i don't think the 2 things above need to be tied into whether the choice is right for trove to be doing this (or if it should do this or...)
16:33:19 <harlowja_at_home> but my guess is others believe differently :)
16:34:44 <dhellmann> I'd be afraid that adding the feature would encourage more anti-patterns like this, but I don't object strongly enough to -2 it
16:34:48 <amrith> So, in the answer of an answer of 'yes' I'm going to assume that the answer is 'no' and look for other alternatives. if that's not the case, please let me know.
16:34:58 <sileht> personaly I don't see the point to add hook when we can do the same without hook
16:35:08 <dhellmann> that's also a good point
16:35:14 <amrith> sileht, that's not true; it is not the same with a decorator
16:35:28 <amrith> upgrade and cross release issues are MUCH easier if this is handled at a level below the project.
16:35:52 <harlowja_at_home> sileht, do  you mean by not using oslo.messaging (and switching to rest?)
16:35:52 <amrith> if it is up to the project, with decorators, then the case of a client without the code to handle a signature will fail
16:35:57 <amrith> because of the extra parameter.
16:36:24 <amrith> no harlowja_at_home he's talking about using a decorator as I had illustrated in the ML
16:36:40 <sileht> harlowja_at_home, about https://review.openstack.org/#/c/394389/
16:36:43 <amrith> the issue with that is that to handle it in the consumer (trove) signature will have to be a parameter in kwargs
16:37:01 <amrith> and that means that a client without the code to handle it would barf at the extra parameter
16:37:06 <harlowja_at_home> ah, haven't seen that one yet, created this morning, ha
16:37:23 <dhellmann> if you aren't doing versioning of that api, you already have a problem trying to change it in any way
16:37:35 <amrith> dhellmann, this isn't about versioning an API
16:37:45 <amrith> this is about allowing coexistance
16:37:57 <amrith> and allowing a version that has no support understand a message that has an extra parameter
16:38:14 <sileht> amrith, that what oslo.messaging version  is for
16:38:17 <amrith> and this isn't an API, this is every RPC call.
16:38:54 <sileht> the decorator version is an API change while the hooks version is not
16:39:53 <amrith> sileht, your @verifier is the thing that pops the signature
16:40:08 <amrith> what happens to a server with no @verifier?
16:40:30 <amrith> are you suggesting that we must use version caps on a per guest basis?
16:40:36 <sileht> amrith: you have to use version
16:40:38 <amrith> in the service vm context, that is infeasible
16:40:56 <amrith> you cannot require an operator to upgrade all guest database instances at the same time in order to upgrade the control plane
16:41:21 <dhellmann> are you not going to be signing the messages sent from the control plane to the service vm?
16:41:29 <amrith> yes
16:41:31 <amrith> absolutely
16:41:36 <amrith> that's the prime attack vector
16:41:51 <amrith> one compromised service vm telling another one to go shut itself down, for example.
16:41:55 <dhellmann> so how are you going to handle that upgrade case?
16:42:19 <amrith> if the code is in oslo.messaging, then a guest who doesn't have the code will receive a signature and the old oslo.messaging will just ignore it.
16:42:48 <amrith> that is a luxury of having the verification (or at least the dispatch of verification) being in dispatch
16:42:55 <amrith> and where the signature is not an argument
16:43:12 <amrith> if the code is in the consumer (trove) then all I can do is insert a signature as an argument to the API call
16:43:13 <dhellmann> right, so that's why I brought up API versioning. Because you have this one case where you can put the thing into the library and "hide" the change, but you won't be able to do that for all cases.
16:44:20 <amrith> so we can handle this in the consumer by adding a signature parameter to all API calls
16:44:25 <amrith> literally all API calls
16:44:26 <harlowja_at_home> amrith  dhellmann sileht do the three (and more?) want to continue on this perhaps in #openstack-oslo seems like we may not reach consensus just yet :)
16:44:35 <dhellmann> harlowja_at_home : good idea
16:44:39 <amrith> and in the version that supports signatures, remove it.
16:44:42 <amrith> seems a bit much
16:45:35 <amrith> harlowja_at_home, it seems like we are at a decision
16:45:49 <amrith> and the answer seems to be that consumer projects like trove should handle this with api versioning.
16:45:52 <harlowja_at_home> ok, whats the ruling :-P
16:45:55 <amrith> and hooks/decorators.
16:46:05 <amrith> I'm happy with that decision, just let me know.
16:46:13 <dhellmann> or not using rpc at all :-)
16:46:21 <sileht> dhellmann: ++
16:46:24 <amrith> or not using rpc calls for this.
16:46:45 <amrith> which is a much bigger project, but sure, it is a feasible solution.
16:47:14 <amrith> fyi dhellmann this is not restricted to projects with service vm's.
16:47:20 <harlowja_at_home> okie dokie, amrith do u mind sending out a conclusion to the ML, to at least distribute this decision?
16:47:28 <amrith> with a hypervisor escape, one can compromise the openstack control plane
16:47:51 <dhellmann> amrith : sure
16:48:18 <amrith> harlowja_at_home, sure thing
16:48:20 <amrith> will do that.
16:48:23 <harlowja_at_home> amrith, thanks :)
16:48:30 <harlowja_at_home> alright next topic
16:48:41 <harlowja_at_home> #topic OpenStack goals
16:48:47 <harlowja_at_home> dhellmann, the mic is yours :)
16:48:51 <dhellmann> ah, thanks
16:49:07 <dhellmann> as you all know, the goal this cycle is to move off of incubated code to the libraries
16:49:14 <harlowja_at_home> #link https://governance.openstack.org/goals/ocata/remove-incubated-oslo-code.html
16:49:17 <dhellmann> we have a relatively small number of projects with work to do on that regard
16:49:36 <dhellmann> projects have already started, and the patches should all be visible on this report
16:49:41 <dhellmann> #link https://review.openstack.org/#/q/topic:goal-remove-incubated-oslo-code+is:open
16:49:47 <harlowja_at_home> dhellmann, do these eventually show up in https://governance.openstack.org/goals/ocata/remove-incubated-oslo-code.html#project-teams ?
16:49:56 <harlowja_at_home> under 'Completion Artifacts'?
16:50:02 <dhellmann> I, and I'm sure the other project folks, would appreciate help with reviews of those changes
16:50:22 <dhellmann> harlowja_at_home : usually the artifacts will be links to bugs or blueprints, though some projects have added links to the patches
16:50:32 <harlowja_at_home> k
16:51:06 <dhellmann> so, when you have time, it would help the community a bit to do some of those reviews in case folks get stuck on transitions in any way
16:51:29 <dhellmann> harlowja : that's all I had, thanks for giving me the slot on the agenda
16:51:32 <harlowja_at_home> sounds good to me, the last of the last of the last of that cleanup :)
16:51:54 <harlowja_at_home> cool, thanks dhellmann
16:52:08 <harlowja_at_home> #topic Open discussion
16:52:20 <harlowja_at_home> so just a few other tiny links to share
16:52:36 <harlowja_at_home> #link https://www.openstack.org/ptg/
16:52:43 <harlowja_at_home> and #link https://releases.openstack.org/ocata/schedule.html
16:52:53 <harlowja_at_home> keep those in mind (for folks reviewing, doing changes...)
16:53:14 <harlowja_at_home> Jan 16-20 isn't that far away :)
16:53:25 <harlowja_at_home> (Final release for non-client libraries)
16:53:41 <harlowja_at_home> especially if people go on vacations and ...
16:53:51 <harlowja_at_home> in fact, damn that's really not that far away, lol
16:54:39 * harlowja_at_home that's all i had
16:54:40 <bknudson> this release will consist mostly of requirements changes and switching asserts
16:54:46 <harlowja_at_home> :)
16:54:48 <harlowja_at_home> lol
16:55:17 <harlowja_at_home> and cleanups of some sorta
16:55:20 <harlowja_at_home> *sort
16:56:18 <harlowja_at_home> alright, guess we can end 4 minutes early, since doesn't seem like anyone else has any open discussions :)
16:56:39 <harlowja_at_home> thanks amrith dhellmann sileht and others for the insightful discussion around messaging :)
16:57:03 <harlowja_at_home> as always #openstack-oslo for more fun! :)
16:57:19 <amrith> thx harlowja_at_home
16:57:30 <amrith> oh, one question if we are in open discussion
16:57:44 <harlowja_at_home> 3 minutes :-P
16:57:46 <amrith> if the world comes to an end tomorrow, does oslo have any special plans :)
16:57:48 <harlowja_at_home> ha
16:58:07 <harlowja_at_home> move everyone to canada?
16:58:13 <harlowja_at_home> lol
16:58:20 <harlowja_at_home> i'm thinking i might like vancouver more and more
16:58:25 <kgiusti> unless canada builds a wall...
16:58:26 <harlowja_at_home> ha
16:58:28 <amrith> sure, just follow me y'all ;)
16:59:13 <harlowja_at_home> oh, ya, and get out and vote (plllease, especially those in battleground states) :)
16:59:22 <harlowja_at_home> (for those in US)
16:59:29 <amrith> everyone should ...
16:59:41 <amrith> "vote early, vote often"
17:00:08 <harlowja_at_home> :)
17:00:10 <harlowja_at_home> #endmeeting