12:05:08 <jaosorior> #startmeeting TripleO Security Squad
12:05:09 <openstack> Meeting started Wed Jun  6 12:05:08 2018 UTC and is due to finish in 60 minutes.  The chair is jaosorior. Information about MeetBot at http://wiki.debian.org/MeetBot.
12:05:10 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
12:05:13 <openstack> The meeting name has been set to 'tripleo_security_squad'
12:05:17 <jaosorior> This time it was me being a bit late :D
12:05:27 <moguimar> no worries
12:05:41 <jaosorior> lhinds, Tengu are you around?
12:05:44 <moguimar> #link https://etherpad.openstack.org/p/tripleo-security-squad
12:06:22 <jaosorior> #topic Public TLS by default work update
12:06:56 <jaosorior> As I mentioned yesterday in the TripleO weekly meeting, the public TLS by default work is currently blocked due to some decisions made about the approach we'll take to enable that.
12:07:11 <jaosorior> The first iteration added that to the deploy mistral workflow
12:07:33 <lhinds> hey jaosorior , sorry was delayed
12:07:36 <jaosorior> However, we decided to instead add that logic to the derive_parameters workflow, which is currently not enabled by default
12:08:00 <jaosorior> dervie_parameters is currently used for NFV cases, where more info is needed to be dynamically added to the deployment depending on the hosts
12:08:03 <jaosorior> but it could be generalized
12:08:27 <jaosorior> jaganathan took on the work of making that workflow more general, and even adding the Fixed IP additions we need
12:08:56 <jaosorior> He's working with the following patches
12:09:00 <jaosorior> #link https://review.openstack.org/#/c/572678/
12:09:05 <jaosorior> #link https://review.openstack.org/#/c/572701/
12:09:11 <jaosorior> So please review them if you have some time
12:09:21 <raildo> o/
12:10:15 <ooolpbot> URGENT TRIPLEO TASKS NEED ATTENTION
12:10:15 <ooolpbot> https://bugs.launchpad.net/tripleo/+bug/1775205
12:10:16 <ooolpbot> https://bugs.launchpad.net/tripleo/+bug/1775209
12:10:17 <openstack> Launchpad bug 1775205 in tripleo "Failing jobs are no longer getting the output from gate_test_oooq.sh script" [Critical,Triaged] - Assigned to Ronelle Landy (rlandy)
12:10:18 <openstack> Launchpad bug 1775209 in tripleo "docker permission denied on upgrade prep containers" [Critical,Triaged] - Assigned to Steve Baker (steve-stevebaker)
12:10:18 <jaosorior> After that work merges, we can add the code needed to get TLS by default in the overcloud on top of that
12:10:55 <jaosorior> That's all on that side
12:11:07 <jaosorior> Any questions/feedback?
12:12:16 <jaosorior> #topic Limit TripleO users
12:12:25 <jaosorior> I believe Tengu and lhinds have some updates on this topic
12:13:20 <Tengu> woops, sorry, late.
12:13:25 <lhinds> So Tengu plans to tackle stack user, while I will look at the heat-admin and other users. We also plan to work on a spec as first prior so that a consensus can be gained over the approach.
12:13:32 <lhinds> please add if you like Tengu
12:13:47 <lhinds> there is a ML thread with some good discussion
12:13:56 <lhinds> and giving oslo.privsep a look in
12:14:05 <Tengu> actually two threads, but yep.
12:14:08 <lhinds> (in tripleo pythonclient)
12:14:40 <Tengu> fact is, oslo.privsep isn't easy to tackle, and will require "some" refactoring - but it's all for the best. We really have to write the spec/blueprint first though
12:15:16 <jaosorior> Tengu: what are the main limitations we have currently that prevent us from adopting oslo.privsep?
12:15:22 <Tengu> Following Michael blog posts we can reach a nice state.
12:15:56 <Tengu> jaosorior: well, at least in tripleoclient, the undercloud deploy is called with root - and there are many things that needs to be changed in order to launch it as a user.
12:16:41 <Tengu> mainly, there are file rights to change on the fly in the process, it starts heat daemon as heat and so on. for now, the code itself doesn't care about privileges, so there are os.chown, os.setuid and the like.
12:17:29 <Tengu> also a bunch of subprocess calls without any "sudo" or whatsover. So we will need to git-grep all of those occurences, check WHO has to run them, what are the requested rights, and finaly configure some privsep context.
12:17:39 <jaosorior> Tengu: do we have a list of things we should change so we can track them and address them systematically?
12:17:44 <jaosorior> that would probably be useful
12:17:58 <jaosorior> we already have an etherpad where we were tracking this work. lhinds do you have the link?
12:18:15 <lhinds> yup: https://twitter.com/WHS_Carpet
12:18:20 <lhinds> derp!
12:18:22 <lhinds> one sec..
12:18:26 <jaosorior> lol
12:18:28 <Tengu> jaosorior: not really. I have a beginning of sudoers though, it might be a start. But using rootwrap (privsep) will change that
12:19:13 <lhinds> https://etherpad.openstack.org/p/tripleo-heat-admin-security
12:19:21 <Tengu> fact is, rootwrap is "just" a sudo filter so that we don't have to manage the listing directly in sudo. also, it allows a more fine-grained right management.
12:19:28 <jaosorior> #link https://etherpad.openstack.org/p/tripleo-heat-admin-security
12:19:42 <Tengu> #link https://www.madebymikal.com/how-to-make-a-privileged-call-with-oslo-privsep/
12:19:49 <Tengu> #link https://www.madebymikal.com/adding-oslo-privsep-to-a-new-project-a-worked-example/
12:20:04 <Tengu> those two blogposts are really interesting and put some good practices/ideas.
12:20:14 <jaosorior> OK, I'll take a look at those
12:20:26 <Tengu> it might be interesting to involve Michael Still in "our" spec writing
12:20:56 <lhinds> so the above would cover python code, but we also need to agree an approach on general nix scripts and calls using 'sudo <command>'
12:20:58 <Tengu> as it seems it's a pretty young thing, it's the good time to write a precise spec and implement best practices for privsep use
12:21:02 <jaosorior> Tengu: is the spec you're referring to limited to the undercloud's stack user? or will it be a general approach for the overcloud user as well?
12:21:23 <openstackgerrit> Rafael Folco proposed openstack-infra/tripleo-ci master: Move scenario000-multinode-oooq-container-upgrades to check  https://review.openstack.org/572187
12:21:48 <Tengu> jaosorior: I think we should write two specs: one regarding general python coding style regarding privileges escalation;  a second regarding "standard sudo calls from a shell"
12:22:01 <Tengu> lhinds: -^ this also answers your concern I think :).
12:22:17 <jaosorior> OK
12:22:27 <Tengu> I see those two issues as different issues - so different specs.
12:22:34 <lhinds> That would work, we just need to make sure we co-ordinate the patches for when we switch off :ALL
12:22:46 <Tengu> lhinds: of course we have to.
12:22:47 <jaosorior> two issues that are coverd by this overall topic though :D
12:23:01 <Tengu> jaosorior: yep. but it's best to cut it in smaller parts.
12:23:07 <Tengu> else we won't find a start nor an end.
12:23:17 <lhinds> and yes, use the same topic / branch name (if that would work)
12:23:38 <jaosorior> Tengu: agreed
12:23:49 <Tengu> lhinds: I've never done a spec/blueprint, care to start it somewhere (already on the etherpad I think) and I add my inputs?
12:24:20 <lhinds> Tengu: sure, its quite simple..will kick something off
12:24:26 <jaosorior> Tengu: the etherpad is a good place to brainstorm and track things. you can even put a draft of the spec and subsequently propose it
12:24:31 <Tengu> lhinds: thank you :).
12:24:39 <Tengu> jaosorior: will see to that.
12:24:59 <jaosorior> Alright, that will cover then python privilege escalation
12:25:04 <lhinds> I was thinking we could do a BP with two different specs
12:25:22 <lhinds> I will get onto that, leave it with me.
12:25:24 <jaosorior> lhinds: that sounds like a good idea
12:25:36 <openstackgerrit> John Trowbridge proposed openstack-infra/tripleo-ci master: Add ability to upgrade FROM change under test  https://review.openstack.org/572419
12:25:37 <openstackgerrit> John Trowbridge proposed openstack-infra/tripleo-ci master: Add logic/tests for n->n+1 upgrades  https://review.openstack.org/572420
12:25:40 <Tengu> lhinds: forward the links, and we can work on that.
12:25:52 <lhinds> Tengu: ack, will do
12:25:54 <jaosorior> Regarding scripts and sudo usage, lhinds, wanna share the proposal?
12:27:08 <Tengu> as said - we should write a dedicated spec for that, and enforce it in order to drop that horrible NOPASSWD:ALL :).
12:28:01 <lhinds> yup, so I have a basic script to pull out all sudo calls. Plan is to get this runnning in the various project test suites (tempest) and build a whitelist of calls that are made.
12:28:16 <Tengu> there's a thread about that in the ML, and the etherpad also gives some info - we have to format all of that and separate "python privilege escalation" and "privilege escalation with sudo in scripts".
12:28:42 <lhinds> So the spec will outline how to gather sudo calls (on-going) and regexp strictness needed for whitelisting calls.
12:29:01 <Tengu> +42
12:29:03 <lhinds> and also put together some playbooks that can apply the whitelists
12:29:36 <Tengu> hmm. would it make sense to add sudoers snippets in packages directly?
12:29:51 <lhinds> packages?
12:30:08 <Tengu> err. right, containers now, packages will drop.
12:30:18 <lhinds> sorry, yes...
12:30:35 <lhinds> spoke to jaosorior , it would be made within the image (vm / container)
12:30:45 <jaosorior> Tengu: well, each openstack project already has some sudoers files specific to the project. The main concern here is the heat-admin or tripleo-admin users which we use for the deployment, and the validations user which we use for...well, validations :D
12:31:10 <Tengu> jaosorior: as well as "stack" user I guess (on the undercloud)
12:31:26 <Tengu> I saw some concern regarding its extensives rights on some sites.
12:32:04 <jaosorior> Tengu: well, yeah, although ideally the stack user should only be able to do openstack commands, which we will then limit
12:32:32 <jaosorior> or the minimum set of calls to set up an undercloud
12:32:39 <shardy> it's worth noting that even if you lock down e.g heat-admin for the nodes, all API actors on the undercloud are admin, e.g you can always run arbitrary ansible tasks on the nodes even if you lock down the default user provided via the user-data
12:32:43 <lhinds> actually that's a good point.. the stack user has a lot of sudo calls for hiera work and others.
12:32:45 <Tengu> it's actually more or less the case - there are some needs for yum install - all the rest should be managed by privsep in python.
12:33:00 <Tengu> bah, we should manage that with user GROUPS
12:33:06 <lhinds> so spec needs to be on 'python' mgmt of escaltions, and script management
12:33:14 <lhinds> rather then stack / heat admin
12:33:18 <Tengu> i.e. hieradata owned by root:deploy - stack member of deploy group, and 0640 on the data.
12:33:21 <lhinds> or underclooud /overcloud
12:33:22 <shardy> not saying a bad idea, only that it's a small part of the overall picture with an admin-driven provisioning/management tool
12:33:42 <Tengu> yup. that's why we need some proper specs.
12:33:54 <Tengu> in order to ensure we cover the cases, step by step.
12:34:23 <Tengu> file ownership can also be a (good) solution in some cases in order to avoid so-called privilege escalations.
12:34:53 <openstackgerrit> Chandan Kumar proposed openstack/tripleo-quickstart-extras master: use upstream git url instead of github  https://review.openstack.org/572347
12:35:26 <jpena> I'm trying (unsuccessfully for now) to test the TripleO all-in-one installer on a VM running on my laptop. So far I'm getting http://paste.openstack.org/show/722798/ and the deployment fails. Does that ring a bell?
12:36:06 <jaosorior> Tengu, lhinds: can you sync up in order to write the needed specs? It would be beneficial to have these discussions in the spec, for easier tracking.
12:36:11 <Tengu> jpena: the error seems actually above, not the hiera thingy. nothing before that error?
12:36:19 <Tengu> jaosorior lhinds sure thing.
12:36:29 <lhinds> jaosorior: agree, lets trash the finer points in review
12:36:46 <jaosorior> Tengu, lhinds: Thanks for taking a look at this work
12:36:51 <Tengu> np
12:37:44 <jaosorior> shardy: you'll be added to the spec reviews. your feedback is always very useful :D
12:38:19 <shardy> jaosorior: ack thanks
12:38:25 <openstackgerrit> Arx Cruz proposed openstack/tripleo-quickstart master: Using session in get-overcloud-nodes.py  https://review.openstack.org/572709
12:40:12 <jaosorior> Any more thoughts/feedback/questions about this?
12:41:59 <jpena> Tengu: checking
12:42:05 <jaosorior> #topic Any other business
12:42:20 <jaosorior> Anything else folks want to bring up to the meeting?
12:42:28 <moguimar> yep
12:42:44 <moguimar> me and raildo are making progress with the secrets stuff
12:42:57 <jaosorior> moguimar: can you share a bit of context for the rest of the atendees?
12:43:15 <moguimar> we're implementing backend drivers for oslo.config
12:43:34 <moguimar> to store plaintext secrets elsewhere other than in the plaintext config files
12:43:44 <raildo> #link http://specs.openstack.org/openstack/oslo-specs/specs/queens/oslo-config-drivers.html
12:44:18 <moguimar> the oslo.config work consists of supporting drivers and an INI driver for fetching config from an URL
12:44:32 <moguimar> then we'll add a driver for castellan
12:44:49 <moguimar> so oslo.config will be able to retrieve sensitive config from castellan
12:45:00 <moguimar> after that, we'll try to integrate to tripleO
12:45:25 <moguimar> right now we're 80% done in the oslo.config and starting the work on castellan
12:45:38 <moguimar> anything you want to add raildo ?
12:45:55 <moguimar> would also be nice to bring doug to this meeting next week
12:46:04 <jaosorior> Yeah, that would be useful
12:46:54 <raildo> so, would be great review this current spec: https://review.openstack.org/#/c/474304/ where we should define how that integration with tripleO will be done, probably target release
12:47:04 <raildo> so, any review on that aspect will be appreciated :)
12:47:39 <Tengu> raildo: will read it at least. no +2 from me thoug - can't do that :).
12:48:00 <raildo> Tengu, haha thanks :)
12:48:38 <jaosorior> raildo: I'll check that spec out, however, once we have a bit more details on how the deployment would look like; it would be better to propose a separate spec just for TripleO integration
12:49:10 <openstackgerrit> Luke Hinds proposed openstack/tripleo-specs master: [WIP] Spec placeholder for limit sudo BP  https://review.openstack.org/572760
12:49:22 <raildo> jaosorior, sounds reasonable for me
12:49:50 <jaosorior> raildo: lhinds just uploaded a spec placeholder, so you could use that for reference :) ^^
12:49:57 <Tengu> lhinds: just added myself on your placeholder.
12:50:13 <raildo> awesome, thanks for the hints!
12:50:17 <jaosorior> moguimar, raildo: Do you have an estimate on when you'll finish the castellan driver?
12:50:23 <lhinds> Tengu: cool, there will be another coming for pything stuff
12:50:28 <jaosorior> so we can start thinking about TripleO integration
12:50:33 <lhinds> s/pything/python
12:50:59 <moguimar> not yet
12:50:59 <raildo> jaosorior,  so, the castellan driver will be target for early on Stein release
12:51:17 <jaosorior> OK
12:51:23 <raildo> but not so well defined right now
12:51:24 <moguimar> tks raildo
12:51:55 <raildo> so basically, that's no worries about that on Rocky, we can start discussing about that on PTG-ish
12:52:10 <jaosorior> sure
12:53:20 <jaosorior> raildo, moguimar: Do you want to use next week's meeting to brain storm about TripleO integration?
12:53:31 <raildo> jaosorior, that would be awesome!
12:53:53 <jaosorior> OK, I'll write it up as a topic for next week's meeting
12:53:57 <jaosorior> lets hope redrobot can make it
12:54:15 <jaosorior> raildo, moguimar: thanks for driving this effort
12:54:41 <raildo> no problem :)
12:55:08 <jaosorior> any questions/feedback on this work?
12:55:44 <Tengu> raildo: just read your spec. Pretty nice :). Added a comment for tripleo integration "needs" - I'm a beginner on that topic, so there's possibility I missed something :)
12:56:08 <openstackgerrit> Luke Hinds proposed openstack/tripleo-specs master: [WIP] Spec to improve privilege mgmt in py-client  https://review.openstack.org/572761
12:56:34 <Tengu> lhinds: cool :)
12:56:35 <raildo> Tengu, thanks , I'll take a look on that soon!
12:56:52 <lhinds> so both have the same topic now:
12:56:54 <lhinds> https://review.openstack.org/#/q/topic:priv-mgmt+(status:open+OR+status:merged)
12:57:02 <lhinds> we can use the same topic for patches as well
12:57:06 <lhinds> (easier to track)
12:57:22 <jaosorior> lhinds: awesome!
12:57:26 <lhinds> don't worry about naming and stuff, we can fix that up
12:57:55 <lhinds> Tengu: you can cherry pick the spec down and take over as author or co-author etc.
12:58:08 <lhinds> or just leave as it is, up to you mate
12:58:08 <jaosorior> Well, we're at the end of the meeting now
12:58:12 <Tengu> lhinds: pl
12:58:12 <jaosorior> thanks to everybody for attending
12:58:15 <Tengu> *ok
12:58:17 <lhinds> thanks jaosorior
12:58:22 <raildo> thanks ozz
12:58:33 <jaosorior> #endmeeting