13:02:20 <jaosorior> #startmeeting TripleO Security Squad 13:02:21 <openstack> Meeting started Wed Mar 21 13:02:20 2018 UTC and is due to finish in 60 minutes. The chair is jaosorior. Information about MeetBot at http://wiki.debian.org/MeetBot. 13:02:22 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 13:02:24 <openstack> The meeting name has been set to 'tripleo_security_squad' 13:02:32 <gfidente> Tengu can I see your cmdline? 13:02:51 <jaosorior> Hey! d0ugal, lhinds, owalsh 13:02:52 <Tengu> gfidente: 2s 13:03:00 <Tengu> gfidente: basically I did that: http://paste.openstack.org/show/707512/ 13:03:02 <jaosorior> So, today should be a shorter meeting than last time :D 13:03:09 <d0ugal> :) 13:03:12 <jaosorior> should I wait a bit more for other folks? or should we start already? 13:03:26 <lhinds> I think we can kick off with d0ugal here now 13:03:30 <jaosorior> Alright! 13:03:35 <lhinds> mistral is the first topic 13:03:41 <Tengu> jaosorior: oh, meeting? here? 13:03:42 <jaosorior> #topic Mistral Secret Storage 13:03:45 <d0ugal> apetrich, thrash, rbrady, toure ^ we are going to chat about mistral and secrets if you want to join. 13:03:59 <Tengu> gfidente: do you take part in the meeting? 13:04:00 <thrash> d0ugal: ack 13:04:01 <jaosorior> Tengu: yes. It's the weekly Security Squad meeting 13:04:04 <apetrich> oh dear 13:04:16 <gfidente> Tengu security squad? 13:04:22 <Tengu> jaosorior: oh. I'll go DM with gfidente then :) 13:04:26 <rbrady> d0ugal: ack 13:04:46 <jaosorior> So, we've been talking a while about needing secret storage for mistral 13:04:54 <jaosorior> This is due to the fact that we store a LOT of sensitive information there 13:05:06 <jaosorior> the overcloud private keys and passwords namely 13:05:13 <openstackgerrit> Tim Rozet proposed openstack/puppet-tripleo stable/queens: Fixes incorrect ownership of ODL TLS cert/key https://review.openstack.org/554909 13:05:41 <jaosorior> Being TripleO an active user of mistral, I would like it to "beta" or take into use any solution that we have in mind 13:06:04 <jaosorior> Also, having talked to thrash in the PTG, I also volunteer to help out on the coding side of mistral if more hands are needed. 13:06:22 <jaosorior> But I would like to talk and understand what are the main challenges on this side 13:07:03 <d0ugal> so, first I think we need to clarify exactly what is stored and why. 13:07:09 <jaosorior> sure 13:07:43 <d0ugal> Mistral has a database that is mostly in-flight only. We store all the heat parameters etc. while the workflow is being executed 13:07:52 <d0ugal> They are then stored for 48 hours afterwards 13:08:22 <d0ugal> Mistral does log lots of information, and parameters may be logged at times - but I think this has been reduced (or possibly stopped) 13:08:23 <thrash> I think the more sensitive stuff is stored in a mistral environment, is it not? 13:08:37 <d0ugal> thrash: no, it is stored in Swift now 13:08:41 <thrash> d0ugal: ack 13:08:51 <apetrich> d0ugal, parameters are logged in debug only now 13:09:20 <apetrich> as with most sensitive info AFAIK 13:09:26 <d0ugal> The only information stored in mistral long term is two different "environments" - blobs of json basically 13:09:45 <d0ugal> These are the ssh keys for overcloud nodes, iirc 13:09:51 <d0ugal> and .. 13:10:00 <jaosorior> d0ugal: which environments? 13:10:03 <d0ugal> undercloud_ceilometer_snmpd_password and undercloud_db_password 13:10:12 <d0ugal> tripleo.undercloud-config and "ssh_keys" 13:10:14 <ooolpbot> URGENT TRIPLEO TASKS NEED ATTENTION 13:10:14 <ooolpbot> https://bugs.launchpad.net/tripleo/+bug/1757111 13:10:14 <ooolpbot> https://bugs.launchpad.net/tripleo/+bug/1757174 13:10:15 <openstack> Launchpad bug 1757111 in tripleo " fs020(both queens/master) tempest tests failing while booting an instance" [Critical,Triaged] 13:10:16 <openstack> Launchpad bug 1757174 in tripleo "tripleo-buildimage-overcloud-full-centos-7 failing with diskimage_builder.element_dependencies.MissingElementException: Element 'size=4096'' not found" [Critical,Triaged] 13:10:23 <d0ugal> They can be viewed with... 13:10:23 <jaosorior> d0ugal: why do we specifically store those passwords in mistral and not swift? 13:10:25 <d0ugal> $ mistral environment-get tripleo.undercloud-config 13:10:35 <d0ugal> $ mistral environment-get ssh_keys 13:11:04 <openstackgerrit> Marius Cornea proposed openstack/tripleo-upgrade master: Include connectivity check prepare scripts during FFU https://review.openstack.org/554914 13:11:04 <d0ugal> jaosorior: good question. Mostly for legacy reasoning I think. They could be moved to swift 13:11:05 <owalsh> ssh_keys is the heat-admin key? 13:11:20 <d0ugal> owalsh: I believe so, but I am not sure. 13:11:26 <jaosorior> d0ugal: would be great if we would keep all the passwords in one place. So we can secure that one place at some point. 13:11:44 <Tengu> (use gopass + gpg :D) 13:11:46 <d0ugal> jaosorior: The tripleo.undercloud-config environment is related to the undercloud itself, rather than a plan - I think that is why it is in mistral. 13:12:04 <d0ugal> jaosorior: +1 13:12:31 <openstackgerrit> Carlos Camacho proposed openstack/tripleo-quickstart-extras master: Collect installed cron jobs https://review.openstack.org/554889 13:12:38 <d0ugal> I think the ssh_keys environment was added out of simplicity, we didn't have a better plan at the time. 13:12:59 <jaosorior> thrash, apetrich does anybody know what ssh_keys actually is? is it the keys for heat-admin? 13:13:00 <trozet> can another core help out with reviewing https://review.openstack.org/#/c/553788/1 please? 13:13:16 <dtantsur> folks.. I know it may sound provocative, but is it possible to add configuration steps to a service template that are NOT written in puppet 13:13:18 <dtantsur> ? 13:13:23 <d0ugal> jaosorior: I can find out. 13:13:25 <thrash> jaosorior: I think so. Would need to double check. 13:13:38 <dtantsur> I don't really want to spend half of cycle doing a trivial thing like 'call a command, get its result' 13:13:44 <d0ugal> or shadower and mandre would know if they are around 13:13:53 <openstackgerrit> Harald Jensås proposed openstack/tripleo-heat-templates master: Add ctlplane networking for routed networks https://review.openstack.org/547326 13:14:04 <jaosorior> either way, there's a private key there, which would be considered sensitive info. So we need to secure it somehow 13:14:17 <d0ugal> jaosorior: +1 13:14:35 <hjensas> derekh: ^^ Can you have a look at the python script there? Make sure I don't mess up the ipv6 stuff again? 13:14:36 <dtantsur> EmilienM: hey, maybe you know (re my question above) 13:14:40 <jaosorior> d0ugal, thrash: One option would be to move all that to swift. And rely on swift encryption (which we don't have right now, but we could enable) 13:14:53 <apetrich> jaosorior, during ping test (and I think tempest as well but not 100% sure) the keys to the created servers are stored in an env in mistral 13:15:18 <thrash> jaosorior: +1000 13:15:34 <d0ugal> jaosorior: I didn't know swift had that option, sounds like a good (and easy?) starting point. 13:15:39 <jaosorior> thrash, d0ugal, apetrich: Would you guys be able to dedicate some time to move those to swift? 13:15:50 <thrash> jaosorior: somebody can, yes. :) 13:16:07 <jaosorior> d0ugal, to be able to do that, we probably need barbican in the undercloud, but that's something alee and me can work on. 13:16:11 <d0ugal> jaosorior: we are going to do some planning soon, so we could open a bug for this and consider it then 13:16:50 <jaosorior> d0ugal, apetrich, thrash: So, having moved those environments to be stored in swift. Would that be the last bits of sensitive info stored in mistral? 13:16:54 <owalsh> if it's only used the the pingtest/tempest key do we care? 13:17:29 <thrash> jaosorior: I think from a tripleo perspective, that's a good bet. 13:17:31 <apetrich> owalsh, not only those keys unfortunately 13:17:40 <owalsh> apetrich: ack 13:17:55 <jaosorior> owalsh: it sure depends on the user that pingtest/tempest uses. If it's heat-admin it's problematic, since it's able to do sudo su. 13:18:02 <d0ugal> jaosorior: do you could storing for 48 hours as storing? :) 13:18:19 <owalsh> jaosorior: runs as stack AFAIK 13:18:29 <d0ugal> jaosorior: we also probably need to do some checking of the logs and/or protection there against future leaks 13:18:32 <jaosorior> d0ugal: I need to double check on that one. lhinds what do you think? 13:18:38 <jaosorior> d0ugal: definitely 13:18:47 <lhinds> jaosorior: just reading.. 13:19:41 <lhinds> I guess time could be configurable for now (if that's what you were refering to) 13:20:08 <lhinds> or log integrity? 13:20:38 <jaosorior> Log integrity is something we should cover, so we should report any issues as mistral bugs and get those fixed. 13:20:58 <jaosorior> lhinds: but currently mistral stores the heat environments (which might contain sensitive info) for a limited time (48 hours) 13:21:11 <jaosorior> lhinds: is this something we can live with, or should we also avoid this? 13:21:17 <openstackgerrit> Harald Jensås proposed openstack/tripleo-heat-templates master: Add ctlplane networking for routed networks https://review.openstack.org/547326 13:21:21 <d0ugal> FWIW, fixing this in Mistral will likely be very hard. 13:21:48 <alee> o/ 13:21:51 <lhinds> so it would be difficult to encrypt the heat envs? 13:22:06 <lhinds> (stored in mistal) 13:22:28 <d0ugal> lhinds: I think so, mistral internally duplicates them in a few places to optimize db lookup 13:22:51 <lhinds> d0ugal: ack 13:23:10 <jaosorior> d0ugal: I thought the generated heat environments were all stored in swift. 13:23:21 <lhinds> so i think as far as time periods, any time window is a potential exploit window (although shorted better of course) 13:23:27 <d0ugal> jaosorior: they are - but while the workflow is running and for 48 hours after they are also in Mistral 13:23:45 <jaosorior> d0ugal: is it possible to disable that? 13:24:20 <d0ugal> jaosorior: yes, they could be deleted when the workflow finishes, but it is extremely useful for debugging etc. 13:24:30 <d0ugal> We actually increased the time, the default is 1 hour irrc 13:24:34 <d0ugal> iirc* 13:24:46 <jaosorior> d0ugal: how is it useful for debugging? 13:25:22 <d0ugal> jaosorior: when the execution is stored you can inspect it and find out exactly what happened, what inputs and outputs happened at every point in the workflow 13:25:31 <d0ugal> jaosorior: you can even restart workflows in the middle etc. 13:25:33 <lhinds> has there been any BP / LP for encrypting heat envs stored in mistral (so it's on the radar so to speak). I could take a look at the code, can't promise anything as new to mistral 13:25:55 <d0ugal> it is a bit like having the interactive debugger you have in most programming languages (but via a rest api :)) 13:25:59 <jaosorior> d0ugal: What about making that attribute configurable? In the hardening docs we could then tell folks to lower that time, or disable it entirely. 13:26:14 <lhinds> but with a key in barbican, it should be doable. 13:26:27 <d0ugal> jaosorior: it is configured by instack-undercloud, can users change those puppet settings? 13:26:45 <jaosorior> should be possible 13:26:49 <jaosorior> depending on how it's configured 13:27:30 <jaosorior> Need to double-check if the instack-undercloud hieradata takes precedence or the hieradata overrides do. but it should be doable. 13:27:56 <d0ugal> lhinds: there was a blueprint for mistral for securing secrets. I think both rbrady and thrash had a look at doing it. So they know more about that than me. 13:28:00 <jaosorior> #action For now, we will document how to lower the time mistral stores heat environments and add it to the hardening guide. 13:28:39 <d0ugal> jaosorior: FYI, here is the setting: https://github.com/openstack/instack-undercloud/blob/master/elements/puppet-stack-config/puppet-stack-config.yaml.template#L671 13:28:51 <jaosorior> #link https://github.com/openstack/instack-undercloud/blob/master/elements/puppet-stack-config/puppet-stack-config.yaml.template#L671 13:28:51 <lhinds> d0ugal / rbrady / thrash if you manage to dig it out (the BP) please paste if for me. 13:28:58 <d0ugal> lhinds: looking for it. 13:29:02 <lhinds> thanks d0ugal 13:29:12 <d0ugal> lhinds: https://blueprints.launchpad.net/mistral/+spec/mistral-secure-sensitive-data 13:29:28 <lhinds> so configurable as first port of call, and then ideal future functionaility to encrypt 13:29:31 <d0ugal> See the spec linked at the top and there was a patch, but I think that got stuck. 13:30:45 <lhinds> so there is a fair whack of code there, any reason for the abandon by Brad? 13:31:02 <d0ugal> jaosorior: should I open a bug for the mistral environments? 13:31:09 <jaosorior> d0ugal: that would be great 13:31:14 <d0ugal> k, on it 13:32:07 <alee> d0ugal, I'm having trouble finding the actual spec .. 13:32:27 <d0ugal> alee: https://specs.openstack.org/openstack/mistral-specs/specs/pike/approved/secure-sensitive-data.html 13:32:34 <lhinds> alee: spec has gone missing, but some code here: 13:32:36 <lhinds> https://review.openstack.org/#/c/459747/ 13:32:44 <jaosorior> #link https://specs.openstack.org/openstack/mistral-specs/specs/pike/approved/secure-sensitive-data.html 13:32:46 <alee> ah cool thanks 13:32:47 <d0ugal> I think the spec was moved because it missed the openstack release 13:32:57 <d0ugal> Which is a bad idea it seems :) 13:33:50 <lhinds> k, found the spec: 13:33:50 <jaosorior> Alright, but at least for the short term we have a plan 13:33:53 <lhinds> #link https://github.com/openstack/mistral-specs/blob/master/specs/pike/approved/secure-sensitive-data.rst 13:34:10 <jaosorior> * Move all sensitive data to swift (to have it all in one place) 13:34:23 <lhinds> ok, brad is thrash, got it now 13:34:27 <jaosorior> * Document how to reduce time mistral stores heat environments) 13:34:28 <thrash> lhinds: :D 13:34:32 <d0ugal> #link https://bugs.launchpad.net/tripleo/+bug/1757430 13:34:33 <openstack> Launchpad bug 1757430 in tripleo "The ssh_keys and tripleo.undercloud-config Mistral environments should be move to swift" [High,Confirmed] 13:34:46 <jaosorior> and then we can focus on securing swift instead, which already can encrypt with barbican. 13:35:00 <jaosorior> d0ugal: awesome 13:35:32 <jaosorior> thanks 13:36:28 <d0ugal> np 13:36:33 <jaosorior> Anything else someone wants to bring up about this topic? 13:37:09 <lhinds> nothing from me this week 13:37:18 <jaosorior> ok 13:37:31 <openstackgerrit> Harald Jensås proposed openstack/python-tripleoclient master: Fix Genconfig - no HOME in environment https://review.openstack.org/554678 13:37:41 <jaosorior> Thanks d0ugal, thrash and apetrich for joining 13:37:57 <jaosorior> #topic Work progress udpate 13:38:17 <d0ugal> jaosorior: np, thanks for the input! 13:38:50 <jaosorior> Just a heads up for folks in the squad, there are a bunch of reviews for different items in the etherpad https://etherpad.openstack.org/p/tripleo-security-squad (Maybe we need to come up with an easier way to track those) 13:38:55 <jaosorior> so reviews are appreciated 13:39:26 <jaosorior> Right now, most of the work that I've been doing has been on enabling TLS by default (which hopefully almost merges for the undercloud https://review.openstack.org/#/c/552382/ ) 13:39:47 <jaosorior> I'm also working on enabling it by default in the overcloud, so if someone is intersted in joining that work or testing, let me know. 13:40:13 <jaosorior> that's all on my side. 13:40:35 <alee> jaosorior, I'll probably ping you about joining that work later today or tomorrow 13:40:44 <jaosorior> alee: awesome 13:41:07 <jaosorior> #topic Any other business 13:41:17 <jaosorior> Anything else someone wants to bring up to the squad? 13:41:33 <alee> jaosorior, I think we wanted to do a quick meeting to identify secrets to be secured/ passwords etc. 13:41:50 <alee> jaosorior, did we want to schedule that? 13:41:58 <jaosorior> alee: that would be good. 13:42:17 <jaosorior> alee: Any day/time preference? 13:42:34 <alee> jaosorior, how about tommorow? 13:42:44 <jaosorior> works for me 13:42:49 <openstackgerrit> Martin André proposed openstack/tripleo-common master: Pass connection info via ansible config file https://review.openstack.org/554526 13:42:53 <alee> morning my time -- say 10 am EST? 13:43:08 <jaosorior> alee: that works for me. 2pm UTC 13:43:12 <jaosorior> lhinds: does that work for you? 13:43:34 <lhinds> jaosorior: thats fine for me 13:44:25 <lhinds> I have a work shop thing, but might be able to leave a little early 13:44:32 <lhinds> (it's remote) 13:45:13 <jaosorior> lhinds, alee: I'll poke you tomorrow then before the time. 13:45:16 <jaosorior> Anybody else is welcome to join 13:46:12 <jaosorior> Anything else someone would like to bring up? 13:47:11 <jaosorior> Alright 13:47:15 <jaosorior> thanks everyone for joining! 13:47:17 <jaosorior> #endmeeting