18:00:46 #startmeeting sahara 18:00:47 Meeting started Thu Feb 5 18:00:46 2015 UTC and is due to finish in 60 minutes. The chair is SergeyLukjanov. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:00:48 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 18:00:50 The meeting name has been set to 'sahara' 18:00:52 sahara folks, hey! 18:00:56 o/ 18:00:58 yo/ 18:00:58 hello/ 18:01:00 Hi! 18:01:03 hi 18:01:22 hello 18:01:25 hi 18:01:36 okay, let's start 18:01:37 #link https://wiki.openstack.org/wiki/Meetings/SaharaAgenda 18:01:39 hi 18:01:46 hi 18:01:47 #topic sahara@horizon status (crobertsrh, NikitaKonovalov) 18:01:54 #link https://etherpad.openstack.org/p/sahara-reviews-in-horizon 18:02:05 I've updated the etherpad a few days ago 18:02:20 I had a couple bug fixes merge, but mostly the same status 18:02:33 so, looks like we have a bunch of patches merged and that's awesome 18:02:49 Yes, we will take any sort of progress :) 18:02:55 so, do we have something urgent? 18:03:20 Nothing that I would call "urgent" at the moment that I know of 18:03:26 okay 18:03:35 crobertsrh, anything else on this topic? 18:03:52 Just be sure to take a look at: https://review.openstack.org/#/c/147677/ 18:04:00 That is the "guided cluster creation" 18:04:17 ack, it's on my todo list 18:04:45 #topic News / updates 18:04:49 folks, please 18:05:07 #info kilo-2 is mostly here, /me will release rest of the components today 18:05:36 spark-swift integration is merged, but there is still a question of the jackson version compat issue. We can talk about it in open discussion. Essentially, wait for spark community to fix it (ongoing) or carry our own spark assembly for a while 18:06:16 security doc chapter is coming along, i'm hoping to have a review up soon. i also have some ideas about how we can integrate barbican for some of our credentials storage. 18:06:24 also, the group working on cdh plugin has identified maybe a shortfall in Java EDP support for hbase, where an extra classpath value needs to be set. I am trying to repro on a CDH cluster so I can verify and spec a change if we need one 18:06:29 #link https://bugs.launchpad.net/openstack-manuals/+bug/1415218 18:06:41 for anyone who wants to keep track 18:06:53 patch for quota checks, new integration tests 18:07:09 Working to integrate Sahara with TripleO. There's a puppet review ongoing here: https://review.openstack.org/#/c/145509. Thanks to those who have already reviewed; any more reviews are useful. A tripleo-image-elements commit is likely soon, and a tripleo-heat-templates commit afterward. 18:08:12 Working on Sahara integration test for service - key store value 18:08:17 in https://review.openstack.org/146659 we noticed that launch_command.py (for spark jobs) was not in the MANIFEST.in. Do we need to backport that to Juno? 18:08:41 egafford: I'll take a look tomorrow, I suppose 18:08:56 egafford, yay! 18:09:18 Nikolay_St: Thanks. SergeyLukjanov: :) 18:09:41 any other updates? 18:09:51 let's move on 18:09:54 #topic How to improve Horizon changes 18:10:06 Interested in answer to tmckay's question re: launch_command.py. 18:10:19 so, we've discussed a bit this topic on the last cross-project meeting 18:10:19 we can take it up in open discussion 18:10:27 tmckay: Sane and just. 18:10:38 +1: we should improve horizon changes 18:10:44 :) 18:10:49 and agreed that we need to at least be sure that all our changes we need to merged into horizon 18:11:03 has own blueprints and bugs in horizon targeted to the milestone 18:11:17 it should increase review rate a lot 18:11:39 to do it I think we should make some process of inter-command prioritization 18:12:02 and ping horizon team to accept high-prio things to the milestones 18:12:03 What do you mean by "process of inter-command prioritization" 18:12:19 I mean to discuss the urgency of things inside the sahara team 18:12:34 and then push the list to the horizon team 18:12:44 and always ensure that milestones assigned 18:12:54 Ok, so maybe take our etherpad of outstanding changes, order them by priority and ping #horizon-people 18:13:58 Is there a go-to person on the horizon side? Is an in-channel request all we need, or should we send an email? 18:14:00 and ensure that all of them has own issues and blueprints 18:14:14 crobertsrh: ping me 18:14:23 I know they were talking about this issue, but I missed this week's meeting. 18:14:25 I think that we could initially ask david-lyle to go through the etherpad 18:14:27 thanks david-lyle :) 18:14:32 david-lyle, oh, hey :) 18:14:34 david-lyle, thanks 18:14:45 saw horizon, came running 18:14:52 :) 18:15:20 I just need to be made aware of items you are needing, then I can prioritize them in horizon 18:15:40 crobertsrh, could you please update the etherpad with bugs/blueprints and links to reviews? 18:15:41 Sounds fine, david-lyle. 18:15:49 I can do that 18:16:01 crobertsrh, cool, thx 18:16:07 links to reviews should contain the bp/bugs links in them already 18:16:23 I can add them to the etherpad though if that somehow speeds things up 18:16:27 david-lyle, you're not using specs process in horizon? 18:16:42 not yet, will be moving to that in lemming 18:16:59 lemming, that's the next release? 18:17:00 david-lyle, nice, so, it'll be easier for us before Lemming 18:17:03 tmckay, nope 18:17:20 http://surveymonkey.com/r/openstack-l-naming voting for the next release name 18:17:24 I hope not 18:17:34 closing next week, hurry up to vote 18:17:38 * tmckay wipes brow 18:17:43 there was some talk about lemming being a poor choice on the ml 18:18:01 could be Love, so weigh your options carefully 18:18:10 +1 London 18:18:36 -4 to Love, I don't want to say this word while talking with customers ;) 18:18:48 okay, let's move on 18:18:50 lol 18:18:52 there is a town here called Lizard Lick 18:19:01 * david-lyle goes back to Horizon land 18:19:02 #topic Open discussion 18:19:10 david-lyle, thx for participating 18:19:13 i'd like to talk about deployments 18:19:28 as they pertain to advice we will be giving in the security doc 18:19:34 okay, spark assembly issue for me. and backport manifest tweak necessary for Juno? 18:20:36 are there any objections to us recommending that sahara controllers be deployed into cloud instances? 18:21:14 elmiko, as opposed to running on the openstack controller? 18:21:27 is that bad for some reason? ^^ 18:21:37 maybe after that the daemon splitting is in place... 18:21:43 yea sorta, as opposed to being deployed directly on a host connected to the cloud 18:22:07 i'm being asked to create a recommended deployment strategy 18:22:52 and security-wise it sounds like it's easier to recommend deploying to cloud instances as the security issues will be elevated past the host related issues. 18:22:56 does that make sense? 18:23:31 does anyone have opinions about sahara deployment in production? 18:23:31 elmiko, what do you mean by 'cloud instances'? 18:23:52 alazarev: a server spawned by openstack as opposed to a host connected directly to the cloud 18:24:28 for example 18:24:29 https://mimccune.fedorapeople.org/data_processing.png 18:24:43 i shared that diagram as a basic example of a deployment 18:25:20 the pushback i got was if i was implying that sahara needs to be a separate host connected to the cloud, or could it be an instance within the cloud. 18:25:49 elmiko, do you want to recommend dedicated node for sahara controller? 18:25:51 i don't think it _needs_ to be a separate host, so i don't have a problem recommending installing to a cloud server(was instance) 18:26:21 elmiko, tiny issue that I've seen in a case with Sahara running inside the stack -- the credentials for the admin tenant for the stack are inside of the sahara.conf on the VM. Maybe not a big deal, but the creds are there in addition to only being on the openstack controller node (if you run Sahara as an openstack service) 18:26:23 alazarev: well, i think that _we_ need to recommend something as a team. 18:27:16 i don't see any difference in deploying sahara in single or distributed mode into cloud servers. 18:27:17 elmiko, as I see sahara controller is usually run on all openstack controller nodes, never saw other deployment 18:27:41 alazarev, +1 18:27:42 alazarev: so, on the same machines as say a nova controller? 18:27:59 elmiko, yeap 18:28:08 ok, but here's the question. _why_ does it need to be deployed on controller nodes? 18:28:09 alazarev, at worst it's a vm on the controller node. 18:28:30 we have a dep on the neutron l3-agent somewhere, right? 18:28:37 afaik it just needs access to the control plane 18:28:42 elmiko, why not? 18:28:49 for netns exec 18:29:30 elmiko, doesn't have to be. but if you run it somewhere else, not deployed by openstack, you have to add it to the service catalog. (it may be that I'm not up on my commercial deployments, mostly dev env) 18:29:43 it needs access to control place and to VMs ssh 18:29:48 alazarev: from the brief conversation i had with bdpayne in openstack-security i think he prefers recommending it installed as an instance because it make security auditing easier. (that's my impression, he had to go before we could finish) 18:30:34 mattf: couldn't it still access neutron if it was inside a vm? 18:31:26 elmiko, i don't have details cached in anymore. to handle namespaces you need to be able to netns exec, which means living in the same kernel instance 18:31:27 elmiko, which part of neutron? it would be hard to use netns for example 18:31:40 tmckay: +1; I've had success running Sahara from a separate node. elmiko: I believe it is certainly TripleO's strategy to allow "optional" services like Sahara to run in or outside of controller nodes whenever possible, so if we are strictly dependent in prod on being a process on the controller node itself, this'll be important to know. 18:32:45 ok, so netns access is one solid check in favor of needing to be deployed on a controller instead of a cloud server. does that sound accurate? and can anyone point me towards documentation? 18:33:45 elmiko, also we need admin access to keystone, it can be restricted to private network 18:33:47 so the next question becomes, is there a way to allow netns access from a cloud server, and if there is does that create a competing sec issue 18:34:00 that makes the whole thing not worth doing 18:34:23 sorry folks, need to disconnect right now 18:34:24 i think part of this stems from the impression that sahara is an application that runs on top of the cloud 18:34:29 #chair alazarev 18:34:30 Current chairs: SergeyLukjanov alazarev 18:34:31 alazarev, +1, that touches on the "admin cred" issue I mentioned earlier 18:34:35 #chair tmckay 18:34:36 Current chairs: SergeyLukjanov alazarev tmckay 18:34:55 elmiko, i don't think we have a doc that talks about the deployment considerations when using namespaces. we just mention there's a config you need to set. 18:34:58 alazarev, tmckay, as long as the vm had a route to the identity service it shouldn't matter where it runs 18:35:16 SergeyLukjanov, ciao 18:35:41 elmiko, yes, but this will require additional efforts to make such route 18:35:43 elmiko, I think his point is that if you put it on the controller, you *could* restrict keystone to private net 18:36:06 alazarev: agreed 18:36:09 heh - https://ask.openstack.org/en/question/56906/saharacant-login-to-nodes/ 18:36:15 but it could be done 18:36:28 mattf: nice 18:37:58 netns is not a silver bullet since it can't be used for HA mode 18:38:22 netns seems like a corner case to me, but it's a configuration that might not work well within an instance 18:38:31 this is why I've started indirect access feature 18:38:36 when would you _need_ to use netns? 18:38:52 indirect access seems like a security win to me 18:40:06 elmiko, the only thing I don't like in current implementation is "ssh over ssh' which is too slow, port forwarding would be much faster 18:40:14 yea 18:40:14 indirect access is just awesome 18:40:22 <3 it from the beginning 18:41:00 so, is our recommendation that operators install sahara on a controller node as opposed to a cloud instance? 18:41:37 bdpayne: are you available? 18:42:16 elmiko, I believe so, all installations I saw were in such way 18:42:20 I think we would have to try the cloud instance as a reference architecture and see how it works 18:42:45 ok, i can try it out 18:42:46 but to date, I agree with alazarev. I always thought Sahara was meant to be on the controller nodes 18:43:13 right, we've always installed it on controller nodes. but for a security recommendation is it _required_ to be on controller nodes, that's the real question. 18:43:40 elmiko: If the admin creds weren't on each Sahara node, I'd say it'd be an easy win to suggest Sahara be run in cloud nodes (assuming your config doesn't require netns). Given that the admin creds are on each Sahara node, though, it does seem a lot dodgier of a proposition. 18:43:45 i think the argument is that running things on cloud instances provides an easier way to manage the controllers. 18:43:47 elmiko, I don't think it is _required_ 18:43:50 elmiko, if the cloud instance model is viable and it has demonstrable security benefits, maybe we recommend it. Or maybe we present it as an option with pros and conds 18:44:19 i think this is a recommendation we will need to have in the sec.doc. 18:44:57 egafford: i guess i don't see much difference between the creds being on a vm in the cloud or on a controller attached to the cloud. exploiting either machine would be bad. 18:45:24 keep in mind, i'm not suggesting creating images with the creds preloaded. 18:45:31 elmiko, I'm here now 18:45:51 elmiko: Right, that would be absurd. Don't think anyone's suggesting that. 18:46:13 bdpayne: awesome, we are having some difficulty with the concept of running sahara on a controller as opposed to a cloud instance. would you be able to talk about the pros/cons of those deployments from a sec perspective? 18:46:40 bdpayne: by default we have always run on controllers 18:47:02 hey, I was just reading the backlogs 18:47:07 why are admin creds needed? 18:47:29 I would argue that you don't want the admin creds in an instance 18:47:33 I forget just what errors you get without them. I think it's for auth checks 18:47:38 so that may be what pushes it back onto the control plane 18:47:40 alazarev, ^^ do you know? 18:47:58 at the least we need admin creds for creating trusts 18:48:03 but if you don't need them, all the better and I'd certainly argue for putting this in an instance 18:48:06 that is true 18:48:27 could the trusts be setup once at install time and then just used? 18:48:31 or is this an ongoing need? 18:48:41 ongoing, we create them dynamically under some configs 18:49:11 * tmckay should document the errors that arise when admin creds are missing 18:49:29 interesting 18:49:52 ok, so more generally, of course running a service on the control plane has much greater security impact 18:50:03 if that service is compromised, then it could potentially be a stepping stone to more sensitive parts of the cloud 18:50:10 this is why I tend to prefer things in instances 18:50:26 but, it sounds like that may not be viable with the current design 18:50:36 it may be useful to lay out some of these considerations in the security guide 18:50:42 what do we mean by 'admin' here? sahara admin or openstack admin? 18:50:56 and explain under which situations someone may be able to deploy as an instance 18:50:58 in case people want to do that 18:51:09 I'm talking about openstack admin 18:51:11 i think we'll need to do more testing on the instance model 18:51:32 admin is used to update sahara objects in background, to create trusts, etc. 18:51:55 there is also a config option that uses net namespaces and we're not sure if those will work in a cloud instance 18:53:04 so yeah, I think laying out the security concerns and showing how to best mitigate them is the right path for the security guide 18:53:14 even if the end result is something on the control plane 18:53:52 ok, i'll have to do more digging around the sec doc to see if i can leverage the advice there for control plane installations 18:54:58 bdpayne: thanks for the advice, it gives me some food for thought 18:55:10 np! 18:55:41 only 5 min left, should we talk about tmckay's questions? 18:56:13 alright, opinions on spark assembly class conflicts for swift access. Short answer, people in spark are working on it, but I don't know when it will be in a release. Looks like maybe 1.3 18:56:39 so, are we okay with the current fix, to patch the classpath and hope it works? Or do we carry our own spark assembly for DIB for a little while? 18:56:58 tough question 18:57:01 my gut is to wait for 1.3, and fix on fail if someone comes up with a use case that shows we need a new assembly 18:57:11 i'm ok with that 18:57:16 me too 18:57:17 +1 for wait 18:57:21 I'm ok with that too 18:57:49 okay. it works, but we know it could break, but we haven't seen that. Works for me. 18:57:51 thanks. 18:57:53 Other question 18:58:03 tmckay: Hi tmckay, i have replied your email about the masternotfound error 18:58:12 we left launch_command.py out of MANIFEST.in 18:58:15 huichun, thanks 18:58:45 launch_command.py lives in edp/resources, so I'm guessing it's not in a distro in Juno 18:58:56 do we need to backport that fix? 18:59:01 1 min left 18:59:03 i think so 18:59:13 what is the best way to check, with setuptools on a juno branch and bdist? 18:59:18 or sdist? 18:59:32 tmckay: Yes, we should. 18:59:32 makes me think that spark edp in Juno can't be working 18:59:37 just install to a clean virtualenv and look in the site-packages 18:59:39 but we haven't heard that 18:59:53 On the RH side: https://bugzilla.redhat.com/show_bug.cgi?id=1184522 18:59:59 let's switch to #openstack-sahara 19:00:01 #endmeeting