21:03:36 <ttx> #startmeeting crossproject
21:03:36 <morganfainberg> jeblair, welcome to ping-as-as-service fun facts?
21:03:37 <openstack> Meeting started Tue Feb  3 21:03:36 2015 UTC and is due to finish in 60 minutes.  The chair is ttx. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:03:39 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:03:41 <openstack> The meeting name has been set to 'crossproject'
21:03:42 <mestery> lol
21:03:47 <ttx> Our agenda for today:
21:03:51 <ttx> #link http://wiki.openstack.org/Meetings/CrossProjectMeeting
21:04:04 <ttx> #topic Horizon reviews for project (e.x. Sahara, Trove) panels (SergeyLukjanov)
21:04:09 <nikhil_k> o/
21:04:11 <ttx> SergeyLukjanov: awake ?
21:04:18 <SergeyLukjanov> ttx, yup
21:04:21 <ttx> SergeyLukjanov: you really shouldn't be, but you are
21:04:38 <ttx> So... Some changes in specific panels in Horizon seem to linger a bit
21:04:44 <ttx> I think this is due to a priority mismatch, with Horizon core focusing on the.. well.. core
21:04:53 <ttx> And because it's difficult for them to review the change functionally
21:04:56 <SergeyLukjanov> so, the question is very simple - how to manage change requests from different projects that have own panels to horizon
21:05:03 <ttx> Not sure how to best solve that mismatch...
21:05:18 <ttx> it won't get any better as we add more projects that may want Horizon panels
21:05:21 <SergeyLukjanov> yeah, unfortunately no good proposals on improving it
21:05:35 <jungleboyj> o/
21:05:36 <david-lyle> There are a couple of reasons for the lag generally
21:05:50 <SergeyLukjanov> but it's going to be an issues when new functionality couldn't be supported on the dashboard side
21:06:03 <david-lyle> first, they aren't targeted to any milestone, so we aren't watching for them
21:06:34 <david-lyle> two, the reason above, harder to verify the nuanced changes of other projects without subject matter experts
21:06:45 <david-lyle> and as a bonus item
21:06:55 <david-lyle> we've grown, a lot
21:07:11 <david-lyle> as the community continues to grow
21:07:13 <asalkeld> sounds a bit like tempest reviews
21:07:18 <SergeyLukjanov> the first ones seems to be able to be solved, but the second one sounds like a much bigger problem to solve
21:07:21 <SergeyLukjanov> yeah
21:07:28 <ttx> there is the option of having each project be responsible of their panel, but quality would be lower (and it's generally not the same people writing core code and Horizon panel code)
21:08:01 <dhellmann> ttx: would that be outside of the main horizon tree?
21:08:11 <ttx> dhellmann: a bit like tuskar-ui
21:08:24 <dhellmann> ok
21:08:24 <asalkeld> can you get the same functionality out of tree?
21:08:38 <dhellmann> it seems like that's a reasonable approach, but I don't know if the technical details work out
21:08:39 <SergeyLukjanov> sahara-dashboard was merged into the horizon during the prev. cycle :)
21:08:46 <david-lyle> horizon has a fairly mature plugin mechanism
21:09:18 <david-lyle> so it would be feasible to have UI plugins for services sit in a separate repo and installed onto Horizon nodes
21:09:25 <asalkeld> also there is benefit from having ui experts reviewing your code
21:09:26 <ttx> dhellmann: I think horizon-folk would still have to advise on how to best do one
21:09:38 <david-lyle> issues are UX consistency, translation, quality
21:09:44 <SergeyLukjanov> even if we'll have the project specific stuff out of the tree - will be horizon folks able to review changes to help keep some code quality?
21:09:50 <thingee> ttx: not sure how much has changed with horizon since it has been a while, but if each project's panel is a django app, can't these just live in tree of the projects? Like we plan to do with tempest?
21:09:51 <ttx> but maybe they could directly write some (say for Nova and other core-compute stuff), and support the work of others
21:09:55 <thingee> ttx: maybe I'm crazy
21:09:56 <SergeyLukjanov> david-lyle, ++
21:10:37 <ttx> david-lyle: we have a number of horizontal projects which are moving towards a "handle directly less projects, but provide tools and advice for all" model
21:10:50 <annegentle> to me it's a similar problem to the docs reviews-- and we manage to keep an eye on quite a few repos so far
21:11:00 <annegentle> so yes, what ttx said
21:11:16 <ttx> david-lyle: you would decide who you handle directly
21:11:16 <david-lyle> I worry UX is a different beast
21:11:22 <jeblair> at least having a "foo-dashboard" repo facilitates overlap between foo and horizon teams
21:11:29 <ttx> david-lyle: and provide advice for all the others rather than write them all
21:11:52 <cburgess> I would also point out that there arer panel in horizon today that have to touch multiple projects. The launch instance flow for instances can touch nova, neutron, and cinder. So I'm not sure how you would classify what is in core and what is in a project specific panel.
21:11:59 <david-lyle> ttx, honestly most are written externally now and merged into tree
21:12:14 <ttx> david-lyle: so the problem is more.. maintenance ?
21:12:22 <david-lyle> yes
21:12:23 <jeblair> cburgess: yes, i think those are good candidates for a basic layer supported in horizon tree
21:12:25 <thingee> cburgess: good point
21:12:36 <david-lyle> many projects are providing the maintenance as well
21:12:55 <david-lyle> core review load hasn't scaled to support the diversity
21:13:05 <SergeyLukjanov> ttx, maintening and improving
21:13:06 <cburgess> So to pick on cinder.. I could see a "volume" panel as being project specific, but the launch instance flow stuff is core. So there is defiantly overlap.
21:13:28 <david-lyle> additionally, when we get rather mature code merges, the UX can be inconsistent and harder to maintain
21:13:37 <thingee> cburgess: what about creating a volume from an image :)
21:13:46 <cburgess> Right that too.
21:13:51 <cburgess> It gets complicated.. quickly.
21:14:02 <david-lyle> but by that point it can be too far in the process
21:14:12 <thingee> or backup a volume to an object store
21:14:20 <ttx> david-lyle: it's a complex issue. I don't think we'll find the solution today, but I think that's the larger challenge for Horizon today
21:14:26 <ttx> largest*
21:14:30 <thingee> scratch the backup example
21:14:35 <david-lyle> I agree
21:14:53 <david-lyle> the big-tent scares me in that regard
21:14:56 <ttx> find its role and place in the larger tent
21:15:28 <david-lyle> either we look like we have 17 UI teams that throw something together, or we figure a better way to scale
21:15:36 <ttx> david-lyle: like I said, other horizontal projects have decided to move more in a tools+mentoring mode, but I see how that may not be applicabel to Horizon
21:15:56 <david-lyle> I think the latter is the correct approach, but details are pesky
21:16:39 <ttx> I think it's a discussion you should have with your team and see how to reorganize to support the future
21:16:51 <david-lyle> already have started
21:16:55 <ttx> not sure there is much more we can do today
21:17:02 <ttx> SergeyLukjanov: ?
21:17:12 <SergeyLukjanov> yeah
21:17:24 <david-lyle> it's a good point and not lost on me, been grappling with it for a while
21:17:29 <ttx> anything else on that topic ?
21:17:38 <SergeyLukjanov> david-lyle, could we followup with blueprints and issues?
21:17:46 <david-lyle> sure
21:17:52 <SergeyLukjanov> david-lyle, I mean to ensure that they aren't missed on the milestones?
21:17:57 <ttx> yes, the targeting coordination sounds solvable short-term
21:17:59 <david-lyle> that would be ideal
21:18:26 <ttx> ok, moving on
21:18:27 <ttx> #topic openstack-specs discussion
21:18:28 <SergeyLukjanov> david-lyle, okay, I'll take a look on how it could be done and contact you
21:18:33 <SergeyLukjanov> ttx, david-lyle thx
21:18:36 <ttx> * Add TRACE definition to log guidelines (https://review.openstack.org/#/c/145245/)
21:18:37 <david-lyle> thanks SergeyLukjanov
21:18:41 <ttx> Sergfethx for staying up
21:18:55 <ttx> SergeyLukjanov: ^
21:19:04 <ttx> sdague: around?
21:19:07 <sdague> ttx: yes
21:19:31 <sdague> so this is the first add after the major log guidelines, which is basically to take back the TRACE keyword
21:19:41 <sdague> which we've been randomly jamming into stack traces
21:19:50 <ttx> No opposition so far, so I raise it today to see if we can move on to TC rubberstamping next week
21:19:56 <sdague> and use it for trace level logging (i.e. < DEBUG)
21:20:02 <ttx> or if this needs a few more cycles
21:20:12 <dansmith> sdague: +10^9
21:20:24 <ttx> my impression is that this doesn't have enough PTLs +1 yet
21:20:42 <fungi> i was personally surprised the first time i saw a project logging stack traces/tracebacks as "trace" log level
21:21:02 <sdague> fungi: well, it kind of isn't. It's just jammed into the oslo log format string
21:21:10 <sdague> slightly back doored
21:21:14 <fungi> yeah
21:21:24 <fungi> but yeah, not what that's for
21:21:42 <ttx> sdague: if you do another revision for any reason, i would add trace definition in the bullet list
21:21:51 <sdague> anyway, please express opinions. This will clearly take longer amount of time because we'll need an oslo change
21:21:55 <ttx> but not worth losing the curent votes over
21:22:09 <dhellmann> sdague: did you have a chance to look at the implementation comment I left?
21:22:10 <sdague> but I think it's the right long term direciton
21:22:24 <sdague> dhellmann: I have not yet, I will loop back around on it
21:22:46 <dhellmann> sdague: nothing critical, but we have to be careful about assuming that everything can use oslo.log because of the dependency cycles
21:22:53 <sdague> ok
21:22:59 <ttx> I read the current silence on this as consent, but we actually need +1 for consent :)
21:23:34 <Rockyg> It would be really good to get a couple of seasoned operators' opinions from the operators list on this one.  I can try to get some more attention...
21:23:51 <sdague> Rockyg: this should not impact ops
21:23:51 <ttx> Rockyg: that would be great
21:24:04 <sdague> the expected use of this is at a level way below what ops should set at
21:24:06 * dhellmann hopes operators are not running with that much output
21:24:17 <dansmith> yeah, not sure this is an operator thing
21:24:17 <cburgess> sdague: Well several of us find stack traces very useful. As long as we can still get them without having to always run sub-debug level.
21:24:26 <Rockyg> Most ops are running at debug
21:24:27 <dhellmann> cburgess: this isn't about stack traces, though
21:24:36 <cburgess> OK
21:24:38 <cburgess> Haven't read the spec.
21:24:40 <dansmith> cburgess: I think the point is, not using TRACE for stack traces at all
21:24:43 <cburgess> and yes... we run at debug level.
21:24:50 <cburgess> OK thats cool.
21:24:54 <dansmith> cburgess: we want TRACE back for actual tracing
21:25:01 <jeblair> yeah, stack traces should be at >= error
21:25:03 <Rockyg> ++
21:25:08 <morganfainberg> dansmith, ++
21:25:13 <cburgess> Yeah so I agree with that then. Move stack traces to debug and let operators choose debug if they want.
21:25:14 <ttx> Free TRACE!
21:25:20 <sdague> jeblair: agreed
21:25:24 <cburgess> or error
21:25:36 <sdague> cburgess: no, stack traces should be errors, I think that's in the first log spec
21:25:44 <sdague> because that should never be an ok thing
21:25:45 <cburgess> Thats fine with me.
21:25:51 <jeblair> cburgess: yes, certainly that if you are running at debug, you should see stack traces
21:26:00 <dhellmann> cburgess: http://git.openstack.org/cgit/openstack/openstack-specs/tree/specs/log-guidelines.rst
21:26:08 <cburgess> OK I'm on the same page.
21:26:13 <cburgess> I don't see how this is an operator issue then.
21:26:32 <sdague> cburgess: http://git.openstack.org/cgit/openstack/openstack-specs/tree/specs/log-guidelines.rst#n246 more specifically
21:26:35 <cburgess> stack traces still happen, debug is debug, and trace will be a new thing. Seems fine to me.
21:26:54 <ttx> More on this topic ?
21:27:01 <sdague> just ... go vote on it
21:27:08 <ttx> yeah
21:27:59 <ttx> #topic rootwrap overhead - should we stay on this plan (sdague)
21:28:03 <bknudson> just a warning that code that's not tested (e.g., if trace isn't enabled) often winds up broken
21:28:13 <ttx> For those who missed the previous episodes, rootwrap was built for privilege separation, not really for performance
21:28:22 <ttx> Some services make heavy use of rootwrapped system calls, so the low performance impacts them
21:28:40 <ttx> and apparently as a result, some operators patch it out (although that's the first time I hear of that, I could use some details)
21:28:45 <dhellmann> are those projects adopting the daemon mode work you did?
21:28:49 <ttx> This basically means they are running the rootwrap-using nodes as root, which IMHO is not a great idea
21:28:58 <ttx> dhellmann: no
21:29:01 <ttx> The problem was *already* addressed on oslo.rootwrap side in Juno, with a high-performance daemon mode which is actually even faster than shelling out to sudo.
21:29:10 <ttx> But that change was not picked up by Neutron or Nova yet
21:29:25 <ttx> Neutron has a Kilo spec for it
21:29:26 <cburgess> I'll fall on the grenade here.
21:29:31 <ttx> sdague: but you had other suggestions ?
21:29:41 <sdague> ttx: 2 things
21:29:46 <thingee> ttx: or cinder https://review.openstack.org/#/c/149677/
21:29:52 <cburgess> So we did extensive performance testing of nova-network nova-compute in the G cycle with sudo and rootwrap.
21:29:54 <mestery> ttx: For Neutron, we plan to get rootwrap daemon mode in by Kilo-3.
21:29:56 <cburgess> Its wasn't even close.
21:30:08 <cburgess> So we patched back in the ability to select sudo.
21:30:09 <ttx> cburgess: not the daemon mode, I suspect
21:30:17 <cburgess> Nope not daemon mode. Wasn't an option yet.
21:30:35 <cburgess> So I will freely admit that our current practice is based on old performance data.
21:30:42 <ttx> cburgess: no question the "normal" mode sucks. It was designed for a couple calls in nova
21:30:56 <ttx> Neutron calls it a few hundreds times when creating a network
21:31:09 <cburgess> The daemon mode sound interesting and is something I would like to play with. But until nova and neutron support it (the two heaviest users of rootwrap) I don't know that it should be the only option.
21:31:23 <ttx> See perf impact at https://review.openstack.org/#/c/81798/
21:31:28 <sdague> yeh, nova is going to land a patch to make sudo selectable
21:31:30 <jogo> nova is holding off supporting daemon mode until it lands in neutron
21:31:47 <sdague> the other issue is the rootwrap policies are full of some pretty big holes
21:31:55 * dhellmann hopes mestery doesn't say neutron is waiting for nova
21:32:02 <dansmith> sdague: it's landed
21:32:06 <cburgess> Right so the patch to allow sudo again was added (by me) because a recent setuptools change undid the hack in PBR to make rootwrap perfomant. This resulted in gate failures due to timeouts.
21:32:12 <mestery> lol
21:32:14 <mestery> No, we plan to land that in kilo-3
21:32:27 <mestery> The work slipped from kilo-2, but I have high confidence it will land in kilo-3
21:32:31 <ttx> you realize "running with sudo" is the same as running as root security-wise ?
21:32:33 <dhellmann> mestery: early enough that nova is likely to be able to land it, too?
21:32:34 <sdague> nova-compute's policies actually make 0 sense because there are at least half a dozen own the box holes in it
21:32:47 <ttx> since you grant the user the ability to run anythign under sudo
21:33:04 <mestery> dhellmann: I'm not sure about that. The services split caused the author a slight headache I think, but we have a plan now on it.
21:33:05 <dhellmann> sdague: is that an issue with rootwrap, or with the policy definitions?
21:33:10 <cburgess> ttx: Yup I'm aware.
21:33:17 <ttx> with policy definitions
21:33:24 <dhellmann> sdague: like, do you see it as fundamental?
21:33:34 <sdague> dhellmann: it's an issue with the assumption that some of these services can be priv separated
21:33:37 <ttx> almost everything uses CommandFilter which sucks
21:33:43 * Rockyg I thought ttx was going to say running with sudo was like running with scissors
21:33:45 <cburgess> But as sdague pointed out you  can do that anyways with the current policies. Its a risk we take on and have to be sure we are aware of audit.
21:34:31 <ttx> cburgess: not for all the nodes though. I agree that the network and compute nodes have policies that basically let you do anything, because nobody takes the time to fix them
21:34:39 <dhellmann> sdague: because they need to do so many things as root?
21:34:42 <ttx> but an api or a metadata node are pretty secure
21:34:43 <sdague> dhellmann: yes
21:34:43 <cburgess> ttx: Agreed and we only patch it back in for nova.
21:35:10 <cburgess> ttx: We don't use neutron yet, but I suspect we would for that as well, though I will need to benchmark neutron and the new daemon mode in kilo.
21:35:10 <ttx> I wish that instead of patching it out people would spend more time fixing the policies or adopting the daemon mode
21:35:24 <dhellmann> ok, well, yeah, if there are real cases of that we should figure out if we can just run a root-level service with a small application-specific api of some sort
21:35:39 <dansmith> cburgess: IIRC, you can't use it with nova yet unless you make nova's rootwrap support support daemon mode
21:35:47 <cburgess> dansmith: Right
21:35:57 <dhellmann> sdague: but when I suggested neutron do that instead of building this general purpose thing into rootwrap they said that would be too hard
21:36:08 <dhellmann> the surface area may be different for nova's needs
21:36:11 <dansmith> fixing the policy definitions and supporting daemon mode are one thing,
21:36:14 <cburgess> So seems like we should get nova using daemon mode then do a 3 way performance analysis. If daemon mode is as good as it looks remove the sudo option again.
21:36:29 <dansmith> but I think that precluding support for sudo in the tree is not a necessary policy
21:36:35 <dhellmann> dansmith: ++
21:37:04 <cburgess> dansmith: daemon mode and policy fixes are separate things.
21:37:06 <dhellmann> on the other hand, then we have to ask which mode we'd test with
21:37:17 <cburgess> Also this feel like a nova and maybe neutron issue and not something that has to impact all projects.
21:37:21 <dansmith> cburgess: the two are required to demonstrate any advantage to using rootwrap, was my point
21:37:35 <cburgess> dansmith: Agreed
21:37:46 <ttx> dansmith: it was done at a time when performance was not so much an issue, and we hoped people would just write better filter definitions
21:37:47 <dansmith> dhellmann: we can test with rootwrap if we want, that's fine.. I don't think there is much risk there for the more relaxed case breaking
21:37:55 <dansmith> ttx: yeah, I understand
21:38:09 <ttx> the sad truth is, people don't care taht much about privilege separation
21:38:28 <ttx> neutron (which can still use sudo) is most often run with it
21:38:43 <fungi> it's also worth considering the risk profile of letting these services do things locally as root. in situations where they're the only thing running on the machine, it may not actually be buying you much to prevent giving them control over all the other things that system isn't actually doing
21:38:44 <morganfainberg> ttx, people may not care but that is because there hasn't been a reason to care (yet, thankfully)
21:38:45 <dansmith> well, and there are people that don't want to solve the problem that way anyway
21:39:00 <dansmith> we spend (too much) time getting our SELinux policies super tight
21:39:02 <sdague> well it only helps if you *really* know you've got policy that doesn't let you get out
21:39:17 <ttx> dansmith, sdague: so the patch is not patching out, it's just allowing to use sudo back again ?
21:39:18 <dansmith> fungi: exactly
21:39:23 <morganfainberg> but with the isolation fungi just described (many people run that way) it is less important to have the isolation
21:39:26 <sdague> ttx: correct
21:39:33 <dansmith> ttx: yes
21:39:34 <ttx> sdague: i'm fine with that
21:39:35 <cburgess> ttx: Yes its an option, uses rootwrap by default.
21:39:44 <ttx> matches what Neutron has
21:39:48 <fungi> what rootwrap is mostly buying is safety from someone coercing a service into doing things it wasn't intended to do, but preventing root access is a poor proxy for that sort of protection
21:40:07 <morganfainberg> this is a nice middleground fix allow sudo where people want to use sudo
21:40:09 <dhellmann> sdague, ttx, dansmith : should we push that up into the oslo.rootwrap code itself, so applications only have to deal with one api?
21:40:13 <sdague> fungi: right, agreed. which is why I wanted to raise it as an issue
21:40:22 <ttx> rootwrap doesn't enforce being the only option...
21:40:30 <dansmith> dhellmann: that would be a library option to ... not do anything? :)
21:40:34 <morganfainberg> ttx but nova not supporting anything else does.
21:40:37 <ttx> so Nova can definitely support both options
21:40:43 <morganfainberg> ttx, exactly
21:40:47 <cburgess> My work here is done.
21:40:51 <ttx> it's nova-side code anyway
21:40:52 <dhellmann> dansmith: a library option to use a function that calls sudo instead of our wrapper, so nova doesn't have to care at all
21:41:00 <sdague> because it seems like we're doing a lot of work here, to provide options, but as no one is really auditing the policies effectively, it seems.... not effective
21:41:12 <dhellmann> sdague: ++
21:41:29 <dhellmann> dansmith: and also instead of nova and neutron both having their own version of that option, and wrapper code
21:41:30 <sdague> it's like belt and suspenders, but no pants
21:41:39 <dansmith> dhellmann: well, my point is, it seems weird to have the library have a short-circuit option, because if I want to use sudo I could just snip that import line from nova and be good, instead of having to install the library for no reason, right?
21:41:42 <ttx> It's also worth noting that in many cases, the root call is not even necessary
21:41:50 <ttx> it's just convenient
21:42:04 <jogo> sdague: I once brought this up with one of the 'security' groups in OpenStack and they just shrugged and didn't want to audit
21:42:05 <dhellmann> dansmith: it makes rootwrap become "run this as root somehow" instead of "use your wrapper to run this"
21:42:17 <dansmith> I'd really like to be able to say "use sudo" and not even have the library imported as native support in the tree
21:42:25 <fungi> things like selinux and apparmor (and even just careful application of extended attributes and cgroups-imposed namespacing) are probably closer to where the real safety net should be, but that's a lot more complicated and varying cross-platform
21:42:30 <dhellmann> dansmith: why make every application author implement that?
21:42:42 <dansmith> dhellmann: I suppose, just seems overkill, but meh
21:42:43 <morganfainberg> jogo, "containers or isolation of concerns mostly mitigates the issue" i mean - to be fair if someone compromised neutron they could still do all neutron could do w/ rootwrap.
21:42:44 <ttx> sdague: you mentioned capabilities too
21:42:52 <morganfainberg> which isn't a small amount of breaking things.
21:42:54 <sdague> fungi: bingo, that seems like where the effort should be spent to me, honestly
21:42:58 * dhellmann takes off his Killing Code Duplication hat
21:42:58 <dansmith> fungi: yep
21:43:10 <ttx> sdague: I think it's not completely crazy to grant he neutron user rights over networky stuff
21:43:27 <ttx> and shave a few thousands rootwrap calls
21:43:31 <sdague> ttx: I was told neutron was moving to a model like that at the nova midcycle, maybe mestery can speak up
21:43:36 <ttx> to me it's a complentary approach
21:43:40 <fungi> except that networky stuff can often be parlayed into full root access to the system anyway
21:43:40 <dansmith> dhellmann: code dedup is good, but less-friggin-libraries-required wins for me every time :)
21:43:43 <ttx> complementary*
21:43:54 <bknudson> there's at least a review in progress to get selinux in triploe - https://review.openstack.org/#/c/108168/
21:43:57 <bknudson> tripleo
21:44:01 <ttx> fungi: you stole my bullet :)
21:44:08 <mestery> sdague: To be honest, we're hoping to get rootwrap in and I'm hoping we can discuss these aspects at the summit with the broader community
21:44:09 <dhellmann> dansmith: meh. Work on more than one project for a while.
21:44:18 <mestery> As someone said, we shouldn't solve htis one way in neutron and another in nova
21:44:23 <fungi> definitely one of those classic security arguments where the perfect can be the enemy of the good, but not sure we've really got candidates for either perfect or good
21:44:42 <mestery> fungi: ++
21:45:13 <morganfainberg> fungi, +∞
21:45:39 <fungi> the main thing i see rootwrap doing is letting some deployers feel like they have control over what a service can or can't do, but it's the "false sense of" variety of security
21:46:08 <ttx> I  would pursue all options. Allow nova to run sudo directly for the less security-conscious. Enable rootwrap daemon mode for those who want a decent tradeoff. Add capabilities to shave a thousand rootwrap calls. Improve the rootwrap filter definitions so that they actually filter
21:46:18 <sdague> fungi: yeh, that's a big concern for me
21:46:20 <fungi> and i have doubts that many deployers/operators are fine-tuning the default supplied rootwrap rules to begin with
21:46:20 <sdague> I mean - https://github.com/openstack/nova/blob/master/etc/nova/rootwrap.d/compute.filters#L39
21:46:24 <cburgess> ttx: +1
21:47:11 <ttx> sdague: I've been removing that one at least once in the past
21:47:19 <ttx> it just keeps on reappearing
21:47:35 <cburgess> Dear god that line scares me
21:47:35 <bknudson> needs a hacking check
21:47:44 <sdague> well there is also dd, cp in there
21:47:45 <morganfainberg> cburgess, haha
21:47:50 <ttx> and chown
21:47:58 <cburgess> Yeah
21:48:04 <cburgess> Wow I just really started looking at that file.
21:48:05 <sdague> yeh, like I said, it's kind of pointless
21:48:07 <jeblair> there seems to be a reviewer education problem
21:48:10 <cburgess> I'm going to go back to my safe pretend world now.
21:48:28 <ttx> the nova-compute node is running as root basically
21:48:33 <sdague> ttx: right
21:48:37 <morganfainberg> ttx, shhhh
21:48:41 <morganfainberg> ttx, you'll scare someone
21:48:45 <dansmith> sdague: the qemu-nbd one is probably the worst in there
21:48:47 <ttx> rootwrap just gives you a framlework to fix that, it's not a work-free bullet
21:48:49 <morganfainberg> :P
21:48:55 <dansmith> sdague: export /dev/sda as an nbd device.. game over :)
21:49:08 <morganfainberg> cburgess, what has been seen cannot be unseen. though yelling lalalalala helps sometimes.
21:49:08 <sdague> yeh, there are so many game overs
21:49:26 <sdague> which is why I'm not sure this approach works
21:49:56 <ttx> sdague: buggy filters make it just as unsecure as the option of using sudo directly
21:50:13 <morganfainberg> ttx, might make it worse. because it provides the sense of security falsely
21:50:32 <ttx> morganfainberg: true
21:50:49 <ttx> but most people think "running under a non-root user" makes them safe too
21:51:24 <ttx> The main issue to me is.. even when we fix them (I think I did it twice over the years) people keep on re-breaking them
21:51:24 <mriedem> did i already miss any discussion on bug 967832 ?
21:51:27 <ttx> difficult to gate on it
21:51:29 <sdague> anyway, it seems like on the compute nodes we should just admit it's basically game over, not sugar coat it at this level. Do it at the selinux/apparmor level
21:51:32 <ttx> mriedem: no
21:51:50 <ttx> sdague: or fix the filters
21:51:59 <dansmith> honestly,
21:52:05 <dansmith> there are two valuable things on a comptue node
21:52:11 <dansmith> the images with other people's data, and the mq credentials
21:52:21 <dansmith> if you get the latter, you can do anything you want to the entire deployment
21:52:28 <dansmith> both of those things are readable by nova's user anyway
21:52:57 <morganfainberg> sdague, i have opiinion on selinux and apparmor, but putting the enforcement at that level seems waaaaay more sane [regardless of which] than rootwrap
21:52:57 <sdague> yep
21:53:03 <dansmith> if you're running your company's imap server alongside nova-compute, you're in for more trouble anyway
21:53:12 <lifeless> !
21:53:28 <fungi> from a vmt perspective, having a handle on this and publicizing that it "does not do the things you think" for securing a deployment would help reduce what could otherwise become needless vulnerability reports
21:53:28 <ttx> We have one more topic to cover
21:53:32 <morganfainberg> dansmith, the mq-creds should be guarded with signed messages..
21:53:36 <cburgess> So I shouldn't be running my our payroll processing and credit card processor on the same server and my openstack cloud?
21:53:37 <morganfainberg> dansmith, but thats another story
21:53:41 <dansmith> morganfainberg: should be but they're not
21:53:52 <morganfainberg> dansmith, lets plan to circle back on that at L summit?
21:53:56 <morganfainberg> dansmith, i want to see that gap closed.
21:54:00 <ttx> cburgess: you should run them in containers on the same machine
21:54:06 <cburgess> morganfainberg +1
21:54:07 <morganfainberg> dansmith, will bug you on that front a bit later on this cycle?
21:54:10 <dansmith> morganfainberg: it's been on the list for a long time, but.. yeah :)
21:54:12 <morganfainberg> cburgess, you too.
21:54:18 <cburgess> ttx: Yeah I was joking and we are moving to containerizing everything.
21:54:37 <sdague> because containers never have exploits... :)
21:54:41 <ttx> cburgess: because that is safe (joking too)
21:54:42 <morganfainberg> sdague, ever.
21:54:48 <morganfainberg> sdague, EVAR
21:54:55 <ttx> OK, let's cover the last topic quick
21:54:59 <ttx> #topic Bug 967832 (mriedem)
21:55:00 <cburgess> ttx: I'm terrible at sarcasm. morganfainberg can verify this
21:55:08 <ttx> and I'm French
21:55:11 <ttx> #link https://bugs.launchpad.net/nova/+bug/967832
21:55:14 <ttx> #link http://lists.openstack.org/pipermail/openstack-dev/2015-February/055801.html
21:55:15 <morganfainberg> ttx, he's worse at it :P
21:55:20 <ttx> mriedem: you have 5 min :)
21:55:20 <cburgess> LOL
21:55:30 <mriedem> so morganfainberg already commented on this in the ML,
21:55:38 <mriedem> basically this is a bug across projects since grizzly,
21:55:52 <mriedem> looks like there was an approved spec in juno in neutron that didn't land code and wasn't re-proposed for kilo,
21:56:11 <morganfainberg> i actually like the concept of putting this into middleware that can receive registered "do X on event Y"
21:56:16 <mriedem> wondering if everyone is in agreement about fixing this thing and then looking for best solutions since it's going to be probably similar impl in the various projects
21:56:22 <morganfainberg> i would be 100% behind that.
21:56:41 <cburgess> mriedem: From an operator perspective I can tell you that this bug sucks, badly.
21:56:51 <mriedem> so nova tells middleware to call nova delete on an instance when a project that instance is running under is gone?
21:56:53 <fungi> it might make tempest devs a lot happier too
21:56:56 <morganfainberg> mriedem, yep.
21:57:05 <jogo> mriedem: yeah, callbacks
21:57:06 <morganfainberg> mriedem or spool, it for a periodic
21:57:14 <mtreinish> fungi: yes it would, it makes the cleanup story much easier
21:57:20 <mriedem> but...does nova need the tenant to exist for the delete to happen, or does middlware process the callback before deleting the tenant?
21:57:24 <morganfainberg> mriedem up to nova to figure it all out the best way to not crater itself
21:57:24 <notmyname> mriedem: middleware in what project? nova or keystone?
21:57:34 <lifeless> I think you need a bit of both
21:57:34 <morganfainberg> notmyname, i'd put it as a package in keystonemiddleware
21:57:35 <mriedem> notmyname: keystone
21:57:37 <lifeless> you want events for latency
21:57:38 <morganfainberg> but not in auth_token
21:57:56 <morganfainberg> notmyname, s/package/module
21:57:59 <notmyname> thanks
21:58:04 <cburgess> I don't think you would need the tenant to exist.
21:58:09 <lifeless> and you want either a lossless history or a diff style check to deal with servers that were down etc
21:58:11 <mriedem> did anyone look at the WIPs from neutron in juno? https://review.openstack.org/#/q/status:abandoned+project:openstack/neutron+branch:master+topic:bp/tenant-delete,n,z
21:58:13 <cburgess> Each project would only cleanup its own local resources.
21:58:16 <mriedem> mestery: do you remember those? ^
21:58:20 <jogo> cburgess: agreed, making sure tenant exists makes this a lot more complex
21:58:24 <cburgess> The problem we have is cross project relationships. Like volumes attached to VMs.
21:58:37 <cburgess> But maybe we don't care.
21:58:49 <notmyname> middleware sounds risky since (1) it requires coordination for things that can't be coordinated (eg what happens if an instance fails to delete?) and (2) it will _really_ slow down the delete call to keystone
21:58:51 <cburgess> Maybe we just take destructive do it anyways actions because its all being nuked.
21:59:12 <morganfainberg> notmyname, it's a callback - i assume nova would spool it for a delete job that is periodic
21:59:16 <bknudson> are there any other examples where a server needs to keep track of state in another server?
21:59:16 <mriedem> does multi-tenancy muck this up at all
21:59:17 <morganfainberg> not a 'do it right now'
21:59:17 <mriedem> ?
21:59:24 <morganfainberg> mriedem, not really.
21:59:25 <mriedem> can i have volumes attached to my instance from another project?
21:59:33 <notmyname> an alternate solution would be a background keystone scrubber that looks at what's been deleted and sends the related commands to different services (projects)
21:59:35 <morganfainberg> mriedem, god i hope not
21:59:54 <dhellmann> notmyname: ++
21:59:55 <mriedem> notmyname: yeah, but how long does that keep casting?
21:59:56 <mtreinish> mriedem: I don't think so, that sounds like a recipe for disaster
21:59:57 <morganfainberg> notmyname, sure - i'd just say whatever would be a listener that takes a callback
22:00:14 <notmyname> morganfainberg: I'm still not clear on what you mean by "callback
22:00:22 <morganfainberg> notmyname, so we publish the framework you register a function to do something
22:00:25 <lifeless> notmyname: or the projects could do the checking
22:00:28 <notmyname> and I'm considering what happens when someone is deleted from keystone and has a PB of data in swift
22:00:34 <jogo> notmyname: you register a function for the keystone janitor to call upon tenant deletion
22:00:38 <lifeless> notmyname: making keystone not have to know whats dependent on it
22:00:57 <lifeless> notmyname: which would be better I think
22:00:57 <bknudson> keystone_middleware.on_project_delete(function_to_call)
22:00:58 <ttx> Out of time -- Looks like that discussion can continue on the thread? We can cover it at next week meeting if necessary ?
22:01:08 <morganfainberg> ttx ++
22:01:10 <mriedem> sure
22:01:11 <notmyname> lifeless: but keystone does know. that's the whole point of keystone. it knows everything
22:01:26 <ttx> Feel free to continue to argue on #openstack-dev though
22:01:27 <morganfainberg> notmyname, no keystone doesn't know what depends on it.
22:01:30 <morganfainberg> anyway
22:01:32 <ttx> we just ned to vacate channel
22:01:34 <jogo> notmyname: it doesn't know how much data a user has in swift.
22:01:39 <ttx> No time for open discussion
22:01:44 <ttx> I'll just paste the link to the 1:1 syncs we had today, focused on the kilo-2 tag
22:01:50 <ttx> #link http://eavesdrop.openstack.org/meetings/ptl_sync/2015/ptl_sync.2015-02-03-09.06.html
22:02:03 <ttx> #endmeeting