15:07:53 <mnaser> #startmeeting tc
15:07:54 <openstack> Meeting started Thu May 14 15:07:53 2020 UTC and is due to finish in 60 minutes.  The chair is mnaser. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:07:55 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:07:57 <jungleboyj> :-)
15:07:58 <mnaser> let's keep that open just to record
15:07:58 <openstack> The meeting name has been set to 'tc'
15:08:06 <njohnston> +1 thanks
15:08:14 <mnaser> so afaik all proposed goals have landed
15:08:28 <gmann> yeah, we have two proposed goal merged - https://governance.openstack.org/tc/goals/proposed/index.html
15:08:31 <gmann> #link https://governance.openstack.org/tc/goals/proposed/index.html
15:08:37 <diablo_rojo> The log here are definitely a good idea. We should send anything we decide to the ML for people not in attendance too though.. and give it 24 hours to really finally decide since not everyone could make it and this is kinda impromptu?
15:08:52 <gmann> one goal zuulv3 migration is already selected fr V cycle, we need to select one more
15:09:04 <mnaser> diablo_rojo: i think what we should do is propose a change with those goals we select, and then send out that review which will be under formal-vote rules too
15:09:25 <diablo_rojo> mnaser, not quite, there are still a handful of projects that didn't finish their docs in time
15:09:33 <mnaser> so technically we have no choice but waiting for 7 days to land it anyways, which gives us indirectly community approval?
15:09:35 <diablo_rojo> openstackansible being one of them ;)
15:09:44 <mnaser> diablo_rojo: the patches are up there i promise D:
15:09:49 <gmann> njohnston: any idea if we can get ralonsoh here to know his opinion on single or multi cycle goal for rootwrap ?
15:10:01 <diablo_rojo> mnaser, oh okay I mustve missed them in last audit
15:10:07 <mnaser> diablo_rojo: they came up in the past few days
15:10:25 <ralonsoh> hi
15:10:27 <gmann> rootwrap goal is much user benefits and mock goal is easy and doable.
15:10:34 <gmann> ralonsoh: hi thanks for joining
15:10:36 <mnaser> FWIW, on this topic, njohnston proposed the following change wrt goal proposal, so something to think about -- https://review.opendev.org/#/c/724142/
15:10:38 <diablo_rojo> mnaser, ah alright, cool. Thank you :) I think we are still missing a couple projects though. So its close, but not completed.
15:11:12 <jungleboyj> Yeah, we had talked about rootwrap and mock before and those both seemed good.
15:11:24 <gmann> ralonsoh: we were discussing last time also and today. will rootwrap can be single cycle goal or need multi cycle ?
15:11:33 <mnaser> fwiw, there's nothing stopping us from having 3 goals in total.
15:11:33 <ralonsoh> multi cycle
15:11:40 <jungleboyj> Most likely multi
15:11:48 <ralonsoh> even for one single project, like neutron, we'll need more time
15:12:04 <ricolin> mock goal isn't that hard but require 10+ patches in single project to fix all
15:12:05 <diablo_rojo> Neither one of those is particularly user facing (which I know is a thing we've considered in the past) but I think both are good options.
15:12:05 <mnaser> ralonsoh: i think the good thing is that many projects have already started using privsep though, so the infrastructure is there
15:12:14 <ralonsoh> mnaser, exactly
15:12:26 <ralonsoh> and that makes migration process easier
15:12:47 <gmann> true
15:12:54 <mnaser> something to note about privsep is that it's possible it may be a lot less of a case where we don't control our own destiny
15:13:05 <mnaser> for example, if you migrate something to privsep, and use a library instead of a bunch of system shell outs
15:13:14 <mnaser> and then that library is missing a feature or something, it may block us from progressing forward
15:13:44 <ralonsoh> you can still use a CLI command but with privsep
15:14:13 <mnaser> ralonsoh: right, but to an extent that might not be _that_ much more different than rootwrap, but maybe i may be missing some points
15:14:24 <ralonsoh> exactly
15:14:39 <mnaser> so it wouldn't be security hardening other than renaming to use something else
15:15:04 <jungleboyj> I believe, from what I know, that even using privsep with a CLI is better than rootwrap.  I could be wrong though.
15:15:16 <ralonsoh> well, the security in rootwrap is just because you parse the command
15:15:17 <njohnston> I could imagine an exception list for issues such as that as they come up, much as we have an exception list for projects that need to stay on py27
15:15:43 <ralonsoh> with privsep you are forcing the linux capatibilities to use
15:15:49 <ralonsoh> that's an improvement
15:15:57 <mnaser> actually that reminds me of a post that mikal wrote a while back
15:16:16 <mnaser> https://www.madebymikal.com/how-to-make-a-privileged-call-with-oslo-privsep/
15:16:20 <gmann> yeah and new way present for all projects is at least good
15:16:21 <mnaser> maybe we can reference that somewhere
15:16:55 <ralonsoh> mnaser, I'll take a look at this article
15:17:03 <diablo_rojo> Woudl be a good thing to include in the goal proposal
15:17:12 <fungi> yes, directly translating rootwrap rules to privsep rules doesn't appreciably improve security
15:17:27 <mnaser> dhellman seems to have suggested a while back to add this to docs https://twitter.com/mikal/status/993672440470372352 :)
15:18:14 <mnaser> i want to look at nova nd see how much rootwrap usage they have
15:18:16 <fungi> but at least if the rules are using privsep, that's a step toward more easily tightening them up
15:18:17 <gmann> +1 oslo have these info will be helpful
15:18:48 <jungleboyj> Yeah.
15:19:06 <clarkb> privsep is also a performance benefit too iirc because it isn't forking a new python process each call?
15:19:25 <mnaser> clarkb: technically we had rootwrap-daemon to do the same
15:19:28 <ralonsoh> yes, we spawn a daemon the first time it's called
15:19:52 <ralonsoh> and then we use a socket to this daemon
15:19:52 <clarkb> ah I thought all that work went into privsep, didn't realize rootwrap got similar
15:19:52 <mnaser> but rootwrap daemon couldn't call privilged functions, only shell outto things
15:20:19 <mnaser> so from a performance benefit of "can i use pyroute2 in privsep and use internal apis vs fork out "ip link set...""
15:20:21 <mnaser> it benefits a aton
15:20:29 <njohnston> Given that smcginnis is committed to making the mock change - and it's already well underway - I think that it makes more sense to get started with the privsep goal.  This seems especially true for projects that have a very small contributor base and may need a long time to work on changing patterns from a simple CLI call to a library.
15:20:35 <mnaser> https://github.com/openstack/nova/blob/master/etc/nova/rootwrap.d/compute.filters -- does this tell me rootwrap is fully out of nova btw?
15:20:49 <jungleboyj> njohnston: ++
15:20:51 <mnaser> https://github.com/openstack/nova/commit/909d0de68edcb232c0d0ca28b755806f7f3780bc
15:20:54 <ttx> privsep is faster that the daemon, as it does not spawn a new process for the command called
15:20:56 <mnaser> this patch says nova is alread ydone
15:21:37 <ttx> so rootwrap (python+command) < rootwarp-daemon (command) < privsep (inline python)
15:21:44 <mnaser> off the top of my head, nova and neutron are the two big targets because they need to do things as root.  other projects probably should run in user space anyways?
15:22:09 <gmann> njohnston: +1
15:22:14 <mnaser> maybe cinder too for operations that involve mounting things and doing copies
15:22:24 <ttx> yes nova is fully privsepped. Just needs the phase 2 work (properly rewrite commands into python)
15:22:38 <gmann> nova is done yea, but neutron too is completely migrated ?
15:22:52 <mnaser> https://github.com/openstack/neutron/tree/master/etc/neutron/rootwrap.d -- neutron has not even moved to privsep
15:22:54 <njohnston> mnaser: I wonder if nova just needs to clean some docs up: http://codesearch.openstack.org/?q=rootwrap&i=nope&files=&repos=openstack/nova
15:22:57 <jungleboyj> Cinder is impacted as well.  We are part way through the migration though.
15:22:58 <mnaser> sorry, let me correct
15:23:04 <mnaser> neutron still has rootwrap usage
15:23:05 <ralonsoh> gmann, no, not neutron
15:23:26 <mnaser> ok, how about we split this goal into 2?  goal #1 -- move everything into privsep but not necessarily have to worry about reimplementing it with libraries
15:23:37 <mnaser> goal #2 -- remove all usage of exec() unless explicitly necessary
15:23:47 <mnaser> and #1 should be easy and if nova did it then anyone can do it :P
15:23:50 <ttx> os-brick is also privsepped, but with execs()
15:23:58 <mnaser> (and we have an example of a project which did it too)
15:24:04 <diablo_rojo> That seems like a cleaner approach to making it all one goal and making that span two releases.
15:24:09 <ttx> mnaser: yeah that is what I was thinking
15:24:12 <jungleboyj> mnaser: ++
15:24:18 <jungleboyj> I like that approach.
15:24:19 <diablo_rojo> Only to find out people wait to do part 1 until the end of the second release..
15:24:28 <ttx> You can do a one-cycle goal of moving stuff to privsep that does not involve rewriting every command
15:24:31 <mnaser> cause looking at some of the stuff that neutron will have to do will require a lot of time
15:24:39 <mnaser> lots of dnsmasq interactions and what not
15:24:49 <ttx> then a goal of getting rid of all bad usage of privsep once you are closer  that getting that completed
15:25:07 <ttx> and in the mean time, you can spend a few cycles getting the large ones closer to phase 2
15:25:07 <njohnston> ralonsoh: what do you think about reframing the goal in this way?
15:25:16 <ralonsoh> njohnston, ok with this
15:25:23 <mnaser> yep.  i like that a lot.  it also makes it a lot less intimidating in terms of security and trying to think about how to do it without execs
15:25:27 <ralonsoh> about neutron, just the 1:1 conversion will take more time
15:25:27 <ttx> they do not have to be done one right after the other
15:25:29 <gmann> this also sounds good idea or targeting set of projects in goal phases ?
15:26:00 <ttx> So if be end of phase 1 we identify some very large amounts of work, maybe we can get it started on specific projects before setting it as a goal
15:26:24 <ttx> Goals are good for getting everyone to a given level, not so great when amount of work is vastly different across projects
15:26:44 <mnaser> given the situation we're in, if we split this out to "just convert to privsep and keep execs (unless you want to eliminate them)" + mock + zuul.  i think that's a fair "load" at our contributors
15:26:44 <njohnston> ^^ excellent point ttx
15:26:55 <mnaser> the main reason being many projects have accomplished these goals already to varying degrees
15:27:06 <ttx> I think phase 1 in one cycle is totally doable
15:27:16 <mnaser> afaik mock efforts were already mostly underway in nova, most of nova already have zuul v3 jobs, an privsep already done
15:27:21 <jungleboyj> mnaser: ++
15:27:23 <gmann> mnaser that will be too much for one cycle. mock can  go without goal in that case
15:27:27 <diablo_rojo> Then maybe a gap before phase two to get others with more work ready for it.
15:27:34 <ttx> phase 2... it's doable in one cycle *if* we get a few cycle to have the largest targets caught up
15:27:36 <diablo_rojo> But that can be decided exactly later.
15:27:44 <ttx> diablo_rojo: ++
15:27:45 <gmann> zuul migration can be lot of work for many projects
15:27:48 <mnaser> i dont want to stress our teams but i also feel like some teams literally won't have to do these projects
15:28:01 <mnaser> like i dont think MOST of our services will be affected by rootwrap
15:28:01 <ttx> gmann: hopefully not the same as the privsep one :)
15:28:11 <jungleboyj> ++
15:28:14 <ttx> Like privsep phase 1 is basically done for nova
15:28:22 <mnaser> i.e. i dont think heat ever needed to run things as root ever :)
15:28:29 <gmann> yeah, and adding mock there might be lot of work for them
15:28:29 <mnaser> and heat already had zuul v3 jobs
15:28:35 <ricolin> I think phase1 + zuul +mock indeed sounds like fair load as mnaser mentioned, and we need more discussion on what's detail for phase two IMO
15:28:36 <gmann> in term of review and all.
15:28:43 <ricolin> mnaser, yes
15:28:45 <jungleboyj> Cinder should be able to get phase 1 done pretty easily.
15:28:59 <mnaser> well perhaps we should investigate the affect projects and we might realize that the overall load per project won't be big
15:29:12 <mnaser> cause i think that for the most part, projects will end having to do only 1/3
15:29:17 <mnaser> (cause the other two are likely already done for example)
15:30:05 <mnaser> like i think we need to assume that most project will be noop to 1 or 2 out of these 3.  if most projects will not be noop to these 3 goals, we need to drop one
15:30:07 <gmann> "convert to privsep and keep execs (unless you want to eliminate them)"  + zuul' can be good try though there are still chance project might miss but we are giving them goals in start of cycle
15:31:13 <gmann> there are lot of legacy jobs for many projects. In my xenail->bionic migration it was around >50 % if i remember correctly
15:31:13 <mnaser> right but how many have already moved to zuulv3? and how many even use rootwrap in the first place
15:31:46 <mnaser> gmann: i think maybe we might have a different perception given that you're much closer in your QA interaction though, so i'll take your word on our zuulv3 progress
15:32:14 <mnaser> the way i see it is we have 1 goal everyone will most likely have to do (zuulv3) and two that most people will likely have to do 1 out of
15:32:51 <mnaser> i'm looking at rootwrap usage and it seems minimal as i scroll through projects
15:32:59 <mnaser> a lot of projects list it in their dependencies for some reason, but no actual usage
15:34:09 <mnaser> i just wanted to pick a smaller project like zun that interacts with the system and even they use privsep: https://opendev.org/openstack/zun/src/branch/master/etc/zun/rootwrap.d/zun.filters
15:35:09 <fungi> technically, the privsep phase 1 goal can really be the "get rid of rootwrap" goal, because there might be cases where projects were doing things as root which would be better done in other ways, and don't actually need either rootwrap or privsep
15:35:13 <gmann> zun is one of most active and up to date one you checked  :)
15:35:17 <mnaser> magnum has no rootwrap usage, heat has no rootwrap usage, watcher has no rootwrap usage
15:35:42 <mnaser> i'm trying to pick random arbitrary projects, i may suck at naming random ones but most of them have no actual need to use rootwrap with the exception of system interacting services such as nova/cinder/neutron
15:35:47 <diablo_rojo> Easy goal for them then lol
15:35:58 <njohnston> mnaser: You have to look for utils.execute with run_as_root as true http://codesearch.openstack.org/?q=utils.execute&i=nope&files=&repos=openstack/zun
15:35:59 <mnaser> and nova is already done with it
15:36:32 <jungleboyj> :-)
15:36:40 <mnaser> njohnston: does that mean they have inaccurate filter file? oh
15:37:14 <njohnston> oh, sorry, I did not see it was the filter file you linked to
15:37:14 <mnaser> njohnston: they do use it, it seems: https://opendev.org/openstack/zun/src/branch/master/zun/common/privileged.py
15:37:27 <mnaser> they just have zun.common as a whole privileged space
15:37:40 <mnaser> (not the best idea but at least rootwrap ain't htere)
15:37:47 <mnaser> that's why i feel like adding the phase of rootwrap isn't that much more load, IMHO.
15:37:57 <mnaser> which really leaves 2 goals, mock and zuulv3
15:38:04 <gmann> also if we divide it in two phase, i think doing it in consecutive cylce will be good to get he flow going otherwise it is easy to miss the second phase
15:38:29 <mnaser> I agree.
15:38:30 <njohnston> we might want to add to the goal proposal saying 'best practice is to strictly scope your PrivContext space'
15:38:40 <gmann> and we select the phase2 as the W cycle goal in advance? or preselect
15:38:57 * fungi suggests "recommended practice" or "standard practice" over "best practice"
15:39:29 <njohnston> fungi: sure, sounds good
15:39:45 <mnaser> gmann: I don’t have an opinion on that right now but not opposed to that
15:39:48 <fungi> the term "best practice(s)" always makes me twitch
15:40:52 <gmann> mnaser: yeah something we can discuss in PTG for W.
15:41:15 <mnaser> I agree. We can put it there and nothing stops us from changing it later
15:41:20 <njohnston> manila is an instance where there is a fair - but not overwhelming - number of instances to fix, for example: https://opendev.org/openstack/manila/src/branch/master/etc/manila/rootwrap.d/share.filters http://codesearch.openstack.org/?q=utils.execute&i=nope&files=&repos=openstack/manila
15:41:42 <mnaser> njohnston: ah yes that’s another system interacting one inmissed
15:41:50 <mnaser> Do they use zuul v3?
15:42:13 <mnaser> Like if it’s just one project thatmight be hit harder...
15:42:21 <gmann> ph one more things in V cycle, migration to Ubuntu 20.04. this also need community level effort
15:42:40 <mnaser> 8@ driving to the dealership now
15:42:42 <gmann> though it will be easy after all projects move to zuulv3
15:42:44 <mnaser> Oops. Wrong window.
15:43:24 <gmann> but if we need to do Ubuntu 20.04 migration in parallel that is also some work on projects side
15:43:29 <ricolin> gmann, yeah, that's something better done after zuulv3
15:43:39 <fungi> oh, yep, folks were previously in favor of calling a switch of default job node major version a cycle goal
15:43:53 <gmann> ricolin: but we have to do it in V cycle and cannot do late during release
15:43:57 <fungi> and victoria would by our current rules, need to switch from ubuntu 18.04 lts to 20.04 lts
15:44:30 <gmann> yeah, or can we move it to W cycle so that easy thing to do after zuulv3 goal
15:44:37 <mnaser> I think the system components are a lot more lax now though
15:44:44 <ricolin> gmann, what I proposed is to consider make the dependency between two goals if we need to do both of them in V
15:44:45 <mnaser> It’s not like we switched to systemd
15:45:12 <gmann> ricolin: +1 but doing dependent goal in single cycle is difficult.
15:45:14 <fungi> well, we'd need to revise the rules by which we select our test platform if we push the ubuntu focal switch out an additional cycle
15:45:34 <gmann> humm
15:45:37 <ricolin> gmann, that's true
15:46:22 <gmann> in that case, who about keep "zuulv3 + Ubuntu 20.04 migration" for V and rest all to later cycle
15:46:30 <gmann> s/who/how
15:46:55 <fungi> https://governance.openstack.org/tc/reference/runtimes/victoria.html already lists "Ubuntu 20.04"
15:47:41 <njohnston> I am good with zuulv3 + 20.04 for V, rootwrap/privsep for W and X, and mock done informally
15:47:43 <fungi> it doesn't *have* to be positioned as a cycle goal, but it's the same amount of work either way
15:47:43 <gmann> yeah, we can either move it to W cycle or select this as second goal. coordination of thus migration with zuulv3 might be difficutl though
15:48:40 <fungi> hopefully the focal migration won't be that rough, we already have established patterns for splitting xenial and bionic to different branches
15:48:50 <gmann> fungi: yeah, it need same effort as goal and i am in favor of doing it as goal so that we get enough attention. with zuulv3 it will be more of testing carefully before migration
15:48:55 <mnaser> My vote is to picking both of those and if teams express concern we can revisit it
15:49:30 <mnaser> I don’t want us to plan too far out and start working that far back. We can adjust and adapt
15:50:11 <fungi> also, the focal switch should probably happen asap so teams have all cycle to work out any bugs exposed by it
15:50:55 <gmann> that is something difficult as it will be dependent on zuulv3 migration
15:51:19 <gmann> migrating legacy job to focal and then convert them to zuulv3 is duplicate work
15:51:19 <jungleboyj> mnaser:  ++
15:51:43 <fungi> gmann: like so we can avoid having to update legacy tools/jobs to support running on focal?
15:51:57 <gmann> that is my main worry, when we finish zuulv3 say m-3 then we migrate to focal and we will have risk of breaking things at release time
15:52:21 <fungi> we could tie them together without delaying the default nodeset change
15:52:44 <gmann> fungi: you mean zuulv3 base job to focal and while migrating legycy jobs to zuulv3 will automatic to focal ?
15:52:46 <fungi> just say legacy jobs still run on bionic in master, but v3-native jobs use focal in master
15:53:07 <gmann> yeah, sounds good
15:53:11 <fungi> so switching to v3 native in master also means switching to run on focal
15:53:30 <gmann> +1
15:53:56 <fungi> and provides added incentive for projects with lingering legacy jobs to switch them out by the end of the cycle (otherwise they're not testing on the current ubuntu lts)
15:54:30 <njohnston> and if a project is not sure if a job failure is because of the focal switch or. azuulv3 switch they can always temporarily specify a bionic nodeset for that job to troubleshoot
15:55:02 <gmann> njohnston: yeah, those failure will be less but it can work like you suggested to differentiate the failure
15:55:41 <njohnston> just to preempt the usual engineer objection to changing two things at once
15:55:59 <gmann> ok, 5 min remaining. i need to move to nova meeting after that.
15:56:01 <ricolin> tosky, ^^^
15:56:03 <gmann> what is consensus?
15:56:33 <njohnston> I vote for zuulv3 + ubuntu 20.04 for V
15:57:46 <ricolin> +1 on making ubuntu 20.04 as part of zuul v3 goal if that's possible:)
15:57:47 <evrardjp> I second that
15:58:03 <jungleboyj> I think it makes sense if it is possible.
15:59:24 <gmann> ricolin: not part of zuulv3 but as separate goal. goal1 - zuulv3 + goal2 ubuntu 20.04 for V
15:59:57 <ricolin> Okay, then +1 on  zuulv3 + ubuntu 20.04 for V
16:00:08 <gmann> because that need separate coordination and  testing more on all projects
16:00:28 <ricolin> and pre-select mock, privsep for W
16:01:05 <njohnston> I think we just leave privsep in proposed should be good enough, without overly binding a future TC
16:01:10 <gmann> yeah, we can add that in etherpad to disucss for W. i mean in PTG etherpad
16:01:30 <njohnston> I believe mock is happening with or without the goal status
16:01:33 <njohnston> in V
16:01:43 <gmann> i also think so
16:02:03 <ricolin> sure, as we need to figure out what we gonna do for W in PTG anyway:)
16:02:15 <ricolin> njohnston, yes it is
16:02:57 <tosky> I concur that it's probably better to port the legacy jobs to non-legacy first, and gain fossa support from the base job automagically
16:03:47 <tosky> some tuning in the jobs may be needed, but in the worst case people can temporarily pin the nodeset to a bionic one
16:03:52 <gmann> tosky: yeah, we will not do legacy job to focal
16:03:59 <ricolin> make sense
16:05:54 <ricolin> we need a champion for ubuntu 20.04 one
16:06:42 <ricolin> And a proposal
16:06:47 <gmann> i can do that.
16:08:20 <ricolin> gmann, cool:)
16:08:43 <jungleboyj> Thanks gmann
16:08:48 <njohnston> thanks gmann!
16:09:00 <njohnston> do we have a consensus that this is the direction we want to go?
16:09:44 <njohnston> or do we need to wait for the proposal doc and then rediscuss next week?
16:10:51 <jungleboyj> It think we have more or less reached consensus, but it also seems that people have wandered off.
16:10:52 <gmann> waiting for mnaser, he seems away.
16:10:54 <ricolin> we can do official vote on patch which propose ubuntu 20.04 as v goal under https://governance.openstack.org/tc/goals/selected/victoria/
16:11:00 <gmann> ok
16:11:05 <gmann> we need to end meeting also
16:11:17 <njohnston> thanks all!
16:11:19 <ricolin> but yeah, let mnaser do to call:)
16:11:29 <jungleboyj> Yeah, we should probably propose and vote to make it official.
16:11:45 <njohnston> #endmeeting