15:07:53 #startmeeting tc 15:07:54 Meeting started Thu May 14 15:07:53 2020 UTC and is due to finish in 60 minutes. The chair is mnaser. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:07:55 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:07:57 :-) 15:07:58 let's keep that open just to record 15:07:58 The meeting name has been set to 'tc' 15:08:06 +1 thanks 15:08:14 so afaik all proposed goals have landed 15:08:28 yeah, we have two proposed goal merged - https://governance.openstack.org/tc/goals/proposed/index.html 15:08:31 #link https://governance.openstack.org/tc/goals/proposed/index.html 15:08:37 The log here are definitely a good idea. We should send anything we decide to the ML for people not in attendance too though.. and give it 24 hours to really finally decide since not everyone could make it and this is kinda impromptu? 15:08:52 one goal zuulv3 migration is already selected fr V cycle, we need to select one more 15:09:04 diablo_rojo: i think what we should do is propose a change with those goals we select, and then send out that review which will be under formal-vote rules too 15:09:25 mnaser, not quite, there are still a handful of projects that didn't finish their docs in time 15:09:33 so technically we have no choice but waiting for 7 days to land it anyways, which gives us indirectly community approval? 15:09:35 openstackansible being one of them ;) 15:09:44 diablo_rojo: the patches are up there i promise D: 15:09:49 njohnston: any idea if we can get ralonsoh here to know his opinion on single or multi cycle goal for rootwrap ? 15:10:01 mnaser, oh okay I mustve missed them in last audit 15:10:07 diablo_rojo: they came up in the past few days 15:10:25 hi 15:10:27 rootwrap goal is much user benefits and mock goal is easy and doable. 15:10:34 ralonsoh: hi thanks for joining 15:10:36 FWIW, on this topic, njohnston proposed the following change wrt goal proposal, so something to think about -- https://review.opendev.org/#/c/724142/ 15:10:38 mnaser, ah alright, cool. Thank you :) I think we are still missing a couple projects though. So its close, but not completed. 15:11:12 Yeah, we had talked about rootwrap and mock before and those both seemed good. 15:11:24 ralonsoh: we were discussing last time also and today. will rootwrap can be single cycle goal or need multi cycle ? 15:11:33 fwiw, there's nothing stopping us from having 3 goals in total. 15:11:33 multi cycle 15:11:40 Most likely multi 15:11:48 even for one single project, like neutron, we'll need more time 15:12:04 mock goal isn't that hard but require 10+ patches in single project to fix all 15:12:05 Neither one of those is particularly user facing (which I know is a thing we've considered in the past) but I think both are good options. 15:12:05 ralonsoh: i think the good thing is that many projects have already started using privsep though, so the infrastructure is there 15:12:14 mnaser, exactly 15:12:26 and that makes migration process easier 15:12:47 true 15:12:54 something to note about privsep is that it's possible it may be a lot less of a case where we don't control our own destiny 15:13:05 for example, if you migrate something to privsep, and use a library instead of a bunch of system shell outs 15:13:14 and then that library is missing a feature or something, it may block us from progressing forward 15:13:44 you can still use a CLI command but with privsep 15:14:13 ralonsoh: right, but to an extent that might not be _that_ much more different than rootwrap, but maybe i may be missing some points 15:14:24 exactly 15:14:39 so it wouldn't be security hardening other than renaming to use something else 15:15:04 I believe, from what I know, that even using privsep with a CLI is better than rootwrap. I could be wrong though. 15:15:16 well, the security in rootwrap is just because you parse the command 15:15:17 I could imagine an exception list for issues such as that as they come up, much as we have an exception list for projects that need to stay on py27 15:15:43 with privsep you are forcing the linux capatibilities to use 15:15:49 that's an improvement 15:15:57 actually that reminds me of a post that mikal wrote a while back 15:16:16 https://www.madebymikal.com/how-to-make-a-privileged-call-with-oslo-privsep/ 15:16:20 yeah and new way present for all projects is at least good 15:16:21 maybe we can reference that somewhere 15:16:55 mnaser, I'll take a look at this article 15:17:03 Woudl be a good thing to include in the goal proposal 15:17:12 yes, directly translating rootwrap rules to privsep rules doesn't appreciably improve security 15:17:27 dhellman seems to have suggested a while back to add this to docs https://twitter.com/mikal/status/993672440470372352 :) 15:18:14 i want to look at nova nd see how much rootwrap usage they have 15:18:16 but at least if the rules are using privsep, that's a step toward more easily tightening them up 15:18:17 +1 oslo have these info will be helpful 15:18:48 Yeah. 15:19:06 privsep is also a performance benefit too iirc because it isn't forking a new python process each call? 15:19:25 clarkb: technically we had rootwrap-daemon to do the same 15:19:28 yes, we spawn a daemon the first time it's called 15:19:52 and then we use a socket to this daemon 15:19:52 ah I thought all that work went into privsep, didn't realize rootwrap got similar 15:19:52 but rootwrap daemon couldn't call privilged functions, only shell outto things 15:20:19 so from a performance benefit of "can i use pyroute2 in privsep and use internal apis vs fork out "ip link set..."" 15:20:21 it benefits a aton 15:20:29 Given that smcginnis is committed to making the mock change - and it's already well underway - I think that it makes more sense to get started with the privsep goal. This seems especially true for projects that have a very small contributor base and may need a long time to work on changing patterns from a simple CLI call to a library. 15:20:35 https://github.com/openstack/nova/blob/master/etc/nova/rootwrap.d/compute.filters -- does this tell me rootwrap is fully out of nova btw? 15:20:49 njohnston: ++ 15:20:51 https://github.com/openstack/nova/commit/909d0de68edcb232c0d0ca28b755806f7f3780bc 15:20:54 privsep is faster that the daemon, as it does not spawn a new process for the command called 15:20:56 this patch says nova is alread ydone 15:21:37 so rootwrap (python+command) < rootwarp-daemon (command) < privsep (inline python) 15:21:44 off the top of my head, nova and neutron are the two big targets because they need to do things as root. other projects probably should run in user space anyways? 15:22:09 njohnston: +1 15:22:14 maybe cinder too for operations that involve mounting things and doing copies 15:22:24 yes nova is fully privsepped. Just needs the phase 2 work (properly rewrite commands into python) 15:22:38 nova is done yea, but neutron too is completely migrated ? 15:22:52 https://github.com/openstack/neutron/tree/master/etc/neutron/rootwrap.d -- neutron has not even moved to privsep 15:22:54 mnaser: I wonder if nova just needs to clean some docs up: http://codesearch.openstack.org/?q=rootwrap&i=nope&files=&repos=openstack/nova 15:22:57 Cinder is impacted as well. We are part way through the migration though. 15:22:58 sorry, let me correct 15:23:04 neutron still has rootwrap usage 15:23:05 gmann, no, not neutron 15:23:26 ok, how about we split this goal into 2? goal #1 -- move everything into privsep but not necessarily have to worry about reimplementing it with libraries 15:23:37 goal #2 -- remove all usage of exec() unless explicitly necessary 15:23:47 and #1 should be easy and if nova did it then anyone can do it :P 15:23:50 os-brick is also privsepped, but with execs() 15:23:58 (and we have an example of a project which did it too) 15:24:04 That seems like a cleaner approach to making it all one goal and making that span two releases. 15:24:09 mnaser: yeah that is what I was thinking 15:24:12 mnaser: ++ 15:24:18 I like that approach. 15:24:19 Only to find out people wait to do part 1 until the end of the second release.. 15:24:28 You can do a one-cycle goal of moving stuff to privsep that does not involve rewriting every command 15:24:31 cause looking at some of the stuff that neutron will have to do will require a lot of time 15:24:39 lots of dnsmasq interactions and what not 15:24:49 then a goal of getting rid of all bad usage of privsep once you are closer that getting that completed 15:25:07 and in the mean time, you can spend a few cycles getting the large ones closer to phase 2 15:25:07 ralonsoh: what do you think about reframing the goal in this way? 15:25:16 njohnston, ok with this 15:25:23 yep. i like that a lot. it also makes it a lot less intimidating in terms of security and trying to think about how to do it without execs 15:25:27 about neutron, just the 1:1 conversion will take more time 15:25:27 they do not have to be done one right after the other 15:25:29 this also sounds good idea or targeting set of projects in goal phases ? 15:26:00 So if be end of phase 1 we identify some very large amounts of work, maybe we can get it started on specific projects before setting it as a goal 15:26:24 Goals are good for getting everyone to a given level, not so great when amount of work is vastly different across projects 15:26:44 given the situation we're in, if we split this out to "just convert to privsep and keep execs (unless you want to eliminate them)" + mock + zuul. i think that's a fair "load" at our contributors 15:26:44 ^^ excellent point ttx 15:26:55 the main reason being many projects have accomplished these goals already to varying degrees 15:27:06 I think phase 1 in one cycle is totally doable 15:27:16 afaik mock efforts were already mostly underway in nova, most of nova already have zuul v3 jobs, an privsep already done 15:27:21 mnaser: ++ 15:27:23 mnaser that will be too much for one cycle. mock can go without goal in that case 15:27:27 Then maybe a gap before phase two to get others with more work ready for it. 15:27:34 phase 2... it's doable in one cycle *if* we get a few cycle to have the largest targets caught up 15:27:36 But that can be decided exactly later. 15:27:44 diablo_rojo: ++ 15:27:45 zuul migration can be lot of work for many projects 15:27:48 i dont want to stress our teams but i also feel like some teams literally won't have to do these projects 15:28:01 like i dont think MOST of our services will be affected by rootwrap 15:28:01 gmann: hopefully not the same as the privsep one :) 15:28:11 ++ 15:28:14 Like privsep phase 1 is basically done for nova 15:28:22 i.e. i dont think heat ever needed to run things as root ever :) 15:28:29 yeah, and adding mock there might be lot of work for them 15:28:29 and heat already had zuul v3 jobs 15:28:35 I think phase1 + zuul +mock indeed sounds like fair load as mnaser mentioned, and we need more discussion on what's detail for phase two IMO 15:28:36 in term of review and all. 15:28:43 mnaser, yes 15:28:45 Cinder should be able to get phase 1 done pretty easily. 15:28:59 well perhaps we should investigate the affect projects and we might realize that the overall load per project won't be big 15:29:12 cause i think that for the most part, projects will end having to do only 1/3 15:29:17 (cause the other two are likely already done for example) 15:30:05 like i think we need to assume that most project will be noop to 1 or 2 out of these 3. if most projects will not be noop to these 3 goals, we need to drop one 15:30:07 "convert to privsep and keep execs (unless you want to eliminate them)" + zuul' can be good try though there are still chance project might miss but we are giving them goals in start of cycle 15:31:13 there are lot of legacy jobs for many projects. In my xenail->bionic migration it was around >50 % if i remember correctly 15:31:13 right but how many have already moved to zuulv3? and how many even use rootwrap in the first place 15:31:46 gmann: i think maybe we might have a different perception given that you're much closer in your QA interaction though, so i'll take your word on our zuulv3 progress 15:32:14 the way i see it is we have 1 goal everyone will most likely have to do (zuulv3) and two that most people will likely have to do 1 out of 15:32:51 i'm looking at rootwrap usage and it seems minimal as i scroll through projects 15:32:59 a lot of projects list it in their dependencies for some reason, but no actual usage 15:34:09 i just wanted to pick a smaller project like zun that interacts with the system and even they use privsep: https://opendev.org/openstack/zun/src/branch/master/etc/zun/rootwrap.d/zun.filters 15:35:09 technically, the privsep phase 1 goal can really be the "get rid of rootwrap" goal, because there might be cases where projects were doing things as root which would be better done in other ways, and don't actually need either rootwrap or privsep 15:35:13 zun is one of most active and up to date one you checked :) 15:35:17 magnum has no rootwrap usage, heat has no rootwrap usage, watcher has no rootwrap usage 15:35:42 i'm trying to pick random arbitrary projects, i may suck at naming random ones but most of them have no actual need to use rootwrap with the exception of system interacting services such as nova/cinder/neutron 15:35:47 Easy goal for them then lol 15:35:58 mnaser: You have to look for utils.execute with run_as_root as true http://codesearch.openstack.org/?q=utils.execute&i=nope&files=&repos=openstack/zun 15:35:59 and nova is already done with it 15:36:32 :-) 15:36:40 njohnston: does that mean they have inaccurate filter file? oh 15:37:14 oh, sorry, I did not see it was the filter file you linked to 15:37:14 njohnston: they do use it, it seems: https://opendev.org/openstack/zun/src/branch/master/zun/common/privileged.py 15:37:27 they just have zun.common as a whole privileged space 15:37:40 (not the best idea but at least rootwrap ain't htere) 15:37:47 that's why i feel like adding the phase of rootwrap isn't that much more load, IMHO. 15:37:57 which really leaves 2 goals, mock and zuulv3 15:38:04 also if we divide it in two phase, i think doing it in consecutive cylce will be good to get he flow going otherwise it is easy to miss the second phase 15:38:29 I agree. 15:38:30 we might want to add to the goal proposal saying 'best practice is to strictly scope your PrivContext space' 15:38:40 and we select the phase2 as the W cycle goal in advance? or preselect 15:38:57 * fungi suggests "recommended practice" or "standard practice" over "best practice" 15:39:29 fungi: sure, sounds good 15:39:45 gmann: I don’t have an opinion on that right now but not opposed to that 15:39:48 the term "best practice(s)" always makes me twitch 15:40:52 mnaser: yeah something we can discuss in PTG for W. 15:41:15 I agree. We can put it there and nothing stops us from changing it later 15:41:20 manila is an instance where there is a fair - but not overwhelming - number of instances to fix, for example: https://opendev.org/openstack/manila/src/branch/master/etc/manila/rootwrap.d/share.filters http://codesearch.openstack.org/?q=utils.execute&i=nope&files=&repos=openstack/manila 15:41:42 njohnston: ah yes that’s another system interacting one inmissed 15:41:50 Do they use zuul v3? 15:42:13 Like if it’s just one project thatmight be hit harder... 15:42:21 ph one more things in V cycle, migration to Ubuntu 20.04. this also need community level effort 15:42:40 8@ driving to the dealership now 15:42:42 though it will be easy after all projects move to zuulv3 15:42:44 Oops. Wrong window. 15:43:24 but if we need to do Ubuntu 20.04 migration in parallel that is also some work on projects side 15:43:29 gmann, yeah, that's something better done after zuulv3 15:43:39 oh, yep, folks were previously in favor of calling a switch of default job node major version a cycle goal 15:43:53 ricolin: but we have to do it in V cycle and cannot do late during release 15:43:57 and victoria would by our current rules, need to switch from ubuntu 18.04 lts to 20.04 lts 15:44:30 yeah, or can we move it to W cycle so that easy thing to do after zuulv3 goal 15:44:37 I think the system components are a lot more lax now though 15:44:44 gmann, what I proposed is to consider make the dependency between two goals if we need to do both of them in V 15:44:45 It’s not like we switched to systemd 15:45:12 ricolin: +1 but doing dependent goal in single cycle is difficult. 15:45:14 well, we'd need to revise the rules by which we select our test platform if we push the ubuntu focal switch out an additional cycle 15:45:34 humm 15:45:37 gmann, that's true 15:46:22 in that case, who about keep "zuulv3 + Ubuntu 20.04 migration" for V and rest all to later cycle 15:46:30 s/who/how 15:46:55 https://governance.openstack.org/tc/reference/runtimes/victoria.html already lists "Ubuntu 20.04" 15:47:41 I am good with zuulv3 + 20.04 for V, rootwrap/privsep for W and X, and mock done informally 15:47:43 it doesn't *have* to be positioned as a cycle goal, but it's the same amount of work either way 15:47:43 yeah, we can either move it to W cycle or select this as second goal. coordination of thus migration with zuulv3 might be difficutl though 15:48:40 hopefully the focal migration won't be that rough, we already have established patterns for splitting xenial and bionic to different branches 15:48:50 fungi: yeah, it need same effort as goal and i am in favor of doing it as goal so that we get enough attention. with zuulv3 it will be more of testing carefully before migration 15:48:55 My vote is to picking both of those and if teams express concern we can revisit it 15:49:30 I don’t want us to plan too far out and start working that far back. We can adjust and adapt 15:50:11 also, the focal switch should probably happen asap so teams have all cycle to work out any bugs exposed by it 15:50:55 that is something difficult as it will be dependent on zuulv3 migration 15:51:19 migrating legacy job to focal and then convert them to zuulv3 is duplicate work 15:51:19 mnaser: ++ 15:51:43 gmann: like so we can avoid having to update legacy tools/jobs to support running on focal? 15:51:57 that is my main worry, when we finish zuulv3 say m-3 then we migrate to focal and we will have risk of breaking things at release time 15:52:21 we could tie them together without delaying the default nodeset change 15:52:44 fungi: you mean zuulv3 base job to focal and while migrating legycy jobs to zuulv3 will automatic to focal ? 15:52:46 just say legacy jobs still run on bionic in master, but v3-native jobs use focal in master 15:53:07 yeah, sounds good 15:53:11 so switching to v3 native in master also means switching to run on focal 15:53:30 +1 15:53:56 and provides added incentive for projects with lingering legacy jobs to switch them out by the end of the cycle (otherwise they're not testing on the current ubuntu lts) 15:54:30 and if a project is not sure if a job failure is because of the focal switch or. azuulv3 switch they can always temporarily specify a bionic nodeset for that job to troubleshoot 15:55:02 njohnston: yeah, those failure will be less but it can work like you suggested to differentiate the failure 15:55:41 just to preempt the usual engineer objection to changing two things at once 15:55:59 ok, 5 min remaining. i need to move to nova meeting after that. 15:56:01 tosky, ^^^ 15:56:03 what is consensus? 15:56:33 I vote for zuulv3 + ubuntu 20.04 for V 15:57:46 +1 on making ubuntu 20.04 as part of zuul v3 goal if that's possible:) 15:57:47 I second that 15:58:03 I think it makes sense if it is possible. 15:59:24 ricolin: not part of zuulv3 but as separate goal. goal1 - zuulv3 + goal2 ubuntu 20.04 for V 15:59:57 Okay, then +1 on zuulv3 + ubuntu 20.04 for V 16:00:08 because that need separate coordination and testing more on all projects 16:00:28 and pre-select mock, privsep for W 16:01:05 I think we just leave privsep in proposed should be good enough, without overly binding a future TC 16:01:10 yeah, we can add that in etherpad to disucss for W. i mean in PTG etherpad 16:01:30 I believe mock is happening with or without the goal status 16:01:33 in V 16:01:43 i also think so 16:02:03 sure, as we need to figure out what we gonna do for W in PTG anyway:) 16:02:15 njohnston, yes it is 16:02:57 I concur that it's probably better to port the legacy jobs to non-legacy first, and gain fossa support from the base job automagically 16:03:47 some tuning in the jobs may be needed, but in the worst case people can temporarily pin the nodeset to a bionic one 16:03:52 tosky: yeah, we will not do legacy job to focal 16:03:59 make sense 16:05:54 we need a champion for ubuntu 20.04 one 16:06:42 And a proposal 16:06:47 i can do that. 16:08:20 gmann, cool:) 16:08:43 Thanks gmann 16:08:48 thanks gmann! 16:09:00 do we have a consensus that this is the direction we want to go? 16:09:44 or do we need to wait for the proposal doc and then rediscuss next week? 16:10:51 It think we have more or less reached consensus, but it also seems that people have wandered off. 16:10:52 waiting for mnaser, he seems away. 16:10:54 we can do official vote on patch which propose ubuntu 20.04 as v goal under https://governance.openstack.org/tc/goals/selected/victoria/ 16:11:00 ok 16:11:05 we need to end meeting also 16:11:17 thanks all! 16:11:19 but yeah, let mnaser do to call:) 16:11:29 Yeah, we should probably propose and vote to make it official. 16:11:45 #endmeeting