22:01:12 <corvus> #startmeeting zuul
22:01:12 <openstack> Meeting started Mon Mar 12 22:01:12 2018 UTC and is due to finish in 60 minutes.  The chair is corvus. Information about MeetBot at http://wiki.debian.org/MeetBot.
22:01:13 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
22:01:16 <openstack> The meeting name has been set to 'zuul'
22:01:20 <corvus> #link agenda https://wiki.openstack.org/wiki/Meetings/Zuul
22:01:27 <corvus> #link previous meeting http://eavesdrop.openstack.org/meetings/zuul/2018/zuul.2018-02-19-22.01.html
22:01:32 <SpamapS> o/
22:01:55 <corvus> #topic Action items
22:02:00 <tobiash> o/
22:02:03 <corvus> rbergeron to mail list with times she signed people up for after thorough research and hope for non-conflict to do zuul interviews for yayyyy, content
22:02:15 <corvus> that, or something like it, happened!
22:02:27 <corvus> i think zuul interviews were had at the ptg
22:02:31 <fungi> i didn't see the interviews. anybody have a link?
22:02:37 <corvus> and there should be stuff up on youtube
22:02:45 <corvus> i haven't seen it yet, myself
22:03:02 <tobiash> yes saw the zuul video :)
22:03:05 <Shrews> i saw the jim & jesse show. captivating!
22:03:23 <corvus> there was also a second one
22:03:57 <clarkb> and I did one for infra
22:04:00 <clarkb> wiht pabelanger
22:04:21 <corvus> #link jim and jesse interview https://www.youtube.com/watch?v=t1Ae_gkXOiQ
22:04:53 <fungi> thanks!
22:04:58 <dmsimard> corvus: I think that's the version without the audio fix
22:05:15 <dmsimard> There's an "edited" version with less background noises/air conditioning
22:05:25 <corvus> dmsimard: oh i thought that was the fixed version
22:05:34 <dmsimard> Maybe it is, I'll check
22:05:59 <dmsimard> Doh, it is the fixed version, my bad
22:06:05 <corvus> cool
22:06:28 <corvus> i can't find others right now, i guess let's dig for them later
22:06:46 <corvus> #topic  Possibility to shift meeting back about two hours? (tobiash)
22:07:08 <tobiash> yah, I'd really like to attend the zuul meeting
22:07:27 <tobiash> but it's currently 23:00 for me and 0:00 in summer time
22:07:36 <fungi> yuck
22:07:42 <Shrews> i'm all for earlier times
22:07:48 <fungi> i'm free mondays at 2000z
22:07:50 <corvus> i'll note that no one west of me has announced their presence at this meeting, so we may not have much in the way of objections from this group :)
22:07:56 <tobiash> so I'd kindly ask if it's possible to shift it to an earlier time
22:08:09 <tobiash> :)
22:08:09 <ianw> mostly works for .au ... that will be 7am for the next little while, 6am after daylight savings changes
22:08:37 <corvus> i think when we set the time we had more late-rising apac folks
22:08:40 <fungi> it's still an hour later than we've been doing openstack-infra team meetings, which i'll grant is terribad for apac attendees
22:08:46 <corvus> and fewer eu folks
22:09:08 <clarkb> is jhesketh here too?
22:09:26 <clarkb> that said there probably isn't a time that is good for eu, na, and apac
22:09:54 <fungi> i heard someone say that TristanC is moving back to the americas, so if that's the case he's one less apac-timer
22:10:10 <dmsimard> clarkb: right. Maybe we could post the meeting notes to the mailing list ?
22:10:23 <corvus> i think in the entire history of openstack, the best time we've come up with is 20:00 +/- one hour -- it seems to be about the sweet spot in terms of equalizing the pain.
22:10:48 <fungi> discussing the meeting time change proposal on zuul-discuss would probably be a good way to get more inclusive feedback from other timezones
22:11:13 <corvus> that's why we had the infra, tc, and project-wide meetings on those times
22:11:40 <corvus> fungi: definitely -- i think that's the next step
22:12:37 <corvus> while we're thinking about it, do we want to stick with monday, or move it to tuesday?  there's no longer an openstack conflict at 20:00... :|
22:13:02 <tobiash> I'm fine with both days
22:13:04 <clarkb> moving it to tuesday would certainly simplify my meeting line up
22:13:09 <corvus> the advantage to moving it is that we can be adjacent to the openstack-infra meeting, and we don't get canceled for holidays as much.
22:13:10 <clarkb> but thats not a major concern
22:13:21 <corvus> the advantage to keeping it on monday is that we get to cancel it more often for holidays :)
22:14:25 <Shrews> i wouldn't like tuesday, but that's purely for selfish reasons
22:15:13 <corvus> (all my other meetings are on tuesday, so this would mean i'd get to go back to just writing off one day for meetings, and i'd "gain" a workday :)
22:15:14 <jlk> blarg, I missed the start of the meeting. oops.
22:15:47 <corvus> jlk: a surprisingly on-topic comment! :)
22:15:52 <corvus> okay, let's solicit feedback from the mailing list on both the date and time changes....
22:15:58 <corvus> tobiash: you want to send a mail, or should i?
22:15:58 <fungi> i'd be fine with tuesday at this point. after the tc went long enough not using that slot we officially removed it from the schedule
22:16:04 <jlk> heh I even asked about it before hand. My calendar entry didn't have an alarm associated with it.
22:16:11 <tobiash> I can send it tomorrow
22:16:31 <corvus> (i'm also fine with either day, fwiw)
22:16:51 <corvus> anything else on this topic?
22:17:45 <corvus> #action tobiash send a proposal to the mailing list to discuss moving meeting to 20:00 or thereabouts
22:17:53 <corvus> #topic  Security notification process (corvus)
22:18:32 <corvus> we're about to send out a notification about a vulnerability in zuul we just fixed
22:18:48 <fungi> good thing we didn't release v3 already! ;)
22:19:19 <corvus> fortunately, it's not bad, as it does not seem possible to exploit in the standard configuration
22:19:53 <corvus> but it has pointed out we don't really have a documented process for security issues
22:20:15 <fungi> yes, i think, before we _do_ have our big v3 release splash, it would be swell to have a clear process for reporting and announcing them
22:20:36 <fungi> i think we can likely borrow a lot from the process the openstack vmt uses:
22:20:38 <fungi> #link https://security.openstack.org/vmt-process.html OpenStack Vulnerability Management Process
22:21:09 <fungi> a number of other communities outside openstack have adopted some or most of that since we started maintaining documentation about how it's done
22:21:12 <corvus> fungi: does storyboard support security bugs yet?
22:21:16 <fungi> it does!
22:21:26 <fungi> though there's a bit of a wrinkle which needs some work
22:21:39 <corvus> maybe we can jump straight to that and skip the "private email" option?
22:21:52 <fungi> the biggest missing piece at this point, i think, is setting up some sort of bridge from configuration management to populating teams through the sb api
22:22:14 <clarkb> fungi: to restrict who can see them?
22:22:21 <fungi> right now at least, sb only allows admins to configure (and even view the membership of) its teams
22:22:57 <fungi> so it seems like if we want to start supporting arbitrary teams for adding to private stories, having a way to handle team creation and population through code review would be nice
22:23:20 <fungi> rather than putting team management on the shoulders of a handful of sb site admins
22:24:21 <fungi> the next biggest missing bit is that there's currently no feature to be able to set initial teams to subscribe to a new private story, so that would at least for now rely on us providing some instructions to reporters on how to pick an appropriate team to subscribe
22:24:55 <clarkb> that could lead to "missing" bugs which wouldn't be great
22:24:59 <fungi> e.g., "make sure to add the 'zuul-coresec' team to the story by clicking the ..."
22:25:06 <corvus> i'm not too worried about that -- when you click the "private" button, it's pretty obvious you need to add a team
22:25:10 <corvus> or person
22:25:13 <clarkb> ah
22:25:16 <fungi> right
22:25:26 <fungi> the ui tries to make it hard for you to miss, but...
22:25:38 <fungi> so anyway, those are the extent of the rough edges i'm aware of for doing this with sb
22:25:56 <corvus> in other words, if we've succeeded in getting a person to check the private box, adding the team seems easy.  the big challenge is getting folks to that point.  :)
22:26:05 <fungi> that said, we don't currently have anyone doing it (i only just added a vmt team last week and it's apaprently team id #1)
22:26:56 <corvus> so, um, i'd volunteer to create a team for us, so we can help exercise this.  is that unfair?
22:26:59 <fungi> corvus: oh, so insofar as convincing them to click teh private box, we _can_ also use custom reporting urls which default to private and hide the checkbox for that during story creation if we wanted to have a "go here to report suspected vulnerabilities..." instruction in docs
22:27:33 <corvus> fungi: that sounds good, though "hide the checkbox" worries me unless there's some other confirmation that "yes this bug is private"
22:27:35 <fungi> support for that just merged in storyboard-webclient in the past month
22:28:49 <fungi> i'll have to pull up details on how that bit works
22:29:00 <corvus> fungi: can a private bug be made public later?
22:29:44 <fungi> yep
22:29:53 <fungi> this just hides the checkbox during initial story creation
22:30:11 <corvus> ah, i see the edit option on existing stories, cool
22:30:43 <fungi> something like /#!/story/new?title=my%20vulnerability&force_private=true&description=This%20is%20really%20bad...
22:30:58 <fungi> there are a few settable options through custom reporting urls at the moment
22:31:04 <corvus> clarkb: as infra ptl, i'll ask you to decide whether i should go ahead and create a zuul-security team, or whether we need to block on automating that...
22:31:23 <clarkb> corvus: considering its largely a one time thing its probably fine to just go ahead and make one
22:31:28 <fungi> i wouldn't block on automating it for zuul to be able to use it
22:31:47 <corvus> #action corvus create zuul-security team in storyboard
22:31:56 <clarkb> I assume group membership is relatively easy to change and update later on similar to gerrit
22:32:00 <clarkb> so not too worried about that
22:32:32 <fungi> more that the openstack vmt is going to need lots of per-project coresec teams in sb when we start moving a lot of openstack projects there, and for _that_ purpose i think openstack-infra could stand to have some sb team management functionality which doesn't require teams to go as a sb admin every time they need a team adjusted
22:32:54 <fungi> er, to go ask
22:33:25 <corvus> okay, that gets us reporting, and possibly review for some smaller patches.  i would not have wanted to do patch development in storyboard for the recent patch though -- it was large.  my guess is we'll probably switch to email for that.  but we can coordinate that in storyboard.  this gets us a bootstrapping process at least.
22:33:35 <fungi> one other thing custom reporting urls don't do yet, i think, is allow to prepopulate a project for the initial task
22:33:51 <corvus> fungi: that would be handy.
22:34:01 <fungi> i agree
22:34:06 <fungi> it's been suggested
22:34:52 <fungi> and yeah, i honestly haven't tried pasting diffs into sb comments yet to figure out how that would work... my guess is not well but i don't have evidence to back that up yet
22:35:42 <corvus> skipping down the list a bit -- let's talk about cves... i'm imagining that at some point in the future, we may want to request them.  but we can probably hold off on that at least until we've actually make a release...?
22:35:54 <clarkb> you can use markdown to quote it but ya may not end up being great
22:35:58 <fungi> i recommend holding off until you have a release, tes
22:36:00 <fungi> er, yes
22:36:11 <corvus> ie, can our policy be "don't request cve's for unreleased vulnerabilities"?
22:36:18 <fungi> generally mitre discourages issuing a cve for vulnerabilities only present in unreleased code
22:36:35 <corvus> so maybe even if we introduce a vuln in the future, as long as we don't release it, skip the cve, just fix it (and maybe let folks know on the mailing list)
22:36:44 <SpamapS> +1 for that policy
22:36:53 <fungi> and that is the openstack vmt's policy as well, at least (you'll see in the process document i linked earlier it's one of the classes in our report taxonomy)
22:37:02 <corvus> cool
22:37:34 <fungi> we've found having a clear taxonomy of report classifications is an easy way to be able to explain why something doesn't get a cve and/or advisory
22:37:43 <corvus> now, hypothetically, if we had released last week, and we were working on the current patch checking issue -- should we request a cve for that?  it's not exploitable with bubblewrap in place...
22:38:21 <clarkb> I think you would and then you'd just set the severity appropriately. Probably low/medium in this case?
22:39:00 <fungi> the openstack vmt has requested cve assignment for and sent advisories about vulnerabilities in non-default, non-recommended configurations as long as they're not clearly marked as experimental/debugging or otherwise not for production use
22:40:09 <corvus> here's what we say on the subject: https://docs.openstack.org/infra/zuul/admin/components.html#attr-executor.execution_wrapper
22:40:17 <fungi> for example, if your installation and/or configuration documentation clearly stated that the option to disable bubblewrap was insecure and not intended for production usage, we wouldn't have recommended an advisory and cve
22:40:51 <fungi> and yeah, that documentation in the admin guide seems like exactly that
22:40:55 <clarkb> I guess in that case it does say nullwrap is unsafe
22:41:04 <clarkb> hwoever we also know some users like tobias have to operate in this manner
22:41:20 <corvus> clarkb: tobiash uses bwrap
22:41:29 <clarkb> at the very least we probably want to communicate the situation even if not getting a cve so that users in that position don't have trouble going forward
22:41:39 <clarkb> corvus: oh right he had to deploy his own openshift to get it working
22:41:41 <corvus> he has an openshift with admin privs so he can use privileged containers
22:41:46 <clarkb> ya
22:41:52 <fungi> right, it's never cut-and-dried, so if you know these are risks some users need to take, you adjust your decision based on the information you have
22:41:52 <tobiash> yah ;)
22:42:41 <corvus> i think our conversation earlier suggested we don't know if any users currently required to task those risks...
22:42:41 <fungi> that's also a takeaway for the project's documentation authors if they're marking a feature unsafe yet acknowledge that some people need it... this suggests a problem in the overall design ;)
22:43:11 <clarkb> someone was running on vanilla openshift as a service, maybe rcarrillocruz ?
22:44:04 <corvus> i just can't personally stand behind not using bwrap
22:44:11 <tobiash> yes, rcarrillocruz ran/runs a non-production test env on openshift online
22:44:23 <fungi> so anyway, for this case, if it had been present in 3.x after official release i'd have suggested not bothering with a formal advisory process unless you knew you had users relying on this clearly documented as unsafe feature, unless you just felt like it was warranted anyway
22:45:24 <fungi> in particular, you could also consider that as a deciding factor for whether you need to bother with developing and reviewing the fix secretly under an embargo (if the project even wants to support embargoed vulnerabilities)
22:46:38 <corvus> good point
22:47:11 <fungi> kurt seifried had a good article posted to the red hat security blog about the trade-offs with embargoed vulnerabilities in open software
22:47:27 <fungi> the permalinks i had for that are all screwed up so i'm trying to find it now
22:48:07 <fungi> aha, here we go...
22:48:23 <fungi> #link https://access.redhat.com/blogs/766093/posts/1976653 The hidden costs of embargoes
22:49:59 <corvus> yeah, i'm not sure what the right answer will be for us there...
22:51:14 <fungi> it's a perfectly valid stance to just follow a full-disclosure model with no coordinated disclosure and embargoes
22:51:44 <corvus> just disclose it as soon as a patch is ready?
22:51:53 <fungi> depends on what you feel the predominant risk profile is and the ways in which downstream stakeholders consume the software
22:52:19 <clarkb> if we take this as an example embargos probably do make sense so that you can update right away and don't have to turn zuul off for some unknown period of time
22:52:27 <clarkb> (assuming the default setup wasn't secure)
22:52:58 <clarkb> thankfully zuul makes it relatively easy to restart services
22:53:57 <fungi> yeah, if you feel some vulnerabilities do warrant an embargo and pre-disclosure to some registered subset of stakeholders then i recommend at least having a clear set of rules on how you determine which sorts of reports actually need that level of paranoia
22:54:33 <corvus> the worst thing is probably 'exposure of secrets'
22:54:34 <fungi> so that the people handling them don't unnecessarily impose a costly development overhead on fixes for lower-risk vulnerabilities
22:55:17 <corvus> that's the thing where if there were an exploit for that in the wild, we'd stop zuul and wait for a patch.
22:55:27 <fungi> makes sense
22:56:30 <fungi> but yes, if there are reports where you can make a good case for a public development process then you get the usual benefits of people outside your closed set being able to spot potential issues with your proposed solution, the ability to rely on a public ci system to thoroughly pre-test the fix, and so on
22:56:46 <corvus> fungi: do you have time to start on some zuul-specific docs for this process?
22:57:09 <fungi> corvus: i can make that a priority for this week, sure. where would you like them to live/be published?
22:57:36 <fungi> just so i know to ptopose the prose to the right repo
22:57:37 <corvus> should we put it in the zuul documentation?  maybe in the developer section?
22:57:45 <fungi> wfm
22:57:51 <corvus> (then maybe on the website, we can make a top-level link for it and deep-link to it)
22:57:59 <fungi> do we have a specific section on bug reporting?
22:58:01 <corvus> like "report a bug"  "report a security bug" links or something.
22:58:19 <corvus> fungi: heh, i don't think there's anything more than what's in the contributing file
22:58:52 <fungi> to compare to openstack's process, we actually have two pages one of which is the instructions on reporting a security bug and the other is the process for the people who will be handling the report
22:59:18 <fungi> want to keep the instructions on reporting vulnerabilities short and sweet but also flexible
22:59:32 <corvus> we could put the 'report' section inside the administrators guide even...
22:59:39 <fungi> so we further include e-mail addresses and openpgp keys reporters can use to securely report via e-mail if they prefer
23:00:23 <fungi> yeah, you want the bits on reporting to be as easy to find as possible
23:00:33 <corvus> #action fungi start documenting vulnerability process
23:00:36 <fungi> referencing them from the admin guide at a minimum would be a good idea
23:00:54 <corvus> okay, we're at time now...
23:01:00 <corvus> thanks everyone!
23:01:11 <corvus> #endmeeting