19:00:10 <clarkb> #startmeeting infra
19:00:11 <openstack> Meeting started Tue Oct 24 19:00:10 2017 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:00:12 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:00:14 <openstack> The meeting name has been set to 'infra'
19:00:20 <AJaeger> o/
19:00:21 <ianw> o/
19:00:28 <clarkb> hello infraers
19:00:35 <fungi> helo
19:00:36 <Zara> hi!
19:00:51 <tobiash> hi
19:01:02 <clarkb> #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting
19:01:11 <clarkb> #topic Announcements
19:01:27 <clarkb> #info Summit in two weeks
19:01:48 <fungi> seems like we just had one
19:02:06 <clarkb> This is a general fyi that many of us will be traveling to the summit in Sydney soon. Expect things to slow down starting mid next week
19:02:14 <clarkb> #info You probably need a Visa if you don't have one yet
19:02:25 <fungi> yeah, i have to start my journey a week from tomorrow
19:02:48 <jeblair> or an "electronic travel authorization" if you don't need a "visa"
19:02:53 <clarkb> right that
19:03:17 <clarkb> so make sure you've got one
19:03:36 <clarkb> #info No meeting November 7
19:04:04 <clarkb> I'm just going to go ahead and call it on ^
19:04:10 <jeblair> ++
19:04:34 <clarkb> but I will be around to host the meeting next week
19:04:55 <clarkb> #topic Actions from last meeting
19:05:02 <clarkb> #link http://eavesdrop.openstack.org/meetings/infra/2017/infra.2017-10-17-19.00.txt Minutes from last meeting
19:05:16 <clarkb> fungi did the backup policy on secrets get a doc patch yet?
19:05:31 <fungi> something i was probably supposed to do there, so... no not yet
19:05:35 <clarkb> #action fungi Document secrets backup policy
19:05:43 * Shrews arrives late
19:05:53 <clarkb> ianw: I don't think I've seen backups for the actual secret keys yet either?
19:06:00 <ianw> bah no, will do
19:06:01 <fungi> oh, right, something about making sure it says not to treat us as a backup of your secrets
19:06:15 <clarkb> #action ianw Work to backup zuulv3 secret's keys
19:06:17 <clarkb> fungi: ya that
19:06:34 <clarkb> (fwiw I didn't expect those things had been done as so much effort is still in sorting out bugs/jobfixes/etc)
19:06:37 <fungi> we can't provide them to you if you lose them, but we'll back up our private keys for our own disaster recover of the ci system
19:06:43 <fungi> will do
19:07:00 <clarkb> #topic Priority Efforts
19:07:06 <clarkb> #topic Zuul v3
19:07:31 <clarkb> jeblair had an item about the PTI and job variants here
19:07:40 <AJaeger> Are we ready to remove zuul v2 config from project-config? Then we need to merge first https://review.openstack.org/#/c/513910 and then recheck and merge https://review.openstack.org/#/c/507180/
19:07:46 <jeblair> oh hi
19:07:52 <AJaeger> We first want to tag, don't we?
19:07:58 <jeblair> AJaeger: yeah, i think we're ready
19:08:13 <jeblair> we can always retroactively tag
19:08:27 * AJaeger removes the WIP from 513910...
19:08:56 <jeblair> (anyone want to suggest a commit to tag?  anyone want to actually perform the tag?)
19:08:59 <AJaeger> jeblair: go ahead, sorry for sneaking in front...
19:09:04 <clarkb> I can make the tag
19:09:06 <jeblair> (i don't care enough to suggest a commit :)
19:09:27 <clarkb> jeblair: probably commit before 513910?
19:09:38 <jeblair> clarkb: wfm
19:09:44 <clarkb> or we could even tag 513910 since it doesn't remove the jobs yet
19:10:11 <jeblair> so the pti thing -- i think the migration doc is pretty clear on "don't put python-jobs template in your repo, put it in project-config"
19:10:13 <clarkb> after the meeting I'll pull up a git log and propose a commit to tag but I'm guessing something like master or master~1
19:10:24 <fungi> i'm happy to `git tag -s farewell-jenkins ...` the immediate merge commit parent of whatever change removes our v2 configs
19:10:40 <fungi> as noted, doesn't have to happen pre-merge
19:10:59 <jeblair> but projects may need/want to create project-pipeline variants of the jobs in those templates
19:11:48 <jeblair> (like, use a trusty on this old branch, or adding files or irrelevant files)
19:12:00 <clarkb> or adding required projects like with neutron jobs
19:12:03 <jeblair> ya
19:12:10 <jeblair> generally speaking, those could go either place -- project-config or in-repo.
19:12:17 <jeblair> with one notable exception
19:12:35 <AJaeger> jeblair: I'd like to have a clear policy on what is ok: Do we reject variants in project-config? Don't care? Ask to have them in project-config?
19:13:04 <Shrews> or don't run any py27 jobs (the template adding that also adds pep8, which makes that more complex)
19:13:07 <jeblair> once the project template is applied in project-config, you can't elect not to run a job on a branch, you have to drop the template and add the job specifically for the branch
19:13:13 <jeblair> yah, that :)
19:13:24 <jeblair> it's probably worth treating these as two separate cases
19:13:44 <jeblair> since not running a pti job on a branch is, strictly speaking, a policy violation
19:14:02 <jeblair> even though, in the case which brought it to our attention (nodepool), it's a very forward-looking one :)
19:14:44 <jeblair> anyway, that case requires a project-config change anyway, so it's not going to slip by us.  i think we can set it aside for the moment and focus on the other case -- variants that add projects or file matchers, etc
19:15:04 <jeblair> oh you know what, file matchers may not work either for the same reason
19:15:19 <Shrews> branch matchers do not
19:15:19 <clarkb> jeblair: I think that may be what the release job is running into
19:15:21 <jeblair> (i bet that's what's going on with the thing i'm supposed to look at in the etherpad)
19:15:23 <AJaeger> to reduce our work, we could say: templates have to stay in project-config, any variants go in-repo ;)
19:15:56 <jeblair> AJaeger: i'm inclined to favor that idea too
19:16:35 <AJaeger> if we would go down this road, would we -2 any variant and ask to add to in-repo?
19:16:46 <clarkb> is this something that is considered a bug or just behavior we have to live with?
19:17:02 <clarkb> asking beacuse if it is just something that needs to be fixed do we want to move back to project-config later?
19:17:10 <jeblair> clarkb: this is largely a policy question
19:17:28 <AJaeger> Similar, do we allow new legacy jobs like https://review.openstack.org/514768? I'm inclined to -2 this change that adds a legacy tempest troveclient job
19:18:10 <jeblair> clarkb: ultimately, i'm inclined to say that we should trust projects to manage their own variants
19:18:35 <jeblair> reviewing those isn't really worth our time, and it's desirable that they be in-repo.
19:18:38 <clarkb> jeblair: would we still force the variant to be run via project-config?
19:18:46 <AJaeger> jeblair: then I would love to have a check that we do not run non-voting jobs in gate or voting jobs only in check ;)
19:18:54 <dhellmann> there are rules about defining a job in one repo and attaching it to another repo right?
19:19:00 <clarkb> slightly concerned we'll end up with teh variant of "run no jobs" if not
19:19:21 <jeblair> dhellmann: yes.  one of the rules is you can totally do that.
19:19:40 <dhellmann> so I can define a job in releases and attach it to nova without their consent?
19:19:45 <jeblair> clarkb: yeah, having the project-template attached to the project in project-config will cause the job to always run.
19:20:00 <clarkb> jeblair: gotcha so even if the variant is defined elsewhere we'll force it to run
19:20:10 <jeblair> dhellmann: nope.  another rule is you can only attach it to your own project.  only project-config is exempt from that rule.
19:20:14 <clarkb> in that case ya I agree that allowing projects to define their own variants seems to be simplest
19:20:34 <dhellmann> jeblair : ok, that's what I thought, thanks
19:20:42 <AJaeger> dhellmann: you could define a template in release and if nova uses it, any changes in release repo, would be used...
19:21:01 <AJaeger> dhellmann: but then nova could remove the template usage
19:21:17 <dhellmann> AJaeger : yeah, that'll work fine. I was hoping to put the list of which projects use which release jobs in one place. it sounds like that place is still project-config, even if we move the job definitions elsewhere.
19:21:32 <AJaeger> dhellmann: yes
19:21:45 <jeblair> so i think where this is most likely to break down is in the competing interest between having the template in project-config saying "always run pep8" and a project wanting to say "don't run pep8 on these files".
19:22:02 <jeblair> basically, as soon as any variant matches a change, the job is going to run.  you can't undo that
19:22:31 <clarkb> jeblair: would the main job with no exclusions match too?
19:22:35 <jeblair> this is both in the case of the 'don't run on this branch' example from earlier, but now that i think about it, the more mundane 'don't run on this file'.
19:22:48 <dhellmann> so the job settings aren't applied in "layers"?
19:23:05 <jeblair> dhellmann: they are, but once a job is selected to run, it can't be unselected
19:23:14 <AJaeger> jeblair: so, to not run a job, I just add a "files: this-file-does-not-exist" ?;)
19:23:22 <jeblair> dhellmann: cause basically, that's just not applying extra layers
19:23:37 <jeblair> let me rephrase
19:23:56 <dhellmann> jeblair : I guess you're saying the settings in the project tree don't override the settings in project-config
19:24:07 <fungi> using a first-match algorithm on the decision to run a job
19:24:12 <dhellmann> they may extend them to add jobs, but not change the rules for selecting jobs to run
19:25:01 <jeblair> the openstack-python-jobs template in project-config is attached to all projects.  if a change matches any of the matchers are on *that template*, the jobs will run, regardless of whether a project adds their own variant to their own pipeline.
19:25:14 * dhellmann nods
19:25:26 <jeblair> dhellmann: yeah.  they can even alter the way the job is run (ie, increase timeout, add projects, change nodes).  the only thing they can't do is cause it not to run.
19:25:42 <clarkb> ya that is why releases py35 job runs even though there is a variant that specifies different irrelevant files
19:25:53 <dhellmann> I guess that's not surprising and is probably a feature.
19:25:55 <AJaeger> jeblair: so, irrelevant-files/files cannot be added in a variant?
19:26:05 <jeblair> AJaeger: only in an additive capacity
19:26:16 <fungi> so adding files would work fine
19:26:21 <dhellmann> files would work but irrelevant-files wouldn't?
19:26:31 <jeblair> you can use them to say "also run this job on this file".  or, "extend the timeout if the change touches this file"
19:26:40 <dhellmann> but not "don't run this, even though another rule says to"
19:26:48 <jeblair> dhellmann: right
19:26:53 <dhellmann> got it
19:26:57 <fungi> if you wanted to note additional files which should also cause the job to run (assuming the job was previously set to run only under more limited circumstances)?
19:27:08 <jeblair> fungi: yep
19:27:14 <AJaeger> jeblair: thanks for this discussion, this is helpful!
19:27:35 <dhellmann> so, tl;dr, there's no way to make the py35 job not run for data files that don't change the test results?
19:27:41 <jeblair> so i think if our policy is "you have to have the pti template attached to your project in project-config"  there's some implicit policy around when those jobs run.
19:27:51 <clarkb> dhellmann: you could modify the base py35 job
19:27:56 <jeblair> dhellmann: i think there are 2 things we can do (while continuing this policy)
19:27:56 <clarkb> dhellmann: but otherwise I dn't think so
19:28:04 * clarkb lets jeblair explain better
19:28:09 <jeblair> we can try to have a good set of matchers on that project-template
19:28:14 <jeblair> and i think we actuall do that now?
19:28:16 <dhellmann> I wonder why the releases repo falls under the pti we use for product repos.
19:28:24 <clarkb> jeblair: ya there are already some irrelevant files on the project-template
19:28:29 <jeblair> ie, i think the project template does have irrelevant-files
19:28:39 <jeblair> copied from the nova/neutron copypasta from v2 i think
19:28:55 <jeblair> so we can continue to manage additions to that and try to find a happy medium for everyone
19:28:55 <fungi> dhellmann: currently the pti is getting applied to every deliverable of an official team. we may want to decide at the tc level that the pti is only applicable to some subset?
19:29:21 <jeblair> or we could adopt a policy that lets folks remove the project-template from their project in project-config, perhaps only if they also add back the individual jobs but with the matchers they want.
19:29:30 <dhellmann> fungi : Yeah, I think that's a conversation worth having. Not just for this case, which it seems we can solve regardless.
19:29:45 <fungi> or, rather, we at the governance level have done a poor job of defining the extent to which deliverables have to conform to the pti
19:29:58 <Shrews> jeblair: that is what nodepool has done
19:30:07 <fungi> and so project-config-core reviewers are taking the safest and most literal interpretation
19:30:12 <dhellmann> fungi: yeah, I've noticed several times that different people have a different definition of "all repos" :-)
19:30:14 <Shrews> though i expect, once we merge feature back to master, we can re-apply the template
19:30:51 <jeblair> also, to be fair, the pti doesn't say 'you have to have a zuul project-template'.  it just says 'you have to run these jobs'.  it's probably fair for releases to still follow the pti, and for us to adapt our thinking on how that happens.
19:31:21 <jeblair> (and more importantly, if you run these jobs, they run this way)
19:31:36 <dhellmann> does the infra team *want* to enforce the PTI for all projects? or would you prefer that happens some other way?
19:31:59 <fungi> jeblair: in fact this came up for swift not too far back. their unit tests include tests requiring some special local filesystems provided which results in them needing to run an alternate job for them. the decision at the governance level was that the sort of deviation they required was acceptable
19:32:09 <jeblair> i'm not sure i'd characterize our desire as that to 'enforce' as much as 'facilitate'.
19:32:15 <dhellmann> sure, ok
19:32:35 <AJaeger> We're not enforcing all PTI templates on jobs - I'm not -2 a change that removes python-jobs or a repo creation without python-jobs...
19:32:51 <clarkb> ya we've also given leeway in the past with eg noop jobs to start
19:32:59 <fungi> dhellmann: basically we'll do our best to review what's tested based on guidance provided in project governance
19:33:27 <fungi> but, definitely not the police
19:33:54 <dhellmann> ok. so it sounds like we should clarify that, and independently the infra team wants to consider the approach to apply the results
19:34:13 <jeblair> so maybe the happy medium is:  1) use the template if you can.  2) if you want to change the matchers on the template to improve them and it would work for everyone, do that and we'll merge the change.  3) if that doesn't work for you, it's okay to drop the template in project-config, but then (a) add the jobs back in project-config (b) add the jobs back in your own repo.
19:34:23 <jeblair> (i don't know which of 3a or 3b we should suggest)
19:34:47 <clarkb> jeblair: does adding the jobs back in project-config outside of a template work? eg this behavior is specific to templates?
19:35:00 <fungi> probably 3a vs 3b can depend on whether the project has (yet) started to do any in-repo config
19:35:04 <dhellmann> if part of the point of being able to set the jobs up outside of project-config is to lower the infra team's review burden, it seems like (b) is the approach to go with
19:35:09 <jeblair> clarkb: yeah, and when you do that, you can add whatever (project-specific) matchers you want.
19:35:15 <clarkb> got it
19:35:38 <clarkb> fungi: ya that seems like a reasonable place to start
19:36:00 <fungi> i agree 3b is preferred but if a project doesn't have or doesn't want in-repo configuration at that point then 3a is fine they just agree to put up with our review bottleneck
19:36:11 <dhellmann> that seems fair
19:36:42 <clarkb> its not like that bottleneck is a regression and generally should be improved if >0 projects do it in repo
19:36:50 <fungi> in most cases i expect teams to prefer 3b as a solution over 3a anyway
19:37:09 <fungi> once they get comfortable with the notion
19:37:42 <jeblair> okay, i think i have a feeling for this, thanks.  this has been really useful to discuss in a meeting setting :)  i'll write it up and send it to the list
19:37:58 <fungi> awesome, thanks jeblair!
19:38:00 <AJaeger> thanks, jeblair !
19:38:08 <fungi> i agree, very helpful
19:38:27 <AJaeger> One related question: Do we allow new legacy jobs like https://review.openstack.org/514768? I'm inclined to -2 this change that adds a legacy tempest troveclient job.
19:38:50 <clarkb> AJaeger: I'm inclined to allow it if only because it is easy to copy what saharaclient did and this appeas to be a bug in the migration itself
19:38:53 <AJaeger> Or do we handle this like 3a  vs 3b - if in-repo exists...
19:39:02 <jeblair> i'm maybe inclined to let that slide for a few weeks.  maybe a month or so.
19:39:45 <AJaeger> So, encourage them to go in-repo but let them make their own decision for some time - ok, let's try...
19:39:46 <jeblair> just because the v3 native replacements for complicated jobs are only now becoming available, and we're impinging on peoples' schedules enough as it is :)
19:40:15 <jeblair> maybe stop doing that after sydney
19:40:27 <fungi> and also that allows teams comfortable with writing v3-native jobs to lead the way, and provide a larger body of examples and experience for other teams to learn from
19:40:29 <clarkb> wfm
19:40:46 <AJaeger> wfm
19:40:58 <clarkb> ok anything else zuul? we are running out of time and have a few other items on the list
19:41:18 <clarkb> onward
19:41:19 <fungi> nothing from me
19:41:22 <clarkb> #topic General topics
19:41:32 <clarkb> #topic IRC highlight string for project-config cores
19:41:52 <clarkb> config-core has been proposed as a replacement for project-config-core to reduce the amount of typing necessary
19:42:11 <fungi> i'm happy to match on the alternative there
19:42:29 <fungi> whatever has some loose consensus is fine by me at any rate
19:42:35 <clarkb> ya probbly best thing is to add both sets of highlights and that way users can get away with teh short version or use the long one if already learned
19:42:47 <AJaeger> that's what I did earlier today...
19:43:05 <clarkb> lets do that then. It is easy
19:43:49 <clarkb> #agreed Add both config-core and project-config-core highlights to irc clients
19:44:05 <clarkb> please disagree if there is a reason not to
19:44:17 <ianw> config-core: i agree
19:44:28 <clarkb> perfect
19:44:32 <ianw> hopefully that highlighted for you :)
19:44:33 <AJaeger> thanks for the test, ianw ;)
19:44:39 <clarkb> #topic Remove old/EOL branches
19:44:44 <clarkb> tonyb: this one is yours
19:45:00 <AJaeger> tonyb: did you make it?
19:45:09 <AJaeger> Otherwise I can step in...
19:45:35 <AJaeger> tonyb has proposed two topics for branch removal: stable/newton and old numeric ones
19:45:52 <AJaeger> the agenda lists the two emails.
19:46:04 <AJaeger> The numeric one got no feedback at all AFAIU.
19:46:26 <clarkb> AJaeger: the projects using the numberic branches are who you were looking for feedback from right?
19:46:27 <AJaeger> I'd like to see the removal done to make our jobs easier - less to migrate
19:46:49 <AJaeger> clarkb: yes, those where what tonyb looked for feedback.
19:47:15 <AJaeger> No feedback can mean those projects are dead, haven't read it, whatever - but he only asked for disagreement and nothing happened
19:48:16 <clarkb> worst case we'll have tags for the branches and can restore them
19:48:23 <clarkb> I think it is probably fine to ask for forgiveness in this case
19:48:35 <ianw> if they were important they'd be in a branch-override in project-config or something, which they're not afaics?
19:49:53 <AJaeger> So, two questions basically: Shall we do it? And who volunteers? jhesketh_ did it the last few times AFAIR but he seems not be around
19:50:00 <clarkb> ++ to doing it
19:50:40 <clarkb> AJaeger: is this time sensitive as far as running the script?
19:50:41 <ianw> i'm happy to work with tonyb on it, since we're timezone buddies
19:51:20 <clarkb> ianw: that would be great. I'm happy to help too so feel free to ping if I can help
19:51:25 <AJaeger> clarkb: not really time sensitive - but we normally clean up jobs only after removal. And I would prefer to simplify jobs earlier instead of seeing them migrated in-repo...
19:51:37 <clarkb> AJaeger: good point so should be done soon
19:52:12 <clarkb> AJaeger: does that cover it? anything else you and tonyb want to go over on this topic?
19:52:27 <AJaeger> this covers everything from my side
19:52:38 <clarkb> #topic Sydney team evening ideas
19:52:41 <AJaeger> (but I expected tonyb to be here, so don't know whether it covers everything from his)
19:52:56 <clarkb> AJaeger: ok we can follow up with tonyb later in the day (when tonyb is awake)
19:53:05 <clarkb> #link https://ethercalc.openstack.org/lx7zv5denrb9
19:53:10 <clarkb> ianw has put together a thing ^
19:53:23 <clarkb> if you are going to be in sydney and are interested in walking around and finding food and drink you should take a look
19:53:27 <ianw> i realised the tuesday is melbourne cup day
19:53:43 <clarkb> ianw: does that mean everyone is going to be in the bars watching a sporting event?
19:54:12 <ianw> the event is at like 3pm, but many will continue.  i'm not sure if the conference has organised anything
19:54:19 <ianw> it is "the race that stops a nation", as they say
19:54:37 <clarkb> sounds fun :)
19:54:58 <clarkb> but ya just wanted to amke sure people saw that ethercalc, there is an infra list thread. Please take a look if interested
19:55:08 <clarkb> #topic Project Renames
19:55:09 <jeblair> does "X" mean i am available?
19:55:14 <clarkb> #undo
19:55:15 <openstack> Removing item from minutes: #topic Project Renames
19:55:21 <clarkb> jeblair: that is how I was treating it
19:55:24 <ianw> jeblair: yes
19:55:38 <clarkb> #info X on ethercalc means you are available that night for team thing
19:55:41 <jeblair> cool
19:55:46 <clarkb> #topic Project Renames
19:56:09 <clarkb> These didn't happen beacuse we were focused on helping the release team. I think it is probably a good idea to get more settled into zuulv3 before we try to schedule this
19:56:24 <ianw> ++ agree
19:56:29 <ianw> #link https://etherpad.openstack.org/p/rename-2017-10-20
19:56:30 <clarkb> it also came up that we may have to modify our "playbook" for performing these due to zuulv3
19:56:36 <ianw> a preliminary plan
19:56:48 <clarkb> Probably post sydney we can resync and go from there
19:56:54 <ianw> but i think we decided the early zuul steps were wrong?
19:57:10 <clarkb> ianw: ya I think there were concerns about renaming things in a way that kept zuul happy
19:57:27 <clarkb> lets pick it back up after the summit
19:57:31 <clarkb> #topic Open Discussion
19:57:36 <clarkb> quick you have 1.5 minutes
19:58:12 <clarkb> I did something to one of my molars last Friday eating a brick that we called toast and have a dentist visit tomorrow morning (I was just there too...)
19:58:37 <fungi> right, renaming projects without breaking zuul's configuraiton loading is the tricky bit we don't have answers to yet, i think (short of forcing the change in while zuul is offline)
19:59:01 <fungi> changeS
19:59:01 <clarkb> ok, thank you everyone
19:59:06 <clarkb> see you next week then in sydney
19:59:08 <clarkb> #endmeeting