20:01:47 <ttx> #startmeeting tc
20:01:48 <jaypipes> o/
20:01:49 <openstack> Meeting started Tue Mar  1 20:01:47 2016 UTC and is due to finish in 60 minutes.  The chair is ttx. Information about MeetBot at http://wiki.debian.org/MeetBot.
20:01:50 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
20:01:52 <abregman> o/
20:01:52 <openstack> The meeting name has been set to 'tc'
20:02:04 <ttx> Good $timeofday everyone
20:02:10 <ttx> Our agenda for today:
20:02:14 <ttx> #link https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee
20:02:23 <ttx> #topic Add a "stable:follows-policy" tag
20:02:29 <ttx> #link https://review.openstack.org/277918
20:02:30 * edleafe hides in the shadows
20:02:38 * rockyg squeezes into the back of the room
20:02:41 <ttx> So this is a tag that the stable maintenance team can use to certify that a given deliverable stable branches are following the common stable branch policy
20:02:45 * sridhar_ram1 lurks
20:02:53 <ttx> That is made necessary because in the big tent there are a lot of stable/* branches that do not really follow policy those days
20:03:04 <ttx> (and we still want to call them stable/* for other reasons)
20:03:12 <ttx> So it is a pretty important piece of information to communicate to those who consume our deliverables
20:03:21 <ttx> missing a few votes
20:03:36 <ttx> I think mriedem addressed anne's remark
20:03:45 <ttx> questions ?
20:03:49 <sdague> lgtm
20:03:54 <flaper87> lgtm
20:04:00 * dhellmann wonders if we want to start migrating to series/* instead of stable/*
20:04:14 <ttx> dhellmann: there are lots of things that encode stable/*
20:04:19 <dhellmann> yeah
20:04:19 <sdague> dhellmann: there would be a pretty big cost to all our tooling for that
20:04:23 <ttx> (hardcode)
20:04:39 <ttx> dhellmann: but yes you are right, that's what it means today, hence the need for the tag
20:04:44 * thingee lurks
20:04:45 <jeblair> still something about this seems wrong
20:04:59 <jeblair> i don't like a tag that says something follows a policy
20:05:00 <amrith> hello ... is this the TC meeting? (https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee)
20:05:10 <russellb> amitry: yes.
20:05:13 <flaper87> amrith: yes
20:05:19 <russellb> oops.
20:05:24 <dhellmann> jeblair : we have the tag for following the deprecation policy, too. isn't this the same?
20:05:29 <mestery> dhellmann: ++
20:05:33 <thingee> dhellmann: +1
20:05:33 <sdague> remembering the 3 weeks of trying to get the ceilometer grenade bits into a passing state because gnocchi had a custom branch structure
20:05:44 <ttx> dhellmann: slightly different, the other one is an assertion by the project itself
20:06:00 <jaypipes> jeblair: isn't that what the assert:XXX tags are?
20:06:04 <jaypipes> kinda? :)
20:06:18 <dhellmann> ttx: I suppose that's a reasonable distinction to make.
20:06:34 <ttx> here the stable team grants it, so it is indeed a bit of a carrot/stick for the stable team to encourage policy following
20:06:51 <ttx> not sure that is what jeblair objects to
20:07:01 <jeblair> the reason we hardcoded stable in many places was because it meant the same thing
20:07:17 <sdague> honestly, the stable team job is one of the more thankless jobs out there. If they feel this helps them accomplish it, I'm all for it.
20:07:22 <ttx> jeblair: currently in infra it means "series"
20:07:23 <dtroyer> o/
20:07:30 <ttx> sdague: ++
20:07:33 <flaper87> sdague: ++
20:07:33 <annegentl_> sdague: yep
20:07:46 <jeblair> why not say "stable/" means this thing, and if you do something that doesn't mean that, call your branch something else?
20:07:50 <dhellmann> yeah, we have a conflict between the way it has ended up being used and the way it was intended to be used
20:08:09 <ttx> jeblair: I proposed that but that would break a lot of things apparently
20:08:09 <jaypipes> jeblair: I could definitely get behind that.
20:08:19 <jeblair> ttx: what does it break?
20:08:24 <dhellmann> could we rename existing branches?
20:08:24 <ttx> jeblair: I think I raised it at an infra meeting and you shot it down
20:08:37 <jeblair> at least i'm consistent ;)
20:08:46 <flaper87> lol
20:08:50 <sdague> I also think the lever there is weird
20:08:52 <jeblair> ttx: i'm still trying to understand this though
20:09:03 <ttx> jeblair: if not you, maybe fungi
20:09:03 <jeblair> (i'm exploring, not advocating)
20:09:09 <sdague> because project/foo thinks they are following stable branch policy
20:09:18 <dhellmann> we have 3 types of stable branches: actually stable, using stable/$series for convenience, and using stable/$version
20:09:20 <sdague> so they have stable/mitaka
20:09:29 <sdague> stable maint team notices they aren't
20:09:44 <sdague> stable maint team has authority to delete project/foo stable/mitaka branch?
20:10:02 <flaper87> It'd be better for stable/* to actually be stable, fwiw.
20:10:06 <dhellmann> so yeah, even if we don't rename things we want to give the stable team a flag to say "yes, we agree that this is stable"
20:10:12 <jeblair> maybe instead of deleting the branch, we remove the project
20:10:15 <sdague> dhellmann: right, which is that
20:10:23 <sdague> jeblair: o_O
20:10:37 <flaper87> not all projects must follow the stable policy
20:10:38 <sdague> the only project we've removed before was one that was dead for 2 cycles
20:10:40 <ttx> as if we could remove a project
20:10:45 <dhellmann> jeblair : rename to openstack-not-actually-stable/$project ?
20:10:47 <fungi> i think i didn't so much object, as express that devstack-gate's current branch mapping would get waaaay more complex if you needed to try to assert that your stable/1.3.0 or foo/bar branch should be tested against stable/liberty of some other project
20:10:57 <annegentl_> wobbly/1.1.0
20:10:58 <sdague> right, I don't believe removing a project is a thing that actually exists as a TC power
20:11:01 <ttx> fungi: right
20:11:02 <dhellmann> annegentl_ : ++
20:11:02 <jeblair> fungi: absolutely
20:11:04 <russellb> sure it is
20:11:07 <russellb> (removing a project)
20:11:09 <jeblair> sdague: oh it definitely does
20:11:14 <jeblair> if we can add, we can remove
20:11:16 <russellb> we just never do it
20:11:17 <sdague> ok, I'll believe it when it happens
20:11:20 <jaypipes> stable-but-having-experimental-feature-backports/$project is too long of a branch name.
20:11:21 <russellb> heh, right
20:11:24 <dhellmann> sdague : I think he just means as an official project, not from gerrit?
20:11:30 <ttx> I'm not saying this can't be implemented differently, I'm just saying a tag is the most convenient way to fix it for now
20:11:36 <dhellmann> jaypipes : abbreviate it?
20:11:42 <jaypipes> dhellmann: :)
20:11:44 <jeblair> to try to get back onto the topic --
20:11:46 <sdague> dhellmann: it is an un-unit tested piece of policy
20:11:47 <flaper87> I like the tag as an immediate solution
20:11:52 <fungi> the last time i remember it coming up was when some project (maybe an oslo lib) wanted more than one stable/x.y.z that would cross-test against a specific stable branch of other repos in devstack
20:11:54 <flaper87> if we find a better solution, we can always remove the tag
20:11:56 <dhellmann> ttx: I agree. I think we want this tag regardless, because there's the "we're trying to be stable" state to consider.
20:11:57 <ttx> and I'd like to fix it now before we cut stable/mitaka all around
20:12:03 <ttx> starting tomorrow
20:12:04 <jeblair> the thing that's sticking with me is that we're giving our users an inconsistent story
20:12:15 <flaper87> I like the idea of renaming fake stable branches, fwiw
20:12:19 <mordred> jeblair: ++
20:12:29 <annegentl_> jeblair: or is it quality checking for truth?
20:12:37 <mordred> it's bad enough that people think that "stable/liberty" is some how "more hardend"
20:12:37 <ttx> jeblair: I get your point. I think we can fix it better tomorrow, but it's more work
20:12:42 <jeblair> we're saying "stable/*" may or may not be stable
20:12:48 <annegentl_> jeblair: I mean, I see it either way, yeah.
20:12:50 <flaper87> jeblair: it's inconsistent now, anyway. The benefit of this tag is that it's provided by a team we trust
20:12:51 <mordred> rather than "has a different policy for landing patches"
20:12:53 <sdague> jeblair: we are admitting the truth
20:13:12 <mordred> but as soon as we even step away from _that_ with things that have the openstack title on it
20:13:15 <ttx> If we decide to replace the tag with some ownership of a branch space, we can do that another day
20:13:20 <sdague> because the truth is stable/* may or may not be that today, which is what triggered it
20:13:28 <ttx> But I expect that would run into a lot of implementation issues
20:13:38 <sdague> so, step 1 - acknowledge the state of the world, write it down
20:13:42 <ttx> like... projects can create branches
20:13:46 <russellb> sdague: ++
20:13:49 <dhellmann> right, it would be useful to be able to declare what is actually stable and then go through the exercise of fixing the things that aren't
20:13:50 <sdague> step 2 - figure out how you want to change it
20:13:54 <flaper87> I also proposed in the review that we could find a way to tag each stable branch individually
20:13:58 <ttx> sdague: yep, baby steps
20:14:03 <flaper87> but again, I think the current proposal is fine for now
20:14:11 <flaper87> and we should evolve from there
20:14:13 <mestery> Nicely put sdague, +1
20:14:25 <jeblair> sdague: sure, i'm just more aspirational and would like us to describe what we want things to be, rather than to follow the project and try to clean things up behind it.
20:14:35 <jeblair> i typed that a while ago
20:14:44 <jeblair> sdague: i guess i'm jumping to step 2 :)
20:15:02 <ttx> OK, let's approve step 1 and then someone can propose step 2
20:15:04 <sdague> jeblair: right, and if stable maint wasn't already such a giant thankless job, that would be fine
20:15:08 <flaper87> Ideally, I'd like each stable branch to actually be stable and follow the stable policy but that's not the goal right now
20:15:24 <sdague> but I think it needs a lot more people piled into helping on stable maint before step 2 is doable
20:15:28 <ttx> I agree with that long-term goal
20:15:35 <jeblair> sdague: yeah.  i don't think a -1 from me will help step 2 happen.
20:15:37 <ttx> sdague: exxxactlk
20:15:39 <ttx> y
20:15:46 <sdague> jeblair: right
20:15:53 <ttx> ok, let's move on
20:16:03 <ttx> we have 9 YES
20:16:15 <ttx> jeblair: want to write down your aspirational next step on the review before I approve ?
20:16:21 <annegentl_> jeblair: ayup
20:16:55 <jeblair> ttx: done
20:17:02 <ttx> alrighty approved
20:17:12 <ttx> #topic Clarify test-only license requirements
20:17:16 <ttx> #link https://review.openstack.org/279999
20:17:28 <ttx> So this is definitely a good clarification to add, and it now has enough votes to be approved
20:17:52 <ttx> do we have lifeless around
20:17:54 <dhellmann> I'd like lifeless to expand on his point, if we have a minute for that?
20:18:15 <ttx> lifeless mentions that this does not help with Scapy (being considered at https://review.openstack.org/#/c/277893/)
20:18:19 <dhellmann> because even with this clarification, I'm not comfortable approving gpl additions to the requirements list
20:18:29 <ttx> I suspect his argument being we can't import libraries without creating derivative work, be it test-time or run-time code ?
20:18:53 <ttx> dhellmann: right, or leaving MySQL-python in
20:18:59 <dhellmann> right
20:19:04 <ttx> It appears to still be used in trove:
20:19:05 <ttx> http://git.openstack.org/cgit/openstack/trove/tree/requirements.txt#n38
20:19:09 <ttx> Although I couldn't find an import so it might just be a leftover
20:19:23 <dhellmann> did you look in oslo.db?
20:19:40 <mordred> I do not htink lifeless point is relevant
20:19:53 <ttx> dhellmann: I'd say that the clarification is still a clarification, although it doesn't really clarify whether test-only deps are ok
20:20:03 <dhellmann> mordred : not relevant to this, or not relevant to the patch adding scapy to the requirements list?
20:20:04 <mordred> the license terms are about terms for cpoying
20:20:09 <mordred> not relevant to the patch adding scapy
20:20:28 <mordred> I do not think that openstack projects using scapy in testing produces any license violations
20:20:31 <mordred> imho
20:20:43 <dhellmann> ok, I'm not really worried about that case per se
20:20:47 <ttx> mordred: that is where I was too
20:20:53 <dhellmann> I'm worried about the fact that we have no way to enforce it going further than that
20:20:57 <dims> mordred : yes, if it's in the base infrastructure itself. not in the project requirements
20:21:00 <jeblair> mordred: i agree (regardless of where you fall on the import==derivative question)
20:21:16 <sdague> mordred: I would agree, we don't however have any mechanism for keeping things out of requirements.txt once it's in global-requirements.txt
20:21:19 <ttx> I think that's another discussion though. Difficult to have without lifeless to sustain his argument
20:21:26 <sdague> gr had been a bit more of a firewall up until this point
20:21:27 <mordred> I think if people have not read the recent article by eben related to the ubuntu+zfs case
20:21:30 <mordred> you should
20:21:30 <ttx> sdague: taht's another technical issue
20:21:47 <mordred> because eben makes some REALLY execellent points hwich are analogously relevant here
20:21:50 <sdague> I'm still +2 on that, but it's a thing people concerned about this should raise
20:22:19 <annegentl_> mordred: linkety?
20:22:19 <ttx> OK, I propose we approve this and raise that discussion on another forum, like legal-discuss, with lifeless around
20:22:35 <flaper87> ttx: ++
20:22:43 <annegentl_> ttx: sounds good
20:22:43 <mordred> https://www.softwarefreedom.org/resources/2016/linux-kernel-cddl.html
20:22:44 <jeblair> #link https://www.softwarefreedom.org/resources/2016/linux-kernel-cddl.html
20:22:50 <annegentl_> thanks!
20:22:51 <fungi> worth noting that if the unified global requirements list remains our sole gatekeeping tool for this, then it's not actually effective if we start comingling test-time dependencies not held to the same standard in that list and we'll need to come up with a more fine-grained solution
20:23:10 <jeblair> fungi: yes, that's the point dims raised
20:23:18 <fungi> oh, yep, i missed that comment
20:23:24 <ttx> yes, we should have one at some point. For the moment I'll probably do due diligence on problematic deps
20:23:26 <jeblair> i don't actually think we intended it to be that, however
20:23:44 <dhellmann> fungi : yeah, we'll have to talk about that at some point soon, I think.
20:23:56 <ttx> ok, approving now, and deferring taht discussion for when lifeless and mordred are here at the same time
20:24:41 <ttx> done
20:24:50 <ttx> #topic Applying Tacker for Big Tent
20:24:58 <ttx> #link https://review.openstack.org/276417
20:25:06 <ttx> Round 3... As I said in the meeting reminder, would be good if we could come to a final decision here
20:25:18 <ttx> All stated objections hinge on whether we trust Tacker can become a generally-useful OpenStack glue service rather than an NFV-infeodated purpose-built narrow application
20:25:25 <ttx> We can either trust that they will become that and approve them now
20:25:33 <ttx> Or we can apply caution and ask that they demonstrate being generally-useful first, and delay the application
20:25:51 <ttx> (or assert that they can't become that and reject them outright)
20:26:03 <ttx> I'm slightly leaning on the "trusting" side because I fear the catch-22 (not being able to truly be generally useful until it's officially adopted)
20:26:15 <ttx> That said, the lack of TC member votes and enthusiasm around it should push us to caution
20:26:22 <mestery> I think we should in general lean to trust
20:26:31 <ttx> Still missing 6 votes, current expressed votes lean towards including it
20:26:32 <mestery> So I agree with you ttx
20:26:36 <flaper87> I'm on the trusting side. We've applied the same logic on other projects and we can evaluate again later
20:27:08 <annegentl_> generally trust as well
20:27:13 <jaypipes> I have not changed my mind on Tacker. I continue to see it as a purpose-built application for telco that consumes a variety of OpenStack APIs and services.
20:27:14 <dhellmann> I'm OK with trusting them, but would like to understand what it means if they don't?
20:27:22 <dtroyer> I think they have shown good intentions and practice, I think we can trust the team so far
20:27:28 <dtroyer> But I am still not certain this needs to be an OpenStack project.  It looks like an application layer to me.
20:27:33 <ttx> missing lifeless, mordred, dtroyer, jeblair, sdague, dhellmann
20:28:01 <mordred> yah. you are missing me
20:28:09 <ttx> mordred: I do.
20:28:10 <annegentl_> dtroyer: I feel like that's the case for trove then too
20:28:11 <dhellmann> do we set a time-frame? do we revisit this at some point?
20:28:12 <russellb> it sounds like we should have a more general conversation about where we'd like to draw an upper scope bound
20:28:16 <mordred> I'm having trouble developing enough of an opinion on this to vote, tbh
20:28:16 <russellb> and write that down
20:28:19 <annegentl_> russellb: yeah
20:28:23 <dhellmann> russellb : ++
20:28:29 <dtroyer> annegentl_: maybe?  although Trove is more general.
20:28:31 <russellb> because other than that, this is fine to me
20:28:32 <sdague> mordred: same
20:28:37 <annegentl_> I'd like to separate that discussion about scopes/bounds from this review though.
20:28:42 <mordred> here are my thoughts, in no particular order
20:28:47 <ttx> mordred: and I feel bad for voting as well. I have not nearly the same conviction as jaypipes has, yet my vote carries the same weight
20:29:01 <ttx> and yet we need to, that's what we were elected for
20:29:03 <mordred> a) in the big tent, I care most about humans who are like us doing development on things kind of like we do development on things
20:29:17 <mordred> I don't particularly pass a lot of judgement on the validity of the thing itself
20:29:21 * jeblair expands voting scale 10x for ttx to register a 1/10 vote
20:29:24 <bswartz> +1 (with conviction) ?
20:29:28 <jaypipes> ttx: the weight of one's conviction has never been a good indicator of correctness :)
20:29:49 <ttx> +0
20:29:56 <mordred> b) there is obviously an upper bound where things are just apps on top of openstack ... but I'm not sure if I have a good basis to judge that at th emoment
20:30:02 <russellb> i've tried to balance out jaypipes' conviction with equal and opposite conviction, to make it totally unclear where to go next
20:30:07 <annegentl_> jaypipes: heh
20:30:12 <jeblair> russellb: my hero
20:30:31 <mestery> heh
20:30:44 <dhellmann> russellb : helping!
20:30:51 <ttx> russellb: and then I summarized it with plenty of doubts and registered my +0
20:31:00 <russellb> this is in an upper bound grey area, but it's far from alone there
20:31:04 <ttx> recipe for success and quick decision
20:31:06 <mordred> c) I would like to do things to help our opnfv friends
20:31:06 <russellb> IMO..
20:31:20 <russellb> ttx: :)
20:31:27 <russellb> ¯\_(ツ)_/¯
20:31:36 <mestery> mordred: teamwork!
20:31:57 <ttx> we can totally decide that we need to think about it a lot more and defer the decision to next cycle
20:31:58 <mordred> so, I thinkn I'm leaning towards a yes
20:32:13 <mordred> largely because I think that to reject something from the big tent I need to have a specific reason for the no
20:32:21 <ttx> we can also say yes but with the damocles sword of being removed based on that future discussion on scope
20:32:22 <russellb> i'd propose yes for now, with an increased priority for discussing scope
20:32:29 <markmcclain> I still think caution is warranted and we should defer... would give team time to implement the integrations they've committed to
20:32:31 <russellb> and an understanding that we may revisit the projects in the grey area based on the outcome
20:32:36 <mestery> russellb: +1
20:32:50 <dtroyer> russellb: that is to me a reason to either defer or decide no
20:32:58 <jaypipes> I'm not at ALL against helping out our OPNFV friends. I'm just saying I don't believe a purpose-built Telco MANO application "is OpenStack" and furthers the mission of OpenStack to be the ubiquitous cloud computing infrastructure for private and public clouds.
20:33:41 * flaper87 will learn all the acronyms before this discussion ends... promised!
20:33:43 <ttx> russellb: "provisionally approved pending further discussion on scope" ?
20:33:44 <jeblair> mordred's *rousing* speech seems to have gathered a few votes
20:34:10 <sdague> I'm formally abstaining
20:34:14 <mestery> I tend to agree with mordred that under the big tent I'm not seeing how I'd vote no on this one.
20:34:16 <ttx> remember Donald Trump will soon be POTUS and the end of the world is near anyway
20:34:20 <dougwig> This bike shed is starting to look like a psychedelic barber pole.  That was lit on fire.
20:34:20 <sdague> I did register it as such
20:34:27 <anteaya> ttx: cory booker is in the wings
20:34:35 <anteaya> ttx: I have hope for him post trump
20:34:48 <mordred> jaypipes: I think that if it helps NFV folks use openstack as their cloud, then it helps that. I do not know if it will help that, but I do not think it will explicitly hurt that
20:35:21 <mordred> jaypipes: so worst case it seems to hurt no more than any of the other margianlly helpful things we have, and best case maybe it helps a little?
20:35:29 <russellb> mordred: that's my feeling
20:35:32 <mestery> 8 rollcall +1 votes now
20:35:33 <ttx> OK, we have 8 votes yes at this stage, probably because it's easier to say yes than no.
20:35:36 <rockyg> mordred:  ++ makes ubiquitous openstack more ubiquitous
20:35:44 <mestery> ttx: Isn't that the big tent's premise?
20:35:49 <ttx> I'm fine with making it clear this is provisional approval
20:35:56 <ttx> pending more discussion
20:35:59 <mestery> ttx: Have we done a provisional approval before?
20:36:01 <ttx> no
20:36:04 <mestery> OK
20:36:07 <ttx> all approvals are proviosnal
20:36:07 <russellb> welcome to big tent era
20:36:10 <mestery> lol
20:36:11 <mestery> True
20:36:30 <ttx> it's just that we need to psychologically prepare projects to be kicked out
20:36:35 <ttx> I have my list
20:36:44 <jaypipes> rockyg: or does it make NFV architecture and ETSI more ubiquitous?
20:36:59 <rockyg> jaypipes, Both?
20:37:00 <ttx> some projects are going horribly wrong and should definitely not be openstack
20:37:05 <jaypipes> rockyg: one is not the other
20:37:06 <russellb> these folks are rallying around openstack, and i think we should be supporting that
20:37:25 <jaypipes> rockyg: and it is that equivocy that I see as dangerous.
20:37:26 <markmcclain> russellb: but they're rallying around openstack on their terms
20:37:29 <mestery> Right, this makes openstack AND NFV/ETSI more ubiquitous IMHO
20:37:30 <ttx> so we might not have kicked out anyone but it may happen. Especially now that we don't even have to rename the repos
20:37:32 <russellb> markmcclain: huh?
20:37:34 <anteaya> ttx: so happy to hear you say that
20:37:42 <jaypipes> rockyg: the whole-hog consumption of OPenStack by the telco industry consortium is a real danger.
20:37:53 * dhellmann can't wait to see ttx's list
20:38:02 <flaper87> dhellmann: ++
20:38:10 <rockyg> jaypipes, very true.
20:38:28 <ttx> dhellmann: projects without an install guide are up there on my list
20:38:29 <sdague> ttx: if it's not tested it's broken?
20:38:35 <dhellmann> ttx: ++
20:38:55 <ttx> anyway, back on topic
20:39:03 <flaper87> should we move on? I think there are enough votes to make a decision
20:39:09 <mestery> flaper87: ++
20:39:10 <markmcclain> russellb: basically what japypipes said... telcos want a very specific vision for OpenStack
20:39:17 <ttx> dtroyer: you should register your vote as RollCall
20:39:18 <jaypipes> rockyg: if you don't see the danger that a single industry driving feature requests and product direction is, please have a chat with any nova core contributor about how NUMA, CPU pinning, SR-IOV and PCI functionality has gone.
20:39:23 <rockyg> jaypipes, can enterprise world adoption help balance the scales?
20:39:25 <russellb> markmcclain: everyone has a vision
20:39:29 <dtroyer> ttx: thanks, done
20:39:32 <russellb> anyway, moving on
20:40:19 <ttx> 8 vs. 3, one abstain
20:40:36 <ttx> I think that means yes
20:40:58 <flaper87> it's missing lifeless' vote but that won't change the result
20:41:18 <flaper87> Unless lifeless convinces all +1's to be -1's
20:41:20 <ttx> Although I agree with dtroyer's point. We are creating a larger grey mess we'll have to fix one day
20:41:21 <flaper87> :D
20:41:45 <flaper87> I think it's fair to say we'll revisit that grey mess in Newton
20:41:51 <flaper87> why not? Or every cycle
20:42:02 <ttx> I think revisiting that grey mess should be our priority in Newton
20:42:08 <mestery> ttx: ++
20:42:11 <flaper87> ttx: I'd agree with that
20:42:12 <mestery> Spring cleaning
20:42:14 <flaper87> :)
20:42:15 <ttx> We had two grey things being submitted recently
20:42:19 <russellb> fine with me.
20:42:29 <ttx> previously that was easier
20:42:29 <annegentl_> grey matter for grey mess
20:42:32 <russellb> i'm sure we'll have plenty to argue around that
20:42:55 <ttx> alright, closing the vote now
20:43:40 <ttx> It is in
20:43:51 <ttx> #topic Open discussion
20:43:55 <ttx> * OpenStack CI resources vs. project growth (sdague)
20:43:59 <ttx> sdague: floor is yours
20:44:19 <sdague> sure, this is something I wanted to bring up in looking at the delays and turn around time we're hitting as we hit milestones
20:44:25 <sdague> we've got project growth
20:44:30 <sdague> growth of testing on projects
20:44:48 <sdague> and are actually currently down on cloud resources from the last 3 releases
20:45:14 <sdague> and while that last one may get addressed, I think the growth is always going to be pushing boundaries
20:45:37 <sdague> for instance, there was a nodepool issue over the weekend, which basically put an 8 hour delay into everything yesterday until it could grind overnight
20:45:38 <annegentl_> sdague: so so true
20:45:50 <annegentl_> for other cross project concerns also
20:46:09 <ttx> annegentl_: I think infra is where the growth is creating the most impact though
20:46:11 <dhellmann> we've even been hampered a bit with releases because of the check queue depth
20:46:15 <sdague> so any hiccup in the system can trigger a cascade that taks a couple of days to clear
20:46:42 <annegentl_> ttx: yeah except for lack of install guides will cause longer-term delay in adoption. CI resources are easier to measure direct hit
20:47:00 <sdague> I think it's too late to address anything for mitaka, but I wanted to get a conversation going around this
20:47:09 <dhellmann> sdague : are you building up to a proposal, or just starting the conversation?
20:47:10 <fungi> yeah, the real shame is that the nodepool issue was addressed hours before we got into peak usage. it could have been far worse if it were just a little later in the day
20:47:18 <annegentl_> not saying one's more "valuable" but that one's easier to visualize on a timeline
20:47:38 <sdague> dhellmann: just the conversation
20:47:43 <ttx> annegentl_: sure, "most visible impact"
20:48:08 <ttx> sdague: so... solutions include: more test hosts, less tests/project, less projects
20:48:18 <ttx> any other solution ?
20:48:24 <sdague> more guidance on use of test resources?
20:48:27 <fungi> or finding a way to deprioritize jobs for some projects and prioritize others
20:48:36 <sdague> right, prioritization on peaks
20:48:41 <sdague> that's a 5th thing
20:48:51 <jeblair> fungi: interestingly we looked at that a long time ago, and the use of ci resources by non "core" projects was almost negligible
20:48:55 <ttx> FOr example, non-voting jobs seem to be completely ignored by most, yet consume a lot of resources
20:48:58 <dhellmann> yeah, I was thinking if we reserved CI for official projects during peak times that might help, but only during those times and we'd move the peaks from other projects to other times
20:49:08 <jeblair> i actually doubt it is that much different now
20:49:13 <dhellmann> jeblair : does that still hold?
20:49:15 <fungi> gearman hampers us there a bit since the protocol has a designed-in 3-level precedence implementation
20:49:16 <annegentl_> one question I have is, has anyone measured 3rd party impact
20:49:28 <mordred> annegentl_: what do you mean
20:49:30 <mordred> ?
20:49:34 <annegentl_> and would more 3rd party testing help with resources?
20:49:35 <anteaya> annegentl_: what impact
20:49:45 <sdague> so, again, I think we come back to step 1 - do we have a real problem?
20:49:46 <anteaya> how would you invision?
20:49:47 <jeblair> dhellmann: my gut says it mostly still holds.  but actual measurement might be useful.  :)
20:49:49 <fungi> jeblair: yeah, i suspect not much has changed there but it would be interesting to see updated numbers
20:50:00 <dhellmann> jeblair, fungi : yeah
20:50:05 <anteaya> annegentl_: the testing is different, proprietary vs open
20:50:14 <annegentl_> anteaya: wondering if we look at all testing as a pie, is 3rd party testing able to take on more of a large pie?
20:50:24 <anteaya> only to benefit themselves
20:50:26 <annegentl_> anteaya: ok
20:50:41 <anteaya> third party testing has no benefit to openstack trunk code
20:50:43 <ttx> sdague: I think we do have a problem yes. We could ramp up CI resources to accommodate growth before, and we are struggling to do that currently
20:50:44 <jeblair> sdague: i think this is timely -- we are basically at capacity right now (where my definition of at capacity means our ability to finish 24h worth of jobs in 24h)
20:50:44 <jbryce> when we have these periodic bottlenecks, how under-resourced are we? (ballpark)
20:51:17 <fungi> annegentl_: anteaya: right, the "third-party" solution is additional parties with available cloud resources donating quota to us so we can use that to run first-party (upstream) jobs there
20:51:24 <jbryce> 10%? 50%? 100%?
20:51:38 <annegentl_> jeblair: time machine?
20:51:55 <ttx> I would say 50%, but fungi should know better
20:52:00 <sdague> jeblair: right, I think we have different definitions of at capacity, because if you can't turn around results on your patch in a timely manner it hampers the ability to address issues only seen in our gate
20:52:00 <anteaya> jbryce: we would use any resources given to us to their maximum quota
20:52:01 <russellb> and do you think paying some subset of providers for extra capacity would hurt our ability to get resources donated?
20:52:08 <mordred> jbryce: we were hitting capacity before HP sunset
20:52:25 <sdague> mordred: we were, it's definitely a lot more cramped since then though
20:52:29 <fungi> jbryce: it's hard to put a hard number on, because the impact is "how much longer does it take to get certain kinds ofg job results"
20:52:31 <jbryce> anteaya: i understand that, but i’m wondering at what level of additional resources we would not be hitting capacity
20:52:36 <mordred> jbryce: so, I'd say we're at least under capacity by 600 VMs at peak times - but probably more. we are obviously working on finding more capacity
20:52:48 <jbryce> mordred: ok cool
20:52:55 <sdague> mordred: yeh, that's about as good an estimate as any
20:53:05 <jbryce> how predictable is the arrival of peak times? it seems at least some of them we can see coming, eh?
20:53:14 <fungi> we have graphs
20:53:20 <anteaya> jbryce: every milestone and feature freeze
20:53:22 <fungi> that can be trended fairly accurately
20:53:25 <jeblair> we can define it more precisely if we agree on a service expectation (jobs report in 24h?  8h?  2h?)
20:53:27 <anteaya> you can set a watch by it
20:53:45 <anteaya> jeblair: right, we have never set service expecations
20:53:52 * jbryce is mulling over some way to get temporary quota increases from providers within an agreed upon timeframe
20:53:53 <russellb> jbryce: i think it's the kind of thing worth spending $ on
20:53:59 <sdague> jeblair: right, maybe that's part of the conversation
20:54:00 <anteaya> we just live with what we have and communicate it
20:54:00 <jbryce> yeah exactly
20:54:04 <flaper87> russellb: ++
20:54:07 <jeblair> currently, lacking any other metric, we've been going by 24h
20:54:21 <sdague> because once we get past 2 hrs turn around, it definitely negative impacts development
20:54:22 <annegentl_> might as well do 24 hours, not sure smaller increments is meaningful for global work
20:54:23 <fungi> there are daily, weekly and release cycle patterns which each have their own envelope
20:54:34 <annegentl_> any link to graphs?
20:54:35 <dhellmann> jbryce: it would be useful to have more capacity around the 2nd and 3rd milestones, and during the RC1 period.
20:54:36 <anteaya> sdague: 2 hours turn around for what?
20:54:39 <anteaya> for the gate?
20:54:45 <sdague> especially with global teams where you are only getting a couple of hrs of overlap to solve things
20:54:49 <sdague> anteaya: check queue
20:55:01 <anteaya> okay check queue
20:55:03 <markmcclain> when we have peak demand can we not automatically disable non-voting jobs? for instance neutron 8 of 18 and manilla 10 of 16 are non voting
20:55:30 <anteaya> markmcclain: every project uses non-voting jobs to mean different things
20:55:32 <fungi> #link http://grafana.openstack.org/dashboard/db/zuul-status the zuul job queue is probably the best indicator of demand
20:55:36 <dhellmann> that seems like a high % even during non-peak times
20:55:37 <dhellmann> both to
20:55:37 <dhellmann> do
20:55:51 <anteaya> despite what we would like, folks interpret them differently within their group
20:56:04 <jeblair> markmcclain: i guess the question is -- are non-voting jobs valuable?  if they are, we should run them; if they are not, we should not run them in check.
20:56:26 <anteaya> non-voting is supposed to be a step towards voting, but that isn't the reality for some
20:56:35 <ttx> some/most
20:56:37 <sdague> anyway, this was an intent not to solve the issue right here, because I think seeing our growth curves it's an important conversation to be having over the next cycle, with the TC folks being part of it
20:56:47 <ttx> sdague: and thanks for raising it
20:56:53 <annegentl_> sdague: yep thanks
20:56:53 <dhellmann> anteaya : do you have any idea what % of the jobs we run are non-voting overall?
20:57:07 <jeblair> it's also worth noting that we have recently onlined some more clouds, and have others in the pipelin
20:57:17 <sdague> jeblair: yep, which definitely helps
20:57:17 <fungi> also prioritization is hard to tweak, because as soon as we prioritize one thing (say, gating) then we discover how much people relied on faster turn-around on others (check results, post-merge publication)
20:57:21 <anteaya> dhellmann: I do not, but I bet AJeager would have those numbers quickly
20:57:27 <mordred> sdague: I love everyone on the TC - but do you have ideas of which ways are you thinking the TC can be helpful?
20:57:29 <anteaya> dhellmann: can I get back to you on that?
20:57:31 <jeblair> including infra-cloud (which ran 1 some jobs last week!)
20:57:41 <dhellmann> anteaya : sure, I expect this is going to be a topic of discussion for a while
20:57:42 <sdague> but our project and test growth rates seem to be accellerating faster than that
20:57:52 <anteaya> dhellmann: I'll take that as an action item
20:58:02 <dhellmann> mordred : I expect we're going to need to set some usage policies, don't you?
20:58:07 <sdague> mordred: honestly, this feels like a more important issue to get a hold of than new tags. Maybe that's just me :)
20:58:19 <dhellmann> sdague : no, I'm on board with that, too
20:58:47 <mordred> sdague: I agree it's super important - but I also just want to make sure we have an idea of how we think productive looks. perhaps usage policies for sure
20:58:49 <flaper87> sdague: dhellmann count me in
20:59:00 <ttx> No more time to discuss stale reviews, we can do that next week
20:59:00 <sdague> mordred: usage policies and norms
20:59:15 <sdague> and leadership back in projects we're a part of
20:59:15 <dhellmann> it would be interesting to look at things like whether we could combine any of the shorter jobs so they run together, in parallel with the longer jobs, to use fewer nodes
20:59:16 <ttx> Small heads-up, please review project team guide changes ! We have a number of changes stuck in that queue
20:59:19 <ttx> #link https://review.openstack.org/#/q/status:open+project:openstack/project-team-guide
20:59:20 <mordred> sdague: k. that's also helpful in infra being able to prepare/provide input data that can help those
20:59:31 <sdague> yep, for sure
20:59:33 <mordred> like, depending on the quality of te discussion, different reports and data might be useful
21:00:14 <dhellmann> will this be the subject of an infra session at the summit?
21:00:18 <dhellmann> or cross-project?
21:00:24 <rockyg> ++
21:00:34 <jeblair> dhellmann: that suggestion feels more like a technical infra suggestion than a tc topic?
21:00:42 <flaper87> jeblair: ++
21:00:46 <jeblair> dhellmann: (your combining jobs idea)
21:00:47 <ttx> ok we are out of time
21:00:57 <ttx> Thanks everyone
21:01:02 <flaper87> o/
21:01:04 <dhellmann> jeblair : yeah, definitely
21:01:05 <flaper87> cheers
21:01:06 <flaper87> tty next week
21:01:09 <mestery> thanks ttx
21:01:11 <ttx> #endmeeting