20:01:47 #startmeeting tc 20:01:48 o/ 20:01:49 Meeting started Tue Mar 1 20:01:47 2016 UTC and is due to finish in 60 minutes. The chair is ttx. Information about MeetBot at http://wiki.debian.org/MeetBot. 20:01:50 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 20:01:52 o/ 20:01:52 The meeting name has been set to 'tc' 20:02:04 Good $timeofday everyone 20:02:10 Our agenda for today: 20:02:14 #link https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee 20:02:23 #topic Add a "stable:follows-policy" tag 20:02:29 #link https://review.openstack.org/277918 20:02:30 * edleafe hides in the shadows 20:02:38 * rockyg squeezes into the back of the room 20:02:41 So this is a tag that the stable maintenance team can use to certify that a given deliverable stable branches are following the common stable branch policy 20:02:45 * sridhar_ram1 lurks 20:02:53 That is made necessary because in the big tent there are a lot of stable/* branches that do not really follow policy those days 20:03:04 (and we still want to call them stable/* for other reasons) 20:03:12 So it is a pretty important piece of information to communicate to those who consume our deliverables 20:03:21 missing a few votes 20:03:36 I think mriedem addressed anne's remark 20:03:45 questions ? 20:03:49 lgtm 20:03:54 lgtm 20:04:00 * dhellmann wonders if we want to start migrating to series/* instead of stable/* 20:04:14 dhellmann: there are lots of things that encode stable/* 20:04:19 yeah 20:04:19 dhellmann: there would be a pretty big cost to all our tooling for that 20:04:23 (hardcode) 20:04:39 dhellmann: but yes you are right, that's what it means today, hence the need for the tag 20:04:44 * thingee lurks 20:04:45 still something about this seems wrong 20:04:59 i don't like a tag that says something follows a policy 20:05:00 hello ... is this the TC meeting? (https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee) 20:05:10 amitry: yes. 20:05:13 amrith: yes 20:05:19 oops. 20:05:24 jeblair : we have the tag for following the deprecation policy, too. isn't this the same? 20:05:29 dhellmann: ++ 20:05:33 dhellmann: +1 20:05:33 remembering the 3 weeks of trying to get the ceilometer grenade bits into a passing state because gnocchi had a custom branch structure 20:05:44 dhellmann: slightly different, the other one is an assertion by the project itself 20:06:00 jeblair: isn't that what the assert:XXX tags are? 20:06:04 kinda? :) 20:06:18 ttx: I suppose that's a reasonable distinction to make. 20:06:34 here the stable team grants it, so it is indeed a bit of a carrot/stick for the stable team to encourage policy following 20:06:51 not sure that is what jeblair objects to 20:07:01 the reason we hardcoded stable in many places was because it meant the same thing 20:07:17 honestly, the stable team job is one of the more thankless jobs out there. If they feel this helps them accomplish it, I'm all for it. 20:07:22 jeblair: currently in infra it means "series" 20:07:23 o/ 20:07:30 sdague: ++ 20:07:33 sdague: ++ 20:07:33 sdague: yep 20:07:46 why not say "stable/" means this thing, and if you do something that doesn't mean that, call your branch something else? 20:07:50 yeah, we have a conflict between the way it has ended up being used and the way it was intended to be used 20:08:09 jeblair: I proposed that but that would break a lot of things apparently 20:08:09 jeblair: I could definitely get behind that. 20:08:19 ttx: what does it break? 20:08:24 could we rename existing branches? 20:08:24 jeblair: I think I raised it at an infra meeting and you shot it down 20:08:37 at least i'm consistent ;) 20:08:46 lol 20:08:50 I also think the lever there is weird 20:08:52 ttx: i'm still trying to understand this though 20:09:03 jeblair: if not you, maybe fungi 20:09:03 (i'm exploring, not advocating) 20:09:09 because project/foo thinks they are following stable branch policy 20:09:18 we have 3 types of stable branches: actually stable, using stable/$series for convenience, and using stable/$version 20:09:20 so they have stable/mitaka 20:09:29 stable maint team notices they aren't 20:09:44 stable maint team has authority to delete project/foo stable/mitaka branch? 20:10:02 It'd be better for stable/* to actually be stable, fwiw. 20:10:06 so yeah, even if we don't rename things we want to give the stable team a flag to say "yes, we agree that this is stable" 20:10:12 maybe instead of deleting the branch, we remove the project 20:10:15 dhellmann: right, which is that 20:10:23 jeblair: o_O 20:10:37 not all projects must follow the stable policy 20:10:38 the only project we've removed before was one that was dead for 2 cycles 20:10:40 as if we could remove a project 20:10:45 jeblair : rename to openstack-not-actually-stable/$project ? 20:10:47 i think i didn't so much object, as express that devstack-gate's current branch mapping would get waaaay more complex if you needed to try to assert that your stable/1.3.0 or foo/bar branch should be tested against stable/liberty of some other project 20:10:57 wobbly/1.1.0 20:10:58 right, I don't believe removing a project is a thing that actually exists as a TC power 20:11:01 fungi: right 20:11:02 annegentl_ : ++ 20:11:02 fungi: absolutely 20:11:04 sure it is 20:11:07 (removing a project) 20:11:09 sdague: oh it definitely does 20:11:14 if we can add, we can remove 20:11:16 we just never do it 20:11:17 ok, I'll believe it when it happens 20:11:20 stable-but-having-experimental-feature-backports/$project is too long of a branch name. 20:11:21 heh, right 20:11:24 sdague : I think he just means as an official project, not from gerrit? 20:11:30 I'm not saying this can't be implemented differently, I'm just saying a tag is the most convenient way to fix it for now 20:11:36 jaypipes : abbreviate it? 20:11:42 dhellmann: :) 20:11:44 to try to get back onto the topic -- 20:11:46 dhellmann: it is an un-unit tested piece of policy 20:11:47 I like the tag as an immediate solution 20:11:52 the last time i remember it coming up was when some project (maybe an oslo lib) wanted more than one stable/x.y.z that would cross-test against a specific stable branch of other repos in devstack 20:11:54 if we find a better solution, we can always remove the tag 20:11:56 ttx: I agree. I think we want this tag regardless, because there's the "we're trying to be stable" state to consider. 20:11:57 and I'd like to fix it now before we cut stable/mitaka all around 20:12:03 starting tomorrow 20:12:04 the thing that's sticking with me is that we're giving our users an inconsistent story 20:12:15 I like the idea of renaming fake stable branches, fwiw 20:12:19 jeblair: ++ 20:12:29 jeblair: or is it quality checking for truth? 20:12:37 it's bad enough that people think that "stable/liberty" is some how "more hardend" 20:12:37 jeblair: I get your point. I think we can fix it better tomorrow, but it's more work 20:12:42 we're saying "stable/*" may or may not be stable 20:12:48 jeblair: I mean, I see it either way, yeah. 20:12:50 jeblair: it's inconsistent now, anyway. The benefit of this tag is that it's provided by a team we trust 20:12:51 rather than "has a different policy for landing patches" 20:12:53 jeblair: we are admitting the truth 20:13:12 but as soon as we even step away from _that_ with things that have the openstack title on it 20:13:15 If we decide to replace the tag with some ownership of a branch space, we can do that another day 20:13:20 because the truth is stable/* may or may not be that today, which is what triggered it 20:13:28 But I expect that would run into a lot of implementation issues 20:13:38 so, step 1 - acknowledge the state of the world, write it down 20:13:42 like... projects can create branches 20:13:46 sdague: ++ 20:13:49 right, it would be useful to be able to declare what is actually stable and then go through the exercise of fixing the things that aren't 20:13:50 step 2 - figure out how you want to change it 20:13:54 I also proposed in the review that we could find a way to tag each stable branch individually 20:13:58 sdague: yep, baby steps 20:14:03 but again, I think the current proposal is fine for now 20:14:11 and we should evolve from there 20:14:13 Nicely put sdague, +1 20:14:25 sdague: sure, i'm just more aspirational and would like us to describe what we want things to be, rather than to follow the project and try to clean things up behind it. 20:14:35 i typed that a while ago 20:14:44 sdague: i guess i'm jumping to step 2 :) 20:15:02 OK, let's approve step 1 and then someone can propose step 2 20:15:04 jeblair: right, and if stable maint wasn't already such a giant thankless job, that would be fine 20:15:08 Ideally, I'd like each stable branch to actually be stable and follow the stable policy but that's not the goal right now 20:15:24 but I think it needs a lot more people piled into helping on stable maint before step 2 is doable 20:15:28 I agree with that long-term goal 20:15:35 sdague: yeah. i don't think a -1 from me will help step 2 happen. 20:15:37 sdague: exxxactlk 20:15:39 y 20:15:46 jeblair: right 20:15:53 ok, let's move on 20:16:03 we have 9 YES 20:16:15 jeblair: want to write down your aspirational next step on the review before I approve ? 20:16:21 jeblair: ayup 20:16:55 ttx: done 20:17:02 alrighty approved 20:17:12 #topic Clarify test-only license requirements 20:17:16 #link https://review.openstack.org/279999 20:17:28 So this is definitely a good clarification to add, and it now has enough votes to be approved 20:17:52 do we have lifeless around 20:17:54 I'd like lifeless to expand on his point, if we have a minute for that? 20:18:15 lifeless mentions that this does not help with Scapy (being considered at https://review.openstack.org/#/c/277893/) 20:18:19 because even with this clarification, I'm not comfortable approving gpl additions to the requirements list 20:18:29 I suspect his argument being we can't import libraries without creating derivative work, be it test-time or run-time code ? 20:18:53 dhellmann: right, or leaving MySQL-python in 20:18:59 right 20:19:04 It appears to still be used in trove: 20:19:05 http://git.openstack.org/cgit/openstack/trove/tree/requirements.txt#n38 20:19:09 Although I couldn't find an import so it might just be a leftover 20:19:23 did you look in oslo.db? 20:19:40 I do not htink lifeless point is relevant 20:19:53 dhellmann: I'd say that the clarification is still a clarification, although it doesn't really clarify whether test-only deps are ok 20:20:03 mordred : not relevant to this, or not relevant to the patch adding scapy to the requirements list? 20:20:04 the license terms are about terms for cpoying 20:20:09 not relevant to the patch adding scapy 20:20:28 I do not think that openstack projects using scapy in testing produces any license violations 20:20:31 imho 20:20:43 ok, I'm not really worried about that case per se 20:20:47 mordred: that is where I was too 20:20:53 I'm worried about the fact that we have no way to enforce it going further than that 20:20:57 mordred : yes, if it's in the base infrastructure itself. not in the project requirements 20:21:00 mordred: i agree (regardless of where you fall on the import==derivative question) 20:21:16 mordred: I would agree, we don't however have any mechanism for keeping things out of requirements.txt once it's in global-requirements.txt 20:21:19 I think that's another discussion though. Difficult to have without lifeless to sustain his argument 20:21:26 gr had been a bit more of a firewall up until this point 20:21:27 I think if people have not read the recent article by eben related to the ubuntu+zfs case 20:21:30 you should 20:21:30 sdague: taht's another technical issue 20:21:47 because eben makes some REALLY execellent points hwich are analogously relevant here 20:21:50 I'm still +2 on that, but it's a thing people concerned about this should raise 20:22:19 mordred: linkety? 20:22:19 OK, I propose we approve this and raise that discussion on another forum, like legal-discuss, with lifeless around 20:22:35 ttx: ++ 20:22:43 ttx: sounds good 20:22:43 https://www.softwarefreedom.org/resources/2016/linux-kernel-cddl.html 20:22:44 #link https://www.softwarefreedom.org/resources/2016/linux-kernel-cddl.html 20:22:50 thanks! 20:22:51 worth noting that if the unified global requirements list remains our sole gatekeeping tool for this, then it's not actually effective if we start comingling test-time dependencies not held to the same standard in that list and we'll need to come up with a more fine-grained solution 20:23:10 fungi: yes, that's the point dims raised 20:23:18 oh, yep, i missed that comment 20:23:24 yes, we should have one at some point. For the moment I'll probably do due diligence on problematic deps 20:23:26 i don't actually think we intended it to be that, however 20:23:44 fungi : yeah, we'll have to talk about that at some point soon, I think. 20:23:56 ok, approving now, and deferring taht discussion for when lifeless and mordred are here at the same time 20:24:41 done 20:24:50 #topic Applying Tacker for Big Tent 20:24:58 #link https://review.openstack.org/276417 20:25:06 Round 3... As I said in the meeting reminder, would be good if we could come to a final decision here 20:25:18 All stated objections hinge on whether we trust Tacker can become a generally-useful OpenStack glue service rather than an NFV-infeodated purpose-built narrow application 20:25:25 We can either trust that they will become that and approve them now 20:25:33 Or we can apply caution and ask that they demonstrate being generally-useful first, and delay the application 20:25:51 (or assert that they can't become that and reject them outright) 20:26:03 I'm slightly leaning on the "trusting" side because I fear the catch-22 (not being able to truly be generally useful until it's officially adopted) 20:26:15 That said, the lack of TC member votes and enthusiasm around it should push us to caution 20:26:22 I think we should in general lean to trust 20:26:31 Still missing 6 votes, current expressed votes lean towards including it 20:26:32 So I agree with you ttx 20:26:36 I'm on the trusting side. We've applied the same logic on other projects and we can evaluate again later 20:27:08 generally trust as well 20:27:13 I have not changed my mind on Tacker. I continue to see it as a purpose-built application for telco that consumes a variety of OpenStack APIs and services. 20:27:14 I'm OK with trusting them, but would like to understand what it means if they don't? 20:27:22 I think they have shown good intentions and practice, I think we can trust the team so far 20:27:28 But I am still not certain this needs to be an OpenStack project. It looks like an application layer to me. 20:27:33 missing lifeless, mordred, dtroyer, jeblair, sdague, dhellmann 20:28:01 yah. you are missing me 20:28:09 mordred: I do. 20:28:10 dtroyer: I feel like that's the case for trove then too 20:28:11 do we set a time-frame? do we revisit this at some point? 20:28:12 it sounds like we should have a more general conversation about where we'd like to draw an upper scope bound 20:28:16 I'm having trouble developing enough of an opinion on this to vote, tbh 20:28:16 and write that down 20:28:19 russellb: yeah 20:28:23 russellb : ++ 20:28:29 annegentl_: maybe? although Trove is more general. 20:28:31 because other than that, this is fine to me 20:28:32 mordred: same 20:28:37 I'd like to separate that discussion about scopes/bounds from this review though. 20:28:42 here are my thoughts, in no particular order 20:28:47 mordred: and I feel bad for voting as well. I have not nearly the same conviction as jaypipes has, yet my vote carries the same weight 20:29:01 and yet we need to, that's what we were elected for 20:29:03 a) in the big tent, I care most about humans who are like us doing development on things kind of like we do development on things 20:29:17 I don't particularly pass a lot of judgement on the validity of the thing itself 20:29:21 * jeblair expands voting scale 10x for ttx to register a 1/10 vote 20:29:24 +1 (with conviction) ? 20:29:28 ttx: the weight of one's conviction has never been a good indicator of correctness :) 20:29:49 +0 20:29:56 b) there is obviously an upper bound where things are just apps on top of openstack ... but I'm not sure if I have a good basis to judge that at th emoment 20:30:02 i've tried to balance out jaypipes' conviction with equal and opposite conviction, to make it totally unclear where to go next 20:30:07 jaypipes: heh 20:30:12 russellb: my hero 20:30:31 heh 20:30:44 russellb : helping! 20:30:51 russellb: and then I summarized it with plenty of doubts and registered my +0 20:31:00 this is in an upper bound grey area, but it's far from alone there 20:31:04 recipe for success and quick decision 20:31:06 c) I would like to do things to help our opnfv friends 20:31:06 IMO.. 20:31:20 ttx: :) 20:31:27 ¯\_(ツ)_/¯ 20:31:36 mordred: teamwork! 20:31:57 we can totally decide that we need to think about it a lot more and defer the decision to next cycle 20:31:58 so, I thinkn I'm leaning towards a yes 20:32:13 largely because I think that to reject something from the big tent I need to have a specific reason for the no 20:32:21 we can also say yes but with the damocles sword of being removed based on that future discussion on scope 20:32:22 i'd propose yes for now, with an increased priority for discussing scope 20:32:29 I still think caution is warranted and we should defer... would give team time to implement the integrations they've committed to 20:32:31 and an understanding that we may revisit the projects in the grey area based on the outcome 20:32:36 russellb: +1 20:32:50 russellb: that is to me a reason to either defer or decide no 20:32:58 I'm not at ALL against helping out our OPNFV friends. I'm just saying I don't believe a purpose-built Telco MANO application "is OpenStack" and furthers the mission of OpenStack to be the ubiquitous cloud computing infrastructure for private and public clouds. 20:33:41 * flaper87 will learn all the acronyms before this discussion ends... promised! 20:33:43 russellb: "provisionally approved pending further discussion on scope" ? 20:33:44 mordred's *rousing* speech seems to have gathered a few votes 20:34:10 I'm formally abstaining 20:34:14 I tend to agree with mordred that under the big tent I'm not seeing how I'd vote no on this one. 20:34:16 remember Donald Trump will soon be POTUS and the end of the world is near anyway 20:34:20 This bike shed is starting to look like a psychedelic barber pole. That was lit on fire. 20:34:20 I did register it as such 20:34:27 ttx: cory booker is in the wings 20:34:35 ttx: I have hope for him post trump 20:34:48 jaypipes: I think that if it helps NFV folks use openstack as their cloud, then it helps that. I do not know if it will help that, but I do not think it will explicitly hurt that 20:35:21 jaypipes: so worst case it seems to hurt no more than any of the other margianlly helpful things we have, and best case maybe it helps a little? 20:35:29 mordred: that's my feeling 20:35:32 8 rollcall +1 votes now 20:35:33 OK, we have 8 votes yes at this stage, probably because it's easier to say yes than no. 20:35:36 mordred: ++ makes ubiquitous openstack more ubiquitous 20:35:44 ttx: Isn't that the big tent's premise? 20:35:49 I'm fine with making it clear this is provisional approval 20:35:56 pending more discussion 20:35:59 ttx: Have we done a provisional approval before? 20:36:01 no 20:36:04 OK 20:36:07 all approvals are proviosnal 20:36:07 welcome to big tent era 20:36:10 lol 20:36:11 True 20:36:30 it's just that we need to psychologically prepare projects to be kicked out 20:36:35 I have my list 20:36:44 rockyg: or does it make NFV architecture and ETSI more ubiquitous? 20:36:59 jaypipes, Both? 20:37:00 some projects are going horribly wrong and should definitely not be openstack 20:37:05 rockyg: one is not the other 20:37:06 these folks are rallying around openstack, and i think we should be supporting that 20:37:25 rockyg: and it is that equivocy that I see as dangerous. 20:37:26 russellb: but they're rallying around openstack on their terms 20:37:29 Right, this makes openstack AND NFV/ETSI more ubiquitous IMHO 20:37:30 so we might not have kicked out anyone but it may happen. Especially now that we don't even have to rename the repos 20:37:32 markmcclain: huh? 20:37:34 ttx: so happy to hear you say that 20:37:42 rockyg: the whole-hog consumption of OPenStack by the telco industry consortium is a real danger. 20:37:53 * dhellmann can't wait to see ttx's list 20:38:02 dhellmann: ++ 20:38:10 jaypipes, very true. 20:38:28 dhellmann: projects without an install guide are up there on my list 20:38:29 ttx: if it's not tested it's broken? 20:38:35 ttx: ++ 20:38:55 anyway, back on topic 20:39:03 should we move on? I think there are enough votes to make a decision 20:39:09 flaper87: ++ 20:39:10 russellb: basically what japypipes said... telcos want a very specific vision for OpenStack 20:39:17 dtroyer: you should register your vote as RollCall 20:39:18 rockyg: if you don't see the danger that a single industry driving feature requests and product direction is, please have a chat with any nova core contributor about how NUMA, CPU pinning, SR-IOV and PCI functionality has gone. 20:39:23 jaypipes, can enterprise world adoption help balance the scales? 20:39:25 markmcclain: everyone has a vision 20:39:29 ttx: thanks, done 20:39:32 anyway, moving on 20:40:19 8 vs. 3, one abstain 20:40:36 I think that means yes 20:40:58 it's missing lifeless' vote but that won't change the result 20:41:18 Unless lifeless convinces all +1's to be -1's 20:41:20 Although I agree with dtroyer's point. We are creating a larger grey mess we'll have to fix one day 20:41:21 :D 20:41:45 I think it's fair to say we'll revisit that grey mess in Newton 20:41:51 why not? Or every cycle 20:42:02 I think revisiting that grey mess should be our priority in Newton 20:42:08 ttx: ++ 20:42:11 ttx: I'd agree with that 20:42:12 Spring cleaning 20:42:14 :) 20:42:15 We had two grey things being submitted recently 20:42:19 fine with me. 20:42:29 previously that was easier 20:42:29 grey matter for grey mess 20:42:32 i'm sure we'll have plenty to argue around that 20:42:55 alright, closing the vote now 20:43:40 It is in 20:43:51 #topic Open discussion 20:43:55 * OpenStack CI resources vs. project growth (sdague) 20:43:59 sdague: floor is yours 20:44:19 sure, this is something I wanted to bring up in looking at the delays and turn around time we're hitting as we hit milestones 20:44:25 we've got project growth 20:44:30 growth of testing on projects 20:44:48 and are actually currently down on cloud resources from the last 3 releases 20:45:14 and while that last one may get addressed, I think the growth is always going to be pushing boundaries 20:45:37 for instance, there was a nodepool issue over the weekend, which basically put an 8 hour delay into everything yesterday until it could grind overnight 20:45:38 sdague: so so true 20:45:50 for other cross project concerns also 20:46:09 annegentl_: I think infra is where the growth is creating the most impact though 20:46:11 we've even been hampered a bit with releases because of the check queue depth 20:46:15 so any hiccup in the system can trigger a cascade that taks a couple of days to clear 20:46:42 ttx: yeah except for lack of install guides will cause longer-term delay in adoption. CI resources are easier to measure direct hit 20:47:00 I think it's too late to address anything for mitaka, but I wanted to get a conversation going around this 20:47:09 sdague : are you building up to a proposal, or just starting the conversation? 20:47:10 yeah, the real shame is that the nodepool issue was addressed hours before we got into peak usage. it could have been far worse if it were just a little later in the day 20:47:18 not saying one's more "valuable" but that one's easier to visualize on a timeline 20:47:38 dhellmann: just the conversation 20:47:43 annegentl_: sure, "most visible impact" 20:48:08 sdague: so... solutions include: more test hosts, less tests/project, less projects 20:48:18 any other solution ? 20:48:24 more guidance on use of test resources? 20:48:27 or finding a way to deprioritize jobs for some projects and prioritize others 20:48:36 right, prioritization on peaks 20:48:41 that's a 5th thing 20:48:51 fungi: interestingly we looked at that a long time ago, and the use of ci resources by non "core" projects was almost negligible 20:48:55 FOr example, non-voting jobs seem to be completely ignored by most, yet consume a lot of resources 20:48:58 yeah, I was thinking if we reserved CI for official projects during peak times that might help, but only during those times and we'd move the peaks from other projects to other times 20:49:08 i actually doubt it is that much different now 20:49:13 jeblair : does that still hold? 20:49:15 gearman hampers us there a bit since the protocol has a designed-in 3-level precedence implementation 20:49:16 one question I have is, has anyone measured 3rd party impact 20:49:28 annegentl_: what do you mean 20:49:30 ? 20:49:34 and would more 3rd party testing help with resources? 20:49:35 annegentl_: what impact 20:49:45 so, again, I think we come back to step 1 - do we have a real problem? 20:49:46 how would you invision? 20:49:47 dhellmann: my gut says it mostly still holds. but actual measurement might be useful. :) 20:49:49 jeblair: yeah, i suspect not much has changed there but it would be interesting to see updated numbers 20:50:00 jeblair, fungi : yeah 20:50:05 annegentl_: the testing is different, proprietary vs open 20:50:14 anteaya: wondering if we look at all testing as a pie, is 3rd party testing able to take on more of a large pie? 20:50:24 only to benefit themselves 20:50:26 anteaya: ok 20:50:41 third party testing has no benefit to openstack trunk code 20:50:43 sdague: I think we do have a problem yes. We could ramp up CI resources to accommodate growth before, and we are struggling to do that currently 20:50:44 sdague: i think this is timely -- we are basically at capacity right now (where my definition of at capacity means our ability to finish 24h worth of jobs in 24h) 20:50:44 when we have these periodic bottlenecks, how under-resourced are we? (ballpark) 20:51:17 annegentl_: anteaya: right, the "third-party" solution is additional parties with available cloud resources donating quota to us so we can use that to run first-party (upstream) jobs there 20:51:24 10%? 50%? 100%? 20:51:38 jeblair: time machine? 20:51:55 I would say 50%, but fungi should know better 20:52:00 jeblair: right, I think we have different definitions of at capacity, because if you can't turn around results on your patch in a timely manner it hampers the ability to address issues only seen in our gate 20:52:00 jbryce: we would use any resources given to us to their maximum quota 20:52:01 and do you think paying some subset of providers for extra capacity would hurt our ability to get resources donated? 20:52:08 jbryce: we were hitting capacity before HP sunset 20:52:25 mordred: we were, it's definitely a lot more cramped since then though 20:52:29 jbryce: it's hard to put a hard number on, because the impact is "how much longer does it take to get certain kinds ofg job results" 20:52:31 anteaya: i understand that, but i’m wondering at what level of additional resources we would not be hitting capacity 20:52:36 jbryce: so, I'd say we're at least under capacity by 600 VMs at peak times - but probably more. we are obviously working on finding more capacity 20:52:48 mordred: ok cool 20:52:55 mordred: yeh, that's about as good an estimate as any 20:53:05 how predictable is the arrival of peak times? it seems at least some of them we can see coming, eh? 20:53:14 we have graphs 20:53:20 jbryce: every milestone and feature freeze 20:53:22 that can be trended fairly accurately 20:53:25