19:00:48 #startmeeting 19:00:49 Meeting started Tue Nov 15 19:00:48 2011 UTC. The chair is mtaylor. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:00:50 Useful Commands: #action #agreed #help #info #idea #link #topic. 19:00:57 I'm here :) 19:01:08 i'm here too 19:01:16 check that out - it's a party 19:01:18 yay 19:01:19 yay ci 19:01:33 #idea beer 19:01:38 hehe 19:02:05 you know what - I think I want to start with ... 19:02:13 #topic rearranging servers a little bit 19:02:50 we're really close to rolling some code into production that creates and deletes cloud servers accounts as needed 19:03:07 so it seems like a terrible idea to have things arranged such that a bug in a script could just delete the jenkins server 19:03:19 or the gerrit server 19:03:28 ha! 19:03:50 to that end, we've set up a couple of additional cloud servers accounts - one to hold ephemeral script-created servers 19:04:24 and one to hold dev resources - jenkins, gerrit, the servers that hold the dev docs and I think the ones that hold the irc bots 19:05:05 the other ones web resources that are in the current openstacak account I think should stay there - because they're not really things that fall under openstack-ci team management 19:05:22 anybody have any general dissent to that plan or concerns about it? 19:05:58 awesome 19:06:23 #agreed we'll move the servers around into a set of web-resources, a set of dev-driving machines, and a set of ephemeral machines 19:06:46 #topic state of integration testing 19:06:58 jeblair: you wanna update us on where integration testing is? 19:07:02 sure 19:08:04 i'm quite close to having on-demand devstack configured vm testing of changes we can use for gating 19:08:11 basically that means: 19:08:48 we'll have a job that launches a vm, has devstack configure it, and run tests on it, then destroy it 19:09:06 for now, the tests we're going to run are just the "exercises.sh" tests in devstack 19:09:16 but that's really a placeholder for the openstack-integration-test suite 19:09:25 once that's ready for gating, we'll switch to that 19:09:42 for the moment, i'm only focusing on the stable/diablo branch 19:09:57 since i don't believe devstack can configure trunk at the moment 19:10:13 fantastic 19:10:34 i'm working with jesse on moving devstack into gerrit, and i believe he'll be ready to focus on trunk soon 19:10:36 are there plans to run off of essex1? 19:11:14 the test job will be just like the others, eventually it will run on every branch 19:11:30 cool. :) 19:11:41 so that'll be master, and milestone proposed 19:11:59 but not specifically essex1, which is now just a tag since it was released 19:12:15 (future development should happen on master) 19:12:36 the scripts are all here: https://github.com/openstack/openstack-ci/tree/master/slave_scripts 19:12:39 devstack-* 19:12:56 and the jenkins job is here: https://jenkins.openstack.org/view/All/job/dev-gate-integration-tests-devstack-vm/ 19:13:02 #link https://github.com/openstack/openstack-ci/tree/master/slave_scripts 19:13:08 #link https://jenkins.openstack.org/view/All/job/dev-gate-integration-tests-devstack-vm/ 19:13:11 what's the timeline for gating with openstack-integration-tests? 19:13:15 it's obviously not done yet 19:13:23 as in, i'm literally in the middle of configuring it 19:13:59 but it is demonstrating a couple of new things we're doing with jenkins: 19:14:17 it's triggered on multiple projects; so a change to any of nova, glance, keystone will all trigger the job 19:14:22 and it's running in 'silent mode' 19:14:35 which means that it does not report its output back to gerrit, and does not vote on changes 19:15:00 that's an important feature for us to get complicated jobs like this up and running without interfering with ongoing development 19:15:23 the multiple-job trigger is pretty bad-ass 19:15:27 dolphm: i'm not sure anyone has dates, but: 19:16:27 i think this setup should be ready to go this week. we want to get python-novaclient and devstack into gerrit before we get too serious, hopefully this week or next. after that, as soon as openstack-integration-tests say they're ready 19:17:05 we want all the important components, devstack included, gated on this job so that we know if someone changes devstack, it still works for testing 19:17:43 mtaylor: yes, since we can trigger on multiple jobs, that means this one jenkins job can be used to gate all the projects 19:17:52 (all the projects we're interested in gating, that is) 19:18:13 jeblair: that's so much nicer than having a bazillion different jobs 19:18:25 and it means we can ensure changes to them are serialized across all projects 19:18:43 (is that clear, or should i elaborate on that?) 19:18:57 makes sense to me 19:19:11 very clear 19:19:21 (and awesome) 19:19:41 cool. so that's the current status. sorry i don't have green lights in jenkins to show yet. but at this point all the individual components have been tested. just assembling them now. 19:19:55 oh, one more thing 19:20:15 we're going to try an idea ttx had at uds: 19:20:29 if a change fails the devstack driven integration tests, 19:20:42 we install the developer's key on the vm and hand it to them. 19:21:00 so the vm isn't deleted? 19:21:07 they have, say, 24 hours to work on the very VM where the test failed and see what went wrong and try to fix it. 19:21:10 but is deleted on success? 19:21:17 dolphm: that's the idea 19:21:24 that's pretty damned cool 19:21:26 (up to a point) 19:21:39 yeah, i think it'll be pretty neat. :) 19:22:17 any other questions about this? 19:22:31 if the dev is asleep for those 24 hours, how do they re-run the job to get a failed vm back up? 19:23:08 you can re-trigger a job in jenkins, or by +2'ing it again in gerrit 19:23:59 no everybody can +2.. can everybody retrigger? 19:24:26 not* 19:24:49 dolphm: I'm not positive about that. 19:25:21 I do not believe everyone can retrigger 19:25:22 BUT 19:25:33 I betcha we can figure something out there 19:25:44 i'm just thinking, as this becomes a popular/necessary workflow, if 24 hrs isn't sufficient, I'd like devs to be able to get a failed vm on their own, with minimal effort 19:25:47 jeblair: have a branch that works with trunk fyi 19:25:59 vishy: awesome 19:26:03 * mtaylor makes out with vishy 19:26:17 ... 19:26:26 * dolphm hides 19:26:28 had to disable one test because python-novaclient is out of date 19:27:01 do we have a date for moving python-novaclient over yet? 19:27:09 we should have devstack tagged diablo/stable and have a master running trunk in the next couple days 19:27:22 no, i proposed tomorrow morning to sandy, but haven't heard back 19:27:57 I'm going to try to get the git-based pip-requires up and going for all the projects this week too, and finish moving the other projects to using pip-based unittest builders 19:28:07 so we should be consistent across the board real soon now 19:28:22 vishy: i emailed jesse about moving devstack to gerrit; we need to do that soon too, before we can start using it to gate 19:29:15 mtaylor: yay! 19:32:34 so I think that's where we're at with just about everything 19:33:03 #topic anything else 19:34:06 I have a gerrit question 19:34:15 (not sure if this is the right forum) 19:34:21 go for it! 19:34:39 when is -1 vs -2 appropriate? 19:35:07 -2 functionally blocks it from being merged 19:35:11 forever? 19:35:19 until the -2 is lifted/changed 19:35:33 by another review from that same person? 19:35:35 in fact - -2 sticks around even after a new change is pushed 19:35:38 yes 19:35:43 ok 19:35:55 so -2 is throwing your weight down and saying NO 19:36:25 but, "this is mostly ok but fix A,B,and C" should probably be -1, then? 19:36:48 probably 19:36:57 i think so, and your -1 will be automatically removed when there is a new patchset. 19:37:06 and -2 is "no, I don't think adding the ability to make toast is a good feature to ever add to nova" 19:37:54 notmyname: you might consider using it also as a way to ensure that you review a subsequent change 19:38:03 ok. makes sense. thanks 19:38:12 as in: "I want you to fix this, and make sure that it's fixed to my satisfaction before it goes in" 19:38:15 we could consider suggesting a policy around -2 similar to +2 ... 19:38:42 like, people vote in general, and a core member +2's or -2's based on the general consensus of the voting? 19:39:03 but yeah - we can also just try playing with it for a while and seeing how we like it 19:39:09 mtaylor: i want core people to all be able to +2 eventually 19:39:21 ie, we should support multiple +2 votes 19:39:32 once we have the gerrit trigger plugin updated 19:39:35 jeblair: yes. 19:39:53 so i don't want to build too much policy around what is currently a technical limitation :/ 19:40:06 ++ 19:40:07 or if we do, understand where we're going in the future 19:40:55 I also have a gerrit question / feature request 19:41:37 vishy: go for it 19:41:45 i find it quite annoying that repushing covers up previous reviews, especially when the push is just a rebase 19:42:18 yes. I do too 19:42:20 i don't have a solution, but it would be nice to be able to say, just a rebase when merging 19:42:22 right, the trivial rebase problem... 19:42:29 we have a todo list item on that 19:42:29 and it keeps the previous +1s 19:42:52 https://bugs.launchpad.net/openstack-ci/+bug/881184 19:42:54 Launchpad bug 881184 in openstack-ci "consider trivial rebase hook for gerrit" [Wishlist,Confirmed] 19:42:54 #action mtaylor translate all of the gerrit/jenkins todo list items into bugs/blueprints 19:43:08 wow, mtaylor, that was quick! :) 19:43:12 (all the "trivial" rebasing also makes the master branch look ugly) 19:43:40 so what that hook does is: 19:43:42 "This hook is designed to detect when a patchset uploaded to Gerrit has the same git patch-id as the previous patchset. It then reapplies reviews onto the new patchset using the Gerrit SuExec and Approve commands." 19:43:52 there's also been talk about a UI patch to make some of the review data show more prominiently by default, rather than needing to expand a section 19:44:01 notmyname: I disagree with your comment about toast 19:44:01 * vishy has been waiting for the toast patch for a while now 19:44:03 which i believe would solve vishy's problem. 19:44:33 obviously it should be easy to install, we just need to make sure it does what it advertises 19:44:42 and works with our workflow 19:45:24 vishy: well that's where the -1 or -2 levels come in ;-) 19:46:47 Hmmm. Toast... 19:47:32 * mtaylor is hungry now 19:48:29 mtaylor: turns out that brazillian food in brazil is exactly the same as brazillian food in the US. I suggest you go get some churisca (sp? whatever.) tonight 19:48:48 notmyname: yes. it's quite tasty isn't it? 19:48:56 meeting over? :) 19:49:11 I think so 19:49:14 #endmeeting