19:00:48 <mtaylor> #startmeeting
19:00:49 <openstack> Meeting started Tue Nov 15 19:00:48 2011 UTC.  The chair is mtaylor. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:00:50 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic.
19:00:57 <mtaylor> I'm here :)
19:01:08 <jsavak> i'm here too
19:01:16 <mtaylor> check that out - it's a party
19:01:18 <jeblair> yay
19:01:19 <annegentle> yay ci
19:01:33 <jsavak> #idea beer
19:01:38 <mtaylor> hehe
19:02:05 <mtaylor> you know what - I think I want to start with ...
19:02:13 <mtaylor> #topic rearranging servers a little bit
19:02:50 <mtaylor> we're really close to rolling some code into production that creates and deletes cloud servers accounts as needed
19:03:07 <mtaylor> so it seems like a terrible idea to have things arranged such that a bug in a script could just delete the jenkins server
19:03:19 <jeblair> or the gerrit server
19:03:28 <jsavak> ha!
19:03:50 <mtaylor> to that end, we've set up a couple of additional cloud servers accounts - one to hold ephemeral script-created servers
19:04:24 <mtaylor> and one to hold dev resources - jenkins, gerrit, the servers that hold the dev docs and I think the ones that hold the irc bots
19:05:05 <mtaylor> the other ones web resources that are in the current openstacak account I think should stay there - because they're not really things that fall under openstack-ci team management
19:05:22 <mtaylor> anybody have any general dissent to that plan or concerns about it?
19:05:58 <mtaylor> awesome
19:06:23 <mtaylor> #agreed we'll move the servers around into a set of web-resources, a set of dev-driving machines, and a set of ephemeral machines
19:06:46 <mtaylor> #topic state of integration testing
19:06:58 <mtaylor> jeblair: you wanna update us on where integration testing is?
19:07:02 <jeblair> sure
19:08:04 <jeblair> i'm quite close to having on-demand devstack configured vm testing of changes we can use for gating
19:08:11 <jeblair> basically that means:
19:08:48 <jeblair> we'll have a job that launches a vm, has devstack configure it, and run tests on it, then destroy it
19:09:06 <jeblair> for now, the tests we're going to run are just the "exercises.sh" tests in devstack
19:09:16 <jeblair> but that's really a placeholder for the openstack-integration-test suite
19:09:25 <jeblair> once that's ready for gating, we'll switch to that
19:09:42 <jeblair> for the moment, i'm only focusing on the stable/diablo branch
19:09:57 <jeblair> since i don't believe devstack can configure trunk at the moment
19:10:13 <mtaylor> fantastic
19:10:34 <jeblair> i'm working with jesse on moving devstack into gerrit, and i believe he'll be ready to focus on trunk soon
19:10:36 <jsavak> are there plans to run off of essex1?
19:11:14 <jeblair> the test job will be just like the others, eventually it will run on every branch
19:11:30 <jsavak> cool. :)
19:11:41 <jeblair> so that'll be master, and milestone proposed
19:11:59 <jeblair> but not specifically essex1, which is now just a tag since it was released
19:12:15 <jeblair> (future development should happen on master)
19:12:36 <jeblair> the scripts are all here: https://github.com/openstack/openstack-ci/tree/master/slave_scripts
19:12:39 <jeblair> devstack-*
19:12:56 <jeblair> and the jenkins job is here: https://jenkins.openstack.org/view/All/job/dev-gate-integration-tests-devstack-vm/
19:13:02 <jeblair> #link https://github.com/openstack/openstack-ci/tree/master/slave_scripts
19:13:08 <jeblair> #link https://jenkins.openstack.org/view/All/job/dev-gate-integration-tests-devstack-vm/
19:13:11 <dolphm> what's the timeline for gating with openstack-integration-tests?
19:13:15 <jeblair> it's obviously not done yet
19:13:23 <jeblair> as in, i'm literally in the middle of configuring it
19:13:59 <jeblair> but it is demonstrating a couple of new things we're doing with jenkins:
19:14:17 <jeblair> it's triggered on multiple projects; so a change to any of nova, glance, keystone will all trigger the job
19:14:22 <jeblair> and it's running in 'silent mode'
19:14:35 <jeblair> which means that it does not report its output back to gerrit, and does not vote on changes
19:15:00 <jeblair> that's an important feature for us to get complicated jobs like this up and running without interfering with ongoing development
19:15:23 <mtaylor> the multiple-job trigger is pretty bad-ass
19:15:27 <jeblair> dolphm: i'm not sure anyone has dates, but:
19:16:27 <jeblair> i think this setup should be ready to go this week.  we want to get python-novaclient and devstack into gerrit before we get too serious, hopefully this week or next.  after that, as soon as openstack-integration-tests say they're ready
19:17:05 <jeblair> we want all the important components, devstack included, gated on this job so that we know if someone changes devstack, it still works for testing
19:17:43 <jeblair> mtaylor: yes, since we can trigger on multiple jobs, that means this one jenkins job can be used to gate all the projects
19:17:52 <jeblair> (all the projects we're interested in gating, that is)
19:18:13 <mtaylor> jeblair: that's so much nicer than having a bazillion different jobs
19:18:25 <jeblair> and it means we can ensure changes to them are serialized across all projects
19:18:43 <jeblair> (is that clear, or should i elaborate on that?)
19:18:57 <mtaylor> makes sense to me
19:19:11 <dolphm> very clear
19:19:21 <dolphm> (and awesome)
19:19:41 <jeblair> cool.  so that's the current status.  sorry i don't have green lights in jenkins to show yet.  but at this point all the individual components have been tested.  just assembling them now.
19:19:55 <jeblair> oh, one more thing
19:20:15 <jeblair> we're going to try an idea ttx had at uds:
19:20:29 <jeblair> if a change fails the devstack driven integration tests,
19:20:42 <jeblair> we install the developer's key on the vm and hand it to them.
19:21:00 <dolphm> so the vm isn't deleted?
19:21:07 <jeblair> they have, say, 24 hours to work on the very VM where the test failed and see what went wrong and try to fix it.
19:21:10 <dolphm> but is deleted on success?
19:21:17 <jeblair> dolphm: that's the idea
19:21:24 <dolphm> that's pretty damned cool
19:21:26 <jeblair> (up to a point)
19:21:39 <jeblair> yeah, i think it'll be pretty neat. :)
19:22:17 <jeblair> any other questions about this?
19:22:31 <dolphm> if the dev is asleep for those 24 hours, how do they re-run the job to get a failed vm back up?
19:23:08 <jeblair> you can re-trigger a job in jenkins, or by +2'ing it again in gerrit
19:23:59 <dolphm> no everybody can +2.. can everybody retrigger?
19:24:26 <dolphm> not*
19:24:49 <jeblair> dolphm: I'm not positive about that.
19:25:21 <mtaylor> I do not believe everyone can retrigger
19:25:22 <mtaylor> BUT
19:25:33 <mtaylor> I betcha we can figure something out there
19:25:44 <dolphm> i'm just thinking, as this becomes a popular/necessary workflow, if 24 hrs isn't sufficient, I'd like devs to be able to get a failed vm on their own, with minimal effort
19:25:47 <vishy> jeblair: have a branch that works with trunk fyi
19:25:59 <jeblair> vishy: awesome
19:26:03 * mtaylor makes out with vishy
19:26:17 <jsavak> ...
19:26:26 * dolphm hides
19:26:28 <vishy> had to disable one test because python-novaclient is out of date
19:27:01 <mtaylor> do we have a date for moving python-novaclient over yet?
19:27:09 <vishy> we should have devstack tagged diablo/stable and have a master running trunk in the next couple days
19:27:22 <jeblair> no, i proposed tomorrow morning to sandy, but haven't heard back
19:27:57 <mtaylor> I'm going to try to get the git-based pip-requires up and going for all the projects this week too, and finish moving the other projects to using pip-based unittest builders
19:28:07 <mtaylor> so we should be consistent across the board real soon now
19:28:22 <jeblair> vishy: i emailed jesse about moving devstack to gerrit; we need to do that soon too, before we can start using it to gate
19:29:15 <jeblair> mtaylor: yay!
19:32:34 <mtaylor> so I think that's where we're at with just about everything
19:33:03 <mtaylor> #topic anything else
19:34:06 <notmyname> I have a gerrit question
19:34:15 <notmyname> (not sure if this is the right forum)
19:34:21 <mtaylor> go for it!
19:34:39 <notmyname> when is -1 vs -2 appropriate?
19:35:07 <jeblair> -2 functionally blocks it from being merged
19:35:11 <notmyname> forever?
19:35:19 <mtaylor> until the -2 is lifted/changed
19:35:33 <notmyname> by another review from that same person?
19:35:35 <mtaylor> in fact - -2 sticks around even after a new change is pushed
19:35:38 <mtaylor> yes
19:35:43 <notmyname> ok
19:35:55 <mtaylor> so -2 is throwing your weight down and saying NO
19:36:25 <notmyname> but, "this is mostly ok but fix A,B,and C" should probably be -1, then?
19:36:48 <mtaylor> probably
19:36:57 <jeblair> i think so, and your -1 will be automatically removed when there is a new patchset.
19:37:06 <notmyname> and -2 is "no, I don't think adding the ability to make toast is a good feature to ever add to nova"
19:37:54 <jeblair> notmyname: you might consider using it also as a way to ensure that you review a subsequent change
19:38:03 <notmyname> ok. makes sense. thanks
19:38:12 <jeblair> as in: "I want you to fix this, and make sure that it's fixed to my satisfaction before it goes in"
19:38:15 <mtaylor> we could consider suggesting a policy around -2 similar to +2 ...
19:38:42 <mtaylor> like, people vote in general, and a core member +2's or -2's based on the general consensus of the voting?
19:39:03 <mtaylor> but yeah - we can also just try playing with it for a while and seeing how we like it
19:39:09 <jeblair> mtaylor: i want core people to all be able to +2 eventually
19:39:21 <jeblair> ie, we should support multiple +2 votes
19:39:32 <jeblair> once we have the gerrit trigger plugin updated
19:39:35 <mtaylor> jeblair: yes.
19:39:53 <jeblair> so i don't want to build too much policy around what is currently a technical limitation :/
19:40:06 <mtaylor> ++
19:40:07 <jeblair> or if we do, understand where we're going in the future
19:40:55 <vishy> I also have a gerrit question / feature request
19:41:37 <jeblair> vishy: go for it
19:41:45 <vishy> i find it quite annoying that repushing covers up previous reviews, especially when the push is just a rebase
19:42:18 <mtaylor> yes. I do too
19:42:20 <vishy> i don't have a solution, but it would be nice to be able to say, just a rebase when merging
19:42:22 <jeblair> right, the trivial rebase problem...
19:42:29 <mtaylor> we have a todo list item on that
19:42:29 <vishy> and it keeps the previous +1s
19:42:52 <jeblair> https://bugs.launchpad.net/openstack-ci/+bug/881184
19:42:54 <uvirtbot> Launchpad bug 881184 in openstack-ci "consider trivial rebase hook for gerrit" [Wishlist,Confirmed]
19:42:54 <mtaylor> #action mtaylor translate all of the gerrit/jenkins todo list items into bugs/blueprints
19:43:08 <jeblair> wow, mtaylor, that was quick! :)
19:43:12 <notmyname> (all the "trivial" rebasing also makes the master branch look ugly)
19:43:40 <jeblair> so what that hook does is:
19:43:42 <jeblair> "This hook is designed to detect when a patchset uploaded to Gerrit has the same git patch-id as the previous patchset. It then reapplies reviews onto the new patchset using the Gerrit SuExec and Approve commands."
19:43:52 <mtaylor> there's also been talk about a UI patch to make some of the review data show more prominiently by default, rather than needing to expand a section
19:44:01 <vishy> notmyname: I disagree with your comment about toast
19:44:01 * vishy has been waiting for the toast patch for a while now
19:44:03 <jeblair> which i believe would solve vishy's problem.
19:44:33 <jeblair> obviously it should be easy to install, we just need to make sure it does what it advertises
19:44:42 <jeblair> and works with our workflow
19:45:24 <notmyname> vishy: well that's where the -1 or -2 levels come in ;-)
19:46:47 <ttx> Hmmm. Toast...
19:47:32 * mtaylor is hungry now
19:48:29 <notmyname> mtaylor: turns out that brazillian food in brazil is exactly the same as brazillian food in the US. I suggest you go get some churisca (sp? whatever.) tonight
19:48:48 <mtaylor> notmyname: yes. it's quite tasty isn't it?
19:48:56 <jeblair> meeting over? :)
19:49:11 <mtaylor> I think so
19:49:14 <mtaylor> #endmeeting