19:00:52 <mtaylor> #startmeeting
19:00:53 <openstack> Meeting started Tue May 22 19:00:52 2012 UTC.  The chair is mtaylor. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:00:54 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:07 <heckj> notmyname: okay - I'll kick you email and we'll grab some time. Sorry for slow/lack of responses, have been a bit under the gun lately
19:01:25 * mtaylor punches heckj for his laziness!
19:01:31 <notmyname> heckj: no worries. no rush
19:01:43 <mtaylor> that's the kind of discipline we hand out during the CI meeting ;)
19:01:45 * heckj throws pies back at mtaylor
19:01:57 * mtaylor goes to take a quick shower...
19:02:02 <jeblair> that makes it sound like used pied.
19:02:04 <jeblair> pies.
19:02:09 <dolphm> these meeting transitions can be rough
19:02:14 <heckj> heh
19:02:18 <jeblair> and messy
19:02:21 <mtaylor> dolphm: make sure you always wear body armor
19:02:32 <clarkb> and give mtaylor lots of room
19:02:35 <dolphm> lol
19:02:42 * mtaylor begins krumping
19:03:05 * dolphm switches to heavy armor
19:03:09 * LinuxJedi has no idea what krumping is but it sounds dirty
19:03:17 * mtaylor waits while people google krumping so they realize is a verb mtaylor could never pull off
19:03:36 <mtaylor> and with that ... who'd like to talk about CI stuff?
19:03:46 * clarkb is here
19:03:51 <LinuxJedi> *crickets*
19:04:01 <mtaylor> yeah. that's what I thought
19:04:13 <mtaylor> LinuxJedi: why don't you start us off ...
19:04:19 <mtaylor> #topic stackforge updates
19:04:27 <LinuxJedi> so, Stackforge....
19:04:56 <LinuxJedi> first up mail...
19:05:22 <LinuxJedi> if you were following last week you will know I spent some effort moving the Gerrit server to a new node
19:05:32 <mtaylor> yup. fun times
19:05:33 <LinuxJedi> this is because outgoing mail was getting blocked
19:05:47 <LinuxJedi> after I did that we found it was happening again
19:06:11 <LinuxJedi> it turns out that HP Cloud has rate limiting on port 25.  This is to stop people using it as spam bots
19:06:18 <LinuxJedi> which is awesome, except for us
19:06:31 <LinuxJedi> we are working with teams to fix this for us, but it isn't going to be overnight
19:06:46 <LinuxJedi> so in the mean time Stackforge probably won't send you notifications for code review
19:07:18 <LinuxJedi> In other news we managed to add Heat to Stackforge and as of about 6 minutes ago it was green lighting on Stackforge Jenkins too
19:07:35 <LinuxJedi> mtaylor: Jenkins Jobs next?
19:07:37 <mtaylor> woohoo!
19:07:50 <mtaylor> well, I was just going to mention for the folks following along at home...
19:08:00 <mtaylor> LinuxJedi: what is it that's different about heat there LinuxJedi ?
19:08:15 <LinuxJedi> ah!  I'm glad you asked (almost as if we scripted it!)
19:08:20 * mtaylor bows
19:08:48 * jeblair is still working on his script
19:08:49 <LinuxJedi> well, heat doesn't use a stackforge/project URL in github.  It is using the repo it was born with (heat-api/heat)
19:09:07 <LinuxJedi> which caused a few minor issues which I have just fixed
19:09:17 <LinuxJedi> so in theory we could do this with other projects in the future
19:09:36 <LinuxJedi> (other Stackforge projects at least, Openstack has the CLA complication stopping us)
19:09:42 <clarkb> so no changes to github?
19:10:01 <soren> What the heck is heat?
19:10:03 <mtaylor> the bug links in gerrit are, of course, not correct... so we may want to eventually get that per-project bug-tracker feature coded up
19:10:03 <LinuxJedi> clarkb: the only change to github is the Stackforge org doesn't own the repo
19:10:08 * soren is late to the game
19:10:13 <clarkb> LinuxJedi: I see
19:10:21 <jeblair> I don't believe we should do that for openstack projects.  we chose not to for good reason.
19:10:22 <LinuxJedi> soren: it is an api: https://github.com/heat-api/heat
19:10:25 <mtaylor> clarkb: they had to add the gerrit user to their repo with push access
19:10:29 <mtaylor> I agree with jeblair
19:10:38 <mtaylor> I think it's more relevant for stackforge projets
19:10:40 <mtaylor> projects
19:10:54 <LinuxJedi> ++
19:11:10 * LinuxJedi was going to explain that bit better but didn't finish the script in time
19:11:21 <mtaylor> where they may not as much be using launchpad, etc.
19:11:21 <LinuxJedi> soren: they have #heat IRC channel if you want to chat to them
19:11:59 <mtaylor> the question of allowing any openid for stackforge also came up ... which also seems like it might be a more relevant choice, but isn't super high prior right now
19:12:15 <LinuxJedi> mtaylor: oh yea, I had that down to bring up in this meeting
19:12:23 * mtaylor keeps stepping on LinuxJedi's toes
19:12:32 <LinuxJedi> hehe, no its ok, I should read my notes :)
19:12:56 <LinuxJedi> so, what mtaylor said.  I don't know if we got to a consensus on this yet
19:13:37 <LinuxJedi> and I haven't figured it if there will be technical problems with it
19:13:53 <LinuxJedi> and it has been low priority for me to figure it out to be honest
19:14:08 <mtaylor> yup. I think we might get space to think bout that in a few weeks
19:14:25 <mtaylor> any questions on stackforge stuff?
19:14:34 <LinuxJedi> oh, one minor other thing with Stackforge, the sync and expire scripts work without manual intervention now
19:14:46 <mtaylor> oh good
19:15:25 <mtaylor> #topic job filler
19:15:31 <LinuxJedi> my notes are completely out of order for this meeting so I all probably miss something somewhere
19:15:58 * mtaylor should probably do the whole agenda thing again now that we have, you know, people
19:16:12 <LinuxJedi> ok, Jenkins Job Filler is a magic pony (or Python script as us devs call it) which given a bit of fairy dust (YAML) will spray Jenkins with jobs for projects
19:16:49 <LinuxJedi> it now actually works properly and is being rolled out onto real Openstack projects
19:17:10 <LinuxJedi> I've been assisting mtaylor with that (fixing what he breaks)
19:17:14 <mtaylor> cinder, python-cinderclient and python-swiftclient are all using it at the moment ...
19:17:23 * mtaylor can break things good
19:17:39 <LinuxJedi> and also making sure it is working on Stackforge too.  99% of Stackforge uses this and the 1% can probably be sorted tomorrow
19:18:25 <mtaylor> woot
19:18:41 <LinuxJedi> and the custom Jenkins library has been ditched now since everyone but me can get the stock library working ;)
19:18:41 * mtaylor is excited... it's getting us much closer to our consistent jobs goal
19:18:47 <mtaylor> hehehe
19:18:54 <clarkb> and the yaml fairy dust can be found in openstack-ci-repo?
19:19:04 <mtaylor> openstack-ci-puppet
19:19:04 <clarkb> er
19:19:08 <clarkb> ya that
19:19:12 <LinuxJedi> clarkb: absolutely, in the modules/jenkins_jobs/files dir
19:19:27 <mtaylor> for instance: https://github.com/openstack/openstack-ci-puppet/tree/master/modules/jenkins_jobs/files/projects/openstack
19:20:06 <LinuxJedi> it can do templating for projects that fit a box or it also supports oddities like Heat
19:20:23 <LinuxJedi> at some point it will support doing both at the same time for a project
19:21:10 <LinuxJedi> mtaylor: only other thing I got is on the Openstack Jenkins slave upgrades
19:21:36 <mtaylor> so, we're holding off on rolling it out for the rest of the openstack projects until we get something sorted with our pypi mirror (I need to test how bad the external links thing is with nova)
19:21:53 <mtaylor> LinuxJedi: let's come back to slave upgrades
19:21:58 <mtaylor> #topic pypi mirror
19:22:01 <mtaylor> Shrews: you around?
19:22:03 <LinuxJedi> oh, awesome.
19:22:06 <LinuxJedi> mtaylor: sure thing
19:22:06 <Shrews> no!
19:22:16 <LinuxJedi> he is asquare
19:22:35 <Shrews> so, it turns out the pypi mirror software is not doing some things that we expect it to do, and doing some things that we don't expect it to do
19:22:44 <mtaylor> I love software
19:23:02 <Shrews> e.g.) doesn't appear to follow links, and only pulls the latest versions of packages
19:23:10 <Shrews> so that needs sorting out
19:23:42 <mtaylor> yah
19:24:00 <Shrews> but it can still be used with the proper pip.conf
19:24:22 <Shrews> installation via distutils/easy_install will likely be fail
19:25:01 <Shrews> end communication
19:25:15 <mtaylor> cool. thank ya
19:25:22 <mtaylor> while we're bugging you ...
19:25:28 <mtaylor> #topic work in progress
19:25:53 <Shrews> We will have a Work In Progress change state in Gerrit "real soon now"
19:26:07 <mtaylor> that's so exciting
19:26:12 <jeblair> sounds like it's a work in progress
19:26:13 <Shrews> a change in WIP will not appear in any reviewer's Review Requests
19:26:39 <Shrews> we can currently set it with "gerrit review", but can't do it through the UI quite yet
19:26:50 <Shrews> damn java
19:27:39 <mtaylor> Shrews: but gwt makes things easier!
19:27:47 <Shrews> so "gerrit review --workinprogress <id>"
19:28:01 <Shrews> mtaylor: uh huh, sure
19:28:22 <mtaylor> excellent. when we have that, I want to have our jenkins do --workinprogress when it sets a -1 VRFY review...
19:28:27 <clarkb> can you push to a WIP ref as well? so that it can start as a work in progress?
19:28:35 <LinuxJedi> mtaylor: we should make the expire script do that once it is done
19:28:46 <Shrews> clarkb: no
19:28:51 <Shrews> oh, and also
19:29:05 <Shrews> it will automatically go back to Review in Progress on a new patchset push
19:29:16 <jeblair> it sounds like we may need an extra acl, if we want some users to be able to WIP something, but not everyone
19:29:27 <jeblair> so we can give it to jenkins and project-core
19:29:31 <Shrews> jeblair: right now, if you can abandon a change, you can WIP it
19:29:40 <Shrews> "WIP it good!"
19:29:43 <LinuxJedi> jeblair: ++
19:29:48 <jeblair> Shrews: i believe that's sysadmin+owner
19:30:25 * Shrews forgets offhand, but that sounds right. project owner too
19:30:33 <mtaylor> yeah, I think jeblair is right - one of the problems with abandon has been that more people can't use it. if we can grant perms to core team and to jenkins, that would be stellar
19:31:04 <Shrews> mtaylor: i'll figure that out… after the button fiasco
19:31:10 <mtaylor> Shrews: ++
19:31:31 * LinuxJedi will buy Shrews a mug of Java when he finishes
19:31:45 <mtaylor> hahaha
19:31:47 <Shrews> die. k thx
19:31:57 <LinuxJedi> :)
19:32:27 <mtaylor> alrighty...
19:32:38 <mtaylor> #topic better dashboard
19:32:46 <mtaylor> clarkb: how's that?
19:32:58 <clarkb> I think the first basic version is working
19:33:38 <mtaylor> sweet. is it worth popping into review status and landing in its current state?
19:33:55 <clarkb> basically it gives you three lists of patchsets. Changes you have submitted, changes you have starred + changes you are a reviewer on less patches whose latest patchset you have reviewed
19:34:07 <clarkb> finally you get a list of the patchsets for open changes that you have recently reviewed
19:34:27 <clarkb> if you think that functionality is worthwhile then sure
19:34:52 <mtaylor> honestly, even just the list of things I should be reviewing without the things I've reviewed is a great step forward
19:35:07 <clarkb> There are two missing pieces (potentaily just one though). Do we want to include watched changes in the second list?
19:35:24 <mtaylor> clarkb, Shrews: once WIP is landed, we might want to add "list of my WIP patches" somewhere (I know I always had trouble finding my WIP branches in lp)
19:35:26 <clarkb> the issue I have with that is projects are watched not individual changes so I htink that could get noisy
19:35:57 <mtaylor> clarkb: well, I think watched projects is currently used more than individual review requests
19:36:14 <mtaylor> clarkb: so I'd vote for yes ... anybody else?
19:36:46 <LinuxJedi> ++
19:36:57 <jeblair> so....
19:36:59 <jeblair> "changes you are a reviewer on less patches whose latest patchset you have reviewed"
19:37:06 <jeblair> that seems like the new thing here, yeah?
19:37:11 <clarkb> yes
19:37:38 <jeblair> but since we don't normally add people as reviewers, instead, relying on watched changes
19:37:46 <jeblair> that's not exactly what we need right now, right?
19:38:00 <clarkb> no, the watched changes should be added
19:38:19 <clarkb> but it will help as you become a reviwer when you submit comments
19:38:20 <jeblair> ok, i see you said that up there
19:38:44 <jeblair> so i think if that's added, i'm in support of giving it a test drive
19:39:13 <clarkb> the other missing piece is sorting the resulting list like ttx proposed
19:39:20 <clarkb> so that the most important changes show up first
19:39:47 <mtaylor> yeah, I definitely think we can land it before we get ttx sorting. that seems like a v2 feature
19:39:49 <jeblair> clarkb: if the "patches whose latest patchset you have reviewed" can be generalized and made into an item that can be generally used in gerrit queries, that could be really useful
19:40:14 <clarkb> jeblair: so that you could do status:open is:reviewed ?
19:40:21 <clarkb> or something like that
19:40:25 <jeblair> yeah, you can already do a lot of things like:
19:40:26 <jeblair> https://review.openstack.org/#/q/reviewer:corvus,n,z
19:41:16 <jeblair> so your last pane could be something like "reviewer:corvus and not latestreviewed" or something.  i forget the grammar.  :)
19:41:17 <clarkb> so it sounds like leaving it as a draft is probably good for now. I will work on adding watched changes and look into the query stuff
19:41:33 <mtaylor> cool
19:42:04 <mtaylor> #topic new devstack-gate madness
19:42:13 <mtaylor> jeblair: so you've been busy ...
19:42:27 <jeblair> i'm going to combine that and zuul, because that's how i wrote my script
19:42:32 <jeblair> i have a wall of text.  i will try to paste it in such a way that i'm not kickbanned.
19:42:32 <mtaylor> great
19:42:42 <jeblair> i've been working on speculative execution of jenkins jobs
19:42:43 <jeblair> the idea is to be able to run gating jobs in parallel, but still have them merged in series.
19:42:43 <jeblair> so they are still tested in a strict order
19:42:43 <jeblair> i completed a simple proof of concept for most of the ideas using the gerrit trigger plugin
19:42:48 <jeblair> but in writing that, i realized that some of the things we wanted to do might be a little out
19:42:48 <jeblair> of scope for the gerrit trigger plugin, and further, some of the ideas are quite difficult to
19:42:48 <jeblair> implement in the trigger plugin, because its data structures aren't exactly set up in a way
19:42:48 <jeblair> that facilitates this.
19:43:17 <jeblair> .
19:43:18 <jeblair> so i thought it would be easier to implement this idea if the control for running tests and
19:43:18 <jeblair> merging changes was moved outside of gerrit trigger plugin, and into a new system that's
19:43:18 <jeblair> easier for us to maintain.  i'm working on a python daemon that will monitor gerrit, trigger
19:43:18 <jeblair> jenkins jobs, report back, and merge changes as appropriate.  it's actually going to give us
19:43:29 <jeblair> a lot more flexibility in the gating system, in that it will be able to make decisions about
19:43:29 <jeblair> whether to merge changes based on input not available to the gerrit trigger plugin, for example,
19:43:29 <jeblair> whether another change has been merged.  it also will have fabulous logging.
19:43:29 <jeblair> i've made very good progress on this system.  it's working name is "zuul".
19:43:36 <jeblair> .
19:43:47 <jeblair> any questions about that?
19:43:58 * mtaylor supports our new zuul overlords
19:44:33 <LinuxJedi> jeblair: Ghostbusters?
19:44:36 <jeblair> t
19:44:51 <Shrews> i'm against the fabulous logging (/me prefers just-ok-logging), but ++ !!!
19:45:21 <jeblair> Shrews: it'll be configurable.  i'll look into adding an option to randomly drop important entries.
19:45:32 <Shrews> jeblair: awesomeness
19:45:46 <jeblair> okay, moving on to devstack-gate madness:
19:45:52 <jeblair> since the end goal is that we should be able to fire LOTS of tests in parallel in jenkins
19:45:52 <jeblair> (if we turned this on today, assuming enough nodes, we could see upwards of 50 tests running in parallel)
19:45:52 <jeblair> there's a problem with the way devstack-gate has been running.  if we allowed it to run
19:45:52 <jeblair> in parallel, it would run as many jobs as there were executors for, and we might quickly run
19:45:52 <jeblair> out of nodes (which would fail tests).
19:46:02 <jeblair> .
19:46:03 <jeblair> so i took a page out of the jclouds playbook
19:46:03 <jeblair> (which is working toward being able to replace devstack-gate, but isn't there yet), and made
19:46:03 <jeblair> devstack-gate spin up machines built as jenkins slaves instead of dumb nodes.  when they
19:46:03 <jeblair> come online, devstack-gate adds them to jenkins as first class slaves (with a 'devstack-oneiric'
19:46:10 <jeblair> node tag).  the devstack integration test job is now configured only to run on 'devstack-oneiric'
19:46:11 <jeblair> nodes.  when the job starts, the node is disabled so that no further jobs run on it.
19:46:11 <jeblair> the test itself now runs on the slave, rather than ssh-ing to the devstack node.
19:46:11 <jeblair> and when it is complete, the node is removed from jenkins and deleted.
19:46:23 <jeblair> .
19:46:25 <jeblair> the upshot of that is that in the near term, we should never get another error due
19:46:25 <jeblair> to being out of jenkins nodes, because jobs that need them will just queue up.
19:46:25 <jeblair> and in the long term, we'll be able to trigger lots of runs of the integration tests
19:46:25 <jeblair> and execute them in parallel as fast as we can spin up nodes.
19:46:40 <jeblair> <end>
19:47:40 <mtaylor> the first thing about zuul was cool
19:47:44 <jeblair> oh, and i should add, the switch from gerrit trigger plugin to zuul will have to happen all at once, so having, the jenkins job filler in place will make that much nicer.
19:47:46 <mtaylor> the second thing about jenkins slaves is really great
19:48:16 <jeblair> yes, in a couple weeks, all these things are going to come together, and we will have a very shiny machine.  :)
19:48:57 <mtaylor> any more questions for jeblair?
19:49:06 <LinuxJedi> ever wonder by creating fantastic machines if we are going to make ourselves redundant? ;)
19:49:28 <mtaylor> LinuxJedi: I hope so
19:49:39 <LinuxJedi> yay!  rise of the machines!
19:49:44 <jeblair> i don't want my job to be "merge changes".  i'm happy to have a machine do that.  :)
19:49:56 <clarkb> skynet will run on jenkins?
19:49:59 <LinuxJedi> true
19:50:14 <mtaylor> related to that ...
19:50:16 <LinuxJedi> clarkb: Nova is Skynet, didn't you get the memo?
19:50:19 <mtaylor> #topic jenkins slaves
19:50:52 <mtaylor> LinuxJedi upgraded the slaves, yeah?
19:51:09 <LinuxJedi> oh, so yes...
19:51:29 <LinuxJedi> one of the Jenkins slaves (oneiric5) started failing jobs overnight
19:51:33 <LinuxJedi> it was due to OOM
19:51:57 <LinuxJedi> I've upgrade oneiric 4,5,6 to have double the RAM.  The others I'll probably do tomorrow now
19:52:05 <LinuxJedi> since I'm in meetings until I sleep
19:52:44 <LinuxJedi> it shouldn't affect anyone but we will only have half the nodes for an hour tomorrow
19:53:07 <mtaylor> cool
19:53:49 <mtaylor> related to that ... I've been working on getting jenkins slaves spun up via jclouds
19:53:57 <LinuxJedi> yay! :)
19:55:56 <mtaylor> I'm working on getting a base image built that we can use to spin up nodes - because running puppet from scratch, well, takes some time
19:56:15 <mtaylor> we've also got a couple of patches that need to land in jclouds plugin - because right now puppet is creating the jenkins user...
19:56:24 <mtaylor> and the jclouds plugin is ALSO trying to do this :)
19:57:36 <mtaylor> anyhow - I've submitted some patches to try to take care of that, and hopefully we'll have that going soon
19:58:07 <mtaylor> additionally, the auto image creation code is in jclouds 1.5 beta... so as soon as that gets released, it should be available for the plugin
19:58:15 <mtaylor> I tihnk that's all i've got there
19:58:30 <mtaylor> and we're about out of time - anybody else got anything real quick?
19:58:31 <LinuxJedi> mtaylor: time to wrap up anyway, call in 1.5 minutes
19:58:44 <mtaylor> great.
19:58:46 <mtaylor> #endmeeting