19:14:56 <LinuxJedi> #startmeeting
19:14:57 <openstack> Meeting started Tue Apr  3 19:14:56 2012 UTC.  The chair is LinuxJedi. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:14:58 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic.
19:15:05 <LinuxJedi> #topic CI meeting
19:15:29 <LinuxJedi> mtaylor: meeting started, you have the chair ;)
19:15:37 <mtaylor> yay!
19:15:41 <mtaylor> how's everybody doing?
19:15:49 <LinuxJedi> good thanks
19:15:53 <mtaylor> swet
19:15:53 <jeblair> super awesome
19:16:19 <mtaylor> so - let's start with LinuxJedi - wanna talk about jenkins job templates and api stuffs?
19:16:31 <LinuxJedi> sure...
19:17:08 <LinuxJedi> so, I can't remember if I covered this last week but now in puppet we have a new module which will automatically spray Jenkins with our default jobs for any given project
19:17:40 <LinuxJedi> there is one project on Openstack now using this and mtaylor has since fixed a few minor bugs and made some improvements
19:18:03 <mtaylor> it's the bees knees
19:18:12 <LinuxJedi> the current problem is any new project or change to any of the building blocks that makes the jobs means Jenkins is restarted
19:18:37 <LinuxJedi> so I'm working on getting it to fire the API call to reload Jenkins instead (I have a prototype of this working)
19:19:08 <jeblair> once that's taken care of, i think we can roll it out to more openstack projects; but we're going to have to do that carefully since they aren't all standardized yet.
19:19:22 <mtaylor> jeblair: good way to ensure they're standardized :)
19:19:27 <LinuxJedi> next steps are: 1. load Jenkins jobs in using the API instead of spraying files. 2. support builders other than oneiric.  3. support jobs other than python
19:19:41 <mtaylor> ++
19:19:58 <mtaylor> LinuxJedi: add to todo list, an ensure param for the job class
19:20:10 <LinuxJedi> mtaylor: job class?
19:20:14 <mtaylor> LinuxJedi: that can toggle the diabled button, and can also ensure gone
19:20:28 <LinuxJedi> mtaylor: ++
19:20:31 <mtaylor> LinuxJedi: so that we can ensure => present , ensure => diabled, ensure => absent
19:20:47 * mtaylor realized we need this with the gate-* rename patch
19:20:52 <LinuxJedi> mtaylor: that will be easier once I switch to Ruby instead of Puppet scripting
19:21:01 <mtaylor> ++
19:21:10 * LinuxJedi is 90% there with that
19:21:15 <mtaylor> awesome
19:21:31 <mtaylor> will you remind me (and the viewing public) what the ruby scripting was in support of?
19:22:14 <LinuxJedi> oh, so puppet being non-procedural doesn't support things like foreach...
19:22:23 <LinuxJedi> there is a way around this using defined types and arrays
19:22:34 <LinuxJedi> but it only works for one level of "foreach"
19:22:42 <LinuxJedi> after that arrays get flattened
19:22:57 <LinuxJedi> plus we can do API stuff
19:23:19 <LinuxJedi> we need that so that we can have oneiric and natty jobs for certain projects
19:23:41 <jeblair> LinuxJedi: is it worth doing this in puppet/ruby, or would everyone be happier if we had puppet just call out to a python program that did the work? :)
19:23:56 <mtaylor> awesome. makes sense.
19:24:11 <LinuxJedi> jeblair: I would be a lot happier if I didn't get my hands dirty with Ruby here ;)
19:24:33 <mtaylor> don't we have to do it in ruby though because this is about job template expansion?
19:25:01 <LinuxJedi> mtaylor: that depends on whether you want puppet to own the XML
19:25:04 <mtaylor> so you can say "class job { 'python-glance-client': target => [oneiric,natty] }
19:25:15 <mtaylor> and have target do something sensible
19:25:22 <LinuxJedi> mtaylor: yes, that is the plan
19:25:28 <mtaylor> (or something - I made that up of course)
19:25:56 <LinuxJedi> mtaylor: the bit that makes it harder is we would need an oneiric and natty job for some but not others (such as tarball)
19:26:11 <LinuxJedi> mtaylor: I have a way of expressing that too which works
19:27:10 <mtaylor> cool
19:28:16 <LinuxJedi> mtaylor: I think it will end up with jobs called 'gate-python-glance-client-oneiric-python27'
19:28:26 <LinuxJedi> and likewise for natty
19:28:39 <mtaylor> I think that's fine if that's not how I actually have to manage them
19:28:41 <mtaylor> :)
19:29:17 <LinuxJedi> mtaylor: if there was a switch in the job to automatically change between them (no idea how we would do that) it would solve a lot of problems
19:29:43 <LinuxJedi> mtaylor: or if a job could build on multiple simultaneously (again I have no idea how to do that)
19:29:51 <jeblair> there's matrix jobs
19:30:00 <LinuxJedi> matrix jobs?
19:30:27 <jeblair> https://jenkins.openstack.org/job/devstack-launch-vms/
19:30:28 <jeblair> there's one
19:30:57 <LinuxJedi> arse, ok I have been trying to solve a problem that doesn't exist then :)
19:31:04 <mtaylor> jeblair: didn't we not like how that reported back to gerrit though?
19:31:13 <jeblair> the variable is provider there, but build slave is also something that can be varied
19:31:21 <jeblair> mtaylor: i'm not sure we ever had a matrix job report to gerrit?
19:31:30 <mtaylor> jeblair: we tested it
19:31:50 <jeblair> mtaylor: okay, i don't recall that
19:31:52 <mtaylor> jeblair: it reports the url of the top level job, and then it's several clicks to the specific thing
19:32:09 <mtaylor> so you can't tell from gerrit whether it failed on natty or on oneiric
19:32:28 <mtaylor> (we were thinking about it for python26/python27 IIRC)
19:32:50 <mtaylor> and the thought was that we might want to have gerrit trigger grok matrix jobs differently or something
19:32:53 <jeblair> mtaylor: okay, i'll take your word for it.  that's certainly true for post-build triggers, seems reasonable it would be for matrix jobs too
19:33:33 <LinuxJedi> so stick with multiple jobs instead of matrix jobs?
19:33:44 <jeblair> we didn't have both 26/27 jobs until a few weeks ago, so it hasn't been a problem until now
19:33:47 <annegentle> do I want matrix jobs for doc jobs?
19:34:03 <mtaylor> annegentle: for the essex/diablo thing? or?
19:34:22 <annegentle> mtaylor: yep, for essex diablo "root" or /trunk
19:34:32 <jeblair> annegentle: i don't think so.  i haven't fully thought about your question, but my initial thought is we should treat it like the tarball jobs
19:34:35 <mtaylor> annegentle: I believe we can do that more naturally
19:34:50 <annegentle> jeblair: oh, interesting, yeah a lump of docs
19:34:56 <jeblair> which know to build tarballs differently for different branches
19:35:33 <jeblair> LinuxJedi: so it sounds like matrix jobs would solve your problem, but not in a gerrit-friendly way
19:36:01 <LinuxJedi> jeblair: sure, so, should I not use it?
19:36:34 <jeblair> i think not -- gerrit friendliness is important
19:36:39 <LinuxJedi> jeblair: my personal opinion would be it would be a lot cleaner (and easier to implement) at the cost that we may have to patch gerrit at some point
19:36:45 <LinuxJedi> damn :)
19:36:56 <jeblair> it's not gerrit that needs to be patched as much as the gerrit trigerr plugin
19:37:01 <jeblair> that's possibly easier
19:37:02 <mtaylor> yeah - I think it would be easier for us, but less good experience for the devs
19:37:25 <jeblair> if we did that, it might be an opportunity to solve another problem i'd like to solve
19:37:34 <LinuxJedi> ko, I will finish implementing the way I was doing it for now then
19:37:57 <jeblair> i actually want only one job triggered in jenkins for a change, and i want it to trigger other jobs, but i want the report back to look nice.
19:37:58 <mtaylor> cool. - but let's definitely keep our eyes on fixing the trigger plugin at some point
19:38:16 <LinuxJedi> mtaylor: in other news, I may bump finishing the meetbot replacement up my todo list ;)
19:38:17 <mtaylor> jeblair: oh my, you're getting fancy
19:38:46 <jeblair> i think those are _nearly_ the same issue as far as the plugin goes -- matrix jobs and job-triggered jobs.  they both produce composite output
19:39:04 <jeblair> mtaylor: it's the only way i can think of to ensure the correct ordering of testing and merging of changes.
19:39:33 <jeblair> we don't do what we intend to at the moment, it just happens to work like that most of the time.  as we get more jobs that run longer, we'll get more changes merged out of order.
19:39:58 <mtaylor> jeblair: it also would allow us to test merge before we bother testing other things
19:40:05 <jeblair> yes
19:40:20 <LinuxJedi> I like that idea :)
19:40:36 <mtaylor> should that be your next major dev project?
19:40:54 <jeblair> i think it would be a good one
19:41:08 <mtaylor> k. works for me
19:41:38 <LinuxJedi> other things I have done this week: meetbot broke and I restarted it ;)
19:41:39 <jeblair> LinuxJedi: i'm not sure this will get done in time to be useful for your current problem though.
19:41:59 <LinuxJedi> jeblair: it is OK, I can always switch the way it is done quite easily afterwards
19:42:06 <jeblair> cool
19:42:55 <jeblair> meetbot still needs to be puppetized, right?
19:43:25 <LinuxJedi> jeblair: yep, I almost finished that in Seattle, but lots of more important things keep coming up
19:43:55 <LinuxJedi> jeblair: I have a branch in which it "works" but there needs to be a few finishing touches IIRC
19:45:26 <LinuxJedi> mtaylor: if it is OK with you, once the Jenkins Jobs improvements are out of the way I'd like to finish meetbot
19:45:47 <LinuxJedi> mtaylor: one main advantage is puppet will make sure it is running on every run ;)
19:46:13 <jeblair> that seems useful and timely.  :)
19:46:49 <LinuxJedi> ++
19:47:18 <annegentle> so for Thursday, release day, should I just update the doc jobs I have or try to get help with treating them like tar ball jobs?
19:47:40 <annegentle> do any of you have the bandwidth for helping me this week?
19:48:02 <LinuxJedi> mtaylor: oh, whilst I remember, this says Ubuntu SSO is any day now: http://meta.askubuntu.com/questions/2837/moving-to-ubuntu-single-sign-on
19:48:04 <jeblair> annegentle: i believe i can help
19:49:00 <annegentle> jeblair: thank you thank  you
19:49:21 <annegentle> I'm sorry if I'm interrupting your meeting flow :)
19:49:45 <jeblair> annegentle: well, the chair doesn't seem to object.  :)
19:49:52 <LinuxJedi> mtaylor: so a job possibly for one of the new guys is to investigate moving over to Ubuntu SSO?
19:51:35 <jeblair> so, um, quick update from me:
19:51:42 <jeblair> i typed this up while LinuxJedi was typing. :)
19:51:42 <jeblair> new version of devstack gate that uses nodes from multiple providers is in production
19:51:42 <jeblair> we're much less likely to see node exhaustion
19:51:42 <jeblair> though at least one provider seems to be producing servers that run so slow we time out
19:51:42 <jeblair> more work is needed to document and collect stats on that
19:51:43 <jeblair> the process has, so far, identified a number of operation problems with the providers, and i've been providing feedback
19:51:59 <jeblair> and
19:52:01 <jeblair> i opened a pull request for the ref-updated branch of the gerrit-trigger-plugin
19:52:01 <jeblair> it still has outstanding issues that need to be addressed before it's likely to be merged
19:52:01 <jeblair> but at least the request is for the code we're using
19:52:01 <jeblair> eventually we'll probably need to do some more work to get it merged
19:52:06 <jeblair> and finally
19:52:06 <jeblair> i'm planning on moving gerrit to a new (oneiric) server, but am waiting on a quota increase in the cloud servers account
19:52:43 <mtaylor> awesome.
19:53:10 <mtaylor> is there any benefit in installing puppetmaster before you install the new gerrit server?
19:53:23 <LinuxJedi> ttx: that would explain why your devstack timed out earlier ^
19:53:48 <jeblair> LinuxJedi: thoughts on mtaylor's question?
19:54:16 <LinuxJedi> mtaylor, jeblair: it is not hard to switch between the two so I'm happy either way
19:54:30 <LinuxJedi> mtaylor: any news on devananda's gerrit tests?
19:54:38 <mtaylor> devananda: ?
19:55:05 <devananda> LinuxJedi: i improved the GroupAdmin page load time by stopping the extra SET & SHOW commands, but would like it to reuse connections, too ...
19:55:20 <devananda> it's opening 3k connections to mysql for that one page
19:55:35 <LinuxJedi> devananda: that is an awesome start :)
19:55:40 <devananda> i'm testing latest google branch of 2.3 now, then digging into the code to see if i can fix
19:56:03 <jeblair> we mostly need to be off of natty soon; i don't mind doing further upgrades as they're ready.
19:56:24 <LinuxJedi> devananda: if you can make it use connection pooling like pretty much anything else written in Java I think it would be perfect ;)
19:56:51 <LinuxJedi> jeblair: ++ natty's puppet doesn't support a lot of what we are trying to do nowadays
19:59:23 <mtaylor> when it becomes useful - I have a branch with our patches ported in to the 2.3 tree
19:59:46 <LinuxJedi> mtaylor: I suspect it will be useful to devananda now ;)
20:00:03 <mtaylor> BUT - I was thinking - perhaps we should import stock 2.3 into a branch in our gerrit, then propose each of our cherry-pick forward ports as a change?
20:00:16 <mtaylor> I had to edit two of jeblair's patches in doing the rebase
20:01:02 <mtaylor> which I _think_ I got right, but might not have
20:01:11 <LinuxJedi> mtaylor: should we overrun the meeting due to the false start?  I believe PPB should be now
20:01:25 <jeblair> we should really upstream them.  :)
20:01:29 <mtaylor> anybody know - is the PPB meeting?
20:01:33 <devananda> mtaylor: yea that may be useful soon. I'll poke you when I resolve the reconnect issue (would rather have a clean codebase to work on until then)
20:01:36 <mtaylor> jeblair: well, yes
20:01:44 <mtaylor> devananda: agree
20:02:06 <mtaylor> jeblair: just trying to figure out how to model carrying local changes with tracking upstream in gerrit :)
20:02:54 <jeblair> i'd like to be able to continue to merge
20:04:26 <notmyname> ppb meeting today?
20:04:29 <mtaylor> I've got my next thing ... I'm going to need to call this one a wrap from me
20:04:45 <LinuxJedi> mtaylor: cool, shall I endmeeting?
20:05:18 <LinuxJedi> anyone have anything before I do?
20:05:25 <jeblair> nope
20:05:27 <mtaylor> nope
20:05:37 <LinuxJedi> #endmeeting