19:01:46 <jeblair> #startmeeting ci
19:01:47 <openstack> Meeting started Tue Oct  2 19:01:46 2012 UTC.  The chair is jeblair. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:48 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:49 <openstack> The meeting name has been set to 'ci'
19:02:07 <jeblair> (looking up last meeting)
19:02:37 <jeblair> #topic items from last meeting
19:02:51 <jeblair> I believe this one is unchanged:
19:02:54 <jeblair> #action jbryce provide a test foundation server
19:03:06 <fungi> i have an update there
19:03:21 <jeblair> awesome: go for it
19:03:30 <fungi> i pinged him and reed via e-mail later in the week
19:03:55 <fungi> no word on progress yet, but reed expressed interest in taking over doing that if he gets time
19:03:59 <reed> I have talked to Todd today, he hopefully will hop online
19:04:09 <jeblair> #action reed provide a test foundation server
19:04:09 <fungi> reed: awesome!
19:04:22 <jeblair> #action todd provide a test foundation server
19:04:29 <jeblair> #action anyone provide a test foundation server
19:05:13 <jeblair> cool, if todd shows up, we can chat about that more, or any time in #-infra
19:05:29 <fungi> otherwise, we're still in a holding pattern. it works on review-dev and can be demo'd there
19:05:45 <jeblair> (and seems to work)
19:06:09 <jeblair> clarkb: i believe you proposed the translation session for the summit?
19:06:19 <clarkb> jeblair: I did. It has been preapproved too
19:06:25 <jeblair> groovy!
19:07:01 <jeblair> pabelanger added the puppet lint job, which rocks
19:07:44 <fungi> thanks pabelanger!
19:08:05 <clarkb> it tells us how bad we are
19:08:16 <jeblair> it is indicating we have quite a ways to go before we're lint-free
19:08:38 <fungi> the first time i saw it, i thought for a moment my browser was going to lock up trying to show the entire log ;)
19:09:06 <jeblair> #link http://idle.slashdot.org/story/12/10/01/1215213/us-agricultural-economists-say-bacon-shortage-is-hogwash
19:09:21 * fungi breathes a sigh of relief
19:09:22 <jeblair> ^ seems like we're off the hook there
19:09:22 <uvirtbot> jeblair: Error: "seems" is not a valid command.
19:09:53 <jeblair> and mordred did document use cases for bug 995604
19:09:53 <fungi> ^ seems uvirtbot likes this topic
19:09:54 <uvirtbot> Launchpad bug 995604 in openstack-ci "use symbolic-ref from gerrit to provide moving codename targets" [High,In progress] https://launchpad.net/bugs/995604
19:09:55 <uvirtbot> fungi: Error: "seems" is not a valid command.
19:10:05 <jeblair> #topic symbolic-ref
19:10:33 <jeblair> we chatted a bit earlier about symbolic refs.  i don't think it's a slam dunk for what we want to do
19:10:44 <fungi> jeblair: yes, i updated the bug report with details
19:11:06 <jeblair> but we have a lot of good information thanks to fungi's work
19:11:17 <fungi> as noted there, i did a proof-of-concept following his suggestion, but tags do indeed seem to fill the void in a better way
19:11:52 <jeblair> cool, my feeling is it's sort of a back-burner idea until we chat at the summit and figure out what will happen with branches next time around
19:12:02 <fungi> during or after the ods session(s) on branch organization and release process, we need to talk about how we automate the tag updates
19:12:25 <fungi> presumably that happens zuul-side?
19:12:41 <fungi> and not directly on gerrit's copies?
19:13:03 <jeblair> fungi: maybe it's a gerrit hook script on ref-updated?
19:13:05 <fungi> since we wouldn't want to tag a commit until it's merged successfully
19:13:21 <fungi> jeblair: that could work
19:13:55 <fungi> but i'll definitely try to catch any ods discussions around that as well
19:14:21 <jeblair> fungi: anyway, i also don't want to presume we will need this; it's possible (granted, unlikely) we could come up with a branch model that satisfies the use case.
19:14:31 <fungi> that would be ideal, i agree
19:15:20 <jeblair> #topic zuul
19:15:30 <jeblair> i gave a talk at the jenkins user conference about zuul
19:15:52 <jeblair> and plugged job-builder, gerrit, the openstack-ci-puppet repo, and our wiki/documentation too
19:16:15 <jeblair> some folks have popped into #-infra as a result of that, which is great
19:17:05 <jeblair> i'd love to get patches from outside of openstack for anything we do.  :)
19:17:06 <pabelanger> Thanks
19:17:11 <jeblair> oh, and git-review too.
19:17:25 <clarkb> I added a minor feature to zuul to have it report the zuul status link when it starts gate jobs
19:17:44 <clarkb> that started a short conversation on per change status pages
19:17:46 <fungi> saw that--clickyconvenient
19:17:51 <jeblair> zuul has some missing documentation and some things that make it hard to set up initially, so i have one patch in review to help with that, and another one coming
19:18:16 <clarkb> I think zuul may need to be restarted to pick up the change though
19:18:24 <jeblair> clarkb: yep
19:18:47 <jeblair> so per-change status pages are totally doable, it sort of depends on how much we want zuul to be a web app
19:19:29 <clarkb> maybe giving zuul a json api and allowing the webab to be decoupled is a better way to handle this problem?
19:19:51 <jeblair> clarkb: that's not a bad idea.
19:21:45 <jeblair> clarkb: yeah, i think i like that.  i'll cipher on that.  btw, anyone here love writing web apps?
19:22:39 * fungi thought the web was a passing fad for longer than he cares to admit, but will volunteer for anything
19:22:57 <jeblair> fungi: you still do, don't you? :)
19:23:12 <fungi> i'm just hoping beyond hope at this stage
19:23:31 <jeblair> anyone else have a #topic?
19:23:41 <fungi> gerrit whitespace change detection
19:23:48 <clarkb> oneiric -> precise upgrades
19:24:09 <jeblair> #topic gerrit whitespace change detection
19:24:10 <fungi> asps. very dangerous. clarkb goes first
19:24:13 <fungi> or me
19:24:15 <jeblair> whoops.  :)
19:24:48 <fungi> markmc was asking about a feature to have the trivial-rebase hook module no longer reapply reviews on whitespace changes between patchsets
19:25:06 <fungi> #link https://bugs.launchpad.net/openstack-ci/+bug/1057506
19:25:07 <uvirtbot> Launchpad bug 1057506 in openstack-ci "Gerrit trivial rebase detection ignores whitespace" [Medium,Fix released]
19:25:22 <fungi> merged yesterday: #link https://review.openstack.org/13775
19:25:42 <fungi> keep an eye out in case it does unexpected things, but i kicked the tires pretty thoroughly already
19:26:01 <jeblair> cool!
19:26:20 <fungi> your turn, clarkb
19:26:31 <jeblair> #topic asps; oneiric -> precise upgrades
19:26:57 <clarkb> so a few of the infrastructure hosts run oneiric and we have wanted to upgrade them to precise. notably the jenkins and gerrit hosts
19:27:20 <jeblair> mordred: ping
19:27:28 <clarkb> I have started cleaning up the Jenkins puppet modules to make it easy to stand up jenkins on a brand new host.
19:27:56 <clarkb> once https://review.openstack.org/#/c/13926/ merges I think we will be ready to start building a new jenkins-dev host
19:28:21 <clarkb> the motivation for building new hosts is to move to rackspace's new openstack cloud where we can have ipv6
19:28:51 <jeblair> the jenkins stuff looks great, btw.
19:29:01 <jeblair> i have a question about the gerrit modules though....
19:29:29 <jeblair> is our gerrit puppet config so complete that when we spin up a new gerrit host, will it immediately start replicating empty repositories to github?
19:29:47 <clarkb> o_O I hope not but I think that is a valid concern
19:30:06 <clarkb> the gerrit module has a lot more moving parts particularly for configuration
19:30:36 <clarkb> I only just started poking at gerrit today, definitely need to look at it pretty closely
19:30:45 <fungi> i avoided using it when i built my test gerrit vm because i was unsure how much of a loaded gun it was, after acasual glance
19:30:56 <jeblair> i guess it probably wouldn't actually have the empty repos in gerrit yet, right?
19:31:24 <jeblair> (even though it would probably have a working replication config, and possibly empty git repos in /var for the apache mirrors?)
19:31:26 <clarkb> jeblair: no, but it has the project lists and the replication config
19:32:56 <clarkb> a similar concern when we get to jenkins.o.o will be making sure that we don't have two masters fighting for the same sets of slaves
19:32:58 <jeblair> all things being equal, it would be swell if we did not replicate empty repos to github, so we should either confirm that there is a key piece missing from puppet (like the git repos in gerrit), or engineer something in (like a do-not-configure-replication flag file or something)
19:33:57 <jeblair> yeah, in both cases, there will be a shutdown/rsync/startup process, where i'm sure we won't want puppet agent running on either side.  :)
19:34:25 <jeblair> sorry, an rsync/shutdown/rsync/startup process.  :)
19:34:42 <jeblair> we have moved both of them before, and it's not too hairy.
19:34:45 <clarkb> gerrit will need DB syncing too. lots of little moving parts. should be fun. The module cleanup is just the first tiny step
19:35:12 <jeblair> yep
19:35:29 <jeblair> we'll probably want to schedule these moves for a saturday or sunday.
19:35:58 <clarkb> maybe after the summit? do people usually get back to work with a passion or are they hungover/
19:36:11 <clarkb> that assumes we can be ready in two weeks
19:36:54 <jeblair> clarkb: before or after summit shouldn't matter, esp on a weekend.
19:37:00 <clarkb> ok
19:37:29 <jeblair> it shouldn't be a huge amount of downtime (like .5 hour)
19:37:59 <jeblair> (we'll schedule more for planning/safety)
19:38:32 <jeblair> also, advantage of moving servers vs in-place upgrade is there is an obvious and not-too-difficult roll-back plan.  :)
19:39:03 <clarkb> I should probaby start an etherpad with a list of things that need to be done
19:39:19 <fungi> speaking of which, who handles the dns records (and how/where)?
19:39:54 <jeblair> fungi: we do, via the rs openstack cloud servers account
19:40:14 <fungi> jeblair: okay, so low latency on getting those changed. awesome
19:40:40 <jeblair> fungi: yes, very low.  :)
19:40:55 <fungi> that does indeed make cut-over and fall-back quick
19:41:20 <jeblair> fungi: and we can drop the ttl's to 300 a fews days before we start
19:41:58 <fungi> all the better. i assume you don't get the opportunity to shuffle ip addressing between the vms though
19:42:25 <jeblair> fungi: very unlikely considering the move from legacy -> nova
19:42:51 <fungi> but as long as a half hour window is acceptable, then dns is certainly doable
19:43:39 <clarkb> we took longer to upgrade jenkins last time :)
19:43:54 <jeblair> and that had no notice.  :)
19:43:57 * fungi spent too much time doing zero-downtime planned maintenance swaps in a former life
19:44:22 <fungi> yeah, emergencies can't be helped. i think you guys knocked that out remarkably quickly though
19:44:41 <jeblair> weekends really aren't that busy (except sundays seem busier than saturdays), and advance notice goes a long way.  :)
19:45:20 <jeblair> any more #topics?
19:45:39 <fungi> a quick note on the jenkins doc job conversions
19:46:00 <jeblair> #topic jenkins job builder job conversions
19:46:40 <fungi> per annegentle a few minutes ago, the artifact-id job-builder needs is apparently buried in each root pom.xml
19:46:57 <fungi> i need to go back through the ones i converted earlier and double-check them
19:47:21 <fungi> hopefully i either guessed correctly or they didn't break anything being wrong
19:47:44 <fungi> since they're already merged now
19:47:46 <clarkb> fungi: I am willing to bet if it set it wrong jenkins just updated the values afte rthe first run
19:48:11 <clarkb> and the job builder doesn't make changes very often so may not be a major issue
19:48:39 <fungi> hrm. so is that field actually needed in the job-builder configs if jenkins detects it with magic anyway?
19:49:04 <clarkb> that I don't know. I assume it is set in the job builder to provide a complete xml output
19:49:12 <fungi> i wondered about that, seeing as how there was no field in the webui to add it
19:49:41 <clarkb> actually I think it does need it to avoid jenkins jobs making needless changes
19:50:08 <jeblair> clarkb: jenkins jobs keeps its own cache, and only makes changes when the job is different than its cache
19:50:23 <jeblair> clarkb: so it won't update a job if it changes in jenkins
19:50:43 <jeblair> clarkb: but if the yaml changes, it would presumably remove the artifact id if it wasn't in the yaml
19:50:48 <jeblair> maybe that's what you were saying
19:51:21 <clarkb> I wasn't thinking of the cache, but the second scenario is still a small problem
19:51:57 <jeblair> but if it doesn't hurt anything, getting rid of needless configuration in job builder would be nice
19:52:28 <clarkb> might need to dig into jenkins maven source to sort it all out though
19:52:34 <fungi> and there is definitely some discrepancy between them in the doc jobs. i found some use the same id but others use the job name for that instead
19:53:09 <jeblair> while we're on the subject, it's probably worth mentioning for the record that fungi, clarkb, annegentle, and I are all spent time last week and this week getting the last jenkins jobs that aren't defined by job builder converted.
19:53:32 <fungi> i've claimed another half-dozen and am chewing on them now
19:53:46 <jeblair> awesome!
19:54:06 <fungi> thus the realization about the artifact-id field
19:56:01 <fungi> anyway, that's all i had on that
19:56:41 <clarkb> random log grepping fact. zuul started over 2k jobs on September 20th
19:56:57 <fungi> yikes!
19:56:58 <jeblair> yowza!
19:57:35 <clarkb> grep 'Build .* started' /var/log/zuul/zuul.log.2012-09-20 | wc -l
19:57:35 <jeblair> (graphing data from zuul would be fun, i think)
19:58:16 <jeblair> and on that bombshell...
19:58:18 <jeblair> #endmeeting