19:01:46 #startmeeting ci 19:01:47 Meeting started Tue Oct 2 19:01:46 2012 UTC. The chair is jeblair. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:48 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:49 The meeting name has been set to 'ci' 19:02:07 (looking up last meeting) 19:02:37 #topic items from last meeting 19:02:51 I believe this one is unchanged: 19:02:54 #action jbryce provide a test foundation server 19:03:06 i have an update there 19:03:21 awesome: go for it 19:03:30 i pinged him and reed via e-mail later in the week 19:03:55 no word on progress yet, but reed expressed interest in taking over doing that if he gets time 19:03:59 I have talked to Todd today, he hopefully will hop online 19:04:09 #action reed provide a test foundation server 19:04:09 reed: awesome! 19:04:22 #action todd provide a test foundation server 19:04:29 #action anyone provide a test foundation server 19:05:13 cool, if todd shows up, we can chat about that more, or any time in #-infra 19:05:29 otherwise, we're still in a holding pattern. it works on review-dev and can be demo'd there 19:05:45 (and seems to work) 19:06:09 clarkb: i believe you proposed the translation session for the summit? 19:06:19 jeblair: I did. It has been preapproved too 19:06:25 groovy! 19:07:01 pabelanger added the puppet lint job, which rocks 19:07:44 thanks pabelanger! 19:08:05 it tells us how bad we are 19:08:16 it is indicating we have quite a ways to go before we're lint-free 19:08:38 the first time i saw it, i thought for a moment my browser was going to lock up trying to show the entire log ;) 19:09:06 #link http://idle.slashdot.org/story/12/10/01/1215213/us-agricultural-economists-say-bacon-shortage-is-hogwash 19:09:21 * fungi breathes a sigh of relief 19:09:22 ^ seems like we're off the hook there 19:09:22 jeblair: Error: "seems" is not a valid command. 19:09:53 and mordred did document use cases for bug 995604 19:09:53 ^ seems uvirtbot likes this topic 19:09:54 Launchpad bug 995604 in openstack-ci "use symbolic-ref from gerrit to provide moving codename targets" [High,In progress] https://launchpad.net/bugs/995604 19:09:55 fungi: Error: "seems" is not a valid command. 19:10:05 #topic symbolic-ref 19:10:33 we chatted a bit earlier about symbolic refs. i don't think it's a slam dunk for what we want to do 19:10:44 jeblair: yes, i updated the bug report with details 19:11:06 but we have a lot of good information thanks to fungi's work 19:11:17 as noted there, i did a proof-of-concept following his suggestion, but tags do indeed seem to fill the void in a better way 19:11:52 cool, my feeling is it's sort of a back-burner idea until we chat at the summit and figure out what will happen with branches next time around 19:12:02 during or after the ods session(s) on branch organization and release process, we need to talk about how we automate the tag updates 19:12:25 presumably that happens zuul-side? 19:12:41 and not directly on gerrit's copies? 19:13:03 fungi: maybe it's a gerrit hook script on ref-updated? 19:13:05 since we wouldn't want to tag a commit until it's merged successfully 19:13:21 jeblair: that could work 19:13:55 but i'll definitely try to catch any ods discussions around that as well 19:14:21 fungi: anyway, i also don't want to presume we will need this; it's possible (granted, unlikely) we could come up with a branch model that satisfies the use case. 19:14:31 that would be ideal, i agree 19:15:20 #topic zuul 19:15:30 i gave a talk at the jenkins user conference about zuul 19:15:52 and plugged job-builder, gerrit, the openstack-ci-puppet repo, and our wiki/documentation too 19:16:15 some folks have popped into #-infra as a result of that, which is great 19:17:05 i'd love to get patches from outside of openstack for anything we do. :) 19:17:06 Thanks 19:17:11 oh, and git-review too. 19:17:25 I added a minor feature to zuul to have it report the zuul status link when it starts gate jobs 19:17:44 that started a short conversation on per change status pages 19:17:46 saw that--clickyconvenient 19:17:51 zuul has some missing documentation and some things that make it hard to set up initially, so i have one patch in review to help with that, and another one coming 19:18:16 I think zuul may need to be restarted to pick up the change though 19:18:24 clarkb: yep 19:18:47 so per-change status pages are totally doable, it sort of depends on how much we want zuul to be a web app 19:19:29 maybe giving zuul a json api and allowing the webab to be decoupled is a better way to handle this problem? 19:19:51 clarkb: that's not a bad idea. 19:21:45 clarkb: yeah, i think i like that. i'll cipher on that. btw, anyone here love writing web apps? 19:22:39 * fungi thought the web was a passing fad for longer than he cares to admit, but will volunteer for anything 19:22:57 fungi: you still do, don't you? :) 19:23:12 i'm just hoping beyond hope at this stage 19:23:31 anyone else have a #topic? 19:23:41 gerrit whitespace change detection 19:23:48 oneiric -> precise upgrades 19:24:09 #topic gerrit whitespace change detection 19:24:10 asps. very dangerous. clarkb goes first 19:24:13 or me 19:24:15 whoops. :) 19:24:48 markmc was asking about a feature to have the trivial-rebase hook module no longer reapply reviews on whitespace changes between patchsets 19:25:06 #link https://bugs.launchpad.net/openstack-ci/+bug/1057506 19:25:07 Launchpad bug 1057506 in openstack-ci "Gerrit trivial rebase detection ignores whitespace" [Medium,Fix released] 19:25:22 merged yesterday: #link https://review.openstack.org/13775 19:25:42 keep an eye out in case it does unexpected things, but i kicked the tires pretty thoroughly already 19:26:01 cool! 19:26:20 your turn, clarkb 19:26:31 #topic asps; oneiric -> precise upgrades 19:26:57 so a few of the infrastructure hosts run oneiric and we have wanted to upgrade them to precise. notably the jenkins and gerrit hosts 19:27:20 mordred: ping 19:27:28 I have started cleaning up the Jenkins puppet modules to make it easy to stand up jenkins on a brand new host. 19:27:56 once https://review.openstack.org/#/c/13926/ merges I think we will be ready to start building a new jenkins-dev host 19:28:21 the motivation for building new hosts is to move to rackspace's new openstack cloud where we can have ipv6 19:28:51 the jenkins stuff looks great, btw. 19:29:01 i have a question about the gerrit modules though.... 19:29:29 is our gerrit puppet config so complete that when we spin up a new gerrit host, will it immediately start replicating empty repositories to github? 19:29:47 o_O I hope not but I think that is a valid concern 19:30:06 the gerrit module has a lot more moving parts particularly for configuration 19:30:36 I only just started poking at gerrit today, definitely need to look at it pretty closely 19:30:45 i avoided using it when i built my test gerrit vm because i was unsure how much of a loaded gun it was, after acasual glance 19:30:56 i guess it probably wouldn't actually have the empty repos in gerrit yet, right? 19:31:24 (even though it would probably have a working replication config, and possibly empty git repos in /var for the apache mirrors?) 19:31:26 jeblair: no, but it has the project lists and the replication config 19:32:56 a similar concern when we get to jenkins.o.o will be making sure that we don't have two masters fighting for the same sets of slaves 19:32:58 all things being equal, it would be swell if we did not replicate empty repos to github, so we should either confirm that there is a key piece missing from puppet (like the git repos in gerrit), or engineer something in (like a do-not-configure-replication flag file or something) 19:33:57 yeah, in both cases, there will be a shutdown/rsync/startup process, where i'm sure we won't want puppet agent running on either side. :) 19:34:25 sorry, an rsync/shutdown/rsync/startup process. :) 19:34:42 we have moved both of them before, and it's not too hairy. 19:34:45 gerrit will need DB syncing too. lots of little moving parts. should be fun. The module cleanup is just the first tiny step 19:35:12 yep 19:35:29 we'll probably want to schedule these moves for a saturday or sunday. 19:35:58 maybe after the summit? do people usually get back to work with a passion or are they hungover/ 19:36:11 that assumes we can be ready in two weeks 19:36:54 clarkb: before or after summit shouldn't matter, esp on a weekend. 19:37:00 ok 19:37:29 it shouldn't be a huge amount of downtime (like .5 hour) 19:37:59 (we'll schedule more for planning/safety) 19:38:32 also, advantage of moving servers vs in-place upgrade is there is an obvious and not-too-difficult roll-back plan. :) 19:39:03 I should probaby start an etherpad with a list of things that need to be done 19:39:19 speaking of which, who handles the dns records (and how/where)? 19:39:54 fungi: we do, via the rs openstack cloud servers account 19:40:14 jeblair: okay, so low latency on getting those changed. awesome 19:40:40 fungi: yes, very low. :) 19:40:55 that does indeed make cut-over and fall-back quick 19:41:20 fungi: and we can drop the ttl's to 300 a fews days before we start 19:41:58 all the better. i assume you don't get the opportunity to shuffle ip addressing between the vms though 19:42:25 fungi: very unlikely considering the move from legacy -> nova 19:42:51 but as long as a half hour window is acceptable, then dns is certainly doable 19:43:39 we took longer to upgrade jenkins last time :) 19:43:54 and that had no notice. :) 19:43:57 * fungi spent too much time doing zero-downtime planned maintenance swaps in a former life 19:44:22 yeah, emergencies can't be helped. i think you guys knocked that out remarkably quickly though 19:44:41 weekends really aren't that busy (except sundays seem busier than saturdays), and advance notice goes a long way. :) 19:45:20 any more #topics? 19:45:39 a quick note on the jenkins doc job conversions 19:46:00 #topic jenkins job builder job conversions 19:46:40 per annegentle a few minutes ago, the artifact-id job-builder needs is apparently buried in each root pom.xml 19:46:57 i need to go back through the ones i converted earlier and double-check them 19:47:21 hopefully i either guessed correctly or they didn't break anything being wrong 19:47:44 since they're already merged now 19:47:46 fungi: I am willing to bet if it set it wrong jenkins just updated the values afte rthe first run 19:48:11 and the job builder doesn't make changes very often so may not be a major issue 19:48:39 hrm. so is that field actually needed in the job-builder configs if jenkins detects it with magic anyway? 19:49:04 that I don't know. I assume it is set in the job builder to provide a complete xml output 19:49:12 i wondered about that, seeing as how there was no field in the webui to add it 19:49:41 actually I think it does need it to avoid jenkins jobs making needless changes 19:50:08 clarkb: jenkins jobs keeps its own cache, and only makes changes when the job is different than its cache 19:50:23 clarkb: so it won't update a job if it changes in jenkins 19:50:43 clarkb: but if the yaml changes, it would presumably remove the artifact id if it wasn't in the yaml 19:50:48 maybe that's what you were saying 19:51:21 I wasn't thinking of the cache, but the second scenario is still a small problem 19:51:57 but if it doesn't hurt anything, getting rid of needless configuration in job builder would be nice 19:52:28 might need to dig into jenkins maven source to sort it all out though 19:52:34 and there is definitely some discrepancy between them in the doc jobs. i found some use the same id but others use the job name for that instead 19:53:09 while we're on the subject, it's probably worth mentioning for the record that fungi, clarkb, annegentle, and I are all spent time last week and this week getting the last jenkins jobs that aren't defined by job builder converted. 19:53:32 i've claimed another half-dozen and am chewing on them now 19:53:46 awesome! 19:54:06 thus the realization about the artifact-id field 19:56:01 anyway, that's all i had on that 19:56:41 random log grepping fact. zuul started over 2k jobs on September 20th 19:56:57 yikes! 19:56:58 yowza! 19:57:35 grep 'Build .* started' /var/log/zuul/zuul.log.2012-09-20 | wc -l 19:57:35 (graphing data from zuul would be fun, i think) 19:58:16 and on that bombshell... 19:58:18 #endmeeting