19:02:45 #startmeeting ci 19:02:46 Meeting started Tue Sep 4 19:02:45 2012 UTC. The chair is jeblair. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:02:47 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:02:48 The meeting name has been set to 'ci' 19:03:18 anyone want to talk about ci stuff? 19:03:21 sure 19:03:49 absolutely 19:03:59 #topic trivial rebase hook 19:04:08 fungi: why don't you tell folks about what you've been up to? 19:04:34 i've got a commit in for review to get the hook pushed 19:04:59 we need a dummy account to which we attribute the comments on changed commit ids first 19:05:15 so it's set to wip until i get that id # and update the commit 19:05:21 #action jeblair create gerrit account for trivial rebase hook 19:05:39 notes on precisely what we need are i the current commit message 19:06:00 https://review.openstack.org/#/c/12373/ 19:06:31 #link https://review.openstack.org/#/c/12373/ 19:06:41 ah, yes, i forget the tags 19:07:04 what is the expected behavior of this hook? 19:07:10 anyway, it's fully tested on my gerrit dev vm, and i believe all your requested adjustments/fixes are integrated 19:07:11 so, i've been dealing with other stuff this week, so i guess i've missed out on the purpose of this change? 19:07:12 fungi: awesome! i'm very exciting about this. 19:07:22 expected behaviors: 19:08:23 fungi: also you should add "fixes bug 881184" to the commit message -- LP will be automatically updated that way 19:08:24 Launchpad bug 881184 in openstack-ci "consider trivial rebase hook for gerrit" [Medium,Triaged] https://launchpad.net/bugs/881184 19:08:25 1. when a second or subsequent patchset is committed for review, if the git patch-id is calculated to be the same as the previous commit, code review flags are re-added for each of the prior reviewers 19:09:50 2. except if the commit message has been changed, in which case code reviews are not re-added, but a comment is made by the "Trivial Rebase" (trivial-rebase) role account noting that observation 19:10:09 that's basically it 19:10:22 so this way reviewers don't have to figure out that a new patchset is just a rebase and re-review. 19:10:48 correct. about a dozen patches were made to the script, for which i have a local git branch against upstream we can try to get merged later 19:11:29 patches are already atomic and thoroughly documented, so they should be okay with most if not all 19:11:42 Shrews, clarkb: all caught up? 19:11:48 jeblair: yup. thx 19:11:50 I think that covers it 19:11:56 good change 19:12:00 cool, anything else about trivial rebase? 19:12:09 not from me 19:13:34 #topic jenkins "improvements" 19:14:09 we're continuing to make changes to our jenkins configuration to abate performance problems and generally make things suck less 19:14:33 clarkb: you want to talk about the scp plugin and related things? 19:15:11 sure. I have modified the upstream SCP plugin to copy console logs of builds and to add an option to copy files when a build fails 19:15:47 this allows us to copy the build artifacts of all builds to another server, which we are now doing (things get copied to logs.openstack.org) 19:16:22 these files are then served by apache and access to them is much quicker than through jenkins 19:17:21 there have been a couple bugs in my changes to the SCP plugin. It wasn't always copying the entirety of the console log because it needed to wait longer (the fix was to spawn a new thread that can wait for everything else to finish) 19:18:01 and it doesn't currently handle null workspaces which happen when slaves die in unexpected ways. The fix for this is currently on jenkins-dev and seems to be working. Will put it on jenkins.o.o when opportunity arises. 19:18:24 clarkb: cool, so with that in place, i think we can start reducing the retention time for jenkins builds to 1 day 19:18:53 clarkb: and also stop jenkins-archiving some artifacts (like devstack logs) since they are getting copied to the new site 19:19:36 semi related to this is the nose html output plugin. It captures unit test results in an html file that we can serve via apache on logs.o.o. 19:19:55 It is able to capture skipped tests now too which is cool 19:20:20 clarkb: think they'll accept your scp plugin changes upstream? 19:20:43 Shrews: I have submitted a pull request for that, but haven't heard back on it. I do need to update the pull request to include these bug fixes 19:21:05 #action mtaylor get in touch with appropriate jenkins buddies about clarkb's scp plugin pull request 19:21:26 clarkb: link? 19:21:38 Shrews: I haven't created a project on review.o.o because I would really like to upstream the changes 19:21:58 #link https://github.com/jenkinsci/scp-plugin/pull/4 19:22:28 yes, i think that's the way to go; only if that fails should we start maintaining a fork. :( 19:22:46 ++ 19:23:19 so, a while ago, we talked a while ago about explicitly noting when a unit test tries to "sudo" during a jenkins run (which will fail), and failing the test. 19:23:40 we had a lot of sudo related cronspam this weekend, which moved that up my priority list a bit... 19:23:52 so i'm about to merge this change: 19:23:54 #link https://review.openstack.org/#/c/12366/ 19:24:05 which will start failing tests if sudo has been run during the test 19:24:50 that won't actually stop us from getting spammed, but it will at least automate the process of letting people know their tests are trying to sudo 19:25:42 jeblair: semi-related to the Heat are going to run their 'sudo' tests on a separate Jenkins that will +1/-1 verify vote 19:25:59 is there any option to also alias sudo to /usr/bin/false for the user running the tests? 19:26:03 * LinuxJedi is not sure if they are the cause or not but wouldn't be surprised 19:26:45 fungi: wouldn't be surprised if the tests run sudo with the full path 19:26:59 okay, good point 19:27:04 fungi: and from a fork/exec from within python, etc... 19:27:30 yeah, so no way to solve that short of chroot 19:27:30 LinuxJedi: perhaps, but this has certainly come up with the core projects, so it's a general solution 19:28:18 #topic devstack gate 19:28:37 We're now running two devstack-gate jobs, one with cinder, and one with nova-volume 19:29:04 they are both going to be supported in folsom, so we're testing both 19:30:12 in grizzly, n-vol should be deprecated, so i think we can just test cinder then 19:31:00 anything else on this subject? 19:31:43 #topic zuul 19:32:41 jeblair: it appaers that the refactoring has corrected the bug(s) we were seeing 19:32:46 I refactored some of the code in zuul so that some of the more complex things we're doing with early dequeuing of changes, etc, should be easier to comprehend, and hopefully doesn't have some of the edge cases that we were running into last week 19:32:50 clarkb: whew :) 19:33:14 main user-visible change is that the ordering of the status queue is the reverse of what it used to be... 19:33:21 #link https://jenkins.openstack.org/zuul/status 19:33:39 that and changes shouldn't get stuck because zuul doesn't know what to do with them 19:33:51 with the head of the queue at the top/left, with following changes trailing down and right 19:34:39 also, as soon as a change at the head of the queue fails, it is immediately dequeued, but stays visible on the status screen, disconnected from the rest of the queue, while the remainder of its jobs finish 19:35:04 that's called a "severed head". 19:35:56 #topic job builder 19:36:23 based on yun mao's work with pylint for nova 19:36:56 i added a job that generates the diff of xml output from the jenkins job builder 19:37:02 see it in action here: 19:37:04 #link https://review.openstack.org/#/c/12288/ 19:37:36 so we can examine how any commit to jenkins job builder will affect the xml for the openstack jobs 19:38:04 i'll also set up a similar test to the openstack yaml for the job builder, so we can see how any change to the yaml will change the xml 19:38:30 it's a non-voting job, with a custom message -- so it doesn't say FAILURE/SUCCESS, but rather "xml has (or has not) changed" 19:38:39 awesome :) 19:39:28 #topic doc jobs 19:39:46 clarkb: you've been working with annegentle on docs jobs -- anything you folks want to fill us in on? 19:40:11 we are trying to start using jenkins jobs builder for those jobs 19:40:44 the existing jobs use environment variables not injected by zuul so the builder code in jenkins job builder will need to be upadted to do that 19:41:00 can I ask whether anyone is using sonar to manage openstack code quality? 19:41:28 I can work on that, but may have questions for jeblair and LinuxJedi. We'll see how far I get 19:41:49 but other than that the jobs are very similar to the other jobs created by jenkins job builder 19:41:54 iopenstack: in a minute 19:42:30 clarkb: if you want to take extending job builder to do properties, great 19:42:41 the jobs run as post merge jobs and compile documents (from markdown or docbook) and publish the build artifacts 19:42:53 jeblair: yup I will do that. 19:42:58 clarkb: i think with that addition, most of the docs jobs could be put into job builder 19:43:11 #action clarkb extend jenkins job builder to support build properties 19:43:20 Thanks Jeblair 19:43:42 clarkb: also, feel free to consider alternate ways of supporting those jobs -- we have tools now we didn't back then. 19:43:58 k 19:44:24 #topic open disussion 19:44:56 I have a couple things for open discussion. First gerritbot is working now. Hopefully it isn't too spammy. If it is we can tweak it 19:45:11 clarkb: awesome stuff :) 19:45:44 and the other thing was I pointed puppet labs folks at jenkins job builder and they complained about the lack of documentation 19:45:58 would be good to add some of that especially now that it is an independent project 19:46:38 a very good point, I think we did have some in the openstack-ci-puppet tree but there probably needs to be a separate set of docs 19:46:52 clarkb: indeed. if i ever get a spare moment, i will write some. i'm not even sure it's completely de-openstacked yet. 19:47:07 clarkb: in fact, i'm sure it's not. 19:47:11 I am a little bit new to automation builders: what is the difference between gerritbot and jenkins? 19:47:21 we have non-openstack people begging to use it, so we will need to do that at some point 19:47:34 iopenstack: this is the CI/infrastructure team meeting -- you are welcome, and your question is relevant; but since you just dropped in, and i haven't seen you before, i wanted to make sure you are where you want to be. could you introduce yourself? 19:48:59 yes. sorry this is third times I am hearing on openstack meeting. I am Dan from the UK, university of surrey. I am working for EC ICT project for many years and now focus on bigData project. 19:49:11 #action jeblair finish de-openstacking job builder 19:49:46 iopenstack: thanks for dropping by; i haven't heard of anyone using sonar. are you? 19:50:00 I like to learn something from your guys and get to find a place to contribute if I can. 19:50:22 iopenstack: in answer to your second question, gerritbot is an IRC bot which tells us about changes. Not really comparable to Jenkins 19:50:46 not yet. but I would like to try to use it. Maybe it would be a good idea to use it/ 19:51:00 iopenstack: a lot of our work is documented here: http://ci.openstack.org/ though it can certainly be updated 19:51:02 LinuxJedi, thanks. I will take a look at. 19:51:13 thanks. jeblair. 19:51:21 iopenstack: https://github.com/openstack/openstack-ci-puppet 19:51:54 iopenstack: that's the puppet repo where the code and configuration for all of the CI and infrastructure systems resides -- you may be able to answer questions that the docs don't answer by looking there... 19:52:15 Thanks Jeblair. all are catched. 19:52:34 iopenstack: if you think that sonar is useful, i would recommend talking to some of the PTLs for the core projects about it, or bringing it up on the mailing list 19:53:33 iopenstack: and if people decide it's worth running centrally, we can help you get it going. 19:53:36 iopenstack: any further CI/infra questions after the meeting can be directed to the #openstack-infra and other openstack development questions the #openstack-dev channel 19:54:07 Jeblair: Thanks. I will do that after I have done a little bit research 19:54:08 iopenstack: personally, i have no experience with it -- so i can't offer an opinion right now as to whether we can or should run it. 19:54:24 time is short; anyone have anything else? 19:54:49 ok. thanks. If I got finding something useful, I will be back in the next meeting. 19:55:05 we should have mtaylor and a couple of others back for next week's meeting 19:55:26 #action iopenstack looking into sonar 19:55:54 or whatever remains of them after burning man. :) 19:56:12 indeed :) 19:56:26 he better be back, I have stuff that needs approving :) 19:56:33 okay, thanks everyone! 19:56:35 #endmeeting