19:02:45 <jeblair> #startmeeting ci
19:02:46 <openstack> Meeting started Tue Sep  4 19:02:45 2012 UTC.  The chair is jeblair. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:02:47 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:02:48 <openstack> The meeting name has been set to 'ci'
19:03:18 <jeblair> anyone want to talk about ci stuff?
19:03:21 <clarkb> sure
19:03:49 <fungi> absolutely
19:03:59 <jeblair> #topic trivial rebase hook
19:04:08 <jeblair> fungi: why don't you tell folks about what you've been up to?
19:04:34 <fungi> i've got a commit in for review to get the hook pushed
19:04:59 <fungi> we need a dummy account to which we attribute the comments on changed commit ids first
19:05:15 <fungi> so it's set to wip until i get that id # and update the commit
19:05:21 <jeblair> #action jeblair create gerrit account for trivial rebase hook
19:05:39 <fungi> notes on precisely what we need are i the current commit message
19:06:00 <fungi> https://review.openstack.org/#/c/12373/
19:06:31 <jeblair> #link https://review.openstack.org/#/c/12373/
19:06:41 <fungi> ah, yes, i forget the tags
19:07:04 <clarkb> what is the expected behavior of this hook?
19:07:10 <fungi> anyway, it's fully tested on my gerrit dev vm, and i believe all your requested adjustments/fixes are integrated
19:07:11 <Shrews> so, i've been dealing with other stuff this week, so i guess i've missed out on the purpose of this change?
19:07:12 <jeblair> fungi: awesome!  i'm very exciting about this.
19:07:22 <fungi> expected behaviors:
19:08:23 <jeblair> fungi: also you should add "fixes bug 881184" to the commit message -- LP will be automatically updated that way
19:08:24 <uvirtbot> Launchpad bug 881184 in openstack-ci "consider trivial rebase hook for gerrit" [Medium,Triaged] https://launchpad.net/bugs/881184
19:08:25 <fungi> 1. when a second or subsequent patchset is committed for review, if the git patch-id is calculated to be the same as the previous commit, code review flags are re-added for each of the prior reviewers
19:09:50 <fungi> 2. except if the commit message has been changed, in which case code reviews are not re-added, but a comment is made by the "Trivial Rebase" (trivial-rebase) role account noting that observation
19:10:09 <fungi> that's basically it
19:10:22 <jeblair> so this way reviewers don't have to figure out that a new patchset is just a rebase and re-review.
19:10:48 <fungi> correct. about a dozen patches were made to the script, for which i have a local git branch against upstream we can try to get merged later
19:11:29 <fungi> patches are already atomic and thoroughly documented, so they should be okay with most if not all
19:11:42 <jeblair> Shrews, clarkb: all caught up?
19:11:48 <Shrews> jeblair: yup. thx
19:11:50 <clarkb> I think that covers it
19:11:56 <Shrews> good change
19:12:00 <jeblair> cool, anything else about trivial rebase?
19:12:09 <fungi> not from me
19:13:34 <jeblair> #topic jenkins "improvements"
19:14:09 <jeblair> we're continuing to make changes to our jenkins configuration to abate performance problems and generally make things suck less
19:14:33 <jeblair> clarkb: you want to talk about the scp plugin and related things?
19:15:11 <clarkb> sure. I have modified the upstream SCP plugin to copy console logs of builds and to add an option to copy files when a build fails
19:15:47 <clarkb> this allows us to copy the build artifacts of all builds to another server, which we are now doing (things get copied to logs.openstack.org)
19:16:22 <clarkb> these files are then served by apache and access to them is much quicker than through jenkins
19:17:21 <clarkb> there have been a couple bugs in my changes to the SCP plugin. It wasn't always copying the entirety of the console log because it needed to wait longer (the fix was to spawn a new thread that can wait for everything else to finish)
19:18:01 <clarkb> and it doesn't currently handle null workspaces which happen when slaves die in unexpected ways. The fix for this is currently on jenkins-dev and seems to be working. Will put it on jenkins.o.o when opportunity arises.
19:18:24 <jeblair> clarkb: cool, so with that in place, i think we can start reducing the retention time for jenkins builds to 1 day
19:18:53 <jeblair> clarkb: and also stop jenkins-archiving some artifacts (like devstack logs) since they are getting copied to the new site
19:19:36 <clarkb> semi related to this is the nose html output plugin. It captures unit test results in an html file that we can serve via apache on logs.o.o.
19:19:55 <clarkb> It is able to capture skipped tests now too which is cool
19:20:20 <Shrews> clarkb: think they'll accept your scp plugin changes upstream?
19:20:43 <clarkb> Shrews: I have submitted a pull request for that, but haven't heard back on it. I do need to update the pull request to include these bug fixes
19:21:05 <jeblair> #action mtaylor get in touch with appropriate jenkins buddies about clarkb's scp plugin pull request
19:21:26 <jeblair> clarkb: link?
19:21:38 <clarkb> Shrews: I haven't created a project on review.o.o because I would really like to upstream the changes
19:21:58 <clarkb> #link https://github.com/jenkinsci/scp-plugin/pull/4
19:22:28 <jeblair> yes, i think that's the way to go; only if that fails should we start maintaining a fork.  :(
19:22:46 <Shrews> ++
19:23:19 <jeblair> so, a while ago, we talked a while ago about explicitly noting when a unit test tries to "sudo" during a jenkins run (which will fail), and failing the test.
19:23:40 <jeblair> we had a lot of sudo related cronspam this weekend, which moved that up my priority list a bit...
19:23:52 <jeblair> so i'm about to merge this change:
19:23:54 <jeblair> #link https://review.openstack.org/#/c/12366/
19:24:05 <jeblair> which will start failing tests if sudo has been run during the test
19:24:50 <jeblair> that won't actually stop us from getting spammed, but it will at least automate the process of letting people know their tests are trying to sudo
19:25:42 <LinuxJedi> jeblair: semi-related to the Heat are going to run their 'sudo' tests on a separate Jenkins that will +1/-1 verify vote
19:25:59 <fungi> is there any option to also alias sudo to /usr/bin/false for the user running the tests?
19:26:03 * LinuxJedi is not sure if they are the cause or not but wouldn't be surprised
19:26:45 <jeblair> fungi: wouldn't be surprised if the tests run sudo with the full path
19:26:59 <fungi> okay, good point
19:27:04 <jeblair> fungi: and from a fork/exec from within python, etc...
19:27:30 <fungi> yeah, so no way to solve that short of chroot
19:27:30 <jeblair> LinuxJedi: perhaps, but this has certainly come up with the core projects, so it's a general solution
19:28:18 <jeblair> #topic devstack gate
19:28:37 <jeblair> We're now running two devstack-gate jobs, one with cinder, and one with nova-volume
19:29:04 <jeblair> they are both going to be supported in folsom, so we're testing both
19:30:12 <jeblair> in grizzly, n-vol should be deprecated, so i think we can just test cinder then
19:31:00 <jeblair> anything else on this subject?
19:31:43 <jeblair> #topic zuul
19:32:41 <clarkb> jeblair: it appaers that the refactoring has corrected the bug(s) we were seeing
19:32:46 <jeblair> I refactored some of the code in zuul so that some of the more complex things we're doing with early dequeuing of changes, etc, should be easier to comprehend, and hopefully doesn't have some of the edge cases that we were running into last week
19:32:50 <jeblair> clarkb: whew  :)
19:33:14 <jeblair> main user-visible change is that the ordering of the status queue is the reverse of what it used to be...
19:33:21 <jeblair> #link https://jenkins.openstack.org/zuul/status
19:33:39 <clarkb> that and changes shouldn't get stuck because zuul doesn't know what to do with them
19:33:51 <jeblair> with the head of the queue at the top/left, with following changes trailing down and right
19:34:39 <jeblair> also, as soon as a change at the head of the queue fails, it is immediately dequeued, but stays visible on the status screen, disconnected from the rest of the queue, while the remainder of its jobs finish
19:35:04 <jeblair> that's called a "severed head".
19:35:56 <jeblair> #topic job builder
19:36:23 <jeblair> based on yun mao's work with pylint for nova
19:36:56 <jeblair> i added a job that generates the diff of xml output from the jenkins job builder
19:37:02 <jeblair> see it in action here:
19:37:04 <jeblair> #link https://review.openstack.org/#/c/12288/
19:37:36 <jeblair> so we can examine how any commit to jenkins job builder will affect the xml for the openstack jobs
19:38:04 <jeblair> i'll also set up a similar test to the openstack yaml for the job builder, so we can see how any change to the yaml will change the xml
19:38:30 <jeblair> it's a non-voting job, with a custom message -- so it doesn't say FAILURE/SUCCESS, but rather "xml has (or has not) changed"
19:38:39 <LinuxJedi> awesome :)
19:39:28 <jeblair> #topic doc jobs
19:39:46 <jeblair> clarkb: you've been working with annegentle on docs jobs -- anything you folks want to fill us in on?
19:40:11 <clarkb> we are trying to start using jenkins jobs builder for those jobs
19:40:44 <clarkb> the existing jobs use environment variables not injected by zuul so the builder code in jenkins job builder will need to be upadted to do that
19:41:00 <iopenstack> can I ask whether anyone is using sonar to manage openstack code quality?
19:41:28 <clarkb> I can work on that, but may have questions for jeblair and LinuxJedi. We'll see how far I get
19:41:49 <clarkb> but other than that the jobs are very similar to the other jobs created by jenkins job builder
19:41:54 <jeblair> iopenstack: in a minute
19:42:30 <jeblair> clarkb: if you want to take extending job builder to do properties, great
19:42:41 <clarkb> the jobs run as post merge jobs and compile documents (from markdown or docbook) and publish the build artifacts
19:42:53 <clarkb> jeblair: yup I will do that.
19:42:58 <jeblair> clarkb: i think with that addition, most of the docs jobs could be put into job builder
19:43:11 <jeblair> #action clarkb extend jenkins job builder to support build properties
19:43:20 <iopenstack> Thanks Jeblair
19:43:42 <jeblair> clarkb: also, feel free to consider alternate ways of supporting those jobs -- we have tools now we didn't back then.
19:43:58 <clarkb> k
19:44:24 <jeblair> #topic open disussion
19:44:56 <clarkb> I have a couple things for open discussion. First gerritbot is working now. Hopefully it isn't too spammy. If it is we can tweak it
19:45:11 <LinuxJedi> clarkb: awesome stuff :)
19:45:44 <clarkb> and the other thing was I pointed puppet labs folks at jenkins job builder and they complained about the lack of documentation
19:45:58 <clarkb> would be good to add some of that especially now that it is an independent project
19:46:38 <LinuxJedi> a very good point, I think we did have some in the openstack-ci-puppet tree but there probably needs to be a separate set of docs
19:46:52 <jeblair> clarkb: indeed.  if i ever get a spare moment, i will write some.  i'm not even sure it's completely de-openstacked yet.
19:47:07 <jeblair> clarkb: in fact, i'm sure it's not.
19:47:11 <iopenstack> I am a little bit new to automation builders: what is the difference between gerritbot and jenkins?
19:47:21 <LinuxJedi> we have non-openstack people begging to use it, so we will need to do that at some point
19:47:34 <jeblair> iopenstack: this is the CI/infrastructure team meeting -- you are welcome, and your question is relevant; but since you just dropped in, and i haven't seen you before, i wanted to make sure you are where you want to be.  could you introduce yourself?
19:48:59 <iopenstack> yes. sorry this is third times I am hearing on openstack meeting. I am Dan from the UK, university of surrey. I am working for EC ICT project for many years and now focus on bigData project.
19:49:11 <jeblair> #action jeblair finish de-openstacking job builder
19:49:46 <jeblair> iopenstack: thanks for dropping by; i haven't heard of anyone using sonar.  are you?
19:50:00 <iopenstack> I like to learn something from your guys and get to find a place to contribute if I can.
19:50:22 <LinuxJedi> iopenstack: in answer to your second question, gerritbot is an IRC bot which tells us about changes.  Not really comparable to Jenkins
19:50:46 <iopenstack> not yet. but I would like to try to use it. Maybe it would be a good idea to use it/
19:51:00 <jeblair> iopenstack: a lot of our work is documented here: http://ci.openstack.org/  though it can certainly be updated
19:51:02 <iopenstack> LinuxJedi, thanks. I will take a look at.
19:51:13 <iopenstack> thanks. jeblair.
19:51:21 <jeblair> iopenstack: https://github.com/openstack/openstack-ci-puppet
19:51:54 <jeblair> iopenstack: that's the puppet repo where the code and configuration for all of the CI and infrastructure systems resides -- you may be able to answer questions that the docs don't answer by looking there...
19:52:15 <iopenstack> Thanks Jeblair. all are catched.
19:52:34 <jeblair> iopenstack: if you think that sonar is useful, i would recommend talking to some of the PTLs for the core projects about it, or bringing it up on the mailing list
19:53:33 <jeblair> iopenstack: and if people decide it's worth running centrally, we can help you get it going.
19:53:36 <LinuxJedi> iopenstack: any further CI/infra questions after the meeting can be directed to the #openstack-infra and other openstack development questions the #openstack-dev channel
19:54:07 <iopenstack> Jeblair: Thanks. I will do that after I have done a little bit research
19:54:08 <jeblair> iopenstack: personally, i have no experience with it -- so i can't offer an opinion right now as to whether we can or should run it.
19:54:24 <jeblair> time is short; anyone have anything else?
19:54:49 <iopenstack> ok. thanks. If I got finding something useful, I will be back in the next meeting.
19:55:05 <LinuxJedi> we should have mtaylor and a couple of others back for next week's meeting
19:55:26 <jeblair> #action iopenstack looking into sonar
19:55:54 <jeblair> or whatever remains of them after burning man.  :)
19:56:12 <LinuxJedi> indeed :)
19:56:26 <LinuxJedi> he better be back, I have stuff that needs approving :)
19:56:33 <jeblair> okay, thanks everyone!
19:56:35 <jeblair> #endmeeting