19:01:54 #startmeeting 19:01:55 Meeting started Tue Aug 16 19:01:54 2011 UTC. The chair is mtaylor. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:56 Useful Commands: #action #agreed #help #info #idea #link #topic. 19:02:18 #topic Actions from last meeting 19:02:30 #action jaypipes design upgrade path jenkins job with mtaylor 19:02:37 (that's my way of saying we still haven't done that) 19:02:53 jeblair did do an auto-closing pull request hook though 19:03:03 jeblair: is that deployed now? 19:03:03 i did... 19:03:20 here it is: https://github.com/openstack/openstack-ci/blob/master/gerrit/close_pull_requests.py 19:03:35 and yes, it's deployed. it should be running in production for keystone and glance 19:03:43 #link https://github.com/openstack/openstack-ci/blob/master/gerrit/close_pull_requests.py 19:03:46 sweet 19:03:53 it runs out of cron every 5 mins 19:04:11 that's all from last week then 19:04:21 #topic Discuss automated testing and configuration coverage 19:04:27 someone last week volunteered to take that and see about adding support for creating gerrit changes 19:04:39 #link http://eavesdrop.openstack.org/meetings/openstack-meeting/2011/openstack-meeting.2011-08-15-20.07.log.html 19:04:42 oh yeah - who was that 19:04:45 anybody remember? 19:04:53 heckj: was that you? 19:05:27 19:44:36 <_0x44> vi, and bengrue is volunteering to write the second-round hook to auto-integrate gerrit and github. 19:05:29 mtaylor: sorry, no 19:05:44 bengrue. that's him 19:06:03 #action bengrue write the second-round gerrit github hook 19:06:09 how's that? :) 19:06:14 :) 19:06:29 oh, cool, I got an action item... 19:06:35 so - soren, jaypipes and deshantm: what are we supposed to pow-wow about? 19:06:44 future bengrue, if you're reading this, let me know if you need anything. :) 19:06:46 gimme a sec, catching up. 19:07:03 caught up. 19:07:05 * jaypipes reading back.. 19:07:31 I think most of the things have been discussed. Xen.org/Citrix is here to help with automation/testing 19:07:39 sweet. we're in favor of that 19:07:41 working thing into our workflow as needed 19:07:43 mtaylor: Is that the action item from yesterday? 19:07:49 yeah 19:07:53 Ok. 19:08:00 deshantm: have you and mtaylor chatted about how to set up a jenkins slave? 19:08:15 jaypipes: not yet 19:08:23 we theoretically have a citrix slave, actually :) 19:08:25 we run jenkins at Citrix and Xen.org 19:08:30 although it's been offline for a bit 19:08:55 deshantm: yup, understood. we'd like to have our master jenkins box trigger builds on your local jenkins. 19:08:55 mtaylor: I must admit, I'm not completely sure. I wasn't following the meeting, my name was just mentioned and I chimed in, but I didn't completely get the context. 19:09:09 I have limited experience, but have created a job successfully 19:09:17 jaypipes: just to be clear (and because people keep saying similar words) ... 19:09:21 jaypipes that neat 19:09:25 deshantm: no worries, mtaylor can help explain that process. 19:09:31 we're actually NOT interested in having our master jenkins trigger jobs on other people's jenkins 19:09:34 because that doesn't work 19:09:41 Bus, meet mtaylor. :P 19:09:45 we're interested in having people provide jenkins slaves, and running jobs there 19:09:47 mtaylor: no? 19:09:54 jaypipes: no. it is not an option 19:10:14 mtaylor: hmm, that's odd. I thought that was the whole purpose of having a slave builder? 19:10:21 mtaylor and I can take the details of the discussion offline though 19:10:22 jaypipes: yes. a slave builder 19:10:33 jaypipes: slave builders are "dumb". 19:10:36 is there anything at a high level that we need to discuss though? 19:10:37 mtaylor: sorry, that's what I was referring to.. 19:10:39 jaypipes: but a slave builder is not another jenkins ... it's a slave hanging off our our jenkins 19:10:44 I'm dropping off in 5 19:10:45 jaypipes: They're not full Jenkins installs, just workers. 19:10:50 mtaylor: sorry, got my semantics wrong... 19:10:53 soren: got it. 19:10:54 jaypipes: I figured - I just wanted to be clear here because some folks have been getting confused :) 19:11:05 mtaylor: appreciate that :) 19:11:06 * soren included 19:11:08 :) 19:11:17 deshantm: cool - well let's definitely talk offline about getting some useful jobs set up on your slave 19:11:35 * jaypipes replaces slave with builder in his mind... 19:11:38 deshantm: I'm also working with alandman and primeminiterp from msft/novell with their slave 19:11:58 deshantm: so hopefully we can put together a generic/reusable process here 19:14:04 ok sounds good. we may already be testing useful things that can help with automated testing for things like XenAPI support 19:14:13 I'd love that 19:14:16 I'll have to check 19:14:27 #action dshantm and mtaylor will talk jenkins slaves 19:14:44 the end goal is to fill in that hypervisor support matrix automagically 19:14:59 ooh - that _would_ be magical 19:15:04 Yeah. I'm working on that, too. 19:15:07 Differently, though. 19:15:16 ok, i'm off 19:15:19 thanks guys 19:15:20 take care 19:15:28 thanks! 19:15:35 So, I have a stack of tests that exercise all the various things hypervisor drivers are supposed to do. 19:15:38 (unit tests) 19:16:01 Whenever a NotImplementedError is raised, I log that. 19:16:21 ...so when this massive clean-up task is done, that should provide reliable information to fill in such a matrix. 19:16:40 ...and then it's just a matter of extracting the info from these logs and turn it into beautiful html. 19:16:50 mmmm 19:16:59 * mtaylor thinks that sounds wonderful 19:17:10 It's coming along reasonably well. 19:17:11 soren: this may sound like an odd question, but ... 19:17:14 Shoot. 19:17:30 soren: do you actually have information on installed cluster setup prereqs for running those tests? 19:17:43 19:15 < soren> (unit tests) 19:17:51 ah 19:17:54 :) 19:17:57 sorry - missed that 19:18:04 I was about to get wet 19:18:20 and on that note ... 19:18:29 #topic Open Discussion 19:18:52 anybody gots anythings? 19:19:05 I can go into a bit more detail about this stuff I'm doing if anyone wants it? 19:19:30 ..but maybe it's more suitable for an ml post. 19:19:39 (sorry about being late; this meeting is right at lunchtime here.) 19:19:49 Anything new of note? 19:19:53 soren: I'm happy to listen - but also happy to read mailing list post 19:20:01 bengrue: there's an action with your name on it :) 19:20:07 oh? 19:20:28 url? 19:20:28 bengrue: jeblair made the auto-closing pull-request hook thing - so we made you an action for poking at the create-a-gerrit-review bit 19:20:46 where is gerrit installed 19:20:46 ? 19:21:17 it'll be in the meeting notes. gerrit is installed at review.openstack.org ... the hook is at https://github.com/openstack/openstack-ci/blob/master/gerrit/close_pull_requests.py 19:21:24 we have a dev box: review-dev.openstack.org 19:21:43 will I need an account on the dev box? 19:21:49 nope 19:22:06 you can do everything you need to externally 19:22:28 its has a test project set up on it 19:22:32 is the dev box auto-updated on push? 19:22:37 that's linked to https://github.com/gtest-org/test 19:23:04 so what you might do is send pull requests to that project, and then have your script check for them and create gerrit changes 19:23:25 you can submit the changes as your own gerrit user (it syncs with launchpad same as production) 19:24:07 and yes, both dev and production run the latest code out of the github repo 19:24:18 so when you're done, and propose the change in gerrit, when it's approved, it'll go into production 19:24:29 Okay. 19:24:40 lemme know if you need anything 19:24:44 Other than that, were any decisions made/discussed today? 19:24:47 Sure thing. 19:25:43 not really 19:28:28 jeblair: can you think of anything else we should talk with folks about? 19:29:27 nope 19:34:15 So, I have some general questions about CI workflow and openstack. 19:34:43 great! bring em on 19:34:57 What tests are the set of acceptance tests for something to be put into trunk? 19:35:24 acceptance tests being the set of tests that are required to be green. 19:35:52 currently, it's the unit tests in the tree 19:36:08 Because I've been having issues with the smoketests on my dev box, and they seem to be the only ones that actually integrate various subsystems. 19:36:11 I see. 19:36:14 bengrue: for glance, it is the set of unit and functional tests inside glance/tests/ 19:36:16 that is correct 19:36:28 we're currently working on getting smoketests added to the list of things that are acceptance tests 19:36:39 So, are there any full integrstion tests between nova/glance/stance? 19:36:42 ... 19:36:46 swift. 19:36:48 stance, wtf. 19:37:18 hehe. stance 19:38:10 bengrue: also currently in work 19:38:11 Also, what's the level of code coverage? I've been removing whole functions from my dev and the unit tests have still been passing. 19:38:56 Which has been frustrating, because I was removing the functions to see the set of tests that went red to learn about said functions. 19:39:10 bengrue: https://jenkins.openstack.org/view/Nova/job/nova-coverage/ 19:39:24 bengrue: where are you putting your code? 19:39:35 bengrue: is is also an open task to get the code coverage jobs to block coverage decreasing 19:40:20 where am I putting my code? My code is currently local, I didn't commit this stuff. 19:41:12 I'm not the biggest fan of blocking commits if code coverage decreases. 19:41:15 The one that comes to mind is the iscsi discovery code. I made it return at the top with a false to see what tests were covering it. 19:41:19 (none were) 19:41:24 bengrue: ok. there are a number of projects on github that are trying to address integrated testing: https://github.com/cloudbuilders/kong and https://github.com/rackspace-titan/stacktester/ 19:42:04 I'm not a fan of code coverage as a hard metric either, but I was surprised that such systems didnt seem to have tests at all, so I was hoping to get a high level picture of how safe I should feel because of the tests. 19:42:07 jaypipes: Smokestack runs stacktester now. 19:42:20 Also. Mtaylor. I can make nova-vpc run it too if your interested. 19:42:27 dprince: figured as much :) 19:42:28 dprince: please do 19:42:28 I'm working at integrating kong with piston's workflow right now. 19:42:39 Will take a bit of work to make those tests pass for libvirt though. 19:42:41 dprince: blocking based on decreasing coverage was a request from NTT from last ODS 19:42:45 bengrue: could you elaborate on what Piston's workflow is? 19:43:05 Yeah. I was at that session during the conference. Just saying it worries me. 19:43:36 Seems like it motivates people to write tests for coverage which wouldn't always be good tests that we need. 19:43:55 right now, jay, it's being invented! 19:44:01 dprince: soren is working on un-f-king the unit tests and removing the excessive use of stubs... 19:44:13 great! 19:44:40 Still. I hate to block a commit on code coverage as a metric. 19:44:51 bengrue: OK, well I want to make sure that we all work together and that all these new tests get brought under a single umbrella project that everyone can benefit from :) too little communication going on up until now, so there's a myriad projects duplicating efforts.. 19:45:35 jaypipes: a thousand flowers blooming? 19:45:42 I can elaborate more in a few days; I'm currently working on a CI project internally so we can work on our proprietary stuff alongside the open stuff. 19:45:47 dprince: I sent an email to Gabe this morning about trying to get Kong and stacktester merged and everyone working on the same stuff. Sorry, didn't include you because it was more high-level discussion trying to get everyone on same page first... 19:45:57 heckj: indeed, my friend :) 19:47:44 wait - there are flowers? 19:48:01 jaypipes: Sure. As far as test suites go I'm Okay with there being multiple options out there. I sort of see them as weapons. If they work well at finding bugs people will use them. Right now I'm running 3 suites: Ruby Openstack Compute v1.0, Nova Smoke Tests, StackTester v1.1 19:48:33 So essentially all 3 API/versions are getting some coverage on every commit and/or branch we run in SmokeStack. 19:48:43 Haven't heard of Kong but I'll take a look. 19:50:01 I appears to be 5 days old. (Initial release). 19:51:16 ?!? I thought we adjourned this meeting 20 minute ago? 19:51:18 Doh. 19:51:38 soren: whatever gave you that idea? 19:51:49 Sorry. I was late. And people were still talking. So... 19:53:25 mtaylor: On another topic. The nova tarmac job is getting hung every couple days. 19:53:59 If I see its been running for more than 10 hours in the morning I kill it, a new one starts, and code gets merged again. 19:54:18 dprince: yeah - I've been seeing that. have I mentioned how much I can wait to move from tarmac to gerrit so that we can actually see _where_ it's getting hung? 19:54:35 s/can/can't/ 19:54:42 Sure. Just trying to keep things moving along until we make the switch. 19:55:17 mtaylor: the 5 minutes of silence, perhaps? :) 19:55:32 dprince: thank you very much! if you kill the job, probably looking at the console output of the job for the last branch attempted to merge and setting that to work in progress with a note about hanging would prevent the re-hang (I would hope) 19:56:13 I've done that a couple times. Sometimes the branch it hangs on appears to already have merged though. Its wierd. 19:56:24 very weird 19:56:32 I will very much look forward to tracking that down better 19:57:09 Also. I've been gone.... Last week or a couple of weeks ago I saw you made a comment about the nova-vpc job being red a lot. The test_004_metadata test had some issues (I think those have been fixed now). 19:57:39 Anyway. The job is red today... because of another nova bug. So I think it is working pretty well. 19:57:58 The nova smoke tests still need a bit of tuning IMHO. 19:58:24 ++ 19:58:40 2min left 19:58:52 dprince: well, the question is - once we can do this (post gerrit) - do you think that nova-vpc is solid enough to add to the pre-merge blockers? 19:58:58 We may want to wait a bit longer for an instance to boot. Adding an option to increase the instance boot timeout would do the trick. But its getting to the point where I would consider gating commits on a job like this possible. 19:59:02 dprince: or do you think it shoudl stay post-merge 19:59:07 Its close. 19:59:11 dprince: cool 19:59:26 well - we're at least 3 weeks out from being able to make that choice anyway - so sweet 20:00:02 alright - I think we're out of time here 20:00:09 thanks everybody! 20:00:11 #endmeeting