19:08:45 #startmeeting ci 19:08:46 Meeting started Tue Oct 23 19:08:45 2012 UTC. The chair is jeblair. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:08:47 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:08:48 The meeting name has been set to 'ci' 19:09:53 #topic actions from last meeting 19:10:31 fungi: we're still waiting on the foundation server for the CLA thing, yeah? 19:11:14 yes, toddmorey e-mailed me last week to say that he has something mostly working but was too busy to meet up with the summit going on 19:11:37 so i'm waiting to hear back from him now that things are hopefully cooling down again 19:11:42 #action toddmorey provide a test foundation server 19:12:39 #topic summit 19:12:52 So we were all pretty busy at the summit 19:12:57 #link https://etherpad.openstack.org/state-of-ci 19:13:16 wait, we were all at the summit? 19:13:18 clarkb: Thanks for the link 19:13:20 Monty is going to translate those items into bugs 19:13:23 I am 19:13:26 I'm so happy about it 19:13:29 #action mordred bugify summit actions 19:13:36 if people could go through other todo list items they got from the summit 19:13:44 and add them to that (or just make bugs from them) 19:13:47 I'd appreciate it 19:14:03 I'm also going through the current grizzly list and getting rid of or re-prioritizing cruft 19:14:18 #action everyone collect action items from other summit session etherpads and register as bugs 19:14:49 look at that 1 commit behind all of redhat for CI commits 19:14:50 woot 19:14:56 russellb: ^ 19:15:13 pabelanger: nice showing! 19:15:54 any particular summit-related topics we should discuss here? 19:16:31 i've renewed work on the continuously updating tags for branches on gerrit 19:16:41 based on (brief) discussions at the summit 19:16:57 more or less people agreeing with what jeblair and i already talked through 19:17:05 jeblair: the idea for a hosted log server might be worth bringing up 19:17:23 fungi: +1, and i saw you updated the bug 19:17:23 clarkb: YES PLEASE 19:17:41 clarkb: yeah, why don't you give an overview? 19:18:44 everyone is testing openstack, but other than smokestack and us no one is reporting test results. Apparently companies make it hard to host their own results, but it was suggested that if we could provide a place to put logs people could just upload them there 19:19:00 I'd be interested in getting more information about salt and puppet from the summit. If somebody has a URL / presentation handy 19:19:54 also, we really need to make logs.o.o more easily consumed. The potential is there to fix these two problems with one handy log server setup 19:20:26 currently I am looking at logstash because it seems fairly configurable and there aer puppet modules to make it go (also their logo is awesome) 19:20:27 ahh, i remember the testlog server discussions now. i was confusing that with the proposal for a central logserver for our infrastructure 19:20:57 fungi: perhaps those needs overlap a bit. 19:21:04 jeblair: ++ 19:21:17 jeblair: potentially 19:21:39 if anyone has had experience with logstash or similar I am definitely interested in chatting 19:22:01 last time i set up logservers it was rsyslogd and tenshi 19:22:02 pabelanger: nothing's coming to mind. i'm sure lots of people were talking about those things, but i wasn't there. 19:22:39 pabelanger: other than some nice conversations we had with Ryan_Lane, but i think you know most of that. 19:22:56 pabelanger: my only recollection of salt/puppet getting mentioned from a ci perspective was in the "state of ci" session 19:23:16 yeah - the only thing is what we've talked about in channel 19:23:29 that being, we'd like to start using salt to orchestrate our puppet runs 19:23:50 so that we can make zuul trigger salt to immediately make changes on merge, rather than waiting for a double cron 19:24:04 jeblair: cool. 19:24:11 fungi: /nods 19:25:07 objections to moving on to server status/upgrades? 19:25:21 seconded 19:25:32 pabelanger: the only slides I can find are from a demonstration Dan Bode gave: http://www.slideshare.net/bodepd/openstack-grizzley-puppettalk 19:25:32 #topic server upgrades 19:25:58 So, we upgraded servers! 19:26:11 we did! it was awesome and we got 99% of it all working 19:26:20 olaph: Ooo, checking it out. Thanks 19:26:41 the gerrit and jenkins master servers (all four) are now on rackspace nova 19:26:45 jeblair: clarkb congrats 19:26:49 and have been upgraded to precise 19:26:54 wq00t! 19:27:19 as is etherpad.o.o, which was also migrated from a sekrit rackspace server to one publicly managed by the infra team 19:27:30 at this point there are no known regressions, right? just some new features we can't implement quite yet 19:27:30 in theory this also gives us ipv6 connectivity but openssh server apparently tickles a bug in ovs so as a precaution we have not created AAAA records for the two production servers yet 19:28:03 fungi: yup. I haven't heard any screaming. jgriffith did lose a pad but the link had the wrong url 19:28:19 saw that. didn't sound like a bug, just a typo 19:28:40 yup 19:29:02 and the new etherpad hasn't melted down yet which is good 19:29:19 aside from the usual jenkins bugginess, since reverted 19:29:19 clarkb: and we have system-level monitoring of it with cacti! 19:29:41 * fungi is very excited about cacti 19:29:44 #link http://cacti.openstack.org/cacti/graph_view.php 19:30:38 btw, for ram, i just pulled a number out of a hat when i made the etherpad host. it said 4g. 19:30:51 it looks like that was generous. 19:30:59 is cacti running on a dedicated vm? 19:31:04 fungi: yes 19:31:21 so if we also wanted to throw nagios or something on it down the road, that would probably be doable 19:31:44 fungi: yep 19:31:49 spiffy 19:32:25 #topic grenade / quantum 19:32:27 fungi: shouldn't be too hard. I'd be able to help with that. Use it with our asterisk instances 19:32:50 pabelanger: cool. used it a ton myself over the years (long before it changed its name) 19:33:06 two devstack variants have been requested: grenade upgrade testing, and quantum. 19:33:27 i'm very close to being able to run grenade (next step: move the grenade project into openstack-dev) 19:33:57 and i have set up a silent job for quantum so that nachi can finish getting that working 19:34:59 on a related note, we should find out if people are okay with us not running n-vol tests for master, since n-vol and cinder are in folsom, i think n-vol may be deprecated now? 19:35:09 jgriffith, vishy: ^ do you have an opinion on that? 19:35:45 anotherjesse said that we could go ahead and till that 19:35:46 kill 19:36:05 but continue running them for stable/folsom and stable/essex updates right? 19:36:40 jeblair: I'm ok with it (I think) 19:36:47 fungi: i reckon so. 19:36:57 jeblair: we shouldn't be making changes to nova-vol any longer so... 19:37:31 #action jeblair deconfigure n-vol testing on master 19:38:09 #topic testr and friends 19:38:35 #link https://review.openstack.org/#/c/14611/ 19:39:18 I figured we needed to do a small proof of concept for testr and friends on a project that would be easy enough that this whole thing won't lose traction 19:39:19 testr and testtools and fixtures are super cool 19:39:41 so with lifeless' help I hacked the above change for python-novaclient to use testr instead of nose 19:39:58 clarkb: good plan 19:40:02 supersexy 19:40:14 o/ 19:40:23 lifeless: we like your libraries 19:40:45 lifeless is an awesome librarian 19:40:52 I had to switch from unittest2 to testtools, update the base test class to capture stdout and save it in the subunit stream, correct setUp and tearDown methods (need to call parent class methods) 19:41:08 and some boilerplate config stuff in tox.ini and testr.conf 19:41:36 The current things we are missing are coverage, realtime output of test results, and human readable test logs 19:41:39 clarkb: the changes to do testtools and setUp/tearDown should be landable without testr yeah? 19:42:00 clarkb: technically you get realtime output 19:42:08 clarkb: its just less IN YOUR FACE 19:42:19 mordred: yes, I think I should pull those out into another change and rebase things 19:42:25 lifeless: s/realtime/IN YOUR FACE ANSI COLOR realtime output/ 19:42:40 lifeless: what realtime output is there currently? 19:42:48 jeblair: it outputs errors as they happen. 19:42:55 and failures 19:43:02 jeblair: it does what standard nose/unittest runs do without the openstack nose plugin 19:43:03 all classes of things going wrong. 19:43:12 jeblair: ....F...E... etc 19:43:18 mordred: nope 19:43:23 mordred: it doesn't 19:43:25 ok 19:43:37 I thought I saw .'s yesterday 19:43:55 jeblair: it outputs anything outcome that will make the run be considered not-ok, as and when it happens. 19:43:59 with all its attachments etc. 19:44:15 so what other requirements are there driving the desire to implement something else? 19:44:38 i'm not sure blindly re-implementing openstack-nose-plugin without targeting specific goals is the best idea. 19:44:51 prior experience I had trying to get us to plain nose before 19:45:25 the people who have approval access have keeping that output as a requirement before they'd merge the change 19:45:38 it's possible that we can convince them that the new features outweight the loss 19:45:45 mordred: yeah, but i seem to recall one specific objection was, eg, outputting either a "." or a test description rather than the test name. 19:46:19 mordred: so since testr-default-output != plain-nose-default-output, we should re-evaluate. :) 19:46:43 indeed ... and I think super-quick test runs might please people ... but I also think that writing an ansi color filter for subunit output shouldn't be terrible 19:46:47 HOWEVER, you make a good point 19:47:03 so when we have the rest of everything working, we'll have a chat with people to determine whether or not the color thing is important to them 19:47:24 i think identification of slow tests may have been a use-case too. 19:47:29 indeed 19:47:30 testr slowest 19:47:33 for the other two issues: we can potentially get away with nose for coverage and subunit logs as the log files in the short term 19:47:34 (and that one in particular driving the color thing) 19:47:35 just run that :) 19:48:27 yeah, so lets see what the actual issues are, and perhaps there might be a more intelligent way to solve them other than expecting nova devs to stare at a screen identifying colors for 11 minutes while 3800 tests run. 19:48:31 yeah ... the more we have to teach them how to run something new ... 19:48:34 vishy: ping 19:49:02 mordred: we can tack on things like "testr slowest" in the tox config, right? 19:49:19 jeblair: yes, however tox short circuits on failure 19:49:21 vishy: if we could reduce test run time by 10x or more and still had sensible output for slow things and failures, but lost the current color output formatting, would that be a non-starter? 19:50:12 jeblair: so we may need a lightweight wrapper like 'bash -c 'testr run --parallel ; testr slowest' 19:50:24 clarkb: you probably want pipefail there 19:50:40 clarkb: or perhaps something a little more sophisticated - preserve $? from testr run --parallel 19:50:40 danwent, around? 19:50:42 clarkb: _if_ people care about slow test runs on failure (they might?) 19:50:47 lifeless: good point 19:50:59 jeblair: I think they should care on each run 19:51:09 there is also the possibility that if hte system is sane and all, we can just show people how to do stuff with testr directly ... 19:51:20 I just don't want to get bogged down in perception of regression 19:51:28 the tl;dr is yes we can add it to tox. it will just be a little ugly 19:51:53 however you don't see that ugly when running tox -efoo 19:52:19 koolhead17: what's up? 19:53:07 danwent, i changed the doc repo to released ubuntu cloud repo, it was testing at time guide was written. 19:53:25 koolhead17: we're having a ci meeting in this channel 19:53:44 jeblair, srry :( 19:54:14 anyone else have something to say about testr? 19:54:39 nope 19:54:42 #topic open discussion 19:54:46 5 mins left! 19:54:53 any bacon shortage updates? 19:55:01 I currently have no bacon 19:55:09 that sounds like a bug 19:55:33 i am fresh out. but i will be eating more iberian ham soon, which makes me happy. 19:55:54 * clarkb is going to try and take the afternoon off if mordred doesn't scream about it 19:56:01 i hope to have the commit-merged hook module for gerrit to update tags on review-dev checked in for review later this afternoon 19:56:15 sweet. 19:56:20 then you can all kick it around 19:56:36 i'm playing around with graphite for collecting and displaying stats for devstack jobs (and maybe later, all jenkins jobs). 19:57:07 i'm changing commit-merged into a .erb though so we can pass vars into it. hopefully that doesn't meet with concern 19:57:41 fungi: that should be fine in principle. 19:57:44 since it will technically affect both review and review-dev from that perspective 19:58:05 oh, and jenkins-job-builder is getting more contributors! 19:58:15 saw that. very exciting 19:58:26 loops! 19:58:44 fungi: and loops and loops and loops... 19:59:04 heh 19:59:16 #endmeeting