19:11:24 <jeblair> #startmeeting ci
19:11:25 <openstack> Meeting started Tue Dec  4 19:11:24 2012 UTC.  The chair is jeblair. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:11:27 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:11:29 <openstack> The meeting name has been set to 'ci'
19:11:59 <jeblair> #topic actions from last meeting
19:12:10 <jeblair> fungi: any news on the foundation server front?
19:12:28 <fungi> nothing new from toddmorey on his piece
19:12:51 <fungi> i started digging into the documentation updates and putting together a list, not in etherpad yet so nothing to link
19:13:10 <fungi> bug update script change proposed as requested: #link https://review.openstack.org/#/c/17488/
19:13:20 <jeblair> mordred: you were working with todd on publishing the source for the site, right?
19:13:29 <mordred> yes. and I have not heard an update from him on that
19:13:52 <fungi> i saw nothing in the bp update script which needed adjustment, but i could be misunderstanding what we wanted there
19:14:04 <jeblair> mordred: would you like to pester him?
19:14:38 <jeblair> fungi: yeah, i don't think the bp script needs updating for this
19:14:48 <fungi> groovy
19:15:09 <clarkb> I checked the bp script too and that was my understadning
19:15:32 <fungi> thanks clarkb!
19:15:35 <jeblair> #action jeblair propose a system for linking revierifies to bugs
19:15:39 <jeblair> that still needs to happen
19:16:01 <jeblair> #action mordred work with toddmorey on foundation server/code
19:16:31 <jeblair> #topic testrepository
19:16:35 <jeblair> clarkb: how's that going?
19:16:49 <clarkb> clarkb: slower than I would like, but it seems to be getting traction
19:16:53 <clarkb> er
19:17:01 <clarkb> ya ignore the fact that I addressed that to myself
19:17:05 <mordred> never
19:17:07 <clarkb> I have parallel coverage working
19:17:30 <clarkb> and got four changes to nova merged that fixed broken tests
19:18:31 <clarkb> there are more broken tests that fail when run in parallel with testr. Current major issues are concurrent DB access and tests not sharing ip ports nicely
19:19:04 <jeblair> clarkb: also, do we want to look into having jenkins randomize the order when it runs tests?
19:19:42 <clarkb> I like the idea, but am beginning to think that getting all tests to pass if that is done will be a major headache
19:20:01 <jeblair> clarkb: it sounded nice, except since it will create more failures, i think an ability to reproduce the exact sequence is necessary in order to fix those bugs
19:20:13 <mordred> ++
19:20:22 <jeblair> anyway, we can certainly defer it for now
19:20:29 <clarkb> yes, and it is possible to reproduce the sequence, but not necessarily the run time of each test
19:20:43 <clarkb> makes reproducing contention difficult
19:21:05 * mordred is still really unclear as to how any of these are doing concurrent db access
19:21:24 <clarkb> mordred: it is possible that something more devious is happening there
19:21:39 <clarkb> like maybe the base DB file is being updated or something
19:22:01 <mordred> clarkb: we do not use file-based sqlite
19:22:08 <mordred> whoever has been saying that we do is quite mistaken
19:22:20 <clarkb> mordred: I htought the in memory DB is populated from a file though
19:22:24 <mordred> nope
19:22:38 <clarkb> ok, I think that means there is a different DB related bug
19:22:41 <mordred> the in-memory db is first created by running through the migrations on an empty db
19:22:50 <mordred> then the results of that are saved into a sql script
19:23:05 <mordred> which is then run on an empty db in the setup of each test case
19:23:22 <clarkb> mordred: actually maybe that is the cause of the bug. since the test that is failing runs through the migrations and is finding that a column already exists
19:23:38 <clarkb> (we may need to special case that test and have it tear down the DB in a way that it accepts)
19:24:13 <clarkb> long story short, moving test fixtures to proper fixture objects has been fixing a lot of problems
19:24:38 <clarkb> we need to do the same for the DB (which mordred did) and anything that listens on sockets will need to be handled that way as well to avoid port conflicts
19:25:15 <jeblair> clarkb: any special considerations for jenkins we should be thinking about?
19:25:47 <clarkb> jeblair: eventually there will be some cleanup needed to remove the 0 tests check
19:25:55 <clarkb> also, we will need to copy different results files
19:26:02 <jeblair> clarkb: doesn't testr store data between runs (test run times)?  where? and should we try to preserve that? or make sure it's not preserved?
19:26:33 <clarkb> jeblair: it does store it in the .testrepository dir as a sequence of numbered files.
19:26:49 <clarkb> saving that data would be useful if we need to improve test run times even more
19:27:07 <clarkb> (that log data is used by testr to partition tests)
19:27:11 <jeblair> okay, so if we do nothing, it will be wiped out by git-clean, which has the advantage of determinism...
19:27:16 <clarkb> correct
19:27:27 <jeblair> though it may be less fast.
19:28:11 <clarkb> we will also want to treat the subunit stream as a log we care about
19:28:18 <jeblair> okay, we'll doing nothing and letting it get wiped out is probably the best first step.  we'd need some extra engineering to preserve that.
19:28:25 <jeblair> ok
19:28:29 <clarkb> (that is the source of test sequencing and stdout/stderr/log)
19:29:17 <clarkb> there was a suggestion to run a non voting testr test against nova to help reviewers see the broken tests
19:29:27 <jeblair> so it sounds like those things can all be merged after the testr change(s) land(s)
19:29:35 <mordred> yah
19:29:43 <clarkb> if we do that I think it might be most appropriate to install testr and dependencies on the systems and not in the venv
19:30:01 <clarkb> jeblair: correct
19:30:45 <jeblair> clarkb: do you think running a non-voting job would be helpful?
19:30:48 <clarkb> but the overall work should be minimal https://review.openstack.org/#/c/15078/ runs under testr just fine when it can merge
19:31:25 <clarkb> jeblair: I think its usefullness is really dependent on buy in from code reviewers
19:31:45 <clarkb> if that info gets vishy et al into the mindset of fixing those errors then yes, it is worth it
19:32:37 <clarkb> mordred: maybe you and I should drop into the next nova meeting and discuss this
19:32:37 <jeblair> clarkb: why do you think testrepository should be installed on the system?
19:32:58 <clarkb> jeblair: because I think there will be resistance adding it as a test-requires before tox/run_tests actually makes use of it
19:33:24 <clarkb> though now that you have me thinking about it we can just add a new tox env and run it out of there
19:33:32 <clarkb> (with its own special deps list)
19:33:48 <jeblair> yeah, i think that's the way to go -- better migration path
19:34:00 <clarkb> yup. I retract my statement about system wide installs of testr
19:35:48 <jeblair> clarkb: i am skeptical that having a non-voting job will focus attention.
19:35:58 <clarkb> me too
19:36:23 <clarkb> will probably have better luck bringnig up particularly hairy failures to nova directly and ask for help that way
19:36:57 <jeblair> clarkb: yeah; and your idea of raising the issue in the nova meeting seems like a good one.
19:37:03 <clarkb> however, they want reproduceability, so adding a special tox venv may be useful even if we don't have a non voting test
19:37:17 <jeblair> true
19:38:01 <jeblair> clarkb: end of topic?
19:38:07 <clarkb> I think so
19:38:08 <mordred> clarkb also, the new env should help give them a way to try the new system out
19:38:20 <jeblair> #topic grenade/quantum
19:38:43 <jeblair> these are languishing.  I pinged dtroyer about grenade and haven't heard back.
19:38:54 <jeblair> progress on quantum seems to be slow, but nachi is still working on it.
19:39:24 <jeblair> i can probably merge my grenade-running change...
19:39:28 <jeblair> set up a job that runs it...
19:39:42 <jeblair> and then point dtroyer at that.
19:40:28 <clarkb> ++
19:40:56 <jeblair> #topy pypi uploads
19:40:59 <jeblair> #topic pypi uploads
19:41:14 <jeblair> clarkb: this is in place for all pypi-uploading projects now?
19:41:17 <fungi> topy sounds like the name of our next pipy project
19:41:27 <clarkb> jeblair: not yet
19:41:31 <clarkb> #link https://review.openstack.org/#/c/17417/
19:41:41 <clarkb> that change needs approval then it will be in place for all projects
19:41:55 <jeblair> fungi, mordred: ^ ! :)
19:42:05 <clarkb> the new upload process is working for zuul, gerritlib, and JJB so I feel fairly confident putting it in place across the baord
19:42:08 <fungi> just pulled up a tab for that one ;)
19:42:28 <jeblair> clarkb: cool!  thanks for doing that!
19:42:44 <fungi> i'll review it as soon as the meeting wraps
19:42:51 <jeblair> #topic git-review
19:43:14 <jeblair> so we have released a tested and working version of git-review...
19:43:29 <jeblair> there are a bunch of changes for it in gerrit that aren't getting a lot of attention...
19:43:39 <mordred> need test suite
19:44:16 <jeblair> mordred: do you want to hold off reviewing changes until there is a test suite?
19:44:25 <mordred> jeblair: well, I'm really torn on that, honestly
19:44:36 <jeblair> i've volunteered to do release-time manual testing...
19:44:37 <clarkb> I think for some of the changes it depends.
19:44:51 <jeblair> so we can at least make progress and release known-working things...
19:44:55 <mordred> ah - ok. that might be a short-term solution
19:45:22 <clarkb> I think refactoring type stuff should wait, but things like https://review.openstack.org/#/c/16128/ which fix bugs might be valuable even with manual testing
19:45:22 <jeblair> i think i can review some changes to git-review as well...
19:45:32 <fungi> the slew of exit code changes are the ones languishing most, and i think those drive back to an overall design question
19:46:17 <jeblair> yeah, i think the project could use a lead to make some decisions about the direction...
19:46:46 <jeblair> especially since there might even be a few incompatible changes in the queue?
19:47:24 <mordred> yah.
19:47:33 <mordred> and I think that most of us think it's mostly fine as is right now
19:48:23 <jeblair> yeah, i'm personally not inclined to change it much, so i'm a bad choice to help git-review through the current situation...
19:48:36 <jeblair> my head is also full of other things.
19:48:51 <jeblair> does anyone want to volunteer to help steer it?
19:49:28 <jeblair> i suspect that if we do nothing, it'll end up getting forked.
19:49:49 <olaph> i can volunteer to help whoever /is/ steering... :)
19:50:09 <fungi> olaph: we can team up on that i guess
19:50:30 <fungi> i'll try to dig through the existing open changes and wishlist bugs
19:50:42 <clarkb> I know I have already reviewed some of the design changes and disagreed with them on gerrit. I can take that back up again, but I think I will try to maintain that stance
19:50:45 <clarkb> which may not be helpful
19:50:59 <jeblair> clarkb: that is helpful.  having any opinion is helpful i think.
19:51:18 <fungi> having not been around for its inception, i don't want to step on anyone's toes with regard to direction of the tool
19:51:46 <jeblair> fungi: i only have one requirement: "git review" on a newly committed change to a newly cloned repo should "just work" with no interaction.  :)
19:51:53 <jeblair> i don't really care what else it does.
19:52:29 <fungi> i agree with jeblair though on the having opinions part. overcomplicating its design to make it a more useful backend tool is going in the wrong direction in my opinion
19:52:45 <fungi> so i think you were spot on with that, clarkb
19:52:59 <clarkb> sounds like we have a driver ^ :)
19:53:04 <jeblair> yep
19:53:16 <fungi> i'll see what headway i can make this week
19:53:24 <jeblair> #action fungi steer git-review development
19:53:26 <clarkb> awesome. I will gladly rereview things
19:53:31 <fungi> i'm still a little fuzzy on how to automate testing for it
19:53:36 <jeblair> fungi: thanks, and me too.
19:53:39 <fungi> the list jeblair put together is a great start
19:54:03 <fungi> but we can take that discussion back to -infra
19:54:17 <clarkb> fungi: I think that if you setup git repo fixtures and use GitPython you can come up with a reasonable test suite
19:55:16 <jeblair> fungi: i also don't think you have to write the test suite.  it's definitely not our highest priority.  i mostly just don't like the idea of changes sitting in gerrit but no project direction.
19:55:36 <fungi> jeblair: right. i saw that as to complimentary tasks
19:55:40 <jeblair> k
19:55:40 <fungi> er, two
19:56:08 <jeblair> #topic new-project howto
19:56:12 <jeblair> clarkb: ^?
19:56:34 <clarkb> #link http://ci.openstack.org/stackforge.html
19:56:55 <jeblair> yay!
19:57:22 <clarkb> I think that is mostly done. The generic this is how it works documentation has been in place since we added project management through puppet, but we were missing the how to do stackforge from scratch
19:57:47 <clarkb> The new stackforge doc should be fairly complete and comes with a blurb explaining the purpose of stackforge
19:58:24 <jeblair> clarkb: that's great!
19:58:53 <jeblair> and that's time.  thanks!
19:58:58 <jeblair> #endmeeting