20:17:14 <jbryce> #startmeeting
20:17:15 <openstack> Meeting started Tue Nov 15 20:17:14 2011 UTC.  The chair is jbryce. Information about MeetBot at http://wiki.debian.org/MeetBot.
20:17:16 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic.
20:17:24 <mtaylor> anotherjesse: but once we get this first one up - I really want to start spending more time on the plan of getting better/fuller hardware/environment coverage from vendors, which we've got good ideas on doing
20:17:41 <anotherjesse> nice
20:17:42 <jbryce> my hard drive started rattling like a pair of castanets. i think that's a bad sign. yay for backups, though
20:17:52 <anotherjesse> jbryce: backup to the cloud
20:17:59 <mtaylor> jbryce: unless you like castanets
20:18:08 <jbryce> mtaylor: true...
20:18:10 <jbryce> http://wiki.openstack.org/Governance/PPB - agenda
20:18:20 <jbryce> #topic Foundation update
20:18:41 <jbryce> thierry asked about an update on foundation progress since things have been quiet on the mailing list
20:19:31 <jbryce> since we made the announcement, we've seen a lot of interest from a lot of different people. we've been meeting with different companies and foundations and getting feedback and trying to get as many of them as possible to join up on the mailing list also
20:20:17 <jbryce> one consistent piece of feedback that we've gotten is that we should come forward with a drafting process and timeline for putting it all together, so that's something we've been working on with several lawyers from rackspace and elsewhere
20:20:53 <ttx> jbryce: should we encourage the presentation of existing models on the ML ?
20:21:37 <ttx> (copying an existing model is a good way to reduce the base work)
20:21:54 <ttx> jbryce: or are your private discussions already past that ?
20:22:18 <jbryce> mark or i will be sending out a more detailed email to the foundation list but the basic process is to publish a draft of the high level goals and philosophies and get feedback on that over the rest of November, then move into an actual legal document drafting process. that will probably consist of 2 drafting periods where we publish a draft and accept comments over the course of several weeks and the re-draft
20:23:11 <jbryce> ttx: discussing existing models is definitely something we can do on the list
20:23:14 <ttx> jbryce: do you want the "goals and philosophies" defined first, before the actual governance and foundation model ? Or in parallel ?
20:23:49 <jbryce> i think it makes sense to get the high level done first as it will affect the details of the legal structure
20:23:55 <ttx> (because the existing models are obviously more about governance structure than about goals/scope/philosophy)
20:24:09 <jbryce> true
20:24:43 <ttx> so imho that's two separate discussions, dunno if you want to have them in parallel or define the goals first
20:25:01 <jbryce> i think i'd prefer goals first
20:25:35 <ttx> jbryce: ok, would be good to send out some progress note saying the first step is to discuss/define goals/philosophy/scope
20:25:39 <jbryce> in summary, there's been a small amount of progress in actual documentation, but quite a bit of input and feedback that has been offered from a variety of source that we're trying to aggregate and publish
20:26:21 <ttx> (so that the discussion can focus on that first)
20:26:44 <jbryce> good feedback
20:27:21 <jbryce> anything else? question or suggestions?
20:28:09 <jbryce> ok. moving on
20:28:17 <jbryce> #topic Quality for Essex
20:28:24 <mtaylor> so...
20:28:39 <mtaylor> first of all: http://eavesdrop.openstack.org/meetings/openstack-meeting/2011/openstack-meeting.2011-11-15-19.00.log.html
20:28:41 <ttx> +1
20:29:03 <mtaylor> we had a great summary of where we were on this during the CI meeting in the last hour
20:29:52 <mtaylor> as a much briefer summary - current status is that jeblair has the infrastructure/integration pieces working to launch fresh cloud servers, run devstack on them and then run exercise.sh against those
20:30:36 <mtaylor> currently immediately outstanding todo items to get those tests up and going are getting devstack working against trunk (which vishy apparently already has done) and getting devstack and python-novaclient moved over to gerrit
20:31:07 <anotherjesse> mtaylor: also devstack has a config.ini generator
20:31:08 <mtaylor> (we actually don't need trunk working before we start - those are just the immediate tasks)
20:31:17 <anotherjesse> so you can run the openstack-integration-tests against them
20:31:21 <anotherjesse> (we do that on our jenkins)
20:31:24 <mtaylor> great
20:31:55 <anotherjesse> mtaylor: was there talk about how to run larger scale tests (launch vms non-stop for a week, ...)
20:32:01 <mtaylor> that's the next step after the above, is once we're satisfied that exercise.sh can run consistently, we'll start adding openstack-integration-tests
20:32:47 <jaypipes> why are we using 2 jenkins again?
20:32:51 <mtaylor> anotherjesse: at the moment I think running tests of that nature are going to need folks to pony up labs in which to run them in
20:33:01 <anotherjesse> jaypipes: we push everything we do back to mtaylor / jeblair
20:33:02 <jbryce> how much of the core functionality do you think will be covered by the time we start getting to the end of essex?
20:33:03 <mtaylor> jaypipes: anotherjesse has been helping prototype devstack jenkins stuff
20:33:08 <anotherjesse> jaypipes: should we stop?
20:33:17 <anotherjesse> we don't want to break the core jenkins server
20:33:20 <jaypipes> just dont understand why there is a need for 2 jenkins...
20:33:36 <anotherjesse> jaypipes: everyone else runs their own jenkins - is it bad for us to as well?
20:33:54 <mtaylor> jaypipes: too many cooks in the kitchen - basically that way anotherjesse can do things and show them to us
20:33:55 <jbryce> #info will need better lab environments to do larger scale and longer-running tests
20:34:06 <jaypipes> anotherjesse: the direction should be the central jenkins out to secondary jenkins, not the other way around.
20:34:12 <anotherjesse> mtaylor: getting the integration tests running and alerting of basic failures is great, but then there is things like the rax cloud hit in early essex - after many vm launches it various api commands got really slow
20:34:16 <mtaylor> jaypipes: dude. totally different thing
20:34:25 <jaypipes> mtaylor: no, it's not.
20:34:28 <mtaylor> jaypipes: it is
20:34:36 <mtaylor> anotherjesse's jenkins is a dev jenkins
20:34:51 <jaypipes> mtaylor: and what is the point of that?
20:34:54 <mtaylor> it's a sandbox where he can play with different jobs and not worry about breaking the core element of the entire dev process
20:35:19 <mtaylor> doing tons of dev directly on the core jenkins is, well - if things break, then things BREAK, right?
20:35:27 <jbryce> it's staging vs production...don't break the live one that gates the development process for everyone
20:35:41 <anotherjesse> jaypipes / mtaylor we can move this talk about ways we can help the ci process externally
20:36:00 <anotherjesse> since we had asked mtaylor / jeblair at the last summit and we thought it was a good idea
20:36:08 <anotherjesse> move this talk to another time - outside pbb
20:36:13 <jaypipes> sure
20:36:18 <mtaylor> yeah. perhaps next week CI?
20:36:33 <anotherjesse> mtaylor: ya - setting alarm now - didn't update time
20:36:39 <jbryce> so the process sounds great, but can i re-ask my question? = )
20:36:41 <jbryce> how much of the core functionality do you think will be covered by the time we start getting to the end of essex?
20:36:52 <mtaylor> jbryce: that's actually a question that's out of my scope
20:36:59 <anotherjesse> jbryce: I worry about testing large scale VMs launches
20:37:15 <anotherjesse> but - there is lots of parties (hp, rax and others) talking about it
20:37:16 <mtaylor> jbryce: my main bit is making sure stuff gets run consistently - the integration tests guys will need to get good coverage in their work
20:37:19 <anotherjesse> we have a great plan for the general stuff
20:37:24 <mtaylor> and then the thing anotherjesse is mentioning
20:37:43 <mtaylor> because honestly the jenkins control-center model isn't going to be terribly useful for week-long tets
20:37:45 <mtaylor> tests
20:37:46 <jbryce> mtaylor: that's fine. not saying it's your scope, but i'd love to know that someone is thinking about it.
20:37:50 <mtaylor> totally
20:38:03 <ttx> do we know how the integration-tests development is going so far ?
20:38:30 <mtaylor> I think it's going to wind up being a discussion between folks like rax, dell and hp - folks who have the resources to spin up a lab somewhere that can run such a test
20:38:34 <anotherjesse> ttx: bcwaldon had gotten the basics working
20:38:35 <jaypipes> mtaylor: to be blunt, it's pointless trying to design large scale performance testing of Nova until a deployment Jenkins job that uses real deployment things like chef/puppet is completed.
20:38:40 <jbryce> and also find out if we should go really prod some people to work on this as a form of the strategic contributions that ttx has talked about
20:38:53 <mtaylor> and then we have to figure out how to collaborate on that between us
20:38:56 <ttx> jbryce: the teams are up and are open
20:38:59 <mtaylor> and how to report back findings in a sensible manner
20:39:04 <ttx> anyone can join and help
20:39:18 <mtaylor> jaypipes: again, I disagree - I think we can design the process modularly
20:39:24 <anotherjesse> mtaylor: agreed
20:39:32 <mtaylor> jaypipes: the long-term test is just something that needs to run against a cloud
20:39:36 <anotherjesse> mtaylor: the process of deploy is independant of the tests
20:39:39 <mtaylor> how that cloud got there is a different engineering problem to solve
20:39:41 <jaypipes> mtaylor: ok, I'll reserve (any more) judgment until I see that then
20:39:42 <anotherjesse> yes
20:40:19 <mtaylor> and in this case
20:40:51 <mtaylor> even though we're actually doing a good job along the lines of automated deployment of cloud onto bare metal from jenkins- that work specifically isn't going to be as directly applicable to a week-long test
20:41:03 <jbryce> i know that the teams are open and anyone can help, but if it's not happening and we get to the end of essex and these tests are not helping us have a much better release, i think that's a big failure on our part.
20:41:04 <mtaylor> because I don't want a week-long test taking up threads in my crazy java app called jenkins :)
20:41:22 <jaypipes> jbryce: couldn't agree more.
20:41:34 <ttx> jbryce: that's why I was asking on progress on the integration-tests side -- see if more resources are needed there and a call for help is warranted
20:41:38 <mtaylor> jbryce: I would say that if we don't have integration tests running by essex-2 we should all book time in a hotel somewhere, get together and have a celebrity death match
20:41:56 <mtaylor> integration-tests is a little chicken-and-egg at the moment
20:42:19 <jaypipes> mtaylor: the point is that you won't be taking up threads on *your* jenkins. The tests will be running on large HP (or other) clusters. But those clusters absolutely are a no-go until deployment jobs that use real ddeployment methods are done.
20:42:21 <mtaylor> as soon as we get this next jenkins job deployed, then we'll have somewhere we can point their work at so that they can get consistent feedback on their tests running
20:42:23 <anotherjesse> mtaylor: really? we've been running them in jenkins for a couple weeks  - will ping you after
20:42:27 <ttx> mtaylor: the devil is in the number of those tests, not their simple existence.
20:42:37 <mtaylor> anotherjesse: I meant dev-process-wise
20:42:46 <anotherjesse> mtaylor: ?
20:42:53 <anotherjesse> oh - in gating them
20:42:59 <anotherjesse> yes - understood now
20:43:00 <mtaylor> anotherjesse: yes
20:43:16 <jaypipes> ttx: for integration tests, dwalleck, westmaas, wwkeyboard, bcwaldon, nati, and a number of others have been contributing to them.
20:43:26 <ttx> jaypipes: is more help needed ?
20:43:47 <ttx> jaypipes: or are you confident we can have decent coverage before the last milestones of Essex ?
20:43:47 <jaypipes> ttx: of course. that's what we discuss in our wednesday meetings...
20:44:00 <anotherjesse> mtaylor: is there a philosophy about broken tests in the integration suite?
20:44:07 <mtaylor> anotherjesse: yes
20:44:13 <anotherjesse> mtaylor: eg, when we write a test, can it merge if it is "broken"
20:44:21 <jaypipes> ttx: I'm confident we'll have decent coverage of some stuff... depends on how much APIs and implementations change between now and then.
20:44:22 <westmaas> ttx: decent coverage by last milestone of essex is well within reach
20:44:23 <anotherjesse> but the test isn't what is broken, it could be underneath
20:44:26 <mtaylor> anotherjesse: the thing we discussed at the ods was having a blessed suite of gating tests
20:44:28 <westmaas> but more help is useful
20:44:31 <anotherjesse> mtaylor: cools
20:44:44 <ttx> jaypipes: ok
20:44:53 <mtaylor> anotherjesse: so that withing the test suite, there will be the full thing, and the meta-suite that's what we believe is solid enough to block trunk on
20:45:03 <anotherjesse> mtaylor: cool
20:45:08 <mtaylor> anotherjesse: and hopefully a simple process for promoting test cases
20:45:31 <mtaylor> oh, btw...
20:45:33 <ttx> jbryce: so yes, I guess I could call for more practical "put your foot where your mouth is and join the QA team"
20:45:55 <mtaylor> so, one of the pieces that has come out of jeblair's work this past week on this are jobs in jenkins that are triggered by multiple gerrit projects
20:46:10 * jaypipes noticed commits to kong recently, well after kong was integrated into the main test suite...
20:46:17 <ttx> jbryce: but convincing all the companies around the PPB table sounds more efficient than a blogpost on my personal website :)
20:46:33 <mtaylor> so - in this case, we can have one that gates nova, keystone, glance, devstack and openstack-integration-tests ... so that a change to any of them is tested against trunk of the others
20:46:37 <mtaylor> in a very elegant manner
20:47:03 <jbryce> ttx: i agree. part of why i wanted to raise it here
20:47:19 <jbryce> ttx: if you blog though, i bet you'll get some decent retweeting of that post. = )
20:47:22 <zns> Won't we need to be able to submit cross-project patches for that to really work?
20:47:39 <jaypipes> also, there's commits to cloudbuilders/openstack-puppet dated Oct 30, while the main openstack/openstack-puppet hasn't changed since 20th Oct.
20:47:42 <ttx> jbryce: let's try that, should improve my klout score.
20:48:34 <jbryce> ok
20:48:46 <jbryce> let me try to summarize and see if i've got it all straight
20:48:47 <mtaylor> zns: shouldn't need to. if you add apis to one thing, then consume them in the next patch... ensures that people are writing to specs cross-project
20:48:50 <vishy> jaypipes: i don't think the deploy team has confidence in openstack-ci yet
20:48:58 <jaypipes> vishy: awesome
20:48:59 <mtaylor> zns: BUT - we will definitely keep our eyes on that
20:49:20 <zns> mtaylor: ok
20:49:37 <jbryce> 1) we have continuous integration testing infrastructure in place now that is able to do basic testing and gate contributions on things really functionining
20:50:15 <mtaylor> jbryce: (1) yes - give or take a couple of days, we've got a little cross-team coordination to do
20:50:21 <jbryce> 2) we are going to start including the intergration-tests in to that gating which will improve the coverage and give better testing
20:51:19 <jbryce> 3) we could use additional contributions to the integration-tests and should reach out and try to get more involvement from people. getting more tests here will be a big help in improving the quality before release time comes around
20:51:51 <jbryce> 4) there's some basic work going on to do not just functional tests (item A works) but also longer running and larger scale tests, but it doesn't sound like that effort is as defined or as far along yet
20:51:51 <jaypipes> jbryce: good summary
20:52:11 <ttx> +1
20:53:15 <jbryce> ok. i'm going to go lobby for this on my own with the companies and people i talk to. i think we should all do it. i'm really keen to make essex the one that we can be super proud of
20:53:32 <jbryce> #info 1) we have continuous integration testing infrastructure in place now that is able to do basic testing and gate contributions on things really functionining
20:53:38 <jbryce> #info 2) we are going to start including the intergration-tests in to that gating which will improve the coverage and give better testing
20:53:47 <jbryce> #info 3) we could use additional contributions to the integration-tests and should reach out and try to get more involvement from people. getting more tests here will be a big help in improving the quality before release time comes around
20:53:54 <jbryce> #info 4) there's some basic work going on to do not just functional tests (item A works) but also longer running and larger scale tests, but it doesn't sound like that effort is as defined or as far along yet
20:54:10 <jbryce> grr...should have done that the first time around, but better late than never
20:54:16 <mtaylor> hehe
20:54:52 <jbryce> i think we should discuss this again when we get towards essex-2 and see how things are progressing
20:54:59 <jaypipes> ++
20:56:10 <jbryce> i really worry about not having a drastic improvement and how it will impact the community and credibility of openstack. i'd even be open to considering treating some level of functional quality testing like an essential blueprint that could delay the release.
20:57:03 <ttx> jbryce: one issue is that it's never really finished
20:57:31 <jbryce> ttx: i know. i agree on that for sure. i like to ship, but there's a different between never really finished and really not working
20:57:46 <ttx> and there are organizational things around development that can't support long delays. Like design summits.
20:58:13 <ttx> so we should get our act together and just make it shine in the timeframe we have
20:58:21 <jbryce> i agree on that too. that's why i want to raise it now. if we get ahead of it, there shouldn't need to be a delay.
20:58:33 <ttx> it won't be perfect, but it needs to be better !
20:59:05 <jbryce> thanks for all the info. i'm going to go on a campaign. = )
20:59:12 <jbryce> anyone have anything for our final 40 seconds?
20:59:56 <jbryce> thanks everyone
20:59:56 <ttx> nope
20:59:58 <jbryce> #endmeeting