20:17:14 #startmeeting 20:17:15 Meeting started Tue Nov 15 20:17:14 2011 UTC. The chair is jbryce. Information about MeetBot at http://wiki.debian.org/MeetBot. 20:17:16 Useful Commands: #action #agreed #help #info #idea #link #topic. 20:17:24 anotherjesse: but once we get this first one up - I really want to start spending more time on the plan of getting better/fuller hardware/environment coverage from vendors, which we've got good ideas on doing 20:17:41 nice 20:17:42 my hard drive started rattling like a pair of castanets. i think that's a bad sign. yay for backups, though 20:17:52 jbryce: backup to the cloud 20:17:59 jbryce: unless you like castanets 20:18:08 mtaylor: true... 20:18:10 http://wiki.openstack.org/Governance/PPB - agenda 20:18:20 #topic Foundation update 20:18:41 thierry asked about an update on foundation progress since things have been quiet on the mailing list 20:19:31 since we made the announcement, we've seen a lot of interest from a lot of different people. we've been meeting with different companies and foundations and getting feedback and trying to get as many of them as possible to join up on the mailing list also 20:20:17 one consistent piece of feedback that we've gotten is that we should come forward with a drafting process and timeline for putting it all together, so that's something we've been working on with several lawyers from rackspace and elsewhere 20:20:53 jbryce: should we encourage the presentation of existing models on the ML ? 20:21:37 (copying an existing model is a good way to reduce the base work) 20:21:54 jbryce: or are your private discussions already past that ? 20:22:18 mark or i will be sending out a more detailed email to the foundation list but the basic process is to publish a draft of the high level goals and philosophies and get feedback on that over the rest of November, then move into an actual legal document drafting process. that will probably consist of 2 drafting periods where we publish a draft and accept comments over the course of several weeks and the re-draft 20:23:11 ttx: discussing existing models is definitely something we can do on the list 20:23:14 jbryce: do you want the "goals and philosophies" defined first, before the actual governance and foundation model ? Or in parallel ? 20:23:49 i think it makes sense to get the high level done first as it will affect the details of the legal structure 20:23:55 (because the existing models are obviously more about governance structure than about goals/scope/philosophy) 20:24:09 true 20:24:43 so imho that's two separate discussions, dunno if you want to have them in parallel or define the goals first 20:25:01 i think i'd prefer goals first 20:25:35 jbryce: ok, would be good to send out some progress note saying the first step is to discuss/define goals/philosophy/scope 20:25:39 in summary, there's been a small amount of progress in actual documentation, but quite a bit of input and feedback that has been offered from a variety of source that we're trying to aggregate and publish 20:26:21 (so that the discussion can focus on that first) 20:26:44 good feedback 20:27:21 anything else? question or suggestions? 20:28:09 ok. moving on 20:28:17 #topic Quality for Essex 20:28:24 so... 20:28:39 first of all: http://eavesdrop.openstack.org/meetings/openstack-meeting/2011/openstack-meeting.2011-11-15-19.00.log.html 20:28:41 +1 20:29:03 we had a great summary of where we were on this during the CI meeting in the last hour 20:29:52 as a much briefer summary - current status is that jeblair has the infrastructure/integration pieces working to launch fresh cloud servers, run devstack on them and then run exercise.sh against those 20:30:36 currently immediately outstanding todo items to get those tests up and going are getting devstack working against trunk (which vishy apparently already has done) and getting devstack and python-novaclient moved over to gerrit 20:31:07 mtaylor: also devstack has a config.ini generator 20:31:08 (we actually don't need trunk working before we start - those are just the immediate tasks) 20:31:17 so you can run the openstack-integration-tests against them 20:31:21 (we do that on our jenkins) 20:31:24 great 20:31:55 mtaylor: was there talk about how to run larger scale tests (launch vms non-stop for a week, ...) 20:32:01 that's the next step after the above, is once we're satisfied that exercise.sh can run consistently, we'll start adding openstack-integration-tests 20:32:47 why are we using 2 jenkins again? 20:32:51 anotherjesse: at the moment I think running tests of that nature are going to need folks to pony up labs in which to run them in 20:33:01 jaypipes: we push everything we do back to mtaylor / jeblair 20:33:02 how much of the core functionality do you think will be covered by the time we start getting to the end of essex? 20:33:03 jaypipes: anotherjesse has been helping prototype devstack jenkins stuff 20:33:08 jaypipes: should we stop? 20:33:17 we don't want to break the core jenkins server 20:33:20 just dont understand why there is a need for 2 jenkins... 20:33:36 jaypipes: everyone else runs their own jenkins - is it bad for us to as well? 20:33:54 jaypipes: too many cooks in the kitchen - basically that way anotherjesse can do things and show them to us 20:33:55 #info will need better lab environments to do larger scale and longer-running tests 20:34:06 anotherjesse: the direction should be the central jenkins out to secondary jenkins, not the other way around. 20:34:12 mtaylor: getting the integration tests running and alerting of basic failures is great, but then there is things like the rax cloud hit in early essex - after many vm launches it various api commands got really slow 20:34:16 jaypipes: dude. totally different thing 20:34:25 mtaylor: no, it's not. 20:34:28 jaypipes: it is 20:34:36 anotherjesse's jenkins is a dev jenkins 20:34:51 mtaylor: and what is the point of that? 20:34:54 it's a sandbox where he can play with different jobs and not worry about breaking the core element of the entire dev process 20:35:19 doing tons of dev directly on the core jenkins is, well - if things break, then things BREAK, right? 20:35:27 it's staging vs production...don't break the live one that gates the development process for everyone 20:35:41 jaypipes / mtaylor we can move this talk about ways we can help the ci process externally 20:36:00 since we had asked mtaylor / jeblair at the last summit and we thought it was a good idea 20:36:08 move this talk to another time - outside pbb 20:36:13 sure 20:36:18 yeah. perhaps next week CI? 20:36:33 mtaylor: ya - setting alarm now - didn't update time 20:36:39 so the process sounds great, but can i re-ask my question? = ) 20:36:41 how much of the core functionality do you think will be covered by the time we start getting to the end of essex? 20:36:52 jbryce: that's actually a question that's out of my scope 20:36:59 jbryce: I worry about testing large scale VMs launches 20:37:15 but - there is lots of parties (hp, rax and others) talking about it 20:37:16 jbryce: my main bit is making sure stuff gets run consistently - the integration tests guys will need to get good coverage in their work 20:37:19 we have a great plan for the general stuff 20:37:24 and then the thing anotherjesse is mentioning 20:37:43 because honestly the jenkins control-center model isn't going to be terribly useful for week-long tets 20:37:45 tests 20:37:46 mtaylor: that's fine. not saying it's your scope, but i'd love to know that someone is thinking about it. 20:37:50 totally 20:38:03 do we know how the integration-tests development is going so far ? 20:38:30 I think it's going to wind up being a discussion between folks like rax, dell and hp - folks who have the resources to spin up a lab somewhere that can run such a test 20:38:34 ttx: bcwaldon had gotten the basics working 20:38:35 mtaylor: to be blunt, it's pointless trying to design large scale performance testing of Nova until a deployment Jenkins job that uses real deployment things like chef/puppet is completed. 20:38:40 and also find out if we should go really prod some people to work on this as a form of the strategic contributions that ttx has talked about 20:38:53 and then we have to figure out how to collaborate on that between us 20:38:56 jbryce: the teams are up and are open 20:38:59 and how to report back findings in a sensible manner 20:39:04 anyone can join and help 20:39:18 jaypipes: again, I disagree - I think we can design the process modularly 20:39:24 mtaylor: agreed 20:39:32 jaypipes: the long-term test is just something that needs to run against a cloud 20:39:36 mtaylor: the process of deploy is independant of the tests 20:39:39 how that cloud got there is a different engineering problem to solve 20:39:41 mtaylor: ok, I'll reserve (any more) judgment until I see that then 20:39:42 yes 20:40:19 and in this case 20:40:51 even though we're actually doing a good job along the lines of automated deployment of cloud onto bare metal from jenkins- that work specifically isn't going to be as directly applicable to a week-long test 20:41:03 i know that the teams are open and anyone can help, but if it's not happening and we get to the end of essex and these tests are not helping us have a much better release, i think that's a big failure on our part. 20:41:04 because I don't want a week-long test taking up threads in my crazy java app called jenkins :) 20:41:22 jbryce: couldn't agree more. 20:41:34 jbryce: that's why I was asking on progress on the integration-tests side -- see if more resources are needed there and a call for help is warranted 20:41:38 jbryce: I would say that if we don't have integration tests running by essex-2 we should all book time in a hotel somewhere, get together and have a celebrity death match 20:41:56 integration-tests is a little chicken-and-egg at the moment 20:42:19 mtaylor: the point is that you won't be taking up threads on *your* jenkins. The tests will be running on large HP (or other) clusters. But those clusters absolutely are a no-go until deployment jobs that use real ddeployment methods are done. 20:42:21 as soon as we get this next jenkins job deployed, then we'll have somewhere we can point their work at so that they can get consistent feedback on their tests running 20:42:23 mtaylor: really? we've been running them in jenkins for a couple weeks - will ping you after 20:42:27 mtaylor: the devil is in the number of those tests, not their simple existence. 20:42:37 anotherjesse: I meant dev-process-wise 20:42:46 mtaylor: ? 20:42:53 oh - in gating them 20:42:59 yes - understood now 20:43:00 anotherjesse: yes 20:43:16 ttx: for integration tests, dwalleck, westmaas, wwkeyboard, bcwaldon, nati, and a number of others have been contributing to them. 20:43:26 jaypipes: is more help needed ? 20:43:47 jaypipes: or are you confident we can have decent coverage before the last milestones of Essex ? 20:43:47 ttx: of course. that's what we discuss in our wednesday meetings... 20:44:00 mtaylor: is there a philosophy about broken tests in the integration suite? 20:44:07 anotherjesse: yes 20:44:13 mtaylor: eg, when we write a test, can it merge if it is "broken" 20:44:21 ttx: I'm confident we'll have decent coverage of some stuff... depends on how much APIs and implementations change between now and then. 20:44:22 ttx: decent coverage by last milestone of essex is well within reach 20:44:23 but the test isn't what is broken, it could be underneath 20:44:26 anotherjesse: the thing we discussed at the ods was having a blessed suite of gating tests 20:44:28 but more help is useful 20:44:31 mtaylor: cools 20:44:44 jaypipes: ok 20:44:53 anotherjesse: so that withing the test suite, there will be the full thing, and the meta-suite that's what we believe is solid enough to block trunk on 20:45:03 mtaylor: cool 20:45:08 anotherjesse: and hopefully a simple process for promoting test cases 20:45:31 oh, btw... 20:45:33 jbryce: so yes, I guess I could call for more practical "put your foot where your mouth is and join the QA team" 20:45:55 so, one of the pieces that has come out of jeblair's work this past week on this are jobs in jenkins that are triggered by multiple gerrit projects 20:46:10 * jaypipes noticed commits to kong recently, well after kong was integrated into the main test suite... 20:46:17 jbryce: but convincing all the companies around the PPB table sounds more efficient than a blogpost on my personal website :) 20:46:33 so - in this case, we can have one that gates nova, keystone, glance, devstack and openstack-integration-tests ... so that a change to any of them is tested against trunk of the others 20:46:37 in a very elegant manner 20:47:03 ttx: i agree. part of why i wanted to raise it here 20:47:19 ttx: if you blog though, i bet you'll get some decent retweeting of that post. = ) 20:47:22 Won't we need to be able to submit cross-project patches for that to really work? 20:47:39 also, there's commits to cloudbuilders/openstack-puppet dated Oct 30, while the main openstack/openstack-puppet hasn't changed since 20th Oct. 20:47:42 jbryce: let's try that, should improve my klout score. 20:48:34 ok 20:48:46 let me try to summarize and see if i've got it all straight 20:48:47 zns: shouldn't need to. if you add apis to one thing, then consume them in the next patch... ensures that people are writing to specs cross-project 20:48:50 jaypipes: i don't think the deploy team has confidence in openstack-ci yet 20:48:58 vishy: awesome 20:48:59 zns: BUT - we will definitely keep our eyes on that 20:49:20 mtaylor: ok 20:49:37 1) we have continuous integration testing infrastructure in place now that is able to do basic testing and gate contributions on things really functionining 20:50:15 jbryce: (1) yes - give or take a couple of days, we've got a little cross-team coordination to do 20:50:21 2) we are going to start including the intergration-tests in to that gating which will improve the coverage and give better testing 20:51:19 3) we could use additional contributions to the integration-tests and should reach out and try to get more involvement from people. getting more tests here will be a big help in improving the quality before release time comes around 20:51:51 4) there's some basic work going on to do not just functional tests (item A works) but also longer running and larger scale tests, but it doesn't sound like that effort is as defined or as far along yet 20:51:51 jbryce: good summary 20:52:11 +1 20:53:15 ok. i'm going to go lobby for this on my own with the companies and people i talk to. i think we should all do it. i'm really keen to make essex the one that we can be super proud of 20:53:32 #info 1) we have continuous integration testing infrastructure in place now that is able to do basic testing and gate contributions on things really functionining 20:53:38 #info 2) we are going to start including the intergration-tests in to that gating which will improve the coverage and give better testing 20:53:47 #info 3) we could use additional contributions to the integration-tests and should reach out and try to get more involvement from people. getting more tests here will be a big help in improving the quality before release time comes around 20:53:54 #info 4) there's some basic work going on to do not just functional tests (item A works) but also longer running and larger scale tests, but it doesn't sound like that effort is as defined or as far along yet 20:54:10 grr...should have done that the first time around, but better late than never 20:54:16 hehe 20:54:52 i think we should discuss this again when we get towards essex-2 and see how things are progressing 20:54:59 ++ 20:56:10 i really worry about not having a drastic improvement and how it will impact the community and credibility of openstack. i'd even be open to considering treating some level of functional quality testing like an essential blueprint that could delay the release. 20:57:03 jbryce: one issue is that it's never really finished 20:57:31 ttx: i know. i agree on that for sure. i like to ship, but there's a different between never really finished and really not working 20:57:46 and there are organizational things around development that can't support long delays. Like design summits. 20:58:13 so we should get our act together and just make it shine in the timeframe we have 20:58:21 i agree on that too. that's why i want to raise it now. if we get ahead of it, there shouldn't need to be a delay. 20:58:33 it won't be perfect, but it needs to be better ! 20:59:05 thanks for all the info. i'm going to go on a campaign. = ) 20:59:12 anyone have anything for our final 40 seconds? 20:59:56 thanks everyone 20:59:56 nope 20:59:58 #endmeeting