19:04:59 <mtaylor> #startmeeting
19:05:00 <openstack> Meeting started Tue Aug 23 19:04:59 2011 UTC.  The chair is mtaylor. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:05:01 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic.
19:05:08 <jaypipes> bengrue, anotherjesse: no worries.
19:05:20 <jaypipes> bengrue: feeling better today I hope?
19:05:29 <bengrue> Yeah.  In the office.
19:05:55 <jaypipes> OK, so mtaylor, before you start... lemme get something in.
19:06:36 * mtaylor punches jaypipes
19:06:43 <mtaylor> #topic jaypipes talking to people about things
19:06:48 <jaypipes> All: I'm putting together a project plan that has action items for folks on my team (mtaylor, soren, and jeblair) as well as some folks on other teams like anotherjesse's.
19:07:07 <jaypipes> Hoped to get it done by this meeting time, but failed unfortunately.
19:07:58 <jaypipes> The basic gist of the plan is that mtaylor and jeblair will have Jenkins firing off an automated deployment of OpenStack onto 5 lab servers by Friday, and this deployment cluster will have a set of functional tests run against it.
19:08:18 <jaypipes> The tests will be just the nova smoketests and possibly some stuff from smokestack or kong (for this Friday)
19:08:54 * soren is back
19:09:07 <jaypipes> mtaylor is at Burning Man all next week so jeblair will be working on documenting a bunch of stuff next week around the CI infrastructure and soren will be documenting and working on QA/QE stuff (the actual test cases and suites running on a deployed cluster)
19:09:38 <soren> Burning man? Will he be alright?
19:09:44 <mtaylor> soren: always am
19:09:50 <jaypipes> termie_ and sleepsonthefloor got Djeep installed on one of the lab servers late yesterday and jeblair now has access to that (thank you termie_ and sleepsonthefloor!_
19:09:54 <mtaylor> soren: I will, however, be COMPLETELY unreachable
19:10:35 <jaypipes> what anotherjesse wants most is to have all the code (puppet modules, glue code, etc) in a public repo that is tested and code reviewed like any othe ropenstack project. We are all in agreement on that!
19:11:03 <anotherjesse> jaypipes: I'm cool with the CI team who is doing the draft not having to get approval at the begining ;)
19:11:15 <mtaylor> anotherjesse: ++
19:11:16 <anotherjesse> but putting it in a repo we can watch and help would be muy bueno
19:11:28 <jaypipes> anotherjesse: sure
19:11:37 <anotherjesse> mtaylor: that doesn't mean 1 huge drop - if you don't commit at least daily, no burning man for you
19:11:48 <mtaylor> anotherjesse: so far, depending on what it is, we put everything we do in either openstack/openstack-ci or openstack/openstack-ci-puppet
19:11:54 <mtaylor> anotherjesse: hehe
19:12:10 <anotherjesse> mtaylor: will that be added to ci.openstack.org - which repo is for what
19:12:14 <mtaylor> anotherjesse: and we actually already do gerrit reviews and some testing on each other ... so we're set
19:12:19 <jaypipes> I think that there are a number of GitHub repos already existing that contain puppet modules and chef recipes. These need to be identified and anotherjesse and others need to let mtaylor and jeblair know which puppet/chef modules we need to fire from Jenkins on each build to get continuos deployment testing better
19:12:42 <mtaylor> anotherjesse: openstack-ci-puppet is for the puppet modules we use to manage the jenkins slaves and other servers
19:12:55 <mtaylor> anotherjesse: openstack-ci is for supporting scripts/glue code/docs/whatnot
19:12:56 <anotherjesse> mtaylor: for the website - not for me ;)
19:13:02 <jaypipes> one sec guys, can I just finish up?
19:13:03 <mtaylor> anotherjesse: yup.
19:13:08 <mtaylor> jaypipes: NO
19:13:09 <mtaylor> jaypipes: yes
19:14:11 <jaypipes> OK, so the Djeep installation on the deployment test cluster is temporary. Over the next week, jeblair is going to be looking into the puppet modules (site.pp IIRC) that Djeep generates/builds and converting the automation code to run cobbler instead of having the Djeep stuff on the deployment cluster.
19:14:28 <jaypipes> This is going to be done with an eye to adaptability
19:14:43 <jaypipes> so that in the future, things like Crowbar can also be used to deploy
19:14:54 <jaypipes> anotherjesse: does above ^^ ok with you?
19:14:57 <anotherjesse> jaypipes: cooleo - fyi djeep builds yaml files - and so puppet can run without it
19:15:27 <jaypipes> anotherjesse: ok, well that is something that jeblair is going to document and analyze next week
19:15:35 <jeblair> er, i'm at rookie-o next week
19:15:42 <jaypipes> but the bottom line for all is that by Friday, we WILL have a deployment cluster that is at least:
19:15:44 <jeblair> i'm not sure i'll have much time to work on this
19:15:50 <jeblair> (next week)
19:15:50 <anotherjesse> jeblair: take your laptop
19:15:53 <jaypipes> jeblair: hold on, one sec :)
19:16:03 <jaypipes> but the bottom line for all is that by Friday, we WILL have a deployment cluster that is at least:
19:16:13 <jaypipes> a) testing *some* deployment packaging
19:16:23 <jaypipes> b) firing *some* tests against the deployed cluster
19:16:27 <jaypipes> OK.. .now
19:17:13 <jaypipes> jeblair: documenting/analyzing stuff you can do while on the plane, etc.. Rookie-O isn't an all-consuming thing the entire week, and I'm trying to get us out of it a bit early so we can actually get some stuff done...
19:18:01 <jaypipes> jeblair: if you don't get the analyze/document stuff done next week, it's ok... as long as we're moving forward and have a plan, I'm happy.
19:18:02 <anotherjesse> jaypipes: fyi we took our laptops and sugarbear told us when we had to make sure to pay 100% attention (lawyers, hr, ...)
19:18:29 <jaypipes> anotherjesse: yeah, but we're there the entire week...
19:18:33 <jeblair> i just want to set expectations about how much time i'm really going to have to work on this stuff next week.
19:18:34 <jaypipes> anyway, back on track...
19:18:46 <jaypipes> jeblair: expectation is that you'll do the best you can, nothing more.
19:18:56 <jaypipes> jeblair: same expectation for anyone else.. :)
19:19:15 <jaypipes> the point is that we have a) a plan and b) we're moving forward and not spinning wheels
19:19:27 <jaypipes> bengrue, anotherjesse, jeblair, mtaylor: agreed? ^^
19:19:39 <jeblair> lgtm
19:19:40 <mtaylor> works for me
19:19:57 <anotherjesse> jaypipes: agreed - mtaylor question about a
19:20:06 <mtaylor> #agreed we have a) a plan and b) we're moving forward and not spinning wheels
19:20:12 <mtaylor> anotherjesse: talk to me
19:20:19 <bengrue> sounds clearer to me; where will the CI server be on friday?
19:20:20 <anotherjesse> if we need to change the preseed or puppet rules - is that stuff that will be in the bzr repo?
19:20:24 <jaypipes> as for soren's part in this, I will work with soren over the next few days to identify some low-hanging fruit re: funtional test suite and identifying/agreeing on OUTPUTs, etc
19:20:26 <anotherjesse> (what it is pointing to)
19:20:38 <jaypipes> anotherjesse: git repo, but yes
19:20:45 <anotherjesse> jaypipes: cooleo - either way
19:20:48 <mtaylor> anotherjesse: what jaypipes said
19:20:51 <jaypipes> mtaylor: answer for bengrue ?
19:20:56 <bengrue> (ie, what should I be paying attention to / eagerly awaiting with baited breath?)
19:21:01 <anotherjesse> grue have seen your email from jay outlining a&b?
19:21:07 <mtaylor> bengrue: I'm not sure I understand the question?
19:21:07 <anotherjesse> (and cdefgh)
19:21:17 <jaypipes> mtaylor: asking where he can see the deploy cluster
19:21:24 <mtaylor> bengrue: as in - where will it physically be?
19:21:25 <mtaylor> ah
19:21:35 <mtaylor> well - jenkins.openstack.org will be the thing driving this puppy
19:21:37 <bengrue> I have the email, scanned it so far, haven't responded yet.
19:21:43 <jaypipes> mtaylor: and the code that is running/deploying the cluster, I assume
19:21:55 <bengrue> Oh, the link is there.
19:22:10 <bengrue> Well, the link to the documentation?  http://ci.openstack.org
19:22:43 <jaypipes> bengrue: that is mtaylor and jeblair are dumping docs on this stuff (and dumping more docs in the next days and weeks)
19:23:02 <bengrue> are all of the ci jobs going to be prefixed with ci- ?
19:23:23 <bengrue> Will there be more added in the next week?  Or will everything be added into ci-puppet?
19:23:40 <jaypipes> bengrue: more of what? sorry, docs or code?
19:23:42 <mtaylor> sorry man - I'm still just not following the questions
19:24:05 <jaypipes> yes, I'm not quite following you either, bengrue
19:24:22 <anotherjesse> mtaylor: request for the ci.openstack.org site - can you put a "how to contribute" section?  With links to repos and whatnot?
19:24:23 <mtaylor> there will be a job, something like say "nova-baremetal" (it hasn't been named yet) that will launch the deploy/test stuff
19:24:30 <mtaylor> anotherjesse: yes
19:24:34 <jaypipes> anotherjesse: ++
19:24:42 <mtaylor> #action mtaylor Add how to contribute section to ci.openstack.org
19:24:53 <jaypipes> mtaylor: openstack-deploy? :) nova is only one piece :)
19:24:56 <anotherjesse> mtaylor: and perhaps a link to the ci site on the jenkins site?
19:25:02 <bengrue> You're saying that by friday, there will be a deployment cluster that is testing deployment packaging and firing tests against it.  You're saying that the ci server is jenkins.openstack.org.  I assume that jenkins will be triggering the cluster deployment via new jobs, yes?
19:25:12 <mtaylor> jaypipes: sure. like I said - I have not gotten there yet
19:25:14 <jaypipes> bengrue: yep
19:25:19 <mtaylor> bengrue: yes
19:25:20 <bengrue> And then coordinating the testing against the cluster?
19:25:30 <mtaylor> bengrue: yes
19:25:32 <jaypipes> bengrue: well, firing a command that runs the tests, yes
19:25:38 <bengrue> Will the new jenkins jobs be prefixed with ci- ?
19:25:41 <mtaylor> no
19:25:42 <jaypipes> no
19:25:50 <jaypipes> All jenkins jobs are CI...
19:25:55 <bengrue> where should I be looking on Jenkins for the new stuff?
19:26:04 <mtaylor> I do not know what they will be prefixed with ... I'll send out a mailing list message
19:26:07 <bengrue> Oh
19:26:07 <bengrue> https://jenkins.openstack.org/job/ci/
19:26:16 <bengrue> Is this what I should be stalking?
19:26:21 <mtaylor> no
19:26:36 <jaypipes> co
19:26:38 <jaypipes> no
19:26:39 <bengrue> so tbd then, I'll wait for the msg.
19:26:47 <mtaylor> that is merely a job that gerrit uses to check commits we make to openstack-ci - which are a collection of helper scripts
19:26:59 <mtaylor> yeah - msg will make it clearer
19:27:10 <mtaylor> also- I'd like to be clear on something ...
19:27:12 <jaypipes> mtaylor: let's just determine the name now... can we call the master job openstack-deploy?
19:27:21 <jaypipes> CI != QA :)
19:27:37 <bengrue> I'm not sure I follow.
19:27:44 <mtaylor> openstack-deploy is a bit encompasing - as I expect to have one of these jobs for msft/novel, one for citrix, etc.
19:27:52 <mtaylor> so, how about openstack-deploy-rax
19:27:58 <jaypipes> mtaylor: sure
19:28:00 <mtaylor> since it's the one deploying to the rackspace hardware
19:28:03 <anotherjesse> jaypipes: CI  != QA - can you elaborate?
19:28:08 <bengrue> In my experience, the whole point of CI/CD _is_ to play a large amount of the traditional QA role.
19:28:25 <anotherjesse> jaypipes: this might be the disconnect?
19:28:26 <anotherjesse> ;)
19:28:32 <anotherjesse> since I agree with ben
19:28:51 <mtaylor> CI is a tool/method used for QA
19:28:53 <jaypipes> anotherjesse, bengrue: CI is the automation of continuous builds triggered by commits to trunk. QA is the tests and test suites that get executed against deployed clusteres
19:29:18 <jaypipes> anotherjesse, bengrue: I explained this disconnect in terminology in my email to you both :)
19:29:33 <jaypipes> anotherjesse, bengrue: sooo, in other words...
19:30:25 <jaypipes> CI == Gerrit/Jenkins/Tarmac, CD == CI + deployment scripts and modules, QA == functional, unit, and integration test suites running on the clusters deployed by CD.
19:30:38 <jaypipes> that make more sense?
19:30:56 <jaypipes> mtaylor: would you agree with the above simplification?
19:31:01 <anotherjesse> sure.. but the point of caring about CI is so you can do CD/QA
19:31:08 <anotherjesse> not CI just for the sake of it
19:31:15 <bengrue> I saw that, and I'm unsure that I agree with the premise.  Technically the infrastructure to run tests are seperate from the tests, but both are necessary parts of the whole.
19:31:16 <nati> jaypipes++
19:31:22 <bengrue> This is the CI/Testing team, right?
19:31:28 <anotherjesse> so CI requirements are driven by CD/QA ?
19:31:37 <jaypipes> anotherjesse: sure, but our team has expertise in CI and a bit in CD, and we need your team's expertise in D to get good CD and QA ;)
19:31:53 <bengrue> OMG acronym BBQ
19:32:05 <anotherjesse> jaypipes: so - perhaps this is silly but imho most devs can't have a dev environment that even simulates a deployment
19:32:15 <jaypipes> anotherjesse: the point being that mtaylor and jeblair own the glue code to fire off deployment modules that your team creates.
19:32:18 <anotherjesse> since it is rather complicated to have a network toplogy
19:32:22 <nati> NTT can also provide staff for QA in D ;)
19:32:30 <anotherjesse> so - CI is needed to do development if you do TDD
19:32:38 <jaypipes> anotherjesse: mtaylor and jeblair can't be responsible for creating working deployment/puppet modules...
19:33:02 <bengrue> How can you have a stable testbed without working deployment scripts?
19:33:03 <anotherjesse> not asking for that - asking for ability to not restrict the modules / scripts / … (which we have said we aren't!)
19:33:04 <mtaylor> so... this is great ... I have the feeling we might not solve the acronym discussion right here
19:33:07 <jaypipes> anotherjesse: but they *can* be responsible for making sure that the puppet modules your team and others create get automatically built on hardware linked in the CI environemt
19:33:08 <anotherjesse> (they have said they aren't
19:33:21 <anotherjesse> agree
19:33:29 <jaypipes> anotherjesse: k.
19:33:39 <jaypipes> anotherjesse: just trying to delineate who is owning what :)
19:33:43 <anotherjesse> bengrue: I think he is saying that they will make the scripts that we collaborate on
19:33:45 <anotherjesse> work
19:33:52 <anotherjesse> but we (stackers) own the scripts
19:33:56 <jaypipes> anotherjesse: yes!
19:33:59 <jaypipes> :)
19:34:25 <bengrue> just the individual stacker teams?
19:34:25 <jaypipes> anotherjesse: and we are relying on your team to create those puppet modules and make sure they actually work :)
19:34:27 <anotherjesse> I've not thought otherwise - I've just not seen what are the assumptions about the environment our scripts need to run within ;)
19:34:32 <bengrue> What about us sattelite  companies? ; )
19:34:39 <anotherjesse> bengrue: you are a stacker
19:34:48 <jaypipes> anotherjesse: and you are relying on us to make sure that jenkins continuously executes those puppet modules to test deployment and packaging.
19:34:50 <bengrue> oh, openstack, derp.
19:35:04 <bengrue> racker / stacker conflation
19:35:10 <jaypipes> anotherjesse: we good on that distinction?
19:35:13 <anotherjesse> bengrue: and so the CI environment needs to be chainable / …so you can kick off things (this is where monty/jim/jay can speak up since I don't even know the words)
19:35:21 <anotherjesse> jaypipes: I've agreed with that and continue to agree with that
19:35:33 * soren needs to run for 3 minutes.
19:35:44 <mtaylor> anotherjesse: I think what you said may be part of a conversation we should have at some point re: "assumptions about the environment"
19:35:52 <jaypipes> bengrue: outside partners and individuals such as yourself are welcome to contribute to any and all aspects of this, but IMHO, the most pressing needs are in the development of additional test cases.
19:35:58 <mtaylor> anotherjesse: I think we may each be expecting to see those assumptions from each other :)
19:36:43 <jaypipes> anotherjesse: re: chainable stuff, that's a task for mtaylor and jeblair to document: How to stand up a Jenkins builder in your own lab...
19:37:14 <mtaylor> anotherjesse, bengrue: and yes. the intent is to allow all of the satelite companies to provide resources to ensure that specific hardware combinations/configurations that are important to them are tested
19:37:17 * soren returns
19:37:32 <jaypipes> mtaylor: ok, now that we've agreed to agree on the agreement about responsibilities, can we get an update on where we stand on the deployment cluster?
19:38:01 <mtaylor> jaypipes: jeblair is working with the djeep system that termie and sleepsonthefloor just handed off to him
19:38:12 <jaypipes> mtaylor: by "working with", what do we mean?
19:39:01 <jeblair> evaluating that it works as expected, and we have access.  it does, and i do
19:39:18 <jaypipes> jeblair: awesome. next steps?
19:39:21 <bengrue> So, just for my general edification, when will the "assumptions about the environment" cease to be assumptions and start to be an agreed upon, explicit set of facts?  Just looking for a general timeframe here, nothing binding.
19:39:40 <mtaylor> hang on guys - one topic at a time please
19:39:40 <bengrue> Is that a conversation for this coming week, etc?
19:40:16 <jeblair> currently it's based on a puppetmaster server.  we generally don't use that approach, and run puppet out of a local git repo.  we may want to adjust the system to that approach
19:40:29 <anotherjesse> bengrue: I think mtaylor as he deploys the first environment might have a heavily constrained first environment --- and then they are going to work on loosening it
19:40:35 <jeblair> and get the puppet config in a git repo where we can all hack on it
19:40:40 <mtaylor> anotherjesse: ++
19:40:42 <anotherjesse> bengrue: making it so there are multiple - and it evolves to multiple architectures
19:40:48 <bengrue> Okay, that makes sense.
19:41:00 <anotherjesse> i'm hoping mtaylor will be guilted into lots of docs in ci.openstack.org ;)  or beered into it
19:41:01 <jeblair> we also need to start working on the jenkins job that fires off rebuilding the cluster
19:41:13 <mtaylor> anotherjesse: definitely.
19:41:19 <bengrue> I am a fan of spiking.
19:41:24 <jaypipes> beer works best.
19:41:35 <bengrue> make sure autocorrect is on.
19:41:43 <bengrue> ; )
19:42:16 <jeblair> so i think that's the main thrust of work on the cluster, there are dependencies:
19:42:21 <jaypipes> jeblair: the puppetmaster/git puppet thing can wait? or is that something you want to get done before Friday?
19:42:31 <mtaylor> although just to set expectations ... priority #1 for the week is getting this working by friday. priority #2 is piles of docs. I want both, but I think 1 without 2 will be met with way more approval than 2 without 1 ... so if something has to slip to next week, it's going to be #2
19:42:50 <jaypipes> mtaylor: yes
19:42:57 <anotherjesse> jeblair: the assumption is that we are working towards multiple architectures eventually (like VLAN mode vs LXC  vs ...)
19:42:58 <anotherjesse> right?
19:43:14 <anotherjesse> so - tests will run against 1 or more architectures
19:43:30 <mtaylor> yes
19:43:45 <jaypipes> anotherjesse: yes, absolutely.
19:44:00 <jaypipes> anotherjesse: there would be little point to this if that were not the case ;)
19:44:01 <jeblair> yes -- the djeep setup presupposes one architecture, so that's probably a post-djeep step
19:44:51 <jaypipes> jeblair, mtaylor: OK, so could you outline in steps exactly what you need to do before Friday to get this all going? Any blockers that anotherjesse can assist with?
19:45:13 <anotherjesse> djeep pre-supposes nothing - but the preseeds/recipes it uses do … hence the roles / rolemapper - but for sake of arguement it doesn't change anything
19:45:57 <jeblair> anotherjesse mentioned his team was continuing work on puppet recipies (and removing dependence on puppetmaster) -- any progress there we can use?
19:46:22 <anotherjesse> jeblair: not yet - we've been focusing on d4 the last week
19:47:27 <jaypipes> jeblair: is that a blocker for Friday or something post Rookie-O?
19:48:48 <anotherjesse> jeblair: imho if you have it kicking off with puppet solo but it blows up
19:48:53 <anotherjesse> the community can fix it
19:49:47 <anotherjesse> getting it so a button causes it to: spin up servers, kick off a script (that runs puppet or ?), then runs tests … if each of the steps is in CI, we can all pitch in if we can look at how it blows up
19:50:08 <jeblair> we'll need the puppet config in git for that to work
19:50:14 <jaypipes> anotherjesse: a Big Red Button?
19:50:26 <anotherjesse> jeblair: isn't that what we are doing?
19:50:28 <jeblair> so that's definitely the goal
19:50:56 <jaypipes> jeblair: where is the puppet config that is used to build the cluster now?
19:51:14 <jeblair> it's on the cluster -- anotherjesse's team built it somewhat as a one-off
19:51:26 <jeblair> so i'm not sure it's ready to go right into a git repo as-is
19:51:44 <anotherjesse> jeblair: our puppet config is at github.com/cloudbuilders/openstack-puppet
19:51:55 <anotherjesse> we don't even have a commit key on the server
19:52:25 <anotherjesse> jeblair: so - the "missing" site.pp
19:52:42 <anotherjesse> is because the whole point of djeep is 2 components
19:52:54 <anotherjesse> 1) something that maps a server to a network / netboot config
19:53:04 <anotherjesse> these configs get written to yaml/json files
19:53:20 <anotherjesse> 2) puppet server that uses the node classification configuration
19:53:27 <anotherjesse> that load those json/yaml
19:53:39 <anotherjesse> so you can just use the puppet recipes with json & a classifier
19:53:43 <anotherjesse> or you can use a site.pp
19:54:05 <anotherjesse> or you can use everything if you are doing dev and switch between xen and kvm and different net models and blow away servers daily
19:54:15 <anotherjesse> jeblair: make more sense?
19:54:31 <anotherjesse> jeblair: I can show you how to manually run the node classifier on the djeep server
19:54:47 <anotherjesse> djeep doesn't even have to be running after it writes the yaml files - it is pure puppet
19:55:24 <jaypipes> anotherjesse: they are formulating their response :)
19:55:49 * jaypipes happy we are finally getting to the nitty gritty :)
19:55:57 <anotherjesse> jaypipes: sorry for the sploid
19:56:11 <jaypipes> hey, no worries, this is exactly what I was hoping for.
19:56:23 <jeblair> for the moment, i don't think we need to change the configuration, right?
19:56:29 <jaypipes> these are precisely the conversations we need to have.
19:56:39 <anotherjesse> in our master we set:
19:56:39 <anotherjesse> [master]
19:56:40 <anotherjesse> node_terminus = exec
19:56:40 <anotherjesse> external_nodes = /root/djeep/etc/puppet/node_classes.py
19:56:41 <jeblair> so we don't need to re-run the node classifier
19:56:57 <anotherjesse> jeblair: node classifier is a way for puppet master to not have a sitepp
19:58:20 <jeblair> i guess what i was trying to say is that we can leave the djeep configuration static for the time being
19:59:40 <anotherjesse> jeblair: here is an example of output:
19:59:40 <anotherjesse> http://paste.openstack.org/show/2259/
19:59:45 <anotherjesse> sanitized
19:59:52 <jbryce> meeting time is almost up. = )
20:00:12 <anotherjesse> if site.pp wants to hardcode 356602-staging-cpu2 to have various global variables and classes you can
20:00:18 <anotherjesse> it just is VERY verbose
20:00:27 <anotherjesse> having a simple classifier is much easier
20:00:29 <bengrue> Were there non-open-discussion topics that needed to be discussed?
20:00:34 <mtaylor> out of time
20:00:35 <mtaylor> :)
20:00:44 <anotherjesse> we can continue in #dev
20:00:50 <bengrue> Let's.
20:00:51 <jaypipes> anotherjesse, jeblair: please do continue in #dev...
20:00:53 <jeblair> i'll dig more into puppet and ask anotherjesse with questions
20:01:29 <mtaylor> thanks guys! easily the densest ci meeting we've had in a while!
20:01:31 <mtaylor> #endmeeting