19:02:42 <mtaylor> #startmeeting
19:02:43 <openstack> Meeting started Tue Jun 14 19:02:42 2011 UTC.  The chair is mtaylor. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:02:44 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic.
19:02:47 <termie> HOYOO
19:03:17 <markwash> :-)
19:03:24 <mtaylor> #info Added an agenda overview to the wiki
19:03:29 <mtaylor> #link http://wiki.openstack.org/Meetings/CITeamMeeting
19:04:14 <mtaylor> So - I guess let's get this puppy rolling
19:04:47 <mtaylor> #topic Discuss/Present overall CIPlan as it currently stands
19:04:48 <jaypipes> o/
19:04:54 <termie> so, starting with ciplan
19:05:14 <mtaylor> yup. seemed like a good idea to start at the beginning
19:05:18 <termie> which smoke testing are you referring to?
19:05:34 <mtaylor> #link http://wiki.openstack.org/CIPlan
19:05:46 <mtaylor> well, right now to the jenkins job which is useless
19:05:51 <termie> the stuff in /smoketests only?
19:06:25 <mtaylor> I was _actually_ just talking about the infrastructure to run it - but yes, to run the stuff in nova's /smoketests
19:07:32 <termie> so, for those the general plan is: start a vm, provision with chef, point smoketests at them
19:07:54 <termie> at the moment ex-anso has been using vagrant because it automates a good portion of the things we need to do to set up a multinode cluster
19:08:04 <mtaylor> yup. although I'd say the general plan is "start a vm, install/provision with something, test them"
19:08:05 <termie> but you seem to be set on using external cloud servers
19:08:15 <termie> i am asking for a more concrete plan
19:08:24 <mtaylor> indeed.
19:08:35 <termie> "something" == chef scripts from openstack-recipes
19:08:37 <termie> in my opinion
19:08:38 <mtaylor> what I was going to say is that for now "install/provision with something" seems to be chef as a best practice
19:08:41 <mtaylor> yes
19:08:57 <termie> if there is no disagreement lets start saying htat
19:08:59 <termie> instead of something
19:09:14 <mtaylor> we were starting at the beginning here today, so I thought we'd be clear
19:09:22 <termie> we are clarlifying
19:09:23 <nati> Will chef scripts  be standard openstack deployment tool?
19:09:52 <termie> nati: i think they will be one of them, the puppet guys will also have a set of tools
19:09:56 <mtaylor> nati: that's certainly a possibility
19:10:17 <mtaylor> yeah - and getting in to chef v. puppet is certainly not something I care to do anytime in my personal near future
19:10:26 <nati> I think OpenStack should have an official deployment tool, and Smoketesting tool should use the official one.
19:10:40 <termie> nati: agreed, but we can't make the political statement right now
19:10:46 <mtaylor> ++
19:10:46 <nati> Because we should test deployment tool also.
19:10:47 <termie> nati: for now we are going to use the chef recipes
19:10:56 <mtaylor> right. because they are what's there
19:10:56 <termie> and our docs will describe the usage of those
19:11:02 <termie> and our tools will depend on them
19:11:03 <nati> agreed
19:11:11 <termie> the puppet guys will do owrk to make those tools able to use puppet as well
19:11:17 <termie> but that onus is on them
19:11:28 <termie> we like them we just have more inertia on chef right now
19:11:47 <termie> so, for the vm stuff, shall we say we use rackspace cloud servers?
19:11:59 <termie> or vagrant?
19:12:05 <mtaylor> yes. I think that gets us the best ability to do many of these in parallel
19:12:09 <jaypipes> RCS.
19:12:21 <termie> we also can get them more cheaply than amazon i assume
19:12:28 <mtaylor> yes. yes we can :)
19:12:43 <dprince> Chef cookbooks written properly should work on bare metal, cloud, or a personal VM on your laptop (vagrant).
19:12:51 <mtaylor> which then gets us to getting the damn thing working
19:12:52 <termie> dprince: yup, that's the goal
19:12:56 <mtaylor> dprince: yes. agree
19:13:14 <mtaylor> the two approaches on the table in my head for making this happen are:
19:13:17 <dprince> So the way I see us collaboration working here is we collaborate on good config management practices.
19:13:45 <mtaylor> openstack_vpc fired from jenkins - or just firing up a node and doing the chef bits on it by hand
19:13:56 <termie> what is openstack_vpc?
19:14:00 <mtaylor> I am, at this point, fine with either
19:14:10 <mtaylor> openstack_vpc is the thing that powers smokestack
19:14:10 <jaypipes> termie: https://github.com/dprince/openstack_vpc
19:14:23 <termie> cool, will that kick of chef pieces?
19:14:28 <mtaylor> yes
19:14:30 <dprince> Yep.
19:14:35 <termie> perfect, lets use that
19:14:38 <mtaylor> the main question I have for folks there is
19:14:49 <termie> we can make dprince make it ddo what we want :))
19:14:49 <jaypipes> termie: https://github.com/rackspace/chef_vpc_toolkit
19:14:55 <nati> Is there any limitations of VPC?
19:14:56 <dprince> Termie, I forked the Anso cookbooks and have made a couple minor changes to get things working in the cloud.
19:15:16 <dprince> nati: The limitations of VPC are the same limitations you'd hit on Cloud Servers.
19:15:17 <mtaylor> is requires redis, which also is used in the nova unit tests - is there any reason to think that running openstack_vpc on the jenkins box will bork the unit tests
19:15:18 <termie> dprince: bug xtoddx to get them integrated?
19:15:36 <termie> where is the redis requirement?
19:15:53 <termie> nova doesn't use redis anywhere anymore i thought
19:16:11 <jaypipes> termie: smokestack does...
19:16:14 <mtaylor> great! I just know that there were already redis processes on the jenkins box, and I didn't want to step on anything
19:16:14 <dprince> termie: Smokestack uses --> Cloud Servers VPC. Both of these are Rails apps for which I'm using a Redis backed job queue (Resque).
19:16:28 <markwash> mtaylor: I don't think openstack_vpc needs redis
19:16:38 <markwash> mtaylor: just the stuff built on top of it
19:16:48 <mtaylor> problem solved then - redis on jenkins box is historical and non-conflicting
19:16:48 <dprince> mtaylor: I'm happy to let you have access to the Cloud Servers VPC instance we are using on Titan.
19:16:53 <termie> dprince: so vpc is not a command-line tool
19:16:54 <termie> ?
19:17:23 <dprince> termie: openstack_vpc is a set of config files and rake tasks to spin up testing groups.
19:17:26 <mtaylor> dprince: well - main thing is that I need for jenkins to be able to fire the commands - probably easier to just install it on the jenkins box, yeah?
19:17:34 <dprince> termie: it is command line driven.
19:17:39 <dprince> Smokestack scripts it.
19:17:45 <termie> dprince: but we have to have a server running all the time that hits that server?
19:17:52 <termie> erm
19:17:57 <termie> and a cli that hits that server
19:18:32 <dprince> termie: openstack_vpc uses Cloud Servers VPC to spin up cloud servers.
19:18:49 <mtaylor> termie: Cloud Servers VPC is an always running rails app - so yes
19:19:13 <termie> and what does its always running get us over having it just do a one-shot thing?
19:19:13 <mtaylor> dprince: I couldn't figure out how to get openstack_vpc to talk to a cloud servers vpc on another box ... was I just missing something?
19:19:34 <termie> the queuing and such can be handled by jenkins, is what i am getting at
19:19:40 <mtaylor> termie: yes
19:19:56 <mtaylor> so - eventually I want to just have jenkins spin up the servers and do the chef bits
19:20:12 <termie> agreed, but i think we should be using a common tool for that
19:20:17 <mtaylor> but having jenkins call openstack_vpc should get us further today
19:20:19 <termie> and it sounds like vpc has the pieces for it
19:20:22 <mtaylor> yes
19:20:33 <termie> i just don't want another long running process to maintain
19:20:38 <dprince> So I'm already running each trunk commit in SmokeStack (via a Jenkins). I'm happy to move that move that bit over to the public Jenkins now if you guys would like.
19:21:07 <termie> that would be grand
19:21:11 <mtaylor> ++
19:21:24 <termie> have mtaylor make you admin'y?
19:21:29 <mtaylor> there are things I would like to do with that setup long term - but that gets us a thing we need today
19:21:37 <mtaylor> and we can talk about the other long term things later
19:21:41 <dprince> Yes.
19:21:42 <mtaylor> dprince: are you dprince on launchpad?
19:21:42 <termie> so
19:21:52 <dprince> I'm dan-prince.
19:21:59 <termie> we set up an openstackvpc instance (probably just steal titan's for now)
19:22:16 <mtaylor> dprince: ok. you are now an admin on the openstack jenkins
19:22:31 <dprince> Also. I'm happy to make Cloud Servers VPC available for anyone else to use as well.
19:22:37 <termie> we deploy a single and multinode setup as per anso's old vagrant testing, and run the nova smoketests against ahtat
19:22:41 <nati> Can we test VLANManager on VPC?
19:22:43 <termie> smokestack is just doing openstack api, right>?
19:23:03 <dprince> termie: I'm now running the nova smoketests too.
19:23:07 <termie> ooo
19:23:10 <mtaylor> great
19:23:13 <termie> so
19:23:14 <dprince> termie: minus volume tests.
19:23:31 <termie> ah, do we think there is a way to integrate those?
19:23:53 <dprince> The issue is I can't get iscsi-target to run on Cloud Servers.
19:23:54 <termie> i'd say in that case lets just ditch the first section of CIPlan and replace it with smokestack
19:23:58 <mtaylor> dprince: is it possible to run them with the xunit xml output and get those copied back to the jenkins box?
19:24:00 <termie> ah
19:24:21 <dprince> mtaylor: Sure. We can do that.
19:24:34 <termie> it sounds like titan has a strong lead on most of this
19:24:37 <mtaylor> termie: well - the whole thing here is that we want to get this integrated with jenkins so that we can eventually add it to the tarmac stuff
19:24:42 <mtaylor> yup
19:24:55 <dprince> So. On that front I actually have mixed feelings.
19:25:02 <termie> mtaylor: right, just saying integrate smokestack instead of writing your own bits
19:25:06 <mtaylor> totally
19:25:16 <termie> dprince: oh? do tell :)
19:25:21 <dprince> Is the idea that we would prevent (block the trunk commit) if the functional tests fail?
19:25:25 <mtaylor> if he's already got that work done on a different jenkins
19:25:34 <mtaylor> maybe - the idea is that I want to be able for us to make that choice
19:25:39 <mtaylor> physically
19:25:48 <mtaylor> so that the choice is policy and not lack of ability
19:25:52 <dprince> The model we've been following on Titan is we run branches in merge prop. Heavily. Its kind of a review tool.
19:25:57 <termie> dprince: i think that is a tarmac config issue rather than a jenkins config issue
19:26:07 <termie> dprince: we want that as well
19:26:18 <mtaylor> so on _that_ one _I_ have mixed feelings
19:26:22 <termie> dprince: we == ansoy people at least
19:26:29 <mtaylor> which is a security thing
19:26:39 <mtaylor> running heavily on branches from known folks is one thing
19:26:46 <termie> right
19:26:52 <mtaylor> running code from _any_ unreviewed merge prop is, well, fail
19:26:52 <dprince> Right. But what I'm getting at is any time you have functional tests there is a chance it could fail due to an external dependency. Something like the network was down, launchpad, GitHub, etc.
19:26:59 <mtaylor> totally
19:27:22 <dprince> I'd hate to hold up a trunk commit if there is a long lived dependency failure.
19:27:32 <mtaylor> I have no issues holding up a trunk committ
19:27:43 <mtaylor> the whole idea is to keep trunk in good shape
19:27:45 <termie> dprince: well, once we're on github some of those things will be a little bit easier to work around
19:27:47 <mtaylor> but that's a whole other thing
19:28:04 <termie> dprince: it is relatively easy to integrate things into your dev tree and take them back out later
19:28:11 <mtaylor> I don't think any part of this has anything to do with github v. launchpad - but that's _also_ a different conversation
19:28:13 <dprince> Sure. I just wanted to make sure everyone was on the same page.
19:28:29 <mtaylor> yup. for now...
19:28:42 <mtaylor> dprince: can I put you down with an action of getting smokestack jenkins job done?
19:28:43 <dprince> Also the tests as is take 15 minutes. Is that everyone acceptable with that?
19:29:01 <dprince> mtaylor: Sure. I can take that.
19:29:12 <mtaylor> #action dprince make smokestack jenkins job
19:29:24 <termie> for now we don't automatically run against merge props and will try gating merges on them with the option that we might need to turn that off
19:29:35 <mtaylor> yes.
19:29:36 <termie> titan can continue doing it for their own stuff, obvs
19:30:00 <mtaylor> although gating merges will have to wait for now on some jenkins/tarmac work - but that should be done soon enough - for now, notification will be a huge win
19:30:33 <termie> step 1, public smokestack, step 2, tarmac integration
19:30:38 <mtaylor> yup
19:30:50 <termie> want to move to baremetal bits?
19:30:50 <dprince> Sure. So just to be clear. You want me to make a simple Jenkins job that runs each time we have a trunk commit. This gives you the indicator light on Jenkins you want?
19:31:01 <mtaylor> yes
19:31:02 <mtaylor> and
19:31:06 <termie> dprince: we'll want to get nosexunit out
19:31:14 <mtaylor> dprince: also - take a peek at the nova-sometest job at the nosexunit output stuff
19:31:19 <mtaylor> because we want that too
19:31:28 <dprince> Sure.
19:31:33 <mtaylor> baller
19:31:48 * mtaylor will purchase for dan prince a beer
19:31:58 <dprince> One the bare metal thing. I've got 4 nodes that I'm working on getting integrated with XenServer testing.
19:32:07 <termie> are we moving to baremetal now?
19:32:12 <termie> meetingbot
19:32:18 <mtaylor> #topic Jenkins job to integrate/fire bare metal testing
19:32:36 <termie> "< dprince> One the bare metal thing. I've got 4 nodes that I'm working on getting integrated with XenServer  testing.
19:33:00 <termie> i am wondering what rpath is offering
19:33:18 <mtaylor> rpath is offereing something for this similar to what dprince just offered for functional
19:33:23 <termie> we had a nicely working pxe + ubuntu bootstrap setup working already
19:33:44 <termie> that only needed the chef provisioning bits on the client side
19:33:45 <dprince> For bare metal testing I'm actually very interested in using Dells Crowbar project.
19:33:49 <mtaylor> they currently have a jenkins running jobs that inject images in to cobbler and stuff
19:33:55 <termie> dprince: everybody is but it doens't exist yet
19:34:07 <mtaylor> I'm less interested in crowbar unless it's only one of the solutions
19:34:13 <termie> dprince: and many of us are a bit tired of waiting on them
19:34:15 <mtaylor> because just testing dell is mega uninteresting to me
19:34:16 <dprince> termie: I'm asking for the bits as we speak...
19:34:33 <dprince> (via emails, etc).
19:34:37 <termie> the chef scripts would theoretically be the same
19:34:41 <dprince> Sure.
19:34:57 <termie> so all we really need is a simple script to kick off chef clientside
19:35:03 <dprince> I'm working on a XenServer cookbook to help setup the dom0 and domu.
19:35:11 <termie> cool
19:35:26 <termie> our side doesn't know how to do that, we're still kvm-ville
19:36:01 <termie> for the moment though i'd love to get what we already had nearly working finished
19:36:40 <nati> cool
19:36:46 <mtaylor> yup. and I can turn my attention to that now that we're considering functional done via smokestack
19:37:00 <mtaylor> termie: did you have a reason for doing pxe stuff directly and not using cobbler or something similar?
19:37:24 <termie> mtaylor: it was dead simple and didn't require learning much as we were already doing it for nasa
19:37:27 <dprince> How many machines do you have access to?
19:37:34 <termie> dprince: i think 20
19:37:37 <mtaylor> 10
19:37:39 <mtaylor> we have 10 machines
19:38:03 <termie> you have 10 or we gave you 10?
19:38:13 <dprince> Cool. Well we have 4 on titan. Not as many but they are big boys. 48 Gigs of memory, lots of disk.
19:38:14 <mtaylor> someone gave me access to 10
19:38:40 <termie> mtaylor: was it us who gave you access? i am trying to ask whether you are uysing the ones we allotted or some other mystery set
19:38:53 <mtaylor> termie: the ones from jesse - so probably
19:39:03 <mtaylor> termie: "us" is very nebulous to me most of the time
19:39:41 <termie> mtaylor: ex-anso is my usual us when i am acting as spokesperson
19:39:58 <termie> we don't really have a good line of distinction
19:40:05 <mtaylor> fair - I guess I should be more clear - that doesn't mean anything to me either ... but it's really not important right now
19:40:07 <termie> but anso is the easiest
19:40:44 <mtaylor> in any case- I have access to 10 machines in the equinix facility in the bay area- I believe these are from "you"
19:40:51 <termie> okies
19:41:25 <termie> so i will assume we can't give you any more
19:41:34 <mtaylor> I'm also assuming that
19:42:16 <mtaylor> the thing we talked about at ODS was eventually making a setup that had 5 machines allocated to swift, 4 to nova and 1 to glance which would also be the driver of the thing
19:42:36 <mtaylor> that, obviously, will not be what we do in the short term - but that's the aim
19:42:50 <dprince> Why so many to swift?
19:43:06 <termie> i'd say 4 nova, 4 swift 1 glance and one orchestrator
19:43:21 <dprince> I'd go for more nova myself.
19:43:22 <termie> 4 swift just to test replication and have a separate proxy
19:43:34 <dprince> Okay. I see.
19:43:40 <mtaylor> there was a reason they said 5 instead of 4 - jaypipes do you remember what it was?
19:43:50 <mtaylor> because I'd love to have the orchestrator be separate
19:44:14 <termie> same here
19:44:27 <termie> because the other machines hsould probably be in a different network config
19:44:45 <mtaylor> ok. let's say for sake of argument that we do that until such a time as someone comes in and says "dear god! we have to have 5 for swift"
19:45:36 <jaypipes> mtaylor: no, I can't remember why.
19:45:50 <mtaylor> we also have, while we're on the subject, an offer of machines from the Novel/Microsoft Interoperability Lab to ensure that we have a deployment testing the hyperv stuff
19:46:46 <mtaylor> should probably be easier to make use of that once we have the first set of these up and going
19:48:02 <termie> so action items for monty?
19:48:08 <mtaylor> so - actions coming out of this week on this topic are that I'm going to get a jenkins job that fires off termie's pxe boot work. we're waiting on proper multi-machine chef recipes, yeah?
19:48:21 <termie> nope, we have those
19:48:28 <termie> but they will probably have to be updated for baremtetal
19:48:46 <termie> the stuff i gave you kicks off a chefserver in lxc
19:48:56 <mtaylor> #action mtaylor jenkins job that fires off pxe
19:48:57 <mtaylor> yup
19:49:00 <mtaylor> saw that
19:49:22 <termie> we have probably additional knowledge on some of that now
19:49:26 <mtaylor> the key there then is getting the right chef stuff running on the machines once they are booted
19:49:27 <mtaylor> great!
19:49:29 <termie> so you can ask us questions
19:49:40 <termie> yeah, but it sounds like dprince might know some of that
19:49:44 <mtaylor> I'd love to - as chef makes me want to rip my arms off
19:49:46 <termie> that was where i handed things to you
19:49:58 <termie> so i don't know how to actually kick off a chef client
19:50:02 <termie> it makes me want to rip my arms also
19:50:11 <termie> although they claim it is getting better
19:50:12 <mtaylor> dprince: I'll be coming at you for questions on that
19:50:28 <dprince> mtaylor: Sure. Happy to help.
19:50:33 <mtaylor> termie: I keep expecing something like a command line "chef nova-node"
19:50:42 <termie> yeah me too
19:50:45 <mtaylor> termie: but instead apparetnly I have to write json files :)
19:50:48 <termie> but it is like, get this key from somewhere
19:50:56 <mtaylor> ok. as long as it's not just me
19:51:14 <termie> vagrant does it, too, if you want to read some ruby
19:51:15 <mtaylor> for now, I'll make sure I get the first bits up and kicking, and then I'll bug dan on the next step
19:51:42 <mtaylor> it does - and there's too much magic in the vagrant files to help me figure out what to do on the command line
19:51:47 <mtaylor> darned "helpful" things
19:52:07 <mtaylor> whatever - I'm sure I'll be a chef expert by the time this is all said and done
19:53:16 <mtaylor> next agenda section probably shouldn't be its own section - but I just wanted to make sure to mention that at some point figuring out how we verify the chef and puppet stuff is probably a good idea - but I don't feel like solving that today
19:53:33 <termie> i think we effectively punted that earlier
19:53:34 <mtaylor> we're running short on time - anybody got anything else on bare metal for now?
19:53:49 <termie> only thing i would asdd is due dilligence to see what loren has
19:54:05 <mtaylor> loren?
19:54:25 <termie> USC guy who has done a bunch of baremetal stuff
19:54:29 <termie> actually probably not useful for now
19:54:35 <termie> it is for provisioning baremetal from nova
19:54:41 <termie> let's skip
19:54:44 <mtaylor> ah. cool.
19:54:46 <termie> i'll keep up with him
19:55:01 <mtaylor> I'm going to combine a couple of topics real quick:
19:55:07 <mtaylor> #topic Jenkins Plugins
19:55:14 <mtaylor> anybody want to do any Java hacking?
19:55:18 <termie> we don't need any right now, right?
19:55:43 <termie> you want to replace tarmac, i guess, but do we need to?
19:55:51 <mtaylor> well - we need some at some point - and I always want to ask if there's anyone lurking who wants to help
19:56:00 <mtaylor> we're getting close to the point where we need to
19:56:28 <termie> why do we need to?
19:56:35 <mtaylor> as we add additional tests for things in Jenkins, as long as we're using tarmac or roundabout, we have to double implement those outside of jenkins if we want them as part of the gatekeeping
19:56:52 <termie> what is being double implemented?
19:57:15 <mtaylor> well, nothing right now - we're just not adding those things to the gatekeeper as of yet
19:57:29 <mtaylor> but our friends at ntt wanted to add code coverage metric counts to the gatekeeping
19:57:53 <mtaylor> and since we're already tracking that in jenkins - if dependent jobs worked for us, that would be a piece of cake
19:58:23 <mtaylor> but the tarmac/roundabout is ignorant of jenkins
19:59:09 <termie> how is it ignorant of jenkins?
19:59:11 <mtaylor> in any case - obviously not this week - but I wanted to check with folks. however, as is usually the case, there are not 100s of java devs clamoring to help out - so we'll get to it when we get to it
19:59:12 <termie> it rungs the tests
19:59:25 <termie> so, not that i want to
19:59:30 <termie> but i can be a java devloper
19:59:30 <mtaylor> it runs tests that are defined in a config file on the file system - the only interaction is that jenkins runs tarmac
19:59:35 <termie> but i don't think we need it
19:59:45 <ttx> one minute left :)
19:59:58 <mtaylor> tarmac can't pass/fail things based on the outcome of jenkins jobs
20:00:10 <termie> really? i thought that was exactly what it did
20:00:13 <mtaylor> nope
20:00:22 <termie> that is what roundabout does
20:00:25 <mtaylor> nope
20:00:35 <mtaylor> or - is it really?
20:00:37 <termie> ... yeah
20:00:51 <termie> it runs the tests in jenkins and then if it fails it dumps info into th elof
20:00:54 <termie> log
20:00:56 <termie> s/log/issue
20:01:01 <mtaylor> ok - I'll re-look at it - I thought it did the same thing that tarmac did
20:01:03 <termie> pretty sure
20:01:07 <termie> plz double check though
20:01:09 <jbryce> time to wrap up?
20:01:10 <mtaylor> I will
20:01:11 <termie> yeah
20:01:22 <mtaylor> #action mtaylor will look at roundabout jenkins triggering
20:01:27 <mtaylor> ok guys - till next week
20:01:28 <termie> okay wrap up
20:01:29 <mtaylor> #endmeeting