19:01:07 <mtaylor> #startmeeting
19:01:08 <openstack> Meeting started Tue Mar 20 19:01:07 2012 UTC.  The chair is mtaylor. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:09 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic.
19:01:15 <mtaylor> #topic OpenStack CI Stuff
19:01:30 <LinuxJedi> yay :)
19:02:33 <mtaylor> sup?
19:03:06 <jeblair> mtaylor: you lookin at me?
19:03:08 <soren> o/
19:03:15 <mtaylor> so, I'm going to try to power through - I'm finalizing some slides for a talk I'm giving in a few hours
19:03:17 <LinuxJedi> \o/
19:03:18 <mtaylor> :)
19:03:53 <mtaylor> jeblair: which made me think - SUPER low priority - but if we're keeping our talks in slidy in git, we could actually publish them somewhere
19:04:05 <LinuxJedi> mtaylor: you mean you just started some slides for a talk in a few hours? ;)
19:04:34 <mtaylor> LinuxJedi: nah. not _just_ started - I am cannibalizing from some previous slides
19:04:40 <LinuxJedi> lol :)
19:05:06 <mtaylor> LinuxJedi, devananda: not sure if we pointed either of you at it, but we have sets of slides at http://github.com/openstack-ci/publications
19:05:21 <mtaylor> all in html :)
19:05:26 <LinuxJedi> I didn't know that, but awesome :)
19:05:33 <devananda> nice
19:06:02 <mtaylor> SO ... status of things ... jeblair is working on devstack-gate stuff, and linuxjedi has puppet/css stuff
19:06:02 <jeblair> we should probably add a readme to make this explicit, but i feel very generous about those, i totally think people should cannibalize, reuse them, (contribute fixes!) as they see fit.
19:06:12 <mtaylor> jeblair: ++
19:06:24 <mtaylor> jeblair: perhaps there's an appropriate createive commons license for that?
19:06:37 <jeblair> mtaylor: i bet there is.
19:06:47 * LinuxJedi would be surprised if there isn't
19:06:58 <mtaylor> jeblair, LinuxJedi: you guys wanna tell folks about the cool stuff you've been doing?
19:07:07 <mtaylor> my only status update is that I've been breaking things
19:07:17 <LinuxJedi> sure, who first?
19:07:17 <mtaylor> and working on hiring more people so that I don't have to touch things and break them any more
19:07:22 <mtaylor> LinuxJedi: you go
19:07:23 <LinuxJedi> mtaylor: you fixed something today!
19:07:29 <LinuxJedi> ok, so...
19:07:29 <mtaylor> accident
19:08:00 <LinuxJedi> for those who weren't here last week, I have been skinning Jenkins and Gerrit
19:08:26 <LinuxJedi> the skins for this can be seen on https://jenkins-dev.openstack.org/ and https://review-dev.openstack.org/
19:08:41 <LinuxJedi> this is currently pending code review and will go live ASAP
19:09:18 <mtaylor> LinuxJedi: I keep meaning to look - it's parameterized such that you could use your nifty stackforge logo on stackforge jenkins/gerrit, yeah?
19:09:22 <LinuxJedi> In other news... I have been working on getting puppet to automagically create jobs when given a new project name
19:09:43 <LinuxJedi> mtaylor: it isn't yet, because I need other things reviewed and to land first
19:10:05 <LinuxJedi> mtaylor: I have 4 things pending review for the puppet repo now ;)
19:11:01 <LinuxJedi> I have a prototype for this which seems to work and is pending review also.  It sets up all the jobs we are typically usign right now with tox
19:12:03 <LinuxJedi> the final thing I have been working on is getting our puppet scripts to work with Openstack and Stackforge.  That is also pending review, and will have some extra stuff to go in once everything else lands (like a cool logo for Stackforge)
19:12:36 <mtaylor> all of that will be quite sweet
19:12:53 <mtaylor> I've been punching myself in the face with slides today, so I haven't gotten to fully review your new template jobs work
19:12:57 <LinuxJedi> I may finally get the CI docs stuff I've been putting off for the last few weeks done this week ;)
19:13:04 <mtaylor> LinuxJedi: woot!
19:13:20 <mtaylor> LinuxJedi: I betcha that working through some of those things with devananda will be good motivation to do that :)
19:13:42 <LinuxJedi> mtaylor: no worries, but I am getting to the point soon where I can't do much more puppet stuff until they land
19:13:55 <LinuxJedi> mtaylor: very much so
19:14:10 <mtaylor> speaking of - for those playing along at home - we're still working on getting devananda access to things, but he's going to investigate gerrit 2.3 for us once we have him some machine resources :)
19:14:22 <LinuxJedi> ++
19:14:49 <soren> You can't spare him a cloud instance?
19:14:53 <mtaylor> soren: have you been following any of our chatting about driving the standard jenkins jobs for the projects from our puppet modules?
19:15:04 <LinuxJedi> soren: we maxxed out mtaylor's account already ;)
19:15:07 <soren> mtaylor: I... no. I don't think so.
19:15:08 <mtaylor> soren: I just got him his own cloud account! (it just took a while)
19:15:28 <mtaylor> soren: since you also poke at things, it would be nice to get your thoughts on LinuxJedi's patch
19:16:03 <soren> mtaylor: I'm guessing the goal is to more declaratively be able to add more projets?
19:16:03 <LinuxJedi> soren: https://review.openstack.org/5531
19:16:15 <mtaylor> soren: yes - and to manage them long term consistently
19:16:26 <mtaylor> soren: and have those things all code reviewed :)
19:17:00 <soren> Cool. I've been trying to do something similar just with Jenkins.. It's not a lot of fun.
19:17:58 <LinuxJedi> oh, also high up on my todo list is to knock up a quick homepage for Stackforge, I had everything ready to do that and other things got in the way :)
19:18:20 <mtaylor> ++
19:19:02 <mtaylor> soren, LinuxJedi I imagine we'll go through a couple of iterations before we get totally satisfied - but I think LinuxJedi's work thus far is a huge step towards perfection
19:19:20 <mtaylor> cool.
19:19:30 <mtaylor> jeblair: how about you? wanna talk about devstack-gate?
19:19:31 <soren> mtaylor: I'm not very comfortable with it landing right now, though.
19:19:33 <soren> Release is soon.
19:19:53 <soren> I'd like us to wait until a couple of days after (to leave room for a potential quick point release).
19:19:56 <LinuxJedi> soren: well, even when it lands it doesn't have to be live for Openstack, it can be used on Stackforge first
19:19:58 <mtaylor> soren: agree. I think it can land in the puppet repo without us actually configuring it to do things
19:20:04 <mtaylor> what LinuxJedi said
19:20:05 <soren> Ah.
19:20:07 <soren> Ok
19:20:19 <LinuxJedi> but we need stackforge puppet to land before I can do that
19:20:20 <soren> That's somewhat comforting.
19:20:24 <mtaylor> stackforge is becoming really useful as a staging environment for openstack-ci :)
19:20:31 <LinuxJedi> hehe :)
19:20:49 <jeblair> we've already not upgraded gerrit because of what we learned from stackforge.  :)
19:21:22 <LinuxJedi> oh yea, no news on the gerrit 2.2.2 performance bug (I don't expect it will be fixed with high priority).  I'm hoping 2.3 will be better
19:21:38 <soren> Question: We deploy a set of services and run some tests against it for gating, right? On metal?
19:21:58 <mtaylor> soren: against what?
19:21:58 <soren> I'm mostly thinking of Nova here. As I do.
19:22:04 <mtaylor> ah. yes
19:22:12 <soren> Exactly what set of tests?
19:22:14 <jeblair> we don't test on metal anymore
19:22:23 <mtaylor> (but would like to again at some point)
19:22:31 <soren> And if I can deliver something else against which the same set of tests passes, can we gate on that?
19:22:38 <soren> jeblair: Oh.
19:22:41 <soren> jeblair: How come?
19:23:00 <jeblair> we lost access to those machines, no one at rax stepped up to help.
19:23:03 <LinuxJedi> scrap metal merchants gave us a good deal
19:23:15 <soren> Ah.
19:23:21 <soren> .(
19:23:23 <soren> :(
19:23:41 <mtaylor> soren: we have a new set of machines at hp
19:23:49 <mtaylor> soren: getting them bootstrapped is on my todo list
19:24:27 <soren> Ok.
19:24:40 <soren> These feel like stupid questions, but I don't know anymore how to find the answers myself.
19:24:48 <devananda> mtaylor, might ^ be a good project for me?
19:24:56 <soren> ..but what is the current set of tests run before anything gets merged into Nova?
19:25:21 <jeblair> unit tests(26 and 27), pep8, and devstack exercise.sh
19:25:34 <jeblair> i think we're really close to adding devstack-configured tempest to that
19:25:47 <soren> So we spin up a cloud instance, run devstack and its exercise.sh.
19:25:53 <jeblair> yep
19:25:56 <soren> Cool.
19:26:03 * soren makes sure his stuff passes the same set of tests.
19:26:03 <jeblair> that's the "devstack-gate"
19:26:34 <soren> jeblair: I see no such job.
19:26:41 <jeblair> soren: https://review.openstack.org/#change,5554
19:26:56 <jeblair> you can look at any merged change to nova, and see the jenkins report
19:27:03 <jeblair> it will have links to all the gating jobs
19:27:17 <jeblair> the actual job is gate-integration-tests-devstack-vm.  :)
19:27:39 <LinuxJedi> incidentally the job that we had issues with for the RC1 release this morning ;)
19:27:40 <soren> Where is this configured (so that I won't have to ask these silly questions another time)?
19:27:51 <mtaylor> devananda: YES
19:28:01 <LinuxJedi> soren: https://jenkins.openstack.org/view/Gate/job/gate-integration-tests-devstack-vm/
19:28:16 <soren> LinuxJedi: Right, right. I mean..
19:28:17 <soren> er...
19:28:29 <jeblair> soren: answering the question of "what are the current gating jobs for a project"?
19:28:32 <soren> I mean, how could I have determined that that was one of the jobs that would be..
19:28:35 <soren> Right.
19:28:38 <soren> Taht.s
19:28:39 <soren> That.
19:28:44 <jeblair> soren: that's trickier.  really the easiest way is to look at the most recent merged change.
19:28:58 <soren> jeblair: Let's pretend that I wanted to add something to the list.
19:29:05 <soren> (I don't, but I think this is the asnwer I'm looking for)
19:29:10 <jeblair> because the actual answer is "whatever jobs are configured with a gerrit trigger"
19:29:17 <soren> Ah.
19:29:23 <LinuxJedi> soren: the job watches gerrit, so it is whatever jobs are watching
19:29:41 <jeblair> soren: so that's why we've been naming them with gate-
19:29:42 <LinuxJedi> and it isn't easy to reverse-lookup
19:29:45 <soren> Is it Jenkins that ultimately merges?
19:30:02 <soren> Or Gerrit?
19:30:23 <jeblair> if you search for jobs titled "gate-nova-*" you'll get all the nova gating jobs, but devstack violates that naming convention since it watches _many_ projects.
19:30:32 <jeblair> jenkins tells gerrit to merge
19:30:36 <soren> If Gerrit, how does it know to trust (only) our Jenkins?
19:30:46 <jeblair> the jenkins user is the only one with that permission
19:30:51 <vishy> jeblair / mtaylor : where and what is this stackforge thing?
19:30:57 * soren is trying to catch up with how everything works.
19:31:02 <LinuxJedi> vishy: I can answer that...
19:31:22 <soren> jeblair: Ah, I see. Ok.
19:31:23 <LinuxJedi> vishy: stackforge is like Openstack Jenkins/Gerrit but for satellite type projects
19:31:34 <soren> jeblair: Cool. Thanks.
19:31:35 <devananda> soren, this is a good discussion for me as well, I'm still learning all these interdependencies ...
19:31:39 <jeblair> soren: i think that's great.  we could probably use an updated set of "here's the design philosophy for the system" type docs.  we've been very focused on operational ones.
19:31:46 <LinuxJedi> vishy: currently only one project on it, a second coming very soon
19:31:59 <vishy> LinuxJedi: where is it?
19:32:02 <LinuxJedi> vishy: jenkins.stackforge.org and review.stackforge.org
19:32:19 <LinuxJedi> vishy: I'm hopefully creating a homepage for it this week and there are other subdomains to come
19:32:29 <LinuxJedi> vishy: RedDwarf is next to go on it
19:32:56 <vishy> MRaaS nice
19:33:30 <LinuxJedi> vishy: we are also using it to test things like Gerrit 2.2.2 before it goes to Openstack
19:33:58 <LinuxJedi> and we are using puppet master there instead of invoking with cron
19:35:08 <jeblair> so here's what i was up to last week: https://review.openstack.org/#change,5481
19:35:34 <jeblair> that updates the devstack gate job to change the way we build the devstack vm images
19:35:46 <jeblair> and adds support for multiple image versions and providers
19:36:09 <jeblair> so we should be ready to handle things like "run this branch on precise, and this branch on oneiric" if that comes up
19:36:23 <soren> Cool.
19:36:42 <jeblair> and we now have accounts in rackspace legacy, the rackspace nova beta-test system, and hpcloud
19:36:55 <jeblair> so next up is getting devstack nodes in all three of those providers
19:37:23 <jeblair> we'll use all of them all the time, and we'll effectively have an HA system
19:37:30 <soren> "devstack nodes"?
19:37:32 <jeblair> so if one fails, we'll continue using nodes from others.
19:38:01 <jeblair> soren: throwaway vms that are booted, have devstack installed and run on them, run exercise.sh (and soon tempest), then deleted.
19:38:14 <vishy> jeblair: that would be awesome
19:38:34 <jeblair> we boot them ahead of time and maintain a pool of them that are ready for immediate use by the test scripts.
19:38:40 <vishy> jeblair: so tired of devstack gate blow up because raxcloud is borked
19:38:56 <soren> Oh, gotcha. I misunderstood. Thought you meant we'd have some sort of persistent thing called "devstack nodes" at each provider.
19:39:00 <jeblair> vishy: yes, and pvo and i are both excited about helping to test an actual nova deployment.  :)
19:39:23 <LinuxJedi> s/test/destroy
19:39:34 <jeblair> soren: nope, though we do generate images...
19:40:00 <jeblair> that have all of the .debs, python packages, test images, and code repos cached ahead of time
19:40:18 <jeblair> so that we minimize network dependencies and startup time
19:40:38 <soren> Does the OpenStack cloud at RAX let you upload your own images?
19:40:46 * soren hasn't been blessed with an account
19:41:11 <jeblair> i don't know, but rackspace legacy and hpcloud both don't, so we have to work with that.
19:41:30 <jeblair> instead, the scripts start with a provider base image, cache everything, then take a snapshot.
19:41:31 <soren> So what /do/ you build images for?
19:41:36 <soren> Oh.
19:41:38 <soren> Gotcha.
19:41:47 <jeblair> then we spin up the nodes from that snapshot
19:42:12 <jeblair> so the challenge for me this week is that there is no single library that can work with all three providers
19:42:18 <pvo> soren: not yet.
19:42:33 <jeblair> novaclient works with hpcloud and rax-nova, but not rax-legacy
19:42:59 <jeblair> libcloud works with rax-legacy and (trunk version:) rax-nova, but not hpcloud
19:42:59 <pvo> jeblair: it sorta works… not 100% though
19:43:05 <pvo> I can get to do some basic things
19:43:19 <jeblair> pvo: novaclient and rax-legacy?
19:44:15 <pvo> jeblair: yea, I can get it to sorta work.
19:44:43 <jeblair> oh, i didn't think it worked at all after they removed the v1_0 stuff.
19:45:13 <vishy> jeblair: the old python-openstack-compute code works
19:45:26 <vishy> jeblair: I use that when I'm talking to legacy
19:45:43 <mtaylor> vishy: yeah - problem is wanting to have one script which talks to all three
19:47:56 <jeblair> we need to list images, sizes, and machines; boot machines; create an image from a running machine; and delete machines
19:48:32 <vishy> mtaylor: patch legacy compatibility back into novaclient :)
19:48:41 <jeblair> vishy: i actually looked into that
19:48:48 <jeblair> i'm not sure it ever supported creating an image
19:49:24 <chmouel> jeblair: yeah it did
19:49:35 <soren> If it's the thing based on jacobian's thing, then it did.
19:49:48 <mtaylor> vishy: I poked bcwaldon a while back about adding support back in
19:49:50 <jeblair> chmouel: cool; did it have a different name than the method used in 1.1?
19:50:16 <chmouel> jeblair: let me check
19:50:55 <jeblair> i saw the "backup" method, but that seemed to be about setting backup schedules, not immediately creating a named image...
19:50:59 <chmouel> jeblair: just the standard create https://github.com/chmouel/python-cloudservers/blob/master/cloudservers/images.py#L39
19:51:11 <jeblair> oh. of course.
19:51:14 <jeblair> it's on the image class. :)
19:52:05 <jeblair> servers.create_image exists in 1.1, so i assumed if 1.0 had it, it would be something similar
19:52:13 <jeblair> okay, so novaclient is back on the menu.  :)
19:52:29 <mtaylor> woohoo
19:52:40 <jeblair> i could add 1.0 support back in and see if that works with all three.
19:52:58 <jeblair> or i can work on getting libcloud to work with hpcloud (and rax-nova if further work is needed)
19:52:59 <mtaylor> that would get novaclient inline with what we've talked about for it moving foward - which is that it should just know how to talk to openstack clouds
19:53:00 <chmouel> we probably need just a python-cloudservers with custom auth url if novaclient doesn't want 1.0 support
19:53:27 * mtaylor thinks it would be way more friendly if novaclient could do it without killing people
19:53:34 <jeblair> chmouel: well, i don't want to use two libraries to talk to all our providers
19:53:46 <jeblair> (i mean, we can already do that today with libcloud and novaclient if we wanted to)
19:54:12 <mtaylor> jeblair: I vote on getting buy on on adding 1.0 support to novaclient ... and if that's a non-starter, then fix libcloud
19:54:15 <chmouel> ok make sense
19:54:29 <mtaylor> (we should make our own tools kickass where we can)
19:54:36 <jeblair> i agree
19:55:35 <jeblair> that was very helpful, thanks chmouel, vishy, pvo.
19:55:50 <LinuxJedi> anything else before we hit the hour?
19:56:13 * jeblair EOLs
19:56:44 <mtaylor> thanks everybody! that was super helpful
19:56:48 <mtaylor> #endmeeting