13:28:40 <esberglu> #startmeeting powervm_ci_meeting
13:28:41 <openstack> Meeting started Thu Jan 19 13:28:40 2017 UTC and is due to finish in 60 minutes.  The chair is esberglu. Information about MeetBot at http://wiki.debian.org/MeetBot.
13:28:42 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
13:28:44 <openstack> The meeting name has been set to 'powervm_ci_meeting'
13:28:58 <thorst> o/
13:29:21 <wangqwsh> :)
13:31:13 <esberglu> #topic status
13:31:38 <efried> o/
13:31:59 <esberglu> Yesterday I worked through some fixes for adreznec's DNS / apt patch
13:32:00 <xia> o/
13:32:11 <esberglu> It's pretty much working now, a couple small changes are still needed
13:32:26 <thorst> esberglu: pretty much working in that test runs have used it or...
13:32:53 <esberglu> Yeah. Test runs have used it on staging
13:33:00 <thorst> and?!?
13:33:04 <thorst> faster stronger?
13:35:03 <esberglu> They were faster, mostly finishing in under an hour
13:35:10 <thorst> weird...and neat
13:35:21 <thorst> can you PM me the zuul from one of them?
13:35:28 <thorst> I'd like to compare devstack times.
13:35:56 <esberglu> But the times are usually a little different between staging and prod
13:36:08 <thorst> yeah, but devstack times shouldn't be too different
13:36:10 <thorst> hopefully
13:36:16 <thorst> run times...definitely different
13:37:37 <esberglu> As part of deploying that patch I realized that the CI nodes (mgmt, ctrl, log) were 14.04, I thought we had upgraded a while back
13:37:40 <esberglu> But apparently not
13:37:43 <adreznec> Yeah, we'll have to do a longer term comparison to be sure
13:37:48 <adreznec> How did the upgrade go esberglu?
13:37:57 <esberglu> Not as well as I was hoping
13:38:02 <esberglu> The ctrl node is fine
13:38:04 <thorst> upgrade?
13:38:06 <adreznec> Yeah, was afraid of that
13:38:09 <thorst> o...
13:38:10 <esberglu> To 16.04
13:38:11 <adreznec> 16.04
13:38:16 <esberglu> On the staging CI servers
13:38:22 <thorst> did we capture the VM images for the controller nodes before upgrading?
13:38:26 <thorst> I could just snapshot them
13:38:32 <thorst> so you can fall back safely
13:38:51 <thorst> too late for staging, but maybe something for prod?
13:38:52 <adreznec> Well worst case scenario we just run the playbooks again right? :)
13:39:00 <thorst> word...that's why we built them
13:39:14 <esberglu> Seeing some issues with the log and mgmt nodes
13:39:30 <adreznec> What went awry? bad config? missing packages?
13:39:40 <esberglu> For some reason puppet is downloading all the modules to the / directory on the log node
13:39:55 <esberglu> When running install_modules.sh
13:40:21 <adreznec> Hmm. Maybe the path it was trying to use doesn't exist or something...
13:40:31 <esberglu> The mgmt node had some bad packages and some stuff with starting/stopping services
13:41:00 <esberglu> I just upgraded them last night so I haven't dug too deep into the problems yet
13:41:33 <esberglu> #action esberglu: Get staging CI nodes running on 16.04
13:41:57 <adreznec> Ok. Yeah these are pretty typical issues due to the upstart/systemd conversion
13:42:20 <esberglu> And after that start hitting the dual CI
13:42:48 <esberglu> #topic Dual CI
13:42:54 <esberglu> I have some questions about this
13:43:04 <esberglu> How do we want to handle voting?
13:43:24 <thorst> no vote yet.  Similar to what nova-powervm does (once stable of course...don't publish until stable)
13:43:37 <thorst> different CI name though.  Something that indicates in tree status
13:45:08 <esberglu> Okay. Has somebody been testing the in tree with devstack?
13:45:14 <esberglu> So I can steal their local.conf?
13:45:25 * thorst looks to efried
13:45:35 <adreznec> The PowerVM Oak CI. Or maybe maple? elm?
13:45:50 <efried> I have a local.conf
13:45:51 <thorst> har har har...
13:46:36 <efried> I don't get it
13:46:49 <efried> oh
13:46:52 <efried> tree
13:46:56 <thorst> yah...
13:47:00 <efried> got it
13:47:21 <efried> bore (in-tree) vs bark (oot)
13:47:32 <efried> or is it bole?
13:47:59 <efried> bole
13:48:02 <efried> anywhooo
13:48:32 <adreznec> lol
13:48:53 <adreznec> Ok, so efried is going to give his local.conf to esberglu and we'll go from there
13:49:15 <esberglu> That's pretty much all I have for CI. Get the staging nodes up and get the dual CI running there
13:49:29 <esberglu> I might have some q's as I get further into the dual CI stuff
13:49:35 <thorst> xia: I know you were working with wangqwsh.  I know that wangqwsh is helping efried with the in tree driver, but has experience with the CI.
13:49:42 <efried> not sure how we handle the fact that you have to spawn with no network
13:49:47 <thorst> wondering out loud if it would make sense for you to become familiar with the CI as well.
13:50:00 <thorst> efried: we can deploy w/ network, that network just won't get attached to the VM.
13:50:04 <thorst> so it'll fake work
13:50:13 <efried> it broke when i tried it
13:50:28 <thorst> what the what?  OK...I'll check out the error later.
13:50:28 <efried> maybe a code change is needed.
13:50:33 <efried> thx
13:50:37 <wangqwsh> throw a lot of error about networking
13:51:06 <thorst> ahh, I bet I know what it is...bet you'd get around it using the SEA agent for now...
13:51:19 <efried> #action efried & thorst to investigate spawn "with" network on the first few in-tree change sets
13:51:20 <thorst> no, that'd be broken too.
13:52:16 <efried> back to #topic wangqwsh helping with CI?
13:52:46 <thorst> well, actually it would be xia getting some experience there from wangqwsh
13:52:57 <thorst> we clearly need backups for esberglu when he decides to go on vacation
13:53:00 <wangqwsh> um.
13:53:12 <thorst> and not just backups, but also driving some of that CI dev
13:53:22 <esberglu> Yeah with all 5 days I have left for the year...
13:53:24 <thorst> not sure its right to do that now...but at some point.
13:53:34 <thorst> something to consider.
13:54:16 <esberglu> I have been adding wangqwsh to the neo-os-ci reviews. I can add xia as well. Just to give some exposure
13:54:39 <thorst> yeah, exposure is a good start.  I know wangqwsh and xia, you're both also in the driver code as well
13:54:47 <thorst> so if that's the focus now...keep it as the primary
13:55:25 <thorst> that's all I had  :-)
13:56:03 <esberglu> Anyone have anything else?
13:56:51 <efried> not I
13:57:03 <wangqwsh> nope
13:58:20 <esberglu> #endmeeting