13:28:40 <esberglu> #startmeeting powervm_ci_meeting 13:28:41 <openstack> Meeting started Thu Jan 19 13:28:40 2017 UTC and is due to finish in 60 minutes. The chair is esberglu. Information about MeetBot at http://wiki.debian.org/MeetBot. 13:28:42 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 13:28:44 <openstack> The meeting name has been set to 'powervm_ci_meeting' 13:28:58 <thorst> o/ 13:29:21 <wangqwsh> :) 13:31:13 <esberglu> #topic status 13:31:38 <efried> o/ 13:31:59 <esberglu> Yesterday I worked through some fixes for adreznec's DNS / apt patch 13:32:00 <xia> o/ 13:32:11 <esberglu> It's pretty much working now, a couple small changes are still needed 13:32:26 <thorst> esberglu: pretty much working in that test runs have used it or... 13:32:53 <esberglu> Yeah. Test runs have used it on staging 13:33:00 <thorst> and?!? 13:33:04 <thorst> faster stronger? 13:35:03 <esberglu> They were faster, mostly finishing in under an hour 13:35:10 <thorst> weird...and neat 13:35:21 <thorst> can you PM me the zuul from one of them? 13:35:28 <thorst> I'd like to compare devstack times. 13:35:56 <esberglu> But the times are usually a little different between staging and prod 13:36:08 <thorst> yeah, but devstack times shouldn't be too different 13:36:10 <thorst> hopefully 13:36:16 <thorst> run times...definitely different 13:37:37 <esberglu> As part of deploying that patch I realized that the CI nodes (mgmt, ctrl, log) were 14.04, I thought we had upgraded a while back 13:37:40 <esberglu> But apparently not 13:37:43 <adreznec> Yeah, we'll have to do a longer term comparison to be sure 13:37:48 <adreznec> How did the upgrade go esberglu? 13:37:57 <esberglu> Not as well as I was hoping 13:38:02 <esberglu> The ctrl node is fine 13:38:04 <thorst> upgrade? 13:38:06 <adreznec> Yeah, was afraid of that 13:38:09 <thorst> o... 13:38:10 <esberglu> To 16.04 13:38:11 <adreznec> 16.04 13:38:16 <esberglu> On the staging CI servers 13:38:22 <thorst> did we capture the VM images for the controller nodes before upgrading? 13:38:26 <thorst> I could just snapshot them 13:38:32 <thorst> so you can fall back safely 13:38:51 <thorst> too late for staging, but maybe something for prod? 13:38:52 <adreznec> Well worst case scenario we just run the playbooks again right? :) 13:39:00 <thorst> word...that's why we built them 13:39:14 <esberglu> Seeing some issues with the log and mgmt nodes 13:39:30 <adreznec> What went awry? bad config? missing packages? 13:39:40 <esberglu> For some reason puppet is downloading all the modules to the / directory on the log node 13:39:55 <esberglu> When running install_modules.sh 13:40:21 <adreznec> Hmm. Maybe the path it was trying to use doesn't exist or something... 13:40:31 <esberglu> The mgmt node had some bad packages and some stuff with starting/stopping services 13:41:00 <esberglu> I just upgraded them last night so I haven't dug too deep into the problems yet 13:41:33 <esberglu> #action esberglu: Get staging CI nodes running on 16.04 13:41:57 <adreznec> Ok. Yeah these are pretty typical issues due to the upstart/systemd conversion 13:42:20 <esberglu> And after that start hitting the dual CI 13:42:48 <esberglu> #topic Dual CI 13:42:54 <esberglu> I have some questions about this 13:43:04 <esberglu> How do we want to handle voting? 13:43:24 <thorst> no vote yet. Similar to what nova-powervm does (once stable of course...don't publish until stable) 13:43:37 <thorst> different CI name though. Something that indicates in tree status 13:45:08 <esberglu> Okay. Has somebody been testing the in tree with devstack? 13:45:14 <esberglu> So I can steal their local.conf? 13:45:25 * thorst looks to efried 13:45:35 <adreznec> The PowerVM Oak CI. Or maybe maple? elm? 13:45:50 <efried> I have a local.conf 13:45:51 <thorst> har har har... 13:46:36 <efried> I don't get it 13:46:49 <efried> oh 13:46:52 <efried> tree 13:46:56 <thorst> yah... 13:47:00 <efried> got it 13:47:21 <efried> bore (in-tree) vs bark (oot) 13:47:32 <efried> or is it bole? 13:47:59 <efried> bole 13:48:02 <efried> anywhooo 13:48:32 <adreznec> lol 13:48:53 <adreznec> Ok, so efried is going to give his local.conf to esberglu and we'll go from there 13:49:15 <esberglu> That's pretty much all I have for CI. Get the staging nodes up and get the dual CI running there 13:49:29 <esberglu> I might have some q's as I get further into the dual CI stuff 13:49:35 <thorst> xia: I know you were working with wangqwsh. I know that wangqwsh is helping efried with the in tree driver, but has experience with the CI. 13:49:42 <efried> not sure how we handle the fact that you have to spawn with no network 13:49:47 <thorst> wondering out loud if it would make sense for you to become familiar with the CI as well. 13:50:00 <thorst> efried: we can deploy w/ network, that network just won't get attached to the VM. 13:50:04 <thorst> so it'll fake work 13:50:13 <efried> it broke when i tried it 13:50:28 <thorst> what the what? OK...I'll check out the error later. 13:50:28 <efried> maybe a code change is needed. 13:50:33 <efried> thx 13:50:37 <wangqwsh> throw a lot of error about networking 13:51:06 <thorst> ahh, I bet I know what it is...bet you'd get around it using the SEA agent for now... 13:51:19 <efried> #action efried & thorst to investigate spawn "with" network on the first few in-tree change sets 13:51:20 <thorst> no, that'd be broken too. 13:52:16 <efried> back to #topic wangqwsh helping with CI? 13:52:46 <thorst> well, actually it would be xia getting some experience there from wangqwsh 13:52:57 <thorst> we clearly need backups for esberglu when he decides to go on vacation 13:53:00 <wangqwsh> um. 13:53:12 <thorst> and not just backups, but also driving some of that CI dev 13:53:22 <esberglu> Yeah with all 5 days I have left for the year... 13:53:24 <thorst> not sure its right to do that now...but at some point. 13:53:34 <thorst> something to consider. 13:54:16 <esberglu> I have been adding wangqwsh to the neo-os-ci reviews. I can add xia as well. Just to give some exposure 13:54:39 <thorst> yeah, exposure is a good start. I know wangqwsh and xia, you're both also in the driver code as well 13:54:47 <thorst> so if that's the focus now...keep it as the primary 13:55:25 <thorst> that's all I had :-) 13:56:03 <esberglu> Anyone have anything else? 13:56:51 <efried> not I 13:57:03 <wangqwsh> nope 13:58:20 <esberglu> #endmeeting