14:08:38 <esberglu> #startmeeting powervm_driver_meeting 14:08:39 <openstack> Meeting started Tue Nov 29 14:08:38 2016 UTC and is due to finish in 60 minutes. The chair is esberglu. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:08:40 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:08:42 <openstack> The meeting name has been set to 'powervm_driver_meeting' 14:08:43 <adreznec> Yep 14:08:45 <adreznec> That was it 14:08:45 <adreznec> :P 14:09:01 <thorst_> o/ 14:09:07 <thorst_> efried is off today I think 14:09:33 <esberglu> #topic Status 14:10:32 <esberglu> The latest CI run went through last night with only 6 failures 14:10:49 <thorst_> I'm reviewing those. Some seem like legit nova failures 14:11:18 <esberglu> Run in question if anyone needs it 14:11:20 <esberglu> http://184.172.12.213/34/403934/1/check/nova-powervm-pvm-dsvm-tempest-full/57319c1/ 14:11:37 <esberglu> It looked like most of them were the same failure 14:11:52 <esberglu> I haven't looked into the logs yet though 14:12:02 <adreznec> Ah ok 14:12:08 <adreznec> Was going to ask if any were new tests 14:12:21 <esberglu> Yeah one of them is a new one that I need to disable 14:12:26 <adreznec> Were the runs that failed all against master? 14:12:59 <esberglu> Yeah 14:13:00 <thorst_> yeah, this is a nova failure (at least one of them) 14:13:07 <thorst_> I'll go investigate that. 14:13:11 <thorst_> propose something up... 14:13:22 <thorst_> o wait...no, this is on us 14:13:25 <esberglu> #action esberglu: Disable unsupported test 14:13:43 <thorst_> #action thorst: fix confirm_migration in nova_powervm 14:14:15 <adreznec> test_delete_server seems like one that should pass... 14:14:19 <adreznec> Unless the name is bad 14:14:48 <thorst_> yeah, they're failing because of a bug 14:14:52 <thorst_> they changed a driver signature 14:14:56 <adreznec> Ah 14:14:59 <thorst_> we didn't notice (no CI :-) 14:15:09 <adreznec> Yeah 14:15:20 <adreznec> Sorry, was still unzipping the n-cpu log 14:15:22 <thorst_> nothing to worry about 14:15:26 <thorst_> I got it 14:15:30 <adreznec> Ok 14:15:42 <thorst_> hooray for CI though 14:16:42 <thorst_> so, next steps are to get that fixed...see if we can't get a few solid runs through... 14:16:45 <thorst_> then party? 14:16:53 <adreznec> You're sure this delete one is related? The 404 back from REST? 14:17:02 <esberglu> Yeah I think that's a different issue 14:17:18 <thorst_> maybe not...but the tempest-DeleteServersTestJSON hit this 14:17:25 <thorst_> TypeError: confirm_migration() takes exactly 4 arguments (5 given) 14:17:36 <thorst_> and I see that 4 times in the logs 14:17:43 <thorst_> each on different instances. 14:18:00 <esberglu> And the 5th failure is the one I need to disable 14:18:03 <thorst_> so its at least one of the things. 14:18:14 <adreznec> Right 14:18:17 <esberglu> So all we have left is the first test_delete_server failure 14:18:19 <adreznec> Ok, lets start there then I guess 14:18:27 <esberglu> Which is something about a security group in use 14:19:57 <thorst_> I wonder if that is a side effect of these other servers still existing though 14:20:02 <thorst_> cause we failed to delete them 14:20:12 <thorst_> and its just one of those concurrency things 14:20:52 <openstackgerrit> Drew Thorstensen (thorst) proposed openstack/nova-powervm: Update to new confirm_migration signature https://review.openstack.org/404234 14:21:27 <esberglu> Could be. Btw did you guys see that change for the arch import statement? 14:22:03 <esberglu> Well at least thorst_ did 14:22:12 <adreznec> I saw the review come in, didn't have a chance to look at it yet 14:22:17 <thorst_> yeah, I'm ready to W+1 that 14:22:25 <thorst_> adreznec: should take 2 seconds and would unwedge 14:22:53 <esberglu> https://review.openstack.org/#/c/403934/ 14:23:21 <esberglu> And that's the patch this CI run is from 14:23:27 <adreznec> lgtm 14:25:11 <thorst_> so what's next? 14:25:27 <thorst_> I propose that we update the requests library in the tempest VMs, the novalinks, and the undercloud 14:25:54 <thorst_> because efried told me we saw some of those requests bugs in a few runs a while back...so I'd like to squash that completely. 14:26:12 <adreznec> thorst_: do we have enough confidence that's solving real issues? 14:26:26 <adreznec> If so, we should probably get our dependency updated 14:26:27 <thorst_> adreznec: I think it is several issues 14:26:36 <adreznec> On requests >= 2.12.1 14:26:38 <thorst_> we have a fix from efried that we need to finish 14:26:42 <thorst_> that's solid 14:26:43 <adreznec> I was going to test that out today 14:26:52 <adreznec> Make sure it doesn't break anything 14:27:02 <adreznec> If it doesn't I'll put up a nova-powervm req change for it 14:27:09 <thorst_> the requests >= 2.12.1 has worked much better for kriskend's three runs of 10 deploys 14:27:28 <thorst_> adreznec: do we want it in nova-powervm or just install it and instead pursue with glanceclient? 14:27:52 <adreznec> We could try 14:27:52 <thorst_> this is one of those situations where its not us... 14:28:04 <adreznec> That would mean a g-r bump probably 14:28:13 <adreznec> So it could take while to filter through 14:28:14 <thorst_> or a g-r exclude 14:28:19 <thorst_> right 14:28:29 <thorst_> but I think that's the right approach 14:28:32 <adreznec> Do we think this is an issue specifically with 2.11.1? 14:28:39 <thorst_> unclear 14:28:47 <thorst_> that's definitely what people have when we hit the issue 14:28:49 <adreznec> Ok... probably need testing there then 14:28:59 <adreznec> To confirm if we need an exclude or a minimum bump 14:29:24 <thorst_> yeah. I'd say for now we just add it to the CI env and workaround from there 14:29:27 <openstackgerrit> Merged openstack/nova-powervm: Change arch to fields.Architecture https://review.openstack.org/403934 14:29:33 <adreznec> Sure 14:29:38 <thorst_> actually 14:29:44 <thorst_> upper-constraints will kill us there 14:29:51 <thorst_> they have it tagged to 2.11.1 14:30:08 <thorst_> so even if we put it in the CI, devstack will detect that and then update it 14:30:14 <thorst_> so I think we need to have a g-r debate. 14:30:37 <thorst_> #action thorst to open bug to get upper-constraints updated for requests package 14:31:13 <adreznec> Fun fun 14:31:19 <thorst_> moving on :-D 14:31:20 <esberglu> I could remove the upper in the same way that we are removing the pypowervm constraints temporarily 14:31:35 <thorst_> esberglu: lets wait it out and see if its bad 14:31:39 <esberglu> Ok 14:31:48 <thorst_> I'd rather go through formal process 14:32:34 <thorst_> any other items to discuss? I'm assuming OSA progress has been limited due to Thanksgiving and then focus on stablizing core CI? 14:32:52 <esberglu> I haven't done anything with it since before break 14:32:53 <adreznec> Yeah, I don't think there's been any movement on CI there 14:33:06 <thorst_> wangqwsh: do you have any updates on it? 14:33:20 <adreznec> and the only upstream patch I'm aware of is the one I have open for fixing the interfaces template 14:33:40 <wangqwsh> no, still block on crt_trunk_with_free_vlan 14:34:40 <thorst_> the block there is? 14:34:48 <thorst_> (sorry - my memory is hazy from holiday) 14:35:33 <wangqwsh> i am trying to boot a vm by manual, however the nova compute said KeyError: '06CC64F5-608A-4D4D-8C95-65B80CDFA96D' 14:36:04 <wangqwsh> i found this error was from pypowervm/tasks/cna.py - crt_trunk_with_free_vlan 14:36:35 <thorst_> can you send a pastebin with the stack? 14:36:46 <thorst_> not urgent, but I should probably dig into that 14:37:11 <wangqwsh> sure. 14:37:34 <thorst_> I'm also thinking that wangqwsh should learn from esberglu how to redeploy the CI 14:37:49 <thorst_> so that esberglu can take a break every once and a while (ex. go on holiday) 14:38:06 <wangqwsh> yes, 14:38:15 <esberglu> Especially since I am gonna be out for 2+ weeks after christmas 14:38:38 <wangqwsh> pleasure :) 14:38:45 <thorst_> esberglu: Can you take an action to set up a jump box (that has the patches that aren't yet merged) and train wangqwsh on how to do the deploys? 14:39:11 <esberglu> #action esberglu: Teach wangqwsh how to deploy CI 14:39:16 <thorst_> wangqwsh: you'll get thrown in on this quickly, cause mainline CI is our highest priority (though we have a light at end of tunnel) 14:39:40 <wangqwsh> ok 14:41:44 <thorst_> cool 14:42:08 <esberglu> thorst_: Once we get your change in, what projects should I open up the CI for? The *-powervm projects don't have much action right now, but the core projects will have a ton 14:43:10 <esberglu> Just kick off a few runs and make sure they are alright and then start opening up to the core projects after? 14:43:12 <thorst_> well, we want core once we've stablized 14:43:19 <thorst_> three phases 14:43:25 <thorst_> 1) Stabilize in powervm projects 14:43:39 <thorst_> 2) Open up to other projects for higher velocity -> check results 14:43:49 <thorst_> 3) Publish logs (but do not vote) for nova project 14:44:01 <thorst_> and for now, other projects should probably just be nova 14:44:55 <adreznec> Yeah 14:45:08 <adreznec> I vote *-powervm first, then nova, then neutron/ceilometer 14:45:26 <adreznec> then OSA once we get the base CI stable 14:46:00 <thorst_> I'd really like 3 being done by Jan 1 14:46:02 <thorst_> :-) 14:46:05 <thorst_> if not sooner 14:46:20 <esberglu> I think we can get it done before then 14:46:36 <thorst_> optimism 14:46:39 <thorst_> :-) 14:47:09 <esberglu> Anything else? 14:47:49 <thorst_> not from me 14:47:54 <thorst_> lots of good progress though! 14:48:21 <adreznec> Yeah, that's all I've got 14:48:26 <adreznec> Good to see things back running again :) 14:48:35 <esberglu> Man I forgot to change the topic the whole meeting. Oh well 14:48:39 <esberglu> #endmeeting