13:00:05 #startmeeting powervm_driver_meeting 13:00:06 Meeting started Tue Mar 28 13:00:05 2017 UTC and is due to finish in 60 minutes. The chair is esberglu. Information about MeetBot at http://wiki.debian.org/MeetBot. 13:00:07 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 13:00:09 The meeting name has been set to 'powervm_driver_meeting' 13:00:15 o/ 13:01:38 o/ 13:02:25 #topic In Tree Driver 13:02:54 efried: I was looking through the changesets and CI really didn't like power on/off and after 13:03:02 Oh wait nvm 13:03:14 Yeah, I don't think that was us. 13:03:32 Something fundamental done broke. 13:03:43 Yeah I think it was an issue with certain CIs 13:03:54 But looks like verified +1s now 13:04:04 So I owe you a couple reviews 13:04:58 I don't know if you caught this comment last week, but... 13:05:13 If you ever feel down about our CI success rate, just look at freakin xenserver. 13:05:19 That guy fails like 50% of the time. 13:05:29 :-) 13:05:53 And that one is gating. 13:07:23 Does anyone have an OOT setup with recent (last 2 weeks, say) nova code underneath it where we can verify that glance bug? 13:07:47 Yeah I think we are fine from a success rate view. At least most days :-) 13:07:51 I'm still restacking my test system to debug it. But I need to prove that it affects OOT so I can open a launchpad bug. 13:07:58 efried: I don't 13:09:14 efried: Nope, still on Ocata over here 13:09:37 efried: I'm reconfiguring neo34 for OOT, got stuck with issues, which I'm looking into .. 13:09:46 Okay. 13:10:12 Otherwise in-tree just need reviews, at least up to 'console'. (I don't want to move SSP into the ready list until we figure this bug out.) 13:10:33 thorst I may need your help with the bug 13:11:19 "Monkey patch the glance API code in nova" is my only solution right now. 13:11:28 uhhh, that's awful 13:11:34 if you have a setup where it's borked I can take a peak 13:11:36 Yeah, I assume that's not a viable solution. 13:11:42 totes not viable. 13:12:00 thorst I ought to have that by the time this meeting is over. Stacking now. And that always succeeds. 13:12:07 cool 13:12:12 Should be ready in 10 minutes then efried 13:12:13 :) 13:12:48 btw, wanna queue up a topic for after the meeting: I have a sneaking suspicion that, when a system has been running for a long time, things go pear-shaped. 13:13:01 noted 13:13:10 That sounds bad, but ok 13:13:37 yeah, curious about that too...because we've been running CI for months 13:13:47 but...post scrum topic 13:14:24 #topic OOT Driver 13:14:36 Anyone have anything here? 13:14:49 Wellll... 13:15:00 I've been accumulating changes from in-tree to backport to OOT. 13:15:10 I have some of them in a (no-yet-proposed) commit. 13:15:21 But some things have come up that will require a much wider effort. 13:15:26 Like autospeccing. 13:15:31 I know Shyama will be proposing fixes for LPM w.r.t. Cinder and File backed volumes. 13:15:41 she's taking over a change set from me 13:16:03 I guess I don't really have an action item to propose here, but I do want to announce that I'll be requiring new UT to autospec anything coming from pypowervm from this point forward. 13:16:19 fair enough... 13:16:31 And it won't hurt my feelings if people want to go retrofit existing UTs with autospec. 13:16:50 The ephemeral file support is still on hold until we can get those pesky REST changes implemented. Probably a couple sprints out still tbh 13:17:13 adreznec: and then we need pypowervm updates? 13:17:18 Yeah 13:17:27 Once the REST side is done 13:17:29 good thing we have a new versioning approach there 13:17:33 :) 13:17:52 Yeah we'll have to keep that as a topic 13:18:07 Deciding when we need to do a version bump there 13:20:46 FYI it looks like the change to add a global-reqs job for nova-powervm got stuck (https://review.openstack.org/#/c/440852/) 13:21:01 The corresponding deps merged but it didn't go in. Just bumped it 13:21:21 Do we want to add g-r jobs for networking-powervm and ceilometer-powervm? 13:21:40 Probably 13:21:42 we should I'd think 13:21:52 Ok 13:22:01 I can toss those up a bit later here 13:22:07 Fairly straightforward 13:22:08 Cool 13:22:29 Anything else OOT before we move on? 13:24:16 #topic CI 13:24:23 I've got a bunch of stuff here 13:25:17 I believe we are ready to move up the IT CI patches to console? 13:25:34 And then add the corresponding whitelist change 13:26:23 Sounds like it 13:26:47 Then we can start getting some volume through and hunt down any issues 13:26:54 So I will put up that patch today 13:27:20 Other than that there are a few things I want to get working 13:27:29 I want to get all branches running on master tempest 13:27:37 ocata and master are fine 13:27:48 newton is passing everything but 3 tests 13:28:16 So I need to figure those failures out and then we can move it up for newton 13:28:33 I also want to get the undercloud moved from newton to ocata 13:28:59 It seems like we have a lull where I can try to get that going on staging 13:29:31 I'm guessing it's going to be a bigger endeavor than just checking out a different branch 13:29:58 Then the last big change is to fix the goofy networking stuff 13:30:16 Right now the IT and OOT networking is different 13:30:56 did we ever dig up that OVS note? 13:31:02 And OOT networks are being created in prep_devstack.sh while IT is using the os_ci_tempest.sh 13:31:06 And its just bad 13:31:42 thorst: Was gonna talk to you about that today if you have time 13:32:16 I'm free between 12-3 to chat about that 13:32:25 just need to find that note...I have no idea where that thing is :-) 13:32:37 Okay I'll hunt it down after this 13:32:42 I seem to remember me thinking it was brilliant at the time, but I've since forgotten what that idea is 13:34:49 That's all I have for CI 13:35:38 esberglu: We don't have any tests specific to SVC + FC in CI right ? 13:37:07 jayasankar_: we do not. 13:37:12 no cinder in the CI 13:37:50 Okay.. 13:37:54 Yep. That's why we are having you take a look 13:38:00 jayasankar_: The only storage in the CI today is SSP 13:38:15 And using remote upload, at that. 13:38:39 which is why we didn't see problems three weeks ago. 13:39:20 #topic Open Discussion 13:39:30 efried: You had something here? 13:39:55 My test system was up, not doing anything, for a couple of weeks. 13:40:02 When I got back to it, it was broken. 13:40:26 I've been looking at it while we've been talking, here, and I believe I've narrowed it down to the VIOS being hosed. 13:40:36 I know at least the cluster is screwed. 13:41:01 At the moment I'm trying to figure out if it could be because another system was in the cluster, and it may have inadvertently used the cluster disks for something. 13:41:01 Networking issues maybe? 13:41:34 adreznec: networking never fails 13:41:35 Mm, could be part of it, I suppose. Got a weird error listing the cluster - it was saying the localhost was only reachable through the repository disk. 13:42:02 Anyway, purely anecdotally, this isn't the first time I've exerienced this - left a neo alone for "a while" and come back to find it borked. 13:42:12 We've had systems up and running for many weeks without notable issues 13:42:23 efried: could be shared disk issues. 13:42:25 Okay, we have? Then I'm happy. 13:42:37 Yeah. 13:42:51 I need to be reminded where that SAN is so I can make sure those disks are gone from the other neo. 13:43:55 And I'll contact Uma to see if she can recover it to some normal state. I can't get anything going wrt the cluster right now. 13:43:59 esberglu: I need help on to configure tempest in OSA. I stuck there last few weeks. 13:45:28 nbante: I'm in the same boat. I just got an OSA deployment to complete the full run_playbooks script yesterday for the first time since picking OSA back up 13:45:28 efried: I can send you the v7k 13:45:50 nice.. 13:46:28 I faced so many issue while setup but now stuck in tempest 13:46:34 nbante: esberglu are these AIO? 13:46:45 Yeah mine is 13:47:09 If so, one the AIO is running you should just be able to use the gate-check-commit.sh script in the OSA repo I think 13:47:20 A subset of which is running tempest agains tthe AIO 13:47:26 AIO? 13:47:30 All in One 13:47:33 All in one 13:47:34 ok 13:48:41 That'll do a bit more than just tempest, but it'll be the same level of testing they'd do in the gate 13:48:47 Which is what we'd ideally want 13:50:04 adreznec: do you have any link where I can get that script. I'll try to run tht as well. 13:50:27 nbante: It's in the scripts subdirectory of the main OSA repo 13:50:28 https://github.com/openstack/openstack-ansible/blob/master/scripts/gate-check-commit.sh 13:50:39 So if you have OSA cloned down, you should already have it 13:50:48 in openstack-ansible/scripts/ 13:51:42 I already cloned down. It should. will try to run and share you result 13:56:27 Any final topics before I end the meeting? 13:57:51 is there any planned schedule for IT deliverable ? 13:58:38 both IT and OOT ? or it is like by 2Q we have to complete both ? 14:01:16 jayasankar_: the OOT is there today. IT needs to be done as patches are proposed up 14:01:28 the core reviewers hold the key to when things get merged in... 14:01:33 (we are not core reviewers) 14:01:56 so the net is, IT needs to be tested as efried proposes them up :-) 14:02:42 Okay. 14:03:17 Thanks for joining 14:03:21 #endmeeting