13:00:05 <esberglu> #startmeeting powervm_driver_meeting
13:00:06 <openstack> Meeting started Tue Mar 28 13:00:05 2017 UTC and is due to finish in 60 minutes.  The chair is esberglu. Information about MeetBot at http://wiki.debian.org/MeetBot.
13:00:07 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
13:00:09 <openstack> The meeting name has been set to 'powervm_driver_meeting'
13:00:15 <thorst> o/
13:01:38 <efried> o/
13:02:25 <esberglu> #topic In Tree Driver
13:02:54 <esberglu> efried: I was looking through the changesets and CI really didn't like power on/off and after
13:03:02 <esberglu> Oh wait nvm
13:03:14 <efried> Yeah, I don't think that was us.
13:03:32 <efried> Something fundamental done broke.
13:03:43 <esberglu> Yeah I think it was an issue with certain CIs
13:03:54 <esberglu> But looks like verified +1s now
13:04:04 <esberglu> So I owe you a couple reviews
13:04:58 <efried> I don't know if you caught this comment last week, but...
13:05:13 <efried> If you ever feel down about our CI success rate, just look at freakin xenserver.
13:05:19 <efried> That guy fails like 50% of the time.
13:05:29 <thorst> :-)
13:05:53 <efried> And that one is gating.
13:07:23 <efried> Does anyone have an OOT setup with recent (last 2 weeks, say) nova code underneath it where we can verify that glance bug?
13:07:47 <esberglu> Yeah I think we are fine from a success rate view. At least most days :-)
13:07:51 <efried> I'm still restacking my test system to debug it.  But I need to prove that it affects OOT so I can open a launchpad bug.
13:07:58 <thorst> efried: I don't
13:09:14 <adreznec> efried: Nope, still on Ocata over here
13:09:37 <jayasankar_> efried: I'm reconfiguring neo34 for OOT, got stuck with issues, which I'm looking into ..
13:09:46 <efried> Okay.
13:10:12 <efried> Otherwise in-tree just need reviews, at least up to 'console'.  (I don't want to move SSP into the ready list until we figure this bug out.)
13:10:33 <efried> thorst I may need your help with the bug
13:11:19 <efried> "Monkey patch the glance API code in nova" is my only solution right now.
13:11:28 <thorst> uhhh, that's awful
13:11:34 <thorst> if you have a setup where it's borked I can take a peak
13:11:36 <efried> Yeah, I assume that's not a viable solution.
13:11:42 <thorst> totes not viable.
13:12:00 <efried> thorst I ought to have that by the time this meeting is over.  Stacking now.  And that always succeeds.
13:12:07 <thorst> cool
13:12:12 <adreznec> Should be ready in 10 minutes then efried
13:12:13 <adreznec> :)
13:12:48 <efried> btw, wanna queue up a topic for after the meeting: I have a sneaking suspicion that, when a system has been running for a long time, things go pear-shaped.
13:13:01 <esberglu> noted
13:13:10 <adreznec> That sounds bad, but ok
13:13:37 <thorst> yeah, curious about that too...because we've been running CI for months
13:13:47 <thorst> but...post scrum topic
13:14:24 <esberglu> #topic OOT Driver
13:14:36 <esberglu> Anyone have anything here?
13:14:49 <efried> Wellll...
13:15:00 <efried> I've been accumulating changes from in-tree to backport to OOT.
13:15:10 <efried> I have some of them in a (no-yet-proposed) commit.
13:15:21 <efried> But some things have come up that will require a much wider effort.
13:15:26 <efried> Like autospeccing.
13:15:31 <thorst> I know Shyama will be proposing fixes for LPM w.r.t. Cinder and File backed volumes.
13:15:41 <thorst> she's taking over a change set from me
13:16:03 <efried> I guess I don't really have an action item to propose here, but I do want to announce that I'll be requiring new UT to autospec anything coming from pypowervm from this point forward.
13:16:19 <thorst> fair enough...
13:16:31 <efried> And it won't hurt my feelings if people want to go retrofit existing UTs with autospec.
13:16:50 <adreznec> The ephemeral file support is still on hold until we can get those pesky REST changes implemented. Probably a couple sprints out still tbh
13:17:13 <thorst> adreznec: and then we need pypowervm updates?
13:17:18 <adreznec> Yeah
13:17:27 <adreznec> Once the REST side is done
13:17:29 <thorst> good thing we have a new versioning approach there
13:17:33 <adreznec> :)
13:17:52 <adreznec> Yeah we'll have to keep that as a topic
13:18:07 <adreznec> Deciding when we need to do a version bump there
13:20:46 <adreznec> FYI it looks like the change to add a global-reqs job for nova-powervm got stuck (https://review.openstack.org/#/c/440852/)
13:21:01 <adreznec> The corresponding deps merged but it didn't go in. Just bumped it
13:21:21 <adreznec> Do we want to add g-r jobs for networking-powervm and ceilometer-powervm?
13:21:40 <esberglu> Probably
13:21:42 <thorst> we should I'd think
13:21:52 <adreznec> Ok
13:22:01 <adreznec> I can toss those up a bit later here
13:22:07 <adreznec> Fairly straightforward
13:22:08 <esberglu> Cool
13:22:29 <esberglu> Anything else OOT before we move on?
13:24:16 <esberglu> #topic CI
13:24:23 <esberglu> I've got a bunch of stuff here
13:25:17 <esberglu> I believe we are ready to move up the IT CI patches to console?
13:25:34 <esberglu> And then add the corresponding whitelist change
13:26:23 <adreznec> Sounds like it
13:26:47 <esberglu> Then we can start getting some volume through and hunt down any issues
13:26:54 <esberglu> So I will put up that patch today
13:27:20 <esberglu> Other than that there are a few things I want to get working
13:27:29 <esberglu> I want to get all branches running on master tempest
13:27:37 <esberglu> ocata and master are fine
13:27:48 <esberglu> newton is passing everything but 3 tests
13:28:16 <esberglu> So I need to figure those failures out and then we can move it up for newton
13:28:33 <esberglu> I also want to get the undercloud moved from newton to ocata
13:28:59 <esberglu> It seems like we have a lull where I can try to get that going on staging
13:29:31 <esberglu> I'm guessing it's going to be a bigger endeavor than just checking out a different branch
13:29:58 <esberglu> Then the last big change is to fix the goofy networking stuff
13:30:16 <esberglu> Right now the IT and OOT networking is different
13:30:56 <thorst> did we ever dig up that OVS note?
13:31:02 <esberglu> And OOT networks are being created in prep_devstack.sh while IT is using the os_ci_tempest.sh
13:31:06 <esberglu> And its just bad
13:31:42 <esberglu> thorst: Was gonna talk to you about that today if you have time
13:32:16 <thorst> I'm free between 12-3 to chat about that
13:32:25 <thorst> just need to find that note...I have no idea where that thing is  :-)
13:32:37 <esberglu> Okay I'll hunt it down after this
13:32:42 <thorst> I seem to remember me thinking it was brilliant at the time, but I've since forgotten what that idea is
13:34:49 <esberglu> That's all I have for CI
13:35:38 <jayasankar_> esberglu: We don't have any tests specific to SVC + FC in CI right ?
13:37:07 <thorst> jayasankar_: we do not.
13:37:12 <thorst> no cinder in the CI
13:37:50 <jayasankar_> Okay..
13:37:54 <esberglu> Yep. That's why we are having you take a look
13:38:00 <adreznec> jayasankar_: The only storage in the CI today is SSP
13:38:15 <efried> And using remote upload, at that.
13:38:39 <efried> which is why we didn't see problems three weeks ago.
13:39:20 <esberglu> #topic Open Discussion
13:39:30 <esberglu> efried: You had something here?
13:39:55 <efried> My test system was up, not doing anything, for a couple of weeks.
13:40:02 <efried> When I got back to it, it was broken.
13:40:26 <efried> I've been looking at it while we've been talking, here, and I believe I've narrowed it down to the VIOS being hosed.
13:40:36 <efried> I know at least the cluster is screwed.
13:41:01 <efried> At the moment I'm trying to figure out if it could be because another system was in the cluster, and it may have inadvertently used the cluster disks for something.
13:41:01 <adreznec> Networking issues maybe?
13:41:34 <thorst> adreznec: networking never fails
13:41:35 <efried> Mm, could be part of it, I suppose.  Got a weird error listing the cluster - it was saying the localhost was only reachable through the repository disk.
13:42:02 <efried> Anyway, purely anecdotally, this isn't the first time I've exerienced this - left a neo alone for "a while" and come back to find it borked.
13:42:12 <adreznec> We've had systems up and running for many weeks without notable issues
13:42:23 <thorst> efried: could be shared disk issues.
13:42:25 <efried> Okay, we have?  Then I'm happy.
13:42:37 <efried> Yeah.
13:42:51 <efried> I need to be reminded where that SAN is so I can make sure those disks are gone from the other neo.
13:43:55 <efried> And I'll contact Uma to see if she can recover it to some normal state.  I can't get anything going wrt the cluster right now.
13:43:59 <nbante> esberglu: I need help on to configure tempest in OSA. I stuck there last few weeks.
13:45:28 <esberglu> nbante: I'm in the same boat. I just got an OSA deployment to complete the full run_playbooks script yesterday for the first time since picking OSA back up
13:45:28 <thorst> efried: I can send you the v7k
13:45:50 <nbante> nice..
13:46:28 <nbante> I faced so many issue while setup but now stuck in tempest
13:46:34 <adreznec> nbante: esberglu are these AIO?
13:46:45 <esberglu> Yeah mine is
13:47:09 <adreznec> If so, one the AIO is running you should just be able to use the gate-check-commit.sh script in the OSA repo I think
13:47:20 <adreznec> A subset of which is running tempest agains tthe AIO
13:47:26 <nbante> AIO?
13:47:30 <adreznec> All in One
13:47:33 <esberglu> All in one
13:47:34 <nbante> ok
13:48:41 <adreznec> That'll do a bit more than just tempest, but it'll be the same level of testing they'd do in the gate
13:48:47 <adreznec> Which is what we'd ideally want
13:50:04 <nbante> adreznec: do you have any link where I can get that script. I'll try to run tht as well.
13:50:27 <adreznec> nbante: It's in the scripts subdirectory of the main OSA repo
13:50:28 <adreznec> https://github.com/openstack/openstack-ansible/blob/master/scripts/gate-check-commit.sh
13:50:39 <adreznec> So if you have OSA cloned down, you should already have it
13:50:48 <adreznec> in openstack-ansible/scripts/
13:51:42 <nbante> I already cloned down. It should. will try to run and share you result
13:56:27 <esberglu> Any final topics before I end the meeting?
13:57:51 <jayasankar_> is there any planned schedule for IT deliverable ?
13:58:38 <jayasankar_> both IT and OOT ? or it is like by 2Q we have to complete both ?
14:01:16 <thorst> jayasankar_: the OOT is there today.  IT needs to be done as patches are proposed up
14:01:28 <thorst> the core reviewers hold the key to when things get merged in...
14:01:33 <thorst> (we are not core reviewers)
14:01:56 <thorst> so the net is, IT needs to be tested as efried proposes them up  :-)
14:02:42 <jayasankar_> Okay.
14:03:17 <esberglu> Thanks for joining
14:03:21 <esberglu> #endmeeting