13:00:56 <esberglu> #startmeeting powervm_driver_meeting
13:00:57 <openstack> Meeting started Tue Jul 25 13:00:56 2017 UTC and is due to finish in 60 minutes.  The chair is esberglu. Information about MeetBot at http://wiki.debian.org/MeetBot.
13:00:58 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
13:01:00 <openstack> The meeting name has been set to 'powervm_driver_meeting'
13:01:10 <mdrabe> o/
13:01:11 <edmondsw> o/
13:01:45 <esberglu> #topic In Tree Driver
13:01:48 <efried> \o
13:01:56 <esberglu> #link https://etherpad.openstack.org/p/powervm-in-tree-todos
13:02:07 <esberglu> Oh and here's the agenda
13:02:08 <esberglu> #link https://etherpad.openstack.org/p/powervm-in-tree-todos
13:02:18 <esberglu> #link https://etherpad.openstack.org/p/powervm_driver_meeting_agenda
13:02:39 <esberglu> No topics for in tree.
13:03:13 <edmondsw> yeah, I don't have anything for IT
13:03:13 <esberglu> Reminder that this week (thursday) is the pike 3 milestone (feature freeze)
13:03:37 <efried> https://review.openstack.org/#/c/476945/
13:04:13 <efried> Keeps failing CI.  I imagine it's for whatever reason everything is failing CI, and we'll get to that later?
13:04:31 <esberglu> efried: Yeah. I'll update on that status later
13:04:58 <edmondsw> efried should we open a bug with which to associate that change?
13:05:18 <efried> Wouldn't think so.  It's not a bug.
13:05:27 <edmondsw> k
13:05:33 <efried> and we don't need to "backport" it.
13:05:40 <edmondsw> I thought it fixed something, but must be wrong
13:05:49 <efried> even if it doesn't make Pike, we don't need to backport it to Pike.
13:05:55 <edmondsw> fine
13:06:44 <esberglu> edmondsw: efried: I've been seeing errors about power_off_progressive in CI, assuming this is related somehow?
13:06:58 <efried> oh?
13:07:05 <efried> Yes, that would be related.
13:07:08 <efried> I haven't seen those.
13:07:10 <efried> Well, shit.
13:07:18 <efried> Guess we spend some time on that soon.
13:07:20 <efried> But maybe not too soo.
13:07:23 <efried> Because feature freeze.
13:08:30 <esberglu> efried: I don't have logs handy atm, I'll link them when I get to CI status
13:08:58 <efried> Thought we ran that bad boy through when it was first proposed.  But mebbe not.
13:09:14 <efried> Twill suck if power_off_progressive is STILL broken.
13:09:27 <efried> Hate that code.
13:09:36 <esberglu> efried: The related error is not hitting every run so we may have ran it and just not hit the issue
13:10:13 <efried> We would have needed to run with a develop pypowervm and in-flight nova[-powervm] at the time, which I guess we might have done.  But anyway...
13:10:55 <efried> I don't have anything else in-tree.  Moving on?
13:11:01 <esberglu> Anyone have anything OOT or ready to move straight to PCI passthrough?
13:11:27 <mdrabe> NVRAM stuff
13:11:41 <mdrabe> efried: Discuss that later perhaps?
13:12:14 <efried> thorst_afk is home with sick kid, would like him involved in both of those discussions.
13:12:30 <efried> So the NVRAM improvement was what, again?  3.4%?
13:12:37 <mdrabe> Mhm
13:12:43 <efried> That's...
13:12:52 <mdrabe> Annoying
13:12:53 <edmondsw> #topic Out Of Tree Driver
13:13:01 <efried> ...not nothing, but pretty marginal.
13:13:13 <thorst_afk> I'm here.
13:13:22 <thorst_afk> slowly
13:13:54 <efried> mdrabe And this was run through a pvc suite that actually exercises the NVRAM/slotmgr code paths?
13:14:15 <efried> Has it been run through pvc fvt kind of thing that hammers those code paths for functionality?
13:14:18 <mdrabe> thorst: This is about https://review.openstack.org/#/c/471926/
13:14:42 <mdrabe> It can't run through fvt until it merges
13:14:44 <thorst> that is ... quite the change set.
13:14:53 <efried> Yeah, that's the problem.
13:14:57 <thorst> mdrabe: it'd have to be cherry picked.
13:15:16 <mdrabe> thorst: Not necessarily within nova-powervm
13:15:17 <efried> It completely swizzles the way nvram and slot management is stored & retrieved in swift.
13:15:31 <thorst> for a 3.4% improvement in deploy path?
13:15:32 <efried> Backward compatible, right mdrabe?  Upgrade-able?
13:15:39 <thorst> overall deploy path
13:15:44 <mdrabe> Yes backward compatible
13:15:59 <mdrabe> It doesn't affect the slot manager path greatly
13:16:21 <efried> By which I mean: if you have pvc running the old code, so objects are stored in swift the old way, then you upgrade pvc, it still works on the old objects?
13:16:53 <thorst> and if you have some nova computes running at old level and new nova computes running at new level
13:16:56 <thorst> does it all just work...
13:17:03 <thorst> cause agents don't get upgraded all at once.
13:17:03 <mdrabe> efried: The objects aren't stored any differently
13:17:28 <thorst> I think I need to spend some time reviewing this (I'm just asking naive questions)
13:17:41 <mdrabe> It's still keyed off the instance UUID regardless
13:17:43 <efried> Oh, right, they were always keyed by UUID in swift - the difference is the way they were passed around nova-powervm.
13:18:53 <efried> So yeah, thorst, that's the debate.  This is a nontrivial change across code that we chucked together at the last minute during that one TCC to cover the RR emergency, so I'm kinda scared of touching it.
13:19:04 <mdrabe> Actually efried this isn't that code
13:19:17 <thorst> yeah, that nightmare is in pypowervm mostly
13:19:18 <mdrabe> Most of the NVRAM stuff was already there, we slapped together the slot manager stuff
13:19:38 <thorst> this nvram stuff was from kyleh long ago
13:20:05 <thorst> so the basic change here is - move from list to set, and allow passing in the uuid or the instance, instead of just the instance.
13:20:14 <thorst> thus reducing the number of instance lookups I assume
13:20:24 <efried> That's the gist
13:20:46 <mdrabe> thorst: Yes, the performance improvement comes from avoiding getting the instance object in the NVRAM event handler
13:20:47 <thorst> let me review in more detail, but I think its a solid change.  The net though is, yeah, it needs some good test pre-merge
13:21:04 <thorst> so we've got to figure out a plan to do that....patch onto some PVC's I assume
13:21:08 <thorst> since PVC always has this turned on
13:21:30 <mdrabe> It might be worth noting there's also performance improvement in idle compute time
13:21:41 <efried> ?
13:21:51 <mdrabe> ? to me or thorst?
13:22:00 <thorst> I think he's asking for an explanation
13:22:02 <thorst> and if he isn't, I am
13:22:19 <mdrabe> Any NVRAM events that come in won't grab the instance object
13:22:30 <thorst> that's ... nice
13:22:30 <mdrabe> NVRAM events aren't exclusive to deploys
13:22:39 <thorst> right right...
13:23:53 <mdrabe> thorst: I'm not the biggest fan of the caching that's done in the change at the moment
13:23:54 <efried> Okay, so the plan is to move forward carefully, get some good pvc fvt (somehow)
13:24:08 <thorst> +1
13:24:18 <mdrabe> I'd be curious on your review thoughts for that bit
13:24:35 <thorst> I'll review today
13:25:45 <esberglu> Move on to PCI?
13:25:57 <edmondsw> I think so
13:26:00 <esberglu> #topic PCI Passthrough
13:26:00 <thorst> k
13:26:14 <efried> No progress on my end since last week.
13:26:17 <edmondsw> I don't have anything new here, but I wanted to make sure this is a standing item on the agenda going forward
13:26:41 <edmondsw> so that's all for today then, probably
13:27:15 <esberglu> Okay on to CI
13:27:17 <esberglu> #link https://etherpad.openstack.org/p/powervm_ci_todos
13:27:27 <esberglu> #topic Devstack generated tempest.conf
13:27:43 <esberglu> I've got 3 changes out in devstack, need a couple more
13:27:47 <esberglu> https://review.openstack.org/#/c/486629/
13:27:57 <esberglu> https://review.openstack.org/#/c/486671/
13:28:05 <esberglu> https://review.openstack.org/#/c/486701/
13:28:57 <esberglu> Then just need to backport those as needed and add the new options to the local.conf files
13:29:18 <esberglu> At that point we should be ready to use the generated tempest conf
13:29:24 <edmondsw> sweet
13:29:32 <esberglu> #topic CI Status
13:30:01 <esberglu> The last couple weeks the networking issues have been affecting the CI, taking down systems and leading to poor performance
13:30:26 <esberglu> Now that all of that seems to have settled down (fingers crossed) I'm gonna take inventory of tempest failures
13:30:28 <efried> zat what's been causing the timeouts?
13:31:12 <esberglu> efried: Seems like they have been way up when the network has been poor. Anecdotal though
13:31:36 <efried> k, well, we'll keep an eye on it.
13:31:41 <efried> Progress on the mystery 500?
13:32:07 <esberglu> Anyhow I'll let you guys know what I find when looking through the failures.
13:32:40 <esberglu> efried: Haven't looked back into it
13:33:43 <efried> And the power_off_progressive thing - you'll pop me links when you see it again?
13:33:59 <esberglu> efried: Yep
13:34:01 <efried> That's prolly all we can do here for CI, then.
13:34:49 <esberglu> mdrabe: Any progress on the jenkins/nodepool upgrade stuff? Haven't touched base on that in a while
13:35:20 <mdrabe> esberglu Nah been entirely pvc of late
13:35:23 <mdrabe> but I wanted to ask
13:35:45 <mdrabe> How tough would it be to go to all zuul?
13:35:56 <mdrabe> It's zuulv3 right?
13:36:23 <esberglu> mdrabe: Last time I checked zuul v3 wasn't ready for 3rd party CI's. But that may have changed
13:36:42 <esberglu> Once zuul v3 is ready I want to move to it
13:37:11 <esberglu> I shouldn't be too hard. Mostly just moving our jenkins jobs definitions over to zuul (where they would be defined with ansible instead)
13:38:29 <esberglu> Alright let's move on
13:38:36 <esberglu> #topic Driver Testing
13:38:40 <esberglu> Developments here?
13:40:05 <efried> jay1_ chhavi How's iSCSI going?
13:40:29 <efried> thorst wanted to make sure we got an update here.
13:40:34 <jay1_> Yeah, Last week encountered the volume attach issue
13:41:06 <chhavi> efried: ravi and jay is looking into it, i am not currently working on it.
13:41:32 <jay1_> and by the time Chhavi/Ravi debug it further system UI was not loading might be bcz of the NW issue we got last week
13:42:05 <jay1_> so, I reconfigured the system and again facing the Volume creation issue
13:42:20 <jay1_> Ravi is looking into it
13:45:06 <edmondsw> from talking to ravi, he has very limited time to spend on this
13:45:42 <edmondsw> so we're going to have to figure that out
13:49:03 <edmondsw> next topic?
13:49:20 <esberglu> #topic Open Discussion
13:49:34 <esberglu> #subtopic Test subtopic
13:50:20 <esberglu> Just curious if that command is a thing or not
13:50:34 <esberglu> I don't have anything further
13:50:44 <esberglu> Thanks for joining
13:50:56 <esberglu> #endmeeting