13:31:23 <esberglu> #startmeeting ci_scrum
13:31:24 <openstack> Meeting started Thu Nov 17 13:31:23 2016 UTC and is due to finish in 60 minutes.  The chair is esberglu. Information about MeetBot at http://wiki.debian.org/MeetBot.
13:31:25 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
13:31:28 <openstack> The meeting name has been set to 'ci_scrum'
13:31:40 <esberglu> #topic status
13:32:02 <adreznec> In the future we should do powervm ci scrum, then all the logs go under the same directory out on eavesdrop
13:32:19 <efried> o/
13:32:24 <thorst_> \o
13:32:36 <adreznec> o
13:32:51 <adreznec> Haven't had enough coffee to raise hand
13:33:35 <esberglu> I looked at the dir online and it just says ci_scrum?
13:34:05 <adreznec> Yeah, in the past I was doing powervm ci meeting
13:34:15 <adreznec> So they were going to http://eavesdrop.openstack.org/meetings/powervm_ci_meeting/
13:34:28 <adreznec> If you keep giving it the same name each time they all go to the same directory
13:34:47 <adreznec> Not a big deal
13:35:03 <esberglu> Oh I had the wrong bookmark
13:35:06 <esberglu> http://eavesdrop.openstack.org/meetings/ci_scrum/
13:35:22 <esberglu> We must have done it there once or something
13:35:31 <esberglu> Anyways
13:35:44 <thorst_> Status - stop everything until CI is running?
13:35:56 <esberglu> I did end up getting a run to go through last night
13:36:03 <esberglu> But still seeing the same behavior
13:36:24 <thorst_> elaborate?  The pypowervm issue certainly isn't fixed
13:36:47 <esberglu> Yeah. But I put USE_CONSTRAINTS=false
13:36:54 <thorst_> that worked?
13:36:56 <esberglu> Didn't we think that would work around?
13:37:00 <esberglu> It did not work
13:37:06 <thorst_> we thought it might.
13:37:13 <thorst_> is everyone aware what the pypowervm issue is?
13:37:21 <thorst_> adreznec?  I'm not sure if you were yet
13:37:28 <adreznec> So we're still seeing constraints issues?
13:37:47 <adreznec> Why here and not in the upstream gate?
13:37:51 <thorst_> devstack installs from the requirements file.  Which then blows away our local install (with the local2remote patch)
13:38:50 <adreznec> Hmm
13:39:05 <thorst_> before we dive into that (and I'd like to do a deep dive on that)
13:39:13 <thorst_> are there any other items we need to chase esberglu?
13:39:23 <thorst_> otherwise the meeting will get away on us with that rabbit hole
13:39:48 <esberglu> I still need to do work on the post stack vm cleaner script
13:40:49 <esberglu> And neo 6,7,8 stop working last night. Probably just need to restart vio_daemon
13:41:03 <thorst_> esberglu: was that just the lu clean up?
13:41:11 <adreznec> Just... stopped working?
13:41:20 <adreznec> without the CI running/doing anything?
13:41:23 <thorst_> adreznec: apearson is adding in a 'heartbeat' of sorts for that.
13:41:37 * adreznec sighs
13:41:42 <thorst_> yuh
13:41:53 <thorst_> so manual restarts for now, update of novalink in future should help
13:41:58 <esberglu> thorst_: It looked like the LU thing. It happened after I was clearing out the nodes so I could deploy with USE_CONSTRAINTS
13:42:46 <esberglu> But it wouldn't let me delete the LUs or the VMs
13:44:26 <esberglu> I didn't spend too much time looking at it, was more interested in getting a run through the CI
13:45:00 <thorst_> esberglu: alright.
13:45:30 <thorst_> #action thorst to follow up with apearson on vio_daemon thing
13:45:54 <thorst_> #action esberglu to work on the clean up script for lu.  Possibly restart vio_daemon depending on how far along that is in NL
13:46:52 <thorst_> any other items?
13:48:35 <esberglu> Not from me
13:49:14 <esberglu> You guys ready to talk about the pypowervm versioning then?
13:49:47 <thorst_> esberglu: do you have a link to a run that we saw this issue/
13:50:11 <adreznec> So if I understand correctly it's happening because pypowervm is now a global-req, which because our stuff now as a real requirement.txt requirement on pypowervm is causing pypowervm to be installed down the develop branch path I put in, erasing our pre-installed version of pypowervm
13:50:49 <thorst_> adreznec: well, it doesn't erase it
13:50:52 <esberglu> 9.114.251.93:8080/job/nova-powervm-pvm-dsvm-tempest-full/304/consoleFull
13:50:55 <thorst_> but yeah, effectively.
13:51:13 <esberglu> Crap not what I meant to paste
13:51:27 <thorst_> heh
13:51:53 <esberglu> http://184.172.12.213/15/328315/20/check/nova-powervm-pvm-dsvm-tempest-full/84aad2b/
13:52:08 <adreznec> Does it not install over the top of it?
13:52:22 <thorst_> adreznec: Search for this timestamp: 2016-11-16 20:27:46.703
13:52:36 <efried> It would be relatively easy to fix this in our CI - our scripts download the requirements project first, then hack the pypowervm lines out of global-requirements.txt and upper-constraints.txt, then proceed with the rest of devstack - but fixing it for our devstack in general is going to be tougher.
13:52:37 <efried> Here's one suggestion, not particularly pretty:
13:52:37 <efried> We start by re-syncing 1.0.0.4 into 1.0.0.3.9.
13:52:37 <efried> Then moving forward, we write any new pypowervm function into nova-powervm itself (or whichever out-of-tree driver needs it), while continuing to develop against 1.0.0.4.
13:52:37 <efried> Then periodically, we release a new version of pypowervm to pip (1.0.0.3.9.1, 1.0.0.3.9.2, ... just kidding) and update the global requirements.
13:52:37 <efried> Once that merges, we update nova-powervm accordingly, removing that functionality and repointing its usage from the temporary nova-powervm code to the released pypowervm code.
13:53:25 <thorst_> the challenge with that is that we don't want frequent updates to the global-requirements.txt.
13:53:45 <efried> Right, so when I say "periodically" above, we're talking once or twice per release.
13:54:02 <efried> (openstack release, that is)
13:54:13 <thorst_> then we lose the flexibility that we have now of getting fixes across multiple components (in a short period of time) for the out of tree driver
13:54:21 <efried> Yup.
13:54:26 <adreznec> Yeah, that's kind of a pain
13:54:50 <efried> It's pretty rare that we need something from pypowervm in multiple out-of-tree drivers, neh?
13:54:54 <thorst_> I think that's 100% right for in-tree...in fact I'd like the in-tree to use 1.0.0.4 basically once that goes out, even after pypowervm goes to 1.0.0.5
13:55:04 <thorst_> at least support it.
13:59:43 <thorst_> I like the idea of having the out of tree potentially overwrite the req.
14:00:11 <efried> Can we accept that it won't work for devstack in general?  Only for CI?
14:00:16 <thorst_> no
14:00:20 <thorst_> we need a devstack'd solution
14:00:55 <thorst_> I'd accept that we don't have the right answer for now...and maybe the work around is that we have something in the nova-powervm (or c/n) devstack scripts tweak the upper-constraints
14:01:05 <thorst_> and that we need to dwell on this a bit.
14:01:56 <efried> Would need to confirm this, but I believe the devstack scripts run really early, as soon as the project is collected.
14:02:31 <efried> The problem there is that we don't know for a given devstack run whether nova-powervm is collected before or after the requirements project.
14:02:50 <efried> It's not deterministic.
14:03:24 <efried> Which is kind of the point of global requirements - to at least make the levels of everything deterministic regardless of the order they happen to come in.
14:03:34 <thorst_> so it only installs for projects that require it.  Right now that's nova-powervm, networking-powervm, and ceilometer-powervm (and at some point nova)
14:03:42 <thorst_> so it would be called before that
14:03:47 <thorst_> but potentially, quickly changed.
14:04:00 <thorst_> because nova would be the first thing called.
14:04:20 <thorst_> so it's deterministic right now...
14:04:54 <thorst_> I think the 'hack' (and it is a hack) is two stages.
14:04:58 <efried> We can't get away with having nova's devstack scripts muck with the requirements.
14:05:06 <thorst_> 1) our devstacks tweak the upper-limits
14:05:39 <thorst_> 2) once in nova, we know that nova will go before nova-powervm.  So there nova-powervm would have to uninstall what nova put in, then let the other install go through.
14:05:59 <thorst_> the other install being what we developed in 1
14:06:11 <thorst_> core nova won't muck with it.  That's a no go.
14:06:15 <thorst_> nor should it.
14:06:21 <thorst_> that'd be...awful.
14:08:16 <efried> thorst_, I think we can implement both pieces at the same time.
14:08:37 <efried> Basically *-powervm uninstalls any pypowervm that happens to be there and installs the right version.
14:08:47 <thorst_> efried: possibly, but if we separate out and get one before the other...then I'd like to do what's quickest
14:09:10 <thorst_> (unless its like a couple hour difference)
14:09:48 <adreznec> Uninstalling 'any pypowervm that happens to be there' basically assumes it was pip installed
14:10:03 <adreznec> Unless we're going to get in the game of doing apt/yum removes, etc
14:10:08 <adreznec> Which sounds gross
14:11:55 <thorst_> adreznec: well, we'd just pip uninstall
14:11:59 <thorst_> not apt-get uninstall
14:12:10 <adreznec> I think part of this comes back to our pypowervm versioning being broken
14:12:13 <adreznec> again
14:12:16 <efried> yup
14:12:42 <adreznec> Right, because if we had proper versioning this wouldn't be a real issue
14:13:03 <adreznec> We could bump upper-constraints to something reasonable
14:13:31 <adreznec> and not worry about it ripping this version out from under us
14:14:29 <efried> adreznec, wouldn't we still have the moving-target problem?
14:14:44 <efried> Assuming we can't update upper-constraints twice a week whenever we put something new in pypowervm
14:14:57 <thorst_> and wouldn't it be well know that version 2.0.x doesn't exist, so how can that realistically be an upper-constraint
14:15:03 <adreznec> Sure, but so does everyone else in openstack... it makes things a bit less flexible, I agree
14:15:29 <efried> Right, I have to assume you can't put a version number that doesn't exist into requirements.  Or that'd be a snap.
14:15:40 <thorst_> unless you can?
14:16:03 <adreznec> Even if you could, I'd imagine the reqs team would -1 that patchset if they caught it
14:16:34 <adreznec> Otherwise it kind of defeats the entire purpose of upper-constraints, no?
14:16:39 <efried> For sure.
14:16:43 <efried> That's what we're trying to do.
14:16:47 <adreznec> Look, our upper-constraint is version 9000!
14:16:48 <efried> Defeat the entire purpose of upper-constraints.
14:16:50 <adreznec> Right
14:17:17 <efried> I'm still finding it tough to believe that nobody else runs into this on a regular basis.
14:17:18 <adreznec> Our goals are very different from their goals here since this is basically a development branch of an out-of-tree driver
14:18:07 <adreznec> Yeah, I looked at the compute-hyperv driver
14:18:10 <thorst_> back to the 'out-of-tree driver should isolate its weirdness in itself'
14:18:16 <adreznec> But they just bump os-win
14:18:34 <adreznec> and requirements
14:18:42 <efried> How frequently?
14:19:05 <adreznec> https://github.com/openstack/os-win/releases
14:19:09 <adreznec> Not that frequently
14:19:53 <efried> So thorst_, give it to me again why the lockstep approach wouldn't work?  (Accumulating function in nova-powervm and then, a couple times a release, "moving" it over to pypowervm.)
14:20:57 <thorst_> efried: well that was my request earlier.  Can we get more frequent releases of pypowervm without driving the whole train along with it.
14:21:01 <thorst_> which perhaps we can.
14:21:06 <esberglu> I know this won't work for all cases. But for now in the CI couldn't we just update the pypowervm requirement everywhere we need to before devstack even starts
14:21:24 <efried> esberglu, yes, but we need a solution that works for devstack in general.
14:21:26 <thorst_> esberglu: no, cause its too high a version, so upper constraints installs over it
14:21:41 <thorst_> esberglu: sorry, I misread.
14:21:57 <esberglu> efried: Yeah I just meant for now. So we can get CI up
14:22:56 <thorst_> esberglu: yeah, lets I guess do it for now.
14:23:25 <esberglu> #action esberglu: Change pypowervm requirements before CI runs
14:25:18 <thorst_> alright...lets do that for CI, cool off and figure out how the heck we do this moving forward.
14:25:41 <esberglu> Sounds good. Anything else before I end the meeting? I know we are way long
14:26:12 <efried> It looks like, if we wanted to do this in the *-powervm projects' devstack scripts, it would install over the top of 1.0.0.3.9.
14:26:41 <efried> ...which gets pulled down *with* each *-powervm in a way we have no control over.
14:27:12 <thorst_> yep.  Lets close up meeting here
14:27:15 <thorst_> we can still discuss
14:27:28 <esberglu> #endmeeting