| *** thorst has joined #openstack-powervm | 00:11 | |
| *** edmondsw has joined #openstack-powervm | 00:15 | |
| *** thorst has quit IRC | 00:15 | |
| *** edmondsw has quit IRC | 00:19 | |
| *** thorst has joined #openstack-powervm | 00:21 | |
| *** thorst has quit IRC | 00:28 | |
| *** zerick has quit IRC | 00:40 | |
| *** zerick has joined #openstack-powervm | 00:42 | |
| *** thorst has joined #openstack-powervm | 00:52 | |
| *** thorst has quit IRC | 00:52 | |
| *** thorst has joined #openstack-powervm | 01:26 | |
| *** thorst has quit IRC | 01:26 | |
| *** chas has joined #openstack-powervm | 02:16 | |
| *** chas has quit IRC | 02:21 | |
| *** zerick has quit IRC | 03:16 | |
| *** jay1_ has joined #openstack-powervm | 03:32 | |
| *** adreznec has quit IRC | 04:26 | |
| *** zerick has joined #openstack-powervm | 04:26 | |
| *** adreznec has joined #openstack-powervm | 04:27 | |
| *** shyama has joined #openstack-powervm | 04:32 | |
| *** chas has joined #openstack-powervm | 05:16 | |
| *** jay1_ has quit IRC | 05:16 | |
| *** chas has quit IRC | 05:20 | |
| *** jay1_ has joined #openstack-powervm | 05:30 | |
| *** esberglu has joined #openstack-powervm | 05:49 | |
| *** esberglu has quit IRC | 05:53 | |
| *** openstackgerrit has joined #openstack-powervm | 06:12 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/nova-powervm master: Updated from global requirements https://review.openstack.org/454506 | 06:12 |
|---|---|---|
| *** jay1_ has quit IRC | 07:21 | |
| *** shyama has quit IRC | 07:25 | |
| *** k0da has joined #openstack-powervm | 07:37 | |
| *** jay1_ has joined #openstack-powervm | 07:39 | |
| *** chas has joined #openstack-powervm | 07:40 | |
| *** chas has quit IRC | 07:41 | |
| *** chas has joined #openstack-powervm | 07:41 | |
| *** chas has quit IRC | 07:44 | |
| *** chas has joined #openstack-powervm | 07:44 | |
| *** chas has quit IRC | 07:49 | |
| *** shyama has joined #openstack-powervm | 07:51 | |
| *** shyama_ has joined #openstack-powervm | 07:57 | |
| *** shyama has quit IRC | 07:59 | |
| *** shyama_ is now known as shyama | 07:59 | |
| *** shyama has quit IRC | 08:28 | |
| *** esberglu has joined #openstack-powervm | 08:32 | |
| *** esberglu has quit IRC | 08:36 | |
| *** esberglu has joined #openstack-powervm | 09:29 | |
| *** esberglu has quit IRC | 09:34 | |
| *** chas has joined #openstack-powervm | 09:45 | |
| *** chas has quit IRC | 09:50 | |
| *** jay1_ has quit IRC | 09:51 | |
| *** chas has joined #openstack-powervm | 11:04 | |
| *** chas has quit IRC | 11:09 | |
| *** openstackgerrit has quit IRC | 11:18 | |
| *** thorst has joined #openstack-powervm | 11:47 | |
| *** shyama has joined #openstack-powervm | 11:58 | |
| *** edmondsw has joined #openstack-powervm | 12:09 | |
| *** shyama_ has joined #openstack-powervm | 12:12 | |
| *** shyama has quit IRC | 12:15 | |
| *** shyama_ is now known as shyama | 12:15 | |
| *** jpasqualetto has joined #openstack-powervm | 12:22 | |
| *** jpasqualetto has quit IRC | 12:27 | |
| *** jpasqualetto has joined #openstack-powervm | 12:40 | |
| *** esberglu has joined #openstack-powervm | 12:43 | |
| *** chas has joined #openstack-powervm | 13:05 | |
| *** smatzek has joined #openstack-powervm | 13:10 | |
| *** chas has quit IRC | 13:10 | |
| *** apearson has joined #openstack-powervm | 13:10 | |
| *** smatzek has quit IRC | 13:12 | |
| *** smatzek has joined #openstack-powervm | 13:12 | |
| *** mdrabe has joined #openstack-powervm | 13:14 | |
| thorst | esberglu: do we have the capability to get certain runs through on the CI? | 13:28 |
| thorst | I noticed the nodes weren't cleaning up still...but I think you had a manual way to do that? | 13:29 |
| esberglu | I mean I can go through and manually wipe them out. But I took it back down to debug this morning | 13:29 |
| thorst | I really want efried's fixes through today, and we need to fix the big ceilometer-powervm backlog we have at the moment. | 13:29 |
| thorst | we've got kind of a mess in the ceilometer space atm... | 13:29 |
| thorst | maybe we just enable for the -powervm projects during this interim while we debug overall? | 13:30 |
| esberglu | Yeah. I wish I would have kept the staging up so we could just look at the results there | 13:30 |
| esberglu | Cuz now I can't get that back up either | 13:30 |
| esberglu | Also some of the runs going through were failing like everything | 13:31 |
| *** shyama has quit IRC | 13:31 | |
| esberglu | So I don't really know that trying to get it back up is a good solution | 13:31 |
| thorst | esberglu: well, everything will fail until we get efried's change delivered. | 13:32 |
| thorst | they'll just hang indefinitely | 13:32 |
| esberglu | Oh yeah duh | 13:32 |
| thorst | so we need to push that through to get out of the suck | 13:32 |
| esberglu | How about this | 13:32 |
| esberglu | I try to get staging back up this morning in the background | 13:33 |
| esberglu | While I continue looking into the production issue | 13:33 |
| esberglu | If I don't make progress by midday | 13:33 |
| esberglu | I deploy prod with just the *-powervm projects | 13:33 |
| thorst | OK - my goals today...get efried's patch in and get the crap in ceilometer-powervm fixed up and delivered... | 13:34 |
| esberglu | Depends on how fast we need efried's fix in today I guess | 13:34 |
| thorst | so a mid day go round for that is good for me | 13:34 |
| thorst | mid day maybe being 11 AM CST? | 13:34 |
| thorst | cause the CI will take 90 min or so to go through for his change | 13:34 |
| thorst | and then the ceilometer-powervm ones need to run behind that | 13:35 |
| esberglu | Maybe I should just go ahead and do it now | 13:35 |
| esberglu | Because deploying management will take an hour | 13:35 |
| esberglu | and then building the image template will take another 40 | 13:36 |
| thorst | and while that's going you can debug staging as well? | 13:36 |
| esberglu | Yeah | 13:36 |
| thorst | I'm good with that | 13:36 |
| esberglu | Okay | 13:36 |
| thorst | not sure about efried...he may not like it | 13:36 |
| thorst | but he's not here to argue atm | 13:36 |
| thorst | :-) | 13:36 |
| esberglu | I'll let you know when it's ready | 13:36 |
| thorst | thx dude | 13:36 |
| esberglu | But this means CI probably won't be truly fixed until early next week | 13:37 |
| thorst | understood. | 13:37 |
| thorst | we just need a way to get the -powervm backlog solved | 13:37 |
| thorst | cause my queue kept me awake last night. | 13:37 |
| esberglu | Yep. Sounds like a plan | 13:37 |
| *** adreznec has quit IRC | 13:39 | |
| *** dwayne has quit IRC | 13:47 | |
| *** adreznec has joined #openstack-powervm | 14:00 | |
| *** thorst has quit IRC | 14:15 | |
| *** thorst has joined #openstack-powervm | 14:18 | |
| *** adreznec has quit IRC | 14:23 | |
| *** smatzek has quit IRC | 14:37 | |
| *** thorst has quit IRC | 14:39 | |
| *** dwayne has joined #openstack-powervm | 14:40 | |
| efried | Hey guys, sorry, had an awards assembly for a kid. It's not a diaper, but it takes longer ;-) | 14:41 |
| efried | esberglu You need anything from me atm? | 14:41 |
| *** adreznec has joined #openstack-powervm | 14:42 | |
| *** thorst has joined #openstack-powervm | 14:43 | |
| *** thorst has quit IRC | 14:43 | |
| *** thorst has joined #openstack-powervm | 14:44 | |
| efried | esberglu Is the CI in a state where I can try a recheck on https://review.openstack.org/#/c/453233/ ? | 14:47 |
| esberglu | efried: No we were just discussing what to do since CI was down and we need to get some changes through | 14:47 |
| esberglu | Here's the plan | 14:47 |
| esberglu | Nodepool still isn't deleting nodes | 14:47 |
| esberglu | Meaning I have to manually clean them up until it is fixed | 14:47 |
| esberglu | For the time being we have limited production CI to only the *-powervm projects | 14:48 |
| efried | But that doesn't affect the ability to do runs, right? | 14:48 |
| esberglu | To limit the amount of manual cleanup needed | 14:48 |
| efried | Just means you have to babysit the pool? | 14:48 |
| esberglu | Basically. | 14:48 |
| efried | Okay, so let's do that at least long enough to get https://review.openstack.org/#/c/453233/ through. | 14:48 |
| esberglu | I redeployed the prod pool again, nodes are spawning as we speak | 14:48 |
| esberglu | You can kick off a recheck now | 14:48 |
| esberglu | zuul will pick it up | 14:48 |
| esberglu | but it will still be maybe 20 min before a node is ready to run it on | 14:49 |
| esberglu | In the meantime I'm working on staging CI to see if this nodepool stuff is hitting us there | 14:49 |
| esberglu | To determine whether it's an environmental thing | 14:49 |
| esberglu | or not | 14:49 |
| esberglu | But long story short you can get runs through on anything *-powervm | 14:50 |
| esberglu | But not nova for now | 14:50 |
| thorst | esberglu: and the ready nodes have the latest pypowervm? | 14:51 |
| *** adreznec has quit IRC | 14:52 | |
| esberglu | Yeah the out of tree runs pick up develop | 14:52 |
| *** adreznec has joined #openstack-powervm | 14:53 | |
| thorst | efried: my Ceilometer gameplan... | 14:58 |
| thorst | 1) get your fix in for CI. Make sure it works. | 14:58 |
| thorst | 2) I think gordc's change here is ready to go in once it passes CI: https://review.openstack.org/#/c/448134/2 | 14:59 |
| thorst | 3) Then his update here: https://review.openstack.org/#/c/448249/ | 14:59 |
| thorst | I'll rebase that one once 2 is in | 14:59 |
| thorst | 4) Merge this in once CI is good: https://review.openstack.org/#/c/452712/ | 14:59 |
| *** mdrabe has quit IRC | 14:59 | |
| thorst | 5) Rebase Gautam's change with gordc's from items 2/3 | 15:00 |
| thorst | I suspect item 5 won't be much at that point, I'll connect up with Gautam offline to see if there are still changes to be made | 15:00 |
| esberglu | efried: thorst: Kicked off a recheck for 1) | 15:04 |
| thorst | thx | 15:05 |
| *** mdrabe has joined #openstack-powervm | 15:06 | |
| *** chas has joined #openstack-powervm | 15:06 | |
| efried | esberglu thorst Looking at the console log for the last failing run, funkiness is happening with pypowervm. | 15:06 |
| efried | To the point where I'm not sure we're getting the version we want. | 15:06 |
| efried | First of all, it's grabbing the develop branch into /opt/stack/pypowervm. Which would be okay, since develop is past 1.1.1 - but I think it's still not what we want. | 15:07 |
| thorst | efried: we did that because we had to patch things into pypowervm for the remotability aspects | 15:07 |
| efried | But then later on, as networking-powervm is installing, it claims to be using pypowervm from /usr/local/lib/python2.7/dist-packages/pypowervm | 15:09 |
| efried | And I have no idea what version that guy is. | 15:09 |
| efried | Same thing happens when installing ceilometer-powervm | 15:09 |
| efried | Point is, by the time stack is done, I'm not sure which instance we're actually using, or what version it's at. | 15:10 |
| efried | Ima check the compute log now and see if I can figure that out. | 15:11 |
| *** chas has quit IRC | 15:11 | |
| thorst | efried: yeah...super ... odd | 15:11 |
| efried | compute is using /usr/local/lib... | 15:11 |
| efried | esberglu thorst Ah, I think we may have a stale marker LU. | 15:12 |
| efried | Or did, at the time. | 15:13 |
| thorst | I have to defer to esbergluhere... | 15:13 |
| esberglu | Should be good then this run. I ran the cleaner that wipes all the vms and lus | 15:13 |
| efried | okay, good. | 15:13 |
| esberglu | Before deploying mgmt | 15:13 |
| esberglu | efried: Nodes just came online, run just started | 15:15 |
| efried | The bad(ish) news is that we don't actually know whether that pypowervm is 1.1.1 (or later). esberglu is there a node running that's already stacked that I can log into quick? | 15:15 |
| *** shyama has joined #openstack-powervm | 15:15 | |
| efried | PM me an IP? | 15:16 |
| esberglu | se | 15:16 |
| esberglu | sec | 15:16 |
| efried | The good(ish) news is that networking- and ceilometer-powervm are still reqing pypowervm without a version, so it ain't gonna blow away whatever version is there. | 15:16 |
| efried | But it's still a mystery why it ends up in /usr/local/lib | 15:16 |
| efried | esberglu After this shitstorm has abated, I think we should fix it not to use develop - just to use the requirements version. We can still patch in local2remote. If it's a matter of finding out where it's installed, we can do that too. | 15:17 |
| esberglu | I'm already doing that for in-tree | 15:18 |
| esberglu | So no problem | 15:18 |
| esberglu | We are using the u-c for in-tree | 15:18 |
| esberglu | Do we want that or g-r? | 15:19 |
| esberglu | I think they are the same right now | 15:19 |
| esberglu | But don't have to be | 15:19 |
| efried | esberglu I don't think we should be doing anything - let pypowervm get pulled in from the nova req. | 15:20 |
| efried | Ohh. | 15:20 |
| esberglu | We can't | 15:20 |
| efried | We need pypowervm - including remote - before we stack. | 15:20 |
| esberglu | Yep | 15:20 |
| esberglu | So we have to wipe the reqs before stacking | 15:20 |
| efried | Do we need it before we clone nova-powervm? | 15:20 |
| efried | and/or nova? | 15:21 |
| esberglu | Yeah we need it to be in for the ready node script | 15:21 |
| esberglu | Which is before that | 15:21 |
| efried | Well, we could still get around that. | 15:21 |
| efried | esberglu So if we wanted to make sure we get the nova-powervm (or whatever) requirements.txt version of pypowervm (and everything else), and we wanted to do it up front before cloning anything, we can do something like: | 15:28 |
| efried | sudo pip install -r https://git.openstack.org/cgit/openstack/nova-powervm/tree/requirements.txt | 15:28 |
| thorst | efried: I thought we actually already do that for each project | 15:29 |
| efried | I just tried that and it works. | 15:29 |
| thorst | as part of the nightly build or something | 15:29 |
| efried | thorst During stacking. This is before that. | 15:29 |
| thorst | no, I thought as part of nightly | 15:29 |
| thorst | so that during stacking it took less time | 15:29 |
| efried | Now, we might want to be a tad smarter about it and try to figure out how to make a URL explicitly to the change set we're testing. | 15:29 |
| efried | thorst All I can tell you is the console log for a given run shows us explicitly cloning and installing pypowervm. | 15:29 |
| thorst | efried: right...but I think that's because we ripped pypowervm out of the requirements.txt | 15:30 |
| esberglu | Let me explain what happens now | 15:30 |
| thorst | I'm just saying, if we don't rip pypowervm out of the requirements it may already be there... | 15:30 |
| esberglu | There seems to be confusion | 15:30 |
| esberglu | So during the prepare_node_powervm script (nightly image build) | 15:31 |
| esberglu | We install develop pypowervm | 15:31 |
| esberglu | and apply local2remote | 15:32 |
| esberglu | because we need it for the ready node script | 15:32 |
| esberglu | Then when a run gets kicked off | 15:32 |
| esberglu | For in-tree: We install pypowervm from requirements upper-constraints | 15:33 |
| esberglu | For out of tree: we install develop | 15:33 |
| esberglu | Both with the local2remote applied | 15:34 |
| efried | In both cases? | 15:34 |
| esberglu | Yeah right now it installs again for OOT (dumb, we don't need to do that, already installed) | 15:34 |
| efried | Okay, that's cool. So we're replacing the image build's develop+local2remote during the run anyway. | 15:35 |
| efried | But | 15:35 |
| esberglu | Actually not dumb | 15:35 |
| esberglu | In case new changes are in develop that we want | 15:35 |
| esberglu | Anyhow | 15:35 |
| esberglu | After it gets installed we wipe pypowervm from | 15:35 |
| efried | But we're removing the pypowervm line from requirements (just g-r, or also project requirements files?) | 15:35 |
| esberglu | requirements: upper-constraints and global-requirements | 15:36 |
| esberglu | As well as nova and nova-powervm requirements | 15:36 |
| esberglu | Otherwise | 15:36 |
| esberglu | When we stack, devstack thinks it's a version that isn't allowed | 15:36 |
| esberglu | and overwrites | 15:36 |
| efried | Okay. Nothing during stacking itself actually needs pypowervm, though (right?) | 15:36 |
| esberglu | I don't believe so | 15:37 |
| efried | And your script gets control again after devstack so it can do more stuff. | 15:37 |
| *** k0da has quit IRC | 15:37 | |
| esberglu | Yep | 15:37 |
| efried | So I kinda suspect if there's a requirements bump of pypowervm within one of our change sets, it won't get honored anyway, as we're currently set up. | 15:37 |
| efried | That said, we shouldn't be getting requirements bumps in random change sets anymore anyway - just from the bot. | 15:38 |
| efried | But | 15:38 |
| efried | If we ever happen to need one of those to make CI succeed, we'll be scrood. | 15:38 |
| efried | esberglu Does the per-run stuff need pypowervm+local2remote *before* stacking? | 15:39 |
| esberglu | Yeah | 15:39 |
| efried | Let's table that for a sec - if it does, we'll just have an extra step. | 15:40 |
| efried | So what I think we should do is, first thing after devstack, explicitly pip install -r <project that we're testing>/requirements.txt and then apply local2remote to whatever pypowervm exists on the other side of that. | 15:41 |
| efried | (We should also put an explicit version of pypowervm into networking-powervm and ceilometer-powervm requirements.txt - I think that's going to come once the bot stuff is active.) | 15:41 |
| thorst | efried: yeah...I agree that networking-powervm and ceilometer-powervm can now be pinned | 15:41 |
| thorst | instead of develop | 15:42 |
| thorst | #GrowingUp | 15:42 |
| efried | And if it turns out we don't need pypowervm+local2remote *before* devstack, we can just take that part out. | 15:42 |
| *** mdrabe has left #openstack-powervm | 15:42 | |
| *** mdrabe has joined #openstack-powervm | 15:42 | |
| * efried resists making another diaper joke | 15:42 | |
| * thorst remembers the 4 am projectile from 8 hours ago... | 15:44 | |
| efried | adreznec - Once this mess is sorted out, there's change sets in the pipe to add networking- and ceilometer-powervm to the requirements update bot, right? | 15:45 |
| adreznec | yep | 15:45 |
| thorst | omg adreznec is here? | 15:46 |
| efried | Okay, so how's that plan sound esberglu? | 15:46 |
| thorst | quick! pile on work! | 15:46 |
| adreznec | https://review.openstack.org/#/c/452220/ and https://review.openstack.org/#/c/452222/, then https://review.openstack.org/#/c/452226/ | 15:46 |
| adreznec | One part of it already merged | 15:46 |
| efried | esberglu ==> First thing after devstack, explicitly pip install -r <project that we're testing>/requirements.txt and then apply local2remote (and any other custom patches) to whatever pypowervm exists on the other side of that. | 15:47 |
| *** shyama has quit IRC | 15:47 | |
| esberglu | efried: Sounds reasonable. I'm looking at the scripts quick to make sure we aren't missing anything | 15:48 |
| adreznec | efried: esberglu I know we've had issues in the past with needing updates to the patch to match latest develop... | 15:49 |
| adreznec | How are we planning to handle multiple patch versions | 15:49 |
| efried | adreznec We're at a place where we shouldn't be using anything from develop except local2remote anymore. | 15:50 |
| adreznec | Sure | 15:51 |
| esberglu | efried: Won't we need to apply the patch before pip installing (after stacking)? | 15:51 |
| efried | Swhy we went through all the trouble of breaking the pypowervm release process free. | 15:51 |
| adreznec | But what if 1.1.1 and 1.1.2 have changes in between that require different patch versions | 15:51 |
| adreznec | and Pike reqs 1.1.1 and Queens reqs 1.1.2 | 15:51 |
| esberglu | Remind me why we can't just get the local2remote stuff included in pypowervm? | 15:53 |
| thorst | esberglu: local2remote basically makes it so we look at the RMC connection and identify the IP address to remote to | 15:54 |
| thorst | but then we mask it to look like its local | 15:54 |
| thorst | even though its remote... | 15:54 |
| efried | thorst It could possibly be done, though right now we've got some defaults in the patch that we should remove (like username/password for th REST server) | 15:55 |
| esberglu | It would make this all a lot easier | 15:55 |
| efried | True dat. | 15:55 |
| efried | Course it wouldn't help us until we release 1.1.2 ;-) | 15:56 |
| esberglu | It would knock out all the issues we are discussing right now | 15:56 |
| thorst | removing it? Sure. But yeah, that | 15:56 |
| thorst | and also, how to find user name/password | 15:56 |
| efried | Let me look into it. | 15:56 |
| thorst | and I don't want the nova-powervm code having options to pass that in | 15:56 |
| thorst | because then it looks like we support running nova-powervm from a remote server | 15:56 |
| thorst | and we really don't...just for CI | 15:56 |
| adreznec | At one point we talked about possibly having pypowervm look for those creds in a config file | 15:57 |
| thorst | adreznec: I don't remember that...but I don't hate that idea. | 15:57 |
| adreznec | That could be dropped into a CI environment | 15:57 |
| adreznec | Can't remember why we didn't go that route though | 15:58 |
| esberglu | Yeah and I could just ansible vault that file | 15:58 |
| thorst | probably because if someone knew about it they could still make a case for remotability | 15:58 |
| thorst | where as this was so hidden (because we were literally patching the code) | 15:58 |
| thorst | but it's biting us as we mature. | 15:58 |
| esberglu | Yeah. It will get especially sloppy once we are running multiple branches for in-tree and out-of-tree | 15:59 |
| thorst | yeah, so I'mgood with something for 1.1.2 that mimicks this behavior | 16:01 |
| thorst | will just be a pain supporting newton, pike, etc... | 16:02 |
| efried | thorst esberglu We _do_ support having the creds in a config file, but we generate the file with defaults if we don't find it. And I'm pretty sure that's the code path we're taking in the CI, cause we're not shipping that file, and the setup is only getting done once. | 16:02 |
| thorst | efried: when did that support drop in? | 16:02 |
| thorst | and what about an IP | 16:02 |
| efried | I dunno, I remember working on it a few months ago, but I don't remember if I introduced it. | 16:03 |
| efried | The IP is detected based on RMC something-or-other. | 16:03 |
| thorst | we have to give it an address to use...though I suspect we could hard code that to 192.128.168.1 or something | 16:03 |
| thorst | whoa...then maybe problem solved? | 16:03 |
| efried | Well, here's what I'm thinking. | 16:03 |
| efried | I'd like to put the vast majority of this code into a separate .py that lives in the CI project. | 16:04 |
| efried | And have the pypowervm setup do a conditional local import of that .py maybe based on some env var (_PYPOWERVM_LOCAL2REMOTE_DO_NOT_USE_THIS_UNLESS_YOU_ARE_US=1). | 16:05 |
| efried | Better yet: _PYPOWERVM_LOCAL2REMOTE_DO_NOT_USE_THIS_UNLESS_YOU_ARE_US=/path/to/local2remote.py | 16:06 |
| thorst | a dev flag? | 16:06 |
| thorst | adreznec: how's that sit with you? | 16:06 |
| thorst | I can be convinced of that personally | 16:07 |
| thorst | (does this push us out to not having the nova-powervm change merged today?) | 16:08 |
| efried | Nono, separate discussion. | 16:08 |
| thorst | k | 16:08 |
| thorst | good | 16:08 |
| efried | The nova-powervm change should be CIing right now, and it has all the pieces in place from what esberglu and I can tell. | 16:08 |
| adreznec | thorst: efried seems fair, though we'll have to support the existing patch mechanism for a while I guess due to Newton/Ocata | 16:08 |
| thorst | yep... | 16:09 |
| thorst | chip away at it | 16:09 |
| efried | Roger wilco. Will need esberglu's help figuring out how to ship the files. esberglu Where do you want them located? | 16:10 |
| efried | I think I'll prolly want to ship a .py and a .sh | 16:10 |
| esberglu | They should go in | 16:10 |
| efried | Since all of the host discovery nonsense is shell stuff anyway. | 16:10 |
| esberglu | neo-os-ci/ci-ansible/roles/ci-management/templates/scripts/ | 16:11 |
| efried | esberglu rgr. On it. | 16:11 |
| esberglu | This is fantastic | 16:11 |
| thorst | a rare fantastic. :-) | 16:17 |
| *** shyama has joined #openstack-powervm | 16:24 | |
| *** chas has joined #openstack-powervm | 17:07 | |
| *** chas has quit IRC | 17:12 | |
| thorst | efried: no love on your change | 17:23 |
| thorst | looks to me like a timeout again | 17:23 |
| efried | eff | 17:23 |
| efried | Results not posted yet? | 17:23 |
| thorst | o, duh | 17:24 |
| thorst | I'm a moron | 17:24 |
| efried | You looking at a different branch? | 17:24 |
| thorst | I was looking at Apr 6th results | 17:24 |
| thorst | I assumed they had to be done by now | 17:24 |
| efried | gooood. | 17:24 |
| thorst | whoopsy | 17:24 |
| efried | Though I would have expected results by now. | 17:24 |
| efried | which doesn't bode well. | 17:24 |
| efried | Should take <2h, neh? | 17:24 |
| thorst | 2 hour 10 min by now | 17:25 |
| thorst | (I'm on the jenkins) | 17:25 |
| thorst | just hopped on the node | 17:26 |
| thorst | not timing out, but hitting some exceptions | 17:26 |
| thorst | HTTP error 400 for method PUT on path /rest/api/web/File/contents/c4fc0760-6c99-412f-8743-6854534560fe: Bad Request -- REST002C Content-Length specified in header does not match that of the meta file: 104,857,600 | 17:26 |
| thorst | so uhh...looks to me like it found a legit bug? | 17:27 |
| thorst | efried: want on the system? | 17:36 |
| thorst | while it's still running? | 17:36 |
| efried | thorst Was that in config drive? | 17:37 |
| efried | What do we have that's 100MB? | 17:37 |
| efried | We're not using a dummy image, are we? | 17:37 |
| thorst | that's the ole dummy image | 17:38 |
| thorst | which is 100 MB | 17:38 |
| thorst | but I mean, images can be a couple megs | 17:38 |
| efried | k, thought we were using a real image these days. | 17:38 |
| thorst | we just increase the size to 1 GB later | 17:38 |
| efried | So this is a PITA; I don't really want the REST server checking File sizes. | 17:39 |
| efried | And I think we can get around it. | 17:39 |
| efried | Looking... | 17:39 |
| efried | Damn. Not without another pypowervm change. | 17:40 |
| efried | least, not for SSP. | 17:40 |
| efried | Damn, damn, damn. Why wouldn't this show up in testing? | 17:40 |
| *** jpasqualetto has quit IRC | 17:43 | |
| thorst | you used whole round numbers | 17:44 |
| efried | thorst as opposed to what? | 17:45 |
| efried | It's a byte size at this point. We don't have fractional bytes. | 17:47 |
| efried | This is the byte size we get directly from glance. | 17:47 |
| efried | It passes alll the way through. | 17:47 |
| thorst | hmm... | 17:53 |
| thorst | well, that should be fine, right? | 17:59 |
| thorst | why would the content length be different then? | 17:59 |
| thorst | is it because we round up to a gig? | 18:00 |
| thorst | that may be something we can fix on the REST side... | 18:00 |
| thorst | thoughts? | 18:02 |
| efried | thorst Yah, but we wouldn't want to have to require a new pvm-rest back to newton. | 18:02 |
| efried | would we? | 18:02 |
| thorst | hm | 18:02 |
| thorst | not sure...I mean, its not like you can go back once a new one is out | 18:02 |
| efried | There's no rounding, btw. | 18:02 |
| thorst | well, maybe we just need Hsien to look asap | 18:03 |
| efried | I looked at this with apearson already. | 18:03 |
| thorst | and the fix is in rest or? | 18:03 |
| efried | thorst Wait, is that node still running?? | 18:03 |
| efried | I need to look at the REST logs quick. | 18:04 |
| thorst | no, finished finally | 18:04 |
| thorst | kick off a recheck | 18:04 |
| thorst | wait an hour | 18:04 |
| thorst | then you have two hours with it | 18:04 |
| thorst | although, REST logs are on the NL | 18:04 |
| thorst | and that's still alive | 18:04 |
| esberglu | The node is still alive though | 18:04 |
| efried | ohh, riight. | 18:04 |
| esberglu | Since they arent deleting | 18:04 |
| efried | Yeah, I need the neo. PM me that? | 18:04 |
| thorst | I have the neo IP | 18:04 |
| thorst | sending your way | 18:05 |
| efried | Okay yeah. | 18:05 |
| esberglu | I found something suspicious that might be behind the nodepool stuff | 18:06 |
| thorst | doooo tell | 18:06 |
| esberglu | When I check the status of jenkins | 18:06 |
| efried | So this is a problem we were seeing with the requests library defaulting Content-Length to zero when it detects a certain type of input stream. | 18:06 |
| efried | I thought I had it sussed. | 18:06 |
| efried | And I tested it, dammit. | 18:06 |
| esberglu | The status is | 18:07 |
| esberglu | Active: active (exited) | 18:07 |
| esberglu | I'm looking at what that means | 18:07 |
| esberglu | But I think that should be | 18:07 |
| esberglu | active (running) | 18:07 |
| thorst | actively exited :-) | 18:07 |
| efried | So assuming it really doesn't work the way I've got it in pypowervm now, the possible solutions are: | 18:08 |
| efried | In pypowervm, quit populating the stupid f_size field on the File. apearson asserts that, if that field is left empty, they don't do that check. | 18:08 |
| efried | Or | 18:08 |
| efried | In REST, accept Content-Length of zero. | 18:09 |
| efried | apearson proposed a change to do just that, a few months ago, but we decided we didn't need it, so we abandoned it. | 18:09 |
| thorst | efried: maybe we could make a local2zero patch that we put in pypowervm as part of the CI? | 18:09 |
| thorst | (kidding) | 18:09 |
| efried | I can't imagine this has anything to do with local vs remote. | 18:09 |
| thorst | if we do the content length of zero...is the change in nova-powervm? | 18:09 |
| efried | That's the thing - local or remote should now be using the exact same code path - IO_STREAM and API upload. | 18:10 |
| thorst | (which I like more now...) | 18:10 |
| thorst | my joke was lost on you...disregard the patch thing | 18:10 |
| efried | No, but it's a valid thing to think through anyway. | 18:11 |
| efried | We can't finagle it from nova-powervm. Change has to be in pypowervm. | 18:11 |
| thorst | great... | 18:11 |
| thorst | I'd say we should stop filling in the field on the file | 18:12 |
| thorst | and try that | 18:12 |
| thorst | MAYBE do a patch to get CI running until 1.1.2 goes out? | 18:12 |
| thorst | (which may need to be sooner rather than later...) | 18:12 |
| efried | esberglu How long to get a new quick-and-dirty pypowervm patch pulled in? | 18:13 |
| esberglu | Instantly | 18:13 |
| *** jpasqualetto has joined #openstack-powervm | 18:13 | |
| esberglu | Just have to reload the jobs | 18:13 |
| efried | esberglu 5109 (thorst) | 18:14 |
| efried | The real solution will be somewhat less sledgehammery. | 18:14 |
| efried | Simpler might be to get apearson to put out a new pvm-rest with the Content-Length=0 tolerance and respin the node image with it. | 18:16 |
| esberglu | efried: Jobs are updated. Shoot | 18:17 |
| efried | Shoot what? Recheck? | 18:17 |
| efried | thorst The other thing we could do is reintroduce IterableToFileAdapter. Then the solution would be in nova-powervm only. We only ever saw this problem through the use-the-iterable-directly-from-glance path (even though I f**king FIXED it). | 18:18 |
| thorst | well, maybe that's our solution for earlier releases | 18:20 |
| thorst | and maybe even this release | 18:20 |
| thorst | until we do 1.1.2 properly... | 18:20 |
| efried | The earlier releases do have IterableToFileAdapter, I believe. | 18:20 |
| efried | yeah. | 18:20 |
| efried | Can't remember exactly why. | 18:20 |
| esberglu | efried: Yeah you can recheck, 5109 is in | 18:21 |
| *** smatzek has joined #openstack-powervm | 18:21 | |
| efried | Because we thought we could make them work with an older pypowervm - IO_STREAM + coordinated? | 18:21 |
| efried | k, recheck going. | 18:21 |
| esberglu | Nice we got a +2 on the 1st in-tree patch | 18:32 |
| efried | I think he's taking it back. | 18:33 |
| *** smatzek has quit IRC | 18:33 | |
| esberglu | well shit | 18:33 |
| esberglu | Small stuff? I didn't look yet | 18:33 |
| esberglu | Assuming it must be | 18:33 |
| efried | yeah. | 18:35 |
| efried | esberglu Will in-tree CI pass if we put up a new patch there? | 18:35 |
| thorst | they don't translate log messages anymore...interesting! | 18:35 |
| esberglu | It won't even run | 18:35 |
| efried | esberglu Can it be MADE to run? | 18:36 |
| efried | Or maybe we just stay quiet about it and let it NOT run. | 18:36 |
| efried | thorst See comment about differentiating between our driver and libvirt+phyp | 18:36 |
| efried | I didn't even know that existed. | 18:37 |
| efried | Or what it means. | 18:37 |
| efried | Can you help me come up with some text quick to explain that in our docstring? | 18:37 |
| thorst | yeah...the libvirt+phyp project is something that I believe is dead, calls back to an HMC, and I have no clue how it does virtual I/O | 18:37 |
| thorst | adreznec knows a bit more. | 18:37 |
| esberglu | efried: I could potentially turn on nova runs long enough to just let your run through then disable right after | 18:37 |
| efried | esberglu Stand by with that. | 18:38 |
| adreznec | libvirt+phyp is basically just it sshing into HMC/IVM and running CLI | 18:38 |
| adreznec | hasn't been updated in ages afaik | 18:39 |
| efried | Should I just add "...using NovaLink"? | 18:39 |
| efried | Or "PowerVM NovaLink implementation of Compute Driver."? | 18:39 |
| thorst | yeah, but I do wonder if those questions will persist. But I don't feel like code is the right place to sort that out | 18:40 |
| adreznec | mhm | 18:41 |
| efried | thorst adreznec Do virt drivers include any kind of readme in the code tree? Wouldn't have thought so. | 18:41 |
| thorst | I don't think so | 18:42 |
| adreznec | I mean you could definitely add Novalink in there to clarify | 18:42 |
| thorst | but we do have our powervm wiki | 18:42 |
| adreznec | Not that I've seen | 18:42 |
| efried | Kind of place where we would say, "Requires NovaLink with such-and-such packages blah blah" | 18:42 |
| thorst | https://wiki.openstack.org/wiki/PowerVM | 18:42 |
| thorst | and maybe reference it there? | 18:42 |
| thorst | we can put a blurb in that... | 18:42 |
| thorst | and then in the code, maybe a link to the wiki? That still feels a bit weird but... | 18:42 |
| adreznec | Well | 18:42 |
| adreznec | There's this for hyper-v | 18:42 |
| adreznec | https://docs.openstack.org/ocata/config-reference/compute/hypervisor-hyper-v.html | 18:42 |
| adreznec | Maybe we could do something similar for powervm? | 18:43 |
| thorst | right, but in time? | 18:43 |
| thorst | like...we'd need a while to get to the point to do that | 18:43 |
| thorst | like more than skeleton code :-) | 18:43 |
| adreznec | well... maybe it's just a skeleton doc | 18:43 |
| adreznec | :P | 18:43 |
| thorst | peace with that | 18:43 |
| efried | Should I include that PowerVM link in the docstring? | 18:44 |
| adreznec | I mean the lxc one is pretty short | 18:44 |
| adreznec | https://docs.openstack.org/ocata/config-reference/compute/hypervisor-lxc.html | 18:44 |
| efried | Yabut those are not in the code tree. | 18:45 |
| efried | I think mriedem just wants the docstring to make it clear that this isn't libvirt+phyp. | 18:45 |
| efried | I don't think he's asking for a full reference doc here. | 18:45 |
| efried | I'm looking short-term, just in this change set. | 18:45 |
| thorst | k | 18:46 |
| adreznec | I vote we just add the Novalink word like you said | 18:47 |
| thorst | then yeah, call out that it requires PowerVM NovaLink | 18:47 |
| adreznec | and link to the novalink page | 18:47 |
| adreznec | and be done with it then | 18:47 |
| esberglu | efried: Not gonna turn on CI quick for in tree #1? | 18:51 |
| efried | thorst What do you think? | 18:51 |
| thorst | if it'll pass...OK | 18:52 |
| efried | We've demonstrated that it passes. Not sure mriedem is going to be looking for it again at this point. | 18:52 |
| efried | Long as it doesn't show up failed. | 18:52 |
| esberglu | I think we just leave it | 18:52 |
| esberglu | If he asks, just say our CI is having issues | 18:52 |
| esberglu | We know that it is fine at this point | 18:52 |
| esberglu | It has passed like 20 times | 18:52 |
| esberglu | Well not that many but still | 18:52 |
| esberglu | And it's already +2 again | 18:53 |
| thorst | k...so back to the CI fiasco! | 18:55 |
| thorst | :-) | 18:55 |
| efried | phew | 18:56 |
| * efried is rebasing the whole pile again, and getting rid of _L* | 18:57 | |
| *** k0da has joined #openstack-powervm | 18:58 | |
| thorst | so who is doing something for CI now? | 18:59 |
| thorst | what's the two sentence there? | 18:59 |
| *** thorst has quit IRC | 19:03 | |
| *** chas has joined #openstack-powervm | 19:08 | |
| esberglu | Still have no real idea what's going on with the nodepool deletions | 19:08 |
| esberglu | But it is also hitting staging now | 19:08 |
| esberglu | Which helps narrow it down | 19:08 |
| efried | thorst bailed on us. | 19:09 |
| efried | I'm getting ready to test the local2remote stuff. | 19:09 |
| efried | But first, lunch. | 19:09 |
| efried | When thorst gets back, remind me to tell him about TaskFlow's PrintingDurationListener. Think it means we can get rid of BaseTask. | 19:10 |
| esberglu | Geez late lunch. Will do | 19:11 |
| *** chas has quit IRC | 19:12 | |
| *** thorst has joined #openstack-powervm | 19:32 | |
| thorst | looks like POK net just died... | 19:44 |
| efried | thorst TaskFlow has a way to do something like: | 19:53 |
| efried | with PrintingDurationListener(engine, LOG.info): | 19:54 |
| efried | engine.run() | 19:54 |
| efried | And that'll log the duration of each task for us | 19:54 |
| efried | So we could get rid of most of our base PowerVMTask. | 19:55 |
| thorst | neat. | 19:55 |
| efried | The only thing it would theoretically still need would be the instance var we're requiring in the constructor, used for logging. But we could just make that mandatory in the tasks themselves. | 19:55 |
| thorst | yeah...I do like having it in oslo.log though | 19:56 |
| efried | Having what in oslo.log? | 19:56 |
| thorst | instance=self.instance | 19:56 |
| thorst | it's so nice for a good log viewer | 19:56 |
| thorst | just putting in the instance.uuid and getting all the messages related to that | 19:56 |
| efried | Oh, yeah, we can still do that, just saying we would change the class sigs for our Tasks to accept that explicitly. | 19:57 |
| thorst | if you lose that...it makes debug harder. | 19:57 |
| thorst | even for printing the amount of time each task took? | 19:57 |
| efried | Oh, you want the instance ID in that one too... | 19:57 |
| efried | Well, we could extend the listener. | 19:57 |
| efried | Anyway, wishlist. | 19:58 |
| efried | I'll open a lp bug so we don't forget. | 19:58 |
| efried | Cause it ain't happenin now. | 19:58 |
| efried | https://bugs.launchpad.net/nova-powervm/+bug/1680947 FYI | 20:02 |
| openstack | Launchpad bug 1680947 in nova-powervm "Use PrintingDurationListener, get rid of PowerVMTask" [Wishlist,New] | 20:02 |
| *** apearson has quit IRC | 20:16 | |
| thorst | fair enough | 20:17 |
| *** apearson has joined #openstack-powervm | 20:18 | |
| *** esberglu has quit IRC | 20:27 | |
| *** esberglu has joined #openstack-powervm | 20:30 | |
| *** apearson has quit IRC | 20:34 | |
| *** apearson has joined #openstack-powervm | 20:37 | |
| efried | thorst Apparently I can't have >=1.1.0 in nova requirements if g-r says >=1.1.1. | 20:50 |
| efried | That's not what they done taught me in 3rd grade math, but whatever. | 20:50 |
| thorst | well, makes sense to me | 20:51 |
| *** apearson has quit IRC | 20:53 | |
| efried | >=1.1.1 is >=1.1.0. And I woulda thought g-r would go first. | 20:53 |
| efried | what, as I say, ever. | 20:53 |
| esberglu | It goes the other way I think | 20:53 |
| esberglu | So you can have >=1.1.2 in nova reqs? | 20:54 |
| esberglu | Not really sure how all that works though | 20:54 |
| esberglu | I just though of something that might be responsible for the nodepool stuff | 20:55 |
| esberglu | Jenkins has this ssh slaves plugin that we use | 20:55 |
| esberglu | And it published a warning about man-in-the-middle attacks and recommended upgrading | 20:55 |
| esberglu | Jenkins has this thing where you can delay the upgrade until next restart | 20:55 |
| *** apearson has joined #openstack-powervm | 20:55 | |
| esberglu | So I did that and kind of forgot | 20:55 |
| esberglu | But the timeframe makes sense | 20:55 |
| esberglu | So once the network is back up I'm gonna try downgrading that to the previous version | 20:56 |
| esberglu | (it turns out staging is NOT broken, just production) | 20:56 |
| esberglu | And staging has the old version | 20:57 |
| esberglu | Downgrading isn't a good plan long term because of that vulnerability | 20:57 |
| esberglu | But hopefully it will at least confirm my suspicion | 20:57 |
| thorst | and maybe let us fix it in staging instead of trying to fix it in production | 21:01 |
| thorst | :-) | 21:01 |
| *** edmondsw has quit IRC | 21:02 | |
| *** edmondsw has joined #openstack-powervm | 21:02 | |
| *** edmondsw has quit IRC | 21:07 | |
| *** chas has joined #openstack-powervm | 21:09 | |
| *** apearson has quit IRC | 21:13 | |
| *** chas has quit IRC | 21:14 | |
| esberglu | efried: You still around? | 21:14 |
| esberglu | Looks like your run didn't go any better this time around | 21:14 |
| esberglu | Same error it looks like.... | 21:15 |
| esberglu | Huh that patch didn't get into pypowervm... | 21:16 |
| esberglu | I can't get onto anything to see why not right now though | 21:18 |
| esberglu | efried: I'm heading out for the day. But if the network comes back up at some point I can recheck this weekend | 21:19 |
| *** esberglu has quit IRC | 21:19 | |
| *** thorst has quit IRC | 21:22 | |
| *** apearson has joined #openstack-powervm | 21:26 | |
| *** jpasqualetto has quit IRC | 21:34 | |
| *** jpasqualetto has joined #openstack-powervm | 21:51 | |
| *** k0da has quit IRC | 21:52 | |
| *** dwayne has quit IRC | 22:02 | |
| *** esberglu has joined #openstack-powervm | 22:07 | |
| *** esberglu has quit IRC | 22:12 | |
| *** jpasqualetto has quit IRC | 22:13 | |
| *** mdrabe has quit IRC | 22:14 | |
| *** apearson has quit IRC | 22:20 | |
| *** thorst has joined #openstack-powervm | 22:22 | |
| *** esberglu has joined #openstack-powervm | 22:28 | |
| *** thorst has quit IRC | 22:41 | |
| *** esberglu has quit IRC | 23:03 | |
| *** chas has joined #openstack-powervm | 23:09 | |
| *** chas has quit IRC | 23:14 | |
| *** esberglu has joined #openstack-powervm | 23:34 | |
| *** esberglu has quit IRC | 23:38 | |
| *** thorst has joined #openstack-powervm | 23:39 | |
| *** thorst has quit IRC | 23:43 | |
| *** thorst has joined #openstack-powervm | 23:56 | |
| *** esberglu has joined #openstack-powervm | 23:56 | |
| *** thorst has quit IRC | 23:57 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!