| *** esberglu has joined #openstack-powervm | 00:18 | |
| *** esberglu has quit IRC | 00:23 | |
| *** thorst has quit IRC | 00:33 | |
| *** esberglu has joined #openstack-powervm | 00:39 | |
| *** esberglu has quit IRC | 00:43 | |
| *** esberglu has joined #openstack-powervm | 01:14 | |
| *** esberglu has quit IRC | 01:19 | |
| *** thorst has joined #openstack-powervm | 01:34 | |
| *** thorst has quit IRC | 01:37 | |
| *** thorst has joined #openstack-powervm | 01:37 | |
| *** thorst has quit IRC | 01:37 | |
| *** esberglu has joined #openstack-powervm | 02:08 | |
| *** esberglu has quit IRC | 02:13 | |
| *** thorst has joined #openstack-powervm | 02:38 | |
| *** thorst has quit IRC | 02:57 | |
| *** esberglu has joined #openstack-powervm | 03:03 | |
| *** esberglu has quit IRC | 03:07 | |
| *** tjakobs__ has joined #openstack-powervm | 03:53 | |
| *** thorst has joined #openstack-powervm | 03:55 | |
| *** esberglu has joined #openstack-powervm | 03:57 | |
| *** thorst has quit IRC | 03:59 | |
| *** esberglu has quit IRC | 04:01 | |
| *** shyama_ has joined #openstack-powervm | 04:39 | |
| *** shyama has quit IRC | 04:42 | |
| *** shyama_ is now known as shyama | 04:42 | |
| *** esberglu has joined #openstack-powervm | 04:51 | |
| *** esberglu has quit IRC | 04:55 | |
| *** thorst has joined #openstack-powervm | 04:55 | |
| *** thorst has quit IRC | 05:00 | |
| *** tjakobs__ has quit IRC | 05:17 | |
| *** shyama_ has joined #openstack-powervm | 05:25 | |
| *** shyama has quit IRC | 05:27 | |
| *** shyama_ is now known as shyama | 05:27 | |
| *** esberglu has joined #openstack-powervm | 05:45 | |
| *** esberglu has quit IRC | 05:49 | |
| *** thorst has joined #openstack-powervm | 05:56 | |
| *** thorst has quit IRC | 06:00 | |
| *** esberglu has joined #openstack-powervm | 06:39 | |
| *** esberglu has quit IRC | 06:44 | |
| *** thorst has joined #openstack-powervm | 06:57 | |
| *** esberglu has joined #openstack-powervm | 07:00 | |
| *** thorst has quit IRC | 07:01 | |
| *** esberglu has quit IRC | 07:04 | |
| *** k0da has joined #openstack-powervm | 07:29 | |
| *** esberglu has joined #openstack-powervm | 07:35 | |
| *** esberglu has quit IRC | 07:39 | |
| *** thorst has joined #openstack-powervm | 07:58 | |
| *** thorst has quit IRC | 08:17 | |
| *** shyama has quit IRC | 08:43 | |
| *** shyama has joined #openstack-powervm | 08:43 | |
| *** thorst has joined #openstack-powervm | 09:14 | |
| *** thorst has quit IRC | 09:18 | |
| *** esberglu has joined #openstack-powervm | 09:23 | |
| *** esberglu has quit IRC | 09:28 | |
| *** esberglu has joined #openstack-powervm | 10:18 | |
| *** esberglu has quit IRC | 10:22 | |
| *** esberglu has joined #openstack-powervm | 11:11 | |
| *** thorst has joined #openstack-powervm | 11:15 | |
| *** esberglu has quit IRC | 11:16 | |
| *** thorst has quit IRC | 11:20 | |
| *** openstackgerrit has joined #openstack-powervm | 11:36 | |
| openstackgerrit | Shyama proposed openstack/nova-powervm master: FileIO adapter does not remove mappings on detach https://review.openstack.org/455269 | 11:36 |
|---|---|---|
| *** smatzek has joined #openstack-powervm | 11:48 | |
| *** thorst has joined #openstack-powervm | 11:55 | |
| *** esberglu has joined #openstack-powervm | 12:06 | |
| *** jpasqualetto has joined #openstack-powervm | 12:08 | |
| *** esberglu has quit IRC | 12:10 | |
| *** edmondsw has joined #openstack-powervm | 12:16 | |
| *** mdrabe has joined #openstack-powervm | 12:31 | |
| *** dwayne has quit IRC | 12:31 | |
| *** jpasqualetto has quit IRC | 12:55 | |
| *** jpasqualetto has joined #openstack-powervm | 13:17 | |
| *** jpasqualetto has joined #openstack-powervm | 13:17 | |
| thorst | efried: are we still held up on CI? I don't see esberglu on, but I'm 99% sure that's where we left off on Friday | 13:28 |
| efried | thorst I posted a recheck last night... | 13:29 |
| efried | but it failed. | 13:29 |
| thorst | going to look it up... | 13:29 |
| efried | In a different way than last time. | 13:29 |
| efried | http://184.172.12.213/33/453233/3/check/nova-powervm-out-of-tree-pvm/8abbd10/powervm_os_ci.html | 13:29 |
| efried | I'll recheck it again now, but this may be something esberglu needs to look at. | 13:30 |
| thorst | yeah...recheck that again. Possibly new keystone specific tests | 13:30 |
| thorst | sigh...I feel like even OOT may want to go whiteliest | 13:30 |
| thorst | however, you have some legit failures in there too | 13:30 |
| thorst | s/you've/we've | 13:31 |
| thorst | ServerDiskConfigTestJSON | 13:31 |
| thorst | HTTP error 400 for method PUT on path /rest/api/web/File/contents/a2335cef-4520-4657-87e3-7dd7eee1cc3d: Bad Request -- REST002C Content-Length specified in header does not match that of the meta file: 104,857,600 | 13:32 |
| *** openstackgerrit has quit IRC | 13:33 | |
| efried | Same as before. | 13:34 |
| *** esberglu has joined #openstack-powervm | 13:34 | |
| thorst | efried: do we handle that with the Iterable facade thing up in nova-powervm | 13:34 |
| *** esberglu has quit IRC | 13:34 | |
| thorst | while we work a proper fix for pypowervm 1.1.2 | 13:34 |
| *** esberglu has joined #openstack-powervm | 13:35 | |
| efried | Well, theoretically. | 13:35 |
| *** smatzek has quit IRC | 13:35 | |
| efried | That is, we never saw this problem when using IterableToFileAdapter. | 13:35 |
| thorst | might as well try it... | 13:35 |
| efried | I'm buggered if I understand why it's different from the façade I have in pypowervm. | 13:35 |
| efried | It also appears as though that pypowervm patch is still NOT in play here. | 13:36 |
| efried | Because the File element still has the ExpectedFileSizeInBytes field in it. | 13:36 |
| thorst | efried: yeah...we were trying a pypowervm patch that would remove that...but I think that we really want that facade anyway so we support older versions of nova-powervm | 13:37 |
| thorst | right? | 13:37 |
| efried | thorst Does not compute. | 13:37 |
| thorst | yeah...ok. Let me rephrase | 13:38 |
| efried | If the façade is in nova-powervm, how are we supporting older versions of nova-powervm? | 13:38 |
| efried | You mean older versions of pypowervm? | 13:38 |
| thorst | we can backport the facade into ocata and newton...which should work with the older versions of pypowervm | 13:38 |
| efried | thorst That's already done. | 13:38 |
| *** tjakobs__ has joined #openstack-powervm | 13:38 | |
| thorst | I thought the older ones were still using coordinated? | 13:39 |
| thorst | they aren't? | 13:39 |
| efried | https://review.openstack.org/#/c/453905/ (ocata) and https://review.openstack.org/#/c/453907/ (newton) are using IO_STREAM instead of FUNC. | 13:39 |
| efried | And will use coordinated or streaming purely depending on which pypowervm is under 'em. | 13:40 |
| thorst | I'll recheck those... | 13:41 |
| efried | tjakobs__ and I tested both of those last week. Coordinated was a hair slower, but both worked. Right tjakobs__ ? I can't remember which one(s) hung. | 13:41 |
| tjakobs__ | newer pypowervm with older nova_powervm was hanging. With backport/new nova_powervm patch I haven't seen it hang, but the deploys took an extra 15-20 ish seconds (of like 20-30 before) | 13:43 |
| tjakobs__ | of a 2G image | 13:43 |
| thorst | the previous runs (April 7th) timed out...so I'm re-running now | 13:43 |
| thorst | efried: could those be stuck because of a stale marker lu? | 13:46 |
| thorst | esberglu: did you just wipe out the current runs? | 13:47 |
| esberglu | No? | 13:48 |
| thorst | o, they all just came back as 'aborted' | 13:48 |
| thorst | all the current rechecks | 13:48 |
| thorst | also, it does look like we have stale marker LU's in some of the SSPs | 13:48 |
| esberglu | Still having net issues? | 13:48 |
| thorst | they said we were...but I'm not seeing any | 13:48 |
| esberglu | Hmm idk, I guess just kick off another round of checks | 13:49 |
| thorst | can you check for stale marker LU's/ | 13:49 |
| *** dwayne has joined #openstack-powervm | 13:50 | |
| esberglu | Yeah | 13:52 |
| efried | thorst esberglu FYI, I opened a screen session on my pok victim on Friday evening and it's still alive now. So the network couldn't have been TOO bad since then. | 13:53 |
| thorst | k. based on logs I'm betting there are some. I can then recheck them after we clean them out (unless I'm wrong) | 13:53 |
| efried | I want to get this resolved so I can bask in the glow of the TaskFlow change I made. | 13:54 |
| efried | Really chuffed about that. | 13:54 |
| thorst | efried: I want to get this resolved so that we can fix the 10+ merges we have in the backlog | 13:54 |
| efried | Yeah yeah, that too. | 13:54 |
| adreznec | efried: Interesting, all my SSH connections from RCH->POK died over the weekend | 13:54 |
| efried | adreznec SSH connections seem to die when idle for too long. Screen session was continuously posting changes, so stayed alive. | 13:55 |
| efried | Or perhaps rch-pok conn had separate problems. | 13:55 |
| thorst | efried adreznec: Yeah, I think connections within POK are OK | 13:56 |
| thorst | but from outside in weren't | 13:56 |
| efried | I'm connecting from aus | 13:56 |
| adreznec | efried: Yeah, I have tweaks to my SSH config that usually stops connections from dropping | 13:56 |
| efried | adreznec You'll have to share those with me. That's annoying. | 13:56 |
| efried | Seems like it didn't used to do that. Not sure what changed. | 13:56 |
| efried | Never looked into it. | 13:56 |
| efried | thorst Reinstating the stupid IterableToFileAdapter in master... | 13:57 |
| adreznec | efried: It's mostly just setting the ServerAliveInterval/ClientAliveInterval and setting TCPKeepAlive | 13:58 |
| efried | adreznec Yeah, okay, I don't even know where that config file lives. Never used it before. | 13:58 |
| *** smatzek has joined #openstack-powervm | 13:59 | |
| adreznec | Usually /etc/ssh/sshd_config on the server and ~/.ssh/config on the client | 13:59 |
| esberglu | thorst: Stale LUs are cleaned up | 14:04 |
| thorst | running a recheck | 14:04 |
| thorst | esberglu: can you send a note with steps on how we clean up dead jenkins nodes so that we can back you up a bit while we're debugging | 14:05 |
| esberglu | thorst: All I have been doing is logging onto the mgmt node | 14:09 |
| esberglu | And running | 14:09 |
| esberglu | sudo nodepool delete <node number> | 14:09 |
| esberglu | The node number is the number at the end of the instance name | 14:09 |
| esberglu | PowerVM_CI-PowerVM_DevStacked-1290 | 14:10 |
| esberglu | So 1290 | 14:10 |
| thorst | ahh, easy enough | 14:10 |
| *** openstackgerrit has joined #openstack-powervm | 14:13 | |
| openstackgerrit | Eric Fried proposed openstack/nova-powervm master: create_disk_from_image: IO_STREAM instead of FUNC https://review.openstack.org/453233 | 14:13 |
| efried | thorst esberglu ^^ with IterableToFileAdapter. | 14:14 |
| thorst | k | 14:14 |
| thorst | lets see how stable/ocata goes first | 14:14 |
| efried | Shrug, the patch set should kick off its own check. | 14:15 |
| thorst | true | 14:16 |
| *** apearson has joined #openstack-powervm | 14:20 | |
| *** tjakobs__ has quit IRC | 14:40 | |
| thorst | efried esberglu: the jobs are failing again | 14:48 |
| thorst | looks like it failed while running testr... | 14:49 |
| esberglu | By failing do you mean it aborted again? | 14:49 |
| thorst | yeah, look at run 144 | 14:49 |
| esberglu | Looks like the master patch made it all the way through but failed everything | 14:55 |
| esberglu | Not sure what is going on for 144 | 14:55 |
| thorst | I'm watching 145 now... | 14:56 |
| thorst | but meh | 14:56 |
| esberglu | efried: Why don't we need 5109 anymore? | 14:58 |
| efried | esberglu Well, the error we were seeing - the one about file size mismatch in Content-Length header - that check is only done if a) the File object is created with the ExpectedFileSizeInBytes set, and b) the Content-Length header is set in the upload HTTP request. | 14:59 |
| efried | For some reason, when we use IterableToFileAdapter, Content-Length doesn't get set by the requests python module, so it's aaight. | 15:00 |
| efried | But when we use the façade in pypowervm that does substantially the same thing, the requests module sets Content-Length to zero. | 15:00 |
| mdrabe | efried: Is http://morpheus.pok.stglabs.ibm.com:8080/#/c/4975/ gonna fix what you're talking about? | 15:01 |
| efried | I just updated the master branch change set - the one that fixes the hang, where we go from FUNC to IO_STREAM - to use IterableToFileAdapter again. | 15:01 |
| efried | So - if the theory holds - we won't see that error now, because the requests module won't set Content-Length. | 15:01 |
| efried | mdrabe No. | 15:02 |
| efried | mdrabe It ought to fix a bug we have where we don't want to do retries on a partially-consumed stream. | 15:02 |
| efried | But otherwise I don't think it affects anything. | 15:03 |
| efried | We still need to finish that dialog with thorst though. | 15:03 |
| mdrabe | Yea I'd like to get that through, forgot about it for a bit | 15:04 |
| esberglu | efried: We are still seeing that error on patch set 4 of the master changeset | 15:06 |
| esberglu | The newest patchset | 15:07 |
| efried | esberglu The Content-Length mismatch error?? | 15:07 |
| esberglu | nvm | 15:08 |
| esberglu | I was looking at the wrong results | 15:08 |
| efried | phew | 15:08 |
| esberglu | efried: It doesn't look good though | 15:09 |
| esberglu | http://184.172.12.213/33/453233/4/check/nova-powervm-out-of-tree-pvm/5ca001b/ | 15:09 |
| thorst | well, are they all identity exceptions? | 15:09 |
| esberglu | Yeah | 15:09 |
| efried | esberglu Yeah, this doesn't look like problems in our code. | 15:09 |
| thorst | then that looks GREAT | 15:09 |
| efried | This is what we needed - bugs in other projects piled on top. | 15:10 |
| thorst | wait | 15:10 |
| thorst | not all identify | 15:10 |
| thorst | AggregatesAdminTestJSON | 15:10 |
| esberglu | Looks like it can't connect to the mysql db | 15:10 |
| thorst | esberglu: do we capture any database logs? | 15:12 |
| esberglu | No | 15:13 |
| *** tjakobs__ has joined #openstack-powervm | 15:14 | |
| thorst | do we know if other CI's are hitting this? | 15:15 |
| thorst | I suspect this is a root for many things | 15:15 |
| thorst | I'm wondering if we're running out of memory | 15:15 |
| esberglu | I think that this all ties back to the production env. issues | 15:16 |
| esberglu | I kicked off another recheck on the master set | 15:16 |
| esberglu | And will look at the results on staging | 15:16 |
| thorst | esberglu: PM us the IP its running on | 15:17 |
| thorst | we will want to look at two things | 15:17 |
| thorst | 1) Disk spaces | 15:17 |
| thorst | 2) Memory capacity | 15:17 |
| thorst | the dstat-csv gives us insight into memory... | 15:17 |
| thorst | yeah...I think we're out of memory on these things | 15:22 |
| thorst | can we bump them up to 12 GB memory? | 15:22 |
| thorst | maybe at the start of a run...and see if that helps | 15:22 |
| thorst | the 8 GB may not cut it anymore... | 15:22 |
| adreznec | I think last month there were issues in the gate with memory consumption | 15:23 |
| thorst | buff is down to 4 MB at end, cache is down to 166 MB at end, and free is down to 200 MB at end... | 15:23 |
| thorst | that seems...too low to me | 15:24 |
| adreznec | They tried constraining mysql/rabbit memory, but iirc it didn't go well | 15:24 |
| thorst | and disk ops just skyrocket at 9:40 (on this run) | 15:24 |
| thorst | well, we have enough memory now...we just upgraded all nodes | 15:25 |
| *** esberglu has quit IRC | 15:25 | |
| thorst | so 12 or hell, even 16 to see if that fixes it... | 15:25 |
| *** esberglu has joined #openstack-powervm | 15:25 | |
| thorst | crap...I need to run to a call now. efried / esberglu - if you want to hop on a node and try bumping for the next run...it seems worthwhile IMO to try that | 15:26 |
| *** esberglu has quit IRC | 15:27 | |
| *** shyama has quit IRC | 15:28 | |
| efried | I don't know how to do that. adreznec ? | 15:32 |
| thorst | bleh...we can't update the max memory | 15:32 |
| thorst | I just tried it while I'm on hold. | 15:32 |
| thorst | we'll need to update the flavor and rebuild the ready nodes... | 15:32 |
| thorst | I'm hopping into the under cloud to see if I can't update the flavor...then we'll need to clean out some nodes. | 15:33 |
| thorst | horizon so slow.... | 15:33 |
| adreznec | Yeah, you should just be able to edit the flavor directly, then delete all the nodes and they'll get recreated | 15:33 |
| thorst | undercloud is basically out of disk space. | 15:34 |
| adreznec | Like | 15:34 |
| adreznec | the SSP is out of disk space | 15:34 |
| adreznec | Or the controller is out of disk space | 15:34 |
| thorst | controller | 15:34 |
| adreznec | Ok | 15:35 |
| adreznec | Let me ping the guy in control of disk space for that node | 15:35 |
| thorst | I have to jump... adreznec...could you take a peak | 15:35 |
| adreznec | thorst: The controller is out of disk space | 15:35 |
| thorst | I think it just needs cleanup | 15:35 |
| *** thorst is now known as thorst_afk | 15:35 | |
| adreznec | I have like 20 minutes here, I'll look and see what's up | 15:35 |
| thorst_afk | thx dude | 15:36 |
| adreznec | Yup... /dev/mapper/novalink--ci--vg-root 181G 166G 6.4G 97% / | 15:37 |
| *** jpasqualetto has quit IRC | 15:37 | |
| adreznec | Whatever I fix we'll have to make sure to get a cleanup utility for it into the CI deployment | 15:37 |
| thorst_afk | logs didn't seem too gross | 15:37 |
| thorst_afk | so I'm wondering if its glance images | 15:38 |
| thorst_afk | horizon UI still won't boot up for me | 15:38 |
| adreznec | thorst_afk: Yeah, it's all images | 15:53 |
| adreznec | We had 6 15G images on there | 15:54 |
| adreznec | Not sure why nodepool isn't cleaning up the old images... | 15:55 |
| adreznec | Maybe something to do with all the rebuilds | 15:55 |
| *** k0da has quit IRC | 15:59 | |
| *** shyama has joined #openstack-powervm | 15:59 | |
| *** shyama has quit IRC | 16:03 | |
| *** shyama has joined #openstack-powervm | 16:04 | |
| thorst_afk | adreznec: did you update the flavor? | 16:07 |
| thorst_afk | I just updated it | 16:10 |
| adreznec | thorst_afk: Sorry, people stopped in my office | 16:11 |
| adreznec | thorst_afk: did you bump it to 12 or 16? | 16:12 |
| adreznec | nvm, looks like 16 | 16:12 |
| thorst_afk | I also bumped disk | 16:18 |
| thorst_afk | I think I'm going to wipe the nodepool VMs now | 16:18 |
| thorst_afk | so that they rebuild at the new size | 16:18 |
| adreznec | Ok | 16:18 |
| *** jpasqualetto has joined #openstack-powervm | 16:25 | |
| *** jpasqualetto has joined #openstack-powervm | 16:25 | |
| thorst_afk | adreznec: now its complaining that it can't find the flavor | 16:27 |
| adreznec | thorst_afk: Did the flavor id change or something? Not sure how nodepool is actually looking for the flavor | 16:28 |
| thorst_afk | I think nodepool just needs to be restarted | 16:29 |
| *** shyama has quit IRC | 16:29 | |
| thorst_afk | can I just do a nodepool service restart? I seem to remember there was funkiness with that | 16:29 |
| adreznec | thorst_afk: I think so... but not 100% sure. Guess you could always manually stop/start if it fails | 16:30 |
| thorst_afk | adreznec: I think it just needed a restart in order to cache a new flavor | 16:34 |
| adreznec | thorst_afk: Did you actually delete and recreate the flavor? | 16:34 |
| thorst_afk | no, I hit edit... | 16:34 |
| thorst_afk | but if I remember right, you can't edit flavors | 16:34 |
| thorst_afk | they actually create a new one under the covers and soft delete the old | 16:34 |
| adreznec | Hmm yeah, that sounds right | 16:34 |
| thorst_afk | as there are VMs using the old | 16:34 |
| thorst_afk | huh...still stuck in building | 16:54 |
| thorst_afk | efried: I learned something interesting. | 17:03 |
| thorst_afk | during a long upload, the compute service stops responding | 17:04 |
| thorst_afk | at least with the undercloud | 17:04 |
| thorst_afk | though that looks like its coordinated. | 17:05 |
| *** jpasqualetto has quit IRC | 17:15 | |
| *** thorst_afk is now known as thorst | 17:16 | |
| *** shyama has joined #openstack-powervm | 17:22 | |
| *** jpasqualetto has joined #openstack-powervm | 17:28 | |
| *** jpasqualetto has quit IRC | 17:33 | |
| *** jpasqualetto has joined #openstack-powervm | 17:36 | |
| *** jpasqualetto has quit IRC | 17:44 | |
| thorst | efried: new runs actually have VMs...running | 17:51 |
| *** jpasqualetto has joined #openstack-powervm | 17:57 | |
| *** esberglu_ has joined #openstack-powervm | 18:00 | |
| *** esberglu_ has quit IRC | 18:02 | |
| *** esberglu_ has joined #openstack-powervm | 18:02 | |
| esberglu_ | thorst: efried: Have you guys been manually cleaning out the nodepool nodes? | 18:08 |
| esberglu_ | Or are they somehow magically deleting themselves again | 18:08 |
| *** esberglu_ is now known as esberglu | 18:15 | |
| thorst | esberglu: well, I did a mass delete | 18:18 |
| thorst | also it looks like at least ONE went through as a success | 18:18 |
| thorst | adreznec did clear out some images | 18:19 |
| esberglu | Yeah ocata and newton runs both passed | 18:19 |
| thorst | its not clear to me if the runs that just finished cleaned themselves up afterwards | 18:19 |
| thorst | but hey - 150 and 151 went through ok! | 18:19 |
| thorst | progress!!!! | 18:19 |
| esberglu | They did not clean up | 18:19 |
| thorst | esberglu: yeah, looks like we still have the offline issue | 18:19 |
| thorst | yep... | 18:20 |
| thorst | so 'derp' | 18:20 |
| thorst | I'm still of mindset maybe in production we hammer this and add something that calls off to the nodepool and does a nodepool delete | 18:20 |
| thorst | as the final part of the jenkins job | 18:20 |
| esberglu | Aborted connection 2518 to db: 'nodepool' user: 'nodepool' host: 'localhost' (Got an error reading communication packets) | 18:22 |
| esberglu | That's all over the mysql error log | 18:22 |
| esberglu | That might have something to do with it | 18:23 |
| thorst | esberglu: to get the successful runs, we did have to change two things | 18:23 |
| thorst | 1) Increased memory to 16 GB | 18:23 |
| thorst | 2) Increased disk to 30 GB. | 18:23 |
| thorst | so I think you'll want to push a change to neo-os-ci that reflects that | 18:23 |
| esberglu | I was planning on doing 1 anyways since we got the mem upgrades | 18:23 |
| esberglu | Just hadn't gotten to it yet | 18:24 |
| esberglu | I will push that up | 18:24 |
| thorst | k | 18:24 |
| *** shyama has quit IRC | 18:28 | |
| *** edmondsw_ has joined #openstack-powervm | 19:15 | |
| *** edmondsw_ has quit IRC | 19:15 | |
| thorst | efried: you back yet? | 19:16 |
| efried | thorst Sorry, just saw this. Sup? | 19:24 |
| thorst | basically, runs are going through...we'll want to make sure we're primed to merge them | 19:24 |
| thorst | still some hokiness to them | 19:25 |
| efried | thorst "compute service stops responding" smells like that threading issue. | 19:27 |
| thorst | yeah, once we get these patches in...we will want to update the undercloud with your changes | 19:27 |
| efried | thorst So should we merge the ocata & newton changes? | 19:29 |
| thorst | lets wait for master to finish | 19:29 |
| thorst | then bust through all three at once | 19:29 |
| efried | ight | 19:29 |
| thorst | then bust through some ceilometer-powervm ones | 19:30 |
| esberglu | master just passed | 19:51 |
| esberglu | efried: thorst: ^ | 19:52 |
| thorst | yay! | 19:52 |
| efried | Merge, quick! | 19:52 |
| thorst | efried: I already +2'd | 19:52 |
| thorst | you want the W+1 glory/ | 19:52 |
| efried | done | 19:53 |
| thorst | after that, I'll hit up the ceilometer-powervm rechecks | 19:53 |
| efried | thorst I'm going to need a consult on some of mriedem's comments on #2. | 19:55 |
| thorst | link? | 19:56 |
| efried | https://review.openstack.org/#/c/438598/ | 19:57 |
| efried | Great, a merge conflict too. | 19:57 |
| efried | crap! on #1 too! | 19:57 |
| efried | stupid cursive | 19:58 |
| openstackgerrit | Merged openstack/nova-powervm master: create_disk_from_image: IO_STREAM instead of FUNC https://review.openstack.org/453233 | 19:58 |
| *** jpasqualetto has quit IRC | 20:03 | |
| thorst | efried: put some comments in | 20:08 |
| efried | thorst thx | 20:09 |
| thorst | he totally gets all the frustrations I had | 20:09 |
| thorst | xag - wtf is that | 20:09 |
| thorst | adapter.helpers - totally not clear that it returns a copy | 20:09 |
| thorst | efried: you have a chance to help Jay? | 20:17 |
| thorst | *do you have time to help him | 20:17 |
| thorst | with devstack issues | 20:17 |
| efried | thorst I can try. Kind of the blind leading the blind. | 20:18 |
| efried | What does he need help with? | 20:18 |
| thorst | let me PM you the error...its large :-) | 20:18 |
| thorst | esberglu: I just kicked off a bunch of rechecks | 20:19 |
| *** k0da has joined #openstack-powervm | 20:19 | |
| esberglu | throst: try again, none of them got picked up.... | 20:24 |
| esberglu | zuul died | 20:24 |
| esberglu | thorst: ^ | 20:25 |
| thorst | why did zuul die? I see you added a recheck comment already to at least one | 20:26 |
| esberglu | idk | 20:26 |
| esberglu | The production management seems to be hosed | 20:26 |
| thorst | disk space? I wonder if we should just scrap it and let it rebuild from scratch... | 20:27 |
| esberglu | thorst: I wouldn't be against that at this point | 20:27 |
| thorst | lets get this recheck through and then maybe that's what we bite off tomorrow | 20:27 |
| esberglu | Yeah | 20:27 |
| esberglu | It has plenty of disk | 20:28 |
| thorst | https://review.openstack.org/#/c/448134/ --> Does that need a recheck again? | 20:28 |
| esberglu | 160G available | 20:28 |
| esberglu | Nope that went through on my recheck | 20:29 |
| esberglu | That's the only one going | 20:29 |
| esberglu | So if there are other ones you kicked off, they will need another recheck comment | 20:29 |
| thorst | k. I just did 3 more | 20:30 |
| esberglu | Yep I see them | 20:30 |
| thorst | k | 20:31 |
| *** smatzek has quit IRC | 20:33 | |
| esberglu | thorst: 448134 failed because it couldn't resolve github.com | 21:00 |
| esberglu | Kicked off another one | 21:00 |
| *** thorst has quit IRC | 21:03 | |
| *** edmondsw has quit IRC | 21:08 | |
| *** edmondsw has joined #openstack-powervm | 21:10 | |
| *** edmondsw has quit IRC | 21:15 | |
| *** jay1_ has joined #openstack-powervm | 21:22 | |
| jay1_ | efied: I have been getting the error "Error [Errno 2] No such file or directory: '/opt/stack/pypowervm/' while executing command python setup.py" while stacking with Install_Pypowervm=false | 21:23 |
| jay1_ | any idea to by pass this ? | 21:24 |
| efried | jay1_ I'm working on it. | 21:31 |
| efried | Something is causing pvm-novalink stuff to get uninstalled during stacking. | 21:31 |
| efried | That's gonna break the world. | 21:31 |
| efried | adreznec You around to help diagnose deb pkg dependency boggle? | 21:31 |
| efried | oh, but wait, stacking actually succeeded this time around. | 21:32 |
| jay1_ | efried: sure.. | 21:32 |
| efried | None of the novalink services started properly because pvm-rest-app ain't there. | 21:33 |
| efried | Ima see if they'll start after I install that. | 21:33 |
| jay1_ | ok.. | 21:33 |
| efried | But mebbe not cause not sure if the pvm_admin group is gonna show up right. | 21:33 |
| jay1_ | user neo is already member of pvm_admin now. | 21:36 |
| efried | I was able to start the compute and networking services. But several of the services are showing RPC timeouts - something still wrong with rabbitmq I think. | 21:37 |
| jay1_ | I can see the current status of rabbitmq-server as Active | 21:39 |
| efried | I suspect there's a dep conflict between arping and iputils-arping. The former seems to be a pkg req of pvm-novalink; the latter of... something stacky. | 21:39 |
| *** smatzek has joined #openstack-powervm | 22:02 | |
| *** smatzek has quit IRC | 22:06 | |
| jay1_ | efried: pls let me know once the issue is fixed. | 22:08 |
| efried | jay1_ It's not going to be a simple fix. | 22:08 |
| *** esberglu has quit IRC | 22:09 | |
| jay1_ | ah.. | 22:09 |
| *** esberglu has joined #openstack-powervm | 22:09 | |
| *** esberglu has quit IRC | 22:13 | |
| *** esberglu has joined #openstack-powervm | 22:22 | |
| *** esberglu has quit IRC | 22:26 | |
| *** apearson has quit IRC | 22:29 | |
| *** jay1_ has quit IRC | 22:32 | |
| *** k0da has quit IRC | 22:42 | |
| *** tjakobs__ has quit IRC | 22:52 | |
| *** edmondsw has joined #openstack-powervm | 22:53 | |
| *** edmondsw has quit IRC | 22:57 | |
| *** thorst has joined #openstack-powervm | 23:02 | |
| openstackgerrit | Merged openstack/networking-powervm master: Remove INSTALL_PYPOWERVM https://review.openstack.org/454343 | 23:09 |
| openstackgerrit | Merged openstack/nova-powervm master: FileIO adapter does not remove mappings on detach https://review.openstack.org/455269 | 23:10 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!