| *** thorst_ has quit IRC | 00:14 | |
| *** thorst_ has joined #openstack-powervm | 00:15 | |
| *** thorst_ has quit IRC | 00:19 | |
| *** thorst_ has joined #openstack-powervm | 00:34 | |
| *** thorst_ has quit IRC | 00:34 | |
| *** thorst_ has joined #openstack-powervm | 00:35 | |
| *** thorst_ has quit IRC | 00:43 | |
| *** esberglu has quit IRC | 00:50 | |
| *** k0da has joined #openstack-powervm | 01:05 | |
| *** k0da has quit IRC | 01:13 | |
| *** thorst_ has joined #openstack-powervm | 01:37 | |
| *** esberglu has joined #openstack-powervm | 01:47 | |
| *** esberglu has quit IRC | 01:59 | |
| *** tblakes has joined #openstack-powervm | 02:20 | |
| *** tblakes has quit IRC | 02:22 | |
| *** esberglu has joined #openstack-powervm | 02:24 | |
| *** thorst_ has quit IRC | 02:25 | |
| *** thorst_ has joined #openstack-powervm | 02:25 | |
| *** esberglu has quit IRC | 02:29 | |
| *** apearson has joined #openstack-powervm | 02:32 | |
| *** thorst_ has quit IRC | 02:34 | |
| *** smatzek has joined #openstack-powervm | 02:40 | |
| *** esberglu has joined #openstack-powervm | 02:45 | |
| *** esberglu has quit IRC | 02:46 | |
| *** apearson has quit IRC | 02:49 | |
| *** smatzek has quit IRC | 02:52 | |
| *** thorst_ has joined #openstack-powervm | 03:04 | |
| *** thorst_ has quit IRC | 03:12 | |
| *** thorst_ has joined #openstack-powervm | 03:29 | |
| *** thorst_ has quit IRC | 03:30 | |
| *** esberglu has joined #openstack-powervm | 03:53 | |
| *** esberglu has quit IRC | 03:54 | |
| *** thorst_ has joined #openstack-powervm | 04:31 | |
| *** thorst_ has quit IRC | 04:39 | |
| *** thorst_ has joined #openstack-powervm | 05:37 | |
| *** esberglu has joined #openstack-powervm | 05:41 | |
| *** thorst_ has quit IRC | 05:45 | |
| *** esberglu has quit IRC | 05:45 | |
| *** thorst_ has joined #openstack-powervm | 06:42 | |
| *** thorst_ has quit IRC | 06:50 | |
| *** k0da has joined #openstack-powervm | 06:53 | |
| *** svenkat has joined #openstack-powervm | 07:21 | |
| *** svenkat has quit IRC | 07:26 | |
| *** esberglu has joined #openstack-powervm | 07:34 | |
| *** esberglu has quit IRC | 07:39 | |
| *** thorst_ has joined #openstack-powervm | 07:47 | |
| *** thorst_ has quit IRC | 07:55 | |
| *** svenkat has joined #openstack-powervm | 08:26 | |
| *** svenkat has quit IRC | 08:30 | |
| *** k0da has quit IRC | 08:52 | |
| *** thorst_ has joined #openstack-powervm | 08:54 | |
| *** thorst_ has quit IRC | 08:59 | |
| *** svenkat has joined #openstack-powervm | 09:13 | |
| *** svenkat has quit IRC | 09:18 | |
| *** esberglu has joined #openstack-powervm | 09:26 | |
| *** esberglu has quit IRC | 09:30 | |
| *** adi___ has quit IRC | 09:37 | |
| *** adi___ has joined #openstack-powervm | 09:37 | |
| *** thorst_ has joined #openstack-powervm | 09:57 | |
| *** thorst_ has quit IRC | 10:05 | |
| *** svenkat has joined #openstack-powervm | 10:17 | |
| *** svenkat has quit IRC | 10:22 | |
| *** svenkat has joined #openstack-powervm | 11:02 | |
| *** thorst_ has joined #openstack-powervm | 11:03 | |
| *** svenkat has quit IRC | 11:06 | |
| *** thorst_ has quit IRC | 11:09 | |
| *** esberglu has joined #openstack-powervm | 12:04 | |
| *** svenkat has joined #openstack-powervm | 12:06 | |
| *** esberglu has quit IRC | 12:08 | |
| *** svenkat has quit IRC | 12:11 | |
| *** seroyer has joined #openstack-powervm | 12:14 | |
| *** svenkat has joined #openstack-powervm | 12:47 | |
| *** thorst_ has joined #openstack-powervm | 12:50 | |
| *** seroyer has quit IRC | 13:02 | |
| *** svenkat has quit IRC | 13:13 | |
| *** adreznec has quit IRC | 13:17 | |
| *** adreznec has joined #openstack-powervm | 13:20 | |
| *** kylek3h has joined #openstack-powervm | 13:24 | |
| *** kylek3h has quit IRC | 13:28 | |
| *** tblakes has joined #openstack-powervm | 13:33 | |
| *** seroyer has joined #openstack-powervm | 13:33 | |
| *** jwcroppe has quit IRC | 13:35 | |
| *** jwcroppe has joined #openstack-powervm | 13:36 | |
| *** seroyer has quit IRC | 13:38 | |
| *** jwcroppe has quit IRC | 13:40 | |
| *** edmondsw has joined #openstack-powervm | 13:47 | |
| *** openstackgerrit has quit IRC | 13:48 | |
| *** openstackgerrit has joined #openstack-powervm | 13:48 | |
| *** jwcroppe has joined #openstack-powervm | 13:57 | |
| *** esberglu has joined #openstack-powervm | 13:57 | |
| *** esberglu has quit IRC | 13:58 | |
| *** esberglu has joined #openstack-powervm | 13:58 | |
| *** svenkat has joined #openstack-powervm | 14:00 | |
| *** seroyer has joined #openstack-powervm | 14:05 | |
| *** seroyer has quit IRC | 14:14 | |
| *** mdrabe has joined #openstack-powervm | 14:14 | |
| *** esberglu has quit IRC | 14:20 | |
| *** smatzek has joined #openstack-powervm | 14:26 | |
| *** esberglu has joined #openstack-powervm | 14:33 | |
| *** esberglu has quit IRC | 14:33 | |
| *** esberglu has joined #openstack-powervm | 14:33 | |
| *** smatzek has quit IRC | 14:36 | |
| esberglu | thorst: efried: Logs from one of the runs with the pypowervm patch | 14:40 |
|---|---|---|
| esberglu | http://184.172.12.213/79/370679/3/silent/nova-pvm-dsvm-tempest-full/9f09a37/ | 14:40 |
| thorst_ | hmm | 14:40 |
| thorst_ | 49 failures | 14:40 |
| thorst_ | that seems bad :-) | 14:40 |
| esberglu | Bunch of errors in n-cpu.log | 14:40 |
| esberglu | Mostly | 14:41 |
| esberglu | [01;31m2016-11-07 19:39:32.758 TRACE pypowervm.tasks.cluster_ssp [01;35m[00mError: Unexpected error: AttributeError for PUT https://172.16.0.1:12443/rest/api/web/File/contents/9b19efc2-32fa-4d31-8b24-129750d6d8db: 'function' object has no attribute 'read' | 14:41 |
| thorst_ | I blame efried | 14:41 |
| *** svenkat has quit IRC | 14:42 | |
| *** kylek3h has joined #openstack-powervm | 14:43 | |
| efried | esberglu, I will *share* the blame with thorst_. | 14:45 |
| efried | thorst_, you fixing or should I? | 14:46 |
| *** seroyer has joined #openstack-powervm | 14:49 | |
| thorst_ | efried: is it really as simple as we just missed the read method in the RESTAPIPipe? | 14:52 |
| efried | thorst_, no, I don't think so. | 14:52 |
| efried | The context manager is returning the stream, which oughtta have a read method on it. The read method should therefore be unnecessary. | 14:53 |
| efried | oh | 14:53 |
| efried | Hold on, let me try something. | 14:53 |
| efried | thorst_, no, it's right. | 14:56 |
| efried | the context manager bit, that is. | 14:56 |
| efried | thorst_, take a look at line 419 - that's returning a thing that oughtta have a .read method, right? | 14:57 |
| *** smatzek has joined #openstack-powervm | 14:57 | |
| efried | thorst_, let's make sure the logic is straight. We want to call the RESTAPIPipe path if we're *not* local and if upload_type is FUNC - right? | 14:58 |
| thorst_ | no... | 14:59 |
| thorst_ | well, let me look | 14:59 |
| thorst_ | yeah, it should... | 15:00 |
| efried | thorst_, I don't think we're hitting the RESTAPIPipe path here. | 15:02 |
| thorst_ | so I didn't initially either | 15:03 |
| thorst_ | which is why I asked that. | 15:03 |
| thorst_ | but, Ido now. The chunkreader is when you upload to the REST API | 15:03 |
| *** apearson has joined #openstack-powervm | 15:03 | |
| efried | But I think that's trying to happen through the regular DISK_IMAGE path. | 15:04 |
| efried | _copy_func is not in the stack. | 15:05 |
| thorst_ | huh...what the heck sets filehandle in adapter.py | 15:07 |
| thorst_ | yeah, and I realize that...but its because its a different thread. | 15:07 |
| efried | _copy_func is nowhere in the log | 15:08 |
| efried | If the problem was the RESTAPIPipe context manager not returning a read()able handle, there would be a stack trace that included _copy_func. | 15:08 |
| efried | So that ain't it. | 15:08 |
| efried | Is this running against latest master of nova-powervm? | 15:09 |
| efried | seems like it. | 15:10 |
| efried | So | 15:10 |
| thorst_ | the stack is clearly incomplete... | 15:11 |
| efried | get_or_upload_image_lu is passing FUNC to upload_lu, and the io_stream is the 'upload' method that does the glance API image download deal. | 15:11 |
| thorst_ | you don't just start at request... | 15:11 |
| efried | nah, see 2016-11-07 19:39:34.619 | 15:11 |
| efried | that's the complete stack. | 15:11 |
| thorst_ | brilliant | 15:11 |
| efried | not sure why it's truncated above. | 15:11 |
| thorst_ | probably something catches it and does a LOG.exception(e), raise | 15:12 |
| efried | mm | 15:12 |
| efried | So upload_lu is getting DISK_IMAGE, hopefully, because not local. | 15:13 |
| thorst_ | well, we landed in _upload_stream | 15:14 |
| thorst_ | so that's good. | 15:14 |
| efried | But because DISK_IMAGE - as opposed to DISK_IMAGE_COORDINATED - it winds up in _upload_stream_api | 15:14 |
| thorst_ | wait a tick...nothing in there points to upload coordinated | 15:15 |
| efried | which is NOT where we have this fancy new stuff. | 15:15 |
| thorst_ | yeah...yeah... | 15:15 |
| efried | Yeah, coordinated gets set in the File create only if local. | 15:15 |
| thorst_ | ahh snap | 15:15 |
| efried | So we _could_ fix by again special-casing that guy. But I think a refactor may be in order here. | 15:15 |
| thorst_ | but this is remote. | 15:16 |
| efried | yeah, so it sets not-coordinated | 15:16 |
| efried | which triggers | 15:16 |
| efried | if vio_file.enum_type == vf.FileType.DISK_IMAGE_COORDINATED: | 15:16 |
| efried | _upload_stream_coordinated(vio_file, io_handle, upload_type) <====not this | 15:16 |
| efried | else: | 15:16 |
| efried | # Upload the file directly to the REST API server. | 15:16 |
| efried | _upload_stream_api(vio_file, io_handle) <====but this | 15:16 |
| thorst_ | so we need to get that set to coordinated. | 15:17 |
| thorst_ | basically that's it | 15:17 |
| efried | Yeeeaaahhh... | 15:17 |
| efried | We can do that real quick to see if it works, but I'd like to reorganize this code so it's less pastariffic. | 15:18 |
| efried | thorst_, in light of RESTAPIPipe, do we ever need _upload_stream_api? | 15:20 |
| thorst_ | sorry...on call arguing | 15:24 |
| efried | putting up a quick-fix patch set. | 15:26 |
| efried | thorst_, esberglu: it's up. Try r out. | 15:28 |
| efried | (incidentally, this change set hangs jenkins, so abort builds if you put up patch sets) | 15:29 |
| efried | (until we fix UT) | 15:29 |
| thorst_ | nice fix | 15:38 |
| *** mdrabe has quit IRC | 15:38 | |
| *** mdrabe has joined #openstack-powervm | 15:46 | |
| *** tjakobs has joined #openstack-powervm | 15:50 | |
| *** svenkat has joined #openstack-powervm | 16:18 | |
| *** chas_ has joined #openstack-powervm | 16:32 | |
| thorst_ | efried esberglu: finally out of calls...so that new fix is getting validated? | 16:56 |
| esberglu | Yeah. Base image just finished building, nodes are spawning off of that | 16:56 |
| thorst_ | rockin | 16:57 |
| *** efried has quit IRC | 17:20 | |
| *** chas_ has quit IRC | 17:24 | |
| *** chas_ has joined #openstack-powervm | 17:36 | |
| *** chas_ has quit IRC | 17:40 | |
| thorst_ | adreznec: are we having the meeting later tonight, or just transitioning to new IRC scrum style? | 17:58 |
| *** k0da has joined #openstack-powervm | 18:14 | |
| *** k0da has quit IRC | 18:52 | |
| esberglu | efried: thorst_: A bunch of the runs are failing to stack because of read-only filesystems. About to check out the nodes and see if they are full or something | 19:12 |
| esberglu | Other ones stack, and then loop on this forever | 19:12 |
| esberglu | 13:05:16 Looking for an existing Image LU with checksum '2f282b84e7e608d5852449ed940bfc51' | 19:12 |
| esberglu | 13:05:22 No matching Image LU found. | 19:12 |
| esberglu | Until the agent goes offline | 19:12 |
| esberglu | A bunch of other ones are in progress, didn't hit either of the issues above | 19:13 |
| *** k0da has joined #openstack-powervm | 19:23 | |
| thorst_ | esberglu: did the SSP go tails up? | 19:36 |
| esberglu | thorst_: Yeah looks like it | 19:53 |
| thorst_ | fantastic... | 19:54 |
| thorst_ | yeah, I'm hearing that there are network 'blips' in POK | 19:54 |
| thorst_ | esberglu: is it better at all? | 20:17 |
| openstackgerrit | Drew Thorstensen (thorst) proposed openstack/nova-powervm: Fix rollback op's search call https://review.openstack.org/393801 | 20:17 |
| *** efried has joined #openstack-powervm | 20:18 | |
| esberglu | thorst_: I think I'm gonna have to restart the VIOSes. Last time I was seeing similar errors in the FFDC log I was told I had to either reboot or restart the vio_daemon | 20:25 |
| thorst_ | restart the vio_daemon | 20:32 |
| thorst_ | way easier than restarting the VIOSes | 20:32 |
| esberglu | Command for that? | 20:34 |
| efried | oem_setup_env | 20:37 |
| efried | stopsrc -s vio_daemon | 20:37 |
| efried | startsrc -s vio_daemon | 20:37 |
| efried | esberglu ^^ | 20:37 |
| esberglu | thanks | 20:37 |
| esberglu | efried: A bunch of the long runs are also now spinning looking for existing lu with checksum 'xxx' | 20:43 |
| efried | esberglu, need to see that. | 20:44 |
| efried | It shouldn't take a long time to look for that LU unless the VIOS is hanging. | 20:44 |
| esberglu | Yeah its probably just related to my other vios problems then | 20:45 |
| efried | yuh | 20:45 |
| thorst_ | but if its neo6, everything responded very quick... | 20:48 |
| esberglu | Its everywhere. Neo 6, 19, 21 ,38, 39 plus more probably | 20:53 |
| thorst_ | ugh... | 20:54 |
| thorst_ | if this keeps up after the restarts...lets see about a SSP reconfigure. | 20:54 |
| thorst_ | bleh | 20:54 |
| esberglu | Looks like there are errors about releasing a lock in the FFDC | 20:55 |
| *** svenkat has quit IRC | 21:16 | |
| *** smatzek has quit IRC | 21:29 | |
| *** chas_ has joined #openstack-powervm | 21:32 | |
| *** apearson has quit IRC | 22:29 | |
| *** thorst_ has quit IRC | 22:32 | |
| *** apearson has joined #openstack-powervm | 22:32 | |
| *** tblakes has quit IRC | 22:33 | |
| *** edmondsw has quit IRC | 22:38 | |
| *** thorst_ has joined #openstack-powervm | 22:55 | |
| *** esberglu has quit IRC | 22:56 | |
| *** esberglu has joined #openstack-powervm | 22:58 | |
| *** efried has quit IRC | 23:01 | |
| *** esberglu has quit IRC | 23:03 | |
| *** apearson has quit IRC | 23:04 | |
| *** mdrabe has quit IRC | 23:04 | |
| *** thorst_ has quit IRC | 23:07 | |
| *** esberglu has joined #openstack-powervm | 23:11 | |
| *** k0da has quit IRC | 23:17 | |
| *** tjakobs has quit IRC | 23:17 | |
| *** jwcroppe has quit IRC | 23:27 | |
| *** jwcroppe has joined #openstack-powervm | 23:27 | |
| *** jwcroppe has quit IRC | 23:31 | |
| *** seroyer has quit IRC | 23:32 | |
| *** chas_ has quit IRC | 23:33 | |
| *** chas_ has joined #openstack-powervm | 23:38 | |
| *** chas_ has quit IRC | 23:43 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!