| *** tonyb has quit IRC | 00:33 | |
| *** tonyb has joined #openstack-powervm | 00:35 | |
| *** esberglu has joined #openstack-powervm | 01:53 | |
| *** esberglu has quit IRC | 03:01 | |
| *** edmondsw has joined #openstack-powervm | 04:43 | |
| *** edmondsw_ has joined #openstack-powervm | 04:44 | |
| *** edmondsw has quit IRC | 04:48 | |
| *** edmondsw_ has quit IRC | 04:59 | |
| *** edmondsw has joined #openstack-powervm | 05:41 | |
| *** edmondsw has quit IRC | 05:45 | |
| *** edmondsw has joined #openstack-powervm | 06:02 | |
| *** edmondsw has quit IRC | 06:06 | |
| *** edmondsw has joined #openstack-powervm | 07:24 | |
| *** edmondsw has quit IRC | 07:28 | |
| *** edmondsw has joined #openstack-powervm | 07:45 | |
| *** edmondsw has quit IRC | 07:49 | |
| *** AlexeyAbashkin has joined #openstack-powervm | 08:05 | |
| *** edmondsw has joined #openstack-powervm | 08:05 | |
| *** edmondsw has quit IRC | 08:09 | |
| *** edmondsw has joined #openstack-powervm | 08:26 | |
| *** edmondsw has quit IRC | 08:30 | |
| *** adi_____ has quit IRC | 12:15 | |
| *** adi_____ has joined #openstack-powervm | 12:15 | |
| *** efried has joined #openstack-powervm | 12:16 | |
| *** edmondsw has joined #openstack-powervm | 12:28 | |
| *** edmondsw_ has joined #openstack-powervm | 12:29 | |
| *** edmondsw has quit IRC | 12:33 | |
| *** edmondsw_ has quit IRC | 12:44 | |
| *** edmondsw has joined #openstack-powervm | 12:48 | |
| *** edmondsw has quit IRC | 12:52 | |
| *** edmondsw has joined #openstack-powervm | 12:59 | |
| *** edmondsw has quit IRC | 13:01 | |
| *** edmondsw has joined #openstack-powervm | 13:27 | |
| *** edmondsw has quit IRC | 13:31 | |
| *** esberglu has joined #openstack-powervm | 13:40 | |
| *** edmondsw has joined #openstack-powervm | 13:59 | |
| esberglu | efried: You around? | 14:00 |
|---|---|---|
| efried | esberglu: yup. | 14:00 |
| esberglu | efried: I was wrong yesterday. It isn't just an issue with our mocks bmd.is_volume doesn't work live either | 14:01 |
| esberglu | *bdm.is_volume | 14:01 |
| efried | oo | 14:02 |
| esberglu | Pops an attribute error "AttributeError: 'dict' object has no attribute 'is_volume'" | 14:02 |
| efried | esberglu: Yeah, it looked to me like DriverVolumeBlockDevice takes a BlockDeviceMapping and strips away all the special-ness. | 14:03 |
| efried | esberglu: And I can't figure out why we would do that. | 14:03 |
| esberglu | efried: But the actual bdm being passed into that method doesn't have that special-ness either | 14:04 |
| *** edmondsw has quit IRC | 14:04 | |
| esberglu | From what I can tell | 14:04 |
| efried | right, swhat I'm saying. So either we need to re-bdm-ify the thing so we can ask is_volume, or we need to duplicate the logic, which is by asking if that particular field is set to "volume". | 14:04 |
| *** edmondsw has joined #openstack-powervm | 14:04 | |
| efried | You should bring this question to mriedem. | 14:05 |
| *** edmondsw has quit IRC | 14:09 | |
| *** edmondsw has joined #openstack-powervm | 14:09 | |
| *** tjakobs__ has joined #openstack-powervm | 14:13 | |
| *** edmondsw has quit IRC | 14:14 | |
| *** AlexeyAbashkin has quit IRC | 14:25 | |
| *** edmondsw has joined #openstack-powervm | 14:35 | |
| *** edmondsw has quit IRC | 14:40 | |
| *** edmondsw has joined #openstack-powervm | 14:42 | |
| *** edmondsw has quit IRC | 14:46 | |
| *** esberglu has quit IRC | 15:04 | |
| *** esberglu has joined #openstack-powervm | 15:05 | |
| *** AlexeyAbashkin has joined #openstack-powervm | 15:08 | |
| esberglu | efried: git restack question | 15:19 |
| esberglu | I checked out the localdisk IT patch and ran git restack | 15:19 |
| esberglu | And have done all of the edits on the vSCSI change | 15:19 |
| esberglu | And git added all of them | 15:20 |
| esberglu | Then what do I do? | 15:20 |
| esberglu | git rebase --continue says that I have uncommitted changes in my working tree | 15:21 |
| esberglu | git commit --amend takes me into the localdisk commit message | 15:21 |
| efried | It sounds like you weren't actually on the VSCSI change when you did your edits. | 15:21 |
| efried | git restack brings you into an editor. You change `pick` to `edit` for the commit you want to fix up. Did you do that for the VSCSI commit? | 15:21 |
| esberglu | efried: Yes | 15:22 |
| efried | Then git commit --amend should have put you in the VSCSI commit message. | 15:22 |
| efried | Something got messed up in there. | 15:22 |
| efried | How hard to redo the vscsi changes? | 15:22 |
| esberglu | efried: I'll just copy all the files I thouched | 15:23 |
| efried | You can also save your branch (rename it to something to move it out of the way) and then re-download and re-restack. | 15:23 |
| efried | Then you've got the branch with your changes and you can diff/patch, which may be easier. | 15:24 |
| esberglu | git stash to the rescue | 15:29 |
| efried | beuat | 15:32 |
| efried | beaut | 15:32 |
| efried | dude, that's like the fourth time this morning I've put my 'u' in the wrong place. | 15:32 |
| efried | I wonder if I'm having a stroke. | 15:32 |
| *** edmondsw has joined #openstack-powervm | 15:33 | |
| esberglu | efried: You remember why we couldn't test vSCSI in CI? | 15:37 |
| esberglu | ^ or edmondsw if you're there | 15:37 |
| esberglu | Something to do with remote? | 15:39 |
| efried | I don't think I ever understood that, no, sorry. | 15:39 |
| esberglu | efried: New vSCSI commit is up | 15:42 |
| efried | ack | 15:43 |
| efried | esberglu: snapshot was a manual rebase? | 15:43 |
| edmondsw | esberglu efried I'm sitting next to thorst, I'll get him to sign in and explain | 15:43 |
| *** thorst has joined #openstack-powervm | 15:43 | |
| esberglu | efried: 1 line was conflicting... | 15:44 |
| efried | lovely. | 15:44 |
| thorst | we don't have a SSP cinder driver. We have a storwize one, but we don't have a good cleanup mechanism for failed CI jobs and our v7k would eventually die (and also they can't handle 100 servers SSH'ing to it at the same time) | 15:44 |
| thorst | I think that's the answer to the question I can't see :-) | 15:44 |
| edmondsw | thorst question was "why we can't test vSCSI in CI" | 15:45 |
| thorst | yeah, it's basically we need a cinder driver issue | 15:45 |
| esberglu | thorst: Yep thank you | 15:45 |
| thorst | I think that there are some opportunities on the roadmap, but may be a few months out | 15:45 |
| edmondsw | thorst how does SSP relate to this? | 15:49 |
| edmondsw | I thought it was more to do with how our CI is structured | 15:51 |
| thorst | vscsi can connect to either SSP or a FC backend (and more - like GPFS and what not). So you just need a cinder backend that is reasonable for the CI to test vSCSI | 16:00 |
| *** esberglu has quit IRC | 16:05 | |
| *** esberglu has joined #openstack-powervm | 16:05 | |
| *** esberglu has quit IRC | 16:10 | |
| *** thorst has quit IRC | 16:30 | |
| *** AlexeyAbashkin has quit IRC | 17:01 | |
| *** edmondsw has quit IRC | 17:04 | |
| *** edmondsw has joined #openstack-powervm | 17:56 | |
| *** edmondsw_ has joined #openstack-powervm | 18:03 | |
| *** edmonds__ has joined #openstack-powervm | 18:05 | |
| *** edmonds__ is now known as edmondsw__ | 18:06 | |
| *** edmondsw__ is now known as edmondsw___ | 18:06 | |
| *** edmondsw has quit IRC | 18:06 | |
| *** edmondsw___ is now known as edmondsw | 18:06 | |
| *** edmondsw_ has quit IRC | 18:09 | |
| *** esberglu has joined #openstack-powervm | 18:17 | |
| *** esberglu has quit IRC | 18:17 | |
| *** esberglu has joined #openstack-powervm | 18:18 | |
| efried | esberglu: Have we reproduced in tree yet? | 18:45 |
| efried | esberglu: Are you running in tree with the pypowervm patch? | 18:45 |
| esberglu | efried: Yeah right now both IT and OOT CI are running with patched pypowervm | 18:46 |
| esberglu | efried: Not reproduced IT yet | 18:46 |
| esberglu | Wait do you mean reproduced as in have we hit the issue IT? If so then yes | 18:47 |
| esberglu | We haven't pushed a nova patch yet | 18:47 |
| efried | right, that's what I was asking. | 18:47 |
| efried | Okay, so definitely affects nova too. | 18:47 |
| esberglu | Yes | 18:48 |
| esberglu | efried: edmondsw: I'll be taking down production CI later this week to upgrade the undercloud to queens | 19:02 |
| esberglu | Probably friday morning | 19:02 |
| esberglu | Shouldn't be any longer than a normal redeploy assuming no roadblocks | 19:03 |
| edmondsw | esberglu ack | 19:07 |
| *** esberglu has quit IRC | 19:11 | |
| *** esberglu has joined #openstack-powervm | 19:14 | |
| efried | edmondsw: I meant to talk to you about something else here. | 19:17 |
| efried | edmondsw: We're no longer constrained by the requirements project/bot. Does that apply to queens as well? | 19:17 |
| efried | edmondsw: If so, we don't need that monkey-patch that Gautam put up this morning. | 19:17 |
| edmondsw | efried what monkey patch? | 19:18 |
| edmondsw | I thought the requirements changes were only for rocky and forward | 19:18 |
| edmondsw | so probably moot | 19:19 |
| efried | okay. | 19:22 |
| efried | edmondsw, esberglu: 6505 is back up for review. | 19:34 |
| esberglu | ack | 19:34 |
| efried | bbiab | 19:34 |
| *** edmondsw_ has joined #openstack-powervm | 21:11 | |
| *** edmondsw has quit IRC | 21:13 | |
| esberglu | efried: Just got around to verifying the is_volume change | 21:32 |
| esberglu | https://review.openstack.org/#/c/526094/46/nova/virt/powervm/driver.py@545 | 21:32 |
| efried | well shit | 21:33 |
| efried | esberglu: I think I know why. | 21:33 |
| efried | esberglu: I think you need to skip https://review.openstack.org/#/c/526094/46/nova/virt/powervm/driver.py@213 and pass block_device_info into _vol_drv_iter. | 21:34 |
| efried | esberglu: Hum, maybe not. | 21:37 |
| esberglu | efried: Why? I'm not following. block_device_info contains multiple block_device_mappings, and is_volume could be different between them | 21:37 |
| efried | esberglu: Yeah, we need to figure out wtf block_device_info is, once and for all. | 21:39 |
| efried | esberglu: Got a way to pdb into this sucker? | 21:39 |
| esberglu | Can you pdb into a running openstack? | 21:39 |
| efried | esberglu: Yes, as long as you're running the relevant process in the foreground. | 21:40 |
| efried | esberglu: Or you could use remotepdb | 21:40 |
| efried | which would probably be better considering n-cpu is threaded. | 21:40 |
| efried | import remotepdb; remotepdb.set_trace() | 21:41 |
| efried | Watch the logs for a message that tells you it hit. The message will give you a port number. Telnet to that port number on localhost and you'll be attached to the pdb session at that point in that thread. | 21:41 |
| efried | You'll probably have to install remotepdb | 21:41 |
| efried | pip install remote-pdb | 21:42 |
| edmondsw_ | oh wow, there is an equivalent to rpdb that handles connections properly? awesome | 21:47 |
| edmondsw_ | looks like it's remote_pdb though (missing underscore) | 21:50 |
| efried | oh, my bad | 21:53 |
| *** edmondsw_ has quit IRC | 21:55 | |
| esberglu | efried: Looks like block_device_info is just a standard dict | 21:56 |
| esberglu | http://paste.openstack.org/show/719872/ | 21:56 |
| efried | esberglu: Well wtf. | 21:57 |
| efried | esberglu: so you're sitting there right before you would encounter the error, yes? | 21:58 |
| esberglu | efried: Yeah | 21:58 |
| efried | esberglu: Okay, stand by, let me noodle this. | 21:58 |
| esberglu | efried: Ok. I'm relocating, should be back online in 15 or so | 22:00 |
| efried | k | 22:00 |
| *** esberglu has quit IRC | 22:04 | |
| *** edmondsw has joined #openstack-powervm | 22:05 | |
| *** edmondsw has quit IRC | 22:05 | |
| *** tjakobs__ has quit IRC | 22:09 | |
| *** esberglu has joined #openstack-powervm | 22:17 | |
| efried | esberglu: | 22:17 |
| efried | from nova.objects import block_device as bdmobj | 22:17 |
| efried | step into _vol_drv_iter, into the loop over bdms; | 22:17 |
| efried | Run: | 22:17 |
| efried | bdm = bdmobj.BlockDeviceMapping(**bdm) | 22:17 |
| efried | ...and then see if the is_volume line passes. | 22:17 |
| efried | This would be something of a hack that shouldn't be necessary, but if it works... | 22:17 |
| esberglu | efried: on it | 22:18 |
| efried | thx | 22:18 |
| esberglu | efried: It works, but the connection_info is a dict, so it has to be converted to a string first (required by BlockDeviceMappin) | 22:29 |
| efried | esberglu: This is bizarre. The block_device_info is coming straight from the spawn call. In the ironic code path they're *also* not touching it before doing that _bdm_obj.is_volume business on it. So why are we getting a dict where they're getting a DriverVolumeBlockDevice? | 22:29 |
| esberglu | Wait no it still failed sorry | 22:29 |
| esberglu | efried: I have no idea. I'll post another note on the patch and we can see what mriedem has to say | 22:30 |
| efried | esberglu: I left a note on the review. It's bizarre to me that block_device_info is a plain dict. I think we wait for mriedem to look again. | 22:30 |
| efried | yeah, that. | 22:30 |
| efried | I'm out. | 22:31 |
| *** efried has quit IRC | 22:42 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!