| *** thorst has quit IRC | 00:31 | |
| *** thorst has joined #openstack-powervm | 00:31 | |
| *** thorst has quit IRC | 00:35 | |
| *** esberglu has quit IRC | 00:44 | |
| *** esberglu has joined #openstack-powervm | 00:44 | |
| *** esberglu has quit IRC | 00:49 | |
| *** esberglu has joined #openstack-powervm | 00:51 | |
| *** esberglu has quit IRC | 00:52 | |
| *** esberglu has joined #openstack-powervm | 00:52 | |
| *** esberglu has quit IRC | 00:56 | |
| *** thorst has joined #openstack-powervm | 01:07 | |
| *** thorst has quit IRC | 01:15 | |
| *** thorst has joined #openstack-powervm | 02:15 | |
| *** thorst has quit IRC | 02:35 | |
| *** jwcroppe has quit IRC | 02:50 | |
| *** jwcroppe has joined #openstack-powervm | 02:51 | |
| *** jwcroppe_ has joined #openstack-powervm | 03:11 | |
| *** jwcroppe has quit IRC | 03:13 | |
| *** esberglu has joined #openstack-powervm | 04:21 | |
| *** esberglu has quit IRC | 04:26 | |
| *** thorst has joined #openstack-powervm | 04:32 | |
| *** thorst has quit IRC | 04:37 | |
| *** shyama has joined #openstack-powervm | 04:38 | |
| *** esberglu has joined #openstack-powervm | 05:16 | |
| *** esberglu has quit IRC | 05:20 | |
| *** thorst has joined #openstack-powervm | 05:33 | |
| *** thorst has quit IRC | 05:38 | |
| *** tjakobs_ has joined #openstack-powervm | 05:53 | |
| *** tjakobs_ has quit IRC | 05:57 | |
| *** esberglu has joined #openstack-powervm | 06:10 | |
| *** esberglu has quit IRC | 06:15 | |
| *** openstackgerrit has joined #openstack-powervm | 06:18 | |
| openstackgerrit | Shyama proposed openstack/nova-powervm master: Update File i/o to support live-migration and use slot map https://review.openstack.org/432322 | 06:18 |
|---|---|---|
| openstackgerrit | Shyama proposed openstack/nova-powervm master: Update File i/o to support live-migration and use slot map https://review.openstack.org/432322 | 06:19 |
| openstackgerrit | Gautam Prasad proposed openstack/ceilometer-powervm master: Adhere to updated Inspector interface https://review.openstack.org/453024 | 06:27 |
| *** thorst has joined #openstack-powervm | 06:34 | |
| *** thorst has quit IRC | 06:38 | |
| *** thorst has joined #openstack-powervm | 07:34 | |
| *** thorst has quit IRC | 07:39 | |
| *** esberglu has joined #openstack-powervm | 07:59 | |
| *** esberglu has quit IRC | 08:04 | |
| *** k0da has joined #openstack-powervm | 08:35 | |
| *** thorst has joined #openstack-powervm | 08:35 | |
| *** thorst has quit IRC | 08:55 | |
| *** edmondsw has joined #openstack-powervm | 09:14 | |
| *** edmondsw has quit IRC | 09:18 | |
| *** esberglu has joined #openstack-powervm | 09:48 | |
| *** thorst has joined #openstack-powervm | 09:52 | |
| *** esberglu has quit IRC | 09:52 | |
| *** thorst has quit IRC | 09:56 | |
| *** k0da has quit IRC | 09:56 | |
| *** thorst has joined #openstack-powervm | 11:30 | |
| *** smatzek has joined #openstack-powervm | 11:52 | |
| *** edmondsw has joined #openstack-powervm | 12:06 | |
| *** jpasqualetto has joined #openstack-powervm | 12:11 | |
| *** jpasqualetto has quit IRC | 12:21 | |
| *** esberglu has joined #openstack-powervm | 12:31 | |
| *** jpasqualetto has joined #openstack-powervm | 12:34 | |
| a1fisher | Morning guys, I'm seeking some clarification on getting a nova-link vm to talk to an existng openstack controller. I've had some success getting my controller to see nova-link and can almost deploy an instance from Horizon but I'd like to check my approach and clarify what openstack components are needed to be installed on nova-link. I installed nova-compute and neutron-openvswitch-agent. Is this the right approach or have I | 12:36 |
| a1fisher | gone down the wrong path? | 12:36 |
| *** esberglu has quit IRC | 12:36 | |
| a1fisher | Sorry, should have added that I have installed the networking-powervm and nova-powervm drivers (with networking-powervm also going on the controller) | 12:39 |
| thorst | a1fisher: well, it really depends on the environment that you want to create. Are you doing typical I/O for PowerVM (meaning a Shared Ethernet Adapter on the VIOS)? | 12:41 |
| thorst | or are you trying to do some SDN'y things? | 12:42 |
| waler | it's a simple config thort - single VIOS, single SEA, one SSP | 12:42 |
| thorst | waler: OK - so then you need networking-powervm installed. There is a pvm-sea-agent in there to run | 12:42 |
| thorst | also, due to how neutron works, if you're not running Linux Bridge or OVS, then you need to also install the 'agent package' on the neutron server. | 12:43 |
| waler | (we kept things simple, in the hope it would avoid any additional complications) | 12:43 |
| thorst | so install networking-powervm on the controller as well. You don't need to do anything with it there, it just tells the neutron server what a SEA is | 12:43 |
| thorst | what version of OpenStack are you using? | 12:43 |
| thorst | I ask because we have a known bug in trunk that efried is working on :-( | 12:44 |
| a1fisher | thats been done. The bit I think I'm missing is running pvm-sea-agent. We're using Mitaka | 12:44 |
| thorst | Mitaka should be fine. I think around Ocata we did add the ability to use the Open vSwitch agent if you'd like. But in Mitaka, stick with the SEA agent | 12:45 |
| a1fisher | so, all we need to do from a networking point of view is install networking-powervm and run the agent? | 12:45 |
| thorst | pretty much. | 12:45 |
| thorst | unless you have multiple SEAs | 12:45 |
| waler | not at the moment :) | 12:46 |
| thorst | if you have multiple SEAs, you need to start specifying the bridge mappings and what not | 12:46 |
| a1fisher | cool, I'll give that a go. | 12:46 |
| thorst | don't forget - install it on both the novalink and your controller | 12:46 |
| thorst | run the agent just on the novalink | 12:46 |
| thorst | but install the package on both | 12:46 |
| thorst | (and restart neutron-server after you installed it on the controller) | 12:46 |
| waler | thorst : does it add much to the OpenStack setup/config, if you go from a single VIOS, to a redundant pair of VIOS? | 12:47 |
| a1fisher | OK. regarding the nova-compute part, does that need to be on the nova-link vm? | 12:47 |
| a1fisher | thats the only way I can think that nova-link will talk to the controller | 12:47 |
| thorst | waler: It depends. But generally no. Yes if you start doing multiples of things (multiple SSPs for instance) | 12:47 |
| thorst | a1fisher: nova-compute is meant to run on the novalink vm | 12:47 |
| thorst | it is not supported in any other fashion. | 12:47 |
| a1fisher | thanks | 12:48 |
| a1fisher | I'll give that a go and see far we can get | 12:48 |
| waler | many thanks thorst | 12:48 |
| thorst | good luck! | 12:48 |
| *** jwcroppe_ has quit IRC | 12:52 | |
| *** nbante has joined #openstack-powervm | 12:55 | |
| *** apearson has joined #openstack-powervm | 12:56 | |
| *** esberglu has joined #openstack-powervm | 12:57 | |
| esberglu | #startmeeting powervm_driver_meeting | 13:00 |
| openstack | Meeting started Tue Apr 4 13:00:41 2017 UTC and is due to finish in 60 minutes. The chair is esberglu. Information about MeetBot at http://wiki.debian.org/MeetBot. | 13:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 13:00 |
| openstack | The meeting name has been set to 'powervm_driver_meeting' | 13:00 |
| thorst | o/ | 13:00 |
| esberglu | #topic Out Of Tree Driver | 13:03 |
| esberglu | ocata is broken | 13:03 |
| thorst | esberglu: the upload thing? | 13:03 |
| esberglu | Yep | 13:03 |
| thorst | crap...how did that get back ported... | 13:03 |
| thorst | I thought we tested it earlier :-/ | 13:04 |
| thorst | in the staging env... | 13:04 |
| esberglu | So CI is still down, I can redeploy again with newton or we can get this figured out and be without CI until then | 13:05 |
| thorst | esberglu: well it seems like we're going to need a new pypowervm for it | 13:05 |
| esberglu | I think I may have missed some of the convo yesterday | 13:05 |
| thorst | and we're going to have to bump that back in ocata...which is awful | 13:05 |
| esberglu | But it looked like efried was expecting this | 13:05 |
| thorst | well, efried thought this could occur. I didn't think it could | 13:06 |
| thorst | it was based off of whether or not whatever broke us was back ported | 13:06 |
| thorst | efried will be on in 15 min and we can discuss more then | 13:06 |
| esberglu | Do you know what broke us? | 13:07 |
| thorst | nope | 13:07 |
| thorst | that's the irony...and what I'm frustrated about | 13:07 |
| thorst | we're trying to fix for something that we don't know why we broke | 13:07 |
| esberglu | Well lets look at what got into ocata in the last week | 13:07 |
| thorst | +2 | 13:08 |
| esberglu | Because I deployed ocata on staging at the end of last week no problem | 13:08 |
| esberglu | There's nothing that has gone into ocata since I deployed on staging successfully | 13:10 |
| esberglu | in the 3 powervm projects | 13:10 |
| thorst | right. | 13:10 |
| thorst | it'd be in nova itself | 13:10 |
| esberglu | Yep I'm looking there now | 13:10 |
| adreznec | It would have to be a bugfix at that point that broke us, right? | 13:10 |
| adreznec | I mean Ocata's been cut for some time now | 13:10 |
| thorst | I'd assume | 13:10 |
| thorst | https://github.com/openstack/nova/commits/stable/ocata | 13:10 |
| thorst | nothing much in the past week there tho | 13:11 |
| *** mdrabe has joined #openstack-powervm | 13:11 | |
| adreznec | Right... | 13:11 |
| thorst | thought maybe a global req change | 13:11 |
| esberglu | None of that looks suspect | 13:11 |
| thorst | but nothing much there | 13:11 |
| thorst | pbr updated... | 13:11 |
| adreznec | thorst: yeah, once ocata gets cut reqs are pretty much frozen | 13:11 |
| adreznec | Unless there's a major breaking issue | 13:11 |
| *** jwcroppe has joined #openstack-powervm | 13:12 | |
| thorst | esberglu: I'm now curious if newton is hosed. | 13:12 |
| esberglu | I hope not. That means CI is down for the count until this is resolved | 13:13 |
| thorst | right. | 13:13 |
| thorst | adreznec: does concurrent.futures use greenlet or eventlet? | 13:13 |
| thorst | do you know? | 13:14 |
| adreznec | thorst: not sure offhand | 13:15 |
| adreznec | wait | 13:16 |
| adreznec | isn't concurrent.futures a stdlib | 13:16 |
| thorst | adreznec: yeah, but that's where efried sees us hanging | 13:17 |
| adreznec | neither eventlet or greenlet are builtins, so it can't use either by default | 13:17 |
| thorst | so efried wants to switch to all eventlet I think... | 13:17 |
| *** efried has joined #openstack-powervm | 13:18 | |
| efried | Howdy. | 13:18 |
| thorst | so the net is...I think that is now our highest priority | 13:18 |
| efried | Not sure if I missed anything, but I have a plan for the broken upload in ocata. | 13:18 |
| thorst | and we should work it first oot | 13:18 |
| thorst | rather than it (so we don't create a bunch of misc reviews for the core side) | 13:18 |
| thorst | efried: net is, CI is down | 13:18 |
| thorst | what I think we're curious about is did something change in OpenStack...or was it perhaps even lower than that | 13:19 |
| thorst | like eventlet or somewhere else | 13:19 |
| adreznec | thorst: is this futures in py2.7 or py3? | 13:19 |
| esberglu | Staging CI is still up. So if we need to run something through we can still do it there | 13:19 |
| efried | For OOT ocata, we need a bug opened; then we a) port to ocata OOT the change that moves from FUNC back to IO_STREAM; b) update the pypowervm requirement to 1.1.1. And, of course, we'll need to release 1.1.1. | 13:19 |
| adreznec | I'll admit I'm not totally in the loop on the upload issue | 13:19 |
| thorst | efried: what is the 'fix' | 13:20 |
| thorst | down in pypowervm | 13:20 |
| efried | Yeah, so something changed in eventlet recently - I still haven't nailed down exactly what, but sdague gave me some vague pointers last week. | 13:20 |
| thorst | well shit | 13:20 |
| adreznec | efried: something that would have changed since ocata was released? | 13:20 |
| thorst | that'll affect things way back | 13:20 |
| efried | The fix is two-sided, unfortunately. In pypowervm, we have to kill coordinated upload. In community, we have to kill FUNC. | 13:21 |
| adreznec | otherwise shouldn't it be pinned by version in reqs? | 13:21 |
| efried | Because there's no way to do FUNC without threads, and there's no way to do coordinated without threads. | 13:21 |
| efried | The alternative is to retool pypowervm to use eventlet instead of futures. | 13:21 |
| thorst | efried: so is concurrent.futures just dead? | 13:21 |
| thorst | because of a change in eventlet? | 13:22 |
| efried | No, it's just incompatible with greenlet. | 13:22 |
| efried | Although that might not be entirely true. | 13:22 |
| thorst | so func is still viable, just not in an OpenStack env. | 13:22 |
| thorst | and it also calls into question if we need a change for your VIOS Task thingy | 13:22 |
| thorst | which also uses concurrent.futures | 13:22 |
| efried | I don't disagree with that. | 13:22 |
| thorst | non-disagreement is as close as we can hope for a resounding agreement | 13:23 |
| thorst | efried: so are you focused on that change today? | 13:23 |
| thorst | and we just keep CI down while we fix that? | 13:23 |
| efried | Which change? Get rid of FUNC and coordinated in ocata OOT? | 13:23 |
| efried | Or convert to greenlet? | 13:23 |
| efried | Perhaps we should take a couple of minutes and go over what I (sort of, maybe) know so far about the underlying cause. | 13:24 |
| thorst | efried: yes, lets do that | 13:24 |
| efried | So from my research with mdrabe yesterday, I *think* it goes like this: | 13:24 |
| efried | There's two kinds of multiprocessing models available: threads and greenlets. | 13:24 |
| adreznec | So just to clarify | 13:24 |
| adreznec | What version of pypowervm is this using | 13:25 |
| adreznec | With Ocata | 13:25 |
| adreznec | 1.1.0? | 13:25 |
| efried | I don't fully understand the difference between them, but they're totally different animals, not just different implementations on top of the same underlying threading model. | 13:25 |
| efried | Openstack uses greenlets throughout. They even have a hacking check in place to make sure you're using eventlet through their nova.utils wrapper of it. | 13:25 |
| efried | (adreznec, not sure, but to fix this we'll need to release 1.1.1 and bump the ocata req to that. Is that even legal?) | 13:26 |
| adreznec | Uh | 13:26 |
| esberglu | ocata is using 1.0.0.4 I believe | 13:26 |
| adreznec | We can technically do that for Ocata | 13:26 |
| adreznec | I guess... | 13:26 |
| adreznec | in the future... please, let's never have to deal with that | 13:27 |
| efried | Yeah, the req bump will need to happen regardless of which way we fix this. Unless we can figure out some as-yet-unknown way to fix it purely in the community code. | 13:27 |
| adreznec | I'm just wondering if this is only broken with some combination of versions | 13:27 |
| adreznec | e.g. only with pypowervm 1.1.0 because that's where we require futures>=3.0 | 13:27 |
| adreznec | vs just "futures" | 13:28 |
| adreznec | with no version req | 13:28 |
| efried | Mm. And presumably openstack doesn't require futures? | 13:28 |
| adreznec | nope | 13:28 |
| adreznec | sorry, yes, they do | 13:28 |
| adreznec | that's where the >3.0 req came from | 13:28 |
| adreznec | (walking to a meeting) | 13:29 |
| efried | Anyway, threading in python apparently has this GIL (global interpreter lock) which actually makes it so that literally only one thread runs at a time - the others are stopped. | 13:29 |
| thorst | right. | 13:30 |
| efried | Normally this is okay because threads can yield and allow other threads to run, so as long as your actual programming doesn't have deadlocks in it, you're aaight. | 13:30 |
| efried | But | 13:30 |
| efried | This sucker is blocking on a syscall. | 13:30 |
| efried | Which doesn't yield. | 13:30 |
| efried | So all the other threads - including the greenlets, including the one that would kick the REST server to do its open, which would unblock the write side - are frozen. | 13:31 |
| efried | Now, there is apparently a way to explicitly release the GIL. | 13:31 |
| thorst | ? | 13:32 |
| efried | That might be the least disruptive path, if we can figure out how to do it. But a) it's going to be a hack (more on that in a bit), and b) it might not work in the context we would need to do it in - that is, it might only work if we can do it right at that open() call, which is in code we don't own. | 13:33 |
| thorst | efried: to me...lets just fix it proper... | 13:33 |
| thorst | greenlet (not eventlet)? | 13:33 |
| efried | Yeah, so I don't know what the difference is there - those are different libs - but they use greenlets / green threads under the covers. | 13:35 |
| efried | Whereas anything that says "thread" - like the native thread library, or concurrent.futures - uses the other kind of threads. | 13:35 |
| thorst | I'd assume we can use greenlets for most things...but the pipe may not be able to use greenlet... | 13:36 |
| efried | That's an unknown at this point. But I imagine it's gotta be possible. | 13:37 |
| thorst | ok... | 13:37 |
| thorst | so the net is, due to this, we need to bump pypowervm...get a bug for nova-powervm...and possibly back port this way (enough) back | 13:37 |
| efried | However, given that we're going to need to release a new pypowervm and bump the OOT req to it anyway, I would just as soon do the fix that avoids threading altogether. | 13:38 |
| thorst | waler is testing on Mitaka, so he could actually probably tell us if Mitaka is impacted :-) | 13:38 |
| *** tjakobs_ has joined #openstack-powervm | 13:38 | |
| thorst | efried: for the upload? Sure. VIOS Feed Task stuff...not so sure | 13:38 |
| efried | thorst Agree, but I think what saves us there is that what's running in that thread is non-blocking. | 13:39 |
| thorst | alright...so I guess that's priority 1... | 13:39 |
| thorst | yep...I agree that is probably not highly impacted. | 13:39 |
| efried | So maybe it hitches the process while it's doing that POST, but as soon as the POST comes back, we keep truckin. | 13:39 |
| thorst | yeah, kinda ick, not uber ick | 13:40 |
| efried | Not ideal, and perhaps something we should look into for the future, but not first priority, right. | 13:40 |
| thorst | alright...so gameplan build out here... | 13:40 |
| thorst | 1) esberglu - would you be willing to make the bug and tag at least back to ocata | 13:40 |
| esberglu | Sure | 13:41 |
| thorst | I honestly suspect that newton / mitaka may still be impacted...would love to know that if you have time to redeploy with newton | 13:41 |
| thorst | 2) efried you're updating the pypowervm bits? | 13:41 |
| thorst | 3) should I do the nova-powervm bits to swap off func? | 13:42 |
| esberglu | thorst: I should be able to do that in the background today | 13:42 |
| efried | thorst Actually, let me do it. | 13:43 |
| efried | Take a look at this delta: https://review.openstack.org/#/c/443189/15..16/nova/virt/powervm/disk/ssp.py | 13:43 |
| thorst | yeah | 13:43 |
| efried | It'll be like that, except we won't actually need the IterableToFileAdapter. | 13:43 |
| thorst | we would need to basically revert into that change for both localdisk | 13:43 |
| thorst | and ssp | 13:43 |
| thorst | why not? | 13:43 |
| efried | Because I'm gonna make a change to pypowervm ;-) | 13:43 |
| efried | Since we're going to need a new version of that anyway. | 13:43 |
| efried | Backward compatible. | 13:44 |
| thorst | don't make it even more complicated :-) | 13:44 |
| efried | But eliminating the need for IterableToFileAdapter. | 13:44 |
| efried | It makes it less complicated, really. | 13:44 |
| efried | The HTTP request expects an iterable. Glance gives us an iterable. For some reason we had pypowervm expecting a file and converting it to an iterable, so the community had to convert the iterable to a file just so pypowervm could convert it back. | 13:45 |
| efried | which is stupid. | 13:45 |
| thorst | hmm...ok | 13:45 |
| thorst | well, I guess I'll let you do magic and be on point to be a reviewer? | 13:45 |
| thorst | how do we make actions in the meeting? | 13:46 |
| thorst | (for the meeting minutes) | 13:46 |
| esberglu | #action esberglu: Open bug for upload issue | 13:46 |
| *** k0da has joined #openstack-powervm | 13:46 | |
| efried | I'm going to use 5083, which is already most of the way there. Just need to add the iterable killer. | 13:48 |
| thorst | #action efried drive pypowervm and nova-powervm fixes for upload issue | 13:48 |
| thorst | #action esberglu determine if newton is impacted | 13:48 |
| thorst | efried: 5083 - we should tag that with the bug that esberglu is making | 13:50 |
| thorst | so we have one bug capturing this whole nightmare... | 13:50 |
| thorst | #action adreznec ship out a new pypowervm once this whole fiasco is solved :-) | 13:50 |
| thorst | (heh) | 13:50 |
| adreznec | lol | 13:50 |
| adreznec | might need to sync that with julio | 13:50 |
| thorst | what else do we have for the meeting? | 13:53 |
| esberglu | Cool sounds like we have a plan. Meeting is almost up, anyone have anything else? I don't have anything for CI | 13:53 |
| esberglu | efried: anything in-tree? | 13:53 |
| esberglu | Just waiting for reviewers at this point correct? | 13:53 |
| thorst | nbante and jay are still doing testing... I know jay is hitting issues, I'm trying to help out once I get in the env. nbante I thik is stuck on something with tempest in OSA. | 13:53 |
| efried | I think by the time we hit the SSP change set, we'll need to bump the in-tree reqs to the new pypowervm. | 13:53 |
| *** tjakobs_ has quit IRC | 13:54 | |
| nbante | correct..adreznec: check once if we have to uncomment anything in user_config.yml to get that work | 13:55 |
| adreznec | nbante: you'll likely have to experiment on your own there today. I'm pretty much swamped in meetings until later this afternoon | 13:56 |
| nbante | sure..I already tried most of parts. But will give shot in 2-3 hours. If not work, will send you note | 13:57 |
| esberglu | nbante: I may have time to take a look today. I will let you know | 13:58 |
| thorst | nbante: one thought I had... | 13:59 |
| nbante | sure..thanks | 13:59 |
| thorst | I don't think we care if tempest is deployed via OSA | 13:59 |
| thorst | or run from a separate server... | 13:59 |
| thorst | you have a cloud, we just need to run tempest against it | 13:59 |
| thorst | :-) | 13:59 |
| thorst | so that gives you options to try | 13:59 |
| thorst | but esberglu is more familiar than I am...so maybe he'll figure it out in 2 mins | 14:00 |
| nbante | I never tried tempest so not sure how it worked. In SVT, we have own framework | 14:00 |
| *** jwcroppe has quit IRC | 14:00 | |
| esberglu | thorst: I haven't got tempest working yet either.... so I'm pretty much in the same boat as nbante right now for OSA CI | 14:00 |
| thorst | esberglu: right...but my thought is | 14:00 |
| thorst | we have tempest working for IT CI | 14:01 |
| thorst | or OOT CI | 14:01 |
| thorst | so...uh...how'd we set it up there? | 14:01 |
| thorst | and can we do the same here? | 14:01 |
| *** jwcroppe has joined #openstack-powervm | 14:02 | |
| esberglu | At least some tweaks will be needed. We can discuss more when I really dive into it | 14:02 |
| thorst | awesome | 14:03 |
| thorst | nbante: are you deployed with Cinder? | 14:03 |
| nbante | no | 14:03 |
| nbante | using local disk only | 14:03 |
| thorst | so we're just getting back to where we were? | 14:03 |
| nbante | after local disk, I worked on tempest where I stuck | 14:04 |
| nbante | do you want me to parallely work on configuring cinder? | 14:05 |
| thorst | ok | 14:05 |
| thorst | ahh, right...so we were getting tempest and then moving to iSCSI cinder | 14:05 |
| nbante | correct | 14:05 |
| thorst | sorry, I'm getting my wires wrong :-) | 14:05 |
| nbante | :) | 14:06 |
| thorst | OK - I'll also catch up with Jay... | 14:06 |
| thorst | can you check with him to get his IRC working? | 14:06 |
| nbante | sure..will check | 14:07 |
| thorst | ok - I didn't have anything else. | 14:07 |
| esberglu | #endmeeting | 14:11 |
| openstack | Meeting ended Tue Apr 4 14:11:13 2017 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 14:11 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/powervm_driver_meeting/2017/powervm_driver_meeting.2017-04-04-13.00.html | 14:11 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/powervm_driver_meeting/2017/powervm_driver_meeting.2017-04-04-13.00.txt | 14:11 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/powervm_driver_meeting/2017/powervm_driver_meeting.2017-04-04-13.00.log.html | 14:11 |
| shyama | thorst: efried https://review.openstack.org/#/c/432322 the is_volume_on_vios is used by slot.py to rebuild vscsi slots | 14:16 |
| efried | shyama See new review | 14:34 |
| *** mdrabe has quit IRC | 14:37 | |
| *** mdrabe has joined #openstack-powervm | 14:43 | |
| efried | thorst adreznec esberglu 5083 PS2 | 14:47 |
| esberglu | efried: https://bugs.launchpad.net/nova-powervm/+bug/1679720 | 14:47 |
| openstack | Launchpad bug 1679720 in nova-powervm "Openstack image upload failure" [Undecided,New] | 14:47 |
| esberglu | I basically just summarized what happened in IRC | 14:48 |
| thorst | efried: loking | 14:48 |
| efried | thorst How do you want me to deprecate FUNC? With a comment, or actually check for its usage and issue DeprecationWarning? | 14:48 |
| thorst | comment... | 14:48 |
| esberglu | I tried to just summarize the IRC chat for that bug, might be missing some stuff since I don't understand the issue as well as you | 14:48 |
| esberglu | Feel free to update | 14:48 |
| efried | thorst k, will address in PS3; meanwhile please review PS2 for any other changes needed. | 14:49 |
| *** tjakobs_ has joined #openstack-powervm | 14:49 | |
| efried | esberglu Regarding the bug - does the upload actually fail with a 500? Doesn't just hang? Can you put a stack trace in the bug? | 14:54 |
| esberglu | efried: Yeah it got a 500 error when I was trying to stack yesterday. But I'm dumb and forgot to save off the logs before trying to deploy on newton | 14:55 |
| efried | That's... weird. | 14:55 |
| esberglu | So I don't have a stack trace, and it looks like newton is working | 14:55 |
| efried | I need to see that 500. Could be a separate issue. | 14:55 |
| efried | Assume it was during the primer LU creation | 14:56 |
| efried | We might not need to fix ocata. Which would be kinda nice. | 14:56 |
| esberglu | Well once I confirm newton works, I can go back and try ocata again | 14:56 |
| esberglu | But here's what happened | 14:56 |
| esberglu | It downloaded the image from the remote location | 14:56 |
| esberglu | Then tried to run the openstack create image command | 14:57 |
| esberglu | And got a 500 error there | 14:57 |
| efried | openstack create image doesn't touch our code. That 500 would have had to be from glance. Does that sound familiar? | 14:57 |
| efried | thorst esberglu adreznec Until we nail that down, I'm going to modify this bug to represent the hang we know about. | 14:59 |
| esberglu | Okay. I'm restacking with ocata now | 15:00 |
| *** jwcroppe has quit IRC | 15:05 | |
| *** jwcroppe has joined #openstack-powervm | 15:07 | |
| *** nbante has quit IRC | 15:07 | |
| efried | thorst Feedback on PS2? | 15:16 |
| efried | Still working at it? | 15:16 |
| efried | esberglu We're going to have a tricky-ish time testing this, because the nova-powervm change set is going to have a requirements update to pypowervm 1.1.1, which hasn't been released yet; but we don't want to release that until we've tested it. Can you finagle a way around that? | 15:18 |
| esberglu | efried: Yeah should be able to just pull in the pypowervm patch and then remove pypowervm from the reqs | 15:19 |
| efried | k | 15:19 |
| thorst | I +1'd it | 15:20 |
| efried | k | 15:20 |
| efried | thorst PS3 is up. | 15:20 |
| efried | Added Partial-Bug to the commit message too. | 15:20 |
| *** shyama_ has joined #openstack-powervm | 15:22 | |
| thorst | LGTM now | 15:24 |
| *** shyama has quit IRC | 15:24 | |
| *** shyama_ is now known as shyama | 15:24 | |
| openstackgerrit | Eric Fried proposed openstack/nova-powervm master: create_disk_from_image: IO_STREAM instead of FUNC https://review.openstack.org/453233 | 15:45 |
| efried | thorst adreznec esberglu ^^ | 15:45 |
| thorst | efried: ok. I'll review that while you review shyama's (if you haven't already) | 15:45 |
| efried | Note that we're going to have to bump g-r pypowervm also before this merges, or our shiny new bot is going to propose a reversion to 1.1.0 | 15:45 |
| efried | thorst I did. Needs more changes. | 15:46 |
| efried | bbiab | 15:46 |
| thorst | shyama: FYI | 15:46 |
| efried | esberglu See if you can pull ^^ (453233) and pypowervm 5083 into the CI. | 15:46 |
| esberglu | efried: Okay. I will do a manual run on staging. That's really my only option right now | 15:47 |
| esberglu | efried: Manual run is started. Should see results in 30 min or so | 16:08 |
| esberglu | It also looks like the ocata stack is going to go through for CI undercloud. Not done yet, but past the point it failed previously | 16:08 |
| esberglu | Just a weird coincidence that I hit it during the image upload | 16:09 |
| *** shyama has quit IRC | 16:17 | |
| efried | esberglu Does that guy use the same SSP as others? | 16:18 |
| esberglu | No it has its own | 16:18 |
| efried | Okay, good. Cause we won't test the code path in question unless we're actually uploading a new image LU. | 16:20 |
| *** shyama has joined #openstack-powervm | 16:33 | |
| openstackgerrit | Shyama proposed openstack/nova-powervm master: Update File i/o to support live-migration and use slot map https://review.openstack.org/432322 | 16:46 |
| shyama | efried: done with the changes. please check the new change set https://review.openstack.org/#/c/432322 | 16:47 |
| esberglu | efried: Passed the run | 16:56 |
| thorst | shyama: I posted a comment in there | 17:02 |
| thorst | small one, otherwise looks good to me | 17:03 |
| efried | thorst https://review.openstack.org/#/c/453233/ | 17:04 |
| efried | shyama I posted new comments as well. | 17:07 |
| *** nbante has joined #openstack-powervm | 17:19 | |
| shyama | efried: https://review.openstack.org/#/c/432322/7/nova_powervm/virt/powervm/volume/driver.py@120 I will make it " Needs to be implemented within the subclass to support rebuild." does that sound ok? | 17:20 |
| efried | shyama Ah, yes, that makes more sense. Thanks. | 17:20 |
| openstackgerrit | Shyama proposed openstack/nova-powervm master: Update File i/o to support live-migration and use slot map https://review.openstack.org/432322 | 17:35 |
| efried | thorst esberglu FYI, got a bug in Content-Length header. Looks like glance's iterable answers to len() with a zero. | 17:40 |
| thorst | efried: still think we can get away without a facade? | 17:41 |
| efried | You suck | 17:41 |
| thorst | efried: wasn't being sassy | 17:41 |
| thorst | just asking :-) | 17:41 |
| efried | Thing is, even if it returned 0, the requests code should still be doing the right thing. | 17:42 |
| efried | Must be something else. Still looking... | 17:42 |
| *** apearson has quit IRC | 18:00 | |
| *** apearson has joined #openstack-powervm | 18:03 | |
| efried | thorst I am stumped. | 18:05 |
| efried | When I reinstate IterableToFileAdapter and use chunkreader(), it works. | 18:05 |
| efried | I even tried sending the glance iterable through chunkreader(). | 18:05 |
| efried | Which should be NO different. | 18:06 |
| thorst | so the IterableToFileAdapter had some weird-ness to it | 18:08 |
| thorst | that I didnt' see until like 5th pass throgh it | 18:08 |
| thorst | specifically around that 'remaining_data' bit. | 18:08 |
| thorst | if it passed size in, it would get part of it, but otherwise it got all of it... | 18:09 |
| efried | Well, we are passing size in. | 18:09 |
| efried | And it's being hidden through chunkreader() anyway. | 18:10 |
| thorst | hmm | 18:10 |
| thorst | do you catch the StopIteration? | 18:11 |
| efried | Yeah. | 18:12 |
| efried | Think I've got a version that works now. | 18:12 |
| thorst | not sure then... | 18:12 |
| efried | I think it's because the glance thingy isn't really an iterable per se. | 18:13 |
| efried | I mean, it is and it isn't. | 18:13 |
| thorst | right. | 18:13 |
| thorst | they're multi meg chunks | 18:13 |
| efried | 64K | 18:14 |
| efried | or... whatever. | 18:14 |
| thorst | hmmm...I thought they made them bigger | 18:14 |
| efried | oh, perhaps so, and the IterableToFileAdapter is forcing them to 64K | 18:15 |
| efried | Rather, the chunkreader is forcing them to 64K through the IterableToFileAdapter. | 18:16 |
| thorst | right. | 18:17 |
| efried | Anyway, I believe I've got a version that works without IterableToFileAdapter. toxing it now, will post in a bit. | 18:17 |
| efried | thorst Check out PS4 | 18:20 |
| efried | thorst adapter.py best compared against base. | 18:20 |
| thorst | so you're basically saying upload the whole chunk unless they have a read method... | 18:21 |
| thorst | but then we ignore the chunksize parameter altogether that way... | 18:22 |
| thorst | seems like we'd want to specify that somehow... | 18:22 |
| esberglu | BergySpecial1 | 18:22 |
| esberglu | Damn | 18:22 |
| efried | Time to change your password. | 18:22 |
| adreznec | Now's my chance! | 18:23 |
| esberglu | Time to change my gmail password | 18:23 |
| efried | thorst I'm not a fan of the overhead associated with rechunking. We're getting rid of that by getting rid of IterableToFileAdapter. | 18:23 |
| * adreznec furiously logs in as esberglu everywhere | 18:23 | |
| *** apearson has quit IRC | 18:23 | |
| adreznec | :P | 18:23 |
| efried | thorst And chunksize isn't overridden anywhere. That 64K is pretty arbitrary, almost certainly an artifact left over from the k2operator days. | 18:24 |
| *** nbante has quit IRC | 18:25 | |
| thorst | lulz | 18:26 |
| thorst | efried: totes | 18:26 |
| thorst | Just thinking its another thing to note in the method signature | 18:26 |
| thorst | (which is non-existent at present) | 18:26 |
| efried | thorst Tested with SSP, works. Gonna test with localdisk now. | 18:26 |
| efried | thorst Yeah, not a terrible idea to add a docstring there. | 18:27 |
| efried | thorst But it's gonna be a big 'un. | 18:28 |
| thorst | efried: just capture the delta. | 18:30 |
| thorst | don't boil the adapter.py docstring ocean | 18:30 |
| efried | thorst But, but, but... my OCD!!! | 18:30 |
| *** mdrabe has quit IRC | 18:30 | |
| thorst | handle your OCD in a separate change set. | 18:31 |
| *** shyama has quit IRC | 18:33 | |
| efried | thorst PS5 | 18:45 |
| efried | Man, we don't have a lot of logging on the localdisk side. | 18:45 |
| thorst | we probably never cleaned it up like we did ssp | 18:46 |
| thorst | it's just such a lower used path | 18:46 |
| efried | thorst Oh, crap, sonar. | 18:46 |
| *** jwcroppe has quit IRC | 18:47 | |
| *** jwcroppe has joined #openstack-powervm | 18:48 | |
| *** apearson has joined #openstack-powervm | 18:49 | |
| *** mdrabe has joined #openstack-powervm | 18:50 | |
| *** jwcroppe has quit IRC | 18:52 | |
| efried | thorst Interesting stat: upload to localdisk too 434s. Versus about 100s less to SSP. | 18:52 |
| thorst | is localdisk truly using a localdisk? | 18:53 |
| thorst | or a hdisk backed by FC? | 18:53 |
| thorst | the SAS drives aren't as fast as your FC. | 18:53 |
| efried | thorst Dunno. | 18:54 |
| efried | But I would have expected the "network" to be the bottleneck anyway. | 18:54 |
| efried | The RMC limiter | 18:54 |
| thorst | heh, well maybe it isn't | 18:55 |
| thorst | :-) | 18:55 |
| efried | thorst Sonar-fixin PS6 posted. | 18:57 |
| efried | Retesting for sanity with localdisk. | 18:57 |
| efried | ahcrap. | 18:57 |
| efried | hold on. | 18:58 |
| thorst | efried: I've got a meeting for 60 min | 18:58 |
| thorst | so I'll recheck then :-) | 18:59 |
| efried | thorst Okay, PS7 is read now. | 18:59 |
| efried | ready | 18:59 |
| efried | swhat I get for developing from two different machines. | 18:59 |
| efried | adreznec We're going to need to release pypowervm 1.1.1 now. You got a handle on that? | 19:00 |
| adreznec | efried: I'd need to talk with Julio on the specifics required now with the new flows | 19:01 |
| adreznec | He handled everything the last time around | 19:01 |
| efried | thorst I'd like to get 5037 in there as well. Please add that to your review queue | 19:01 |
| adreznec | Not sure how he handled tagging/version bumping | 19:01 |
| *** mdrabe has quit IRC | 19:02 | |
| *** mdrabe has joined #openstack-powervm | 19:04 | |
| efried | esberglu So ocata is green? | 19:10 |
| esberglu | efried: Finishing up the stacks on the compute nodes, hit some package versioning issues there | 19:11 |
| esberglu | But yeah it appears to be | 19:11 |
| efried | esberglu Has it made the primer LU yet? | 19:11 |
| esberglu | efried: No not yet | 19:22 |
| efried | esberglu Okay, so we won't know if we're busted there until that happens. | 19:22 |
| openstackgerrit | Eric Fried proposed openstack/nova-powervm master: create_disk_from_image: IO_STREAM instead of FUNC https://review.openstack.org/453233 | 19:50 |
| thorst | efried: hit up the reviews | 19:54 |
| thorst | +2 | 19:54 |
| thorst | I think we need to push a new pypowervm before we push https://review.openstack.org/#/c/453233 through | 19:55 |
| efried | thorst Yes, before that will verify. I wouldn't mind a look-see now, though. | 19:55 |
| efried | thorst mdrabe Do we want to get 4975 in? | 19:57 |
| efried | That guy has (or will have) some fixage to the retry logic. | 19:59 |
| efried | Right now, if a streamed upload fails partway, we'll retry it. | 19:59 |
| efried | ...with a stream that's partially consumed. | 19:59 |
| efried | Which is bogus. | 19:59 |
| efried | thorst And what about 5052, which appears to be ready? | 20:00 |
| thorst | efried: 5052 - yes | 20:01 |
| thorst | I can submit that now | 20:01 |
| efried | k | 20:01 |
| efried | mdrabe How close are we on 4975? | 20:08 |
| *** thorst has quit IRC | 20:35 | |
| mdrabe | efried: Sorry been in a meeting, you're dependant on 4975 ya? | 20:36 |
| *** smatzek has quit IRC | 20:37 | |
| *** mdrabe has quit IRC | 20:40 | |
| *** mdrabe has joined #openstack-powervm | 20:48 | |
| *** thorst has joined #openstack-powervm | 21:07 | |
| esberglu | efried: Hit a few more hiccups getting the compute nodes to stack | 21:15 |
| esberglu | Image template is built now, ready nodes are spawning | 21:15 |
| *** tjakobs_ has quit IRC | 21:20 | |
| efried | mdrabe Not dependent; but it did uncover a definite bug. | 21:24 |
| efried | mdrabe It's pretty low-probability, and if the upload breaks partway through, we'd wind up failing anyway. Your fix would just fail us sooner and probably prettier. | 21:25 |
| efried | So it's not urgent or strictly necessary. | 21:25 |
| mdrabe | K should have it up in a bit | 21:26 |
| mdrabe | If UT is will play | 21:26 |
| efried | mdrabe See my response to thorst comment. | 21:29 |
| efried | Anyway, no hurry; I don't think we care too much about getting this into 1.1.1 | 21:29 |
| mdrabe | efried So you thinkin keep all the retries then? | 21:30 |
| efried | mdrabe Keep the retry on the func path - unless thorst still disagrees. | 21:30 |
| efried | We may want to talk it out first so we don't end up thrashing the change set. | 21:31 |
| mdrabe | Ight I'll leave that be for now | 21:31 |
| *** edmondsw has quit IRC | 21:31 | |
| *** edmondsw has joined #openstack-powervm | 21:33 | |
| *** edmondsw has quit IRC | 21:38 | |
| *** smatzek has joined #openstack-powervm | 21:58 | |
| *** esberglu has quit IRC | 21:58 | |
| *** thorst has quit IRC | 22:01 | |
| *** mdrabe has quit IRC | 22:03 | |
| *** smatzek has quit IRC | 22:12 | |
| *** esberglu has joined #openstack-powervm | 22:13 | |
| *** k0da has quit IRC | 22:13 | |
| *** tjakobs_ has joined #openstack-powervm | 22:25 | |
| *** tjakobs__ has joined #openstack-powervm | 22:29 | |
| *** apearson has quit IRC | 22:29 | |
| *** tjakobs_ has quit IRC | 22:32 | |
| *** apearson has joined #openstack-powervm | 22:38 | |
| *** tjakobs__ has quit IRC | 22:42 | |
| *** esberglu has left #openstack-powervm | 22:45 | |
| *** jpasqualetto has quit IRC | 22:53 | |
| *** apearson has quit IRC | 22:53 | |
| *** thorst has joined #openstack-powervm | 23:01 | |
| *** jpasqualetto has joined #openstack-powervm | 23:05 | |
| *** thorst has quit IRC | 23:06 | |
| *** jpasqualetto has quit IRC | 23:13 | |
| *** jpasqualetto has joined #openstack-powervm | 23:26 | |
| *** jpasqualetto has quit IRC | 23:28 | |
| *** svenkat has joined #openstack-powervm | 23:45 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!