| *** esberglu has quit IRC | 00:12 | |
| *** edmondsw has quit IRC | 00:46 | |
| *** apearson has joined #openstack-powervm | 00:58 | |
| *** thorst_ has quit IRC | 01:24 | |
| *** thorst_ has joined #openstack-powervm | 01:25 | |
| *** thorst_ has quit IRC | 01:33 | |
| *** svenkat has joined #openstack-powervm | 01:46 | |
| *** thorst_ has joined #openstack-powervm | 02:14 | |
| *** thorst_ has quit IRC | 02:14 | |
| *** thorst_ has joined #openstack-powervm | 02:14 | |
| *** thorst_ has quit IRC | 02:23 | |
| *** svenkat has quit IRC | 02:26 | |
| *** svenkat has joined #openstack-powervm | 02:27 | |
| *** thorst_ has joined #openstack-powervm | 02:32 | |
| *** thorst_ has quit IRC | 02:33 | |
| *** thorst_ has joined #openstack-powervm | 02:39 | |
| *** thorst_ has quit IRC | 02:39 | |
| *** svenkat has quit IRC | 03:15 | |
| *** seroyer has quit IRC | 03:29 | |
| *** thorst_ has joined #openstack-powervm | 03:40 | |
| *** thorst_ has quit IRC | 03:48 | |
| *** thorst_ has joined #openstack-powervm | 04:48 | |
| *** thorst_ has quit IRC | 04:54 | |
| *** thorst_ has joined #openstack-powervm | 05:52 | |
| *** thorst_ has quit IRC | 05:59 | |
| *** apearson has quit IRC | 06:29 | |
| *** apearson has joined #openstack-powervm | 06:34 | |
| *** thorst_ has joined #openstack-powervm | 06:58 | |
| *** thorst_ has quit IRC | 07:04 | |
| *** thorst_ has joined #openstack-powervm | 08:03 | |
| *** thorst_ has quit IRC | 08:09 | |
| *** k0da has joined #openstack-powervm | 08:41 | |
| *** thorst_ has joined #openstack-powervm | 09:07 | |
| *** thorst_ has quit IRC | 09:14 | |
| *** Cartoon has joined #openstack-powervm | 09:53 | |
| *** Cartoon has quit IRC | 09:53 | |
| *** Cartoon has joined #openstack-powervm | 09:55 | |
| *** thorst_ has joined #openstack-powervm | 10:13 | |
| *** thorst_ has quit IRC | 10:19 | |
| *** thorst_ has joined #openstack-powervm | 11:17 | |
| *** thorst_ has quit IRC | 11:17 | |
| *** thorst_ has joined #openstack-powervm | 11:17 | |
| *** viclarson has joined #openstack-powervm | 11:29 | |
| *** kylek3h_away has quit IRC | 11:55 | |
| *** edmondsw has joined #openstack-powervm | 12:08 | |
| *** thorst_ has quit IRC | 12:37 | |
| *** esberglu has joined #openstack-powervm | 12:54 | |
| *** tblakes has joined #openstack-powervm | 12:54 | |
| *** mdrabe has joined #openstack-powervm | 13:20 | |
| *** kriskend_ has joined #openstack-powervm | 13:28 | |
| *** efried has joined #openstack-powervm | 14:16 | |
| *** tjakobs has joined #openstack-powervm | 14:21 | |
| *** seroyer has joined #openstack-powervm | 14:24 | |
| *** seroyer has quit IRC | 14:28 | |
| *** smatzek has joined #openstack-powervm | 14:35 | |
| *** apearson has quit IRC | 14:50 | |
| *** smatzek has quit IRC | 14:51 | |
| *** smatzek has joined #openstack-powervm | 15:06 | |
| *** smatzek has quit IRC | 15:07 | |
| *** mdrabe has quit IRC | 15:10 | |
| *** thorst_ has joined #openstack-powervm | 15:11 | |
| *** thorst_ has quit IRC | 15:11 | |
| *** thorst_ has joined #openstack-powervm | 15:13 | |
| *** mdrabe has joined #openstack-powervm | 15:18 | |
| *** seroyer has joined #openstack-powervm | 15:21 | |
| *** kylek3h has joined #openstack-powervm | 15:25 | |
| *** kylek3h has quit IRC | 15:51 | |
| *** seroyer has quit IRC | 16:00 | |
| *** AlexeyAbashkin has quit IRC | 16:01 | |
| *** seroyer has joined #openstack-powervm | 16:03 | |
| *** viclarson has quit IRC | 16:12 | |
| adreznec | esberglu: thorst_ if the SSP is full I'd expect errors like that | 16:17 |
|---|---|---|
| adreznec | How much space do we have out there? | 16:17 |
| esberglu | adreznec: thorst: SSP is not full | 16:17 |
| thorst_ | output from pvmctl? | 16:17 |
| thorst_ | and what's the lu output? | 16:17 |
| esberglu | +-----------+----------+------------+----------+ | 16:17 |
| esberglu | | Name | Cap (GB) | Free Space | OC Space | | 16:17 |
| esberglu | +-----------+----------+------------+----------+ | 16:17 |
| esberglu | | ssp_stage | 249.88 | 222.86 | 0.0 | | 16:17 |
| esberglu | +-----------+----------+------------+----------+ | 16:17 |
| thorst_ | efried: what is OC Space? | 16:18 |
| esberglu | +-----------+---------------------------------------------------------------+----------+-------+------+---------------------------------------------------------------+--------+ | 16:18 |
| esberglu | | SSP | Name | Cap (GB) | Type | Thin | Clone | In use | | 16:18 |
| esberglu | +-----------+---------------------------------------------------------------+----------+-------+------+---------------------------------------------------------------+--------+ | 16:18 |
| esberglu | | ssp_stage | image_template_PowerVM_Ubuntu_Base_1476973324_c04c707c85afd5> | 60.0 | Image | True | image_template_PowerVM_Ubuntu_Base_1476973324_c04c707c85afd5> | False | | 16:18 |
| thorst_ | that SSP is smaller than I thought it was. | 16:18 |
| esberglu | | | part10253365image_template_PowerVM_Ubuntu_Base_1476973324_c0> | 0.01 | Image | True | part10253365image_template_PowerVM_Ubuntu_Base_1476973324_c0> | False | | 16:18 |
| esberglu | +-----------+---------------------------------------------------------------+----------+-------+------+---------------------------------------------------------------+--------+ | 16:18 |
| thorst_ | heh, we need to use more pastebin. | 16:18 |
| efried | thorst_, not sure - ask clbush. | 16:18 |
| *** openstackgerrit has quit IRC | 16:18 | |
| esberglu | Ugh that looks awful. You guys need a pastebin for it? | 16:19 |
| thorst_ | nah | 16:19 |
| adreznec | I got it, but yeah, IRC doesn't handle tables well | 16:19 |
| thorst_ | so if it can't find the right LU, I'd probably ask efried to take a peak? | 16:19 |
| *** openstackgerrit has joined #openstack-powervm | 16:19 | |
| thorst_ | he's mr. SSP & OpenStack | 16:19 |
| adreznec | Poor guy... err... lucky, I meant lucky | 16:20 |
| thorst_ | hey, I'm just saying that SSP has linked clones. | 16:21 |
| adreznec | kriskend_: any more luck with testing the new glance code? | 16:25 |
| efried | esberglu, what do you want me to look at? | 16:26 |
| kriskend_ | adreznec haven't yet figured out why it isn't running | 16:26 |
| adreznec | Crud, ok | 16:28 |
| esberglu | efried: Nodes aren't spawning on the staging CI. Seeing this in the FFDC log | 16:40 |
| esberglu | http://paste.openstack.org/show/586606/ | 16:40 |
| esberglu | By aren't spawning I mean they are just hanging, not erroring out | 16:40 |
| efried | esberglu, that FFDC stuff doesn't mean anything to me. | 16:41 |
| efried | Have you asked someone in REST? | 16:41 |
| efried | I can take a look at the compute logs if you like. Hanging is weird, sounds like another marker LU problem. | 16:42 |
| esberglu | Delete is hanging too | 16:45 |
| esberglu | Its on neo14 | 16:45 |
| thorst_ | efried: as you get a chance...can you look at launchpad bug 1634963 in networking-powervm. Its due to a change you made and I've got an exploiter asking for status on it. | 16:48 |
| openstack | Launchpad bug 1634963 in networking-powervm "provision_devices called with empty set of requests" [Undecided,New] https://launchpad.net/bugs/1634963 | 16:48 |
| efried | thorst_, looking. | 16:48 |
| *** seroyer has quit IRC | 16:49 | |
| *** AlexeyAbashkin has joined #openstack-powervm | 16:49 | |
| *** AlexeyAbashkin has quit IRC | 16:49 | |
| thorst_ | efried: thx dude | 16:52 |
| kriskend_ | adreznec Hmm stepping thru this new glance code... apparently it is running... | 16:54 |
| *** AlexeyAbashkin has joined #openstack-powervm | 16:54 | |
| *** k0da has quit IRC | 16:55 | |
| *** dwayne has joined #openstack-powervm | 16:57 | |
| efried | thorst_, that bug is pretty bogus. The overhead in provision_devices for an empty list is negligible. | 16:57 |
| thorst_ | does it call get_device_details at all? | 16:58 |
| efried | no | 16:58 |
| thorst_ | then I agree. | 16:58 |
| thorst_ | I'm good to reject it | 16:58 |
| thorst_ | lets just let the exploiter know | 16:58 |
| thorst_ | I don't see arnoldje in here... | 16:58 |
| efried | It builds a couple of empty dicts. Other than that, it's just the method call overhead itself. | 16:58 |
| efried | oh | 16:59 |
| thorst_ | the only reason I question it is because its arnoldje... his analysis is the best analysis | 16:59 |
| thorst_ | "the best" | 16:59 |
| thorst_ | (channeling donald) | 16:59 |
| efried | I wasn't looking at the (non-networking-powervm) exploiter code. | 16:59 |
| thorst_ | ahh, lol | 16:59 |
| efried | But I would challenge that we shouldn't call the method at all. | 16:59 |
| thorst_ | well, meh | 16:59 |
| efried | Yeah, if they want to no-op on zero requests, they can check that at the beginning of their stuff. | 17:00 |
| thorst_ | yep. | 17:00 |
| efried | How do I know whether every exploiter wants to act on an empty list or not? | 17:00 |
| thorst_ | you don't | 17:00 |
| thorst_ | reject it | 17:00 |
| efried | nod | 17:01 |
| efried | thorst_, all that said, I'm not sure I really see the value in calling the provision at all on an empty list. | 17:04 |
| efried | So, meh, I wouldn't object to "fixing" it in the community. | 17:04 |
| efried | thoughts? | 17:04 |
| thorst_ | Its minor, but yeah. I'm fine either way | 17:04 |
| thorst_ | your call | 17:04 |
| thorst_ | adreznec: in our OSA install...I *think* I see the issue. | 17:05 |
| thorst_ | the neutron-agents-container...its linking br-int to 'br-provider'. Which doesn't seem to actually exist within the container | 17:05 |
| *** apearson has joined #openstack-powervm | 17:07 | |
| *** apearson has quit IRC | 17:07 | |
| *** apearson has joined #openstack-powervm | 17:10 | |
| thorst_ | adreznec: I can manually throw it on...but we'd lose it with the OSA rebuild | 17:15 |
| adreznec | thorst_: Sorry, was grabbing lunch | 17:24 |
| adreznec | Hmm, ok. br-provider was actually a bit of post-setup I did, it wasn't part of the OSA build at all. | 17:26 |
| adreznec | As far as I can tell the OSA OVCS setup doesn't actually connect br-int to anything | 17:26 |
| adreznec | *OVS | 17:26 |
| thorst_ | adreznec: I think there is a bug in OSA for OVS | 17:29 |
| thorst_ | because it doesn't do that setup | 17:29 |
| thorst_ | :-) | 17:29 |
| adreznec | Agreed. For now I think we can just set it up manually and doc the requirement | 17:31 |
| adreznec | And push stuff up for fixes next week at the summit | 17:31 |
| thorst_ | bleh | 17:31 |
| *** seroyer has joined #openstack-powervm | 18:07 | |
| -openstackstatus- NOTICE: The Gerrit service on review.openstack.org is being restarted now in an attempt to resolve some mismatched merge states on a few changes, but should return momentarily. | 18:09 | |
| thorst_ | adreznec: can we try to have each VM connect to another? | 18:13 |
| adreznec | Did you figure out that fw issue? | 18:13 |
| thorst_ | did you get the 10.0.0.x applied to another VM? | 18:13 |
| thorst_ | no, but I want to see if VM to VM is OK | 18:13 |
| adreznec | No, I can do that quick | 18:13 |
| adreznec | Was still looking at config drive stuff | 18:14 |
| *** tblakes has quit IRC | 18:23 | |
| adreznec | thorst_: Ok, I can ping 10.0.0.4 from 10.0.0.8 | 18:28 |
| thorst_ | OK...so its definitely this network node | 18:28 |
| thorst_ | I can't seem to get it working right. | 18:28 |
| thorst_ | I think we'll need to connect with automagically at Barcelona about it...but for now...I think we may want to redeploy with the network node using LB | 18:29 |
| adreznec | Hmm ok, let me poke at it for a minute | 18:29 |
| adreznec | What changes did you make so far to get traffic flowing? | 18:29 |
| thorst_ | FYI - I currently have the firewall driver turned to noop | 18:29 |
| thorst_ | adreznec: Changes to get it into the host. I did: | 18:33 |
| thorst_ | 1) ovs-vsctl add-port br-provider eth1 | 18:33 |
| thorst_ | 2) ifconfig eth1 up | 18:33 |
| thorst_ | 3) ifconfig eth1 0.0.0.0 | 18:34 |
| thorst_ | Step 2 was a br-provider up... | 18:34 |
| thorst_ | 4) ifconfig br-provider <ip> netmask 255.255.252.0 | 18:34 |
| thorst_ | 5) restart agent | 18:34 |
| thorst_ | and a few iterations have been messing with the conf files....but nothing drastic | 18:34 |
| adreznec | Ok | 18:35 |
| *** apearson has quit IRC | 18:41 | |
| *** tobias_ has joined #openstack-powervm | 18:55 | |
| *** tobias_ has quit IRC | 18:59 | |
| thorst_ | adreznec: anything I should be doing atm? | 19:23 |
| *** k0da has joined #openstack-powervm | 19:23 | |
| adreznec | thorst_: You could try digging into the config drive stuff I guess. I've been playing with the network node config and trying to recover our other OSA controller, haven't had time to dig any farther as to why the interfaces data isn''t getting populated correctly | 19:28 |
| thorst_ | adreznec: can we deploy? | 19:29 |
| thorst_ | VIOSes are RMC busy, are they not? | 19:29 |
| adreznec | Uhh... good question | 19:30 |
| adreznec | Do we need to restart the vio daemons? | 19:30 |
| thorst_ | yeah... | 19:30 |
| thorst_ | did mmandell's upload ever finish? | 19:31 |
| adreznec | Not sure | 19:32 |
| kriskend_ | thorst: any reason this code would be much faster? | 19:52 |
| *** seroyer has quit IRC | 19:52 | |
| thorst_ | kriskend_: yes. But how much faster are we talking | 19:53 |
| thorst_ | if its like 99% faster, then I'm not sure its copying the bits :-) | 19:53 |
| thorst_ | if its 50% faster...I could believe that | 19:53 |
| kriskend_ | not sure yet... we will see | 19:53 |
| thorst_ | kriskend_: the suspense... | 19:54 |
| kriskend_ | Task crt_disk_from_img completed in 217 seconds for instance KAx-3-c1da6a77-pvm | 19:56 |
| kriskend_ | 10G image | 19:56 |
| adreznec | What was it before | 19:56 |
| *** seroyer has joined #openstack-powervm | 19:57 | |
| thorst_ | and make sure it's still doing three at a time...and not one by one | 19:58 |
| kriskend_ | i think it was around 400 seconds or so | 19:58 |
| kriskend_ | it grew if you did a bunch... | 19:59 |
| kriskend_ | I am going to have to remove some breakpoints... | 19:59 |
| thorst_ | kriskend_: so naturally one will be faster than 3 at a time...so that's expected. But another thing I'd expect...lower CPU usage in the nova-powervm process while it runs. | 19:59 |
| kriskend_ | I am doing 7 deploys I think 3 at a time are still happening and the total time for all of them is not going to be nearly as long as it was before | 20:00 |
| kriskend_ | half is probably about where it will land... | 20:00 |
| thorst_ | rockin | 20:01 |
| thorst_ | monitor the CPU utilization of the nova_powervm process too | 20:01 |
| thorst_ | what's that look like | 20:01 |
| thorst_ | (I believe it killed a core initially) | 20:02 |
| kriskend_ | like 5% | 20:02 |
| thorst_ | smells like a winner | 20:03 |
| kriskend_ | oh but I think it was past the copy part already | 20:03 |
| thorst_ | o bleh. | 20:03 |
| kriskend_ | last couple VMs are booting now | 20:03 |
| thorst_ | that's the part I care about | 20:03 |
| kriskend_ | so 20 minutes to do 7 VMs...not bad | 20:04 |
| kriskend_ | and there was a bit of extra delay cuz of my breakpoints | 20:04 |
| kriskend_ | I want to try this on the system that always has issues with multi deploys | 20:04 |
| kriskend_ | so going to load it there and try it out | 20:04 |
| thorst_ | kriskend_: sounds good | 20:06 |
| thorst_ | I made a bug for the other issue you pinged me on | 20:06 |
| adreznec | Which system is that | 20:06 |
| thorst_ | https://bugs.launchpad.net/nova-powervm/+bug/1635385 | 20:06 |
| openstack | Launchpad bug 1635385 in nova-powervm "True error can get lost in getting partition state" [Undecided,New] | 20:06 |
| kriskend_ | hmm anyone know if Hsien's delete patch got merged... | 20:10 |
| kriskend_ | or how I would go about getting it? | 20:11 |
| adreznec | the vopt delete on kriskend_ ? | 20:12 |
| adreznec | *one | 20:12 |
| adreznec | If so, that's merged | 20:13 |
| kriskend_ | yes | 20:15 |
| openstackgerrit | Drew Thorstensen (thorst) proposed openstack/nova-powervm: Support additional responses for qp's https://review.openstack.org/389339 | 20:19 |
| thorst_ | efried: I consider that one ^^ to be backport candidate to newton. | 20:21 |
| *** thorst_ has quit IRC | 20:22 | |
| *** mdrabe has quit IRC | 20:25 | |
| *** apearson has joined #openstack-powervm | 20:26 | |
| *** apearson has quit IRC | 20:29 | |
| *** apearson has joined #openstack-powervm | 20:29 | |
| kriskend_ | 23 minutes to deploy and boot 10 10G VMs | 20:49 |
| kriskend_ | use to take about 45 minutes with this controller and compute | 20:49 |
| adreznec | All successful? | 20:49 |
| kriskend_ | yeah but this is not on the system it use to fail on | 20:49 |
| kriskend_ | trying that next... | 20:49 |
| adreznec | Ah ok | 20:50 |
| *** apearson has quit IRC | 20:53 | |
| *** apearson has joined #openstack-powervm | 20:53 | |
| *** thorst_ has joined #openstack-powervm | 20:56 | |
| *** edmondsw has quit IRC | 21:02 | |
| efried | thorst_, we should never see 406. | 21:08 |
| thorst_ | o we do | 21:09 |
| efried | thorst_ Tell me how. 406 should be if we send the wrong Accept header for the response payload. If we're seeing it for any other reason, that's a REST bug. | 21:12 |
| adreznec | Why would we see a 406? That seems like we're sending a bad request | 21:13 |
| adreznec | Also do and should are very different things :P | 21:13 |
| efried | adreznec, agree on both points. | 21:14 |
| efried | If we're sending a bad request, we need to fix that bug. | 21:14 |
| efried | If *do* get a response we *should* not, REST needs to fix that bug. | 21:15 |
| efried | But we *should* not be band-aiding the community code because one or both of the above bugs exists. | 21:15 |
| adreznec | Right, unless REST is sending back 406s for things that really shouldn't be 406s then "handling" a 406 could lead to bad behavior down the road | 21:15 |
| *** tjakobs has quit IRC | 21:49 | |
| thorst_ | efried: http://pastebin.com/RKqfBChf | 21:51 |
| thorst_ | try on your own. I thought 406 may have applied due to the QuickProperty thing | 21:51 |
| efried | thorst_, okay, I can buy that the REST server is doing "the right thing" there - because the URI is invalid, it wanted to send us an XML error response with a 400 or 500; but we special-case 'quick' to accept application/json because (when we get the URI right) that's what we get back. | 21:54 |
| efried | So I suppose the same thing is happening with a 404. | 21:54 |
| efried | And what we really ought to be doing is accepting either application/json or application/atom+xml. | 21:55 |
| efried | Not sure if there's a way to get the Accept header to take more than one type... | 21:55 |
| *** thorst_ has quit IRC | 21:56 | |
| *** thorst_ has joined #openstack-powervm | 21:57 | |
| efried | thorst_, did you catch any of that? | 21:57 |
| *** dwayne has quit IRC | 22:00 | |
| efried | thorst_, I have a fix in pypowervm that we really should have made a looong time ago. | 22:04 |
| *** thorst_ has quit IRC | 22:05 | |
| *** thorst_ has joined #openstack-powervm | 22:20 | |
| thorst_ | efried: I think I did | 22:20 |
| thorst_ | and I don't think there is a way to say both... | 22:20 |
| efried | thorst_ - 10s | 22:21 |
| efried | thorst_, 4308 | 22:21 |
| efried | thorst_, I tested this live, and it does The Right Thing. I got 400 and 404 to come through as proper HttpErrors with 400/404 in 'em; but the real, valid qp response comes through when you get the URI right. | 22:22 |
| efried | We should've fixed this three years ago. | 22:22 |
| *** thorst_ has quit IRC | 22:24 | |
| *** kriskend_ has quit IRC | 22:34 | |
| *** k0da has quit IRC | 22:53 | |
| *** dwayne has joined #openstack-powervm | 23:10 | |
| *** thorst_ has joined #openstack-powervm | 23:24 | |
| *** thorst_ has quit IRC | 23:33 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!