| *** chas_ has quit IRC | 00:00 | |
| *** chas_ has joined #openstack-powervm | 00:02 | |
| *** thorst_ has joined #openstack-powervm | 00:06 | |
| *** chas_ has quit IRC | 00:07 | |
| *** esberglu has joined #openstack-powervm | 00:09 | |
| *** thorst_ has quit IRC | 00:16 | |
| *** thorst_ has joined #openstack-powervm | 00:17 | |
| *** esberglu has quit IRC | 00:19 | |
| *** thorst_ has quit IRC | 00:25 | |
| *** chas_ has joined #openstack-powervm | 00:36 | |
| *** chas_ has quit IRC | 00:40 | |
| *** jwcroppe has joined #openstack-powervm | 00:42 | |
| *** seroyer has joined #openstack-powervm | 00:58 | |
| *** smatzek has joined #openstack-powervm | 01:04 | |
| *** thorst_ has joined #openstack-powervm | 01:23 | |
| *** thorst_ has quit IRC | 01:26 | |
| *** thorst_ has joined #openstack-powervm | 01:27 | |
| *** chas_ has joined #openstack-powervm | 01:36 | |
| *** chas_ has quit IRC | 01:41 | |
| *** tblakes has quit IRC | 01:49 | |
| *** smatzek has quit IRC | 01:54 | |
| *** seroyer has quit IRC | 02:17 | |
| *** jwcroppe has quit IRC | 02:55 | |
| *** jwcroppe has joined #openstack-powervm | 02:55 | |
| openstackgerrit | Drew Thorstensen (thorst) proposed openstack/ceilometer-powervm: Update the init of the collector to take in a conf https://review.openstack.org/395918 | 03:13 |
|---|---|---|
| *** thorst_ has quit IRC | 04:08 | |
| *** thorst_ has joined #openstack-powervm | 04:08 | |
| *** thorst_ has quit IRC | 04:17 | |
| *** tjakobs has joined #openstack-powervm | 04:29 | |
| *** tjakobs has quit IRC | 04:33 | |
| *** chas_ has joined #openstack-powervm | 04:39 | |
| *** chas_ has quit IRC | 04:43 | |
| *** thorst_ has joined #openstack-powervm | 05:15 | |
| *** adi___ has quit IRC | 05:20 | |
| *** adi___ has joined #openstack-powervm | 05:21 | |
| *** thorst_ has quit IRC | 05:22 | |
| *** k0da has joined #openstack-powervm | 05:40 | |
| *** thorst_ has joined #openstack-powervm | 06:21 | |
| *** thorst_ has quit IRC | 06:27 | |
| *** chas_ has joined #openstack-powervm | 06:40 | |
| *** chas_ has quit IRC | 06:45 | |
| *** esberglu has joined #openstack-powervm | 07:27 | |
| *** esberglu has quit IRC | 07:31 | |
| *** openstackgerrit has quit IRC | 07:48 | |
| *** openstackgerrit has joined #openstack-powervm | 07:48 | |
| *** madhaviy has joined #openstack-powervm | 08:10 | |
| *** esberglu has joined #openstack-powervm | 08:26 | |
| *** thorst_ has joined #openstack-powervm | 08:28 | |
| *** esberglu has quit IRC | 08:31 | |
| *** thorst_ has quit IRC | 08:37 | |
| *** chas_ has joined #openstack-powervm | 08:42 | |
| *** chas_ has quit IRC | 08:47 | |
| *** chas_ has joined #openstack-powervm | 08:53 | |
| *** esberglu has joined #openstack-powervm | 09:21 | |
| *** esberglu has quit IRC | 09:26 | |
| *** thorst_ has joined #openstack-powervm | 09:35 | |
| *** thorst_ has quit IRC | 09:42 | |
| *** k0da has quit IRC | 10:10 | |
| *** madhaviy has quit IRC | 10:37 | |
| *** madhaviy has joined #openstack-powervm | 10:38 | |
| *** thorst_ has joined #openstack-powervm | 10:40 | |
| *** thorst_ has quit IRC | 10:47 | |
| *** esberglu has joined #openstack-powervm | 11:06 | |
| *** smatzek has joined #openstack-powervm | 11:07 | |
| *** esberglu has quit IRC | 11:10 | |
| *** edmondsw has joined #openstack-powervm | 11:26 | |
| *** madhaviy has quit IRC | 11:33 | |
| *** edmondsw has quit IRC | 11:34 | |
| *** edmondsw has joined #openstack-powervm | 11:35 | |
| *** thorst_ has joined #openstack-powervm | 11:45 | |
| *** thorst_ has quit IRC | 11:53 | |
| *** esberglu has joined #openstack-powervm | 12:21 | |
| *** esberglu has quit IRC | 12:25 | |
| *** thorst_ has joined #openstack-powervm | 12:47 | |
| *** thorst_ has quit IRC | 12:47 | |
| *** thorst_ has joined #openstack-powervm | 12:47 | |
| *** kylek3h has quit IRC | 12:50 | |
| *** esberglu has joined #openstack-powervm | 13:00 | |
| thorst_ | adreznec: I added you to 395560. The z team added me due to discussions at the summit, but I think you did all the infra setup and probably can do a better technical evaluation of that than I | 13:04 |
| *** svenkat has joined #openstack-powervm | 13:18 | |
| *** svenkat_ has joined #openstack-powervm | 13:21 | |
| *** svenkat has quit IRC | 13:25 | |
| *** svenkat_ is now known as svenkat | 13:25 | |
| adreznec | thorst_: Ok, I'll take a look later | 13:29 |
| *** kylek3h has joined #openstack-powervm | 13:29 | |
| adreznec | #startmeeting PowerVM CI Meeting | 13:30 |
| openstack | Meeting started Thu Nov 10 13:30:34 2016 UTC and is due to finish in 60 minutes. The chair is adreznec. Information about MeetBot at http://wiki.debian.org/MeetBot. | 13:30 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 13:30 |
| openstack | The meeting name has been set to 'powervm_ci_meeting' | 13:30 |
| *** wangqwsh has joined #openstack-powervm | 13:30 | |
| adreznec | All right. Roll call? | 13:30 |
| thorst_ | o/ | 13:30 |
| wangqwsh | start? | 13:31 |
| esberglu | yep | 13:31 |
| adreznec | All right, looks like we have enough to get started | 13:31 |
| adreznec | #topic Current Status | 13:31 |
| adreznec | thorst_ or esberglu, want to kick things off here? | 13:32 |
| efried | o/ | 13:32 |
| thorst_ | so I think my (and really efried's) contribution is that the fifo_pipo is almost done | 13:32 |
| thorst_ | I had a working version yesterday (no UT yet), then efried did a rev | 13:33 |
| thorst_ | I'm not sure if we've validated that rev yet. efried? | 13:33 |
| efried | No chance it's working now ;-) | 13:33 |
| thorst_ | I have full faith! | 13:33 |
| *** dwayne_ has quit IRC | 13:33 | |
| thorst_ | but want proof as well | 13:33 |
| thorst_ | :-) | 13:33 |
| efried | Can you try it out with whatever setup you used yesterday thorst_? | 13:33 |
| thorst_ | efried: yep - can do | 13:34 |
| efried | coo | 13:34 |
| adreznec | AWESOME | 13:34 |
| adreznec | Oops | 13:34 |
| thorst_ | then we'll ask esberglu to take it in the CI while we work UT and what not | 13:34 |
| esberglu | I loaded the devstack CI with the version Drew put up yesterday | 13:34 |
| thorst_ | well it is awesome. | 13:34 |
| thorst_ | oooo | 13:35 |
| adreznec | Not quite that awesome, but still good to get it tested | 13:35 |
| thorst_ | how'd that go? | 13:35 |
| esberglu | The tempest runs are still just looping looking for a matching LU with checksum xxx | 13:35 |
| esberglu | http://184.172.12.213/02/396002/1/silent/neutron-pvm-dsvm-tempest-full/3aed2d8/ | 13:35 |
| efried | Right, I was going to look at that this morning. | 13:35 |
| thorst_ | bah | 13:36 |
| esberglu | So its not even getting to the tempest stuff at this point | 13:36 |
| efried | uh, is that the right log? | 13:37 |
| efried | oh, is it the os_ci_tempest.sh that's doing that? | 13:37 |
| thorst_ | efried: right...we pre-seed the image in the SSP | 13:37 |
| esberglu | Yeah | 13:37 |
| adreznec | yikes | 13:37 |
| adreznec | super early | 13:37 |
| efried | oh, we've seen this before. | 13:38 |
| thorst_ | #action thorst to validate efried's rev of remote patch | 13:38 |
| efried | #action efried to diag/debug "looking for LU with checksum" loop | 13:38 |
| adreznec | We have? Hmm, must be forgetting | 13:39 |
| adreznec | Ok | 13:39 |
| esberglu | #action esberglu to put remote patch in CI when ready | 13:39 |
| adreznec | What other status do we have? wangqwsh I saw an email from you with boot speed issues? | 13:39 |
| esberglu | Yay I have an action this time | 13:39 |
| wangqwsh | yes | 13:40 |
| wangqwsh | not sure the reason | 13:40 |
| adreznec | How slow are we talking here? | 13:40 |
| thorst_ | wangqwsh: how slow of a boot are we talking/ | 13:40 |
| thorst_ | lol | 13:40 |
| adreznec | echo echo echo | 13:40 |
| wangqwsh | more than 3 hours | 13:40 |
| wangqwsh | they are still in deleting | 13:41 |
| thorst_ | whoa. | 13:41 |
| adreznec | Wow, that's a lot worse than I expected | 13:41 |
| esberglu | Ahh I bet they hit the marker LU thing | 13:41 |
| thorst_ | I thought we'd be talking a couple mins here. | 13:41 |
| adreznec | Ok, and this is on neo14? | 13:41 |
| thorst_ | but its deleting? | 13:41 |
| wangqwsh | ye, neo14 | 13:41 |
| wangqwsh | yes | 13:41 |
| wangqwsh | i am trying to deleting them. | 13:41 |
| wangqwsh | because of spawning error. | 13:42 |
| efried | Okay, the thing esberglu is seeing is because there's a stale marker LU somewhere that's blocking the "seed LPAR" creation. | 13:42 |
| esberglu | Yeah. I will go in and delete that marker lu now | 13:43 |
| efried | And by "stale" I mean "who knows?" | 13:43 |
| adreznec | wangqwsh: I take it they never finished booting? | 13:43 |
| efried | partf59362e8image_base_os_2f282b84e7e608d5852449ed940bfc51 | 13:43 |
| wangqwsh | yes. | 13:43 |
| adreznec | Ok | 13:43 |
| wangqwsh | never.. | 13:43 |
| adreznec | Yeah it sounds like they got blocked | 13:43 |
| efried | So people. | 13:44 |
| efried | Is it possible for the compute process to die completely without cleaning up? | 13:44 |
| efried | Possible of course theoretically - but in the CI environment | 13:44 |
| efried | Cause Occam's razor says that's the most likely way we end up in this stale-marker-LU state. | 13:45 |
| thorst_ | efried: sure - we hit the timeout limit...it'll leave it around | 13:45 |
| thorst_ | but we have a cleanup process | 13:45 |
| thorst_ | now whether or not that cleanup process is cleaning out the markers... | 13:46 |
| efried | Right - the 'finally' block will still get executed. | 13:46 |
| efried | Yes, 'finally' cleans up the marker LU. | 13:46 |
| thorst_ | no no... | 13:46 |
| thorst_ | I mean, we can just straight kill from zuul the run | 13:46 |
| thorst_ | shut down the VM | 13:46 |
| thorst_ | mid process | 13:46 |
| thorst_ | could I guess leave the marker around... | 13:47 |
| adreznec | Yeah | 13:47 |
| efried | oh, for sure. | 13:47 |
| thorst_ | I'd think that'd be rare | 13:47 |
| adreznec | An edge case, but definitely possible | 13:47 |
| efried | Well, I can assure you our cleanup process doesn't delete marker LUs. | 13:47 |
| efried | At that level. | 13:47 |
| efried | Cause how would we know which ones were safe to delete? | 13:48 |
| thorst_ | efried: agree. | 13:48 |
| efried | But it seems we're seeing this very frequently. | 13:49 |
| thorst_ | though I think that the CI could have a job that says 'hey...if we have a marker lu that's been around for more than 60 minutes...its time to delete it' | 13:49 |
| thorst_ | that would be 100% specific to the CI though | 13:49 |
| efried | yeah | 13:49 |
| thorst_ | because we know that nothing in there should take more than an hour. | 13:49 |
| efried | But would that mask if we had a real bug that causes a real upload hang? | 13:49 |
| thorst_ | but I'd rather try to find the root cause (I don't think we have yet) before we do that | 13:50 |
| thorst_ | efried: yep...exactly | 13:50 |
| efried | no, we definitely have not identified the root cause. | 13:50 |
| thorst_ | I wonder if we should add to the CI job something that prints out all the LUs before it runs | 13:50 |
| thorst_ | before the devstack even... | 13:50 |
| esberglu | I can add that in it would be super easy | 13:51 |
| adreznec | You're thinking for debug? | 13:51 |
| thorst_ | adreznec: yah | 13:51 |
| adreznec | Seems like a fair place to start | 13:51 |
| thorst_ | just to know, was it there before we even started | 13:51 |
| efried | And at the end of a run, grep the log for certain messages pertaining to marker LUs. | 13:51 |
| adreznec | Is there anything else we'd want to add to that debug info | 13:51 |
| thorst_ | and *maybe* to the pypowervm bit | 13:51 |
| thorst_ | as part of a warning log, when we detect a marker lu...but I thought we had that. | 13:51 |
| thorst_ | down in pypowervm itself... | 13:52 |
| efried | 2016-11-10 02:53:12.099 INFO pypowervm.tasks.cluster_ssp [req-dcce7f01-a1fd-470b-a174-9fc51f9a4a05 admin admin] Waiting for in-progress upload(s) to complete. Marker LU(s): ['partf59362e8image_base_os_2f282b84e7e608d5852449ed940bfc51'] | 13:52 |
| efried | I'm not seeing pypowervm DEBUG turned on in this log, btw, esberglu. | 13:52 |
| thorst_ | yeah... | 13:52 |
| efried | I thought we made that change. | 13:52 |
| thorst_ | efried: we had it off because of all the gorp. Now that gorp is gone...did we make that change in CI yet? | 13:53 |
| adreznec | I thought so... | 13:53 |
| esberglu | We did. I wonder if I didn't pull down the newest neo-os-ci last redeploy | 13:53 |
| efried | Merged on 11/3 | 13:54 |
| efried | 4420 | 13:54 |
| thorst_ | #action esberglu to figure out where the pypowervm debug logs are | 13:54 |
| esberglu | I definitely did not pull down the newest... | 13:54 |
| adreznec | That sounds bad | 13:54 |
| esberglu | If I go into jenkins and manually kill jobs would that leave the marker LUs around? | 13:55 |
| adreznec | Does zuul delete the VM if the job ends? | 13:55 |
| adreznec | Have to step away for a couple minutes, please continue without me | 13:56 |
| efried | esberglu, I think the answer is yes | 13:56 |
| thorst_ | esberglu: well, if its stuck in an upload | 13:56 |
| thorst_ | efried: I wonder if we could/should get the VM's name in the marker lu | 13:57 |
| efried | Anything that aborts the compute process. | 13:57 |
| esberglu | Ahh then I'm probably to blame. I do that sometimes when I need to redeploy but don't want to wait for all the runs to finish | 13:57 |
| thorst_ | esberglu: when we redeploy though, I thought we cleaned out the SSP? | 13:57 |
| esberglu | Or just redeploy the management playbook which also just kills stuff | 13:57 |
| esberglu | Yeah redeploys do | 13:57 |
| esberglu | clean out the ssp | 13:57 |
| efried | And since upload takes "a while", you can easily catch a thread that's in the middle of one. | 13:57 |
| thorst_ | maybe we need to add something to the management playbook to clean out the SSPs. | 13:58 |
| thorst_ | but efried, thoughts on adding a VM name? | 13:58 |
| efried | Right - I had considered it, but it's kinda tough with name length restrictions. | 13:58 |
| thorst_ | yeah | 13:58 |
| efried | I can make it work if we think it's critical. | 13:59 |
| thorst_ | not sure...lets see how these other actions pan out? | 13:59 |
| efried | Though I was going to make it some part of the MTMS | 13:59 |
| *** apearson has joined #openstack-powervm | 13:59 | |
| efried | The VM name doesn't really help us much. | 13:59 |
| thorst_ | host + lpar id... | 13:59 |
| thorst_ | that'd be a start | 13:59 |
| thorst_ | changing it though has implications for in the field...though I think this is low use code at the moment | 13:59 |
| efried | But yeah, if esberglu is interrupting runs, that's going to be the culprit most of the time. | 14:00 |
| thorst_ | esberglu: can we add something in the management playbook that cleans out the compute node markers (or ssps)? | 14:00 |
| thorst_ | I mean, honestly, we want to clear out all of the computes | 14:01 |
| thorst_ | but just not redeploy. | 14:01 |
| esberglu | So the computes get cleaned out when the compute playbook is run | 14:01 |
| esberglu | But not in the management playbook | 14:01 |
| efried | This is something that covers all the nodes sharing a given SSP? | 14:01 |
| thorst_ | efried: it'd be all nodes in the given cloud | 14:01 |
| efried | Cool. | 14:02 |
| efried | Are we close to done? I've got another meeting. | 14:02 |
| thorst_ | esberglu: yeah, its weird...because its the management playbook. But I think you're running that just to build new images. | 14:02 |
| thorst_ | so, I think you want to pull the clean of the compute nodes into that...or make a new playbook altogether for it | 14:02 |
| esberglu | Yeah exactly | 14:02 |
| thorst_ | efried: yeah, me too | 14:02 |
| esberglu | #action esberglu find a way to clean compute nodes from management playbook | 14:03 |
| esberglu | The only other thing I had was for wangqwsh | 14:04 |
| esberglu | The read only filesystem was fixed | 14:04 |
| thorst_ | is that an assertion or a question? | 14:04 |
| wangqwsh | esberglu: cool | 14:05 |
| esberglu | And I redeployed the staging env. with the newest versions of both OSA CI patches and the latest local2remote. I'm still getting stuck trying to build the wheels | 14:05 |
| esberglu | It looks like you got past all of the bootstrap stuff and to the tempest part? | 14:05 |
| wangqwsh | yes, at the tempest | 14:06 |
| wangqwsh | you can use this review: | 14:06 |
| wangqwsh | let me find it | 14:07 |
| wangqwsh | http://morpheus.rch.stglabs.ibm.com/#/c/4033/ | 14:08 |
| esberglu | That review is already deployed in the environment | 14:08 |
| wangqwsh | i add some variables and scripts for osa | 14:08 |
| esberglu | Yeah I have the latest version of that deployed | 14:09 |
| esberglu | I will send an email with more info about what I am hitting, too much info for irc | 14:10 |
| esberglu | #endmeeting | 14:10 |
| adreznec | Ok | 14:10 |
| wangqwsh | ok | 14:10 |
| adreznec | Sounds like we're done then | 14:10 |
| adreznec | Thanks all | 14:10 |
| adreznec | #endmeeting | 14:10 |
| wangqwsh | i can take a look at it | 14:10 |
| thorst_ | good job team | 14:10 |
| openstack | Meeting ended Thu Nov 10 14:10:36 2016 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 14:10 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/powervm_ci_meeting/2016/powervm_ci_meeting.2016-11-10-13.30.html | 14:10 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/powervm_ci_meeting/2016/powervm_ci_meeting.2016-11-10-13.30.txt | 14:10 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/powervm_ci_meeting/2016/powervm_ci_meeting.2016-11-10-13.30.log.html | 14:10 |
| adreznec | P.S. only the host can do that for the first 60 minutes esberglu | 14:10 |
| esberglu | Haha I though everyone else was gone | 14:11 |
| *** tblakes has joined #openstack-powervm | 14:11 | |
| adreznec | Oh man | 14:12 |
| adreznec | ocata-1 is next week | 14:12 |
| adreznec | Crazy | 14:12 |
| thorst_ | adreznec: yeah...you see os-vif is actually going in | 14:12 |
| thorst_ | and being used | 14:12 |
| adreznec | Yeah... and I don't think we've really done much testing with it. Not that it shouldn't just work | 14:13 |
| *** dwayne_ has joined #openstack-powervm | 14:17 | |
| *** mdrabe has joined #openstack-powervm | 14:21 | |
| *** smatzek has quit IRC | 14:23 | |
| thorst_ | efried: this on your radar? https://bugs.launchpad.net/pypowervm/+bug/1637243 | 14:24 |
| openstack | Launchpad bug 1637243 in pypowervm "Failure in TaskFlow returns large WrappedFailure exception to the user" [Undecided,New] - Assigned to Eric Fried (efried) | 14:24 |
| thorst_ | I'm getting pinged on it. | 14:24 |
| *** esberglu has quit IRC | 14:25 | |
| *** esberglu has joined #openstack-powervm | 14:38 | |
| *** seroyer has joined #openstack-powervm | 14:44 | |
| efried | thorst_, yeah, I was working on it yesterday in between the upload business. I should have a change set up for review today, hopefully. | 15:04 |
| *** seroyer has quit IRC | 15:09 | |
| *** smatzek has joined #openstack-powervm | 15:11 | |
| esberglu | thorst_: efried: adreznec: I put up change 4486 for the management playbook to also clean out the vms | 15:13 |
| *** wangqwsh has quit IRC | 15:20 | |
| *** tjakobs has joined #openstack-powervm | 15:25 | |
| thorst_ | efried: your patch seems to be working...I just need to verify the clean up bits. Otherwise though, looking solid | 15:28 |
| efried | nice. | 15:28 |
| efried | I put up a minor edit for the vopt thing. | 15:28 |
| efried | You wouldn't have hit the bug in the remote env, but would've in local. | 15:28 |
| thorst_ | k | 15:30 |
| thorst_ | this actually is nice...it uses next to no CPU | 15:30 |
| thorst_ | I really like it | 15:30 |
| *** adreznec has quit IRC | 15:34 | |
| *** adreznec has joined #openstack-powervm | 15:37 | |
| *** kriskend has joined #openstack-powervm | 15:38 | |
| *** seroyer has joined #openstack-powervm | 16:45 | |
| *** seroyer has quit IRC | 16:48 | |
| *** mdrabe has quit IRC | 17:00 | |
| thorst_ | efried: change looks solid | 17:09 |
| thorst_ | esberglu: you should probably integrate that change while we then fix up UT | 17:10 |
| esberglu | Will do | 17:10 |
| *** mdrabe has joined #openstack-powervm | 17:10 | |
| *** dwayne_ has quit IRC | 17:34 | |
| *** dwayne_ has joined #openstack-powervm | 17:36 | |
| thorst_ | efried: small q on 4487 | 17:46 |
| thorst_ | efried: also this one https://review.openstack.org/#/c/393801/ | 17:54 |
| *** k0da has joined #openstack-powervm | 18:17 | |
| efried | thorst_, 4487 - did you want me to replace the tabs with spaces, or remove them entirely, or...? | 19:08 |
| thorst_ | well, it just meant the first line of the new exception would have a tab | 19:12 |
| thorst_ | and the rest of the lines wouldn't | 19:12 |
| thorst_ | and that seemed...weird to me. | 19:12 |
| efried | thorst_, I don't think so. | 19:24 |
| efried | self.msg_fmt = _("FeedTask %(ft_name)s experienced multiple " | 19:24 |
| efried | "exceptions:\n\t%(concat_msgs)s") | 19:24 |
| efried | concat_msgs = '\n\t'.join([fail.exception_str | 19:24 |
| efried | So like | 19:25 |
| efried | FeedTask foo experienced multiple exceptions\n\tFirst message\n\tSecond message\n\tThird message. | 19:25 |
| efried | Here's what the formatted string in the test case looks like: | 19:27 |
| efried | FeedTask ft experienced multiple exceptions: | 19:27 |
| efried | this is an exception on lpar1! | 19:27 |
| efried | this is an exception on lpar2! | 19:27 |
| thorst_ | what happens to the exceptions stack? Just hidden? | 19:41 |
| efried | thorst_, I'm logging the wrapped exceptions via LOG.exception before I raise the MultipleExceptionsInFeedTask one (which is the one that has the concatenated message). | 19:45 |
| thorst_ | got it | 19:45 |
| thorst_ | OK - excellent | 19:45 |
| efried | And the MultipleExceptionsInFeedTask exception itself also has a wrapped_failure attribute | 19:45 |
| efried | which contains the original WrappedFailure object, which can be dissected to get right at the stack traces, messages, exception objects, etc. | 19:45 |
| efried | ...if the caller wanted to trap it via try/except. | 19:46 |
| efried | thorst_ Let's talk about 4475 too if you have a sec. | 19:47 |
| thorst_ | let me look it up | 19:48 |
| *** chmod666org has joined #openstack-powervm | 19:48 | |
| *** openstackgerrit has quit IRC | 19:48 | |
| thorst_ | k | 19:48 |
| thorst_ | fire it up | 19:48 |
| thorst_ | I do not have a suggestion. In my head I was thinking a string that resolved to a class name and have the adapter load that | 19:48 |
| thorst_ | but that's gross. | 19:48 |
| *** openstackgerrit has joined #openstack-powervm | 19:49 | |
| efried | thorst_, that IS gross. | 19:55 |
| thorst_ | agree. | 19:55 |
| efried | But how do you feel about me just parsing the XML "by hand" in the vios_busy_retry_helper? | 19:55 |
| efried | It's not too many LOC, fairly straightforward. | 19:55 |
| efried | pity not to use the wrapper class, but if you think the benefits outweigh... | 19:56 |
| thorst_ | efried: to me...I think I'm OK with that... | 19:56 |
| thorst_ | I hate having imports in weird places | 19:56 |
| efried | I just don't want you to think that *any* helper will have this kind of import problem. | 19:56 |
| thorst_ | right. | 19:56 |
| efried | Although tbh, they kinda will. | 19:56 |
| thorst_ | right. | 19:56 |
| efried | Cause what other kind of thing would we be looking for in a Response? | 19:57 |
| thorst_ | not sure. | 19:57 |
| efried | ...other than something we have a wrapper for | 19:57 |
| efried | So anyway, those are the options. | 19:57 |
| efried | I can't think of a reasonable way to get the import jankitude into the Adapter. | 19:57 |
| thorst_ | I can just shut up about the imports | 19:57 |
| thorst_ | I'm being kinda a princess on it | 19:57 |
| efried | It'll actually compile okay with an unknown symbol in the kwarg value - but as soon as you run it, you better have that symbol resolved. | 19:57 |
| thorst_ | and if the other way isn't better...no reason for you to do it for me to just whine about that. | 19:58 |
| efried | It's only better because it avoids the deferred imports. | 19:58 |
| thorst_ | I can just resolve that this is the best solution... | 19:58 |
| thorst_ | maybe a comment as to why we did this awful thing | 19:58 |
| thorst_ | and be done at that | 19:58 |
| thorst_ | (if not already there) | 19:58 |
| efried | I won't advocate or veto either one. I'd almost prefer the direct-parsing solution. Your call. | 19:58 |
| efried | btw, this isn't a problem for consumers of the adapter/helpers, since they'll be importing both - no circularity. As proven by community code. | 19:59 |
| thorst_ | I defer to you. | 20:00 |
| efried | thorst_, what about 4487? We good there at this point? | 20:07 |
| efried | PM took a rather strict interpretation of our conversation yesterday and put a due date of 11am today on it. | 20:08 |
| thorst_ | need a bit...chatting with someone on sdn | 20:08 |
| *** chas_ has quit IRC | 20:18 | |
| *** kylek3h has quit IRC | 20:28 | |
| *** k0da has quit IRC | 20:33 | |
| *** k0da has joined #openstack-powervm | 20:34 | |
| *** thorst_ has quit IRC | 20:51 | |
| *** thorst_ has joined #openstack-powervm | 20:51 | |
| *** thorst_ has quit IRC | 20:56 | |
| *** seroyer has joined #openstack-powervm | 21:02 | |
| *** seroyer has quit IRC | 21:09 | |
| *** chas_ has joined #openstack-powervm | 21:19 | |
| *** smatzek has quit IRC | 21:21 | |
| *** chas_ has quit IRC | 21:23 | |
| *** kylek3h has joined #openstack-powervm | 21:29 | |
| kriskend | just figured out how to get the Guru Meditation log for nova compute. It is very helpful for debugging hang conditions, because it shows you the stack trace for nova threads | 21:30 |
| kriskend | run | 21:30 |
| kriskend | sudo kill -USR2 32934 | 21:30 |
| kriskend | for the nova process | 21:30 |
| kriskend | in devstack, the output should go to nova compute log | 21:31 |
| *** thorst_ has joined #openstack-powervm | 21:46 | |
| esberglu | efried: thorst: At some point while the prod. CI was down, master devstack broke for us. | 21:56 |
| thorst_ | lol | 21:57 |
| thorst_ | undercloud or AIO? | 21:57 |
| esberglu | AIO. We use stable/newton for undercloud now | 21:57 |
| thorst_ | boo... | 21:58 |
| *** smatzek has joined #openstack-powervm | 22:01 | |
| thorst_ | yeah...we'll need to debug. I think efried just fought through a bunch of this... | 22:02 |
| *** svenkat has quit IRC | 22:02 | |
| *** smatzek has quit IRC | 22:02 | |
| thorst_ | but he may have ultimately rolled back to stable/newton | 22:02 |
| efried | Yeah, I gave up on master. | 22:05 |
| *** apearson has quit IRC | 22:06 | |
| esberglu | Crap. There are some stable/newton runs going so we can see what happens there | 22:09 |
| esberglu | Not good | 22:11 |
| esberglu | http://184.172.12.213/41/396341/1/check/nova-powervm-pvm-dsvm-tempest-full/c85509c/ | 22:11 |
| *** smatzek has joined #openstack-powervm | 22:14 | |
| efried | esberglu: Have we had a successful CI run with the latest 4458? | 22:14 |
| efried | ever | 22:14 |
| efried | 2016-11-10 16:06:41.682 ERROR oslo.messaging._drivers.impl_rabbit [-] [5b362dca-295c-41c1-9c66-65308f610e9b] AMQP server on 172.16.0.1:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds. Client port: 38080 | 22:15 |
| efried | 2016-11-10 16:06:41.735 ERROR oslo.messaging._drivers.impl_rabbit [-] [d9b8fdfd-7084-403e-9ef9-9a6f0cb5601f] AMQP server 172.16.0.1:5672 closed the connection. Check login credentials: Socket closed | 22:15 |
| efried | 2016-11-10 16:06:41.775 ERROR oslo_messaging.rpc.server [-] Can not acknowledge message. Skip processing | 22:15 |
| efried | This breaks the pipe and kills the fifo_reader. | 22:15 |
| esberglu | Nope. Whatever broke master devstack came in while I was redeploying with the latest. That was the first run to go through with it in | 22:16 |
| efried | o. So whatever thorst_ said was looking good earlier... wasn't CI. | 22:17 |
| esberglu | Correct | 22:17 |
| efried | thorst_, esberglu: Looks like we need to ignore-and-retry on EINTR. Which is... weird. | 22:21 |
| efried | https://www.python.org/dev/peps/pep-0475/ | 22:21 |
| efried | Will happen automagically in py3 | 22:21 |
| efried | ye gods, this would get ugly fast. | 22:25 |
| efried | If we have to trap and ignore EINTR on every bloody open() and read() and write() and close()... | 22:25 |
| *** seroyer has joined #openstack-powervm | 22:29 | |
| efried | thorst_, esberglu: Uploaded a new patch set to 4458 which should work around this EINTR thing - which oughtta be intermittent anyway. I don't like it. | 22:31 |
| *** seroyer has quit IRC | 22:32 | |
| esberglu | Alright. I have to go take care of some lease stuff. And then gerrit is down so I will do that 1st thing tomorrow | 22:35 |
| *** esberglu has quit IRC | 22:36 | |
| *** esberglu has joined #openstack-powervm | 22:37 | |
| thorst_ | efried esberglu: OK | 22:41 |
| *** esberglu has quit IRC | 22:41 | |
| *** kylek3h has quit IRC | 22:46 | |
| *** esberglu has joined #openstack-powervm | 22:50 | |
| *** esberglu has quit IRC | 22:54 | |
| *** chas_ has joined #openstack-powervm | 22:58 | |
| *** chas_ has quit IRC | 23:02 | |
| *** chas_ has joined #openstack-powervm | 23:04 | |
| *** smatzek has quit IRC | 23:07 | |
| *** mdrabe has quit IRC | 23:08 | |
| *** chas_ has quit IRC | 23:09 | |
| *** kriskend has quit IRC | 23:10 | |
| *** tblakes has quit IRC | 23:11 | |
| *** kylek3h has joined #openstack-powervm | 23:15 | |
| *** k0da has quit IRC | 23:17 | |
| *** tjakobs has quit IRC | 23:19 | |
| *** esberglu has joined #openstack-powervm | 23:29 | |
| *** esberglu has quit IRC | 23:29 | |
| *** esberglu has joined #openstack-powervm | 23:30 | |
| *** esberglu has quit IRC | 23:34 | |
| *** seroyer has joined #openstack-powervm | 23:43 | |
| *** seroyer has quit IRC | 23:45 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!