*** dwayne has joined #openstack-powervm | 00:01 | |
*** k0da has quit IRC | 00:03 | |
*** kylek3h has joined #openstack-powervm | 00:36 | |
*** esberglu has quit IRC | 01:21 | |
*** esberglu has joined #openstack-powervm | 01:52 | |
*** esberglu has quit IRC | 01:56 | |
*** esberglu has joined #openstack-powervm | 03:05 | |
*** esberglu has quit IRC | 03:25 | |
*** esberglu has joined #openstack-powervm | 03:25 | |
*** efried has quit IRC | 03:28 | |
*** esberglu has quit IRC | 03:30 | |
*** efried has joined #openstack-powervm | 03:41 | |
*** tlian has quit IRC | 03:59 | |
*** thorst_ has quit IRC | 04:00 | |
*** thorst_ has joined #openstack-powervm | 05:00 | |
*** tjakobs has joined #openstack-powervm | 05:03 | |
*** thorst_ has quit IRC | 05:04 | |
*** tjakobs has quit IRC | 05:09 | |
*** thorst_ has joined #openstack-powervm | 05:31 | |
*** thorst_ has quit IRC | 05:35 | |
*** thorst_ has joined #openstack-powervm | 07:31 | |
*** thorst_ has quit IRC | 07:36 | |
*** esberglu has joined #openstack-powervm | 07:50 | |
*** esberglu has quit IRC | 07:55 | |
*** k0da has joined #openstack-powervm | 08:18 | |
*** thorst_ has joined #openstack-powervm | 09:32 | |
*** thorst_ has quit IRC | 09:38 | |
*** esberglu has joined #openstack-powervm | 09:39 | |
*** esberglu has quit IRC | 09:44 | |
*** efried has quit IRC | 10:06 | |
*** edmondsw has joined #openstack-powervm | 10:30 | |
*** thorst_ has joined #openstack-powervm | 10:34 | |
*** edmondsw has quit IRC | 10:35 | |
*** esberglu has joined #openstack-powervm | 10:39 | |
*** thorst_ has quit IRC | 10:39 | |
*** esberglu has quit IRC | 10:44 | |
*** esberglu has joined #openstack-powervm | 12:24 | |
*** esberglu has quit IRC | 12:28 | |
*** dwayne has quit IRC | 12:45 | |
*** thorst_ has joined #openstack-powervm | 12:45 | |
*** edmondsw has joined #openstack-powervm | 12:46 | |
*** edmondsw has quit IRC | 12:46 | |
*** edmondsw has joined #openstack-powervm | 12:46 | |
*** tblakes has joined #openstack-powervm | 13:09 | |
*** efried has joined #openstack-powervm | 13:19 | |
*** newbie|3 has joined #openstack-powervm | 13:32 | |
*** dwayne has joined #openstack-powervm | 13:47 | |
*** apearson has joined #openstack-powervm | 14:03 | |
*** tlian has joined #openstack-powervm | 14:07 | |
*** esberglu has joined #openstack-powervm | 14:13 | |
*** mdrabe has joined #openstack-powervm | 14:13 | |
*** esberglu has quit IRC | 14:13 | |
*** esberglu has joined #openstack-powervm | 14:14 | |
*** esberglu_ has joined #openstack-powervm | 14:18 | |
*** esberglu has quit IRC | 14:21 | |
thorst_ | esberglu_: lots of failures in CI. How we holding up? | 14:25 |
---|---|---|
*** apearson has quit IRC | 14:28 | |
*** apearson has joined #openstack-powervm | 14:30 | |
*** esberglu has joined #openstack-powervm | 14:35 | |
*** esberglu_ has quit IRC | 14:39 | |
*** kjw3 has joined #openstack-powervm | 14:43 | |
esberglu | thorst_: Here are the failures since I turned it back on yesterday. (not including bad patches / our in tree runs) | 14:56 |
esberglu | http://paste.openstack.org/show/597197/ | 14:56 |
esberglu | I'm thinking we may still have an issue with the power off stuff | 14:56 |
thorst_ | ugh...really? | 15:01 |
thorst_ | efried: do you have any time to chase those? | 15:01 |
esberglu | A lot of the rebuild tests may be explained by that plus there are some that are still explicitly failing to power off | 15:01 |
adreznec | http://nooooooooooooooo.com/ | 15:01 |
efried | We're talking about the in-tree CI failing on power-off? | 15:02 |
thorst_ | oot | 15:02 |
efried | hmph. | 15:02 |
thorst_ | heh | 15:02 |
efried | Okay, yeah, I can take a look. | 15:02 |
esberglu | No the out of tree | 15:02 |
thorst_ | thx dude | 15:03 |
esberglu | I think some of the others may be bad patches or not supported. | 15:04 |
esberglu | p.s. I put the results from the in-tree runs at the bottom of that paste | 15:05 |
esberglu | Not all of them are from the latest patch, but just to get an idea where it's at | 15:06 |
esberglu | Seeing about 5-7 failures a run so far there | 15:07 |
esberglu | I need to dig in and see which are unsupported and if there are real failures | 15:08 |
efried | So hold on | 15:08 |
efried | what is it I'm looking at? | 15:08 |
efried | Specifically just those two power-off failures? | 15:08 |
efried | thorst_ esberglu ^^ | 15:09 |
thorst_ | yeah, lets see if there are power off failures...and then figure out why... | 15:09 |
thorst_ | other random failures, consider noise for the time being | 15:09 |
thorst_ | or a separate thing to chase | 15:09 |
efried | okay. Two power-off failures on just two runs in two separate test cases (though both rebuild) seems like random noise. | 15:10 |
efried | But I'll look. | 15:10 |
esberglu | Those two yeah. But I think some of the rebuilds might be hitting the same thing | 15:10 |
efried | So I'll reiterate what I said yesterday: I expect the in-tree to hit those power-off failures, because pypowervm 1.0.0.4 doesn't have the latest fix. | 15:11 |
thorst_ | eww... | 15:12 |
thorst_ | so...how do we get rid of that noise? | 15:12 |
thorst_ | patch it in as part of the run...like we did with the local2remote | 15:12 |
thorst_ | until we can bump to 1.0.0.5? | 15:12 |
efried | Well, we could disable the new timeout code in-tree. | 15:12 |
efried | Or that, but that seems a tad dishonest. | 15:12 |
thorst_ | I don't think dishonest. | 15:13 |
thorst_ | I think it will be pushed through by the time we're actually core reviewed | 15:13 |
thorst_ | I just don't want to be seeing failures for a month | 15:13 |
efried | Shrug, okay. So esberglu - wherever you're pulling in 2382, pull in 4761 as well. | 15:14 |
efried | Ugh, the CI env doesn't just conveniently patch those in from git, does it? It has, like, a copy of the diff in a text file checked in separately? | 15:15 |
esberglu | Yeah... | 15:15 |
esberglu | But that doesn't explain the power off issues for oot | 15:16 |
efried | Nono, separate issue. | 15:16 |
esberglu | Okay just making sure we were on the same page | 15:16 |
efried | btw, esberglu, not sure how you're doing your data collection, but I'm seeing two power-off failures in http://184.172.12.213/34/412634/24/check/nova-out-of-tree-pvm/88e7cc8/powervm_os_ci.html | 15:16 |
esberglu | That list definitely isn't perfect. Just gives a rough idea of what we are seeing | 15:18 |
esberglu | I want to make some sort of tool to automate this, I just haven't looked into it yet | 15:18 |
efried | yuh. | 15:19 |
*** apearson has quit IRC | 15:21 | |
esberglu | Maybe jenkins has a plugin already | 15:21 |
*** kriskend has joined #openstack-powervm | 15:31 | |
*** apearson has joined #openstack-powervm | 15:32 | |
*** tjakobs has joined #openstack-powervm | 15:32 | |
efried | esberglu thorst_ - So from what I'm seeing, this is trying the power-off unforced with a 60s timeout, and then when that fails, it's *not* retrying. | 15:39 |
efried | The code path that I believe leads to this is when the LPAR doesn't have RMC. | 15:40 |
thorst_ | well then my pypowervm 'fix' wasn't good | 15:40 |
efried | Yeah, your pypowervm fix was fine - we just weren't hitting it. | 15:40 |
thorst_ | hrmmm... | 15:40 |
thorst_ | I didn't think about RMC. | 15:40 |
efried | I'm questioning whether we should declare VMPowerOffFailure if we timeout that path (no RMC). | 15:40 |
efried | Or just ignore it. | 15:41 |
efried | Although... | 15:41 |
efried | We don't add the immediate flag when RMC is dead - just change from osshutdown to shutdown. | 15:41 |
efried | Sooo... I wonder if we should force-retry when that guy fails. | 15:42 |
efried | Add another noodle to that spaghetti code. | 15:42 |
efried | This is really getting to the point where I'm gonna refactor the sonofab!tch | 15:42 |
efried | It's out of control. | 15:42 |
thorst_ | efried: I'd support that. But the net is, if RMC is down, an osshutdown doesn't work, then a shutdown doesn't work...rip the damn power out. | 15:43 |
thorst_ | I think it was built the way it was because someone wanted to keep LoC down. | 15:43 |
efried | support which? The force-retry in that case, the refactor, or both? | 15:43 |
thorst_ | all | 15:43 |
efried | Okay. | 15:43 |
efried | Yeah, at this point I believe refactoring will actually reduce the LOC. | 15:44 |
thorst_ | yep | 15:44 |
efried | Thinking recursive. | 15:44 |
efried | instead of loop | 15:44 |
efried | Anyway, I'll do it in two steps. | 15:44 |
efried | First change set I'll add the force flag and retry in that scenario. | 15:44 |
efried | Then I'll do a refactor next. | 15:44 |
efried | Off I go. | 15:44 |
*** newbie|3 has quit IRC | 15:54 | |
thorst_ | cool beans | 15:55 |
*** jpasqualetto has joined #openstack-powervm | 15:56 | |
efried | thorst_ 4767 | 16:04 |
efried | esberglu: What does it take to pick up ^^? Does it need to merge, do things need to be rebuilt/redeployed, ...? | 16:04 |
esberglu | Management node redeploy (~2 hours) | 16:06 |
thorst_ | and getting it into the VMs | 16:06 |
thorst_ | so you have to flush ready nodes? | 16:06 |
esberglu | Yeah | 16:06 |
efried | gross | 16:06 |
efried | Sho would be nice if we had a way to try these pypowervm patches in the CI before they merge. | 16:07 |
esberglu | I could kick off an image build and flush the nodes manually, but it ends up taking just as long | 16:07 |
thorst_ | staging env... | 16:08 |
thorst_ | but...in this case... | 16:08 |
esberglu | I could do a manual run on the staging env | 16:09 |
thorst_ | I'd say go prod | 16:10 |
thorst_ | for throughput | 16:10 |
esberglu | Fine by me | 16:11 |
*** mdrabe has quit IRC | 16:23 | |
*** mdrabe has joined #openstack-powervm | 16:29 | |
*** k0da has quit IRC | 16:35 | |
esberglu | efried: thorst_: Sorry I wasn't clear before. 4767 still needs to be merged to test on the production environment | 16:48 |
efried | Yeah. | 16:48 |
* efried grumbles | 16:48 | |
esberglu | Then patch in both 4761 and 4767 for in-tree | 16:50 |
*** kjw3 has quit IRC | 17:00 | |
efried | esberglu Yeah. As well as local2remote. | 17:01 |
esberglu | And abort fifo | 17:01 |
*** kriskend has quit IRC | 17:28 | |
*** tjakobs has quit IRC | 17:28 | |
*** burgerk has quit IRC | 17:33 | |
*** kylek3h is now known as kylek3h_away | 17:34 | |
*** esberglu has quit IRC | 17:55 | |
*** esberglu has joined #openstack-powervm | 17:56 | |
*** esberglu_ has joined #openstack-powervm | 17:57 | |
*** esberglu has quit IRC | 18:00 | |
*** esberglu_ has quit IRC | 18:01 | |
*** esberglu has joined #openstack-powervm | 18:09 | |
*** esberglu has quit IRC | 18:09 | |
*** esberglu has joined #openstack-powervm | 18:09 | |
*** apearson has quit IRC | 18:29 | |
*** apearson has joined #openstack-powervm | 18:32 | |
*** apearson has quit IRC | 18:44 | |
*** apearson has joined #openstack-powervm | 18:46 | |
*** apearson has quit IRC | 18:50 | |
*** apearson has joined #openstack-powervm | 19:12 | |
esberglu | Reminder that tomorrow is the 1st release candidate for *-powervm projects. This means that a stable/ocata branch will be created. | 19:25 |
*** efried has quit IRC | 19:25 | |
esberglu | Any further changes for the ocata release will have to be made on the master branch and backported to the stable/ocata | 19:25 |
*** apearson has quit IRC | 19:28 | |
thorst_ | wooo! | 19:30 |
*** efried has joined #openstack-powervm | 19:48 | |
*** jpasqualetto has quit IRC | 19:48 | |
*** k0da has joined #openstack-powervm | 19:55 | |
*** apearson has joined #openstack-powervm | 20:02 | |
*** jpasqualetto has joined #openstack-powervm | 20:05 | |
*** jpasqualetto has quit IRC | 20:35 | |
*** edmondsw has quit IRC | 21:44 | |
*** edmondsw has joined #openstack-powervm | 21:53 | |
*** thorst_ has quit IRC | 22:09 | |
*** thorst_ has joined #openstack-powervm | 22:16 | |
*** thorst_ has quit IRC | 22:20 | |
*** thorst_ has joined #openstack-powervm | 22:40 | |
*** edmondsw has quit IRC | 22:41 | |
*** thorst_ has quit IRC | 22:44 | |
*** tblakes has quit IRC | 22:46 | |
*** tlian has quit IRC | 22:51 | |
*** tjakobs has joined #openstack-powervm | 22:51 | |
*** tlian has joined #openstack-powervm | 22:58 | |
*** esberglu has quit IRC | 23:01 | |
*** dwayne has quit IRC | 23:02 | |
*** tlian2 has joined #openstack-powervm | 23:10 | |
*** mdrabe has quit IRC | 23:10 | |
*** tlian has quit IRC | 23:13 | |
*** tlian2 has quit IRC | 23:15 | |
*** k0da has quit IRC | 23:17 | |
*** tlian has joined #openstack-powervm | 23:24 | |
*** apearson has quit IRC | 23:36 | |
*** tjakobs has quit IRC | 23:51 | |
*** dwayne has joined #openstack-powervm | 23:57 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!