| *** dwayne has joined #openstack-powervm | 00:01 | |
| *** k0da has quit IRC | 00:03 | |
| *** kylek3h has joined #openstack-powervm | 00:36 | |
| *** esberglu has quit IRC | 01:21 | |
| *** esberglu has joined #openstack-powervm | 01:52 | |
| *** esberglu has quit IRC | 01:56 | |
| *** esberglu has joined #openstack-powervm | 03:05 | |
| *** esberglu has quit IRC | 03:25 | |
| *** esberglu has joined #openstack-powervm | 03:25 | |
| *** efried has quit IRC | 03:28 | |
| *** esberglu has quit IRC | 03:30 | |
| *** efried has joined #openstack-powervm | 03:41 | |
| *** tlian has quit IRC | 03:59 | |
| *** thorst_ has quit IRC | 04:00 | |
| *** thorst_ has joined #openstack-powervm | 05:00 | |
| *** tjakobs has joined #openstack-powervm | 05:03 | |
| *** thorst_ has quit IRC | 05:04 | |
| *** tjakobs has quit IRC | 05:09 | |
| *** thorst_ has joined #openstack-powervm | 05:31 | |
| *** thorst_ has quit IRC | 05:35 | |
| *** thorst_ has joined #openstack-powervm | 07:31 | |
| *** thorst_ has quit IRC | 07:36 | |
| *** esberglu has joined #openstack-powervm | 07:50 | |
| *** esberglu has quit IRC | 07:55 | |
| *** k0da has joined #openstack-powervm | 08:18 | |
| *** thorst_ has joined #openstack-powervm | 09:32 | |
| *** thorst_ has quit IRC | 09:38 | |
| *** esberglu has joined #openstack-powervm | 09:39 | |
| *** esberglu has quit IRC | 09:44 | |
| *** efried has quit IRC | 10:06 | |
| *** edmondsw has joined #openstack-powervm | 10:30 | |
| *** thorst_ has joined #openstack-powervm | 10:34 | |
| *** edmondsw has quit IRC | 10:35 | |
| *** esberglu has joined #openstack-powervm | 10:39 | |
| *** thorst_ has quit IRC | 10:39 | |
| *** esberglu has quit IRC | 10:44 | |
| *** esberglu has joined #openstack-powervm | 12:24 | |
| *** esberglu has quit IRC | 12:28 | |
| *** dwayne has quit IRC | 12:45 | |
| *** thorst_ has joined #openstack-powervm | 12:45 | |
| *** edmondsw has joined #openstack-powervm | 12:46 | |
| *** edmondsw has quit IRC | 12:46 | |
| *** edmondsw has joined #openstack-powervm | 12:46 | |
| *** tblakes has joined #openstack-powervm | 13:09 | |
| *** efried has joined #openstack-powervm | 13:19 | |
| *** newbie|3 has joined #openstack-powervm | 13:32 | |
| *** dwayne has joined #openstack-powervm | 13:47 | |
| *** apearson has joined #openstack-powervm | 14:03 | |
| *** tlian has joined #openstack-powervm | 14:07 | |
| *** esberglu has joined #openstack-powervm | 14:13 | |
| *** mdrabe has joined #openstack-powervm | 14:13 | |
| *** esberglu has quit IRC | 14:13 | |
| *** esberglu has joined #openstack-powervm | 14:14 | |
| *** esberglu_ has joined #openstack-powervm | 14:18 | |
| *** esberglu has quit IRC | 14:21 | |
| thorst_ | esberglu_: lots of failures in CI. How we holding up? | 14:25 |
|---|---|---|
| *** apearson has quit IRC | 14:28 | |
| *** apearson has joined #openstack-powervm | 14:30 | |
| *** esberglu has joined #openstack-powervm | 14:35 | |
| *** esberglu_ has quit IRC | 14:39 | |
| *** kjw3 has joined #openstack-powervm | 14:43 | |
| esberglu | thorst_: Here are the failures since I turned it back on yesterday. (not including bad patches / our in tree runs) | 14:56 |
| esberglu | http://paste.openstack.org/show/597197/ | 14:56 |
| esberglu | I'm thinking we may still have an issue with the power off stuff | 14:56 |
| thorst_ | ugh...really? | 15:01 |
| thorst_ | efried: do you have any time to chase those? | 15:01 |
| esberglu | A lot of the rebuild tests may be explained by that plus there are some that are still explicitly failing to power off | 15:01 |
| adreznec | http://nooooooooooooooo.com/ | 15:01 |
| efried | We're talking about the in-tree CI failing on power-off? | 15:02 |
| thorst_ | oot | 15:02 |
| efried | hmph. | 15:02 |
| thorst_ | heh | 15:02 |
| efried | Okay, yeah, I can take a look. | 15:02 |
| esberglu | No the out of tree | 15:02 |
| thorst_ | thx dude | 15:03 |
| esberglu | I think some of the others may be bad patches or not supported. | 15:04 |
| esberglu | p.s. I put the results from the in-tree runs at the bottom of that paste | 15:05 |
| esberglu | Not all of them are from the latest patch, but just to get an idea where it's at | 15:06 |
| esberglu | Seeing about 5-7 failures a run so far there | 15:07 |
| esberglu | I need to dig in and see which are unsupported and if there are real failures | 15:08 |
| efried | So hold on | 15:08 |
| efried | what is it I'm looking at? | 15:08 |
| efried | Specifically just those two power-off failures? | 15:08 |
| efried | thorst_ esberglu ^^ | 15:09 |
| thorst_ | yeah, lets see if there are power off failures...and then figure out why... | 15:09 |
| thorst_ | other random failures, consider noise for the time being | 15:09 |
| thorst_ | or a separate thing to chase | 15:09 |
| efried | okay. Two power-off failures on just two runs in two separate test cases (though both rebuild) seems like random noise. | 15:10 |
| efried | But I'll look. | 15:10 |
| esberglu | Those two yeah. But I think some of the rebuilds might be hitting the same thing | 15:10 |
| efried | So I'll reiterate what I said yesterday: I expect the in-tree to hit those power-off failures, because pypowervm 1.0.0.4 doesn't have the latest fix. | 15:11 |
| thorst_ | eww... | 15:12 |
| thorst_ | so...how do we get rid of that noise? | 15:12 |
| thorst_ | patch it in as part of the run...like we did with the local2remote | 15:12 |
| thorst_ | until we can bump to 1.0.0.5? | 15:12 |
| efried | Well, we could disable the new timeout code in-tree. | 15:12 |
| efried | Or that, but that seems a tad dishonest. | 15:12 |
| thorst_ | I don't think dishonest. | 15:13 |
| thorst_ | I think it will be pushed through by the time we're actually core reviewed | 15:13 |
| thorst_ | I just don't want to be seeing failures for a month | 15:13 |
| efried | Shrug, okay. So esberglu - wherever you're pulling in 2382, pull in 4761 as well. | 15:14 |
| efried | Ugh, the CI env doesn't just conveniently patch those in from git, does it? It has, like, a copy of the diff in a text file checked in separately? | 15:15 |
| esberglu | Yeah... | 15:15 |
| esberglu | But that doesn't explain the power off issues for oot | 15:16 |
| efried | Nono, separate issue. | 15:16 |
| esberglu | Okay just making sure we were on the same page | 15:16 |
| efried | btw, esberglu, not sure how you're doing your data collection, but I'm seeing two power-off failures in http://184.172.12.213/34/412634/24/check/nova-out-of-tree-pvm/88e7cc8/powervm_os_ci.html | 15:16 |
| esberglu | That list definitely isn't perfect. Just gives a rough idea of what we are seeing | 15:18 |
| esberglu | I want to make some sort of tool to automate this, I just haven't looked into it yet | 15:18 |
| efried | yuh. | 15:19 |
| *** apearson has quit IRC | 15:21 | |
| esberglu | Maybe jenkins has a plugin already | 15:21 |
| *** kriskend has joined #openstack-powervm | 15:31 | |
| *** apearson has joined #openstack-powervm | 15:32 | |
| *** tjakobs has joined #openstack-powervm | 15:32 | |
| efried | esberglu thorst_ - So from what I'm seeing, this is trying the power-off unforced with a 60s timeout, and then when that fails, it's *not* retrying. | 15:39 |
| efried | The code path that I believe leads to this is when the LPAR doesn't have RMC. | 15:40 |
| thorst_ | well then my pypowervm 'fix' wasn't good | 15:40 |
| efried | Yeah, your pypowervm fix was fine - we just weren't hitting it. | 15:40 |
| thorst_ | hrmmm... | 15:40 |
| thorst_ | I didn't think about RMC. | 15:40 |
| efried | I'm questioning whether we should declare VMPowerOffFailure if we timeout that path (no RMC). | 15:40 |
| efried | Or just ignore it. | 15:41 |
| efried | Although... | 15:41 |
| efried | We don't add the immediate flag when RMC is dead - just change from osshutdown to shutdown. | 15:41 |
| efried | Sooo... I wonder if we should force-retry when that guy fails. | 15:42 |
| efried | Add another noodle to that spaghetti code. | 15:42 |
| efried | This is really getting to the point where I'm gonna refactor the sonofab!tch | 15:42 |
| efried | It's out of control. | 15:42 |
| thorst_ | efried: I'd support that. But the net is, if RMC is down, an osshutdown doesn't work, then a shutdown doesn't work...rip the damn power out. | 15:43 |
| thorst_ | I think it was built the way it was because someone wanted to keep LoC down. | 15:43 |
| efried | support which? The force-retry in that case, the refactor, or both? | 15:43 |
| thorst_ | all | 15:43 |
| efried | Okay. | 15:43 |
| efried | Yeah, at this point I believe refactoring will actually reduce the LOC. | 15:44 |
| thorst_ | yep | 15:44 |
| efried | Thinking recursive. | 15:44 |
| efried | instead of loop | 15:44 |
| efried | Anyway, I'll do it in two steps. | 15:44 |
| efried | First change set I'll add the force flag and retry in that scenario. | 15:44 |
| efried | Then I'll do a refactor next. | 15:44 |
| efried | Off I go. | 15:44 |
| *** newbie|3 has quit IRC | 15:54 | |
| thorst_ | cool beans | 15:55 |
| *** jpasqualetto has joined #openstack-powervm | 15:56 | |
| efried | thorst_ 4767 | 16:04 |
| efried | esberglu: What does it take to pick up ^^? Does it need to merge, do things need to be rebuilt/redeployed, ...? | 16:04 |
| esberglu | Management node redeploy (~2 hours) | 16:06 |
| thorst_ | and getting it into the VMs | 16:06 |
| thorst_ | so you have to flush ready nodes? | 16:06 |
| esberglu | Yeah | 16:06 |
| efried | gross | 16:06 |
| efried | Sho would be nice if we had a way to try these pypowervm patches in the CI before they merge. | 16:07 |
| esberglu | I could kick off an image build and flush the nodes manually, but it ends up taking just as long | 16:07 |
| thorst_ | staging env... | 16:08 |
| thorst_ | but...in this case... | 16:08 |
| esberglu | I could do a manual run on the staging env | 16:09 |
| thorst_ | I'd say go prod | 16:10 |
| thorst_ | for throughput | 16:10 |
| esberglu | Fine by me | 16:11 |
| *** mdrabe has quit IRC | 16:23 | |
| *** mdrabe has joined #openstack-powervm | 16:29 | |
| *** k0da has quit IRC | 16:35 | |
| esberglu | efried: thorst_: Sorry I wasn't clear before. 4767 still needs to be merged to test on the production environment | 16:48 |
| efried | Yeah. | 16:48 |
| * efried grumbles | 16:48 | |
| esberglu | Then patch in both 4761 and 4767 for in-tree | 16:50 |
| *** kjw3 has quit IRC | 17:00 | |
| efried | esberglu Yeah. As well as local2remote. | 17:01 |
| esberglu | And abort fifo | 17:01 |
| *** kriskend has quit IRC | 17:28 | |
| *** tjakobs has quit IRC | 17:28 | |
| *** burgerk has quit IRC | 17:33 | |
| *** kylek3h is now known as kylek3h_away | 17:34 | |
| *** esberglu has quit IRC | 17:55 | |
| *** esberglu has joined #openstack-powervm | 17:56 | |
| *** esberglu_ has joined #openstack-powervm | 17:57 | |
| *** esberglu has quit IRC | 18:00 | |
| *** esberglu_ has quit IRC | 18:01 | |
| *** esberglu has joined #openstack-powervm | 18:09 | |
| *** esberglu has quit IRC | 18:09 | |
| *** esberglu has joined #openstack-powervm | 18:09 | |
| *** apearson has quit IRC | 18:29 | |
| *** apearson has joined #openstack-powervm | 18:32 | |
| *** apearson has quit IRC | 18:44 | |
| *** apearson has joined #openstack-powervm | 18:46 | |
| *** apearson has quit IRC | 18:50 | |
| *** apearson has joined #openstack-powervm | 19:12 | |
| esberglu | Reminder that tomorrow is the 1st release candidate for *-powervm projects. This means that a stable/ocata branch will be created. | 19:25 |
| *** efried has quit IRC | 19:25 | |
| esberglu | Any further changes for the ocata release will have to be made on the master branch and backported to the stable/ocata | 19:25 |
| *** apearson has quit IRC | 19:28 | |
| thorst_ | wooo! | 19:30 |
| *** efried has joined #openstack-powervm | 19:48 | |
| *** jpasqualetto has quit IRC | 19:48 | |
| *** k0da has joined #openstack-powervm | 19:55 | |
| *** apearson has joined #openstack-powervm | 20:02 | |
| *** jpasqualetto has joined #openstack-powervm | 20:05 | |
| *** jpasqualetto has quit IRC | 20:35 | |
| *** edmondsw has quit IRC | 21:44 | |
| *** edmondsw has joined #openstack-powervm | 21:53 | |
| *** thorst_ has quit IRC | 22:09 | |
| *** thorst_ has joined #openstack-powervm | 22:16 | |
| *** thorst_ has quit IRC | 22:20 | |
| *** thorst_ has joined #openstack-powervm | 22:40 | |
| *** edmondsw has quit IRC | 22:41 | |
| *** thorst_ has quit IRC | 22:44 | |
| *** tblakes has quit IRC | 22:46 | |
| *** tlian has quit IRC | 22:51 | |
| *** tjakobs has joined #openstack-powervm | 22:51 | |
| *** tlian has joined #openstack-powervm | 22:58 | |
| *** esberglu has quit IRC | 23:01 | |
| *** dwayne has quit IRC | 23:02 | |
| *** tlian2 has joined #openstack-powervm | 23:10 | |
| *** mdrabe has quit IRC | 23:10 | |
| *** tlian has quit IRC | 23:13 | |
| *** tlian2 has quit IRC | 23:15 | |
| *** k0da has quit IRC | 23:17 | |
| *** tlian has joined #openstack-powervm | 23:24 | |
| *** apearson has quit IRC | 23:36 | |
| *** tjakobs has quit IRC | 23:51 | |
| *** dwayne has joined #openstack-powervm | 23:57 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!