| *** esberglu has joined #openstack-powervm | 00:14 | |
| *** kylek3h has joined #openstack-powervm | 00:15 | |
| *** kylek3h has quit IRC | 00:16 | |
| *** kylek3h has joined #openstack-powervm | 00:52 | |
| *** kylek3h has quit IRC | 00:53 | |
| *** edmondsw has joined #openstack-powervm | 00:54 | |
| *** edmondsw has quit IRC | 00:58 | |
| *** kylek3h has joined #openstack-powervm | 01:29 | |
| *** kylek3h has quit IRC | 01:29 | |
| *** esberglu has quit IRC | 01:43 | |
| *** thorst has joined #openstack-powervm | 01:49 | |
| *** thorst has quit IRC | 01:49 | |
| *** kylek3h has joined #openstack-powervm | 02:23 | |
| *** kylek3h_ has joined #openstack-powervm | 02:30 | |
| *** kylek3h has quit IRC | 02:33 | |
| *** esberglu has joined #openstack-powervm | 02:38 | |
| *** esberglu has quit IRC | 02:42 | |
| *** edmondsw has joined #openstack-powervm | 02:42 | |
| *** kylek3h_ has quit IRC | 02:45 | |
| *** edmondsw has quit IRC | 02:46 | |
| *** thorst has joined #openstack-powervm | 03:22 | |
| *** thorst has quit IRC | 03:30 | |
| *** chhavi has joined #openstack-powervm | 04:31 | |
| *** k0da has joined #openstack-powervm | 04:57 | |
| *** thorst has joined #openstack-powervm | 05:26 | |
| *** thorst has quit IRC | 05:33 | |
| *** k0da has quit IRC | 05:51 | |
| *** k0da has joined #openstack-powervm | 05:58 | |
| *** thorst has joined #openstack-powervm | 06:01 | |
| *** thorst has quit IRC | 06:05 | |
| *** esberglu has joined #openstack-powervm | 06:16 | |
| *** edmondsw has joined #openstack-powervm | 06:18 | |
| *** esberglu has quit IRC | 06:21 | |
| *** edmondsw has quit IRC | 06:23 | |
| *** miltonm has quit IRC | 06:31 | |
| *** k0da has quit IRC | 06:50 | |
| *** thorst has joined #openstack-powervm | 08:02 | |
| *** esberglu has joined #openstack-powervm | 08:04 | |
| *** edmondsw has joined #openstack-powervm | 08:06 | |
| *** thorst has quit IRC | 08:06 | |
| *** esberglu has quit IRC | 08:08 | |
| *** edmondsw has quit IRC | 08:11 | |
| *** k0da has joined #openstack-powervm | 08:36 | |
| *** esberglu has joined #openstack-powervm | 09:52 | |
| *** edmondsw has joined #openstack-powervm | 09:54 | |
| *** esberglu has quit IRC | 09:57 | |
| *** edmondsw has quit IRC | 09:58 | |
| *** thorst has joined #openstack-powervm | 10:03 | |
| *** thorst has quit IRC | 10:07 | |
| *** thorst has joined #openstack-powervm | 10:24 | |
| *** thorst has quit IRC | 10:28 | |
| *** thorst has joined #openstack-powervm | 10:47 | |
| *** thorst has quit IRC | 10:51 | |
| *** smatzek has joined #openstack-powervm | 10:57 | |
| *** esberglu has joined #openstack-powervm | 11:40 | |
| *** kylek3h has joined #openstack-powervm | 11:42 | |
| *** kylek3h has quit IRC | 11:42 | |
| *** edmondsw has joined #openstack-powervm | 11:42 | |
| *** esberglu has quit IRC | 11:44 | |
| *** edmondsw has quit IRC | 11:47 | |
| *** thorst has joined #openstack-powervm | 11:54 | |
| *** kylek3h has joined #openstack-powervm | 12:02 | |
| *** kylek3h has quit IRC | 12:11 | |
| *** chhavi has quit IRC | 12:32 | |
| *** edmondsw has joined #openstack-powervm | 12:33 | |
| *** esberglu has joined #openstack-powervm | 13:23 | |
| *** esberglu has quit IRC | 13:24 | |
| *** esberglu has joined #openstack-powervm | 13:24 | |
| *** apearson has joined #openstack-powervm | 13:27 | |
| *** k0da has quit IRC | 13:27 | |
| *** efried_zzz is now known as efried | 13:40 | |
| *** miltonm has joined #openstack-powervm | 13:57 | |
| *** kjw3 has joined #openstack-powervm | 14:11 | |
| *** apearson has quit IRC | 15:13 | |
| *** tjakobs_ has joined #openstack-powervm | 15:17 | |
| *** apearson has joined #openstack-powervm | 15:20 | |
| *** apearson has quit IRC | 15:38 | |
| *** edmondsw has quit IRC | 15:42 | |
| *** apearson has joined #openstack-powervm | 15:42 | |
| *** edmondsw has joined #openstack-powervm | 15:50 | |
| esberglu | edmondsw: Next time you try to get that system stacking make sure you run | 15:58 |
|---|---|---|
| esberglu | sudo ppc64_cpu --smt=off | 15:58 |
| esberglu | Before ./stack.sh | 15:58 |
| edmondsw | esberglu yeah, I saw y'all talking about that yesterday | 15:58 |
| edmondsw | t | 15:58 |
| edmondsw | tx | 15:58 |
| efried | should add that to the top of the etherpad | 16:02 |
| *** apearson has quit IRC | 16:26 | |
| *** apearson has joined #openstack-powervm | 16:27 | |
| *** apearson has quit IRC | 16:31 | |
| *** apearson has joined #openstack-powervm | 16:31 | |
| *** apearson has quit IRC | 16:32 | |
| *** apearson has joined #openstack-powervm | 17:25 | |
| *** apearson has quit IRC | 17:35 | |
| *** apearson has joined #openstack-powervm | 18:16 | |
| *** tjakobs_ has quit IRC | 19:42 | |
| *** tjakobs_ has joined #openstack-powervm | 19:42 | |
| *** smatzek has quit IRC | 20:15 | |
| thorst | esberglu: see #openstack-nova | 20:51 |
| thorst | edmondsw efried: ^^ | 20:51 |
| thorst | they noted that https://review.openstack.org/#/c/487246/ broke the PowerVM CI | 20:51 |
| thorst | efried: some of those logs look like we're just having trouble connecting to novalink? | 20:53 |
| efried | thorst That's pretty normal, I think, innit? | 20:56 |
| edmondsw | nice that they noticed | 20:56 |
| efried | It looks to me like they're just killing the compute process too soon. | 20:56 |
| thorst | efried: why wouldn't we be able to connect...like instantly? | 20:56 |
| efried | Yeah, dunno, should be able to. | 20:57 |
| efried | Where's esberglu ? | 20:57 |
| thorst | I don't know how we do the pypowervm thing now, but I didn't think it would be connecting to localhost. | 20:58 |
| thorst | and that's probably why it's hitting this thing | 20:58 |
| thorst | not sure where esberglu is atm | 20:58 |
| thorst | net I think to nova is, we'll be investigating. | 20:58 |
| esberglu | Catching up | 20:58 |
| thorst | esberglu: the question I'd have is - are we setting this on the CI? | 21:02 |
| thorst | https://github.com/powervm/pypowervm/blob/master/pypowervm/adapter.py#L171-L175 | 21:02 |
| efried | If we weren't, we never would have gotten started. Which we kinda clearly didn't in this case. | 21:03 |
| thorst | yeah. I haven't been paying super close attention to CI (obviously) but the statement that init_host takes ten minutes shocks me. | 21:04 |
| thorst | we spent tons of effort getting the CI time down... | 21:04 |
| thorst | it almost sounded like we tried localhost for X minutes and then flipped over to this other config? | 21:04 |
| esberglu | thorst: So up until now we have been stacking, then changing the environment file setting for the relevant devstack services | 21:04 |
| esberglu | And then restarting those services | 21:05 |
| esberglu | at which point the load with 'PYPOWERVM_SESSION_CONFIG' set | 21:05 |
| thorst | o. | 21:05 |
| thorst | well. | 21:05 |
| thorst | yikes? | 21:05 |
| esberglu | Problem is that the file we are modifying to do so is created during the stack | 21:05 |
| esberglu | So we can't just update the env. file before stacking because it doesn't exist | 21:06 |
| thorst | right right. Sec... | 21:06 |
| esberglu | We might be able to create the env. file prior to stacking, but idk if stacking will overwrite that | 21:06 |
| thorst | https://github.com/openstack/nova-powervm/blob/master/nova_powervm/virt/powervm/driver.py#L162 | 21:06 |
| thorst | seems to me that maybe we should make "conn_tries" a conf option | 21:06 |
| thorst | and you set that to like 2 or 3. | 21:07 |
| thorst | edmondsw efried: thoughts? | 21:07 |
| esberglu | I don't see how that helps with this | 21:07 |
| efried | sowait, esberglu walk me through the timing again? | 21:08 |
| efried | So before, they weren't waiting for compute to come up, and they just went right by that part, and then after stack was done, we would swap out the adapter config to enable remote, and then we would start up compute again? | 21:09 |
| efried | But now they're waiting for compute to come up, which it never will while we're trying at localhost. | 21:09 |
| edmondsw | I'm probably not much help at the moment... I have a splitting headache, can't think straight | 21:09 |
| efried | And when they time out, they kill the compute service and fail the stack. | 21:10 |
| edmondsw | I'm trying to keep up but I have to read everything 3 times to parse it | 21:10 |
| efried | edmondsw Go put a hot bag of rice over your eyes. We've got it covered. | 21:10 |
| edmondsw | tx | 21:11 |
| thorst | efried: I think it just died after it timed out | 21:12 |
| thorst | but we restarted it or something | 21:12 |
| thorst | I guess I'm not sure why we can't put down the session conf file before stacking? | 21:12 |
| thorst | I'm sure I'm missing something obvious there. | 21:12 |
| efried | ++ | 21:12 |
| esberglu | efried: Here comes my blurb. But yeah that's pretty much it | 21:13 |
| esberglu | We set PYPOWERVM_SESSION_CONFIG in /etc/environment which is needed for remote | 21:13 |
| esberglu | Then we stack. When stacking all of the devstack services get a service file (/etc/systemd/system/devstack@service.service) | 21:13 |
| esberglu | We need to set the EnvironmentFile var in that service file to /etc/environment so that it knows what session config to load | 21:13 |
| esberglu | Previously we were letting the stack finish, then updating the service file, then restarting the services at which point remote would work | 21:13 |
| esberglu | If I understand everything correctly the problem is that we need remote working for the stack to complete. | 21:13 |
| efried | Yup. | 21:13 |
| efried | I mean, they're reverting that change, but I imagine they're gonna try to bring it back in some form eventually. | 21:14 |
| thorst | esberglu: yeah, remote working pre-stack. Which I think means we need to somehow set the os.environ attribute pre-stack | 21:14 |
| esberglu | So we either need to have the service files in place before stacking (but they might get overwritten, haven't looked for this code yet) | 21:14 |
| thorst | efried: even if they revert, I'd imagine this would speed up our CI as is | 21:14 |
| thorst | maybe not...if its non-blocking | 21:14 |
| efried | thorst Well, I'm thinking we probably do the service file edit and restart n-cpu right after stack finishes, so it would be negligible. | 21:15 |
| efried | Compute was sitting there failing to start while stack finished, but should start right up when we did that restart. | 21:15 |
| esberglu | efried: Yeah that's accurate | 21:15 |
| efried | It'll shave a hair off, but not significant. | 21:15 |
| efried | But regardless, we want to get remote set up so it takes effect during stack. | 21:16 |
| efried | esberglu I think I would start by figuring out where those service files are generated in devstack, and see if we can hook in there. | 21:17 |
| efried | What's the path to the compute service file? | 21:17 |
| esberglu | /etc/systemd/system/devstack@n-cpu.service | 21:17 |
| efried | esberglu It looks like if you set up that file before stack, it won't get overwritten. | 21:19 |
| efried | Needs to be owned by root | 21:19 |
| efried | I'm looking at devstack/functions-common:1443 | 21:20 |
| efried | write_user_unit_file | 21:20 |
| esberglu | efried: Yep was just getting there. I'll test out a fix | 21:20 |
| thorst | for reference - how is the CI holding up? | 21:21 |
| thorst | are we passing when we should be passing? | 21:21 |
| efried | thorst It hasn't seemed very good to me. I've got four, five rechecks in a row on a couple changes. | 21:21 |
| thorst | esberglu: are we identifying the root causes on those? And importantly, pulling in the right people and forcing ownership to solve that :-) | 21:22 |
| esberglu | efried: thorst: Yeah failure rates are higher than normal. There was a bunch failing due to discover_hosts timeouts and bad marker LUs that are now fixed | 21:22 |
| esberglu | Tomorrow I was planning on diving into the timeouts that are hitting us | 21:22 |
| thorst | I need to head out, but if we need a renewed focus on CI...that's a tablestakes function. New development basically goes on pause while we solve it. | 21:23 |
| *** thorst has quit IRC | 21:25 | |
| *** thorst has joined #openstack-powervm | 21:26 | |
| *** thorst_ has joined #openstack-powervm | 21:27 | |
| *** thorst has quit IRC | 21:30 | |
| *** esberglu has quit IRC | 21:31 | |
| *** thorst_ has quit IRC | 21:32 | |
| *** apearson has quit IRC | 21:43 | |
| *** esberglu has joined #openstack-powervm | 21:45 | |
| *** thorst has joined #openstack-powervm | 21:45 | |
| *** esberglu has quit IRC | 21:49 | |
| *** thorst has quit IRC | 21:50 | |
| *** esberglu has joined #openstack-powervm | 21:51 | |
| *** edmondsw has quit IRC | 22:06 | |
| *** edmondsw has joined #openstack-powervm | 22:07 | |
| *** edmondsw has quit IRC | 22:12 | |
| *** thorst has joined #openstack-powervm | 22:13 | |
| *** thorst has quit IRC | 22:14 | |
| *** openstack has joined #openstack-powervm | 22:44 | |
| *** thorst has joined #openstack-powervm | 23:15 | |
| *** thorst has quit IRC | 23:21 | |
| *** edmondsw has joined #openstack-powervm | 23:37 | |
| *** edmondsw has quit IRC | 23:41 | |
| *** tjakobs_ has quit IRC | 23:44 | |
| *** thorst has joined #openstack-powervm | 23:46 | |
| *** thorst has quit IRC | 23:55 | |
| *** thorst has joined #openstack-powervm | 23:56 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!