| *** thorst_afk has joined #openstack-powervm | 00:11 | |
| *** svenkat has joined #openstack-powervm | 00:14 | |
| *** svenkat has quit IRC | 00:49 | |
| *** thorst_afk has quit IRC | 00:52 | |
| *** svenkat has joined #openstack-powervm | 01:05 | |
| *** svenkat has quit IRC | 01:30 | |
| *** edmondsw has joined #openstack-powervm | 01:42 | |
| *** thorst_afk has joined #openstack-powervm | 01:42 | |
| *** edmondsw has quit IRC | 01:46 | |
| *** apearson has joined #openstack-powervm | 02:10 | |
| *** thorst_a_ has joined #openstack-powervm | 02:18 | |
| *** thorst_a_ has quit IRC | 02:18 | |
| *** thorst_afk has quit IRC | 02:20 | |
| *** edmondsw has joined #openstack-powervm | 03:30 | |
| *** edmondsw has quit IRC | 03:34 | |
| *** apearson has quit IRC | 03:34 | |
| *** http_GK1wmSU has joined #openstack-powervm | 03:48 | |
| *** http_GK1wmSU has left #openstack-powervm | 03:49 | |
| *** thorst_afk has joined #openstack-powervm | 04:19 | |
| *** edmondsw has joined #openstack-powervm | 04:31 | |
| *** thorst_afk has quit IRC | 04:33 | |
| *** chhavi has joined #openstack-powervm | 04:35 | |
| *** edmondsw has quit IRC | 04:36 | |
| *** thorst_afk has joined #openstack-powervm | 06:01 | |
| *** thorst_afk has quit IRC | 06:05 | |
| *** edmondsw has joined #openstack-powervm | 06:19 | |
| *** edmondsw has quit IRC | 06:24 | |
| *** thorst_afk has joined #openstack-powervm | 08:02 | |
| *** thorst_afk has quit IRC | 08:06 | |
| *** thorst_afk has joined #openstack-powervm | 09:03 | |
| *** thorst_afk has quit IRC | 09:07 | |
| *** edmondsw has joined #openstack-powervm | 09:55 | |
| *** edmondsw has quit IRC | 10:00 | |
| *** wilkm has joined #openstack-powervm | 10:08 | |
| *** openstackgerrit has quit IRC | 10:18 | |
| *** edmondsw has joined #openstack-powervm | 10:36 | |
| *** edmondsw has quit IRC | 10:40 | |
| *** thorst_afk has joined #openstack-powervm | 11:04 | |
| *** thorst_afk has quit IRC | 11:08 | |
| *** smatzek has joined #openstack-powervm | 11:14 | |
| *** svenkat has joined #openstack-powervm | 11:42 | |
| *** svenkat has quit IRC | 11:46 | |
| *** svenkat_ has joined #openstack-powervm | 11:46 | |
| *** edmondsw has joined #openstack-powervm | 11:47 | |
| *** thorst_afk has joined #openstack-powervm | 11:51 | |
| *** thorst_afk has quit IRC | 11:52 | |
| *** edmondsw has quit IRC | 11:52 | |
| *** thorst has joined #openstack-powervm | 12:06 | |
| *** edmondsw has joined #openstack-powervm | 12:23 | |
| *** kylek3h has joined #openstack-powervm | 12:54 | |
| *** apearson has joined #openstack-powervm | 13:20 | |
| *** esberglu has joined #openstack-powervm | 13:25 | |
| esberglu | edmondsw: thorst: efried_zzz: Last night I was looking at the prod. controller and there were a bunch of old openstack processes running | 13:35 |
|---|---|---|
| esberglu | I don't think they were getting cleaned up properly when unstacking during the deploy | 13:36 |
| esberglu | To avoid this in the future I am just going to run ./clean.sh every deployment. It will increase deployment time slightly but we have been doing that less and less | 13:37 |
| thorst | seems reasonable | 13:49 |
| *** efried_zzz is now known as fried_rice | 13:49 | |
| *** wilkm has quit IRC | 13:59 | |
| *** apearson has quit IRC | 14:10 | |
| *** smatzek has quit IRC | 14:13 | |
| *** smatzek has joined #openstack-powervm | 14:13 | |
| *** apearson has joined #openstack-powervm | 14:30 | |
| esberglu | fried_rice: Got time to talk through timeout issues today? | 14:32 |
| fried_rice | esberglu Yeah, but gimme 30. | 14:33 |
| esberglu | fried_rice: Sure | 14:33 |
| esberglu | fried_rice: edmondsw: thorst: 5648 should fix the cell issues in CI | 15:33 |
| edmondsw | sweet | 15:34 |
| *** apearson has quit IRC | 15:36 | |
| edmondsw | esberglu hmm... "after the stack". So we remove that once sdague's change goes in? | 15:39 |
| edmondsw | or are we still possibly broken after his change goes in? | 15:39 |
| edmondsw | because "after the stack" is too late to mean anything once his change goes in, so it's one of those 2 | 15:40 |
| esberglu | edmondsw: We could remove this after his change goes in if his change fixes the issue | 15:42 |
| esberglu | But it doesn't hurt to leave | 15:43 |
| esberglu | it | 15:43 |
| esberglu | If the hosts are already discovered while stacking this won't do anything | 15:43 |
| esberglu | If they don't get discovered while stacking it gives us one more chance to discover them | 15:43 |
| edmondsw | esberglu my point is that it no longer gives us one more chance after his change goes in, because the stack would fail, right? | 15:44 |
| esberglu | edmondsw: Let me take another look at the change, but I didn't think that it would fail the stack | 15:46 |
| edmondsw | esberglu maybe I misunderstood/misremember... but if not then why did it break anyone? | 15:47 |
| esberglu | edmondsw: Because discover_hosts was being run too soon I believe | 15:48 |
| esberglu | So they are now waiting until the compute service is actually up to run it | 15:48 |
| edmondsw | esberglu oh, right | 15:50 |
| edmondsw | yeah, I don't see anything in wait_on_compute that would actually bail on the stack | 15:51 |
| esberglu | So sdagues change _should_ fix us | 15:52 |
| esberglu | But no reason to remove the second discover_hosts after that merges because it doesn't hurt anything and could potentially save a run that still fails the stack's discover_hosts | 15:52 |
| edmondsw | esberglu so something was blowing up with that `openstack compute service list` call because of the missing EnvironmentFile, but we've fixed that, and should be good now? | 15:53 |
| esberglu | Yep. Previously we weren't setting up remote until after the stack which was causing that to fail | 15:54 |
| esberglu | Now we do the remote setup before stacking | 15:54 |
| edmondsw | k, left comments | 15:56 |
| edmondsw | I agree with fried_rice, we need to figure out why this is even necessary and what we can do about it | 15:56 |
| fried_rice | edmondsw esberglu To make sure everyone understands: sdague's *original* change would *fail* the stack if the compute service didn't come alive within the timeout in mid-stack. | 15:59 |
| fried_rice | With his *new* patch, it logs a warning, but proceeds anyway. | 15:59 |
| *** openstackgerrit has joined #openstack-powervm | 16:00 | |
| openstackgerrit | Merged openstack/nova-powervm master: force python2.7 for tox pep8 https://review.openstack.org/490014 | 16:00 |
| fried_rice | So his change won't in fact fix us, other than by giving our compute service a little longer to come up and be discoverable. | 16:02 |
| fried_rice | In a situation where our compute service doesn't come up in that 60s window, it is pretty unlikely to come up at all, so the new discover_hosts at the end is likewise pretty unlikely to change anything. | 16:03 |
| fried_rice | So to summarize: | 16:03 |
| fried_rice | Until sdague's change merges, esberglu's change makes sense. After sdague's change merges, it's almost certainly a no-op, though it doesn't hurt anything. | 16:04 |
| fried_rice | esberglu I'm ready to look into timeouts now. | 16:04 |
| edmondsw | esberglu fried_rice should we merge this, then, or just carry it as a patch until sdague's change goes in? | 16:04 |
| edmondsw | I kinda lean toward the latter | 16:04 |
| fried_rice | esberglu Can we "carry it as a patch"? | 16:05 |
| edmondsw | oh, maybe we can't... | 16:05 |
| fried_rice | (esberglu In true programmer style, I hope you multiplied my time estimate by three.) | 16:05 |
| fried_rice | edmondsw esberglu I'd be okay with adding a TODO to remove it once (link to sdague's patch) merges. | 16:05 |
| edmondsw | +1 | 16:06 |
| esberglu | fried_rice: Was just about to take lunch. Dig in this afternoon or would you rather do it now? No big deal either way for me | 16:12 |
| fried_rice | esberglu Whenever you like. I got plenty to keep me busy. | 16:12 |
| esberglu | fried_rice: I have a quick thing now and then I'll take lunch | 16:13 |
| fried_rice | ndo | 16:13 |
| fried_rice | sho | 16:13 |
| esberglu | http://184.172.12.213/22/490722/1/check/nova-in-tree-pvm/74ebdbe/ | 16:13 |
| esberglu | So initially I thought this was timing out on the rebuild | 16:13 |
| esberglu | But it looks like it is actually timing out on a show_server rest request | 16:14 |
| esberglu | Just wanted to make sure you were seeing the same. I went through the logs and the server in question goes through the flow as it should based on the test | 16:15 |
| esberglu | And actually reaches the correct status. The request to check the status is what seems to be the timeout source | 16:15 |
| *** esberglu is now known as esberglu_lunch | 16:25 | |
| edmondsw | AndyWojo you were asking about the OpenStack User Survey the other day... I did get out a blog about that last week if you want to check it out: https://developer.ibm.com/powervc/2017/07/27/openstack-user-survey-time/ | 17:19 |
| *** edmondsw_ has joined #openstack-powervm | 17:23 | |
| edmondsw_ | fried_rice is there a way to search eavesdrop? | 17:24 |
| fried_rice | edmondsw_ If you figure it out, let me know. | 17:24 |
| edmondsw_ | yeah, that's what I figured | 17:24 |
| fried_rice | edmondsw_ Wouldn't be too hard to write a crawler | 17:25 |
| fried_rice | which means someone must have done it. | 17:25 |
| *** edmondsw has quit IRC | 17:26 | |
| *** openstackstatus has quit IRC | 17:27 | |
| *** openstack has joined #openstack-powervm | 17:29 | |
| fried_rice | esberglu_lunch I agree with your assessment. And I can find no indication as to why that request timed out. | 17:49 |
| fried_rice | The n-api side looks fine. | 17:49 |
| *** esberglu_lunch is now known as esberglu | 17:49 | |
| fried_rice | But 99% of the time we get timeouts, this test case is one of the culprits, nah? | 17:50 |
| esberglu | fried_rice: Yeah | 17:50 |
| fried_rice | esberglu Can we run with tempest patches? | 17:51 |
| esberglu | fried_rice: Yeah | 17:51 |
| esberglu | Toss a patch up and start dumping some more info? | 17:52 |
| fried_rice | Well, I'm really not sure how more info would help us. | 17:52 |
| fried_rice | But it might be worth figuring out if the hitch is permanent or temporary. | 17:52 |
| fried_rice | So what I was thinking is: put an infinite retry loop around that show_server rest request. | 17:53 |
| fried_rice | esberglu https://review.openstack.org/491003 | 18:00 |
| fried_rice | esberglu So if the condition is permanent, the test itself will eventually time out. | 18:01 |
| fried_rice | But my guess is it ain't, and this will make that class of problem vanish. | 18:01 |
| esberglu | fried_rice: Nice beat me to it | 18:01 |
| fried_rice | esberglu Aaand now I'm gonna go to lunch. | 18:02 |
| esberglu | fried_rice? | 18:02 |
| fried_rice | I wish. | 18:02 |
| AndyWojo | edmondsw_: thank you. I already completed the survey. | 18:03 |
| edmondsw_ | AndyWojo sure. I think you can still update your answers until it closes, if you wanted to. | 18:04 |
| AndyWojo | I answered everything correctly. | 18:10 |
| edmondsw_ | AndyWojo ++ | 18:11 |
| AndyWojo | I recently got my OpenStack certification, and the SuSE OpenStack cert. Very surprised not to see any IBM certs centered around OpenStack/PowerVC | 18:16 |
| *** openstackstatus has quit IRC | 18:26 | |
| *** openstack has joined #openstack-powervm | 18:28 | |
| *** esberglu has quit IRC | 18:43 | |
| *** esberglu has joined #openstack-powervm | 18:43 | |
| *** esberglu has quit IRC | 18:48 | |
| *** esberglu has joined #openstack-powervm | 19:01 | |
| *** thorst has quit IRC | 19:03 | |
| openstackgerrit | Eric Berglund proposed openstack/nova-powervm master: DNM: ci check https://review.openstack.org/328315 | 19:04 |
| *** thorst has joined #openstack-powervm | 19:06 | |
| *** thorst has quit IRC | 19:10 | |
| *** chhavi has quit IRC | 19:18 | |
| *** thorst has joined #openstack-powervm | 19:18 | |
| *** thorst has quit IRC | 19:20 | |
| *** thorst has joined #openstack-powervm | 19:20 | |
| esberglu | edmondsw_: I'm dumb and messed up the patching format | 19:24 |
| esberglu | 5651 | 19:24 |
| edmondsw_ | esberglu np | 19:24 |
| esberglu | Added an example for future dumb me's lol | 19:25 |
| edmondsw_ | lol... +2 | 19:25 |
| edmondsw_ | could you just put a comment in the conf file itself? | 19:26 |
| esberglu | edmondsw_: Atm no. I was gonna add a TODO for that and to make the list of changes comma separated | 19:26 |
| edmondsw_ | sure | 19:26 |
| openstackgerrit | Eric Berglund proposed openstack/nova-powervm master: DNM: CI Check2 https://review.openstack.org/328317 | 19:27 |
| *** apearson has joined #openstack-powervm | 19:32 | |
| -openstackstatus- NOTICE: Gerrit is being restarted to pick up CSS changes and should be back momentarily | 20:36 | |
| *** smatzek has quit IRC | 20:50 | |
| *** apearson has quit IRC | 21:00 | |
| *** svenkat_ has quit IRC | 21:08 | |
| *** apearson has joined #openstack-powervm | 21:08 | |
| *** thorst has quit IRC | 21:21 | |
| *** thorst has joined #openstack-powervm | 21:22 | |
| *** thorst has quit IRC | 21:26 | |
| *** kylek3h has quit IRC | 21:38 | |
| *** edmondsw_ has quit IRC | 21:47 | |
| *** apearson has quit IRC | 22:05 | |
| *** apearson has joined #openstack-powervm | 22:06 | |
| *** apearson has quit IRC | 22:06 | |
| *** fried_rice is now known as efried_WEEKEND | 22:16 | |
| *** thorst has joined #openstack-powervm | 22:22 | |
| *** thorst has quit IRC | 22:28 | |
| *** esberglu has quit IRC | 22:28 | |
| *** esberglu has joined #openstack-powervm | 22:29 | |
| *** esberglu has quit IRC | 22:33 | |
| *** efried_WEEKEND has quit IRC | 22:48 | |
| *** efried_WEEKEND has joined #openstack-powervm | 22:58 | |
| *** edmondsw has joined #openstack-powervm | 23:36 | |
| *** edmondsw has quit IRC | 23:40 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!