| *** dwayne_ has quit IRC | 00:01 | |
| *** chas_ has joined #openstack-powervm | 00:15 | |
| *** thorst has joined #openstack-powervm | 00:19 | |
| *** chas_ has quit IRC | 00:20 | |
| *** thorst has quit IRC | 00:26 | |
| *** chas_ has joined #openstack-powervm | 01:16 | |
| *** chas_ has quit IRC | 01:20 | |
| *** thorst has joined #openstack-powervm | 01:25 | |
| *** tjakobs has joined #openstack-powervm | 01:30 | |
| *** tjakobs has quit IRC | 01:30 | |
| *** thorst has quit IRC | 01:32 | |
| *** thorst has joined #openstack-powervm | 01:51 | |
| *** thorst has quit IRC | 01:56 | |
| *** thorst has joined #openstack-powervm | 01:57 | |
| *** thorst has quit IRC | 02:05 | |
| *** chas_ has joined #openstack-powervm | 02:17 | |
| *** chas_ has quit IRC | 02:21 | |
| *** thorst has joined #openstack-powervm | 04:03 | |
| *** thorst has quit IRC | 04:10 | |
| *** chas_ has joined #openstack-powervm | 04:18 | |
| *** chas_ has quit IRC | 04:23 | |
| *** thorst has joined #openstack-powervm | 05:08 | |
| *** thorst has quit IRC | 05:15 | |
| *** thorst has joined #openstack-powervm | 06:15 | |
| *** thorst has quit IRC | 06:20 | |
| *** AlexeyAbashkin has joined #openstack-powervm | 06:51 | |
| *** AlexeyAbashkin has quit IRC | 06:57 | |
| *** thorst has joined #openstack-powervm | 07:19 | |
| *** thorst has quit IRC | 07:25 | |
| *** AlexeyAbashkin has joined #openstack-powervm | 07:56 | |
| *** thorst has joined #openstack-powervm | 08:22 | |
| *** thorst has quit IRC | 08:30 | |
| *** k0da has joined #openstack-powervm | 08:54 | |
| *** chas_ has joined #openstack-powervm | 08:57 | |
| *** AlexeyAbashkin has quit IRC | 09:04 | |
| *** madhaviy has joined #openstack-powervm | 09:23 | |
| *** thorst has joined #openstack-powervm | 09:28 | |
| *** thorst has quit IRC | 09:34 | |
| *** openstackgerrit has quit IRC | 09:47 | |
| *** openstackgerrit has joined #openstack-powervm | 09:48 | |
| *** thorst has joined #openstack-powervm | 10:32 | |
| *** thorst has quit IRC | 10:40 | |
| *** AlexeyAbashkin has joined #openstack-powervm | 11:27 | |
| *** chas_ has quit IRC | 11:29 | |
| *** thorst has joined #openstack-powervm | 11:37 | |
| *** thorst has quit IRC | 11:45 | |
| *** smatzek has joined #openstack-powervm | 11:50 | |
| *** seroyer has joined #openstack-powervm | 12:25 | |
| *** AlexeyAbashkin has quit IRC | 12:35 | |
| *** seroyer has quit IRC | 12:39 | |
| *** seroyer has joined #openstack-powervm | 12:41 | |
| *** thorst has joined #openstack-powervm | 12:50 | |
| *** svenkat has joined #openstack-powervm | 12:50 | |
| *** thorst_ has joined #openstack-powervm | 12:54 | |
| *** apearson has quit IRC | 12:54 | |
| *** thorst has quit IRC | 12:54 | |
| *** kylek3h has quit IRC | 13:00 | |
| *** edmondsw has joined #openstack-powervm | 13:03 | |
| *** seroyer has quit IRC | 13:15 | |
| *** chas_ has joined #openstack-powervm | 13:22 | |
| *** apearson has joined #openstack-powervm | 13:24 | |
| *** mdrabe has joined #openstack-powervm | 13:42 | |
| *** seroyer has joined #openstack-powervm | 13:43 | |
| *** AlexeyAbashkin has joined #openstack-powervm | 13:49 | |
| *** kylek3h has joined #openstack-powervm | 13:51 | |
| *** tblakes has joined #openstack-powervm | 13:54 | |
| *** seroyer has quit IRC | 13:58 | |
| *** AlexeyAbashkin has quit IRC | 14:02 | |
| *** esberglu has joined #openstack-powervm | 14:07 | |
| thorst_ | esberglu: where are we at CI wise? | 14:15 |
|---|---|---|
| thorst_ | cause see https://review.openstack.org/#/c/381772/ | 14:16 |
| openstackgerrit | Sridhar Venkat proposed openstack/networking-powervm: ProvisionRequest does not distinguish event source https://review.openstack.org/396467 | 14:16 |
| esberglu | thorst_: I'm just looking at the runs from the latest deploy now. Still getting timeouts across the board on spawns | 14:17 |
| esberglu | http://184.172.12.213/10/396510/6/silent/nova-pvm-dsvm-tempest-full/954b85a/ | 14:17 |
| thorst_ | which is stuck here: "Waiting for in-progress upload(s) to complete." | 14:18 |
| esberglu | Yep | 14:19 |
| esberglu | efried: ^^ | 14:19 |
| thorst_ | alright. And the first VM hits that. | 14:19 |
| thorst_ | this has got to be leaving a bunch of junk in the env. | 14:19 |
| thorst_ | Is there a way we can clean the environment and just have a single test go through...and we can maybe step through that? | 14:20 |
| thorst_ | I basically want to see the logs where we create the marker... | 14:21 |
| *** chas_ has quit IRC | 14:24 | |
| *** chas_ has joined #openstack-powervm | 14:25 | |
| esberglu | thorst_: For a clean env. we would have to redeploy. And then I could just disable all of the projects. Then turn a project on just long enough to manually kick a run off | 14:25 |
| thorst_ | what about the staging env? | 14:26 |
| thorst_ | can we do there? | 14:26 |
| esberglu | It would wipe all of the OSA CI stuff. Which is fine by me but idk if qing wu has anything on there right now he doesn't want to lose | 14:27 |
| esberglu | Actually I may be able to just kick off a single CI run | 14:28 |
| esberglu | The env. should be clean because it has just been OSA dev | 14:28 |
| thorst_ | yeah...that'd be neat. or maybe if it has a ready node...I could hop on there quick? | 14:28 |
| esberglu | And I can just enable a non-osa project for a second | 14:28 |
| thorst_ | well, if it has a ready node...with the patches | 14:29 |
| thorst_ | then I could hop on that and take a peak before you let something through | 14:29 |
| *** chas_ has quit IRC | 14:29 | |
| esberglu | We would have to rebuild the image and all that for the patches to get in | 14:29 |
| esberglu | Or just apply the patches manually | 14:30 |
| thorst_ | esberglu: lets apply manually | 14:30 |
| thorst_ | just try to recreate ourselves | 14:30 |
| esberglu | I'm just trying to think if there is anything from the OSA deploy that would cause issues | 14:32 |
| esberglu | I don't think so | 14:32 |
| *** smatzek has quit IRC | 14:32 | |
| esberglu | thorst_: There are 2 ready nodes on neo14 that we can toy with | 14:38 |
| thorst_ | is efried1 in yet? | 14:38 |
| thorst_ | also, POK network is awful ATM | 14:38 |
| thorst_ | so I'm investigating that a bit. | 14:38 |
| thorst_ | nope, efried1 is out. | 14:42 |
| thorst_ | esberglu: PM me the private nodes? | 14:43 |
| *** chas_ has joined #openstack-powervm | 14:45 | |
| *** kriskend has joined #openstack-powervm | 14:48 | |
| *** seroyer has joined #openstack-powervm | 14:48 | |
| *** chas_ has quit IRC | 14:50 | |
| *** smatzek has joined #openstack-powervm | 14:56 | |
| *** tjakobs has joined #openstack-powervm | 15:22 | |
| thorst_ | esberglu: the pvmctl packages on these ready nodes is quite old | 15:32 |
| esberglu | thorst_: I thought there was logic that got the newest in one of the scripts? Hold on I will check | 15:36 |
| thorst_ | I can confirm its freezing in efried's code. | 15:42 |
| thorst_ | :-) | 15:42 |
| esberglu | thorst_: It installs "neo-cli-latest" from gsa. I assumed that it was something that gets updated due to the name, but it does not. And yeah its super old | 15:44 |
| thorst_ | lol | 15:44 |
| esberglu | :q | 15:46 |
| adreznec | Yeah... | 15:47 |
| adreznec | We were supposed to have open sourced the cli code by now | 15:47 |
| adreznec | Making that whole flow unnecessary | 15:47 |
| adreznec | But it's never made it onto Chris' plate | 15:47 |
| esberglu | I will get a new version up on GSA | 15:48 |
| thorst_ | esberglu: I need to make a new patch of efried1's... | 15:51 |
| esberglu | thorst_: Okay. Let me know when it is ready | 15:52 |
| thorst_ | it'll be a bit | 15:54 |
| thorst_ | efried1 forgot how threads work :-) | 15:54 |
| thorst_ | esberglu: somethin just kicked me off of the ready node | 16:05 |
| thorst_ | looks like something stole its IP | 16:05 |
| esberglu | What? I disabled all of the projects on the staging CI, so nothing should be happening. | 16:07 |
| esberglu | But even if runs were going through idk what would do that | 16:07 |
| thorst_ | I'm assuming that something from prod is overlapping the IPs? | 16:08 |
| thorst_ | i'm kinda blocked on testing till we figure it out :-) | 16:11 |
| esberglu | thorst_: This what you see if you try to log in again? | 16:15 |
| esberglu | PowerVM_CI-PowerVM_DevStacked-25758 | 16:15 |
| thorst_ | isn't that way too high a number for the staging env? | 16:17 |
| thorst_ | this is what it should be: powervm-ci-powervm-devstacked-13581 | 16:17 |
| esberglu | Yeah theres a prod node with the same ip | 16:17 |
| thorst_ | kill it please? | 16:17 |
| thorst_ | and we need to get you a different set of IPs for staging. | 16:17 |
| thorst_ | that'll lead to nightmares. | 16:17 |
| esberglu | Yeah I thought we did that when we very first made the staging env? | 16:18 |
| esberglu | I will look into it | 16:18 |
| thorst_ | and I stand corrected... efried1 did have threads working right. | 16:20 |
| thorst_ | esberglu: where is the code we use to 'seed' the image into the SSP? | 16:22 |
| *** dwayne has joined #openstack-powervm | 16:22 | |
| thorst_ | esberglu: OK - so efried1's patch works. I'm thinking that we have environment clean out issues. Do you think we can do a full environment rebuild here? | 16:27 |
| thorst_ | I want to make sure we've properly cleaned out the SSPs... | 16:27 |
| *** chas_ has joined #openstack-powervm | 16:27 | |
| thorst_ | anyway to clean, verify that everything is clean, then re-open? | 16:27 |
| esberglu | We should just be able to run vm_cleaner.sh on all of the nodes to clean them | 16:32 |
| thorst_ | what about existing runs? | 16:32 |
| esberglu | Zuul is disabled rn, no runs are going | 16:33 |
| thorst_ | ok | 16:35 |
| thorst_ | can you clean it out? | 16:35 |
| thorst_ | then we'll want to step through each one individually | 16:35 |
| esberglu | Yep. I put up a change for the ip allocations | 16:36 |
| thorst_ | +2'd...can we roll that out quickly? | 16:37 |
| thorst_ | going to grab lunch....I'll check where we are when I get back. But intention is to go to each host and make sure there are no LU's there after we clean it out completely | 16:42 |
| thorst_ | not even ready nodes... | 16:42 |
| thorst_ | then open it back up | 16:42 |
| *** k0da has quit IRC | 16:50 | |
| *** madhaviy has quit IRC | 17:02 | |
| *** chas__ has joined #openstack-powervm | 17:10 | |
| *** chas_ has quit IRC | 17:12 | |
| efried1 | Hey thorst_, esberglu - I'm on now. Anything urgent I can help with? | 17:23 |
| esberglu | efried1: We found a couple issues with the CI. We had IPs overlapping in staging/production. And an outdated neo-cli | 17:24 |
| esberglu | I'm redeploying with fixes for both | 17:24 |
| efried1 | cool | 17:24 |
| *** chas__ has quit IRC | 17:34 | |
| *** chas_ has joined #openstack-powervm | 17:35 | |
| *** chas_ has quit IRC | 17:39 | |
| thorst_ | esberglu: did we get it cleaned out? | 18:16 |
| esberglu | Yeah. And I confirmed that all the LUs were gone. I ended up doing a full cloud redeploy (still running) | 18:18 |
| thorst_ | esberglu: OK - cool | 18:22 |
| *** chas_ has joined #openstack-powervm | 18:30 | |
| thorst_ | esberglu: we'll be able to get the ready nodes set up, but not accept jobs yet? | 18:32 |
| thorst_ | (and then maybe open it to just one job quick) | 18:32 |
| *** chas__ has joined #openstack-powervm | 18:34 | |
| *** chas_ has quit IRC | 18:34 | |
| *** chas__ has quit IRC | 18:38 | |
| esberglu | thorst_: Yep I can do that. Did you still have a change to that patch? Or did it end up being okay? | 18:40 |
| thorst_ | esberglu: its OK | 18:41 |
| thorst_ | I think there are optimizations | 18:41 |
| thorst_ | but it shouldn't need it | 18:41 |
| esberglu | Okay. About to kick off the mgmt playbook, then get some food | 18:42 |
| thorst_ | esberglu: k | 18:44 |
| thorst_ | thx | 18:44 |
| *** chas_ has joined #openstack-powervm | 19:28 | |
| *** k0da has joined #openstack-powervm | 19:29 | |
| *** mdrabe has quit IRC | 19:53 | |
| *** mdrabe has joined #openstack-powervm | 19:53 | |
| *** kylek3h is now known as kylek3h_away | 20:12 | |
| *** mdrabe_ has joined #openstack-powervm | 20:30 | |
| *** kylek3h_away is now known as kylek3h | 20:30 | |
| *** mdrabe has quit IRC | 20:33 | |
| *** mdrabe_ has quit IRC | 20:45 | |
| esberglu | thorst_: We've got jenkins nodes ready to go on production | 21:05 |
| thorst_ | can you fire one test job off? | 21:05 |
| thorst_ | preferably one on nova-powervm | 21:05 |
| *** kriskend_ has joined #openstack-powervm | 21:06 | |
| *** kriskend has quit IRC | 21:06 | |
| *** smatzek has quit IRC | 21:10 | |
| thorst_ | esberglu: this run is looking pretty clean... | 21:49 |
| *** svenkat has quit IRC | 21:52 | |
| thorst_ | although it does appear to be leaking LU's | 21:53 |
| *** kriskend_ has quit IRC | 21:55 | |
| *** kriskend_ has joined #openstack-powervm | 21:56 | |
| esberglu | thorst_: Not getting stuck on the waiting for upload to complete thing though? | 21:58 |
| *** smatzek has joined #openstack-powervm | 22:00 | |
| thorst_ | well, the 'part' isn't there anymore | 22:01 |
| thorst_ | but it is leaking a bunch of disks | 22:01 |
| *** smatzek has quit IRC | 22:02 | |
| *** smatzek has joined #openstack-powervm | 22:02 | |
| *** edmondsw has quit IRC | 22:02 | |
| *** mdrabe has joined #openstack-powervm | 22:17 | |
| *** kylek3h has quit IRC | 22:23 | |
| *** smatzek has quit IRC | 22:25 | |
| *** thorst_ has quit IRC | 22:30 | |
| *** tblakes has quit IRC | 22:37 | |
| *** thorst has joined #openstack-powervm | 22:53 | |
| *** kylek3h has joined #openstack-powervm | 22:56 | |
| *** kylek3h has quit IRC | 22:56 | |
| *** kylek3h has joined #openstack-powervm | 22:57 | |
| *** thorst has quit IRC | 22:57 | |
| *** seroyer has quit IRC | 23:09 | |
| *** kylek3h_ has joined #openstack-powervm | 23:13 | |
| *** kylek3h has quit IRC | 23:13 | |
| *** kriskend_ has quit IRC | 23:15 | |
| *** kylek3h_ has quit IRC | 23:16 | |
| *** esberglu has quit IRC | 23:16 | |
| *** kylek3h has joined #openstack-powervm | 23:16 | |
| *** esberglu has joined #openstack-powervm | 23:16 | |
| *** apearson has quit IRC | 23:17 | |
| *** chas_ has quit IRC | 23:17 | |
| *** chas_ has joined #openstack-powervm | 23:18 | |
| *** chas_ has quit IRC | 23:20 | |
| *** chas_ has joined #openstack-powervm | 23:20 | |
| *** esberglu has quit IRC | 23:21 | |
| *** chas_ has quit IRC | 23:25 | |
| *** kylek3h is now known as kylek3h_away | 23:27 | |
| *** esberglu has joined #openstack-powervm | 23:29 | |
| *** tjakobs has quit IRC | 23:37 | |
| *** k0da has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!