| *** edmondsw has joined #openstack-powervm | 00:03 | |
| *** edmondsw has quit IRC | 00:08 | |
| *** waler has quit IRC | 01:39 | |
| *** waler has joined #openstack-powervm | 01:51 | |
| *** waler has quit IRC | 02:26 | |
| *** waler has joined #openstack-powervm | 02:26 | |
| *** esberglu has joined #openstack-powervm | 02:35 | |
| openstackgerrit | zhangyanxian proposed openstack/ceilometer-powervm master: Optimize the link address for fetching git code https://review.openstack.org/447762 | 02:35 |
|---|---|---|
| openstackgerrit | zhangyanxian proposed openstack/ceilometer-powervm master: Optimize the link address for fetching git code https://review.openstack.org/447762 | 02:37 |
| *** esberglu has quit IRC | 02:39 | |
| openstackgerrit | zhangyanxian proposed openstack/networking-powervm master: Optimize the link address for fetching git code https://review.openstack.org/447763 | 02:39 |
| openstackgerrit | zhangyanxian proposed openstack/networking-powervm master: Optimize the link address for fetching git code https://review.openstack.org/447763 | 02:40 |
| *** waler has quit IRC | 04:20 | |
| *** esberglu has joined #openstack-powervm | 04:24 | |
| *** esberglu has quit IRC | 04:29 | |
| *** waler has joined #openstack-powervm | 04:32 | |
| *** waler has quit IRC | 04:59 | |
| *** waler has joined #openstack-powervm | 05:00 | |
| *** esberglu has joined #openstack-powervm | 06:13 | |
| *** esberglu has quit IRC | 06:17 | |
| *** k0da has joined #openstack-powervm | 07:46 | |
| *** a1fisher has joined #openstack-powervm | 07:58 | |
| *** esberglu has joined #openstack-powervm | 08:02 | |
| *** esberglu has quit IRC | 08:06 | |
| -openstackstatus- NOTICE: Wiki is broken with database problems, we are working to resolve it | 08:32 | |
| *** ChanServ changes topic to "Wiki is broken with database problems, we are working to resolve it" | 08:32 | |
| *** k0da has quit IRC | 08:35 | |
| *** ChanServ changes topic to "This channel is for PowerVM-related development and discussion. For general OpenStack support, please use #openstack." | 08:40 | |
| -openstackstatus- NOTICE: Wiki problems have been fixed, it's up and running | 08:40 | |
| *** openstackgerrit has quit IRC | 09:03 | |
| *** k0da has joined #openstack-powervm | 09:16 | |
| *** esberglu has joined #openstack-powervm | 09:51 | |
| *** esberglu has quit IRC | 09:56 | |
| *** k0da has quit IRC | 10:46 | |
| *** smatzek has joined #openstack-powervm | 11:09 | |
| *** edmondsw has joined #openstack-powervm | 12:00 | |
| *** edmondsw has quit IRC | 12:05 | |
| *** jpasqualetto has joined #openstack-powervm | 12:08 | |
| *** jpasqualetto has quit IRC | 12:14 | |
| *** svenkat has joined #openstack-powervm | 12:25 | |
| *** jpasqualetto has joined #openstack-powervm | 12:26 | |
| *** edmondsw has joined #openstack-powervm | 12:26 | |
| *** kjw3 has quit IRC | 12:29 | |
| *** kjw3 has joined #openstack-powervm | 12:30 | |
| *** kylek3h has joined #openstack-powervm | 12:37 | |
| *** efried has joined #openstack-powervm | 12:52 | |
| *** esberglu has joined #openstack-powervm | 12:58 | |
| esberglu | #startmeeting powervm_driver_meeting | 13:01 |
| openstack | Meeting started Tue Mar 21 13:01:45 2017 UTC and is due to finish in 60 minutes. The chair is esberglu. Information about MeetBot at http://wiki.debian.org/MeetBot. | 13:01 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 13:01 |
| openstack | The meeting name has been set to 'powervm_driver_meeting' | 13:01 |
| efried | o/ | 13:01 |
| esberglu | #topic In Tree Driver | 13:02 |
| efried | Four change sets are rebased and updated. | 13:02 |
| esberglu | Yep looking at those is first on my list this morning | 13:02 |
| esberglu | Mostly just rebasing or functional changes? | 13:03 |
| efried | Bottom two passed CI; top two failed. I just submitted rechecks for 'em. #4 failed one test, rebuild in error state or something. #3 failed that same one and two more. | 13:03 |
| esberglu | Yeah will talk about CI failures later in the meeting | 13:03 |
| efried | Actually almost no functional changes - a couple of method signatures and the way flavors are passed around. The vast majority was UT changes. | 13:03 |
| *** apearson has joined #openstack-powervm | 13:04 | |
| efried | So I don't expect CI failures due to my changes. | 13:04 |
| efried | Fact that #2 succeeded supports that. | 13:04 |
| esberglu | Yeah those failures are unrelated | 13:05 |
| efried | Cause the method signature changes started there. | 13:05 |
| efried | So today I plan to rebase the SSP change set. Then jayasankar_ can retest that guy and we can see if there's still that weird glance bug to nail down. | 13:05 |
| efried | Say, is jayasankar_ on this meeting notice? | 13:05 |
| esberglu | No I will add him | 13:06 |
| efried | Thanks. | 13:06 |
| efried | And Nilesh too | 13:06 |
| esberglu | Yep | 13:06 |
| efried | I don't have anything else. | 13:06 |
| esberglu | I think jayasankar_ hit some issues testing in-tree, I haven't looked into yet but we may have an issue there | 13:07 |
| esberglu | Gonna help him with that today if hes online | 13:07 |
| esberglu | That's all I have for in-tree | 13:07 |
| esberglu | #topic Out of Tree Driver | 13:07 |
| esberglu | Guessing there isn't gonna be much to talk about here | 13:07 |
| adreznec | We need to get that new pypowervm release out | 13:08 |
| efried | I've still been backporting some of the changes from in-tree. | 13:08 |
| esberglu | Rechecked all changes to the stable/* branches, so the backlog is cleared there | 13:08 |
| adreznec | Currently our reqs issue is blocking an OSA SHA release | 13:08 |
| efried | I'll put up a new OOT change set once I'm done with the SSP stuff. | 13:08 |
| adreznec | The PTL come to talk to me about it yesterday | 13:09 |
| efried | Guess you'll be on jfoliva's ass today. Sh*t rolls downhill. | 13:09 |
| adreznec | I pinged Julio and I believe he and Dom merged the change, but getting that out today is my priority | 13:09 |
| adreznec | So we can merge https://review.openstack.org/#/c/440811/ | 13:09 |
| efried | We'll have to put up a new patch set on that. | 13:10 |
| efried | but yeah | 13:10 |
| adreznec | Yep | 13:10 |
| adreznec | I'll drive that | 13:10 |
| *** mdrabe has joined #openstack-powervm | 13:11 | |
| efried | #action adreznec to hold the whip on jfoliva to get the new pypowervm release out. | 13:11 |
| efried | #action adreznec to close https://review.openstack.org/#/c/440811/ | 13:11 |
| *** tblakes has joined #openstack-powervm | 13:12 | |
| esberglu | #topic CI | 13:13 |
| esberglu | We are getting Http 500 Errors, it is hitting multiple tests | 13:14 |
| esberglu | I'm guessing that's what hung up your patches efried | 13:14 |
| esberglu | http://184.172.12.213/52/445652/3/check/nova-out-of-tree-pvm/ab6525c/powervm_os_ci.html | 13:14 |
| esberglu | ^ example | 13:14 |
| esberglu | I haven't had a chance to debug yet, been putting out other CI fires for the last week or so | 13:14 |
| esberglu | But it's on my radar for today | 13:15 |
| esberglu | I know that it was hitting some of the "rebuild" tests as well | 13:15 |
| esberglu | The other thing I wanted to talk about is how to determine which tests to run for in-tree and how/when we are going to put those out | 13:17 |
| efried | unknown internal error with half a dozen blank lines in the <Message> | 13:17 |
| efried | That's gonna be fun to debug. | 13:17 |
| efried | We don't save off the REST logs, do we. | 13:17 |
| esberglu | No | 13:17 |
| efried | whee. | 13:17 |
| esberglu | We could | 13:17 |
| efried | But I don't think my failures were the same, fwiw. | 13:17 |
| efried | The 500 showed up in the html report here, but the same wasn't happening in my failures. | 13:18 |
| efried | Anyway, leave it to you to debug. Let me know if you need help. | 13:18 |
| esberglu | Sounds good | 13:18 |
| esberglu | Anyway back to the in-tree whitelist | 13:18 |
| efried | Check with apearson whether it's okay to publish REST logs publicly. | 13:18 |
| esberglu | K. The problem with the whitelist is that there isn't a good way to know which tests will be supported when we add certain functionality | 13:19 |
| esberglu | What I have been doing | 13:19 |
| efried | esberglu One strategy is: with each new change set, make sure OOT is stable, then run the in-tree change set with the OOT config. | 13:19 |
| efried | Whatever passes, makes the whitelist. | 13:19 |
| esberglu | Yep that's what I have been doing | 13:19 |
| efried | It's simple and easy. But it's not particularly scientific. | 13:19 |
| esberglu | Exactly | 13:19 |
| efried | What we really should be doing (ugh) is inspecting each test and seeing if it *should* pass or fail. | 13:20 |
| efried | If it should fail and it passes, we should open a bug for that, and potentially fix the test. | 13:20 |
| esberglu | Yeah, but that's a HUGE time investment | 13:20 |
| efried | I suspect there are a number of tests that fit that category. | 13:20 |
| efried | If it should pass and it fails, likewise nail that down. | 13:20 |
| efried | Yes. Possibly something jayasankar_ and nilesh could help out with. | 13:21 |
| esberglu | The other thing related to this is when we are going to put changes out | 13:22 |
| efried | to the in-tree whitelist? | 13:22 |
| esberglu | Right now we are pulling in the 1st patch for all in-tree runs | 13:22 |
| esberglu | Once that 1st one merges, are we going to start pulling in the 2nd patch and test that for all in-tree? | 13:22 |
| efried | Well, I don't think the methodology has to do with change-by-change so much as by when (at which change set boundaries) the whitelist changes. | 13:23 |
| efried | The whitelist may actually be the same for the first four or five change sets. | 13:23 |
| efried | Is it? | 13:24 |
| esberglu | No | 13:24 |
| esberglu | power on/off will have an associated whitelist change | 13:24 |
| efried | Okay, so #1 and #2 have the same whitelist, then it changes for power on/off, then it stays the same again until SSP, I'm guessing. | 13:25 |
| efried | or maybe console. | 13:25 |
| esberglu | Yep I don't think any of the 4 spawn/destroy will change anything | 13:26 |
| esberglu | But my issue is still present when we get to SSP | 13:26 |
| esberglu | We can't change the whitelist until that change is in | 13:26 |
| esberglu | Meaning we won't get much CI volume on that change | 13:26 |
| esberglu | Unless we do what we are doing with PS1 right now and pull it in for every in-tree run | 13:26 |
| efried | Unless we change our setup to pull in that guy | 13:26 |
| efried | right. | 13:26 |
| efried | But | 13:26 |
| efried | Then with the lower change sets we would have to figure out how to allow them to revert down the tree. | 13:27 |
| *** smatzek has quit IRC | 13:27 | |
| efried | Until they merge. | 13:27 |
| efried | Which is gonna be a while, way things are going. | 13:27 |
| efried | So maybe we should look into enabling that. | 13:27 |
| efried | Then we could essentially move up the whitelist and the baseline change set any time we have a change set we consider stable. | 13:28 |
| efried | Right now we're doing something like, "If the path from the current change set back to tip of master contains our baseline change set, don't apply our baseline change set." | 13:29 |
| *** jpasqualetto has quit IRC | 13:30 | |
| esberglu | Okay I like that going forward. For now I think we are stable through the 4th spawn delete (full flavor). So are we ready to move our baseline to that? | 13:30 |
| esberglu | And yeah that's what we are currently doing | 13:30 |
| efried | We would have to do that, but also, "If the path from our baseline back to the tip of master contains the current change set, don't apply our baseline change set." | 13:30 |
| efried | Yes, I'm happy to move our baseline to that once we have the above logic working. | 13:31 |
| efried | But not until then. | 13:31 |
| efried | Cause otherwise change sets 1-3 will fail. | 13:31 |
| efried | They won't fail the CI run - they'll fail on the git shuffle. | 13:31 |
| esberglu | Okay. I can look into that on staging, might bug you later if I have questions | 13:32 |
| efried | Cause you'll be trying to cherry-pick, say, #2 onto the tip of a chain that already contains #2. | 13:32 |
| esberglu | I think that --allow-empty gets past that no? | 13:32 |
| esberglu | or --keep-redundant | 13:32 |
| esberglu | I think | 13:32 |
| efried | It may. But I'm not sure. There's a more explicit way to do it. | 13:32 |
| esberglu | #action esberglu: Add CI logic for applying proper changesets | 13:34 |
| efried | if git log --pretty=format:%H origin/${ZUUL_BRANCH}..HEAD | grep -q $commit; then | 13:34 |
| efried | That's what we're doing today. | 13:34 |
| efried | The new thing is going to be a little more complicated because we don't yet have $commit downloaded. | 13:34 |
| efried | oo, it's even more complicated because we're theoretically looping through and cherry-picking multiple commits. | 13:35 |
| efried | This is gonna be fun. | 13:35 |
| *** mdrabe has quit IRC | 13:36 | |
| efried | Anyway, | 13:36 |
| efried | we don't need to solve it here. | 13:36 |
| efried | We can actually do the git fetch and compare origin/${ZUUL_BRANCH} to FETCH_HEAD | 13:36 |
| *** mdrabe has joined #openstack-powervm | 13:37 | |
| efried | And just skip the cherry-pick. | 13:37 |
| efried | So yeah, we don't need to solve it here. | 13:37 |
| efried | But that solves it. | 13:37 |
| esberglu | lol | 13:37 |
| esberglu | Okay. The only other thing I have for CI is OSA CI | 13:38 |
| * efried backs away slowly | 13:38 | |
| esberglu | Same... | 13:38 |
| esberglu | Currently just trying to get run_playbooks script to work | 13:39 |
| esberglu | Which is what runs basically the entire set of ansible playbooks | 13:40 |
| esberglu | It took forever to get that env. stable, but I think it finally is and I can start grinding through the failures there | 13:40 |
| esberglu | That's all for me today. Any other topics/thoughts? | 13:41 |
| adreznec | Any specific issues on the OSA CI at this point? | 13:41 |
| esberglu | It's failing to install pip | 13:41 |
| adreznec | That... seems odd | 13:42 |
| esberglu | Yeah. I've only tried the script once and didn't have time to debug yet | 13:42 |
| esberglu | So it might be something trivial | 13:42 |
| adreznec | Ok | 13:42 |
| esberglu | I'll keep you posted | 13:43 |
| efried | esberglu #3 and #4 failed in-tree CI again. | 13:44 |
| esberglu | Ugh. Same thing? | 13:44 |
| efried | no | 13:44 |
| efried | test_multiple_create_with_reservation_return | 13:44 |
| efried | and test_multiple_create | 13:45 |
| efried | respectively. | 13:45 |
| esberglu | Http 500 or not? | 13:45 |
| efried | not from us. | 13:45 |
| efried | Looks like might be neutron glitches. | 13:46 |
| efried | Want me to recheck again, or leave 'em? | 13:46 |
| esberglu | Looks like the test_multiple_create is hitting all of the OOT runs since last night | 13:48 |
| efried | is it new? | 13:49 |
| efried | the test, that is? | 13:49 |
| efried | sorry, we shouldn't derail the meeting with this. Is the meeting over? | 13:49 |
| esberglu | Yeah I think so | 13:49 |
| esberglu | #endmeeting | 13:49 |
| openstack | Meeting ended Tue Mar 21 13:49:30 2017 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 13:49 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/powervm_driver_meeting/2017/powervm_driver_meeting.2017-03-21-13.01.html | 13:49 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/powervm_driver_meeting/2017/powervm_driver_meeting.2017-03-21-13.01.txt | 13:49 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/powervm_driver_meeting/2017/powervm_driver_meeting.2017-03-21-13.01.log.html | 13:49 |
| esberglu | And no I don't believe that is a new test | 13:49 |
| esberglu | But I'm checking now | 13:49 |
| *** jpasqualetto has joined #openstack-powervm | 13:50 | |
| esberglu | Nope not new | 13:51 |
| esberglu | I will start debugging | 13:51 |
| efried | coo | 13:51 |
| *** jpasqualetto has quit IRC | 13:52 | |
| *** smatzek has joined #openstack-powervm | 13:59 | |
| *** jpasqualetto has joined #openstack-powervm | 14:07 | |
| *** tjakobs has joined #openstack-powervm | 14:09 | |
| *** burgerk has joined #openstack-powervm | 14:22 | |
| esberglu | efried: That test_multiple_create is failing with "Multiple Possible Networks Found" which has been very difficult to debug in the past | 14:25 |
| esberglu | Still looking | 14:26 |
| esberglu | But nothing obvious in the logs so far | 14:26 |
| *** k0da has joined #openstack-powervm | 14:42 | |
| *** openstackgerrit has joined #openstack-powervm | 14:48 | |
| openstackgerrit | gordon chung proposed openstack/ceilometer-powervm master: use new instance polling interface https://review.openstack.org/448134 | 14:49 |
| *** mdrabe has quit IRC | 14:50 | |
| esberglu | efried: I think the test_multiple_create just comes down to the way we have our net. configured | 14:51 |
| *** tblakes has quit IRC | 14:55 | |
| *** mdrabe has joined #openstack-powervm | 14:58 | |
| *** tblakes has joined #openstack-powervm | 15:19 | |
| esberglu | efried: You have any time to take a look? | 15:25 |
| *** jayasankar_ has quit IRC | 15:49 | |
| efried | esberglu I could try... | 15:53 |
| efried | What are we looking at? | 15:53 |
| esberglu | http://184.172.12.213/76/40087B6/31/check/nova-out-of-tree-pvm/31196b7/powervm_os_ci.html | 15:53 |
| esberglu | So it's finding multiple possible networks and doesn't know which one to use | 15:54 |
| esberglu | Which we have hit problems with before | 15:54 |
| efried | I get a 404 on that link. | 15:54 |
| efried | ah | 15:55 |
| efried | There's an extra B in the URL | 15:55 |
| esberglu | Oh theres an extra B in there | 15:55 |
| efried | Is this in-tree? | 15:55 |
| esberglu | OOT | 15:55 |
| esberglu | It's hitting every run right now | 15:55 |
| efried | And we've had multiple networks in the past when some other test has been creating them and something went wrong? | 15:56 |
| esberglu | Yeah, we've hit issues with multiple possible networks found a few times | 15:56 |
| esberglu | Leading us to change our network config. to shared | 15:57 |
| esberglu | I think. Or maybe it was after we changed it to shared that we saw it | 15:57 |
| esberglu | There is a section in the skip list with the comment "Failing becuase we are using shared networks" | 15:59 |
| esberglu | And I think they were tests also hitting this multiple networks issue | 16:00 |
| esberglu | And we tried to debug for a while and didn't get anywhere | 16:00 |
| esberglu | In this case, I don't think there are other tests having issues. I think it is a timing thing and there are actually just multiple networks at that point in time | 16:02 |
| esberglu | And it doesn't know which to choose | 16:02 |
| esberglu | Hard to say though, and I have been struggling to find anything useful in the logs | 16:03 |
| efried | Others must have this issue. Maybe we can ask around on #openstack-qa? | 16:12 |
| efried | I suppose attempting a run with concurrency=1 would take a nontrivial redeploy of something-or-other? | 16:12 |
| esberglu | Actually I don't think it would. I could put up a patch to os_ci_tempest.sh and pull it in as part of the jenkins job without redeploying I think | 16:15 |
| esberglu | Let me confirm quick | 16:15 |
| esberglu | Yeah I could just reload the jenkins config to get it in | 16:16 |
| esberglu | Think we should try it? | 16:18 |
| *** a1fisher has quit IRC | 16:32 | |
| efried | esberglu If you're seeing that failure regularly, then yeah, I think it would be informative, if it's not too much trouble. Of course, the run will take a lot longer, so it's not a viable long-term solution, but it should give us a better baseline for debugging. | 16:34 |
| openstackgerrit | Eric Berglund proposed openstack/nova-powervm master: DNM: ci check https://review.openstack.org/328315 | 16:43 |
| esberglu | efried: ^ Testing on that patch | 16:44 |
| efried | coo | 16:45 |
| openstackgerrit | gordon chung proposed openstack/ceilometer-powervm master: use new instance polling interface https://review.openstack.org/448134 | 16:58 |
| *** IRCFrEAK has joined #openstack-powervm | 17:50 | |
| *** IRCFrEAK has quit IRC | 17:52 | |
| *** adreznec has quit IRC | 17:55 | |
| *** adreznec has joined #openstack-powervm | 17:57 | |
| *** jpasqualetto has quit IRC | 19:04 | |
| *** jpasqualetto has joined #openstack-powervm | 19:16 | |
| openstackgerrit | gordon chung proposed openstack/ceilometer-powervm master: use new vnic polling interface https://review.openstack.org/448249 | 19:27 |
| *** esberglu has quit IRC | 20:00 | |
| *** esberglu has joined #openstack-powervm | 20:00 | |
| *** esberglu has quit IRC | 20:05 | |
| *** esberglu has joined #openstack-powervm | 20:32 | |
| *** smatzek has quit IRC | 20:33 | |
| esberglu | efried: Bad news. Still failing test_multiple_create with concurrency=1 | 20:34 |
| efried | esberglu That's not actually bad news. Means we don't have a timing problem with other tests. | 20:34 |
| esberglu | True. I'm looking into the test now to see what could be going wrong | 20:35 |
| *** svenkat has quit IRC | 20:57 | |
| esberglu | efried: Okay I think I see what's happening | 21:15 |
| efried | talk t ome | 21:15 |
| esberglu | https://github.com/openstack/tempest/commit/176e7cb66b8e131993b2e6a8bfb1652e17452a44 | 21:15 |
| esberglu | So you can see there they changed to call compute.create_test_server | 21:15 |
| esberglu | When it was calling self.create_test_server | 21:16 |
| esberglu | It would eventually get to here | 21:16 |
| esberglu | https://github.com/openstack/tempest/blob/master/tempest/test.py#L642 | 21:16 |
| esberglu | Which would set the network based on the fixed_network_name in the conf | 21:16 |
| efried | Do we have such a thing? | 21:17 |
| efried | I thought we did. | 21:17 |
| esberglu | But there isn't any equivalent of that when calling compute.create_test_server | 21:17 |
| efried | oh, yeah, sorry, I was backwards. | 21:17 |
| efried | Bug? | 21:18 |
| *** apearson has quit IRC | 21:19 | |
| esberglu | I think so. Making sure there isn't something else missing on our side | 21:21 |
| efried | esberglu Other CIs aren't failing because.... they only have one network defined?? | 21:36 |
| efried | adreznec You got a problem with https://review.openstack.org/#/c/447762/ ? | 21:38 |
| adreznec | efried: Looking | 21:41 |
| *** jpasqualetto has quit IRC | 21:41 | |
| efried | adreznec And should we do that for the other projects? | 21:41 |
| efried | (he did it for networking-powervm - added you) | 21:42 |
| *** burgerk has quit IRC | 21:43 | |
| adreznec | efried: I guess it's fine. Not really a big deal to me | 21:46 |
| *** svenkat has joined #openstack-powervm | 21:46 | |
| adreznec | Why'd it fail CI? | 21:46 |
| efried | Yeah, I'm not seeing the problem with using http. | 21:46 |
| *** tblakes has quit IRC | 21:46 | |
| efried | Presumably because of the thing esberglu is looking into. | 21:46 |
| adreznec | Ah I see | 21:47 |
| adreznec | Hadn't read backscroll | 21:47 |
| *** jpasqualetto has joined #openstack-powervm | 21:57 | |
| *** tjakobs has quit IRC | 22:00 | |
| *** mdrabe has quit IRC | 22:04 | |
| *** k0da has quit IRC | 22:55 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!