Tuesday, 2016-11-29

*** esberglu has quit IRC00:12
*** esberglu has joined #openstack-powervm00:12
*** esberglu has quit IRC00:16
*** thorst_ has joined #openstack-powervm00:26
thorst_efried: The CI...its so pretty00:27
*** thorst_ has quit IRC00:27
*** kylek3h has joined #openstack-powervm00:31
*** seroyer has joined #openstack-powervm01:08
*** thorst_ has joined #openstack-powervm01:35
*** thorst_ has quit IRC01:41
*** seroyer has quit IRC01:52
*** thorst_ has joined #openstack-powervm02:39
*** thorst_ has quit IRC02:40
*** thorst_ has joined #openstack-powervm02:40
*** thorst_ has quit IRC02:41
*** mdrabe_ has joined #openstack-powervm02:58
*** svenkat has quit IRC03:05
*** tlian has quit IRC03:06
*** mdrabe_ has quit IRC03:06
*** svenkat has joined #openstack-powervm03:06
*** thorst_ has joined #openstack-powervm03:08
*** tlian has joined #openstack-powervm03:15
*** thorst_ has quit IRC03:16
*** tlian has quit IRC03:31
*** svenkat has quit IRC03:41
*** esberglu has joined #openstack-powervm03:41
*** esberglu has quit IRC03:46
*** apearson has joined #openstack-powervm03:48
*** apearson has quit IRC03:59
*** apearson has joined #openstack-powervm04:00
*** tjakobs has joined #openstack-powervm04:12
*** esberglu has joined #openstack-powervm04:12
*** thorst_ has joined #openstack-powervm04:14
*** jwcroppe has joined #openstack-powervm04:21
*** esberglu has quit IRC04:22
*** thorst_ has quit IRC04:22
*** esberglu has joined #openstack-powervm04:22
*** esberglu has quit IRC04:26
*** apearson has quit IRC05:03
*** thorst_ has joined #openstack-powervm05:20
*** thorst_ has quit IRC05:26
*** tjakobs has quit IRC05:57
*** thorst_ has joined #openstack-powervm06:25
*** jwcroppe has quit IRC06:30
*** thorst_ has quit IRC06:32
*** thorst_ has joined #openstack-powervm07:28
*** thorst_ has quit IRC07:36
*** esberglu has joined #openstack-powervm07:52
*** esberglu has quit IRC07:59
*** thorst_ has joined #openstack-powervm08:34
*** thorst_ has quit IRC08:41
*** esberglu has joined #openstack-powervm08:52
*** esberglu has quit IRC08:56
*** thorst_ has joined #openstack-powervm09:39
*** k0da has joined #openstack-powervm09:40
*** thorst_ has quit IRC09:47
*** esberglu has joined #openstack-powervm09:47
*** openstackgerrit has quit IRC09:48
*** openstackgerrit has joined #openstack-powervm09:49
*** esberglu has quit IRC09:51
*** esberglu has joined #openstack-powervm10:42
*** thorst_ has joined #openstack-powervm10:45
*** esberglu has quit IRC10:46
*** wangqwsh has joined #openstack-powervm10:48
*** thorst_ has quit IRC10:51
*** esberglu has joined #openstack-powervm11:37
*** esberglu has quit IRC11:41
*** thorst_ has joined #openstack-powervm11:49
*** thorst_ has quit IRC11:57
*** smatzek has joined #openstack-powervm12:00
*** esberglu has joined #openstack-powervm12:27
*** seroyer has joined #openstack-powervm12:28
*** esberglu has quit IRC12:31
*** svenkat has joined #openstack-powervm12:34
*** svenkat has quit IRC12:37
*** svenkat has joined #openstack-powervm12:37
*** thorst_ has joined #openstack-powervm12:49
*** tlian has joined #openstack-powervm12:55
thorst_efried: did you get neo-40 and 41?13:10
*** edmondsw has joined #openstack-powervm13:11
*** svenkat has quit IRC13:19
*** esberglu has joined #openstack-powervm13:22
*** esberglu has quit IRC13:26
*** esberglu has joined #openstack-powervm13:27
*** esberglu has quit IRC13:31
*** mdrabe_ has joined #openstack-powervm13:48
*** thorst_ has quit IRC13:54
*** svenkat has joined #openstack-powervm13:54
*** thorst_ has joined #openstack-powervm13:54
*** esberglu has joined #openstack-powervm14:01
thorst_CI scrum time?14:05
esbergluYeah14:06
esbergluadreznec: What's the right meeting name again?14:07
adreznecAh shit, sorry guys, my calendar invite didn't pop up14:08
esberglu#startmeeting powervm_driver_meeting14:08
openstackMeeting started Tue Nov 29 14:08:38 2016 UTC and is due to finish in 60 minutes.  The chair is esberglu. Information about MeetBot at http://wiki.debian.org/MeetBot.14:08
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.14:08
openstackThe meeting name has been set to 'powervm_driver_meeting'14:08
adreznecYep14:08
adreznecThat was it14:08
adreznec:P14:08
thorst_o/14:09
thorst_efried is off today I think14:09
esberglu#topic Status14:09
esbergluThe latest CI run went through last night with only 6 failures14:10
thorst_I'm reviewing those.  Some seem like legit nova failures14:10
esbergluRun in question if anyone needs it14:11
esbergluhttp://184.172.12.213/34/403934/1/check/nova-powervm-pvm-dsvm-tempest-full/57319c1/14:11
esbergluIt looked like most of them were the same failure14:11
esbergluI haven't looked into the logs yet though14:11
adreznecAh ok14:12
adreznecWas going to ask if any were new tests14:12
esbergluYeah one of them is a new one that I need to disable14:12
adreznecWere the runs that failed all against master?14:12
esbergluYeah14:12
thorst_yeah, this is a nova failure (at least one of them)14:13
thorst_I'll go investigate that.14:13
thorst_propose something up...14:13
thorst_o wait...no, this is on us14:13
esberglu#action esberglu: Disable unsupported test14:13
thorst_#action thorst: fix confirm_migration in nova_powervm14:13
adreznectest_delete_server seems like one that should pass...14:14
adreznecUnless the name is bad14:14
thorst_yeah, they're failing because of a bug14:14
thorst_they changed a driver signature14:14
adreznecAh14:14
thorst_we didn't notice (no CI  :-)14:14
adreznecYeah14:15
adreznecSorry, was still unzipping the n-cpu log14:15
thorst_nothing to worry about14:15
thorst_I got it14:15
adreznecOk14:15
thorst_hooray for CI though14:15
thorst_so, next steps are to get that fixed...see if we can't get a few solid runs through...14:16
thorst_then party?14:16
adreznecYou're sure this delete one is related? The 404 back from REST?14:16
esbergluYeah I think that's a different issue14:17
thorst_maybe not...but the tempest-DeleteServersTestJSON hit this14:17
thorst_TypeError: confirm_migration() takes exactly 4 arguments (5 given)14:17
thorst_and I see that 4 times in the logs14:17
thorst_each on different instances.14:17
esbergluAnd the 5th failure is the one I need to disable14:18
thorst_so its at least one of the things.14:18
adreznecRight14:18
esbergluSo all we have left is the first test_delete_server failure14:18
adreznecOk, lets start there then I guess14:18
esbergluWhich is something about a security group in use14:18
thorst_I wonder if that is a side effect of these other servers still existing though14:19
thorst_cause we failed to delete them14:20
thorst_and its just one of those concurrency things14:20
openstackgerritDrew Thorstensen (thorst) proposed openstack/nova-powervm: Update to new confirm_migration signature  https://review.openstack.org/40423414:20
esbergluCould be. Btw did you guys see that change for the arch import statement?14:21
esbergluWell at least thorst_ did14:22
adreznecI saw the review come in, didn't have a chance to look at it yet14:22
thorst_yeah, I'm ready to W+1 that14:22
thorst_adreznec: should take 2 seconds and would unwedge14:22
esbergluhttps://review.openstack.org/#/c/403934/14:22
esbergluAnd that's the patch this CI run is from14:23
adrezneclgtm14:23
*** tblakes has joined #openstack-powervm14:23
thorst_so what's next?14:25
thorst_I propose that we update the requests library in the tempest VMs, the novalinks, and the undercloud14:25
thorst_because efried told me we saw some of those requests bugs in a few runs a while back...so I'd like to squash that completely.14:25
adreznecthorst_: do we have enough confidence that's solving real issues?14:26
adreznecIf so, we should probably get our dependency updated14:26
thorst_adreznec: I think it is several issues14:26
adreznecOn requests >= 2.12.114:26
thorst_we have a fix from efried that we need to finish14:26
thorst_that's solid14:26
adreznecI was going to test that out today14:26
adreznecMake sure it doesn't break anything14:26
adreznecIf it doesn't I'll put up a nova-powervm req change for it14:27
thorst_the requests >= 2.12.1 has worked much better for kriskend's three runs of 10 deploys14:27
thorst_adreznec: do we want it in nova-powervm or just install it and instead pursue with glanceclient?14:27
adreznecWe could try14:27
thorst_this is one of those situations where its not us...14:27
adreznecThat would mean a g-r bump probably14:28
adreznecSo it could take while to filter through14:28
thorst_or a g-r exclude14:28
thorst_right14:28
thorst_but I think that's the right approach14:28
adreznecDo we think this is an issue specifically with 2.11.1?14:28
thorst_unclear14:28
thorst_that's definitely what people have when we hit the issue14:28
adreznecOk... probably need testing there then14:28
adreznecTo confirm if we need an exclude or a minimum bump14:28
thorst_yeah.  I'd say for now we just add it to the CI env and workaround from there14:29
openstackgerritMerged openstack/nova-powervm: Change arch to fields.Architecture  https://review.openstack.org/40393414:29
adreznecSure14:29
thorst_actually14:29
thorst_upper-constraints will kill us there14:29
thorst_they have it tagged to 2.11.114:29
thorst_so even if we put it in the CI, devstack will detect that and then update it14:30
thorst_so I think we need to have a g-r debate.14:30
thorst_#action thorst to open bug to get upper-constraints updated for requests package14:30
*** smatzek has quit IRC14:30
adreznecFun fun14:31
thorst_moving on   :-D14:31
esbergluI could remove the upper in the same way that we are removing the pypowervm constraints temporarily14:31
thorst_esberglu: lets wait it out and see if its bad14:31
esbergluOk14:31
thorst_I'd rather go through formal process14:31
thorst_any other items to discuss?  I'm assuming OSA progress has been limited due to Thanksgiving and then focus on stablizing core CI?14:32
esbergluI haven't done anything with it since before break14:32
adreznecYeah, I don't think there's been any movement on CI there14:32
thorst_wangqwsh: do you have any updates on it?14:33
adreznecand the only upstream patch I'm aware of is the one I have open for fixing the interfaces template14:33
wangqwshno, still block on crt_trunk_with_free_vlan14:33
thorst_the block there is?14:34
thorst_(sorry - my memory is hazy from holiday)14:34
wangqwshi am trying to boot a vm by manual, however the nova compute said KeyError: '06CC64F5-608A-4D4D-8C95-65B80CDFA96D'14:35
wangqwshi found this error was from pypowervm/tasks/cna.py - crt_trunk_with_free_vlan14:36
thorst_can you send a pastebin with the stack?14:36
thorst_not urgent, but I should probably dig into that14:36
wangqwshsure.14:37
thorst_I'm also thinking that wangqwsh should learn from esberglu how to redeploy the CI14:37
thorst_so that esberglu can take a break every once and a while (ex. go on holiday)14:37
wangqwshyes,14:38
*** edmondsw_ has joined #openstack-powervm14:38
esbergluEspecially since I am gonna be out for 2+ weeks after christmas14:38
wangqwshpleasure :)14:38
thorst_esberglu: Can you take an action to set up a jump box (that has the patches that aren't yet merged) and train wangqwsh on how to do the deploys?14:38
esberglu#action esberglu: Teach wangqwsh how to deploy CI14:39
thorst_wangqwsh: you'll get thrown in on this quickly, cause mainline CI is our highest priority (though we have a light at end of tunnel)14:39
wangqwshok14:39
thorst_cool14:41
esbergluthorst_: Once we get your change in, what projects should I open up the CI for? The *-powervm projects don't have much action right now, but the core projects will have a ton14:42
esbergluJust kick off a few runs and make sure they are alright and then start opening up to the core projects after?14:43
thorst_well, we want core once we've stablized14:43
thorst_three phases14:43
thorst_1) Stabilize in powervm projects14:43
thorst_2) Open up to other projects for higher velocity -> check results14:43
thorst_3) Publish logs (but do not vote) for nova project14:43
thorst_and for now, other projects should probably just be nova14:44
adreznecYeah14:44
adreznecI vote *-powervm first, then nova, then neutron/ceilometer14:45
adreznecthen OSA once we get the base CI stable14:45
thorst_I'd really like 3 being done by Jan 114:46
thorst_:-)14:46
thorst_if not sooner14:46
esbergluI think we can get it done before then14:46
thorst_optimism14:46
thorst_:-)14:46
esbergluAnything else?14:47
thorst_not from me14:47
thorst_lots of good progress though!14:47
adreznecYeah, that's all I've got14:48
adreznecGood to see things back running again :)14:48
esbergluMan I forgot to change the topic the whole meeting. Oh well14:48
esberglu#endmeeting14:48
openstackMeeting ended Tue Nov 29 14:48:39 2016 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)14:48
openstackMinutes:        http://eavesdrop.openstack.org/meetings/powervm_driver_meeting/2016/powervm_driver_meeting.2016-11-29-14.08.html14:48
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/powervm_driver_meeting/2016/powervm_driver_meeting.2016-11-29-14.08.txt14:48
openstackLog:            http://eavesdrop.openstack.org/meetings/powervm_driver_meeting/2016/powervm_driver_meeting.2016-11-29-14.08.log.html14:48
*** edmondsw_ has quit IRC14:53
*** edmondsw_ has joined #openstack-powervm14:53
*** edmondsw has quit IRC14:55
*** edmondsw_ has quit IRC14:55
*** jwcroppe has joined #openstack-powervm14:55
*** edmondsw has joined #openstack-powervm14:55
*** smatzek has joined #openstack-powervm14:57
thorst_adreznec: https://review.openstack.org/#/c/404234/15:00
thorst_we'll see what that does to CI15:00
thorst_but if its better...may need a quick +2 / W+115:00
adreznecAck15:03
*** wangqwsh has quit IRC15:10
esbergluthorst_: adreznec: The CI run didn't get through the stack...15:19
thorst_esberglu: ruh roh15:19
adreznecPanic!15:19
thorst_I'm going to re-run it15:21
thorst_that's odd15:21
esbergluIt's looking for a network named public. But we don't create the networks until after the stack15:21
esbergluIt hit the same thing on your first patch too15:21
thorst_devstack change?15:21
esbergluLooks like it15:22
esbergluhttps://github.com/openstack-dev/devstack/commit/42a21ec58dd9a5f4251aec34931844ac7dcdcd3615:23
esbergluHere's the error15:23
esberglu10:14:06 2016-11-29 15:14:33.681 | +++lib/tempest:configure_tempest:245        [m openstack network show -f value -c id public15:23
esberglu10:14:09 2016-11-29 15:14:36.068 | Error while executing command: No Network found for public15:23
*** tjakobs has joined #openstack-powervm15:25
*** apearson has joined #openstack-powervm15:30
thorst_are we back to changing devstack to create the networks, then we delete and create as we want15:31
esbergluthorst_: I could try it. But I think we stopped doing that because we were hitting issues with devstack failing to create the networks15:36
*** tlian has quit IRC15:46
*** tlian has joined #openstack-powervm15:53
thorst_hmm15:55
thorst_not sure what we want to do then...15:55
esbergluthorst_: I'm going into an instance manually right now and trying to have devstack create the networks15:55
thorst_k15:56
*** k0da has quit IRC16:17
esbergluthorst_: Got the same error with devstack creating the networks. It only created a private network. I'm made some local.conf changes and am trying again16:26
adreznecYeah...16:27
adreznecDevstack network config is a pain16:27
adreznecthorst_: what do we want to do with https://review.openstack.org/#/c/399372/116:27
adreznecIt's kind of... meh?16:27
adreznecI mean kind of correct since it's an official project16:27
openstackgerritMerged openstack/ceilometer-powervm: Show team and repo badges on README  https://review.openstack.org/40243916:31
thorst_adreznec: I'm OK with it.16:34
thorst_but github.com?16:34
thorst_is that right?16:34
adreznecSigh16:35
adreznecProbably not?16:35
adreznecNow what it should actually be is another question16:35
adreznecNormally its a docs.o.o link16:36
thorst_what does nova do16:36
adreznecdocs.o.o16:37
adreznecWhich doesn't exist for ceilometer-powervm16:37
thorst_hmm16:37
adreznecCould point to http://ceilometer-powervm.readthedocs.io/en/latest/16:37
adreznecI guess16:37
adreznecMore useful than github maybe16:38
thorst_yeah16:39
adreznecMade a review comment16:39
adreznecTrying to get through review backlog here16:40
thorst_:-)16:41
*** dwayne has quit IRC17:38
adreznecthorst_: esberglu are we good to merge in the devstack updates to the *-powervm projects that Eric put up?17:40
adreznecThat move the pypowervm install flow after driver setup?17:40
thorst_I think so17:40
thorst_otherwise the CI runs would've hit it17:41
thorst_the hold up there was that it get past CI17:41
esbergluI don't think CI runs have gone through on those patches17:41
thorst_o, then no?17:41
adreznecUh17:42
adreznecK17:42
adreznecYou guys had +1s on them17:42
adreznecSo figured I'd ask17:42
thorst_right17:42
adreznecThe patches were pretty straightforward17:42
thorst_yeah, we +1'd but when efried and I discussed we both wanted them to go through CI first17:43
esbergluYeah I don't see any reason they wouldn't work, just haven't actually tested in CI yet17:43
thorst_so maybe just +2 and I'll W+1 once we're through CI17:43
esbergluBut I'm still battling devstack17:43
thorst_we can wait  :-)17:51
*** apearson has quit IRC18:27
*** seroyer has quit IRC19:01
*** apearson has joined #openstack-powervm19:06
*** mdrabe_ has quit IRC19:33
*** mdrabe has joined #openstack-powervm19:41
*** k0da has joined #openstack-powervm19:46
*** dwayne has joined #openstack-powervm19:49
esbergluthorst_: I was able to get the AIO to stack, but I had to modify the neutron code to do so. Still looking for a config option to do what I'm trying to do20:05
thorst_modify the neutron code...that sounds bad20:09
adreznecesberglu: thorst_ so in this scenario we're trying to get devstack to create the networks for us, right?20:53
adreznec(just getting a chance to dig into this)20:54
thorst_adreznec: we had asked devstack earlier NOT to create the networks for us20:54
esbergluYeah. And we are creating them by hand after the stack20:54
adreznecSo which behavior do we want20:54
adreznecNot to create them?20:54
esbergluNot create them20:55
adreznecHave we tried setting NEUTRON_CREATE_INITIAL_NETWORKS=False20:55
esbergluYeah. We have that set to false.20:55
adreznecThat's supposed to disable creating both the private and external networks...20:56
adreznecWe're sure it's staying set?20:56
esbergluYep. So when we get to this part of devstack, it tries to find a public network, but there isn't one created20:56
esbergluhttps://github.com/openstack-dev/devstack/blob/master/lib/tempest#L24520:56
thorst_and that used to work20:56
thorst_doesn't anymore...so I was thinking this is just a devstack bug20:57
thorst_because most people use devstack to create the actual networks...20:57
adreznecYeah20:57
esbergluJust add an or there to check that NEUTRON_CREATE_INITIAL_NETWORKS is True?20:57
adreznecAnd we're not creating a public network here, only private, right?20:57
esbergluWe aren't creating either20:57
adreznecBecause we don't support floating IPs20:57
adreznecDon't we need a private network?20:58
esbergluWell devstack doesn't create either, we create both20:58
adreznecOh20:58
adreznecI see20:58
adreznecPost-stack though20:58
adreznecGot it20:58
esbergluYep20:58
thorst_so we can create them the way we want without contorting to some odd/weird devstack conf options20:59
thorst_we just want an empty openstack...20:59
adreznecRight21:00
adreznecDigging through the code here quick21:01
*** seroyer has joined #openstack-powervm21:01
adreznecYeah I found a bug where dragonflow is hitting this as well21:02
adreznecBut they worked around it with a hack21:02
thorst_shouldn't we just post it up and see what they say?  :-)21:03
adreznecpost what up21:03
thorst_a devstack review to add that check21:03
esbergluthorst_: I'm currently stacking with that check in place. If it works I can propose a change21:27
thorst_rockin21:27
thorst_and a bug report and all that jazz21:27
*** smatzek has quit IRC21:29
*** smatzek has joined #openstack-powervm22:02
*** smatzek has quit IRC22:06
*** tlian2 has joined #openstack-powervm22:11
*** tlian has quit IRC22:14
*** tlian2 has quit IRC22:15
*** jwcroppe has quit IRC22:16
*** jwcroppe has joined #openstack-powervm22:17
*** jwcroppe has quit IRC22:21
*** chas has joined #openstack-powervm22:30
*** tblakes has quit IRC22:31
*** dwayne has quit IRC22:32
*** chas has quit IRC22:34
*** seroyer has quit IRC22:38
*** edmondsw has quit IRC22:50
*** tlian has joined #openstack-powervm22:50
*** thorst_ has quit IRC22:53
*** tjakobs has quit IRC22:54
*** mdrabe has quit IRC23:03
*** esberglu has quit IRC23:04
*** svenkat has quit IRC23:04
*** esberglu has joined #openstack-powervm23:04
*** esberglu has quit IRC23:09
*** tjakobs has joined #openstack-powervm23:18
*** esberglu has joined #openstack-powervm23:25
*** esberglu has quit IRC23:41
*** thorst_ has joined #openstack-powervm23:42
*** thorst_ has quit IRC23:46
*** k0da has quit IRC23:51
*** thorst_ has joined #openstack-powervm23:56
*** thorst_ has quit IRC23:58
*** apearson has quit IRC23:58

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!