Wednesday, 2017-04-05

*** thorst has joined #openstack-powervm00:02
*** thorst has quit IRC00:07
*** svenkat has quit IRC00:22
*** thorst has joined #openstack-powervm00:34
*** thorst has quit IRC00:38
*** thorst has joined #openstack-powervm01:38
*** edmondsw has joined #openstack-powervm01:41
*** thorst has quit IRC01:43
*** edmondsw has quit IRC01:46
*** apearson has joined #openstack-powervm02:31
*** thorst has joined #openstack-powervm02:40
*** thorst has quit IRC02:58
*** efried has quit IRC03:04
*** efried has joined #openstack-powervm03:14
*** edmondsw has joined #openstack-powervm03:30
*** edmondsw has quit IRC03:34
*** apearson has quit IRC04:04
*** apearson has joined #openstack-powervm04:05
*** shyama has joined #openstack-powervm04:06
openstackgerritShyama proposed openstack/nova-powervm master: Update File i/o to support live-migration and use slot map  https://review.openstack.org/43232204:12
*** apearson has quit IRC04:35
*** thorst has joined #openstack-powervm04:55
*** thorst has quit IRC05:00
*** tonyb_ is now known as tonyb05:25
*** thorst has joined #openstack-powervm05:56
*** thorst has quit IRC06:01
*** thorst has joined #openstack-powervm06:57
*** thorst has quit IRC07:01
*** k0da has joined #openstack-powervm07:29
*** thorst has joined #openstack-powervm07:57
*** thorst has quit IRC08:17
*** thorst has joined #openstack-powervm09:14
*** thorst has quit IRC09:18
*** thorst has joined #openstack-powervm11:15
*** thorst has quit IRC11:19
*** thorst has joined #openstack-powervm11:41
*** smatzek has joined #openstack-powervm11:44
*** smatzek has quit IRC11:46
*** smatzek has joined #openstack-powervm11:46
*** edmondsw has joined #openstack-powervm12:10
efriedthorst Any idea how we bring https://review.openstack.org/#/c/453301/ to the attention of someone who can merge it?12:18
*** jpasqualetto has joined #openstack-powervm12:18
efried(Is tonyb such a one?)12:18
efriedthorst https://review.openstack.org/#/c/432322/ is good by me.  Need a final nod from you.12:20
*** jpasqualetto has quit IRC12:27
thorstefried: looking ... sorry for being slow...debugging another gigantic broadcast storm in our SVT env...12:31
thorstefried: on 453301, I'd say tonyb.12:35
efriedthorst Oh, tonyb is on medical leave.12:38
thorsthmm...not sure then12:39
*** jpasqualetto has joined #openstack-powervm12:39
thorstadreznec: ?12:39
efriedI'll ask sdague.  He's the one who mentioned eventlet changes.12:39
thorstahh, good idea.12:40
efriedthorst Still needing you to review https://review.openstack.org/#/c/453233/ when you get a chance.  As soon as that requirements change goes through, that's eligible.12:42
thorstwhat does IMAGE_API.download return back to us there?12:43
thorstI had actually reviewed it...but didn't hit +2 yet12:43
efriedThe iterable chunks thingy.12:43
efriedthorst https://github.com/openstack/nova/blob/master/nova/image/glance.py#L296 the result of that guy12:44
thorstthe chunky monkey...got it12:44
*** svenkat has joined #openstack-powervm12:44
efriedsdague +2ed the reqs change.12:47
openstackgerritMerged openstack/nova-powervm master: Update File i/o to support live-migration and use slot map  https://review.openstack.org/43232212:48
thorstnice12:48
efriedshyama ^^ FYI12:50
-openstackstatus- NOTICE: The Gerrit service on http://review.openstack.org is being restarted to address hung remote replication tasks, and should return to an operable state momentarily12:52
*** svenkat has quit IRC12:52
*** svenkat has joined #openstack-powervm12:52
BorD__quick question, if you select default processor compatibility mode with a POWER8 server, and you boot VIOS, what processor mode should it be running in?13:12
walerpower713:13
adreznecBorD__: I believe it runs in POWER7 proc compat mode by default13:13
BorD__is there a reason for that?13:13
waleryep13:14
walerit's based on AIX6.113:14
adreznecYep, that's the reason13:14
walerAIX 6.1 supported POWER8, but didn't exploit it (whereas 7.x did)13:15
BorD__thanks!  on another note, have you seen a concise document that gives HMC and NIM ports for use through a firewall?  I have seen http://www-01.ibm.com/support/docview.wss?uid=isg3T1011808 and http://www-01.ibm.com/support/docview.wss?uid=nas8N101911113:16
BorD__I was hoping for something I could just hand off to firewall guys instead of making them read all of this.13:17
*** mdrabe has joined #openstack-powervm13:17
*** esberglu has joined #openstack-powervm13:18
esbergluefried: thorst: Production CI spawned the ready nodes, but the local2remote stuff is broken13:18
efriedesberglu Right, I shoulda thought of that.  Let me get you a fix right quick.13:19
efriedesberglu Wait, when you say it's broken, do you mean the patch won't apply, or it applies but doesn't work?13:20
esbergluefried: It applies but doesn't work13:22
efriedesberglu What's the symptom13:22
esbergluefried: Something about the novalink packages not being installed. One sec and I can get the actual message13:23
esbergluUnable to connect to the NovaLink services. Check to make sure the pvm-core and pvm-rest services are running (systemctl status), and verify that this partition has PowerVM management authority. If you are currently updating pvm-novalink, you may see this error. Wait for the update to finish, then try again.13:24
esbergluWhen trying to do pvmctl commands13:25
efriedesberglu Are you sure you're updating the right copy of pypowervm?13:25
efriedNote that 1.1.1 was published last night, if that makes a difference to ya.13:25
esbergluNah this is on develop13:25
*** tjakobs__ has joined #openstack-powervm13:25
walerBorD__, that NIM URL has a good breakdown list a fair way into the document. The HMC one is a bit more catch-all, as it can depend on what you hoping to work "behind the firewall."13:25
BorD__yeah I agree, however I am running into firewall guys that do not want to read, and i was hoping i could just point to an ibm blessed page that just had the list.  We will make do.  Thanks again!13:26
esbergluefried: I rebased the patch last night but that made no difference13:30
efriedesberglu Yeah, it shouldn't have been a merge conflict.13:31
efriedDo you have a system where this is happening?13:31
efriedesberglu Yeah, so pvmctl is actually using the pypowervm in /usr/local/lib, not the one in /opt/stack13:35
efriedjenkins@powervm-ci-powervm-devstacked-9356:/home/powervmci$ python -c 'import pypowervm; print pypowervm.__file__'13:36
efriedokeeeeey, not sure why the second line isn't printing.13:36
efriedAlso pip show pypowervm13:37
efriedIf you're gonna clone a pypowervm into /opt/stack, you've gotta pip install -e it.13:37
esbergluThe one in /usr/local/lib has the patch13:37
efriedeaux13:37
openstackgerritMerged openstack/nova-powervm master: Updated from global requirements  https://review.openstack.org/45104513:39
*** jpasqualetto_ has joined #openstack-powervm13:43
*** jpasqualetto has quit IRC13:43
*** jpasqualetto_ has quit IRC13:44
*** jpasqualetto has joined #openstack-powervm13:44
*** smatzek has quit IRC13:46
*** tjakobs__ has quit IRC13:59
a1fisherHi Guys, quick question on running the powervm-sea-agent. I'm assuming I need to run 'networking-powervm-sea-agent'? If so I'm getting a14:08
a1fishersorry an error "MultiBridgeNoMapping: The system has more than one network bridge, but the bridge_mappings have not been specified.  Please configure the bridge_mappings before proceeding." I have bridge_mapping set in the mls ini file.14:09
*** smatzek has joined #openstack-powervm14:10
thorsta1fisher: paste your bridge_mapping (its a slightly different format).  Also make sure you specify --config-file when you start the process14:10
*** nbante has joined #openstack-powervm14:11
a1fisherAh, looks like it was the --config-file option that was missing. Seems to be working with ./networking-powervm-sea-agent --config-file /etc/neutron/plugins/ml2/ml2_conf.ini14:17
a1fisherDo you guys run the agent as a service normally?14:17
thorsta1fisher: all depends on the distribution...but I believe most run as a service14:18
thorstour devstacks run as a screen...but that's generally only useful for development and is awful for the real world14:18
a1fisherok, thanks for your help. On to the next issue :-)14:18
thorstheh14:21
*** apearson has joined #openstack-powervm14:25
walera1fisher, I'm sure once we get the first compute integrated, things will become more obvious how things are supposed to work together (he says being optimistic)14:26
*** shyama has quit IRC14:34
thorstwaler: what could go wrong14:41
thorst;-)14:41
efriedthorst  https://review.openstack.org/453301 is merging now.  Since https://review.openstack.org/453233 has a Depends-On, it could get +W and it would wait behind that guy anyway.14:47
efriedThough I wouldn't mind seeing the latter go through CI first (esberglu)14:47
thorstefried: yeah, maybe lets let it run through CI?14:49
esbergluefried: The ready nodes are about to start spawning. So assuming installing pvm-rest-server-dev fixed the last of the problems (I think it did)14:49
esbergluWe should be able to get runs through within the hour14:49
thorsthttps://pbs.twimg.com/media/Bp11KmKCUAAp9pz.jpg14:50
*** efried has quit IRC14:52
*** shyama has joined #openstack-powervm14:56
*** tjakobs__ has joined #openstack-powervm15:00
*** mdrabe has quit IRC15:13
*** mdrabe has joined #openstack-powervm15:20
esbergluefried: thorst: CI is back up15:28
thorstesberglu: exceeding expectations once again15:32
thorstI'll kick off a recheck since efried has gone AWOL15:33
*** chas has joined #openstack-powervm15:33
*** chas has quit IRC15:35
thorstesberglu: didn't the recheck command change recently?  Its not 'recheck powervm' anymore?15:35
esberglupowervm: recheck15:35
*** chas has joined #openstack-powervm15:36
thorstmakes sense15:36
esbergluor powevm-recheck, powervm recheck, powervm:recheck15:36
esbergluWith any sort of whitespace in between15:36
esbergluBecause recheck powervm was kicking off rechecks for pretty much everything15:36
thorstyep15:37
thorstI remembered that bit  :)15:37
esbergluRuns are failing right now, one of the pypowervm patches we apply needs a rebase15:38
esbergluI'm fixing it now15:38
thorstesberglu: ok15:38
*** chas has quit IRC15:40
*** k0da has quit IRC15:42
esbergluthorst: I'm having a weird issue trying to push the rebase up15:42
esbergluhttp://paste.openstack.org/show/605521/15:43
esbergluIt says 4726 is closed15:43
esbergluBut the actual change is number 472715:43
esbergluthorst: Worked around it15:47
esbergluShould be back in business now15:48
openstackgerritEric Berglund proposed openstack/nova-powervm master: DNM: ci check  https://review.openstack.org/32831515:51
*** nbante has quit IRC15:53
*** efried has joined #openstack-powervm16:07
*** chas has joined #openstack-powervm16:11
efriedesberglu We shouldn't need 4727 (Abort the fifo pipe) anymore.16:13
*** chas has quit IRC16:15
efriedesberglu Just kill it.16:17
esbergluYeah I was just about to ask you about that16:17
efriedI'm proposing.16:19
efriedesberglu 509716:19
efriedUntil https://review.openstack.org/#/c/453233/ merges, OOT CI runs (on unrelated changes) may be broken.  But they woulda been anyway.16:20
efriedFortunately (!?) you got a reprieve because https://review.openstack.org/#/c/453301/ glitched in the gate while merging.16:21
*** chas has joined #openstack-powervm16:21
*** shyama has quit IRC16:23
*** esberglu has quit IRC16:24
*** esberglu has joined #openstack-powervm16:24
*** chas has quit IRC16:26
*** esberglu has quit IRC16:29
*** esberglu has joined #openstack-powervm16:43
*** apearson has quit IRC16:46
*** apearson has joined #openstack-powervm16:48
*** apearson has quit IRC16:50
*** apearson has joined #openstack-powervm16:52
*** shyama has joined #openstack-powervm16:56
*** apearson has quit IRC17:19
esbergluthorst: adreznec: Man CI just isn't cooperating right now. Nodepool is failing to detect and delete nodes that have been used already17:21
thorstis the cleanup just not working?17:21
esbergluI think it is a problem with nodepool, not with our post run cleanup script. If that's what you mean17:25
esbergluNormally post run nodepool detects that the run completed and deletes the node from jenkins then deletes the instance17:25
esberglu(at least I'm pretty sure it's nodepool responsible for that)17:26
*** dwayne__ has quit IRC17:27
*** apearson has joined #openstack-powervm17:33
esbergluWe aren't on the latest nodepool, I wonder if we need to upgrade. Looking into what it takes to move from 0.3.0 to 0.4.017:35
thorstesberglu: we do have those in VMs and we could backup the image.17:36
esbergluthorst: Have what in vms?17:37
thorstesberglu: the nodepool17:39
esbergluI'm not following what you mean17:39
thorstbasically, I could snapshot the VM.  Then if you screw up nodepool we could revert17:46
thorst(although, we have playbooks to rebuild the nodepool, so it doesn't really matter)17:46
esbergluYeah I was just gonna do a management redeploy17:47
thorstahh, OK17:48
esbergluLooks like we have to set up a "zookeeper cluster" with nodepool 0.4.017:48
esbergluNo idea what that entails17:48
esbergluReading through the email chain from the 0.4.0 release17:48
adreznecesberglu: should we just pin on an older release for now?18:00
esbergluWe are pinned on an older release18:00
esberglu0.3.018:00
adreznecAh ok18:01
adreznecthat's what I thought18:01
esbergluIdk why it suddenly isn't working18:01
adreznecI figured maybe we unpinned or something since you talked about 0.4.018:01
esbergluI wonder if something just got messed up during the deploy. I'm gonna try redeploying mgmt while I look into 0.4.018:02
adreznecSeems reasonable18:03
*** jpasqualetto has quit IRC18:06
esbergluI'm killing all of the current runs. Shouldn't affect anyone since they were gonna fail anyway until 5099 gets in18:07
*** dwayne__ has joined #openstack-powervm18:20
*** jpasqualetto has joined #openstack-powervm18:22
*** jpasqualetto has quit IRC18:34
*** chas has joined #openstack-powervm18:36
*** chas has quit IRC18:40
*** jpasqualetto has joined #openstack-powervm18:50
thorstefried: see comment on 509619:11
esbergluStill trying to get the CI back up, just one thing after another at this point19:15
esbergluNow mysql won't start on the mgmt node19:15
*** shyama has quit IRC19:26
*** k0da has joined #openstack-powervm19:30
*** mdrabe_ has joined #openstack-powervm19:30
efriedthorst 5096 back atcha.19:31
*** mdrabe has quit IRC19:31
*** smatzek has quit IRC19:40
*** apearson has quit IRC19:57
*** apearson has joined #openstack-powervm20:03
*** smatzek has joined #openstack-powervm20:07
esbergluI'm hoping that the issues deleting the nodes was somehow tied into the mysql issue20:07
esbergluI got past the mysql thing so *fingers crossed* CI is coming back up20:08
*** mdrabe__ has joined #openstack-powervm20:36
*** mdrabe_ has quit IRC20:37
*** mdrabe__ is now known as mdrabe20:47
openstackgerritOpenStack Proposal Bot proposed openstack/nova-powervm master: Updated from global requirements  https://review.openstack.org/45386520:49
*** thorst has quit IRC20:50
*** thorst has joined #openstack-powervm20:51
*** thorst has quit IRC20:55
efriedadreznec Guess we take ^^ and remove same from 45323321:00
*** svenkat has quit IRC21:01
adreznecefried: Yup, seems reasonable21:04
openstackgerritMerged openstack/nova-powervm master: Updated from global requirements  https://review.openstack.org/45386521:07
*** k0da has quit IRC21:12
*** smatzek has quit IRC21:13
*** thorst has joined #openstack-powervm21:18
*** k0da has joined #openstack-powervm21:20
*** thorst has quit IRC21:22
efriedadreznec esberglu (tjakobs__ thorst): https://review.openstack.org/#/q/topic:bug/1679720 -- removed requirements.txt change, manually cherry-picked back to ocata & newton.  tjakobs__ and I tested the backports with stock pypowervm relevant to each release; and with 1.1.1.21:40
esbergluAwesome21:40
esbergluI'm still trying to figure out why nodepool won't delete the damn nodes anymore21:42
esbergluBut your runs should go through21:42
esberglu*are going through21:42
efriedgood21:43
esbergluJust hope they pass first time through lol21:44
*** thorst has joined #openstack-powervm21:53
*** efried has quit IRC22:07
*** thorst has quit IRC22:12
*** k0da has quit IRC22:14
*** apearson has quit IRC22:16
*** edmondsw has quit IRC22:22
*** mdrabe has quit IRC22:23
*** edmondsw has joined #openstack-powervm22:24
*** edmondsw has quit IRC22:29
*** chas has joined #openstack-powervm22:36
*** chas has quit IRC22:41
*** tjakobs__ has quit IRC22:53
*** chas has joined #openstack-powervm22:57
*** thorst has joined #openstack-powervm22:59
*** chas has quit IRC23:01
*** thorst has quit IRC23:04
*** chas has joined #openstack-powervm23:33
*** thorst has joined #openstack-powervm23:34
*** chas has quit IRC23:38
*** apearson has joined #openstack-powervm23:48
*** thorst has quit IRC23:49
*** apearson has quit IRC23:50
*** apearson has joined #openstack-powervm23:51
*** apearson has quit IRC23:52

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!