Monday, 2015-08-17

*** sdake_ has joined #tripleo00:04
*** sdake has quit IRC00:07
*** sdake has joined #tripleo00:20
*** shadower has quit IRC00:23
*** shadower has joined #tripleo00:23
*** sdake_ has quit IRC00:23
*** olaph has joined #tripleo00:38
*** Goneri has quit IRC00:43
*** mestery has joined #tripleo00:59
*** yamahata has joined #tripleo01:27
*** mestery has quit IRC01:36
*** mestery has joined #tripleo01:36
*** mestery has quit IRC01:37
*** yamahata has quit IRC01:47
*** al has quit IRC01:59
*** al has joined #tripleo02:01
*** panda has quit IRC02:09
*** panda has joined #tripleo02:10
*** aukhan has joined #tripleo02:41
*** untriaged-bot has joined #tripleo03:00
untriaged-botUntriaged bugs so far:03:00
untriaged-bothttps://bugs.launchpad.net/os-collect-config/+bug/148251003:00
openstackLaunchpad bug 1482510 in heat "OS::Heat::SoftwareDeployment failed due SSL certificate verification error" [Medium,Triaged] - Assigned to Rico Lin (rico-lin)03:00
untriaged-bothttps://bugs.launchpad.net/diskimage-builder/+bug/146603703:00
openstackLaunchpad bug 1466037 in diskimage-builder "Signed Fedora and Ubuntu user image built by DIB can`t boot on HP DL380 Gen8 server for lack of mpt2sas driver" [Undecided,Incomplete]03:00
uvirtbotLaunchpad bug 1482510 in os-collect-config "OS::Heat::SoftwareDeployment failed due SSL certificate verification error" [Undecided,New]03:00
uvirtbotLaunchpad bug 1482510 in os-collect-config "OS::Heat::SoftwareDeployment failed due SSL certificate verification error" [Undecided,New] https://launchpad.net/bugs/148251003:00
untriaged-bothttps://bugs.launchpad.net/diskimage-builder/+bug/148338503:00
openstackLaunchpad bug 1483385 in diskimage-builder "install_grub failing for centos7" [Undecided,In progress] - Assigned to Abel Lopez (al592b)03:00
uvirtbotLaunchpad bug 1466037 in diskimage-builder "Signed Fedora and Ubuntu user image built by DIB can`t boot on HP DL380 Gen8 server for lack of mpt2sas driver" [Undecided,Incomplete]03:00
untriaged-bothttps://bugs.launchpad.net/diskimage-builder/+bug/147180203:00
openstackLaunchpad bug 1471802 in diskimage-builder "ironic-agent element hardcodes interfaces names for DHCP." [Undecided,Fix committed] - Assigned to Om Kumar (om-kumar)03:00
uvirtbotLaunchpad bug 1466037 in diskimage-builder "Signed Fedora and Ubuntu user image built by DIB can`t boot on HP DL380 Gen8 server for lack of mpt2sas driver" [Undecided,Incomplete] https://launchpad.net/bugs/146603703:00
uvirtbotLaunchpad bug 1483385 in diskimage-builder "install_grub failing for centos7" [Undecided,In progress]03:00
*** untriaged-bot has quit IRC03:00
uvirtbotLaunchpad bug 1483385 in diskimage-builder "install_grub failing for centos7" [Undecided,In progress] https://launchpad.net/bugs/148338503:00
uvirtbotLaunchpad bug 1471802 in diskimage-builder "ironic-agent element hardcodes interfaces names for DHCP." [Undecided,Fix committed]03:00
uvirtbotLaunchpad bug 1471802 in diskimage-builder "ironic-agent element hardcodes interfaces names for DHCP." [Undecided,Fix committed] https://launchpad.net/bugs/147180203:00
*** al has quit IRC03:08
*** al has joined #tripleo03:10
*** openstack has joined #tripleo04:19
*** masco has joined #tripleo05:20
*** jprovazn has joined #tripleo05:52
*** bvandenh has joined #tripleo06:06
*** lsmola has joined #tripleo06:12
*** Marga_ has joined #tripleo06:12
*** Marga_ has quit IRC06:13
*** Marga_ has joined #tripleo06:14
*** Marga_ has quit IRC06:17
*** Marga_ has joined #tripleo06:17
*** ifarkas has joined #tripleo06:40
*** sdake_ has joined #tripleo06:44
*** sdake has quit IRC06:47
marios" It's just you. http://review.openstack.org is up. "  :/06:54
-openstackstatus- NOTICE: Gerrit is currently under very high load and may be unresponsive. infra are looking into the issue.07:07
*** sdake_ has quit IRC07:08
mariosso perhaps not just me then07:09
*** jtomasek has joined #tripleo07:09
*** pblaho has joined #tripleo07:15
openstackgerritYanis Guenane proposed openstack/tripleo-heat-templates: [test] Ensuring ha job is working when stonith is fully disabled  https://review.openstack.org/21256607:25
*** pblaho has quit IRC07:32
*** pblaho has joined #tripleo07:32
*** sthillma has joined #tripleo07:35
*** aufi has joined #tripleo07:35
*** sthillma_ has joined #tripleo07:36
*** matbu has joined #tripleo07:38
*** sthillma has quit IRC07:39
*** sthillma_ is now known as sthillma07:39
*** yog_ has joined #tripleo07:41
*** sthillma_ has joined #tripleo07:41
*** sthillma has quit IRC07:44
*** sthillma_ is now known as sthillma07:44
*** lucasagomes has joined #tripleo07:53
*** stendulker has joined #tripleo07:55
*** matbu has quit IRC07:57
*** matbu has joined #tripleo08:02
*** matbu has quit IRC08:07
*** sthillma has quit IRC08:07
*** shardy has joined #tripleo08:09
*** derekh has joined #tripleo08:14
derekhAnybody looking at the failures in devtest/CI ?08:15
spredzyderekh, I've been investigating the HA job failure for few days08:19
spredzyno luck yet08:19
spredzyAs it only happens in the CI but not on my setup :/08:19
derekhspredzy: ok, I'm gonna try the nonha job first to see if I can figure that one out08:20
spredzyIssue are : Sometimes cluster not forming itself. Sometimes it does but Galera can't create the cluster08:20
spredzyderekh, I think puppet-nonha job was green last time I checked08:21
*** jistr has joined #tripleo08:21
* spredzy dealing with gerrit atm is really painful08:21
derekhspredzy: yup it is, the one I'm looking at is overcloud-f21-nonha08:21
*** matbu has joined #tripleo08:28
*** shardy_ has joined #tripleo08:33
*** shardy has quit IRC08:34
*** matbu has quit IRC08:35
*** adrianopetrich has quit IRC08:36
*** matbu has joined #tripleo08:37
*** shardy_ has quit IRC08:38
*** shardy has joined #tripleo08:39
*** mcornea has joined #tripleo08:39
*** mbound has joined #tripleo08:47
*** gfidente has joined #tripleo08:48
*** regebro has joined #tripleo08:57
*** untriaged-bot has joined #tripleo09:00
untriaged-botUntriaged bugs so far:09:00
untriaged-bothttps://bugs.launchpad.net/os-collect-config/+bug/148251009:00
openstackLaunchpad bug 1482510 in heat "OS::Heat::SoftwareDeployment failed due SSL certificate verification error" [Medium,Triaged] - Assigned to Rico Lin (rico-lin)09:00
untriaged-bothttps://bugs.launchpad.net/diskimage-builder/+bug/146603709:00
uvirtbotLaunchpad bug 1482510 in os-collect-config "OS::Heat::SoftwareDeployment failed due SSL certificate verification error" [Undecided,New]09:00
openstackLaunchpad bug 1466037 in diskimage-builder "Signed Fedora and Ubuntu user image built by DIB can`t boot on HP DL380 Gen8 server for lack of mpt2sas driver" [Undecided,Incomplete]09:00
untriaged-bothttps://bugs.launchpad.net/diskimage-builder/+bug/148338509:00
uvirtbotLaunchpad bug 1466037 in diskimage-builder "Signed Fedora and Ubuntu user image built by DIB can`t boot on HP DL380 Gen8 server for lack of mpt2sas driver" [Undecided,Incomplete]09:00
uvirtbotLaunchpad bug 1482510 in os-collect-config "OS::Heat::SoftwareDeployment failed due SSL certificate verification error" [Undecided,New] https://launchpad.net/bugs/148251009:00
openstackLaunchpad bug 1483385 in diskimage-builder "install_grub failing for centos7" [Undecided,In progress] - Assigned to Abel Lopez (al592b)09:00
untriaged-bothttps://bugs.launchpad.net/diskimage-builder/+bug/147180209:00
openstackLaunchpad bug 1471802 in diskimage-builder "ironic-agent element hardcodes interfaces names for DHCP." [Undecided,Fix committed] - Assigned to Om Kumar (om-kumar)09:00
uvirtbotLaunchpad bug 1466037 in diskimage-builder "Signed Fedora and Ubuntu user image built by DIB can`t boot on HP DL380 Gen8 server for lack of mpt2sas driver" [Undecided,Incomplete] https://launchpad.net/bugs/146603709:00
uvirtbotLaunchpad bug 1483385 in diskimage-builder "install_grub failing for centos7" [Undecided,In progress]09:00
*** untriaged-bot has quit IRC09:00
uvirtbotLaunchpad bug 1483385 in diskimage-builder "install_grub failing for centos7" [Undecided,In progress] https://launchpad.net/bugs/148338509:00
uvirtbotLaunchpad bug 1471802 in diskimage-builder "ironic-agent element hardcodes interfaces names for DHCP." [Undecided,Fix committed]09:00
uvirtbotLaunchpad bug 1471802 in diskimage-builder "ironic-agent element hardcodes interfaces names for DHCP." [Undecided,Fix committed] https://launchpad.net/bugs/147180209:00
*** pelix has joined #tripleo09:06
*** Marga_ has quit IRC09:08
*** Marga_ has joined #tripleo09:08
*** kbyrne has quit IRC09:12
*** kbyrne has joined #tripleo09:17
*** bvandenh has quit IRC09:18
*** akrivoka has joined #tripleo09:19
*** athomas has joined #tripleo09:23
*** akrivoka has quit IRC09:34
*** akrivoka has joined #tripleo09:36
*** Guest47951 is now known as d0ugal09:49
*** d0ugal has quit IRC09:49
*** d0ugal has joined #tripleo09:49
*** Marga_ has quit IRC09:54
*** Marga_ has joined #tripleo09:55
*** panda has quit IRC10:09
*** panda has joined #tripleo10:10
*** leanderthal has quit IRC10:20
*** leanderthal has joined #tripleo10:22
-openstackstatus- NOTICE: review.openstack.org (aka gerrit) is going down for an emergency restart10:23
*** ChanServ changes topic to "review.openstack.org (aka gerrit) is going down for an emergency restart"10:23
*** bvandenh has joined #tripleo10:39
*** ChanServ changes topic to "CI failing on https://bugs.launchpad.net/tripleo/+bug/1483706 and https://bugs.launchpad.net/tripleo/+bug/1482195 | Deploying OpenStack Using OpenStack | https://wiki.openstack.org/wiki/TripleO"10:50
-openstackstatus- NOTICE: Gerrit restart has resolved the issue and systems are back up and functioning10:50
*** yog_ has quit IRC10:56
spredzyderekh, ping10:59
spredzyderekh, would you happen to know if there are any kind of multicast filtering in our CI system ?10:59
*** regebro has quit IRC11:00
*** regebro has joined #tripleo11:01
derekhspredzy: none that I'm aware of11:02
*** Marga_ has quit IRC11:02
spredzyderekh, ack. I am starting to run out of idea then :)11:07
*** aukhan has quit IRC11:11
derekhspredzy: I'll see if I can set you up a VM on the ci cloud to reproduce11:12
*** paramite has joined #tripleo11:13
spredzyderekh, thanks that would be awesome. Question: Can devtest run in a VM ?11:13
spredzyWhen I tried months ago it has to be run on a baremetal AFAIK11:13
derekhspredzy: it can run in a VM but needs to control VM's that run on baremetal (i.e. the undercloud/overcloud can't be nested virt instances)11:15
spredzyack no nested virt. That was what I tried11:15
spredzythanks for clearing this out11:16
derekhspredzy: can you point me at a public key for you, I've spun you up a VM11:17
derekhspredzy: I'll then start a screen session to show you what I'm going to do11:17
spredzyderekh, ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+ZFQv3MyjtL1BMpSA0o0gIkzLVVC711rthT29hBNeORdNowQ7FSvVWUdAbTq00U7Xzak1ANIYLJyn+0r7olsdG4XEiUR0dqgC99kbT/QhY5mLe5lpl7JUjW9ctn00hNmt+TswpatCKWPNwdeAJT2ERynZaqPobENgvIq7jfOFWQIVew7qFeZygxsPVn36EUr2Cdq7Nb7U0XFXh3x1p0v0+MbL4tiJwPlMAGvFTKIMt+EaA+AsRIxiOo9CMk5ZuOl9pT8h5vNuEOcvS0qx4v44EAD2VOsCVCcrPNMcpuSzZP8dRTGU9wRREAWXngD0Zq9YJMH38VTxHiskoBw1NnPz spredzy@murcia.yanisguenane.11:18
spredzyack11:18
derekhspredzy: got it11:18
derekhssh fedora@66.187.229.6611:18
derekhspredzy: ^11:18
spredzyderekh, in11:19
derekhspredzy: ok, screen -x11:19
spredzyderekh, in11:19
derekhspredzy: this is a vanilla f21 cloud instance, the first thing that infra do is install a bunch of stuff to make it a nodepool template, one sec and I'll make a script11:20
*** mburned_` is now known as mburned_out11:20
*** mburned_out is now known as mburned11:20
*** stendulker has quit IRC11:20
derekhspredzy: essentially this gets run11:21
derekhhttp://paste.openstack.org/show/419081/11:21
spredzyderekh, ack (looking at the screen session)11:22
derekhspredzy: hmm, running it twice is a problem, gonna remove a few things11:23
derekhspredzy: ok, so amunst other things that clones a shed load of git repositories, its takes about 30 minutes (maybe a little longer, so we can get back to this once its finished)11:25
spredzyok11:25
spredzyderekh, ^11:25
*** lucasagomes is now known as lucas-hungry11:28
*** shardy_ has joined #tripleo11:43
*** shardy has quit IRC11:45
*** Marga_ has joined #tripleo11:49
*** shardy_ has quit IRC11:49
*** shardy has joined #tripleo11:49
spredzyderekh, ping. process seems over11:53
*** rhallisey has joined #tripleo11:53
derekhspredzy: yup, ok so what normally happens now (in nodepool) is that a snapshot is taken, this snapshot is what all CI tests will start from  (stop me if I'm telling you things you already know)11:54
spredzyderekh, nop at all I am tootally unfamiliar with this process11:55
*** funzo has joined #tripleo11:55
derekhspredzy: ok, I've SU'd to jenkins, everything else will run as jenkins11:55
spredzyderekh, ok11:56
derekhspredzy: we also need to configure eth111:56
derekhspredzy: eth1 is on a special "test" network, it will alowe us to talk to bm host that are used to being up instances11:58
spredzyderekh, ack11:58
*** chlong has quit IRC11:59
derekhspredzy: so, nearly there ;-) , nodepool/zuul kicks off jobs with a load of job specific stuff defined11:59
*** adrianopetrich has joined #tripleo11:59
*** funzo has quit IRC12:00
spredzyderekh, what is the last command you ran. It went out pretty fast :/12:02
*** adrianopetrich_ has joined #tripleo12:02
derekhspredzy: I essentiall went through this (I had it in some notes) http://paste.openstack.org/show/419108/12:02
derekhspredzy: and changed the joba name to the ha job12:03
derekhspredzy: this sets things up and then calls "gate_hook",12:03
derekhspredzy: I've stubbed out gate_hook so we can run it seperatly in a minute12:04
spredzyderekh, ok. I think it failed atm12:04
derekhspredzy: checking12:04
*** adrianopetrich has quit IRC12:04
spredzyderekh, timeout -s 9 m vs. timeout -s 9m12:06
spredzyno ?12:06
spredzywell I see some command not found above also12:06
derekhspredzy: yup possibly but I *think* its ok in this case12:06
derekhspredzy: so I'm going to try the tripleo command to see how it goes12:07
derekhspredzy: nope, I'm wrong its not ok, /opt/stack/new wasn't set up12:08
derekhspredzy: I gotta pop away for about 30 minutes can we reconvene after that ? (or at some stage that suits you?)12:09
spredzysure12:09
spredzyderekh, ping me when you're back12:09
derekhspredzy: will do12:10
spredzythx12:10
*** jayg|g0n3 is now known as jayg12:15
*** shardy_ has joined #tripleo12:28
*** shardy has quit IRC12:29
*** lucas-hungry is now known as lucasagomes12:31
*** shardy_ has quit IRC12:33
*** rbrady has joined #tripleo12:34
*** shardy has joined #tripleo12:34
*** noslzzp has joined #tripleo12:41
*** Marga_ has quit IRC12:47
*** matbu has quit IRC12:47
*** adrianopetrich_ has quit IRC12:51
*** matbu has joined #tripleo12:51
*** dprince has joined #tripleo12:54
*** Marga_ has joined #tripleo12:56
*** sdake has joined #tripleo13:00
derekhspredzy: back, just taking a look now to see what I screwed up13:01
spredzyderekh, ack I am around13:01
derekhspredzy: I recloned openstack-infra/devstack-gate into $WORKSPACE13:04
*** adrianopetrich_ has joined #tripleo13:05
derekhspredzy: I must have done it wrong the first time13:05
*** matbu has quit IRC13:05
spredzyderekh, ok let see where it leads us now13:05
*** lifeless has quit IRC13:05
derekhspredzy: ok, that worked, moving into the tripleo-ci directory to run tripleo ci ha job13:08
*** rlandy has joined #tripleo13:08
derekhspredzy: when a ci job finishes the instances are fried up for another CI job, that sleep keeps them around until you kill the script13:09
spredzyderekh, ack13:09
spredzyderekh, so now the regular CI job is running, correct ?13:10
derekhspredzy: then we run this command, the overcloud-puppet ha ci job is running13:10
derekhspredzy: yup13:10
spredzyso once it is done - failed mostly - system won't be tear down by the sleep you just put in13:11
spredzyderekh, some from there I'll be able to debug/break/change things13:11
derekhspredzy: exactly, it will only be released once the sleep is finished/killed13:11
spredzyderekh, yeah thats about 16h13:12
spredzythats gives me plenty of time :)13:12
spredzyderekh, thanks for setting this up for me13:12
derekhspredzy: you might need to change one or two things so the script can be rerun (not all of it is idepotent) but that should esentially be it13:12
derekhspredzy: no prob, image download seems slow but lets see how far this gets and we can see if I've forgotten anything ;-)13:13
*** devvesa has joined #tripleo13:14
*** sdake_ has joined #tripleo13:15
derekhhmmm, image download isn't usually this slow13:16
*** karume has joined #tripleo13:16
derekhspredzy: its getting slower, going to cancel it and start over13:17
spredzyderekh, ok13:17
derekhspredzy: going to change a few things so it can be rerun multiple times13:17
*** yog_ has joined #tripleo13:18
*** sdake has quit IRC13:19
*** sdake_ has quit IRC13:19
derekhspredzy: 3 changes there, 1. remove the check to specifically stop us reusing nodes, 2. add "|| true" after the ci-branch is created (as it errors if the branch already exists)13:20
derekhand 3  don't download the fedora image each time13:20
spredzyack13:21
*** matbu has joined #tripleo13:22
*** hewbrocca has joined #tripleo13:23
spredzyderekh, was that cause by the || true on the ci-branch thingy ?13:24
derekhspredzy: it was because of the ci-branch yes but I don't think the "|| true" did it13:25
derekhspredzy: looks like I gotta change a few more things so this can be rerun bear with me13:25
derekhthis script is normally only run once and VM is thrown away13:26
*** funzo has joined #tripleo13:26
*** lifeless has joined #tripleo13:28
spredzyderekh, now it fails with "AttributeError: 'module' object has no attribute 'wraps'"13:29
*** absubram has quit IRC13:29
derekhspredzy: yup, the first pass through this script must have updating something (in six...?)13:30
*** julim has joined #tripleo13:31
*** funzo has quit IRC13:31
jistrbtw i looked into this a bit as well on my local setup, didn't find anything yet (still unable to reproduce it) but here are some info points:13:32
jistrthe most weird thing is that CI doesn't execute "Disable STONITH" exec, as spredzy pointed out before13:33
jistrthe cause could be that this unless condition is not met for some reason13:33
jistrhttps://github.com/redhat-openstack/puppet-pacemaker/blob/master/manifests/stonith.pp#L513:33
spredzyjistr, weirdest thing is that each CI run gives a different kind of error13:34
spredzySometimes corosync can't form the cluster13:34
spredzySometime cororsync is ok, pacemaker is ok but galera fails13:34
*** mcornea has quit IRC13:34
spredzyjistr, if you look at the logs from this https://review.openstack.org/#/c/212566/13:34
*** adrianopetrich_ has quit IRC13:35
spredzythe mysqld.log for each node, you'll see cluster forms itself (3 nodes), then 1 leave, then another leave, then it its brought back up. 1 then 2, then 1 leave again like non stop13:35
jistrbut "Disable STONITH" is never run, is it?13:35
jistron my local env it's always run13:35
jistri noticed that there's an updated corosync RPM in Fedora which my image didn't contain13:36
jistrso i run yum -y update with a firstboot script when deploying13:36
jistrbut it still didn't reproduce13:36
jistrand disable stonith was run13:36
spredzy+ sudo pcs stonith show --full13:37
spredzy+ which crm_verify13:37
spredzyso not sure what to think13:37
*** mcornea has joined #tripleo13:37
derekhspredzy: ok, its getting further now, currently building the ramdisk, hopefully its gets further now13:37
jistrstonith show won't give any relevant info afaik13:37
spredzyjistr, what about + sudo pcs stonith show --full13:38
spredzy+ which crm_verify13:38
spredzyooops13:38
spredzyworry13:38
spredzy<nvpair id="cib-bootstrap-options-stonith-enabled" name="stonith-enabled" value="false"/>13:38
jistryeah that looks like it's disabled then13:38
jistr(pcs property show stonith-enabled would show from cmdline)13:38
jistrbut i still wonder how the difference between CI and my local env happens...13:39
jistrhow come it's disabled in CI even though the "disable stonith" exec isn't being run13:39
spredzylooking at the corosync logs and the galera logs I have the feeling one node os the cluster is flapping (in and out the cluster)13:40
spredzyand I have no idea what could be the cause of that13:40
spredzyderekh, ack thx13:40
spredzyjistr, ^13:40
spredzyjistr, on the run I am talking about it did run I can see '/Stage[main]/Pacemaker::Stonith/Exec[Disable STONITH]/returns: executed successfully' in the os-collect-config log13:41
jistrhmm interesting13:41
spredzy(review: https://review.openstack.org/#/c/212566/)13:41
spredzybut before the recheck that wasn't the case13:41
*** sseago has joined #tripleo13:41
jistrone thing which flew by is if the CI job cluster could interfere with other CI job clusters, i.e. if we're using multicast13:42
jistrbut we're using unicast13:42
jistrtransport: udpu13:42
spredzyyeah, by default when you run pcs cluster start it will use --transport udpu13:42
jistr(which doesn't disprove the interference hypothesis, just a data point)13:42
spredzyjistr, I've run it this week end in the CI while no other tripleo-heat-templates job were running13:43
spredzystill had the same issue :(13:43
spredzyjistr, have you ever experiences nodes flapping in/out the cluster ?13:43
*** bvandenh has quit IRC13:43
spredzys/experiences/experienced13:43
jistrno i haven't13:44
spredzyjistr, http://logs.openstack.org/66/212566/3/check-tripleo/gate-tripleo-ironic-overcloud-f21puppet-ha/9fe3de0/logs/overcloud-controller-2_logs/corosync.txt.gz13:44
spredzyif you grep Retransmit13:44
spredzyyou'll see quite some messages ... this worries me as a fact that it might show some cluster difunctionement13:45
spredzybut I still can't pin point the actual issue13:45
*** sseago has quit IRC13:46
*** Goneri has joined #tripleo13:48
*** masco has quit IRC13:48
*** sdake has joined #tripleo13:49
*** sdake has quit IRC13:49
*** sdake has joined #tripleo13:49
*** matbu has quit IRC13:55
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Switch nova temprevert to a cherrypick  https://review.openstack.org/21371413:57
*** matbu has joined #tripleo13:57
derekhThat ^^^ I think is why the non puppet jobs are failing13:57
dprincederekh: ack13:57
dprincederekh: fast tracking the switch to instack underclouds would help us here too13:58
derekhspredzy: seed has booted on that manuall ci job13:58
*** pradk has joined #tripleo13:59
*** spzala has joined #tripleo13:59
derekhdprince: yup, I'm back looking at it today should have a good idea how close we are in a bit13:59
spredzyderekh, ok14:00
dprincederekh: cool, yeah we don't use the "ephemeral" partitions for those jobs. Or we shouldn't need to if we do14:00
derekhdprince: ack14:00
*** chlong has joined #tripleo14:03
jistrspredzy: well guess what :) after yum -y update i get the same behavior you posted with flapping nodes (previously i was focusing just on the "disable stonith" exec)14:04
hewbroccahey fellas!14:04
jistrand the deployment seems stuck14:05
jistrspredzy: so our suspect is corosync/corosynclib RPMs i'd say14:05
jistrhewbrocca: hello14:05
*** lblanchard has joined #tripleo14:08
spredzyjistr, thhat would confirm my guess about corosync being the guily, but then I would suspect the issue to be 'network' related and not specifically configuration related14:08
spredzy(I asked also on #clusterlabs) see if they get any idea14:08
spredzyhewbrocca, o/14:08
*** panda has quit IRC14:09
*** panda has joined #tripleo14:10
*** sdake has quit IRC14:10
gfidentespredzy, any trace of network manager?14:11
jistrhere's the full list of suspects http://fpaste.org/255866/14398206/raw/14:13
*** w_ has joined #tripleo14:14
spredzyjistr, ahaha it couldnt be shorter :)14:14
spredzy:D14:14
spredzygfidente, apparently nop (http://logs.openstack.org/66/212566/3/check-tripleo/gate-tripleo-ironic-overcloud-f21puppet-ha/9fe3de0/logs/overcloud-controller-0_logs/host_info.txt.gz)14:14
* spredzy out for the next 4 hours.14:15
*** olaph has quit IRC14:16
*** spredzy is now known as spredzy|afk14:16
*** bvandenh has joined #tripleo14:17
*** sdake has joined #tripleo14:17
*** olaph has joined #tripleo14:20
*** w_ has quit IRC14:21
*** mcornea has quit IRC14:23
*** mcornea has joined #tripleo14:24
*** funzo has joined #tripleo14:27
*** adrianopetrich has joined #tripleo14:32
*** funzo has quit IRC14:32
*** olaph has quit IRC14:33
openstackgerritJiri Stransky proposed openstack/diskimage-builder: Fedora: install older corosync  https://review.openstack.org/21373614:38
*** paramite is now known as paramite|afk14:41
openstackgerritJiri Stransky proposed openstack-infra/tripleo-ci: Fedora: install older corosync  https://review.openstack.org/21373714:45
jistrspredzy|afk, derekh: ^ tried to pin down corosync and corosynclib to see if it fixes us14:45
*** paramite|afk is now known as paramite14:46
*** mburned is now known as mburned_out14:52
*** mburned_out is now known as mburned14:52
*** bvandenh has quit IRC14:56
*** shardy_ has joined #tripleo14:59
*** untriaged-bot has joined #tripleo15:00
untriaged-botUntriaged bugs so far:15:00
untriaged-bothttps://bugs.launchpad.net/os-collect-config/+bug/148251015:00
openstackLaunchpad bug 1482510 in heat "OS::Heat::SoftwareDeployment failed due SSL certificate verification error" [Medium,Triaged] - Assigned to Rico Lin (rico-lin)15:00
uvirtbotLaunchpad bug 1482510 in os-collect-config "OS::Heat::SoftwareDeployment failed due SSL certificate verification error" [Undecided,New]15:00
untriaged-bothttps://bugs.launchpad.net/diskimage-builder/+bug/146603715:00
openstackLaunchpad bug 1466037 in diskimage-builder "Signed Fedora and Ubuntu user image built by DIB can`t boot on HP DL380 Gen8 server for lack of mpt2sas driver" [Undecided,Incomplete]15:00
uvirtbotLaunchpad bug 1482510 in os-collect-config "OS::Heat::SoftwareDeployment failed due SSL certificate verification error" [Undecided,New] https://launchpad.net/bugs/148251015:00
uvirtbotLaunchpad bug 1466037 in diskimage-builder "Signed Fedora and Ubuntu user image built by DIB can`t boot on HP DL380 Gen8 server for lack of mpt2sas driver" [Undecided,Incomplete]15:00
untriaged-bothttps://bugs.launchpad.net/diskimage-builder/+bug/148338515:00
openstackLaunchpad bug 1483385 in diskimage-builder "install_grub failing for centos7" [Undecided,In progress] - Assigned to Abel Lopez (al592b)15:00
uvirtbotLaunchpad bug 1466037 in diskimage-builder "Signed Fedora and Ubuntu user image built by DIB can`t boot on HP DL380 Gen8 server for lack of mpt2sas driver" [Undecided,Incomplete] https://launchpad.net/bugs/146603715:00
untriaged-bothttps://bugs.launchpad.net/diskimage-builder/+bug/147180215:00
openstackLaunchpad bug 1471802 in diskimage-builder "ironic-agent element hardcodes interfaces names for DHCP." [Undecided,Fix committed] - Assigned to Om Kumar (om-kumar)15:00
uvirtbotLaunchpad bug 1483385 in diskimage-builder "install_grub failing for centos7" [Undecided,In progress]15:00
*** untriaged-bot has quit IRC15:00
uvirtbotLaunchpad bug 1483385 in diskimage-builder "install_grub failing for centos7" [Undecided,In progress] https://launchpad.net/bugs/148338515:00
uvirtbotLaunchpad bug 1471802 in diskimage-builder "ironic-agent element hardcodes interfaces names for DHCP." [Undecided,Fix committed]15:00
uvirtbotLaunchpad bug 1471802 in diskimage-builder "ironic-agent element hardcodes interfaces names for DHCP." [Undecided,Fix committed] https://launchpad.net/bugs/147180215:00
openstackgerritDan Prince proposed openstack/os-net-config: os-net-config: ensure ifup is called just once  https://review.openstack.org/21374615:00
*** shardy has quit IRC15:01
*** dsneddon has joined #tripleo15:03
*** dsneddon has quit IRC15:03
*** dsneddon has joined #tripleo15:03
*** funzo has joined #tripleo15:04
*** shardy_ has quit IRC15:04
*** shardy has joined #tripleo15:05
*** mcornea has quit IRC15:07
*** dsneddon has quit IRC15:08
*** dsneddon has joined #tripleo15:08
openstackgerritDan Prince proposed openstack/tripleo-image-elements: os-net-config: add configure_safe_defaults  https://review.openstack.org/21374815:09
*** dsneddon has quit IRC15:11
openstackgerritgreghaynes proposed openstack/diskimage-builder: create growroot element  https://review.openstack.org/20663615:11
*** dsneddon has joined #tripleo15:11
*** yamahata has joined #tripleo15:11
*** paramite is now known as paramite|afk15:11
*** paramite|afk is now known as paramite15:12
*** Marga_ has quit IRC15:17
*** chlong has quit IRC15:18
*** dprince has quit IRC15:20
*** mbound has quit IRC15:22
*** spzala has quit IRC15:30
*** chlong has joined #tripleo15:31
*** paramite is now known as paramite|afk15:32
*** chlong has quit IRC15:38
*** trown is now known as trown|lunch15:40
*** chlong has joined #tripleo15:40
*** aufi has quit IRC15:42
*** Marga_ has joined #tripleo15:43
*** lazy_prince has joined #tripleo15:45
*** jprovazn has quit IRC15:52
*** mestery has joined #tripleo15:53
*** ifarkas has quit IRC15:56
*** rwsu has joined #tripleo15:57
*** yog_ has quit IRC16:00
*** dprince has joined #tripleo16:03
*** lucasagomes is now known as lucas-brb16:04
*** lazy_prince has quit IRC16:06
*** alop has joined #tripleo16:08
*** pbourke has quit IRC16:10
*** pbourke has joined #tripleo16:10
*** adrianopetrich has quit IRC16:11
*** regebro has quit IRC16:12
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Switch nova temprevert to a cherrypick  https://review.openstack.org/21371416:12
*** david-ly_ is now known as david-lyle16:15
*** Marga_ has quit IRC16:17
*** Marga_ has joined #tripleo16:18
*** lazy_prince has joined #tripleo16:18
*** jistr has quit IRC16:19
*** spzala has joined #tripleo16:20
openstackgerritgreghaynes proposed openstack/diskimage-builder: Fix init-scripts element path munging and deps  https://review.openstack.org/21318016:20
derekhspredzy|afk: that run has finsihed now, I've logged into one of the controllers16:20
openstackgerritgreghaynes proposed openstack/diskimage-builder: Install-static depends on rsync  https://review.openstack.org/21330916:21
openstackgerritgreghaynes proposed openstack/diskimage-builder: create growroot element  https://review.openstack.org/20663616:21
derekhspredzy|afk: if you need to redo it later when your back you can rerun the devtest command and when its done log in like this : http://paste.openstack.org/show/419343/16:21
*** sthillma has joined #tripleo16:26
*** Marga_ has quit IRC16:30
*** olaph has joined #tripleo16:32
*** yamahata has quit IRC16:35
*** sthillma has quit IRC16:35
*** matbu has quit IRC16:37
*** olaph has quit IRC16:42
*** matbu has joined #tripleo16:42
*** trown|lunch is now known as trown16:44
*** devvesa has quit IRC16:46
*** tzumainn has joined #tripleo16:49
*** lucas-brb is now known as lucasagomes16:54
*** derekh has quit IRC16:57
*** lazy_prince has quit IRC16:57
*** yamahata has joined #tripleo16:58
*** dsneddon has quit IRC17:01
*** karume has quit IRC17:05
*** bvandenh has joined #tripleo17:14
*** mestery has quit IRC17:15
*** sdake_ has joined #tripleo17:20
*** athomas has quit IRC17:21
*** sdake has quit IRC17:24
*** mestery has joined #tripleo17:26
*** lsmola has quit IRC17:31
*** sdake has joined #tripleo17:31
*** mestery has quit IRC17:34
*** sdake_ has quit IRC17:35
*** morazi has joined #tripleo17:37
*** sthillma has joined #tripleo17:48
openstackgerritDan Prince proposed openstack/tripleo-heat-templates: Docker compute role configured via Puppet  https://review.openstack.org/20950517:57
openstackgerritDan Prince proposed openstack/tripleo-heat-templates: Docker compute role configured via Puppet  https://review.openstack.org/20950517:57
*** dsneddon has joined #tripleo17:58
*** sthillma has quit IRC18:07
*** sthillma has joined #tripleo18:07
*** panda has quit IRC18:10
*** panda has joined #tripleo18:10
*** lucasagomes is now known as lucas-dinner18:10
*** pelix has quit IRC18:11
*** lucas-dinner has quit IRC18:20
*** bvandenh has quit IRC18:24
*** yamahata has quit IRC18:28
*** shivrao has joined #tripleo18:29
*** spzala has quit IRC18:39
*** olaph has joined #tripleo18:41
*** matbu has quit IRC18:43
*** karume has joined #tripleo18:45
*** olaph has quit IRC18:51
*** olaph has joined #tripleo18:53
*** Marga_ has joined #tripleo18:58
*** spzala has joined #tripleo19:02
*** matbu has joined #tripleo19:03
*** olaph has quit IRC19:09
*** karume has quit IRC19:16
*** matbu has quit IRC19:32
*** matbu has joined #tripleo19:34
*** adrianopetrich has joined #tripleo19:34
*** Goneri has quit IRC19:37
*** penick has joined #tripleo19:58
*** jayg is now known as jayg|g0n320:02
*** noslzzp has quit IRC20:17
*** Goneri has joined #tripleo20:28
openstackgerritRob Pothier proposed openstack/tripleo-heat-templates: Enable Cisco Nexus and UCSM plugins  https://review.openstack.org/19875420:29
*** paramite|afk is now known as paramite20:36
*** paramite has quit IRC20:38
*** akrivoka has quit IRC20:40
*** spredzy|afk is now known as spredzy20:41
*** shardy has quit IRC20:50
*** trown is now known as trown|outttypeww20:53
*** matbu has quit IRC20:54
*** untriaged-bot has joined #tripleo21:00
untriaged-botUntriaged bugs so far:21:00
untriaged-bothttps://bugs.launchpad.net/os-collect-config/+bug/148251021:00
openstackLaunchpad bug 1482510 in heat "OS::Heat::SoftwareDeployment failed due SSL certificate verification error" [Medium,Triaged] - Assigned to Rico Lin (rico-lin)21:00
uvirtbotLaunchpad bug 1482510 in os-collect-config "OS::Heat::SoftwareDeployment failed due SSL certificate verification error" [Undecided,New]21:00
untriaged-bothttps://bugs.launchpad.net/diskimage-builder/+bug/146603721:00
openstackLaunchpad bug 1466037 in diskimage-builder "Signed Fedora and Ubuntu user image built by DIB can`t boot on HP DL380 Gen8 server for lack of mpt2sas driver" [Undecided,Incomplete]21:00
uvirtbotLaunchpad bug 1482510 in os-collect-config "OS::Heat::SoftwareDeployment failed due SSL certificate verification error" [Undecided,New] https://launchpad.net/bugs/148251021:00
uvirtbotLaunchpad bug 1466037 in diskimage-builder "Signed Fedora and Ubuntu user image built by DIB can`t boot on HP DL380 Gen8 server for lack of mpt2sas driver" [Undecided,Incomplete]21:00
untriaged-bothttps://bugs.launchpad.net/diskimage-builder/+bug/148338521:00
openstackLaunchpad bug 1483385 in diskimage-builder "install_grub failing for centos7" [Undecided,In progress] - Assigned to Abel Lopez (al592b)21:00
uvirtbotLaunchpad bug 1466037 in diskimage-builder "Signed Fedora and Ubuntu user image built by DIB can`t boot on HP DL380 Gen8 server for lack of mpt2sas driver" [Undecided,Incomplete] https://launchpad.net/bugs/146603721:00
untriaged-bothttps://bugs.launchpad.net/diskimage-builder/+bug/147180221:00
openstackLaunchpad bug 1471802 in diskimage-builder "ironic-agent element hardcodes interfaces names for DHCP." [Undecided,Fix committed] - Assigned to Om Kumar (om-kumar)21:00
uvirtbotLaunchpad bug 1483385 in diskimage-builder "install_grub failing for centos7" [Undecided,In progress]21:00
*** untriaged-bot has quit IRC21:00
uvirtbotLaunchpad bug 1483385 in diskimage-builder "install_grub failing for centos7" [Undecided,In progress] https://launchpad.net/bugs/148338521:00
uvirtbotLaunchpad bug 1471802 in diskimage-builder "ironic-agent element hardcodes interfaces names for DHCP." [Undecided,Fix committed]21:00
uvirtbotLaunchpad bug 1471802 in diskimage-builder "ironic-agent element hardcodes interfaces names for DHCP." [Undecided,Fix committed] https://launchpad.net/bugs/147180221:00
*** yamahata has joined #tripleo21:01
*** lblanchard has quit IRC21:03
*** sthillma_ has joined #tripleo21:04
*** sthillma has quit IRC21:06
*** sthillma_ is now known as sthillma21:06
*** jtomasek has quit IRC21:09
openstackgerritDan Sneddon proposed openstack/tripleo-heat-templates: Remove hardcoded bridge name in bonded compute NIC config  https://review.openstack.org/21386121:11
*** mestery has joined #tripleo21:15
*** julim has quit IRC21:17
*** gfidente has quit IRC21:41
*** Marga_ has quit IRC21:46
*** uvirtbot has quit IRC21:50
*** mestery has quit IRC22:01
*** mestery has joined #tripleo22:04
*** mestery has quit IRC22:09
*** panda has quit IRC22:09
*** sdake_ has joined #tripleo22:10
openstackgerritYanis Guenane proposed openstack-infra/tripleo-ci: Adding more CPU in the HA scenario  https://review.openstack.org/21388522:10
*** panda has joined #tripleo22:10
*** sdake has quit IRC22:13
openstackgerritOpenStack Proposal Bot proposed openstack/tuskar: Updated from global requirements  https://review.openstack.org/18693922:13
*** pradk has quit IRC22:14
*** dprince has quit IRC22:21
openstackgerritTomoki Sekiyama proposed openstack/os-net-config: Support multiple addresses assignment with ifcfg  https://review.openstack.org/21390222:26
openstackgerritTomoki Sekiyama proposed openstack/os-net-config: Support multiple addresses assignment with eni  https://review.openstack.org/21390322:26
*** Goneri has quit IRC22:33
*** chlong has quit IRC22:34
*** shivrao has quit IRC22:41
*** shivrao has joined #tripleo22:53
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Nothing to see here  https://review.openstack.org/11101122:56
*** sdake_ is now known as sdake23:01
*** rhallisey has quit IRC23:11
*** mestery has joined #tripleo23:13
*** sdake_ has joined #tripleo23:23
*** sdake has quit IRC23:26
*** spzala has quit IRC23:38
*** mestery has quit IRC23:55

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!