Wednesday, 2017-06-07

*** ooolpbot has joined #tripleo00:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION00:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168019500:10
openstackLaunchpad bug 1680195 in tripleo "Random ovb-ha ping test failures" [Critical,Triaged] - Assigned to John Trowbridge (trown)00:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169590400:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169609400:10
*** ooolpbot has quit IRC00:10
openstackLaunchpad bug 1695904 in tripleo "stable/ocata scenario002 broken due to volume went to status error" [Critical,Triaged]00:10
openstackLaunchpad bug 1696094 in tripleo "CI: ovb-ha promotion job fails with 504 gateway timeout, neutron-server create-subnet timing out" [Critical,Triaged]00:10
*** pmannidi has quit IRC00:11
*** artom has joined #tripleo00:13
*** rlandy has quit IRC00:17
*** bfournie has joined #tripleo00:20
*** bfournie has quit IRC00:22
*** bfournie has joined #tripleo00:25
openstackgerritBob Fournier proposed openstack/tripleo-heat-templates master: Add pre-selected Control Plane IP support.  https://review.openstack.org/41327800:29
*** pmannidi has joined #tripleo00:31
*** itlinux has joined #tripleo00:44
*** jbadiapa has quit IRC00:52
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Start at block-device mounting/unmounting cleanup  https://review.openstack.org/47118300:54
*** itlinux has quit IRC00:55
*** itlinux has joined #tripleo00:56
*** itlinux has quit IRC00:56
*** rhallisey has quit IRC00:58
*** jbadiapa has joined #tripleo00:59
*** mdnadeem has joined #tripleo01:05
*** japestinho has joined #tripleo01:06
*** ooolpbot has joined #tripleo01:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION01:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168019501:10
openstackLaunchpad bug 1680195 in tripleo "Random ovb-ha ping test failures" [Critical,Triaged] - Assigned to John Trowbridge (trown)01:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169590401:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169609401:10
*** ooolpbot has quit IRC01:10
openstackLaunchpad bug 1695904 in tripleo "stable/ocata scenario002 broken due to volume went to status error" [Critical,Triaged]01:10
openstackLaunchpad bug 1696094 in tripleo "CI: ovb-ha promotion job fails with 504 gateway timeout, neutron-server create-subnet timing out" [Critical,Triaged]01:10
openstackgerritIan Wienand proposed openstack/diskimage-builder master: PPC bootloader; install to root partition  https://review.openstack.org/47152501:11
openstackgerritIan Wienand proposed openstack/diskimage-builder master: PPC bootloader; install to boot partition  https://review.openstack.org/47152501:16
*** trozet has quit IRC01:17
*** tzumainn has quit IRC01:17
*** trozet has joined #tripleo01:17
*** tzumainn has joined #tripleo01:17
openstackgerritVictoria Martinez de la Cruz proposed openstack/tripleo-heat-templates master: Containerize Manila Share service  https://review.openstack.org/47152701:18
openstackgerritIan Wienand proposed openstack/diskimage-builder master: PPC bootloader; install to boot partition  https://review.openstack.org/47152501:24
*** gvrangan_ has quit IRC01:29
*** fzdarsky_ has joined #tripleo01:30
*** fzdarsky has quit IRC01:33
*** atoth has quit IRC01:34
*** itlinux has joined #tripleo01:39
*** itlinux has quit IRC01:45
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Add a keep-output flag for functional tests  https://review.openstack.org/47153201:48
openstackgerritVu Cong Tuan proposed openstack/tripleo-common master: Using assertIsNone(xxx) instead of assertEqual(None, xxx)  https://review.openstack.org/47061001:50
*** chlong has quit IRC01:50
openstackgerritSteve Baker proposed openstack/tripleo-common master: Make overcloud_containers.yaml template driven  https://review.openstack.org/47153701:55
*** itlinux has joined #tripleo01:57
*** yamahata has quit IRC01:59
*** gbarros has joined #tripleo02:04
*** itlinux has quit IRC02:05
*** ooolpbot has joined #tripleo02:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION02:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168019502:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169590402:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169609402:10
*** ooolpbot has quit IRC02:10
openstackLaunchpad bug 1680195 in tripleo "Random ovb-ha ping test failures" [Critical,Triaged] - Assigned to John Trowbridge (trown)02:10
openstackLaunchpad bug 1695904 in tripleo "stable/ocata scenario002 broken due to volume went to status error" [Critical,Triaged]02:10
openstackLaunchpad bug 1696094 in tripleo "CI: ovb-ha promotion job fails with 504 gateway timeout, neutron-server create-subnet timing out" [Critical,Triaged]02:10
*** mdnadeem has quit IRC02:11
*** gbarros has quit IRC02:21
*** mdnadeem has joined #tripleo02:22
*** cdearborn_ has quit IRC02:50
*** links has joined #tripleo02:52
*** egafford has joined #tripleo03:00
*** egafford has quit IRC03:04
*** ooolpbot has joined #tripleo03:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION03:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168019503:10
openstackLaunchpad bug 1680195 in tripleo "Random ovb-ha ping test failures" [Critical,Triaged] - Assigned to John Trowbridge (trown)03:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169590403:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169609403:10
*** ooolpbot has quit IRC03:10
openstackLaunchpad bug 1695904 in tripleo "stable/ocata scenario002 broken due to volume went to status error" [Critical,Triaged]03:10
openstackLaunchpad bug 1696094 in tripleo "CI: ovb-ha promotion job fails with 504 gateway timeout, neutron-server create-subnet timing out" [Critical,Triaged]03:10
*** tzumainn has quit IRC03:12
*** thrash is now known as thrash|g0ne03:17
*** dbecker has quit IRC03:19
*** jmelvin has quit IRC03:21
*** itlinux has joined #tripleo03:22
*** owalsh_ has joined #tripleo03:24
*** psahoo has joined #tripleo03:27
*** owalsh has quit IRC03:27
openstackgerritzhongshengping proposed openstack/puppet-tripleo master: Fix the default port for Panko API  https://review.openstack.org/47156703:29
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Update tracing in block_device_create_config_file  https://review.openstack.org/47116303:31
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Move ppc block-device default to right $ARCH  https://review.openstack.org/47117003:31
*** dbecker has joined #tripleo03:33
openstackgerritVu Cong Tuan proposed openstack/python-tripleoclient master: Remove support for py34  https://review.openstack.org/47157203:42
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Pass all blockdevices to bootloader  https://review.openstack.org/47118303:49
openstackgerritIan Wienand proposed openstack/diskimage-builder master: PPC bootloader; install to boot partition  https://review.openstack.org/47152503:49
openstackgerritVu Cong Tuan proposed openstack/tripleo-common master: Remove support for py34  https://review.openstack.org/47157403:50
*** atheurer has quit IRC03:54
*** japestinho has quit IRC03:55
*** yprokule has joined #tripleo03:57
*** ykarel_ has joined #tripleo03:59
*** ramishra has quit IRC03:59
openstackgerritzhongshengping proposed openstack/instack-undercloud master: Fix the port for Panko API  https://review.openstack.org/47157704:02
openstackgerritMichael Chapman proposed openstack/tripleo-quickstart master: Add virt power to root auth keys when qemu system  https://review.openstack.org/47157804:03
*** atheurer has joined #tripleo04:04
*** ooolpbot has joined #tripleo04:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION04:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168019504:10
openstackLaunchpad bug 1680195 in tripleo "Random ovb-ha ping test failures" [Critical,Triaged] - Assigned to John Trowbridge (trown)04:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169590404:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169609404:10
*** ooolpbot has quit IRC04:10
openstackLaunchpad bug 1695904 in tripleo "stable/ocata scenario002 broken due to volume went to status error" [Critical,Triaged]04:10
openstackLaunchpad bug 1696094 in tripleo "CI: ovb-ha promotion job fails with 504 gateway timeout, neutron-server create-subnet timing out" [Critical,Triaged]04:10
*** pmannidi has quit IRC04:12
*** skramaja has joined #tripleo04:13
openstackgerritMerged openstack/diskimage-builder master: Updated from global requirements  https://review.openstack.org/47045104:15
openstackgerritMerged openstack/diskimage-builder master: Use https in docs links  https://review.openstack.org/47079104:16
*** pmannidi has joined #tripleo04:29
*** janki has joined #tripleo04:31
openstackgerritzhongshengping proposed openstack/puppet-tripleo master: Fix the port for Panko API  https://review.openstack.org/47156704:34
*** dparkes has quit IRC04:40
*** Aakash has quit IRC04:44
*** atheurer has quit IRC04:46
*** itlinux has quit IRC04:49
*** florianf has joined #tripleo04:52
*** pmannidi has quit IRC04:54
*** pcaruana has joined #tripleo04:56
*** ooolpbot has joined #tripleo05:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION05:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168019505:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169590405:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169609405:10
*** ooolpbot has quit IRC05:10
openstackLaunchpad bug 1680195 in tripleo "Random ovb-ha ping test failures" [Critical,Triaged] - Assigned to John Trowbridge (trown)05:10
openstackLaunchpad bug 1695904 in tripleo "stable/ocata scenario002 broken due to volume went to status error" [Critical,Triaged]05:10
openstackLaunchpad bug 1696094 in tripleo "CI: ovb-ha promotion job fails with 504 gateway timeout, neutron-server create-subnet timing out" [Critical,Triaged]05:10
*** pmannidi has joined #tripleo05:12
*** pgadiya has joined #tripleo05:12
*** dsariel_ has joined #tripleo05:13
*** pcaruana has quit IRC05:14
*** bkopilov has joined #tripleo05:14
*** prateek has joined #tripleo05:15
*** ykarel__ has joined #tripleo05:16
*** ykarel_ has quit IRC05:18
*** nyechiel has joined #tripleo05:20
*** pgadiya has quit IRC05:26
openstackgerritSaravanan KR proposed openstack/tripleo-common master: Move passwords from mistral environment to swift environment  https://review.openstack.org/46790905:27
*** mdnadeem has quit IRC05:28
yolandathrash|g0ne, bnemec , that is true, we pass the repos in an env var. I did on https://review.openstack.org/471481. But without delorean-repo element, these vars are not used anywhere, it'd delorean-repo eelment that uses them and installs the repo on the image05:34
*** paramite has joined #tripleo05:35
*** aufi has joined #tripleo05:35
*** numan_ has joined #tripleo05:36
yolandabnemec i have been testing on a vm, the combination of 471479 and 471481 produced a successful image05:39
openstackgerritSaravanan KR proposed openstack/tripleo-heat-templates master: Enable DPDK on boot using PreNetworkConfig  https://review.openstack.org/46749005:42
*** lmiccini has joined #tripleo05:44
*** mdnadeem has joined #tripleo05:45
*** yprokule has quit IRC05:47
*** nicky has joined #tripleo05:48
*** agurenko has joined #tripleo05:51
openstackgerrityolanda.robla proposed openstack/tripleo-common master: Add delorean-repo element to centos  https://review.openstack.org/47147905:52
*** dparkes has joined #tripleo05:53
*** ratailor has joined #tripleo05:55
*** noslzzp has joined #tripleo05:55
*** mdnadeem has quit IRC06:05
openstackgerritDavid Sariel proposed openstack/tripleo-heat-templates master: Increase default for CinderLVMLoopDeviceSize  https://review.openstack.org/47074506:06
openstackgerritDavid Sariel proposed openstack/tripleo-heat-templates master: Increase default for CinderLVMLoopDeviceSize  https://review.openstack.org/47074506:08
*** ooolpbot has joined #tripleo06:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION06:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168019506:10
openstackLaunchpad bug 1680195 in tripleo "Random ovb-ha ping test failures" [Critical,Triaged] - Assigned to John Trowbridge (trown)06:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169590406:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169609406:10
*** ooolpbot has quit IRC06:10
openstackLaunchpad bug 1695904 in tripleo "stable/ocata scenario002 broken due to volume went to status error" [Critical,Triaged]06:10
openstackLaunchpad bug 1696094 in tripleo "CI: ovb-ha promotion job fails with 504 gateway timeout, neutron-server create-subnet timing out" [Critical,Triaged]06:10
*** jaosorior_away is now known as jaosorior06:11
*** bkopilov has quit IRC06:17
openstackgerritMichele Baldessari proposed openstack/tripleo-heat-templates master: Configuration for containerized MySQL clients  https://review.openstack.org/46653106:18
*** udesale has joined #tripleo06:19
*** jprovazn has joined #tripleo06:19
*** mdnadeem has joined #tripleo06:20
*** rcernin has joined #tripleo06:21
*** mcornea has joined #tripleo06:23
*** aufi has quit IRC06:25
*** radeks has joined #tripleo06:26
yolandabnemec, so i updated the change with comments, please take a look. My only concern is that i'd really like to include delorean-repo conditionally, as it was done before. But i don't see how to do it with yaml06:31
*** anshulbehl has joined #tripleo06:31
*** anshulbehl has left #tripleo06:32
*** noslzzp has quit IRC06:38
*** hjensas has quit IRC06:42
*** b00tcat_ has joined #tripleo06:48
*** b00tcat has quit IRC06:48
openstackgerritMerged openstack/diskimage-builder master: Add a keep-output flag for functional tests  https://review.openstack.org/47153206:50
*** athomas has joined #tripleo06:53
*** ffiore has joined #tripleo06:53
*** bogdando has joined #tripleo07:00
*** afazekas has quit IRC07:01
*** afazekas has joined #tripleo07:01
*** pcaruana has joined #tripleo07:03
*** ooolpbot has joined #tripleo07:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION07:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168019507:10
openstackLaunchpad bug 1680195 in tripleo "Random ovb-ha ping test failures" [Critical,Triaged] - Assigned to John Trowbridge (trown)07:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169590407:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169609407:10
*** ooolpbot has quit IRC07:10
openstackLaunchpad bug 1695904 in tripleo "stable/ocata scenario002 broken due to volume went to status error" [Critical,Triaged]07:10
openstackLaunchpad bug 1696094 in tripleo "CI: ovb-ha promotion job fails with 504 gateway timeout, neutron-server create-subnet timing out" [Critical,Triaged]07:10
*** limao has quit IRC07:10
*** tesseract has joined #tripleo07:11
*** limao has joined #tripleo07:11
*** mdnadeem has quit IRC07:11
*** jlinkes has joined #tripleo07:12
*** mdnadeem has joined #tripleo07:12
*** flepied has quit IRC07:15
*** jaganathan has joined #tripleo07:16
*** sid1 has joined #tripleo07:18
*** amoralej|off is now known as amoralej07:18
*** bkopilov has joined #tripleo07:20
*** shardy has joined #tripleo07:21
*** aufi has joined #tripleo07:23
openstackgerritChristian Schwede proposed openstack/puppet-tripleo master: Fix Swift ring management in container deploymnents  https://review.openstack.org/46879007:24
openstackgerritMichele Baldessari proposed openstack/puppet-pacemaker master: WIP Fix pacemaker remote resource creation with pcs 0.9.158  https://review.openstack.org/47160707:26
openstackgerritMerged openstack/tripleo-heat-templates master: Fix upgrade tasks to use correct service name  https://review.openstack.org/47142107:27
openstackgerritMerged openstack/tripleo-ui master: Fix a few forgotten PropTypes imports  https://review.openstack.org/46514607:28
openstackgerritAlfredo Moralejo proposed openstack-infra/tripleo-ci master: DNM: testing something for gate  https://review.openstack.org/47136107:28
*** limao has quit IRC07:28
*** noslzzp has joined #tripleo07:29
*** limao has joined #tripleo07:29
*** ccamacho has quit IRC07:30
*** fragatina has joined #tripleo07:32
*** ccamacho has joined #tripleo07:33
*** hjensas has joined #tripleo07:33
*** oidgar has joined #tripleo07:33
*** jpena|off is now known as jpena07:34
*** numan_ has quit IRC07:35
*** iranzo has quit IRC07:39
*** iranzo has joined #tripleo07:39
*** iranzo has joined #tripleo07:39
openstackgerrityolanda.robla proposed openstack/python-tripleoclient master: DNM: Testing experimental  https://review.openstack.org/47021107:42
*** egonzalez has joined #tripleo07:45
*** aufi has quit IRC07:45
*** flepied has joined #tripleo07:51
*** rcernin has quit IRC07:52
*** rcernin has joined #tripleo07:52
*** jpich has joined #tripleo07:53
openstackgerritChristian Schwede proposed openstack/puppet-tripleo master: Fix Swift ring management in container deployments  https://review.openstack.org/46879007:56
*** zzzeek has quit IRC08:00
*** zzzeek has joined #tripleo08:00
*** fragatina has quit IRC08:00
*** fragatina has joined #tripleo08:01
*** pmannidi has quit IRC08:02
*** fragatina has quit IRC08:06
bogdandofolks, could we merge custom undercloud patch? it seems passing extended CI well https://review.openstack.org/#/c/466322/08:08
bogdandojistr could you +1 ? you was asking for checking upgrades job for it08:09
bogdandohttps://review.openstack.org/#/c/450792/ ^^08:09
bogdandoweshay: ^^08:09
*** ooolpbot has joined #tripleo08:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION08:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168019508:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169590408:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169609408:10
*** ooolpbot has quit IRC08:10
openstackLaunchpad bug 1680195 in tripleo "Random ovb-ha ping test failures" [Critical,Triaged] - Assigned to John Trowbridge (trown)08:10
openstackLaunchpad bug 1695904 in tripleo "stable/ocata scenario002 broken due to volume went to status error" [Critical,Triaged]08:10
openstackLaunchpad bug 1696094 in tripleo "CI: ovb-ha promotion job fails with 504 gateway timeout, neutron-server create-subnet timing out" [Critical,Triaged]08:10
bogdandoshardy, mandre: PTAL, I'd like to have someone's +1/2 from containers DFG please08:10
shardybogdando: I'm not a quickstart expert but lgtm08:14
*** japestinho has joined #tripleo08:15
bogdandoshardy: the main thing for our dfg is README with an example guide I tried to provide, perhaps. And the install script08:15
bogdandoshardy: thanks!08:15
bogdandonote, the script is based on the doit.sh the team uses, AFAIK08:16
*** ramishra has joined #tripleo08:16
*** lucas-afk is now known as lucasagomes08:16
openstackgerritMerged openstack/diskimage-builder master: Update tracing in block_device_create_config_file  https://review.openstack.org/47116308:18
openstackgerritJaganathan Palanisamy proposed openstack/tripleo-common master: Derive Params (part 1): Initial workflow to start  https://review.openstack.org/45646908:20
*** numans has joined #tripleo08:22
openstackgerritMichele Baldessari proposed openstack/puppet-tripleo master: Only set the stonith property on the pacemaker_master node  https://review.openstack.org/47163008:24
yolandabnemec, around?08:26
*** ffiore_ has joined #tripleo08:26
yolandaor thrash|g0ne ?08:27
*** ffiore has quit IRC08:27
*** zoli|gone is now known as zoli08:27
*** aufi has joined #tripleo08:28
openstackgerritMichele Baldessari proposed openstack/puppet-tripleo master: Only set the stonith property on the pacemaker_master node  https://review.openstack.org/47163008:29
EmilienMyolanda: they are US based, I hope they are sleeping now08:30
yolandaEmilienM, i'm having this problem with image builds that are broken08:31
EmilienMwhere?08:31
yolandasee https://review.openstack.org/47021108:31
yolandayesterday we switched from using the legacy builds from tripleoclient to the new ones based on yaml files, on the ci08:32
*** oneswig has joined #tripleo08:32
yolandathis uncovered a problem, that the delorean-repo, that was added by code on the legacy build, is not added on the images08:32
yolandaso they fail08:32
yolandai tested and https://review.openstack.org/471479 solves it, but it also means adding delorean-repo everywhere, not optionally with the flag, as was done before08:32
yolandaso i wanted to discuss the best way to do it, i was thinking on update delorean-repo element, to only do any action depending on the old USE_DELOREAN_TRUNK flag, or something similar08:33
*** iranzo has quit IRC08:33
EmilienMyolanda: does it affect all tripleo CI now?08:34
yolandayes08:34
EmilienMyolanda: what jobs are failing now?08:34
yolandawell, the buildimae jos, that are on tripleoclient and diskimagebuilder08:34
EmilienMjust gate-tripleo-buildimage-overcloud-full-centos-7 ?08:34
yolandayes08:34
EmilienMok, not the ovb jobs, right?08:34
yolandano08:34
EmilienMI'm asking because we are in the middle of a release08:34
*** florianf has quit IRC08:34
EmilienMand the last thing I want is a broken CI08:34
yolandai know...08:34
EmilienMI see this: http://logs.openstack.org/11/470211/2/check/gate-tripleo-buildimage-overcloud-full-centos-7/f434bba/console.html#_2017-06-07_08_08_16_54589808:35
EmilienMnot sure we want delorean-repo element everywhere08:35
yolandabut that is the want that configures repo for images, and without that, the build fails, as cannot install the packages: http://logs.openstack.org/11/470211/2/check/gate-tripleo-buildimage-overcloud-full-centos-7/f434bba/console.html#_2017-06-07_08_06_06_75986408:36
EmilienMit sounds like the element doesn't work with old environment, iiuc, which is not backward compatible :(08:36
EmilienMmaybe updating the element08:36
yolandathat's why i was thinking, we could include it, but update delorean-repo element, to only configure things if USE_DELOREAN_TRUNK=108:36
*** iranzo has joined #tripleo08:37
*** derekh has joined #tripleo08:37
*** florianf has joined #tripleo08:38
yolandaEmilienM, or can you think on another alternative?08:38
openstackgerritMerged openstack/tripleo-docs master: Document hardware types for the undercloud and virtualbmc  https://review.openstack.org/46891008:38
shardyI think in CI we've typically used DIB_YUM_REPO_CONF to point at the repos on the host?08:39
shardyhttps://github.com/openstack-infra/tripleo-ci/blob/master/scripts/tripleo.sh#L65408:39
* shardy hasn't looked at how oooqs does it but I assume similar08:39
*** tosky has joined #tripleo08:40
yolandaso i could update the buildimage job, to configure the repo inside the host, then point to it with DIB_YUM_REPO_CONF ? the delorean element is deprecated?08:41
shardyyolanda: yeah that may be more consistent with the other ci jobs I think, but perhaps check with sshnaidm|afk or adarazs re what quickstart does for image building08:42
yolandathey use DIB_YUM_REPO_CONF, yes08:43
*** ckyriakidou has joined #tripleo08:43
yolandaactually at some point we could change this buildimage job, and use quickstart on it08:43
yolandaanyway, going to fix the buildimage job to reflect it08:43
shardyyolanda: sounds good, thanks!08:44
*** yprokule has joined #tripleo08:45
*** limao has quit IRC08:45
yolandathx for the pointers08:45
*** limao has joined #tripleo08:45
jistrbogdando: unfortunately it looks like it does break the upgrade :( https://review.openstack.org/#/c/450792/35/roles/undercloud-deploy/tasks/create-scripts.yml08:46
*** oneswig has quit IRC08:46
bogdandohm, jistr but CI has passed it seems https://review.openstack.org/#/c/466322/, wasn't that case covered by CI?08:46
jistrbogdando: it didn't pass, in the link above you pasted, see gate-tripleo-ci-centos-7-containers-multinode-upgrades-nv08:47
bogdandojistr, oh, right08:47
EmilienMyolanda: yeah, I don't have any other alternative in mind, please go ahead with this proposal08:48
jistrbogdando: that permission denied looks a bit as if the jenkins user was trying to delete the t-h-t dir, but it was maybe owned by root or something. Not sure if this is what happened and how, but it looks a bit like it.08:48
bogdandojistr: is it possible to retrigger only that case?08:48
*** oneswig has joined #tripleo08:49
jistrbogdando: i don't think so, we can retrigger everything if we want, but the error doesn't look like an intermittent one...08:49
openstackgerritMartin André proposed openstack/tripleo-heat-templates master: [WIP] Add docker templates for octavia services  https://review.openstack.org/46725408:49
bogdandojistr: strange, I see nothing that includes the role with become http://codesearch.openstack.org/?q=undercloud-deploy&i=nope&files=&repos=08:50
openstackgerritEmilien Macchi proposed openstack/tripleo-image-elements master: Release 7.0.0.0b2  https://review.openstack.org/47164208:53
bogdandojistr: this https://review.openstack.org/#/c/450792/35/roles/overcloud-prep-config/tasks/custom_tht.yml@13 breaks...08:54
jistrbogdando: that means it'll run as root, right?08:55
bogdandoyup08:55
bogdandoI think I will remove sudo. But that restricts the overcloud_templates_path for home dirs08:56
bogdandobut it always have been so, as it appears08:56
jistrright, the original doesn't have the become https://review.openstack.org/#/c/450792/35/roles/overcloud-prep-config/tasks/main.yml08:56
bogdandoI added ti cuz I was testing with paths like /usr08:56
bogdandobut I think it is ok to use only home, /tmp, /var/tmp other places with sticky bits08:56
*** salmankhan has joined #tripleo08:57
bogdandoI will update docs then08:57
jistrbogdando: yea i hope it shouldn't matter. The whole play runs as some deployment user (should be the owner of the directory where the t-h-t will be written). So if the deployment user was root (instead of stack or jenkins), it would have root permissions even without `become`.08:57
*** dparkes has quit IRC08:57
*** akrivoka has joined #tripleo08:57
bogdandojistr: no, the idea was to allow non root users to install custom tht by any paths, with sudo08:58
bogdandobut that doesn't play well with ci08:58
*** oneswig has left #tripleo08:58
jistrbogdando: ok.. i hope not having the right to write into any paths wouldn't be a big issue... The user which does the deployment and wants to clone the t-h-t, should always be able to clone into their home dir, right? I mean, i don't see the benefit in cloning it somewhere else.08:59
bogdandojistr: yes, should not be an issue, given that that's the current restriction anyway08:59
openstackgerritMartin André proposed openstack/tripleo-heat-templates master: DO NOT MERGE Testing containerized tacker service  https://review.openstack.org/47164609:00
jistrbogdando: i have one more question at https://review.openstack.org/#/c/450792/35/roles/undercloud-deploy/templates/undercloud-deploy.sh.j209:02
jistrbogdando: oh i see it in the readme now, i'll read it :)09:03
* jistr didn't scroll to the bottom there09:03
openstackgerritBogdan Dobrelya proposed openstack/tripleo-quickstart-extras master: Custom containerized undercloud deployments  https://review.openstack.org/45079209:06
openstackgerritMerged openstack/tripleo-common master: Parse roles from roles_data.yaml directly  https://review.openstack.org/46767809:07
openstackgerritMerged openstack/tripleo-common master: Updated from global requirements  https://review.openstack.org/47014709:08
openstackgerritMerged openstack/tripleo-heat-templates master: Updated from global requirements  https://review.openstack.org/47056509:08
*** sshnaidm|afk is now known as sshnaidm09:09
bogdandojistr: clarified the case with required vs update packages09:09
*** ooolpbot has joined #tripleo09:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION09:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168019509:10
openstackLaunchpad bug 1680195 in tripleo "Random ovb-ha ping test failures" [Critical,Triaged] - Assigned to John Trowbridge (trown)09:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169590409:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169609409:10
*** ooolpbot has quit IRC09:10
openstackLaunchpad bug 1695904 in tripleo "stable/ocata scenario002 broken due to volume went to status error" [Critical,Triaged]09:10
openstackLaunchpad bug 1696094 in tripleo "CI: ovb-ha promotion job fails with 504 gateway timeout, neutron-server create-subnet timing out" [Critical,Triaged]09:10
*** florianf has quit IRC09:11
openstackgerritmathieu bultel proposed openstack/tripleo-common master: Implement upgrade workflow  https://review.openstack.org/46376509:12
*** dparkes has joined #tripleo09:15
skramajajtomasek: hey. i had to drop off yesterday in the middle of the dicussion, is there anything still open, which we should discuss now?09:15
jistrbogdando: ack i see, thanks for the explanation. Maybe the installs/updates of packages could at some point be a separate script from `undercloud-deploy.sh`, but now i see how it makes sense as a feature.09:18
jistrbogdando: i'm ready to +2 when https://review.openstack.org/#/c/466322/ is green09:19
bogdandojistr, thanks!09:19
openstackgerritmathieu bultel proposed openstack/python-tripleoclient master: Implement Upgrade workflow via the tripleo cli  https://review.openstack.org/46372809:20
openstackgerritmathieu bultel proposed openstack/tripleo-common master: Implement upgrade workflow  https://review.openstack.org/46376509:22
yolandaEmilienM, shardy : https://review.openstack.org/#/c/471647/09:22
*** florianf has joined #tripleo09:22
openstackgerritLingyong Xu proposed openstack/instack master: Drop py34 target in tox.ini  https://review.openstack.org/47166309:23
EmilienMyolanda: +109:27
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Ability to enable/disable debug mode per OpenStack service  https://review.openstack.org/46769809:27
EmilienMshardy: upgrade jobs are voting, I wonder if we want them voting09:27
EmilienMerr09:27
EmilienMshardy: upgrade jobs are passing (except one scenario), I wonder if we want them voting09:27
openstackgerritMerged openstack/instack-undercloud master: Remove neutron ml2 plugin linuxbridge mechanism_driver  https://review.openstack.org/47115209:28
shardyEmilienM: yes I think that would be good09:28
ccamachoEmilienM they should, right?09:29
ccamachoIs there a reason why not?09:29
*** limao has quit IRC09:29
openstackgerritDmitry Tantsur proposed openstack/tripleo-common master: When enrolling Dell nodes, use drac_address instead of deprecated drac_host  https://review.openstack.org/46991809:29
openstackgerritMartin André proposed openstack/tripleo-heat-templates master: DO NOT MERGE Testing containerized congress service  https://review.openstack.org/47167409:30
*** dparkes has quit IRC09:30
openstackgerritDmitry Tantsur proposed openstack/tripleo-common master: Add workflow for IPMI nodes discovery  https://review.openstack.org/46257709:30
EmilienMccamacho, shardy: there are reasons to NOT make them voting. Like the fact they are not part of the promotion pipeline09:30
EmilienMwhich means they can break tomorrow09:31
EmilienMbecause they don't vote in promotion pipeline09:31
EmilienMso if a bad patch merges in Nova, our gate will be broken09:31
EmilienMbecause upgrade jobs will fail and be voting09:31
EmilienMand it will happen for sure09:32
shardyEmilienM: IMO we should have upgrade coverage in the promotion pipeline09:33
shardyeverything in our (voting) gate should also run for periodic promotion?09:34
*** ebarrera has joined #tripleo09:34
EmilienMshardy: it's not so easy09:34
EmilienMshardy: sshnaidm is working on having multinode jobs in the promotion pipeline09:34
EmilienMshardy: but it's not so easy because multinode jobs are run in infra clouds and they need access to the promotion server09:34
shardyEmilienM: Ok, I guess then in the short term we'll have to deal with the regressions when they happen09:35
EmilienMshardy: so for security reasons we can't open ports from all IPs09:35
shardyEmilienM: can't we have a job on the promotion servers which polls for the status of some infra jobs?09:35
EmilienMshardy: I think sshnaidm can tell more about what he's preparing09:35
jtomasekskramaja: so there are 2 things with the current workflow. It will probably be better if I comment on the review09:36
sshnaidmshardy, we'll have script on mirror-server that will pull results from openstack-health about status of multinode jobs09:36
shardysshnaidm: ah, yeah that's the sort of thing I imagined, sounds good09:36
EmilienMsshnaidm: which means we need to run tempest on the multinode jobs, right?09:36
EmilienMopenstack-health only works with tempest AFIK no?09:37
shardyhmm09:37
EmilienMunless we fake the subunit file with fake results09:37
EmilienMwe could do that in the meantime09:37
sshnaidmEmilienM, no, but with subunit. I created subunit results for pingtest too :)09:37
EmilienMfake a subunit file if job didn't fail09:37
EmilienMahah you rocks09:37
* EmilienM waves on sshnaidm09:37
shardysounds good :)09:37
*** mdnadeem has quit IRC09:38
sshnaidmhttp://status.openstack.org/openstack-health/#/g/project/openstack-infra~2Ftripleo-ci09:38
sshnaidmfirst jobs were registered today as I see09:38
EmilienMthat's really cool09:38
EmilienMso now we can subscribe to RSS when it fails :D09:39
EmilienMopenstack-health is really cool for these who don't know it, I suggest to have a look09:39
EmilienMsshnaidm, shardy, ccamacho: so in conclusion. Once we have promotion jobs working with multinode (sounds super close) - let's add upgrade jobs09:40
EmilienMand also all scenarios09:40
EmilienMand then I'm ok to make them voting09:40
EmilienMshardy: so everytime something breaks in upgrades, we can escalate to the right team while our CI keeps working09:40
EmilienMI prefer to not have a promotion during 5 days instead of having our CI full red during 5 days09:41
EmilienMwhat do you think?09:41
*** limao has joined #tripleo09:43
shardyEmilienM: yes I agree09:43
*** owalsh_ is now known as owalsh09:44
EmilienMcool09:45
openstackgerritFlorian Fuchs proposed openstack/tripleo-validations master: Pre-upgrade: Check disk space  https://review.openstack.org/47168609:45
*** karimb has joined #tripleo09:46
openstackgerritOpenStack Proposal Bot proposed openstack/tripleo-ui master: Imported Translations from Zanata  https://review.openstack.org/47168909:47
*** limao has quit IRC09:47
EmilienMjtomasek: how are we doing with ui today? :)09:49
owalshbogdando, jaosorior: FYI I raised a bug for the /etc/ssh/ssh_known_hosts issue on the undercloud - https://bugs.launchpad.net/tripleo/+bug/169628309:50
openstackLaunchpad bug 1696283 in tripleo "Containerized undercloud fails to mount /etc/ssh/ssh_known_hosts when /etc/ is mounted readonly" [Medium,Triaged] - Assigned to Oliver Walsh (owalsh)09:50
jtomasekEmilienM: RDO should be up to date now, I think we're ready to release09:50
EmilienMyeah?09:50
openstackgerritFlorian Fuchs proposed openstack/tripleo-validations master: Add pre-upgrade group to RAM/disk space validations  https://review.openstack.org/47169109:50
owalshbogdando, jaosorior: I'm not missing anything obvious, am I? This looks like a docker bug - maybe specific to devicemapper.09:51
jtomasekI'll give you final confirmation when I get honza's and florianf's confirmation09:51
jtomasekEmilienM: ^09:51
EmilienMok ok ok09:51
EmilienMjtomasek: also, please patch tripleo-ui with tag bump if not done already09:52
bogdandoowalsh: it seems that undercloud is configured by CI scripts to use overlay2. While the overlcoud nodes' overlay2 is ensured by t-h-t docker service template09:52
EmilienMjtomasek: don't forget to update releasenotes/source/conf.py also09:52
bogdandoowalsh: but yeah, by default undercloud deployed with devicemapper, by quickstart09:52
jtomasekEmilienM: ok09:53
*** fzdarsky_ is now known as fzdarsky|lunch09:58
*** fragatina has joined #tripleo10:00
openstackgerritMerged openstack/diskimage-builder master: Move ppc block-device default to right $ARCH  https://review.openstack.org/47117010:02
ebarreraEmilienM, can you take a quick look to https://bugzilla.redhat.com/show_bug.cgi?id=1455503 ?10:07
openstackbugzilla.redhat.com bug 1455503 in openstack-tripleo-heat-templates "external ip not crrectly set for apache on 1+1 plus isolated ctlplane + external" [Unspecified,New] - Assigned to emacchi10:07
*** ffiore_ has quit IRC10:08
ebarreraEmilienM, I'm not sure if I'm doing something wrong or it's really a bug.. some help would be nice to move the case forward10:08
openstackgerrityolanda.robla proposed openstack/python-tripleoclient master: DNM: Testing experimental  https://review.openstack.org/47021110:09
*** ooolpbot has joined #tripleo10:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION10:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168019510:10
openstackLaunchpad bug 1680195 in tripleo "Random ovb-ha ping test failures" [Critical,Triaged] - Assigned to John Trowbridge (trown)10:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169590410:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169609410:10
*** ooolpbot has quit IRC10:10
openstackLaunchpad bug 1695904 in tripleo "stable/ocata scenario002 broken due to volume went to status error" [Critical,Triaged]10:10
openstackLaunchpad bug 1696094 in tripleo "CI: ovb-ha promotion job fails with 504 gateway timeout, neutron-server create-subnet timing out" [Critical,Triaged]10:10
owalshbogdando, jaosorior: reproduced https://bugs.launchpad.net/tripleo/+bug/1696283 using overlay210:13
openstackLaunchpad bug 1696283 in tripleo "Containerized undercloud fails to mount /etc/ssh/ssh_known_hosts when /etc/ is mounted readonly" [Medium,Triaged] - Assigned to Oliver Walsh (owalsh)10:13
*** lucasagomes has quit IRC10:15
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates master: Stop/disable l3 agent in docker service upgrade_tasks  https://review.openstack.org/46992410:16
numansEmilienM, shardy Hi, can you please add this to your review queue - https://review.openstack.org/#/c/436767/10:16
*** ffiore_ has joined #tripleo10:17
openstackgerritGabriele Cerami proposed openstack-infra/tripleo-ci master: quickstart transition: add promotion logic  https://review.openstack.org/44546710:18
*** lucasagomes has joined #tripleo10:19
EmilienMnumans: we still don't have testing for ovn, I see your patch still in progress10:22
numansEmilienM, sorry for the wrong link - https://review.openstack.org/#/c/436767/10:23
shardynumans: does OVN really need it's own VIP vs just routing traffic via one of the per-network ones?10:23
openstackgerritGabriele Cerami proposed openstack-infra/tripleo-ci master: quickstart transition: add promotion logic  https://review.openstack.org/44546710:23
shardywe've got one for Redis, but that was a special case and not something we really want to replicate10:23
EmilienMnumans: I think the links was good. My point is that we still don't have tests for OVN in our CI and you're asking me to review something that we haven't test10:25
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates master: Add host logging for redis service template  https://review.openstack.org/47170910:26
numansEmilienM, From what i spoke with the rdo team, they want to have a update job in CI to test from ovs 2.6 to ovs 2.7 before using ovs 2.710:26
numansEmilienM, i will follow up with them.10:27
*** sai has quit IRC10:27
numansshardy, I think it needs its own VIP. However I will think more on it and see if the existing VIP can be used and comeback on it10:28
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: DONT REVIEW: test ci patch  https://review.openstack.org/47171010:28
shardynumans: ack, it'd be better to avoid it if possible10:29
*** rook has quit IRC10:29
shardynumans: context is I'm working composable networking atm, and special-cases like this are kind of a hassle in that model10:29
openstackgerritGabriele Cerami proposed openstack/tripleo-quickstart master: Create configuration for periodic jobs in featureset002  https://review.openstack.org/46373910:30
*** rook has joined #tripleo10:30
shardyvs just mapping a service to a vip via ServiceNetMapDefaults10:30
*** sai has joined #tripleo10:30
numansshardy, ok.10:30
*** rook is now known as Guest5259010:30
*** agurenko_ has joined #tripleo10:30
numansshardy, this is the puppet tripelo patch for OVN HA. Basically i want ovn server to be started on the vip node.10:31
ccamachoEmilienM shardy **sorry was in a meeting, yeahp I agree with having it voting after having working the promotion jobs with multinodes. thanks10:31
numansshardy, i guess in the case where existing vip is used, I need to make sure that VIP is created before ovn::ocf resource is created.10:32
shardynumans: we create all the vips before any services/nodes10:32
openstackgerritBogdan Dobrelya proposed openstack-infra/tripleo-ci master: DO NOT MERGE testing 450792  https://review.openstack.org/46632210:33
numansshardy, ok. cool. in that case, creating ovn::ocf resource and setting the proper colocation constraints would do. i will do the testing10:33
openstackgerritFlorian Fuchs proposed openstack/tripleo-validations master: Add pre-upgrade group to undercloud RAM validation  https://review.openstack.org/47169110:33
numansshardy, thanks for the comments.10:33
*** agurenko has quit IRC10:34
shardynumans: thanks, I added a comment on the review but we can revisit if there's a specific reason ServiceNetMapDefaults won't work in this case10:34
*** ubijtsa is now known as assassin10:37
openstackgerritMerged openstack/tripleo-quickstart-extras master: collect-logs: Remove collection of docker etc/ dirs  https://review.openstack.org/44717610:38
numansshardy, for ovn there is already a map in the ServiceNetMapDefaults - https://github.com/openstack/tripleo-heat-templates/blob/master/network/service_net_map.j2.yaml#L8010:39
shardyEmilienM: hey I wanted to draw your attention to https://review.openstack.org/#/c/468292/ - sounds like we may have to change the release model for os-*-config10:40
shardyIf you're OK with that I can update the review10:40
EmilienMshardy: big +110:42
*** ffiore_ has quit IRC10:43
*** japestinho has quit IRC10:45
*** dsariel_ has quit IRC10:45
*** ffiore_ has joined #tripleo10:47
openstackgerritOliver Walsh proposed openstack/tripleo-heat-templates master: Ensure /etc/ssh/ssh_known_hosts exist in docker config-data.  https://review.openstack.org/47147510:47
openstackgerritOliver Walsh proposed openstack/tripleo-heat-templates master: Map /etc/ssh/ssh_known_hosts to all containers  https://review.openstack.org/46846110:48
*** jprovazn has quit IRC10:49
*** nicky has quit IRC10:49
owalshmandre, bogdando, jaosorior: FYI ^^^10:49
owalshit only occurs when the files does not already exist in config-data10:50
owalshseems you can mount a file over an existing ro file, but can't mount a new file into an existing ro directory10:53
bogdandoowalsh: an interesting finding!10:55
*** rhallisey has joined #tripleo10:56
owalshbogdando: yea. I've not hit this one before, but then I wouldn't be using ro mounts very often10:56
bogdandoowalsh: ro mounts are safe for concurrent access10:57
bogdandowe want this for the most of the config data cases10:57
bogdandoactually for all cases...10:58
shardybandini, dciabrin: Hey, yesterday we discussed getting HA coverage in the container CI, any idea if there are patches up already to enable that, or do we need to start it?10:58
bandinishardy: I believe we would need to start it10:59
owalshbogdando: ack. It's only an issue if /etc is mounted. Usually we mount /etc/<foo>10:59
shardyjistr: ^^ just in case it's something you already looked at10:59
shardybandini: ack, it'd be good to get that started if the pieces we need have landed, are we still blocked on some RDO things?11:00
owalshbogdando: I guess it makes sense, creating a new file in /etc/ssh would need to modify the directory entry in the /etc/:ro mount11:00
jankimarios, hey11:01
bandinishardy: yeah we are still waiting on the rdo pcmk stuff (should happen rather soon now, today or tomorrow)11:01
shardybandini: Ok, thanks, pls let me know when it's ready and we can sync again re CI :)11:02
jistrshardy, bandini, dciabrin: haven't looked into it yet, but assuming we don't need to do anything else than including the right services, it should be easy from the CI side. We just need to switch the composable_scenario. Do you have a job in mind where we could do this? Maybe the multinode job? It runs in 1.5 hours so it has some space for experiments.11:02
bandinishardy: yess! thanks :)11:02
bandinijistr: ideally yes a multinode job with 3 controllers11:02
marioso/ janki11:03
*** agurenko_ is now known as agurenko11:03
shardyjistr: yeah I was thinking the multinode job, then later we need pacemaker plus minor updates which has the same requirement11:03
jankimarios, I am still stuck at the odl_user error same as last time. I checked all puppet manifests for odl_username and it is mentioned properly11:03
shardye.g 3 controllers to test a rolling minor update w/pacemaker11:03
jankimarios, on second note, could this be because of firewall. pasting logs on pastebin11:04
jistrshardy, bandini: right. I hope we could start with 1 all-in-one undercloud w/ "pseudo HA" and then expand the number of nodes and add updates.11:04
*** lucasagomes is now known as lucas-hungry11:04
openstackgerritBeth Elwell proposed openstack/tripleo-heat-templates master: TEST ONLY: Check horizon container upgrades  https://review.openstack.org/47172111:05
shardyjistr: sure we can start with 1 as a first step11:05
jistrshardy, bandini: err s/undercloud/overcloud/11:05
mariosjanki: so, i don't recall the details of what you were doing (deploying with ODL)... I seem to recall there was a missing parameter and we were digging into the puppet module and found where it was defaulted11:05
*** shardy is now known as shardy_lunch11:06
*** agurenko is now known as agurenko_11:06
jankimarios, https://pastebin.com/MawgVuQt. Check the firewall rules. all ipv4 and ipv6 drop11:06
jankimarios, correct. that's what we were doing. I understand that parameter (username) mentioend correctly at all required positions11:07
jankimarios, the failure could be because of ip rules11:07
mariosjanki: so it could be that, but the error in the paste (has failures) may just be because an earlier dependency failed (I'd need more info to check) log all ipv4]: Skipping because of failed dependencies for example11:09
jankimarios, ohk. Where would I find more logs, like it says (truncated, view all with --long)11:10
marios    Error: Could not find a suitable provider for odl_user still seems to be the root from that paste11:10
*** ooolpbot has joined #tripleo11:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION11:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168019511:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169590411:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169609411:10
*** ooolpbot has quit IRC11:10
openstackLaunchpad bug 1680195 in tripleo "Random ovb-ha ping test failures" [Critical,Triaged] - Assigned to John Trowbridge (trown)11:10
openstackLaunchpad bug 1695904 in tripleo "stable/ocata scenario002 broken due to volume went to status error" [Critical,Triaged]11:10
openstackLaunchpad bug 1696094 in tripleo "CI: ovb-ha promotion job fails with 504 gateway timeout, neutron-server create-subnet timing out" [Critical,Triaged]11:10
mariosjanki: so that is during the puppet run... looks like control1?11:10
*** jkilpatr has joined #tripleo11:10
jankimarios, are these puppet files fetched at run time? because a patch was added to puppet-opendaylight about username yesterday11:11
mariosjanki: so look on that box for more info on the puppet error and dig at specific service logs if you think that is the problem. but again, there still seems from that paste at least something to do with Error: Could not find a suitable provider for odl_user are you sure it is being passed correctly tripleo heat templates <--> puppet-module11:11
mariosjanki: no they will be whatever is on your nodes when you deployed them (the pupet modules i mean, unless you manually update them )11:11
jankimarios, I deployed it today right from undercloud11:12
*** pkovar has joined #tripleo11:12
mariosjanki: heh well whatever is on your overcloud-full image anyway.11:12
* marios biab coffee11:12
jankimarios, how to add log files in puppet and get those logs effective while deploying?11:12
*** agurenko_ is now known as agurenko11:12
yolandaso, green light on buildimage: https://review.openstack.org/#/c/470211/11:13
skramajajtomasek: ack. i will wait for your comments on the review.11:15
openstackgerritJiri Stransky proposed openstack/tripleo-quickstart master: Switch containers multinode CI to deploy HA  https://review.openstack.org/47172411:15
bandinijistr: ack works for me as a first pass11:16
openstackgerritOliver Walsh proposed openstack/tripleo-heat-templates master: Ensure /etc/ssh/ssh_known_hosts exist in docker config-data.  https://review.openstack.org/47147511:16
openstackgerritOliver Walsh proposed openstack/tripleo-heat-templates master: Map /etc/ssh/ssh_known_hosts to all containers  https://review.openstack.org/46846111:17
openstackgerritJiri Stransky proposed openstack/tripleo-quickstart master: Switch containers multinode CI to deploy HA  https://review.openstack.org/47172411:17
jistrbandini, shardy_lunch: on CI side i'd expect something like this -- https://review.openstack.org/471724/ -- though we may need to tweak how environments/docker.yaml and environments/puppet-pacemaker.yaml interact. Maybe we need environments/docker-pacemaker.yaml instead? Anyway, i think we fortunately don't have many problems anymore that would be specific only to CI or using nodepool. So once we can get11:20
jistrdocker+pacemaker to deploy, and then get it to deploy via OOOQ, the CI should just click into place.11:20
yolandahi thrash|g0ne , so i seem to be building my jobs correctly now, but i see that experimental jobs for ramdisk are broken now: http://logs.openstack.org/11/470211/3/experimental/gate-tripleo-buildimage-agent-ramdisk-centos-7-nv/cf3ab6b/console.html11:21
yolandait tries to create the image name agent-ramdisk, but that's not defined on the overcloud yamls11:22
*** dsariel_ has joined #tripleo11:22
yolandaactually there is no ramdisk definition there, just overcloud11:22
bandinijistr: I have a review dor docker-ha.yaml up somewhere11:22
bandinijistr: I need to clean it up though11:22
jistrbandini: ah nice, so yea i think we'd need to tweak OOOQ a bit to be able to deploy with that11:23
mariosjanki: so the puppet aply logs will be on the node itself, i'd probably start with the journal or even vi /var/log/messages11:23
bandinijistr: https://review.openstack.org/#/c/471384/ right now am duplicating docker.yaml so the operator does not have to do -e docker and -e ha-docker, but maybe we just prefer not duplicating the services and use both environments in the right order11:23
jankimarios, will do11:23
bandinijistr: dciabrin just finished convincing me to not duplicate stuff in the yaml file and have the op specify -e docker *and* -e ha-docker, so I will do that later today11:24
*** pkovar has quit IRC11:26
*** b00tcat_ has quit IRC11:26
jistrbandini: yea sounds a bit better to me too... Eventually, i assume we'd switch the entries in the default resource registry anyway, so docker.yaml would essentially go away. So yea i'd optimize for the scenario which includes both.11:26
jistrand maybe at some point we'd also make docker-ha.yaml to be the default too, and both would go away11:27
bandinijistr: 2 against 1. I will totally change it now :D11:27
*** jaganathan has quit IRC11:27
bandinijistr: agreed, makes sense11:27
*** athomas has quit IRC11:33
dciabrinbandini, oh that's great :)11:34
*** athomas has joined #tripleo11:34
jankimarios, thanks :)11:37
*** fragatina has quit IRC11:38
*** fzdarsky|lunch is now known as fzdarsky11:38
*** zoli is now known as zoli|brb11:39
*** oanson has quit IRC11:40
*** Guest52590 is now known as rook11:40
*** ffiore_ has quit IRC11:41
rookshardy_lunch zaneb -- os-collect-config - when i see ErrorResponse on the overcloud nodes, does that mean there was a issue reading from Swift?11:41
*** milan has joined #tripleo11:41
*** oanson has joined #tripleo11:42
*** jpena is now known as jpena|lunch11:42
sshnaidmarxcruz, have you tried multinode with tempest?11:42
*** shardy_lunch is now known as shardy11:43
shardyrook: depends on the error but possibly - what did it say?11:43
shardyrook: my normal test is to grab the request URL from the config file and download it with curl11:44
*** thrash|g0ne is now known as thrash11:44
rook<ErrorResponse><Error><Message>The request processing has failed due to an internal error:Timed out waiting for a reply to message ID11:44
*** karimb has quit IRC11:44
rookI think I remember hitting something like this, and it was due to Swift11:44
shardyrook: yeah, sounds like something isn't working on the undercloud - do swift CLI commands work OK there?11:44
thrashyolanda: I'm pretty sure that job can just go away11:44
rookshardy: this isn't a env that I have access to, just one I am trying to help remotely debug.11:45
*** karimb has joined #tripleo11:45
*** dparkes has joined #tripleo11:45
thrashopenstackgerrit: or, it needs to be renamed ironic-python-agent11:45
rookshardy the operator is having issue scaling up - always fails ~50 nodes (newton).11:45
thrashinstead of ramdisk-agent11:45
shardyrook: ack, I'd start by checking the status of the swift services on the undercloud11:45
rookshardy I wonder if there is a issue with how swift is configured... Maybe this was a earlier release.11:45
rookI know we hit swift issues early, but got them fixed.11:46
shardyrook: not running out of resources perhaps as the deploy gets bigger, and having something get killed on the undercloud?11:46
*** agurenko is now known as agurenko_11:46
*** karimb has quit IRC11:46
thrashyolanda: so s/agent-ramdisk/python-ironic-agent/11:46
openstackgerritMerged openstack/tripleo-ui master: Imported Translations from Zanata  https://review.openstack.org/47168911:47
thrashyolanda: and deploy-ramdisk just needs to go away11:48
thrashyolanda: sending up a patch now11:48
rookshardy: thanks!11:48
*** karimb has joined #tripleo11:48
*** jprovazn has joined #tripleo11:49
jankimarios, think found the root Error: /Stage[main]/Opendaylight::Install/File_line[java_options_systemd]: Could not evaluate: No such file or directory - /usr/lib/systemd/system/opendaylight.service11:50
*** atoth has joined #tripleo11:50
openstackgerritFlavio Percoco proposed openstack/tripleo-heat-templates master: Don't create networks if neutron is not enabled  https://review.openstack.org/47173611:53
*** abishop has joined #tripleo11:54
mariosjanki: cool... so looks like the service isn't in the image? You may just manually install it for now as workaround?11:54
jankimarios, yes. will do that. BUt i did give correct environment argument to overcloud-deploy11:55
jankimarios, where else to enforce that the service gets included in the image11:56
*** florianf has quit IRC11:56
mariosjanki: so you may have included the env files/the service manifest (assume you have the service in roles data etc etc but if it was never installed on the image then you can't use it)11:57
jankimarios, how to get it installed in the image?11:58
mariosjanki: so grep tells me it is already there11:58
mariosjanki: it needs to go to tripleo-puppet-modules sec finding link for you11:59
jankimarios, where and what did you grep (sorry bugging you)11:59
*** sid1 has quit IRC11:59
adarazsEmilienM: do you know which bugs are still in the way of a successful promotion?12:00
mariosjanki: unless you used tripleoquickstart stable/ocata some older version that doesn't have this@ https://github.com/openstack/tripleo-puppet-elements/blob/master/elements/overcloud-controller/install.d/package-installs-overcloud-controller#L6612:00
*** dparkes has quit IRC12:00
jankimarios, I used master-tripleo-ci. checking the link12:01
*** pkovar has joined #tripleo12:01
EmilienMadarazs: https://bugs.launchpad.net/tripleo/+bug/169609412:01
openstackLaunchpad bug 1696094 in tripleo "CI: ovb-ha promotion job fails with 504 gateway timeout, neutron-server create-subnet timing out" [Critical,Triaged]12:01
adarazsEmilienM: thanks!12:01
jankimarios, I am assuming these packages come from RDO12:01
ansiwenshardy: the stable/ocata backport is still failing (since three weeks), and actually now I don't find the problem in the logs. https://review.openstack.org/#/c/462542/12:01
*** chlong has joined #tripleo12:02
honzajtomasek: so, i shouldn't try and get a few last minute patches in today?  i thought releases were usually cut on thursdays?12:04
*** noslzzp has quit IRC12:04
openstackgerritMerged openstack/tripleo-validations master: Updated from global requirements  https://review.openstack.org/47014812:05
openstackgerritSven Anderson proposed openstack/tripleo-common master: Add EC2-API container image  https://review.openstack.org/46948612:05
*** sid1 has joined #tripleo12:05
*** karimb has quit IRC12:06
oidgarhi, seems like my undercloud heat engine is stuck, i got the following message:12:06
oidgar2017-06-07 12:00:32.425 8061 INFO heat.engine.stack [req-8dacc439-326a-41bc-8f32-c60d682c9bd4 - admin - - -] Stack DELETE FAILED (overcloud): Resource DELETE failed: resources.AllNodesDeploySteps: Engine went do12:06
oidgarwn during stack DELETE12:06
*** ccamacho is now known as ccamacho|lunch12:06
oidgaranyone knows how to fix such issue?12:06
shardyansiwen: it looks like the pingtest stack timed out - EmilienM hey you mentioned scenario002 was still broken, do you have a bug reference?12:06
*** morazi has joined #tripleo12:06
owalshratailor: did the os_traits packing happen?12:06
EmilienMshardy: this one ? https://bugs.launchpad.net/tripleo/+bug/169590412:07
openstackLaunchpad bug 1695904 in tripleo "stable/ocata scenario002 broken due to volume went to status error" [Critical,Triaged]12:07
ansiwenmandre: can you +2 this? https://review.openstack.org/#/c/469494/12:07
jtomasekhonza: we definitely can release tomorrow12:07
shardyEmilienM: ah, thanks12:07
owalshratailor: FYI http://logs.openstack.org/61/468461/7/check/gate-tripleo-ci-centos-7-containers-multinode-upgrades-nv/34ba4af/logs/undercloud/var/log/nova/nova-manage.log.txt.gz#_2017-06-07_11_53_43_88212:07
EmilienMshardy: it's for ocata. We have another bug on master, affecting HA jobs: https://bugs.launchpad.net/tripleo/+bug/1680195 (randomly AFIK)12:07
openstackLaunchpad bug 1680195 in tripleo "Random ovb-ha ping test failures" [Critical,Triaged] - Assigned to John Trowbridge (trown)12:07
*** ebarrera has quit IRC12:07
ansiwenshardy or EmilienM, maybe you as well? It's a trivial image addition. https://review.openstack.org/#/c/469494/12:08
*** ebarrera has joined #tripleo12:08
*** lucas-hungry is now known as lucasagomes12:09
*** ooolpbot has joined #tripleo12:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION12:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168019512:10
openstackLaunchpad bug 1680195 in tripleo "Random ovb-ha ping test failures" [Critical,Triaged] - Assigned to John Trowbridge (trown)12:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169590412:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169609412:10
*** ooolpbot has quit IRC12:10
openstackLaunchpad bug 1695904 in tripleo "stable/ocata scenario002 broken due to volume went to status error" [Critical,Triaged]12:10
openstackLaunchpad bug 1696094 in tripleo "CI: ovb-ha promotion job fails with 504 gateway timeout, neutron-server create-subnet timing out" [Critical,Triaged]12:10
ansiwenshardy: thanks for +2!12:11
*** karimb has joined #tripleo12:11
openstackgerritJiri Tomasek proposed openstack/tripleo-ui master: Update code to React Router v4 API  https://review.openstack.org/46674612:11
EmilienMansiwen: why gate-tripleo-ci-centos-7-ovb-containers-oooq-nv is red?12:12
*** ratailor has quit IRC12:12
owalshratailor: seems it was just added yesterday https://github.com/rdo-packages/nova-distgit/commit/361e542183d21e71695a7e9f8ad7dde488f8e82712:12
*** fragatina has joined #tripleo12:12
owalshEmilienM: ^^^ will this cause promotion to fail?12:13
*** trown|outtypewww is now known as trown12:13
EmilienMowalsh: I don't know, do we have python-os-traits in rdo?12:14
*** liverpooler has quit IRC12:14
owalshEmilienM: yea - https://review.rdoproject.org/r/#/c/6870/12:15
*** florianf has joined #tripleo12:16
owalshEmilienM: I think it's just the docker images in containers-multinode-upgrades that are affected12:16
*** jaganathan has joined #tripleo12:17
*** sid2 has joined #tripleo12:19
ansiwenEmilienM: i have no idea, but it's always red for all jobs I ever saw12:19
EmilienMansiwen: because it's always red doesn't mean we want to ignore it12:20
*** sid1 has quit IRC12:20
EmilienMansiwen: I won't +2 it until we know why it's red12:20
ansiwenEmilienM: even here: https://review.openstack.org/#/c/470147/12:21
shardyhttp://tripleo.org/cistatus.html shows it was green for some time, but the last few jobs have failed12:21
ansiwenEmilienM: it's non-voting for a reason, no?12:21
ansiwenEmilienM: all the recent merges had it red12:21
* shardy sighs12:21
EmilienMnon voting doesn't mean we don't care12:22
dciabrinmandre, hey, got some time for some kolla rebuild question? It seems when I rebuild my images locally package puppet is not installed anymore in the images :/12:22
ansiwenEmilienM: no, but merging anyway does? I don't say we shouldn't look what is going on there, I just say it's not related to my change12:23
openstackgerritJiri Tomasek proposed openstack/tripleo-ui master: Show Node Introspection Data  https://review.openstack.org/47131812:24
openstackgerritJiri Tomasek proposed openstack/tripleo-ui master: Add Introspection status to Nodes  https://review.openstack.org/46787212:24
EmilienMansiwen: the problem is that everyone wants their patches merged to get their work done, but a very few folks actually care of the CI12:24
EmilienMansiwen: if more folks doing features would do little help to maintain CI jobs, jobs would be more stable for sure12:24
EmilienMinstead of ignoring and approving patches even when jobs merge12:25
mandredciabrin: can you show me how you build the images?12:25
EmilienMmandre: do you know why the ovb-containers update is failing now? once it's green again, we should make it voting12:25
*** ratailor has joined #tripleo12:26
owalshEmilienM: timeout?12:26
ebarreraEmilienM, can you take a quick look to https://bugzilla.redhat.com/show_bug.cgi?id=1455503 ?12:26
openstackbugzilla.redhat.com bug 1455503 in openstack-tripleo-heat-templates "external ip not crrectly set for apache on 1+1 plus isolated ctlplane + external" [Unspecified,New] - Assigned to emacchi12:26
shardyNo they're all failing after about 30mins12:26
shardylooking at logs..12:26
*** rlandy has joined #tripleo12:26
shardythe undercloud install is broken12:27
dciabrinmandre, http://paste.openstack.org/show/611692/ using the tripleo_kolla_template_overrides from tripleo-common12:27
EmilienMebarrera: I'm assigned because I'm the default assignee for this projects - but I'm not actually looking at this bug now, it's out of my scope for now.12:27
shardyhttp://logs.openstack.org/74/471674/1/check/gate-tripleo-ci-centos-7-containers-multinode-upgrades-nv/6a36fb5/logs/undercloud/home/jenkins/undercloud_install.log.txt.gz#_2017-06-07_09_57_3812:27
ebarreraEmilienM, ack, do you know someone able to help ?12:28
shardyowalsh: seems to be nova-manage cellsv2 stuff12:28
ansiwenEmilienM: it's not a regression for sure, because master was failing as well. And I'm happy if it is blocked in general, and not only on my changes. :-)12:28
shardydid anything change in that recently?12:28
owalshshardy: missing the os-traits dependency12:28
mandreEmilienM: another issue we've seen recently, https://bugs.launchpad.net/tripleo/+bug/1694709 but I don't think that's it12:28
openstackLaunchpad bug 1694709 in tripleo "Reaching disk space capacity for containers job" [High,Triaged] - Assigned to Dan Prince (dan-prince)12:28
owalshshardy: I was just talking about that :-)12:28
shardyowalsh: aha12:28
shardyI assumed it'd just block promotion12:28
owalshshardy: are we building docker images from source? Nova change that requires this landed yesterday AFAIK12:29
shardyowalsh: we're building them from packages but the push to dockerhub is till manual atm12:30
shardythis is the undercloud install though, so not containerized12:30
mandredciabrin: maybe it's better for you to work from a kolla checkout rather than rpm install if you're rebuilding frequently12:31
shardyowalsh: perhaps we're using an old version, they're logged here:12:31
shardyhttp://logs.openstack.org/74/471674/1/check/gate-tripleo-ci-centos-7-containers-multinode-upgrades-nv/6a36fb5/logs/undercloud/var/log/messages.txt.gz12:32
*** zoli|brb is now known as zoli12:32
shardyopenstack-nova-common-16.0.0-0.20170607093107.0fc64f0.el7.centos.noarch12:32
*** jprovazn has quit IRC12:33
ansiwenEmilienM: you should make it voting then, because now you are blocking the wrong changes and not the causing ones.12:33
owalshshardy: this adds the requires python-nova-16.0.0-0.20170607093107.0fc64f0.el7.centos.noarch12:34
owalshEmilienM, shardy: so that just got promoted?12:34
shardyowalsh: yeah it seems so12:34
owalshhrmm, with nova api db_sync is failing?12:35
ratailorowalsh, this dependency has been added in nova.spec https://github.com/rdo-packages/nova-distgit/blob/rpm-master/openstack-nova.spec#L45112:36
sshnaidmbnemec, why not to add OVB code to openstack projects?12:36
owalshratailor: yea, but it's a bit late12:36
openstackgerritChristian Schwede proposed openstack/puppet-tripleo master: Fix Swift ring management in container deployments  https://review.openstack.org/46879012:36
EmilienMowalsh: https://dashboards.rdoproject.org/rdo-dev - we haven't had a promotion in 17 days12:37
EmilienMansiwen: I'm trying to get some help, but obviously it's hard to get some.12:37
arxcruzsshnaidm: not in a long time12:37
ratailorowalsh, couldn't get time last week because of tripleo bootcamp. it got merged today only. https://review.rdoproject.org/r/#/c/6974/212:38
jtomasekhonza: what patches do you want me to review primarily?12:40
shardyEmilienM: ah, yeah I was looking at the ocata pin12:40
shardywhy are we installing latest nova then?  Hmm..12:41
EmilienMshardy: in upgrade jobs?12:42
shardyhttp://logs.openstack.org/74/471674/1/check/gate-tripleo-ci-centos-7-containers-multinode-upgrades-nv/6a36fb5/logs/undercloud/etc/yum.repos.d/delorean.repo.txt.gz12:42
shardythat repo is wrong12:42
shardyit's pointing to current for delorean and delorean-current12:42
openstackgerritFlavio Percoco proposed openstack/tripleo-heat-templates master: WIP: Deploy kubernetes using TripleO on the overcloud  https://review.openstack.org/47175912:42
shardyjistr: ^^ any idea where we set those repos?12:42
ansiwenEmilienM: it's not that I don't want to help, I'd love to actually, I just don't have the ability. I am overwhelmed by the logs and don't see what's going on. :-(12:42
thrashyolanda: https://review.openstack.org/47175812:42
*** jaganathan has quit IRC12:43
flaper87shardy: EmilienM fultonj therve https://review.openstack.org/471759 no need to review, it's WIP but just a heads up12:43
*** karimb has quit IRC12:43
socialoh hi, can I get someone to review this https://review.openstack.org/#/c/468960 I'm quite blocked on this one12:43
*** agurenko_ is now known as agurenko12:44
EmilienMshardy: looks like a quickstart thing, since this job is not managed by tripleo-ci scripts12:45
jistrhttps://github.com/openstack-infra/tripleo-ci/blob/master/toci-quickstart/playbooks/multinode.yml#L54-L6212:45
jistrhttps://github.com/openstack/tripleo-quickstart/tree/master/roles/repo-setup12:45
jistrhttps://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/master.yml12:45
* EmilienM invokes jistr :D12:45
jistrshardy: ^ that's how i *think* it should work. There could be some bad switch along the way...12:46
shardyhttps://trunk.rdoproject.org/centos7/current-tripleo/ points to different nova version, so we're somehow writing or overwriting the wrong repo12:47
jistroh wait we're talking about upgrade job12:48
EmilienMyeah12:48
*** jpena|lunch is now known as jpena12:48
EmilienMI was looking at the code which upgrade repos12:48
shardyyeah, the undercloud install of the upgrade job12:48
EmilienMshardy: the nova version in tripleo repo is consistent between openstack-nova-* and python-nova, I don't see the problem12:49
shardyEmilienM: we're installing nova from -current not the -current-tripleo pin12:49
EmilienMwhat12:50
shardybecause the repo files for -current and -current-tripleo both point to the same repo hash12:50
shardyhttp://logs.openstack.org/74/471674/1/check/gate-tripleo-ci-centos-7-containers-multinode-upgrades-nv/6a36fb5/logs/undercloud/etc/yum.repos.d/delorean.repo.txt.gz12:50
EmilienMindeed12:50
shardyhttp://mirror.regionone.osic-cloud1.openstack.org:8080/rdo/centos7/0f/c6/0fc64f0ab663edb2564cba5868113ad6529d078b_6cc62d7012:50
EmilienMyeah12:50
shardyhttp://logs.openstack.org/74/471674/1/check/gate-tripleo-ci-centos-7-containers-multinode-upgrades-nv/6a36fb5/logs/undercloud/etc/yum.repos.d/delorean-current.repo.txt.gz12:51
*** jmelvin has joined #tripleo12:51
* EmilienM digs in quickstart code12:51
* owalsh biab, gotta grab some lunch12:52
*** atheurer has joined #tripleo12:52
jistri don't see anything that would do repo-setup on undercloud in multinode.yml12:52
*** karimb has joined #tripleo12:52
jistrthere's https://github.com/openstack-infra/tripleo-ci/blob/master/toci-quickstart/playbooks/multinode.yml#L212:52
jistrwhich points to https://github.com/openstack/tripleo-quickstart/blob/master/playbooks/quickstart.yml12:53
jistrbut i don't see any UC repo-setup either there12:53
mandreansiwen: do you have the link to the patch using the nova-consoleauth and nova-novncproxy images handy?12:54
yolandathrash, just flipped words12:55
mandreansiwen: aha, found it https://review.openstack.org/#/c/469576/, sorry for the noise12:55
thrashyolanda: oops. :D12:55
EmilienMflaper87: ack12:56
*** jaganathan has joined #tripleo12:56
*** liverpooler has joined #tripleo12:56
ansiwenmandre: yes, and https://review.openstack.org/#/c/469541/12:56
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci master: TEST: DONT RECHECK or REVIEW: periodic jobs  https://review.openstack.org/35921512:57
*** ccamacho|lunch is now known as ccamacho12:57
*** dparkes has joined #tripleo12:57
*** trown is now known as trown|brb12:58
*** pkovar has quit IRC12:58
thrashyolanda: updated. Good catch.12:58
ansiwenmandre: thanks for +212:59
*** zoli is now known as zoli|afk13:00
*** agopi has joined #tripleo13:00
*** ffiore_ has joined #tripleo13:00
*** agurenko is now known as agurenko_13:01
openstackgerritBrent Eagles proposed openstack/tripleo-docs master: Post deployment installation instructions for Octavia  https://review.openstack.org/44749613:02
shardyhttps://github.com/openstack/tripleo-quickstart/blob/master/roles/repo-setup/defaults/main.yml#L2413:02
shardyshouldn't that be delorean-current.repo?13:03
openstackgerritwes hayutin proposed openstack/tripleo-quickstart master: Add support for external network bridges  https://review.openstack.org/40642113:03
*** pkovar has joined #tripleo13:04
shardyhttp://logs.openstack.org/74/471674/1/check/gate-tripleo-ci-centos-7-containers-multinode-upgrades-nv/6a36fb5/logs/undercloud/home/jenkins/repo_setup.log.txt.gz13:04
shardythat shows we download the right files, but somehow copy the wrong delorean.repo file from repo_role_tmp13:05
openstackgerritmathieu bultel proposed openstack/tripleo-common master: Implement upgrade workflow  https://review.openstack.org/46376513:05
jistrshardy: possibly it should be delorean-current.repo, but i think the `repos` variable is just completely overriden by this https://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/master.yml13:06
jistrshardy: so it shouldn't be a problem13:06
jistrshardy: i still don't see how we run a repo-setup on undercloud though, i'm starting to think we don't...13:07
shardyAh, I was trying to match the log output with repo_setup.sh.j2 and that perhaps explains why it doesn't match except for delorean-deps13:07
shardyjistr: doesn't that log output show it happening?13:08
jistrshardy: ah right it does13:08
* jistr looks at console.html13:08
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates master: Fix the disable expirer to remove crontab  https://review.openstack.org/47035713:08
shardythe repo_setup.log shows downloading repo files, and it shows the wrong delorean.repo at the end13:08
shardyI just haven't seen exactly where/why13:09
openstackgerritCarlos Camacho proposed openstack/python-tripleoclient master: Add openstack undercloud backup  https://review.openstack.org/46621313:09
jistrshardy: found it https://github.com/openstack/tripleo-quickstart-extras/blob/a7b34d30656b9d9bb3030954419aeb8ce75eef7b/roles/undercloud-setup/tasks/main.yml#L13-L1513:09
*** ooolpbot has joined #tripleo13:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION13:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168019513:10
openstackLaunchpad bug 1680195 in tripleo "Random ovb-ha ping test failures" [Critical,Triaged] - Assigned to John Trowbridge (trown)13:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169590413:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169609413:10
*** ooolpbot has quit IRC13:10
openstackLaunchpad bug 1695904 in tripleo "stable/ocata scenario002 broken due to volume went to status error" [Critical,Triaged]13:10
openstackLaunchpad bug 1696094 in tripleo "CI: ovb-ha promotion job fails with 504 gateway timeout, neutron-server create-subnet timing out" [Critical,Triaged]13:10
openstackgerritmathieu bultel proposed openstack/tripleo-common master: Implement upgrade workflow  https://review.openstack.org/46376513:10
*** ffiore has joined #tripleo13:11
*** ffiore_ has quit IRC13:11
EmilienMshardy: just a quick fyi on upgrade jobs that are all passing (except the container one): https://review.openstack.org/#/c/46769813:11
EmilienMit would be nice to keep it going :D13:11
EmilienMmaking them voting and having them in promotion pipeline will certainly help13:12
shardyEmilienM: Nice!  So much green! :)13:12
*** toure|biab is now known as toure13:13
*** trown|brb is now known as trown13:13
openstackgerritOpenStack Proposal Bot proposed openstack/python-tripleoclient master: Updated from global requirements  https://review.openstack.org/47055513:13
jistrshardy: can you explain what do you mean by 'copy wrong delorean.repo'?13:14
jistrshardy: i'm looking at the script and it looks correct to me... http://logs.openstack.org/74/471674/1/check/gate-tripleo-ci-centos-7-containers-multinode-upgrades-nv/6a36fb5/logs/undercloud/home/jenkins/repo_setup.sh.txt.gz13:14
EmilienMyolanda: i self approved the t-i-e patch, please update the releases patch when merged13:14
openstackgerritDmitry Tantsur proposed openstack/python-tripleoclient master: [WIP] Add 'overcloud node discover' command  https://review.openstack.org/47176713:14
*** tzumainn has joined #tripleo13:15
* jistr double checking the logs13:16
shardyjistr: search for "repo: delorean" and "repo: delorean-deps" in the repo_setup log13:17
shardythey are the same13:17
shardyboth pointing to -current13:17
shardy0fc64...13:17
bogdandojistr: bad luck for the patch http://logs.openstack.org/22/466322/4/check/gate-tripleo-ci-centos-7-containers-multinode-upgrades-nv/f44b8aa/logs/undercloud/home/jenkins/undercloud_install.log.txt.gz#_2017-06-07_10_58_5813:17
shardyearlier in the log we download current-tripleo/delorean.repo13:17
shardy38/47/38471d4ccf3914805fafaa56b21db2cc83755e95_5d6d179f13:18
shardybut for some reason, we copy current/delorean.repo to both repo files under /etc13:18
bogdandoEmilienM, jistr: oh, the same bug as well13:19
bogdando> 15:11:47 - EmilienM: shardy: just a quick fyi on upgrade jobs that are all passing (except the container one): https://review.openstack.org/#/c/46769813:19
*** gbarros has joined #tripleo13:19
openstackgerritCarlos Camacho proposed openstack/python-tripleoclient master: Add openstack undercloud backup  https://review.openstack.org/46621313:20
jistrshardy: oh i see it, fix coming right up. It's the mirror URL replacement logic, there's copypasta mistake.13:20
mandrejistr: looks like https://review.openstack.org/#/c/462211/ is mergeable13:20
shardyjistr: aha, well spotted!13:21
*** ebarrera has quit IRC13:21
shardymandre: yeah would be good to land that one I think, I'm re-testing the md5sum one locally now so can rebase after that lands13:22
*** ebarrera has joined #tripleo13:22
openstackgerritDan Trainor proposed openstack/instack-undercloud master: Add proxy and config endpoints of ironic-inspector  https://review.openstack.org/47005813:22
openstackgerritBrent Eagles proposed openstack/tripleo-docs master: Post deployment installation instructions for Octavia  https://review.openstack.org/44749613:23
openstackgerritMichael Chapman proposed openstack/tripleo-quickstart master: qemu://system fixes  https://review.openstack.org/47157813:24
mandreshardy: cool! I've blocked some time for code reviews, anything special you need me to look at?13:25
jistrshardy: is there a bug reported yet that i can tag in the commit message?13:25
jistrif not i could perhaps just send it as-is13:25
shardyjistr: no I've not raised one yet13:26
shardymandre: not yet, the md5sum one is failing ci and I'm still testing some follow-up patches locally, thanks though :)13:26
shardywill be good to land the ansible one as it's been around for a while13:26
*** pradk has quit IRC13:27
openstackgerritCarlos Camacho proposed openstack/python-tripleoclient master: Add openstack undercloud backup  https://review.openstack.org/46621313:28
openstackgerritJiri Stransky proposed openstack/tripleo-quickstart master: Fix repo mirror logic for tripleo-ci/master-undercloud-ocata-overcloud  https://review.openstack.org/47177413:28
*** jcoufal has joined #tripleo13:29
jistrshardy, EmilienM ^^13:29
*** egafford has joined #tripleo13:29
karthiksshardy,  can you please review https://review.openstack.org/#/c/466693/ and https://review.openstack.org/#/c/469164/13:30
EmilienMjistr: nice shot13:30
EmilienMjistr: thanks13:30
*** links has quit IRC13:30
*** skramaja has quit IRC13:31
*** jprovazn has joined #tripleo13:31
*** amoralej is now known as amoralej|lunch13:31
jistrmandre: wow so green13:31
* jistr reviews13:31
*** jaganathan has quit IRC13:32
openstackgerritwes hayutin proposed openstack/tripleo-quickstart master: enable the rhsm/cdn yum repos in repo-setup  https://review.openstack.org/47015013:32
openstackgerritwes hayutin proposed openstack/tripleo-quickstart master: add a config for building images  https://review.openstack.org/45637313:33
openstackgerritwes hayutin proposed openstack/tripleo-quickstart-extras master: add steps to enable tripleo_ui  https://review.openstack.org/45906113:33
openstackgerritwes hayutin proposed openstack/tripleo-quickstart master: Add ssh tunneling for TripleO-UI  https://review.openstack.org/40773313:33
openstackgerritwes hayutin proposed openstack/tripleo-quickstart master: baseos-undercloud release configuration for all supported releases.  https://review.openstack.org/46403313:33
*** ansmith has joined #tripleo13:34
openstackgerritCarlos Camacho proposed openstack/python-tripleoclient master: Add openstack undercloud backup  https://review.openstack.org/46621313:37
openstackgerritMartin André proposed openstack/tripleo-heat-templates master: Make container names consistent  https://review.openstack.org/47177813:38
*** Goneri has joined #tripleo13:39
*** psahoo has quit IRC13:39
*** jmelvin is now known as jmelvin|brb13:40
*** florianf has quit IRC13:41
openstackgerritDmitry Tantsur proposed openstack/python-tripleoclient master: [WIP] Add 'overcloud node discover' command  https://review.openstack.org/47176713:42
shardykarthiks: done!13:42
karthiksshardy,  Thanks13:43
*** chlong has quit IRC13:43
*** ratailor has quit IRC13:43
*** jaganathan has joined #tripleo13:44
*** noslzzp has joined #tripleo13:44
mandrejistr: I know! make sure to put your sunglasses on13:44
jistr:D13:44
openstackgerritChristian Schwede proposed openstack/puppet-tripleo master: Fix Swift ring management in container deployments  https://review.openstack.org/46879013:45
*** jmelvin|brb has quit IRC13:45
trownlol... if you review with rose colored glasses, you can ignore red CI13:46
openstackgerritCarlos Camacho proposed openstack/python-tripleoclient master: Add openstack undercloud backup  https://review.openstack.org/46621313:47
*** rcrit has quit IRC13:48
jistrshardy++ nice work @ https://review.openstack.org/46221113:51
jistrthis is going to be easier to explain on the deep dive. The ordering of actions is much more apparent in the playbook13:51
*** gkadam has joined #tripleo13:52
openstackgerritMarios Andreou proposed openstack/tripleo-validations master: Add undercloud service status check before update/upgrade  https://review.openstack.org/46291813:53
openstackgerritMichele Baldessari proposed openstack/tripleo-common master: Do not use multiple overrides with the same name  https://review.openstack.org/47178113:54
*** pchavva has joined #tripleo13:57
openstackgerritMichele Baldessari proposed openstack/tripleo-common master: Do not use multiple overrides with the same name  https://review.openstack.org/47178113:57
openstackgerritCarlos Camacho proposed openstack/python-tripleoclient master: Add openstack undercloud backup  https://review.openstack.org/46621313:57
*** florianf has joined #tripleo13:58
bogdandofolks, updated comments for https://bugs.launchpad.net/tripleo/+bug/169470914:00
openstackLaunchpad bug 1694709 in tripleo "Reaching disk space capacity for containers job" [High,Triaged] - Assigned to Dan Prince (dan-prince)14:00
bogdandoresluts are frustrating14:00
openstackgerritmathieu bultel proposed openstack/python-tripleoclient master: Implement Upgrade workflow via the tripleo cli  https://review.openstack.org/46372814:01
shardyjistr: thanks, yeah that was what I was aiming for14:01
florianfbnemec: Hi, I put up a patch for a pre-upgrade validation you suggested (disk space): https://review.openstack.org/#/c/471686/14:01
shardyjistr: and it's much easier to run the entire deploy sequence on each node (for minor updates)14:01
florianfbnemec: Just wanted to make sure I picked up the 10GB requirement right.14:01
matbushardy: hey, can you check my comment in this review, when you have a minute https://review.openstack.org/#/c/44422414:02
*** nyechiel has quit IRC14:02
jistrshardy: nice14:03
*** dparkes has quit IRC14:05
*** amoralej|lunch is now known as amoralej14:06
shardyjistr: replied to your comment - it does depend on the host puppet step114:06
shardythanks for the review :)14:06
*** hjensas has quit IRC14:06
*** agurenko has joined #tripleo14:06
jistrbogdando, mandre: this might have been made worse by me pushing the keystone container image (to fix the critical CI issue earlier this week) separately from the other container images. I didn't want to rebuild them all because 1) i have no idea if my env is correct for it, so i wanted to make the smallest possible change. 2) dprince mentioned at some point that he baked in some special HA ingredients from14:06
jistrbandini/dciabrin to make testing with pacemaker possible, and i wasn't sure if that's still the case, so i didn't want to undo the changes 3) it was getting late in the evening and i wanted to go home14:06
jistrbogdando: that's re the issue that we're running out of space for images14:07
*** bfournie has quit IRC14:07
shardymatbu: +1 on starting with the CLI approach14:07
shardymatbu: FYI I started a SwiftObject plugin for heat here: https://review.openstack.org/#/c/469039/14:07
shardyI'll try to finish it this week14:07
jistrbogdando, mandre: having a separately built keystone image means it doesn't share the layers below with the other images14:07
bogdandojistr, mandre: hm, that's actually a good point. The way we rebuild images and reuse a common base14:08
mandrejistr: it's ok, I'll push a new batch of container images very soon14:08
jistrmandre: great, thanks a lot! i think that could mitigate the space issue14:08
*** dparkes has joined #tripleo14:08
mandrejistr: it depends on how you build the images, there is actually a way to reuse existing layers in kolla but you need to have them in your docker cache14:08
bogdandojistr: right, and we really should double-check all of the images use the common centos base and properly layered14:09
bogdandoall the time we push rebuild14:09
*** agurenko_ has quit IRC14:09
tdasilvatrown: hello, have you seen this error before: http://paste.openstack.org/show/611718/ ?14:09
*** ykarel__ has quit IRC14:09
*** ooolpbot has joined #tripleo14:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION14:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168019514:10
openstackLaunchpad bug 1680195 in tripleo "Random ovb-ha ping test failures" [Critical,Triaged] - Assigned to John Trowbridge (trown)14:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169590414:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169609414:10
*** ooolpbot has quit IRC14:10
openstackLaunchpad bug 1695904 in tripleo "stable/ocata scenario002 broken due to volume went to status error" [Critical,Triaged]14:10
jistrmandre: nice, i didn't know we could do that14:10
openstackLaunchpad bug 1696094 in tripleo "CI: ovb-ha promotion job fails with 504 gateway timeout, neutron-server create-subnet timing out" [Critical,Triaged]14:10
*** jmelvin has joined #tripleo14:12
trowntdasilva: I have not, but that seems like you have an old checkout of quickstart https://github.com/openstack/tripleo-quickstart/blob/master/roles/parts/ovs/tasks/main.yml14:12
trowntdasilva: could you delete the .quickstart dir and run again? or run with --clean (which does the same thing)14:12
*** gkadam has quit IRC14:13
tdasilvatrown: that's the weird part, i always delete .quickstart AND run with --clean14:13
openstackgerritMartin André proposed openstack/tripleo-common master: Add workaround for missing dep in octavia image  https://review.openstack.org/47178814:13
*** bfournie has joined #tripleo14:14
matbushardy: ack cool thx :)14:14
bogdandojistr, mandre: so let's fix that then :) we know the problem now14:14
*** noslzzp has quit IRC14:14
trowntdasilva: but from a local git checkout with --no-clone? The code in the paste does not match the code in the upstream repo14:15
matbushardy: do you think its doable to add it in pike M2 ? (i mena the cli patch)14:15
matbus/mena/mean/14:15
openstackgerritCarlos Camacho proposed openstack/python-tripleoclient master: Add openstack undercloud backup  https://review.openstack.org/46621314:15
tdasilvatrown: been running like this: ./quickstart.sh --requirements quickstart-extras-requirements.txt --playbook quickstart-extras.yml --config ~/.quickstart/config/general_config/containers_minimal.yml --teardown all --tags all --clean --release master 127.0.0.214:16
*** janki has quit IRC14:16
shardymatbu: the first step is probably to fix the tests etc, the unit test failures are real14:17
trowntdasilva: ah I see it is just bad error message from ansible... code is at https://github.com/openstack/tripleo-quickstart/blob/master/roles/environment/setup/tasks/main.yml#L214:17
openstackgerritAlex Schultz proposed openstack/tripleo-heat-templates master: Standardize example role definitions  https://review.openstack.org/44568714:17
shardym2 is now, so I wouldn't expect it to necessarily land in time for that14:17
trowntdasilva: odd thing is that has not changed in a couple months... something else must be happening in your environment14:18
matbushardy: ack, i can fix pep8 and UT asap14:18
tdasilvatrown: correct, i noticed that too, that code was last introduced back in dec, i think14:18
tdasilvaand i've run oooq many times since ;)14:19
mandrejistr, shardy: can you do a very quick review before I start re-building the images? https://review.openstack.org/#/c/471781/ and https://review.openstack.org/#/c/471788/14:19
openstackgerritmathieu bultel proposed openstack/python-tripleoclient master: Implement Upgrade workflow via the tripleo cli  https://review.openstack.org/46372814:19
openstackgerritAlex Schultz proposed openstack/tripleo-heat-templates master: Standardize example role definitions  https://review.openstack.org/44568714:19
trowntdasilva: can you try without adding --requirements? I think that would be overriding all of the requirements files, and we did recently change quickstart to use everything from the venv, so you might be missing stuff14:20
shardymatbu: also I wonder if we might consider something more generic and not upgrade specific14:20
shardymatbu: like openstack overcloud config download ?14:20
shardythat could output all the config data from role_data and be useful for more than just upgrade14:20
matbushardy: hmm yep14:21
tdasilvatrown: running now14:21
*** zoli|afk is now known as zoli|wfh14:21
*** prateek has quit IRC14:22
matbushardy: so you mean starting a new overcloud_config.py, where we can manage all those config related stuffs?14:22
shardymatbu: yes, just something which can download the various configs from the overcloud and write them to disk14:23
openstackgerritAlex Schultz proposed openstack/tripleo-heat-templates master: Standardize example role definitions  https://review.openstack.org/44568714:23
shardye.g not only upgrade_tasks14:23
jpichakrivoka: What do you think about bringing up the "stop using mistral env" patches at the next meeting? It has so many implications including for upgrades, I think it'd be better to merge earlier rather than later14:23
shardybut the step_config puppet manifest, config_settings variables, etc14:23
jpichIf we want it in Pike (I kinda should have thought about this yesterday, sorry!)14:24
shardyall of role_data for each role basically, but perhaps reformatted into a useful format on disk14:24
akrivokajpich: sure, makes sense14:24
*** cdearborn has joined #tripleo14:24
*** gbarros has quit IRC14:24
matbushardy: ack i'll rework that14:24
shardyas it's already possible to dump it as yaml/json directly with openstack stack output show14:24
*** pradk has joined #tripleo14:24
shardymatbu: thanks - that would mean we could use the same approach for manual minor updates potentially14:25
jpichakrivoka: Ok! Adding it now so I don't forget :)14:25
akrivokajpich: thank you :)14:25
jistrmandre: is python-cotyledon packages? I see it mentioned in ocata RDO somewhere, but i don't see it in http://trunk.rdoproject.org/centos7/current-tripleo/14:26
jistrs/packages/packaged/14:26
*** ebarrera_ has joined #tripleo14:26
jistrah i forgot to check delorean-deps14:27
jpichakrivoka: Gah I just noticed there's yet another merge conflict as well, I'll look into it as well14:27
jistrmandre: yea it's there, sorry for false alarm +2'd both14:27
*** pradk has quit IRC14:28
mandrejistr: ack, thanks14:28
akrivokajpich: seems like gerrit doesn't send email notifications on merge conflicts? :\14:29
jpichakrivoka: This is weird, I see a merge conflict on the dashboard but not when I look at the patch directly... O_o14:29
mandreFWIW I think we can just land them, because these files are not tested in CI yet14:29
*** gbarros has joined #tripleo14:29
sshnaidmhrybacki, jaosorior  https://trello.com/c/Irj8mWDK14:29
akrivokajpich: I'm seeing a red "Cannot Merge" on the patch14:30
jaosoriorsshnaidm: makes sense to me14:30
openstackgerritCarlos Camacho proposed openstack/python-tripleoclient master: Add openstack undercloud backup  https://review.openstack.org/46621314:30
jpichakrivoka: Huh. doesn't show for me even after refresh. ¯\_(ツ)_/¯14:30
*** pradk has joined #tripleo14:31
akrivokajpich: I can resolve the conflict if that's ok with you?14:31
openstackgerritDimitri Savineau proposed openstack/puppet-tripleo master: Cinder: Add NexentaStor Support  https://review.openstack.org/37095614:31
*** jaosorior is now known as jaosorior_away14:31
jpichakrivoka: Oh sure, be my guest! Doesn't look to be a too mean one14:32
*** karimb has quit IRC14:32
*** limao has joined #tripleo14:36
hrybackisshnaidm: ack, thank you!14:36
akrivokajpich: ah the conflict seems due to another one of your commits :) https://github.com/openstack/tripleo-common/commit/c824b58a22b6824eb4f4f1d3bc50ebc443e440f914:37
jpichakrivoka: Sorryyyy14:37
akrivokajpich: lol no worries :)14:37
jpich:)14:37
*** ebarrera_ has quit IRC14:37
openstackgerritCarlos Camacho proposed openstack/python-tripleoclient master: Add openstack undercloud backup  https://review.openstack.org/46621314:38
openstackgerritYedidyah Bar-David proposed openstack/tripleo-quickstart master: Do not log virt_power private key  https://review.openstack.org/47180214:38
*** limao_ has joined #tripleo14:40
*** limao has quit IRC14:40
*** sri_ has joined #tripleo14:42
*** didib has joined #tripleo14:42
*** pkovar has quit IRC14:43
*** pkovar has joined #tripleo14:44
*** karimb has joined #tripleo14:46
ansiwenmandre: why did you +2 this: https://review.openstack.org/#/c/469576/  but not this: https://review.openstack.org/#/c/469541/ ?14:47
pandaSlower: by default kolla installs packages from the host repository ?14:47
ansiwenmandre: I locally deployed both and actually got a authorized VNC connection to a booted instance. so it is working, that much I know :-)14:47
ansiwenmandre: I'm just trying to understand what you are looking at, because I'm still lost in all the CI logs14:48
mandreansiwen: I'm waiting to see the containers be up in CI, the job for the nova-consoleauth container timed out and didn't complete log collection14:51
openstackgerritAna Krivokapic proposed openstack/tripleo-common master: Use Swift to store Plan environment  https://review.openstack.org/45229114:52
akrivokajpich: ^14:52
*** lmiccini has quit IRC14:53
jpichakrivoka: Yay, thanks!14:53
pradkmandre, so regarding the tacker errors.. this seems like something new we picked up in the recent builds14:54
ansiwenmandre: which job exactly? there are many, and for the other one that you +2 also14:58
*** didib has quit IRC15:00
mandrepradk: ok, this will likely be fixed with newer container image15:00
*** karimb has quit IRC15:01
*** mdnadeem has joined #tripleo15:01
mandrepradk: I'll recheck when I'm done rebuilding15:01
*** isq has quit IRC15:01
mandreansiwen: I'm looking at gate-tripleo-ci-centos-7-ovb-containers-oooq-nv result15:02
*** karimb has joined #tripleo15:02
mandreansiwen: it seems we do not enable nova-consoleauth in the multinode container job15:02
openstackgerritCarlos Camacho proposed openstack/python-tripleoclient master: Add openstack undercloud backup  https://review.openstack.org/46621315:04
ansiwenmandre: what about all the other container jobs? I can see 415:05
*** lmiccini has joined #tripleo15:05
*** hjensas has joined #tripleo15:06
*** hjensas has quit IRC15:06
*** hjensas has joined #tripleo15:06
mandreansiwen: they don't test your container as far as I can tell15:07
ansiwenmandre: and that's a bug or a feature? ;-) do I have to add it somwhere else beside docker.yaml?15:07
mandreansiwen: it is tested by gate-tripleo-ci-centos-7-ovb-containers-oooq-nv, the one I'm waiting the result for15:09
mandreansiwen: :)15:09
*** ooolpbot has joined #tripleo15:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION15:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168019515:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169590415:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169609415:10
*** ooolpbot has quit IRC15:10
openstackLaunchpad bug 1680195 in tripleo "Random ovb-ha ping test failures" [Critical,Triaged] - Assigned to John Trowbridge (trown)15:10
openstackLaunchpad bug 1695904 in tripleo "stable/ocata scenario002 broken due to volume went to status error" [Critical,Triaged]15:10
openstackLaunchpad bug 1696094 in tripleo "CI: ovb-ha promotion job fails with 504 gateway timeout, neutron-server create-subnet timing out" [Critical,Triaged]15:10
ansiwenmandre: yes, but my question is, where is it configured what is tested? because as far as I can tell it should be in the defailt services for the controller role. so if the other jobs don't deploy it, I am wondering why.15:10
openstackgerritMerged openstack/tripleo-heat-templates master: Stop/disable l3 agent in docker service upgrade_tasks  https://review.openstack.org/46992415:11
*** gvrangan has joined #tripleo15:11
*** SlickNik- has joined #tripleo15:11
*** rcernin has quit IRC15:11
*** SlickNik has quit IRC15:12
openstackgerritMerged openstack/tripleo-image-elements master: Release 7.0.0.0b2  https://review.openstack.org/47164215:12
*** eggmaster has joined #tripleo15:15
openstackgerritCarlos Camacho proposed openstack/python-tripleoclient master: Add openstack undercloud backup  https://review.openstack.org/46621315:16
mandrejistr: did you know heat-api is doa on the multinode-upgrade job? http://logs.openstack.org/41/469541/3/check/gate-tripleo-ci-centos-7-containers-multinode-upgrades-nv/1a720f8/logs/subnode-2/var/log/extra/docker/containers/heat_api/stdout.log.txt.gz15:19
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-quickstart-extras master: Ensure liberty support for images preparation role.  https://review.openstack.org/47182415:19
mandreansiwen: I'm not too familiar with it, but I think the multinode jobs only enable a subset of services, and ultimately everything is tested via the different scenarios15:20
owalshansiwen: controller services are overridden for multinode e.g https://github.com/openstack/tripleo-heat-templates/blob/master/ci/environments/multinode-containers.yaml#L1815:21
mandreansiwen: see for example https://github.com/openstack/tripleo-heat-templates/blob/master/ci/environments/scenario002-multinode-containers.yaml15:21
ansiwenowalsh: thanks, that's exactly the file that I was now finally looking at :-)15:22
*** noslzzp has joined #tripleo15:22
openstackgerritMarios Andreou proposed openstack/tripleo-validations master: Add undercloud service status check before update/upgrade  https://review.openstack.org/46291815:23
jistrmandre: no i didn't. We should probably add pingtest after upgrade, though some of the jobs complete around 2.5h mark, so i'm a bit worried that we'd start hitting timeouts soon... Might be still worth a shot though.15:24
sshnaidmbnemec, does OVB support creating swap on the host?15:25
sshnaidmrlandy, the question raised because if we want to use separate node for undercloud, we need enough memory15:25
sshnaidmrlandy, right now we have 8G swap in addition to 8G of host15:26
openstackgerritMerged openstack/tripleo-quickstart master: include tqe changes in basic.sh gate jobs  https://review.openstack.org/47144415:26
*** chlong has joined #tripleo15:26
sshnaidmrlandy, so I'm not sure ci.m1.large will be enough..15:26
sshnaidmrlandy, how much memory do you have in your jobs?15:27
rlandysshnaidm, ci.m1.xlarge?15:27
rlandywould that work?15:27
sshnaidmrlandy, yeah, but it's crazy waste of disk space15:27
*** dsneddon has joined #tripleo15:27
sshnaidm160GB of disk, omg15:28
rlandysshnaidm: problem is we don't have control over flavors15:28
sshnaidmrlandy, yeah, and I'm not sure we don't exceed quotas quickly using xlarge15:28
sshnaidmswap seems a not bad workaround for that15:28
rlandywe waste some form of quota meeting some other requirement all the time with fixed flavors15:29
rlandyI'd agree15:29
sshnaidmrlandy, but maybe it improved - how much memory do you have in jobs on undercloud?15:29
bnemecI believe swap today is created as a file on the undercloud host.15:30
bnemecBut we really don't want to be using swap if we can help it.15:30
*** gvrangan has quit IRC15:30
*** dparkes has quit IRC15:31
*** itlinux has joined #tripleo15:34
*** karimb has quit IRC15:35
ansiwenmandre: I always thought, the scenarios are for the opt-in services, wasn't aware that we also remove default services in CI runs15:36
*** udesale has quit IRC15:36
*** karimb has joined #tripleo15:36
openstackgerritMerged openstack/tripleo-quickstart-extras master: Migrate quickstart-extras.yml --> TQE  https://review.openstack.org/46961415:37
openstackgerritMerged openstack/tripleo-quickstart master: Migrate quickstart-extras.yml --> TQE  https://review.openstack.org/46961115:37
*** agurenko has quit IRC15:38
*** itlinux has quit IRC15:42
*** atheurer has quit IRC15:42
*** yprokule has quit IRC15:44
sshnaidmpanda, ping15:44
*** panda is now known as panda|bbl15:45
panda|bblsshnaidm: I'll be back in ~2 hours15:45
panda|bblsshnaidm: really have to run now15:46
panda|bblsorry15:46
abishophey experts, I need help with an upgrade issue when cinder-volume is controlled by pacemaker15:46
abishopproblem is after upgrade it's being restarted *before* puppet restarts the other cinder services15:46
abishopI'm hoping there's a way for pacemaker to restart cinder-volume after puppet15:46
sshnaidmpanda|bbl, np, I will write in review15:46
*** ckyriakidou has quit IRC15:46
*** ckyriakidou has joined #tripleo15:47
*** fragatina has quit IRC15:48
openstackgerritMarios Andreou proposed openstack/tripleo-validations master: Add undercloud service status check before update/upgrade  https://review.openstack.org/46291815:48
openstackgerritBrent Eagles proposed openstack/tripleo-heat-templates stable/newton: DO NOT MERGE: Testing a fix  https://review.openstack.org/47138115:49
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates stable/ocata: Expose metric delay processing metric  https://review.openstack.org/46921715:50
*** oidgar has quit IRC15:50
openstackgerritMartin André proposed openstack/tripleo-quickstart-extras master: Allow deleting all of docker cache  https://review.openstack.org/47183215:51
radezdtantsur: ping, making some progress on the ironic and arm stuff have a question if you're around?15:53
dtantsurradez: I am, hi!15:54
radezhey dtantsur, so here's where I'm at... I've proivisioned my overcloud nodes and trying to get them to boot15:54
radezif I direct boot via libvirt with overcloud and kernel/rd I can boot ok, but if i boot with ironic I get avcs15:54
radezI tried disabling selinux by hand and then I can boot15:55
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-quickstart master: Fix 1ctlr deployment.  https://review.openstack.org/47183515:55
openstackgerritMartin André proposed openstack-infra/tripleo-ci master: Workaround for limited disk space on undercloud  https://review.openstack.org/47183615:55
radezis there a way as a temp workaround to pass enforcing=0 to all my nodes so I can boot and investigate the avcs?15:55
openstackgerritBrent Eagles proposed openstack/puppet-tripleo master: Remove unnecessary references to neutron core plugin hiera  https://review.openstack.org/47183715:56
dtantsurradez: you were changing some files in /tftpboot, right? I think you may need to update its attributes15:56
openstackgerritPetr Blaho proposed openstack/tripleo-quickstart-extras master: Adds 'exit 1' to validate-overcloud-ipmi-connection template  https://review.openstack.org/47183815:56
dtantsurradez: hmm, or are the denials on the nodes themselves? is it in the under or overcloud?15:56
radezdtantsur: in the nodes themselves in the overcloud, I'm able to pxe and provision successfully now15:56
radezand the overcloud nodes boot ok15:56
*** iranzo has quit IRC15:56
radezby boot I mean the kernel loads and the drives mount properly15:57
dtantsurso at which stage to you get the denial?15:57
radezbut in the systemd process I get the avcs on the overcloud nodes15:57
mandrebogdando: let's see if https://review.openstack.org/471832 and https://review.openstack.org/471836 help with the disk space issue15:57
radezdtantsur: lemme get a log15:57
mandrebogdando: it's a bit brutal but it should work :)15:57
dtantsurradez: otherwise you can pass selinux=0 to the kernel command line15:58
radezdtantsur: that was my short term workaround idea, is there a way to set that as a property on the ironic nodes?15:58
openstackgerritwes hayutin proposed openstack/tripleo-quickstart master: enable the rhsm/cdn yum repos in repo-setup  https://review.openstack.org/47015015:58
trowndtantsur: we were able to do that manually... how do we tell ironic to do that?15:58
openstackgerritwes hayutin proposed openstack/tripleo-quickstart master: enable the rhsm/cdn yum repos in repo-setup  https://review.openstack.org/47015015:59
dtantsurradez, trown, if you're using local boot (the default in ooo), it's not so easy15:59
openstackgerritAttila Darazs proposed openstack/tripleo-quickstart-extras master: GATE CHECK  https://review.openstack.org/43027716:00
*** atheurer has joined #tripleo16:00
radezdtantsur: :(16:00
radezdtantsur: happen to have any docs that could help me through it?16:00
trowndtantsur: ya localboot, we have to do it in the configdrive? was looking for where we do that?16:00
dtantsurradez, trown, you can use whole-disk images (ask yolanda how), then you're in control of grub arguments16:00
trowndtantsur: ah k.. configdrive is too late?16:01
dtantsurtrown: yep. it's read after the kernel is booted16:01
radezdtantsur: maybe I'll try and just catch the nodes manually to get the deploy to run and then investigate?16:01
*** fzdarsky is now known as fzdarsky|afk16:01
*** mcornea has quit IRC16:01
dtantsurradez: yeah, maybe this issue is easy to fix, and you won't need to mess with kernel arguments16:01
dtantsurI suspect we need to whitelist something somewhere in the end16:02
*** jlinkes has quit IRC16:02
radezdtantsur: is there a place I could just hack into the ironic code selinux=0 to get it up and running?16:02
trowndtantsur: I am suspicious that config drive is actually causing the avcs... do you know where we create that?16:02
dtantsurradez: it's on the ramdisk, it may be quite hard to hack..16:02
dtantsurtrown: this is possible, though it's strange that it only happens on ARM16:03
radezdtantsur: ah, kk16:03
trowndtantsur: ya... maybe something very low level has bad packaging though16:03
beaglesshardy: did Or ping you about missing neutron /etc/neutron/conf.d subdirectories?16:03
dtantsurtrown: maybe worth checking openstack-selinux package16:03
beaglesshardy: was just wondering what the problem/solution was - running into it myself right now16:04
radezdtantsur: k, lemme see if I can work around it manually and get into the machines to look at the avcs more in depth16:04
dtantsurradez, trown, you can temporary disable local boot (change boot_option capability to netboot on both flavor on nodes)16:04
dtantsurradez, trown, in this case the nodes will always PXE boot16:04
trownradez: ^ seems like a good immediate workaround16:04
*** gkadam has joined #tripleo16:04
dtantsurand you can hack this option: https://github.com/openstack/ironic/blob/master/etc/ironic/ironic.conf.sample#L328216:04
radezand when they pxe they will just fall back to disk because they're already provisioned?16:05
trownradez: at least would allow you to move forward and see if that is the only issue16:05
dtantsurradez: no, ironic will provide kernel and ramdisk, then they'll continue booting from disk16:05
radezack16:05
shardybeagles: hey, I think mandre has been taking a look and helping with the octavia patches16:06
radezkk, I'll put that in my pocket too, I think I have a couple options to help me move forward here, thx for the help dtantsur trown I'll circle back after I try them16:06
shardybeagles: sorry been a bit busy so not yet looked into it myself16:06
beaglesshardy: ah cool16:06
beaglesshardy: no problem16:06
*** noslzzp has quit IRC16:07
*** ooolpbot has joined #tripleo16:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION16:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168019516:10
openstackLaunchpad bug 1680195 in tripleo "Random ovb-ha ping test failures" [Critical,Triaged] - Assigned to John Trowbridge (trown)16:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169590416:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169609416:10
*** ooolpbot has quit IRC16:10
openstackLaunchpad bug 1695904 in tripleo "stable/ocata scenario002 broken due to volume went to status error" [Critical,Triaged]16:10
openstackLaunchpad bug 1696094 in tripleo "CI: ovb-ha promotion job fails with 504 gateway timeout, neutron-server create-subnet timing out" [Critical,Triaged]16:10
*** ffiore has quit IRC16:11
*** ebarrera has quit IRC16:12
*** aufi has quit IRC16:13
bogdandomandre: interesting, I'm not sure that works, never tried before to run a container from an image then just remove the ground from beneath its legs so brutally :D16:13
bogdandomandre: I thought that docker run-time COW layer is based on top of the stored image16:14
bogdandoand if stopped docker, removed the image and started, the container will never fly again16:14
*** cylopez has joined #tripleo16:15
*** karimb has quit IRC16:16
*** lmiccini has quit IRC16:17
*** florianf has quit IRC16:17
openstackgerritCarlos Camacho proposed openstack/python-tripleoclient master: Add openstack undercloud restore  https://review.openstack.org/47102116:22
*** ffiore has joined #tripleo16:28
*** trown is now known as trown|lunch16:30
*** limao_ has quit IRC16:32
openstackgerritCyril Lopez proposed openstack/tripleo-docs master: Explain how to use firewall in tripleo  https://review.openstack.org/46625216:32
*** tesseract has quit IRC16:34
*** jpich has quit IRC16:36
*** jkilpatr has quit IRC16:36
*** milan has quit IRC16:36
*** lucasagomes is now known as lucas-afk16:38
*** bogdando has quit IRC16:39
*** milan has joined #tripleo16:40
*** jkilpatr has joined #tripleo16:40
*** ramishra has quit IRC16:41
*** dparkes has joined #tripleo16:42
ansiwenmandre: still around? this time it got deployed correctly (although the job is failing): http://logs.openstack.org/41/469541/3/check-tripleo/gate-tripleo-ci-centos-7-ovb-containers-oooq-nv/cdcb4e0/logs/oooq/overcloud-controller-0/var/log/containers/nova/nova-consoleauth.log.txt.gz16:45
*** ffiore has quit IRC16:45
ansiwenmandre: http://logs.openstack.org/41/469541/3/check-tripleo/gate-tripleo-ci-centos-7-ovb-containers-oooq-nv/cdcb4e0/logs/oooq/overcloud-controller-0/var/log/extra/docker/containers/nova_consoleauth/docker_info.log.txt.gz16:46
*** atheurer has quit IRC16:47
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates master: Containerize Tacker Services  https://review.openstack.org/46630616:48
*** flepied has quit IRC16:48
*** salmankhan has quit IRC16:49
*** itlinux has joined #tripleo16:50
*** ckyriakidou has quit IRC16:50
*** milan has quit IRC16:51
*** tvignaud has quit IRC16:53
openstackgerritAlan Bishop proposed openstack/tripleo-heat-templates master: WIP: Experiment upgrading cinder-volume under pacemaker  https://review.openstack.org/47144716:55
*** egonzalez has quit IRC16:55
mandreansiwen: great, +216:55
*** derekh has quit IRC16:56
ansiwenmandre: the problem seems to be a normal timeout, 145 minutes are just not enough, it's killed while validating16:57
ansiwenmandre: this ansible step takes almost an hour: http://logs.openstack.org/41/469541/3/check-tripleo/gate-tripleo-ci-centos-7-ovb-containers-oooq-nv/cdcb4e0/console.html#_2017-06-07_14_38_36_30019916:58
mandreansiwen: correct, by using the multinode CI we can speed things up a lot, problem is we don't yet have parity17:00
mandreansiwen: there is a lot of work going on on the containers CI, mostly by jistr17:01
ansiwenmandre: but don't we have to increase the timeout then for now?17:01
mandreansiwen: that's the time infra gives us to run out tests (we can't have more), we need to speed up our CI jobs17:02
*** sri_ has quit IRC17:03
ansiwenmandre: ok, so we can't make that job voting as EmilienM wants to, as long as we can't figure that out17:03
* EmilienM dancing17:05
ansiwenEmilienM: are you still around? so it's a simple timeout, could you remove the -1 then?17:05
ansiwenthe service is actually deployed17:05
EmilienMI'll look at it in a few. I'm afk now17:05
openstackgerritCarlos Camacho proposed openstack/python-tripleoclient master: Add openstack undercloud restore  https://review.openstack.org/47102117:05
*** tvignaud has joined #tripleo17:06
ansiwenEmilienM: afk, I see, you and your fancy speech-recognition IRC client... ;-)17:06
EmilienMYeah I'm on my phone17:06
openstackgerritCarlos Camacho proposed openstack/python-tripleoclient master: Add openstack undercloud restore  https://review.openstack.org/47102117:07
ansiwenEmilienM: ha, you are as crazy as me, I also have an IRC client on my phone... :-)17:08
*** cylopez has quit IRC17:08
EmilienMProbably my biggest mistake17:09
*** atheurer has joined #tripleo17:09
*** cylopez has joined #tripleo17:10
*** ooolpbot has joined #tripleo17:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION17:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168019517:10
openstackLaunchpad bug 1680195 in tripleo "Random ovb-ha ping test failures" [Critical,Triaged] - Assigned to John Trowbridge (trown)17:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169590417:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169609417:10
*** ooolpbot has quit IRC17:10
openstackLaunchpad bug 1695904 in tripleo "stable/ocata scenario002 broken due to volume went to status error" [Critical,Triaged]17:10
openstackLaunchpad bug 1696094 in tripleo "CI: ovb-ha promotion job fails with 504 gateway timeout, neutron-server create-subnet timing out" [Critical,Triaged]17:10
ansiwenmandre: same here, btw. https://review.openstack.org/#/c/469486/17:10
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates master: DO NOT MERGE Testing containerized tacker service  https://review.openstack.org/47164617:10
*** gbarros has quit IRC17:12
openstackgerritSven Anderson proposed openstack/tripleo-heat-templates master: DONT MERGE - Test for Ib96040c2e27ad76b1fa6ecb9468bb9d97b3c4518  https://review.openstack.org/46920117:13
openstackgerritSven Anderson proposed openstack/tripleo-heat-templates master: Add Ec2Api to container based deployment  https://review.openstack.org/46920017:13
tdasilvatrown|lunch: just wanted to give you a feedback that removing the --requirements flag worked for me, thanks!17:14
*** agopi has quit IRC17:16
owalshmwhahaha: https://review.openstack.org/471319 ... this is essentially just a workaround until I can containerize polkitd17:16
owalshmwhahaha: trying to land a big puppet-tripleo refactor to get live-migraiton working, so ^^^ can wait till pike-317:16
*** shardy has quit IRC17:18
mwhahahaowalsh: ok just wondering if we have to have that. Let me know because the code isn't terrible if it's a short term fix but I just want to make sure we're not carrying the tech debt longer than we have to17:20
*** fragatina has joined #tripleo17:20
owalshmwhahaha: alternative is to override the kolla nova uid so that it matches what we use on the rpm17:20
owalshmwhahaha: but dprince and I agreed on the polkit/puppet-tripleo option17:20
owalshmwhahaha: ultimately we are dependant on the host polkitd, so I think it's a must fix for pike-317:21
*** jpena is now known as jpena|off17:21
*** itlinux has quit IRC17:24
openstackgerritMerged openstack/tripleo-quickstart master: Container featuresets for scenarios 001-004  https://review.openstack.org/46669617:27
*** agopi has joined #tripleo17:28
openstackgerritMerged openstack/tripleo-quickstart master: Container upgrade featuresets for scenarios 001-004  https://review.openstack.org/46594717:31
*** rcernin has joined #tripleo17:31
openstackgerritMerged openstack/tripleo-heat-templates master: Convert puppet and docker steps to ansible  https://review.openstack.org/46221117:31
openstackgerritMerged openstack/tripleo-common master: Use the mistral-lib actions  https://review.openstack.org/47126817:31
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates master: Containerize Congress API service  https://review.openstack.org/46573417:32
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates master: DO NOT MERGE Testing containerized congress service  https://review.openstack.org/47167417:32
mwhahahaowalsh: right but can we just deliver that in a different way than via puppet (ie package)?17:34
mwhahahaowalsh: because what is there isn't really something that needs puppet17:34
*** fragatina has quit IRC17:35
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates master: Containerize Congress API service  https://review.openstack.org/46573417:37
*** trown|lunch is now known as trown17:37
trowntdasilva: nice glad to hear it17:37
EmilienMansiwen: back on computer17:46
EmilienMansiwen: recheck17:46
EmilienMansiwen: if ci pass I'll approve17:46
EmilienMansiwen: thanks for your patience17:46
*** dtantsur is now known as dtantsur|afk17:50
*** agopi has quit IRC17:51
*** itlinux has joined #tripleo17:54
openstackgerritMerged openstack/tripleo-heat-templates master: Ability to enable/disable debug mode per OpenStack service  https://review.openstack.org/46769817:56
openstackgerritMerged openstack/tripleo-quickstart master: Fix repo mirror logic for tripleo-ci/master-undercloud-ocata-overcloud  https://review.openstack.org/47177417:56
openstackgerritMerged openstack/puppet-tripleo stable/newton: Restrict nova migration ssh tunnel  https://review.openstack.org/47143317:57
openstackgerritMerged openstack/tripleo-quickstart-extras master: Send mail tool  https://review.openstack.org/42334017:57
*** zoli|wfh is now known as zoli|gone17:57
openstackgerritMerged openstack/tripleo-heat-templates master: Ensure /etc/ssh/ssh_known_hosts exist in docker config-data.  https://review.openstack.org/47147517:57
openstackgerritMerged openstack/tripleo-heat-templates master: Map /etc/ssh/ssh_known_hosts to all containers  https://review.openstack.org/46846117:57
*** gkadam has quit IRC17:59
openstackgerritBob Fournier proposed openstack/tripleo-heat-templates master: Add pre-selected Control Plane IP support.  https://review.openstack.org/41327818:03
*** lblanchard has joined #tripleo18:03
*** lblanchard has quit IRC18:04
*** agopi has joined #tripleo18:04
*** lblanchard has joined #tripleo18:04
*** itlinux has quit IRC18:05
*** sid2 has quit IRC18:06
openstackgerritJames Slagle proposed openstack/tripleo-heat-templates master: Use Deployment actions for blacklist  https://review.openstack.org/47187618:06
*** gbarros has joined #tripleo18:09
openstackgerritwes hayutin proposed openstack/tripleo-quickstart master: WIP: only updates local_workspace if checkouts exist  https://review.openstack.org/47187818:09
*** ooolpbot has joined #tripleo18:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION18:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168019518:10
openstackLaunchpad bug 1680195 in tripleo "Random ovb-ha ping test failures" [Critical,Triaged] - Assigned to John Trowbridge (trown)18:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169590418:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169609418:10
*** ooolpbot has quit IRC18:10
openstackLaunchpad bug 1695904 in tripleo "stable/ocata scenario002 broken due to volume went to status error" [Critical,Triaged]18:10
openstackLaunchpad bug 1696094 in tripleo "CI: ovb-ha promotion job fails with 504 gateway timeout, neutron-server create-subnet timing out" [Critical,Triaged]18:10
*** salmankhan has joined #tripleo18:11
*** itlinux has joined #tripleo18:11
openstackgerritMerged openstack/tripleo-common master: Do not use multiple overrides with the same name  https://review.openstack.org/47178118:14
*** iranzo has joined #tripleo18:14
*** iranzo has joined #tripleo18:14
openstackgerritwes hayutin proposed openstack/tripleo-quickstart master: only updates local_workspace if checkouts exist  https://review.openstack.org/47187818:15
*** salmankhan has quit IRC18:15
*** lblanchard has quit IRC18:16
*** lblanchard has joined #tripleo18:16
*** lblanchard1 has joined #tripleo18:16
*** lblanchard has quit IRC18:16
*** lblanchard has joined #tripleo18:18
*** lblanchard1 has quit IRC18:18
*** pkovar has quit IRC18:19
openstackgerritJon Bernard proposed openstack/tripleo-heat-templates master: Move iscsid to a container  https://review.openstack.org/46253818:24
openstackgerritwes hayutin proposed openstack/tripleo-quickstart master: only updates local_workspace if checkouts exist  https://review.openstack.org/47187818:24
*** mdnadeem has quit IRC18:25
*** pchavva has quit IRC18:29
*** jaganathan has quit IRC18:29
*** pchavva has joined #tripleo18:33
openstackgerritwes hayutin proposed openstack/tripleo-quickstart master: only updates local_workspace if checkouts exist  https://review.openstack.org/47187818:38
*** trown is now known as trown|mtg18:41
openstackgerritwes hayutin proposed openstack/tripleo-quickstart master: only updates local_workspace if checkouts exist  https://review.openstack.org/47187818:41
*** gbarros has quit IRC18:44
*** gbarros has joined #tripleo18:44
openstackgerritJohn Fulton proposed openstack/tripleo-common master: Add Mistral workbook to call ceph-ansible to install Ceph  https://review.openstack.org/46964418:46
openstackgerritJohn Fulton proposed openstack/tripleo-common master: Add Mistral workbook to call ceph-ansible to install Ceph  https://review.openstack.org/46964418:48
openstackgerritwes hayutin proposed openstack/tripleo-quickstart master: WIP, adjust ansible ssh settings for performance  https://review.openstack.org/47188618:48
*** amoralej is now known as amoralej|off18:54
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates master: Add support to configure Num of Storage sacks  https://review.openstack.org/47133418:54
openstackgerritCarlos Camacho proposed openstack/python-tripleoclient master: Add openstack undercloud restore  https://review.openstack.org/47102118:54
openstackgerritMerged openstack/tripleo-common stable/newton: Add MigrationSshKey to generated passwords  https://review.openstack.org/46640218:57
openstackgerritMerged openstack/tripleo-puppet-elements stable/newton: Include openstack-nova-migration in the overcloud images  https://review.openstack.org/46640518:58
openstackgerritwes hayutin proposed openstack/tripleo-quickstart master: WIP, adjust ansible ssh settings for performance  https://review.openstack.org/47188618:58
*** itlinux has quit IRC19:00
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates master: Fix the disable expirer to remove crontab  https://review.openstack.org/47035719:00
*** itlinux has joined #tripleo19:00
*** gbarros has quit IRC19:00
*** gbarros has joined #tripleo19:01
openstackgerritCarlos Camacho proposed openstack/python-tripleoclient master: Add openstack undercloud restore  https://review.openstack.org/47102119:02
openstackgerritwes hayutin proposed openstack/tripleo-quickstart master: only updates local_workspace if checkouts exist  https://review.openstack.org/47187819:02
openstackgerritMerged openstack/tripleo-heat-templates stable/newton: Add migration SSH tunneling support  https://review.openstack.org/47147819:04
openstackgerritMerged openstack/tripleo-heat-templates stable/newton: Restrict nova migration ssh tunnel  https://review.openstack.org/47143519:04
openstackgerritwes hayutin proposed openstack/tripleo-quickstart master: only updates local_workspace if checkouts exist  https://review.openstack.org/47187819:05
openstackgerritJon Bernard proposed openstack/tripleo-common master: Add missing cinder container images  https://review.openstack.org/46940119:06
*** lblanchard has quit IRC19:09
*** ebarrera has joined #tripleo19:09
*** ooolpbot has joined #tripleo19:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION19:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168019519:10
openstackLaunchpad bug 1680195 in tripleo "Random ovb-ha ping test failures" [Critical,Triaged] - Assigned to John Trowbridge (trown)19:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169590419:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169609419:10
*** ooolpbot has quit IRC19:10
openstackLaunchpad bug 1695904 in tripleo "stable/ocata scenario002 broken due to volume went to status error" [Critical,Triaged]19:10
openstackLaunchpad bug 1696094 in tripleo "CI: ovb-ha promotion job fails with 504 gateway timeout, neutron-server create-subnet timing out" [Critical,Triaged]19:10
*** dsariel_ has quit IRC19:10
*** paramite has quit IRC19:11
*** rcernin has quit IRC19:17
*** dsneddon has quit IRC19:18
*** rcrit has joined #tripleo19:19
*** lblanchard has joined #tripleo19:21
*** dsneddon has joined #tripleo19:22
*** lblanchard has quit IRC19:22
*** lblanchard1 has joined #tripleo19:22
openstackgerritCarlos Camacho proposed openstack/python-tripleoclient master: Add openstack undercloud restore  https://review.openstack.org/47102119:24
rcritI've forgotten a lot of my ruby setup stuff, can someone help me run the puppet-tripleo tests?19:25
rcritI did a bundle install to get puppet-openstack_spec_helper but rspec can't find it19:26
* rcrit notes that it would be nice if the package had a testing section19:26
openstackgerritRyan Brady proposed openstack/tripleo-common master: Adds ansible actions  https://review.openstack.org/47002119:27
openstackgerritLee Yarwood proposed openstack/tripleo-quickstart master: devmode: Add release switch  https://review.openstack.org/47189319:27
*** trown|mtg is now known as trown19:28
*** fultonj has quit IRC19:28
*** flepied has joined #tripleo19:30
*** fultonj has joined #tripleo19:35
*** rcernin has joined #tripleo19:35
*** panda|bbl is now known as panda19:37
*** ebarrera_ has joined #tripleo19:38
rookI haven't played with the container work, or kept up with it, but i am curious, if we have a worker count set to > 1 do we launch a container per worker, or a single container with multiple processes (i know that is a no-no in the container world, right?)19:42
*** trown is now known as trown|brb19:42
rookEmilienM ^19:42
*** gbarros has quit IRC19:42
bnemecrcrit: I use https://docs.openstack.org/developer/puppet-openstack-guide/testing.html19:43
mwhahaharcrit: bundle install; bundle exec rake spec19:43
rcritthanks, I'll take a look19:43
rcritmight be nice to mention it in the project code I guess19:43
*** flepied has quit IRC19:43
rcritI just want to contribute a test and can't get anything running at all19:44
mwhahaharcrit: to be fair, there doesn't seem to be an documentation on running any tests python or puppet :D19:47
rcritbut the python stuff I was able to discover pretty quicky, the ruby docs had me doing all sorts of stupid things19:50
*** iranzo has quit IRC19:50
EmilienMrook: puppet runs inside a container during the deployment, so facter will look for /proc/cpuinfo19:51
mwhahaharcrit: eh i've had equally infuriating experiances with python tests. but the puppet-openstack docs cover how to run the tests19:51
rcritI guess, I'm looking at puppet-tripleo so didn't think to look elsewhere for docs19:52
*** dsariel_ has joined #tripleo19:52
EmilienMrook: so if we mount /proc (which I think is the case), I guess facter will take the values from there and configure openstack services19:52
*** trown|brb is now known as trown19:55
*** jcoufal has quit IRC19:56
*** agopi has quit IRC19:59
*** agopi has joined #tripleo20:00
*** gbarros has joined #tripleo20:00
sshnaidmbnemec, around?20:02
sshnaidmbnemec, do you know what is min-ready parameter in cloud node definition? http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/nodepool.yaml#n27420:02
sshnaidmrlandy, ^^20:02
*** lblanchard1 has quit IRC20:03
rlandysshnaidm: panda put the answer on #oooq20:03
EmilienMyolanda: you still around?20:03
EmilienMyolanda: promotion jobs fail on http://logs.openstack.org/15/359215/107/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-ha/14b94c4/console.html#_2017-06-07_13_59_14_02124720:04
*** gbarros has quit IRC20:04
*** lblanchard has joined #tripleo20:05
openstackgerritwes hayutin proposed openstack/tripleo-quickstart-extras master: add steps to enable tripleo_ui  https://review.openstack.org/45906120:06
*** lblanchard has quit IRC20:06
*** rbowen has quit IRC20:06
*** rbowen has joined #tripleo20:06
-openstackstatus- NOTICE: The Gerrit service on review.openstack.org is being restarted now to clear some excessive connection counts while we debug the intermittent request failures reported over the past few minutes20:07
*** paramite has joined #tripleo20:08
pradkhuh whats with all the merge conflict failures  .. i guess related to the above message?20:10
*** ooolpbot has joined #tripleo20:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION20:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168019520:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169590420:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169609420:10
*** ooolpbot has quit IRC20:10
openstackLaunchpad bug 1680195 in tripleo "Random ovb-ha ping test failures" [Critical,Triaged] - Assigned to John Trowbridge (trown)20:10
openstackLaunchpad bug 1695904 in tripleo "stable/ocata scenario002 broken due to volume went to status error" [Critical,Triaged]20:10
openstackLaunchpad bug 1696094 in tripleo "CI: ovb-ha promotion job fails with 504 gateway timeout, neutron-server create-subnet timing out" [Critical,Triaged]20:10
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates master: Containerize Tacker Services  https://review.openstack.org/46630620:11
*** akrivoka has quit IRC20:12
*** jprovazn has quit IRC20:13
openstackgerritVictoria Martinez de la Cruz proposed openstack/tripleo-heat-templates master: Containerize Manila API service  https://review.openstack.org/44929220:14
trownEmilienM: do you know what the neutron fix is so I can try it?20:15
EmilienMit didn't help AFIK20:15
EmilienMbut let me find the link20:15
trownEmilienM: ah k... I tried earlier today and still got failure to ping in the pingtest (though stack deployed fine)20:16
EmilienMtrown: ihar can confirm but https://review.openstack.org/#/c/471446/ is the patch20:16
trownpingtest stack deployed fine that is20:16
EmilienMpingtest was fine?20:16
EmilienMwow20:16
EmilienMit should fail to create the network20:16
EmilienMneutron-server should fail20:16
trownhmm in my test the stack created fine, but ping failed20:17
EmilienMtrown: but could you talk to neutron?20:18
EmilienMlike creating net/subnet?20:18
*** cylopez has left #tripleo20:18
trownya, have to do that in order to create the stack... maybe my reproducer was not valid20:18
openstackgerritAlex Schultz proposed openstack/tripleo-heat-templates master: Add host logging for redis service template  https://review.openstack.org/47170920:19
openstackgerritAlex Schultz proposed openstack/tripleo-heat-templates master: Mount /var/run/libvirt on ceilo agent compute  https://review.openstack.org/46993520:19
openstackgerritAlex Schultz proposed openstack/tripleo-heat-templates master: Standardize example role definitions  https://review.openstack.org/44568720:20
openstackgerritAlex Schultz proposed openstack/tripleo-heat-templates master: Fix IronicInspectorAdmin to be https  https://review.openstack.org/46643220:20
openstackgerritAlex Schultz proposed openstack/tripleo-heat-templates stable/newton: Addition of firewall rules for Nuage  https://review.openstack.org/46228620:20
*** Goneri has quit IRC20:20
*** gbarros has joined #tripleo20:21
openstackgerritAlex Schultz proposed openstack/tripleo-heat-templates master: Dell SC: Add exclude_domain_ip option  https://review.openstack.org/45541320:21
*** flepied has joined #tripleo20:21
openstackgerritMerged openstack/tripleo-heat-templates master: Don't create networks if neutron is not enabled  https://review.openstack.org/47173620:22
rookEmilienM: so that means, each container will have multiple Eventlet/WSGI processes20:22
openstackgerritMerged openstack-infra/tripleo-ci master: Use bool with ansible boolean variables  https://review.openstack.org/46511120:22
trownEmilienM: ah I think the config I tried to use is not updating overcloud images... so indeed my reproducer is not valid20:22
rookhttps://review.openstack.org/#/c/471889/20:23
EmilienMrook: probably. Of course we might need to change that20:23
rooksorry wrong channel ^^20:23
rookEmilienM I think it is OK, but Container people might bark/throwup20:23
EmilienMtrown: now promotion jobs fail on packaging: http://logs.openstack.org/15/359215/107/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-ha/14b94c4/console.html#_2017-06-07_13_59_14_02124720:23
*** dsariel_ has quit IRC20:23
EmilienMrook: I don't think we want them to be sick20:24
EmilienMtrown: nevermind, they fixed it. see #rdo20:24
EmilienMtrown: (talking about the packaging thing)20:24
rookEmilienM: Roger - then we might need a way to launch N containers per N service workers.20:24
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci master: TEST: DONT RECHECK or REVIEW: periodic jobs  https://review.openstack.org/35921520:24
*** gbarros has quit IRC20:24
EmilienMrook: I agree20:24
rookEmilienM: looking forward to trying it!20:25
rookTrying Pike that is.20:25
EmilienMrook: keep in mind there is always room for improvment, specially in tripleo20:25
EmilienMrook: I would ask you to maybe file a bug and triage it, so container team can look at it20:25
EmilienMso we can track this discussion20:25
*** agopi has quit IRC20:29
EmilienMjrist: any blocker before I release tripleo-ui?20:33
EmilienMjrist: can you submit the patch in tripleo-ui that update reno / release bump?20:34
*** dougbtv_ has joined #tripleo20:34
EmilienMrook: can you ack and confirm you'll create a bug report? just making sure it happens before I leave computer20:35
*** liverpooler has quit IRC20:37
openstackgerritwes hayutin proposed openstack/tripleo-quickstart-extras master: add steps to enable tripleo_ui  https://review.openstack.org/45906120:37
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates master: DO NOT MERGE Testing containerized congress service  https://review.openstack.org/47167420:38
*** dougbtv has quit IRC20:38
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates stable/newton: Expose metric delay processing metric  https://review.openstack.org/46922620:39
*** agopi has joined #tripleo20:39
*** atoth has quit IRC20:44
*** lblanchard has joined #tripleo20:45
*** ansmith has quit IRC20:46
*** thrash is now known as thrash|bbl20:51
*** pchavva has quit IRC20:51
*** trown is now known as trown|outtypewww20:52
*** pcaruana has quit IRC20:53
*** pcaruana has joined #tripleo20:53
weshayrlandy, I suspect packaging21:02
rlandyweshay: yep - can't be ovb-specific at that point  - so we must see it elsewhere21:02
*** ebarrera_ has quit IRC21:06
*** morazi has quit IRC21:06
*** sid1 has joined #tripleo21:06
openstackgerritCarlos Camacho proposed openstack/python-tripleoclient master: Add openstack undercloud restore  https://review.openstack.org/47102121:06
*** lblanchard has quit IRC21:09
weshayrlandy, http://logstash.openstack.org/#dashboard/file/logstash.json?query=%5C%22%2Fetc%2Fgnocchi%2Fgnocchi.conf%20%20returned%201%20instead%20of%20one%20of%20%5B0%5D%5C%2221:09
*** ooolpbot has joined #tripleo21:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION21:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168019521:10
openstackLaunchpad bug 1680195 in tripleo "Random ovb-ha ping test failures" [Critical,Triaged] - Assigned to John Trowbridge (trown)21:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169590421:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169609421:10
*** ooolpbot has quit IRC21:10
openstackLaunchpad bug 1695904 in tripleo "stable/ocata scenario002 broken due to volume went to status error" [Critical,Triaged]21:10
openstackLaunchpad bug 1696094 in tripleo "CI: ovb-ha promotion job fails with 504 gateway timeout, neutron-server create-subnet timing out" [Critical,Triaged]21:10
weshaymeh.. bad url21:12
*** agopi has quit IRC21:15
*** pcaruana has quit IRC21:16
*** sid1 has quit IRC21:17
*** egafford has quit IRC21:18
*** abishop has quit IRC21:22
*** funzo_ has joined #tripleo21:22
*** fultonj has quit IRC21:22
*** fragatina has joined #tripleo21:22
*** spredzy_ has joined #tripleo21:23
*** bfournie1 has joined #tripleo21:23
*** Tyrantelf_ has joined #tripleo21:25
*** Hazelesque_ has joined #tripleo21:26
*** Baggypan1s12000 has joined #tripleo21:26
*** bfournie has quit IRC21:29
*** funzo has quit IRC21:29
*** spredzy has quit IRC21:29
*** Baggypants12000 has quit IRC21:29
*** Hazelesque has quit IRC21:29
*** Tyrantelf has quit IRC21:29
*** spredzy_ is now known as spredzy21:29
*** SlickNik- has quit IRC21:32
*** SlickNik has joined #tripleo21:34
*** fultonj has joined #tripleo21:34
*** SlickNik has left #tripleo21:35
*** SlickNik has joined #tripleo21:36
*** SlickNik has left #tripleo21:36
*** ebarrera has quit IRC21:39
*** ansmith has joined #tripleo21:40
*** radeks has quit IRC21:41
*** rcernin has quit IRC21:44
*** panda is now known as panda|off21:47
*** SlickNik has joined #tripleo21:48
*** flepied has quit IRC21:48
*** yamahata has joined #tripleo21:49
*** cdearborn has quit IRC21:50
*** dtrainor has quit IRC21:53
*** dtrainor has joined #tripleo21:53
openstackgerritwes hayutin proposed openstack/tripleo-quickstart master: DO NOT REVIEW test ovb-ha against RDO latest builds  https://review.openstack.org/47148521:59
openstackgerritwes hayutin proposed openstack/tripleo-quickstart master: DO NOT REVIEW test ovb-ha against RDO latest builds  https://review.openstack.org/47148522:02
openstackgerritSteve Baker proposed openstack/tripleo-common master: Make overcloud_containers.yaml template driven  https://review.openstack.org/47153722:05
*** ecerquei has quit IRC22:05
openstackgerritLuigi Toscano proposed openstack/tripleo-heat-templates master: Containerize Sahara  https://review.openstack.org/46268122:08
*** ooolpbot has joined #tripleo22:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION22:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168019522:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169590422:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169609422:10
*** ooolpbot has quit IRC22:10
openstackLaunchpad bug 1680195 in tripleo "Random ovb-ha ping test failures" [Critical,Triaged] - Assigned to John Trowbridge (trown)22:10
openstackLaunchpad bug 1695904 in tripleo "stable/ocata scenario002 broken due to volume went to status error" [Critical,Triaged]22:10
openstackLaunchpad bug 1696094 in tripleo "CI: ovb-ha promotion job fails with 504 gateway timeout, neutron-server create-subnet timing out" [Critical,Triaged]22:10
*** agopi has joined #tripleo22:15
*** tosky has quit IRC22:19
*** jmelvin has quit IRC22:20
*** pradk has quit IRC22:20
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates stable/ocata: Add ignore_projects to filter gnocchi events  https://review.openstack.org/46920222:25
*** sid1 has joined #tripleo22:26
*** itlinux has quit IRC22:31
*** chlong has quit IRC22:32
*** jkilpatr has quit IRC22:34
*** rlandy has quit IRC22:35
*** sid1 has quit IRC22:36
openstackgerritOliver Walsh proposed openstack/puppet-tripleo master: Refactor nova migration config into client & target profiles  https://review.openstack.org/47195022:41
*** itlinux has joined #tripleo22:41
*** sid1 has joined #tripleo22:42
*** dsneddon has quit IRC22:42
openstackgerritMerged openstack/tripleo-quickstart master: devmode: Add release switch  https://review.openstack.org/47189322:43
*** itlinux has quit IRC22:54
*** sid1 has quit IRC22:55
openstackgerritOliver Walsh proposed openstack/tripleo-heat-templates master: Run the nova-compute container as the nova user  https://review.openstack.org/46846622:55
*** sid1 has joined #tripleo22:56
openstackgerritwes hayutin proposed openstack/tripleo-quickstart master: DO NOT REVIEW test ovb-ha against RDO latest builds  https://review.openstack.org/47148523:05
openstackgerritOliver Walsh proposed openstack/puppet-tripleo master: Refactor nova migration config into client & target profiles  https://review.openstack.org/47195023:07
*** fragatina has quit IRC23:09
*** ooolpbot has joined #tripleo23:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION23:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168019523:10
openstackLaunchpad bug 1680195 in tripleo "Random ovb-ha ping test failures" [Critical,Triaged] - Assigned to John Trowbridge (trown)23:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169590423:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169609423:10
*** ooolpbot has quit IRC23:10
openstackLaunchpad bug 1695904 in tripleo "stable/ocata scenario002 broken due to volume went to status error" [Critical,Triaged]23:10
openstackLaunchpad bug 1696094 in tripleo "CI: ovb-ha promotion job fails with 504 gateway timeout, neutron-server create-subnet timing out" [Critical,Triaged]23:10
*** openstack has joined #tripleo23:13
*** sid1 has quit IRC23:17
openstackgerritSteve Baker proposed openstack/tripleo-common master: Make overcloud_containers.yaml template driven  https://review.openstack.org/47153723:18
*** sid1 has joined #tripleo23:18
openstackgerritOliver Walsh proposed openstack/tripleo-heat-templates master: Add support for nova live/cold-migration with containers  https://review.openstack.org/47195623:19
openstackgerritOliver Walsh proposed openstack/puppet-tripleo master: Refactor nova migration config into client & target profiles  https://review.openstack.org/47195023:23
*** sid1 has quit IRC23:24
*** thrash|bbl is now known as thrash23:27
*** pmannidi has joined #tripleo23:30
openstackgerritJohn Fulton proposed openstack/tripleo-common master: Add Mistral workbook to call ceph-ansible to install Ceph  https://review.openstack.org/46964423:40
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Use class as super() argument  https://review.openstack.org/47196123:44
openstackgerritOliver Walsh proposed openstack/tripleo-heat-templates master: Add support for nova live/cold-migration with containers  https://review.openstack.org/47195623:47
openstackgerritOliver Walsh proposed openstack/tripleo-heat-templates master: Add support for nova live/cold-migration with containers  https://review.openstack.org/47195623:50
openstackgerritOliver Walsh proposed openstack/tripleo-heat-templates master: Modify libvirtd container command line when TLS is enabled  https://review.openstack.org/47196323:57

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!