Tuesday, 2017-08-29

*** achadha has quit IRC00:00
*** leitan has joined #tripleo00:01
*** ooolpbot has joined #tripleo00:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION00:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168955600:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169140300:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295500:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171291000:10
openstackLaunchpad bug 1689556 in tripleo "CI: containers overcloud can't start vm when deployed with multinic network isolation" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)00:10
*** ooolpbot has quit IRC00:10
openstackLaunchpad bug 1691403 in tripleo "containerized overcloud - neutron-openvswitchagent fails to start" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)00:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel)00:10
openstackLaunchpad bug 1712910 in tripleo "oooq virt-customize repo setup logs are not saved " [Critical,Triaged] - Assigned to wes hayutin (weshayutin)00:10
*** rhallisey has quit IRC00:12
*** toure is now known as toure_biab00:21
*** garyk has joined #tripleo00:24
*** lblanchard has joined #tripleo00:41
*** limao has joined #tripleo00:43
*** ecerquei has joined #tripleo00:43
*** Hazelesque has quit IRC00:44
*** Hazelesque has joined #tripleo00:46
*** leitan has quit IRC00:52
*** psahoo has joined #tripleo00:53
*** ooolpbot has joined #tripleo01:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION01:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168955601:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169140301:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295501:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171291001:10
openstackLaunchpad bug 1689556 in tripleo "CI: containers overcloud can't start vm when deployed with multinic network isolation" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)01:10
*** ooolpbot has quit IRC01:10
openstackLaunchpad bug 1691403 in tripleo "containerized overcloud - neutron-openvswitchagent fails to start" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)01:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel)01:10
openstackLaunchpad bug 1712910 in tripleo "oooq virt-customize repo setup logs are not saved " [Critical,Triaged] - Assigned to wes hayutin (weshayutin)01:10
*** achadha has joined #tripleo01:11
*** ecerquei has quit IRC01:14
*** achadha has quit IRC01:15
openstackgerritMerged openstack/tripleo-common master: Add stunnel to redis image for tls  https://review.openstack.org/49651901:22
openstackgerritMerged openstack/tripleo-heat-templates master: manila: set "host" to "hostgroup"  https://review.openstack.org/49748201:23
openstackgerritMerged openstack/tripleo-heat-templates stable/pike: Enable ceilometer compute agent  https://review.openstack.org/49811901:24
openstackgerritMerged openstack/puppet-tripleo master: Update reno>=2.5.0  https://review.openstack.org/49571501:24
*** ebarrera has quit IRC01:33
openstackgerritMerged openstack/tripleo-quickstart master: Change Delorean repo link to consistent for single job pipelines  https://review.openstack.org/49798201:35
*** achadha has joined #tripleo01:45
*** achadha has quit IRC01:49
*** achadha has joined #tripleo02:02
*** yamahata has quit IRC02:08
*** ooolpbot has joined #tripleo02:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION02:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168955602:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169140302:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295502:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171291002:10
openstackLaunchpad bug 1689556 in tripleo "CI: containers overcloud can't start vm when deployed with multinic network isolation" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)02:10
*** ooolpbot has quit IRC02:10
openstackLaunchpad bug 1691403 in tripleo "containerized overcloud - neutron-openvswitchagent fails to start" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)02:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel)02:10
openstackLaunchpad bug 1712910 in tripleo "oooq virt-customize repo setup logs are not saved " [Critical,Triaged] - Assigned to wes hayutin (weshayutin)02:10
*** achadha_ has joined #tripleo02:12
*** achadha has quit IRC02:15
openstackgerritAlex Schultz proposed openstack/puppet-tripleo stable/pike: Enable config for docker daemon debug  https://review.openstack.org/49865302:20
*** gbarros has joined #tripleo02:30
*** gbarros has quit IRC02:34
*** achadha_ has quit IRC02:37
*** ramishra has joined #tripleo02:38
*** lblanchard has quit IRC02:43
*** tzumainn has quit IRC02:45
*** rlandy|bbl is now known as rlandy02:54
*** yamahata has joined #tripleo03:06
*** garyk has quit IRC03:09
*** ooolpbot has joined #tripleo03:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION03:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168955603:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169140303:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295503:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171291003:10
openstackLaunchpad bug 1689556 in tripleo "CI: containers overcloud can't start vm when deployed with multinic network isolation" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)03:10
*** ooolpbot has quit IRC03:10
openstackLaunchpad bug 1691403 in tripleo "containerized overcloud - neutron-openvswitchagent fails to start" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)03:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel)03:10
openstackLaunchpad bug 1712910 in tripleo "oooq virt-customize repo setup logs are not saved " [Critical,Triaged] - Assigned to wes hayutin (weshayutin)03:10
*** limao has quit IRC03:11
*** itlinux has joined #tripleo03:13
*** pradk has quit IRC03:13
openstackgerritEmilien Macchi proposed openstack/tripleo-common stable/pike: Add stunnel to redis image for tls  https://review.openstack.org/49866203:27
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates stable/pike: manila: set "host" to "hostgroup"  https://review.openstack.org/49866403:27
*** achadha has joined #tripleo03:27
*** achadha has quit IRC03:27
*** achadha has joined #tripleo03:28
*** achadha has quit IRC03:32
*** pradk has joined #tripleo03:33
*** gkadam_ has joined #tripleo03:41
*** jbcraig1 has quit IRC03:49
*** udesale has joined #tripleo03:50
openstackgerritwes hayutin proposed openstack/tripleo-heat-templates master: DNM, for testing purposes only  https://review.openstack.org/49867103:56
openstackgerritwes hayutin proposed openstack/tripleo-heat-templates master: DNM, for testing purposes only  https://review.openstack.org/49867103:59
openstackgerritSteve Baker proposed openstack/tripleo-common master: Upload remove default for pull_source  https://review.openstack.org/48983704:01
openstackgerritSteve Baker proposed openstack/tripleo-common master: Discover a versioned container image tag  https://review.openstack.org/49867804:01
EmilienMstevebaker: how things are going04:02
stevebakerEmilienM: hey, good04:02
*** links has joined #tripleo04:03
EmilienMstevebaker: do you need any help or review?04:04
stevebakerEmilienM: I'm about to propose a fix to this, its a new command though, so even though its a "feature" I still think its worth landing for rc2 https://bugs.launchpad.net/tripleo/+bug/170896704:06
openstackLaunchpad bug 1708967 in tripleo "prepare command needs to discover the versioned tag from the latest tag" [High,In progress] - Assigned to Steve Baker (steve-stevebaker)04:06
*** pdeore has joined #tripleo04:06
EmilienMstevebaker: I was more interested by the 2 current CI alerts that you're assigned of04:07
EmilienMhttps://bugs.launchpad.net/tripleo/+bug/168955604:07
openstackLaunchpad bug 1689556 in tripleo "CI: containers overcloud can't start vm when deployed with multinic network isolation" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)04:07
EmilienMhttps://bugs.launchpad.net/tripleo/+bug/169140304:07
openstackLaunchpad bug 1691403 in tripleo "containerized overcloud - neutron-openvswitchagent fails to start" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)04:07
stevebakerEmilienM: this change should fix both, just waiting to merge https://review.openstack.org/#/c/493695/04:07
EmilienMstevebaker: cool. We have https://bugs.launchpad.net/tripleo/+bug/1712910 which is blocking our gate right now... weshay is on it04:09
openstackLaunchpad bug 1712910 in tripleo "oooq virt-customize repo setup logs are not saved " [Critical,Triaged] - Assigned to wes hayutin (weshayutin)04:09
EmilienMstevebaker: please keep doing recheck, we have no fix now and the issue is random04:09
*** ooolpbot has joined #tripleo04:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION04:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168955604:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169140304:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295504:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171291004:10
*** ooolpbot has quit IRC04:10
openstackLaunchpad bug 1689556 in tripleo "CI: containers overcloud can't start vm when deployed with multinic network isolation" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)04:10
openstackLaunchpad bug 1691403 in tripleo "containerized overcloud - neutron-openvswitchagent fails to start" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)04:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel)04:10
openstackLaunchpad bug 1712910 in tripleo "oooq virt-customize repo setup logs are not saved " [Critical,Triaged] - Assigned to wes hayutin (weshayutin)04:10
stevebakerEmilienM: OK, you mean recheck my change until it lands?04:11
EmilienMstevebaker: yea04:11
stevebakeroki04:11
EmilienMand we all count on weshay to fix the world04:11
EmilienMweshay: no pressure04:11
openstackgerritwes hayutin proposed openstack/instack-undercloud master: DNM, test commit only  https://review.openstack.org/49868004:12
weshay:P04:12
stevebakerlooks like weshay is just committing random crap now 0_o04:13
weshaystevebaker, like that's a change in behavior ;)04:14
stevebaker:)04:15
stevebakerweshay: so the virt-customize fails randomly and we're not capturing enough yet to know why?04:15
weshaystevebaker, k.. waiting on the status page to look again04:16
stevebakerweshay: ok, i'll keep an eye out04:16
weshaydang.. overall we're at 91.4% pass rate EmilienM :)04:17
EmilienMstevebaker: that's exactly that04:17
EmilienMweshay: on which one04:17
weshayoverall04:17
EmilienMweshay: we had so much failures today04:17
EmilienMah04:17
EmilienMthat's good man04:17
weshaythe undercloud-oooq job is failing 30% of the time04:17
EmilienMthat's a lot04:17
weshayyes04:18
weshayEmilienM, thanks for doing the math for me04:18
EmilienMweshay: I'm working on running tempest on more jobs04:19
EmilienMweshay: good progress today04:19
*** itlinux has quit IRC04:19
weshayI saw04:19
EmilienMweshay: I hope before ptg we can run tempest on all scenarios (container + baremetal)04:19
weshayEmilienM, stevebaker it's not random04:21
EmilienMstevebaker: could you review weshay's work on https://review.openstack.org/#/c/498074/ please ?04:21
weshayit's only failing on instack-undercloud changes04:21
EmilienMweshay: ah I thought, sorry04:21
EmilienMweshay: good work btw on it ^04:22
stevebakerEmilienM: sure04:22
EmilienMthx04:22
weshaythanks for the reviews..04:22
*** ykarel has joined #tripleo04:23
weshayEmilienM, the dlrn issue only hit us once? in 30 days? is this correct04:26
weshayhttp://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22ERROR:dlrn:Received%20exception%20Error%20in%5C%22%20AND%20tags:%5C%22console%5C%22%20AND%20build_queue:%5C%22gate%5C%22%20AND%20voting:1&from=864000s04:26
EmilienMweshay: let me check04:27
EmilienMweshay: which launchpad is it?04:27
EmilienMhttps://bugs.launchpad.net/tripleo/+bug/1708832 ?04:28
openstackLaunchpad bug 1708832 in tripleo "DLRN build failures in gate" [High,In progress]04:28
EmilienMyeah04:28
weshayBug 1708832 - DLRN build failures in gate04:28
EmilienM0 fails in 24 hrs / 1 fails in 10 days04:28
EmilienMsounds like yes04:28
EmilienMweshay: can you update the LP please (assignment + link the patch)04:28
EmilienMso we can keep track of what who where04:28
EmilienMand thanks :)04:28
weshayhrm.. there are two bugs on it.. will I mess up the elestic recheck if I mark it a dupe of https://bugs.launchpad.net/tripleo/+bug/171313004:31
openstackLaunchpad bug 1713130 in tripleo "pip install DLRN don't git clone from github.com" [High,In progress] - Assigned to wes hayutin (weshayutin)04:31
EmilienMweshay: yeah use the ES one true04:31
EmilienMor good I mean04:32
weshayok.. I have to update the commit message then04:32
EmilienMweshay: no need04:33
openstackgerritSteve Baker proposed openstack/python-tripleoclient master: Command to discover the versioned tag from latest  https://review.openstack.org/49868304:33
EmilienMweshay: just link it in comment04:33
EmilienMso we can merge it04:33
weshaythat is done04:33
EmilienMok thx04:34
*** morazi has quit IRC04:37
openstackgerritSteve Baker proposed openstack/python-tripleoclient master: Command to discover the versioned tag from latest  https://review.openstack.org/49868304:38
*** achadha has joined #tripleo04:40
EmilienMstevebaker: if you can review https://review.openstack.org/#/c/498440/2 please04:43
*** achadha has quit IRC04:44
*** achadha has joined #tripleo04:44
openstackgerritwes hayutin proposed openstack-infra/tripleo-ci master: the kolla container playbook needs release variables  https://review.openstack.org/49868804:48
EmilienMscenario004 is not working in container because the neutron plugin is not part of the container04:51
EmilienMhttp://logs.openstack.org/75/498575/1/check/gate-tripleo-ci-centos-7-scenario004-multinode-oooq-container/ffd691c/logs/subnode-2/var/log/containers/neutron/neutron-server.log.txt.gz#_2017-08-28_21_13_54_32004:51
EmilienMstevebaker: can you help with this one today if you have time? Otherwise I'll look at it tomorrow04:51
EmilienMthat's part of this effort: https://review.openstack.org/#/c/498575/04:52
openstackgerritFlavio Percoco proposed openstack/tripleo-common master: Encode stdout/stderr to generate readable logs  https://review.openstack.org/49739504:55
EmilienMI reported the issue: https://bugs.launchpad.net/tripleo/+bug/171361204:56
openstackLaunchpad bug 1713612 in tripleo "When neutron is containerized, No 'neutron.service_plugins' driver found" [High,Triaged]04:56
*** shreshtha has joined #tripleo04:57
*** bcafarel has quit IRC04:57
*** skramaja has joined #tripleo04:57
*** jschlueter has quit IRC04:58
*** mburned has quit IRC04:58
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart master: fs016: run tempest like fs005  https://review.openstack.org/49856904:58
*** shreshtha has quit IRC04:58
*** pdeore has quit IRC04:58
*** bcafarel has joined #tripleo04:59
*** shreshtha has joined #tripleo04:59
*** limao has joined #tripleo04:59
*** achadha_ has joined #tripleo05:00
*** mburned has joined #tripleo05:02
*** jschlueter|znc has joined #tripleo05:02
*** achadha has quit IRC05:03
weshayso.. EmilienM does this need to come from a mirror? https://github.com/openstack/tripleo-quickstart-extras/blame/master/roles/build-images/defaults/main.yml#L705:05
* weshay looks at toci05:05
*** Vijayendra has joined #tripleo05:06
*** achadha_ has quit IRC05:07
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Drop step_config as top level docker requirement  https://review.openstack.org/44271605:07
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: docker: Add unit tests on service_name  https://review.openstack.org/44275505:07
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Add docker templates to configure Ironic inspector  https://review.openstack.org/45782205:07
EmilienMweshay: I'll look later05:08
EmilienMweshay: go to bed now, ttyl05:08
EmilienMjaosorior: https://review.openstack.org/#/c/498440/205:09
*** ooolpbot has joined #tripleo05:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION05:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168955605:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169140305:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295505:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171291005:10
openstackLaunchpad bug 1689556 in tripleo "CI: containers overcloud can't start vm when deployed with multinic network isolation" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)05:10
*** ooolpbot has quit IRC05:10
openstackLaunchpad bug 1691403 in tripleo "containerized overcloud - neutron-openvswitchagent fails to start" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)05:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel)05:10
openstackLaunchpad bug 1712910 in tripleo "oooq virt-customize repo setup logs are not saved " [Critical,Triaged] - Assigned to wes hayutin (weshayutin)05:10
jaosoriorEmilienM: sounds like a reasonable change. +205:10
jaosorior* looks05:10
*** dparkes has quit IRC05:11
*** achadha has joined #tripleo05:11
*** pdeore has joined #tripleo05:14
*** achadha has quit IRC05:15
*** dpawar has joined #tripleo05:23
*** ratailor has joined #tripleo05:24
*** masco has joined #tripleo05:28
openstackgerritFlavio Percoco proposed openstack/tripleo-heat-templates master: WIP: Deploy OpenShift using OOO on the overcloud  https://review.openstack.org/49447005:31
*** pgadiya has joined #tripleo05:36
*** janki has joined #tripleo05:37
*** daidv has quit IRC05:41
*** jfrancoa has joined #tripleo05:41
*** daidv has joined #tripleo05:42
*** mdnadeem has joined #tripleo05:44
*** iranzo has joined #tripleo05:46
*** garyk has joined #tripleo05:49
*** ykarel_ has joined #tripleo05:49
*** jfrancoa has quit IRC05:50
*** jfrancoa has joined #tripleo05:50
*** ykarel has quit IRC05:51
*** yprokule has joined #tripleo05:54
openstackgerritMerged openstack/diskimage-builder master: Use [[ for =~ matches  https://review.openstack.org/49833305:59
*** ykarel_ is now known as ykarel06:00
openstackgerritMerged openstack/tripleo-heat-templates master: Set docker-puppet --health-cmd = /bin/true  https://review.openstack.org/49813706:01
*** dparkes has joined #tripleo06:04
*** cschwede has joined #tripleo06:07
*** dsneddon has quit IRC06:09
*** ooolpbot has joined #tripleo06:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION06:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168955606:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169140306:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295506:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171291006:10
openstackLaunchpad bug 1689556 in tripleo "CI: containers overcloud can't start vm when deployed with multinic network isolation" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)06:10
*** ooolpbot has quit IRC06:10
openstackLaunchpad bug 1691403 in tripleo "containerized overcloud - neutron-openvswitchagent fails to start" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)06:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel)06:10
openstackLaunchpad bug 1712910 in tripleo "oooq virt-customize repo setup logs are not saved " [Critical,Triaged] - Assigned to wes hayutin (weshayutin)06:10
*** marios has joined #tripleo06:10
*** iranzo has quit IRC06:16
*** agurenko has joined #tripleo06:17
*** pgadiya has quit IRC06:18
*** iranzo has joined #tripleo06:20
*** rlandy has quit IRC06:27
*** pgadiya has joined #tripleo06:34
openstackgerritJose Luis Franco proposed openstack/tripleo-heat-templates master: Add yaml validation in docker upgrade_tasks.  https://review.openstack.org/49794006:35
openstackgerritJose Luis Franco proposed openstack/tripleo-heat-templates master: Add tags to baremetal cron removal tasks  https://review.openstack.org/49793606:35
*** ffiore has joined #tripleo06:39
*** pcaruana has joined #tripleo06:39
*** Vijayendra has quit IRC06:41
*** fzdarsky|afk has joined #tripleo06:41
*** Vijayendra has joined #tripleo06:42
*** jaganathan has joined #tripleo06:44
jaosorioradarazs: could you check this out https://review.openstack.org/#/c/496639/ ?06:48
*** pgadiya has quit IRC06:54
*** aditya_r has joined #tripleo06:58
*** fzdarsky|afk is now known as fzdarsky07:04
*** pgadiya has joined #tripleo07:06
*** ebarrera has joined #tripleo07:06
*** rcernin has joined #tripleo07:10
*** ooolpbot has joined #tripleo07:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION07:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168955607:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169140307:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295507:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171291007:10
openstackLaunchpad bug 1689556 in tripleo "CI: containers overcloud can't start vm when deployed with multinic network isolation" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)07:10
*** ooolpbot has quit IRC07:10
openstackLaunchpad bug 1691403 in tripleo "containerized overcloud - neutron-openvswitchagent fails to start" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)07:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel)07:10
openstackLaunchpad bug 1712910 in tripleo "oooq virt-customize repo setup logs are not saved " [Critical,Triaged] - Assigned to wes hayutin (weshayutin)07:10
*** jprovazn has joined #tripleo07:10
*** florianf has joined #tripleo07:13
*** aufi has joined #tripleo07:13
*** cylopez has joined #tripleo07:15
sshnaidm|afkmwhahaha, hi, can you please un-workflow this patch? https://review.openstack.org/#/c/498562 it requires a discussion..07:18
*** sshnaidm|afk is now known as sshnaidm07:19
thervejaosorior, Hi, you around? Wanted to chat about that swift ssl issue?07:21
*** bkopilov has quit IRC07:21
jaosoriortherve: I'm around07:22
jaosoriorwhat's up?07:22
thervejaosorior, So it's https://bugzilla.redhat.com/show_bug.cgi?id=148518907:23
openstackbugzilla.redhat.com bug 1485189 in rhosp-director "rhosp-director: HA overcloud +SSL+ceph fails when undercloud was also deployed with SSL: os-collect-config[3057]: requests.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)" [Unspecified,New] - Assigned to gfidente07:23
*** dpawar has quit IRC07:23
jaosoriortherve: well, regarding that issue. I did comment on the BZ, giving the CAMap option, which is what you would anyway have to do if you have an external swift endpoint (the overcloud has to trust it)07:23
jaosoriortherve: check comment #807:23
therveYou mentioned it should be using internal endpoints for that though07:23
therveI was wondering why you thought that07:23
jaosoriortherve: well, if it's swift deployed on the undercloud, it should probably be using the internal endpoint.07:24
thervejaosorior, But it's contacted by the overcloud07:24
jaosoriortherve: although.... now that I think about it, there's very little chance that the undercloud will be configured with network isolation. So really it's not that big of a deal.07:24
*** tesseract has joined #tripleo07:25
therveSo arguably you'd want the communication to be secured07:25
therveNot sure I get your comment about network isolation07:25
therveYou mean the IPs will be the same between internal and public?07:26
jaosoriortherve: they'll be in the same network.07:26
thervejaosorior, Can you configure the undercloud with network isolation?07:26
*** bkopilov has joined #tripleo07:26
jaosoriortherve: that's what I just mentioned, I don't think you can (and if you can I don't know how)07:27
therveBasically, I want to know if there is a code change to be made to use internal endpoint07:27
therveOr if the answer is to properly use CaMap for security07:27
jaosoriortherve: * CAMap07:27
*** jpena|off is now known as jpena07:28
*** dpawar has joined #tripleo07:28
jaosoriortherve: so... here's the situation. ideally we should be using SSL everywhere for the undercloud. we're not cause of some constraints (long story). as a fallback we rely on that network being private (which seems to be just fine for many deployers)07:29
*** anshul has joined #tripleo07:29
therveRight07:30
jaosoriortherve: due to the current situation, I don't have a very strong opinion on using public vs. internal endpoint there. I think using an internal endpoint will make things easier, since it is an internal service anyway (as far as I understood).07:30
therveWell07:30
jaosoriortherve: it's not?07:31
thervejaosorior, Critical information is circulating on that communication though. If you care about undercloud ssl, that's probably what you want to protect07:31
therveOtherwise not sure why you'd want undercloud ssl07:31
jaosoriortherve: for horizon access, from somewhere to the undercloud.07:31
therveOk, that's fair07:32
*** jpich has joined #tripleo07:32
therveI guess you have more pressing issues if someone can read data between your nodes and the undercloud07:32
*** dpawar has quit IRC07:32
*** ykarel is now known as ykarel|lunch07:33
jaosoriortherve: and we can't really enable TLS everywhere, cause that ends up breaking services like ironic... we need to modify the IPA images to trust the CA, and we can't cause then folks loose support07:34
therve Ah07:34
thervejaosorior, So ironic is already using the internal endpoints?07:34
jaosoriortherve: it is07:34
therveEven from remote nodes07:34
jaosoriortherve: well, for the overcloud deployment, yeah07:35
therveOK that's an interesting point07:35
jaosoriortherve: I came up with this some time ago https://review.openstack.org/#/c/358457/  but I could never get the time to move it forward, and apparently folks are OK with that.07:35
thervejaosorior, So let's assume it should be using internal endpoints here07:35
thervejaosorior, Should we use them all the time, or allow configuration?07:35
jaosoriortherve: we should allow configuration07:35
therveConsidering it can work if you setup it properly07:35
* therve nods07:36
jaosoriortherve: well, it's a matter of usability too07:36
*** egonzalez has joined #tripleo07:36
jaosoriortherve: if you decide to use a remote swift endpoint. (swift deployed outside of the undercloud... which someone mentioned you can do... though I don't really know how) then you do need to use the CAMap anyway, and use public endpoints07:36
*** jtomasek has quit IRC07:37
*** jtomasek_ has joined #tripleo07:37
therveLooking at the code I don't see how's that possible :)07:37
therveBut ok I'll look into doing that07:37
*** yamahata has quit IRC07:37
therveThanks for the input!07:37
jaosoriortherve: I thought that using the CAMap and making the undercloud's CA cert trusted by the overcloud would be the easier path so I proposed this for quickstart https://review.openstack.org/#/c/496639/07:38
jaosoriorbut as feedback I was told that would be quite a burden for deployers07:38
jaosoriorI don't really think it is, as long as we document it07:38
jaosoriorusing the CAMap, stuff works, and no code changes are needed for tripleo or the services :D07:39
therve:)07:39
jaosorioronly to quickstart and infrared, and the docs07:39
jaosoriorwhich is fine by me07:39
therveWell I'll let you push that path, because I think it's worth it too07:39
therveI'll see if I can make the endpoints configurable07:39
*** ratailor is now known as ratailor|Lunch07:39
jaosoriortherve: soo... if you have influence in this, lets use the CAMap. we use TLS there; it's something that should be done for remote Swift deployments anyway, and it seems like the easier solution07:40
thervejaosorior, Not sure about influence, but I have inertia if I don't write the code :D07:41
jaosoriortherve: are you acquainted with infrared? there could be code written for that to allow using CAMap :D07:41
therveHum no I don't know about it at all07:41
jaosoriortherve: also, you said you looked at the code and you didn't see how that was possible. How so?07:42
jaosoriorregarding the IPA patch.07:42
thervejaosorior, I don't see how is it possible to use an external swift07:42
jaosoriortherve: aaaah07:43
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart master: WIP create default settings for environment types  https://review.openstack.org/49540707:44
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart master: Change devmode to deploy containerized services by default  https://review.openstack.org/47241207:45
*** oidgar has joined #tripleo07:46
*** mcornea has joined #tripleo07:47
jaosoriortherve: commented on the patch.07:51
jaosorior* the BZ, not the patch07:51
openstackgerritJose Luis Franco proposed openstack/tripleo-heat-templates master: Add yaml validation in docker upgrade_tasks.  https://review.openstack.org/49794007:52
jaosoriortherve: is zaqar using mongodb by default?07:52
thervejaosorior, Depends where :)07:52
therveYes in the overcloud, no in the undercloud07:52
jaosoriortherve: did you see the mail of folks suggesting to get rid of mongodb?07:53
thervejaosorior, Ah thanks. I was in that thread but missed the last update07:54
*** dpawar has joined #tripleo07:56
matbudciabrin: hello, can you take a look at https://review.openstack.org/#/c/49849907:57
matbuwhen you have a moment07:57
*** paramite has joined #tripleo08:02
*** ffiore has quit IRC08:04
*** suuuper has joined #tripleo08:07
*** ooolpbot has joined #tripleo08:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION08:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168955608:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169140308:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295508:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171291008:10
openstackLaunchpad bug 1689556 in tripleo "CI: containers overcloud can't start vm when deployed with multinic network isolation" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)08:10
*** ooolpbot has quit IRC08:10
openstackLaunchpad bug 1691403 in tripleo "containerized overcloud - neutron-openvswitchagent fails to start" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)08:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel)08:10
openstackLaunchpad bug 1712910 in tripleo "oooq virt-customize repo setup logs are not saved " [Critical,Triaged] - Assigned to wes hayutin (weshayutin)08:10
*** ffiore has joined #tripleo08:10
*** dsariel has joined #tripleo08:14
matbuone easy review plz https://review.openstack.org/#/c/480461/08:20
matbuand another one: https://review.openstack.org/#/c/479212/08:21
*** milan has joined #tripleo08:26
*** stendulker has joined #tripleo08:28
*** ffiore has quit IRC08:28
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Use integers for rabbitmq ports  https://review.openstack.org/49871108:28
openstackgerritMerged openstack/tripleo-docs master: Typo: removed backslash which ended up in the generated HTML version  https://review.openstack.org/49793908:28
*** bogdando has joined #tripleo08:29
*** ratailor|Lunch is now known as ratailor08:29
honzamwhahaha: apetrich: d0ugal: i just got back from vacation, is there anything i can do to help move this patch across the finish line?  https://review.openstack.org/#/c/469608/  I feel rather unqualified to resolve the CI issues but I'm happy to learn.08:30
sshnaidmwho is Steve Baker on IRC?08:31
apetrichhonza, I'm on it. The CI is not logging the file that we need to understand the problem, I'm trying to reproduce the CI run locally so I can look at that file08:31
apetrichd0ugal, ^^08:31
honzaapetrich: perfect, thank you08:32
mandresshnaidm: try stevebaker :)08:32
sshnaidmstevebaker, hi08:32
sshnaidmmandre, thanks :)08:32
mandresshnaidm: he's in NZ, so his night now08:33
sshnaidmmandre, oh, ok.. 8 PM08:34
mandresshnaidm: anything I can help you with maybe?08:35
sshnaidmmandre, no, just wanted him to remove +w from some patch..08:35
sshnaidmnevermind08:35
openstackgerritLukas Bezdicka proposed openstack/python-tripleoclient master: Add setup-heat-outputs option to get heat stack outputs  https://review.openstack.org/49565808:35
mandresshnaidm: iiuc if someone else -2 or -w over it, that won't merge08:36
sshnaidmmandre, good to know, thanks08:37
d0ugalapetrich: what file isn't logged?08:38
openstackgerritSagi Shnaidman proposed openstack/tripleo-heat-templates master: DONT REVIEW: test build patch  https://review.openstack.org/49871208:39
apetrichd0ugal, so it fails on a disk thingmagick manipulation that is logged as  > /var/lib/oooq-images/master/repo_setup.sh.log and var/lib is not on the logs08:40
apetrichd0ugal, it is set to collect but didn't08:40
skramajahave a question on container image building.. do we use directly kolla-build command to build images [or] invoked via "openstack overcloud container image build " command?08:42
*** bogdando has quit IRC08:48
d0ugalapetrich: oh, weird08:48
*** gfidente has joined #tripleo08:49
*** gfidente has quit IRC08:49
*** gfidente has joined #tripleo08:49
*** bogdando has joined #tripleo08:53
*** tosky has joined #tripleo08:55
mrungeskramaja: the guide says openstack overcloud container image build08:56
mrungeskramaja: but internally, it calls kolla-build08:56
mrungeas far as I found out08:57
*** ykarel|lunch is now known as ykarel08:57
skramajathanks mrunge i am trying to find how the templates-override options is provided via "overcloud container image build" as i am unable to find the related option..08:58
mrungeskramaja: good Q.08:58
*** lucas-afk is now known as lucasagomes08:59
mrungeI've always used kolla-build, and you can edit /etc/kolla/kolla-build.conf08:59
skramajamrunge: the doc here https://docs.openstack.org/tripleo-docs/latest/install/containers_deployment/overcloud.html, referes to kolla-build command..08:59
mrungeright08:59
mrungeskramaja: there is more: https://docs.openstack.org/tripleo-docs/latest/install/containers_deployment/overcloud.html09:01
mrungemeh, same link09:01
*** ratailor_ has joined #tripleo09:01
*** ratailor_ has quit IRC09:01
*** ratailor_ has joined #tripleo09:02
*** hewbrocca_afk is now known as hewbrocca09:04
*** ratailor has quit IRC09:04
*** garyk1 has joined #tripleo09:07
skramajamrunge: hmm.. an option to do it, from dprince helper scripts - https://github.com/dprince/undercloud_containers/blob/master/build_kolla.sh#L2309:10
*** ooolpbot has joined #tripleo09:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION09:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168955609:10
openstackLaunchpad bug 1689556 in tripleo "CI: containers overcloud can't start vm when deployed with multinic network isolation" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)09:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169140309:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295509:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171291009:10
*** ooolpbot has quit IRC09:10
openstackLaunchpad bug 1691403 in tripleo "containerized overcloud - neutron-openvswitchagent fails to start" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)09:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel)09:10
openstackLaunchpad bug 1712910 in tripleo "oooq virt-customize repo setup logs are not saved " [Critical,Triaged] - Assigned to wes hayutin (weshayutin)09:10
*** limao has quit IRC09:11
*** aditya_r has quit IRC09:16
*** aditya_r has joined #tripleo09:16
*** sshnaidm is now known as sshnaidm|afk09:18
*** fzdarsky has quit IRC09:20
*** ratailor has joined #tripleo09:22
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo master: Enable TLS for rabbitmq's replication traffic  https://review.openstack.org/49832409:23
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Rabbitmq: Enable Erlang distribution TLS and add CA file  https://review.openstack.org/49832509:23
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Use FQDN for nodename in rabbitmq configuration  https://review.openstack.org/49872609:25
*** ratailor_ has quit IRC09:26
adarazsjaosorior: https://review.openstack.org/496639 looks good, but gfidente has a question on it that's not answered yet. I can merge it otherwise.09:28
mrungeskramaja: so, he's basically providing a kolla-build.config09:28
skramajamrunge: yes. and the conf file is provided as input to the "overcloud container image build" command09:29
mrungeexactly09:29
*** fzdarsky has joined #tripleo09:30
*** ffiore has joined #tripleo09:40
*** pkovar has joined #tripleo09:43
*** psahoo has quit IRC09:47
adarazssshnaidm|afk: can you please take a look at this change: https://review.openstack.org/498688 -- we should have it in ASAP.09:50
gfidenteleseb with jprovazn there is an env where the osd bootstrap seems to fail09:51
gfidenteapparently the bootstrap osd keyring is missing in the osd container09:52
gfidentebut I couldn't really figure out the sequence of operations from entrypoint.sh down to the command which failed09:52
gfidenteyou have time to check with him?09:52
lesebgfidente yes09:53
jprovaznleseb: cool, let me send you details09:54
*** udesale has quit IRC10:02
bogdandoweshay: hi. Could you please re-vote https://review.openstack.org/#/c/492178/ ?10:03
*** rook is now known as rook|pto10:06
*** ratailor has quit IRC10:07
*** ratailor has joined #tripleo10:08
jprovazngfidente: ansible tasks were logged in mistral executor.log on UC?10:09
gfidentehey sorry guys10:09
*** ooolpbot has joined #tripleo10:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION10:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168955610:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169140310:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295510:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171291010:10
openstackLaunchpad bug 1689556 in tripleo "CI: containers overcloud can't start vm when deployed with multinic network isolation" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)10:10
*** ooolpbot has quit IRC10:10
openstackLaunchpad bug 1691403 in tripleo "containerized overcloud - neutron-openvswitchagent fails to start" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)10:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel)10:10
openstackLaunchpad bug 1712910 in tripleo "oooq virt-customize repo setup logs are not saved " [Critical,Triaged] - Assigned to wes hayutin (weshayutin)10:10
gfidenteleseb jprovazn read the tmux chat10:15
lesebgfidente looks like the problem is that the disk had partitions already10:15
lesebcpeh partitions10:15
leseb*ceph10:15
gfidenteso yeah oooq reuses the same disks which are pre-partitioned and potentially with ceph if there has been a previous attempt10:15
gfidenteright ironic can do the cleanup it'st just not enabled by default10:15
gfidentecause it's a bit time consuming10:15
gfidenteso oooq does not10:16
gfidentelet me find the setting10:16
lesebsure10:16
jprovaznshould ceph-ansible repartition the disk (at least if an option is used)?10:17
gfidentejprovazn no I think that is risky10:17
jprovazngfidente: if there is an option "--i-really-mean-it"?10:17
gfidenteyeah but then if you pass a parameter like that from tht10:17
sshnaidm|afkadarazs, looking..10:17
gfidenteit's stuck in heat for every stack update10:17
gfidenteI think cleanup on deletion is good approach10:18
lesebgfidente +110:18
jprovazngfidente: well, many times I re-run oooq with an existing OC deployed10:18
jprovazngfidente: so I'm afraid that cleanup in THT will not help10:18
gfidentein ironic?10:19
jprovazngfidente: it depends when deletion of ironic nodes is done?10:19
gfidentewhen the node is released10:19
*** sshnaidm|afk is now known as sshnaidm10:19
sshnaidmadarazs, a few questions about https://review.openstack.org/#/c/498688/1/scripts/shell-builders/rdo-infra-config/build-containers-images.sh10:20
jprovazngfidente: if it covers scenario when I have fully deployed oooq and then jsut run oooq again with -T all, then it's fine10:20
jprovazngfidente: aha, ok10:20
sshnaidmadarazs, why do we set trunk_repository and then unconditionally override it?10:20
jprovaznback in 3010:21
sshnaidmadarazs, that patch means we never run kolla for master, is it intentionally?10:21
gfidentejprovazn automated_clean=true10:21
gfidentejprovazn in [conductor] should do10:21
gfidentejprovazn https://docs.openstack.org/ironic/latest/admin/cleaning.html10:21
openstackgerritJanki Chhatbar proposed openstack/python-tripleoclient master: Add --neutron-drvier argument to prepare command  https://review.openstack.org/49874310:23
*** dtantsur|afk is now known as dtantsur10:23
*** garyk has quit IRC10:24
jankimandre, ^10:24
adarazssshnaidm: hmm, I saw $STABLE_RELEASE defaulting to master if not set, but maybe that's not going to happen in that script.10:26
*** aditya_r has quit IRC10:27
mandrejanki: this is great but we can even automate it a bit more by inspecting the list of services that are enabled, so you don't have to introduce a new --neutron-driver option10:28
mandrejanki: does it make sense?10:28
sshnaidmadarazs, afaik it's never master, because it's *stable* branch10:29
adarazssshnaidm: I'll rework this.10:29
jankimandre, I am submitting corresponding ooo-common patch. lets discuss there? Ya. it makes sense but I am not sure how to achieve that10:29
openstackgerritOpenStack Proposal Bot proposed openstack/tripleo-ui stable/pike: Imported Translations from Zanata  https://review.openstack.org/49874710:29
sshnaidmadarazs, we use in scripts ${STABLE_BRANCH:-'master'}10:29
adarazssshnaidm: yeah, I saw some places where it's defaulted to master, but I don't think it propagates here from any of those scripts.10:30
*** skramaja has quit IRC10:32
openstackgerritJanki Chhatbar proposed openstack/tripleo-common master: Add ODL container images to TripleO-common.  https://review.openstack.org/49838310:32
jankimandre, ^10:32
adarazssshnaidm: thanks for the review.10:32
mandrejanki: ack, all the tripleo-common cares about is that the neutron_driver variable is set, let's discuss the best way to set it in your python-tripleoclient patch10:32
jankimandre, sure10:33
numansjanki, mandre can i follow this patch  https://review.openstack.org/498383 and do the same for ovn ?10:38
jankinumans, lets decide on how to pass neutron-driver to ooo-common first. currently the patch is half done. You can do that now or can do it later once the logic is fully implemented.10:40
openstackgerritMichele Baldessari proposed openstack/tripleo-common master: Add clustercheck healthcheck  https://review.openstack.org/49746810:40
*** akrivoka has joined #tripleo10:45
*** jkilpatr has quit IRC10:46
*** ykarel is now known as ykarel|afk10:46
*** akrivoka has quit IRC10:46
openstackgerritAttila Darazs proposed openstack-infra/tripleo-ci master: Use release specific repos during containers build  https://review.openstack.org/49868810:47
adarazssshnaidm: ^ can you check it out again?10:47
sshnaidmadarazs, it's playbook, you can't use env vars like  "$RELEASE", you need the lookup('env', 'RELEASE') etc10:48
adarazssshnaidm: it's being cat-ed and the variable is defined. let me fetch the config for it.10:49
sshnaidmadarazs, oh, right, sorry10:49
*** akrivoka has joined #tripleo10:50
adarazssshnaidm: might be more elegant with a lookup though, but it shouldn't make a practical difference here :)10:51
sshnaidmadarazs, do you know where is $RELEASE defined?10:51
sshnaidmadarazs, yeah, you're right10:51
adarazssshnaidm: yes, fetching that part of the config.10:51
adarazsjust gitweb is slow.10:51
sshnaidmadarazs, I use usually github for it, it highlights the lines10:52
mandrejanki: I pointed to code example in https://review.openstack.org/#/c/498743, I think you can easily reuse the service_filter variable10:52
adarazssshnaidm: https://review.rdoproject.org/r/gitweb?p=config.git;a=blob;f=jobs/tripleo-upstream.yml;h=e883555e504d88c82ad6649f3fed4de32a1e165e;hb=HEAD#l7610:52
jankimandre, checking the commit. let me try doing it this way.10:52
adarazssshnaidm: yeah, but do we have this rdoproject config repo anywhere mirrored to github? :)10:52
sshnaidmadarazs, should be..10:53
*** fzdarsky is now known as fzdarsky|lunch10:54
sshnaidmadarazs, is it only for me - when I open any link the rdo.review page it logs me in and redirect to "my changes", without going to page which I wanted to open10:55
sshnaidmadarazs, and without any chance to go to the original page..10:55
adarazssshnaidm: yeah, I just realized I have the same problem. I bookmarked the gitweb page of this repo and it doesn't work.10:56
mandrenumans, janki: maybe we can add the ovn neutron image in https://review.openstack.org/#/c/498383/ as well since we'll have https://review.openstack.org/#/c/498743/ set neutron_driver to ovn when appropriate10:56
sshnaidmadarazs, yeah, I need to open link twice there..10:56
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Use FQDN for nodename in rabbitmq configuration  https://review.openstack.org/49872610:56
adarazssshnaidm: second time it works, yep. not the best UX :)10:56
adarazssshnaidm: that top bar iframe messes with things.10:57
jprovazngfidente: I wonder which ironic clean step  should be used for cleaning up non-root disk partitions in case of manual cleanup10:58
jprovazndtantsur: ^ maybe you might know ?10:58
sshnaidmadarazs, +210:59
dtantsurjprovazn: ironic cleaning cleans all disks10:59
dtantsurwell, all it can find10:59
sshnaidmadarazs, we push containers to trunk.registry.rdoproject.org, right?10:59
adarazssshnaidm: yeah. I think we will need to push them to docker.io as well, but for now there.10:59
sshnaidmok10:59
adarazssshnaidm: also it seems to be working for the master jobs.11:00
adarazssshnaidm: crap, now that I pushed that change I can't +2 it I guess.11:00
sshnaidmadarazs, you can merge it actually, ci jobs don't make sense there11:00
adarazssshnaidm: I'm not talking about CI jobs, just another pair of eyes. but yeah, I think we could get it going. not much to lose but time. :)11:01
dtantsurjprovazn: http://tripleo.org/install/mistral-api/mistral-api.html#cleaning-nodes11:02
*** jkilpatr has joined #tripleo11:04
lvdombrkrguys, im trying enable ssl for overcloud, in ofic documentation i can read: To start, it is necessary to have a predictable public VIP. As of this writing, that means using network isolation.11:05
*** ffiore has quit IRC11:05
lvdombrkris there any posibility enable ssl without network isolation?11:06
*** ansmith has quit IRC11:06
jprovazndtantsur: nice. thanks11:07
*** garyk1 has quit IRC11:08
*** ooolpbot has joined #tripleo11:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION11:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168955611:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169140311:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295511:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171291011:10
openstackLaunchpad bug 1689556 in tripleo "CI: containers overcloud can't start vm when deployed with multinic network isolation" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)11:10
*** ooolpbot has quit IRC11:10
openstackLaunchpad bug 1691403 in tripleo "containerized overcloud - neutron-openvswitchagent fails to start" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)11:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel)11:10
openstackLaunchpad bug 1712910 in tripleo "oooq virt-customize repo setup logs are not saved " [Critical,Triaged] - Assigned to wes hayutin (weshayutin)11:10
*** stendulker has quit IRC11:15
gfidentedtantsur again on the automated_clean default11:17
gfidentedo I remember correctly it is enabled by default in ironic, it is enabled by default in undercloud.conf but oooq is disabling it?11:17
lvdombrkrcan my network. yaml look like this http://paste.openstack.org/raw/619745/11:17
lvdombrkrif i want just add ssl11:18
lvdombrkr?11:18
jprovazngfidente: http://tripleo.org/install/mistral-api/mistral-api.html#cleaning-nodes says " As TripleO disables automated cleaning,...."11:18
lvdombrkr guys, im trying enable ssl for overcloud, can i do this without network isolation? or with simple .yaml like http://paste.openstack.org/raw/619745/11:23
lvdombrkr?11:23
*** Vijayendra has quit IRC11:24
gfidentejprovazn ah https://github.com/openstack/instack-undercloud/blob/1d8ad41db9cba8f23ccba4a78f7a13405e57ef5d/instack_undercloud/undercloud.py#L37311:24
jprovazngfidente: dtantsur: I can imagine that combination of disabled cleanup (by default) and ceph-ansible may cause quite a lot of pain to tripleo deployers11:25
gfidentebut it is true in ironic https://github.com/openstack/ironic/blob/master/ironic/conf/conductor.py#L11911:26
gfidenteso I think the default in tripleo is a safe measure11:28
sshnaidmadarazs, do you know why was it added? https://review.openstack.org/#/c/475747/108/toci-quickstart/playbooks/multinode.yml11:29
gfidentewe should probably add in the doc a comment about that pointing to how to cleanup the node disks in case of repeated ceph deployments11:29
sshnaidmadarazs, why do we need to build images in multinode and how is it related to that containers job?11:29
dtantsurgfidente, jprovazn, it's disabled by default (mostly historically)11:30
dtantsurI have nothing against enabling, but 1. it will make enrolling nodes longer, 2. it will make people who wipe their deployments very sad, as recovery won't be possible :)11:30
*** eck` is now known as eck`gone11:31
*** Vijayendra has joined #tripleo11:32
*** garyk has joined #tripleo11:32
jaosoriordciabrin, bandini: would using FQDNs for rabbitmq's NODENAME work? or will pacemaker have issues with it?11:34
bandinimmh hard to say. I'd be surprised if something did not break tbh11:35
jaosorior:(11:36
openstackgerritJose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Heat]  https://review.openstack.org/49139911:36
openstackgerritJose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Aodh]  https://review.openstack.org/49141111:36
openstackgerritJose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Mistral]  https://review.openstack.org/49347111:36
openstackgerritJose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Ceilometer]  https://review.openstack.org/49039211:36
openstackgerritJose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Neutron]  https://review.openstack.org/48995711:36
openstackgerritJose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Gnocchi]  https://review.openstack.org/49143311:36
openstackgerritJose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Mixed-1]  https://review.openstack.org/49519511:36
*** udesale has joined #tripleo11:36
*** ratailor has quit IRC11:38
*** pdeore has quit IRC11:38
*** pgadiya has quit IRC11:39
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: Multinode jobs should not build images  https://review.openstack.org/49877211:40
adarazssshnaidm: this is an interesting question. so we have a couple of multinode promotion jobs that are not child jobs of the image building job. don't they need images for the overcloud nodeS?11:42
sshnaidmadarazs, neither of multinode jobs needs images, we use deployed server ^^11:42
jaosoriorbandini: I think we'll need a cluster_nodes option like the galera resource agent :/11:42
jaosoriordciabrin: ^^11:42
bandinioh boy11:43
*** pdeore has joined #tripleo11:43
adarazssshnaidm: hm, okay. then it's probably not necessary. I checked some multinode job and indeed it doesn't build the image.11:43
sshnaidmadarazs, maybe we need to build them for promotion only..? but we do it in ovb jobs, never did this in multinode, because no way to test them11:43
*** shreshtha has quit IRC11:44
sshnaidmadarazs, actually it's not configured right for building images and it fails every patch for tripleo-common, etc11:44
adarazssshnaidm: looks like the image building job is running on multinode and that needs it: https://review.rdoproject.org/jenkins/job/periodic-tripleo-centos-7-master-images-build/136/consoleFull11:45
*** ykarel|afk is now known as ykarel11:46
*** rhallisey has joined #tripleo11:46
*** trown|outtypewww is now known as trown11:47
sshnaidmadarazs, why does it build them?11:47
adarazssshnaidm: then it also fetches them, sooo I'm a bit confused. :)11:47
trownsshnaidm: it builds images in periodic job11:47
sshnaidmtrown, why does multinode job build images?11:47
trownsshnaidm: https://review.rdoproject.org/jenkins/job/periodic-tripleo-centos-7-master-images-build/11:47
trownsshnaidm: that job11:48
adarazstrown, weshay: FYI we merged this: https://review.openstack.org/498688 -- let me know if it doesn't look all right, I can probably still -w it..11:48
trownadarazs: nice, that was the first thing I was going to look at this morning11:49
sshnaidmtrown, I see, but it doesn't answer the question - why *multinode* job?11:49
trownadarazs: saw that pike image build "worked", but built master images11:50
trownsshnaidm: because the job I pointed to is a multinode job11:50
sshnaidmtrown, it broke CI for tripleo-common, instack-undercloud and other repos that usually build images, if it's required for this job we need to limit it for rdo cloud only11:50
adarazssshnaidm: why does it matter which job builds the image? it doesn't use them, just builds them.11:50
trownsshnaidm: but there is really no reason to build the image there... since we are not testing it there anyways11:50
sshnaidmtrown, adarazs ovb job build the images and test them by deploying overcloud, multinode jobs don't do it11:51
trownsshnaidm: but then we need some way to upload from the ovb jobs in periodic pipeline11:51
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates master: Remove puppet run and workarounds from tripleo_upgrade_node.sh  https://review.openstack.org/49877611:51
trownsshnaidm: ya the job I pointed to is just a parent job which builds and sub jobs which test11:51
trownsshnaidm: but we didnt add jobs which test qcow image only containers11:51
sshnaidmtrown, adarazs so we build images that nobody uses? I'm confused..11:51
trownsshnaidm: take a deep breath11:52
adarazssshnaidm: I am a bit as well. I assumed that we will chain up the OVB jobs on the image build job eventually and use those built images.11:52
trownadarazs: sshnaidm, ^ that was the plan11:52
trownadarazs: sshnaidm, but we dont have to stick with that...11:52
adarazsbut now until the image build job was working it was easier to build in each job :)11:52
sshnaidmtrown, so what is the current plan?11:53
trownsshnaidm: s/was/is currently/  just saying we can change it if we have a way to upload from other jobs11:53
trownsshnaidm: but right now we upload from the job I linked, so if we dont build images there, we have no images to cache at the end of periodic11:54
sshnaidmtrown, why do we need to cache them in the end of periodic..? seems like I miss the whole picture11:55
trownsshnaidm: I was asking for your feedback on that review for weeks fwiw11:55
trownsshnaidm: umm... so we have images to use when we promote?11:55
sshnaidmtrown, yeah, and I for a weeks asked for explanation of this review11:55
adarazstrown: where are we uploading them?11:55
trownadarazs: to images.rdoproject.org11:56
adarazstrown: just found the upload part in the console log. so they are these, right? https://images.rdoproject.org/master/delorean/tripleo-upstream/testing/11:57
trownadarazs: ya11:57
*** abishop has joined #tripleo11:57
jaosoriorjistr, mwhahaha, EmilienM: Can you check this out https://review.openstack.org/#/c/498711/ ?11:57
trownadarazs: then on promote we would move those to passed-ci dir or the like11:57
sshnaidmtrown, can't we upload from ovb job?11:57
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo master: Enable TLS for rabbitmq's replication traffic  https://review.openstack.org/49832411:57
adarazstrown: ack.11:58
trownsshnaidm: can you spend some time to understand what we have setup in rdoproject.org?11:58
sshnaidmtrown, yeah, if you have some information for me11:58
*** lucasagomes is now known as lucas-hungry11:58
trownsshnaidm: we could upload from an ovb job... but how do we know the status of all jobs from a single job11:58
trownsshnaidm: but I guess in this case we dont care about that actually... so ya we could add to one of the ovb jobs to upload its images to testing link11:59
trownsshnaidm: promote is a later step anyways12:00
sshnaidmtrown, ok, anyway, I need either to remove build image part from multinode playbook, or limit it for rdocloud only and configure it correctly12:00
sshnaidmtrown, right now it's bug12:00
sshnaidmtrown, so let's decide either we use multinode job or not12:00
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Rabbitmq: Enable Erlang distribution TLS  https://review.openstack.org/49832512:01
trownsshnaidm: well if you remove it you have to fix jjb in rdoproject.org12:01
trownsshnaidm: the upload is called from the jjb and it will fail if there are not images to upload12:01
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Rabbitmq: Enable Erlang distribution TLS  https://review.openstack.org/49832512:01
*** ffiore has joined #tripleo12:02
sshnaidmtrown, just curios, do you have the current workflow described anywhere that I can see it?12:03
trownsshnaidm: ya rdoproject.org jjb12:04
sshnaidmtrown, it's code, not docs12:04
sshnaidmit's not even code, it's *jjb*12:05
*** fzdarsky|lunch is now known as fzdarsky12:07
*** ffiore has quit IRC12:09
*** ooolpbot has joined #tripleo12:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION12:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168955612:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169140312:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295512:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171291012:10
openstackLaunchpad bug 1689556 in tripleo "CI: containers overcloud can't start vm when deployed with multinic network isolation" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)12:10
*** ooolpbot has quit IRC12:10
openstackLaunchpad bug 1691403 in tripleo "containerized overcloud - neutron-openvswitchagent fails to start" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)12:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel)12:10
openstackLaunchpad bug 1712910 in tripleo "oooq virt-customize repo setup logs are not saved " [Critical,Triaged] - Assigned to wes hayutin (weshayutin)12:10
*** jlabarre has joined #tripleo12:11
openstackgerritOr Idgar proposed openstack/tripleo-common master: WIP Add Octavia post deployment steps to tripleo workflows  https://review.openstack.org/48745012:11
*** morazi has joined #tripleo12:11
weshaytrown, sshnaidm reading through just checking https://review.openstack.org/#/c/498772/1/toci-quickstart/playbooks/multinode.yml addresses https://bugs.launchpad.net/tripleo/+bug/1713167 ?12:12
openstackLaunchpad bug 1713167 in tripleo "build-images : Throw error msg if image build is failing *** Image Build Error ***" [High,Triaged]12:12
*** atoth has joined #tripleo12:13
*** jpena is now known as jpena|lunch12:14
sshnaidmweshay, yes12:16
weshaysshnaidm, trown k.. thanks guys12:16
*** pchavva has joined #tripleo12:19
adarazsweshay, sshnaidm: it addresses like cutting off a leg when you get a wound -- this would result in the periodic rdoproject.org jobs not building and uploading images for now and fail when trying to upload them. not that we currently use that images, but I think we should tune the to_build script to return stuff properly instead of removing it.12:20
*** MVenesio has joined #tripleo12:20
*** ffiore has joined #tripleo12:20
sshnaidmadarazs, weshay  I don't so like the idea to build images with multinode job12:21
weshayadarazs, sshnaidm ok.. that is what env is for :)12:21
adarazssshnaidm: why does it matter what nodes the image build runs on if we verify them in other jobs?12:22
sshnaidmweshay, depends.. env couldn't help if it's calculated dynamically as it happens now12:22
*** dtantsur is now known as dtantsur|busy12:23
sshnaidmadarazs, if we verify them in other jobs it doesn't matter, but it will save us this step12:23
sshnaidmadarazs, actually we never built images in multinode, so need to check it before12:24
weshaythe check vs periodic vs. rdo soft factory jobs could all run w/ diff env files12:24
*** aditya_r has joined #tripleo12:24
*** bkopilov has quit IRC12:25
*** aditya_r has quit IRC12:25
*** bkopilov has joined #tripleo12:25
*** aditya_r has joined #tripleo12:25
weshayadarazs, where are we failing on upload, do you have a log?12:28
adarazsweshay: we're not, but if sshnaidm would take out the part that builds the images then we would fail in the rdoproject.org image building jobs on the upload.12:28
*** oidgar has quit IRC12:28
*** aditya_r has quit IRC12:29
sshnaidmweshay, shouldn't we have this workflow described anywhere? like diagram or chart..12:29
sshnaidmusing jjb as source of information is a nightmare12:30
weshaysshnaidm, that would certainly be helpful12:30
sshnaidmjjb is a labyrinth12:30
weshayya..12:30
weshaysshnaidm, adarazs could we not just set to_build appropriately in a env config?12:32
*** pdeore has quit IRC12:32
weshaymajor changes at this point of the release is not ideal12:33
*** pdeore has joined #tripleo12:33
adarazsweshay: I think that would be better, if we can.12:33
weshaywell, I know we can.. adarazs sshnaidm want to hash it out?12:34
* adarazs looks at the to_build script.12:34
*** aditya_r has joined #tripleo12:36
sshnaidmweshay, to_build builds according to zuul changes, how is it related to building periodic?12:36
*** ykarel has quit IRC12:36
*** janki has quit IRC12:37
sshnaidmweshay, you want to build images always in periodic jobs, right?12:37
adarazssshnaidm: well I assume that now that the to_build is added to the multinode job we have to just make a case for "when multinode don't build an image, unless it's periodic/promotion"?12:37
adarazsor unless it's that image building job specificially12:38
sshnaidmadarazs, weshay do we have a var that identifies rdo cloud?12:38
weshaysshnaidm, adarazs taking a step back, has anyone confirmed that fetch_images is actually getting the centos image12:38
weshayatm the error afaict is that the image was not found12:38
sshnaidmweshay, no, it doesn't build correctly at all, it's next step..12:39
weshayok12:39
weshaysshnaidm, and why is it not building correctly?12:39
*** gkadam_ has quit IRC12:39
weshaywhat is the error?12:39
sshnaidmweshay, because it's not configured correctly12:39
weshaysshnaidm, have an example?12:40
sshnaidmit's not configured at all and tries to build as it was in libvirt..12:40
sshnaidmweshay, the logs you sent are an example12:40
*** pdeore_ has joined #tripleo12:40
*** rlandy has joined #tripleo12:41
weshaysshnaidm, so which log file indicates to you it's not configured correctly?12:41
rnoriegaEmilienM, ping12:41
sshnaidmweshay, all of them :)12:41
adarazssshnaidm: do we have a variable that is multinode/ovb so that we can check on it in to_build?12:41
sshnaidmadarazs, mm.. not sure I get it, what do you want to do..?12:41
*** pdeore has quit IRC12:42
jaosoriordciabrin: any reason why this was added https://github.com/openstack/puppet-tripleo/blob/master/manifests/profile/pacemaker/database/redis_bundle.pp#L136-L140 ?12:42
weshaysshnaidm, all I can see from the logs is that the isolation image is not present12:42
adarazssshnaidm: wrap this in a "if multinode set_env false", else: https://github.com/openstack-infra/tripleo-ci/blob/master/scripts/to_build#L12-L1712:42
jaosoriorbandini: is it possible to restart a service from a specific node with pacemaker?12:43
sshnaidmweshay, it's a symptom, not a problem12:43
sshnaidmweshay, this how the images build should be configured: https://github.com/openstack-infra/tripleo-ci/blob/3a18965dae750f17154eb10121266f6ce72255d4/toci-quickstart/config/testenv/ovb.yml#L24-L3312:43
*** eck`gone is now known as eck`12:43
*** tzumainn has joined #tripleo12:44
weshaysshnaidm, ah.. so build_image_isolated should be false?12:44
sshnaidmadarazs, let's clarify the condition when we build - periodic=true && rdocliud=true && ... something else?12:45
sshnaidmweshay, of course12:45
weshaysshnaidm, ya.. we already have some variables defined to distinguish between jobs in upstream vs rdo software factory12:46
weshayand periodic vs. check12:46
sshnaidmweshay, adarazs should we build in every periodic rdocloud job?12:46
sshnaidmor on specific job?12:47
adarazssshnaidm: I think only in the image build job. /me looks at the beloved JJB definitions :)12:47
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci master: WIP -- switch multinode composable upgrade job to oooq  https://review.openstack.org/46358012:47
sshnaidmor maybe just let's build on ovb jobs..12:47
weshaysshnaidm, atm that is what we were doing because there was no parent / child jobs upstream12:47
sshnaidmadarazs, then maybe let's take TOCI_JOBTYPE into consideration12:47
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci master: WIP -- switch multinode composable upgrade job to oooq  https://review.openstack.org/46358012:47
weshayyou can't build the image in one job, upload it and then have other jobs consume it w/ the older style tripleo promotion jobs12:47
*** cdearborn has joined #tripleo12:48
weshaysshnaidm, so afaict.. there wasn't much of a choice12:48
sshnaidmweshay, it saves us 30 mins and brings problems for a days..12:48
sshnaidmweshay, does it worth this 30 min?12:48
weshaysshnaidm, it's not something we could change afaict while both set of jobs are running12:49
weshayit would have to be done after we bring sf jobs online12:49
weshayboth sets have to be functional atm12:50
*** ecerquei has joined #tripleo12:50
sshnaidmweshay, sorry, I don't understand what you mean..12:51
jaosorioreck`: I just tested out the patches (non-containerized, containerized testing coming next) and they work. Now, the only issue is that I couldn't enable verification because pacemaker tracks shortnames... and we can only issue certificates for FQDNs12:51
weshaysshnaidm, let's bluejeans real quick12:51
bandinijaosorior: pcs resource restart foo node-bar12:52
weshayadarazs, trown fyi.. if you are interested in discussing the image build issue w/ the isolated image12:52
*** ansmith has joined #tripleo12:52
jaosorioreck`: so, the current state would meet the minimum requirement (encryption everywhere); but it's not as secure as I wish it was. So alternatively we need to add cluster-names for the rabbitmq resource agent, as we do for mysql12:52
adarazsweshay: joining12:52
eck`jaosorior: awesome, i will try to sanity check ipv6+tls if you haven't done so already12:52
weshaymy bluejeans12:52
jaosoriorbandini: how disruptive is that? thinking about using that for certmonger's post-save command12:53
jaosorioreck`: please do, I haven't checked that.12:53
jaosorioreck`: what do you think of the commits overall though?12:53
eck`jaosorior: looked fine to me!12:54
jaosoriorgreat!12:54
*** jmelvin has joined #tripleo12:55
bandinijaosorior: meh, actually you cannot do it for bundle it appears12:55
jaosoriorbandini: no??? :(12:56
jaosoriorbandini: why?12:56
bandininot sure where the limitation comes from, but pcs for sure is not liking that12:56
jaosoriorshit fuck12:57
jaosoriorX_X12:57
jaosoriorbandini: know who I can ask?12:57
bandinijaosorior: I just asked andrew, but he is in .au, so it will have to wait until tonight12:58
jaosoriorok12:59
jaosoriorthanks dude12:59
jaosoriorbandini: now... if we could get restarts for bundles in the near future, that would be reaally neat :D12:59
jaosoriorlets see what answer we get13:00
*** ebarrera has quit IRC13:00
bandiniack13:00
jaosoriorbandini: need reviews?13:00
*** number80 has quit IRC13:00
bandinino thanks, am debugging some resource agents changes13:00
*** lucas-hungry is now known as lucasagomes13:01
*** jcoufal has joined #tripleo13:01
*** toure_biab is now known as toure13:01
*** ebarrera has joined #tripleo13:02
*** pblaho has quit IRC13:02
*** number80 has joined #tripleo13:02
cdearbornHey folks, I've run into a problem where I have a node that I want to deploy ceph to, and it has 2 disk controllers in it.13:04
cdearbornThe order in which the disks are enumerated is either all disks on controller A, then all disks on controller B or vice versa13:05
cdearbornThis makes it so the OSD/journal configuration ends up being messed up13:05
cdearbornbecause the OS drive could be /dev/sda or /dev/sds13:05
cdearbornI've been trying to find a solution to this13:06
cdearbornI was wondering if it is possible to specify the OSD/journal mapping separately for each ceph node?13:06
*** garyk has quit IRC13:07
*** aditya_r has quit IRC13:07
*** aditya_r has joined #tripleo13:07
*** aditya_r has quit IRC13:08
*** aufi has quit IRC13:08
openstackgerritMartin André proposed openstack/tripleo-common master: [WIP] Make curl healthchecks work with internal TLS  https://review.openstack.org/49880513:08
*** morazi has quit IRC13:08
*** ooolpbot has joined #tripleo13:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION13:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168955613:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169140313:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295513:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171291013:10
openstackLaunchpad bug 1689556 in tripleo "CI: containers overcloud can't start vm when deployed with multinic network isolation" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)13:10
*** ooolpbot has quit IRC13:10
openstackLaunchpad bug 1691403 in tripleo "containerized overcloud - neutron-openvswitchagent fails to start" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)13:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel)13:10
openstackLaunchpad bug 1712910 in tripleo "oooq virt-customize repo setup logs are not saved " [Critical,Triaged] - Assigned to wes hayutin (weshayutin)13:10
*** morazi has joined #tripleo13:10
*** aufi has joined #tripleo13:11
jistrcdearborn: hi, personally i haven't tried this, but i think this may solve the issue http://tripleo.org/install/advanced_deployment/node_specific_hieradata.html13:11
jistrcdearborn: assuming you deploy stable, not Pike yet13:12
*** leitan has joined #tripleo13:12
cdearbornjistr: Thanks!  I'm actually using Newton.  Any idea if this capability is in that release?13:13
gfidentecdearborn jistr++ that does it yes13:14
adarazssshnaidm: FYI this is what we should remove to not upload images: https://review.rdoproject.org/r/gitweb?p=config.git;a=blob;f=jobs/tripleo-upstream.yml;h=e883555e504d88c82ad6649f3fed4de32a1e165e;hb=HEAD#l23313:14
gfidentethanks13:14
*** dpawar has quit IRC13:14
gfidentecdearborn note the uuid you need to use is not the ironic node uuid, it is the system uuid13:14
jistrcdearborn: yea the file is present in stable/newton so i think it should work https://github.com/openstack/tripleo-heat-templates/blob/stable/newton/puppet/extraconfig/pre_deploy/per_node.yaml13:14
cdearborngfidente, jistr: thanks!  my thought was i could probe the ceph nodes and build an OSD/journal config using /dev/disk/by-id/* device names13:16
*** eck` is now known as eck`gone13:16
cdearborngfidente, jistr: any other thoughts on how to work around the issue?13:16
gfidentecdearborn you can also use the dev aliases but not in newton I think13:17
cdearborngfidente: can you clarify?13:17
gfidentecdearborn when specifying the disk names, you can refer to the aliases which udev creates13:18
gfidenteinstead of /dev/sdX13:18
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: Multinode jobs should not build images  https://review.openstack.org/49877213:18
*** links has quit IRC13:18
cdearborngfidente, right, that's what I meant above by using /dev/disk/by-id/*13:18
gfidentecdearborn yeah but then you'd still need per-node data right?13:19
*** catintheroof has joined #tripleo13:19
cdearborngfidente: yes, would love to solve this without having to use per-node data13:19
*** eck`gone is now known as eck`13:19
*** ecerquei_ has joined #tripleo13:20
gfidentecdearborn yeah but I don't think that is possible because I don't think we have control over the disk discovery13:20
cdearborngfidente: I looked into using /dev/disk/by-path/* for device names, but those names are specific by node too in RHEL 7.4 because they include the SASAddress of the dirve13:20
*** jpena|lunch is now known as jpena13:20
gfidentecdearborn in pike you should be able to use regexp13:21
gfidentebut earlier I think the only option is per-node data13:21
cdearborngfidente, really!  regexp would be very nice13:22
*** ecerquei has quit IRC13:22
gfidentecdearborn yeah though that is pike13:22
*** lblanchard has joined #tripleo13:24
cdearborngfidente, jistr, The only other solution I could come up with is to modify the udev rules so that the by-path names generated are names that are the same across nodes - no SASAddresses in names, maybe bay numbers or something like that13:25
*** lucasagomes is now known as lucas-brb13:26
cdearborngfidente, jistr, maybe switch udev rules back to what they were in RHEL 7.3, because the by-path names did not include the SASAddresses in that version of RHEL13:26
cdearborngfidente, jistr, any thoughts on this approach?13:26
gfidentecdearborn this is actually interesting13:26
gfidenteas I didn't know the behavior changed in newer RHEL13:27
*** oidgar has joined #tripleo13:27
*** oidgar has quit IRC13:27
*** oidgar has joined #tripleo13:28
cdearborngfidente, hmmm - one other thing that changed was the generation of server that I'm using, so i guess that could have changed due to the driver and not due to RHEL version.  will check into that13:29
*** udesale has quit IRC13:30
*** mrcc has joined #tripleo13:30
gfidentecdearborn++ thanks let us know how it goes13:32
*** masco has quit IRC13:32
*** ykarel has joined #tripleo13:34
*** morazi has quit IRC13:35
*** morazi has joined #tripleo13:36
*** garyk has joined #tripleo13:37
cdearborngfidente, ah, i see the difference.  rhel 7.4, but the by-path names in one server are scsi, and the by-path names in the other server are sas with SASAddress13:38
cdearborngfidente, the scsi names are the same across older server, but the sas names are unique per node on newer servers.  ugh13:39
gfidentecdearborn so you don't have good aliases to use anymore13:39
gfidenteI see13:39
gfidentewell if you have different 'group' of servers13:40
*** dparkes has quit IRC13:40
gfidenteanother option is to use a custom role13:40
gfidenteand pass two different list of disks to each role type13:40
cdearborngfidente, that's a thought...  that would help with the 2 different generations of the servers, but still wouldn't solve the original problem of the OS drive moving around in the enumeration on the newer server13:42
*** ykarel has quit IRC13:42
EmilienMhi13:42
gfidentecdearborn yeah13:42
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates stable/pike: Set docker-puppet --health-cmd = /bin/true  https://review.openstack.org/49881213:43
gfidentecdearborn or wait for pike13:43
*** gbarros has joined #tripleo13:44
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart master: WIP create default settings for environment types  https://review.openstack.org/49540713:44
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart master: Change devmode to deploy containerized services by default  https://review.openstack.org/47241213:44
cdearborngfidente very true.  will dig in to the per node config and see what i can do there13:44
gfidentecdearborn you can try upstream code13:45
gfidenteto check if the regexp works13:45
*** anshulbehl has joined #tripleo13:45
gfidenteit's basically ceph-ansible13:45
*** anshul has quit IRC13:46
*** ffiore has quit IRC13:47
cdearborngfidente, ok, unfortunately I'm constrained to using Newton for now13:48
*** aditya_r has joined #tripleo13:53
*** agurenko has quit IRC13:53
EmilienMtripleo meeting in 5 min, here, you can bring topics on https://etherpad.openstack.org/p/tripleo-meeting-items13:55
*** garyk has quit IRC13:56
cdearborngfidente, jistr, thanks for all your help!13:56
*** ykarel has joined #tripleo13:57
*** jprovazn has quit IRC13:58
*** ffiore has joined #tripleo13:59
EmilienMmeeting time14:00
EmilienM#startmeeting tripleo14:00
openstackMeeting started Tue Aug 29 14:00:57 2017 UTC and is due to finish in 60 minutes.  The chair is EmilienM. Information about MeetBot at http://wiki.debian.org/MeetBot.14:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.14:00
*** openstack changes topic to " (Meeting topic: tripleo)"14:01
openstackThe meeting name has been set to 'tripleo'14:01
EmilienM#topic agenda14:01
*** openstack changes topic to "agenda (Meeting topic: tripleo)"14:01
EmilienM* review past action items14:01
EmilienM* CI status14:01
EmilienM* bugs14:01
EmilienM* Projects releases or stable backports14:01
EmilienM* Specs / blueprints14:01
openstackgerritMartin André proposed openstack/tripleo-common master: Make curl healthchecks work with internal TLS  https://review.openstack.org/49880514:01
EmilienM* one off agenda items14:01
EmilienM* open discussion14:01
EmilienMAnyone can use the #link, #action and #info commands, not just the moderatorǃ14:01
*** aufi has quit IRC14:01
EmilienMHi everyone! who is around today?14:01
mwhahahahi14:01
adarazs14:01
*** garyk has joined #tripleo14:01
jkilpatr\o14:02
sshnaidm8/14:02
mandrehello14:02
pradko/14:02
jristo/14:02
cdearborn\o14:02
jaosorioro/14:02
EmilienMthere is no action taken recorded in last week meeting14:02
marioso/14:02
EmilienMso I'll skip "review past action items" item in the agenda14:02
EmilienM#topic CI status14:03
*** openstack changes topic to "CI status (Meeting topic: tripleo)"14:03
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates master: Add support for deploying RGW with ceph-ansible  https://review.openstack.org/48080414:03
EmilienMallright, we have a bunch of alerts which affect CI status, let's review it14:03
EmilienMhttps://bugs.launchpad.net/tripleo/+bug/168955614:03
openstackLaunchpad bug 1689556 in tripleo "CI: containers overcloud can't start vm when deployed with multinic network isolation" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)14:03
EmilienM#link https://review.openstack.org/#/c/493695/ is the fix14:03
EmilienMso this one is covered14:03
EmilienMhttps://bugs.launchpad.net/tripleo/+bug/169140314:04
openstackLaunchpad bug 1691403 in tripleo "containerized overcloud - neutron-openvswitchagent fails to start" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)14:04
saneaxo/14:04
trowno/14:04
*** japestinho has joined #tripleo14:04
EmilienMthis one needs review14:04
openstackgerritJaganathan Palanisamy proposed openstack/tripleo-common master: Derive Params OvsPmdCoreList compute issue  https://review.openstack.org/49881614:04
beagleso/14:04
EmilienM#action team to help with reviewing https://review.openstack.org/#/q/topic:bug/1691403 (CI alert fix)14:04
EmilienMif you're familiar with docker & networking, please have a look ^14:05
EmilienMthere are some patches on quickstart as well, sshnaidm, trown, adarazs ^14:05
EmilienMnext one: https://bugs.launchpad.net/tripleo/+bug/170295514:06
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel)14:06
EmilienMmatbu: any update on this one?14:06
* adarazs is looking14:06
EmilienMthe last one will be https://bugs.launchpad.net/tripleo/+bug/171291014:06
openstackLaunchpad bug 1712910 in tripleo "oooq virt-customize repo setup logs are not saved " [Critical,Triaged] - Assigned to wes hayutin (weshayutin)14:06
EmilienMwhich is I think wip by weshay and ssh14:06
EmilienMsshnaidm: ^14:06
sshnaidmEmilienM, yeah, patch is on the gate I think14:07
EmilienMsshnaidm: https://review.openstack.org/#/c/498772 ?14:07
sshnaidmEmilienM, oh, yeah14:07
abishopo/14:07
EmilienMsshnaidm: when merged and fixed, please update the launchpad14:07
sshnaidmEmilienM, sure14:08
EmilienMdo we have any other CI issues at this time?14:08
EmilienMalright - is there anything else to discuss about CI this week?14:09
openstackgerritJiri Tomasek proposed openstack/tripleo-ui master: Remove semicolons  https://review.openstack.org/49882214:09
EmilienMreminder: we have a 1h meeting, focused on tripleo ci work, on thursday. ping adarazs or me to get details14:09
EmilienMmoving on14:10
EmilienM#topic bugs14:10
*** openstack changes topic to "bugs (Meeting topic: tripleo)"14:10
EmilienM#link https://launchpad.net/tripleo/+milestone/pike-rc214:10
*** jaganathan has quit IRC14:10
*** ooolpbot has joined #tripleo14:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION14:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168955614:10
openstackLaunchpad bug 1689556 in tripleo "CI: containers overcloud can't start vm when deployed with multinic network isolation" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)14:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169140314:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295514:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171291014:10
openstackLaunchpad bug 1691403 in tripleo "containerized overcloud - neutron-openvswitchagent fails to start" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)14:10
*** ooolpbot has quit IRC14:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel)14:10
openstackLaunchpad bug 1712910 in tripleo "oooq virt-customize repo setup logs are not saved " [Critical,In progress] - Assigned to wes hayutin (weshayutin)14:10
adarazsCI Squad meeting details: https://etherpad.openstack.org/p/tripleo-ci-squad-meeting14:10
EmilienMadarazs: thanks14:10
*** skramaja has joined #tripleo14:10
EmilienM76 Triaged, 62 In Progress, currently for RC2 - it's a lot14:10
*** skramaja has quit IRC14:11
EmilienMI would use some help to triage them otherwise the script will move them to queens-1 automatically when we release final pike14:11
EmilienMbut we want critical / high bugs fixed and backported to stable/pike, as a goal14:11
*** jmelvin has quit IRC14:11
EmilienMany bug to discuss this week in particular?14:12
EmilienMno bug? perfect. Let's move on ;-)14:12
EmilienM#topic projects releases or stable backports14:12
*** openstack changes topic to "projects releases or stable backports (Meeting topic: tripleo)"14:12
EmilienMpike: we released RC1 last week (thanks mwhahaha for the huge help) - which means we have stable/pike branch everywhere in tripleo now14:13
openstackgerritHonza Pokorny proposed openstack/tripleo-ui master: Add 'discover and enroll nodes' workflow  https://review.openstack.org/48852614:13
EmilienMwhich means: please take care of backporting patches you want in stable/pike (need FFE for blueprints or a launchpad bug for bugs)14:13
*** aufi has joined #tripleo14:13
EmilienMwe aim to release RC2 by next week, we still have a bunch of things to land14:14
EmilienMany question about Pike release?14:14
EmilienMif any, ping me or mwhahaha anytime14:14
*** cylopez has quit IRC14:14
*** garyk has quit IRC14:15
EmilienMI also plan to release new tags for Newton and Ocata - probably this week14:15
EmilienMjschlueter|znc: ^ fyi14:15
*** jschlueter|znc is now known as jschlueter14:15
EmilienMany question / feedback about release management and backports?14:15
jschlueterEmilienM: ack14:15
EmilienM#topic specs / blueprints14:16
*** openstack changes topic to "specs / blueprints (Meeting topic: tripleo)"14:16
EmilienM#link https://review.openstack.org/#/q/project:openstack/tripleo-specs+status:open14:16
EmilienM#link https://launchpad.net/tripleo/+milestone/pike-rc214:16
EmilienMso we still have 5 open blueprints14:16
*** ecerquei__ has joined #tripleo14:16
EmilienMDell blueprints are going to be implemented in Pike, some patches are missing, but all in gate now.14:17
EmilienMhttps://blueprints.launchpad.net/tripleo/+spec/websocket-logging will be closed in Pike iiuc14:17
*** jjoyce has joined #tripleo14:17
EmilienMhttps://blueprints.launchpad.net/tripleo/+spec/node-discovery-by-range as well (closed in Pike) though the UI part will be postponed to queens14:18
EmilienMhonza, jrist ^ correct?14:18
jristyes14:18
EmilienMgood14:18
*** ecerquei_ has quit IRC14:18
jristliterally only 1 patch for websocket-logging14:18
EmilienMplease update the launchpad when done, otherwise I'll look myself next week14:18
EmilienMjrist: good!14:18
jristok not true. a backport :(14:18
honza#link https://blueprints.launchpad.net/tripleo/+spec/gui-node-discovery-by-range14:18
EmilienMhonza: excellent14:19
honzanew node discovery gui part blueprint14:19
EmilienMhonza: approved14:19
EmilienMfor queens-1, to start14:19
jrist:)14:20
EmilienMand https://blueprints.launchpad.net/tripleo/+spec/node-discovery-by-range closed14:20
EmilienMsorry I've lost internet 1min14:21
EmilienMcan you see my msg?14:21
mwhahahayea14:21
EmilienMok thx14:21
EmilienMa last note about blueprints (I'll send a note to ML as well)14:22
EmilienM#action EmilienM sends a note about Queens blueprints / specs on ML14:22
EmilienMplease prepare your blueprints / specs prior to the PTG14:22
EmilienMit really help our team when things are written and prepared before, so we can get into the right topics during face to face meetings14:22
EmilienMso if you have something you want to do in Queens, please create a blueprint (or a spec if you like) - before the PTG, and share it on the ML if you want14:23
EmilienMany question?14:24
EmilienMmoving on14:24
EmilienM#topic one off agenda items14:24
*** openstack changes topic to "one off agenda items (Meeting topic: tripleo)"14:24
EmilienM#link https://etherpad.openstack.org/p/tripleo-meeting-items14:24
EmilienMmwhahaha: go ahead14:24
mwhahahaAs previously mentioned, the PTG is soon. Please see the ML post by EmilienM about the agenda.14:24
mwhahaha#link http://lists.openstack.org/pipermail/openstack-dev/2017-August/121574.html14:24
mwhahahaAlso please take some time to review the proposed topics.14:24
mwhahaha#link https://etherpad.openstack.org/p/tripleo-ptg-queens14:24
mwhahahathat is all14:24
EmilienMthanks for the reminder14:24
EmilienMcan i use uppercase?14:24
EmilienMPLEASE READ THIS ETHERPAD AND CHECK THE SCHEDULE14:25
mwhahaha:o14:25
EmilienMthat would be super helpful14:25
EmilienMI try to bring sessions together and create some slots that make sense14:25
EmilienMlike CI topics or Heat / mistral things etc14:25
*** aditya_r has quit IRC14:26
EmilienMANY feedback on the agenda / schedule is very welcome, you can ping me on IRC or email or on the ML in the thread14:26
EmilienMthat is all14:26
EmilienMsshnaidm: go ahead please14:26
*** anshulbehl has quit IRC14:26
sshnaidmjust a few reminders of merging policy in tripleo-quickstart/-extras and tripleo-ci repos14:26
sshnaidmif you gonna merge patches into oooq, oooq-extras or tripleo-ci repos, please attend our tripleo-ci-squad meetings every Thursday 14.30 UTC, so that you'll get the overview of what is going in CI, please take into account: http://lists.openstack.org/pipermail/openstack-dev/2017-July/119766.html14:27
sshnaidmif it's possible, please avoid from merging patches without review of one of oooq-cores, please remember that this framework is used both for a *few* CI systems and for running locally by tripleo developers, you may affect all of them.14:27
sshnaidmand general point of course - please be cautious with merging stuff that you don't have full understanding of it, even if it's something urgent14:27
EmilienMsshnaidm: it reminds me a patch ;-) let's discuss about this meeting14:27
sshnaidmok14:27
*** pdeore__ has joined #tripleo14:28
EmilienM#topic open discussion14:28
*** openstack changes topic to "open discussion (Meeting topic: tripleo)"14:28
EmilienMany question or general feedback?14:28
EmilienMok14:29
EmilienM#endmeeting14:29
*** openstack changes topic to "CI Status: YELLOWish (undercloud-oooq failures) | "you can't land a Cessna with workarounds, so don't try in Quickstart" - TripleO : http://tripleo.org/ | https://wiki.openstack.org/wiki/TripleO | Meetings On Tuesdays at 14:00 UTC here."14:29
openstackMeeting ended Tue Aug 29 14:29:31 2017 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)14:29
openstackMinutes:        http://eavesdrop.openstack.org/meetings/tripleo/2017/tripleo.2017-08-29-14.00.html14:29
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/tripleo/2017/tripleo.2017-08-29-14.00.txt14:29
EmilienMnow go back to work :P14:29
openstackLog:            http://eavesdrop.openstack.org/meetings/tripleo/2017/tripleo.2017-08-29-14.00.log.html14:29
EmilienMsshnaidm: I would love to get a review on https://review.openstack.org/#/c/498573/ as soon as we can, we got tempest working on scenario002-container (basically a copy paste from the other featureset)14:30
*** pdeore_ has quit IRC14:30
*** dprince has joined #tripleo14:32
sshnaidmEmilienM, looking.. also arxcruz and chandankumar are welcome to look, as it contains tempest config14:32
EmilienMsshnaidm: fine but we need to get this one merged really asap, if not today14:33
* arxcruz loves to see tempest on scenarios :)14:34
EmilienMsshnaidm: tbh, the time to get reviews on oooq is really slow14:34
sshnaidmEmilienM, this patch doesn't have reviewers added14:35
EmilienMsshnaidm: would it help if we add reviewers in gerrit?14:35
sshnaidmEmilienM, +w14:35
sshnaidmEmilienM, well, I get mail about it at least14:35
sshnaidmEmilienM, and usually firstly review where I've been added14:36
sshnaidmEmilienM, but not everybody do like this.. you can ping cores in IRC like now14:36
sshnaidmEmilienM, and if it's not urgent, we have squad meeting14:37
sshnaidmEmilienM, and if it's something big and complex, we have even oooq review meetings14:37
EmilienMsshnaidm: ok, fair enough14:37
*** jmelvin has joined #tripleo14:38
EmilienMfor these who go to the PTG, I would like to share a nice reading about organization in Denver: https://ttx.re/queens-ptg.html - please take 5 min14:38
*** chlong has joined #tripleo14:41
*** lucas-brb is now known as lucasagomes14:41
lvdombrkrhelo folks, if anyone tried deploy overcloud with ssl without network isalation?14:42
*** mrcc has quit IRC14:42
jpichUseful link, thanks. Nice work on the proposed schedule as well!14:42
jristjtomasek_: ew ;)14:44
jristjtomasek_: do you agree with the no semicolons thing?14:44
jristjtomasek_: I like explicit14:44
jtomasek_jrist: I have created the patch to start the discussion. I am not entirely convinced that we need to get rid of semicolons, on the other hand, the semicolons are not needed at all:)14:47
jristyeah14:47
jristI commented14:47
jristThis seems like complication for the sake of complication. The curmudgeon inside me is especially disturbed.  I'm not particularly against it though.14:48
*** itlinux has joined #tripleo14:49
*** Goneri has joined #tripleo14:50
jtomasek_jrist: the issue in the first link has a good discussion and feels discouraging, although when you look at the discussion under other 2 links, the problems are resolved. Also the problems are tied to prettier implementation rather than regular js without semicolons usage14:50
*** ratailor has joined #tripleo14:51
jpichjtomasek_: Seems like a lot of churn for not much gain. It's also going to cause a million conflicts and make the git history (git blame ;)) (for debugging) useless for a while14:51
jtomasek_jpich: right, that is my biggest concern. good point14:52
lvdombrkr helo folks, if anyone tried deploy overcloud with ssl without network isalation?14:57
*** iranzo has quit IRC15:01
*** yamahata has joined #tripleo15:01
*** jprovazn has joined #tripleo15:02
openstackgerritHarry Rybacki proposed openstack/tripleo-quickstart-extras master: Update validate-tempest role docs  https://review.openstack.org/49883615:02
*** iranzo has joined #tripleo15:03
*** iranzo has quit IRC15:03
*** iranzo has joined #tripleo15:03
*** ecerquei__ has quit IRC15:03
*** ecerquei has joined #tripleo15:04
*** limao has joined #tripleo15:09
*** ooolpbot has joined #tripleo15:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION15:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168955615:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169140315:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295515:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171291015:10
*** ooolpbot has quit IRC15:10
openstackLaunchpad bug 1689556 in tripleo "CI: containers overcloud can't start vm when deployed with multinic network isolation" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)15:10
openstackLaunchpad bug 1691403 in tripleo "containerized overcloud - neutron-openvswitchagent fails to start" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)15:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel)15:10
openstackLaunchpad bug 1712910 in tripleo "oooq virt-customize repo setup logs are not saved " [Critical,In progress] - Assigned to wes hayutin (weshayutin)15:10
lvdombrkrhelo folks, if anyone tried deploy overcloud with ssl without network isalation?15:11
*** trown is now known as trown|brb15:12
*** cdearborn_ has joined #tripleo15:13
*** dmarlin has joined #tripleo15:13
*** limao_ has joined #tripleo15:13
abishopsshnaidm: hoping you can review https://review.openstack.org/474981 and https://review.openstack.org/48617515:15
*** itlinux has quit IRC15:15
openstackgerritMerged openstack/tripleo-heat-templates master: Add DockerPuppetProcessCount defaults to 3  https://review.openstack.org/49813915:15
*** limao has quit IRC15:15
openstackgerritMerged openstack/tripleo-heat-templates stable/newton: Don't disable satellite repo after registration  https://review.openstack.org/49545215:17
openstackgerritMerged openstack/tripleo-heat-templates stable/newton: Workaround for RHEL registration as "localhost"  https://review.openstack.org/49545315:17
bnemeclvdombrkr: That would be a very uncommon configuration.  I think we used to do it in ci so it should work though.15:17
*** yprokule has quit IRC15:22
*** stee_3_ has joined #tripleo15:22
*** limao_ has quit IRC15:22
*** bugzy_ has joined #tripleo15:23
*** limao has joined #tripleo15:23
*** aufi has quit IRC15:25
*** stee_3 has quit IRC15:26
*** bugzy has quit IRC15:26
*** cdearborn has quit IRC15:26
*** trown|brb is now known as trown15:28
*** lucasagomes is now known as lucas-afk15:28
*** pdeore__ has quit IRC15:30
*** pdeore__ has joined #tripleo15:30
openstackgerritAlex Schultz proposed openstack/tripleo-heat-templates master: Add missing Docker service  https://review.openstack.org/49884215:31
*** rlandy is now known as rlandy|mtg15:33
*** limao has quit IRC15:34
*** limao has joined #tripleo15:34
*** iranzo has quit IRC15:36
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates master: WIP: Remove package if service stopped and disabled  https://review.openstack.org/47988615:37
*** mdnadeem has quit IRC15:37
lvdombrkrbnemec: thanks, unfortinetly it didnt work for me...i generete certeficate and key, place both in enable-tls.yaml than create .yaml with ControlFixedIPs15:38
*** beagles has left #tripleo15:38
bnemeclvdombrkr: What is failing?15:38
lvdombrkr bnemec: my deploy command looks like: openstack overcloud deploy --templates -e one.yaml -e tls-endpoints-public-ip.yaml -e /home/stack/ssl-heat-templates/environments/ssl/enable-tls.yaml -e   --compute-scale 115:38
lvdombrkrbnemec: and there is no luck15:39
lvdombrkrbnemec: nothing fails, i just cant reach dashboard by :44315:39
bnemeclvdombrkr: That may be because you aren't using network isolation so it isn't listening on a public address.15:41
bnemecWithout net-iso it only listens on the provisioning network, which frequently isn't routable.15:41
lvdombrkrbnemec: thanks. can i use such a simple network isalation .yaml only with sine parametrs: <bneme15:43
lvdombrkrbnemec: http://paste.openstack.org/raw/619745/15:43
bogdandofolks, I have an odd puppet augeas error making docker restart to not happen https://pastebin.com/SyLSkQQy15:45
*** limao_ has joined #tripleo15:45
bogdandoI wonder, what's wrong? the config file looks correct15:45
*** ffiore has quit IRC15:46
*** limao has quit IRC15:47
*** limao has joined #tripleo15:48
mwhahahabogdando: it's erroring on parsing the sysconfig file15:48
mwhahahabogdando: /etc/sysconfig/docker-storage has a parse error in it, what does yours look like?15:49
bogdandomwhahaha: DOCKER_STORAGE_OPTIONS=-s overlay215:49
bogdandobut I'm using standard packages :/15:49
*** limao_ has quit IRC15:49
mwhahahabogdando: so what added that, shouldn't it be quoted?15:49
bogdandomwhahaha: good question15:50
openstackgerritBen Nemec proposed openstack/tripleo-heat-templates master: Add ability to generate an environment index  https://review.openstack.org/49192515:50
openstackgerritBen Nemec proposed openstack/tripleo-heat-templates master: Begin adding environments with all params for a service  https://review.openstack.org/47592415:50
openstackgerritBen Nemec proposed openstack/tripleo-heat-templates master: Update generated ssl environments  https://review.openstack.org/49324515:50
openstackgerritBen Nemec proposed openstack/tripleo-heat-templates master: Convert enable-internal-tls.yaml to be generated  https://review.openstack.org/49324615:50
mwhahahaor do you know what added that15:50
bnemecWe need the two ssl-related changes for Pike. ^15:50
bnemecI just added a bug reference so we can backport them once they merge.15:50
bogdandomwhahaha: oh yeah, indeed. That's my user data script suxx15:51
bogdandothanks!!15:51
bnemeclvdombrkr: It is possible to deploy with just an external network, yes.15:51
mwhahahabogdando: np15:51
lvdombrkrls15:51
bnemeclvdombrkr: I have an example of doing that here: https://github.com/cybertron/tripleo-network-templates#simple15:51
bnemecThe actual files are in https://github.com/cybertron/tripleo-network-templates/tree/master/simple15:51
*** dtantsur|busy is now known as dtantsur|afk15:53
sshnaidmabishop, I think it would help to review your patches if you pasted your oooq run where this works. Because it's not tested in CI. Some pastebin in the patch that shows the successful run with your patch.15:55
*** ffiore has joined #tripleo15:56
*** limao has quit IRC15:56
*** ratailor has quit IRC15:56
*** limao has joined #tripleo15:57
*** rlandy|mtg is now known as rlandy16:01
*** marios has quit IRC16:02
*** marrusl has joined #tripleo16:03
lvdombrkrbnemec: thanks, i will try right now16:04
*** egonzalez has quit IRC16:06
*** limao has quit IRC16:07
*** limao has joined #tripleo16:08
*** dsneddon has joined #tripleo16:09
*** ooolpbot has joined #tripleo16:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION16:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168955616:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169140316:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295516:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171291016:10
openstackLaunchpad bug 1689556 in tripleo "CI: containers overcloud can't start vm when deployed with multinic network isolation" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)16:10
*** ooolpbot has quit IRC16:10
openstackLaunchpad bug 1691403 in tripleo "containerized overcloud - neutron-openvswitchagent fails to start" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)16:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel)16:10
openstackLaunchpad bug 1712910 in tripleo "oooq virt-customize repo setup logs are not saved " [Critical,In progress] - Assigned to wes hayutin (weshayutin)16:10
*** paramite has quit IRC16:12
*** tesseract has quit IRC16:15
*** thrash is now known as thrash|biab16:16
*** garyk has joined #tripleo16:16
*** fzdarsky is now known as fzdarsky|afk16:17
*** limao has quit IRC16:19
*** limao has joined #tripleo16:19
*** fzdarsky|afk has quit IRC16:22
*** itlinux has joined #tripleo16:23
openstackgerritwes hayutin proposed openstack/tripleo-quickstart-extras master: use pip to install delorean  https://review.openstack.org/49807416:23
openstackgerritEmilien Macchi proposed openstack/puppet-tripleo stable/newton: Release 5.6.2 (newton)  https://review.openstack.org/49885816:25
*** suuuper has quit IRC16:25
openstackgerritMichael  Henkel proposed openstack/os-net-config master: This patch adds initial support for the Contrail vRouter interface  https://review.openstack.org/49249216:26
*** achadha has joined #tripleo16:26
*** achadha has quit IRC16:27
*** achadha has joined #tripleo16:27
*** limao_ has joined #tripleo16:29
*** ykarel has quit IRC16:30
lvdombrkrbnemec: no luck...if i understood right PublicVirtualFixedIPs should ping after deployment despite any problems/mistakes with certeficates/keys?16:31
*** achadha_ has joined #tripleo16:31
*** achadha has quit IRC16:31
*** limao has quit IRC16:33
*** achadha_ has quit IRC16:33
*** achadha has joined #tripleo16:33
abishopsshnaidm: sure, I'll do that. Thanks for feedback16:35
*** achadha has quit IRC16:36
*** achadha has joined #tripleo16:36
*** achadha has quit IRC16:36
*** achadha has joined #tripleo16:37
*** gkadam_ has joined #tripleo16:37
*** achadha has quit IRC16:37
*** jpich has quit IRC16:38
*** achadha has joined #tripleo16:38
EmilienMsshnaidm: if something is not tested by CI, I'm not sure it's really fair to ask authors to test themselves and past results in Gerrit16:38
EmilienMsshnaidm: I think we want to automate all the things and ensure CI covers what we need16:38
sshnaidmEmilienM, I think anyway who prepare the patch should test it firstly16:40
sshnaidmEmilienM, it doesn't contradict of course that we need to cover as much as possible in CI16:40
*** mcornea has quit IRC16:40
*** milan has quit IRC16:43
*** pdeore__ has quit IRC16:44
mwhahahasshnaidm: yes they should test it, but then asking for verification because of a lack of CI seems odd.  If you're concerned about it, perhaps you should take the time to make sure it works correctly if that's something you want to see before you approve it.16:46
*** rcernin has quit IRC16:46
openstackgerritwes hayutin proposed openstack/tripleo-quickstart-extras master: do not use a custom scripts dir in dlrn builds  https://review.openstack.org/49886616:46
sshnaidmmwhahaha, this is very special case for re-running quickstart locally twice, I doubt it will be covered in CI, unfortunately not every use case it's possible to cover. I ask actually for people to do CI work so that will help to merge patches.16:48
sshnaidmmwhahaha, I don't insist, it's not a requirement, but I think it could be helpful16:48
*** itlinux has quit IRC16:48
*** pcaruana has quit IRC16:48
*** itlinux has joined #tripleo16:49
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates stable/pike: Add DockerPuppetProcessCount defaults to 3  https://review.openstack.org/49886716:49
*** ffiore has quit IRC16:49
bnemeclvdombrkr: Yes, the ip should exist regardless of what happens with the certificates.16:50
EmilienMdprince: doing backports for you on your docker patches, if I miss one please do it yourself16:50
dprinceEmilienM: cool, thanks man16:53
*** links has joined #tripleo16:54
*** jpena is now known as jpena|off16:59
*** hewbrocca is now known as hewbrocca_afk17:01
*** jfrancoa has quit IRC17:01
*** limao_ has quit IRC17:05
*** gbarros has quit IRC17:09
*** ramishra has quit IRC17:10
*** milan has joined #tripleo17:10
*** ooolpbot has joined #tripleo17:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION17:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168955617:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169140317:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295517:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171291017:10
*** ooolpbot has quit IRC17:10
openstackLaunchpad bug 1689556 in tripleo "CI: containers overcloud can't start vm when deployed with multinic network isolation" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)17:10
openstackLaunchpad bug 1691403 in tripleo "containerized overcloud - neutron-openvswitchagent fails to start" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)17:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel)17:10
openstackLaunchpad bug 1712910 in tripleo "oooq virt-customize repo setup logs are not saved " [Critical,In progress] - Assigned to wes hayutin (weshayutin)17:10
*** pkovar has quit IRC17:11
*** gbarros has joined #tripleo17:11
*** gbarros has quit IRC17:12
*** gkadam_ has quit IRC17:15
*** sid1 has joined #tripleo17:20
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Use Python to compute release notes version  https://review.openstack.org/49888217:21
*** trown is now known as trown|lunch17:26
EmilienMbnemec: thx for your review, I was wondering if I missed something in setup.cfg but it looks good: http://docs-draft.openstack.org/82/498882/1/check/gate-tripleo-heat-templates-releasenotes/2cd0973//releasenotes/build/html/17:26
bnemecEmilienM: Yeah, my review is obviously assuming ci passes. :-)17:27
dprinceEmilienM: as for backporting containers patches I think we probably do want to be careful not to add new containers at this point17:27
EmilienMbnemec: it does (for reno jobs)17:27
EmilienMdprince: which ones are missing?17:27
dprinceEmilienM: the cron-container was newish, and will now require some work to get it built downstream17:28
dprinceEmilienM: cron-container was used by logrotate17:28
EmilienMdprince: ack for this one17:28
EmilienMdprince: anything else?17:28
EmilienMdprince: https://bugs.launchpad.net/tripleo/+bug/1713612 - I would like this one fixed17:28
openstackLaunchpad bug 1713612 in tripleo "When neutron is containerized, No 'neutron.service_plugins' driver found" [High,Triaged]17:28
EmilienMrnoriega: ^ any progress?17:28
EmilienMbogdando: FYI about what dprince said17:29
dprinceEmilienM: so long as neutron patch doesn't require a net-new container I think it would be a backport candidate17:29
EmilienMdprince: I hope not17:31
bogdandodprince, EmilienM: ack17:31
bogdandoI'm not really know much of the downstream part of the things :(17:31
bogdandobut we definitely need logrotate backported for pike17:32
bogdando(but I hoped it made it to the pike, didn't it?)17:33
EmilienM(I don't know)17:33
*** yamahata has quit IRC17:33
EmilienMdprince: I think yes we need a new container17:33
EmilienMdprince: because the bgp / l2gw things are external of neutron17:33
EmilienMthey are neutron-l2gw something17:33
EmilienMlet me check if we already have containers for them17:33
dprinceEmilienM: then, I would suggest not backporting these fixes then17:34
dprinceEmilienM: and just target them towards Queens17:34
*** links has quit IRC17:34
EmilienMdprince: ack, I'll warn neutron folks17:35
dprinceEmilienM: should be no surprise I think. It is just a bit late to be backporting net-new containers at this point I think17:36
EmilienMyes I agree17:37
*** sid1 has quit IRC17:37
EmilienMdprince: I figured bgp/l2gw wasn't working in containers when enabling tests on scenario00417:37
openstackgerritBen Nemec proposed openstack/tripleo-validations master: Lazy load the entire stack instead of single outputs  https://review.openstack.org/49888417:39
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Use Python to compute release notes version  https://review.openstack.org/49888217:47
*** catintheroof has quit IRC17:49
*** tongl has joined #tripleo17:50
*** thrash|biab is now known as thrash17:52
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Use Python to compute release notes version  https://review.openstack.org/49888217:56
*** bogdando has quit IRC17:57
*** yamahata has joined #tripleo18:09
*** ooolpbot has joined #tripleo18:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION18:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168955618:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169140318:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295518:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171291018:10
openstackLaunchpad bug 1689556 in tripleo "CI: containers overcloud can't start vm when deployed with multinic network isolation" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)18:10
*** ooolpbot has quit IRC18:10
openstackLaunchpad bug 1691403 in tripleo "containerized overcloud - neutron-openvswitchagent fails to start" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)18:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel)18:10
openstackLaunchpad bug 1712910 in tripleo "oooq virt-customize repo setup logs are not saved " [Critical,In progress] - Assigned to wes hayutin (weshayutin)18:10
openstackgerritRonelle Landy proposed openstack/tripleo-common master: DO NOT MERGE - Add no-op change for containers update testing  https://review.openstack.org/49889818:12
*** iranzo has joined #tripleo18:15
*** iranzo has quit IRC18:15
*** iranzo has joined #tripleo18:15
*** liverpooler has joined #tripleo18:15
*** catintheroof has joined #tripleo18:23
*** cschwede has quit IRC18:29
EmilienMbnemec, mwhahaha : https://review.openstack.org/#/c/498882/ is ready for review again18:31
pabelangerEmilienM: weshay: sshnaidm: what is the status of 498074? Are we okay with moving that forward today? Another gate failure on github.com: http://logs.openstack.org/40/498440/2/gate/gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container/e7dd2c4/console.html#_2017-08-29_17_53_58_45389118:32
weshaypabelanger, I've addressed sshnaidm's concerns, I was looking at that yesterday in recheck, and I only saw one failure in the last 30 days18:33
*** milan has quit IRC18:33
weshayah.. a few from today18:34
weshaypabelanger, so I have two patches now18:34
EmilienMin a mtg, bbl18:34
weshaypabelanger, two reviews now https://review.openstack.org/#/c/498866/18:35
pabelangergreat18:35
*** trown|lunch is now known as trown18:36
pabelangerartg_rdoinfo_repo_url: "https://github.com/redhat-openstack/rdoinfo" is still a potential issue18:36
pabelangerbut I think we need to have RDO publish that to https://trunk.rdoproject.org so we can use the reverse proxy cache18:36
weshayrlandy, Slower https://trunk.rdoproject.org/centos7-master/c3/f6/c3f6ee141b463c781df9beade10a7d4086ce1968_548cd9ec/18:36
*** dprince has quit IRC18:36
weshaypabelanger, agree18:45
pabelangeranybody better at mistral? http://logs.openstack.org/08/498308/1/gate/gate-tripleo-ci-centos-7-containers-multinode/f0b31ec/logs/undercloud/home/jenkins/overcloud_deploy.log.txt.gz just failed in the gate, but I have no idea what that means18:48
*** iranzo has quit IRC18:54
mwhahahapabelanger: that's a mighty fine Unexpected error you have there18:56
*** gbarros has joined #tripleo18:56
openstackgerritVictoria Martinez de la Cruz proposed openstack/tripleo-heat-templates master: Add storage backends env files for containerized deployment  https://review.openstack.org/49537018:56
mwhahahapabelanger: zaqar failed to put stuff in swift it looks like http://logs.openstack.org/08/498308/1/gate/gate-tripleo-ci-centos-7-containers-multinode/f0b31ec/logs/undercloud/var/log/zaqar/zaqar-server.log.txt.gz#_2017-08-29_18_33_21_13418:57
mwhahahaand of course we're not capturing swift logs18:57
tonglHi, I have a quick question. How to change the overcloud neutron core_plugins in yaml for quickstart?18:58
mwhahahamaybe they are in messages18:58
*** jtomasek_ has quit IRC18:59
tonglI am trying to setup a test env with other neutron plugins.18:59
*** dobson has quit IRC18:59
*** jobewan has joined #tripleo19:00
*** jobewan has quit IRC19:00
*** dparkes has joined #tripleo19:01
*** dobson has joined #tripleo19:03
*** achadha has quit IRC19:04
*** achadha has joined #tripleo19:05
mwhahahatongl: changing the core_plugins would traditionally be something done via THT, the question i guess would be how do you include custom THT templates to the deploy command via quickstart19:05
*** limao has joined #tripleo19:08
tonglyes, should we include it in the yaml config file? Is there any sample yaml config that uses thirdparty neutron plugin?19:08
*** achadha has quit IRC19:09
mwhahahatongl: i don't know, i'm trying to figure that out myself19:09
*** dprince has joined #tripleo19:09
*** ooolpbot has joined #tripleo19:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION19:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168955619:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169140319:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295519:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171291019:10
openstackLaunchpad bug 1689556 in tripleo "CI: containers overcloud can't start vm when deployed with multinic network isolation" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)19:10
*** ooolpbot has quit IRC19:10
openstackLaunchpad bug 1691403 in tripleo "containerized overcloud - neutron-openvswitchagent fails to start" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)19:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel)19:10
openstackLaunchpad bug 1712910 in tripleo "oooq virt-customize repo setup logs are not saved " [Critical,In progress] - Assigned to wes hayutin (weshayutin)19:10
tonglmwhahaha: thanks. we added our neutron plugin in pike release and would like to validate if everything works fine.19:10
*** hjensas has joined #tripleo19:12
*** hjensas has quit IRC19:12
*** limao has quit IRC19:12
mwhahahatongl: so quickstart offers a few config options that lets you specify additional -e files during the deploy, example https://github.com/openstack/tripleo-quickstart/blob/master/config/general_config/featureset031.yml#L4219:14
mwhahahatongl: the question (which i don't know the answer to) is how do you provide those when you run quickstart so they get copied to the undercloud when you want to deploy19:15
*** ebarrera has quit IRC19:15
mwhahahatongl: effectively the examples i found seem to only use already existing files from either the tripleo-heat-template repo or tripleo-ci19:15
mwhahahaweshay, trown, sshnaidm: do you have an answer as to how to provide your own THT yaml to quickstart so it gets copied (and applied) -^ i couldn't find examples19:16
*** Goneri has quit IRC19:17
*** garyk has quit IRC19:17
tonglmwhahaha: Yeh, trying to see how can I provide yaml so that undercloud can deploy overcloud with thirdparty neutron plugin and customized configs.19:19
*** achadha has joined #tripleo19:22
*** achadha has quit IRC19:26
*** gfidente is now known as gfidente|afk19:27
*** catintheroof has quit IRC19:30
*** achadha has joined #tripleo19:31
*** dprince has quit IRC19:32
*** catintheroof has joined #tripleo19:36
weshaymwhahaha, let me know if this helps https://github.com/openstack/tripleo-quickstart-extras/tree/master/roles/overcloud-prep-config19:38
mwhahahaweshay: not really19:39
mwhahahaweshay: so is there not a way to specify 'copy this file over' via quickstart?19:39
*** pchavva has quit IRC19:41
weshayhttps://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/overcloud-prep-config/templates/overcloud-custom-tht-script.sh.j219:41
mwhahahaoh there it is extra_tht_configs19:41
mwhahahatongl: extra_tht_configs is what you would define witha list of yaml files which should get included in the deploy19:43
*** jprovazn has quit IRC19:43
mwhahahatongl: https://github.com/openstack/tripleo-heat-templates/blob/f3e084dc85c0d95fadd078c6297a40136779ee5c/environments/networking/neutron-midonet.yaml#L50 is ane xample how you can set the core_plugins via THT19:43
openstackgerritAndy Smith proposed openstack/tripleo-heat-templates master: OpenStack containerized qdrouterd service  https://review.openstack.org/47904919:48
*** dprince has joined #tripleo19:51
weshayarxcruz, there are two failing tempest tests on pike FYI19:52
arxcruzweshay: where's the logs?19:53
weshayarxcruz, http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-nonha-tempest-oooq-pike/25b6333/logs/stackviz/#/stdin19:54
*** japestinho has quit IRC19:54
*** iranzo has joined #tripleo19:54
*** iranzo has quit IRC19:54
*** iranzo has joined #tripleo19:54
*** Goneri has joined #tripleo20:08
tonglmwhahaha: awesome. thanks. I will have a look at it.20:09
*** ooolpbot has joined #tripleo20:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION20:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168955620:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169140320:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295520:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171291020:10
openstackLaunchpad bug 1689556 in tripleo "CI: containers overcloud can't start vm when deployed with multinic network isolation" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)20:10
*** ooolpbot has quit IRC20:10
openstackLaunchpad bug 1691403 in tripleo "containerized overcloud - neutron-openvswitchagent fails to start" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)20:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel)20:10
openstackLaunchpad bug 1712910 in tripleo "oooq virt-customize repo setup logs are not saved " [Critical,In progress] - Assigned to wes hayutin (weshayutin)20:10
*** sid1 has joined #tripleo20:11
*** eck` is now known as eck`gone20:12
openstackgerritJustin Kilpatrick proposed openstack/tripleo-quickstart-extras master: WIP: Disruption detection and some stack update roles  https://review.openstack.org/49795020:14
pabelangermwhahaha: okay, I'll check logstash.o.o and see if that has happen before20:15
openstackgerritThomas Herve proposed openstack/tripleo-common master: WIP Make some tweaks to swift logging  https://review.openstack.org/49893020:20
*** iranzo has quit IRC20:24
*** iranzo has joined #tripleo20:25
*** iranzo has joined #tripleo20:25
*** jkilpatr has quit IRC20:28
*** iranzo has quit IRC20:31
*** ansmith has quit IRC20:33
pabelangermwhahaha: another failure in the gate: http://logs.openstack.org/96/498296/1/gate/gate-tripleo-ci-centos-7-scenario002-multinode-oooq-container/ebc5774/logs/undercloud/var/log/zaqar/zaqar-server.log.txt.gz20:35
mwhahahapabelanger: seems like an ordering problem cause i looked in /var/log/messages and the post for that container is after the failure20:36
*** ebarrera has joined #tripleo20:36
mwhahahapabelanger:  seems like an intermittent zaqar/mistral issue20:36
pabelangermwhahaha: ya, going to setup an elastic-recheck query if I can find the pattern20:36
stevebakermorning20:37
stevebakersshnaidm: hi20:37
stevebakerEmilienM: anything you would like me to look at?20:38
openstackgerritAna Krivokapic proposed openstack/tripleo-common master: Add workflow that wraps register, introspect and provide  https://review.openstack.org/49893620:38
*** leitan has quit IRC20:46
*** leitan has joined #tripleo20:47
*** jkilpatr has joined #tripleo20:47
*** leitan has quit IRC20:47
*** leitan has joined #tripleo20:47
openstackgerritBen Nemec proposed openstack/tripleo-validations master: Prevent unnecessary calls to dynamic inventory host()  https://review.openstack.org/49893920:47
*** leitan has quit IRC20:47
*** leitan has joined #tripleo20:48
*** leitan has quit IRC20:52
*** eck`gone is now known as eck`20:53
*** florianf has quit IRC20:59
*** ecerquei has quit IRC21:00
*** trown is now known as trown|outtypewww21:02
*** gfidente|afk has quit IRC21:06
arxcruzweshay: one of the tests are legitm failing, but the luks one should not be running, because the job doesn't have encrypt enabled21:07
*** lblanchard has quit IRC21:07
*** ooolpbot has joined #tripleo21:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION21:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168955621:10
openstackLaunchpad bug 1689556 in tripleo "CI: containers overcloud can't start vm when deployed with multinic network isolation" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)21:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169140321:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295521:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171291021:10
*** ooolpbot has quit IRC21:10
openstackLaunchpad bug 1691403 in tripleo "containerized overcloud - neutron-openvswitchagent fails to start" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)21:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel)21:10
openstackLaunchpad bug 1712910 in tripleo "oooq virt-customize repo setup logs are not saved " [Critical,In progress] - Assigned to wes hayutin (weshayutin)21:10
*** rhallisey has quit IRC21:12
*** artom has quit IRC21:13
pabelanger3rd failure: http://logs.openstack.org/93/489393/9/gate/gate-tripleo-ci-centos-7-scenario002-multinode-oooq-container/043a4f5/logs/undercloud/var/log/zaqar/zaqar-server.log.txt.gz21:14
pabelangerhitting it pretty often now in gate21:14
mwhahahapabelanger: please file a bug with alert ci tags21:17
bnemecWow, 48 patches in the tripleo gate queue.  That's...impressive.21:18
*** jcoufal has quit IRC21:18
toskylike https://images-na.ssl-images-amazon.com/images/I/41m3ebMv5XL.jpg ?21:19
tosky:)21:19
*** abishop has quit IRC21:19
*** sid1 has quit IRC21:20
openstackgerritSteve Baker proposed openstack/tripleo-quickstart master: enable pingtest on fs019  https://review.openstack.org/49857521:25
pabelangermwhahaha: message:"u'subscription_create" AND message:"u'Object PUT failed" AND tags:"console" AND voting:121:25
pabelangeris the query21:25
pabelanger17 hits over 10 days21:26
pabelangermwhahaha: what would you like the bug to be? I'm not sure what is logic is doing :)21:26
*** ansmith has joined #tripleo21:27
*** jlabarre has quit IRC21:27
mwhahahapabelanger: me neither so i usually just throw what i've figured out into a bug and let someone else RCA it up. it looks to be that zaqar is erroring when trying to post status to a swift container or something21:27
pabelangerk21:27
mwhahahapabelanger: is it just the scenario002 containers jobs?21:28
pabelangerno, it is a mix21:28
*** akrivoka has quit IRC21:29
*** morazi has quit IRC21:29
mwhahahaok yea just throw what you've found in the bug. it seems to be more of a downstream mistral/zaqar ordering issue21:29
mwhahahabut you never know21:29
mwhahahai think it happens when we're doing introspection21:30
pabelangerbug 171383221:30
openstackbug 1713832 in tripleo "Object PUT failed for zaqar_subscription" [Undecided,New] https://launchpad.net/bugs/171383221:30
pabelangerif you want to triage it21:30
pabelangercreating elastic-recheck now21:30
mwhahahak21:31
* mwhahaha adds it to the list o things to try and do today21:31
*** catintheroof has quit IRC21:34
pabelangermwhahaha: so, the bad thing here is, we've been seeing this issue in check since http://logs.openstack.org/43/496243/1/check/gate-tripleo-ci-centos-7-scenario004-multinode-oooq-container/0a075c5/console.html.gz but we didn't properly debug the failure. 2017-08-24 08:54:5921:37
*** dprince has quit IRC21:38
EmilienMstevebaker: https://review.openstack.org/#/c/498575/ will fail because of https://bugs.launchpad.net/bugs/171361221:41
openstackLaunchpad bug 1713612 in tripleo "Containers missing for Neutron L2GW and BPGVPN-API" [High,Triaged] - Assigned to Ricardo Noriega (rnoriega-b)21:41
arxcruzweshay: this should fix the tests https://review.rdoproject.org/r/#/c/9070/21:42
*** artom has joined #tripleo21:42
*** artom has quit IRC21:42
*** artom has joined #tripleo21:42
*** tbarron is now known as tbarron|errand21:46
*** pradk has quit IRC21:46
stevebakerEmilienM: ok, it sounds like we should align upstream with downstream and not containerise neutron21:47
EmilienMstevebaker: neutron-server you mean?21:49
EmilienMstevebaker: or just disable these 2 services on scenario004-container21:50
EmilienMfor pike. It should be fine as well21:50
EmilienMtherve: if you around, we need your help on https://bugs.launchpad.net/tripleo/+bug/171383221:50
openstackLaunchpad bug 1713832 in tripleo "Object PUT failed for zaqar_subscription" [Critical,Triaged]21:50
* EmilienM realizes it's super late in europe21:51
weshaylong zuul queues21:51
EmilienMmwhahaha, weshay: what are the patches we need RIGHT NOW for CI?21:51
EmilienMI'm going to kill the zuul queue21:51
EmilienMhttps://review.openstack.org/#/c/498772/ for sure21:52
weshayyes21:52
weshaythat one21:52
weshayhttps://review.openstack.org/#/c/498688/21:52
EmilienMweshay: that's all?21:52
weshaylooking21:52
EmilienMweshay: that one is critical?21:53
EmilienMweshay: I don't care about promotion now21:53
weshaywould help the container effort21:53
EmilienMweshay: I don't want to help anyone now, I want to see how we can bring the gate back21:53
EmilienMsounds like the zaqar issue and this patch https://review.openstack.org/#/c/498772/ are needed21:53
weshaya couple more... sec21:53
EmilienMok21:53
EmilienMtell me because I'm about to kill the gate21:54
EmilienMmwhahaha: any thoughts before I proceed?21:54
weshayEmilienM, https://review.openstack.org/#/c/498074/21:54
EmilienMprobably too late for sshnaidm to review21:55
EmilienMweshay: but it sounds like it worked on his test patch21:56
EmilienMweshay: how many hits do we have for this one? https://review.openstack.org/#/c/498074/21:56
weshayEmilienM, I have a follow up that address his concerns21:56
EmilienMin logstash21:56
EmilienMnot as many iirc21:56
weshayEmilienM, paul pinged me today21:56
EmilienMok21:56
*** ecerquei has joined #tripleo21:56
EmilienMmwhahaha: please review https://review.openstack.org/#/c/498074/ when you can21:56
weshay<pabelanger> EmilienM: weshay: sshnaidm: what is the status of 498074? Are we okay with moving that forward today? Another gate failure on github.com: http://logs.openstack.org/40/498440/2/gate/gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container/e7dd2c4/console.html#_2017-08-29_17_53_58_45389121:56
stevebakerEmilienM, weshay: I think it worked, not sure about that stderr though http://logs.openstack.org/12/498712/1/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha-oooq/6516c0f/logs/ara_oooq/result/9621bc1f-97f3-46c2-a314-377b353409de/21:57
EmilienMstevebaker: which one?21:57
stevebaker" packages not built correctly: not updating the consistent symlink"21:57
weshaystevebaker, look at a current build, and you'll see the same21:57
weshaywas going to follow up w/ that too21:57
weshayyou'll see that in all the jobs atm21:57
stevebakerlooks like dlrn runs though, so I think the pip install works21:57
weshayEmilienM, sshnaidm wanted this https://review.openstack.org/#/c/498866/121:58
stevebakerhttp://logs.openstack.org/12/498712/1/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha-oooq/6516c0f/logs/delorean_logs/e8/38/e838261c783eb7743d4999df5fd8284081dffb38_dev/21:58
stevebaker ^^ dlrn logs from sshnaidm's trigger21:59
EmilienMweshay: I'm asking for critical issues we want merged ASAP like now21:59
EmilienMweshay: is https://review.openstack.org/#/c/498866 part of it?21:59
EmilienMthere is no bug in the commit message so I think no21:59
weshayno21:59
EmilienMok good21:59
weshayjust referencing sshnaidm's objection21:59
*** jmelvin has quit IRC21:59
EmilienMso please (again) give me link of things we need merged now21:59
EmilienMand we need some help on https://bugs.launchpad.net/tripleo/+bug/171383222:00
openstackLaunchpad bug 1713832 in tripleo "Object PUT failed for zaqar_subscription" [Critical,Triaged]22:00
weshayso what is holding up the gates?22:00
mwhahahacontainer-oooq fails and then this zaqar fail thing22:00
mwhahahaand also possibly dlrn fails22:00
EmilienMmwhahaha: container-oooq ?22:00
EmilienMwhat exactly?22:01
mwhahahathe remove builds from multinode22:01
EmilienMah indeed22:01
EmilienMso let's focus on these 3 issues22:01
weshayya.. that should merge soon ISH22:01
EmilienMand I'm going to kill all other patches in the gate22:01
EmilienM@all please do not approve any patch22:01
weshayremove builds from multinode is here: https://review.openstack.org/#/c/498688/22:02
EmilienMhttps://review.openstack.org/#/c/498772/ just failed in gate22:02
stevebakerEmilienM: could https://review.openstack.org/#/c/493695/ be next after the critical ones?22:02
EmilienMstevebaker: tbh I don't know right now we're trying to solve gate issue, so we can keep merging stuffs22:03
EmilienMstevebaker: because right now we don't merge anything22:03
EmilienMlast patch which merged was 7h ago22:03
EmilienMeverything between failed in gate22:04
weshayEmilienM, did tempest get enabled on the container-multinode job?22:05
* weshay looks22:06
EmilienMweshay: yes, one of them the main one22:07
weshayya.. it worked, just looking at a very odd console log22:08
EmilienMok gate is purged22:09
*** Goneri has quit IRC22:10
*** ooolpbot has joined #tripleo22:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION22:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168955622:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169140322:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295522:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171291022:10
openstackLaunchpad bug 1689556 in tripleo "CI: containers overcloud can't start vm when deployed with multinic network isolation" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)22:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171383222:10
*** ooolpbot has quit IRC22:10
openstackLaunchpad bug 1691403 in tripleo "containerized overcloud - neutron-openvswitchagent fails to start" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)22:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel)22:10
openstackLaunchpad bug 1712910 in tripleo "oooq virt-customize repo setup logs are not saved " [Critical,In progress] - Assigned to wes hayutin (weshayutin)22:10
openstackLaunchpad bug 1713832 in tripleo "Object PUT failed for zaqar_subscription" [Critical,Triaged]22:10
weshaylooking for other jobs failing like http://logs.openstack.org/72/498772/2/gate/gate-tripleo-ci-centos-7-containers-multinode/701b480/console.html22:12
*** Goneri has joined #tripleo22:13
*** itlinux has quit IRC22:14
weshayEmilienM, I don't know why https://review.openstack.org/#/c/498772/ failed.. recheck?22:14
EmilienMweshay: I'm trying to get it promoted to the gate on #openstack-infra22:16
EmilienMcan someone look at the zaqar thing?22:16
weshayk.. it just timed out22:16
EmilienMdid we merge something that broke us?22:16
EmilienMok 498772 promoted to gate now22:17
EmilienMI'm starting to investigate the zaqar problem22:17
EmilienMyeah, like pabelanger we have the issue since a bunch of days now :(22:19
EmilienMit could have happened in a promotion22:19
*** Goneri has quit IRC22:20
mwhahahait's probably a promotion22:20
mwhahahaEmilienM: it looked like it's trying to read the queue before it gets created in swift22:20
mwhahahabut i don't know the order in which that stuff happens so it's all magic to me22:20
pabelangerdid we just abandon a bunch of things?22:23
mwhahahayea22:23
EmilienMpabelanger: I'm clearing the gate22:24
EmilienMpabelanger: so we keep focus on 3 problems:22:24
EmilienM1) getting https://review.openstack.org/498772 merged22:24
EmilienM2) https://bugs.launchpad.net/tripleo/+bug/171383222:25
openstackLaunchpad bug 1713832 in tripleo "Object PUT failed for zaqar_subscription" [Critical,Triaged]22:25
EmilienMand 3) https://review.openstack.org/49807422:25
EmilienMpabelanger: could you promote 498074 to the gate as well please, I just approved this patch22:25
pabelanger1 sec22:26
EmilienMpabelanger: 498772 was promoted by clarkb afik22:26
EmilienMok it's in gate now22:26
EmilienMpabelanger: nevermind22:26
*** dsneddon has quit IRC22:26
*** tongl has quit IRC22:29
EmilienMmwhahaha: first time zaqar issue appeared was Aug 2122:31
EmilienMmwhahaha: http://logs.openstack.org/59/471759/65/check/gate-tripleo-ci-centos-7-3nodes-multinode-nv/e045834/console.html#_2017-08-21_12_19_25_50997522:32
EmilienMthis one right?22:32
mwhahahayea22:32
EmilienMk22:32
pabelangerEmilienM: https://review.openstack.org/498954/ is the elastic-recheck query22:33
EmilienMI nkow22:33
EmilienMbut thanks22:33
EmilienMlet's see what we changed between August 20 and 2222:34
*** achadha_ has joined #tripleo22:35
mwhahahaEmilienM: since this is the validation workflow, did we update that then?22:36
*** morazi has joined #tripleo22:37
EmilienMmwhahaha: I'm doing rpm diffs and looking22:37
EmilienMhttps://www.diffchecker.com/JvMExY0X22:37
EmilienMmaybe https://github.com/openstack/instack-undercloud/commit/3b4d191d986ee6a7f92e3314798057e58cf17dd922:38
EmilienMbut not sure at all, I'm guessing at this stage22:38
*** thrash is now known as thrash|g0ne22:38
EmilienMis it always the same workflow?22:38
*** achadha has quit IRC22:38
*** rlandy is now known as rlandy|bbl22:38
EmilienMyes22:39
EmilienMStarted Mistral Workflow tripleo.validations.v1.check_pre_deployment_validations. Execution ID: ebb5bed9-fae1-4e61-8617-3fb425527ffa22:39
EmilienMalways this one22:39
*** achadha_ has quit IRC22:39
mwhahaharevert it22:41
EmilienMok22:42
EmilienMI think that's the one22:42
EmilienMbut I'm really unsure22:42
EmilienMmwhahaha: is the error random?22:42
mwhahahayes22:42
EmilienMprobably yes since gate is not 100% broken22:42
mwhahahait's a race condition22:42
EmilienMkk22:42
EmilienMrevert22:42
mwhahahait's trying to query the results before the swift container gets created in zaqar22:42
openstackgerritEmilien Macchi proposed openstack/instack-undercloud master: Revert "Adds execution of post-upgrade validations on undercloud upgrade"  https://review.openstack.org/49897522:43
openstackgerritEmilien Macchi proposed openstack/instack-undercloud master: Revert "Adds execution of post-upgrade validations on undercloud upgrade"  https://review.openstack.org/49897522:43
EmilienMok done ^22:44
EmilienMmwhahaha: https://review.openstack.org/#/c/498975/22:45
EmilienMpabelanger: my last request, promoting ^ to the gate please22:46
EmilienMpabelanger: https://review.openstack.org/#/c/498772/ is failing again22:48
*** dmarlin has quit IRC22:49
EmilienMhttp://logs.openstack.org/72/498772/2/gate/gate-tripleo-ci-centos-7-containers-multinode/3d94563/console.html#_2017-08-29_22_43_24_37351722:49
EmilienMis it a new one?22:49
stevebakerEmilienM: do you know what the story is with this missing ovs_vxlan_bridge command? http://logs.openstack.org/72/498772/2/gate/gate-tripleo-ci-centos-7-undercloud-oooq/a8af258/logs/undercloud/home/jenkins/vxlan_networking.sh.log.txt.gz22:49
EmilienMstevebaker: sounds like we're talking about the same thing now22:50
pabelangerEmilienM: haven't seen that before22:50
EmilienMpabelanger: http://logs.openstack.org/72/498772/2/gate/gate-tripleo-ci-centos-7-containers-multinode/3d94563/logs/undercloud/home/jenkins/vxlan_networking.sh.log.txt.gz#_2017-08-29_22_43_2422:50
EmilienM2017-08-29 22:43:24 | /home/jenkins/vxlan_networking.sh: line 91: ovs_vxlan_bridge: command not found22:50
openstackgerritBob Fournier proposed openstack/tripleo-common master: Accept glance image ID in addition to name  https://review.openstack.org/49897822:51
pabelangerya, first time I seen that22:51
EmilienMI'm pretty sure it's a change in devstack-gate22:51
EmilienMthat just merged22:51
pabelangerOh, ya.22:51
pabelangerwe just changed something22:51
EmilienMhttps://review.openstack.org/43593322:51
EmilienMand broke us22:51
pabelangerlets move to #openstack-infra22:51
EmilienMso now we have a 4th issue22:51
*** mwhahaha sets mode: +o EmilienM22:54
*** EmilienM changes topic to "CI Status: Red - please do not approve any patch"22:55
*** achadha has joined #tripleo22:58
*** cdearborn_ has quit IRC23:03
EmilienMpabelanger: I'm filling a bug for the 4th issue23:05
EmilienMso we can have alerts here23:05
EmilienMpabelanger: https://bugs.launchpad.net/tripleo/+bug/171386823:08
openstackLaunchpad bug 1713868 in tripleo "devstack-gate doesn't provide ovs_vxlan_bridge function anymore" [Critical,Triaged] - Assigned to Paul Belanger (pabelanger)23:08
EmilienMpabelanger: instead of using the function, would it be simpler to re-use the same bash code23:10
*** ooolpbot has joined #tripleo23:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION23:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168955623:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169140323:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295523:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171291023:10
openstackLaunchpad bug 1689556 in tripleo "CI: containers overcloud can't start vm when deployed with multinic network isolation" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)23:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171383223:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171386823:10
*** ooolpbot has quit IRC23:10
openstackLaunchpad bug 1691403 in tripleo "containerized overcloud - neutron-openvswitchagent fails to start" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)23:10
EmilienMpabelanger: until we switch to an ansible task?23:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel)23:10
openstackLaunchpad bug 1712910 in tripleo "oooq virt-customize repo setup logs are not saved " [Critical,In progress] - Assigned to wes hayutin (weshayutin)23:10
openstackLaunchpad bug 1713832 in tripleo "Object PUT failed for zaqar_subscription" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)23:10
openstackLaunchpad bug 1713868 in tripleo "devstack-gate doesn't provide ovs_vxlan_bridge function anymore" [Critical,Triaged] - Assigned to Paul Belanger (pabelanger)23:10
EmilienMweshay: removing alert on https://bugs.launchpad.net/tripleo/+bug/1713868 - no activity since more than a month23:10
EmilienMnot this one23:10
EmilienMhttps://bugs.launchpad.net/tripleo/+bug/170295523:11
weshayk23:11
mwhahahait's still a problem23:12
mwhahahajust no one seems to have time to fix it23:13
* mwhahaha shrugs23:13
openstackgerritPaul Belanger proposed openstack-infra/tripleo-ci master: In-tree ovs_vxlan_bridge function from devstack-gate  https://review.openstack.org/49898523:13
pabelangerEmilienM: ^ should be our fix23:14
mwhahahaif only there was a code search folks to use to determine if things were being used outside of their project before removing them completely23:15
* mwhahaha sighs and goes to flip tables elsewhere23:15
*** tosky has quit IRC23:15
EmilienMweshay: please assign https://bugs.launchpad.net/tripleo/+bug/1708832 to yourself23:16
openstackLaunchpad bug 1708832 in tripleo "DLRN build failures in gate" [High,In progress]23:16
weshaydone23:16
EmilienMweshay: thank you23:17
EmilienMpabelanger: reviewing23:18
pabelangerEmilienM: hold up, I think swap is broken too23:18
EmilienM:(23:19
pabelangerokay, I'll have to abandon, restore23:21
pabelangerbut want to make sure vxlan is setup before I do that23:21
pabelangerif we pass that, we can likey enqueue right into gate23:21
EmilienMok23:22
EmilienMand swap?23:22
pabelangerhttp://status.openstack.org/elastic-recheck/#170610323:22
pabelangerjust bad luck23:22
pabelangerknown issue23:22
EmilienMah ok23:22
EmilienMI thought it was something in tripleo again23:22
EmilienM:)23:22
*** jeblair has joined #tripleo23:22
EmilienMjeblair: o/23:23
*** yamahata has quit IRC23:34
openstackgerritPaul Belanger proposed openstack/tripleo-quickstart-extras master: In-tree function ovs_vxlan_bridge  https://review.openstack.org/49898623:35
pabelangerEmilienM: okay, ^ is the other place it needs to be fixed23:35
*** lblanchard has joined #tripleo23:35
EmilienMpabelanger: ok23:36
EmilienMpabelanger: if we need it at multiple places, the patches won't land I guess23:37
EmilienMwell or maybe23:37
EmilienMsince we run all gate jobs in oooq now23:37
pabelangerEmilienM: ya, possible something will need to be force merged to unwedge23:38
EmilienMpabelanger: how can we do that? infra roots?23:38
pabelangerlets get 498986 green in check first23:38
EmilienMk23:38
pabelangerEmilienM: ya, I can force merge, just need to sudo in gerrit23:38
*** yamahata has joined #tripleo23:40
EmilienMok23:40
EmilienMpabelanger: it would really help to force merge https://review.openstack.org/#/c/498772/23:41
EmilienMweshay: can you remind me, https://review.openstack.org/#/c/498772/ is fixing the instack-undercloud gate, right?23:41
EmilienMpabelanger: https://review.openstack.org/#/c/498074/ as well23:42
EmilienMpabelanger: and https://review.openstack.org/#/c/498975/ - this is a revert23:42
pabelangerEmilienM: I'm confident that is going to pass, so once we fix networking, I can enqueue / promote again, and you can start +A again23:42
EmilienMI'm pretty sure it's safe to force merge at this stage23:42
pabelangerwell, lets not force merge until it is needed. I don't really like doing that unless gate is wedged23:43
pabelangereverything should pass, with proper depends-On headers23:43
EmilienMpabelanger: gats is really wedge right now23:43
EmilienMpabelanger: I'm not sure it will pass23:43
EmilienMit's a spaguetti23:43
EmilienMbut as you wish...23:43
pabelangerright, we need to make sure 498986 is good23:43
pabelangerbecause it is untested right now23:44
pabelangerso, as long as something sets up vxlan, then we can +A, promote and enqueue23:44
EmilienMpabelanger: I didn't ask to force merge this one23:44
pabelangerthen, we can do same with other patches23:44
EmilienMI asked https://review.openstack.org/#/c/498772/ https://review.openstack.org/#/c/498975/ and https://review.openstack.org/#/c/498074/23:44
pabelangerright, there is no need to force merge those, they don't wedge the gate23:45
weshayEmilienM, several of the gates, yes23:45
weshaybut not all are affected23:45
EmilienMweshay: k23:45
weshayit's a must have atm23:46
weshaythat is the top one from us23:46
EmilienMpabelanger: it would save time to force merge them imho23:46
EmilienMso we would just have to focus on the d-g thing23:46
EmilienMand we wouldn't hit the 3 other problems23:46
pabelangerEmilienM: once they are enqueue into gate, anything behind them will work23:46
pabelangerwe just need to make sure 498986,1 is good23:47
EmilienMk23:47
EmilienMpabelanger: before you leave for your end of day, please let me know23:48
EmilienM(I know it's late for you)23:48
pabelangerEmilienM: don't worry, I am here until this is all fixed23:48
EmilienMthanks ...23:49
pabelangerblarg23:51
pabelangerI see an issue23:51
pabelanger1 sec23:51
openstackgerritPaul Belanger proposed openstack/tripleo-quickstart-extras master: In-tree function ovs_vxlan_bridge  https://review.openstack.org/49898623:52
EmilienMok still lgtm23:54
pabelangerhttps://review.rdoproject.org/jenkins/job/gate-tripleo-ci-centos-7-undercloud-oooq-master-nv/231/console23:55
pabelangerthat is surprisingly fast23:55
pabelangernot sure why23:55
pabelangerEmilienM: okay, vxlan was setup23:58
*** achadha has quit IRC23:59
*** achadha has joined #tripleo23:59
EmilienMyep indeed23:59

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!