Wednesday, 2019-10-30

*** rlandy has quit IRC00:02
*** panda has quit IRC00:02
openstackgerritMerged openstack/tripleo-ansible master: Add support for --limit option into ceph-ansible call  https://review.opendev.org/69137900:05
openstackgerritMerged openstack/tripleo-ansible master: Ensure plugin path is correct  https://review.opendev.org/69191200:05
*** panda has joined #tripleo00:05
*** tkajinam has joined #tripleo00:16
*** ricolin has quit IRC00:45
openstackgerritMerged openstack/diskimage-builder master: Revert "Drop vhdutil dependency, use qemu-img"  https://review.opendev.org/69199800:52
*** ricolin has joined #tripleo00:57
openstackgerritMerged openstack/tripleo-heat-templates stable/stein: Normalise all pacemaker resource upgrade tasks for staged upgrades  https://review.opendev.org/68914901:06
openstackgerritMerged openstack/tripleo-ci master: Remove scenario012 multinode job  https://review.opendev.org/69173901:29
openstackgerritMerged openstack/tripleo-ci master: Avoid execution failure when mirror_info.sh is missing  https://review.opendev.org/69167101:29
openstackgerritMerged openstack/tripleo-ansible master: Run scenario004 for changes to tripleo-ceph-* roles  https://review.opendev.org/69143501:42
*** bfournie has joined #tripleo01:55
*** slaweq has joined #tripleo02:22
*** slaweq has quit IRC02:27
*** slaweq has joined #tripleo02:37
*** slaweq has quit IRC02:52
*** slaweq has joined #tripleo02:57
*** mschuppert has quit IRC03:01
*** slaweq has quit IRC03:12
*** slaweq has joined #tripleo03:17
*** slaweq has quit IRC03:22
*** slaweq_ has joined #tripleo03:32
*** slaweq_ has quit IRC03:37
*** psachin has joined #tripleo03:38
*** rh-jelabarre has quit IRC03:38
*** slaweq_ has joined #tripleo03:42
*** avivgta has quit IRC03:44
openstackgerritMerged openstack/tripleo-heat-templates master: Add experimental support to deploy containers without Paunch  https://review.opendev.org/68765103:44
*** slaweq_ has quit IRC03:47
*** slaweq_ has joined #tripleo03:52
*** slaweq_ has quit IRC03:57
*** ykarel|pto has joined #tripleo04:01
*** ykarel|pto is now known as ykarel04:01
*** epoojad1 has joined #tripleo04:08
openstackgerritMerged openstack/tripleo-ansible master: tripleo-container-manage: few improvements  https://review.opendev.org/69183604:18
openstackgerrityatin proposed openstack/tripleo-quickstart master: [DNM] Add CloudSIG stein/train release files  https://review.opendev.org/66595904:22
*** epoojad1 has quit IRC04:32
*** udesale has joined #tripleo04:35
openstackgerrityatin proposed openstack/tripleo-ci master: [DNM] Allow to specify custom version_hash  https://review.opendev.org/66534504:54
*** soniya29 has joined #tripleo04:59
*** ykarel is now known as ykarel|afk05:02
*** ykarel|afk has quit IRC05:02
*** bandini has joined #tripleo05:07
*** skramaja has joined #tripleo05:15
*** janki has joined #tripleo05:17
*** ykarel has joined #tripleo05:25
openstackgerritChandan Kumar (raukadah) proposed openstack/tripleo-quickstart master: Move fs020 to os_tempest from train onwards  https://review.opendev.org/68746005:31
openstackgerritChandan Kumar (raukadah) proposed openstack/tripleo-quickstart master: Port fs021 to os_tempest  https://review.opendev.org/68908505:31
*** ratailor has joined #tripleo05:47
*** sanjayu_ has joined #tripleo05:47
*** lmiccini has joined #tripleo06:22
*** dking_desktop has quit IRC06:27
*** hjensas has quit IRC06:35
*** kopecmartin|off is now known as kopecmartin06:40
TenguEmilienM: ah, yeah, maybe/probably - though I found out that, in my env, python3-molecule with python3 ansible wanted python2-pyyaml.... go figure.06:43
*** yprokule has joined #tripleo06:49
openstackgerritRajesh Tailor proposed openstack/instack-undercloud stable/queens: Delete nova-cert service during install/update  https://review.opendev.org/69093006:49
*** marios has joined #tripleo06:54
*** marios is now known as marios|ruck06:54
*** jbadiapa has joined #tripleo06:57
openstackgerrityatin proposed openstack/tripleo-ci master: Do not rely on system ansible  https://review.opendev.org/66024706:58
*** ykarel is now known as ykarel|afk06:59
*** sanjayu_ has quit IRC07:02
*** saneax has joined #tripleo07:02
*** sanjayu_ has joined #tripleo07:04
*** saneax has quit IRC07:05
*** sanjayu__ has joined #tripleo07:07
*** waleedm has joined #tripleo07:07
*** sanjayu__ has quit IRC07:08
*** pcaruana has joined #tripleo07:09
*** sanjayu_ has quit IRC07:09
*** saneax has joined #tripleo07:10
*** jcapitao|afk has joined #tripleo07:10
*** jcapitao|afk is now known as jcapitao07:11
*** sanjayu_ has joined #tripleo07:12
*** jcapitao has quit IRC07:13
*** saneax has quit IRC07:13
*** cfontain_ has quit IRC07:23
*** dciabrin has quit IRC07:24
openstackgerritJuan Badia Payno proposed openstack/ansible-role-openstack-operations master: [WIP] - Added rear backup  https://review.opendev.org/69205507:25
*** rascasoft has quit IRC07:29
*** mschuppert has joined #tripleo07:46
*** aedc has quit IRC07:49
*** pkopec has joined #tripleo07:55
*** bogdando has joined #tripleo07:56
*** florianf has joined #tripleo08:02
*** tkajinam has quit IRC08:03
*** ykarel|afk is now known as ykarel08:03
openstackgerritCédric Jeanneret (Tengu) proposed openstack/tripleo-heat-templates master: Add in-flight validation for keystone container  https://review.opendev.org/69182708:03
openstackgerritCédric Jeanneret (Tengu) proposed openstack/tripleo-heat-templates master: Add in-flight validation for keystone container  https://review.opendev.org/69182708:04
*** avivgta has joined #tripleo08:06
*** ratailor has quit IRC08:16
*** ratailor has joined #tripleo08:18
*** chandankumar has quit IRC08:19
*** chandankumar has joined #tripleo08:20
*** holser has joined #tripleo08:22
*** slaweq_ has joined #tripleo08:30
*** gfidente has joined #tripleo08:34
*** pbandark has joined #tripleo08:35
*** hjensas has joined #tripleo08:37
*** jpena|off is now known as jpena08:37
*** florianf_ has joined #tripleo08:37
*** florianf has quit IRC08:38
*** florianf_ is now known as florianf08:44
*** amoralej|off is now known as amoralej08:46
*** janki has quit IRC08:46
*** jchhatbar has joined #tripleo08:47
*** lucasagomes has joined #tripleo08:51
openstackgerrityatin proposed openstack/tripleo-heat-templates master: [DNM] collect image prepare debug data  https://review.opendev.org/69207808:51
openstackgerritCarlos Camacho proposed openstack/tripleo-heat-templates master: Ansible lint check in THT  https://review.opendev.org/68747908:56
*** lmiccini has quit IRC08:57
*** rpittau|afk is now known as rpittau08:58
*** avivgta has quit IRC08:59
*** dtantsur|afk is now known as dtantsur09:01
*** jpich has joined #tripleo09:01
*** ykarel is now known as ykarel|lunch09:03
*** lmiccini has joined #tripleo09:06
openstackgerritSergii Golovatiuk proposed openstack/tripleo-upgrade stable/stein: Bump cirros version to 0.4.0  https://review.opendev.org/69208309:09
*** iurygregory has joined #tripleo09:10
*** suuuper has joined #tripleo09:16
openstackgerritSergii Golovatiuk proposed openstack/tripleo-upgrade stable/rocky: Bump cirros version to 0.4.0  https://review.opendev.org/69208609:17
*** slaweq_ is now known as slaweq09:18
openstackgerritSergii Golovatiuk proposed openstack/tripleo-upgrade stable/queens: Bump cirros version to 0.4.0  https://review.opendev.org/69208709:19
*** sshnaidm|afk is now known as sshnaidm09:21
*** jfrancoa has joined #tripleo09:23
openstackgerritSergii Golovatiuk proposed openstack/tripleo-upgrade stable/pike: Bump cirros version to 0.4.0  https://review.opendev.org/69208909:27
*** jfrancoa has quit IRC09:28
openstackgerritNatal Ngétal proposed openstack/tripleo-heat-templates stable/train: Workaround rhel8,0 iptables bug causing neutron_ovs_agent to restart continuously  https://review.opendev.org/69209009:29
*** paramite has joined #tripleo09:38
*** ratailor_ has joined #tripleo09:38
*** tesseract has joined #tripleo09:38
openstackgerritBogdan Dobrelya (bogdando) proposed openstack/tripleo-ci master: Collect generated grapg for container images prepare  https://review.opendev.org/69185109:38
openstackgerritSorin Sbarnea proposed openstack/tripleo-heat-templates master: Add a scenario000-standalone environment  https://review.opendev.org/69096009:41
openstackgerritSorin Sbarnea proposed openstack/tripleo-heat-templates master: Add a scenario000-standalone environment  https://review.opendev.org/69096009:41
*** ratailor has quit IRC09:41
*** dciabrin has joined #tripleo09:42
*** aedc has joined #tripleo09:46
*** cgoncalves has quit IRC09:49
openstackgerritSorin Sbarnea proposed openstack/tripleo-ci master: add tripleo-ci-centos-8-scenario000-standalone job  https://review.opendev.org/69096109:55
*** cgoncalves has joined #tripleo09:58
*** holser has quit IRC10:11
*** ykarel|lunch is now known as ykarel10:13
*** holser has joined #tripleo10:13
*** holser has quit IRC10:13
*** holser has joined #tripleo10:14
*** Tengu has quit IRC10:18
*** holser has quit IRC10:19
*** holser__ has joined #tripleo10:19
*** Tengu has joined #tripleo10:19
*** morazi has quit IRC10:24
*** rcernin has quit IRC10:27
*** florianf has quit IRC10:28
*** pcaruana has quit IRC10:35
marios|rucksshnaidm: reviews please https://review.opendev.org/#/c/655423/ it is exercised there https://review.opendev.org/#/c/651828/ (tobiko tobiko-tripleo-update-standalone )10:41
marios|ruck(sorry sshnaidm )10:41
marios|ruckslaweq: ^^10:41
openstackgerrityogananth subramanian proposed openstack/puppet-tripleo stable/queens: Fix upper case checks for SRIOV interface  https://review.opendev.org/69210110:41
*** csmart has quit IRC10:44
*** jtomasek has quit IRC10:44
*** openstackstatus has quit IRC10:44
*** jtomasek has joined #tripleo10:48
*** paramite has quit IRC10:50
*** hkominos has joined #tripleo10:53
*** rfolco|off has joined #tripleo10:54
openstackgerritTelles Mota Vidal Nóbrega proposed openstack/tripleo-ansible master: Adding setup_rear and setup_nfs task  https://review.opendev.org/69195010:59
*** panda is now known as panda|pto11:00
openstackgerritSagi Shnaidman proposed openstack/tripleo-ansible master: Adds new molecule testing for tripleo-container-manage  https://review.opendev.org/69185711:01
sshnaidmEmilienM, take a look if it's correct please ^^11:01
EmilienMsshnaidm: ok11:01
EmilienMsshnaidm: it looks excellent, thanks for the help11:03
*** csmart has joined #tripleo11:05
openstackgerrityogananth subramanian proposed openstack/puppet-tripleo stable/rocky: Fix upper case checks for SRIOV interface  https://review.opendev.org/69210511:08
openstackgerrityogananth subramanian proposed openstack/puppet-tripleo stable/stein: Fix upper case checks for SRIOV interface  https://review.opendev.org/69210611:09
*** arxcruz is now known as arxcruz|lunch11:10
EmilienMsshnaidm: it passed, +211:10
EmilienMsshnaidm: maybe cédric and you can have a last pass so we expediate it11:10
EmilienMnice team work!11:11
slaweqmarios|ruck: thx a lot11:11
sshnaidmEmilienM, it's passed11:12
sshnaidmEmilienM, don't you have this js applet that brings you CI results right after every job finishes?11:13
*** jchhatbar has quit IRC11:14
*** fultonj has quit IRC11:14
EmilienMsshnaidm: I have a secret and therefore I can't share it :P11:14
*** fultonj has joined #tripleo11:16
*** pbandark has quit IRC11:16
*** rh-jelabarre has joined #tripleo11:16
*** aedc has quit IRC11:17
marios|ruckack np slaweq11:18
*** sshnaidm has quit IRC11:23
openstackgerritMerged openstack/tripleo-quickstart-extras master: [os_tempest] Generate skiplist only for overcloud  https://review.opendev.org/68910611:24
openstackgerritMerged openstack/ansible-role-tripleo-modify-image master: Python3 support for dev_install  https://review.opendev.org/69048411:24
*** florianf has joined #tripleo11:27
openstackgerritJose Luis Franco proposed openstack/tripleo-upgrade master: Remove qemu packages to run Leapp upgrade with rhos-release.  https://review.opendev.org/69176311:27
*** rfolco|off has quit IRC11:35
*** morazi has joined #tripleo11:37
*** morazi has quit IRC11:39
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/tripleo-ansible stable/train: Revert "Workaround for ansible-lint installation failure"  https://review.opendev.org/69210911:39
openstackgerritHarald Jensås proposed openstack/diskimage-builder master: WIP: Add IPv6 support in dhcp-all-interfaces  https://review.opendev.org/69211011:40
odyssey4meCould I get some eyes on https://review.opendev.org/691837 please to fix updates for master, which needs a backport to stable/train too?11:41
EmilienModyssey4me: yes11:42
openstackgerritHarald Jensås proposed openstack/diskimage-builder master: WIP: Add IPv6 support in dhcp-all-interfaces  https://review.opendev.org/69211011:43
*** sshnaidm has joined #tripleo11:43
*** raildo has joined #tripleo11:44
*** ramishra has quit IRC11:46
openstackgerritGiulio Fidente proposed openstack/tripleo-common stable/rocky: Bump ceph-container version to 3.2.7  https://review.opendev.org/69211111:46
*** ramishra has joined #tripleo11:47
*** morazi has joined #tripleo11:48
openstackgerritGiulio Fidente proposed openstack/tripleo-common stable/queens: Bump ceph-container to 3.2.7  https://review.opendev.org/69211311:49
openstackgerritKamil Sambor proposed openstack/puppet-tripleo stable/train: Add configurable monitor timeouts for ovn dbs  https://review.opendev.org/69211411:49
odyssey4metyvm EmilienM11:51
openstackgerritDougal Matthews proposed openstack/tripleo-specs master: Add spec for the Mistral to Ansible work  https://review.opendev.org/67927211:52
openstackgerritMerged openstack/tripleo-heat-templates stable/train: deploy-steps-tasks: remove duplicate "when" for paunch debug  https://review.opendev.org/69200211:53
*** dciabrin has quit IRC11:55
*** ratailor_ has quit IRC11:56
openstackgerrityogananth subramanian proposed openstack/puppet-tripleo stable/train: Fix upper case checks for SRIOV interface  https://review.opendev.org/69211711:58
openstackgerritFrancesco Pantano proposed openstack/tripleo-heat-templates master: Adds ceph-systemd tag in external_upgrade_tasks for systemd units  https://review.opendev.org/69138011:58
*** jpena is now known as jpena|lunch11:59
*** ekultails has joined #tripleo11:59
*** marios|ruck is now known as marios|call12:00
*** pcaruana has joined #tripleo12:00
*** ykarel is now known as ykarel|afk12:02
cloudnullmornings12:03
*** paramite has joined #tripleo12:09
*** rlandy has joined #tripleo12:13
openstackgerritKamil Sambor proposed openstack/puppet-tripleo stable/stein: Add configurable monitor timeouts for ovn dbs  https://review.opendev.org/69211912:14
openstackgerritHarald Jensås proposed openstack/diskimage-builder master: WIP: Add IPv6 support in dhcp-all-interfaces  https://review.opendev.org/69211012:17
*** arxcruz|lunch is now known as arxcruz12:18
openstackgerritKamil Sambor proposed openstack/puppet-tripleo stable/rocky: Add configurable monitor timeouts for ovn dbs  https://review.opendev.org/69212012:20
arxcruzfultonj: gfidente any news regarding the tags on ceph-ansible?12:21
gfidentearxcruz nah sorry12:22
gfidentewill send the email now12:23
arxcruzgfidente: ok, please cc me :)12:24
*** udesale has quit IRC12:26
openstackgerritKamil Sambor proposed openstack/puppet-tripleo stable/queens: Add configurable monitor timeouts for ovn dbs  https://review.opendev.org/69212312:26
*** alexmcleod has joined #tripleo12:30
*** aedc has joined #tripleo12:39
*** hkominos has quit IRC12:40
*** aakarsh has quit IRC12:47
*** Goneri has joined #tripleo12:49
openstackgerritKamil Sambor proposed openstack/tripleo-heat-templates stable/train: Add configurable monitor timeouts for ovn dbs  https://review.opendev.org/69212712:50
openstackgerritKamil Sambor proposed openstack/tripleo-heat-templates stable/stein: Add configurable monitor timeouts for ovn dbs  https://review.opendev.org/69213012:55
*** rfolco has joined #tripleo13:01
*** jpena|lunch is now known as jpena13:02
openstackgerritRonelle Landy proposed openstack/ansible-role-collect-logs master: WIP: Add functionality to sanitize collected logs  https://review.opendev.org/69199413:02
*** psachin has quit IRC13:02
openstackgerritKamil Sambor proposed openstack/tripleo-heat-templates stable/rocky: Add configurable monitor timeouts for ovn dbs  https://review.opendev.org/69213313:05
*** hkominos has joined #tripleo13:05
*** ramishra has quit IRC13:07
openstackgerritKamil Sambor proposed openstack/tripleo-heat-templates stable/queens: Add configurable monitor timeouts for ovn dbs  https://review.opendev.org/69213413:10
*** soniya29 has quit IRC13:11
slagleEmilienM: hey, you may want to look at this job: https://zuul.opendev.org/t/openstack/build/f569583a37a34b578121f8c1d47c6b8613:11
openstackgerritMerged openstack/tripleo-upgrade master: Bump cirros version to 0.4.0  https://review.opendev.org/69189013:11
slagleEmilienM: a container failed to start at step 3, and instead of failing, the job just eventually times out. could be an issue with the new paunch handling13:11
*** amoralej is now known as amoralej|lunch13:12
slagleEmilienM: you can see the container failure for nova_api_db_sync here: https://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/zuul_opendev_logs_f56/690154/2/check/tripleo-ci-centos-7-containers-multinode/f569583/logs/subnode-1/var/log/paunch.log.txt.gz13:12
*** xek has joined #tripleo13:13
mwhahahawe don't have any time limits on the db sync containers13:15
mwhahahai've seen it just hang because mysql is unavailable13:15
mwhahahaseems like something we should add (it's not a new issue)13:16
EmilienMslagle: the new paunch handling hasn't been merged yet (if you speak about https://review.opendev.org/#/c/691528)13:17
EmilienMmwhahaha: right we should have a time limit on the containers13:17
mwhahahawe used to have exec time limits in puppet13:18
mwhahahabut we lost that when we run the commands straight in a container13:18
slagleEmilienM: the new paunch module is already in use though13:18
*** pkopec has quit IRC13:19
EmilienMslagle: right. The outcome should be the same though, it's calling the same functions (paunch as a library vs CLI)13:19
EmilienMI'll look at it in case13:19
*** pkopec has joined #tripleo13:19
slagleEmilienM: when paunch logged that error, it should have failed the task13:19
slaglei would think anyway13:20
mwhahahait retries13:20
mwhahahaand the retry hung13:20
mwhahahahence the need for a timeout13:20
slaglewhere is the evidence of that?13:20
slaglei see the timeout, but nothing about a retry13:21
mwhahahaansible ate it13:21
slaglemwhahaha: you're saying the ansible task is supposed to retry?13:22
slagleb/c there's nothing about a retry in the paunch log or the nova_api_db_sync log13:23
EmilienMthe retry is in the container_puppet.py, right?13:23
mwhahahayea13:23
slaglethis isn't container_puppet.py. it's paunch starting containers for step 313:24
slaglei only see a single attempt of the db sync in the nova manage log as well13:24
EmilienMah right13:24
EmilienMthe CLI called the paunch builder without retries, so it didn't change with Ansible I think13:25
EmilienMand I don't think we have retries in paunch13:25
mwhahahaso what's probably hanging is another db sync13:25
mwhahahawe don't just fail on first failure when starting containers13:25
*** bfournie has quit IRC13:26
slagleyea that could be13:26
EmilienMwe do have retry for container image pulling and some systemd stuffs13:26
mwhahaha4358f6833a5d  192.168.24.1:8787/tripleomaster/centos-binary-keystone:ac658f8293a71605f7db8f1acde3fdf635aa6b60_67a09fe9-updated-20191029220713                    /usr/bin/bootstra...  54 minutes ago     Up 54 minutes ago                    keystone_db_sync                             119kB (virtual 1.09GB)13:26
mwhahahaso keystone db sync hangs13:26
mwhahahahttps://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/zuul_opendev_logs_f56/690154/2/check/tripleo-ci-centos-7-containers-multinode/f569583/logs/subnode-1/var/log/extra/podman/podman_allinfo.log.txt.gz13:26
mwhahahathis is why i said we need a timeout13:27
mwhahahabecause nova api db sync failed, but keystone just hangs13:27
slagleright, ok. that explains it13:27
mwhahahahttps://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/zuul_opendev_logs_f56/690154/2/check/tripleo-ci-centos-7-containers-multinode/f569583/logs/subnode-1/var/log/containers/keystone/keystone.log.txt.gz13:28
mwhahahamysql was unavailable13:28
EmilienMweird13:28
mwhahahathere really should be a limit in the db syncs themselves, but here we are13:28
EmilienMcould  be a start_order thing?13:28
mwhahahahaproxy probably had it down13:29
slagleyea not sure what the exact issue is13:29
slaglebut if paunch were to log when it's starting a container, it might help here13:29
mwhahahapffft logging13:30
mwhahahawho needs that13:30
slaglemaybe we should switch to kubernetes. i heard it can start containers too13:31
openstackgerritMarios Andreou proposed openstack/tripleo-docs master: tripleo-ci ruck|rover and promotions docs  https://review.opendev.org/68953013:33
openstackgerritMerged openstack/tripleo-ansible stable/train: Revert "Workaround for ansible-lint installation failure"  https://review.opendev.org/69210913:34
openstackgerritMarios Andreou proposed openstack/tripleo-docs master: tripleo-ci ruck|rover and promotions docs  https://review.opendev.org/68953013:36
EmilienMlol13:37
*** TrevorV has joined #tripleo13:38
slaglewe should probably INFO log by default in paunch instead of WARNING. so we could add some helpful messages13:40
EmilienM+113:41
slagleok, will do something13:41
slaglei don't know what. but i'll do something13:41
*** ramishra has joined #tripleo13:41
slagleas soon as i figure out why i don't have a database13:43
*** aakarsh has joined #tripleo13:44
*** ykarel|afk is now known as ykarel13:45
*** cdearborn has joined #tripleo13:45
*** bfournie has joined #tripleo13:46
EmilienMyou said you would convert our tooling to deploy containers with k8s13:47
*** aakarsh|2 has joined #tripleo13:47
*** kopecmartin is now known as kopecmartin|scho13:48
*** artom has quit IRC13:48
*** aakarsh has quit IRC13:50
*** dciabrin has joined #tripleo13:50
zbrykarel: re queens and iptable, is that in the right direction https://review.opendev.org/#/c/686192/ ?13:51
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates stable/train: Testing for train jobs  https://review.opendev.org/69214413:51
ykarelzbr, nope that's not right, ^^ is disabling iptables to late in overcloud prep13:52
ykarelissue is at place where container login is done13:52
ykarelbefore running quickstart13:52
mwhahahaslagle: btw haproxy shows mysql as down https://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/zuul_opendev_logs_f56/690154/2/check/tripleo-ci-centos-7-containers-multinode/f569583/logs/subnode-1/var/log/extra/haproxy-stats.txt.gz13:52
zbrykarel: do you happen to have patch or know where to I should inject it?13:53
ykarelzbr, container-registry role should do it, i think u already have a patch around it13:53
ykarelhttps://review.opendev.org/#/c/686173/13:53
ykarelbut this merged13:54
ykarelso u didn't tested that?13:54
ykareland it's still failing?13:54
openstackgerritMerged openstack/tripleo-heat-templates master: Remove EC2 api  https://review.opendev.org/69010313:54
zbrykarel: i need to ask, weshay after the meeting, if i remember well he told me that on queens our code is *not* using this role.13:56
mwhahahaquickstart would use that role13:57
mwhahahathe deploy doesn't13:57
ykarelzbr, that role is used and that's why is issue is happending13:57
ykarelafter enabling registry login at those jobs13:57
*** amoralej|lunch is now known as amoralej13:58
slaglemwhahaha: yea, i see services down. and mysql restarted a few times in its log13:58
zbrmwhahaha: so https://review.opendev.org/#/c/686192/ is still needed (after i fix the code itself)13:59
mwhahahaof course the logs are less than helpful13:59
mwhahahazbr: no13:59
mwhahahathe puppet bits should do that13:59
mwhahahai think14:00
mwhahahathe queens stuff didn't do lookup bits (if i recall correctly) so it's not the same i think14:00
*** ykarel is now known as ykarel|meeting14:04
openstackgerritAde Lee proposed openstack/tripleo-ansible master: WIP Add role to add ipa services for tls-everywhere  https://review.opendev.org/69190614:05
*** alexmcleod has quit IRC14:05
*** alexmcleod has joined #tripleo14:06
ade_leemwhahaha, EmilienM https://review.opendev.org/#/c/691887 please14:09
ade_leemwhahaha, ah - just saw your comment about needing a bug.  adding one14:09
openstackgerritMerged openstack/tripleo-heat-templates master: Correct ovn-dbs upgrade image id variable  https://review.opendev.org/69183714:20
*** waleedm has quit IRC14:20
*** jaosorior has joined #tripleo14:21
openstackgerritAde Lee proposed openstack/tripleo-heat-templates stable/train: Restart certmnonger after registering system with IPA  https://review.opendev.org/69188414:24
mjturekhate to bother but can anyone take a look at https://review.opendev.org/#/c/691698/ ? ppc container image build job is currently blocked by it14:25
openstackgerritAde Lee proposed openstack/tripleo-heat-templates stable/stein: Restart certmnonger after registering system with IPA  https://review.opendev.org/69188614:25
openstackgerritAde Lee proposed openstack/tripleo-heat-templates stable/queens: Restart certmnonger after registering system with IPA  https://review.opendev.org/69188814:27
ade_leemwhahaha, EmilienM updated backports with bug number ^^14:27
*** mcornea has joined #tripleo14:28
openstackgerritRodolfo Alonso Hernandez proposed openstack/tripleo-heat-templates master: Add "radvd_user" config parameter to Neutron L3 container  https://review.opendev.org/69215214:29
mwhahahaade_lee: you missed stable/rocky14:30
openstackgerritAde Lee proposed openstack/tripleo-heat-templates stable/rocky: Restart certmnonger after registering system with IPA  https://review.opendev.org/69188714:32
ade_leemwhahaha, thanks - did it but forgot to publish edit14:32
*** dasp has quit IRC14:32
gfidentearxcruz it's unclear to me though why we need describe --tags to build the rpm14:34
*** artom has joined #tripleo14:34
gfidenteand also why it is even supposed to point to latest 4.x tag, which is created from a commit in stable-4.0 not master14:34
gfidenteso what it is exactly that we should ask the guys?14:34
arxcruzjpena: can you help me out here? ^14:37
jpenagfidente: so the issue is that the latest tag in master has a lower value than what we have in the stable branches. The makefile included in the repo uses "git describe --tags" to get the version for the rpm, so we end up with a lower version in master than in stable-4.x14:38
gfidentejpena yeah I get it14:39
gfidentebut so they tagged from master14:39
gfidenteby mistake?14:39
jpenait looks like rc tags where done in master, and after that they did it from stable14:39
gfidentejpena because there wasn't stable- branch until rc yes14:40
jpenawe could change the way the makefile is defining the version, so it increases the y version (assuming a post-tag commit). But I'm not sure if the team will want that14:40
jpenawe do something similar in DLRN: if you have a post-tag commit, we auto-increase the version number14:41
jpenahttps://github.com/softwarefactory-project/DLRN/blob/master/scripts/build_srpm.sh#L98-L11014:41
*** dasp has joined #tripleo14:42
openstackgerritHarald Jensås proposed openstack/diskimage-builder master: WIP: Add IPv6 support in dhcp-all-interfaces  https://review.opendev.org/69211014:43
openstackgerritHarald Jensås proposed openstack/python-tripleoclient master: FQDN in insecure registries, and write hosts early  https://review.opendev.org/69080514:45
*** Goneri has quit IRC14:49
*** leanderthal has joined #tripleo14:50
openstackgerritEmilien Macchi proposed openstack/paunch master: Fix backward compatibility for old config startup files  https://review.opendev.org/69152814:58
*** ykarel|meeting is now known as ykarel15:01
openstackgerritEmilien Macchi proposed openstack/tripleo-ansible master: paunch: use utils.load_config()  https://review.opendev.org/69165515:01
*** xek_ has joined #tripleo15:03
*** amoralej is now known as amoralej|off15:03
*** jpena is now known as jpena|off15:04
*** marios|call is now known as marios15:04
*** xek has quit IRC15:05
*** jaosorior has quit IRC15:06
*** weshay is now known as weshay|rover15:08
*** rfolco is now known as rfolco|ruck15:08
*** sanjayu_ has quit IRC15:10
*** marios is now known as marios|ruck15:11
*** rfolco|ruck is now known as rfolco|rucker15:13
openstackgerritFrancesco Pantano proposed openstack/tripleo-heat-templates master: [DNM] Enable Dashboard on scenario001  https://review.opendev.org/67769915:17
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/tripleo-heat-templates stable/train: Correct ovn-dbs upgrade image id variable  https://review.opendev.org/69216915:22
*** rfolco|rucker has quit IRC15:26
*** florianf has quit IRC15:29
*** hjensas has quit IRC15:32
*** dciabrin has quit IRC15:37
*** Goneri has joined #tripleo15:49
*** jaosorior has joined #tripleo15:51
*** florianf has joined #tripleo15:52
*** ykarel is now known as ykarel|away15:55
openstackgerritMichele Baldessari proposed openstack/tripleo-heat-templates master: Fix duplicate mount point in neutron_dhcp  https://review.opendev.org/69218216:08
openstackgerritJaganathan Palanisamy proposed openstack/tripleo-heat-templates stable/rocky: Derives NovaPCIPassthrough per SR-IOV node  https://review.opendev.org/69218416:13
*** TrevorV has quit IRC16:17
*** kopecmartin|scho is now known as kopecmartin|off16:24
*** social has quit IRC16:24
*** marios|ruck is now known as marios|out16:25
openstackgerritEmilien Macchi proposed openstack/tripleo-ansible master: zuul/container-manage: trigger molecule jobs on more files  https://review.opendev.org/69218816:26
openstackgerritDougal Matthews proposed openstack/tripleo-ansible master: Baremetal Introspection Playbook  https://review.opendev.org/68919416:27
openstackgerritDougal Matthews proposed openstack/tripleo-ansible master: Add support for introspecting all_manageable nodes  https://review.opendev.org/68953216:27
openstackgerritDougal Matthews proposed openstack/tripleo-ansible master: Baremetal Configure Playbook  https://review.opendev.org/69166716:27
openstackgerritDougal Matthews proposed openstack/tripleo-ansible master: OpenStack Node Provide playbook  https://review.opendev.org/69218916:27
*** dking_desktop has joined #tripleo16:30
openstackgerritMerged openstack/tripleo-quickstart master: Move fs020 to os_tempest from train onwards  https://review.opendev.org/68746016:31
dking_desktopThis may be a silly question, but if I'm seeing the following in an overcloud deploy with --debug, does it mean that the deploy has failed already by this point:   REQ: curl -g -i ... -X POST https://10.64.0.2:13989/v2/executions ... -H "content-type: application/json" -d '{"input": "{\"plan\": \"overcloud-one\"}", "workflow_name": "tripleo.deployment.v1.set_deployment_status_failed", "description": ""}'16:32
*** TrevorV has joined #tripleo16:32
*** leanderthal has quit IRC16:33
openstackgerritBogdan Dobrelya (bogdando) proposed openstack/tripleo-common master: Make upload workers faster on processing layers  https://review.opendev.org/68728816:36
*** alexmcleod has quit IRC16:36
openstackgerritMerged openstack/tripleo-ansible master: Adds new molecule testing for tripleo-container-manage  https://review.opendev.org/69185716:38
*** marios|out has quit IRC16:41
*** ricolin has quit IRC16:41
cloudnullnice!16:43
openstackgerritBogdan Dobrelya (bogdando) proposed openstack/tripleo-common master: Make upload workers faster on processing layers  https://review.opendev.org/68728816:43
dking_desktop?16:44
openstackgerritSorin Sbarnea proposed openstack/tripleo-quickstart-extras master: lint: removed 601, 602, corrected flake8  https://review.opendev.org/69219016:45
mwhahahadking_desktop: i don't think so, i think it's setting it as running or something. would have to dig into the code16:48
openstackgerritBogdan Dobrelya (bogdando) proposed openstack/tripleo-common master: Lock image layers for local operations as well  https://review.opendev.org/69219116:48
openstackgerritBogdan Dobrelya (bogdando) proposed openstack/tripleo-common master: Store cross-references of layers in global view  https://review.opendev.org/69219216:48
*** hjensas has joined #tripleo16:49
dking_desktopI'm running through the output with --debug and I'm having trouble finding a place where it actually fails. The eventual failure is that shortly after it tries to send that, I get a python error thrown: "ValueError: No JSON object could be decoded"16:50
mwhahahaonce the deploy gets started, the client basically just polls the status from mistral16:50
mwhahahayou'd need to check the mistral logs to see if something goes wrong16:51
mwhahahawell not completely accurate, mistral runs and publishes status/info to zaqar and the client polls zaqar16:51
*** morazi has quit IRC16:51
openstackgerritRonelle Landy proposed openstack/ansible-role-collect-logs master: WIP: Add functionality to sanitize collected logs  https://review.opendev.org/69199416:51
mwhahahai've seen the error you're talking about if the mistral executor crashes or there's an exception in it so it never completed teh workflow16:51
*** hkominos has quit IRC16:53
dking_desktopDo you know what log to check for that? I'm currently looking at the logs that were output on the command line as well as looking at /var/lib/mistral/<cloud name>/ansible.log16:53
*** lucasagomes has quit IRC16:54
*** TrevorV has quit IRC16:54
*** morazi has joined #tripleo16:54
*** TrevorV has joined #tripleo16:55
dking_desktopHere's the last thing when the error is thrown: http://paste.openstack.org/show/785661/16:58
*** rpittau is now known as rpittau|afk16:59
*** klemorali has quit IRC17:00
mwhahahadking_desktop: if >= Rocky, /var/log/containers/mistral/ has the service logs17:01
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Install and start Rsyslog on the Undercloud & Standalone  https://review.opendev.org/69219517:01
dking_desktopThanks. I'll take a look there netxt.17:02
*** openstackstatus has joined #tripleo17:04
*** ChanServ sets mode: +v openstackstatus17:04
*** bogdando has quit IRC17:04
*** rfolco has joined #tripleo17:04
*** jpich has quit IRC17:22
openstackgerritSorin Sbarnea proposed openstack/tripleo-quickstart master: WIP: iptables and queens  https://review.opendev.org/69219617:25
*** Garyx_ has quit IRC17:25
*** xek_ has quit IRC17:29
*** florianf has quit IRC17:38
*** avivgta has joined #tripleo17:39
*** dtantsur is now known as dtantsur|afk17:49
ykarel|awayzbr, queens overcloud deployment succeeded with the test patch:- https://logs.rdoproject.org/71/20171/69/check/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-queens/c5a5a45/logs/tempest.html.gz17:49
ykarel|awaytempest failed though, but that's unrelated17:49
ykarel|awayso u can formalize the usage of that new var17:49
ykarel|awayfor queens jobs17:49
*** morazi has quit IRC17:50
ykarel|awayweshay|rover, fyi ^^17:51
*** Garyx has joined #tripleo17:51
weshay|roverykarel|away, aye.. thanks I've been chatting w/ zbr about it17:52
ykarel|awayokk cool17:52
*** morazi has joined #tripleo17:53
ykarel|awaymwhahaha, can u check u requested for the debug enablement for stuck container-image-prepare as done with https://review.opendev.org/#/c/692078/17:55
ykarel|awayit didn't reproduced yet with couple of rechecks17:55
ykarel|awayalso running some jobs with https://review.rdoproject.org/r/#/c/23122/17:55
mwhahahawhat are you trying to do?17:55
mwhahahawill check back in a bit after lunch17:55
mwhahahai don't thinkt hat's right17:56
ykarel|away<mwhahaha> yea it didn't OOM, so we'd need debug to be enabled17:56
ykarel|away<mwhahaha> then we could see what it's hanging o17:56
ykarel|awaymwhahaha, ^^17:56
ykarel|awaymwhahaha, okk please check if it's wrong, and update to the correct17:56
openstackgerritwes hayutin proposed openstack/tripleo-quickstart master: iptables and queens  https://review.opendev.org/69219617:56
ykarel|awayweshay|rover, ^^ should not work, as we need that var at job level17:57
ykarel|awayas that role is executed before quickstart run17:57
weshay|roverk. k.. /me looking.. thank you17:58
weshay|roverhttps://opendev.org/openstack/ansible-role-container-registry/src/branch/master/tasks/install-engine.yml#L1717:58
weshay|roversee it17:58
mwhahahaykarel|away: debug is on for non-ovb jobs17:58
mwhahahaykarel|away: there's another setting somewhere else17:58
mwhahahabut i'll find it after lunch17:58
weshay|rovermwhahaha, ya.. in tqe17:58
ykarel|awaymwhahaha, okk Thanks17:59
weshay|roverzbr, ^17:59
rfolcocloudnull, pls review this when you get a chance https://review.opendev.org/#/c/691967/18:04
* cloudnull looking now 18:04
*** avivgta has quit IRC18:05
cloudnullrfolco lgtm, someone else will need to approve.18:08
rfolcocloudnull, thx, I did manual test and forced an exception, _exception is the right one from what I could check18:09
*** pcaruana has quit IRC18:10
cloudnull++18:10
cloudnullidk where I got exception_info from but if this works better +1 from me :)18:10
cloudnullwe might want to backport that review to train, once its approved18:11
*** dsneddon has joined #tripleo18:12
*** holser__ has quit IRC18:14
*** chandankumar is now known as raukadah18:17
*** yprokule has quit IRC18:18
*** dciabrin has joined #tripleo18:18
openstackgerritMerged openstack/tripleo-quickstart master: Port fs021 to os_tempest  https://review.opendev.org/68908518:19
*** mburned is now known as mburned_out18:23
* ykarel|away out18:25
*** ykarel|away has quit IRC18:31
*** mburned_out is now known as mburned18:34
openstackgerritSagi Shnaidman proposed openstack/tripleo-quickstart-extras master: Use docker registry mirror in overcloud  https://review.opendev.org/69191918:35
*** aedc has quit IRC18:36
*** dciabrin has quit IRC18:42
*** morazi has quit IRC18:45
*** Goneri has quit IRC18:46
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Install and start Rsyslog on the Undercloud & Standalone  https://review.opendev.org/69219518:51
EmilienMcloudnull, mwhahaha : https://review.opendev.org/#/c/692188/118:53
cloudnull+218:54
*** zaneb has quit IRC19:00
*** zaneb has joined #tripleo19:00
openstackgerritMerged openstack/tripleo-ansible master: zuul/container-manage: trigger molecule jobs on more files  https://review.opendev.org/69218819:03
*** tesseract has quit IRC19:03
*** pcaruana has joined #tripleo19:05
openstackgerritMerged openstack/tripleo-quickstart-extras master: undercloud-setup: fixed two runtime warnings  https://review.opendev.org/69183219:14
*** artom has quit IRC19:21
*** jaosorior has quit IRC19:21
*** suuuper has quit IRC19:23
*** aakarsh|2 has quit IRC19:25
*** suuuper has joined #tripleo19:27
*** gfidente is now known as gfidente|afk19:28
*** pcaruana has quit IRC19:29
*** morazi has joined #tripleo19:33
openstackgerritAlex Schultz proposed openstack/tripleo-heat-templates stable/queens: Use hiera interpolation for memcached_network  https://review.opendev.org/56163719:33
openstackgerritTelles Mota Vidal Nóbrega proposed openstack/tripleo-heat-templates master: [DNM] Adding ReaR THT  https://review.opendev.org/69007419:36
*** Goneri has joined #tripleo19:36
openstackgerritTelles Mota Vidal Nóbrega proposed openstack/tripleo-heat-templates master: [DNM] Adding ReaR THT  https://review.opendev.org/69007419:37
*** pcaruana has joined #tripleo19:39
*** aedc has joined #tripleo19:43
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates stable/queens: DO-NOT-MERGE Test new ceph-container builds (OSP queens)  https://review.opendev.org/56221520:08
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates stable/rocky: DO-NOT-MERGE Test new ceph-container builds (OSP rocky)  https://review.opendev.org/60159820:08
*** pcaruana has quit IRC20:09
*** gfidente|afk has quit IRC20:11
*** rfolco has quit IRC20:12
*** aedc has quit IRC20:14
*** mcornea has quit IRC20:18
*** Goneri has quit IRC20:22
*** avivgta has joined #tripleo20:25
openstackgerritEmilien Macchi proposed openstack/tripleo-ansible master: WIP - tripleo-container-manage - support container updates & cleanup  https://review.opendev.org/68917220:28
*** holser has joined #tripleo20:32
*** mcornea has joined #tripleo20:34
*** artom has joined #tripleo20:45
openstackgerritwes hayutin proposed openstack/tripleo-quickstart-extras master: ci queens needs docker iptables off  https://review.opendev.org/69222320:49
openstackgerritwes hayutin proposed openstack/tripleo-quickstart-extras master: ci queens needs docker iptables off  https://review.opendev.org/69222320:52
weshay|rovermwhahaha, you cool w/ https://review.opendev.org/#/q/topic:queens-docker-iptables+(status:open+OR+status:merged)20:52
mwhahahai guess?20:53
*** dciabrin has joined #tripleo20:54
weshay|roverheh20:54
*** morazi has quit IRC20:54
*** cdearborn has quit IRC20:55
*** dciabrin_ has joined #tripleo20:58
*** dciabrin has quit IRC21:02
*** avivgta has quit IRC21:05
*** morazi has joined #tripleo21:06
*** panda|pto has quit IRC21:14
*** panda has joined #tripleo21:18
openstackgerritAlex Schultz proposed openstack/tripleo-heat-templates stable/queens: Use hiera interpolation for memcached_network  https://review.opendev.org/56163721:21
openstackgerritAlex Schultz proposed openstack/tripleo-heat-templates stable/queens: Add missing hiera interpolation calls  https://review.opendev.org/69222821:21
openstackgerritAlex Schultz proposed openstack/tripleo-heat-templates stable/queens: Add validation for hiera interpolation in services  https://review.opendev.org/69222921:21
openstackgerritAlex Schultz proposed openstack/tripleo-heat-templates stable/queens: Track global failures flag for YAML validations  https://review.opendev.org/69223021:21
*** holser has quit IRC21:34
*** TrevorV has quit IRC21:36
openstackgerritIan Wienand proposed openstack/diskimage-builder master: vhd-util : note on Xen/RAX images  https://review.opendev.org/69223421:36
*** iurygregory has quit IRC21:40
*** holser has joined #tripleo21:44
*** raildo has quit IRC21:53
openstackgerritHarald Jensås proposed openstack/diskimage-builder master: WIP: Add IPv6 support in dhcp-all-interfaces  https://review.opendev.org/69211021:59
openstackgerritKevin Carter (cloudnull) proposed openstack/tripleo-ansible master: Remove old Heat hosts entries in tripleo-hosts-entries  https://review.opendev.org/69196921:59
openstackgerritKevin Carter (cloudnull) proposed openstack/tripleo-heat-templates master: Convert firewall rules to use TripleO-Ansible  https://review.opendev.org/67723722:01
openstackgerritKevin Carter (cloudnull) proposed openstack/tripleo-heat-templates master: Configure time using tripleo-ansible  https://review.opendev.org/67116622:02
*** mcornea has quit IRC22:07
openstackgerritKevin Carter (cloudnull) proposed openstack/tripleo-common master: Write ansible log content to a different log file  https://review.opendev.org/68413122:09
openstackgerritKevin Carter (cloudnull) proposed openstack/tripleo-heat-templates master: Move basic global functions into ansible tasks  https://review.opendev.org/68328622:26
openstackgerritKevin Carter (cloudnull) proposed openstack/tripleo-heat-templates master: Convert subprocess to a reusable method  https://review.opendev.org/68342922:28
openstackgerritKevin Carter (cloudnull) proposed openstack/tripleo-heat-templates master: Move globals into main level 0 case  https://review.opendev.org/68359022:28
openstackgerritKevin Carter (cloudnull) proposed openstack/tripleo-heat-templates master: Move config processing into consolodated methods  https://review.opendev.org/68367022:28
*** dtrainor has quit IRC22:29
*** pkopec has quit IRC22:33
openstackgerritSagi Shnaidman proposed openstack/ansible-role-collect-logs master: Run Sova patterns parsing in CI jobs  https://review.opendev.org/67863022:37
*** holser has quit IRC22:42
openstackgerritSteve Baker proposed openstack/python-tripleoclient master: Support undercloud fake-hardware ironic driver  https://review.opendev.org/68719522:50
*** pkopec has joined #tripleo22:52
*** rlandy has quit IRC23:00
*** tkajinam has joined #tripleo23:01
*** tkajinam has quit IRC23:01
*** tkajinam has joined #tripleo23:04
*** slaweq has quit IRC23:05
*** slaweq has joined #tripleo23:10
*** slaweq has quit IRC23:16
openstackgerritMerged openstack/puppet-tripleo stable/train: Fix upper case checks for SRIOV interface  https://review.opendev.org/69211723:16
openstackgerritKevin Carter (cloudnull) proposed openstack/tripleo-ansible master: [WIP] Convert all of the fedora28 container usage to centos8  https://review.opendev.org/69224223:18
openstackgerritKevin Carter (cloudnull) proposed openstack/tripleo-ansible master: [WIP] Convert all of the fedora28 container usage to centos8  https://review.opendev.org/69224223:19
openstackgerritKevin Carter (cloudnull) proposed openstack/tripleo-ansible master: Add the skeleton role as an irrelevant file for scenarios  https://review.opendev.org/69224323:27
openstackgerritKevin Carter (cloudnull) proposed openstack/tripleo-ansible master: [WIP] Convert all of the fedora28 container usage to centos8  https://review.opendev.org/69224223:27
*** EmilienM has quit IRC23:47
*** EmilienM has joined #tripleo23:48
*** ChanServ sets mode: +v EmilienM23:48

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!