Thursday, 2019-12-12

*** dsneddon has quit IRC00:02
*** slaweq has joined #tripleo00:03
openstackgerritMerged openstack/puppet-tripleo stable/train: Revert "Replace hiera('service_names') by hiera('enabled_services')"  https://review.opendev.org/69746600:05
*** dsneddon has joined #tripleo00:06
*** holser has quit IRC00:06
*** slaweq has quit IRC00:08
*** holser has joined #tripleo00:10
*** dsneddon has quit IRC00:12
*** dsneddon has joined #tripleo00:12
*** brault has quit IRC00:29
*** brault has joined #tripleo00:30
openstackgerritMerged openstack/tripleo-ansible stable/train: Move localhost_address to net_ip_map hieradata  https://review.opendev.org/69841400:34
*** holser has quit IRC00:38
openstackgerritMerged openstack/paunch stable/train: Exit 1 if a container fails and return the error  https://review.opendev.org/69857000:42
*** artom has quit IRC01:07
*** ysandeep has quit IRC01:24
openstackgerritMerged openstack/paunch stable/stein: Exit 1 if a container fails and return the error  https://review.opendev.org/69857101:26
*** epoojad1 has quit IRC01:46
*** dsneddon has quit IRC02:06
*** dsneddon has joined #tripleo02:29
*** rfolco has quit IRC02:32
openstackgerritJames Slagle proposed openstack/tripleo-common master: Add deploy_steps_tasks to PER_TASK_STEPS  https://review.opendev.org/69861402:39
openstackgerritJames Slagle proposed openstack/tripleo-heat-templates master: Execute deploy_steps_tasks per step  https://review.opendev.org/69861502:41
openstackgerritJames Slagle proposed openstack/tripleo-heat-templates master: Execute deploy_steps_tasks per step  https://review.opendev.org/69861502:43
*** mschuppert has quit IRC02:48
*** apetrich has quit IRC03:09
*** rh-jelabarre has quit IRC03:12
*** psachin has joined #tripleo03:36
*** rlandy|bbl has quit IRC03:44
*** udesale has joined #tripleo03:45
*** ykarel|away has joined #tripleo03:49
*** bhagyashris has joined #tripleo04:22
openstackgerritKevin Carter (cloudnull) proposed openstack/tripleo-ansible master: Update firewall role to use an action plugin  https://review.opendev.org/69862004:23
openstackgerritKevin Carter (cloudnull) proposed openstack/tripleo-ansible master: Update firewall role to use an action plugin  https://review.opendev.org/69862004:23
*** ekultails has quit IRC04:24
cloudnullslagle ^04:26
*** ykarel|away has quit IRC04:27
*** ykarel|away has joined #tripleo04:40
*** ykarel|away is now known as ykarel04:41
*** ysandeep has joined #tripleo04:53
*** ysandeep has quit IRC04:54
openstackgerritJaganathan Palanisamy proposed openstack/tripleo-common stable/train: OvS DPDK parameters failing in NIC Partitioning  https://review.opendev.org/69862105:00
*** bhagyashris has quit IRC05:06
*** shyamb has joined #tripleo05:07
*** brault has quit IRC05:13
*** bhagyashris has joined #tripleo05:13
*** brault has joined #tripleo05:14
*** epoojad1 has joined #tripleo05:14
*** surpatil has joined #tripleo05:24
openstackgerritRabi Mishra proposed openstack/tripleo-heat-templates master: Try deleting container for failures too  https://review.opendev.org/69862205:29
*** SurajPatil has joined #tripleo05:39
*** surpatil has quit IRC05:42
openstackgerritJaganathan Palanisamy proposed openstack/tripleo-common master: Derive network config fails  https://review.opendev.org/69862305:45
openstackgerritJaganathan Palanisamy proposed openstack/tripleo-common master: Derive network config fails  https://review.opendev.org/69862305:47
*** surpatil has joined #tripleo05:57
*** dsneddon has quit IRC05:58
*** SurajPatil has quit IRC05:59
*** dsneddon has joined #tripleo06:02
*** raukadah is now known as chkumar|ruck06:05
*** dsneddon has quit IRC06:07
*** yprokule has joined #tripleo06:09
*** soniya29 has joined #tripleo06:11
*** pcaruana has joined #tripleo06:18
*** waleedm_ has joined #tripleo06:18
*** marios|ruck has joined #tripleo06:26
*** zainub_wahid has joined #tripleo06:26
*** SurajPatil has joined #tripleo06:26
*** shyamb has quit IRC06:26
openstackgerritLuca Miccini proposed openstack/tripleo-heat-templates stable/train: Properly set loopback_users via rabbitmq::loopback_users  https://review.opendev.org/69862806:27
*** lmiccini has joined #tripleo06:28
*** surpatil has quit IRC06:29
*** surpatil has joined #tripleo06:31
*** SurajPatil has quit IRC06:34
*** jtomasek has joined #tripleo06:40
openstackgerritSagi Shnaidman proposed openstack/tripleo-heat-templates master: Don't use pacemaker in ironic sa job  https://review.opendev.org/68241006:42
*** soniya29 has quit IRC06:47
*** jfrancoa has joined #tripleo06:48
*** dsneddon has joined #tripleo06:49
*** ykarel is now known as ykarel|lunch06:57
*** shyamb has joined #tripleo07:02
openstackgerritMadhuri Kumari proposed openstack/diskimage-builder master: Fix wrong URL in ironic-agent element  https://review.opendev.org/69863507:05
*** jbadiapa has joined #tripleo07:06
*** shyamb has quit IRC07:10
*** shyamb has joined #tripleo07:10
*** gkadam has joined #tripleo07:13
*** gkadam has quit IRC07:13
*** bandini has joined #tripleo07:14
*** chkumar|ruck is now known as chkumar|rover07:32
openstackgerritLuca Miccini proposed openstack/tripleo-heat-templates stable/stein: Properly set loopback_users via rabbitmq::loopback_users  https://review.opendev.org/69864107:39
*** mschuppert has joined #tripleo07:41
*** shyamb has quit IRC07:41
*** bhagyashris has quit IRC07:43
*** pkopec has joined #tripleo07:43
*** hberaud|gone is now known as hberaud07:45
openstackgerritMichele Baldessari proposed openstack/tripleo-heat-templates master: Increase rabbitmq tcp backlog  https://review.opendev.org/69682707:45
*** jidar has quit IRC07:48
*** yprokule has quit IRC07:48
*** yprokule has joined #tripleo07:50
*** jidar has joined #tripleo07:51
*** tesseract has joined #tripleo07:59
*** jpena|off is now known as jpena08:01
*** dpawlik has joined #tripleo08:02
openstackgerritGregory Thiemonge proposed openstack/tripleo-heat-templates master: Set octavia services' stop grace period to 300sec  https://review.opendev.org/69801408:03
*** tkajinam has quit IRC08:04
*** yprokule has quit IRC08:08
*** yprokule_ has joined #tripleo08:08
*** yprokule_ is now known as yprokule08:09
*** bhagyashris has joined #tripleo08:12
*** jpich has joined #tripleo08:14
openstackgerritLuca Miccini proposed openstack/tripleo-heat-templates stable/rocky: Properly set loopback_users via rabbitmq::loopback_users  https://review.opendev.org/69865308:15
*** slaweq has joined #tripleo08:15
*** tmazur has joined #tripleo08:15
openstackgerritLuca Miccini proposed openstack/tripleo-heat-templates stable/queens: Properly set loopback_users via rabbitmq::loopback_users  https://review.opendev.org/69865508:17
*** amoralej|off is now known as amoralej08:26
*** tosky has joined #tripleo08:27
*** akahat has joined #tripleo08:30
*** dsneddon has quit IRC08:32
*** bhagyashris has quit IRC08:36
*** slaweq has quit IRC08:36
openstackgerritDaniel Bengtsson proposed openstack/tripleo-quickstart-extras master: Use python -m pip instead pip command.  https://review.opendev.org/69331008:37
openstackgerritDaniel Bengtsson proposed openstack/tripleo-quickstart-extras master: The file option should not require environment variable.  https://review.opendev.org/69402108:38
*** ykarel|lunch is now known as ykarel08:48
*** tonyb has quit IRC08:56
*** tonyb has joined #tripleo08:56
*** bandini has quit IRC08:57
*** SurajPatil has joined #tripleo08:58
*** dsneddon has joined #tripleo08:58
*** slaweq has joined #tripleo08:59
*** surpatil has quit IRC09:00
*** rpittau|afk is now known as rpittau09:00
*** surpatil has joined #tripleo09:01
*** shyamb has joined #tripleo09:01
*** SurajPatil has quit IRC09:03
ramishraHi! anyone knows this failure https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_c85/698622/1/check/tripleo-ci-centos-7-containers-multinode/c85b120/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz /09:03
ramishraError and traceback does not say much:/09:03
*** dsneddon has quit IRC09:05
Tenguramishra: ah... I've seen that on another change for another container.... wondering if it's a transcient issue then09:07
*** bandini has joined #tripleo09:07
chkumar|roverramishra: Tengu I have seen yesterday in the same job just one time09:07
Tenguhttps://review.opendev.org/#/c/698515/  https://60da32d022af524db511-31f87add5c30895815b1e3afff00e73d.ssl.cf2.rackcdn.com/698515/2/check/tripleo-ci-centos-7-containers-undercloud-minion/28e4d3d/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz09:08
Tenguother change, other job, other container09:08
ramishraintermittent probably, annoyingly we don't have any sane error message. Also we've the minion voting job without logs captured, unless I don't know where to look. https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_a6e/698622/1/check/tripleo-ci-centos-7-containers-undercloud-minion/a6e14e1/logs/quickstart_install.log09:09
Tenguramishra: well, "mine" is the minion one with some logs.. ?09:09
Tenguah, the minion itself. meh..09:10
ramishraTengu: I don't see /home/zuul/minion_install.log09:10
ramishraI guess that's where it's expected to write.. It's voting without logs is kind of strange:)09:11
*** redrobot has quit IRC09:11
*** dpawlik has quit IRC09:11
*** bhagyashris has joined #tripleo09:12
*** suuuper has joined #tripleo09:13
*** lucasagomes has joined #tripleo09:13
*** dpawlik has joined #tripleo09:13
*** soniya29 has joined #tripleo09:13
*** redrobot has joined #tripleo09:13
*** holser has joined #tripleo09:18
*** derekh has joined #tripleo09:19
openstackgerritSagi Shnaidman proposed openstack/tripleo-quickstart master: Bump ansible to 2.9 for quickstart  https://review.opendev.org/69274809:30
openstackgerritSagi Shnaidman proposed openstack/tripleo-quickstart master: Bump ansible to 2.9 for quickstart  https://review.opendev.org/69274809:30
openstackgerritSagi Shnaidman proposed openstack/tripleo-ci master: DNM: test ansible 2.9  https://review.opendev.org/69327409:30
openstackgerritSagi Shnaidman proposed openstack/tripleo-quickstart-extras master: DNM: Run ansible 2.9 jobs  https://review.opendev.org/69275009:30
*** iurygregory has joined #tripleo09:31
*** ccamacho has joined #tripleo09:35
openstackgerritmathieu bultel proposed openstack/tripleo-quickstart-extras master: Add when check for validate services include  https://review.opendev.org/69866309:37
*** dsneddon has joined #tripleo09:40
openstackgerritmathieu bultel proposed openstack/tripleo-upgrade stable/train: Update: Ensure parallel execution  https://review.opendev.org/69751909:43
*** ccamacho is now known as ccamacho|pto09:48
*** saneax has joined #tripleo09:48
*** dtantsur|afk is now known as dtantsur09:52
*** gfidente has joined #tripleo09:53
*** surpatil has quit IRC09:56
openstackgerritChandan Kumar (raukadah) proposed openstack/tripleo-quickstart master: Bump tempest_concurrency from 3 to 2 to fix race issue  https://review.opendev.org/69866409:57
*** shyamb has quit IRC09:57
*** shyamb has joined #tripleo09:59
*** mmethot has quit IRC10:03
*** mmethot has joined #tripleo10:04
*** ansmith_ has joined #tripleo10:17
*** zainub_wahid has quit IRC10:19
*** ansmith has quit IRC10:19
*** waleedm has joined #tripleo10:21
*** dsneddon has quit IRC10:24
*** waleedm_ has quit IRC10:24
*** pkopec has quit IRC10:26
*** pkopec has joined #tripleo10:26
*** shyamb has quit IRC10:30
*** alexmcleod has joined #tripleo10:32
*** mmethot has quit IRC10:34
openstackgerritChandan Kumar (raukadah) proposed openstack/tripleo-quickstart master: Move ostempest job definition withi n featureset 52  https://review.opendev.org/69821110:37
openstackgerritChandan Kumar (raukadah) proposed openstack/tripleo-quickstart master: Move ostempest job definition with in featureset 52  https://review.opendev.org/69821110:38
openstackgerritChandan Kumar (raukadah) proposed openstack/tripleo-ci master: Remove tempest fsoverrides in favor of featureset vars  https://review.opendev.org/69821410:39
*** dsneddon has joined #tripleo10:40
*** mmethot has joined #tripleo10:42
*** shyamb has joined #tripleo10:43
*** shyamb has quit IRC10:44
*** shyamb has joined #tripleo10:44
*** dsneddon has quit IRC10:46
*** waleedm has quit IRC10:47
*** waleedm_ has joined #tripleo10:47
*** surpatil has joined #tripleo10:50
*** ykarel is now known as ykarel|afk10:55
*** rcernin has quit IRC10:56
*** hberaud is now known as hberaud|lunch10:56
*** pcaruana has quit IRC11:03
*** pcaruana has joined #tripleo11:06
*** dtantsur is now known as dtantsur|brb11:09
openstackgerritFrancesco Pantano proposed openstack/tripleo-heat-templates master: Add CephGrafana resource to the ControllerStorageNFS role  https://review.opendev.org/69851411:12
*** dsneddon has joined #tripleo11:13
*** udesale has quit IRC11:14
*** shyamb has quit IRC11:19
*** ykarel|afk is now known as ykarel11:20
slaweqmarios|ruck: hi11:20
marios|ruckslaweq: o/11:20
slaweqmarios|ruck: according to the bug with dhcp failures11:21
marios|ruckslaweq: hey man me and chkumar|rover are trying to get you some good logs11:21
slaweqcan we maybe reproduce this on some nodes and ssh to it to debug there?11:21
marios|ruckslaweq: yeah so that is what we can do with https://review.rdoproject.org/r/#/c/23919/11:21
slaweqas I'm not 100% sure that those additional logs will be enough to triage this issue11:21
openstackgerritFrancesco Pantano proposed openstack/tripleo-heat-templates master: Add CephGrafana resource to the ControllerStorageNFS role  https://review.opendev.org/69851411:21
marios|ruckslaweq: i mean we once we get some logs there , if it isn't helpful, then we rerun it and ask rdo folks to hold the nodes running that11:22
marios|ruckslaweq: does it make sense?11:22
slaweqmarios|ruck: ok, would be great :)11:22
slaweqthx a lot11:22
marios|ruckslaweq: i just replied on the cix board check there when you get some time, and i updated https://bugs.launchpad.net/tripleo/+bug/1853652/comments/20 earlier11:22
openstackLaunchpad bug 1853652 in tripleo "openstack overcloud node provide --all-manageable timing out and failing periodic rhel-8-ovb-3ctlr_1comp-featureset001-master" [Critical,Triaged]11:22
marios|ruckslaweq: i'll add another note on cix about holding node sec11:22
slaweqok11:23
*** shyamb has joined #tripleo11:24
*** sshnaidm|afk is now known as sshnaidm11:32
openstackgerritMadhuri Kumari proposed openstack/diskimage-builder master: Fix wrong URL in ironic-agent element  https://review.opendev.org/69863511:40
openstackgerritFrancesco Pantano proposed openstack/tripleo-heat-templates master: Add CephGrafana resource to the ControllerStorageNFS role  https://review.opendev.org/69851411:45
openstackgerritSergii Golovatiuk proposed openstack/paunch master: Fix container config IDs lookup by lists vs str  https://review.opendev.org/69728011:48
*** mmethot has quit IRC11:52
*** mmethot has joined #tripleo11:52
*** mmethot has quit IRC11:58
*** SurajPatil has joined #tripleo12:00
*** rfolco has joined #tripleo12:01
*** surpatil has quit IRC12:02
openstackgerritChandan Kumar (raukadah) proposed openstack/tripleo-ci master: Replace ostempest job in favor of plain standalone  https://review.opendev.org/69799212:03
*** jpena is now known as jpena|lunch12:04
openstackgerritMerged openstack/tripleo-ansible stable/train: Copy all_nodes.json instead of rendering template  https://review.opendev.org/69841512:07
*** waleedm_ has quit IRC12:08
*** mmethot has joined #tripleo12:09
*** shyamb has quit IRC12:11
openstackgerritSagi Shnaidman proposed openstack/tripleo-ansible master: tripleo-container-manage: include more arguments for container exec  https://review.opendev.org/69848012:11
openstackgerritC├ędric Jeanneret (Tengu) proposed openstack/tripleo-common master: Remove workdir only if it's a temporary one  https://review.opendev.org/69870912:14
sshnaidmcloudnull, take a look please: https://review.opendev.org/#/c/698550/12:15
*** jpich has quit IRC12:19
*** jpich has joined #tripleo12:19
*** hberaud|lunch is now known as hberaud12:20
*** dpawlik has quit IRC12:22
*** dsneddon has quit IRC12:28
*** raildo has joined #tripleo12:28
openstackgerritSagi Shnaidman proposed openstack/tripleo-ansible master: Fix typo that prevents to run tests locally  https://review.opendev.org/69871112:30
*** Goneri has quit IRC12:31
*** epoojad1 is now known as epoojad1|afk12:33
sshnaidmowalsh, ^^12:33
*** ansmith_ has quit IRC12:34
*** akahat has quit IRC12:35
*** sshnaidm has quit IRC12:37
*** akahat has joined #tripleo12:37
*** epoojad1|afk has quit IRC12:38
*** sshnaidm has joined #tripleo12:38
*** dpawlik has joined #tripleo12:46
*** soniya29 has quit IRC12:55
*** rlandy has joined #tripleo12:56
*** bfournie has quit IRC12:57
openstackgerritmathieu bultel proposed openstack/tripleo-quickstart-extras master: Add when check for validate services include  https://review.opendev.org/69866312:58
*** dtantsur|brb is now known as dtantsur12:58
*** shyamb has joined #tripleo12:59
*** dsneddon has joined #tripleo13:00
*** dmacpher has quit IRC13:01
*** amoralej is now known as amoralej|lunch13:02
*** dmacpher has joined #tripleo13:02
*** rh-jelabarre has joined #tripleo13:04
*** bhagyashris has quit IRC13:08
openstackgerritTakashi Kajinami proposed openstack/puppet-tripleo master: Add support to configure virtlogd  https://review.opendev.org/68534913:10
EmilienMsshnaidm: yes go for squash13:14
*** surpatil has joined #tripleo13:15
*** jpena|lunch is now known as jpena13:15
*** shyamb has quit IRC13:15
EmilienMdciabrin_: I think I know what caused https://bugs.launchpad.net/tripleo/+bug/185609013:16
openstackLaunchpad bug 1856090 in tripleo "paunch in master generates invalid config_id when creating containers" [Undecided,New]13:16
dciabrin_EmilienM, oh13:16
EmilienMdciabrin_: https://bugs.launchpad.net/tripleo/+bug/185509013:17
openstackLaunchpad bug 1855090 in tripleo "Minor updates in train and U fail being applied by paunch" [High,In progress] - Assigned to Sergii Golovatiuk (sgolovatiuk)13:17
EmilienMhttps://review.opendev.org/#/c/697280/13:17
*** SurajPatil has quit IRC13:17
EmilienMit looks like the same bug as you13:17
*** sshnaidm has quit IRC13:18
openstackgerritDougal Matthews proposed openstack/tripleo-common master: Remove workdir only if it's a temporary one  https://review.opendev.org/69870913:19
*** sshnaidm has joined #tripleo13:19
openstackgerritDougal Matthews proposed openstack/tripleo-common master: Remove workdir only if it's a temporary one  https://review.opendev.org/69870913:19
openstackgerritDougal Matthews proposed openstack/tripleo-common master: Remove workdir only if it's a temporary one  https://review.opendev.org/69870913:20
EmilienMdciabrin_: bogdan isn't online but I think the issue comes from https://review.opendev.org/#/c/695921/4/tripleo_ansible/ansible_plugins/modules/paunch.py13:20
EmilienMand he's workaround it by doing another ps with the '['foo']'13:20
EmilienMwhich is imho not a good idea and is expensive13:21
dciabrin_EmilienM, ok either way when you have an env and some time, if we can discuss a bit the restart of service, yesterday i tried something and it didn't work as I expected13:22
*** ade_lee_ has joined #tripleo13:23
*** mmethot has quit IRC13:24
*** ade_lee has quit IRC13:26
*** mcornea has joined #tripleo13:26
*** ade_lee_ has quit IRC13:26
*** ade_lee_ has joined #tripleo13:26
openstackgerritDougal Matthews proposed openstack/tripleo-common master: Remove workdir only if it's a temporary one  https://review.opendev.org/69870913:28
*** mmethot has joined #tripleo13:30
*** ade_lee_ has quit IRC13:32
*** ade_lee_ has joined #tripleo13:32
*** amoralej|lunch is now known as amroalej13:32
*** amroalej is now known as amoralej13:32
EmilienMchem, ccamacho|pto, matbu : https://review.opendev.org/#/c/697280/2/paunch/runner.py FYI see my comment13:33
cloudnullmornings13:33
cloudnullsshnaidm sure13:33
openstackgerritDougal Matthews proposed openstack/tripleo-common master: Remove workdir only if it's a temporary one  https://review.opendev.org/69870913:34
weshaypanda, do you have logs w/ errors from quay?13:35
*** ansmith has joined #tripleo13:36
chemEmilienM: noted. This is low priority in any case, so better have a less costly solution, you're right.13:36
cloudnullsshnaidm done13:36
*** ade_lee_ has quit IRC13:37
*** fmount has quit IRC13:37
*** fmount has joined #tripleo13:37
sshnaidmcloudnull, and this please https://review.opendev.org/#/c/698711/13:40
openstackgerritEmilien Macchi proposed openstack/tripleo-common master: Fix ironic-pxe container healthcheck  https://review.opendev.org/69871913:41
*** morazi has joined #tripleo13:44
*** Goneri has joined #tripleo13:45
pandaweshay: they are in the temp promoter, but mostly look like this13:47
pandaweshay: 2019-12-12 13:24:03,782 p=6796 u=centos |  failed: [localhost] (item=[u'current-tripleo-rdo', u'redis']) => {"ansible_loop_var": "item", "changed": false, "item": ["current-tripleo-rdo", "redis"], "msg": "Error pushing image quay.io/tripleoqueens/centos-binary-redis: received unexpected HTTP status: 500 Internal Server Error"}13:47
pandaweshay: or 2019-12-12 13:22:50,125 p=6796 u=centos |  failed: [localhost] (item=[u'current-tripleo-rdo', u'qdrouterd']) => {"ansible_loop_var": "item", "changed": false, "item": ["current-tripleo-rdo", "qdrouterd"], "msg": "Error pushing image quay.io/tripleoqueens/centos-binary-qdrouterd: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out."}13:47
weshayEmilienM, ^13:49
*** ykarel is now known as ykarel|afk13:49
EmilienMweshay: we need to report it to quay.io13:49
EmilienMI have contacts13:49
*** bfournie has joined #tripleo13:50
openstackgerritKevin Carter (cloudnull) proposed openstack/tripleo-heat-templates master: scale: fixes for compute scale down  https://review.opendev.org/69851513:54
*** dmacpher has quit IRC13:56
openstackgerritDougal Matthews proposed openstack/python-tripleoclient master: WIP Start moving the import CLI command to the Ansible implementation  https://review.opendev.org/69872313:57
*** pcaruana has quit IRC13:57
*** dpawlik has quit IRC13:58
*** sshnaidm has quit IRC14:04
openstackgerritFrancesco Pantano proposed openstack/tripleo-validations master: Add a validation to fail if a Ceph dependency is not installed  https://review.opendev.org/69745314:06
*** ade_lee has joined #tripleo14:09
slaweqmarios|ruck: this job is failing with different reasons all the time, like e.g. https://logs.rdoproject.org/19/23919/3/check/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-master/8466f20/job-output.txt :/14:10
marios|ruckslaweq: yeah it still fails on the checksum thing14:10
openstackgerritSoniya Murlidhar Vyas proposed openstack/tripleo-quickstart-extras master: [train] Remove passing tests from skiplist  https://review.opendev.org/69872414:10
slaweqmarios|ruck: yes, now I see14:10
marios|ruckslaweq: i will try the alternative in a sec14:10
marios|ruckchkumar|rover: ^^^ still failed there fyi (will add dependencies instead ) https://review.rdoproject.org/r/#/c/23919/14:11
chkumar|rovermarios|ruck: yes all 314:11
*** hjensas has quit IRC14:11
marios|ruckchkumar|rover: ack14:11
chkumar|rovercontainer build, ipa and overcloud image14:11
chkumar|rovermarios|ruck: I am taking a look at overcloud image build14:11
marios|ruckchkumar|rover: why container build?14:12
marios|ruckchkumar|rover: i think just overcloud and ipa14:12
*** dsneddon has quit IRC14:13
openstackgerritLuca Miccini proposed openstack/tripleo-heat-templates stable/queens: Properly set loopback_users via rabbitmq::loopback_users  https://review.opendev.org/69865514:13
openstackgerritLuca Miccini proposed openstack/tripleo-heat-templates stable/rocky: Properly set loopback_users via rabbitmq::loopback_users  https://review.opendev.org/69865314:13
*** ykarel|afk has quit IRC14:13
*** ade_lee_ has joined #tripleo14:15
*** ade_lee has quit IRC14:15
*** epoojad1 has joined #tripleo14:16
openstackgerritSoniya Murlidhar Vyas proposed openstack/tripleo-quickstart-extras master: [Train] Remove passing tests from skiplist  https://review.opendev.org/69638714:17
*** aedc has joined #tripleo14:17
*** ade_lee_ has quit IRC14:17
*** ade_lee_ has joined #tripleo14:17
openstackgerritKevin Carter (cloudnull) proposed openstack/tripleo-common master: Revert "Update hostvars lookup to fix regression"  https://review.opendev.org/69872614:18
*** ade_lee_ has quit IRC14:19
*** ade_lee_ has joined #tripleo14:19
marios|ruckchkumar|rover: wdyt https://review.rdoproject.org/r/#/c/23919/4/.zuul.yaml  cc slaweq fyi let's see :/14:20
*** akahat has quit IRC14:20
*** ade_lee_ has quit IRC14:21
openstackgerritKevin Carter (cloudnull) proposed openstack/tripleo-common stable/train: Revert "Update hostvars lookup to fix regression"  https://review.opendev.org/69872714:21
*** ade_lee_ has joined #tripleo14:21
*** ade_lee__ has joined #tripleo14:24
openstackgerritJames Slagle proposed openstack/tripleo-common master: Revert "Update hostvars lookup to fix regression"  https://review.opendev.org/69872614:25
*** ade_lee_ has quit IRC14:26
*** ekultails has joined #tripleo14:27
*** dsneddon has joined #tripleo14:30
openstackgerritKevin Carter (cloudnull) proposed openstack/tripleo-ansible master: Update firewall role to use an action plugin  https://review.opendev.org/69862014:30
openstackgerritKevin Carter (cloudnull) proposed openstack/tripleo-common stable/train: Revert "Update hostvars lookup to fix regression"  https://review.opendev.org/69872714:32
*** dmacpher has joined #tripleo14:34
*** dsneddon has quit IRC14:35
*** pcaruana has joined #tripleo14:36
openstackgerritJames Slagle proposed openstack/tripleo-common master: Add deploy_steps_tasks to PER_TASK_STEPS  https://review.opendev.org/69861414:42
*** TrevorV has joined #tripleo14:43
openstackgerritKevin Carter (cloudnull) proposed openstack/tripleo-ansible master: Update firewall role to use an action plugin  https://review.opendev.org/69862014:48
*** sshnaidm has joined #tripleo14:49
chkumar|rovermarios|ruck: looks good,14:51
marios|ruckchkumar|rover: oh ffs it fails image build on the network thing14:51
marios|ruckslaweq: :/ http://logs.rdoproject.org/19/23919/4/check/periodic-tripleo-rhel-8-buildimage-overcloud-full-master/a8e2de1/build.log14:52
marios|ruckchkumar|rover: ^14:52
marios|ruckchkumar|rover: 2019-12-12 14:43:29.467 | Failed to open connection to "system" message bus: Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory14:52
marios|ruckchkumar|rover: slaweq: so now we are blocked on https://bugs.launchpad.net/tripleo/+bug/1853028 (yet another bug)14:52
openstackLaunchpad bug 1853028 in tripleo "Build overcloud image for rhel8 fails sometimes on in_target.d/post-install.d/51-enable-network-service" [Critical,Triaged]14:52
* marios|ruck sigh recheck 14:53
*** ccamacho|pto has quit IRC14:54
*** hjensas has joined #tripleo14:56
chkumar|rovermarios|ruck: I will have reproducer by tomororw afternoon working on reproducing it14:56
*** florianf has quit IRC15:00
slaweqmarios|ruck: ok, can You ping me when it will be ready to debug for us?15:02
marios|ruckslaweq: yeah of course15:02
slaweqmarios|ruck: thx a lot15:02
slaweqbtw. I have another question about tripleo jobs15:02
slaweqin neutron we have one tripleo non-voting job which is failing 100% times since some time https://zuul.opendev.org/t/openstack/build/7a37089a26584c588555b1e049a93f7b15:03
slaweqit's failing because it's using python 2.7 and neutron now can't run on it anymore15:03
slaweqis there any way how I can/should modify this job to use python3 on cenots 7 (or 8)?15:03
*** epoojad1 has quit IRC15:03
openstackgerritJames Slagle proposed openstack/tripleo-common master: Add deploy_steps_tasks to PER_TASK_STEPS  https://review.opendev.org/69861415:04
chkumar|roverslaweq: interesting15:04
marios|ruckslaweq: well centos8 once we get it should be ok. we actually had to pin neutron for centos because of that15:05
chkumar|roverslaweq: time to change the job definition to move to validate temepst15:05
marios|ruckslaweq: https://bugs.launchpad.net/tripleo/+bug/1855655/comments/115:05
openstackLaunchpad bug 1855655 in tripleo "FS01 master overcloud image prepare time out due to 'module' object has no attribute 'abc' in dhcp agent" [Critical,In progress]15:05
chkumar|rovermarios|ruck: failed at validate-tempest15:05
chkumar|roverproposing a patch15:05
slaweqchkumar|rover: I will check how to switch this job to use validate tempest role15:06
paramiteHey guys, could we please move forward with https://review.opendev.org/#/c/695859/ and https://review.opendev.org/#/c/696071/ ?15:06
marios|ruckchkumar|rover: slaweq: ah so it isn't cos of py3?15:06
marios|ruckchkumar|rover: or it is just tempest specific?15:06
chkumar|roverslaweq: proposing a patch now15:06
slaweqmarios|ruck: we can't pin it as we want to run it on every patch so that's not an option for us :)15:06
slaweqchkumar|rover: thx15:06
marios|ruckslaweq: yeah i realise i was just saying we hit an issue with neutron/py3 recently sounds related15:07
chkumar|rovermarios|ruck: failing on validate-tempest15:08
*** dsneddon has joined #tripleo15:08
*** dtantsur is now known as dtantsur|brb15:10
*** artom has joined #tripleo15:15
chkumar|rovermarios|ruck: slaweq https://review.opendev.org/69873715:20
chkumar|roverif the error still come, I need to get it fixed then15:20
marios|ruckchkumar|rover: ack15:22
openstackgerritSagi Shnaidman proposed openstack/tripleo-ansible master: Introduce tripleo-keystone-resources role  https://review.opendev.org/69716115:22
sshnaidmEmilienM, ^^15:22
weshayTengu, I never say no to help.. I can share notes and bring you into context if you want15:23
Tenguweshay: ok :)15:25
EmilienMsshnaidm: thank you15:25
sshnaidmEmilienM, cloudnull it will require fixes in role itself https://review.opendev.org/#/c/697161/ but we'll have a real testing with live keystone container15:25
Tenguweshay: I have to talk with you anyway for some CI things about selinux :)15:25
EmilienMsshnaidm: wow15:25
EmilienMsshnaidm: that's awesome15:25
EmilienMnicely done15:26
*** leanderthal has joined #tripleo15:27
*** florianf has joined #tripleo15:28
*** saneax has quit IRC15:31
*** dsneddon has quit IRC15:32
*** surpatil has quit IRC15:33
*** saneax has joined #tripleo15:33
*** bhagyashris has joined #tripleo15:33
openstackgerritMarios Andreou proposed openstack/tripleo-quickstart-extras master: DNM test https://tree.taiga.io/project/tripleo-ci-board/task/1315  https://review.opendev.org/68726715:42
*** ykarel|afk has joined #tripleo15:48
*** ramishra has quit IRC15:48
*** chkumar|rover is now known as raukadah|r15:49
*** ramishra has joined #tripleo15:50
*** raukadah|r is now known as raukadah15:50
*** leanderthal has quit IRC15:51
*** leanderthal has joined #tripleo15:51
*** dtantsur|brb is now known as dtantsur15:52
EmilienMsshnaidm: sounds like it's failing for a valid reason15:53
EmilienM msg: 'Unsupported parameters for (os_keystone_role) module: domain_id Supported parameters include: api_timeout, auth, auth_type, availability_zone, ca_cert, client_cert, client_key, cloud, interface, name, region_name, state, timeout, validate_certs, wait'15:53
EmilienMsshnaidm: i'll fix15:53
sshnaidmEmilienM, yep, no such argument15:53
openstackgerritEmilien Macchi proposed openstack/tripleo-ansible master: Introduce tripleo-keystone-resources role  https://review.opendev.org/69716115:54
*** bhagyashris_ has joined #tripleo15:54
*** bhagyashris has quit IRC15:55
openstackgerritJames Slagle proposed openstack/tripleo-heat-templates master: Execute deploy_steps_tasks per step  https://review.opendev.org/69861515:56
EmilienMsshnaidm: why do you create clouds.yaml here https://review.opendev.org/#/c/697161/34/tripleo_ansible/roles/tripleo-keystone-resources/molecule/default/prepare.yml15:57
EmilienMsshnaidm: this task can do it for you https://review.opendev.org/#/c/697161/34/tripleo_ansible/roles/tripleo-keystone-resources/tasks/clouds.yml15:57
EmilienMwe use that ^15:57
EmilienMwe should* use that ^15:57
sshnaidmEmilienM, first calls to ansible use "cloud: openstack"15:58
sshnaidmEmilienM, so you need it in the beginning15:58
*** bhagyashris_ has quit IRC15:59
EmilienMsshnaidm: i don't get it15:59
sshnaidmEmilienM, first os_keystone task uses "cloud: openstack"16:00
EmilienMsshnaidm: in a customer call now, brb16:01
sshnaidmEmilienM, I'm in mtg too16:01
openstackgerritMarios Andreou proposed openstack/tripleo-quickstart-extras master: DNM test https://tree.taiga.io/project/tripleo-ci-board/task/1315  https://review.opendev.org/68726716:01
openstackgerritmathieu bultel proposed openstack/tripleo-quickstart-extras master: Add when check for validate services include  https://review.opendev.org/69866316:03
EmilienMsshnaidm: what I mean is that we need to call the clouds.yaml tasks before "Create Keystone Admin resources"16:04
EmilienMbefore the main16:04
*** hjensas has quit IRC16:04
EmilienMmaybe we need to call clouds.yaml from main16:04
EmilienMthat would be easier I guess16:04
openstackgerritMerged openstack/tripleo-ansible master: Rename undercloud-backup playbook  https://review.opendev.org/69558316:05
openstackgerritMerged openstack/tripleo-ansible master: Fix typo that prevents to run tests locally  https://review.opendev.org/69871116:05
openstackgerritMerged openstack/tripleo-ansible master: Bump molecule to 3.0a4  https://review.opendev.org/69855016:05
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates stable/queens: DNM test https://tree.taiga.io/project/tripleo-ci-board/task/1315  https://review.opendev.org/69874716:05
*** jpena is now known as jpena|off16:07
openstackgerritLukas Bezdicka proposed openstack/tripleo-upgrade stable/train: Update: Run all roles update at once  https://review.opendev.org/69752016:10
*** leanderthal has quit IRC16:12
*** leanderthal has joined #tripleo16:13
*** florianf has quit IRC16:14
*** irclogbot_1 has quit IRC16:14
*** irclogbot_0 has joined #tripleo16:16
*** dsneddon has joined #tripleo16:19
EmilienMsshnaidm: i'm going to send a patch16:26
*** leanderthal has quit IRC16:26
openstackgerritEmilien Macchi proposed openstack/tripleo-ansible master: Introduce tripleo-keystone-resources role  https://review.opendev.org/69716116:38
*** florianf_ has joined #tripleo16:42
*** marios|ruck has quit IRC16:45
openstackgerritMerged openstack/diskimage-builder master: Set correct python version for non-chroot scripts  https://review.opendev.org/69721116:48
*** tesseract has quit IRC16:48
*** lmiccini has quit IRC16:51
*** holser has quit IRC16:51
*** lucasagomes has quit IRC16:52
*** holser has joined #tripleo16:52
*** bhagyashris has joined #tripleo16:55
*** alexmcleod has quit IRC16:56
openstackgerritJose Luis Franco proposed openstack/tripleo-upgrade stable/train: Add LEAPP_SKIP_CHECK_OS_RELEASE variable in UC upgrade.  https://review.opendev.org/69875816:56
openstackgerritOliver Walsh proposed openstack/tripleo-ansible master: Add role/playbook to manage nova image cache  https://review.opendev.org/67625216:56
*** yprokule has quit IRC16:57
EmilienMweshay: have you reached out to support@quay.io yet?17:01
*** ykarel|afk has quit IRC17:02
*** raildo has quit IRC17:05
*** florianf_ has quit IRC17:05
*** raildo has joined #tripleo17:05
weshayEmilienM, no.. I need to chat w/ panda17:05
*** raildo has quit IRC17:06
*** raildo has joined #tripleo17:06
*** bandini has quit IRC17:07
*** jtomasek has quit IRC17:08
*** iurygregory has quit IRC17:11
*** raildo has quit IRC17:11
*** raildo has joined #tripleo17:13
*** sshnaidm has quit IRC17:18
openstackgerritOliver Walsh proposed openstack/tripleo-ansible master: Add role/playbook to manage nova image cache  https://review.opendev.org/67625217:20
*** jpena|off is now known as jpena17:27
openstackgerritCarlos Goncalves proposed openstack/diskimage-builder master: Update bindep for RHEL/CentOS 8  https://review.opendev.org/69844817:29
*** iurygregory has joined #tripleo17:32
openstackgerritKevin Carter (cloudnull) proposed openstack/tripleo-ansible master: Update firewall role to use an action plugin  https://review.opendev.org/69862017:33
openstackgerritBrent Eagles proposed openstack/tripleo-common master: WIP: Allow different namespace and version tag of octavia container images  https://review.opendev.org/69877517:34
*** dmacpher_ has joined #tripleo17:35
*** dtantsur is now known as dtantsur|afk17:37
*** jpich has quit IRC17:37
*** dmacpher has quit IRC17:38
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart master: fs010/undercloud: disable paunch  https://review.opendev.org/69407517:41
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: CI/fs010: disable Paunch on the overcloud  https://review.opendev.org/69407617:41
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: CI/fs010: disable Paunch on the overcloud  https://review.opendev.org/69407617:41
*** bhagyashris has quit IRC17:43
*** suuuper has quit IRC17:45
openstackgerritEmilien Macchi proposed openstack/tripleo-ansible master: Introduce tripleo-keystone-resources role  https://review.opendev.org/69716117:48
*** rpittau is now known as rpittau|afk17:49
*** raildo has quit IRC17:50
*** dmacpher_ has quit IRC17:50
*** raildo has joined #tripleo17:51
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Deploy /etc/openstack/clouds.yaml with Ansible  https://review.opendev.org/69639017:53
*** hberaud is now known as hberaud|gone17:53
openstackgerritOliver Walsh proposed openstack/tripleo-ansible master: Add role/playbook to manage nova image cache  https://review.opendev.org/67625217:55
mrungeHi, can I get another +2/+A for a patch to make legacy-telemetry installable? https://review.opendev.org/#/c/698213/17:58
mrunge*pretty please*?17:58
*** derekh is now known as derekh_afk18:00
*** psachin has quit IRC18:09
*** ayoung has quit IRC18:09
openstackgerritKevin Carter (cloudnull) proposed openstack/tripleo-heat-templates stable/train: scale: fixes for compute scale down  https://review.opendev.org/69878418:15
*** gfidente has quit IRC18:21
*** tmazur has quit IRC18:29
openstackgerritEmilien Macchi proposed openstack/tripleo-ansible master: Introduce tripleo-keystone-resources role  https://review.opendev.org/69716118:29
m1pHi, what's the preferred way to modify a container (e.g. horizon to add a theme) in a persistent way?18:30
*** jpena is now known as jpena|off18:36
*** saneax has quit IRC18:37
EmilienMm1p: this should help you: https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/deployment/container_image_prepare.html#modify-with-custom-dockerfile18:41
openstackgerritMerged openstack/tripleo-heat-templates master: Increase rabbitmq tcp backlog  https://review.opendev.org/69682718:41
m1pEmilienM: ok, so custom container. I was afraid there's no easier way. Thank's a lot!18:42
*** amoralej is now known as amoralej|off18:43
EmilienMm1p: also there is a nice new blog post about building containers: https://www.redhat.com/sysadmin/building-buildah18:44
m1pnice, didn't use buildah before, will give it a try18:46
EmilienMm1p: buildah and podman are replacing docker since openstack stein in tripleo18:52
EmilienMm1p: they also have an IRC channel #podman if you need specific support.18:54
m1pthanks. still using docker with train, I'm waiting for centos 8 to replace it entirely with podman18:57
ekultailsAny Ceph experts around? In what release did we switch to only having ceph-tools installed in the ceph-container? Was it Train?18:58
ekultailsThis is in regards to https://review.gerrithub.io/c/rhos-infra/cloud-config/+/47784818:58
ekultails^ fultonj18:59
fultonjekultails: stein19:00
fultonjekultails: i can comment in the review19:01
openstackgerritEmilien Macchi proposed openstack/tripleo-ansible master: Introduce tripleo-keystone-resources role  https://review.opendev.org/69716119:01
ekultailsfultonj: Great, thanks for the clarification!19:04
fultonjekultails: i added a comment about the container name too19:04
fultonjusing ceph-mon-`hostname` will work on the nodes which have the CephMon service19:04
openstackgerritOliver Walsh proposed openstack/tripleo-ansible master: Add role/playbook to manage nova image cache  https://review.opendev.org/67625219:04
fultonjpodman exec ceph-mon-`hostname` ceph -s19:05
fultonjfor example ^19:05
fultonjthat's what ceph-ansible names the container19:05
ekultails+1 I appreciate the clarification/help.19:05
*** morazi has quit IRC19:07
*** sshnaidm has joined #tripleo19:09
*** tosky has quit IRC19:21
*** iurygregory has quit IRC19:22
*** iurygregory has joined #tripleo19:23
openstackgerritMerged openstack/tripleo-ansible master: tripleo-config: add tripleo_config_remote_src option  https://review.opendev.org/69808419:23
openstackgerritCarlos Goncalves proposed openstack/diskimage-builder master: DNM: Fix yumdownloader cache dir  https://review.opendev.org/69878819:23
openstackgerritJames Slagle proposed openstack/tripleo-heat-templates master: Execute deploy_steps_tasks per step  https://review.opendev.org/69861519:27
*** ramishra has quit IRC19:28
*** artom has quit IRC19:30
*** mmethot has quit IRC19:40
openstackgerritEmilien Macchi proposed openstack/tripleo-ansible master: Introduce tripleo-keystone-resources role  https://review.opendev.org/69716119:43
openstackgerritAlex Schultz proposed openstack/python-tripleoclient master: Retry the connection to heat-api when 500 code  https://review.opendev.org/69793919:46
EmilienMmwhahaha: i stopped counting how many times we write a new function to retry something19:50
mwhahahaat this point tenacity should probably be on all the functions19:51
* mwhahaha dies a bit more19:51
EmilienMwhat I mean is that we should stop encourage these functions19:51
EmilienMand use tenacity when possible19:51
EmilienMI think it's the way to go19:51
mwhahahaor start addressing the real problem and not hiding it with retries19:51
EmilienMi think we agree here19:52
cloudnullqq - looking at https://review.opendev.org/#/c/698620/ its failing the multinode job because the module I'm adding couldn't be found. Though when I test in standalone it seems to work just fine. From what I can tell, I don't think the multinode job is rebuilding the package with the change? Is there a way around this?19:53
cloudnull-cc weshay ^19:53
mwhahahathe multinodes should get the rebuilt package19:53
mwhahahamaybe it's broken for tripleo-ansible? though seems odd19:54
cloudnullhttps://1f60b37dcbd00092b7dc-4a63bffb654cd83f9f61f31a50d77a27.ssl.cf5.rackcdn.com/698620/5/check/tripleo-ci-centos-7-containers-multinode/51ce463/19:54
cloudnullthat's the job failure19:54
cloudnullhttps://1f60b37dcbd00092b7dc-4a63bffb654cd83f9f61f31a50d77a27.ssl.cf5.rackcdn.com/698620/5/check/tripleo-ci-centos-7-containers-multinode/51ce463/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz19:54
cloudnullhas the error at the bottom of the file19:54
mwhahahanot in the container?19:55
cloudnullthat one hasn't finished yet, but its running longer than the multi-node job so I suspect it got further19:55
cloudnullI've been running this patch in standalone locally19:55
cloudnulloh mwhahaha -- I see what you're saying.19:58
* cloudnull was thinking you were talking about - tripleo-ci-centos-7-undercloud-containers 19:58
cloudnullso how do we get modules we write into the container during testing ?19:59
mwhahahait usually should be19:59
mwhahahaunless we're not running the update containers bits on tripleo-ansible19:59
cloudnullbecause that would make some sense if the module wasn't found in the mistral container when the playbook executes19:59
mwhahahait might be a version issue as well19:59
cloudnullis there something I can go poke at?20:01
mwhahahathe logs? :D20:03
* mwhahaha looks20:03
openstackgerritMerged openstack/tripleo-ansible master: tripleo-container-manage: include more arguments for container exec  https://review.opendev.org/69848020:04
mwhahahatripleo-container-image-prepare.log specifically20:04
mwhahahathat'll have the yum update output20:04
mwhahahato make sure it' supdating the container with tripleo-ansible20:04
mwhahahaso mistral-executor got it20:05
mwhahaha2019-12-12 18:24:05,868 19566 INFO tripleo_common.image.image_uploader [  ]       tripleo-ansible.noarch 0:1.0.1-0.20191212180346.e9cdd81.el720:06
mwhahahaso now we need to check to make sure the updated executor container was used20:06
mwhahahac9c6df39cdc8  192.168.24.1:8787/tripleomaster/centos-binary-mistral-executor:38c4e3104abdeb4699cfbe7a78fa2f37d7a863b4_93bde36c-updated-20191212175625           kolla_start           24 minutes ago  Up 24 minutes ago                 mistral_executor                                         52.8MB (virtual 1.73GB)20:07
mwhahahaso maybe it's not getting picked up in tripleo-common? do we need to ensure it ends up in a specific spot?20:07
* mwhahaha shrugs20:07
cloudnullthe module should be installed in normal spot20:08
*** iurygregory has quit IRC20:08
cloudnullwe have the package module in their too20:08
cloudnullit should be in share/ansible/plugins/20:09
openstackgerritLuke Short proposed openstack/paunch master: Retry deleting containers.  https://review.opendev.org/69769620:09
openstackgerritLuke Short proposed openstack/paunch master: Retry deleting containers.  https://review.opendev.org/69769620:10
cloudnullhttps://github.com/openstack/tripleo-common/blob/e84323ad1c0efef7accedb89e38b6e93499f5f9f/tripleo_common/actions/ansible.py#L6520:10
cloudnullwhich looks like it should be looking in the right place20:10
mwhahahahttps://1f60b37dcbd00092b7dc-4a63bffb654cd83f9f61f31a50d77a27.ssl.cf5.rackcdn.com/698620/5/check/tripleo-ci-centos-7-containers-multinode/51ce463/logs/undercloud/var/log/extra/podman/containers/mistral_executor/podman_info.log.txt.gz so it installed it from gating-repo which is the built package20:13
mwhahahamy assumption is that it didn't end up in the package for whatever reason but that should show up on the the undercloud or something20:14
*** pcaruana has quit IRC20:19
openstackgerritEmilien Macchi proposed openstack/tripleo-ansible master: Introduce tripleo-keystone-resources role  https://review.opendev.org/69716120:22
cloudnullmwhahaha is there a way I could reproduce that locally ?20:24
cloudnullor do we think its a matter of just rechecking ?20:24
openstackgerritEmilien Macchi proposed openstack/tripleo-ansible master: Introduce tripleo-keystone-resources role  https://review.opendev.org/69716120:24
openstackgerritLuke Short proposed openstack/paunch master: Retry deleting containers.  https://review.opendev.org/69769620:24
mwhahahacloudnull: if you installed an undercloud and built the rpm with the change20:25
mwhahahacloudnull: rdo cloud might reproduce it (not sure the state of the reproducer)20:25
openstackgerritCarlos Goncalves proposed openstack/diskimage-builder master: Install rng-tools in Red Hat family distro images  https://review.opendev.org/69718320:26
cloudnullmwhahaha would running this (with a built rpm) in standalone be enough ?20:26
cloudnullor will there need to be an overcloud to trigger the error ?20:27
mwhahahano20:27
mwhahahayou need to deploy an overcloud20:27
cloudnullOK20:27
cloudnullwill go poke at it20:27
*** jfrancoa has quit IRC20:27
mwhahahai thought we captured the rpm in the logs that you could use20:27
mwhahahathat might have only been for ovb tho20:27
*** holser has quit IRC20:32
ade_lee__mwhahaha, hey -- I'm trying to do an overcloud install and got stuck doing a tripleo-container-image-prepare20:38
ade_lee__mwhahaha, seems to have exceeded memory and something was oom-killed20:38
ade_lee__mwhahaha, I have 16G on my undercloud -- any reason that might have happened?20:38
mwhahahaadd swap20:38
mwhahahathe min requirement is like 24g20:39
mwhahahahas been since rocky20:39
mwhahahaor add swap (like a good person)20:39
ade_lee__mwhahaha, yeeesh .. ok - will do20:39
*** derekh_afk has quit IRC20:48
*** pkopec has quit IRC20:57
*** morazi has joined #tripleo21:00
weshaymwhahaha, cloudnull the rpm is not saved, just the build logs21:00
openstackgerritJames Slagle proposed openstack/tripleo-common master: Make ()'s around step|int optional  https://review.opendev.org/69880421:01
weshaycloudnull, did you get it worked out?  tripleo-ansible-1.0.1-0.20191212180346.e9cdd81.el7.x86_6421:01
*** TrevorV has quit IRC21:02
cloudnullweshay not yet21:02
cloudnullI can build the RPM if you want it ?21:02
weshaycloudnull, so.. I should read the scrollback more carefully.. are you saying it should be installed on the overcloud?21:03
weshayhttps://1f60b37dcbd00092b7dc-4a63bffb654cd83f9f61f31a50d77a27.ssl.cf5.rackcdn.com/698620/5/check/tripleo-ci-centos-7-containers-multinode/51ce463/logs/subnode-1/var/log/extra/rpm-list.txt.gz21:03
cloudnullI think it needs to be installed in the mistral container21:04
weshayoh21:04
* weshay looks21:04
cloudnullhttps://review.opendev.org/#/c/698620/ - so it passed the containers job but not the multinode one21:04
weshaytripleo-ansible.noarch             1.0.1-0.20191212180346.e9cdd81.el7  @gating-repo21:05
weshaylooks like it is21:05
* weshay gets better log21:05
weshayhttps://zuul.opendev.org/t/openstack/build/51ce4636ee4f4e1185ccd433496fa50b/log/logs/undercloud/var/log/extra/podman/containers/mistral_executor/podman_info.log.txt.gz#92621:06
weshaycloudnull, ^21:06
dciabrin_EmilienM, mwhahaha so it looks like https://review.opendev.org/#/c/698580/ failed in the gate in a update job (https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_6f3/698580/1/gate/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/6f39b33/), and I think it's due to the infamous podman bug https://bugzilla.redhat.com/show_bug.cgi?id=175784521:09
openstackbugzilla.redhat.com bug 1757845 in podman "You have to remove that container to be able to reuse that name.: that name is already in use (due to exec user process caused "no such file or directory")" [High,Assigned] - Assigned to nalin21:09
*** EmilienM is now known as EmilienM|mtg21:10
*** mcornea has quit IRC21:14
*** EmilienM|mtg is now known as EmilienM21:15
EmilienMdciabrin_: yeah ekultails is workign on a paunch patch https://review.opendev.org/69769621:15
EmilienMmight be helpful21:16
openstackgerritLuke Short proposed openstack/tripleo-heat-templates master: Use paunch to handle container removal.  https://review.opendev.org/69880621:16
dciabrin_EmilienM, no i think it won't. the container has gone missing in "podman ps -a", i think it's a leftover in container storage, which prevents creating new containers with the same name21:17
EmilienM--storage would help?21:17
mwhahahaweshay: is the rhel8 build job run on a rhel8 host or centos7 host?21:20
dciabrin_most probably yes. and then sometimes the "rm -f --storage" might fail if we're unlucky as in https://bugzilla.redhat.com/show_bug.cgi?id=1757845#c3121:20
openstackbugzilla.redhat.com bug 1757845 in podman "You have to remove that container to be able to reuse that name.: that name is already in use (due to exec user process caused "no such file or directory")" [High,Assigned] - Assigned to nalin21:20
EmilienMdciabrin_: lol21:20
* EmilienM depressed21:21
EmilienMekultails: ^ fyi21:21
*** rfolco has quit IRC21:21
ekultailsI'm following along.21:21
dciabrin_EmilienM,  :/ but yeah to get rid of the error in the job, we must run 'rm -f --storage' if 'rm -f' fails21:21
ekultailsThere are a lot of gotchas here.21:21
dciabrin_ekultails, o/21:21
ekultailsdciabrin_: Hey there :-)21:21
weshaymwhahaha, anything rhel8 is on a rhel8 host21:22
mwhahahahrm k21:22
dciabrin_EmilienM, ekultails i wonder if we have a logstash instance or anything which categorizes the failure in the gate, i'm curious if we're hittin that podman bug regularly or not21:22
mwhahahaweshay: 8.1 or 8.021:22
weshayah ya.. was going to mention that..21:22
weshaymwhahaha, overcloud images? is using a rhel8.0 base image qcow2 atm21:23
mwhahahaweshay: zuul claims 8.1?21:23
weshaymwhahaha, the node pool nodes... it's weird and I need to check.. I think internally we have 8.1 and rdo is on 8.021:23
mwhahaha  ansible_distribution_version: '8.1'21:23
mwhahahahttp://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-rhel-8-buildimage-overcloud-full-master/b7e6593/zuul-info/host-info.primary.yaml21:23
* weshay looks21:24
ekultailsdciabrin_: Running `rm --storage` will only work if the container was actually deleted and happens to have orphaned storage. Otherwise that command errors out. And it does not touch the container, only the storage. It refuses to delete the storage if the container exists.21:24
weshaymwhahaha, ya.. but21:24
weshaymwhahaha, http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-rhel-8-buildimage-overcloud-full-master/b7e6593/build_images.sh21:24
weshayexport DIB_LOCAL_IMAGE=/home/zuul/workspace/rhel-8.0-guest-rhui.qcow221:24
weshaymwhahaha, need that updated to 8.1?21:25
mwhahahaweshay: probably?21:25
dciabrin_ekultails, right i remember some weird things like that. in fact some time ago i worked on a workaround in container-puppet to use rm --storage only in the condition you describe.21:26
* dciabrin_ digs in his code21:26
EmilienMdciabrin_: in mtg, brb21:26
ekultailsMaybe if `podman rm` fails we can check to see if there is a storage overlay with that UUID and then run `podman rm --storage`?21:26
ekultailsYeah, I remember seeing that. I need to also take a closer look.21:26
*** Goneri has quit IRC21:27
weshaymwhahaha, you looking at the "no valid mapping"?21:27
mwhahahano the network stuff21:28
weshayChandan was looking at the rhel8 overcloud image build errors earlier.. was helping him get setup21:28
mwhahahawhat's the valid mapping?21:28
weshay2019-12-12 00:27:47.177 | ERROR:root:overcloud-base has no valid mapping for package openssl-perl21:28
ekultailsdciabrin_: It looks like you just unconditionally run `podman rm --storage` after a `podman rm`, right?21:28
mwhahahafailed to glob pattern /etc/rc0.d/[SK][0-9][0-9]network: No such file or directory21:28
weshay2019-12-12 00:27:47.220 | ERROR:root:overcloud-base has no valid mapping for package lvm221:28
weshaymwhahaha, oh.. ok.. I think chandan was going after that earlier21:28
ekultailsdciabrin_: We're trying to move that logic into Paunch so I can update the code in there to do something similar.21:28
dciabrin_ekultails, you need to duplicate the workarounds then, because the error often happens in container-puppet, when we spawn container concurrently like crazy21:29
ekultailslol Good to know, okay I can do that.21:29
weshaymwhahaha, I'll upload an 8.121:29
mwhahahaweshay: selinux is enforcing on the build job21:30
mwhahahaweshay: and we don't grab those logs, can we?21:30
weshaymwhahaha, oof.. it's in our standard collect-logs21:30
EmilienMdciabrin_: I can look for a logstash query21:30
weshaybut SOMEONE didn't want to use that21:31
* weshay runs21:31
ekultailsBy the way, our CentOS 7 builds for Train use docker, right? Not podman?21:31
weshaycentos-7 should be docker21:31
ekultails++ Thanks!21:31
mwhahahaweshay: yea can we jsut cp /var/log/audit/* rather than the world21:31
weshayit's a totally valid point21:32
dciabrin_ekultails, i only touched container-puppet, and here when we 'podman rm', we know that the container is stopped already all the time. so it's safe to rm --storage in sequence, because "we can't check whether the container is still visible in the container-storage after a successful 'rm' command" :/21:32
EmilienMekultails, weshay : what? centos7 + train == podman21:32
EmilienMyou meant ci jobs?21:33
ekultailsdciabrin_: Okay, good to know. That was a concern of mine as well. :-/21:33
ekultailsEmilienM: Mainly inquiring about CI.21:35
weshayEmilienM, oh... my bad.. I was thinking buildah vs. docker21:35
*** ansmith has quit IRC21:35
EmilienMekultails: everything is podman starting in Stein except OVB jobs they run OVB because centos7+pacemaker is not compatibile with podman21:35
weshaycentos7 is built w/ docker, but uses podman21:35
weshayrhel 8 built w/ buildah, uses podman21:36
weshaymy bad all21:36
ekultailsEmilienM: Thanks for the clarification! Wasn't 100% sure if docker was still a priority target.21:37
EmilienMekultails: no problem, let me know21:39
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Deploy /etc/openstack/clouds.yaml with Ansible  https://review.opendev.org/69639021:58
EmilienMekultails: FYI my comment on https://review.opendev.org/#/c/69880621:59
ekultailsEmilienM: I'll rebase my patch ontop of that and s/rm_container/paunch remove_container/g there.22:06
openstackgerritKevin Carter (cloudnull) proposed openstack/tripleo-quickstart-extras master: Add additional path data to our override configuration file  https://review.opendev.org/69881222:11
openstackgerritKevin Carter (cloudnull) proposed openstack/tripleo-ansible master: Update firewall role to use an action plugin  https://review.opendev.org/69862022:11
EmilienMekultails++22:11
EmilienMsshnaidm: https://review.opendev.org/#/c/697161/ is good for review now, thanks22:13
sshnaidmEmilienM, already +2 :)22:13
EmilienMgracias22:13
sshnaidmdenada22:13
*** morazi has quit IRC22:14
*** raildo has quit IRC22:14
*** ansmith has joined #tripleo22:14
*** rh-jelabarre has quit IRC22:30
*** aakarsh has joined #tripleo22:33
*** aakarsh has quit IRC22:46
*** rfolco has joined #tripleo22:46
openstackgerritCarlos Goncalves proposed openstack/diskimage-builder master: DNM: Fix yumdownloader cache dir  https://review.opendev.org/69878822:48
*** ade_lee_ has joined #tripleo22:48
*** rfolco has quit IRC22:50
*** ade_lee__ has quit IRC22:51
openstackgerritLuke Short proposed openstack/tripleo-heat-templates master: Use paunch to handle container removal.  https://review.opendev.org/69880622:53
*** ade_lee_ has quit IRC22:56
*** ade_lee_ has joined #tripleo22:56
*** rfolco has joined #tripleo23:01
*** ade_lee_ has quit IRC23:02
*** rfolco has quit IRC23:06
*** tkajinam has joined #tripleo23:06
openstackgerritMerged openstack/tripleo-heat-templates master: Enable additional Swift healtchecks  https://review.opendev.org/65120723:07
*** dmsimard has quit IRC23:07
*** dmsimard has joined #tripleo23:08
*** dmellado has quit IRC23:09
*** osmanlicilegi has quit IRC23:09
*** zbitter has joined #tripleo23:10
*** zaneb has quit IRC23:10
*** osmanlicilegi has joined #tripleo23:10
*** irclogbot_0 has quit IRC23:11
*** sshnaidm is now known as sshnaidm|off23:11
*** dmellado has joined #tripleo23:11
*** irclogbot_1 has joined #tripleo23:13
*** rfolco has joined #tripleo23:15
*** holser has joined #tripleo23:21
openstackgerritJames Slagle proposed openstack/tripleo-heat-templates master: Use include_tasks instead of import_tasks  https://review.opendev.org/69751023:21
openstackgerritJames Slagle proposed openstack/tripleo-heat-templates master: Execute deploy_steps_tasks per step  https://review.opendev.org/69861523:21
openstackgerritJames Slagle proposed openstack/tripleo-heat-templates master: Use async tasks for long running common tasks  https://review.opendev.org/69881523:21
slagle^^ magic pixie dust to go fast23:21
*** saneax has joined #tripleo23:22
openstackgerritMerged openstack/tripleo-heat-templates master: Change default value for NeutronPluginExtensions  https://review.opendev.org/69820923:23
mwhahahawat23:26
openstackgerritMerged openstack/tripleo-upgrade master: Add LEAPP_SKIP_CHECK_OS_RELEASE variable in UC upgrade.  https://review.opendev.org/69852323:27
*** bfournie has quit IRC23:29
*** slaweq has quit IRC23:30
*** slaweq has joined #tripleo23:37
*** jbadiapa has quit IRC23:37
openstackgerritMerged openstack/paunch master: builder: include environment when running an exec  https://review.opendev.org/69832923:41
openstackgerritMerged openstack/tripleo-upgrade master: Fix the undefined operation_type variable.  https://review.opendev.org/69822823:41
openstackgerritMerged openstack/tripleo-heat-templates master: Add CephGrafana resource to the ControllerStorageNFS role  https://review.opendev.org/69851423:41
*** rfolco has quit IRC23:47
openstackgerritMerged openstack/tripleo-heat-templates master: scale: fixes for compute scale down  https://review.opendev.org/69851523:54
openstackgerritMerged openstack/tripleo-heat-templates master: Remove ceph backend by default for gnocchi  https://review.opendev.org/69821323:54
openstackgerritMerged openstack/tripleo-quickstart-extras master: Render containerized-prepare-parameters.yaml during upgrade if not found.  https://review.opendev.org/69822523:54
openstackgerritMerged openstack/tripleo-common master: Revert "Update hostvars lookup to fix regression"  https://review.opendev.org/69872623:54
openstackgerritMerged openstack/tripleo-common master: Add deploy_steps_tasks to PER_TASK_STEPS  https://review.opendev.org/69861423:54
*** holser has quit IRC23:58

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!