Thursday, 2020-03-05

openstackgerritSteve Baker proposed openstack/tripleo-quickstart master: DNM switch off nova/glance in featureset001  https://review.opendev.org/71136400:04
*** sgw has quit IRC00:09
openstackgerritMerged openstack/tripleo-heat-templates master: nova-compute-container: add missing condition for ksmdisabled  https://review.opendev.org/71100700:11
openstackgerritMerged openstack/puppet-tripleo master: Revert "Add replication_probe_interval for ovsdbs"  https://review.opendev.org/71120100:11
openstackgerritMerged openstack/tripleo-heat-templates master: Use command to set pythonintepreter  https://review.opendev.org/71098000:11
*** slaweq has joined #tripleo00:11
openstackgerritEmilien Macchi proposed openstack/tripleo-common master: deployment: set DeployIdentifier to '' if --skip-deploy-identifier is used  https://review.opendev.org/71136600:13
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates stable/train: nova-compute-container: add missing condition for ksmdisabled  https://review.opendev.org/71136800:14
openstackgerritEmilien Macchi proposed openstack/python-tripleoclient master: overcloud: set DeployIdentifier to '' if --skip-deploy-identifier is used  https://review.opendev.org/71136900:15
*** Tengu has quit IRC00:20
*** Tengu has joined #tripleo00:22
*** mcornea has quit IRC00:26
*** slaweq has quit IRC00:27
*** sgw has joined #tripleo00:28
EmilienMsshnaidm|afk, weshay|ruck: have we seen errors on scenario001-centos8?00:33
EmilienMrlandy|bbl: ^00:33
EmilienMhttps://c6bc260fb8a50c93c7b6-115c1089095738e3e088969e8724f0ca.ssl.cf2.rackcdn.com/711230/1/check/tripleo-ci-centos-8-scenario001-standalone/2ad61f9/logs/undercloud/var/log/containers/stdouts/gnocchi_db_sync.log00:33
EmilienMcradox.ObjectNotFound: [errno 2] error calling conf_read_file00:33
weshay|ruckEmilienM, yes.. we need a promotion00:33
weshay|ruckhrm.. not sure if thats the same error we saw earlier00:33
EmilienMweshay|ruck: you found the same error? just to make sure we don't miss another issue00:33
* weshay|ruck checks00:33
weshay|ruckya.. sec00:33
weshay|ruckchecking00:34
weshay|ruckEmilienM, different00:34
weshay|ruckalso busted due to https://review.opendev.org/#/c/709701/00:34
EmilienMit's in the context of https://review.opendev.org/#/c/704443/00:34
EmilienMbut I don't think it's related to it ^00:34
EmilienMgnocchi is configured by puppet00:34
weshay|ruckya.. a puppet change could have busted it up00:35
weshay|ruckEmilienM, https://review.opendev.org/#/c/711320/00:35
EmilienMweshay|ruck: why don't we replace00:36
EmilienMinstead of adding00:36
weshay|ruckEmilienM, we're not adding00:36
weshay|rucklook at the branch excludes00:36
EmilienMah00:37
EmilienMright00:37
weshay|ruck:)00:37
EmilienMalso00:37
EmilienMgetting this on centos8:00:37
EmilienMError: error checking path "/run/libvirt": stat /run/libvirt: no such file or directory00:37
weshay|ruckwhere?00:37
EmilienMhttps://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_688/711230/1/check/tripleo-ci-centos-7-scenario012-standalone/6882c20/logs/undercloud/home/zuul/standalone_deploy.log00:38
weshay|ruckEmilienM, btw.. I need to explain this to you at some point http://dashboard-ci.tripleo.org/d/UDA4H3aZk/component-pipeline?orgId=100:38
weshay|ruckwill keep bugs off df's queue00:38
EmilienMI need to eat00:38
EmilienMor she'll kill me00:38
EmilienM:/00:38
weshay|ruckbu by00:39
EmilienMI'll come back eventually00:39
EmilienMbaie00:39
EmilienMI have 10 tabs of issues00:39
EmilienMyou can all find them in https://review.opendev.org/#/c/711230/00:39
EmilienM(check zuul.o.o)00:39
weshay|ruckscen012 is f..'d00:39
weshay|ruckhas been00:39
EmilienMk but still00:39
EmilienMthis is libvirt00:39
EmilienManyway00:39
EmilienMaurevoir00:39
weshay|ruckbu by00:39
weshay|ruckmf.. didn't even vote00:40
*** dsneddon has quit IRC00:40
*** dsneddon has joined #tripleo00:45
*** jmasud has joined #tripleo00:47
*** jmasud has quit IRC00:54
*** jmasud has joined #tripleo00:56
*** slaweq has joined #tripleo01:11
*** slaweq has quit IRC01:16
*** lbragstad_ has joined #tripleo01:34
*** lbragstad has quit IRC01:36
*** nhicher has quit IRC01:38
*** nhicher has joined #tripleo01:38
*** tkajinam has quit IRC01:49
*** tkajinam has joined #tripleo01:50
openstackgerritBrent Eagles proposed openstack/tripleo-heat-templates master: Do not install client packages in overcloud on octavia deploy  https://review.opendev.org/68895801:55
*** fultonj has joined #tripleo01:58
*** avivgt has quit IRC02:02
*** Goneri has quit IRC02:24
openstackgerritMerged openstack/tripleo-ci master: Handle cloudsig releases  https://review.opendev.org/71091002:31
openstackgerritMerged openstack/python-tripleoclient master: Remove the mistral from the check_deprecated_parameters workflow  https://review.opendev.org/70914102:31
*** sanjayu_ has joined #tripleo02:42
openstackgerritMerged openstack/python-tripleoclient master: Remove mistral when creating the overcloudrc file.  https://review.opendev.org/69413802:44
openstackgerritMerged openstack/paunch master: Update zuul layout to reflect recent changes in TripleO  https://review.opendev.org/71082402:44
openstackgerritMerged openstack/ansible-role-collect-logs master: Add pattern for image prepare error  https://review.opendev.org/71134102:44
openstackgerritMerged openstack/tripleo-ci master: Use centos8 built images  https://review.opendev.org/71128002:44
*** psachin has joined #tripleo02:51
*** sanjayu_ has quit IRC02:53
*** raukadah is now known as chandankumar03:02
*** lbragstad_ has quit IRC03:09
*** slaweq has joined #tripleo03:11
*** slaweq has quit IRC03:16
openstackgerritMerged openstack/tripleo-ansible master: Fix typo in logs  https://review.opendev.org/71133503:21
openstackgerritMerged openstack/tripleo-ci master: Add c8 multinode containers job  https://review.opendev.org/71017903:46
*** dsneddon has quit IRC03:51
*** apetrich has quit IRC04:00
*** udesale has joined #tripleo04:09
*** slaweq has joined #tripleo04:11
*** slaweq has quit IRC04:16
openstackgerritMerged openstack/tripleo-heat-templates master: Cleanup tasks for container-puppet.py  https://review.opendev.org/71050604:16
*** social has quit IRC04:20
*** rlandy|bbl is now known as rlandy04:29
*** skramaja has joined #tripleo04:29
openstackgerritRonelle Landy proposed openstack/tripleo-ci master: Add centos-8 scenario010 standalone jobs.  https://review.opendev.org/71128704:32
openstackgerritMerged openstack/tripleo-heat-templates master: Fixed package names for CentOS-8 octavia deployment  https://review.opendev.org/71091304:40
*** jdandrea has joined #tripleo04:41
openstackgerritwes hayutin proposed openstack/tripleo-ci master: DNM: Add centos-8 scenario000-multinode update job  https://review.opendev.org/71087604:45
*** rlandy has quit IRC04:53
*** ykarel|away is now known as ykarel04:55
*** ade_lee has quit IRC04:58
*** ade_lee has joined #tripleo04:59
*** slaweq has joined #tripleo05:11
*** udesale has quit IRC05:31
*** evrardjp has quit IRC05:35
*** evrardjp has joined #tripleo05:35
*** slaweq has quit IRC05:38
*** udesale has joined #tripleo05:39
*** jdandrea has quit IRC05:40
openstackgerritMerged openstack/tripleo-ci master: Improve build images role for CentOS8  https://review.opendev.org/70859905:54
*** mburned is now known as mburned_out06:03
*** mburned_out is now known as mburned06:03
*** dsneddon has joined #tripleo06:12
*** sri_ has joined #tripleo06:21
*** mburned is now known as mburned_out06:27
*** mburned_out is now known as mburned06:27
*** marios has joined #tripleo06:28
*** bandini has joined #tripleo06:35
*** weshay|ruck has quit IRC06:36
*** saneax has joined #tripleo06:36
*** weshay has joined #tripleo06:36
*** weshay has quit IRC06:40
*** weshay has joined #tripleo06:40
*** otherwiseguy has quit IRC06:41
*** rnoriega_ has quit IRC06:41
*** otherwiseguy has joined #tripleo06:42
*** waleedm has joined #tripleo06:45
*** rnoriega_ has joined #tripleo06:46
*** threestrands has quit IRC06:48
*** jfrancoa has joined #tripleo06:52
openstackgerritHarald JensÃ¥s proposed openstack/tripleo-heat-templates master: Add NeutronDhcpAgentDnsmasqEnableAddr6List param  https://review.opendev.org/71034206:55
openstackgerritFrancesco Pantano proposed openstack/tripleo-heat-templates master: Add the certificate specs in ceph_mgr service  https://review.opendev.org/70284606:55
*** dwilde has quit IRC06:56
*** dsneddon has quit IRC07:01
*** ccamacho has quit IRC07:02
*** abdysn has joined #tripleo07:10
*** weshay has quit IRC07:13
*** weshay_ has joined #tripleo07:16
*** jcapitao_off has joined #tripleo07:20
openstackgerrityatin proposed openstack/ansible-role-tripleo-modify-image master: Remove unavailable repos from update  https://review.opendev.org/71073807:20
openstackgerritChandan Kumar (raukadah) proposed openstack/tripleo-ci master: Adds tripleo-ci-centos-8-containers-undercloud-minion  https://review.opendev.org/71096107:20
*** jcapitao_off is now known as jcapitao07:22
openstackgerritRabi Mishra proposed openstack/python-tripleoclient master: Skip network check with stack env when validations disabled  https://review.opendev.org/71139207:22
*** pcaruana has joined #tripleo07:24
*** jbadiapa has joined #tripleo07:24
ykarelchandankumar, looks like https://review.opendev.org/#/c/709141/7/tripleoclient/workflows/parameters.py@130 has caused the multinode failures07:27
ykarelmerged few hours back, i noticed failure in c8 job upstream too07:27
*** hjensas has quit IRC07:28
ykarellooks like something else is wrong, as that is list, but error says dict07:33
ykarelme seeing https://2714a4271cb20642317a-491b37cd909c997f4e833a309b30267e.ssl.cf1.rackcdn.com/711287/4/check/tripleo-ci-centos-8-containers-multinode/6a2bec0/logs/undercloud/home/zuul/overcloud_deploy.log07:34
*** otherwiseguy has quit IRC07:34
openstackgerritFrancesco Pantano proposed openstack/tripleo-ansible master: Add a new set of tasks to build the openstack pool and key list  https://review.opendev.org/70908307:34
*** weshay_ has quit IRC07:34
*** weshay_ has joined #tripleo07:34
*** otherwiseguy has joined #tripleo07:35
*** ykarel is now known as ykarel|lunch07:35
*** dmacpher__ has quit IRC07:36
*** lhinds has quit IRC07:38
*** rnoriega_ has quit IRC07:39
*** mburned has quit IRC07:39
*** weshay_ has quit IRC07:39
*** haleyb|away has quit IRC07:39
*** otherwiseguy has quit IRC07:40
*** abdysn has quit IRC07:50
*** waleedm has quit IRC07:50
*** waleedm has joined #tripleo07:51
*** abdysn has joined #tripleo07:51
*** tesseract has joined #tripleo07:52
*** tremble has joined #tripleo07:53
*** rnoriega_ has joined #tripleo07:55
*** otherwiseguy has joined #tripleo07:56
*** weshay_ has joined #tripleo07:56
*** iurygregory has joined #tripleo07:56
*** lhinds has joined #tripleo07:58
*** cylopez has joined #tripleo08:02
openstackgerritMerged openstack/tripleo-common stable/train: Fix memcached healthcheck regression  https://review.opendev.org/71134008:03
openstackgerritMerged openstack/tripleo-quickstart-extras master: Remove unnecessary yum calls  https://review.opendev.org/71098208:03
*** hjensas has joined #tripleo08:08
*** ccamacho has joined #tripleo08:11
*** tkajinam has quit IRC08:16
*** weshay_ has quit IRC08:24
*** rnoriega_ has quit IRC08:25
*** weshay has joined #tripleo08:25
*** jpena|off is now known as jpena08:26
*** rnoriega_ has joined #tripleo08:28
*** saneax has quit IRC08:29
*** saneax has joined #tripleo08:30
*** ykarel|lunch is now known as ykarel08:32
*** rainleander has joined #tripleo08:32
jbadiapagood morning!08:33
*** leanderthal has quit IRC08:35
*** amoralej|off is now known as amoralej08:45
ykarelchandankumar, commented on the patch wrt ^^ issue08:45
chandankumarykarel, looking08:45
*** rpittau|afk is now known as rpittau08:46
*** bogdando has joined #tripleo08:46
*** marios has quit IRC08:56
*** pojadhav is now known as pojadhav|lunch09:01
*** jpich has joined #tripleo09:01
*** pojadhav|lunch is now known as pojadhav09:01
*** lucasagomes has joined #tripleo09:02
*** ade_lee has quit IRC09:07
*** sai_p___ has quit IRC09:07
*** mwhahaha has quit IRC09:07
*** mmagr has quit IRC09:08
*** abishop has quit IRC09:08
*** stevebaker has quit IRC09:08
*** mrunge has quit IRC09:08
*** StevenK has quit IRC09:08
openstackgerritMichele Baldessari proposed openstack/puppet-pacemaker master: WIP enforce firewall rules before pacemaker-auth  https://review.opendev.org/71140509:17
*** cylopez has quit IRC09:18
*** cylopez has joined #tripleo09:18
*** apetrich has joined #tripleo09:24
*** gfidente|afk is now known as gfidente09:25
*** tosky has joined #tripleo09:25
*** suuuper has joined #tripleo09:28
*** holser has joined #tripleo09:28
openstackgerrityatin proposed openstack/tripleo-quickstart master: Fix convert image package list  https://review.opendev.org/71140809:29
ykarelchandankumar, weshay issue caused by last patch ^^09:29
ykarelwith https://review.opendev.org/#/c/711078/109:30
chandankumarnit pick above ^^09:34
ykarellooking09:34
ykarelchandankumar, any specific advantage of using that09:35
ykarelis version thing09:35
chandankumargood to use version compare09:35
chandankumarhttps://docs.ansible.com/ansible/latest/user_guide/playbooks_tests.html#version-comparison09:36
*** sshnaidm|afk is now known as sshnaidm09:37
*** alexmcleod has joined #tripleo09:37
openstackgerritSagi Shnaidman proposed openstack/tripleo-quickstart-extras master: DNM: fail before introspection  https://review.opendev.org/69597509:39
openstackgerrityatin proposed openstack/tripleo-quickstart master: Fix convert image package list  https://review.opendev.org/71140809:41
*** ade_lee has joined #tripleo09:43
*** sai_p___ has joined #tripleo09:43
*** mwhahaha has joined #tripleo09:43
*** mmagr has joined #tripleo09:43
*** abishop has joined #tripleo09:43
*** stevebaker has joined #tripleo09:43
*** mrunge has joined #tripleo09:43
*** StevenK has joined #tripleo09:43
*** orwell.freenode.net sets mode: +ov mwhahaha mwhahaha09:43
sshnaidmcores, please review https://review.opendev.org/#/c/711253/ fix for flaky test09:43
openstackgerritSagi Shnaidman proposed openstack/tripleo-ansible master: WIP: try podman driver  https://review.opendev.org/70885109:44
*** openstackstatus has quit IRC09:45
openstackgerritSagi Shnaidman proposed openstack/tripleo-quickstart-extras master: DNM: clean nodes on OVB  https://review.opendev.org/70981609:46
openstackgerritSagi Shnaidman proposed openstack/tripleo-quickstart master: DNM: use vexxhost registry for promotion jobs  https://review.opendev.org/71013509:48
*** bogdando has quit IRC09:55
*** bogdando has joined #tripleo09:56
*** sri_ has quit IRC10:03
openstackgerritChristophe Fontaine proposed openstack/tripleo-ansible master: 'tripleo_ovs_dpdk_lcore_list' should be optional  https://review.opendev.org/71136010:11
openstackgerritChristophe Fontaine proposed openstack/tripleo-ansible master: CI: mount /lib/modules  https://review.opendev.org/71142010:11
openstackgerritSagi Shnaidman proposed openstack/tripleo-ansible master: WIP: try podman driver  https://review.opendev.org/70885110:13
*** derekh has joined #tripleo10:13
openstackgerritMerged openstack/tripleo-ci master: add the upstream centos-8 undercloud job  https://review.opendev.org/71085610:16
*** marios has joined #tripleo10:17
openstackgerritJuan Badia Payno proposed openstack/tripleo-ansible master: WIP: BaR backup /var/lib/ceph ceph mon and mgr  https://review.opendev.org/71142310:19
*** jcapitao has quit IRC10:20
*** jcapitao has joined #tripleo10:21
*** psachin has quit IRC10:23
*** dtantsur|afk is now known as dtantsur10:24
openstackgerritMerged openstack/tripleo-ci master: Add centos-8 scenario007 standalone jobs.  https://review.opendev.org/71133010:25
openstackgerrityatin proposed openstack/ansible-role-tripleo-modify-image master: Remove unavailable repos from update  https://review.opendev.org/71073810:28
*** hjensas has quit IRC10:31
*** jcapitao has quit IRC10:32
*** jcapitao has joined #tripleo10:33
openstackgerritBogdan Dobrelya (bogdando) proposed openstack/paunch stable/stein: Cleanup containers in the same loop as they are created  https://review.opendev.org/71142610:37
*** alexmcleod has quit IRC10:43
*** hjensas has joined #tripleo10:44
*** udesale has quit IRC10:49
*** alexmcleod has joined #tripleo10:50
openstackgerritChandan Kumar (raukadah) proposed openstack/tripleo-quickstart-extras master: Add job.repos to container updates for component jobs  https://review.opendev.org/71115610:51
*** alexmcleod has quit IRC10:55
*** alexmcleod has joined #tripleo10:56
*** cylopez has quit IRC10:59
openstackgerritBogdan Dobrelya (bogdando) proposed openstack/paunch stable/queens: Cleanup containers in the same loop as they are created  https://review.opendev.org/71143210:59
*** jmasud has quit IRC11:04
*** jmasud has joined #tripleo11:04
openstackgerritBogdan Dobrelya (bogdando) proposed openstack/paunch stable/queens: Cleanup containers in the same loop as they are created  https://review.opendev.org/71143211:05
openstackgerritChandan Kumar (raukadah) proposed openstack/tripleo-ci master: Add centos-8 scenario010 standalone jobs.  https://review.opendev.org/71128711:06
openstackgerritBogdan Dobrelya (bogdando) proposed openstack/paunch stable/stein: Cleanup containers in the same loop as they are created  https://review.opendev.org/71142611:07
openstackgerritRabi Mishra proposed openstack/tripleo-common master: Move RotateFernetKeysAction functionality to utils  https://review.opendev.org/71143411:11
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates stable/queens: Enable objects versioning for Ceph RadosGW  https://review.opendev.org/71020211:14
openstackgerritRabi Mishra proposed openstack/tripleo-ansible master: WIP Use only ansible for fernet key rotation  https://review.opendev.org/71143611:18
*** slaweq has joined #tripleo11:18
openstackgerritSorin Sbarnea proposed openstack/tripleo-ci master: lint: enable black  https://review.opendev.org/70398011:19
openstackgerritSagi Shnaidman proposed openstack/tripleo-ansible master: WIP: try podman driver  https://review.opendev.org/70885111:21
*** rpittau is now known as rpittau|bbl11:27
*** chem has quit IRC11:35
*** tesseract-RH has joined #tripleo11:40
openstackgerritFrancesco Pantano proposed openstack/tripleo-ansible master: Add a new set of tasks to build the openstack pool and key list  https://review.opendev.org/70908311:41
openstackgerritFrancesco Pantano proposed openstack/tripleo-heat-templates master: Add CephBasePoolVars and CephKeyVars structures  https://review.opendev.org/70929711:42
*** tesseract has quit IRC11:43
openstackgerritJose Luis Franco proposed openstack/tripleo-heat-templates stable/train: WIP: Move Stein upgrade tasks to ffwd tasks.  https://review.opendev.org/70610711:50
*** hjensas has quit IRC11:54
*** jcapitao is now known as jcapitao_lunch11:54
*** yprokule has joined #tripleo11:56
*** marios is now known as marios|call12:01
*** rlandy has joined #tripleo12:02
*** amoralej is now known as amoralej|lunch12:06
*** raildo has joined #tripleo12:08
*** jpena is now known as jpena|lunch12:09
*** dpawlik has quit IRC12:14
*** cylopez has joined #tripleo12:15
*** dpawlik has joined #tripleo12:15
openstackgerritLuca Miccini proposed openstack/puppet-tripleo stable/queens: set x-forwarded-port header for proxied requests  https://review.opendev.org/71144212:16
*** slaweq has quit IRC12:18
*** jcapitao_lunch has quit IRC12:19
openstackgerritJose Luis Franco proposed openstack/tripleo-upgrade master: Remove Congress service from roles_data.  https://review.opendev.org/71024212:20
*** jcapitao_lunch has joined #tripleo12:21
*** pcaruana has quit IRC12:22
*** hjensas has joined #tripleo12:22
*** dpawlik has quit IRC12:22
openstackgerritJose Luis Franco proposed openstack/tripleo-heat-templates master: Add always tags for hieradata render in external upgrade.  https://review.opendev.org/71032012:26
*** cylopez has quit IRC12:30
gfidentecloudnull I think I need help understanding what is going on with https://review.opendev.org/#/c/697990/12:30
openstackgerritChandan Kumar (raukadah) proposed openstack/tripleo-ci master: Add centos-8 scenario010 standalone jobs.  https://review.opendev.org/71128712:30
gfidentethe skeleton_test fails, but not the role test itself12:30
gfidenteso I can't reproduce it locally12:31
gfidenteand the failure seems just a file not found :(12:31
gfidentecan you help understand what is happening there and why it fails?12:31
cgoncalvesgfidente, hey! you might be able to help me with a TripleO Ceph related question :) I have a patch that adds a new upgrade job where Octavia and Ceph are enabled. problem is the job times out in what appears to be while upgrading Ceph. would you have any hints?12:33
cgoncalvesthe upgrade process appears to halt here: https://zuul.opendev.org/t/openstack/build/125e42ce57724b67b368a3f8f15f4c21/log/logs/undercloud/home/zuul/standalone_upgrade.log#867312:33
openstackgerritMichele Baldessari proposed openstack/puppet-pacemaker master: Enforce firewall rules before pacemaker-auth  https://review.opendev.org/71140512:34
*** yprokule has quit IRC12:34
gfidentecgoncalves hi12:34
*** pcaruana has joined #tripleo12:35
gfidentecgoncalves so you don't have the logs location, cause we have a bug in the task name12:35
gfidentebut I suspect this is just refusing to run because we have a single osd12:35
*** dpawlik has joined #tripleo12:36
gfidentelet me find the logs12:36
cgoncalvesthis? https://zuul.opendev.org/t/openstack/build/125e42ce57724b67b368a3f8f15f4c21/log/logs/undercloud/home/zuul/standalone-ansible-T9r6F_/ceph-ansible/ceph_ansible_command.log12:37
cgoncalvesFAILED - RETRYING: waiting for the monitor(s) to form the quorum... (10 retries left).12:37
gfidenteyeah that https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_125/710473/2/check/tripleo-ci-centos-7-scenario010-standalone-upgrade/125e42c/logs/undercloud/home/zuul/standalone-ansible-T9r6F_/ceph-ansible/ceph_ansible_command.log12:39
*** ansmith_ has quit IRC12:40
*** jpich has quit IRC12:40
cgoncalvesgfidente, thanks. so two questions: why does it fail on single osd only during upgrade? and what would be your suggestion to make the job work?12:41
*** jpich has joined #tripleo12:41
gfidentecgoncalves it should be running rolling_update though, not site-container12:41
gfidentecgoncalves the command.sh says it was running site-container12:41
gfidenteis this the initial deployment or the actual upgrade attempt?12:41
cgoncalvesgfidente, upgrade12:42
*** cylopez has joined #tripleo12:42
gfidenteso this is external-run --tags ceph ?12:42
cgoncalvesI'm not familiar with that site-container thingy12:42
gfidenteI mean at which step of the job is this failing>12:43
gfidenteso site-container playbook should work with a single mon/osd12:43
gfidenterolling_update won't because you can't keep the cluster up and this is a requirement from ceph folks12:43
gfidenteso site-container itself must be failing for a different reason, what is the change?12:44
cgoncalvesgfidente, I think it fails at external deployment step 212:45
cgoncalveshttps://zuul.opendev.org/t/openstack/build/125e42ce57724b67b368a3f8f15f4c21/log/logs/undercloud/home/zuul/standalone_upgrade.log#819912:45
cgoncalvesgfidente, this is the change: https://review.opendev.org/#/c/710473/12:45
cgoncalvesthe associated feature set is https://review.opendev.org/#/c/710556/1/config/general_config/featureset064.yml12:45
*** morazi has joined #tripleo12:46
cgoncalvesand the associated scenario is https://github.com/openstack/tripleo-heat-templates/blob/master/ci/environments/scenario010-standalone.yaml12:46
gfidentebut that works already with ceph12:50
gfidentein other jobs it seems12:50
gfidentescenario010 I mean12:50
cgoncalvescorrect. the problem manifests itself during standalone upgrade12:50
*** bfournie has quit IRC12:53
gfidentecgoncalves so looking at the ceph daemon logs12:54
gfidentehttps://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_125/710473/2/check/tripleo-ci-centos-7-scenario010-standalone-upgrade/125e42c/logs/undercloud/var/log/extra/podman/containers/12:54
gfidenteit did deploythe initial overcloud standalone12:54
openstackgerritMerged openstack/tripleo-quickstart-extras master: Remove yum calls from images prepare script  https://review.opendev.org/71127312:54
cgoncalvesgfidente, +1. it even passed tempest tests12:56
openstackgerritMerged openstack/tripleo-upgrade stable/train: Change roles_data/update_serial without using shell  https://review.opendev.org/71126412:57
openstackgerritMerged openstack/tripleo-upgrade stable/rocky: Allow to switch from parallel to serial update.  https://review.opendev.org/71072412:57
*** mburned has joined #tripleo12:58
openstackgerritSagi Shnaidman proposed openstack/tripleo-ci master: DNM use tripleo-ci-testing images for c8  https://review.opendev.org/71145112:59
openstackgerritBrent Eagles proposed openstack/tripleo-heat-templates master: Do not install client packages in overcloud on octavia deploy  https://review.opendev.org/68895813:00
*** ahosam has joined #tripleo13:01
*** rainleander is now known as leanderthal13:02
*** hjensas has quit IRC13:07
*** rh-jelabarre has joined #tripleo13:13
*** udesale has joined #tripleo13:14
*** jcapitao_lunch is now known as jcapitao13:15
*** rh-jelabarre has quit IRC13:21
*** rh-jelabarre has joined #tripleo13:21
*** hjensas has joined #tripleo13:27
openstackgerritEmilien Macchi proposed openstack/tripleo-common stable/train: deployment: set DeployIdentifier to '' if --skip-deploy-identifier is used  https://review.opendev.org/71145413:29
*** ansmith_ has joined #tripleo13:36
*** ansmith has joined #tripleo13:38
*** avivgt has joined #tripleo13:38
*** bfournie has joined #tripleo13:39
*** ansmith_ has quit IRC13:40
*** cgoncalves has quit IRC13:41
*** mcornea has joined #tripleo13:41
*** cgoncalves has joined #tripleo13:45
*** rpittau|bbl is now known as rpittau13:45
*** jdandrea has joined #tripleo13:46
*** fmount has quit IRC13:47
openstackgerritJuan Badia Payno proposed openstack/tripleo-ansible master: WIP:BaR Always do MySQL dumps even the services wont be stopped  https://review.opendev.org/71145813:47
*** fmount has joined #tripleo13:48
*** avivgt has quit IRC13:52
*** jcoufal has joined #tripleo13:54
*** jcoufal has quit IRC13:54
openstackgerritMerged openstack/tripleo-ci master: Adds tripleo-ci-centos-8-containers-undercloud-minion  https://review.opendev.org/71096113:55
*** jcoufal has joined #tripleo13:55
openstackgerritLukas Bezdicka proposed openstack/tripleo-heat-templates master: [Q->T] Introduce Queens to Train Upgrade  https://review.opendev.org/71145914:02
openstackgerritChristophe Fontaine proposed openstack/tripleo-ansible master: 'tripleo_ovs_dpdk_lcore_list' should be optional  https://review.opendev.org/71136014:02
*** jdandrea has quit IRC14:03
*** avivgt has joined #tripleo14:03
*** jdandrea_ has joined #tripleo14:04
cloudnullgfidente will take a look shortly14:09
gfidentecloudnull thanks14:09
cloudnullgfidente i actually have no idea what that test is doing, looks like its trying to exercise a role that doesn't exist14:12
*** lbragstad has joined #tripleo14:13
cloudnulland i dont see that in the tripleo-ansible zuul config ...14:13
cloudnullweshay ^ is that something that was added to tripleo-ci ?14:14
cloudnullhttps://zuul.opendev.org/t/openstack/build/99187eafaac24323b0b331dbb190008014:14
cloudnullhttps://zuul.opendev.org/t/openstack/job/tripleo-ansible-centos-7-molecule-skeleton_test14:15
cloudnulllooks like that job is now gone ?14:15
openstackgerritKevin Carter (cloudnull) proposed openstack/tripleo-ansible master: Add tripleo-lvmfilter role to restrict visible block devices for LVM  https://review.opendev.org/69799014:15
cloudnullgfidente I rebased it just to see if it was something from an old run, sadly it looks like that test is back14:16
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Remove all ignore_errors to avoid confusion when debugging  https://review.opendev.org/71105914:18
EmilienMbogdando: hey, is it better you think? https://review.opendev.org/#/c/711059/3..4/deployment/undercloud/undercloud-upgrade.yaml14:18
bogdandoEmilienM: yeah, checking for rc should work better14:18
cloudnullgfidente ah - https://review.opendev.org/#/c/697990/24/zuul.d/molecule.yaml@11814:19
bogdandoEmilienM: but for that particular case, it should be ==014:19
cloudnullgfidente is that something maybe you had in a local checkout ?14:19
bogdandoor != 0 for the failed check14:19
EmilienMbogdando: oh right14:19
bogdandoEmilienM: for the idempotency Gods you can also add changed_when: false sometimes :)14:19
cloudnullgfidente it looks like your lvmfilter change is adding the "skeleton_test" role to the zuul config14:20
bogdandomay be in followup14:20
EmilienMbogdando: yeah14:20
*** avivgt has quit IRC14:20
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Remove all ignore_errors to avoid confusion when debugging  https://review.opendev.org/71105914:22
EmilienMbogdando: I can't change the failed_when, or the task will fail if OVS upgrade isn't needed14:22
EmilienMbogdando: however I can add changed_when, in a followup14:22
bogdandosure, that works14:22
gfidentecloudnull aaaaaah that's because I run the newrole playbook to create it14:23
openstackgerritChristophe Fontaine proposed openstack/tripleo-ansible master: 'tripleo_ovs_dpdk_lcore_list' should be optional  https://review.opendev.org/71136014:24
cloudnulldid the new role playbook generate that role test ?14:24
*** marios has joined #tripleo14:25
cloudnullor did you run that playbook with the option `role_name=skeleton_test`?14:25
*** marios|call has quit IRC14:25
weshaycloudnull, need a few14:25
cloudnullweshay disregard I think we found the issue14:26
*** ykarel is now known as ykarel|away14:27
*** ekultails has joined #tripleo14:29
*** jpena|lunch is now known as jpena14:29
*** amoralej|lunch is now known as amoralej14:31
*** Goneri has joined #tripleo14:31
openstackgerritMerged openstack/tripleo-upgrade master: Initital ffwd2 patch to automate OS upgrade.  https://review.opendev.org/70490514:32
openstackgerritMerged openstack/paunch master: Cleanup containers in the same loop as they are created  https://review.opendev.org/70740814:32
*** TrevorV has joined #tripleo14:32
gfidentecloudnull I don't think I ran it with name=skeleton_test no14:32
gfidentecloudnull so shall I just go ahead and remove it from zuul in an update?14:33
*** chem has joined #tripleo14:33
*** sgw has quit IRC14:33
openstackgerritLukas Bezdicka proposed openstack/tripleo-common master: [Q->T] Introduce Queens to Train Upgrade  https://review.opendev.org/71146814:33
*** haleyb has joined #tripleo14:36
*** ahosam has quit IRC14:38
*** cgoncalves has quit IRC14:39
*** cgoncalves has joined #tripleo14:40
gfidentecloudnull I think I might have pulled it in rebasing one of the old patchsets then carried it forward into the new ones14:41
gfidentecloudnull so I'll clear it no14:41
gfidente*now14:41
*** jcapitao has quit IRC14:42
*** waleedm has quit IRC14:42
*** jcapitao has joined #tripleo14:44
openstackgerritGiulio Fidente proposed openstack/tripleo-ansible master: Add tripleo-lvmfilter role to restrict visible block devices for LVM  https://review.opendev.org/69799014:45
openstackgerritwes hayutin proposed openstack/tripleo-quickstart master: Adding molecule test for fetch-images role  https://review.opendev.org/68121114:48
*** sgw has joined #tripleo14:49
*** haleyb is now known as haleyb|away14:50
cloudnullgfidente that last review looks good !14:54
gfidentethe -modules job got moved around probably for the same reason14:54
gfidenteI can put it back to clear the gerrit change14:55
cloudnull++ the add role playbook will resort the jobs14:55
*** tremble has quit IRC14:55
cloudnullso you will see movement when its run, that's normal14:55
gfidenteI insisted on running the tests locally14:56
gfidentedidn't even figure it was a different module14:56
gfidenteskeleton_thingie14:56
*** abdysn has quit IRC14:57
gfidentethanks again14:57
cloudnullanytime, sorry for the confusion14:57
openstackgerritwes hayutin proposed openstack/tripleo-heat-templates master: update tht standalone scenario jobs for centos-8  https://review.opendev.org/71081114:59
openstackgerritBogdan Dobrelya (bogdando) proposed openstack/paunch stable/queens: Cleanup containers in the same loop as they are created  https://review.opendev.org/71143215:01
openstackgerritwes hayutin proposed openstack/tripleo-heat-templates master: update tht standalone scenario jobs for centos-8  https://review.opendev.org/71081115:02
openstackgerritwes hayutin proposed openstack/tripleo-common master: replace centos-7 with centos-8 in scenario jobs  https://review.opendev.org/71082115:02
*** d0ugal has joined #tripleo15:07
*** gfidente has quit IRC15:08
*** tremble has joined #tripleo15:11
*** jaosorior has quit IRC15:20
*** ramishra has quit IRC15:21
*** ramishra has joined #tripleo15:23
weshaycloudnull++ d0ugal++15:24
* d0ugal looks around curiously15:25
openstackgerritArx Cruz proposed openstack/python-tripleoclient master: Fix KeyError exception on check_deprecated_parameters  https://review.opendev.org/71148015:29
weshayanyone seen this in centos-8-containers-multinode15:39
weshay2020-03-05 12:21:07 |     heat_resource_tree_params = heat_resource_tree['parameters']15:39
weshay2020-03-05 12:21:07 | KeyError: 'parameters'15:39
weshay2020-03-05 12:21:07 | 'parameters'15:39
weshayhttps://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_879/710961/3/gate/tripleo-ci-centos-8-containers-multinode/879bf81/logs/undercloud/home/zuul/overcloud_deploy.log15:39
weshayhttps://bugs.launchpad.net/tripleo/+bug/186618415:40
openstackLaunchpad bug 1866184 in tripleo "centos-8 multinode overcloud deploy failed with heat_resource_tree_params = heat_resource_tree['parameters'] KeyError: 'parameters'" [Critical,In progress] - Assigned to Arx Cruz (arxcruz)15:40
weshaynrmind15:40
weshayarxcruz|rover++15:40
weshayEmilienM, cloudnull please review asap containers-multinode is down https://review.opendev.org/#/c/711480/15:40
cloudnullweshay - https://review.opendev.org/#/c/711291/15:41
weshaycloudnull, or should I double check w/ centos-815:42
weshayjust looking for confidence level15:42
openstackgerritwes hayutin proposed openstack/tripleo-ci master: make centos-8 containers-multinode voting/gating  https://review.opendev.org/71148815:46
arxcruz|roverweshay: not sure if it will work because of other issue with dict not being iterable15:46
ykarel|awaycloudnull, will ^^ also fixes unhashable dict error at 'user_defined': i in user_params.keys()15:47
arxcruz|rovernevermind, it will work because i in user_params.keys will return true :)15:47
sshnaidmweshay, another one: https://bugs.launchpad.net/tripleo/+bug/186620415:47
openstackLaunchpad bug 1866204 in tripleo "OVB is stuck on providing nodes" [Critical,Triaged]15:47
ykarel|awayhttps://review.opendev.org/#/c/709141/7/tripleoclient/workflows/parameters.py@12815:47
ykarel|awaycloudnull, ^^15:47
sshnaidmweshay, actually it can be a promotion blocker15:47
*** ramishra has quit IRC15:48
cloudnullykarel|away weshay EmilienM I'm happy to abandon my review and go with https://review.opendev.org/#/c/71148015:48
openstackgerritwes hayutin proposed openstack/python-tripleoclient master: update layout for centos-8 containers-multinode  https://review.opendev.org/71148915:49
ykarel|awaycloudnull, me not sure if this will fix unhashable dict issue15:49
sshnaidmcloudnull, apetrich fyi: https://bugs.launchpad.net/tripleo/+bug/1866204  I think cleaning node by default was introduced in one of latest mistral_to_ansible patches. We don't need to clean nodes in CI, need to make sure it's not by default when running nodes "provide"15:49
openstackLaunchpad bug 1866204 in tripleo "OVB is stuck on providing nodes" [Critical,Triaged]15:49
ykarel|awayif u are sure any will work15:49
weshaycloudnull, ykarel|away let me know.. once it's green.. it needs be returned to voting asap15:50
weshaysshnaidm, is centos-8 stuck on the same thing?15:51
sshnaidmweshay, yeah15:51
cloudnulltripleo-ci-centos-7-containers-multinode passed https://review.opendev.org/#/c/711291 are we seeing an issue with "unhashable dict error at 'user_defined': i in user_params.keys()" ?15:51
weshaysshnaidm, https://review.rdoproject.org/r/#/c/25332/15:51
weshayk15:51
cloudnullsshnaidm i dont think that the cleaning patches merged just yet ?15:51
ykarel|awaycloudnull, issue is in centos815:51
ykarel|awaywith python315:51
ykarel|awaypython2 jobs are fine15:51
cloudnulloic15:51
cloudnullthen lets go with arxcruz|rover's review w15:51
sshnaidmcloudnull, maybe register patch..?15:51
cloudnulli'll abandon mine15:51
ykarel|awaycloudnull, so that review will fix unhashable dict error?15:52
weshayekultails, cix https://bugs.launchpad.net/tripleo/+bug/186620415:52
openstackLaunchpad bug 1866204 in tripleo "OVB is stuck on providing nodes" [Critical,Triaged]15:52
sshnaidmweshay, it's here: https://review.rdoproject.org/r/#/c/25666/15:52
sshnaidmweshay, the job you pointed to is waiting for image building fixes, chandankumar is on it15:53
arxcruz|rovercloudnull: sorry, didn't knew you were working on it, just saw the bug filled :)15:53
cloudnullits all good15:53
cloudnullykarel|away it looks like it15:53
cloudnullhttps://pasted.tech/pastes/fab38cc0c5109a888bf1b385e1e086c867cfa7f215:53
weshayso which patch we going w/?15:53
cloudnullarxcruz|rover 's review will ensure both default to a dict15:54
ykarel|awaycloudnull, instead of 'i' try a dict15:54
ykarel|awayi think that's the issue15:54
ykarel|awayin python2 that will pass15:54
ykarel|awaywith py3 fails15:54
cloudnulli just tested it with py3715:54
ykarel|awaycloudnull, but u tested string15:54
arxcruz|roverykarel|away: but i should not be a dict...15:54
ykarel|awaytest dict15:55
ykarel|awayarxcruz|rover, somehow it's coming as dict15:55
ykarel|awaythat's why the issue imo15:55
ykarel|awaypython3 -c "x={1:1};{} in x.keys()"15:56
*** jcapitao is now known as jcapitao_afk15:56
ykarel|awayvs python2 -c "x={1:1};{} in x.keys()"15:56
arxcruz|roverykarel|away: yeah you right, let me test something15:57
cloudnullinteresting. TIL15:57
weshayarxcruz|rover++ ykarel|away++15:57
cloudnullprobably need to use a sentinel for py3, might need to handle that loop two ways depending on py version15:58
openstackgerritwes hayutin proposed openstack/tripleo-ci master: make centos-8 containers-multinode voting/gating  https://review.opendev.org/71148815:58
openstackgerritwes hayutin proposed openstack/python-tripleoclient master: update layout for centos-8 containers-multinode  https://review.opendev.org/71148915:59
weshaymwhahaha, https://review.opendev.org/#/c/711320/16:00
cloudnullsshnaidm the register patch hasn't merged yet either - https://review.opendev.org/#/c/709583/616:00
*** dsneddon has joined #tripleo16:01
*** Trevor_V has joined #tripleo16:01
openstackgerritwes hayutin proposed openstack/tripleo-ansible master: update containers-multinode to centos-8  https://review.opendev.org/71149416:04
*** TrevorV has quit IRC16:05
openstackgerritwes hayutin proposed openstack/tripleo-common master: update containers-multinode to centos-8  https://review.opendev.org/71149616:06
sshnaidmcloudnull, I see, will look more..16:10
*** jcapitao_afk is now known as jcapitao16:10
sshnaidmrlandy, do you remember if we run nodes clean by default in ovb?16:10
*** jcapitao has quit IRC16:12
rlandysshnaidm: I disabled that on baremetal because it was taking forever - by default on ovb, I think yes - will have to check that16:12
rlandygetting16:12
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates master: Remove topic suffix from default value  https://review.opendev.org/71012316:12
*** morazi has quit IRC16:13
rlandysshnaidm: so checking the settings for OVB - yes - we clean nodes in releases after pike (other than the vexxhost settings you have)16:16
sshnaidmrlandy, where is it defined?16:16
rlandysshnaidm: search for undercloud_clean_nodes16:16
rlandyit's in undercloud.conf16:17
rlandywe set that in some of the featuresets16:17
rlandyand then you have a vexxhost setting in tripleo-ci16:17
rlandyhttp://git.app.eng.bos.redhat.com/git/tripleo-environments.git/tree/hardware_environments/dell_fc430_envA/network_configs/single_nic_vlans/env_settings.yml#n30 - that is where we disable for bm - downstream only16:18
*** jcapitao has joined #tripleo16:18
openstackgerritKevin Carter (cloudnull) proposed openstack/python-tripleoclient master: Remove the mistral from the deploy_plan workflow  https://review.opendev.org/71047916:19
sshnaidmrlandy, hmm.. I see it's disabled in vexx16:20
sshnaidmrlandy, didn't think it's done in undercloud part16:20
rlandysshnaidm: yeah - not sure why that would be16:20
rlandyit's set that16:20
rlandythere16:20
rlandythe only reason to disable it would be timing or issues with the overcloud nodes disks16:21
openstackgerritwes hayutin proposed openstack/tripleo-heat-templates master: update containers-multinode to centos-8  https://review.opendev.org/71149916:23
openstackgerritSagi Shnaidman proposed openstack/tripleo-ci master: Sync vexxhost settings for OVB  https://review.opendev.org/71150016:24
sshnaidmrlandy, let's see ^^16:24
rlandysshnaidm: for some reason it's also set in multinode on vexxhost? https://github.com/openstack/tripleo-ci/blob/master/toci-quickstart/config/testenv/multinode-vexxhost.yml#L5716:25
openstackgerritArx Cruz proposed openstack/python-tripleoclient master: Fix KeyError exception on check_deprecated_parameters  https://review.opendev.org/71148016:26
arxcruz|roverykarel|away: now it will work :) ^^16:26
openstackgerritSagi Shnaidman proposed openstack/tripleo-ci master: Sync vexxhost and rdo cloud settings  https://review.opendev.org/71150016:28
sshnaidmrlandy, ack, changed it too16:28
rlandyk - let's see the damage16:28
*** jcoufal has quit IRC16:29
arxcruz|roverweshay: ^16:30
arxcruz|roverweshay: heading home, ping me if you need something :)16:30
openstackgerritwes hayutin proposed openstack/python-tripleoclient master: update containers-multinode to centos-8  https://review.opendev.org/71150316:32
weshayarxcruz|rover, thanks16:32
weshayarxcruz|rover, confident in that last revision?16:32
*** ramishra has joined #tripleo16:33
openstackgerritChandan Kumar (raukadah) proposed openstack/tripleo-ci master: Add centos8 support for emit_releases_script  https://review.opendev.org/71044116:33
*** aedc_ has joined #tripleo16:34
openstackgerritChandan Kumar (raukadah) proposed openstack/tripleo-quickstart-extras master: Revert "Set container build hash based on dlrn md5 checksum"  https://review.opendev.org/71130516:34
chandankumarrlandy, ^^16:34
rlandychandankumar: yeah - better - the other way around would be a disaster :(16:35
chandankumarrlandy, rechecked here https://review.rdoproject.org/r/#/c/25694/16:35
rlandychandankumar: ack - thanks - will keep an eye on that16:36
*** aedc has quit IRC16:36
openstackgerritMerged openstack/python-tripleoclient master: Avoid TZ issues in test_image test  https://review.opendev.org/71125316:38
openstackgerritMarios Andreou proposed openstack/tripleo-ci master: WIP Add centos-8-scenario010-ovn-provider-standalone job definition  https://review.opendev.org/71150716:40
rlandymarios: did you see this at all in the last few days ...  Data could not be sent to remote host "127.0.0.2". Make sure this host can be reached over ssh: kex_exchange_identification: Connection closed by remote host16:43
openstackgerritMichele Baldessari proposed openstack/puppet-tripleo master: Enforce firewall rules before pacemaker-auth  https://review.opendev.org/71150916:43
rlandyhitting the component pipeline in a few places16:43
rlandyafter the tempest run16:44
rlandyarxcruz|rover: ^^?16:44
rlandyrechecking those jobs16:44
*** udesale has quit IRC16:44
*** cfontain has joined #tripleo16:46
openstackgerritwes hayutin proposed openstack/tripleo-ci master: update periodics for centos-8  https://review.opendev.org/71151016:46
*** morazi has joined #tripleo16:46
*** Trevor_V has quit IRC16:47
*** ccamacho has quit IRC16:48
openstackgerritJose Luis Franco proposed openstack/tripleo-upgrade master: Adapt FFWD2 Undercloud upgrade workflow.  https://review.opendev.org/70623816:48
*** aedc_ has quit IRC16:49
*** TrevorV has joined #tripleo16:50
openstackgerritwes hayutin proposed openstack/python-tripleoclient master: update containers-multinode, minion to centos-8  https://review.opendev.org/71150316:51
*** aedc_ has joined #tripleo16:51
*** cfontain has quit IRC16:52
openstackgerritJose Luis Franco proposed openstack/tripleo-upgrade master: Adapt FFWD2 Undercloud upgrade workflow.  https://review.opendev.org/70623816:53
*** rcernin has quit IRC16:55
openstackgerritwes hayutin proposed openstack/tripleo-ansible master: update more jobs to centos-8  https://review.opendev.org/71149416:56
*** aedc_ has quit IRC16:56
*** marios is now known as marios|out16:59
*** cylopez has quit IRC17:00
openstackgerritMerged openstack/tripleo-ansible stable/train: Tune-up package install  https://review.opendev.org/71134917:01
*** tesseract-RH has quit IRC17:01
*** lucasagomes has quit IRC17:02
*** bogdando has quit IRC17:06
openstackgerritwes hayutin proposed openstack/tripleo-heat-templates master: update containers-multinode, minion to centos-8  https://review.opendev.org/71149917:07
openstackgerritwes hayutin proposed openstack/tripleo-ci master: make centos-8 undercloud-containers voting  https://review.opendev.org/71151617:10
*** suuuper has quit IRC17:15
*** jpich has quit IRC17:22
*** rpittau is now known as rpittau|afk17:25
openstackgerritMerged openstack/tripleo-ansible stable/train: Creates container_config_data module  https://review.opendev.org/71134717:29
openstackgerritMerged openstack/tripleo-ansible master: Only add host entries for role_networks  https://review.opendev.org/71057417:32
openstackgerritwes hayutin proposed openstack/tripleo-common master: update jobs to centos-8  https://review.opendev.org/71149617:32
*** evrardjp has quit IRC17:35
*** marios|out has quit IRC17:35
*** evrardjp has joined #tripleo17:35
*** holser has quit IRC17:36
weshayekultails, sorry for the confusion.. you have the right log and review now https://logserver.rdoproject.org/66/25666/7/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master/4fc1241/job-output.txt17:37
weshaythanks for the help17:37
*** dtantsur is now known as dtantsur|afk17:40
ekultailsweshay: No worries. Thanks for the update. :-)17:46
openstackgerritSagi Shnaidman proposed openstack/tripleo-ci master: lint: enable black  https://review.opendev.org/70398017:49
openstackgerritMerged openstack/tripleo-ansible master: update tripleo-ansible ci for centos-8  https://review.opendev.org/71081017:59
*** derekh has quit IRC18:00
openstackgerritMichele Baldessari proposed openstack/puppet-tripleo master: Make sure we create stonith resources before stonith levels  https://review.opendev.org/71152718:06
*** bandini has quit IRC18:06
*** jcapitao is now known as jcapitao_off18:10
*** slaweq has joined #tripleo18:11
openstackgerritArx Cruz proposed openstack/python-tripleoclient master: Fix KeyError exception on check_deprecated_parameters  https://review.opendev.org/71148018:13
*** jcapitao_off has quit IRC18:15
*** chandankumar is now known as raukadah18:18
*** ccamacho has joined #tripleo18:20
*** Goneri has quit IRC18:23
*** jbadiapa has quit IRC18:23
*** jpena is now known as jpena|off18:26
*** Goneri has joined #tripleo18:27
openstackgerritMerged openstack/tripleo-heat-templates stable/train: nova-compute-container: add missing condition for ksmdisabled  https://review.opendev.org/71136818:31
openstackgerritMerged openstack/tripleo-common master: deployment: set DeployIdentifier to '' if --skip-deploy-identifier is used  https://review.opendev.org/71136618:31
openstackgerritMerged openstack/python-tripleoclient master: overcloud: set DeployIdentifier to '' if --skip-deploy-identifier is used  https://review.opendev.org/71136918:31
openstackgerritMerged openstack/python-tripleoclient master: Skip network check with stack env when validations disabled  https://review.opendev.org/71139218:31
sshnaidmekultails, centos8 bug is here https://bugs.launchpad.net/tripleo/+bug/186620218:32
openstackLaunchpad bug 1866202 in tripleo "OVB on centos8 fails because of networking failures" [Critical,Triaged]18:32
sshnaidmekultails, the patch that runs centos8 is  https://review.rdoproject.org/r/#/c/25666/18:33
ekultailssshnaidm: Awesome, thanks for sharing. Yeah, I was just looking at the NIC configs on one of these broken CI jobs.18:35
*** amoralej is now known as amoralej|off18:35
sshnaidmekultails, br-ex is a red herring, the problem starts before that..18:35
sshnaidmekultails, maybe changes with network manager, a lot of logs from it18:35
sshnaidmekultails, centos8 btw has renamed interfaces to "ensX", not sure how it's in images, need to check18:36
ekultailssshanidm: https://logserver.rdoproject.org/66/25666/7/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master/4fc1241/logs/overcloud-controller-0/etc/sysconfig/network-scripts/18:37
ekultailsI was wondering if that was the problem. There's both ifcfg-eth* and a ifcfg-ens3.18:37
ekultailssshnaidm: This is my latest update/comment from the CI bug I've been looking into: https://bugs.launchpad.net/tripleo/+bug/1866204/comments/718:38
openstackLaunchpad bug 1866204 in tripleo "OVB is stuck on providing nodes" [Critical,Triaged]18:38
ekultailsI think we're on the right track. It might just bee something in TripleO Quickstart that needs to be updated.18:39
weshaysshnaidm, ekultails we can't really fix the ifnames issue from ci.. would have to be in the dib18:41
sshnaidmekultails, this is a different bug18:41
sshnaidmekultails, if you're talking about "yum", it's wip, it's a wrong patch18:41
sshnaidmekultails, just updated bugs with comments, please take a look18:41
sshnaidmweshay, I just checked, and seems like interfaces are ok in ovb18:42
weshaysshnaidm, ik18:42
weshayok18:42
sshnaidmweshay, they have ethX18:42
weshaysshnaidm, can you set the default to download and not build images?18:42
openstackgerritAlex Schultz proposed openstack/tripleo-common stable/stein: Use config for layers  https://review.opendev.org/71070518:42
weshaymaybe in the release file18:42
weshayfor tripleo-ci/CentOS-8/master.yml18:43
sshnaidmweshay, it is, but periodic job usually build and test them18:43
weshaysshnaidm, ya.. not needed though18:43
ekultailssshnaidm: The yum comment can be disregarded. I guess the OVB cleaning is probably different than the NIC configuration issue, right?18:43
weshaybecause of the job that actually builds them18:43
weshaybetter to skip the build in ovb jobs for now18:43
sshnaidmweshay, anyway, raukadah should make this script working for c8, it's wip in https://review.opendev.org/#/c/711081/18:44
weshayuntil we kill the tqe role18:44
sshnaidmweshay, if we have building jobs in periodic we can configure them18:44
weshaysshnaidm, we have centos-8 overcloud-full and ipa landed18:44
sshnaidmweshay, because it was failing before when didn't find tripleo-ci-testing images..18:44
weshayk.. we will now18:44
weshaysshnaidm, next run we should see it18:45
sshnaidmweshay, ok, then it's fine to run it with "to_build: false"18:45
weshay++18:45
weshayyes18:45
*** slaweq has quit IRC18:47
*** rmccabe has joined #tripleo18:53
rmccabeanyone know if there's a known good procedure for deploying from upstream master? trying to do some testing and i've not been able to get a good deploy18:54
*** jaosorior has joined #tripleo19:08
sshnaidmekultails, can you please move your comments to c8 bug? It will help to concentrate the all info in a right place19:17
*** mcornea has quit IRC19:18
openstackgerritBrent Eagles proposed openstack/tripleo-ansible master: octavia move openstack api calls to undercloud  https://review.opendev.org/71109419:23
ekultailssshnaidm: Will do.19:27
openstackgerritwes hayutin proposed openstack/tripleo-ansible master: WIP: here we go.. centos-7 -> 8 for molecule  https://review.opendev.org/71153519:28
weshaycloudnull, should be interesting ^19:28
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: Add job.repos to container updates for component jobs  https://review.opendev.org/71115619:29
openstackgerritHarald JensÃ¥s proposed openstack/tripleo-heat-templates master: Drop resource registry override in ip-from-pool  https://review.opendev.org/71153719:33
openstackgerritHarald JensÃ¥s proposed openstack/tripleo-docs master: Predictable IPs, no resource_registry override  https://review.opendev.org/71153919:37
cloudnullweshay nice! The roles should work on RHEL / cent 8. Most are already testing in cent8 containers.19:40
weshayok.. cloudnull we were wondering if the upgrade roles need to run on centos-7 still19:40
weshaynot quite sure what to make of that was going to ping those guys19:40
weshaymatbu, ^ may be able to provide some insight19:40
openstackgerritHarald JensÃ¥s proposed openstack/tripleo-heat-templates master: Drop resource registry override in ip-from-pool  https://review.opendev.org/71153719:41
openstackgerritRyan McCabe proposed openstack/tripleo-heat-templates master: Add STF environment  https://review.opendev.org/69462319:42
sshnaidmrlandy, seems like cleaning fails on vexx: https://logserver.rdoproject.org/00/711500/2/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master-vexxhost/1324126/logs/undercloud/var/log/extra/baremetal_list.txt.gz19:42
*** saneax has quit IRC19:42
sshnaidmrlandy, need to fix that..19:42
sshnaidmweshay, ^^19:42
*** pcaruana has quit IRC19:42
rlandysshnaidm: is there a difference in the nodes between vexx and rdocloud?19:43
rlandylooking at error19:43
sshnaidmrlandy, should not be, except of openstack version if the cloud19:43
openstackgerritwes hayutin proposed openstack/python-tripleoclient master: Fix KeyError exception on check_deprecated_parameters  https://review.opendev.org/71148019:43
*** cfontain has joined #tripleo19:43
sshnaidmrlandy, same settings, and I think we disabled cleaning before because it was failing..19:44
rlandysshnaidm: you probably had a reason to make cleaning false19:44
weshaysshnaidm, can't we turn cleaning off19:44
sshnaidmrlandy, yeah, and forgot about it later19:44
sshnaidmweshay, it decreases our coverage19:44
sshnaidmweshay, somehow it works on rdo cloud19:45
rlandyhad some problems in baremetal - but the overcloud nodes there are not clean when we start19:45
sshnaidmrlandy, vms should be clean, just created19:45
rlandyright - here you should not have that issue19:46
mmagrHey guys, is there anybody able to find time for review of https://review.opendev.org/#/c/704080/ or https://review.opendev.org/#/c/709106/ please?19:46
rlandyintrospection does ok19:46
sshnaidmrlandy, introspection didn't start, there are no manageable nodes19:46
sshnaidmrlandy, actually I think maybe we need to fail it in this case..19:47
cloudnullweshay no idea19:47
rlandy2020-03-05 17:44:23 |   msg: No nodes are manageable at this time. 2020-03-05 17:47:58 | 4 node(s) successfully moved to the "available" state.19:47
rlandythat should fail19:48
rlandysshnaidm: ^^ yeah definitely19:48
rlandythen we would catch it there19:48
mwhahahacloudnull:  openstack overcloud node introspect --all-manageable --provide is broken19:49
*** cfontain has quit IRC19:49
mwhahahacloudnull: http://paste.openstack.org/show/790358/19:49
cloudnullI dont think the provide patch merged yet ?19:50
mwhahahawe shouldn't have removed the ansible bits then19:51
cloudnullhttps://review.opendev.org/#/c/710308/19:51
mwhahahausing a uuid doesn't work either19:51
mwhahahahttp://pastebin.test.redhat.com/84242219:51
cloudnulllooks like its concatenating the uuids - ebc0d52f-1c93-4c72-8b93-5b9e052ddc60 0cacc47d-7bf4-447c-a6d1-ef637ed05dbd 0b3f9b16-3027-49f5-994a-bdf830f8837f19:52
rlandysshnaidm: have you checked with the vexx guys about the node configs. We usually see this failure when there is a disk issue in cleaning19:53
mwhahahacloudnull: no it errors with one node too19:53
cloudnullmwhahaha https://review.opendev.org/#/c/710308/5/tripleoclient/workflows/baremetal.py - we're not using the provide playbook yet, not until that patch is merged19:54
mwhahahathis seems to indicate otherwise?19:54
* mwhahaha shrugs19:54
mwhahahathis is the same workflow i've been using for years19:54
mwhahahaso kinda bad it no longer works19:54
* mwhahaha files a bug19:54
sshnaidmmwhahaha, we have a problem with this recently19:55
sshnaidmmwhahaha, I think it's related to bug that was opened today19:55
sshnaidmmwhahaha, but in here  http://paste.openstack.org/show/790358/ nodes are in "available" state19:55
sshnaidmnot in "managealbe"19:56
mwhahaharight that's what --provide is supposed to do19:56
mwhahahaif i remember correctly19:56
mwhahahaso that's the missing piece19:57
mwhahaha--provide has no effect19:57
mwhahahawait ---provide moves them to available19:57
sshnaidmmwhahaha, yep19:58
sshnaidmmwhahaha, provide does move to available19:58
sshnaidmmwhahaha, registration should move to manageable, introspection just introspects, cleaning leaves nodes in manageable, and then "provide" make them available19:59
sshnaidmiirc19:59
cloudnullso we're good ?20:01
mwhahahai don't think that's what happens20:01
sshnaidmmwhahaha, but maybe you hit https://bugs.launchpad.net/tripleo/+bug/186620420:01
openstackLaunchpad bug 1866204 in tripleo "OVB is stuck on providing nodes" [Critical,Triaged]20:01
mwhahahano the registration was successful20:02
sshnaidmit's not clear yet what happens, but definitely some problem is there20:02
openstackgerritEmilien Macchi proposed openstack/paunch stable/train: Cleanup containers in the same loop as they are created  https://review.opendev.org/70976020:02
sshnaidmmwhahaha, what are nodes state right after registration?20:02
mwhahahalet me nuke them and see20:03
mwhahahai was also going to try osp16 as well20:03
sshnaidmsomething should call "manage" for nodes to make them manageable, and maybe it was part of registration in mistral.. there are so much duplicates in these calls20:04
mwhahahaok so import makes them manageable20:04
sshnaidmaha20:04
sshnaidmmwhahaha, and then you run introspection?20:05
mwhahahayea20:05
mwhahahai usually run introspect --all-manageable --provide20:05
sshnaidmmwhahaha, and it doesn't find manageabler nodes?20:05
mwhahahaok so it's --provide20:06
mwhahahait's running before introspection20:06
mwhahahaif i do introspect <uuids> it's running20:06
sshnaidmcloudnull, maybe we switched the order? ^20:07
cloudnullof all managable and provide ?20:08
mwhahaha--provide20:08
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates stable/train: Cleanup tasks for container-puppet.py  https://review.opendev.org/71154220:08
mwhahaha--all-manageable just doesn't work20:08
mwhahahaunless --provide is running prior to --all-manageable lookups20:09
openstackgerritKevin Carter (cloudnull) proposed openstack/tripleo-ansible master: Revert "Modify the introspection playbook to simplify client interactions"  https://review.opendev.org/71154320:09
mwhahahai'll check that next20:09
cloudnull^ that should fix the all-managable issue20:09
mwhahahaso i had successful introspection w/o --provide20:09
cloudnullhttps://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/v2/overcloud_node.py#L89-L98 - it looks like provide is still using the old mistral functions still,20:11
sshnaidmcloudnull, oh, I see it remove all-manageable option20:11
mwhahahai think it still ran when the introspect failed20:11
mwhahahaso that might have been the problem20:12
cloudnullthe provide playbook isn't yet afaict20:12
mwhahaha--all-manageable was a noop, and we then ran provide20:12
cloudnull****isn't used yet afaict20:12
sshnaidmcloudnull, so it might solve also https://bugs.launchpad.net/tripleo/+bug/186620420:14
openstackLaunchpad bug 1866204 in tripleo "OVB is stuck on providing nodes" [Critical,Triaged]20:14
sshnaidmcloudnull, I'd be glad to be added to reviews for any introspection part20:14
sshnaidmcloudnull, anyway I deal with it in OVB jobs20:14
cloudnullif we can get https://review.opendev.org/#/c/707348 in, then that revert wouldn't be needed.20:15
openstackgerritAlex Schultz proposed openstack/tripleo-ansible master: Revert "Modify the introspection playbook to simplify client interactions"  https://review.opendev.org/71154320:15
mwhahahaadded closes bug20:15
mwhahahai think that solves that bug20:15
cloudnull++20:15
mwhahahabecause what happened is that --all-manageable basically did nothing but the provide did run20:15
mwhahahawhich makes subsequent runs bad20:15
mwhahahabecause it's already been moved to available even without introspection20:16
cloudnullmakes sense.20:16
mwhahahaseems like we shouldn't be able to provide a node if it hasn't been introspected20:16
mwhahahathat seems like a bug in ironic?20:16
mwhahahaunless it's valid to be able to skip introspection?20:16
openstackgerritRonelle Landy proposed openstack/tripleo-ci master: DNM: Add centos-8 scenario000-multinode update job  https://review.opendev.org/71087620:18
*** cylopez has joined #tripleo20:23
*** sgw has quit IRC20:23
weshayhow the heck did it pass two ovb jobs to begin w/20:24
sshnaidmmwhahaha, we can skip introspection totally20:25
sshnaidmmwhahaha, and cleaning20:25
lbragstadhey folks, i have an environment using RDO master and i verified https://review.opendev.org/#/c/702031/13 and https://review.opendev.org/#/c/705041/3 exist on the undercloud20:25
sshnaidmmwhahaha, cloudnull but I think maybe worth to fail a playbook if no node was introspected?20:25
mwhahahafor that command, yea20:26
cloudnullprobably a good idea20:26
mwhahahawe have a seperate provide command20:26
lbragstadmy overcloud nodes have a processor count of 4 (using lscpu), but only two keystone processes are running in the keystone container20:26
sshnaidmmwhahaha, and maybe worth to separate these commands, it's too much dups20:26
mwhahahalbragstad: bet you don't have those in overcloud-full20:26
lbragstadam i missing something or should there be at least 4?20:26
mwhahahasshnaidm: i perfer the single shot command but that's me20:27
cloudnullsshnaidm once we revert the revert, the playbook will require that the node uuids be provided.20:27
cloudnullso that should guard against no nodes being introspected ?20:27
sshnaidmcloudnull, we still need --all-manageable20:27
cloudnullfor sure, that will be handled in the client20:27
sshnaidmcloudnull, and if it doesn't find manageable nodes, need to fail20:27
lbragstadmwhahaha overcloud-full? for additional context, i'm using a deployed-server environment20:28
mwhahahalbragstad: ok, same diff. do the overcloud nodes have the patches20:28
cloudnullsshnaidm https://review.opendev.org/#/c/707348/9/tripleoclient/workflows/baremetal.py@16920:28
sshnaidmcloudnull, oh, I recall that you want to sort manageable nodes in client, right?20:28
lbragstadmwhahaha checking20:28
cloudnullthat playbook should fail if the list is empty20:28
mwhahahalbragstad: also do you have the THT that switches it to be os_workers_keystone, i think it's os_workers by default in THT20:29
cloudnullthough we could add a specific fail task to ensure20:29
sshnaidmcloudnull, yeah, that seems totally fine20:29
sshnaidmcloudnull, and this as well20:29
cloudnullbut for now the revert seems like the best course of action20:29
cloudnullwe have a bunch of changes to get through before https://review.opendev.org/#/c/707348 is in20:30
mwhahahalbragstad: https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/deployment/keystone/keystone-container-puppet.yaml#L112 needs to be updated20:30
sshnaidmcloudnull, maybe it will solve problems with ovb failing there..20:30
cloudnullpotentially20:30
sshnaidmcloudnull, let's rerun check-rdo on all of them after this is merged20:30
lbragstadmwhahaha that explains it20:30
lbragstadmwhahaha i deployed by overriding KeystoneWorkers and it worked... i'll get a patch up20:31
*** cfontain has joined #tripleo20:31
openstackgerritKevin Carter (cloudnull) proposed openstack/python-tripleoclient master: Fix delete plan  https://review.opendev.org/71154820:32
mwhahahahrm loading local files seems broken with overcloud deploy command20:33
mwhahaha-e from the local path doesn't seem to be found20:33
mwhahahai don't recall any patches around our template stuff lately, does anyone know of recent changes?20:33
mwhahahai take that back20:35
fultonjmwhahaha: i don't. that sounds scary. how would such a break pass any of the ci jobs20:35
mwhahahaEmilienM: we might want to revert https://review.opendev.org/#/c/642032/1520:35
openstackgerritLance Bragstad proposed openstack/tripleo-heat-templates master: Update the number of keystone workers  https://review.opendev.org/71154920:35
lbragstadmwhahaha done ^20:35
EmilienMyeah?20:35
mwhahahafun now i get     heat_resource_tree_params = heat_resource_tree['parameters']20:35
* mwhahaha flips tables20:35
mwhahahaand openstack overcloud pland elete is broken20:36
mwhahahafjdkhlfhdksjfhdasljfdshjkf20:36
EmilienMfaeioujefaoiaefiaefio20:36
mwhahahaAttributeError: 'ClientManager' object has no attribute 'create_mistral_context'20:36
EmilienMoops20:36
EmilienMhttps://review.opendev.org/#/c/705678/ ?20:36
mwhahahayes that doesn't work20:37
openstackgerritwes hayutin proposed openstack/paunch master: move centos-7 jobs to centos-8 where possible  https://review.opendev.org/71081620:37
mwhahahaso glad we're testing things20:37
EmilienMapparently not20:37
EmilienMd0ugal: ^ fyi should we revert as well?20:37
cloudnullmwhahaha Fix delete plan https://review.opendev.org/71154820:38
* mwhahaha files bug20:38
openstackgerritwes hayutin proposed openstack/paunch master: move centos-7 jobs to centos-8 where possible  https://review.opendev.org/71081620:38
EmilienMnice20:38
EmilienMcloudnull reads minds20:38
* cloudnull just ran into that issue today 20:39
mwhahahaok so now i can't deploy20:39
mwhahaha    heat_resource_tree_params = heat_resource_tree['parameters']20:39
mwhahahais failing on parameters20:39
mwhahahathat's teh check deprecate parameters function20:39
*** sgw has joined #tripleo20:39
mwhahahai vaguely recall a cix about that20:39
cloudnullmwhahaha arxcruz|rover has a fix for that  - https://review.opendev.org/#/c/711480/20:40
mwhahahawhy don't we have this key anymore20:41
mwhahahawonder if it's related to that previous heat patch20:41
openstackgerritAlex Schultz proposed openstack/tripleo-common master: Revert "Modify roles to remove unused services"  https://review.opendev.org/71155120:42
* mwhahaha tests20:42
cloudnullit looks like the key isn't there on first run?20:42
mwhahahabut there was20:42
mwhahahalike this just came up after it wasn't finding my files in the local dir that were not fully qualified20:42
* mwhahaha shrugs20:42
cloudnullI ran into that issue the other day as well, and posted https://review.opendev.org/#/c/711291/ but arxcruz|rover review was better.20:43
mwhahahasomething weird is going on20:43
mwhahahai guess i should deploy an overcloud by hand more often20:44
weshayhand built overcloud's a delicious treat20:44
mwhahahait's like getting punched in the face every command run20:44
*** TrevorV has quit IRC20:44
fultonjartisanal overclouds20:44
mwhahahaok so that unused service thing is not related to teh parameters20:45
*** sshnaidm is now known as sshnaidm|afk20:48
*** slaweq has joined #tripleo20:48
weshaythat's some sneaky shit right there https://review.opendev.org/#/c/711543/20:49
mwhahahaok so we have 0 coverage on including a file from the current working directory20:49
weshaylolz20:49
weshayha ha20:50
mwhahahaso who knows when this broke20:50
weshayjust saw that too20:50
* weshay throws things at cloudnull 20:50
weshayah.. I wonder if rdo triggered20:50
openstackgerritJose Luis Franco proposed openstack/tripleo-upgrade stable/train: Initital ffwd2 patch to automate OS upgrade.  https://review.opendev.org/71155220:50
weshaybut is overloaded20:50
weshaymwhahaha, https://review.opendev.org/#/c/710356/20:50
weshaybecause that had two jobs run20:51
mwhahahaweshay: i don't think we use --all-manageable in ci20:51
* mwhahaha shrugs20:51
weshaymwhahaha, ya.. it's running20:51
weshayhttps://review.rdoproject.org/zuul/status20:51
cloudnullI think there are differences due to 001 being novaless ?20:51
mwhahahahttps://bugs.launchpad.net/tripleo/+bug/186624920:53
openstackLaunchpad bug 1866249 in tripleo "using an environment file from the current working directory no longer works for overcloud deploy" [High,Triaged]20:53
* mwhahaha goes digging20:53
*** cgoncalves has quit IRC20:53
*** rcernin has joined #tripleo20:54
openstackgerritMerged openstack/tripleo-ansible master: Revert "Modify the introspection playbook to simplify client interactions"  https://review.opendev.org/71154320:54
*** cgoncalves has joined #tripleo20:56
*** jfrancoa has quit IRC20:57
openstackgerritKevin Carter (cloudnull) proposed openstack/tripleo-ansible master: Revert "Revert "Modify the introspection playbook to simplify client interactions""  https://review.opendev.org/71155320:57
* mwhahaha throws things at cloudnull 20:59
cloudnullwhat did i do now ?21:00
mwhahahai think youre cleanup of the os.makedirs in https://review.opendev.org/#/c/671878/44/tripleoclient/v1/overcloud_deploy.py broke the local file loading21:00
* mwhahaha is testing theory21:00
cloudnullbut that was months ago ?21:01
cloudnullwe're just seeing that now ?21:01
mwhahahayea cause we don't test non-fullyqualified files21:01
mwhahahain CI21:01
* mwhahaha flips tables21:01
cloudnullfun21:01
openstackgerritMerged openstack/tripleo-docs master: Fix a typo  https://review.opendev.org/71134621:02
*** ansmith has quit IRC21:02
mwhahahathis is why i always deploy by hand :/21:02
mwhahahauncovers so many of these regressions21:02
mwhahahatoo bad i hadn't done it in some time21:03
cloudnullsomething easy to fix ?21:04
*** cylopez has quit IRC21:04
mwhahahayea revert all the changes in that file21:06
mwhahahaspecifically i think it's the _Deploy_tripleo_heat_templates_tmpdir part21:07
*** xek_ has quit IRC21:07
mwhahahacause reverting that function seems to have fixed it21:07
mwhahahaan hour and 20mins later i'm finally deploying an overcloud21:08
*** cgoncalves has quit IRC21:10
*** cgoncalves has joined #tripleo21:11
cloudnullmwhahaha do you have a bug handy for the relative path thing ?21:16
*** alexmcleod has quit IRC21:17
mwhahahai got it21:17
*** rcernin has quit IRC21:18
*** dpawlik has quit IRC21:18
openstackgerritAlex Schultz proposed openstack/python-tripleoclient master: Partial Revert "Replace ansible shell with python runner"  https://review.opendev.org/71155621:19
mwhahahacloudnull: -^21:19
cloudnull++21:20
openstackgerritAlex Schultz proposed openstack/tripleo-common stable/stein: Use config for layers  https://review.opendev.org/71070521:24
openstackgerritFlavio Fernandes proposed openstack/tripleo-heat-templates master: Add Octavia OVN Provider configuration  https://review.opendev.org/70769521:30
openstackgerritSteve Baker proposed openstack/tripleo-ansible master: module tripleo_baremetal_expand_roles  https://review.opendev.org/70641721:30
openstackgerritSteve Baker proposed openstack/tripleo-ansible master: module tripleo_baremetal_check_existing  https://review.opendev.org/70704321:30
openstackgerritSteve Baker proposed openstack/tripleo-ansible master: module tripleo_baremetal_populate_environment  https://review.opendev.org/70729521:30
openstackgerritSteve Baker proposed openstack/tripleo-ansible master: module metalsmith_instances  https://review.opendev.org/70641821:30
openstackgerritSteve Baker proposed openstack/tripleo-ansible master: Playbooks for overcloud node (un)provision  https://review.opendev.org/70704421:30
openstackgerritSteve Baker proposed openstack/tripleo-ansible master: Auto-detect default overcloud-full image  https://review.opendev.org/71135921:30
*** slaweq has quit IRC21:33
openstackgerritSteve Baker proposed openstack/tripleo-quickstart-extras master: Do a local image copy if there is no glance  https://review.opendev.org/71135421:37
cloudnullstevebaker - i gave your reviews another spin last night, at least for provisioning, it seemed to be working great.21:42
ekultailsstevebaker: Hey, good morning Steve. I was wondering if you've encountered anything similar to this / what your thoughts on this bug are (see my last comment): https://bugs.launchpad.net/tripleo/+bug/186620421:42
openstackLaunchpad bug 1866204 in tripleo "OVB is stuck on providing nodes" [Critical,Triaged]21:42
*** raildo has quit IRC21:47
*** dsneddon has quit IRC21:48
*** bfournie has quit IRC21:50
*** cfontain_ has joined #tripleo21:58
*** cfontain has quit IRC22:01
*** cfontain_ has quit IRC22:03
*** ansmith has joined #tripleo22:10
openstackgerritwes hayutin proposed openstack/python-tripleoclient master: update layout for centos-8 containers-multinode  https://review.opendev.org/71148922:18
*** iurygregory has quit IRC22:18
openstackgerritKevin Carter (cloudnull) proposed openstack/tripleo-common master: Remove invalid mistral envocation and references  https://review.opendev.org/71156222:24
*** jmasud has quit IRC22:26
stevebakercloudnull: cool, thanks22:35
*** bfournie has joined #tripleo22:39
stevebakerekultails: my dev environment is OVB, and I occasionally have node clean failures/timeouts which could be what you're seeing. I haven't looked into it though, I just manually "manage" -> "provide" to trigger clean again.22:41
stevebakerekultails: if you have any leads I could investigate the next time it happens22:41
*** tkajinam has joined #tripleo22:45
ekultailsstevebaker: Cool, thanks. It's interesting that you occasionally run into it, too. I normally use pre-deployed servers so the error isn't something I've experienced before. I don't think we have much to go on besides the one CI failure. I'll get an e-mail thread going with you (and maybe some network folks?) once I can find out some more details.22:50
*** morazi has quit IRC22:50
*** rh-jelabarre has quit IRC23:11
*** cfontain_ has joined #tripleo23:18
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates stable/train: Remove comment about tripleo_container_manage being experimental  https://review.opendev.org/71156323:21
openstackgerritKevin Carter (cloudnull) proposed openstack/python-tripleoclient master: Remove mistral when running the validate_nodes  https://review.opendev.org/70955223:21
openstackgerritKevin Carter (cloudnull) proposed openstack/python-tripleoclient master: Remove mistral when running the register_or_update workflow  https://review.opendev.org/70958323:21
openstackgerritKevin Carter (cloudnull) proposed openstack/python-tripleoclient master: Remove mistral from the undeploy_roles workflow  https://review.opendev.org/70961423:21
openstackgerritKevin Carter (cloudnull) proposed openstack/python-tripleoclient master: Remove mistral from the configure workflow  https://review.opendev.org/70961623:21
openstackgerritKevin Carter (cloudnull) proposed openstack/python-tripleoclient master: Remove mistral from the configure_manageable_nodes workflow  https://review.opendev.org/70963323:21
openstackgerritKevin Carter (cloudnull) proposed openstack/python-tripleoclient master: Remove the mistral from the clean_.*_nodes workflow  https://review.opendev.org/70975523:21
openstackgerritKevin Carter (cloudnull) proposed openstack/python-tripleoclient master: Remove the mistral from the create_raid_configuration workflow  https://review.opendev.org/70985823:21
openstackgerritKevin Carter (cloudnull) proposed openstack/python-tripleoclient master: Remove mistral from the bios_configuration workflows  https://review.opendev.org/70986123:21
openstackgerritKevin Carter (cloudnull) proposed openstack/python-tripleoclient master: Client code to do node import with ansible instead of mistral  https://review.opendev.org/70456123:21
openstackgerritKevin Carter (cloudnull) proposed openstack/python-tripleoclient master: Remove mistral from the introspection workflows  https://review.opendev.org/70734823:21
openstackgerritKevin Carter (cloudnull) proposed openstack/python-tripleoclient master: Remove mistral when running the provide workflows  https://review.opendev.org/71030823:21
openstackgerritKevin Carter (cloudnull) proposed openstack/python-tripleoclient master: Remove the mistral from the deploy_plan workflow  https://review.opendev.org/71047923:22
openstackgerritKevin Carter (cloudnull) proposed openstack/python-tripleoclient master: Remove the mistral from the create_plan workflow  https://review.opendev.org/71156423:22
openstackgerritKevin Carter (cloudnull) proposed openstack/python-tripleoclient master: Remove mistral when running the update_plan workflow  https://review.opendev.org/71156523:22
EmilienMpoor CI23:23
cloudnullsadly had to rebase after some of the changes that went in today :(23:24
openstackgerritKevin Carter (cloudnull) proposed openstack/tripleo-ansible master: Ensure the os-cloud flag is used for CLI commands  https://review.opendev.org/71126023:29
openstackgerritChristophe Fontaine proposed openstack/tripleo-ansible master: 'tripleo_ovs_dpdk_lcore_list' should be optional  https://review.opendev.org/71136023:37
EmilienMade_lee, weshay https://review.opendev.org/#/c/711114/23:45
stevebakercloudnull, EmilienM: hey what is the plan for ansiblizing the overcloud deploy command. baremetal (un)provision are standalone commands but are also optionally integrated into the overcloud deploy command. I'd like to just swap out the workflow call with a playbook call but that would violate our .v1 .v2 pattern23:49
EmilienMstevebaker: you mean the tripleo-operator-ansible role for overcloud deploy?23:50
stevebakerEmilienM: I don't know!23:51
openstackgerritMerged openstack/tripleo-upgrade stable/train: Replace Docker service by Podman in roles_data.  https://review.opendev.org/71095523:55
stevebakerEmilienM: I mean what is the plan for "openstack overcloud deploy" not to call mistral? It looks like tripleo-operator-ansible/roles/tripleo_overcloud_deploy still calls it23:59
EmilienMoh23:59
cloudnullstevebaker for deploy_plan I just converted the workflow - https://review.opendev.org/#/c/710479/ - its now just executed all through the client, in a future iteration I'd like to make that, and others, a playbook, but my first priority was to get rid of mistral.23:59
EmilienMyeah what cloudnull  said23:59

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!