Friday, 2019-12-13

openstackgerritAlex Schultz proposed openstack/tripleo-heat-templates stable/stein: Add ntpdate to ensure hwclock sync  https://review.opendev.org/69881700:08
EmilienMsshnaidm|off, cloudnull: this missing interpreter line bug is weird00:23
EmilienMhttps://2d9f46ad209753f76f32-38806e64f2d3daae89d7ad3776e6aee4.ssl.cf1.rackcdn.com/696390/44/check/tripleo-ci-centos-7-containers-multinode/5b1c754/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz00:23
EmilienMand I depends-on your patch on config-template00:23
EmilienMI'm preparing a reproducer, maybe we can debug tomorrow00:28
*** bfournie has joined #tripleo00:34
*** ysandeep has joined #tripleo00:35
*** rcernin has joined #tripleo00:48
*** ekultails has quit IRC00:49
*** saneax has quit IRC00:51
*** stevebaker_ is now known as pequbot-test00:51
*** ekultails has joined #tripleo00:52
*** pequbot-test is now known as stevebaker_00:54
*** ade_lee has joined #tripleo00:55
*** ade_lee has quit IRC00:59
*** ade_lee has joined #tripleo00:59
sshnaidm|offEmilienM, try to run this task with -vvvv01:00
*** ysandeep has quit IRC01:01
sshnaidm|offEmilienM, and ANSIBLE_DEBUG=101:01
*** ade_lee has quit IRC01:03
*** ysandeep has joined #tripleo01:03
*** ade_lee has joined #tripleo01:03
*** ayoung has joined #tripleo01:10
openstackgerritMerged openstack/tripleo-common master: Fix ironic-pxe container healthcheck  https://review.opendev.org/69871901:23
cloudnullEmilienM ++ i'd like to nail that down01:27
cloudnullI spent some time trying to make the issue happen locally but I've not been able to01:27
openstackgerritLuke Short proposed openstack/tripleo-specs master: Add spec for AWX integration.  https://review.opendev.org/69882801:43
openstackgerritLuke Short proposed openstack/tripleo-specs master: Add spec for AWX integration.  https://review.opendev.org/69882801:44
*** hberaud|gone has quit IRC01:51
*** zbitter is now known as zaneb02:12
openstackgerritMerged openstack/tripleo-heat-templates stable/stein: [stein only] WIP Make podman update from 1.0.5 to 1.4.2.  https://review.opendev.org/69714402:29
openstackgerritMerged openstack/tripleo-heat-templates stable/stein: Add Octavia driver agent service  https://review.opendev.org/69613902:29
openstackgerritMerged openstack/tripleo-heat-templates master: Add netcontrold service for DPDK roles  https://review.opendev.org/69476502:29
*** mschuppert has quit IRC02:47
*** ysandeep has quit IRC02:55
*** nkinder has quit IRC02:57
*** ayoung has quit IRC02:59
*** ayoung has joined #tripleo03:00
openstackgerritMerged openstack/tripleo-ansible master: Introduce tripleo-keystone-resources role  https://review.opendev.org/69716103:02
*** ysandeep has joined #tripleo03:13
*** aakarsh has joined #tripleo03:31
*** psachin has joined #tripleo03:33
*** ysandeep has quit IRC03:36
*** ade_lee has quit IRC03:43
*** ade_lee has joined #tripleo03:43
*** skramaja has quit IRC03:44
*** bhagyashris has joined #tripleo03:54
*** rlandy has quit IRC04:07
*** udesale has joined #tripleo04:29
*** ykarel has joined #tripleo04:37
*** psachin has quit IRC04:48
*** epoojad1 has joined #tripleo04:53
*** pequbot-test has joined #tripleo04:53
pequbot-testURGENT TRIPLEO TASKS NEEDING ATTENTION: 1204:53
pequbot-testhttps://bugs.launchpad.net/tripleo/+bug/181770504:54
openstackLaunchpad bug 1817705 in tripleo "periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master fails tempest test related to DHCP agent" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)04:54
pequbot-testhttps://bugs.launchpad.net/tripleo/+bug/181917504:54
pequbot-testhttps://bugs.launchpad.net/tripleo/+bug/184003604:54
openstackLaunchpad bug 1819175 in tripleo "containers-multinode unable to pull images" [High,Triaged]04:54
pequbot-testhttps://bugs.launchpad.net/tripleo/+bug/184061604:54
pequbot-testhttps://bugs.launchpad.net/tripleo/+bug/184778704:54
openstackLaunchpad bug 1840036 in tripleo "[check][container-build] RHEL-8 patch pulled, needs to be rebased against kolla patch running in check" [Critical,Triaged] - Assigned to chandan kumar (chkumar246)04:54
openstackLaunchpad bug 1840616 in tripleo "Master/stein check and promotion OVB jobs are randomly giving ansible time out while overcloud deploy" [Critical,Triaged] - Assigned to Ronelle Landy (rlandy)04:54
openstackLaunchpad bug 1847787 in tripleo "refresh old CentOS-7 image settings w/ the latest" [Critical,In progress] - Assigned to Marios Andreou (marios-b)04:54
stevebaker_hey, pequbot-test is my new bot. It will run every 4 hours04:55
stevebaker_I'll kill it in ~16 hours, I just want to see how it runs overnight04:56
*** raukadah is now known as chkumar|rover05:08
*** zainub_wahid has joined #tripleo05:11
*** holser has joined #tripleo05:29
*** surpatil has joined #tripleo05:35
*** ade_lee has quit IRC05:39
*** holser has quit IRC05:46
*** skramaja has joined #tripleo06:00
*** pcaruana has joined #tripleo06:02
*** soniya29 has joined #tripleo06:08
*** yprokule has joined #tripleo06:14
*** bandini has joined #tripleo06:18
*** udesale has quit IRC06:25
*** lmiccini has joined #tripleo06:25
*** marios|ruck has joined #tripleo06:31
*** jbadiapa has joined #tripleo06:43
*** ramishra has joined #tripleo06:45
openstackgerritMatthias Runge proposed openstack/tripleo-heat-templates stable/train: Move the legacy telemetry environment template to correct location  https://review.opendev.org/69766506:48
openstackgerritMatthias Runge proposed openstack/tripleo-heat-templates stable/train: Remove ceph backend by default for gnocchi  https://review.opendev.org/69883606:48
*** jfrancoa has joined #tripleo06:51
openstackgerritSoniya Murlidhar Vyas proposed openstack/tripleo-quickstart-extras master: [Train] Remove passing tests from skiplist  https://review.opendev.org/69638707:17
openstackgerritFrancesco Pantano proposed openstack/tripleo-heat-templates stable/train: Add CephGrafana resource to the ControllerStorageNFS role  https://review.opendev.org/69884007:20
openstackgerritCédric Jeanneret (Tengu) proposed openstack/tripleo-common master: Remove workdir only if it's a temporary one  https://review.opendev.org/69870907:24
*** soniya29 has quit IRC07:39
*** mschuppert has joined #tripleo07:39
*** soniya29 has joined #tripleo07:40
*** hberaud has joined #tripleo07:40
*** rcernin has quit IRC07:41
*** tesseract has joined #tripleo07:43
*** saneax has joined #tripleo07:44
*** ykarel is now known as ykarel|lunch07:47
*** pkopec has joined #tripleo07:57
*** jtomasek has joined #tripleo08:03
*** slaweq has quit IRC08:21
openstackgerritMartin Schuppert proposed openstack/puppet-tripleo master: DNM neutron placement config  https://review.opendev.org/69884308:23
*** amoralej|off is now known as amoralej08:27
*** jpich has joined #tripleo08:29
*** iurygregory has joined #tripleo08:33
openstackgerritmathieu bultel proposed openstack/tripleo-quickstart-extras master: Add when check for validate services include  https://review.opendev.org/69866308:35
*** tosky has joined #tripleo08:40
*** slaweq has joined #tripleo08:42
openstackgerritmathieu bultel proposed openstack/tripleo-ci master: Do not override featureset value for UC upgrade  https://review.opendev.org/69884908:42
openstackgerritmathieu bultel proposed openstack/tripleo-quickstart-extras master: Add when check for validate services include  https://review.opendev.org/69866308:43
*** ykarel|lunch is now known as ykarel08:45
*** tmazur has joined #tripleo08:48
pequbot-testURGENT TRIPLEO TASKS NEEDING ATTENTION: 1108:53
pequbot-testhttps://bugs.launchpad.net/tripleo/+bug/184003608:53
openstackLaunchpad bug 1840036 in tripleo "[check][container-build] RHEL-8 patch pulled, needs to be rebased against kolla patch running in check" [Critical,Triaged] - Assigned to chandan kumar (chkumar246)08:53
pequbot-testhttps://bugs.launchpad.net/tripleo/+bug/184061608:53
pequbot-testhttps://bugs.launchpad.net/tripleo/+bug/184778708:53
openstackLaunchpad bug 1840616 in tripleo "Master/stein check and promotion OVB jobs are randomly giving ansible time out while overcloud deploy" [Critical,Triaged] - Assigned to Ronelle Landy (rlandy)08:53
pequbot-testhttps://bugs.launchpad.net/tripleo/+bug/184856408:53
pequbot-testhttps://bugs.launchpad.net/tripleo/+bug/185301208:53
openstackLaunchpad bug 1847787 in tripleo "refresh old CentOS-7 image settings w/ the latest" [Critical,In progress] - Assigned to Marios Andreou (marios-b)08:53
openstackLaunchpad bug 1848564 in tripleo "rhel8 buildimage oc failing: incorrect deps repo" [Critical,Triaged] - Assigned to Rafael Folco (rafaelfolco)08:53
openstackLaunchpad bug 1853012 in tripleo "Standalone upgrade job is failing - On ovn upgrade" [Critical,In progress] - Assigned to mathieu bultel (mat-bultel)08:53
*** jpena|off is now known as jpena08:54
openstackgerritDamien Ciabrini proposed openstack/tripleo-common stable/train: Fix ironic-pxe container healthcheck  https://review.opendev.org/69885109:04
marios|ruckbandini: o/ commented @ https://review.opendev.org/#/c/698628/1 it's another bug am afraid but you might get lucky09:05
marios|ruckbandini: i think if you sacrifice a gfidente should be good09:05
marios|rucklmiccini: ^09:06
*** lucasagomes has joined #tripleo09:06
bandinimarios|ruck: I think https://review.opendev.org/#/c/698851/ should do it?09:06
marios|ruckbandini: looking .. .nice matbu fyi ^^09:07
marios|ruckbandini: matbu was looking into that too09:07
marios|ruckbandini: thanks (can we sacrifice a gfidente anyway though?)09:07
bandinialways ;)09:07
marios|ruckbandini: but then why does it sometimes pass09:07
bandinii have not dug into it all too much tbh09:08
marios|ruckbandini: k thanks for pointer anyway... i may mark the new bug as duplicate then ... but not sure mainly because this is happening only some of the time for train09:09
*** rcernin has joined #tripleo09:15
*** gfidente has joined #tripleo09:15
openstackgerritDougal Matthews proposed openstack/tripleo-common master: Remove workdir only if it's a temporary one  https://review.opendev.org/69870909:18
*** holser has joined #tripleo09:18
*** tkajinam has quit IRC09:19
openstackgerritDougal Matthews proposed openstack/tripleo-common master: Remove workdir only if it's a temporary one  https://review.opendev.org/69870909:19
openstackgerritMerged openstack/puppet-tripleo master: Enable deployment with external loadbalancers  https://review.opendev.org/69823909:22
openstackgerritMerged openstack/tripleo-heat-templates master: Add ClientRetryLimit parameter for heat  https://review.opendev.org/69800809:22
openstackgerritMerged openstack/tripleo-heat-templates stable/stein: Relax filtering in krb-service-principals jinja  https://review.opendev.org/69759809:22
openstackgerritMerged openstack/tripleo-heat-templates stable/stein: Add Octavia anti-affinity parameters  https://review.opendev.org/69824709:22
*** derekh has joined #tripleo09:30
openstackgerritDaniel Bengtsson proposed openstack/tripleo-upgrade master: Clean up ansible variables.  https://review.opendev.org/56295709:32
*** suuuper has joined #tripleo09:33
openstackgerritwangfaxin proposed openstack/puppet-tripleo master: Update the community page.  https://review.opendev.org/69885609:35
chkumar|roverEmilienM: https://bugs.launchpad.net/tripleo/+bug/1856278 please have a look when free, thanks!09:36
openstackLaunchpad bug 1856278 in tripleo "RHEL8 scenario 1 standalone deployment failed with The following containers failed validations and were not started: collectd" for master and train" [Critical,Confirmed]09:36
*** pbandark has joined #tripleo09:37
openstackgerritFrancesco Pantano proposed openstack/tripleo-heat-templates stable/train: Add CephGrafana resource to the ControllerStorageNFS role  https://review.opendev.org/69884009:45
*** alexmcleod has joined #tripleo09:45
openstackgerritDaniel Bengtsson proposed openstack/tripleo-upgrade master: Clean up ansible variables.  https://review.opendev.org/56295709:47
*** rcernin has quit IRC09:53
Tenguchkumar|rover: heya! https://bugs.launchpad.net/tripleo/+bug/1856278/comments/1  have probably found the RCA of the new promotion blocker.10:10
openstackLaunchpad bug 1856278 in tripleo "RHEL8 scenario 1 standalone deployment failed with The following containers failed validations and were not started: collectd" for master and train" [Critical,Confirmed]10:10
openstackgerritMartin Schuppert proposed openstack/puppet-tripleo master: DNM neutron placement config  https://review.opendev.org/69884310:10
chkumar|roverTengu: :-) patch got merged 9 weeks ago, so many promotion happens :-)10:11
Tenguoh. wait. what10:12
Tengupfrrrt. should clean my glasses.10:12
chkumar|roverTengu: if you mean this review https://review.opendev.org/#/c/682039/ then10:13
Tenguchkumar|rover: yeah -.- I've seen 13 which was matchin today - month got dropped in my reading for some reason10:14
chkumar|roverTengu: it may be a side effect of the patch https://review.opendev.org/#/c/697666/10:14
*** ansmith_ has joined #tripleo10:15
Tenguwell, at least it fails for a good reason. Now I'm wondering why /var/log/journal isn't there o_O10:16
Tenguso that patch you mention did put the issue into light imho.10:17
Tenguthe rca is still the modification I'm pointing10:17
*** ansmith has quit IRC10:18
*** marios|ruck has quit IRC10:18
chkumar|roverTengu: yup :-)10:18
Tenguand I'm a bit surprised this log file isn't present... there's something fishy.10:19
Tenguoh..... my undercloud being deployed right now doesn't have that logfile (rhel-8 though)10:19
Tenguthat's.... annoying10:19
Tenguso - YES, that's the real issue :)10:19
Tenguwondering if it shouldn't be /var/log/messages instead.10:23
*** florianf has joined #tripleo10:31
openstackgerritSaravanan KR proposed openstack/tripleo-heat-templates stable/train: Add netcontrold service for DPDK roles  https://review.opendev.org/69886910:34
paramiteHey guys, could we please move forward with https://review.opendev.org/#/c/695859/ and https://review.opendev.org/#/c/696071/ ?10:37
*** marios|ruck has joined #tripleo10:43
openstackgerritGael Chamoulaud proposed openstack/python-tripleoclient master: TripleO Validations Logging CLI Introduction  https://review.opendev.org/69539710:51
*** yolanda has quit IRC10:53
*** ykarel is now known as ykarel|afk10:57
*** hberaud is now known as hberaud|lunch11:00
*** ansmith has joined #tripleo11:17
*** ansmith_ has quit IRC11:17
openstackgerritChandan Kumar (raukadah) proposed openstack/tripleo-ci master: Put selinux_enforcing flag to fs override  https://review.opendev.org/69887611:18
*** yolanda has joined #tripleo11:25
openstackgerritMarios Andreou proposed openstack/tripleo-ci master: DNM DEBUG testing https://review.rdoproject.org/r/#/c/23919/  https://review.opendev.org/69888311:33
marios|ruckchkumar|rover: ^^11:33
openstackgerritChandan Kumar (raukadah) proposed openstack/tripleo-ci master: set selinux to permissive for image build  https://review.opendev.org/69887611:33
chkumar|rovermarios|ruck: cool11:33
*** raildo has joined #tripleo11:34
marios|ruckchkumar|rover: you posted same then https://review.opendev.org/#/c/698876/2/roles/oooci-build-images/templates/build-images.sh.j2 :D11:34
marios|ruckchkumar|rover: race ?11:34
marios|ruck;)11:34
marios|ruckhttps://review.rdoproject.org/r/#/c/23919/ updated lets see11:34
marios|ruckchkumar|rover: ^11:34
chkumar|roveryes11:35
chkumar|rovermarios|ruck: sorry the fs ovrides needs to be removed from aboev patch?11:37
*** epoojad1 is now known as epoojad1|afk11:39
*** ykarel|afk is now known as ykarel11:42
marios|ruckchkumar|rover: ?11:43
marios|ruckchkumar|rover: i removed it11:43
marios|ruckchkumar|rover: 13:37 < chkumar|rover> marios|ruck: sorry the fs ovrides needs to be removed from aboev patch?11:44
marios|ruckwut?11:44
*** epoojad1|afk has quit IRC11:44
*** hrw has joined #tripleo11:46
hrwhi11:46
hrwKolla in 'base' image has base_rhel_package_installation block used only by tripleo. How often you do builds with "install_type != binary or install_metatype != rdo"?11:48
*** zainub_wahid has quit IRC11:51
openstackgerritMerged openstack/python-tripleoclient master: Retry the connection to heat-api when 500 code  https://review.opendev.org/69793911:51
*** bhagyashris has quit IRC11:52
*** rfolco has joined #tripleo11:58
*** dtantsur|afk is now known as dtantsur12:02
EmilienMchkumar|rover: I'm not working on collectd things unless I have to, I would contact those who maintain its integration, like Martin cc paramite12:03
EmilienMparamite: if not already the case, please urgently look at https://bugs.launchpad.net/tripleo/+bug/185627812:03
openstackLaunchpad bug 1856278 in tripleo "RHEL8 scenario 1 standalone deployment failed with The following containers failed validations and were not started: collectd" for master and train" [Critical,Confirmed]12:03
*** suuuper has quit IRC12:06
paramitechecking12:08
paramiteyeah we need that to be mounted, already discussing this with cjeanner12:09
openstackgerritSoniya Murlidhar Vyas proposed openstack/tripleo-quickstart-extras master: [Master] Remove passing tests from skiplist  https://review.opendev.org/69888712:09
paramiteSo solution could be to create the path before spawning the container ... EmilienM any objection to that?12:09
*** lucasagomes has quit IRC12:09
*** EmilienM has quit IRC12:12
*** EmilienM has joined #tripleo12:13
*** ChanServ sets mode: +v EmilienM12:13
openstackgerritLee Yarwood proposed openstack/tripleo-heat-templates master: Mount /boot from the host within the nova-compute container  https://review.opendev.org/69888812:14
*** EmilienM is now known as EvilienM12:17
*** kozhukalov has joined #tripleo12:25
*** jpena is now known as jpena|lunch12:29
openstackgerritMartin Mágr proposed openstack/tripleo-heat-templates master: Create journal directory ahead  https://review.opendev.org/69888912:31
paramiteEvilienM, ^12:32
paramiteEvilienM, got a bad day today? :)12:32
EvilienMparamite: please add bug number 185627812:32
EvilienMparamite: ahah no it's 'just' my casual friday nickname12:32
paramiteah yes, sorry12:33
openstackgerritMartin Mágr proposed openstack/tripleo-heat-templates master: Create journal directory ahead  https://review.opendev.org/69888912:33
openstackgerritJose Luis Franco proposed openstack/tripleo-heat-templates master: Force facts cache refreshing before upgrade.  https://review.opendev.org/69889112:42
chkumar|roverparamite: EvilienM thank for the fix12:42
chkumar|roverparamite: we need to backport that patch to train also12:42
paramiteyup12:42
openstackgerritJose Luis Franco proposed openstack/tripleo-heat-templates master: Force facts cache refreshing before upgrade.  https://review.opendev.org/69889112:43
*** hberaud|lunch is now known as hberaud12:45
*** rh-jelabarre has joined #tripleo12:46
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: deploy-steps: use lookup() to get pre|post_deployments_role variables  https://review.opendev.org/69889312:47
openstackgerritJose Luis Franco proposed openstack/tripleo-heat-templates stable/stein: Force facts cache refreshing before upgrade.  https://review.opendev.org/69889412:48
*** ykarel is now known as ykarel|away12:48
*** kozhukalov has quit IRC12:48
*** kozhukalov has joined #tripleo12:48
EvilienMdciabrin_: do we want https://review.opendev.org/#/c/698851/ in stein as well?12:51
dciabrin_EvilienM, i think it doesn't harm, yes12:51
openstackgerritMerged openstack/tripleo-heat-templates stable/train: Add healthcheck for nova-virtlogd container  https://review.opendev.org/69710912:52
EvilienMdciabrin_: ok I'll do it12:52
*** lbragstad_ has quit IRC12:53
EvilienMdciabrin_: ah not needed apaprently12:53
EvilienMit's doing:12:53
EvilienMbind_host=$(ps -ef | grep -oe '--address \([0-9]\+\.\?\)\+' | awk '{print $2}')12:53
EvilienMvs bind_host=$(hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml)12:53
* EvilienM moves on12:53
pequbot-testURGENT TRIPLEO TASKS NEEDING ATTENTION: 1212:53
pequbot-testhttps://bugs.launchpad.net/tripleo/+bug/184003612:53
openstackLaunchpad bug 1840036 in tripleo "[check][container-build] RHEL-8 patch pulled, needs to be rebased against kolla patch running in check" [Critical,Triaged] - Assigned to chandan kumar (chkumar246)12:53
pequbot-testhttps://bugs.launchpad.net/tripleo/+bug/184061612:53
pequbot-testhttps://bugs.launchpad.net/tripleo/+bug/184778712:53
openstackLaunchpad bug 1840616 in tripleo "Master/stein check and promotion OVB jobs are randomly giving ansible time out while overcloud deploy" [Critical,Triaged] - Assigned to Ronelle Landy (rlandy)12:53
pequbot-testhttps://bugs.launchpad.net/tripleo/+bug/184856412:53
pequbot-testhttps://bugs.launchpad.net/tripleo/+bug/185301212:53
openstackLaunchpad bug 1847787 in tripleo "refresh old CentOS-7 image settings w/ the latest" [Critical,In progress] - Assigned to Marios Andreou (marios-b)12:53
openstackLaunchpad bug 1848564 in tripleo "rhel8 buildimage oc failing: incorrect deps repo" [Critical,Triaged] - Assigned to Rafael Folco (rafaelfolco)12:53
openstackLaunchpad bug 1853012 in tripleo "Standalone upgrade job is failing - On ovn upgrade" [Critical,In progress] - Assigned to mathieu bultel (mat-bultel)12:53
*** lmiccini has quit IRC12:53
dciabrin_EvilienM, ah boom. excellent12:54
EvilienMweshay: omg can we NOT have the bot please ^12:54
EvilienMchkumar|rover: what is that bot ^12:55
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates stable/train: Enable additional Swift healtchecks  https://review.opendev.org/69889612:56
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates stable/stein: Enable additional Swift healtchecks  https://review.opendev.org/69889712:56
chkumar|roverEvilienM: stevebaker_ brought it, he will silence it in 16 hours12:56
chkumar|roverwhen we comes online12:56
*** morazi has joined #tripleo12:58
*** jpich has quit IRC12:59
*** akahat has joined #tripleo12:59
*** ykarel_ has joined #tripleo13:00
openstackgerritKamil Sambor proposed openstack/tripleo-heat-templates stable/train: Change default value for NeutronPluginExtensions  https://review.opendev.org/69889913:00
openstackgerritEmilien Macchi proposed openstack/python-tripleoclient stable/train: Retry the connection to heat-api when 500 code  https://review.opendev.org/69890013:01
openstackgerritEmilien Macchi proposed openstack/python-tripleoclient stable/stein: Retry the connection to heat-api when 500 code  https://review.opendev.org/69890113:01
openstackgerritEmilien Macchi proposed openstack/python-tripleoclient stable/rocky: Retry the connection to heat-api when 500 code  https://review.opendev.org/69890213:01
openstackgerritEmilien Macchi proposed openstack/python-tripleoclient stable/queens: Retry the connection to heat-api when 500 code  https://review.opendev.org/69890313:01
*** ykarel|away has quit IRC13:03
*** ykarel_ has quit IRC13:34
*** rlandy has joined #tripleo13:35
*** jpena|lunch is now known as jpena13:37
weshayEvilienM, is that the alert bot?13:37
weshaythat was turned off for quite some time, I thought you guys turned it off... I can and should be off13:37
weshayI'm not sure where it's running tbh13:37
weshayEvilienM, we could kickban pequbot-test13:43
*** Goneri has joined #tripleo13:45
*** soniya29 has quit IRC13:47
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates stable/stein: deploy-steps: use lookup() to get pre|post_deployments_role variables  https://review.opendev.org/69890714:00
cloudnullmornings14:01
openstackgerritBrent Eagles proposed openstack/tripleo-quickstart-extras master: WIP: Allow full path to be specified for amphora destination  https://review.opendev.org/69285814:07
openstackgerritBrent Eagles proposed openstack/tripleo-quickstart-extras master: WIP: Allow full path to be specified for amphora destination  https://review.opendev.org/69285814:08
openstackgerritGael Chamoulaud proposed openstack/python-tripleoclient master: TripleO Validator CLI Refactor  https://review.opendev.org/69355214:09
openstackgerritGael Chamoulaud proposed openstack/python-tripleoclient master: Replace ansible shell with python runner  https://review.opendev.org/67187814:09
openstackgerritGael Chamoulaud proposed openstack/python-tripleoclient master: Overload stdout callback plugin  https://review.opendev.org/69801514:09
openstackgerritGael Chamoulaud proposed openstack/python-tripleoclient master: TripleO Validations Logging CLI Introduction  https://review.opendev.org/69539714:09
*** skramaja has quit IRC14:13
*** mmethot has joined #tripleo14:13
*** epoojad1 has joined #tripleo14:13
openstackgerritMerged openstack/tripleo-common stable/train: Revert "Update hostvars lookup to fix regression"  https://review.opendev.org/69872714:15
*** amoralej is now known as amoralej|lunch14:15
openstackgerritJose Luis Franco proposed openstack/tripleo-upgrade stable/train: Fix the undefined operation_type variable.  https://review.opendev.org/69891114:16
marios|ruckslaweq: o/ can i have pub keys please cc jpena14:18
marios|ruckslaweq: am asking jpena to hold the node https://review.rdoproject.org/r/#/c/23919/ periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-master14:19
marios|ruckjpena: me @ https://github.com/marios.keys please14:19
marios|ruckslaweq: we finally got round the image build and the fs1 is running looks liike its gonna timeout on the issue we want to check14:20
Tengufolks, can we get a merge for https://review.opendev.org/698232 and https://review.opendev.org/698238  ? they need to be backported... please? :)14:20
slaweqmarios|ruck: my public key http://paste.openstack.org/show/787558/14:21
slaweqmarios|ruck: but I'm not sure if I will be able to check that today14:21
marios|ruckslaweq: ack jpena maybe its better to delete it before you leave today then ? and we can request again on monday?14:22
marios|ruckslaweq: adding your key now (thanks jpena am in)14:22
slaweqmarios|ruck: now I have meeting14:23
slaweqI will try to check that after 4 pm14:23
slaweqplease give me host IP address14:23
marios|ruckslaweq: can you check just for ssh access please ssh zuul@38.145.34.714:24
marios|ruckslaweq: ack i have meeting in 5 mins too14:24
slaweqmarios|ruck: it works, thx14:24
marios|ruckslaweq: but we can avoid holding it until monday though ... perhaps we can release it before jpena leaves (is it soon?) otherwise we can try again on monday14:24
marios|ruckslaweq: at least we have a way around the image build thing14:24
marios|ruckack thanks slaweq thanks jpena just ping us when you want to leave and you can clear the node14:24
marios|ruckno need to hold it whole weekend for nothing14:25
*** aedc_ has joined #tripleo14:25
slaweqsure, we can unhold it today and try again on monday14:25
slaweqbut if possible to keep it on hold for next 1,5h, I will try to check today14:25
marios|ruckslaweq: hmmm getenforce says 'Enforcing' i wonder if that is the same issue14:25
openstackgerritJose Luis Franco proposed openstack/tripleo-heat-templates master: Force facts cache refreshing before upgrade.  https://review.opendev.org/69889114:25
marios|ruckslaweq: https://bugs.launchpad.net/tripleo/+bug/1853028/comments/1314:26
openstackLaunchpad bug 1853028 in tripleo "Build overcloud image for rhel8 fails sometimes on in_target.d/post-install.d/51-enable-network-service" [Critical,Triaged]14:26
marios|ruckchkumar|rover: maybe same issue for fs1 ... indeed it is enforcing on that node14:26
openstackgerritCédric Jeanneret (Tengu) proposed openstack/tripleo-heat-templates master: Ensure /var/log/journal exists as soon as possible  https://review.opendev.org/69891414:27
*** aedc has quit IRC14:27
cloudnullweshay - https://review.opendev.org/#/c/698620 seems to be getting further now (multi-node job). It failed tempest, maybe a fluke I'm rechecking to see what's up (but its not failing like it was yesterday) - so it looks like https://review.opendev.org/#/c/698812 has an impact on the multi-node jobs, when mistral is invoked -cc mwhahaha sshnaidm|off14:29
*** TrevorV has joined #tripleo14:29
*** mcornea has joined #tripleo14:30
weshaycloudnull, /me looks at ovb in a sec.. but multinode failed .. https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_5f8/698620/6/check/tripleo-ci-centos-7-containers-multinode/5f83bbc/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz14:35
weshay2019-12-13 00:11:33 |         "2019-12-13 00:11:29,392 ERROR: 25576 -- Failed running container for crond",14:35
weshay2019-12-13 00:11:33 |         "Error: unable to pull 192.168.24.1:8787/tripleomaster/centos-binary-cron:38c4e3104abdeb4699cfbe7a78fa2f37d7a863b4_93bde36c-updated-20191212223733: unable to pull image: Error initializing source docker://192.168.24.1:8787/tripleomaster/centos-binary-cron:38c4e3104abdeb4699cfbe7a78fa2f37d7a863b4_93bde36c-updated-20191212223733: pinging docker registry returned: Get http://192.168.24.1:8787/v2/: dial tcp14:36
weshay192.168.24.1:8787: connect: no route to host",14:36
weshaythat looks like it should be transient14:36
*** dmellado has quit IRC14:36
cloudnull++14:36
cloudnullit got past the [tripleo-firewall : Manage firewall rules] task14:36
weshaycloudnull, the fs001 ovb jobs are getting killed by ironic changes14:37
weshaywe've escalated networking and ironic on that14:37
*** ramishra has quit IRC14:37
cloudnull++14:37
weshayhow was dinner?14:37
chkumar|rovermarios|ruck: fs1 train rhel8?14:38
*** dmellado has joined #tripleo14:39
chkumar|rovercloudnull: please have a look at this one http://logs.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-train/e2b80c8/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz I am not sure what is going on, the selinux is permissive there14:40
EvilienMsshnaidm|off, cloudnull : re https://review.opendev.org/#/c/698812/1/roles/overcloud-deploy/templates/ansible.cfg.j214:41
mwhahahachkumar|rover: i think it's enforcing at that point14:41
EvilienMsshnaidm|off, cloudnull : IMHO we should stop overriding ansible.cfg, it's confusing and also error prone; config-download (tripleo-common code) already generates that file14:41
EvilienMsshnaidm|off, cloudnull : if we need "ara" support in it, then we need to add an option in tripleo-common14:41
cloudnullmwhahaha chkumar|rover ++14:41
mwhahahachkumar|rover: hm i see the selinux bits are run before the failure14:42
EvilienMand have a toggle in CI to enable it14:42
cloudnullEvilienM ++14:42
cloudnullwe could just add it to the config14:42
cloudnullif the directories or callbacks are not found ansible will simply omit them14:42
sshnaidm|offEvilienM, special option for ara only?14:43
EvilienMis there a lazy import somehow?14:44
chkumar|rovermwhahaha: Does selinux bits needs adjustment? or we need to set permissive on the undercloud itself?14:44
cloudnullreally no need. we can do the reverse of what I did here - https://review.opendev.org/#/c/69881214:44
mwhahahachkumar|rover: no it's failing ont eh remote systems14:44
cloudnulljust add the ara path/callbacks and if they're found they'll be used.14:44
EvilienMcloudnull: esrever eht?14:45
mwhahahachkumar|rover: one thing we could do is do a debug print of the contents of /etc/rc0.d to ensure it exists14:45
cloudnullEvilienM WAT?14:45
sshnaidm|offcloudnull, to add by default to ansible.cfg generated by triple-common?14:45
chkumar|rovermwhahaha: will do that14:45
cloudnull++14:45
EvilienMcloudnull: the reverse :D14:45
* cloudnull pushing a review 14:45
sshnaidm|offI don't think it's good idea to hardcode external plugin config in our code, what if something changes in ara in next versions14:46
* dmsimard ara spider senses tingling14:46
sshnaidm|offdmsimard, :D do you highlight "ara"?14:46
dmsimardsshnaidm|off: yes :p14:46
cloudnullsshnaidm|off its effectively hard-coded now in oooq14:46
EvilienMsshnaidm|off: my problem is with overriding in oooq something we already generate in the product itself14:47
mwhahahaEvilienM: it's supposed to be an override file meaning it gets merged14:47
sshnaidm|offcloudnull, yeah, it's just a temporary CI framework14:47
mwhahahait's designed this way14:47
mwhahahaspecifically for this14:47
EvilienMit's just confusing, all i'm saying14:47
sshnaidm|offEvilienM, we actually add, not override iirc14:47
*** derekh has quit IRC14:48
EvilienMbut anyway, let's move on14:48
*** amoralej|lunch is now known as amoralej14:48
cloudnullmwhahaha I dont think its merging14:48
mwhahahait's an override file14:48
mwhahahaso who knows14:48
sshnaidm|offBut the problem that it's only for "overcloud deploy"14:48
sshnaidm|offit's not for undercloud or standalone14:48
cloudnullwithout https://review.opendev.org/#/c/698812 this review fails https://review.opendev.org/#/c/698620 (failing to find the plugins)14:48
*** artom has joined #tripleo14:50
chkumar|rovertbarron: Hello14:50
cloudnullso it seems, when the override is provided, it replaces what we generate in tripleo-common; on the overcloud14:50
chkumar|rovertbarron: please have a look at manila temepst test failure http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-scenario004-standalone-master/5049a8c/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz14:51
sshnaidm|offcloudnull, https://github.com/openstack/tripleo-common/blob/master/tripleo_common/actions/ansible.py#L142:L14214:51
tbarronchkumar|rover: will do14:51
mwhahahaEvilienM, cloudnull, sshnaidm|off: https://github.com/openstack/tripleo-common/blob/master/tripleo_common/actions/ansible.py#L138  correct it does not merge14:51
chkumar|rovertbarron: from here http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-scenario004-standalone-master/5049a8c/logs/undercloud/var/log/containers/manila/manila-scheduler.log.txt.gz -> ] Failed to schedule create_share: No valid host was found. Failed to find a weighted host, the last executed filter was AvailabilityZoneFilter.: NoValidHost:14:51
chkumar|roverNo valid host was found. Failed to find a weighted host, the last executed filter was AvailabilityZoneFilter.14:51
sshnaidm|offcloudnull, it really overrides14:51
mwhahahaso if we update tripleo-common it also needs to be updated in quickstart currently14:52
EvilienMchkumar|rover: "Share type not found"14:52
cloudnullmwhahaha++14:52
EvilienMchkumar|rover: https://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-scenario004-standalone-master/5049a8c/logs/undercloud/var/log/containers/manila/manila-api.log.txt.gz#_2019-12-13_10_47_16_36814:52
mwhahahai thought we added bits to allow for tweaking and not complete overriding14:52
*** aakarsh has quit IRC14:52
*** iurygregory has quit IRC14:52
cloudnull¯\_(ツ)_/¯14:52
mwhahahahttps://github.com/openstack/tripleo-common/blob/master/tripleo_common/actions/ansible.py#L14214:52
sshnaidm|offI also remember "merge" somewhere, maybe confused with something..14:52
mwhahahait overwrites everything14:53
mwhahahaif defined14:53
chkumar|rovermarios|ruck: ^^14:53
mwhahahaso you can overwrite a specific setting completely14:53
mwhahahaso if we're tweaking paths in quickstart, it needs to be updated to include whatever we have defined in tripleo-common14:53
cloudnullhttps://review.opendev.org/#/c/698812 ^14:54
sshnaidm|offmaybe to change it to smart merge?14:54
mwhahahabut then you can't disable stuff14:54
mwhahahaso 'override' pretty much means take this value14:55
EvilienMdo we actually use this a-r-a on the overcloud in CI ? I thought we stopped it because it was taking too much time14:56
sshnaidm|offmaybe to add env var support, like if we need to add path, we can use like ANSIBLE_.._PATH=${ANSIBLE_.._PATH}14:56
sshnaidm|offEvilienM, yeah, we do14:56
*** ade_lee has joined #tripleo14:56
sshnaidm|offEvilienM, and want to use it on undercloud too14:56
*** kozhukalov has quit IRC14:56
sshnaidm|offEvilienM, otherwise we don't have data about tasks length14:57
chkumar|roverslaweq: Hello, please have a look at this comment https://bugs.launchpad.net/tripleo/+bug/1855985/comments/12 need some help on how to proceed there14:57
openstackLaunchpad bug 1855985 in tripleo "tempest network test_list_agent test failed with mismatch error in train fs020" [Critical,Confirmed]14:57
* cloudnull really likes ARA for the purpose of debugging 14:57
sshnaidm|offwhich would be very useful for performance degradation measurement14:57
sshnaidm|offcloudnull, yes, and this is too14:58
EvilienM++ me too I like it, i just don't like the way we configure ansible.cfg in oooq14:58
tbarronEvilienM: ty14:58
cloudnullEvilienM ++14:58
sshnaidm|offEvilienM, yeah, seems like it requires redesign14:58
sshnaidm|offthis exists as a hack now, and with all mistral removing..14:59
tbarronchkumar|rover: as EvilienM indicated, failure on 'share type not found', so we need to see what changed since last time it worked, do you have a pointer to prev successful run14:59
EvilienMmwhahaha, cloudnull : see my tmate, the clouds.yaml worked14:59
EvilienMso it's really in CI that is the problem, same problem as cloudnull with firewall15:00
cloudnull++15:00
tbarronchkumar|rover: i'm assuming this is a *new* issue15:00
chkumar|rovertbarron: pulling from here https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-scenario004-standalone-master15:00
tbarronchkumar|rover: ack, ty15:02
openstackgerritAlex Schultz proposed openstack/paunch stable/train: builder: include environment when running an exec  https://review.opendev.org/69892815:04
openstackgerritAlex Schultz proposed openstack/paunch stable/stein: builder: include environment when running an exec  https://review.opendev.org/69892915:05
weshayEvilienM, agree re: ansible.cfg15:06
weshayI don't like how we're doing that either15:07
openstackgerritJose Luis Franco proposed openstack/tripleo-heat-templates master: Force facts cache refreshing before upgrade.  https://review.opendev.org/69889115:07
slaweqchkumar|rover: I just replied to https://bugs.launchpad.net/tripleo/+bug/1855985/comments/1215:07
openstackLaunchpad bug 1855985 in tripleo "tempest network test_list_agent test failed with mismatch error in train fs020" [Critical,Confirmed]15:07
weshaycloudnull, EvilienM ideally we find a way IMHO to incorporate ara more into tripleo and less of a CI thing15:08
openstackgerritJose Luis Franco proposed openstack/tripleo-heat-templates stable/stein: Force facts cache refreshing before upgrade.  https://review.opendev.org/69889415:09
weshaybut others will probably not agree15:09
dmsimardweshay: would be easier on EL815:10
dmsimardwhen is tripleo moving to EL8 ?15:12
weshaydmsimard, work will start Jan 315:12
mwhahahais ara a shipped package yet?15:13
dmsimardmwhahaha: in fedora 30, 31 and 3215:14
mwhahahaso that's a no15:14
weshaydmsimard, get er done15:14
chkumar|roverslaweq: sure will do that15:14
dmsimardmwhahaha, weshay: would love help for EL8 but there's a story for it https://tree.taiga.io/project/morucci-software-factory/us/314715:14
*** leanderthal has joined #tripleo15:15
dmsimardarrfab (from centos infrastructure team) is also interested in this but he is using virtualenv for now15:15
mwhahahathe issue being that i'm not particularly interested in taking on the software ownership in tripleo for ara so there'd need to be someone officially on the hook for it when it breaks15:15
weshayagree15:16
*** ykarel_ has joined #tripleo15:16
dmsimardthe complicated part of the packaging is the API server -- the ansible plugin bits are easy and might already be packageable15:16
dmsimardthe API is stateless (when using a remote database) so it's a good fit for living in a container i.e, https://review.opendev.org/#/c/687905/2/contrib/containers/fedora-rawhide-ara-api.sh15:17
*** rainleander has joined #tripleo15:17
dmsimardmwhahaha: understandable :)15:17
weshaydmsimard, ok.. asking Alan about it15:18
dmsimardalso -- something else to consider would be some sort of "ara as a service"15:18
weshayaraaas15:18
dmsimardwhere tripleo only installs the callback and the callback takes care of sending data to a central server15:19
weshaydmsimard, right.. part of software factory15:19
dmsimardfor example, yes15:19
*** leanderthal has quit IRC15:20
weshaydmsimard, that would cool as well..  I'll take what I can get really15:20
dmsimardweshay: already did a poc of that with sshnaidm|off a while back15:20
weshaybut I do see value in having it working directly w/ TripleO/OSP for operators to see their deployment15:21
weshaybut.. I'll take what I can get15:21
dmsimardyes, but I'm also just one human with other responsabilities :p15:21
sshnaidm|offwould be nice to have interface for external plugins/callbacks15:22
sshnaidm|offoverride-ansible-cfg doesn't fit this apparently15:22
weshaydmsimard I thought you were three people tbh.. but ya.. so we need to have a conversation w/ the delivery team15:22
openstackgerritAlex Schultz proposed openstack/paunch stable/rocky: builder: include environment when running an exec  https://review.opendev.org/69893215:23
*** ayoung has quit IRC15:23
slaweqmarios|ruck: I just commented in https://bugs.launchpad.net/tripleo/+bug/185365215:24
openstackLaunchpad bug 1853652 in tripleo "openstack overcloud node provide --all-manageable timing out and failing periodic rhel-8-ovb-3ctlr_1comp-featureset001-master" [Critical,Triaged]15:24
slaweqit seems for me that it's SELinux issue15:24
*** iurygregory has joined #tripleo15:24
*** ayoung has joined #tripleo15:24
*** bandini has quit IRC15:25
openstackgerritAlex Schultz proposed openstack/paunch stable/stein: builder: allow to pass a dict to 'environment'  https://review.opendev.org/69893315:26
openstackgerritAlex Schultz proposed openstack/paunch stable/stein: builder: include environment when running an exec  https://review.opendev.org/69892915:26
*** rfolco is now known as rfolco|doctor15:27
marios|ruckslaweq: thanks, amheading out a bit early today have been fighting some virus thing since the morning15:27
marios|ruckslaweq: fantastic15:27
marios|ruckselinux it is then chkumar|rover also for fs1 rhel cc weshay15:28
marios|ruckslaweq: can we clear the node then?15:28
slaweqmarios|ruck: yes, You can clear it now15:28
slaweqI think we should be good after selinux issue will be solved15:28
marios|ruckslaweq: so we don't hold ack cool we can reproduce if needed and work out selinux monday15:28
marios|ruckslaweq: \o/15:28
marios|ruckjpena: ^^ no need to hold the node thanks very much15:28
slaweqif not, we can hold node again and check it one more time15:28
marios|ruckjpena: sorry for harrassing you so much today !15:28
jpenaack, I'll remove it15:29
jpenano worries marios|ruck !15:29
slaweqmarios|ruck: jpena thx for the nodes to debug :)15:29
openstackgerritDaniel Bengtsson proposed openstack/tripleo-heat-templates master: Run update without yum update to apply hotfixes.  https://review.opendev.org/69893615:29
marios|ruckslaweq: ack we can do it as we need next week depending on jpena to help us :D15:29
EvilienMsshnaidm|off: are we good with https://review.opendev.org/#/c/698812/ ?15:31
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Deploy /etc/openstack/clouds.yaml with Ansible  https://review.opendev.org/69639015:32
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Deploy /etc/openstack/clouds.yaml with Ansible  https://review.opendev.org/69639015:32
sshnaidm|offEvilienM, I think so, although it's not a full solution..15:32
EvilienMsshnaidm|off: no but we can iterate later15:33
sshnaidm|offEvilienM, agree15:33
*** marios|ruck is now known as marios|ruck|out15:35
*** ksambor has quit IRC15:35
openstackgerritGael Chamoulaud proposed openstack/tripleo-ansible master: Create Validations logging directory  https://review.opendev.org/69625915:39
*** ykarel_ is now known as ykarel|away15:39
*** marios|ruck|out has quit IRC15:44
openstackgerritJose Luis Franco proposed openstack/tripleo-heat-templates master: Force facts cache refreshing before upgrade.  https://review.opendev.org/69889115:45
*** epoojad1 has quit IRC15:45
*** kozhukalov has joined #tripleo15:46
*** ykarel|away has quit IRC15:46
openstackgerritJose Luis Franco proposed openstack/tripleo-heat-templates master: Force facts cache refreshing before upgrade.  https://review.opendev.org/69889115:47
*** jbadiapa has quit IRC15:48
*** surpatil has quit IRC15:49
openstackgerritJose Luis Franco proposed openstack/tripleo-heat-templates stable/stein: Force facts cache refreshing before upgrade.  https://review.opendev.org/69889415:49
*** chkumar|rover is now known as raukadah15:53
*** akahat has quit IRC15:54
*** kozhukalov has quit IRC15:56
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: WIP - Manage Keystone resources with Ansible for Nova API  https://review.opendev.org/69698016:02
*** ykarel|away has joined #tripleo16:04
*** bnemec is now known as beekneemech16:06
*** fmount has quit IRC16:13
*** fmount has joined #tripleo16:13
openstackgerritLee Yarwood proposed openstack/tripleo-heat-templates master: Mount /boot from the host within the nova-compute container  https://review.opendev.org/69888816:14
openstackgerritAlex Schultz proposed openstack/tripleo-heat-templates master: Bump heat_template_version to ussuri  https://review.opendev.org/69894716:17
openstackgerritAlex Schultz proposed openstack/tripleo-heat-templates stable/train: Bump heat_template_version to train  https://review.opendev.org/69894816:19
openstackgerritAlex Schultz proposed openstack/tripleo-heat-templates stable/stein: Bump heat_template_version to stein  https://review.opendev.org/69894916:20
*** rainleander has quit IRC16:22
openstackgerritJose Luis Franco proposed openstack/tripleo-heat-templates stable/stein: Force facts cache refreshing before upgrade.  https://review.opendev.org/69889416:31
openstackgerritDaniel Bengtsson proposed openstack/tripleo-heat-templates master: Run update without yum update to apply hotfixes.  https://review.opendev.org/69893616:33
openstackgerritJose Luis Franco proposed openstack/tripleo-heat-templates master: Force facts cache refreshing before upgrade.  https://review.opendev.org/69889116:34
*** kozhukalov has joined #tripleo16:41
mwhahahaweshay: are we tracking the upgrade job fails on stable/rocky somewhere?16:41
weshaymwhahaha, aye.. http://dashboard-ci.tripleo.org/d/PhzG6KaWk/upgrades-cockpit?orgId=116:43
mwhahahaweshay: i mean the job failures http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-undercloud-upgrades16:44
mwhahahaeg a bug16:44
mwhahahathat job is broken on queens/rocky16:44
weshaymwhahaha, matbu said the patch to fix it was in the gate16:44
* weshay looks16:44
weshaythis morning16:44
*** ykarel|away has quit IRC16:44
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: WIP - Manage Keystone resources with Ansible for Nova API  https://review.opendev.org/69698016:46
weshayhrm.. mwhahaha this one was for train https://review.opendev.org/#/c/698851/16:46
weshaysorry16:46
weshaymwhahaha, /me looks into it16:46
mwhahahaquickstart problem or something16:48
mwhahaha2019-12-13 14:25:00.918220 | primary | The error appears to be in '/home/zuul/workspace/.quickstart/usr/local/share/ansible/roles/container-update/tasks/main.yml': line 9, column 11, but may16:48
mwhahahathis job is not containerized16:48
mwhahahathis shouldn't be running16:48
mwhahaha(the ansible bits)16:48
weshaymwhahaha, ok.. so sounds like we need to immediately mark the rocky version non-voting, non-gating and bug it16:49
mwhahahaand queens16:49
weshayaye16:49
weshayon it16:49
raukadahis there a way to check whether systemd-devel or libsystemd is used by parsing header files?16:52
*** hberaud is now known as hberaud|school-r16:52
*** tesseract has quit IRC16:53
pequbot-testURGENT TRIPLEO TASKS NEEDING ATTENTION: 1116:53
pequbot-testhttps://bugs.launchpad.net/tripleo/+bug/184003616:53
openstackLaunchpad bug 1840036 in tripleo "[check][container-build] RHEL-8 patch pulled, needs to be rebased against kolla patch running in check" [Critical,Triaged] - Assigned to chandan kumar (chkumar246)16:53
pequbot-testhttps://bugs.launchpad.net/tripleo/+bug/184061616:53
pequbot-testhttps://bugs.launchpad.net/tripleo/+bug/184856416:53
openstackLaunchpad bug 1840616 in tripleo "Master/stein check and promotion OVB jobs are randomly giving ansible time out while overcloud deploy" [Critical,Triaged] - Assigned to Ronelle Landy (rlandy)16:53
pequbot-testhttps://bugs.launchpad.net/tripleo/+bug/185301216:53
pequbot-testhttps://bugs.launchpad.net/tripleo/+bug/185397816:53
openstackLaunchpad bug 1848564 in tripleo "rhel8 buildimage oc failing: incorrect deps repo" [Critical,Triaged] - Assigned to Rafael Folco (rafaelfolco)16:53
openstackLaunchpad bug 1853012 in tripleo "Standalone upgrade job is failing - On ovn upgrade" [Critical,In progress] - Assigned to mathieu bultel (mat-bultel)16:53
openstackgerritDamien Ciabrini proposed openstack/tripleo-heat-templates master: podman: force rm --storage and unlink lingering overlayfs  https://review.opendev.org/69896216:53
openstackLaunchpad bug 1853978 in tripleo "periodic train rhel8 ovb overcloud deployment failed with Could not find class ::tripleo::profile::base::neutron::ovn_metadata_agent_wrappers" [Critical,Triaged]16:53
openstackgerritAlex Schultz proposed openstack/tripleo-quickstart-extras master: Ensure undercloud is containerized  https://review.opendev.org/69896316:54
mwhahahaweshay: -^ that's the fix (i think)16:54
openstackgerritJose Luis Franco proposed openstack/tripleo-upgrade master: Move undercloud_os_upgrade.yaml to tasks/common.  https://review.opendev.org/69851616:55
weshayjfrancoa, fyi ^16:55
weshaymwhahaha, ok.. will hold for those results and watch it16:55
jfrancoaweshay: thanks for the heads up, I'll add myself as reviewer16:56
openstackgerritAlex Schultz proposed openstack/tripleo-common stable/rocky: Quote output values for overcloudrc  https://review.opendev.org/69748316:56
mwhahahaweshay: -^ there's a depends-on16:56
*** hjensas has joined #tripleo17:03
*** suuuper has joined #tripleo17:04
openstackgerritMerged openstack/tripleo-heat-templates master: Use include_tasks instead of import_tasks  https://review.opendev.org/69751017:04
openstackgerritMerged openstack/tripleo-heat-templates master: Update environment var for keystone bootstrap  https://review.opendev.org/69833217:04
openstackgerritMerged openstack/tripleo-heat-templates master: Configure additional info during bootstrap  https://review.opendev.org/69771417:04
*** hberaud|school-r is now known as hberaud17:06
*** yprokule has quit IRC17:07
*** hberaud is now known as hberaud|gone17:11
*** hrw has left #tripleo17:15
*** paramite has quit IRC17:20
*** aedc_ has quit IRC17:28
*** openstackgerrit has quit IRC17:29
*** kozhukalov has quit IRC17:35
*** boazel_ has joined #tripleo17:36
*** boazel has quit IRC17:38
*** openstackgerrit has joined #tripleo17:39
openstackgerritMerged openstack/tripleo-heat-templates stable/stein: Add ntpdate to ensure hwclock sync  https://review.opendev.org/69881717:39
*** dtantsur is now known as dtantsur|afk17:40
*** boazel_ has quit IRC17:45
*** mcornea has quit IRC17:49
*** pkopec has quit IRC17:49
openstackgerritMerged openstack/tripleo-common master: Add horizon health check  https://review.opendev.org/69858717:50
*** amoralej is now known as amoralej|off17:51
*** jpena is now known as jpena|off17:53
openstackgerritMerged openstack/tripleo-heat-templates master: Enable horizon healthcheck  https://review.opendev.org/69858817:55
openstackgerritMerged openstack/tripleo-upgrade stable/train: Update: Run all roles update at once  https://review.opendev.org/69752017:55
*** iurygregory has quit IRC17:55
*** melwitt is now known as jgwentworth17:57
raukadahmwhahaha: needs +w on this https://review.opendev.org/#/c/698914/17:59
raukadahmwhahaha: Dhanyabad :-)18:00
raukadahwe may seen green on monday train/master promotion18:01
raukadahTengu: if you around, can we backport https://review.opendev.org/#/c/698914/ to train also?18:01
*** mcornea has joined #tripleo18:04
*** holser has quit IRC18:09
openstackgerritMerged openstack/tripleo-common master: Remove workdir only if it's a temporary one  https://review.opendev.org/69870918:16
*** soniya29 has joined #tripleo18:27
*** gfidente has quit IRC18:40
*** pbandark has quit IRC18:42
openstackgerritAlex Schultz proposed openstack/tripleo-heat-templates master: Bump heat_template_version to ussuri  https://review.opendev.org/69894718:42
*** yolanda has quit IRC18:43
openstackgerritAlex Schultz proposed openstack/tripleo-heat-templates stable/train: Bump heat_template_version to train  https://review.opendev.org/69894818:43
*** aedc has joined #tripleo18:43
openstackgerritAlex Schultz proposed openstack/tripleo-heat-templates stable/stein: Bump heat_template_version to stein  https://review.opendev.org/69894918:43
openstackgerritBrent Eagles proposed openstack/tripleo-quickstart-extras master: WIP: Allow full path to be specified for amphora destination  https://review.opendev.org/69285818:45
*** jfrancoa has quit IRC18:48
*** saneax has quit IRC18:48
openstackgerritMerged openstack/tripleo-heat-templates stable/train: Change default value for NeutronPluginExtensions  https://review.opendev.org/69889918:54
*** tosky has quit IRC18:57
*** morazi has quit IRC19:01
*** tmazur has quit IRC19:02
*** florianf has quit IRC19:08
openstackgerritLuke Short proposed openstack/paunch master: podman: force rm --storage and unlink lingering overlayfs  https://review.opendev.org/69899919:10
*** soniya29 has quit IRC19:11
ekultails^ EmilienM, dciabrin_: Here's the Paunch version of the podman storage workaround.19:12
ekultailsI'm looking into adding that logic to the podman_container Ansible module now, too.19:20
EvilienMcloudnull, mwhahaha : yeah !! https://review.opendev.org/#/c/696390/19:22
EvilienMekultails: excellent, I'll review it19:22
cloudnullEvilienM very nice19:23
EvilienMcloudnull: ovb failing, will look why19:23
openstackgerritMerged openstack/tripleo-ci master: Add scenario007 standalone to run for train branch too  https://review.opendev.org/69587819:23
EvilienMnot related19:24
openstackgerritJames Slagle proposed openstack/tripleo-heat-templates master: Delegate and run once debug start-at-task messages  https://review.opendev.org/69833919:29
openstackgerritJames Slagle proposed openstack/tripleo-heat-templates master: Move some common tasks to step 1  https://review.opendev.org/69834819:29
openstackgerritJames Slagle proposed openstack/tripleo-heat-templates master: Execute deploy_steps_tasks per step  https://review.opendev.org/69861519:29
openstackgerritJames Slagle proposed openstack/tripleo-heat-templates master: Use async tasks for long running common tasks  https://review.opendev.org/69881519:29
*** jtomasek has quit IRC19:31
openstackgerritAlex Schultz proposed openstack/tripleo-heat-templates stable/rocky: Add ntpdate to ensure hwclock sync  https://review.opendev.org/69900319:32
openstackgerritAlex Schultz proposed openstack/tripleo-heat-templates stable/queens: Add ntpdate to ensure hwclock sync  https://review.opendev.org/69900419:35
openstackgerritAlex Schultz proposed openstack/tripleo-common stable/train: Add horizon health check  https://review.opendev.org/69900519:35
openstackgerritAlex Schultz proposed openstack/tripleo-heat-templates stable/train: Enable horizon healthcheck  https://review.opendev.org/69900619:35
*** suuuper has quit IRC19:38
openstackgerritAlex Schultz proposed openstack/tripleo-common stable/train: Remove workdir only if it's a temporary one  https://review.opendev.org/69900719:38
openstackgerritJames Slagle proposed openstack/tripleo-heat-templates master: Use async tasks for long running common tasks  https://review.opendev.org/69881519:43
EvilienMslagle: I like https://review.opendev.org/#/c/698815/ - and the fact that the new tripleo-container-manage role also start containers with async in batch. However I noticed that running more than 2 podman run commands at the same time isn't efficient because of podman locks19:55
EvilienMfor tasks which doesn't involve locks, I +1 the usage of async19:55
openstackgerritAlex Schultz proposed openstack/tripleo-quickstart master: Set containerized_undercloud to false  https://review.opendev.org/69901019:57
openstackgerritAlex Schultz proposed openstack/tripleo-quickstart master: Set containerized_undercloud to false  https://review.opendev.org/69901019:57
EvilienMslagle: if you want to review what we did for the containers managed without paunch: https://github.com/openstack/tripleo-ansible/tree/master/tripleo_ansible/roles/tripleo-container-manage/tasks -- I would appreciate feedback19:57
openstackgerritAlex Schultz proposed openstack/tripleo-common stable/rocky: Quote output values for overcloudrc  https://review.opendev.org/69748319:57
slagleEvilienM: sure, i'll take a look19:59
vkmcEvilienM, o/19:59
EvilienMslagle: ty20:00
EvilienMvkmc: hey, what's up!20:00
vkmcif you have a moment later... I'm trying to debug a container (horizon) that is respawning every 1min20:00
vkmcthere is some issue with kolla_start20:00
EvilienMwhich version?20:00
vkmclatest image build20:00
vkmcI know that we can modify the .json kolla file and restart the container... put a sleep before the main command and try to inspect the state before failing20:01
mwhahahavkmc: check the stdout, i noticed yesterday that in one version i saw some transiation errors20:03
mwhahahavkmc: /var/log/containers/stdout/horizon.log20:03
mwhahahatranslation errors20:03
slagleEvilienM: main thing we need to remember going forward is how expensive looping can be when including roles/tasks20:03
EvilienMslagle: it's expensive when tasks are skipped right?20:04
vkmcmwhahaha, yes, I saw that as well20:05
vkmcI believe that those are being removed somewhere when running kolla_start20:05
slagleEvilienM: not just skipping. there's overhead for every task20:06
slaglepaunch starting containers at step X, is presently one task20:06
EvilienMslagle: the # of task in that role is imo minimal for now20:07
slagleif there's 5 containers to start at step X, and now 5 tasks to run to start 1 container, and 100 nodes, that's...2500 tasks?20:07
slagleas opposed to 100 tasks20:07
EvilienMok so what you suggest is to have the same task running more containers (example with containers)20:07
EvilienMvs batching the data to executing the same tasks with batched data20:08
slagleEvilienM: yes, or use an action plugin as cloudnull did for tripleo-firewall: https://review.opendev.org/#/c/69862020:08
EvilienMrun task ABC with 10 containers vs running 5x2 tasks with 1 container20:08
slagleEvilienM: right20:08
EvilienMgotcha20:08
EvilienMI'm going to look that next20:09
EvilienMnote that I'm consuming podman_container.py ansible module20:09
EvilienMsshnaidm|off: ^ for our roadmap ^20:10
*** kozhukalov has joined #tripleo20:10
sshnaidm|offnot sure I understand the issue..20:15
EvilienMsshnaidm|off: right now we create a container in one task, the idea is to have one task which can craete more containers20:18
mwhahahatl;dr more python, less task yaml20:19
sshnaidm|offok, will look into20:21
sshnaidm|offEvilienM, one task for one step?20:22
EvilienMI'm in mtg brb20:23
*** Goneri has quit IRC20:27
*** rfolco|doctor is now known as rfolco20:31
*** kozhukalov has quit IRC20:31
EvilienMsshnaidm|off: basically if we could do concurrency with the podman_container module20:34
EvilienMlet's say we would call the module to manage 3 containers at a time20:34
EvilienMwe would save 2 ansible tasks because only one tasks would be enough to deploy 3 containers instead of 320:34
EvilienMsshnaidm|off: I'm in a call with kevin, to talk about action plugins20:37
*** kozhukalov has joined #tripleo20:37
EvilienMsshnaidm|off: i don't think we'll have to patch podman_container module, we'll write an action plugin in tripleo-ansible20:40
*** ayoung has quit IRC20:44
*** ayoung has joined #tripleo20:45
pequbot-testURGENT TRIPLEO TASKS NEEDING ATTENTION: 1120:53
pequbot-testhttps://bugs.launchpad.net/tripleo/+bug/184003620:53
openstackLaunchpad bug 1840036 in tripleo "[check][container-build] RHEL-8 patch pulled, needs to be rebased against kolla patch running in check" [Critical,Triaged] - Assigned to chandan kumar (chkumar246)20:53
pequbot-testhttps://bugs.launchpad.net/tripleo/+bug/184061620:53
pequbot-testhttps://bugs.launchpad.net/tripleo/+bug/184856420:53
openstackLaunchpad bug 1840616 in tripleo "Master/stein check and promotion OVB jobs are randomly giving ansible time out while overcloud deploy" [Critical,Triaged] - Assigned to Ronelle Landy (rlandy)20:53
pequbot-testhttps://bugs.launchpad.net/tripleo/+bug/185301220:53
pequbot-testhttps://bugs.launchpad.net/tripleo/+bug/185397820:53
openstackLaunchpad bug 1848564 in tripleo "rhel8 buildimage oc failing: incorrect deps repo" [Critical,Triaged] - Assigned to Rafael Folco (rafaelfolco)20:53
openstackLaunchpad bug 1853012 in tripleo "Standalone upgrade job is failing - On ovn upgrade" [Critical,In progress] - Assigned to mathieu bultel (mat-bultel)20:53
openstackLaunchpad bug 1853978 in tripleo "periodic train rhel8 ovb overcloud deployment failed with Could not find class ::tripleo::profile::base::neutron::ovn_metadata_agent_wrappers" [Critical,Triaged]20:53
mwhahahagood to see we resurrected an annoying bot20:54
EvilienMcan you ban it?20:57
EvilienMI can't20:57
*** EvilienM is now known as EmilienM20:57
openstackgerritJames Slagle proposed openstack/tripleo-heat-templates master: Use async tasks for long running common tasks  https://review.opendev.org/69881520:57
openstackgerritAlex Schultz proposed openstack/tripleo-heat-templates stable/train: Update environment var for keystone bootstrap  https://review.opendev.org/69901720:58
*** TrevorV has quit IRC20:58
*** raildo has quit IRC20:59
openstackgerritAlex Schultz proposed openstack/tripleo-heat-templates stable/stein: Update environment var for keystone bootstrap  https://review.opendev.org/69901921:05
openstackgerritAlex Schultz proposed openstack/tripleo-heat-templates stable/rocky: Update environment var for keystone bootstrap  https://review.opendev.org/69902021:09
*** holser has joined #tripleo21:10
*** rlandy has quit IRC21:10
*** ayoung has quit IRC21:23
sshnaidm|offEmilienM, but action plugin still executes them in serial, doesn't it?21:25
mwhahahait stops having the overhead of the task engine21:27
mwhahahathe loop goes way faster and is run on the ansible executor host. i'm not sure how it's splitup for the inventory tho21:28
mwhahahawe saw a massive speedup on the iptables bits which seem similar21:28
openstackgerritMichael Turek proposed openstack/tripleo-ci master: ppc64le: Add altarch_excludes var for non x86 container builds  https://review.opendev.org/69902421:31
*** holser has quit IRC21:35
*** alexmcleod has quit IRC21:39
sshnaidm|offall this can be solved with mitogen21:49
slagledo we know of any plans for mitogen to be supported by the ansible core community?21:53
sshnaidm|offslagle, what kind of support?21:53
slagleofficially supported as a strategy plugin21:54
slagleso that we (tripleo) don't have to be on the hook to maintain it21:54
*** kozhukalov has quit IRC21:54
sshnaidm|offnot that I'm aware of21:55
sshnaidm|offbut you need just include it21:55
sshnaidm|offlike that https://review.opendev.org/#/c/659909/  https://review.opendev.org/#/c/659907/21:56
*** holser has joined #tripleo22:10
*** slaweq has quit IRC22:14
*** mcornea has quit IRC22:16
openstackgerritAlex Schultz proposed openstack/tripleo-heat-templates stable/stein: Revert "Add ntpdate to ensure hwclock sync"  https://review.opendev.org/69903322:17
*** rfolco has quit IRC22:19
*** holser has quit IRC22:26
*** Goneri has joined #tripleo22:34
*** hjensas has quit IRC22:45
*** pcaruana has quit IRC22:48
*** rfolco has joined #tripleo22:50
*** rfolco has quit IRC22:55
openstackgerritJohn Fulton proposed openstack/tripleo-validations master: Introduce Ceph placement group validation  https://review.opendev.org/69467323:25
*** rfolco has joined #tripleo23:49
openstackgerritMerged openstack/tripleo-heat-templates stable/stein: deploy-steps: use lookup() to get pre|post_deployments_role variables  https://review.opendev.org/69890723:54

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!