Wednesday, 2018-11-21

openstackgerritRedHat RDO CI proposed openstack/tripleo-heat-templates master: GATE CHECK for TripleO  https://review.openstack.org/60429800:00
openstackgerritIan Wienand proposed openstack/diskimage-builder master: simple-init: allow for NetworkManager support  https://review.openstack.org/61912000:07
openstackgerritAlex Schultz proposed openstack/python-tripleoclient master: Add version to python execution  https://review.openstack.org/61912400:09
*** ooolpbot has joined #tripleo00:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION00:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180354700:10
*** ooolpbot has quit IRC00:10
openstackLaunchpad bug 1803547 in tripleo "tempest run failed with 'The specified regex doesn't match with anything ' on tripleo-ci-centos-7-scenario001-multinode-oooq-container" [Critical,In progress] - Assigned to chandan kumar (chkumar246)00:10
*** aedc has joined #tripleo00:11
*** tosky has quit IRC00:12
*** boazel has joined #tripleo00:17
openstackgerritIan Wienand proposed openstack/diskimage-builder master: simple-init: allow for NetworkManager support  https://review.openstack.org/61912000:19
*** boazel_ has quit IRC00:19
*** vinaykns has quit IRC00:21
*** toure is now known as toure|gone00:30
*** gfidente has quit IRC00:34
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: WIP: Rewrite nodepool-setup role to be used with v3 playbooks  https://review.openstack.org/61785500:38
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: WIP: Add a new role to clone repos and include zuul changes  https://review.openstack.org/61799700:39
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: Create reproducer_extra_vars.yml with vars used in run-v3.yaml  https://review.openstack.org/61699300:39
openstackgerritIan Wienand proposed openstack/diskimage-builder master: simple-init: allow for NetworkManager support  https://review.openstack.org/61912000:41
openstackgerritMerged openstack/tripleo-quickstart-extras master: Use a shorter default domain name  https://review.openstack.org/61858800:41
*** hamzy has joined #tripleo00:44
openstackgerritMerged openstack/tripleo-quickstart-extras master: Correct buggy typo  https://review.openstack.org/61894600:52
openstackgerritIan Wienand proposed openstack/diskimage-builder master: simple-init: allow for NetworkManager support  https://review.openstack.org/61912000:57
openstackgerritMerged openstack/tripleo-quickstart-extras master: Fixed logstash file name for tempest  https://review.openstack.org/61791300:59
openstackgerritMerged openstack/tripleo-heat-templates master: Honor --skip-deploy-identifier in common deploy tasks for updates  https://review.openstack.org/61825401:00
*** ooolpbot has joined #tripleo01:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION01:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180354701:10
openstackLaunchpad bug 1803547 in tripleo "tempest run failed with 'The specified regex doesn't match with anything ' on tripleo-ci-centos-7-scenario001-multinode-oooq-container" [Critical,In progress] - Assigned to chandan kumar (chkumar246)01:10
*** ooolpbot has quit IRC01:10
openstackgerritEmilien Macchi proposed openstack/tripleo-common stable/queens: Ansible json error callback plugin  https://review.openstack.org/61912901:34
openstackgerritJohn Fulton proposed openstack/tripleo-common stable/rocky: Update swift_backup workflow success status message  https://review.openstack.org/61913001:35
*** tongl has quit IRC01:40
openstackgerritEmilien Macchi proposed openstack/tripleo-common stable/queens: Workflow and action for deployment failures  https://review.openstack.org/61913301:46
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: Create reproducer_extra_vars.yml with vars used in run-v3.yaml  https://review.openstack.org/61699301:56
*** sambetts_ has quit IRC02:02
*** sambetts_ has joined #tripleo02:06
*** ooolpbot has joined #tripleo02:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION02:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180354702:10
openstackLaunchpad bug 1803547 in tripleo "tempest run failed with 'The specified regex doesn't match with anything ' on tripleo-ci-centos-7-scenario001-multinode-oooq-container" [Critical,In progress] - Assigned to chandan kumar (chkumar246)02:10
*** ooolpbot has quit IRC02:10
*** chem has quit IRC02:13
*** chem has joined #tripleo02:13
openstackgerritJohn Fulton proposed openstack/tripleo-quickstart-extras master: WIP/DNM: Add split_control_plane_{controller,compute} booleans  https://review.openstack.org/61736802:21
openstackgerritJohn Fulton proposed openstack-infra/tripleo-ci master: WIP/DNM: Add split-controlplane standalone non-voting job definition  https://review.openstack.org/61598802:21
*** mrsoul has joined #tripleo02:30
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: WIP: Rewrite nodepool-setup role to be used with v3 playbooks  https://review.openstack.org/61785502:31
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: WIP: Add a new role to clone repos and include zuul changes  https://review.openstack.org/61799702:32
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: Create reproducer_extra_vars.yml with vars used in run-v3.yaml  https://review.openstack.org/61699302:32
*** mschuppert has quit IRC02:33
*** ramishra has quit IRC02:41
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: Create reproducer_extra_vars.yml with vars used in run-v3.yaml  https://review.openstack.org/61699302:52
openstackgerritMerged openstack/tripleo-ha-utils master: Remove trailing white spaces  https://review.openstack.org/61524903:02
openstackgerritTong Liu proposed openstack/tripleo-heat-templates master: Add config files/templates to integrate nsx plugin with container  https://review.openstack.org/60505603:04
*** jaganathan has joined #tripleo03:05
*** ooolpbot has joined #tripleo03:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION03:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180354703:10
openstackLaunchpad bug 1803547 in tripleo "tempest run failed with 'The specified regex doesn't match with anything ' on tripleo-ci-centos-7-scenario001-multinode-oooq-container" [Critical,In progress] - Assigned to chandan kumar (chkumar246)03:10
*** ooolpbot has quit IRC03:10
*** zaneb has quit IRC03:23
*** Chaserjim has joined #tripleo03:27
*** Chaserjim has quit IRC03:41
*** zaneb has joined #tripleo03:42
*** vinaykns has joined #tripleo03:51
*** tzumainn has quit IRC04:03
*** ooolpbot has joined #tripleo04:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION04:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180354704:10
openstackLaunchpad bug 1803547 in tripleo "tempest run failed with 'The specified regex doesn't match with anything ' on tripleo-ci-centos-7-scenario001-multinode-oooq-container" [Critical,In progress] - Assigned to chandan kumar (chkumar246)04:10
*** ooolpbot has quit IRC04:10
*** ykarel|away has joined #tripleo04:15
*** psachin has joined #tripleo04:15
*** ykarel|away is now known as ykarel04:15
*** udesale has joined #tripleo04:17
*** numans has quit IRC04:18
*** saneax has joined #tripleo04:19
*** vinaykns has quit IRC04:23
*** rlandy has quit IRC04:31
*** itlinux has joined #tripleo04:33
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: Create reproducer_extra_vars.yml with vars used in run-v3.yaml  https://review.openstack.org/61699304:40
*** janki has joined #tripleo04:42
*** ratailor has joined #tripleo05:00
*** zbitter has joined #tripleo05:05
openstackgerritMerged openstack/tripleo-heat-templates master: Revert "Revert "Convert *tasks from bootstrap_nodeid to short_bootstrap_node_name""  https://review.openstack.org/61180005:08
openstackgerritMerged openstack/tripleo-heat-templates stable/rocky: Honor --skip-deploy-identifier in common deploy tasks for updates  https://review.openstack.org/61825505:08
openstackgerritMerged openstack/tripleo-validations stable/rocky: Fix the undercloud-process-count validation  https://review.openstack.org/61898605:08
*** zaneb has quit IRC05:08
*** ooolpbot has joined #tripleo05:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION05:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180354705:10
openstackLaunchpad bug 1803547 in tripleo "tempest run failed with 'The specified regex doesn't match with anything ' on tripleo-ci-centos-7-scenario001-multinode-oooq-container" [Critical,In progress] - Assigned to chandan kumar (chkumar246)05:10
*** ooolpbot has quit IRC05:10
openstackgerritIan Wienand proposed openstack/diskimage-builder master: simple-init: allow for NetworkManager support  https://review.openstack.org/61912005:21
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: Create reproducer_extra_vars.yml with vars used in run-v3.yaml  https://review.openstack.org/61699305:28
*** chandankumar has joined #tripleo05:36
*** chandankumar is now known as chkumar|ruck05:37
*** Chaserjim has joined #tripleo05:38
*** Chaserjim has quit IRC05:43
*** sambetts_ has quit IRC05:44
*** sambetts_ has joined #tripleo05:44
*** ade_lee has quit IRC05:48
*** d0ugal has quit IRC06:05
*** ooolpbot has joined #tripleo06:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION06:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180354706:10
openstackLaunchpad bug 1803547 in tripleo "tempest run failed with 'The specified regex doesn't match with anything ' on tripleo-ci-centos-7-scenario001-multinode-oooq-container" [Critical,In progress] - Assigned to chandan kumar (chkumar246)06:10
*** ooolpbot has quit IRC06:10
chkumar|ruckykarel, Hello06:11
chkumar|ruckykarel, I am trying fs035 reproducer on rdocloud if it reproduces tempestconfig issue we will good to debug06:11
ykarelchkumar|ruck, okk, so facing some issue?06:14
chkumar|ruckykarel, just kicked the reproducer06:15
ykarelchkumar|ruck, ack06:17
*** numans has joined #tripleo06:18
*** noama has joined #tripleo06:22
noamagood morning06:23
noamais there some known issue with "chronyd" on master06:23
noama?06:23
noamai get "Unable to restart service chronyd" when deploying overcloud06:23
Tenguhello there06:31
Tengunoama: hey, yes, there's something about that, I saw a couple of issues - there's something on launchpad I think.06:31
noamanothing pops when i search chronyd06:31
Tenguhmmm. maybe its on redhat BZ then. gimme a few minutes to boot up my brain :)06:33
noamalol06:33
noamasorry,06:33
noamacoffee first06:33
Tenguweird... can't find back in my logs. although it was discussed yesterday, that's sure.06:38
Tenguanyway, the issue discussed yesterday was a string in a wrong format in the chrony configuration.06:38
Tengunoama: as a workaround, you can push this in a local registry: OS::TripleO::Services::Timesync: OS::Heat::None06:40
noamaoh06:40
noamado you know if there is a patch ?06:40
*** apetrich has joined #tripleo06:40
Tengunoama: I think mwhahaha is working on it, but he's in the US, so deeply sleeping right now. He's the guy able to answer you, he's the one that added the chrony thingy :)06:41
Tengunoama: he should pop up in about 7-8 hours from now.06:42
noamaoh, but meanwhile i can revert to ntp?06:42
Tenguor you can disable timesync. I do hope you're not deploying prod on master, are you?06:43
noamano, its for testing:)06:43
Tenguand in order to either revert to ntp, or just disable timesync, the registry entry I pasted earlier will do the job :)06:44
noamathx !06:44
Tenguif you put "OS::Heat::None" as value, it disable - if you put <I don't remember what> instead it switches to ntp.06:45
Tenguykarel: hello! would you be available for a small chat about podman integration to fs021 and the like? :)06:46
ykarelTengu, sure06:47
Tenguykarel: here on BJ?06:47
Tengu*or06:47
ykarelBJ will also work06:47
Tenguthat would avoid my fingers messing :D06:48
ykarel:)06:48
hjensasnoama: here's the bug regarding chronyd https://bugs.launchpad.net/tripleo/+bug/1801162 - The fix merged 2 weeks ago, but I guess we did'nt promote to packages/containers yet.07:03
openstackLaunchpad bug 1801162 in tripleo "output of group vars that are not string/ints may include python encoding" [High,Fix released] - Assigned to Alex Schultz (alex-schultz)07:03
*** rfolco has quit IRC07:06
*** ooolpbot has joined #tripleo07:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION07:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180354707:10
*** ooolpbot has quit IRC07:10
openstackLaunchpad bug 1803547 in tripleo "tempest run failed with 'The specified regex doesn't match with anything ' on tripleo-ci-centos-7-scenario001-multinode-oooq-container" [Critical,In progress] - Assigned to chandan kumar (chkumar246)07:10
*** jfrancoa has joined #tripleo07:11
*** ahosam has joined #tripleo07:14
openstackgerritCédric Jeanneret proposed openstack/tripleo-quickstart-extras master: Correct some commands from the "Collect container info" tasks  https://review.openstack.org/61897507:15
openstackgerritCédric Jeanneret proposed openstack/tripleo-quickstart-extras master: Allow to collect HAProxy stats and log them in a file  https://review.openstack.org/61852907:15
noamahjensas when will it be promoted? i need work containerize only07:16
hjensasnoama: not sure, the reason it did'nt promote is most likely some CI issue in the promote jobs. It will promote once those issues are resolved.07:18
Tengu24d late for master promotion07:19
hjensasnoama: for now, use the workaround Tengu mentioned. Override to use the ntp service instead. resource_registry override -  OS::TripleO::Services::Timesync: OS::TripleO::Services::Ntp07:21
*** skramaja has joined #tripleo07:23
*** gkadam has joined #tripleo07:25
*** yprokule has joined #tripleo07:26
*** rcernin has quit IRC07:26
*** quiquell|off is now known as quiquell07:28
*** agurenko has joined #tripleo07:31
*** ksambor has joined #tripleo07:32
*** mschuppert has joined #tripleo07:33
*** agurenko has quit IRC07:33
openstackgerritQuique Llorente proposed openstack-infra/tripleo-ci master: Add standalone scenario002 job  https://review.openstack.org/61870907:34
openstackgerritQuique Llorente proposed openstack/tripleo-heat-templates master: Add env file for standalone scenario002  https://review.openstack.org/61853707:34
*** Chaserjim has joined #tripleo07:39
*** Chaserjim has quit IRC07:43
*** saneax is now known as saneax|lunch07:44
*** ykarel is now known as ykarel|lunch07:45
openstackgerritQuique Llorente proposed openstack-infra/tripleo-ci master: Add standalone scenario002 job  https://review.openstack.org/61870907:49
Tengucan anyone have a look at this one and, if zuul is happy, make it merge? https://review.openstack.org/#/c/618529/   thank you :)07:50
Tenguquiquell: hey, you're core now right? congrats :)07:50
openstackgerritQuique Llorente proposed openstack-infra/tripleo-ci master: Add standalone scenario002 job  https://review.openstack.org/61870907:50
quiquellTengu: nah not yet07:50
Tenguquiquell: oh. well. should be soon ;).07:51
*** numans has quit IRC07:52
*** quiquell is now known as quiquell|brb07:53
*** ccamacho has joined #tripleo07:59
openstackgerritJose Luis Franco proposed openstack/python-tripleoclient stable/rocky: Add docker bridge ip option to the undercloud.conf  https://review.openstack.org/61916707:59
*** jtomasek has joined #tripleo07:59
openstackgerritRedHat RDO CI proposed openstack/tripleo-quickstart-extras master: GATE CHECK for quickstart-extras  https://review.openstack.org/56044508:00
openstackgerritRedHat RDO CI proposed openstack/tripleo-heat-templates master: GATE CHECK for TripleO  https://review.openstack.org/60429808:00
openstackgerritRedHat RDO CI proposed openstack/tripleo-heat-templates stable/rocky: GATE CHECK for TripleO  https://review.openstack.org/60429308:00
openstackgerritzhufl proposed openstack/diskimage-builder master: Add missing ws separator between words  https://review.openstack.org/61916908:02
*** cylopez has joined #tripleo08:03
*** artom has quit IRC08:08
*** ooolpbot has joined #tripleo08:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION08:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180354708:10
openstackLaunchpad bug 1803547 in tripleo "tempest run failed with 'The specified regex doesn't match with anything ' on tripleo-ci-centos-7-scenario001-multinode-oooq-container" [Critical,In progress] - Assigned to chandan kumar (chkumar246)08:10
*** ooolpbot has quit IRC08:10
*** cylopez has quit IRC08:11
openstackgerritCarlos Camacho proposed openstack/instack-undercloud stable/rocky: Get the stack name if possible before running the validations group  https://review.openstack.org/61911208:12
*** artom has joined #tripleo08:12
*** artom has quit IRC08:13
*** artom has joined #tripleo08:13
*** quiquell|brb is now known as quiquell08:15
*** numans has joined #tripleo08:26
openstackgerritYurii Prokulevych proposed openstack/tripleo-upgrade master: Run online data migration before update/upgrade converge.  https://review.openstack.org/61896108:28
*** cylopez has joined #tripleo08:29
*** shardy has joined #tripleo08:35
*** sri_ has quit IRC08:35
*** shardy has quit IRC08:39
*** dtantsur|afk is now known as dtantsur08:42
openstackgerritQuique Llorente proposed openstack/tripleo-quickstart-extras master: Install correct python version packages at tempest  https://review.openstack.org/61872108:49
*** maufart__ has joined #tripleo08:50
openstackgerritFlorian Fuchs proposed openstack/tripleo-validations master: Fix containerized services task  https://review.openstack.org/61917608:51
openstackgerritThomas Herve proposed openstack/tripleo-quickstart-extras master: Use a custom roles file for undercloud deploy  https://review.openstack.org/61917708:51
*** ykarel|lunch is now known as ykarel08:52
*** jpena|off is now known as jpena08:52
*** holser_ has joined #tripleo08:53
*** hakhande_ has quit IRC08:56
*** hakhande_ has joined #tripleo08:57
chkumar|ruckDo we have any fs where we are testing ipv6?08:57
*** tosky has joined #tripleo08:57
*** saneax|lunch is now known as saneax08:57
*** sri_ has joined #tripleo08:58
*** hakhande_ has quit IRC08:59
marioschkumar|ruck: grepr ipv6.*true config/general_config/* config/general_config/featureset024.yml config/general_config/featureset035.yml08:59
*** hakhande_ has joined #tripleo08:59
*** maufart__ has quit IRC09:01
chkumar|ruckmarios, thanks :-)09:01
marioschkumar|ruck: np :)09:02
*** ccamacho has quit IRC09:02
*** yolanda__ has joined #tripleo09:03
*** ccamacho has joined #tripleo09:04
*** yolanda__ is now known as yolanda09:05
*** rdopiera has joined #tripleo09:07
*** rfolco has joined #tripleo09:09
*** ooolpbot has joined #tripleo09:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION09:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180354709:10
openstackLaunchpad bug 1803547 in tripleo "tempest run failed with 'The specified regex doesn't match with anything ' on tripleo-ci-centos-7-scenario001-multinode-oooq-container" [Critical,In progress] - Assigned to chandan kumar (chkumar246)09:10
*** ooolpbot has quit IRC09:10
*** d0ugal has joined #tripleo09:11
*** ratailor has quit IRC09:12
*** ratailor has joined #tripleo09:13
*** rdopiera has quit IRC09:14
*** chkumar|ruck has quit IRC09:18
*** chkumar|ruck has joined #tripleo09:18
openstackgerritMerged openstack/ansible-role-tripleo-modify-image master: Set WORKDIR / in the dockerfiles  https://review.openstack.org/61744209:20
*** d0ugal has quit IRC09:22
openstackgerritMarios Andreou proposed openstack-infra/tripleo-ci master: Add new tripleo-ci-centos-7-scenario001-standalone job  https://review.openstack.org/61687209:23
*** jaosorior has quit IRC09:26
*** d0ugal has joined #tripleo09:26
openstackgerritFlorian Fuchs proposed openstack/tripleo-validations master: Skip schema check in network validation  https://review.openstack.org/61918409:28
*** bogdando has joined #tripleo09:28
*** derekh has joined #tripleo09:30
*** hakhande_ has quit IRC09:35
*** hakhande_ has joined #tripleo09:35
*** suuuper has joined #tripleo09:36
*** ahosam has quit IRC09:36
*** ahosam has joined #tripleo09:36
*** ahosam has quit IRC09:40
*** cylopez has quit IRC09:44
openstackgerritMartin André proposed openstack/tripleo-heat-templates master: Configure registry to use gluster  https://review.openstack.org/61895509:47
openstackgerritHarald Jensås proposed openstack/tripleo-heat-templates master: Add MTU to neutron networks and nic-config templates  https://review.openstack.org/59959309:47
*** pcaruana has joined #tripleo09:48
chkumar|ruckTengu, Hello09:51
chkumar|ruckTengu, http://logs.openstack.org/86/617686/1/check/tripleo-ci-centos-7-containers-multinode/29b7317/logs/undercloud/home/zuul/undercloud_install.log.txt.gz#_2018-11-21_01_23_1909:51
*** shardy has joined #tripleo09:51
chkumar|ruckTengu, container create failed: container_linux.go:336: starting container process caused \"process_linux.go:293: applying cgroup configuration for process caused \\\"open /sys/fs/cgroup/cpuset/machine.slice/cpuset.cpus: no such file or directory\\\"\"",09:51
chkumar|ruck2018-11-21 01:23:19 |     ": internal libpod error",09:51
chkumar|ruckTengu, Failed running docker-puppet.py for mysql"09:51
Tenguchkumar|ruck: I think I saw something about that.09:52
openstackgerritKartikeya Jain proposed openstack/diskimage-builder master: Adding support for SLES 15 in element 'sles'  https://review.openstack.org/61918609:53
Tenguchkumar|ruck: but that's a random failure, again.09:55
Tenguit hits. or not. but maybe.09:55
Tenguyay.09:55
Tenguwe should use podman as a random number generator......09:55
chkumar|ruckTengu, https://review.openstack.org/#/c/617686/ it got broke on this one09:56
Tenguchkumar|ruck: yeah, so apparently panda|rover found this on 2018-04-2709:56
Tengurandomness is random.09:57
Tenguseems to hit every now and then. will check how sever this is in the logstash.09:58
chkumar|ruckTengu, sure09:58
Tenguok. 1 time for the last 48h.09:59
*** kopecmartin|off is now known as kopecmartin10:00
Tenguhmmm. apparently 6 times on the last 7 days....10:00
chkumar|ruckTengu, it is serious then10:00
Tenguthat said, I'm pretty sure EmilienM talked about that.10:00
Tenguerr..10:00
TenguI'd not mark that as "serious" though.10:00
chkumar|ruckok10:00
evrardjpEmilienM: hey -- I checked a part of your email about tripleo feedback. I was thinking of something: If you were to use ansible-config_template action plugin, we could get somewhere... config_template at some point uses copy module to transfer the file to destination. And copy module accepts "validate" where you can pass your own arguments for validation (could be upgrade checks, or other checks). Just10:00
evrardjpsaying ;)10:00
Tenguwe're talking of 6 times over I don't know how many podman deploys ;).10:01
*** sshnaidm|afk is now known as sshnaidm10:02
Tenguchkumar|ruck: https://bugs.launchpad.net/tripleo/+bug/180393710:02
openstackLaunchpad bug 1803937 in tripleo "internal libpod error open /sys/fs/cgroup/cpuset/machine.slice/cpuset.cpus: no such file or directory " [Undecided,New]10:02
evrardjpbut yeah that would also mean a complete mindshift10:02
*** jaosorior has joined #tripleo10:03
Tenguchkumar|ruck: will add a comment in the LP pointing the frequency.10:03
chkumar|ruckTengu, sure10:03
*** ahosam has joined #tripleo10:03
Tenguchkumar|ruck: updated - will talk to Rabi when he connects.10:05
*** krypto has joined #tripleo10:05
kryptohi all in tripleO,will adding new compute nodes to cluster cause any service restart on openstack controller10:06
*** ooolpbot has joined #tripleo10:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION10:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180354710:10
*** ooolpbot has quit IRC10:10
openstackLaunchpad bug 1803547 in tripleo "tempest run failed with 'The specified regex doesn't match with anything ' on tripleo-ci-centos-7-scenario001-multinode-oooq-container" [Critical,In progress] - Assigned to chandan kumar (chkumar246)10:10
*** salmankhan has joined #tripleo10:14
*** akrivoka has joined #tripleo10:15
*** salmankhan1 has joined #tripleo10:16
*** salmankhan has quit IRC10:18
*** salmankhan1 is now known as salmankhan10:18
jaosoriorkrypto: it shouldn't10:18
openstackgerritQuique Llorente proposed openstack-infra/tripleo-ci master: Merge featureset custom env files and override  https://review.openstack.org/61901010:18
openstackgerritQuique Llorente proposed openstack-infra/tripleo-ci master: Add standalone scenario002 job  https://review.openstack.org/61870910:21
openstackgerritKamil Sambor proposed openstack/tripleo-quickstart master: Remove openvswitch tasks  https://review.openstack.org/61919610:22
openstackgerritDerek Higgins proposed openstack/tripleo-heat-templates master: Don't set boot_option=local on baremetal flavors  https://review.openstack.org/61872710:23
openstackgerritNatal Ngétal proposed openstack/tripleo-quickstart master: [Configuration] Remove mitaka file.  https://review.openstack.org/61767010:24
openstackgerritKarthik S proposed openstack/os-net-config master: WIP: Add max_tx_rate, min_tx_rate setting for sriov_vf  https://review.openstack.org/61919910:29
*** dsneddon has quit IRC10:30
openstackgerritQuique Llorente proposed openstack-infra/tripleo-ci master: Merge featureset custom env files and override  https://review.openstack.org/61901010:31
openstackgerritQuique Llorente proposed openstack-infra/tripleo-ci master: Add standalone scenario002 job  https://review.openstack.org/61870910:31
*** ahosam has quit IRC10:43
*** maufart__ has joined #tripleo10:45
*** ahosam has joined #tripleo10:51
*** shardy has quit IRC10:52
*** shardy has joined #tripleo10:53
shardymarios: Hey o/10:53
shardymarios: I see the Revert-Revert ref 611800 finally landed, wanted to check you're OK with the result before I rebase the remainder of the series10:54
shardyAFAICS the upgrade jobs look green in cistatus but wanted to confirm :)10:54
*** ahosam has quit IRC10:55
chkumar|ruckjaosorior, Hello10:59
chkumar|ruckjaosorior, I am seeing this issue The task includes an option with an undefined variable. The error was: 'haproxy_public_tls_inject_short_bootstrap_node_name' is undefined\10:59
chkumar|ruckjaosorior, https://logs.rdoproject.org/48/17348/5/check/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master/24f7424/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-11-21_10_37_3710:59
chkumar|ruckon overcloud deploy10:59
jaosoriorshort_bootstrap_node_name....11:02
jaosoriorwhere is that coming from?11:02
jaosoriorshardy: weren't there changes recently to the bootstrap nodes code? ^^11:03
chkumar|ruckTengu, https://logs.rdoproject.org/48/17348/5/check/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master/24f7424/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-11-21_10_37_3411:04
shardyjaosorior: yeah, sounds like we need to revert revert revert11:04
shardyargh!11:04
shardyI'd like to understand why the job is green before merging though11:04
shardythere must be a fault in the job11:04
chkumar|ruckTengu, after creating persistant directory [create persistent logs directory] and it failed at glance logs readm11:05
jaosoriorshardy: probably when it merged at the time, ovb was broken.11:05
chkumar|ruckTengu, please have a look11:05
shardychkumar|ruck: sec let me look, I think it's related to https://review.openstack.org/#/c/611800/11:05
shardyjaosorior: yeah but I'm still not clear why it's failing, perhaps we're using a very old tripleo-common or something11:06
* shardy looks at logs11:06
chkumar|ruckjaosorior, I was debugging tempest fs025 jobs then I applied the patch in rdo ci, there it is coming11:07
chkumar|ruck*fs03511:07
chkumar|ruckshardy, let me try with depends on my job11:08
chkumar|rucksorry let me check if patch is there in my deployment11:08
*** ooolpbot has joined #tripleo11:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION11:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180354711:10
*** ooolpbot has quit IRC11:10
openstackLaunchpad bug 1803547 in tripleo "tempest run failed with 'The specified regex doesn't match with anything ' on tripleo-ci-centos-7-scenario001-multinode-oooq-container" [Critical,In progress] - Assigned to chandan kumar (chkumar246)11:10
hjensasI'm struggling with a THT issue. Error in ControllerServiceChain output role_data: Cannot merge <type 'unicode'> with <type 'tuple'>11:10
hjensasThe change is in puppet/services/apache.j2.yaml, I'm passing list as parameter to apache::mod::remoteip::proxy_ips: .. see http://paste.openstack.org/show/735883/11:11
hjensasshardy: therve: can you take a look? ^^11:12
chkumar|ruckshardy, https://github.com/openstack/tripleo-heat-templates/commit/07241f33d105d8fb0321fa70e4526fbe192b5c1e is available in my deployment and rpm used openstack-tripleo-heat-templates-10.0.1-0.20181121051529.07241f3.el7.noarch11:13
chkumar|ruckshardy, do we need to revert that patch?11:13
shardychkumar|ruck: we've already reverted it - let me check that tripleo-common looks wrong11:14
shardysec11:14
Tenguchkumar|ruck: no. the error is not the missing directory.11:16
derekhCan anybody with some time take a look at reviewing the ironic ci job, https://review.openstack.org/#/c/509728/ and https://review.openstack.org/#/c/582294/11:17
Tenguchkumar|ruck: it's "ignored" by the playbook. you want to go a bit bellow, and you will find the cause in TASK [get DeployedSSLCertificatePath attributes] *******************************11:17
Tenguchkumar|ruck: precisely: "The error was: 'haproxy_public_tls_inject_short_bootstrap_node_name' is undefined"11:18
chkumar|ruckTengu, shardy is looking into that ^^11:18
Tenguyeah, reading.11:19
shardychkumar|ruck: do you have a bug for this yet?  I can see why it's failing but still finding the root cause11:19
shardywe're missing haproxy_public_tls_inject_short_bootstrap_node_name in https://logs.rdoproject.org/48/17348/5/check/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master/24f7424/logs/undercloud/var/lib/mistral/overcloud/group_vars/overcloud.yaml.txt.gz11:19
shardywhich should be provided due to https://review.openstack.org/#/c/605046/11:19
shardywe may have a service name mismatch, checking now11:20
mariosshardy: o/ sorry in a call panda|rover can you please check https://tree.taiga.io/project/tripleo-ci-board/issue/219 if you get a chance (I'll have a look when i finish here)11:20
chkumar|ruckshardy, nope11:20
chkumar|ruckshardy, seen right now11:20
chkumar|ruckshardy, filing  abug11:20
openstackgerritCarlos Camacho proposed openstack/instack-undercloud stable/rocky: Get the stack name if possible before running the validations group  https://review.openstack.org/61911211:20
mariosshardy: panda|rover the revert revert finally merged ... we should check if the job is ok and make it voting again if it is11:20
*** sambetts_ is now known as sambetts|afk11:21
Tenguwe're getting time outs in zuul.11:21
chkumar|ruckmarios, the revert revert failed here https://logs.rdoproject.org/48/17348/5/check/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master/24f7424/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-11-21_10_37_3711:21
Tengujust sayin'11:21
shardymarios: chkumar|ruck has an issue which may be related, I'm investigating now11:21
shardyit's not the undercloud_upgrades job this time tho, tls related11:22
chkumar|ruckshardy, https://bugs.launchpad.net/tripleo/+bug/180443311:28
openstackLaunchpad bug 1804433 in tripleo "[fs035][master] multinode periodic job failed at overcloud deploy giving haproxy_public_tls_inject_short_bootstrap_node_name is undefined" [Critical,Triaged]11:28
*** udesale has quit IRC11:28
shardychkumar|ruck: thanks, I can see a possible workaround which is to use haproxy_short_bootstrap_node_name but still looking for the root-cause11:29
chkumar|ruckshardy, sure11:30
shardychkumar|ruck: can we test a fix for this in the check queue?11:30
chkumar|ruckshardy, yes11:30
chkumar|ruckshardy, pass me the fix11:30
shardychkumar|ruck: Ok sec I'll push a WIP patch we can test while I look into the root-cause11:30
chkumar|ruckshardy, https://review.rdoproject.org/r/#/c/17348/ edit this patch and added a depends on11:31
chkumar|ruckit will trigger fs035 job11:31
*** cylopez has joined #tripleo11:32
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates master: WIP adjust haproxy-public-tls-inject.yaml bootstrap variable  https://review.openstack.org/61921511:33
shardychkumar|ruck: ^^ that's a possible workaround, I'll ping you when I've figured out the root-cause11:33
chkumar|ruckshardy, let me add this patch to depends on11:36
*** xek has quit IRC11:39
shardychkumar|ruck, jaosorior: OK I see the root-cause, it's because OS::TripleO::Services::HAProxyPublicTLS is enabled via ./puppet/services/haproxy.yaml11:40
shardywhich is an anti-pattern in terms of composable services11:40
shardybut if we want to keep it that way, my fix is correct, e.g we have to use the haproxy_short_bootstrap_node_name11:40
*** holser_ has quit IRC11:40
*** Chaserjim has joined #tripleo11:40
shardyI'll update the bug with details, lets see what CI says about the WIP fix11:41
jaosoriorshardy: I like your fix better. In reality, that public TLS bit is not a service, but an option within the haproxy service.11:41
jaosoriorthanks for digging into it!11:41
shardyjaosorior: yeah it's kind of a corner-case11:41
shardyit's my fault for sed-replacing based on the service_name ;)11:42
openstackgerritQuique Llorente proposed openstack-infra/tripleo-ci master: Add standalone scenario002 job  https://review.openstack.org/61870911:42
openstackgerritCarlos Camacho proposed openstack/instack-undercloud stable/rocky: Get the stack name if possible before running the validations group  https://review.openstack.org/61911211:43
*** Chaserjim has quit IRC11:45
chkumar|ruckfyi! Queue has started growing , we need to purge the gates11:45
jaosoriorchkumar|ruck: do we have any insight on why it's growing?11:46
ykareljaosorior, shardy, any idea where tripleo-common's scripts/container-update.py is used, and how it's different from container-check11:47
chkumar|ruckjaosorior, tripleo-ci-centos-7-containers-multinode is timing out and failing a lot11:47
ykareli mean https://github.com/imain/container-check/blob/master/container-check#L6311:47
chkumar|ruckI donot have much insight11:48
jaosoriorchkumar|ruck: is anybody checking that? Do we have a bug?11:48
chkumar|ruckjaosorior, https://bugs.launchpad.net/tripleo/+bug/180393711:48
openstackLaunchpad bug 1803937 in tripleo "internal libpod error open /sys/fs/cgroup/cpuset/machine.slice/cpuset.cpus: no such file or directory " [Undecided,New]11:48
chkumar|ruckjaosorior, I am still looking in the failed one11:49
jaosoriorok, lets add an alert for that bug11:49
*** janki has quit IRC11:50
*** mugsie has joined #tripleo11:50
chkumar|ruckjaosorior, I am going to into that11:51
jaosoriorthanks chkumar|ruck11:51
chkumar|ruckjaosorior, sorry done11:51
chkumar|ruckjaosorior, Do we want to update the channel message donot workflow something like that?11:52
*** nawar has joined #tripleo11:52
nawarhi11:52
jaosoriorchkumar|ruck: we do, but I don't have the permission to do so at the moment.11:52
nawari'm trying network isolation but i'm getting NotSupported: resources.VipPort: Update to properties network of VipPort (OS::TripleO::Network::Ports::ControlPlaneVipPort) is not supported11:53
dtantsurfolks, a trivial backport if someone has time https://review.openstack.org/#/c/618979/11:54
dtantsurjaosorior: hi! could you check https://review.openstack.org/#/c/604761/ when you have a moment? I haven't updated it with our novajoin discussion, everything else should be okay (IIRC, it has been a while)11:55
jaosoriordtantsur: will do11:56
*** janki has joined #tripleo11:56
*** holser|lunch has joined #tripleo11:58
*** raildo has joined #tripleo11:59
*** psachin has quit IRC12:01
shardynawar: are you trying to update an existing deployment to enable network-isolation?12:01
nawaryes12:02
shardynawar: that is not supported, you have to choose it from the initial deploy12:02
nawarI see12:02
nawarok I will clean the nodes and try it again12:03
nawarthanks12:03
shardynawar: np!12:03
*** xek has joined #tripleo12:04
nawaranother thing every time I delete a stack I got Error occurred during stack delete None, but the delete process continue as normal12:06
*** dprince has joined #tripleo12:07
*** ooolpbot has joined #tripleo12:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION12:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180354712:10
openstackLaunchpad bug 1803547 in tripleo "tempest run failed with 'The specified regex doesn't match with anything ' on tripleo-ci-centos-7-scenario001-multinode-oooq-container" [Critical,In progress] - Assigned to chandan kumar (chkumar246)12:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180393712:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180443312:10
*** ooolpbot has quit IRC12:10
openstackLaunchpad bug 1803937 in tripleo "internal libpod error open /sys/fs/cgroup/cpuset/machine.slice/cpuset.cpus: no such file or directory " [Critical,Triaged]12:10
openstackLaunchpad bug 1804433 in tripleo "[fs035][master] multinode periodic job failed at overcloud deploy giving haproxy_public_tls_inject_short_bootstrap_node_name is undefined" [Critical,In progress] - Assigned to Steven Hardy (shardy)12:10
chkumar|ruckEmilienM, mwhahaha can we force merge these two patches https://review.openstack.org/#/c/618994/ and https://review.openstack.org/#/c/618674/ in tempest fixes12:11
*** janki has quit IRC12:12
jaosoriorchkumar|ruck: ok, gate clear12:14
*** sshnaidm is now known as sshnaidm|afk12:15
openstackgerritRafael Folco proposed openstack/tripleo-heat-templates master: Add a new standalone scenario003 env file  https://review.openstack.org/61876012:19
openstackgerritRafael Folco proposed openstack-infra/tripleo-ci master: Add new tripleo-ci-centos-7-scenario003-standalone job  https://review.openstack.org/61876112:20
*** kopecmartin has quit IRC12:23
*** kopecmartin has joined #tripleo12:25
Tenguchkumar|ruck: re: podman failure: remote issue created: https://github.com/containers/libpod/issues/1841  - I'll try to add the "podman run" command in extenso.12:28
openstackgerritJohn Fulton proposed openstack/tripleo-quickstart-extras master: WIP/DNM: Add split_control_plane_{controller,compute} booleans  https://review.openstack.org/61736812:29
openstackgerritJohn Fulton proposed openstack-infra/tripleo-ci master: WIP/DNM: Add split-controlplane standalone non-voting job definition  https://review.openstack.org/61598812:29
EmilienMhi12:30
EmilienMTengu: how often do we hit https://github.com/containers/libpod/issues/1841 in the gate?12:30
TenguEmilienM: about 6 times on the last 7 days. Not that much.... but apparently, enough to mark it as "critical", although I'm not that sure about that level.12:32
TenguEmilienM: https://bugs.launchpad.net/tripleo/+bug/180393712:33
openstackLaunchpad bug 1803937 in tripleo "internal libpod error open /sys/fs/cgroup/cpuset/machine.slice/cpuset.cpus: no such file or directory " [Critical,Triaged]12:33
EmilienMwow ok I thought it was really bad12:33
Tengusame reaction :)12:33
EmilienMit's not minor, we need to fix it, but it's not major.12:33
*** krypto has quit IRC12:33
Tenguyep12:33
EmilienMchkumar|ruck: please remove the alert at this point. No need of that unless it becomes critical12:33
Tenguwhat I said in my first comment on the LP ;)12:34
openstackgerritJohn Fulton proposed openstack/tripleo-quickstart-extras master: WIP/DNM: Add split_control_plane_{controller,compute} booleans  https://review.openstack.org/61736812:34
EmilienMTengu: which LP12:34
TenguEmilienM: linked just before: https://bugs.launchpad.net/tripleo/+bug/180393712:34
openstackLaunchpad bug 1803937 in tripleo "internal libpod error open /sys/fs/cgroup/cpuset/machine.slice/cpuset.cpus: no such file or directory " [Critical,Triaged]12:34
EmilienMTengu: ah there is also https://bugs.launchpad.net/tripleo/+bug/180443412:34
openstackLaunchpad bug 1804434 in tripleo "podman fails to find image: error getting image" [High,New] - Assigned to Gabriele Cerami (gcerami)12:34
EmilienMwhich is the one I just saw12:34
Tenguah, that's another story for that one.12:35
EmilienMTengu: this is not the real problem12:38
EmilienMthe message is not why the job failed12:38
chkumar|ruckEmilienM, done12:39
thervehjensas: Looking12:39
*** jpena is now known as jpena|lunch12:39
EmilienMchkumar|ruck: I'm investigating https://bugs.launchpad.net/tripleo/+bug/180443412:39
openstackLaunchpad bug 1804434 in tripleo "podman fails to find image: error getting image" [High,New] - Assigned to Gabriele Cerami (gcerami)12:39
EmilienMTengu: ^ see my comment fyi12:39
jaosoriorEmilienM: could you update the status? seems CI is hosed :(12:40
EmilienMpanda|rover: I'm also removing the alert until I'm sure it's actually hitting a lot12:40
*** EmilienM sets mode: +o jaosorior12:40
EmilienMjaosorior: goa head12:40
chkumar|ruckEmilienM, jaosorior can we add a msg like "If hitted timedout on voting job, ping us on #tripleo channel on multiple recheck12:41
*** jaosorior changes topic to "CI Status: Red, sad sad red - do not recheck nor approve patches. If you hit a timeout, let us know | https://docs.openstack.org/tripleo-docs/latest/"12:42
TenguEmilienM: ah, good catch, I didn't see that one.12:42
chkumar|ruckjaosorior, thanks :-)12:42
*** thrash|g0ne is now known as thrash12:44
chkumar|ruckshardy, looks like fs01 also got affected12:45
*** dtantsur is now known as dtantsur|brb12:47
openstackgerritRafael Folco proposed openstack-infra/tripleo-ci master: Remove legacy multinode-networking playbook  https://review.openstack.org/61922812:47
openstackgerritQuique Llorente proposed openstack/tripleo-heat-templates master: Add env file for standalone scenario002  https://review.openstack.org/61853712:50
openstackgerritQuique Llorente proposed openstack/tripleo-heat-templates master: Add 'podman' as option at DockerCli  https://review.openstack.org/61923012:50
thervehjensas: You'd like it to be a list instead of a tuple?12:50
*** ramishra has joined #tripleo12:51
EmilienMjaosorior: we need to have https://review.openstack.org/#/c/619099/ ASAP12:51
jaosoriorEmilienM: done.12:52
hjensastherve: afaict the data is a list. btw, it works on stack create.12:52
EmilienMjaosorior: thx12:53
thervehjensas: When does it fail then?12:53
hjensastherve: on stack update, after applying the change to apache.j2.yaml. I think I may have found something, there is an issue with https://review.openstack.org/#/c/613459/7/network/networks.j2.yaml - the yaql converts to a string. I will change that to expression: switch(not $.data => 'disabled', $.data => $.data)12:55
openstackgerritChandan Kumar proposed openstack/tripleo-quickstart master: Run tempest.api.identity.v3 tests in fs016  https://review.openstack.org/61867412:55
hjensastherve: I will have to test, but having that fix in on the initial stack create may fix the update issue.12:55
thervehjensas: Not sure what you mean by create/update12:56
therveWhat do you put in create, and what in update?12:56
openstackgerritQuique Llorente proposed openstack-infra/tripleo-ci master: Add standalone scenario002 job  https://review.openstack.org/61870912:56
hjensastherve: I mean overcloud stack CREATE|UPDATE.12:56
thervehjensas: Sure, but what's the template in create?12:58
therveIt doesn't mean much to say it works if you don't test it :D12:58
*** jcoufal has joined #tripleo12:59
*** leanderthal has joined #tripleo12:59
*** morazi has joined #tripleo13:02
openstackgerritQuique Llorente proposed openstack-infra/tripleo-ci master: Cleanup images directory recollection  https://review.openstack.org/61874213:03
hjensastherve: I think the issue is the yaql did string convert on the list with the template used to create the stack. yaql string function looks like a tuple? http://paste.openstack.org/show/735895/ I will have to go back and re-do the tests.13:04
thervehjensas: That may be right yeah13:05
therveThough something would have to cast back to tuple13:05
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-common master: Persist region name in overcloudrc  https://review.openstack.org/61923613:06
*** ooolpbot has joined #tripleo13:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION13:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180354713:10
openstackLaunchpad bug 1803547 in tripleo "tempest run failed with 'The specified regex doesn't match with anything ' on tripleo-ci-centos-7-scenario001-multinode-oooq-container" [Critical,In progress] - Assigned to chandan kumar (chkumar246)13:10
*** ooolpbot has quit IRC13:10
*** rh-jelabarre has joined #tripleo13:10
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-common master: Persist region name in overcloudrc  https://review.openstack.org/61923613:13
*** zul has joined #tripleo13:13
*** amoralej is now known as amoralej|lunch13:13
EmilienMmschuppert: I think https://review.openstack.org/#/c/610966 brought https://bugs.launchpad.net/tripleo/+bug/180443413:19
openstackLaunchpad bug 1804434 in tripleo "podman fails to find image: error getting image" [High,In progress] - Assigned to Emilien Macchi (emilienm)13:19
EmilienMhttps://bugs.launchpad.net/tripleo/+bug/180443413:20
EmilienMnew title13:20
mschuppertEmilienM: checking13:21
openstackgerritQuique Llorente proposed openstack/tripleo-heat-templates master: Add 'podman' as allowed option at ContainerCli  https://review.openstack.org/61923013:24
openstackgerritQuique Llorente proposed openstack/tripleo-heat-templates master: Add env file for standalone scenario002  https://review.openstack.org/61853713:25
openstackgerritQuique Llorente proposed openstack-infra/tripleo-ci master: Add standalone scenario002 job  https://review.openstack.org/61870913:25
openstackgerritQuique Llorente proposed openstack/tripleo-heat-templates master: Add 'podman' as allowed option at ContainerCli  https://review.openstack.org/61923013:25
openstackgerritQuique Llorente proposed openstack/tripleo-heat-templates master: Add env file for standalone scenario002  https://review.openstack.org/61853713:25
EmilienMquiquell: you can abandon https://review.openstack.org/#/c/61923013:28
quiquellEmilienM: Yep, what a brain frat13:29
quiquells/frat/fart/13:29
quiquellEmilienM: maybe you can help us with scenario002 at standalone13:29
quiquellwell 001/002/003 have same issues13:30
quiquellmarios, EmilienM: ^ looks like they just add the paremter to fix the podman thing13:30
quiquellI mean rfolco13:31
mariosquiquell: so we need the podman for haproxy cli thing?13:32
mariosquiquell: and wherever else we don't have it yet13:32
mariosquiquell: did you get a run with that?13:33
quiquellmarios: Yep but they have already merge the change so it's fixed13:33
quiquellmarios: I am into it13:33
quiquellmarios, rfolco: just rebase env file13:33
mariosslightly related but anyone seen this from rdo cloud for standalone reproducer?13:33
marios*=* 15:15:55 *=*=*= " Cleanup images directory recollection " https://review.openstack.org/#/c/618742/ 1If4e1b88403cf0478fc2c315ae2592ae382f51a8c tripleo-ci13:33
chkumar|ruckjaosorior, https://review.openstack.org/#/c/619236/ please make sure tempest passes in nv jobs as on region name tempest tests depends on that13:33
mariossorry this is what i am getting... quiquell seen this before? Create_Failed: Resource CREATE failed: StackValidationFailed: resources.subnodes.resources[0].resources.subnode_floating_ip: Property error: subnode_floating_ip.Properties.port_id: Multiple port matches found for name '', use an ID to be more specific13:33
quiquellmarios: Humm that's different13:34
mariosquiquell: wondering if it is because i already have a standalone on rdo cloud, maybe there is a conflict if we have 213:34
mariosquiquell: yeah this is me trying to reproduce it so i can debug13:34
mariosquiquell: it isn't related to the scenario error/podman thing13:34
quiquellmarios: Ahh ok,  yep, you have to do cleanup13:35
quiquellmarios: nope, it find more than one network, so you have to be more specific13:35
quiquellmarios: It means go an do cleanup13:35
openstackgerritQuique Llorente proposed openstack/tripleo-heat-templates master: Add env file for standalone scenario002  https://review.openstack.org/61853713:35
mariosquiquell: oh it just merged or something? /me reading https://review.openstack.org/#/c/619230/3/docker/services/pacemaker/haproxy.yaml@10013:36
jaosoriorchkumar|ruck: can you ellaborate?13:36
quiquellmarios: Yep just merged13:36
*** dciabrin has quit IRC13:36
mariosquiquell: ok cool so recheck then13:36
quiquellmarios: And rebase before13:36
mschuppertEmilienM: I have not seen this before. need to check if I can reproduce it manually13:36
mariosquiquell: ack13:36
*** dciabrin has joined #tripleo13:36
chkumar|ruckshardy, https://logs.rdoproject.org/48/17348/6/check/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master/a77fb04/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-11-21_13_17_0513:37
chkumar|ruckshardy, I am not sure it is related to your patch13:37
openstackgerritQuique Llorente proposed openstack/tripleo-heat-templates master: Add env file for standalone scenario002  https://review.openstack.org/61853713:37
chkumar|ruckshardy, but overcloud deploy failed13:37
*** rlandy has joined #tripleo13:37
mariosquiquell: i don't find it in master tht yet did it merge? do you have the review please13:38
quiquellmarios: This is the review that fixed it like 20 hours ago https://review.openstack.org/#/c/617833/13:38
mariosquiquell: thx13:39
quiquellmarios: we are so lucky :-)13:39
quiquellEmilienM: thanks13:39
EmilienMmschuppert: if you have time to answer on https://github.com/containers/libpod/issues/1844#issuecomment-44066438313:40
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates master: Initial conversion of scenario001 to standalone  https://review.openstack.org/61659213:40
mschuppertEmilienM: will do13:41
*** jpena|lunch is now known as jpena13:42
*** Chaserjim has joined #tripleo13:42
*** Chaserjim has quit IRC13:47
*** ratailor has quit IRC13:49
*** quiquell is now known as quiquell|brb13:53
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-heat-templates master: Remove validation part of the online database migration pre-upgrade.  https://review.openstack.org/61924713:57
openstackgerritJavier Peña proposed openstack-infra/tripleo-ci master: Use venv path for rdopkg call  https://review.openstack.org/61850913:57
*** janki has joined #tripleo13:59
jtomasekhonza: I've added some comments to https://review.openstack.org/#/c/619071/14:00
*** matbu has quit IRC14:00
*** mcornea has joined #tripleo14:00
openstackgerritBeth Elwell proposed openstack/tripleo-common stable/rocky: Select message IDs properly when publishing to swift  https://review.openstack.org/61924914:01
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: Create reproducer_extra_vars.yml with vars used in run-v3.yaml  https://review.openstack.org/61699314:01
*** trown|outtypewww is now known as trown14:02
*** matbu has joined #tripleo14:02
openstackgerritBeth Elwell proposed openstack/tripleo-common stable/rocky: Exit when container is missing or empty when downloading logs  https://review.openstack.org/61925014:02
chkumar|ruckjaosorior, https://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/utils.py#L235 sets up regionname for each of the service which gets dumped in tripleo deployer input conf file and which is used python-tempestconf to generate tempest.conf14:04
chkumar|ruckjaosorior, just wanted to make sure https://review.openstack.org/#/c/619236/ doesnot break anything tempest side14:05
jaosoriorchkumar|ruck: it keeps using the tripleo default./14:05
chkumar|ruckjaosorior, ok, then we are good there14:05
jaosoriorchkumar|ruck: all that patch does it persist the value that is given through the KeystoneRegion parameter14:06
*** Chaserjim has joined #tripleo14:06
*** ykarel is now known as ykarel|away14:06
openstackgerritKamil Sambor proposed openstack/tripleo-heat-templates master: Check and enable openvswitch package  https://review.openstack.org/61903114:06
*** zbitter is now known as zaneb14:06
*** dtantsur|brb is now known as dtantsur14:07
jaosoriorchkumar|ruck: at some point we need to make that region from the bit you showed me to be configurable.14:08
*** ooolpbot has joined #tripleo14:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION14:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180354714:10
*** ooolpbot has quit IRC14:10
openstackLaunchpad bug 1803547 in tripleo "tempest run failed with 'The specified regex doesn't match with anything ' on tripleo-ci-centos-7-scenario001-multinode-oooq-container" [Critical,In progress] - Assigned to chandan kumar (chkumar246)14:10
*** Chaserjim has quit IRC14:10
*** chkumar|ruck has quit IRC14:12
*** amoralej|lunch is now known as amoralej14:13
rfolcoquiquell|brb, marios so just rebase/recheck or any changes to scen-stand?14:13
mariosrfolco: yeah rebase your environment change (tht) and recheck the job (is what i did anyway)14:14
rfolcomarios, thx14:14
mariosrfolco: https://review.openstack.org/#/c/617833/4 is the change we want and it merged last night14:14
rfolcomarios, cool, thanks for sharing14:15
mariosthanks to quiquell|brb for digging rfolco14:15
*** toure|gone is now known as toure14:17
*** quiquell|brb is now known as quiquell14:19
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: Create reproducer_extra_vars.yml with vars used in run-v3.yaml  https://review.openstack.org/61699314:20
openstackgerritRadomir Dopieralski proposed openstack/tripleo-heat-templates master: Don't force Horizon's secure cookies to disabled  https://review.openstack.org/61925614:21
*** tzumainn has joined #tripleo14:22
*** jaosorior has quit IRC14:25
*** quiquell is now known as quiquell|brb14:32
*** vinaykns has joined #tripleo14:33
openstackgerritDerek Higgins proposed openstack/tripleo-docs master: Remove redundant mentions of boot_option  https://review.openstack.org/61926114:35
*** chem has quit IRC14:35
*** agopi|brb has quit IRC14:37
openstackgerritMartin Mágr proposed openstack/tripleo-heat-templates master: Fix logging for metrics_qdr  https://review.openstack.org/61926314:39
*** udesale has joined #tripleo14:39
*** nawar has quit IRC14:41
openstackgerritJohn Fulton proposed openstack/tripleo-common stable/rocky: Update swift_backup workflow success status message  https://review.openstack.org/61913014:44
*** skramaja has quit IRC14:47
*** quiquell|brb is now known as quiquell14:48
openstackgerritJiri Stransky proposed openstack/tripleo-heat-templates master: Stop upgrade if a task on one node fails  https://review.openstack.org/61926514:50
openstackgerritBrent Eagles proposed openstack/tripleo-heat-templates master: neutron-cleanup skips ports marked skip_cleanup=true  https://review.openstack.org/61906014:51
openstackgerritDavid Peacock proposed openstack/ansible-role-redhat-subscription master: add support for unregistration  https://review.openstack.org/61654714:54
*** ykarel|away has quit IRC14:55
*** chem has joined #tripleo15:00
jankiEmilienM, hi15:00
*** leanderthal has quit IRC15:01
*** jfrancoa has quit IRC15:04
honzajtomasek: some??? :)15:07
*** agopi has joined #tripleo15:08
mwhahahaso why is CI listed as red but we only have a single alert  that has been open for 6 days15:08
mwhahahaEmilienM: -^15:08
EmilienMI don't know15:09
Tengumwhahaha: topic set by jaosorior apparently.15:09
EmilienMwe need to stop creating alerts for everything15:09
*** ykarel|away has joined #tripleo15:10
mwhahahaand we cleared the gate because...15:10
*** ooolpbot has joined #tripleo15:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION15:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180354715:10
*** ooolpbot has quit IRC15:10
openstackLaunchpad bug 1803547 in tripleo "tempest run failed with 'The specified regex doesn't match with anything ' on tripleo-ci-centos-7-scenario001-multinode-oooq-container" [Critical,In progress] - Assigned to chandan kumar (chkumar246)15:10
*** ykarel|away is now known as ykarel15:10
Tengumwhahaha: because 2 patches had to be "force merged"15:10
mwhahahabut why15:10
Tenguwait15:10
Tenguwell, 17h lag in gate first, I'd say.15:10
mwhahahai don't know why those tempest changes needed to be forced15:10
mwhahahabut that's not a reason to clear it without actionable thigns15:11
mwhahahaor at least bugs15:11
mwhahahabecause now we have no plan forward?15:11
*** chandankumar has joined #tripleo15:11
Tengu[13:11] < chkumar|ruck> EmilienM, mwhahaha can we force merge these two patches https://review.openstack.org/#/c/618994/ and https://review.openstack.org/#/c/618674/ in tempest fixes  // [13:14] < jaosorior> chkumar|ruck: ok, gate clear15:11
Tenguthat's all I can say.15:12
Tenguchandankumar: ping? :)15:12
chandankumarTengu, Hello15:12
mwhahahathe scenarios aren't even voting for master15:13
* mwhahaha is confused15:13
chandankumarTengu, scenario001 in gate jobs are voting15:13
mwhahahanot for master15:13
Tenguhmm, and jaosorior didn't send any mail about the red status :(15:13
mwhahahathey shouldn't be15:13
chandankumarmwhahaha, few hours back, the zuul queue has reached 20 hours15:14
mwhahahaok and do we know why?15:14
mwhahahais it timeouts? podman failures? where are the bugs?15:14
mwhahahajust because the queue is long doesn't mean we just kick everything out15:15
chandankumarmwhahaha, nope, I have started tracking down timedout, not yet bugs15:15
mwhahahathen we shouldn't have acted15:15
* mwhahaha sighs15:15
openstackgerritDavid Peacock proposed openstack/ansible-role-redhat-subscription master: add support for unregistration  https://review.openstack.org/61654715:16
*** jfrancoa has joined #tripleo15:17
Tenguwow. I think I never saw that one until now: http://logs.openstack.org/75/618975/4/check/tripleo-ci-centos-7-containers-multinode-queens/3f8f18c/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-11-21_13_31_4815:17
openstackgerritDavid Peacock proposed openstack/ansible-role-redhat-subscription master: add support for unregistration  https://review.openstack.org/61654715:18
chandankumarmwhahaha, please change the channel topic15:18
Tenguah. error 500 somewhere. anyway.15:18
EmilienMprobably mistral or heat15:18
EmilienMlet me see15:18
Tengubtw, EmilienM: https://review.openstack.org/#/c/618529/  care to push that one? :)15:18
* Tengu really, really wants haproxy stats in the CI logs15:18
*** mwhahaha changes topic to "CI Status: ORANGE? No loading up the gate as we're investigating | https://docs.openstack.org/tripleo-docs/latest/"15:19
EmilienMhttp://logs.openstack.org/75/618975/4/check/tripleo-ci-centos-7-containers-multinode-queens/3f8f18c/logs/undercloud/var/log/mistral/api.log.txt.gz#_2018-11-21_13_31_29_87815:19
EmilienMmistral timeout ^15:19
chandankumarmwhahaha, thanks :-)15:19
Tenguyay15:19
Tenguanyway, that one is a nice to have for podman log investigation. Would be cool to see it merged rather sooner than later ;).15:19
jtomasekhonza: it added up lol15:21
jtomasekhonza: sorry15:21
honzajtomasek: np, thanks for looking at it so quickly, i clearly didn't understand very well15:22
*** chandankumar has quit IRC15:22
*** chandankumar has joined #tripleo15:24
*** jaosorior has joined #tripleo15:24
*** Vorrtex has joined #tripleo15:27
openstackgerritCarlos Camacho proposed openstack/instack-undercloud stable/rocky: Get the stack name if possible before running the validations group  https://review.openstack.org/61911215:31
*** Chaserjim has joined #tripleo15:33
*** suuuper has quit IRC15:34
*** chandankumar has quit IRC15:39
*** quiquell is now known as quiquell|off15:39
*** ramishra has quit IRC15:45
*** Chaserjim has quit IRC15:46
*** ratailor has joined #tripleo15:48
openstackgerritChristian Schwede proposed openstack/tripleo-heat-templates master: Disable deprecated parameter ControllerEnableSwiftStorage  https://review.openstack.org/61927615:48
mwhahahaok so there is a blocking issue with the container-selinuxs tuff15:49
mwhahahaEmilienM http://logs.openstack.org/29/618529/4/gate/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/1bd3e5b/logs/undercloud/home/zuul/install_packages.sh.log.txt.gz#_2018-11-21_15_41_4615:49
mwhahahathe updates job is hosed15:49
EmilienMmwhahaha: https://bugs.launchpad.net/tripleo/+bug/180447515:50
openstackLaunchpad bug 1804475 in tripleo "[master][vexhost][fs001]container-selinux package failed to install during install packages step" [Critical,Triaged]15:50
EmilienMmwhahaha: I pinged the maintainer, he's looking at it15:50
mwhahahak15:50
mwhahahaalso broke kolla's gate too15:51
EmilienMgood15:51
EmilienMso we aren't alone :D15:51
Tenguerf... so that's http://logs.openstack.org/29/618529/4/gate/tripleo-ci-centos-7-standalone/596fa1c/job-output.txt.gz#_2018-11-21_15_45_45_534447 I guess?15:51
mwhahahayes15:51
Tenguyay.15:51
*** mwhahaha changes topic to "CI Status:REEEEEEEEEEEEEEED and stuff. no approving anything | https://docs.openstack.org/tripleo-docs/latest/"15:52
*** ykarel is now known as ykarel|away15:52
Tenguguess it's a sign for me to call it a day, and to go doing some terraform + freeipa deploy tests.15:52
*** noama has quit IRC15:52
*** udesale has quit IRC15:58
jaganathanjaosorior, hi15:59
*** dsneddon has joined #tripleo16:02
*** ooolpbot has joined #tripleo16:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION16:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180354716:10
openstackLaunchpad bug 1803547 in tripleo "tempest run failed with 'The specified regex doesn't match with anything ' on tripleo-ci-centos-7-scenario001-multinode-oooq-container" [Critical,In progress] - Assigned to chandan kumar (chkumar246)16:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180447516:10
*** ooolpbot has quit IRC16:10
openstackLaunchpad bug 1804475 in tripleo "[master][vexhost][fs001]container-selinux package failed to install during install packages step" [Critical,Triaged]16:10
openstackgerritDmitry Tantsur proposed openstack/tripleo-docs master: Document using direct deploy with ironic  https://review.openstack.org/61928716:10
*** ksambor has quit IRC16:12
openstackgerritNatal Ngétal proposed openstack/tripleo-quickstart master: [Configuration] Remove mitaka file.  https://review.openstack.org/61767016:15
*** agopi is now known as agopi|food16:17
*** ratailor has quit IRC16:24
*** chem has quit IRC16:24
*** kopecmartin is now known as kopecmartin|off16:25
*** ykarel|away has quit IRC16:27
*** chem has joined #tripleo16:30
openstackgerritMerged openstack/tripleo-heat-templates stable/queens: Ensure octavia flavor gets configured if composable octavia  https://review.openstack.org/61880516:32
*** saneax has quit IRC16:36
*** janki has quit IRC16:36
openstackgerritThomas Herve proposed openstack/tripleo-quickstart-extras master: Use a custom roles file for undercloud deploy  https://review.openstack.org/61917716:38
*** Baggypants12000 has joined #tripleo16:39
Baggypants12000I'm having trouble with the tripleo quickstart, It gets to the part where it's customising overcloud-full.qcow2 and fails.16:40
*** gkadam has quit IRC16:40
Baggypants12000in _home_stack_repo_setup.sh.log it reports something about an selinux package conflict and suggests running with --allowerasing16:41
Baggypants12000has anyone else met this?16:41
mwhahahathere's currently an upstream issue16:41
mwhahahaaround that package16:41
mwhahahais not related to quickstart16:41
Baggypants12000Excellent16:41
mwhahahawe're tracking it with  https://bugs.launchpad.net/tripleo/+bug/180447516:42
openstackLaunchpad bug 1804475 in tripleo "[master][vexhost][fs001]container-selinux package failed to install during install packages step" [Critical,Triaged]16:42
openstackgerritEmilien Macchi proposed openstack/paunch master: runner: run inspect quietly  https://review.openstack.org/61931316:43
Baggypants12000hmm, looks like I picked the wrong day to install openstack.16:43
Baggypants12000Thanks mwhahaha16:43
mwhahaha:D16:43
mwhahahayou could try an older (non-master) version16:44
mwhahahai don't think we pull in that repo for stable versions, though we might16:44
*** cylopez has quit IRC16:47
*** Chaserjim has joined #tripleo16:48
*** paramite has quit IRC16:48
*** pcaruana has quit IRC16:50
*** dsneddon has quit IRC16:51
*** agopi|food is now known as agopi16:51
*** xek_ has joined #tripleo16:54
*** dsneddon has joined #tripleo16:57
*** xek has quit IRC16:57
openstackgerritMichele Baldessari proposed openstack/puppet-tripleo master: WIP Fix up redis haproxy check  https://review.openstack.org/61931716:57
*** shardy has quit IRC17:01
*** yprokule has quit IRC17:02
Chaserjim.17:02
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: upgrade: remove Docker containers now managed by Podman  https://review.openstack.org/61520917:03
*** jfrancoa has quit IRC17:05
openstackgerritJohn Fulton proposed openstack/tripleo-quickstart-extras master: WIP/DNM: Add split_control_plane_{controller,compute} booleans  https://review.openstack.org/61736817:07
*** shardy has joined #tripleo17:07
*** bogdando has quit IRC17:09
*** ooolpbot has joined #tripleo17:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION17:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180354717:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180447517:10
*** ooolpbot has quit IRC17:10
openstackLaunchpad bug 1803547 in tripleo "tempest run failed with 'The specified regex doesn't match with anything ' on tripleo-ci-centos-7-scenario001-multinode-oooq-container" [Critical,In progress] - Assigned to chandan kumar (chkumar246)17:10
openstackLaunchpad bug 1804475 in tripleo "[master][vexhost][fs001]container-selinux package failed to install during install packages step" [Critical,Triaged]17:10
openstackgerritDavid Peacock proposed openstack/ansible-role-redhat-subscription master: add support for unregistration  https://review.openstack.org/61654717:10
*** cgoncalves has quit IRC17:12
*** sambetts|afk has quit IRC17:14
*** cgoncalves has joined #tripleo17:14
*** betherly has joined #tripleo17:15
*** sambetts_ has joined #tripleo17:16
*** xek_ is now known as xek17:18
*** betherly has quit IRC17:21
*** hakhande_ has quit IRC17:21
openstackgerritClark Boylan proposed openstack/diskimage-builder master: package-installs: provide for skip from env var  https://review.openstack.org/61911917:21
openstackgerritClark Boylan proposed openstack/diskimage-builder master: simple-init: allow for NetworkManager support  https://review.openstack.org/61912017:21
*** Chaserjim has quit IRC17:25
*** Chaserjim has joined #tripleo17:25
*** morazi has quit IRC17:27
*** dprince has quit IRC17:29
*** morazi has joined #tripleo17:35
*** Vorrtex has quit IRC17:38
*** jaosorior has quit IRC17:41
*** cgoncalves has quit IRC17:42
*** cgoncalves has joined #tripleo17:42
openstackgerritMichele Baldessari proposed openstack/tripleo-common master: Generating fencing for capitalized MACs  https://review.openstack.org/61399017:44
*** florianf has quit IRC17:45
*** cylopez has joined #tripleo17:48
*** dtantsur is now known as dtantsur|afk17:49
*** derekh has quit IRC17:50
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates master: Adjust haproxy-public-tls-inject.yaml bootstrap variable  https://review.openstack.org/61921517:50
*** maufart__ has quit IRC17:51
*** holser|lunch has quit IRC17:52
*** dprince has joined #tripleo18:01
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: Create reproducer_extra_vars.yml with vars used in run-v3.yaml  https://review.openstack.org/61699318:04
*** jcoufal_ has joined #tripleo18:06
*** mcornea has quit IRC18:09
*** jcoufal has quit IRC18:10
*** ooolpbot has joined #tripleo18:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION18:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180354718:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180447518:10
*** ooolpbot has quit IRC18:10
openstackLaunchpad bug 1803547 in tripleo "tempest run failed with 'The specified regex doesn't match with anything ' on tripleo-ci-centos-7-scenario001-multinode-oooq-container" [Critical,In progress] - Assigned to chandan kumar (chkumar246)18:10
openstackLaunchpad bug 1804475 in tripleo "[master][vexhost][fs001]container-selinux package failed to install during install packages step" [Critical,Triaged]18:10
*** salmankhan has quit IRC18:11
*** chem has quit IRC18:11
*** jpena is now known as jpena|off18:13
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: WIP: Add a new role to clone repos and include zuul changes  https://review.openstack.org/61799718:16
*** mcornea has joined #tripleo18:16
*** florianf has joined #tripleo18:16
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: Create reproducer_extra_vars.yml with vars used in run-v3.yaml  https://review.openstack.org/61699318:18
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: WIP: Rewrite nodepool-setup role to be used with v3 playbooks  https://review.openstack.org/61785518:19
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: WIP: Add a new role to clone repos and include zuul changes  https://review.openstack.org/61799718:19
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: Create reproducer_extra_vars.yml with vars used in run-v3.yaml  https://review.openstack.org/61699318:20
*** hberaud has quit IRC18:24
openstackgerritRafael Folco proposed openstack-infra/tripleo-ci master: Deprecate unsupported/old stable branch jobs  https://review.openstack.org/61933718:29
*** trown is now known as trown|lunch18:33
openstackgerritRonelle Landy proposed openstack-infra/tripleo-ci master: WIP - DNM - Run toci_gate_test with tmux in the reproducer  https://review.openstack.org/61865418:33
*** shardy has quit IRC18:34
*** shardy has joined #tripleo18:36
*** dsneddon has quit IRC18:39
openstackgerritHonza Pokorny proposed openstack/tripleo-common master: Add a way to update ContainerImagePrepare params  https://review.openstack.org/61907118:42
openstackgerritHonza Pokorny proposed openstack/tripleo-common master: Add a way to update ContainerImagePrepare params  https://review.openstack.org/61907118:47
ChaserjimHi tripleo - has anyone seen this before when performing a scale-out of a compute?18:49
ChaserjimWorkflow 'tripleo.deployment.v1.config_download_deploy' [RUNNING -> ERROR, msg=Failure caused by error in tasks: send_message18:49
ChaserjimNot very descriptive18:49
*** ahosam has joined #tripleo18:53
*** shardy has quit IRC18:56
*** amoralej is now known as amoralej|off18:56
*** dsneddon has joined #tripleo18:57
openstackgerritMichele Baldessari proposed openstack/puppet-tripleo master: mysql: fix root password update for containerized mysql  https://review.openstack.org/60249919:00
*** xek has quit IRC19:00
*** apetrich has quit IRC19:00
openstackgerritMichele Baldessari proposed openstack/puppet-tripleo master: WIP Make the neutron-haproxy-wrapper compatible with haproxy 1.8  https://review.openstack.org/60791619:02
*** morazi has quit IRC19:07
*** holser_ has joined #tripleo19:07
*** xek has joined #tripleo19:08
*** dprince has quit IRC19:08
*** ooolpbot has joined #tripleo19:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION19:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180354719:10
openstackLaunchpad bug 1803547 in tripleo "tempest run failed with 'The specified regex doesn't match with anything ' on tripleo-ci-centos-7-scenario001-multinode-oooq-container" [Critical,In progress] - Assigned to chandan kumar (chkumar246)19:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180447519:10
*** ooolpbot has quit IRC19:10
openstackLaunchpad bug 1804475 in tripleo "[master][vexhost][fs001]container-selinux package failed to install during install packages step" [Critical,Triaged]19:10
openstackgerritMichele Baldessari proposed openstack/puppet-pacemaker master: WIP Initial pcs 0.10 support  https://review.openstack.org/60787319:12
*** apetrich has joined #tripleo19:15
*** dsneddon has quit IRC19:17
bandinimwhahaha: I have not yet looked at the python3 ansible-pacemaker stuff, I'll add it on my todo19:17
mwhahahabandini: k i worked around it by creating a fake python-lxml package for rhel819:18
bandinimwhahaha: so is the issue that it requires python-lxml which does not exist or is it more a general code issue? (or a simple packaging problem)?19:18
mwhahahabandini: likely both19:18
bandiniack19:19
mwhahahabandini: it likely does not work in python3 because of packaging, but i got around the install at least by faking it19:19
bandinialrighty, I'll investigate19:19
bandinisurely it will be another lovely bag of fail19:19
mwhahahaeh might just need a python3 conversion in the spec19:20
mwhahahabut who knows :D19:20
bandiniehehe19:20
openstackgerritJohn Fulton proposed openstack/tripleo-quickstart-extras master: WIP/DNM: Add split_control_plane_{controller,compute} booleans  https://review.openstack.org/61736819:21
*** raildo has quit IRC19:43
*** raildo has joined #tripleo19:43
*** trown|lunch is now known as trown19:44
openstackgerritCarlos Camacho proposed openstack/instack-undercloud stable/rocky: Get the stack name if possible before running the validations group  https://review.openstack.org/61911219:57
Chaserjimanyone ever seen this issue during a scale-out of ovs compute? ERROR mistral.executors.default_executor KeyError: u'23f1abe0-aa3f-46c3-a712-d3216b9b297d'19:58
openstackgerritCarlos Camacho proposed openstack/instack-undercloud stable/rocky: Get the stack name if possible before running the validations group  https://review.openstack.org/61911219:59
*** apetrich has quit IRC20:01
*** dsneddon has joined #tripleo20:04
*** krypto has joined #tripleo20:08
kryptoThanks jaosorior20:08
*** ooolpbot has joined #tripleo20:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION20:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180354720:10
openstackLaunchpad bug 1803547 in tripleo "tempest run failed with 'The specified regex doesn't match with anything ' on tripleo-ci-centos-7-scenario001-multinode-oooq-container" [Critical,In progress] - Assigned to chandan kumar (chkumar246)20:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180447520:10
*** ooolpbot has quit IRC20:10
openstackLaunchpad bug 1804475 in tripleo "[master][vexhost][fs001]container-selinux package failed to install during install packages step" [Critical,Triaged]20:10
kryptoin TripleO if i want to add a new compute node to the cluster ,can i avoid executing all the unnecessary tasks on other nodes.I feel on other compute nodes only /etc/hosts need updation. Is it possible20:11
*** vpickard is now known as vpickard_20:12
mwhahahakrypto: depends on the version, we do have some black listing capabilities which will prevent extra stuff, there's also skip deployment identifier which should only execute things when there is a config change20:15
kryptothanks mwhahaha is it possible on newton release20:17
mwhahahano20:17
kryptomay i know from which release its supported ?20:17
mwhahahapike20:17
mwhahahaonward20:18
mwhahahathat's the only place where it really works20:18
fultonjChaserjim: possibly replated to "msg=Failure caused by error in tasks: send_message" https://bugs.launchpad.net/tripleo/+bug/178968020:20
openstackLaunchpad bug 1789680 in tripleo "mistral MessagingTimeout correlates with containerized undercloud uptime" [Critical,Fix released] - Assigned to Dougal Matthews (d0ugal)20:20
ChaserjimIt looks like its more related here: Which is odd since I have this fix in my code base20:21
Chaserjimhttps://bugs.launchpad.net/tripleo/+bug/179360520:21
openstackLaunchpad bug 1793605 in tripleo "GetOvercloudConfig fails when a node is blacklisted: server_id isn't found" [High,Fix released] - Assigned to Emilien Macchi (emilienm)20:21
kryptomwhahaha thats not what i was expecting ... :P i am on Newton release20:23
mwhahahakrypto: sorry but from an upstream perspective newton is EOL and we didnt add that functionality until pike20:23
mwhahahaand it wasn't trivial20:24
kryptowill it restart controller services as well?20:24
mwhahahait shouldn't unless there are changes20:24
*** ahosam has quit IRC20:24
kryptocould you tell me how it decides whether there are  changes or not.I am new to triple-O and previously i was using mirantis fuel and it had this feature20:25
mwhahahakrypto: same way fuel did, if there are config changes. we run puppet under the covers so if you are managing things only with tripleo it should be idempotent20:26
mwhahahakrypto: if you are doing any post-install configurations it may update your configs and trigger restarts20:26
openstackgerritRonelle Landy proposed openstack-infra/tripleo-ci master: Make hard linking tripleo-ci idempotent  https://review.openstack.org/61829920:29
openstackgerritRonelle Landy proposed openstack-infra/tripleo-ci master: WIP - DNM - Run toci_gate_test with tmux in the reproducer  https://review.openstack.org/61865420:29
kryptoif i recollect correctly in fuel it uses puppet hashes to check for changes.my concern we have added some custom patches to the binaries after installing from rpm files.My gut feeling is that this will not be reverted since for puppet package is already installed20:30
mwhahahakrypto: no it didn't (i know cause i dealt with it). tripleo doesn't care about the binary bits just the configs20:31
*** akrivoka_ has joined #tripleo20:33
*** raildo has quit IRC20:35
mwhahahajillr: bug 1804311 is a dupe of bug 1801162 which is already fix but we need a promotion20:35
openstackbug 1801162 in tripleo "output of group vars that are not string/ints may include python encoding" [High,Fix released] https://launchpad.net/bugs/1801162 - Assigned to Alex Schultz (alex-schultz)20:35
*** akrivoka has quit IRC20:36
openstackgerritCarlos Camacho proposed openstack/instack-undercloud stable/rocky: Get the stack name if possible before running the validations group  https://review.openstack.org/61911220:41
kryptoThanks mwhahaha is there a way to list all the variables that are managed by puppet,i forgot the hiera commands to get it.I am not good in puppet either :)20:45
mwhahahakrypto: all of it? not sure what you mean20:46
mwhahahakrypto: fyi the hiera data is in /etc/puppet/hieradata on the nodes20:46
mwhahahabut the changes in your environment files would be what triggers service restarts20:46
*** dsneddon has quit IRC20:48
kryptoi have many custom changes to our sdn controller which should not be reverted when i add compute nodes.if the variable is not in /etc/puppet/hieradata/ is it safe in assuming that it is not managed by puppet20:48
mwhahahano20:49
mwhahahawhat type of sdn changes20:49
kryptosome sdn specific rbac,BGP asn20:50
mwhahaharight i don't know what kinda config changes those would be20:50
mwhahahaso sorry can't really help you. I would hope you have a staging environment that you could test beforehand20:50
kryptothere is a way to pass asn in heat template,but i couldnt do it at the time of deployment.So it configured the default ASN .Post deployment we changed it on the SDN controller.Also cpu pinning parameters(i know this will be reverted )20:52
kryptoby the way your name is so familiar :)20:52
*** dsneddon has joined #tripleo20:53
mwhahahaso yea if you don't replicate the config into heat it'll probably get reverted20:54
mwhahahabut it depends on where that ends up20:54
mwhahahai'm not sure where that config lives20:54
*** akrivoka_ has quit IRC20:55
*** devep has joined #tripleo20:59
hjensasmwhahaha: We set 'apache::mod::remoteip::proxy_ips' here: https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/services/apache.j2.yaml#L101-L102 . But I don't see remoteip.conf in etc/httpd/conf.modules.d. Am I looking in the wrong place?21:00
*** devep_ has joined #tripleo21:00
*** devep has quit IRC21:00
*** devep has joined #tripleo21:00
*** holser_ has quit IRC21:01
openstackgerritAlex Schultz proposed openstack/tripleo-common master: Specify python interpreter for undercloud  https://review.openstack.org/61935721:02
itlinuxhappy TG tomorrow to everyone! Ciao21:03
mwhahahahjensas: what version21:03
hjensasmwhahaha: master, or whatever we promoted ... I was looking for a reference in a CI job and see nothing. - https://review.openstack.org/61903821:04
mwhahahahjensas: it would be in a container i think21:04
mwhahahahjensas: and that would assume we enable that mod21:04
EmilienMdo we include that module?21:05
mwhahahai don't think so?21:06
hjensasmwhahaha: yeah, I looked under var/log/config-data/ in the CI job.21:06
hjensasok, we would have a .load file for the module if we included it? I don't see that either.21:08
mwhahahawe only include it for horizon21:09
mwhahahahttp://git.openstack.org/cgit/openstack/puppet-tripleo/tree/manifests/profile/base/horizon.pp#n8721:10
*** ooolpbot has joined #tripleo21:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION21:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180354721:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180447521:10
*** ooolpbot has quit IRC21:10
openstackLaunchpad bug 1803547 in tripleo "tempest run failed with 'The specified regex doesn't match with anything ' on tripleo-ci-centos-7-scenario001-multinode-oooq-container" [Critical,In progress] - Assigned to chandan kumar (chkumar246)21:10
openstackLaunchpad bug 1804475 in tripleo "[master][vexhost][fs001]container-selinux package failed to install during install packages step" [Critical,Triaged]21:10
hjensasmwhahaha: and I was already hoping I could push a patch to remove the remotip stuff from the apache service tempalte. Guess not then. :)21:12
*** betherly has joined #tripleo21:13
EmilienMmwhahaha: is there anything to force merge for thez tempest alert?21:15
EmilienMI can ping infra21:15
mwhahahathe tempest alert has been open for a while and isn't that serious21:17
mwhahahathe scenario jobs aren't voting in master21:17
EmilienMok21:18
mwhahahai wouldn't bother forcing anything21:18
EmilienMmwhahaha: ok so podman21:18
EmilienMmwhahaha: the untag removed the broken rpm in cbs21:18
EmilienMbut it takes roughly 2h to sync with buildlogs21:18
EmilienMit should happen very soon21:18
EmilienMthe only thing we could do is to replace buildlogs by cbs in our CI21:18
EmilienMbut i'm not sure about the impact21:19
EmilienMcbs is up to date now, for this rpm but i'm not sure we would have other troubles with cbs vs buildlogs21:19
mwhahahado we have a cbs proxy21:19
*** holser_ has joined #tripleo21:20
EmilienMthat's exactly what I'm looking at now21:20
mwhahahais there a virt repo on cbs or would we be adding everything21:21
EmilienMhttps://cbs.centos.org/repos/virt7-container-common-candidate/x86_64/21:21
EmilienMwe would have that ^21:21
mwhahahathat should be fine21:21
mwhahahait's just podman/docker/skopeo21:22
EmilienMNODEPOOL_CBS_CENTOS_PROXY21:22
*** radeks_ has quit IRC21:22
EmilienMlet me see if I can quickly switch that21:23
*** dsneddon has quit IRC21:23
*** betherly has quit IRC21:23
*** dsneddon has joined #tripleo21:25
EmilienMhttp://mirror.sjc1.vexxhost.openstack.org:8080/cbs.centos/virt7-container-common-candidate/x86_64/21:26
EmilienMwe could use that21:26
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart master: master/hack: Switch to CBS instead of buildlogs  https://review.openstack.org/61935921:30
EmilienMmwhahaha: ^21:30
*** devep__ has joined #tripleo21:33
*** devep has quit IRC21:37
*** devep_ has quit IRC21:37
*** devep has joined #tripleo21:38
mmethotstevebaker, hey about adding swap stuff to podman. Swap isnt something configurable for kube or so I recall21:39
mmethotor am I crazy here?21:39
stevebakermmethot: I see it in podman run --help, so I'm assuming its implemented21:40
*** devep_ has joined #tripleo21:40
*** xek has quit IRC21:40
stevebakermmethot: kube and podman are not really related, except that a CRI-O backed kubernetes shares some core libraries with podman21:41
mmethotstevebaker, yeah from what Im googling now apparently so. Alright thanks21:42
mmethotI just assumed they were related close enough, that they would of had the same mentality21:42
*** ahosam has joined #tripleo21:42
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart master: master/hack: Switch to CBS instead of buildlogs  https://review.openstack.org/61935921:42
*** devep__ has quit IRC21:42
mmethotstevebaker, from the paunch docs though we say: "The intention is for the format to evolve to faithfully implement existing formats such as the `Kubernetes Pod format <https://kubernetes.io/docs/concepts/workloads/pods/pod/>`_.21:43
*** devep__ has joined #tripleo21:43
mmethotshould we not work towards that? ie not handle swap?21:43
stevebakermmethot: we haven't started the pod effort yet, hopefully paunch can just delegate to podman's pod management. Since podman supports swap I'm not really seeing the problem21:45
*** jjoyce has quit IRC21:45
*** holser_ has quit IRC21:45
*** jjoyce has joined #tripleo21:45
jillrmwhahaha: oh, ok thx - it was just reported to me yesterday21:45
*** devep_ has quit IRC21:45
*** devep has quit IRC21:45
*** devep has joined #tripleo21:46
mmethotstevebaker, aiight. I don't have a problem with that I'm just double checking21:46
*** thrash is now known as thrash|g0ne21:46
*** agopi is now known as agopi|brb21:46
mmethotI'm more in agreement with the kubes mindset on swap I guess21:47
openstackgerritAlex Schultz proposed openstack/python-tripleoclient master: Improve bulk symlink error handling  https://review.openstack.org/61936321:48
*** trown is now known as trown|outtypewww21:48
*** jcoufal_ has quit IRC21:50
*** agopi|brb has quit IRC21:51
*** jjoyce has quit IRC21:53
stevebakermmethot: I just think having paunch implement 100% of the compose v1 format is an orthogonal issue to switching to the pod format21:54
*** betherly has joined #tripleo21:55
openstackgerritMarc Methot proposed openstack/paunch master: Allow to set limit memory on containers  https://review.openstack.org/61590921:55
*** jjoyce has joined #tripleo21:56
mmethotstevebaker, sure and that's alright. Made the changes accordingly21:57
stevebakermmethot: ok, thanks21:57
*** betherly has quit IRC22:00
*** betherly has joined #tripleo22:02
*** devep__ has quit IRC22:03
*** devep has quit IRC22:03
*** jjoyce has quit IRC22:04
*** jjoyce has joined #tripleo22:04
*** betherly has quit IRC22:06
*** rcernin has joined #tripleo22:09
*** vinaykns has quit IRC22:09
*** ooolpbot has joined #tripleo22:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION22:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180354722:10
openstackLaunchpad bug 1803547 in tripleo "tempest run failed with 'The specified regex doesn't match with anything ' on tripleo-ci-centos-7-scenario001-multinode-oooq-container" [Critical,In progress] - Assigned to chandan kumar (chkumar246)22:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180447522:10
*** ooolpbot has quit IRC22:10
openstackLaunchpad bug 1804475 in tripleo "[master][vexhost][fs001]container-selinux package failed to install during install packages step" [Critical,Triaged]22:10
EmilienMmwhahaha: https://review.openstack.org/#/c/619359 is working22:14
EmilienMI see undercloud being deployed in zuul22:14
mwhahahak22:15
EmilienMmwhahaha: if by EOD buildlogs isn't updated... should we land that?22:15
mwhahahai'm kinda concerned this is our default repo config in general22:16
EmilienMmeh I'll see if that works22:16
EmilienMyes22:16
mwhahahai wish we could roll some of these into rdo22:16
mwhahahabut yea we probably should switch22:16
*** agopi|brb has joined #tripleo22:19
openstackgerritHarald Jensås proposed openstack/tripleo-heat-templates master: DNM - Debug - aodh service does not have ServiceData  https://review.openstack.org/61937022:21
openstackgerritSteve Baker proposed openstack/tripleo-common master: Implement python based uploader  https://review.openstack.org/61601822:22
openstackgerritSteve Baker proposed openstack/tripleo-common master: WIP make python uploader the default  https://review.openstack.org/61601922:22
hjensasugh, I meant for that debug job to only run tripleo-ci-centos-7-scenario001-multinode-oooq-container. But I see it's running all jobs in Zuul, and the job I want not at all.22:28
hjensasDo I need a depends on patch in quickstart-extras or somthing to get it to run  tripleo-ci-centos-7-scenario001-multinode-oooq-container?22:28
hjensasnevermind, it does run the job ... (I need coffe.)22:31
*** rpioso is now known as rpioso|afk22:36
*** dtrainor_ has joined #tripleo22:37
*** dtrainor has quit IRC22:40
*** cylopez has quit IRC22:47
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: WIP: Copy the zuul inventory to run the reproducer  https://review.openstack.org/61663722:49
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: WIP: Modify the zuul inventory to pass to the reproducer  https://review.openstack.org/61740122:49
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: WIP: Rewrite nodepool-setup role to be used with v3 playbooks  https://review.openstack.org/61785522:49
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: WIP: Add a new role to clone repos and include zuul changes  https://review.openstack.org/61799722:49
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: Create reproducer_extra_vars.yml with vars used in run-v3.yaml  https://review.openstack.org/61699322:50
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: WIP: Rewrite nodepool-setup role to be used with v3 playbooks  https://review.openstack.org/61785522:51
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: WIP: Add a new role to clone repos and include zuul changes  https://review.openstack.org/61799722:51
*** betherly has joined #tripleo22:51
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: Create reproducer_extra_vars.yml with vars used in run-v3.yaml  https://review.openstack.org/61699322:51
*** betherly has quit IRC22:57
*** slagle has quit IRC22:58
*** toure is now known as toure|gone23:03
*** krypto has quit IRC23:03
*** betherly has joined #tripleo23:08
*** ooolpbot has joined #tripleo23:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION23:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180354723:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180447523:10
*** ooolpbot has quit IRC23:10
openstackLaunchpad bug 1803547 in tripleo "tempest run failed with 'The specified regex doesn't match with anything ' on tripleo-ci-centos-7-scenario001-multinode-oooq-container" [Critical,In progress] - Assigned to chandan kumar (chkumar246)23:10
openstackLaunchpad bug 1804475 in tripleo "[master][vexhost][fs001]container-selinux package failed to install during install packages step" [Critical,Triaged]23:10
*** vinaykns has joined #tripleo23:10
*** dtrainor__ has joined #tripleo23:12
*** dtrainor_ has quit IRC23:15
*** rlandy is now known as rlandy|bbl23:17
*** betherly has quit IRC23:18
*** mcornea has quit IRC23:18
*** ahosam has quit IRC23:19
openstackgerritHonza Pokorny proposed openstack/tripleo-common master: Add a way to update ContainerImagePrepare params  https://review.openstack.org/61907123:22
*** florianf has quit IRC23:38
openstackgerritHonza Pokorny proposed openstack/tripleo-common master: Add a way to update ContainerImagePrepare params  https://review.openstack.org/61907123:41
*** slaweq has quit IRC23:45
*** betherly has joined #tripleo23:48
*** Chaserjim has quit IRC23:51
*** zul has quit IRC23:51
*** betherly has quit IRC23:53

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!