| *** pcaruana has quit IRC | 00:03 | |
| *** pkopec has quit IRC | 00:03 | |
| *** rfolco has quit IRC | 00:26 | |
| *** rh-jelabarre has quit IRC | 00:43 | |
| *** rh-jelabarre has joined #tripleo | 00:44 | |
| *** rh-jelabarre has quit IRC | 00:48 | |
| *** gouthamr_ has quit IRC | 00:59 | |
| *** gouthamr_ has joined #tripleo | 01:05 | |
| *** Goneri has quit IRC | 01:18 | |
| *** gouthamr_ has quit IRC | 01:25 | |
| *** gouthamr_ has joined #tripleo | 01:37 | |
| *** d34dh0r53 has quit IRC | 01:38 | |
| *** d34dh0r53 has joined #tripleo | 01:41 | |
| *** rlandy|bbl is now known as rlandy | 01:49 | |
| *** rlandy has quit IRC | 02:04 | |
| *** apetrich has quit IRC | 02:13 | |
| *** gouthamr_ has quit IRC | 02:52 | |
| *** gouthamr_ has joined #tripleo | 02:56 | |
| *** pojadhav is now known as pojadhav|SickLea | 02:57 | |
| *** pojadhav|SickLea is now known as pojadhav|Sick | 02:57 | |
| *** rcernin has quit IRC | 03:00 | |
| *** rcernin has joined #tripleo | 03:03 | |
| *** gouthamr_ has quit IRC | 03:26 | |
| *** dmacpher has quit IRC | 03:27 | |
| *** gouthamr_ has joined #tripleo | 03:32 | |
| *** saneax has joined #tripleo | 03:35 | |
| *** saneax has quit IRC | 03:40 | |
| *** saneax has joined #tripleo | 03:41 | |
| *** gouthamr_ has quit IRC | 04:00 | |
| *** saneax has quit IRC | 04:00 | |
| *** dmacpher has joined #tripleo | 04:00 | |
| *** saneax has joined #tripleo | 04:01 | |
| *** saneax has quit IRC | 04:03 | |
| *** saneax has joined #tripleo | 04:03 | |
| *** gouthamr_ has joined #tripleo | 04:12 | |
| *** soniya29 has joined #tripleo | 04:16 | |
| *** saneax has quit IRC | 04:17 | |
| *** saneax has joined #tripleo | 04:18 | |
| *** ratailor has joined #tripleo | 04:22 | |
| *** saneax has quit IRC | 04:22 | |
| *** saneax has joined #tripleo | 04:22 | |
| *** saneax has quit IRC | 04:24 | |
| *** saneax has joined #tripleo | 04:25 | |
| *** ykarel has joined #tripleo | 04:27 | |
| *** gouthamr_ has quit IRC | 04:29 | |
| *** gouthamr_ has joined #tripleo | 04:30 | |
| openstackgerrit | yatin proposed openstack/tripleo-ansible stable/train: [DNM] Debug c8 train iptables issue https://review.opendev.org/740153 | 04:32 |
|---|---|---|
| *** evrardjp has quit IRC | 04:33 | |
| *** evrardjp has joined #tripleo | 04:33 | |
| *** gouthamr_ has quit IRC | 04:35 | |
| *** gouthamr_ has joined #tripleo | 04:40 | |
| openstackgerrit | yatin proposed openstack/tripleo-ci master: Branchful C8 Train jobs https://review.opendev.org/737682 | 04:50 |
| *** saneax has quit IRC | 04:52 | |
| *** saneax has joined #tripleo | 04:53 | |
| *** saneax has quit IRC | 04:55 | |
| *** saneax has joined #tripleo | 04:55 | |
| *** abdysn has joined #tripleo | 05:02 | |
| *** ysandeep|away is now known as ysandeep|rover | 05:03 | |
| *** saneax has quit IRC | 05:03 | |
| *** saneax has joined #tripleo | 05:03 | |
| *** bandini has joined #tripleo | 05:06 | |
| openstackgerrit | Merged openstack/tripleo-ansible stable/ussuri: Introduce an Action Plugin to manage systemd services for containers https://review.opendev.org/739998 | 05:09 |
| *** udesale has joined #tripleo | 05:13 | |
| *** saneax has quit IRC | 05:17 | |
| *** saneax has joined #tripleo | 05:19 | |
| *** zzzeek has quit IRC | 05:19 | |
| *** saneax has quit IRC | 05:20 | |
| *** zzzeek has joined #tripleo | 05:23 | |
| *** jfrancoa has joined #tripleo | 05:33 | |
| *** lmiccini has joined #tripleo | 05:33 | |
| openstackgerrit | Chandan Kumar (raukadah) proposed openstack/tripleo-ci master: Add support for authfile in buildcontainers role https://review.opendev.org/739710 | 05:36 |
| *** marios has joined #tripleo | 05:38 | |
| *** psahoo has joined #tripleo | 05:56 | |
| *** saneax has joined #tripleo | 06:15 | |
| *** pcaruana has joined #tripleo | 06:37 | |
| *** slaweq has joined #tripleo | 06:39 | |
| *** xek_ has joined #tripleo | 06:44 | |
| openstackgerrit | Merged openstack/tripleo-common stable/ussuri: Implement a new Container Images layout https://review.opendev.org/736751 | 06:48 |
| openstackgerrit | Merged openstack/python-tripleoclient stable/ussuri: Run tripleo-build-containers-ubi-8 under specific conditions https://review.opendev.org/739597 | 06:48 |
| *** sanjayu_ has joined #tripleo | 06:49 | |
| *** gouthamr_ has quit IRC | 06:50 | |
| *** saneax has quit IRC | 06:52 | |
| *** gouthamr_ has joined #tripleo | 06:52 | |
| *** sri_ has quit IRC | 06:54 | |
| *** slaweq has quit IRC | 06:55 | |
| *** sri_ has joined #tripleo | 06:56 | |
| *** csatari has quit IRC | 06:58 | |
| *** csatari has joined #tripleo | 07:01 | |
| *** gouthamr_ has quit IRC | 07:04 | |
| *** gouthamr_ has joined #tripleo | 07:05 | |
| *** skramaja has joined #tripleo | 07:06 | |
| *** amoralej|off is now known as amoralej | 07:07 | |
| *** gouthamr_ has quit IRC | 07:15 | |
| *** jcapitao has joined #tripleo | 07:20 | |
| *** gouthamr_ has joined #tripleo | 07:22 | |
| *** zbr7 is now known as zbr|ruck | 07:23 | |
| *** bogdando has joined #tripleo | 07:26 | |
| openstackgerrit | Grzegorz Grasza proposed openstack/tripleo-quickstart-extras master: Ensure that DHCP doesn't overwite DNS settings https://review.opendev.org/739905 | 07:29 |
| openstackgerrit | Grzegorz Grasza proposed openstack/tripleo-quickstart-extras master: Ensure that DHCP doesn't overwite DNS settings https://review.opendev.org/739905 | 07:32 |
| *** sanjayu__ has joined #tripleo | 07:34 | |
| *** sanjayu_ has quit IRC | 07:36 | |
| openstackgerrit | Michele Baldessari proposed openstack/tripleo-heat-templates master: WIP add locking for the restart bundles https://review.opendev.org/740173 | 07:38 |
| *** xek_ has quit IRC | 07:38 | |
| *** sanjayu__ has quit IRC | 07:40 | |
| *** soniya29 has quit IRC | 07:40 | |
| *** saneax has joined #tripleo | 07:43 | |
| *** gouthamr_ has quit IRC | 07:46 | |
| *** sanjayu_ has joined #tripleo | 07:47 | |
| *** slaweq has joined #tripleo | 07:47 | |
| *** saneax has quit IRC | 07:50 | |
| Tengu | hello folks - care to +A this quick patch? https://review.opendev.org/#/c/739748/ thanks! | 07:50 |
| *** sanjayu__ has joined #tripleo | 07:51 | |
| *** ysandeep|rover is now known as ysandeep|lunch | 07:52 | |
| *** gouthamr_ has joined #tripleo | 07:52 | |
| *** ysandeep|lunch is now known as ysandeep | 07:53 | |
| ysandeep | Folks, fyi.. undercloud minion jobs are failing in validation step , under investigation - https://bugs.launchpad.net/tripleo/+bug/1886914 | 07:53 |
| openstack | Launchpad bug 1886914 in tripleo "Undercloud minion failing validation ERROR: Heat Engine host count is 1 or less." [Critical,Triaged] | 07:53 |
| *** ysandeep is now known as ysandeep|lunch | 07:53 | |
| *** sanjayu_ has quit IRC | 07:54 | |
| *** sshnaidm has joined #tripleo | 07:55 | |
| *** sanjayu_ has joined #tripleo | 07:55 | |
| *** suuuper has joined #tripleo | 07:57 | |
| *** gouthamr_ has quit IRC | 07:57 | |
| *** sanjayu__ has quit IRC | 07:59 | |
| *** udesale_ has joined #tripleo | 08:00 | |
| *** udesale has quit IRC | 08:03 | |
| *** gouthamr_ has joined #tripleo | 08:09 | |
| *** jpich has joined #tripleo | 08:10 | |
| *** iurygregory has quit IRC | 08:10 | |
| *** udesale_ has quit IRC | 08:10 | |
| *** gfidente has joined #tripleo | 08:13 | |
| *** lucasagomes has joined #tripleo | 08:15 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/diskimage-builder master: support non-x86_64 centos7 to change DIB_DISTRIBUTION_MIRROR variable https://review.opendev.org/740183 | 08:18 |
| *** sanjayu_ has quit IRC | 08:19 | |
| openstackgerrit | yatin proposed openstack/tripleo-quickstart-extras master: [DNM] Debug c8 train iptables https://review.opendev.org/740185 | 08:22 |
| *** iurygregory has joined #tripleo | 08:24 | |
| *** maciejjozefczyk has joined #tripleo | 08:28 | |
| *** shyamb has joined #tripleo | 08:31 | |
| bhagyashris | bandini, Hi, | 08:32 |
| bhagyashris | bandini, i have reported this bug https://bugs.launchpad.net/tripleo/+bug/1886829 do you have any idea about this failure ? | 08:32 |
| openstack | Launchpad bug 1886829 in tripleo "Error: 'rabbitmqctl status | grep -F "{rabbit,"' returned 1 instead of one of [0] failing scenario004-standalone-train job" [Medium,Triaged] | 08:32 |
| openstackgerrit | Merged openstack/tripleo-common stable/ussuri: tcib/ceilometer-base: remove panko dependency https://review.opendev.org/739532 | 08:34 |
| *** ysandeep|lunch is now known as ysandeep|rover | 08:34 | |
| *** derekh has joined #tripleo | 08:36 | |
| openstackgerrit | Chandan Kumar (raukadah) proposed openstack/tripleo-ci master: Add support for authfile in buildcontainers role https://review.opendev.org/739710 | 08:38 |
| shyamb | Hi ykarel: I have updated the url names | 08:40 |
| shyamb | and some other places | 08:40 |
| shyamb | to TrilioDatamverApi instead of TrilioDmApi | 08:40 |
| shyamb | Tried overcloud deployment again | 08:40 |
| shyamb | but still, Endpoint Map of Trilio is not getting loaded | 08:41 |
| shyamb | What could be the missing piece here? | 08:41 |
| shyamb | For any third party service, if we append endpoint map entries to endpoint map yaml file, does those get loaded in environment? | 08:42 |
| *** pkopec has joined #tripleo | 08:43 | |
| openstackgerrit | Chandan Kumar (raukadah) proposed openstack/tripleo-common master: Remove the duplicated entry of designate-base https://review.opendev.org/740189 | 08:44 |
| openstackgerrit | Michele Baldessari proposed openstack/tripleo-heat-templates master: WIP add locking for the restart bundles https://review.opendev.org/740173 | 08:45 |
| *** apetrich has joined #tripleo | 08:49 | |
| *** shyamb has quit IRC | 08:58 | |
| bandini | bhagyashris: o/ I commented on the LP | 09:00 |
| bhagyashris | bandini, ack thanks ! | 09:01 |
| bandini | bhagyashris: any chance we can either hold the VM for me to inspect or enable puppet debug logs? | 09:01 |
| bhagyashris | bandini, i will check with rdo team | 09:03 |
| bhagyashris | ykarel, hey can you do this ^ | 09:03 |
| openstackgerrit | Merged openstack/tripleo-common stable/ussuri: Run standalone jobs with locally built images https://review.opendev.org/739533 | 09:08 |
| *** shyamb has joined #tripleo | 09:09 | |
| *** tosky has joined #tripleo | 09:10 | |
| ykarel | shyamb, seems u missed some steps | 09:12 |
| *** udesale has joined #tripleo | 09:12 | |
| ykarel | have u added endpoint in network/endpoints/endpoint_map and data? | 09:13 |
| *** Tengu has quit IRC | 09:13 | |
| *** saneax has joined #tripleo | 09:13 | |
| ykarel | if it's ^^, i think it should have been loaded when deploying | 09:13 |
| ykarel | bhagyashris, looking | 09:13 |
| ykarel | bhagyashris, okk i can hold the node | 09:14 |
| bhagyashris | bandini, ^^ | 09:14 |
| bhagyashris | ykarel, thank you :) | 09:15 |
| ykarel | bhagyashris, can u rerun the job | 09:15 |
| ykarel | and share patch id | 09:15 |
| openstackgerrit | Bogdan Dobrelya (bogdando) proposed openstack/tripleo-specs master: Add spec for renaming rules https://review.opendev.org/740013 | 09:15 |
| *** Tengu has joined #tripleo | 09:15 | |
| bandini | bhagyashris, ykarel: ack thanks. ping me when you have the node and I will look at it https://github.com/mbaldessari.keys | 09:15 |
| ykarel | ack | 09:15 |
| bhagyashris | bandini, will need this patch --> enable puppet debug logs | 09:17 |
| openstackgerrit | Sergii Golovatiuk proposed openstack/python-tripleoclient master: Add "yes" prompt for update/upgrades commands https://review.opendev.org/730325 | 09:17 |
| bhagyashris | bandini, or should i just rerun the job | 09:18 |
| bandini | bhagyashris: if i have the env that will be enough, i.e. i don't need anything else | 09:18 |
| bhagyashris | bandini, ykarel this is the testproject patch to test sc004 train job https://review.rdoproject.org/r/#/c/28452/ | 09:19 |
| bhagyashris | bandini, ack | 09:19 |
| *** rcernin has quit IRC | 09:23 | |
| *** dtantsur|afk is now known as dtantsur | 09:29 | |
| *** shyam89 has joined #tripleo | 09:30 | |
| *** shyamb has quit IRC | 09:30 | |
| openstackgerrit | Sergii Golovatiuk proposed openstack/tripleo-upgrade master: Add --yes into update commands to autoconfirm. https://review.opendev.org/730328 | 09:34 |
| *** xek has joined #tripleo | 09:40 | |
| ysandeep|rover | Folks, Undercloud is not uploading container to its registry with "modify_append_tag" suffix - because of this minion jobs are failing because minion node is trying to pull container with tag(that consists modify_append_tag ) from undercloud. Are you aware of any recent changes that can cause this. | 09:50 |
| ysandeep|rover | Today logs - https://16a586043fd88ddd8ac7-0aa6565d1e0c06a8a7c8c1f2b9ada9fc.ssl.cf5.rackcdn.com/724734/22/check/tripleo-ci-centos-8-containers-undercloud-minion/3fe43e7/logs/undercloud/var/log/extra/podman/podman_allinfo.log | 09:51 |
| ysandeep|rover | Old job log - https://73c3fc2c634b9a95041f-b66311f00e65e72370f624798f3cdac4.ssl.cf1.rackcdn.com/739863/1/gate/tripleo-ci-centos-8-containers-undercloud-minion/6a3a719/logs/undercloud/var/log/extra/podman/podman_allinfo.log | 09:51 |
| ysandeep|rover | chandankumar, ykarel zbr|ruck if you are aware ^^ ? | 09:52 |
| ysandeep|rover | fyi.. https://bugs.launchpad.net/tripleo/+bug/1886914 | 09:52 |
| openstack | Launchpad bug 1886914 in tripleo "Undercloud minion failing validation ERROR: Heat Engine host count is 1 or less." [Critical,Triaged] | 09:52 |
| zbr | Version: 1.6.4 -- seems like ancient to me. | 09:53 |
| ykarel | nope /me not aware about recent | 09:54 |
| ykarel | related to tag change | 09:54 |
| *** shyamb has joined #tripleo | 09:55 | |
| zbr | ignore my last commetn | 09:56 |
| ysandeep|rover | zbr, yes old job where it was working correctly also have same version | 09:56 |
| ysandeep|rover | ykarel, thanks! I will continue looking | 09:57 |
| *** shyam89 has quit IRC | 09:57 | |
| *** xek has quit IRC | 10:05 | |
| *** hjensas is now known as hjensas|afk | 10:12 | |
| *** maciejjozefczyk has quit IRC | 10:13 | |
| *** maciejjozefczyk has joined #tripleo | 10:14 | |
| openstackgerrit | Sergii Golovatiuk proposed openstack/python-tripleoclient master: Add "yes" prompt for update/upgrades commands https://review.opendev.org/730325 | 10:17 |
| openstackgerrit | Marios Andreou proposed openstack/python-tripleoclient master: Fix package manager used during undercloud packages update https://review.opendev.org/740019 | 10:25 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-ci master: Add centos-8-containerized-undercloud-upgrades job ussuri/master https://review.opendev.org/739218 | 10:35 |
| *** ysandeep|rover is now known as ysandeep|afk | 10:42 | |
| *** xek has joined #tripleo | 10:45 | |
| *** psahoo has quit IRC | 10:52 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates master: Prevent skip package fact to run on all steps. https://review.opendev.org/740240 | 10:52 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates master: Prevent ovn dbs related facts to run on each step. https://review.opendev.org/740241 | 10:52 |
| *** ysandeep|afk is now known as ysandeep|rover | 10:58 | |
| *** tkajinam has quit IRC | 11:02 | |
| *** jcapitao is now known as jcapitao_lunch | 11:06 | |
| *** jpich has quit IRC | 11:08 | |
| *** jpich has joined #tripleo | 11:09 | |
| *** psahoo has joined #tripleo | 11:12 | |
| openstackgerrit | yatin proposed openstack/tripleo-heat-templates stable/train: [DNM] enable just iptables not nftables https://review.opendev.org/739963 | 11:22 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-common master: Add a strict option to per role tasks file generation. https://review.opendev.org/739083 | 11:24 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-common master: add update and post update tasks and a extra debug step. https://review.opendev.org/740247 | 11:24 |
| *** shyamb has quit IRC | 11:25 | |
| openstackgerrit | Michele Baldessari proposed openstack/tripleo-heat-templates master: WIP add locking for the restart bundles https://review.opendev.org/740173 | 11:27 |
| *** soniya29 has joined #tripleo | 11:28 | |
| *** shyamb has joined #tripleo | 11:33 | |
| *** shyam89 has joined #tripleo | 11:50 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-ansible master: container_systemd: improve logging https://review.opendev.org/740252 | 11:52 |
| *** shyamb has quit IRC | 11:53 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-ansible master: tripleo_states: add missing space https://review.opendev.org/740253 | 11:54 |
| ysandeep|rover | EmilienM, cloudnull whenever you have time could you please take a look at https://bugs.launchpad.net/tripleo/+bug/1886914 | 11:55 |
| openstack | Launchpad bug 1886914 in tripleo "Undercloud minion failing validation ERROR: Heat Engine host count is 1 or less." [Critical,Triaged] | 11:55 |
| ysandeep|rover | In master branch - Undercloud is not uploading container to its registry with "modify_append_tag" suffix - because of this minion jobs are failing because minion node is trying to pull container with tag(that consists modify_append_tag ) from undercloud. Are you aware of any recent changes that can cause this. | 11:55 |
| *** rh-jelabarre has joined #tripleo | 11:57 | |
| *** shyam89 has quit IRC | 12:02 | |
| *** jcapitao_lunch is now known as jcapitao | 12:02 | |
| *** bhagyashris is now known as bhagyashris|brb | 12:04 | |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-ansible master: Fix introspection restart with unlocking node https://review.opendev.org/740256 | 12:05 |
| *** zbr_ has quit IRC | 12:05 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-common stable/ussuri: (TCIB) Remove validation related things from mistral container https://review.opendev.org/740257 | 12:06 |
| *** amoralej is now known as amoralej|lunch | 12:07 | |
| *** xek has quit IRC | 12:14 | |
| *** rlandy has joined #tripleo | 12:23 | |
| *** raildo has joined #tripleo | 12:23 | |
| *** rlandy is now known as rlandy|ruck2 | 12:24 | |
| openstackgerrit | Cédric Jeanneret (Tengu) proposed openstack/tripleo-operator-ansible master: Add missing "-o pipefail" to generated scripts https://review.opendev.org/740260 | 12:30 |
| *** psahoo_ has joined #tripleo | 12:31 | |
| *** psahoo has quit IRC | 12:35 | |
| openstackgerrit | mathieu bultel proposed openstack/tripleo-validations master: Add tripleo-validation.py as script in setup.cfg https://review.opendev.org/740261 | 12:35 |
| *** ysandeep|rover is now known as ysandeep|afk | 12:37 | |
| *** bhagyashris|brb is now known as bhagyashris | 12:37 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: CI test - never merge https://review.opendev.org/740264 | 12:40 |
| cloudnull | Good morning. | 12:41 |
| weshay_ruck | EmilienM, cloudnull anyone other than Alex have experience w/ minion? https://bugs.launchpad.net/tripleo/+bug/1886914 | 12:41 |
| openstack | Launchpad bug 1886914 in tripleo "Undercloud minion failing validation ERROR: Heat Engine host count is 1 or less." [Critical,Triaged] | 12:41 |
| EmilienM | weshay_ruck: we're debugging it | 12:41 |
| EmilienM | weshay_ruck: with Tengu | 12:41 |
| *** chem has quit IRC | 12:42 | |
| EmilienM | there are multiple problems | 12:42 |
| weshay_ruck | danka | 12:42 |
| weshay_ruck | ysandeep|afk, ^ | 12:42 |
| *** raildo has quit IRC | 12:47 | |
| *** raildo has joined #tripleo | 12:50 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-common master: Add update and post update tasks to step generated file. https://review.opendev.org/740247 | 12:53 |
| EmilienM | weshay_ruck, Tengu : for some reason (i didn't find it yet why) but the undercloud isn't pulling -updated container images that were built in modify-image role | 12:54 |
| openstackgerrit | Grace Byers proposed openstack/tripleo-operator-ansible master: This is a basic playbook to install undercloud minions on an already provisioned system. https://review.opendev.org/740268 | 12:55 |
| cloudnull | EmilienM anything I can help with ? | 12:56 |
| cloudnull | -cc Tengu | 12:56 |
| EmilienM | cloudnull: debugging 1886914 | 12:56 |
| cloudnull | di | 12:57 |
| cloudnull | ** si | 12:57 |
| *** ysandeep|afk is now known as ysandeep | 12:58 | |
| cloudnull | do we have a tmate session / meet to debug that ? | 12:58 |
| EmilienM | no i'm reading logs now | 13:00 |
| EmilienM | and setting up a reproducer | 13:00 |
| *** morazi has quit IRC | 13:00 | |
| cloudnull | k | 13:00 |
| EmilienM | there are 2 issues : | 13:01 |
| EmilienM | 1) Ansible didn't fail when the minion failed to be deployed (tripleoclient RC is 1) | 13:01 |
| EmilienM | 2) Undercloud main node wasn't deployed with -updated containers | 13:02 |
| *** morazi has joined #tripleo | 13:06 | |
| EmilienM | Tengu, cloudnull : The container issue is because yum_update playbook didn't run in tripleo container image prepare, so the -updated images were not created. | 13:07 |
| EmilienM | I'm looking at why it doesn't run for that job anymore | 13:07 |
| EmilienM | (i checked and it's running for the other jobs) | 13:07 |
| *** ekultails has joined #tripleo | 13:08 | |
| *** hjensas|afk is now known as hjensas | 13:09 | |
| EmilienM | cloudnull: Tengu did https://review.opendev.org/740260 in case we didn't catch a pipe fail | 13:09 |
| EmilienM | but we don't have pipes.. we only have redirections | 13:09 |
| openstackgerrit | Lance Bragstad proposed openstack/tripleo-heat-templates master: Update undercloud TLS template with propery keytab group https://review.opendev.org/740275 | 13:09 |
| *** TrevorV has joined #tripleo | 13:11 | |
| *** amoralej|lunch is now known as amoralej | 13:12 | |
| openstackgerrit | Lance Bragstad proposed openstack/tripleo-heat-templates master: Update undercloud TLS template with proper keytab group https://review.opendev.org/740275 | 13:14 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Revert "Collapse deploy steps" https://review.opendev.org/740277 | 13:14 |
| *** xek has joined #tripleo | 13:17 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Revert "Collapse deploy steps" https://review.opendev.org/740277 | 13:17 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Revert "Collapse host prep tasks" https://review.opendev.org/740280 | 13:18 |
| *** Goneri has joined #tripleo | 13:20 | |
| *** soniya29 has quit IRC | 13:24 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Revert "Collapse host prep tasks" https://review.opendev.org/740280 | 13:26 |
| *** skramaja has quit IRC | 13:29 | |
| EmilienM | weshay_ruck: when did we update new images? | 13:32 |
| EmilienM | I think I know why the images weren't modified | 13:32 |
| EmilienM | we use modify_only_with_labels | 13:32 |
| EmilienM | and it would only update our kolla images | 13:32 |
| EmilienM | but if we switched to the new ones, they wouldn't be updated | 13:32 |
| EmilienM | which is why it started to fail yesterday | 13:32 |
| weshay_ruck | I think yesterday | 13:32 |
| EmilienM | HA | 13:32 |
| EmilienM | I think this is it | 13:33 |
| EmilienM | (not 100% sure yet) | 13:33 |
| EmilienM | https://opendev.org/openstack/kolla/src/branch/master/docker/base/Dockerfile.j2#L21 | 13:35 |
| EmilienM | yeah | 13:35 |
| EmilienM | I forgot about that one | 13:35 |
| EmilienM | https://opendev.org/openstack/tripleo-common/src/branch/master/tripleo_common/image/kolla_builder.py#L342-L344 | 13:36 |
| EmilienM | so we need to add a new tag to our images, rebuild them and allow this tag to be filtered | 13:36 |
| EmilienM | cloudnull, weshay_ruck, Tengu : I updated launchpad with the 3 issues: https://bugs.launchpad.net/tripleo/+bug/1886914/comments/10 | 13:41 |
| openstack | Launchpad bug 1886914 in tripleo "Undercloud minion failing validation ERROR: Heat Engine host count is 1 or less." [Critical,Triaged] | 13:41 |
| Tengu | EmilienM: will check - sorry, was in an mtg for the past hour or so | 13:42 |
| weshay_ruck | aye.. thanks EmilienM also in a mtg.. chandankumar and I will look in a bit | 13:42 |
| EmilienM | busy people | 13:43 |
| Tengu | yeah.. | 13:43 |
| openstackgerrit | Merged openstack/tripleo-validations master: Add "mock" in test-requirements and lower-constraints https://review.opendev.org/739748 | 13:45 |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-validations master: Ensure virtualenv is installed. https://review.opendev.org/739476 | 13:46 |
| openstackgerrit | Miguel proposed openstack/tripleo-common stable/queens: Add python-pyngus package to ceilometer notification image https://review.opendev.org/740293 | 13:47 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-common master: Add "tcib_version" LABEL into all our images https://review.opendev.org/740296 | 13:50 |
| weshay_ruck | bhagyashris, ping sbaker or email re: paunch and c8 | 13:52 |
| weshay_ruck | c8 train | 13:53 |
| bhagyashris | weshay_ruck, ack | 13:53 |
| EmilienM | weshay_ruck: what happens with paunch? Steve isn't working on Paunch anymore; | 13:54 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart-extras master: containers: add tcib_version label to authorized images to be updated https://review.opendev.org/740298 | 13:57 |
| EmilienM | weshay_ruck: ^ this should fix part of the issue | 13:59 |
| EmilienM | we'll need container rebuild | 13:59 |
| EmilienM | and it should bring the minion job back | 13:59 |
| *** saneax has quit IRC | 14:00 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed openstack/ansible-role-collect-logs master: WIP: logstash https://review.opendev.org/740299 | 14:03 |
| *** ratailor has quit IRC | 14:04 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-ci master: Include train for Centos8 Jobs https://review.opendev.org/738375 | 14:06 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates master: Complete refactor update: use step file, add information for user. https://review.opendev.org/740302 | 14:09 |
| sshnaidm | EmilienM, cloudnull please https://review.opendev.org/#/c/740256 something I messed up | 14:10 |
| EmilienM | sshnaidm: have you already looked at the idempotency issue with podman_container/volumes? | 14:10 |
| sshnaidm | EmilienM, yep, will a submit a patch soon | 14:10 |
| EmilienM | sshnaidm: awesome | 14:10 |
| EmilienM | sshnaidm: +2 | 14:10 |
| cloudnull | done | 14:10 |
| *** abdysn has quit IRC | 14:24 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-ci master: Fix issue with scenario007 multinode job https://review.opendev.org/736468 | 14:25 |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-tripleo stable/train: HA: fix rabbitmq readiness check for rabbitmq-server 3.8 https://review.opendev.org/740307 | 14:27 |
| *** lbragstad_ has joined #tripleo | 14:29 | |
| *** lbragstad has quit IRC | 14:29 | |
| openstackgerrit | Merged openstack/tripleo-ansible master: Fix introspection restart with unlocking node https://review.opendev.org/740256 | 14:31 |
| openstackgerrit | Merged openstack/puppet-tripleo stable/train: Enable to modify params of logrotate-crond.conf https://review.opendev.org/739091 | 14:31 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed openstack/ansible-role-collect-logs master: WIP: logstash https://review.opendev.org/740299 | 14:31 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-upgrade master: Remove --update-plan-only from UPGRADE converge. https://review.opendev.org/731659 | 14:38 |
| openstackgerrit | Bogdan Dobrelya (bogdando) proposed openstack/python-tripleoclient master: Log error when fail_on_rc disabled https://review.opendev.org/740312 | 14:40 |
| openstackgerrit | amolkahat proposed openstack/tripleo-quickstart master: [DNM] Testing python-tempestconf octavia chagnes https://review.opendev.org/740314 | 14:42 |
| EmilienM | cloudnull: do you remember why you added tcib_meta? https://opendev.org/openstack/python-tripleoclient/src/branch/master/tripleoclient/v2/tripleo_container_image.py#L458 | 14:45 |
| EmilienM | it doesn't seem to be used anywhere | 14:45 |
| cloudnull | there was a reason. i dont remember it though :D | 14:46 |
| cloudnull | is that causing issues? | 14:46 |
| EmilienM | no | 14:46 |
| openstackgerrit | Merged openstack/tripleo-operator-ansible master: Add missing "-o pipefail" to generated scripts https://review.opendev.org/740260 | 14:47 |
| *** soniya29 has joined #tripleo | 14:47 | |
| cloudnull | looking across tripleo-common and tripleoclient, it doesn't seem to be used anywhere. so maybe not needed any longer? | 14:48 |
| *** soniya29 has quit IRC | 14:51 | |
| ykarel | bandini, hi | 14:52 |
| *** bhagyashris is now known as bhagyashris|dinn | 14:52 | |
| ykarel | bandini, can u check https://review.opendev.org/#/c/739963/ | 14:52 |
| ykarel | u added some fix earlier for nftables | 14:53 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-common master: TCIB: add "kolla_version" for backward compatibility https://review.opendev.org/740296 | 14:54 |
| EmilienM | cloudnull, weshay_ruck : I think the quickest way to get the minion job working is to merge https://review.opendev.org/740296 and rebuild TCIB images | 14:54 |
| openstackgerrit | amolkahat proposed openstack/tripleo-quickstart master: Removing profile overrides from featureset063 https://review.opendev.org/740314 | 14:54 |
| weshay_ruck | k | 14:54 |
| openstackgerrit | Michele Baldessari proposed openstack/tripleo-heat-templates master: WIP add locking for the restart bundles https://review.opendev.org/740173 | 14:56 |
| *** mnasiadka has left #tripleo | 14:59 | |
| EmilienM | cloudnull: could you please file a new launchpad for this ansible rc thing? | 15:07 |
| EmilienM | to separate it from https://bugs.launchpad.net/tripleo/+bug/1886914 | 15:07 |
| openstack | Launchpad bug 1886914 in tripleo "Undercloud minion failing validation ERROR: Heat Engine host count is 1 or less." [Critical,Triaged] | 15:07 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed openstack/ansible-role-collect-logs master: WIP: Correct logstash.txt generation https://review.opendev.org/740299 | 15:07 |
| cloudnull | sure | 15:07 |
| cloudnull | https://bugs.launchpad.net/tripleo/+bug/1886971 | 15:16 |
| openstack | Launchpad bug 1886971 in tripleo "Refactor _standalone_deploy " [Undecided,New] | 15:16 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-common master: Relax filter_images_with_labels() and re-add kolla_version https://review.opendev.org/740296 | 15:25 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates master: FFU support for ceph_nfs https://review.opendev.org/739172 | 15:27 |
| *** ysandeep is now known as ysandeep|away | 15:30 | |
| sshnaidm | rlandy|ruck2, worth to ask hjensas and bfournie if they need this reproducer | 15:30 |
| sshnaidm | I think they mostly were dealing with https://bugs.launchpad.net/tripleo/+bug/1879472 | 15:31 |
| openstack | Launchpad bug 1879472 in tripleo "OVB overcloud deploy fails sporadically with "not enough free physical ports" error" [Critical,Triaged] | 15:31 |
| rlandy|ruck2 | hjensas: bfournie: ^^ we have nodes held showing the deploy issues if you want them. | 15:32 |
| rlandy|ruck2 | sshnaidm: on a held node, have you tried to rerun deploy again? | 15:33 |
| rlandy|ruck2 | just asking if it's worth it | 15:33 |
| *** ykarel is now known as ykarel|away | 15:33 | |
| hjensas | rlandy|ruck2: I can take a look. Did we see issue again with https://review.opendev.org/739970 ? Maby we can run a job to excercice that change several times? | 15:34 |
| sshnaidm | rlandy|ruck2, I didn't | 15:35 |
| rlandy|ruck2 | hjensas: I didn;t try https://review.opendev.org/#/c/739970/ | 15:35 |
| rlandy|ruck2 | I can do on the next run | 15:35 |
| rlandy|ruck2 | hjensas: if you send me your keys I can add you to the zuul node or tmate you in - whatever you prefer | 15:35 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-upgrade master: Do not fail If stack has no controllers https://review.opendev.org/732072 | 15:37 |
| rlandy|ruck2 | running again with https://review.opendev.org/#/c/739970/ | 15:38 |
| openstackgerrit | Douglas Mendizábal proposed openstack/tripleo-heat-templates master: Update Nova VNC puppet variables https://review.opendev.org/723920 | 15:38 |
| hjensas | rlandy|ruck2: https://github.com/hjensas.keys | 15:40 |
| rlandy|ruck2 | hjensas: zuul@10.0.108.218 | 15:42 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart-extras master: containers: add tcib_managed label to authorized images to be updated https://review.opendev.org/740298 | 15:44 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-common master: tcib: rename kolla_version to tcib_managed https://review.opendev.org/740323 | 15:45 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-ci master: set minion job to nv for now https://review.opendev.org/740324 | 15:46 |
| weshay_ruck | EmilienM, rlandy|ruck2 ^ | 15:46 |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-ansible master: Update podman_container with better idempotency https://review.opendev.org/740325 | 15:47 |
| sshnaidm | EmilienM, ^ | 15:47 |
| EmilienM | weshay_ruck: good, I starred it in case we forgot to revert | 15:47 |
| EmilienM | sshnaidm: awesome, let me review it | 15:47 |
| *** bhagyashris|dinn is now known as bhagyashris | 15:49 | |
| EmilienM | sshnaidm: lgtm | 15:51 |
| *** bogdando has quit IRC | 15:51 | |
| EmilienM | sshnaidm: no molecule tests? | 15:51 |
| sshnaidm | EmilienM, yeah, I think I'll update it in a different patch | 15:51 |
| sshnaidm | need to port them from collections, but it's tested there | 15:51 |
| EmilienM | ok | 15:52 |
| cloudnull | do we have the tests within a collection ? | 15:52 |
| sshnaidm | EmilienM, https://github.com/containers/ansible-podman-collections/pull/78/commits/0767d49a598efaeec2ba44f7da21376c4c1e3e79 | 15:52 |
| sshnaidm | cloudnull, ^ | 15:52 |
| cloudnull | if so, no need to recreate them here? | 15:52 |
| sshnaidm | cloudnull, the problem with tests is a podman version | 15:52 |
| cloudnull | just an integration test would be all that we should need (assuming the coverage in the official collection is good) ? | 15:53 |
| sshnaidm | collection tests atm don't cover all podman versions | 15:53 |
| *** ykarel|away has quit IRC | 15:53 | |
| cloudnull | k | 15:53 |
| EmilienM | cloudnull: looking at https://bugs.launchpad.net/bugs/1886971 - so we would have this bug for some time and didn't catch it? | 15:54 |
| openstack | Launchpad bug 1886971 in tripleo "Refactor _standalone_deploy " [High,Triaged] | 15:54 |
| sshnaidm | when I start building podman there and run in matrix of versions, it will be fine | 15:54 |
| cloudnull | EmilienM yes. the issue has been there, even before we moved to ansible-runner. | 15:54 |
| EmilienM | cloudnull: how didn't we catch it before? | 15:55 |
| hjensas | rlandy|ruck2: in this job one of 4 BM nodes timed out when being provisioned. Do we have the console log of the virtual BM's? | 15:55 |
| EmilienM | cloudnull: how do you explain that tripleoclient returned 1 and ansible didn't fail? | 15:56 |
| rlandy|ruck2 | hjensas: you can still access the console and logs ... sending you creds | 15:57 |
| cloudnull | ansible is specifically told not to fail | 15:58 |
| EmilienM | cloudnull: but the client failed | 15:58 |
| EmilienM | i'm taling about tripleo operator ansible | 15:58 |
| EmilienM | talking* | 15:58 |
| EmilienM | which ran "openstack undercloud minion install | 15:58 |
| EmilienM | " | 15:58 |
| cloudnull | oh, that I dont know. | 15:58 |
| EmilienM | tripleoclient reported RC 1 | 15:59 |
| EmilienM | cloudnull: that's the core issue | 15:59 |
| EmilienM | cloudnull: https://16a586043fd88ddd8ac7-0aa6565d1e0c06a8a7c8c1f2b9ada9fc.ssl.cf5.rackcdn.com/724734/22/check/tripleo-ci-centos-8-containers-undercloud-minion/3fe43e7/job-output.txt | 15:59 |
| EmilienM | [tripleo.operator.tripleo_undercloud_minion_install : undercloud minion install] | 15:59 |
| EmilienM | changed: [subnode-1] | 15:59 |
| EmilienM | cloudnull: that's what we're debugging :D | 15:59 |
| cloudnull | however, the client is running all playbooks in the set and then failing when the RC != 0 | 15:59 |
| *** marios has quit IRC | 16:00 | |
| cloudnull | so while I dont know why the operator-ansible parts didn't raise, I do know that the client is compounding the issue | 16:00 |
| EmilienM | cloudnull: https://16a586043fd88ddd8ac7-0aa6565d1e0c06a8a7c8c1f2b9ada9fc.ssl.cf5.rackcdn.com/724734/22/check/tripleo-ci-centos-8-containers-undercloud-minion/3fe43e7/logs/subnode-1/home/zuul/minion_install.log | 16:01 |
| EmilienM | cloudnull: what issue is compounded? | 16:06 |
| *** udesale_ has joined #tripleo | 16:08 | |
| *** jpich has quit IRC | 16:08 | |
| cloudnull | to start the RC shouldn't be 1, it should 2 (ansible raises). also it looks like it continued to execute a set of playbooks even after the minion install failed. | 16:09 |
| *** udesale has quit IRC | 16:10 | |
| *** d34dh0r53 has quit IRC | 16:10 | |
| EmilienM | cloudnull: where do you see that? | 16:10 |
| cloudnull | which is consistent with https://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/v1/tripleo_deploy.py#L1309 | 16:11 |
| cloudnull | the RC is clearly seen in the logs. | 16:11 |
| *** amoralej is now known as amoralej|off | 16:12 | |
| EmilienM | I wonder what other "operation" would be run after the deploy | 16:13 |
| cloudnull | https://pasted.tech/pastes/82839937a6af917612c103b2d454d006c4a139d2 | 16:13 |
| EmilienM | this isn't an operation | 16:13 |
| EmilienM | part of tripleoclient | 16:13 |
| EmilienM | this is a one-off script run *after* running "openstack undercloud minion install" | 16:13 |
| EmilienM | and btw if the operation fails, we leave the loop: https://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/v1/tripleo_deploy.py#L1326-L1327 | 16:14 |
| cloudnull | and then go into the finally, which may or may not reraise | 16:14 |
| EmilienM | I disagree, it raised | 16:15 |
| EmilienM | https://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/v1/tripleo_deploy.py#L1368 | 16:15 |
| EmilienM | raise exceptions.DeploymentError('Deployment failed.') | 16:15 |
| EmilienM | https://16a586043fd88ddd8ac7-0aa6565d1e0c06a8a7c8c1f2b9ada9fc.ssl.cf5.rackcdn.com/724734/22/check/tripleo-ci-centos-8-containers-undercloud-minion/3fe43e7/logs/subnode-1/home/zuul/minion_install.log | 16:15 |
| cloudnull | and it didnt stop | 16:15 |
| EmilienM | we can see it in the logs | 16:15 |
| EmilienM | 2020-07-09 04:17:22.085 40274 ERROR openstack [ ] Deployment failed. | 16:16 |
| EmilienM | (however rc isn't returned) | 16:16 |
| cloudnull | I see that , I also see it continued executing finally raising rc1 | 16:16 |
| cloudnull | not rc 2 | 16:16 |
| cloudnull | maybe its not the client? but if that's the case then something else in path is trapping or ignoring the error | 16:17 |
| EmilienM | it should not matter if it's 1 2 or 128 | 16:18 |
| EmilienM | https://github.com/openstack/python-tripleoclient/blob/111f3ca79f72e99be7becdbf494212d41603a8d5/tripleoclient/v1/tripleo_deploy.py#L1411 | 16:18 |
| EmilienM | we check if not 0, we raise | 16:18 |
| cloudnull | there would be a stacktrace , but there's not one | 16:18 |
| EmilienM | there is one | 16:18 |
| EmilienM | 2020-07-09 04:17:22.085 40274 ERROR openstack [ ] Deployment failed. | 16:18 |
| cloudnull | that's not a stacktrace | 16:18 |
| EmilienM | it's https://github.com/openstack/python-tripleoclient/blob/111f3ca79f72e99be7becdbf494212d41603a8d5/tripleoclient/v1/tripleo_deploy.py#L1412 | 16:18 |
| EmilienM | yeah beacuse we only raise "msg" I think | 16:19 |
| cloudnull | I think that log is https://github.com/openstack/python-tripleoclient/blob/111f3ca79f72e99be7becdbf494212d41603a8d5/tripleoclient/v1/tripleo_deploy.py#L1365-L1367 | 16:19 |
| EmilienM | no | 16:19 |
| EmilienM | or maybe | 16:19 |
| EmilienM | there is the "Deployment Failed!" thing | 16:20 |
| *** jcapitao has quit IRC | 16:20 | |
| cloudnull | that's the only ERROR log message in that method. | 16:20 |
| EmilienM | h we get the DEPLOY_FAILURE_MESSAGE | 16:21 |
| EmilienM | so we reached https://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/v1/tripleo_deploy.py#L1365-L1368 | 16:21 |
| cloudnull | it looks like it | 16:21 |
| cloudnull | but there's no stack | 16:21 |
| cloudnull | maybe its in a different log message? | 16:22 |
| EmilienM | anyway, it means that tripleo_deploy does its job, returns an error | 16:22 |
| EmilienM | cloudnull: would we need to re-raise in finally here? https://github.com/openstack/python-tripleoclient/blob/111f3ca79f72e99be7becdbf494212d41603a8d5/tripleoclient/v1/tripleo_deploy.py#L1419 | 16:23 |
| *** jfrancoa has quit IRC | 16:23 | |
| cloudnull | yea. the execution should stop, the finally when failed stuff should go into the exception block. finally should just perform cleanup regardless of the RC information | 16:24 |
| cloudnull | then we wouldn't need to reraise or worry about any of this | 16:25 |
| cloudnull | that said, if the minion validataion isn't part of the operations being executed, then idk what's the root cause of the failure. | 16:27 |
| *** udesale_ has quit IRC | 16:27 | |
| cloudnull | maybe something in quickstart? | 16:27 |
| EmilienM | cloudnull: I wonder if https://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/v1/tripleo_deploy.py#L1404 should be indented one tab more | 16:29 |
| EmilienM | so we always return rc | 16:29 |
| *** d34dh0r53 has joined #tripleo | 16:30 | |
| EmilienM | ah nevermind it's in the finally already | 16:31 |
| EmilienM | cloudnull: I'll propose a patch | 16:34 |
| cloudnull | https://pasted.tech/pastes/338c5a727b905ac73ebda8fc0ee83745ece29fba | 16:34 |
| cloudnull | it looks like the return would be Null in the event of a failure | 16:35 |
| cloudnull | so maybe that's the issue? | 16:35 |
| openstackgerrit | Luke Short proposed openstack/tripleo-heat-templates master: Expose the Apache WSGI timeout https://review.opendev.org/738925 | 16:35 |
| cloudnull | https://pasted.tech/pastes/c9ed2cc0ccb65b86c4bb65cec7f216bb7bbe86c6 - another test | 16:37 |
| cloudnull | if the finally doens't reraise we don't actually trap an exception | 16:38 |
| *** lucasagomes has quit IRC | 16:39 | |
| openstackgerrit | Luke Short proposed openstack/tripleo-heat-templates master: Expose the Apache WSGI timeout https://review.opendev.org/738925 | 16:39 |
| *** jfrancoa has joined #tripleo | 16:40 | |
| openstackgerrit | Emilien Macchi proposed openstack/python-tripleoclient master: tripleo_deploy: improve exceptions catching https://review.opendev.org/740332 | 16:43 |
| EmilienM | cloudnull: ^ wdyt? | 16:44 |
| cloudnull | Looking | 16:44 |
| openstackgerrit | Emilien Macchi proposed openstack/python-tripleoclient master: tripleo_deploy: improve exceptions catching https://review.opendev.org/740332 | 16:44 |
| *** andreaf has quit IRC | 16:48 | |
| *** m1p_ has joined #tripleo | 16:48 | |
| *** simondodsley_ has joined #tripleo | 16:48 | |
| *** mwhahaha_ has joined #tripleo | 16:48 | |
| *** ChanServ sets mode: +v mwhahaha_ | 16:48 | |
| cloudnull | LGTM. if that successfully traps the error and raises the expected failure +1 | 16:48 |
| cloudnull | Do we need to clear the cmd line on failure ? | 16:49 |
| cloudnull | maybe we just remove the try/finally their and allow the calling method to raise? | 16:49 |
| *** dasp_ has joined #tripleo | 16:49 | |
| *** simondodsley has quit IRC | 16:55 | |
| *** mwhahaha has quit IRC | 16:55 | |
| *** m1p has quit IRC | 16:55 | |
| *** dasp has quit IRC | 16:55 | |
| *** hberaud has quit IRC | 16:55 | |
| *** gchamoul has quit IRC | 16:55 | |
| EmilienM | cloudnull: yeah because ansible/podman run, so we always need to clear cmd line I think | 16:55 |
| *** m1p_ is now known as m1p | 16:55 | |
| *** mwhahaha_ is now known as mwhahaha | 16:55 | |
| *** simondodsley_ is now known as simondodsley | 16:55 | |
| *** gchamoul has joined #tripleo | 16:56 | |
| *** lmiccini has quit IRC | 16:57 | |
| *** hberaud has joined #tripleo | 16:58 | |
| *** derekh has quit IRC | 16:59 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed openstack/ansible-role-collect-logs master: WIP: Correct logstash.txt generation https://review.opendev.org/740299 | 17:00 |
| cloudnull | ok | 17:05 |
| cloudnull | then +2 | 17:05 |
| cloudnull | EmilienM where is the ansible-operator-minion-install called ? | 17:05 |
| cloudnull | from quickstart? | 17:06 |
| EmilienM | extras | 17:08 |
| EmilienM | let me pull it for you | 17:08 |
| EmilienM | cloudnull: https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/undercloud-minion-deploy/tasks/main.yml#L24 | 17:08 |
| cloudnull | oic | 17:10 |
| *** sshnaidm is now known as sshnaidm|afk | 17:12 | |
| cloudnull | EmilienM https://github.com/openstack/tripleo-operator-ansible/blob/master/roles/tripleo_undercloud_minion_install/tasks/main.yml#L38-L49 | 17:15 |
| cloudnull | it looks like that's never executed | 17:15 |
| cloudnull | are we setting https://github.com/openstack/tripleo-operator-ansible/blob/master/roles/tripleo_undercloud_minion_install/tasks/main.yml#L48-L49 somewhere? | 17:15 |
| cloudnull | and not running the generated script before trying to execute the validation task? | 17:16 |
| EmilienM | what? | 17:16 |
| EmilienM | it is executed? | 17:16 |
| EmilienM | https://16a586043fd88ddd8ac7-0aa6565d1e0c06a8a7c8c1f2b9ada9fc.ssl.cf5.rackcdn.com/724734/22/check/tripleo-ci-centos-8-containers-undercloud-minion/3fe43e7/job-output.txt | 17:16 |
| EmilienM | 2020-07-09 04:12:54.118652 | primary | TASK [tripleo.operator.tripleo_undercloud_minion_install : Write reproducer script] *** | 17:16 |
| EmilienM | 2020-07-09 04:12:54.118776 | primary | Thursday 09 July 2020 04:12:54 +0000 (0:00:00.059) 0:47:05.630 ********* | 17:16 |
| EmilienM | 2020-07-09 04:12:54.685879 | primary | changed: [subnode-1] | 17:16 |
| cloudnull | 2020-07-09 04:17:28.927091 | primary | | 17:17 |
| cloudnull | 2020-07-09 04:17:28.927127 | primary | TASK [undercloud-minion-deploy : Install the minion] *************************** | 17:17 |
| cloudnull | 2020-07-09 04:17:28.927180 | primary | Thursday 09 July 2020 04:17:28 +0000 (0:00:00.046) 0:51:40.439 ********* | 17:17 |
| cloudnull | 2020-07-09 04:17:28.952740 | primary | skipping: [subnode-1] | 17:17 |
| cloudnull | 2020-07-09 04:17:28.957846 | primary | | 17:17 |
| cloudnull | 2020-07-09 04:17:28.957883 | primary | PLAY [Validate minion] ********************************************************* | 17:17 |
| cloudnull | 2020-07-09 04:17:28.978869 | primary | | 17:17 |
| cloudnull | its skipped | 17:18 |
| EmilienM | it's the legacy way of running minions | 17:18 |
| EmilienM | without tripleo operators ... | 17:18 |
| cloudnull | oh. | 17:18 |
| EmilienM | https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/undercloud-minion-deploy/tasks/main.yml#L29-L34 | 17:18 |
| EmilienM | when: release not in ['master'] | 17:18 |
| cloudnull | ah. https://github.com/openstack/tripleo-operator-ansible/blob/master/plugins/modules/tripleo_shell_script.py and thats whats running it | 17:19 |
| * cloudnull i follow | 17:19 | |
| cloudnull | nvmd - https://github.com/openstack/tripleo-operator-ansible/blob/master/plugins/modules/tripleo_shell_script.py#L113-L131 - looks like it only writes a file | 17:20 |
| EmilienM | yes, it writes the shell script | 17:20 |
| EmilienM | which is later executed | 17:20 |
| cloudnull | where is the file executed? | 17:21 |
| *** lbragstad_ is now known as lbragstad | 17:21 | |
| EmilienM | this is how tripleo-operator-ansible works | 17:21 |
| EmilienM | we wrap a cli, write a script and run the script | 17:21 |
| lbragstad | o/ hey folks - i'm curious if anyone would be interested in taking a look at the series starting at https://review.opendev.org/#/c/729076/ | 17:21 |
| cloudnull | I get that, but where is the script run ? | 17:21 |
| cloudnull | looking at the logs https://pasted.tech/pastes/b5ca559308b665480a64046851d295dd5480b597 | 17:21 |
| cloudnull | it seems like it generates the script, then we validate it ? | 17:22 |
| cloudnull | I see - 2020-07-09 04:12:54.705579 | primary | TASK [tripleo.operator.tripleo_undercloud_minion_install : undercloud minion install] *** | 17:23 |
| cloudnull | but it seems unlikely that the minion install completed in <5 min and didn't fail? | 17:23 |
| EmilienM | why not? | 17:24 |
| EmilienM | https://16a586043fd88ddd8ac7-0aa6565d1e0c06a8a7c8c1f2b9ada9fc.ssl.cf5.rackcdn.com/724734/22/check/tripleo-ci-centos-8-containers-undercloud-minion/3fe43e7/logs/subnode-1/home/zuul/undercloud_minion_install.log | 17:24 |
| EmilienM | stack created at 04:13:05.203 | 17:24 |
| EmilienM | ansible done (and failed) at 04:17:20.805954 | 17:25 |
| EmilienM | it's 4min and 15s | 17:25 |
| EmilienM | it failed to pull containers, which is why it failed early | 17:25 |
| cloudnull | the task didn't fail, it "changed" | 17:26 |
| EmilienM | which task? | 17:26 |
| EmilienM | tripleo_undercloud_minion_install ? | 17:26 |
| cloudnull | 2020-07-09 04:12:54.705579 | primary | TASK [tripleo.operator.tripleo_undercloud_minion_install : undercloud minion install] *** | 17:26 |
| EmilienM | yes, probably due to tripleoclient | 17:27 |
| EmilienM | my patch etc? | 17:27 |
| cloudnull | maybe. | 17:28 |
| * cloudnull trying to look at this from another direction. | 17:28 | |
| cloudnull | maybe the script isn't failing ? | 17:28 |
| *** slaweq has quit IRC | 17:28 | |
| EmilienM | my theory is that it isn't failing because the finally didn't raise the error and it returned 0 | 17:29 |
| cloudnull | ++ | 17:29 |
| cloudnull | I tend to agree | 17:29 |
| cloudnull | just trying to make sure I cross all my I's and dot all my T's | 17:29 |
| *** psahoo_ has quit IRC | 17:30 | |
| EmilienM | cloudnull: i'll let you investigate further, I need to go soon and I've been on this since 7am | 17:30 |
| cloudnull | +1 | 17:30 |
| openstackgerrit | Sergii Golovatiuk proposed openstack/python-tripleoclient master: Add "yes" prompt for update/upgrades commands https://review.opendev.org/730325 | 17:30 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-ansible master: container_status: "exec" checks status https://review.opendev.org/739839 | 17:41 |
| openstackgerrit | Sergii Golovatiuk proposed openstack/python-tripleoclient master: Add "yes" prompt for update/upgrades commands https://review.opendev.org/730325 | 17:41 |
| *** dtantsur is now known as dtantsur|afk | 17:45 | |
| *** bandini has quit IRC | 17:48 | |
| openstackgerrit | Sergii Golovatiuk proposed openstack/tripleo-upgrade master: Add --yes into update commands to autoconfirm. https://review.opendev.org/730328 | 18:00 |
| *** morazi has quit IRC | 18:26 | |
| *** pkopec has quit IRC | 18:29 | |
| openstackgerrit | Michele Baldessari proposed openstack/tripleo-heat-templates master: WIP add locking for the restart bundles https://review.opendev.org/740173 | 18:29 |
| *** EmilienM is now known as EmilienM|PTO | 18:38 | |
| *** morazi has joined #tripleo | 18:44 | |
| *** morazi has quit IRC | 18:47 | |
| *** morazi has joined #tripleo | 18:47 | |
| *** morazi has quit IRC | 18:49 | |
| *** morazi has joined #tripleo | 18:50 | |
| *** xek has quit IRC | 18:57 | |
| *** rlandy|ruck2 is now known as rlandy|brb | 19:15 | |
| *** tosky has quit IRC | 19:26 | |
| *** rlandy|brb is now known as rlandy | 19:36 | |
| *** morazi has quit IRC | 19:46 | |
| *** morazi has joined #tripleo | 19:46 | |
| *** morazi has quit IRC | 19:50 | |
| openstackgerrit | Merged openstack/tripleo-upgrade master: Fix conditional for python3-tripleoclient in undercloud pre-reqs https://review.opendev.org/739946 | 19:52 |
| *** xek has joined #tripleo | 20:07 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/train: [TRAIN-ONLY] Add keystone_resources for Panko https://review.opendev.org/737356 | 20:09 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/train: [TRAIN ONLY] Provide way to initialize Leapp https://review.opendev.org/736072 | 20:12 |
| openstackgerrit | Merged openstack/tripleo-validations master: Ensure virtualenv is installed. https://review.opendev.org/739476 | 20:12 |
| *** morazi has joined #tripleo | 20:16 | |
| openstackgerrit | Michele Baldessari proposed openstack/tripleo-heat-templates master: WIP add locking for the restart bundles https://review.opendev.org/740173 | 20:17 |
| *** paramite has quit IRC | 20:29 | |
| *** gfidente has quit IRC | 20:30 | |
| *** slaweq has joined #tripleo | 20:33 | |
| *** xek has quit IRC | 20:36 | |
| *** jfrancoa has quit IRC | 20:50 | |
| *** jbadiapa has quit IRC | 20:58 | |
| *** TrevorV has quit IRC | 21:06 | |
| *** maciejjozefczyk has quit IRC | 21:12 | |
| *** yolanda has quit IRC | 21:16 | |
| *** yolanda has joined #tripleo | 21:17 | |
| *** raildo has quit IRC | 21:22 | |
| weshay_ruck | rlandy, https://logserver.rdoproject.org/23/740323/1/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-vexxhost/caabf81/logs/overcloud-controller-0/etc/yum.repos.d/ | 21:25 |
| *** suuuper has quit IRC | 21:36 | |
| *** csibbitt has quit IRC | 21:38 | |
| weshay_ruck | stevebaker, do you happen to know what repo the overcloud-base image element exists? | 21:57 |
| weshay_ruck | don't see it in tripleo-image-elements | 21:57 |
| weshay_ruck | stevebaker, found it :) nvrmind | 22:03 |
| stevebaker | weshay_ruck: you're welcome | 22:04 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-puppet-elements master: WIP: add buildah to overcloud-base https://review.opendev.org/740378 | 22:04 |
| weshay_ruck | :) | 22:05 |
| *** slaweq has quit IRC | 22:21 | |
| *** rh-jelabarre has quit IRC | 22:30 | |
| *** slaweq has joined #tripleo | 22:38 | |
| *** jamesdenton has quit IRC | 22:40 | |
| cloudnull | weshay_ruck hows the CI coming along ? anything new or on going (besides the minion gate) that needs looking into ? | 22:53 |
| *** ekultails has quit IRC | 22:59 | |
| *** tkajinam has joined #tripleo | 23:00 | |
| *** rfolco has joined #tripleo | 23:03 | |
| *** rfolco has quit IRC | 23:07 | |
| *** pkopec has joined #tripleo | 23:38 | |
| *** dmellado has quit IRC | 23:39 | |
| openstackgerrit | Merged openstack/tripleo-upgrade master: Do not fail If stack has no controllers https://review.opendev.org/732072 | 23:40 |
| *** rlandy has quit IRC | 23:58 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates master: WIP: Complete refactor update: use step file, add information for user. https://review.opendev.org/740302 | 23:59 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!