| openstackgerrit | RedHat RDO CI proposed openstack/tripleo-quickstart-extras master: GATE CHECK for quickstart-extras https://review.openstack.org/560445 | 00:00 |
|---|---|---|
| openstackgerrit | RedHat RDO CI proposed openstack/tripleo-heat-templates master: GATE CHECK for TripleO https://review.openstack.org/604298 | 00:00 |
| *** rascasoft has joined #tripleo | 00:04 | |
| *** gfidente|afk has quit IRC | 00:04 | |
| *** ooolpbot has joined #tripleo | 00:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1811004 | 00:10 |
| openstack | Launchpad bug 1811004 in tripleo "tripleo-ci-centos-7-scenario003-standalone fails tempest.scenario.test_minimum_basic.TestMinimumBasicScenario" [Critical,In progress] - Assigned to Michele Baldessari (michele) | 00:10 |
| *** ooolpbot has quit IRC | 00:10 | |
| *** jbadiapa has quit IRC | 00:10 | |
| *** rascasoft has quit IRC | 00:11 | |
| *** cfontain has joined #tripleo | 00:12 | |
| *** ansmith has quit IRC | 00:17 | |
| *** rascasoft has joined #tripleo | 00:18 | |
| *** wolverineav has quit IRC | 00:18 | |
| *** rascasoft has quit IRC | 00:25 | |
| *** tosky has quit IRC | 00:32 | |
| *** rascasoft has joined #tripleo | 00:32 | |
| *** wolverineav has joined #tripleo | 00:35 | |
| *** rascasoft has quit IRC | 00:38 | |
| *** lblanchard has joined #tripleo | 00:38 | |
| *** cfontain has quit IRC | 00:46 | |
| *** zaneb has joined #tripleo | 00:46 | |
| *** wolverineav has quit IRC | 00:48 | |
| *** wolverineav has joined #tripleo | 00:50 | |
| *** wolverineav has quit IRC | 00:55 | |
| *** rascasoft has joined #tripleo | 00:56 | |
| *** ansmith has joined #tripleo | 00:59 | |
| *** rascasoft has quit IRC | 01:02 | |
| *** ooolpbot has joined #tripleo | 01:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1811004 | 01:10 |
| openstack | Launchpad bug 1811004 in tripleo "tripleo-ci-centos-7-scenario003-standalone fails tempest.scenario.test_minimum_basic.TestMinimumBasicScenario" [Critical,In progress] - Assigned to Michele Baldessari (michele) | 01:10 |
| *** ooolpbot has quit IRC | 01:10 | |
| *** rascasoft has joined #tripleo | 01:13 | |
| *** rascasoft has quit IRC | 01:18 | |
| *** fragatina has quit IRC | 01:34 | |
| *** wolverineav has joined #tripleo | 01:35 | |
| *** wolverin_ has joined #tripleo | 01:38 | |
| *** wolverineav has quit IRC | 01:39 | |
| *** wolverin_ has quit IRC | 01:43 | |
| *** rascasoft has joined #tripleo | 01:43 | |
| *** rascasoft has quit IRC | 01:49 | |
| *** ansmith has quit IRC | 01:54 | |
| *** rascasoft has joined #tripleo | 01:59 | |
| *** rascasoft has quit IRC | 02:05 | |
| *** ooolpbot has joined #tripleo | 02:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1811004 | 02:10 |
| *** ooolpbot has quit IRC | 02:10 | |
| openstack | Launchpad bug 1811004 in tripleo "tripleo-ci-centos-7-scenario003-standalone fails tempest.scenario.test_minimum_basic.TestMinimumBasicScenario" [Critical,In progress] - Assigned to Michele Baldessari (michele) | 02:10 |
| openstackgerrit | zhongshengping proposed openstack/puppet-tripleo master: Modify puppet version 4 to 5 https://review.openstack.org/630858 | 02:13 |
| openstackgerrit | zhongshengping proposed openstack/puppet-pacemaker master: Modify puppet version 4 to 5 https://review.openstack.org/630864 | 02:19 |
| *** rascasoft has joined #tripleo | 02:26 | |
| *** saneax has joined #tripleo | 02:31 | |
| *** mschuppert has quit IRC | 02:32 | |
| *** rascasoft has quit IRC | 02:32 | |
| *** lblanchard has quit IRC | 02:37 | |
| *** rascasoft has joined #tripleo | 03:07 | |
| *** ooolpbot has joined #tripleo | 03:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1811004 | 03:10 |
| openstack | Launchpad bug 1811004 in tripleo "tripleo-ci-centos-7-scenario003-standalone fails tempest.scenario.test_minimum_basic.TestMinimumBasicScenario" [Critical,In progress] - Assigned to Michele Baldessari (michele) | 03:10 |
| *** ooolpbot has quit IRC | 03:10 | |
| *** sanjayu_ has joined #tripleo | 03:14 | |
| *** rascasoft has quit IRC | 03:15 | |
| *** saneax has quit IRC | 03:16 | |
| *** apetrich has quit IRC | 03:16 | |
| *** wolverineav has joined #tripleo | 03:27 | |
| *** cfontain has joined #tripleo | 03:28 | |
| *** wolverineav has quit IRC | 03:32 | |
| *** rascasoft has joined #tripleo | 03:36 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart master: add a noop-upgrade variable for fs051 https://review.openstack.org/630400 | 03:39 |
| *** ykarel|away has joined #tripleo | 03:42 | |
| *** rascasoft has quit IRC | 03:44 | |
| *** rascasoft has joined #tripleo | 04:00 | |
| *** tzumainn has quit IRC | 04:00 | |
| *** cfontain has quit IRC | 04:01 | |
| *** udesale has joined #tripleo | 04:05 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/rocky: Fix Octavia hieradata keys https://review.openstack.org/623171 | 04:06 |
| *** rascasoft has quit IRC | 04:07 | |
| *** ooolpbot has joined #tripleo | 04:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1811004 | 04:10 |
| *** ooolpbot has quit IRC | 04:10 | |
| openstack | Launchpad bug 1811004 in tripleo "tripleo-ci-centos-7-scenario003-standalone fails tempest.scenario.test_minimum_basic.TestMinimumBasicScenario" [Critical,In progress] - Assigned to Michele Baldessari (michele) | 04:10 |
| *** thrash is now known as thrash|g0ne | 04:11 | |
| *** udesale has quit IRC | 04:15 | |
| *** udesale has joined #tripleo | 04:16 | |
| *** rascasoft has joined #tripleo | 04:21 | |
| *** udesale has quit IRC | 04:26 | |
| *** rascasoft has quit IRC | 04:27 | |
| *** radeks has joined #tripleo | 04:33 | |
| *** fragatina has joined #tripleo | 04:45 | |
| *** rh-jelabarre has quit IRC | 04:47 | |
| *** fragatina has quit IRC | 04:47 | |
| *** fragatina has joined #tripleo | 04:48 | |
| *** fragatina has quit IRC | 04:51 | |
| *** wolverineav has joined #tripleo | 04:51 | |
| *** fragatina has joined #tripleo | 04:51 | |
| *** rascasoft has joined #tripleo | 04:52 | |
| *** fragatina has quit IRC | 04:54 | |
| *** _fragatina_ has joined #tripleo | 04:54 | |
| *** _fragatina_ has quit IRC | 04:56 | |
| *** fragatina has joined #tripleo | 04:56 | |
| *** rascasoft has quit IRC | 04:57 | |
| *** fragatina has quit IRC | 04:58 | |
| *** fragatina has joined #tripleo | 04:58 | |
| *** wolverineav has quit IRC | 05:02 | |
| *** wolverineav has joined #tripleo | 05:03 | |
| *** fragatina has quit IRC | 05:03 | |
| *** fragatina has joined #tripleo | 05:04 | |
| openstackgerrit | Merged openstack/tripleo-common master: Use Timesync service instead of Ntp https://review.openstack.org/609657 | 05:07 |
| *** ooolpbot has joined #tripleo | 05:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1811004 | 05:10 |
| *** ooolpbot has quit IRC | 05:10 | |
| openstack | Launchpad bug 1811004 in tripleo "tripleo-ci-centos-7-scenario003-standalone fails tempest.scenario.test_minimum_basic.TestMinimumBasicScenario" [Critical,In progress] - Assigned to Michele Baldessari (michele) | 05:10 |
| *** rascasoft has joined #tripleo | 05:10 | |
| *** fragatina has quit IRC | 05:11 | |
| *** fragatina has joined #tripleo | 05:11 | |
| *** fragatina has quit IRC | 05:13 | |
| *** fragatina has joined #tripleo | 05:13 | |
| *** fragatina has quit IRC | 05:16 | |
| *** fragatina has joined #tripleo | 05:16 | |
| *** rascasoft has quit IRC | 05:16 | |
| *** fragatina has quit IRC | 05:17 | |
| *** fragatina has joined #tripleo | 05:17 | |
| *** fragatina has quit IRC | 05:19 | |
| *** fragatina has joined #tripleo | 05:19 | |
| *** fragatina has quit IRC | 05:27 | |
| *** fragatina has joined #tripleo | 05:28 | |
| *** fragatina has quit IRC | 05:30 | |
| *** fragatina has joined #tripleo | 05:30 | |
| *** fragatina has quit IRC | 05:32 | |
| *** fragatina has joined #tripleo | 05:32 | |
| *** fragatina has quit IRC | 05:36 | |
| *** fragatina has joined #tripleo | 05:36 | |
| *** rascasoft has joined #tripleo | 05:37 | |
| *** chkumar|out is now known as chandankumar | 05:38 | |
| *** ykarel|away is now known as ykarel | 05:38 | |
| *** rascasoft has quit IRC | 05:43 | |
| *** ykarel has quit IRC | 05:43 | |
| *** fragatina has quit IRC | 05:45 | |
| *** fragatina has joined #tripleo | 05:46 | |
| *** fragatina has quit IRC | 05:51 | |
| *** fragatina has joined #tripleo | 05:52 | |
| *** fragatina has quit IRC | 05:59 | |
| *** fragatina has joined #tripleo | 05:59 | |
| *** ykarel has joined #tripleo | 06:00 | |
| *** fragatina has quit IRC | 06:01 | |
| *** fragatina has joined #tripleo | 06:01 | |
| openstackgerrit | Merged openstack/python-tripleoclient master: Use template for lower-constraints https://review.openstack.org/627151 | 06:03 |
| openstackgerrit | Merged openstack/tripleo-common stable/queens: Fail node cleaning on timeout https://review.openstack.org/628971 | 06:03 |
| *** udesale has joined #tripleo | 06:05 | |
| *** ooolpbot has joined #tripleo | 06:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1811004 | 06:10 |
| *** ooolpbot has quit IRC | 06:10 | |
| openstack | Launchpad bug 1811004 in tripleo "tripleo-ci-centos-7-scenario003-standalone fails tempest.scenario.test_minimum_basic.TestMinimumBasicScenario" [Critical,In progress] - Assigned to Michele Baldessari (michele) | 06:10 |
| *** udesale has quit IRC | 06:12 | |
| *** udesale has joined #tripleo | 06:13 | |
| *** ratailor has joined #tripleo | 06:13 | |
| *** fragatina has quit IRC | 06:13 | |
| *** fragatina has joined #tripleo | 06:14 | |
| *** fragatina has quit IRC | 06:18 | |
| *** fragatina has joined #tripleo | 06:18 | |
| *** fragatina has quit IRC | 06:22 | |
| *** fragatina has joined #tripleo | 06:23 | |
| *** fragatina has quit IRC | 06:23 | |
| *** fragatina has joined #tripleo | 06:24 | |
| *** fragatina has quit IRC | 06:25 | |
| *** fragatina has joined #tripleo | 06:25 | |
| *** fragatina has quit IRC | 06:26 | |
| *** fragatina has joined #tripleo | 06:26 | |
| *** fragatina has quit IRC | 06:27 | |
| *** fragatina has joined #tripleo | 06:27 | |
| *** itlinux_ has joined #tripleo | 06:28 | |
| *** wolverineav has quit IRC | 06:28 | |
| *** wolverineav has joined #tripleo | 06:29 | |
| *** fragatina has quit IRC | 06:29 | |
| *** fragatina has joined #tripleo | 06:29 | |
| *** itlinux has quit IRC | 06:30 | |
| *** fragatina has quit IRC | 06:31 | |
| *** fragatina has joined #tripleo | 06:32 | |
| *** fragatina has quit IRC | 06:32 | |
| *** wolverineav has quit IRC | 06:33 | |
| *** fragatina has joined #tripleo | 06:33 | |
| *** udesale has quit IRC | 06:33 | |
| *** udesale has joined #tripleo | 06:33 | |
| *** brault has quit IRC | 06:33 | |
| *** _fragatina_ has joined #tripleo | 06:34 | |
| *** _fragatina_ has quit IRC | 06:37 | |
| *** _fragatina_ has joined #tripleo | 06:37 | |
| *** gfidente has joined #tripleo | 06:38 | |
| *** fragatina has quit IRC | 06:38 | |
| *** rascasoft has joined #tripleo | 06:38 | |
| *** cfontain has joined #tripleo | 06:41 | |
| *** _fragatina_ has quit IRC | 06:44 | |
| *** fragatina has joined #tripleo | 06:44 | |
| *** rascasoft has quit IRC | 06:47 | |
| *** fragatina has quit IRC | 06:47 | |
| *** fragatina has joined #tripleo | 06:48 | |
| *** fragatina has quit IRC | 06:48 | |
| *** itlinux_ has quit IRC | 06:48 | |
| *** fragatina has joined #tripleo | 06:48 | |
| *** fragatina has quit IRC | 06:49 | |
| *** fragatina has joined #tripleo | 06:50 | |
| *** fragatina has quit IRC | 06:52 | |
| *** fragatina has joined #tripleo | 06:53 | |
| *** ksambor has joined #tripleo | 06:54 | |
| *** avivgt has joined #tripleo | 06:57 | |
| *** rcernin has quit IRC | 06:59 | |
| *** rascasoft has joined #tripleo | 07:02 | |
| *** fragatina has quit IRC | 07:07 | |
| *** fragatina has joined #tripleo | 07:08 | |
| *** rascasoft has quit IRC | 07:09 | |
| *** rascasoft has joined #tripleo | 07:10 | |
| *** ooolpbot has joined #tripleo | 07:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1811004 | 07:10 |
| openstack | Launchpad bug 1811004 in tripleo "tripleo-ci-centos-7-scenario003-standalone fails tempest.scenario.test_minimum_basic.TestMinimumBasicScenario" [Critical,In progress] - Assigned to Michele Baldessari (michele) | 07:10 |
| *** ooolpbot has quit IRC | 07:10 | |
| *** jfrancoa has joined #tripleo | 07:10 | |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-heat-templates master: Ensure logs folder is created in prep hosts tasks https://review.openstack.org/630699 | 07:12 |
| *** cfontain has quit IRC | 07:14 | |
| *** jbadiapa has joined #tripleo | 07:15 | |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart-extras master: Use os_tempest for running tempest on standalone https://review.openstack.org/628415 | 07:15 |
| openstackgerrit | Chandan Kumar proposed openstack-infra/tripleo-ci master: Run os_tempest with standalone for tripleo-specs https://review.openstack.org/627500 | 07:16 |
| *** fragatina has quit IRC | 07:18 | |
| *** fragatina has joined #tripleo | 07:19 | |
| *** mschuppert has joined #tripleo | 07:23 | |
| *** fragatina has quit IRC | 07:25 | |
| *** fragatina has joined #tripleo | 07:26 | |
| *** florianf_ has joined #tripleo | 07:28 | |
| *** florianf has joined #tripleo | 07:29 | |
| *** fragatina has quit IRC | 07:31 | |
| *** fragatina has joined #tripleo | 07:31 | |
| *** kopecmartin|off is now known as kopecmartin | 07:35 | |
| *** cylopez has joined #tripleo | 07:40 | |
| *** wolverineav has joined #tripleo | 07:43 | |
| *** fragatina has quit IRC | 07:44 | |
| *** fragatina has joined #tripleo | 07:44 | |
| *** cfontain has joined #tripleo | 07:46 | |
| *** cfontain has quit IRC | 07:47 | |
| *** wolverineav has quit IRC | 07:47 | |
| *** apetrich has joined #tripleo | 07:48 | |
| *** fragatina has quit IRC | 07:49 | |
| *** _fragatina_ has joined #tripleo | 07:49 | |
| *** ccamacho has joined #tripleo | 07:56 | |
| openstackgerrit | waleed mousa proposed openstack/puppet-tripleo stable/queens: Configure switchedv while there is no vfs bounded https://review.openstack.org/630605 | 07:57 |
| *** _fragatina_ has quit IRC | 07:59 | |
| *** fragatina has joined #tripleo | 08:00 | |
| openstackgerrit | RedHat RDO CI proposed openstack/tripleo-heat-templates stable/rocky: GATE CHECK for TripleO https://review.openstack.org/604293 | 08:00 |
| *** therve has joined #tripleo | 08:00 | |
| *** radeks has quit IRC | 08:08 | |
| *** aufi has joined #tripleo | 08:08 | |
| *** ooolpbot has joined #tripleo | 08:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1811004 | 08:10 |
| openstack | Launchpad bug 1811004 in tripleo "tripleo-ci-centos-7-scenario003-standalone fails tempest.scenario.test_minimum_basic.TestMinimumBasicScenario" [Critical,In progress] - Assigned to Michele Baldessari (michele) | 08:10 |
| *** ooolpbot has quit IRC | 08:10 | |
| *** gfidente has quit IRC | 08:12 | |
| *** fragatina has quit IRC | 08:13 | |
| *** fragatina has joined #tripleo | 08:14 | |
| *** panda|off is now known as panda | 08:19 | |
| panda | mwhahaha: thanks, added to the doc | 08:22 |
| *** fragatina has quit IRC | 08:22 | |
| *** hberaud has joined #tripleo | 08:23 | |
| *** fragatina has joined #tripleo | 08:23 | |
| *** ccamacho has quit IRC | 08:25 | |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-quickstart-extras master: Update tht,tripleoclient and tc before standalone_upgrade. https://review.openstack.org/630228 | 08:25 |
| *** fragatina has quit IRC | 08:26 | |
| *** fragatina has joined #tripleo | 08:27 | |
| *** avivgt has quit IRC | 08:31 | |
| *** jpena|off is now known as jpena | 08:33 | |
| *** boazel has joined #tripleo | 08:34 | |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-specs master: New spec for stein: os_tempest tripleo integration https://review.openstack.org/630654 | 08:34 |
| *** holser_ has joined #tripleo | 08:35 | |
| *** boazel_ has quit IRC | 08:37 | |
| *** jpich has joined #tripleo | 08:38 | |
| *** gfidente has joined #tripleo | 08:39 | |
| *** tosky has joined #tripleo | 08:44 | |
| *** fragatina has quit IRC | 08:45 | |
| *** fragatina has joined #tripleo | 08:46 | |
| *** fragatina has quit IRC | 08:46 | |
| *** fragatina has joined #tripleo | 08:46 | |
| openstackgerrit | Michele Baldessari proposed openstack/tripleo-heat-templates master: Fix scenario003-standalone https://review.openstack.org/630751 | 08:49 |
| *** ykarel is now known as ykarel|lunch | 08:55 | |
| *** chem has joined #tripleo | 08:55 | |
| *** dtantsur|afk is now known as dtantsur | 08:56 | |
| *** xek has joined #tripleo | 09:00 | |
| *** fragatina has quit IRC | 09:01 | |
| *** fragatina has joined #tripleo | 09:01 | |
| *** fragatina has quit IRC | 09:02 | |
| *** fragatina has joined #tripleo | 09:03 | |
| *** fragatina has quit IRC | 09:04 | |
| *** fragatina has joined #tripleo | 09:05 | |
| *** paramite has joined #tripleo | 09:07 | |
| *** ccamacho has joined #tripleo | 09:08 | |
| *** ooolpbot has joined #tripleo | 09:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1811004 | 09:10 |
| openstack | Launchpad bug 1811004 in tripleo "tripleo-ci-centos-7-scenario003-standalone fails tempest.scenario.test_minimum_basic.TestMinimumBasicScenario" [Critical,In progress] - Assigned to Michele Baldessari (michele) | 09:10 |
| *** ooolpbot has quit IRC | 09:10 | |
| *** bandini has joined #tripleo | 09:19 | |
| openstackgerrit | Kamil Sambor proposed openstack/tripleo-quickstart master: WIP: Remove openvswitch tasks https://review.openstack.org/619196 | 09:22 |
| *** bogdando has joined #tripleo | 09:26 | |
| *** pcaruana has joined #tripleo | 09:30 | |
| *** derekh has joined #tripleo | 09:33 | |
| *** wolverineav has joined #tripleo | 09:36 | |
| *** wolverineav has quit IRC | 09:41 | |
| openstackgerrit | Natal Ngétal proposed openstack/tripleo-docs master: [Documentation] Typo. https://review.openstack.org/630916 | 09:44 |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart master: Add os_tempest role as a TQE extras https://review.openstack.org/628421 | 09:46 |
| openstackgerrit | Chandan Kumar proposed openstack-infra/tripleo-ci master: Run os_tempest with standalone for tripleo-specs https://review.openstack.org/627500 | 09:50 |
| chandankumar | quiquell: ^^ testing here | 09:50 |
| quiquell | chandankumar: ack | 09:52 |
| *** ratailor_ has joined #tripleo | 09:52 | |
| openstackgerrit | Chandan Kumar proposed openstack-infra/tripleo-ci master: Run os_tempest with standalone for tripleo-specs https://review.openstack.org/627500 | 09:53 |
| *** ratailor has quit IRC | 09:55 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates master: Be explicit when passing vars into deploy steps https://review.openstack.org/613300 | 10:03 |
| *** waleedm has joined #tripleo | 10:07 | |
| *** jaosorior has joined #tripleo | 10:09 | |
| *** ooolpbot has joined #tripleo | 10:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1811004 | 10:10 |
| openstack | Launchpad bug 1811004 in tripleo "tripleo-ci-centos-7-scenario003-standalone fails tempest.scenario.test_minimum_basic.TestMinimumBasicScenario" [Critical,In progress] - Assigned to Michele Baldessari (michele) | 10:10 |
| *** ooolpbot has quit IRC | 10:10 | |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-common master: Add a role for data presistence and restoration https://review.openstack.org/630921 | 10:22 |
| *** fragatina has quit IRC | 10:23 | |
| *** fragatina has joined #tripleo | 10:24 | |
| *** udesale has quit IRC | 10:25 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates stable/rocky: DO-NOT-MERGE Test new ceph-container builds (OSP queens) https://review.openstack.org/601598 | 10:25 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates stable/queens: DO-NOT-MERGE Test new ceph-container builds (OSP queens) https://review.openstack.org/562215 | 10:26 |
| *** avivgt has joined #tripleo | 10:26 | |
| *** aedc has joined #tripleo | 10:28 | |
| *** fragatina has quit IRC | 10:28 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-common master: Bump the Ceph container image to 3.2.1 https://review.openstack.org/630925 | 10:29 |
| *** paramite has quit IRC | 10:29 | |
| *** fragatina has joined #tripleo | 10:29 | |
| openstackgerrit | Michele Baldessari proposed openstack/tripleo-heat-templates master: Fix scenario003-standalone https://review.openstack.org/630751 | 10:30 |
| *** paramite has joined #tripleo | 10:30 | |
| *** chandankumar has quit IRC | 10:31 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-quickstart-extras master: Bump the Ceph container image to 3.2.1 https://review.openstack.org/630927 | 10:32 |
| *** salmankhan has joined #tripleo | 10:33 | |
| *** arxcruz|ruck has quit IRC | 10:34 | |
| *** udesale has joined #tripleo | 10:35 | |
| *** chandan_kumar has joined #tripleo | 10:35 | |
| *** jtomasek has quit IRC | 10:40 | |
| *** jtomasek has joined #tripleo | 10:41 | |
| *** salmankhan has quit IRC | 10:42 | |
| *** salmankhan has joined #tripleo | 10:42 | |
| *** ykarel|lunch is now known as ykarel | 10:42 | |
| *** salmankhan has quit IRC | 10:46 | |
| *** salmankhan has joined #tripleo | 10:53 | |
| *** brault has joined #tripleo | 10:54 | |
| *** apetrich has quit IRC | 10:54 | |
| openstackgerrit | Martin Mágr proposed openstack/tripleo-heat-templates stable/rocky: Enable fluentd health check https://review.openstack.org/612373 | 10:55 |
| openstackgerrit | Martin Mágr proposed openstack/tripleo-heat-templates stable/rocky: Enable health check for OVN containers https://review.openstack.org/612374 | 10:56 |
| openstackgerrit | Martin Mágr proposed openstack/tripleo-heat-templates stable/rocky: Enable ceilometer-agent-compute health check https://review.openstack.org/612372 | 10:57 |
| openstackgerrit | Lee Yarwood proposed openstack/tripleo-common master: placement: Use the less generic PlacementApi service name https://review.openstack.org/630708 | 11:00 |
| *** udesale has quit IRC | 11:01 | |
| *** arxcruz has joined #tripleo | 11:04 | |
| *** apetrich has joined #tripleo | 11:07 | |
| *** holser_ is now known as holser|lunch | 11:09 | |
| *** ratailor__ has joined #tripleo | 11:10 | |
| *** ooolpbot has joined #tripleo | 11:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1811004 | 11:10 |
| openstack | Launchpad bug 1811004 in tripleo "tripleo-ci-centos-7-scenario003-standalone fails tempest.scenario.test_minimum_basic.TestMinimumBasicScenario" [Critical,In progress] - Assigned to Michele Baldessari (michele) | 11:10 |
| *** ooolpbot has quit IRC | 11:10 | |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-quickstart master: Add some fixes for libvirt https://review.openstack.org/629839 | 11:10 |
| *** ratailor_ has quit IRC | 11:12 | |
| openstackgerrit | Sorin Sbarnea proposed openstack/tripleo-quickstart master: Run all linters via pre-commit https://review.openstack.org/627545 | 11:14 |
| *** morazi has quit IRC | 11:14 | |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-quickstart master: Add some fixes for libvirt https://review.openstack.org/629839 | 11:21 |
| openstackgerrit | Rabi Mishra proposed openstack/tripleo-heat-templates master: Use net=none for *_init_log(s) containers https://review.openstack.org/630631 | 11:22 |
| openstackgerrit | Rabi Mishra proposed openstack/tripleo-heat-templates master: Remove configuration for cni0 bridge https://review.openstack.org/630625 | 11:22 |
| openstackgerrit | Rabi Mishra proposed openstack/tripleo-heat-templates master: Create /run/netns if does not exist https://review.openstack.org/630936 | 11:22 |
| *** openstackgerrit has quit IRC | 11:22 | |
| ykarel | jfrancoa, hi | 11:27 |
| jfrancoa | ykarel: hey | 11:28 |
| ykarel | jfrancoa, where is the job running to upgrade queens to rocky, i wanted to see the job def | 11:28 |
| ykarel | overcloud | 11:28 |
| jfrancoa | ykarel: bufff...I think there was something in the rdo jobs...but I think it's ages I don't see them running, let me look for it as we started relying on the standalone jobs | 11:29 |
| jfrancoa | ykarel: I put myself the objective to resurrect those upgrades jobs, but I don't find the moment to do it | 11:29 |
| ykarel | hmm me don't find there in rdo such job | 11:30 |
| *** openstackgerrit has joined #tripleo | 11:30 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: Collect logs: Handle errors and timeout https://review.openstack.org/617617 | 11:30 |
| jfrancoa | ykarel: it used to be legacy-tripleo-ci-centos-7-container-to-container-upgrades-master , when rocky was master but then we focused on the standalone jobs and it was not created any legacy-tripleo-ci-centos-7-container-to-container-upgrades-rocky job | 11:31 |
| *** hberaud is now known as hberaud|lunch | 11:33 | |
| ykarel | jfrancoa, okk so now these are removed | 11:33 |
| jfrancoa | ykarel: I don't have jobs to resurrect then .. :-D | 11:34 |
| openstackgerrit | Rabi Mishra proposed openstack/tripleo-heat-templates stable/pike: Fix typo in octavia upgrade_tasks https://review.openstack.org/630941 | 11:34 |
| ykarel | jfrancoa, ack, Thanks for checking | 11:35 |
| *** udesale has joined #tripleo | 11:36 | |
| jfrancoa | ykarel: yw | 11:37 |
| *** two_oes has joined #tripleo | 11:37 | |
| openstackgerrit | Surya Prakash (spsurya) proposed openstack/tripleo-heat-templates master: Update the min tox version to 2.0 https://review.openstack.org/630945 | 11:40 |
| *** apetrich has quit IRC | 11:44 | |
| openstackgerrit | Surya Prakash (spsurya) proposed openstack/tripleo-common master: Update tox min version to 2.0 https://review.openstack.org/630946 | 11:44 |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-quickstart master: Add some fixes for libvirt https://review.openstack.org/629839 | 11:46 |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-validations master: Fix minor Ansible syntax https://review.openstack.org/630949 | 11:50 |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-validations master: Fix minor Ansible syntax https://review.openstack.org/630949 | 11:52 |
| *** noama has quit IRC | 11:52 | |
| *** fragatina has quit IRC | 11:54 | |
| *** holser|lunch is now known as holser_ | 11:57 | |
| *** ansmith has joined #tripleo | 11:59 | |
| *** factor has quit IRC | 12:00 | |
| *** apetrich has joined #tripleo | 12:01 | |
| *** jpena is now known as jpena|lunch | 12:03 | |
| *** rbrady has joined #tripleo | 12:04 | |
| openstackgerrit | Rabi Mishra proposed openstack/tripleo-heat-templates master: Create /run/netns if does not exist https://review.openstack.org/630936 | 12:04 |
| openstackgerrit | Rabi Mishra proposed openstack/tripleo-heat-templates master: Use net=none for *_init_log(s) containers https://review.openstack.org/630631 | 12:04 |
| openstackgerrit | Rabi Mishra proposed openstack/tripleo-heat-templates master: Remove configuration for cni0 bridge https://review.openstack.org/630625 | 12:04 |
| jaosorior | ramishra: have you tried out the /run/nets creation patch? | 12:05 |
| ramishra | jaosorior: you mean the one I proposed or something else? | 12:06 |
| jaosorior | the one you just proposed | 12:06 |
| jaosorior | ramishra: I was just concerned about the SELinux context | 12:06 |
| jaosorior | mandre: around? | 12:07 |
| ramishra | jaosorior: Hmm..Does not have much idea about the implications though. Feel free to comment on it. We had a rdo patch for bug 1799638, but it did not fix it AFAICT. | 12:09 |
| openstack | bug 1799638 in tripleo "/run/netns doesn't exist until we create a network namespace" [High,Fix committed] https://launchpad.net/bugs/1799638 - Assigned to Cédric Jeanneret (cjeanner) | 12:09 |
| openstackgerrit | Sorin Sbarnea proposed openstack/tripleo-quickstart-extras master: Corrected openstack-virtual-baremetal repo location https://review.openstack.org/630953 | 12:10 |
| *** ooolpbot has joined #tripleo | 12:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1811004 | 12:10 |
| openstack | Launchpad bug 1811004 in tripleo "tripleo-ci-centos-7-scenario003-standalone fails tempest.scenario.test_minimum_basic.TestMinimumBasicScenario" [Critical,In progress] - Assigned to Michele Baldessari (michele) | 12:10 |
| *** ooolpbot has quit IRC | 12:10 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Conditionalize docker socket bind-mount https://review.openstack.org/630955 | 12:14 |
| EmilienM | Tengu: ^ | 12:14 |
| *** ansmith has quit IRC | 12:16 | |
| Tengu | on it | 12:17 |
| jaosorior | Tengu: could you check out https://review.openstack.org/#/c/630936/2 ? | 12:19 |
| Tengu | jaosorior: errrr.... wait... | 12:19 |
| Tengu | that thing looks a lot like something I did months ago. | 12:20 |
| Tengu | I created a placeholder in a systemd unit, you remember? | 12:20 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: DNM - podman testing https://review.openstack.org/628379 | 12:20 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: DNM - podman testing https://review.openstack.org/628380 | 12:20 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: DNM - podman testing https://review.openstack.org/628381 | 12:20 |
| jaosorior | Tengu: which one? | 12:20 |
| Tengu | jaosorior: ramishra I'll -1 that review, gimme a second to find my patch | 12:20 |
| EmilienM | Tengu: ^ the patches I use to test your tht code and see if we still hit the podman bug | 12:20 |
| Tengu | jaosorior: https://review.openstack.org/#/c/630936/2 | 12:20 |
| Tengu | EmilienM: good | 12:21 |
| jaosorior | Tengu: this one too http://bit.ly/2rllTOo | 12:21 |
| Tengu | jaosorior: err? | 12:21 |
| jaosorior | tried to rick-roll you | 12:21 |
| jaosorior | anyway | 12:22 |
| ramishra | Tengu: that rdo patch of yours does not work see http://logs.openstack.org/31/630631/4/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/a56584b/logs/undercloud/home/zuul/undercloud_install.log.txt.gz#_2019-01-14_14_58_10 | 12:22 |
| Tengu | ramishra: it worked until now at least. | 12:22 |
| Tengu | so there's something else. | 12:22 |
| Tengu | lemme comment. | 12:23 |
| ramishra | Tengu: yeah, it's because we were using cni0 bridge in many containers | 12:23 |
| ramishra | now when I changed those to net=node it exposed the bug I guess | 12:23 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-quickstart master: Use the tripleo-standalone-scenarios-full template https://review.openstack.org/630247 | 12:24 |
| ramishra | Tengu: https://review.openstack.org/#/c/630631/ | 12:24 |
| ramishra | Tengu: your patch just enables the service, without rebooting the node (starting the placeholder service) it won't be created I think | 12:25 |
| Tengu | can't find back my patch -.-. maybe you want to tweak it instead of creating this directory, as it won't persist reboots anyway. | 12:26 |
| *** hberaud|lunch is now known as hberaud | 12:27 | |
| Tengu | ahhh https://review.rdoproject.org/r/#/c/17078/ | 12:27 |
| *** jcoufal has joined #tripleo | 12:27 | |
| chandan_kumar | EmilienM: jaosorior please have a look when time https://review.openstack.org/630654 | 12:28 |
| ramishra | Tengu: I've seen that one, but I don't think we should do it in rdo packages at all | 12:28 |
| openstackgerrit | Martin Mágr proposed openstack/tripleo-heat-templates stable/rocky: Enable ceilometer-agent-compute health check https://review.openstack.org/612372 | 12:28 |
| Tengu | ramishra: well, we have to do something like that anyway. as said, it won't persist reboot. | 12:28 |
| bandini | weshay, marios: I updated https://bugs.launchpad.net/tripleo/+bug/1811004 I think we need some network help now (cannot see any rabbit errors after the last iteration of my patch) | 12:29 |
| openstack | Launchpad bug 1811004 in tripleo "tripleo-ci-centos-7-scenario003-standalone fails tempest.scenario.test_minimum_basic.TestMinimumBasicScenario" [Critical,In progress] - Assigned to Michele Baldessari (michele) | 12:29 |
| EmilienM | chandan_kumar: ok | 12:29 |
| Tengu | ramishra: setting the directory like you're doing will crash at some point. so maybe it's an issue of order things are made, or there's something else. But I can't let you create that directory like that. | 12:29 |
| marios | bandini: thanks very much i didn't check back after zuul reported but noted it was failing on tempest thanks for looking there! | 12:30 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/puppet-tripleo master: [WIP] Variables to reprovision pacemaker cluster one node at a time. https://review.openstack.org/630956 | 12:30 |
| marios | bandini: not sure if its the same tempest issue on the bug or what but will dig some more in a while thanks | 12:30 |
| *** radeks has joined #tripleo | 12:31 | |
| ramishra | Tengu: crash? Don't we do that for other stuff, may be I'm missing selinux context, your fix does not fix the bug AFAICT, also I don't think having an additional task that helps when it does not exist won't possibly hurt? | 12:31 |
| Tengu | ramishra: the issue is, this /run/netns will be erased upon reboot. First cause of crash; more over, it's a sensible directory, accessed by the host itself, and it should keep its default setype - the right way to create that directory is to actually create a namespace and drop it. Like the unit I've pushed does it. So maybe it's not located in the right place, at the time I made it it was the correct | 12:34 |
| Tengu | location/repo/package. | 12:34 |
| *** raildo has joined #tripleo | 12:34 | |
| Tengu | ramishra: i.e. host boots, systemd kicks in with its unit PRIOR any container startup, and we get the directory created. | 12:35 |
| openstackgerrit | Merged openstack/tripleo-specs master: New spec for stein: os_tempest tripleo integration https://review.openstack.org/630654 | 12:35 |
| mandre | jaosorior: here now, what's up? | 12:36 |
| ramishra | Tengu: yeah, but it's expected to be created without the node reboot | 12:36 |
| Tengu | ramishra: so maybe it's just lacking the initial "start" once it's installed. | 12:36 |
| jaosorior | mandre: can you check this out? would this be an issue for openshift? https://review.openstack.org/#/c/630625/7 (I don't know if openshift would even use that service template at all) | 12:36 |
| ramishra | It works with docker but not podman | 12:36 |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-quickstart master: Add some fixes for libvirt https://review.openstack.org/629839 | 12:37 |
| Tengu | and that part is weird, as, afaik, there weren't any crash in the CI due to the lack of netns... so I'm wondering why it's happening now... ? | 12:37 |
| Tengu | ramishra: docker creates the directories when they are missing. | 12:37 |
| Tengu | but still...... please do NOT create that directory like that. | 12:37 |
| ramishra | Tengu: I know. btw, where would you start that service? May be we can do that in host_prep_tasks (same place as the current patch) | 12:38 |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/tripleo-ci master: Corrected openstack-virtual-baremetal repo location https://review.openstack.org/630958 | 12:38 |
| Tengu | just ensure the unit is restarted or something like that, if you want. | 12:38 |
| Tengu | ramishra: -^^ | 12:38 |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-quickstart master: Add some fixes for libvirt https://review.openstack.org/629839 | 12:39 |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-quickstart-extras master: Rewrite nodepool-setup for the zuul repro https://review.openstack.org/629872 | 12:39 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: Collect logs: Handle errors and timeout https://review.openstack.org/617617 | 12:39 |
| jaosorior | Tengu: what was the patch you had for rdo? | 12:40 |
| ramishra | Tengu: let me see what kind of issues I see with the current patch. We have made the fix distribution specific where as the issue is with podman and our templates | 12:40 |
| Tengu | jaosorior: linked in my comment on ramishra's | 12:40 |
| *** ratailor__ has quit IRC | 12:40 | |
| ramishra | I actually don't like fixing these things in rdo | 12:41 |
| mandre | jaosorior: this should be fine as far as openshift is concerned | 12:41 |
| jaosorior | mandre: alright! thanks for checking it out | 12:41 |
| Tengu | ramishra: well.... you might not need to edit my rdo patch, maybe just call things in t-h-t. | 12:42 |
| mandre | jaosorior: no, thank *you* | 12:42 |
| Tengu | as said, maybe you just need to ensure the service kicks in at least once. | 12:42 |
| *** zul has joined #tripleo | 12:43 | |
| ramishra | Tengu: having that service is a pre-req which is not good IMO | 12:44 |
| *** naturalblue has joined #tripleo | 12:45 | |
| *** morazi has joined #tripleo | 12:46 | |
| naturalblue | hey guys/gals. Does anyone know if its possible to use tripleo on a proxmox cluster. I just want to do a few POC's of openstack before changing over. Would I need to provision the VMs first or can ironic/baremetal be used to provision through proxmox. Any advce is appreciated. | 12:46 |
| *** panda is now known as panda|lunch | 12:47 | |
| Tengu | ramishra: still. it's the right way to do things. unless you want to "netns create" in t-h-t, but that makes little sens since there is the systemd thingy in place, managing it just fine. | 12:47 |
| ramishra | Tengu: I would probably disagree that it's the right thing to do, but what do I know:) | 12:49 |
| Tengu | ramishra: at least using the file ansible module is wrong for that case. | 12:49 |
| openstackgerrit | Sorin Sbarnea proposed openstack/tripleo-heat-templates master: DNM: test include_role adoption https://review.openstack.org/630962 | 12:50 |
| *** dprince has joined #tripleo | 12:52 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-common master: Workflow for provisioning bare metals without Nova https://review.openstack.org/576856 | 12:55 |
| *** chandan_kumar has quit IRC | 12:58 | |
| *** jpena|lunch is now known as jpena | 12:59 | |
| *** chandan_kumar has joined #tripleo | 12:59 | |
| *** radeks has quit IRC | 13:00 | |
| *** radeks has joined #tripleo | 13:01 | |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-quickstart-extras master: Rewrite nodepool-setup for the zuul repro https://review.openstack.org/629872 | 13:01 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/puppet-tripleo master: [WIP] Variables to reprovision pacemaker cluster one node at a time. https://review.openstack.org/630956 | 13:03 |
| openstackgerrit | Jiri Stransky proposed openstack/python-tripleoclient master: Support '--tags' for 'overcloud upgrade run' https://review.openstack.org/630967 | 13:04 |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-quickstart-extras master: Rewrite nodepool-setup for the zuul repro https://review.openstack.org/629872 | 13:07 |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-quickstart master: Add some fixes for libvirt https://review.openstack.org/629839 | 13:08 |
| *** ooolpbot has joined #tripleo | 13:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1811004 | 13:10 |
| *** ooolpbot has quit IRC | 13:10 | |
| openstack | Launchpad bug 1811004 in tripleo "tripleo-ci-centos-7-scenario003-standalone fails tempest.scenario.test_minimum_basic.TestMinimumBasicScenario" [Critical,In progress] - Assigned to Michele Baldessari (michele) | 13:10 |
| openstackgerrit | Lee Yarwood proposed openstack/puppet-tripleo master: placement: Initial extraction of the Placement service from Nova https://review.openstack.org/624335 | 13:10 |
| *** wolverineav has joined #tripleo | 13:12 | |
| *** rh-jelabarre has joined #tripleo | 13:15 | |
| *** wolverineav has quit IRC | 13:16 | |
| *** mcornea has joined #tripleo | 13:18 | |
| openstackgerrit | Adriano Petrich proposed openstack/python-tripleoclient master: WIP Remove execution from workflow message send https://review.openstack.org/630970 | 13:18 |
| *** bfournie has joined #tripleo | 13:18 | |
| *** rlandy has joined #tripleo | 13:19 | |
| *** arxcruz is now known as arxcruz|ruck | 13:21 | |
| *** radeks has quit IRC | 13:29 | |
| *** thrash|g0ne is now known as thrash | 13:29 | |
| *** trown|outtypewww is now known as trown | 13:29 | |
| quiquell | rlandy: o/ | 13:30 |
| *** gfidente has quit IRC | 13:30 | |
| rlandy | quiquell: hi | 13:30 |
| rlandy | quiquell: need to chat re:launcher | 13:30 |
| quiquell | rlandy: yep, just tell me when you are ready | 13:31 |
| rlandy | quiquell; 5 mins - just responding to upshift tenant inquiries | 13:31 |
| quiquell | rlandy: ack | 13:32 |
| openstackgerrit | Cédric Jeanneret proposed openstack/tripleo-heat-templates master: Mount system modules when calling system iptables https://review.openstack.org/630973 | 13:33 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/puppet-tripleo master: [WIP] Variables to reprovision pacemaker cluster one node at a time. https://review.openstack.org/630956 | 13:33 |
| *** udesale has quit IRC | 13:36 | |
| *** udesale has joined #tripleo | 13:36 | |
| *** lblanchard has joined #tripleo | 13:39 | |
| rlandy | quiquell: ready to meet when you are | 13:40 |
| quiquell | rlandy: ack | 13:40 |
| quiquell | rlandy: ack going to your blue | 13:41 |
| rlandy | quiquell: k | 13:41 |
| *** panda|lunch is now known as panda | 13:43 | |
| *** ade_lee has quit IRC | 13:46 | |
| openstackgerrit | Merged openstack/paunch stable/rocky: Fix comparing dictionaries for changes detection https://review.openstack.org/630750 | 13:53 |
| openstackgerrit | Lee Yarwood proposed openstack/puppet-tripleo master: placement: Initial extraction of the Placement service from Nova https://review.openstack.org/624335 | 13:55 |
| weshay | jaosorior, mwhahaha so.. the podman error continues to haunt gate and promotion jobs :( | 13:55 |
| openstackgerrit | Lee Yarwood proposed openstack/puppet-tripleo master: placement: Initial extraction of the Placement service from Nova https://review.openstack.org/624335 | 13:56 |
| weshay | will mention it the mtg today I guess | 13:56 |
| *** raildo has quit IRC | 13:56 | |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-common master: Add a role for data presistence and restoration https://review.openstack.org/630921 | 13:59 |
| jaosorior | #startmeeting tripleo | 14:00 |
| openstack | Meeting started Tue Jan 15 14:00:37 2019 UTC and is due to finish in 60 minutes. The chair is jaosorior. Information about MeetBot at http://wiki.debian.org/MeetBot. | 14:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 14:00 |
| *** openstack changes topic to " (Meeting topic: tripleo)" | 14:00 | |
| openstack | The meeting name has been set to 'tripleo' | 14:00 |
| jaosorior | #topic agenda | 14:00 |
| *** openstack changes topic to "agenda (Meeting topic: tripleo)" | 14:00 | |
| jaosorior | * Review past action items | 14:00 |
| jaosorior | * One off agenda items | 14:00 |
| jaosorior | * Squad status | 14:00 |
| jaosorior | * Bugs & Blueprints | 14:00 |
| jaosorior | * Projects releases or stable backports | 14:00 |
| jaosorior | * Specs | 14:00 |
| jaosorior | * open discussion | 14:00 |
| jaosorior | Anyone can use the #link, #action and #info commands, not just the moderatorǃ | 14:01 |
| jaosorior | Hey folks! who's around? | 14:01 |
| dpeacock | o/ | 14:01 |
| lyarwood | \o | 14:01 |
| abishop | o/ | 14:01 |
| marios | o/ | 14:01 |
| beagles | o/ | 14:01 |
| jrist | o/~ | 14:01 |
| ccamacho | o/ | 14:02 |
| rfolco | o/ | 14:02 |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-common master: Add a role for data presistence and restoration https://review.openstack.org/630921 | 14:03 |
| *** raildo has joined #tripleo | 14:03 | |
| jaosorior | #topic review past action items | 14:04 |
| *** openstack changes topic to "review past action items (Meeting topic: tripleo)" | 14:04 | |
| jaosorior | None! :D | 14:04 |
| Tengu | lazy. | 14:04 |
| Tengu | ;) | 14:04 |
| jaosorior | :P | 14:04 |
| jaosorior | #topic one off agenda items | 14:04 |
| *** openstack changes topic to "one off agenda items (Meeting topic: tripleo)" | 14:04 | |
| jaosorior | First one: (lyarwood) Would it be possible to upload the placement-api container to docker.io earliy ahead of the next TripleO promotion? | 14:05 |
| jaosorior | Oh, sorry: | 14:05 |
| jaosorior | #link https://etherpad.openstack.org/p/tripleo-meeting-items | 14:05 |
| lyarwood | ^ yeah so this came up in here yesterday and after speaking to the folks in #oooq they wanted me to bring it up in this meeting | 14:05 |
| lyarwood | it shouldn't have any impact on CI as nothing is currently using it | 14:06 |
| lyarwood | it just unblocks my work https://review.openstack.org/#/q/topic:tripleo-placement-extraction on extracting placement | 14:06 |
| jaosorior | lyarwood: that's fine on my side. I'm unsure on what the process is on uploading the image though. marios do you know? | 14:07 |
| *** apetrich has quit IRC | 14:07 | |
| lyarwood | jaosorior: panda is aware, I think they just wanted the idea to be blessed during this meeting | 14:07 |
| jaosorior | nice, if panda knows how to do it, lets get that work going. thanks for bringing it up lyarwood | 14:08 |
| lyarwood | jaosorior: ack np, thanks! | 14:08 |
| jaosorior | panda, weshay: what's the state of the promotions though? | 14:08 |
| marios | jaosorior: not been following that work (new promotion images) and yeah panda did that last time we needed :) | 14:08 |
| weshay | shi tay ? | 14:08 |
| marios | jaosorior: (via http://dashboard-ci.tripleo.org/d/cEEjGFFmz/cockpit?orgId=1 ) maste didn't promote since new years | 14:09 |
| panda | jaosorior: I can push the image to the repo | 14:09 |
| jaosorior | panda: thanks | 14:09 |
| lyarwood | panda: thanks :) | 14:09 |
| *** ansmith has joined #tripleo | 14:10 | |
| panda | lyarwood: if you have the image built and pushed somewhere we'll get it done fast | 14:10 |
| *** ooolpbot has joined #tripleo | 14:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1811004 | 14:10 |
| openstack | Launchpad bug 1811004 in tripleo "tripleo-ci-centos-7-scenario003-standalone fails tempest.scenario.test_minimum_basic.TestMinimumBasicScenario" [Critical,In progress] - Assigned to Michele Baldessari (michele) | 14:10 |
| *** ooolpbot has quit IRC | 14:10 | |
| *** tzumainn has joined #tripleo | 14:10 | |
| lyarwood | panda: I don't, I'll move over to #oooq to talk to you about this now quickly | 14:10 |
| panda | lyarwood: ok | 14:10 |
| openstackgerrit | Nagasai Vinaykumar Kapalavai proposed openstack/tripleo-heat-templates master: Update to the ceilometer publisher list https://review.openstack.org/600843 | 14:11 |
| jaosorior | #topic Squad status | 14:12 |
| *** openstack changes topic to "Squad status (Meeting topic: tripleo)" | 14:12 | |
| jaosorior | ci | 14:12 |
| jaosorior | #link https://etherpad.openstack.org/p/tripleo-ci-squad-meeting | 14:12 |
| *** mmethot has joined #tripleo | 14:12 | |
| jaosorior | upgrade | 14:12 |
| jaosorior | #link https://etherpad.openstack.org/p/tripleo-upgrade-squad-status | 14:12 |
| jaosorior | containers | 14:12 |
| jaosorior | #link https://trello.com/b/S8TmOU0u/tripleo-podman | 14:12 |
| jaosorior | integration | 14:12 |
| jaosorior | #link https://etherpad.openstack.org/p/tripleo-integration-squad-status | 14:12 |
| *** apetrich has joined #tripleo | 14:12 | |
| jaosorior | ui/cli | 14:12 |
| jaosorior | #link https://etherpad.openstack.org/p/tripleo-ui-cli-squad-status | 14:13 |
| jaosorior | validations | 14:13 |
| jaosorior | #link https://etherpad.openstack.org/p/tripleo-validations-squad-status | 14:13 |
| jaosorior | networking | 14:13 |
| jaosorior | #link https://etherpad.openstack.org/p/tripleo-networking-squad-status | 14:13 |
| jaosorior | security | 14:13 |
| jaosorior | #link https://etherpad.openstack.org/p/tripleo-security-squad | 14:13 |
| jaosorior | edge | 14:13 |
| jaosorior | #link https://etherpad.openstack.org/p/tripleo-edge-squad-status | 14:13 |
| jaosorior | Any squad wanting to bring up their status, or a topic for the general public? | 14:13 |
| jaosorior | #topic bugs & blueprints | 14:15 |
| *** openstack changes topic to "bugs & blueprints (Meeting topic: tripleo)" | 14:15 | |
| jaosorior | #link https://launchpad.net/tripleo/+milestone/stein-3 | 14:15 |
| jaosorior | For Stein we currently have 27 (for stein-2) and 4 (for stein-3) blueprints open in Launchpad. | 14:15 |
| jaosorior | Bugs: 619 stein-3. 105 (+1) open Storyboard bugs. | 14:15 |
| openstack | bug 619 in a2ps (Ubuntu) "-Pdisplay does not work" [Medium,Fix released] https://launchpad.net/bugs/619 - Assigned to MOTU (motu) | 14:15 |
| jaosorior | #link https://storyboard.openstack.org/#!/project_group/76 | 14:15 |
| jaosorior | Please folks, move your blueprints from stein-2 to stein-3 | 14:15 |
| jaosorior | If your blueprint is still being worked on. | 14:15 |
| *** gfidente has joined #tripleo | 14:16 | |
| *** psachin has joined #tripleo | 14:16 | |
| openstackgerrit | Ben Nemec proposed openstack/openstack-virtual-baremetal master: Fix pep8 error https://review.openstack.org/630983 | 14:16 |
| openstackgerrit | Ben Nemec proposed openstack/openstack-virtual-baremetal master: Add .gitreview https://review.openstack.org/630984 | 14:16 |
| jaosorior | #topic projects releases or stable backports | 14:17 |
| *** openstack changes topic to "projects releases or stable backports (Meeting topic: tripleo)" | 14:17 | |
| jaosorior | We just had a release for stein-2 | 14:17 |
| jaosorior | Big thanks to EmilienM for doing that | 14:17 |
| jaosorior | #topic specs | 14:19 |
| *** openstack changes topic to "specs (Meeting topic: tripleo)" | 14:19 | |
| jaosorior | #link https://review.openstack.org/#/q/project:openstack/tripleo-specs+status:open | 14:19 |
| abishop | there are (new?) issues installing stable/rocky undercloud | 14:19 |
| EmilienM | jaosorior: my pleasure | 14:19 |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-common master: nova-less-deploy: deploy_instances and undeploy_instances workflows https://review.openstack.org/576856 | 14:19 |
| abishop | (I have details, not sure where we are in mtg) | 14:20 |
| jaosorior | #topic open discussion | 14:20 |
| *** openstack changes topic to "open discussion (Meeting topic: tripleo)" | 14:20 | |
| jaosorior | Anything else that folks want to bring up to the meeting? | 14:20 |
| jaosorior | abishop: I think you could bring it up now. | 14:20 |
| abishop | looking at https://bugs.launchpad.net/tripleo/+bug/1811713, found it's an selinux issue I can fix by adding ":z" to /etc/iscsi mount in puppet_config | 14:20 |
| openstack | Launchpad bug 1811713 in tripleo "rocky undercloud fails to install" [High,Triaged] | 14:20 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/puppet-tripleo master: [WIP] Variables to reprovision pacemaker cluster one node at a time. https://review.openstack.org/630956 | 14:20 |
| abishop | I _think_ it's due to https://review.openstack.org/625262 | 14:20 |
| abishop | looking at that patch, I see ci's undercloud.conf sets "undercloud_enable_selinux = false" :-/ | 14:20 |
| abishop | then, but last night released new python2-tripleo-repos rpm, and today I get another error installing rocky undercloud | 14:21 |
| abishop | "stderr: chown: cannot read directory '/var/log/rabbitmq': Permission denied" | 14:21 |
| jaosorior | EmilienM: did you mean to backport https://review.openstack.org/#/c/625262/ to rocky? | 14:22 |
| EmilienM | jaosorior: yes | 14:22 |
| *** shyamb has joined #tripleo | 14:22 | |
| EmilienM | it actually enables selinux in containers | 14:23 |
| jaosorior | EmilienM: while I'm generally happy about us enabling SELinux. Seems to be like rocky wasn't quite ready. At least from the bug that abishop is talking about. | 14:23 |
| abishop | http://logs.openstack.org/62/625262/4/check/tripleo-ci-centos-7-undercloud-oooq/6d70b06/logs/undercloud/home/zuul/undercloud.conf.txt.gz | 14:23 |
| abishop | doesn't test with undercloud_enable_selinux = true | 14:24 |
| openstackgerrit | mathieu bultel proposed openstack/tripleo-common master: Catch heat exception in package_update action https://review.openstack.org/630986 | 14:24 |
| EmilienM | abishop: I doubt CI has selinux enforced | 14:24 |
| shyamb | Hi | 14:26 |
| jaosorior | EmilienM: yeah, from what I can tell it's not enabled on CI. I still don't know where we're setting that. | 14:26 |
| weshay | jaosorior, we're in open discussion still? | 14:26 |
| jaosorior | weshay: yeah | 14:26 |
| weshay | k.. just want to highlight for folks.. http://status.openstack.org/elastic-recheck/ we've been able to merge queries and it's very helpful | 14:26 |
| jaosorior | EmilienM, Tengu: any chance any of you could take a look at the issue that abishop is reporting? | 14:27 |
| weshay | in understanding why your ci job fail | 14:27 |
| shyamb | jaosorior: In one of my custom openstack service container I am mounting nfs share | 14:27 |
| weshay | http://status.openstack.org/elastic-recheck/#1805186 and http://status.openstack.org/elastic-recheck/#1804434 are the two top issues I'm seeing atm for us | 14:27 |
| abishop | to recap, I could easily reproduce https://bugs.launchpad.net/tripleo/+bug/1811713 | 14:28 |
| openstack | Launchpad bug 1811713 in tripleo "rocky undercloud fails to install" [High,Triaged] | 14:28 |
| weshay | abishop, what dlrn hash are you using? | 14:28 |
| abishop | yesterday, I could fix by adding ":z" to iscsid's puppet_config | 14:28 |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-common master: Add a role for data presistence and restoration https://review.openstack.org/630921 | 14:28 |
| shyamb | we want this share to be accesible on host also | 14:28 |
| abishop | weshay: yesterday, the one in the lp | 14:28 |
| shyamb | how can I do that? Any ideas? | 14:28 |
| abishop | today there's a new one, and I get another error even w/ my local fix | 14:28 |
| shyamb | bind mount does the vice versa | 14:28 |
| weshay | ah k | 14:29 |
| shyamb | jaosorior: Any ideas? | 14:29 |
| jaosorior | shyamb: not at the moment really. | 14:29 |
| weshay | abishop, can you double check for me if that ends up being https://trunk.rdoproject.org/centos7-rocky/current-tripleo/ | 14:29 |
| shyamb | jaosorior: ok | 14:30 |
| openstackgerrit | Ben Nemec proposed openstack/openstack-virtual-baremetal master: Fix pep8 error https://review.openstack.org/630983 | 14:30 |
| openstackgerrit | Ben Nemec proposed openstack/openstack-virtual-baremetal master: Add .gitreview https://review.openstack.org/630984 | 14:30 |
| openstackgerrit | Ben Nemec proposed openstack/openstack-virtual-baremetal master: Add Zuul config https://review.openstack.org/630993 | 14:30 |
| abishop | weshay: I may need to poke around (I don't typically watch that stuff), but it's python2-tripleo-repos-0.0.1-0.20190114223251.6470f0c.el7.noarch followed by " sudo -E tripleo-repos -b rocky current ceph" | 14:31 |
| *** vinaykns has joined #tripleo | 14:33 | |
| abishop | weshay: full url https://trunk.rdoproject.org/centos7/current/python2-tripleo-repos-0.0.1-0.20190114223251.6470f0c.el7.noarch.rpm | 14:33 |
| *** two_oes has quit IRC | 14:35 | |
| *** moguimar has quit IRC | 14:36 | |
| *** moguimar has joined #tripleo | 14:36 | |
| openstackgerrit | Cédric Jeanneret proposed openstack/tripleo-heat-templates master: Mount system modules when calling system iptables https://review.openstack.org/630973 | 14:37 |
| openstackgerrit | Nagasai Vinaykumar Kapalavai proposed openstack/tripleo-heat-templates master: metrics_qdr: Add TLS support https://review.openstack.org/630736 | 14:39 |
| jaosorior | #endmeeting | 14:39 |
| *** openstack changes topic to "CI Status: Green | https://docs.openstack.org/tripleo-docs/latest/" | 14:39 | |
| openstack | Meeting ended Tue Jan 15 14:39:46 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 14:39 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/tripleo/2019/tripleo.2019-01-15-14.00.html | 14:39 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/tripleo/2019/tripleo.2019-01-15-14.00.txt | 14:39 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/tripleo/2019/tripleo.2019-01-15-14.00.log.html | 14:39 |
| rfolco | CI community meeting (office hours) starts now at https://bluejeans.com/4113567798 | 14:40 |
| aedc | [stack@manager ~]$ grep container_images_file undercloud.conf | 14:41 |
| aedc | container_images_file = $HOME/containers-prepare-parameter.yaml | 14:41 |
| aedc | [stack@manager ~]$ openstack undercloud install | 14:41 |
| aedc | no such option container_images_file in group [DEFAULT] | 14:41 |
| aedc | Normal for you ? | 14:41 |
| aedc | => python-tripleoclient-10.6.1-0.20190110030345.ce7f606.el7.noarch | 14:42 |
| vinaykns | hello channel, I'm working on adding ssl support to metrics_qdr service. Need review for the patch https://review.openstack.org/#/c/630736/ | 14:42 |
| EmilienM | mwhahaha: on our TODO for later, we'll probably need to update THT to build puppet5 compatible hieradata https://review.openstack.org/#/c/630977/1/functions | 14:45 |
| openstackgerrit | Cédric Jeanneret proposed openstack/tripleo-heat-templates master: Ensure we get the file directly in the container when running podman https://review.openstack.org/630287 | 14:46 |
| weshay | abishop, cat /etc/yum.repos.d/delorean.repo | 14:46 |
| openstackgerrit | Cédric Jeanneret proposed openstack/tripleo-heat-templates master: DNM - podman testing https://review.openstack.org/628379 | 14:47 |
| jaosorior | lyarwood: around? | 14:47 |
| openstackgerrit | Cédric Jeanneret proposed openstack/tripleo-heat-templates master: DNM - podman testing https://review.openstack.org/628380 | 14:47 |
| openstackgerrit | Cédric Jeanneret proposed openstack/tripleo-heat-templates master: DNM - podman testing https://review.openstack.org/628381 | 14:47 |
| lyarwood | jaosorior: yes | 14:48 |
| abishop | weshay: that python2-tripleo-repos I mentioned has delorean-rocky.repo with baseurl=https://trunk.rdoproject.org/centos7-rocky/22/a4/22a47cd90380b3fd0ea5d256523264f1906e8525_df8abf35 | 14:48 |
| jaosorior | lyarwood: I realize I made the wrong assumptions when saying that it would be fine to just push the placement container to the registry. | 14:48 |
| lyarwood | jaosorior: ah, how so? | 14:48 |
| jaosorior | lyarwood: How critical is that container to keep working on what you're doing? Do you have that container built somewhere already? | 14:49 |
| lyarwood | jaosorior: I don't at the moment, I looked into building it locally this morning but it would be much easier to have something I could use in CI tbh | 14:50 |
| jaosorior | lyarwood: If you have that container built somewhere already and have already tested it out. Then sure, it could be pushed out. | 14:50 |
| jaosorior | lyarwood: however, the CI team is quite overloaded at the moment. And if they're the ones that have to build the container manually. I would rather let them focus on fixing the current issues we're dealing with. | 14:50 |
| lyarwood | jaosorior: I really don't get that logic tbh, nothing would use the container outside of my changes | 14:50 |
| aedc | ok, it's my fault, but the message is not very explicit ;) | 14:51 |
| lyarwood | jaosorior: yeah I appreciate that | 14:51 |
| jaosorior | I understand that; all I'm saying here is that I would let the CI team focus on the current critial issues (e.g. promotion) | 14:51 |
| jaosorior | lyarwood: if you can build that container and pass it on to panda, and if the new container is only used by your patches. Then sure, pushing it to the registry could be arranged. | 14:52 |
| *** artom has quit IRC | 14:52 | |
| lyarwood | jaosorior: kk, I'll go and try this again | 14:52 |
| lyarwood | jaosorior: there's zero docs for this outside of the tripleo-common zuul jobs right? | 14:52 |
| *** shyamb has quit IRC | 14:53 | |
| lyarwood | jaosorior: building overcloud containers from kolla that is | 14:53 |
| jaosorior | lyarwood: there are <some> docs in tripleo.org about that. | 14:53 |
| lyarwood | jaosorior: ack, I'll try to hunt them down, thanks! | 14:53 |
| * lyarwood missed them earlier | 14:53 | |
| jaosorior | lyarwood: http://tripleo.org/install/containers_deployment/3rd_party.html?highlight=kolla%20build#building-new-containers-with-kolla-build | 14:54 |
| lyarwood | jaosorior: awesome, thanks! | 14:55 |
| *** artom has joined #tripleo | 14:56 | |
| openstackgerrit | Natal Ngétal proposed openstack/paunch master: [Configuration] Switch to stestr. https://review.openstack.org/629421 | 14:57 |
| openstackgerrit | mathieu bultel proposed openstack/python-tripleoclient master: Use a single queue for update/upgrade https://review.openstack.org/629103 | 14:58 |
| mwhahaha | dprince: can you review https://review.openstack.org/#/c/614639/ as a work around for our podman exec failure issues. I think ultimately we want https://review.openstack.org/#/c/630287/ but in the short term to stop the gate/promotion failures i think we need a retry | 14:58 |
| dprince | mwhahaha: on it | 14:58 |
| dtantsur | oh god, podman doesn't have the cp command, does it? | 14:59 |
| Tengu | dtantsur: nope it doesn't. | 14:59 |
| Tengu | dtantsur: you need to podman create, podman mount, mkdir/cp, podman umount, podman start | 15:00 |
| *** artom has quit IRC | 15:00 | |
| dtantsur | Tengu: do we at least use normal 'podman start' now? I recall 'docker start' did not work, because we bolted a lot of things on top of it. | 15:00 |
| *** artom has joined #tripleo | 15:00 | |
| Tengu | dtantsur: I'm doing that here if you want an example: https://review.openstack.org/#/c/630287/2/docker/docker-puppet.py | 15:00 |
| Tengu | dtantsur: at least podman start seems to work in my case. | 15:01 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/puppet-tripleo master: [WIP] Variables to reprovision pacemaker cluster one node at a time. https://review.openstack.org/630956 | 15:01 |
| Tengu | dtantsur: *maybe* a "podman cp" will be done - it will basically do the mount/mkdir|cp/umount under the hood. | 15:01 |
| dtantsur | yeah, but I need to figure out which arguments to pass | 15:01 |
| dtantsur | i.e. I need to hack on an existing container | 15:02 |
| dprince | mwhahaha: so hey, I get why you want it. But could we add a 'retry' option to docker-puppet.py and only use it in the cases that seem to be problematic? | 15:02 |
| Tengu | dtantsur: hmmmm so in that case I'm not sure you can mount a running container | 15:02 |
| dtantsur | and I guess I cannot mount something to a running container, can I? | 15:02 |
| mwhahaha | dprince: you mean, all podman interactions | 15:02 |
| Tengu | dtantsur: :D | 15:02 |
| *** waleedm has quit IRC | 15:02 | |
| mwhahaha | dprince: ie, there | 15:02 |
| dprince | mwhahaha: the general retry 5 times thing concerns me with podman | 15:02 |
| Tengu | dtantsur: you can try: podman mount <container name|id> it will return you the mounted path | 15:02 |
| dtantsur | for love's sake... the development for tripleo was barely possible even with docker... | 15:02 |
| dprince | mwhahaha: sorry, perhaps I misunderstood. I thought it the retry was only needed in certain cases | 15:03 |
| Tengu | dtantsur: well, if you have to push files inside a running container, maybe there's an issue elsewhere? | 15:03 |
| dtantsur | Tengu: I'm not sure I get it. I'm trying to test a patch against tripleo-common.. | 15:03 |
| Tengu | oh | 15:03 |
| dtantsur | you know, that thing that takes `sudo pip install -e .` usually :) | 15:03 |
| mwhahaha | dprince: it's needed when the container launch fails because of the exec unable ot find file, which can happen for any of the entry point things. I don't like the retry in principle because we shouldn't be having this but we've suffered for weeks because of it | 15:03 |
| Tengu | dtantsur: if you want I'm in Bridge room, just come over :). | 15:03 |
| dtantsur | Tengu: Bridge room where? :D | 15:04 |
| Tengu | dtantsur: Brno, old building | 15:04 |
| dtantsur | Tengu: Brno is 7 hours by train nowadays :) will you stay there next week? | 15:04 |
| dprince | mwhahaha: well, my concern is this will also slow down general error detection in the case where we do have an error in puppet, etc. | 15:04 |
| *** ykarel has quit IRC | 15:04 | |
| dtantsur | I'll come for devconf.cz | 15:04 |
| Tengu | dtantsur: oh, I thought you were in Brno - sorry! | 15:05 |
| mwhahaha | dprince: it shouldn't because if it's a legit failure it fails quickly | 15:05 |
| mwhahaha | dprince: can you think of a case where the launching takes an extended period of time? | 15:05 |
| Tengu | dtantsur: I'm leaving on Friday ;). | 15:05 |
| dtantsur | Tengu: used to be. moved to Berlin a bit more than a year ago. no worries, I don't think everyone is up-to-date on my location :) | 15:05 |
| Tengu | dtantsur: :) | 15:05 |
| dprince | mwhahaha: not always, like the keystone init tasks take awhile. retrying those would be painful | 15:05 |
| Tengu | anyway. back to your issue: try to mount the running container - afaik it should be OK for a running container, as it mounts an overlay. | 15:06 |
| Tengu | dtantsur: just tested - you can mount a running container :D | 15:06 |
| mwhahaha | dprince: i'm open to alternative suggestions but this failure rate is terrible http://status.openstack.org/elastic-recheck/index.html#1804434 | 15:06 |
| * dtantsur dances \o/ | 15:06 | |
| Tengu | dtantsur: so basically: podman mount <container name|id>, copy the things you want in it, and "podman umount <container name|id>" | 15:07 |
| dprince | mwhahaha: thanks for pointing it out. let me ponder a bit | 15:07 |
| *** quiquell is now known as quiquell|off | 15:07 | |
| Tengu | dtantsur: fact is - I find the podman way for that far more usable and elegant than the docker cp. | 15:07 |
| dtantsur | well, if it works - I'll agree | 15:07 |
| Tengu | ;) | 15:07 |
| dtantsur | hmm, I see some files | 15:07 |
| dtantsur | Tengu: sad I'm not in Brno, I'd give you a beer for your discovery, thanks! | 15:08 |
| Tengu | ;) | 15:08 |
| Tengu | next time, no problem :D | 15:08 |
| EmilienM | dtantsur: he'll get his beer no worries ;-) | 15:08 |
| Tengu | :D | 15:09 |
| dtantsur | heh, I hope so | 15:09 |
| dtantsur | I loved it when so many familiar faces were in Berlin last November | 15:09 |
| Tengu | wasn't there :(. but this year I'll try to get PTG and other things :). | 15:09 |
| dtantsur | Tengu: so, just to check my understanding: 'podman mount' mounts the whole filesystem of the container to a host location? | 15:10 |
| *** ooolpbot has joined #tripleo | 15:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1811004 | 15:10 |
| openstack | Launchpad bug 1811004 in tripleo "tripleo-ci-centos-7-scenario003-standalone fails tempest.scenario.test_minimum_basic.TestMinimumBasicScenario" [Critical,In progress] - Assigned to Michele Baldessari (michele) | 15:10 |
| *** ooolpbot has quit IRC | 15:10 | |
| Tengu | dtantsur: ah, you probably won't be able to "podman umount" in fact, as it's a running container | 15:10 |
| dtantsur | Tengu: I was able to podman umount, I hope I did not break everything :D | 15:10 |
| Tengu | dtantsur: ok. well, maybe it was only on my side. I never did it before ,). | 15:11 |
| Tengu | and MAYBE it's due to the fact I'm using latest podman 1.0 | 15:11 |
| Tengu | :) | 15:11 |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-quickstart master: Add some fixes for libvirt https://review.openstack.org/629839 | 15:16 |
| dtantsur | Tengu: I feel bad, but this works: http://paste.openstack.org/show/742646/ | 15:17 |
| * dtantsur <3 podman more than docker for now | 15:18 | |
| Tengu | hehehe | 15:18 |
| Tengu | at least we have ONE dude happy with docker :D | 15:18 |
| Tengu | EmilienM: -^^ | 15:19 |
| Tengu | **podman | 15:19 |
| dtantsur | well, I used to have retries for 'docker cp' in this script, because it kept failing 2 times of 3 for now reason.. | 15:19 |
| Tengu | uhuhu | 15:19 |
| dtantsur | I guess it's Web Scale (tm)? | 15:20 |
| *** paramite has quit IRC | 15:22 | |
| Tengu | jaosorior: any thoughts on https://bugs.launchpad.net/tripleo/+bug/1810422 ? | 15:23 |
| openstack | Launchpad bug 1810422 in tripleo "SELinux context is wrong after reboot for haproxy logs" [Medium,Triaged] - Assigned to Cédric Jeanneret (cjeanner) | 15:23 |
| *** ykarel has joined #tripleo | 15:23 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: WIP: Reload haproxy when certificate is renewed https://review.openstack.org/623353 | 15:25 |
| *** aufi has quit IRC | 15:25 | |
| Tengu | jaosorior: hmmmm we actually do need to reload haproxy upon certificate change? I'm not that sure in fact. | 15:25 |
| *** aufi has joined #tripleo | 15:26 | |
| jaosorior | Tengu: we do, haproxy doesn't do it by itself. | 15:26 |
| Tengu | jaosorior: hmm ok. maybe newest version are more clever. | 15:26 |
| Tengu | jaosorior: anyway - do you have a minute for the issue I just linked? Just wanting to get a second point of view on that one :) | 15:27 |
| openstackgerrit | David Peacock proposed openstack/tripleo-heat-templates master: flatten sahara service configuration https://review.openstack.org/625270 | 15:28 |
| jaosorior | Tengu: gimme 5min | 15:28 |
| openstackgerrit | Alex Schultz proposed openstack/paunch stable/queens: Fix comparing dictionaries for changes detection https://review.openstack.org/631005 | 15:28 |
| Tengu | jaosorior: thanks :) | 15:28 |
| openstackgerrit | David Peacock proposed openstack/tripleo-heat-templates master: flatten cinder service configuration https://review.openstack.org/626937 | 15:34 |
| dtantsur | hjensas: do you have an idea why a neutron port would not receive an IP (like, any fixed_ips)? My undercloud may not be able to deploy because of it.. | 15:37 |
| *** abishop is now known as abishop|afk | 15:37 | |
| dtantsur | Cannot apply dhcp option server-ip- | 15:38 |
| dtantsur | address because it's ip_version 4 is not in port's address IP versions | 15:38 |
| hjensas | dtantsur: if the port is on a routed provider network ip allocation can be deffered if not physical-network attribute is set. | 15:38 |
| hjensas | dtantsur: basically neutron does | 15:38 |
| dtantsur | hjensas: well, it's the default undercloud | 15:38 |
| *** sudeepbatra has joined #tripleo | 15:39 | |
| dtantsur | whatever we set by default is set | 15:39 |
| hjensas | 'nt know which network segment. So it need to wait until it knows. Undercloud does not have routed provider network by default. | 15:39 |
| hjensas | dtantsur: but, it's you undercloud that is'nt deploying? | 15:39 |
| jaosorior | Tengu: does the haproxy package come with syslog rules? | 15:39 |
| dtantsur | hjensas: thinking about it, I might have enabled routed networks previously | 15:40 |
| dtantsur | I'll redeploy without it. but in any case it's quite dangerous: deployment is broken by default with it | 15:40 |
| Tengu | jaosorior: nah, we push them from t-h-t. | 15:41 |
| hjensas | dtantsur: it depends how you import nodes. I think if you don't provide any port data in instackenv.json, and leave it to introspection to discover ports you could end up without physnet info on the bm ports. | 15:41 |
| dtantsur | hjensas: the default quickstart, nothing added by me | 15:42 |
| Tengu | jaosorior: so basically, I've made things so that rsyslog writes haproxy container logs in the /var/log/containers, but the setype doesn't allow that, and there's probably a rule that ensure the setype is set to container_file_t | 15:42 |
| dtantsur | hjensas: it enables routed_networks by default | 15:42 |
| weshay | abishop|afk, ya.. so.. I'll have to go poke at the tripleo's repo code.. but if you are blocked use hash https://trunk.rdoproject.org/centos7-rocky/current-tripleo/delorean.repo | 15:42 |
| dtantsur | which makes me wonder if anybody actually uses quickstart :D | 15:42 |
| weshay | abishop|afk, that one will work | 15:42 |
| Tengu | jaosorior: so this prevent rsyslog to write in the location - that's why I propose to move the haproxy logs back in /var/log/ directly, and symlink it into containers/ location. Like we've done for swift. | 15:42 |
| weshay | ugh.. how do I stumble into these | 15:42 |
| weshay | dtantsur, what's the issue? | 15:43 |
| dtantsur | weshay: quickstart enables routed networks by default, but the generated instackenv.json is not sufficient for deployment with routed networks to work | 15:43 |
| hjensas | dtantsur: :D yes, I think I enabled it to ensure we get a little bit of coverage on that path in CI. (But I thought I only did that for CI ... not the default quickstart.) | 15:43 |
| dtantsur | at least that's what hjensas just told me | 15:43 |
| dtantsur | hjensas: the generated instackenv.json only has MACs | 15:43 |
| weshay | which change hjensas ? | 15:44 |
| *** mjturek has joined #tripleo | 15:44 | |
| hjensas | dtantsur: then it should set ctlplane as physnet by default. | 15:44 |
| dtantsur | hjensas: oh, it actually does. which gets us back to the initial question: why wouldn't neutron assign an IP to a port? | 15:44 |
| dtantsur | weshay: hold on, something else is broken in the default quickstart | 15:45 |
| hjensas | weshay: long time, back in queens time. I can dig, but ^^ :) | 15:45 |
| weshay | dtantsur, k.. hrm.. queens eh.. k | 15:45 |
| *** salmankhan has quit IRC | 15:45 | |
| dtantsur | all I can say is that Ironic seems to be broken on my deployment | 15:46 |
| *** fragatina has joined #tripleo | 15:46 | |
| dtantsur | I can try rebuilding it with the latest quickstart, but it's pretty new | 15:46 |
| jaosorior | Tengu: well, I think we do gotta find out what's setting it | 15:46 |
| weshay | our latest jobs on queens https://review.openstack.org/#/c/567224/ | 15:46 |
| weshay | dtantsur, this is related to ironic? | 15:46 |
| dtantsur | weshay: it's related to something we broke in the undercloud. | 15:47 |
| Tengu | jaosorior: I suspect some selinux policy applied on directory - I don't remember the name. sefile? se.... damn. | 15:47 |
| dtantsur | weshay: tl;dr neutron creates ports without IP addresses, so no deployment is possible | 15:47 |
| weshay | k.. I saw some bug on that.. I think | 15:47 |
| weshay | which settings in the undercloud? | 15:47 |
| weshay | btw | 15:47 |
| Tengu | jaosorior: fcontext. wasn't that far. | 15:47 |
| dtantsur | hjensas: actually you can reproduce it by just creating a port | 15:47 |
| dtantsur | try it if you have an undercloud | 15:48 |
| weshay | https://logs.rdoproject.org/24/567224/168/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/73c22e1/logs/undercloud/home/zuul/undercloud.conf.txt.gz | 15:48 |
| hjensas | dtantsur: without routed networks enabled as well? | 15:48 |
| dtantsur | hjensas: I don't know, I don't have an environment without them | 15:48 |
| hjensas | dtantsur: me neiter. But if routed networks is enabled, a port create won't allocate an IP unless enough binding info is provided. | 15:49 |
| Tengu | jaosorior: damn. apparently nothing shows up with semanage fcontext -l -.-' | 15:49 |
| weshay | # Enable support for routed ctlplane networks | 15:49 |
| weshay | enable_routed_networks = true | 15:49 |
| openstackgerrit | mathieu bultel proposed openstack/tripleo-common master: Catch heat exception in package_update action https://review.openstack.org/630986 | 15:50 |
| EmilienM | mwhahaha, dprince, Tengu : the only alternative I have is https://review.openstack.org/629546 | 15:50 |
| hjensas | dtantsur: If you provide the the subnet to use in a fixed_ips request for the port it will allocate an ip. | 15:50 |
| EmilienM | mwhahaha, dprince, Tengu : which increases the deployment time a lot | 15:50 |
| Tengu | EmilienM: did you hit that thing with the processcount set to 3? | 15:51 |
| EmilienM | Tengu: yes | 15:51 |
| mwhahaha | i would rather add retries | 15:51 |
| Tengu | sh*t. | 15:51 |
| mwhahaha | because then we might take longer to fail | 15:51 |
| hjensas | dtantsur: or you need to add the binding info, which compute and then neutron will look up the physnet mapping and allocate from a subnet in a network segment where the "hypervisor" is connected. | 15:51 |
| mwhahaha | then increasing deployment time | 15:51 |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: support cracklib in pam for Gentoo's musl profile https://review.openstack.org/631009 | 15:51 |
| dprince | EmilienM: have you tried a middle ground number there | 15:52 |
| dprince | EmilienM: like 3. I sometimes use that locally on my slower boxes | 15:52 |
| Tengu | dprince: yep he tried, and got the issue | 15:52 |
| EmilienM | dprince: 3 failed | 15:52 |
| Tengu | that's what I asked just before ^^' | 15:52 |
| dprince | ouch, podman!!! | 15:53 |
| Tengu | dprince: :D | 15:53 |
| dprince | EmilienM: this isn't good. It means podman isn't ready I think | 15:53 |
| hjensas | dtantsur: http://paste.openstack.org/show/742656/ | 15:53 |
| EmilienM | dprince: yeah so what? | 15:54 |
| EmilienM | we always knew it was unstable | 15:54 |
| EmilienM | but tried to have it tested anyway so we can iterate | 15:54 |
| *** dciabrin_ has joined #tripleo | 15:54 | |
| EmilienM | it isn't new that we have these kind of issues with podman | 15:54 |
| dprince | EmilienM: so maybe go back to docker until we resolve it? | 15:54 |
| EmilienM | we have managed to get podman tested almost everywhere in our CI | 15:55 |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/tripleo-ci master: Corrected openstack-virtual-baremetal repo location https://review.openstack.org/630958 | 15:55 |
| EmilienM | and switched tripleoclient to deploy podman by default | 15:55 |
| EmilienM | it would be a huge step backward just for one bug | 15:55 |
| EmilienM | if Tengu's patch can work, we'll go that route | 15:55 |
| EmilienM | but we need a temporary solution | 15:55 |
| *** ksambor has quit IRC | 15:56 | |
| Tengu | hm. tbh I'm not sure my patch is better..... and I have the feeling it will probably create other issues. | 15:56 |
| *** Vorrtex has joined #tripleo | 15:57 | |
| *** moguimar has quit IRC | 15:57 | |
| *** radeks has joined #tripleo | 15:57 | |
| dprince | EmilienM: do we have fine grained data on which docker-puppet.py tasks are failing? Like is it totally random? | 15:57 |
| mwhahaha | totally random | 15:57 |
| Tengu | result is rand(rand(container)) | 15:57 |
| *** morazi has quit IRC | 15:57 | |
| dtantsur | hjensas: well, there is only one subnet | 15:58 |
| *** suuuper has joined #tripleo | 15:58 | |
| dtantsur | and anyway, it breaks creating a port in a simple way | 15:58 |
| EmilienM | dprince: it's in any podman run command | 15:58 |
| EmilienM | and just in that script | 15:58 |
| EmilienM | when the entrypoint is bind mounted | 15:58 |
| dtantsur | which needs to substantially complicates the patch to deploy without nova | 15:58 |
| dprince | I personally would set the run count to 1 as a work around. and deal with the slowness until the proper fix. | 15:58 |
| dprince | EmilienM: ^ that is my vote | 15:58 |
| EmilienM | dprince: look at https://github.com/containers/libpod/issues/1844 | 15:58 |
| EmilienM | we tried everything... really | 15:58 |
| EmilienM | slowness will increases the change of timeouts in CI | 15:59 |
| dtantsur | hjensas: how do we detect which subnet to use if there are several? | 15:59 |
| Tengu | *chance | 15:59 |
| dprince | EmilienM: setting the run count is configurable, it is clean, and highlights that podman is the problem | 15:59 |
| *** moguimar has joined #tripleo | 15:59 | |
| mwhahaha | we know podman is the problem | 15:59 |
| Tengu | dprince: we already know podman is a serious problem in fact..... | 15:59 |
| *** moguimar is now known as moguimar_afk | 15:59 | |
| dtantsur | hjensas: as to "binding info", ironic adds it, it does not seem to be enough.. | 15:59 |
| EmilienM | lol I think we all know podman has been a problem | 15:59 |
| Tengu | and making the CI slower is not a good idea, knowing podman is *already* slower than docker. | 15:59 |
| EmilienM | we have reported around a hundred of github issue since August | 16:00 |
| openstackgerrit | Ben Nemec proposed openstack/openstack-virtual-baremetal master: Add Zuul config https://review.openstack.org/630993 | 16:00 |
| openstackgerrit | Ben Nemec proposed openstack/openstack-virtual-baremetal master: Add .gitreview https://review.openstack.org/630984 | 16:00 |
| hjensas | dtantsur: you should be able to create the port, then when ironic updates it with binding info it should update the port and allocate an ip. | 16:00 |
| dtantsur | hjensas: yeah, here is the problem: it does not | 16:00 |
| dtantsur | hjensas: here is a bound port without IP: http://paste.openstack.org/show/742657/ | 16:01 |
| dprince | EmilienM: this is better, https://review.openstack.org/#/c/629546/5. But do it with a 1 line change to environments/podman.yaml. You shouldn't be modifying the defaults for a podman bug IMO | 16:01 |
| openstackgerrit | Sorin Sbarnea proposed openstack/tripleo-quickstart-extras master: Corrected openstack-virtual-baremetal repo location https://review.openstack.org/630953 | 16:02 |
| hjensas | dtantsur: ok, is there an agent? openstack network agent list | grep 3afd4c71-f946-45f7-9667-4a05fedd874e ? | 16:02 |
| EmilienM | dprince: (again) https://review.openstack.org/#/c/629546 will increase the risk of timeouts in our CI, most of our CI jobs run with podman now | 16:02 |
| EmilienM | dprince: we have maybe 1 or 2 jobs running with docker | 16:03 |
| dtantsur | hjensas: yeah http://paste.openstack.org/show/742658/ | 16:03 |
| *** radeks has quit IRC | 16:04 | |
| dtantsur | (the 3rd from the bottom) | 16:04 |
| Tengu | dprince: also, the bug has a "low rate hit" overall. something like 10 or 15% - slowering the whole CI for that is probably a bit overkill. | 16:04 |
| dprince | EmilienM: I get it, then we'll have to stop the long running jobs too. I give preference to clean solutions here I think. My concern with the retry patch is it will cause some timeouts too | 16:04 |
| mwhahaha | dprince: how is that going to cause timeouts? | 16:04 |
| hjensas | dtantsur: then do openstack network agent show -c configuration -f yaml <agent-id> ( brigde_mappings ?) | 16:04 |
| Tengu | dprince: probably less timeouts than lowering the process count.. | 16:04 |
| EmilienM | right | 16:04 |
| EmilienM | because process count is for all runs | 16:04 |
| EmilienM | while retry is only where there is a failure | 16:04 |
| *** salmankhan has joined #tripleo | 16:04 | |
| EmilienM | and again, it's a patch we want to revert ASAP | 16:05 |
| dprince | mwhahaha: well, in the case where the puppet manifest legitematly fails multiple times. Many cases that could happen I think | 16:05 |
| * Tengu feels some pressures for his own patch | 16:05 | |
| EmilienM | we are working with the podman team on it, and we want podman 1.0 in tripleo ci to see improvements | 16:05 |
| dtantsur | hjensas: http://paste.openstack.org/show/742659/ | 16:05 |
| mwhahaha | dprince: but that'll stop at the step and won't continue | 16:05 |
| *** morazi has joined #tripleo | 16:05 | |
| dprince | sure, but after retrying 5 times right? | 16:05 |
| mwhahaha | meaning instead of 30 mins it takes 35 | 16:06 |
| dprince | perhaps. all depends on the length of the task and the timeout for each iteration. My point is that either solution causes problems too. The retry is also going to make the logs a real mess if there is thrashing | 16:07 |
| hjensas | dtantsur: it all looks good. I think we need to debug neutron. https://github.com/openstack/neutron/blob/master/neutron/db/ipam_backend_mixin.py#L637-L639 | 16:07 |
| *** errr has quit IRC | 16:07 | |
| Tengu | more over, there's a fair limit of 5 retries, with a 3s delay (we might want to lower that, EmilienM ?) - if we hit a puppet failure, the probability it will hit early at catalog compilation is pretty higher than anything, dprince | 16:07 |
| EmilienM | we can reduce the retry to 3? | 16:07 |
| hjensas | dtantsur: according to that docstring it should allocate/update the IPs once enough binding data is available. | 16:07 |
| Tengu | EmilienM: maybe even 1... ? | 16:07 |
| dtantsur | hjensas: yeah, I wonder why | 16:08 |
| dtantsur | I guess we don't see it regularly because nova provides a subnet? | 16:08 |
| dprince | Tengu: the machine is likely stressed when it hits these. Perhaps swapping or CPU bound. The retry might not help as much as you think it will | 16:08 |
| Tengu | dprince: well, seeing how the queue is managed, it won't really make a difference regarding the number of processes running afaik. It will keep the thread locked until either final failure, or success. | 16:09 |
| weshay | EmilienM, ooff.. that is making the standalone jobs run for 2hrs almost :( | 16:10 |
| Tengu | dprince: more over, the issue is probably NOT related to I/O or system load or whatever - we did try a lot of things in order to reproduce it, without any hit :(. | 16:10 |
| *** ooolpbot has joined #tripleo | 16:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1811004 | 16:10 |
| openstack | Launchpad bug 1811004 in tripleo "tripleo-ci-centos-7-scenario003-standalone fails tempest.scenario.test_minimum_basic.TestMinimumBasicScenario" [Critical,In progress] - Assigned to Michele Baldessari (michele) | 16:10 |
| *** ooolpbot has quit IRC | 16:10 | |
| hjensas | dtantsur: yes, in THT we set which subnet to use in fixed_ips for each Role. | 16:10 |
| Tengu | dprince: slowering disks, decraising RAM in favor of SWAP, decreasing CPU.... we just never could reproduce it outside of the CI.... | 16:10 |
| *** errr has joined #tripleo | 16:10 | |
| dprince | Tengu: have you tried connecting to the CI boxes as they are runnning? | 16:10 |
| EmilienM | weshay: which one? | 16:11 |
| EmilienM | weshay: the ProcessCount to 1? | 16:11 |
| Tengu | dprince: no. because the failure appears sooo randomly we can't connect to each box. we usually see it once it's over. | 16:11 |
| weshay | https://review.openstack.org/#/c/629546/ | 16:11 |
| weshay | ya | 16:11 |
| EmilienM | yes I told you | 16:11 |
| dtantsur | hjensas: I guess a straightforward fix is to support subnets in metalsmith and just pass them.. but I still wonder why this happens. | 16:11 |
| hjensas | dtantsur: I'd love to see us not need to add the subnet. | 16:12 |
| EmilienM | dprince: the failure is so random and low in stats that it's impossible for us to ssh and debug | 16:12 |
| EmilienM | dprince: because even if you do, the podman command will work then | 16:12 |
| dtantsur | hjensas: right. let's move to #openstack-neutron? | 16:12 |
| hjensas | dtantsur: neutron should figure this out and set the IP once we bind. | 16:12 |
| hjensas | dtantsur: ok. | 16:12 |
| mwhahaha | dprince: can we just retry 3 instead of 5 as a compromise? | 16:15 |
| openstackgerrit | mathieu bultel proposed openstack/tripleo-common master: Catch heat exception in package_update action https://review.openstack.org/630986 | 16:15 |
| *** dougbtv has joined #tripleo | 16:15 | |
| *** ccamacho has quit IRC | 16:16 | |
| dprince | EmilienM: commented on how to make this a bit better since most like it https://review.openstack.org/#/c/614639/ | 16:18 |
| EmilienM | k | 16:18 |
| dprince | So long as we agree we can't cut a release with this code. I.e its temporary I'm fine with it. | 16:18 |
| EmilienM | I think that's what we said | 16:19 |
| dprince | EmilienM: well, the patch log message doesn't say that. Added an explicit (this is horrible) note to the changelog might help someone realize the intent of this patch even better | 16:20 |
| dprince | EmilienM: and like I said in my first comment. Making this configurable in some way would improve the situation if you wanted to temporarily disable it for some jobs... | 16:21 |
| dprince | EmilienM: similar to DockerPuppetProcess count. Having an option if we are going this route would be better | 16:22 |
| dprince | all depends on how long live it will be in CI I guess. | 16:22 |
| *** ykarel is now known as ykarel|away | 16:23 | |
| openstackgerrit | Emilien Macchi proposed openstack/python-tripleoclient master: utils: pass python interpreter to run_ansible_playbook https://review.openstack.org/630684 | 16:24 |
| *** fragatina has quit IRC | 16:25 | |
| *** fragatina has joined #tripleo | 16:26 | |
| EmilienM | dprince: on it | 16:27 |
| *** cylopez has quit IRC | 16:29 | |
| Tengu | dprince: making that might make this patch permanent if it's too comfortable to tweak it ;). | 16:29 |
| openstackgerrit | Nagasai Vinaykumar Kapalavai proposed openstack/tripleo-heat-templates master: Update to the ceilometer publisher list https://review.openstack.org/600843 | 16:32 |
| *** ansmith_ has joined #tripleo | 16:32 | |
| Tengu | jaosorior: re: the haproxy selnux issue for its log: I think the thing to do is to push a policy in openstack-selinux | 16:34 |
| *** ansmith has quit IRC | 16:35 | |
| openstackgerrit | Lee Yarwood proposed openstack/tripleo-heat-templates master: WIP placement: Extract the service from Nova https://review.openstack.org/630644 | 16:36 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: docker-puppet: retry container run command https://review.openstack.org/614639 | 16:39 |
| EmilienM | dprince, mwhahaha, Tengu ^ please review | 16:39 |
| EmilienM | https://review.openstack.org/#/c/614639/10..11 | 16:39 |
| *** ade_lee has joined #tripleo | 16:41 | |
| openstackgerrit | Michele Baldessari proposed openstack/tripleo-heat-templates master: Transitioning to HAProxy 1.8 https://review.openstack.org/607856 | 16:41 |
| *** mcornea has quit IRC | 16:42 | |
| *** mcornea has joined #tripleo | 16:42 | |
| *** hamzy_ has quit IRC | 16:43 | |
| *** fragatina has quit IRC | 16:47 | |
| *** fragatina has joined #tripleo | 16:48 | |
| *** fragatina has quit IRC | 16:48 | |
| *** wolverineav has joined #tripleo | 16:49 | |
| *** fragatina has joined #tripleo | 16:49 | |
| *** salmankhan has quit IRC | 16:50 | |
| openstackgerrit | Martin Schuppert proposed openstack/tripleo-heat-templates master: Add functionality to disable KSM on compute nodes per role https://review.openstack.org/631021 | 16:50 |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Add in-place execution mode for reproducer https://review.openstack.org/627633 | 16:50 |
| *** salmankhan has joined #tripleo | 16:50 | |
| Tengu | jaosorior: could that have your blessing? https://github.com/redhat-openstack/openstack-selinux/pull/20 | 16:52 |
| *** fragatina has quit IRC | 16:53 | |
| *** wolverineav has quit IRC | 16:53 | |
| *** fragatina has joined #tripleo | 16:53 | |
| weshay | ssbarnea|rover, the last number of gate failures were not found by elastic-recheck | 16:53 |
| weshay | ssbarnea|rover, that may be something to note in http://dashboard-ci.tripleo.org/d/cEEjGFFmz/cockpit?orgId=1 | 16:54 |
| weshay | under failing gate jobs | 16:54 |
| weshay | whether or not the failure is associated w/ elastic recheck | 16:54 |
| *** ansmith has joined #tripleo | 16:54 | |
| weshay | ssbarnea|rover, can explain that more if you like | 16:55 |
| openstackgerrit | waleed mousa proposed openstack/puppet-tripleo stable/queens: Configure switchedv while there is no vfs bounded https://review.openstack.org/630605 | 16:55 |
| ssbarnea|rover | weshay: going for (late) lunch, i will ping you when I am bakc. | 16:55 |
| weshay | ssbarnea|rover, no prob.. can be tomorrow too | 16:55 |
| *** ansmith_ has quit IRC | 16:56 | |
| *** fragatina has quit IRC | 16:58 | |
| *** fragatina has joined #tripleo | 16:59 | |
| *** aufi has quit IRC | 16:59 | |
| *** udesale has quit IRC | 17:04 | |
| *** fragatina has quit IRC | 17:06 | |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates master: WIP: testing for https://bugs.launchpad.net/tripleo/+bug/1811004 https://review.openstack.org/631023 | 17:07 |
| openstack | Launchpad bug 1811004 in tripleo "tripleo-ci-centos-7-scenario003-standalone fails tempest.scenario.test_minimum_basic.TestMinimumBasicScenario" [Critical,In progress] - Assigned to Michele Baldessari (michele) | 17:07 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-docs master: WIP: Add high level overview of ci jobs/check vs gate and promote https://review.openstack.org/631024 | 17:09 |
| *** jpich has quit IRC | 17:09 | |
| *** ooolpbot has joined #tripleo | 17:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1811004 | 17:10 |
| *** ooolpbot has quit IRC | 17:10 | |
| openstack | Launchpad bug 1811004 in tripleo "tripleo-ci-centos-7-scenario003-standalone fails tempest.scenario.test_minimum_basic.TestMinimumBasicScenario" [Critical,In progress] - Assigned to Michele Baldessari (michele) | 17:10 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates master: [WIP] Hook in a place in hiera for upgrade to add varibles. https://review.openstack.org/631025 | 17:10 |
| openstackgerrit | Cédric Jeanneret proposed openstack/tripleo-heat-templates master: Ensure we get the file directly in the container when running podman https://review.openstack.org/630287 | 17:11 |
| openstackgerrit | Cédric Jeanneret proposed openstack/tripleo-heat-templates master: DNM - podman testing https://review.openstack.org/628379 | 17:12 |
| openstackgerrit | Cédric Jeanneret proposed openstack/tripleo-heat-templates master: DNM - podman testing https://review.openstack.org/628380 | 17:12 |
| openstackgerrit | Cédric Jeanneret proposed openstack/tripleo-heat-templates master: DNM - podman testing https://review.openstack.org/628381 | 17:12 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Add gate/check layout for scenario-standalone in a template https://review.openstack.org/630239 | 17:13 |
| *** ekultails has joined #tripleo | 17:13 | |
| weshay | Tengu, you have a minute? | 17:14 |
| weshay | want to talk about how we're testing podman | 17:14 |
| openstackgerrit | Martin Schuppert proposed openstack/tripleo-heat-templates master: Add functionality to disable KSM on compute nodes per role https://review.openstack.org/631021 | 17:15 |
| *** morazi has quit IRC | 17:15 | |
| EmilienM | weshay: we are leaving the room, 6.15pm here | 17:17 |
| EmilienM | weshay: send email | 17:17 |
| * EmilienM away | 17:18 | |
| *** abishop|afk is now known as abishop | 17:19 | |
| *** derekh has quit IRC | 17:22 | |
| *** morazi has joined #tripleo | 17:23 | |
| *** avivgt has quit IRC | 17:26 | |
| *** dtantsur is now known as dtantsur|afk | 17:28 | |
| openstackgerrit | Merged openstack/openstack-virtual-baremetal master: Add Zuul config https://review.openstack.org/630993 | 17:29 |
| openstackgerrit | Merged openstack/openstack-virtual-baremetal master: Add .gitreview https://review.openstack.org/630984 | 17:30 |
| *** sudeepbatra has quit IRC | 17:32 | |
| *** dtantsur|afk has quit IRC | 17:36 | |
| *** ykarel has joined #tripleo | 17:37 | |
| *** ykarel|away has quit IRC | 17:38 | |
| openstackgerrit | Merged openstack/python-tripleoclient master: [Tests] Fix warning duplicate key. https://review.openstack.org/629118 | 17:41 |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-common master: nova-less-deploy: deploy_instances and undeploy_instances workflows https://review.openstack.org/576856 | 17:41 |
| openstackgerrit | Simon Dodsley proposed openstack/tripleo-heat-templates master: Add missing entries for Pure Storage Cinder Backend and fix typos https://review.openstack.org/619608 | 17:43 |
| openstackgerrit | Simon Dodsley proposed openstack/tripleo-heat-templates master: Add missing entries for Pure Storage Cinder Backend and fix typos https://review.openstack.org/619608 | 17:46 |
| openstackgerrit | Simon Dodsley proposed openstack/tripleo-heat-templates master: Add missing entries for Pure Storage Cinder Backend and fix typos https://review.openstack.org/619608 | 17:50 |
| *** sanjayu_ has quit IRC | 17:50 | |
| *** trown is now known as trown|lunch | 17:52 | |
| *** fragatina has joined #tripleo | 17:54 | |
| *** wolverineav has joined #tripleo | 17:54 | |
| *** wolverineav has quit IRC | 17:57 | |
| *** wolverineav has joined #tripleo | 17:57 | |
| *** jpena is now known as jpena|off | 17:59 | |
| *** gfidente is now known as gfidente|afk | 18:00 | |
| *** pcaruana has quit IRC | 18:01 | |
| *** psachin has quit IRC | 18:03 | |
| *** salmankhan1 has joined #tripleo | 18:04 | |
| *** salmankhan has quit IRC | 18:07 | |
| *** holser_ is now known as holser|eod | 18:08 | |
| *** salmankhan1 has quit IRC | 18:08 | |
| *** ooolpbot has joined #tripleo | 18:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1806966 | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1811004 | 18:10 |
| *** ooolpbot has quit IRC | 18:10 | |
| openstack | Launchpad bug 1806966 in tripleo "mistral.rpc.base MessagingTimeout: Timed out waiting for a reply to message ID" [Critical,Triaged] | 18:10 |
| openstack | Launchpad bug 1811004 in tripleo "tripleo-ci-centos-7-scenario003-standalone fails tempest.scenario.test_minimum_basic.TestMinimumBasicScenario" [Critical,In progress] - Assigned to Michele Baldessari (michele) | 18:10 |
| *** holser|eod has quit IRC | 18:12 | |
| *** brault has quit IRC | 18:14 | |
| *** brault has joined #tripleo | 18:15 | |
| *** itlinux has joined #tripleo | 18:16 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Be explicit when passing vars into deploy steps https://review.openstack.org/613300 | 18:16 |
| *** panda is now known as panda|off | 18:19 | |
| *** brault has quit IRC | 18:20 | |
| EmilienM | mwhahaha: completely untested, but a starting point: https://review.openstack.org/631043 | 18:23 |
| EmilienM | feedback is welcome | 18:23 |
| *** ykarel has quit IRC | 18:25 | |
| *** bogdando has quit IRC | 18:26 | |
| *** ansmith_ has joined #tripleo | 18:26 | |
| *** ansmith has quit IRC | 18:29 | |
| *** hberaud is now known as hberaud|gone | 18:32 | |
| *** wolverineav has quit IRC | 18:34 | |
| *** wolverineav has joined #tripleo | 18:35 | |
| *** morazi has quit IRC | 18:35 | |
| *** wolverineav has quit IRC | 18:40 | |
| *** chem has quit IRC | 18:43 | |
| *** wolverineav has joined #tripleo | 18:43 | |
| *** dciabrin_ has quit IRC | 18:44 | |
| *** hamzy_ has joined #tripleo | 18:44 | |
| *** wolverineav has quit IRC | 18:48 | |
| *** ansmith_ has quit IRC | 18:52 | |
| *** kopecmartin is now known as kopecmartin|off | 18:52 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/rocky: Bind mount /var/lib/iscsi in containers using iSCSI https://review.openstack.org/630321 | 18:55 |
| dprince | mwhahaha: lets try it I guess https://review.openstack.org/#/c/614639/11 | 18:57 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras master: WIP, add dlrn build log to log stash https://review.openstack.org/620958 | 18:57 |
| *** hamzy_ has quit IRC | 18:59 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-heat-templates stable/queens: DNM, wes GATE CHECK for TripleO https://review.openstack.org/631051 | 18:59 |
| *** hamzy_ has joined #tripleo | 19:00 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-heat-templates stable/queens: DNM, wes GATE CHECK for TripleO https://review.openstack.org/631051 | 19:00 |
| *** ansmith_ has joined #tripleo | 19:01 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras master: add dlrn build log to log stash https://review.openstack.org/620958 | 19:01 |
| *** wolverineav has joined #tripleo | 19:02 | |
| *** ooolpbot has joined #tripleo | 19:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1806966 | 19:10 |
| openstack | Launchpad bug 1806966 in tripleo "mistral.rpc.base MessagingTimeout: Timed out waiting for a reply to message ID" [Critical,Triaged] | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1811004 | 19:10 |
| *** ooolpbot has quit IRC | 19:10 | |
| openstack | Launchpad bug 1811004 in tripleo "tripleo-ci-centos-7-scenario003-standalone fails tempest.scenario.test_minimum_basic.TestMinimumBasicScenario" [Critical,In progress] - Assigned to Michele Baldessari (michele) | 19:10 |
| *** Chaserjim has quit IRC | 19:11 | |
| *** hamzy_ has quit IRC | 19:13 | |
| *** rlandy is now known as rlandy|brb | 19:14 | |
| openstackgerrit | Merged openstack/puppet-tripleo master: Allow mysql options to be set for the HA bundle https://review.openstack.org/625192 | 19:18 |
| *** trown|lunch is now known as trown | 19:19 | |
| *** hamzy_ has joined #tripleo | 19:31 | |
| *** jfrancoa has quit IRC | 19:31 | |
| *** rlandy|brb is now known as rlandy | 19:33 | |
| *** Chaserjim has joined #tripleo | 19:37 | |
| *** dougbtv has quit IRC | 19:44 | |
| *** ekultails has quit IRC | 19:47 | |
| *** wolverineav has quit IRC | 19:48 | |
| *** wolverineav has joined #tripleo | 19:48 | |
| *** wolverineav has quit IRC | 19:51 | |
| *** wolverineav has joined #tripleo | 19:51 | |
| *** wolverineav has quit IRC | 19:52 | |
| *** wolverineav has joined #tripleo | 19:52 | |
| *** wolverineav has quit IRC | 19:58 | |
| openstackgerrit | Merged openstack/ansible-role-container-registry master: Fix linting issues https://review.openstack.org/630460 | 20:00 |
| openstackgerrit | Merged openstack/puppet-tripleo stable/queens: Add additional tuning options for mariadb/galera https://review.openstack.org/625197 | 20:00 |
| *** ekultails has joined #tripleo | 20:06 | |
| *** brault has joined #tripleo | 20:06 | |
| *** ooolpbot has joined #tripleo | 20:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1806966 | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1811004 | 20:10 |
| *** ooolpbot has quit IRC | 20:10 | |
| openstack | Launchpad bug 1806966 in tripleo "mistral.rpc.base MessagingTimeout: Timed out waiting for a reply to message ID" [Critical,Triaged] | 20:10 |
| openstack | Launchpad bug 1811004 in tripleo "tripleo-ci-centos-7-scenario003-standalone fails tempest.scenario.test_minimum_basic.TestMinimumBasicScenario" [Critical,In progress] - Assigned to Michele Baldessari (michele) | 20:10 |
| *** brault has quit IRC | 20:11 | |
| *** hamzy_ has quit IRC | 20:12 | |
| *** rfolco has quit IRC | 20:28 | |
| openstackgerrit | mathieu bultel proposed openstack/python-tripleoclient master: Use a single queue for update/upgrade https://review.openstack.org/629103 | 20:33 |
| *** morazi has joined #tripleo | 20:35 | |
| *** itlinux_ has joined #tripleo | 20:36 | |
| *** itlinux has quit IRC | 20:37 | |
| *** wolverineav has joined #tripleo | 20:39 | |
| *** itlinux_ has quit IRC | 20:39 | |
| *** wolverineav has quit IRC | 20:43 | |
| *** sudeepbatra has joined #tripleo | 20:51 | |
| sudeepbatra | overcloud deployment failed again using quickstart | 20:52 |
| sudeepbatra | TASK [did the deployment pass or fail?] *************************************************************************************** task path: /home/toor/.quickstart/playbooks/quickstart-extras-overcloud.yml:44 Tuesday 15 January 2019 15:37:03 -0500 (0:00:00.156) 2:57:21.724 ******* fatal: [localhost]: FAILED! => { "failed_when_result": true, "overcloud_deploy_result": "failed" | 20:52 |
| sudeepbatra | rlandy, any suggestions ^ | 20:52 |
| weshay | sudeepbatra, ssh to the undercloud and look at the deploy logs | 20:54 |
| weshay | ansible is going to tell you anything important | 20:54 |
| sudeepbatra | yeah , did that , had been observing the overcloud_deploy.log | 20:54 |
| weshay | arxcruz|ruck, tomorrow.. please read my comments in https://bugs.launchpad.net/tripleo/+bug/1811682 | 20:54 |
| openstack | Launchpad bug 1811682 in tripleo "centos-binary-swift-object failed with cannot load glue library: libibverbs.so.1: cannot open shared object file: No such file or directory"" [Critical,Incomplete] | 20:54 |
| sudeepbatra | DeploymentError: Heat Stack create failed | 20:54 |
| *** wolverineav has joined #tripleo | 20:55 | |
| weshay | you need to get your log files up where folks can see the whole thing for us to help | 20:55 |
| weshay | snippets of errors are not actionable, sorry but true | 20:55 |
| sudeepbatra | [overcloud.Controller.0.Controller]: CREATE_FAILED ResourceInError: resources.Controller: Went to status ERROR due to "Message: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance bd8d5dda-de0a-4c29-b21c-1cb150e36275., Code: 500" | 20:55 |
| rlandy | what weshay said ... | 20:56 |
| sudeepbatra | I tried as you suggested > ./tripleo-quickstart/quickstart.sh --clean --release master --teardown all --tags all -e undercloud_undercloud_ntp_servers=147.117.251.28 -e undercloud_clean_nodes=false -e @/home/toor/deploy-config.yaml 127.0.0.2 | 20:56 |
| sudeepbatra | rlandy ^ | 20:56 |
| rlandy | ok - so it got you as far as deploy | 20:56 |
| rlandy | we need to see why your overcloud deployment failed | 20:57 |
| weshay | sudeepbatra, upload the deploy log | 20:57 |
| weshay | rlandy, we should write a tool to some free service for that | 20:57 |
| rlandy | "Exhausted all hosts available for retrying build failures for instance" | 20:57 |
| weshay | lolz | 20:57 |
| sudeepbatra | ok, where can I upload pls | 20:57 |
| rlandy | looks like you don;t have enough capacity to deploy the overcloud | 20:57 |
| rlandy | are your overcloud nodes big enough | 20:58 |
| weshay | sudeepbatra, I use chunk.io | 20:58 |
| *** wolverineav has quit IRC | 21:00 | |
| *** wolverineav has joined #tripleo | 21:00 | |
| sudeepbatra | 16Core, 64GB , 250SSD and 2 TB HDD | 21:01 |
| sudeepbatra | cant do quickstart on this ? :) | 21:01 |
| weshay | ya | 21:01 |
| weshay | do you have other vm's running? | 21:01 |
| weshay | sudeepbatra, compare it https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-promote-master-current-tripleo-delorean-minimal-446/undercloud/home/stack/overcloud_deploy.log.gz | 21:03 |
| sudeepbatra | no, TripleO is the only reason I am using it.. | 21:04 |
| rlandy | overcloud node flavors - how big are the VMs | 21:04 |
| sudeepbatra | default, I am just using the quickstart.sh code | 21:04 |
| sudeepbatra | ./tripleo-quickstart/quickstart.sh --clean --release master --teardown all --tags all -e undercloud_undercloud_ntp_servers=147.117.251.28 -e undercloud_clean_nodes=false -e @/home/toor/deploy-config.yaml 127.0.0.2 | 21:05 |
| *** dprince has quit IRC | 21:05 | |
| sudeepbatra | After 2 weeks with TripleO, I must say I had better days with Openstack-helm :( | 21:08 |
| mwhahaha | all your failures have been environmental tho | 21:09 |
| * mwhahaha shrugs | 21:09 | |
| sudeepbatra | :) | 21:09 |
| sudeepbatra | I just want to get it up | 21:10 |
| *** ooolpbot has joined #tripleo | 21:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1806966 | 21:10 |
| openstack | Launchpad bug 1806966 in tripleo "mistral.rpc.base MessagingTimeout: Timed out waiting for a reply to message ID" [Critical,Triaged] | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1811004 | 21:10 |
| *** ooolpbot has quit IRC | 21:10 | |
| openstack | Launchpad bug 1811004 in tripleo "tripleo-ci-centos-7-scenario003-standalone fails tempest.scenario.test_minimum_basic.TestMinimumBasicScenario" [Critical,In progress] - Assigned to Michele Baldessari (michele) | 21:10 |
| *** florianf_ has quit IRC | 21:11 | |
| mwhahaha | sudeepbatra: try https://github.com/cjeanner/tripleo-lab | 21:12 |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart-extras master: WIP: Design new role for zuul-based reproducer https://review.openstack.org/631067 | 21:15 |
| *** hamzy_ has joined #tripleo | 21:21 | |
| *** salmankhan has joined #tripleo | 21:23 | |
| *** xek has quit IRC | 21:24 | |
| sudeepbatra | rlandy,weshay, uploaded the log (overcloud_deploy.log at the end of https://bugs.launchpad.net/tripleo/+bug/1810966 (not getting response from chunk.io) | 21:29 |
| openstack | Launchpad bug 1810966 in tripleo "tripleo-quickstart does not work" [Undecided,Incomplete] | 21:29 |
| *** hamzy_ has quit IRC | 21:29 | |
| openstackgerrit | Merged openstack/tripleo-common master: Add nova serialproxy image https://review.openstack.org/628367 | 21:36 |
| openstackgerrit | Merged openstack/python-tripleoclient master: Remove unused class vars from overcloud_deploy.py https://review.openstack.org/629093 | 21:38 |
| *** trown is now known as trown|outtypewww | 21:41 | |
| openstackgerrit | Alan Bishop proposed openstack/tripleo-heat-templates stable/queens: Bind mount /var/lib/iscsi in containers using iSCSI https://review.openstack.org/631072 | 21:42 |
| *** ekultails has quit IRC | 21:42 | |
| rlandy | looking | 21:44 |
| openstackgerrit | mathieu bultel proposed openstack/tripleo-common master: Catch heat exception in package_update action https://review.openstack.org/630986 | 21:46 |
| *** ekultails has joined #tripleo | 21:48 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Conditionalize docker socket bind-mount https://review.openstack.org/630955 | 21:49 |
| *** mburned is now known as mburned_out | 21:51 | |
| *** mburned_out is now known as mburned | 21:51 | |
| rlandy | the problem here is the ntp servers on the overcloud | 21:51 |
| *** Vorrtex has quit IRC | 21:52 | |
| rlandy | weshay: ^^ same issue with overcloud deployment | 21:52 |
| *** ansmith_ has quit IRC | 21:54 | |
| *** wolverineav has quit IRC | 21:58 | |
| Tengu | weshay: sorry, was off, still need me? | 21:58 |
| *** raildo has quit IRC | 21:58 | |
| *** wolverineav has joined #tripleo | 21:58 | |
| *** lblanchard has quit IRC | 22:00 | |
| sudeepbatra | but I specified , --ntp-server 147.117.251.28 inside the config.yaml | 22:00 |
| sudeepbatra | rlandy ^ | 22:00 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates master: step2: flatten the neutron service configurations https://review.openstack.org/630513 | 22:01 |
| rlandy | sudeepbatra: I know - I have a similar problem on my internal systems with ntpupdate failing | 22:01 |
| rlandy | I am investigating | 22:01 |
| rlandy | I was discussing it with mwhahaha a few days ago | 22:02 |
| sudeepbatra | ok , thanks. appreciate help. | 22:02 |
| rlandy | 2019-01-15 14:11:04 | ok: [overcloud-compute-0] => { | 22:02 |
| rlandy | 2019-01-15 14:11:04 | "ansible_facts": { | 22:02 |
| rlandy | 2019-01-15 14:11:04 | "ntp_install_packages": false, | 22:02 |
| rlandy | 2019-01-15 14:11:09 | "ntp_serveOvercloud configuration failed. | 22:02 |
| rlandy | ^^ I see that | 22:02 |
| rlandy | 2019-01-15 14:11:09 | fatal: [overcloud-compute-0]: FAILED! => { | 22:02 |
| rlandy | 2019-01-15 14:11:09 | 15 Jan 14:11:09 ntpdate[21133]: no server suitable for synchronization found | 22:02 |
| sudeepbatra | where did u see that > overcloud_deploy.log ? | 22:04 |
| rlandy | [heat-admin@overcloud-controller-0 ~]$ sudo ntpdate -u 10.5.26.10 | 22:05 |
| rlandy | 15 Jan 17:05:07 ntpdate[19011]: adjust time server 10.5.26.10 offset 0.053704 sec | 22:05 |
| rlandy | works on the controllers but not on the compute nodes | 22:05 |
| rlandy | mwhahaha: ^^ | 22:05 |
| rlandy | yes in my overcloud_deploy.log | 22:06 |
| rlandy | I specified --ntp-server in the deploy command | 22:06 |
| rlandy | so the controllers can connect but the compute node cannot | 22:06 |
| *** owalsh_ has joined #tripleo | 22:07 | |
| rlandy | works with i[ address and clock.* on controllers but no access on compute | 22:07 |
| mwhahaha | right and we said turn on masqurade on the undercloud | 22:09 |
| mwhahaha | computes don't have external access by default | 22:10 |
| *** owalsh has quit IRC | 22:10 | |
| *** ooolpbot has joined #tripleo | 22:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1806966 | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1811004 | 22:10 |
| *** ooolpbot has quit IRC | 22:10 | |
| openstack | Launchpad bug 1806966 in tripleo "mistral.rpc.base MessagingTimeout: Timed out waiting for a reply to message ID" [Critical,Triaged] | 22:10 |
| openstack | Launchpad bug 1811004 in tripleo "tripleo-ci-centos-7-scenario003-standalone fails tempest.scenario.test_minimum_basic.TestMinimumBasicScenario" [Critical,In progress] - Assigned to Michele Baldessari (michele) | 22:10 |
| rlandy | but masquerade is on | 22:10 |
| mwhahaha | but the default route isn't the undercloud anymore right? | 22:10 |
| mwhahaha | aren't you providing a different route | 22:11 |
| rlandy | should this not work be default? | 22:13 |
| *** owalsh_ is now known as owalsh | 22:13 | |
| mwhahaha | i don't think your network config is the default | 22:14 |
| mwhahaha | i believe it does work by default | 22:14 |
| rlandy | it's a baremetal config so it may need an update | 22:14 |
| mwhahaha | right | 22:14 |
| sudeepbatra | I didnt see ntp error in my overcloud_deploy.log though | 22:15 |
| rlandy | you pointed to this bug so I thought you had the an ntp error | 22:16 |
| rlandy | https://bugs.launchpad.net/tripleo/+bug/1810966 | 22:16 |
| openstack | Launchpad bug 1810966 in tripleo "tripleo-quickstart does not work" [Undecided,Incomplete] | 22:16 |
| rlandy | however that discusses overcloud | 22:16 |
| rlandy | undercloud | 22:16 |
| sudeepbatra | oh sorry, I meant the last statement only.. | 22:17 |
| sudeepbatra | export OPT_TEARDOWN="all" ./tripleo-quickstart/quickstart.sh --clean --release master --teardown all --tags all -e undercloud_undercloud_ntp_servers=147.117.251.28 -e undercloud_clean_nodes=false -e @/home/toor/deploy-config.yaml 127.0.0.2 ERROR : Went to status ERROR due to "Message: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 604c1477-33a5-4a92-9d48-47afc9b75394., | 22:17 |
| sudeepbatra | for which I uploaded the log | 22:18 |
| mwhahaha | that's a provisioning problem, check ironic/nova logs as to why it couldn't provision the nodes | 22:18 |
| *** factor has joined #tripleo | 22:22 | |
| *** jcoufal has quit IRC | 22:22 | |
| *** gfidente|afk has quit IRC | 22:24 | |
| Tengu | weshay: just read your email - can't say more than mwhahaha and EmilienM. | 22:25 |
| Tengu | mwhahaha: btw, my patch injecting the docker-puppet.sh script inside the container *before* it actually runs doesn't fix the exec error....... | 22:25 |
| mwhahaha | lovely | 22:25 |
| Tengu | I just hit it in plain on my very patch. and it's not a mistake in the change, as other containers were running fine -.- | 22:26 |
| mwhahaha | i think we need to work on trying to reproduce the bug outside of docker-puppet | 22:26 |
| mwhahaha | i wonder if it's related to the number of mounts we use | 22:26 |
| Tengu | mwhahaha: EmilienM and I tried very hard for that. | 22:26 |
| mwhahaha | did you keep track of what you tried? | 22:26 |
| Tengu | mwhahaha: I don't think so. more over, my patch actually drop a mount | 22:26 |
| mwhahaha | right but we still do a lot of mounts | 22:27 |
| Tengu | mwhahaha: so we tried to load our own builder, by lowering either the CPU and/or the RAM (in favor of swap), and blasting the disks in order to get slow I/O. That was mostly EmilienM's game last week. | 22:27 |
| mwhahaha | k | 22:27 |
| Tengu | mwhahaha: wel yeah. but it's really weird it's always THAT part that explodes. some other "prod" containers also have a lot of mounts. | 22:28 |
| Tengu | and even if we don't make a mount for the entrypoint, THAT specific thing explodes..... if it were the mounts, I'd think "puppet can't find that module" would also be a possible issue | 22:28 |
| Tengu | mwhahaha: example: http://paste.openstack.org/show/742692/ | 22:29 |
| Tengu | we clearly see the steps: create the container, copy the file, start the container. | 22:29 |
| Tengu | the copy has the "mount, cp, umount" steps embedded | 22:30 |
| * mwhahaha shrugs | 22:30 | |
| Tengu | I'll poke on #podman. they should be here now :) | 22:30 |
| sudeepbatra | was trying this link to troubleshoot, my nova list shows as STATUS=ERROR for both overcloud-controller and compute | 22:32 |
| sudeepbatra | so where do I check the ironic/nova logs | 22:33 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Mount system modules when calling system iptables https://review.openstack.org/630973 | 22:33 |
| openstackgerrit | Merged openstack/tripleo-quickstart master: reduce standalone tempest workers to 3 https://review.openstack.org/628024 | 22:33 |
| *** mcornea has quit IRC | 22:33 | |
| Tengu | mwhahaha: hmmmm apparently MAYBE the error I have now can help the container team. Let's be positive :) | 22:36 |
| Tengu | EmilienM: -^^ | 22:40 |
| *** ekultails has quit IRC | 22:40 | |
| mwhahaha | we can dream | 22:41 |
| Tengu | mwhahaha: you're send bad waves. that's not cool ;) | 22:42 |
| sudeepbatra | mwhahaha, was following this link > https://docs.openstack.org/tripleo-docs/latest/install/troubleshooting/troubleshooting-overcloud.html | 22:43 |
| mwhahaha | Tengu: containers will do that to you | 22:43 |
| Tengu | mwhahaha: ;_; | 22:43 |
| mwhahaha | speaking of containers | 22:44 |
| * mwhahaha goes crying into kolla | 22:44 | |
| Tengu | uhuhuhuhu | 22:44 |
| Tengu | I heard some things today indeed. | 22:44 |
| *** brault has joined #tripleo | 22:45 | |
| *** brault has quit IRC | 22:45 | |
| *** rcernin has joined #tripleo | 22:46 | |
| openstackgerrit | mathieu bultel proposed openstack/tripleo-common master: Catch heat exception in package_update action https://review.openstack.org/630986 | 22:48 |
| sudeepbatra | $ heat resource-show overcloud Compute > | resource_status_reason | CREATE aborted (Task create from ResourceGroup "Compute" Stack "overcloud" [c9e86d7f-bcbd-4fe4-9905-0cb9efe34b7c] Timed out) | 22:49 |
| *** vinaykns has quit IRC | 22:50 | |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Update tht,tripleoclient and tc before standalone_upgrade. https://review.openstack.org/630228 | 22:52 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Don't force Horizon's secure cookies to disabled https://review.openstack.org/619256 | 22:52 |
| openstackgerrit | Merged openstack/puppet-tripleo stable/queens: Configure switchedv while there is no vfs bounded https://review.openstack.org/630605 | 22:52 |
| sudeepbatra | but the nova list shows the overcloud nodes as Status = ERROR, so I cant do ssh heat-admin@compute to the overcloud nodes and check nova/ironic logs | 22:52 |
| mwhahaha | sudeepbatra: the nova/ironic logs are ont he undercloud | 22:55 |
| *** suuuper has quit IRC | 22:56 | |
| rlandy | mwhahaha: I think I found my missing update - thanks | 22:56 |
| sudeepbatra | [stack@undercloud ~]$ sudo ls /var/log/nova/ ls: cannot access /var/log/nova/: No such file or directory | 22:57 |
| openstackgerrit | Cédric Jeanneret proposed openstack/tripleo-heat-templates master: Ensure we get the file directly in the container when running podman https://review.openstack.org/630287 | 22:58 |
| mwhahaha | sudeepbatra: /var/log/containers/ | 22:59 |
| openstackgerrit | Cédric Jeanneret proposed openstack/tripleo-heat-templates master: DNM - podman testing https://review.openstack.org/628379 | 22:59 |
| openstackgerrit | Cédric Jeanneret proposed openstack/tripleo-heat-templates master: DNM - podman testing https://review.openstack.org/628380 | 22:59 |
| openstackgerrit | Cédric Jeanneret proposed openstack/tripleo-heat-templates master: DNM - podman testing https://review.openstack.org/628381 | 22:59 |
| sudeepbatra | oh I see, this is Queens, so both undercloud and overcloud are containerized ..hmm | 23:00 |
| mwhahaha | sudeepbatra: no, if it's queens the undercloud should be non-containerized | 23:00 |
| mwhahaha | if rocky, it's containerized | 23:00 |
| Tengu | ok, midnight, I really should sleep now. see you folks! | 23:01 |
| *** jtomasek has quit IRC | 23:01 | |
| sudeepbatra | hmm, when I ran ./tripleo-quickstart/quickstart.sh --clean --release master --teardown all --tags all -e undercloud_undercloud_ntp_servers=147.117.251.28 -e undercloud_clean_nodes=false -e @/home/toor/deploy-config.yaml 127.0.0.2 | 23:01 |
| sudeepbatra | release master is default queens ? | 23:01 |
| mwhahaha | no | 23:01 |
| mwhahaha | we're in stein | 23:01 |
| mwhahaha | master is current | 23:01 |
| mwhahaha | so containerized undercloud | 23:01 |
| sudeepbatra | ok , thats good | 23:02 |
| sudeepbatra | I shd be checking nova-conductor and ironic-conductor right ? | 23:04 |
| mwhahaha | nova scheduler/placement probably | 23:04 |
| mwhahaha | just look in all the logs | 23:05 |
| *** toure is now known as toure|gone | 23:06 | |
| *** rascasoft has quit IRC | 23:06 | |
| *** jcoufal has joined #tripleo | 23:06 | |
| *** ooolpbot has joined #tripleo | 23:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1806966 | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1811004 | 23:10 |
| *** ooolpbot has quit IRC | 23:10 | |
| openstack | Launchpad bug 1806966 in tripleo "mistral.rpc.base MessagingTimeout: Timed out waiting for a reply to message ID" [Critical,Triaged] | 23:10 |
| openstack | Launchpad bug 1811004 in tripleo "tripleo-ci-centos-7-scenario003-standalone fails tempest.scenario.test_minimum_basic.TestMinimumBasicScenario" [Critical,In progress] - Assigned to Michele Baldessari (michele) | 23:10 |
| sudeepbatra | cant see any meaningful error - checked ironic-conductor,app.log,nova-conductor,nova-scheduler | 23:14 |
| sudeepbatra | nova-placement > 2019-01-15 18:53:24.950 18 DEBUG nova.api.openstack.placement.wsgi [-] logging_exception_prefix = %(asctime)s.%(msecs)03d %(process)d ERROR %(name)s %(instance)s log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2571 | 23:18 |
| sudeepbatra | +mwhahaha, doesnt seem fatal ^ | 23:19 |
| *** ade_lee has quit IRC | 23:22 | |
| *** bfournie has quit IRC | 23:22 | |
| mwhahaha | that's just debug info | 23:22 |
| mwhahaha | go look for retries or timeouts | 23:22 |
| sudeepbatra | ok | 23:24 |
| *** rlandy has quit IRC | 23:33 | |
| *** rascasoft has joined #tripleo | 23:34 | |
| *** rascasoft has quit IRC | 23:39 | |
| *** jcoufal has quit IRC | 23:45 | |
| *** salmankhan has quit IRC | 23:56 | |
| *** wolverineav has quit IRC | 23:56 | |
| *** ade_lee has joined #tripleo | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!