| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart master: DNM, test ovs-2.9 https://review.openstack.org/555873 | 00:01 |
|---|---|---|
| weshay | somehow | 00:01 |
| weshay | some [where|how] | 00:02 |
| *** gvrangan has quit IRC | 00:02 | |
| weshay | mwhahaha, EmilienM we look green across master and queens, waiting on tempest which I hear is a full pass on the last hash and ovn | 00:04 |
| EmilienM | woot woot | 00:04 |
| weshay | if I could just get my patch to update the containers w/ ovs29 | 00:05 |
| mwhahaha | A likely story | 00:05 |
| weshay | MUTHA FAKA | 00:05 |
| EmilienM | is that a new dialect? | 00:05 |
| weshay | leetard | 00:05 |
| openstackgerrit | Merged openstack/os-net-config master: Don't restart ivs/nfvswitch in os-net-config https://review.openstack.org/555369 | 00:06 |
| weshay | dang.. that's already in urbandictionary | 00:06 |
| weshay | thought I was clever.. NOPE | 00:06 |
| EmilienM | he's funny | 00:07 |
| *** chkumar246 has joined #tripleo | 00:09 | |
| *** ooolpbot has joined #tripleo | 00:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757556 | 00:10 |
| *** ooolpbot has quit IRC | 00:10 | |
| openstack | Launchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged] | 00:10 |
| *** rlandy is now known as rlandy|bbl | 00:10 | |
| *** jobewan has joined #tripleo | 00:11 | |
| *** rcernin has quit IRC | 00:19 | |
| *** rcernin has joined #tripleo | 00:19 | |
| *** liverpooler has joined #tripleo | 00:19 | |
| *** jobewan has quit IRC | 00:23 | |
| openstackgerrit | Marius Cornea proposed openstack/tripleo-upgrade master: Support custom stack names during FFU https://review.openstack.org/555975 | 00:25 |
| openstackgerrit | Marius Cornea proposed openstack/tripleo-upgrade master: WIP: Add support for per role FFU upgrade https://review.openstack.org/556687 | 00:25 |
| openstackgerrit | Emilien Macchi proposed openstack/python-tripleoclient master: add lower-constraints job https://review.openstack.org/556195 | 00:27 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-validations master: add lower-constraints job https://review.openstack.org/556267 | 00:30 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-common master: add lower-constraints job https://review.openstack.org/556265 | 00:31 |
| *** SlickNik has joined #tripleo | 00:32 | |
| openstackgerrit | Emilien Macchi proposed openstack/os-apply-config master: add lower-constraints job https://review.openstack.org/556040 | 00:32 |
| openstackgerrit | Emilien Macchi proposed openstack/os-collect-config master: add lower-constraints job https://review.openstack.org/556043 | 00:33 |
| openstackgerrit | Emilien Macchi proposed openstack/os-refresh-config master: add lower-constraints job https://review.openstack.org/556044 | 00:33 |
| *** SlickNik has quit IRC | 00:40 | |
| *** fragatin_ has quit IRC | 00:40 | |
| *** fragatina has joined #tripleo | 00:41 | |
| *** fragatina has quit IRC | 00:41 | |
| *** fragatina has joined #tripleo | 00:41 | |
| *** fragatina has quit IRC | 00:41 | |
| *** lebauce has quit IRC | 00:42 | |
| *** fragatina has joined #tripleo | 00:43 | |
| *** SlickNik has joined #tripleo | 00:44 | |
| *** fragatina has quit IRC | 00:48 | |
| *** dmacpher has joined #tripleo | 00:48 | |
| *** jobewan has joined #tripleo | 00:50 | |
| *** lebauce has joined #tripleo | 00:51 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: undercloud: configure mtu when creating neutron network https://review.openstack.org/553026 | 00:53 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Run ovb-ha (fs001) with a containerized undercloud https://review.openstack.org/542556 | 00:54 |
| openstackgerrit | Merged openstack/tripleo-docs master: Remove the lacp bond mode constraints https://review.openstack.org/555151 | 00:56 |
| *** jobewan has quit IRC | 00:58 | |
| *** psahoo has joined #tripleo | 01:01 | |
| *** liverpooler has quit IRC | 01:08 | |
| *** ooolpbot has joined #tripleo | 01:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757556 | 01:10 |
| *** ooolpbot has quit IRC | 01:10 | |
| openstack | Launchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged] | 01:10 |
| *** jobewan has joined #tripleo | 01:18 | |
| *** trozet has quit IRC | 01:24 | |
| EmilienM | trown, weshay: please remove -2 on https://review.rdoproject.org/r/#/c/12774/ | 01:25 |
| *** gyankum has joined #tripleo | 01:27 | |
| *** cshastri has joined #tripleo | 01:31 | |
| weshay | EmilienM, done, and we should have ovn results soon :) | 01:32 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart master: bump fs20 tempest to 3 workers https://review.openstack.org/556695 | 01:36 |
| *** gvrangan has joined #tripleo | 01:36 | |
| *** gvrangan has quit IRC | 01:37 | |
| *** gvrangan has joined #tripleo | 01:37 | |
| openstackgerrit | Marius Cornea proposed openstack/tripleo-upgrade master: WIP: Add support for per role FFU upgrade https://review.openstack.org/556687 | 01:39 |
| openstackgerrit | Marius Cornea proposed openstack/tripleo-upgrade master: WIP: Add support for per role FFU upgrade https://review.openstack.org/556687 | 01:40 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras master: bump ci overcloud flavor for faster job time https://review.openstack.org/556697 | 01:41 |
| *** gvrangan has quit IRC | 01:42 | |
| *** gvrangan has joined #tripleo | 01:42 | |
| *** thrash is now known as thrash|g0ne | 01:45 | |
| *** liverpooler has joined #tripleo | 01:48 | |
| *** gvrangan has quit IRC | 01:51 | |
| *** jobewan has quit IRC | 01:51 | |
| *** gvrangan has joined #tripleo | 01:52 | |
| openstackgerrit | Marius Cornea proposed openstack/tripleo-upgrade master: WIP: Add support for per role FFU upgrade https://review.openstack.org/556687 | 01:55 |
| *** gvrangan has quit IRC | 01:55 | |
| EmilienM | weshay: nice! :) | 01:59 |
| weshay | The container update worked and updated openvswitch to 2.9 | 02:03 |
| weshay | Mar 27 01:06:38 Updated: 1:openvswitch-2.9.0-3.el7.x86_64 | 02:03 |
| weshay | http://logs.openstack.org/73/555873/10/check/tripleo-ci-centos-7-scenario007-multinode-oooq-container/18cfbb8/logs/subnode-2/var/log/extra/docker/containers/ovn_metadata_agent/log/yum.log.txt.gz | 02:03 |
| weshay | oh and the job passed :) | 02:03 |
| weshay | http://logs.openstack.org/73/555873/10/check/tripleo-ci-centos-7-scenario007-multinode-oooq-container/18cfbb8/logs/tempest.html.gz | 02:03 |
| weshay | EmilienM, ^ | 02:03 |
| *** bkopilov has quit IRC | 02:09 | |
| *** ooolpbot has joined #tripleo | 02:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757556 | 02:10 |
| *** ooolpbot has quit IRC | 02:10 | |
| openstack | Launchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged] | 02:10 |
| *** jmelvin has joined #tripleo | 02:10 | |
| *** lblanchard has quit IRC | 02:12 | |
| *** dmsimard|cave is now known as dmsimard | 02:29 | |
| *** liverpooler has quit IRC | 02:29 | |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Remove adjust-interface-mtus script https://review.openstack.org/546216 | 02:32 |
| *** daidv has joined #tripleo | 02:34 | |
| *** ykarel|away has joined #tripleo | 02:35 | |
| *** ramishra has joined #tripleo | 02:53 | |
| *** jmelvin has quit IRC | 02:55 | |
| *** fragatina has joined #tripleo | 03:02 | |
| openstackgerrit | Merged openstack/tripleo-validations stable/pike: Fix the nova config file location https://review.openstack.org/556457 | 03:04 |
| openstackgerrit | Merged openstack/tripleo-upgrade master: Support custom stack names during FFU https://review.openstack.org/555975 | 03:04 |
| *** fragatina has quit IRC | 03:08 | |
| EmilienM | weshay: nice | 03:10 |
| EmilienM | weshay: looking your email now | 03:10 |
| *** ooolpbot has joined #tripleo | 03:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757556 | 03:10 |
| *** ooolpbot has quit IRC | 03:10 | |
| openstack | Launchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged] | 03:10 |
| *** ramishra has quit IRC | 03:16 | |
| *** ramishra has joined #tripleo | 03:18 | |
| *** rlandy|bbl is now known as rlandy | 03:19 | |
| *** rlandy has quit IRC | 03:19 | |
| *** kmy has quit IRC | 03:21 | |
| *** kmy has joined #tripleo | 03:21 | |
| *** fragatina has joined #tripleo | 03:30 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Honor {{role.name}}ExtraConfig with deprecated params https://review.openstack.org/555795 | 03:32 |
| openstackgerrit | Merged openstack/tripleo-image-elements stable/queens: Increase the image size to 23G https://review.openstack.org/555730 | 03:32 |
| openstackgerrit | Merged openstack/tripleo-common master: Updated from global requirements https://review.openstack.org/555687 | 03:32 |
| *** anande has joined #tripleo | 03:37 | |
| *** bkopilov has joined #tripleo | 03:40 | |
| openstackgerrit | Sai Ram Peesapati proposed openstack/tripleo-heat-templates stable/ocata: Enable Neutron LBaaS Integration https://review.openstack.org/556741 | 03:40 |
| openstackgerrit | Sai Ram Peesapati proposed openstack/tripleo-heat-templates stable/ocata: Enable Neutron LBaaS Integration https://review.openstack.org/556741 | 03:43 |
| openstackgerrit | Sai Ram Peesapati proposed openstack/tripleo-heat-templates stable/ocata: Enable Neutron LBaaS Integration https://review.openstack.org/556741 | 03:44 |
| openstackgerrit | Sai Ram Peesapati proposed openstack/tripleo-heat-templates stable/ocata: Enable Neutron LBaaS Integration https://review.openstack.org/556741 | 03:51 |
| *** fragatina has quit IRC | 03:51 | |
| *** shreshtha has joined #tripleo | 03:53 | |
| *** shreshtha has quit IRC | 03:53 | |
| *** shreshtha has joined #tripleo | 03:54 | |
| *** shreshtha is now known as shreshtha|wfh | 03:54 | |
| *** ykarel|away is now known as ykarel | 03:54 | |
| *** udesale has joined #tripleo | 03:55 | |
| openstackgerrit | Sai Ram Peesapati proposed openstack/tripleo-heat-templates stable/ocata: Enable Neutron LBaaS Integration https://review.openstack.org/556741 | 03:56 |
| *** khrystoph has quit IRC | 03:59 | |
| *** khrystoph has joined #tripleo | 03:59 | |
| *** fragatina has joined #tripleo | 03:59 | |
| openstackgerrit | Sai Ram Peesapati proposed openstack/tripleo-heat-templates stable/newton: Enable Neutron LBaaS Integration https://review.openstack.org/556743 | 04:00 |
| openstackgerrit | Sai Ram Peesapati proposed openstack/tripleo-heat-templates stable/newton: Enable Neutron LBaaS Integration https://review.openstack.org/556743 | 04:01 |
| *** ykarel has quit IRC | 04:04 | |
| *** ooolpbot has joined #tripleo | 04:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757556 | 04:10 |
| *** ooolpbot has quit IRC | 04:10 | |
| openstack | Launchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged] | 04:10 |
| openstackgerrit | Merged openstack/tripleo-validations master: Fix docker img filter in rabbitmq validation https://review.openstack.org/555306 | 04:13 |
| openstackgerrit | Merged openstack/python-tripleoclient master: Updated from global requirements https://review.openstack.org/555234 | 04:13 |
| *** karthiks has quit IRC | 04:18 | |
| *** Tyrantelf has quit IRC | 04:22 | |
| *** Tyrantelf has joined #tripleo | 04:23 | |
| *** yamahata has joined #tripleo | 04:24 | |
| openstackgerrit | Merged openstack/python-tripleoclient master: Don't delete the tripleo Zaqar queue https://review.openstack.org/555193 | 04:25 |
| openstackgerrit | Merged openstack/tripleo-common master: Add and fix healthcheck scripts for Octavia services https://review.openstack.org/554946 | 04:25 |
| *** psachin has joined #tripleo | 04:28 | |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: Change the GENTOO_PORTAGE_CLEANUP variable default https://review.openstack.org/556749 | 04:28 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/queens: Add support for Dell EMC XtremIO Cinder ISCSI Backend https://review.openstack.org/554331 | 04:32 |
| openstackgerrit | Merged openstack/python-tripleoclient master: Use openstackdocstheme for python-tripleoclient https://review.openstack.org/554139 | 04:32 |
| openstackgerrit | Merged openstack/tripleo-validations master: Fix overcloud services connectivity validation https://review.openstack.org/553832 | 04:32 |
| *** tzumainn has quit IRC | 04:33 | |
| openstackgerrit | Sai Ram Peesapati proposed openstack/tripleo-heat-templates stable/ocata: Allows Configuration of Neutron Parameter https://review.openstack.org/556752 | 04:33 |
| openstackgerrit | Sai Ram Peesapati proposed openstack/tripleo-heat-templates stable/ocata: Allows Configuration of Neutron Parameter https://review.openstack.org/556752 | 04:34 |
| openstackgerrit | Nguyen Hai proposed openstack/python-tripleoclient master: Follow the new PTI for document build [Updated] https://review.openstack.org/556753 | 04:39 |
| openstackgerrit | Sai Ram Peesapati proposed openstack/tripleo-heat-templates stable/newton: Allows Configuration of Neutron Parameter https://review.openstack.org/556754 | 04:40 |
| *** anande has quit IRC | 04:40 | |
| openstackgerrit | Merged openstack/tripleo-image-elements stable/pike: Increase the /var partition from 7G to 10G https://review.openstack.org/553768 | 04:44 |
| openstackgerrit | Merged openstack/python-tripleoclient stable/queens: Pass plan/stack name into config_download workflow https://review.openstack.org/553753 | 04:44 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/queens: Configure deploy logs for Ironic https://review.openstack.org/551241 | 04:44 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Enable nova-metadata health check https://review.openstack.org/550706 | 04:44 |
| *** agopi|out has quit IRC | 04:45 | |
| *** agopi|out has joined #tripleo | 04:46 | |
| *** ykarel has joined #tripleo | 04:46 | |
| Tengu | hello there | 04:47 |
| *** janki has joined #tripleo | 04:48 | |
| *** gvrangan has joined #tripleo | 04:48 | |
| *** gvrangan has quit IRC | 04:48 | |
| openstackgerrit | Merged openstack/tripleo-common master: Add healthcheck script for Nova metadata https://review.openstack.org/550441 | 04:49 |
| openstackgerrit | Merged openstack/tripleo-quickstart master: add the ability to set the root passwd on the overcloud-full image https://review.openstack.org/547163 | 04:49 |
| *** gvrangan has joined #tripleo | 04:49 | |
| jaosorior | good morning folks! | 04:53 |
| *** gvrangan has quit IRC | 04:54 | |
| openstackgerrit | Merged openstack/tripleo-common master: Install python2-networking-baremetal in neutron-server https://review.openstack.org/545452 | 04:57 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-common master: Refactor setting default CA https://review.openstack.org/554835 | 04:58 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-common master: Read bytes for default CA https://review.openstack.org/554836 | 04:58 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-common master: TLS by default for the overcloud if no certificate provided https://review.openstack.org/554926 | 04:58 |
| *** agopi|out has quit IRC | 05:00 | |
| *** dpawar has joined #tripleo | 05:02 | |
| *** marios has joined #tripleo | 05:03 | |
| *** dmacpher has quit IRC | 05:07 | |
| *** dmacpher has joined #tripleo | 05:08 | |
| *** ooolpbot has joined #tripleo | 05:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757556 | 05:10 |
| *** ooolpbot has quit IRC | 05:10 | |
| openstack | Launchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged] | 05:10 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-common master: TLS by default for the overcloud if no certificate provided https://review.openstack.org/554926 | 05:17 |
| *** ratailor has joined #tripleo | 05:18 | |
| *** fragatina has quit IRC | 05:20 | |
| *** fragatina has joined #tripleo | 05:20 | |
| *** ratailor is now known as rtailor | 05:21 | |
| *** pdeore has joined #tripleo | 05:26 | |
| *** karthiks has joined #tripleo | 05:28 | |
| *** etingof has joined #tripleo | 05:29 | |
| *** moshele has joined #tripleo | 05:41 | |
| *** dsariel has joined #tripleo | 05:50 | |
| *** jfrancoa has joined #tripleo | 05:53 | |
| *** nyechiel_ has joined #tripleo | 05:54 | |
| *** dpawar has quit IRC | 06:00 | |
| *** moshele has quit IRC | 06:00 | |
| *** ooolpbot has joined #tripleo | 06:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757556 | 06:10 |
| openstack | Launchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged] | 06:10 |
| *** ooolpbot has quit IRC | 06:10 | |
| Tengu | hello! is there a way to limite max PG in ceph using puppet-ceph? Apparently the CephPoolDefaultPgNum is only for ceph-ansible -.- | 06:13 |
| *** agurenko has joined #tripleo | 06:14 | |
| *** psachin has quit IRC | 06:14 | |
| *** mrunge has quit IRC | 06:15 | |
| *** moshele has joined #tripleo | 06:16 | |
| *** mrunge has joined #tripleo | 06:17 | |
| *** pgadiya has joined #tripleo | 06:19 | |
| *** pblaho has joined #tripleo | 06:19 | |
| *** psachin has joined #tripleo | 06:24 | |
| *** moshele has quit IRC | 06:24 | |
| *** anilvenkata has quit IRC | 06:31 | |
| *** dbecker has quit IRC | 06:31 | |
| *** yurii has joined #tripleo | 06:34 | |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart-extras master: Refactored validate-tempest role for undercloud and containers https://review.openstack.org/551441 | 06:39 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Use ironic::inspector::dnsmasq_ip_subnets https://review.openstack.org/543582 | 06:42 |
| *** anilvenkata has joined #tripleo | 06:44 | |
| *** dbecker has joined #tripleo | 06:46 | |
| *** pliu_ has joined #tripleo | 06:46 | |
| *** zoli is now known as zoli|trng | 06:47 | |
| *** zoli|trng is now known as zoli | 06:48 | |
| *** dpawar has joined #tripleo | 06:49 | |
| *** moshele has joined #tripleo | 06:51 | |
| *** pcaruana has joined #tripleo | 06:53 | |
| *** moshele has quit IRC | 06:57 | |
| *** radeks has joined #tripleo | 06:57 | |
| *** jaosorior has quit IRC | 07:02 | |
| openstackgerrit | Damien Ciabrini proposed openstack/puppet-tripleo master: HAProxy: expose stats socket in HA containerized deployment https://review.openstack.org/556766 | 07:04 |
| *** dsariel has quit IRC | 07:05 | |
| *** nyechiel has joined #tripleo | 07:06 | |
| *** nyechiel_ has quit IRC | 07:06 | |
| *** aufi has joined #tripleo | 07:09 | |
| *** ooolpbot has joined #tripleo | 07:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757556 | 07:10 |
| *** ooolpbot has quit IRC | 07:10 | |
| openstack | Launchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged] | 07:10 |
| *** cylopez has joined #tripleo | 07:12 | |
| *** cylopez has left #tripleo | 07:13 | |
| *** bogdando has joined #tripleo | 07:14 | |
| *** tesseract has joined #tripleo | 07:15 | |
| *** jtomasek has joined #tripleo | 07:15 | |
| *** ebarrera has joined #tripleo | 07:19 | |
| *** aufi has quit IRC | 07:21 | |
| *** aufi has joined #tripleo | 07:25 | |
| *** rcernin has quit IRC | 07:25 | |
| *** dsariel has joined #tripleo | 07:28 | |
| *** gfidente has joined #tripleo | 07:29 | |
| *** akrivoka has joined #tripleo | 07:43 | |
| *** pblaho has quit IRC | 07:44 | |
| *** psachin has quit IRC | 07:46 | |
| *** ykarel is now known as ykarel|lunch | 07:49 | |
| *** kopecmartin has joined #tripleo | 07:49 | |
| *** kopecmartin has joined #tripleo | 07:49 | |
| openstackgerrit | Damien Ciabrini proposed openstack/tripleo-heat-templates master: HAProxy: expose stats socket in HA deployment after upgrade https://review.openstack.org/556772 | 07:52 |
| *** jaosorior has joined #tripleo | 07:53 | |
| *** tosky has joined #tripleo | 07:54 | |
| *** gkadam has joined #tripleo | 07:55 | |
| openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart-extras master: GATE CHECK for quickstart-extras https://review.openstack.org/472607 | 07:55 |
| *** numans has quit IRC | 07:57 | |
| *** elgxl has joined #tripleo | 07:58 | |
| *** ktibi has joined #tripleo | 07:59 | |
| *** numans has joined #tripleo | 08:00 | |
| *** lucas-afk is now known as lucasagomes | 08:01 | |
| *** mdnadeem has joined #tripleo | 08:01 | |
| chkumar246 | arxcruz|rover: Hello | 08:03 |
| chkumar246 | arxcruz|rover: regarding this review https://review.openstack.org/#/c/551441/ | 08:03 |
| chkumar246 | and tempest_dir | 08:03 |
| *** jpich has joined #tripleo | 08:05 | |
| *** derekh has joined #tripleo | 08:09 | |
| *** ooolpbot has joined #tripleo | 08:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757556 | 08:10 |
| *** ooolpbot has quit IRC | 08:10 | |
| openstack | Launchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged] | 08:10 |
| *** suuuper has joined #tripleo | 08:18 | |
| *** cylopez has joined #tripleo | 08:18 | |
| *** pdeore has quit IRC | 08:22 | |
| *** ccamacho has joined #tripleo | 08:23 | |
| *** ccamacho has quit IRC | 08:23 | |
| *** pdeore has joined #tripleo | 08:25 | |
| *** ccamacho has joined #tripleo | 08:27 | |
| *** ccamacho is now known as ccamacho|PTO | 08:28 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates master: undercloud: configure mtu when creating neutron network https://review.openstack.org/553026 | 08:28 |
| *** ykarel|lunch is now known as ykarel | 08:30 | |
| *** matbu has quit IRC | 08:33 | |
| *** matbu has joined #tripleo | 08:33 | |
| *** mdnadeem_ has joined #tripleo | 08:36 | |
| *** gkadam_ has joined #tripleo | 08:37 | |
| *** mdnadeem has quit IRC | 08:37 | |
| *** pgadiya has quit IRC | 08:37 | |
| *** pgadiya has joined #tripleo | 08:38 | |
| *** pblaho has joined #tripleo | 08:39 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-quickstart master: Run ovb-ha (fs001) with a containerized undercloud https://review.openstack.org/542556 | 08:39 |
| *** gkadam has quit IRC | 08:39 | |
| *** holser__ has joined #tripleo | 08:40 | |
| *** gyankum has quit IRC | 08:41 | |
| *** gyan_ has joined #tripleo | 08:41 | |
| *** gkadam_ has quit IRC | 08:42 | |
| *** gkadam_ has joined #tripleo | 08:42 | |
| *** gyan__ has joined #tripleo | 08:44 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Upgrade data on disk on mysql major upgrade - non HA https://review.openstack.org/555480 | 08:44 |
| *** yamahata has quit IRC | 08:44 | |
| *** cshastri has quit IRC | 08:44 | |
| *** gyan_ has quit IRC | 08:47 | |
| *** bogdando has quit IRC | 08:50 | |
| *** bogdando has joined #tripleo | 08:50 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-upgrade stable/queens: DNM: Testing FFU Patches. https://review.openstack.org/556787 | 08:56 |
| openstackgerrit | Damien Ciabrini proposed openstack/tripleo-heat-templates stable/queens: Upgrade data on disk on mysql major upgrade - non HA https://review.openstack.org/556790 | 08:59 |
| *** mdnadeem_ has quit IRC | 08:59 | |
| *** mdnadeem has joined #tripleo | 08:59 | |
| bogdando | PTAL https://review.openstack.org/#/c/550081/ | 09:02 |
| bogdando | shardy, jaosorior: ^^ | 09:02 |
| *** holser__ has quit IRC | 09:03 | |
| *** hjensas has quit IRC | 09:03 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/python-tripleoclient master: Generate inventory with tripleo-admin user https://review.openstack.org/554545 | 09:04 |
| *** holser__ has joined #tripleo | 09:04 | |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-validations stable/newton: Sanitize validations output https://review.openstack.org/537419 | 09:06 |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-quickstart-extras master: Modify failed tripleo validations log file name https://review.openstack.org/522239 | 09:07 |
| *** ooolpbot has joined #tripleo | 09:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757556 | 09:10 |
| *** ooolpbot has quit IRC | 09:10 | |
| openstack | Launchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged] | 09:10 |
| derekh | Can anybody tell me if this is a good/bad idea in CI ? https://review.openstack.org/#/c/555737/1 | 09:18 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates master: Default environment/services/* to docker https://review.openstack.org/556793 | 09:19 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates master: Delete not-used services-docker files https://review.openstack.org/556794 | 09:19 |
| derekh | looks like we're not testing the change for dib and t-c | 09:19 |
| shardy | derekh: isn't tripleo-common in the -current whitelist, e.g we're already running trunk dlrn packages for that? | 09:20 |
| derekh | shardy: yup, id imagine so, so we shouldn't be pip installing it at all | 09:21 |
| shardy | https://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/master.yml#L60..L67 | 09:21 |
| shardy | derekh: yeah I think it's the same for dib | 09:21 |
| shardy | derekh: I'm not sure we should be doing any of that repo setup or pip install but perhaps there's a reason for it | 09:23 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates master: Default environment/services/* to docker https://review.openstack.org/550060 | 09:24 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates master: Delete not-used services-docker files https://review.openstack.org/556794 | 09:24 |
| *** elgxl has quit IRC | 09:24 | |
| derekh | shardy: ok, I'll see if I can figure out whats going on, /me may have gotten sent down the wrong route looking at pip | 09:24 |
| shardy | derekh: ack, tbh I'm not really familiar with the tripleo-buildimage stuff but I always thought quickstart should be run on a clean host, even when we're using deployed server for the multinode jobs | 09:26 |
| shardy | maybe sshnaidm or weshay will have some comments | 09:26 |
| *** cylopez has left #tripleo | 09:26 | |
| derekh | shardy: ack , sshnaidm, weshay any thoughts on above? ^ | 09:27 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/python-tripleoclient master: Default environment/services/* to docker https://review.openstack.org/556797 | 09:27 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-quickstart master: Default environment/services/* to docker https://review.openstack.org/556798 | 09:27 |
| *** yurii has quit IRC | 09:28 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-quickstart-extras master: Default environment/services/* to docker https://review.openstack.org/556799 | 09:29 |
| bogdando | jaosorior, slagle, mandre: PTAL https://review.openstack.org/#/q/topic:containers-default+(status:open+OR+status:merged) | 09:30 |
| *** nyechiel has quit IRC | 09:30 | |
| sshnaidm | derekh, can you bring a little bit context please? I'm not familiar with that | 09:31 |
| bogdando | can we merge https://review.openstack.org/#/c/550081 as is please? | 09:31 |
| derekh | sshnaidm: the original problem happened when this patch merged https://review.openstack.org/#/c/539947/ | 09:32 |
| derekh | sshnaidm: image building stopped working and ianw looked into it, the conclusion was that we were using the new yaml files but because the python was getting pip installed the required pythong code change wasn't being used | 09:33 |
| derekh | sshnaidm: which led to this patch https://review.openstack.org/#/c/554705/ | 09:34 |
| derekh | sshnaidm: now I am attempting to backport the 4k patch to queens, but first need to fix the same problem for stable branches | 09:35 |
| *** anilvenkata has quit IRC | 09:36 | |
| shardy | bogdando: I left a comment - dprince and I disagree on this, personally I don't think we should make it easy to deploy combinations of containers/baremetal that we won't actually test | 09:37 |
| shardy | not blocking but perhaps we can get feedback from others on that | 09:38 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-quickstart master: Fix Fast Forward featureset documentation. https://review.openstack.org/556801 | 09:38 |
| sshnaidm | derekh, I see.. btw, you need to add queens repos to this path, it has pike only.. | 09:40 |
| sshnaidm | s/path/patch | 09:40 |
| derekh | looking | 09:41 |
| sshnaidm | derekh, besides of that looks ok, but I can miss something, it's old legacy code that I've never touched. | 09:42 |
| derekh | sshnaidm: ok, thanks, I'll update the patch to add the queens repo, so it a step closer ;-) | 09:43 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci master: Install the zuul-cloned dib and tripleo-common https://review.openstack.org/555737 | 09:45 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-common master: TLS by default for the overcloud https://review.openstack.org/554926 | 09:48 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-common master: TLS by default for the overcloud https://review.openstack.org/554926 | 09:55 |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-quickstart-extras master: Overcloud update CI with tripleo-upgrade role https://review.openstack.org/542201 | 09:55 |
| *** salmankhan has joined #tripleo | 09:56 | |
| openstackgerrit | Arx Cruz proposed openstack/tripleo-quickstart-extras master: Refactored validate-tempest role for undercloud and containers https://review.openstack.org/551441 | 10:00 |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-quickstart-extras master: Overcloud update CI with tripleo-upgrade role https://review.openstack.org/542201 | 10:00 |
| openstackgerrit | Arx Cruz proposed openstack/tripleo-quickstart-extras master: Refactored validate-tempest role for undercloud and containers https://review.openstack.org/551441 | 10:01 |
| *** ykarel_ has joined #tripleo | 10:03 | |
| *** psahoo_ has joined #tripleo | 10:03 | |
| *** mdnadeem_ has joined #tripleo | 10:03 | |
| *** gyan_ has joined #tripleo | 10:03 | |
| *** ratailor_ has joined #tripleo | 10:03 | |
| *** gkadam__ has joined #tripleo | 10:03 | |
| *** chandankumar has joined #tripleo | 10:03 | |
| *** udesale_ has joined #tripleo | 10:03 | |
| *** pdeore_ has joined #tripleo | 10:03 | |
| *** salmankhan has quit IRC | 10:04 | |
| *** chkumar246 has quit IRC | 10:04 | |
| *** rtailor has quit IRC | 10:04 | |
| *** ykarel has quit IRC | 10:04 | |
| *** pdeore has quit IRC | 10:04 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates master: WIP: Migrate Ceph deployment to external_deploy_task https://review.openstack.org/546966 | 10:04 |
| *** mdnadeem has quit IRC | 10:04 | |
| *** gyan__ has quit IRC | 10:05 | |
| *** udesale__ has joined #tripleo | 10:05 | |
| *** pdeore__ has joined #tripleo | 10:05 | |
| *** ykarel__ has joined #tripleo | 10:05 | |
| *** gyan__ has joined #tripleo | 10:05 | |
| *** ratailor__ has joined #tripleo | 10:06 | |
| *** gkadam has joined #tripleo | 10:06 | |
| *** gkadam_ has quit IRC | 10:07 | |
| *** udesale has quit IRC | 10:07 | |
| *** psahoo has quit IRC | 10:07 | |
| *** udesale_ has quit IRC | 10:08 | |
| *** gyan_ has quit IRC | 10:08 | |
| *** pdeore_ has quit IRC | 10:08 | |
| *** ykarel_ has quit IRC | 10:08 | |
| *** chandankumar has quit IRC | 10:08 | |
| openstackgerrit | Numan Siddique proposed openstack/tripleo-quickstart master: Enable OVN HA profile in fset30 https://review.openstack.org/523804 | 10:08 |
| *** ratailor_ has quit IRC | 10:08 | |
| *** gkadam__ has quit IRC | 10:08 | |
| *** mdnadeem_ has quit IRC | 10:08 | |
| *** psahoo_ has quit IRC | 10:09 | |
| *** ooolpbot has joined #tripleo | 10:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757556 | 10:10 |
| *** ooolpbot has quit IRC | 10:10 | |
| openstack | Launchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged] | 10:10 |
| *** ykarel__ is now known as ykarel | 10:11 | |
| *** derekh is now known as derekh_afk | 10:13 | |
| *** mdnadeem_ has joined #tripleo | 10:20 | |
| *** chandankumar has joined #tripleo | 10:20 | |
| therve | shardy, reg https://review.openstack.org/#/c/555312/5 , the resource just doesn't exist | 10:20 |
| *** psahoo_ has joined #tripleo | 10:21 | |
| *** anilvenkata has joined #tripleo | 10:22 | |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart master: Run tempest on undercloud on fs003 https://review.openstack.org/554522 | 10:23 |
| *** tosky__ has joined #tripleo | 10:25 | |
| *** tosky is now known as Guest72548 | 10:25 | |
| *** tosky__ is now known as tosky | 10:25 | |
| shardy | therve: Ah, I thought we aliased both *Deployment types, fair enough :) | 10:26 |
| akrivoka | hey jtomasek could you help me debug something when you have a moment | 10:26 |
| *** psahoo_ has quit IRC | 10:26 | |
| therve | shardy, That would have solved some issues :) | 10:27 |
| therve | shardy, wdyt of the solution btw? | 10:29 |
| shardy | therve: It seems OK but it's kind of a shame we can't handle it in a generic way via the inventory | 10:31 |
| shardy | therve: I added a small nit to the -common patch | 10:31 |
| shardy | therve: thanks for looking into this, I guess the important thing is we have a workable solution now :) | 10:31 |
| therve | shardy, Yeah I'll write a fix in the mean time. We could if we were earlier in the cycle... | 10:31 |
| *** gkadam has quit IRC | 10:32 | |
| shardy | s/inventory/registry | 10:32 |
| * shardy has been doing too much ansible today | 10:32 | |
| therve | shardy, Hum your nit is interesting, it probably means it can't work as is | 10:32 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-quickstart master: Add missing Featureset037 documentation. https://review.openstack.org/556814 | 10:33 |
| openstackgerrit | Thomas Herve proposed openstack/tripleo-common master: Add custom subclass to revert mapping https://review.openstack.org/556491 | 10:33 |
| chem | jistr: jfrancoa ^ | 10:33 |
| chem | jistr: jfrancoa I've done it on top of my featureset update, I hate conflict | 10:34 |
| *** gyan__ has quit IRC | 10:34 | |
| *** gkadam has joined #tripleo | 10:35 | |
| *** mdnadeem has joined #tripleo | 10:36 | |
| *** bkopilov has quit IRC | 10:36 | |
| *** gyan__ has joined #tripleo | 10:36 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Container Undercloud - Enable neutron segments https://review.openstack.org/555424 | 10:37 |
| openstackgerrit | Merged openstack/tripleo-upgrade stable/queens: Include connectivity check prepare scripts during FFU https://review.openstack.org/556573 | 10:37 |
| *** salmankhan has joined #tripleo | 10:37 | |
| *** mdnadeem_ has quit IRC | 10:38 | |
| Tengu | fultonj: hello! Apparently the first step is on a good track for my migration. I just had some issues with ceph itself, as the deployed one was broken (only 2 ceph servers, and didn't put the correct configuration for pool size and so on). I owe you a couple of beers if this succeeds :) | 10:40 |
| *** marios has quit IRC | 10:42 | |
| *** marios has joined #tripleo | 10:43 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Default NodeTLSData to always attempt setting TLS cert https://review.openstack.org/556817 | 10:45 |
| *** moshele has joined #tripleo | 10:46 | |
| *** moshele has quit IRC | 10:48 | |
| yolanda | hi, i'm trying to compose some documentation about tripleo cleanup. I see that puppet-nova offers a purge_shadow_tables cron. But is there some parameter to manage that from TripleO ? | 10:49 |
| *** udesale__ has quit IRC | 10:51 | |
| *** salmankhan has quit IRC | 10:51 | |
| shardy | yolanda: we may be missing that one but various nova::cron things are configured in puppet/services/nova-base.yaml | 10:51 |
| shardy | so that could be added if needed | 10:51 |
| yolanda | shardy, so i can see the code in manifests/cron/purge_shadow_tables.pp, but i cannot see it being called in nova-base, or managed by tripleo | 10:52 |
| yolanda | do you know if there is an effort to add it? | 10:52 |
| yolanda | i was asked by a partner to get info on all the possible cleaning steps that could be done in database | 10:53 |
| shardy | yolanda: it's not in nova-base AFAICS, and I'm not aware of any effort but it seems reasonable to add it | 10:53 |
| yolanda | ok, i will open a BZ | 10:53 |
| shardy | owalsh may be able to comment/help ^^ | 10:53 |
| *** salmankhan has joined #tripleo | 10:53 | |
| yolanda | a launchpad bug as well | 10:53 |
| shardy | yolanda: ack, probably a fairly easy addition I guess | 10:53 |
| yolanda | yep, doesn't look complex | 10:54 |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart-extras master: Switch to tempest run cli for running tempest tests https://review.openstack.org/556014 | 10:54 |
| shardy | yolanda: for now they can use e.g ControllerExtraConfig overrides to configure it | 10:54 |
| yolanda | i saw some efforts for ccamacho to use that, but on the undercloud | 10:54 |
| shardy | yolanda: Ah yes I see https://review.openstack.org/#/q/topic:bp-db-cleanup+(status:open+OR+status:merged) | 10:55 |
| shardy | https://review.openstack.org/#/c/551966/11/puppet/services/nova-base.yaml seems to do what you need? | 10:55 |
| owalsh | yolanda, shardy: yea, ^^^ | 10:56 |
| yolanda | ah, missed that one, i only could see the change in the undercloud | 10:56 |
| yolanda | nice | 10:56 |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart-extras master: Switch to tempest run cli for running tempest tests https://review.openstack.org/556014 | 10:56 |
| shardy | heh always nice when a problem is already solved :) | 10:56 |
| shardy | thanks owalsh sorry for the interrupt | 10:56 |
| yolanda | thx for the pointer there | 10:57 |
| owalsh | np | 10:57 |
| jtomasek | akrivoka: hi, sure | 10:57 |
| yolanda | btw, shardy, owalsh , i'm getting constant RDO third party CI failures on one patch. Is that expected? is mostly cirros vms timeout | 10:58 |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart master: Run tempest on fs027 using containerized tempest https://review.openstack.org/551442 | 10:58 |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart master: Run tempest on fs027 using containerized tempest https://review.openstack.org/551442 | 10:59 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Add HostsEntry to undercloud's /etc/hosts https://review.openstack.org/555041 | 10:59 |
| *** moshele has joined #tripleo | 11:00 | |
| owalsh | yolanda: this one? https://bugs.launchpad.net/tripleo/+bug/1757556 | 11:00 |
| openstack | Launchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged] | 11:00 |
| yolanda | going to check tempest logs, in console i can see tempest.lib.exceptions.SSHTimeout: Connection to the 10.0.0.105 via SSH timed out | 11:03 |
| openstackgerrit | Chandan Kumar proposed openstack/python-tripleoclient master: Added fake tempest docker service to undercloud config https://review.openstack.org/551271 | 11:03 |
| yolanda | owalsh, looks similar yes | 11:04 |
| yolanda | https://logs.rdoproject.org/59/555259/1/openstack-check/gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset042-master-tht/Zed570ac9f3e84b2ba84b9514c08ad2a3/undercloud/home/jenkins/tempest/tempest.log.txt.gz | 11:04 |
| *** janki has quit IRC | 11:06 | |
| *** lucasagomes is now known as lucas-hungry | 11:08 | |
| *** dpawar has quit IRC | 11:09 | |
| *** hjensas has joined #tripleo | 11:09 | |
| *** hjensas has quit IRC | 11:09 | |
| *** hjensas has joined #tripleo | 11:09 | |
| *** ooolpbot has joined #tripleo | 11:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757556 | 11:10 |
| *** ooolpbot has quit IRC | 11:10 | |
| openstack | Launchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged] | 11:10 |
| Tengu | hmmm, can anyone from ceph and/or swift have a quick look on this issue? https://bugs.launchpad.net/tripleo/+bug/1759235 I'm pretty sure it's related to my env inclusion, but… | 11:11 |
| openstack | Launchpad bug 1759235 in tripleo "duplicated definition for Keystone_user[swift]" [Undecided,New] | 11:11 |
| openstackgerrit | Merged openstack/tripleo-common master: Fix minor updates for deployed server environments https://review.openstack.org/554484 | 11:12 |
| *** yprokule has joined #tripleo | 11:13 | |
| akrivoka | jtomasek: hey so, I was trying to figure out why swift returns success to the UI even when the tar extraction fails | 11:13 |
| akrivoka | jtomasek: (wrt bz https://bugzilla.redhat.com/show_bug.cgi?id=1560579) | 11:14 |
| openstack | bugzilla.redhat.com bug 1560579 in openstack-swift "Tarball extraction fails but Swift API returns status 200" [Unspecified,Closed: notabug] - Assigned to thiago | 11:14 |
| akrivoka | jtomasek: so I tried plan update from the UI, gave it a bad tar file (e.g. just take a txt file and rename it as .tar.gz) - the PUT call to Swift retuns 200 OK | 11:15 |
| akrivoka | jtomasek: then I tried the same API call directly on the undercloud using curl (literally did "copy as cURL" from Chrome dev tools, and pasted it on the undercloud - the only thing I had to change is add the request data with "-T badfile.tar.gz"), and this time the API correctly returns 400 bad request, error message correctly states the file is not a valid tar archive | 11:15 |
| openstackgerrit | Merged openstack/tripleo-quickstart master: Add tripleo-upgrade specific parameters. https://review.openstack.org/556567 | 11:15 |
| akrivoka | jtomasek: (of course, I used the same badfile.tar.gz that I gave to the browser locally, scp-ed it to the undercloud) | 11:16 |
| jtomasek | akrivoka: ok, I'll check what happens in UI side | 11:16 |
| akrivoka | jtomasek: any idea why/how this could happen...? I'm puzzled :) | 11:16 |
| akrivoka | jtomasek: this is the API call I'm talking about btw: https://github.com/openstack/tripleo-ui/blob/master/src/js/services/SwiftApiService.js#L73 | 11:17 |
| akrivoka | jtomasek: thanks! | 11:17 |
| *** nyechiel has joined #tripleo | 11:18 | |
| jtomasek | akrivoka: I think that 'catch' on line 83 is going to be the problem | 11:18 |
| * jtomasek tests | 11:19 | |
| akrivoka | jtomasek: but even just looking at raw HTTP requests (Network tab of the Chrome dev tools) it doesn't make sense | 11:19 |
| openstackgerrit | yatin proposed openstack/tripleo-heat-templates stable/queens: Correctly set NeutronMl2PluginBase for OVN scenario https://review.openstack.org/556823 | 11:21 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-common master: TLS by default for the overcloud https://review.openstack.org/554926 | 11:22 |
| *** bfournie has quit IRC | 11:27 | |
| *** abishop has joined #tripleo | 11:27 | |
| *** morazi has quit IRC | 11:27 | |
| *** pkovar has joined #tripleo | 11:30 | |
| *** sshnaidm has quit IRC | 11:30 | |
| *** leanderthal has joined #tripleo | 11:31 | |
| *** openstackgerrit has quit IRC | 11:33 | |
| *** ansmith has quit IRC | 11:33 | |
| *** dtantsur|afk is now known as dtantsur | 11:34 | |
| *** gyan__ has quit IRC | 11:34 | |
| *** bkopilov has joined #tripleo | 11:34 | |
| jtomasek | akrivoka: there has to be something wrong with cors, as that is the only difference between issuing the request from browser and cURLing | 11:36 |
| akrivoka | jtomasek: I have no idea how to debug that :( | 11:40 |
| hjensas | mwhahaha: EmilienM: can you have a look at this bp: https://blueprints.launchpad.net/tripleo/+spec/enrich-ironic-networking-integration (Small change, and code already up for review.) | 11:42 |
| *** dpawar has joined #tripleo | 11:44 | |
| *** adarazs is now known as adarazs_lunch | 11:46 | |
| *** udesale has joined #tripleo | 11:47 | |
| *** openstackgerrit has joined #tripleo | 11:47 | |
| openstackgerrit | Merged openstack/tripleo-common master: Remove extra ssh args passed to ansible https://review.openstack.org/554525 | 11:47 |
| jtomasek | akrivoka: I think that in browser, the real error is covered by the CORS error, which we let pass | 11:49 |
| *** jlabarre has joined #tripleo | 11:51 | |
| *** ysandeep has joined #tripleo | 11:53 | |
| *** paramite_ is now known as paramite | 11:53 | |
| *** pdeore__ has quit IRC | 11:53 | |
| ysandeep | ping #tripleo, hi Guys I have a Small Query - Undercloud by default listens for snmp polls on external interfaces, Anyone aware how we can disable it and shouldn't be it disabled by default ? | 11:54 |
| ysandeep | I understood snmp is used to collect celiometer metric from overcloud nodes, To me it should be enabled on management network interfaces but not for external network interfaces because that leads to security vulnerabilities. | 11:54 |
| *** pradk has joined #tripleo | 11:54 | |
| jtomasek | akrivoka: in firefox, it is possible to see the correct response for the request | 11:55 |
| chem | social: let's continue here | 11:55 |
| *** quiquell_ has joined #tripleo | 11:57 | |
| akrivoka | jtomasek: huh, yeah I see | 11:59 |
| jtomasek | but no idea why it shows 200 in network tab | 11:59 |
| openstackgerrit | Merged openstack/tripleo-common master: Cleanup old MariaDB cruft from override file https://review.openstack.org/532793 | 12:00 |
| openstackgerrit | Merged openstack/puppet-tripleo master: HAProxy: expose stats socket in HA containerized deployment https://review.openstack.org/556766 | 12:00 |
| jtomasek | akrivoka: IMHO the root of the problem is what is in the comment in GUI code - "Swift doesn't add CORS headers to successful PUT requests" | 12:00 |
| *** lblanchard has joined #tripleo | 12:00 | |
| shardy | ysandeep: Hi, can you clarify what you see listening on the undercloud? | 12:00 |
| *** pgadiya has quit IRC | 12:00 | |
| openstackgerrit | Damien Ciabrini proposed openstack/puppet-tripleo stable/queens: HAProxy: expose stats socket in HA containerized deployment https://review.openstack.org/556833 | 12:02 |
| shardy | ysandeep: I can see ceilometer is configured to listen for snmp but the telemetry services are disabled by default - possibly this is a bug specific to when they are enabled? | 12:02 |
| ysandeep | shardy, snmp responds on ext vlan interface of undercloud node, I want to use ceilometer | 12:03 |
| *** pchavva has joined #tripleo | 12:03 | |
| *** ratailor__ is now known as ratailor | 12:03 | |
| *** rfolco|off is now known as rfolco|ruckk | 12:03 | |
| *** rfolco|ruckk is now known as rfolco|ruck | 12:03 | |
| ysandeep | shardy, We do want to use ceilometer , however it should happen over ADMIN network which is separate and not external network. We only want to disable SNMP over external network that is visible to outside of our cloud because that leads to security vulnerabilities. | 12:04 |
| *** openstackgerrit has quit IRC | 12:04 | |
| shardy | ysandeep: Ok sounds like a possible configuration bug, but FYI we've been trying to discourage use of telemetry services on the undercloud so we can reduce the footprint and also the undercloud isn't highly available so you can't rely on it to always collect metrics etc | 12:04 |
| shardy | ysandeep: understood - could you please raise a bug? | 12:05 |
| shardy | ysandeep: if you believe it is a vulnerability please mark it as such | 12:05 |
| *** quiquell_ has quit IRC | 12:06 | |
| ysandeep | shardy, bug for the master branch, do you think by chance it can backported to newton | 12:06 |
| shardy | ysandeep: potentially, lets discuss the fix for master first | 12:06 |
| shardy | ysandeep: in general bugfixes can be backported | 12:07 |
| ysandeep | shardy, thank you ,Surely i will raise a bug , In the mean can you think of any possible workaround | 12:08 |
| *** openstackgerrit has joined #tripleo | 12:08 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-common master: TLS by default for the overcloud https://review.openstack.org/554926 | 12:08 |
| *** janki has joined #tripleo | 12:08 | |
| shardy | ysandeep: the workaround is to figure out the puppet configuration needed to bind on a specific network, then pass it via the hieradata_override interface in the undercloud.conf | 12:09 |
| *** sshnaidm has joined #tripleo | 12:09 | |
| shardy | ysandeep: the final fix will be to apply the same data in the normal undercloud install | 12:09 |
| jistr | social: https://review.openstack.org/#/c/548635/5/docker/services/gnocchi-api.yaml what about the yum module issue with asterisks? | 12:09 |
| shardy | ysandeep: perhaps pradk can help work out that config when he's around | 12:09 |
| *** ooolpbot has joined #tripleo | 12:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757556 | 12:10 |
| *** ooolpbot has quit IRC | 12:10 | |
| openstack | Launchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged] | 12:10 |
| ysandeep | shardy, I will raise a bug in the mean thank you for your time | 12:10 |
| ysandeep | shardy, Its a pleasure to talk with you, I read your blogs :) | 12:11 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-common master: TLS by default for the overcloud https://review.openstack.org/554926 | 12:11 |
| shardy | ysandeep: np, thanks for raising the issue, hopefully we can find a quick fix for you :) | 12:11 |
| *** bfournie has joined #tripleo | 12:11 | |
| *** liverpooler has joined #tripleo | 12:12 | |
| *** dprince has joined #tripleo | 12:16 | |
| *** lucas-hungry is now known as lucasagomes | 12:16 | |
| bogdando | stevebaker: , folks, PTAL https://review.openstack.org/#/c/554539/ | 12:18 |
| bogdando | https://review.openstack.org/#/c/555338/ is needed for heat installer and underclouds | 12:18 |
| *** nyechiel_ has joined #tripleo | 12:19 | |
| *** nyechiel has quit IRC | 12:19 | |
| *** raildo has joined #tripleo | 12:20 | |
| jaosorior | bogdando: need other reviews? I was not available when I asked a bit ago, but now I have some time. | 12:20 |
| *** thrash|g0ne is now known as thrash | 12:20 | |
| bogdando | jaosorior: thanks! will post a few more shortly) | 12:20 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Run playbooks with custom args https://review.openstack.org/553474 | 12:21 |
| *** pdeore__ has joined #tripleo | 12:21 | |
| *** suuuper has quit IRC | 12:22 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack/python-tripleoclient master: Always hard link http boot contents for Ironic https://review.openstack.org/556516 | 12:23 |
| bogdando | jaosorior: https://review.openstack.org/#/q/topic:containers-default+(status:open+OR+status:merged) please | 12:24 |
| bogdando | folks, I wonder why this fails http://logs.openstack.org/26/553026/7/check/tripleo-ci-centos-7-undercloud-containers/947b156/logs/undercloud/home/zuul/undercloud_install.log.txt.gz ? | 12:26 |
| *** atoth has joined #tripleo | 12:26 | |
| bogdando | the range contstraing seems valid, isn't it?.. | 12:26 |
| bogdando | constraint* | 12:26 |
| bogdando | https://review.openstack.org/#/c/553026/7/extraconfig/post_deploy/undercloud_post.yaml | 12:27 |
| jaosorior | bogdando: I guess it's because the constraint is written wrong | 12:28 |
| bogdando | hm | 12:28 |
| bogdando | https://docs.openstack.org/heat/latest/template_guide/hot_spec.html#range ... but... | 12:28 |
| *** tzumainn has joined #tripleo | 12:29 | |
| jaosorior | bogdando: commented on your patch | 12:29 |
| bogdando | oh!.. thanks jaosorior :) | 12:30 |
| jaosorior | no biggie | 12:30 |
| Tengu | bogdando: thanks for your comment on my ceph/rgw/swift/docker issue :). Might be good to push some node in some doc though? | 12:30 |
| Tengu | *note, not node, sorry. | 12:31 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates master: undercloud: configure mtu when creating neutron network https://review.openstack.org/553026 | 12:31 |
| bogdando | Tengu: may be it is, just not sure if it would be Heat docs? :) | 12:32 |
| *** nyechiel_ has quit IRC | 12:32 | |
| bogdando | should* | 12:32 |
| Tengu | bogdando: no idea - my overview is limited for that field now. I'll know far more once I start working on the deploy framework. In May :) | 12:33 |
| *** rlandy has joined #tripleo | 12:36 | |
| gfidente | slagle I have a run from https://review.openstack.org/#/c/546966/ in CI | 12:40 |
| gfidente | slagle should be http://logs.openstack.org/66/546966/9/check/tripleo-ci-centos-7-scenario004-multinode-oooq-container/c5d4dc8/ | 12:41 |
| gfidente | slagle but I can't find the ansible-playbook run logs? | 12:41 |
| gfidente | slagle or maybe I am not sure where to look? | 12:41 |
| slagle | gfidente: checking | 12:43 |
| jistr | matbu, chem: can you please check https://review.openstack.org/#/c/555250/ ? it blocks upgrades to master | 12:45 |
| slagle | gfidente: it didn't use config-download. will need to look into the CI patch | 12:45 |
| matbu | jistr: yep | 12:46 |
| *** kmy has quit IRC | 12:46 | |
| slagle | gfidente: scenario004 is featureset019, which we didn't update | 12:48 |
| *** derekh_afk is now known as derekh | 12:48 | |
| fultonj | Tengu: just saw your update. glad to hear it's going well so far. if you need to update pool PG numbers it's not too hard to do that manually post deploy | 12:49 |
| *** morazi has joined #tripleo | 12:49 | |
| slagle | gfidente: scenario004 is multiple featuresets? | 12:49 |
| slagle | gfidente: it's also featureset008 and 15 and 19 | 12:50 |
| slagle | isn't it obvious!? | 12:50 |
| *** dpawar has quit IRC | 12:50 | |
| Tengu | fultonj: had another issue with the order of environments - but eventually it should succeed. | 12:51 |
| ratailor | Has anybody successfully deployed 1ctlr_1comp_1ceph with fs024 for rhos-12 branch ? | 12:51 |
| ratailor | I am getting timeout on introspection for controller node. | 12:52 |
| ratailor | I have also tried the same with stable/pike branch, but got the same result. | 12:52 |
| ratailor | jaosorior, jistr, Tengu ^^ Do you have any suggestions ? | 12:54 |
| *** moshele has quit IRC | 12:54 | |
| openstackgerrit | Merged openstack/tripleo-common master: Add healthcheck script for libvirtd https://review.openstack.org/550492 | 12:56 |
| Tengu | ratailor: hmm, using pike on my side, and introspection is OK - maybe some network issue? | 12:56 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-common master: TLS by default for the overcloud https://review.openstack.org/554926 | 12:56 |
| Tengu | ratailor: but using the good old "openstack overcloud ..." commands for now. | 12:57 |
| *** quiquell is now known as quiquell|lunch | 12:58 | |
| *** dpawar has joined #tripleo | 12:59 | |
| *** ansmith has joined #tripleo | 12:59 | |
| *** mcornea has joined #tripleo | 13:00 | |
| *** trown|outtypewww is now known as trown | 13:00 | |
| social | jistr: yum module * issue will be adressed in another commit | 13:01 |
| chandankumar | weshay: arxcruz|rover by the way tempest is running on undercloud http://logs.openstack.org/22/554522/4/check/tripleo-ci-centos-7-undercloud-oooq/fcabece/logs/undercloud/home/zuul/tempest_output.log.txt.gz | 13:01 |
| social | jistr: there are quite many other services with that so I'll do it together | 13:01 |
| chandankumar | I need to find a set of actual tests to run | 13:01 |
| jistr | social: ok makes sense | 13:01 |
| *** anilvenkata has quit IRC | 13:01 | |
| *** toure|biab is now known as toure | 13:01 | |
| ratailor | Tengu, I tried this on stable/pike and on rdo-cloud as well, but got the same issue. | 13:02 |
| *** aufi_ has joined #tripleo | 13:03 | |
| ratailor | Tengu, the ceph-storage node is getting local_gb=7 instead of default 50. | 13:03 |
| arxcruz|rover | chandankumar: http://www.quickmeme.com/meme/3vcq2n | 13:03 |
| *** panda|off is now known as panda | 13:03 | |
| *** jcoufal has joined #tripleo | 13:03 | |
| ratailor | Tengu, even using infrared, the introspection failed for controller node after retrying 3 times. | 13:04 |
| Tengu | ratailor: on baremeatal, or virtual machines? | 13:05 |
| jistr | thanks matbu for the review | 13:05 |
| *** aufi has quit IRC | 13:05 | |
| *** jmelvin has joined #tripleo | 13:05 | |
| *** nyechiel_ has joined #tripleo | 13:07 | |
| *** jmelvin_ has joined #tripleo | 13:07 | |
| ratailor | Tengu, on a single baremetal using quickstart. | 13:08 |
| Tengu | ratailor: hm, I don't use quickstart. but if you run a single baremetal, you'll provision virtual machines for each role right? | 13:09 |
| ratailor | Tengu, yes | 13:09 |
| *** ooolpbot has joined #tripleo | 13:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 13:10 |
| ratailor | Tengu, I have also done the same with infrared, if you have access to that, I can provide jenkins job url. | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757556 | 13:10 |
| *** ooolpbot has quit IRC | 13:10 | |
| openstack | Launchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged] | 13:10 |
| *** jmelvin has quit IRC | 13:10 | |
| Tengu | ratailor: I know ironic will need to get the MAC address for virtual machin introspection - especially if using virtualBMC. but as I'm not using it, I can't say. and I don't have accesses yet. | 13:10 |
| ratailor | Tengu, anyways np, Thanks! | 13:11 |
| Tengu | ^^' | 13:12 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/queens: Consume ceph-container project's new style of tags https://review.openstack.org/556446 | 13:14 |
| *** Goneri has joined #tripleo | 13:17 | |
| *** quiquell|lunch is now known as quiquell | 13:21 | |
| openstackgerrit | John Trowbridge proposed openstack/tripleo-quickstart master: [DNM] Testing keystone-only featureset000 https://review.openstack.org/556630 | 13:23 |
| *** dpawar has quit IRC | 13:24 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Default NodeTLSData to always attempt setting TLS cert https://review.openstack.org/556817 | 13:24 |
| openstackgerrit | Merged openstack/tripleo-docs master: Refactor backup and restore docs https://review.openstack.org/544975 | 13:29 |
| *** alee_ has quit IRC | 13:29 | |
| *** maufart__ has joined #tripleo | 13:30 | |
| *** moshele has joined #tripleo | 13:33 | |
| *** aufi_ has quit IRC | 13:33 | |
| *** jaosorior has quit IRC | 13:33 | |
| *** aputtur_ has joined #tripleo | 13:35 | |
| *** jmelvin_ is now known as jmelvin_bomgr | 13:36 | |
| *** moshele has quit IRC | 13:37 | |
| *** moshele has joined #tripleo | 13:39 | |
| chem | jfrancoa: rlandy has filed a lp for the current pike error https://bugs.launchpad.net/tripleo/+bug/1759060 | 13:41 |
| openstack | Launchpad bug 1759060 in tripleo "gate-tripleo-ci-centos-7-containers-multinode-upgrades-pike-nv fails - No such file or directory /home/jenkins/overcloud-prep-containers.sh" [High,Triaged] | 13:41 |
| chem | jfrancoa: let's have a debug session in 15 min (need to finish some downstream work) | 13:42 |
| chem | jfrancoa: the error seems rather obvious | 13:43 |
| jfrancoa | chem: I saw that this morning, yes. But once we solve that, we need to deal then with this other one: https://bugs.launchpad.net/tripleo/+bug/1757969 | 13:43 |
| openstack | Launchpad bug 1757969 in tripleo "[CI ocata->pike] Undefined variable 'deploy_config_name'" [High,New] - Assigned to Jose Luis Franco (jfrancoa) | 13:43 |
| jfrancoa | chem: ok, cool. let me know when you'll have some time | 13:43 |
| shardy | jtomasek: Hi, wanted to discuss your comments re messages/status - would it be enough if we always write a top-level latest.yaml, so you just read that and get the last message? | 13:44 |
| shardy | jtomasek: then for the IN_PROGRESS case we just send another message at the start of the workflow, which looks like: http://paste.openstack.org/show/715403/ | 13:45 |
| jtomasek | shardy: Hi, thing is that the same latest.yaml would have to be written by all affected workflows, not one latest.yaml per different workflow. Because in that case the client has to know which workflows affect deployment state, fetch their latest.yaml and then identify which of them is latest of them to get current status | 13:46 |
| shardy | we'd probably also need a periodic task to catch the case where the mistral engine restarts killing the executions | 13:46 |
| *** dpawar has joined #tripleo | 13:47 | |
| shardy | jtomasek: yes that is what I'm suggesting - also write a top-level "latest.yaml" or status.yaml which is just the most recently written message | 13:47 |
| shardy | regardless of workflow | 13:47 |
| openstackgerrit | James Slagle proposed openstack/tripleo-quickstart master: WIP: config-download for scenario's 001 and 004 https://review.openstack.org/548306 | 13:47 |
| shardy | for a single plan the deploy workflow is pretty much a sequence so that should work, right? | 13:47 |
| shardy | I don't think we make much use of branching etc | 13:47 |
| jtomasek | shardy: yes, that's very much what I am asking, why tie it to the message then? Why not make it a separate action? | 13:48 |
| shardy | jtomasek: then you just read planname-messages/latest.yaml and you get a status to display | 13:48 |
| shardy | jtomasek: because regardless of some new status abstraction, we need to send the messages | 13:48 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates master: WIP: Migrate Ceph deployment to external_deploy_task https://review.openstack.org/546966 | 13:48 |
| shardy | jtomasek: I'm trying to avoid writing the same data twice, except in the message abstraction provided by dougal | 13:49 |
| shardy | and also keep a consistent format | 13:49 |
| shardy | jtomasek: if you can provide an example of what would be different about a non-message "status" data structure that would be helpful | 13:49 |
| jtomasek | shardy: we would not need to send the starting messages with a separate action | 13:49 |
| shardy | AFAICS all we need is the state, the reason and where the state happened, e.g which workflow? | 13:49 |
| *** moshele has quit IRC | 13:49 | |
| shardy | jtomasek: we'd need to send an equivalent status to swift, so I'm not sure I get the difference | 13:50 |
| shardy | I suppose there's less change the client gets an unexpected message (which appears to break tripleoclient, but IMO that's a bug) | 13:50 |
| shardy | jtomasek: I'm just trying to understand the difference between sending a starting message and what you're proposing | 13:51 |
| shardy | to me it seems like adding an IN_PROGRESS message is less work than adding an IN_PROGRESS status, and also COMPLETE/FAILED status for every possible path of all workflows | 13:52 |
| shardy | because we already send messages for everything except the IN_PROGRESS part? | 13:52 |
| shardy | (which is probably a historical mistake I think?) | 13:52 |
| shardy | mandre: ^^ interested in your thoughts | 13:52 |
| jtomasek | shardy: right, I think all we need for start is a string describing the status. but this includes 'Udeploying' too. so if we have a global latest.yaml which is basically a swift object which tracks the status. The difference is IMHO that we won't have one file per message/workflow type but one file per 'deployment status' so that makes it disconnected from the messages | 13:52 |
| shardy | jtomasek: yes, so we'd put it outside of the directories per workflow in the *-messages container | 13:53 |
| shardy | jtomasek: it's not disconnected from the messages, it's just the most recent | 13:53 |
| mandre | shardy: reading backlog | 13:53 |
| rasca | EmilienM, mwhahaha, hey guys, is all setup for this one https://review.openstack.org/#/c/555830/ to merge? What else need to be done? | 13:54 |
| shardy | jtomasek: for example, this is what I did to add an in-progress message to the deploy workflow http://paste.openstack.org/show/715404/ | 13:54 |
| EmilienM | rasca: waiting | 13:54 |
| shardy | jtomasek: I'm worried adding a new "status" action for this will be more work, particularly since dougal already spent time getting this message abstraction to work | 13:55 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-quickstart master: Update overcloud-prep-containers script name. https://review.openstack.org/556860 | 13:55 |
| *** cdearborn has joined #tripleo | 13:55 | |
| jtomasek | shardy: I see, right. That solution makes it slightly harder for client, because client gets the information about latest workflow and the status, but then it needs to know how to behave based on the type of workflow it is. e.g.: if workflow is 'undeploy_plan', status: SUCCESS, it needs to identify that the deployment is not deployed any more | 13:55 |
| shardy | jtomasek: Ok, how would having a new action help with that? Could we instead have a new "get plan status" action with whatever logic you need in the client? | 13:56 |
| mwhahaha | reminder meeting in 3 mins, agenda is here: https://etherpad.openstack.org/p/tripleo-meeting-items | 13:57 |
| jfrancoa | chem: done with the first O->P issue https://review.openstack.org/#/c/556860/ :-D, whenever you want we can go with the second one | 13:57 |
| shardy | jtomasek: I guess my question is, won't you require that logic anyway, to decide if a plan still exists or whatever? | 13:57 |
| jtomasek | get plan status action would be helpful. it would basically extract the logic from actions triggered from workflows to single actions | 13:58 |
| jtomasek | *action | 13:58 |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates stable/queens: Honor {{role.name}}ExtraConfig with deprecated params https://review.openstack.org/556862 | 13:59 |
| jtomasek | dedicated action triggered from undeploy_plan would be: status: UNDEPLOYED. So client would just have to know how to react to different states | 13:59 |
| jtomasek | shardy: but you're right that it is probably a detail, since with your solution it is a combination if workflow type+status | 13:59 |
| mwhahaha | #startmeeting tripleo | 14:00 |
| mwhahaha | #topic agenda | 14:00 |
| mwhahaha | * Review past action items | 14:00 |
| mwhahaha | * One off agenda items | 14:00 |
| mwhahaha | * Squad status | 14:00 |
| openstack | Meeting started Tue Mar 27 14:00:14 2018 UTC and is due to finish in 60 minutes. The chair is mwhahaha. Information about MeetBot at http://wiki.debian.org/MeetBot. | 14:00 |
| mwhahaha | * Bugs & Blueprints | 14:00 |
| mwhahaha | * Projects releases or stable backports | 14:00 |
| mwhahaha | * Specs | 14:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 14:00 |
| mwhahaha | * open discussion | 14:00 |
| shardy | jtomasek: Ok, perhaps we could put that logic into a get_plan_status action then as it reduces the impact on the existing workflows, and it can be added after we get the basic message data working as needed | 14:00 |
| mwhahaha | Anyone can use the #link, #action and #info commands, not just the moderatorǃ | 14:00 |
| mwhahaha | Hi everyone! who is around today? | 14:00 |
| *** openstack changes topic to " (Meeting topic: tripleo)" | 14:00 | |
| openstack | The meeting name has been set to 'tripleo' | 14:00 |
| *** openstack changes topic to "agenda (Meeting topic: tripleo)" | 14:00 | |
| jtomasek | o/ | 14:00 |
| EmilienM | hello | 14:00 |
| myoung | o/ | 14:00 |
| beagles | o/ | 14:00 |
| jpich | o/ | 14:00 |
| fultonj | o/ | 14:00 |
| matbu | o/ | 14:00 |
| shardy | o/ | 14:00 |
| jfrancoa | o/ | 14:00 |
| thrash | o/ | 14:00 |
| chem | o/ | 14:00 |
| cdearborn | o/ | 14:00 |
| mandre | o/ | 14:00 |
| rfolco|ruck | o/ | 14:01 |
| owalsh | o/ | 14:01 |
| marios | o/ | 14:01 |
| panda | o/ | 14:01 |
| dpeacock | o/ | 14:01 |
| mwhahaha | alright lets go | 14:02 |
| mwhahaha | #topic review past action items | 14:02 |
| mwhahaha | mwhahaha to send an ML note about making sure to review current status of assigned bugs, also review bugs by squad tags - DONE | 14:02 |
| mwhahaha | #link http://lists.openstack.org/pipermail/openstack-dev/2018-March/128557.html | 14:02 |
| *** openstack changes topic to "review past action items (Meeting topic: tripleo)" | 14:02 | |
| mwhahaha | didn't see any replies, so i hope everyone is looking at bugs | 14:02 |
| mwhahaha | that's it on the action items. | 14:02 |
| *** alee_ has joined #tripleo | 14:03 | |
| mwhahaha | #topic one off agenda items | 14:03 |
| mwhahaha | #link https://etherpad.openstack.org/p/tripleo-meeting-items | 14:03 |
| *** openstack changes topic to "one off agenda items (Meeting topic: tripleo)" | 14:03 | |
| mwhahaha | the agenda is empty, anyone have anything? | 14:03 |
| *** jaosorior has joined #tripleo | 14:03 | |
| holser__ | o/ | 14:03 |
| mwhahaha | sounds like not so much. moving on then | 14:04 |
| mwhahaha | #topic Squad status | 14:04 |
| mwhahaha | ci | 14:04 |
| mwhahaha | #link https://etherpad.openstack.org/p/tripleo-ci-squad-meeting | 14:04 |
| mwhahaha | upgrade | 14:04 |
| mwhahaha | #link https://etherpad.openstack.org/p/tripleo-upgrade-squad-status | 14:04 |
| mwhahaha | containers | 14:04 |
| mwhahaha | #link https://etherpad.openstack.org/p/tripleo-containers-squad-status | 14:04 |
| *** openstack changes topic to "Squad status (Meeting topic: tripleo)" | 14:04 | |
| mwhahaha | config-download | 14:04 |
| mwhahaha | #link https://etherpad.openstack.org/p/tripleo-config-download-squad-status | 14:04 |
| mwhahaha | integration | 14:04 |
| mwhahaha | #link https://etherpad.openstack.org/p/tripleo-integration-squad-status | 14:04 |
| mwhahaha | ui/cli | 14:04 |
| mwhahaha | #link https://etherpad.openstack.org/p/tripleo-ui-cli-squad-status | 14:04 |
| mwhahaha | validations | 14:04 |
| mwhahaha | #link https://etherpad.openstack.org/p/tripleo-validations-squad-status | 14:04 |
| mwhahaha | networking | 14:04 |
| mwhahaha | #link https://etherpad.openstack.org/p/tripleo-networking-squad-status | 14:04 |
| mwhahaha | workflows | 14:04 |
| mwhahaha | #link https://etherpad.openstack.org/p/tripleo-workflows-squad-status | 14:04 |
| mwhahaha | security | 14:04 |
| mwhahaha | #link https://etherpad.openstack.org/p/tripleo-security-squad | 14:04 |
| jrist | o/ | 14:04 |
| jistr | o/ | 14:05 |
| *** ramishra has quit IRC | 14:05 | |
| bogdando | o/ | 14:05 |
| mwhahaha | any status issues or things people would like to raise for visibility? | 14:05 |
| jrist | mwhahaha: will update ui/cli - was out yesterday | 14:05 |
| jrist | mwhahaha: there was talk about migrating tripleo ui to story board | 14:06 |
| mwhahaha | arxcruz|rover, rfolco|ruck, weshay : where are we at with promotions? | 14:06 |
| jrist | but I haven't seen an update of that | 14:06 |
| jrist | any word? | 14:06 |
| mwhahaha | jrist: last i heard it was stuck on the import, they are working on it | 14:06 |
| jrist | k | 14:06 |
| jrist | thanks | 14:06 |
| mwhahaha | should be done in the next few days is what I read | 14:06 |
| chandankumar | \o/ | 14:06 |
| radeks | o/ | 14:06 |
| mwhahaha | not so sure how i feel about it honestly | 14:06 |
| * mwhahaha is not looking forward to having to track bug numbers across 2 systems | 14:06 | |
| jrist | me either :) wasn't my decision | 14:06 |
| jrist | (3) | 14:07 |
| EmilienM | so storyboard migration is making good progress | 14:07 |
| mwhahaha | i try and imagine that bz doesn't exist | 14:07 |
| mwhahaha | i fail daily | 14:07 |
| EmilienM | they had issue with the mysql database and some bugs with weird caracters | 14:07 |
| mwhahaha | pffft who needs utf8 | 14:07 |
| EmilienM | FYI they are importing tripleo bugs with "ui" tag | 14:07 |
| EmilienM | so please continue to use this tag in launchpad heavily or your bugs won't be imported | 14:07 |
| beagles | the "side car" containers have been going well, we are working on getting some more extensive ad-hoc testing using a small group of neutron devs/qe experts. We currently have one outlier (radvd) that needs a patch to the binary to support containerization. In general things are looking pretty good | 14:08 |
| jtomasek | EmilienM: yes, we use that consistently | 14:08 |
| EmilienM | jtomasek: excellent | 14:08 |
| beagles | side-car containers for neutron agents that is | 14:08 |
| jpich | thanks for the update. And yeah a bunch of bugs are tagged i18n also :) | 14:08 |
| weshay | rfolco|ruck, ? | 14:08 |
| EmilienM | jpich: we can pass the information on #storyboard and import these ones | 14:09 |
| jrist | we're a good test case, right jpich ? | 14:09 |
| weshay | mwhahaha, last night.. it was scen007 and the full tempest run failing | 14:09 |
| jrist | or edge case as it were | 14:09 |
| *** anilvenkata has joined #tripleo | 14:09 | |
| weshay | mwhahaha, latest run results were worse | 14:09 |
| mwhahaha | weshay: sc007 still failing? | 14:09 |
| *** abishop has quit IRC | 14:09 | |
| mwhahaha | i need to check on my fs020 reproducer and see what that lookedl ike | 14:09 |
| *** ysandeep has quit IRC | 14:09 | |
| jpich | EmilienM: Ah, sorry - I meant they're tagged both ui and i18n, and that probably explains the issue with importing unusual characters | 14:09 |
| weshay | mwhahaha, I believe apevec and rdo need to update to ovs 2.9.0.3 for that | 14:10 |
| mwhahaha | weshay: k | 14:10 |
| *** ooolpbot has joined #tripleo | 14:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757556 | 14:10 |
| openstack | Launchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged] | 14:10 |
| *** ooolpbot has quit IRC | 14:10 | |
| rfolco|ruck | mwhahaha, fs020 is being worked with larger flavor and more workers... timing out +5h run | 14:10 |
| mwhahaha | beagles: did you figure out a solution for side card container management after the fact? ie cleanup | 14:11 |
| mwhahaha | rfolco|ruck: yea we shouldn't have to increase the flavor, would like to understand where the perf regression is so we can raise the issue with the correct team | 14:11 |
| beagles | mwhahaha, yup "--rm=true" to the docker run command causes the container to automagically clean up when it is stopped | 14:11 |
| mwhahaha | beagles: oh how handy | 14:11 |
| beagles | mwhahaha, indeed! | 14:11 |
| mwhahaha | anyway sounds like stuff's getting worked on. any other items? | 14:12 |
| EmilienM | mwhahaha: we've got a first successful OVB job running with a containerized undercloud. bogdando and I are working on cleaning up things but I think we can make the switch right after we have a promotion | 14:13 |
| *** kopecmartin has quit IRC | 14:13 | |
| mwhahaha | EmilienM: that's some lofty goals :o | 14:13 |
| mwhahaha | EmilienM: and by switch you mean one featureset or all | 14:14 |
| * mwhahaha would prefer not all initially | 14:14 | |
| bogdando | just fs001 for now | 14:14 |
| mwhahaha | k | 14:14 |
| EmilienM | mwhahaha: yeah just fs001 | 14:14 |
| EmilienM | (and I would like to switch fs010 as well) | 14:15 |
| weshay | mwhahaha, made the promotion results clear in https://review.rdoproject.org/etherpad/p/ruckrover-sprint11 | 14:15 |
| mwhahaha | weshay: ok i'll review that later | 14:15 |
| *** mcornea has quit IRC | 14:15 | |
| *** mcornea has joined #tripleo | 14:15 | |
| mwhahaha | alright moving on to bugs | 14:15 |
| mwhahaha | #topic bugs & blueprints | 14:15 |
| mwhahaha | #link https://launchpad.net/tripleo/+milestone/rocky-1 | 14:15 |
| mwhahaha | For Rocky we currently have 66 (+1) blueprints and about 635 (+0) open bugs. 628 rocky-1, 2 rocky-2, 5 rocky-3 | 14:15 |
| *** openstack changes topic to "bugs & blueprints (Meeting topic: tripleo)" | 14:15 | |
| mwhahaha | Reminder about the deadline for blueprints for Rocky is April 3, 2018. | 14:16 |
| mwhahaha | #link http://lists.openstack.org/pipermail/openstack-dev/2018-March/128217.html | 14:16 |
| mwhahaha | that's next week | 14:16 |
| mwhahaha | so if you would like your blueprint in rocky, update it | 14:16 |
| mwhahaha | or i'm moving it next week | 14:16 |
| mwhahaha | any pressing bugs (besides ssh timeouts) | 14:17 |
| weshay | openvswitch-2.9.0-3.el7.x86_64.rpm2018-03-27 13:416.1M is in dlrn-deps now | 14:17 |
| mwhahaha | weshay: so new promotion runs will get that? does that also include queens? | 14:18 |
| weshay | mwhahaha, nope.. it missed it I'm being told atm | 14:20 |
| mwhahaha | k | 14:20 |
| mwhahaha | moving on | 14:20 |
| mwhahaha | #topic projects releases or stable backports | 14:20 |
| *** openstack changes topic to "projects releases or stable backports (Meeting topic: tripleo)" | 14:20 | |
| mwhahaha | I think we're releasing queens this week. saw a patch for it | 14:20 |
| mwhahaha | EmilienM: is that right? | 14:20 |
| EmilienM | we released newton ocata pike queens yesterday | 14:21 |
| mwhahaha | k | 14:21 |
| EmilienM | I'm now working on the rdo part | 14:21 |
| jrist | newton EOL soon? | 14:21 |
| mwhahaha | yea i suppose we should EOL it | 14:21 |
| EmilienM | well we still have backports | 14:22 |
| mwhahaha | we said we'd do it in jan or feb | 14:22 |
| mwhahaha | why don't we EOl it at Rocky M1 | 14:22 |
| mwhahaha | and just put a fork in it | 14:22 |
| openstackgerrit | Marius Cornea proposed openstack/tripleo-upgrade stable/queens: Support custom stack names during FFU https://review.openstack.org/556574 | 14:22 |
| EmilienM | tbh, I don't see much reason to EOL now, the cost of maintenance is pretty low and we still have backports | 14:22 |
| jrist | fine with me, just not typical | 14:22 |
| mwhahaha | ok i guess | 14:23 |
| mwhahaha | we'll revisit this one later | 14:23 |
| EmilienM | if we eol we'll force people to do downstream backports while they don't need too now | 14:23 |
| mwhahaha | well they still have to | 14:23 |
| mwhahaha | cause imports are manualish | 14:23 |
| mwhahaha | anyway we can sort that out later | 14:24 |
| mwhahaha | let's finish this meeting so we can get back to work on more pressing things | 14:24 |
| mwhahaha | #topic specs | 14:24 |
| mwhahaha | #link https://review.openstack.org/#/q/project:openstack/tripleo-specs+status:open | 14:24 |
| *** openstack changes topic to "specs (Meeting topic: tripleo)" | 14:24 | |
| mwhahaha | any specs that need attention? | 14:24 |
| mwhahaha | i believe we've merged a few in the last few weeks so i'm not sure much is out there | 14:24 |
| mwhahaha | dsneddon: can you please update https://review.openstack.org/#/c/451584/ to Rocky | 14:25 |
| mwhahaha | that's it on the spec bits | 14:25 |
| mwhahaha | #topic open discussion | 14:25 |
| *** openstack changes topic to "open discussion (Meeting topic: tripleo)" | 14:26 | |
| mwhahaha | anything else? | 14:26 |
| *** artom has quit IRC | 14:27 | |
| mwhahaha | sounds like nope | 14:28 |
| mwhahaha | thanks everyone | 14:28 |
| mwhahaha | #endmeeting | 14:28 |
| *** openstack changes topic to "Welcome to Rocky. CI status - Promotions: RED; check/gate: GREEN; RDO CI jobs: Questionable | http://tripleo.org/ | https://docs.openstack.org/tripleo-docs/latest/" | 14:28 | |
| openstack | Meeting ended Tue Mar 27 14:28:11 2018 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 14:28 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/tripleo/2018/tripleo.2018-03-27-14.00.html | 14:28 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/tripleo/2018/tripleo.2018-03-27-14.00.txt | 14:28 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/tripleo/2018/tripleo.2018-03-27-14.00.log.html | 14:28 |
| openstackgerrit | Jose Luis Franco proposed openstack-infra/tripleo-ci master: Run playbooks with custom args https://review.openstack.org/553474 | 14:28 |
| openstackgerrit | Jose Luis Franco proposed openstack-infra/tripleo-ci master: Use tripleo-upgrade role in undercloud upgrades job. https://review.openstack.org/548974 | 14:28 |
| jrist | thanks mwhahaha | 14:28 |
| fultonj | mwhahaha++ | 14:28 |
| *** aputtur_ has quit IRC | 14:29 | |
| *** ykarel is now known as ykarel|away | 14:29 | |
| *** florianf has joined #tripleo | 14:31 | |
| jaosorior | EmilienM, mwhahaha: Hey, can you help me out with some packaging issue I have? This patch to tripleo-common introduces a new script https://review.openstack.org/#/c/554926/ it fails because the RPM doesn't have that script available. This patch to tripelo-common-distgit introduces the script to the spec file, but it fails cause the script isn't available... how should I set up the dependencies there? | 14:31 |
| fultonj | gfidente: i like in ps9 of https://review.openstack.org/#/c/546966 how you make the group_vars dir | 14:32 |
| fultonj | gfidente: inventory looks good but i see heat-admin in place of tripleo-admin in my env | 14:32 |
| *** abishop has joined #tripleo | 14:33 | |
| mwhahaha | jaosorior: i have no idea, ask #rdo folks? | 14:33 |
| EmilienM | jaosorior: oh, we had this problem before | 14:34 |
| EmilienM | jaosorior: let me show you an example, a sec | 14:34 |
| *** gkadam_ has joined #tripleo | 14:36 | |
| EmilienM | jaosorior: in THT, we solved like this https://review.rdoproject.org/r/#/c/11170/2/openstack-tripleo-heat-templates.spec | 14:37 |
| *** Goneri has quit IRC | 14:38 | |
| EmilienM | jaosorior: you can add a if I think | 14:38 |
| jaosorior | oooh I see | 14:38 |
| jaosorior | EmilienM: thanks! I'll give it a try | 14:38 |
| *** gkadam has quit IRC | 14:38 | |
| *** ratailor has quit IRC | 14:39 | |
| openstackgerrit | Merged openstack/tripleo-common master: Add ironic-neutron-agent container https://review.openstack.org/547321 | 14:40 |
| *** aputtur_ has joined #tripleo | 14:44 | |
| *** yamahata has joined #tripleo | 14:46 | |
| *** ykarel|away has quit IRC | 14:46 | |
| fultonj | gfidente: ps9 looked like it made an inventory that should work, but i ran into these issues: http://paste.openstack.org/show/715411 i'll debug and try to help. ping me if you want to see it directly | 14:50 |
| EmilienM | trown: please remove -2 on https://review.rdoproject.org/r/#/c/12774/ | 14:50 |
| *** Goneri has joined #tripleo | 14:51 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack/python-tripleoclient master: Hard link http boot contents for Ironic https://review.openstack.org/556516 | 14:57 |
| EmilienM | trown: thanks | 14:57 |
| trown | EmilienM: np, thanks for fixing it | 14:57 |
| *** pkovar has quit IRC | 14:58 | |
| *** percevalbot` has quit IRC | 14:58 | |
| *** yamahata has quit IRC | 14:59 | |
| bogdando | EmilienM, jaosorior: it still mysteriously fails http://logs.openstack.org/26/553026/8/check/tripleo-ci-centos-7-undercloud-containers/894e20b/logs/undercloud/home/zuul/undercloud_install.log.txt.gz | 15:01 |
| bogdando | it does not like heat ranges | 15:01 |
| bogdando | um "expected a list" ... | 15:02 |
| jaosorior | bogdando: what was the patch? | 15:02 |
| *** hamzy has quit IRC | 15:02 | |
| bogdando | https://review.openstack.org/553026 | 15:02 |
| bogdando | jaosorior: ^^ | 15:02 |
| jaosorior | aaah shit | 15:03 |
| jaosorior | bogdando: so, gerrit transformed my - into a bullet point | 15:03 |
| jaosorior | but yeah, it should be a list | 15:03 |
| myoung | o/ tripleo CI community meeting is happening now if interested: https://bluejeans.com/7071866728 | 15:03 |
| bogdando | woot | 15:03 |
| *** rook is now known as rook-mtg | 15:03 | |
| bogdando | (╯°□°）╯︵ ┻━┻ | 15:04 |
| jaosorior | bogdando: should be - range: {... } | 15:04 |
| jaosorior | so, it was missing the dash | 15:04 |
| jaosorior | I tried pointing it out in my review... but gerrit changed the dash | 15:04 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates master: undercloud: configure mtu when creating neutron network https://review.openstack.org/553026 | 15:05 |
| bogdando | indeed | 15:05 |
| bogdando | EmilienM, jaosorior: updated :) | 15:05 |
| bogdando | thanks! | 15:05 |
| jaosorior | that looks about right | 15:05 |
| EmilienM | jrist: any update on https://etherpad.openstack.org/p/tripleo-ui-cli-squad-status ? or nothing this week? | 15:06 |
| *** hamzy has joined #tripleo | 15:06 | |
| jrist | EmilienM: I was gone yesterday - started an update but need to ping other people | 15:06 |
| jrist | EmilienM: that's part of why I asked about bug migration | 15:06 |
| *** pkovar has joined #tripleo | 15:07 | |
| EmilienM | jrist: no prob, I'll wait a bit for it then | 15:07 |
| jrist | honza, jpich, jtomasek - any specific updates for ui/cli squad status? | 15:08 |
| jpich | not from me | 15:08 |
| mandre | shardy: did you reach a conclusion with jtomasek about what needs to be persisted in swift? what is the issue exactly? from what I understand, the persisted latest.yaml is not enough to give an accurate status of the deployment | 15:08 |
| *** florianf has quit IRC | 15:08 | |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart-extras master: Refactored validate-tempest role for undercloud and containers https://review.openstack.org/551441 | 15:09 |
| *** pcaruana has quit IRC | 15:10 | |
| shardy | mandre: yes I'm proposing we add a new "top level" latest.yaml, or perhaps a "plan_status.yaml" which can be written optionally for workflows which modify the plan state | 15:10 |
| *** ooolpbot has joined #tripleo | 15:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757556 | 15:10 |
| *** ooolpbot has quit IRC | 15:10 | |
| openstack | Launchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged] | 15:10 |
| shardy | mandre: the issue is it's more work for the client to figure out the plan status by looking at all the different workflows | 15:10 |
| shardy | vs having a single yaml file we figure out in mistral and write to swift | 15:10 |
| shardy | I'm working on a follow-up patch which does that, provided it seems like a reasonable solution | 15:11 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Increase time for log collection in OVB https://review.openstack.org/556884 | 15:11 |
| *** marios has quit IRC | 15:11 | |
| sshnaidm | weshay, ^^ | 15:11 |
| mandre | shardy: IIUC top level latest.yaml won't give us the needed information, it will only give us the status for the latest workflow, not necessarily the global status | 15:11 |
| *** agopi|out has joined #tripleo | 15:12 | |
| *** jaosorior has quit IRC | 15:12 | |
| openstackgerrit | yolanda.robla proposed openstack/tripleo-docs master: WIP: Add FFU docs https://review.openstack.org/549892 | 15:12 |
| shardy | mandre: yes that's what I was trying to clarify with jtomasek, exactly what information will be missing? | 15:12 |
| *** marios has joined #tripleo | 15:12 | |
| shardy | if we call it plan_status.yaml we could add some additional data, but if every plan state transition is managed by a workflow, I expected the last persisted message to be close the what is required | 15:13 |
| *** yprokule has quit IRC | 15:13 | |
| shardy | the main gap I can think of is if the mistral engine dies, we need a periodic task to reset the status for any IN_PROGRESS executions and mark the plan deployment as failed | 15:13 |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart master: Ignore empty values for dlrn hashes https://review.openstack.org/554882 | 15:14 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-quickstart-extras master: Add missing dash in repo-setup task. https://review.openstack.org/556886 | 15:14 |
| mandre | for example, tripleo.deployment.v1.deploy_plan has status SUCCESS, but that's really just the beginning of the deployment, the global status should be IN_PROGRESS or similar | 15:15 |
| *** thrash is now known as thrash|biab | 15:15 | |
| shardy | mandre: yes that's why I was proposing to add a new IN_PROGRESS message | 15:15 |
| shardy | mandre: and when the config_download flag is enabled we'd get an accurate status | 15:15 |
| *** ykarel|away has joined #tripleo | 15:15 | |
| mandre | shardy: I think adding a top level latest.yaml will be useful in any case but it won't be sufficient | 15:15 |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart master: Run tempest on fs027 using containerized tempest https://review.openstack.org/551442 | 15:16 |
| *** pliu_ has quit IRC | 15:16 | |
| shardy | mandre: I agree the existing deploy plan workflow is weird in that it declares COMPLETE when nothing is really complete, but https://review.openstack.org/#/c/554224/ is the workflow I'm focussing on which doesn't do that anymore | 15:17 |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart master: Run tempest on undercloud on fs003 https://review.openstack.org/554522 | 15:17 |
| chandankumar | arxcruz|rover: testing venv in above patch | 15:17 |
| chandankumar | ^^ | 15:17 |
| openstackgerrit | Jaganathan Palanisamy proposed openstack/tripleo-common master: Fails to get DPDK feature status. https://review.openstack.org/556887 | 15:18 |
| *** percevalbot has joined #tripleo | 15:19 | |
| jrist | EmilienM: just the one note that I know of | 15:19 |
| mandre | shardy: alright, let's suppose the latest workflow accurately report the status of the deployment... then if we send a message when the workflow start with IN_PROGRESS then probably the global latest.yaml is enough | 15:20 |
| *** pkovar1 has joined #tripleo | 15:20 | |
| *** pkovar has quit IRC | 15:20 | |
| shardy | mandre: Ok cool, I'll push a patch which does that then perhaps you can jtomasek can review and we'll iterate as needed | 15:21 |
| shardy | mandre: thanks for the feedback :) | 15:21 |
| * jtomasek reads back | 15:21 | |
| *** colonwq has quit IRC | 15:21 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates master: Implement MasqueradeNetworks services https://review.openstack.org/553427 | 15:22 |
| mandre | shardy: let's just hope we won't find any other weird workflows | 15:22 |
| *** maufart__ has quit IRC | 15:22 | |
| shardy | mandre: yeah - I think the current plan deploy workflow is kind of an anti-pattern if we want to avoid polling, so I guess we'll have to fix any we find anyway | 15:22 |
| mandre | sounds like a plan | 15:23 |
| shardy | I'm also not sure if we need the "all" dir in the patch from d0ugal but I'll leave his patch alone for now and post a followup | 15:24 |
| jtomasek | shardy: sounds good | 15:24 |
| *** almondjoy has joined #tripleo | 15:24 | |
| mandre | shardy: yeah, probably we don't need the "all" directory | 15:25 |
| weshay | sshnaidm++ | 15:26 |
| *** artom has joined #tripleo | 15:26 | |
| weshay | mwhahaha, https://review.openstack.org/#/c/556884/1 | 15:27 |
| *** bogdando has quit IRC | 15:28 | |
| *** leanderthal has quit IRC | 15:30 | |
| *** marios has quit IRC | 15:32 | |
| *** moshele has joined #tripleo | 15:32 | |
| *** marios has joined #tripleo | 15:33 | |
| *** rook-mtg is now known as rook | 15:33 | |
| *** colonwq has joined #tripleo | 15:35 | |
| derekh | can somebody take a look at this to see if installing the zuul cloned dib and tirpleo-common makes sense here ? https://review.openstack.org/#/c/555737/ some context from earlier http://eavesdrop.openstack.org/irclogs/%23tripleo/%23tripleo.2018-03-27.log.html#t2018-03-27T09:18:31 | 15:35 |
| *** moshele has quit IRC | 15:35 | |
| *** agurenko has quit IRC | 15:36 | |
| *** ebarrera has quit IRC | 15:38 | |
| *** thrash|biab is now known as thrash | 15:45 | |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates master: Designate Integration https://review.openstack.org/518998 | 15:45 |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates master: Containerized Designate https://review.openstack.org/535489 | 15:45 |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates master: Add environment to enable Designate https://review.openstack.org/555006 | 15:45 |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates master: Deploy Designate in scenario003 https://review.openstack.org/555007 | 15:45 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-quickstart master: Fix Fast Forward featureset documentation. https://review.openstack.org/556801 | 15:48 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-quickstart master: Add missing Featureset037 documentation. https://review.openstack.org/556814 | 15:48 |
| bnemec | So for the past couple of days, all of my undercloud installs are failing on http://paste.openstack.org/show/715422/ | 15:50 |
| bnemec | Is anyone else seeing this? | 15:50 |
| *** chlong has joined #tripleo | 15:51 | |
| hjensas | bnemec: using master repos? | 15:52 |
| bnemec | hjensas: Yeah | 15:52 |
| *** derekh has quit IRC | 15:53 | |
| *** derekh has joined #tripleo | 15:53 | |
| openstackgerrit | Marius Cornea proposed openstack/tripleo-upgrade master: WIP: Add support for per role FFU upgrade https://review.openstack.org/556687 | 15:54 |
| openstackgerrit | Christian Schwede proposed openstack/tripleo-common master: Fix missing permissions on Swift rebalance playbook https://review.openstack.org/556928 | 15:54 |
| hjensas | bnemec: Haven't seen that specific one, but looks similar to other issues we've had with client side changes before? | 15:55 |
| hjensas | bnemec: I'll do a test and see if I see it as well. | 15:56 |
| bnemec | Oh: http://192.168.24.2:13000/ | 15:56 |
| bnemec | That's a bad address. | 15:56 |
| bnemec | Port 13000 requires https. | 15:57 |
| bnemec | (that's from the keystone log, ftr) | 15:57 |
| bnemec | Something is making a bad request. | 15:58 |
| * bnemec wonders how this is working in ci | 15:58 | |
| *** marios has quit IRC | 15:58 | |
| *** fragatina has quit IRC | 15:58 | |
| *** ktibi has quit IRC | 16:03 | |
| bnemec | Ah: generate_service_certificate = False | 16:04 |
| bnemec | ci is overriding the new default | 16:04 |
| bnemec | That's...bad | 16:04 |
| *** trozet has joined #tripleo | 16:04 | |
| openstackgerrit | Sai Ram Peesapati proposed openstack/tripleo-heat-templates stable/ocata: Enable Neutron LBaaS Integration https://review.openstack.org/556741 | 16:05 |
| *** dpawar has quit IRC | 16:05 | |
| openstackgerrit | Sai Ram Peesapati proposed openstack/tripleo-heat-templates stable/ocata: Enable Neutron LBaaS Integration https://review.openstack.org/556741 | 16:06 |
| *** lblanchard has quit IRC | 16:06 | |
| *** lblanchard has joined #tripleo | 16:07 | |
| *** ooolpbot has joined #tripleo | 16:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757556 | 16:10 |
| openstack | Launchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged] | 16:10 |
| *** ooolpbot has quit IRC | 16:10 | |
| *** salmankhan has quit IRC | 16:11 | |
| *** jfrancoa has quit IRC | 16:16 | |
| bnemec | https://bugs.launchpad.net/tripleo/+bug/1759317 | 16:18 |
| openstack | Launchpad bug 1759317 in tripleo "Undercloud install fails with new default SSL setting" [Critical,Triaged] | 16:18 |
| *** salmankhan has joined #tripleo | 16:18 | |
| *** derekh has quit IRC | 16:21 | |
| openstackgerrit | Gabriele Cerami proposed openstack/tripleo-specs master: Modify tech-debt policy https://review.openstack.org/545392 | 16:21 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart master: add feature_name variable to config https://review.openstack.org/556962 | 16:21 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras master: allow full tempest fs020 more resources https://review.openstack.org/556965 | 16:23 |
| *** artom has quit IRC | 16:23 | |
| *** pcaruana has joined #tripleo | 16:25 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates master: WIP: Migrate Ceph deployment to external_deploy_task https://review.openstack.org/546966 | 16:25 |
| *** suuuper has joined #tripleo | 16:26 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras master: allow full tempest fs020 more resources https://review.openstack.org/556965 | 16:26 |
| *** zoli is now known as zoli|gone | 16:26 | |
| *** zoli|gone is now known as zoli | 16:26 | |
| *** derekh has joined #tripleo | 16:27 | |
| *** moshele has joined #tripleo | 16:33 | |
| *** lucasagomes is now known as lucas-afk | 16:33 | |
| hjensas | bnemec: my undercloud installed fine, instack-undercloud-8.3.1-0.20180327001318.7a7717e.el7.centos.noarch generate_service_certificate = true in my config. | 16:33 |
| *** holser__ has quit IRC | 16:33 | |
| *** chem is now known as chem|eod | 16:34 | |
| bnemec | hjensas: Hmm, that's the same version I have. | 16:34 |
| *** radeks has quit IRC | 16:36 | |
| hjensas | bnemec: The repo I used is https://trunk.rdoproject.org/centos7/62/d0/62d0d75229fa07e33ca24ba5c59d63d66a9182a6_f1a4ee5a - my config: http://paste.openstack.org/show/715429/ | 16:38 |
| *** trown is now known as trown|lunch | 16:40 | |
| bnemec | hjensas: Hmm, you are setting a lot more stuff than I am. I'm mostly taking the defaults, except for disabling a few things and setting the hostname and mtu. | 16:41 |
| *** udesale has quit IRC | 16:41 | |
| *** morazi has quit IRC | 16:42 | |
| hjensas | bnemec: ack, most of them are uncommented defaults. (Or was the defaults ... i've used that config for a while.) | 16:43 |
| *** jpich has quit IRC | 16:43 | |
| *** moshele has quit IRC | 16:44 | |
| *** morazi has joined #tripleo | 16:44 | |
| *** janki has quit IRC | 16:49 | |
| openstackgerrit | John Trowbridge proposed openstack/tripleo-quickstart master: [DNM] Testing keystone-only featureset000 https://review.openstack.org/556630 | 16:51 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart master: add feature_name variable to config https://review.openstack.org/556962 | 16:56 |
| *** fragatina has joined #tripleo | 16:56 | |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-validations master: Add validation for checking baremetal boot images https://review.openstack.org/555325 | 16:57 |
| *** bkopilov has quit IRC | 16:57 | |
| *** shreshtha|wfh has quit IRC | 16:57 | |
| *** derekh has quit IRC | 16:59 | |
| *** gkadam_ has quit IRC | 17:00 | |
| *** paramite has quit IRC | 17:03 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras master: allow full tempest fs020 more resources https://review.openstack.org/556965 | 17:08 |
| *** fragatina has quit IRC | 17:09 | |
| *** ooolpbot has joined #tripleo | 17:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757556 | 17:10 |
| *** ooolpbot has quit IRC | 17:10 | |
| openstack | Launchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged] | 17:10 |
| *** bkopilov has joined #tripleo | 17:11 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras master: allow full tempest fs020 more resources https://review.openstack.org/556965 | 17:12 |
| *** artom has joined #tripleo | 17:25 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/queens: Upgrade data on disk on mysql major upgrade - non HA https://review.openstack.org/556790 | 17:29 |
| *** morazi has quit IRC | 17:29 | |
| *** morazi has joined #tripleo | 17:30 | |
| openstackgerrit | Marius Cornea proposed openstack/tripleo-upgrade master: WIP: Add support for per role FFU upgrade https://review.openstack.org/556687 | 17:30 |
| *** fragatina has joined #tripleo | 17:30 | |
| *** morazi has quit IRC | 17:31 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates master: WIP: Migrate Ceph deployment to external_deploy_task https://review.openstack.org/546966 | 17:34 |
| *** mdnadeem has quit IRC | 17:34 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Allow stacks to remove old update resources https://review.openstack.org/555250 | 17:37 |
| openstackgerrit | Merged openstack/python-tripleoclient master: Generate inventory with tripleo-admin user https://review.openstack.org/554545 | 17:37 |
| openstackgerrit | Merged openstack/instack-undercloud master: Add purge shadow tables to instack-undercloud https://review.openstack.org/555308 | 17:39 |
| *** morazi has joined #tripleo | 17:39 | |
| *** tesseract has quit IRC | 17:40 | |
| *** artom has quit IRC | 17:41 | |
| openstackgerrit | Liz Blanchard proposed openstack/tripleo-ui master: Move Logout under User Name Dropdown https://review.openstack.org/556991 | 17:42 |
| *** trown|lunch is now known as trown | 17:45 | |
| jrist | ah lookie here ^ | 17:46 |
| jrist | lblanchard: ^ all the attention | 17:46 |
| jrist | ALL | 17:46 |
| jrist | :) | 17:46 |
| lblanchard | :) | 17:47 |
| lblanchard | nothing to see here! These aren't the droids you're looking for… | 17:47 |
| *** panda is now known as panda|off | 17:48 | |
| *** pcaruana has quit IRC | 17:51 | |
| *** dparkes has joined #tripleo | 17:55 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates master: WIP: Migrate Ceph deployment to external_deploy_task https://review.openstack.org/546966 | 17:56 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates master: WIP: Migrate Ceph deployment to external_deploy_task https://review.openstack.org/546966 | 17:57 |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Share overcloud Heat templates for undercloud conf https://review.openstack.org/555338 | 18:05 |
| openstackgerrit | Merged openstack/tripleo-common master: Remove /etc/os-net-config/config.json in os-apply-config https://review.openstack.org/555452 | 18:05 |
| openstackgerrit | Bob Fournier proposed openstack/tripleo-common stable/queens: Remove /etc/os-net-config/config.json in os-apply-config https://review.openstack.org/556997 | 18:07 |
| openstackgerrit | Bob Fournier proposed openstack/os-net-config stable/queens: Don't restart ivs/nfvswitch in os-net-config https://review.openstack.org/556998 | 18:07 |
| *** shardy has quit IRC | 18:09 | |
| *** ooolpbot has joined #tripleo | 18:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757556 | 18:10 |
| *** ooolpbot has quit IRC | 18:10 | |
| openstack | Launchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged] | 18:10 |
| *** psachin has joined #tripleo | 18:10 | |
| *** pkovar1 has quit IRC | 18:11 | |
| openstackgerrit | Marius Cornea proposed openstack/tripleo-upgrade master: WIP: Add support for per role FFU upgrade https://review.openstack.org/556687 | 18:12 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: FFU: Fix gnocchi FFU tasks https://review.openstack.org/548635 | 18:12 |
| openstackgerrit | Merged openstack/tripleo-quickstart master: bump fs20 tempest to 3 workers https://review.openstack.org/556695 | 18:12 |
| openstackgerrit | Merged openstack/puppet-tripleo stable/queens: HAProxy: expose stats socket in HA containerized deployment https://review.openstack.org/556833 | 18:12 |
| openstackgerrit | Merged openstack/tripleo-upgrade stable/queens: Support custom stack names during FFU https://review.openstack.org/556574 | 18:12 |
| *** dparkes has quit IRC | 18:13 | |
| *** fragatina has quit IRC | 18:22 | |
| *** psachin has quit IRC | 18:25 | |
| *** dtantsur is now known as dtantsur|afk | 18:25 | |
| openstackgerrit | Merged openstack/puppet-tripleo master: Add missing cron jobs for Overcloud cleanup https://review.openstack.org/555362 | 18:28 |
| *** anilvenkata has quit IRC | 18:29 | |
| *** anilvenkata has joined #tripleo | 18:30 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras master: rdocloud-ci.yml is not used https://review.openstack.org/557000 | 18:31 |
| openstackgerrit | wes hayutin proposed openstack-infra/tripleo-ci master: the rdocloud flavors are defined in the tebroker https://review.openstack.org/557002 | 18:37 |
| *** ebarrera has joined #tripleo | 18:37 | |
| openstackgerrit | Merged openstack/os-apply-config master: add lower-constraints job https://review.openstack.org/556040 | 18:37 |
| *** fragatina has joined #tripleo | 18:46 | |
| *** pchavva has quit IRC | 18:48 | |
| openstackgerrit | Marius Cornea proposed openstack/tripleo-upgrade master: WIP: Add support for per role FFU upgrade https://review.openstack.org/556687 | 18:48 |
| *** etingof has quit IRC | 18:48 | |
| *** ebarrera has quit IRC | 18:50 | |
| hjensas | arxcruz|rover bug/1757556 are we any closer on that bug? Is any tests being done to see if https://review.openstack.org/549331 helps? | 18:50 |
| *** moshele has joined #tripleo | 18:51 | |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-heat-templates master: Add purge to Nova cleanup cron. https://review.openstack.org/551966 | 18:52 |
| openstackgerrit | Liz Blanchard proposed openstack/tripleo-ui master: Move Logout under User Name Dropdown https://review.openstack.org/556991 | 18:54 |
| *** ykarel|away has quit IRC | 18:54 | |
| *** moshele has quit IRC | 18:54 | |
| *** trozet has quit IRC | 18:55 | |
| *** Goneri has quit IRC | 18:59 | |
| bnemec | hjensas: So if I override the ip ranges then it works. I wonder if this is related to the problem slagle reported with default values not being propagated correctly. | 19:00 |
| openstackgerrit | wes hayutin proposed openstack-infra/tripleo-ci master: if fs20 use m1.ci.xlarge https://review.openstack.org/557015 | 19:02 |
| *** salmankhan has quit IRC | 19:02 | |
| *** gfidente has quit IRC | 19:02 | |
| *** almondjoy has quit IRC | 19:02 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras master: allow full tempest fs020 more resources https://review.openstack.org/556965 | 19:08 |
| hjensas | bnemec: hmm, how minimal is your config? Did you override all the options, or just the dhcp_start/end and/or inspection range? | 19:09 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Run ovb-ha (fs001) with a containerized undercloud https://review.openstack.org/542556 | 19:09 |
| openstackgerrit | Marius Cornea proposed openstack/tripleo-upgrade master: WIP: Add support for per role FFU upgrade https://review.openstack.org/556687 | 19:10 |
| *** ooolpbot has joined #tripleo | 19:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757556 | 19:10 |
| *** ooolpbot has quit IRC | 19:10 | |
| openstack | Launchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged] | 19:10 |
| mwhahaha | weshay: soo i'm running the tempest run for fs020 and it seems like the extra load is from httpd | 19:10 |
| weshay | wha? | 19:11 |
| weshay | really | 19:11 |
| mwhahaha | yea | 19:11 |
| weshay | horizon?' | 19:11 |
| mwhahaha | httpd is the top 2 things | 19:11 |
| mwhahaha | not sure | 19:11 |
| weshay | mwhahaha, can you share a tmate or something | 19:11 |
| weshay | I'd like to see how one does perf monitoring w/ containers | 19:12 |
| mwhahaha | yea sec | 19:12 |
| bnemec | httpd is probably keystone or one of the other api services. | 19:12 |
| mwhahaha | yea it's likely keystone | 19:12 |
| bnemec | hjensas: The working config looks like http://paste.openstack.org/show/715452/, the failing one is http://paste.openstack.org/show/715453/ | 19:13 |
| hjensas | bnemec: ok, I'll try to reproduce using the minimal config. | 19:16 |
| *** trown is now known as trown|brb | 19:17 | |
| openstackgerrit | Damien Ciabrini proposed openstack/tripleo-heat-templates stable/queens: HAProxy: expose stats socket in HA deployment after upgrade https://review.openstack.org/557019 | 19:18 |
| weshay | mwhahaha, rook is looking at the perf too | 19:25 |
| mwhahaha | yea some of the httpd is keystone | 19:25 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Fix fluentd upgrade tasks during FFU. https://review.openstack.org/554653 | 19:26 |
| rook | I don't see much from http vs neutron-server, neutron-ovs-agent | 19:26 |
| *** itlinux has joined #tripleo | 19:26 | |
| rook | howve, i have not insight into what tempest is doing right now. | 19:26 |
| *** ebarrera has joined #tripleo | 19:31 | |
| *** agopi|out has quit IRC | 19:31 | |
| mwhahaha | i'm seeing a lot in neutron-rootwrap | 19:32 |
| mwhahaha | which seems weird | 19:32 |
| mwhahaha | weshay: we need to get back to https://review.openstack.org/#/c/554377/ but it seems like the linter doesn't like the shell usage. is there an ignore option for ansible-lint? | 19:35 |
| weshay | mwhahaha, ya.. I saw your comment ealier today. Going to address that today | 19:35 |
| mwhahaha | hrybacki: do you know if it's a problem that we have max_active_keys set to 5 but only have 2 actual keys? is there any perf issue with this? | 19:44 |
| mwhahaha | seeing a bunch of warnings in keystone | 19:44 |
| mwhahaha | 2018-03-27 19:41:04.382 18 DEBUG keystone.common.token_utils [req-6784e26b-a991-4519-bb6f-5c81bf213ab1 2f69c84ed35448508d3f627de85e4087 6ffa97007e1a47c3997f38fef1da82a9 - default default] Loaded 2 Fernet keys from /etc/keystone/fernet-keys, but `[fernet_tokens] max_active_keys = 5`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? load_keys | 19:44 |
| mwhahaha | /usr/lib/python2.7/site-packages/keystone/common/token_utils.py:306 | 19:44 |
| mwhahaha | also why do we keep reloading the keys | 19:45 |
| rook | mwhahaha: I only see 8 rootwraps | 19:45 |
| *** Goneri has joined #tripleo | 19:45 | |
| mwhahaha | rook: yea it's not quanity but seems to be a recuring cpu consumer | 19:45 |
| mwhahaha | rook: since they are spawned in the neutron containers they aren't persistent | 19:46 |
| rook | http://norton.perf.lab.eng.rdu.redhat.com:3000/d/ekWONB6kz/openstack-controller-general-system-performance?orgId=1&var-Cloud=openstack-oooq&var-Node=overcloud-controller-0&var-Disk=disk-vda&var-Interface=interface-br-ex&var-OSD=All&var-Pool=All&var-cpus0=All&var-cpus00=All&from=1522176411746&to=1522180011746 | 19:46 |
| *** akrivoka has quit IRC | 19:48 | |
| hjensas | bnemec: http://paste.openstack.org/show/715455/ <- this config install succeed on my test. | 19:49 |
| *** dprince has quit IRC | 19:50 | |
| mwhahaha | rook: i don't see neutron-rootwrap in there | 19:50 |
| rook | mwhahaha: yeah - I am on my own machine looking | 19:51 |
| rook | but yeah | 19:51 |
| mwhahaha | yea i'm on a reproducer watching top | 19:51 |
| mwhahaha | i like those graphs tho, very pretty :D | 19:51 |
| rook | neutron-openvswitch is my biggest hitter. | 19:52 |
| *** jtomasek has quit IRC | 19:52 | |
| bnemec | hjensas: wtf? Okay, I'll keep looking. | 19:52 |
| bnemec | rook: That's probably https://bugs.launchpad.net/tripleo/+bug/1755605 | 19:53 |
| openstack | Launchpad bug 1755605 in tripleo "openvswitch-agent eating 100% cpu" [High,Triaged] | 19:53 |
| rook | mwhahaha: swift is also busy | 19:53 |
| bnemec | It's supposedly fixed on the neutron side, but maybe we haven't promoted yet? | 19:53 |
| mwhahaha | not really for me, it's mostly neutron/httpd in terms of cpu usage | 19:53 |
| rook | bnemec: this is the overcloud. | 19:53 |
| bnemec | rook: It also happens there. | 19:53 |
| rook | so, is this the issue them/ | 19:53 |
| bnemec | I just initially noticed it on the undercloud. | 19:53 |
| mwhahaha | i'm using a promotion dlrn from yesterday | 19:54 |
| rook | mwhahaha: I am using the dlrn hash weshay sent me | 19:54 |
| mwhahaha | doesn't have the ovs stuff | 19:54 |
| rook | ah, i think i do. | 19:54 |
| rook | ovs2.9? | 19:54 |
| mwhahaha | yea i don't have that | 19:54 |
| rook | openvswitch-2.9.0-3.el7.x86_64 | 19:54 |
| hjensas | I suspect bug/1755605 as well, it keeps my t460 going very hot! | 19:54 |
| mwhahaha | openvswitch-2.8.2-1.el7.x86_64 | 19:54 |
| bnemec | Oh man, I need one of those Northern Hawk Owls in my neighborhood. So many voles. | 19:57 |
| rook | In the docker world | 20:01 |
| rook | what does : (unhealhty) mean? | 20:01 |
| *** trozet has joined #tripleo | 20:01 | |
| rook | 23426c7fd051 192.168.24.1:8787/tripleomaster/centos-binary-neutron-server:current-tripleo "kolla_start" 2 hours ago Up 2 hours (unhealthy) neutron_api | 20:01 |
| *** bkopilov has quit IRC | 20:01 | |
| mwhahaha | the healthcheck is reporting failure | 20:01 |
| mwhahaha | but chances are our healthcheck is bad | 20:01 |
| mwhahaha | cause like memcache is unhealthy :D | 20:01 |
| mwhahaha | it doesn't do anything except run | 20:01 |
| hjensas | mwhahaha: rook: the fix for that ovs agent eating cpu is not in openvswitch, it is in neutron/agent/linux/ip_conntrack.py -- but if you have a promotion since yesterday the patch should be there ... | 20:03 |
| mwhahaha | but that would be the agent | 20:04 |
| mwhahaha | not rootwrap or server right? | 20:04 |
| bnemec | If we're only going to configure 2 keys we should set max_active_keys to that. | 20:05 |
| bnemec | Scroll fail. :-( | 20:05 |
| bnemec | Although still valid. :-) | 20:05 |
| bnemec | https://github.com/openstack/instack-undercloud/blob/master/elements/puppet-stack-config/puppet-stack-config.yaml.template#L226 | 20:05 |
| mwhahaha | bnemec: i know that does seem like the better choice | 20:05 |
| hjensas | mwhahaha: yes, it would be the agent. | 20:06 |
| mwhahaha | https://github.com/openstack/tripleo-heat-templates/blob/88daf0d5da305935573e28db315184cc07668f56/puppet/services/keystone.yaml#L136-L139 | 20:06 |
| rook | just the openvswitch-agent mwhahaha not the neutron-server... nor the rootwrap :/ | 20:06 |
| rook | but | 20:06 |
| rook | neutron-openvswitch-agent is stillconsuming ~60% | 20:06 |
| rook | with the latest / and ovs2.9 | 20:07 |
| rook | swift is pretty hangry too | 20:07 |
| *** liverpooler has quit IRC | 20:08 | |
| *** toure is now known as toure|gone | 20:09 | |
| *** ooolpbot has joined #tripleo | 20:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757556 | 20:10 |
| *** ooolpbot has quit IRC | 20:10 | |
| openstack | Launchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged] | 20:10 |
| weshay | hjensas, the latest neutron code is tested in every attempt at a promotion | 20:10 |
| weshay | hjensas, every 4-5 hours | 20:10 |
| weshay | rook, you are looking at http://norton.perf.lab.eng.rdu.redhat.com:3000/d/ekWONB6kz/openstack-controller-general-system-performance?orgId=1&var-Cloud=openstack-oooq&var-Node=overcloud-controller-0&var-Disk=disk-vda&var-Interface=interface-br-ex&var-OSD=All&var-Pool=All&var-cpus0=All&var-cpus00=All&from=1522174962124&to=1522178562125&panelId=88&fullscreen ? | 20:12 |
| openstackgerrit | Ronelle Landy proposed openstack-infra/tripleo-ci master: DNM: Define overcloud_docker_registry_namespace for rdocloud https://review.openstack.org/557035 | 20:13 |
| *** athomas has quit IRC | 20:14 | |
| *** ebarrera has quit IRC | 20:15 | |
| *** bkopilov has joined #tripleo | 20:15 | |
| hjensas | weshay: ok, thanks. Then it probably is not that. | 20:18 |
| *** moshele has joined #tripleo | 20:23 | |
| *** tosky has quit IRC | 20:25 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras master: deprecate older general configs https://review.openstack.org/554263 | 20:25 |
| *** moshele has quit IRC | 20:30 | |
| rook | oh, here we go mwhahaha | 20:30 |
| rook | rootwrap hell | 20:30 |
| mwhahaha | rook: :D so i turned up debugging on rootwrap | 20:30 |
| *** tosky has joined #tripleo | 20:30 | |
| mwhahaha | it's a bunch of iptables-save iptables-restore | 20:30 |
| mwhahaha | at least for ovs agent | 20:32 |
| rook | mwhahaha: i wonder if this is already fixed based on what hjensas mentioned. | 20:33 |
| mwhahaha | maybe | 20:33 |
| rook | weshay: yes, i was. | 20:33 |
| weshay | k.. very cool stuff btw | 20:33 |
| rook | well, we need to get rid of the ops-tools repo quickstart ships with | 20:34 |
| weshay | rook, was that on a deployed overcloud? | 20:34 |
| rook | :D | 20:34 |
| rook | that was the only change. | 20:34 |
| rook | that is on the oooq overcloud | 20:34 |
| weshay | nice | 20:34 |
| weshay | the ops-tools repo? | 20:34 |
| rook | there was a quickstart-opstools repo | 20:34 |
| rook | that i nuked | 20:34 |
| weshay | k | 20:35 |
| *** suuuper has quit IRC | 20:35 | |
| weshay | rook, what did you deploy with? which hash? | 20:35 |
| weshay | we can check | 20:35 |
| rook | errr | 20:35 |
| rook | cfaab0cbdc34c064c468bf0157a334ef22b6645d_e3f032d8 | 20:37 |
| *** morazi has quit IRC | 20:38 | |
| weshay | hjensas, sorry what was the patch we're looking for in neutron? | 20:40 |
| hjensas | weshay: https://review.openstack.org/549331 | 20:40 |
| weshay | thank you! | 20:40 |
| mwhahaha | wait | 20:41 |
| mwhahaha | that's from 10 days ago | 20:41 |
| mwhahaha | right about the time this starting to perform worse? | 20:41 |
| weshay | I can check, pretty close | 20:43 |
| *** dparkes has joined #tripleo | 20:44 | |
| mwhahaha | also kinda lame that the worker count is fixed | 20:44 |
| mwhahaha | at 8 | 20:44 |
| weshay | rook, so you have the neutron patch and the latest ovs | 20:45 |
| weshay | https://trunk.rdoproject.org/centos7-master/cf/aa/cfaab0cbdc34c064c468bf0157a334ef22b6645d_e3f032d8/ | 20:45 |
| weshay | openstack-neutron-12.0.0-0.20180323180756.0982729.el7.centos.noarch.rpm | 20:45 |
| weshay | which is from https://github.com/openstack/neutron/commit/0982729f56d70b10863ef56b8a6464cf151c3e26#diff-e900a9020bbb5db562fe3b148873a78d | 20:45 |
| *** olivierbourdon38 has quit IRC | 20:46 | |
| *** olivierbourdon38 has joined #tripleo | 20:46 | |
| weshay | the patch hjensas is referring to landed on the 17 | 20:46 |
| *** ansmith has quit IRC | 20:47 | |
| weshay | mwhahaha, full tempest was running to completion before and after that neutron patch | 20:51 |
| mwhahaha | k | 20:51 |
| weshay | it was failing, but | 20:51 |
| *** aputtur_ has quit IRC | 20:53 | |
| weshay | mwhahaha, ok.. it started here https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master/fb86c5b/undercloud/home/jenkins/tempest_output.log.txt.gz | 20:56 |
| *** Goneri has quit IRC | 20:56 | |
| mwhahaha | weshay: that run? | 20:56 |
| weshay | hrm.. | 20:56 |
| weshay | sec.. | 20:56 |
| *** fpan has joined #tripleo | 20:56 | |
| weshay | ok.. so.. | 20:57 |
| weshay | https://review.rdoproject.org/jenkins/job/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master/ | 20:58 |
| weshay | build 789 790 timeout on tempest | 20:58 |
| *** hjensas has quit IRC | 20:58 | |
| weshay | 791 passes | 20:58 |
| weshay | 792+ timeout | 20:58 |
| mwhahaha | weshay: 790 and 791 have the same pkgs | 21:03 |
| *** chkumar246 has joined #tripleo | 21:04 | |
| *** chandankumar has quit IRC | 21:05 | |
| *** moshele has joined #tripleo | 21:05 | |
| *** etingof has joined #tripleo | 21:06 | |
| weshay | mwhahaha, ya | 21:07 |
| weshay | you check rpm-qa.txt | 21:08 |
| mwhahaha | yea | 21:08 |
| openstackgerrit | John Trowbridge proposed openstack/tripleo-quickstart master: [DNM] Testing keystone-only featureset000 https://review.openstack.org/556630 | 21:08 |
| weshay | WAT DA FUK | 21:08 |
| *** chkumar246 has quit IRC | 21:09 | |
| *** ooolpbot has joined #tripleo | 21:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757556 | 21:10 |
| *** ooolpbot has quit IRC | 21:10 | |
| openstack | Launchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged] | 21:10 |
| *** raildo has quit IRC | 21:10 | |
| openstackgerrit | Ronelle Landy proposed openstack-infra/tripleo-ci master: DNM: Define overcloud_docker_registry_namespace for rdocloud https://review.openstack.org/557035 | 21:15 |
| *** itlinux has quit IRC | 21:15 | |
| mwhahaha | weshay: i got dstat from a partial tempest run (i gave up eventually). it's like 50% user | 21:16 |
| mwhahaha | so nothing in particular jumps out as a problem | 21:16 |
| *** bfournie has quit IRC | 21:18 | |
| *** itlinux has joined #tripleo | 21:19 | |
| weshay | heh | 21:21 |
| *** chkumar246 has joined #tripleo | 21:21 | |
| weshay | mwhahaha, what about rook's graphana | 21:21 |
| mwhahaha | well that had more cpus than our CI | 21:22 |
| mwhahaha | there wasn't really anything that stands out | 21:22 |
| weshay | that's the problem w/ perf guys, they always have better hardware than you do :) | 21:22 |
| mwhahaha | i know right? :D | 21:23 |
| mwhahaha | oh no it didn't | 21:24 |
| mwhahaha | it was 2 cpu | 21:24 |
| mwhahaha | yea so similar numbers | 21:24 |
| mwhahaha | so neutron is definitely the big offender in terms of processor usage | 21:25 |
| rook | lol | 21:27 |
| rook | this is a virt oc bros | 21:27 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Deploy container-multinode (fs010) with a containerized undercloud https://review.openstack.org/542906 | 21:30 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Deploy container-multinode (fs010) with a containerized undercloud https://review.openstack.org/542906 | 21:30 |
| *** cdearborn has quit IRC | 21:34 | |
| *** rfolco|ruck is now known as rfolco|off | 21:35 | |
| *** fragatina has quit IRC | 21:35 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/queens: Correctly set NeutronMl2PluginBase for OVN scenario https://review.openstack.org/556823 | 21:35 |
| *** fragatina has joined #tripleo | 21:35 | |
| *** ansmith has joined #tripleo | 21:36 | |
| *** moshele has quit IRC | 21:36 | |
| *** holser__ has joined #tripleo | 21:37 | |
| *** mcornea has quit IRC | 21:38 | |
| EmilienM | mwhahaha: I think https://github.com/puppetlabs/puppetlabs-apache/blob/3f68bac435838b20af3682a31d85670328689e98/manifests/mod/prefork.pp#L2 is killing us | 21:38 |
| mwhahaha | well it's not helping | 21:39 |
| EmilienM | I haven't checked how it works but do we load the prefork or it it automatic? | 21:39 |
| mwhahaha | yea | 21:39 |
| mwhahaha | it's automatic | 21:39 |
| mwhahaha | i think that's the default | 21:39 |
| EmilienM | it's very opinionated | 21:39 |
| mwhahaha | so any service we front with httpd gets those number of processes launched in their containers | 21:39 |
| mwhahaha | since it's not httpd for the host but rather per container | 21:39 |
| mwhahaha | so keystone gets that, nova gets that, etc etc | 21:40 |
| mwhahaha | and apache is not exactly the lightest weight thing int he world | 21:40 |
| EmilienM | yeah | 21:40 |
| EmilienM | for sure we should stop using the prefork param per container | 21:40 |
| EmilienM | I'm trying to find where do we include this class | 21:41 |
| rook | mwhahaha: lol and each container has its own apache :D | 21:42 |
| rook | yah for sharing | 21:42 |
| mwhahaha | rook: yea | 21:42 |
| mwhahaha | there by losing anything we gained by using apache | 21:42 |
| mwhahaha | might as well go back to eventlet | 21:42 |
| mwhahaha | EmilienM: i think it comes in via https://github.com/puppetlabs/puppetlabs-apache/blob/d2e326b6c78dffbb94a70cda4ca840829275847c/manifests/default_mods.pp#L114 or something | 21:43 |
| *** jmelvin_bomgr has quit IRC | 21:43 | |
| * rook wonders if mwhahaha is kidding or not... | 21:43 | |
| mwhahaha | maybe i am, maybe i'm not | 21:44 |
| mwhahaha | YOU'LL NEVER KNOW | 21:44 |
| *** lblanchard has quit IRC | 21:44 | |
| mwhahaha | EmilienM: https://github.com/puppetlabs/puppetlabs-apache/blob/master/manifests/params.pp#L85 | 21:45 |
| EmilienM | yeah | 21:45 |
| mwhahaha | so that's why there's 120+ httpd processes on the controller | 21:45 |
| mwhahaha | rather than like 10 | 21:45 |
| * mwhahaha sighs | 21:45 | |
| EmilienM | I guess the simplest way is to override apache::mod::prefork hiera params | 21:45 |
| rook | is it a easy flag to switch? | 21:45 |
| EmilienM | and reduce the default | 21:45 |
| mwhahaha | well for containers do we want prefork? | 21:45 |
| EmilienM | I'm afraid we can't easily disable the mod right? | 21:45 |
| EmilienM | yeah that's the question | 21:46 |
| mwhahaha | rook: which is the least questionable apache process method from a perf perspective? | 21:46 |
| EmilienM | the question is do we need MPM for openstack APIs | 21:46 |
| EmilienM | and I think we need it | 21:46 |
| mwhahaha | it's all getting piped back to a specific wsgi config anyway | 21:46 |
| mwhahaha | so it's more for handling the http part | 21:46 |
| mwhahaha | so i have a feeling we should definately reduce the startservers and min servers | 21:47 |
| EmilienM | if we need MPM we can just override apache::mod::prefork::startservers (and others) in puppet/services/apache.j2.yaml | 21:47 |
| EmilienM | mwhahaha: let's give it a try | 21:49 |
| *** mcornea has joined #tripleo | 21:49 | |
| mwhahaha | EmilienM: let me propose something | 21:49 |
| EmilienM | oki doc | 21:49 |
| rook | MPM would be preferred from a OSP perspective mwhahaha -- I think there was a rhos-dev discussing other http solutions like unicorn | 21:52 |
| mwhahaha | rook: yea i meant like prefork vs whatever the thread one is | 21:52 |
| bnemec | If we drop the number of processes, is that going to cause concurrency issues for the apis? | 21:53 |
| rook | yes | 21:53 |
| rook | totally -- we already have seen that. | 21:53 |
| rook | mwhahaha: worker i think is what we want | 21:53 |
| bnemec | I know we had a problem where we were basically running a single worker because they were multiple threads in a single process. | 21:53 |
| * rook looks thing sup | 21:53 | |
| bnemec | \o/ GIL | 21:53 |
| mwhahaha | so there's multiple worker configs | 21:53 |
| mwhahaha | in this case it's teh httpd config not the wsgi config | 21:53 |
| mwhahaha | so we're fronting say 2 keystone workers with 8 httpd workers | 21:54 |
| bnemec | Ah, okay. That does seem unnecessary. | 21:54 |
| mwhahaha | so we're fronting each api with at least 5 httpd instances | 21:54 |
| mwhahaha | because defaults | 21:54 |
| mwhahaha | so this is purely the number of httpd instances and not the actual wsgi service configs | 21:54 |
| mwhahaha | so things waiting around for httpd requests | 21:55 |
| mwhahaha | once it gets busy, apache will launch new ones | 21:55 |
| EmilienM | the defaults in puppetlabs module are terrible this time. | 21:55 |
| mwhahaha | but by default with 0 traffic we have at min 5* number of api endpoints | 21:55 |
| mwhahaha | because each container has it's own httpd :/ | 21:55 |
| rook | right | 21:55 |
| rook | nothing is shared.. which was the whole reason to move to wsgi | 21:55 |
| mwhahaha | where on baremetal it used to be 5 httpd for all api servers | 21:55 |
| mwhahaha | dat resource usage | 21:56 |
| rook | sharing is caring | 21:56 |
| *** alee_ has quit IRC | 21:56 | |
| *** fpan has quit IRC | 21:57 | |
| EmilienM | so we change the defaults and switch to worker ? | 21:57 |
| EmilienM | mpm_module seems to be the default for debian but for redhat it's prefork | 21:58 |
| mwhahaha | EmilienM: yea that was my question | 21:58 |
| mwhahaha | which is better for redhat | 21:58 |
| EmilienM | in my containerized undercloud I have 85 process of httpd :-O | 21:59 |
| mwhahaha | prefork has been the default for forever | 21:59 |
| *** trozet has quit IRC | 21:59 | |
| EmilienM | mwhahaha: http://paste.openstack.org/show/EQG443WPYEbGwD0Ty2B0/ | 21:59 |
| mwhahaha | unfortunately like each of these need to be tunable for each service now | 21:59 |
| EmilienM | I reduced StartServers from 8 to 2 | 22:00 |
| mwhahaha | let's see if this helps before we go down that route | 22:00 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-heat-templates master: Lower the number of apache servers https://review.openstack.org/557073 | 22:00 |
| mwhahaha | yea i lowered it to 2/2/4 | 22:00 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart master: setup the virt-bmc in tq https://review.openstack.org/554377 | 22:00 |
| *** jlabarre has quit IRC | 22:00 | |
| EmilienM | mwhahaha: we'll be able to compare with what we have now: http://logs.openstack.org/66/551966/12/check/tripleo-ci-centos-7-undercloud-containers/40f1eb5/logs/undercloud/var/log/config-data/keystone/etc/httpd/conf.modules.d/prefork.conf.txt.gz | 22:01 |
| mwhahaha | yea | 22:02 |
| rook | mwhahaha we have discussed internally to look at different http servers and even apache configs... at the time not all services were using wsgi/http -- so maybe now is a good time to re-engage this. | 22:02 |
| *** hjensas has joined #tripleo | 22:02 | |
| *** hjensas has quit IRC | 22:02 | |
| *** hjensas has joined #tripleo | 22:02 | |
| mwhahaha | let me go retry my fs020 with that patch | 22:02 |
| EmilienM | we'll have to backport that down to Pike I think | 22:02 |
| mwhahaha | EmilienM: yes likely | 22:02 |
| rook | did this only come about due to neutron becoming acontainer. | 22:02 |
| *** thrash is now known as thrash|g0ne | 22:03 | |
| mwhahaha | no | 22:03 |
| mwhahaha | it's due to containers | 22:03 |
| * mwhahaha with holds commentary | 22:03 | |
| mwhahaha | side effect of containerization | 22:03 |
| rook | Well 12 was containerized.. :/ | 22:03 |
| mwhahaha | yea an no one noticed :D | 22:04 |
| rook | It seems like Neutron was the tipping point | 22:04 |
| * rook likes to blame Networking | 22:04 | |
| rook | :D | 22:04 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras master: remove virt-bmc setup for tqe https://review.openstack.org/554376 | 22:04 |
| mwhahaha | well it might be our default switch to containers | 22:04 |
| mwhahaha | that seems to keep uncovering stuff | 22:04 |
| * mwhahaha shrugs | 22:04 | |
| * EmilienM containerizes mwhahaha | 22:04 | |
| mwhahaha | i'll try relaunching my fs020 w/ that patch | 22:05 |
| *** itlinux has quit IRC | 22:05 | |
| weshay | mwhahaha, which patch? | 22:08 |
| EmilienM | weshay: https://review.openstack.org/557073 | 22:08 |
| weshay | cool | 22:08 |
| mwhahaha | weshay: what do you put in ZUUL_CHANGES for it to work? | 22:08 |
| weshay | mwhahaha, you'll see a variable called ZUUL_CHANGES in the script | 22:09 |
| mwhahaha | yea do you use the ref/changes... stuf | 22:09 |
| weshay | just add that patch in the proper format | 22:09 |
| mwhahaha | right | 22:09 |
| mwhahaha | what's the format | 22:09 |
| mwhahaha | HEH | 22:09 |
| weshay | mwhahaha, you may have some changes in there | 22:09 |
| weshay | but getting | 22:09 |
| *** ooolpbot has joined #tripleo | 22:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757556 | 22:10 |
| openstack | Launchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged] | 22:10 |
| *** ooolpbot has quit IRC | 22:10 | |
| weshay | openstack/tripleo-heat-templates:refs/changes/73/557073/1 | 22:10 |
| * weshay grabs doc | 22:10 | |
| mwhahaha | k | 22:10 |
| weshay | openstack/tripleo-heat-templates:master:refs/changes/73/557073/1 | 22:10 |
| mwhahaha | yea | 22:11 |
| mwhahaha | i found an example in another ci run | 22:11 |
| mwhahaha | since it's 4pm | 22:11 |
| mwhahaha | i'll just check that tomorrow | 22:11 |
| * mwhahaha sighs | 22:11 | |
| EmilienM | mwhahaha: do we have a bug report? | 22:12 |
| mwhahaha | for the proc count? now | 22:12 |
| mwhahaha | no | 22:12 |
| EmilienM | ok | 22:12 |
| EmilienM | rook: please provide early feedback on https://review.openstack.org/#/c/557073/ | 22:12 |
| EmilienM | if it's provide results that we expect we'll probably land it fast | 22:13 |
| mwhahaha | we can land it tomorrow | 22:14 |
| rook | weshay how can i take ^^ and deploy with oooq? | 22:14 |
| mwhahaha | we'll see how this reproducer comes in | 22:14 |
| rook | I can run against my current cloud i have with oooq -- and be able to compare perf/utilization | 22:14 |
| mwhahaha | rook: i grabbbed https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master/3e4eee6/reproducer-quickstart.sh and then updated teh ZUUL_CHANGES with openstack/tripleo-heat-templates:master:refs/changes/73/557073/1 | 22:15 |
| weshay | rook, you can do that or I can get in your libvirt job | 22:15 |
| EmilienM | rook: let me show you the script | 22:15 |
| rook | sweet 3 different ways | 22:15 |
| *** trozet has joined #tripleo | 22:15 | |
| EmilienM | rook: do you have RDO cloud access? | 22:15 |
| rook | :D | 22:15 |
| rook | No - I have a baremetal libvirt cloud | 22:15 |
| rook | cloud/machine | 22:15 |
| rook | that I have working with oooq... | 22:16 |
| *** rcernin has joined #tripleo | 22:16 | |
| rook | ok, mwhahaha i think I can follow your example | 22:16 |
| EmilienM | rook: http://paste.openstack.org/show/715477/ | 22:16 |
| rook | weshay ^ just set ZUUL_CHANGES | 22:16 |
| EmilienM | that will be the reproducer script for his patch | 22:16 |
| EmilienM | ZUUL_CHANGES="openstack/tripleo-heat-templates:master:refs/changes/73/557073/1" | 22:17 |
| rook | ack | 22:17 |
| rook | yeah | 22:17 |
| mwhahaha | anyway i'll let this thing cook and check in on it later | 22:17 |
| EmilienM | I'm also testing it in my env right now | 22:17 |
| *** holser__ has quit IRC | 22:17 | |
| * mwhahaha wanders off | 22:17 | |
| weshay | rook, bash devmode.sh -c config/general_config/featureset020.yml -w /var/tmp/test/ <virthost> | 22:17 |
| rook | I have tempest finishing up, once it completes, i will run some perf tests against this cloud | 22:17 |
| weshay | rook, it will prompt you for the zuul change | 22:18 |
| rook | ack weshay | 22:18 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/python-tripleoclient master: Updated from global requirements https://review.openstack.org/557082 | 22:21 |
| *** fragatina has quit IRC | 22:21 | |
| *** fragatina has joined #tripleo | 22:21 | |
| *** artom has joined #tripleo | 22:24 | |
| *** yolanda has quit IRC | 22:25 | |
| *** yolanda has joined #tripleo | 22:27 | |
| *** lblanchard has joined #tripleo | 22:28 | |
| *** yamahata has joined #tripleo | 22:31 | |
| openstackgerrit | Merged openstack/tripleo-quickstart master: deprecate old general configurations https://review.openstack.org/554255 | 22:32 |
| *** alee_ has joined #tripleo | 22:33 | |
| *** ebarrera has joined #tripleo | 22:36 | |
| *** lblanchard has quit IRC | 22:41 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/pike: A keystone only controller deployment https://review.openstack.org/542968 | 22:46 |
| *** chlong has quit IRC | 22:49 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart master: setup the virt-bmc in tq https://review.openstack.org/554377 | 22:53 |
| *** fragatina has quit IRC | 22:58 | |
| *** fragatina has joined #tripleo | 22:58 | |
| *** jcoufal has quit IRC | 23:03 | |
| *** ebarrera has quit IRC | 23:06 | |
| EmilienM | weshay: I tested the patch in my env | 23:07 |
| EmilienM | I now have 32 process of httpd | 23:07 |
| EmilienM | comparing to 85 before | 23:07 |
| EmilienM | and each container has this httpd config now: http://paste.openstack.org/show/715507/ | 23:08 |
| EmilienM | rook, mwhahaha ^ fyi as well | 23:08 |
| *** ooolpbot has joined #tripleo | 23:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757556 | 23:10 |
| openstack | Launchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged] | 23:10 |
| *** ooolpbot has quit IRC | 23:10 | |
| *** bfournie has joined #tripleo | 23:11 | |
| *** fragatina has quit IRC | 23:14 | |
| *** fragatina has joined #tripleo | 23:14 | |
| *** mcornea has quit IRC | 23:20 | |
| weshay | EmilienM, k cool mine is still running | 23:23 |
| EmilienM | weshay: containerized undercloud man :) | 23:25 |
| *** tosky has quit IRC | 23:25 | |
| *** chandankumar has joined #tripleo | 23:25 | |
| *** chkumar246 has quit IRC | 23:26 | |
| *** mcornea has joined #tripleo | 23:26 | |
| *** chandankumar has quit IRC | 23:30 | |
| openstackgerrit | Merged openstack/paunch master: Allow to limit cgroup cpu shares https://review.openstack.org/554539 | 23:30 |
| EmilienM | so indeed with mwhahaha's patch we can see the new config: http://logs.openstack.org/73/557073/1/check/tripleo-ci-centos-7-undercloud-containers/088c596/logs/undercloud/var/log/config-data/keystone/etc/httpd/conf.modules.d/prefork.conf.txt.gz | 23:34 |
| EmilienM | looking at dstat now | 23:34 |
| *** mcornea has quit IRC | 23:36 | |
| EmilienM | we can see a little win on memory (by 250MB) | 23:37 |
| *** tbonds has joined #tripleo | 23:38 | |
| EmilienM | with alex's patch: http://logs.openstack.org/73/557073/1/check/tripleo-ci-centos-7-undercloud-containers/088c596/logs/undercloud/var/log/extra/dstat.html.gz | 23:38 |
| EmilienM | without: http://logs.openstack.org/02/557002/1/check/tripleo-ci-centos-7-undercloud-containers/e6d2678/logs/undercloud/var/log/extra/dstat.html.gz | 23:38 |
| EmilienM | I guess we can see the difference on the overcloud with more requests from Tempest | 23:39 |
| EmilienM | we'll see what other CI jobs tell us | 23:39 |
| *** chandankumar has joined #tripleo | 23:43 | |
| *** yamahata has quit IRC | 23:44 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Run ovb-ha (fs001) with a containerized undercloud https://review.openstack.org/542556 | 23:48 |
| openstackgerrit | Emilien Macchi proposed openstack/python-tripleoclient master: Manage upgrades to a containerized undercloud https://review.openstack.org/549624 | 23:48 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: DNM - test containerized undercloud upgrades https://review.openstack.org/553633 | 23:48 |
| openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart-extras master: GATE CHECK for quickstart-extras https://review.openstack.org/472607 | 23:55 |
| *** rbrady-afk has quit IRC | 23:57 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!