| *** dsariel has joined #tripleo | 00:01 | |
| *** jbadiapa_ has quit IRC | 00:04 | |
| *** ooolpbot has joined #tripleo | 00:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 00:10 |
|---|---|---|
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1718387 | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1719123 | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720220 | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720458 | 00:10 |
| openstack | Launchpad bug 1718387 in tripleo "ping test is periodically failing for the gate-tripleo-ci-centos-7-nonha-multinode-oooq " [Critical,Triaged] - Assigned to Sofer Athlan-Guyot (sofer-athlan-guyot) | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720556 | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720721 | 00:10 |
| *** ooolpbot has quit IRC | 00:10 | |
| openstack | Launchpad bug 1719123 in tripleo "tempest fails on overcloud keystone admin tenant error" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz) | 00:10 |
| openstack | Launchpad bug 1720220 in tripleo "CI: Most/all legacy-tripleo jobs failing on gate" [Critical,Triaged] | 00:10 |
| openstack | Launchpad bug 1720458 in tripleo "Lastest delorean pip package causes error in autodoc" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 00:10 |
| openstack | Launchpad bug 1720556 in tripleo "tracker: upstream zuul is not triggering periodic jobs" [Critical,Triaged] - Assigned to wes hayutin (weshayutin) | 00:10 |
| openstack | Launchpad bug 1720721 in tripleo "CI: OVB jobs fail because can't install XStatic from PyPI mirror on rh1 cloud" [Critical,Triaged] - Assigned to Paul Belanger (pabelanger) | 00:10 |
| *** jbadiapa_ has joined #tripleo | 00:19 | |
| *** achadha has joined #tripleo | 00:19 | |
| *** achadha has quit IRC | 00:19 | |
| *** achadha has joined #tripleo | 00:20 | |
| *** achadha_ has joined #tripleo | 00:25 | |
| *** tongl has quit IRC | 00:26 | |
| *** gbarros has quit IRC | 00:28 | |
| *** achadha has quit IRC | 00:28 | |
| *** ansmith has quit IRC | 00:31 | |
| *** rbrady has joined #tripleo | 00:33 | |
| *** rbrady has joined #tripleo | 00:33 | |
| *** itlinux has joined #tripleo | 00:35 | |
| *** yamahata has quit IRC | 00:37 | |
| *** thrash is now known as thrash|g0ne | 00:48 | |
| gregwork | Does anyone know why after a successful deployment of the overcloud (ocata release) that connecting to horizon I get a directory index instead of the splash page ? | 01:02 |
| *** snecklifter has quit IRC | 01:04 | |
| *** snecklifter has joined #tripleo | 01:05 | |
| *** ooolpbot has joined #tripleo | 01:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1718387 | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1719123 | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720220 | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720458 | 01:10 |
| openstack | Launchpad bug 1718387 in tripleo "ping test is periodically failing for the gate-tripleo-ci-centos-7-nonha-multinode-oooq " [Critical,Triaged] - Assigned to Sofer Athlan-Guyot (sofer-athlan-guyot) | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720556 | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720721 | 01:10 |
| *** ooolpbot has quit IRC | 01:10 | |
| openstack | Launchpad bug 1719123 in tripleo "tempest fails on overcloud keystone admin tenant error" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz) | 01:10 |
| openstack | Launchpad bug 1720220 in tripleo "CI: Most/all legacy-tripleo jobs failing on gate" [Critical,Triaged] | 01:10 |
| openstack | Launchpad bug 1720458 in tripleo "Lastest delorean pip package causes error in autodoc" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 01:10 |
| openstack | Launchpad bug 1720556 in tripleo "tracker: upstream zuul is not triggering periodic jobs" [Critical,Triaged] - Assigned to wes hayutin (weshayutin) | 01:10 |
| openstack | Launchpad bug 1720721 in tripleo "CI: OVB jobs fail because can't install XStatic from PyPI mirror on rh1 cloud" [Critical,Triaged] - Assigned to Paul Belanger (pabelanger) | 01:10 |
| openstackgerrit | Ben Nemec proposed openstack/puppet-tripleo master: Switch from ipv6-icmp to icmpv6 https://review.openstack.org/509041 | 01:18 |
| *** achadha_ has quit IRC | 01:23 | |
| *** achadha has joined #tripleo | 01:23 | |
| *** aputtur has joined #tripleo | 01:24 | |
| *** cshastri has joined #tripleo | 01:26 | |
| *** achadha has quit IRC | 01:28 | |
| *** itlinux has quit IRC | 01:29 | |
| *** itlinux has joined #tripleo | 01:29 | |
| *** dmacpher has joined #tripleo | 01:33 | |
| *** itlinux has quit IRC | 01:48 | |
| *** fzdarsky_ has joined #tripleo | 01:48 | |
| *** fzdarsky|afk has quit IRC | 01:52 | |
| *** rbrady has quit IRC | 01:54 | |
| *** achadha has joined #tripleo | 02:07 | |
| *** ooolpbot has joined #tripleo | 02:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1718387 | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1719123 | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720220 | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720458 | 02:10 |
| openstack | Launchpad bug 1718387 in tripleo "ping test is periodically failing for the gate-tripleo-ci-centos-7-nonha-multinode-oooq " [Critical,Triaged] - Assigned to Sofer Athlan-Guyot (sofer-athlan-guyot) | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720556 | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720721 | 02:10 |
| *** ooolpbot has quit IRC | 02:10 | |
| openstack | Launchpad bug 1719123 in tripleo "tempest fails on overcloud keystone admin tenant error" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz) | 02:10 |
| openstack | Launchpad bug 1720220 in tripleo "CI: Most/all legacy-tripleo jobs failing on gate" [Critical,Triaged] | 02:10 |
| openstack | Launchpad bug 1720458 in tripleo "Lastest delorean pip package causes error in autodoc" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 02:10 |
| openstack | Launchpad bug 1720556 in tripleo "tracker: upstream zuul is not triggering periodic jobs" [Critical,Triaged] - Assigned to wes hayutin (weshayutin) | 02:10 |
| openstack | Launchpad bug 1720721 in tripleo "CI: OVB jobs fail because can't install XStatic from PyPI mirror on rh1 cloud" [Critical,Triaged] - Assigned to Paul Belanger (pabelanger) | 02:10 |
| *** bobh_ has joined #tripleo | 02:17 | |
| *** tongl has joined #tripleo | 02:17 | |
| *** aputtur has quit IRC | 02:24 | |
| *** daidv has joined #tripleo | 02:27 | |
| *** itlinux has joined #tripleo | 02:31 | |
| *** bnemec has quit IRC | 02:39 | |
| *** itlinux has quit IRC | 02:44 | |
| *** bobh_ has quit IRC | 02:59 | |
| *** noslzzp has joined #tripleo | 03:06 | |
| *** ooolpbot has joined #tripleo | 03:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1718387 | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1719123 | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720220 | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720458 | 03:10 |
| openstack | Launchpad bug 1718387 in tripleo "ping test is periodically failing for the gate-tripleo-ci-centos-7-nonha-multinode-oooq " [Critical,Triaged] - Assigned to Sofer Athlan-Guyot (sofer-athlan-guyot) | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720556 | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720721 | 03:10 |
| *** ooolpbot has quit IRC | 03:10 | |
| openstack | Launchpad bug 1719123 in tripleo "tempest fails on overcloud keystone admin tenant error" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz) | 03:10 |
| openstack | Launchpad bug 1720220 in tripleo "CI: Most/all legacy-tripleo jobs failing on gate" [Critical,Triaged] | 03:10 |
| openstack | Launchpad bug 1720458 in tripleo "Lastest delorean pip package causes error in autodoc" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 03:10 |
| openstack | Launchpad bug 1720556 in tripleo "tracker: upstream zuul is not triggering periodic jobs" [Critical,Triaged] - Assigned to wes hayutin (weshayutin) | 03:10 |
| openstack | Launchpad bug 1720721 in tripleo "CI: OVB jobs fail because can't install XStatic from PyPI mirror on rh1 cloud" [Critical,Triaged] - Assigned to Paul Belanger (pabelanger) | 03:10 |
| *** ramishra has joined #tripleo | 03:21 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: WIP: Fix TripleO CI jobs https://review.openstack.org/508660 | 03:29 |
| *** gbarros has joined #tripleo | 03:32 | |
| *** aditya_r has joined #tripleo | 03:32 | |
| *** achadha has quit IRC | 03:37 | |
| *** abregman|afk has quit IRC | 03:40 | |
| *** masco has joined #tripleo | 03:44 | |
| *** udesale has joined #tripleo | 03:45 | |
| *** Dinesh_Bhor has joined #tripleo | 03:47 | |
| *** gkadam has joined #tripleo | 03:52 | |
| *** dbecker has quit IRC | 03:53 | |
| *** tongl has quit IRC | 03:59 | |
| *** mrunge has joined #tripleo | 04:00 | |
| *** tongl has joined #tripleo | 04:00 | |
| *** mrunge_ has quit IRC | 04:02 | |
| *** ykarel has joined #tripleo | 04:07 | |
| *** dbecker has joined #tripleo | 04:09 | |
| *** ooolpbot has joined #tripleo | 04:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1718387 | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1719123 | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720220 | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720458 | 04:10 |
| openstack | Launchpad bug 1718387 in tripleo "ping test is periodically failing for the gate-tripleo-ci-centos-7-nonha-multinode-oooq " [Critical,Triaged] - Assigned to Sofer Athlan-Guyot (sofer-athlan-guyot) | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720556 | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720721 | 04:10 |
| *** ooolpbot has quit IRC | 04:10 | |
| openstack | Launchpad bug 1719123 in tripleo "tempest fails on overcloud keystone admin tenant error" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz) | 04:10 |
| openstack | Launchpad bug 1720220 in tripleo "CI: Most/all legacy-tripleo jobs failing on gate" [Critical,Triaged] | 04:10 |
| openstack | Launchpad bug 1720458 in tripleo "Lastest delorean pip package causes error in autodoc" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 04:10 |
| openstack | Launchpad bug 1720556 in tripleo "tracker: upstream zuul is not triggering periodic jobs" [Critical,Triaged] - Assigned to wes hayutin (weshayutin) | 04:10 |
| openstack | Launchpad bug 1720721 in tripleo "CI: OVB jobs fail because can't install XStatic from PyPI mirror on rh1 cloud" [Critical,Triaged] - Assigned to Paul Belanger (pabelanger) | 04:10 |
| *** oanson has quit IRC | 04:21 | |
| *** gbarros has quit IRC | 04:28 | |
| *** shreshtha has joined #tripleo | 04:29 | |
| *** pdeore has joined #tripleo | 04:29 | |
| *** oanson has joined #tripleo | 04:30 | |
| *** colonwq has joined #tripleo | 04:32 | |
| *** yamahata has joined #tripleo | 04:33 | |
| *** mdnadeem has joined #tripleo | 04:39 | |
| *** morazi has quit IRC | 04:49 | |
| *** ramishra has quit IRC | 04:54 | |
| *** ramishra has joined #tripleo | 04:55 | |
| openstackgerrit | Pranali Deore proposed openstack/tripleo-common stable/newton: Adding env variable OS_TENANT_NAME back in overcloudrc https://review.openstack.org/509079 | 04:57 |
| *** abregman has joined #tripleo | 04:59 | |
| *** pcaruana has joined #tripleo | 05:01 | |
| *** dparkes has quit IRC | 05:03 | |
| *** psachin has joined #tripleo | 05:04 | |
| *** ratailor has joined #tripleo | 05:07 | |
| *** ooolpbot has joined #tripleo | 05:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1718387 | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1719123 | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720220 | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720458 | 05:10 |
| openstack | Launchpad bug 1718387 in tripleo "ping test is periodically failing for the gate-tripleo-ci-centos-7-nonha-multinode-oooq " [Critical,Triaged] - Assigned to Sofer Athlan-Guyot (sofer-athlan-guyot) | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720556 | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720721 | 05:10 |
| *** ooolpbot has quit IRC | 05:10 | |
| openstack | Launchpad bug 1719123 in tripleo "tempest fails on overcloud keystone admin tenant error" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz) | 05:10 |
| openstack | Launchpad bug 1720220 in tripleo "CI: Most/all legacy-tripleo jobs failing on gate" [Critical,Triaged] | 05:10 |
| openstack | Launchpad bug 1720458 in tripleo "Lastest delorean pip package causes error in autodoc" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 05:10 |
| openstack | Launchpad bug 1720556 in tripleo "tracker: upstream zuul is not triggering periodic jobs" [Critical,Triaged] - Assigned to wes hayutin (weshayutin) | 05:10 |
| openstack | Launchpad bug 1720721 in tripleo "CI: OVB jobs fail because can't install XStatic from PyPI mirror on rh1 cloud" [Critical,Triaged] - Assigned to Paul Belanger (pabelanger) | 05:10 |
| *** rwsu has quit IRC | 05:10 | |
| *** yprokule has joined #tripleo | 05:17 | |
| *** iranzo has joined #tripleo | 05:18 | |
| *** dpawar has joined #tripleo | 05:19 | |
| *** stendulker has joined #tripleo | 05:21 | |
| *** pcaruana has quit IRC | 05:23 | |
| *** tongl has quit IRC | 05:24 | |
| *** skramaja has joined #tripleo | 05:25 | |
| *** marios has joined #tripleo | 05:27 | |
| *** rwsu has joined #tripleo | 05:33 | |
| *** oanson has quit IRC | 05:36 | |
| *** oanson has joined #tripleo | 05:36 | |
| *** psahoo has joined #tripleo | 05:37 | |
| *** achadha has joined #tripleo | 05:38 | |
| *** pgadiya has joined #tripleo | 05:40 | |
| *** achadha has quit IRC | 05:42 | |
| sshnaidm|afk | does anybody know about error of undercloud install? Error: Parameter proto failed on Firewall[001 accept all icmp ipv6]: Invalid value "ipv6-icmp". Valid values are ip, ! ip, tcp, | 05:42 |
| *** noslzzp has quit IRC | 05:43 | |
| jaosorior | no, but it seems like a puppet error. Maybe the firewall module was updated and has a bug? | 05:45 |
| sshnaidm|afk | yeah, looking in last patches.. | 05:45 |
| sshnaidm|afk | jaosorior, puppet-tripleo, right? | 05:45 |
| jaosorior | uhm | 05:46 |
| jaosorior | sshnaidm|afk: can I see the logs? | 05:46 |
| sshnaidm|afk | jaosorior, https://bugs.launchpad.net/tripleo/+bug/1720938 | 05:47 |
| openstack | Launchpad bug 1720938 in tripleo "CI: undercloud install fails with error: Parameter proto failed on Firewall[001 accept all icmp ipv6]" [Critical,Triaged] | 05:47 |
| jaosorior | sshnaidm|afk: I think it might even be from the package that actually provides the Firewall resource (which is upstream puppet) | 05:47 |
| jaosorior | oh | 05:47 |
| openstackgerrit | yatin proposed openstack/tripleo-quickstart-extras master: Use shell instead of command module as using &> operator https://review.openstack.org/509084 | 05:47 |
| jaosorior | well, we could use icmpv6 | 05:48 |
| *** nyechiel_ has joined #tripleo | 05:50 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo master: Use icmpv6 instead of ipv6-icmp https://review.openstack.org/509086 | 05:51 |
| jaosorior | sshnaidm|afk: maybe that should do the trick ^^ | 05:51 |
| jaosorior | sshnaidm|afk: gonna investigate it a bit more to have a better commit message | 05:51 |
| *** rcernin has joined #tripleo | 05:53 | |
| jaosorior | sshnaidm|afk: ok, the interface changed https://github.com/puppetlabs/puppetlabs-firewall/commit/41fb8205b68aec8a3683e1b4f53ec9de2246d504 :/ | 05:55 |
| sshnaidm|afk | jaosorior, thanks, will try your patch now | 05:56 |
| sshnaidm|afk | upstream puppet seems completely ignoring backward compatibility, stability etc | 05:56 |
| *** sshnaidm|afk is now known as sshnaidm | 05:56 | |
| jaosorior | sshnaidm: wouldn't be the first time :( | 05:57 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo master: Use icmpv6 instead of ipv6-icmp https://review.openstack.org/509086 | 05:57 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: WIP: Fix TripleO CI jobs https://review.openstack.org/508660 | 05:58 |
| *** agurenko has joined #tripleo | 06:01 | |
| sshnaidm | jaosorior, maybe we need to gate this changes from upstream anyhow, instead of just upgrading to last one | 06:01 |
| *** janki has joined #tripleo | 06:02 | |
| *** janki has quit IRC | 06:02 | |
| *** janki has joined #tripleo | 06:03 | |
| jaosorior | mandre: around yet? | 06:08 |
| *** jprovazn has joined #tripleo | 06:08 | |
| *** ooolpbot has joined #tripleo | 06:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1718387 | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1719123 | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720220 | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720458 | 06:10 |
| openstack | Launchpad bug 1718387 in tripleo "ping test is periodically failing for the gate-tripleo-ci-centos-7-nonha-multinode-oooq " [Critical,Triaged] - Assigned to Sofer Athlan-Guyot (sofer-athlan-guyot) | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720556 | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720721 | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720938 | 06:10 |
| *** ooolpbot has quit IRC | 06:10 | |
| openstack | Launchpad bug 1719123 in tripleo "tempest fails on overcloud keystone admin tenant error" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz) | 06:10 |
| openstack | Launchpad bug 1720220 in tripleo "CI: Most/all legacy-tripleo jobs failing on gate" [Critical,Triaged] | 06:10 |
| openstack | Launchpad bug 1720458 in tripleo "Lastest delorean pip package causes error in autodoc" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 06:10 |
| openstack | Launchpad bug 1720556 in tripleo "tracker: upstream zuul is not triggering periodic jobs" [Critical,Triaged] - Assigned to wes hayutin (weshayutin) | 06:10 |
| openstack | Launchpad bug 1720721 in tripleo "CI: OVB jobs fail because can't install XStatic from PyPI mirror on rh1 cloud" [Critical,Triaged] - Assigned to Paul Belanger (pabelanger) | 06:10 |
| openstack | Launchpad bug 1720938 in tripleo "CI: undercloud install fails with error: Parameter proto failed on Firewall[001 accept all icmp ipv6]" [Critical,In progress] - Assigned to Juan Antonio Osorio Robles (juan-osorio-robles) | 06:10 |
| *** pgadiya has quit IRC | 06:14 | |
| *** dparkes has joined #tripleo | 06:17 | |
| *** ramishra has quit IRC | 06:20 | |
| openstackgerrit | yatin proposed openstack/tripleo-quickstart-extras master: Add reference of image for pike release in rdocloud env https://review.openstack.org/509090 | 06:21 |
| *** ramishra has joined #tripleo | 06:22 | |
| mandre | jaosorior: here now, good morning | 06:23 |
| jaosorior | mandre: sup dude. haha I actually found the answer I needed faster than I thought :O | 06:24 |
| mandre | jaosorior: nice :) | 06:26 |
| *** fzdarsky_ is now known as fzdarsky | 06:26 | |
| *** jtomasek has joined #tripleo | 06:30 | |
| *** cylopez has joined #tripleo | 06:40 | |
| *** pcaruana has joined #tripleo | 06:44 | |
| *** pmannidi has quit IRC | 06:45 | |
| *** cylopez has quit IRC | 06:45 | |
| *** cylopez has joined #tripleo | 06:46 | |
| jaosorior | mandre: what is /usr/local/bin/kolla_set_configs for? | 06:47 |
| *** dciabrin has joined #tripleo | 06:49 | |
| *** dmacpher has quit IRC | 06:55 | |
| *** jlinkes has joined #tripleo | 06:56 | |
| jaosorior | sshnaidm: lol, lets keep shaming that commit :P | 06:58 |
| sshnaidm | jaosorior, yeah, more blaming! | 06:59 |
| sshnaidm | jaosorior, just found the quote in net: "The silent reproach of a million tear-stained eyes" | 06:59 |
| jaosorior | sshnaidm: https://youtu.be/EA6H_2mJKRE?t=1m29s | 07:00 |
| sshnaidm | :D | 07:04 |
| *** jfrancoa has joined #tripleo | 07:06 | |
| bandini | jaosorior: https://bugs.launchpad.net/tripleo/+bug/1720938 is a duplicate of https://bugs.launchpad.net/bugs/1720918 (FYI in case you had not spotted it) | 07:06 |
| openstack | Launchpad bug 1720938 in tripleo "CI: undercloud install fails with error: Parameter proto failed on Firewall[001 accept all icmp ipv6]" [Critical,In progress] - Assigned to Juan Antonio Osorio Robles (juan-osorio-robles) | 07:06 |
| openstack | Launchpad bug 1720918 in tripleo "puppet-firewall changed the ipv6-icmp rule type name" [Critical,In progress] - Assigned to Ben Nemec (bnemec) | 07:06 |
| *** ebarrera has joined #tripleo | 07:08 | |
| *** ooolpbot has joined #tripleo | 07:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1718387 | 07:10 |
| openstack | Launchpad bug 1718387 in tripleo "ping test is periodically failing for the gate-tripleo-ci-centos-7-nonha-multinode-oooq " [Critical,Triaged] - Assigned to Sofer Athlan-Guyot (sofer-athlan-guyot) | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1719123 | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720220 | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720458 | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720556 | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720721 | 07:10 |
| openstack | Launchpad bug 1719123 in tripleo "tempest fails on overcloud keystone admin tenant error" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz) | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720938 | 07:10 |
| *** ooolpbot has quit IRC | 07:10 | |
| openstack | Launchpad bug 1720220 in tripleo "CI: Most/all legacy-tripleo jobs failing on gate" [Critical,Triaged] | 07:10 |
| openstack | Launchpad bug 1720458 in tripleo "Lastest delorean pip package causes error in autodoc" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 07:10 |
| openstack | Launchpad bug 1720556 in tripleo "tracker: upstream zuul is not triggering periodic jobs" [Critical,Triaged] - Assigned to wes hayutin (weshayutin) | 07:10 |
| openstack | Launchpad bug 1720721 in tripleo "CI: OVB jobs fail because can't install XStatic from PyPI mirror on rh1 cloud" [Critical,Triaged] - Assigned to Paul Belanger (pabelanger) | 07:10 |
| openstack | Launchpad bug 1720938 in tripleo "CI: undercloud install fails with error: Parameter proto failed on Firewall[001 accept all icmp ipv6]" [Critical,In progress] - Assigned to Juan Antonio Osorio Robles (juan-osorio-robles) | 07:10 |
| *** shardy has joined #tripleo | 07:11 | |
| mandre | jaosorior: it's the script that copies the files to the specified place in the container and sets ownership/permissions | 07:13 |
| *** migi_ is now known as migi | 07:13 | |
| jaosorior | bandini: didn't know :O | 07:13 |
| mandre | jaosorior: for example, /usr/local/bin/kolla_set_configs takes care of https://github.com/openstack/tripleo-heat-templates/blob/master/docker/services/aodh-api.yaml#L90-L98 | 07:14 |
| bandini | jaosorior: I'll mark it as a duplicate | 07:14 |
| bandini | :) | 07:14 |
| jaosorior | bandini: thanks | 07:16 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: WIP: Add option for keystone containers to log to stdout/stderr https://review.openstack.org/508517 | 07:17 |
| jaosorior | mandre: aaah! cool cool | 07:17 |
| *** tesseract has joined #tripleo | 07:17 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: WIP: Add option for keystone containers to log to stdout/stderr https://review.openstack.org/508517 | 07:21 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: WIP: Add option for keystone containers to log to stdout/stderr https://review.openstack.org/508517 | 07:22 |
| jaosorior | dciabrin: did you get a chance to talk about the rabbitmq logging issue I was seeing? | 07:24 |
| *** ebarrera has quit IRC | 07:25 | |
| *** ebarrera has joined #tripleo | 07:25 | |
| *** cylopez has quit IRC | 07:26 | |
| *** brault has joined #tripleo | 07:28 | |
| *** ffiore has joined #tripleo | 07:29 | |
| jaosorior | sshnaidm: soo... shit's pretty broken :/ http://logs.openstack.org/86/509086/2/check/legacy-puppet-tripleo-puppet-unit-4.8-centos-7/dd01909/job-output.txt.gz#_2017-10-03_06_10_02_867185 | 07:30 |
| sshnaidm | jaosorior, yeah... | 07:31 |
| *** gfidente has joined #tripleo | 07:36 | |
| *** jpena|off is now known as jpena | 07:39 | |
| shardy | d0ugal: Hey, good morning, are you around? | 07:45 |
| shardy | d0ugal: I've been chatting with pdeore about https://review.openstack.org/#/c/410255/ apparently this broke some glance things reference https://review.openstack.org/#/c/509079/ | 07:46 |
| *** brault has quit IRC | 07:46 | |
| shardy | just trying to figure out if reverting to OS_TENANT_NAME is the right thing to do here | 07:46 |
| *** brault has joined #tripleo | 07:46 | |
| *** sshnaidm is now known as sshnaidm|afk | 07:47 | |
| *** aufi has joined #tripleo | 07:48 | |
| *** egonzalez has joined #tripleo | 07:48 | |
| *** pdeore is now known as pdeore|lunch | 07:49 | |
| *** sshnaidm|afk has quit IRC | 07:52 | |
| *** ccamacho has joined #tripleo | 07:52 | |
| *** jpich has joined #tripleo | 07:54 | |
| *** etingof has quit IRC | 07:56 | |
| *** rwsu has quit IRC | 07:57 | |
| *** bandini has quit IRC | 07:57 | |
| *** aditya_ra has joined #tripleo | 07:58 | |
| *** bandini has joined #tripleo | 08:02 | |
| *** aditya_r has quit IRC | 08:02 | |
| dciabrin | jaosorior, hey, not yesterday, eck will come back online on later today (EST tz) | 08:02 |
| *** masco is now known as masco_afk | 08:03 | |
| jaosorior | ok | 08:04 |
| *** rwsu has joined #tripleo | 08:09 | |
| *** ooolpbot has joined #tripleo | 08:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1718387 | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1719123 | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720220 | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720458 | 08:10 |
| openstack | Launchpad bug 1718387 in tripleo "ping test is periodically failing for the gate-tripleo-ci-centos-7-nonha-multinode-oooq " [Critical,Triaged] - Assigned to Sofer Athlan-Guyot (sofer-athlan-guyot) | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720556 | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720721 | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720918 | 08:10 |
| *** ooolpbot has quit IRC | 08:10 | |
| openstack | Launchpad bug 1719123 in tripleo "tempest fails on overcloud keystone admin tenant error" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz) | 08:10 |
| openstack | Launchpad bug 1720220 in tripleo "CI: Most/all legacy-tripleo jobs failing on gate" [Critical,Triaged] | 08:10 |
| openstack | Launchpad bug 1720458 in tripleo "Lastest delorean pip package causes error in autodoc" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 08:10 |
| openstack | Launchpad bug 1720556 in tripleo "tracker: upstream zuul is not triggering periodic jobs" [Critical,Triaged] - Assigned to wes hayutin (weshayutin) | 08:10 |
| openstack | Launchpad bug 1720721 in tripleo "CI: OVB jobs fail because can't install XStatic from PyPI mirror on rh1 cloud" [Critical,Triaged] - Assigned to Paul Belanger (pabelanger) | 08:10 |
| openstack | Launchpad bug 1720918 in tripleo "puppet-firewall changed the ipv6-icmp rule type name" [Critical,In progress] - Assigned to Ben Nemec (bnemec) | 08:10 |
| *** athomas has joined #tripleo | 08:13 | |
| d0ugal | shardy: Hey - running a bit late today. Just looking at the patches. | 08:14 |
| shardy | d0ugal: np - I see jaosorior commented also - I don't have a newton env to test but just trying to figure out the safest option - I'm a bit wary of just removing OS_PROJECT_NAME after 10 months as it could break some other client | 08:15 |
| d0ugal | shardy: yeah, I am really confused why this has come up now :) | 08:16 |
| *** oanson has quit IRC | 08:21 | |
| *** oanson has joined #tripleo | 08:22 | |
| *** aditya_ra has quit IRC | 08:23 | |
| *** florianf has quit IRC | 08:24 | |
| *** apevec has joined #tripleo | 08:26 | |
| *** etingof has joined #tripleo | 08:26 | |
| *** lucas-afk is now known as lucasagomes | 08:30 | |
| *** dsneddon has quit IRC | 08:30 | |
| *** sshnaidm|afk has joined #tripleo | 08:32 | |
| *** sid1 has joined #tripleo | 08:33 | |
| *** mcornea has joined #tripleo | 08:38 | |
| *** ffiore has quit IRC | 08:40 | |
| *** sshnaidm|afk is now known as sshnaidm | 08:41 | |
| *** spectr has joined #tripleo | 08:41 | |
| *** ffiore has joined #tripleo | 08:42 | |
| *** derekh has joined #tripleo | 08:44 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: WIP: Fix TripleO CI jobs https://review.openstack.org/508660 | 08:46 |
| openstackgerrit | Martin Mágr proposed openstack/tripleo-heat-templates master: List all unhealthy containers https://review.openstack.org/509110 | 08:46 |
| *** pdeore|lunch is now known as pdeore | 08:50 | |
| *** tosky has joined #tripleo | 08:57 | |
| *** salmankhan has joined #tripleo | 08:58 | |
| openstackgerrit | Pranali Deore proposed openstack/tripleo-common stable/newton: Adding env variable OS_TENANT_NAME back in overcloudrc https://review.openstack.org/509079 | 09:00 |
| pdeore | shardy, d0ugal, I've updated the patch by adding both OS_PROJECT_NAME and OS_TENANT_NAME https://review.openstack.org/509079 | 09:02 |
| shardy | pdeore: ack, thanks - sounds like it may be safer provided that works for you | 09:03 |
| openstackgerrit | David Sariel proposed openstack/tripleo-validations master: Add containers_sanity_check https://review.openstack.org/483403 | 09:04 |
| pdeore | shardy, yeah this works and would not break any other client :) | 09:07 |
| shardy | pdeore: ack, need the bug reference, otherwise +2 | 09:09 |
| pdeore | shardy, sure will do that.. :) | 09:09 |
| shardy | pdeore: is it also worth adding a comment to explain why we have the duplicate variables? | 09:10 |
| *** ooolpbot has joined #tripleo | 09:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1718387 | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1719123 | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720220 | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720458 | 09:10 |
| openstack | Launchpad bug 1718387 in tripleo "ping test is periodically failing for the gate-tripleo-ci-centos-7-nonha-multinode-oooq " [Critical,Triaged] - Assigned to Sofer Athlan-Guyot (sofer-athlan-guyot) | 09:10 |
| shardy | otherwise someone may do a clean up in future and remove the tenant name | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720556 | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720721 | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720918 | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720973 | 09:10 |
| *** ooolpbot has quit IRC | 09:10 | |
| openstack | Launchpad bug 1719123 in tripleo "tempest fails on overcloud keystone admin tenant error" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz) | 09:10 |
| openstack | Launchpad bug 1720220 in tripleo "CI: Most/all legacy-tripleo jobs failing on gate" [Critical,Triaged] | 09:10 |
| openstack | Launchpad bug 1720458 in tripleo "Lastest delorean pip package causes error in autodoc" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 09:10 |
| openstack | Launchpad bug 1720556 in tripleo "tracker: upstream zuul is not triggering periodic jobs" [Critical,Triaged] - Assigned to wes hayutin (weshayutin) | 09:10 |
| openstack | Launchpad bug 1720721 in tripleo "CI: OVB jobs fail because can't install XStatic from PyPI mirror on rh1 cloud" [Critical,Triaged] - Assigned to Paul Belanger (pabelanger) | 09:10 |
| openstack | Launchpad bug 1720918 in tripleo "puppet-firewall changed the ipv6-icmp rule type name" [Critical,In progress] - Assigned to Ben Nemec (bnemec) | 09:10 |
| openstack | Launchpad bug 1720973 in tripleo "CI: infras cirros image has wrong permission" [Critical,Triaged] - Assigned to Paul Belanger (pabelanger) | 09:10 |
| pdeore | shardy, yes I'm gonna add the comment too .. | 09:11 |
| *** aditya_r has joined #tripleo | 09:11 | |
| openstackgerrit | Raoul Scarazzini proposed openstack/tripleo-quickstart-extras master: Add optional tcpdump to introspection https://review.openstack.org/509118 | 09:19 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-specs master: Queens - all nodes ansible-playbook upgrade workflow https://review.openstack.org/507620 | 09:25 |
| *** ebarrera has quit IRC | 09:27 | |
| *** ebarrera has joined #tripleo | 09:28 | |
| openstackgerrit | Pranali Deore proposed openstack/tripleo-common stable/newton: Adding env variable OS_TENANT_NAME back in overcloudrc https://review.openstack.org/509079 | 09:29 |
| *** suuuper has joined #tripleo | 09:35 | |
| *** rakhmerov has joined #tripleo | 09:37 | |
| jprovazn | gfidente, hi | 09:42 |
| *** apetrich has quit IRC | 09:42 | |
| gfidente | jprovazn hey | 09:43 |
| jprovazn | gfidente, I'd like to add pass a variable to ceph-ansible (from heat param), specifically, I'd like to add something like this: "- ceph_nfs_bind_addr: {get_param: [NetVipMap, {get_param: StorageNFSNetName}]}" into https://github.com/openstack/tripleo-heat-templates/blob/master/docker/services/ceph-ansible/ceph-base.yaml#L277 | 09:46 |
| gfidente | jprovazn auch | 09:46 |
| gfidente | we can't pull the binding ip address yet from tht | 09:46 |
| jprovazn | gfidente, and I was wondering what is the mechanism now how to bubble such composable network param down to this template? | 09:46 |
| gfidente | can you pass it a network to listen on instead of an IP ? | 09:47 |
| shardy | jprovazn: can you use the ServiceNetMap instead of a special StorageNFSNetName? | 09:47 |
| gfidente | yeah ^^ that will give you the network to bind on | 09:48 |
| jprovazn | shardy, gfidente: the goal here is to pass to ceph-ansible VIP of a composable network which nfs-ganesha should listen on, if I can get this VIP from ServiceNetMap I think I can use ServiceNetMap | 09:48 |
| gfidente | (assuming we add something like GaneshaNFS there) | 09:48 |
| jprovazn | gfidente, this WIP patch adds StorageNFS network https://review.openstack.org/#/c/471245/ but you can imagine there is any other existing composable network | 09:50 |
| gfidente | jprovazn is the plan still to deploy nfs with legs on two networks, 'storage' to be able to connect to the ceph cluster and another reachable by the guests? | 09:50 |
| jprovazn | gfidente, yes | 09:50 |
| gfidente | jprovazn so I think in ceph-ansible we should add ceph_nfs_address_block | 09:51 |
| gfidente | which would work like ceph_mon_address_block or _rgw_address_block | 09:51 |
| *** udesale has quit IRC | 09:51 | |
| gfidente | that is the subnet (eventually taken from ServiceNetMap) where you want the NFS frontend to be exposed | 09:51 |
| gfidente | and ceph-ansible picks up the IP on the local node | 09:51 |
| gfidente | on the given network | 09:51 |
| gfidente | the list of monitors instead is computed by ceph-ansible itself basing on the inventory and pasted into the ganesha config | 09:52 |
| gfidente | like it does for the clients already | 09:52 |
| jprovazn | "<gfidente> and ceph-ansible picks up the IP on the local node" - well, I *think* I need it to pick up the virtual IP (which will be set/managed by pacemaker) | 09:53 |
| jprovazn | gfidente, IOW I need to configure nfs-ganesha to listen on this specific VIP, I think using network or interface instead is not possible based on this doc: https://github.com/nfs-ganesha/nfs-ganesha/wiki/CoreParams but I'll check with ganesha guys | 09:54 |
| gfidente | so for the other services which use a vip we have haproxy | 09:55 |
| gfidente | looks like in this case you want the service to follow the VIP via pacemaker? | 09:55 |
| jprovazn | gfidente, yes, exactly | 09:56 |
| gfidente | but you need to tell ganesha what is the VIP | 09:56 |
| jprovazn | gfidente, I have these bits done, but after switching from configure ganesha with puppet to ceph-ansible, this VIP is problematic :) | 09:56 |
| jprovazn | gfidente, shardy: I guess I could fetch it from puppet hieradata in worst case | 09:57 |
| gfidente | hmmm so I was thinking to create the VIP from within the ganesha resource itself | 09:57 |
| gfidente | but I don't think we can because heat would create one for each instance in the resource group | 09:57 |
| jprovazn | gfidente, yes, I do exactly this | 09:57 |
| jprovazn | gfidente, but I need also pass this VIP to ceph-ansible as nfs-ganesha is configure (but not started) by it | 09:58 |
| gfidente | I guess the closest thing we have is redis | 09:58 |
| *** ykarel is now known as ykarel|afk | 10:01 | |
| *** sid1 has quit IRC | 10:03 | |
| gfidente | looks like it is dumped into hieradata https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/all-nodes-config.yaml#L205 | 10:03 |
| gfidente | created in overcloud.j2 | 10:03 |
| gfidente | but we have this generic structure ServiceData | 10:04 |
| gfidente | I think you could create the VIP in overcloud.j2 | 10:04 |
| gfidente | append it into ServiceData | 10:04 |
| gfidente | and consume it within the ceph-ansible template for ganesha | 10:04 |
| shardy | I don't think we want to add any more special vips for specific services | 10:04 |
| gfidente | shardy ^^ ?? | 10:04 |
| shardy | it's kind of the opposite of all the composable patterns we've been implementing, isn't it? | 10:04 |
| jprovazn | gfidente, it's supposed to be vip on composable network | 10:05 |
| jprovazn | so I guesss I want to avoid hardcodeding dependency on it into overcloud | 10:05 |
| jprovazn | overcloud.yaml | 10:05 |
| gfidente | shardy jprovazn right so if we don't want per-service but per-network | 10:05 |
| gfidente | still, it's created in overcloud.j2 out of a composable network and appended into ServiceData | 10:05 |
| gfidente | maybe together with the other network VIPs | 10:05 |
| shardy | Most services reference the vips via the EndpointMap, is that an option here? | 10:05 |
| shardy | Or we can pass the name of the network in via config_settings, and the vip->IP mappings? | 10:06 |
| * shardy finds links | 10:06 | |
| jprovazn | shardy, maybe, I'm not sure how this EndpointMap works | 10:06 |
| shardy | https://github.com/openstack/tripleo-heat-templates/blob/master/overcloud.j2.yaml#L679 | 10:07 |
| shardy | https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/all-nodes-config.yaml#L170 | 10:07 |
| shardy | so we already calculate the vip for every enabled service, and add that to the hieradata, so I wonder if we can just use that in ansible? | 10:08 |
| shardy | gfidente: does allNodesConfig get added to the merged config_settings? | 10:08 |
| * shardy checks | 10:08 | |
| *** yamahata has quit IRC | 10:08 | |
| gfidente | shardy but why would we dump anyting into hiera | 10:08 |
| gfidente | it was necessary to resolve local IPs but not VIPs | 10:09 |
| shardy | gfidente: I'm saying the service -> VIP IP mapping already exists in hiera | 10:09 |
| shardy | so you just need to pass it to ansible | 10:09 |
| gfidente | if we add the service into EndpointMap like mysql it can be resolved within the ceph-ansible template | 10:09 |
| *** ooolpbot has joined #tripleo | 10:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1718387 | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1719123 | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720220 | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720458 | 10:10 |
| openstack | Launchpad bug 1718387 in tripleo "ping test is periodically failing for the gate-tripleo-ci-centos-7-nonha-multinode-oooq " [Critical,Triaged] - Assigned to Sofer Athlan-Guyot (sofer-athlan-guyot) | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720556 | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720721 | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720918 | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720973 | 10:10 |
| *** ooolpbot has quit IRC | 10:10 | |
| openstack | Launchpad bug 1719123 in tripleo "tempest fails on overcloud keystone admin tenant error" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz) | 10:10 |
| openstack | Launchpad bug 1720220 in tripleo "CI: Most/all legacy-tripleo jobs failing on gate" [Critical,Triaged] | 10:10 |
| openstack | Launchpad bug 1720458 in tripleo "Lastest delorean pip package causes error in autodoc" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 10:10 |
| openstack | Launchpad bug 1720556 in tripleo "tracker: upstream zuul is not triggering periodic jobs" [Critical,Triaged] - Assigned to wes hayutin (weshayutin) | 10:10 |
| openstack | Launchpad bug 1720721 in tripleo "CI: OVB jobs fail because can't install XStatic from PyPI mirror on rh1 cloud" [Critical,Triaged] - Assigned to Paul Belanger (pabelanger) | 10:10 |
| openstack | Launchpad bug 1720918 in tripleo "puppet-firewall changed the ipv6-icmp rule type name" [Critical,In progress] - Assigned to Ben Nemec (bnemec) | 10:10 |
| openstack | Launchpad bug 1720973 in tripleo "CI: infras cirros image has wrong permission" [Critical,Triaged] - Assigned to Paul Belanger (pabelanger) | 10:10 |
| shardy | jprovazn, gfidente: check /etc/puppet/hieradata/vip_data.json - that's what you need right? | 10:10 |
| gfidente | shardy meh again why hieradata | 10:11 |
| gfidente | for rgw we're just getting it from endpoint https://github.com/openstack/tripleo-heat-templates/blob/master/docker/services/ceph-ansible/ceph-rgw.yaml#L77 | 10:11 |
| shardy | gfidente: yeah I already said EndpointMap is an option | 10:12 |
| jprovazn | shardy, it's not problem to get this value in hieradata - I do this already, the problem is I don't know how to pass this to ceph-ansible too | 10:12 |
| * gfidente likes endpointmap more | 10:12 | |
| jprovazn | cool, so I could create a pseudo endpoint entry | 10:12 |
| shardy | I'm talking about hieradata because you pass the hieradata keys into the workflow, and earlier you started talking about making template changes which I don't think is needed, we already have the service -> vip mapping in several places | 10:12 |
| gfidente | there should be one for mysql | 10:12 |
| jprovazn | I wonder if it has some side-effects (e.g. this endpoint is added into keystone)? | 10:13 |
| jprovazn | shardy, aha, I think I didn't know about this bit: "hieradata keys into the workflow" | 10:13 |
| gfidente | jprovazn yes all hieradata are into the execution environment | 10:14 |
| *** sshnaidm is now known as sshnaidm|lnch | 10:14 | |
| jprovazn | shardy, so I thought I should somehow get this param passed to ceph-ansible template first | 10:14 |
| gfidente | but I don't like resorting to querying hieradata when we can push same setting into the template | 10:14 |
| shardy | jprovazn: yeah well gfidente added https://github.com/openstack/tripleo-heat-templates/blob/master/overcloud.j2.yaml#L414 but I don't think it yet considers the AllNodesConfig only the service config_settings | 10:14 |
| jprovazn | gfidente, cool, so I think I don't have any problem except how to get this hieradata in tripleo-common workbook, which I can probably figure out | 10:14 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Nova] https://review.openstack.org/499108 | 10:14 |
| shardy | gfidente: yeah I'm not saying query hieradata, only that you can wire the same data in via the templates | 10:15 |
| shardy | e.g make AllNodesConfig available to the workflow | 10:15 |
| jprovazn | gfidente, shardy: oh, so allNodesConfig can't be accessible from the ceph workbook? | 10:16 |
| gfidente | jprovazn yeah it can't because it's not appended | 10:16 |
| *** iranzo has quit IRC | 10:16 | |
| gfidente | which I think is what shardy is suggesting to do | 10:16 |
| shardy | yeah but we could probably add it | 10:16 |
| shardy | yup, seems like a fairly clean solution, and may be useful for other things? | 10:16 |
| *** sid1 has joined #tripleo | 10:17 | |
| shardy | then the ansible/workflow stuff gets the exact same data we use for the heat+puppet architecture | 10:17 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Octavia] https://review.openstack.org/502038 | 10:17 |
| gfidente | shardy so if we append the data into the execution env | 10:19 |
| shardy | skramaja: hey, when you have time I've got a question re the code that detects deprecated parameters in tripleo-common | 10:19 |
| shardy | https://github.com/openstack/tripleo-common/blob/master/workbooks/plan_management.yaml#L448 | 10:19 |
| gfidente | the ceph-ansible param should be resolved within the workflow though | 10:19 |
| skramaja | yes shardy | 10:19 |
| shardy | skramaja: am I reading that correctly that it only considers the parameters from the plan_environment parameter_defaults section? | 10:20 |
| gfidente | if we add the entry in endpointmap instead it can be resolved within the service template | 10:20 |
| gfidente | I mean, maybe we should still append allnodesconfig into execution environment | 10:20 |
| shardy | skramaja: I hit an error because I'm moving to putting the parametes into individual environment files, but wanted to check I wasn't missing something before trying to fix it | 10:20 |
| gfidente | but to set the ceph-ansible param wouldn't endpointmap still be nicer? :D | 10:20 |
| shardy | https://github.com/openstack/tripleo-common/blob/master/tripleo_common/actions/parameters.py#L69 | 10:21 |
| shardy | skramaja: seems that's where we get the params, so it relies on the tripleoclient merging | 10:21 |
| skramaja | shardy: we get the list of parameters from the parameter_defaults and compare it with the template parameters, to figure out if there is any unused parameter. | 10:22 |
| *** sid1 has quit IRC | 10:23 | |
| shardy | skramaja: Ok, the issue is ref bug #1635409 we shouldn't be merging all the environments together in tripleoclient, which this approach seems to require | 10:23 |
| openstack | bug 1635409 in tripleo "parameter_merge_strategies: merge behaves like overwrite" [High,In progress] https://launchpad.net/bugs/1635409 - Assigned to Steven Hardy (shardy) | 10:23 |
| openstackgerrit | Oliver Walsh proposed openstack/tripleo-heat-templates master: Fix race in cell_v2 discovery with containers https://review.openstack.org/508930 | 10:23 |
| shardy | skramaja: instead we probably need to iterate over each environment in the plan environment environments list | 10:23 |
| shardy | and deprecate the parameter_defaults section of the plan_environment | 10:23 |
| openstackgerrit | Oliver Walsh proposed openstack/tripleo-heat-templates master: Fix race in cell_v2 discovery with containers https://review.openstack.org/508930 | 10:23 |
| *** suuuper has quit IRC | 10:23 | |
| skramaja | yes understood. we need to modify it. | 10:25 |
| shardy | skramaja: Ok thanks for confirming, I'll take a look and see if I can adjust it | 10:25 |
| skramaja | shardy: and we need to do int only for the user-environemnts.. | 10:26 |
| shardy | Fixing that bug has been really hard because so many places expect the client-side merging :( | 10:26 |
| shardy | skramaja: well we could also warn if some environments/foo.yaml has deprecated parameters? | 10:26 |
| shardy | I think we could just iterate over the entire list, not only user-environments? | 10:27 |
| skramaja | shardy: *unused parameters.. | 10:27 |
| skramaja | shardy: yes for both.. | 10:27 |
| shardy | skramaja: Ok I'll give it a try, thanks! | 10:27 |
| skramaja | sure shardy let me know if any thing required from me.. | 10:29 |
| *** brault has quit IRC | 10:29 | |
| *** jkilpatr has quit IRC | 10:34 | |
| *** oanson has quit IRC | 10:38 | |
| *** oanson has joined #tripleo | 10:40 | |
| *** agurenko has quit IRC | 10:41 | |
| *** sshnaidm|lnch has quit IRC | 10:43 | |
| openstackgerrit | Oliver Walsh proposed openstack/tripleo-heat-templates master: Fix race in cell_v2 discovery with containers https://review.openstack.org/508930 | 10:45 |
| *** cshastri has quit IRC | 10:50 | |
| *** ffiore has quit IRC | 10:51 | |
| *** ffiore has joined #tripleo | 10:51 | |
| openstackgerrit | Oliver Walsh proposed openstack/tripleo-heat-templates master: Fix race in cell_v2 discovery with containers https://review.openstack.org/508930 | 10:52 |
| *** ansmith has joined #tripleo | 10:53 | |
| *** cdearborn has joined #tripleo | 10:53 | |
| *** agurenko has joined #tripleo | 10:53 | |
| jaosorior | owalsh: does nova-conductor need to start in step_4? another option would be to move it to step 3 | 10:53 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates master: Add allNodesConfig into role_merged_configs https://review.openstack.org/509146 | 10:56 |
| gfidente | shardy I think ^^ is what you mentioned | 10:57 |
| owalsh | jaosorior: don't think so. db_syncs are in step 3, nova api etc... services start in step 4. I could maybe move the api services to the end of step 3 but I think they would not be composible if I did | 10:58 |
| *** ansmith has quit IRC | 10:58 | |
| owalsh | jaosorior: ... well... maybe it would work, but I expect it would make things even more racy as services fail and retry until everything gets in sync | 10:59 |
| *** achadha has joined #tripleo | 10:59 | |
| owalsh | jaosorior; but yea, adding a 6th step isn't ideal... patch set 3 avoids it - https://review.openstack.org/#/c/508930/3, but I assume running multiple concurrent discover hosts is racy too | 11:00 |
| owalsh | jaosorior: between https://github.com/openstack/nova/blob/master/nova/objects/host_mapping.py#L185 and the save() call | 11:01 |
| *** amoralej is now known as amoralej|lunch | 11:02 | |
| *** jlinkes has quit IRC | 11:02 | |
| jaosorior | owalsh: well, the start_order of the db_syncs is usually 0. So, adding a high number would ideally make running nova-conductor on step 3 a non-issue? :D | 11:02 |
| owalsh | jaosorior: what if we split conductor and api out on to different nodes? | 11:03 |
| *** udesale has joined #tripleo | 11:03 | |
| owalsh | jaosorior: start order wouldn't apply AFAIK | 11:03 |
| *** jkilpatr has joined #tripleo | 11:03 | |
| *** achadha has quit IRC | 11:03 | |
| owalsh | jaosorior: that's what I meant by not being composible | 11:04 |
| jaosorior | oh right, and db_sync doesn't actually run on the conductor node iehter | 11:04 |
| jaosorior | yeah | 11:04 |
| jaosorior | well shit | 11:04 |
| owalsh | yup... other option I was thinking of is patch set 3, but ignore exit code 1 from discover hosts | 11:04 |
| jaosorior | uhm... | 11:06 |
| jaosorior | damn dude, not an easy one | 11:06 |
| *** jlinkes has joined #tripleo | 11:06 | |
| owalsh | jaosorior: yea, step_6 looks like the least evil option to me | 11:06 |
| *** spectr has quit IRC | 11:07 | |
| jaosorior | owalsh: I see what you mean now | 11:08 |
| *** ooolpbot has joined #tripleo | 11:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1718387 | 11:10 |
| openstack | Launchpad bug 1718387 in tripleo "ping test is periodically failing for the gate-tripleo-ci-centos-7-nonha-multinode-oooq " [Critical,Triaged] - Assigned to Sofer Athlan-Guyot (sofer-athlan-guyot) | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1719123 | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720220 | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720458 | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720556 | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720721 | 11:10 |
| openstack | Launchpad bug 1719123 in tripleo "tempest fails on overcloud keystone admin tenant error" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz) | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720918 | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720973 | 11:10 |
| *** ooolpbot has quit IRC | 11:10 | |
| openstack | Launchpad bug 1720220 in tripleo "CI: Most/all legacy-tripleo jobs failing on gate" [Critical,Triaged] | 11:10 |
| openstack | Launchpad bug 1720458 in tripleo "Lastest delorean pip package causes error in autodoc" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 11:10 |
| openstack | Launchpad bug 1720556 in tripleo "tracker: upstream zuul is not triggering periodic jobs" [Critical,Triaged] - Assigned to wes hayutin (weshayutin) | 11:10 |
| openstack | Launchpad bug 1720721 in tripleo "CI: OVB jobs fail because can't install XStatic from PyPI mirror on rh1 cloud" [Critical,Triaged] - Assigned to Paul Belanger (pabelanger) | 11:10 |
| openstack | Launchpad bug 1720918 in tripleo "puppet-firewall changed the ipv6-icmp rule type name" [Critical,In progress] - Assigned to Ben Nemec (bnemec) | 11:10 |
| openstack | Launchpad bug 1720973 in tripleo "CI: infras cirros image has wrong permission" [Critical,Triaged] - Assigned to Paul Belanger (pabelanger) | 11:10 |
| gfidente | jistr regarding https://review.openstack.org/#/c/509001/1/workbooks/access.yaml | 11:12 |
| gfidente | do I understand correctly we can use wait_for_connection outside the list of ansible tasks? | 11:12 |
| *** aditya_ra has joined #tripleo | 11:12 | |
| owalsh | jaosorior: still waiting on a few local runs to confirm that the timing is ok - we need a few seconds for the computes to register before running discover hosts so it's always going to a potential race | 11:12 |
| owalsh | s/going to a/going to be a/ | 11:14 |
| *** aditya_r has quit IRC | 11:15 | |
| *** aditya_ra has quit IRC | 11:16 | |
| *** aditya_r has joined #tripleo | 11:16 | |
| *** lucasagomes is now known as lucas-hungry | 11:16 | |
| *** pkovar has joined #tripleo | 11:17 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-common master: Ensure ssh key is active before returning from create_admin_via_nova https://review.openstack.org/509001 | 11:17 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-docs master: Update documentation for O to P upgrade and update https://review.openstack.org/496223 | 11:21 |
| *** stendulker_ has joined #tripleo | 11:24 | |
| *** aditya_r has quit IRC | 11:25 | |
| *** jkilpatr has quit IRC | 11:25 | |
| *** aditya_r has joined #tripleo | 11:25 | |
| *** ykarel|afk is now known as ykarel | 11:26 | |
| *** sshnaidm|lnch has joined #tripleo | 11:26 | |
| *** pradk has joined #tripleo | 11:26 | |
| *** stendulker has quit IRC | 11:27 | |
| jistr | gfidente: no i think it needs to be in tasks | 11:27 |
| jistr | it's a normal module | 11:27 |
| gfidente | jistr so I put it in the tasks in the update | 11:27 |
| gfidente | though the example in ansible docs had it outside too | 11:27 |
| jistr | gfidente: i think you can get rid of the delay -- it could slow things down needlessly? but LGTM | 11:28 |
| *** pkovar has quit IRC | 11:29 | |
| *** sshnaidm|lnch is now known as sshnaidm | 11:29 | |
| gfidente | jistr but I think occ is polling every 30 secs? | 11:29 |
| *** stendulker_ has quit IRC | 11:30 | |
| *** brian83 has joined #tripleo | 11:30 | |
| *** bobh has joined #tripleo | 11:31 | |
| jistr | gfidente: yea but you might be lucky at least with some nodes. E.g. i think rook had intermittent success with ~15 nodes with no delays at all. | 11:32 |
| jistr | though there was some gap before ceph-ansible started executing there perhaps | 11:32 |
| *** suuuper has joined #tripleo | 11:32 | |
| rook | jistr: right, 16 compute nodes had 80% success rate... 26 compute nodes had 0% success rate. | 11:33 |
| jistr | gfidente: so IDK imo it's slightly better to do 3x5s of polling than 15 secs of unconditional waiting upfront, but it's not a big deal so i +2'd anyway. If you change it i'll +2 again, up to you though. | 11:33 |
| gfidente | jistr ok thanks | 11:33 |
| gfidente | sec on a call | 11:33 |
| *** ffiore has quit IRC | 11:34 | |
| *** shreshtha has quit IRC | 11:34 | |
| *** suuuper has quit IRC | 11:34 | |
| rook | jistr gfidente IMHO both areas need to double check connectivity - OOO before handing off to ceph-ansible, and ceph-ansible... We are too optimistic about the accessibility state of hosts. | 11:36 |
| *** yprokule_ has joined #tripleo | 11:38 | |
| *** mrunge has quit IRC | 11:39 | |
| *** yprokule has quit IRC | 11:39 | |
| *** yprokule_ is now known as yprokule | 11:39 | |
| *** mrunge has joined #tripleo | 11:40 | |
| *** jkilpatr has joined #tripleo | 11:40 | |
| *** thrash|g0ne is now known as thrash | 11:45 | |
| *** jkilpatr_ has joined #tripleo | 11:46 | |
| *** jkilpatr has quit IRC | 11:47 | |
| *** rbrady has joined #tripleo | 11:50 | |
| *** sid1 has joined #tripleo | 11:50 | |
| *** brault has joined #tripleo | 11:50 | |
| *** jkilpatr_ has quit IRC | 11:54 | |
| *** milan has joined #tripleo | 11:54 | |
| *** abishop has joined #tripleo | 11:56 | |
| *** morazi has joined #tripleo | 11:56 | |
| *** udesale has quit IRC | 11:57 | |
| *** artom has joined #tripleo | 11:57 | |
| *** chem has quit IRC | 11:59 | |
| *** apetrich has joined #tripleo | 12:00 | |
| *** liverpooler has joined #tripleo | 12:00 | |
| *** liverpooler has quit IRC | 12:00 | |
| *** chem has joined #tripleo | 12:00 | |
| *** liverpooler has joined #tripleo | 12:01 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Add option for keystone containers to log to stdout/stderr https://review.openstack.org/508517 | 12:01 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Add option for nova containers to log to stdout/stderr https://review.openstack.org/509157 | 12:01 |
| *** raildo has joined #tripleo | 12:01 | |
| *** artom has quit IRC | 12:03 | |
| Tengu | jaosorior: hello! how are you? I was wondering if novajoin-server could add a new right in order to let the principal access the Vault service. That would allow a smooth integration with that service, in order to share certificates and other secrets :) | 12:03 |
| *** aditya_r has quit IRC | 12:03 | |
| *** aditya_r has joined #tripleo | 12:03 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-common master: Ensure ssh key is active before returning from create_admin_via_nova https://review.openstack.org/509001 | 12:03 |
| jaosorior | Tengu: you mean with IPA Vault? | 12:04 |
| *** jlinkes_ has joined #tripleo | 12:04 | |
| Tengu | jaosorior: yep :) | 12:05 |
| Tengu | as it's been included in freeipa 4.5 | 12:05 |
| *** jlinkes has quit IRC | 12:06 | |
| *** pdeore has quit IRC | 12:06 | |
| *** pblaho1 has joined #tripleo | 12:07 | |
| *** rcrit has joined #tripleo | 12:07 | |
| *** jkilpatr_ has joined #tripleo | 12:08 | |
| *** chem has quit IRC | 12:08 | |
| *** bfournie has quit IRC | 12:08 | |
| *** bfournie has joined #tripleo | 12:08 | |
| jaosorior | Tengu: rcrit could probably answer that better. | 12:09 |
| *** pblaho has quit IRC | 12:09 | |
| Tengu | jaosorior: I've opened an issue on novajoin for that, might be the place to discuss it :) | 12:09 |
| jaosorior | Tengu: can you pass the link? | 12:09 |
| lyarwood | I assume others are also getting a NET::ERR_CERT_DATE_INVALID error against https://images.rdoproject.org/ ? | 12:09 |
| *** pkovar has joined #tripleo | 12:09 | |
| Tengu | https://bugs.launchpad.net/novajoin/+bug/1721003 | 12:09 |
| openstack | Launchpad bug 1721003 in novajoin "Authorize nova created ipa account to access Vault" [Undecided,New] | 12:09 |
| *** jlinkes_ has quit IRC | 12:10 | |
| *** ooolpbot has joined #tripleo | 12:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1718387 | 12:10 |
| openstack | Launchpad bug 1718387 in tripleo "ping test is periodically failing for the gate-tripleo-ci-centos-7-nonha-multinode-oooq " [Critical,Triaged] - Assigned to Sofer Athlan-Guyot (sofer-athlan-guyot) | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1719123 | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720220 | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720458 | 12:10 |
| openstack | Launchpad bug 1719123 in tripleo "tempest fails on overcloud keystone admin tenant error" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz) | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720556 | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720721 | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720918 | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720973 | 12:10 |
| openstack | Launchpad bug 1720220 in tripleo "CI: Most/all legacy-tripleo jobs failing on gate" [Critical,Triaged] | 12:10 |
| *** ooolpbot has quit IRC | 12:10 | |
| openstack | Launchpad bug 1720458 in tripleo "Lastest delorean pip package causes error in autodoc" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 12:10 |
| openstack | Launchpad bug 1720556 in tripleo "tracker: upstream zuul is not triggering periodic jobs" [Critical,Triaged] - Assigned to wes hayutin (weshayutin) | 12:10 |
| openstack | Launchpad bug 1720721 in tripleo "CI: OVB jobs fail because can't install XStatic from PyPI mirror on rh1 cloud" [Critical,Triaged] - Assigned to Paul Belanger (pabelanger) | 12:10 |
| openstack | Launchpad bug 1720918 in tripleo "puppet-firewall changed the ipv6-icmp rule type name" [Critical,In progress] - Assigned to Ben Nemec (bnemec) | 12:10 |
| openstack | Launchpad bug 1720973 in tripleo "CI: infras cirros image has wrong permission" [Critical,Triaged] - Assigned to Paul Belanger (pabelanger) | 12:10 |
| Tengu | jaosorior: don't know if the title is correct though. I'm trying to understand how to use the libipa and ipaclient, reading novajoin code, as it's not really documented… It's not that easy. | 12:10 |
| *** raildo has quit IRC | 12:11 | |
| rcrit | ok, I"ll comment in LP | 12:11 |
| jaosorior | Tengu: can you expand on how your setup looks like? | 12:11 |
| *** raildo has joined #tripleo | 12:11 | |
| Tengu | rcrit: hello :). Thanks for comment :) | 12:11 |
| *** jlinkes has joined #tripleo | 12:11 | |
| *** bobh has quit IRC | 12:12 | |
| Tengu | jaosorior: I'm trying to embed IPA Vault access in certodia, and to use the nova keytab in order to access that. For now, the rights don't allow that, of course | 12:12 |
| *** bfournie has quit IRC | 12:13 | |
| Tengu | jaosorior: that would allow to get either a plain custodia, or, as we can already link hosts in IPA, use its internal vault. I think it's a pretty good thing, as it will also allow to get strong auth, encrypted connection and so on. | 12:13 |
| Tengu | and that would prevent the need to install a custodia if we already have a freeipa :) | 12:14 |
| eck` | jaosorior: i hear you have a rabbit log question | 12:14 |
| *** sid1 has quit IRC | 12:15 | |
| *** sid1 has joined #tripleo | 12:15 | |
| rook | jistr gfidente guess it isn't a good opinion to have :) | 12:15 |
| jistr | rook: haha what, the connectivity checks? :) | 12:15 |
| jaosorior | eck`: right, I want to get the rabbitmq container to log to stdout (at least the mean log, the rest of the logs can keep going to separate files). | 12:16 |
| rook | jistr: yup | 12:16 |
| jaosorior | eck`: tries setting up the RABBITMQ_LOGS environment variable to /dev/stdout; and that didn't seem to work out. | 12:16 |
| *** pblaho1 is now known as pblaho | 12:16 | |
| *** pblaho has quit IRC | 12:16 | |
| *** pblaho has joined #tripleo | 12:17 | |
| *** lucas-hungry is now known as lucasagomes | 12:17 | |
| eck` | jaosorior: hmm i've actually never tried this before | 12:17 |
| eck` | let me play around with it and see if i can get it to work | 12:18 |
| jaosorior | eck`: that would be great | 12:18 |
| *** jpena is now known as jpena|lunch | 12:19 | |
| jistr | rook: so the issue we hit was that the os-collect-config didn't create the tripleo-admin user & authorized key yet. Once we check for its completion in the enable_ssh_admin workflow, we should not hit this issue any time later. It still may be valid to e.g. test connectivity in ceph-ansible to e.g. spot whether the inventory contains existing hosts. (But i'm not too familiar w/ ceph-ansible personally.) As | 12:20 |
| jistr | for guarding against temporary network issues maybe we could try the retry parameter that ansible has? Validating won't help much with that as network issues may occur after validation too. | 12:20 |
| *** udesale has joined #tripleo | 12:22 | |
| gfidente | rook jistr like adding a SSH_RETRIES into ceph-ansible ? | 12:22 |
| *** dprince has joined #tripleo | 12:23 | |
| jistr | gfidente, rook: yea maybe this http://docs.ansible.com/ansible/latest/intro_configuration.html#retries | 12:23 |
| *** spectr has joined #tripleo | 12:24 | |
| gfidente | like 3 retries? | 12:25 |
| *** cylopez has joined #tripleo | 12:26 | |
| *** cylopez has quit IRC | 12:26 | |
| *** cylopez has joined #tripleo | 12:26 | |
| jistr | yea maybe something like that :) tbh in my personal usage i never needed this so i'm not sure what would be a good value. (I only deal with small environments though.) | 12:26 |
| *** anshul has joined #tripleo | 12:27 | |
| *** jlinkes_ has joined #tripleo | 12:27 | |
| *** amoralej|lunch is now known as amoralej | 12:28 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-common master: Ensure ssh key is active before returning from create_admin_via_nova https://review.openstack.org/509001 | 12:28 |
| gfidente | rook ^^ | 12:28 |
| openstackgerrit | Oliver Walsh proposed openstack/tripleo-heat-templates master: Fix race in cell_v2 discovery with containers https://review.openstack.org/508930 | 12:29 |
| *** jlinkes has quit IRC | 12:29 | |
| jistr | +2, ship it :D | 12:30 |
| *** abregman is now known as abregman|mtg | 12:30 | |
| *** masco_afk has quit IRC | 12:31 | |
| *** atoth has quit IRC | 12:33 | |
| gfidente | ahahaha | 12:33 |
| gfidente | zuul | 12:34 |
| eck` | jaosorior: here you go... | 12:34 |
| eck` | # If $RABBITMQ_LOGS is '-', send all log messages to stdout. This is | 12:34 |
| eck` | # particularly useful for Docker images. | 12:34 |
| jaosorior | eck`: yep, that's what I tried | 12:34 |
| jaosorior | wait | 12:34 |
| jaosorior | so I tried that | 12:34 |
| jaosorior | using | 12:34 |
| jaosorior | '-' | 12:34 |
| jaosorior | and when that didn't work out | 12:34 |
| *** skramaja has quit IRC | 12:34 | |
| jaosorior | I tried setting that to /dev/stdout, and even tried /dev/stderr | 12:34 |
| eck` | hmm | 12:35 |
| eck` | maybe this is a newer feature that we don't have in rdo | 12:35 |
| * eck` looks | 12:35 | |
| jaosorior | eck`: that's why I was asking dciabrin if the resource agent that starts rabbitmq redirects the output or something of the sort. | 12:35 |
| *** aufi has quit IRC | 12:36 | |
| *** catintheroof has joined #tripleo | 12:36 | |
| *** pblaho has quit IRC | 12:36 | |
| *** catintheroof has quit IRC | 12:36 | |
| *** catintheroof has joined #tripleo | 12:36 | |
| eck` | jaosorior: ohh yeah it does | 12:37 |
| eck` | jaosorior: it goes to /var/log/rabbitmq/startup_log and startup_err | 12:37 |
| eck` | so maybe it does work but it's landing in those files | 12:37 |
| jaosorior | right | 12:38 |
| jaosorior | eck`: where is that happening? | 12:38 |
| jaosorior | eck`: would be great to get an option to disable that redirection :D | 12:38 |
| *** gkadam has quit IRC | 12:38 | |
| *** rhallisey has joined #tripleo | 12:39 | |
| *** rhallisey has quit IRC | 12:39 | |
| *** rhallisey has joined #tripleo | 12:40 | |
| eck` | https://github.com/ClusterLabs/resource-agents/blob/master/heartbeat/rabbitmq-cluster#L209 | 12:40 |
| *** leitan has joined #tripleo | 12:40 | |
| *** rlandy has joined #tripleo | 12:41 | |
| *** catintheroof has quit IRC | 12:41 | |
| eck` | i'll put in a PR to add an option to disable that | 12:41 |
| jaosorior | eck`: thanks! | 12:41 |
| *** ansmith has joined #tripleo | 12:41 | |
| *** paramite_ has quit IRC | 12:42 | |
| eck` | although... | 12:42 |
| jaosorior | O_O | 12:42 |
| eck` | i'm not sure it will actually end up in the docker logs still | 12:42 |
| jaosorior | why not? | 12:43 |
| eck` | because pacemaker-remote is pid1 inside the container | 12:43 |
| openstackgerrit | Ryan Brady proposed openstack/tripleo-common master: Enhances Roles List https://review.openstack.org/508514 | 12:43 |
| eck` | and then it forks off a separate process to handle reaping orphan child processes | 12:43 |
| jaosorior | uhm... well, if it all goes to stderr it would end up in the logs, no? | 12:43 |
| jaosorior | aww shit | 12:43 |
| eck` | and then under that is the actual rabbit process | 12:44 |
| eck` | so i don't know if the rabbit process stdout/stderr inherits the same fd from pacemaker-remote | 12:44 |
| eck` | if it does, then it would probably work... if not, then no | 12:44 |
| eck` | jaosorior: if you have an environment handy, you can just modify the resource agent by hand to remote the redirection and see if it works | 12:45 |
| *** ecerquei has joined #tripleo | 12:45 | |
| jaosorior | let me try that | 12:45 |
| eck` | i don't have a containerized environment right now so i would have to build one first :) | 12:45 |
| jaosorior | eck`: do I have to reload pacemaker somehow after removing the redirection? | 12:46 |
| *** bfournie has joined #tripleo | 12:46 | |
| eck` | jaosorior: no it should pick it up automatically | 12:46 |
| *** jlinkes_ has quit IRC | 12:46 | |
| *** jcoufal has joined #tripleo | 12:46 | |
| jaosorior | eck`: where was the resource agent located again? | 12:46 |
| *** atoth has joined #tripleo | 12:47 | |
| eck` | jaosorior: /usr/lib/ocf/resource.d/heartbeat/rabbitmq-cluster | 12:47 |
| *** mdnadeem has quit IRC | 12:48 | |
| *** cylopez has quit IRC | 12:48 | |
| *** pblaho has joined #tripleo | 12:48 | |
| *** aufi has joined #tripleo | 12:49 | |
| *** toure_biab is now known as toure | 12:49 | |
| *** deadnull has joined #tripleo | 12:49 | |
| *** panda|bbl is now known as panda | 12:49 | |
| *** pchavva has joined #tripleo | 12:50 | |
| *** jmelvin has joined #tripleo | 12:51 | |
| *** udesale has quit IRC | 12:52 | |
| *** trown|outtypewww is now known as trown | 12:54 | |
| *** catintheroof has joined #tripleo | 12:54 | |
| *** paramite has joined #tripleo | 12:54 | |
| therve | Hi | 12:55 |
| *** pdeore has joined #tripleo | 12:55 | |
| therve | Is it supported to run undercloud install several times in the same host? | 12:55 |
| shardy | therve: Hey, yes but I ran into some race/idempotency issues when I was testing like that | 12:55 |
| trown | therve: yes it is supposed to be idempotent | 12:55 |
| trown | therve: if not it is a bug | 12:55 |
| shardy | we fixed things so it should work, but re-running isn't well tested at this point | 12:55 |
| shardy | so yeah what trown said :) | 12:56 |
| therve | OK :) | 12:56 |
| therve | We'll it's ocata, so not sure how much I should pursue it | 12:56 |
| trown | but ya wouldnt doubt that could have bugs... because we dont have a test for it | 12:56 |
| therve | Seems that puppet-mongodb is misbehaving or something | 12:56 |
| trown | though I thought the undercloud only job did two installs | 12:56 |
| *** ratailor has quit IRC | 12:57 | |
| shardy | trown: We did have an undercloud idempotency test but I'm not sure if the undercloud-containers job does it | 12:57 |
| *** ffiore has joined #tripleo | 12:57 | |
| shardy | Oh wait | 12:57 |
| shardy | If we're talking about the normal puppet undercloud install then you're right trown | 12:58 |
| shardy | for some reason I read therve's question as about the container based undercloud deploy | 12:58 |
| * shardy needs coffee | 12:58 | |
| therve | Yeah that's the puppet one I'm talking about | 12:58 |
| shardy | Ah, yeah that definitely should work | 12:59 |
| *** tzumainn has joined #tripleo | 13:00 | |
| *** Goneri has joined #tripleo | 13:01 | |
| *** jmelvin has quit IRC | 13:02 | |
| *** jmelvin has joined #tripleo | 13:02 | |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-common master: Fixup upgrade-non-controller to also accept IP address https://review.openstack.org/505624 | 13:06 |
| marios | jistr: when you next get a chance please can you check ^ https://review.openstack.org/#/c/505624/4..5/scripts/upgrade-non-controller.sh | 13:07 |
| marios | jistr: oh, i just saw your updated comment, never mind reading that :) | 13:08 |
| *** mdnadeem has joined #tripleo | 13:08 | |
| *** achadha has joined #tripleo | 13:08 | |
| jaosorior | dciabrin, bandini: You guys around? | 13:09 |
| eck` | (run, hide, scream...) | 13:09 |
| jaosorior | eck`: it's tripleo related, those are part of the everyday work :D | 13:09 |
| trown | sshnaidm: do you know if there is a good way to see what remaining zuulv3 issues are being worked on by infra team (other than scrolling through #openstack-infra)? | 13:09 |
| jistr | marios: was just thinking to get rid of the evals because if `hostname="trololo; rm -rf whatever"` then the eval won't be friendly. But given that it's just operator-facing and not user facing this might not be a big deal. | 13:10 |
| *** ooolpbot has joined #tripleo | 13:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1718387 | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1719123 | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720220 | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720458 | 13:10 |
| openstack | Launchpad bug 1718387 in tripleo "ping test is periodically failing for the gate-tripleo-ci-centos-7-nonha-multinode-oooq " [Critical,Triaged] - Assigned to Sofer Athlan-Guyot (sofer-athlan-guyot) | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720556 | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720721 | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720918 | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720973 | 13:10 |
| *** ooolpbot has quit IRC | 13:10 | |
| openstack | Launchpad bug 1719123 in tripleo "tempest fails on overcloud keystone admin tenant error" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz) | 13:10 |
| openstack | Launchpad bug 1720220 in tripleo "CI: Most/all legacy-tripleo jobs failing on gate" [Critical,Triaged] | 13:10 |
| openstack | Launchpad bug 1720458 in tripleo "Lastest delorean pip package causes error in autodoc" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 13:10 |
| openstack | Launchpad bug 1720556 in tripleo "tracker: upstream zuul is not triggering periodic jobs" [Critical,Triaged] - Assigned to wes hayutin (weshayutin) | 13:10 |
| openstack | Launchpad bug 1720721 in tripleo "CI: OVB jobs fail because can't install XStatic from PyPI mirror on rh1 cloud" [Critical,Triaged] - Assigned to Paul Belanger (pabelanger) | 13:10 |
| openstack | Launchpad bug 1720918 in tripleo "puppet-firewall changed the ipv6-icmp rule type name" [Critical,In progress] - Assigned to Ben Nemec (bnemec) | 13:10 |
| openstack | Launchpad bug 1720973 in tripleo "CI: infras cirros image has wrong permission" [Critical,Triaged] - Assigned to Paul Belanger (pabelanger) | 13:10 |
| *** chlong has joined #tripleo | 13:10 | |
| *** chlong_ has joined #tripleo | 13:10 | |
| jistr | marios: +2'd fwiw | 13:11 |
| sshnaidm | trown, unfortunately no | 13:11 |
| *** achadha has quit IRC | 13:12 | |
| marios | jistr: ack, thanks. so i replied again... i'll quickly try the global params way too and if testing doesn't cause any obvious problems i'll post it | 13:12 |
| *** achadha has joined #tripleo | 13:12 | |
| marios | jistr: i used to do the 'print to return' way but we starting echo error/log messages and that broke down, esp when we want to assign 2 things in one function | 13:12 |
| jistr | marios: yea, had the same issue when working w/ bash | 13:13 |
| jaosorior | larsks, jbadiapa_: Got here keystone and nova logging (mostly) to stdout in their respective containers, with the extras (apache's access log) being logged to a fifo and picked up by a sidecar container https://review.openstack.org/#/q/topic:tripleo-logging+(status:open+OR+status:merged) | 13:13 |
| *** aufi has quit IRC | 13:13 | |
| jaosorior | larsks, jbadiapa_: better link https://review.openstack.org/#/q/topic:tripleo-logging+status:open | 13:14 |
| *** dsariel has quit IRC | 13:15 | |
| bandini | jaosorior: o/ | 13:15 |
| jaosorior | bandini: hey! soo.... I'm trying to get the rabbitmq container to log to stdout (so it gets picked up by docker logs). And, as eck` found out, it seems that pacemaker forks rabbitmq in such a way that it's not possible for it to output its logs to stdout (or stderr). Any way around it that you can think of? | 13:17 |
| *** deadnull has quit IRC | 13:17 | |
| * bandini looks | 13:17 | |
| eck` | bandini: i think somewhere along the way that pacemaker-remote forks off the "child-reaper" and then the actual target process, it doesn't clone the file descriptors | 13:18 |
| dmsimard | sshnaidm: how are things looking this morning, want to give me a tl;dr ? | 13:18 |
| eck` | but i haven't looked yet to verify that | 13:18 |
| eck` | the end result is that rabbit does correctly log to stdout, but its stdout is not the same stdout that docker is reading its logs from | 13:18 |
| bandini | eck`: oh so not necessarily in the RA, but in pcmk-remote pid 1? | 13:18 |
| *** liverpooler has quit IRC | 13:19 | |
| sshnaidm | dmsimard, well.. jobs finish now, but everything fails with lost connections to nodes | 13:19 |
| dmsimard | sshnaidm: is that from your review with fixes ? | 13:19 |
| eck` | bandini: i think so, yes | 13:19 |
| sshnaidm | dmsimard, in the morning some of jobs reached stage of tempest (wow), but failed because wrong permissions of image (fixed) | 13:19 |
| sshnaidm | dmsimard, yep | 13:19 |
| eck` | i had jaosorior modify the RA so it doesn't redirect to the files | 13:19 |
| eck` | the end result is zero log files, also zero output on the docker side | 13:20 |
| bandini | eck`: bummer, I was so keen on blaming it on the RA (aka on *you*) ;) | 13:20 |
| eck` | but if we log into the container and run `rabbitmq-server` manually, you see the logs on stdout | 13:20 |
| sshnaidm | dmsimard, so the ball is on infras side, I don't think we have something to add today to our code | 13:20 |
| sshnaidm | trown, ^ | 13:20 |
| sshnaidm | if nothing broke in last few hours of course | 13:20 |
| dmsimard | sshnaidm: I'm helping infra but need visibility on the issues :) | 13:20 |
| *** lblanchard has joined #tripleo | 13:21 | |
| *** mdnadeem has quit IRC | 13:21 | |
| dmsimard | sshnaidm: where is the image permission fix ? is that inside your review ? | 13:21 |
| sshnaidm | dmsimard, in last run it was all because lost connection | 13:21 |
| pabelanger | sshnaidm: looking at mirror now | 13:21 |
| *** bnemec has joined #tripleo | 13:21 | |
| sshnaidm | dmsimard, I have a workaround for it, and https://review.openstack.org/#/c/509116/ | 13:22 |
| *** cylopez has joined #tripleo | 13:23 | |
| sshnaidm | dmsimard, trown, and today in morning puppet firewall module upstream broke the interface and failed our jobs.. like it's not enough current issues :D | 13:23 |
| *** jlinkes has joined #tripleo | 13:23 | |
| *** aditya_r has quit IRC | 13:23 | |
| bandini | eck`: so https://github.com/ClusterLabs/pacemaker/commit/8abdd82ba85e ? | 13:23 |
| bandini | no that is the reaping only | 13:23 |
| sshnaidm | https://bugs.launchpad.net/tripleo/+bug/1720918 | 13:23 |
| openstack | Launchpad bug 1720918 in tripleo "puppet-firewall changed the ipv6-icmp rule type name" [Critical,In progress] - Assigned to Ben Nemec (bnemec) | 13:23 |
| trown | dmsimard: all the post_failure jobs here show the current connection lost issue though https://review.openstack.org/#/c/508660/ | 13:24 |
| dmsimard | sshnaidm: over 2h on those jobs... Timeout ? I'll dig in a few minutes | 13:24 |
| sshnaidm | dmsimard, yeah, they were stuck and reached jobs timeout | 13:24 |
| *** jpena|lunch is now known as jpena | 13:24 | |
| trown | sshnaidm: should we add in the firewall fix to the depends on in https://review.openstack.org/#/c/508660/ | 13:25 |
| trown | sshnaidm: I can do it | 13:25 |
| openstackgerrit | John Trowbridge proposed openstack-infra/tripleo-ci master: WIP: Fix TripleO CI jobs https://review.openstack.org/508660 | 13:26 |
| *** aufi has joined #tripleo | 13:26 | |
| trozet | shardy: hey wanted to continue discussion about the dynamic network stuff | 13:26 |
| *** apetrich has quit IRC | 13:26 | |
| sshnaidm | trown, it is | 13:27 |
| trown | sshnaidm: oh whoops there are two patches for it, and the one you linked is not attached to the bug... but it does have unit tests ... | 13:27 |
| openstackgerrit | John Trowbridge proposed openstack-infra/tripleo-ci master: WIP: Fix TripleO CI jobs https://review.openstack.org/508660 | 13:28 |
| *** apetrich has joined #tripleo | 13:28 | |
| trown | reverted my edit | 13:28 |
| sshnaidm | trown, I think they do the same.. | 13:29 |
| *** dsariel has joined #tripleo | 13:29 | |
| *** marrusl has joined #tripleo | 13:29 | |
| trown | sshnaidm: ya, just got confused because I saw ben's patch first | 13:30 |
| trown | sshnaidm: but the one you linked is better in that it has a spec test | 13:30 |
| openstackgerrit | Tim Rozet proposed openstack/tripleo-heat-templates master: Fixes missing type for heat param TenantNetName https://review.openstack.org/508920 | 13:31 |
| *** achadha has quit IRC | 13:32 | |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-common master: Fixup upgrade-non-controller to also accept IP address https://review.openstack.org/505624 | 13:32 |
| marios | jistr: thanks :) evals have been abolished ^ | 13:32 |
| marios | jistr: (you've spent a while on it already so tomorrow is fine, just fyi ;) thanks) | 13:33 |
| jistr | marios: yea that's fine, looking | 13:33 |
| EmilienM | hello | 13:34 |
| *** ykarel has quit IRC | 13:34 | |
| *** alee has joined #tripleo | 13:34 | |
| jistr | marios: nice, thanks! +2'd | 13:35 |
| jistr | hi EmilienM | 13:36 |
| marios | jistr: thanks, i spotted one thing and will update but it is very minor (line 137) so you can revote tomorrow/shouldnt need much review thanks man | 13:36 |
| marios | jistr: it doesn't break anything the nit i mean it is just 'nicer' | 13:36 |
| jistr | it's fine, i often check updates on reviews several times a day when waiting for deploy or something of that sort | 13:37 |
| EmilienM | sshnaidm: we could pin puppetlabs-firewall if needed (temporary) | 13:37 |
| jistr | unless i'm totally swamped, on those days i sometimes don't get to reviews at all :/ | 13:37 |
| sshnaidm | EmilienM, we have fixes (even 2) | 13:38 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-common master: Fixup upgrade-non-controller to also accept IP address https://review.openstack.org/505624 | 13:38 |
| sshnaidm | EmilienM, but seems like it's not first time we are broken by patches in upstream puppet | 13:38 |
| sshnaidm | EmilienM, need to think how to avoid such cases... | 13:39 |
| pabelanger | sshnaidm: okay, try now the pypi mirror for tripleo-test-cloud-rh1 | 13:39 |
| *** brault has quit IRC | 13:39 | |
| marios | thanks for the reviews then jistr. i like it better now +1 no more eval (though doing assignment like that in bash is pretty cool, it probably isn't very safe) | 13:39 |
| sshnaidm | pabelanger, jobs are in queue, let's see | 13:39 |
| *** brault has joined #tripleo | 13:39 | |
| sshnaidm | pabelanger, queue of check-tripleo is 91... | 13:40 |
| *** mdnadeem has joined #tripleo | 13:40 | |
| sshnaidm | next week we can check | 13:40 |
| pabelanger | the URLs in the bug are working now | 13:40 |
| pabelanger | I've also added what the fix was | 13:41 |
| sshnaidm | pabelanger, thanks! | 13:41 |
| pabelanger | now to understand why we had to do it | 13:41 |
| EmilienM | sshnaidm: yeah, amoralej proposed different solutions, he can talk more about it | 13:41 |
| *** aditya_r has joined #tripleo | 13:42 | |
| amoralej | EmilienM, in fact that passed gate jobs | 13:42 |
| amoralej | but we didn't cover firewall | 13:42 |
| *** cylopez has quit IRC | 13:43 | |
| *** rhallisey has quit IRC | 13:43 | |
| *** rhallisey has joined #tripleo | 13:43 | |
| amoralej | sshnaidm, do you want to pin firewall module? | 13:43 |
| *** brault has quit IRC | 13:44 | |
| *** aditya_r has quit IRC | 13:44 | |
| sshnaidm | amoralej, we have patches that should fix it, so I'm not sure we need to pin it.. | 13:44 |
| *** aditya_r has joined #tripleo | 13:44 | |
| sshnaidm | but no strong objections | 13:44 |
| amoralej | sshnaidm, yeah, i've seen the patch | 13:44 |
| amoralej | but gate is broken, right? | 13:44 |
| sshnaidm | amoralej, yeah | 13:44 |
| sshnaidm | we have 4 patches in pipeline, so plus/minus one.. | 13:45 |
| EmilienM | sshnaidm: yeah not sure it will merge anytime soon | 13:45 |
| EmilienM | amoralej: +1 for pin | 13:45 |
| EmilienM | we can unpin when things are calm | 13:45 |
| amoralej | EmilienM, gimme a while, i'm triaging issue in heat with ipv6 | 13:46 |
| amoralej | do you know if someone has been workig on that? | 13:46 |
| amoralej | periodic jobs have been failing because of it | 13:46 |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-validations master: Unit testing for ini https://review.openstack.org/473522 | 13:46 |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui master: Refactor Loader component https://review.openstack.org/509181 | 13:48 |
| EmilienM | amoralej: not afik | 13:48 |
| rook | gfidente coming back.. jistr I set the retries to 5... | 13:51 |
| *** sid2 has joined #tripleo | 13:51 | |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-validations master: Add a validation that checks for available update https://review.openstack.org/476517 | 13:52 |
| mwhahaha | reminder, we have a meeting in here in ~8 mins https://etherpad.openstack.org/p/tripleo-meeting-items | 13:52 |
| *** jcoufal_ has joined #tripleo | 13:53 | |
| *** liverpooler has joined #tripleo | 13:53 | |
| *** liverpooler has quit IRC | 13:53 | |
| lvdombrkr | folks, what mean this looping lines on overcloud compute node while deployment in progress IN=br-ex OUT, SRC...DST... https://pasteboard.co/GNe0HKa.jpg | 13:53 |
| *** liverpooler has joined #tripleo | 13:54 | |
| *** pdeore has quit IRC | 13:54 | |
| *** liverpooler has quit IRC | 13:54 | |
| *** raildo has quit IRC | 13:54 | |
| *** sid1 has quit IRC | 13:55 | |
| *** raildo has joined #tripleo | 13:55 | |
| *** jcoufal has quit IRC | 13:55 | |
| *** liverpooler has joined #tripleo | 13:55 | |
| *** brault has joined #tripleo | 13:57 | |
| *** sshnaidm is now known as sshnaidm|mtg | 13:58 | |
| *** spectr has quit IRC | 13:58 | |
| *** spectr has joined #tripleo | 13:59 | |
| *** cdearborn_ has joined #tripleo | 13:59 | |
| *** cylopez has joined #tripleo | 13:59 | |
| mwhahaha | #startmeeting tripleo | 13:59 |
| mwhahaha | #topic agenda | 13:59 |
| mwhahaha | * Review past action items | 13:59 |
| mwhahaha | * One off agenda items | 13:59 |
| mwhahaha | * Squad status | 13:59 |
| mwhahaha | * Bugs & Blueprints | 13:59 |
| openstack | Meeting started Tue Oct 3 13:59:37 2017 UTC and is due to finish in 60 minutes. The chair is mwhahaha. Information about MeetBot at http://wiki.debian.org/MeetBot. | 13:59 |
| mwhahaha | * Projects releases or stable backports | 13:59 |
| mwhahaha | * Specs | 13:59 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 13:59 |
| mwhahaha | * open discussion | 13:59 |
| mwhahaha | Anyone can use the #link, #action and #info commands, not just the moderatorǃ | 13:59 |
| *** openstack changes topic to " (Meeting topic: tripleo)" | 13:59 | |
| openstack | The meeting name has been set to 'tripleo' | 13:59 |
| *** openstack changes topic to "agenda (Meeting topic: tripleo)" | 13:59 | |
| EmilienM | o/ | 13:59 |
| fultonj | o/ | 13:59 |
| slagle | hi | 13:59 |
| jfrancoa | o/ | 14:00 |
| trown | o/ | 14:00 |
| bandini | o/ | 14:00 |
| mandre | o/ | 14:00 |
| cdearborn_ | \o | 14:00 |
| marios | o/ | 14:00 |
| beagles | o/ | 14:00 |
| jistr | o/ | 14:00 |
| matbu_ | o/ | 14:00 |
| jtomasek | o/ | 14:00 |
| atoth | o/ | 14:00 |
| ccamacho | o/ | 14:00 |
| jrist | \o | 14:00 |
| mwhahaha | alright let's do this | 14:01 |
| mwhahaha | #topic review past action items | 14:01 |
| *** openstack changes topic to "review past action items (Meeting topic: tripleo)" | 14:01 | |
| mwhahaha | shardy to look at how to reduce # of services deployed on ovb (continued) | 14:01 |
| mwhahaha | shardy, did you get a chance to do this? not that it matters because CI is hosed | 14:01 |
| adarazs | o/ | 14:02 |
| *** sid2 has quit IRC | 14:02 | |
| mwhahaha | I'll take that as a no, i'll follow up with steve later | 14:03 |
| owalsh | o/ | 14:03 |
| mwhahaha | #action mwhahaha to follow up with shardy about services and ovb jobs | 14:03 |
| mwhahaha | review newton backports in gerrit | 14:03 |
| EmilienM | I think we did it | 14:03 |
| mwhahaha | as a reminder upstream eol is soon | 14:03 |
| EmilienM | but some of them are still under review | 14:03 |
| mwhahaha | still trying to figure out what that exactly means for tripleo but make sure to take a look at newton stuff | 14:04 |
| EmilienM | and it won't make progress until this zuul v3 stops to kill us | 14:04 |
| mwhahaha | yup | 14:04 |
| fultonj | regarding newton i have a bug i am working on i hope to backport to newton but it has to go further https://bugs.launchpad.net/tripleo/+bug/1720787 | 14:04 |
| openstack | Launchpad bug 1720787 in tripleo "TripleO deploys ceph client keyring with 644 permissions" [High,In progress] - Assigned to John Fulton (jfulton-org) | 14:04 |
| shardy | mwhahaha: Hey sorry, running late, no progress re ovb jobs yet, as you say CI has been down | 14:04 |
| fultonj | just an fyi | 14:04 |
| shardy | mwhahaha: still plan to take a look, but pre-zuul3 issues the OVB jobs weren't timing out as much so I deproritized reducing the services | 14:05 |
| EmilienM | fultonj: it will be - in theory we release newton / ocata / pike every 2 weeks | 14:05 |
| mwhahaha | shardy: ok. yea i think it's still a good idea to evaulate what we're checking to also help with excessive coverage, etc | 14:05 |
| fultonj | EmilienM: ack | 14:05 |
| openstackgerrit | Tim Rozet proposed openstack/tripleo-heat-templates master: Dynamically renders network isolation resource registry https://review.openstack.org/509190 | 14:06 |
| shardy | mwhahaha: Yeah agreed | 14:06 |
| *** abregman|mtg is now known as abregman | 14:06 | |
| mwhahaha | ok moving on to the next stuff | 14:06 |
| mwhahaha | #topic one off agenda items | 14:06 |
| *** openstack changes topic to "one off agenda items (Meeting topic: tripleo)" | 14:06 | |
| mwhahaha | #link https://etherpad.openstack.org/p/tripleo-meeting-items | 14:06 |
| mwhahaha | looks empty, anyone have anything they wish to address? | 14:06 |
| fultonj | one-off... | 14:07 |
| fultonj | https://etherpad.openstack.org/p/tripleo-integration-squad-status | 14:07 |
| fultonj | TripleO Integration Squad Status | 14:07 |
| mwhahaha | fultonj: we have a section for squad status (next) :D | 14:07 |
| fultonj | whops | 14:07 |
| *** ykarel has joined #tripleo | 14:07 | |
| mwhahaha | but thanks i'll just link that | 14:07 |
| EmilienM | fultonj: nice etherpad, thanks! | 14:07 |
| fultonj | thanks | 14:07 |
| fultonj | thanks gfidente | 14:07 |
| mwhahaha | ok moving on to squad status since we have some | 14:08 |
| mwhahaha | #topic Squad status | 14:08 |
| *** openstack changes topic to "Squad status (Meeting topic: tripleo)" | 14:08 | |
| mwhahaha | integration | 14:08 |
| mwhahaha | #link https://etherpad.openstack.org/p/tripleo-integration-squad-status | 14:08 |
| fultonj | we talked about these items at PTG | 14:08 |
| EmilienM | fultonj: do we target them all for Queens? | 14:08 |
| fultonj | no progress on multiple ceph clusters (would be expansion of composable roles capability) | 14:08 |
| *** dsneddon has joined #tripleo | 14:09 | |
| fultonj | only multiple ceph pools and luminous | 14:09 |
| fultonj | 2 out of the 3 | 14:09 |
| fultonj | we hope to land changes for multiple ceph clusters now, but not fully deliver feature until post-queens | 14:09 |
| marios | mwhahaha: i just put this down (we spoke about it in dfg upgrades not sure if someone got round to it already) but https://etherpad.openstack.org/p/tripleo-upgrade-squad-status | 14:09 |
| fultonj | now --> during the queens cycle | 14:09 |
| mwhahaha | marios: thanks! | 14:09 |
| marios | matbu_: can you update with the minor updates remaining https://etherpad.openstack.org/p/tripleo-upgrade-squad-status | 14:10 |
| *** ooolpbot has joined #tripleo | 14:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1718387 | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1719123 | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720220 | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720458 | 14:10 |
| openstack | Launchpad bug 1718387 in tripleo "ping test is periodically failing for the gate-tripleo-ci-centos-7-nonha-multinode-oooq " [Critical,Triaged] - Assigned to Sofer Athlan-Guyot (sofer-athlan-guyot) | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720556 | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720721 | 14:10 |
| marios | mwhahaha: i'll ask folks to update it | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720918 | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720973 | 14:10 |
| openstack | Launchpad bug 1719123 in tripleo "tempest fails on overcloud keystone admin tenant error" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz) | 14:10 |
| *** ooolpbot has quit IRC | 14:10 | |
| openstack | Launchpad bug 1720220 in tripleo "CI: Most/all legacy-tripleo jobs failing on gate" [Critical,Triaged] | 14:10 |
| openstack | Launchpad bug 1720458 in tripleo "Lastest delorean pip package causes error in autodoc" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 14:10 |
| openstack | Launchpad bug 1720556 in tripleo "tracker: upstream zuul is not triggering periodic jobs" [Critical,Triaged] - Assigned to wes hayutin (weshayutin) | 14:10 |
| openstack | Launchpad bug 1720721 in tripleo "CI: OVB jobs fail because can't install XStatic from PyPI mirror on rh1 cloud" [Critical,Triaged] - Assigned to Paul Belanger (pabelanger) | 14:10 |
| openstack | Launchpad bug 1720918 in tripleo "puppet-firewall changed the ipv6-icmp rule type name" [Critical,In progress] - Assigned to Ben Nemec (bnemec) | 14:10 |
| openstack | Launchpad bug 1720973 in tripleo "CI: infras cirros image has wrong permission" [Critical,Triaged] - Assigned to Paul Belanger (pabelanger) | 14:10 |
| mwhahaha | marios: sounds good i'll use that to refer to status durring the meetings going forward | 14:10 |
| matbu_ | marios: yep /me tries to click on the link | 14:10 |
| EmilienM | fultonj: do you have upgrades working from ocata - puppet-ceph managed to pike ceph-ansible managed? (I know CI is down now, but I would like to see some CI job voting here once it's back) | 14:10 |
| fultonj | from ocata working, but no ci (must add ci) | 14:11 |
| EmilienM | fultonj: maybe you can add it in the etherpad, I would like it to be a priority | 14:11 |
| fultonj | EmilienM: yes, will do | 14:11 |
| EmilienM | thanks | 14:11 |
| fultonj | #actionitem | 14:11 |
| mandre | EmilienM: I think that may be covered in the CI work we do in the containers squad | 14:11 |
| fultonj | #action ceph-upgrade ci | 14:11 |
| EmilienM | mandre: most probably, I just want to make sure this is done at some point | 14:12 |
| mandre | talking about the container squad, we also prepared an etherpad :) https://etherpad.openstack.org/p/tripleo-containers-squad-status | 14:12 |
| mwhahaha | mandre: thanks! i'll add that to the list for next time as well | 14:12 |
| matbu_ | marios: done | 14:12 |
| mwhahaha | ci - weshay trown sshnaidm|mtg - any status that you can share around CI? | 14:13 |
| mwhahaha | ui/cli - jrist - any status you can share around UI? (or validations as well) | 14:13 |
| trown | mwhahaha: other than it is totally borked due to zuulv3 migration not a ton this week | 14:13 |
| EmilienM | mwhahaha: status is here https://etherpad.openstack.org/p/tripleo-ci-squad-meeting | 14:13 |
| marios | ty matbu_ | 14:13 |
| mwhahaha | EmilienM: yea but they are getting rid of the squad meeting so is that still valid? | 14:13 |
| jrist | nothing in particular except that we are beginning to work on some ci and upstream automation | 14:13 |
| weshay | mwhahaha, we're putting on the finishing touches to tripleo promotion jobs in rdo software factory | 14:14 |
| *** cdearborn has quit IRC | 14:14 | |
| EmilienM | mwhahaha: or here even https://etherpad.openstack.org/p/tripleo-ci-squad-scrum | 14:14 |
| trown | we are working on a new process for organizing our squad and will have more formal status etherpad going forward | 14:14 |
| mwhahaha | trown: sounds good, just let me know so I can add it to the list | 14:14 |
| mwhahaha | workflows - thrash - any status you can share? | 14:15 |
| mwhahaha | networking - beagles - any status you can share? | 14:15 |
| weshay | tripleo-ci-squad sprint details can be found https://www.redhat.com/archives/rdo-list/2017-September/msg00068.html | 14:15 |
| *** sid1 has joined #tripleo | 14:15 | |
| thrash | mwhahaha: working on some rfe's for ui | 14:16 |
| mwhahaha | thrash: sounds good, thanks. let us know if you need anything (reviews/etc) | 14:16 |
| thrash | mwhahaha: ack | 14:17 |
| mwhahaha | ok thanks everyone for the status. let's move on | 14:17 |
| mwhahaha | #topic bugs & blueprints | 14:17 |
| mwhahaha | #link https://launchpad.net/tripleo/+milestone/queens-1 | 14:17 |
| *** openstack changes topic to "bugs & blueprints (Meeting topic: tripleo)" | 14:17 | |
| mwhahaha | We currently have 62 blueprints and about 476 open bugs. Please take some time to review your blueprint status and make sure it is properly up to date. | 14:18 |
| *** dpawar has quit IRC | 14:18 | |
| mwhahaha | please be aware that queens-1 is in about 2 weeks | 14:18 |
| mwhahaha | so please move stuff that you aren't going to land by queens-1 out to queens-2 | 14:18 |
| *** artom has joined #tripleo | 14:18 | |
| mwhahaha | anyone have any blueprints or bugs they want to raise for visibility? | 14:19 |
| *** dpawar has joined #tripleo | 14:19 | |
| slagle | i filed a blueprint for the ansible work | 14:19 |
| slagle | just fyi | 14:19 |
| EmilienM | I hope we can merge something before the 2 weeks ^^ | 14:19 |
| mwhahaha | ok | 14:20 |
| *** bobh has joined #tripleo | 14:20 | |
| marios | mwhahaha: its already in upgrades squad status, but for visibility, spec for the Q upgrades here https://review.openstack.org/507620 | 14:20 |
| marios | mwhahaha: captures what we discussed in ptg | 14:21 |
| mwhahaha | marios: sounds good | 14:21 |
| EmilienM | mwhahaha: https://blueprints.launchpad.net/tripleo/+spec/ansible-config-download-ci and https://blueprints.launchpad.net/tripleo/+spec/ansible-config-download | 14:21 |
| mwhahaha | slagle: do you think those will be landed in the next two weeks? | 14:22 |
| EmilienM | it's possible | 14:22 |
| slagle | mwhahaha: i think it's possible | 14:22 |
| slagle | yea, we'll go with "possible" :) | 14:22 |
| mwhahaha | provided ci stops being on fire :D | 14:22 |
| slagle | it's also possible not a single patch will get landed in the next 2 weeks :) | 14:22 |
| EmilienM | we already tested the bits on our envs, it works fine (module some changes) - if our CI is back this week... | 14:22 |
| *** itlinux has joined #tripleo | 14:23 | |
| EmilienM | we'll move it to queens-2 otherwise, but we want to get ci coverage asap for this feature | 14:23 |
| *** psahoo has quit IRC | 14:23 | |
| mwhahaha | makes sense | 14:23 |
| mwhahaha | Ok any other bugs/blueprints? | 14:23 |
| shardy | 476 open bugs - I wonder what we can do about that - I think the trend is upwards so perhaps we can prune/prioritize or de-duplicate better there? | 14:23 |
| mwhahaha | shardy: i've started to go through and clean some up | 14:23 |
| mwhahaha | shardy: we're about +6 for the week | 14:24 |
| shardy | mwhahaha: ack OK I'll see if I can spend some time helping | 14:24 |
| EmilienM | we could target 450 by queens-1, and maybe 400 by queens-2 | 14:24 |
| mwhahaha | there's a bunch of old stuff that is no longer valid i'm sure | 14:24 |
| shardy | mwhahaha: I find it useful to have a clearly defined (smaller) set of priority things for a milestone, then it's easier to know which reviews to prioritize | 14:24 |
| mwhahaha | so we could move all <= medium to queens-2 for visibility | 14:25 |
| shardy | mwhahaha: yeah something like that might be good | 14:25 |
| mwhahaha | #action mwhahaha to move bugs <= medium to queens-2 and review > medium for validity | 14:25 |
| mwhahaha | i'll do that this week | 14:25 |
| EmilienM | mwhahaha: +1 | 14:25 |
| mwhahaha | moving on | 14:25 |
| mwhahaha | #topic projects releases or stable backports | 14:25 |
| *** openstack changes topic to "projects releases or stable backports (Meeting topic: tripleo)" | 14:25 | |
| EmilienM | mwhahaha: I can do it with the script | 14:26 |
| mwhahaha | EmilienM: i'll get it | 14:26 |
| EmilienM | k | 14:26 |
| *** itlinux has quit IRC | 14:26 | |
| mwhahaha | so we have pending stable releases but there's some issues around th stable-policy stuff | 14:26 |
| mwhahaha | Given the status of CI there isn't much point in talking backport patches | 14:27 |
| mwhahaha | so let's move on | 14:27 |
| mwhahaha | #topic specs | 14:27 |
| mwhahaha | #link https://review.openstack.org/#/q/project:openstack/tripleo-specs+status:open | 14:27 |
| *** openstack changes topic to "specs (Meeting topic: tripleo)" | 14:27 | |
| mwhahaha | given that queens-1 is in ~2 weeks, we need to make sure any queens specs get merged asap | 14:27 |
| mwhahaha | please take a look at the open ones and review | 14:28 |
| mwhahaha | as marios pointed out the upgrade spec is here: | 14:28 |
| mwhahaha | #link https://review.openstack.org/507620 | 14:28 |
| EmilienM | can we merge specs? lol | 14:28 |
| mwhahaha | well it starts with reviewing them :D | 14:28 |
| EmilienM | ah yeah, jobs pass, at least something that works | 14:28 |
| marios | mwhahaha: thanks, sorry for the noise before (I didn't read the context, i thought you were asking for updates on Q stuff, not blueprints/bugs targetting apologies ) | 14:28 |
| mwhahaha | marios: it's all good, it's better to have more info then none at all :D | 14:29 |
| mwhahaha | Ok so on to the open discussion | 14:30 |
| mwhahaha | #topic open discussion | 14:30 |
| *** openstack changes topic to "open discussion (Meeting topic: tripleo)" | 14:30 | |
| *** janki has quit IRC | 14:30 | |
| mwhahaha | Anything else that folks would like to talk about? | 14:30 |
| EmilienM | we haven't talked much about CI | 14:30 |
| bandini | is there any super rough/high level eta for fixing zuulv3? | 14:30 |
| EmilienM | it's unclear to me where we are now | 14:30 |
| mwhahaha | dmsimard, pabelanger: any updates on CI | 14:30 |
| *** tongl has joined #tripleo | 14:30 | |
| marios | right, i was weary about asking... do we know anything more about it :) seems other share the sentiment | 14:30 |
| shardy | bandini: that's been asked on the ML but I don't see any reply yet | 14:31 |
| EmilienM | we've been very patient until now but things are getting bad now | 14:31 |
| bandini | shardy: yeah saw the mail, thought maybe some tripleo ci overlords had some additional thoughts :) | 14:31 |
| EmilienM | pabelanger, dmsimard: this CI downtime is having a critical effect on TripleO delivery, fyi | 14:31 |
| *** chlong has quit IRC | 14:31 | |
| *** chlong_ has quit IRC | 14:31 | |
| dmsimard | mwhahaha, EmilienM: sshnaidm|mtg gave me an update earlier, it looks like we're almost clear and hitting some sort of timeout right now | 14:31 |
| dmsimard | let me pull up the review, hang on. | 14:32 |
| EmilienM | shardy: which email? | 14:32 |
| *** cdearborn_ is now known as cdearborn | 14:32 | |
| EmilienM | [openstack-dev] [all] Update on Zuul v3 Migration - and what to do about issues | 14:32 |
| EmilienM | this one? | 14:32 |
| shardy | [openstack-dev] [all] Update on Zuul v3 Migration - and what to do about issues | 14:32 |
| dmsimard | mwhahaha, EmilienM: https://review.openstack.org/#/c/508660/ | 14:32 |
| *** chlong has joined #tripleo | 14:32 | |
| trown | dmsimard: we are hitting a timeout because the connection to the nodepool node drops | 14:32 |
| amoralej | EmilienM, but if we ping puppet-firewall, we'll be unable to merge the fix, right? | 14:32 |
| shardy | yeah that one, dansmith asked about mitigation as did sdague yesterday | 14:33 |
| bnemec | True | 14:33 |
| dmsimard | trown: yes, I am looking into it right now | 14:33 |
| EmilienM | amoralej: let's pin it anyway for now, so we remove one issue | 14:33 |
| amoralej | ok | 14:33 |
| EmilienM | dmsimard, pabelanger : so back to the initial question, any ETA? | 14:33 |
| dmsimard | mwhahaha, EmilienM: there are different ongoing discussions about considering a rollback but it's not a simple task because some projects (tripleo included) have had to introduce changes in their projects to support zuul v3 and doing a rollback would mean breaking those projects again | 14:34 |
| EmilienM | why haven't we tested thing before? | 14:34 |
| EmilienM | it's a paradox that the projects in charge of testing wasn't tested against tripleo before | 14:35 |
| shardy | yeah it's kinda surprising there wasn't some sort of paralell migration strategy so jobs could be switched over gradually | 14:35 |
| shardy | EmilienM: well it sounds like it's not only TripleO | 14:35 |
| *** janki has joined #tripleo | 14:35 | |
| tosky | the problem is that some of the fixes are stuck in the queue, but they will land | 14:36 |
| tosky | I'm not sure that rolling back now wouldn't cause even more disruption | 14:36 |
| pabelanger | we are having scale issues, which are harder to test then migration process | 14:36 |
| shardy | tosky: yeah, it'd just be nice to have better visibility of the status/progress I guess | 14:36 |
| *** tongl has quit IRC | 14:37 | |
| pabelanger | EmilienM: no ETA, suggest watching https://etherpad.openstack.org/p/zuulv3-issues and #openstack-infra. | 14:37 |
| EmilienM | anything we can help? | 14:37 |
| dmsimard | EmilienM: I heavily recommended a gradual opt-in of selected projects, especially deployment projects such as puppet-openstack, openstack-ansible and tripleo but the decision ended otherwise | 14:37 |
| dmsimard | EmilienM: right now we are working with https://review.openstack.org/#/c/508660/ | 14:38 |
| dmsimard | and I am looking into what seems to be a timeout related issue | 14:38 |
| pabelanger | just patience and what ever needs done on etherpad | 14:38 |
| dmsimard | seeing as the jobs ran for over 2 hours, I'm hopeful there are no more issues related to tripleo or the jobs themselves | 14:38 |
| EmilienM | pabelanger: patience ✓ already ;-) | 14:39 |
| *** aputtur has joined #tripleo | 14:39 | |
| EmilienM | last patch merged in tripleo was Sep 28th | 14:39 |
| EmilienM | do we have a failover plan if by end of week it's not fixed? | 14:39 |
| EmilienM | we can't stop merging code duringe more than a week, we have to find a plan b | 14:40 |
| dmsimard | EmilienM: there's ongoing discussion around that topic and I anticipate an infra core to chime in the openstack-dev thread | 14:40 |
| mwhahaha | weshay: said there might be an option with software factory | 14:40 |
| EmilienM | I don't think they have enough resources tbh | 14:40 |
| mwhahaha | time to figure out what would qualify as a critical minimal subset of CI testing | 14:41 |
| EmilienM | do we have someone looking at that^ ? | 14:41 |
| EmilienM | weshay: ^ | 14:41 |
| EmilienM | mwhahaha: anyway, we can move on I guess | 14:42 |
| shardy | we did that once before, disabled a bunch of jobs and merged code with a subset - it took weeks to fix all the regressions after | 14:42 |
| shardy | so IMO it'd be best avoided if possible | 14:42 |
| EmilienM | shardy: no way we do that again | 14:42 |
| * mwhahaha shrugs | 14:43 | |
| mwhahaha | it's an option if we can't get anything goig | 14:43 |
| mwhahaha | going | 14:43 |
| dmsimard | it seems jobs are timing out on a specific play: | 14:43 |
| dmsimard | http://logs.openstack.org/60/508660/13/check/legacy-tripleo-ci-centos-7-scenario001-multinode-oooq-puppet/c92e76d/job-output.txt.gz#_2017-10-03_10_59_26_310226 | 14:43 |
| dmsimard | anyone familiar would know why ? | 14:44 |
| mwhahaha | we can look after that | 14:44 |
| shardy | mwhahaha: yeah, cool, just saying probably should be a last resort :) | 14:44 |
| EmilienM | on "Add virthost to inventory" ? | 14:44 |
| dmsimard | maybe it's not a specific thing, another job timed out on a different task http://logs.openstack.org/60/508660/13/check/legacy-tripleo-ci-centos-7-undercloud-oooq/5dc7681/job-output.txt.gz#_2017-10-03_10_46_17_231725 | 14:44 |
| mwhahaha | ok so it sounds like we need to actively work on CI, so lets close out the meeting and go focus on CI | 14:44 |
| mwhahaha | thanks everyone | 14:44 |
| mwhahaha | #endmeeting | 14:45 |
| *** openstack changes topic to "CI Status: RED - zuul v3'd (see alerts) | http://tripleo.org/ | https://docs.openstack.org/tripleo-docs/latest/" | 14:45 | |
| openstack | Meeting ended Tue Oct 3 14:45:01 2017 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 14:45 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/tripleo/2017/tripleo.2017-10-03-13.59.html | 14:45 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/tripleo/2017/tripleo.2017-10-03-13.59.txt | 14:45 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/tripleo/2017/tripleo.2017-10-03-13.59.log.html | 14:45 |
| EmilienM | dmsimard: https://git.openstack.org/cgit/openstack/tripleo-quickstart/tree/playbooks/build-images.yml#n2 | 14:45 |
| dmsimard | mwhahaha, EmilienM: it's been mostly sshnaidm|mtg and I looking at failures, if we can get more people actively looking we might be more productive | 14:45 |
| mwhahaha | trown: can you help out since it looks like oooq stuff -^ | 14:46 |
| dmsimard | all jobs are timeouting in https://review.openstack.org/#/c/508660/ and we need to figure out why | 14:46 |
| shardy | dmsimard: That looks like the subnode is just down or not reachable so the ssh fails? | 14:46 |
| trown | mwhahaha: dmsimard ... I have also been looking | 14:46 |
| jrist | thanks mwhahaha | 14:46 |
| trown | dmsimard: and as explained it timed out because we lost connection | 14:46 |
| matbu_ | shardy: yep looks like | 14:46 |
| dmsimard | shardy: that's a good hint, hang on | 14:46 |
| trown | dmsimard: I have said that 3 times this morning already | 14:47 |
| *** leitan has quit IRC | 14:47 | |
| dmsimard | trown: look at the timestamps, those tasks are probably not legitimately supposed to run for ~2h by themselves | 14:47 |
| dmsimard | there's something else going on | 14:48 |
| trown | yesterday the jobs could not even run... so saying we need more help looking is a bit frustrating | 14:48 |
| trown | there has been nothing to look at | 14:48 |
| *** ffiore has quit IRC | 14:49 | |
| trown | and there still is nothing to look at in the current jobs becuase they lost connection and there is no logs | 14:49 |
| trown | not like I can just stand up my own zuulv3 and reproduce the issue | 14:49 |
| jschlueter|znc | what would cause a line like "2017-10-03 08:34:42Z [0]: CREATE_IN_PROGRESS state changed" to show up in overcloud_install.log ? | 14:51 |
| *** jschlueter|znc is now known as jschlueter | 14:51 | |
| dmsimard | trown: I don't exactly have any more tools or access than you do and I've been working on this since last week through the weekend | 14:51 |
| bnemec | jschlueter: A resource named 0 started the creation process. | 14:52 |
| jschlueter | was looking a CI job and saw that odd message show up | 14:52 |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-validations stable/pike: Fix name of controller token validation https://review.openstack.org/509195 | 14:52 |
| trown | dmsimard: k... just dont know what you are asking for more eyes on it... there is nothing to look at | 14:52 |
| shardy | jschlueter: it's an event from one of the ResourceGroups - "0" being the resource name for e.g a node in a resource group | 14:52 |
| trown | dmsimard: also... this is not a problem created by tripleo-ci | 14:52 |
| shardy | jschlueter: IMO it's an expected message | 14:52 |
| jschlueter | "0" is a not a very good descriptive name... | 14:53 |
| shardy | jschlueter: note that in recent versions we print the whole path, so it's easier to see what the resource is | 14:53 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates stable/newton: Keep floating ip reachability during pacemaker migration. https://review.openstack.org/474967 | 14:53 |
| dmsimard | trown: there is absolutely nothing that corellates all of those 10 jobs timing out together from what you can see ? they've all just lost connection ? | 14:53 |
| trown | dmsimard: I am willing to help however I can... and have been trying to, but there is nothing to look at | 14:53 |
| shardy | jschlueter: yeah in recent versions it'd be someting like overcloud.Controller.0 | 14:53 |
| jschlueter | shardy: this is Pike ... would that have it? | 14:53 |
| shardy | jschlueter: Yeah, in that case it's not expected, is it from the overcloud deploy output? | 14:53 |
| jschlueter | ahh overcloud.Controller.0 | 14:53 |
| dmsimard | trown: I found something that perhaps explains multinode issues but would not explain single node issues | 14:53 |
| dmsimard | and I'm working on a patch | 14:54 |
| jschlueter | yes this is from an overcloud deploy containerized on pike as of a week ago | 14:54 |
| *** agurenko has quit IRC | 14:54 | |
| shardy | jschlueter: Ok let me see if I can replicate it | 14:54 |
| shardy | maybe a bug crept in somewhere | 14:54 |
| *** sshnaidm|mtg is now known as sshnaidm | 14:54 | |
| dmsimard | trown: if multinode doesn't pass after this patch I'll ask infra to hold a set of nodes | 14:55 |
| jschlueter | shardy: it from an OSP 12 CI job and I can get you link if that's helpful | 14:55 |
| dmsimard | I have no idea for the single node job. | 14:55 |
| trown | dmsimard: thanks | 14:55 |
| shardy | jschlueter: sure, yes please | 14:55 |
| *** dpawar has quit IRC | 14:55 | |
| sshnaidm | dmsimard, afaics zuul lose connection to all nodes - single, multi.. | 14:56 |
| jschlueter | shardy: ack | 14:56 |
| trown | dmsimard: btw... I think a revert would be the correct thing to do for infra here... I know that is not going to go anywhere though | 14:56 |
| dmsimard | trown: they've been discussing the different options since this morning | 14:56 |
| *** dpawar has joined #tripleo | 14:56 | |
| dmsimard | and they've started preparing things in case of a rollback such as https://review.openstack.org/#/c/509196/ | 14:57 |
| *** brault has quit IRC | 14:57 | |
| shardy | dmsimard: yeah the timestamps do look suspicious, e.g 2017-10-03 10:59:26.310226 | primary | PLAY [Add virthost to inventory] | 14:57 |
| shardy | then the next output is 2017-10-03 12:58:01.628571 | PLAY RECAP | 14:58 |
| shardy | so it hung for nearly an hour - I wonder if that's just the ssh connection hanging or something else? | 14:58 |
| shardy | the tasks prior to that seemed to run OK AFAICS | 14:58 |
| tosky | dmsimard: so how is the likelyhood of a rollback right now? | 14:58 |
| dmsimard | tosky: high ? | 14:58 |
| tosky | sigh | 14:58 |
| dmsimard | tosky: too much heat and pitchforks | 14:59 |
| *** ecerquei has quit IRC | 14:59 | |
| dmsimard | tosky: they're discussing the different options to keep pushing forward while unbreaking projects | 14:59 |
| shardy | dmsimard: out of interest, what was the reason for the flag day migration vs a phased migration? | 14:59 |
| dmsimard | it's not easy | 14:59 |
| *** ebarrera has quit IRC | 14:59 | |
| tosky | dmsimard: say that some projects are already working, can they decide to stay? | 14:59 |
| tosky | on v3, I mean | 14:59 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates master: Remove package if service stopped and disabled https://review.openstack.org/479886 | 15:00 |
| *** brault has joined #tripleo | 15:00 | |
| dmsimard | tosky: not with the currently discussed option, projects would return to v2 gating. There are projects that have fully migrated to native v3 jobs as well in the meantime | 15:00 |
| *** dpawar has quit IRC | 15:01 | |
| sshnaidm | tosky, afaics there are problem with zuul itself too, not with configs only | 15:01 |
| jrist | nice haircut EmilienM | 15:01 |
| EmilienM | jrist: well, I didn't expect doing it, but it's a long story | 15:01 |
| jrist | haha | 15:01 |
| *** chem has joined #tripleo | 15:01 | |
| *** ansmith_ has joined #tripleo | 15:02 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates master: Set host name explicitly for telemetry https://review.openstack.org/509199 | 15:03 |
| dmsimard | shardy: I'm not 100% sure, I believe the migration process is very complex and running it selectively makes it even more so. The bulk of the migration occurs here and it's 175k lines https://review.openstack.org/#/c/507176/ | 15:03 |
| *** gvrangan has joined #tripleo | 15:04 | |
| *** leitan has joined #tripleo | 15:05 | |
| shardy | Wow. 175k lines of auto-generated ansible, I mean that had to just work, right?! :D | 15:05 |
| * mwhahaha resists urge to troll | 15:06 | |
| dmsimard | 99% of those are shell tasks running JJB builders verbatim | 15:07 |
| dmsimard | there's not so much "auto generated ansible" as it is feeding whatever the job ran in terms of JJB builders, publishers, in a playbook with shell passthrough | 15:07 |
| *** dmarlin has joined #tripleo | 15:08 | |
| shardy | dmsimard: yeah, just joking really, sure the pain will be worth it, but I've been bitten by flag-day migrations too many times now, it always goes wrong IME | 15:08 |
| *** sshnaidm is now known as sshnaidm|afk | 15:08 | |
| * shardy likes words like "incremental" ;) | 15:09 | |
| dmsimard | I agree. I tried to test deployment projects ahead of the migration but I got some push back. Example: https://review.openstack.org/#/c/505921/ | 15:09 |
| *** sid1 has quit IRC | 15:09 | |
| dmsimard | FWIW that particular review, just by submitting it, allowed us to find several problems ahead of the migration | 15:09 |
| *** ooolpbot has joined #tripleo | 15:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1718387 | 15:10 |
| openstack | Launchpad bug 1718387 in tripleo "ping test is periodically failing for the gate-tripleo-ci-centos-7-nonha-multinode-oooq " [Critical,Triaged] - Assigned to Sofer Athlan-Guyot (sofer-athlan-guyot) | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1719123 | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720220 | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720458 | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720556 | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720721 | 15:10 |
| openstack | Launchpad bug 1719123 in tripleo "tempest fails on overcloud keystone admin tenant error" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz) | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720918 | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720973 | 15:10 |
| *** ooolpbot has quit IRC | 15:10 | |
| openstack | Launchpad bug 1720220 in tripleo "CI: Most/all legacy-tripleo jobs failing on gate" [Critical,Triaged] | 15:10 |
| openstack | Launchpad bug 1720458 in tripleo "Lastest delorean pip package causes error in autodoc" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 15:10 |
| openstack | Launchpad bug 1720556 in tripleo "tracker: upstream zuul is not triggering periodic jobs" [Critical,Triaged] - Assigned to wes hayutin (weshayutin) | 15:10 |
| openstack | Launchpad bug 1720721 in tripleo "CI: OVB jobs fail because can't install XStatic from PyPI mirror on rh1 cloud" [Critical,Triaged] - Assigned to Paul Belanger (pabelanger) | 15:10 |
| openstack | Launchpad bug 1720918 in tripleo "puppet-firewall changed the ipv6-icmp rule type name" [Critical,In progress] - Assigned to Ben Nemec (bnemec) | 15:10 |
| openstack | Launchpad bug 1720973 in tripleo "CI: infras cirros image has wrong permission" [Critical,Triaged] - Assigned to Paul Belanger (pabelanger) | 15:10 |
| bnemec | I think we may have set a new record for alert bugs. | 15:10 |
| bnemec | \o/ ? | 15:10 |
| pabelanger | Xstatic one is done, so the alert can be removed | 15:11 |
| *** yprokule has quit IRC | 15:11 | |
| *** gfidente has quit IRC | 15:12 | |
| *** rcernin has quit IRC | 15:12 | |
| *** abregman is now known as abregman|mtg | 15:12 | |
| dmsimard | trown: it turns out the fix I thought I had for multinode was already in | 15:13 |
| *** gfidente has joined #tripleo | 15:13 | |
| *** mdnadeem has quit IRC | 15:14 | |
| dmsimard | trown: sshnaidm|afk beat me to it with https://github.com/openstack-infra/openstack-zuul-jobs/commit/ccb5ee2ba95c3a85adc54a5d73a009fd1c2ac5cc | 15:14 |
| *** ffiore has joined #tripleo | 15:15 | |
| *** apetrich has quit IRC | 15:18 | |
| *** pcaruana has quit IRC | 15:20 | |
| trown | zuul status page is down now? | 15:25 |
| *** brault has quit IRC | 15:27 | |
| *** abregman|mtg is now known as abregman | 15:28 | |
| *** abregman is now known as abregman|afk | 15:28 | |
| *** achadha has joined #tripleo | 15:29 | |
| *** cdearborn has quit IRC | 15:29 | |
| *** itlinux has joined #tripleo | 15:29 | |
| weshay | trown, looks that way http://zuulv3.openstack.org/ | 15:29 |
| * trown gets lunch | 15:30 | |
| *** trown is now known as trown|lunch | 15:30 | |
| *** itlinux has quit IRC | 15:31 | |
| *** cdearborn has joined #tripleo | 15:32 | |
| *** ecerquei has joined #tripleo | 15:32 | |
| *** achadha has quit IRC | 15:33 | |
| *** pblaho has quit IRC | 15:34 | |
| *** chlong has quit IRC | 15:36 | |
| *** dmarlin has quit IRC | 15:40 | |
| *** aufi has quit IRC | 15:41 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates stable/newton: Keep floating ip reachability during pacemaker migration. https://review.openstack.org/474967 | 15:41 |
| *** dparkes has quit IRC | 15:43 | |
| *** itlinux has joined #tripleo | 15:44 | |
| *** leitan has quit IRC | 15:44 | |
| *** brault has joined #tripleo | 15:45 | |
| *** leitan has joined #tripleo | 15:45 | |
| *** archit has joined #tripleo | 15:45 | |
| *** psachin has quit IRC | 15:45 | |
| *** leitan has quit IRC | 15:46 | |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates master: Fix some missed hard-coded network references https://review.openstack.org/508947 | 15:49 |
| *** abregman|afk has quit IRC | 15:49 | |
| *** janki has quit IRC | 15:50 | |
| *** dmarlin has joined #tripleo | 15:51 | |
| *** tongl has joined #tripleo | 15:51 | |
| *** archit has quit IRC | 15:52 | |
| dmsimard | EmilienM, shardy, trown|lunch, weshay, mwhahaha: there'll be communications about an imminent partial rollback going out to openstack-dev soon | 15:53 |
| shardy | dmsimard: ack thanks for the update | 15:53 |
| *** gbarros has joined #tripleo | 15:53 | |
| dmsimard | tl;dr, every project is going back to full v2 gating, v3 will run with limited capacity only on the check pipeline | 15:53 |
| *** shardy is now known as shardy_afk | 15:53 | |
| dmsimard | apevec ^ | 15:53 |
| mwhahaha | dmsimard: oh you mean the way it should have been in the beginning? sweet | 15:54 |
| dmsimard | ¯\_(ツ)_/¯ | 15:54 |
| dmsimard | afk food | 15:54 |
| *** cdearborn__ has joined #tripleo | 15:57 | |
| *** lucasagomes is now known as lucas-afk | 16:01 | |
| amoralej | sshnaidm|afk, EmilienM, puppet-firewall is reverted in current repo | 16:02 |
| EmilienM | good | 16:02 |
| EmilienM | thanks | 16:02 |
| *** pkovar has quit IRC | 16:03 | |
| *** tosky has quit IRC | 16:04 | |
| *** egonzalez has quit IRC | 16:04 | |
| *** gvrangan has quit IRC | 16:05 | |
| *** archit_ has joined #tripleo | 16:05 | |
| *** jfrancoa has quit IRC | 16:06 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates master: Special treatment for os-net-config upgrade. https://review.openstack.org/509214 | 16:06 |
| *** milan has quit IRC | 16:09 | |
| *** ooolpbot has joined #tripleo | 16:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1718387 | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1719123 | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720220 | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720458 | 16:10 |
| openstack | Launchpad bug 1718387 in tripleo "ping test is periodically failing for the gate-tripleo-ci-centos-7-nonha-multinode-oooq " [Critical,Triaged] - Assigned to Sofer Athlan-Guyot (sofer-athlan-guyot) | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720556 | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720918 | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720973 | 16:10 |
| *** ooolpbot has quit IRC | 16:10 | |
| openstack | Launchpad bug 1719123 in tripleo "tempest fails on overcloud keystone admin tenant error" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz) | 16:10 |
| openstack | Launchpad bug 1720220 in tripleo "CI: Most/all legacy-tripleo jobs failing on gate" [Critical,Triaged] | 16:10 |
| openstack | Launchpad bug 1720458 in tripleo "Lastest delorean pip package causes error in autodoc" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 16:10 |
| openstack | Launchpad bug 1720556 in tripleo "tracker: upstream zuul is not triggering periodic jobs" [Critical,Triaged] - Assigned to wes hayutin (weshayutin) | 16:10 |
| openstack | Launchpad bug 1720918 in tripleo "puppet-firewall changed the ipv6-icmp rule type name" [Critical,In progress] - Assigned to Ben Nemec (bnemec) | 16:10 |
| openstack | Launchpad bug 1720973 in tripleo "CI: infras cirros image has wrong permission" [Critical,Triaged] - Assigned to Paul Belanger (pabelanger) | 16:10 |
| *** cdearborn has quit IRC | 16:10 | |
| openstackgerrit | Lee Yarwood proposed openstack/tripleo-quickstart-extras master: DNM: Allow the configuration of virtualbmc on Newton https://review.openstack.org/509218 | 16:11 |
| openstackgerrit | Lee Yarwood proposed openstack/tripleo-quickstart master: Allow the confgiuration of virtualbmc on Newton https://review.openstack.org/509219 | 16:12 |
| *** marios has quit IRC | 16:12 | |
| *** ffiore has quit IRC | 16:12 | |
| *** ccamacho has quit IRC | 16:13 | |
| *** yamahata has joined #tripleo | 16:14 | |
| *** aditya_r has quit IRC | 16:16 | |
| *** nyechiel_ has quit IRC | 16:16 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates master: Special treatment for os-net-config upgrade. https://review.openstack.org/509214 | 16:17 |
| *** brault has quit IRC | 16:19 | |
| *** nyechiel_ has joined #tripleo | 16:24 | |
| *** yamahata has quit IRC | 16:25 | |
| *** gbarros has quit IRC | 16:27 | |
| *** leitan has joined #tripleo | 16:29 | |
| *** trown|lunch is now known as trown | 16:36 | |
| *** gvrangan has joined #tripleo | 16:37 | |
| *** gvrangan has quit IRC | 16:37 | |
| *** tesseract has quit IRC | 16:37 | |
| *** gvrangan has joined #tripleo | 16:37 | |
| *** shreshtha has joined #tripleo | 16:38 | |
| *** gvrangan has quit IRC | 16:38 | |
| *** gvrangan has joined #tripleo | 16:38 | |
| *** gvrangan has quit IRC | 16:41 | |
| *** cylopez has quit IRC | 16:42 | |
| *** aditya_r has joined #tripleo | 16:43 | |
| *** gvrangan has joined #tripleo | 16:43 | |
| *** aditya_ra has joined #tripleo | 16:46 | |
| *** aditya_ra has quit IRC | 16:46 | |
| *** jpich has quit IRC | 16:47 | |
| *** gvrangan has quit IRC | 16:47 | |
| *** aditya_r has quit IRC | 16:48 | |
| *** aditya_ra has joined #tripleo | 16:49 | |
| openstackgerrit | Honza Pokorny proposed openstack/tripleo-common master: Remove scsi-target-utils from centos source packages https://review.openstack.org/509225 | 16:52 |
| openstackgerrit | Honza Pokorny proposed openstack/tripleo-common master: Correct names of mariadb packages for source builds https://review.openstack.org/509226 | 16:52 |
| *** aditya_ra has quit IRC | 16:52 | |
| *** derekh has quit IRC | 16:55 | |
| *** nyechiel_ has quit IRC | 16:56 | |
| *** yamahata has joined #tripleo | 16:58 | |
| *** salmankhan has quit IRC | 16:59 | |
| *** mcornea has quit IRC | 17:03 | |
| *** pblaho has joined #tripleo | 17:05 | |
| *** ykarel has quit IRC | 17:06 | |
| *** shreshtha has quit IRC | 17:09 | |
| *** ooolpbot has joined #tripleo | 17:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1718387 | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1719123 | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720220 | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720458 | 17:10 |
| openstack | Launchpad bug 1718387 in tripleo "ping test is periodically failing for the gate-tripleo-ci-centos-7-nonha-multinode-oooq " [Critical,Triaged] - Assigned to Sofer Athlan-Guyot (sofer-athlan-guyot) | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720556 | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720918 | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720973 | 17:10 |
| *** ooolpbot has quit IRC | 17:10 | |
| openstack | Launchpad bug 1719123 in tripleo "tempest fails on overcloud keystone admin tenant error" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz) | 17:10 |
| openstack | Launchpad bug 1720220 in tripleo "CI: Most/all legacy-tripleo jobs failing on gate" [Critical,Triaged] | 17:10 |
| openstack | Launchpad bug 1720458 in tripleo "Lastest delorean pip package causes error in autodoc" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 17:10 |
| openstack | Launchpad bug 1720556 in tripleo "tracker: upstream zuul is not triggering periodic jobs" [Critical,Triaged] - Assigned to wes hayutin (weshayutin) | 17:10 |
| openstack | Launchpad bug 1720918 in tripleo "puppet-firewall changed the ipv6-icmp rule type name" [Critical,In progress] - Assigned to Ben Nemec (bnemec) | 17:10 |
| openstack | Launchpad bug 1720973 in tripleo "CI: infras cirros image has wrong permission" [Critical,Triaged] | 17:10 |
| *** jpena is now known as jpena|off | 17:10 | |
| openstackgerrit | Michele Baldessari proposed openstack/tripleo-quickstart master: pm_addr in instack should not be localhost https://review.openstack.org/509229 | 17:12 |
| *** mcornea has joined #tripleo | 17:13 | |
| *** gvrangan has joined #tripleo | 17:19 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras master: WIP: work with containers across builds https://review.openstack.org/509232 | 17:21 |
| *** thrash is now known as thrash|biab | 17:27 | |
| *** gvrangan has quit IRC | 17:30 | |
| *** gvrangan has joined #tripleo | 17:31 | |
| *** gvrangan has quit IRC | 17:33 | |
| *** gvrangan has joined #tripleo | 17:34 | |
| *** abregman has joined #tripleo | 17:35 | |
| *** gvrangan has quit IRC | 17:37 | |
| *** gvrangan has joined #tripleo | 17:37 | |
| *** itlinux has quit IRC | 17:41 | |
| *** achadha has joined #tripleo | 17:41 | |
| *** achadha has quit IRC | 17:41 | |
| *** achadha has joined #tripleo | 17:42 | |
| *** itlinux has joined #tripleo | 17:42 | |
| *** dprince has quit IRC | 17:46 | |
| *** achadha_ has joined #tripleo | 17:46 | |
| *** achadha has quit IRC | 17:47 | |
| *** itlinux has quit IRC | 17:48 | |
| *** achadha_ has quit IRC | 17:48 | |
| *** gvrangan has quit IRC | 17:48 | |
| *** achadha has joined #tripleo | 17:50 | |
| *** gvrangan has joined #tripleo | 17:50 | |
| *** gvrangan has quit IRC | 17:54 | |
| *** itlinux has joined #tripleo | 17:54 | |
| *** bnemec has quit IRC | 17:56 | |
| *** itlinux has quit IRC | 17:58 | |
| *** bnemec has joined #tripleo | 18:01 | |
| *** amoralej is now known as amoralej|off | 18:01 | |
| *** dprince has joined #tripleo | 18:04 | |
| *** milan has joined #tripleo | 18:08 | |
| *** hrybacki is now known as hrybacki|brb | 18:09 | |
| *** thrash|biab is now known as thrash | 18:10 | |
| *** ooolpbot has joined #tripleo | 18:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1718387 | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1719123 | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720220 | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720458 | 18:10 |
| openstack | Launchpad bug 1718387 in tripleo "ping test is periodically failing for the gate-tripleo-ci-centos-7-nonha-multinode-oooq " [Critical,Triaged] - Assigned to Sofer Athlan-Guyot (sofer-athlan-guyot) | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720556 | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720918 | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720973 | 18:10 |
| *** ooolpbot has quit IRC | 18:10 | |
| openstack | Launchpad bug 1719123 in tripleo "tempest fails on overcloud keystone admin tenant error" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz) | 18:10 |
| openstack | Launchpad bug 1720220 in tripleo "CI: Most/all legacy-tripleo jobs failing on gate" [Critical,Triaged] | 18:10 |
| openstack | Launchpad bug 1720458 in tripleo "Lastest delorean pip package causes error in autodoc" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 18:10 |
| openstack | Launchpad bug 1720556 in tripleo "tracker: upstream zuul is not triggering periodic jobs" [Critical,Triaged] - Assigned to wes hayutin (weshayutin) | 18:10 |
| openstack | Launchpad bug 1720918 in tripleo "puppet-firewall changed the ipv6-icmp rule type name" [Critical,In progress] - Assigned to Ben Nemec (bnemec) | 18:10 |
| openstack | Launchpad bug 1720973 in tripleo "CI: infras cirros image has wrong permission" [Critical,Triaged] | 18:10 |
| *** aditya_r has joined #tripleo | 18:16 | |
| *** hrybacki|brb is now known as hrybacki | 18:17 | |
| *** itlinux has joined #tripleo | 18:21 | |
| *** gvrangan has joined #tripleo | 18:31 | |
| *** gvrangan has quit IRC | 18:31 | |
| *** gvrangan has joined #tripleo | 18:31 | |
| *** dparkes has joined #tripleo | 18:32 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart master: add team structure doc https://review.openstack.org/508325 | 18:32 |
| *** nyechiel_ has joined #tripleo | 18:32 | |
| *** noslzzp has joined #tripleo | 18:32 | |
| *** aditya_r has quit IRC | 18:34 | |
| *** jmelvin has quit IRC | 18:35 | |
| *** noslzzp has quit IRC | 18:38 | |
| slagle | weshay: trown : hi. in oooq...do I already have access to the subnodes hostname and ip somewhere? | 18:41 |
| weshay | slagle, is the ssh.config.ansible file | 18:41 |
| weshay | s/in/is | 18:42 |
| slagle | weshay: is that from groups["overcloud"]? | 18:43 |
| weshay | slagle, pm | 18:46 |
| alee | weshay, pm | 18:49 |
| bnemec | pms for everyone! :-) | 18:56 |
| EmilienM | weshay: please don't pm, i'm very interested by this topic | 18:56 |
| EmilienM | weshay: I'm going to work with slagle on CI'ing config-download work | 18:57 |
| weshay | EmilienM, I was sending examples of the ssh config and hosts file | 18:57 |
| EmilienM | please share it here | 18:57 |
| weshay | I'd use upstream ci, but .. ooops | 18:57 |
| EmilienM | lol | 18:58 |
| *** gvrangan has quit IRC | 18:58 | |
| slagle | weshay: if i want to test a change to the overcloud-deploy role for a multinode job, what is the best way to do that? | 18:58 |
| slagle | weshay: given i don't want to "guess and check" with actual upstream ci | 18:59 |
| weshay | slagle, you need to test it w/ nodepool or just multiple nodes | 18:59 |
| slagle | weshay: i have multiple nodes, that's no problem | 19:00 |
| weshay | if you need to use the same exact deployment with nodepool traas is going to be your best bet | 19:00 |
| slagle | weshay: the part i'm missing is ... how do I just "reapply" the overcloud-deploy role | 19:00 |
| weshay | featureset001/002 and a few others will deploy locally with multiple nodes, but you have to specify the topology file | 19:00 |
| weshay | slagle, this is an example of a step by step deployment https://docs.openstack.org/tripleo-quickstart/latest/getting-started.html | 19:01 |
| weshay | note the -I and --teardown none | 19:02 |
| slagle | weshay: great thanks, i think that's the missing piece | 19:02 |
| slagle | we need to modify overcloud-deploy to generate a few new environments and pass them to the deploy command | 19:03 |
| slagle | and I think this will let us quickly test those changes out | 19:03 |
| weshay | cool | 19:04 |
| slagle | weshay: as for the subnodes thing...it looks like the subnodes are dynamically added to the invnentory, so maybe I can grab the hostname/ip from those facts? | 19:05 |
| slagle | i'll try that anyway | 19:05 |
| weshay | ya it will have it slurped in if gather_facts is executed on those nodes. I'd need to double check that | 19:06 |
| weshay | collect_logs runs on them but I'd need to look at the other roles | 19:07 |
| *** gfidente has quit IRC | 19:07 | |
| weshay | EmilienM, get a haircut please | 19:07 |
| EmilienM | weshay: I bought this new thing and set it to 4mm. It was too short. | 19:08 |
| EmilienM | weshay: you know she killed me | 19:08 |
| weshay | lol | 19:08 |
| EmilienM | i sleep on the couch now | 19:08 |
| *** Goneri has quit IRC | 19:08 | |
| *** jtomasek has quit IRC | 19:10 | |
| *** ooolpbot has joined #tripleo | 19:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1718387 | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1719123 | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720220 | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720458 | 19:10 |
| openstack | Launchpad bug 1718387 in tripleo "ping test is periodically failing for the gate-tripleo-ci-centos-7-nonha-multinode-oooq " [Critical,Triaged] - Assigned to Sofer Athlan-Guyot (sofer-athlan-guyot) | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720556 | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720973 | 19:10 |
| *** ooolpbot has quit IRC | 19:10 | |
| openstack | Launchpad bug 1719123 in tripleo "tempest fails on overcloud keystone admin tenant error" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz) | 19:10 |
| openstack | Launchpad bug 1720220 in tripleo "CI: Most/all legacy-tripleo jobs failing on gate" [Critical,Triaged] | 19:10 |
| openstack | Launchpad bug 1720458 in tripleo "Lastest delorean pip package causes error in autodoc" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 19:10 |
| openstack | Launchpad bug 1720556 in tripleo "tracker: upstream zuul is not triggering periodic jobs" [Critical,Triaged] - Assigned to wes hayutin (weshayutin) | 19:10 |
| openstack | Launchpad bug 1720973 in tripleo "CI: infras cirros image has wrong permission" [Critical,Triaged] | 19:10 |
| *** ramishra has quit IRC | 19:11 | |
| *** aputtur has quit IRC | 19:19 | |
| *** catintheroof has quit IRC | 19:32 | |
| *** catintheroof has joined #tripleo | 19:32 | |
| Slower | slagle: did you get rid of that tmp dir for the playbook download? | 19:33 |
| slagle | Slower: no. that's not in my patch actually | 19:34 |
| slagle | Slower: it's in here: https://review.openstack.org/#/c/487496 | 19:35 |
| slagle | Slower: reviews and comments still appropriate :) | 19:35 |
| EmilienM | dmsimard, mwhahaha: just fyi I noticed that tripleo jobs fail on quickstart jobs, not on legacy-tripleo-ci | 19:37 |
| Slower | slagle: ok I'll check it out thanks! | 19:37 |
| *** jmelvin has joined #tripleo | 19:37 | |
| EmilienM | see https://review.openstack.org/#/c/487496 for example | 19:37 |
| EmilienM | legacy-tripleo-ci-centos-7-undercloud-containers worked fine | 19:37 |
| EmilienM | but oooq multinode jobs failed - have we figured out something related I guess? | 19:37 |
| dmsimard | EmilienM: some of the changes sshnaidm|afk or others might have introduced are not necessarily compatible with both v2 and v3 | 19:38 |
| *** jprovazn has quit IRC | 19:39 | |
| EmilienM | are they merged? | 19:39 |
| dmsimard | EmilienM: need to pick up kids at school, changes should mostly be documented here: https://bugs.launchpad.net/tripleo/+bug/1720220 | 19:40 |
| openstack | Launchpad bug 1720220 in tripleo "CI: Most/all legacy-tripleo jobs failing on gate" [Critical,Triaged] | 19:40 |
| dmsimard | brb | 19:40 |
| *** chlong_ has joined #tripleo | 19:43 | |
| *** jcoufal_ has quit IRC | 19:43 | |
| *** chlong has joined #tripleo | 19:44 | |
| *** chlong has quit IRC | 19:46 | |
| *** nyechiel_ has quit IRC | 19:49 | |
| gregwork | browbeat seems pretty amazing as a project/toolkit | 19:49 |
| gregwork | does anyone here use it ? | 19:49 |
| *** bnemec has quit IRC | 19:49 | |
| slagle | rook does :) | 19:51 |
| weshay | lol | 19:51 |
| weshay | tripleo seems like a cool project, anyone here use it? | 19:51 |
| rook | hahaha weshay | 19:55 |
| dmsimard | EmilienM: I re-read what you said just now and I misunderstood what you meant | 19:55 |
| rook | gregwork what are you looking for... we do have our own channel | 19:55 |
| *** archit_ is now known as archit | 19:55 | |
| dmsimard | EmilienM: I thought you already had v2 jobs run again on a patch and v3 jobs passed but v2 jobs failed | 19:55 |
| dmsimard | EmilienM: I am not currently hunting down any tripleo v3 issues, the only hint I had this morning was a false alarm | 19:56 |
| *** dparkes has quit IRC | 19:58 | |
| *** jkilpatr_ has quit IRC | 20:00 | |
| *** chem` has joined #tripleo | 20:04 | |
| *** chem` has quit IRC | 20:04 | |
| *** chem` has joined #tripleo | 20:04 | |
| *** chem has quit IRC | 20:06 | |
| *** liverpooler has quit IRC | 20:08 | |
| *** ooolpbot has joined #tripleo | 20:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1718387 | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1719123 | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720220 | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720458 | 20:10 |
| openstack | Launchpad bug 1718387 in tripleo "ping test is periodically failing for the gate-tripleo-ci-centos-7-nonha-multinode-oooq " [Critical,Triaged] - Assigned to Sofer Athlan-Guyot (sofer-athlan-guyot) | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720556 | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720973 | 20:10 |
| *** ooolpbot has quit IRC | 20:10 | |
| openstack | Launchpad bug 1719123 in tripleo "tempest fails on overcloud keystone admin tenant error" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz) | 20:10 |
| openstack | Launchpad bug 1720220 in tripleo "CI: Most/all legacy-tripleo jobs failing on gate" [Critical,Triaged] | 20:10 |
| openstack | Launchpad bug 1720458 in tripleo "Lastest delorean pip package causes error in autodoc" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 20:10 |
| openstack | Launchpad bug 1720556 in tripleo "tracker: upstream zuul is not triggering periodic jobs" [Critical,Triaged] - Assigned to wes hayutin (weshayutin) | 20:10 |
| openstack | Launchpad bug 1720973 in tripleo "CI: infras cirros image has wrong permission" [Critical,Triaged] | 20:10 |
| *** pchavva has quit IRC | 20:10 | |
| *** catintheroof has quit IRC | 20:10 | |
| *** catintheroof has joined #tripleo | 20:11 | |
| openstackgerrit | Michele Baldessari proposed openstack/tripleo-quickstart-extras master: WIP Initial autofencing support https://review.openstack.org/502297 | 20:13 |
| *** milan has quit IRC | 20:14 | |
| *** jkilpatr has joined #tripleo | 20:20 | |
| *** artom has quit IRC | 20:21 | |
| *** toure is now known as toure_biab | 20:21 | |
| *** itlinux has quit IRC | 20:22 | |
| *** achadha has quit IRC | 20:24 | |
| *** Goneri has joined #tripleo | 20:24 | |
| *** achadha has joined #tripleo | 20:25 | |
| mwhahaha | so are we back on v2 and is everything ok? | 20:27 |
| *** achadha has quit IRC | 20:29 | |
| *** itlinux has joined #tripleo | 20:32 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-specs master: introduce a ci team structure policy https://review.openstack.org/509280 | 20:34 |
| dmsimard | mwhahaha: I've started seeing jenkins report back at least 15 minutes ago https://review.openstack.org/#/c/508791/ | 20:35 |
| mwhahaha | dmsimard: yea i meant for tripleo jobs :D but yea it seems that v2 is picking up stuf | 20:36 |
| * mwhahaha test approves something | 20:36 | |
| *** trown is now known as trown|outtypewww | 20:38 | |
| *** ansmith has quit IRC | 20:38 | |
| *** ansmith_ has quit IRC | 20:39 | |
| gregwork | rook: i see, I am trying to come up with some benchmarks of this rhosp hci build I have. We shined it up with nvme for ceph and mellanox 100gbe cards using ovs-dpdk | 20:39 |
| gregwork | rook: browbeat looked like an interesting toolkit to generate those reports | 20:39 |
| gregwork | rook: just trying to get it installed now, not all of the install scripts respect environment: proxy_env (our lab is dark outside of squid proxy) | 20:40 |
| weshay | mwhahaha, our no-op test change is 472607 I kicked it.. and it's visible in http://status.openstack.org/zuul/ | 20:42 |
| *** catintheroof has quit IRC | 20:42 | |
| *** achadha has joined #tripleo | 20:42 | |
| *** catintheroof has joined #tripleo | 20:42 | |
| mwhahaha | yea i approved 507970 and it's looking ok so far | 20:43 |
| mwhahaha | it only has nonha-multipnode + containers-multinode + pep8 | 20:43 |
| rook | gregwork we are at openstack-browbeat | 20:43 |
| mwhahaha | so we'll see in like 90 mins if all went well | 20:43 |
| *** archit has quit IRC | 20:45 | |
| openstackgerrit | Lokesh Jain proposed openstack/tripleo-heat-templates master: Addition of Nuage bridge MTU parameter https://review.openstack.org/509284 | 20:45 |
| *** pblaho has quit IRC | 20:46 | |
| *** achadha has quit IRC | 20:46 | |
| *** nyechiel_ has joined #tripleo | 20:48 | |
| *** achadha has joined #tripleo | 20:49 | |
| *** itlinux has quit IRC | 20:57 | |
| *** lblanchard has quit IRC | 21:00 | |
| *** achadha has quit IRC | 21:00 | |
| *** tosky has joined #tripleo | 21:00 | |
| *** achadha has joined #tripleo | 21:01 | |
| *** chlong_ has quit IRC | 21:03 | |
| *** itlinux has joined #tripleo | 21:03 | |
| *** numans has quit IRC | 21:03 | |
| *** numans has joined #tripleo | 21:07 | |
| *** dparkes has joined #tripleo | 21:07 | |
| *** rhallisey has quit IRC | 21:07 | |
| *** alee has quit IRC | 21:08 | |
| *** ecerquei has quit IRC | 21:08 | |
| *** ooolpbot has joined #tripleo | 21:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1718387 | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1719123 | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720220 | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720458 | 21:10 |
| openstack | Launchpad bug 1718387 in tripleo "ping test is periodically failing for the gate-tripleo-ci-centos-7-nonha-multinode-oooq " [Critical,Triaged] - Assigned to Sofer Athlan-Guyot (sofer-athlan-guyot) | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720556 | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720973 | 21:10 |
| *** ooolpbot has quit IRC | 21:10 | |
| openstack | Launchpad bug 1719123 in tripleo "tempest fails on overcloud keystone admin tenant error" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz) | 21:10 |
| openstack | Launchpad bug 1720220 in tripleo "CI: Most/all legacy-tripleo jobs failing on gate" [Critical,Triaged] | 21:10 |
| openstack | Launchpad bug 1720458 in tripleo "Lastest delorean pip package causes error in autodoc" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 21:10 |
| openstack | Launchpad bug 1720556 in tripleo "tracker: upstream zuul is not triggering periodic jobs" [Critical,Triaged] - Assigned to wes hayutin (weshayutin) | 21:10 |
| openstack | Launchpad bug 1720973 in tripleo "CI: infras cirros image has wrong permission" [Critical,Triaged] | 21:10 |
| *** nyechiel_ has quit IRC | 21:14 | |
| openstackgerrit | Ian Main proposed openstack/python-tripleoclient master: WIP: Use ansible to deploy undercloud. https://review.openstack.org/509289 | 21:16 |
| *** Goneri has quit IRC | 21:16 | |
| Slower | slagle: dprince: ^^ FYI | 21:17 |
| *** raildo has quit IRC | 21:17 | |
| dprince | Slower: ack, will review in a bit | 21:18 |
| Slower | dprince: it's working. Getting DB errors again | 21:18 |
| *** abishop has quit IRC | 21:18 | |
| dprince | Slower: that means one of the software deployments likely isn't getting converted or executed in the playbook | 21:19 |
| Slower | right | 21:19 |
| dprince | Slower: is your inventory the same as the other day with slagle's fixes | 21:19 |
| Slower | I thought I had all the bits in place | 21:19 |
| Slower | maybe I missed something | 21:19 |
| dprince | Slower: yeah, maybe slagle' needs to work his magic again | 21:20 |
| Slower | oh I need a newer patch from slagle | 21:20 |
| Slower | that was it | 21:20 |
| Slower | I bet | 21:20 |
| Slower | I'll try it | 21:20 |
| dprince | Slower: yes! rebase those and I bet you'll get it :) | 21:21 |
| openstackgerrit | John Fulton proposed openstack/tripleo-heat-templates master: WIP: Set file mode permission for Ceph keyrings in containers https://review.openstack.org/508975 | 21:22 |
| *** bfournie has quit IRC | 21:24 | |
| slagle | weshay: for the command here: https://docs.openstack.org/tripleo-quickstart/latest/getting-started.html#deploy-the-tripleo-overcloud | 21:28 |
| *** mcornea has quit IRC | 21:28 | |
| slagle | weshay: what directory do you run that from? | 21:28 |
| *** bobh has quit IRC | 21:28 | |
| slagle | weshay: i have an environment setup to match a ci job (with traas) | 21:28 |
| weshay | from the tripleo-quickstart dir | 21:29 |
| slagle | weshay: ok...quickstart-extras-overcloud.yml does not have to exist in that dir? | 21:30 |
| Slower | dprince: sadly I accidently ran cleanup_everything.sh today | 21:31 |
| weshay | slagle, it just needs to be in the local working dir ... e.g. what you specify w/ -w or default to ~/.quickstart | 21:31 |
| Slower | so I may be missing something that was in there.. although I thought I put it all in doit.sh | 21:31 |
| dprince | Slower: I never liked/run that script :/ | 21:32 |
| dprince | Slower: remove it! | 21:32 |
| Slower | I was going to add a requried '-f' arg or something | 21:32 |
| Slower | we should do something tho | 21:32 |
| slagle | weshay: ok, that helps a bit | 21:33 |
| *** morazi has quit IRC | 21:33 | |
| owalsh | mwhahaha: would you mind pulling the +W trigger on https://review.openstack.org/506594 when (if?) the gate looks good? | 21:33 |
| weshay | slagle, you can take tq and tqe source a virtenv and run python setup.py install on each and all the files land into place | 21:33 |
| mwhahaha | owalsh: sure | 21:33 |
| * mwhahaha adds it to the list | 21:34 | |
| slagle | weshay: yea, i've got that much setup by the ci job | 21:34 |
| slagle | weshay: but now am getting: ERROR! the file_name '/home/centos/tripleo-root/workspace/.quickstart/config/release/master.yml' does not exist, or is not readable | 21:34 |
| owalsh | mwhahaha: thanks, looks good so far at least :-) | 21:34 |
| slagle | weshay: I guess somethign to do with passing -R master? | 21:34 |
| slagle | but i have no config/release dir at all | 21:34 |
| weshay | sounds like the setup was not run, is the virtenv in .quickstart? | 21:35 |
| Slower | fatal: [overcloud]: FAILED! => {"failed": true, "msg": "Unexpected templating type error occurred on ({{ vars[item] | to_json }}): Undefined is not JSON serializable"} | 21:35 |
| Slower | that's new.. | 21:35 |
| Slower | anywho.. | 21:35 |
| slagle | Slower: we saw that on Friday. it's likely a problem with your inventory | 21:36 |
| slagle | Slower: need to add the host to the group named after the role | 21:36 |
| slagle | so that it picks up what's in group_vars/<role> | 21:37 |
| Slower | oh right | 21:40 |
| Slower | I thought I did that tho | 21:40 |
| Slower | slagle: https://da.gd/FI3Mz | 21:41 |
| *** ansmith has joined #tripleo | 21:45 | |
| *** ansmith_ has joined #tripleo | 21:46 | |
| *** ecerquei has joined #tripleo | 21:49 | |
| *** jlinkes has quit IRC | 21:50 | |
| *** itlinux has quit IRC | 21:53 | |
| *** jmelvin has quit IRC | 21:59 | |
| EmilienM | weshay: we are running iterative deployment and it fails on fatal: [undercloud]: FAILED! => {"changed": true, "failed": true, "msg": "Destination directory /home/stack does not exist"} | 22:00 |
| EmilienM | weshay: we run on rdo cloud so of course /home/stack doesn't exist | 22:00 |
| EmilienM | we need to override working_dir ? | 22:00 |
| *** ooolpbot has joined #tripleo | 22:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1718387 | 22:10 |
| openstack | Launchpad bug 1718387 in tripleo "ping test is periodically failing for the gate-tripleo-ci-centos-7-nonha-multinode-oooq " [Critical,Triaged] - Assigned to Sofer Athlan-Guyot (sofer-athlan-guyot) | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1719123 | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720220 | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720458 | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720556 | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720973 | 22:10 |
| openstack | Launchpad bug 1719123 in tripleo "tempest fails on overcloud keystone admin tenant error" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz) | 22:10 |
| *** ooolpbot has quit IRC | 22:10 | |
| openstack | Launchpad bug 1720220 in tripleo "CI: Most/all legacy-tripleo jobs failing on gate" [Critical,Triaged] | 22:10 |
| openstack | Launchpad bug 1720458 in tripleo "Lastest delorean pip package causes error in autodoc" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 22:10 |
| openstack | Launchpad bug 1720556 in tripleo "tracker: upstream zuul is not triggering periodic jobs" [Critical,Triaged] - Assigned to wes hayutin (weshayutin) | 22:10 |
| openstack | Launchpad bug 1720973 in tripleo "CI: infras cirros image has wrong permission" [Critical,Triaged] | 22:10 |
| * mwhahaha waits for the first merge in 5 days | 22:16 | |
| mwhahaha | YOU CAN DOO EEET ZUUL V2 | 22:17 |
| mwhahaha | like watching paint dry | 22:21 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Fix logging for docker-puppet https://review.openstack.org/507970 | 22:23 |
| mwhahaha | \o/ (also you're welcome Slower) | 22:24 |
| *** mwhahaha changes topic to "CI Status: GREENish - zuul v2 back (v3 still broken) (see alerts) | http://tripleo.org/ | https://docs.openstack.org/tripleo-docs/latest/" | 22:24 | |
| Slower | mwhahaha: woot! thanks :) | 22:25 |
| *** pmannidi has joined #tripleo | 22:25 | |
| EmilienM | weshay: we need your help, probably tomorrow morning | 22:28 |
| EmilienM | weshay: running: ~/tripleo-root/tripleo-quickstart/quickstart.sh -w ~/tripleo-root/workspace/.quickstart/ -R master --no-clone --tags all --nodes ~/tripleo-root/tripleo-quickstart/config/nodes/1ctlr.yml -I --teardown none -p quickstart-extras-overcloud.yml --extra-vars ~/tripleo-root/tripleo-quickstart/config/general_config/featureset-multinode-common.yml --release tripleo-ci/master --environment | 22:29 |
| EmilienM | ~/tripleo-root/tripleo-ci/toci-quickstart/config/testenv/multinode.yml --config ~/tripleo-root/tripleo-quickstart/config/general_config/featureset004.yml localhost | 22:29 |
| EmilienM | weshay: and we have this task error: http://paste.openstack.org/show/622589/ | 22:30 |
| EmilienM | weshay: what file did I miss? | 22:30 |
| *** ecerquei_ has joined #tripleo | 22:31 | |
| mwhahaha | just as a comment, we need to figure out what to do with the ci alerts for v3 | 22:32 |
| mwhahaha | probably should drop the alert (if they aren't impacting anymore) and tag with zuulv3 or something | 22:33 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-common master: Config download support for all deployments https://review.openstack.org/508189 | 22:33 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: fs10: deploy steps with ansible https://review.openstack.org/508307 | 22:33 |
| *** ecerquei has quit IRC | 22:33 | |
| EmilienM | mwhahaha: yes, good idea | 22:33 |
| openstackgerrit | Merged openstack/tripleo-docs master: Update deprecated NovaCompute parameter names https://review.openstack.org/507800 | 22:38 |
| *** apevec has left #tripleo | 22:39 | |
| *** dmarlin has quit IRC | 22:40 | |
| *** ecerquei_ has quit IRC | 22:41 | |
| *** ecerquei has joined #tripleo | 22:41 | |
| Slower | ok I'm back to the db errors now :) | 22:43 |
| Slower | yay me! | 22:43 |
| *** rlandy is now known as rlandy|bbl | 22:48 | |
| weshay | EmilienM, k | 22:49 |
| *** bfournie has joined #tripleo | 22:50 | |
| *** achadha has quit IRC | 22:50 | |
| *** achadha has joined #tripleo | 22:50 | |
| openstackgerrit | Merged openstack/tripleo-docs master: Fix use of CephAnsibleExtraConfig and CephConfigOverrides https://review.openstack.org/507882 | 22:51 |
| *** bfournie has quit IRC | 22:51 | |
| *** achadha_ has joined #tripleo | 22:52 | |
| mwhahaha | weshay: https://bugs.launchpad.net/tripleo/+bug/1719123 is that not a thing anymore? (the patch was merged) | 22:53 |
| openstack | Launchpad bug 1719123 in tripleo "tempest fails on overcloud keystone admin tenant error" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz) | 22:53 |
| weshay | mwhahaha, ya.. new error on tempest.. I close this one | 22:53 |
| mwhahaha | k thanks | 22:53 |
| *** achadha_ has quit IRC | 22:54 | |
| *** achadha_ has joined #tripleo | 22:54 | |
| *** achadha has quit IRC | 22:55 | |
| jrist | dmsimard: any ideas on this one? I clicked the command on the failed line http://i.share.pho.to/7e653c12_o.png | 22:56 |
| dmsimard | jrist: have a link to the report ? | 22:56 |
| jrist | http://logs.openstack.org/03/508903/1/check/legacy-releasenotes/46d6430/ara/ | 22:57 |
| dmsimard | jrist: lots of files not found in there, not just that one task | 22:58 |
| dmsimard | jrist: yikes http://logs.openstack.org/03/508903/1/check/legacy-releasenotes/46d6430/job-output.txt.gz#_2017-10-02_15_44_38_299976 | 22:58 |
| jrist | ah that must be the problem | 22:58 |
| jrist | no space! | 22:58 |
| jrist | :( | 22:59 |
| weshay | EmilienM, slagle http://logs.openstack.org/66/497766/3/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/f1f6b8e/console.html#_2017-10-03_20_14_44_280101 | 22:59 |
| *** achadha_ has quit IRC | 22:59 | |
| *** achadha has joined #tripleo | 22:59 | |
| *** leitan has quit IRC | 22:59 | |
| jrist | dmsimard: TIL you could link to lines in these logs | 22:59 |
| dmsimard | ಠ_ಠ | 22:59 |
| jrist | so | 22:59 |
| jrist | forgive me for the stupid question | 22:59 |
| jrist | but what do we do about this? | 23:00 |
| *** achadha has quit IRC | 23:00 | |
| dmsimard | jrist: the disk space issue was resolved yesterday by infra and should not happen again | 23:00 |
| jrist | woot thanks | 23:00 |
| jrist | I'll doa recheck | 23:00 |
| *** achadha has joined #tripleo | 23:00 | |
| dmsimard | jrist: sure thing | 23:01 |
| dmsimard | jrist: should work | 23:01 |
| *** catintheroof has quit IRC | 23:01 | |
| jrist | dmsimard: man, ara is looking good | 23:01 |
| jrist | that logo! :) | 23:01 |
| dmsimard | jrist: ikr | 23:01 |
| dmsimard | jrist: imagine if we had a sick 404/500 page too | 23:02 |
| dmsimard | :P | 23:02 |
| jrist | yeah yeah working on it :) | 23:02 |
| dmsimard | no worries, 1.0 is still a way off anyway | 23:02 |
| dmsimard | more work than I thought | 23:02 |
| jrist | always | 23:03 |
| *** brian83 has quit IRC | 23:04 | |
| jrist | dmsimard: do you think this is fixed? | 23:04 |
| jrist | http://logs.openstack.org/60/508860/1/check/legacy-tripleo-ci-centos-7-undercloud-oooq/870370b/job-output.txt.gz#_2017-10-02_17_30_22_092133 | 23:04 |
| dmsimard | jrist: the disk space issue? definitely, asked infra about it | 23:04 |
| jrist | dmsimard: no, this is ssh host keys issue | 23:04 |
| jrist | known_hosts rather | 23:04 |
| *** achadha has quit IRC | 23:05 | |
| dmsimard | jrist: that's an interesting issue but should no longer impact you since it's a zuul v3 job and zuul v3 is no longer gating, we've rolled everything back to v2 | 23:05 |
| jrist | ah that explains it | 23:05 |
| dmsimard | there's a thread or two on openstack-dev explaining the partial rollback | 23:05 |
| jrist | I must have missed those | 23:06 |
| jrist | *booggle* | 23:06 |
| jrist | dmsimard: a third issuehttp://logs.openstack.org/02/504902/4/check/legacy-tripleo-ci-centos-7-undercloud-oooq/b10bbaa/job-output.txt.gz#_2017-10-03_04_07_32_449145 | 23:06 |
| jrist | http://logs.openstack.org/02/504902/4/check/legacy-tripleo-ci-centos-7-undercloud-oooq/b10bbaa/job-output.txt.gz#_2017-10-03_04_07_32_449145 rather | 23:06 |
| jrist | "Quickstart completed." | 23:07 |
| jrist | and then it says ERROR | 23:07 |
| jrist | right after that | 23:07 |
| dmsimard | disregard any legacy-* jobs for the time being, we'll eventually get to them | 23:07 |
| jrist | ah | 23:07 |
| jrist | but all of ours are legacy at the moment :) | 23:08 |
| dmsimard | jrist: if you do a recheck you'll get non-legacy jobs chiming in | 23:09 |
| jrist | k | 23:09 |
| dmsimard | legacy jobs will eventually chime in as well but they're non-voting | 23:09 |
| *** ooolpbot has joined #tripleo | 23:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1718387 | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1719123 | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1720458 | 23:10 |
| *** ooolpbot has quit IRC | 23:10 | |
| openstack | Launchpad bug 1718387 in tripleo "ping test is periodically failing for the gate-tripleo-ci-centos-7-nonha-multinode-oooq " [Critical,Triaged] - Assigned to Sofer Athlan-Guyot (sofer-athlan-guyot) | 23:10 |
| openstack | Launchpad bug 1719123 in tripleo "tempest fails on overcloud keystone admin tenant error" [Critical,Fix committed] - Assigned to Arx Cruz (arxcruz) | 23:10 |
| openstack | Launchpad bug 1720458 in tripleo "Lastest delorean pip package causes error in autodoc" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 23:10 |
| *** tzumainn has quit IRC | 23:15 | |
| *** itlinux has joined #tripleo | 23:21 | |
| *** tosky has quit IRC | 23:22 | |
| *** achadha has joined #tripleo | 23:24 | |
| *** bnemec has joined #tripleo | 23:25 | |
| *** achadha has quit IRC | 23:29 | |
| *** etingof has quit IRC | 23:32 | |
| *** itlinux has quit IRC | 23:32 | |
| *** tongl has quit IRC | 23:41 | |
| *** jrist is now known as jrist-afk | 23:42 | |
| jrist-afk | crap | 23:42 |
| *** jrist-afk is now known as jrist | 23:42 | |
| slagle | weshay: thanks...and if we wanted to just redo overcloud-deploy would we just s/multinode.yaml/quickstart-extras-overcloud.yml/ ? | 23:52 |
| slagle | weshay: or perhaps just passing the overcloud-deploy to --tags might be the better approach? | 23:53 |
| *** etingof has joined #tripleo | 23:54 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
