| *** saneax is now known as saneax-_-|AFK | 00:00 | |
| *** karimb has quit IRC | 00:06 | |
| *** jcoufal has joined #tripleo | 00:10 | |
| *** ooolpbot has joined #tripleo | 00:11 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 00:11 |
|---|---|---|
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1654611 | 00:11 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1654615 | 00:11 |
| *** ooolpbot has quit IRC | 00:11 | |
| openstack | Launchpad bug 1654611 in tripleo "Mitaka nonha ping test failing to upload image to glance" [Critical,Triaged] | 00:11 |
| openstack | Launchpad bug 1654615 in tripleo "Mitaka ha ping test failing to ping vm" [Critical,Triaged] - Assigned to Ben Nemec (bnemec) | 00:11 |
| *** jcoufal has quit IRC | 00:12 | |
| *** saneax-_-|AFK is now known as saneax | 00:16 | |
| *** dprince has quit IRC | 00:22 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras: WIP: Use openstack deploy to install the undercloud https://review.openstack.org/419040 | 00:25 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras: WIP: Use openstack deploy to install the undercloud https://review.openstack.org/419040 | 00:26 |
| *** ansmith_ has joined #tripleo | 00:29 | |
| *** ansmith has joined #tripleo | 00:30 | |
| *** inc0 has quit IRC | 00:32 | |
| *** jkilpatr has joined #tripleo | 00:40 | |
| *** dougbtv has quit IRC | 00:42 | |
| *** rbrady has quit IRC | 00:43 | |
| *** mburned_out is now known as mburned | 00:43 | |
| *** saneax is now known as saneax-_-|AFK | 00:46 | |
| *** limao has joined #tripleo | 00:47 | |
| *** salmankhan has joined #tripleo | 00:47 | |
| *** salmankhan has quit IRC | 00:52 | |
| *** dougbtv has joined #tripleo | 00:59 | |
| *** salmankhan has joined #tripleo | 01:03 | |
| *** salmankhan has quit IRC | 01:08 | |
| *** ooolpbot has joined #tripleo | 01:11 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 01:11 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1654611 | 01:11 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1654615 | 01:11 |
| *** ooolpbot has quit IRC | 01:11 | |
| openstack | Launchpad bug 1654611 in tripleo "Mitaka nonha ping test failing to upload image to glance" [Critical,Triaged] | 01:11 |
| openstack | Launchpad bug 1654615 in tripleo "Mitaka ha ping test failing to ping vm" [Critical,Triaged] - Assigned to Ben Nemec (bnemec) | 01:11 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: DONT MERGE: WIP: Multinode periodic tempest https://review.openstack.org/376237 | 01:17 |
| *** ansmith has quit IRC | 01:20 | |
| *** ansmith_ has quit IRC | 01:20 | |
| *** dsneddon has quit IRC | 01:54 | |
| EmilienM | bnemec: back online. CI still looks broken | 02:01 |
| EmilienM | well this patch passed well: https://review.openstack.org/#/c/398634/ | 02:02 |
| EmilienM | bnemec: puppet ci is broken as well | 02:05 |
| EmilienM | the ryu thing :( | 02:06 |
| *** ooolpbot has joined #tripleo | 02:11 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 02:11 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1654611 | 02:11 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1654615 | 02:11 |
| *** ooolpbot has quit IRC | 02:11 | |
| openstack | Launchpad bug 1654611 in tripleo "Mitaka nonha ping test failing to upload image to glance" [Critical,Triaged] | 02:11 |
| openstack | Launchpad bug 1654615 in tripleo "Mitaka ha ping test failing to ping vm" [Critical,Triaged] - Assigned to Ben Nemec (bnemec) | 02:11 |
| *** salmankhan has joined #tripleo | 02:27 | |
| *** salmankhan has quit IRC | 02:31 | |
| *** yamahata has quit IRC | 02:43 | |
| *** ooolpbot has joined #tripleo | 03:11 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 03:11 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1654611 | 03:11 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1654615 | 03:11 |
| *** ooolpbot has quit IRC | 03:11 | |
| openstack | Launchpad bug 1654611 in tripleo "Mitaka nonha ping test failing to upload image to glance" [Critical,Triaged] | 03:11 |
| openstack | Launchpad bug 1654615 in tripleo "Mitaka ha ping test failing to ping vm" [Critical,Triaged] - Assigned to Ben Nemec (bnemec) | 03:11 |
| *** bkopilov has quit IRC | 03:13 | |
| openstackgerrit | Lars Kellogg-Stedman proposed openstack/tripleo-heat-templates: implement a collectd composable service https://review.openstack.org/411048 | 03:18 |
| *** ramishra has quit IRC | 03:25 | |
| *** salmankhan has joined #tripleo | 03:27 | |
| *** ramishra has joined #tripleo | 03:27 | |
| *** apetrich has quit IRC | 03:28 | |
| *** apetrich has joined #tripleo | 03:28 | |
| openstackgerrit | RedHat RDO CI proposed openstack/tripleo-heat-templates: GATE TEST, please ignore https://review.openstack.org/365449 | 03:31 |
| *** salmankhan has quit IRC | 03:31 | |
| openstackgerrit | Ben Kero proposed openstack/tripleo-quickstart-extras: (undercloud|overcloud)-deploy: Add deployed_server script support https://review.openstack.org/417944 | 03:35 |
| *** links has joined #tripleo | 03:39 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras: WIP: Use openstack deploy to install the undercloud https://review.openstack.org/419040 | 03:44 |
| *** tzumainn has quit IRC | 03:56 | |
| openstackgerrit | Ben Kero proposed openstack-infra/tripleo-ci: WIP: Add OOOQ multinode test support https://review.openstack.org/416110 | 04:04 |
| *** ooolpbot has joined #tripleo | 04:11 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 04:11 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1654611 | 04:11 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1654615 | 04:11 |
| *** ooolpbot has quit IRC | 04:11 | |
| openstack | Launchpad bug 1654611 in tripleo "Mitaka nonha ping test failing to upload image to glance" [Critical,Triaged] | 04:11 |
| openstack | Launchpad bug 1654615 in tripleo "Mitaka ha ping test failing to ping vm" [Critical,Triaged] - Assigned to Ben Nemec (bnemec) | 04:11 |
| *** gkadam has joined #tripleo | 04:12 | |
| *** psahoo has joined #tripleo | 04:12 | |
| openstackgerrit | avnish proposed openstack/tripleo-quickstart: Use upper-constraints for tox envs https://review.openstack.org/419261 | 04:14 |
| *** sudipto has joined #tripleo | 04:29 | |
| *** sudipto_ has joined #tripleo | 04:29 | |
| *** skramaja has joined #tripleo | 04:30 | |
| *** psahoo_ has joined #tripleo | 04:34 | |
| *** nmathew- has joined #tripleo | 04:35 | |
| *** bkopilov has joined #tripleo | 04:38 | |
| *** salmankhan has joined #tripleo | 04:39 | |
| *** karthiks has joined #tripleo | 04:40 | |
| *** salmankhan has quit IRC | 04:42 | |
| *** karthiks has quit IRC | 04:46 | |
| *** salmankhan has joined #tripleo | 04:52 | |
| *** salmankhan has quit IRC | 04:56 | |
| *** saneax-_-|AFK is now known as saneax | 05:01 | |
| *** fzdarsky|afk has joined #tripleo | 05:01 | |
| *** karthiks has joined #tripleo | 05:06 | |
| *** fzdarsky|afk has quit IRC | 05:09 | |
| *** ooolpbot has joined #tripleo | 05:11 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 05:11 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1654611 | 05:11 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1654615 | 05:11 |
| *** ooolpbot has quit IRC | 05:11 | |
| openstack | Launchpad bug 1654611 in tripleo "Mitaka nonha ping test failing to upload image to glance" [Critical,Triaged] | 05:11 |
| openstack | Launchpad bug 1654615 in tripleo "Mitaka ha ping test failing to ping vm" [Critical,Triaged] - Assigned to Ben Nemec (bnemec) | 05:11 |
| *** prateek has joined #tripleo | 05:13 | |
| *** sudipto has quit IRC | 05:14 | |
| *** sudipto_ has quit IRC | 05:14 | |
| openstackgerrit | Ben Kero proposed openstack-infra/tripleo-ci: WIP: Add OOOQ multinode test support https://review.openstack.org/416110 | 05:17 |
| *** yamahata has joined #tripleo | 05:21 | |
| *** Vijayendra has quit IRC | 05:55 | |
| *** sudipto has joined #tripleo | 05:58 | |
| *** sudipto_ has joined #tripleo | 05:58 | |
| *** salmankhan has joined #tripleo | 05:59 | |
| *** salmankhan has quit IRC | 06:04 | |
| *** ooolpbot has joined #tripleo | 06:11 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 06:11 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1654611 | 06:11 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1654615 | 06:11 |
| *** ooolpbot has quit IRC | 06:11 | |
| openstack | Launchpad bug 1654611 in tripleo "Mitaka nonha ping test failing to upload image to glance" [Critical,Triaged] | 06:11 |
| openstack | Launchpad bug 1654615 in tripleo "Mitaka ha ping test failing to ping vm" [Critical,Triaged] - Assigned to Ben Nemec (bnemec) | 06:11 |
| *** Vijayendra has joined #tripleo | 06:11 | |
| *** absubram has quit IRC | 06:17 | |
| *** karthiks has quit IRC | 06:19 | |
| *** skramaja has quit IRC | 06:26 | |
| *** skramaja has joined #tripleo | 06:26 | |
| *** karthiks has joined #tripleo | 06:34 | |
| *** bana_k has joined #tripleo | 06:35 | |
| *** masco has joined #tripleo | 06:55 | |
| *** ealcaniz has joined #tripleo | 06:57 | |
| *** ccamacho has quit IRC | 07:00 | |
| *** ooolpbot has joined #tripleo | 07:11 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 07:11 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1654611 | 07:11 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1654615 | 07:11 |
| *** ooolpbot has quit IRC | 07:11 | |
| openstack | Launchpad bug 1654611 in tripleo "Mitaka nonha ping test failing to upload image to glance" [Critical,Triaged] | 07:11 |
| openstack | Launchpad bug 1654615 in tripleo "Mitaka ha ping test failing to ping vm" [Critical,Triaged] - Assigned to Ben Nemec (bnemec) | 07:11 |
| *** ealcaniz has quit IRC | 07:13 | |
| *** florianf has joined #tripleo | 07:14 | |
| *** tesseract has joined #tripleo | 07:14 | |
| *** iranzo has joined #tripleo | 07:20 | |
| *** ealcaniz has joined #tripleo | 07:21 | |
| *** prateek_ has joined #tripleo | 07:22 | |
| *** prateek has quit IRC | 07:22 | |
| *** achadha has joined #tripleo | 07:22 | |
| *** pgadiya has joined #tripleo | 07:23 | |
| *** rasca has joined #tripleo | 07:25 | |
| rasca | Buongiorno folks! | 07:26 |
| *** psahoo has quit IRC | 07:27 | |
| *** psahoo has joined #tripleo | 07:27 | |
| *** anton has quit IRC | 07:31 | |
| *** anton has joined #tripleo | 07:32 | |
| *** yprokule has joined #tripleo | 07:32 | |
| *** tremble has joined #tripleo | 07:34 | |
| *** tremble has quit IRC | 07:34 | |
| *** tremble has joined #tripleo | 07:34 | |
| *** sudipto has quit IRC | 07:36 | |
| *** sudipto_ has quit IRC | 07:36 | |
| *** bana_k has quit IRC | 07:36 | |
| *** cylopez has joined #tripleo | 07:37 | |
| *** achadha has quit IRC | 07:45 | |
| *** anton has quit IRC | 07:46 | |
| *** achadha has joined #tripleo | 07:47 | |
| *** achadha has quit IRC | 07:49 | |
| *** achadha has joined #tripleo | 07:49 | |
| *** anton has joined #tripleo | 07:51 | |
| *** mcornea has joined #tripleo | 08:00 | |
| *** mhenkel has joined #tripleo | 08:02 | |
| *** ccamacho has joined #tripleo | 08:02 | |
| *** Vijayendra has quit IRC | 08:03 | |
| *** ebarrera has joined #tripleo | 08:07 | |
| openstackgerrit | Raoul Scarazzini proposed openstack/tripleo-quickstart-extras: Add external repo and img roles to bm undercloud https://review.openstack.org/418328 | 08:07 |
| *** flepied has quit IRC | 08:07 | |
| *** mcornea has quit IRC | 08:09 | |
| *** ooolpbot has joined #tripleo | 08:11 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 08:11 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1654611 | 08:11 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1654615 | 08:11 |
| *** ooolpbot has quit IRC | 08:11 | |
| openstack | Launchpad bug 1654611 in tripleo "Mitaka nonha ping test failing to upload image to glance" [Critical,Triaged] | 08:11 |
| openstack | Launchpad bug 1654615 in tripleo "Mitaka ha ping test failing to ping vm" [Critical,Triaged] - Assigned to Ben Nemec (bnemec) | 08:11 |
| *** zoli|gone has quit IRC | 08:11 | |
| *** mcornea has joined #tripleo | 08:12 | |
| *** aufi has joined #tripleo | 08:12 | |
| *** pgadiya has quit IRC | 08:14 | |
| *** zoliXXL has joined #tripleo | 08:15 | |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Use the overcloudrc generated in a Mistral action https://review.openstack.org/410272 | 08:15 |
| *** jlinkes has joined #tripleo | 08:21 | |
| *** jpena|off is now known as jpena | 08:21 | |
| *** amoralej|off is now known as amoralej | 08:21 | |
| *** hjensas has quit IRC | 08:24 | |
| *** Vijayendra has joined #tripleo | 08:25 | |
| *** abehl has joined #tripleo | 08:30 | |
| *** sudipto_ has joined #tripleo | 08:31 | |
| *** sudipto has joined #tripleo | 08:31 | |
| *** jpich has joined #tripleo | 08:31 | |
| d0ugal | Seeing quite a bit of red on cistatus. | 08:31 |
| *** jpena is now known as jpena|off | 08:34 | |
| *** yamahata has quit IRC | 08:35 | |
| *** jpich has quit IRC | 08:37 | |
| *** jprovazn has joined #tripleo | 08:38 | |
| *** jpich has joined #tripleo | 08:38 | |
| *** jpena|off is now known as jpena | 08:40 | |
| *** kberger has quit IRC | 08:41 | |
| *** nmathew- has quit IRC | 08:43 | |
| *** arxcruz has quit IRC | 08:45 | |
| *** arxcruz has joined #tripleo | 08:46 | |
| *** pgadiya has joined #tripleo | 08:47 | |
| *** flepied has joined #tripleo | 08:48 | |
| *** hjensas has joined #tripleo | 08:49 | |
| *** hjensas has quit IRC | 08:49 | |
| *** hjensas has joined #tripleo | 08:49 | |
| *** zoliXXL is now known as zoli|brb | 08:49 | |
| *** pmannidi has quit IRC | 08:49 | |
| *** skramaja has quit IRC | 08:52 | |
| *** yprokule_ has joined #tripleo | 08:57 | |
| *** athomas has joined #tripleo | 08:58 | |
| *** yprokule has quit IRC | 08:58 | |
| *** yprokule_ is now known as yprokule | 08:58 | |
| *** zzzeek has quit IRC | 09:00 | |
| *** zzzeek has joined #tripleo | 09:00 | |
| *** david-lyle has quit IRC | 09:01 | |
| *** pgadiya has quit IRC | 09:03 | |
| *** pgadiya has joined #tripleo | 09:06 | |
| d0ugal | Any guesses what this failure means? http://logs.openstack.org/14/417314/3/check/gate-tripleo-ci-centos-7-nonha-multinode/306fe03/console.html#_2017-01-12_09_06_54_442220 | 09:09 |
| d0ugal | oh, I see a bit further down "not enough hosts" | 09:10 |
| d0ugal | Is this one I should just be rechecking? It has already been rechecked a couple of times (but different errors AFAICT) | 09:10 |
| *** ooolpbot has joined #tripleo | 09:11 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 09:11 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1654611 | 09:11 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1654615 | 09:11 |
| *** ooolpbot has quit IRC | 09:11 | |
| openstack | Launchpad bug 1654611 in tripleo "Mitaka nonha ping test failing to upload image to glance" [Critical,Triaged] | 09:11 |
| openstack | Launchpad bug 1654615 in tripleo "Mitaka ha ping test failing to ping vm" [Critical,Triaged] - Assigned to Ben Nemec (bnemec) | 09:11 |
| openstackgerrit | Ben Kero proposed openstack/tripleo-quickstart-extras: (undercloud|overcloud)-deploy: Add deployed_server script support https://review.openstack.org/417944 | 09:20 |
| *** nmathew has joined #tripleo | 09:21 | |
| *** hewbrocca_afk is now known as hewbrocca | 09:22 | |
| *** lucas-afk is now known as lucasagomes | 09:22 | |
| *** yprokule_ has joined #tripleo | 09:24 | |
| *** pgadiya has quit IRC | 09:24 | |
| *** david-lyle has joined #tripleo | 09:24 | |
| *** shardy has joined #tripleo | 09:25 | |
| *** yprokule has quit IRC | 09:25 | |
| *** yprokule_ is now known as yprokule | 09:25 | |
| *** panda|Zz is now known as panda | 09:26 | |
| *** gfidente has joined #tripleo | 09:27 | |
| *** gfidente has quit IRC | 09:27 | |
| *** gfidente has joined #tripleo | 09:27 | |
| *** pgadiya has joined #tripleo | 09:40 | |
| *** zoli|brb is now known as zoli | 09:41 | |
| *** pgadiya_ has joined #tripleo | 09:41 | |
| *** nyechiel_ has joined #tripleo | 09:44 | |
| *** pgadiya has quit IRC | 09:45 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/tripleo-common: Updated from global requirements https://review.openstack.org/419360 | 09:46 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/tripleo-heat-templates: Updated from global requirements https://review.openstack.org/419361 | 09:46 |
| matbu | shardy: hey morning, can you rebase your reviews for neutron and nova upgrade ? | 09:46 |
| *** owalsh-afk has joined #tripleo | 09:47 | |
| matbu | shardy: then i would be able to add reviews on top of yours | 09:47 |
| *** salmankhan has joined #tripleo | 09:47 | |
| shardy | matbu: sure, I wanted to get glance passing but I can rebase the whole series this morning | 09:47 |
| chem | shardy: hi, I was about to rebase 405240 (glance) for https://bugs.launchpad.net/tripleo/+bug/1655651 but it seems there is a chain of dep before to rebase | 09:48 |
| openstack | Launchpad bug 1655651 in tripleo "Unable to rerun Ansible upgrade tasks for services which contain wildcards for their names" [Critical,Confirmed] - Assigned to Sofer Athlan-Guyot (sofer-athlan-guyot) | 09:48 |
| openstackgerrit | Flavio Percoco proposed openstack/python-tripleoclient: not merge: undercloud container https://review.openstack.org/413732 | 09:49 |
| openstackgerrit | Flavio Percoco proposed openstack/python-tripleoclient: Add `--keep-running` flag to undercloud deploy https://review.openstack.org/414490 | 09:49 |
| shardy | chem: why does the glance patch need rebasing? | 09:49 |
| shardy | chem: I rebased that one yesterday to test the upgrade CI | 09:49 |
| *** owalsh-afk is now known as owalsh | 09:49 | |
| chem | shardy: cannot merge today | 09:49 |
| shardy | but I'll rebase the whole series now so we've got a branch to test from | 09:49 |
| shardy | https://review.openstack.org/#/c/405240/ " Merge if Necessary" | 09:50 |
| chem | shardy: sorry swift | 09:50 |
| chem | shardy: not glance | 09:50 |
| shardy | chem: Ok, yeah I only rebased glance, I'll do the whole branch, sec | 09:50 |
| chem | shardy: oki, thanks. | 09:50 |
| * shardy was trying to avoid too much CI load as we had a lot of queueing jobs yesterday | 09:50 | |
| d0ugal | shardy: cistatus is looking pretty red this morning. | 09:52 |
| d0ugal | for the obv jobs anyway | 09:53 |
| shardy | d0ugal: Yeah, not got to looking at those yet | 09:53 |
| shardy | I think I'll have to rebase this series anyway as folks need a branch to test | 09:53 |
| d0ugal | I am attempting to try and understand it :) | 09:53 |
| d0ugal | sure | 09:53 |
| * cmyster tells a mistrall joke: why did json crossed the road except the road had an exception | 09:55 | |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Add snmp service support for composable upgrades https://review.openstack.org/405250 | 09:57 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Add mistral service support for composable upgrades https://review.openstack.org/405246 | 09:57 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Add sahara service support for composable upgrades https://review.openstack.org/405247 | 09:57 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Add ironic service support for composable upgrades https://review.openstack.org/405248 | 09:57 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Add heat service support for composable upgrades https://review.openstack.org/405244 | 09:57 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Add telemetry service support for composable upgrades https://review.openstack.org/405249 | 09:57 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Add swift service support for composable upgrades https://review.openstack.org/405245 | 09:57 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Add neutron service support for composable upgrades https://review.openstack.org/405242 | 09:57 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Add cinder service support for composable upgrades https://review.openstack.org/405243 | 09:57 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Add glance service support for composable upgrades https://review.openstack.org/405240 | 09:57 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Add nova service support for composable upgrades https://review.openstack.org/405241 | 09:57 |
| shardy | matbu, chem: ^^ | 09:57 |
| chem | shardy: thanks | 09:57 |
| shardy | chem: I've not fixed the swift patch yet, feel free to push a fix if you have it | 09:57 |
| matbu | shardy: thx | 09:57 |
| chem | shardy: yep, will do in a sec | 09:57 |
| shardy | matbu: I'll try enabling more services in CI, enough that we can run a basic pingtest | 09:58 |
| cmyster | shardy: any chance you can cipher something for me? | 09:58 |
| cmyster | something that passes ^ but breaks down | 09:58 |
| matbu | shardy: yes, i have a look at your review (sanity check), looks good | 09:58 |
| shardy | matbu: I was going to implement sanity tests for each service, but it may be quicker if we just enable a bunch of services then run pingtest | 09:58 |
| matbu | shardy: i think we can land the neutron review | 09:58 |
| matbu | shardy: and maybe pacemaker | 09:59 |
| openstackgerrit | Shashi proposed openstack/os-cloud-config: Removes unnecessary utf-8 encoding https://review.openstack.org/419381 | 09:59 |
| matbu | shardy: i was able to get a working upgrade with pacemaker and network isolation | 09:59 |
| marios | matbu: shardy chem o/ hey guys | 09:59 |
| matbu | shardy: neutron needs a workaround | 09:59 |
| matbu | marios: morning | 10:00 |
| shardy | matbu: Ok that's great news :) | 10:00 |
| shardy | matbu: do we need an etherpad to track which patches you used, then we can hopefully get reviews and land them? | 10:00 |
| chem | marios: hi | 10:00 |
| matbu | shardy: yep so now i'm deploying with a compute nodes | 10:00 |
| marios | shardy: matbu chem what do you guys think about doing this https://review.openstack.org/#/c/414598/ (i.e. sanity check before running the upgrades). it extends the basic checks we had before (which were mostly check cluster is ok) | 10:00 |
| matbu | shardy: sure good idea | 10:00 |
| shardy | then we can work on getting the upgrade CI to test w/pacemaker etc | 10:00 |
| shardy | hey marios | 10:01 |
| *** salmankhan has quit IRC | 10:01 | |
| owalsh | shardy: just occurred to me - UpgradeLevelNovaCompute will not be set to auto until after the Ocata upgrade | 10:01 |
| *** salmankhan has joined #tripleo | 10:01 | |
| marios | shardy: matbu chem its been in review for a while and i know shardy had some questions on whether it is a sane thing to do. chem has a comment about not using 'service' and instead just trying to detect | 10:01 |
| marios | chem: ideall yes this is what i wanted to do but i found that the 'check and fail/exit' isn't really a pattern in the ansible modules | 10:01 |
| marios | chem: i.e. it is like 'make it so' not just check | 10:01 |
| shardy | owalsh: We set it during the upgrade with crudini | 10:01 |
| shardy | same as when we pinned it | 10:01 |
| shardy | owalsh: then puppet will later write the same config | 10:02 |
| shardy | Or, that's how I expect it to work at least :) | 10:02 |
| owalsh | shardy: but we would need the computes to have this before the upgrade | 10:02 |
| *** gkadam is now known as gkadam|mtg | 10:02 | |
| chem | marios: in the comment I provide a ansible snippet that would do the job, If I remember correctly | 10:02 |
| owalsh | shardy: if we wanted to do an all in one upgrade | 10:02 |
| marios | owalsh: shardy so in previous upgrades we upgrade+pin controllers, upgrade+pin compute, unpin everything (converge) | 10:03 |
| marios | owalsh: i.e. previously we pin controllers first as they are upgraded first | 10:03 |
| *** achadha has quit IRC | 10:04 | |
| owalsh | marios: yea | 10:04 |
| shardy | owalsh: Oh right :( | 10:04 |
| marios | chem: thanks for that i will try it .. i was more asking if people thought it is worth adding this check for all the services. i thought it was cool that we could do this now with composable | 10:04 |
| *** nijaba has quit IRC | 10:04 | |
| shardy | owalsh: honestly, I'm not seeing a good way to do this other than run a minor update which does things on the computes, then the major upgrade | 10:04 |
| *** achadha has joined #tripleo | 10:04 | |
| *** achadha has quit IRC | 10:04 | |
| matbu | marios: chem i'm commenting the ansible snippet in the review :) | 10:04 |
| shardy | which is kinda dissapointing given we've just figured out how to remove at least one stack update from the upgrade experience | 10:05 |
| *** nijaba has joined #tripleo | 10:05 | |
| *** nijaba has quit IRC | 10:05 | |
| *** nijaba has joined #tripleo | 10:05 | |
| marios | shardy: how about an extra softwareconfig that the upgrade init depends on for example to deliver any cluster wide stuff? like this rpc pin && deliver non controller upgrade scripts | 10:05 |
| shardy | marios: I think the pre-flight checks are fine | 10:05 |
| shardy | marios: yeah I guess we could do this in the code that delivers the compute upgrade script | 10:06 |
| matbu | marios: chem but i don't mind, i'm more in favor of making iteration than having the perfect review that took 1 month to get merged :) we can improve the code later | 10:06 |
| openstackgerrit | Ben Kero proposed openstack/tripleo-quickstart-extras: (undercloud|overcloud)-deploy: Add deployed_server script support https://review.openstack.org/417944 | 10:06 |
| shardy | marios: that's kinda ugly though, as the whole point of all our discussions yesterday was to not touch the computes | 10:06 |
| *** fzdarsky|afk has joined #tripleo | 10:07 | |
| shardy | now we're saying we upgrade them a little bit, then stop | 10:07 |
| shardy | despite a nice flag saying upgrade is disabled for computes ... | 10:07 |
| marios | shardy: yeah :( for the record, i'm still not clear what the problem is i just take from the earlier irc from owalsh that we need to pin compute first for rpc | 10:07 |
| openstackgerrit | Ben Kero proposed openstack-infra/tripleo-ci: WIP: Add OOOQ multinode test support https://review.openstack.org/416110 | 10:07 |
| marios | shardy: this time round for _reasons_ | 10:07 |
| shardy | marios: http://lists.openstack.org/pipermail/openstack-dev/2017-January/109875.html | 10:08 |
| *** derekh has joined #tripleo | 10:08 | |
| marios | shardy: if it is a matter of one-line bash crudini to do that in the same software config that also delivers the upgrade script, might be ok... | 10:08 |
| marios | shardy: thanks looking at the link | 10:08 |
| shardy | tl;dr nova has introduced some inconvenient requirements around upgrades, meaning you have to do stuff on the computes, then the controller, then the computes | 10:08 |
| marios | shardy: thanks | 10:09 |
| owalsh | marios, shardy: also http://docs.openstack.org/developer/nova/placement.html#deployment | 10:10 |
| owalsh | "The placement-api service must be deployed at some point after you have upgraded to the 14.0.0 Newton release but before you can upgrade to the 15.0.0 Ocata release." | 10:10 |
| openstackgerrit | Shashi proposed openstack/os-net-config: Removes unnecessary utf-8 encoding https://review.openstack.org/419390 | 10:11 |
| *** ooolpbot has joined #tripleo | 10:11 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 10:11 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1654611 | 10:11 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1654615 | 10:11 |
| *** ooolpbot has quit IRC | 10:11 | |
| openstack | Launchpad bug 1654611 in tripleo "Mitaka nonha ping test failing to upload image to glance" [Critical,Triaged] | 10:11 |
| marios | shardy: i am going to poke at the non-controller script delivery later/plan to but right now am trying to work out why i get "argument of type 'NoneType' is not iterable" trying to run https://review.openstack.org/#/c/418344/ (the generated templates look fine to me so not sure yet what/why/if related to one of the other reviews I applied in the env) | 10:11 |
| openstack | Launchpad bug 1654615 in tripleo "Mitaka ha ping test failing to ping vm" [Critical,Triaged] - Assigned to Ben Nemec (bnemec) | 10:11 |
| sshnaidm|away | is neutron issue "No module named tinyrpc.server" known for somebody? | 10:11 |
| marios | owalsh: thanks | 10:11 |
| marios | owalsh: shardy so in fact do we need to deploy/configure a new service as well | 10:12 |
| *** limao has quit IRC | 10:12 | |
| *** sshnaidm|away is now known as sshnaidm | 10:12 | |
| owalsh | marios, shardy: current plan is to configure the newton computes (clients), then just deploy the service on the controller in the ocata upgrade | 10:12 |
| marios | owalsh: shardy OK... still a bit more than a simple crudini though i mean 'configure newton computes' ... am just thinking about it in terms of it happening as part of the 'deliver the upgrade scripts' step... | 10:14 |
| openstackgerrit | Itzik Brown proposed openstack/tripleo-heat-templates: Removes deprecated OpenDaylight L2 only deployments https://review.openstack.org/419394 | 10:15 |
| owalsh | marios: this all depends on running puppet during the controller upgrade (so no more converge step) | 10:15 |
| owalsh | marios: so we can't use nova version pinning | 10:16 |
| marios | owalsh: "this all depends" ... the setup of the placement service on controllers you mean? and for that to happen we need it also configured on the computes first | 10:16 |
| *** dsariel has joined #tripleo | 10:17 | |
| owalsh | marios: computes must be configured so that they connect to the placement service once it's running | 10:18 |
| owalsh | marios: otherwise we are looking at downtime until all of the computes are upgraded to ocata | 10:18 |
| marios | owalsh: ack thanks owalsh so the pinning is no longer a concern but we do need to configure compute to talk to placement service, before we actually deploy placement during the controller upgrade run | 10:19 |
| *** salmankhan has quit IRC | 10:19 | |
| marios | owalsh: so whilst we don't need to upgrade the computes/services on computes first we do need to do that config first on the compute service. | 10:20 |
| shardy | marios: yeah that sounds right | 10:20 |
| owalsh | marios: yes, exactly, need to placement api creds https://review.openstack.org/#/c/406300/14/manifests/placement.pp | 10:20 |
| shardy | marios: and, we have to run puppet as part of the upgrade of the controller, so it configures and starts the placement API | 10:20 |
| shardy | then, we should be good to have folks upgrade the computes as before | 10:21 |
| *** zoli is now known as zoli|relocating | 10:21 | |
| matbu | marios: your NoneType issue looks not related to the review ^ | 10:23 |
| *** tosky has joined #tripleo | 10:23 | |
| *** owalsh is now known as owalsh-brb | 10:24 | |
| matbu | marios: it's about a bad loop i guess | 10:24 |
| marios | matbu: do you mean you also saw it somewhere else/without that review? | 10:24 |
| marios | owalsh-brb: shardy thanks for the explanation it is clearer now | 10:25 |
| matbu | marios: nop, but may be you have another patch applied which cause the issue | 10:26 |
| marios | matbu: ah yeah i hope so/otherwise cant quickly see the error (I'm playing with the rendered major_upgrade_steps.yaml ( shardy the tox -e templates is very cool thanks for the headsup on that) | 10:27 |
| *** apetrich has quit IRC | 10:28 | |
| *** salmankhan has joined #tripleo | 10:29 | |
| *** salmankhan has quit IRC | 10:33 | |
| *** owalsh-brb is now known as owalsh | 10:37 | |
| *** gfidente has quit IRC | 10:42 | |
| *** nmathew has quit IRC | 10:42 | |
| *** pgadiya_ has quit IRC | 10:47 | |
| *** shardy has quit IRC | 10:51 | |
| *** xuao has joined #tripleo | 10:51 | |
| *** jd__ has quit IRC | 10:52 | |
| *** saneax is now known as saneax-_-|AFK | 10:55 | |
| jpich | jtomasek, florianf: Looks like the behaviour of some of the workflows is going to change to properly fail when there is an issue ( https://review.openstack.org/#/q/topic:bug/1651704 ), is there something we need to change from the UI side to take this into account? | 10:56 |
| *** salmankhan has joined #tripleo | 10:57 | |
| jtomasek | jpich: we will, afaik the validations will need to reflect that | 10:57 |
| jpich | jtomasek: Ok, I'll open a high priority bug targeted to ocata-3 that references this then so we keep track of it | 10:59 |
| jtomasek | d0ugal: are those changes making the workflow status to failed? | 10:59 |
| *** paramite has joined #tripleo | 10:59 | |
| jtomasek | d0ugal: did you verify that such failed workflows still produce output? as I think that was the issue I looked at it last time | 11:00 |
| d0ugal | jtomasek: what workflow output do you want/neet? | 11:00 |
| d0ugal | need* | 11:00 |
| jtomasek | d0ugal: https://review.openstack.org/#/c/418968/1 | 11:01 |
| jtomasek | d0ugal: when validation fails, it needs to produce output | 11:01 |
| d0ugal | jtomasek: Can you give me an example? I'm not really sure how to test it. | 11:01 |
| jtomasek | d0ugal: when we worked on the workflow and we made that workflow fail, it did not return any output because of it | 11:02 |
| *** salmankhan has quit IRC | 11:02 | |
| *** salmankhan has joined #tripleo | 11:02 | |
| d0ugal | jtomasek: Don't you just read messages from Zaqar? | 11:02 |
| *** pgadiya_ has joined #tripleo | 11:03 | |
| jtomasek | d0ugal: not all the time. for example when we list all validations, we use workflow executions to track last result of that workflow to get the status of that validation | 11:03 |
| d0ugal | jtomasek: okay, I'll see how different the output looks. | 11:04 |
| d0ugal | I wasn't aware the UI read execution from Mistral. | 11:05 |
| jtomasek | d0ugal: looking at line 62 https://review.openstack.org/#/c/418968/1/workbooks/validations.yaml | 11:08 |
| jtomasek | d0ugal: I am confused why fail workflow on-success of send-message | 11:09 |
| d0ugal | jtomasek: :) | 11:09 |
| d0ugal | jtomasek: http://www.dougalmatthews.com/2017/Jan/09/mistral-flow-control/ | 11:09 |
| jtomasek | d0ugal: I am probably missinterpreting that code... | 11:09 |
| d0ugal | jtomasek: that may help explain the logic | 11:09 |
| jtomasek | d0ugal: thanks | 11:10 |
| d0ugal | jtomasek: but tl;dr "fail_workflow" is called if the expression on the right is true | 11:10 |
| jtomasek | d0ugal: aha! TIL | 11:10 |
| d0ugal | jtomasek: so fail_workflow is only called if the message status is "FAILED" | 11:10 |
| jtomasek | d0ugal: very nice | 11:10 |
| d0ugal | It's useful, but it is confusing. | 11:10 |
| *** ooolpbot has joined #tripleo | 11:11 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 11:11 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1654611 | 11:11 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1654615 | 11:11 |
| *** ooolpbot has quit IRC | 11:11 | |
| openstack | Launchpad bug 1654611 in tripleo "Mitaka nonha ping test failing to upload image to glance" [Critical,Triaged] | 11:11 |
| openstack | Launchpad bug 1654615 in tripleo "Mitaka ha ping test failing to ping vm" [Critical,Triaged] - Assigned to Ben Nemec (bnemec) | 11:11 |
| *** xuao has quit IRC | 11:11 | |
| sshnaidm | FYI, https://bugs.launchpad.net/tripleo/+bug/1655919 | 11:11 |
| openstack | Launchpad bug 1655919 in tripleo "Openvswitch agent fails because can't import tinyrpc.server module" [Undecided,New] | 11:11 |
| jpich | d0ugal: Do you know if the plan is to backport? If it's changing expectations from the workflows consumer side there may be issues... (probably at least worth a release note for master, too!) | 11:11 |
| d0ugal | jpich: no plans to backport. A release note probably makes sense if the result is different, testing now. | 11:12 |
| jtomasek | d0ugal: so the way validations worked so far is that even though validation itself failed, it did not make the workflow fail (workflow succeeds but validation fails - makes quite sense). Also this allowed us to keep the output of the validation in workflow execution | 11:13 |
| *** milan has joined #tripleo | 11:13 | |
| jtomasek | d0ugal: I am trying to figure out if your patch changes that behavior or not | 11:13 |
| d0ugal | jtomasek: it will make the workflow fail, if the validation fails - which I think makes more sense. | 11:14 |
| d0ugal | so I am checking the workflow output | 11:14 |
| *** shardy has joined #tripleo | 11:14 | |
| jpich | d0ugal: Cool, that's one less worry! We already know part of the result (the workflow status) is different so, I think that'd be a useful FYI for other workflow consumers regardless in case people were relying on it | 11:14 |
| jtomasek | d0ugal: ok, yes makes sense, problem is that afaik in that case the execution won't have any output which means we won't be able to display reason of failure of that validation | 11:15 |
| jtomasek | d0ugal: unless something changed in mistral in the meantime... thanks for looking into it | 11:15 |
| d0ugal | jpich: yeah, makes sense. it does raise the question about what is considered a backwards compatible change and what is allowed. | 11:15 |
| d0ugal | jtomasek: sure, if needed I'll block the validation workflow change and try to change this in Mistral. | 11:16 |
| jtomasek | d0ugal: thanks | 11:16 |
| d0ugal | jtomasek, jpich - how do I run a validation? :) | 11:16 |
| d0ugal | (without the UI!) | 11:16 |
| jpich | d0ugal: http://tripleo.org/validations/validations.html :) | 11:17 |
| jpich | http://tripleo.org/validations/validations.html#running-a-single-validation | 11:17 |
| jtomasek | impressive! | 11:17 |
| d0ugal | oh my, docs! | 11:18 |
| *** akrivoka has joined #tripleo | 11:18 | |
| jpich | shadower++ | 11:18 |
| *** masco has quit IRC | 11:19 | |
| *** jd__ has joined #tripleo | 11:19 | |
| jtomasek | d0ugal: actually when I think about it, it would be quite nice in general if a failed workflow could produce an output. I am not sure why it would not | 11:19 |
| *** gfidente has joined #tripleo | 11:20 | |
| *** gfidente has quit IRC | 11:20 | |
| *** gfidente has joined #tripleo | 11:20 | |
| d0ugal | jtomasek: agreed. I feel like output might even be more important for a failed workflow :) | 11:20 |
| jpich | d0ugal: We do have versioning for our workflows but no clear idea of when/how much change is appropriate to bump them... Maybe it'd make sense to do it here? (That'd only mean updating the client in addition to this, and give other consumers time to adapt as needed/make sure they're not affected?) I'd assume huge changes would be more likely to result in new workflows/workbooks, rather than version bumps of the existing ones... I could be wrong though! | 11:20 |
| d0ugal | jpich: doesn't bumping the workflow mean duplicating it and then maintaining two versions? :( | 11:21 |
| *** sudipto has quit IRC | 11:22 | |
| *** sudipto_ has quit IRC | 11:22 | |
| jpich | d0ugal: Oh... Hrm. Wouldn't it be ok to mark the old one as deprecated and focus on the new one though? | 11:23 |
| jpich | if the backwards-incompatible changes remain minor, consumers will move to the new one to get the newer stuff | 11:23 |
| jpich | ...or this is too naive :) | 11:23 |
| *** fzdarsky|afk is now known as fzdarsky | 11:25 | |
| *** zoli|relocating is now known as zoli | 11:28 | |
| d0ugal | jtomasek: http://paste.openstack.org/show/594679/ | 11:28 |
| *** zoli is now known as zoliXXL | 11:28 | |
| *** dtantsur|afk is now known as dtantsur | 11:28 | |
| d0ugal | jpich: How do we tell them the original is deprecated? | 11:28 |
| d0ugal | jtomasek, jpich: ^ so it looks like stdout is still there. there is now an extra "result" | 11:29 |
| jpich | d0ugal: workflow description update + release note? I suppose there's no way to print deprecation warnings... | 11:29 |
| d0ugal | jpich: who reads release notes? :) | 11:30 |
| d0ugal | jpich: I think it is going to be a tricky problem to solve well | 11:30 |
| * jtomasek checks GUI code | 11:31 | |
| d0ugal | jpich: I guess we could add an deprecated action that logs/prints a deprecation warning. | 11:31 |
| d0ugal | jpich: we could even send zaqar messages "This workflow will self destruct..." | 11:32 |
| jpich | d0ugal: In the Mistral logs? | 11:32 |
| jpich | lol | 11:32 |
| *** zoliXXL is now known as zoli|lunch | 11:32 | |
| d0ugal | jpich: yeah, mistral logs, which isn't that useful - but somebody *might* see it | 11:32 |
| jtomasek | d0ugal: ok, so it looks great, the output is there, all good:) thanks for checking that | 11:33 |
| jpich | d0ugal: I'm not sure about the Zaqar messages, since this would affect users rather than deployers/whoever wrote the code to consume the workflow | 11:33 |
| *** salmankhan has quit IRC | 11:33 | |
| d0ugal | jtomasek: yay, that is nice and easy then. | 11:33 |
| *** salmankhan has joined #tripleo | 11:33 | |
| d0ugal | jpich: deprecation warnings are fairly common on the CLI | 11:33 |
| jtomasek | it must have been changed in mistral quite recently | 11:34 |
| jpich | d0ugal: True, but that's stuff you can act on as a user like "use that other command instead" | 11:34 |
| d0ugal | jpich: not always, sometimes the deprecations are from other python packages we use. I openstackclient made quite a fuss a while ago when they moved to osc_lib | 11:35 |
| d0ugal | but anyway, I think we mostly agree. I just don't know of a good solution :( | 11:35 |
| jpich | d0ugal: Yeah! I agree it's a tricky problem | 11:35 |
| *** akrivoka has quit IRC | 11:36 | |
| *** akrivoka has joined #tripleo | 11:39 | |
| jpich | d0ugal: If it doesn't break any of our 3 known consumers (UI, CLI, manageiq?) maybe we're ok with just the "FYI BTW" release note about the workflow status change for this time, because it generally has a low impact? | 11:39 |
| *** gkadam|mtg has quit IRC | 11:40 | |
| d0ugal | jpich: Yeah, makes sense. I'll wait for EmilienM to add reno then :) | 11:41 |
| *** fzdarsky has quit IRC | 11:41 | |
| d0ugal | EmilienM: also, FWIW, sign me up for helping with release notes for tripleoclient/tripleo-common. | 11:41 |
| jpich | d0ugal: Looks like it's here but failing - https://review.openstack.org/#/c/419140/ * poking around * | 11:41 |
| d0ugal | jpich: oh, I missed that. | 11:42 |
| jpich | EmilienM is fast!! | 11:42 |
| d0ugal | "2017-01-11 22:20:44.496370 | ImportError: No module named tripleocommon" | 11:42 |
| d0ugal | hmm | 11:42 |
| d0ugal | it is missing the _ | 11:43 |
| d0ugal | but I don't know where that comes from | 11:43 |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common: Add ReNo support https://review.openstack.org/419140 | 11:44 |
| jpich | d0ugal: Well spotted!! | 11:46 |
| jpich | I'll review when we have the releasenotes job result | 11:47 |
| d0ugal | jpich: thanks | 11:47 |
| *** prateek has joined #tripleo | 11:47 | |
| openstackgerrit | Radoslaw Smigielski proposed openstack/tripleo-heat-templates: Add new HAProxyDefaultTimeout option https://review.openstack.org/419433 | 11:48 |
| *** jkilpatr has quit IRC | 11:49 | |
| *** shardy has quit IRC | 11:55 | |
| *** gfidente has quit IRC | 11:55 | |
| *** shardy has joined #tripleo | 11:56 | |
| *** gfidente has joined #tripleo | 11:56 | |
| *** gfidente has quit IRC | 11:56 | |
| *** gfidente has joined #tripleo | 11:56 | |
| *** fzdarsky has joined #tripleo | 11:57 | |
| panda | sshnaidm: are we collecting /home/jenkins in tripleo-ci jobs ? I don't rememer and I don't seem to see it anywhere | 11:58 |
| *** saneax-_-|AFK is now known as saneax | 11:58 | |
| sshnaidm | panda, in non-oooq? | 11:58 |
| panda | sshnaidm: yes | 11:59 |
| sshnaidm | panda, nope, only /etc and /var/log | 11:59 |
| panda | sshnaidm: great ... I have to reproduce manually then | 11:59 |
| *** bkopilov has quit IRC | 11:59 | |
| *** fzdarsky has quit IRC | 11:59 | |
| sshnaidm | panda, reproduce what? | 11:59 |
| *** jprovazn has quit IRC | 12:00 | |
| *** fzdarsky has joined #tripleo | 12:00 | |
| panda | sshnaidm: https://bugs.launchpad.net/tripleo/+bug/1655938 | 12:01 |
| openstack | Launchpad bug 1655938 in tripleo "CI: overcloud image creation fails on all periodic jobs" [Critical,Triaged] | 12:01 |
| *** salmankhan has quit IRC | 12:01 | |
| *** gfidente has quit IRC | 12:01 | |
| sshnaidm | panda, I think it's copied to /var/log | 12:01 |
| sshnaidm | panda, Juan even did a patch for saving it there | 12:02 |
| *** gfidente has joined #tripleo | 12:02 | |
| *** gfidente has quit IRC | 12:02 | |
| sshnaidm | panda, iirc there is "tee" too | 12:02 |
| panda | sshnaidm: we're copying something more than http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-nonha/68f0aac/logs/undercloud/var/log/image_build.txt.gz ? | 12:02 |
| *** fultonj has joined #tripleo | 12:03 | |
| panda | 13:03 $ grep -r overcloud-image-build.log * | 12:03 |
| panda | scripts/tripleo.sh: tee -a overcloud-image-build.log | 12:03 |
| panda | in tripleo-ci dir | 12:03 |
| *** jprovazn has joined #tripleo | 12:03 | |
| sshnaidm | panda, whoops, seems like we do save /home/jenkins in tar.gz file, only these build image files | 12:04 |
| sshnaidm | panda, I thought they're copied to /var/log.. | 12:04 |
| jpich | d0ugal: Looks like the fun continues http://logs.openstack.org/40/419140/2/check/gate-tripleo-common-releasenotes/d0f9974/console.html#_2017-01-12_11_48_58_783712 | 12:04 |
| *** jpena is now known as jpena|lunch | 12:05 | |
| sshnaidm | panda, tar.gz contains everything you need | 12:05 |
| d0ugal | jpich: already updating the patch | 12:05 |
| jpich | d0ugal: :D | 12:05 |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common: Add ReNo support https://review.openstack.org/419140 | 12:06 |
| sshnaidm | panda, but it's definitely better to save them in /var/log | 12:06 |
| *** fzdarsky has quit IRC | 12:06 | |
| panda | sshnaidm: you mean http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-nonha/68f0aac/logs/undercloud.tar.xz has everything I need ? | 12:06 |
| sshnaidm | panda, yep | 12:06 |
| panda | sshnaidm: looking | 12:07 |
| jpich | d0ugal: Passing for me locally now :) | 12:08 |
| panda | sshnaidm: lol | 12:08 |
| d0ugal | jpich: nice | 12:08 |
| d0ugal | jpich: I was too lazy to try that :P | 12:08 |
| panda | sshnaidm: Command '['disk-image-create', '-a', 'amd64', '-o', u'/home/jenkins/overcloud-full.qcow2', '-t', 'qcow2', '-p', 'python-psutil,python-debtcollector,plotnetcfg,sos,device-mapper-multipath,python-heat-agent-puppet,python-heat-agent-hiera,python-heat-agent-apply-config,python-heat-agent-ansible,yum-plugin-priorities', '--min-tmpfs', '5', 'centos7', 'baremetal', 'dhcp-all-interfaces', | 12:08 |
| panda | 'overcloud-agent', 'overcloud-full', 'overcloud-controller', 'overcloud-compute', 'overcloud-ceph-storage', 'puppet-modules', 'hiera', 'os-net-config', 'stable-interface-names', 'grub2', 'element-manifest', 'dynamic-login', 'enable-packages-install', 'pip-and-virtualenv-override', 'selinux-permissive']' returned non-zero exit status 1 | 12:08 |
| panda | sshnaidm: ah, overcloud-full.qcow2.log contains a lot more | 12:08 |
| sshnaidm | Error downloading packages: | 12:09 |
| sshnaidm | 1:openstack-dashboard-11.0.0-0.20170112045845.04f88bd.el7.centos.noarch: [Errno 256] No more mirrors to try. | 12:09 |
| sshnaidm | 1:python-django-horizon-11.0.0-0.20170112045845.04f88bd.el7.centos.noarch: [Errno 256] No more mirrors to try. | 12:09 |
| sshnaidm | https://trunk.rdoproject.org/centos7/d5/7c/d57ce1dda839865c8060458760fa27dbbd7e2aee_73ab70fb/python-django-horizon-11.0.0-0.20170112045845.04f88bd.el7.centos.noarch.rpm: [Errno 14] HTTPS Error 403 - Forbidden | 12:10 |
| sshnaidm | forbidden, how is rude | 12:11 |
| *** ooolpbot has joined #tripleo | 12:11 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 12:11 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1654611 | 12:11 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1654615 | 12:11 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1655919 | 12:11 |
| *** ooolpbot has quit IRC | 12:11 | |
| openstack | Launchpad bug 1654611 in tripleo "Mitaka nonha ping test failing to upload image to glance" [Critical,Triaged] | 12:11 |
| openstack | Launchpad bug 1654615 in tripleo "Mitaka ha ping test failing to ping vm" [Critical,Triaged] - Assigned to Ben Nemec (bnemec) | 12:11 |
| openstack | Launchpad bug 1655919 in tripleo "CI: Openvswitch agent fails because can't import tinyrpc.server module" [Critical,Triaged] | 12:11 |
| *** links has quit IRC | 12:13 | |
| *** derekh has quit IRC | 12:16 | |
| *** derekh has joined #tripleo | 12:16 | |
| EmilienM | hello | 12:17 |
| EmilienM | d0ugal: ack | 12:17 |
| EmilienM | jpich: yes because I missed to add a note (otherwise it fails to build) | 12:17 |
| EmilienM | omg CI status today | 12:18 |
| d0ugal | lol | 12:18 |
| jpich | EmilienM: d0ugal fixed it all I think, we're good :) Just waiting on CI results | 12:18 |
| EmilienM | should I go back to bed? | 12:18 |
| d0ugal | EmilienM: go to the pub | 12:18 |
| EmilienM | :) | 12:19 |
| sshnaidm | EmilienM, ovb job problem is handled by amoralej, it's openvswitch packaging problem as I undertand (let amoralej correct me) | 12:20 |
| amoralej | the problem is that new ryu package requires new library tinyrpc, i'm preparing a new package | 12:21 |
| EmilienM | sshnaidm: it's not only OVB it's also multinode I guess | 12:21 |
| EmilienM | amoralej: thanks :) | 12:21 |
| *** jkilpatr has joined #tripleo | 12:21 | |
| EmilienM | and FYI it's also Puppet CI: https://ci.centos.org/view/rdo/view/weirdo-pipelines/view/weirdo-promote-puppet/ | 12:21 |
| amoralej | EmilienM, that's where i found it | 12:23 |
| EmilienM | amoralej: nice :) | 12:24 |
| EmilienM | d0ugal: thx for the reno thing, it seems to work now http://docs-draft.openstack.org/40/419140/3/check/gate-tripleo-common-releasenotes/9ab16c6//releasenotes/build/html/unreleased.html | 12:24 |
| openstackgerrit | Ben Kero proposed openstack-infra/tripleo-ci: WIP: Add OOOQ multinode test support https://review.openstack.org/416110 | 12:24 |
| d0ugal | EmilienM: no worries, thanks for doing 99% of the work :) | 12:24 |
| marios | matbu: not sure if i understood the comment at https://review.openstack.org/#/c/414598/3/puppet/services/haproxy.yaml (I will update this anyway to not even use 'service' and use shell like chem suggests) | 12:24 |
| *** owalsh is now known as owalsh-lunch | 12:25 | |
| *** lucasagomes is now known as lucas-hungry | 12:26 | |
| *** ipsecguy has quit IRC | 12:29 | |
| openstackgerrit | John Trowbridge proposed openstack/tripleo-quickstart-extras: (undercloud|overcloud)-deploy: Add deployed_server script support https://review.openstack.org/417944 | 12:29 |
| *** zoli|lunch is now known as zoli | 12:31 | |
| panda | sshnaidm: if that bug is affecting undercloud installation, why the job did not stop ad undercloud install setp ? | 12:32 |
| sshnaidm | panda, why does it affect undercloud installation? | 12:35 |
| *** dprince has joined #tripleo | 12:35 | |
| panda | sshnaidm: what is setting up neutron-openvswitch-agent ? undercloud install ? | 12:36 |
| sshnaidm | panda, I think so | 12:37 |
| sshnaidm | panda, but service just restarts itself in the loop | 12:37 |
| sshnaidm | panda, seems like neutron doesn't care about it until it tries to create a port | 12:38 |
| panda | oh, that's why install doesn't fail, ok | 12:39 |
| *** tommylikehu has quit IRC | 12:39 | |
| *** tommylikehu has joined #tripleo | 12:40 | |
| *** pkovar has joined #tripleo | 12:41 | |
| openstackgerrit | Flavio Percoco proposed openstack/tripleo-quickstart-extras: WIP: Use openstack deploy to install the undercloud https://review.openstack.org/419040 | 12:42 |
| *** salmankhan has joined #tripleo | 12:42 | |
| EmilienM | flaper87: if you need tht/tools in the rpm; please patch https://github.com/rdo-packages/tripleo-heat-templates-distgit/blob/rpm-master/openstack-tripleo-heat-templates.spec#L42 | 12:44 |
| *** tzumainn has joined #tripleo | 12:47 | |
| sshnaidm | panda, http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-nonha/68f0aac/console.html#_2017-01-12_06_12_19_032197 | 12:48 |
| flaper87 | EmilienM: well, TBH, that's a quick and dirty fix but I'd rather not do that. I guess we can patch it for now just to move on | 12:48 |
| flaper87 | EmilienM: the real fix is to refactor tripleo_common | 12:48 |
| EmilienM | flaper87: okidoki | 12:48 |
| flaper87 | EmilienM: actually, I'll do it for now just to be able to move on with this and then I think we can remove it but I'll sync with dprince | 12:48 |
| sshnaidm | panda, and http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-nonha/68f0aac/console.html#_2017-01-12_06_12_19_206915 | 12:49 |
| sshnaidm | panda, consistent and current have the same hash | 12:49 |
| marios | chem: the only downside of using the bash/shell to do the check is the error sucks (like for check haproxy as the example you get a generic [WARNING]: Could not create retry file 'leplaybook.retry'. [Errno 2] No such file or directory: '' | 12:50 |
| marios | chem: but | 12:50 |
| chem | marios: this is not du to the shell module, it's a misconfiguration of ansible | 12:51 |
| marios | chem: agree we should live with that rather than try to start it and check if it was started | 12:51 |
| marios | chem: ah ok | 12:51 |
| panda | sshnaidm: we should check if they have the same hash also when current is broken or not. | 12:51 |
| chem | marios: basically, ansible has it leplaybook.retry set to '' | 12:51 |
| chem | marios: should be a directory's name there I guess | 12:51 |
| panda | sshnaidm: I'm putting up a patch, let's see how are the feedbacks. I think oooq has started using consistent too. | 12:52 |
| *** psahoo has quit IRC | 12:52 | |
| chem | marios: as an error msg you should have some gibberish half-json with the output of the command in error inside it | 12:53 |
| panda | sshnaidm: oh, uhm those jobs are supposed to be on the same hour as the failing periodic jobs. | 12:53 |
| chem | marios: not pretty, but more useful :) | 12:53 |
| marios | chem: ok, just checking (I had a similar comment when i tried exit1 at https://review.openstack.org/#/c/408631/10/puppet/services/tripleo-packages.yaml | 12:53 |
| marios | chem: because it could be quite important to say why we failed when we say 'failed on a preupgrade check' | 12:53 |
| panda | sshnaidm: maybe it's not only a question of hash, not sure what changes there. | 12:53 |
| sshnaidm | panda, I'm not sure, but the same hash for these repos seems suspicious to me.. | 12:54 |
| chem | marios: so it's either the ansible configuration is missing or something in the way we trigger ansible | 12:55 |
| *** fzdarsky has joined #tripleo | 12:56 | |
| marios | chem: great that did it thanks! | 12:56 |
| marios | chem: ++ | 12:56 |
| marios | chem: via https://github.com/ansible/ansible/issues/16365 i added /uncomment in ansible.cfg the retry_files_save_path | 12:57 |
| marios | chem: and now i get the proper message/error from ansible-playbook leplaybook.yaml | 12:57 |
| chem | marios: cool :) | 12:57 |
| *** jpena|lunch is now known as jpena | 12:58 | |
| *** dtantsur is now known as dtantsur|brb | 12:58 | |
| *** jprovazn has quit IRC | 12:59 | |
| EmilienM | thrash: hey, regarding the mistral testing thing. I think debug is already at maximum, isn't? see http://logs.openstack.org/20/418420/1/check/gate-puppet-openstack-integration-4-scenario003-tempest-centos-7/c3da45a/logs/mistral/engine.txt.gz we have INFO | 12:59 |
| EmilienM | thrash: please look at config and tell me what you would need more | 12:59 |
| *** jprovazn has joined #tripleo | 13:00 | |
| thrash | EmilienM: let me look. | 13:00 |
| thrash | EmilienM: but I think you're right. | 13:00 |
| *** fzdarsky has quit IRC | 13:01 | |
| *** morazi has joined #tripleo | 13:01 | |
| thrash | EmilienM: yep. That's as debug as it gets. :) | 13:02 |
| EmilienM | thrash: the best I can do tell you how to reproduce | 13:02 |
| EmilienM | thrash: on your laptop | 13:02 |
| thrash | EmilienM: that'd be awesome. :D | 13:02 |
| *** lucas-hungry is now known as lucasagomes | 13:02 | |
| EmilienM | thrash: let me pastbin something | 13:04 |
| *** morazi has quit IRC | 13:06 | |
| yolanda | hi, i updated the bug i created for dracut in TripleO with my findings: https://bugs.launchpad.net/tripleo/+bug/1655681 | 13:06 |
| openstack | Launchpad bug 1655681 in tripleo "Ramdisk for overcloud image shall include lvm, crypt drivers , or add the possibility to include extra ones" [Undecided,New] | 13:06 |
| yolanda | does anyone know how to better add modules inside the dracut image? | 13:07 |
| *** arxcruz has quit IRC | 13:07 | |
| *** rbrady has joined #tripleo | 13:07 | |
| *** rbrady has joined #tripleo | 13:07 | |
| EmilienM | thrash: http://paste.openstack.org/show/594692/ | 13:08 |
| thrash | EmilienM: thanks! | 13:08 |
| EmilienM | thrash: the github URL is missing https:// sorry | 13:08 |
| thrash | EmilienM: no worries. | 13:08 |
| openstackgerrit | Raoul Scarazzini proposed openstack/tripleo-quickstart-extras: Add external repo and img roles to bm undercloud https://review.openstack.org/418328 | 13:08 |
| *** jayg|g0n3 is now known as jayg | 13:10 | |
| *** ooolpbot has joined #tripleo | 13:11 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 13:11 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1654611 | 13:11 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1654615 | 13:11 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1655919 | 13:11 |
| *** ooolpbot has quit IRC | 13:11 | |
| openstack | Launchpad bug 1654611 in tripleo "Mitaka nonha ping test failing to upload image to glance" [Critical,Triaged] | 13:11 |
| *** gfidente has joined #tripleo | 13:11 | |
| openstack | Launchpad bug 1654615 in tripleo "Mitaka ha ping test failing to ping vm" [Critical,Triaged] - Assigned to Ben Nemec (bnemec) | 13:11 |
| openstack | Launchpad bug 1655919 in tripleo "CI: Openvswitch agent fails because can't import tinyrpc.server module" [Critical,Triaged] | 13:11 |
| *** trown|outtypewww is now known as trown | 13:11 | |
| *** salmankhan has quit IRC | 13:12 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci: Use "consistent" delorean repo instead of "current" https://review.openstack.org/419455 | 13:12 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates: Add swift service support for composable upgrades https://review.openstack.org/405245 | 13:13 |
| *** jprovazn has quit IRC | 13:13 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates: Add swift service support for composable upgrades https://review.openstack.org/405245 | 13:16 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates: Add swift service support for composable upgrades https://review.openstack.org/405245 | 13:17 |
| *** dtantsur|brb has quit IRC | 13:17 | |
| *** ansmith_ has joined #tripleo | 13:17 | |
| *** rasca has quit IRC | 13:18 | |
| *** yolanda has quit IRC | 13:18 | |
| *** dtantsur has joined #tripleo | 13:18 | |
| *** rasca has joined #tripleo | 13:18 | |
| *** ansmith has joined #tripleo | 13:18 | |
| *** yolanda has joined #tripleo | 13:18 | |
| *** morazi has joined #tripleo | 13:19 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates: Add swift service support for composable upgrades https://review.openstack.org/405245 | 13:19 |
| sshnaidm | panda, dlrn repos just named "current", they are not related to "tripleo-current" | 13:21 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates: Adds a pre-upgrade check that service is running (step0) https://review.openstack.org/414598 | 13:22 |
| *** ipsecguy has joined #tripleo | 13:23 | |
| openstackgerrit | Martin Mágr proposed openstack/tripleo-heat-templates: Improve SSL support for Sensu https://review.openstack.org/419460 | 13:23 |
| panda | sshnaidm: here https://github.com/openstack-infra/tripleo-ci/blob/master/scripts/tripleo.sh#L301 and https://github.com/openstack-infra/tripleo-ci/blob/master/scripts/tripleo.sh#L291 for example | 13:23 |
| marios | chem: matbu shardy i updated https://review.openstack.org/#/c/414598/ if you think its ok I can update for more services | 13:23 |
| panda | hm, known good | 13:23 |
| marios | chem: i used "=active" because 'active' also matches for 'inactive' :) | 13:24 |
| sshnaidm | panda, yeah, it's for stable | 13:24 |
| sshnaidm | panda, current repo is for newest packages, it shouldn't be "consistent" | 13:24 |
| *** kberger has joined #tripleo | 13:25 | |
| chem | marios: hehe, sorry. Maybe '\Wactive\W' would be even safer | 13:25 |
| *** links has joined #tripleo | 13:25 | |
| marios | chem: ah yeah that would work, i also considered '-v inactive' | 13:26 |
| chem | marios: with \W maybe egrep should be used | 13:27 |
| sshnaidm | panda, see for example "includepkgs" parameter there, it's newest tripleo-related packages | 13:27 |
| marios | chem: '=active' should be ok though? | 13:27 |
| sshnaidm | panda, why do you think it should be "consistent"? | 13:28 |
| chem | marios: =activeunknwon ? | 13:29 |
| chem | marios: with typo and all :) | 13:29 |
| *** pkovar1 has joined #tripleo | 13:29 | |
| *** pkovar has quit IRC | 13:29 | |
| marios | chem: is there such a state i mean activeunknown? | 13:29 |
| panda | sshnaidm: I'm trying to understand if there's any case where current may be used. If consistent contains the same thing as current, but with the warranty that it's free from ftbfs, then ti's better to remove it completely | 13:30 |
| sshnaidm | panda, it shouldn't be same as consistent | 13:30 |
| sshnaidm | panda, it's probably rdo issue with repo links | 13:31 |
| chem | marios: I doubt it, but '\W..\W' would guard against such postfix. | 13:31 |
| marios | chem: ack ok then i will update the review | 13:31 |
| marios | chem: sec | 13:31 |
| sshnaidm | panda, but worth to check if current really contains the last built packages, I think there was a problem with it before that they were outdated | 13:31 |
| *** amoralej is now known as amoralej|off | 13:32 | |
| panda | sshnaidm: also, stable branches are not using consistent at all, and maybe they should instead | 13:32 |
| *** amoralej|off is now known as amoralej|lunch | 13:32 | |
| openstackgerrit | Tim Rozet proposed openstack/tripleo-heat-templates: Removes deprecated OpenDaylight L2 only deployments https://review.openstack.org/419394 | 13:32 |
| sshnaidm | panda, does "consistent" exist for stable branches at all? | 13:33 |
| panda | sshnaidm: yes https://trunk.rdoproject.org/centos7-mitaka/consistent/ | 13:33 |
| *** pgadiya_ has quit IRC | 13:33 | |
| sshnaidm | panda, also consider that "current" in tripleo ci scripts may be different from what apevec means | 13:33 |
| panda | sshnaidm: I know, I'm especially suspicious of the mirror uris, but I'll see how the gates go | 13:34 |
| ansiwen | trown: I installed the undercloud with quickstart and release master-tripleo-ci, and now the "openstack overcloud node import instackenv.json" fails with: No valid host was found. Reason: No conductor service registered which supports driver pxe_ipmitool. (HTTP 400) | 13:34 |
| ansiwen | trown: any idea? | 13:34 |
| marios | chem: apparently it is "grep '\bactive\b'" (couldn't get \W to work maybe i was doing sthing else wrong) | 13:36 |
| matbu | marios: hey man sorry for my comment :) i'm more in favor of your previous change | 13:36 |
| *** sudipto_ has joined #tripleo | 13:37 | |
| *** sudipto has joined #tripleo | 13:37 | |
| trown | ansiwen: that sounds like ironic conductor is not started | 13:37 |
| marios | matbu: np thanks for clearing the confusion i will add a comment in a sec and lets discuss there | 13:37 |
| ansiwen | trown: and why is that? did I forget an essential step? | 13:37 |
| openstackgerrit | Steven Hardy proposed openstack-infra/tripleo-ci: Implement major upgrade for Newton to Ocata https://review.openstack.org/404831 | 13:37 |
| trown | ansiwen: that is the harder question :P | 13:37 |
| openstackgerrit | Steven Hardy proposed openstack-infra/tripleo-ci: Add basic pre/post upgrade sanity checks https://review.openstack.org/411846 | 13:37 |
| openstackgerrit | Steven Hardy proposed openstack-infra/tripleo-ci: Add glance to major upgrade CI https://review.openstack.org/412475 | 13:37 |
| ansiwen | trown: which part in the process is meant to start the conductor? | 13:38 |
| trown | ansiwen: it should be started during the puppet run of `openstack undercloud install` | 13:38 |
| chem | matbu: marios I'm kind against changing the state of the cluster for a check | 13:38 |
| ansiwen | trown: which again is run by quickstart, right? | 13:38 |
| chem | matbu: marios "check it but not change it" | 13:38 |
| trown | ansiwen: ya | 13:39 |
| matbu | chem: marios yep i agree, this is the cons of the service module | 13:39 |
| trown | ansiwen: what was your quickstart invocation, I can try to reproduce it | 13:39 |
| *** bkopilov has joined #tripleo | 13:39 | |
| marios | chem: matbu right... i lean towards just checking even though it is less 'ansibly' | 13:39 |
| ansiwen | trown: bash tripleo-quickstart/quickstart.sh --teardown all --release master-tripleo-ci -c tripleo-quickstart/config/general_config/minimal.yml 192.168.178.96 | 13:40 |
| marios | matbu: chem this is what i set out to do originally but found that the 'best' way without just using cmd/shell is to try and start it and check if it changed (but really we dont want to be doing that) | 13:40 |
| marios | matbu: chem 'best' i mean most 'ansibly' | 13:40 |
| matbu | marios: chem https://github.com/ansible/ansible-modules-core/pull/3316 | 13:40 |
| marios | matbu: right cool | 13:40 |
| marios | matbu: that would be good | 13:41 |
| matbu | marios: yep i'm looking if the PR exist in the main repo | 13:41 |
| marios | matbu: so in a sense the 'check_and_fail' for the ansible-pacemaker is kind of antipattern it seems :) | 13:41 |
| matbu | marios: lets start with the shell | 13:41 |
| matbu | marios: yes right :) | 13:41 |
| *** rlandy has joined #tripleo | 13:41 | |
| ansiwen | trown: and i removed the .quickstart dir before that | 13:43 |
| trown | ansiwen: can you try adding "--tags all"? I will try in parallel your way ... we have had a similar bug in the past where a restart of ironic was being done in later steps then the default tags so it hides that the default tags needed it | 13:43 |
| trown | ansiwen: cool, there is a --clean option that will do that for you too... just fyi | 13:43 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates: Adds a pre-upgrade check that service is running (step0) https://review.openstack.org/414598 | 13:44 |
| ansiwen | trown: good to know, thatnks | 13:44 |
| chem | matbu: is there a "noop" argument that can be passed to an ansible resource ? | 13:44 |
| chem | matbu: just wondering | 13:45 |
| ansiwen | trown: your undercloud-config ansible role was very helpful, btw. I have a nice config now for myself | 13:45 |
| matbu | chem: i'm looking | 13:45 |
| chem | matbu: like --check, but just attached to only resource | 13:45 |
| matbu | chem: i'll make a PR for the service module | 13:46 |
| matbu | chem: it's stupid to not have this behavior, it's really helpful | 13:46 |
| *** owalsh-lunch is now known as owalsh | 13:46 | |
| matbu | chem: maybe there is a design reason behind that | 13:46 |
| chem | matbu: maybe nobody use ansible as a monitoring system :) | 13:47 |
| matbu | chem: hehe yep, but since systemctl provide a "status" option | 13:47 |
| *** hjensas has quit IRC | 13:47 | |
| ansiwen | trown: I also had certificate failures when I wanted to use upload-puppet-modules, I had to set REQUESTS_CA_BUNDLE env accordingly | 13:47 |
| *** salmankhan has joined #tripleo | 13:48 | |
| chem | matbu: does ansible has the concept of "virtual" module, where service is systemctl on systemctl based system and something else elsewhere ? | 13:48 |
| chem | matbu: I kinda remember that back in the days it didn't have that "by design" | 13:49 |
| chem | matbu: like the yum/apt resource for packages | 13:49 |
| ansiwen | trown: I already restarted quickstart without tags all, befor you told me so, let's see how that goes | 13:49 |
| matbu | chem: hm i don't think | 13:49 |
| chem | matbu: oki | 13:50 |
| *** arxcruz has joined #tripleo | 13:50 | |
| *** lblanchard has joined #tripleo | 13:51 | |
| *** absubram has joined #tripleo | 13:51 | |
| *** salmankhan has quit IRC | 13:51 | |
| *** salmankhan has joined #tripleo | 13:52 | |
| *** Vijayendra has quit IRC | 13:54 | |
| ansiwen | shardy: can you give me a hint what is missing, when my endpoint uris are still empty, like "ec2api::keystone::auth::internal_url": "://:" , although I added ec2api to network/service_net_map.j2.yaml and network/endpoints/endpoint_map.yaml? What am I missing there? | 13:54 |
| ansiwen | shardy: context: http://logs.openstack.org/67/406167/8/check-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/bd774f2/ | 13:55 |
| *** pradk has joined #tripleo | 13:55 | |
| openstackgerrit | Florian Fuchs proposed openstack/tripleo-ui: [WIP] Add i18n to Notifications https://review.openstack.org/419469 | 13:56 |
| *** leanderthal has joined #tripleo | 13:58 | |
| flaper87 | EmilienM: https://review.rdoproject.org/r/#/c/4288/ | 13:58 |
| florianf | jtomasek: I've thought a bit about the best way to handle translations in ui notifications. ^ here's a WIP patch with one possible approach. If you have some time to look at it, let me know what you think. | 13:59 |
| florianf | jtomasek: It just highlights the approach, so it's really very small. | 13:59 |
| *** absubram has quit IRC | 14:00 | |
| *** absubram has joined #tripleo | 14:00 | |
| *** colonwq has joined #tripleo | 14:01 | |
| jtomasek | florianf: ok, I'll check it | 14:02 |
| jtomasek | thanks | 14:02 |
| florianf | jtomasek: thanks! | 14:02 |
| *** gkadam has joined #tripleo | 14:03 | |
| chem | EmilienM, shardy, marios, matbu so, what do you think about merging the bp/overcloud-upgrades-per-service topic so that we can ease the path for testing it ? | 14:03 |
| *** prateek_ has quit IRC | 14:03 | |
| akrivoka | EmilienM: how can I record the deep dive? seems I don't have the permissions | 14:03 |
| *** prateek has quit IRC | 14:03 | |
| *** rhallisey has joined #tripleo | 14:03 | |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-tripleo: Initial pacemaker remote profile support https://review.openstack.org/400967 | 14:04 |
| *** jcoufal has joined #tripleo | 14:05 | |
| *** fzdarsky has joined #tripleo | 14:05 | |
| *** shardy has quit IRC | 14:05 | |
| *** leanderthal has quit IRC | 14:06 | |
| *** shardy has joined #tripleo | 14:06 | |
| *** colonwq has quit IRC | 14:07 | |
| *** colonwq has joined #tripleo | 14:07 | |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common: Add support for calling validations in deploy and introspect workflows https://review.openstack.org/416284 | 14:08 |
| *** hjensas has joined #tripleo | 14:09 | |
| *** hjensas has quit IRC | 14:09 | |
| *** hjensas has joined #tripleo | 14:09 | |
| *** aufi has quit IRC | 14:10 | |
| shardy | chem: I think we can merge the upgrade series pretty soon, but we can't merge anything until we fix CI | 14:11 |
| *** ooolpbot has joined #tripleo | 14:11 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 14:11 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1654611 | 14:11 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1654615 | 14:11 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1655919 | 14:11 |
| *** ooolpbot has quit IRC | 14:11 | |
| openstack | Launchpad bug 1654611 in tripleo "Mitaka nonha ping test failing to upload image to glance" [Critical,Triaged] | 14:11 |
| openstack | Launchpad bug 1654615 in tripleo "Mitaka ha ping test failing to ping vm" [Critical,Triaged] - Assigned to Ben Nemec (bnemec) | 14:11 |
| openstack | Launchpad bug 1655919 in tripleo "CI: Openvswitch agent fails because can't import tinyrpc.server module" [Critical,Triaged] | 14:11 |
| shardy | does anyone know why we're getting no valid host found on almost all jobs? | 14:11 |
| shardy | I don't see a bug report yet | 14:11 |
| chem | shardy: arghh ... ack | 14:11 |
| bnemec | shardy: I assume it's https://bugs.launchpad.net/tripleo/+bug/1655919 | 14:11 |
| *** aufi has joined #tripleo | 14:12 | |
| bnemec | Although again I have to wonder how we're being broken by this. The promotion pipeline is supposed to protect us from this sort of thing. | 14:12 |
| shardy | chem: in the meantime I'm getting some ci patches ready which extend the CI test to deploy/upgrade more services, so hopefully we'll have some CI coverage by the time the patches for the services land | 14:12 |
| *** limao has joined #tripleo | 14:13 | |
| bnemec | Oh, this is ryu again. | 14:13 |
| weshay | is the master pingtest failing on all builds? | 14:14 |
| sshnaidm | EmilienM, trown about CI squad, is it still planned to have squads? Do we need to start CI-squad meetings, wdyt? | 14:14 |
| shardy | weshay: No, almost everything is failing during the deployment | 14:14 |
| sshnaidm | weshay, overcloud should fail: https://bugs.launchpad.net/tripleo/+bug/1655919 | 14:14 |
| openstack | Launchpad bug 1655919 in tripleo "CI: Openvswitch agent fails because can't import tinyrpc.server module" [Critical,Triaged] | 14:14 |
| *** akrivoka has quit IRC | 14:14 | |
| EmilienM | sshnaidm: yes it's still the plan to have squads. Meetings are up to the squad, if it's something that is needed | 14:15 |
| *** [1]cdearborn has joined #tripleo | 14:15 | |
| EmilienM | shardy: CI should be fixed by today | 14:15 |
| EmilienM | shardy: current blocker is 1655919 | 14:15 |
| *** limao_ has joined #tripleo | 14:15 | |
| trozet | EmilienM, shardy: Can you please review https://review.openstack.org/#/c/419394 ? | 14:15 |
| shardy | EmilienM: ack, thanks for confirming | 14:15 |
| shardy | trozet: will do | 14:16 |
| *** limao has quit IRC | 14:16 | |
| trozet | shardy: ty | 14:16 |
| *** saneax is now known as saneax-_-|AFK | 14:16 | |
| trown | sshnaidm: ya, CI-squad meeting is definitely needed imo... adarazs has fearlessly volunteered to be our squad liason | 14:16 |
| shardy | ansiwen: sorry I'm not sure why you're seeing that, your patch looks OK but there must be a mistake somewhere, I'll try to take a closer look later | 14:16 |
| *** apetrich has joined #tripleo | 14:16 | |
| trown | sshnaidm: maybe we could replace our thursday quickstart transition meeting with a tripleo-ci squad meeting | 14:16 |
| sshnaidm | trown, I dunno, transition meeting is pretty long to add there topics :) | 14:17 |
| adarazs | trown, sshnaidm: so should we denote this weeks tripleo-ci-transition meeting as the ci squad meeting? I think most people on the squad list will attend. | 14:17 |
| *** amoralej|lunch is now known as amoralej | 14:17 | |
| openstackgerrit | Ben Kero proposed openstack-infra/tripleo-ci: WIP: Add OOOQ multinode test support https://review.openstack.org/416110 | 14:18 |
| adarazs | rbrady: you're on that list too, so if you'd like to join, it would be probably useful. :) (meaning this: https://etherpad.openstack.org/p/tripleo-squads -> CI) | 14:18 |
| shardy | trozet: to clarify, is odl branched or in some way aligned with openstack release branches? | 14:18 |
| shardy | trozet: at first glance this doesn't look super great from a stable-maint perspective | 14:19 |
| *** Goneri has joined #tripleo | 14:19 | |
| trozet | shardy: yeah I agree | 14:19 |
| *** leanderthal has joined #tripleo | 14:19 | |
| shardy | but if the current configuration is actually broken, I suppose we've got no choice | 14:19 |
| *** jprovazn has joined #tripleo | 14:20 | |
| *** liverpooler has quit IRC | 14:20 | |
| *** egafford has joined #tripleo | 14:20 | |
| panda | trown: +1 | 14:20 |
| rbrady | adarazs: where's the meeting location? | 14:20 |
| adarazs | rlandy: I send you the invitation | 14:21 |
| panda | sshnaidm: it will become squad meeting after the transition, one of the main goals of the squad is the transition anyway at the moment :) | 14:21 |
| *** panda is now known as super-sleuth | 14:21 | |
| trown | lol | 14:21 |
| sshnaidm | super-sleuth, I thought ci-squad meeting is more for current/urgent issues that should be discussed | 14:23 |
| super-sleuth | sshnaidm: so it shouldn't be a regular meeting ? | 14:23 |
| *** limao_ has quit IRC | 14:24 | |
| EmilienM | all: RDO folks are building python-tinyrpc which is the reason why all CI is currently red (latest python-ryu build requires it) - no ETA but it should be fixed very soon | 14:24 |
| sshnaidm | super-sleuth, if nothing to talk about, maybe no.. but it's always something to discuss in CI :) | 14:24 |
| *** abehl has quit IRC | 14:26 | |
| rlandy | adarazs: sorry meeting - looking for invite | 14:26 |
| adarazs | rlandy: oops, I meant rbrady, hehe :) | 14:26 |
| adarazs | rbrady: so I sent you an invite with the info :) | 14:26 |
| rlandy | adarazs: ha - lucky I checked | 14:27 |
| *** jmelvin has joined #tripleo | 14:27 | |
| *** limao has joined #tripleo | 14:28 | |
| *** super-sleuth is now known as panda | 14:28 | |
| trown | sshnaidm: I am pretty sure there will always be enough CI related stuff to warrant a weekly meeting whether it is urgent stuff needing to be addressed or planning job coverage ... we would also come up with our report for the larger tripleo meeting in the squad meeting | 14:30 |
| openstackgerrit | David Moreau Simard proposed openstack/tripleo-quickstart-extras: Add ARA to the collected log directories https://review.openstack.org/418602 | 14:31 |
| openstackgerrit | David Moreau Simard proposed openstack/tripleo-quickstart: Add the ARA Ansible callback generate a static playbook reports https://review.openstack.org/418601 | 14:31 |
| dmsimard | adarazs: I submitted new patchsets, let me know how it looks ^ | 14:32 |
| openstackgerrit | yolanda.robla proposed openstack/tripleo-image-elements: Add element to rebuild dracut enabling modules https://review.openstack.org/419486 | 14:35 |
| *** gkadam_ has joined #tripleo | 14:35 | |
| *** gkadam has quit IRC | 14:35 | |
| *** alee has quit IRC | 14:39 | |
| *** prateek_ has joined #tripleo | 14:40 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: Use already patched repos cloned by zuul for build https://review.openstack.org/419491 | 14:41 |
| *** gkadam_ has quit IRC | 14:41 | |
| *** prateek has joined #tripleo | 14:41 | |
| openstackgerrit | yolanda.robla proposed openstack/tripleo-image-elements: Add element to rebuild dracut enabling modules https://review.openstack.org/419486 | 14:42 |
| *** liverpooler has joined #tripleo | 14:43 | |
| *** akrivoka has joined #tripleo | 14:44 | |
| *** ealcaniz has quit IRC | 14:45 | |
| openstackgerrit | John Trowbridge proposed openstack-infra/tripleo-ci: Use quickstart-extras checked out from zuul refs https://review.openstack.org/418123 | 14:47 |
| *** mcornea is now known as mcornea|afk | 14:48 | |
| beagles | holy cow that neutron/ryu thing turned into a mess | 14:48 |
| openstackgerrit | yolanda.robla proposed openstack/tripleo-image-elements: Add element to rebuild dracut enabling modules https://review.openstack.org/419486 | 14:48 |
| jtomasek | jkilpatr: you asked how tripleo-ui interacts with Mistral workflows? | 14:50 |
| jkilpatr | jtomasek, more how it takes user input -> triple settings, looking at the github it looks like it just calls api commands at least once a cloud is up. I assume it bakes config files for things like overcloud deploys? | 14:51 |
| jkilpatr | sorry I didn't hangout, right into another meeting. | 14:51 |
| *** flepied has quit IRC | 14:52 | |
| sshnaidm | trown, adarazs maybe it's worth to have agend with topics for ci-squad meetings? in some etherpad for example.. | 14:52 |
| adarazs | sshnaidm: okay, I'll create an etherpad. | 14:53 |
| jtomasek | jkilpatr: yep, right, but it is done on the backend in tripleo-common, GUI is just a client. business logic itself lives in tripleo-common, so all clients use the same logic | 14:53 |
| jkilpatr | jtomasek, does tripleo-common handle defaults or are those defined by the client? | 14:54 |
| jtomasek | jkilpatr: in general, certain actions and workflows update mistral environment, which stores all configuration changes done by clients and for deployment, this information is transformed into single environment file which is together with tripleo-heat-templates used to run the deployment | 14:54 |
| d0ugal | jkilpatr: most of the defaults come from tripleo-heat-templates. tripleo-common does a few things like generating passwords | 14:55 |
| jtomasek | jkilpatr: all defaults are defined in tripleo-heat-templates template set. we call this 'deployment plan'. A deployment plan is a single source of truth for overcloud configuration | 14:55 |
| jkilpatr | ok then, so the same place I would look doing a 'normal' CLI deploy. | 14:56 |
| jtomasek | yep | 14:56 |
| jkilpatr | cool, just wanted to make sure if my team was making defaults suggestions (worker numbers or whatever based on perf data) we didn't need to go to each UI and inform them too. | 14:57 |
| jtomasek | jkilpatr: no, you'd propose a change to tripleo-heat-templates and GUI then consumes it automatically | 14:57 |
| *** rook has quit IRC | 14:59 | |
| openstackgerrit | yolanda.robla proposed openstack/tripleo-image-elements: Add element to rebuild dracut enabling modules https://review.openstack.org/419486 | 14:59 |
| *** lucasagomes has quit IRC | 14:59 | |
| *** sai has quit IRC | 15:00 | |
| *** lucasagomes has joined #tripleo | 15:00 | |
| *** chlong has joined #tripleo | 15:02 | |
| ansiwen | trown: just to let you know, I redeployed the undercloud... and everything seems to be fine :-/ | 15:04 |
| trown | ansiwen: well that is good I suppose :)... we have had a bug in the past that sounds similar but it reproduced very consistently | 15:05 |
| *** rook has joined #tripleo | 15:05 | |
| *** sai has joined #tripleo | 15:05 | |
| ansiwen | trown: I pulled quickstart inbetween... so maybe something has been fixed? | 15:05 |
| *** rook is now known as Guest8298 | 15:05 | |
| *** limao has quit IRC | 15:06 | |
| openstackgerrit | Brad P. Crochet proposed openstack/python-tripleoclient: Add support for calling validations from introspection https://review.openstack.org/419506 | 15:07 |
| trown | ansiwen: hmm last merge was an update to README yesterday... not sure | 15:07 |
| EmilienM | amoralej: I assigned https://bugs.launchpad.net/tripleo/+bug/1655919 to you | 15:08 |
| openstack | Launchpad bug 1655919 in tripleo "CI: Openvswitch agent fails because can't import tinyrpc.server module" [Critical,Triaged] - Assigned to Alfredo Moralejo (amoralej) | 15:08 |
| *** absubram has quit IRC | 15:08 | |
| EmilienM | panda: can you assign https://bugs.launchpad.net/tripleo/+bug/1655938 to yourself? | 15:09 |
| openstack | Launchpad bug 1655938 in tripleo "CI: overcloud image creation fails on all periodic jobs" [Critical,Triaged] | 15:09 |
| openstackgerrit | Harry Rybacki proposed openstack/tripleo-quickstart-extras: Add default `ansible_user` to collect-logs https://review.openstack.org/416741 | 15:09 |
| panda | EmilienM: yep | 15:09 |
| *** jdennis1 has joined #tripleo | 15:10 | |
| *** flepied has joined #tripleo | 15:11 | |
| *** ooolpbot has joined #tripleo | 15:11 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 15:11 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1654611 | 15:11 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1654615 | 15:11 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1655919 | 15:11 |
| *** ooolpbot has quit IRC | 15:11 | |
| openstack | Launchpad bug 1654611 in tripleo "Mitaka nonha ping test failing to upload image to glance" [Critical,Triaged] | 15:11 |
| openstack | Launchpad bug 1654615 in tripleo "Mitaka ha ping test failing to ping vm" [Critical,Triaged] - Assigned to Ben Nemec (bnemec) | 15:11 |
| openstack | Launchpad bug 1655919 in tripleo "CI: Openvswitch agent fails because can't import tinyrpc.server module" [Critical,Triaged] - Assigned to Alfredo Moralejo (amoralej) | 15:11 |
| beagles | EmilienM: just in case : https://review.openstack.org/#/c/419508/ | 15:11 |
| *** jdennis has quit IRC | 15:12 | |
| EmilienM | beagles: well, we already updated ryu in RDO, so I guess it's too late | 15:12 |
| beagles | EmilienM: if we got that in and reverted the ryu package | 15:13 |
| EmilienM | except if we revert it | 15:13 |
| beagles | EmilienM: right | 15:13 |
| EmilienM | I think apevec wants to move forward (which I understand) and fix all the things | 15:13 |
| beagles | EmilienM: ack | 15:13 |
| *** aufi has quit IRC | 15:13 | |
| EmilienM | beagles: we need to update ryu anyway. We don't know which downside we can hit again | 15:14 |
| EmilienM | beagles: end of ocata is really close, let's clean things before | 15:14 |
| beagles | EmilienM: yup | 15:14 |
| *** numans has quit IRC | 15:14 | |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common: Add support for calling validations in deploy and introspect workflows https://review.openstack.org/416284 | 15:14 |
| *** salmankhan has quit IRC | 15:15 | |
| *** anshul has joined #tripleo | 15:17 | |
| *** yamahata has joined #tripleo | 15:20 | |
| *** numans has joined #tripleo | 15:20 | |
| chem | gfidente: hi, it's my ceph question of the day | 15:20 |
| gfidente | chem hey | 15:21 |
| chem | gfidente: on ceph version 10.2 (for osp9) what is the best way to restart the radosgw ? | 15:21 |
| gfidente | wait we're not supposed to be using 10 in osp9 | 15:21 |
| gfidente | but anyway you can just restard the systemd instance | 15:21 |
| chem | gfidente: ah ... that could be (customized installation) | 15:22 |
| chem | gfidente: systemctl restart ceph-radosgw don't work | 15:22 |
| chem | gfidente: missing unit | 15:22 |
| *** salmankhan has joined #tripleo | 15:22 | |
| gfidente | yeah it's a named instance | 15:22 |
| gfidente | ceph-radosgw@$instance-id | 15:22 |
| gfidente | like ceph-osd@1 | 15:22 |
| *** numans has quit IRC | 15:22 | |
| *** jaosorior has joined #tripleo | 15:22 | |
| chem | gfidente: oki | 15:22 |
| jaosorior | bnemec: hey dude, you around? | 15:22 |
| chem | gfidente: what version should we have on osp9 ? | 15:22 |
| *** numans has joined #tripleo | 15:22 | |
| gfidente | hammer is the one deployed by tripleo | 15:23 |
| bnemec | jaosorior: Yeah. | 15:23 |
| gfidente | 0.94 | 15:23 |
| jaosorior | bnemec: hey dude, so the TLS job took more than 10 hours (last time I saw, probably even more) to run | 15:23 |
| jaosorior | and in the end it didn't even run :/ | 15:23 |
| chem | gfidente: That was the ceph question of the days. Thanks :) | 15:23 |
| jaosorior | couldn't get an env | 15:23 |
| jaosorior | bnemec: do you have any ideas how could I test that faster | 15:23 |
| gfidente | chem I think systemctl -a should tell you what was the instance named | 15:23 |
| jaosorior | getting one run a day is really rough | 15:23 |
| jaosorior | and I'm getting really behind :/ | 15:24 |
| chem | gfidente: ack | 15:24 |
| bnemec | jaosorior: Yeah, the down side of experimental jobs is that they get last priority, and we've had a queue all day lately so they basically only run at night. | 15:24 |
| *** numans has quit IRC | 15:24 | |
| bnemec | Which is why I've been trying to find ways to stop queuing so badly during the day. | 15:25 |
| *** numans has joined #tripleo | 15:25 | |
| bnemec | jaosorior: One option to get faster results would be to submit a tripleo-ci patch that replaces one of the regular jobs with tls. | 15:25 |
| jaosorior | bnemec: could you help me out with that? I'm not sure how to do it | 15:25 |
| jaosorior | bnemec: also, I ran it quite early in the morning, and it couldn't get an env (again) http://logs.openstack.org/38/391738/82/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-fakeha-caserver/20c781f/console.html#_2017-01-12_13_53_19_577811 | 15:26 |
| bnemec | jaosorior: Hmm, that's strange. I wonder if there's an issue with the te-broker changes related to this. Let me check the logs. | 15:26 |
| jaosorior | bnemec: thanks man | 15:27 |
| *** milan has quit IRC | 15:27 | |
| amoralej | EmilienM: ack | 15:28 |
| EmilienM | we have our TripleO CI squad meeting: https://etherpad.openstack.org/p/tripleo-ci-squad-meeting - feel free to join | 15:28 |
| bnemec | jaosorior: Ah, think I see the problem: 2017-01-12 04:02:55,560 - testenv-worker-9445 - INFO - Received job : {"ssh_key": "\"ssh-rsa", "callback_name": "callback_5932a9ea4e4545ffbe1ae75fe90c07ad", "envsize": "3", "timeout": "10200", "create_undercloud": "true", "ucinstance": "d8293022-2526-408f-bafe-3d587912b246"} | 15:28 |
| bnemec | That ssh_key value looks bogus. | 15:28 |
| *** Guest8298 is now known as rook-LOLLLL | 15:28 | |
| jaosorior | bnemec: it wasn't quoted correctly so it seems | 15:28 |
| jaosorior | bnemec: here's ohw the call ends up like http://logs.openstack.org/38/391738/82/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-fakeha-caserver/20c781f/console.html#_2017-01-12_13_53_13_479327 | 15:29 |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart-extras: WIP: Creating docs for repo setup role https://review.openstack.org/419523 | 15:29 |
| jaosorior | bnemec: and line 309 here https://review.openstack.org/#/c/391738/82/toci_gate_test.sh is how it's added to the arguments | 15:30 |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart-extras: Use full path to run-os-net-config.sh in locally copied nic-configs https://review.openstack.org/410908 | 15:30 |
| jaosorior | bnemec: so for one it seems the way I tried to add the quotes is wrong, but for some reason it's not passing the rest of the ssh_key :/ | 15:32 |
| bnemec | jaosorior: I might stick the cat output in a variable so you can quote it and not have it get split at the space. | 15:33 |
| bnemec | e.g. SSH_KEY="$(cat ~/.ssh/id_rsa.pub)" | 15:33 |
| bnemec | I'm still not exactly sure what quoting would be needed when passing it to --ssh-key though. | 15:33 |
| bnemec | I would mess around with a test script locally and see what seems to do the right thing. | 15:34 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: TLS everywhere job introduction https://review.openstack.org/391738 | 15:34 |
| jaosorior | bnemec: what about that? | 15:34 |
| jaosorior | bnemec: but yeah, I would still use some help getting a job that replaces a regular one for the tls job | 15:35 |
| bnemec | jaosorior: Basically remove the existing section (say, ha) and replace it with the bits from the two new sections you added. Like http://paste.openstack.org/show/594720/ | 15:37 |
| bnemec | Then the "ha" job on the change should be running your new job instead. | 15:37 |
| *** egonzalez has joined #tripleo | 15:38 | |
| egonzalez | dprince: ping | 15:38 |
| openstackgerrit | Bob Fournier proposed openstack/diskimage-builder: Handle failure of carrier check in dhcp-all-interfaces.sh https://review.openstack.org/419527 | 15:38 |
| *** alee has joined #tripleo | 15:38 | |
| jaosorior | bnemec: what if I change the whole name of the job (lets say the ovb ha job) for the tls job? | 15:39 |
| jaosorior | before we get to that case | 15:39 |
| *** salmankhan has quit IRC | 15:40 | |
| bnemec | jaosorior: The job variables are passed in by infra. I'd be hesitant to mess with them. | 15:40 |
| *** salmankhan has joined #tripleo | 15:40 | |
| bnemec | It also won't change the name of the job in the results. | 15:40 |
| jaosorior | bnemec: it won't | 15:40 |
| jaosorior | bnemec: but I could get the result of the TLS job there | 15:41 |
| jaosorior | something like export TOCI_JOBTYPE=gate-tripleo-ci-centos-7-ovb-fakeha-caserver | 15:41 |
| bnemec | jaosorior: You can certainly try it. | 15:42 |
| jaosorior | sshnaidm: you had at some point skipped all the other tests based on the test name | 15:42 |
| jaosorior | sshnaidm: how was that again? | 15:42 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: DO NOT MERGE - testing TLS everywhere job https://review.openstack.org/419529 | 15:45 |
| jaosorior | bnemec: like this ^^ | 15:45 |
| *** apetrich has quit IRC | 15:45 | |
| openstackgerrit | Merged openstack/diskimage-builder: Run dhcp-interface@.service after network.target https://review.openstack.org/416664 | 15:47 |
| jaosorior | bnemec: how did your patch for the nodecount go? | 15:47 |
| jaosorior | bnemec: saw it failed but didn't check why | 15:47 |
| jaosorior | was it related? | 15:47 |
| bnemec | jaosorior: Everything is failing right now, so no useful results. | 15:48 |
| jaosorior | crap | 15:48 |
| larsks | I saw a recent CI failure in which the stack deployed correctly but the pingtest failed to spawn a nova instance due to a "No valid host" error. Should I just recheck or is there something going on? | 15:51 |
| larsks | Or maybe I should read my scrollback first. bnemec, "everything is failing" == ci is currently broken? | 15:52 |
| shardy | http://tripleo.org/cistatus.html | 15:52 |
| shardy | larsks: that's a good place to check, everything is red atm, there's a critical bug which isn't yet fixed | 15:53 |
| larsks | shardy, thanks for the link. | 15:53 |
| shardy | https://bugs.launchpad.net/tripleo/+bug/1655919 | 15:53 |
| openstack | Launchpad bug 1655919 in tripleo "CI: Openvswitch agent fails because can't import tinyrpc.server module" [Critical,Triaged] - Assigned to Alfredo Moralejo (amoralej) | 15:53 |
| shardy | larsks: ^^ | 15:53 |
| openstackgerrit | Brent Eagles proposed openstack/tripleo-common: Add support for deploying Octavia https://review.openstack.org/419536 | 15:54 |
| openstackgerrit | Brent Eagles proposed openstack/tripleo-common: Add support for deploying Octavia https://review.openstack.org/419536 | 15:54 |
| *** bkopilov has quit IRC | 15:56 | |
| *** tremble has quit IRC | 15:57 | |
| *** pcaruana has joined #tripleo | 15:58 | |
| *** salmankhan has quit IRC | 15:59 | |
| *** absubram has joined #tripleo | 15:59 | |
| *** salmankhan has joined #tripleo | 16:00 | |
| *** sanagikoki has joined #tripleo | 16:04 | |
| *** sudipto has quit IRC | 16:05 | |
| *** sudipto_ has quit IRC | 16:05 | |
| jaosorior | bnemec: so, seems that the trick to pass it to ha worked | 16:05 |
| jaosorior | but | 16:05 |
| jaosorior | I still have issues quoting that | 16:05 |
| *** sudipto_ has joined #tripleo | 16:05 | |
| *** sudipto has joined #tripleo | 16:05 | |
| dprince | egonzalez: hi, back in a few | 16:06 |
| egonzalez | dprince: just to know if there is a bp, spec or whatever about containerization in tripleO, willing to know the idea/design under this to help | 16:07 |
| *** sanagikoki has quit IRC | 16:09 | |
| *** sanagikoki has joined #tripleo | 16:10 | |
| *** rook-LOLLLL is now known as rook | 16:10 | |
| *** ooolpbot has joined #tripleo | 16:11 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 16:11 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1654611 | 16:11 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1654615 | 16:11 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1655919 | 16:11 |
| *** ooolpbot has quit IRC | 16:11 | |
| openstack | Launchpad bug 1654611 in tripleo "Mitaka nonha ping test failing to upload image to glance" [Critical,Triaged] | 16:11 |
| openstack | Launchpad bug 1654615 in tripleo "Mitaka ha ping test failing to ping vm" [Critical,Triaged] - Assigned to Ben Nemec (bnemec) | 16:11 |
| openstack | Launchpad bug 1655919 in tripleo "CI: Openvswitch agent fails because can't import tinyrpc.server module" [Critical,Triaged] - Assigned to Alfredo Moralejo (amoralej) | 16:11 |
| *** ebarrera has quit IRC | 16:12 | |
| *** rajinir has joined #tripleo | 16:14 | |
| *** fpan_ has joined #tripleo | 16:15 | |
| *** fpan has quit IRC | 16:16 | |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common: Add support for calling validations in deploy and introspect workflows https://review.openstack.org/416284 | 16:16 |
| *** pradk has quit IRC | 16:16 | |
| ccamacho | larsks the current CI issue affects not only CI.. from the no valid host message also have this in nova-conductor " Insufficient compute resources: Free disk 0.00 GB < requested 40 GB.\n']"" | 16:17 |
| larsks | ccamacho, thanks. I'm just glad it's not me (I had just pushed changes that fixed a legitimate ci failure...) | 16:17 |
| *** fpan_ has quit IRC | 16:19 | |
| ansiwen | my overcloud deploy fails with "No valid host was found." although the OvercloudXXXFlavor is set to the same name as is shown in the capabilities in the ironic nodes. the flavors with these names also exist.... what am I doing wrong? >_< | 16:19 |
| *** fpan has joined #tripleo | 16:20 | |
| chem | gfidente: he, the second ceph question of the day. I don't see a mecanism to upgrade ceph rados gw in osp10, only mon/osd, am i missing something ? | 16:21 |
| *** sai is now known as smalleni | 16:21 | |
| bnemec | jaosorior: Like I said, for the quoting I would set up two scripts locally - one that reads id_rsa.pub and passes it to the other one. The other script would just dump out the value it receives so you can see if it's what you want. | 16:22 |
| gfidente | chem yeah we didn't support rgw in osp9 | 16:22 |
| gfidente | or better in tripleo/newton | 16:22 |
| chem | gfidente: oki, this make sense then. | 16:23 |
| chem | gfidente: thanks | 16:23 |
| gfidente | I guess you could hack the upgrade scripts | 16:23 |
| *** ccamacho has quit IRC | 16:23 | |
| *** aufi has joined #tripleo | 16:24 | |
| *** prateek_ has quit IRC | 16:24 | |
| chem | gfidente: not there yet. Would it be just possible to include the proper newton template so that it just work ? | 16:25 |
| *** prateek has quit IRC | 16:25 | |
| openstackgerrit | Paul Belanger proposed openstack/tripleo-specs: Add debootstrap-minimal element for diskimage-builder https://review.openstack.org/419139 | 16:25 |
| gfidente | chem to include the ocata template? | 16:26 |
| gfidente | you'd need the updated puppet-tripleo too | 16:26 |
| colonwq | chem, it would take some tripleo puppet changes as the swift bits are not properly wrapped in a flag check. | 16:26 |
| gfidente | doesn't sound easy | 16:26 |
| *** ccamacho has joined #tripleo | 16:27 | |
| ccamacho | ansiwen check in the undercloud the nova-conductor logs | 16:27 |
| ccamacho | grep for Insufficient | 16:27 |
| chem | gfidente: nope the ceph-rgw available newton | 16:28 |
| *** panda is now known as panda|bbl | 16:28 | |
| gfidente | chem then I think I am not understanding the request | 16:29 |
| gfidente | you couldn't deploy rgw in newton at all | 16:29 |
| gfidente | from tripleo | 16:29 |
| chem | gfidente: ah ... so I must be looking at the wrong branch then | 16:29 |
| gfidente | chem in ocata you can | 16:29 |
| *** prateek has joined #tripleo | 16:30 | |
| *** yprokule has quit IRC | 16:30 | |
| *** prateek_ has joined #tripleo | 16:30 | |
| adarazs | EmilienM: thanks for joining the ci squad meeting, I think it was very useful for us. | 16:30 |
| dsavineau | gfidente: you can deploy radosgw in newton | 16:30 |
| dsavineau | https://github.com/openstack/tripleo-heat-templates/blob/stable/newton/environments/ceph-radosgw.yaml | 16:31 |
| EmilienM | adarazs: it was! | 16:32 |
| EmilienM | adarazs: can you maybe send notes on the ML? | 16:32 |
| yolanda | hi, are the | 16:33 |
| yolanda | there problems on the ci? | 16:33 |
| adarazs | EmilienM: all right. though I will filter it down and make a summary during the next TripleO meeting too. | 16:33 |
| *** bkopilov has joined #tripleo | 16:33 | |
| yolanda | i just got all the tests failed on a change | 16:33 |
| gfidente | dsavineau chem right ... newton had roles already | 16:33 |
| gfidente | mitaka didn't | 16:34 |
| dsavineau | exactly | 16:34 |
| gfidente | so taking on a side release naming issue | 16:34 |
| gfidente | I don't think it'd be trivial to port the newton rgw templates/puppet into mitaka | 16:34 |
| sshnaidm | jaosorior, https://review.openstack.org/#/c/416259/15/toci_gate_test.sh, first lines there | 16:34 |
| *** achadha has joined #tripleo | 16:35 | |
| *** achadha has quit IRC | 16:35 | |
| *** achadha has joined #tripleo | 16:35 | |
| chem | gfidente: so in newton we have that https://github.com/openstack/tripleo-heat-templates/blob/stable/newton/environments/ceph-radosgw.yaml | 16:36 |
| *** links has quit IRC | 16:37 | |
| chem | gfidente: hehe, np. It hapen to me all the time (had to check on the web) | 16:38 |
| *** rlandy is now known as rlandy|mtg | 16:39 | |
| *** fzdarsky has quit IRC | 16:39 | |
| gfidente | chem dsavineau yep thanks | 16:39 |
| dprince | egonzalez: we have an old spec for containers but it is evolving a bit in this new patch https://review.openstack.org/#/c/416421/ | 16:41 |
| dprince | egonzalez: our approach uses Kolla containers, configured w/ puppet and then orchestrated via Heat | 16:42 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: TLS everywhere job introduction https://review.openstack.org/391738 | 16:42 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: DO NOT MERGE - testing TLS everywhere job https://review.openstack.org/419529 | 16:42 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: [WIP] Respond to multiple messages from workflow executions https://review.openstack.org/406178 | 16:42 |
| openstackgerrit | Ben Kero proposed openstack-infra/tripleo-ci: WIP: Add OOOQ multinode test support https://review.openstack.org/416110 | 16:44 |
| *** achadha has quit IRC | 16:45 | |
| *** achadha has joined #tripleo | 16:46 | |
| *** blinky_ghost has joined #tripleo | 16:47 | |
| blinky_ghost | hi maybe this is the right channel to ask: I'm trying to deploy tripleo newton and I'm getting this error in ironic when trying to discover hw: instack.itcenter.local dnsmasq-dhcp[4098]: DHCPDISCOVER(tap2e59947b-28) a4:ba:db:40:29:33 no address available. Any Hint? thanks | 16:47 |
| dprince | dtantsur, lucasagomes: that question ^? | 16:48 |
| dtantsur | blinky_ghost, wild guess: something misconfigued NIC on undercloud? also, wdym by "discover hw"? introspection? | 16:49 |
| *** achadha_ has joined #tripleo | 16:49 | |
| * lucasagomes reads | 16:50 | |
| blinky_ghost | dtantsur: yes, introspection. Not misconfigured NIC because I deploy this before. However I deleted my overcloud and tried again to deploy and now I'm getting this | 16:50 |
| dtantsur | blinky_ghost, anything in openstack-ironic-inspector-dnsmasq logs? | 16:51 |
| *** achadha has quit IRC | 16:51 | |
| egonzalez | dprince: thanks for the info, sounds like funny time coming ;) | 16:52 |
| blinky_ghost | dtantsur: I only see this: Jan 12 16:16:43 instack.itcenter.local dnsmasq-dhcp[1515]: DHCPREQUEST(br-ctlplane) 192.168.21.63 a4:ba:db:40:29:33 | 16:52 |
| blinky_ghost | Jan 12 16:16:43 instack.itcenter.local dnsmasq-dhcp[1515]: DHCPACK(br-ctlplane) 192.168.21.63 a4:ba:db:40:29:33 localhost | 16:52 |
| *** chlong has quit IRC | 16:52 | |
| *** achadha_ has quit IRC | 16:54 | |
| dtantsur | so, these are the right logs (previous ones were from neutron) | 16:54 |
| blinky_ghost | dtantsur: I think if I delete neutron db and redeploy undercloud it will work, but I have to test it to make sure. | 16:55 |
| blinky_ghost | *I mean reinstall the undercloud | 16:55 |
| dtantsur | neutron is not really related, I don't think it's going to help | 16:56 |
| dtantsur | blinky_ghost, what is the actual error you see? | 16:56 |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder: Bump fedora-minimal release to 25 https://review.openstack.org/419559 | 16:56 |
| blinky_ghost | dtantsur: the server doesn't get ip from dhcp after some time boots to HD | 16:56 |
| ansiwen | ccamacho: thanks! Free disk 10.00 GB < requested 40 GB. So the default flavors are wrong? | 16:57 |
| *** jmelvin has quit IRC | 16:57 | |
| dtantsur | blinky_ghost, hmm. this DHCPACK you pasted above - was it from this introspection run or some previous? | 16:57 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: CI test - never merge https://review.openstack.org/419122 | 16:58 |
| blinky_ghost | dtantsur: it was from now | 16:58 |
| ccamacho | ansiwen I bet you have free space.. the thing is that somehow not sure why free_disk_mb is reported wrong.. | 16:58 |
| * ccamacho crying | 16:58 | |
| blinky_ghost | dtantsur: but I'll test again, just a sec | 16:59 |
| * ansiwen padding ccamachos shoulder | 16:59 | |
| ccamacho | there was something related to the nova placement API merged recently... that might be hitting us.. | 16:59 |
| ansiwen | ccamacho: ouch... | 17:00 |
| ccamacho | ansiwen but if you run nova hypervisor-stats you will have the space displayed correctly.. | 17:01 |
| * ccamacho at least in my case.. | 17:01 | |
| ansiwen | ccamacho: free_disk_gb | 100 | 17:01 |
| ccamacho | ;) | 17:02 |
| ansiwen | but does that help? | 17:02 |
| ansiwen | nova list | 17:02 |
| ansiwen | oops :-) | 17:02 |
| ansiwen | ccamacho: so, anything I can do? | 17:04 |
| owalsh | ansiwen: Free disk 10.00 GB < requested 40 GB is normal I think, VMs are 50GB & the flavors require 40, leaving 10 | 17:05 |
| *** flepied has quit IRC | 17:06 | |
| owalsh | ansiwen: nova should try the next host, unless the filters have removed them | 17:06 |
| ansiwen | owalsh: but I have two ironic nodes, and wanna deploy only two nodes (control and compute) so it should be sufficient, no? | 17:06 |
| ccamacho | hey owalsh this look like an error to me "Insufficient compute resources: Free disk 0.00 GB < requested 40 GB. " | 17:06 |
| ansiwen | owalsh: so you think there is another problem? | 17:06 |
| *** fpan_ has joined #tripleo | 17:06 | |
| *** sudswas__ has joined #tripleo | 17:07 | |
| sanagikoki | Could someone review this ? It needs another +2. https://review.openstack.org/#/c/404897/ | 17:07 |
| *** sudipto_ has quit IRC | 17:07 | |
| *** fpan has quit IRC | 17:07 | |
| *** chem has quit IRC | 17:07 | |
| *** bana_k has joined #tripleo | 17:07 | |
| ansiwen | ccamacho, owalsh: I also cound the PortBindingFailed error in the compute logs | 17:07 |
| *** chem has joined #tripleo | 17:07 | |
| ansiwen | s/cound/found/ | 17:07 |
| *** sudipto has quit IRC | 17:08 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: TLS everywhere job introduction https://review.openstack.org/391738 | 17:08 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: DO NOT MERGE - testing TLS everywhere job https://review.openstack.org/419529 | 17:08 |
| ansiwen | ccamacho: I don't understand what hypervisor-stats actually means in case of ironic backend... totals of all ironic nodes together? | 17:08 |
| *** sudipto has joined #tripleo | 17:08 | |
| *** thrash is now known as thrash|biab | 17:09 | |
| blinky_ghost | dtantsur: you're right, that's an old log, I don't see anything passing by openstack-ironic-inspector-dnsmasq: sudo journalctl -l -u openstack-ironic-inspector-dnsmasq -f. I have debug=True | 17:09 |
| ansiwen | ccamacho: seems so, because my ironic nodes have 50gb each | 17:09 |
| dtantsur | blinky_ghost, do you see DHCP requests coming to the undercloud in via e.g. tcpdump | 17:11 |
| *** hewbrocca is now known as hewbrocca_afk | 17:11 | |
| *** ooolpbot has joined #tripleo | 17:11 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 17:11 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1654611 | 17:11 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1654615 | 17:11 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1655919 | 17:11 |
| *** ooolpbot has quit IRC | 17:11 | |
| openstack | Launchpad bug 1654611 in tripleo "Mitaka nonha ping test failing to upload image to glance" [Critical,Triaged] | 17:11 |
| openstack | Launchpad bug 1654615 in tripleo "Mitaka ha ping test failing to ping vm" [Critical,Triaged] - Assigned to Ben Nemec (bnemec) | 17:11 |
| openstack | Launchpad bug 1655919 in tripleo "CI: Openvswitch agent fails because can't import tinyrpc.server module" [Critical,In progress] - Assigned to Alfredo Moralejo (amoralej) | 17:11 |
| blinky_ghost | dtantsur: a second | 17:12 |
| openstackgerrit | John Trowbridge proposed openstack-infra/tripleo-ci: Use quickstart-extras checked out from zuul refs https://review.openstack.org/418123 | 17:14 |
| *** achadha has joined #tripleo | 17:14 | |
| jaosorior | bnemec: yes! I think I got it | 17:16 |
| *** trown is now known as trown|lunch | 17:16 | |
| jaosorior | 2017-01-12 17:15:56.034779 | +(/opt/stack/new/tripleo-ci/toci_gate_test.sh:337): ./testenv-client -b 192.168.103.254:4730 -t 10200 --envsize 3 --ucinstance b10480da-1267-453d-a3fb-41796fa462dc --create-undercloud --ssh-key 'ssh-rsa | 17:17 |
| jaosorior | AAAAB3NzaC1yc2EAAAADAQABAAABAQDBiFFAopMUeeYG7IR/zdKbhSS/MWa8VSMrRyNqOsDlOBwFg3yhdL7vpZxj3FZWkd8lKNFY+faQ7hu2GtzJ6cRS2PF1nG2E6tMyFdtmF0667K+8g/sU8MooRldQ70bdQC2WEik6Hx3iKtzZRHSzoWEgoG2d/2tL5o3rlqhUuUBrxVPH6jDaweuY6uU7h/R1iQj71lzQkAc4Igh7HRqnvq/NvhbKekKZr5hDRAIioc9yXmZYIlyNJSlsxy4kyZ/cF6wXuJyLnpSh80qKLJyFKdDw10Ju/VT4zKr4iZ+6Ej3okWw4nd7AQf5cmyJ9KcQVwfdpft+MF+Nf/2fYn1yhUg/N | 17:17 |
| jaosorior | jenkins@tripleo-centos-7-tripleo-test-cloud-rh1-6629952' -- ./toci_instack_ovb.sh | 17:17 |
| *** salmankhan has quit IRC | 17:17 | |
| *** zoli is now known as zoli|gone | 17:18 | |
| ansiwen | bnemec: blind guess re: https://bugs.launchpad.net/tripleo/+bug/1654611 | 17:18 |
| openstack | Launchpad bug 1654611 in tripleo "Mitaka nonha ping test failing to upload image to glance" [Critical,Triaged] | 17:18 |
| *** sdake_ has joined #tripleo | 17:18 | |
| *** cylopez has quit IRC | 17:18 | |
| *** achadha has quit IRC | 17:18 | |
| bnemec | jaosorior: Looks good in the te-broker logs too. | 17:18 |
| sdake_ | dprince note I updated the work items in the blueprint your working on here: https://blueprints.launchpad.net/kolla/+spec/apache-packages-for-apis | 17:18 |
| sdake_ | dprince if you could keep them up to date (TODO/INPROGRESS/DONE) that is how we know when a blueprint is finished for the release - or if we need to move it to the next release | 17:19 |
| *** jlinkes has quit IRC | 17:19 | |
| sdake_ | DONE may be completed - not certain | 17:19 |
| bnemec | ansiwen: I'm taking all guesses, blind or otherwise, at this point. :-) | 17:19 |
| ansiwen | bnemec: I had certificate issues before with the requests package before in different scenarios. and I saw requests involved. check if setting the env REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt | 17:20 |
| ansiwen | help | 17:20 |
| ansiwen | s | 17:20 |
| dprince | sdake_: I think once the 3 patches land we are good for now | 17:20 |
| sdake_ | dprince ok - if you want to revise the work items, it w ould be helpful - not sure which ones you need or don't need | 17:20 |
| dprince | sdake_: I updated your updates to reflect what needs to land. My aodh and gnocchi patches weren't required after all. (that was me being overly ambitious there) | 17:21 |
| bnemec | ansiwen: Where would you set that? | 17:21 |
| bnemec | On the ceph node? | 17:21 |
| dprince | sdake_: over time, more API's will likely switch to Apache/mod_wsgi I think | 17:21 |
| dprince | sdake_: so there may be more of this coming... but for now I think we are initially okay | 17:21 |
| blinky_ghost | dtantsur: Yes I see tcpdump requests: 17:20:57.356729 B a4:ba:db:3e:02:d4 ethertype IPv4 (0x0800), length 592: 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from a4:ba:db:3e:02:d4, length 548 | 17:21 |
| *** achadha has joined #tripleo | 17:21 | |
| ansiwen | bnemec: I couldn't do upload-puppet-modules without that... | 17:21 |
| ansiwen | bnemec: on the side that initiates the https connection | 17:22 |
| dprince | sdake_: so if we leave the BP open for another release it might be fine. Or we could create a new one at that time | 17:22 |
| dtantsur | blinky_ghost, interesting, and still nothing in the dnsmasq logs? | 17:22 |
| ansiwen | bnemec: as I said, it's a blind guess, because I saw the requests package involved... | 17:22 |
| blinky_ghost | dtantsur: nothing | 17:22 |
| *** flepied has joined #tripleo | 17:22 | |
| dtantsur | blinky_ghost, hmm. how do you start introspection? | 17:23 |
| bnemec | ansiwen: Oh, yeah I don't think it's actually related to ssl. It looks like ceph isn't running on the cephstorage node, so glance has nothing to talk to when it tries to save the image. | 17:23 |
| sdake_ | dprince typically we close out blueprints and open new ones up | 17:23 |
| blinky_ghost | dtantsur: openstack overcloud node introspect c3c84353-4afc-4400-80fb-8752bf9f65f1 --provide | 17:23 |
| dprince | sdake_: that is good with me | 17:23 |
| sdake_ | dprince just so we know what work got done when :) | 17:23 |
| *** dsneddon_afk is now known as dsneddon | 17:23 | |
| bnemec | This is happening with ssl turned off too. | 17:23 |
| ansiwen | bnemec: and hit this problem twice in the last two days... | 17:23 |
| sdake_ | dprince also reno is helpful - although for this it probably doesn't make a ton of sense | 17:23 |
| ansiwen | bnemec: ok.... then... | 17:23 |
| dtantsur | blinky_ghost, do you see node's MAC blocked in 'iptables -L' during introspection? | 17:24 |
| dprince | sdake_: ack, this is just a piddly package sort of thing | 17:24 |
| *** rlandy|mtg is now known as rlandy | 17:24 | |
| blinky_ghost | dtantsur: Chain ironic-inspector (1 references) | 17:24 |
| blinky_ghost | target prot opt source destination | 17:24 |
| blinky_ghost | DROP all -- anywhere anywhere MAC A4:BA:DB:3E:02:D4 | 17:24 |
| blinky_ghost | ACCEPT all -- anywhere anywhere | 17:24 |
| dtantsur | hmm, it's blacklisted for some reason.. is introspection still "running" for it? | 17:26 |
| *** chlong has joined #tripleo | 17:26 | |
| blinky_ghost | dtantsur: yes | 17:26 |
| dtantsur | blinky_ghost, huh.. but restarting introspection doesn't help, does it? | 17:27 |
| blinky_ghost | dtantsur: I aborted it and it's still there: openstack baremetal introspection abort c3c84353-4afc-4400-80fb-8752bf9f65f1 | 17:27 |
| blinky_ghost | dtantsur: nopes | 17:27 |
| dtantsur | well, after aborting introspection this record should be in place | 17:27 |
| dtantsur | you should not see this DROP when introspection is still active | 17:28 |
| *** ccamacho has quit IRC | 17:28 | |
| blinky_ghost | dtantsur: you mean it should be there, and when i start the introspection it should disappear? | 17:30 |
| dtantsur | yep | 17:30 |
| blinky_ghost | the DROP | 17:30 |
| jaosorior | bnemec: hey dude, I'm having some issue with the TLS job; http://logs.openstack.org/29/419529/4/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/f5b82a0/console.html#_2017-01-12_17_27_51_025492 I need to run some stuff with sudo, and even if I use the -t option it's failing :/ | 17:31 |
| *** bana_k has quit IRC | 17:31 | |
| *** egonzalez has left #tripleo | 17:31 | |
| blinky_ghost | dtantsur: I rebooted undercloud and the record on iptables wasn't there. Then I started introspection I see the drop again | 17:33 |
| *** aufi has quit IRC | 17:34 | |
| dtantsur | blinky_ghost, check 'ironic port-list', is it possible you have the same MAC for another node? | 17:34 |
| *** pradk has joined #tripleo | 17:34 | |
| blinky_ghost | dtantsur: nopes [stack@instack ~]$ ironic port-list | 17:35 |
| blinky_ghost | +--------------------------------------+-------------------+ | 17:35 |
| blinky_ghost | | UUID | Address | | 17:35 |
| blinky_ghost | +--------------------------------------+-------------------+ | 17:35 |
| blinky_ghost | | ed1d57e0-660b-4133-b734-cd8a589b7b23 | a4:ba:db:3e:02:d4 | | 17:35 |
| blinky_ghost | | 55ebc7f2-cee2-4c8b-b39a-d1f5e1ad1187 | a4:ba:db:40:29:33 | | 17:35 |
| blinky_ghost | +--------------------------------------+-------------------+ | 17:35 |
| *** bana_k has joined #tripleo | 17:36 | |
| jaosorior | sshnaidm: if you have time, can you help me out figuring out some issue with the TLS job? http://logs.openstack.org/29/419529/4/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/f5b82a0/console.html#_2017-01-12_17_27_51_025492 I need to run some stuff with sudo, and even if I use the -t option it's failing :/ | 17:36 |
| openstackgerrit | Jason E. Rist proposed openstack/tripleo-ui: Add i18n strings to the login page https://review.openstack.org/419045 | 17:37 |
| sshnaidm | jaosorior, I usually use -ttt | 17:37 |
| sshnaidm | jaosorior, the more t the better :) | 17:37 |
| jaosorior | haha alright | 17:38 |
| jaosorior | gonna try it out | 17:38 |
| dtantsur | blinky_ghost, anything suspicious in openstack-ironic-inspector logs? | 17:38 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: TLS everywhere job introduction https://review.openstack.org/391738 | 17:38 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: DO NOT MERGE - testing TLS everywhere job https://review.openstack.org/419529 | 17:38 |
| dtantsur | blinky_ghost, also please make sure a4:ba:db:3e:02:d4 actually belongs to the correct node | 17:38 |
| dtantsur | blinky_ghost, if e.g. the MACs are swapped, it will cause this effect | 17:38 |
| blinky_ghost | dtantsur: ok a second | 17:39 |
| *** salmankhan has joined #tripleo | 17:43 | |
| *** yamahata has quit IRC | 17:43 | |
| *** derekh has quit IRC | 17:43 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Test with scheduler hints https://review.openstack.org/378040 | 17:44 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Add feature testing matrix to readme https://review.openstack.org/399269 | 17:44 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Add support for testing predictable placement https://review.openstack.org/378014 | 17:44 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Test hostname map https://review.openstack.org/378017 | 17:44 |
| *** anshul has quit IRC | 17:44 | |
| *** achadha has quit IRC | 17:44 | |
| ansiwen | shardy: if you have the chance to look again at https://review.openstack.org/#/c/398634/ | 17:45 |
| EmilienM | bnemec: it's odd that we have a matrix here https://review.openstack.org/#/c/399269 and in THT for scenarios... wdyt? | 17:45 |
| ansiwen | shardy: or any hint how to proceed debugging that would be highly appreciated :-) | 17:45 |
| bnemec | EmilienM: I see it as a little different because the feature enablement is done in tripleo-ci, whereas the service stuff is being managed in tht. | 17:46 |
| bnemec | Basically, to add a service to a multinode job you submit a patch to tht. | 17:47 |
| EmilienM | bnemec: true | 17:47 |
| bnemec | To add a feature to a job, you submit a patch to tripleo-ci. | 17:47 |
| bnemec | (in general) | 17:47 |
| EmilienM | yeah, I guess that's fine now | 17:47 |
| EmilienM | as long as our devs know where to find what | 17:47 |
| shardy | ansiwen: I'll try to check it out, but basically you have to start with heat resource-show overcloud EndpointMap | 17:48 |
| shardy | ansiwen: see if the data is wrong there, and if so then work backwards via the references in the endpoint map template | 17:48 |
| *** ebarrera has joined #tripleo | 17:48 | |
| *** karthiks has quit IRC | 17:48 | |
| shardy | somewhere an empty string is being passed in, or perhaps there's a missing defintion | 17:48 |
| *** jpich has quit IRC | 17:49 | |
| shardy | ansiwen: I thought it was not defining the network in ServiceNetMapDefaults initially, but I checked your patch and that looks OK | 17:49 |
| *** lucasagomes is now known as lucas-afk | 17:50 | |
| ansiwen | shardy: might it be a problem that the Network and the ...Admin ...Public ...Internal both are set to Ec2Api? which the other service do a bit differently | 17:50 |
| ansiwen | shardy: I have to run "heat resource-show.." after the failed overcloud deployment? | 17:51 |
| *** owalsh has quit IRC | 17:51 | |
| shardy | ansiwen: well that just means they should get the network assigned to Ec2ApiNetwork | 17:52 |
| shardy | https://review.openstack.org/#/c/398634/16/network/service_net_map.j2.yaml | 17:52 |
| shardy | which you do there | 17:52 |
| ansiwen | shardy: right, how I did, right? | 17:52 |
| ansiwen | yes | 17:52 |
| openstackgerrit | Paul Belanger proposed openstack/tripleo-specs: Add debootstrap-minimal element for diskimage-builder https://review.openstack.org/419139 | 17:52 |
| shardy | ansiwen: yes, that's how you'll get more details about almost any failed deployment - introspect the failed stack and compare the outputs from heat with the templates you gave it | 17:53 |
| openstackgerrit | Jason E. Rist proposed openstack/tripleo-ui: Add i18n strings to the login page https://review.openstack.org/419045 | 17:53 |
| openstackgerrit | Jason E. Rist proposed openstack/tripleo-ui: Add i18n strings to the login page https://review.openstack.org/419045 | 17:54 |
| ansiwen | shardy: the problem is: I can't deploy it locally because of another problem... :-/ | 17:54 |
| shardy | ansiwen: Ok, well I guess you have to fix the other problem first ;) | 17:54 |
| *** colonwq has quit IRC | 17:55 | |
| shardy | honestly, this stuff is far, far easier to debug when you've got a local deployment to poke at | 17:55 |
| *** thrash|biab is now known as thrash | 17:55 | |
| ansiwen | shardy: No valid host was found. it seems like there is a PortBindingFailed coming from neutron | 17:55 |
| shardy | ansiwen: did you update today? Is it https://bugs.launchpad.net/tripleo/+bug/1655919 ? | 17:56 |
| openstack | Launchpad bug 1655919 in tripleo "CI: Openvswitch agent fails because can't import tinyrpc.server module" [Critical,In progress] - Assigned to Alfredo Moralejo (amoralej) | 17:56 |
| shardy | all of CI is failing with a similar error but looks like it should be fixed soon | 17:57 |
| shardy | there's packages referenced you can pull locally and try | 17:57 |
| ansiwen | shardy: oh, yes, I deployed freshly today... | 17:57 |
| *** masco has joined #tripleo | 17:57 | |
| ansiwen | shardy: I just learned how to update the puppet modules... no idea yeat how to include RDO packages... :-/ | 17:58 |
| shardy | ansiwen: you just download and install them? | 17:59 |
| shardy | like on any system | 17:59 |
| ansiwen | shardy: oh, directly in the undercloud? they don't have to be in the images? | 17:59 |
| ansiwen | I see | 17:59 |
| shardy | ansiwen: well it sounds like it's the undercloud that's broken | 17:59 |
| ansiwen | right | 18:00 |
| jaosorior | sshnaidm: crap, even adding -ttt I stumbled upon the same issue | 18:00 |
| shardy | ansiwen: if you need to put the same packages into overcloud-full.qcow2 you can use virt-customize to do it, but I'd test the fixes on the undercloud first | 18:01 |
| shardy | you can prove the undercloud neutron is fixed and you can at least start the deploy | 18:01 |
| jaosorior | sshnaidm: http://logs.openstack.org/29/419529/5/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/0d175ad/console.html#_2017-01-12_17_59_42_250478 | 18:03 |
| jaosorior | sshnaidm: and this is where I'm doing the call https://review.openstack.org/#/c/391738/87/scripts/deploy.sh | 18:03 |
| *** rasca has quit IRC | 18:03 | |
| blinky_ghost | dtantsur: you're right it was swapped macs, I had some issues with my ILOs and swapped them somehow. Thank you for your time :) | 18:04 |
| sshnaidm | jaosorior, hmm..lemme see | 18:04 |
| *** prateek_ has quit IRC | 18:05 | |
| *** colonwq has joined #tripleo | 18:05 | |
| openstackgerrit | Harry Rybacki proposed openstack/tripleo-quickstart: Remove unused roles from OOOQ https://review.openstack.org/418998 | 18:05 |
| dtantsur | blinky_ghost, yvw :) | 18:08 |
| sshnaidm | jaosorior, is user in sudoers file? | 18:08 |
| jaosorior | sshnaidm: that's a great question, it pretty much is just newly deployed node | 18:09 |
| chem | given a valid json file is there a easy way to give it to os-collect-config for debug purpose ? | 18:09 |
| jaosorior | it should AFAIK | 18:09 |
| jaosorior | bnemec: ^^ | 18:09 |
| sshnaidm | jaosorior, can you check it? in worst case you can connect and run it as root | 18:10 |
| jaosorior | sshnaidm: not sure how | 18:10 |
| jaosorior | sshnaidm: when I tried that (deploying the extra node with ovb) in rh1, it was in the sudoers file | 18:10 |
| *** ooolpbot has joined #tripleo | 18:11 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 18:11 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1654611 | 18:11 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1654615 | 18:11 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1655919 | 18:11 |
| *** ooolpbot has quit IRC | 18:11 | |
| openstack | Launchpad bug 1654611 in tripleo "Mitaka nonha ping test failing to upload image to glance" [Critical,Triaged] | 18:11 |
| sshnaidm | jaosorior, ssh root@... "cat /etc/sudoers /etc/sudoers.d/* /etc/sudo.conf" | 18:11 |
| openstack | Launchpad bug 1654615 in tripleo "Mitaka ha ping test failing to ping vm" [Critical,Triaged] - Assigned to Ben Nemec (bnemec) | 18:11 |
| openstack | Launchpad bug 1655919 in tripleo "CI: Openvswitch agent fails because can't import tinyrpc.server module" [Critical,In progress] - Assigned to Alfredo Moralejo (amoralej) | 18:11 |
| jaosorior | sshnaidm: I tried to connect to root initially, but the default key is not added there | 18:11 |
| jaosorior | sshnaidm: so I get a failure trying to connect | 18:11 |
| sshnaidm | jaosorior, ok | 18:11 |
| ansiwen | shardy: obviously I'm to retarded, to find these packages on https://trunk.rdoproject.org >_< | 18:12 |
| EmilienM | ok, tripleo CI should be green again | 18:12 |
| EmilienM | please do not recheck everything in the same time | 18:13 |
| EmilienM | and recheck only patches that are urgent to have | 18:13 |
| ansiwen | good call | 18:13 |
| *** weshay is now known as weshay_bbiab | 18:13 | |
| ansiwen | amoralej: I'm too stupid, where can I find python-ryu-4.9-2 rpm? | 18:13 |
| shardy | http://cbs.centos.org/koji/packageinfo?packageID=2330 | 18:14 |
| shardy | ansiwen: you can grab it from koji if it's not hit the repos yet | 18:14 |
| ansiwen | shardy: awsome, thanks | 18:14 |
| amoralej | ansiwen, can get it from http://buildlogs.centos.org/centos/7/cloud/x86_64/openstack-ocata/?C=M;O=D also | 18:15 |
| jaosorior | sshnaidm: quick and dirty patch would be to do sudo su inside the script I'm calling, though I'm not too keen on that | 18:15 |
| amoralej | but in fact it's in the repo already so, it should be automatically fetched when installing | 18:15 |
| sshnaidm | jaosorior, I don't think it's a cause, but maybe try to run it as 'sudo /bin/bash /absolute_path/freeipa-setup.sh' | 18:16 |
| ansiwen | amoralej: if I install the overcloud again? yes, I wanted to avoid that | 18:16 |
| sshnaidm | jaosorior, maybe, but if sudo doesn't work.. not sure sudo su will work | 18:16 |
| ansiwen | amoralej: sorry, undercloud | 18:16 |
| jaosorior | sshnaidm: not sure what changes, when I ran toci_gate_test.sh from rh1 I didn't have this issue :/ | 18:17 |
| amoralej | ok ansiwen, then you can just run yum update python2-ryu | 18:17 |
| amoralej | but you'll need new packages in overcloud images too | 18:17 |
| ansiwen | amoralej: thanks! | 18:17 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: TLS everywhere job introduction https://review.openstack.org/391738 | 18:18 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: DO NOT MERGE - testing TLS everywhere job https://review.openstack.org/419529 | 18:18 |
| sshnaidm | jaosorior, it's weird, because -ttt should have done the trick.. maybe try this: http://paste.openstack.org/show/594750/ | 18:19 |
| *** chlong has quit IRC | 18:19 | |
| jaosorior | sshnaidm: something like this http://paste.openstack.org/show/594752/ ? | 18:21 |
| sshnaidm | jaosorior, let's give a try.. I hope if it's in sudoers, it has passwordless sudo | 18:21 |
| *** jpena is now known as jpena|off | 18:22 | |
| *** yamahata has joined #tripleo | 18:22 | |
| jaosorior | sshnaidm: or you meant the printing of the sudoers file? | 18:23 |
| sshnaidm | jaosorior, I doubt if you can read it without sudo | 18:23 |
| *** ebarrera has quit IRC | 18:24 | |
| *** akrivoka has quit IRC | 18:24 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: TLS everywhere job introduction https://review.openstack.org/391738 | 18:25 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: DO NOT MERGE - testing TLS everywhere job https://review.openstack.org/419529 | 18:25 |
| jaosorior | sshnaidm: trying it out ^^ | 18:25 |
| *** salmankhan has quit IRC | 18:29 | |
| *** rbowen has quit IRC | 18:31 | |
| *** chlong has joined #tripleo | 18:32 | |
| *** rbowen has joined #tripleo | 18:32 | |
| *** dougbtv has quit IRC | 18:32 | |
| *** colonwq has quit IRC | 18:32 | |
| *** openstackgerrit has quit IRC | 18:33 | |
| *** florianf has quit IRC | 18:33 | |
| *** pcaruana has quit IRC | 18:34 | |
| *** pkovar1 has quit IRC | 18:36 | |
| *** dougbtv has joined #tripleo | 18:37 | |
| *** openstackgerrit has joined #tripleo | 18:38 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-docs: Link to ironic docs on "no valid host found" instead of duplicating them https://review.openstack.org/419596 | 18:38 |
| *** colonwq has joined #tripleo | 18:40 | |
| *** apetrich has joined #tripleo | 18:40 | |
| *** ipsecguy has quit IRC | 18:41 | |
| *** jkilpatr has quit IRC | 18:43 | |
| EmilienM | amoralej: tripleo bug closed - I confirm it works fine in CI now | 18:43 |
| EmilienM | sshnaidm: we can run manual promotion thing now | 18:43 |
| shardy | thanks amoralej! :) | 18:43 |
| EmilienM | yeah thanks amoralej !! nice work | 18:43 |
| sshnaidm | EmilienM, ok | 18:43 |
| amoralej | glad to help! | 18:44 |
| EmilienM | (like usual ;-)) | 18:44 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: TEST: DONT RECHECK or REVIEW: periodic jobs https://review.openstack.org/359215 | 18:44 |
| *** colonwq has quit IRC | 18:44 | |
| openstackgerrit | Harry Rybacki proposed openstack/tripleo-quickstart-extras: DO NOT MERGE - Testing out change to gate-quickstart https://review.openstack.org/419605 | 18:45 |
| openstackgerrit | Harry Rybacki proposed openstack/tripleo-quickstart-extras: Remove unused dependency from roles https://review.openstack.org/419007 | 18:45 |
| *** trown|lunch is now known as trown | 18:45 | |
| bnemec | Do we need a promotion? Wasn't this a problem in a repo we don't pin? | 18:45 |
| slagle | pradk: howdy, i think we need to move aodh-db-sync to the -common package | 18:46 |
| jaosorior | sshnaidm: 2017-01-12 18:45:40.599309 | Password: su: Authentication failure :/ | 18:46 |
| slagle | pradk: currently it is in -api, we run aodh-db-sync in step 3, but don't include the aodh api.pp until step 4, which installs openstack-aodh-api | 18:47 |
| sshnaidm | jaosorior, uh, that's a problem, need to be passwordless sudoer. | 18:47 |
| sshnaidm | jaosorior, should be a way to log in as root or use boot script | 18:47 |
| jaosorior | bnemec: ^^ | 18:48 |
| pradk | slagle, hmm could we just move the db sync to step 4 before api? | 18:48 |
| jaosorior | bnemec: having issues running commands on the extra node that require privileges | 18:48 |
| EmilienM | bnemec: we don't *need* a promotion per-se but it's blocking things we're working on in tripleo | 18:48 |
| sshnaidm | jaosorior, maybe using user-metadata with nova | 18:48 |
| EmilienM | bnemec: ex: placement api (a bug was fixed in nova last night) | 18:48 |
| *** dougbtv_ has joined #tripleo | 18:49 | |
| jaosorior | sshnaidm: ?? | 18:49 |
| bnemec | jaosorior: Hmm, you're using the infra template for the instance? | 18:49 |
| *** jkilpatr has joined #tripleo | 18:49 | |
| jaosorior | bnemec: it's the extra node that I get from ovb | 18:50 |
| *** dougbtv has quit IRC | 18:50 | |
| bnemec | jaosorior: Yeah, but what image is it booted from? | 18:50 |
| slagle | pradk: possibly, but then it's out of sync with every other service | 18:51 |
| bnemec | Oh, we aren't specifying an image. | 18:51 |
| slagle | pradk: i was just going to patch the spec file | 18:51 |
| sshnaidm | bnemec, isn't it blank centos image? | 18:51 |
| jaosorior | bnemec: good question | 18:51 |
| jaosorior | bnemec: right, I didn't modify that | 18:51 |
| *** tesseract has quit IRC | 18:51 | |
| jaosorior | bnemec: should I? | 18:51 |
| bnemec | Heh, this happens to work because we have an image named CentOS-7-x86_64-GenericCloud in that cloud, and that's the OVB default too. | 18:52 |
| *** rhallisey has quit IRC | 18:52 | |
| bnemec | Strange though. I thought the base cloud images had password-less sudo. | 18:52 |
| pradk | slagle, yea spec file change is easy enough .. want me to do it? | 18:52 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Add heat service support for composable upgrades https://review.openstack.org/405244 | 18:52 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Add swift service support for composable upgrades https://review.openstack.org/405245 | 18:52 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Add neutron service support for composable upgrades https://review.openstack.org/405242 | 18:52 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Add cinder service support for composable upgrades https://review.openstack.org/405243 | 18:52 |
| jaosorior | bnemec: well, it worked when I tried it in RH1. Now it seems to be failing on CI | 18:52 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Add nova service support for composable upgrades https://review.openstack.org/405241 | 18:52 |
| jaosorior | for some reason | 18:52 |
| bnemec | jaosorior: You're logging in as the centos user, right? | 18:53 |
| slagle | pradk: nah, i've got it in progress | 18:53 |
| pradk | cool | 18:53 |
| *** dtantsur is now known as dtantsur|afk | 18:53 | |
| *** rhallisey has joined #tripleo | 18:54 | |
| *** colonwq has joined #tripleo | 18:54 | |
| jaosorior | bnemec: yes | 18:55 |
| jaosorior | bnemec: this is what I'm doing https://review.openstack.org/#/c/391738/89/scripts/deploy.sh | 18:55 |
| bnemec | jaosorior: I'm betting it isn't sudo that's failing. Line 52 is: su - | 18:57 |
| jaosorior | bnemec: ths was failing as well https://review.openstack.org/#/c/391738/87/scripts/deploy.sh | 18:58 |
| jaosorior | bnemec: I was trying to do sudo bash | 18:58 |
| jaosorior | bnemec: and it was failing with: Pseudo-terminal will not be allocated because stdin is not a terminal. | 18:58 |
| jaosorior | bnemec: even though I added -ttt | 18:59 |
| *** blinky_ghost has quit IRC | 19:00 | |
| *** jmelvin has joined #tripleo | 19:00 | |
| bnemec | jaosorior: I wonder if the sudo within a sudo is a problem. | 19:00 |
| jaosorior | bnemec: which sudo within a sudo? | 19:00 |
| bnemec | line 22 of https://review.openstack.org/#/c/391738/87/scripts/deploy.sh would get run within the sudo at line 51. | 19:01 |
| jaosorior | bnemec: that's not what's running on the extra node | 19:01 |
| bnemec | I'm kind of guessing here since I thought -ttt would fix the terminal error. | 19:01 |
| *** colonwq has quit IRC | 19:01 | |
| bnemec | But maybe the second sudo is not seeing the terminal for some reason? | 19:01 |
| jaosorior | bnemec: that's deploy.sh, where we trigger the command, and subsequently the undercloud installation and such | 19:02 |
| *** anshul has joined #tripleo | 19:02 | |
| jaosorior | bnemec: this is the script that's actually ran https://github.com/openstack/tripleo-heat-templates/blob/master/ci/scripts/freeipa_setup.sh | 19:02 |
| *** tosky has quit IRC | 19:03 | |
| bnemec | jaosorior: Crap, you're right. I was thinking that the shell script was being created there, but it's just the .env file. | 19:03 |
| jaosorior | bnemec: sorry if it's a bit confusing | 19:04 |
| *** sudipto has quit IRC | 19:05 | |
| *** sudswas__ has quit IRC | 19:05 | |
| *** sdake_ is now known as sdake | 19:06 | |
| jaosorior | bnemec: any ideas? | 19:08 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Add heat service support for composable upgrades https://review.openstack.org/405244 | 19:08 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Add swift service support for composable upgrades https://review.openstack.org/405245 | 19:08 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Add neutron service support for composable upgrades https://review.openstack.org/405242 | 19:08 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Add cinder service support for composable upgrades https://review.openstack.org/405243 | 19:08 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Add nova service support for composable upgrades https://review.openstack.org/405241 | 19:08 |
| jaosorior | should I switch to using another image? | 19:08 |
| *** ooolpbot has joined #tripleo | 19:11 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 19:11 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1654611 | 19:11 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1654615 | 19:11 |
| *** ooolpbot has quit IRC | 19:11 | |
| openstack | Launchpad bug 1654611 in tripleo "Mitaka nonha ping test failing to upload image to glance" [Critical,Triaged] | 19:11 |
| openstack | Launchpad bug 1654615 in tripleo "Mitaka ha ping test failing to ping vm" [Critical,Triaged] - Assigned to Ben Nemec (bnemec) | 19:11 |
| openstackgerrit | Steven Hardy proposed openstack-infra/tripleo-ci: Add nova and neutron to major upgrade CI https://review.openstack.org/419614 | 19:12 |
| bnemec | jaosorior: Not sure. I did come up with a situation where I could sudo on an ssh connection even though I got a "Pseudo-terminal will not be allocated because stdin is not a terminal.", so maybe that wasn't actually the problem? | 19:13 |
| bnemec | I don't think a different image would make a difference though. | 19:13 |
| jaosorior | bnemec: so, trying this manually on rh1 this was actually working :/ | 19:13 |
| jaosorior | with the sudo bash freeipa-setup.sh | 19:14 |
| *** colonwq has joined #tripleo | 19:16 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: TLS everywhere job introduction https://review.openstack.org/391738 | 19:16 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: DO NOT MERGE - testing TLS everywhere job https://review.openstack.org/419529 | 19:16 |
| *** amoralej is now known as amoralej|off | 19:17 | |
| *** shardy has quit IRC | 19:18 | |
| bnemec | jaosorior: Yeah, but you were running it from a terminal in that case. | 19:19 |
| EmilienM | sshnaidm: promotion seems to fail no? gate-tripleo-ci-centos-7-ovb-ha failed | 19:19 |
| EmilienM | sshnaidm: in 359215 | 19:19 |
| jaosorior | bnemec: right | 19:19 |
| sshnaidm | EmilienM, no, it's previous run | 19:20 |
| jaosorior | bnemec: well, I was calling toci_gate_test.sh to eventually get there | 19:20 |
| jaosorior | bnemec: but anyway, I guess your point still stands | 19:20 |
| sshnaidm | EmilienM, we run now 31 patch | 19:20 |
| bnemec | Yeah, somewhere in the chain there was a real terminal. In a job there isn't. | 19:20 |
| bnemec | A bunch of jobs failed on creating ovb stacks. | 19:20 |
| bnemec | It looks like we've got way too many stacks on rh1 again. | 19:20 |
| bnemec | [heat-admin@overcloud-controller-0 ~]$ heat stack-list | grep baremetal | wc -l | 19:20 |
| bnemec | 76 | 19:20 |
| EmilienM | sshnaidm: yeah but 31 is failing, see in zuul. no? | 19:20 |
| jaosorior | bnemec: what about your patch to exit the job if it's not avialable on zuul anymore, would that help? that might be part of the issue | 19:21 |
| bnemec | jaosorior: I'll be darned, that even passed ci: https://review.openstack.org/#/c/418615/ | 19:22 |
| bnemec | At least the jobs that are working. | 19:22 |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart: Add composable deployment config files and mixed release https://review.openstack.org/410831 | 19:22 |
| bnemec | EmilienM: We should merge https://review.openstack.org/#/c/418615/ to hopefully help with the quota issues on rh1 | 19:23 |
| openstackgerrit | Harry Rybacki proposed openstack/tripleo-quickstart-extras: ensure extras-common depends on tq common https://review.openstack.org/417195 | 19:23 |
| openstackgerrit | Harry Rybacki proposed openstack/tripleo-quickstart-extras: Ensure that each role in extras has a dependency on extras-common https://review.openstack.org/413800 | 19:23 |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart-extras: Add composable upgrade workflow for tripleo-quickstart https://review.openstack.org/416480 | 19:24 |
| sshnaidm | EmilienM, bnemec yes, it failed to get free ovb stack: http://logs.openstack.org/15/359215/31/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/a7dda53/console.html#_2017-01-12_19_03_13_637470 | 19:24 |
| bnemec | We've had 74 instances fail on quota issues in the past hour+. | 19:25 |
| *** masco has quit IRC | 19:25 | |
| EmilienM | bnemec: approved | 19:25 |
| bnemec | EmilienM: Thanks | 19:25 |
| *** weshay_bbiab is now known as weshay | 19:25 | |
| sshnaidm | bnemec, is it memory quota? | 19:25 |
| bnemec | sshnaidm: Yes | 19:25 |
| *** dougbtv_ is now known as dougbtv | 19:25 | |
| *** rajinir has quit IRC | 19:26 | |
| openstackgerrit | Chris Jones proposed openstack/python-tripleoclient: Add baremetal_deploy.ConfigureFencing https://review.openstack.org/417898 | 19:28 |
| *** rhallisey has quit IRC | 19:30 | |
| *** rhallisey has joined #tripleo | 19:30 | |
| *** mcornea|afk has quit IRC | 19:37 | |
| *** fultonj has quit IRC | 19:37 | |
| *** jmelvin has quit IRC | 19:37 | |
| *** migi has quit IRC | 19:39 | |
| *** chlong has quit IRC | 19:41 | |
| *** lucas-afk has quit IRC | 19:41 | |
| openstackgerrit | Harry Rybacki proposed openstack/tripleo-quickstart-extras: Remove unused dependency from roles https://review.openstack.org/419007 | 19:42 |
| *** jkilpatr has quit IRC | 19:43 | |
| EmilienM | jaosorior: dude, is it a blocker your comment on https://review.openstack.org/#/c/406309/ ? | 19:43 |
| EmilienM | jaosorior: or can we use nova api network? | 19:43 |
| *** rook has quit IRC | 19:43 | |
| *** paramite has quit IRC | 19:43 | |
| *** smalleni has quit IRC | 19:44 | |
| *** rook has joined #tripleo | 19:44 | |
| *** lucasagomes has joined #tripleo | 19:44 | |
| openstackgerrit | Harry Rybacki proposed openstack/tripleo-quickstart-extras: Remove unused dependency from roles https://review.openstack.org/419007 | 19:44 |
| *** rook is now known as Guest87706 | 19:44 | |
| *** rajinir has joined #tripleo | 19:45 | |
| *** Guest87706 is now known as rook-lollll | 19:45 | |
| openstackgerrit | Harry Rybacki proposed openstack/tripleo-quickstart: Remove unused roles from OOOQ https://review.openstack.org/418998 | 19:45 |
| *** sai has joined #tripleo | 19:45 | |
| Ng | so I know that at least quickstart is deploying with vbmc now, and overcloud nodes are registered in Ironic with pxe_ipmitool instead of pxe_ssh - is that officially the future and nobody should be caring about pxe_ssh anymore? or is that just a quickstart feature? | 19:47 |
| *** bana_k has quit IRC | 19:50 | |
| jaosorior | EmilienM: not a blocker | 19:50 |
| jaosorior | EmilienM: thought I had +2ed already | 19:51 |
| EmilienM | jaosorior: ok | 19:54 |
| *** jkilpatr has joined #tripleo | 19:55 | |
| *** jkilpatr has joined #tripleo | 19:55 | |
| *** chlong has joined #tripleo | 19:56 | |
| openstackgerrit | Tim Rozet proposed openstack/tripleo-heat-templates: Parameterizes Nova API default floating IP pool https://review.openstack.org/419631 | 19:58 |
| *** artom has quit IRC | 20:01 | |
| *** artom has joined #tripleo | 20:01 | |
| jaosorior | bnemec: alright, I'm queuing another attempt just using sudo | 20:04 |
| jaosorior | to see if there's some other cause to the failure | 20:04 |
| jaosorior | bnemec: is there a need to kill stacks or instances on rh1? | 20:05 |
| *** ipsecguy has joined #tripleo | 20:06 | |
| openstackgerrit | John Trowbridge proposed openstack-infra/tripleo-ci: WIP: Add OOOQ multinode test support https://review.openstack.org/416110 | 20:06 |
| openstackgerrit | John Trowbridge proposed openstack-infra/tripleo-ci: Add OOOQ multinode test support https://review.openstack.org/416110 | 20:08 |
| *** ooolpbot has joined #tripleo | 20:11 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 20:11 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1654611 | 20:11 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1654615 | 20:11 |
| *** ooolpbot has quit IRC | 20:11 | |
| openstack | Launchpad bug 1654611 in tripleo "Mitaka nonha ping test failing to upload image to glance" [Critical,Triaged] | 20:11 |
| -openstackstatus- NOTICE: Gerrit will be offline between now and 20:30 for scheduled maintenance: http://lists.openstack.org/pipermail/openstack-dev/2017-January/109910.html | 20:11 | |
| openstack | Launchpad bug 1654615 in tripleo "Mitaka ha ping test failing to ping vm" [Critical,Triaged] - Assigned to Ben Nemec (bnemec) | 20:11 |
| *** ChanServ changes topic to "Gerrit will be offline between now and 20:30 for scheduled maintenance: http://lists.openstack.org/pipermail/openstack-dev/2017-January/109910.html" | 20:11 | |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras: Ensure extras-common depends on common https://review.openstack.org/417008 | 20:11 |
| leifmadsen | Question about firewall management of the overcloud post-deploy. What is the correct way to manage new firewall rules for persistence on reboot on an overcloud node? | 20:11 |
| *** pblaho has quit IRC | 20:15 | |
| EmilienM | leifmadsen: something with https://github.com/openstack/puppet-tripleo/blob/master/manifests/firewall.pp#L66-L74 | 20:16 |
| leifmadsen | EmilienM: but that's done during deploy right? | 20:17 |
| leifmadsen | or is that run somehow on boot? | 20:17 |
| EmilienM | brb in a mtg | 20:17 |
| leifmadsen | this is a post deploy reboot | 20:17 |
| leifmadsen | np | 20:17 |
| jaosorior | bnemec: also, do you think using su -c would work? | 20:22 |
| *** jmelvin has joined #tripleo | 20:23 | |
| beagles | leifmadsen: unless I'm mistaken, puppet firewall stuff sets the rules up to be persistent | 20:30 |
| leifmadsen | beagles: right ok so this is my main question -- where are the rules stored for persistence? | 20:30 |
| leifmadsen | so I've setup my environment, I've logged into the overcloud node, I've done "iptables -I ... " and now I reboot | 20:30 |
| leifmadsen | that rule goes away right? | 20:30 |
| leifmadsen | (this is what I'm being led to believe from someone who asked, I'm waiting on a cloud to spin up to confirm) | 20:31 |
| beagles | leifmadsen: certainly all of the firewall rules defined in the templates are stored on the nodes /etc/sysconfig/iptables file | 20:31 |
| *** anshul has quit IRC | 20:31 | |
| leifmadsen | beagles: that's what I read as well, but they say when they do 'service iptables save' on reboot the rules are gone... | 20:31 |
| beagles | leifmadsen: yeah.. I think so. The iptables.conf is not configured to store the contents of the iptables on reboot/shutdown | 20:31 |
| *** bana_k has joined #tripleo | 20:32 | |
| leifmadsen | sorry, I should have been more clear that a 'service iptables save' is also run | 20:32 |
| leifmadsen | so from their viewpoint, it seems like that isn't the source of the iptables rules, so I was wondering if some sort of puppet magictry was actually run on reboots? | 20:32 |
| leifmadsen | and loaded from hiera? | 20:32 |
| *** jmelvin has quit IRC | 20:33 | |
| *** jrist has quit IRC | 20:33 | |
| *** fultonj has joined #tripleo | 20:34 | |
| *** jmelvin has joined #tripleo | 20:34 | |
| -openstackstatus- NOTICE: Updated: Gerrit will be offline until 20:45 for scheduled maintenance (running longer than anticipated): http://lists.openstack.org/pipermail/openstack-dev/2017-January/109910.html | 20:36 | |
| *** ChanServ changes topic to "Updated: Gerrit will be offline until 20:45 for scheduled maintenance (running longer than anticipated): http://lists.openstack.org/pipermail/openstack-dev/2017-January/109910.html" | 20:36 | |
| EmilienM | leifmadsen: https://github.com/puppetlabs/puppetlabs-firewall#create-firewall-rules | 20:40 |
| EmilienM | "Rules are persisted automatically between reboots, although there are known issues with ip6tables on older Debian/Ubuntu distributions. There are also known issues with ebtables." | 20:41 |
| leifmadsen | EmilienM: thanks for that -- the issue I'm having is that how do I make changes post-boot, and subsequently keep them after boot | 20:41 |
| leifmadsen | is the intention to redeploy the stack from the undercloud? | 20:41 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Respond to multiple messages from workflow executions https://review.openstack.org/406178 | 20:42 |
| leifmadsen | scenario: I deploy my cloud with the default rules, I want to add a new rule on the overcloud node, and then keep it persistent after reboot | 20:42 |
| leifmadsen | so the changes are post-deploy | 20:42 |
| EmilienM | let me show you after my mtg | 20:42 |
| leifmadsen | kk, cuz I'm not really sure how this puppet module is called, and where I should expect the persistent data to exist | 20:43 |
| leifmadsen | btw if I'm not here, it's because I had to leave a bit early to get my wife a birthday card and pick up my kids | 20:43 |
| *** jmelvin has quit IRC | 20:43 | |
| *** chlong has quit IRC | 20:44 | |
| *** anshul has joined #tripleo | 20:44 | |
| *** Goneri has quit IRC | 20:46 | |
| *** jrist has joined #tripleo | 20:46 | |
| *** jrist has joined #tripleo | 20:46 | |
| openstackgerrit | John Trowbridge proposed openstack/tripleo-quickstart-extras: [gate-quickstart] Workaround for ansible bug 5504 https://review.openstack.org/419639 | 20:48 |
| openstack | bug 1681 in Launchpad itself "duplicate for #5504 Viewing a translation page fails in unix2newlines" [High,Fix released] https://launchpad.net/bugs/1681 - Assigned to Carlos Perelló MarÃn (carlos) | 20:48 |
| *** dhill_ has quit IRC | 20:49 | |
| *** dhill__ has joined #tripleo | 20:49 | |
| *** dhill__ has quit IRC | 20:50 | |
| *** dhill_ has joined #tripleo | 20:50 | |
| *** dciabrin has quit IRC | 20:50 | |
| bnemec | jaosorior: Unfortunately I can't kill heat stacks because there's no good way to tell which ones map to a dead job at this point. | 20:50 |
| bnemec | I did bump the quota in the hopes that being overcommitted would be better than failing to create heat stacks. | 20:51 |
| bnemec | It looks like things are back under control again though. | 20:51 |
| jaosorior | yay | 20:51 |
| bnemec | I have no idea about su -c. I'm not sure why it's failing, so I'm not sure how to fix it. | 20:51 |
| jaosorior | well, lets see | 20:52 |
| jaosorior | right now, I'm trying it out with sudo | 20:52 |
| jaosorior | and lets see if the logs say anything | 20:52 |
| *** d0ugal has quit IRC | 20:52 | |
| bnemec | [heat-admin@overcloud-controller-0 ~]$ heat stack-list | grep baremetal | wc -l | 20:52 |
| bnemec | 41 | 20:52 |
| bnemec | Now we've only got 2/3 of capacity being used | 20:53 |
| *** jmelvin has joined #tripleo | 20:53 | |
| *** d0ugal has joined #tripleo | 20:54 | |
| *** d0ugal has quit IRC | 20:54 | |
| *** d0ugal has joined #tripleo | 20:54 | |
| *** ChanServ changes topic to "TripleO : http://tripleo.org/ | https://wiki.openstack.org/wiki/TripleO | Meetings On Tuesdays at 14:00 UTC in #openstack-meeting-alt" | 20:54 | |
| jaosorior | bnemec: there was a recent restarting of zuul, is there anything we have to do? | 20:56 |
| bnemec | Oh, that sucks. | 20:56 |
| jaosorior | what? the restarting? | 20:57 |
| rlandy | dprince: do you have any thoughts on https://bugs.launchpad.net/heat-templates/+bug/1650625? | 20:57 |
| openstack | Launchpad bug 1650625 in Heat Templates "Relative path to run-os-net-config will fail for custom nic-configs saved in a different directory" [Undecided,New] | 20:57 |
| openstackgerrit | Harry Rybacki proposed openstack/tripleo-quickstart-extras: Remove unused dependency from roles https://review.openstack.org/419007 | 20:57 |
| *** chlong has joined #tripleo | 20:57 | |
| rlandy | could be considered as blocking CI w/o if we don't apply one of the workarounds | 20:57 |
| *** panda|bbl is now known as panda | 20:58 | |
| bnemec | jaosorior: I'm pretty sure that means we get to recheck all the things. | 20:59 |
| bnemec | The check-tripleo queue got emptied by the restart | 20:59 |
| jaosorior | bnemec: I was told in the infra channel that they would requeue jobs soon | 20:59 |
| bnemec | And that means rh1 is about to be very unhappy because there are 40 jobs already running that no longer correspond to a zuul job. | 21:00 |
| jaosorior | bnemec: maybe it would be worth to kill them :/ | 21:00 |
| jaosorior | bnemec: at lesat the zuul status shows that all the tripleo jobs that are currently queuing haven't started yet; so maybe it's fine | 21:01 |
| *** david-lyle has quit IRC | 21:01 | |
| dprince | rlandy: commented on the bug. | 21:01 |
| bnemec | jaosorior: Yeah, I guess we can kill all the current heat stacks. | 21:01 |
| *** dciabrin has joined #tripleo | 21:01 | |
| *** david-lyle has joined #tripleo | 21:02 | |
| *** d0ugal has quit IRC | 21:03 | |
| *** chlong has quit IRC | 21:04 | |
| *** prateek has quit IRC | 21:04 | |
| *** jprovazn has quit IRC | 21:04 | |
| rlandy | dprince: thanks - left a response on the bug | 21:06 |
| *** rbowen has quit IRC | 21:10 | |
| openstackgerrit | Tim Rozet proposed openstack/tripleo-heat-templates: [WIP] Adds service for OpenDaylight DPDK https://review.openstack.org/419648 | 21:10 |
| *** ooolpbot has joined #tripleo | 21:11 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 21:11 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1654611 | 21:11 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1654615 | 21:11 |
| openstack | Launchpad bug 1654611 in tripleo "Mitaka nonha ping test failing to upload image to glance" [Critical,Triaged] | 21:11 |
| *** ooolpbot has quit IRC | 21:11 | |
| *** rbowen has joined #tripleo | 21:11 | |
| openstack | Launchpad bug 1654615 in tripleo "Mitaka ha ping test failing to ping vm" [Critical,Triaged] - Assigned to Ben Nemec (bnemec) | 21:11 |
| jaosorior | bnemec: did you kill the stacks? | 21:13 |
| jaosorior | bnemec: for a strange reason it seems that the tripleo jobs are queuing, not sure if it's normal though; with the restart being recent and all | 21:13 |
| bnemec | jaosorior: I'm kind of waiting for jobs to start running again. | 21:14 |
| bnemec | If the current jobs end up clearing out before the new ones come in then that would be good. | 21:14 |
| bnemec | I'm not super excited about deleting heat stacks out from under jobs if I don't have to. | 21:15 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Add jaosorior's key to tripleo-cd-admins https://review.openstack.org/419653 | 21:16 |
| *** chlong has joined #tripleo | 21:17 | |
| *** openstackgerrit has quit IRC | 21:18 | |
| *** openstackgerrit has joined #tripleo | 21:18 | |
| openstackgerrit | Andreas Florath proposed openstack/diskimage-builder: Easy diskimage-builder development environment setup https://review.openstack.org/419655 | 21:18 |
| *** rbrady is now known as rbrady-afk | 21:19 | |
| *** iranzo has quit IRC | 21:20 | |
| *** jayg is now known as jayg|g0n3 | 21:22 | |
| *** leanderthal has quit IRC | 21:23 | |
| *** nyechiel_ has quit IRC | 21:25 | |
| *** athomas has quit IRC | 21:26 | |
| *** jkilpatr has quit IRC | 21:33 | |
| bnemec | Well, stacks are creating, but I see no jobs running in zuul yet. | 21:33 |
| jaosorior | bnemec: yep, everything is queuing still | 21:34 |
| *** anshul has quit IRC | 21:34 | |
| *** jmelvin has quit IRC | 21:37 | |
| *** jmelvin has joined #tripleo | 21:37 | |
| jaosorior | bnemec: yay! one job started | 21:38 |
| *** gfidente has quit IRC | 21:39 | |
| bnemec | jaosorior: I think the status is running behind. There are 9 heat stacks in rh1 now, and those are only created after the job starts. | 21:40 |
| bnemec | But stuff is running in any case. | 21:40 |
| jaosorior | well, we're good then | 21:41 |
| *** mhenkel has quit IRC | 21:41 | |
| *** mhenkel has joined #tripleo | 21:42 | |
| *** ansmith_ has quit IRC | 21:44 | |
| *** ansmith has quit IRC | 21:44 | |
| *** rhallisey has quit IRC | 21:44 | |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Add docker Glance API service https://review.openstack.org/400870 | 21:45 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Containerize MySQL https://review.openstack.org/414601 | 21:45 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: docker: eliminate copy-json.py in favor of json-file https://review.openstack.org/416420 | 21:45 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: docker: new hybrid deployment architecture and configuration https://review.openstack.org/416421 | 21:45 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Add docker heat services https://review.openstack.org/417639 | 21:45 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Add docker keystone service https://review.openstack.org/416605 | 21:45 |
| *** mhenkel has quit IRC | 21:46 | |
| *** ccamacho has joined #tripleo | 21:47 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Stop running if zuul job goes away https://review.openstack.org/418615 | 21:51 |
| *** jcoufal has quit IRC | 21:51 | |
| bnemec | \o/ | 21:51 |
| *** panda is now known as panda|Zz | 21:53 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Re-enable graphite metrics https://review.openstack.org/411460 | 21:54 |
| openstackgerrit | Merged openstack/tripleo-common: A script that pulls the latest puppet source https://review.openstack.org/416355 | 21:54 |
| *** yamahata has quit IRC | 21:56 | |
| *** yamahata has joined #tripleo | 21:56 | |
| *** egafford has quit IRC | 21:59 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/tripleo-common: Updated from global requirements https://review.openstack.org/419360 | 22:00 |
| *** trown is now known as trown|outtypewww | 22:01 | |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Add docker Glance API service https://review.openstack.org/400870 | 22:06 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Containerize MySQL https://review.openstack.org/414601 | 22:07 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Add docker heat services https://review.openstack.org/417639 | 22:07 |
| *** d0ugal has joined #tripleo | 22:07 | |
| *** absubram has quit IRC | 22:07 | |
| EmilienM | leifmadsen: back | 22:09 |
| *** lblanchard has quit IRC | 22:09 | |
| EmilienM | leifmadsen: so right now if you want to add custom firewall rules, you need to do something like | 22:09 |
| EmilienM | leifmadsen: a sec | 22:09 |
| EmilienM | https://github.com/openstack/tripleo-heat-templates/blob/master/ci/environments/scenario001-multinode.yaml#L62-L63 | 22:10 |
| EmilienM | use ControllerExtraConfig | 22:10 |
| EmilienM | where you would use tripleo::firewall::rules | 22:10 |
| EmilienM | it's an hash of rules, you have an example on: https://github.com/openstack/puppet-tripleo/blob/master/manifests/firewall.pp#L66-L74 | 22:10 |
| EmilienM | if you do that, rules will apply to controller and will be consistent. | 22:10 |
| *** ooolpbot has joined #tripleo | 22:11 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 22:11 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1654611 | 22:11 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1654615 | 22:11 |
| *** ooolpbot has quit IRC | 22:11 | |
| openstack | Launchpad bug 1654611 in tripleo "Mitaka nonha ping test failing to upload image to glance" [Critical,Triaged] | 22:11 |
| openstack | Launchpad bug 1654615 in tripleo "Mitaka ha ping test failing to ping vm" [Critical,Triaged] - Assigned to Ben Nemec (bnemec) | 22:11 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: TEST: DONT RECHECK or REVIEW: periodic jobs https://review.openstack.org/359215 | 22:12 |
| EmilienM | sshnaidm: trying promotion again ^ | 22:13 |
| *** jkilpatr has joined #tripleo | 22:14 | |
| *** fultonj has quit IRC | 22:19 | |
| jaosorior | bnemec: alright, got to the failure again http://logs.openstack.org/29/419529/8/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/d8d335c/console.html#_2017-01-12_22_15_09_756817 | 22:24 |
| jaosorior | bnemec: with this being what ran https://review.openstack.org/#/c/391738/90/scripts/deploy.sh | 22:27 |
| *** chlong has quit IRC | 22:29 | |
| jaosorior | EmilienM: if you're still around, I would really use some help with that :/ else we canc heck it tomorrow | 22:33 |
| EmilienM | jaosorior: here | 22:34 |
| *** dprince has quit IRC | 22:34 | |
| jaosorior | bnemec: basically the stuff I told to bnemec before | 22:35 |
| jaosorior | EmilienM: ^^ | 22:35 |
| jaosorior | sorry haha | 22:35 |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Extract /etc from subnode logs for easy browser viewing https://review.openstack.org/417708 | 22:35 |
| jaosorior | EmilienM: but yeah, trying to get the TLS everywhere job working with the new ovb mechanism (putting up an extra node before the undercloud has started, and using that to deploy the CA) | 22:35 |
| jaosorior | EmilienM: this is what I'm running https://review.openstack.org/#/c/391738/90/scripts/deploy.sh and this is where it's failing http://logs.openstack.org/29/419529/8/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/d8d335c/console.html#_2017-01-12_22_15_09_756817 :/ | 22:36 |
| EmilienM | running sudo over ssh :) | 22:37 |
| *** chlong has joined #tripleo | 22:37 | |
| EmilienM | -ttt is used, mhh | 22:37 |
| bnemec | jaosorior: So, just discovered something interesting. | 22:37 |
| bnemec | -ttt seems to undo the terminal setting. | 22:38 |
| bnemec | -tt works | 22:38 |
| bnemec | At least in my local Jenkins. | 22:38 |
| EmilienM | :-O | 22:38 |
| EmilienM | TIL | 22:38 |
| jaosorior | whaaaa | 22:40 |
| jaosorior | bnemec: trying it out | 22:41 |
| jaosorior | bnemec: wait, you run a local Jenkins? | 22:41 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: TLS everywhere job introduction https://review.openstack.org/391738 | 22:41 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: DO NOT MERGE - testing TLS everywhere job https://review.openstack.org/419529 | 22:41 |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder: Add spec for debootstrap-minimal element https://review.openstack.org/419679 | 22:41 |
| bnemec | jaosorior: Yeah, I set it up back in the gerrithub days. | 22:41 |
| jaosorior | bnemec: what do you do with it? | 22:42 |
| bnemec | Right now it basically just builds the test rpms at https://repos.fedorapeople.org/repos/openstack-m/jenkins/rpm-build/ | 22:42 |
| jaosorior | wow | 22:42 |
| bnemec | And periodically comes in handy when I need to test something like running a script without a terminal. ;-) | 22:43 |
| jaosorior | hahaha | 22:43 |
| jaosorior | dude, thanks a lot of checking this out | 22:44 |
| jaosorior | if this works, and we merge the TLS job tomorrow, I'm gonna get free shots for a lot of people on the weekend O_O | 22:45 |
| jaosorior | (I'm in Mexico currently, stuff is cheaper than in Finland :P) | 22:45 |
| bnemec | My uncle tells me even the cheap tequila there is amazing. :-) | 22:46 |
| bnemec | And as someone who has recently discovered he's a tequila snob (I thought I just didn't like it, until I tried some good stuff), I approve. | 22:46 |
| jaosorior | bnemec: DUDE, should have known earlier! Could have brought a good bottle to the summit | 22:47 |
| jaosorior | bnemec: you also wanna try craft mezcals | 22:48 |
| bnemec | jaosorior: Fortunately summits are generally well-supplied with free alcohol. | 22:48 |
| bnemec | And apparently on those rare occasions where I do have to buy my own, I just leave and make EmilienM foot the bill. | 22:48 |
| jaosorior | mezcal is pretty much tequila, except they can't call it that way cause it's not done in the same region. But it's based on Agave as well | 22:48 |
| bnemec | (in my defense, I still almost missed the bus) | 22:49 |
| bnemec | That seems to be the story behind a lot of similar alcohols. | 22:49 |
| jaosorior | hahaha crap | 22:49 |
| bnemec | Same ingredients, same process, different regions. | 22:49 |
| jaosorior | yep | 22:49 |
| EmilienM | bnemec: :-D | 22:50 |
| jaosorior | bnemec: anyway, this month I'll take the train to Tequila (which is in the state of Jalisco, where I'm currently staying at), I'll let you know how is it | 22:50 |
| jaosorior | as far as I've been told, that whole train ride is free samples of tequila, and it can get pretty rough O_O | 22:51 |
| jroll | um, that's awesome | 22:51 |
| jaosorior | haha lets see | 22:52 |
| * jroll finds out where those run from | 22:52 | |
| jaosorior | jroll: from Guadalajara | 22:52 |
| jaosorior | jroll: if you drop by Guadalajara, let me know, I'm here the whole January | 22:52 |
| jroll | aha, cool | 22:53 |
| jroll | I won't be any time soon but am thinking about mexico for a honeymoon later this year | 22:53 |
| jaosorior | jroll: well, it depends on what you like to do | 22:53 |
| jroll | jaosorior: the goal is strictly to sit on a beach for a week, but if there's a scenic route that includes tequila samples, I may be in on that :) | 22:54 |
| jaosorior | jroll: haha, for beaches, the best ones are on the other side of the country | 22:54 |
| jroll | ah, yes, maps say you are right | 22:55 |
| jaosorior | jroll: so something around the Yucatan peninsula is what you're looking for | 22:55 |
| jroll | indeed | 22:55 |
| * jaosorior is mexican | 22:55 | |
| jaosorior | jroll: then you'll also get close to some nice pyramids and everything, so, that's where you wanna go | 22:56 |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder: Start with an empty environment for chroot https://review.openstack.org/413868 | 22:56 |
| jaosorior | they've got some cool stuff there | 22:56 |
| jroll | oh nice | 22:56 |
| EmilienM | I would never try to play with jaosorior at the bar | 22:56 |
| jaosorior | :P | 22:58 |
| jroll | heh | 22:58 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras: WIP: Use openstack deploy to install the undercloud https://review.openstack.org/419040 | 22:58 |
| *** owalsh has joined #tripleo | 23:00 | |
| * bnemec kicks of rechecks and bails | 23:02 | |
| bnemec | *off | 23:02 |
| jaosorior | bnemec: have a good one dude | 23:02 |
| jaosorior | bnemec: and thanks for the help | 23:02 |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder: Start with an empty environment for chroot https://review.openstack.org/413868 | 23:02 |
| bnemec | np | 23:04 |
| *** morazi has quit IRC | 23:06 | |
| openstackgerrit | Merged openstack/diskimage-builder: Fix Gentoo builds on Ubuntu 16.04 Xenial hosts https://review.openstack.org/413395 | 23:07 |
| *** jmelvin has quit IRC | 23:08 | |
| *** yamahata has quit IRC | 23:10 | |
| *** ooolpbot has joined #tripleo | 23:11 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 23:11 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1654611 | 23:11 |
| openstack | Launchpad bug 1654611 in tripleo "Mitaka nonha ping test failing to upload image to glance" [Critical,Triaged] | 23:11 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1654615 | 23:11 |
| *** ooolpbot has quit IRC | 23:11 | |
| openstack | Launchpad bug 1654615 in tripleo "Mitaka ha ping test failing to ping vm" [Critical,Triaged] - Assigned to Ben Nemec (bnemec) | 23:11 |
| *** yamahata has joined #tripleo | 23:12 | |
| *** colonwq has quit IRC | 23:14 | |
| *** saneax-_-|AFK is now known as saneax | 23:19 | |
| *** jaosorior has quit IRC | 23:22 | |
| *** kberger has quit IRC | 23:35 | |
| *** rook-lollll is now known as rook | 23:37 | |
| openstackgerrit | Andreas Florath proposed openstack/diskimage-builder: Docker matrix build https://review.openstack.org/414347 | 23:45 |
| *** [1]cdearborn has quit IRC | 23:47 | |
| leifmadsen | EmilienM: ping? (I know it's late) | 23:50 |
| leifmadsen | just back, going to read your backscroll | 23:50 |
| *** pmannidi has joined #tripleo | 23:50 | |
| *** mhenkel has joined #tripleo | 23:51 | |
| EmilienM | leifmadsen: sure, let me know if you got it | 23:51 |
| *** owalsh has quit IRC | 23:51 | |
| leifmadsen | sure -- so the person I'm working with is trying to do post-deploy firewall control (not really the OOO way, and I'm working to fix that, but I'm trying to understand kinda how you would if you for some reason had to) | 23:51 |
| leifmadsen | from what I saw briefly, that looks like how you would control it via the deploy method with THT right? | 23:52 |
| EmilienM | leifmadsen: yes, in postconfig thing | 23:54 |
| *** lblanchard has joined #tripleo | 23:55 | |
| EmilienM | leifmadsen: http://tripleo.org/advanced_deployment/node_config.html | 23:55 |
| *** mhenkel has quit IRC | 23:55 | |
| leifmadsen | EmilienM: sorry... I mean, outside of OOO | 23:55 |
| EmilienM | I'm not sure I understand the question. | 23:56 |
| EmilienM | is your question about tripleo or not? | 23:56 |
| leifmadsen | well, it is about where the data is dealt with on the node after OOO has done it's thing | 23:56 |
| leifmadsen | the scenario is, after OOO deploy, said person made a change (say manually) on the node, rebooted it, and the changes were reverted, so I'm mostly just trying to understand how the revert happened | 23:57 |
| EmilienM | which change? | 23:58 |
| leifmadsen | a change to iptables | 23:58 |
| EmilienM | this person made this rule persistent? | 23:58 |
| leifmadsen | a 'service iptables save' was run | 23:58 |
| EmilienM | and the rule wasn't persistent at reboot? | 23:58 |
| leifmadsen | correct | 23:58 |
| EmilienM | because Puppet is not run at boot time | 23:58 |
| leifmadsen | hrmm | 23:58 |
| EmilienM | so it's impossible TripleO removed the rule | 23:58 |
| EmilienM | this person probably missed how to save an iptables rule on the system. | 23:59 |
| EmilienM | but I would highly suggest to manage this rule with TripleO. | 23:59 |
| EmilienM | by the method I told you | 23:59 |
| EmilienM | and I strongly suggest to avoid manual things | 23:59 |
| EmilienM | because they overlap with that puppet is doing in tripleo and it's out of control of tripleo if you do manual things | 23:59 |
| leifmadsen | ok, the only reason I was confused was because I was involved in another project where it seemed like the same type of thing happened, and a theory (that I hadn't yet had a chance to confirm, I was waiting for my cloud to deploy) was that something happened at boot to put the node back to the same state as defined by OOO | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!