| openstackgerrit | RedHat RDO CI proposed openstack/tripleo-heat-templates stable/queens: GATE CHECK for TripleO https://review.openstack.org/567224 | 00:00 |
|---|---|---|
| openstackgerrit | RedHat RDO CI proposed openstack/tripleo-heat-templates stable/ocata: GATE CHECK for TripleO https://review.openstack.org/564291 | 00:00 |
| *** rwsu has quit IRC | 00:00 | |
| *** mjturek has quit IRC | 00:02 | |
| *** ooolpbot has joined #tripleo | 00:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783762 | 00:10 |
| openstack | Launchpad bug 1783762 in tripleo "Containerized Ironic BM (ILO) provisioning issue: DriverLoadError: Driver, hardware type or interface ilo-pxe could not be loaded. Reason: [Errno 13] Permission denied: '/var/lib/ironic/httpboot/" [Critical,Triaged] | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784078 | 00:10 |
| *** ooolpbot has quit IRC | 00:10 | |
| openstack | Launchpad bug 1784078 in tripleo "scenario jobs in stable/queens are broken with ansible file issues: could not locate file in lookup: Controller/step_config.pp" [Critical,Triaged] | 00:10 |
| *** etingof has joined #tripleo | 00:11 | |
| *** artom has quit IRC | 00:12 | |
| *** Goneri has joined #tripleo | 00:12 | |
| *** agopi|brb has joined #tripleo | 00:20 | |
| openstackgerrit | Merged openstack/diskimage-builder master: cache-url requires curl https://review.openstack.org/542350 | 00:20 |
| openstackgerrit | Goutham Pacha Ravi proposed openstack/paunch master: Switch docs to openstackdocstheme https://review.openstack.org/587256 | 00:21 |
| *** agopi|brb is now known as agopi | 00:23 | |
| *** jtcressy has quit IRC | 00:27 | |
| *** Goneri has quit IRC | 00:28 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Move LVM cleanup phase into cleanup https://review.openstack.org/579102 | 00:30 |
| *** rlandy has quit IRC | 00:31 | |
| *** medberry has joined #tripleo | 00:37 | |
| *** medberry has quit IRC | 00:37 | |
| *** medberry has joined #tripleo | 00:37 | |
| *** edmondsw has joined #tripleo | 00:42 | |
| *** lblanchard has joined #tripleo | 00:44 | |
| *** edmondsw has quit IRC | 00:48 | |
| *** medberry has quit IRC | 00:50 | |
| *** lblanchard has quit IRC | 00:54 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/queens: Copy-in redis certs via kolla extended/start https://review.openstack.org/586978 | 00:58 |
| openstackgerrit | Merged openstack/tripleo-common master: Update failures listing to use latest ansible-errors.json location https://review.openstack.org/583293 | 00:58 |
| *** rfolco|off is now known as rfolco|ruck | 01:01 | |
| openstackgerrit | Merged openstack/tripleo-quickstart master: save curl errors in separate file for future debug https://review.openstack.org/580214 | 01:05 |
| *** lblanchard has joined #tripleo | 01:06 | |
| *** mrsoul has joined #tripleo | 01:08 | |
| *** ooolpbot has joined #tripleo | 01:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783762 | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784078 | 01:10 |
| *** ooolpbot has quit IRC | 01:10 | |
| openstack | Launchpad bug 1783762 in tripleo "Containerized Ironic BM (ILO) provisioning issue: DriverLoadError: Driver, hardware type or interface ilo-pxe could not be loaded. Reason: [Errno 13] Permission denied: '/var/lib/ironic/httpboot/" [Critical,Triaged] | 01:10 |
| openstack | Launchpad bug 1784078 in tripleo "scenario jobs in stable/queens are broken with ansible file issues: could not locate file in lookup: Controller/step_config.pp" [Critical,Triaged] | 01:10 |
| *** mrsoul` has quit IRC | 01:10 | |
| *** lblanchard has quit IRC | 01:13 | |
| *** Petersingh has joined #tripleo | 01:16 | |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Search for last public image for reproducer https://review.openstack.org/583198 | 01:16 |
| openstackgerrit | Merged openstack/tripleo-quickstart master: Add option to install additional roles https://review.openstack.org/576816 | 01:16 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Fix parameter name used to create the Manila CephX keyring https://review.openstack.org/585562 | 01:18 |
| openstackgerrit | Merged openstack/puppet-tripleo stable/pike: Remove INSECURE_REGISTRY from docker_registry.pp https://review.openstack.org/587190 | 01:18 |
| *** shreshtha has quit IRC | 01:26 | |
| *** wolverineav has joined #tripleo | 01:30 | |
| *** wolverineav has quit IRC | 01:34 | |
| *** rbrady_ has joined #tripleo | 01:35 | |
| *** rbrady_ has quit IRC | 01:35 | |
| *** rbrady_ has joined #tripleo | 01:35 | |
| *** rbrady has quit IRC | 01:37 | |
| openstackgerrit | Goutham Pacha Ravi proposed openstack/paunch master: Switch docs to openstackdocstheme https://review.openstack.org/587256 | 01:46 |
| *** rh-jelabarre has quit IRC | 01:53 | |
| openstackgerrit | Goutham Pacha Ravi proposed openstack/paunch master: Switch docs to openstackdocstheme https://review.openstack.org/587256 | 01:54 |
| gouthamr | :| | 01:54 |
| *** yamahata has quit IRC | 01:54 | |
| *** rfolco|ruck is now known as rfolco|off | 01:55 | |
| openstackgerrit | Steve Baker proposed openstack/tripleo-docs master: Document "openstack tripleo container image prepare" https://review.openstack.org/553104 | 02:01 |
| *** dparkes has quit IRC | 02:03 | |
| *** ooolpbot has joined #tripleo | 02:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783762 | 02:10 |
| openstack | Launchpad bug 1783762 in tripleo "Containerized Ironic BM (ILO) provisioning issue: DriverLoadError: Driver, hardware type or interface ilo-pxe could not be loaded. Reason: [Errno 13] Permission denied: '/var/lib/ironic/httpboot/" [Critical,Triaged] | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784078 | 02:10 |
| *** ooolpbot has quit IRC | 02:10 | |
| openstack | Launchpad bug 1784078 in tripleo "scenario jobs in stable/queens are broken with ansible file issues: could not locate file in lookup: Controller/step_config.pp" [Critical,Triaged] | 02:10 |
| *** threestrands has quit IRC | 02:10 | |
| *** karthiks has quit IRC | 02:30 | |
| *** karthiks has joined #tripleo | 02:42 | |
| *** ramishra has joined #tripleo | 02:51 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: upgrade: move removal of services at step 3 (leftovers) https://review.openstack.org/587270 | 02:56 |
| EmilienM | mwhahaha: ^ we'll need this one asap | 02:56 |
| *** psachin has joined #tripleo | 02:56 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: fs050: enable TLS https://review.openstack.org/585333 | 02:57 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: fs050: enable TLS https://review.openstack.org/585333 | 02:57 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-docs master: Document "openstack tripleo container image prepare" https://review.openstack.org/553104 | 02:59 |
| EmilienM | stevebaker: fixed some whitespace and +2 ^ | 03:00 |
| *** ooolpbot has joined #tripleo | 03:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783762 | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784078 | 03:10 |
| *** ooolpbot has quit IRC | 03:10 | |
| openstack | Launchpad bug 1783762 in tripleo "Containerized Ironic BM (ILO) provisioning issue: DriverLoadError: Driver, hardware type or interface ilo-pxe could not be loaded. Reason: [Errno 13] Permission denied: '/var/lib/ironic/httpboot/" [Critical,Triaged] | 03:10 |
| openstack | Launchpad bug 1784078 in tripleo "scenario jobs in stable/queens are broken with ansible file issues: could not locate file in lookup: Controller/step_config.pp" [Critical,Triaged] | 03:10 |
| *** links has joined #tripleo | 03:15 | |
| *** dparkes has joined #tripleo | 03:19 | |
| *** itlinux_ has joined #tripleo | 03:21 | |
| *** medberry has joined #tripleo | 03:22 | |
| *** medberry has quit IRC | 03:22 | |
| *** medberry has joined #tripleo | 03:22 | |
| *** threestrands has joined #tripleo | 03:26 | |
| *** medberry has quit IRC | 03:31 | |
| *** rbrady__ has joined #tripleo | 03:31 | |
| openstackgerrit | Merged openstack/puppet-pacemaker master: Pin net-telnet to 0.1.1 https://review.openstack.org/587202 | 03:32 |
| *** rbrady_ has quit IRC | 03:34 | |
| *** mmethot has quit IRC | 03:41 | |
| *** Petersingh is now known as Petersingh|afk | 03:45 | |
| *** Haresh has joined #tripleo | 03:49 | |
| *** skramaja has joined #tripleo | 03:50 | |
| *** psahoo has joined #tripleo | 03:50 | |
| *** khyr0n has joined #tripleo | 03:51 | |
| *** dparkes has quit IRC | 03:58 | |
| *** skramaja_ has joined #tripleo | 04:00 | |
| *** skramaja has quit IRC | 04:00 | |
| *** jaganathan has joined #tripleo | 04:00 | |
| *** jaganathan has quit IRC | 04:02 | |
| *** jaganathan has joined #tripleo | 04:02 | |
| *** jaganathan has joined #tripleo | 04:03 | |
| *** gkadam has joined #tripleo | 04:05 | |
| *** mschuppert has joined #tripleo | 04:06 | |
| *** shadower has quit IRC | 04:06 | |
| *** homegrown has quit IRC | 04:07 | |
| *** psachin` has joined #tripleo | 04:08 | |
| *** psachin has quit IRC | 04:08 | |
| *** shreshtha has joined #tripleo | 04:08 | |
| *** yolanda has quit IRC | 04:09 | |
| *** shreshtha has quit IRC | 04:09 | |
| *** shreshtha has joined #tripleo | 04:09 | |
| openstackgerrit | Merged openstack/tripleo-common master: Revert "Add disable-nouveau element to tripleo images" https://review.openstack.org/586591 | 04:10 |
| *** homegrown has joined #tripleo | 04:14 | |
| *** mmethot has joined #tripleo | 04:18 | |
| *** Petersingh|afk is now known as Petersingh | 04:18 | |
| *** shadower has joined #tripleo | 04:21 | |
| *** ykarel|away has joined #tripleo | 04:23 | |
| *** yolanda has joined #tripleo | 04:24 | |
| itlinux_ | hello all, do we have openflow enable by default.. my understanding was that was off. Thanks | 04:28 |
| *** pdeore has joined #tripleo | 04:28 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: upgrade: move removal of services at step 3 (leftovers) https://review.openstack.org/587270 | 04:30 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: fs050: enable TLS https://review.openstack.org/585333 | 04:30 |
| *** ade_lee has quit IRC | 04:36 | |
| *** ade_lee has joined #tripleo | 04:36 | |
| stevebaker | EmilienM: thanks | 04:37 |
| *** janki has joined #tripleo | 04:50 | |
| *** udesale has joined #tripleo | 04:50 | |
| openstackgerrit | yatin proposed openstack/tripleo-quickstart-extras master: Remove single quote from docker inspect command in tempest run https://review.openstack.org/586213 | 04:51 |
| ykarel|away | chandankumar, ^^ trying seperately | 04:51 |
| ykarel|away | we need to unblock promotions and fix things seperately | 04:52 |
| ykarel|away | chandankumar, also trying ^^ with a promotion job | 04:53 |
| *** ade_lee has quit IRC | 05:06 | |
| chandankumar | ykarel|away: ok | 05:08 |
| *** bogdando has joined #tripleo | 05:10 | |
| *** dparkes has joined #tripleo | 05:21 | |
| *** skramaja_ is now known as skramaja | 05:21 | |
| *** ykarel|away has quit IRC | 05:26 | |
| *** jfrancoa has joined #tripleo | 05:29 | |
| *** ratailor has joined #tripleo | 05:35 | |
| openstackgerrit | Merged openstack/tripleo-docs master: Document "openstack tripleo container image prepare" https://review.openstack.org/553104 | 05:38 |
| *** ksambor has joined #tripleo | 05:42 | |
| *** ykarel|away has joined #tripleo | 05:55 | |
| *** nyechiel has joined #tripleo | 05:58 | |
| jaosorior | dciabrin: yoo | 06:02 |
| jaosorior | itlinux_: it's off AFAIK | 06:02 |
| *** gkadam has quit IRC | 06:04 | |
| *** gkadam has joined #tripleo | 06:05 | |
| *** Petersingh is now known as Petersingh|afk | 06:06 | |
| *** waleedm has joined #tripleo | 06:09 | |
| *** avivgt has joined #tripleo | 06:17 | |
| *** Petersingh|afk is now known as Petersingh | 06:17 | |
| *** quiquell has joined #tripleo | 06:18 | |
| *** agurenko has joined #tripleo | 06:19 | |
| *** moguimar has joined #tripleo | 06:19 | |
| *** ccamacho has joined #tripleo | 06:21 | |
| *** ade_lee has joined #tripleo | 06:29 | |
| *** khyr0n has quit IRC | 06:29 | |
| *** ade_lee has quit IRC | 06:31 | |
| openstackgerrit | Cédric Jeanneret proposed openstack/python-tripleoclient master: New function: run_ansible_playbook https://review.openstack.org/586538 | 06:32 |
| Tengu | #moreTests :] | 06:32 |
| bandini | chandankumar: is this tempest failure known? http://logs.openstack.org/48/583648/3/gate/tripleo-ci-centos-7-undercloud-containers/0a5cdea/logs/undercloud/home/zuul/tempest.log.txt.gz#_2018-07-30_22_02_11 | 06:33 |
| * bandini scours through open LP bugs | 06:33 | |
| *** zaneb has quit IRC | 06:34 | |
| *** zaneb has joined #tripleo | 06:34 | |
| *** ykarel|away is now known as ykarel | 06:35 | |
| openstackgerrit | Quique Llorente proposed openstack-infra/tripleo-ci master: Take playbooks from ansible https://review.openstack.org/586513 | 06:36 |
| *** pcaruana has joined #tripleo | 06:37 | |
| *** agurenko has quit IRC | 06:37 | |
| chandankumar | bandini: it is a timed out issue | 06:37 |
| *** yamahata has joined #tripleo | 06:38 | |
| bandini | chandankumar: ah ok. /me rechecks then. thanks! | 06:39 |
| *** agurenko has joined #tripleo | 06:40 | |
| *** bogdando has quit IRC | 06:44 | |
| *** yprokule has joined #tripleo | 06:48 | |
| openstackgerrit | Quique Llorente proposed openstack-infra/tripleo-ci master: Take playbooks from ansible https://review.openstack.org/586513 | 06:48 |
| openstackgerrit | Quique Llorente proposed openstack-infra/tripleo-ci master: Take playbooks from ansible https://review.openstack.org/586513 | 06:49 |
| openstackgerrit | Quique Llorente proposed openstack-infra/tripleo-ci master: Cleanup variables and toci_jobtype leftovers https://review.openstack.org/584465 | 06:55 |
| *** yamahata has quit IRC | 06:55 | |
| openstackgerrit | Quique Llorente proposed openstack-infra/tripleo-ci master: Cleanup variables and toci_jobtype leftovers https://review.openstack.org/584465 | 06:55 |
| *** yamahata has joined #tripleo | 06:56 | |
| openstackgerrit | Quique Llorente proposed openstack-infra/tripleo-ci master: Take playbooks from ansible https://review.openstack.org/586513 | 06:59 |
| *** ukalifon has joined #tripleo | 07:00 | |
| openstackgerrit | Quique Llorente proposed openstack-infra/tripleo-ci master: Cleanup variables and toci_jobtype leftovers https://review.openstack.org/584465 | 07:03 |
| openstackgerrit | Quique Llorente proposed openstack-infra/tripleo-ci master: Replace TAGS with ansible var https://review.openstack.org/584508 | 07:03 |
| *** quiquell is now known as quiquell|bbl | 07:04 | |
| *** jaganathan has quit IRC | 07:09 | |
| *** jaganathan has joined #tripleo | 07:09 | |
| *** peereb has joined #tripleo | 07:10 | |
| dciabrin | jaosorior, hey o/ | 07:11 |
| jaosorior | dciabrin: how's it going? Did the deployment work? | 07:13 |
| *** links has quit IRC | 07:13 | |
| dciabrin | jaosorior, the 3-node ha worked, i'm currently doing the non-HA. let me ping you when it's sorted out | 07:14 |
| openstackgerrit | Merged openstack/ansible-role-tripleo-modify-image master: Zuul: add missing jobs for release https://review.openstack.org/586380 | 07:16 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Add support for containerized networking-ansible ML2 plugin https://review.openstack.org/585194 | 07:16 |
| openstackgerrit | Arx Cruz proposed openstack/tripleo-quickstart-extras master: Fix ssl warnings on tempest containerized https://review.openstack.org/580384 | 07:17 |
| *** pmannidi has quit IRC | 07:18 | |
| openstackgerrit | Tony Breeds proposed openstack/python-tripleoclient stable/queens: Allow passing an agent-name as a command line argument https://review.openstack.org/587305 | 07:21 |
| openstackgerrit | Tony Breeds proposed openstack/python-tripleoclient stable/queens: Introduce trivial helpers to manage image_files and names https://review.openstack.org/587306 | 07:21 |
| openstackgerrit | Tony Breeds proposed openstack/python-tripleoclient stable/queens: Always define properties for image even if empty https://review.openstack.org/587307 | 07:21 |
| openstackgerrit | Tony Breeds proposed openstack/python-tripleoclient stable/queens: Add --architecture support when uploading images https://review.openstack.org/587308 | 07:21 |
| openstackgerrit | Tony Breeds proposed openstack/python-tripleoclient stable/queens: Add --platform support when uploading images https://review.openstack.org/587309 | 07:21 |
| openstackgerrit | Tony Breeds proposed openstack/python-tripleoclient stable/queens: Actually use the arch and platform to select appropriate images https://review.openstack.org/587310 | 07:21 |
| openstackgerrit | Tony Breeds proposed openstack/python-tripleoclient stable/queens: Switch to using assert_has_calls() instead of assertEqual() https://review.openstack.org/587311 | 07:21 |
| openstackgerrit | Tony Breeds proposed openstack/python-tripleoclient stable/queens: Always set a hardware architecture on uploaded images https://review.openstack.org/587312 | 07:21 |
| openstackgerrit | Merged openstack/tripleo-common master: Remove unused clients from container images https://review.openstack.org/582637 | 07:21 |
| openstackgerrit | Merged openstack/tripleo-common master: Allow deferred undercloud push_destination detection https://review.openstack.org/585589 | 07:21 |
| openstackgerrit | Merged openstack/python-tripleoclient master: Prepare defaults, defer detecting undercloud registry https://review.openstack.org/585590 | 07:21 |
| openstackgerrit | Merged openstack/ansible-role-container-registry master: Fix package README errors https://review.openstack.org/587205 | 07:21 |
| openstackgerrit | Yurii Prokulevych proposed openstack/tripleo-upgrade stable/queens: Workload flavor customization. https://review.openstack.org/587313 | 07:21 |
| *** rcernin has quit IRC | 07:22 | |
| openstackgerrit | waleed mousa proposed openstack/tripleo-heat-templates master: Removing the deprecated environment files for hw offloading https://review.openstack.org/573102 | 07:23 |
| openstackgerrit | waleed mousa proposed openstack/puppet-tripleo master: Adding support for VF LAG in SR-IOV for Mellanox interfaces https://review.openstack.org/558411 | 07:23 |
| *** bogdando has joined #tripleo | 07:24 | |
| *** amoralej|off is now known as amoralej | 07:35 | |
| *** ykarel is now known as ykarel|lunch | 07:36 | |
| *** Petersingh is now known as Petersingh|lunch | 07:37 | |
| *** janki has quit IRC | 07:38 | |
| *** nenad has joined #tripleo | 07:40 | |
| *** pblaho has joined #tripleo | 07:40 | |
| *** tesseract has joined #tripleo | 07:42 | |
| *** janki has joined #tripleo | 07:42 | |
| *** florianf has joined #tripleo | 07:43 | |
| *** threestrands has quit IRC | 07:45 | |
| *** shardy has joined #tripleo | 07:47 | |
| openstackgerrit | Quique Llorente proposed openstack-infra/tripleo-ci master: Cleanup variables and toci_jobtype leftovers https://review.openstack.org/584465 | 07:48 |
| *** udesale has quit IRC | 07:49 | |
| *** udesale has joined #tripleo | 07:49 | |
| chandankumar | bogdando: sshnaidm|afk mandre_away https://review.openstack.org/#/c/586213/9 we need to merge this | 07:50 |
| openstackgerrit | Quique Llorente proposed openstack-infra/tripleo-ci master: [WIP] Move nodes_file and nodes_args to common.yaml https://review.openstack.org/585714 | 07:51 |
| *** jpena|off is now known as jpena | 07:53 | |
| *** tosky has joined #tripleo | 07:55 | |
| openstackgerrit | Tony Breeds proposed openstack/tripleo-common stable/queens: IPMI doesn't mandate a username or password so tripleo shouldn't either https://review.openstack.org/587328 | 07:57 |
| openstackgerrit | Tony Breeds proposed openstack/tripleo-common stable/queens: Allow platform in instackenv.json https://review.openstack.org/587329 | 07:57 |
| *** mhenkel_ has joined #tripleo | 07:57 | |
| openstackgerrit | RedHat RDO CI proposed openstack/tripleo-heat-templates stable/queens: GATE CHECK for TripleO https://review.openstack.org/567224 | 08:00 |
| openstackgerrit | RedHat RDO CI proposed openstack/tripleo-quickstart-extras master: GATE CHECK for quickstart-extras https://review.openstack.org/560445 | 08:00 |
| *** threestrands has joined #tripleo | 08:00 | |
| openstackgerrit | Martin Schuppert proposed openstack/puppet-tripleo master: Move nova-metadata api to httpd wsgi https://review.openstack.org/582622 | 08:00 |
| *** threestrands has quit IRC | 08:03 | |
| *** Haresh has quit IRC | 08:04 | |
| openstackgerrit | Honza Pokorny proposed openstack/tripleo-quickstart master: Add a flag to create the reproducer script https://review.openstack.org/586843 | 08:06 |
| *** quiquell|bbl is now known as quiquell | 08:07 | |
| *** jpich has joined #tripleo | 08:10 | |
| *** derekh has joined #tripleo | 08:10 | |
| openstackgerrit | Quique Llorente proposed openstack-infra/tripleo-ci master: Replace TAGS with ansible var https://review.openstack.org/584508 | 08:13 |
| *** udesale has quit IRC | 08:13 | |
| *** links has joined #tripleo | 08:13 | |
| *** udesale has joined #tripleo | 08:16 | |
| *** ykarel|lunch is now known as ykarel | 08:18 | |
| openstackgerrit | Quique Llorente proposed openstack-infra/tripleo-ci master: [WIP] Move nodes_file and nodes_args to common.yaml https://review.openstack.org/585714 | 08:20 |
| yprokule | jaosorior: o/ | 08:20 |
| yprokule | jaosorior: how it's going ? | 08:20 |
| openstackgerrit | Merged openstack/diskimage-builder master: Only detach device if all partitions have been cleaned https://review.openstack.org/576876 | 08:21 |
| *** Petersingh|lunch is now known as Petersingh | 08:26 | |
| Tengu | heya! anyone with some experience with python "mock" library? I'm having a small issue in some unit testing for my new change in python-tripleoclient.... | 08:27 |
| *** gfidente has joined #tripleo | 08:30 | |
| *** gfidente has quit IRC | 08:30 | |
| *** gfidente has joined #tripleo | 08:30 | |
| *** gkadam has quit IRC | 08:31 | |
| therve | Tengu: shoot | 08:35 |
| *** bugzy_ is now known as bugzy | 08:35 | |
| Tengu | therve: how to mock 2x the same function, in my case I have two "os.path.exists()" calls with two different files. I want to assert they are both called with proper path | 08:36 |
| therve | Tengu: So a function stays mock unless you tell it not ot | 08:36 |
| therve | to | 08:37 |
| *** gkadam has joined #tripleo | 08:37 | |
| Tengu | therve: 1http://paste.openstack.org/show/726918/ easier with the current code I guess :). | 08:37 |
| Tengu | I tried to use the "wraps" option, but apparently I'm either not doing it right, or it's not what I thought it is | 08:38 |
| therve | Hum I never used that | 08:38 |
| Tengu | ksambor: -^^ if you want to check what I'm doing as well | 08:39 |
| therve | Tengu: I don't think you can call patch twice for the same function | 08:39 |
| openstackgerrit | Quique Llorente proposed openstack-infra/tripleo-ci master: [WIP] Remove featureset BASH variables https://review.openstack.org/587340 | 08:39 |
| therve | Call it once, and do the asserts accordingly | 08:39 |
| Tengu | hm, the second assert fails | 08:40 |
| *** ade_lee has joined #tripleo | 08:40 | |
| Tengu | the function code is: https://review.openstack.org/#/c/586538/7/tripleoclient/utils.py - calls are on 100 and 110 | 08:40 |
| therve | How does the second assert look like? | 08:40 |
| Tengu | therve: they are both in the pasted cod | 08:41 |
| Tengu | *code | 08:41 |
| ksambor | Tengu: using assert_has_calls() is not enough for this case? | 08:41 |
| *** gkadam has quit IRC | 08:41 | |
| Tengu | ksambor: want to ensure they are called with the right paths :/ | 08:42 |
| therve | Tengu: So even if you remove mock_exists_playbook ? | 08:42 |
| *** mdnadeem has joined #tripleo | 08:42 | |
| *** mdnadeem_ has joined #tripleo | 08:42 | |
| Tengu | 2s, seeing something weird | 08:43 |
| Tengu | damn. wrong order for the mock_* in test function params. anyway. http://paste.openstack.org/show/726919/ the errir | 08:45 |
| Tengu | *error | 08:45 |
| *** mdnadeem has quit IRC | 08:47 | |
| therve | Tengu: And you're sure exists is called ? | 08:47 |
| therve | Looking at the code it shouldn't be | 08:48 |
| therve | You're in the first condition | 08:48 |
| openstackgerrit | Quique Llorente proposed openstack-infra/tripleo-ci master: Replace PLAYBOOKS_DRY_RUN with ansible dry_run https://review.openstack.org/587344 | 08:48 |
| Tengu | yup. at least it should as we pass an ansible_config, that one being an absolute path | 08:48 |
| therve | Ah no | 08:48 |
| therve | Tengu: Can you commit the code? | 08:48 |
| Tengu | therve: which code? the tests? it's in a non-working state :'( | 08:49 |
| therve | Tengu: Yes | 08:49 |
| therve | Well that's ok you don't pay gerrit fees for that :) | 08:49 |
| Tengu | sure, but... well, just trying another thing. | 08:50 |
| Tengu | -.- | 08:50 |
| openstackgerrit | Cédric Jeanneret proposed openstack/python-tripleoclient master: New function: run_ansible_playbook https://review.openstack.org/586538 | 08:50 |
| Tengu | therve: -^^ | 08:50 |
| ksambor | Tengu: you can check with what argument calls are made and even order if you want https://docs.python.org/3/library/unittest.mock.html#unittest.mock.Mock.assert_has_calls | 08:51 |
| Tengu | ksambor: oh. hmm. | 08:51 |
| Tengu | might do the trick then. | 08:51 |
| Tengu | ksambor: \o/ | 08:53 |
| Tengu | just have to understand how it works, but... | 08:54 |
| *** salmankhan has joined #tripleo | 08:54 | |
| *** brault has joined #tripleo | 08:55 | |
| therve | Tengu: Yeah or assert_any_call | 08:56 |
| therve | Personally I check .mock_calls to see what it's in there | 08:56 |
| Tengu | I'm not far from success I think. | 09:01 |
| *** sshnaidm|afk has quit IRC | 09:02 | |
| *** sshnaidm has joined #tripleo | 09:04 | |
| *** paramite has joined #tripleo | 09:04 | |
| Tengu | therve, ksambor: thanks a lot for your valuable inputs! I now have a working test :). | 09:04 |
| ksambor | Tengu: np | 09:05 |
| openstackgerrit | Cédric Jeanneret proposed openstack/python-tripleoclient master: New function: run_ansible_playbook https://review.openstack.org/586538 | 09:06 |
| *** akrivoka has joined #tripleo | 09:10 | |
| *** moguimar has quit IRC | 09:11 | |
| openstackgerrit | Feilong Wang proposed openstack/os-collect-config stable/queens: Add region support https://review.openstack.org/587351 | 09:12 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates stable/queens: Fix parameter name used to create the Manila CephX keyring https://review.openstack.org/587353 | 09:15 |
| skramaja | jaosorior: shardy EmilienM could you take a look at https://review.openstack.org/#/c/581352/? | 09:18 |
| gfidente | ganesha ftw https://review.openstack.org/#/c/585695/ | 09:18 |
| gfidente | pls :D | 09:18 |
| *** ratailor has quit IRC | 09:20 | |
| skramaja | thanks jaosorior | 09:20 |
| openstackgerrit | yatin proposed openstack/tripleo-quickstart-extras master: Fix Subsequent run of tempest-setup.sh https://review.openstack.org/587356 | 09:21 |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart-extras master: Use local docker registry host for tempest container https://review.openstack.org/584368 | 09:21 |
| *** ratailor has joined #tripleo | 09:24 | |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart-extras master: Use local docker registry host for tempest container https://review.openstack.org/584368 | 09:29 |
| *** zoli is now known as zoli|lunch | 09:32 | |
| *** ksambor has quit IRC | 09:33 | |
| openstackgerrit | Quique Llorente proposed openstack-infra/tripleo-ci master: [WIP] Replace toci ENV_VARS by ansible env_vars https://review.openstack.org/587360 | 09:34 |
| *** leanderthal has joined #tripleo | 09:37 | |
| *** ksambor has joined #tripleo | 09:40 | |
| openstackgerrit | Cédric Jeanneret proposed openstack/python-tripleoclient master: New function: run_ansible_playbook https://review.openstack.org/586538 | 09:40 |
| *** moguimar has joined #tripleo | 09:44 | |
| *** panda|rover|off is now known as panda|rover | 09:48 | |
| *** ratailor has quit IRC | 09:51 | |
| *** ratailor has joined #tripleo | 09:52 | |
| *** jaosorior has quit IRC | 09:53 | |
| *** jaosorior has joined #tripleo | 09:54 | |
| *** akrivoka has quit IRC | 09:54 | |
| panda|rover | gah, where's ooolpbot ? | 09:54 |
| *** akrivoka has joined #tripleo | 09:54 | |
| *** ratailor has quit IRC | 09:56 | |
| *** ratailor has joined #tripleo | 09:57 | |
| numans | jaosorior, Hi, can you please take a look at this when you get some time - https://review.openstack.org/#/c/583306/ | 09:58 |
| shardy | jaosorior: Hey I wanted to chat about hostname/fqdn questions when you get a moment, IIRC you did some work on that for tls-everywhere? | 09:59 |
| openstackgerrit | Marios Andreou proposed openstack-infra/tripleo-ci master: Soften guard for ceph loop device in ci script so we can move it https://review.openstack.org/587369 | 10:00 |
| shardy | The question I'm being asked is why does /etc/hostname refer to the short name vs hostname -f which uses the /etc/hosts e.g overcloud-novacompute-0.localdomain | 10:00 |
| shardy | jaosorior: AFAIK that's expected but I wanted to check if I'm missing anything | 10:00 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/python-tripleoclient master: Make undercloud user-provided cert take precedence https://review.openstack.org/587370 | 10:02 |
| jaosorior | shardy: not sure I got the question | 10:02 |
| jaosorior | ah, well, I'm honestly not sure why /etc/hostname uses the shortname to be honest. Thought it would use the FQDN | 10:02 |
| *** ksambor has quit IRC | 10:03 | |
| gfidente | jaosorior shardy $hostname fwiw has implications on ceph so if we need to change it we might need test upgrade | 10:03 |
| shardy | jaosorior: Ok, that's the question basically - they changed the undercloud via hostnamectl to an fqdn but AFAIK that won't have any impact on the overcloud, and AFAIK /etc/hostname is supposed to be a short name, vs hostname -f which is derived from the DNS name (or in our case /etc/hosts) | 10:03 |
| gfidente | not that I know why it's not using fqdn, just saying that ceph relies on it | 10:03 |
| gfidente | shardy jaosorior I think the domain part is set by the neutron dhcp in undercloud | 10:04 |
| gfidente | and I think we default that to '' | 10:04 |
| shardy | gfidente: ack yeah at this point I'm just trying to understand what's exppected, and if anything changed in this area for Queens | 10:04 |
| shardy | mhenkel_: ^^ discussing your issue FYI | 10:04 |
| mhenkel_ | shardy: thx | 10:05 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: Collect logs: split and move the script creation https://review.openstack.org/587103 | 10:06 |
| shardy | mhenkel_: FYI I was looking at https://bugzilla.redhat.com/show_bug.cgi?id=1475383 because the domain for overcloud nodes should come via dhcp, which is derived from your undercloud config | 10:06 |
| openstack | bugzilla.redhat.com bug 1475383 in instack-undercloud "[osp12] overcloud node FQDN is incorrect when domain name is set via DHCP on external network" [Medium,Closed: errata] - Assigned to jslagle | 10:06 |
| mhenkel_ | the docu for undercloud installation (tripleo and osp13) is saying to set the hostname to fqdn with either hostnamectl or by editing undercloud.conf | 10:07 |
| shardy | but AFAICS that is working OK | 10:07 |
| mhenkel_ | so, I feel that the same should be done on the overcloud | 10:07 |
| mhenkel_ | shardy: yes, I read that BZ | 10:07 |
| shardy | mhenkel_: ack, I'm just trying to clarify if it's a docs or functional issue - AFAIK this hasn't changed recently but have you observed different behavior on previous releases? | 10:08 |
| mhenkel_ | in the past (AFAIK between RHEL 7.3 and 7.4) there was a change in behavior with the hostname tool | 10:08 |
| mhenkel_ | and I think to remember that `hostname` gave back the fqdn in the past. But I'm not entirely sure | 10:09 |
| shardy | mhenkel_: Ok thanks for the info - it'd be good to confirm if you do have any older environments you can check | 10:09 |
| *** Petersingh has quit IRC | 10:10 | |
| openstackgerrit | Sorin Sbarnea proposed openstack/tripleo-quickstart master: bump ansible from 2.5.5 to 2.5.7 https://review.openstack.org/587371 | 10:11 |
| mhenkel_ | shardy: I don't have a system but I have this: https://review.opencontrail.org/#/c/34705/ | 10:12 |
| shardy | mhenkel_: ack so it may be independent of any tripleo change then, thanks | 10:14 |
| mhenkel_ | shardy: and this is what I found in my past emails: http://paste.openstack.org/show/726930/ | 10:15 |
| *** ksambor has joined #tripleo | 10:15 | |
| mhenkel_ | there you can see that between 7.3 and 7.4 the output of `hostname` changed from fqdn to short name | 10:16 |
| openstackgerrit | Marios Andreou proposed openstack-infra/tripleo-ci master: Remove tripleo.sh --bootstrap-subnodes add ceph loop device https://review.openstack.org/583195 | 10:19 |
| jtomasek_ | shardy: hi, this expression is failing for me when $root.role_tags is null (which is in case when there are no role_tags defined in plan-environment.yaml https://github.com/openstack/tripleo-common/blob/master/workbooks/plan_management.yaml#L1408 | 10:24 |
| openstackgerrit | Martin Schuppert proposed openstack/tripleo-heat-templates master: Add NovaResumeGuestsStateOnHostBoot and NovaResumeGuestsShutdownTimeout https://review.openstack.org/577414 | 10:24 |
| shardy | mhenkel_: Ack, Ok looking at the 7.4 release notes there were a bunch of changes around dhcp & fqdn but I've not spotted the exact change that caused this | 10:24 |
| owalsh | shardy: cloud-init? | 10:25 |
| shardy | my guess is the rhel folks will consider the old behavior a bug since hostname -f still works, but you're welcome to raise a bz so we can discuss it | 10:25 |
| openstackgerrit | Cédric Jeanneret proposed openstack/python-tripleoclient master: New function: run_ansible_playbook https://review.openstack.org/586538 | 10:25 |
| shardy | owalsh: between 7.3 and 7.4 apparaently the default configuration of /etc/hostname (and therefore unqualified "hostname" CLI and puppet hostname facts) from fqdn to a short name | 10:26 |
| shardy | which AFAICS is correct, but it's caused some issues for mhenkel_ on upgrade I believe | 10:26 |
| shardy | I'm not entirely sure what changed but there's a few references to init script changes in the release notes | 10:26 |
| shardy | could also be cloud-init I guess as I know that sets the initial hostname | 10:27 |
| shardy | may present some interesting challenges when you upgrade then add new nodes I guess :( | 10:27 |
| owalsh | shardy: there are some known issues with the hostname changes... https://bugs.launchpad.net/tripleo/+bug/1763322 is the biggest problem I think | 10:31 |
| openstack | Launchpad bug 1763322 in tripleo "M->Newton upgrade causes all fip to disappear." [High,In progress] - Assigned to Sofer Athlan-Guyot (sofer-athlan-guyot) | 10:31 |
| jtomasek_ | shardy: nm, I think I have a fix for that | 10:31 |
| openstackgerrit | Arx Cruz proposed openstack/tripleo-quickstart master: Running containerized tempest only in containerized environment https://review.openstack.org/577780 | 10:31 |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-common master: Fix available roles listing https://review.openstack.org/587380 | 10:42 |
| shardy | owalsh: ack thanks for the link, not sure that's related to this issue but good to have the context | 10:42 |
| shardy | jtomasek_: ack thanks, I think we'll have to default it | 10:42 |
| jtomasek_ | shardy: ^^ | 10:42 |
| numans | shardy, Hi, can you plz take a look at this when you get some time - https://review.openstack.org/#/c/583306/. Thanks | 10:45 |
| *** ksambor is now known as ksambor|lunch | 10:46 | |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui master: Fix error message in case of available roles listing failure https://review.openstack.org/587381 | 10:46 |
| *** zoli|lunch is now known as zoli | 10:47 | |
| owalsh | shardy: ack, /me scrolling back. So we had a similar issue with nova/neutron/ceilo not setting host in the conf files. They then default to socket.gethostname() which may or may not return an fqdn.... | 10:48 |
| owalsh | then I changed t-h-t to always set host to the fqdn, but this caused problems for upgrades.... | 10:49 |
| shardy | owalsh: ack thanks, yeah that sounds similar to what mhenkel_ is reporting, so it may just be a known issue I guess | 10:50 |
| owalsh | shardy: but this is for ceph? | 10:50 |
| shardy | owalsh: No I think it's impacted some contrail integration | 10:50 |
| owalsh | ah, ok... I expect it's similar issue where the hostname is used as the uuid, so when it changes we get multiple services in the DB and any existing references refer to the old entry | 10:53 |
| waleedm | Hi guys, I still have problem with introspection for baremetal nodes, as following: | 10:55 |
| waleedm | Failed to prepare node def465cf-cf64-4ff4-8f90-af2b3b1dc6fc for cleaning: Failed to create neutron ports for any PXE enabled port on node | 10:55 |
| waleedm | Could anyone help with that, please ? | 10:55 |
| *** tzumainn has joined #tripleo | 10:57 | |
| openstackgerrit | Sorin Sbarnea proposed openstack/tripleo-quickstart master: Fix missing --requirements argument on quickstart.sh https://review.openstack.org/587384 | 11:00 |
| sri_ | waleedm, Hi can you try running this from your undercloud ipmitool -I lanplus -U <username> -P <password> -H <host IPMI ip> power status | 11:00 |
| sri_ | ipmitool -I lanplus -U <username> -P <password> -H <host IPMI ip> power status | 11:00 |
| sri_ | waleedm, that will conform everything configured properly from IPMI side | 11:01 |
| sri_ | waleedm, and also make sure your overcloud nodes booting from right nic | 11:02 |
| openstackgerrit | Sorin Sbarnea proposed openstack/tripleo-quickstart master: Fix --requirements argument on quickstart.sh https://review.openstack.org/587384 | 11:02 |
| waleedm | sri_: Chassis Power is off | 11:04 |
| sri_ | waleedm, that means everything looks good from IPMI side | 11:05 |
| waleedm | sri_: actually the nodes are introspected successfully but after that it goes to clean failed error | 11:05 |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Remove single quote from docker inspect command in tempest run https://review.openstack.org/586213 | 11:06 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/queens: Revert "Delete not-used services-docker files" https://review.openstack.org/586480 | 11:06 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/queens: Revert "Move remnants of environments/services-docker" https://review.openstack.org/586484 | 11:06 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/queens: Fix errors using multiple-nics templates w/o VLANs defined https://review.openstack.org/586722 | 11:06 |
| *** salmankhan has quit IRC | 11:06 | |
| waleedm | sri_: after the node introspected and shutdown I got that error | 11:07 |
| sri_ | waleedm: hmmm, that is strange | 11:07 |
| flaper87 | trown|outtypewww: thanks for working on the 3.10 patch | 11:08 |
| waleedm | sri_: and that happened only with physical machines but for vms it finished successfully | 11:08 |
| sri_ | waleedm, can you look into Ironic inspecter logs | 11:09 |
| openstackgerrit | Sorin Sbarnea proposed openstack/tripleo-quickstart master: Fix --requirements argument on quickstart.sh https://review.openstack.org/587384 | 11:10 |
| sri_ | waleedm, can you also look into this https://access.redhat.com/solutions/3349081 | 11:13 |
| *** ksambor|lunch is now known as ksambor | 11:17 | |
| *** salmankhan has joined #tripleo | 11:17 | |
| waleedm | sri_: I'll disable the cleaning now and check | 11:17 |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui master: Enable sorting of environments https://review.openstack.org/587385 | 11:18 |
| Tengu | gfidente or fultonj: Heya! small question: what's the latest docker tag for ceph containers on Pike? | 11:21 |
| *** rh-jelabarre has joined #tripleo | 11:22 | |
| *** pradk has joined #tripleo | 11:23 | |
| *** ansmith has quit IRC | 11:29 | |
| *** wolverineav has joined #tripleo | 11:33 | |
| *** bfournie has quit IRC | 11:36 | |
| *** morazi has joined #tripleo | 11:39 | |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-quickstart-extras master: [DNM] To test cleanup toci_jobtype leftovers https://review.openstack.org/587394 | 11:40 |
| waleedm | sri_: ther undercloud is containerized now, how to save the changes to the ironic_conductor container ? | 11:41 |
| *** lvdombrkr has joined #tripleo | 11:44 | |
| Tengu | Soo. good people of tripleo - care to have a look at this review? https://review.openstack.org/#/c/586538/ the CI is running on it, but seeing the unit tests and pep8 working, and as it's a new function not yet in use, I'm pretty condident :). | 11:45 |
| *** nenad has quit IRC | 11:46 | |
| *** jpena is now known as jpena|lunch | 11:46 | |
| *** adrianc_ has joined #tripleo | 11:51 | |
| *** raildo has joined #tripleo | 11:53 | |
| openstackgerrit | Cédric Jeanneret proposed openstack/python-tripleoclient master: New preflight check for the undercloud: disk space https://review.openstack.org/586541 | 11:53 |
| *** amoralej is now known as amoralej|lunch | 11:55 | |
| sri_ | waleedm, I don't have undercloud in a containers, but i believe Ironic volume is mounted to undercloud right ? | 11:55 |
| openstackgerrit | Quique Llorente proposed openstack-infra/tripleo-ci master: Replace TAGS with ansible var https://review.openstack.org/584508 | 11:55 |
| *** abishop has joined #tripleo | 11:58 | |
| openstackgerrit | Quique Llorente proposed openstack-infra/tripleo-ci master: [WIP] Run emit_releases_file.py for all featuresets https://review.openstack.org/585649 | 12:03 |
| *** leanderthal has quit IRC | 12:04 | |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-heat-templates stable/queens: [DNM] To test emit_releases_file.py https://review.openstack.org/585656 | 12:04 |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-heat-templates stable/pike: [DNM] To test emit_releases_file.py https://review.openstack.org/585657 | 12:05 |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-heat-templates stable/ocata: [DNM] To test emit_releases_file.py https://review.openstack.org/585658 | 12:05 |
| openstackgerrit | Quique Llorente proposed openstack-infra/tripleo-ci master: Remove STATS vars from TOCI https://review.openstack.org/587398 | 12:07 |
| *** artom has joined #tripleo | 12:08 | |
| *** rfolco|off is now known as rfolco|ruck | 12:12 | |
| *** waleedm has quit IRC | 12:13 | |
| *** waleedm has joined #tripleo | 12:14 | |
| *** Haresh has joined #tripleo | 12:14 | |
| openstackgerrit | Quique Llorente proposed openstack-infra/tripleo-ci master: Refactor DRY_RUN, STATS_* and COLLECT_CONF vars https://review.openstack.org/587344 | 12:17 |
| *** rbrady__ is now known as rbrady | 12:18 | |
| *** rbrady has joined #tripleo | 12:18 | |
| *** panda|rover is now known as panda|rover|lunc | 12:18 | |
| *** waleedm has left #tripleo | 12:20 | |
| *** waleedm has joined #tripleo | 12:20 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Add OVS-DPDK parameter as part of roles file https://review.openstack.org/581352 | 12:21 |
| openstackgerrit | Merged openstack/puppet-tripleo master: Prevent triggering firewall actions while configuring HA services https://review.openstack.org/583648 | 12:21 |
| waleedm | Hi guys, is there a way to deploy tripleo-quickstart in non containerized way ? | 12:22 |
| *** eck` is now known as eck`gone | 12:24 | |
| *** edmondsw has joined #tripleo | 12:24 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: upgrade: move removal of services at step 3 (leftovers) https://review.openstack.org/587270 | 12:24 |
| EmilienM | marios: ^ done - thanks for the review! | 12:25 |
| *** eck`gone is now known as eck` | 12:27 | |
| numans | EmilienM, Hi. Is the gate up now. I can do few rechecks right ? | 12:27 |
| *** bfournie has joined #tripleo | 12:28 | |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-quickstart-extras master: [DNM] To test TAGs removal https://review.openstack.org/587402 | 12:29 |
| EmilienM | numans: status says otherwise, I would double check with ruck/rover | 12:29 |
| EmilienM | panda|rover|lunc: ^ (not sure who is ruck this week) | 12:31 |
| EmilienM | rfolco|ruck: ^ | 12:31 |
| rfolco|ruck | EmilienM, numans: gate should be ok, let me know if any issues | 12:32 |
| *** ratailor has quit IRC | 12:32 | |
| *** jcoufal has joined #tripleo | 12:33 | |
| numans | rfolco|ruck, EmilienM thanks | 12:33 |
| marios | ack EmilienM | 12:33 |
| numans | EmilienM, can you please have a look at this patch when you get some time - https://review.openstack.org/#/c/583306/ | 12:33 |
| *** rlandy has joined #tripleo | 12:35 | |
| jistr | thanks for all the reviews gfidente | 12:35 |
| gfidente | jistr I think I should at least review them | 12:36 |
| gfidente | don't make me feel bad | 12:36 |
| numans | EmilienM, is it fine for this patch of yours to be backported to queens https://review.openstack.org/#/c/586531/ ? this is actually needed by - https://review.openstack.org/#/c/586533/ | 12:39 |
| *** ykarel is now known as ykarel|away | 12:40 | |
| EmilienM | numans: yes for both | 12:40 |
| *** agopi has quit IRC | 12:40 | |
| *** leanderthal has joined #tripleo | 12:43 | |
| *** rwsu has joined #tripleo | 12:43 | |
| openstackgerrit | Carlos Camacho proposed openstack/python-tripleoclient master: Create folders if they don't exist when creating output files https://review.openstack.org/587405 | 12:44 |
| *** nenad has joined #tripleo | 12:45 | |
| *** mcornea has joined #tripleo | 12:48 | |
| *** jpena|lunch is now known as jpena | 12:48 | |
| jistr | gfidente: hah no don't feel bad :D We didn't have the framework in place for this type of update, and along with making it i wanted to test it on something. | 12:48 |
| waleedm | Hi guys, is there a way to deploy tripleo-quickstart in non containerized way ? | 12:53 |
| *** pchavva has joined #tripleo | 12:55 | |
| openstackgerrit | Merged openstack/puppet-tripleo stable/pike: Do not target Redis over TLS when using HA Redis profile https://review.openstack.org/586862 | 12:56 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/pike: Do not target Redis over TLS when using HA Redis profile https://review.openstack.org/586863 | 12:56 |
| *** agopi has joined #tripleo | 13:01 | |
| *** cgoncalves has quit IRC | 13:01 | |
| *** cgoncalves has joined #tripleo | 13:02 | |
| *** ykarel|away has quit IRC | 13:02 | |
| *** ansmith has joined #tripleo | 13:07 | |
| fultonj | jistr: reviewing your ceph upgrade changes in on my list too and sorry i haven't gotten to it. | 13:09 |
| jistr | fultonj: no worries and thanks :) | 13:09 |
| *** links has quit IRC | 13:09 | |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-upgrade master: Adjust roles_data also for updates. https://review.openstack.org/587413 | 13:10 |
| waleedm | guys, how to disable autamted_clean in containerized undercloud ? | 13:12 |
| openstackgerrit | Dan Radez proposed openstack/tripleo-heat-templates master: adding vxlan into Neutuon drivers for networking-ansible https://review.openstack.org/586244 | 13:12 |
| sri_ | waleedm, I am not able to ping you | 13:14 |
| waleedm | sri_: ok now I'm here, I couldn't be able to disable the autamted_clean in the container | 13:16 |
| *** mwhahaha changes topic to "Welcome to Rocky | CI Status: GREENish, OVB questionable | https://docs.openstack.org/tripleo-docs/latest/" | 13:16 | |
| waleedm | sri_: and I'm not sure if there is a way to run master underlcoud without containers | 13:16 |
| sri_ | waleedm, I don't know, I've never run the quickstart, maybe if you try to run older version | 13:18 |
| shardy | sri_, waleedm: You can run master undercloud without containers but you'll need https://review.openstack.org/#/c/585344/ | 13:18 |
| shardy | that said, even with that fix I did run into issues with ssl so it may be best to stick with containers now we have no CI coverage of the old model | 13:19 |
| *** Goneri has joined #tripleo | 13:19 | |
| waleedm | shardy: thanks, I'll try it | 13:20 |
| sri_ | shardy, when we run containers is the (nova, Ironic )configuration mounted to host right ? | 13:20 |
| openstackgerrit | Cédric Jeanneret proposed openstack/tripleo-heat-templates master: Allow to remove xinetd service https://review.openstack.org/586057 | 13:22 |
| Tengu | EmilienM: updated - thanks for your comments :) -^^ | 13:22 |
| shardy | sri_: yes the config is mounted from /var/lib/config-data on the host | 13:22 |
| sri_ | shardy, got it, thanks :) | 13:23 |
| *** bnemec has joined #tripleo | 13:24 | |
| *** waleedm has quit IRC | 13:25 | |
| *** links has joined #tripleo | 13:26 | |
| *** lblanchard has joined #tripleo | 13:28 | |
| *** zaneb has quit IRC | 13:30 | |
| *** zaneb has joined #tripleo | 13:30 | |
| openstackgerrit | Merged openstack/puppet-pacemaker master: Remove the pcmk_is_remote fact https://review.openstack.org/583482 | 13:32 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/python-tripleoclient master: Allow custom UC/standalone passwords https://review.openstack.org/587431 | 13:32 |
| *** thrash|g0ne is now known as thrash | 13:33 | |
| *** psahoo has quit IRC | 13:34 | |
| *** myoung has joined #tripleo | 13:37 | |
| *** skramaja has quit IRC | 13:38 | |
| EmilienM | jaosorior: didn't we switch the containerized undercloud to deploy TLS by default? | 13:39 |
| EmilienM | jaosorior: can you remind us what was the default in instack-undercloud in Queens? | 13:39 |
| *** chuck_ has joined #tripleo | 13:41 | |
| *** yrabl has joined #tripleo | 13:42 | |
| *** chuck_ is now known as zul | 13:42 | |
| *** amoralej|lunch is now known as amoralej | 13:43 | |
| openstackgerrit | Brent Eagles proposed openstack/tripleo-heat-templates master: WIP: Migrate neutron aux processes from baremetal to container on upgrades https://review.openstack.org/584981 | 13:43 |
| *** mjturek has joined #tripleo | 13:45 | |
| openstackgerrit | Brent Eagles proposed openstack/tripleo-heat-templates master: WIP: Migrate neutron aux processes from baremetal to container on upgrades https://review.openstack.org/584981 | 13:47 |
| openstackgerrit | Lukas Bezdicka proposed openstack/tripleo-heat-templates stable/queens: Fix GlanceRegistry FFU cleanup https://review.openstack.org/587441 | 13:48 |
| *** mjturek has quit IRC | 13:55 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates stable/queens: nova: add parameters to configure nova::cron::archive_deleted_rows https://review.openstack.org/587446 | 13:57 |
| *** cdearborn has joined #tripleo | 13:58 | |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-heat-templates master: [WIP] Add support for the Mistral notifier services https://review.openstack.org/584853 | 13:59 |
| fultonj | tripleo meeting? | 14:00 |
| mwhahaha | #startmeeting tripleo | 14:00 |
| mwhahaha | #topic agenda | 14:00 |
| openstack | Meeting started Tue Jul 31 14:00:55 2018 UTC and is due to finish in 60 minutes. The chair is mwhahaha. Information about MeetBot at http://wiki.debian.org/MeetBot. | 14:00 |
| mwhahaha | * Review past action items | 14:00 |
| mwhahaha | * One off agenda items | 14:00 |
| mwhahaha | * Squad status | 14:00 |
| mwhahaha | * Bugs & Blueprints | 14:00 |
| mwhahaha | * Projects releases or stable backports | 14:00 |
| mwhahaha | * Specs | 14:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 14:00 |
| mwhahaha | * open discussion | 14:00 |
| mwhahaha | Anyone can use the #link, #action and #info commands, not just the moderatorǃ | 14:00 |
| mwhahaha | Hi everyone! who is around today? | 14:00 |
| *** openstack changes topic to " (Meeting topic: tripleo)" | 14:00 | |
| openstack | The meeting name has been set to 'tripleo' | 14:00 |
| dpeacock | o/ | 14:00 |
| *** openstack changes topic to "agenda (Meeting topic: tripleo)" | 14:01 | |
| Tengu | «o/ | 14:01 |
| jtomasek_ | o/ | 14:01 |
| EmilienM | hello | 14:01 |
| ksambor | o/ | 14:01 |
| rfolco|ruck | o/ | 14:01 |
| myoung | o/ | 14:01 |
| panda|rover|lunc | o/ | 14:01 |
| marios | o/ | 14:01 |
| *** panda|rover|lunc is now known as panda|rover | 14:01 | |
| *** quiquell is now known as quiquell|bbl | 14:01 | |
| florianf | o/ | 14:02 |
| mwhahaha | alright let's move along, not much on the agenda | 14:03 |
| jfrancoa | o/ | 14:03 |
| mwhahaha | #topic review past action items | 14:03 |
| mwhahaha | team to review outstanding UI changes for config download - https://review.openstack.org/#/q/topic:bp/config-download-ui+(status:open+OR+status:merged) - ALMOST DONE | 14:03 |
| mwhahaha | Please take a moment (tripleo-ui folks) to finish the reviews. | 14:03 |
| *** openstack changes topic to "review past action items (Meeting topic: tripleo)" | 14:03 | |
| weshay | 0/ | 14:03 |
| mwhahaha | that's it on the previous action items | 14:03 |
| fultonj | o/ | 14:03 |
| mwhahaha | looks like the config download ui bits are almost done | 14:03 |
| mwhahaha | moving on to the one off items | 14:03 |
| mwhahaha | #topic one off agenda items | 14:04 |
| mwhahaha | #link https://etherpad.openstack.org/p/tripleo-meeting-items | 14:04 |
| *** openstack changes topic to "one off agenda items (Meeting topic: tripleo)" | 14:04 | |
| mwhahaha | (myoung) CI Community Meeting starts immediately upon this meeting closing @ https://bluejeans.com/7050859455. All are welcome, ask/discuss anything. https://etherpad.openstack.org/p/tripleo-ci-squad-meeting | 14:04 |
| jrist | o/ | 14:04 |
| owalsh | o/ | 14:04 |
| beagles | o/ | 14:04 |
| mwhahaha | an other one off items? | 14:04 |
| Tengu | still in need of review on the validation framework spec :) | 14:05 |
| Tengu | #link https://review.openstack.org/583475 | 14:05 |
| Tengu | but I think we're almost there. | 14:05 |
| mwhahaha | sounds good | 14:06 |
| mwhahaha | moving on to squad status bits | 14:06 |
| mwhahaha | #topic Squad status | 14:06 |
| mwhahaha | ci | 14:06 |
| mwhahaha | #link https://etherpad.openstack.org/p/tripleo-ci-squad-meeting | 14:06 |
| mwhahaha | upgrade | 14:06 |
| mwhahaha | #link https://etherpad.openstack.org/p/tripleo-upgrade-squad-status | 14:06 |
| mwhahaha | containers | 14:06 |
| mwhahaha | #link https://etherpad.openstack.org/p/tripleo-containers-squad-status | 14:06 |
| mwhahaha | config-download | 14:06 |
| mwhahaha | #link https://etherpad.openstack.org/p/tripleo-config-download-squad-status | 14:06 |
| *** openstack changes topic to "Squad status (Meeting topic: tripleo)" | 14:06 | |
| mwhahaha | integration | 14:06 |
| mwhahaha | #link https://etherpad.openstack.org/p/tripleo-integration-squad-status | 14:06 |
| mwhahaha | ui/cli | 14:06 |
| mwhahaha | #link https://etherpad.openstack.org/p/tripleo-ui-cli-squad-status | 14:06 |
| mwhahaha | validations | 14:06 |
| mwhahaha | #link https://etherpad.openstack.org/p/tripleo-validations-squad-status | 14:06 |
| mwhahaha | networking | 14:06 |
| mwhahaha | #link https://etherpad.openstack.org/p/tripleo-networking-squad-status | 14:06 |
| mwhahaha | workflows | 14:06 |
| mwhahaha | #link https://etherpad.openstack.org/p/tripleo-workflows-squad-status | 14:06 |
| mwhahaha | security | 14:06 |
| mwhahaha | #link https://etherpad.openstack.org/p/tripleo-security-squad | 14:06 |
| mwhahaha | any specific status related items to raise? | 14:06 |
| EmilienM | for the container squad, most of the work is around stabilization and finishing what we started in the cycle, and heavy testing | 14:07 |
| mwhahaha | looks like a bunch of upgrade reviews are pending to remove the ceph bits | 14:07 |
| fultonj | ? | 14:08 |
| mwhahaha | fultonj: https://review.openstack.org/#/c/585368/ & related | 14:08 |
| mwhahaha | guess there was previously s a bunch of ceph specific items | 14:08 |
| mwhahaha | might want to take a look :D | 14:08 |
| fultonj | mwhahaha: oh that, yes i saw it | 14:08 |
| fultonj | i misunderstood what you meant | 14:09 |
| mwhahaha | fultonj: didn't you get the memo? we're dropping ceph support | 14:09 |
| fultonj | lol | 14:09 |
| fultonj | exactly | 14:09 |
| mwhahaha | no one needs it anyway | 14:09 |
| fultonj | ;) | 14:09 |
| mwhahaha | enough trolling | 14:09 |
| mwhahaha | any other status related items? | 14:09 |
| mwhahaha | sounds like nope, moving on | 14:11 |
| *** bogdando has quit IRC | 14:11 | |
| mwhahaha | topic bugs & blueprints | 14:11 |
| mwhahaha | #link https://launchpad.net/tripleo/+milestone/rocky-rc1 | 14:11 |
| mwhahaha | For Rocky we currently have 42 (-7) blueprints and about 286 open Launchpad bugs. 286 rocky-rc1, 438 stein-1. 105 (+3) open Storyboard bugs. | 14:11 |
| mwhahaha | #link https://storyboard.openstack.org/#!/project_group/76 | 14:11 |
| mwhahaha | Please note we are now in the RC phase so we should be working on fixing bugs and not landing additional features unless they have an FFE. | 14:11 |
| *** agurenko has quit IRC | 14:11 | |
| mwhahaha | please help us make sure CI is stable and review the alerts | 14:11 |
| mwhahaha | we've reverted a bunch of items this week so CI looks better now | 14:12 |
| mwhahaha | let's focus on getting promotions and fixing critical/high issues | 14:12 |
| EmilienM | +1 | 14:12 |
| mwhahaha | any specific bugs or blueprints folks want to talk about? | 14:12 |
| weshay | have a question about health checks | 14:12 |
| weshay | have a question about failing undercloud or overcloud deployments based on the container health check status. Just looking for more information about the criteria to fail the deployment | 14:13 |
| mwhahaha | weshay: so we rolled back the change that failed based on unhealthy containers | 14:13 |
| weshay | and plans for that moving forward | 14:13 |
| mwhahaha | weshay: we'll likely not reimplement that until Stein | 14:13 |
| weshay | yes.. but it's a good idea, just want to inform myself.. ah k | 14:13 |
| weshay | thanks | 14:14 |
| mwhahaha | it does raise larger issues as to why we're getting unhealthy containers in CI | 14:14 |
| mwhahaha | since it destabilized CI, it's not something we should probably add back in during Rocky | 14:14 |
| weshay | everything is unhealthy in ci, we smoke and eat fried food | 14:14 |
| myoung | don't forget excessive drinking | 14:14 |
| mwhahaha | i'd like to add it back during Rocky but IMHO it's to late for such a change | 14:15 |
| mwhahaha | we do check container health elsewhere but just not in upstream CI | 14:15 |
| mwhahaha | so we'll likely end up fixing those issues later anyway | 14:15 |
| mwhahaha | any other bug/blueprint/questions? | 14:16 |
| weshay | we could have health checks reports on each job, that have the option to fail or not to fail the deployment | 14:16 |
| weshay | meh | 14:16 |
| mwhahaha | well unless it fails, no one goes looking | 14:16 |
| mwhahaha | cause it's already available in the docker logs | 14:16 |
| weshay | unless it's collected and reported over time | 14:17 |
| weshay | which I think is what we're looking for in this case.. | 14:17 |
| weshay | anyhoo.. can chat about it another time | 14:17 |
| mwhahaha | the healthy bits are captured in the docker info log so we can probably check logstash | 14:17 |
| mwhahaha | (if it's pulled in) | 14:17 |
| mwhahaha | anyway | 14:17 |
| weshay | +1 | 14:17 |
| mwhahaha | moving on | 14:17 |
| mwhahaha | #topic projects releases or stable backports | 14:17 |
| *** openstack changes topic to "projects releases or stable backports (Meeting topic: tripleo)" | 14:17 | |
| mwhahaha | EmilienM: stable releases this week? | 14:17 |
| EmilienM | yes | 14:18 |
| EmilienM | it's on my list | 14:18 |
| mwhahaha | cool | 14:18 |
| mwhahaha | #topic specs | 14:18 |
| mwhahaha | #link https://review.openstack.org/#/q/project:openstack/tripleo-specs+status:open | 14:18 |
| *** quiquell|bbl is now known as quiquell | 14:18 | |
| *** openstack changes topic to "specs (Meeting topic: tripleo)" | 14:18 | |
| mwhahaha | as previously mentioned, there are specs for review | 14:19 |
| mwhahaha | PTG is in a few weeks so take some time to look over open specs | 14:19 |
| mwhahaha | and propose things | 14:19 |
| mwhahaha | I think the specs repo is prepared for stein but i'll check. there's usually a random bit of prep work required at the start of the next cycle | 14:19 |
| panda|rover | is there a deadline for spec proposals ? | 14:19 |
| mwhahaha | #action mwhahaha to review specs repo preparedness for stein | 14:20 |
| Tengu | mwhahaha: can confirm, I prepared the repo for stein specs. | 14:20 |
| mwhahaha | #undo | 14:20 |
| openstack | Removing item from minutes: #action mwhahaha to review specs repo preparedness for stein | 14:20 |
| mwhahaha | never mind then | 14:20 |
| Tengu | meaning they are listed on the page. | 14:20 |
| Tengu | and there's a place-holder that should be removed once we get the first spec in there :). | 14:20 |
| mwhahaha | panda|rover: there's not a specific deadline but we usually like to make sure they are reviewed/merged prior to M1 if possible | 14:20 |
| panda|rover | mwhahaha: ok thanks. | 14:21 |
| mwhahaha | any other spec related questions? | 14:22 |
| *** holser_ has joined #tripleo | 14:24 | |
| mwhahaha | sounds like nope | 14:24 |
| mwhahaha | #topic open discussion | 14:24 |
| *** openstack changes topic to "open discussion (Meeting topic: tripleo)" | 14:24 | |
| mwhahaha | anything else? | 14:24 |
| openstackgerrit | Marios Andreou proposed openstack-infra/tripleo-ci master: Soften guard for ceph loop device in ci script so we can move it https://review.openstack.org/587369 | 14:24 |
| EmilienM | PTL elections end very shortly | 14:24 |
| EmilienM | if there are more candidates out there, you still have time but do it asap | 14:25 |
| fultonj | jaosorior++ | 14:25 |
| mwhahaha | and by time, i think today is it | 14:25 |
| *** quiquell is now known as quiquell|off | 14:26 | |
| mwhahaha | anything else? | 14:26 |
| mwhahaha | alright thanks everyone | 14:27 |
| mwhahaha | #endmeeting | 14:27 |
| *** openstack changes topic to "Welcome to Rocky | CI Status: GREENish, OVB questionable | https://docs.openstack.org/tripleo-docs/latest/" | 14:27 | |
| openstack | Meeting ended Tue Jul 31 14:27:42 2018 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 14:27 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/tripleo/2018/tripleo.2018-07-31-14.00.html | 14:27 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/tripleo/2018/tripleo.2018-07-31-14.00.txt | 14:27 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/tripleo/2018/tripleo.2018-07-31-14.00.log.html | 14:27 |
| *** dparkes has quit IRC | 14:29 | |
| *** ksambor has quit IRC | 14:32 | |
| *** tzumainn has left #tripleo | 14:34 | |
| *** tzumainn has joined #tripleo | 14:34 | |
| Tengu | see you on Thursday :). | 14:36 |
| *** itlinux_ has quit IRC | 14:38 | |
| *** links has quit IRC | 14:41 | |
| openstackgerrit | Amir Mohamad proposed openstack/tripleo-heat-templates master: This files intorduces OVN as a controller. I've cloned the neutron-sfc.yaml file and added to the parameter_defaults NeutronSfcDriver:'ovn' There could be other param defaults needs to be added and i'm not aware of. https://review.openstack.org/587491 | 14:42 |
| weshay | jaosorior, hrybacki does this error w/ certmonger https://bugs.launchpad.net/tripleo/+bug/1772807 look familiar? | 14:42 |
| openstack | Launchpad bug 1772807 in tripleo "default containerized undercloud install with local CA fails with "Error org.freedesktop.DBus.Error.TimedOut"" [High,Triaged] | 14:42 |
| *** rpioso|afk is now known as rpioso | 14:42 | |
| *** agurenko has joined #tripleo | 14:43 | |
| hrybacki | weshay: need to look into the LP more but same error as https://bugzilla.redhat.com/show_bug.cgi?id=1569122 | 14:48 |
| openstack | bugzilla.redhat.com bug 1569122 in dbus "Undercloud installation fails with "Execution of '/bin/getcert list' returned 1: Error org.freedesktop.DBus.Error.TimedOut"" [High,New] - Assigned to dking | 14:48 |
| *** ccamacho has quit IRC | 14:48 | |
| *** gfidente has quit IRC | 14:49 | |
| *** waleedm has joined #tripleo | 14:49 | |
| *** tcw has quit IRC | 14:50 | |
| openstackgerrit | Sergii Golovatiuk proposed openstack/tripleo-heat-templates master: Add block to step_0 Add block to step_6 for neutron-api.yaml Add block to step_1 for nova-compute.yaml https://review.openstack.org/587235 | 14:50 |
| *** tcw has joined #tripleo | 14:51 | |
| *** Haresh has quit IRC | 14:52 | |
| *** agurenko has quit IRC | 14:53 | |
| *** jfrancoa has quit IRC | 14:55 | |
| openstackgerrit | Sorin Sbarnea proposed openstack/tripleo-quickstart-extras master: make reproducer bash syntax more portable https://review.openstack.org/581012 | 14:57 |
| *** jfrancoa has joined #tripleo | 14:58 | |
| *** links has joined #tripleo | 15:00 | |
| openstackgerrit | Sorin Sbarnea proposed openstack/tripleo-quickstart master: Fix typo in coverage ignore pattern https://review.openstack.org/587495 | 15:02 |
| *** holser_ has quit IRC | 15:03 | |
| openstackgerrit | Sorin Sbarnea proposed openstack/tripleo-quickstart master: fix typos in documentation https://review.openstack.org/585487 | 15:04 |
| *** peereb has quit IRC | 15:06 | |
| *** rrubins has quit IRC | 15:08 | |
| *** holser_ has joined #tripleo | 15:09 | |
| *** nenad has quit IRC | 15:09 | |
| *** pcaruana has quit IRC | 15:10 | |
| *** links has quit IRC | 15:12 | |
| *** rrubins has joined #tripleo | 15:12 | |
| hamzy | When you deploy the overcloud, should some networks be already created for you? "openstack network list" is empty | 15:16 |
| EmilienM | hamzy: no tenant network is created for you afik | 15:20 |
| hamzy | thanks | 15:21 |
| openstackgerrit | Juan Badia Payno proposed openstack/tripleo-heat-templates stable/pike: Fluentd configures rsyslog to send logs properly https://review.openstack.org/587496 | 15:22 |
| openstackgerrit | wangqi proposed openstack/python-tripleoclient master: Use jsonutils of oslo.serialization https://review.openstack.org/587497 | 15:23 |
| openstackgerrit | Raoul Scarazzini proposed openstack/tripleo-quickstart master: Add feature set file to run HA validations https://review.openstack.org/538952 | 15:23 |
| *** udesale has quit IRC | 15:23 | |
| sri_ | dsneddon, Hi i just wanted to let you know about my question yesterday, when that issue has happened I was creating ovs bonds manually, I accidentally put two ovs bonds on one bridge so that it created a L2 loop. it's all clear now. | 15:26 |
| EmilienM | weshay: to test new version of ansible, it's easier to do https://review.rdoproject.org/r/#/c/14960/ FYI | 15:28 |
| EmilienM | weshay: however the package needs to be in CBS (apevec is maintaining this rpm now) | 15:29 |
| jaosorior | weshay: it does look familiar, needed to restart certmonger as far as I remember | 15:30 |
| EmilienM | jaosorior: did you see my questions re: tls/undercloud? | 15:30 |
| jaosorior | EmilienM: just saw it | 15:31 |
| EmilienM | good, take your time, just making sure :D | 15:31 |
| jaosorior | EmilienM: yes, we switched to TLS by default in the undercloud | 15:31 |
| jaosorior | (containerized too) | 15:31 |
| EmilienM | jaosorior: do you have the patch handy? | 15:32 |
| jaosorior | I can search for it | 15:32 |
| jaosorior | there was also a release note somewhere and a mail in the mailing list too | 15:32 |
| EmilienM | the last time I manually deployed I didn't have TLS | 15:32 |
| weshay | EmilienM, hrm.. k thanks | 15:32 |
| jaosorior | EmilienM: how did you deploy? | 15:32 |
| EmilienM | but I probably did something wrong, let me respin | 15:32 |
| EmilienM | jaosorior: by hand | 15:32 |
| EmilienM | openstack undercloud install | 15:32 |
| *** dpointk has joined #tripleo | 15:32 | |
| *** holser_ has quit IRC | 15:33 | |
| jaosorior | EmilienM: Change-Id: Iab02ad0ec1e117447afed17c07e870143017e72b commit hash d25729787463f35eb150468ad8a25f91c0bb3c45 | 15:34 |
| *** zaneb has quit IRC | 15:34 | |
| dpointk | Hello all, I have an issue with minor upgrade of Ocata. During the upgrade, I can see 10-ironic_wsgi.conf is being copied to controller and httpd is trying to come up. this is competing with openstack-ironic-api . I've asked in ironic-openstack and they told me to ask here. Is that normal ? What should be the server that is up in Ocata? the openstack-ironic-api or the httpd with wsgi ? | 15:35 |
| *** zaneb has joined #tripleo | 15:35 | |
| *** sshnaidm is now known as sshnaidm|off | 15:36 | |
| *** gfidente has joined #tripleo | 15:36 | |
| *** gfidente has quit IRC | 15:36 | |
| *** gfidente has joined #tripleo | 15:36 | |
| EmilienM | jaosorior: will look | 15:40 |
| EmilienM | jaosorior: oh yes now I remember | 15:41 |
| EmilienM | thanks | 15:41 |
| *** leanderthal has quit IRC | 15:44 | |
| openstackgerrit | Sam Doran proposed openstack/tripleo-heat-templates master: Use ternary filter for configs https://review.openstack.org/587506 | 15:46 |
| *** links has joined #tripleo | 15:49 | |
| *** links has quit IRC | 15:51 | |
| *** links has joined #tripleo | 15:52 | |
| *** links has quit IRC | 15:55 | |
| *** links has joined #tripleo | 15:55 | |
| *** yamahata has quit IRC | 15:56 | |
| jaosorior | owalsh: are you around? | 15:57 |
| openstackgerrit | Jon Schlueter proposed openstack/python-tripleoclient master: Fixup hard-coded hw_architecture in unittest https://review.openstack.org/587509 | 15:57 |
| jaosorior | or anybody that works in Compute? | 15:58 |
| *** links has quit IRC | 15:58 | |
| jschlueter | https://bugs.launchpad.net/tripleo/+bug/1784662 hard coded assumption about hw_architecture in unittest for tripleoclient | 15:58 |
| openstack | Launchpad bug 1784662 in tripleo "tripleoclient unittests failing if hw_architecture not equal x86_64" [Undecided,New] | 15:58 |
| *** zoli is now known as zoli|gone | 15:58 | |
| *** links has joined #tripleo | 15:58 | |
| mwhahaha | tonyb: -^ | 15:58 |
| *** zoli|gone is now known as zoli | 15:58 | |
| openstackgerrit | Amir Mohamad proposed openstack/tripleo-heat-templates master: This files intorduces OVN as a controller. I've cloned the neutron-sfc.yaml file and added to the parameter_defaults NeutronSfcDriver:'ovn' There could be other param defaults needs to be added and i'm not aware of. https://review.openstack.org/587491 | 15:59 |
| openstackgerrit | RedHat RDO CI proposed openstack/tripleo-heat-templates stable/queens: GATE CHECK for TripleO https://review.openstack.org/567224 | 16:00 |
| openstackgerrit | RedHat RDO CI proposed openstack/tripleo-quickstart-extras master: GATE CHECK for quickstart-extras https://review.openstack.org/560445 | 16:00 |
| openstackgerrit | RedHat RDO CI proposed openstack/tripleo-heat-templates stable/pike: GATE CHECK for TripleO https://review.openstack.org/564285 | 16:00 |
| *** links has quit IRC | 16:01 | |
| *** links has joined #tripleo | 16:01 | |
| *** jfrancoa has quit IRC | 16:07 | |
| *** shardy has quit IRC | 16:08 | |
| *** marios is now known as marios|gonehome | 16:08 | |
| *** marios|gonehome is now known as marios | 16:09 | |
| *** links has quit IRC | 16:10 | |
| *** adrianc_ has quit IRC | 16:13 | |
| weshay | pradk, https://bugs.launchpad.net/tripleo/+bug/1784015 | 16:13 |
| openstack | Launchpad bug 1784015 in tripleo "ovb image build broken due to diskimage_builder.element_dependencies.MissingElementException: Element 'disable-nouveau' not found" [Critical,Fix released] - Assigned to Alex Schultz (alex-schultz) | 16:13 |
| weshay | pradk, so that merged 12hrs ago | 16:14 |
| pradk | weshay, ah gotcha.. ok lemme rebase his patch and try again | 16:14 |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-common master: Make MetricsQdr service use qdrouterd image https://review.openstack.org/578749 | 16:15 |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates master: QDR for metrics collection purposes https://review.openstack.org/572312 | 16:15 |
| weshay | pradk, I think this depends-on is the issue https://review.openstack.org/#/c/578749/ | 16:16 |
| weshay | pradk, ya.. you got it | 16:17 |
| pradk | yeah i rebased both, lets see | 16:17 |
| pradk | thx again weshay .. i'll keep an eye on this | 16:17 |
| weshay | pradk, it takes a village | 16:17 |
| pradk | :) | 16:17 |
| *** ukalifon has left #tripleo | 16:19 | |
| *** links has joined #tripleo | 16:21 | |
| *** mdnadeem_ has quit IRC | 16:21 | |
| *** links has quit IRC | 16:24 | |
| *** myoung is now known as myoung|lunch | 16:24 | |
| *** links has joined #tripleo | 16:25 | |
| *** links has quit IRC | 16:27 | |
| *** links has joined #tripleo | 16:28 | |
| *** lvdombrkr has quit IRC | 16:29 | |
| *** links has quit IRC | 16:30 | |
| *** links has joined #tripleo | 16:31 | |
| openstackgerrit | Sorin Sbarnea proposed openstack/tripleo-quickstart-extras master: avoid Inappropriate ioctl for device with pause module https://review.openstack.org/583965 | 16:31 |
| *** yamahata has joined #tripleo | 16:32 | |
| *** khyr0n has joined #tripleo | 16:32 | |
| *** links has quit IRC | 16:33 | |
| *** sdake has quit IRC | 16:33 | |
| *** sdake has joined #tripleo | 16:33 | |
| *** sdake has quit IRC | 16:33 | |
| *** sdake has joined #tripleo | 16:33 | |
| *** links has joined #tripleo | 16:33 | |
| openstackgerrit | Sorin Sbarnea proposed openstack/tripleo-quickstart master: bump ansible from 2.5.5 to 2.5.7 https://review.openstack.org/587371 | 16:33 |
| *** jpich has quit IRC | 16:35 | |
| *** jtcressy has joined #tripleo | 16:35 | |
| *** itlinux has joined #tripleo | 16:35 | |
| *** links has quit IRC | 16:35 | |
| openstackgerrit | Sorin Sbarnea proposed openstack/tripleo-quickstart master: test new version of ansible https://review.openstack.org/584432 | 16:36 |
| *** links has joined #tripleo | 16:36 | |
| jtcressy | mwhahaha: So I just discovered split stack (https://docs.openstack.org/tripleo-docs/latest/install/advanced_deployment/split_stack.html) and i'm freaking out about how cool this is. Can I use this to split ceph into its own stack? | 16:36 |
| jtcressy | maybe also a question for fultonj | 16:36 |
| mwhahaha | unlikely | 16:37 |
| mwhahaha | it'd probably better to just use external ceph | 16:37 |
| mwhahaha | at that point | 16:37 |
| *** salmankhan has quit IRC | 16:37 | |
| jtcressy | yeah. Thats what me and anderbubble have been thinking. I've been unable to update my stack because the ceph-ansible playbook kept getting hung up on a trivial error. | 16:38 |
| fultonj | jtcressy: are you cleaning your disks before the deployment? | 16:38 |
| *** links has quit IRC | 16:38 | |
| fultonj | https://docs.openstack.org/ironic/latest/admin/cleaning.html#starting-manual-cleaning-via-openstack-baremetal-cli | 16:39 |
| fultonj | openstack baremetal node clean <node> --clean-steps '[{"interface": "deploy", "step": "erase_devices_metadata"}]' | 16:39 |
| *** links has joined #tripleo | 16:39 | |
| fultonj | if you deploy ceph on a new node, you clean the disks first ^ | 16:39 |
| jtcressy | ceph-ansible likes to re-create the OSD partition tables in-place during a stack update (it's not erasing anything, so no data loss) but for some reason one of my nodes keeps erroring out with "unable to inform kernel of changes on disk. you may need to reboot to apply changes" or whatever, and there's no problem with it. It should work if you ignore the error, but since the command returns non-zero, the ansible playbook fails and cascades | 16:39 |
| jtcressy | the failure. | 16:39 |
| jtcressy | for good measure i "ceph -s" the cluster after a failed stack update and everything was fine. | 16:40 |
| fultonj | ceph-ansible needs to create parititons on your disk adn expects a clean disk going in | 16:41 |
| fultonj | clean your disk before deployment | 16:41 |
| jtcressy | So, by the way, yes I am cleaning my nodes before every __new__ deployment. But I should not have to clean my nodes during a stack update. | 16:41 |
| jtcressy | i.e. it's deployed but i'm scaling up a compute node | 16:41 |
| *** links has quit IRC | 16:41 | |
| jtcressy | which is one of the things I tried yesterday. Scaled up with "openstack overcloud deploy" + my templates and it failed on ceph-ansible with what i've described above. | 16:42 |
| *** links has joined #tripleo | 16:42 | |
| fultonj | you shouldn't have to clean existing ceph nodes during a compute node scale up | 16:42 |
| jtcressy | exactly | 16:42 |
| jtcressy | I'm going to deem it something wrong with the node maybe? but I can't for the life of me imagine why it keeps happening on that node. | 16:43 |
| *** sshnaidm|off has quit IRC | 16:43 | |
| mwhahaha | jtcressy: you did try rebooting the node right | 16:43 |
| fultonj | which node did it try to change partitions on? | 16:43 |
| jtcressy | well i can't very well reboot the node in the middle of a stack update, right? | 16:43 |
| jtcressy | Here's the full output from the last time I ran it: | 16:43 |
| jtcressy | Error: Partition(s) 1, 2 on /dev/sdc have been written, but we have been unable to inform the kernel of the change, probably because it/they are in use. As a result, the old partition(s) will remain in use. You should reboot now before making further changes. | 16:43 |
| fultonj | jtcressy: are you doing auto discovery of OSDs or passing it a list of OSDs as aprtiiton | 16:43 |
| jtcressy | passing it a list of OSD's | 16:44 |
| fultonj | is sdc an existing OSD on that node | 16:44 |
| jtcressy | but i'm giving it the raw device, e.g. /dev/sdb, /dev/sdc. And yes, /dev/sdc is an existing OSD on that node. | 16:44 |
| mwhahaha | jtcressy: if the update isn't running, you could try rebooting it | 16:44 |
| *** ramishra has quit IRC | 16:44 | |
| mwhahaha | jtcressy: cause it kinda sounds like you're stuck on a kernel bug | 16:44 |
| *** links has quit IRC | 16:44 | |
| *** links has joined #tripleo | 16:45 | |
| jtcressy | I had tried the stack update multiple times before, failing on the same error on the same node, so i guess the kernel bug is triggered by something with the node. I've been looking into the node itself and cant find anything wrong, so it's just haunted LOL. OH! and "clean failed" on the node every time I try to clean it. I have to manually boot the instance and SSH in to do "parted -s /dev/sdc mklabel gpt". | 16:46 |
| jtcressy | I must emphasize that it's only this one node. | 16:47 |
| *** yprokule has quit IRC | 16:47 | |
| mwhahaha | yea sounds like an issue with that hardware | 16:47 |
| mwhahaha | firmware up to date, etc? | 16:47 |
| jtcressy | system firmware as up to date as it can be. not sure about individual drives though, and that's a pandoras box. | 16:48 |
| *** links has quit IRC | 16:48 | |
| jtcressy | these are dell R710's, so not the latest stuff. | 16:48 |
| *** amoralej is now known as amoralej|off | 16:48 | |
| jtcressy | prone to problems i guess. | 16:48 |
| openstackgerrit | Raoul Scarazzini proposed openstack/tripleo-quickstart-extras master: Integrate Validate-HA work into OVB jobs https://review.openstack.org/537669 | 16:48 |
| *** links has joined #tripleo | 16:48 | |
| *** panda|rover is now known as panda|rover|off | 16:49 | |
| *** links has quit IRC | 16:51 | |
| *** links has joined #tripleo | 16:51 | |
| fultonj | jtcressy: if that node is missing that disk you can black list just that disk | 16:52 |
| fultonj | https://docs.openstack.org/tripleo-docs/latest/install/advanced_deployment/node_specific_hieradata.html | 16:52 |
| fultonj | so that node gets passed a list of all the disks that would be OSDs except the one that's missing | 16:53 |
| jtcressy | the disk is present and functional as long as I manually wipe it with "parted -s /dev/sdc mklabel gpt" so i'm really confused as to what its problem could be. | 16:53 |
| fultonj | ironic should clean it too | 16:53 |
| fultonj | ahh but you don't want to clean the full box | 16:53 |
| jtcressy | ironic fails to clean it, likely with the same error. | 16:54 |
| *** links has quit IRC | 16:54 | |
| jtcressy | but I cant tell what error it gets when ironic fails to clean it because it shuts off for some reason. | 16:54 |
| fultonj | so you can clean it manually and try again OR just exclude it from the deployment | 16:54 |
| fultonj | via node spec override as per that doc ^ | 16:54 |
| jtcressy | I thought ironic keeps failed cleaning nodes powered on for diagnostic. | 16:54 |
| *** links has joined #tripleo | 16:54 | |
| fultonj | clean failed --> doesn't go into available, so yes | 16:55 |
| jtcressy | yet here we are, the node status is "powered off" and provisioning state is "clean failed" with maintenance=True | 16:56 |
| jtcressy | it never keeps it powered on. | 16:56 |
| *** links has quit IRC | 16:56 | |
| *** links has joined #tripleo | 16:57 | |
| jtcressy | one way around this might be to ssh as heat-admin before deleting the stack and wiping /dev/sdc and ONLY sdc. after which, delete the stack and re-deploy. | 16:57 |
| fultonj | or exclude sdc as it has odd behavior | 16:58 |
| fultonj | as far as ironic and this node is concerned | 16:58 |
| fultonj | which is problematic for your deployment | 16:58 |
| fultonj | e.g. i'm doing a deploy right now | 16:58 |
| jtcressy | So in "NodeDataLookup" where I put the uuid, should I include all functional OSD's and omit /dev/sdc? | 16:58 |
| fultonj | yes | 16:58 |
| *** adrianc_ has joined #tripleo | 16:58 | |
| fultonj | 30 nodes and one of them has a bad disk | 16:58 |
| fultonj | so i just exclude it | 16:58 |
| openstackgerrit | Raoul Scarazzini proposed openstack/tripleo-quickstart-extras master: Integrate Validate-HA work into OVB jobs https://review.openstack.org/537669 | 16:59 |
| fultonj | http://paste.openstack.org/show/726959/ | 16:59 |
| *** links has quit IRC | 17:00 | |
| *** derekh has quit IRC | 17:00 | |
| fultonj | jtcressy: you can get the machine specific id with `dmidecode -s system-uuid` | 17:00 |
| fultonj | it's not the same as the UUID you get from doing 'openstack baremetal node list' | 17:00 |
| jtcressy | oooh ok. | 17:00 |
| fultonj | it's also in the introspection data as per the doc | 17:01 |
| jtcressy | there was also this, yeah openstack baremetal introspection data save NODE-ID | jq .extra.system.product.uuid | 17:01 |
| fultonj | exactly | 17:01 |
| *** links has joined #tripleo | 17:01 | |
| fultonj | basically list all the disks of your many many nodes for devices in hopes that everything works out | 17:02 |
| fultonj | then bad snow flakes pop out of the mis and end up the overrides yaml | 17:02 |
| fultonj | because as the nodes go up the probability of bad disks increases | 17:02 |
| jtcressy | This is what I got: {"4C4C4544-004D-5010-8046-CAC04F424D31": {"devices": ["/dev/sdb", "/dev/sdd", "/dev/sde", "/dev/sdf", "/dev/sdg"]}} | 17:02 |
| jtcressy | i have/had 6 OSD's on that node. | 17:03 |
| fultonj | looks reasonable | 17:03 |
| *** tesseract has quit IRC | 17:03 | |
| *** links has quit IRC | 17:03 | |
| *** links has joined #tripleo | 17:04 | |
| *** mhenkel__ has joined #tripleo | 17:04 | |
| itlinux | hello guys, where do I find the default policy.json now since the file is empty. I remember that something has changed thanks | 17:04 |
| *** mhenkel_ has quit IRC | 17:05 | |
| *** jpena is now known as jpena|off | 17:05 | |
| *** slaweq has quit IRC | 17:05 | |
| mwhahaha | itlinux: it's in code now | 17:06 |
| *** links has quit IRC | 17:06 | |
| * fultonj needs to step out for now | 17:06 | |
| *** links has joined #tripleo | 17:07 | |
| *** mhenkel__ has quit IRC | 17:09 | |
| *** links has quit IRC | 17:09 | |
| *** links has joined #tripleo | 17:10 | |
| *** ooolpbot has joined #tripleo | 17:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784017 | 17:10 |
| *** ooolpbot has quit IRC | 17:10 | |
| openstack | Launchpad bug 1784017 in tripleo "Build of instance was re-scheduled: invalid argument: could not find capabilities for domaintype=kvm" [Critical,Triaged] | 17:10 |
| *** links has quit IRC | 17:12 | |
| *** links has joined #tripleo | 17:12 | |
| itlinux | mwhahaha: what's the best way to extract that? my understanding is that you can still use the policy.json since i need to create a new role. Thanks | 17:13 |
| * mwhahaha has no idea | 17:13 | |
| weshay | rasca, <jaosorior> weshay: it does look familiar, needed to restart certmonger as far as I remember | 17:14 |
| weshay | rasca, so we need to figure out .. after you get off the quickstart.sh call and on to the proper ovb call | 17:14 |
| weshay | rasca, if your image was old | 17:14 |
| rasca | weshay, after discussing with rlandy we changed a little bit of approach | 17:14 |
| rasca | weshay, I'm testing it right now, let me se if this works | 17:15 |
| *** links has quit IRC | 17:15 | |
| *** gbarros has joined #tripleo | 17:15 | |
| *** links has joined #tripleo | 17:15 | |
| *** bfournie has quit IRC | 17:17 | |
| *** psachin` has quit IRC | 17:17 | |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Adds reproducer check+exit+warning dependencies - virtualenv+others https://review.openstack.org/578081 | 17:18 |
| openstackgerrit | Merged openstack/tripleo-ui master: Update way to determine deployment status in plans list https://review.openstack.org/583219 | 17:18 |
| openstackgerrit | Merged openstack/tripleo-common master: Add default for ceph_ansible_playbook in update/upgrade prepare workflow https://review.openstack.org/585367 | 17:18 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: OVN: Add exec puppet tag to ovn-controller service file https://review.openstack.org/583306 | 17:18 |
| openstackgerrit | Merged openstack/puppet-tripleo master: Enable HAProxy mode http for Swift and Ceph RGW https://review.openstack.org/582991 | 17:18 |
| openstackgerrit | Merged openstack/tripleo-upgrade stable/queens: Workload flavor customization. https://review.openstack.org/587313 | 17:18 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/queens: Copy-in libvirt certs via kolla extended/start https://review.openstack.org/586977 | 17:18 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/queens: Fluentd configures rsyslog to send logs properly https://review.openstack.org/583478 | 17:18 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/queens: Add FFU upgrade steps to Sahara services https://review.openstack.org/582472 | 17:18 |
| *** links has quit IRC | 17:18 | |
| openstackgerrit | Merged openstack/tripleo-upgrade stable/queens: Make service check output more verbose. https://review.openstack.org/580685 | 17:18 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/queens: Adding HeatEngineVolumes and HeatEngineOptEnvVars support https://review.openstack.org/577737 | 17:18 |
| *** links has joined #tripleo | 17:18 | |
| weshay | WOOOT | 17:19 |
| *** links has quit IRC | 17:21 | |
| *** links has joined #tripleo | 17:21 | |
| rlandy | weshay: wrt rasca's comments above ... we have an issue with the way we are calling the list of playbooks | 17:22 |
| rlandy | weshay: as of now - all ovb calls the same series of playbooks | 17:22 |
| rlandy | if we modify the ha-validate playbook to only run with a set var - as with browbeat, | 17:23 |
| rlandy | we will mess with the way the playbook is being called via infra red etc. | 17:23 |
| rlandy | there are ways around this problem | 17:23 |
| rlandy | calling the full playbook (like fs001) or adding a new section like updates | 17:24 |
| *** links has quit IRC | 17:24 | |
| rlandy | any preferences? | 17:24 |
| *** links has joined #tripleo | 17:24 | |
| weshay | rlandy, can you show me? | 17:25 |
| rlandy | weshay: https://review.openstack.org/#/c/537669/14/playbooks/baremetal-full-deploy.yml - this is how was can control which playbook runs if we kick off the full playbook set | 17:25 |
| rlandy | https://github.com/openstack-infra/tripleo-ci/blob/master/toci_gate_test.sh#L146 | 17:26 |
| rlandy | is where we would have to add the validate-ha playbook | 17:26 |
| rlandy | but it would kick with each run on ovb | 17:26 |
| weshay | rlandy, did you mean to point to ovb? | 17:26 |
| rasca | rdocloud: 504 Gateway Time-out | 17:26 |
| rasca | I give up for today | 17:27 |
| *** links has quit IRC | 17:27 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/queens: Adds docker SFC and cleans up ODL envs https://review.openstack.org/575122 | 17:27 |
| openstackgerrit | Merged openstack/tripleo-quickstart master: Fix --requirements argument on quickstart.sh https://review.openstack.org/587384 | 17:27 |
| openstackgerrit | Merged openstack/tripleo-ui master: List deployment failures on deployment failure https://review.openstack.org/578712 | 17:27 |
| rlandy | https://github.com/openstack-infra/tripleo-ci/blob/master/toci_gate_test.sh#L132 | 17:27 |
| rlandy | weshay: ^^ sorry | 17:27 |
| EmilienM | jaosorior: I confirm tls is default, just tested. Thx | 17:27 |
| *** links has joined #tripleo | 17:27 | |
| rlandy | if we added fs 041 to $QUICKSTART_SH_JOBS | 17:27 |
| rlandy | we could kick the full playbook | 17:27 |
| rlandy | I had not considered validate-ha outside of our CI platforms | 17:28 |
| rlandy | OR ... | 17:28 |
| rlandy | we could do what upgrades does | 17:28 |
| rlandy | and create a new case | 17:28 |
| rlandy | OR .. | 17:28 |
| rlandy | we override the playbooks that run | 17:28 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-heat-templates master: adding vxlan into Neutuon drivers for networking-ansible https://review.openstack.org/586244 | 17:28 |
| *** links has quit IRC | 17:30 | |
| weshay | rlandy, so this isn't quite lining up w/ my understanding | 17:30 |
| *** links has joined #tripleo | 17:30 | |
| rlandy | weshay: how so? | 17:30 |
| weshay | rlandy, does rasca use those playbooks w/ infrared? | 17:30 |
| rlandy | weshay: I don;t know | 17:31 |
| rlandy | never touched infrared | 17:31 |
| rlandy | bit iiuc, that playbook must be run in its form | 17:31 |
| *** colonwq has quit IRC | 17:31 | |
| rlandy | weshay: but I guess now I understand the upgrades decisions a bit better | 17:31 |
| weshay | rlandy, oh.. ovb doesn't have the tag section | 17:32 |
| *** bfournie has joined #tripleo | 17:32 | |
| weshay | rlandy, my thought is this.. | 17:33 |
| jtcressy | mwhahaha: is there a document that explains how to enable extra services like designate, LBaaS, murano, (and that one container orchestration thing which name escapes me) | 17:33 |
| *** colonwq has joined #tripleo | 17:33 | |
| *** links has quit IRC | 17:33 | |
| weshay | rlandy, a lot of this going to change quite soon | 17:33 |
| weshay | rlandy, if you can get his workflow to run w/ https://github.com/openstack-infra/tripleo-ci/blob/master/toci_gate_test.sh#L130 | 17:33 |
| weshay | by adding it to QUICKSTART_SH_JOBS, thats fine | 17:33 |
| mwhahaha | jtcressy: designate is a only supported in Rocky, lbaas (octavia) should be ok in queens, no murno, no magnum | 17:33 |
| weshay | rlandy, wdyt | 17:33 |
| rlandy | weshay: ack - good enough for now | 17:33 |
| *** links has joined #tripleo | 17:33 | |
| weshay | rlandy, ya.. and then we can torch this bs | 17:34 |
| jtcressy | mwhahaha: whoa, that's kinda news. I thought designate was around for longer than rocky | 17:34 |
| mwhahaha | jtcressy: it is, but the support in tripleo is not there | 17:34 |
| rlandy | weshay: I thought we should be able to override playbook per jobs | 17:34 |
| * rlandy hopes | 17:34 | |
| mwhahaha | jtcressy: bnemec is working on getting it properly supported as part of Rocky | 17:34 |
| *** myoung|lunch is now known as myoung | 17:34 | |
| jtcressy | mwhahaha: ah, I see. Btw, when does tripleo rocky go GA/release? (or when does it plan to) | 17:35 |
| mwhahaha | ga is end of next month so like 3-4 weeks | 17:35 |
| rlandy | weshay: we can discuss more after the zuul v3 discussion ... | 17:35 |
| jtcressy | sweet. I'll probably start experimenting with it during this school year. | 17:35 |
| EmilienM | panda|rover|off: hey, I thought we fixed https://bugs.launchpad.net/tripleo/+bug/1784670 | 17:35 |
| openstack | Launchpad bug 1784670 in tripleo "CI jobs are not making docker info logs (with healthchecks) available in logstash" [Critical,In progress] - Assigned to Gabriele Cerami (gcerami) | 17:35 |
| bnemec | Speaking of which, I should see if my patches have all passed ci. | 17:36 |
| *** links has quit IRC | 17:36 | |
| *** links has joined #tripleo | 17:37 | |
| EmilienM | panda|rover|off: oh... in logstash | 17:37 |
| EmilienM | panda|rover|off: I'm not sure it's critical | 17:37 |
| EmilienM | it's a good thing to have tho | 17:37 |
| *** links has quit IRC | 17:39 | |
| weshay | EmilienM, aye.. to mwhahaha's point in the mtg | 17:39 |
| *** links has joined #tripleo | 17:40 | |
| *** links has quit IRC | 17:42 | |
| *** links has joined #tripleo | 17:42 | |
| *** gbarros has quit IRC | 17:43 | |
| *** dpointk has quit IRC | 17:43 | |
| *** gbarros has joined #tripleo | 17:45 | |
| *** links has quit IRC | 17:45 | |
| *** links has joined #tripleo | 17:46 | |
| *** florianf has quit IRC | 17:46 | |
| *** ade_lee has quit IRC | 17:48 | |
| *** ade_lee has joined #tripleo | 17:48 | |
| openstackgerrit | Rafael Folco proposed openstack-infra/tripleo-ci master: DNM: Run 3nodes-multinode job only https://review.openstack.org/587567 | 17:49 |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates master: Add support to set cors config in gnocchi templates https://review.openstack.org/576220 | 17:49 |
| *** gbarros has quit IRC | 17:50 | |
| *** links has quit IRC | 17:52 | |
| *** ade_lee has quit IRC | 17:53 | |
| panda|rover|off | EmilienM: it's the thing I'm working on. It bypassed other bugs so I put it in critical. feel free to downgrade it. | 17:53 |
| *** waleedm has quit IRC | 17:54 | |
| *** mhenkel_ has joined #tripleo | 17:54 | |
| *** gbarros has joined #tripleo | 17:54 | |
| *** slaweq has joined #tripleo | 17:55 | |
| *** agurenko has joined #tripleo | 17:56 | |
| *** agurenko has quit IRC | 17:56 | |
| *** gbarros has quit IRC | 17:57 | |
| *** mhenkel_ has quit IRC | 17:58 | |
| openstackgerrit | Rafael Folco proposed openstack/tripleo-quickstart-extras master: WIP: Fix libvirt_type kvm for overcloud deploy https://review.openstack.org/587572 | 18:00 |
| *** florianf has joined #tripleo | 18:01 | |
| weshay | rfolco|ruck, there were some other reviews around kvm vs qemu | 18:05 |
| weshay | probably good to reference them | 18:05 |
| rfolco|ruck | hmm really? I missed that | 18:05 |
| rfolco|ruck | :-/ | 18:05 |
| rfolco|ruck | searching | 18:05 |
| weshay | https://review.openstack.org/#/c/584380/ | 18:06 |
| weshay | rfolco|ruck, ^ | 18:06 |
| *** shreshtha has quit IRC | 18:07 | |
| rfolco|ruck | weshay, thanks | 18:08 |
| weshay | rfolco|ruck, http://logs.openstack.org/28/585528/8/check/tripleo-ci-centos-7-3nodes-multinode/cd65ccf/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-07-27_08_32_24 | 18:09 |
| weshay | openstack overcloud deploy --templates /usr/share/openstack-tripleo-heat-templates --libvirt-type kvm | 18:09 |
| rfolco|ruck | weshay, this tells the nova placement to find a compute that has kvm virt type | 18:10 |
| *** ooolpbot has joined #tripleo | 18:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784017 | 18:10 |
| openstack | Launchpad bug 1784017 in tripleo "Build of instance was re-scheduled: invalid argument: could not find capabilities for domaintype=kvm" [Critical,In progress] - Assigned to Rafael Folco (rafaelfolco) | 18:10 |
| *** ooolpbot has quit IRC | 18:10 | |
| weshay | rfolco|ruck, can you show me? | 18:10 |
| weshay | rfolco|ruck, is that file used for tempest? | 18:10 |
| *** ade_lee has joined #tripleo | 18:12 | |
| rfolco|ruck | hmmmmmmm wait a min | 18:12 |
| *** holser_ has joined #tripleo | 18:12 | |
| rfolco|ruck | this rings a bell, dejavou from ppc times | 18:13 |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: reproducer: cherry pick zuul changes https://review.openstack.org/582963 | 18:14 |
| weshay | ssbarnea, ^ | 18:16 |
| weshay | wow.. things are actually merging in the same day | 18:16 |
| ssbarnea | wow | 18:17 |
| weshay | right | 18:17 |
| mwhahaha | pffft the gate seems to be ok since we reverted all the bits causing problems :D | 18:21 |
| mwhahaha | considering we're just down to one alert from like 8 :D | 18:21 |
| *** itlinux has quit IRC | 18:24 | |
| *** salmankhan has joined #tripleo | 18:29 | |
| *** gbarros has joined #tripleo | 18:29 | |
| *** holser_ has quit IRC | 18:33 | |
| *** srini has joined #tripleo | 18:33 | |
| *** salmankhan has quit IRC | 18:33 | |
| *** srini has quit IRC | 18:34 | |
| *** gbarros has quit IRC | 18:36 | |
| *** jaganathan has quit IRC | 18:36 | |
| *** gbarros has joined #tripleo | 18:40 | |
| *** yamahata__ has joined #tripleo | 18:41 | |
| *** gbarros has quit IRC | 18:42 | |
| *** isq_ has quit IRC | 18:42 | |
| *** mhenkel_ has joined #tripleo | 18:43 | |
| *** gbarros has joined #tripleo | 18:43 | |
| ssbarnea | ar treb sa ma duc pt maine | 18:43 |
| ssbarnea | ca nu o sa am ce manca | 18:43 |
| ssbarnea | plec acum | 18:43 |
| *** gbarros has quit IRC | 18:45 | |
| *** gbarros has joined #tripleo | 18:55 | |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui master: Convert NewPlan form to redux-form https://review.openstack.org/587592 | 18:56 |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui master: Update plans state management https://review.openstack.org/587593 | 18:56 |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui master: Convert EditPlan form to redux-form https://review.openstack.org/587594 | 18:56 |
| *** itlinux has joined #tripleo | 19:02 | |
| openstackgerrit | Ronelle Landy proposed openstack-infra/tripleo-ci master: WIP: DNM: Enable upstream testing of tripleo-ha-utils https://review.openstack.org/587603 | 19:03 |
| *** mhenkel__ has joined #tripleo | 19:09 | |
| *** mhenkel_ has quit IRC | 19:09 | |
| *** ooolpbot has joined #tripleo | 19:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784017 | 19:10 |
| *** ooolpbot has quit IRC | 19:10 | |
| openstack | Launchpad bug 1784017 in tripleo "Build of instance was re-scheduled: invalid argument: could not find capabilities for domaintype=kvm" [Critical,In progress] - Assigned to Rafael Folco (rafaelfolco) | 19:10 |
| *** adrianc_ has quit IRC | 19:10 | |
| *** gbarros has quit IRC | 19:11 | |
| *** morazi has quit IRC | 19:13 | |
| *** gbarros has joined #tripleo | 19:14 | |
| *** gbarros has quit IRC | 19:17 | |
| *** gbarros has joined #tripleo | 19:23 | |
| trozet | dsneddon: ping? | 19:40 |
| dsneddon | trozet, Hello | 19:41 |
| trozet | dsneddon: hey man | 19:41 |
| trozet | dsneddon: is there any way to make all of the FQDN settings in OOO just IP instead? Like https://github.com/openstack/tripleo-heat-templates/blob/stable/queens/puppet/services/neutron-metadata.yaml#L113 | 19:41 |
| *** slagle has quit IRC | 19:45 | |
| *** mjturek has joined #tripleo | 19:46 | |
| *** slaweq has quit IRC | 19:47 | |
| *** jcoufal has quit IRC | 19:52 | |
| dsneddon | trozet, I don't think so, I think this is some kind of Nova weirdness. All I know is that I've always had to use FQDN and then rely on /etc/hosts to contain the list of FQDNs. | 19:53 |
| dsneddon | trozet, Sorry, meant Neutron weirdness, but you get the point. | 19:54 |
| trozet | dsneddon: yeah the problem is, even if the entry is in /etc/hosts, the python dhcp metadata agent code still tries to do dns lookup, and i dont have any external network access | 19:54 |
| trozet | dsneddon: https://bugzilla.redhat.com/show_bug.cgi?id=1504279 | 19:54 |
| openstack | bugzilla.redhat.com bug 1504279 in openstack-neutron "Metadata service takes a long time to response" [High,Closed: notabug] - Assigned to dalvarez | 19:54 |
| dsneddon | trozet, Ugh, stupid Python. | 19:54 |
| *** slaweq has joined #tripleo | 19:55 | |
| dsneddon | trozet, You might get better results if you used 127.0.0.1 as the only entry in /etc/resolv.conf | 19:55 |
| trozet | dsneddon: it takes 30 seconds to get a metadata curl response | 19:55 |
| dsneddon | trozet, That way at least it will fail fast. | 19:55 |
| dsneddon | trozet, But why not just use the undercloud as a DNS server? Or does your undercloud not have external net access either? | 19:55 |
| trozet | dsneddon: yeah i tried that at first, but it was still taking 30 seconds for some reason. Then I tried to change hosts.conf to "order hosts", but that ditn work either. At that point I thought maybe i need to restart linux networking, now lots of things are broken :) | 19:56 |
| trozet | dsneddon: well yeah normally we use undercloud. But in this case there is no undercloud anymore, thse are snapshots of an already deployed overcloud that we bring up in another environment | 19:56 |
| dsneddon | trozet, I see the problem. | 19:56 |
| trozet | dsneddon: so we shut the vms down and save them then bring them back up later, and they have no external network access...yeah | 19:57 |
| trozet | dsneddon: but you are right all the entries for the FQDNs are in /etc/hosts, so its kind of dumb that it is trying to do dns lookup | 19:57 |
| trozet | i dont get why it is doing that | 19:57 |
| dsneddon | trozet, Don't you need to modify /etc/nsswitch.conf to disable DNS and use only /etc/hosts? | 19:57 |
| dsneddon | trozet, Change the hosts: setting there | 19:58 |
| trozet | dsneddon: yeah i changed that to be hosts: files myhostname | 19:58 |
| trozet | dsneddon: do you need to restart a systemd service after that? | 19:58 |
| dsneddon | trozet, And that is probably why you tried to restart networking... | 19:58 |
| trozet | dsneddon: haha yeah | 19:58 |
| dsneddon | trozet, I think you need to restart nss-lookup and nss-user-lookup | 19:59 |
| trozet | dsneddon: ah | 19:59 |
| dsneddon | trozet, Or maybe not, I don't see those services on my Fedora 28 box. | 20:00 |
| trozet | dsneddon: yeah not seeing them on my centos 7 | 20:00 |
| dsneddon | trozet, There have been some changes to this subsystem that I'm not up to speed on. | 20:00 |
| *** tvignaud has quit IRC | 20:00 | |
| trozet | dsneddon: so I'm thinking for now, since other services seem to work and it is jus this metadata...I may just change the value to be an IP in the conf file post deployment before we save the VMs | 20:01 |
| dsneddon | trozet, Give it a try. | 20:01 |
| dsneddon | trozet, I'd like to know if that works, since I've wondered why we always use names | 20:01 |
| trozet | dsneddon: oh I can say it does work. I manually changed it to IP and got response right away | 20:02 |
| dsneddon | trozet, Then I don't know why we are using names, possibly a carryover from when that was required. | 20:02 |
| trozet | dsneddon: maybe the TLS stuff? | 20:02 |
| *** akrivoka has quit IRC | 20:04 | |
| weshay | mwhahaha, EmilienM did something recently change w/ overcloud validation prior to deployments? | 20:05 |
| *** yrabl has quit IRC | 20:05 | |
| mwhahaha | weshay: maybe | 20:05 |
| mwhahaha | weshay: whay | 20:05 |
| EmilienM | who knows | 20:05 |
| weshay | resources.InternalApiNetwork: Property error: resources.InternalApiSubnet.properties.allocation_pools[0].start: "172.17.0.10" does not validate ip_addr (constraint not found) | 20:05 |
| weshay | across several jobs | 20:05 |
| weshay | https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload/80c01cd/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-07-31_19_01_07 | 20:05 |
| weshay | https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master/fa46166/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-07-31_19_02_56 | 20:05 |
| * weshay pokes | 20:06 | |
| mwhahaha | check with hjensas, dsneddon and bfournie | 20:06 |
| mwhahaha | sounds like network config related stuff | 20:06 |
| weshay | tis | 20:06 |
| weshay | k | 20:06 |
| weshay | thanks | 20:06 |
| mwhahaha | that's not normal validations but rather network config things | 20:06 |
| *** jtcressy has quit IRC | 20:07 | |
| *** v1a4 has joined #tripleo | 20:07 | |
| *** v1a4 has quit IRC | 20:07 | |
| *** ooolpbot has joined #tripleo | 20:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784017 | 20:10 |
| *** ooolpbot has quit IRC | 20:10 | |
| openstack | Launchpad bug 1784017 in tripleo "Build of instance was re-scheduled: invalid argument: could not find capabilities for domaintype=kvm" [Critical,In progress] - Assigned to Rafael Folco (rafaelfolco) | 20:10 |
| dsneddon | weshay, mwhahaha: I'm taking a look, but not sure what is causing that yet | 20:11 |
| *** jtcressy has joined #tripleo | 20:12 | |
| * weshay doesn't think it's anything here https://github.com/openstack/tripleo-heat-templates/commits/master/ci/environments/network | 20:12 | |
| weshay | dsneddon, https://github.com/openstack/tripleo-heat-templates/commit/dfc09b6ff1141d05d8c0c11ff2f5ac5e4dddc87f#diff-4f66a19e2e67818f73251bbe5c1b942f | 20:14 |
| weshay | ? | 20:14 |
| weshay | the default being removed maybe? | 20:14 |
| dsneddon | weshay, No, that wouldn't have this effect. | 20:14 |
| dsneddon | weshay, There were a few other small changes recently | 20:15 |
| weshay | rfolco|ruck, /me going to write up a bug | 20:16 |
| rfolco|ruck | weshay, you want me to do so? | 20:17 |
| weshay | rfolco|ruck, I got it | 20:17 |
| rfolco|ruck | ok thanks | 20:17 |
| *** wolverineav has quit IRC | 20:17 | |
| *** wolverineav has joined #tripleo | 20:18 | |
| weshay | odd.. just the external on this one https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/0ec0f44/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-07-31_19_03_30 | 20:18 |
| mwhahaha | weshay: is this deployed server? jillr, slagle was looking into this because we were using the wrong notation for the subnetcidr | 20:19 |
| mwhahaha | it is fall out from switching all the stuff over to get_attr | 20:19 |
| weshay | mwhahaha, ya.. deployed_server | 20:19 |
| weshay | mwhahaha, rdo cloud must have been belly up when these merged | 20:20 |
| mwhahaha | weshay: preeeetty much | 20:20 |
| weshay | ya | 20:20 |
| weshay | FAK | 20:20 |
| dsneddon | weshay, mwhahaha: Symptoms are similar to this bug, but maybe with a different cause: https://bugs.launchpad.net/heat/+bug/1616027 | 20:21 |
| openstack | Launchpad bug 1616027 in OpenStack Heat "Error on creating port with fixed ip" [Low,Incomplete] - Assigned to Dmitriy (duvarenkov) | 20:21 |
| dsneddon | also similar to this (related) bug: https://bugs.launchpad.net/kolla/+bug/1628572 | 20:21 |
| openstack | Launchpad bug 1628572 in kolla "heat need python-croniter" [Low,Invalid] - Assigned to halfss (ljvsss) | 20:21 |
| *** morazi has joined #tripleo | 20:22 | |
| mwhahaha | dsneddon: yea if i understand correctly it's because assumptions were with those get_attr patch that the port information is actually a neutron port but in deployed server, we don't have neutron ports | 20:22 |
| mwhahaha | so we fake it | 20:22 |
| dsneddon | mwhahaha, Oh, I see. Yeah, that would be a problem. | 20:22 |
| dsneddon | mwhahaha, What happens if you manually set ControlPlaneSubnetCidr to '24'? | 20:23 |
| mwhahaha | dsneddon: that's what we currently do, so what you're seeing :D | 20:23 |
| *** pchavva has quit IRC | 20:23 | |
| mwhahaha | dsneddon: https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master/fa46166/logs/undercloud/home/zuul/network-environment.yaml.txt.gz | 20:23 |
| dsneddon | mwhahaha, Oh, I was hoping that maybe if that were set, it wouldn't try and resolve the get_attr. | 20:23 |
| mwhahaha | yea jillr was looking at a THT patch to fix up the get_attr when the Cidr options were not a proper ip, but i don't know the status of that | 20:24 |
| *** lblanchard has quit IRC | 20:25 | |
| dsneddon | mwhahaha, I can see this code at L467 is supposed to check to see if that param is set: https://github.com/openstack/tripleo-heat-templates/commit/dfc09b6ff1141d05d8c0c11ff2f5ac5e4dddc87f#diff-f620b68f0bb72070d60fe8e1f3b0238cR468 | 20:25 |
| mwhahaha | it doesn't seem to be working correctly | 20:25 |
| weshay | mwhahaha, https://review.openstack.org/#/c/579582/ | 20:25 |
| weshay | ya.. no 3rd party | 20:26 |
| *** wolverineav has quit IRC | 20:26 | |
| weshay | this one worked w/ 3rd party https://review.openstack.org/#/c/579580/ | 20:26 |
| weshay | https://review.openstack.org/#/c/579579/ | 20:27 |
| weshay | and that | 20:27 |
| mwhahaha | so maybe it's something else that is just exposed because of those changes | 20:27 |
| dsneddon | mwhahaha, weshay: At one point, https://review.openstack.org/#/c/579582/ was passing 3rd-party featureset001-master, but it looks like it got merged with that failing? | 20:28 |
| mwhahaha | it might be a promotion problem actually | 20:29 |
| mwhahaha | because CI is currently fine, but the periodic jobs are promotions right? | 20:29 |
| weshay | ya.. periodic == promotion | 20:29 |
| mwhahaha | weshay: https://github.com/openstack/heat/commit/5e52ee706c853f146b8adca3524cae1ef458361c | 20:31 |
| mwhahaha | https://review.openstack.org/#/c/575265/ | 20:32 |
| weshay | mwhahaha, bah.. and we have nothing on heat https://review.openstack.org/#/c/575265/ | 20:32 |
| *** ansmith has quit IRC | 20:32 | |
| weshay | how am I just realizing that :( | 20:33 |
| mwhahaha | zaneb, therve -^ | 20:33 |
| weshay | zaneb, therve would be super cool to add a tripleo job to the check jobs there | 20:34 |
| zaneb | weshay: feel free to propose one | 20:36 |
| weshay | zaneb, will do.. looking at your other jobs.. 1.5 hours seems to be a reasonable run time | 20:36 |
| weshay | guess we'll see if the community thinks so | 20:36 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: upgrade: move removal of services at step 3 (leftovers) https://review.openstack.org/587270 | 20:36 |
| zaneb | yeah, not so much worse than the grenade-multinode | 20:37 |
| openstackgerrit | Merged openstack/tripleo-quickstart master: Add fs055 to run refstack tests https://review.openstack.org/570884 | 20:37 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/queens: Fix GlanceRegistry FFU cleanup https://review.openstack.org/587441 | 20:37 |
| zaneb | mwhahaha: do we need to revert https://review.openstack.org/#/c/575265/ ? because I was never comfortable with that change anyway | 20:37 |
| *** zul has quit IRC | 20:37 | |
| *** zul has joined #tripleo | 20:38 | |
| zaneb | or is it just the new library that is the problem? | 20:38 |
| *** mhenkel__ has quit IRC | 20:38 | |
| zaneb | oh, it's the new library https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/0ec0f44/logs/undercloud/var/log/containers/heat/heat-engine.log.txt.gz#_2018-07-31_18_26_42_846 | 20:41 |
| mwhahaha | zaneb: no idea, but it used to work (current ci) and doesn't seem to be working in promotion | 20:41 |
| zaneb | https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/0ec0f44/logs/undercloud/var/log/containers/heat/heat-engine.log.txt.gz?level=WARNING#_2018-07-31_18_26_42_846 | 20:41 |
| zaneb | mwhahaha: you're missing the new requirement, neutronlib, in the container | 20:42 |
| jillr | mwhahaha: weshay: still working on it, I believe there's more than one place that needs to be changed, | 20:42 |
| jillr | if you can specify the full cidr in DeployedServerPortMap and it should work for now | 20:43 |
| jtcressy | fultonj: If one of my ceph servers has 5 OSD's and the other two each have 6 OSD's, is that a reason for my cluster to be in HEALTH_ERR? I omitted one of the OSD's from a server so it has one less than the other two. | 20:46 |
| fultonj | jtcressy: no you don't have to have the same number of OSDs per server | 20:46 |
| jtcressy | it tells me that 7/15 objects are degraded | 20:47 |
| fultonj | http://docs.ceph.com/docs/giant/rados/troubleshooting/troubleshooting-osd/ | 20:48 |
| fultonj | some of the commands above are diff though as ceph is in contianers | 20:49 |
| jtcressy | looks like /var/log/ceph is empty on the controller nodes. | 20:49 |
| fultonj | because of how ceph contianers are | 20:49 |
| jtcressy | and there's no ceph in /var/log/containers/ | 20:49 |
| fultonj | yes | 20:50 |
| fultonj | that's true | 20:50 |
| jtcressy | oh, so they dont log to a file on the host | 20:50 |
| fultonj | sudo journalctl CONTAINER_NAME=ceph-mon-overcloud-controller-0 | 20:50 |
| fultonj | or similar to get the logs | 20:50 |
| *** janki has quit IRC | 20:51 | |
| fultonj | so processes to get logs or stop/start will differ if ceph in containers | 20:51 |
| fultonj | but the concepts in ^ doc are the same | 20:51 |
| fultonj | jtcressy: you may wish to 'docker exec -ti $id /bin/bash ' | 20:52 |
| fultonj | for the id of your monitor | 20:52 |
| fultonj | and then see what's going on | 20:52 |
| fultonj | are the pg numbers good? | 20:52 |
| fultonj | what's going on as per 'ceph -w' | 20:52 |
| jtcressy | Here's ceph -s: https://hastebin.com/onenahuzeb | 20:52 |
| mwhahaha | weshay: zaneb ok so it seems like it's just a missing dep int he heat packaging/containers | 20:53 |
| jtcressy | ceph -w shows a line: Health check update: 22 slow requests are blocked > 32 sec | 20:53 |
| zaneb | mwhahaha: yep. | 20:53 |
| jtcressy | I assume the assessment here is that pg numbers are not great | 20:54 |
| *** wolverineav has joined #tripleo | 20:54 | |
| weshay | mwhahaha, EmilienM fyi 1 gate failure in the last 30hr or so hours | 20:55 |
| weshay | rdo on the other hand | 20:55 |
| jtcressy | Health check update: 27 stuck requests are blocked > 4096 sec | 20:55 |
| fultonj | looks like the cluster is trying to recover but getting stuck | 20:55 |
| *** wolverineav has quit IRC | 20:55 | |
| jtcressy | It doesn't seem to tell me which OSD's are dead | 20:56 |
| fultonj | you might want to ask if anyone wants to help in #ceph | 20:56 |
| fultonj | they're all up and in | 20:56 |
| fultonj | 0 dead as per that ceph-s | 20:56 |
| fultonj | objects just getting degraded unable to replicate | 20:56 |
| fultonj | jtcressy: if it's just a test cluster | 20:57 |
| itlinux | hello all, is there a way to use the ansible to gen the /etc/hosts file for the overcloud.. | 20:58 |
| itlinux | from the undercloud | 20:59 |
| *** florianf has quit IRC | 20:59 | |
| *** wolverin_ has joined #tripleo | 21:00 | |
| *** edmondsw has quit IRC | 21:00 | |
| *** jtomasek has joined #tripleo | 21:00 | |
| *** raildo has quit IRC | 21:00 | |
| *** edmondsw has joined #tripleo | 21:01 | |
| *** jangutter has quit IRC | 21:01 | |
| *** edmondsw has quit IRC | 21:01 | |
| *** edmondsw has joined #tripleo | 21:01 | |
| jtcressy | fultonj: BTW, if there is supposed to be 1 OSD per disk, I _should_ have 17 OSD's... not 13. | 21:02 |
| jtcressy | 18 disks, omit one because it was haunted. | 21:02 |
| fultonj | ah | 21:02 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/queens: Support containerized DVR in compute role https://review.openstack.org/573199 | 21:02 |
| openstackgerrit | Merged openstack/puppet-tripleo stable/ocata: remove scenario005 from experimental https://review.openstack.org/587216 | 21:02 |
| fultonj | docker ps | grep osd | 21:02 |
| fultonj | to find problem OSDs | 21:03 |
| *** jtomasek_ has quit IRC | 21:03 | |
| fultonj | or osd map | 21:03 |
| fultonj | ssh heat-admin@lab-controller01 "ceph node ls osd" | 21:04 |
| jtcressy | Used docker ps against the three ceph nodes: https://hastebin.com/rigobolufu.rb | 21:05 |
| jtcressy | so maybe ceph-storage-0 is having major issues? | 21:06 |
| *** gbarros has quit IRC | 21:07 | |
| *** Goneri has quit IRC | 21:08 | |
| fultonj | thats the one who losd some osds | 21:08 |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart master: Add feature set file to run HA validations https://review.openstack.org/538952 | 21:08 |
| jtcressy | It seems some of the containers were never created on ceph-storage-0 | 21:08 |
| jtcressy | maybe a stack-update will kick it into gear? | 21:08 |
| fultonj | jtcressy: maybe they didn't get enough ram? | 21:08 |
| fultonj | starved | 21:08 |
| fultonj | https://docs.openstack.org/tripleo-docs/latest/install/advanced_deployment/ceph_config.html#configure-container-settings-with-ceph-ansible | 21:09 |
| jtcressy | the containers should still show if they were attempted to be started right? they would show exited if created and failed? | 21:09 |
| *** mhenkel_ has joined #tripleo | 21:09 | |
| fultonj | you'd see the existed ones | 21:09 |
| *** rfolco|ruck is now known as rfolco|off | 21:09 | |
| fultonj | might they be flapping? | 21:09 |
| fultonj | jtcressy: give each osd 5g of ram as per that doc ^ | 21:10 |
| jtcressy | The node they're running on has 2x12C CPU and 48GB of ram | 21:10 |
| *** ooolpbot has joined #tripleo | 21:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784017 | 21:10 |
| *** ooolpbot has quit IRC | 21:10 | |
| openstack | Launchpad bug 1784017 in tripleo "Build of instance was re-scheduled: invalid argument: could not find capabilities for domaintype=kvm" [Critical,In progress] - Assigned to Rafael Folco (rafaelfolco) | 21:10 |
| fultonj | ceph-ansible 3.10 defaults it to 5g per osd | 21:10 |
| fultonj | ceph-ansible 3.1...rc10 i mean | 21:10 |
| fultonj | each container has an upper limit you can set | 21:10 |
| fultonj | if you're having a resource problem | 21:11 |
| fultonj | vi /etc/systemd/system/ceph-osd\@.service | 21:11 |
| fultonj | restart=always --> restart=never | 21:12 |
| fultonj | if you have flapping problem | 21:12 |
| jtcressy | no flapping going on | 21:12 |
| fultonj | then you can debug an OSD to see why it dies | 21:12 |
| fultonj | good | 21:12 |
| fultonj | get the logs of a dead OSD and see what its problem is | 21:13 |
| fultonj | or start a new one and watch the logs | 21:13 |
| fultonj | if it's a test cluster reduce your replica requirements | 21:13 |
| fultonj | low-mem-usage reduces it a lot | 21:13 |
| fultonj | but then you have no recourse if you loose osds | 21:13 |
| fultonj | parameter_defaults: | 21:13 |
| fultonj | CephPoolDefaultSize: 3 | 21:13 |
| fultonj | CephPoolDefaultPgNum: 128 | 21:13 |
| fultonj | many thnings | 21:13 |
| *** bfournie has quit IRC | 21:14 | |
| fultonj | could go wrong | 21:14 |
| *** bfournie has joined #tripleo | 21:14 | |
| fultonj | i would follow ceph troubleshooting guide to find | 21:14 |
| *** bfournie has quit IRC | 21:14 | |
| mwhahaha | weshay: wheres the bug for that heat thing, i'll update the distgit | 21:16 |
| openstackgerrit | Kaio Kassiano Moura Oliveira proposed openstack/diskimage-builder master: Fix typo in installation.rst https://review.openstack.org/587626 | 21:19 |
| weshay | mwhahaha, /me gets | 21:20 |
| *** wolverin_ has quit IRC | 21:20 | |
| weshay | sorry had to make a sandwhich | 21:20 |
| weshay | mwhahaha, https://bugs.launchpad.net/tripleo/+bug/1784712 | 21:21 |
| openstack | Launchpad bug 1784712 in tripleo "ExternalNetwork, InternalApiNetwork, StorageNetwork fail to validate ip_addr" [Critical,Triaged] | 21:21 |
| mwhahaha | k | 21:21 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates stable/queens: nova: add parameters to configure nova::cron::archive_deleted_rows https://review.openstack.org/587446 | 21:22 |
| jtcressy | fultonj: the osds are up on overcloud-cephstorage-0 but I dont see them in the ceph -s command yet in the controllers. | 21:25 |
| *** yolanda has quit IRC | 21:25 | |
| jtcressy | the log on each OSD just has this one line, and one line only: "2018-07-31 21:24:22 /entrypoint.sh: static: does not generate config" | 21:26 |
| *** myoung has quit IRC | 21:28 | |
| *** ansmith has joined #tripleo | 21:30 | |
| *** agopi has quit IRC | 21:30 | |
| jtcressy | fultonj: I'm just going to run a stack update with the updated resources settings you were talking about. 5GB per OSD this time. I don't know what else to try on this. The containers just disappear even though i set resetart=never. | 21:35 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras master: Use local docker registry host for tempest container https://review.openstack.org/584368 | 21:40 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Update/upgrade prepare/converge in config-download era https://review.openstack.org/583320 | 21:50 |
| *** bnemec has quit IRC | 21:51 | |
| *** avivgt has quit IRC | 21:52 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-common master: Make MetricsQdr service use qdrouterd image https://review.openstack.org/578749 | 21:54 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-heat-templates master: QDR for metrics collection purposes https://review.openstack.org/572312 | 21:55 |
| *** mhenkel_ has quit IRC | 21:56 | |
| tonyb | @mwhahaha: So sorry :( | 22:05 |
| mwhahaha | tonyb: it's all good, jschlueter proposed a fix (its in the gate) | 22:06 |
| tonyb | mwhahaha: Yup saw that, it's pretty simple, I have no excuse for missing that | 22:06 |
| mwhahaha | tonyb: need some non x86_64 test nodes | 22:07 |
| tonyb | mwhahaha: Yes Yes we do! | 22:08 |
| tonyb | mwhahaha: I *probably* shoudl ahev run the tests locally on ppc64le at least that would have helped :/ | 22:08 |
| tonyb | mwhahaha: When I backport the buggy chnage shoudl I squash Jon's fix or keep the hstory clear? | 22:09 |
| *** itlinux has quit IRC | 22:09 | |
| mwhahaha | tonyb: just backport both at the same time | 22:10 |
| tonyb | mwhahaha: Okay | 22:10 |
| *** paramite has quit IRC | 22:10 | |
| mwhahaha | (not squashed) | 22:10 |
| *** ooolpbot has joined #tripleo | 22:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784017 | 22:10 |
| *** ooolpbot has quit IRC | 22:10 | |
| openstack | Launchpad bug 1784017 in tripleo "Build of instance was re-scheduled: invalid argument: could not find capabilities for domaintype=kvm" [Critical,In progress] - Assigned to Rafael Folco (rafaelfolco) | 22:10 |
| mwhahaha | it didn't block anything upstream so we can land them seperately | 22:10 |
| tonyb | mwhahaha: cool | 22:10 |
| *** mcornea has quit IRC | 22:14 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Fix manila update, upgrade, and FFU tasks https://review.openstack.org/586623 | 22:15 |
| *** slaweq has quit IRC | 22:16 | |
| *** bfournie has joined #tripleo | 22:17 | |
| *** jtomasek has quit IRC | 22:21 | |
| *** ade_lee has quit IRC | 22:22 | |
| *** edmondsw has quit IRC | 22:23 | |
| *** edmondsw has joined #tripleo | 22:23 | |
| *** rcernin has joined #tripleo | 22:26 | |
| *** rrubins has quit IRC | 22:29 | |
| *** edmondsw has quit IRC | 22:30 | |
| *** rrubins has joined #tripleo | 22:30 | |
| *** edmondsw has joined #tripleo | 22:31 | |
| *** tvignaud has joined #tripleo | 22:51 | |
| openstackgerrit | Jill Rouleau proposed openstack/ansible-role-tripleo-cookiecutter master: Add zuul config, tox files, and lint script. https://review.openstack.org/587654 | 22:54 |
| *** mhenkel_ has joined #tripleo | 23:01 | |
| *** cdearborn has quit IRC | 23:03 | |
| *** mschuppert has quit IRC | 23:04 | |
| *** mhenkel_ has quit IRC | 23:05 | |
| *** ooolpbot has joined #tripleo | 23:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784017 | 23:10 |
| *** ooolpbot has quit IRC | 23:10 | |
| openstack | Launchpad bug 1784017 in tripleo "Build of instance was re-scheduled: invalid argument: could not find capabilities for domaintype=kvm" [Critical,In progress] - Assigned to Rafael Folco (rafaelfolco) | 23:10 |
| *** edmondsw has quit IRC | 23:11 | |
| openstackgerrit | Merged openstack/tripleo-quickstart master: fs050: enable TLS https://review.openstack.org/585333 | 23:12 |
| *** agopi has joined #tripleo | 23:16 | |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart-extras master: Integrate Validate-HA work into OVB jobs https://review.openstack.org/537669 | 23:21 |
| *** pmannidi has joined #tripleo | 23:21 | |
| openstackgerrit | Alex Schultz proposed openstack/ansible-role-tripleo-cookiecutter master: Switch README to RST and add lint https://review.openstack.org/587218 | 23:23 |
| *** threestrands has joined #tripleo | 23:24 | |
| *** threestrands has quit IRC | 23:24 | |
| *** threestrands has joined #tripleo | 23:24 | |
| *** gfidente has quit IRC | 23:34 | |
| *** tosky has quit IRC | 23:35 | |
| openstackgerrit | Steve Baker proposed openstack/tripleo-common master: Create script tripleo-container-image-prepare https://review.openstack.org/587239 | 23:40 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-common master: Action to perform container image prepare https://review.openstack.org/585913 | 23:40 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-common master: Add container_image_prepare_params to deployment workflow https://review.openstack.org/586418 | 23:40 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-heat-templates master: Break out image prepare into its own "service" https://review.openstack.org/581918 | 23:40 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-quickstart-extras master: Switch to workflow driven image prepare https://review.openstack.org/573476 | 23:42 |
| *** itlinux has joined #tripleo | 23:42 | |
| *** toure is now known as toure|gone | 23:50 | |
| *** rpioso is now known as rpioso|afk | 23:52 | |
| *** jtcressy has quit IRC | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!