| *** brault has joined #tripleo | 00:00 | |
| *** gregwork has quit IRC | 00:00 | |
| *** gregwork has joined #tripleo | 00:03 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [WIP] use networkx https://review.openstack.org/467419 | 00:04 |
|---|---|---|
| *** brault has quit IRC | 00:05 | |
| *** ooolpbot has joined #tripleo | 00:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1680195 | 00:10 |
| openstack | Launchpad bug 1680195 in tripleo "Random ovb-ha ping test failures" [Critical,Triaged] | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1690132 | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1690373 | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1692503 | 00:10 |
| *** ooolpbot has quit IRC | 00:10 | |
| openstack | Launchpad bug 1690132 in tripleo "Newton HA failure: unable to push cib, cib_apply_diff failed" [Critical,Triaged] | 00:10 |
| openstack | Launchpad bug 1690373 in tripleo "stable/newton gate-tripleo-ci-centos-7-nonha-multinode-oooq broken" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 00:10 |
| openstack | Launchpad bug 1692503 in tripleo "CI: containers job fails with "oci runtime error..."" [Critical,Triaged] | 00:10 |
| *** ecerquei has quit IRC | 00:12 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [WIP] use networkx https://review.openstack.org/467419 | 00:22 |
| *** itlinux has joined #tripleo | 00:24 | |
| *** ecerquei has joined #tripleo | 00:30 | |
| *** gbarros has joined #tripleo | 00:52 | |
| *** eck`gone is now known as eck` | 01:06 | |
| *** itlinux has quit IRC | 01:06 | |
| *** ooolpbot has joined #tripleo | 01:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1680195 | 01:10 |
| openstack | Launchpad bug 1680195 in tripleo "Random ovb-ha ping test failures" [Critical,Triaged] | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1690132 | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1690373 | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1692503 | 01:10 |
| *** ooolpbot has quit IRC | 01:10 | |
| openstack | Launchpad bug 1690132 in tripleo "Newton HA failure: unable to push cib, cib_apply_diff failed" [Critical,Triaged] | 01:10 |
| openstack | Launchpad bug 1690373 in tripleo "stable/newton gate-tripleo-ci-centos-7-nonha-multinode-oooq broken" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 01:10 |
| openstack | Launchpad bug 1692503 in tripleo "CI: containers job fails with "oci runtime error..."" [Critical,Triaged] | 01:10 |
| *** ecerquei_ has joined #tripleo | 01:18 | |
| openstackgerrit | Ihar Hrachyshka proposed openstack/tripleo-heat-templates master: Enable arp_accept for all interfaces https://review.openstack.org/467430 | 01:19 |
| openstackgerrit | Merged openstack/instack-undercloud master: Add unit test and release note for tuskar package removal https://review.openstack.org/466798 | 01:20 |
| openstackgerrit | Merged openstack/instack-undercloud master: Clarify the enable_cinder option description https://review.openstack.org/466722 | 01:20 |
| openstackgerrit | Merged openstack/instack-undercloud master: Remove hardcoded libvirt default network iptalbes rules https://review.openstack.org/464633 | 01:20 |
| *** ecerquei has quit IRC | 01:21 | |
| *** rbrady has quit IRC | 01:35 | |
| *** itlinux has joined #tripleo | 01:35 | |
| *** ramishra has joined #tripleo | 01:35 | |
| *** gbarros has quit IRC | 01:37 | |
| *** tzumainn has quit IRC | 01:46 | |
| *** rbrady has joined #tripleo | 01:46 | |
| *** rbrady has joined #tripleo | 01:46 | |
| *** itlinux has quit IRC | 01:52 | |
| *** ramishra has quit IRC | 02:00 | |
| *** gbarros has joined #tripleo | 02:05 | |
| *** gbarros has quit IRC | 02:10 | |
| *** ooolpbot has joined #tripleo | 02:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1680195 | 02:10 |
| openstack | Launchpad bug 1680195 in tripleo "Random ovb-ha ping test failures" [Critical,Triaged] | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1690132 | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1690373 | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1692503 | 02:10 |
| *** ooolpbot has quit IRC | 02:10 | |
| openstack | Launchpad bug 1690132 in tripleo "Newton HA failure: unable to push cib, cib_apply_diff failed" [Critical,Triaged] | 02:10 |
| openstack | Launchpad bug 1690373 in tripleo "stable/newton gate-tripleo-ci-centos-7-nonha-multinode-oooq broken" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 02:10 |
| openstack | Launchpad bug 1692503 in tripleo "CI: containers job fails with "oci runtime error..."" [Critical,Triaged] | 02:10 |
| *** atoth has quit IRC | 02:23 | |
| *** rbrady has quit IRC | 02:42 | |
| *** yamahata has quit IRC | 03:00 | |
| *** gbarros has joined #tripleo | 03:07 | |
| *** ooolpbot has joined #tripleo | 03:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1680195 | 03:10 |
| openstack | Launchpad bug 1680195 in tripleo "Random ovb-ha ping test failures" [Critical,Triaged] | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1690132 | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1690373 | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1692503 | 03:10 |
| *** ooolpbot has quit IRC | 03:10 | |
| openstack | Launchpad bug 1690132 in tripleo "Newton HA failure: unable to push cib, cib_apply_diff failed" [Critical,Triaged] | 03:10 |
| openstack | Launchpad bug 1690373 in tripleo "stable/newton gate-tripleo-ci-centos-7-nonha-multinode-oooq broken" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 03:10 |
| openstack | Launchpad bug 1692503 in tripleo "CI: containers job fails with "oci runtime error..."" [Critical,Triaged] | 03:10 |
| *** michapma_dsk has joined #tripleo | 03:10 | |
| *** itlinux has joined #tripleo | 03:10 | |
| *** gbarros has quit IRC | 03:23 | |
| *** michapma_dsk has quit IRC | 03:27 | |
| *** michapma_dsk has joined #tripleo | 03:29 | |
| *** dmacpher is now known as dmacpher-afk | 03:31 | |
| *** masco has joined #tripleo | 03:35 | |
| *** ecerquei_ has quit IRC | 03:37 | |
| *** psahoo has joined #tripleo | 03:40 | |
| *** eck` is now known as eck`gone | 03:46 | |
| *** ykarel has joined #tripleo | 03:52 | |
| *** rbrady has joined #tripleo | 03:52 | |
| *** rbrady has joined #tripleo | 03:52 | |
| *** jaganathan has joined #tripleo | 04:07 | |
| *** itlinux has quit IRC | 04:08 | |
| *** ooolpbot has joined #tripleo | 04:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1680195 | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1690132 | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1690373 | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1692503 | 04:10 |
| openstack | Launchpad bug 1680195 in tripleo "Random ovb-ha ping test failures" [Critical,Triaged] | 04:10 |
| *** ooolpbot has quit IRC | 04:10 | |
| openstack | Launchpad bug 1690132 in tripleo "Newton HA failure: unable to push cib, cib_apply_diff failed" [Critical,Triaged] | 04:10 |
| openstack | Launchpad bug 1690373 in tripleo "stable/newton gate-tripleo-ci-centos-7-nonha-multinode-oooq broken" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 04:10 |
| openstack | Launchpad bug 1692503 in tripleo "CI: containers job fails with "oci runtime error..."" [Critical,Triaged] | 04:10 |
| -openstackstatus- NOTICE: Sufficient free space has been reclaimed that jobs are passing again; any POST_FAILURE results can now be rechecked. | 04:10 | |
| *** ChanServ changes topic to "TripleO : http://tripleo.org/ | https://wiki.openstack.org/wiki/TripleO | Meetings On Tuesdays at 14:00 UTC in #openstack-meeting-alt" | 04:10 | |
| *** ramishra has joined #tripleo | 04:10 | |
| *** ratailor has joined #tripleo | 04:13 | |
| *** gbarros has joined #tripleo | 04:20 | |
| *** udesale has joined #tripleo | 04:24 | |
| *** yamahata has joined #tripleo | 04:32 | |
| *** gbarros has quit IRC | 04:34 | |
| *** dmacpher-afk is now known as dmacpher | 04:44 | |
| *** bkopilov has joined #tripleo | 04:55 | |
| *** jaosorior_away is now known as jaosorior | 04:55 | |
| *** mdnadeem has joined #tripleo | 04:55 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Use networkx for digraph https://review.openstack.org/467419 | 04:58 |
| *** mdnadeem has quit IRC | 05:01 | |
| *** flepied has joined #tripleo | 05:02 | |
| *** jaganathan_ has joined #tripleo | 05:02 | |
| *** jaganathan_ has joined #tripleo | 05:03 | |
| *** jaganathan has quit IRC | 05:05 | |
| openstackgerrit | Rajat Sharma proposed openstack/tripleo-common master: Replace assertRaisesRegexp with assertRaisesRegex https://review.openstack.org/467484 | 05:07 |
| *** ooolpbot has joined #tripleo | 05:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1680195 | 05:10 |
| openstack | Launchpad bug 1680195 in tripleo "Random ovb-ha ping test failures" [Critical,Triaged] | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1690132 | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1690373 | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1692503 | 05:10 |
| *** ooolpbot has quit IRC | 05:10 | |
| openstack | Launchpad bug 1690132 in tripleo "Newton HA failure: unable to push cib, cib_apply_diff failed" [Critical,Triaged] | 05:10 |
| openstack | Launchpad bug 1690373 in tripleo "stable/newton gate-tripleo-ci-centos-7-nonha-multinode-oooq broken" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 05:10 |
| openstack | Launchpad bug 1692503 in tripleo "CI: containers job fails with "oci runtime error..."" [Critical,Triaged] | 05:10 |
| *** skramaja has joined #tripleo | 05:11 | |
| *** prateek__ has joined #tripleo | 05:17 | |
| *** links has joined #tripleo | 05:20 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Set manifest permissions in the image https://review.openstack.org/466593 | 05:21 |
| *** masco has quit IRC | 05:23 | |
| *** agurenko has joined #tripleo | 05:24 | |
| *** mdnadeem has joined #tripleo | 05:30 | |
| *** gbarros has joined #tripleo | 05:30 | |
| *** yprokule has joined #tripleo | 05:34 | |
| *** mdnadeem has quit IRC | 05:40 | |
| *** gbarros has quit IRC | 05:45 | |
| *** dparkes has joined #tripleo | 05:46 | |
| *** lmiccini has joined #tripleo | 05:47 | |
| *** flepied has quit IRC | 05:47 | |
| *** masco has joined #tripleo | 05:54 | |
| *** iranzo has joined #tripleo | 05:54 | |
| *** iranzo has joined #tripleo | 05:54 | |
| *** mdnadeem has joined #tripleo | 05:54 | |
| *** brault has joined #tripleo | 06:00 | |
| *** rcernin has quit IRC | 06:00 | |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-heat-templates master: WIP: Enable DPDK on boot using PreNetworkConfig https://review.openstack.org/467490 | 06:00 |
| *** brault has quit IRC | 06:04 | |
| *** ooolpbot has joined #tripleo | 06:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1680195 | 06:10 |
| openstack | Launchpad bug 1680195 in tripleo "Random ovb-ha ping test failures" [Critical,Triaged] | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1690132 | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1690373 | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1692503 | 06:10 |
| *** ooolpbot has quit IRC | 06:10 | |
| openstack | Launchpad bug 1690132 in tripleo "Newton HA failure: unable to push cib, cib_apply_diff failed" [Critical,Triaged] | 06:10 |
| openstack | Launchpad bug 1690373 in tripleo "stable/newton gate-tripleo-ci-centos-7-nonha-multinode-oooq broken" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 06:10 |
| openstack | Launchpad bug 1692503 in tripleo "CI: containers job fails with "oci runtime error..."" [Critical,Triaged] | 06:10 |
| *** ebarrera has joined #tripleo | 06:12 | |
| *** ebarrera has quit IRC | 06:13 | |
| chandankumar | flaper87: hello, May i pm? | 06:13 |
| *** jprovazn has joined #tripleo | 06:17 | |
| *** rcernin has joined #tripleo | 06:17 | |
| *** maeca1 has quit IRC | 06:20 | |
| *** gfidente has quit IRC | 06:21 | |
| *** brault has joined #tripleo | 06:27 | |
| *** chem has quit IRC | 06:32 | |
| *** mandre|away is now known as mandre | 06:35 | |
| *** florianf has joined #tripleo | 06:40 | |
| *** jaganathan_ has quit IRC | 06:41 | |
| *** gbarros has joined #tripleo | 06:42 | |
| *** ffiore has joined #tripleo | 06:43 | |
| *** gbarros has quit IRC | 06:52 | |
| *** aufi has joined #tripleo | 06:53 | |
| *** bkopilov has quit IRC | 06:55 | |
| *** bogdando has joined #tripleo | 06:57 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Set manifest permissions in the image https://review.openstack.org/466593 | 06:58 |
| *** jlinkes has joined #tripleo | 07:00 | |
| *** ffiore has quit IRC | 07:00 | |
| *** ebarrera has joined #tripleo | 07:01 | |
| *** bkopilov has joined #tripleo | 07:08 | |
| *** ooolpbot has joined #tripleo | 07:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1680195 | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1690132 | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1690373 | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1692503 | 07:10 |
| openstack | Launchpad bug 1680195 in tripleo "Random ovb-ha ping test failures" [Critical,Triaged] | 07:10 |
| *** ooolpbot has quit IRC | 07:10 | |
| openstack | Launchpad bug 1690132 in tripleo "Newton HA failure: unable to push cib, cib_apply_diff failed" [Critical,Triaged] | 07:10 |
| openstack | Launchpad bug 1690373 in tripleo "stable/newton gate-tripleo-ci-centos-7-nonha-multinode-oooq broken" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 07:10 |
| openstack | Launchpad bug 1692503 in tripleo "CI: containers job fails with "oci runtime error..."" [Critical,Triaged] | 07:10 |
| *** ffiore has joined #tripleo | 07:12 | |
| *** jroll has quit IRC | 07:12 | |
| *** shardy has joined #tripleo | 07:14 | |
| *** athomas has joined #tripleo | 07:15 | |
| *** yamahata has quit IRC | 07:15 | |
| *** pcaruana has joined #tripleo | 07:16 | |
| *** anshulbehl has joined #tripleo | 07:21 | |
| *** dbecker has joined #tripleo | 07:21 | |
| *** chem has joined #tripleo | 07:23 | |
| openstackgerrit | Juan Badia Payno proposed openstack/tripleo-heat-templates master: WIP - Containerized Fluentd client https://review.openstack.org/467072 | 07:24 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-common master: Add octavia container images https://review.openstack.org/467239 | 07:29 |
| flaper87 | chandankumar: hey, sure, fire away | 07:31 |
| chandankumar | flaper87: Thanks :-) | 07:32 |
| *** agurenko has quit IRC | 07:34 | |
| *** karimb has joined #tripleo | 07:35 | |
| *** dtantsur|afk is now known as dtantsur | 07:36 | |
| *** oanson has quit IRC | 07:36 | |
| *** oanson has joined #tripleo | 07:37 | |
| *** suuuper has joined #tripleo | 07:37 | |
| *** jpena|off is now known as jpena | 07:38 | |
| skramaja | shardy: can you please take a look at https://review.openstack.org/#/c/446093/ ? | 07:39 |
| shardy | skramaja: morning, yes will do | 07:39 |
| *** agurenko has joined #tripleo | 07:39 | |
| skramaja | good morning. and thanks. | 07:40 |
| ccamacho | hello guys good morning | 07:40 |
| *** agurenko has quit IRC | 07:41 | |
| *** jaganathan_ has joined #tripleo | 07:44 | |
| *** agurenko has joined #tripleo | 07:45 | |
| *** egonzalez has joined #tripleo | 07:45 | |
| shardy | skramaja: looks good, but shouldn't HostCpusList be prefixed so we don't risk some future name collision? | 07:47 |
| shardy | skramaja: I suppose we already missed that, but perhaps we could fix it in future? | 07:47 |
| skramaja | shardy: the idea is to move it to role-specific | 07:48 |
| *** jpich has joined #tripleo | 07:48 | |
| *** dsariel has joined #tripleo | 07:48 | |
| skramaja | shardy: it is specific to host of the role and not associated with feature. | 07:48 |
| *** gbarros has joined #tripleo | 07:49 | |
| shardy | skramaja: Ok, but what is "host process" in this case, it's the CPUs we allow dpdk processing on right, e.g some subset of the host CPUs? | 07:49 |
| shardy | skramaja: I'm happy to land this now, just wanted to make those comments | 07:50 |
| skramaja | shardy: the same set of cpus will be used by all the host process.. for dpdk we need to specific explity, the same list of cores will be writedn to CPUAffinity in /etc/systemd/system.conf | 07:51 |
| *** colonwq has quit IRC | 07:52 | |
| skramaja | shardy: all the related parameters are here - https://github.com/krsacme/tht-dpdk/blob/master/osp10_ovs26/network-environment.yaml#L61 | 07:52 |
| shardy | skramaja: Ok, I see, so we might e.g reserve some CPUs for pinning to VMs or whatever | 07:52 |
| shardy | skramaja: thanks for clarifying, approved :) | 07:53 |
| skramaja | shardy: thanks. there are lot of issues around it and we are hoping that deive parameters will ease it for the operators. | 07:54 |
| mrunge | hello, is there any recommendation on how to recover services on a oooq deployed machine? | 07:59 |
| mrunge | at least in my case, no service survived a reboot. | 07:59 |
| mrunge | does that mean, I have to re-provision? | 07:59 |
| *** zzzeek has quit IRC | 08:00 | |
| mrunge | undercloud, overcloud and compute vms are stopped | 08:00 |
| *** zzzeek has joined #tripleo | 08:00 | |
| mrunge | after working around the issue that the stack user can not create the /run/user/<id>/libvirt, one can start the vms again. but that doesn't help | 08:01 |
| shardy | mrunge: I raised a bug about that yesterday, sec | 08:01 |
| shardy | https://bugs.launchpad.net/tripleo/+bug/1692976 | 08:01 |
| openstack | Launchpad bug 1692976 in tripleo "tripleo-quickstart VMs can't survive a host reboot" [Undecided,Triaged] | 08:01 |
| mrunge | awesome shadower | 08:01 |
| mrunge | err, shardy | 08:01 |
| mrunge | shardy, at least for that, I have a workaround | 08:02 |
| mrunge | well, sort of | 08:02 |
| shardy | mrunge: I've not fully investigated yet, but basically the undercloud qcow is still under ~/stack, but the magic doing unpriveliged virt is broken | 08:02 |
| mrunge | right | 08:02 |
| mrunge | well | 08:02 |
| shardy | mrunge: FWIW what I want is to make all that unpriveleged stuff optional, and just deploy directly as the root user, then you can reboot the host and just start the VMs in virt-manager | 08:03 |
| mrunge | shardy, comment that XDG_RUNTIME stuff in .bashrc | 08:03 |
| shardy | there's another bug related to that not working: | 08:03 |
| shardy | https://bugs.launchpad.net/tripleo/+bug/1692987 | 08:03 |
| openstack | Launchpad bug 1692987 in tripleo "tripleo-quickstart deploying VMs as root user doesn't work" [Undecided,Triaged] | 08:03 |
| *** nyechiel has joined #tripleo | 08:03 | |
| *** amoralej|off is now known as amoralej | 08:03 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/instack-undercloud stable/ocata: Clarify the enable_cinder option description https://review.openstack.org/467513 | 08:04 |
| shardy | mrunge: maybe sshnaidm adarazs or one of the other quickstart experts can take a look and advise how this is expected to work | 08:05 |
| mrunge | shardy, so the current recommendation would be to redeploy? | 08:05 |
| shardy | mrunge: that's what I've been doing, but given the image is still there it must be recoverable | 08:05 |
| mrunge | that'd be good, otherwise it'd cost me another few hours | 08:06 |
| ebarrera | ping tripleo, I'm having problem with a sample environemnt, I'm trying to make a 1+1 env with two isolated networks ctlplane and external, making only horizon available to the external network. Without using VIP nothing is listening on the external, with VIp everything is listening on the external http://pastebin.test.redhat.com/487172 | 08:06 |
| *** gbarros has quit IRC | 08:06 | |
| mrunge | I'm still looking for the error I was seeing, maybe that's a duplicate | 08:06 |
| mrunge | shardy, that issue might be another reincarnation of https://bugs.launchpad.net/tripleo/+bug/1577809 | 08:08 |
| openstack | Launchpad bug 1577809 in tripleo "virsh error: error: no valid connection error: Cannot create user runtime directory '/run/user/1000/libvirt': Permission denied" [High,Triaged] | 08:08 |
| mrunge | well, probably not | 08:08 |
| *** colonwq has joined #tripleo | 08:09 | |
| *** ooolpbot has joined #tripleo | 08:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1680195 | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1690132 | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1690373 | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1692503 | 08:10 |
| *** ooolpbot has quit IRC | 08:10 | |
| openstack | Launchpad bug 1680195 in tripleo "Random ovb-ha ping test failures" [Critical,Triaged] | 08:10 |
| shardy | mrunge: ack, thanks that does look similar | 08:10 |
| openstack | Launchpad bug 1690132 in tripleo "Newton HA failure: unable to push cib, cib_apply_diff failed" [Critical,Triaged] | 08:10 |
| openstack | Launchpad bug 1690373 in tripleo "stable/newton gate-tripleo-ci-centos-7-nonha-multinode-oooq broken" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 08:10 |
| openstack | Launchpad bug 1692503 in tripleo "CI: containers job fails with "oci runtime error..."" [Critical,Triaged] | 08:10 |
| shardy | ebarrera: is it a VM based environment? | 08:12 |
| ebarrera | shardy, yes, running on top of infrared | 08:12 |
| shardy | ebarrera: I'm wondering if https://github.com/openstack/instack-undercloud/commit/9261a868b681d2aaa4fc673041073e453f0148c8 is related, as we had some forwarding rules that were recently removed | 08:12 |
| *** chem has quit IRC | 08:12 | |
| ebarrera | shardy, let me check.... | 08:13 |
| shardy | ebarrera: by default without net-iso external traffic gets routed via the undercloud, so I wonder if the issue is you have the ctlplane without any isolated internal_api network? | 08:13 |
| *** lucas-afk is now known as lucasagomes | 08:16 | |
| ebarrera | shardy, ahh ok, do you mean that in the templates I pasted the translation is made by iptables rules, no haproxy involved ? | 08:16 |
| shardy | ebarrera: possibly, I'm just wondering if the traffic from the external vip is getting mangled due to the nat | 08:17 |
| shardy | ebarrera: I guess you can traceroute and find out? | 08:17 |
| shardy | ebarrera: FWIW what you're trying isn't a normal net-iso setup, normally it'd be either everything on ctlplane or everything isolated | 08:18 |
| shardy | if you were to enable the internal_api network too I suspect it may work | 08:18 |
| *** hjensas has quit IRC | 08:22 | |
| *** mcornea has joined #tripleo | 08:23 | |
| openstackgerrit | Damien Ciabrini proposed openstack/tripleo-heat-templates master: Containerize HAProxy for HA https://review.openstack.org/459597 | 08:28 |
| ebarrera | shardy, i think i didn't explain it correctly... haproxy is not listening on the external network http://pastebin.test.redhat.com/487172 not sure if we are talking about the same | 08:29 |
| shardy | ebarrera: Ah, sorry, perhaps I misunderstood then | 08:30 |
| ebarrera | shardy, there is more details on the pastebin link | 08:30 |
| openstackgerrit | Damien Ciabrini proposed openstack/tripleo-heat-templates master: Containerize Redis for HA https://review.openstack.org/464958 | 08:31 |
| ebarrera | shardy, the think is I'm not sure if I'm doing something wrong... | 08:31 |
| openstackgerrit | Merged openstack/tripleo-docs master: Add deleting Overcloud section https://review.openstack.org/467152 | 08:33 |
| *** nyechiel has quit IRC | 08:33 | |
| openstackgerrit | Damien Ciabrini proposed openstack/tripleo-heat-templates master: Containerize MySQL for HA https://review.openstack.org/455348 | 08:33 |
| mrunge | shardy, do I report oooq-issues on https://bugs.launchpad.net/tripleo, or is there a more specific place? | 08:34 |
| adarazs | mrunge: it's there. | 08:34 |
| adarazs | mrunge: just add the "quickstart" tag. | 08:34 |
| mrunge | ack, thanks adarazs | 08:34 |
| shardy | mrunge: yup the tripleo LP where I reported those two bugs | 08:34 |
| adarazs | mrunge, shardy: going to read back an take a look at what you were talking about. | 08:34 |
| shardy | mrunge: our issues seem simiar tho, so perhaps add to those? | 08:35 |
| openstackgerrit | Damien Ciabrini proposed openstack/tripleo-heat-templates master: Containerize RabbitMQ for HA https://review.openstack.org/452889 | 08:35 |
| mrunge | shardy, adarazs have you seen such messages before? That might be related: http://paste.openstack.org/show/610522/ | 08:36 |
| *** dtantsur is now known as dtantsur|lunch | 08:36 | |
| mrunge | there is a change in ansible in the virt module, autostart became a parameter, not a command itself | 08:37 |
| mrunge | ... in ansible 2.3 | 08:37 |
| *** derekh has joined #tripleo | 08:37 | |
| mrunge | I see, that was already reported: https://bugs.launchpad.net/tripleo/+bug/1689620 | 08:39 |
| openstack | Launchpad bug 1689620 in tripleo "RHEL virthost unable to autostart vm's with ansible module" [High,In progress] - Assigned to wes hayutin (weshayutin) | 08:39 |
| ebarrera | shardy, in fact the IP addresses on haproxy seems swapped, it binds on an IP in ctlplane and forwards it to the external | 08:40 |
| adarazs | mrunge: I think we have a change that will make that a virsh command instead. | 08:40 |
| mrunge | adarazs, I'd prefer the new ansible 2.3 syntax. | 08:41 |
| *** zoli|gone is now known as zoli | 08:41 | |
| mrunge | you could just add autostart as parameter to this block here: https://github.com/openstack/tripleo-quickstart/blob/master/roles/libvirt/setup/undercloud/tasks/main.yml#L282 | 08:42 |
| adarazs | not sure how that's a problem though then, we're still pinned to ansible 2.2 | 08:43 |
| mrunge | adarazs, http://docs.ansible.com/ansible/virt_module.html | 08:43 |
| mrunge | I'm sure I have ansible 2.3 installed here as well | 08:43 |
| *** salmankhan has joined #tripleo | 08:43 | |
| *** dciabrin has quit IRC | 08:44 | |
| mrunge | it *should* use ansible from virtenv though | 08:46 |
| adarazs | mrunge: if you used quickstart.sh it should install ansible 2.2 in the virtenv: https://github.com/openstack/tripleo-quickstart/blob/master/requirements.txt#L2 | 08:46 |
| mrunge | yes, I saw that adarazs | 08:47 |
| mrunge | I was currently trying to get a proof that it is really being used, instead of the system wide one | 08:47 |
| mrunge | the other question is: why does it get autostart as return value, and shouldn't it accept that as well? | 08:48 |
| ebarrera | shardy, actually can the provisioning network and internal api netw coexist in the same network interface without using vlans ? | 08:49 |
| *** dciabrin has joined #tripleo | 08:59 | |
| *** oneswig has joined #tripleo | 08:59 | |
| rnoriega | Hello guys!! | 09:01 |
| openstackgerrit | Carlos Camacho proposed openstack/python-tripleoclient master: Add openstack undercloud backup https://review.openstack.org/466213 | 09:01 |
| rnoriega | ccamacho, shardy, sorry to contact you again, but this is the same as yesterday's patch, but for BGPVPN: https://review.openstack.org/#/c/467124/ | 09:01 |
| rnoriega | easy peasy | 09:01 |
| rnoriega | ccamacho, shardy, just if you have a minute. No need to rush! | 09:01 |
| *** hewbrocca_afk is now known as hewbrocca | 09:02 | |
| *** gbarros has joined #tripleo | 09:03 | |
| *** dparkes has quit IRC | 09:05 | |
| *** gfidente has joined #tripleo | 09:07 | |
| openstackgerrit | Carlos Camacho proposed openstack/python-tripleoclient master: Add openstack undercloud backup https://review.openstack.org/466213 | 09:07 |
| *** ooolpbot has joined #tripleo | 09:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1680195 | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1690132 | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1690373 | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1692503 | 09:10 |
| openstack | Launchpad bug 1680195 in tripleo "Random ovb-ha ping test failures" [Critical,Triaged] | 09:10 |
| *** ooolpbot has quit IRC | 09:10 | |
| openstack | Launchpad bug 1690132 in tripleo "Newton HA failure: unable to push cib, cib_apply_diff failed" [Critical,Triaged] | 09:10 |
| openstack | Launchpad bug 1690373 in tripleo "stable/newton gate-tripleo-ci-centos-7-nonha-multinode-oooq broken" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 09:10 |
| openstack | Launchpad bug 1692503 in tripleo "CI: containers job fails with "oci runtime error..."" [Critical,Triaged] | 09:10 |
| openstackgerrit | Carlos Camacho proposed openstack/python-tripleoclient master: Add openstack undercloud backup https://review.openstack.org/466213 | 09:13 |
| *** ahrechan has joined #tripleo | 09:13 | |
| *** dbecker has quit IRC | 09:13 | |
| ahrechan | hi! Does anyone knows, how to delete old environment using tripleo-quickstart ? | 09:17 |
| marios | ahrechan: there is an option I believe -T all will delete the old env | 09:19 |
| adarazs | ahrechan: by default it removes the current deployment, but -T all cleans up every libvirt bit as well before recreating them. | 09:20 |
| ahrechan | @marios, adarazs: thx a lot! | 09:21 |
| *** anshulbehl has quit IRC | 09:21 | |
| *** gbarros has quit IRC | 09:21 | |
| openstackgerrit | Carlos Camacho proposed openstack/python-tripleoclient master: Add openstack undercloud backup https://review.openstack.org/466213 | 09:22 |
| *** chem has joined #tripleo | 09:26 | |
| *** ckyriakidou has joined #tripleo | 09:27 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Role Specific paramaeter support for neutron-ovs-dpdk-agent service https://review.openstack.org/446093 | 09:33 |
| marios | ack ahrechan thanks adarazs | 09:33 |
| *** zoli is now known as zoli|lunch | 09:34 | |
| *** milan has joined #tripleo | 09:36 | |
| *** hjensas has joined #tripleo | 09:36 | |
| *** dparkes has joined #tripleo | 09:44 | |
| *** dciabrin has quit IRC | 09:49 | |
| *** fzdarsky has joined #tripleo | 09:52 | |
| openstackgerrit | yolanda.robla proposed openstack/tripleo-image-elements master: Add overcloud-secure element https://review.openstack.org/449122 | 09:53 |
| *** adarazs is now known as adarazs_lunch | 09:53 | |
| openstackgerrit | yolanda.robla proposed openstack/tripleo-common master: Add creation of security hardened images https://review.openstack.org/448528 | 09:55 |
| *** links has quit IRC | 09:55 | |
| openstackgerrit | Arx Cruz proposed openstack/tripleo-quickstart-extras master: Send mail tool https://review.openstack.org/423340 | 09:57 |
| *** chem has quit IRC | 09:58 | |
| *** hewbrocca is now known as hewbrocca_afk | 10:01 | |
| openstackgerrit | yolanda.robla proposed openstack/tripleo-image-elements master: Add overcloud-secure element https://review.openstack.org/449122 | 10:02 |
| openstackgerrit | Matthias Runge proposed openstack/tripleo-quickstart master: Adjust handling of autostart https://review.openstack.org/467564 | 10:02 |
| *** dbecker has joined #tripleo | 10:02 | |
| *** anshulbehl has joined #tripleo | 10:05 | |
| openstackgerrit | yolanda.robla proposed openstack/tripleo-image-elements master: Add overcloud-secure element https://review.openstack.org/449122 | 10:05 |
| openstackgerrit | yolanda.robla proposed openstack/tripleo-common master: Add creation of security hardened images https://review.openstack.org/448528 | 10:06 |
| *** links has joined #tripleo | 10:08 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-quickstart-extras master: Add undercloud_heat_public_endpoints switch https://review.openstack.org/467569 | 10:09 |
| *** fzdarsky has quit IRC | 10:09 | |
| *** ooolpbot has joined #tripleo | 10:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1680195 | 10:10 |
| openstack | Launchpad bug 1680195 in tripleo "Random ovb-ha ping test failures" [Critical,Triaged] | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1690132 | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1690373 | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1692503 | 10:10 |
| *** ooolpbot has quit IRC | 10:10 | |
| openstack | Launchpad bug 1690132 in tripleo "Newton HA failure: unable to push cib, cib_apply_diff failed" [Critical,Triaged] | 10:10 |
| openstack | Launchpad bug 1690373 in tripleo "stable/newton gate-tripleo-ci-centos-7-nonha-multinode-oooq broken" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 10:10 |
| openstack | Launchpad bug 1692503 in tripleo "CI: containers job fails with "oci runtime error..."" [Critical,Triaged] | 10:10 |
| bogdando | slagle: please take a look the patch ^^ | 10:11 |
| *** anshulbehl has quit IRC | 10:11 | |
| bogdando | I'm following Pre_Provisioned_Nodes docs, deployed_server docs, and trying to put things together for my deployment RFE :) | 10:11 |
| bogdando | It is still not clear to me what and how needs to be configured by quickstart to address pre-previsioned openstack provider multinode envs... sigh | 10:12 |
| bogdando | some magic needs to happen to create deployed-server stack on undercloud and continue with overcloud deploy | 10:13 |
| bogdando | I've almost managed to master that a dark spell | 10:14 |
| *** gbarros has joined #tripleo | 10:19 | |
| *** anshulbehl has joined #tripleo | 10:20 | |
| *** jroll has joined #tripleo | 10:25 | |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-heat-templates master: Fix race condition between docker-puppet and container steps https://review.openstack.org/467572 | 10:25 |
| jistr | bogdando, sshnaidm, mandre: ^^ i hope that could fix the critical CI bug | 10:25 |
| bogdando | oh, race conditions are lovely | 10:26 |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-pacemaker master: Initial bundle resource support https://review.openstack.org/463170 | 10:26 |
| openstackgerrit | Damien Ciabrini proposed openstack/puppet-tripleo master: Puppet module to deploy RabbitMQ bundle for HA https://review.openstack.org/455189 | 10:26 |
| sshnaidm | jistr, thanks | 10:26 |
| *** dtantsur|lunch is now known as dtantsur | 10:26 | |
| bogdando | jistr: well done, really. How did you come to that tricky thing?.. | 10:27 |
| jistr | bogdando: thanks :) nowadays anytime i see an intermittent problem in the deployment, i'm already on lookout for missing dependency relationships somewhere.. And i noticed in the log that it was container deployment step 1 that failed, and GenerateConfigDeployment was aborted (i.e. unfinished), and realized that the missing memcached config file should have been generated by the GenerateConfigDeployment, so | 10:29 |
| jistr | went to check the relationships on those | 10:29 |
| bogdando | I wish I were able to meditate while looking a heat viz graphs to see such races and missing anchors | 10:30 |
| jistr | hehe | 10:30 |
| *** anshulbehl has quit IRC | 10:31 | |
| *** gbarros has quit IRC | 10:33 | |
| jistr | yea true that could help quite a bit if we were able to scope things right | 10:34 |
| shardy | FWIW I'm hoping this will be much easier to understand when it's just one ansible playbook | 10:34 |
| shardy | vs $many inter-related SoftwareDeployments | 10:34 |
| * shardy rebases patch that does that | 10:35 | |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates master: Convert puppet and docker steps to ansible https://review.openstack.org/462211 | 10:36 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates master: Write md5sum for service config directories https://review.openstack.org/467581 | 10:36 |
| shardy | jistr, bogdando: ^^ reviews welcome :) | 10:37 |
| bogdando | shardy: hope so | 10:37 |
| bogdando | at least we will know a new place to look for missing anchors haha | 10:38 |
| shardy | https://review.openstack.org/#/c/462211/3/docker/deploy-steps-playbook.yaml | 10:38 |
| *** udesale__ has joined #tripleo | 10:38 | |
| *** jkilpatr has quit IRC | 10:38 | |
| shardy | bogdando: haha, I guess my point is there aren't many depends_on, it's just a series of tasks in a list | 10:38 |
| bogdando | right, this is a straightforward and less fragile | 10:39 |
| bogdando | and easier to read | 10:39 |
| shardy | yeah, and easier to debug because you can more easily repeat the steps outside heat | 10:40 |
| *** udesale has quit IRC | 10:41 | |
| gfidente | shardy you mentioned that is running ansible on localhost right? | 10:41 |
| shardy | gfidente: for now, yes | 10:42 |
| gfidente | shardy for now ... leaves open space for not in the future | 10:42 |
| *** anshulbehl has joined #tripleo | 10:42 | |
| *** udesale__ has quit IRC | 10:43 | |
| shardy | gfidente: well, when we have a playbook per role, it would be possible to just write it to swift, and enable running it with the dynamic inventory from the undercloud | 10:43 |
| shardy | gfidente: that's not my current goal though | 10:43 |
| *** udesale has joined #tripleo | 10:44 | |
| gfidente | and ansible is running puppet at this point anyway | 10:44 |
| shardy | gfidente: yes my patch already has ansible running puppet | 10:45 |
| gfidente | ack | 10:45 |
| gfidente | tks | 10:45 |
| *** dsariel has quit IRC | 10:48 | |
| *** hewbrocca_afk is now known as hewbrocca | 10:49 | |
| gfidente | shardy so I am sticking with the workflow / mistral-ansible-actions for now | 10:50 |
| gfidente | but it's open for review inthe future | 10:50 |
| openstackgerrit | Numan Siddique proposed openstack/tripleo-heat-templates master: Pacemaker HA suport for OVN DB servers https://review.openstack.org/436767 | 10:50 |
| shardy | gfidente: ack yup sounds good | 10:50 |
| *** oneswig has quit IRC | 10:51 | |
| *** oneswig has joined #tripleo | 10:52 | |
| *** ansmith has quit IRC | 10:52 | |
| d0ugal | dtantsur: Hey, if you get a chance I'd appreciate input on https://review.openstack.org/#/c/462916/ | 10:54 |
| d0ugal | dtantsur: I had the question (in a PM) if this is just a hack/workaround and the Ironic should better handle the errors. Or if we should retry only on specific errors? | 10:54 |
| dtantsur | d0ugal: ironic-inspector cannot really handle retries. I think the workflows to drive it is the right place for such things. | 10:55 |
| *** oneswig has quit IRC | 10:56 | |
| dtantsur | d0ugal: next, given that 99% of errors are due to timeout (which is usually due to network problems), retrying may be fine | 10:56 |
| *** jkilpatr has joined #tripleo | 10:56 | |
| dtantsur | d0ugal: the only problem I see is that the default timeout is 1 hour. you have to make it lower, if you want to go with retries | 10:56 |
| d0ugal | dtantsur: right, I've only seen timeout errors (and aborts) | 10:56 |
| *** flepied has joined #tripleo | 10:57 | |
| dtantsur | d0ugal: oh, yeah, we should not retry aborts | 10:57 |
| dtantsur | d0ugal: I'd go as far as going: "if we don't get success or abort in %d minutes, restart introspection %d times", where both %d are configurable | 10:58 |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-tripleo master: Puppet module to deploy HAProxy bundle for HA https://review.openstack.org/458832 | 10:58 |
| d0ugal | dtantsur: is the only way to check the error by comparing the error text? | 10:58 |
| dtantsur | d0ugal: in this case we don't need to change the global timeout, and we will be able to provide a --timeout argument in CLI | 10:58 |
| dtantsur | d0ugal: unfortunately, yes | 10:58 |
| d0ugal | dtantsur: right, so I should handle the new timeout in the workflow itself. That does mean we will have two places to configure timeouts, which is unfortunate | 10:59 |
| dtantsur | d0ugal: I see it as a benefit that users can reduce the timeout without reconfiguring the undercloud | 11:00 |
| d0ugal | True, that is a bit nicer | 11:00 |
| b00tcat | if I want to set up a virtual environment with tripleo-quickstart but I already have al my VMs and networks created previously in the physical host - is it possible to *only* set up the undercloud and nothing else? | 11:00 |
| d0ugal | dtantsur: Okay, I'll do some work on these updates. thanks. | 11:00 |
| dtantsur | d0ugal: np, ping me when you're ready for the next review | 11:01 |
| d0ugal | will do! | 11:01 |
| d0ugal | dtantsur: is there a list of possible errors? or should I just do != 'User aborted' (I need to check the exact wording) | 11:02 |
| *** zoli|lunch is now known as zoli | 11:04 | |
| dtantsur | d0ugal: no, not really :( we should have invented error codes, probably, but we haven't | 11:04 |
| *** masco is now known as masco_afk | 11:05 | |
| *** ooolpbot has joined #tripleo | 11:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1680195 | 11:10 |
| openstack | Launchpad bug 1680195 in tripleo "Random ovb-ha ping test failures" [Critical,Triaged] | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1690132 | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1690373 | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1692503 | 11:10 |
| *** ooolpbot has quit IRC | 11:10 | |
| openstack | Launchpad bug 1690132 in tripleo "Newton HA failure: unable to push cib, cib_apply_diff failed" [Critical,Triaged] | 11:10 |
| openstack | Launchpad bug 1690373 in tripleo "stable/newton gate-tripleo-ci-centos-7-nonha-multinode-oooq broken" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 11:10 |
| openstack | Launchpad bug 1692503 in tripleo "CI: containers job fails with "oci runtime error..."" [Critical,In progress] - Assigned to Jiřà Stránský (jistr) | 11:10 |
| d0ugal | dtantsur: so I guess I need to hope this never changes and users don't have translations :( https://github.com/openstack/ironic-inspector/blob/42963fa9f75313f253527baaeda39249a89fe185/ironic_inspector/introspect.py#L216 | 11:11 |
| *** dsariel has joined #tripleo | 11:12 | |
| *** pkovar has joined #tripleo | 11:12 | |
| *** rhallisey has joined #tripleo | 11:13 | |
| dtantsur | d0ugal: yeah, this is quite bad.. I guess we may need a new flag like "aborted". milan wdyt ^^^? | 11:14 |
| *** lucasagomes is now known as lucas-pto | 11:17 | |
| *** abishop has quit IRC | 11:18 | |
| d0ugal | that would work for me, but I guess adding a flag isn't that much easier than adding error codes which might be more generally useful. | 11:18 |
| dtantsur | yeah.. I need to discuss it with other inspector folks, starting with milan who is driving it nowadays | 11:20 |
| d0ugal | makes sense | 11:21 |
| sshnaidm | FYI https://bugs.launchpad.net/tripleo/+bug/1693174 | 11:21 |
| openstack | Launchpad bug 1693174 in tripleo "CI: ovb-ha jobs fail because of lack of memory" [Critical,Triaged] | 11:21 |
| *** liverpooler has quit IRC | 11:22 | |
| dtantsur | oh, not again... | 11:23 |
| *** fragatina has joined #tripleo | 11:28 | |
| *** atoth has joined #tripleo | 11:28 | |
| *** fragatina has quit IRC | 11:29 | |
| jaosorior | sshnaidm: funky, and we even removed mongodb recently from the defaults :/ | 11:29 |
| *** fragatina has joined #tripleo | 11:29 | |
| sshnaidm | it started yesterday about 16.00 UTC | 11:30 |
| *** gbarros has joined #tripleo | 11:30 | |
| *** ffiore has quit IRC | 11:30 | |
| *** pkovar has quit IRC | 11:30 | |
| *** nyechiel has joined #tripleo | 11:32 | |
| *** udesale has quit IRC | 11:34 | |
| *** udesale has joined #tripleo | 11:35 | |
| *** nyechiel has quit IRC | 11:38 | |
| *** adarazs_lunch is now known as adarazs | 11:38 | |
| *** thrash|g0ne is now known as thrash | 11:40 | |
| *** ipsecguy_ is now known as ipsecguy | 11:40 | |
| *** gbarros has quit IRC | 11:42 | |
| mandre | jaosorior: do you know if octavia runs on top of apache? it seems not | 11:44 |
| mandre | jaosorior: my question really is about whether or now we should implement a proxy container for TLS everywhere when containerizing octavia | 11:45 |
| jaosorior | mandre: I have no idea | 11:45 |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates master: Include mongodb in scenario002 job https://review.openstack.org/467353 | 11:46 |
| openstackgerrit | yolanda.robla proposed openstack/tripleo-common master: Add creation of security hardened images https://review.openstack.org/448528 | 11:46 |
| *** Tyrantelf has quit IRC | 11:49 | |
| *** masco_afk is now known as masco | 11:50 | |
| *** Tyrantelf has joined #tripleo | 11:50 | |
| jaosorior | mandre: however, testing it out and developing should be way easier nowadays thanks to hrybacki :D | 11:51 |
| jaosorior | hrybacki: are there any docs or a blog around about using the supplemental node to deploy TLS everywhere? | 11:51 |
| *** gfidente has quit IRC | 11:52 | |
| *** kashyap has joined #tripleo | 11:56 | |
| *** abishop has joined #tripleo | 11:59 | |
| hrybacki | jaosorior mandre (on phone) I do! I'll be in the office within an hour and will get you some links straight away | 12:00 |
| *** bfournie has quit IRC | 12:00 | |
| mandre | jaosorior: ack, thanks, so should we just leave it out for now? | 12:04 |
| openstackgerrit | Honza Pokorny proposed openstack/tripleo-ui master: Add webpack-merge dependency https://review.openstack.org/467602 | 12:05 |
| openstackgerrit | Honza Pokorny proposed openstack/tripleo-ui master: Run minification and tree shaking on build https://review.openstack.org/463824 | 12:05 |
| *** pkovar has joined #tripleo | 12:05 | |
| *** jayg|g0n3 is now known as jayg | 12:06 | |
| dtantsur | folks, could you please check a small docs review https://review.openstack.org/#/c/464656/ ? | 12:06 |
| *** morazi has joined #tripleo | 12:09 | |
| openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart-extras master: Explicitly use bash for pipefail shell blocks https://review.openstack.org/466316 | 12:10 |
| *** ooolpbot has joined #tripleo | 12:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1680195 | 12:10 |
| openstack | Launchpad bug 1680195 in tripleo "Random ovb-ha ping test failures" [Critical,Triaged] | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1690132 | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1690373 | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1692503 | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1693174 | 12:10 |
| *** ooolpbot has quit IRC | 12:10 | |
| openstack | Launchpad bug 1690132 in tripleo "Newton HA failure: unable to push cib, cib_apply_diff failed" [Critical,Triaged] | 12:10 |
| openstack | Launchpad bug 1690373 in tripleo "stable/newton gate-tripleo-ci-centos-7-nonha-multinode-oooq broken" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 12:10 |
| openstack | Launchpad bug 1692503 in tripleo "CI: containers job fails with "oci runtime error..."" [Critical,In progress] - Assigned to Jiřà Stránský (jistr) | 12:10 |
| openstack | Launchpad bug 1693174 in tripleo "CI: ovb-ha jobs fail because of lack of memory" [Critical,Triaged] | 12:10 |
| openstackgerrit | Merged openstack/tripleo-docs master: Document hardware types for ironic in the overcloud https://review.openstack.org/464656 | 12:11 |
| *** trown|outtypewww is now known as trown | 12:15 | |
| *** dciabrin has joined #tripleo | 12:17 | |
| *** dprince has joined #tripleo | 12:18 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: ceilometer-expirer: remove the crontab during upgrade https://review.openstack.org/466096 | 12:23 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: CI test - never merge https://review.openstack.org/461000 | 12:23 |
| * milan reads scroll backs having had a coffee :P | 12:24 | |
| *** mdnadeem has quit IRC | 12:25 | |
| *** gfidente has joined #tripleo | 12:27 | |
| *** gfidente has quit IRC | 12:27 | |
| *** gfidente has joined #tripleo | 12:27 | |
| dtantsur | milan: tl;dr being able to distinguish nodes with failed and aborted introspection. ideally, also distinguish timeout and other failures. | 12:27 |
| milan | dtantsur, I thought about it, we could also handle this by a separate state | 12:28 |
| milan | timeout instead of error | 12:28 |
| dtantsur | milan: maybe a separate field.. otherwise the state machine will blow up | 12:29 |
| dtantsur | and from its point of view, timeout is no different from any other error | 12:29 |
| milan | right but would have the benefit of easy filtering straight from the web api | 12:29 |
| *** kashyap has left #tripleo | 12:29 | |
| dtantsur | a new field is also easy, and won't affect the state machine | 12:30 |
| milan | state=in:timeout or so | 12:30 |
| milan | right | 12:30 |
| milan | maybe a spec? | 12:30 |
| milan | so that folks can also review? | 12:30 |
| dtantsur | I mean, I don't care much how exactly we do it, but I'll probably won't survive another spec :D | 12:30 |
| *** ecerquei_ has joined #tripleo | 12:30 | |
| milan | lol | 12:31 |
| milan | :D | 12:31 |
| milan | OK | 12:31 |
| milan | would be short one ;) | 12:31 |
| milan | :D | 12:31 |
| *** ansmith has joined #tripleo | 12:31 | |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui master: Update code to React Router v4 API https://review.openstack.org/466746 | 12:32 |
| *** rbowen has joined #tripleo | 12:33 | |
| milan | d0ugal, so what is the workflow for you with the introspection? | 12:33 |
| *** rlandy has joined #tripleo | 12:34 | |
| jtomasek | honza: do you plan to change the dev sourcemap as part of the patch https://review.openstack.org/#/c/463824/ or is it ready for review? | 12:35 |
| honza | jtomasek: crap, i forgot about that --- one sec | 12:35 |
| *** mdnadeem has joined #tripleo | 12:36 | |
| jtomasek | honza: thanks! | 12:36 |
| openstackgerrit | Carlos Camacho proposed openstack/python-tripleoclient master: Add openstack undercloud backup https://review.openstack.org/466213 | 12:36 |
| openstackgerrit | Honza Pokorny proposed openstack/tripleo-ui master: Run minification and tree shaking on build https://review.openstack.org/463824 | 12:36 |
| honza | jtomasek: ready, sorry about that | 12:37 |
| *** ratailor has quit IRC | 12:37 | |
| openstackgerrit | Florian Fuchs proposed openstack/tripleo-validations master: Add stack health check validation https://review.openstack.org/467607 | 12:38 |
| *** gbarros has joined #tripleo | 12:39 | |
| *** ffiore has joined #tripleo | 12:40 | |
| *** bfournie has joined #tripleo | 12:41 | |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-quickstart-extras master: Support composable roles for deploying overcloud https://review.openstack.org/455719 | 12:42 |
| *** jpena is now known as jpena|lunch | 12:44 | |
| * milan sort of feels timed-out, aborted and generic error are all different states, e.g timed-out with a message doesn't make much sense, neither aborted does... my 2C | 12:44 | |
| dtantsur | milan: we'll still have a message in the "error" field, but ok. | 12:45 |
| milan | dtantsur, right the trick would be to change the state machine transitions there, code need not be so much affected; question what's more straightforward for an user | 12:46 |
| * milan wouldn't mind either too much, so if d0ugal has an opinion or other folks do | 12:46 | |
| milan | so a spec may be the right way to go | 12:47 |
| * milan OK to write one | 12:47 | |
| dtantsur | yes, assuming people are going to review it | 12:47 |
| milan | dtantsur, that, lol, yeah :D | 12:48 |
| *** zoli is now known as zoli|gone | 12:48 | |
| dtantsur | if it's again the four of us reviewing it, maybe we can just decide it on IRC :D | 12:48 |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-quickstart-extras master: Remove overcloud_services default value https://review.openstack.org/467614 | 12:50 |
| *** gbarros has quit IRC | 12:50 | |
| *** Goneri has joined #tripleo | 12:52 | |
| *** michapma_dsk has quit IRC | 12:55 | |
| *** liverpooler has joined #tripleo | 12:55 | |
| *** trown is now known as trown|brb | 12:56 | |
| openstackgerrit | Marios Andreou proposed openstack/instack-undercloud stable/ocata: Add unit test and release note for tuskar package removal https://review.openstack.org/467617 | 12:57 |
| *** psahoo has quit IRC | 12:58 | |
| openstackgerrit | Merged openstack/tripleo-common stable/ocata: Updated from global requirements https://review.openstack.org/464892 | 12:58 |
| *** tzumainn has joined #tripleo | 12:58 | |
| *** nyechiel has joined #tripleo | 12:59 | |
| *** pkovar has quit IRC | 13:00 | |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-heat-templates master: Remove duplicate PreConfig definition https://review.openstack.org/467618 | 13:02 |
| *** pkovar has joined #tripleo | 13:04 | |
| hrybacki | mandre: here is a blog post that I made (prior to patches landing yesterday) https://hairycode.org/2017/05/08/standing-up-openstack-with-freeipa-and-tls-everywhere-using-tripleo-quickstart/ Now that they have landed you can basically ignore the sections related to checking out the patches though. I'll be updating this post with more details soon | 13:05 |
| *** dsariel has quit IRC | 13:06 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Fix failures in prepare ovb servers scripts https://review.openstack.org/467620 | 13:06 |
| hrybacki | mandre: are you wanting to use OOOQ to provision/deploy the FreeIPA server or use an existing FreeIPA server in a new OOOQ deployment? | 13:06 |
| *** amoralej is now known as amoralej|lunch | 13:07 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates master: TEST ONLY: Check telemetry container upgrades https://review.openstack.org/467621 | 13:08 |
| *** ooolpbot has joined #tripleo | 13:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1680195 | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1690132 | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1690373 | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1692503 | 13:10 |
| openstack | Launchpad bug 1680195 in tripleo "Random ovb-ha ping test failures" [Critical,Triaged] | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1693174 | 13:10 |
| *** ooolpbot has quit IRC | 13:10 | |
| openstack | Launchpad bug 1690132 in tripleo "Newton HA failure: unable to push cib, cib_apply_diff failed" [Critical,Triaged] | 13:10 |
| openstack | Launchpad bug 1690373 in tripleo "stable/newton gate-tripleo-ci-centos-7-nonha-multinode-oooq broken" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 13:10 |
| openstack | Launchpad bug 1692503 in tripleo "CI: containers job fails with "oci runtime error..."" [Critical,In progress] - Assigned to Jiřà Stránský (jistr) | 13:10 |
| openstack | Launchpad bug 1693174 in tripleo "CI: ovb-ha jobs fail because of lack of memory" [Critical,Triaged] | 13:10 |
| *** pradk has quit IRC | 13:11 | |
| *** rbrady has quit IRC | 13:13 | |
| openstackgerrit | Carlos Camacho proposed openstack/python-tripleoclient master: Add openstack undercloud backup https://review.openstack.org/466213 | 13:14 |
| *** twi99y has joined #tripleo | 13:15 | |
| *** trown|brb is now known as trown | 13:16 | |
| twi99y | . /nick eqvist | 13:17 |
| *** twi99y is now known as eqvist | 13:18 | |
| *** ebarrera_ has joined #tripleo | 13:18 | |
| *** egafford has joined #tripleo | 13:20 | |
| sshnaidm | trown, adarazs, bnemec, EmilienM could you please take a look: https://review.openstack.org/467620 it fixes bugs in current ovb scripts, CI jobs has no much value there, it's used only when creating ovb CI servers | 13:21 |
| *** liverpooler has quit IRC | 13:22 | |
| *** liverpooler has joined #tripleo | 13:22 | |
| *** salmankhan has quit IRC | 13:22 | |
| adarazs | sshnaidm: looks okay. | 13:23 |
| *** salmankhan has joined #tripleo | 13:25 | |
| openstackgerrit | Carlos Camacho proposed openstack/python-tripleoclient master: Add openstack undercloud backup https://review.openstack.org/466213 | 13:26 |
| *** ffiore has quit IRC | 13:26 | |
| *** jprovazn has quit IRC | 13:27 | |
| *** rbrady has joined #tripleo | 13:27 | |
| trown | ping slagle, does the minimal bootstrap script not work on newton? https://github.com/openstack-infra/tripleo-ci/blob/master/toci_gate_test-oooq.sh#L222 | 13:30 |
| *** prateek__ has quit IRC | 13:30 | |
| *** ykarel has quit IRC | 13:30 | |
| trown | slagle: I think the install of tripleo-image-elements on the overcloud nodes via instack-undercloud dep is breaking stable/newton jobs | 13:30 |
| trown | https://github.com/openstack-infra/tripleo-ci/blob/master/scripts/bootstrap-overcloud-full.sh#L36 | 13:30 |
| *** ebarrera has quit IRC | 13:31 | |
| *** ebarrera has joined #tripleo | 13:31 | |
| *** mcornea has quit IRC | 13:32 | |
| *** lblanchard has joined #tripleo | 13:32 | |
| *** mcornea has joined #tripleo | 13:32 | |
| *** lblanchard has quit IRC | 13:33 | |
| *** ebarrera_ has quit IRC | 13:33 | |
| *** lblanchard has joined #tripleo | 13:33 | |
| *** eck`gone is now known as eck` | 13:34 | |
| slagle | trown: it does not work on newton, so we use the full bootstrap via instack that requires t-i-e | 13:34 |
| slagle | is this what bandini/EmilienM where chasing down last week with the %post script in t-i-e? | 13:35 |
| slagle | there was a packaging patch to remove that in newton | 13:35 |
| trown | slagle: if we remove tie after we use it to construct our package list would that work? | 13:36 |
| trown | slagle: ya trying to solve V | 13:36 |
| trown | https://bugs.launchpad.net/tripleo/+bug/1690373 | 13:36 |
| openstack | Launchpad bug 1690373 in tripleo "stable/newton gate-tripleo-ci-centos-7-nonha-multinode-oooq broken" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 13:36 |
| *** pkovar has quit IRC | 13:36 | |
| slagle | trown: yea, removing t-i-e at the end of bootstrap-overcloud-full.sh ought to be fine. not sure if it will fix the actual issue though | 13:37 |
| slagle | but we shouldn't use the packaged files for anything else after the bootstrap is done | 13:37 |
| trown | slagle: k, will put up a wip to try that while looking for the packaging change | 13:37 |
| trown | thanks | 13:37 |
| *** nyechiel has quit IRC | 13:39 | |
| *** jaganathan_ has quit IRC | 13:39 | |
| *** ffiore has joined #tripleo | 13:40 | |
| openstackgerrit | Carlos Camacho proposed openstack/python-tripleoclient master: Add openstack undercloud backup https://review.openstack.org/466213 | 13:41 |
| openstackgerrit | Alan Bishop proposed openstack/tripleo-common master: Derive Params (part 2): Get services and feature list https://review.openstack.org/456471 | 13:41 |
| openstackgerrit | John Trowbridge proposed openstack-infra/tripleo-ci master: [multinode] Remove t-i-e from the overcloud after bootstrap https://review.openstack.org/467627 | 13:43 |
| hrybacki | mandre: here is a script with my workflow: https://gist.github.com/HarryRybacki/6cf1e6bc32085dd358365f44267f7188 | 13:43 |
| *** pkovar has joined #tripleo | 13:44 | |
| *** jaganathan has joined #tripleo | 13:44 | |
| *** jpena|lunch is now known as jpena | 13:45 | |
| *** amoralej|lunch is now known as amoralej | 13:46 | |
| *** gbarros has joined #tripleo | 13:47 | |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-heat-templates master: Modify PreNetworkConfig config inline with role-specific parameters https://review.openstack.org/447051 | 13:48 |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-heat-templates master: WIP: Enable DPDK on boot using PreNetworkConfig https://review.openstack.org/467490 | 13:48 |
| openstackgerrit | zgdeng proposed openstack/diskimage-builder master: WIP: Add zip element as s390 architecture bootloader https://review.openstack.org/443548 | 13:48 |
| *** oneswig has joined #tripleo | 13:49 | |
| *** zoli|gone is now known as zoli|wfh | 13:50 | |
| *** Vijayendra has quit IRC | 13:51 | |
| *** skramaja has quit IRC | 13:53 | |
| *** yamahata has joined #tripleo | 13:53 | |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates master: Docker service for Cinder Volume https://review.openstack.org/466564 | 13:54 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates master: Docker services for Cinder Backup https://review.openstack.org/457820 | 13:54 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates master: Docker services for Cinder Api and Scheduler https://review.openstack.org/457011 | 13:54 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates master: Move iscsid to a container https://review.openstack.org/462538 | 13:54 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates master: Containerize multipathd https://review.openstack.org/465989 | 13:54 |
| *** oneswig_ has joined #tripleo | 13:55 | |
| openstackgerrit | Merged openstack/tripleo-common master: Add horizon container https://review.openstack.org/458027 | 13:56 |
| *** prateek has joined #tripleo | 13:57 | |
| openstackgerrit | Carlos Camacho proposed openstack/python-tripleoclient master: Add openstack undercloud backup https://review.openstack.org/466213 | 13:57 |
| mandre | hrybacki: nice, is it already all integrated in quickstart? | 13:57 |
| hrybacki | mandre: as of yesterday it has, yep! | 13:58 |
| *** oneswig has quit IRC | 13:58 | |
| *** gbarros has quit IRC | 13:58 | |
| mandre | wonderful | 13:59 |
| *** Vijayendra has joined #tripleo | 13:59 | |
| *** gbarros has joined #tripleo | 14:00 | |
| jaosorior | bandini, jistr: if you guys are interested in trying out TLS everywhere. check hrybacki's link above ^^ | 14:01 |
| *** jpena is now known as jpena|brb | 14:02 | |
| openstackgerrit | Carlos Camacho proposed openstack/python-tripleoclient master: Add openstack undercloud backup https://review.openstack.org/466213 | 14:03 |
| trozet | dprince: if i want to modify the neutron api container with some more packages, I can use https://github.com/openstack/tripleo-common/blob/master/container-images/tripleo_kolla_template_overrides.j2 | 14:03 |
| trozet | dprince: is that correct? | 14:03 |
| openstackgerrit | Merged openstack/tripleo-quickstart master: Set explicit disabling tempest in featureset https://review.openstack.org/466805 | 14:04 |
| trozet | flaper87: maybe you know ^^ | 14:04 |
| *** jaganathan has quit IRC | 14:04 | |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Set explicit exclusion of mitaka,liberty https://review.openstack.org/466730 | 14:06 |
| *** jmelvin has joined #tripleo | 14:06 | |
| prateek | trown, can you please look into https://review.openstack.org/#/c/465536/ ? | 14:06 |
| *** dsariel has joined #tripleo | 14:08 | |
| *** eqvist has quit IRC | 14:08 | |
| trown | sshnaidm: no job here actually runs tempest ya? https://review.openstack.org/465536 | 14:08 |
| sshnaidm | trown, yep, I tested this in different patch with tempest: https://review.openstack.org/#/c/466830/ | 14:09 |
| *** jlinkes has quit IRC | 14:10 | |
| *** ooolpbot has joined #tripleo | 14:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1680195 | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1690132 | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1690373 | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1692503 | 14:10 |
| openstack | Launchpad bug 1680195 in tripleo "Random ovb-ha ping test failures" [Critical,Triaged] | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1693174 | 14:10 |
| *** ooolpbot has quit IRC | 14:10 | |
| openstack | Launchpad bug 1690132 in tripleo "Newton HA failure: unable to push cib, cib_apply_diff failed" [Critical,Triaged] | 14:10 |
| openstack | Launchpad bug 1690373 in tripleo "stable/newton gate-tripleo-ci-centos-7-nonha-multinode-oooq broken" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 14:10 |
| openstack | Launchpad bug 1692503 in tripleo "CI: containers job fails with "oci runtime error..."" [Critical,In progress] - Assigned to Jiřà Stránský (jistr) | 14:10 |
| openstack | Launchpad bug 1693174 in tripleo "CI: ovb-ha jobs fail because of lack of memory" [Critical,Triaged] | 14:10 |
| *** twi99y has joined #tripleo | 14:11 | |
| trown | sshnaidm: it fails on that test patch :P | 14:11 |
| openstackgerrit | Carlos Camacho proposed openstack/python-tripleoclient master: Add openstack undercloud backup https://review.openstack.org/466213 | 14:11 |
| sshnaidm | trown, it just doesn't have a time :) | 14:12 |
| sshnaidm | trown, but it starts ok, just not all tests finish in time | 14:12 |
| sshnaidm | trown, and ocata has time.. that gives us a food for thought | 14:12 |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-pacemaker master: Release 0.6.0 https://review.openstack.org/467630 | 14:13 |
| sshnaidm | every next release seems to take more and more time | 14:13 |
| trown | ya... that has been the trend for a couple years now | 14:13 |
| EmilienM | bandini: releasing puppet-pacemaker today | 14:13 |
| EmilienM | bandini: any last minute change? | 14:13 |
| openstackgerrit | Andy Smith proposed openstack/puppet-tripleo master: Create qdr HA mesh https://review.openstack.org/448279 | 14:14 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Fix race condition between docker-puppet and container steps https://review.openstack.org/467572 | 14:14 |
| *** oneswig_ has quit IRC | 14:16 | |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-heat-templates master: Standardize example role definitions https://review.openstack.org/445687 | 14:17 |
| bandini | EmilienM: sweet. nope not that I am aware of! (Thanks so much, btw) | 14:17 |
| EmilienM | bandini: can you /join #puppet-openstack plz? | 14:17 |
| EmilienM | bandini: nevermind | 14:18 |
| EmilienM | bandini: we figured :D | 14:18 |
| bandini | :D | 14:18 |
| bandini | let me know if needed | 14:18 |
| bandini | am in a call, but around | 14:18 |
| bandini | jaosorior: nice, noted | 14:18 |
| *** pradk has joined #tripleo | 14:23 | |
| *** prateek has quit IRC | 14:25 | |
| *** egafford has quit IRC | 14:26 | |
| *** egafford1 has joined #tripleo | 14:26 | |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-pacemaker master: Release 0.6.0 https://review.openstack.org/467630 | 14:30 |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-pacemaker master: Update tox configuration https://review.openstack.org/467638 | 14:30 |
| *** mdnadeem has quit IRC | 14:30 | |
| EmilienM | mwhahaha: ^ plz | 14:30 |
| *** egafford1 has quit IRC | 14:30 | |
| *** atheurer has quit IRC | 14:35 | |
| *** pchavva has joined #tripleo | 14:35 | |
| *** chlong has joined #tripleo | 14:38 | |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-pacemaker master: Update tox configuration https://review.openstack.org/467638 | 14:38 |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-pacemaker master: Release 0.6.0 https://review.openstack.org/467630 | 14:38 |
| *** yamahata has quit IRC | 14:39 | |
| *** dsariel has quit IRC | 14:41 | |
| EmilienM | sshnaidm: https://bugs.launchpad.net/tripleo/+bug/1693174 is random or persistent? it looks persistent to me | 14:42 |
| openstack | Launchpad bug 1693174 in tripleo "CI: ovb-ha jobs fail because of lack of memory" [Critical,Triaged] | 14:42 |
| EmilienM | sshnaidm: I'll investigate it | 14:43 |
| *** ccamacho is now known as ccamacho|brb | 14:43 | |
| sshnaidm | EmilienM, I blame ceilometer | 14:43 |
| trown | EmilienM: ya I kicked off an HA run on libvirt to see if it reproduces there | 14:43 |
| EmilienM | I'm looking at logstash | 14:44 |
| bandini | dprince: did you have a review up for the cinder-volume stuff? we wanted to get started on that | 14:44 |
| EmilienM | IMHO we should remove telemetry from HA job | 14:44 |
| EmilienM | we already have coverage in scenario001 | 14:44 |
| bandini | EmilienM: +1 | 14:44 |
| sshnaidm | EmilienM, it takes 4.5 percents of memory on controller, dump logs with high resolution and seems like don't succeed to connect to gnocchi and takes a lot of resources | 14:44 |
| sshnaidm | EmilienM, big +1 | 14:45 |
| *** ykarel has joined #tripleo | 14:45 | |
| sshnaidm | EmilienM, and it's not first time it happens.. | 14:45 |
| EmilienM | I'm looking at logstash now to see when it started exactly | 14:45 |
| *** itlinux has joined #tripleo | 14:45 | |
| sshnaidm | EmilienM, yesterday about 15:30 UTC | 14:47 |
| trown | I am still confused how it could affect gates and not just promote | 14:47 |
| sshnaidm | this is the first job that got memory fail: http://logs.openstack.org/51/464951/7/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha-oooq/c5247c7/console.html | 14:47 |
| EmilienM | sshnaidm: it's weird, the rpm -qa is not sorted anymore, see http://logs.openstack.org/51/464951/7/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha-oooq/c5247c7/logs/overcloud-controller-0/var/log/host_info.txt.gz | 14:48 |
| dprince | bandini: here https://etherpad.openstack.org/p/tripleo-containers-cinder | 14:50 |
| sshnaidm | EmilienM, we can pass it through sort | 14:50 |
| dprince | bandini: pair in an hour or so? | 14:50 |
| EmilienM | sshnaidm: we used to do it | 14:50 |
| dprince | bandini: would love to help push this over the line.... | 14:50 |
| *** atheurer has joined #tripleo | 14:50 | |
| EmilienM | sshnaidm: I got it | 14:51 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates master: Convert puppet and docker steps to ansible https://review.openstack.org/462211 | 14:51 |
| *** dprince has quit IRC | 14:51 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart-extras master: get_host_info: sort rpm -qa https://review.openstack.org/467648 | 14:51 |
| EmilienM | sshnaidm: ^ | 14:51 |
| sshnaidm | EmilienM, it's new, but wasn't copied to collect-logs | 14:52 |
| sshnaidm | EmilienM, so maybe nova and neutorn stuff too: https://github.com/openstack-infra/tripleo-ci/commit/846bd480a5b497e33ecc4856f08d7a7b751fbc5f | 14:53 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates master: Convert puppet and docker steps to ansible https://review.openstack.org/462211 | 14:55 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates master: Write md5sum for service config directories https://review.openstack.org/467581 | 14:55 |
| EmilienM | sshnaidm, trown: I think I found out why HA is now failing | 14:57 |
| EmilienM | python2-eventlet-0.18.4-2.el7.noarch was upgraded to python2-eventlet-0.20.1-1.el7.noarch | 14:57 |
| EmilienM | we had the EXACT same issue in Puppet CI | 14:58 |
| EmilienM | amoralej fixed it by adding more swap | 14:58 |
| EmilienM | I'm pretty sure it's related | 14:58 |
| bnemec | Boo swap | 14:58 |
| *** jpena|brb is now known as jpena | 14:58 | |
| trown | ah that would make sense why it snuck past promote | 14:58 |
| amoralej | mmm, let me check logs | 14:58 |
| trown | we dont promote the base packages (but probably should) | 14:58 |
| EmilienM | trown: see https://review.openstack.org/#/c/467129/ | 14:59 |
| EmilienM | I think it's not Telemetry but eventlet | 14:59 |
| mwhahaha | do we even enable swap in our ci? | 15:00 |
| *** ykarel has quit IRC | 15:01 | |
| *** ykarel has joined #tripleo | 15:01 | |
| sshnaidm | EmilienM, when was it updated? | 15:01 |
| amoralej | eventlet was updated yesterday sshnaidm | 15:02 |
| amoralej | for master only | 15:02 |
| sshnaidm | I wonder if can switch off telemetry and not increase swap | 15:02 |
| *** bfournie has quit IRC | 15:03 | |
| *** bfournie has joined #tripleo | 15:03 | |
| EmilienM | sshnaidm: telemetry doesn't use eventlet afik | 15:03 |
| sshnaidm | Because we never had it on overcloud nodes, either I miss something | 15:03 |
| sshnaidm | EmilienM, no, it just takes enormous memory | 15:04 |
| *** atheurer_ has joined #tripleo | 15:04 | |
| *** dprince has joined #tripleo | 15:05 | |
| *** atheurer has quit IRC | 15:05 | |
| *** jlinkes has joined #tripleo | 15:08 | |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-tripleo master: Puppet module to deploy Redis bundle for HA https://review.openstack.org/464951 | 15:09 |
| openstackgerrit | David Moreau Simard proposed openstack/tripleo-common master: Test container images from trunk.registry.rdoproject.org https://review.openstack.org/467651 | 15:09 |
| *** ooolpbot has joined #tripleo | 15:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1680195 | 15:10 |
| openstack | Launchpad bug 1680195 in tripleo "Random ovb-ha ping test failures" [Critical,Triaged] | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1690132 | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1690373 | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1693174 | 15:10 |
| *** ooolpbot has quit IRC | 15:10 | |
| openstack | Launchpad bug 1690132 in tripleo "Newton HA failure: unable to push cib, cib_apply_diff failed" [Critical,Triaged] | 15:10 |
| openstack | Launchpad bug 1690373 in tripleo "stable/newton gate-tripleo-ci-centos-7-nonha-multinode-oooq broken" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 15:10 |
| openstack | Launchpad bug 1693174 in tripleo "CI: ovb-ha jobs fail because of lack of memory" [Critical,Triaged] | 15:10 |
| *** dmarlin has joined #tripleo | 15:10 | |
| openstackgerrit | Merged openstack/puppet-pacemaker master: Update tox configuration https://review.openstack.org/467638 | 15:12 |
| openstackgerrit | Merged openstack/puppet-pacemaker master: Release 0.6.0 https://review.openstack.org/467630 | 15:13 |
| sshnaidm | bnemec, is it ok? https://review.openstack.org/#/c/467620/ | 15:13 |
| *** jprovazn has joined #tripleo | 15:16 | |
| *** rcernin has quit IRC | 15:18 | |
| *** dparkes has quit IRC | 15:19 | |
| *** hewbrocca is now known as hewbrocca_afk | 15:20 | |
| fultonj | gfidente colonwq we were talking about deploying a containerized overcloud which uses an external ceph cluster | 15:20 |
| fultonj | we had -e environments/docker.yaml -e environments/puppet-ceph-external.yaml | 15:21 |
| gfidente | fultonj yeah we need to mount /etc/ceph in those containers which use ceph | 15:21 |
| fultonj | within the compute contianer we found /etc/ceph not mounted | 15:21 |
| fultonj | right | 15:21 |
| gfidente | I am not sure if we can describe this in the CephExternal service or if it has to go in the NovaCompute service | 15:22 |
| *** fragatina has quit IRC | 15:22 | |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-tripleo master: Puppet module to deploy Redis bundle for HA https://review.openstack.org/464951 | 15:22 |
| openstackgerrit | Merged openstack/instack-undercloud stable/ocata: Clarify the enable_cinder option description https://review.openstack.org/467513 | 15:23 |
| gfidente | maybe we can use call CephClient from within NovaCompute to merge the outputs | 15:23 |
| gfidente | shardy ^^ ? | 15:23 |
| gfidente | would that work when CephClient is None ? | 15:23 |
| *** shardy has quit IRC | 15:23 | |
| gfidente | auch! :D | 15:23 |
| openstackgerrit | Merged openstack/instack-undercloud stable/ocata: Add unit test and release note for tuskar package removal https://review.openstack.org/467617 | 15:23 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Create priority order for configuration files https://review.openstack.org/465632 | 15:29 |
| *** egafford has joined #tripleo | 15:30 | |
| fultonj | gfidente: so we generage ceph.conf on the host and have it bound mount by adding lines to https://github.com/openstack/tripleo-heat-templates/blob/master/docker/services/nova-libvirt.yaml#L101 and https://github.com/openstack/tripleo-heat-templates/blob/master/docker/services/nova-compute.yaml#L96 | 15:32 |
| openstackgerrit | Tim Rozet proposed openstack/tripleo-heat-templates master: Adds service for OVS and enables ODL DPDK deployments https://review.openstack.org/419648 | 15:32 |
| fultonj | generate* | 15:32 |
| *** itlinux has quit IRC | 15:33 | |
| *** tvignaud is now known as tvignaud_pto | 15:34 | |
| *** yamahata has joined #tripleo | 15:35 | |
| *** twi99y has quit IRC | 15:35 | |
| gfidente | fultonj that's an option yes | 15:35 |
| *** itlinux has joined #tripleo | 15:36 | |
| *** twi99y has joined #tripleo | 15:39 | |
| *** agurenko has quit IRC | 15:39 | |
| *** atheurer_ has quit IRC | 15:41 | |
| fultonj | gfidente: so colonwq suggested updating this https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/services/nova-compute.yaml#L106 | 15:41 |
| fultonj | to add CephClient | 15:42 |
| gfidente | right that is the other option | 15:43 |
| gfidente | I guess if sourcing ../CephClient.yaml works | 15:43 |
| gfidente | that is best, we'd have to define which directories have to be bind mounted in CephExternal | 15:44 |
| gfidente | (or CephClient, if we want to use a shared service) | 15:44 |
| dmsimard | Slower: mandre just bursted my bubble | 15:45 |
| dmsimard | Slower: we have another problematic use case | 15:45 |
| Slower | uh oh :) | 15:45 |
| dmsimard | Sort of a chicken-and-egg when adding a new image to https://github.com/openstack/tripleo-common/blob/master/container-images/overcloud_containers.yaml | 15:46 |
| dmsimard | Today I understand that we cheat by uploading the image to Docker Hub first and then adding it to the file so it works | 15:46 |
| dmsimard | But we're trying to transition the image building to the CI pipeline/private registry | 15:47 |
| dmsimard | So I mean technically we could still cheat and build this one image ahead of time but it's not something I've thought about | 15:48 |
| *** aufi has quit IRC | 15:49 | |
| fultonj | gfidente: colonwq want me to test the CephClient.yaml change? | 15:49 |
| fultonj | colonwq: do you want to put up a WIP of it? | 15:49 |
| colonwq | fultonj, I can make a WIP patch for this change | 15:50 |
| fultonj | colonwq: ok, thanks let me know | 15:50 |
| Slower | dmsimard: mmm, I see | 15:50 |
| Slower | dmsimard: well there's a few instances where we need to rebuild all the containers. actually I would say they're all in tripleo-common though | 15:51 |
| Slower | dmsimard: if the kolla overrides or container list yaml changes we should rebuild them all.. | 15:51 |
| Slower | dprince: what did we decide last call about generating the to-be-built container list from overcloud-containers? | 15:51 |
| dmsimard | Slower: yikes, in the gate ? That'd be nuts | 15:52 |
| dmsimard | Slower: I don't disagree but that's going to bump the job time a bunch | 15:52 |
| *** fragatina has joined #tripleo | 15:52 | |
| ebarrera | slagle, what documentation are you refferin on https://youtu.be/fspnjD-1DNI min 27:50 ? | 15:52 |
| *** itlinux has quit IRC | 15:52 | |
| dmsimard | Slower: for image list we can probably just rebuild the delta.. for the overrides, yes, we'd need to rebuild everything | 15:53 |
| Slower | dmsimard: it doesn't happen too often, but probably we should try to rebuild with promoted rpms and have a new set of promoted containers asap when that happens? | 15:53 |
| *** mcornea has quit IRC | 15:53 | |
| dmsimard | brb | 15:53 |
| Slower | dmsimard: if people remember to add it to the build list before they add templates to start the service it'll all be fine :) | 15:53 |
| Slower | dmsimard: which I think people would get actually | 15:54 |
| Slower | I was just talking to trozet who's adding a service and he just did that but via dockerhub | 15:54 |
| *** panda is now known as panda|bbl | 15:55 | |
| openstackgerrit | Keith Schincke proposed openstack/tripleo-heat-templates master: WIP Update NovaCompute to include CephClient resrouce. https://review.openstack.org/467665 | 15:56 |
| *** bfournie has quit IRC | 15:56 | |
| colonwq | fultonj, ^^^ | 15:56 |
| dprince | Slower: re. the overcloud-containers.yaml I think we decided that people may want to have additions to that file | 15:56 |
| *** atheurer has joined #tripleo | 15:56 | |
| dprince | Slower: and that is okay, but that we'd still maintain it as a static file perhaps for now | 15:56 |
| fultonj | colonwq: thanks, let me try that... | 15:57 |
| dprince | Slower: I'm on the fence a bit on this one. I could see us auto-generating it as well | 15:57 |
| Slower | dprince: so my concern was that we were talkinag about generating the one in tripleo-common | 15:57 |
| Slower | dprince: but that's not on the table at all is that right? | 15:57 |
| yolanda | hi, can i get reviews on https://review.openstack.org/449122 and https://review.openstack.org/448528? thx | 15:57 |
| dprince | Slower: mandre had a patch for autogenerating this file already I think | 15:57 |
| Slower | dprince: because it sounds like we need to have a step procedure for adding new containers.. it needs to be added there first and let RDO build it before we can fire it up | 15:57 |
| *** yamahata has quit IRC | 16:00 | |
| openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart-extras master: GATE CHECK https://review.openstack.org/430277 | 16:00 |
| gfidente | colonwq the different output sections of the paren services are inherited by the "caller" so that should work fine | 16:00 |
| gfidente | *paret | 16:00 |
| gfidente | oh, parent | 16:00 |
| colonwq | gfidente, someone needs a vacation. :) | 16:01 |
| *** bfournie has joined #tripleo | 16:01 | |
| amoralej | EmilienM, sshnaidm, re. memory increase with eventlet-0.20.1, real problem is in python-idna, i hope we can bump it soon but it may take a while | 16:01 |
| gfidente | colonwq exactly | 16:01 |
| *** fragatina has quit IRC | 16:01 | |
| trown | amoralej: any chance we could revert the bump to eventlet in the meantime? what required that bump? | 16:02 |
| amoralej | it's required for mysql TLS, which is one of the features for pike | 16:02 |
| EmilienM | amoralej: why taking a while? | 16:03 |
| EmilienM | python-idna deps? | 16:03 |
| amoralej | because we use it from centos base | 16:03 |
| amoralej | python-idna is now in centos base repo, i've requested to get it updated in https://bugzilla.redhat.com/show_bug.cgi?id=1455078 | 16:03 |
| openstack | bugzilla.redhat.com bug 1455078 in python-idna "Update python-idna to version 2.5" [Unspecified,New] - Assigned to npmccallum | 16:03 |
| amoralej | but i guess we'll have to include it in rdo deps | 16:03 |
| amoralej | because updates in base take long, specially rebases | 16:04 |
| *** michapma_dsk has joined #tripleo | 16:04 | |
| *** fragatina has joined #tripleo | 16:05 | |
| amoralej | EmilienM, if you are getting oom now that means probably you were close before new eventlet, maybe adding some swap is not a bad idea in any case | 16:05 |
| *** fragatina has quit IRC | 16:06 | |
| *** fragatina has joined #tripleo | 16:06 | |
| *** ykarel has quit IRC | 16:06 | |
| trown | I would prefer we just disable telemetry in HA job | 16:06 |
| sshnaidm | amoralej, we didn't add swap before, I'd prefer not to start.. | 16:07 |
| trown | that seems better than adding swap | 16:07 |
| amoralej | ok | 16:07 |
| sshnaidm | trown, +1 | 16:07 |
| trown | then when python-idna fix hits we will just be that much faster :) | 16:07 |
| amoralej | i'll keep you updated about any progress with it | 16:07 |
| EmilienM | +1 for removing telemetry from ovb-ha | 16:08 |
| EmilienM | go for it | 16:08 |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Add support for new parameters in python-tempestconf https://review.openstack.org/465536 | 16:08 |
| EmilienM | pradk: ^ thought? | 16:08 |
| pradk | EmilienM, yea we cover it in scenario jobs.. so fine by me | 16:09 |
| trown | pradk: specifically... is there any need for coverage of telemetry specific to ha | 16:09 |
| trown | would we lose ssl coverage in that case though? | 16:10 |
| *** ooolpbot has joined #tripleo | 16:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1680195 | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1690132 | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1690373 | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1693174 | 16:10 |
| openstack | Launchpad bug 1680195 in tripleo "Random ovb-ha ping test failures" [Critical,Triaged] | 16:10 |
| *** ooolpbot has quit IRC | 16:10 | |
| EmilienM | and ipv6 | 16:10 |
| openstack | Launchpad bug 1690132 in tripleo "Newton HA failure: unable to push cib, cib_apply_diff failed" [Critical,Triaged] | 16:10 |
| openstack | Launchpad bug 1690373 in tripleo "stable/newton gate-tripleo-ci-centos-7-nonha-multinode-oooq broken" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 16:10 |
| openstack | Launchpad bug 1693174 in tripleo "CI: ovb-ha jobs fail because of lack of memory" [Critical,Triaged] | 16:10 |
| *** brault has quit IRC | 16:10 | |
| trown | working on it ooolp | 16:10 |
| pradk | trown, it would be nice to have .. is this due to the load? | 16:10 |
| trown | pradk: ya, not telemetry fault, just trying to figure out how to reduce the load instead of adding swap | 16:11 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-quickstart-extras master: Document overcloud deployment methods https://review.openstack.org/467668 | 16:11 |
| EmilienM | imho ovb-ha jobs should only test basic compute: keystone / glance / nova / neutron / cinder | 16:12 |
| *** anshulbehl has quit IRC | 16:12 | |
| pradk | understand | 16:12 |
| bnemec | EmilienM: Except that job provides coverage of things that could be useful to other services too (net-iso, ha). | 16:15 |
| bnemec | It's actually a hole in our test coverage that not everything gets deployed in an ha job at some point. | 16:15 |
| bnemec | I'm not saying that's realistic, but I also don't think our goal should be to strip down the ha job to the bare minimum either. | 16:16 |
| *** lmiccini has quit IRC | 16:16 | |
| *** udesale has quit IRC | 16:16 | |
| *** yprokule has quit IRC | 16:17 | |
| bnemec | rlandy: I've noticed a couple of things in my investigation: First, ipxe-boot is missing the os_shutdown_timeout metadata, so it takes 60(?) seconds to stop the instances when they're registered with ironic. | 16:17 |
| bnemec | That doesn't seem like it should be a problem in and of itself, but it's something we probably want to get fixed. | 16:17 |
| bnemec | Second, I don't see any requests in the ironic logs to actually start the instances. | 16:18 |
| bnemec | Should I? Or does inspector make the IPMI calls directly? | 16:18 |
| *** bogdando has quit IRC | 16:18 | |
| EmilienM | bnemec: I agree but we haven't split the ha job into different services coverage | 16:19 |
| *** dtantsur is now known as dtantsur|afk | 16:23 | |
| *** pcaruana has quit IRC | 16:24 | |
| rlandy | bnemec: thanks for looking into this. wrt powering on, looking at the logs ... I see power off and power status commands and 'During sync_power_state, node <> state does not match expected state 'power on'. Updating recorded state to 'power off'.' Don't see an explicit power on though | 16:24 |
| *** bkopilov has quit IRC | 16:25 | |
| rlandy | the power status commands issued by ipmi are recorded | 16:26 |
| rlandy | bnemec: as a comparison, here is a similar log from qeos7: https://thirdparty-logs.rdoproject.org/jenkins-poc-tripleo-quickstart-master-ovb-qeos7-public-bond-12/undercloud/var/log/ironic/ironic-conductor.log.gz | 16:27 |
| bnemec | rlandy: Oh, interesting. I just noticed that there was eventually a timeout from ironic in my environment: http://paste.openstack.org/show/610580/ | 16:27 |
| rlandy | in the qeos7 log, there is the power on command | 16:28 |
| bnemec | I don't understand the timeout. The bmc was definitely reporting that the node was off, and I believe the ironic node status was correct as well. | 16:30 |
| rlandy | yes - I have checked the ironic status manually while the nodes are off | 16:30 |
| rlandy | and it matches | 16:30 |
| rlandy | which is why it confuses me that the power on is never executed | 16:31 |
| bnemec | rlandy: Do you see the timeout in your logs too? | 16:31 |
| rlandy | yes ... ERROR ironic.drivers.modules.ipmitool [-] IPMI power off timed out after 4 retries on node <> | 16:31 |
| bnemec | I bet there's some sort of race that is being triggered by the longer shutdown timeout. | 16:32 |
| bnemec | I wonder if I can reproduce this in my cloud by removing os_shutdown_timeout from my image too. | 16:33 |
| rlandy | I was looking for a possible setting we could adjust to extend the time to check for node status | 16:33 |
| *** milan has left #tripleo | 16:34 | |
| *** bkopilov has joined #tripleo | 16:34 | |
| *** florianf has quit IRC | 16:35 | |
| *** egonzalez has quit IRC | 16:36 | |
| bnemec | rlandy: Well, ultimately we want to set os_shutdown_timeout on the image which I suspect will make this go away. I'm testing without os_shutdown_timeout locally to see if it happens to me too. | 16:37 |
| bnemec | But it is mildly concerning that 60 seconds is causing problem. I could easily see real world hardware taking that long. | 16:37 |
| bnemec | Although maybe not. If Ironic does a hard shutdown of the node I bet most servers shut off almost immediately. | 16:38 |
| *** jaganathan has joined #tripleo | 16:38 | |
| *** jaganathan has quit IRC | 16:38 | |
| bnemec | rlandy: Do you know who we need to talk to to get the image metadata set correctly? | 16:38 |
| *** anshul has joined #tripleo | 16:39 | |
| rlandy | bnemec: public images are uploaded by the cloud admins - you can ask on internal #rhos-ops. (we can upload private images for test) | 16:40 |
| bnemec | rlandy: Okay, thanks. We could try it with a private image, but we need this set on the public one anyway. | 16:41 |
| rlandy | bnemec: agreed - because anyone deploying with ovb should be starting from the same point | 16:41 |
| *** salmankhan has quit IRC | 16:41 | |
| *** salmankhan has joined #tripleo | 16:42 | |
| rlandy | bnemec: you can open a ticket to get the image metadata fixed on the public image (same address where you requested an account) - which is probably what they will tell us to do if we ask on channel | 16:44 |
| *** ccamacho|brb is now known as ccamacho|out | 16:45 | |
| bnemec | rlandy: Heh, good call. | 16:45 |
| *** ffiore has quit IRC | 16:46 | |
| openstackgerrit | Julie Pichon proposed openstack/tripleo-common master: Parse roles from roles_data.yaml directly https://review.openstack.org/467678 | 16:47 |
| openstackgerrit | John Trowbridge proposed openstack/tripleo-quickstart master: Add interactive prompt to recreate venv in case last run failed https://review.openstack.org/467679 | 16:48 |
| openstackgerrit | Martin André proposed openstack/puppet-tripleo master: Add missing octavia mysql user creation https://review.openstack.org/467680 | 16:49 |
| *** jaganathan has joined #tripleo | 16:49 | |
| *** suuuper has quit IRC | 16:49 | |
| *** ckyriakidou has quit IRC | 16:49 | |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates master: WIP add docker template for octavia services https://review.openstack.org/467254 | 16:49 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates master: Allows use of Mistral workflows during deployment steps https://review.openstack.org/463324 | 16:50 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates master: Provides a list of per-service network IPs to the workflows env https://review.openstack.org/467682 | 16:50 |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates master: WIP add docker template for octavia services https://review.openstack.org/467254 | 16:50 |
| bnemec | rlandy: Ticket is submitted. | 16:50 |
| *** jpich has quit IRC | 16:51 | |
| bnemec | Whoops, scratch that. Just got a delivery failure message. :-( | 16:51 |
| rlandy | bnemec: I am forwarding you the original ticket I entered to add the public ipxe boot image | 16:51 |
| *** trown is now known as trown|lunch | 16:51 | |
| rlandy | bnemec: what address did you use that delivery failed? | 16:52 |
| *** michapma_dsk has quit IRC | 16:55 | |
| *** ramishra has quit IRC | 16:55 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates master: [DO_NOT_MERGE] test service_workflow_tasks https://review.openstack.org/465066 | 16:55 |
| *** derekh has quit IRC | 16:57 | |
| *** fultonj has quit IRC | 16:59 | |
| *** fultonj has joined #tripleo | 17:01 | |
| *** dbecker has quit IRC | 17:03 | |
| *** anshul has quit IRC | 17:04 | |
| *** ooolpbot has joined #tripleo | 17:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1680195 | 17:10 |
| openstack | Launchpad bug 1680195 in tripleo "Random ovb-ha ping test failures" [Critical,Triaged] | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1690132 | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1690373 | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1693174 | 17:10 |
| *** ooolpbot has quit IRC | 17:10 | |
| openstack | Launchpad bug 1690132 in tripleo "Newton HA failure: unable to push cib, cib_apply_diff failed" [Critical,Triaged] | 17:10 |
| openstack | Launchpad bug 1690373 in tripleo "stable/newton gate-tripleo-ci-centos-7-nonha-multinode-oooq broken" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 17:10 |
| openstack | Launchpad bug 1693174 in tripleo "CI: ovb-ha jobs fail because of lack of memory" [Critical,Triaged] | 17:10 |
| slagle | ebarrera: it's here: https://docs.openstack.org/developer/tripleo-docs/advanced_deployment/network_isolation_virt.html | 17:11 |
| slagle | ebarrera: in the demo, i add the ovs port and assign an ip manually. the docs show using an ifcfgvlan10 file instead, but the effect is the same | 17:12 |
| slagle | ifcfg-vlan10 | 17:12 |
| *** nyechiel has joined #tripleo | 17:13 | |
| *** links has quit IRC | 17:16 | |
| *** hjensas has quit IRC | 17:16 | |
| *** jaganathan has quit IRC | 17:17 | |
| ebarrera | slagle, thanks! tomorrow I have remote session with a customer with a virtual env, sure this guide will help me! | 17:18 |
| *** twi99y_ has joined #tripleo | 17:19 | |
| *** twi99y has quit IRC | 17:19 | |
| *** gvrangan_ has joined #tripleo | 17:21 | |
| *** gfidente has quit IRC | 17:22 | |
| *** jaganathan has joined #tripleo | 17:22 | |
| *** athomas has quit IRC | 17:22 | |
| *** jaganathan has quit IRC | 17:23 | |
| *** jaosorior has quit IRC | 17:24 | |
| openstackgerrit | Tim Rozet proposed openstack/tripleo-heat-templates master: Adds service for OVS and enables ODL DPDK deployments https://review.openstack.org/419648 | 17:25 |
| slagle | ebarrera: ok. i'm not sure of their use case, but you wouldn't want to route all your overcloud external traffic through the undercloud for any type of production deployment. | 17:26 |
| slagle | it's really for dev/test setups only | 17:26 |
| ebarrera | slagle, they have problems getting the introspector.ipxe, tomorrow I will run tcpdump and an interactive ipxe client to check it | 17:28 |
| ebarrera | with the guide I will take as example how are the libvirt networks configured | 17:28 |
| slagle | ebarrera: additionally, what is shown there will only work if you're using all the default values from the single-nic-with-vlans templates in t-h-t | 17:29 |
| *** twi99y_ is now known as eqvist | 17:30 | |
| *** AzazeL_ has joined #tripleo | 17:30 | |
| EmilienM | thrash, rbrady: I just found out that all upgrade jobs are now failing on http://logs.openstack.org/00/461000/16/check/gate-tripleo-ci-centos-7-scenario001-multinode-upgrades-nv/c0b8140/console.html#_2017-05-24_14_02_34_443408 | 17:36 |
| bnemec | rlandy: Pretty sure I reproduced the problem in my local cloud by unsetting os_shutdown_timeout. | 17:36 |
| bnemec | So once we get that fixed I think RDO Cloud should be working again. | 17:36 |
| rlandy | bnemec: that would be great - thanks | 17:36 |
| EmilienM | thrash, rbrady : nevermind I think I know. | 17:36 |
| rlandy | the value of testing ovb is majorly reduced with introspection disabled | 17:37 |
| rlandy | bnemec: I won't see the response to your ticket - if you get an update that the image is fixed, could you let me know? | 17:38 |
| bnemec | rlandy: Will do. | 17:38 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: ceilometer-expirer: remove the crontab during upgrade https://review.openstack.org/466096 | 17:38 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: CI test - never merge https://review.openstack.org/461000 | 17:38 |
| *** jaganathan has joined #tripleo | 17:39 | |
| *** fragatina has quit IRC | 17:39 | |
| *** eqvist has quit IRC | 17:41 | |
| *** nyechiel has quit IRC | 17:41 | |
| openstackgerrit | James Slagle proposed openstack/puppet-tripleo master: Install rsync package for galera https://review.openstack.org/467362 | 17:48 |
| *** jmelvin has quit IRC | 17:53 | |
| *** pkovar has quit IRC | 17:54 | |
| *** trown|lunch is now known as trown | 17:54 | |
| *** masco has quit IRC | 17:54 | |
| sshnaidm | bnemec, can we merge this? https://review.openstack.org/#/c/467620/ | 17:58 |
| *** sshnaidm is now known as sshnaidm|afk | 17:58 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: WIP - Ability to enable/disable debug mode per OpenStack service https://review.openstack.org/467698 | 18:00 |
| EmilienM | mwhahaha: early feedback would be appreciated on https://review.openstack.org/467698 | 18:00 |
| *** zoli|wfh is now known as zoli|gone | 18:01 | |
| EmilienM | sshnaidm|afk: wait wait EPEL again? | 18:01 |
| EmilienM | ah for infra nodes | 18:04 |
| *** jpena is now known as jpena|off | 18:04 | |
| *** ooolpbot has joined #tripleo | 18:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1680195 | 18:10 |
| openstack | Launchpad bug 1680195 in tripleo "Random ovb-ha ping test failures" [Critical,Triaged] | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1690132 | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1690373 | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1693174 | 18:10 |
| *** ooolpbot has quit IRC | 18:10 | |
| openstack | Launchpad bug 1690132 in tripleo "Newton HA failure: unable to push cib, cib_apply_diff failed" [Critical,Triaged] | 18:10 |
| openstack | Launchpad bug 1690373 in tripleo "stable/newton gate-tripleo-ci-centos-7-nonha-multinode-oooq broken" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 18:10 |
| openstack | Launchpad bug 1693174 in tripleo "CI: ovb-ha jobs fail because of lack of memory" [Critical,Triaged] | 18:10 |
| openstackgerrit | David Moreau Simard proposed openstack/tripleo-common master: Test container images from trunk.registry.rdoproject.org https://review.openstack.org/467651 | 18:10 |
| *** ramishra has joined #tripleo | 18:11 | |
| *** ihrachys has joined #tripleo | 18:18 | |
| ihrachys | hi folks. | 18:19 |
| ihrachys | I try devmode.sh on rdo cloud | 18:19 |
| ihrachys | and I get this: http://paste.openstack.org/show/610595/ | 18:19 |
| ihrachys | apparently neutronclient is not installed into the env or smth | 18:19 |
| ihrachys | tried to execute devmode.sh from inside venv with qooo installed, and outisde it | 18:20 |
| ihrachys | same issue | 18:20 |
| ihrachys | any ideas what I do wrong? | 18:20 |
| openstackgerrit | Merged openstack/puppet-tripleo master: Enable mistral to run under mod_wsgi https://review.openstack.org/458503 | 18:21 |
| openstackgerrit | Merged openstack/puppet-tripleo master: Add support for autofencing to Pacemaker Remote. https://review.openstack.org/459744 | 18:21 |
| *** jlinkes has quit IRC | 18:22 | |
| ihrachys | bcafarel, ^ have you hit it in your rdocloud experiments? | 18:22 |
| *** ahrechan has quit IRC | 18:24 | |
| *** ramishra has quit IRC | 18:26 | |
| trown | ihrachys: is neutronclient in the venv in ~/.quickstart? | 18:28 |
| ihrachys | trown, no | 18:29 |
| ihrachys | well it's not in reqs.txt in quickstart, so maybe that? | 18:29 |
| ihrachys | lemme add it there | 18:29 |
| *** gbarros has quit IRC | 18:30 | |
| trown | ihrachys: well it might get pulled by one of the other reqs... I have had multiple runs succeed on rdocloud, so there must be something different | 18:30 |
| openstackgerrit | Michele Baldessari proposed openstack/instack-undercloud master: Increase the max_allowed_packet size in mysql's [server] stanza https://review.openstack.org/467707 | 18:30 |
| *** atoth has quit IRC | 18:31 | |
| ihrachys | trown, neutronclient is installed via rpm on the system. maybe that can influence somehow. | 18:31 |
| ihrachys | in any case, if you import it from the repo, the dep should be explicit | 18:33 |
| *** jaganathan has quit IRC | 18:34 | |
| trown | ya it is not quickstart doing that import... it is https://github.com/cybertron/openstack-virtual-baremetal/blob/master/requirements.txt | 18:35 |
| trown | maybe we are not installing that correctly | 18:35 |
| EmilienM | slagle: https://review.openstack.org/#/c/467362 | 18:36 |
| EmilienM | slagle: you have resource dup | 18:36 |
| EmilienM | slagle: because puppet-rsync doesn't use ensure_packages | 18:36 |
| trown | bnemec: any chance you would want to publish releases of https://github.com/cybertron/openstack-virtual-baremetal on pypi? | 18:36 |
| slagle | EmilienM: you told me to use ensure_packages to protect against that :) | 18:36 |
| EmilienM | slagle: you can try something like : if defined(Package['rsync'])) | 18:37 |
| ihrachys | trown, added neutronclient and now it works | 18:37 |
| ihrachys | I will post a patch | 18:37 |
| EmilienM | slagle: the thing with ensure_packages is that all occurences need to use it | 18:37 |
| EmilienM | slagle: if you have ONE resource that use Package['rsync'] in the catalog, you're doomed | 18:37 |
| trown | ihrachys: interesting, a bit confused why that is not failing for me... but patches welcome :) | 18:37 |
| EmilienM | slagle: it should work: if defined(Package['rsync'])) | 18:37 |
| trown | ihrachys: nevermind that just failed for me too... I know panda|bbl had a successful rdocloud deploy earlier today, so something changed | 18:38 |
| EmilienM | slagle: stackoverflow was wrong | 18:40 |
| trown | EmilienM: heresy | 18:40 |
| slagle | i'm just glad we aren't using a script to do this, b/c yikes, that would be so complex | 18:40 |
| EmilienM | :) | 18:41 |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates master: Disabled Ceilometer Api on upgrades https://review.openstack.org/467713 | 18:42 |
| slagle | EmilienM: but i guess i'm wondering why this is a bug if puppet-rsync should have installed the package | 18:42 |
| slagle | EmilienM: perhaps puppet-rsync is included at too late of a step? | 18:42 |
| *** rlandy is now known as rlandy|afk | 18:43 | |
| *** gbarros has joined #tripleo | 18:44 | |
| EmilienM | slagle: yes for sure | 18:45 |
| EmilienM | slagle: it's deployed when swift is deployed | 18:45 |
| *** jaganathan has joined #tripleo | 18:47 | |
| openstackgerrit | Keith Schincke proposed openstack/tripleo-heat-templates master: WIP Update NovaCompute to include CephClientKey https://review.openstack.org/467665 | 18:47 |
| *** iranzo has quit IRC | 18:51 | |
| *** salmankhan has quit IRC | 18:51 | |
| *** itlinux has joined #tripleo | 18:52 | |
| *** dbecker has joined #tripleo | 18:52 | |
| bnemec | trown: It's on my todo list, but currently everything is designed to be run directly from the git repo so it wouldn't work without some changes. | 18:59 |
| trown | bnemec: k, I think in the interim we should probably just use your requirements file when using that repo in quickstart | 19:00 |
| trown | ihrachys: ^ | 19:00 |
| *** itlinux has quit IRC | 19:01 | |
| *** jmelvin has joined #tripleo | 19:02 | |
| *** ooolpbot has joined #tripleo | 19:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1680195 | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1690132 | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1690373 | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1693174 | 19:10 |
| openstack | Launchpad bug 1680195 in tripleo "Random ovb-ha ping test failures" [Critical,Triaged] | 19:10 |
| *** ooolpbot has quit IRC | 19:10 | |
| openstack | Launchpad bug 1690132 in tripleo "Newton HA failure: unable to push cib, cib_apply_diff failed" [Critical,Triaged] | 19:10 |
| openstack | Launchpad bug 1690373 in tripleo "stable/newton gate-tripleo-ci-centos-7-nonha-multinode-oooq broken" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 19:10 |
| openstack | Launchpad bug 1693174 in tripleo "CI: ovb-ha jobs fail because of lack of memory" [Critical,Triaged] | 19:10 |
| *** amoralej is now known as amoralej|off | 19:14 | |
| *** itlinux has joined #tripleo | 19:15 | |
| *** itlinux has quit IRC | 19:23 | |
| *** dweeks has joined #tripleo | 19:31 | |
| *** ecerquei_ has quit IRC | 19:31 | |
| dweeks | If you need to add connectivity to the external connectivity to compute nodes for accessing external Ceph, DNS, NTP, is there a way to add this to the templates and have the compute nodes updated or do I have to do a complete reinstall? | 19:34 |
| dweeks | If you need to add external connectivity to compute nodes for accessing external Ceph, DNS, NTP, is there a way to add this to the templates and have the compute nodes updated or do I have to do a complete reinstall? | 19:34 |
| *** pkovar has joined #tripleo | 19:37 | |
| ihrachys | trown, seems like you have a grasp of the cause | 19:39 |
| ihrachys | trown, maybe I will just leave it for you to solve ;) | 19:39 |
| ihrachys | I mean, you == someone else | 19:39 |
| *** itlinux has joined #tripleo | 19:39 | |
| *** itlinux has quit IRC | 19:39 | |
| mwhahaha | pradk: fyi https://bugs.launchpad.net/tripleo/+bug/1693339 it seems like the ceilometer-upgrade is being improperly run on the compute node by default | 19:40 |
| openstack | Launchpad bug 1693339 in tripleo "basic overcloud deploy fails on ceilometer-upgrade" [High,Triaged] | 19:40 |
| pradk | mwhahaha, looking | 19:40 |
| pradk | mwhahaha, thats weird, upgrade should only run on controller i thought | 19:41 |
| mwhahaha | pradk: me too :D | 19:41 |
| pradk | hmm i think it might be because we inlcude this in tripleo::profile::base::ceilometer | 19:42 |
| pradk | since we deprecated collector and api | 19:42 |
| mwhahaha | it's possible, i'm trying to find out what the catalog looks like | 19:43 |
| pradk | https://github.com/openstack/puppet-tripleo/blob/master/manifests/profile/base/ceilometer.pp#L120-L124 | 19:43 |
| mwhahaha | pradk: ::tripleo::profile::base::ceilometer::agent::polling is what is being included | 19:44 |
| mwhahaha | pradk: step5 catalog looks like: http://paste.openstack.org/show/610602/ | 19:44 |
| pradk | right polling uses compute names space for compute agent | 19:45 |
| pradk | and it includes https://github.com/openstack/puppet-tripleo/blob/master/manifests/profile/base/ceilometer/agent/polling.pp#L52 | 19:45 |
| mwhahaha | pradk: i don't think it should include the base for the agent | 19:46 |
| mwhahaha | pradk: or does the default ceilometer base just drop configs w/o services | 19:47 |
| pradk | hmm wont we need ceilometer::congif for agents | 19:47 |
| mwhahaha | pradk: then i don't think we need to be be including the base in the agents | 19:47 |
| mwhahaha | i'd like to know how this didn't show up for 4 months if that's the problem | 19:48 |
| *** openstackgerrit has quit IRC | 19:48 | |
| pradk | hmm wont we need for containers? | 19:48 |
| pradk | otherwise the config wont be there? | 19:48 |
| mwhahaha | pradk: do the agents use the messaging config? | 19:49 |
| pradk | yea notification agent does | 19:49 |
| mwhahaha | k well we need to find a different spot for https://github.com/openstack/puppet-tripleo/commit/2e89f8ede8369d0a7445599d18ac562ca91e9ad0 | 19:49 |
| mwhahaha | because base gets included by everything | 19:49 |
| pradk | yea | 19:49 |
| mwhahaha | so the upgrade is trying to be run on compute nodes | 19:49 |
| mwhahaha | pradk: actually i wonder how $sync_db is being set to true | 19:50 |
| pradk | yea that should onl;y be run on bootstrap node | 19:51 |
| pradk | compute_hostname should not match that check | 19:51 |
| *** gvrangan_ has quit IRC | 19:52 | |
| *** openstackgerrit has joined #tripleo | 19:52 | |
| openstackgerrit | Damien Ciabrini proposed openstack/tripleo-heat-templates master: Containerize HAProxy for HA https://review.openstack.org/459597 | 19:52 |
| mwhahaha | pradk: ./bootstrap_node.json: "bootstrap_nodeid": "overcloud-novacompute-0", | 19:52 |
| mwhahaha | [root@overcloud-novacompute-0 hieradata]# hiera -c /etc/puppet/hiera.yaml bootstrap_nodeid | 19:53 |
| mwhahaha | overcloud-novacompute-0 | 19:53 |
| mwhahaha | so it looks like we've broken the bootstrap_nodeid or something | 19:53 |
| openstackgerrit | Damien Ciabrini proposed openstack/tripleo-heat-templates master: Containerize MySQL for HA https://review.openstack.org/455348 | 19:53 |
| pradk | mwhahaha, what does that show up on controller? | 19:53 |
| mwhahaha | overcloud-controller-0 | 19:54 |
| openstackgerrit | Damien Ciabrini proposed openstack/tripleo-heat-templates master: Containerize Redis for HA https://review.openstack.org/464958 | 19:54 |
| pradk | weird that seems wrong.. seems like we're setting bootstrap node id = $::hostname | 19:55 |
| mwhahaha | dprince: i don't suppose you guys changed something for the bootstrap node for containers recently? | 19:55 |
| openstackgerrit | Damien Ciabrini proposed openstack/tripleo-heat-templates master: Containerize RabbitMQ for HA https://review.openstack.org/452889 | 19:55 |
| dprince | mwhahaha: we are making use of it, but no hiera changes from us I think | 19:55 |
| dprince | mwhahaha: all of this uses it for example https://review.openstack.org/#/c/465551/ | 19:55 |
| *** trown is now known as trown|brb | 19:56 | |
| mwhahaha | oh wait bootstrap_nodeid is role specific | 19:56 |
| mwhahaha | so there's a different bootstrapt_nodeid for each *role* | 19:56 |
| mwhahaha | not overcloud | 19:56 |
| mwhahaha | pradk yea that upgrade needs to get moved out of base to something else | 19:56 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Fix issues in OVB preparement servers scripts https://review.openstack.org/467620 | 19:57 |
| *** itlinux has joined #tripleo | 19:57 | |
| pradk | mwhahaha, lemme see what i can do.. perhaps i'll add a upgrade.pp and only include this in notification and central agent perhaps | 19:57 |
| *** hjensas has joined #tripleo | 19:59 | |
| *** hjensas has joined #tripleo | 19:59 | |
| sshnaidm|afk | hrybacki, ping | 19:59 |
| panda|bbl | trown|brb: I think we have to pin shade, version 1.21 is not working anymore for us | 20:00 |
| hamzy | Does anyone know where the scripts are to build ironic-python-agent.{initramfs,kernel} and overcloud-full.{initrd,qcow2,vmlinuz}? | 20:00 |
| mwhahaha | hamzy: what do you want to do? | 20:01 |
| hamzy | mwhahaha, build them for ppc64le :) | 20:01 |
| mwhahaha | hamzy: it's built into tripleo-common | 20:01 |
| mwhahaha | hamzy: but it takes a config http://tripleo.org/basic_deployment/basic_deployment_cli.html#get-images | 20:02 |
| hamzy | mwhahaha, from ./image-yaml/overcloud-images-rhel7.yaml ? | 20:02 |
| mwhahaha | hamzy: that's the defualt, you can pass in a different one | 20:02 |
| thrash | hamzy: and they're "stackable" | 20:03 |
| mwhahaha | hamzy: openstack overcloud image build --help | 20:03 |
| hamzy | thanks | 20:03 |
| *** trown|brb is now known as trown | 20:03 | |
| openstackgerrit | Ihar Hrachyshka proposed openstack/tripleo-quickstart master: Added neutronclient dependency https://review.openstack.org/467746 | 20:03 |
| openstackgerrit | Gabriele Cerami proposed openstack/tripleo-quickstart master: requirements: ensure shade version is less than 1.21.0 https://review.openstack.org/467747 | 20:05 |
| panda|bbl | trown: ^ | 20:05 |
| trown | panda|bbl: shade works fine for me if I put neutronclient in the vevn though | 20:05 |
| trown | and panda|bbl looks like ihrachys put up a patch for it https://review.openstack.org/467746 | 20:06 |
| panda|bbl | trown: not for me, it gives me an error during stack deploy | 20:06 |
| *** toure is now known as toure|gone | 20:06 | |
| panda|bbl | trown: \"/tmp/ansible_qsfYmO/ansible_module_os_stack.py\", line 247, in main\n id=stack.id)\nAttributeError: 'bool' object has no attribute 'id'\n", "module_stdout": "", "msg": "MODULE | 20:07 |
| trown | panda|bbl: weird... and you are running from centos? | 20:07 |
| trown | panda|bbl: do you run with -d? | 20:07 |
| ihrachys | that is prolly smth diff | 20:08 |
| *** dbecker has quit IRC | 20:08 | |
| panda|bbl | trown: --delete-all-stack ? | 20:08 |
| panda|bbl | trown: yes | 20:08 |
| *** ooolpbot has joined #tripleo | 20:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1680195 | 20:10 |
| openstack | Launchpad bug 1680195 in tripleo "Random ovb-ha ping test failures" [Critical,Triaged] | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1690132 | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1690373 | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1693174 | 20:10 |
| *** ooolpbot has quit IRC | 20:10 | |
| openstack | Launchpad bug 1690132 in tripleo "Newton HA failure: unable to push cib, cib_apply_diff failed" [Critical,Triaged] | 20:10 |
| openstack | Launchpad bug 1690373 in tripleo "stable/newton gate-tripleo-ci-centos-7-nonha-multinode-oooq broken" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 20:10 |
| openstack | Launchpad bug 1693174 in tripleo "CI: ovb-ha jobs fail because of lack of memory" [Critical,Triaged] | 20:10 |
| openstackgerrit | Damien Ciabrini proposed openstack/tripleo-common master: WIP kolla template overrides for pacemaker HA containers https://review.openstack.org/456174 | 20:11 |
| openstackgerrit | Damien Ciabrini proposed openstack/puppet-tripleo master: WIP cinder-volume bundle for HA deployments https://review.openstack.org/467749 | 20:12 |
| openstackgerrit | Pradeep Kilambi proposed openstack/puppet-tripleo master: Move ceilometer upgrade step out of base https://review.openstack.org/467750 | 20:13 |
| pradk | mwhahaha, ^^ | 20:14 |
| openstackgerrit | Damien Ciabrini proposed openstack/tripleo-heat-templates master: WIP THT-part of cinder-volume for HA https://review.openstack.org/467752 | 20:14 |
| mwhahaha | pradk: so why is this being included with the agents themselves? | 20:16 |
| *** rlandy|afk is now known as rlandy | 20:17 | |
| pradk | mwhahaha, so with collector dead, we need to ensure this is run before the data can be dispatched once central/notification agents gather it | 20:17 |
| pradk | if not, gnocchi wont be initialized for it to handle resource types from ceilo | 20:17 |
| pradk | previously, we did this as part of collector deploy and all was well | 20:18 |
| trown | panda|bbl: have you tried with older shade? seems like stack object is a bool in your env... that seems odd | 20:18 |
| mwhahaha | pradk: so my concern is that if the agent isn't on the controller that's the problem | 20:18 |
| trown | panda|bbl: I have a deploy I started an hour ago that did not hit that issue (I put neutronclient in reqs) | 20:18 |
| pradk | mwhahaha, central agent has to be on controller | 20:18 |
| *** itlinux has quit IRC | 20:18 | |
| pradk | and so is notification | 20:19 |
| mwhahaha | pradk: ok by why include it in the polling agent then | 20:19 |
| *** liverpooler has quit IRC | 20:19 | |
| pradk | mwhahaha, because polling is a generic class uses central as namespace | 20:19 |
| pradk | polling agent == {central, compute} | 20:19 |
| pradk | based on what namespace you pass in | 20:19 |
| mwhahaha | this is convoluted | 20:19 |
| mwhahaha | but ok | 20:19 |
| mwhahaha | as long as a basic deploy works | 20:20 |
| pradk | for example - /usr/bin/ceilometer-polling --polling-namespaces central .. is how central agent is started | 20:20 |
| pradk | yea lets see if this works | 20:21 |
| pradk | mwhahaha, i'm cool with changing it if there is a cleaner approach | 20:21 |
| panda|bbl | trown: if I pin to the older shade, I don't have to add neutronclient, do at least we know what changed. I tested again on centos, and it's ok, seems the error was in a fedora box. | 20:21 |
| panda|bbl | so* | 20:21 |
| mwhahaha | pradk: i'll give a test | 20:22 |
| trown | panda|bbl: no I am quite confused by the neutronclient thing... because in the ovb repo it is pretty obvious we need it... but somehow it was working before today | 20:23 |
| *** ansmith has quit IRC | 20:23 | |
| bkero | Maybe something sourced it in? | 20:24 |
| panda|bbl | trown: shade went from 1.20 to 1.21 this afternoon | 20:24 |
| trown | in any case the patch from ihrachys is the correct thing to do because the ovb repo clearly requires neutronclient | 20:24 |
| *** thrash is now known as thrash|g0ne | 20:25 | |
| bkero | - python-neutronclient \ | 20:25 |
| trown | panda|bbl: ah yep older release of shade had req on neutronclient https://github.com/openstack-infra/shade/blob/15d64f2af5d9f45ba91bc7f4f5f48402ab8c339b/requirements.txt | 20:26 |
| trown | so that is what happened | 20:26 |
| bkero | Yep, got removed in 1.21.0 | 20:26 |
| *** jprovazn has quit IRC | 20:28 | |
| *** gvrangan_ has joined #tripleo | 20:30 | |
| panda|bbl | https://github.com/openstack-infra/shade/commit/81239f6742a22415492fe65873f9606b6734173d | 20:31 |
| panda|bbl | shade doesn't but the ansible module does ? | 20:31 |
| trown | panda|bbl: it is the ovb repo that requires it as in https://github.com/cybertron/openstack-virtual-baremetal | 20:33 |
| trown | panda|bbl: we were just getting it via shade before the latest release so we didnt have to be explicit about it | 20:34 |
| trown | now we do | 20:34 |
| *** gvrangan_ has quit IRC | 20:34 | |
| openstackgerrit | Keith Schincke proposed openstack/tripleo-heat-templates master: WIP Update NovaCompute to include CephClientKey https://review.openstack.org/467665 | 20:36 |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart master: Add python-neutronclient requirement no longer in shade https://review.openstack.org/467758 | 20:38 |
| *** ebarrera has quit IRC | 20:38 | |
| *** ebarrera has joined #tripleo | 20:39 | |
| *** dprince has quit IRC | 20:39 | |
| *** egafford has quit IRC | 20:39 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/puppet-tripleo master: Move ceilometer upgrade step out of base https://review.openstack.org/467750 | 20:45 |
| *** cylopez has quit IRC | 20:47 | |
| *** paramite has joined #tripleo | 20:47 | |
| *** itlinux has joined #tripleo | 20:47 | |
| *** trown is now known as trown|outtypewww | 20:49 | |
| *** paramite has quit IRC | 20:55 | |
| *** ebarrera has quit IRC | 20:56 | |
| *** itlinux has quit IRC | 20:57 | |
| *** bfournie has quit IRC | 21:02 | |
| *** Goneri has quit IRC | 21:02 | |
| *** lblanchard has quit IRC | 21:04 | |
| *** gfidente has joined #tripleo | 21:06 | |
| *** eck` is now known as eck`gone | 21:10 | |
| *** ooolpbot has joined #tripleo | 21:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1680195 | 21:10 |
| openstack | Launchpad bug 1680195 in tripleo "Random ovb-ha ping test failures" [Critical,Triaged] | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1690132 | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1690373 | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1693174 | 21:10 |
| *** ooolpbot has quit IRC | 21:10 | |
| openstack | Launchpad bug 1690132 in tripleo "Newton HA failure: unable to push cib, cib_apply_diff failed" [Critical,Triaged] | 21:10 |
| openstack | Launchpad bug 1690373 in tripleo "stable/newton gate-tripleo-ci-centos-7-nonha-multinode-oooq broken" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 21:10 |
| openstack | Launchpad bug 1693174 in tripleo "CI: ovb-ha jobs fail because of lack of memory" [Critical,Triaged] | 21:10 |
| openstackgerrit | Merged openstack/tripleo-specs master: bug-tagging: add deployment-time tag https://review.openstack.org/467335 | 21:14 |
| mwhahaha | does anyone know how to get a tempurl with openstackclient rather than swiftclient? | 21:14 |
| EmilienM | upgrade jobs are broken on ocata | 21:14 |
| EmilienM | http://logs.openstack.org/50/463050/6/check/gate-tripleo-ci-centos-7-scenario001-multinode-upgrades/e808f4a/logs/subnode-2/var/log/nova/nova-compute.txt.gz#_2017-05-23_15_58_53_696 | 21:14 |
| EmilienM | mwhahaha: sounds like there is a mapping: https://github.com/openstack/python-openstackclient/blob/a3e1b9c31a400105608c9cd3575930dfdfa0b248/doc/source/data/swift.csv#L9 | 21:16 |
| EmilienM | not sure if it's actually supported | 21:16 |
| mwhahaha | well we broke swiftclient on undercloud | 21:16 |
| mwhahaha | so the upload swift artifacts are broken | 21:16 |
| mwhahaha | i was trying to convert it to osc | 21:17 |
| mwhahaha | but that's the one bit that is missing | 21:17 |
| *** morazi has quit IRC | 21:19 | |
| *** abishop has quit IRC | 21:24 | |
| *** pradk has quit IRC | 21:29 | |
| *** pradk has joined #tripleo | 21:31 | |
| *** atheurer has quit IRC | 21:37 | |
| *** atheurer has joined #tripleo | 21:40 | |
| *** rbrady is now known as rbrady-afk | 21:43 | |
| *** fragatina has joined #tripleo | 21:46 | |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart-extras master: Re-enable introspection on rdocloud https://review.openstack.org/467785 | 21:51 |
| bnemec | rlandy: Modulo an mtu issue, I was able to do introspection on rdo cloud with no trouble now. | 21:53 |
| rlandy | bnemec: yes - my test job is deploying the overcloud now - so it's past introspection. Did you give any other glance upload command than I had put in my first ticket request? | 21:55 |
| bnemec | rlandy: No, I just asked them to add the metadata. | 21:56 |
| *** jkilpatr has quit IRC | 22:05 | |
| *** ooolpbot has joined #tripleo | 22:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1680195 | 22:10 |
| openstack | Launchpad bug 1680195 in tripleo "Random ovb-ha ping test failures" [Critical,Triaged] | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1690132 | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1690373 | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1693174 | 22:10 |
| *** ooolpbot has quit IRC | 22:10 | |
| openstack | Launchpad bug 1690132 in tripleo "Newton HA failure: unable to push cib, cib_apply_diff failed" [Critical,Triaged] | 22:10 |
| openstack | Launchpad bug 1690373 in tripleo "stable/newton gate-tripleo-ci-centos-7-nonha-multinode-oooq broken" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 22:10 |
| openstack | Launchpad bug 1693174 in tripleo "CI: ovb-ha jobs fail because of lack of memory" [Critical,Triaged] | 22:10 |
| *** karimb has quit IRC | 22:12 | |
| *** rwsu has quit IRC | 22:15 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates master: Include mongodb in scenario002 job https://review.openstack.org/467353 | 22:18 |
| *** rhallisey has quit IRC | 22:18 | |
| *** jmelvin has quit IRC | 22:22 | |
| *** jkilpatr has joined #tripleo | 22:26 | |
| *** pchavva has quit IRC | 22:29 | |
| openstackgerrit | Merged openstack/tripleo-docs master: Update the guide for pxe_ssh to vbmc migration include pxe_ipmitool https://review.openstack.org/460338 | 22:33 |
| *** itlinux has joined #tripleo | 22:45 | |
| openstackgerrit | Merged openstack/tripleo-quickstart master: Added neutronclient dependency https://review.openstack.org/467746 | 22:52 |
| *** hamzy has quit IRC | 22:53 | |
| *** pradk has quit IRC | 22:57 | |
| *** ooolpbot has joined #tripleo | 23:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1680195 | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1690132 | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1690373 | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1693174 | 23:10 |
| openstack | Launchpad bug 1680195 in tripleo "Random ovb-ha ping test failures" [Critical,Triaged] | 23:10 |
| *** ooolpbot has quit IRC | 23:10 | |
| openstack | Launchpad bug 1690132 in tripleo "Newton HA failure: unable to push cib, cib_apply_diff failed" [Critical,Triaged] | 23:10 |
| openstack | Launchpad bug 1690373 in tripleo "stable/newton gate-tripleo-ci-centos-7-nonha-multinode-oooq broken" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 23:10 |
| openstack | Launchpad bug 1693174 in tripleo "CI: ovb-ha jobs fail because of lack of memory" [Critical,Triaged] | 23:10 |
| *** itlinux has quit IRC | 23:11 | |
| *** dmarlin has quit IRC | 23:12 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/tripleo-common master: Updated from global requirements https://review.openstack.org/467320 | 23:24 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-common master: Replace swiftclient with openstackclient https://review.openstack.org/467800 | 23:35 |
| *** pchavva has joined #tripleo | 23:36 | |
| *** fragatina has quit IRC | 23:48 | |
| openstackgerrit | Merged openstack/instack master: Updated from global requirements https://review.openstack.org/454481 | 23:50 |
| *** ecerquei_ has joined #tripleo | 23:51 | |
| *** yamahata has joined #tripleo | 23:51 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!