| *** http_GK1wmSU has joined #tripleo | 00:04 | |
| *** http_GK1wmSU has left #tripleo | 00:05 | |
| *** ooolpbot has joined #tripleo | 00:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 00:10 |
|---|---|---|
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1708146 | 00:10 |
| openstack | Launchpad bug 1708146 in tripleo "OVB periodic jobs fail in overcloud deploy task with timeout waiting for messages" [Critical,Triaged] | 00:10 |
| *** ooolpbot has quit IRC | 00:10 | |
| *** psahoo has joined #tripleo | 00:26 | |
| openstackgerrit | Dan Sneddon proposed openstack/tripleo-heat-templates master: Convert controller-role.yaml to role.role.j2.yaml https://review.openstack.org/480272 | 00:29 |
| *** rwsu has quit IRC | 00:38 | |
| *** gbarros has joined #tripleo | 00:38 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/tripleo-heat-templates master: Updated from global requirements https://review.openstack.org/488148 | 00:42 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Fix keystone, cinder, heat-api cron containers https://review.openstack.org/489419 | 00:44 |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Enable tempest to run with whitelist https://review.openstack.org/485252 | 00:44 |
| openstackgerrit | Merged openstack/os-net-config stable/newton: Use os-net-config mapping file when interface in bond or bridge https://review.openstack.org/489709 | 00:45 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Update TLS-everywhere docker environment https://review.openstack.org/489479 | 00:45 |
| openstackgerrit | Merged openstack/tripleo-ui master: Implement Zaqar logger adapter https://review.openstack.org/459294 | 00:45 |
| openstackgerrit | Merged openstack/tripleo-quickstart master: Enable tempest on scenario001-multinode job https://review.openstack.org/484788 | 00:45 |
| *** limao has joined #tripleo | 00:46 | |
| openstackgerrit | Merged openstack/tripleo-quickstart master: Prepare pike.yaml config for TripleO CI https://review.openstack.org/490066 | 00:46 |
| *** nyechiel has quit IRC | 00:50 | |
| *** chem has quit IRC | 00:53 | |
| *** psahoo has quit IRC | 00:53 | |
| EmilienM | ianw: tripleoclient job still failing :( | 00:55 |
| *** dmacpher has quit IRC | 00:56 | |
| EmilienM | sounds like diskimage-builder==2.8.0 | 00:56 |
| EmilienM | so we still have a problem :( | 00:56 |
| ianw | EmilienM: ok ... can you point me at the logs? | 00:56 |
| EmilienM | http://logs.openstack.org/88/487888/1/check/gate-tripleo-buildimage-overcloud-full-centos-7/389cfe8/console.html#_2017-08-02_23_54_20_788146 | 00:57 |
| EmilienM | stevebaker: ^ | 00:57 |
| EmilienM | I'm afk 1h, I'm back after | 00:57 |
| stevebaker | ianw: what/when rebuilds the nodepool nodes? | 00:58 |
| *** dixiaoli has joined #tripleo | 00:58 | |
| ianw | stevebaker: oh, well yeah that will happen tomorrow | 00:58 |
| stevebaker | ianw: I mean the images that nodepool nodes boot with | 00:59 |
| ianw | i could kick of an asynchronous build | 00:59 |
| stevebaker | ianw: if that is a done thing, it would be useful | 01:00 |
| EmilienM | are we sure we don't already have the lastest version downloaeded? | 01:01 |
| EmilienM | I see in logs 2.8.0 | 01:01 |
| openstackgerrit | Merged openstack/puppet-tripleo master: Use normal socket file permissions instead of polkit https://review.openstack.org/479816 | 01:02 |
| *** mdnadeem has joined #tripleo | 01:02 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Cinder volume/backup containers shouldn't mount two paths at same point https://review.openstack.org/489825 | 01:02 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Set redis password hiera value in compute agent https://review.openstack.org/487480 | 01:02 |
| stevebaker | EmilienM: the fix is for the image built for nodepool to boot from, not the image built in the CI job itself | 01:03 |
| ianw | EmilienM: but the missing file is on the build host. i mean ... perhaps tripleo should be installing it, which would also work ... but i do think it can be considered part of the base image | 01:03 |
| EmilienM | ah, ok | 01:03 |
| EmilienM | I see, thanks | 01:03 |
| *** psahoo has joined #tripleo | 01:03 | |
| ianw | stevebaker: puppet hasn't rolled out the new version to the builders yet, but when it has, i can trigger a build | 01:04 |
| stevebaker | ianw: ok, thanks | 01:05 |
| stevebaker | EmilienM: in the meantime we could just install policycoreutils-python here http://logs.openstack.org/88/487888/1/check/gate-tripleo-buildimage-overcloud-full-centos-7/389cfe8/console.html#_2017-08-02_23_38_54_346106 | 01:05 |
| *** ooolpbot has joined #tripleo | 01:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1708146 | 01:10 |
| *** ooolpbot has quit IRC | 01:10 | |
| openstack | Launchpad bug 1708146 in tripleo "OVB periodic jobs fail in overcloud deploy task with timeout waiting for messages" [Critical,Triaged] | 01:10 |
| openstackgerrit | Steve Baker proposed openstack/python-tripleoclient master: Install policycoreutils-python https://review.openstack.org/490265 | 01:14 |
| openstackgerrit | Steve Baker proposed openstack/python-tripleoclient master: Add --set argument to prepare command https://review.openstack.org/489083 | 01:18 |
| *** yamahata has quit IRC | 01:20 | |
| openstackgerrit | Steve Baker proposed openstack/python-tripleoclient master: Minor fixes for container image prepare command https://review.openstack.org/484061 | 01:21 |
| *** tzumainn has quit IRC | 01:32 | |
| *** noslzzp has quit IRC | 01:34 | |
| *** gbarros has quit IRC | 01:38 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Make RoleParameters and key_name descriptions consistent https://review.openstack.org/483539 | 01:40 |
| stevebaker | ianw, EmilienM: No need for an async image build I think, this fixes the job for now https://review.openstack.org/#/c/490265 | 01:42 |
| *** ansmith has quit IRC | 01:43 | |
| ianw | stevebaker: i kicked off a build anyway, new version just rolled out -> http://nodepool.openstack.org/dib-image-list | 01:47 |
| stevebaker | ianw: ok, ta | 01:47 |
| *** ramishra has joined #tripleo | 01:51 | |
| *** gbarros has joined #tripleo | 01:57 | |
| *** ansmith has joined #tripleo | 02:02 | |
| *** itlinux has joined #tripleo | 02:02 | |
| *** dmacpher has joined #tripleo | 02:04 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/tripleo-heat-templates master: Updated from global requirements https://review.openstack.org/488148 | 02:07 |
| *** ooolpbot has joined #tripleo | 02:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1707788 | 02:10 |
| openstack | Launchpad bug 1707788 in tripleo "gate-tripleo-buildimage-overcloud-full-centos-7 is broken on missing semanage" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker) | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1708146 | 02:10 |
| *** ooolpbot has quit IRC | 02:10 | |
| openstack | Launchpad bug 1708146 in tripleo "OVB periodic jobs fail in overcloud deploy task with timeout waiting for messages" [Critical,Triaged] | 02:10 |
| *** jrist has joined #tripleo | 02:11 | |
| *** jrist has joined #tripleo | 02:11 | |
| *** trozet has quit IRC | 02:16 | |
| *** itlinux has quit IRC | 02:50 | |
| *** kbyrne has quit IRC | 02:54 | |
| *** kbyrne has joined #tripleo | 02:58 | |
| *** dhill_ has quit IRC | 03:02 | |
| *** dhill_ has joined #tripleo | 03:06 | |
| *** ansmith has quit IRC | 03:06 | |
| *** ansmith has joined #tripleo | 03:07 | |
| *** ooolpbot has joined #tripleo | 03:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1707788 | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1708146 | 03:10 |
| *** ooolpbot has quit IRC | 03:10 | |
| openstack | Launchpad bug 1707788 in tripleo "gate-tripleo-buildimage-overcloud-full-centos-7 is broken on missing semanage" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker) | 03:10 |
| openstack | Launchpad bug 1708146 in tripleo "OVB periodic jobs fail in overcloud deploy task with timeout waiting for messages" [Critical,Triaged] | 03:10 |
| *** mmethot_ has joined #tripleo | 03:18 | |
| *** mmethot_ has quit IRC | 03:19 | |
| *** mmethot_ has joined #tripleo | 03:20 | |
| *** _mmethot_ has quit IRC | 03:22 | |
| *** dixiaoli has quit IRC | 03:28 | |
| *** udesale has joined #tripleo | 03:34 | |
| *** gkadam-afk has joined #tripleo | 03:37 | |
| *** ykarel has joined #tripleo | 03:39 | |
| *** ykarel has quit IRC | 03:46 | |
| *** ykarel has joined #tripleo | 03:46 | |
| *** _mmethot_ has joined #tripleo | 03:49 | |
| *** mmethot_ has quit IRC | 03:53 | |
| *** links has joined #tripleo | 03:54 | |
| *** gbarros has quit IRC | 03:54 | |
| *** dbecker has quit IRC | 03:54 | |
| EmilienM | michapma: what's up with neutron/odl? | 04:03 |
| EmilienM | stevebaker: excellent fix, thank you | 04:03 |
| EmilienM | stevebaker: and thanks for self approving this one, no problem at all | 04:03 |
| stevebaker | EmilienM: its going to need a recheck anyway | 04:03 |
| EmilienM | ah, why? | 04:03 |
| EmilienM | we'll probably need to backport it | 04:04 |
| stevebaker | EmilienM: undercloud-containers job failure | 04:04 |
| *** itlinux has joined #tripleo | 04:06 | |
| *** dbecker has joined #tripleo | 04:06 | |
| *** limao has quit IRC | 04:09 | |
| *** ooolpbot has joined #tripleo | 04:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1707788 | 04:10 |
| openstack | Launchpad bug 1707788 in tripleo "gate-tripleo-buildimage-overcloud-full-centos-7 is broken on missing semanage" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker) | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1708146 | 04:10 |
| *** ooolpbot has quit IRC | 04:10 | |
| openstack | Launchpad bug 1708146 in tripleo "OVB periodic jobs fail in overcloud deploy task with timeout waiting for messages" [Critical,Triaged] | 04:10 |
| *** limao has joined #tripleo | 04:17 | |
| *** limao has quit IRC | 04:18 | |
| *** limao has joined #tripleo | 04:18 | |
| *** limao has quit IRC | 04:19 | |
| *** limao has joined #tripleo | 04:19 | |
| *** limao has quit IRC | 04:20 | |
| *** limao has joined #tripleo | 04:20 | |
| *** limao has quit IRC | 04:21 | |
| *** limao has joined #tripleo | 04:21 | |
| *** limao has quit IRC | 04:21 | |
| *** limao has joined #tripleo | 04:22 | |
| *** ramishra has quit IRC | 04:22 | |
| *** limao has quit IRC | 04:22 | |
| *** limao has joined #tripleo | 04:23 | |
| *** limao has quit IRC | 04:23 | |
| *** ramishra has joined #tripleo | 04:24 | |
| *** limao has joined #tripleo | 04:24 | |
| michapma | EmilienM: the CI job still has that dependency on websockets missing afaik | 04:24 |
| *** limao has quit IRC | 04:24 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Render isolated network templates using jinja2 https://review.openstack.org/475612 | 04:30 |
| EmilienM | michapma: right, nevermind my question :) | 04:31 |
| michapma | EmilienM: no worries :D do you know when the new packages will be promoted? | 04:31 |
| EmilienM | michapma: I hope tonight | 04:32 |
| *** yamahata has joined #tripleo | 04:32 | |
| EmilienM | but it seems like we have 2 bugs in alerts, have you seen them ^ | 04:32 |
| EmilienM | they block promotion I'm pretty sure | 04:32 |
| *** janki has joined #tripleo | 04:34 | |
| *** shreshtha has joined #tripleo | 04:36 | |
| *** ecerquei has quit IRC | 04:36 | |
| michapma | EmilienM: looks like the first one is fixed but the heat one is ongoing. I'm a bit out of my depth on that so I don't think I can help. | 04:36 |
| *** skramaja has joined #tripleo | 04:38 | |
| *** ansmith has quit IRC | 04:46 | |
| *** ansmith has joined #tripleo | 04:48 | |
| weshay_pto | EmilienM, you on | 04:52 |
| weshay_pto | ? | 04:52 |
| *** _mmethot_ has quit IRC | 04:56 | |
| *** _mmethot_ has joined #tripleo | 04:57 | |
| *** limao has joined #tripleo | 04:57 | |
| *** _mmethot_ has quit IRC | 04:58 | |
| *** mmethot has joined #tripleo | 05:03 | |
| *** mmethot has quit IRC | 05:03 | |
| *** mmethot has joined #tripleo | 05:04 | |
| *** udesale__ has joined #tripleo | 05:05 | |
| *** udesale has quit IRC | 05:08 | |
| *** ooolpbot has joined #tripleo | 05:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1707788 | 05:10 |
| openstack | Launchpad bug 1707788 in tripleo "gate-tripleo-buildimage-overcloud-full-centos-7 is broken on missing semanage" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker) | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1708146 | 05:10 |
| *** ooolpbot has quit IRC | 05:10 | |
| openstack | Launchpad bug 1708146 in tripleo "OVB periodic jobs fail in overcloud deploy task with timeout waiting for messages" [Critical,Triaged] | 05:10 |
| *** ratailor has joined #tripleo | 05:11 | |
| *** ansmith has quit IRC | 05:12 | |
| *** pgadiya has joined #tripleo | 05:15 | |
| *** pdeore has joined #tripleo | 05:19 | |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart-extras master: Move subunit test results generation pieces in tempest-results https://review.openstack.org/486965 | 05:25 |
| *** dpawar has joined #tripleo | 05:28 | |
| *** dpawar has quit IRC | 05:28 | |
| *** ansmith has joined #tripleo | 05:30 | |
| *** marios has joined #tripleo | 05:30 | |
| *** prateek has joined #tripleo | 05:31 | |
| jaosorior | bandini: hey dude, I'm a little bit confused on what's missing here https://review.openstack.org/#/c/489132/3/puppet/services/pacemaker.yaml | 05:33 |
| *** dsariel has joined #tripleo | 05:36 | |
| *** itlinux has quit IRC | 05:37 | |
| *** mandre has joined #tripleo | 05:43 | |
| jaosorior | mandre: hey, any idea what's missing from this patch? https://review.openstack.org/#/c/475366/ | 05:49 |
| *** dpawar has joined #tripleo | 05:50 | |
| *** psachin has joined #tripleo | 05:51 | |
| *** cshastri has joined #tripleo | 05:53 | |
| mandre | jaosorior: hey! nothing strikes me as wrong or missing in that patch | 05:55 |
| *** masco has joined #tripleo | 05:55 | |
| jaosorior | mandre: morning dude! | 05:55 |
| jaosorior | mandre: alright. I'm trying it out locally now. Just wanted to know if something obvious is missing | 05:55 |
| mandre | jaosorior: but I just came back from PTO so I don't know about the status of the CI | 05:56 |
| jaosorior | welcome back | 05:56 |
| jaosorior | how did the PTO go? | 05:56 |
| mandre | jaosorior: very well, thank you! | 05:57 |
| *** agurenko has joined #tripleo | 05:57 | |
| mandre | jaosorior: the only thing to keep in mind when changing the from eventlet to apache is to change the user that runs inside the container to root, and I see you did it already | 05:58 |
| jaosorior | right | 05:58 |
| *** brault has joined #tripleo | 05:58 | |
| jaosorior | mandre: so, this patch is on top of the one I just passed you https://review.openstack.org/#/c/489885/ and had better luck in CI (CI had been broken for a bit, but now it's fixed) | 05:59 |
| jaosorior | mandre: and when trying to access nova I get a 503. Meaning the vhost is not actually running | 05:59 |
| jaosorior | mandre: seems there are no httpd logs for the nova container :/ so yeah. something went wrong and I don't know what it is. | 05:59 |
| jaosorior | I guess I'll know when this finishes deploying | 06:00 |
| openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart-extras master: GATE CHECK for quickstart-extras https://review.openstack.org/472607 | 06:00 |
| jaosorior | is there a solution already for getting the logs from containers? I see there is the /var/log/containers folder which has the logs for the service it's running ( keystone.log, nova-compute.log, etc. ) but it doesn't seem to take into account the httpd logs | 06:00 |
| mandre | jaosorior: hmm, no httpd logs you say? has anything changed with log collection? | 06:01 |
| *** marios has quit IRC | 06:01 | |
| openstackgerrit | Debo Zhang proposed openstack/python-tripleoclient master: Change tag link in README to the newest address https://review.openstack.org/490337 | 06:01 |
| *** lmiccini has joined #tripleo | 06:01 | |
| *** marios has joined #tripleo | 06:02 | |
| jaosorior | mandre: not sure. | 06:02 |
| *** mwhahaha has quit IRC | 06:02 | |
| jaosorior | mandre: at least I couldn't find them. | 06:02 |
| mandre | jaosorior: ack, I'll see if there is anything that needs to be done wrt http logs after I've done a first pass on my email inbox | 06:04 |
| *** dsariel has quit IRC | 06:06 | |
| *** jaganathan has joined #tripleo | 06:09 | |
| *** ooolpbot has joined #tripleo | 06:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1707788 | 06:10 |
| openstack | Launchpad bug 1707788 in tripleo "gate-tripleo-buildimage-overcloud-full-centos-7 is broken on missing semanage" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker) | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1708146 | 06:10 |
| *** ooolpbot has quit IRC | 06:10 | |
| openstack | Launchpad bug 1708146 in tripleo "OVB periodic jobs fail in overcloud deploy task with timeout waiting for messages" [Critical,Triaged] | 06:10 |
| marios | /exit | 06:11 |
| *** marios has quit IRC | 06:11 | |
| *** marios has joined #tripleo | 06:11 | |
| *** pgadiya has quit IRC | 06:18 | |
| *** rcernin has joined #tripleo | 06:19 | |
| *** dpawar has left #tripleo | 06:20 | |
| *** dsariel has joined #tripleo | 06:21 | |
| *** gkadam-afk is now known as gkadam | 06:22 | |
| openstackgerrit | Debo Zhang proposed openstack/puppet-tripleo master: Update link addresses in README.md https://review.openstack.org/490339 | 06:22 |
| *** iranzo has joined #tripleo | 06:23 | |
| *** iranzo has joined #tripleo | 06:23 | |
| *** jfrancoa has joined #tripleo | 06:26 | |
| *** pgadiya has joined #tripleo | 06:34 | |
| *** rlandy has joined #tripleo | 06:37 | |
| *** rlandy is now known as rlandy|ci_mtgs | 06:37 | |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates stable/newton: Ensure yum cache is ready before update https://review.openstack.org/490347 | 06:39 |
| *** iranzo has quit IRC | 06:42 | |
| openstackgerrit | Honza Pokorny proposed openstack/tripleo-ui master: Add logging documentation https://review.openstack.org/489134 | 06:43 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Fix CA file bind mounting in containers https://review.openstack.org/489509 | 06:46 |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Better long validation task management https://review.openstack.org/490098 | 06:48 |
| *** pgadiya has quit IRC | 06:48 | |
| *** pcaruana has joined #tripleo | 06:48 | |
| *** jaganathan has quit IRC | 06:51 | |
| *** jaganathan has joined #tripleo | 06:52 | |
| *** coolsvap has joined #tripleo | 06:53 | |
| *** limao has quit IRC | 06:56 | |
| *** pgadiya has joined #tripleo | 07:00 | |
| *** jlinkes has joined #tripleo | 07:01 | |
| *** mcornea has joined #tripleo | 07:03 | |
| *** coolsvap has left #tripleo | 07:04 | |
| *** jtomasek has joined #tripleo | 07:09 | |
| openstackgerrit | Marius Cornea proposed openstack/tripleo-heat-templates master: Stop and disable openstack-nova-compute service on compute nodes https://review.openstack.org/490226 | 07:09 |
| *** ooolpbot has joined #tripleo | 07:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1707788 | 07:10 |
| openstack | Launchpad bug 1707788 in tripleo "gate-tripleo-buildimage-overcloud-full-centos-7 is broken on missing semanage" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker) | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1708146 | 07:10 |
| *** ooolpbot has quit IRC | 07:10 | |
| openstack | Launchpad bug 1708146 in tripleo "OVB periodic jobs fail in overcloud deploy task with timeout waiting for messages" [Critical,Triaged] | 07:10 |
| openstackgerrit | Martin André proposed openstack/puppet-tripleo master: Configure dockerd with --iptables=false https://review.openstack.org/490201 | 07:12 |
| *** tesseract has joined #tripleo | 07:17 | |
| *** udesale__ has quit IRC | 07:18 | |
| *** udesale__ has joined #tripleo | 07:19 | |
| *** ebarrera has joined #tripleo | 07:19 | |
| *** ccamacho has joined #tripleo | 07:20 | |
| *** paramite has joined #tripleo | 07:20 | |
| *** anshulbehl has joined #tripleo | 07:22 | |
| *** limao has joined #tripleo | 07:25 | |
| *** yprokule has joined #tripleo | 07:25 | |
| *** amoralej|off is now known as amoralej | 07:28 | |
| *** snecklifter has quit IRC | 07:30 | |
| openstackgerrit | Arx Cruz proposed openstack/tripleo-quickstart-extras master: Install openstack services tempest plugin https://review.openstack.org/490369 | 07:36 |
| *** jpena|off is now known as jpena | 07:39 | |
| *** cylopez has joined #tripleo | 07:42 | |
| jaosorior | dciabrin: ping | 07:43 |
| jaosorior | dciabrin: did you figure out the issue from yesterday? Or should we open a tmate? | 07:43 |
| *** egonzalez has joined #tripleo | 07:43 | |
| *** ykarel has quit IRC | 07:45 | |
| arxcruz | chandankumar: hello | 07:45 |
| arxcruz | chandankumar: do you know which tempest plugin package has the TelemetryAlarming tests ? | 07:45 |
| *** oidgar has joined #tripleo | 07:47 | |
| openstackgerrit | Arx Cruz proposed openstack/tripleo-quickstart-extras master: Install openstack services tempest plugin https://review.openstack.org/490369 | 07:48 |
| *** ansmith has quit IRC | 07:49 | |
| *** ansmith has joined #tripleo | 07:50 | |
| chandankumar | arxcruz: hello | 07:52 |
| arxcruz | chandankumar: don't worry, I figured out :) | 07:52 |
| arxcruz | thanks | 07:52 |
| chandankumar | arxcruz: python-aodh-tests and python-ceilometer-tests | 07:52 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo master: Use haproxy::endpoint resource for horizon https://review.openstack.org/489592 | 07:52 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo master: Enable TLS in the internal network for horizon https://review.openstack.org/489593 | 07:53 |
| openstackgerrit | Arx Cruz proposed openstack/tripleo-quickstart master: Configure the set of tests to execute in scenario001 https://review.openstack.org/490376 | 07:57 |
| chandankumar | arxcruz: https://review.openstack.org/#/c/490369/2/roles/validate-tempest/tasks/pre-tempest.yml@35 change state to latest | 07:59 |
| chandankumar | if package is already there, then it will update it | 07:59 |
| chandankumar | if not then it will install it | 07:59 |
| openstackgerrit | Arx Cruz proposed openstack/tripleo-quickstart-extras master: Install openstack services tempest plugin https://review.openstack.org/490369 | 08:00 |
| chandankumar | arxcruz: you are so quick | 08:00 |
| *** ramishra has quit IRC | 08:01 | |
| *** mmethot has quit IRC | 08:02 | |
| *** mwhahaha has joined #tripleo | 08:03 | |
| *** limao has quit IRC | 08:05 | |
| *** limao has joined #tripleo | 08:05 | |
| *** ramishra has joined #tripleo | 08:08 | |
| *** limao has quit IRC | 08:09 | |
| jtomasek | honza: have you seen this? I am getting 7 failed tests when I rebase my error handling patch on top of current master http://paste.openstack.org/show/617375/ | 08:10 |
| *** ooolpbot has joined #tripleo | 08:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1707788 | 08:10 |
| openstack | Launchpad bug 1707788 in tripleo "gate-tripleo-buildimage-overcloud-full-centos-7 is broken on missing semanage" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker) | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1708146 | 08:10 |
| *** ooolpbot has quit IRC | 08:10 | |
| openstack | Launchpad bug 1708146 in tripleo "OVB periodic jobs fail in overcloud deploy task with timeout waiting for messages" [Critical,Triaged] | 08:10 |
| honza | jtomasek: looks like a recursive import, this can usually be solved by reordering imports in tests | 08:11 |
| *** limao has joined #tripleo | 08:11 | |
| honza | jtomasek: try moving the store import to the top | 08:11 |
| jtomasek | crazy | 08:11 |
| chandankumar | arxcruz: do we have ping test for undercloud also? | 08:11 |
| honza | jtomasek: but it's weird that it passed the gate --- i'm checking locally now, too | 08:12 |
| jtomasek | honza: I don't see it on master | 08:12 |
| jtomasek | honza: maybe I've moved the store import back to the bottom when I did the rebase | 08:12 |
| honza | jtomasek: ah, sorry, didn't read that properly | 08:12 |
| jtomasek | checking now | 08:12 |
| *** hewbrocca_afk is now known as hewbrocca | 08:12 | |
| jaosorior | mandre: where can I find nowadays the container configuration? | 08:14 |
| jaosorior | for isntance, for nova | 08:15 |
| *** limao has quit IRC | 08:15 | |
| mandre | jaosorior: that would be http://logs.openstack.org/85/489885/1/check-tripleo/gate-tripleo-ci-centos-7-ovb-containers-oooq/06a957f/logs/overcloud-controller-0/var/log/config-data/nova/etc/ | 08:15 |
| honza | yeah, master is fine for me | 08:15 |
| *** panda has joined #tripleo | 08:16 | |
| jaosorior | mandre: I see that keystone has the httpd configuration there (in the config-data directory) but nova doesn't | 08:17 |
| jaosorior | seems I might be missing something :/ | 08:17 |
| *** derekh has joined #tripleo | 08:18 | |
| *** openstackgerrit has quit IRC | 08:18 | |
| *** udesale has joined #tripleo | 08:18 | |
| *** snecklifter has joined #tripleo | 08:19 | |
| mandre | jaosorior: wait, are you using a container image that has apache to generate the configuration? | 08:19 |
| jaosorior | mandre: I'm using the image that was uploaded, as far as I can remember we had enabled httpd there. | 08:20 |
| *** udesale__ has quit IRC | 08:20 | |
| jaosorior | unless we're removing it somewhere? | 08:20 |
| *** ukalifon has joined #tripleo | 08:20 | |
| *** dmacpher has quit IRC | 08:21 | |
| jaosorior | which we're not as far as I can tell | 08:21 |
| *** chem has joined #tripleo | 08:22 | |
| *** bogdando has joined #tripleo | 08:23 | |
| *** openstackgerrit has joined #tripleo | 08:24 | |
| openstackgerrit | Honza Pokorny proposed openstack/tripleo-common master: Add GUI logging workflows https://review.openstack.org/469196 | 08:24 |
| mandre | jaosorior: just had a quick look at the code, nova config is generated using DockerNovaConfigImage which defaults to centos-binary-nova-base | 08:24 |
| arxcruz | chandankumar: I believe yes, but we don't test | 08:25 |
| jaosorior | mandre: | 08:25 |
| jaosorior | dude | 08:25 |
| jaosorior | you're right! | 08:25 |
| mandre | jaosorior: I don't think nova-base has http | 08:25 |
| jaosorior | it doesn't | 08:25 |
| jaosorior | only nova-api does | 08:25 |
| jaosorior | mandre: what do you recommend? should we do a DockerNovaApiConfigImage parameter and use that instead? | 08:26 |
| jaosorior | mandre: yeah, placement uses it's own | 08:26 |
| jaosorior | *its | 08:26 |
| marios | soc_off: o/ hey man was just wondering if this one was forgotten or we didn't want it for some reason https://review.openstack.org/490347 | 08:28 |
| mandre | jaosorior: probably it's better to change the default image for the nova config to be nova-api, I don't see why we would have a different config-volume just for nova-api | 08:28 |
| marios | soc_off: while we still have a stable/newton may as well , but thought to check | 08:28 |
| *** lucas-afk is now known as lucasagomes | 08:28 | |
| *** paken has joined #tripleo | 08:28 | |
| soc_off | marios: you are right | 08:28 |
| mandre | jaosorior: not sure if it's all wired up correctly now, but normally you should just have to update https://github.com/openstack/tripleo-common/blob/master/container-images/overcloud_containers.yaml.j2 | 08:29 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Enable TLS for nova api and placement containers https://review.openstack.org/489885 | 08:29 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Make containerized nova-api run with httpd https://review.openstack.org/475366 | 08:29 |
| soc_off | marios: +1 there, that review should go in | 08:29 |
| jaosorior | mandre: lets try that | 08:29 |
| marios | ack thanks for checking soc_off | 08:30 |
| mandre | jaosorior: just deleting lines 235-237 should do it I think | 08:31 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-common master: Use nova-api image for configuration as well https://review.openstack.org/490385 | 08:32 |
| honza | jtomasek: any luck? | 08:32 |
| jaosorior | mandre: ^^ | 08:32 |
| *** milan has joined #tripleo | 08:32 | |
| jaosorior | what about that? | 08:32 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Enable TLS for nova api and placement containers https://review.openstack.org/489885 | 08:33 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Make containerized nova-api run with httpd https://review.openstack.org/475366 | 08:33 |
| *** jaganathan has quit IRC | 08:33 | |
| *** jaganathan has joined #tripleo | 08:34 | |
| jtomasek | honza: not so much, but I bet it is related to the fact that we import store in different places in UI and we call the logger.setReduxDispatch there | 08:34 |
| jtomasek | honza: I have no idea why this does not occur in master | 08:35 |
| honza | jtomasek: is it broken for you only in tests or also in the browser? | 08:35 |
| jtomasek | honza: my patch is not related to any of this | 08:35 |
| * jtomasek checks | 08:35 | |
| *** bogdando has quit IRC | 08:35 | |
| jtomasek | honza: just tests | 08:36 |
| openstackgerrit | Debo Zhang proposed openstack/tripleo-heat-templates master: Update documentation links in README https://review.openstack.org/490388 | 08:37 |
| jtomasek | honza: app works fine | 08:37 |
| honza | jtomasek: yeah, i have definitely seen this before | 08:37 |
| *** bogdando has joined #tripleo | 08:39 | |
| *** zoli is now known as zli | 08:42 | |
| *** zli is now known as zoli | 08:42 | |
| *** suuuper has joined #tripleo | 08:46 | |
| openstackgerrit | Merged openstack/tripleo-puppet-elements master: Adds puppet-odl module to overcloud image https://review.openstack.org/481433 | 08:48 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Ceilometer] https://review.openstack.org/490392 | 08:53 |
| *** sri_ has joined #tripleo | 08:53 | |
| mandre | jaosorior: commented on the review, your solution won't work | 08:54 |
| rdopiera | did anybody manage to install an older version of osp (like osp10) with tripleo-quickstart? I'm getting 404 errors on trunk.rdoproject.org/centos7 | 08:57 |
| *** apetrich has quit IRC | 08:58 | |
| adarazs | rdopiera: for OSP you need the downstream documentation. | 08:59 |
| rdopiera | adarazs: I'm using it | 08:59 |
| rdopiera | adarazs: but there is no downstream tripleo channel to ask at | 09:00 |
| adarazs | rdopiera: there's #rhos-dev / #rhos-mgt :) | 09:00 |
| *** nyechiel has joined #tripleo | 09:00 | |
| *** ykarel has joined #tripleo | 09:00 | |
| rdopiera | adarazs: yeah, asked there, got ignored | 09:01 |
| adarazs | that's the downstream channel though :) | 09:01 |
| rdopiera | well, it's not for tripleo specifically | 09:01 |
| *** isq_ has joined #tripleo | 09:02 | |
| *** isq has quit IRC | 09:03 | |
| *** ooolpbot has joined #tripleo | 09:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1707788 | 09:10 |
| openstack | Launchpad bug 1707788 in tripleo "gate-tripleo-buildimage-overcloud-full-centos-7 is broken on missing semanage" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker) | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1708146 | 09:10 |
| *** ooolpbot has quit IRC | 09:10 | |
| openstack | Launchpad bug 1708146 in tripleo "OVB periodic jobs fail in overcloud deploy task with timeout waiting for messages" [Critical,Triaged] | 09:10 |
| *** ykarel has quit IRC | 09:12 | |
| *** akrivoka has joined #tripleo | 09:14 | |
| openstackgerrit | Numan Siddique proposed openstack/tripleo-heat-templates master: Support deploying OVN as container services https://review.openstack.org/468860 | 09:16 |
| *** ffiore has joined #tripleo | 09:18 | |
| *** ansmith has quit IRC | 09:25 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/tripleo-ui master: Imported Translations from Zanata https://review.openstack.org/488357 | 09:25 |
| *** ansmith has joined #tripleo | 09:26 | |
| *** iranzo has joined #tripleo | 09:28 | |
| *** bogdando has quit IRC | 09:29 | |
| *** salmankhan has joined #tripleo | 09:29 | |
| *** psahoo has quit IRC | 09:31 | |
| *** ffiore has quit IRC | 09:31 | |
| openstackgerrit | Merged openstack/python-tripleoclient master: Skip empty resource_registry sections https://review.openstack.org/489258 | 09:33 |
| *** iranzo has quit IRC | 09:33 | |
| *** fzdarsky has joined #tripleo | 09:36 | |
| *** stendulker has joined #tripleo | 09:36 | |
| *** apetrich has joined #tripleo | 09:44 | |
| openstackgerrit | Attila Darazs proposed openstack-infra/tripleo-ci master: TEST: DONT RECHECK or REVIEW: periodic jobs https://review.openstack.org/359215 | 09:45 |
| openstackgerrit | Sanjay Upadhyay proposed openstack/tripleo-quickstart master: This adds a variable to add stack user to the undercloud image https://review.openstack.org/490407 | 09:46 |
| *** jaganathan has quit IRC | 09:46 | |
| *** jaganathan has joined #tripleo | 09:47 | |
| *** tosky has joined #tripleo | 09:47 | |
| *** strigazi_OFF has quit IRC | 09:49 | |
| openstackgerrit | Sanjay Upadhyay proposed openstack/tripleo-quickstart master: This adds a variable to add stack user to the undercloud image https://review.openstack.org/490407 | 09:50 |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart-extras master: [WIP]Enable additional params for scenario/tempest ping test. https://review.openstack.org/487127 | 09:50 |
| *** strigazi_OFF has joined #tripleo | 09:50 | |
| *** ramishra has quit IRC | 09:50 | |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Ceilometer] https://review.openstack.org/490392 | 09:51 |
| *** ramishra has joined #tripleo | 09:53 | |
| *** yamahata has quit IRC | 09:55 | |
| openstackgerrit | Lee Yarwood proposed openstack/instack-undercloud master: WIP zaqar: restorecon zaqar.log to ensure httpd has access https://review.openstack.org/490409 | 09:56 |
| *** ramishra has quit IRC | 09:57 | |
| jtomasek | honza: ok, so this is example of a failing test: http://paste.openstack.org/show/617393/ | 09:59 |
| *** ansmith has quit IRC | 09:59 | |
| jtomasek | honza: the stack trace goes by imports | 09:59 |
| jtomasek | honza: if I add import store to RolesActions.tests.js, the test passes | 10:00 |
| honza | jtomasek: yep | 10:00 |
| *** dsariel has quit IRC | 10:00 | |
| jtomasek | honza: but I cant understand why | 10:00 |
| *** ramishra has joined #tripleo | 10:00 | |
| honza | jtomasek: because the store.js file has to be the first one to import the logging service so that the dispatch function is available | 10:02 |
| honza | jtomasek: i think it's caused by the way that imports work/don't work in webpack (or whatever) | 10:02 |
| honza | jtomasek: i.e. it can't detect circular imports accurately | 10:02 |
| honza | jtomasek: but yeah, it's not ideal in tests; we don't want to have to set up any unnecessary state | 10:05 |
| *** katkapilatova has joined #tripleo | 10:09 | |
| *** dtantsur|afk is now known as dtantsur | 10:10 | |
| *** ooolpbot has joined #tripleo | 10:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1707788 | 10:10 |
| openstack | Launchpad bug 1707788 in tripleo "gate-tripleo-buildimage-overcloud-full-centos-7 is broken on missing semanage" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker) | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1708146 | 10:10 |
| *** ooolpbot has quit IRC | 10:10 | |
| openstack | Launchpad bug 1708146 in tripleo "OVB periodic jobs fail in overcloud deploy task with timeout waiting for messages" [Critical,Triaged] | 10:10 |
| honza | jtomasek: one idea to solve this: make the logging service part of the store | 10:10 |
| honza | jtomasek: => you can't have a logger without the store and vice versa | 10:10 |
| honza | jtomasek: am i making any sense? :) | 10:11 |
| jtomasek | honza: I am having hard time to understand this, but am I correct that if we import LoggerService, we get a logger instance | 10:12 |
| *** ansmith has joined #tripleo | 10:12 | |
| jtomasek | honza: but in case when we do this anywhere else than the store, the adapters won't get loaded | 10:12 |
| *** dsariel has joined #tripleo | 10:13 | |
| honza | jtomasek: it's a singleton so you get the same instance everywhere... except the store needs to import it first so that we can set the dispatch function | 10:13 |
| honza | so it's a race condition | 10:13 |
| *** ramishra has quit IRC | 10:13 | |
| jtomasek | honza: right, and that is what tests reveal | 10:14 |
| honza | jtomasek: yes, and it works fine in the browser because we're loading the whole application in order; in tests you only load some tiny pieces that you're testing | 10:15 |
| *** ramishra has joined #tripleo | 10:15 | |
| honza | jtomasek: in tests, we could add some beforeEach where the store is set up properly | 10:15 |
| honza | jtomasek: to mimick the real-world app | 10:16 |
| jtomasek | honza: yeah | 10:16 |
| *** dsariel has quit IRC | 10:19 | |
| jtomasek | honza: I think that if we defined a new middleware for logging it would be better as we could pass the dispatch to the LoggingService | 10:19 |
| *** dciabrin_ has joined #tripleo | 10:20 | |
| jtomasek | honza: rather than reusing redux-logger which does not allow loggers to access dispatch | 10:20 |
| honza | jtomasek: but how? you need the store to create middleware which can access the dispatch function, and you need to create the store with the middleware... | 10:22 |
| honza | jtomasek: that's why i was thinking of making our own custom store instance | 10:22 |
| honza | jtomasek: but to be honest, i haven't looked at the machinery in createStore | 10:23 |
| jtomasek | honza: why you need to create store with the middleware? | 10:23 |
| honza | jtomasek: the applyMiddleware call happens when calling createStore or before, right? | 10:24 |
| honza | jtomasek: can you apply middleware after the store is created? | 10:24 |
| * honza confused | 10:25 | |
| * honza reads about redux middleware | 10:26 | |
| openstackgerrit | Arx Cruz proposed openstack/tripleo-quickstart-extras master: Install openstack services tempest plugin https://review.openstack.org/490369 | 10:28 |
| *** hewbrocca is now known as hewbrocca_afk | 10:32 | |
| *** dsariel has joined #tripleo | 10:32 | |
| honza | jtomasek: interesting, i think i know what you mean now; middleware is exactly what we want | 10:33 |
| *** bkopilov_ has quit IRC | 10:35 | |
| jtomasek | honza: then we'd just need to dispatch an agreed type of action when an error is supposed to be logged which the middleware catches | 10:36 |
| *** dsariel has quit IRC | 10:38 | |
| owalsh | jistr: I think we can rebase https://review.openstack.org/469116 now | 10:39 |
| owalsh | jistr: I can take on this if you're busy | 10:40 |
| *** bkopilov_ has joined #tripleo | 10:41 | |
| *** shreshtha has quit IRC | 10:43 | |
| *** suuuper has quit IRC | 10:43 | |
| *** jkilpatr has quit IRC | 10:44 | |
| *** mdnadeem has quit IRC | 10:47 | |
| *** bogdando has joined #tripleo | 10:47 | |
| *** ckyriakidou has joined #tripleo | 10:50 | |
| *** dsariel has joined #tripleo | 10:50 | |
| *** udesale__ has joined #tripleo | 10:52 | |
| *** udesale has quit IRC | 10:53 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates stable/newton: Adjust for new pcs output. https://review.openstack.org/490422 | 10:54 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates stable/newton: Adjust for new pcs output. https://review.openstack.org/490422 | 10:54 |
| *** dciabrin has quit IRC | 10:54 | |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui master: MistralApiService, SwiftApiService error handling https://review.openstack.org/484382 | 10:55 |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui master: IronicInspectorApiService error handling https://review.openstack.org/486977 | 10:55 |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui master: IronicApiService error handling https://review.openstack.org/486976 | 10:55 |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui master: KeystoneApiService error handling https://review.openstack.org/486979 | 10:55 |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui master: HeatApiService error handling https://review.openstack.org/486978 | 10:55 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates stable/newton: Adjust for new pcs output. https://review.openstack.org/490422 | 10:56 |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui master: Fix plan export URL https://review.openstack.org/488164 | 10:58 |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui master: Update updatePlan action https://review.openstack.org/489196 | 10:59 |
| *** dixiaoli has joined #tripleo | 10:59 | |
| *** dixiaoli has quit IRC | 11:04 | |
| *** amoralej is now known as amoralej|lunch | 11:05 | |
| *** hewbrocca_afk is now known as hewbrocca | 11:06 | |
| *** udesale__ has quit IRC | 11:08 | |
| *** agurenko has quit IRC | 11:08 | |
| *** ooolpbot has joined #tripleo | 11:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1707788 | 11:10 |
| openstack | Launchpad bug 1707788 in tripleo "gate-tripleo-buildimage-overcloud-full-centos-7 is broken on missing semanage" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker) | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1708146 | 11:10 |
| *** ooolpbot has quit IRC | 11:10 | |
| openstack | Launchpad bug 1708146 in tripleo "OVB periodic jobs fail in overcloud deploy task with timeout waiting for messages" [Critical,Triaged] | 11:10 |
| *** ansmith has quit IRC | 11:14 | |
| *** stendulker_ has joined #tripleo | 11:15 | |
| *** jaganathan has quit IRC | 11:16 | |
| *** jkilpatr has joined #tripleo | 11:17 | |
| *** stendulker has quit IRC | 11:18 | |
| *** stendulker_ has quit IRC | 11:19 | |
| openstackgerrit | Lee Yarwood proposed openstack/instack-undercloud master: WIP zaqar: restorecon zaqar.log to ensure httpd has access https://review.openstack.org/490409 | 11:20 |
| adarazs | therve, d0ugal, zaneb: SOS, I just reran the periodic promotion job and it still failed in the timeframe (~7 minutes after the start of the overcloud deploy) that looks like the issue with https://bugs.launchpad.net/tripleo/+bug/1708146 | 11:27 |
| openstack | Launchpad bug 1708146 in tripleo "OVB periodic jobs fail in overcloud deploy task with timeout waiting for messages" [Critical,Triaged] | 11:27 |
| adarazs | we really need a solution for this ASAP, it blocks everything down the line :S | 11:28 |
| jistr | owalsh: ok if you could that would be great. I'm looking into creating ssh user for Ansible on split stack envs | 11:29 |
| therve | adarazs, :/ | 11:29 |
| therve | adarazs, Got the logs | 11:30 |
| therve | ? | 11:30 |
| adarazs | therve: not yet, the job is finishing. | 11:30 |
| openstackgerrit | Lee Yarwood proposed openstack/instack-undercloud master: WIP zaqar: restorecon zaqar.log to ensure httpd has access https://review.openstack.org/490409 | 11:32 |
| oidgar | Hi guys, I installed new undercloud then tried to deploy overcloud with containers but it failed at the beginning | 11:34 |
| oidgar | http://paste.openstack.org/show/617408/ | 11:34 |
| adarazs | therve: same thing again: http://logs.openstack.org/15/359215/122/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha-oooq/4ea32d2/logs/undercloud/home/jenkins/overcloud_deploy.log.txt.gz#_2017-08-03_11_18_45 | 11:34 |
| oidgar | has someone else encountered the same problem? | 11:34 |
| dtantsur | hi folks! may I get 2nd +2 on 3 quite simple patches please: https://review.openstack.org/#/c/489228/ https://review.openstack.org/#/c/487166/ https://review.openstack.org/#/c/487175/ | 11:35 |
| * hewbrocca -2 dtantsur | 11:36 | |
| hewbrocca | sorry dtantsur general principles | 11:36 |
| dtantsur | hewbrocca: if you don't behave, I'll tell Piklz! | 11:36 |
| hewbrocca | O NOES /me runs | 11:36 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates stable/newton: Adjust for new pcs output. https://review.openstack.org/490422 | 11:36 |
| dtantsur | :D | 11:36 |
| therve | adarazs, That doesn't make sense :/ | 11:37 |
| *** dsariel has quit IRC | 11:37 | |
| adarazs | therve: so you're saying that everything master (including heat) should not have this problem anymore? | 11:37 |
| *** lmiccini has quit IRC | 11:38 | |
| therve | adarazs, Yeah see the last logs of https://review.openstack.org/#/c/477047/ | 11:38 |
| therve | Well we can't trust what it runs anymore though | 11:39 |
| *** atoth has joined #tripleo | 11:40 | |
| adarazs | oh wow, these are the old, non-oooq jobs :S | 11:40 |
| *** apetrich has quit IRC | 11:41 | |
| *** lmiccini has joined #tripleo | 11:41 | |
| adarazs | therve: looking at the logs, it looks to me that heat-engine is from the latest upstream here instead of built on the job: | 11:42 |
| adarazs | openstack-heat-engine noarch 1:9.0.0-0.20170803031044.1cdd02c.el7.centos | 11:42 |
| therve | adarazs, What's the latest upstream? | 11:42 |
| therve | That id doesn't match anything | 11:43 |
| adarazs | but maybe I don't understand something about how the old jobs work. but we should have the new oooq jobs in the "check experimental" queue. /me goes and fixes that. | 11:43 |
| therve | adarazs, Well there is http://logs.openstack.org/47/477047/20/experimental/gate-tripleo-ci-centos-7-nonha-multinode-oooq/ee7438d/ | 11:43 |
| adarazs | therve: oh wait, never mind. the job did run around 3am UTC, so this package was generated for this job. | 11:43 |
| therve | But it doesn't seem really trust worthy either | 11:43 |
| adarazs | I'll check that out. | 11:44 |
| hewbrocca | adarazs: did we launch the wrong job? | 11:44 |
| adarazs | hewbrocca: no, I checked that the job that failed was with this hash: https://trunk.rdoproject.org/centos7/b3/b7/b3b7d520396888f05dc497da7a509c58541cef6f_d767b7fe/?C=M;O=D -- which has the latest packages from 10:01 AM UTC, including fresh heat packages from 9:40 AM UTC | 11:46 |
| *** agurenko has joined #tripleo | 11:46 | |
| adarazs | so the periodic jobs did run with the latest consistent hash from master. | 11:46 |
| *** fzdarsky has quit IRC | 11:47 | |
| *** mcornea has quit IRC | 11:47 | |
| *** lucasagomes is now known as lucas-hungry | 11:48 | |
| *** ukalifon has quit IRC | 11:48 | |
| therve | adarazs, So https://review.openstack.org/#/c/490438/ | 11:49 |
| therve | We'd need at leas 4h before it lands though :/ | 11:49 |
| therve | adarazs, We used to be able to remove commits from tripleo CI? can we still do that? | 11:49 |
| *** mcornea has joined #tripleo | 11:50 | |
| adarazs | therve: those gates on the heat change run with the latest *promoted* hash, some packages unpinned + that heat change. so if they pass my only idea is that the problem is not with heat but something else. | 11:50 |
| adarazs | https://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/master.yml#L47-L63 -- these are the packages that are unpinned to latest master on the gates. | 11:50 |
| adarazs | *unpinned from the latest promoted. | 11:50 |
| fultonj | jistr: Regarding https://review.openstack.org/#/c/489613, is it easier for you to tell me the extra parameters I need to use it or do you prefer to just make another patch? | 11:51 |
| jistr | fultonj: we'll need probably 3 more patches | 11:51 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates master: Add logrotate with crond service https://review.openstack.org/490050 | 11:51 |
| jistr | fultonj: it's nothing we can write into the workflow directly | 11:51 |
| therve | adarazs, So. The revert https://review.openstack.org/490075 had CI passing | 11:51 |
| therve | adarazs, master had CI failing: https://review.openstack.org/490194 | 11:51 |
| adarazs | therve: I can't selectively include that in a promotion :/ | 11:52 |
| therve | adarazs, And the supposed fix had CI passing too | 11:52 |
| therve | So the outlier is the promotion job here | 11:52 |
| jistr | fultonj: it has to be connected into the CI. I'm working on a bash script that will be generic enough both for users and for CI to enable ssh admin for deployed server environments. Then we'll plug that into TOCI and OOOQ. | 11:52 |
| adarazs | hmmm. | 11:52 |
| therve | adarazs, Except if non-ooo jobs don't make sense? | 11:52 |
| therve | oooq I mean | 11:52 |
| *** dsariel has joined #tripleo | 11:52 | |
| * adarazs looks at how those jobs failed | 11:54 | |
| adarazs | oh, you're right. they exhibit the problem. | 11:54 |
| therve | adarazs, Can you double check the promo job used the proper heat version? | 11:54 |
| therve | It seemd okay to me, but... I don't trust anything anymore :) | 11:54 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-common master: Add logrotate-crond container images params https://review.openstack.org/490068 | 11:55 |
| adarazs | therve: so do you have a change where these experimental gates passed and it's already merged? | 11:55 |
| therve | adarazs, No. I run again on the test commit. | 11:56 |
| therve | But in theory that's what the check experimental in the patch did | 11:56 |
| adarazs | therve: I'm confused. so what made you think that running on the latest master will work without any revert? | 11:57 |
| fultonj | jistr: So openstack-infra/tripleo-ci, openstack/tripleo-quickstart, and ... THT? | 11:58 |
| therve | adarazs, Which run are you talking about? https://review.openstack.org/490194 without the fix? | 11:58 |
| jistr | fultonj: yes. Working on tht, that should be the first one. | 11:58 |
| adarazs | therve: FWIW the period job did run with the recent heat from master: http://logs.openstack.org/15/359215/122/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha-oooq/4ea32d2/logs/undercloud/var/log/extra/yum-list-installed.txt.gz | 11:58 |
| adarazs | openstack-heat-engine.noarch 1:9.0.0-0.20170803093435.4e56332.el7.centos @delorean | 11:58 |
| fultonj | thank you jistr | 11:58 |
| jistr | sure thing :) it's a pity that we hit this bug. It has nothing to do with Ceph or even containers... | 11:59 |
| *** jchhatbar has joined #tripleo | 11:59 | |
| adarazs | therve: I'm taling about the latest promotion run from master. I understood that the issue is already solved in the latest master without a revert (because a fix got merged during the night), so I can just rerun it and it should work. did I misunderstand you? | 11:59 |
| *** akrivoka has quit IRC | 12:00 | |
| *** jchhatbar has quit IRC | 12:00 | |
| *** jchhatbar has joined #tripleo | 12:00 | |
| therve | adarazs, Ah, because we made related fix in the mean time: https://review.openstack.org/#/c/477047/ | 12:01 |
| therve | Which had CI green | 12:01 |
| jkilpatr | so in recent tripleo builds / quickstart images the perms on /tmp have changed. (a playbook I use had to be updated) was this intentional? | 12:01 |
| *** sbrzozow has joined #tripleo | 12:01 | |
| *** morazi has joined #tripleo | 12:02 | |
| *** pdeore has quit IRC | 12:02 | |
| *** janki has quit IRC | 12:02 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates master: Add logrotate with crond service https://review.openstack.org/490050 | 12:03 |
| adarazs | therve: okay, so I didn't misunderstand you, and I understand your confusion... let me check the packages on that. but yeah, this doesn't make much sense. unless that ovb-ha-tempest-oooq gate is somehow runs different things than the other two promotion job. | 12:03 |
| adarazs | therve: which they might. | 12:03 |
| therve | adarazs, Yeah, there is something going on at least in CI... | 12:04 |
| therve | adarazs, Can we find the actual package bits in delorean? | 12:04 |
| adarazs | therve: for which run? | 12:04 |
| therve | adarazs, The latest promotion test | 12:04 |
| adarazs | therve: it was running with https://trunk.rdoproject.org/centos7/b3/b7/b3b7d520396888f05dc497da7a509c58541cef6f_d767b7fe/?C=M;O=D | 12:05 |
| adarazs | this is what you mean? | 12:06 |
| *** agurenko has quit IRC | 12:06 | |
| *** agurenko has joined #tripleo | 12:07 | |
| therve | adarazs, Oh wow. The code is not there | 12:08 |
| adarazs | how is that possible? | 12:08 |
| therve | adarazs, So the hash is 4e56332 right? Which should be the merge hash? | 12:09 |
| *** amoralej|lunch is now known as amoralej | 12:09 | |
| adarazs | therve: yes | 12:09 |
| adarazs | as far as I know DLRN :) | 12:10 |
| *** ooolpbot has joined #tripleo | 12:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1707788 | 12:10 |
| openstack | Launchpad bug 1707788 in tripleo "gate-tripleo-buildimage-overcloud-full-centos-7 is broken on missing semanage" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker) | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1708146 | 12:10 |
| *** ooolpbot has quit IRC | 12:10 | |
| openstack | Launchpad bug 1708146 in tripleo "OVB periodic jobs fail in overcloud deploy task with timeout waiting for messages" [Critical,Triaged] | 12:10 |
| therve | adarazs, Maybe the forced build didn't operate as you thought? | 12:11 |
| *** dprince has joined #tripleo | 12:12 | |
| hewbrocca | adarazs: can dmsimard help with this? | 12:12 |
| adarazs | hewbrocca: yeah, we need jpena or dmsimard for telling us why things are as they are :) | 12:12 |
| therve | adarazs, Can you double check https://trunk.rdoproject.org/centos7/b3/b7/b3b7d520396888f05dc497da7a509c58541cef6f_d767b7fe/openstack-heat-common-9.0.0-0.20170803093435.4e56332.el7.centos.noarch.rpm ? | 12:12 |
| jpena | adarazs: what do you need? | 12:13 |
| * jpena reads back | 12:13 | |
| *** akrivoka has joined #tripleo | 12:13 | |
| adarazs | therve: I don't have to, you can see that hash is just before the fix: https://github.com/openstack/heat/commits/master | 12:13 |
| adarazs | jpena: let me summarize it. | 12:13 |
| *** pchavva has joined #tripleo | 12:13 | |
| therve | adarazs, I mean, check the content :) | 12:13 |
| therve | Oh snap | 12:13 |
| therve | adarazs, Sorry :/ | 12:13 |
| therve | Ok that was just a stupid mistake on me then | 12:14 |
| adarazs | jpena: so we have this change that merged this morning: https://review.openstack.org/477047 | 12:14 |
| therve | I don't know why it didn't take the latest code though | 12:14 |
| adarazs | jpena: but somehow the latest consistent DLRN build that has the timestamp for way later does not contain that commit. | 12:15 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates master: WIP: Adds pacemaker update_tasks for Pike minor update workflow https://review.openstack.org/486567 | 12:16 |
| adarazs | jpena: https://trunk.rdoproject.org/centos7/consistent/?C=M;O=D -> openstack-heat-engine-9.0.0-0.20170803093435.4e56332.el7.centos.noarch.rpm | 12:16 |
| therve | That's weird but way less than I feared it could be | 12:16 |
| adarazs | so the package is generated from 4e56332 which merged quite a while ago. | 12:16 |
| adarazs | but the timestamp says that the package is built at 093435 | 12:17 |
| therve | adarazs, While we sort that out, you should relaunch, it should pick it up now | 12:17 |
| adarazs | therve: nope, /consistent/ still points to the same package with the commit just right before the fix. it won't work again. | 12:18 |
| therve | Hum, ok | 12:18 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates master: WIP: Adds pacemaker update_tasks for Pike minor update workflow https://review.openstack.org/486567 | 12:18 |
| *** dhill_ has quit IRC | 12:19 | |
| jpena | adarazs: it's still in the build queue: https://trunk.rdoproject.org/centos7-master/queue.html | 12:19 |
| jpena | now I'll check why we have this lag | 12:19 |
| adarazs | phew, okay, we're not going crazy at least :) | 12:19 |
| adarazs | jpena: yeah, I was assuming that the change will be in the DLRN build after a few hours, didn't even think about checking it. let us know what's up. | 12:20 |
| adarazs | hewbrocca: so it seems we ran the promotion with a hash that didn't yet contain the fix because somehow DLRN is having a long build queue. | 12:20 |
| hewbrocca | weeeeeeee | 12:21 |
| hewbrocca | adarazs: OK put that on your "things to check next time" list :) | 12:21 |
| adarazs | yeah, I rarely chase the tail of master this much. :) | 12:22 |
| hewbrocca | LOL chase the tail of master | 12:22 |
| hewbrocca | "there it goes, get it!" | 12:22 |
| hewbrocca | like that | 12:22 |
| *** akrivoka has quit IRC | 12:22 | |
| adarazs | yeah, feels like a dog chasing its own tail sometimes :) | 12:23 |
| jpena | adarazs: so we have a combination of factors: 1 is cinder builds taking 30 minutes each (I'll check that, it's way too much and builds up the queue) | 12:23 |
| *** akrivoka has joined #tripleo | 12:24 | |
| jpena | and 2 is that we have some DB disconnections from the DLRN builder. I've just proposed a review to fix that | 12:24 |
| adarazs | jpena: cool, thanks for looking into it. our heat change is next in line at least \o/ :) | 12:24 |
| therve | Now we still need to fix the experimental jobs on Heat master | 12:27 |
| *** tzumainn has joined #tripleo | 12:27 | |
| d0ugal | adarazs, therve: sorry, I was on lunch - any luck? | 12:27 |
| therve | d0ugal, Yeah we *should* be okay | 12:28 |
| adarazs | d0ugal: we solved some mysteries regarding DLRN, probably we're on track. | 12:28 |
| d0ugal | lol | 12:28 |
| d0ugal | I feel very confident about this | 12:28 |
| therve | Heh heh | 12:28 |
| therve | I'm glad you noticed our skepticism | 12:28 |
| d0ugal | :) | 12:28 |
| hewbrocca | I love this plan! I'm excited to be a part of it! Let's do it! | 12:28 |
| d0ugal | Yes! | 12:29 |
| d0ugal | Wait, what is the plan? | 12:29 |
| therve | Exactly | 12:29 |
| *** eck` is now known as eck`gone | 12:29 | |
| hewbrocca | I think it's | 12:30 |
| hewbrocca | * Let DLRN finish building the change we need | 12:30 |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-heat-templates master: Add script to create tripleo-admin on deployed servers https://review.openstack.org/490458 | 12:30 |
| hewbrocca | * re-kick the promote job | 12:30 |
| jistr | fultonj: ^ tried, tested, trusted :))) | 12:30 |
| jistr | now just to plug it into CI... | 12:30 |
| fultonj | thanks jistr | 12:30 |
| * fultonj looks | 12:30 | |
| hewbrocca | * meantime fix the Heat experimental jobs | 12:30 |
| hewbrocca | (not sure about that last one) | 12:30 |
| adarazs | hewbrocca: yep, and then meanwhile try a parallel job on RDO Phase1 to get more info :) | 12:31 |
| adarazs | I liked panda's summary at our last retrospective: "Every time I have an answer, the question has changed" | 12:31 |
| d0ugal | Great | 12:31 |
| panda | :) | 12:31 |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-heat-templates master: Add script to create tripleo-admin on deployed servers https://review.openstack.org/490458 | 12:32 |
| adarazs | therve: so I you mean we need to fix the experimental jobs because these gates should not have failed on this blank change, right? https://review.openstack.org/490194 | 12:32 |
| therve | adarazs, Well, gate-tripleo-ci-centos-7-nonha-multinode-oooq should have failed | 12:33 |
| therve | adarazs, And apparently the other ones should use oooq ? | 12:33 |
| therve | Not it's expected that they failed, it was before the fix | 12:33 |
| jistr | bogdando: for now i'll go with a way that plugs best into what we already have in TOCI and OOOQ, e.g. https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/overcloud-deploy/tasks/deployed-server.yml | 12:34 |
| adarazs | therve: I think sshnaidm wanted to keep those jobs there. I thought we don't have any oooq jobs there, so the type of jobs ran there should be OK actually. | 12:35 |
| jistr | bogdando: but we can make it use the overcloud_user and overcloud_key variables at some point | 12:35 |
| adarazs | therve: if that change is before the fix, then it should have failed (even the non-oooq ones) | 12:35 |
| jistr | bogdando: it just wouldn't make sense to mix those two right now | 12:35 |
| bogdando | jistr: looking into the patches open | 12:35 |
| therve | adarazs, Yep exactly. It's the oooq one the outlier | 12:35 |
| adarazs | therve: now the thing is that they should pass if you rebase it on top of the fix. so probably the expeirmental gates are working as expected. | 12:35 |
| *** catintheroof has joined #tripleo | 12:36 | |
| jistr | bogdando: we need it also for non-quickstart environments, that's why i did the bash script | 12:36 |
| jistr | deployed server users (not just CIs etc.) might want to configure the overcloud ssh admin user too | 12:36 |
| bogdando | jistr: do you mean the heat-admin user requirement will stay with us? | 12:36 |
| bogdando | no alts? | 12:36 |
| openstackgerrit | Numan Siddique proposed openstack/tripleo-heat-templates master: [WIP] Support OVN DBs docker HA using pacemaker bundle https://review.openstack.org/484376 | 12:37 |
| jistr | bogdando: not heat admin. see the tables in the commit message https://review.openstack.org/#/c/489613/6//COMMIT_MSG | 12:37 |
| *** thrash is now known as thrash|biab | 12:38 | |
| jistr | so split stack envs never had heat-admin, and they won't have it (unless user explicitly wants to create it) | 12:38 |
| numans | bogdando, hi, can you please have a look at this patch - https://review.openstack.org/#/c/468860/ : "Support deploying OVN as container services" | 12:38 |
| *** pgadiya has quit IRC | 12:38 | |
| EmilienM | weshay_pto: yes | 12:40 |
| amoralej | adarazs, the heat fix is now in consistent how could we test it with periodic job? | 12:40 |
| adarazs | yes, I just looked. | 12:40 |
| amoralej | adarazs, how do you do it?, with https://review.openstack.org/#/c/359215/? | 12:41 |
| adarazs | amoralej: yes, with that change. I just rechecked. | 12:42 |
| dmsimard | jpena: cinder build durations are because of the docs (see my email about ftbfs storm) | 12:42 |
| jpena | dmsimard: yep, I'm trying to find some workaround | 12:42 |
| dmsimard | jpena: number80 was open to the idea to only build man docs instead of man+html | 12:43 |
| dmsimard | But I don't know the ramifications | 12:43 |
| jpena | dmsimard: it would just mean the -doc subpackage would not be created | 12:44 |
| *** jpena is now known as jpena|lunch | 12:44 | |
| number80 | well, shipping man in a package is required, doc is recommended, that's the difference :) | 12:44 |
| dmsimard | Let's discuss it on #rdo after lunch | 12:45 |
| adarazs | therve: so now openstack-heat-engine-9.0.0-0.20170803122112.c80011c.el7.centos.noarch.rpm -> that commit hash looks good. :) | 12:45 |
| adarazs | that's the one with the fix. | 12:45 |
| *** dmacpher has joined #tripleo | 12:46 | |
| bogdando | numans: looking | 12:47 |
| chandankumar | EmilienM: mwhahaha https://review.openstack.org/#/c/486965/ | 12:48 |
| number80 | dmsimard: spanish time lunch? :o) | 12:48 |
| chandankumar | please have a look when you are free. | 12:48 |
| dmsimard | number80: ... Yeah | 12:48 |
| *** ecerquei has joined #tripleo | 12:49 | |
| *** gkadam has quit IRC | 12:50 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/tripleo-heat-templates master: Updated from global requirements https://review.openstack.org/488148 | 12:50 |
| *** apetrich has joined #tripleo | 12:52 | |
| *** hewbrocca is now known as hewbrocca_afk | 12:52 | |
| *** hewbrocca_afk is now known as hewbrocca | 12:53 | |
| *** cdearborn has joined #tripleo | 12:53 | |
| *** udesale has joined #tripleo | 12:53 | |
| number80 | 3pm and I have yet to get solid food, I've been jumping from calls to calls | 12:53 |
| *** liverpooler has joined #tripleo | 12:53 | |
| *** Goneri has joined #tripleo | 12:54 | |
| *** lucas-hungry is now known as lucasagomes | 12:54 | |
| *** skramaja has quit IRC | 12:55 | |
| adarazs | hewbrocca, dmsimard: further good news: when I used a fresh (not-yet-promoted) hash the weirdo jobs passed, so if we manage to promote upstream it will likely pass RDO phase1 as well: https://ci.centos.org/view/rdo/view/promotion-pipeline/job/rdo_trunk-promote-master-current-tripleo/179/ | 12:55 |
| *** Vijayendra has quit IRC | 12:56 | |
| openstackgerrit | Arx Cruz proposed openstack/tripleo-quickstart master: Configure the set of tests to execute in scenario001 https://review.openstack.org/490376 | 12:56 |
| *** gbarros has joined #tripleo | 12:58 | |
| hewbrocca | adarazs: ahh that was your time travel experiment from earlier? | 12:58 |
| adarazs | yes. | 12:58 |
| *** ykarel has joined #tripleo | 12:58 | |
| *** milan has quit IRC | 12:58 | |
| *** _milan_ has joined #tripleo | 12:58 | |
| hewbrocca | All right, let's get DLRN to update | 12:58 |
| hewbrocca | go DLRN go! | 12:58 |
| adarazs | hewbrocca: that's done. the periodic promotion jobs are already running. | 12:59 |
| *** noslzzp has joined #tripleo | 13:00 | |
| hewbrocca | oh even better | 13:00 |
| * adarazs gets a coffee. | 13:01 | |
| hewbrocca | good idea | 13:01 |
| *** eck`gone is now known as eck` | 13:01 | |
| *** ansmith has joined #tripleo | 13:03 | |
| *** jchhatbar has quit IRC | 13:04 | |
| morazi | adarazs, that's excellent, thanks! | 13:05 |
| *** Goneri has quit IRC | 13:05 | |
| *** links has quit IRC | 13:06 | |
| openstackgerrit | Jiri Stransky proposed openstack-infra/tripleo-ci master: Set OVERCLOUD_HOSTS for deployed server ssh admin creation https://review.openstack.org/490468 | 13:06 |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-quickstart-extras master: Create tripleo-admin user on deployed servers https://review.openstack.org/490470 | 13:09 |
| jistr | fultonj: i think that should do it ^^ | 13:10 |
| *** ooolpbot has joined #tripleo | 13:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1707788 | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1708146 | 13:10 |
| *** ooolpbot has quit IRC | 13:10 | |
| openstack | Launchpad bug 1707788 in tripleo "gate-tripleo-buildimage-overcloud-full-centos-7 is broken on missing semanage" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker) | 13:10 |
| openstack | Launchpad bug 1708146 in tripleo "OVB periodic jobs fail in overcloud deploy task with timeout waiting for messages" [Critical,Triaged] | 13:10 |
| *** ratailor has quit IRC | 13:10 | |
| fultonj | jistr: thanks! so in https://review.openstack.org/#/c/479288, i should add a depends-on https://review.openstack.org/#/c/490470 ? | 13:11 |
| *** Goneri has joined #tripleo | 13:12 | |
| *** pdeore has joined #tripleo | 13:12 | |
| jistr | fultonj: exactly, i was just typing it :) we also can remove the other depends-ons, as all 4 necessary patches should be pulled in recursively. We actually have to remove dependency on https://review.openstack.org/#/c/483867/ as it would conflict with the other TOCI patch. | 13:13 |
| fultonj | jistr: ok, great. thanks i will update it | 13:13 |
| *** pdeore has quit IRC | 13:13 | |
| *** ramishra has quit IRC | 13:13 | |
| jistr | thanks :) | 13:13 |
| *** psachin has quit IRC | 13:13 | |
| openstackgerrit | Alfredo Moralejo proposed openstack/tripleo-quickstart master: Add consistent-pike.ynml configuration https://review.openstack.org/490472 | 13:14 |
| openstackgerrit | Alfredo Moralejo proposed openstack/tripleo-quickstart master: Add consistent-pike.yml configuration https://review.openstack.org/490472 | 13:15 |
| openstackgerrit | John Fulton proposed openstack/tripleo-heat-templates master: Convert scenario001-multinode-containers job to ceph-ansible https://review.openstack.org/479288 | 13:15 |
| *** ramishra has joined #tripleo | 13:15 | |
| * jistr crosses fingers for CI on 479288 | 13:17 | |
| * fultonj crosses toes | 13:17 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-common master: Add logrotate-crond container images params https://review.openstack.org/490068 | 13:17 |
| *** gbarros has quit IRC | 13:19 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: Add bits to support stable/pike https://review.openstack.org/490077 | 13:20 |
| *** pblaho has quit IRC | 13:22 | |
| *** udesale has quit IRC | 13:22 | |
| *** jcoufal has joined #tripleo | 13:23 | |
| *** udesale has joined #tripleo | 13:23 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: Add bits to support stable/pike https://review.openstack.org/490077 | 13:23 |
| *** psachin has joined #tripleo | 13:24 | |
| fultonj | jistr: in the dependency tree tht depends on quickstart-extras depends on tht ; is that ok? i.e. i can depend on the same repos as long as its not a direct depends on? | 13:25 |
| jistr | fultonj: nope that wouldn't be ok | 13:25 |
| * jistr looks | 13:25 | |
| fultonj | 479288 --> 490470 --> 490458 | 13:25 |
| *** lmiccini has quit IRC | 13:26 | |
| jistr | fultonj: we'll need to rebase 479288 on top of 490458. nice catch :) | 13:27 |
| lyarwood | mwhahaha: hey, owalsh suggested that I ask for your take on https://review.openstack.org/#/c/490409/ if you have any review bandwidth today | 13:28 |
| jistr | fultonj: should i or will you? | 13:28 |
| *** udesale has quit IRC | 13:28 | |
| mwhahaha | lyarwood: k | 13:28 |
| fultonj | jistr: let me try... | 13:28 |
| jistr | ok thanks | 13:28 |
| lyarwood | mwhahaha: thanks | 13:29 |
| openstackgerrit | John Fulton proposed openstack/tripleo-heat-templates master: Convert scenario001-multinode-containers job to ceph-ansible https://review.openstack.org/479288 | 13:29 |
| fultonj | jistr: done | 13:29 |
| *** marios has quit IRC | 13:29 | |
| *** marios has joined #tripleo | 13:30 | |
| fultonj | jistr: now that we have the rebase, the chain still exists; will CI just ignore the dep that 490470 because 479288 provides the file ? | 13:30 |
| openstackgerrit | Adriano Petrich proposed openstack/tripleo-quickstart-extras master: WIP workaround is not needed anymore https://review.openstack.org/485653 | 13:31 |
| fultonj | i.e. if you follow the reviews depends-on down one path you still end up taking 479288 --> 490470 --> 490458 | 13:31 |
| jistr | fultonj: yea. I'm not 100% sure but i do think it should ignore that depends-on. If we removed the depends-on from 490470, it would likely fail CI and become unmergeable. So hopefully we now have it ok. | 13:33 |
| openstackgerrit | Karthik S proposed openstack/tripleo-heat-templates master: NetworkDeploymentActions shall be made role specific https://review.openstack.org/490474 | 13:33 |
| fultonj | jistr: seems about right, let's see what happens. | 13:34 |
| lyarwood | owalsh: https://review.openstack.org/#/c/490226/ btw, not sure if you've seen this. | 13:34 |
| *** lblanchard has joined #tripleo | 13:35 | |
| hewbrocca | adarazs: how soon should we know if that promote will succeed? | 13:36 |
| *** pblaho1 has joined #tripleo | 13:36 | |
| owalsh | lyarwood: ack, LGTM | 13:36 |
| adarazs | hewbrocca: at around 14:15 our time (CEST) | 13:38 |
| *** lmiccini has joined #tripleo | 13:38 | |
| adarazs | so ~half an hour | 13:38 |
| adarazs | if I calculated correctly from the previous run :) | 13:38 |
| *** pblaho1 is now known as pblaho | 13:41 | |
| *** lblanchard has quit IRC | 13:41 | |
| *** jmelvin has joined #tripleo | 13:41 | |
| *** lblanchard has joined #tripleo | 13:41 | |
| hewbrocca | adarazs: coool | 13:41 |
| hewbrocca | LMK, all right? | 13:42 |
| openstackgerrit | Lee Yarwood proposed openstack/instack-undercloud master: WIP zaqar: restorecon zaqar.log to ensure httpd has access https://review.openstack.org/490409 | 13:44 |
| adarazs | hewbrocca: you'll be the first to know, except if somebody else is looking at the telnet job output like me. :) | 13:44 |
| *** Vijayendra has joined #tripleo | 13:45 | |
| *** jpena|lunch is now known as jpena | 13:46 | |
| *** masco has quit IRC | 13:46 | |
| numans | bogdando, thanks | 13:49 |
| *** thrash|biab is now known as thrash | 13:52 | |
| jistr | fultonj: it already failed on unrelated issue | 13:54 |
| jistr | One of the configured repositories failed (Unknown),", "2017-08-03 13:45:58 | and yum doesn't have enough cached data to continue. At this point the only | 13:54 |
| jistr | http://logs.openstack.org/88/479288/20/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq/b295195/console.html#_2017-08-03_13_45_58_273537 | 13:54 |
| jistr | i discovered i probably need to amend the TOCI patch anyway, so i'll do that and we can recheck | 13:54 |
| jistr | (i forgot to export the variable i added) | 13:55 |
| fultonj | jistr: ok, thanks for catching that | 13:55 |
| openstackgerrit | Jiri Stransky proposed openstack-infra/tripleo-ci master: Set OVERCLOUD_HOSTS for deployed server ssh admin creation https://review.openstack.org/490468 | 13:56 |
| *** agopi|away has joined #tripleo | 13:57 | |
| fultonj | jistr: as per ^ i just issued a recheck on https://review.openstack.org/#/c/479288 | 13:58 |
| *** mkovacik__ has joined #tripleo | 13:58 | |
| *** _milan_ has quit IRC | 13:59 | |
| jistr | fultonj: thanks. Not sure if Zuul caught it tough.. when i amended the toci patch, it should have stopped the CI jobs that depended on it, but it didn't. http://status.openstack.org/zuul/ | 13:59 |
| fultonj | yeah, i was just going to ask about that | 13:59 |
| fultonj | e.g. can we tell zuul not to waste it's time and start at the start? | 13:59 |
| openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart-extras master: GATE CHECK for quickstart-extras https://review.openstack.org/472607 | 14:00 |
| adarazs | EmilienM, weshay_pto, panda, rlandy|ci_mtgs, arxcruz: TripleO CI Squad meeting in 30 minutes: https://etherpad.openstack.org/p/tripleo-ci-squad-meeting \o/ | 14:00 |
| *** marios has quit IRC | 14:01 | |
| *** cylopez has quit IRC | 14:01 | |
| rlandy|ci_mtgs | adarazs: won't be able to make it - clashing session at CI meetings | 14:02 |
| adarazs | hewbrocca: 2017-08-03 14:01:58.215102 | TASK [overcloud-deploy : Deploy the overcloud] ********************************* | 14:02 |
| adarazs | if it doesn't fail in the next 8-10 minutes then it should be probably fixed (at least the heat issues) | 14:02 |
| *** pdeore has joined #tripleo | 14:03 | |
| *** marios has joined #tripleo | 14:03 | |
| jistr | fultonj: now it noticed, -1'd the patch, and i rechecked again | 14:03 |
| fultonj | jistr: so it noticed the failure you pointed out and voted a -1 | 14:05 |
| hewbrocca | woop woop | 14:05 |
| EmilienM | adarazs: I'm pretty sure we'll be 2 or 3 :D | 14:05 |
| *** pblaho has quit IRC | 14:05 | |
| jistr | fultonj: it noticed that i pushed the new TOCI patch, which invalidated its dependency chain, so it stopped everything (also on the lower patches) | 14:06 |
| adarazs | EmilienM: yeah. it's more important for talking with panda about handing over his tasks, he'll be out for 4 weeks, starting next week. | 14:06 |
| fultonj | jistr: ok, so that had to percolate up and then it noticed. so then we rechecked | 14:07 |
| *** mrunge has quit IRC | 14:07 | |
| jistr | yea | 14:07 |
| fultonj | thanks for explaining it to me | 14:07 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates master: Drop step_config as top level docker requirement https://review.openstack.org/442716 | 14:08 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates master: docker: Add unit tests on service_name https://review.openstack.org/442755 | 14:08 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates master: Add docker templates to configure Ironic inspector https://review.openstack.org/457822 | 14:08 |
| openstackgerrit | Attila Darazs proposed openstack-infra/tripleo-ci master: Add adarazs to tripleo-cd-admins https://review.openstack.org/490493 | 14:09 |
| adarazs | EmilienM, panda: ^ please vote on it. | 14:09 |
| panda | adarazs: -2 | 14:09 |
| adarazs | panda: that means I get to call you on your cell phone when I need access to those things :) | 14:09 |
| *** ooolpbot has joined #tripleo | 14:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1707788 | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1708146 | 14:10 |
| *** ooolpbot has quit IRC | 14:10 | |
| openstack | Launchpad bug 1707788 in tripleo "gate-tripleo-buildimage-overcloud-full-centos-7 is broken on missing semanage" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker) | 14:10 |
| openstack | Launchpad bug 1708146 in tripleo "OVB periodic jobs fail in overcloud deploy task with timeout waiting for messages" [Critical,Triaged] | 14:10 |
| *** pchavva has quit IRC | 14:10 | |
| panda | adarazs: oh yeah ? -4 ! | 14:11 |
| panda | brb | 14:13 |
| *** panda has quit IRC | 14:13 | |
| adarazs | hewbrocca: still at the "deploy the overcloud" step, previously it failed under 10 minutes. I'd say that's promising. | 14:14 |
| *** panda has joined #tripleo | 14:14 | |
| *** mrunge has joined #tripleo | 14:15 | |
| hewbrocca | nice! | 14:15 |
| jaosorior | mandre: did you see my message on the review? | 14:16 |
| mandre | jaosorior: nope sorry, looking | 14:16 |
| jaosorior | no biggie | 14:16 |
| jaosorior | I'll do it :D | 14:17 |
| adarazs | bnemec: can you do some magic, if it's necessary, on the upstream images server to let me get access when this merges? https://review.openstack.org/490493 | 14:17 |
| openstackgerrit | Numan Siddique proposed openstack/tripleo-common master: Add neutron-server-ovn image to the overcloud containers https://review.openstack.org/489875 | 14:17 |
| jaosorior | so... your suggestion was to use the nova-api image for almost all nova containers? | 14:17 |
| jaosorior | I mean, their configuration | 14:17 |
| jaosorior | mandre: ^^ | 14:17 |
| mandre | jaosorior: oh the one where you're having lunch? I figured it was just momentary :) | 14:18 |
| jaosorior | mandre: it was a long lunch :P | 14:18 |
| jaosorior | I wouldn't have asked if it didn't take long | 14:18 |
| mandre | jaosorior: yeah, use nova-api image for the nova config_volume | 14:19 |
| jaosorior | mandre: but yeah. I'm not sure if doing that is the way to go though. Maybe just having nova-api use the nova-api image for the config is the way to go. That's what nova-placement does anyway | 14:19 |
| *** udesale has joined #tripleo | 14:19 | |
| *** mmethot has joined #tripleo | 14:19 | |
| jaosorior | nova-placement just uses the nova-placement iamge | 14:19 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Make many networking parameters consistent https://review.openstack.org/483540 | 14:19 |
| *** ykarel has quit IRC | 14:20 | |
| mandre | jaosorior: haha, a french sized lunch | 14:20 |
| openstackgerrit | Merged openstack/python-tripleoclient master: Install policycoreutils-python https://review.openstack.org/490265 | 14:20 |
| openstackgerrit | Numan Siddique proposed openstack/tripleo-common master: PLEASE DON'T REVIEW: FOR TESTING ONLY https://review.openstack.org/490495 | 14:21 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates master: Remove baremetal cron jobs on docker upgrade https://review.openstack.org/490496 | 14:21 |
| *** thrash is now known as thrah|biab | 14:21 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Enable TLS for nova api and placement containers https://review.openstack.org/489885 | 14:21 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Make containerized nova-api run with httpd https://review.openstack.org/475366 | 14:21 |
| jaosorior | mandre: like that ^^ | 14:21 |
| jaosorior | mandre: and this being the tripleo-common change https://review.openstack.org/#/c/490385/ | 14:22 |
| *** athomas has quit IRC | 14:23 | |
| jaosorior | bandini, marios: could you check this out https://review.openstack.org/#/c/489891/ ? | 14:23 |
| openstackgerrit | John Fulton proposed openstack/tripleo-heat-templates master: Add support for installing Ceph MDS via ceph-ansible https://review.openstack.org/479426 | 14:23 |
| *** jfrancoa has quit IRC | 14:24 | |
| marios | ack jaosorior added to reviews | 14:24 |
| openstackgerrit | Numan Siddique proposed openstack/tripleo-heat-templates master: DO NOT REVIEW: TESTING ONLY TO TEST OVN CONTAINER SERVICES https://review.openstack.org/489097 | 14:25 |
| mandre | jaosorior: it's not going to work because the nova config_volume is probably still going to use nova-base in the end | 14:25 |
| mandre | jaosorior: for example https://github.com/openstack/tripleo-heat-templates/blob/master/docker/services/nova-consoleauth.yaml#L75 | 14:25 |
| *** athomas has joined #tripleo | 14:26 | |
| jaosorior | mandre: why? We only need it for nova-api | 14:26 |
| mandre | jaosorior: I'd really prefer we don't introduce a new nova-api config volume, and just change the DockerNovaConfigImage to default to nova-api image instead of nova-base | 14:26 |
| mandre | jaosorior: yeah but it's the same 'nova' config_volume, meaning it's a single puppet run and we need to know which image it will use | 14:28 |
| jaosorior | mandre: sorry, I don't mean to be dense but I really don't understand the issue. Can you explain a bit more? | 14:28 |
| mandre | jaosorior: you can't use nova-api in the nova-api template and nova-base somewhere else | 14:28 |
| bogdando | folks, PTAL https://review.openstack.org/#/q/topic:bug/1700912 | 14:28 |
| bogdando | testing it locally | 14:29 |
| jaosorior | mandre: but, as far as I can see that's what nova-placement does | 14:29 |
| bogdando | have issues to make it tested with upstream CI | 14:29 |
| openstackgerrit | Rabi Mishra proposed openstack/tripleo-image-elements master: Create symlink for dib-run-parts https://review.openstack.org/490500 | 14:29 |
| mandre | jaosorior: nova-placement is using a different config_volume, https://github.com/openstack/tripleo-heat-templates/blob/master/docker/services/nova-placement.yaml#L75 | 14:29 |
| *** oidgar has quit IRC | 14:29 | |
| bogdando | jenkins yells on me "This change depends on a change that failed to merge" | 14:29 |
| bogdando | sigh | 14:30 |
| jaosorior | mandre: oooh | 14:30 |
| jaosorior | mandre: now I get it | 14:30 |
| jaosorior | mandre: thanks. Sorry it took too long | 14:30 |
| jaosorior | bogdando: need to rebase the change it depends on | 14:30 |
| EmilienM | tripleo ci squad meeting starting: https://bluejeans.com/u/whayutin/ | 14:30 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/puppet-tripleo master: Add logrotate-crond configuration https://review.openstack.org/490048 | 14:31 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates master: Add logrotate with crond service https://review.openstack.org/490050 | 14:31 |
| mandre | jaosorior: no worries at all, always happy to help | 14:32 |
| *** apetrich has quit IRC | 14:34 | |
| *** links has joined #tripleo | 14:34 | |
| *** links has quit IRC | 14:34 | |
| *** nyechiel has quit IRC | 14:38 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-common master: Use nova-api image for configuration as well https://review.openstack.org/490385 | 14:40 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Enable TLS for nova api and placement containers https://review.openstack.org/489885 | 14:41 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Make containerized nova-api run with httpd https://review.openstack.org/475366 | 14:41 |
| jaosorior | mandre: what about that? ^^ | 14:41 |
| *** pdeore has quit IRC | 14:41 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Change the directory for haproxy certs/keys to be service-specific https://review.openstack.org/489896 | 14:43 |
| *** agurenko has quit IRC | 14:44 | |
| *** pchavva has joined #tripleo | 14:45 | |
| jaosorior | mwhahaha: hey, the pinning of stdlib landed. But there still appear to be some issues :/ http://logs.openstack.org/34/489534/5/check/gate-puppet-pacemaker-puppet-unit-4.5-centos-7/3cdd921/console.html | 14:45 |
| mwhahaha | jaosorior: i'll take a look at it today | 14:46 |
| openstackgerrit | Merged openstack/python-tripleoclient master: Minor fixes for container image prepare command https://review.openstack.org/484061 | 14:46 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/newton: Ensure yum cache is ready before update https://review.openstack.org/490347 | 14:46 |
| *** jfrancoa has joined #tripleo | 14:46 | |
| *** MVenesio has joined #tripleo | 14:47 | |
| hewbrocca | adarazs: ? | 14:48 |
| *** slagle has quit IRC | 14:48 | |
| adarazs | hewbrocca: still deploying the overcloud :) | 14:49 |
| adarazs | hewbrocca: I think the typical job run is around 210 minutes, so it might take another ~40-60 minutes for it to finish. | 14:49 |
| MVenesio | Hi guys, i'm trying to map the networks and ips for my ocata deploy using already deployed servers, so i've created the deployed-ctrl-plane but when i use something like compute01-tenant: to fix a tenant ip for a specific compute, it doesn't work | 14:49 |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui master: MistralApiService, SwiftApiService error handling https://review.openstack.org/484382 | 14:49 |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui master: IronicInspectorApiService error handling https://review.openstack.org/486977 | 14:49 |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui master: IronicApiService error handling https://review.openstack.org/486976 | 14:49 |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui master: KeystoneApiService error handling https://review.openstack.org/486979 | 14:49 |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui master: HeatApiService error handling https://review.openstack.org/486978 | 14:49 |
| *** slagle has joined #tripleo | 14:50 | |
| hewbrocca | adarazs: OK if it's still rolling that is all good news | 14:51 |
| hewbrocca | mburns said something about the upload failing? | 14:51 |
| adarazs | hewbrocca: that was me blocking the promotion on RDO CI phase1 on that not-really-promoted hash. | 14:52 |
| mwhahaha | jaosorior: oh i know, puppet-pacemaker doesn't use p-o-i pins. let me propose a pin just for puppet-pacemaker | 14:53 |
| adarazs | hewbrocca: it looked good, so we should expect the run that I will start on RDO CI to succeed with this hash. | 14:53 |
| openstackgerrit | Emilien Macchi proposed openstack/python-tripleoclient stable/pike: Install policycoreutils-python https://review.openstack.org/490504 | 14:53 |
| openstackgerrit | Emilien Macchi proposed openstack/python-tripleoclient stable/pike: Update .gitreview for stable/pike https://review.openstack.org/487886 | 14:54 |
| jaosorior | mwhahaha: it doesn't? where is that done usually? in project-config? | 14:54 |
| openstackgerrit | Emilien Macchi proposed openstack/python-tripleoclient stable/pike: Update .gitreview for stable/pike https://review.openstack.org/487886 | 14:54 |
| mwhahaha | jaosorior: .fixtures.yaml in the repo (classic puppet module style) | 14:54 |
| openstackgerrit | Emilien Macchi proposed openstack/python-tripleoclient stable/pike: Update UPPER_CONSTRAINTS_FILE for stable/pike https://review.openstack.org/487887 | 14:54 |
| mwhahaha | jaosorior: sec i've got it | 14:54 |
| openstackgerrit | Alex Schultz proposed openstack/puppet-pacemaker master: Pin puppet-stdlib to 4.16.0 https://review.openstack.org/490507 | 14:57 |
| EmilienM | mwhahaha: shouldn't we rather change puppet-pacemaker to use p-o-i stuffs to clone modules? | 14:58 |
| EmilienM | imho we should remove fixtures | 14:58 |
| mwhahaha | not really | 14:58 |
| mwhahaha | cause it just needs stdlib | 14:58 |
| mwhahaha | it's way overkill for it | 14:58 |
| EmilienM | mhh ok | 14:58 |
| amoralej | adarazs, a ovb job has passed overcloud deploy phase with heat patch :) | 15:00 |
| *** gbarros has joined #tripleo | 15:01 | |
| hewbrocca | amoralej: woowoo | 15:02 |
| openstackgerrit | Derek Higgins proposed openstack/tripleo-heat-templates master: Add IronicPxe to the default controller https://review.openstack.org/490512 | 15:04 |
| catintheroof | guys, hi.im having a conceptual issue, let's suppose that i dont know how to configure the my_ip key on neutron's ml2_conf.ini, i change some things and want to re-run the deploy updating the stack, what i see is that updating the stack does nothing updating new values, just seems to do things like deploying new things, is updating the stack on the undercloud what i need to do to try new things ? do i need to delete the stack en re | 15:06 |
| catintheroof | -run it ? or im i missing something conceptually ? | 15:06 |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui master: Fix plan export URL https://review.openstack.org/488164 | 15:08 |
| *** ooolpbot has joined #tripleo | 15:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1708146 | 15:10 |
| openstack | Launchpad bug 1708146 in tripleo "OVB periodic jobs fail in overcloud deploy task with timeout waiting for messages" [Critical,Triaged] | 15:10 |
| *** ooolpbot has quit IRC | 15:10 | |
| *** gbarros has quit IRC | 15:12 | |
| *** paken has quit IRC | 15:12 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates stable/ocata: Remove docker references from stable/ocata scenarios https://review.openstack.org/489874 | 15:15 |
| *** cshastri has quit IRC | 15:17 | |
| *** jlinkes has quit IRC | 15:17 | |
| *** yprokule has quit IRC | 15:20 | |
| openstackgerrit | Alex Schultz proposed openstack/puppet-pacemaker master: Switch requires for wait-for-settle into resource collectors https://review.openstack.org/489534 | 15:21 |
| *** panda has quit IRC | 15:23 | |
| *** dmacpher has quit IRC | 15:24 | |
| *** lmiccini has quit IRC | 15:24 | |
| jaosorior | mwhahaha: thanks for rebasing | 15:26 |
| *** chem has quit IRC | 15:26 | |
| mwhahaha | we'll get that merged (eventually) | 15:26 |
| mwhahaha | :D | 15:26 |
| *** chem has joined #tripleo | 15:27 | |
| *** dprince has quit IRC | 15:27 | |
| *** mkovacik__ is now known as milan | 15:28 | |
| *** nyechiel has joined #tripleo | 15:29 | |
| EmilienM | bnemec: when you have time https://review.openstack.org/#/c/490077/ | 15:29 |
| openstackgerrit | Oliver Walsh proposed openstack/tripleo-heat-templates master: Containerize virtlogd https://review.openstack.org/469116 | 15:32 |
| *** hewbrocca is now known as hewbrocca_afk | 15:34 | |
| *** katkapilatova has left #tripleo | 15:34 | |
| *** tzumainn has quit IRC | 15:34 | |
| *** ckyriakidou has quit IRC | 15:37 | |
| adarazs | bnemec: I used https://review.openstack.org/359215 to start a manual promotion job. now I think everything succeeded but I don't see https://ci.centos.org/view/rdo/view/promotion-pipeline/job/tripleo-dlrn-promote/ triggered by anything. | 15:39 |
| adarazs | bnemec: do you know how that should happen? | 15:39 |
| *** dprince has joined #tripleo | 15:40 | |
| *** jfrancoa has quit IRC | 15:41 | |
| adarazs | anybody else knows? | 15:43 |
| amoralej | adarazs, not all jobs used the same hash :( | 15:45 |
| amoralej | i think that's why | 15:45 |
| adarazs | nooooo | 15:45 |
| adarazs | argh :( | 15:45 |
| amoralej | look at http://logs.openstack.org/15/359215/122/check/gate-tripleo-ci-centos-7-scenario003-multinode-oooq-puppet/4cbfa33/console.html | 15:45 |
| amoralej | d0be7d8b26763196e0d5c35c7255339c7ab9ea4c_31fde871 | 15:46 |
| amoralej | while others have | 15:46 |
| EmilienM | adarazs, amoralej: do we want to promote manually? | 15:46 |
| amoralej | c80011cd91cbbd71c66064dbfdd69fed0fb6d705_65fed62 | 15:46 |
| amoralej | EmilienM, +2 | 15:46 |
| amoralej | i can promote c80011cd91cbbd71c66064dbfdd69fed0fb6d705_65fed62 | 15:46 |
| adarazs | EmilienM: I think so. I feel quite confident in this c80011cd91cbbd71c66064dbfdd69fed0fb6d705_65fed62 hash | 15:46 |
| amoralej | that has the patch from heat | 15:46 |
| EmilienM | ok | 15:46 |
| amoralej | and ovb passed | 15:46 |
| *** tzumainn has joined #tripleo | 15:47 | |
| *** nyechiel has quit IRC | 15:47 | |
| *** ccamacho has quit IRC | 15:47 | |
| adarazs | amoralej: let me know when you did the promote. | 15:47 |
| adarazs | I want to start the rdo job | 15:47 |
| amoralej | adarazs, EmilienM https://ci.centos.org/view/rdo/view/promotion-pipeline/job/tripleo-dlrn-promote/197/ | 15:47 |
| EmilienM | adarazs, amoralej : done | 15:47 |
| adarazs | thanks | 15:47 |
| amoralej | adarazs, did you re-enable build containers job? | 15:48 |
| amoralej | i think it was disabled | 15:48 |
| amoralej | i just enabled | 15:49 |
| adarazs | I didn't disable or enable it :) | 15:49 |
| amoralej | someone disabled, dunno who, but i've enabled, anyway | 15:50 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Add consistent-pike.yml https://review.openstack.org/490574 | 15:50 |
| EmilienM | adarazs, amoralej ^ need your review | 15:50 |
| amoralej | EmilienM, https://review.openstack.org/#/c/490472/ | 15:50 |
| *** anshulbehl has quit IRC | 15:51 | |
| EmilienM | amoralej: +2, I'll let adarazs +A | 15:51 |
| adarazs | looking. | 15:51 |
| jistr | fultonj: whohooo \o/ | 15:51 |
| jistr | 192.168.24.16 : ok=32 changed=6 unreachable=0 failed=1 | 15:51 |
| jistr | that is to say, it still failed http://logs.openstack.org/88/479288/20/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container/667c903/logs/undercloud/var/log/mistral/executor.log.txt.gz | 15:52 |
| amoralej | adarazs, promotion is done | 15:52 |
| jistr | fultonj: but we've moved past the point of overcloud being unreachable | 15:52 |
| fultonj | jistr++ | 15:52 |
| *** ckyriakidou has joined #tripleo | 15:52 | |
| d0ugal | jistr: Lots of red in that file | 15:53 |
| adarazs | amoralej: cool, I see. I started https://ci.centos.org/view/rdo/view/promotion-pipeline/job/rdo_trunk-promote-master-current-tripleo/180/ | 15:53 |
| adarazs | okay, it got the right hash. | 15:53 |
| * adarazs goes to play some squash and later checks on the job. | 15:53 | |
| d0ugal | fultonj, jistr: is there any way to have ansible return structured data that we can format? | 15:54 |
| fultonj | jistr: yes, looks like the playbook itself is failing so that means i have something to debug. thank you! | 15:54 |
| *** mcornea has quit IRC | 15:55 | |
| jistr | fultonj: sure thing :) I gotta run, ttyt | 15:55 |
| *** udesale has quit IRC | 15:55 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates master: Add logrotate with crond service https://review.openstack.org/490050 | 15:55 |
| jistr | d0ugal: i'm not sure about that but we may follow up on that issue | 15:56 |
| *** thrah|biab is now known as thrash | 16:00 | |
| *** psachin has quit IRC | 16:01 | |
| bogdando | PTAL https://review.openstack.org/#/q/topic:bug/1700912 . Works for my local env | 16:01 |
| *** dbecker has quit IRC | 16:01 | |
| EmilienM | bogdando: do we have tests coverage in CI? | 16:02 |
| *** dbecker has joined #tripleo | 16:02 | |
| openstackgerrit | Michael Bayer proposed openstack/puppet-tripleo master: Enable innodb_buffer_pool_size configuration https://review.openstack.org/490046 | 16:03 |
| *** egonzalez has quit IRC | 16:03 | |
| openstackgerrit | Michael Bayer proposed openstack/puppet-tripleo stable/ocata: Enable innodb_buffer_pool_size configuration https://review.openstack.org/490577 | 16:03 |
| EmilienM | fultonj: do you have a moment to debug scenario001 failing to upload an image to glance, when deploying with containers | 16:04 |
| catintheroof | guys, hi.im having a conceptual issue, let's suppose that i dont know how to configure the my_ip key on neutron's ml2_conf.ini, i change some things and want to re-run the deploy updating the stack, what i see is that updating the stack does nothing updating new values, just seems to do things like deploying new things, is updating the stack on the undercloud what i need to do to try new things ? do i need to delete the stack en re | 16:04 |
| catintheroof | -run it ? or im i missing something conceptually ? | 16:04 |
| fultonj | EmilienM: yes | 16:04 |
| EmilienM | fultonj: so glance is containerized but not ceph | 16:04 |
| EmilienM | fultonj: http://logs.openstack.org/10/489710/2/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container/fff73d0/logs/undercloud/home/jenkins/overcloud_validate.log.txt.gz#_2017-08-02_20_11_58 | 16:04 |
| EmilienM | fultonj: I don't see any trace in glance api http://logs.openstack.org/10/489710/2/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container/fff73d0/logs/subnode-2/var/log/containers/glance/api.log.txt.gz | 16:05 |
| EmilienM | fultonj: sounds like http://logs.openstack.org/10/489710/2/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container/fff73d0/logs/subnode-2/var/log/containers/glance/api.log.txt.gz#_2017-08-02_20_09_58_897 | 16:05 |
| EmilienM | fultonj: I've seen that before | 16:05 |
| bogdando | EmilienM: I've modified https://review.openstack.org/#/c/490050/8/ci/environments/multinode-containers.yaml I hope that works as I expect | 16:05 |
| fultonj | EmilienM: is this job using split stack? | 16:05 |
| EmilienM | fultonj: ceph logs are here: http://logs.openstack.org/10/489710/2/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container/fff73d0/logs/subnode-2/var/log/ceph/ | 16:06 |
| EmilienM | fultonj: yes | 16:06 |
| EmilienM | fultonj: but not ceph-ansible | 16:06 |
| *** jchhatbar has joined #tripleo | 16:06 | |
| openstackgerrit | Michael Bayer proposed openstack/puppet-tripleo stable/newton: Enable innodb_buffer_pool_size configuration https://review.openstack.org/490578 | 16:07 |
| slagle | yea, there's some misinformation being spread about ceph and split-stack, that it doesn't work | 16:07 |
| *** Vijayendra has quit IRC | 16:07 | |
| slagle | we've been deploying ceph with splitstack for a while, and it works | 16:07 |
| slagle | it's the new ceph-ansible changes that don't work | 16:07 |
| openstackgerrit | Oliver Walsh proposed openstack/tripleo-heat-templates master: Containerize virtlogd https://review.openstack.org/469116 | 16:07 |
| EmilienM | slagle: I'm testing the puppet ceph now | 16:08 |
| fultonj | slagle: true, they need a user for ansible to run as e.g. https://review.openstack.org/#/c/490470 | 16:08 |
| EmilienM | slagle: context: https://review.openstack.org/#/c/489710 | 16:08 |
| *** marios has quit IRC | 16:08 | |
| EmilienM | fultonj, slagle: I just enabled pingtest on scenario001-containers and it fail | 16:08 |
| catintheroof | fultonj: EmilienM slagle, any help ? | 16:09 |
| EmilienM | let's not talk about ceph-ansible now, this is out of context of my problem | 16:09 |
| EmilienM | fultonj: any idea why glance isn't available when deploying in containers (again except ceph) | 16:09 |
| slagle | EmilienM: i'm not talking about your problem | 16:10 |
| EmilienM | catintheroof: as you can see we're debugging something at this time | 16:10 |
| EmilienM | slagle: :D thanks! | 16:10 |
| slagle | EmilienM: no problem | 16:10 |
| *** apetrich has joined #tripleo | 16:10 | |
| fultonj | EmilienM: i suspect it's because the ceph client configuration for glance, as a container, didn't happen. it should have as per https://review.openstack.org/#/c/482500 | 16:11 |
| fultonj | or didn't complete with success | 16:11 |
| * fultonj will try to figure out why from those logs | 16:11 | |
| catintheroof | EmilienM: ok, please let me know when you can have a min | 16:11 |
| *** rcernin has quit IRC | 16:12 | |
| EmilienM | fultonj: thanks for the help - I'm debugging scenario003 and 004 now | 16:12 |
| EmilienM | (they also have failures in pingtest) | 16:12 |
| *** rlandy|ci_mtgs has quit IRC | 16:12 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Upgrade testing https://review.openstack.org/489710 | 16:13 |
| EmilienM | fultonj: in the meantime, I'm trying again with the patches that merged recently ^ | 16:13 |
| fultonj | ack | 16:14 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Test containers scenarios with pingtest - do not merge https://review.openstack.org/489710 | 16:14 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: CI testing: run tempest in container jobs https://review.openstack.org/490172 | 16:15 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/puppet-tripleo master: Add logrotate-crond configuration https://review.openstack.org/490048 | 16:17 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Test containers scenario001 with tempest - do not merge https://review.openstack.org/490172 | 16:18 |
| akrivoka | jtomasek: you mentioned some patch regarding the introspection workflow, could you link me to it? | 16:19 |
| openstackgerrit | Rajini Karthik proposed openstack/puppet-tripleo master: Support for Dell EMC Unity Manila Driver https://review.openstack.org/490581 | 16:20 |
| jtomasek | akrivoka: sorry, I forgot. it is already merged, check 'introspect' workflow in tripleo-common | 16:21 |
| akrivoka | jtomasek: ack, thanks! | 16:21 |
| *** snecklifter has quit IRC | 16:21 | |
| jtomasek | akrivoka: it calls _introspect workflow for each node separately and sends a message | 16:21 |
| *** nyechiel has joined #tripleo | 16:23 | |
| fultonj | EmilienM: the container host has a key and ceph.conf like it should http://logs.openstack.org/10/489710/2/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container/fff73d0/logs/subnode-2/etc/ceph/ | 16:23 |
| fultonj | my next question is if it made it into the container | 16:24 |
| EmilienM | fultonj: it must be something else, i'm trying to reproduce now | 16:24 |
| EmilienM | fultonj: how is going the work with ceph-ansible now? | 16:24 |
| EmilienM | fultonj: what are blockers? | 16:24 |
| *** ccamacho has joined #tripleo | 16:24 | |
| fultonj | EmilienM: jistr just got the playbook running and it failed and i don't know why yet | 16:24 |
| *** jtomasek has quit IRC | 16:25 | |
| fultonj | it was blocked on the tripleo-admin user, but he got me past that ~20 minutes ago | 16:25 |
| EmilienM | fultonj: ok | 16:25 |
| *** ccamacho has quit IRC | 16:26 | |
| EmilienM | catintheroof: puppet will override the value that it has in catalog | 16:26 |
| EmilienM | catintheroof: that's the whole purpose of using tools like puppet or ansible: they ensure resources exist | 16:26 |
| catintheroof | EmilienM: what i mean is, doing a stack upgrade is the way to go right ? | 16:28 |
| catintheroof | EmilienM: if i want to try if something i changed will change the config files ? | 16:28 |
| openstackgerrit | James Slagle proposed openstack/tripleo-common stable/newton: Add skip_deploy_identifier https://review.openstack.org/489813 | 16:31 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Enable pingtest on scenarios container jobs https://review.openstack.org/490129 | 16:31 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Test containers scenarios with pingtest - do not merge! https://review.openstack.org/489710 | 16:31 |
| EmilienM | catintheroof: if you run stack update, puppet will run again and your custom parameters will be overidden | 16:33 |
| *** Goneri has quit IRC | 16:33 | |
| EmilienM | catintheroof: does it help? | 16:33 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/puppet-tripleo master: Add logrotate-crond configuration https://review.openstack.org/490048 | 16:33 |
| *** dsariel has quit IRC | 16:34 | |
| openstackgerrit | Ian Main proposed openstack/tripleo-docs master: Fix some grammar and add a few notes to docs. https://review.openstack.org/490585 | 16:35 |
| *** nyechiel has quit IRC | 16:36 | |
| *** gkadam has joined #tripleo | 16:37 | |
| *** lucasagomes is now known as lucas-afk | 16:42 | |
| fultonj | how do you debug a container failure by looking at the logs of the subnode from multinode-oooq-container ? any output from http://tripleo.org/containers_deployment/tips_tricks.html#debugging-container-failures or similar available ? | 16:43 |
| fultonj | EmilienM: ^ | 16:43 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates master: Persist containerized services httpd logs https://review.openstack.org/489955 | 16:44 |
| openstackgerrit | James Slagle proposed openstack/python-tripleoclient stable/newton: Add --skip-deploy-identifier https://review.openstack.org/489812 | 16:44 |
| openstackgerrit | John Eckersberg proposed openstack/tripleo-quickstart-extras master: Use shell module when deploying freeipa https://review.openstack.org/490590 | 16:45 |
| EmilienM | fultonj: in a mtg, bbl | 16:47 |
| fultonj | ack | 16:47 |
| catintheroof | EmilienM: yes, it does, but i dont see that happen | 16:48 |
| openstackgerrit | Ian Main proposed openstack/tripleo-heat-templates master: Make our json output prettier. https://review.openstack.org/490591 | 16:48 |
| catintheroof | EmilienM: i've created the deployed-ctrl-plane but when i use something like compute01-tenant: to fix a tenant ip for a specific compute, it doesn't work when i run the stack update | 16:49 |
| fultonj | he's in a meeting | 16:49 |
| bogdando | PTAL this one also please https://review.openstack.org/#/c/489955/ | 16:50 |
| catintheroof | fultonj: oka! can you help ? or is it better to wait for him ? | 16:50 |
| fultonj | catintheroof: network settings are created but not updated | 16:51 |
| fultonj | yes, 'openstack overcloud update' will re-run puppet (or similar idempotent config tools) | 16:51 |
| fultonj | but network settings are exlucded to prevent errors from knocking you off line (that's my understanding of it) | 16:52 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates master: Add logrotate with crond service https://review.openstack.org/490050 | 16:52 |
| catintheroof | fultonj: so, if something was not set right from the deployment stage, and you want to "update" it, how can i do that the right way ? | 16:53 |
| catintheroof | fultonj: by hand ? | 16:53 |
| catintheroof | fultonj: server by server ? | 16:53 |
| fultonj | depends on what somehting is | 16:53 |
| catintheroof | fultonj: eg. my_ip value on ml2_conf.ini on every compute node | 16:54 |
| catintheroof | fultonj: used to close the vxlan tunnels between network nodes and computes | 16:54 |
| *** pcaruana has quit IRC | 16:54 | |
| catintheroof | fultonj: sorry, local_ip | 16:55 |
| fultonj | there may be a way to do it via update, but if the box is experimental, redeployment would be easier | 16:55 |
| fultonj | i know the basics of tripleo network config for my purposes, but it's not my area | 16:56 |
| *** stee_3_ has joined #tripleo | 16:56 | |
| catintheroof | fultonj: is not experimental, what would be the way to do that via update | 16:56 |
| *** Goneri has joined #tripleo | 16:56 | |
| fultonj | since you know tripleo won't update your network (badly) config outside of tripleo shouldn't get broken by updates | 16:57 |
| *** bugzy_ has joined #tripleo | 16:57 | |
| *** dtantsur is now known as dtantsur|afk | 16:57 | |
| fultonj | by badly i mean, in the case of user erro or by differeing the config tools definition to undo your "outside of tripleo change" | 16:58 |
| *** derekh has quit IRC | 16:58 | |
| catintheroof | fultonj: so i should touch it by hand, ok | 16:58 |
| fultonj | errr | 16:58 |
| fultonj | you could | 16:58 |
| fultonj | i don't want to misrepresent what better advice others may have | 16:58 |
| catintheroof | fultonj: last one, for "update" i run this command : openstack overcloud deploy (and the same parameters as the first time) | 16:59 |
| *** bogdando has quit IRC | 16:59 | |
| fultonj | yes | 16:59 |
| catintheroof | fultonj: is that the same as openstack overcloud update ? | 16:59 |
| fultonj | i don't use 'openstack overcloud update' | 17:00 |
| fultonj | i use deploy again and it 'makes it so' | 17:00 |
| *** stee_3 has quit IRC | 17:00 | |
| *** bugzy has quit IRC | 17:00 | |
| fultonj | if i were to make a change like scale a node for example | 17:00 |
| *** bugzy_ has quit IRC | 17:00 | |
| *** tosky has quit IRC | 17:01 | |
| *** bugzy has joined #tripleo | 17:02 | |
| fultonj | 'openstack overcloud update' is for updating versions of the packages | 17:02 |
| fultonj | see https://docs.openstack.org/tripleo-docs/latest/install/post_deployment/package_update.html | 17:02 |
| *** rcernin has joined #tripleo | 17:03 | |
| *** nyechiel has joined #tripleo | 17:03 | |
| catintheroof | fultonj: we are using pre-deployed servers, if i configure on deployed-ctrl-plane eg compute01-tenant would that set the right loca_ip on ml2_conf.ini for the tunnel/tenant networks ? | 17:05 |
| fultonj | i do not know anything about network configuration with pre-deployed servers | 17:05 |
| *** ebarrera has quit IRC | 17:05 | |
| catintheroof | fultonj: no problem, thanks ! | 17:06 |
| fultonj | i don't know if it leaves their network settings unchanged or if it updates them. | 17:06 |
| *** ckyriakidou has quit IRC | 17:07 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Add OPNFV scenario environment https://review.openstack.org/486905 | 17:12 |
| EmilienM | michapma: ^ we got promotion | 17:12 |
| fultonj | how do you debug a container failure by looking at the logs of the subnode from multinode-oooq-container ? any output from http://tripleo.org/containers_deployment/tips_tricks.html#debugging-container-failures or similar available ? | 17:16 |
| fultonj | EmilienM: ^ ? | 17:17 |
| fultonj | e.g. for https://review.openstack.org/#/c/489710 i want to see in http://logs.openstack.org/10/489710/2/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container/fff73d0/ if https://review.openstack.org/#/c/482500/21/docker/services/glance-api.yaml happened | 17:18 |
| EmilienM | fultonj: I'm not sure we export /etc from containers | 17:20 |
| EmilienM | let me look | 17:20 |
| *** atoth has quit IRC | 17:20 | |
| EmilienM | we do in http://logs.openstack.org/10/489710/2/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container/fff73d0/logs/subnode-2/var/log/config-data/ | 17:21 |
| EmilienM | fultonj: http://logs.openstack.org/10/489710/2/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container/fff73d0/logs/subnode-2/var/log/config-data/glance_api/etc/glance/ | 17:21 |
| * fultonj looks | 17:21 | |
| *** atoth has joined #tripleo | 17:22 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Deploy Mistral with Keystone v3 options (authtoken) https://review.openstack.org/461040 | 17:23 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Deploy Mistral with Keystone v3 options (authtoken) https://review.openstack.org/461040 | 17:23 |
| EmilienM | thrash: ^ let's see how it works now | 17:23 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates stable/ocata: Remove docker references from stable/ocata scenarios. https://review.openstack.org/489874 | 17:24 |
| openstackgerrit | Emilien Macchi proposed openstack/python-tripleoclient master: Update reno for stable/pike https://review.openstack.org/487888 | 17:24 |
| *** jpena is now known as jpena|off | 17:25 | |
| *** yamahata has joined #tripleo | 17:25 | |
| michapma | EmilienM: ok thanks. | 17:28 |
| *** milan has quit IRC | 17:33 | |
| openstackgerrit | Victoria Martinez de la Cruz proposed openstack/tripleo-heat-templates master: Containerize Manila Share for HA https://review.openstack.org/482680 | 17:36 |
| *** gkadam has quit IRC | 17:39 | |
| *** gkadam has joined #tripleo | 17:39 | |
| *** paramite has quit IRC | 17:41 | |
| *** ebarrera has joined #tripleo | 17:48 | |
| *** tesseract has quit IRC | 17:49 | |
| *** jchhatbar has quit IRC | 17:50 | |
| *** tosky has joined #tripleo | 17:50 | |
| thrash | EmilienM: ack | 17:53 |
| thrash | EmilienM: it occurred to me that poi should be updated as well... But I'm having some issues with it... | 17:54 |
| thrash | EmilienM: https://review.openstack.org/#/c/490494/ | 17:54 |
| EmilienM | thrash: ok I'll look | 17:55 |
| openstackgerrit | Dan Sneddon proposed openstack/tripleo-heat-templates master: Convert controller-role.yaml to role.role.j2.yaml https://review.openstack.org/480272 | 17:56 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Update capabilities map to match latest environments https://review.openstack.org/483873 | 17:59 |
| openstackgerrit | Merged openstack/python-tripleoclient master: Change tag link in README to the newest address https://review.openstack.org/490337 | 17:59 |
| *** amoralej is now known as amoralej|off | 18:04 | |
| openstackgerrit | Numan Siddique proposed openstack/tripleo-heat-templates master: DO NOT REVIEW: TESTING ONLY TO TEST OVN CONTAINER SERVICES https://review.openstack.org/489097 | 18:12 |
| *** nyechiel has quit IRC | 18:16 | |
| *** akrivoka has quit IRC | 18:18 | |
| EmilienM | numans: instead of re-using an existing job for testing OVN in containers, you should rather push for scenario007-containers, isn't? | 18:23 |
| numans | EmilienM, ack. I just noticed that neutron-server-ovn image is available in tripleoupstream docker hub(which I thought isn't available until now). So I will work on the patches for scenario007-containers. Thanks to dprince for pushing the image | 18:24 |
| EmilienM | numans: cool - you also need to patch project-config to add the experimental job probably | 18:25 |
| numans | EmilienM, right. I worked on it some time back. I need to push the patches :) | 18:25 |
| EmilienM | cool | 18:25 |
| EmilienM | numans: let me know if any help | 18:25 |
| numans | EmilienM, sure. thanks | 18:25 |
| *** gkadam has quit IRC | 18:29 | |
| *** salmankhan has quit IRC | 18:31 | |
| catintheroof | EmilienM: hi! quick question, we have a networking nooped on a pre-deployed servers overcloud, the value of local_ip on the ml2_conf.ini only is getting completed with the ctlplane ip of each node, can you please advise me where to complete the servers ips to be completed the right way for the tenant networks ? | 18:31 |
| openstackgerrit | Justin Kilpatrick proposed openstack/tripleo-quickstart-extras master: WIP: Support for TripleO centralized logging https://review.openstack.org/490623 | 18:43 |
| *** MVenesio has quit IRC | 18:47 | |
| openstackgerrit | Numan Siddique proposed openstack/tripleo-quickstart master: Add fset030 for OVN with containerized oc https://review.openstack.org/490624 | 18:49 |
| *** prateek has quit IRC | 18:51 | |
| *** nyechiel has joined #tripleo | 18:52 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: WIP - use infra mirror for buildlogs repo https://review.openstack.org/490629 | 18:57 |
| EmilienM | pabelanger: ^^doing tripleo | 18:57 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: WIP - use infra mirror for buildlogs repo https://review.openstack.org/490629 | 19:03 |
| EmilienM | thrash: http://logs.openstack.org/94/490494/4/check/gate-puppet-openstack-integration-4-scenario003-tempest-centos-7/61ef58b/logs/puppet.txt.gz#_2017-08-03_16_39_33 | 19:04 |
| EmilienM | Please set password for mistral service user | 19:04 |
| thrash | EmilienM: that's what I don't get... It is set... | 19:04 |
| EmilienM | weird | 19:04 |
| EmilienM | yeah let me loolk | 19:04 |
| EmilienM | thrash: problem in the logic :) | 19:06 |
| EmilienM | thrash: I think it should help https://review.openstack.org/490632 | 19:08 |
| *** rhallisey has quit IRC | 19:10 | |
| thrash | EmilienM: I swear that looked correct to me. haha | 19:10 |
| openstackgerrit | Dan Sneddon proposed openstack/tripleo-heat-templates master: Convert controller-role.yaml to role.role.j2.yaml https://review.openstack.org/480272 | 19:10 |
| openstackgerrit | Dan Sneddon proposed openstack/tripleo-heat-templates master: Convert compute-role.yaml to role.role.j2.yaml https://review.openstack.org/482623 | 19:11 |
| openstackgerrit | Dan Sneddon proposed openstack/tripleo-heat-templates master: Convert controller-role.yaml to role.role.j2.yaml https://review.openstack.org/480272 | 19:11 |
| EmilienM | thrash: we'll see | 19:11 |
| dsneddon | I need some help. I have two patches, A and B. A's parent branch is Master, B's parent branch is A. Whenever I update patch B, it reverts patch A to a much older version. At this point, both A and B have the same version of role.role.j2.yaml, so why does committing changes to B overwrite A with older content? | 19:14 |
| dsneddon | A: https://review.openstack.org/480272 | 19:14 |
| *** jcoufal has quit IRC | 19:15 | |
| dsneddon | B: https://review.openstack.org/482623 | 19:15 |
| dsneddon | I just made these changes to A: https://review.openstack.org/#/c/480272/23..24/puppet/role.role.j2.yaml | 19:15 |
| dsneddon | Copied the same version of the file (role.role.j2.yaml) into change B, did a git review, and this happened to change A: https://review.openstack.org/#/c/480272/23..24/puppet/role.role.j2.yaml | 19:16 |
| dsneddon | EmilienM, thrash, mwhahaha: Any ideas? ^^^ | 19:18 |
| *** pchavva has quit IRC | 19:18 | |
| vkmc | dciabrin_, I'm trying to deploy containerized manila with netapp as the backend and I'm hitting this http://paste.openstack.org/show/617467/ | 19:20 |
| openstackgerrit | Swaminathan Vasudevan proposed openstack/diskimage-builder master: Failed to open parameter YAML error while trying to unmount imagedir https://review.openstack.org/490637 | 19:20 |
| vkmc | dciabrin_, is this because this one is not yet merged? https://review.openstack.org/#/c/482680/ | 19:20 |
| EmilienM | dsneddon: first, you need to rebase on top of master | 19:21 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Convert controller-role.yaml to role.role.j2.yaml https://review.openstack.org/480272 | 19:21 |
| EmilienM | done ^ | 19:21 |
| EmilienM | then let's the the other one | 19:21 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Convert compute-role.yaml to role.role.j2.yaml https://review.openstack.org/482623 | 19:21 |
| EmilienM | dsneddon: can you tell if it's ok? | 19:22 |
| EmilienM | dsneddon: i'm not familiar with the code but please say if it's good now | 19:23 |
| dsneddon | EmilienM, That didn't make a difference. Patch A still has an old version of role.role.j2.yaml. | 19:23 |
| EmilienM | ok brb lunch and I'll look closer | 19:24 |
| dsneddon | EmilienM, Thanks | 19:24 |
| *** fzdarsky has joined #tripleo | 19:24 | |
| mwhahaha | might be how we clone stuff | 19:25 |
| mwhahaha | i have a feeling we don't handle dependencies well | 19:25 |
| * mwhahaha starts looking | 19:25 | |
| bnemec | Are both patches on the same local branch? | 19:28 |
| dsneddon | bnemec, No, A is a branch off of master, B is a branch off of A | 19:28 |
| dsneddon | bnemec, At least I think so, based on the "Parent(s)" branch (Steven Hardy did these changes originally) | 19:29 |
| dsneddon | What I'm trying to do is end up with Patch A having all the files from patchset 23, but with a new role.role.j2.yaml that is the same as patch B patchset 21 | 19:29 |
| mwhahaha | is bad in CI or are we just confusing gerrit | 19:29 |
| dsneddon | mwhahaha, Yeah, I suspect that Gerrit is confused. This is why I have started doing depends-on between changes based on Master, rather than sub-branches. I've run into this before. | 19:30 |
| bnemec | dsneddon: If the patches are being pushed from two different local branches that may well be the problem. | 19:30 |
| bnemec | When working with patch series you need to keep all the changes in one branch or it gets confusing. | 19:31 |
| dsneddon | bnemec, I guess I just don't know how to do that with a series of patches that were posted by someone else. | 19:33 |
| mwhahaha | very painfully | 19:33 |
| mwhahaha | we should merge these and stop adding more :D | 19:33 |
| bnemec | If you have branch foo with patch A, then branch bar with patches A and B, you make changes to A on foo, then changes to B on bar, push foo and bar (in that order), you'll end up with A not having the changes you made because the bar version will overwrite it. | 19:33 |
| openstackgerrit | Bob Fournier proposed openstack/tripleo-heat-templates master: Render VIPs dynamically based on network_data.yaml https://review.openstack.org/486260 | 19:33 |
| *** nyechiel has quit IRC | 19:34 | |
| dsneddon | bnemec, OK, I think I'm getting it. So any changes to A (like those releasenotes I added) need to be duplicated into B, or it will overwrite? | 19:34 |
| bnemec | dsneddon: Go to the last patch in the series, click the Download menu and run the Checkout command. | 19:34 |
| bnemec | dsneddon: Yeah, I _suspect_ that's the problem here. | 19:35 |
| dsneddon | bnemec, OK, so submit the changes in reverse order? That's worth a try, thanks. | 19:35 |
| mwhahaha | dsneddon: well you should rebase them all together and only git review the last one of the change | 19:35 |
| openstackgerrit | James Slagle proposed openstack/python-tripleoclient stable/newton: Add --skip-deploy-identifier https://review.openstack.org/489812 | 19:35 |
| bnemec | Yeah, don't do multiple git reviews. | 19:35 |
| dsneddon | mwhahaha, OK, and that will trigger updating all the other patches, that makes sense. | 19:36 |
| bnemec | Only the last one counts anyway. | 19:36 |
| mwhahaha | you can fix it by pulling down the first then just cherrypicking them all on top of eachother | 19:36 |
| dsneddon | bnemec, Of course, this becomes difficult when you find that one patch at the beginning of the series needs a one-line change. | 19:36 |
| bnemec | dsneddon: git rebase -i HEAD~6 (or however many commits are in the series). | 19:37 |
| mwhahaha | yea that's why when we have these we need to merge them soon and not just let them sit because it's a giant pain (and time suck) to manage this over time | 19:37 |
| bnemec | It sucks because it forces a re-run of ci on everything, but actually making the change shouldn't be that difficult. | 19:38 |
| *** artom_ has quit IRC | 19:38 | |
| *** artom_ has joined #tripleo | 19:38 | |
| dsneddon | bnemec, mwhahaha: Yeah, I'm mostly complaining because this means that I can't split the workload (me editing patches 1-3, Bob editing paches 4-7) | 19:38 |
| dsneddon | But it's not the end of the world. Thanks for your help! | 19:39 |
| mwhahaha | well you can | 19:39 |
| bnemec | Ah, yeah. That's one thing gerrit is terrible at. | 19:39 |
| mwhahaha | but it's just complicated | 19:39 |
| dsneddon | Alright, I'm just going to bang out all these changes | 19:39 |
| mwhahaha | it's harder for the 2nd guy who has to constantly rebase on your patches | 19:39 |
| dsneddon | mwhahaha, That's the situation I'm in while shardy is out on PTO. ;-) | 19:41 |
| mwhahaha | :D | 19:41 |
| mwhahaha | let's figure out controller and get that merged at least | 19:41 |
| mwhahaha | that seems to be the sticking point | 19:41 |
| bnemec | Solution: Don't let shardy go on PTO anymore. :-) | 19:41 |
| mwhahaha | i thought that was a given | 19:41 |
| dsneddon | mwhahaha, That's what I'm trying to do, but then when I thought I was close to getting controller done, I did the next patch in the series, and it wiped out the controller changes. | 19:41 |
| mwhahaha | doh | 19:42 |
| *** jmelvin has quit IRC | 19:42 | |
| mwhahaha | well you can pull down the old one that was close and redo tha tone | 19:42 |
| dsneddon | mwhahaha, That's my next step, I want to see if Controller passes CI before I clone the changes to the rest of the branches | 19:42 |
| * bnemec is concerned that there are still multiple branches in play here | 19:44 | |
| bnemec | Do these patches actually need to be in a series? | 19:44 |
| openstackgerrit | Dan Sneddon proposed openstack/tripleo-heat-templates master: Convert controller-role.yaml to role.role.j2.yaml https://review.openstack.org/480272 | 19:45 |
| mwhahaha | yea it's a rewrite of roles.role.j2.yaml | 19:45 |
| mwhahaha | that's the sticking point | 19:45 |
| mwhahaha | to handle the inconsistencies we have with those | 19:45 |
| mwhahaha | so there's a bunch of controller/compute/etc specific stuff that we still need to handle | 19:45 |
| bnemec | Too bad. | 19:46 |
| mwhahaha | this is why consistency is important and we shouldn't one off things :D | 19:46 |
| dsneddon | We could do it as a single patch, but it wouldn't be clear which changes to role.role.j2.yaml were necessary for each role. | 19:46 |
| mwhahaha | and it's be so hard to review | 19:46 |
| bnemec | That^ | 19:46 |
| dsneddon | mwhahaha, I couldn't agree more. If we had been consistent between Compute and NovaCompute in the first place, SO MANY headaches could have been avoided. | 19:47 |
| * mwhahaha is currently trying to figure out why docker deployments keep coming up no host found :/ | 19:47 | |
| bnemec | Heh, that downstream-only change continues to bite us in the ass. | 19:47 |
| bnemec | What, two and a half years later? | 19:47 |
| mwhahaha | dat techdebt | 19:48 |
| * EmilienM back | 19:52 | |
| *** ebarrera_ has joined #tripleo | 19:56 | |
| EmilienM | thrash: I think I was wrong | 19:58 |
| *** fzdarsky has quit IRC | 19:58 | |
| mwhahaha | should we quote you on that one | 19:58 |
| * mwhahaha documents this time for history | 19:58 | |
| EmilienM | lol | 19:58 |
| *** milan has joined #tripleo | 20:00 | |
| EmilienM | thrash: i'm trying some code ordering | 20:00 |
| mwhahaha | anyone know anything about the pre deployment validations (and why it would think i don't have any nodes available when i do or where that stuff even lives) | 20:02 |
| fultonj | mwhahaha: i know that if you use deployed server you can't use validations | 20:03 |
| mwhahaha | this is an ovb setup | 20:03 |
| mwhahaha | it's erroring say i have 0 nodes but it's saying 5 are available in the json response | 20:04 |
| mwhahaha | so i have no idea wtf it's doing | 20:04 |
| bnemec | Looks like it's a mistral workflow now: https://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/workflows/validations.py#L20 | 20:04 |
| mwhahaha | yea | 20:05 |
| mwhahaha | i'm trying to figure out where this stuff lives :/ | 20:05 |
| *** sbrzozow has quit IRC | 20:05 | |
| mwhahaha | i found the workbook at least | 20:05 |
| * mwhahaha keeps digging | 20:05 | |
| *** salmankhan has joined #tripleo | 20:05 | |
| bnemec | TripleO is like an onion, the more layers you peel away the more you cry. | 20:06 |
| fultonj | lol | 20:06 |
| mwhahaha | also true | 20:06 |
| mwhahaha | it's a work flow to run /usr/bin/run-validation | 20:06 |
| mwhahaha | https://github.com/openstack/tripleo-common/blob/master/tripleo_common/utils/validations.py#L80 | 20:06 |
| * bnemec thinks that needs to be a t-shirt | 20:06 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: DNM - run tempest in scenario001-container https://review.openstack.org/490167 | 20:07 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Test containers scenario001 with tempest - do not merge https://review.openstack.org/490172 | 20:07 |
| EmilienM | bnemec: nice one | 20:07 |
| mwhahaha | soooo the question is what provides run-validation | 20:08 |
| *** liverpooler has quit IRC | 20:08 | |
| mwhahaha | tripleo-validations | 20:08 |
| mwhahaha | that's a new one | 20:08 |
| mwhahaha | hrm can't find a reference to this error there | 20:10 |
| openstackgerrit | Numan Siddique proposed openstack/tripleo-common master: Add OVN images to the overcloud containers https://review.openstack.org/489875 | 20:10 |
| * mwhahaha sighs | 20:10 | |
| *** salmankhan has quit IRC | 20:10 | |
| mwhahaha | ah https://github.com/openstack/tripleo-common/blob/3a9301739650b511774fc9f6a5cde9ace243715e/tripleo_common/actions/validations.py#L424 | 20:10 |
| *** jmelvin has joined #tripleo | 20:11 | |
| openstackgerrit | Numan Siddique proposed openstack/tripleo-common master: Add OVN images to the overcloud containers https://review.openstack.org/489875 | 20:13 |
| *** eck` is now known as eck`gone | 20:14 | |
| *** mmethot_ has joined #tripleo | 20:14 | |
| *** mmethot has quit IRC | 20:17 | |
| *** morazi has quit IRC | 20:17 | |
| mwhahaha | why do we keep changing the node import command | 20:26 |
| * mwhahaha throws things | 20:26 | |
| *** morazi has joined #tripleo | 20:26 | |
| *** dhill_ has joined #tripleo | 20:28 | |
| *** nyechiel has joined #tripleo | 20:31 | |
| openstackgerrit | Rajini Karthik proposed openstack/puppet-tripleo master: Support for Dell EMC VNX Manila Driver https://review.openstack.org/490642 | 20:36 |
| *** nyechiel has quit IRC | 20:36 | |
| openstackgerrit | Rajini Karthik proposed openstack/puppet-tripleo master: Support for Dell EMC VNX Manila Driver https://review.openstack.org/490642 | 20:37 |
| *** ansmith has quit IRC | 20:37 | |
| *** hamzy has quit IRC | 20:37 | |
| stevebaker | morning | 20:42 |
| *** ebarrera has quit IRC | 20:43 | |
| EmilienM | afternoon | 20:46 |
| EmilienM | stevebaker: nice fix for oooclient, it works, we're backporting it to stable/ocata now | 20:46 |
| EmilienM | err, stable/pike | 20:46 |
| openstackgerrit | Emilien Macchi proposed openstack/python-tripleoclient stable/ocata: Install policycoreutils-python https://review.openstack.org/490645 | 20:47 |
| EmilienM | ocata now | 20:47 |
| openstackgerrit | Emilien Macchi proposed openstack/python-tripleoclient stable/newton: Install policycoreutils-python https://review.openstack.org/490646 | 20:47 |
| EmilienM | even newton | 20:47 |
| openstackgerrit | Merged openstack/tripleo-common master: [refactoring] stop using pxe_ssh as the default drivers in tests https://review.openstack.org/487166 | 20:48 |
| *** eck`gone is now known as eck` | 20:48 | |
| *** jkilpatr has quit IRC | 20:48 | |
| EmilienM | therve: sounds like zaqar fails to work when containerized | 20:49 |
| EmilienM | therve: http://logs.openstack.org/10/489710/5/check/gate-tripleo-ci-centos-7-scenario002-multinode-oooq-container/019039b/logs/undercloud/home/jenkins/overcloud_validate.log.txt.gz#_2017-08-03_18_02_19 | 20:50 |
| *** rhallisey has joined #tripleo | 20:51 | |
| EmilienM | dprince: is zaqar suppose to work in containers? | 20:52 |
| openstackgerrit | Steve Baker proposed openstack/python-tripleoclient master: Add --set argument to prepare command https://review.openstack.org/489083 | 20:52 |
| EmilienM | dprince: pingtest doesn't seem to pass | 20:52 |
| EmilienM | and no logs: | 20:52 |
| EmilienM | http://logs.openstack.org/10/489710/5/check/gate-tripleo-ci-centos-7-scenario002-multinode-oooq-container/019039b/logs/subnode-2/var/log/containers/ | 20:52 |
| EmilienM | probably because we need https://review.openstack.org/#/c/490460/1 for logs | 20:53 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Test containers scenarios with pingtest - do not merge! https://review.openstack.org/489710 | 20:53 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Test containers scenarios with pingtest - do not merge! https://review.openstack.org/489710 | 20:53 |
| EmilienM | let's try again with logs | 20:53 |
| *** nyechiel has joined #tripleo | 20:55 | |
| *** _mmethot_ has joined #tripleo | 20:57 | |
| *** milan has quit IRC | 20:58 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: WIP - use infra mirror for buildlogs repo https://review.openstack.org/490629 | 20:59 |
| *** mmethot_ has quit IRC | 21:00 | |
| openstackgerrit | David Hill proposed openstack/diskimage-builder master: Clear /etc/machine-id to avoid duplicate machine-ids https://review.openstack.org/489013 | 21:02 |
| *** itlinux has joined #tripleo | 21:03 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Configure the set of tests to execute in scenario001 https://review.openstack.org/490376 | 21:03 |
| EmilienM | mwhahaha: with autoscaling test ^ | 21:03 |
| EmilienM | brb 20 min | 21:03 |
| *** agopi|away has quit IRC | 21:04 | |
| *** dparkes has joined #tripleo | 21:05 | |
| stevebaker | EmilienM: cool | 21:08 |
| mwhahaha | blah bmc is giving me fits trying to restart my ovb nodes it looks like | 21:10 |
| *** lblanchard has quit IRC | 21:13 | |
| *** mmethot_ has joined #tripleo | 21:14 | |
| openstackgerrit | Lokesh Jain proposed openstack/tripleo-heat-templates master: Addition of Nuage as mechanism driver for ML2 https://review.openstack.org/474788 | 21:15 |
| *** _mmethot_ has quit IRC | 21:16 | |
| *** agopi has joined #tripleo | 21:17 | |
| openstackgerrit | Steve Baker proposed openstack/tripleo-common master: Upload remove defaults for pull_source, push_destination https://review.openstack.org/489837 | 21:19 |
| mwhahaha | bnemec: have you seen introspection fail before when the node is already off? it seems that ipmitool is error when trying to request the node off | 21:20 |
| openstackgerrit | Julia Kreger proposed openstack/diskimage-builder master: Add netbase to ensure /etc/protocols is placed for debian https://review.openstack.org/490656 | 21:20 |
| bnemec | mwhahaha: RDO cloud? I know dtrainor was having issues with his bmc in that cloud recently. | 21:21 |
| mwhahaha | bnemec: rh2 | 21:21 |
| mwhahaha | i'm using my original ovb setup from like 6 months ago | 21:21 |
| mwhahaha | and it used to work | 21:21 |
| mwhahaha | but now introspection is just hanging and all i see in the logs is trying to turn the nodes off | 21:22 |
| openstackgerrit | Merged openstack/tripleo-common master: Update the documentation link for doc migration https://review.openstack.org/486914 | 21:22 |
| openstackgerrit | Merged openstack/tripleo-common master: Update URL home-page in documents according to document migration https://review.openstack.org/483750 | 21:22 |
| bnemec | mwhahaha: Hmm, anything in the bmc console-log? | 21:22 |
| openstackgerrit | Merged openstack/os-net-config master: OvS 2.7 support - dpdk-devargs needs to provided for DPDK devices https://review.openstack.org/480540 | 21:22 |
| mwhahaha | bnemec: just Aug 3 21:22:21 bmc openstackbmc: Getting power state for e04690d1-ddc6-4b17-bf4d-4a8126945ba0 | 21:22 |
| openstackgerrit | David Hill proposed openstack/diskimage-builder master: Clear /etc/machine-id to avoid duplicate machine-ids https://review.openstack.org/489013 | 21:22 |
| *** ansmith has joined #tripleo | 21:22 | |
| mwhahaha | bnemec: ironic-conductor shows http://paste.openstack.org/show/617470/ | 21:23 |
| *** agopi has quit IRC | 21:23 | |
| mwhahaha | bnemec: so it seems that if the nodes aren't getting powered on or something | 21:24 |
| bnemec | mwhahaha: Hmm, I wonder if that's from back when bmc caching was enabled by default. Have you tried rebooting the bmc instance? | 21:24 |
| mwhahaha | if i manually start them introspection fiinsihes | 21:24 |
| openstackgerrit | David Hill proposed openstack/diskimage-builder master: Clear /etc/machine-id to avoid duplicate machine-ids https://review.openstack.org/489013 | 21:24 |
| *** morazi has quit IRC | 21:25 | |
| openstackgerrit | David Hill proposed openstack/diskimage-builder master: Clear /etc/machine-id to avoid duplicate machine-ids https://review.openstack.org/489013 | 21:26 |
| bnemec | mwhahaha: Also, is there any chance this is old enough to not have https://github.com/cybertron/openstack-virtual-baremetal/commit/0b73548248a33baf3167fe59c00e8d520c4f021d ? | 21:26 |
| mwhahaha | bnemec: probably | 21:26 |
| bnemec | It looks like that may be related. | 21:26 |
| mwhahaha | i don't want to upgrade cause it was working :D and everytime i touch this stuff it just falls apart | 21:27 |
| mwhahaha | but i guess i'll do that tomorrow | 21:27 |
| dtrainor | yea, i had issues with it | 21:27 |
| bnemec | mwhahaha: You could hack the bmc code. It's just two lines that need to be removed. | 21:27 |
| mwhahaha | i'm sure i'm missing a bunch of stuff | 21:27 |
| openstackgerrit | David Hill proposed openstack/diskimage-builder master: Clear /etc/machine-id to avoid duplicate machine-ids https://review.openstack.org/489013 | 21:27 |
| mwhahaha | yay my introspection finally finished | 21:28 |
| dtrainor | it has good days, and it has bad days | 21:28 |
| * mwhahaha is too familar with this concept and would like more good than bad | 21:28 | |
| dtrainor | they're talking of applying that nova boot patch, i was supposed to test it today but didn't hear back about it | 21:29 |
| dtrainor | that would alleviate many of those boot related problems during introspection | 21:29 |
| mwhahaha | well this was my rh2 setup which has that patch | 21:30 |
| dtrainor | we'd be less dependent on what the bmc reports because we'd know that the power and boot operation on the node actually worked because the node would start to report introspection data | 21:30 |
| dtrainor | oh, my bad, missed that part | 21:30 |
| mwhahaha | i'm trying to reproduce the ovb-containers (without quickstart) | 21:30 |
| dtrainor | got ya | 21:30 |
| mwhahaha | and when i go to deploy containers it's like you have no nodes | 21:30 |
| dtrainor | go back to ignoring me :) | 21:30 |
| dtrainor | yeah | 21:30 |
| mwhahaha | so my introspection was bad | 21:30 |
| mwhahaha | just fixed tha tbut i think it's still going to fail | 21:30 |
| dtrainor | ohnodes | 21:30 |
| mwhahaha | so now i want to know wtf we're doing by default | 21:30 |
| mwhahaha | Only 0 nodes are exposed to Nova of 5 requests. Check that enough nodes are in "available" state with maintenance mode off. | 21:31 |
| * mwhahaha sighs | 21:31 | |
| fultonj | EmilienM: FYI: it looks to me like containerized glance as a ceph-client in https://review.openstack.org/#/c/489710 is configured cortrectly ; details in http://paste.openstack.org/show/617472 | 21:31 |
| * mwhahaha flips tables | 21:40 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci master: Don't do pre-pingtest cleanup with --skip-pingtest-cleanup https://review.openstack.org/490661 | 21:40 |
| mwhahaha | why is the ComputeFilter excluding all my hosts | 21:43 |
| *** rhallisey has quit IRC | 21:50 | |
| openstackgerrit | Merged openstack/tripleo-quickstart master: Add consistent-pike.yml configuration https://review.openstack.org/490472 | 21:50 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Add bits to support stable/pike https://review.openstack.org/490077 | 21:51 |
| openstackgerrit | Merged openstack/tripleo-quickstart master: Move release notes to the right directory https://review.openstack.org/490127 | 21:51 |
| openstackgerrit | Merged openstack/puppet-tripleo stable/ocata: Remove unnecessary references to neutron core plugin hiera https://review.openstack.org/482211 | 21:52 |
| openstackgerrit | Merged openstack/python-tripleoclient stable/pike: Install policycoreutils-python https://review.openstack.org/490504 | 21:52 |
| openstackgerrit | Merged openstack/python-tripleoclient master: Fail early in case of error in env directory path https://review.openstack.org/475442 | 21:52 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Add environment for setting a custom domain name https://review.openstack.org/476219 | 21:52 |
| openstackgerrit | Merged openstack/python-tripleoclient stable/pike: Update .gitreview for stable/pike https://review.openstack.org/487886 | 21:52 |
| openstackgerrit | Merged openstack/python-tripleoclient stable/pike: Update UPPER_CONSTRAINTS_FILE for stable/pike https://review.openstack.org/487887 | 21:52 |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Install openstack services tempest plugin https://review.openstack.org/490369 | 21:52 |
| openstackgerrit | Merged openstack/python-tripleoclient master: Update reno for stable/pike https://review.openstack.org/487888 | 21:53 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Make UpgradeLevelNovaCompute parameters consistent https://review.openstack.org/484041 | 21:53 |
| dprince | EmilienM: zaqar in containers works for me | 21:53 |
| dprince | EmilienM: on the undercloud it does at least | 21:53 |
| * dprince has to sign off now though | 21:54 | |
| *** dprince has quit IRC | 21:54 | |
| *** cdearborn has quit IRC | 21:54 | |
| *** jmelvin has quit IRC | 21:57 | |
| openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart-extras master: GATE CHECK for quickstart-extras https://review.openstack.org/472607 | 22:00 |
| *** jlabarre has quit IRC | 22:12 | |
| *** lblanchard has joined #tripleo | 22:15 | |
| *** ebarrera_ has quit IRC | 22:17 | |
| *** lblanchard has quit IRC | 22:17 | |
| mwhahaha | we test containers with a baremetal (vm) undercloud right? | 22:17 |
| mwhahaha | it seems as though my nova-compute service was disabled on my undercloud (wtf) | 22:23 |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates master: Update EventPipelinePublisher param description to include zaqar https://review.openstack.org/487861 | 22:38 |
| dtrainor | mwhahaha, disabled, or died? | 22:57 |
| *** rcernin has quit IRC | 22:58 | |
| *** nyechiel has quit IRC | 23:01 | |
| EmilienM | fultonj: but do we have the keys in the container? | 23:05 |
| EmilienM | fultonj: I mean, /etc/ceph/ceph.conf in the glance-api container? | 23:05 |
| stevebaker | EmilienM: hey, I've just sent an FFE email to the list | 23:06 |
| *** gbarros has joined #tripleo | 23:06 | |
| EmilienM | stevebaker: ok, my gmail doesn't have it yet but I'll look | 23:07 |
| mwhahaha | dtrainor: disabled | 23:07 |
| *** tosky has quit IRC | 23:07 | |
| openstackgerrit | Dan Sneddon proposed openstack/tripleo-heat-templates master: Convert controller-role.yaml to role.role.j2.yaml https://review.openstack.org/480272 | 23:07 |
| EmilienM | damn I think ovb jobs are broken again | 23:09 |
| EmilienM | mwhahaha: I think https://review.openstack.org/#/c/481151/ broke us again | 23:09 |
| EmilienM | I'm confused though | 23:09 |
| EmilienM | it passed CI | 23:09 |
| EmilienM | oh maybe we don't have https://review.openstack.org/#/c/481947/ in the image | 23:10 |
| mwhahaha | Blah | 23:11 |
| EmilienM | the new image should have it | 23:14 |
| EmilienM | http://66.187.229.139/builds-master/current-tripleo/metadata.txt | 23:14 |
| EmilienM | hash is c8/00/c80011cd91cbbd71c66064dbfdd69fed0fb6d705_65fed624 | 23:14 |
| EmilienM | https://trunk.rdoproject.org/centos7-master/c8/00/c80011cd91cbbd71c66064dbfdd69fed0fb6d705_65fed624/ | 23:14 |
| EmilienM | https://trunk.rdoproject.org/centos7-master/c8/00/c80011cd91cbbd71c66064dbfdd69fed0fb6d705_65fed624/openstack-mistral-5.0.0-0.20170803070025.c7fe4e8.el7.centos.src.rpm | 23:14 |
| EmilienM | c7fe4e8 is the commit we need, so we should be good | 23:14 |
| EmilienM | but it sounds like it doesn't work, failing jobs have this commit: | 23:15 |
| EmilienM | http://logs.openstack.org/72/480272/26/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha-oooq/08a0163/logs/rpm-qa.txt.gz | 23:15 |
| EmilienM | thrash: ^ | 23:18 |
| EmilienM | any idea on this trace? http://logs.openstack.org/72/480272/26/check-tripleo/gate-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024/4ce0ea6/logs/undercloud/home/jenkins/overcloud_prep_images.log.txt.gz#_2017-08-03_20_53_39 | 23:18 |
| *** Goneri has quit IRC | 23:20 | |
| openstackgerrit | Merged openstack/tripleo-common master: Pass UserKnownHostsFile and deploy ssh key https://review.openstack.org/487650 | 23:21 |
| *** mmethot has joined #tripleo | 23:22 | |
| *** _mmethot_ has joined #tripleo | 23:24 | |
| *** itlinux has quit IRC | 23:25 | |
| *** mmethot_ has quit IRC | 23:25 | |
| *** Goneri has joined #tripleo | 23:27 | |
| *** mmethot has quit IRC | 23:28 | |
| EmilienM | arxcruz: the tempest pushed by quickstart requires cinder, we need to stop that | 23:29 |
| EmilienM | arxcruz: scenario003 doesn't have cinder: http://logs.openstack.org/72/490172/4/check/gate-tripleo-ci-centos-7-scenario003-multinode-oooq-container/a3efcae/logs/undercloud/home/jenkins/tempest_output.log.txt.gz#_2017-08-03_22_27_28 | 23:29 |
| *** tosky has joined #tripleo | 23:31 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Deploy Mistral with Keystone v3 options (authtoken) https://review.openstack.org/461040 | 23:32 |
| *** Goneri has quit IRC | 23:33 | |
| *** hamzy has joined #tripleo | 23:33 | |
| *** _mmethot_ has quit IRC | 23:34 | |
| *** _mmethot_ has joined #tripleo | 23:34 | |
| openstackgerrit | Emilien Macchi proposed openstack/instack-undercloud master: Configure Mistral with authoken parameters https://review.openstack.org/490680 | 23:35 |
| fultonj | EmilienM: regarding your question, do we know if the conf and keys made it into the container... | 23:37 |
| fultonj | I don't see in the subnode-2 logs a way to see how 'docker run' was used or the output of 'docker inspect' | 23:37 |
| fultonj | http://logs.openstack.org/10/489710/5/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container/e2c09a1/logs/subnode-2/ | 23:37 |
| *** tosky has quit IRC | 23:38 | |
| *** nyechiel has joined #tripleo | 23:38 | |
| fultonj | the code to put it there merged on the 25th, so I'm assuming it is present but i agree we should be able to find evidence that it ran ; i am not sure how ; ci collect logs change? | 23:39 |
| EmilienM | re: mistral/introspection, I filed a new alert https://bugs.launchpad.net/tripleo/+bug/1708554 | 23:39 |
| openstack | Launchpad bug 1708554 in tripleo "Introspection is failing with latest Mistral & puppet-mistral" [Critical,Triaged] | 23:39 |
| EmilienM | fultonj: I don't know | 23:39 |
| EmilienM | fultonj: I'm pretty sure there is no code that takes care of pushing ceph.conf file in glance container, when ceph-ansible isn't used, right? | 23:40 |
| fultonj | no, i will show you | 23:40 |
| fultonj | the client change which pushes it in doesn't care which tool made the ceph.conf or keys | 23:40 |
| fultonj | should work with puppet-ceph or ceph-ansible | 23:41 |
| * fultonj gets review | 23:41 | |
| fultonj | https://review.openstack.org/#/c/482500/ | 23:41 |
| fultonj | https://review.openstack.org/#/c/482500/21/docker/services/glance-api.yaml | 23:41 |
| EmilienM | looking | 23:42 |
| EmilienM | fultonj: oh indeed | 23:42 |
| openstackgerrit | Merged openstack/python-tripleoclient stable/ocata: Install policycoreutils-python https://review.openstack.org/490645 | 23:42 |
| EmilienM | sounds like it doesn't work | 23:43 |
| EmilienM | or not like we want | 23:43 |
| fultonj | and since i see the right config in glance-api.conf and the keys and config on the container | 23:43 |
| fultonj | we need to know what's going on though | 23:43 |
| EmilienM | yeah? | 23:43 |
| EmilienM | where do you see the ceph.conf? | 23:43 |
| fultonj | http://logs.openstack.org/10/489710/5/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container/e2c09a1/logs/subnode-2/etc/ceph/ | 23:44 |
| fultonj | clarification | 23:44 |
| fultonj | ... and the keys and config on the container HOST | 23:44 |
| EmilienM | fultonj: ok | 23:45 |
| fultonj | sorry, yes it would be gerat to see it on teh container | 23:45 |
| EmilienM | fultonj: so it should work :D | 23:46 |
| EmilienM | I now understand why you wanted to see docker outputs | 23:46 |
| EmilienM | to see if /etc/ceph is actually mounted to the glance-api container | 23:46 |
| EmilienM | right? | 23:46 |
| EmilienM | stevebaker: nice one :D | 23:46 |
| EmilienM | I don't know if we can | 23:46 |
| EmilienM | tbh | 23:46 |
| fultonj | exactly | 23:46 |
| EmilienM | fultonj: anyway, let's make progress on the ceph-ansible thing | 23:47 |
| EmilienM | fultonj: and hope we can make it work that way | 23:48 |
| EmilienM | fultonj: are we going to support both ways in pike? | 23:48 |
| EmilienM | or just ceph-ansible? | 23:48 |
| fultonj | both ways | 23:48 |
| EmilienM | mhh ok | 23:48 |
| EmilienM | so we have to make this thing work | 23:48 |
| EmilienM | fultonj: can you help me and get some eyes on my problem | 23:48 |
| EmilienM | ? | 23:48 |
| fultonj | yes | 23:48 |
| EmilienM | see why glance doesn't work | 23:48 |
| EmilienM | fultonj: the patch is https://review.openstack.org/#/c/489710/ | 23:49 |
| EmilienM | see all -container jobs failing when pingtest is enabled | 23:49 |
| EmilienM | except for scenario003! wooo | 23:49 |
| fultonj | progress | 23:49 |
| EmilienM | yes :) | 23:50 |
| fultonj | i will ask someone in storage-dfg regarding glance | 23:50 |
| EmilienM | I have to go now but I'm back later after dinner | 23:50 |
| thrash | EmilienM: Looks like a bug that's rearing its ugly head... But I'm not sure why. | 23:50 |
| fultonj | i'll be here | 23:50 |
| fultonj | question: ... | 23:50 |
| EmilienM | thrash: do you think it's related to puppet-mistral patch or the mistral patch? or none? | 23:50 |
| fultonj | why not add a patch that 489710 can depends-on which pulls in those logs from the container? | 23:50 |
| thrash | EmilienM: something keeps introspection from starting... And if that doesn't start, we can't get the actions from the client. | 23:51 |
| EmilienM | thrash: I file https://bugs.launchpad.net/tripleo/+bug/1708554 - feel free to add more debug / thoughts | 23:51 |
| openstack | Launchpad bug 1708554 in tripleo "Introspection is failing with latest Mistral & puppet-mistral" [Critical,Triaged] | 23:51 |
| thrash | EmilienM: not sure yet. Let me dig deeper. | 23:51 |
| EmilienM | thrash: thanks | 23:51 |
| fultonj | perhaps easier said then done, but should we take that approach | 23:51 |
| fultonj | EmilienM: ^? | 23:51 |
| EmilienM | thrash: I need to do some errand, I'm back in a few | 23:51 |
| thrash | EmilienM: ack | 23:51 |
| EmilienM | fultonj: yeah | 23:52 |
| fultonj | ok | 23:52 |
| EmilienM | fultonj: we can do that, feel free to do so if you know how to get these logs | 23:52 |
| EmilienM | fultonj: keep in mind the existing depends-on | 23:52 |
| fultonj | yeah | 23:52 |
| EmilienM | if you have to add one to an existing repo, put it on top of the patch | 23:52 |
| EmilienM | don't adds multiple Depends-on from the same repo | 23:52 |
| EmilienM | it doesn't work | 23:52 |
| fultonj | right | 23:52 |
| EmilienM | use git rebase + 1 depends-on | 23:52 |
| EmilienM | just in case | 23:52 |
| * EmilienM afk now | 23:52 | |
| fultonj | general question for anyone: was larsks looking at this kind of thing? | 23:52 |
| fultonj | "this kind of thing" --> getting info about the running containers into the logs available for someone to examine in the ci results | 23:53 |
| *** rbowen has quit IRC | 23:54 | |
| larsks | fultonj: I was looking into setting up container healthchecks, and I've been mumbling about container logging. I'm not sure that either of those things is what you're asking about, though. | 23:58 |
| fultonj | larsks: ok, yes i saw you eamil about that but wasn't sure if what i was looking for was included | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!