| openstackgerrit | RedHat RDO CI proposed openstack/tripleo-heat-templates stable/queens: GATE CHECK for TripleO https://review.openstack.org/567224 | 00:00 |
|---|---|---|
| openstackgerrit | RedHat RDO CI proposed openstack/tripleo-quickstart-extras master: GATE CHECK for quickstart-extras https://review.openstack.org/560445 | 00:00 |
| *** ooolpbot has joined #tripleo | 00:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1786764 | 00:10 |
| *** ooolpbot has quit IRC | 00:10 | |
| openstack | Launchpad bug 1786764 in tripleo "tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates times out on prepare" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 00:10 |
| weshay | mwhahaha, were you able to combine an openstack change and a distgit change? | 00:17 |
| mwhahaha | weshay: I'm just using it to build the latest of the packages so it would be I'm the data formater to get it working in ci | 00:18 |
| mwhahaha | But that patch is the first step to get there | 00:18 |
| mwhahaha | You could only really do one. I could tweak it so the dist it gets applied last though | 00:19 |
| *** zaneb has quit IRC | 00:19 | |
| mwhahaha | It's an issue with the role itself | 00:19 |
| *** slagle has joined #tripleo | 00:24 | |
| *** agopi has quit IRC | 00:32 | |
| *** zaneb has joined #tripleo | 00:33 | |
| *** thrash is now known as thrash|g0ne | 00:39 | |
| *** rlandy has quit IRC | 00:43 | |
| *** slagle has quit IRC | 00:48 | |
| *** slaweq_ has joined #tripleo | 00:53 | |
| *** slaweq_ has quit IRC | 00:57 | |
| *** agopi has joined #tripleo | 00:57 | |
| weshay | ya.. will take a look, thanks for getting it started | 01:01 |
| *** khyr0n has quit IRC | 01:03 | |
| *** ooolpbot has joined #tripleo | 01:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1786764 | 01:10 |
| *** ooolpbot has quit IRC | 01:10 | |
| openstack | Launchpad bug 1786764 in tripleo "tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates times out on prepare" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 01:10 |
| *** sri_ has joined #tripleo | 01:26 | |
| *** sri_ has joined #tripleo | 01:27 | |
| *** mrsoul has joined #tripleo | 01:28 | |
| *** srini_ has joined #tripleo | 01:39 | |
| *** sri_ has quit IRC | 01:42 | |
| *** gbarros has joined #tripleo | 01:51 | |
| openstackgerrit | Merged openstack/tripleo-common master: Fix Plan creation from Git https://review.openstack.org/588172 | 01:51 |
| *** lblanchard has quit IRC | 01:55 | |
| openstackgerrit | Merged openstack/tripleo-quickstart master: fix if statement for standalone https://review.openstack.org/591678 | 01:56 |
| openstackgerrit | Merged openstack/tripleo-common stable/queens: Revert "Include microversion in OS_VOLUME_API_VERSION" https://review.openstack.org/591674 | 01:56 |
| *** srini_ has quit IRC | 01:56 | |
| *** srini_ has joined #tripleo | 01:56 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Add browbeat-minimal.yml to ovb playbook list https://review.openstack.org/590090 | 02:01 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Fix small nit in enable_validations parameter. https://review.openstack.org/591680 | 02:01 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Add Browbeat env settings to rdocloud https://review.openstack.org/583576 | 02:04 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Add rocky featureset010 to the branchful jobs. https://review.openstack.org/590779 | 02:04 |
| *** morazi has joined #tripleo | 02:06 | |
| *** ooolpbot has joined #tripleo | 02:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1786764 | 02:10 |
| *** ooolpbot has quit IRC | 02:10 | |
| openstack | Launchpad bug 1786764 in tripleo "tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates times out on prepare" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 02:10 |
| *** rfolco|rover is now known as rfolco | 02:12 | |
| *** srini_ has quit IRC | 02:17 | |
| *** srini_ has joined #tripleo | 02:18 | |
| *** openstack has joined #tripleo | 02:33 | |
| *** ChanServ sets mode: +o openstack | 02:33 | |
| *** ramishra has joined #tripleo | 02:37 | |
| *** khyr0n has joined #tripleo | 02:45 | |
| *** srini_ has quit IRC | 03:06 | |
| *** srini_ has joined #tripleo | 03:07 | |
| *** ooolpbot has joined #tripleo | 03:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1786764 | 03:10 |
| *** ooolpbot has quit IRC | 03:10 | |
| openstack | Launchpad bug 1786764 in tripleo "tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates times out on prepare" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 03:10 |
| *** gbarros has quit IRC | 03:18 | |
| *** srini_ has quit IRC | 03:24 | |
| *** srini_ has joined #tripleo | 03:25 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Revert "Switch containerized undercloud upgrade job to nv" https://review.openstack.org/589524 | 03:26 |
| *** srini_ has quit IRC | 03:33 | |
| *** srini_ has joined #tripleo | 03:33 | |
| *** zaneb has quit IRC | 03:35 | |
| *** ooolpbot has joined #tripleo | 04:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1786764 | 04:10 |
| *** ooolpbot has quit IRC | 04:10 | |
| openstack | Launchpad bug 1786764 in tripleo "tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates times out on prepare" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 04:10 |
| *** slaweq_ has joined #tripleo | 04:32 | |
| *** morazi has quit IRC | 04:34 | |
| *** srini_ has quit IRC | 04:35 | |
| *** srini_ has joined #tripleo | 04:36 | |
| *** slaweq_ has quit IRC | 04:36 | |
| honza | EmilienM: I don't ever install/set up the undercloud manually anymore... I can try it if you have some steps handy | 04:41 |
| *** ooolpbot has joined #tripleo | 05:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1786764 | 05:10 |
| *** ooolpbot has quit IRC | 05:10 | |
| openstack | Launchpad bug 1786764 in tripleo "tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates times out on prepare" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 05:10 |
| *** pcaruana has joined #tripleo | 05:12 | |
| *** srini_ has quit IRC | 05:13 | |
| *** sri_ has joined #tripleo | 05:15 | |
| *** maufart__ has joined #tripleo | 05:26 | |
| *** kopecmartin has joined #tripleo | 05:27 | |
| *** sri_ has quit IRC | 05:29 | |
| *** sri_ has joined #tripleo | 05:30 | |
| Tengu | hellothere | 05:33 |
| *** sri_ has quit IRC | 05:36 | |
| *** sri_ has joined #tripleo | 05:37 | |
| *** apetrich has joined #tripleo | 05:42 | |
| *** sri_ has quit IRC | 05:44 | |
| *** sri_ has joined #tripleo | 05:46 | |
| openstackgerrit | Sai Ram Peesapati proposed openstack/tripleo-heat-templates master: Make vlan_transparent in neutron.conf configurable from Undercloud https://review.openstack.org/591904 | 05:50 |
| *** waleedm has joined #tripleo | 05:58 | |
| *** sri_ has quit IRC | 06:09 | |
| *** ooolpbot has joined #tripleo | 06:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1786764 | 06:10 |
| *** ooolpbot has quit IRC | 06:10 | |
| openstack | Launchpad bug 1786764 in tripleo "tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates times out on prepare" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 06:10 |
| *** sri_ has joined #tripleo | 06:10 | |
| *** agurenko has joined #tripleo | 06:11 | |
| *** yprokule has joined #tripleo | 06:22 | |
| *** jaosorior has joined #tripleo | 06:25 | |
| *** dxiri has joined #tripleo | 06:30 | |
| jaosorior | bandini: could I get a review for this https://review.openstack.org/#/c/591681/ | 06:32 |
| *** jaosorior has quit IRC | 06:40 | |
| *** rdopiera has joined #tripleo | 06:50 | |
| openstackgerrit | Andriy Shevchenko proposed openstack/tripleo-repos master: fix tox python3 overrides https://review.openstack.org/591925 | 06:52 |
| *** paramite_ has joined #tripleo | 06:56 | |
| openstackgerrit | Yurii Prokulevych proposed openstack/tripleo-upgrade stable/queens: Use full path to workaround scripts. https://review.openstack.org/591930 | 07:06 |
| *** pliu has joined #tripleo | 07:09 | |
| *** ooolpbot has joined #tripleo | 07:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1786764 | 07:10 |
| *** ooolpbot has quit IRC | 07:10 | |
| openstack | Launchpad bug 1786764 in tripleo "tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates times out on prepare" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 07:10 |
| *** sri_ has quit IRC | 07:10 | |
| *** leanderthal has joined #tripleo | 07:12 | |
| *** sri_ has joined #tripleo | 07:13 | |
| *** khyr0n has quit IRC | 07:15 | |
| *** jaosorior has joined #tripleo | 07:15 | |
| *** noama has joined #tripleo | 07:24 | |
| noama | im trying to run openstack undercloud install | 07:24 |
| *** florianf has joined #tripleo | 07:24 | |
| noama | and getting this "ValueError: u'unreachable' does not appear to be an IPv4 or IPv6 network" on TASK [Output for UndercloudAllNodesValidationDeployment] | 07:25 |
| jaosorior | did you set up an IP with the netmask in local_ip ? | 07:26 |
| noama | no set | 07:27 |
| noama | im just setting the local_interface | 07:27 |
| jaosorior | noama: are you following the install guide? http://tripleo.org/install/index.html | 07:28 |
| *** sri_ has quit IRC | 07:28 | |
| *** sri_ has joined #tripleo | 07:28 | |
| noama | using this https://docs.openstack.org/tripleo-docs/latest/install/installation/installation.html | 07:31 |
| noama | this diff seems to be on cmd 'openstack undercloud install' | 07:33 |
| noama | should i use --use-heat or not? | 07:33 |
| jaosorior | noama: what version are you using? | 07:33 |
| jaosorior | noama: if you're using the latest (rocky) you shouldn't need --use-heat | 07:34 |
| noama | master | 07:34 |
| noama | sudo -E tripleo-repos current-tripleo-dev | 07:34 |
| EmilienM | honza: well, the only step is "openstack undercloud install" | 07:37 |
| honza | EmilienM: I'll see if I can do something today... | 07:39 |
| noama | http://paste.openstack.org/show/728093/ | 07:39 |
| EmilienM | honza: I'm happy to do it with you in a tmate or something | 07:40 |
| EmilienM | honza: but really, 1) deploy tripleoclient on a fresh machine 2) cp /usr/share/python-tripleoclient/undercloud.conf.sample undercloud.conf 3) openstack undercloud install | 07:41 |
| honza | EmilienM: the biggest annoyance for me is the VM setup, after that it's fine :) | 07:41 |
| *** jpich has joined #tripleo | 07:41 | |
| jaosorior | EmilienM: any idea what could be causing? http://paste.openstack.org/show/728093/ | 07:42 |
| *** leanderthal has quit IRC | 07:42 | |
| EmilienM | jaosorior: looking | 07:43 |
| EmilienM | jaosorior: can you please paste /var/lib/heat-config/heat-config-script/346707af-4ecf-4f40-9a12-37015d286fb6 ? | 07:43 |
| EmilienM | the trace is incomplete | 07:43 |
| jaosorior | noama: ^^ | 07:43 |
| noama | http://paste.openstack.org/show/728094/ | 07:45 |
| *** sri_ has quit IRC | 07:46 | |
| EmilienM | networks=$(ip -6 r | grep -v default | cut -d " " -f 1 | grep -v "unreachable") | 07:48 |
| EmilienM | in your deployment, it returns unreachable | 07:48 |
| *** sri_ has joined #tripleo | 07:49 | |
| EmilienM | noama: can you paste your undercloud.conf? | 07:49 |
| noama | mmm i think i get it | 07:49 |
| noama | i run "ip r | grep -v default | cut -d " " -f 1" | 07:49 |
| noama | https://www.irccloud.com/pastebin/2pttwaKL/ | 07:49 |
| noama | ip r | grep -v default | 07:50 |
| noama | 10.209.86.0/24 dev eth0 proto kernel scope link src 10.209.86.29 | 07:50 |
| noama | unreachable 169.254.169.254 scope host | 07:50 |
| noama | 192.168.24.0/24 dev br-ctlplane proto kernel scope link src 192.168.24.1 | 07:50 |
| EmilienM | we should maybe move the grep -v "unreachable" right after the grep default | 07:51 |
| noama | http://paste.openstack.org/show/728094/ line 33 missing unreachable | 07:51 |
| jaosorior | uhm | 07:51 |
| jaosorior | noama: 169.254.169.254 ... are you deploying tripleo on top of an openstack host? | 07:52 |
| jaosorior | (or aws?) | 07:52 |
| noama | its simple kvm vm | 07:52 |
| jaosorior | I see | 07:52 |
| noama | should i delete this route? | 07:54 |
| jaosorior | i would do that, if you're not using it. That's the route to a metadata service. | 07:54 |
| jaosorior | from a virtual machine | 07:55 |
| noama | oh, is this coming from cloud-init? | 07:55 |
| jaosorior | could be | 07:55 |
| noama | sh** | 07:55 |
| jaosorior | what image are you using? | 07:55 |
| noama | centos generic | 07:55 |
| noama | 7.5 | 07:56 |
| *** ooolpbot has joined #tripleo | 08:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1786764 | 08:10 |
| *** ooolpbot has quit IRC | 08:10 | |
| openstack | Launchpad bug 1786764 in tripleo "tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates times out on prepare" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 08:10 |
| *** dxiri has quit IRC | 08:20 | |
| *** akrivoka has joined #tripleo | 08:25 | |
| *** dtantsur|afk is now known as dtantsur | 08:26 | |
| *** derekh has joined #tripleo | 08:31 | |
| *** lennyb has joined #tripleo | 08:38 | |
| *** maufart__ has quit IRC | 08:44 | |
| *** akrivoka has quit IRC | 08:45 | |
| *** akrivoka has joined #tripleo | 08:45 | |
| *** leanderthal has joined #tripleo | 08:48 | |
| openstackgerrit | Rabi Mishra proposed openstack/python-tripleoclient master: Re-use old fernet and credential keys in containerized undercloud https://review.openstack.org/591980 | 09:09 |
| *** ooolpbot has joined #tripleo | 09:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1786764 | 09:10 |
| openstack | Launchpad bug 1786764 in tripleo "tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates times out on prepare" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 09:10 |
| *** ooolpbot has quit IRC | 09:10 | |
| ramishra | jaosorior: ^^for the issue we were discussing yesterday, mostly a hack:/ dunno a better way | 09:10 |
| jaosorior | ramishra: will that work? thought that didn't run as root | 09:11 |
| ramishra | jaosorior: self._set_data_rights() changes the permission on /etc/keystone | 09:12 |
| jaosorior | ramishra: sure, but we run python-tripleoclient from the stack user, not as root AFAIK, so we wouldn't usually be allowed to change those permissions, would we? | 09:15 |
| *** bryan_att_ has joined #tripleo | 09:16 | |
| *** andreaf has quit IRC | 09:16 | |
| ramishra | jaosorior: https://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/v1/tripleo_deploy.py#L105 | 09:17 |
| *** andreaf has joined #tripleo | 09:17 | |
| ramishra | So you're saying stack user can't do that? | 09:17 |
| *** bryan_att has quit IRC | 09:18 | |
| *** bryan_att_ is now known as bryan_att | 09:18 | |
| jaosorior | ramishra: well, it seems it can (though it shouldn't) | 09:18 |
| jaosorior | Tengu: ^^ | 09:19 |
| jaosorior | ramishra: but yeah, I guess that works for now | 09:19 |
| ramishra | yeah, it's a complete hack, but would work | 09:19 |
| *** salmankhan has joined #tripleo | 09:22 | |
| EmilienM | ramishra, jaosorior : hey I was about to work on this bug today | 09:24 |
| EmilienM | I'll review the patch | 09:24 |
| EmilienM | I don't think we want to do that | 09:24 |
| EmilienM | but indeed your solution could work | 09:25 |
| ramishra | EmilienM: Feel free to update it, I could not think of a better solution | 09:25 |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart-extras master: Updated skip list for Tempest https://review.openstack.org/591982 | 09:26 |
| EmilienM | ramishra: thanks for working on it. I'll see how it works | 09:28 |
| ramishra | EmilienM: thanks! | 09:28 |
| jaosorior | EmilienM, ramishra: bogdando has a patch for dealing with the undercloud passwords https://review.openstack.org/#/c/587431/ which also adds the ability to distinguish if it's an upgrade. It would be good that, if we would do this operation, we would restrict it to only do it on upgrades | 09:30 |
| *** gkadam has joined #tripleo | 09:30 | |
| Tengu | jaosorior: hmm ? | 09:33 |
| Tengu | jaosorior: well, that's linked to the privileges escalation thingy ;) | 09:34 |
| Tengu | jaosorior: for the records, even if we run as stack the `openstack overcloud deploy' some actions are calling sudo from within the code directly, and in the undercloud case it calls "sudo openstack undercloud ....." as a first command, more or less ;). | 09:35 |
| *** sri_ has quit IRC | 09:38 | |
| *** waleedm has quit IRC | 09:46 | |
| *** aevoo has joined #tripleo | 09:54 | |
| *** aevoo has quit IRC | 09:55 | |
| *** aevoo has joined #tripleo | 09:55 | |
| *** aevoo has quit IRC | 09:56 | |
| *** aevoo has joined #tripleo | 09:56 | |
| *** ooolpbot has joined #tripleo | 10:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1786764 | 10:10 |
| *** ooolpbot has quit IRC | 10:10 | |
| openstack | Launchpad bug 1786764 in tripleo "tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates times out on prepare" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 10:10 |
| openstackgerrit | Carlos Goncalves proposed openstack/tripleo-common master: Download CentOS-based amphora image if not present https://review.openstack.org/591997 | 10:13 |
| *** agurenko has quit IRC | 10:14 | |
| *** agurenko has joined #tripleo | 10:16 | |
| openstackgerrit | Carlos Goncalves proposed openstack/tripleo-common master: Download CentOS-based amphora image if not present https://review.openstack.org/591997 | 10:33 |
| openstackgerrit | Carlos Goncalves proposed openstack/tripleo-quickstart master: featureset038: run Octavia tempest tests https://review.openstack.org/580119 | 10:39 |
| openstackgerrit | Carlos Goncalves proposed openstack/tripleo-common master: Gate on scenario010 https://review.openstack.org/587015 | 10:41 |
| *** thrash|g0ne is now known as thrash | 10:48 | |
| *** maufart__ has joined #tripleo | 11:04 | |
| *** ooolpbot has joined #tripleo | 11:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1786764 | 11:10 |
| *** ooolpbot has quit IRC | 11:10 | |
| openstack | Launchpad bug 1786764 in tripleo "tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates times out on prepare" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 11:10 |
| *** leanderthal has quit IRC | 11:10 | |
| *** rh-jelabarre has joined #tripleo | 11:14 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-docs master: RHSM: document how we manually scale-down (for now) https://review.openstack.org/592005 | 11:20 |
| openstackgerrit | Rabi Mishra proposed openstack/python-tripleoclient master: Re-use old fernet and credential keys in containerized undercloud https://review.openstack.org/591980 | 11:25 |
| *** panda|ruck is now known as panda|ruck|lunch | 11:25 | |
| *** lblanchard has joined #tripleo | 11:29 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Remove tripleo.sh --bootstrap-subnodes add ceph loop device https://review.openstack.org/583195 | 11:40 |
| *** dprince has joined #tripleo | 11:49 | |
| *** sri_ has joined #tripleo | 11:50 | |
| *** leanderthal has joined #tripleo | 11:52 | |
| *** panda|ruck|lunch is now known as panda|ruck | 11:53 | |
| *** sri_ has quit IRC | 11:54 | |
| *** sri_ has joined #tripleo | 11:54 | |
| *** raildo has joined #tripleo | 11:59 | |
| *** srini_ has joined #tripleo | 12:00 | |
| *** rfolco|rover has joined #tripleo | 12:00 | |
| *** toure|gone is now known as toure | 12:02 | |
| *** sri_ has quit IRC | 12:02 | |
| jaosorior | panda|ruck: hey, could you check these two patches out https://review.openstack.org/#/q/status:open+project:openstack/tripleo-quickstart+branch:master+topic:replace-pase-with-wait-for ? | 12:03 |
| EmilienM | weshay, panda|ruck : any blocker for you if we branch tripleoclient today? | 12:04 |
| panda|ruck | jaosorior: looking | 12:04 |
| *** lblanchard has quit IRC | 12:05 | |
| *** morazi has joined #tripleo | 12:05 | |
| panda|ruck | EmilienM: not sure,, let me check the status of the rocky pipeline, it's merged but don't know if it works correctly and we are not yet actively monitoring it | 12:06 |
| EmilienM | panda|ruck: ok, thx. Please let me know | 12:06 |
| *** abishop has joined #tripleo | 12:08 | |
| *** srini_ has quit IRC | 12:08 | |
| *** srini_ has joined #tripleo | 12:08 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-heat-templates master: Always enable image prepare service for docker clouds https://review.openstack.org/581919 | 12:09 |
| panda|ruck | EmilienM: containers build has been failing for the first 3 builds, so we still don't have any result from the featuresets. https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-centos-7-rocky-containers-build/06fade5/job-output.txt.gz | 12:09 |
| EmilienM | panda|ruck: what are the failures? | 12:10 |
| *** ooolpbot has joined #tripleo | 12:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1786764 | 12:10 |
| *** ooolpbot has quit IRC | 12:10 | |
| openstack | Launchpad bug 1786764 in tripleo "tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates times out on prepare" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 12:10 |
| EmilienM | dprince: hey, if you wanted to see where I am with podman so far, I've been playing with https://review.openstack.org/#/c/590587/ (see all dependencies) and I got: http://logs.openstack.org/87/590587/6/check/tripleo-ci-centos-7-undercloud-containers/447f8d3/logs/undercloud/home/zuul/undercloud_install.log.txt.gz#_2018-08-14_14_07_20 | 12:11 |
| EmilienM | dprince: I also have a local instance and trying to make it work but I wanted to make things visible in CI to share logs | 12:11 |
| dprince | EmilienM: hey, looking... | 12:11 |
| panda|ruck | jaosorior: I think you'll need to rebase 590300 | 12:14 |
| panda|ruck | jaosorior: it modifies a file that you rename on the other | 12:14 |
| panda|ruck | jaosorior: the other is approved now | 12:14 |
| jaosorior | panda|ruck: ok | 12:15 |
| jaosorior | panda|ruck: oh wait; so, those two patches are together | 12:16 |
| jaosorior | one on top of the other | 12:16 |
| dtantsur | is the CI not feeling well? https://review.openstack.org/#/c/585370/ | 12:16 |
| jaosorior | panda|ruck: so https://review.openstack.org/#/c/590306/1 won't merge until https://review.openstack.org/#/c/590300/1 does | 12:16 |
| panda|ruck | jaosorior: ok | 12:17 |
| *** boazel has joined #tripleo | 12:18 | |
| jaosorior | panda|ruck: thanks | 12:20 |
| jaosorior | EmilienM: could you check this out https://review.openstack.org/#/c/591681/ ? | 12:20 |
| *** boazel_ has quit IRC | 12:20 | |
| *** srini_ has quit IRC | 12:21 | |
| *** srini_ has joined #tripleo | 12:22 | |
| *** jcoufal has joined #tripleo | 12:25 | |
| *** trown|outtypewww is now known as trown | 12:26 | |
| jaosorior | Does anybody know how often do the docs in tripleo.org get updated? | 12:26 |
| *** dxiri has joined #tripleo | 12:28 | |
| EmilienM | jaosorior: looking | 12:31 |
| EmilienM | jaosorior: bnemec / dprince probably know that | 12:31 |
| *** rlandy has joined #tripleo | 12:31 | |
| dprince | jaosorior: should be every 15 minutes or so. Is it hung perhaps? | 12:32 |
| dprince | jaosorior: I bet it is... let me check | 12:32 |
| *** srini_ has quit IRC | 12:32 | |
| *** srini_ has joined #tripleo | 12:32 | |
| jaosorior | EmilienM: thanks | 12:33 |
| dprince | it is running now | 12:35 |
| dprince | give it a few minutes | 12:35 |
| jaosorior | dprince: thanks! | 12:37 |
| jaosorior | dprince: where does it run on? | 12:37 |
| *** ykarel has joined #tripleo | 12:37 | |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart master: Add a standalone role for creating the zuul-hosts file https://review.openstack.org/579154 | 12:37 |
| dprince | jaosorior: on an RDO cloud VM | 12:37 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart master: fix bug in standalone ci-script logic https://review.openstack.org/592016 | 12:38 |
| dprince | jaosorior: under my account. | 12:38 |
| beagles | has anyone heard of an issue with the rsync parts of docker-puppet error'ing out in containerized undercloud, breaking the undercloud install? | 12:40 |
| dprince | beagles: no, but sounds interesting | 12:41 |
| dprince | beagles: you should be able to re-run it manually and "tease out" the offending container | 12:41 |
| beagles | I had to hack around it (set +e) for the time being to work on another bug but likely be in a position to reproduce in the next day or so | 12:41 |
| beagles | dprince: it was several containers actually | 12:41 |
| dprince | beagles: are you using selinux? | 12:42 |
| beagles | dprince: I'll gather more details later and file a proper bug report - it's pretty weird though. Seems likely to be something environment specific | 12:42 |
| beagles | dprince: set to permissive | 12:43 |
| beagles | dprince: the only clue I have (that I can remember) is that rsync was complaining files had disappeared | 12:43 |
| dprince | beagles: sounds like a race of some sort | 12:44 |
| beagles | dprince: yeah | 12:44 |
| *** srini_ has quit IRC | 12:44 | |
| beagles | dprince: might be that the way I have my undercloud VM configured might be exposing it | 12:44 |
| *** srini_ has joined #tripleo | 12:45 | |
| dprince | jaosorior: tripleo.org, try now | 12:45 |
| jaosorior | dprince: seems it still doesn't have the latest docs | 12:46 |
| dprince | jaosorior: refresh it maybe :) | 12:47 |
| jaosorior | dprince: I did, and opened it in incognito mode too (and tried another browser) | 12:47 |
| *** ykarel is now known as ykarel|away | 12:49 | |
| dprince | jaosorior: you are checking for your TLS everywhere commit. I don';t see that either. | 12:51 |
| dprince | jaosorior: let me see whats up | 12:51 |
| jaosorior | dprince: correct :D | 12:51 |
| dprince | jaosorior: it was missing python3. I guess the recent change to add it broke this server | 12:52 |
| dprince | jaosorior: Centos 7 doesn't require it. I'll add it to the setup checks perhaps | 12:53 |
| dprince | jaosorior: okay, I installed python34 and python34-devel. I think its good now | 12:57 |
| jaosorior | lets check it out | 12:57 |
| jaosorior | dprince: thanks for looking into this | 12:57 |
| dprince | jaosorior: its probably been broken for quite a while. since we switched to python3 anyways | 12:57 |
| *** crandon has joined #tripleo | 12:59 | |
| *** agurenko has quit IRC | 13:00 | |
| dprince | jaosorior: "Please make sure that the aforementioned configuration options are set in the [DEFAULT] section of undercloud.conf" | 13:01 |
| dprince | jaosorior: boom -> http://tripleo.org/install/advanced_deployment/tls_everywhere.html | 13:02 |
| jaosorior | yeeeah | 13:02 |
| jaosorior | I see it too | 13:02 |
| jaosorior | dprince: thanks! | 13:02 |
| *** srini_ has quit IRC | 13:02 | |
| *** srini_ has joined #tripleo | 13:03 | |
| *** tzumainn has joined #tripleo | 13:03 | |
| crandon | Hi | 13:05 |
| crandon | I'm new to TripleO and quickstart. I'm trying to get quickstart to install overcloud and undercloud into a CentOS 7 VM (running on libvirt/kvm). quickstart is runnign on the VIRTHOST, VIRTHOST variable is set to the external IP of the VM and is accesible via SSH. All looks fine, but overcloud deploy fails in "Deploy the overcloud" step. | 13:07 |
| jaosorior | crandon: crandon you can ssh to the undercloud with: ssh -F ~/.quickstart/ssh.config.ansible undercloud | 13:09 |
| jaosorior | crandon: there, you can do: source stackrc && openstack stack failures list --long overcloud | 13:10 |
| jaosorior | that'll give you the actual error message | 13:10 |
| crandon | The logfile mentioned in the printout is not available on the VIRTHOST in the stak users home: /home/stack/overcloud-deploy.log | 13:10 |
| *** ooolpbot has joined #tripleo | 13:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1786764 | 13:10 |
| openstack | Launchpad bug 1786764 in tripleo "tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates times out on prepare" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 13:10 |
| *** ooolpbot has quit IRC | 13:10 | |
| crandon | jaosorior: Ah, so the undercloud uses the 'stack' user as well and the log is on the undercloud VM... thanks, I'll look into this. One last question though: I'd prefer to get only the undercloud installed and then install the overcloud 'by hand' (so basically via tripleo) on two other VMs (which were not nested unlike to undercloud). What is the straightforward way to do that? | 13:14 |
| jaosorior | how did you initially deploy? | 13:14 |
| jaosorior | crandon: the default tags should leave you in such a state | 13:14 |
| crandon | jaosorior: yes I know, that's not an issue to get the undercloud installed and stop there, but the question is, where can I find docs to install the overcloud without quickstart on 2 other VMs, which are not nested (so the end result will be: Physical server -> VIRTHOST VM -> undercloud VM (nested); Physical Server -> Controller VM; Physical Server -> Compute VM). The trick is, that the undercloud installation will be one layer deeper | 13:17 |
| crandon | than the overcloud nodes. | 13:17 |
| *** ykarel|away has quit IRC | 13:17 | |
| jaosorior | Tengu: ideas? ^^ | 13:18 |
| crandon | jaosorior: I checked the logs and the stack failures and it simply sais "No valid host was found" for both the Controller and Compute node. On the VIRTHOST I can only the undercloud VM running, but with virsh list --all I can see a shut off compute and controller VM (not sure though if they are from the current or any previous quickstart run...) | 13:23 |
| *** leanderthal has quit IRC | 13:23 | |
| *** bfournie has quit IRC | 13:24 | |
| crandon | I also tried to rerun only the failed overcloud deployment, but that failed too: | 13:27 |
| crandon | bash quickstart.sh --teardown none --no-clone -I -p quickstart-extras-overcloud.yml -R queens --tags all $VIRTHOST | 13:27 |
| crandon | ..... | 13:27 |
| crandon | quickstart.sh: line 523: /root/ansible_ssh_env.sh: No such file or directory | 13:27 |
| *** mjturek has joined #tripleo | 13:30 | |
| *** boazel has quit IRC | 13:32 | |
| *** agopi has quit IRC | 13:34 | |
| crandon | jaosorior: The quickstart rerun issue seems to do something with the --no-clone option.... | 13:34 |
| jaosorior | crandon: if you wanna start from scratch you can do use --teardown all | 13:38 |
| crandon | jaosorior: I was willing to continue from oevercloud deploy to see on the run if the overcloud VMs come up at all... | 13:39 |
| crandon | jaosorior: removing --no-clone seem to allow that, however the overcloud VMs still wouldn't start, so I guess it will fail again. | 13:40 |
| *** leanderthal has joined #tripleo | 13:40 | |
| *** gbarros has joined #tripleo | 13:43 | |
| *** vinaykns has joined #tripleo | 13:46 | |
| *** srini_ has quit IRC | 13:50 | |
| *** srini_ has joined #tripleo | 13:50 | |
| *** bfournie has joined #tripleo | 13:55 | |
| *** srini_ has quit IRC | 13:56 | |
| *** florianf has quit IRC | 13:57 | |
| *** srini_ has joined #tripleo | 13:57 | |
| *** rfolco|rover has quit IRC | 13:57 | |
| *** srini_ has quit IRC | 13:57 | |
| mwhahaha | panda|ruck, weshay: where are we at with the scenario000-updates timing out? Do we need to swich it to non-voting? | 14:01 |
| *** mwhahaha changes topic to "Welcome to Rocky | CI Status: ORANGEish - updates job is timingout, OVB is down | https://docs.openstack.org/tripleo-docs/latest/" | 14:02 | |
| * mwhahaha throws things at EmilienM | 14:05 | |
| mwhahaha | EmilienM: https://review.openstack.org/#/c/591273/ master patches hadn't landed | 14:05 |
| mwhahaha | jaosorior: https://review.openstack.org/#/c/589414/ but the queens stuff has landed so we need to either unblock this or revert the queens bit | 14:06 |
| jaosorior | mwhahaha: yeah, TLS everywhere is broken on rocky. the patch was tested in queens, so that works just fine. But I didn't want to introduce extra complexity in getting TLS everywhere working on rocky... waiting for promotion in order to keep testing. | 14:07 |
| mwhahaha | we shouldn't have landed the queens bits then | 14:08 |
| *** agopi has joined #tripleo | 14:09 | |
| *** florianf has joined #tripleo | 14:09 | |
| *** boazel has joined #tripleo | 14:09 | |
| *** ooolpbot has joined #tripleo | 14:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1786764 | 14:10 |
| *** ooolpbot has quit IRC | 14:10 | |
| openstack | Launchpad bug 1786764 in tripleo "tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates times out on prepare" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 14:10 |
| panda|ruck | mwhahaha: jistr suggested we wait for a promotion to get containers in sync. | 14:10 |
| mwhahaha | panda|ruck: we can't merge anything with this job, so i don't want to wait on a promotion unless we are certain the next promotion will have the fix | 14:11 |
| mwhahaha | if we can't guarantee promotion will fix it, it needs to be switched to non-voting ASAP | 14:11 |
| jistr | mwhahaha, panda|ruck: it seems the containers were built with RPMs that don't follow depends-on order of patches that landed, which caused the breakage. I'd think that can't happen but apparently it can. I hit the issue locally too, i redeployed the environment with setting OOOQ variable `update_containers: true`, and i no longer hit the issue. | 14:13 |
| *** zaneb has joined #tripleo | 14:13 | |
| jistr | so that indeed suggests that all we need is up-to-date RPMs in the containers | 14:14 |
| mwhahaha | k so that means a promotion should fix it, but where are we at with the promotion | 14:14 |
| owalsh | mwhahaha: https://review.openstack.org/#/c/589414/5/manifests/haproxy.pp@1089 this shouldn't have passed CI AFAICT, failing downstream | 14:14 |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart-extras master: Updated skip list for Tempest https://review.openstack.org/591982 | 14:14 |
| owalsh | panda|ruck: ^^^ any ideas? | 14:15 |
| mwhahaha | owalsh: so tls is broken for rocky, does it pass queens? | 14:15 |
| owalsh | mwhahaha: yea, it merged | 14:15 |
| *** dxiri has quit IRC | 14:15 | |
| *** dxiri has joined #tripleo | 14:16 | |
| mwhahaha | owalsh: ok well file a bug and let's revert it | 14:16 |
| owalsh | mwhahaha: bug for the false positive from CI? | 14:17 |
| weshay | mwhahaha, I'll take a closer look at that job today | 14:17 |
| mwhahaha | owalsh: i don't know if it's a false positive because we might not be testing tls upstream in a similar fashion | 14:17 |
| mwhahaha | owalsh: i meant for the queens patches. If it's not working then let's revert it | 14:18 |
| owalsh | mwhahaha: only fails when internal TLS is disabled | 14:18 |
| mwhahaha | owalsh: right we might not have that tested in CI (i don't know if we have capacity to add this to a scenario) | 14:18 |
| owalsh | mwhahaha: so we have TLS everywhere enabled on all jobs? | 14:18 |
| mwhahaha | owalsh: not in all jobs, i think only one a few | 14:19 |
| panda|ruck | mwhahaha: jistr latest promotion on master was today, so if there's no change in the gate job, it's better we put it as non-voting, and take additional time to analyze. | 14:20 |
| jistr | panda|ruck, mwhahaha: sure, i'm fine with whatever helps us move forward | 14:21 |
| *** leanderthal has quit IRC | 14:21 | |
| owalsh | mwhahaha: ah, you mean we don't test with novncproxy enabled... that might explain it | 14:22 |
| agonzalez | cgoncalves, about #1609064 there records are deleted automatically when the failover fails | 14:24 |
| *** aevoo has quit IRC | 14:25 | |
| jistr | panda|ruck, mwhahaha: i see the job green on latest patches, e.g. here in gate https://review.openstack.org/#/c/589524/2 and here in check https://review.openstack.org/#/c/591982/1 | 14:25 |
| mwhahaha | it timedout this morning | 14:26 |
| mwhahaha | http://zuul.openstack.org/builds.html?job_name=tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates | 14:26 |
| mwhahaha | latest being at 2018-08-15T08:29:19 | 14:27 |
| mwhahaha | so all THT jobs are timing out | 14:27 |
| mwhahaha | or failing | 14:27 |
| jistr | mwhahaha: hmm weird... i wonder if it could be caused by stale mirrors on some parts of infra cloud | 14:27 |
| mwhahaha | no | 14:27 |
| *** mjturek has quit IRC | 14:27 | |
| mwhahaha | it's likely that there's somethign in THT that broke it | 14:28 |
| weshay | myoung, have we picked a new ruck/rover? | 14:28 |
| *** gbarros has quit IRC | 14:28 | |
| weshay | they need to be reading ^ when they are selected | 14:28 |
| mwhahaha | i think we need to switch it to non-voting | 14:28 |
| mwhahaha | and figure out why it's failign for THT | 14:28 |
| weshay | ok | 14:28 |
| weshay | mwhahaha, I'll put up a change | 14:28 |
| jistr | mwhahaha, panda|ruck: oh i see, it's only failing for tht. Yea let's switch to nv while we figure it out then. | 14:29 |
| openstackgerrit | wes hayutin proposed openstack-infra/tripleo-ci master: update tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates to non-voting https://review.openstack.org/592061 | 14:30 |
| weshay | mwhahaha, ^ myoung for new ruck/rover | 14:30 |
| mwhahaha | do we have a bug for this as well? | 14:30 |
| weshay | leading a mtg.. will add it in a bit | 14:31 |
| weshay | mwhahaha, adding a bug now | 14:33 |
| honza | EmilienM: so, I've been thinking about what you said: if I log into a fresh vm, and install the undercloud manually, then the issue at hand can't possibly be replicated | 14:36 |
| honza | EmilienM: oooq runs from the outside, and should know the public hostname and apply it to the config | 14:37 |
| honza | EmilienM: the bug must be in oooq, and not in the 'openstack undercloud install' | 14:39 |
| *** leanderthal has joined #tripleo | 14:39 | |
| openstackgerrit | Harald Jensås proposed openstack/tripleo-heat-templates master: Fix typo in network_data files https://review.openstack.org/592067 | 14:48 |
| *** jcoufal has quit IRC | 14:49 | |
| openstackgerrit | Oliver Walsh proposed openstack/puppet-tripleo stable/queens: Revert "SSL support for haproxy -> novnc proxy connection" https://review.openstack.org/592072 | 14:54 |
| *** lblanchard has joined #tripleo | 14:55 | |
| openstackgerrit | Oliver Walsh proposed openstack/tripleo-heat-templates stable/queens: Revert "SSL support for haproxy -> novnc proxy connection" https://review.openstack.org/592073 | 14:56 |
| *** d0ugal has joined #tripleo | 14:58 | |
| d0ugal | Workflow Squad meeting in a few mins. https://etherpad.openstack.org/p/tripleo-workflows-squad-status /cc rbrady,apetrich,thrash,toure,jtomasek | 14:58 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: WIP - Podman support for docker-puppet https://review.openstack.org/588655 | 15:00 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: DNM - test podman https://review.openstack.org/590587 | 15:00 |
| EmilienM | dprince: pushed ^ | 15:01 |
| *** betherly-afk is now known as betherly_ | 15:02 | |
| dprince | EmilienM: cool. did you get your podman from https://cbs.centos.org/repos/virt7-container-common-candidate/x86_64/os/Packages/ | 15:02 |
| EmilienM | dprince: not yet but I'm working on a patch in oooq to do that | 15:03 |
| *** ramishra has quit IRC | 15:03 | |
| dprince | EmilienM: cool. yeah according to steve that will get us a more recent podman | 15:03 |
| EmilienM | ++ excellent | 15:03 |
| openstackgerrit | Oliver Walsh proposed openstack/puppet-tripleo master: SSL support for haproxy -> novnc proxy connection https://review.openstack.org/589414 | 15:06 |
| *** maufart__ has quit IRC | 15:06 | |
| *** aufi has joined #tripleo | 15:06 | |
| *** rpioso|afk is now known as rpioso | 15:08 | |
| cgoncalves | agonzalez, the records are not deleted. the status of the amphorae are DELETED, which is different ;) | 15:09 |
| cgoncalves | agonzalez, do you happen to have an env where I could check it? | 15:09 |
| *** ooolpbot has joined #tripleo | 15:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1786764 | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1787226 | 15:10 |
| *** ooolpbot has quit IRC | 15:10 | |
| openstack | Launchpad bug 1786764 in tripleo "tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates times out on prepare" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 15:10 |
| openstack | Launchpad bug 1787226 in tripleo "tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates is timing out" [Critical,Triaged] | 15:10 |
| *** ade_lee has joined #tripleo | 15:10 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: DNM: Enable virt7-container-common-candidate https://review.openstack.org/592079 | 15:14 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: DNM - test podman https://review.openstack.org/590587 | 15:15 |
| openstackgerrit | Oliver Walsh proposed openstack/tripleo-heat-templates master: DNM: get some CI on https://review.openstack.org/589434 https://review.openstack.org/592080 | 15:16 |
| *** dxiri has quit IRC | 15:19 | |
| agonzalez | cgoncalves, yes, correct | 15:19 |
| agonzalez | cgoncalves, yes, moment | 15:19 |
| *** dsneddon has quit IRC | 15:20 | |
| cgoncalves | agonzalez, I'll not have time today to have a look at it, though, so no worries | 15:22 |
| agonzalez | I will send you a mail then tomorrow | 15:22 |
| cgoncalves | thanks! | 15:22 |
| openstackgerrit | Merged openstack/tripleo-docs master: RHSM: document how we manually scale-down (for now) https://review.openstack.org/592005 | 15:24 |
| *** mjturek has joined #tripleo | 15:27 | |
| *** derekh has quit IRC | 15:30 | |
| *** gbarros has joined #tripleo | 15:37 | |
| *** dtrainor has quit IRC | 15:40 | |
| *** dtrainor has joined #tripleo | 15:40 | |
| *** aufi has quit IRC | 15:42 | |
| *** leanderthal has quit IRC | 15:48 | |
| *** yprokule has quit IRC | 15:49 | |
| *** jcoufal has joined #tripleo | 15:52 | |
| openstackgerrit | Carlos Goncalves proposed openstack/tripleo-common master: Fix skip of octavia-undercloud Ansible role https://review.openstack.org/591413 | 15:57 |
| openstackgerrit | Carlos Goncalves proposed openstack/tripleo-common master: Fix skip of octavia-undercloud Ansible role https://review.openstack.org/591413 | 16:00 |
| openstackgerrit | Carlos Goncalves proposed openstack/tripleo-common master: Download CentOS-based amphora image if not present https://review.openstack.org/591997 | 16:01 |
| *** lblanchard has quit IRC | 16:02 | |
| *** vinaykns has quit IRC | 16:06 | |
| *** paramite_ has quit IRC | 16:07 | |
| *** jcoufal has quit IRC | 16:07 | |
| openstackgerrit | Amir Mohamad proposed openstack/tripleo-heat-templates master: This file intorduces OVN as a controller. I've cloned the neutron-sfc.yaml file and added to the parameter_defaults NeutronSfcDriver:'ovn' There could be other param defaults needs to be added and i'm not aware of. https://review.openstack.org/587491 | 16:08 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: DNM: use extra vars for jobs https://review.openstack.org/592100 | 16:08 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: DNM: use extra vars for jobs https://review.openstack.org/592100 | 16:09 |
| *** ooolpbot has joined #tripleo | 16:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1786764 | 16:10 |
| openstack | Launchpad bug 1786764 in tripleo "tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates times out on prepare" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1787226 | 16:10 |
| *** ooolpbot has quit IRC | 16:10 | |
| openstack | Launchpad bug 1787226 in tripleo "tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates is timing out" [Critical,Triaged] | 16:10 |
| openstackgerrit | Oliver Walsh proposed openstack/tripleo-heat-templates stable/queens: Revert "SSL support for haproxy -> novnc proxy connection" https://review.openstack.org/592073 | 16:11 |
| *** rbrady is now known as rbrady-afk | 16:12 | |
| openstackgerrit | Oliver Walsh proposed openstack/puppet-tripleo stable/queens: Revert "SSL support for haproxy -> novnc proxy connection" https://review.openstack.org/592072 | 16:13 |
| *** itlinux has joined #tripleo | 16:14 | |
| mwhahaha | panda|ruck: weshay: heads up new rdoinfo package appears to have hosed ci, i pinged in #rdo http://logs.openstack.org/81/591681/2/gate/tripleo-ci-centos-7-undercloud-oooq/658cd07/job-output.txt.gz#_2018-08-15_15_25_19_090967 | 16:21 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: DNM: use extra vars for jobs https://review.openstack.org/592100 | 16:21 |
| mwhahaha | we not rdoinfo, rdopkg | 16:21 |
| weshay | fun | 16:21 |
| *** panda|ruck is now known as panda|off | 16:22 | |
| *** pradk has joined #tripleo | 16:22 | |
| openstackgerrit | Carlos Goncalves proposed openstack/tripleo-common master: Download CentOS-based amphora image if not present https://review.openstack.org/591997 | 16:22 |
| *** mwhahaha changes topic to "Welcome to Rocky | CI Status: RED - rdopkg has broken everything, updates job is timingout, OVB is down | https://docs.openstack.org/tripleo-docs/latest/" | 16:23 | |
| weshay | rlandy, /me looking at https://github.com/openstack-infra/tripleo-ci/blob/master/playbooks/tripleo-ci/vars/environment_type/ovb.yaml | 16:23 |
| weshay | mwhahaha, rlandy ok.. so we're just pinning back to the older working version? | 16:24 |
| mwhahaha | yea i've got a patch coming shortly | 16:24 |
| rlandy | ok | 16:24 |
| *** jpich has quit IRC | 16:25 | |
| mwhahaha | weshay, rlandy https://bugs.launchpad.net/tripleo/+bug/1787244 | 16:25 |
| openstack | Launchpad bug 1787244 in tripleo "rdopkg release has broken all ci jobs" [Critical,Triaged] | 16:25 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-quickstart-extras master: Pin rdopkg to 0.46.3 https://review.openstack.org/592146 | 16:26 |
| mwhahaha | weshay: rlandy -^ | 16:26 |
| weshay | mwhahaha, rlandy ok.. so we just need someone to post https://goo.gl/images/mYPiL6 | 16:26 |
| weshay | and we're done here | 16:27 |
| *** eggmaster has joined #tripleo | 16:27 | |
| weshay | mwhahaha, rlandy /me looks at what tests jruzica has defined for rdopkg | 16:27 |
| weshay | maybe we can get a tripleo job in there | 16:28 |
| *** rf0lc0 has joined #tripleo | 16:28 | |
| *** dtrainor_ has joined #tripleo | 16:28 | |
| * weshay gets coffee | 16:28 | |
| mwhahaha | i guess i should just purge the gate | 16:29 |
| mwhahaha | oh it's not that long | 16:29 |
| mwhahaha | i'll just let them fail | 16:29 |
| *** dtrainor has quit IRC | 16:30 | |
| rlandy | weshay: ack - would be useful to put a tripleo job closer to where those changes happen | 16:32 |
| mwhahaha | well it should be using the rpm | 16:33 |
| mwhahaha | the problem is we're using pip in quickstart | 16:33 |
| mwhahaha | so i think the better thing is to stop using pip in quickstart and leverage rdopkg from a tested repo | 16:33 |
| *** paramite_ has joined #tripleo | 16:34 | |
| *** jcoufal has joined #tripleo | 16:37 | |
| *** lblanchard has joined #tripleo | 16:37 | |
| weshay | mwhahaha, aye.. still I'd like to see some requirement on rdoinfra projects have some representative tripleo job | 16:38 |
| *** kopecmartin has quit IRC | 16:38 | |
| mwhahaha | right but we need to get to a point where we can share/test the output | 16:38 |
| mwhahaha | the agreed testing format would likely come via an rpm | 16:39 |
| *** florianf has quit IRC | 16:40 | |
| weshay | mwhahaha, how does one build changes to rdopkg ;))) | 16:40 |
| mwhahaha | very carefully | 16:40 |
| * weshay looks at a fix to move to the rpm | 16:40 | |
| mwhahaha | because evidently it can break eeeevvvrrrrryyyyythhhhiiiiinnnnngg | 16:40 |
| weshay | fkrs | 16:41 |
| *** paramite_ has quit IRC | 16:41 | |
| openstackgerrit | Olivier Bourdon proposed openstack/diskimage-builder master: [DNM] Testing CentOS images builds on Ubuntu Xenial https://review.openstack.org/591366 | 16:42 |
| *** noama has quit IRC | 16:50 | |
| *** openstackstatus has joined #tripleo | 16:59 | |
| *** ChanServ sets mode: +v openstackstatus | 16:59 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-heat-templates master: DNM, testing dlrn https://review.openstack.org/584452 | 17:00 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-heat-templates master: DNM, testing dlrn https://review.openstack.org/584452 | 17:00 |
| openstackgerrit | Olivier Bourdon proposed openstack/diskimage-builder master: [DNM] Testing CentOS images builds on Ubuntu Xenial https://review.openstack.org/591366 | 17:04 |
| *** gkadam has quit IRC | 17:05 | |
| *** lblanchard has quit IRC | 17:05 | |
| *** ooolpbot has joined #tripleo | 17:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1786764 | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1787226 | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1787244 | 17:10 |
| *** ooolpbot has quit IRC | 17:10 | |
| openstack | Launchpad bug 1786764 in tripleo "tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates times out on prepare" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 17:10 |
| openstack | Launchpad bug 1787226 in tripleo "tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates is timing out" [Critical,Triaged] | 17:10 |
| openstack | Launchpad bug 1787244 in tripleo "rdopkg release has broken all ci jobs" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) | 17:10 |
| *** salmankhan has quit IRC | 17:13 | |
| rlandy | https://logs.rdoproject.org/46/592146/1/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/14c34c4/job-output.txt.gz#_2018-08-15_16_49_42_863226 | 17:13 |
| rlandy | ovb jobs failing on https://review.openstack.org/#/c/592146/ | 17:14 |
| rlandy | rdocloud jobs rather | 17:14 |
| weshay | rlandy, ya.. I think he updated the wrong thing | 17:18 |
| weshay | rlandy, let's fix this | 17:18 |
| weshay | he's busy | 17:18 |
| weshay | Collecting dlrn== | 17:19 |
| weshay | Could not find a version that satisfies the requirement dlrn== (from versions: 0.1.0, 0.2.0, 0.3.0, 0.4.0, 0.5.0, 0.5.1, 0.6.0, 0.7.0, 0.8.0) | 17:19 |
| weshay | No matching distribution found for dlrn== | 17:19 |
| rlandy | weshay: ack | 17:19 |
| weshay | whayutin•RDO-CI/TRIPLEO-QUICKSTART/tripleo-quickstart-extras ᐅ pip install pip== [standalone] master✔ thinkdoe ⌚ 12:57:17 | 17:19 |
| weshay | Collecting pip== | 17:19 |
| weshay | Could not find a version that satisfies the requirement pip== (from versions: 0.2, 0.2.1, 0.3, 0.3.1, 0.4, 0.5, 0.5.1, 0.6, 0.6.1, 0.6.2, 0.6.3, 0.7, 0.7.1, 0.7.2, 0.8, 0.8.1, 0.8.2, 0.8.3, 1.0, 1.0.1, 1.0.2, 1.1, 1.2, 1.2.1, 1.3, 1.3.1, 1.4, 1.4.1, 1.5, 1.5.1, 1.5.2, 1.5.3, 1.5.4, 1.5.5, 1.5.6, 6.0, 6.0.1, 6.0.2, 6.0.3, 6.0.4, 6.0.5, 6.0.6, 6.0.7, 6.0.8, 6.1.0, 6.1.1, 7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.1.0, 7.1.1, 7.1.2, 8.0.0, 8.0.1, 8.0.2, | 17:19 |
| weshay | 8.0.3, 8.1.0, 8.1.1, 8.1.2, 9.0.0, 9.0.1, 9.0.2, 9.0.3, 10.0.0b1, 10.0.0b2, 10.0.0, 10.0.1, 18.0) | 17:19 |
| rlandy | looking for pip version | 17:19 |
| weshay | whayutin•RDO-CI/TRIPLEO-QUICKSTART/ci-config ᐅ pip install rdopkg== [standalone] master✔ thinkdoe ⌚ 13:19:12 | 17:19 |
| weshay | Collecting rdopkg== | 17:19 |
| weshay | Could not find a version that satisfies the requirement rdopkg== (from versions: 0.23, 0.23.1, 0.24, 0.25, 0.26, 0.27.1, 0.28, 0.28.1, 0.29, 0.29.1, 0.30, 0.31, 0.32, 0.32.1, 0.33, 0.34, 0.35, 0.35.1, 0.36, 0.37, 0.38, 0.39, 0.40, 0.41, 0.41.1, 0.41.2, 0.41.3, 0.41.4, 0.42, 0.43, 0.44.0, 0.44.1, 0.44.2, 0.45.0, 0.46.0, 0.46.1, 0.46.2, 0.46.3, 0.47.0) | 17:19 |
| weshay | rlandy, ^ | 17:19 |
| weshay | rlandy, so on line 59 | 17:20 |
| weshay | we need to march back dlrn to 7.0 | 17:20 |
| weshay | and rdopkg to 46.3 which needs a new pip call | 17:20 |
| *** gkadam has joined #tripleo | 17:20 | |
| weshay | roger that? | 17:21 |
| weshay | tmate? | 17:21 |
| rlandy | yep - should we try out locally? | 17:21 |
| weshay | meh | 17:21 |
| weshay | rlandy, test patch https://review.openstack.org/#/c/584452/ | 17:21 |
| rlandy | fine - queues are not busy | 17:22 |
| weshay | lolz | 17:22 |
| weshay | at all | 17:22 |
| weshay | lol.. YOU ARE ALL SCREWED | 17:22 |
| rlandy | ok - let's tmate | 17:22 |
| weshay | k | 17:22 |
| * mwhahaha sighs | 17:27 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras master: Pin rdopkg to 0.46.3 https://review.openstack.org/592146 | 17:28 |
| *** gbarros has quit IRC | 17:38 | |
| *** dtantsur is now known as dtantsur|afk | 17:40 | |
| mnaser | mwhahaha: please let me know if you can setup a cross-deployment tools thing at the ptg | 17:42 |
| mnaser | esp. around tempest for starters, i see a lot of stuff we can share in terms of blacklisting failing jobs, etc | 17:43 |
| mwhahaha | mnaser: I'll see if I can get something together | 17:43 |
| mnaser | mwhahaha: thank you | 17:44 |
| rlandy | looking better ... | 17:48 |
| rlandy | 2018-08-15 17:45:17.974497 | primary | TASK [build-test-packages : Pip install rdopkg] ******************************** | 17:48 |
| rlandy | 2018-08-15 17:45:17.986947 | primary | Wednesday 15 August 2018 17:45:17 +0000 (0:00:03.978) 0:05:28.155 ****** | 17:48 |
| rlandy | 2018-08-15 17:45:24.499190 | primary | changed: [undercloud] | 17:48 |
| *** pradk has quit IRC | 17:50 | |
| weshay | rlandy, ya.. seems to be working | 17:53 |
| mwhahaha | mnaser: do you know how is managing the ptg schedule? i never actually got any correspondence around it | 17:54 |
| *** trown is now known as trown|lunch | 17:54 | |
| mnaser | i'm assuming kendall (diablo_rojo) is doing most of it (assuming assisted by ttx) | 17:55 |
| *** dsneddon has joined #tripleo | 18:02 | |
| *** akrivoka has quit IRC | 18:04 | |
| *** sshnaidm is now known as sshnaidm|bbl | 18:05 | |
| *** slaweq_ has joined #tripleo | 18:07 | |
| *** ooolpbot has joined #tripleo | 18:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1786764 | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1787226 | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1787244 | 18:10 |
| *** ooolpbot has quit IRC | 18:10 | |
| openstack | Launchpad bug 1786764 in tripleo "tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates times out on prepare" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 18:10 |
| openstack | Launchpad bug 1787226 in tripleo "tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates is timing out" [Critical,Triaged] | 18:10 |
| openstack | Launchpad bug 1787244 in tripleo "rdopkg release has broken all ci jobs" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 18:10 |
| *** ykarel has joined #tripleo | 18:11 | |
| openstackgerrit | Merged openstack/tripleo-quickstart master: fix bug in standalone ci-script logic https://review.openstack.org/592016 | 18:11 |
| *** slaweq_ has quit IRC | 18:15 | |
| *** gkadam has quit IRC | 18:19 | |
| *** mjturek has quit IRC | 18:22 | |
| *** rbrady-afk is now known as rbrady | 18:26 | |
| *** boazel has quit IRC | 18:30 | |
| *** gbarros has joined #tripleo | 18:31 | |
| tonyb | My overcloud deploy failed and openstack stack failures list --long overcloud says: http://paste.openstack.org/show/728133/ | 18:34 |
| tonyb | I'm used to there being a whole lot more verbosity there .... any pointers as to how I can delve into what went wrong? | 18:35 |
| *** boazel has joined #tripleo | 18:35 | |
| *** crandon has quit IRC | 18:35 | |
| *** apetrich has quit IRC | 18:35 | |
| *** salmankhan has joined #tripleo | 18:41 | |
| *** gbarros has quit IRC | 18:43 | |
| *** salmankhan has quit IRC | 18:46 | |
| *** gbarros has joined #tripleo | 18:49 | |
| fultonj | hi tonyb | 18:57 |
| tonyb | fultonj: hi there! | 18:57 |
| fultonj | do you have this file on your UC /var/log/mistral/ceph-install-workflow.log | 18:57 |
| fultonj | ? | 18:57 |
| *** gbarros has quit IRC | 18:58 | |
| tonyb | fultonj: I do not | 18:58 |
| *** hgogada has joined #tripleo | 19:00 | |
| *** hgogada has quit IRC | 19:00 | |
| fultonj | tonyb: does your undercloud.conf have generate_service_certificate = false or true ? | 19:01 |
| fultonj | if it's true are you using inject-trust-anchor.yaml ? | 19:01 |
| tonyb | I didn't set it and the config fil implies flase is the defult | 19:02 |
| fultonj | ahhh but true is the default | 19:02 |
| openstackgerrit | Merged openstack/tripleo-quickstart master: Move scenario009 to the check queue https://review.openstack.org/589157 | 19:03 |
| fultonj | unless you set it to false, then i believe it's true | 19:03 |
| tonyb | fultonj: Oh sorry thise is queens'ish | 19:03 |
| fultonj | is the case for queens too | 19:03 |
| fultonj | was false by dfault in pike | 19:03 |
| fultonj | IIRC | 19:03 |
| fultonj | so..... | 19:03 |
| fultonj | now that we know it's true.... are you using inject-trust-anchor.yaml as per https://docs.openstack.org/tripleo-docs/latest/install/advanced_deployment/ssl.html ? | 19:04 |
| fultonj | also, you're not using spine/leaf are you? | 19:04 |
| *** trown|lunch is now known as trown | 19:05 | |
| fultonj | tonyb: if you don't need a secure connection between your UC and OC, the easiest thing to do is to set generate cert false in your uc.conf | 19:05 |
| fultonj | and not use trust anchor | 19:05 |
| tonyb | I'm not using inject-trust-anchor.yaml, but I wonder if it wouldn't be easier to rebuild the UC with generate_service_certificate=false | 19:06 |
| fultonj | tonyb: that's it exactly | 19:06 |
| tonyb | fultonj: I don;t even know what spine/leaf really is so I doubt it | 19:06 |
| tonyb | fultonj: it's a very small test/dev setup | 19:06 |
| fultonj | tonyb: i agree | 19:06 |
| fultonj | rebuld UC with generate_service_certificate=false | 19:06 |
| tonyb | fultonj: okay, on it. | 19:06 |
| *** rdopiera has quit IRC | 19:07 | |
| fultonj | tonyb: if you want to know what's happening... there's a workbook called access.yaml which tries to make a secure connection to your end point and that workbook is triggere during step2 whenever you deploy w/ octavia or ceph. | 19:08 |
| fultonj | your end point --> your UC's public end point | 19:08 |
| tonyb | fultonj: Ahh okay. Thanks for the help that saved be blind grepping ~/code/openstack ;P | 19:09 |
| fultonj | tonyb: yep | 19:09 |
| *** ooolpbot has joined #tripleo | 19:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1786764 | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1787226 | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1787244 | 19:10 |
| *** ooolpbot has quit IRC | 19:10 | |
| openstack | Launchpad bug 1786764 in tripleo "tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates times out on prepare" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 19:10 |
| openstack | Launchpad bug 1787226 in tripleo "tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates is timing out" [Critical,Triaged] | 19:10 |
| openstack | Launchpad bug 1787244 in tripleo "rdopkg release has broken all ci jobs" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 19:10 |
| *** slaweq_ has joined #tripleo | 19:13 | |
| *** mjturek has joined #tripleo | 19:15 | |
| *** bfournie has quit IRC | 19:18 | |
| *** openstackgerrit has quit IRC | 19:19 | |
| *** openstackgerrit has joined #tripleo | 19:19 | |
| openstackgerrit | Derek Higgins proposed openstack/tripleo-heat-templates master: [WIP] Add scenario 012 - overlcoud baremetal+ansible-ml2 https://review.openstack.org/579603 | 19:19 |
| openstackgerrit | Derek Higgins proposed openstack/tripleo-quickstart master: [WIP] Add featureset 054 - overcloud baremetal+ansible-ml2 https://review.openstack.org/579601 | 19:20 |
| openstackgerrit | Olivier Bourdon proposed openstack/diskimage-builder master: [DNM] Testing CentOS images builds on Ubuntu Xenial https://review.openstack.org/591366 | 19:20 |
| openstackgerrit | Derek Higgins proposed openstack/tripleo-quickstart-extras master: Setup Ironic in Overcloud https://review.openstack.org/509728 | 19:20 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci master: [WIP] Testing a ansible-ml2 job https://review.openstack.org/582294 | 19:21 |
| *** sshnaidm|bbl is now known as sshnaidm | 19:22 | |
| jaosorior | Anybody else getting: resources.ServiceChain: Property error: resources[2].properties: Property DockerCinderConfigImage not assigned ? | 19:28 |
| jaosorior | as an error to the overcloud deploy | 19:29 |
| *** ykarel has quit IRC | 19:29 | |
| mwhahaha | that's an old error | 19:29 |
| mwhahaha | thought we fixed that | 19:29 |
| jaosorior | I just got it when I deployed with quickstart (a couple of hours ago) | 19:30 |
| jaosorior | even deployed with master-tripleo-ci | 19:30 |
| weshay | what is this? | 19:30 |
| weshay | 2018-08-15 18:58:12 | resources.ServiceChain: Property error: resources[2].properties: Property DockerCinderConfigImage not assigned | 19:30 |
| weshay | 2018-08-15 18:58:12 | + status_code=1 | 19:30 |
| *** mjturek has quit IRC | 19:30 | |
| weshay | http://logs.openstack.org/46/592146/2/check/tripleo-ci-centos-7-scenario007-multinode-oooq-container/d5e6d26/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-08-15_18_58_12 | 19:31 |
| mwhahaha | lol that | 19:31 |
| * mwhahaha gives up and goes home | 19:31 | |
| weshay | you are home | 19:31 |
| mwhahaha | stevebaker: should be here soon enough | 19:31 |
| *** mjturek has joined #tripleo | 19:31 | |
| weshay | mwhahaha, known issue/bug or should I look | 19:31 |
| weshay | and write it up | 19:31 |
| mwhahaha | weshay: write it up | 19:31 |
| weshay | fak | 19:31 |
| weshay | k | 19:31 |
| weshay | rlandy, fyi ^ | 19:31 |
| *** dprince has quit IRC | 19:32 | |
| mwhahaha | the last time it happened we needed to add some stuff to tripleo-common to pull in the services | 19:33 |
| *** dprince has joined #tripleo | 19:33 | |
| mwhahaha | it wasn't properly getting dectected when we do the container prepare | 19:33 |
| *** gbarros has joined #tripleo | 19:33 | |
| *** apetrich has joined #tripleo | 19:34 | |
| mwhahaha | i bet it's the image prepare stuff i merged | 19:34 |
| mwhahaha | and i bet we lost a THT patch along the way because of ci | 19:34 |
| mwhahaha | stevebaker, when you pop on please take a look -^ | 19:34 |
| rlandy | weshay: log bug or you did so already? | 19:35 |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart-extras master: DNM - Testing adding properties to flavors https://review.openstack.org/579406 | 19:37 |
| fultonj | can i {get_param: UndercloudTenantId} or something like it? | 19:37 |
| *** boazel has quit IRC | 19:38 | |
| fultonj | some ansible i'm working on which is run during config-download would benefit from having the tenant ID, but I don't know if that information is available in the environment. | 19:38 |
| mwhahaha | fultonj: no | 19:39 |
| mwhahaha | we don't have anything like that | 19:39 |
| *** dbecker has joined #tripleo | 19:40 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-heat-templates master: WIP: Disable Cinder services in scenario 007 container job https://review.openstack.org/592186 | 19:40 |
| fultonj | ok, thanks mwhahaha | 19:40 |
| weshay | mwhahaha, could just turning off cinder be the right thing? | 19:40 |
| weshay | ^ | 19:41 |
| mwhahaha | unlikely | 19:41 |
| fultonj | i got as far as {get_param: [EndpointMap, SwiftPublic, uri]} but was hoping to s/%(tenant_id)s/$tenan_id/g | 19:41 |
| mwhahaha | fultonj: why, what's trying to talk back to the undercloud | 19:41 |
| weshay | stevebaker, fyi.. mwhahaha asked me to ping you re: https://bugs.launchpad.net/tripleo/+bug/1787268 | 19:42 |
| openstack | Launchpad bug 1787268 in tripleo "Property DockerCinderConfigImage not assigned" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 19:42 |
| fultonj | https://review.openstack.org/#/c/582811/8/docker/services/ceph-ansible/ceph-base.yaml | 19:42 |
| mwhahaha | oh god | 19:42 |
| fultonj | mwhahaha: ansible wants to talk to swift | 19:42 |
| fultonj | oh god? | 19:42 |
| * mwhahaha closes laptop and walks away | 19:42 | |
| fultonj | lol | 19:42 |
| fultonj | i don't understand, is it me? | 19:43 |
| fultonj | :) | 19:43 |
| mwhahaha | basically you need a temp url to publish to | 19:43 |
| mwhahaha | we have a mistral workflow that can do that | 19:43 |
| *** hamzy has quit IRC | 19:43 | |
| mwhahaha | we aren't doing in ansible anywhere i'ma ware of | 19:43 |
| mwhahaha | but why are you doing this | 19:43 |
| mwhahaha | this has to be a massive hack to work around something else | 19:44 |
| mwhahaha | also please do not rely on swift on the undercloud (it may not be available) | 19:44 |
| *** mjturek has quit IRC | 19:45 | |
| fultonj | i need to preserve some data so i store it in swift | 19:45 |
| mwhahaha | you need a better way to store state data between runs | 19:45 |
| mwhahaha | if you need this we will never be able to have an undercloud-less deploy with ceph | 19:45 |
| mwhahaha | because there is no swift | 19:45 |
| *** mjturek has joined #tripleo | 19:45 | |
| mwhahaha | so this needs to be solved in a different way | 19:45 |
| fultonj | it was solved in queens this way | 19:46 |
| *** hamzy has joined #tripleo | 19:46 | |
| mwhahaha | it shouldn't have been | 19:46 |
| fultonj | i'm just dealing with the different caller | 19:46 |
| fultonj | so it could be migrated | 19:46 |
| fultonj | but.... | 19:46 |
| fultonj | i understand i could move to a temp url | 19:47 |
| fultonj | but not using swift at all | 19:47 |
| mwhahaha | so imho it's a larger issue that this state information is needed between runs and that we're (tripleo) not aware of it | 19:47 |
| fultonj | not sure what i can do about not needing this state information is needed between runs | 19:49 |
| fultonj | not sure what i can do about not needing this state information between runs | 19:49 |
| fultonj | i mean | 19:50 |
| mwhahaha | this is a ceph-ansible thing isn't it | 19:50 |
| fultonj | yes | 19:50 |
| mwhahaha | the problem is that we need to be able to describe A->B transitions from tripleo so know when we're adding nodes | 19:50 |
| mwhahaha | it sounds like a deficicenty that ceph-ansible needs some sort of state db around it. A place to store it would be on the nodes themselves rather than the ansible running location | 19:51 |
| mwhahaha | because what happens if you lose that info, you can't recreate it | 19:51 |
| fultonj | you can recreate it | 19:51 |
| fultonj | at least i have in the past with sucess | 19:52 |
| mwhahaha | why can't we recreate it on stack updates then? | 19:52 |
| mwhahaha | let me guess it's non-trivial? | 19:52 |
| fultonj | i could try that approach | 19:52 |
| fultonj | i'm not 100% sure i can rerecreate it though | 19:52 |
| mwhahaha | I don't think we're backing this up as part of the undercloud backup process so we should check with carlos on that | 19:52 |
| fultonj | let's assume i can't and keep going with this | 19:53 |
| fultonj | i know we're not backing it up yet | 19:53 |
| fultonj | we can add that | 19:53 |
| mwhahaha | oh god | 19:53 |
| fultonj | "a place to store it would be on the nodes themselves" | 19:55 |
| fultonj | that's what i recreate it from | 19:55 |
| mwhahaha | right that should be a function of ceph-ansible to have initial tasks that construct the current state | 19:56 |
| mwhahaha | then do the transition | 19:56 |
| fultonj | yes, i can see that | 19:56 |
| mwhahaha | imho the ceph ansible bits should be seen as an external thing. we'll provide the nodes but it handles state transitions | 19:56 |
| openstackgerrit | Goutham Pacha Ravi proposed openstack/puppet-tripleo master: Add httpd and tls support to manila-api https://review.openstack.org/592188 | 19:57 |
| *** mjturek has quit IRC | 19:59 | |
| fultonj | so i can pursue this but about the current behavior in queens and transitioning it to rocky | 20:00 |
| fultonj | and what about this swiftless undercloud | 20:00 |
| fultonj | ? | 20:00 |
| mwhahaha | fultonj: so our standalone deployment has no undercloud services | 20:01 |
| mwhahaha | fultonj: so it's purely ansible driven and has no state | 20:01 |
| mwhahaha | meaning it could work with ceph-ansible but not if you are relying on swift for updates | 20:01 |
| fultonj | it could work with ceph-ansible but not for certain ceph-ansible features if you are relying on swift for updates for those features | 20:03 |
| fultonj | like mon scaling | 20:03 |
| mwhahaha | right, scaling is a pretty important feature :D | 20:03 |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart master: [DNM] Add local network-environment file for fs001 https://review.openstack.org/577230 | 20:03 |
| fultonj | i thought of the standalone deployment as a developer tool not a fully supported deploy this way tool, but yes it would be better to not need swift for this stuff | 20:04 |
| mwhahaha | fultonj: it is now, may not be next cycle | 20:04 |
| fultonj | hmmmm | 20:04 |
| mwhahaha | point is we shouldnt' assume undercloud | 20:05 |
| mwhahaha | if we're having to rely on it, that's a problem | 20:05 |
| fultonj | how will it handle swift deployment? | 20:05 |
| mwhahaha | similar problem | 20:05 |
| fultonj | swift deployment relies on being able to back up the keyrings | 20:05 |
| mwhahaha | yea i know | 20:05 |
| fultonj | divide and conquer | 20:05 |
| mwhahaha | it's not good | 20:05 |
| fultonj | ok, i buy your transition state argument and want to go there | 20:05 |
| fultonj | to support this for next release | 20:05 |
| fultonj | next release --> stein | 20:06 |
| fultonj | for rocky i don't want to introduce a regression | 20:06 |
| weshay | everyone else is | 20:06 |
| weshay | all the cool kids are doing it rather | 20:06 |
| *** aevoo has joined #tripleo | 20:06 | |
| fultonj | weshay: everyone else is going there as in not needing swift | 20:07 |
| fultonj | i assume you mean | 20:07 |
| mwhahaha | no one needs distributed storage | 20:07 |
| mwhahaha | it's more headache than it's worth | 20:07 |
| * mwhahaha runs | 20:07 | |
| weshay | I'm just saying.. bah hum bug | 20:07 |
| fultonj | so... | 20:07 |
| mwhahaha | fultonj: so i don't have a solution for you because you'd have to populate the swift info from the client (or ui) at update time | 20:08 |
| mwhahaha | you could probably execute a workflow today to populate that in the plan | 20:08 |
| fultonj | yeah | 20:08 |
| fultonj | that's what i did when mistral did it | 20:09 |
| fultonj | i can go the temp url route | 20:09 |
| mwhahaha | fultonj: no i mean before execution | 20:09 |
| fultonj | i was thinking you meant do what swift does with temp urls | 20:09 |
| mwhahaha | well kinda | 20:09 |
| fultonj | during deployment a container is created to store something on the UC: | 20:10 |
| fultonj | https://github.com/openstack/tripleo-common/blob/master/workbooks/deployment.yaml#L167 | 20:10 |
| mwhahaha | but a workflow upfront that's at the beginning of the deploy workflow | 20:10 |
| fultonj | it's created by a workflow which sets two parameters that ansible can use to access swift | 20:10 |
| fultonj | https://github.com/openstack/tripleo-common/blob/master/workbooks/swift_rings_backup.yaml#L42 | 20:10 |
| *** ooolpbot has joined #tripleo | 20:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1786764 | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1787226 | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1787244 | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1787268 | 20:10 |
| openstack | Launchpad bug 1786764 in tripleo "tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates times out on prepare" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 20:10 |
| *** ooolpbot has quit IRC | 20:10 | |
| openstack | Launchpad bug 1787226 in tripleo "tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates is timing out" [Critical,Triaged] | 20:10 |
| openstack | Launchpad bug 1787244 in tripleo "rdopkg release has broken all ci jobs" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 20:10 |
| openstack | Launchpad bug 1787268 in tripleo "Property DockerCinderConfigImage not assigned" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 20:10 |
| fultonj | a workflow upfront that's at the beginning of the deploy workflow | 20:10 |
| fultonj | like the two i just linked to, right? | 20:10 |
| mwhahaha | yea | 20:11 |
| fultonj | i can do the swift approach | 20:11 |
| fultonj | i just found i had all the credentials i needed | 20:11 |
| *** honza has quit IRC | 20:11 | |
| fultonj | - generated inventory has the token [1] | 20:11 |
| fultonj | [1] http://logs.openstack.org/11/582811/8/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/e2ab585/logs/undercloud/var/lib/mistral/config-download-latest/tripleo-ansible-inventory.yaml.txt.gz | 20:11 |
| fultonj | - generated playbook has the swift commands [2] | 20:11 |
| fultonj | [2] http://logs.openstack.org/11/582811/8/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/e2ab585/logs/undercloud/var/lib/mistral/config-download-latest/external_deploy_steps_tasks.yaml.txt.gz | 20:11 |
| fultonj | HOWEVER... | 20:12 |
| mwhahaha | that's for teh overcloud? | 20:12 |
| fultonj | if this is terrible i can change | 20:12 |
| mwhahaha | oh undercloud_swift_url | 20:12 |
| fultonj | yeah | 20:12 |
| fultonj | swift commands, the command i need which work if i just get that swift_url fixed | 20:13 |
| fultonj | so i just needed the tenant id and i'd be done | 20:13 |
| fultonj | i'm using what's already there, though the tenant id may not be | 20:13 |
| fultonj | as we started the conversation | 20:13 |
| fultonj | how about this... | 20:14 |
| fultonj | rocky: use uc swift the same way using temp urls like oc swift | 20:15 |
| fultonj | stein: ceph-ansible support for not needing state info on uc | 20:15 |
| fultonj | ? | 20:15 |
| fultonj | queens already uses uc swift for ceph and swift | 20:15 |
| mwhahaha | yea | 20:16 |
| *** bfournie has joined #tripleo | 20:16 | |
| stevebaker | mwhahaha, weshay: morning. looking now | 20:16 |
| fultonj | thanks mwhahaha i'll purse the above plans | 20:17 |
| tonyb | fultonj: same error. | 20:18 |
| fultonj | tonyb: do you have this file on your UC /var/log/mistral/ceph-install-workflow.log ? | 20:19 |
| *** mjturek has joined #tripleo | 20:20 | |
| tonyb | fultonj: still no :( | 20:21 |
| tonyb | [stack@director ~]$ grep ^generate undercloud.conf | 20:21 |
| tonyb | generate_service_certificate = false | 20:21 |
| tonyb | so I didn't mess that bit up | 20:21 |
| *** mjturek has quit IRC | 20:21 | |
| weshay | stevebaker, thanks | 20:22 |
| fultonj | WORKFLOW='tripleo.storage.v1.ceph-install' | 20:23 |
| fultonj | UUID=$(mistral execution-list | grep $WORKFLOW | awk {'print $2'} | tail -1) | 20:23 |
| fultonj | tonyb: ^ | 20:23 |
| fultonj | tonyb: actually | 20:23 |
| fultonj | mistral execution-list | grep $WORKFLOW | 20:23 |
| tonyb | [stack@director ~]$ mistral execution-list | grep $WORKFLOW | 20:23 |
| tonyb | ERROR (app) HTTPConnectionPool(host='localhost', port=8989): Max retries exceeded with url: /v2/executions?sort_keys=created_at&limit=100&sort_dirs=asc (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fd608533350>: Failed to establish a new connection: [Errno 111] Connection refused',)) | 20:23 |
| tonyb | but that looks like a UC problem not the Root cause | 20:24 |
| fultonj | tonyb: if mistral is broken it could be the root cause :) | 20:24 |
| fultonj | so... | 20:25 |
| fultonj | tonyb: once mistral is up and running we'll find out though | 20:26 |
| tonyb | fultonj: my bad | 20:26 |
| fultonj | ? | 20:26 |
| tonyb | ... source stackrc *cough* | 20:26 |
| tonyb | (undercloud) [stack@director ~]$ mistral execution-list | grep $WORKFLOW | 20:26 |
| tonyb | | 723c8c0a-3b07-43d2-b253-6e56d351e751 | de71b6b5-a163-4cfc-a0ce-e06abccd865e | tripleo.storage.v1.ceph-install | | sub-workflow execution | | 20:26 |
| fultonj | :) | 20:26 |
| tonyb | d5cb95c0-7c59-4da0-ba8c-1fac4c51e328 | ERROR | Failure caused by error i... | 2018-08-15 20:16:51 | 2018-08-15 20:17:47 | | 20:27 |
| tonyb | (undercloud) [stack@director ~]$ | 20:27 |
| fultonj | UUID=d5cb95c0-7c59-4da0-ba8c-1fac4c51e328 | 20:27 |
| *** morazi has quit IRC | 20:27 | |
| fultonj | mistral task-list $UUID | 20:27 |
| tonyb | empty table | 20:29 |
| tonyb | yyyOA/ | 20:31 |
| tonyb | y | 20:31 |
| tonyb | /q | 20:31 |
| fultonj | let's be sure we're pulling the right one... | 20:31 |
| *** trozet has quit IRC | 20:31 | |
| fultonj | WORKFLOW='tripleo.storage.v1.ceph-install' | 20:31 |
| fultonj | UUID=$(mistral execution-list | grep $WORKFLOW | awk {'print $2'} | tail -1) | 20:32 |
| fultonj | mistral task-list $UUID | 20:32 |
| tonyb | fultonj: different UUID: http://paste.openstack.org/show/728137/ | 20:34 |
| fultonj | collect_nodes_uuid is where you're failing... | 20:35 |
| fultonj | TASK_ID=629792b9-20b2-4f79-8d3c-e6c8619b132b | 20:35 |
| fultonj | mistral task-get-result $TASK_ID | jq . | sed -e 's/\\n/\n/g' -e 's/\\"/"/g' | 20:35 |
| tonyb | "_raw_params": "dmidecode -s system-uuid", \ | 20:36 |
| tonyb | Oh | 20:36 |
| tonyb | ppc64le doesn't so that :/ | 20:36 |
| fultonj | ha ha ha ha | 20:36 |
| fultonj | you found a bug! | 20:36 |
| tonyb | So where would I fix that? | 20:36 |
| fultonj | this means that PPC has no support of node spec hiera data | 20:37 |
| fultonj | so, the workflow needs to be updated to if no ppc, don't run that command | 20:38 |
| tonyb | or run a similar command. | 20:38 |
| fultonj | and we need to update docs like... | 20:38 |
| fultonj | tonyb: sure! | 20:38 |
| tonyb | so its a tripleo-common workflow or somethign in ceph-ansible | 20:38 |
| fultonj | tripleo-common | 20:38 |
| tonyb | okay | 20:39 |
| fultonj | https://docs.openstack.org/tripleo-docs/latest/install/advanced_deployment/node_specific_hieradata.html | 20:39 |
| tonyb | fultonj: Thanks. | 20:39 |
| fultonj | ^ either we add not supported in PPC | 20:39 |
| fultonj | or... | 20:39 |
| fultonj | is there an alternative we could introduce? | 20:39 |
| *** mjturek has joined #tripleo | 20:39 | |
| tonyb | I can get you a systemID just not from dmideode | 20:39 |
| fultonj | nice! | 20:39 |
| fultonj | tonyb: what does ironic do when it introspects them? | 20:40 |
| fultonj | openstack baremetal introspection data save NODE-ID | jq .extra.system.product.uuid | 20:40 |
| fultonj | ^ for a NODE-ID that is a PPC? | 20:40 |
| fultonj | i'm saying you might have a symetric bug in ironic too | 20:40 |
| fultonj | if "inspection_extras = true" supports PPC that is | 20:41 |
| *** mjturek has quit IRC | 20:41 | |
| tonyb | Hmm okay. I thought we'd get that from lshw but perhaps not | 20:42 |
| * tonyb hasn't tried with inspection_extras = true | 20:42 | |
| *** raildo has quit IRC | 20:42 | |
| fultonj | tonyb: food for thought | 20:42 |
| fultonj | https://github.com/openstack/tripleo-common/blob/master/workbooks/ceph-ansible.yaml#L106-L111 | 20:42 |
| fultonj | i think that embeded playbook just needs an extra when | 20:42 |
| fultonj | or two | 20:43 |
| fultonj | dmidecode -s system-uuid when x86 | 20:43 |
| fultonj | other_thing when ppc | 20:43 |
| tonyb | fultonj: Cool. I'll hack it up today | 20:44 |
| fultonj | tonyb: do you want to file a bug or try a patch? | 20:44 |
| fultonj | nice | 20:44 |
| fultonj | tonyb: please add me to the review | 20:44 |
| tonyb | I'll do the fix and then we can decide how to proceed | 20:44 |
| tonyb | fultonj: Will do | 20:45 |
| * tonyb needs more systems! | 20:45 | |
| openstackgerrit | Olivier Bourdon proposed openstack/diskimage-builder master: [DNM] Testing CentOS images builds on Ubuntu Xenial https://review.openstack.org/591366 | 20:54 |
| tonyb | fultonj: Thanks for your help | 20:57 |
| fultonj | tonyb: glad we found what the issue is | 20:58 |
| *** aevoo has quit IRC | 20:58 | |
| fultonj | i think this is the first time for PPC for that worklow | 20:58 |
| *** gbarros has quit IRC | 20:58 | |
| tonyb | fultonj: Yup. 90% of working on ppc is finding these assumptions | 20:59 |
| *** gbarros has joined #tripleo | 20:59 | |
| * tonyb goes for breakfast | 20:59 | |
| openstackgerrit | Steve Baker proposed openstack/tripleo-quickstart-extras master: Switch to workflow driven image prepare https://review.openstack.org/573476 | 21:02 |
| *** trown is now known as trown|outtypewww | 21:03 | |
| *** rh-jelabarre has quit IRC | 21:09 | |
| *** ooolpbot has joined #tripleo | 21:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1786764 | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1787226 | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1787244 | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1787268 | 21:10 |
| openstack | Launchpad bug 1786764 in tripleo "tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates times out on prepare" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 21:10 |
| *** ooolpbot has quit IRC | 21:10 | |
| openstack | Launchpad bug 1787226 in tripleo "tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates is timing out" [Critical,Triaged] | 21:10 |
| openstack | Launchpad bug 1787244 in tripleo "rdopkg release has broken all ci jobs" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 21:10 |
| openstack | Launchpad bug 1787268 in tripleo "Property DockerCinderConfigImage not assigned" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 21:10 |
| stevebaker | weshay: I'm spinning up a reproducer, but I have 2 theories. 1: prepare is supposed to be called with every -e argument that overcloud deploy is called with, but this is not even remotely happening right now, and is really hard the way deploy_args can have non -e arguments | 21:13 |
| *** agopi has quit IRC | 21:15 | |
| stevebaker | weshay: 2: When a role count is zero, I've seen heat sometimes treat its parameters as mandatory even though they are not | 21:17 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: DNM: Enable virt7-container-common-candidate https://review.openstack.org/592079 | 21:25 |
| *** jcoufal has quit IRC | 21:26 | |
| *** rcernin has joined #tripleo | 21:29 | |
| *** dprince has quit IRC | 21:29 | |
| weshay | stevebaker, hrm.. k | 21:39 |
| *** gbarros has quit IRC | 21:40 | |
| *** slaweq_ has quit IRC | 21:42 | |
| *** gbarros has joined #tripleo | 21:43 | |
| *** dbecker has quit IRC | 21:44 | |
| *** zaneb has quit IRC | 21:46 | |
| *** zaneb has joined #tripleo | 21:46 | |
| openstackgerrit | waleed mousa proposed openstack/tripleo-heat-templates master: Removing the deprecated environment files for hw offloading https://review.openstack.org/573102 | 21:47 |
| *** apetrich has quit IRC | 21:55 | |
| *** zaneb has quit IRC | 21:59 | |
| *** zaneb has joined #tripleo | 21:59 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-heat-templates master: WIP: Disable Cinder services in scenario 007 container job https://review.openstack.org/592186 | 22:04 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci master: [WIP] Testing a ansible-ml2 job https://review.openstack.org/582294 | 22:04 |
| *** zaneb has quit IRC | 22:05 | |
| *** ooolpbot has joined #tripleo | 22:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1786764 | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1787226 | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1787244 | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1787268 | 22:10 |
| openstack | Launchpad bug 1786764 in tripleo "tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates times out on prepare" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 22:10 |
| *** ooolpbot has quit IRC | 22:10 | |
| openstack | Launchpad bug 1787226 in tripleo "tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates is timing out" [Critical,Triaged] | 22:10 |
| openstack | Launchpad bug 1787244 in tripleo "rdopkg release has broken all ci jobs" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 22:10 |
| openstack | Launchpad bug 1787268 in tripleo "Property DockerCinderConfigImage not assigned" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 22:10 |
| *** gbarros has quit IRC | 22:18 | |
| bfournie | hi tonyb, just looking at scroll back, in the introspection data you should get system info for ppc because ironic-python-agent uses lshw - https://review.openstack.org/#/c/501799/ | 22:19 |
| *** itlinux has quit IRC | 22:22 | |
| tonyb | bfournie: Yeah I have the introspection data but is seems I didn't get the 'extra' data as: | 22:22 |
| tonyb | (undercloud) [stack@director ~]$ openstack baremetal introspection data save 9f6006f2-0289-41a4-a04d-b1f993551db9 | jq .extra.system.product.uuid | 22:22 |
| tonyb | null | 22:22 |
| tonyb | so step 1) work out how to get a hardware UUID on ppc64le | 22:23 |
| tonyb | step 2) add that into the ceph-ansible code | 22:23 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci master: [WIP] Testing a ansible-ml2 job https://review.openstack.org/582294 | 22:23 |
| tonyb | step 3) add it to ironic/tripleo so that it's gathered as part of the 'extra' data | 22:23 |
| * tonyb needs to do more research on step 3 | 22:24 | |
| bfournie | tonyb: do you have extra_hardware set in /etc/ironic-inspector/inspector.conf ? | 22:25 |
| bfournie | processing_hooks=$default_processing_hooks,extra_hardware,lldp_basic,local_link_connection | 22:25 |
| tonyb | bfournie: it seems I do | 22:26 |
| tonyb | (undercloud) [stack@director ~]$ sudo grep extra_hardware /etc/ironic-inspector/inspector.conf | 22:26 |
| tonyb | processing_hooks=$default_processing_hooks,extra_hardware,lldp_basic,local_link_connection | 22:26 |
| tonyb | Oh perhaps inspection_extras defaults to true (from undercloud.conf) | 22:27 |
| tonyb | (undercloud) [stack@director ~]$ grep inspection_extras undercloud.conf | 22:27 |
| tonyb | #inspection_extras = true | 22:27 |
| tonyb | # inspection_extras set to True. (boolean value) | 22:27 |
| bfournie | tonyb: hmm, ok, I wonder why system data isn't set then, is it just uuid that's not set? | 22:27 |
| tonyb | bfournie: http://paste.openstack.org/show/728140/ | 22:28 |
| bfournie | tonyb: oh, the lshw change just sets vendor, product, serial, and memory it looks like | 22:28 |
| tonyb | so the product.uuid is missing | 22:28 |
| bfournie | tonyb: yeah | 22:28 |
| tonyb | bfournie: Ahh okay I wonder if uuid is available | 22:28 |
| bfournie | tonyb: let me see if its set in x86 | 22:29 |
| tonyb | bfournie: seems not | 22:29 |
| bfournie | tonyb: yep, product": {"serial": "JLRCY12", "vendor": "Dell Inc.", "name": "PowerEdge R320 (SKU=NotProvided;ModelName=PowerEdge R320)", "uuid": "4C4C4544-004C-5210-8043-CAC04F593132"}, | 22:29 |
| tonyb | Hmm okay | 22:30 |
| * tonyb tries real hardware | 22:30 | |
| tonyb | [root@overcloud-controller-0 ~]# lshw -json | grep -Ei $(dmidecode -s system-uuid) | 22:30 |
| tonyb | [root@overcloud-controller-0 ~]# | 22:30 |
| tonyb | but that's a VM | 22:30 |
| tonyb | [root@overcloud-controller-0 ~]# dmidecode -s system-uuid | 22:30 |
| tonyb | DFE7543E-7D20-4232-9C57-1AC2D356CFEF | 22:30 |
| tonyb | bfournie: Oh wait .. you paste is that from ironic or lshw? | 22:32 |
| *** d0ugal has quit IRC | 22:33 | |
| bfournie | tonyb: output of "openstack baremetal introspection data save <node>" | 22:34 |
| tonyb | bfournie: Okay got it | 22:34 |
| tonyb | bfournie: that explains why I'm not geting it ... cause I'm looking at lshw ;P | 22:34 |
| bfournie | :-) | 22:35 |
| tonyb | so even though that data is called UUID does it actually need to be a UUID? | 22:39 |
| tonyb | for now the best I can do on ppc64le is a different string that's tied to the hardware | 22:40 |
| *** d0ugal has joined #tripleo | 22:43 | |
| tonyb | fultonj, bfournie: ^^ | 22:45 |
| bfournie | tonyb: I'm not sure, fultonj would know better. i do think we need a bug for IPA not setting uuid for ppc - http://pastebin.test.redhat.com/632599 | 22:51 |
| tonyb | bfournie: Yup I agree. I'll open one once I find an answer to "does ppc even have that thing" ;P | 22:52 |
| bfournie | tonyb: yeah, that's a good question | 22:53 |
| tonyb | [root@overcloud-novacomputeppc64le-0 device-tree]# lshw -json | jq .serial | 22:54 |
| tonyb | "211729A" | 22:54 |
| tonyb | is as close as I can get right now. That's *seems* to be a serial number for the chassis. | 22:54 |
| weshay | mwhahaha, failed again | 22:55 |
| mwhahaha | weshay: gah | 22:55 |
| mwhahaha | weshay: whait what failed | 22:56 |
| weshay | http://zuul.openstack.org/stream.html?uuid=78c9ae8c0b724f628f817b5c8a019450&logfile=console.log | 22:57 |
| weshay | scenario007 | 22:57 |
| mwhahaha | of course | 22:58 |
| weshay | mwhahaha, let's move scen007 to non-voting | 23:00 |
| weshay | get things back online.. and take it from there | 23:00 |
| *** d0ugal has quit IRC | 23:01 | |
| weshay | also finish the ara patches so we can understand the timeouts | 23:01 |
| mwhahaha | k you'll have to roll scenario007 thing with the rdopkg pin | 23:01 |
| weshay | well.. depends on | 23:02 |
| *** d0ugal has joined #tripleo | 23:02 | |
| openstackgerrit | wes hayutin proposed openstack-infra/tripleo-ci master: move scenario007 containers to nv https://review.openstack.org/592216 | 23:05 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras master: Pin rdopkg to 0.46.3 https://review.openstack.org/592146 | 23:05 |
| weshay | hrm.. | 23:06 |
| mwhahaha | nope | 23:06 |
| mwhahaha | has to be combined | 23:06 |
| mwhahaha | because teh scenario007 won't land due to the pin | 23:07 |
| mwhahaha | plz align stars | 23:07 |
| weshay | ya.. right | 23:07 |
| *** agopi has joined #tripleo | 23:07 | |
| weshay | mwhahaha, ok.. we need to make sure nobody workflows | 23:09 |
| mwhahaha | i make no promises | 23:09 |
| *** ooolpbot has joined #tripleo | 23:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1786764 | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1787226 | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1787244 | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1787268 | 23:10 |
| openstack | Launchpad bug 1786764 in tripleo "tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates times out on prepare" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 23:10 |
| *** ooolpbot has quit IRC | 23:10 | |
| openstack | Launchpad bug 1787226 in tripleo "tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates is timing out" [Critical,Triaged] | 23:10 |
| openstack | Launchpad bug 1787244 in tripleo "rdopkg release has broken all ci jobs" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 23:10 |
| openstack | Launchpad bug 1787268 in tripleo "Property DockerCinderConfigImage not assigned" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 23:10 |
| mwhahaha | weshay: i'll send a note | 23:11 |
| mwhahaha | to the ml | 23:11 |
| weshay | I'm not quite sure how to do this w/o a really big hammer | 23:11 |
| * weshay pokes around some more | 23:12 | |
| weshay | sec | 23:12 |
| *** rpioso is now known as rpioso|afk | 23:12 | |
| mwhahaha | have we figured out the cinder config thing yet | 23:12 |
| mwhahaha | or is that on scenario007 | 23:13 |
| mwhahaha | as an aside, i think i've figured out a way to speed up the build-test-packages role | 23:14 |
| mwhahaha | but that's for another day | 23:14 |
| mwhahaha | let's just blame someone else and go to bed so the people on the other side of the world can fix it when they wake up | 23:16 |
| weshay | ok.. I got this sec | 23:17 |
| weshay | at least to turn off building rpms in tripleo-ci | 23:17 |
| mwhahaha | so switch everything to noop? :D | 23:17 |
| mwhahaha | cause that's basically what that is | 23:17 |
| *** rcernin has quit IRC | 23:18 | |
| *** rcernin has joined #tripleo | 23:19 | |
| openstackgerrit | wes hayutin proposed openstack-infra/tripleo-ci master: move scenario007 containers to nv https://review.openstack.org/592216 | 23:19 |
| weshay | mwhahaha, ok. that should get scen07 to nv | 23:20 |
| *** gbarros has joined #tripleo | 23:20 | |
| weshay | mwhahaha, that also turns off build-test-package | 23:20 |
| mwhahaha | weshay: so line up the patches | 23:20 |
| mwhahaha | weshay: propose a patch to revert that after the quickstart one | 23:21 |
| mwhahaha | and we'll try to get them all in the gate | 23:21 |
| weshay | they are now right? | 23:21 |
| mwhahaha | no | 23:21 |
| weshay | this needs to land first https://review.openstack.org/#/c/592216/1 | 23:21 |
| weshay | then this https://review.openstack.org/#/c/592146/3 | 23:21 |
| weshay | then I can turn it back on in a new patch | 23:21 |
| mwhahaha | right | 23:21 |
| mwhahaha | so propose the patch that depends on 592146 | 23:22 |
| weshay | ok.. man.. chill winston | 23:22 |
| weshay | right | 23:22 |
| weshay | I was just reconsulting my ansible handbook | 23:22 |
| mwhahaha | pffft reference material | 23:22 |
| openstackgerrit | wes hayutin proposed openstack-infra/tripleo-ci master: re-enable build-test-packages https://review.openstack.org/592218 | 23:24 |
| weshay | mwhahaha, at christmas time.. I'm sending you a coffee table book w/ your irc reactions | 23:25 |
| weshay | 1 per page | 23:25 |
| weshay | and some meme next to it | 23:25 |
| mwhahaha | and you have failed me | 23:25 |
| mwhahaha | that patch should depend-on Ib28f20db0aa1db9f2c5b3cbbdae1c85c97bbeba1 | 23:25 |
| weshay | bah fak | 23:26 |
| weshay | it should | 23:26 |
| weshay | I blame the buffer | 23:26 |
| mwhahaha | i blame everyone | 23:27 |
| openstackgerrit | wes hayutin proposed openstack-infra/tripleo-ci master: re-enable build-test-packages https://review.openstack.org/592218 | 23:27 |
| weshay | I blame ruzika's teddy bear | 23:27 |
| weshay | backpack | 23:27 |
| weshay | mwhahaha, ok.. I'll see you in hell and in about 1.5hrs | 23:29 |
| mwhahaha | btw dlrn doesn't work right under python3 | 23:29 |
| weshay | BAH | 23:29 |
| mwhahaha | just fyi | 23:29 |
| weshay | killing me | 23:29 |
| mwhahaha | well it does, until you use --order | 23:29 |
| mwhahaha | then it says NOPE | 23:29 |
| mwhahaha | because it uses cmp, https://docs.python.org/3/howto/sorting.html#the-old-way-using-the-cmp-parameter | 23:29 |
| weshay | mwhahaha, that's what you get for lining up 6 in a row | 23:29 |
| mwhahaha | it's like 12 | 23:29 |
| *** dsneddon has quit IRC | 23:30 | |
| *** dsneddon has joined #tripleo | 23:31 | |
| *** d0ugal has quit IRC | 23:42 | |
| *** rlandy is now known as rlandy|afk | 23:52 | |
| *** d0ugal has joined #tripleo | 23:55 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!