Friday, 2018-08-03

openstackgerritMerged openstack/tripleo-quickstart-extras master: Cleaning up tempest skip tests for pike  https://review.openstack.org/58784400:08
openstackgerritMerged openstack/tripleo-heat-templates master: Expose dnsmasq_local_resolv option  https://review.openstack.org/57793600:08
openstackgerritMerged openstack/tripleo-heat-templates master: Disable recursion in Designate-managed BIND  https://review.openstack.org/57793700:08
openstackgerritMerged openstack/python-tripleoclient master: New function: run_ansible_playbook  https://review.openstack.org/58653800:08
*** hamzy has joined #tripleo00:09
*** ooolpbot has joined #tripleo00:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION00:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178401700:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178482200:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178484700:10
*** ooolpbot has quit IRC00:10
openstackLaunchpad bug 1784017 in tripleo "Build of instance was re-scheduled: invalid argument: could not find capabilities for domaintype=kvm" [Critical,In progress] - Assigned to Rafael Folco (rafaelfolco)00:10
openstackLaunchpad bug 1784822 in tripleo "legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master fails contact ipmi" [Critical,Triaged]00:10
openstackLaunchpad bug 1784847 in tripleo "exception introspecting nodes" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)00:10
*** tonyb has quit IRC00:16
*** edmondsw has joined #tripleo00:37
*** edmondsw has quit IRC00:42
*** jcoufal has joined #tripleo00:49
*** mrsoul has joined #tripleo01:06
*** ooolpbot has joined #tripleo01:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION01:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178401701:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178482201:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178484701:10
*** ooolpbot has quit IRC01:10
openstackLaunchpad bug 1784017 in tripleo "Build of instance was re-scheduled: invalid argument: could not find capabilities for domaintype=kvm" [Critical,In progress] - Assigned to Rafael Folco (rafaelfolco)01:10
openstackLaunchpad bug 1784822 in tripleo "legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master fails contact ipmi" [Critical,Triaged]01:10
openstackLaunchpad bug 1784847 in tripleo "exception introspecting nodes" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)01:10
*** sai_p has quit IRC01:19
*** sai_p has joined #tripleo01:24
*** gbarros has joined #tripleo01:47
*** dtrainor has quit IRC01:54
*** gbarros has quit IRC01:57
*** ramishra has joined #tripleo02:00
*** jcoufal has quit IRC02:05
*** ramishra has quit IRC02:08
*** ooolpbot has joined #tripleo02:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION02:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178401702:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178482202:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178484702:10
*** ooolpbot has quit IRC02:10
openstackLaunchpad bug 1784017 in tripleo "Build of instance was re-scheduled: invalid argument: could not find capabilities for domaintype=kvm" [Critical,In progress] - Assigned to Rafael Folco (rafaelfolco)02:10
openstackLaunchpad bug 1784822 in tripleo "legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master fails contact ipmi" [Critical,Triaged]02:10
openstackLaunchpad bug 1784847 in tripleo "exception introspecting nodes" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)02:10
*** tonyb has joined #tripleo02:22
*** edmondsw has joined #tripleo02:25
*** edmondsw has quit IRC02:30
*** psachin has joined #tripleo02:35
openstackgerritMerged openstack/python-tripleoclient master: Drop incorrect reference to heat_launch  https://review.openstack.org/58785702:44
openstackgerritSteve Baker proposed openstack/ansible-role-tripleo-modify-image master: Add Buildah support  https://review.openstack.org/58839003:07
*** dxiri has joined #tripleo03:08
*** ooolpbot has joined #tripleo03:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION03:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178401703:10
openstackLaunchpad bug 1784017 in tripleo "Build of instance was re-scheduled: invalid argument: could not find capabilities for domaintype=kvm" [Critical,In progress] - Assigned to Rafael Folco (rafaelfolco)03:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178482203:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178484703:10
*** ooolpbot has quit IRC03:10
openstackLaunchpad bug 1784822 in tripleo "legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master fails contact ipmi" [Critical,Triaged]03:10
openstackLaunchpad bug 1784847 in tripleo "exception introspecting nodes" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)03:10
*** rlandy|bbl is now known as rlandy03:14
*** skramaja has joined #tripleo03:22
*** errantekarmico has quit IRC03:26
*** udesale has joined #tripleo03:31
*** itlinux_ has joined #tripleo03:31
openstackgerritwangqi proposed openstack/puppet-tripleo master: Set connect_timeout in mysql  https://review.openstack.org/58843303:39
*** dxiri has quit IRC03:41
openstackgerritChandan Kumar proposed openstack/tripleo-quickstart-extras master: Use local docker registry host for tempest container  https://review.openstack.org/58436803:47
*** ratailor has joined #tripleo03:59
*** ramishra has joined #tripleo04:03
*** ooolpbot has joined #tripleo04:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION04:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178401704:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178482204:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178484704:10
*** ooolpbot has quit IRC04:10
openstackLaunchpad bug 1784017 in tripleo "Build of instance was re-scheduled: invalid argument: could not find capabilities for domaintype=kvm" [Critical,In progress] - Assigned to Rafael Folco (rafaelfolco)04:10
openstackLaunchpad bug 1784822 in tripleo "legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master fails contact ipmi" [Critical,Triaged]04:10
openstackLaunchpad bug 1784847 in tripleo "exception introspecting nodes" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)04:10
*** dxiri has joined #tripleo04:21
openstackgerritSteve Baker proposed openstack/tripleo-docs master: Document new push_destination behaviour  https://review.openstack.org/58843904:25
*** rlandy has quit IRC04:28
*** thrash is now known as thrash|g0ne04:30
*** udesale has quit IRC04:35
*** holser_ has joined #tripleo04:47
*** ykarel has joined #tripleo04:52
jaosoriorbnemec: bummer, when you were available I was gone already (the different timezone are a bit hard)04:54
*** udesale has joined #tripleo04:55
jaosoriorbandini: around?05:06
bandinijaosorior: not for too long ;)05:06
jaosoriorbandini: could you check these out ? https://review.openstack.org/#/c/588151/ https://review.openstack.org/#/c/588136/05:07
*** ooolpbot has joined #tripleo05:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION05:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178401705:10
openstackLaunchpad bug 1784017 in tripleo "Build of instance was re-scheduled: invalid argument: could not find capabilities for domaintype=kvm" [Critical,In progress] - Assigned to Rafael Folco (rafaelfolco)05:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178482205:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178484705:10
*** ooolpbot has quit IRC05:10
openstackLaunchpad bug 1784822 in tripleo "legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master fails contact ipmi" [Critical,Triaged]05:10
openstackLaunchpad bug 1784847 in tripleo "exception introspecting nodes" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)05:10
jaosoriorbandini: thanks05:12
bandininp05:12
*** pmannidi has joined #tripleo05:13
*** dxiri has quit IRC05:14
Tenguhello there :)05:15
*** shyambiradar has joined #tripleo05:17
jaosoriorTengu: good days ir05:23
jaosorior* day sir05:23
shyambiradarjaosorior: Hi05:27
shyambiradarIf I use nova-compute container as base image, will it have nova-compute service started?05:28
jaosoriorshyambiradar: it won't05:28
jaosoriorshyambiradar: the containers only run the services that you configure them to run05:28
jaosoriordid you need the compute service to be running on the same container?05:29
shyambiradarno05:29
shyambiradarI don't want nova-compute service05:29
jhebdenhey all, quick question hopefully - doing a deployment, and getting the following -05:29
jhebdenException creating plan: 404 Client Error: Not Found for url: http://192.168.24.1:8080/v1/AUTH_f3e94f0102fb49a6b343a24bf9b50934/overcloud/docker_config_scripts/nova_statedir_ownership.py05:30
jaosoriorshyambiradar: so then your alright :)05:30
jhebdenI've searched my director host for nova_statedir_ownership.py, it's nowhere to be found - this is on a fresh RHOSP1305:30
shyambiradarjaosorior: I am implementing a nova-compute plugin05:30
shyambiradarso I want all libraries05:30
shyambiradarpackages and other artifacts05:30
jhebdenif anyone has any ideas, I'd appreciate it05:30
jaosoriorthen yes, making your container have nova-compute as the base sounds like the way to go05:31
jaosoriorjhebden: can you pass more logs?05:31
shyambiradargreat05:31
jhebdenjaosorior: absolutely05:31
jhebdenjaosorior: http://paste.openstack.org/show/727204/05:32
jhebdenjaosorior: I've tried with and without docker/docker-ha and I get the same result05:33
shyambiradarjaosorior: Can you please give me nova compute container location in redhat registry?05:33
shyambiradarIf you know05:34
jaosoriorshyambiradar: not sure where that is unfortunately05:34
shyambiradarok05:34
shyambiradarI will search05:34
shyambiradarthanks05:34
jaosoriorjhebden: is this a new deployment or were you updating the plan?05:34
jhebdenjaosorior: this was a new deployment, I had unsuccessfully deployed the overcloud previously05:35
jaosoriorjhebden: you could try deleting the plan05:35
jhebdenjaosorior: will try that now05:35
jhebdenhmm, no plan is showing j05:36
jhebden* jaosorior05:36
jhebdenwith openstack overcloud plan list05:36
jhebdenit seems odd to me that I can't find that file on the director host05:37
openstackgerritCédric Jeanneret proposed openstack/python-tripleoclient master: New preflight validation: disk space on undercloud  https://review.openstack.org/58654105:37
jhebdenit seems that it should be uploaded to swift from the host either during deploy or undercloud install?05:37
jaosoriorjhebden: the plan is updated on deploy05:38
Tenguoopss... wrong commig -.-'.05:39
*** quiquell has joined #tripleo05:39
openstackgerritCédric Jeanneret proposed openstack/python-tripleoclient master: New preflight check for the undercloud: disk space  https://review.openstack.org/58654105:40
*** jaosorior has quit IRC05:40
*** shyambiradar has quit IRC05:41
*** jaosorior has joined #tripleo05:41
*** janki has joined #tripleo05:44
openstackgerritCédric Jeanneret proposed openstack/python-tripleoclient master: New preflight check for the undercloud: disk space  https://review.openstack.org/58654105:46
*** holser_ has quit IRC05:48
*** jrist has quit IRC05:49
*** shyambiradar has joined #tripleo05:50
jhebdenso it seems like my issue is that https://review.openstack.org/#/c/587066/ hasn't landed yet05:54
jhebdenhowever the heat templates refer to scripts which are included in that change05:54
jaosoriorjhebden: oh damn :/ what is referring to that change then?05:56
jaosoriorlets try to get that merged asap05:56
jhebdenjaosorior: it looks like the nova-* templates are expecting this script05:56
jhebdenfor openstack-tripleo-heat-templates05:56
*** apetrich has joined #tripleo05:57
jhebdenjaosorior: docker/services/nova-compute.yaml, docker/services/nova-compute-common.yaml and docker/services/nova-ironic.yaml all seem to refer to the docker_config_scripts05:59
jhebdenjaosorior: this is openstack-tripleo-heat-templates-8.0.4-6.el7ost.src.rpm06:00
jhebdenalso, thank you for you help jaosorior :)06:00
jaosoriorjhebden: well... that's interesting, nova_statedir_ownership.py is available in the t-h-t repo06:01
jhebdenI noticed that, maybe is is just missing from the RPM?06:02
quiquellykarel: I have progress at openshift 3.10, but is failing now at installing origin-node-3.9.006:02
quiquellykarel: http://logs.openstack.org/52/588152/3/check/tripleo-ci-centos-7-scenario009-multinode-oooq/90c035c/logs/undercloud/var/lib/mistral/overcloud/openshift/playbook.log.txt.gz06:02
jaosoriorjhebden: What version are you trying to install?06:02
jhebdenjaosorior: openstack-tripleo-heat-templates-8.0.4-6.el7ost.src.rpm06:02
jhebdenon RHOSP1306:02
jaosoriorjhebden: OK, maybe the package hasn't been updated06:02
jaosoriorhere's the patch that landed for queens https://review.openstack.org/#/c/585762/06:03
ykarelquiquell, checking06:03
quiquellykarel: I think we have to set openshift_pkg_version 3.10 or the like06:04
quiquellykarel: TASK [openshift_version : debug] ***********************************************06:05
quiquellThursday 02 August 2018  13:26:58 +0000 (0:00:00.114)       0:01:32.929 *******06:05
quiquellok: [centos-7-rax-dfw-0001112025] => {06:05
quiquell    "msg": "openshift_pkg_version was not defined. Falling back to -3.9.0"06:05
quiquell}06:05
ykarelquiquell, hmm possibly06:06
quiquellykarel: Going to try06:06
ykarelquiquell, should be 3.10.1806:07
jhebdenjaosorior: helpful if I raise a bug for this?06:08
*** ooolpbot has joined #tripleo06:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION06:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178401706:10
openstackLaunchpad bug 1784017 in tripleo "Build of instance was re-scheduled: invalid argument: could not find capabilities for domaintype=kvm" [Critical,In progress] - Assigned to Rafael Folco (rafaelfolco)06:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178482206:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178484706:10
*** ooolpbot has quit IRC06:10
openstackLaunchpad bug 1784822 in tripleo "legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master fails contact ipmi" [Critical,Triaged]06:10
openstackLaunchpad bug 1784847 in tripleo "exception introspecting nodes" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)06:10
quiquellykarel: Where is the place where we download openshift-ansible from git instead of package system ?06:14
quiquellykarel: You have a patch to revert it06:14
ykarelquiquell, revert patch https://review.openstack.org/#/c/588198/06:16
ykarelquiquell, but looks like until and unless openshift repo is enabled, package install would fail06:16
quiquellykarel: 3.10.18 is the openshift-ansible version and 3.10.0 is the openshift version06:17
quiquellykarel: I am correct ?06:17
quiquellykarel: Learning in the process06:17
ykarelquiquell, /me also no idea, looking into repo06:18
*** yprokule has joined #tripleo06:18
*** jfrancoa has joined #tripleo06:18
quiquellykarel: If we find more problems after upgrading to 3.10, then we can revert, to unlock it06:18
openstackgerritQuique Llorente proposed openstack/tripleo-heat-templates master: Replace openshift_node_labels with openshift_node_group_name  https://review.openstack.org/58815206:19
quiquellykarel: ^06:21
ykarelquiquell, so as i checked repo for openshift-3.10 is not avaiable:- http://mirror.centos.org/centos/7/extras/x86_64/Packages/, so may be we need to enable openshift ansible 3.9 repo, use openshift-ansible 3.10 and use openshift_pkg_version=3.9.006:22
quiquellykarel: https://github.com/openshift/openshift-ansible/issues/839906:24
quiquellykarel: It's from May and they don't have the 3.10 RPMs yet06:24
ykarelquiquell, hmm06:24
quiquellykarel: Maybe we have to include this https://github.com/openshift/openshift-ansible/blob/master/images/installer/origin-extra-root/etc/yum.repos.d/centos-openshift-origin.repo06:25
quiquellykarel: Also in the github they say that RH has deliver 3.1006:25
ykarelquiquell, but that repo do not have 3.1006:25
ykarelso we have to stick to 3.906:25
ykareluntil it's available06:25
quiquellykarel: Yep just checked it... this is the one we have to install to get 3.9 and go back to 3.9 and 3.10 for ansible as you said06:26
shyambiradarHi06:26
shyambiradarI used kolla nova compute container image as base image for my container06:26
shyambiradarOnce I launch my container I am getting following error06:27
shyambiradar()[42436@rhosp13-shyam-dev /]$ id nova06:27
shyambiradaruid=162(nova) gid=162(nova) groups=162(nova),99(nobody),993(libvirt),107(qemu)06:27
shyambiradarid: cannot find name for user ID 4243606:27
shyambiradar----------------------------------06:27
shyambiradarAny thoughts?06:27
jaosoriorjhebden: well, I would wait for the next updates; but if you want you can do so, so it gets some following06:27
quiquellykarel: Lat's try, uncomment flaper87 to get centos-release-openshift-origin3906:35
quiquelllast06:35
ykarelquiquell, ack06:35
openstackgerritQuique Llorente proposed openstack/tripleo-heat-templates master: Replace openshift_node_labels with openshift_node_group_name  https://review.openstack.org/58815206:38
openstackgerritQuique Llorente proposed openstack/tripleo-quickstart master: Activate  centos-release-openshift-origin39-1-1.el7.centos  https://review.openstack.org/58844806:39
quiquellykarel: 2~^06:39
quiquellykarel: Maybe the Depends-On direction is the opposite ?06:40
quiquellI don't see scenario009 running06:40
ykarelquiquell, yup please reverse06:40
quiquellykarel: ack06:40
ykarelquiquell, can run in experimental though06:41
ykarelin tq i see scenario009 is in experimental pipeline06:41
quiquellykarel: But it's in check in other ones...06:42
quiquellykarel: Maybe that part is missing06:42
ykarelpossibly06:42
quiquellykarel: Going to add the Depends-On at TQ and put scenario009 at check06:42
ykarelquiquell, ack06:42
quiquellykarel: If there are changes at featureset03306:42
openstackgerritQuique Llorente proposed openstack/tripleo-heat-templates master: Replace openshift_node_labels with openshift_node_group_name  https://review.openstack.org/58815206:44
quiquellykarel: Nah, let's kee tq as it is, trown was already moving this thing to check06:45
quiquellykarel: At the end we are just testing if upgrade to openshift-ansible 3.10 is possible, is not let's unblock with your patch06:45
*** ccamacho has joined #tripleo06:45
quiquellykarel: I just want to know if we were breaking it with our new CI workflow06:45
ykarelquiquell, ack06:46
openstackgerritQuique Llorente proposed openstack/tripleo-quickstart master: Activate  centos-release-openshift-origin39-1-1.el7.centos  https://review.openstack.org/58844906:47
openstackgerritQuique Llorente proposed openstack/tripleo-heat-templates master: Replace openshift_node_labels with openshift_node_group_name  https://review.openstack.org/58815206:47
openstackgerritQuique Llorente proposed openstack/tripleo-heat-templates master: Replace openshift_node_labels with openshift_node_group_name  https://review.openstack.org/58815206:48
*** rcernin has quit IRC06:54
openstackgerritQuique Llorente proposed openstack-infra/tripleo-ci master: Replace TAGS with ansible var  https://review.openstack.org/58450806:54
jaosoriorbandini: could you also check these out https://review.openstack.org/#/c/588245/ https://review.openstack.org/#/c/588246/ ?06:54
*** shyambiradar has quit IRC06:54
openstackgerritCarlos Camacho proposed openstack/tripleo-upgrade stable/pike: Add the overcloud stack name when calling upgrade-non-controller.sh  https://review.openstack.org/58845006:56
*** shyambiradar has joined #tripleo06:58
ykarelshyambiradar, looks like u are mounting /etc in ur container06:59
shyambiradaryes06:59
ykarelif yes try mounting only the required /etc subfolders06:59
shyambiradarI am mounting06:59
shyambiradarI want /etc/nova07:00
shyambiradarok07:00
shyambiradarthanks07:00
openstackgerritQuique Llorente proposed openstack-infra/tripleo-ci master: Move featureset BASH variables to ansible  https://review.openstack.org/58734007:00
openstackgerritCarlos Camacho proposed openstack/tripleo-quickstart-extras master: Use the TripleO Ansible to run upgrades from tripleo-upgrade  https://review.openstack.org/58845107:01
shyambiradarykarel: working now07:02
openstackgerritQuique Llorente proposed openstack-infra/tripleo-ci master: Move featureset BASH variables to ansible  https://review.openstack.org/58734007:02
shyambiradarThanks a lot ykarel07:02
ykarelshyambiradar, cool07:03
openstackgerritQuique Llorente proposed openstack-infra/tripleo-ci master: Move featureset BASH variables to ansible  https://review.openstack.org/58734007:04
openstackgerritQuique Llorente proposed openstack-infra/tripleo-ci master: Cleanup variables and toci_jobtype leftovers  https://review.openstack.org/58446507:06
openstackgerritQuique Llorente proposed openstack-infra/tripleo-ci master: Replace TAGS with ansible var  https://review.openstack.org/58450807:06
openstackgerritQuique Llorente proposed openstack-infra/tripleo-ci master: Move featureset BASH variables to ansible  https://review.openstack.org/58734007:06
openstackgerritCarlos Camacho proposed openstack/tripleo-quickstart-extras master: Use the TripleO Ansible role from tripleo-upgrade to run upgrades  https://review.openstack.org/58845107:06
*** amoralej|off is now known as amoralej07:09
*** sai_p has quit IRC07:09
*** ooolpbot has joined #tripleo07:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION07:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178401707:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178482207:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178484707:10
*** ooolpbot has quit IRC07:10
openstackLaunchpad bug 1784017 in tripleo "Build of instance was re-scheduled: invalid argument: could not find capabilities for domaintype=kvm" [Critical,In progress] - Assigned to Rafael Folco (rafaelfolco)07:10
openstackLaunchpad bug 1784822 in tripleo "legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master fails contact ipmi" [Critical,Triaged]07:10
openstackLaunchpad bug 1784847 in tripleo "exception introspecting nodes" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)07:10
openstackgerritQuique Llorente proposed openstack-infra/tripleo-ci master: Move nodes_file and nodes_args to common.yaml  https://review.openstack.org/58571407:10
openstackgerritQuique Llorente proposed openstack-infra/tripleo-ci master: Move nodes_file and nodes_args to common.yaml  https://review.openstack.org/58571407:11
openstackgerritQuique Llorente proposed openstack-infra/tripleo-ci master: Refactor DRY_RUN, STATS_* and COLLECT_CONF vars  https://review.openstack.org/58734407:16
*** pmannidi has quit IRC07:16
openstackgerritCarlos Camacho proposed openstack/tripleo-quickstart-extras master: Use the TripleO Ansible role from tripleo-upgrade to run upgrades  https://review.openstack.org/58845107:18
openstackgerritCarlos Camacho proposed openstack/tripleo-quickstart-extras master: Use the TripleO Ansible role from tripleo-upgrade to run upgrades  https://review.openstack.org/58845107:22
openstackgerritQuique Llorente proposed openstack-infra/tripleo-ci master: Replace toci ENV_VARS by ansible env_vars  https://review.openstack.org/58736007:22
*** tosky has joined #tripleo07:26
quiquelljfrancoa, ccamacho: we can +1w this https://review.openstack.org/#/c/577146/07:26
jfrancoaquiquell: my first +1w, what a wonderful moment :-D07:27
quiquelljfrancoa: Hehe congrats :-)07:28
quiquelljfrancoa: Sure it breaks everything, I am the driver of your kaos07:28
openstackgerritArx Cruz proposed openstack/tripleo-quickstart-extras master: Cleaning up tempest skip tests for ocata  https://review.openstack.org/58845807:28
jfrancoaquiquell: thanks, as they were saying in Counter Strike: "The bomb is planted"07:28
*** jtomasek_ has quit IRC07:29
quiquelljfrancoa: Same change in TQ was working find, this will work too07:30
jfrancoaquiquell: yes, I saw that the other one was already merged and so far nothing got broken07:30
*** jtomasek has joined #tripleo07:31
Tengujfrancoa: hey congrats :).07:33
jfrancoaTengu: thanks man07:34
* Tengu opens some bottle of fresh beverage :)07:34
jfrancoaTengu: it's friday + my birthday, so let's open some rum instead :-D07:34
quiquelljfrancoa: Double congrats !!!07:34
Tengujfrancoa: ah, sorry, have only Whiskies in that kind of beverage :)07:34
*** shyambiradar has quit IRC07:44
*** mschuppert has joined #tripleo07:49
*** jpich has joined #tripleo07:53
openstackgerritRedHat RDO CI proposed openstack/tripleo-quickstart-extras master: GATE CHECK for quickstart-extras  https://review.openstack.org/56044508:00
openstackgerritCédric Jeanneret proposed openstack/python-tripleoclient master: Add some logging for the preflight checks  https://review.openstack.org/58628508:00
*** florianf has joined #tripleo08:01
*** dtantsur|afk is now known as dtantsur08:01
*** derekh has joined #tripleo08:03
*** holser_ has joined #tripleo08:08
*** ooolpbot has joined #tripleo08:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION08:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178401708:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178482208:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178484708:10
*** ooolpbot has quit IRC08:10
openstackLaunchpad bug 1784017 in tripleo "Build of instance was re-scheduled: invalid argument: could not find capabilities for domaintype=kvm" [Critical,In progress] - Assigned to Rafael Folco (rafaelfolco)08:10
openstackLaunchpad bug 1784822 in tripleo "legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master fails contact ipmi" [Critical,Triaged]08:10
openstackLaunchpad bug 1784847 in tripleo "exception introspecting nodes" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)08:10
*** lebauce has quit IRC08:11
*** lebauce has joined #tripleo08:12
*** saneax has joined #tripleo08:12
*** jpena|off is now known as jpena08:20
*** leseb_ has quit IRC08:25
quiquellykarel: This could be related to the openshift change ?08:26
quiquellykarel: http://logs.openstack.org/52/588152/8/check/tripleo-ci-centos-7-scenario009-multinode-oooq/eca355d/logs/undercloud/home/zuul/overcloud_prep_containers.log.txt.gz08:26
Tenguflorianf: heya! care to give a (hopefully) last review on https://review.openstack.org/#/c/583475/ ? I added a work regarding the default "running engine" for the validation that should meet the results of our discussions on that topic :).08:28
openstackgerritTom Barron proposed openstack/tripleo-heat-templates stable/queens: Fix manila update, upgrade, and FFU tasks  https://review.openstack.org/58798208:29
Tenguflorianf: if you're OK with that, you might even give the final w+1 (or "call a friend" for review if you feel something still needs some wording/work) :)08:29
ykarelquiquell, don't think so, should be unrelated08:29
openstackgerritQuique Llorente proposed openstack/tripleo-heat-templates master: Replace openshift_node_labels with openshift_node_group_name  https://review.openstack.org/58815208:30
*** shyambiradar has joined #tripleo08:38
*** leanderthal has joined #tripleo08:40
florianfTengu: Yup, I'll have a look!08:41
Tenguflorianf: thank you :)08:41
Tenguhey guys, PTAL? https://review.openstack.org/586541  that would be really nice of you :).08:45
*** salmankhan has joined #tripleo08:48
openstackgerritQuique Llorente proposed openstack-infra/tripleo-ci master: Add a script to compare playbook executions  https://review.openstack.org/58847508:50
quiquellssbarnea: ^08:50
*** leanderthal has quit IRC08:51
openstackgerritQuique Llorente proposed openstack-infra/tripleo-ci master: Add a script to compare playbook executions  https://review.openstack.org/58847508:55
*** ykarel has quit IRC08:57
openstackgerritQuique Llorente proposed openstack-infra/tripleo-ci master: Cleanup variables and toci_jobtype leftovers  https://review.openstack.org/58446508:59
ssbarneatesting the script now09:00
*** jaosorior has quit IRC09:03
quiquellssbarnea: ack09:03
quiquellssbarnea: leftovers is ok , but tags review is not09:03
quiquellssbarnea: Adding some diff mechanism to it09:04
*** leanderthal has joined #tripleo09:05
openstackgerritYurii Prokulevych proposed openstack/tripleo-upgrade stable/pike: Don't invoke multiple migration requests for single vm.  https://review.openstack.org/58847909:05
*** ooolpbot has joined #tripleo09:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION09:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178401709:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178482209:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178484709:10
*** ooolpbot has quit IRC09:10
openstackLaunchpad bug 1784017 in tripleo "Build of instance was re-scheduled: invalid argument: could not find capabilities for domaintype=kvm" [Critical,In progress] - Assigned to Rafael Folco (rafaelfolco)09:10
openstackLaunchpad bug 1784822 in tripleo "legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master fails contact ipmi" [Critical,Triaged]09:10
openstackLaunchpad bug 1784847 in tripleo "exception introspecting nodes" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)09:10
*** gfidente has joined #tripleo09:13
shyambiradarHi, I want to mount /etc/nova from host to container09:16
shyambiradarBut I am not able to access it after mount09:16
shyambiradarbecause of permissions09:16
shyambiradarHow can I handle this scenario?09:17
openstackgerritQuique Llorente proposed openstack-infra/tripleo-ci master: Add a script to compare playbook executions  https://review.openstack.org/58847509:17
openstackgerritSorin Sbarnea proposed openstack-infra/tripleo-ci master: Add a script to compare playbook executions  https://review.openstack.org/58847509:18
shyambiradarAny thought?09:18
ssbarneaquiquell: i made a small improvement on the diff script output to make it easier to read.09:19
quiquellssbarnea: Cool, upload the change09:20
ssbarneaalredy did09:20
ssbarneai was ready to add colors but decided to avoid over engineering from its first version09:21
quiquellssbarnea: Let's find the issues with tags patch09:22
quiquellssbarnea: Going to remove the list from the tripleo card09:22
quiquellssbarnea: Removed the string diff ?09:25
ssbarneanot on putpose, my change was on previous changeset, going to refresh it09:27
quiquellssbarnea: ok09:27
quiquellssbarnea: humm didn't ask you to merge or something ?09:28
quiquellThat's weird09:28
quiquellssbarnea: I mean git review09:28
ssbarneaindeed, i was wondering the same thing,... nothing09:28
quiquellssbarnea: ok np, just re-merge your stuff so we also get the differ thing09:28
quiquellssbarnea: It's broken with your changes, it's comparing only the keys not the conent09:30
quiquellcontent09:30
quiquellssbarnea: going to revert them09:30
ssbarneaahh...09:30
ssbarneaanyway, your pprint use probably fixed the problem,09:30
ssbarneai think i know what happend, there is only one minute between my change and yours, we probably found a bug in gerrit or git-review.09:32
quiquellssbarnea: Ahh yep have to be, ok no problem let's just use the differ09:32
quiquellssbarnea: Going to update with it,09:32
*** zoli is now known as zoli|lunch09:32
quiquellssbarnea: thanks09:33
quiquellupdated09:33
openstackgerritQuique Llorente proposed openstack-infra/tripleo-ci master: Add a script to compare playbook executions  https://review.openstack.org/58847509:33
*** shyambiradar has quit IRC09:35
quiquellssbarnea: Do you want a bj session to explain it to you ?09:35
*** shyambiradar has joined #tripleo09:36
ssbarneafirst let me look at a diff, give me two numbers to check if you can09:36
openstackgerritGabriele Cerami proposed openstack-infra/tripleo-ci master: Te-broker: save all nodes console to html log dir  https://review.openstack.org/58848809:37
openstackgerritCarlos Camacho proposed openstack/tripleo-common stable/queens: Include the stack name when generating the tripleo-ansible-inventory  https://review.openstack.org/58848909:38
shyambiradarHi09:38
shyambiradaropenstack nova compute container is creating 'nova' user with a different it than 16209:39
shyambiradarAny idea?09:39
shyambiradarI was expecting e created with 16209:40
openstackgerritDmitry Tantsur proposed openstack/tripleo-common master: Workflow for provisioning bare metals without Nova  https://review.openstack.org/57685609:40
openstackgerritQuique Llorente proposed openstack-infra/tripleo-ci master: Add a script to compare playbook executions  https://review.openstack.org/58847509:45
*** ykarel has joined #tripleo09:45
*** shyambiradar has quit IRC09:46
ykarelquiquell, just in case u have not seen openshift 3.10 in testing repo: https://lists.centos.org/pipermail/centos-devel/2018-August/016857.html09:47
*** ykarel is now known as ykarel|travel09:48
quiquellykarel|travel: Suppose we don't want to use testing repo09:49
openstackgerritDmitry Tantsur proposed openstack/tripleo-common master: Workflow for provisioning bare metals without Nova  https://review.openstack.org/57685609:49
openstackgerritQuique Llorente proposed openstack-infra/tripleo-ci master: Add a script to compare playbook executions  https://review.openstack.org/58847509:50
*** ykarel|travel has quit IRC09:52
openstackgerritQuique Llorente proposed openstack-infra/tripleo-ci master: Add a script to compare playbook executions  https://review.openstack.org/58847509:52
openstackgerritDmitry Tantsur proposed openstack/tripleo-common master: Switch away from using token_endpoint keystoneauth plugin  https://review.openstack.org/58576509:56
openstackgerritDmitry Tantsur proposed openstack/tripleo-common master: [WIP] Collect required information for deploying without Nova  https://review.openstack.org/58627909:56
openstackgerritQuique Llorente proposed openstack-infra/tripleo-ci master: Replace TAGS with ansible var  https://review.openstack.org/58450809:59
*** agopi has quit IRC10:08
*** ooolpbot has joined #tripleo10:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION10:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178401710:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178482210:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178484710:10
*** ooolpbot has quit IRC10:10
openstackLaunchpad bug 1784017 in tripleo "Build of instance was re-scheduled: invalid argument: could not find capabilities for domaintype=kvm" [Critical,In progress] - Assigned to Rafael Folco (rafaelfolco)10:10
openstackLaunchpad bug 1784822 in tripleo "legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master fails contact ipmi" [Critical,Triaged]10:10
openstackLaunchpad bug 1784847 in tripleo "exception introspecting nodes" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)10:10
panda|roverI need some nova expert10:10
panda|rover!summon nova-expert10:10
openstackpanda|rover: Error: "summon" is not a valid command.10:10
openstackgerritCarlos Camacho proposed openstack/tripleo-common stable/pike: Include the stack name when generating the tripleo-ansible-inventory  https://review.openstack.org/58850610:34
*** zoli|lunch is now known as zoli10:50
openstackgerritSorin Sbarnea proposed openstack-infra/tripleo-ci master: Add a script to compare playbook executions  https://review.openstack.org/58847510:52
openstackgerritQuique Llorente proposed openstack-infra/tripleo-ci master: Move nodes_file and nodes_args to common.yaml  https://review.openstack.org/58571410:52
openstackgerritQuique Llorente proposed openstack-infra/tripleo-ci master: Replace toci ENV_VARS by ansible env_vars  https://review.openstack.org/58736010:52
openstackgerritQuique Llorente proposed openstack-infra/tripleo-ci master: Refactor DRY_RUN, STATS_* and COLLECT_CONF vars  https://review.openstack.org/58734410:53
*** shyambiradar has joined #tripleo10:55
openstackgerritJose Luis Franco proposed openstack/python-tripleoclient master: Handle negative/wrong answer in question when upgrading undercloud.  https://review.openstack.org/58624911:00
openstackgerritCarlos Camacho proposed openstack/tripleo-quickstart-extras master: Use the TripleO Ansible role from tripleo-upgrade to run upgrades  https://review.openstack.org/58845111:00
*** dtrainor has joined #tripleo11:02
*** jpena is now known as jpena|lunch11:03
*** ooolpbot has joined #tripleo11:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION11:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178401711:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178482211:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178484711:10
*** ooolpbot has quit IRC11:10
openstackLaunchpad bug 1784017 in tripleo "Build of instance was re-scheduled: invalid argument: could not find capabilities for domaintype=kvm" [Critical,In progress] - Assigned to Rafael Folco (rafaelfolco)11:10
openstackLaunchpad bug 1784822 in tripleo "legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master fails contact ipmi" [Critical,Triaged]11:10
openstackLaunchpad bug 1784847 in tripleo "exception introspecting nodes" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)11:10
openstackgerritGabriele Cerami proposed openstack/tripleo-quickstart-extras master: Collect logs: get the node console logs  https://review.openstack.org/58851611:10
openstackgerritDmitry Tantsur proposed openstack/tripleo-common master: Workflow for provisioning bare metals without Nova  https://review.openstack.org/57685611:11
openstackgerritQuique Llorente proposed openstack-infra/tripleo-ci master: Replace toci ENV_VARS by ansible env_vars  https://review.openstack.org/58736011:13
*** shyambiradar has quit IRC11:15
*** shyambiradar has joined #tripleo11:15
openstackgerritQuique Llorente proposed openstack-infra/tripleo-ci master: Move nodes_file and nodes_args to common.yaml  https://review.openstack.org/58571411:15
openstackgerritQuique Llorente proposed openstack-infra/tripleo-ci master: Move featureset BASH variables to ansible  https://review.openstack.org/58734011:19
*** rh-jelabarre has joined #tripleo11:26
Tenguheya! on the RDO CI, I have an issue apparently unrelated to my patch: http://logs.rdoproject.org/85/586285/4/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/b4f057b/job-output.txt.gz - specifically: Could not find a version that satisfies the requirement container-check (from versions: )\nNo matching distribution found for container-check11:27
Tenguis it a known issue?11:27
openstackgerritMerged openstack/python-tripleoclient master: Remove usage of resource registry for undercloud TLS  https://review.openstack.org/58815111:28
openstackgerritMerged openstack/tripleo-upgrade stable/pike: Add the overcloud stack name when calling upgrade-non-controller.sh  https://review.openstack.org/58845011:28
openstackgerritMerged openstack/tripleo-quickstart-extras master: Fix centos opstools repo deactivation  https://review.openstack.org/57714611:29
*** lebauce has quit IRC11:30
therveEmilienM: Do we have specific containers for the undercloud or are they the same as the overcloud btw?11:30
openstackgerritGabriele Cerami proposed openstack-infra/tripleo-ci master: [DNM][TEST] dummy change for OVB tests iteration  https://review.openstack.org/58851711:31
*** shyambiradar has quit IRC11:34
*** shyambiradar has joined #tripleo11:35
openstackgerritDerek Higgins proposed openstack/tripleo-heat-templates master: [WIP] Add scenario 012 - overlcoud baremetal+ansible-ml2  https://review.openstack.org/57960311:38
openstackgerritDerek Higgins proposed openstack/tripleo-quickstart master: [WIP] Add featureset 054 - overcloud baremetal+ansible-ml2  https://review.openstack.org/57960111:40
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci master: [WIP] Testing a ansible-ml2 job  https://review.openstack.org/58229411:42
*** ansmith has quit IRC11:45
openstackgerritCarlos Camacho proposed openstack/tripleo-quickstart-extras master: Use the TripleO Ansible role from tripleo-upgrade to run upgrades  https://review.openstack.org/58845111:45
EmilienMtherve: same11:47
therveEmilienM: OK. So it's not possible here11:47
therveEmilienM: What are our options?11:47
*** thrash|g0ne is now known as thrash11:50
*** abishop has joined #tripleo11:52
*** karthiks has joined #tripleo11:53
openstackgerritQuique Llorente proposed openstack-infra/tripleo-ci master: [WIP] Run emit_releases_file.py for all featuresets  https://review.openstack.org/58564911:57
*** jpena|lunch is now known as jpena11:57
*** zul has joined #tripleo12:01
chandankumarmarios: Hey12:01
chandankumarmandre: please have a look at this patch https://review.openstack.org/#/c/584368/1212:02
openstackgerritSorin Sbarnea proposed openstack-infra/tripleo-ci master: Add a script to compare playbook executions  https://review.openstack.org/58847512:02
chandankumarmarios: ^^12:02
*** ratailor has quit IRC12:08
*** shyambiradar has quit IRC12:10
*** ooolpbot has joined #tripleo12:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION12:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178401712:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178482212:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178484712:10
*** ooolpbot has quit IRC12:10
openstackLaunchpad bug 1784017 in tripleo "Build of instance was re-scheduled: invalid argument: could not find capabilities for domaintype=kvm" [Critical,In progress] - Assigned to Rafael Folco (rafaelfolco)12:10
openstackLaunchpad bug 1784822 in tripleo "legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master fails contact ipmi" [Critical,Triaged]12:10
openstackLaunchpad bug 1784847 in tripleo "exception introspecting nodes" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)12:10
mschupperthi, do we have any update on the ipmi issue reported in https://bugs.launchpad.net/tripleo/+bug/1784822 ?12:11
EmilienMtherve: so it's needed only in the heat containers on the undercloud right?12:16
therveEmilienM: yes12:16
EmilienMtherve: is that terrible to install these plugins on the overcloud too?12:16
therveEmilienM: Yes12:16
*** panda|rover is now known as panda|rover|off12:17
openstackgerritChandan Kumar proposed openstack/tripleo-quickstart-extras master: Enable support for running refstack tests in TQE  https://review.openstack.org/57071912:17
therveIt changes basic behavior for server for example12:17
EmilienMmhh12:17
EmilienMin that case I see no other option than creating specific heat containers, that we'll use for the undercloud12:17
EmilienMbut;.. beurk12:17
openstackgerritSorin Sbarnea proposed openstack-infra/tripleo-ci master: Add a script to compare playbook executions  https://review.openstack.org/58847512:18
therveEmilienM: We can do a mount otherwise12:19
*** rbrady-afk is now known as rbrady12:19
therveIe share /usr/lib/heat with the heat-engine container12:20
EmilienMyeah we could do that indeed12:20
EmilienMtherve: with a conditional12:20
EmilienMmount if HeatUndercloud is true12:20
EmilienMfalse by default and true in environment/undercloud.yaml12:20
EmilienMfor ex.12:20
therveYeah12:21
therveEmilienM: So just add a volume conditionally?12:23
EmilienMtherve: yeah12:25
openstackgerritMarios Andreou proposed openstack/tripleo-quickstart master: DNM: testing  https://review.openstack.org/58852712:25
EmilienMI think it's easier and it should work12:25
therveOK I think I see how to do that12:25
openstackgerritQuique Llorente proposed openstack-infra/tripleo-ci master: Add logic/tests for n->n+1 upgrades  https://review.openstack.org/57242012:25
openstackgerritQuique Llorente proposed openstack-infra/tripleo-ci master: Add fs050 job to test from-release-upgrades  https://review.openstack.org/57466512:25
openstackgerritChandan Kumar proposed openstack/tripleo-quickstart-extras master: Use local docker registry host for tempest container  https://review.openstack.org/58436812:25
openstackgerritThomas Herve proposed openstack/tripleo-heat-templates master: Mount /usr/lib/heat on undercloud Heat  https://review.openstack.org/58852912:27
EmilienMtherve: even better :P12:28
EmilienM++ I think it's good12:29
therveIt's a bit cheating but easy at least :)12:30
*** toure|gone is now known as toure12:35
openstackgerritQuique Llorente proposed openstack-infra/tripleo-ci master: Add a script to compare playbook executions  https://review.openstack.org/58847512:39
*** rlandy has joined #tripleo12:41
*** skramaja has quit IRC12:48
openstackgerritQuique Llorente proposed openstack-infra/tripleo-ci master: Add a script to compare playbook executions  https://review.openstack.org/58847512:51
openstackgerritMarios Andreou proposed openstack-infra/tripleo-ci master: Remove tripleo.sh --bootstrap-subnodes add ceph loop device  https://review.openstack.org/58319512:52
*** rfolco|off is now known as rfolco|ruck12:52
*** eck`gone is now known as eck`12:54
*** saneax has quit IRC12:55
openstackgerritQuique Llorente proposed openstack-infra/tripleo-ci master: Add a script to compare playbook executions  https://review.openstack.org/58847512:58
Tenguyprokule: heya! could you give a try to jaosorior patch regarding your upgrade issue with the "enable-tls.yaml" ?12:59
*** edmondsw_ has joined #tripleo13:00
yprokuleTengu: this one worked for me https://review.openstack.org/#/c/588151/13:01
Tenguyprokule: great! sooo, not sure we can close the issue already, or wait for a package to be produced with the fix?13:01
yprokuleTengu: yes, we need that in package13:02
Tenguyprokule: need to import it downstream right? will try to see to that.13:04
yprokuleTengu: jschlueter might help with that13:04
Tenguyprokule: I think I have all what I need :). I'll call for help if needed, but it should be ok.13:06
*** leanderthal has quit IRC13:06
openstackgerritQuique Llorente proposed openstack-infra/tripleo-ci master: Replace TAGS with ansible var  https://review.openstack.org/58450813:07
*** amoralej is now known as amoralej|lunch13:07
openstackgerritAlex Schultz proposed openstack/puppet-pacemaker master: Initial support for adding cluster nodes to an existing cluster  https://review.openstack.org/58515013:09
*** ooolpbot has joined #tripleo13:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION13:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178401713:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178482213:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178484713:10
*** ooolpbot has quit IRC13:10
openstackLaunchpad bug 1784017 in tripleo "Build of instance was re-scheduled: invalid argument: could not find capabilities for domaintype=kvm" [Critical,In progress] - Assigned to Rafael Folco (rafaelfolco)13:10
openstackLaunchpad bug 1784822 in tripleo "legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master fails contact ipmi" [Critical,Triaged]13:10
openstackLaunchpad bug 1784847 in tripleo "exception introspecting nodes" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)13:10
*** agopi has joined #tripleo13:10
*** agopi_ has joined #tripleo13:11
*** gbarros has joined #tripleo13:12
*** ansmith has joined #tripleo13:12
*** quiquell is now known as quiquell|off13:13
*** tzumainn has joined #tripleo13:13
*** agopi has quit IRC13:14
*** agopi_ is now known as agopi13:14
*** quiquell|off has quit IRC13:17
openstackgerritRaoul Scarazzini proposed openstack/tripleo-quickstart-extras master: Integrate Validate-HA work into OVB jobs  https://review.openstack.org/53766913:19
jschlueteryprokule, Tengu: what's up?13:20
Tengujschlueter: nothing, thanks! was about to check backporting an upstream patch downstream, but after some discussions it has to wait - plus, I shouldn't be the one doing that anyway :).13:20
Tengujschlueter: but I have the doc for that kind of task, so I shouldn't get any issue when time comes for me to actually *do* a downstream import :].13:21
mwhahahas/import/cherry-pick13:21
Tenguwell, yes. sorry13:21
openstackgerritDamien Ciabrini proposed openstack/tripleo-heat-templates master: Revert shebang change for InstanceHA startup script  https://review.openstack.org/58855213:21
*** ramishra has quit IRC13:22
* jschlueter nods13:22
mwhahahaweshay: i just saw a patch with full ovb pass from 2am, is it fixed? or was the patch just lucky13:22
weshaymwhahaha, lucky13:23
mwhahahabooo13:23
Tengu:]13:23
EmilienMhas anyone testing the https://github.com/openstack/ansible-role-tripleo-modify-image repo outside of CI?13:23
mwhahahaweshay: did we poke the rdo cloud admins to see if they did any upgrades recently?13:23
Tenguguess it was my patch13:23
EmilienMit doesn't work for me, https://github.com/openstack/ansible-role-tripleo-modify-image13:24
EmilienMhttps://bugs.launchpad.net/tripleo/+bug/178524313:24
openstackLaunchpad bug 1785243 in tripleo "modify-image role: variable error when getting "user" from container config" [Medium,Triaged] - Assigned to Emilien Macchi (emilienm)13:24
weshaymwhahaha, working multiple fronts now atm13:24
mwhahahak13:24
*** jcoufal has joined #tripleo13:24
Tengumwhahaha: you refer to that kind of job? http://logs.rdoproject.org/85/586285/4/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/b4f057b/job-output.txt.gz  - apparently the issue is related to some pip package.13:25
*** agopi_ has joined #tripleo13:25
mwhahahaTengu: no there's an issue with pxe boot13:25
Tenguah, so that's another one :]13:26
mwhahahathough that's an interesting failure13:26
mwhahahalooks like scenario001 in queens might be broken, seeing some repeated TIMED_OUTs13:27
Tengumwhahaha: a recheck is running on the RDO, maybe it was just temporary.13:27
*** agopi has quit IRC13:28
*** agopi_ is now known as agopi13:31
*** psachin has quit IRC13:33
openstackgerritMerged openstack/tripleo-heat-templates master: Update and upgrade tasks for services deployed via external deploy tasks  https://review.openstack.org/57917013:35
openstackgerritMerged openstack/python-tripleoclient stable/pike: [Pike only] use --plan instead of --stack for tripleo-ansible-inventory  https://review.openstack.org/58618413:35
*** ansmith has quit IRC13:40
*** gbarros has quit IRC13:42
*** ansmith has joined #tripleo13:44
weshaymwhahaha, we've moved from 70% fail rate to 43% :/13:47
weshayotp w/ ironic folks atm13:47
mwhahahaprogress? :D13:47
weshaypinche boots13:47
*** jrist has joined #tripleo13:48
jristweird13:49
*** bnemec is now known as beekneemech13:51
*** gbarros has joined #tripleo13:53
openstackgerritJon Schlueter proposed openstack/tripleo-common stable/queens: kolla overrides: Remove yum cache  https://review.openstack.org/58857914:00
*** saneax has joined #tripleo14:04
*** edmondsw_ is now known as edmondsw14:08
*** ooolpbot has joined #tripleo14:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION14:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178401714:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178482214:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178484714:10
*** ooolpbot has quit IRC14:10
openstackLaunchpad bug 1784017 in tripleo "Build of instance was re-scheduled: invalid argument: could not find capabilities for domaintype=kvm" [Critical,In progress] - Assigned to Rafael Folco (rafaelfolco)14:10
*** amoralej|lunch is now known as amoralej14:10
openstackLaunchpad bug 1784822 in tripleo "legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master fails contact ipmi" [Critical,Triaged]14:10
openstackLaunchpad bug 1784847 in tripleo "exception introspecting nodes" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)14:10
openstackgerritJose Luis Franco proposed openstack/python-tripleoclient master: Handle negative/wrong answer in question when upgrading undercloud.  https://review.openstack.org/58624914:13
gfidentemwhahaha maybe you have a link to that patch which removed the osp/ceph packages from the overcloud baremetal image?14:15
gfidenteI am struggling to find it, I remember you were on review14:15
mwhahahayea i wrote it14:16
mwhahahasec14:16
mwhahahawe merged it long ago14:16
mwhahahagfidente: https://review.openstack.org/#/c/563608/14:16
mwhahahagfidente: specifically the work was done via https://review.openstack.org/#/q/status:merged+project:openstack/tripleo-puppet-elements+branch:master+topic:bp/cleanup-overcloud-images14:17
gfidente"i wrote it"14:17
gfidente:D14:17
gfidentemwhahaha++14:17
*** gbarros has quit IRC14:27
*** artom has joined #tripleo14:28
*** janki has quit IRC14:28
*** gbarros has joined #tripleo14:28
openstackgerritToure Dunnon proposed openstack/tripleo-common master: Improve output when registering nodes.  https://review.openstack.org/56981114:32
openstackgerritMerged openstack/tripleo-heat-templates master: fix manila scheduler service typo  https://review.openstack.org/58840714:35
*** jcoufal_ has joined #tripleo14:36
*** jcoufal has quit IRC14:37
*** dxiri has joined #tripleo14:38
openstackgerritSorin Sbarnea proposed openstack-infra/tripleo-ci master: fix tox python3 overrides  https://review.openstack.org/58858714:43
openstackgerritToure Dunnon proposed openstack/tripleo-common master: Clean up node registration output.  https://review.openstack.org/56981114:44
*** janki has joined #tripleo14:51
weshaymwhahaha, rlandy bfournie they are going to reboot a few nodes in rdo.. my stack failed.. holding reattempts for the reboots14:51
openstackgerritSorin Sbarnea proposed openstack-infra/tripleo-ci master: fix pep8 check and made it generic  https://review.openstack.org/58858814:52
openstackgerritMerged openstack/puppet-pacemaker master: Initial support for adding cluster nodes to an existing cluster  https://review.openstack.org/58515014:53
openstackgerritSorin Sbarnea proposed openstack-infra/tripleo-ci master: fix pep8 check and made it generic  https://review.openstack.org/58858814:54
*** rpioso|afk is now known as rpioso14:55
*** dxiri has quit IRC14:58
*** abays has joined #tripleo15:05
openstackgerritMerged openstack/tripleo-common master: Add python-novajoin to nova-api packages  https://review.openstack.org/58813615:08
*** toure has quit IRC15:09
*** ooolpbot has joined #tripleo15:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION15:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178401715:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178482215:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178484715:10
*** ooolpbot has quit IRC15:10
openstackLaunchpad bug 1784017 in tripleo "Build of instance was re-scheduled: invalid argument: could not find capabilities for domaintype=kvm" [Critical,In progress] - Assigned to Rafael Folco (rafaelfolco)15:10
openstackLaunchpad bug 1784822 in tripleo "legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master fails contact ipmi" [Critical,Triaged]15:10
openstackLaunchpad bug 1784847 in tripleo "exception introspecting nodes" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)15:10
*** toure has joined #tripleo15:14
*** jcoufal has joined #tripleo15:15
*** jcoufal_ has quit IRC15:16
openstackgerritCarlos Goncalves proposed openstack/diskimage-builder master: Add netcat to redhat-common map-packages  https://review.openstack.org/58860115:20
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates master: Enable collectd to connect to metrics QDR  https://review.openstack.org/57605715:21
abaysgfidente: Would you have a moment to help with an OSP-12 ceph-ansible deployment issue?15:21
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates master: Enable collectd to connect to metrics QDR  https://review.openstack.org/57605715:23
gfidenteabays let's try but mileage may vary :D15:23
abaysgfidente: Cool, thank you!  So I'm working with someone and they're trying to deploy a cloud with HCI compute nodes15:25
abaysWe keep hitting this error at step 2 of the deployment...15:25
abayshttp://paste.openstack.org/show/727239/15:25
gfidenteabays so is this pike, queens or master?15:26
abaysPike15:26
abaysHe has ceph-ansible installed and we can see the playbooks, etc, in /usr/share/ceph-ansible15:28
*** janki has quit IRC15:29
Tenguhave a great weekend ppl :).15:29
abaysHe had 3.0.27 installed and we upgraded to 3.0.39 hoping that would help, but it didn't15:29
*** zoli is now known as zoli|gone15:38
*** zoli|gone is now known as zoli15:38
openstackgerritAlex Schultz proposed openstack/python-tripleoclient master: Remove instack-undercloud and --use-heat  https://review.openstack.org/58860615:40
openstackgerritMarios Andreou proposed openstack/tripleo-quickstart master: Install openstack-heat-agents for Pike subnodes  https://review.openstack.org/58852715:43
*** itlinux_ has quit IRC15:43
gfidenteabays 3.0.27 should be fine15:46
abaysgfidente: We just noticed that his /etc/ansible/ansible.cfg exists but is completely empty15:46
gfidenteabays I don't think this is the cause of the failure15:46
gfidentedo you see heat failing at workflow_step2 ?15:46
abaysYes15:47
gfidenteabays there should be in /var/log/mistral a log file for the ceph-ansible run15:47
abaysIn executor.log ?15:47
gfidenteno the file is called ceph-ansible-workflow.log15:47
*** yprokule has quit IRC15:47
gfidenteif you don't have it, then mistral didn't get to launching ceph-ansible even15:47
rlandybfournie: weshay:  I managed to reproduce the IPMI failure once - although I probably hit the controller reboot https://paste.fedoraproject.org/paste/1YLCgAWLcG4dvW~oDLfOkg15:48
abaysgfidente: Gotcha -- it's not there, so ceph-ansible didn't run15:48
*** itlinux has joined #tripleo15:49
gfidenteabays it's probably failing at this stage https://github.com/openstack/tripleo-common/blob/stable/pike/workbooks/ceph-ansible.yaml#L68-L8715:49
rlandybfournie: weshay:  I ran again - and now introspection failed on the second and third nodes15:49
rlandyIntrospection of node b9718a5f-5c3c-438a-b502-49decf93f84d timed out.15:49
rlandyIntrospection of node cf2e79cd-0366-44f7-9340-edd7793cb611 timed out.15:49
rlandyRetrying 2 nodes that failed introspection. Attempt 1 of 315:49
gfidenteabays can you see if you spot anything about that task in the mistral logs?15:50
rlandyweshay: bfournie: bmc log from timed out introspection: https://paste.fedoraproject.org/paste/JVbwqTNM9mkBj5YcGyQ30Q15:50
abaysgfidente: Taking a look now15:51
bfourniederekh:  ^15:53
*** abays has quit IRC15:54
openstackgerritMerged openstack/tripleo-heat-templates stable/pike: Clean up env files for Cinder's Netapp backend  https://review.openstack.org/58365015:55
openstackgerritAlex Schultz proposed openstack/tripleo-quickstart-extras master: Add pauch to the includepkgs  https://review.openstack.org/57922315:55
openstackgerritEmilien Macchi proposed openstack/ansible-role-tripleo-modify-image master: Add Buildah support  https://review.openstack.org/58839015:55
*** abays has joined #tripleo15:57
*** ccamacho has quit IRC15:57
abaysgfidente: There's really nothing in the logs concerning "collect_nodes_uuid" other than some debug about validating the expression15:57
bfournierlandy: thanks, can you update bug with that stack trace?15:57
bfournierlandy: are there no libvirt logs?15:58
rlandyI'll look15:58
bfournierlandy: thanks, also anything interesting in nova api log?15:58
rlandybfournie: I used the same setup as before - you can access it if you wish15:59
bfournierlandy: k, will do15:59
rlandyI'll gather what I can15:59
openstackgerritRedHat RDO CI proposed openstack/tripleo-quickstart-extras master: GATE CHECK for quickstart-extras  https://review.openstack.org/56044516:00
openstackgerritRedHat RDO CI proposed openstack/tripleo-heat-templates stable/pike: GATE CHECK for TripleO  https://review.openstack.org/56428516:00
gfidenteabays no ansible output?16:00
*** jfrancoa has quit IRC16:00
abaysgfidente: Just one line about validating the expression16:01
abaysMeaning, I guess, that it's validating the task syntax?16:01
abaysThat was a debug with no error16:01
gfidenteabays judging from you stacktrace16:03
gfidenteamend what I said above I think it's failing in an earlier step even16:03
derekhrlandy: weshay bfournie [84315.724241] openstackbmc[2567]: ClientException: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.16:03
gfidentehttps://github.com/openstack/tripleo-common/blob/stable/pike/workbooks/access.yaml#L14516:03
derekhrlandy: weshay bfournie this is a api error talking to the cloud (rdo cloud?)16:04
-openstackstatus- NOTICE: The infra team is renaming projects in Gerrit. There will be a short ~10 minute Gerrit downtime in a few minutes as a result.16:04
derekhlooks like a problem with the stability of the API on rdo-cloud16:04
gfidentehttps://github.com/openstack/tripleo-common/blob/stable/pike/workbooks/access.yaml#L14016:04
rlandyderekh: yeah - I think hit the rdocloud reboot on that run16:05
gfidenteabays so yeah that is using the default ansible.cfg16:05
abaysgfidente: Okay16:06
abaysSo a blank file would be bad news?16:06
derekhrlandy: ok, can we try again for logs without a reboot as the reboot will cause all kinds of problems16:06
gfidentehttps://github.com/openstack/tripleo-common/blob/stable/pike/tripleo_common/actions/ansible.py#L31-L4416:06
*** rlandy is now known as rlandy|mtg16:07
gfidenteabays yeah looks like it expects defaults section to exist16:07
gfidentehttps://github.com/openstack/tripleo-common/blob/stable/pike/tripleo_common/actions/ansible.py#L3916:07
* abays nods at gfidente 16:07
gfidentenot sure why it's empty there but16:08
gfidenteyes I'd try [defaults]16:08
abaysgfidente: Thank you.  We will get valid cfg content in there and try the deployment again16:09
gfidenteabays cool also keep 0.2716:09
*** ccamacho has joined #tripleo16:09
abaysgfidente: Okay16:09
*** ooolpbot has joined #tripleo16:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION16:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178401716:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178482216:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178484716:10
*** ooolpbot has quit IRC16:10
openstackLaunchpad bug 1784017 in tripleo "Build of instance was re-scheduled: invalid argument: could not find capabilities for domaintype=kvm" [Critical,In progress] - Assigned to Rafael Folco (rafaelfolco)16:10
openstackLaunchpad bug 1784822 in tripleo "legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master fails contact ipmi" [Critical,Triaged]16:10
openstackLaunchpad bug 1784847 in tripleo "exception introspecting nodes" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)16:10
*** holser_ has quit IRC16:10
*** melwitt is now known as jgwentworth16:14
*** holser_ has joined #tripleo16:15
*** gbarros has quit IRC16:16
*** gbarros has joined #tripleo16:18
*** udesale has quit IRC16:20
weshayderekh, do you have the log?16:20
derekhweshay: which log? rlandy|mtg posted one a while back from a bmc https://paste.fedoraproject.org/paste/JVbwqTNM9mkBj5YcGyQ30Q16:21
weshayderekh, rlandy|mtg bfournie and I have been chatting about it. I'm in an out of 1-1's atm.. probably a little out of step w/ rlandy|mtg16:21
derekhweshay: rlandy|mtg thinks that might have been during a reboot16:22
rlandy|mtgweshay: derekh: correct - I caught the reboot - I mentioned that above16:22
weshayah k16:22
weshayfunny how ha doesn't work sometimes16:22
* weshay runs16:22
rlandy|mtgI can get diff results each time I run :)16:23
weshaylolz16:23
derekhweshay: First place I would start is to find a ipmi error and see if that corresponds back to a error on the BMC16:23
weshayk16:23
weshayrlandy|mtg, I think panda|rover|off was adding auth to the bmc node so we could collect logs from it16:23
weshayas derekh makes a good point16:24
derekhweshay: if the rdo-cloud api isn't stable then the reality is there isn't much we can do except wait until it is16:24
rlandy|mtgweshay: sorry - just info transferring to myoung atm - re: zuulv316:24
rlandy|mtgwill get back to this16:24
*** gbarros has quit IRC16:25
*** boazel has joined #tripleo16:26
*** gbarros has joined #tripleo16:28
*** psachin has joined #tripleo16:30
boazelwii boazel16:30
*** jpich has quit IRC16:32
*** holser_ has quit IRC16:32
weshaymwhahaha, to review what deployed-server should or should not do for pike16:33
weshayhttps://bugs.launchpad.net/tripleo/+bug/178506716:33
openstackLaunchpad bug 1785067 in tripleo "without bootstrap-subnodes pike deployment fails for missing os-collect-config" [High,In progress] - Assigned to Marios Andreou (marios-b)16:33
weshayhttps://review.openstack.org/#/c/588527/216:33
weshaydoes tripleo-bootstrap need to be ported back to pike? https://github.com/openstack/tripleo-heat-templates/blob/stable/pike/common/deploy-steps.j216:35
weshayor can we put this yum install into ci16:35
mwhahahaweshay: what did we document for that?16:35
weshayya. that was my next step16:35
mwhahahaif we documented that an end user had to manually do that then sure put it in oooq16:35
mwhahahaif we didn't then it's a bug to be fixed elsewhere16:35
weshaymwhahaha, was deployed server supported in pike?16:35
mwhahahathat's a question for slage tho as i don't remember what we did for pike (he's on pto)16:35
weshayya16:36
mwhahahai believe that was the first one where we did support it16:36
mwhahahayou could maybe check the downstream docs16:36
weshayI am now16:36
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates master: Skip Ceph packages when running tripleo-packages  https://review.openstack.org/58861816:36
mwhahahasince we don't version upstream docs it's hard to tell16:36
weshaymarios, fyi ^16:36
mwhahahawe should really version the upstream docs16:36
openstackgerritGiulio Fidente proposed openstack/tripleo-common master: Remove ceph-osd and update other Ceph packages if deployment succeeded  https://review.openstack.org/58861916:36
*** boazel has quit IRC16:39
openstackgerritRaoul Scarazzini proposed openstack/tripleo-quickstart-extras master: Integrate Validate-HA work into OVB jobs  https://review.openstack.org/53766916:40
weshaymwhahaha, I think it's supported w/ https://github.com/openstack/tripleo-heat-templates/blob/stable/pike/environments/deployed-server-bootstrap-environment-centos.yaml16:40
weshayhttps://github.com/openstack/tripleo-heat-templates/blob/stable/pike/deployed-server/deployed-server-bootstrap-centos.yaml16:41
mwhahahaweshay: so that would be the fix for pike is to enable that16:41
weshaymwhahaha, ya..16:41
* weshay looking for deployed-server-bootstrap-centos.sh16:41
*** jpena is now known as jpena|off16:44
weshaymwhahaha, marios  ok.. https://review.openstack.org/#/c/588527/216:46
weshaysee my comments16:47
*** openstackgerrit has quit IRC16:49
*** gfidente has quit IRC16:50
*** agopi_ has joined #tripleo16:51
*** agonzalez has joined #tripleo16:52
agonzalezHello, with Swift is possible enable Encryption with Barbican, is it possible to do the same with Ceph rgw?16:52
*** agopi has quit IRC16:54
*** agopi_ is now known as agopi16:54
derekhweshay: rlandy|mtg I gotta go soon but will try a reproduce later, is it happening on a specific job?16:55
weshayderekh, featureset001/35 ovb16:56
rlandy|mtgderekh: it's sporadic on various jobs - but I will send you an email with explanations when I am out or meeting16:56
derekhrlandy|mtg: weshay thanks, will mail if I find anything out16:57
derekhalso added some more thoughts to the bug16:57
*** derekh is now known as derekh_afk17:00
*** boazel_ has joined #tripleo17:06
*** boazel_ has joined #tripleo17:07
*** boazel_ has quit IRC17:07
*** boazel_ has joined #tripleo17:08
*** boazel_ has quit IRC17:08
*** boazel_ has joined #tripleo17:09
*** boazel_ has quit IRC17:09
*** ooolpbot has joined #tripleo17:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION17:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178401717:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178482217:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178484717:10
openstackLaunchpad bug 1784017 in tripleo "Build of instance was re-scheduled: invalid argument: could not find capabilities for domaintype=kvm" [Critical,In progress] - Assigned to Rafael Folco (rafaelfolco)17:10
*** ooolpbot has quit IRC17:10
openstackLaunchpad bug 1784822 in tripleo "legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master fails contact ipmi" [Critical,Triaged]17:10
openstackLaunchpad bug 1784847 in tripleo "exception introspecting nodes" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)17:10
*** boazel_ has joined #tripleo17:13
*** florianf has quit IRC17:14
*** boazel_ has joined #tripleo17:14
*** boazel_ has left #tripleo17:15
*** openstackgerrit has joined #tripleo17:16
openstackgerritwes hayutin proposed openstack/tripleo-quickstart master: add required deployed server config for pike  https://review.openstack.org/58862317:16
openstackgerritwes hayutin proposed openstack-infra/tripleo-ci master: Remove tripleo.sh --bootstrap-subnodes add ceph loop device  https://review.openstack.org/58319517:17
openstackgerritwes hayutin proposed openstack-infra/tripleo-ci master: Remove tripleo.sh --bootstrap-subnodes add ceph loop device  https://review.openstack.org/58319517:17
*** dtantsur is now known as dtantsur|afk17:20
*** boazel has joined #tripleo17:20
*** dmellado has quit IRC17:22
*** stevebaker has quit IRC17:23
*** gouthamr has quit IRC17:23
*** boazel has quit IRC17:25
*** gouthamr has joined #tripleo17:26
*** boazel has joined #tripleo17:28
*** psachin has quit IRC17:29
-openstackstatus- NOTICE: Project renames and review.openstack.org downtime are complete without any major issue.17:30
*** gouthamr has quit IRC17:33
*** salmankhan has quit IRC17:46
*** hgibson has joined #tripleo17:56
*** rlandy|mtg is now known as rlandy|brb18:02
*** amoralej is now known as amoralej|off18:03
*** ccamacho has quit IRC18:04
*** ccamacho has joined #tripleo18:05
weshaymwhahaha, derekh_afk the quarterback is toast18:05
derekh_afkweshay: ya, my reproduce script failed to get far and now can't do nova list ;-(18:08
openstackgerritMerged openstack/instack-undercloud master: Set connect_timeout in mysql  https://review.openstack.org/58627518:08
* derekh_afk will try again later18:08
*** ccamacho has quit IRC18:09
*** ooolpbot has joined #tripleo18:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION18:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178401718:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178482218:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178484718:10
*** ooolpbot has quit IRC18:10
openstackLaunchpad bug 1784017 in tripleo "Build of instance was re-scheduled: invalid argument: could not find capabilities for domaintype=kvm" [Critical,In progress] - Assigned to Rafael Folco (rafaelfolco)18:10
openstackLaunchpad bug 1784822 in tripleo "legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master fails contact ipmi" [Critical,Triaged]18:10
openstackLaunchpad bug 1784847 in tripleo "exception introspecting nodes" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)18:10
*** openstackgerrit has quit IRC18:19
*** derekh_afk has quit IRC18:21
*** gbarros has quit IRC18:23
*** gbarros has joined #tripleo18:26
*** gbarros has quit IRC18:31
*** rlandy|brb is now known as rlandy18:34
*** openstackgerrit has joined #tripleo18:35
openstackgerritEmilien Macchi proposed openstack/puppet-tripleo stable/queens: Release 8.3.5 (Queens)  https://review.openstack.org/58863618:35
openstackgerritEmilien Macchi proposed openstack/puppet-tripleo stable/pike: Release 7.4.15 (Pike)  https://review.openstack.org/58863718:36
openstackgerritEmilien Macchi proposed openstack/puppet-tripleo stable/ocata: Release 6.5.15 (Ocata)  https://review.openstack.org/58863818:37
openstackgerritEmilien Macchi proposed openstack/ansible-role-tripleo-modify-image master: Add Buildah support  https://review.openstack.org/58839018:40
*** gbarros has joined #tripleo18:41
openstackgerritJohn Fulton proposed openstack/tripleo-heat-templates master: Change NodeDataLookup type from string to json  https://review.openstack.org/58864018:48
*** gbarros has quit IRC18:51
*** jtomasek has quit IRC18:51
*** gbarros has joined #tripleo18:52
openstackgerritEmilien Macchi proposed openstack/tripleo-common master: ansible: replace yum module by package module when possible  https://review.openstack.org/58448018:52
*** gouthamr has joined #tripleo19:04
openstackgerritMerged openstack/tripleo-common stable/queens: Include the stack name when generating the tripleo-ansible-inventory  https://review.openstack.org/58848919:07
*** ooolpbot has joined #tripleo19:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION19:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178401719:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178482219:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178484719:10
*** ooolpbot has quit IRC19:10
openstackLaunchpad bug 1784017 in tripleo "Build of instance was re-scheduled: invalid argument: could not find capabilities for domaintype=kvm" [Critical,In progress] - Assigned to Rafael Folco (rafaelfolco)19:10
openstackLaunchpad bug 1784822 in tripleo "legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master fails contact ipmi" [Critical,Triaged]19:10
openstackLaunchpad bug 1784847 in tripleo "exception introspecting nodes" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)19:10
*** gbarros has quit IRC19:11
*** derekh has joined #tripleo19:13
*** derekh has quit IRC19:15
*** boazel has quit IRC19:18
*** salmankhan has joined #tripleo19:19
*** colonwq has quit IRC19:21
*** salmankhan has quit IRC19:24
*** jcoufal has quit IRC19:26
*** jcoufal has joined #tripleo19:29
openstackgerritRafael Folco proposed openstack/tripleo-quickstart-extras master: WIP: Fix libvirt_type kvm for overcloud deploy  https://review.openstack.org/58757219:35
openstackgerritRafael Folco proposed openstack-infra/tripleo-ci master: Remove dup config on updates fs037 job  https://review.openstack.org/58865019:49
*** mjturek has joined #tripleo19:52
mwhahahabeekneemech: you around?19:54
beekneemechmwhahaha: For a little while longer.19:54
mwhahahabeekneemech: wanted to see if you wanted to throw things at ovb, or maybe wait until monday19:54
beekneemechmwhahaha: I'm actually on PTO next week, so chances are I'm not going to have time to dig into anything significant until I get back.19:56
mwhahahaoh all rgith19:56
openstackgerritMerged openstack/puppet-tripleo stable/ocata: Release 6.5.15 (Ocata)  https://review.openstack.org/58863819:56
mwhahahai didn't get a chance to launch an undercloud yet, let me see if i can get that done quickly19:56
*** dmellado has joined #tripleo19:56
mwhahahaotherwise i'll poke at it next week19:56
openstackgerritMerged openstack/puppet-tripleo stable/pike: Release 7.4.15 (Pike)  https://review.openstack.org/58863719:59
*** gouthamr has quit IRC20:00
openstackgerritMerged openstack/puppet-tripleo stable/queens: Release 8.3.5 (Queens)  https://review.openstack.org/58863620:01
*** toure is now known as toure|gone20:01
*** rbrady has quit IRC20:06
*** artom has quit IRC20:07
*** ooolpbot has joined #tripleo20:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION20:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178401720:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178482220:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178484720:10
*** ooolpbot has quit IRC20:10
openstackLaunchpad bug 1784017 in tripleo "Build of instance was re-scheduled: invalid argument: could not find capabilities for domaintype=kvm" [Critical,In progress] - Assigned to Rafael Folco (rafaelfolco)20:10
openstackLaunchpad bug 1784822 in tripleo "legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master fails contact ipmi" [Critical,Triaged]20:10
openstackLaunchpad bug 1784847 in tripleo "exception introspecting nodes" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)20:10
*** stevebaker has joined #tripleo20:14
openstackgerritRafael Folco proposed openstack/tripleo-quickstart-extras master: WIP: Fix libvirt_type kvm for overcloud deploy  https://review.openstack.org/58757220:19
mwhahahabeekneemech: ok you still around to troubleshoot some ipxe or are you about to leave20:22
beekneemechmwhahaha: I have a little more time.20:23
*** raildo has quit IRC20:25
openstackgerritJohn Fulton proposed openstack/tripleo-heat-templates stable/queens: DO-NOT-MERGE: testing fix for related bug  https://review.openstack.org/58865420:27
*** jcoufal has quit IRC20:29
*** ansmith has quit IRC20:34
*** gbarros has joined #tripleo20:35
*** abays has quit IRC20:36
*** agopi has quit IRC20:43
*** abishop has quit IRC20:45
openstackgerritDmitry Tantsur proposed openstack/tripleo-common master: Workflow for provisioning bare metals without Nova  https://review.openstack.org/57685620:45
openstackgerritDmitry Tantsur proposed openstack/tripleo-common master: Switch away from using token_endpoint keystoneauth plugin  https://review.openstack.org/58576520:47
openstackgerritDmitry Tantsur proposed openstack/tripleo-common master: [WIP] Collect required information for deploying without Nova  https://review.openstack.org/58627920:47
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: WIP - Podman support for docker-puppet  https://review.openstack.org/58865520:53
*** beekneemech is now known as bnemec-pto20:55
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: WIP - Podman support for docker-puppet  https://review.openstack.org/58865520:56
*** rfolco|ruck is now known as rfolco|off21:01
*** gbarros has quit IRC21:02
openstackgerritEmilien Macchi proposed openstack/tripleo-common master: Add 'podman' rpm to base_centos containers  https://review.openstack.org/58865921:05
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: DNM - test containerized undercloud with podman  https://review.openstack.org/58866021:06
*** ooolpbot has joined #tripleo21:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION21:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178401721:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178482221:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178484721:10
*** ooolpbot has quit IRC21:10
openstackLaunchpad bug 1784017 in tripleo "Build of instance was re-scheduled: invalid argument: could not find capabilities for domaintype=kvm" [Critical,In progress] - Assigned to Rafael Folco (rafaelfolco)21:10
openstackLaunchpad bug 1784822 in tripleo "legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master fails contact ipmi" [Critical,Triaged]21:10
openstackLaunchpad bug 1784847 in tripleo "exception introspecting nodes" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)21:10
*** eck` is now known as eck`gone21:26
weshaybfournie, rlandy ok.. ovb job running introspection now21:31
weshayI have the console up on all nodes21:31
*** gbarros has joined #tripleo21:33
rlandyok21:33
*** apetrich has quit IRC21:34
weshayrlandy, it's hanging on node import21:35
rlandyweshay: that's great21:35
rlandywhat's on your bmc log?21:35
weshayhrm.. do I have access?21:35
weshayrlandy,21:35
rlandyweshay, yse21:37
rlandyyes21:37
rlandyhorizon21:37
rlandyclick on the instance21:37
weshayya. I'm there21:37
rlandyincrease the log length21:37
rlandyI have one node timing out on introspection21:38
*** pcaruana has quit IRC21:38
rlandybut it's trying again ... Retrying 1 nodes that failed introspection. Attempt 1 of21:38
rlandyweshay: see any errors?21:38
rlandyunreachable anything?21:38
weshayno not yet21:39
weshayjust a boot log21:39
weshayon the bmc21:39
rlandy[104870.686499] openstackbmc[2567]: Reporting power state "False" for instance 6eea2153-83e9-41a6-92b4-a5c956f3a9e321:39
rlandy^^ nothing like21:39
weshaynope21:39
rlandyreally?21:40
rlandyone the console log - weird21:40
weshayI can kill the job and rerun the import21:40
rlandyweshay: what is the last command output you see on the undercloud?21:41
weshayopenstack overcloud node import instackenv.json21:42
rlandyif it's making an ipmp call, we should see it on the bmc21:42
weshaytrying again21:42
rlandyok21:42
*** gbarros has quit IRC21:43
*** gouthamr has joined #tripleo21:47
weshayhrm.. not sure rlandy21:48
weshaylet's chat here21:48
weshaynoticing the ovb bmc code fail to load21:48
weshayhttp://paste.openstack.org/show/727312/21:48
weshayhttps://phx2.cloud.rdoproject.org/dashboard/project/instances/52a9b368-7d36-495f-ae8e-8c9fdd981439/console21:48
weshaymwhahaha, fyi21:49
mwhahahathe parse error: Invalid numeric literal at line 1, column 15 lines are not problems21:50
rlandynot sure how that is sporadic21:50
mwhahahathe ip netmask thing is tho21:50
mwhahahaweshay: so yea if the neutron output in the bmc code isn't properly setting up the ips that would explain the introspection problems21:53
mwhahahalet me go see where that comes from21:53
mwhahahaweshay: the prefix len comes from... prefix_len=$(neutron subnet-show -f value -c cidr $private_subnet | awk -F / '{print $2}')21:55
* mwhahaha would like to point out that we shouldn't still be using neutron cli calls but that's for another day21:56
*** rh-jelabarre has quit IRC21:56
weshay neutron net-show -f value -c subnets private-4390121:56
weshayUnable to establish connection to https://phx2.cloud.rdoproject.org:13000/v2.0/tokens: HTTPSConnectionPool(host='phx2.cloud.rdoproject.org', port=13000): Max retries exceeded with url: /v2.0/tokens (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x1eace10>: Failed to establish a new connection: [Errno -2] Name or service not known',))21:56
weshay[  121.954681] cloud-init[2067]: + private_subnet=21:56
mwhahahadns eh21:56
weshayprobably21:57
mwhahahaweshay: what's the dns server on the bmc21:57
mwhahahaif you tell me 8.8.8.8 i'm going to hurt you21:57
weshaylolz21:57
* weshay checks21:58
weshaytoo many console windows up21:58
weshay| dns_nameservers   | 38.145.33.9122:00
weshaytenant provided22:00
mwhahahais that working correctly?22:00
weshayhttp://paste.openstack.org/show/727314/22:00
weshaymwhahaha, sec.. I don't have access to the bmc node22:01
mwhahahaweshay: I would would say try switching the dns servers to 1.1.1.1 and see if it goes away22:01
*** dsneddon has quit IRC22:02
weshay[zuul@undercloud ~]$ cat /etc/resolv.conf22:02
weshay; generated by /usr/sbin/dhclient-script22:02
weshaysearch rdocloud22:02
weshaynameserver 38.145.33.9122:02
weshaynameserver 38.145.32.6622:02
weshay[zuul@undercloud ~]$ ping phx2.cloud.rdoproject.org22:03
weshayPING phx2.cloud.rdoproject.org (38.145.32.1) 56(84) bytes of data.22:03
weshay64 bytes from phx2.cloud.rdoproject.org (38.145.32.1): icmp_seq=1 ttl=63 time=2.68 ms22:03
weshaymwhahaha, if it was dns it was a hiccup22:03
*** dsneddon has joined #tripleo22:03
weshaywhy the f. is it always dns22:03
mwhahahaweshay: because reasons22:03
weshaymwhahaha, bah.. you like the anti budda22:03
weshayyou are like22:03
weshayI always come to you asking questions and you never have any answers ;P22:04
mwhahahaoh i have answers22:04
mwhahahayou just don't like them22:04
weshaylolz22:04
weshayrlandy, /me wonders if any jobs have started pass w/ fs00122:04
weshaysince the reboots22:04
rlandychecking22:04
* weshay needs more data.. and.. f.. access22:04
rlandynot sure if they would get there yet22:04
rlandyreboot only finished22:05
rlandy~1hr22:05
weshayrlandy, so I don't think we're downloading the ssh key for the bmc node in ci..22:05
weshaythat is something I think panda|rover|off was fixing as well22:05
rlandyno22:05
*** dsneddon has quit IRC22:05
*** dsneddon has joined #tripleo22:05
mwhahahaweshay: yea it's the default key22:05
mwhahahaso unless you inject the key into the image, i don't think we have a way of doing that22:06
mwhahahaweshay: so can we switch the dns servers just for giggles?22:06
weshaymwhahaha, I could in my tenant22:06
rlandy 2018-08-03 15:27 || 99 min || /home/zuul/overcloud-prep-images.sh fail. |22:06
weshaybut I atm I can't get on the instance22:06
mwhahahawe'd need to do it in the ci tneant22:06
rlandy^^ but they may be before22:06
*** hgibson has quit IRC22:06
weshayrlandy, k22:07
weshaymwhahaha, meh.. this is my tenant22:07
weshayguess I can do that22:07
mwhahahaweshay: at this point i'm trying to rule out any flakiness in the rdo as much as possible22:08
weshaymwhahaha, so you think the dhcp client will update the dns in a reasonable time or I should retry the whole thing w/ 1..1.1.122:08
mwhahahaweshay: so not using a set of servers hosted in the cloud would be helpful22:08
weshaymwhahaha, it's not a hosted dns service22:08
weshayit's dnsmasque for the private network22:08
weshaythe tenant dns service22:09
weshayfaker22:09
weshayPHAKER22:09
mwhahahano that would be 192.x addresses if that was true22:09
mwhahahathose are set on the subnet22:09
openstackgerritJill Rouleau proposed openstack/tripleo-heat-templates master: WIP: DeployedServer support for cidr via get_attr  https://review.openstack.org/58812222:09
mwhahahaor are we using external addresses as the defaults22:09
*** ooolpbot has joined #tripleo22:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION22:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178401722:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178482222:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178484722:10
*** ooolpbot has quit IRC22:10
openstackLaunchpad bug 1784017 in tripleo "Build of instance was re-scheduled: invalid argument: could not find capabilities for domaintype=kvm" [Critical,In progress] - Assigned to Rafael Folco (rafaelfolco)22:10
openstackLaunchpad bug 1784822 in tripleo "legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master fails contact ipmi" [Critical,Triaged]22:10
openstackLaunchpad bug 1784847 in tripleo "exception introspecting nodes" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)22:10
weshaymwhahaha, rlandy  of course he's right22:11
weshaydang you22:11
mwhahahaDON'T QUESTION ME SON22:11
weshaymwhahaha, ok.. trying w/ 1.1.1.122:11
weshaylolz22:11
weshayI should crack a beer22:11
mwhahaha1.1.1.1 and 1.0.0.122:11
weshayand just retry the introspection22:11
weshay?22:11
weshayor re-doit22:11
mwhahahano you'd need to rebuild the env22:11
mwhahahaor reboot your bmc22:12
mwhahahaor manually re-run the cloud-init stuff22:12
mwhahahabash /var/lib/cloud/instances/*/scripts/part-00122:12
weshayhrm.. still getting dep errors on the service22:15
weshaySPAM22:15
weshaySPAM22:15
weshay322:15
weshay222:15
weshay122:15
weshaySee 'systemctl status config-bmc-ips.service' for details.22:15
weshay[[1;33mDEPEND[0m] Dependency failed for openstack-bmc baremetal-43901_0 Service.22:15
weshay[[1;33mDEPEND[0m] Dependency failed for openstack-bmc baremetal-43901_1 Service.22:15
weshay[[1;33mDEPEND[0m] Dependency failed for openstack-bmc baremetal-43901_2 Service.22:15
weshay[[1;33mDEPEND[0m] Dependency failed for openstack-bmc baremetal-43901_3 Service.[    7.163603] os-net-config[761]: obj = objects.object_from_json(iface_json)22:15
weshay[    7.164878] os-net-config[761]: File "/usr/lib/python2.7/site-packages/os_net_config/objects.py", line 38, in object_from_json22:15
weshay[    7.166767] os-net-config[761]: return Interface.from_json(json)22:15
weshay[    7.167866] os-net-config[761]: File "/usr/lib/python2.7/site-packages/os_net_config/objects.py", line 305, in from_json22:15
weshay[    7.169671] os-net-config[761]: opts = _BaseOpts.base_opts_from_json(json)22:15
weshay[    7.170901] os-net-config[761]: File "/usr/lib/python2.7/site-packages/os_net_config/objects.py", line 256, in base_opts_from_json22:15
* mwhahaha shrugs22:16
mwhahahais that from the reboot or did you rerun the cloud-init stuff22:16
weshayreboot22:17
weshaytrying to import anyway atm22:17
mwhahahayea you probably need to rerun the cloud init bits22:17
mwhahahacheck the ips are properly defined22:17
weshayoh ya.. it turns it off after the first time22:17
weshay 1dot1dot1dot1.cloudflare-dns.com22:20
weshaynice name22:20
mwhahahaspeaking of broken things22:21
mwhahahascenario001 in queens looks to be broken22:21
weshaymwhahaha, /me checks the image22:22
mwhahahanothing but timeouts22:22
mwhahahait's hanging on step322:23
mwhahahalovely22:23
weshayprobably dns22:23
mwhahahawho backported something that broke everything22:23
* mwhahaha flips tables22:23
weshaylolz22:23
mwhahahai bet it's some stupid ceph thing anyway22:24
weshaymwhahaha, what log are you looking at?22:24
weshayI see something else going on in the latest check jobs22:24
mwhahahahttp://logs.openstack.org/53/587353/1/gate/tripleo-ci-centos-7-scenario001-multinode-oooq-container/9f50b78/22:25
mwhahahai'm currently looking at that one22:25
mwhahahabut none of the stable/queens have passed recently on http://zuul.openstack.org/builds.html?job_name=tripleo-ci-centos-7-scenario001-multinode-oooq-container22:25
weshayoh ya I was looking at the wrong thing22:25
* weshay looks again22:25
weshayyes.. see that22:27
rlandythere are a bunch of jobs at the undercloud install stage - so ci should report on introspection soon22:28
mwhahaharlandy: won't do any good, cause dns change won't take effect until the bmc nodes were booted22:28
mwhahahaso likely won't show up until the next batch of ci jobs start22:29
weshaymwhahaha, rlandy I'm trying the dns change first locally22:29
mwhahahapffft22:29
mwhahahawhere's your sense of adventure22:29
weshaytru22:29
mwhahahado it in prod on a friday afternoon22:29
weshaylolz22:29
weshayto quote 12 monkeys22:30
weshayf.. the bozo's22:30
weshaybozos22:30
weshaylolz22:30
mwhahahai mean it's not like it's actually working right now anyway22:30
mwhahahaas long as we don't change another thing till it's been vetted, should be fine :D22:30
*** boazel has joined #tripleo22:32
openstackgerritCarlos Goncalves proposed openstack/diskimage-builder master: Install ca-certificate with redhat-common  https://review.openstack.org/58867622:32
weshay; generated by /usr/sbin/dhclient-script22:33
weshaysearch rdocloud22:33
weshaynameserver 1.1.1.122:33
mwhahahaok so back to scenario001, it's actually failing in step5 it seems22:34
mwhahahaor is just running really long22:34
mwhahahai have no idea what would hang step522:34
mwhahahathat's a new one22:34
* mwhahaha blames gnocchi or something22:34
weshayk.. crap22:34
* weshay will bug it22:34
mwhahahayea so if it's scenario001 and step5 it's likely ceilometer22:35
* mwhahaha blames pradk but he never shows up anymore22:35
mwhahahaweshay: yea gnocchi_db_sync was running for about an hour22:39
mwhahahaweshay: so gnocchi is hanging in queens22:39
* mwhahaha wish he was kidding when he first said gnocchi22:39
mwhahahahttp://logs.openstack.org/53/587353/1/gate/tripleo-ci-centos-7-scenario001-multinode-oooq-container/9f50b78/logs/subnode-2/var/log/extra/docker/docker_allinfo.log.txt.gz22:40
mwhahahahttp://logs.openstack.org/53/587353/1/gate/tripleo-ci-centos-7-scenario001-multinode-oooq-container/9f50b78/logs/subnode-2/var/log/extra/docker/containers/gnocchi_db_sync/stdout.log.txt.gz22:40
mwhahahaceph vs gnocchi22:41
* mwhahaha throws things at fultonj cause he's the only one here22:41
mwhahahaTimedOut: [errno 110] error opening pool 'metrics'22:41
*** rlandy has quit IRC22:42
*** derekh has joined #tripleo22:43
bfourniemwhahaha, weshay: looking at dependency issue in scrollback - wonder if is this due to https://bugzilla.redhat.com/show_bug.cgi?id=157844022:44
openstackbugzilla.redhat.com bug 1578440 in os-net-config "Drop OVS dependency from os-net-config to fix future Entitlement issues" [High,Closed: errata] - Assigned to bfournie22:44
derekhweshay: I've been trying to reproduce the problem, set up a env and have rerun introspection a bunch of times, no failures yet22:44
weshayderekh, seems like the bmc is not starting due to dep issues22:44
mwhahahabfournie: no weshay's error was during the bmc setup which doesn't use that bits. we don't use ovs to configure the bmc22:45
bfourniemwhahaha: k22:45
fultonjdoes the metrics pool exist as per 'ceph df' ; and do we catpure that output in CI22:45
mwhahahafultonj: i don't think we capture that, through it would be a good item to add to the hostinfo22:46
fultonjkeyring looks ok22:47
fultonjhttp://logs.openstack.org/53/587353/1/gate/tripleo-ci-centos-7-scenario001-multinode-oooq-container/9f50b78/logs/subnode-2/etc/ceph/22:47
weshayhttps://phx2.cloud.rdoproject.org/dashboard/project/instances/031a8a5b-1eb6-4476-89b9-2375000af709/console22:48
derekhweshay: If its some dependency its weird that its intermittent22:48
weshayDAM YOU mwhahaha22:48
* weshay putting in a change22:48
mwhahahaweshay: well since it is a sporadic failure thing, we'll only really know at scale :D22:49
weshaytru22:49
mwhahahafultonj: i see calls to info about the metrics pool in the deploy logs http://logs.openstack.org/53/587353/1/gate/tripleo-ci-centos-7-scenario001-multinode-oooq-container/9f50b78/logs/subnode-2/var/log/journal.txt.gz#_Aug_02_17_52_3222:49
* mwhahaha give sup for the day22:50
mwhahahasee you folks on monday22:50
fultonjif the metrics pool was not there we would not have gotten past step422:51
fultonja bad metrics pool will prevent gnochi from standing up at that step22:51
*** agopi has joined #tripleo22:51
mwhahahafultonj: so that's step5 now i thought and that's what it's failing on22:55
fultonjmwhahaha: that's my point is that we got to step5 so therefore the metrics pool was working well enough at that point22:55
* mwhahaha shrugs 22:57
fultonjthat's step5 now22:57
mwhahahayea i think it's hanging on that22:57
mwhahahaanyway22:57
mwhahahawe'll figure it out monday i guess22:57
fultonjyou mean gnochi no longer gets runing at step422:57
fultonjif ghnochi now starts running at 5 instead, then that's a problem in my reasoning22:57
fultonjyeah monday22:57
* fultonj going back to scale lab...22:58
* weshay is going back to cali22:58
openstackgerritwes hayutin proposed openstack/tripleo-quickstart-extras master: update rdocloud dns settings to outside provider  https://review.openstack.org/58867822:59
openstackgerritwes hayutin proposed openstack-infra/tripleo-ci master: update rdo-cloud dns to outside provider  https://review.openstack.org/58867923:00
weshayderekh, ^ we're trying that ^23:01
derekhweshay: worth a try, my bmc has "nameserver 38.145.33.91"23:03
weshayderekh, ya. that's what we were using23:03
weshaymaybe that resolved well enough at times, but not all the time?23:03
*** mschuppert has quit IRC23:03
*** tosky has quit IRC23:04
weshaymwhahaha, ya.. no-op jobs for queens https://review.openstack.org/#/c/56722423:04
weshayscen001 is down for sure23:04
derekhweshay: it would definitely explain intermittent ipmi errors, coupled with recent cloud stability issues23:05
weshayderekh, /me picks an appropriate nirvana song to listen to now23:05
weshaythis job invents new reasons to drink23:07
derekhweshay: I gotta run, afraid I'm not back until Tuesday, hopefully that sorts things out23:07
weshay:)23:07
weshaytake care brotha23:07
derekhweshay: who needs new reason, I've been using the same reason for years23:07
weshay:)23:07
derekhbecause I wanna23:07
derekhttyl, have a good evening23:08
*** derekh has quit IRC23:08
*** ooolpbot has joined #tripleo23:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION23:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178401723:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178482223:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178484723:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178534223:10
openstackLaunchpad bug 1784017 in tripleo "Build of instance was re-scheduled: invalid argument: could not find capabilities for domaintype=kvm" [Critical,In progress] - Assigned to Rafael Folco (rafaelfolco)23:10
*** ooolpbot has quit IRC23:10
openstackLaunchpad bug 1784822 in tripleo "legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master fails contact ipmi" [Critical,Triaged]23:10
openstackLaunchpad bug 1784847 in tripleo "exception introspecting nodes" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)23:10
openstackLaunchpad bug 1785342 in tripleo "public rdo-cloud dns settings are causing bmc setup issues intermittently " [Critical,In progress] - Assigned to wes hayutin (weshayutin)23:10
weshaybfournie, I'll for sure need to buy you a beer or three23:18
weshayci people need a beer budget23:18
*** boazel has quit IRC23:45
*** rpioso is now known as rpioso|afk23:45
bfournieweshay: nice, I'll see you in denver :-) although I didn't do anything to figure out 178534223:48
*** mjturek has quit IRC23:52
weshaybfournie, it's working thus far https://phx2.cloud.rdoproject.org/dashboard/project/instances/031a8a5b-1eb6-4476-89b9-2375000af709/console23:53
weshay2018-08-03 23:55:10 | Successfully introspected 4 node(s).23:55
weshay2018-08-03 23:55:10 |23:55
weshay2018-08-03 23:55:10 | Introspection completed.23:55
weshay2018-08-03 23:55:10 | + openstack overcloud node pr23:55
bfournieweshay: awesome23:56
weshay2018-08-03 23:57:46 | 4 node(s) successfully moved to the "available" state.23:57
weshayhttps://review.openstack.org/#/c/588678/ https://review.openstack.org/#/c/588679/23:58
weshaypanda|rover|off, rfolco|off marios ^23:58

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!