Tuesday, 2019-02-26

openstackgerritRedHat RDO CI proposed openstack/tripleo-heat-templates stable/queens: GATE CHECK for TripleO  https://review.openstack.org/56722400:00
openstackgerritRedHat RDO CI proposed openstack/tripleo-quickstart-extras master: GATE CHECK for quickstart-extras  https://review.openstack.org/56044500:00
openstackgerritRedHat RDO CI proposed openstack/tripleo-heat-templates master: GATE CHECK for TripleO  https://review.openstack.org/60429800:00
openstackgerritRedHat RDO CI proposed openstack/tripleo-heat-templates stable/rocky: GATE CHECK for TripleO  https://review.openstack.org/60429300:00
*** rascasoft has quit IRC00:04
*** wolverineav has quit IRC00:15
*** wolverineav has joined #tripleo00:21
*** cfontain_ has quit IRC00:21
*** ansmith has joined #tripleo00:22
*** tosky has quit IRC00:23
*** ansmith has quit IRC00:27
*** dciabrin__ has quit IRC00:29
*** dciabrin__ has joined #tripleo00:30
*** jobewan has quit IRC00:45
openstackgerritEmilien Macchi proposed openstack/python-tripleoclient master: container image build: introduce --use-buildah  https://review.openstack.org/63432200:46
*** thrash is now known as thrash|g0ne00:55
*** dprince has quit IRC01:16
*** wolverineav has quit IRC01:18
*** jaosorior has quit IRC01:20
*** wolverineav has joined #tripleo01:24
*** rascasoft has joined #tripleo01:26
*** jaosorior has joined #tripleo01:32
*** dmacpher_ has joined #tripleo01:33
*** dmacpher has quit IRC01:34
openstackgerritSteve Baker proposed openstack-infra/tripleo-ci master: Ensure cirros-0.3.6 is referenced consistently  https://review.openstack.org/63900801:40
*** rascasoft has quit IRC01:40
openstackgerritMerged openstack-infra/tripleo-ci master: Add LVM option for ceph-loop-device role  https://review.openstack.org/63719601:47
*** artom has joined #tripleo02:04
*** _fragatina has quit IRC02:05
*** wolverineav has quit IRC02:05
*** wolverineav has joined #tripleo02:08
*** bfournie has joined #tripleo02:09
openstackgerritIan Wienand proposed openstack/diskimage-builder master: [wip] rhel8 beta support  https://review.openstack.org/62313702:11
openstackgerritIan Wienand proposed openstack/diskimage-builder master: [wip] shortcut pip-and-virtualenv for later redhat distros  https://review.openstack.org/63922002:11
*** mschuppert has quit IRC02:15
*** cfontain has joined #tripleo02:18
*** wolverineav has quit IRC02:26
*** wolverineav has joined #tripleo02:27
openstackgerritMerged openstack/tripleo-quickstart-extras master: Rename errors file if it's big  https://review.openstack.org/63917802:27
*** wolverineav has quit IRC02:32
*** cfontain_ has joined #tripleo02:44
*** cfontain has quit IRC02:47
openstackgerritRafael Folco proposed openstack-infra/tripleo-ci master: Add timestamp to containers' build error log  https://review.openstack.org/63908902:49
*** psachin has joined #tripleo02:56
*** cfontain_ has quit IRC02:57
*** wolverineav has joined #tripleo03:02
*** wolverineav has quit IRC03:07
*** ykarel has joined #tripleo03:19
*** cfontain has joined #tripleo03:24
*** abishop has quit IRC03:27
*** rlandy has quit IRC03:28
*** cfontain has quit IRC03:29
*** ade_lee has quit IRC03:33
*** skramaja has joined #tripleo03:34
*** ramishra has joined #tripleo03:36
*** raukadah is now known as chandankumar03:41
*** sdake has joined #tripleo03:49
*** sdake has quit IRC03:49
*** udesale has joined #tripleo03:54
*** cfontain has joined #tripleo03:56
*** ratailor has joined #tripleo03:56
*** shyamb has joined #tripleo03:59
*** shyamb has quit IRC04:06
*** sdake has joined #tripleo04:06
*** sdake has quit IRC04:10
*** sdake has joined #tripleo04:11
*** sdake has quit IRC04:13
*** cfontain has quit IRC04:13
*** sdake has joined #tripleo04:14
*** sdake has quit IRC04:23
*** wolverineav has joined #tripleo04:33
*** rascasoft has joined #tripleo04:41
openstackgerrityatin proposed openstack/tripleo-quickstart-extras master: Move 'extra_tht_configs' tasks to overcloud-deploy role  https://review.openstack.org/62147304:46
*** rascasoft has quit IRC04:53
*** shyamb has joined #tripleo05:05
*** wolverineav has quit IRC05:09
*** wolverineav has joined #tripleo05:11
*** _fragatina has joined #tripleo05:12
*** shyam89 has joined #tripleo05:14
*** shyam89 has quit IRC05:16
*** shyam89 has joined #tripleo05:17
*** shyamb has quit IRC05:17
openstackgerritMerged openstack/ansible-role-tripleo-modify-image master: Speed up execution time of yum_update.sh  https://review.openstack.org/63539805:17
*** janki has joined #tripleo05:26
*** jaganathan has joined #tripleo05:27
*** ramishra has quit IRC05:32
*** saneax has joined #tripleo05:36
*** ramishra has joined #tripleo05:38
openstackgerritMerged openstack/tripleo-common master: Allow tripleo-admin user to have a keypair  https://review.openstack.org/63831605:44
openstackgerritMerged openstack/tripleo-quickstart-extras master: update the reproducer doc, with image import  https://review.openstack.org/63920805:44
*** _fragatina has quit IRC05:47
*** shyam89 has quit IRC05:48
*** wolverineav has quit IRC06:01
*** wolverineav has joined #tripleo06:02
*** wolverineav has quit IRC06:07
*** wolverineav has joined #tripleo06:14
*** shyamb has joined #tripleo06:14
*** shyam89 has joined #tripleo06:17
*** shyamb has quit IRC06:19
*** shyam89 has quit IRC06:19
*** shyamb has joined #tripleo06:20
*** shyamb has quit IRC06:27
*** shyamb has joined #tripleo06:27
*** jchhatbar has joined #tripleo06:43
*** janki has quit IRC06:45
*** cfontain has joined #tripleo07:01
*** ksambor has joined #tripleo07:01
*** florianf has joined #tripleo07:03
*** wolverineav has quit IRC07:06
*** wolverineav has joined #tripleo07:06
*** shyam89 has joined #tripleo07:07
*** yprokule has joined #tripleo07:07
*** rascasoft has joined #tripleo07:09
*** cfontain has quit IRC07:09
*** shyamb has quit IRC07:10
*** skramaja has quit IRC07:11
openstackgerritCédric Jeanneret proposed openstack/python-tripleoclient master: Ensure created symlink is in $PATH  https://review.openstack.org/63907707:13
*** wolverineav has quit IRC07:13
*** shyam89 has quit IRC07:13
jaosoriorTengu: did you check why https://review.openstack.org/#/c/616945/5 failed?07:14
Tenguhey :)007:16
Tengunot yet, have some other patches in the pipe07:16
ykarelfultonj, seeing  ceph_disk.main.Error: Error: Device /dev/loop3 is in use by a device-mapper mapping (dm-crypt?): dm-0,dm-1,dm-2 in all releases ceph jobs07:16
ykarelfultonj, post https://review.openstack.org/#/c/637196/ merged ^^07:17
ykarelfultonj, example logs:- https://logs.rdoproject.org/92/18992/2/check/rdoinfo-tripleo-master-centos-7-scenario001-standalone/9ccec7e/logs/undercloud/var/log/extra/docker/containers/ceph-osd-prepare-standalone-loop3/stdout.log.txt.gz07:17
*** jfrancoa has joined #tripleo07:19
*** kopecmartin|off is now known as kopecmartin07:23
*** shyam89 has joined #tripleo07:24
*** mschuppert has joined #tripleo07:25
*** jtomasek has joined #tripleo07:26
*** skramaja has joined #tripleo07:26
*** ramishra has quit IRC07:31
*** ramishra has joined #tripleo07:32
*** yprokule_ has joined #tripleo07:35
*** yprokule has quit IRC07:35
*** yprokule_ is now known as yprokule07:35
*** _fragatina has joined #tripleo07:41
*** cylopez has joined #tripleo07:43
ykarelfultonj, so i rechecked https://review.openstack.org/#/c/547682/, looks like master will be clear post this, but need to fix older releases07:46
*** wolverineav has joined #tripleo07:47
*** shyam89 has quit IRC07:51
*** shyam89 has joined #tripleo07:51
*** wolverineav has quit IRC07:52
*** chem` has joined #tripleo07:54
*** ykarel is now known as ykarel|lunch07:56
*** rcernin has quit IRC07:56
*** chem has quit IRC07:57
openstackgerritRedHat RDO CI proposed openstack/tripleo-quickstart-extras master: GATE CHECK for quickstart-extras  https://review.openstack.org/56044508:00
openstackgerritCédric Jeanneret proposed openstack/python-tripleoclient master: Ensure created symlink is in $PATH  https://review.openstack.org/63907708:01
*** shyam89 has quit IRC08:01
*** shyam89 has joined #tripleo08:01
*** agurenko has joined #tripleo08:01
*** ccamacho has joined #tripleo08:02
*** holser_ has joined #tripleo08:03
*** quiquell is now known as quiquell|brb08:03
openstackgerritMerged openstack/tripleo-repos master: Add fedora support to tripleo-repos  https://review.openstack.org/63721208:05
*** avivgt has joined #tripleo08:07
*** shyam89 has quit IRC08:07
jbadiapacould any core review https://review.openstack.org/#/c/638143/ to fix the gate, please?08:09
openstackgerritCédric Jeanneret proposed openstack/python-tripleoclient master: Ensure created symlink is in $PATH  https://review.openstack.org/63907708:15
*** tkajinam has quit IRC08:16
*** shardy has joined #tripleo08:16
openstackgerritMerged openstack/ansible-role-openstack-operations master: Fix the gate  https://review.openstack.org/63814308:20
openstackgerritwaleed mousa proposed openstack/os-net-config master: Configure switchdev mode in SR-IOV in os-net-config  https://review.openstack.org/63388508:22
*** quiquell|brb is now known as quiquell08:23
*** _fragatina has quit IRC08:23
*** aufi has joined #tripleo08:27
jbadiapajaosorior: thx08:28
*** wolverineav has joined #tripleo08:29
*** hberaud|gone is now known as hberaud08:31
*** xek has joined #tripleo08:31
*** holser_ has quit IRC08:32
*** holser_ has joined #tripleo08:33
*** wolverineav has quit IRC08:33
*** pcaruana has joined #tripleo08:35
*** aufi_ has joined #tripleo08:35
*** aufi has quit IRC08:38
*** tosky has joined #tripleo08:39
*** jpena|off is now known as jpena08:42
*** wolverineav has joined #tripleo08:44
*** dtantsur|afk is now known as dtantsur08:46
*** wolverineav has quit IRC08:49
*** shyam89 has joined #tripleo08:52
*** jpich has joined #tripleo08:53
*** _fragatina has joined #tripleo08:54
*** shardy has quit IRC08:54
*** dciabrin__ is now known as dciabrin08:58
*** ccamacho has quit IRC08:58
openstackgerritMerged openstack/tripleo-docs master: Add service template sections walkthough (part 3)  https://review.openstack.org/63838808:58
*** bogdando has joined #tripleo09:02
dtantsurmorning folks! wasn't the problem with removing IPA fixed in DIB long ago? https://logs.rdoproject.org/91/637191/10/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/a92e90b/logs/undercloud/home/zuul/overcloud_image_build.log.txt.gz#_2019-02-25_17_51_3509:09
*** ykarel|lunch is now known as ykarel09:10
*** maufart__ has joined #tripleo09:11
*** aufi_ has quit IRC09:13
*** jchhatbar has quit IRC09:16
openstackgerritMartin Mágr proposed openstack/ansible-role-openstack-operations master: Pacemaker backup task  https://review.openstack.org/63550609:16
*** jchhatbar has joined #tripleo09:16
*** jchhatbar has quit IRC09:22
*** jchhatbar has joined #tripleo09:23
holser_Tengu Hobbestigrou https://review.openstack.org/#/c/638399/09:24
openstackgerritQuique Llorente proposed openstack-infra/tripleo-ci master: DNM: Add py3/f28 support to tripleo-container-build-base  https://review.openstack.org/63616009:24
Tenguholser_: hey :). yep, we just discussed that, it's now sorted ;)09:25
holser_Ahh, thanks :D09:25
Tenguholser_: so now https://review.openstack.org/639077 does all the things09:26
*** ksambor has quit IRC09:30
*** gpocentek has quit IRC09:30
*** Hobbestigrou has quit IRC09:30
*** tbarron has quit IRC09:30
openstackgerritLukas Bezdicka proposed openstack/tripleo-heat-templates master: [FFU] Ensure compatibility with ansible 2.6.  https://review.openstack.org/63916909:30
*** Hobbestigrou has joined #tripleo09:30
shyam89Any ideas to debug this issue: http://paste.openstack.org/show/746147/09:31
shyam89debugging overcloud deployment is toughest part09:32
shyam89I went into nested stacks and resources09:32
shyam89still it just one line of error: stack create cancelled09:32
jaosoriorshyam89: stack CREATE cancelled usually means that you pressed Ctrl + C while the deployment was running.09:33
openstackgerritSergii Golovatiuk proposed openstack/tripleo-heat-templates master: [FFU] Ensure compatibility with ansible 2.6.  https://review.openstack.org/63916909:34
shyam89jaosorior: I have not pressed it09:34
jaosoriorshyam89: is there anything in the heat-engine logs?09:34
shyam89It hangs and timeout09:34
*** ekultails has quit IRC09:35
shyam89jaosorior: in osp14 undercloud is containerized09:35
shyam89where can I find heat logs09:35
Tengushyam89: /var/log/containers/09:35
*** ekultails has joined #tripleo09:35
*** _fragatina has quit IRC09:39
*** ccamacho has joined #tripleo09:39
shyam89jaosorior: logs are rotated09:40
shyam89and cleaned also09:40
*** wolverineav has joined #tripleo09:41
shyam89my stack failed yesterday evening09:41
shyam89now I don't see logs of that time09:41
*** ksambor has joined #tripleo09:41
paramiteHey guys, can we please merge this backport: https://review.openstack.org/#/c/633510/ ?09:41
shyam89jaosorior: I see vms are in BUILD state09:41
shyam89and their console show no device to boot09:42
jaosoriorshyam89: it could have been that it timedout waiting for those nodes to build09:42
jaosoriorshyam89: so your next bet is to check the ironic logs09:42
openstackgerritBogdan Dobrelya proposed openstack-infra/tripleo-ci master: Delay deployment jobs until the pep8/unit passes..  https://review.openstack.org/56854309:43
shyam89jaosorior: ironic conductor?09:43
shyam89or api09:43
openstackgerritCédric Jeanneret proposed openstack/tripleo-common master: Make PythonImageUploader understand insecure_registries  https://review.openstack.org/63903709:44
jaosoriorshyam89: I think it would be conductor09:45
*** derekh has joined #tripleo09:46
shyam89jaosorior: looked into logs09:48
shyam89I see ipmitool commands there09:48
shyam89returning 009:48
shyam89it means they are successful09:49
shyam89no 'error' or 'failed' found in the log file09:49
jaosorioryou could try to execute them yourself to verify. but it sure seems like the nodes are not booting09:49
shyam89jaosorior: command is working well09:51
shyam89it is able to get power status of vm09:51
shyam89it means power managment is working09:51
shyam89boot process is not09:52
shyam89I have single disk for each overcloud node09:53
shyam89so not set root disk09:53
shyam89I don't think it is required09:53
openstackgerritBogdan Dobrelya proposed openstack-infra/tripleo-ci master: Delay deployment jobs until the pep8/unit passes..  https://review.openstack.org/56854309:56
shyam89jaosorior: Should I re-run the deployment and monitor heat engine logs?09:56
shyam89any other logs need to monitor09:56
jaosoriorshyam89: heat-engine, nova-compute and ironic are the most relevant in this case09:58
shyam89jaosorior: One more things10:02
shyam89I have create new flavors for deployment with new names10:03
shyam89not delete old10:03
shyam89and then added property to those flavors10:03
shyam89to link profiles10:03
shyam89control, compute10:03
*** sai_p_ has quit IRC10:04
shyam89and also mentioned --control-flavor and --compute-flavor10:04
shyam89in deployment command10:04
shyam89jaosorior: is it enough to make custom flavors work?10:04
paramiteguys, can we please merge this backport: https://review.openstack.org/#/c/633510/ ?10:06
openstackgerritSergii Golovatiuk proposed openstack/tripleo-heat-templates master: [FFU] Ensure compatibility with ansible 2.6.  https://review.openstack.org/63916910:06
jaosoriorshyam89: I'm not sure... I think so10:08
openstackgerritCédric Jeanneret proposed openstack/python-tripleoclient master: Ensure created symlink is in $PATH  https://review.openstack.org/63907710:10
*** shyam89 has quit IRC10:10
openstackgerritSergii Golovatiuk proposed openstack/tripleo-heat-templates master: [FFU] Ensure compatibility with ansible 2.6.  https://review.openstack.org/63916910:11
*** shyam89 has joined #tripleo10:11
*** wolverineav has quit IRC10:14
*** skramaja_ has joined #tripleo10:27
dtantsurhi ovn folks. what is broken here? resources.ServiceChain: Property error: resources[37].properties: Property DockerOvnControllerImage not assigned10:28
dtantsur(essentially the default deployed-server installation)10:28
*** skramaja has quit IRC10:29
dtantsurjaosorior: can it be that the deployed-server roles were not updated ^^^?10:30
openstackgerritNatal Ngétal proposed openstack/tripleo-puppet-elements master: [Documentation] Migrate the format.  https://review.openstack.org/62716510:31
jaosoriordtantsur: where are you seeing that error?10:33
dtantsurjaosorior: a deployed-server deployment locally10:33
jaosoriordtantsur: are you using a custom role?10:33
dtantsurjaosorior: nope. line 23 in https://etherpad.openstack.org/p/tripleo-nova-less-deploy10:33
dtantsurthis used to work on an a bit older undercloud. rebuilt mine today and now getting this.10:34
jaosoriordtantsur: uhm... maybe you have an older tripleo-common10:38
*** _fragatina has joined #tripleo10:39
jaosoriordtantsur: Ovn controller should be part of the newer one10:39
dtantsurjaosorior: rebased yesterday, can rebase again10:39
jaosoriordtantsur: what did you rebase? t-h-t?10:39
dtantsurjaosorior: tripleo-common is the only custom thing I have here10:39
jaosoriordtantsur: how do you update it?10:39
dtantsuryou shouldn't have asked ;) lemme show you10:40
dtantsurjaosorior: http://paste.openstack.org/show/742646/10:40
jaosoriorhaha wowo10:40
jaosoriorwell... maybe rebasing works10:40
jaosoriordtantsur: you might also need a newer t-h-t10:40
dtantsurjaosorior: won't I get today's t-h-t if I build oooq with master-tripleo-ci?10:41
jaosoriorright, you should10:41
*** florianf has quit IRC10:45
openstackgerritCédric Jeanneret proposed openstack/puppet-tripleo master: Ensure we keep enough files, while ensuring their content is clean  https://review.openstack.org/61694510:46
dtantsurjaosorior: I wild guess: I already use a checkout of tripleoclient, maybe I should rebase it as well?10:47
openstackgerritSergii Golovatiuk proposed openstack/tripleo-heat-templates master: [FFU] Ensure compatibility with ansible 2.6.  https://review.openstack.org/63916910:48
dtantsurhmm, seems recent enough as well10:48
openstackgerritQuique Llorente proposed openstack-infra/tripleo-ci master: [DNM] Force periodic  https://review.openstack.org/63930110:50
*** dciabrin has quit IRC10:50
*** dciabrin has joined #tripleo10:50
*** holser_ is now known as holser|lunch10:52
openstackgerritLukas Bezdicka proposed openstack/tripleo-heat-templates stable/rocky: [FFU] Ensure compatibility with ansible 2.6.  https://review.openstack.org/63930310:56
openstackgerritJakub Libosvar proposed openstack-infra/tripleo-ci master: DNM: Trigger 012 scenario  https://review.openstack.org/63930410:56
*** shyamb has joined #tripleo10:57
openstackgerritLukas Bezdicka proposed openstack/tripleo-heat-templates stable/queens: [FFU] Ensure compatibility with ansible 2.6.  https://review.openstack.org/63930510:57
ramishradtantsur: May be you're missing https://review.openstack.org/#/c/604953/ or something related to it, your issue seems similar to https://bugzilla.redhat.com/show_bug.cgi?id=162959910:59
openstackbugzilla.redhat.com bug 1629599 in openstack-tripleo-common "OVN CI jobs failed because DockerOvnControllerImage not assigned" [Urgent,Closed: errata] - Assigned to sbaker10:59
*** shyam89 has quit IRC10:59
*** bfournie has quit IRC11:00
shyambHi jaosorior: ironic and heat logs does not show anything11:01
shyambI mean any failures11:01
shyambboot is not working11:01
shyambDo we still support kvm based overcloud vms?11:02
shyambusing vbmc driver?11:02
*** gfidente has joined #tripleo11:02
shyambosp13 was working well11:02
dtantsurramishra: I definitely have patches from 5 months ago :)11:02
shyambosp14 documentation does not mention about this driver11:02
ramishradtantsur: sure, I thought your custom rebase may have messed it up;)11:03
ramishrabut yeah could be some new issue11:04
dtantsurI'll check, but I haven't had any merge conflicts. My code touches files of its own.11:04
openstackgerritQuique Llorente proposed openstack-infra/tripleo-ci master: DNM: Add py3/f28 support to tripleo-container-build-base  https://review.openstack.org/63616011:06
*** shyamb has quit IRC11:08
paramiteskramaja_, thanks11:08
*** florianf has joined #tripleo11:08
openstackgerritChandan Kumar proposed openstack/tripleo-quickstart-extras master: Standardize os_tempest playbook and set vars dynamically  https://review.openstack.org/63931011:17
*** dciabrin has quit IRC11:24
*** dciabrin_ has joined #tripleo11:24
*** shyamb has joined #tripleo11:24
*** thrash|g0ne is now known as thrash11:28
*** udesale has quit IRC11:29
*** hberaud is now known as hberaud|lunch11:29
openstackgerritwaleed mousa proposed openstack/os-net-config master: Configure switchdev mode in SR-IOV in os-net-config  https://review.openstack.org/63388511:30
*** ratailor has quit IRC11:31
*** dciabrin_ is now known as dciabrin11:32
*** psachin has quit IRC11:40
*** wolverineav has joined #tripleo11:41
openstackgerritMarios Andreou proposed openstack-infra/tripleo-ci master: WIP: Add override repos into tripleo-ci tripleo-repos role  https://review.openstack.org/63865211:43
*** wolverineav has quit IRC11:46
*** ccamacho has quit IRC11:50
EmilienMweshay: when you have time today, please review https://review.openstack.org/#/c/63599111:53
EmilienMmarios: ^11:53
*** jpena is now known as jpena|lunch11:57
openstackgerritDan Macpherson proposed openstack/ansible-role-openstack-operations master: Redis backup, restore, and validation tasks  https://review.openstack.org/63746711:57
bandiniEmilienM: https://review.openstack.org/#/c/638518/ quick one / tested and acked by the networking folks, please11:58
EmilienMbandini: ack11:58
bandiniEmilienM: merci ;)11:58
mariosEmilienM: ack adding in next reviews11:59
panda|ruckcan someone from neutron take a look at https://bugs.launchpad.net/tripleo/+bug/1817705 ? blocking master promotion.12:02
openstackLaunchpad bug 1817705 in tripleo "periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master fails tempest test related to DHCP agent" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)12:02
openstackgerritGael Chamoulaud proposed openstack/tripleo-common master: Use ANSIBLE_BECOME_FLAGS instead of ANSIBLE_SUDO_FLAGS  https://review.openstack.org/63932012:03
*** quiquell is now known as quiquell|lunch12:04
panda|ruckmmmh lunch12:05
Tenguhave a good one :)12:06
Tenguhey folks, btw, PTAL? https://review.openstack.org/#/c/637729/  note: the new check isn't yet active in the current state, would love to get some inputs as to WHERE I can get it working (CI folks, mostly ;))12:06
*** rfolco|rover has quit IRC12:11
*** rfolco|rover has joined #tripleo12:11
*** ccamacho has joined #tripleo12:12
*** shyamb has quit IRC12:13
*** janki has joined #tripleo12:14
*** jchhatbar has quit IRC12:15
*** quiquell|lunch is now known as quiquell12:15
*** raildo has joined #tripleo12:16
openstackgerritMerged openstack/tripleo-heat-templates master: Switch scenario00{1,4}-standalone to Ceph bluestore  https://review.openstack.org/54768212:22
*** holser|lunch is now known as holser_12:22
*** janki has quit IRC12:24
*** janki has joined #tripleo12:24
*** holser__ has joined #tripleo12:25
*** srini_ has joined #tripleo12:26
*** sanjayu_ has joined #tripleo12:26
*** paramite|clone has joined #tripleo12:26
*** rf0lc0 has joined #tripleo12:26
*** dmellado_ has joined #tripleo12:28
*** janki has quit IRC12:29
*** beekneemech has joined #tripleo12:29
*** dprince has joined #tripleo12:30
*** gouthamr has quit IRC12:30
*** rnoriega has quit IRC12:30
*** avivgt has quit IRC12:30
*** cschwede has quit IRC12:30
*** haleyb has quit IRC12:30
*** cgoncalves has quit IRC12:30
*** owalsh has quit IRC12:30
*** toure has quit IRC12:30
*** sri_ has quit IRC12:30
*** dpeacock has quit IRC12:30
*** dmellado has quit IRC12:30
*** zzzeek has quit IRC12:30
*** rfolco|rover has quit IRC12:30
*** gfidente has quit IRC12:30
*** florianf has quit IRC12:30
*** holser_ has quit IRC12:30
*** ramishra has quit IRC12:30
*** saneax has quit IRC12:30
*** jaganathan has quit IRC12:30
*** bnemec has quit IRC12:30
*** paramite has quit IRC12:30
*** hjensas has quit IRC12:30
*** openstackgerrit has quit IRC12:30
*** dmellado_ is now known as dmellado12:30
*** owalsh_ has joined #tripleo12:31
*** gfidente has joined #tripleo12:31
*** hjensas has joined #tripleo12:31
*** zzzeek has joined #tripleo12:31
*** avivgt has joined #tripleo12:31
*** rnoriega has joined #tripleo12:31
*** dxiri has joined #tripleo12:31
*** cschwede_ has joined #tripleo12:32
*** cfontain has joined #tripleo12:32
*** cgoncalves has joined #tripleo12:32
*** gouthamr has joined #tripleo12:33
*** rf0lc0 is now known as rfolco|rover12:33
*** ramishra_ has joined #tripleo12:38
*** shyamb has joined #tripleo12:41
*** dpeacock has joined #tripleo12:42
*** _fragatina has quit IRC12:44
*** shyamb has quit IRC12:46
*** morazi has joined #tripleo12:47
*** hberaud|lunch is now known as hberaud12:48
*** shyamb has joined #tripleo12:50
*** ccamacho has quit IRC12:52
weshayjaosorior I think tihs channel lost it's alerts12:55
Tenguah, that's why it's sooo silent? :) bot's dead?13:01
jaosoriorweshay: did it? who can we poke?13:01
weshayjaosorior whomever has ops13:03
*** gfidente has quit IRC13:04
*** beagles has left #tripleo13:04
*** beagles has joined #tripleo13:04
jaosoriormwhahaha, EmilienM ^^13:05
*** ccamacho has joined #tripleo13:05
EmilienMops to do what13:05
*** EmilienM sets mode: +o jaosorior13:05
*** EmilienM sets mode: +o weshay13:05
*** EmilienM sets mode: +o Tengu13:05
* EmilienM hides13:05
weshayEmilienM the list the alert bugs13:06
* Tengu ducks13:06
EmilienMweshay: isn't it a script run by someone13:06
EmilienMI'm wondering if beekneemech or dprince know13:06
dprinceyeah, not sure. mwhahaha?13:08
EmilienMalex isn't running it for sure13:08
*** agurenko has quit IRC13:08
*** abishop has joined #tripleo13:08
jaosoriorowalsh_: is this still an issue https://bugs.launchpad.net/tripleo/+bug/1759945 ?13:11
openstackLaunchpad bug 1759945 in tripleo "Review wsgi process/thread config" [High,Triaged] - Assigned to Oliver Walsh (owalsh)13:11
jaosoriorquiquell: is this still an issue https://bugs.launchpad.net/tripleo/+bug/1762351 ?13:12
openstackLaunchpad bug 1762351 in tripleo "periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-queens-upload is timeout in virt-customize step" [High,Triaged] - Assigned to Quique Llorente (quiquell)13:12
jaosoriorarxcruz, quiquell: What about this https://bugs.launchpad.net/tripleo/+bug/1762419 ?  still an issue?13:14
openstackLaunchpad bug 1762419 in tripleo "Collect logs task are not collecting the virt-customize logs from periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-$release-upload " [High,Triaged] - Assigned to Matt Young (halcyondude)13:14
*** mcornea has joined #tripleo13:14
bogdandoPTAL https://review.openstack.org/#/c/635161/13:14
*** sdake has joined #tripleo13:15
jaosoriorjfrancoa, jistr: Do you guys know if this is an issue still https://bugs.launchpad.net/tripleo/+bug/1764698 ?13:17
openstackLaunchpad bug 1764698 in tripleo "Deal with "TLS by default" change on Q->R upgrade" [High,Triaged]13:17
quiquelljaosorior: the issue with nester virtualization I think is still there13:17
quiquelljaosorior: very transitory though13:17
quiquelljaosorior: we have same issue with CI at current reproducer13:17
jaosoriorquiquell: uuuuh... which bug number again?13:17
* jaosorior too many tabs open13:18
quiquelljaosorior: https://bugs.launchpad.net/tripleo/+bug/176235113:18
openstackLaunchpad bug 1762351 in tripleo "periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-queens-upload is timeout in virt-customize step" [High,Triaged] - Assigned to Quique Llorente (quiquell)13:18
jistrjaosorior: re TLS i don't know. Personally i haven't hit anything, so from my perspective we could close and reopen if we discover some issue.13:18
jaosoriorjistr: thanks.13:19
*** owalsh_ is now known as owalsh13:19
quiquelljaosorior: some RDO hypervisors have problem with nester virtualization and centos13:19
quiquelljaosorior: https://bugzilla.redhat.com/show_bug.cgi?id=156517913:19
openstackbugzilla.redhat.com bug 1565179 in qemu-kvm-rhev "qemu-kvm fails when run with -machine accel=kvm:tcg nested KVM" [Unspecified,Closed: insufficient_data] - Assigned to pbonzini13:19
owalshjaosorior: it's more of a "we should take a look at this" than a bug13:20
shyambon osp14, during overcloud deployment kvm based vms are not getting powered on13:20
jaosoriorowalsh: anybody planning to do so?13:20
shyambI am using vbmc driver13:20
owalshjaosorior, not that I know of13:21
jaosoriorok, I'll leave that bug be for now.13:21
shyambDoes rocky tripleo supports vbmc based power management?13:21
Tenguhey folks, quick reminder about the Validation Framework meeting in 8 minutes :)13:22
jaosoriormcornea: hey, do you know if this is still an issue https://bugs.launchpad.net/tripleo/+bug/1765552 ?13:23
openstackLaunchpad bug 1765552 in tripleo "FFU: deploy_steps_playbook.yaml playbook fails because rabbitmq_init_bundle container is unable to successfully run Executing: 'rabbitmqctl status | grep -F "{rabbit,"'" [High,Triaged]13:23
*** trown|outtypewww is now known as trown13:24
*** florianf has joined #tripleo13:24
*** agopi has quit IRC13:29
jaosoriorweshay, marios: I'm checking this bug out https://bugs.launchpad.net/tripleo/+bug/1771082 . Aren't we already collecting console logs?13:30
openstackLaunchpad bug 1771082 in tripleo "[rfe] TripleO CI must collect and publish console logs from fake overcloud nodes during introspection and deployment" [High,Triaged]13:30
panda|ruckjaosorior: yes, we do now13:31
jaosoriorpanda|ruck: thanks13:31
mariosjaosorior: ack thanks panda looks like old bug then13:33
*** jpena|lunch is now known as jpena13:33
fultonjykarel: saw your ping from 6 hours ago13:33
*** rlandy has joined #tripleo13:33
fultonjykarel: the commit message for https://review.openstack.org/#/c/638363 explains what you saw. it was expected behavior, thus the jobs were set to non-voting temporarily so both jobs could land.13:34
fultonjboth jobs have merged, we now just need https://review.openstack.org/#/c/638364 to land so 001 and 004 can vote again13:34
*** ansmith has joined #tripleo13:35
ykarelfultonj, see https://bugs.launchpad.net/tripleo/+bug/1817688 and https://review.openstack.org/#/c/637196/6/roles/ceph-loop-device/tasks/main.yml13:35
openstackLaunchpad bug 1817688 in tripleo "job that test ceph in periodic pipeline for stable branches are getting the wrong configuration" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)13:35
ykarelfultonj, master is fine, issue is in stable branches13:35
*** suuuper has joined #tripleo13:35
owalshshyamb: vbmc is used for CI so it should work. If the VM is being created in libvirt but failing to start then there is a debugging guide here - https://kashyapc.fedorapeople.org/virt/Debugging-libvirt-QEMU-in-OpenStack-DevConf-2016.pdf13:35
fultonjykarel: ok, let me dig into this to fix stable13:36
ykarelfultonj, sync with panda|ruck i think he is already on it13:37
ykareljust to avoid duplicate work13:37
fultonjykarel: ack13:37
owalshshyamb: I'd start by looking at the logs on the host - could be something obvious like not enough RAM/DISK etc...13:37
*** dtantsur is now known as dtantsur|afk13:38
*** shyamb has quit IRC13:39
fultonjpanda|ruck: for 1817688 are you working on making the release variable available earlier?13:40
panda|ruckfultonj: yes13:41
fultonjpanda|ruck: anything i can do to help?13:41
panda|ruckfultonj: still not sure if it's possible13:41
fultonjpanda|ruck: for what versions do we have stable jobs?13:41
fultonjrocky, queens could use bluestore13:42
fultonjdo we also need pike though?13:42
fultonji thought i saw ci_branch working before it merged; is the issue that it only worked for master?13:43
panda|ruckfultonj: yes13:44
fultonjis that yes for all 3 questions?13:45
panda|ruckfultonj: yes :) sorry I saw only the last one13:46
panda|ruckfultonj: so anything you can do to help, probably not, I think we can move this role around, since it' used really only during deployment13:47
panda|ruckfultonj: we have stable jobs down to pike13:47
fultonjdown to pike13:47
panda|ruckfultonj: and yes, the job passed because the ci_branch will always be master for changes in tripleo-ci13:48
panda|ruckfultonj: rocky queens pike13:48
panda|ruckykarel: why is pike still relevant ? seems EOL pretty everywhere ?13:49
ykarelpanda|ruck, in upstream it's not EOL afaik13:49
fultonjrocky and queens could go to bluestore as both are being used with ceph-ansible 3.213:49
*** haleyb has joined #tripleo13:50
panda|ruckfultonj: I think it's a start then, I can help you test the bluestore with queens and rocky branches13:50
fultonjpanda|ruck: ok, i can backport13:50
fultonjpanda|ruck: you think it's too dificult to move that var?13:51
fultonj"move it" as in set it earlier in the process13:51
panda|ruckykarel: you're right, but extended maintenance stops 3 match13:51
fultonj5 days13:52
ykarelyeah only few days left13:52
panda|ruckfultonj: not "too" difficult, but the release variable is pivot to a lot of thinks I have to rotate around it.13:52
panda|ruckfultonj: what you need to do to backport ? in tripleo-ci should be enough to remove the branch restriction, and add branchful tests.13:53
fultonjsince LVM is there i backport ^ to rocky and queens and they will work13:54
fultonjpanda|ruck: this is just a proposal to solve the problem differently ^13:55
mcorneajaosorior: hey o/ I think  it's not anymore, it's pretty old one13:56
*** mjturek has joined #tripleo13:56
jaosoriormcornea: ack, will close it.13:56
panda|ruckfultonj: in rocky and queens standalones are not the main test jobs, we still rely on multinode jobs. You'll need to change scenario001/4-multinode.13:56
*** agopi has joined #tripleo13:58
*** beekneemech is now known as bnemec13:59
bnemecEmilienM: weshay: I think it was being run by derekh14:00
jaosorior#startmeeting tripleo14:00
openstackMeeting started Tue Feb 26 14:00:55 2019 UTC and is due to finish in 60 minutes.  The chair is jaosorior. Information about MeetBot at http://wiki.debian.org/MeetBot.14:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.14:00
*** openstack changes topic to " (Meeting topic: tripleo)"14:00
openstackThe meeting name has been set to 'tripleo'14:00
jaosorior#topic agenda14:01
*** openstack changes topic to "agenda (Meeting topic: tripleo)"14:01
jaosorior* Review past action items14:01
jaosorior* One off agenda items14:01
jaosorior* Squad status14:01
jaosorior* Bugs & Blueprints14:01
jaosorior* Projects releases or stable backports14:01
jaosorior* Specs14:01
jaosorior* open discussion14:01
jaosoriorAnyone can use the #link, #action and #info commands, not just the moderatorǃ14:01
jaosoriorHey folks! who's around?14:01
*** aissat-mus has joined #tripleo14:04
jaosorior#topic review past action items14:04
*** openstack changes topic to "review past action items (Meeting topic: tripleo)"14:04
*** zaneb has quit IRC14:04
jaosorior#topic one off agenda items14:05
*** openstack changes topic to "one off agenda items (Meeting topic: tripleo)"14:05
jaosorior#link https://etherpad.openstack.org/p/tripleo-meeting-items14:05
jaosoriorThe first topic is: (rfolco): Quick updates on Fedora 28 containers build: tripleo-repos tool now supports Fedora 28 for setting repos and mirrors. Check job to be added to RDO given the nuances around unmergeable Kolla patches. Epic: https://tree.taiga.io/project/tripleo-ci-board/epic/445.14:05
jaosoriorrfolco|rover: ^^14:06
weshaymarios, ^14:06
rfolco|roverThe goal is to have a check jon in rdo at the end of this sprint to track progress on f28 containers build. That's it. Happy to provide more detail if any questions.14:06
derekhbnemec: EmilienM ya it was me, a old HD died where I was running it, I'll sort it out and get it back,14:06
rfolco|rovercheck job*14:06
mariosjaosorior: rfolco|rover is rover this sprint so we quiquell zbr|out and i are trying to get those merged see https://tree.taiga.io/project/tripleo-ci-board/us/650?milestone=220350 for info and we can discuss some more in the ci meeting next if anyeone is interested in more14:06
mariosjaosorior: quiquell has been trying to get the job to run in rdo jobs for the upstream at https://review.openstack.org/#/c/63616014:07
*** zaneb has joined #tripleo14:07
mariosjaosorior: main news item compared to last week is that we will only be runing in in check non voting and in rdo for now not upstream14:07
rfolco|roverthanks marios14:09
*** vinaykns has joined #tripleo14:10
jaosoriormarios: thanks for the update14:10
jaosorior#topic Squad status14:11
*** openstack changes topic to "Squad status (Meeting topic: tripleo)"14:11
jaosorior#link https://etherpad.openstack.org/p/tripleo-ci-squad-meeting14:12
jaosorior#link https://etherpad.openstack.org/p/tripleo-upgrade-squad-status14:12
jaosorior#link https://trello.com/b/S8TmOU0u/tripleo-podman14:12
jaosorior#link https://etherpad.openstack.org/p/tripleo-integration-squad-status14:12
jaosorior#link https://etherpad.openstack.org/p/tripleo-ui-cli-squad-status14:12
jaosorior#link https://etherpad.openstack.org/p/tripleo-validations-squad-status14:12
jaosorior#link https://etherpad.openstack.org/p/tripleo-networking-squad-status14:12
jaosorior#link https://etherpad.openstack.org/p/tripleo-security-squad14:12
jaosorior#link https://etherpad.openstack.org/p/tripleo-edge-squad-status14:12
jaosoriorAny squad wanting to bring up their status, or a topic for the general public?14:12
TenguValidation: we need reviewers on https://review.openstack.org/636930 and https://review.openstack.org/636581 - those two are the base of the whole work, and it would be good to get them merged sooner than later.14:12
*** ykarel is now known as ykarel|away14:13
TenguAs next steps will involve the addition of ansible-runner to those two changes, we're stuck for now :).14:13
Tenguhelp is really appreciate.14:13
Tengu(end of communication ;))14:13
*** florianf_ has joined #tripleo14:14
jaosoriorTengu: added myself as a reviewer. Thanks for bringing that up14:15
*** florianf has quit IRC14:17
jaosorior#topic bugs & blueprints14:17
*** openstack changes topic to "bugs & blueprints (Meeting topic: tripleo)"14:17
jaosorior#link https://launchpad.net/tripleo/+milestone/stein-314:17
jaosoriorFor Stein we currently have 21 (for stein-3) blueprints open in Launchpad.14:17
jaosoriorBugs: 522 (-45) stein-3 in Launchpad. 104 (0) open Storyboard bugs.14:17
openstackbug 522 in Launchpad itself "translation percentage off the scale" [Medium,Fix released] https://launchpad.net/bugs/522 - Assigned to Carlos Perelló Marín (carlos)14:17
jaosorior#link https://storyboard.openstack.org/#!/project_group/7614:17
*** jcoufal has joined #tripleo14:18
jaosorior#topic projects releases or stable backports14:20
*** openstack changes topic to "projects releases or stable backports (Meeting topic: tripleo)"14:20
*** sdake has quit IRC14:21
jaosorior#topic projects releases or stable backports14:23
*** openstack changes topic to "projects releases or stable backports (Meeting topic: tripleo)"14:23
*** gfidente has joined #tripleo14:24
*** manojawa has joined #tripleo14:24
manojawaHello All... My existing cluster has - 1 X Director, 3 X Controllers & 10 X Compute nodes. All Baremetal. When I tried to add 11th node, it is failing with error "resources.AllNodesDeploySteps: resources.ControllerDeployment_Step5: Error: resources[2]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6 ". Need help in understanding and fixing the issue.14:25
*** ade_lee has joined #tripleo14:25
manojawaI can provide any required details about my setup14:25
*** tzumainn has joined #tripleo14:26
manojawaI searched a lot. I see lot of links with the same error "Deployment exited with non-zero status code: 6", but none of them have any resolution14:27
jaosorior#topic specs14:28
*** openstack changes topic to "specs (Meeting topic: tripleo)"14:28
jaosorior#link https://review.openstack.org/#/q/project:openstack/tripleo-specs+status:open14:28
manojawaor any root cause14:28
jaosoriorAny specs folks want to bring attention to?14:28
fultonjmanojawa: we're in a meeting ATM14:28
manojawaOh.. I am sorry14:28
fultonjmanojawa: no prob, you can ask when we're done :)14:29
jaosorior#topic open discussion14:29
*** openstack changes topic to "open discussion (Meeting topic: tripleo)"14:29
jaosoriorAnything else that folks want to bring up to the meeting?14:29
fultonjjaosorior: https://bugs.launchpad.net/tripleo/+bug/181768814:30
openstackLaunchpad bug 1817688 in tripleo "job that test ceph in periodic pipeline for stable branches are getting the wrong configuration" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)14:30
fultonjjaosorior: panda|ruck and i are working on this14:30
fultonjjobs which are broken are not voting ATM14:30
manojawafultonj: Thank you ! By what time will the meeting be done from now?14:30
fultonjplan is in bug linked above , if any questions we're here but wanted to point it out14:31
fultonjjaosorior: that's all (unless there's questions)14:31
jaosoriorack, thanks for the update.14:31
weshayfultonj I'm going to raise this in the ci community mtg in a few.. if you could join that would be super14:31
panda|ruckfultonj: jaosorior we'll discuss the bug more in the community meeting14:31
fultonjweshay: ok14:31
bogdandoWrt ordering our CI jobs on the linting/unit results, I revived https://review.openstack.org/#/c/568543/ see also  ML topic http://lists.openstack.org/pipermail/openstack-discuss/14:33
bogdando2019-February/003142.html .  My patch basically fails on zuul's "graphs freezing" and I don't know how to proceed with it. So if anyone wish to handover/assist...14:33
EmilienMo/ (late)14:33
bogdando#link http://lists.openstack.org/pipermail/openstack-discuss/003142.html14:33
*** sdake has joined #tripleo14:33
bogdando#link http://lists.openstack.org/pipermail/openstack-discuss/2019-February/003142.html14:34
weshayah thanks14:34
bogdandoIMO we can make openstack-infra folks happy dancing if introduced such an ordering for tripleo CI pools)14:35
weshayI'll raise this as well in the ci community14:36
mariosbogdando: seems sensible btw first link is 404 for me14:37
bogdandomarios: yeah, sorry for that14:37
mariosbogdando: but yeah if we can have same for py27/pep8 etc for example right14:37
mariosbefore even running jobs14:37
mariosthere may be more to do but that is just the simple/obvious case14:37
*** mjturek has quit IRC14:38
panda|ruckI have concens, we don't run lin and tox for every change, only the ones in emit-releases-script14:38
panda|ruckalso the templates are used in other repos than tripleo-ci14:39
bogdandopanda|ruck: sure, the details should be adjusted14:39
bogdandojust that the very idea seems doesn't work for zuul, or that was me doing it wrong14:39
mariosbogdando: we use it in periodics14:40
mariosbogdando: like the container build job is dependency for all the others14:40
bogdandomarios: oh, then should be not a biggie to catch up that for standalone jobs at least14:40
mariosbogdando: eg ^14:41
panda|ruckmarios: yes, but in tripleo-ci it's very complicated, we serve multiple repos, and the jobs listed in the current dependencies are not run every time14:41
mariospanda|ruck: ack sure i just agreed with the idea, doesn't mean it will work everywhere ack on us not running th elinting etc for each change14:42
weshaypanda|ruck are other projects using our project-template?14:42
mariospanda|ruck: but in general i like the idea14:42
weshayor just the jobs.. I think just the job14:42
panda|rucklet's discuss in the community meeting14:42
panda|ruckweshay: templates also14:42
weshaypanda|ruck hrm.. k.. please grab me a link where a project outside of tripleo is using tripleo-multinode-container-minimal14:43
weshayor other example14:43
panda|ruckweshay: tht14:43
panda|ruckweshay: https://github.com/openstack/tripleo-heat-templates/blob/master/zuul.d/layout.yaml#L414:43
bogdandoso more places to amend14:44
weshaythat's not outside of tripleo though14:44
bogdandolets just to fix that freezer cuz warm beer is not cool14:44
panda|ruckmotion to move to community meeting14:46
*** ykarel|away is now known as ykarel14:46
*** yprokule has quit IRC14:49
*** openstack changes topic to "CI Status: Should be green, recheck if needed | https://docs.openstack.org/tripleo-docs/latest/"14:49
openstackMeeting ended Tue Feb 26 14:49:44 2019 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)14:49
openstackMinutes:        http://eavesdrop.openstack.org/meetings/tripleo/2019/tripleo.2019-02-26-14.00.html14:49
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/tripleo/2019/tripleo.2019-02-26-14.00.txt14:49
openstackLog:            http://eavesdrop.openstack.org/meetings/tripleo/2019/tripleo.2019-02-26-14.00.log.html14:49
*** mjturek has joined #tripleo14:50
rfolco|roverCI community meeting (office hours) starts now at https://bluejeans.com/411356779814:51
*** mjturek has quit IRC14:52
*** sdake has quit IRC14:55
*** aissat-mus has quit IRC14:55
*** ykarel_ has joined #tripleo14:59
*** sdake has joined #tripleo15:00
*** ykarel has quit IRC15:00
*** aedc has joined #tripleo15:02
*** mjturek has joined #tripleo15:03
*** mjturek has quit IRC15:03
manojawaHello All... My existing cluster has - 1 X Director, 3 X Controllers & 10 X Compute nodes. All Baremetal. When I tried to add 11th node, it is failing with error "resources.AllNodesDeploySteps: resources.ControllerDeployment_Step5: Error: resources[2]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6 ". Need help in understanding and fixing the issue.15:04
manojawaI searched a lot. I see lot of links with the same error "Deployment exited with non-zero status code: 6", but none of them have any resolution15:04
manojawaSo, need help in fixing the issue15:04
manojawaPlease let me know what all logs/command outputs I need to provide15:05
aedcHello, where can we create a ticket on a bug, apart from https://bugzilla.redhat.com (tripleo-rdo) ?15:06
aedcPMD: net_mlx5: cannot load glue library: libibverbs.so.1: cannot open shared object file: No such file or directory15:06
aedc=> latest15:07
*** Vorrtex has joined #tripleo15:11
jistraedc: hi, the upstream tracker is https://bugs.launchpad.net/tripleo/+filebug15:18
*** rbrady has joined #tripleo15:19
jistrmanojawa: hi, there should be some better output somewhere which should help you uncover the root cause15:19
aedcjistr: thks15:19
*** gbarros has joined #tripleo15:20
jistrmanojawa: ^ some troubleshooting links15:20
*** owalsh_ has joined #tripleo15:24
*** owalsh has quit IRC15:25
*** gbarros has quit IRC15:26
*** owalsh_ is now known as owalsh15:26
*** gbarros has joined #tripleo15:29
openstackLaunchpad bug 1817743 in tripleo "PMD: net_mlx5: cannot load glue library: libibverbs.so.1: cannot open shared object file: No such file or directory" [Undecided,New]15:30
bogdandoweshay, marios, panda|ruck: pushed some updates here https://review.openstack.org/#/q/topic:ci_pipelines+(status:open+OR+status:merged) , but that freezer still fails :/15:31
*** ykarel_ is now known as ykarel|away15:32
*** sanjayu_ has quit IRC15:34
*** sanjayu_ has joined #tripleo15:35
openstackLaunchpad bug 1789236 in tripleo "OVN deployment is failing when used with OVS 2.10" [High,Fix released] - Assigned to Numan Siddique (numansiddique)15:38
*** Chaserjim has joined #tripleo15:41
larsksmwhahaha: does the ftype=0 check in https://github.com/openstack/ansible-role-container-registry/blob/master/tasks/docker.yml#L28 be limited to just checking /var/lib/docker?15:42
larsksWell, the first half and the second half of that sentence obviously didn't talk to eachother before hand, but you get the idea.15:43
chandankumaris there a way to export these values in tqe OS::TripleO::Services::CephMgr in playbook?15:43
chandankumarduring runtime15:43
mwhahahalarsks: yes it just affects docker storage15:44
mwhahahaBut that is the old default for <7.3 I think15:45
mwhahahaNew systems shouldn't run into that problem15:45
*** ccamacho has quit IRC15:51
*** ccamacho has joined #tripleo15:55
*** maufart__ has quit IRC15:56
*** ksambor has quit IRC15:56
*** sdake has quit IRC15:59
*** sdake has joined #tripleo16:01
*** skramaja_ has quit IRC16:01
*** sdake has quit IRC16:02
fultonjregarding bug 1817688 proposal in comment #516:06
openstackbug 1817688 in tripleo "job that test ceph in periodic pipeline for stable branches are getting the wrong configuration" [Critical,Triaged] https://launchpad.net/bugs/1817688 - Assigned to Gabriele Cerami (gcerami)16:06
fultonjjaosorior: ^16:06
fultonjmwhahaha: ^16:06
fultonjweshay: (as per our conversation)16:07
gfidentechandankumar not sure what you're asking exactly16:08
gfidentechandankumar what is it you need to export?16:08
*** gbarros has quit IRC16:10
chandankumargfidente: I want to set these values dynamically based on services imported https://review.openstack.org/#/c/639310/3/playbooks/tempest.yml@416:10
chandankumargfidente: by the way congrats :-)16:10
chandankumarwe are moving to a new role os_tempest for running tempest in future developed by OSA16:11
*** gbarros has joined #tripleo16:16
*** gbarros has quit IRC16:16
*** paramite|clone has quit IRC16:17
*** sdake has joined #tripleo16:18
*** openstackgerrit has joined #tripleo16:20
openstackgerritSagi Shnaidman proposed openstack/tripleo-quickstart master: WIP: run novajoin tempest tests  https://review.openstack.org/63932416:20
*** jfrancoa has quit IRC16:20
*** suuuper has quit IRC16:20
*** gbarros has joined #tripleo16:21
openstackgerritMarios Andreou proposed openstack-infra/tripleo-ci master: WIP: Add override repos into tripleo-ci tripleo-repos role  https://review.openstack.org/63865216:24
weshaypanda|ruck please add a skip list review for https://bugs.launchpad.net/tripleo/+bug/181770516:24
openstackLaunchpad bug 1817705 in tripleo "periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master fails tempest test related to DHCP agent" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)16:24
weshaypanda|ruck and then continue to debug w/ rfolco|rover16:24
*** itlinux has joined #tripleo16:25
*** sdake has quit IRC16:25
*** sdake has joined #tripleo16:26
panda|ruckweshay: ack16:27
gfidentechandankumar so I am not sure what the os_tempest role does16:28
*** quiquell is now known as quiquell|off16:28
gfidentechandankumar but it looks like data should flow the other way around, you enable particular parts of tempest based on how the heat stack is configured16:28
gfidenteinstead of enabling/disabling services based on tempest config16:29
*** gbarros has quit IRC16:30
*** pcaruana has quit IRC16:32
*** sdake has quit IRC16:35
*** sdake has joined #tripleo16:35
*** cylopez has quit IRC16:36
*** sdake has quit IRC16:40
*** sdake_ has joined #tripleo16:41
*** ykarel|away has quit IRC16:41
*** ykarel has joined #tripleo16:42
*** sdake_ has quit IRC16:45
*** sdake has joined #tripleo16:46
*** sdake has quit IRC16:50
*** sdake has joined #tripleo16:52
*** _fragatina has joined #tripleo16:52
*** gfidente is now known as gfidente|afk16:56
openstackgerritGabriele Cerami proposed openstack/tripleo-quickstart-extras master: Adding test to the skiplist  https://review.openstack.org/63938616:58
*** vinaykns has quit IRC16:59
*** moguimar has quit IRC16:59
*** bogdando has quit IRC17:00
*** wolverineav has joined #tripleo17:00
*** sdake has quit IRC17:00
*** ccamacho has quit IRC17:01
*** sdake_ has joined #tripleo17:02
*** vinaykns has joined #tripleo17:03
*** cfontain has quit IRC17:03
*** cfontain has joined #tripleo17:05
*** aedc has quit IRC17:07
*** _fragatina has quit IRC17:08
*** itlinux_ has joined #tripleo17:10
*** _fragatina has joined #tripleo17:10
*** cfontain has quit IRC17:10
openstackgerritGabriele Cerami proposed openstack/tripleo-quickstart-extras master: Adding test to the skiplist for OVN related updates  https://review.openstack.org/63938617:13
*** itlinux has quit IRC17:13
*** sshnaidm is now known as sshnaidm|afk17:17
*** jpich has quit IRC17:17
openstackgerritAlex Schultz proposed openstack/tripleo-heat-templates master: Drop the role service override for ovb  https://review.openstack.org/63760417:21
*** morazi has quit IRC17:21
openstackgerritGabriele Cerami proposed openstack/tripleo-quickstart-extras master: Adding test to the skiplist for OVN related updates  https://review.openstack.org/63938617:22
openstackgerritNicolas PLANEL proposed openstack/tripleo-heat-templates master: Fix skydive_vars evaluation  https://review.openstack.org/63939417:25
openstackgerritAlex Schultz proposed openstack/tripleo-common master: Use the default ContainerImagePrepare if not provided  https://review.openstack.org/63828617:27
panda|ruckfultonj: question about https://review.openstack.org/639351, we are effectively switching the default to bluestore here, do we need to keep a test for the old method in stable branches ?17:31
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Remove environments/baremetal-services.yaml  https://review.openstack.org/62763217:33
fultonjpanda|ruck: ceph-ansible 3.2 defaults to bluestore.17:33
fultonjpanda|ruck: if we could switch on version, that would be nice to keep testing the older filestore17:33
fultonjbut we can't switch on version17:34
fultonj"do we need to keep a test for the old method", no. it would be nice, so if we get the ability to switch on version then we could in a follow up.17:34
fultonjpanda|ruck: ^17:34
panda|ruckfultonj: ok so you're just updating our method to the current ceph default17:34
fultonjpanda|ruck: yes17:34
panda|ruckfultonj: ok thanks. going to +W the patches then17:35
fultonjpanda|ruck: did they come out green yet?17:35
fultonji expect, they will but it's nice to be sure first17:35
*** cfontain has joined #tripleo17:35
panda|ruckfultonj: yes sorry I meant +217:35
fultonjpanda|ruck: ack17:35
*** mjturek has joined #tripleo17:36
panda|ruckfultonj: also it's on THT which I'm not allowed to +W :P17:37
*** avivgt has quit IRC17:37
openstackgerritSergii Golovatiuk proposed openstack/python-tripleoclient stable/queens: Use a single queue for update/upgrade  https://review.openstack.org/63419017:39
*** paramite|clone has joined #tripleo17:40
panda|ruckfultonj: mmh looks like it failed for queens at least http://logs.openstack.org/52/639352/2/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/6b0ab2d/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz17:41
* fultonj looks17:41
panda|ruckand rocky also ...17:42
openstackgerritLukas Bezdicka proposed openstack/tripleo-heat-templates master: [FFU] Ensure compatibility with ansible 2.6.  https://review.openstack.org/63916917:42
*** holser__ has quit IRC17:48
fultonjpanda|ruck: ^ it's as if the LVs were not created. /me looking17:49
openstackgerritLars Kellogg-Stedman proposed openstack/ansible-role-container-registry master: restrict docker xfs ftype check to /var/lib/docker  https://review.openstack.org/63940017:50
openstackgerritMerged openstack/python-tripleoclient master: Remove redundancy in _launch_ansible* methods.  https://review.openstack.org/63692017:50
larsksmwhahaha: https://review.openstack.org/#/c/639400/ restricts the fstype check.17:53
panda|ruckfultonj: http://logs.openstack.org/52/639352/2/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/6b0ab2d/job-output.txt.gz#_2019-02-26_15_02_29_33057717:53
panda|ruckfultonj: creation was skipped17:53
weshaypanda|ruck so for https://bugs.launchpad.net/tripleo/+bug/1817705  I think we can only skip when ovn is in use ya?17:53
openstackLaunchpad bug 1817705 in tripleo "periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master fails tempest test related to DHCP agent" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)17:53
openstackgerritLars Kellogg-Stedman proposed openstack/ansible-role-container-registry master: restrict docker xfs ftype check to /var/lib/docker  https://review.openstack.org/63940017:53
mwhahahalarsks: why do you have ftype not enabled somewhere?17:53
larsksmwhahaha: I am working with a partner who has root filesystems with ftype=0 which is constantly tripping things up.17:54
panda|ruckweshay: yes, means everywhere in master now, except for scenario00717:54
openstackgerritOliver Walsh proposed openstack/tripleo-common master: Skip ssh_known_hosts tasks in check mode  https://review.openstack.org/63940217:54
larsksJust checking /var/lib/docker seems better than telling someone to reformat all their servers.17:54
mwhahahalarsks: we added that logic for some reason17:54
mwhahahabasically if we ever move the container bits (which we will)17:54
larsksYes, it's because docker requires ftype=1 and won't run without it.17:54
larsks...but the existing check causes everything to blow in cases where the system is actually configured appropriately.17:55
larsksIt will trigger on *any* filesystem that happens to have ftype=0.17:55
mwhahahalarsks: why are they running ftype=0 at all17:55
larsksThe change adds a variable (container_registry_docker_fs) which is the directory path we care about.17:55
larsksmwhahaha: I cannot answer that question.17:55
*** bfournie has joined #tripleo17:55
mwhahahalarsks: well you should answer it, because unless this is an upgrade they shouldn't be running it17:56
mwhahahathey can't do anything contrainers that on those disks17:56
mwhahahaer containers17:56
larsksThe *root* filesystem as ftype=0, but /var/lib/docker does not.17:56
larsksEverything works *fine* on these systems.17:56
mwhahahawell that's dangerous because it'll need to be /var/lib/containers in a future release17:56
larsks...in which case we would just change the continer_registry_docker_fs variable.17:56
mwhahahathe point is that it's going to cause them problems and i'd rather not open that up unless there' sa techincal reason17:56
*** trown is now known as trown|lunch17:57
larsksThis just seems *really* broad.  What if you have an old disk mounted with critical data that you can't reformat to be ftype=1? It has nothing to do with docker -- maybe it's /srv/somethingorother -- but it will cause the deploy to bail.17:57
mwhahahathat would be the upgrade problem17:58
larsksI wouldn't mind if it were possible to upgrade an xfs filesyste, but I don't think it is.17:58
mwhahahathere isn't which is why it's important not to install on a system that isn't ftype=117:58
mwhahahaso no, this is critical for the container stuff and we cannot be certain we won't need other paths17:58
larsksI honestly don't see the problem here: with this change, we're still checking for an appropriate filesystem configuration. We're just targeting the filesystem on which we're installing containers, rather than looking at *all* mounted filesystems.17:59
larsksIf the container location changes, we can update the check to match.17:59
mwhahahabut it's too late for customers who already installed18:00
mwhahahathat's the problem18:00
mwhahahaupdating the path solves nothing because there is no fix18:00
mwhahahathis is a UX thing to prevent folks from being screwed in teh future18:00
larsksI'm not following you there.  This check will cause the deploy to fail in situtations in which it should fail.  If someone has already installed...I don't get it; either their install is already broken or I am missing something?18:00
mwhahahai'm sorry but you should ask the partner why ftype=0, it's been =1 since 7.318:00
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: container-image-prepare: redirect all output to logfile  https://review.openstack.org/63940518:01
mwhahahalarsks: if i create a /var/lib/docker that's ftype=1 and everything is ftype=0, upgrades to podman will fail in stein18:01
mwhahahaand user cannot fix their environment18:01
larsksAre we only running this check on initial install and not on upgrade?18:01
larsksOr wouldn't we catch it during the upgrade (if we were to check the new container location)?18:01
mwhahahawe run it all the time, it's part of the container setup18:01
larsksOkay, so the upgrade to podman should be fine, right? Because we'd check the new location and it would either have ftype=1 or not,  and in that case it would fail.18:02
*** wolverineav has quit IRC18:02
mwhahahabut there's no recovery for folks who don't have more disks18:02
*** wolverineav has joined #tripleo18:03
mwhahahasorry but i don't think we should be adjusting this to be less restrictive18:03
larsksI apologize, I may have misunderstood your initial response. I asked if this check could be limited to /var/lib/docker and you said "yes", and I took that as an indication that you would be okay with the change.  I'll drop it.18:03
*** wolverineav has quit IRC18:03
mwhahahawe went thought many iterations of this check, can you get away with it in your case? sure. should we be lifting the restrictions for everyone? no.18:05
mwhahahalarsks: for the sake of the partner it would be better to find out if there's an actual reason for their disk format (performance/security/etc) and then we could investigate more. but an end user can get in a tricky/unrecoverable scenario18:06
*** thrash is now known as thrash|biab18:06
mwhahahasince you can't upgrade xfs, it's really not a good idea to leave it on ftype=0 for new systems18:07
*** cfontain has quit IRC18:09
*** florianf_ has quit IRC18:12
*** itlinux_ has quit IRC18:13
*** kopecmartin is now known as kopecmartin|off18:14
openstackgerritwes hayutin proposed openstack/tripleo-quickstart-extras master: add image sharing instructions  https://review.openstack.org/63940718:14
hamzyI am running RHOSP14.  If docker ps says nova_libvirt is up and healthy, then systemd doesn't control libvirtd right?  What is necessary to restart libvirt? I did "for i in nova_compute nova_libvirt; do sudo docker restart $i; done" on the compute node and "for i in nova_conductor nova_compute nova_api nova_scheduler; do sudo docker restart $i; done" on the controller node.18:15
fultonjpanda|ruck: thanks for explaining "branchful jobs" it's a good horror story. i'll tell my kids at night18:15
*** jpena is now known as jpena|off18:15
*** chandankumar is now known as raukadah18:16
*** chem` has quit IRC18:16
fultonjhamzy: it's better to use paunch to start and stop the containers18:16
mwhahahawaht no18:17
mwhahahadocker restart should be sufficient18:17
*** itlinux has joined #tripleo18:17
mwhahahathat's for running containers that don't exist18:17
mwhahahaonce they exist, docker restart should be fine18:17
fultonjthat's true18:18
hamzyhrm... I am seeing "error : virNetSocketNewConnectUNIX:712 : Failed to connect socket to '/var/run/libvirt/virtlogd-sock': Connection refused" in the syslog18:18
fultonjhe wanted to modify the container and see a change to it18:18
fultonj_I assumed_ he wanted to modify the container and see a change to it18:18
hamzywell restart really18:18
*** wolverineav has joined #tripleo18:18
*** rbrady has quit IRC18:18
hamzyI see libvirt complaining "error: kvm run failed Device or resource busy. This is probably because your SMT is enabled." so I disabled SMT and am trying to restart libvirt18:19
openstackgerritGabriele Cerami proposed openstack-infra/tripleo-ci master: Removing queens and pike from the negative match for ceph-loop role  https://review.openstack.org/63940818:19
panda|ruckfultonj: ^18:20
fultonjpanda|ruck: ack18:20
mwhahahahamzy: owalsh / mschuppert may be able to help you with libvirt problems18:20
*** wolverineav has quit IRC18:22
openstackgerritJohn Fulton proposed openstack/tripleo-heat-templates stable/rocky: Switch scenario00{1,4}-standalone to Ceph bluestore  https://review.openstack.org/63935118:22
*** wolverineav has joined #tripleo18:22
openstackgerritJohn Fulton proposed openstack/tripleo-heat-templates stable/queens: Switch scenario00{1,4}-standalone to Ceph bluestore  https://review.openstack.org/63935218:22
fultonjpanda|ruck: ^18:22
*** wolverineav has quit IRC18:23
openstackgerritNagasai Vinaykumar Kapalavai proposed openstack/puppet-tripleo master: metrics_qdr: TLS support.  https://review.openstack.org/63129718:23
openstackgerritNagasai Vinaykumar Kapalavai proposed openstack/tripleo-heat-templates master: metrics_qdr: Add TLS support  https://review.openstack.org/63073618:23
hamzyooo... mwhahaha owalsh mschuppert, I saw something about virtlogd being a requirement so it looks like "for i in nova_compute nova_virtlogd nova_libvirt; do sudo docker restart $i; done" worked!18:24
*** derekh has quit IRC18:25
openstackgerritJohn Fulton proposed openstack/tripleo-heat-templates stable/queens: Switch scenario00{1,4}-multinode-containers to Ceph bluestore  https://review.openstack.org/63935218:26
*** wolverineav has joined #tripleo18:26
openstackgerritJohn Fulton proposed openstack/tripleo-heat-templates stable/rocky: Switch scenario00{1,4}-multinode-containers to Ceph bluestore  https://review.openstack.org/63935118:26
panda|ruckfultonj: thanks!18:28
fultonjthank you panda|ruck18:28
*** panda|ruck is now known as panda|ruck|off18:28
*** sdake_ has quit IRC18:30
*** sdake has joined #tripleo18:32
openstackgerritNagasai Vinaykumar Kapalavai proposed openstack/puppet-tripleo master: metrics_qdr: TLS support.  https://review.openstack.org/63129718:32
*** sdake has quit IRC18:35
*** sdake has joined #tripleo18:37
openstackgerritNagasai Vinaykumar Kapalavai proposed openstack/tripleo-heat-templates master: metrics_qdr: Add TLS support  https://review.openstack.org/63073618:39
openstackgerritMerged openstack/puppet-tripleo master: Support cephfs_volume_mode parameter  https://review.openstack.org/63877318:43
openstackgerritMerged openstack/tripleo-puppet-elements master: Release note for chrony/ntp switch  https://review.openstack.org/63916318:43
*** sdake has quit IRC18:46
*** rfolco|rover has quit IRC18:46
*** dprince has quit IRC18:48
*** _fragatina has quit IRC18:50
*** sdake has joined #tripleo18:51
*** thrash|biab is now known as thrash18:52
*** sdake has quit IRC18:55
*** sdake_ has joined #tripleo18:56
*** sdake_ has quit IRC18:56
openstackgerritMarc Methot proposed openstack/tripleo-heat-templates master: Adding configurability to the amphora image format  https://review.openstack.org/63941618:57
*** sdake has joined #tripleo18:58
*** sanjayu_ has quit IRC18:59
openstackgerritTom Barron proposed openstack/puppet-tripleo stable/rocky: Support cephfs_volume_mode parameter  https://review.openstack.org/63941718:59
openstackgerritwes hayutin proposed openstack/tripleo-quickstart-extras master: Script to identify tests passing on FS021 but still on skip list  https://review.openstack.org/63752419:00
*** sdake has quit IRC19:00
*** sdake_ has joined #tripleo19:02
*** wolverineav has quit IRC19:03
openstackgerritTom Barron proposed openstack/tripleo-heat-templates master: Support cephfs_volume_mode parameter  https://review.openstack.org/63877519:03
*** wolverineav has joined #tripleo19:04
*** sdake_ has quit IRC19:06
*** sdake has joined #tripleo19:06
*** ramishra_ has quit IRC19:09
*** sdake has quit IRC19:10
VorrtexHey all, I'm trying to get tox running smoothly for tripleo-common on my mac, and I'm not sure how to approach this error:  "ERROR: Error creating virtualenv. Note that some special characters (e.g. ':' and unicode symbols) in paths are not supported by virtualenv. Error details: InvocationError("Failed to get version_info for python3.6: pyenv: python3.6: command not found\n\nThe `python3.6' command exists in these Python19:11
Vorrtex versions:\n  3.6.3\n\n", None)"19:11
*** rfolco has joined #tripleo19:12
*** sdake has joined #tripleo19:12
Vorrtexpy37 works fine, my local version of python is installed via pyenv and is version 3.5.5, and I'm finding that a lot of posts are talking about pyenv having local version clashes, but I can't seem to find where exactly that is happening.  Any ideas?19:13
mwhahahasounds like an error in tox19:14
mwhahahado you have some weird path that you're keeping tripleo-common in?19:14
VorrtexI place it here:  ~/workspace/openstack/tripleo-common19:14
Vorrtexand run tox there as well.19:15
VorrtexI've done tox -r -e py36 (fails), and if I do just a tox -r, it completes everything successfully EXCEPT py3619:15
mwhahahano idea19:15
VorrtexI suppose I could try again with a fresh clone of master or something, but I haven't changed anything on my master branch, outside of pulling latest this morning19:16
mwhahahait doesn't sound like it's getting to the code itself, but rather than environment setup19:16
mwhahahado you have python3.6 installed?19:16
mwhahahathat error seems to point that you're pyenv isn't currently set to 3.19:17
VorrtexMy pyenv is locally set to 3.5.  Not sure why 3.7 would pass if that was an issue.  Also, I thought tox installed its own python virtual environments for whatever version was needed to run?  Bad assumption on my part?19:18
*** trown|lunch is now known as trown19:18
mwhahahais 3.7 the system python19:18
mwhahahaI bet if you pyenv local 3.6.3 in tripleo-common and then tox -e py36 it'd work19:18
Vorrtex3.6.3 is the global19:18
mwhahaharight but the local wins19:19
VorrtexOkay, but that doesn't match up then, why does 3.7 work if the local would win?19:19
mwhahahamagic? no idea what your system setup is19:19
*** itlinux_ has joined #tripleo19:19
Vorrtexha ha, sorry, I don't mean to attack, just looking for understanding19:20
*** sdake has quit IRC19:20
*** gouthamr has quit IRC19:20
*** gouthamr_ has joined #tripleo19:20
*** itlinux has quit IRC19:21
*** sdake_ has joined #tripleo19:22
VorrtexI'm removing my global definition to see if anything changes... Maybe it was clashing with that one.19:22
*** sdake_ has quit IRC19:25
VorrtexDoesn't seem to have changed anything... I'll try cleaning up some other virtual environments.19:28
*** sdake has joined #tripleo19:28
*** rfolco is now known as rfolco|rover19:28
*** ykarel has quit IRC19:33
*** sdake has quit IRC19:35
*** sdake_ has joined #tripleo19:36
*** wolverineav has quit IRC19:42
weshaycgoncalves can we get a wf on https://review.openstack.org/#/c/639083/ ?19:44
*** sdake_ has quit IRC19:45
*** sdake has joined #tripleo19:47
*** ekultails has quit IRC19:47
*** hberaud is now known as hberaud|gone19:48
openstackgerritNagasai Vinaykumar Kapalavai proposed openstack/tripleo-heat-templates master: metrics_qdr: Add TLS support  https://review.openstack.org/63942319:48
*** dprince has joined #tripleo19:48
weshayrlandy mwhahaha https://review.openstack.org/#/c/639341/19:50
weshaysaves about 25mb of space19:50
mwhahahatoo bad atop is actually useful19:51
mwhahahabut sure19:51
weshaymwhahaha we're going to keep in containers-multinode and ovb on rdo19:51
weshayso if you hit perf issues just check the 3rd party jobs19:51
*** itlinux_ has quit IRC19:52
*** sdake has quit IRC19:55
*** itlinux has joined #tripleo19:56
*** wolverineav has joined #tripleo19:56
*** sdake has joined #tripleo19:57
*** derekh has joined #tripleo19:58
*** wolverineav has quit IRC20:01
*** sdake has quit IRC20:03
Vorrtexmwhahaha so now I blew away my virtual environment with 3.6.3 (and any venvs stemming from it), and ended up with the following error:  "ERROR:  py36: InterpreterNotFound: python3.6"20:03
VorrtexSo now I'm really at a loss... its like tox doesn't know what python3.6 is.20:03
*** wolverineav has joined #tripleo20:04
*** ekultails has joined #tripleo20:06
openstackgerritAlex Schultz proposed openstack/tripleo-common master: Modify roles to remove unused services  https://review.openstack.org/62301220:07
*** wolverineav has quit IRC20:09
*** derekh has quit IRC20:09
*** itlinux has quit IRC20:16
*** itlinux has joined #tripleo20:20
openstackgerritwes hayutin proposed openstack-infra/tripleo-ci master: clean up the doc footer in tripleo jobs  https://review.openstack.org/63943020:26
*** dciabrin has quit IRC20:27
*** dciabrin has joined #tripleo20:28
openstackgerritwes hayutin proposed openstack/tripleo-quickstart-extras master: Adding test to the skiplist for OVN related updates  https://review.openstack.org/63938620:31
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: upgrade2podman: prune everything in Docker  https://review.openstack.org/62040520:32
*** wolverineav has joined #tripleo20:37
*** gouthamr_ is now known as gouthamr20:42
*** wolverineav has quit IRC20:42
mwhahahaweshay: did we ever log about for the image building failing on OVB jobs? https://logs.rdoproject.org/86/638286/4/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/b59490e/logs/undercloud/home/zuul/overcloud_image_build.log.txt.gz#_2019-02-26_18_42_2820:43
mwhahahaseems to be happening quite frequently as of late20:44
mwhahahai think it's on the ironic-python-agent20:44
mwhahahawonder if there's a real bug tehre20:44
weshayso.. rdo and especially fs001 where we see that was all messed up yesterday.. jobs were running at like 20% success rate20:44
weshayI've added the error signature to sova to help figure any patterns this morning..20:45
weshayI will continue to debug it w/ rfolco|rover and panda|ruck|off20:45
weshaymwhahaha we have it in alerts.. but alerts are not working in the channel atm20:46
openstackgerritHarry Rybacki proposed openstack/tripleo-heat-templates stable/queens: Move ipa enrollment to host_prep_tasks  https://review.openstack.org/63852320:56
*** itlinux has quit IRC20:59
openstackgerritLee Yarwood proposed openstack/tripleo-heat-templates master: WIP nova: Add nova-status upgrade check call during deployment  https://review.openstack.org/63513921:02
*** spsurya has quit IRC21:03
openstackgerritSteve Baker proposed openstack/tripleo-common master: Expunge the docker uploader  https://review.openstack.org/63943821:07
*** itlinux has joined #tripleo21:07
*** cfontain has joined #tripleo21:08
*** hjensas has quit IRC21:19
openstackgerritNagasai Vinaykumar Kapalavai proposed openstack/puppet-tripleo master: metrics_qdr: TLS support.  https://review.openstack.org/63129721:20
*** wolverineav has joined #tripleo21:21
*** wolverineav has quit IRC21:21
openstackgerritNagasai Vinaykumar Kapalavai proposed openstack/tripleo-heat-templates master: metrics_qdr: Add TLS support  https://review.openstack.org/63942321:21
*** wolverineav has joined #tripleo21:21
*** hjensas has joined #tripleo21:22
*** dxiri has quit IRC21:23
*** derekh has joined #tripleo21:24
*** derekh has quit IRC21:24
*** zbr|out has quit IRC21:27
*** ansmith has quit IRC21:31
*** raildo has quit IRC21:33
*** jcoufal has quit IRC21:37
fultonjweshay: the patches to fix 001 and 004 are now green in CI https://review.openstack.org/#/c/639351 https://review.openstack.org/#/c/63935221:38
weshayfultonj nice21:38
fultonji think we just need some reviews21:40
*** cfontain has quit IRC21:41
weshaymwhahaha might hook you up ^ :)21:43
*** jtomasek has quit IRC21:43
mwhahahachanging stable ci :/21:44
mwhahahasuch is life i guess21:44
fultonjthank you both21:45
*** wolverineav has quit IRC21:46
openstackgerritMerged openstack/python-tripleoclient stable/rocky: Check proxy env vars  https://review.openstack.org/63871021:46
openstackgerritAlex Schultz proposed openstack/tripleo-heat-templates master: Make heat yaql limits configurable  https://review.openstack.org/63945021:47
*** wolverineav has joined #tripleo21:48
*** wolverineav has quit IRC21:48
*** wolverineav has joined #tripleo21:48
fultonjweshay: do you want to review (the depends on of those)? https://review.openstack.org/#/c/639408/21:48
weshayI'm a +2 whore21:49
openstackgerritMerged openstack/python-tripleoclient master: container image build: introduce --use-buildah  https://review.openstack.org/63432221:50
weshayoh snap ^21:50
fultonjthanks weshay21:51
weshayfultonj I may lean on you a bit on where to put some extra coverage upstream21:52
*** paramite|clone has quit IRC21:52
fultonjweshay: ok21:52
*** itlinux has quit IRC21:53
*** mjturek has quit IRC21:54
openstackgerritDavid Peacock proposed openstack/tripleo-heat-templates master: flatten database service Redis  https://review.openstack.org/63593021:55
*** dprince has quit IRC21:59
*** cfontain has joined #tripleo22:02
*** itlinux has joined #tripleo22:04
weshaymwhahaha that image build is working for me locally atm.. I guess I may have to rerun, rerun22:05
weshayjust the image build22:05
*** rcernin has joined #tripleo22:05
*** abishop has quit IRC22:05
weshayit's never worked on your patch I see 63828622:06
mwhahahawhich is weird cause that's a patch for containers not overcloud images22:08
weshayya.. totally.. rerunning..  few min22:08
*** xek has quit IRC22:08
*** wolverineav has quit IRC22:08
*** itlinux has quit IRC22:08
weshaymwhahaha got a hit on another box22:10
*** wolverineav has joined #tripleo22:10
weshayrecreated.. can tmate if you want22:10
mwhahahaseems like a race condition or something22:10
weshaymwhahaha I suppose I can just run the disk-image-create command22:12
weshaythis is from some tmp dir? '-o', './ironic-python-agent'22:12
mwhahahayou could use the image build command to rerun22:13
* mwhahaha digs up docs22:13
weshaydisk-image-create -a amd64 -o ./ironic-python-agent -t qcow2 -p yum-plugin-prioritiespython-hardware-detect --min-tmpfs=7 centos7 selinux-permissive ironic-agent ironic-agent-multipath dynamic-login element-manifest network-gateway enable-packages-install override-pip-and-virtualenv22:14
* weshay reads22:14
*** wolverineav has quit IRC22:14
mwhahahayea you could try the disk-image-create22:15
*** Vorrtex has quit IRC22:15
mwhahahawonder which element it's failing tho22:16
mwhahahaseems to be the ironic-agent one22:16
*** itlinux has joined #tripleo22:17
*** wolverineav has joined #tripleo22:17
weshayya.. -o is output.. I need the right dib path22:17
*** slaweq has quit IRC22:18
weshaythat has to be in the log https://logs.rdoproject.org/86/638286/4/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/b59490e/logs/undercloud/home/zuul/overcloud_image_build.log.txt.gz#_2019-02-26_18_42_3222:18
mwhahahait seems like it's rdocloud22:20
mwhahahacause i don't think i've ever seen that error for the buildimage job22:21
mwhahahaelemens are tripleo-common/t-p-e/t-i-e/heat-agents22:21
* weshay is missing shit http://pastebin.test.redhat.com/72410322:22
* weshay reads22:22
weshaymwhahaha it's not rdo cloud.. I'm on a libvirt node22:22
mwhahahaok then22:22
mwhahahatechincally it's openstack overcloud image build22:22
mwhahahawe might do some stuff under the covers in tripleoclient22:23
weshaysure.. that's what I started w/22:23
* weshay trying to get a real trace22:23
*** bfournie has quit IRC22:23
weshayor log or something22:23
weshaybeen awhile since I messed w/ the image builds22:23
mwhahahapfffft useful logging22:23
mwhahahawho does that22:23
* weshay tries to set the ELEMENT path22:23
*** itlinux has quit IRC22:24
*** itlinux has joined #tripleo22:26
*** slaweq has joined #tripleo22:26
mwhahahai thought there was a patch to DIB to not erase git22:28
mwhahahafrom sorin22:28
mwhahahawonder if we need promotion22:28
*** sdake has joined #tripleo22:29
weshaythere was22:29
weshayI remember that patch22:29
mwhahahacause that's what it's failing on22:29
mwhahahawas from quiquell|off22:29
weshaymissing heat-agents path atm22:29
mwhahahaits i think heat-agents is /usr/share/openstack-heat-agents or something22:29
weshayhrm.. it's not on my box atm22:30
* weshay pulls it22:30
mwhahahawhich would be another reason why it's only ovb22:30
mwhahahacause the other job uses source22:30
*** slaweq has quit IRC22:30
*** sdake has quit IRC22:30
*** itlinux has quit IRC22:31
*** sdake has joined #tripleo22:32
weshayk... it's running22:32
mwhahahait's failing somewhere around https://github.com/openstack/diskimage-builder/blob/e796b3bc1884cbb0a7259be486d835ca114cca9e/diskimage_builder/elements/yum/bin/install-packages#L14522:32
mwhahahawhich seems weird cause it's the end22:33
mwhahahaso something else is failing22:33
weshayfak.. http://paste.openstack.org/show/746371/22:35
* weshay checks image22:35
mwhahaharemove the image and try again22:35
weshayrunning again.. because I don't see the qcow22:37
*** cfontain has quit IRC22:37
weshayit HAS ONE JOB22:37
openstackgerritAlex Schultz proposed openstack/tripleo-heat-templates master: Make heat yaql limits configurable  https://review.openstack.org/63945022:39
mwhahahadat race condition22:39
weshayis that a clean exit?22:39
weshayno image created22:39
mwhahahaoh no it's a fail22:40
*** _fragatina has joined #tripleo22:40
mwhahahamissing a space22:40
mwhahahain your cmdline22:40
weshayah see that22:40
*** sdake has quit IRC22:40
*** sdake_ has joined #tripleo22:41
* weshay missing element python-hardware-detect22:41
* weshay looks22:41
weshaythat package is not avail to me22:43
*** cfontain has joined #tripleo22:43
mwhahahadon't build python3?22:43
weshayya.. just codesearching22:43
mwhahahaalso includes it, weird22:43
mwhahahaanyway i'm kikcing off a full build i guess22:43
weshaymwhahaha where is the element for that though?22:44
mwhahahaactually i'll just cut it down to the ironic-python-agent22:44
mwhahahathat's not an element, it's a package22:44
*** sdake_ has quit IRC22:45
weshayok.. trying w/o22:46
*** sdake has joined #tripleo22:47
mwhahahais how to use the regular build process and only build the ipa22:47
* weshay is an idiot22:47
weshaythat's the easy way :)22:48
mwhahahaset -ex; while true; do openstack overcloud image build --config ipa.yaml; done22:48
mwhahahaactually i think you'd need to remove the output but anyway22:49
weshayweird..  it throws this..22:49
weshay diskimage_builder.element_dependencies.MissingElementException: Element 'yum-plugin-priorities' not found22:49
*** cfontain has quit IRC22:49
*** itlinux has joined #tripleo22:50
weshayI need a comman, not a space or something22:50
mwhahahaoh it's a comma22:50
mwhahahayea i just looked in my output22:50
mwhahahafailed first try fo rme22:51
weshaydecent trace or no?22:51
mwhahahaof course not22:51
weshayk.. /me trying again22:52
weshayreturn code 1, no image no error22:53
* weshay checks the command for verbose22:53
mwhahahaOperation failed: No such file or directory22:53
mwhahahamy least favorite error22:53
mwhahahaWHAT FILE22:53
weshayno debug no verbose?22:54
mwhahahanot for that command22:54
mwhahahai think dib has one22:54
weshayI'm looking the help and doc22:55
*** sdake has quit IRC22:56
mwhahahai wonder if it's the rm -rf /tmp/ironic-python-agent22:56
* mwhahaha tries || true22:56
* weshay has to go22:56
mwhahahaexport DIB_DEBUG_TRACE=122:56
*** itlinux has quit IRC22:57
*** sdake has joined #tripleo22:58
*** agopi has quit IRC22:59
mwhahahasystemctl enable openstack-ironic-python-agent.service23:00
*** Chaserjim has quit IRC23:00
mwhahahafailing on that23:00
*** sdake has quit IRC23:00
*** sdake has joined #tripleo23:02
*** dciabrin has quit IRC23:02
*** tkajinam has joined #tripleo23:02
*** dciabrin has joined #tripleo23:02
*** sdake has quit IRC23:06
*** sdake_ has joined #tripleo23:07
mwhahahaianw: you around?23:07
mwhahahaianw: we're getting ipa failures on build looks like it's erroring enabling the serivce which is weird. not sure if you have any idea, see https://bugs.launchpad.net/tripleo/+bug/181737023:08
openstackLaunchpad bug 1817370 in tripleo "disk-image-create fails in ovb jobs intermittently" [Critical,Triaged]23:08
*** ansmith has joined #tripleo23:09
*** sdake_ has quit IRC23:10
*** sdake has joined #tripleo23:11
mwhahahak so it's failing somewhere around https://github.com/openstack/diskimage-builder/blob/master/diskimage_builder/elements/ironic-agent/post-install.d/80-ironic-agent#L2623:12
mwhahaha4 year old code23:14
* mwhahaha sighs23:14
*** rascasoft has quit IRC23:15
*** dciabrin has quit IRC23:15
*** dciabrin_ has joined #tripleo23:15
*** sanjayu_ has joined #tripleo23:19
*** sdake has quit IRC23:20
*** gfidente|afk has quit IRC23:22
*** sdake has joined #tripleo23:23
*** sdake has quit IRC23:25
*** sdake has joined #tripleo23:26
*** ade_lee has quit IRC23:27
*** sdake has quit IRC23:30
*** sdake has joined #tripleo23:33
*** sdake has quit IRC23:35
mwhahahaweshay: we need a promotion, it's the git removal23:36
*** cfontain has joined #tripleo23:37
*** sdake has joined #tripleo23:37
mwhahahawell that solves the one23:38
mwhahahaand likely why we used to have diskimage-builder in the ci excludes so we always got the latest version23:38
*** aedc has joined #tripleo23:39
*** mwhahaha changes topic to "CI Status: GREENish but OVB broken building images (need promotion) | https://docs.openstack.org/tripleo-docs/latest/"23:39
*** sdake has quit IRC23:40
*** sdake has joined #tripleo23:42
*** sdake has quit IRC23:50
*** sdake has joined #tripleo23:52
*** vinaykns has quit IRC23:54
*** sdake has quit IRC23:55
*** ade_lee has joined #tripleo23:56
*** sdake has joined #tripleo23:56

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!