Monday, 2018-02-19

*** arxcruz has quit IRC00:01
*** arxcruz has joined #tripleo00:01
*** arxcruz has quit IRC00:06
*** ooolpbot has joined #tripleo00:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION00:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/174964100:10
*** ooolpbot has quit IRC00:10
openstackLaunchpad bug 1749641 in tripleo "Overcloud deployment failing in promotion jobs at glance-manage db_sync" [Critical,In progress] - Assigned to Christian Schwede (cschwede)00:10
mwhahahaarxcruz|ruck: looks like shaker got it00:11
mwhahahasbaker even00:11
*** arxcruz has joined #tripleo00:11
*** _jbadiapa has quit IRC00:13
*** arxcruz has quit IRC00:15
*** arxcruz has joined #tripleo00:17
*** gvrangan_ has joined #tripleo00:20
*** arxcruz has quit IRC00:20
*** tosky has quit IRC00:26
*** panda has joined #tripleo00:26
*** gvrangan_ has quit IRC00:32
*** gvrangan has joined #tripleo00:33
*** panda is now known as panda|off00:38
*** maeca has joined #tripleo00:42
*** gvrangan has quit IRC00:48
*** gvrangan has joined #tripleo00:53
*** ooolpbot has joined #tripleo01:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION01:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/174964101:10
openstackLaunchpad bug 1749641 in tripleo "Overcloud deployment failing in promotion jobs at glance-manage db_sync" [Critical,In progress] - Assigned to Christian Schwede (cschwede)01:10
*** ooolpbot has quit IRC01:10
*** dmacpher_ has quit IRC01:13
*** psahoo has joined #tripleo01:15
*** dhill__ has quit IRC01:18
openstackgerritMerged openstack-infra/tripleo-ci master: Collect /var/lib/docker-puppet/ in logs  https://review.openstack.org/54536701:19
*** gvrangan has quit IRC01:40
*** zshi has joined #tripleo01:50
*** itlinux has quit IRC02:02
*** yamahata has joined #tripleo02:03
*** ooolpbot has joined #tripleo02:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION02:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/174964102:10
*** ooolpbot has quit IRC02:10
openstackLaunchpad bug 1749641 in tripleo "Overcloud deployment failing in promotion jobs at glance-manage db_sync" [Critical,In progress] - Assigned to Christian Schwede (cschwede)02:10
*** links has joined #tripleo02:11
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Restrict SNMP to internal network  https://review.openstack.org/54453002:24
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Revert "Disable SNMP service in all CI jobs"  https://review.openstack.org/54499702:24
*** agopi has quit IRC02:36
*** agopi has joined #tripleo02:36
openstackgerritMerged openstack/tripleo-quickstart-extras master: Adding queens release  https://review.openstack.org/54570802:44
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart master: Run ovb-ha (fs001) with a containerized undercloud  https://review.openstack.org/54255602:46
*** sai___ has joined #tripleo02:47
*** dmacpher has joined #tripleo02:48
*** rlandy has quit IRC02:52
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Add enpoints for TripleO UI Config  https://review.openstack.org/52867903:05
*** ooolpbot has joined #tripleo03:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION03:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/174964103:10
*** ooolpbot has quit IRC03:10
openstackLaunchpad bug 1749641 in tripleo "Overcloud deployment failing in promotion jobs at glance-manage db_sync" [Critical,In progress] - Assigned to Christian Schwede (cschwede)03:10
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Add enpoints for TripleO UI Config  https://review.openstack.org/52867903:13
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: docker: add support for TripleO UI  https://review.openstack.org/51549003:15
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: docker: add support for TripleO UI  https://review.openstack.org/51549003:15
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: docker: add support for TripleO UI  https://review.openstack.org/51549003:16
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart master: Run ovb-ha (fs001) with a containerized undercloud  https://review.openstack.org/54255603:21
*** cshastri has joined #tripleo03:32
*** masco has joined #tripleo03:33
*** maeca has quit IRC03:40
*** maeca has joined #tripleo03:42
*** dpawar has joined #tripleo03:49
*** dpawar_ has joined #tripleo03:58
*** udesale has joined #tripleo03:59
*** dpawar has quit IRC04:02
*** dpawar_ has quit IRC04:09
*** ooolpbot has joined #tripleo04:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION04:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/174964104:10
*** ooolpbot has quit IRC04:10
openstackLaunchpad bug 1749641 in tripleo "Overcloud deployment failing in promotion jobs at glance-manage db_sync" [Critical,In progress] - Assigned to Christian Schwede (cschwede)04:10
*** shreshtha has joined #tripleo04:12
*** ykarel has joined #tripleo04:14
*** moshele has joined #tripleo04:18
openstackgerritEmilien Macchi proposed openstack/tripleo-common master: container-images: support cont. u/c w/ HAproxy & keepalived  https://review.openstack.org/54574404:19
*** karthiks has joined #tripleo04:20
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart master: Run ovb-ha (fs001) with a containerized undercloud  https://review.openstack.org/54255604:20
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart master: Run ovb-ha (fs001) with a containerized undercloud  https://review.openstack.org/54255604:21
*** moshele has quit IRC04:23
*** maeca has quit IRC04:24
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: docker: add support for TripleO UI  https://review.openstack.org/51549004:24
openstackgerritMerged openstack/diskimage-builder master: Install systemd earlier for Ubuntu Bionic  https://review.openstack.org/54561104:31
*** karthiks has quit IRC04:34
*** psachin has joined #tripleo04:47
*** karthiks has joined #tripleo04:48
*** sai___ has quit IRC04:57
*** aditya_r has joined #tripleo04:59
*** aditya_r has quit IRC05:00
*** akane_ has joined #tripleo05:03
*** olivierbourdon38 has joined #tripleo05:04
Tenguhello there05:09
*** ooolpbot has joined #tripleo05:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION05:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/174964105:10
*** ooolpbot has quit IRC05:10
openstackLaunchpad bug 1749641 in tripleo "Overcloud deployment failing in promotion jobs at glance-manage db_sync" [Critical,In progress] - Assigned to Christian Schwede (cschwede)05:10
*** anilvenkata has joined #tripleo05:10
*** dpawar has joined #tripleo05:11
*** dpawar has quit IRC05:12
*** dpawar has joined #tripleo05:12
*** pdeore has joined #tripleo05:18
*** pdeore has quit IRC05:18
openstackgerritEmilien Macchi proposed openstack/tripleo-common master: container-images: support cont. u/c w/ HAproxy & keepalived  https://review.openstack.org/54574405:20
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart master: Run ovb-ha (fs001) with a containerized undercloud  https://review.openstack.org/54255605:20
*** masco has quit IRC05:21
*** dpawar has quit IRC05:31
*** shreshtha_ has joined #tripleo05:32
*** moshele has joined #tripleo05:33
*** shreshtha has quit IRC05:34
*** janki has joined #tripleo05:38
EmilienMbandini, cschwede: when you're up, feel free to look https://review.openstack.org/#/c/545623/ (probably need more work but you get the idea...) not sure if that's the right way, but at least you know about this patch. Also feel free to take over.05:45
*** mdnadeem has joined #tripleo05:47
*** shrjoshi has joined #tripleo05:49
*** ramishra has joined #tripleo05:50
*** shreshtha_ has quit IRC05:52
EmilienMpanda|off, mandre: I'm not sure what to do with https://bugs.launchpad.net/tripleo/+bug/1750321 but you probably know better. Thanks for the help06:01
openstackLaunchpad bug 1750321 in tripleo "centos-binary-keepalived container is missing in dockerhub" [High,Triaged]06:01
*** skramaja has joined #tripleo06:03
dtrainorhowdy, Tengu06:07
Tenguhello dtrainor :)06:08
dtrainorgot my ceph deployment figured out, now trying to figure out why the overcloud is having issues with it06:08
Tenguoh. did you deploy the ceph after a complete ovecloud deploy?06:09
Tengu(or even a failed one due to ceph issues)06:09
dtrainorno, as part of the deployment06:09
dtrainoriirc i was talking with you about an ssl trust issue06:09
Tenguyup06:09
dtrainori'm trying to create images and they're all stuck in a "Saving" status06:10
Tenguhad some issues with ceph as well, especially when I add OSDs after "main" deploy (as a stack update)06:10
*** ooolpbot has joined #tripleo06:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION06:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/174964106:10
openstackLaunchpad bug 1749641 in tripleo "Overcloud deployment failing in promotion jobs at glance-manage db_sync" [Critical,In progress] - Assigned to Christian Schwede (cschwede)06:10
*** ooolpbot has quit IRC06:10
dtrainoroh, that sounds like a challenge06:10
Tengu:)06:10
*** threestrands has quit IRC06:10
Tengufact was: my lab ceph VM didn't want to boot correctly in PXE when I attached all the volumes, and if I attach them after the main deploy, OSDs weren't correctly recognized -.-. they were mounted/configured, but apparently there were some issues at some point that prevented any ceph usage.06:12
Tengutempest wasn't happy, and I wasn't happy.06:12
dtrainori see06:12
Tenguin the end, I sorted out the issue in the VM xml file (libvirt), and all was fine and nice and working. the only issue now with tempest: we don't have swift, as we're using rados-gw over ceph forobject store.06:12
Tenguoh and a network glitch due to the way I route the lab. well. that one will probably crash my brain.06:13
dtrainorheh06:16
dtrainormy home lab is ridiculous, i feel your pain06:16
Tengudtrainor: well, imagine a double-NAT network (and a third layer for compute/ceph as they are routed through the undercloud for "internet access")06:17
Tengudtrainor: that's the content of an email I sent a few days ago on the dev ML in fact. the DoubleNO³ :). had to scratch my head some times in order to get a working design, and a working network - still, due to some specificities we have at work, I have some issues -.-.06:19
*** masco has joined #tripleo06:21
dtrainorhah.06:21
dtrainorwell.  this is driving me nuts.06:22
Tengu:)06:23
Tenguare you able to create volumes (cinder) on ceph?06:23
*** threestrands has joined #tripleo06:23
Tenguhave you checked the ceph access from controller(s), and ceph state and so on?06:24
*** jbadiapa has joined #tripleo06:24
*** dparkes_ has quit IRC06:28
dtrainorFailed to run task cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create: No valid backend was found. No weighed backends available: NoValidBackend: No valid backend was found. No weighed backends available06:31
dtrainorhmm06:31
dtrainori'm not familiar with creating volumes in ceph using this configuration, i'm a little green to ceph06:32
dtrainorhttp://paste.openstack.org/show/677393/06:33
Tenguah. not a master of that as well. fortunately, my deploy worked, sooo... ;)06:34
Tenguhmm, apparently your ceph isn't properly configured, but I can't say more :/. there are a couple of ceph commands that might help. something like "ceph status" I think?06:35
Tengudtrainor: http://docs.ceph.com/docs/master/rados/operations/monitoring/  maybe, as a first check06:36
*** psahoo has quit IRC06:36
dtrainoryea i'm consuming that - thank you :)06:36
Tenguyou should be able to run those from a controller I think06:36
dtrainoryep06:39
*** agurenko has joined #tripleo06:41
*** waleedm has joined #tripleo06:44
*** jaganathan has joined #tripleo06:46
jaganathand0ugal, please look into https://review.openstack.org/#/c/527112/06:46
*** suuuper has joined #tripleo06:47
*** dpawar has joined #tripleo06:52
*** psahoo has joined #tripleo06:52
*** dciabrin has quit IRC06:56
*** dmacpher has quit IRC06:58
*** threestrands has quit IRC07:00
openstackgerritSagi Shnaidman proposed openstack/tripleo-quickstart-extras master: Create a generic reproducer script  https://review.openstack.org/54575607:02
*** pgadiya has joined #tripleo07:02
*** psahoo has quit IRC07:08
*** ooolpbot has joined #tripleo07:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION07:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/174964107:10
*** ooolpbot has quit IRC07:10
openstackLaunchpad bug 1749641 in tripleo "Overcloud deployment failing in promotion jobs at glance-manage db_sync" [Critical,In progress] - Assigned to Christian Schwede (cschwede)07:10
*** rcernin has quit IRC07:11
dtrainorTengu, i think i figured out my problem07:14
Tengudtrainor: oh?07:14
* Tengu is preparing a "crisis scenario" for a colleague: the lab has crashed, rebuild it asap" :]07:15
dtrainorSo I'm using UI to do these deployments (also, Pike).  UI deploys with puppet-ceph, which basically fails to configure ceph as a backend07:15
dtrainorif i'm understanding correctly07:15
Tenguhmmm. weird. I'm using puppet-ceph as well07:15
dtrainorin pike?07:16
Tenguyup07:16
dtrainorcontainers?07:16
TenguBM07:16
dtrainorright, but containers on bm, or no containers at all?07:16
Tenguno container at all, pure BM07:16
dtrainorgot ya07:16
Tenguwe did an early deploy with pike, it was just released, and containers.... well, it was all brand new, and we weren't that confident in fact.07:16
*** lvdombrkr has joined #tripleo07:16
dtrainorso if not using a containerized environment, this issue may not apply to you07:17
Tenguprobably then07:17
dtrainorall good07:17
Tengubut I'll need to converge to containers I think.07:17
Tenguand that will be the tricky part.07:17
dtrainori think that would be wise, yes.  it definitely had me pulling out the Docker books.07:17
dtrainorbut short of that, it's been pretty smooth sailing07:17
Tenguoh, well, already using a lot of docker and the like ;).07:18
dtrainorsee, you're already there07:18
Tenguapparently the migration BM->container won't be supported in Queens, so I'll need to do that migration inside Pike.07:18
dtrainoroh i see07:18
Tenguso yep. tricky.07:19
bandiniEmilienM: thanks for the heads up, will chat with the db folks about it07:19
Tenguthat's why I built that lab. Among other reasons :).07:19
Tenguhaving the possibility to test/validate updates is always a good thing. and with Tempest, it's just a breeze.07:19
*** psahoo has joined #tripleo07:20
dtrainoryea, "To deploy Ceph in containers use ceph-ansible, for which only a containerized Ceph deployment is possible. It is not possible to deploy a containerized Ceph with puppet-ceph."07:20
Tengu^^07:22
Tenguso the issue might be in tripleUI ?07:22
openstackgerritMichele Baldessari proposed openstack/tripleo-heat-templates stable/pike: Fix a typo in docker_puppet_apply.sh  https://review.openstack.org/54575907:24
dtrainorat least part of it.  something still fires off ceph-ansible because I see activity related to its use; i just don't know if the end result is the desired one07:29
dtrainordoing a test right now07:29
Tengudtrainor: I think ceph-ansible will be deprecated for good, and shouldn't be involved if puppet ceph module is used. not 100% sure though. might be worth digging a bit indeed.07:30
dtrainorother way around07:30
dtrainorpuppet-ceph is already deprecated https://docs.openstack.org/tripleo-docs/latest/install/advanced_deployment/ceph_config.html07:30
Tenguoh. yes. darn. sorry.07:30
Tenguhave to run after my brain - got up early today -.-'07:31
dtrainori'm up late, running on fumes.  going to take  a nap after this test.07:31
Tengu:)07:31
*** dsariel has joined #tripleo07:43
*** pcaruana has joined #tripleo07:47
openstackgerritMichele Baldessari proposed openstack/tripleo-heat-templates master: Add a ComputeInstanceHA role  https://review.openstack.org/54576107:49
*** bogdando has joined #tripleo07:53
*** links has quit IRC07:55
*** dpawar has quit IRC07:56
*** dpawar has joined #tripleo07:56
openstackgerritAttila Darazs proposed openstack/tripleo-quickstart-extras master: GATE CHECK for quickstart-extras  https://review.openstack.org/47260707:57
*** dpawar_ has joined #tripleo07:59
*** dpawar has quit IRC07:59
*** yprokule has joined #tripleo08:00
*** cylopez has joined #tripleo08:00
*** dciabrin has joined #tripleo08:05
*** ebarrera has joined #tripleo08:06
*** links has joined #tripleo08:06
*** ccamacho has joined #tripleo08:09
*** ooolpbot has joined #tripleo08:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION08:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/174964108:10
*** ooolpbot has quit IRC08:10
openstackLaunchpad bug 1749641 in tripleo "Overcloud deployment failing in promotion jobs at glance-manage db_sync" [Critical,In progress] - Assigned to Christian Schwede (cschwede)08:10
*** florianf has joined #tripleo08:14
*** assassin has quit IRC08:14
openstackgerritMerged openstack/tripleo-common master: Fix overcloud node delete after an upgrade  https://review.openstack.org/54478108:15
*** tesseract has joined #tripleo08:15
*** mrunge has quit IRC08:17
*** assassin has joined #tripleo08:17
*** aufi has joined #tripleo08:25
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-quickstart-extras master: Upgrade UC and OC using tripleo-upgrade role.  https://review.openstack.org/54231108:26
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-quickstart-extras master: Provide a way to change docker registry path during mixed upgrade.  https://review.openstack.org/54047308:27
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-quickstart master: DNM: testing p->m on experimental queue.  https://review.openstack.org/54576208:29
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-quickstart master: DNM: testing p->m on experimental queue.  https://review.openstack.org/54576208:31
*** mrunge has joined #tripleo08:31
*** stendulker has joined #tripleo08:33
*** agurenko has quit IRC08:34
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-quickstart master: DNM: testing p->m on experimental queue.  https://review.openstack.org/54576208:35
*** oidgar has joined #tripleo08:35
*** sshnaidm|off has quit IRC08:35
*** paramite has joined #tripleo08:35
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-heat-templates stable/pike: DNM Testing O->P with tripleo-upgarde  https://review.openstack.org/54390708:38
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-quickstart master: Set tripleo-upgrade vars in featureset.  https://review.openstack.org/53991008:38
*** juriarte has joined #tripleo08:39
*** amoralej|off is now known as amoralej08:41
openstackgerritRabi Mishra proposed openstack/tripleo-common stable/pike: Fix overcloud node delete after an upgrade  https://review.openstack.org/54576308:42
*** jpena|off is now known as jpena08:43
*** oscar____ has joined #tripleo08:44
*** sshnaidm has joined #tripleo08:44
openstackgerritMerged openstack/tripleo-heat-templates master: Add docker service for neutron-ovs-dpdk-agent service  https://review.openstack.org/54385108:47
openstackgerritAthlan-Guyot sofer proposed openstack-infra/tripleo-ci master: Find the right release file for ffu mixed version upgrade.  https://review.openstack.org/52600008:51
openstackgerritAthlan-Guyot sofer proposed openstack-infra/tripleo-ci master: WIP: Take mixed upgrade into account for boostrap_subnode_minimal.  https://review.openstack.org/54424908:51
openstackgerritAthlan-Guyot sofer proposed openstack-infra/tripleo-ci master: DNM: testing ffu upstream as rdo-cloud doesn't collect log when timeout.  https://review.openstack.org/54528108:51
*** dpawar_ has quit IRC08:52
*** dpawar has joined #tripleo08:53
*** holser has joined #tripleo08:54
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-quickstart master: New featureset for fast forward upgrade testing.  https://review.openstack.org/52568608:55
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-heat-templates master: Simplify FastForwardRepoArgs structure used in FFU repo selection.  https://review.openstack.org/54526108:56
*** dbecker has joined #tripleo09:00
*** gfidente has joined #tripleo09:00
*** jtomasek has joined #tripleo09:03
*** jpich has joined #tripleo09:05
*** sshnaidm has quit IRC09:07
openstackgerritCarlos Camacho proposed openstack/tripleo-common master: upgrade-non-controller.sh script does not support alternative plan names  https://review.openstack.org/54577109:08
*** sshnaidm has joined #tripleo09:08
*** ooolpbot has joined #tripleo09:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION09:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/174964109:10
*** ooolpbot has quit IRC09:10
openstackLaunchpad bug 1749641 in tripleo "Overcloud deployment failing in promotion jobs at glance-manage db_sync" [Critical,In progress] - Assigned to Christian Schwede (cschwede)09:10
openstackgerritCarlos Camacho proposed openstack/tripleo-common master: upgrade-non-controller.sh script does not support alternative plan names  https://review.openstack.org/54577109:11
openstackgerritMartin André proposed openstack/tripleo-heat-templates master: Add enpoints for TripleO UI Config  https://review.openstack.org/52867909:19
openstackgerritMartin André proposed openstack/tripleo-heat-templates master: docker: add support for TripleO UI  https://review.openstack.org/51549009:19
openstackgerritMerged openstack/tripleo-heat-templates master: Fix a typo in docker_puppet_apply.sh  https://review.openstack.org/54539809:20
*** derekh has joined #tripleo09:21
openstackgerritMartin André proposed openstack/tripleo-common master: Add tripleo-ui container  https://review.openstack.org/51540709:22
openstackgerritChristian Schwede proposed openstack/tripleo-heat-templates master: Support triggers additions for rolling upgrades  https://review.openstack.org/54562309:22
*** mrch has joined #tripleo09:22
*** fzdarsky has joined #tripleo09:26
*** oidgar has quit IRC09:28
bogdandofolks, did anyone deploy ovb with traas's toci_job type?09:29
bogdandotestenv-client hangs, while it seems to has to be 'Received job'09:30
arxcruz|rucktherve: around?09:31
thervearxcruz|ruck, Yep09:31
*** fzdarsky has quit IRC09:31
openstackgerritSagi Shnaidman proposed openstack/tripleo-quickstart-extras master: Create a generic reproducer script  https://review.openstack.org/54575609:31
arxcruz|rucktherve: some jobs are failing with the following error:09:32
arxcruz|ruckhttps://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens/fa0f376/undercloud/home/jenkins/overcloud_deploy.log.txt.gz#_2018-02-18_22_21_2909:32
thervearxcruz|ruck, https://bugs.launchpad.net/tripleo/+bug/1750270 right?09:32
openstackLaunchpad bug 1750270 in tripleo "[queens promotion] - fs001 fails overcloud deployment with 'MistralOvercloud configuration failed'" [Critical,Triaged]09:32
arxcruz|rucktherve: yes09:32
*** links has quit IRC09:32
therveI commented on it :)09:32
arxcruz|rucktherve: ok, i'll add promotion blocker tag09:33
thervearxcruz|ruck, Do you have an issue way to test something out?09:33
arxcruz|rucktherve: no, i can try the reproducer script09:34
therves/issue/easy woof09:34
arxcruz|rucktherve: but this is happening constantly, and it's blocking promotion09:34
arxcruz|rucktherve: let me try to bring an env up09:34
therveYeah I get it09:34
thervearxcruz|ruck, Great thanks09:34
arxcruz|rucktherve: give me a few... hours hehehe09:35
therveOuch :/09:35
arxcruz|ruckprobably 2 hours, not sure how long it will take to finish the installation09:35
chandankumarsshnaidm: Hello09:36
therveI'll try a simpler reproducer :)09:36
arxcruz|rucktherve: all right then09:36
arxcruz|ruckmeanwhile i'm adding the promotion blocker tag09:36
sshnaidmchandankumar, hi09:36
chandankumarsshnaidm: Do we have a doc on how to write new tht from scratch? i need to write from scratch otherwise tempest containers will be not installed on undercloud09:36
arxcruz|rucknevermind it's already with promotio blocker09:37
sshnaidmchandankumar, I don't know, better to ask DF folks09:37
chandankumarsure09:37
*** florianf has quit IRC09:41
*** dpawar has quit IRC09:42
openstackgerritSaravanan KR proposed openstack/tripleo-heat-templates master: Removed ovs-dpdk workaround to fix the vhost socket permission  https://review.openstack.org/52927309:43
*** oidgar has joined #tripleo09:43
*** links has joined #tripleo09:45
*** oidgar has quit IRC09:47
*** rdopiera_ is now known as rdopiera09:47
*** mrch has quit IRC09:51
*** florianf has joined #tripleo09:54
*** akane_ has quit IRC09:54
*** akane_ has joined #tripleo09:55
*** akane_ has quit IRC09:55
*** akane_ has joined #tripleo09:56
bogdandowrt my question above:09:59
bogdando10:56:47 AM GMT+1 - bogdando: ok, it seems I'm missing the whole te-broker thing, which prolly never worked for traas...09:59
bogdando10:57:52 AM GMT+1 - bogdando: would be really nice to have some repro for RDO jobs, specifically OVB on rdocloud09:59
* bogdando sigh09:59
*** akane__ has joined #tripleo09:59
*** akane_ has quit IRC10:00
openstackgerritCarlos Camacho proposed openstack/tripleo-common master: upgrade-non-controller.sh script does not support alternative plan names  https://review.openstack.org/54577110:00
*** akane__ has quit IRC10:01
*** akane has joined #tripleo10:03
*** akane has quit IRC10:05
bogdandoykarel: thanks! :)10:06
*** akrivoka has joined #tripleo10:09
*** salmankhan has joined #tripleo10:10
*** ooolpbot has joined #tripleo10:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION10:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/174964110:10
*** ooolpbot has quit IRC10:10
openstackLaunchpad bug 1749641 in tripleo "Overcloud deployment failing in promotion jobs at glance-manage db_sync" [Critical,In progress] - Assigned to Christian Schwede (cschwede)10:10
*** mdnadeem has quit IRC10:11
*** shardy has joined #tripleo10:15
openstackgerritMerged openstack/tripleo-upgrade stable/pike: Use the last name of a file path of deployment-files parameter.  https://review.openstack.org/54539110:19
openstackgerritSergii Golovatiuk proposed openstack/tripleo-heat-templates master: Revert "Revert "Introduce wait_backoff, fail_if_empty wrappers""  https://review.openstack.org/54537210:22
*** janki_ has joined #tripleo10:22
lvdombrkrhello folks, can someone clarify, if i create instance on my compute node, and when i try to ssh it, connection going through controller node or directly to compute node skipping controller?10:23
*** akane_ has joined #tripleo10:24
*** mrch has joined #tripleo10:26
*** mdnadeem has joined #tripleo10:27
*** janki_ has quit IRC10:28
*** tosky has joined #tripleo10:29
*** mdnadeem_ has joined #tripleo10:30
*** mdnadeem has quit IRC10:32
*** moshele has quit IRC10:33
*** gkadam has joined #tripleo10:39
*** Sulaiman has joined #tripleo10:41
*** SulaimanRadwan has joined #tripleo10:41
*** panda|off is now known as panda10:45
thervearxcruz|ruck, Sorry I'm stuck, can you try to get an env?10:46
arxcruz|rucktherve: it will probably fail in the same step10:46
arxcruz|ruckgimme a few minutes10:47
thervearxcruz|ruck, I'll have access though?10:47
arxcruz|rucktherve: yup10:47
therveOK so it's worth trying10:47
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates master: Allows for configuration of the Ceph cluster name  https://review.openstack.org/54166410:48
*** agurenko has joined #tripleo10:49
*** mrch has quit IRC10:54
*** dsariel has quit IRC10:54
*** chem has quit IRC10:58
*** zshi has quit IRC10:59
*** salmankhan has quit IRC11:00
*** ebarrera has quit IRC11:01
*** salmankhan has joined #tripleo11:05
*** chem has joined #tripleo11:06
openstackgerritMartin Mágr proposed openstack/tripleo-common master: Add healthcheck script for iSCSId  https://review.openstack.org/54345011:07
openstackgerritSergii Golovatiuk proposed openstack/tripleo-heat-templates master: Revert "Revert "Introduce wait_backoff, fail_if_empty wrappers""  https://review.openstack.org/54537211:08
openstackgerritSergii Golovatiuk proposed openstack/tripleo-heat-templates master: DO NOT MERGE - test scenarios 001&002  https://review.openstack.org/54579711:08
d0ugalI am seeing this when the undercloud attempts to install - any ideas? http://paste.openstack.org/show/677556/11:08
*** ooolpbot has joined #tripleo11:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION11:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/174964111:10
*** ooolpbot has quit IRC11:10
openstackLaunchpad bug 1749641 in tripleo "Overcloud deployment failing in promotion jobs at glance-manage db_sync" [Critical,In progress] - Assigned to Christian Schwede (cschwede)11:10
*** ebarrera has joined #tripleo11:17
*** stendulker has quit IRC11:18
*** lucas-afk is now known as lucasagomes11:18
*** zoli is now known as zoli|lunch11:22
*** zoli|lunch is now known as zoli11:22
*** dtantsur|afk is now known as dtantsur11:23
openstackgerritMerged openstack/tripleo-specs master: Add config-download squad  https://review.openstack.org/54363111:23
*** ccamacho has quit IRC11:26
*** dpawar has joined #tripleo11:26
*** moshele has joined #tripleo11:26
*** ccamacho has joined #tripleo11:28
*** udesale_ has joined #tripleo11:35
*** udesale has quit IRC11:37
*** ebarrera has quit IRC11:38
*** udesale_ has quit IRC11:39
*** bnemec is now known as bnemec-pto11:41
jpichd0ugal: I think you need to paste the lines just above for other people to see? That one can be a bit of a catch-all IIUC. I've been hitting https://bugs.launchpad.net/tripleo/+bug/1690850 lately but haven't found a solution yet unfortunately11:42
openstackLaunchpad bug 1690850 in tripleo "Undercloud install failed." [Medium,Triaged]11:42
d0ugaljpich: yeah, panda pointed me in the right direction. Seems there is a keystone issue.11:42
*** agopi has quit IRC11:42
jpichd0ugal: Ah, maybe something different then11:43
jpichGood luck!11:43
*** ebarrera has joined #tripleo11:44
skramajashardy: dsneddon after the nic configs are jinja templated, any idea how to generate the nic configs and modify it before the deployment?11:44
Tenguhelo there! small question: do I have to add anything in my custom env in order to register overcloud nodes against a Satellite? I found http://tripleo.org/basic_deployment/basic_deployment_cli.html but they only mention the variables - is it sufficient?11:44
shardyskramaja: Hi, there are two possible approaches, one is to run ./tools/proces_templates.py, and optionally copy the resulting templates out of the t-h-t tree11:46
*** ccamacho is now known as ccamacho|lunch11:46
shardyskramaja: the other option is to maintain a copy of the t-h-t tree and copy the j2 template itself11:46
skramajaSulaiman: ^^11:46
shardyclearly the second one is the approach we've chosen for the examples because it means they will work with custom roles and networks11:46
shardybut there's nothing to stop folks maintaining hard-coded non-j2 templates too if they wish11:47
shardyit's just less overhead to provide examples that work for custom roles and networks this way11:47
*** ukalifon has joined #tripleo11:48
shardyTengu: Hi, there are two files enabled via the --rhel-reg option to the deploy command11:49
shardyTengu: the easiest approach is to use those interfaces, but you could also create an environment file based on the files that option enables:11:50
shardyhttps://github.com/openstack/tripleo-heat-templates/blob/master/extraconfig/pre_deploy/rhel-registration/rhel-registration-resource-registry.yaml11:50
shardyhttps://github.com/openstack/tripleo-heat-templates/blob/master/extraconfig/pre_deploy/rhel-registration/environment-rhel-registration.yaml11:50
shardyhttps://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/v1/overcloud_deploy.py#L12411:50
Tengushardy: hmmm ok. "pre-deploy" - does it mean it will not be executed on a stack update for example?11:50
skramajashardy: thanks for the details. most of the usecase have different nic order. its is hard for ppl to transition without docs. any readme planned?11:51
shardyTengu: https://github.com/openstack/tripleo-heat-templates/blob/master/extraconfig/pre_deploy/rhel-registration/rhel-registration.yaml#L12211:51
shardyTengu: we register nodes when they are created, and unregister when they are deleted by default11:51
Tengudang11:51
shardyTengu: on update, already registered nodes shouldn't do anything AFAIK11:52
Tenguand if they aren't registered yet?11:52
shardyTengu: if that's not what you want, change the actions: ['CREATE'] to actions: ['CREATE', 'UPDATE']11:52
Tengufact is: we didn't have a Satellite when we built the overcloud, and now we have one, and want to use it in order to ensure a consistency with versions11:52
Tengushardy: hmm ok.11:52
*** shrjoshi has quit IRC11:52
shardyTengu: ack that should be possible, it's just not the default in the templates11:53
Tengu:)11:53
shardyTengu: you could add a parameter to enable it if you want, or just hack on the template as a workaround11:53
Tenguso basically, I fill up the variables in https://github.com/openstack/tripleo-heat-templates/blob/master/extraconfig/pre_deploy/rhel-registration/environment-rhel-registration.yaml I need, and in my custom registry point OS::TripleO::NodeExtraConfig: to a modified version of the template that enables the action for UPDATE ?11:54
shardyTengu: yes, or push a patch which adds that option to the default template11:55
shardyskramaja: yes docs will be required, I'd check with dsneddon as he may have them in progress already11:55
skramajathanks shardy11:56
Tengushardy: will first provide a custom one, once it's done I'll go back to the default. I think it's the best to do.11:56
shardyTengu: https://github.com/openstack/tripleo-heat-templates/blob/master/extraconfig/pre_deploy/rhel-registration/rhel-registration.yaml#L5611:56
Tenguoh11:56
shardyTengu: like you could add a RHELRegistrationActions parameter if you think others may have this requirement11:56
Tenguah, nah, not the one I thought. ok11:57
shardywhich would avoid carrying the copy of the template, but that is also a perfectly valid approach too11:57
shardyjust means you have to maintain that copy11:57
*** psahoo has quit IRC11:57
TenguI might consider a patch once I'm working at redhat - for now, I'm a still in a hurry ;)11:57
shardyTengu: ack understood :)11:57
Tengudug... s/still/bit11:58
Tengu-.- need more sleep. or coffee. or both11:58
*** mrch has joined #tripleo12:01
openstackgerritArx Cruz proposed openstack/tripleo-quickstart-extras master: Adding tempest_skip_queens.yml file  https://review.openstack.org/54581012:03
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates master: Add RHELRegistrationActions to rhel-registration template  https://review.openstack.org/54581112:04
shardyTengu: ^^ I think that is what you need, untested though12:04
shardyif you find time to try it please leave some review feedback, thanks!12:04
*** raildo has joined #tripleo12:04
openstackgerritMichele Baldessari proposed openstack/tripleo-heat-templates master: Improve the minor update of the docker service  https://review.openstack.org/54182712:04
Tengushardy: hmmm ok, I should be able to test it either today, or tomorrow.12:05
Tengushardy: you dropped a resource in your patch, is it intended?12:06
shardyTengu: yes it's unused, see the commit message12:06
Tenguah, ok. one patch with two things. baad ;)12:07
shardyI can break it up, I was just trying to write you a patch quickly, and that resource is confusing because it's also named DeploymentActions but does nothing12:08
Tengu:D was just joking ;).12:08
openstackgerritLukas Bezdicka proposed openstack/tripleo-quickstart-extras master: Add a hook to run a bash script at the end of the overcloud prep.  https://review.openstack.org/53316712:09
openstackgerritLukas Bezdicka proposed openstack/tripleo-quickstart-extras master: Introduce undercloud_templates_* variables  https://review.openstack.org/53251612:09
openstackgerritLukas Bezdicka proposed openstack/tripleo-quickstart-extras master: Decouple release from overcloud_release  https://review.openstack.org/53424212:09
openstackgerritLukas Bezdicka proposed openstack/tripleo-quickstart-extras master: RDOCloud - don't overwrite image path  https://review.openstack.org/54581212:09
Tengushardy: will test the patch once my colleague has finished the tempest configuration.12:09
arxcruz|rucktherve: it's being hard to get a deployment on rdocloud... need more time12:09
shardyTengu: ack thanks12:09
*** anilvenkata has quit IRC12:10
*** ooolpbot has joined #tripleo12:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION12:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/174964112:10
openstackLaunchpad bug 1749641 in tripleo "Overcloud deployment failing in promotion jobs at glance-manage db_sync" [Critical,In progress] - Assigned to Christian Schwede (cschwede)12:10
*** ooolpbot has quit IRC12:10
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates master: Add RHELRegistrationActions to rhel-registration template  https://review.openstack.org/54581112:12
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates master: Remove unused DeploymentActions resource  https://review.openstack.org/54581312:12
*** pkovar has joined #tripleo12:15
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-quickstart master: Set tripleo-upgrade vars in featureset.  https://review.openstack.org/53991012:16
*** slagle has quit IRC12:16
openstackgerritJiri Stransky proposed openstack/tripleo-quickstart-extras master: Overcloud update CI with tripleo-upgrade role  https://review.openstack.org/54220112:17
*** abishop has quit IRC12:19
openstackgerritSergii Golovatiuk proposed openstack/tripleo-heat-templates master: Revert "Revert "Introduce wait_backoff, fail_if_empty wrappers""  https://review.openstack.org/54537212:19
*** slagle has joined #tripleo12:20
openstackgerritSergii Golovatiuk proposed openstack/tripleo-heat-templates master: DO NOT MERGE - test scenarios 001&002  https://review.openstack.org/54579712:20
openstackgerritJiri Stransky proposed openstack/tripleo-quickstart-extras master: Overcloud update CI with tripleo-upgrade role  https://review.openstack.org/54220112:21
*** stevebaker has quit IRC12:25
*** dmellado has quit IRC12:26
*** dsariel has joined #tripleo12:26
openstackgerritMichele Baldessari proposed openstack/tripleo-heat-templates master: Improve the minor update of the docker service  https://review.openstack.org/54182712:27
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: Allow for multiple playbooks to be run  https://review.openstack.org/52576512:28
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: Run specific jobs with quickstart.sh  https://review.openstack.org/54581512:28
*** mrch has quit IRC12:28
*** stevebaker has joined #tripleo12:29
*** stevebaker has quit IRC12:29
*** stevebaker has joined #tripleo12:29
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-heat-templates stable/pike: Update YAQL queries with groupBy  https://review.openstack.org/54581712:30
*** vpickard has quit IRC12:34
*** stevebaker has quit IRC12:36
*** vpickard has joined #tripleo12:36
*** links has quit IRC12:37
*** jpena is now known as jpena|lunch12:39
*** sshnaidm has quit IRC12:42
*** abishop has joined #tripleo12:46
*** psachin has quit IRC12:51
*** artom__ has quit IRC12:52
*** pkovar has quit IRC12:52
*** pradk has joined #tripleo12:53
*** cylopez has left #tripleo12:54
*** dprince has joined #tripleo12:57
*** rfolco|away is now known as rfolco13:02
*** jlabarre has joined #tripleo13:03
*** deadnull has joined #tripleo13:04
*** thrash|g0ne is now known as thrash13:06
*** dmellado has joined #tripleo13:07
*** ooolpbot has joined #tripleo13:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION13:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/174964113:10
*** ooolpbot has quit IRC13:10
openstackLaunchpad bug 1749641 in tripleo "Overcloud deployment failing in promotion jobs at glance-manage db_sync" [Critical,In progress] - Assigned to Christian Schwede (cschwede)13:10
*** deadnull has quit IRC13:10
*** dougbtv has joined #tripleo13:12
*** suuuper has quit IRC13:13
*** jokke__ is now known as jokke_13:14
*** bfournie has quit IRC13:17
*** bfournie has joined #tripleo13:18
*** ykarel has quit IRC13:20
*** mrch has joined #tripleo13:21
*** mrch has left #tripleo13:21
*** bfournie has quit IRC13:22
*** skramaja has quit IRC13:25
*** dpawar has quit IRC13:25
*** sshnaidm has joined #tripleo13:25
*** pradk has quit IRC13:26
*** ccamacho|lunch has quit IRC13:27
*** ccamacho has joined #tripleo13:27
*** oidgar has joined #tripleo13:28
*** gkadam has quit IRC13:31
openstackgerritMartin Mágr proposed openstack/tripleo-common master: Add healthcheck script for Gnocchi API  https://review.openstack.org/53824613:31
*** jpena|lunch is now known as jpena13:31
openstackgerritMartin Mágr proposed openstack/tripleo-common master: Add healthcheck script for Cinder API  https://review.openstack.org/54289513:31
*** rlandy has joined #tripleo13:32
openstackgerritMartin Mágr proposed openstack/tripleo-common master: Add healthcheck script for etcd  https://review.openstack.org/54280313:32
openstackgerritMartin Mágr proposed openstack/tripleo-common master: Add healthcheck script for Neutron server  https://review.openstack.org/53998613:32
openstackgerritMartin Mágr proposed openstack/tripleo-common master: Add healthchecks for barbican containers  https://review.openstack.org/54523813:32
openstackgerritMartin Mágr proposed openstack/tripleo-common master: Add healthcheck script for Tacker  https://review.openstack.org/53636013:32
openstackgerritMartin Mágr proposed openstack/tripleo-common master: Add healthcheck script for Aodh API  https://review.openstack.org/54284913:32
openstackgerritMartin Mágr proposed openstack/tripleo-common master: Add healthcheck script for gnocchi-metricd  https://review.openstack.org/53822813:32
*** rlandy is now known as rlandy|rover13:33
openstackgerritMartin Mágr proposed openstack/tripleo-common master: Add healthcheck script for memcached  https://review.openstack.org/54400613:33
openstackgerritMartin Mágr proposed openstack/tripleo-common master: Add healthcheck script for gnocchi-statsd  https://review.openstack.org/53825313:33
openstackgerritMartin Mágr proposed openstack/tripleo-common master: Add healthcheck script for Collectd  https://review.openstack.org/53827413:33
openstackgerritMartin Mágr proposed openstack/tripleo-common master: Add healthcheck script for multipathd  https://review.openstack.org/53918513:33
*** tcw has joined #tripleo13:33
*** anilvenkata has joined #tripleo13:33
*** tcw has quit IRC13:34
*** tcw has joined #tripleo13:34
*** pkovar has joined #tripleo13:35
openstackgerritRyan Brady proposed openstack/python-tripleoclient master: Changes Creation Of overcloudrc File to Call Workflow  https://review.openstack.org/54535113:37
*** dmellado has quit IRC13:38
*** pgadiya has quit IRC13:38
openstackgerritRyan Brady proposed openstack/python-tripleoclient master: Changes Delete Plan to Call Workflow  https://review.openstack.org/53904513:39
*** jokke_ has quit IRC13:41
*** jokke_ has joined #tripleo13:41
*** trown|outtypewww is now known as trown13:42
*** pchavva has joined #tripleo13:42
*** pchavva1 has joined #tripleo13:43
*** jcoufal has joined #tripleo13:44
*** fultonj has joined #tripleo13:44
openstackgerritBogdan Dobrelya proposed openstack/tripleo-heat-templates master: Enable multipathd health check  https://review.openstack.org/53918813:46
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-quickstart master: Set docker and image value to the right value in mixed version.  https://review.openstack.org/53992913:48
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-quickstart master: [WIP] Use tripleo-upgrade role for p->m job.  https://review.openstack.org/54007213:48
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-quickstart master: DNM: testing p->m on experimental queue.  https://review.openstack.org/54576213:48
*** liverpooler has joined #tripleo13:50
*** morazi has joined #tripleo13:50
*** cshastri has quit IRC13:51
*** bfournie has joined #tripleo13:51
*** ykarel has joined #tripleo13:51
*** toure|gone is now known as toure13:54
*** jmelvin has joined #tripleo13:54
*** dhill__ has joined #tripleo13:55
*** tzumainn has quit IRC13:55
*** tzumainn has joined #tripleo13:55
openstackgerritAlan Bishop proposed openstack/puppet-tripleo stable/pike: Avoid hard-coded settings in Cinder HA containers  https://review.openstack.org/54583513:57
openstackgerritAlan Bishop proposed openstack/tripleo-heat-templates stable/pike: Unify the Cinder HA and non-HA docker configurations  https://review.openstack.org/54583613:57
gfidenteguys, this is so green it hurts https://review.openstack.org/54166413:57
gfidente(note the depends on)13:57
*** eck`gone is now known as eck`13:59
openstackgerrithanish proposed openstack/tripleo-heat-templates master: Adding support for ComputeLiquidio role in Network jinja file. Since Liquidio compute nodes contain Liquidio smart NIC, tenant and external network ips are assigned in the smart NIC instead of Compute node.  https://review.openstack.org/54583714:00
*** maeca has joined #tripleo14:01
*** artom has joined #tripleo14:01
*** akane_ has quit IRC14:02
*** shrjoshi has joined #tripleo14:04
*** aputtur_ has joined #tripleo14:05
*** florianf has quit IRC14:05
*** shrjoshi has quit IRC14:05
*** shreshtha has joined #tripleo14:06
*** amoralej is now known as amoralej|lunch14:06
*** mmethot has joined #tripleo14:08
*** moshele has quit IRC14:09
larsksHey folks.  I just had a deploy fail; the stack state is CREATED_FAILED, but there are no failed resources (all resources are either CREATE_COMPLETE or INIT_COMPLETE).  What would cause that kind of failure?14:09
*** ooolpbot has joined #tripleo14:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION14:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/174964114:10
*** ooolpbot has quit IRC14:10
openstackLaunchpad bug 1749641 in tripleo "Overcloud deployment failing in promotion jobs at glance-manage db_sync" [Critical,In progress] - Assigned to Christian Schwede (cschwede)14:10
*** etingof has quit IRC14:12
*** mrch has joined #tripleo14:12
*** mrch has left #tripleo14:13
*** Sulaiman has quit IRC14:14
thervelarsks, What's the stack_status_reason?14:15
therveI suspect it timed out14:15
*** florianf has joined #tripleo14:17
flaper87jistr: I was chatting with shardy about https://review.openstack.org/#/c/543933/1/extraconfig/services/openshift-master.yaml and sharing that I feel we should be able to break the inventory/config generation for this into multiple templates. If you look at the openshift-cns.yaml file, it's empty but I believe it should contain the generation of the inventory section for that service14:19
flaper87jistr: one of the ideas we were discussing was doing the generation of the inventory at an earlier step (#1?)14:20
openstackgerritJuan Badia Payno proposed openstack/tripleo-heat-templates master: logging: fluentd deprecated params releasenote  https://review.openstack.org/54387114:20
*** Goneri has joined #tripleo14:20
flaper87and allow for these services to generate their inventory file/section independently and then add them all to the ansible-playbook execution14:20
flaper87jistr: wondering what your thoughts/opinions are on this regard14:21
openstackgerritJuan Badia Payno proposed openstack/tripleo-heat-templates master: logging: fluentd deprecated params releasenote  https://review.openstack.org/54387114:22
larskstherve: let me take a look, just a sec...14:24
*** pchavva1 has quit IRC14:25
larskstherve: stack_status_reason = list index out of range14:25
thervelarsks, https://bugs.launchpad.net/tripleo/+bug/175003214:25
openstackLaunchpad bug 1750032 in tripleo "Undercloud / Overcloud Heat stack fails on: YAQL list index out of range (includes upgrades cases)" [High,Triaged]14:25
jistrflaper87, shardy: that would be an option i think. Either we could generate stub files on filesystem in step 1, and then collect them in step 2 (using the "assemble" ansible module). Or we could generate ansible facts based on some naming scheme in step 1, and again programmatically collect them on step 2 (same, but not writing to filesystem). However, i think keeping the inventory generation in a14:25
jistropenshift-master.yaml is totally fine too.14:25
*** pradk has joined #tripleo14:26
*** etingof has joined #tripleo14:26
larskstherve: ah, thanks.14:27
flaper87jistr: gets tricker when we start adding more variables that are specific to some configs. For example, I'll parametize the glusterfs_devices variable and there are plenty others like that14:27
holsersshnaidm - Good morning14:28
jistrflaper87, shardy: my reasoning is that there's limited amount of "group names" that ansible-openshift can understand, so we don't have to be worried about free composability. We were just discussing the same thing with gfidente w/r/t Ceph. We have a bunch of composable Ceph services, but all the important params can go into ceph-base.yaml because, in the end, we just run *one* installer ceph-ansible, and it14:28
jistrunderstands a limited set of options, so there's no issue in managing all those options from a single composable service in TripleO. We're still driving just one installer, even if we tag nodes via multiple composable services.14:28
holserAre you super busy at the moment?14:28
sshnaidmholser, hi14:28
holsersshnaidm I have a question about gating14:28
gfidentejistr flaper87 shardy yeah and /me would like to discuss that more at PTG14:28
gfidenteansible just assumes a single run14:29
gfidenteand so does people writing the modules14:29
gfidenteso I think a lot of times things could be split better in role vars but just aren't14:29
gfidenteor are re-used all around the place14:29
*** pchavva has quit IRC14:29
sshnaidmholser, sorry, on mtg right now14:29
holsersshnaidm I saw it ran jobs tripleo-ci-centos-7-3nodes-multinode then when it was almost done the status was changed to queued and then it was run againą14:29
jistrflaper87, shardy, gfidente: so yea openshift-master would be pretty big, but i don't think that's an overstep in terms of composability. It's that openshift-ansible is pretty big :) It's still 1:1 mapping even though it may give an illusion of N:1. But i'm definitely open to being persuaded otherwise in case i'm wrong :)14:30
*** akane has joined #tripleo14:30
*** akane has quit IRC14:30
*** ramishra has quit IRC14:31
*** maeca has quit IRC14:31
gfidentejistr shardy one thing I would like to ask for help with is "composability" of those variables by merging key values from mulitple services14:31
jistrflaper87: need to go to mtg now but i'll take another look later14:31
*** jistr is now known as jistr|mtg14:31
sshnaidmholser, it may happen when somebody rebase or update the patch, zuul stop all current jobs and starts new ones14:32
sshnaidmholser, or so any change in dependencies14:32
holsernope nobody touched it14:32
sshnaidms/so/do14:32
flaper87jistr: not entirely wrong. I'm more concerned about future maintenance than the actual mapping with openshift-ansible, tbh. It does map 1:1 quite nicely in terms of execution but it's a bit weird in terms of implementation. It's inconsistent14:32
*** leanderthal has joined #tripleo14:32
holsersshnaidm 544068 - see the status on http://zuul.openstack.org14:33
*** florianf has quit IRC14:34
*** florianf has joined #tripleo14:34
sshnaidmholser, it in final steps as I see14:34
openstackgerritJuan Badia Payno proposed openstack/tripleo-heat-templates master: [WIP]logging: read non-containerized logs  https://review.openstack.org/54584114:34
holserbut that's second or third run sshnaidm14:35
holser5h in total14:35
sshnaidmholser, than maybe of "parent" patches failed, better to ask on #openstack-infra, then should know14:36
*** pkovar has quit IRC14:36
jistr|mtgflaper87, gfidente: well i think we'd be doing a "partially wrong" design by trying to split driving of a single external installer across multiple composable services on our side. Maybe we should think of it as having openshift-ansible.yaml service for running the installer and generating inventory, and then openshift-master, openshift-worker, and openshift-gluster services just for tagging nodes. I think14:37
jistr|mtgsome of the confusion comes from conflating the installer run with tagging nodes as openshift-master?14:37
jistr|mtgcc shardy ^14:37
*** bfournie1 has joined #tripleo14:38
gfidentejistr|mtg I agree we should drive the external tool the way it expects to be, I am just saying the external tool could be more modular14:38
*** udesale has joined #tripleo14:38
*** pkovar has joined #tripleo14:39
*** bfournie has quit IRC14:39
*** udesale has quit IRC14:40
*** links has joined #tripleo14:40
*** udesale has joined #tripleo14:40
*** sovsianikov has joined #tripleo14:41
holsersshnaidm post_failure of one of jobs14:42
*** apetrich has quit IRC14:42
*** apetrich has joined #tripleo14:43
sshnaidmholser, definitely infra problem14:43
holserthanks sshnaidm14:43
*** dmellado has joined #tripleo14:44
EmilienMhello14:44
*** pchavva has joined #tripleo14:44
*** sshnaidm has quit IRC14:46
*** sshnaidm has joined #tripleo14:47
jistr|mtggfidente: yea maybe... but we shouldn't start such work on the tripleo side. Such modularity should first come from the external installer. I think in tripleo we should just try to match the semantics as closely as possible. If we try to invent composability in places where the external installer doesn't have them, we may just put unnecessary work/bugs on us.14:48
gfidenteyeah14:48
*** links has quit IRC14:49
*** stevebaker has joined #tripleo14:49
*** hjensas has quit IRC14:50
raildomandre, hey Did you have any time to think of how to solve? https://bugs.launchpad.net/tripleo/+bug/174821914:54
openstackLaunchpad bug 1748219 in tripleo "keystone integration with LDAP broken in containerized deployment, manually keystone restart needed" [High,Triaged]14:54
EmilienMpradk: when you have time, please review https://review.openstack.org/544530 - and see how it can affect Telemetry services (see usage of internal_api network for SNMP)14:54
pradkEmilienM, sure14:55
*** amoralej|lunch is now known as amoralej14:56
shardyjistr|mtg: Yeah I think conflating the master with the installer run does add to the confusion, but it's probably fine for now14:56
shardyjistr|mtg: eventually it'd be nice to make the "node tagging" templates contribute their own inventory, which I think would fit the ansible model quite well, but I agree we can't expose composability where none exists on the ansible side14:57
*** yamahata has quit IRC14:58
shardyPersonally I find the composition model of ansible quite limited despite roles etc, it's hard to have stable interfaces with the global variable namespace and lack of any schema14:58
*** yamahata has joined #tripleo14:58
mandrehey raildo, sorry I didn't find the time to look at it. Maybe stevebaker or jistr|mtg will have ideas on how to solve it14:58
*** masco has quit IRC14:58
shardybut maybe having a "run installer" service would make things clearer14:58
*** links has joined #tripleo15:01
*** dtantsur is now known as dtantsur|brb15:01
openstackgerritFlavio Percoco proposed openstack/tripleo-heat-templates master: WIP: Deploy OpenShift+CNS  https://review.openstack.org/54393315:02
*** dmellado has quit IRC15:02
*** jistr|mtg is now known as jistr15:02
*** tosky_ has joined #tripleo15:02
*** akane_ has joined #tripleo15:02
*** tosky has quit IRC15:03
openstackgerritAthlan-Guyot sofer proposed openstack-infra/tripleo-ci master: Find the right release file for ffu mixed version upgrade.  https://review.openstack.org/52600015:03
openstackgerritAthlan-Guyot sofer proposed openstack-infra/tripleo-ci master: WIP: Take mixed upgrade into account for boostrap_subnode_minimal.  https://review.openstack.org/54424915:03
openstackgerritAthlan-Guyot sofer proposed openstack-infra/tripleo-ci master: DNM: testing ffu upstream as rdo-cloud doesn't collect log when timeout.  https://review.openstack.org/54528115:03
jistrshardy, flaper87, gfidente: right it's perhaps more ambiguous for the inventory parts than i thought, as there's little potential for change on the external side. The composability mismatch might be a bigger issue with parameters, e.g. some param is only relevant for worker nodes so we'd put it there, and then the installer decides to make that param relevant for all nodes, and we'd have to move it to the15:07
jistr"main" service which runs the installer. On inventory such things are unlikely to happen.15:07
jistrand +1 re ansible composability being a challenge, especially with lack of namespacing :)15:07
*** stevebaker has quit IRC15:07
*** salmankhan has quit IRC15:08
*** ooolpbot has joined #tripleo15:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION15:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/174964115:10
*** ooolpbot has quit IRC15:10
openstackLaunchpad bug 1749641 in tripleo "Overcloud deployment failing in promotion jobs at glance-manage db_sync" [Critical,In progress] - Assigned to Christian Schwede (cschwede)15:10
openstackgerritFlavio Percoco proposed openstack/tripleo-heat-templates master: WIP: Deploy OpenShift+CNS  https://review.openstack.org/54393315:11
jistrshardy, flaper87: from a certain point of view, the nodes already sort of contribute their inventory by the service_name tagging though. The question is whether taking that further, onto the level of actual chunks of the Ansible inventory file, would bring enough benefit to justify the additional mangling needed to glue the pieces...15:11
*** oidgar has quit IRC15:12
flaper87jistr: shardy I don't see this as a way to make the inventory composable but as a way to represent an ansible role and its configs. I do see your point of interpreting external_tasks as a way to map external installers, though.15:14
flaper87and I would be ok with both solutions15:14
flaper87However, putting everything inside a single template might be a bit harder to maintain but not impossible15:15
flaper87it does feel weird to have OpenShift::Worker and OpenShift::GlusterFS, though15:15
-openstackstatus- NOTICE: Zuul has been restarted to pick up latest memory fixes. Queues were saved however patches uploaded after 14:40UTC may have been missed. Please recheck if needed.15:16
dmsimardEmilienM: Do you know if there has been an attempt to upstream whatever makes "OVB" work ? It seems like there's a lot of value in supporting that use case so it feels a bit weird to me that we have to patch that up on top of the clouds we use15:19
* EmilienM in meetings fyi15:20
dmsimardnp, it can wait15:20
*** mdnadeem_ has quit IRC15:20
toure rbrady ping15:21
*** agopi has joined #tripleo15:21
*** salmankhan has joined #tripleo15:22
rbradytoure: ack15:22
*** akane_ has quit IRC15:22
jistrflaper87: "way to represent an ansible role and its configs" -- yea i understand why we'd like to do it that way. The issue is that if the external installer doesn't subscribe to the same philosophy/representation, and we only do this on our side, we may be creating more trouble than benefit. That would probably depend on how particular installers behave. With Ansible-based ones it's hard to make such15:22
jistrseparations a guarantee we can depend on in TripleO. It would have to be driven by developers of the installer, probably by being "pedantic" enough. Just my 2 cents :)15:22
tourewas the original create_container card linked to a bug?15:23
tourerbrady ^^15:23
rbradytoure: https://bugs.launchpad.net/bugs/164043615:24
openstackLaunchpad bug 1640436 in tripleo "Add workflows for all the direct action calls in Mistral" [Medium,In progress] - Assigned to Ryan Brady (rbrady)15:24
tourecool thanks15:24
jistrflaper87: btw regardless which way we pick, we'll probably still need OpenShift::Worker and OpenShift::GlusterFS in the resource registry anyway, to tag the nodes at least15:24
*** aufi has quit IRC15:25
*** oscar____ has quit IRC15:27
openstackgerritCarlos Goncalves proposed openstack/instack-undercloud master: Revert "Add option to install the Octavia client"  https://review.openstack.org/54585215:28
openstackgerritCarlos Goncalves proposed openstack/instack-undercloud master: Revert "Add option to install the Octavia client"  https://review.openstack.org/54585215:31
*** waleedm has quit IRC15:31
openstackgerritMichael Chapman proposed openstack/tripleo-quickstart-extras master: Add opendaylight to collect-logs  https://review.openstack.org/49404315:32
*** dsariel has quit IRC15:33
openstackgerritThomas Herve proposed openstack/tripleo-heat-templates master: Do not use the 3rd argument of yaql groupBy  https://review.openstack.org/54585615:33
thervechem, ^^^15:33
*** bogdando has quit IRC15:33
chemtherve: thanks, looking15:34
thervearxcruz|ruck, I guess no news from my env? :/15:34
*** lucasagomes is now known as lucas-hungry15:34
chemtherve: just wondering how to test that locally, do you have a valid input somewhere ?15:35
thervechem, Depends on which form you want to test :)15:35
therveThere are various structures in play here15:36
jristEmilienM: where's that doc for changes in upstream15:36
EmilienMjrist: that? https://etherpad.openstack.org/p/tripleo-queens-highlights15:37
chemtherve: I see.  I'll get to that another day I guess15:38
jristEmilienM: thanks!15:38
openstackgerritWojciech Dec proposed openstack/tripleo-heat-templates master: Add cisco VTS ML2 template for a dockerized service and default environment settings  https://review.openstack.org/53335115:39
jristEmilienM: sorry to bug youa bout this but do you have pike?15:40
jristEmilienM: off hand?15:40
arxcruz|rucktherve: no :( rdocloud seems to be in maintenance... according rlandy|rover not sure if this is also affectiong the personal tennants15:40
thervearxcruz|ruck, Yeah mine isn't accessible either15:41
jistrraildo: re https://bugs.launchpad.net/tripleo/+bug/1748219 i'm not familiar with the issue but i seem to recall someone mentioned that we do some config still after starting the container, and then it's not picked up...15:41
openstackLaunchpad bug 1748219 in tripleo "keystone integration with LDAP broken in containerized deployment, manually keystone restart needed" [High,Triaged]15:41
jistrmaybe it was for TLS or CA configs15:42
rlandy|roverarxcruz|ruck: therve: rdo cloud is recovering but there are still some issues15:42
jistrraildo: hmm jaosorior isn't around :/ but he might be the person to ask15:42
openstackgerritMartin André proposed openstack/tripleo-heat-templates master: docker: add support for TripleO UI  https://review.openstack.org/51549015:42
jistrlooking into git, he also did the LDAP integration15:43
*** vpickard is now known as vpickard_15:44
raildojistr, yeah, he is on PTO, will come back next wednesday15:45
*** Artue has joined #tripleo15:45
raildojistr, let's wait he come back and see what we can do :) thanks for the update15:46
*** deadnull has joined #tripleo15:47
*** dsariel has joined #tripleo15:48
*** pblaho1 has quit IRC15:53
*** dtantsur|brb is now known as dtantsur15:54
*** hjensas has joined #tripleo15:55
EmilienMjrist: what do you need from pike?15:56
*** maeca has joined #tripleo15:56
openstackgerritMichael Chapman proposed openstack/tripleo-quickstart-extras master: Add opendaylight to collect-logs  https://review.openstack.org/49404315:56
*** ccamacho has quit IRC15:56
*** atoth has quit IRC15:57
*** ccamacho has joined #tripleo15:58
openstackgerritCarlos Camacho proposed openstack/tripleo-common master: upgrade-non-controller.sh script does not support alternative plan names  https://review.openstack.org/54577116:03
openstackgerritMichele Baldessari proposed openstack/tripleo-heat-templates stable/pike: swift_rsync: don't bind mount /run  https://review.openstack.org/54586416:03
openstackgerritYurii Prokulevych proposed openstack/tripleo-upgrade master: Add 'overcloud_images_validate' tag  https://review.openstack.org/54586516:05
*** deadnull has quit IRC16:05
*** colonwq has quit IRC16:07
*** akane_ has joined #tripleo16:07
*** akane_ has quit IRC16:07
*** pcaruana has quit IRC16:07
*** akane_ has joined #tripleo16:07
dmsimardmandre: o/16:10
*** ooolpbot has joined #tripleo16:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION16:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/174964116:10
*** ooolpbot has quit IRC16:10
openstackLaunchpad bug 1749641 in tripleo "Overcloud deployment failing in promotion jobs at glance-manage db_sync" [Critical,In progress] - Assigned to Christian Schwede (cschwede)16:10
dmsimardmandre: for https://review.openstack.org/#/c/526726/ -- are you going to backport that to queens/pike ?16:10
jristEmilienM: just that similar summary of changes16:11
jristEmilienM: it's ok if you don't have it16:11
EmilienMjrist: I'll search today16:11
*** mdnadeem_ has joined #tripleo16:12
openstackgerritTim Rozet proposed openstack/tripleo-quickstart-extras master: Add opendaylight to collect-logs  https://review.openstack.org/49404316:12
dmsimardEmilienM: still in meetings?16:14
EmilienMdmsimard: give me 15min16:15
dmsimardok :P16:15
*** shreshtha has quit IRC16:17
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: Default to docker.io instead of trunk.registry.rdoproject.org  https://review.openstack.org/54587316:19
*** wolverineav has joined #tripleo16:19
sshnaidmdmsimard, ^^16:19
dmsimardsshnaidm: kind of hacky16:20
sshnaidmdmsimard, yeah16:20
dmsimardsshnaidm: why not leave the default to docker.io and just pass an extra var for our promotion jobs to use trunk.registry ?16:20
sshnaidmdmsimard, with current ci it will be even more hacky16:21
dmsimardsshnaidm: why ?16:21
sshnaidmdmsimard, we tried hard to make promotion jobs as much as possible similar to patch jobs16:21
*** udesale has quit IRC16:22
*** mdnadeem_ has quit IRC16:22
*** pblaho1 has joined #tripleo16:24
*** pkovar has quit IRC16:25
dmsimardsshnaidm: tripleo-quickstart is meant to be used outside the gate and promotion jobs right ? the defaults should reflect that and then jobs pass in their own configuration ? I dunno16:25
dmsimardI remember reading about "job reproduction outside the gate" or something like that but I'm not sure how it works16:26
*** akane_ has quit IRC16:26
sshnaidmdmsimard, well, yeah, we can run quickstart to reproduce job - on RDO cloud private tenant16:26
Tengushardy: your patch is being tested right now on my lab :). small note: I'm running Pike, so I had to change the "heat_template_version" value.16:26
Tenguthat said: I didn't check what the script will do in case an already registered node launch it.16:27
*** ykarel is now known as ykarel|away16:29
*** lucas-hungry is now known as lucasagomes16:30
openstackgerritGabriele Cerami proposed openstack/tripleo-quickstart-extras master: Add Null Reproduced generator  https://review.openstack.org/54587516:30
*** leanderthal has quit IRC16:31
openstackgerritCarlos Camacho proposed openstack/tripleo-common master: upgrade-non-controller.sh script does not support alternative plan names  https://review.openstack.org/54577116:32
*** pblaho1 has quit IRC16:32
*** lvdombrkr has quit IRC16:34
*** ukalifon has quit IRC16:34
*** sovsianikov has quit IRC16:34
*** stendulker has joined #tripleo16:36
EmilienMdmsimard: what's up (and sorry for the wait)16:41
dmsimardEmilienM: copypasta because I'm lazy: Do you know if there has been an attempt to upstream whatever makes "OVB" work ? It seems like there's a lot of value in supporting that use case so it feels a bit weird to me that we have to patch that up on top of the clouds we use16:42
*** moshele has joined #tripleo16:42
Tengushardy: oh, btw, would be nice to put default values for the parameters, as all aren't mandatory. Will open an issue about that.16:42
*** jpena is now known as jpena|brb16:42
EmilienMdmsimard: yeah, have you asked to bnemec-pto (which is on pto apparently), he created https://github.com/cybertron/openstack-virtual-baremetal16:43
EmilienMdmsimard: I wish we could make it "upstream" - mwhahaha : any thoughts?16:43
dmsimardEmilienM: I was asking you because he is on PTO yeah :P16:43
dmsimardEmilienM: we probably have enough cores in nova/ironic to ask their opinion about it16:44
shardyTengu: Yeah most are defaulted to empty strings, would be good if we could improve that16:44
mwhahahadmsimard: EmilienM: i know there's been asks for it at least to be moved to gerrit. not sure the status of that16:44
EmilienMdmsimard: i'm not aware of any blockers so far, so I guess it might be just a "do it" thing16:45
dmsimarddtantsur: I saw you contributed code to OVB, do you know why all of this can't be in upstream nova/ironic ?16:45
fultonjEmilienM: the draft calendar has the PTG happening this week, not next week16:45
*** pkovar has joined #tripleo16:45
dtantsurdmsimard: what exactly?16:45
dtantsurgenerating instackenv.json which is tripleo-specific?16:45
EmilienMfultonj: lol16:46
EmilienMfultonj: welcome to my early mornings :D16:46
dmsimarddtantsur: I don't know what exactly, I'm mostly complaining that OpenStack clouds without some custom patching can't support OVB16:46
EmilienMfultonj: and thanks16:46
fultonj:) np, thanks for making calendar16:46
dmsimarddtantsur: it's be great if it was supported everywhere16:46
dtantsurdmsimard: I was patching its build-nodes-json script, it is not cloud-specific16:46
dtantsurotherwise I'm +1 on what you say :)16:46
dmsimarddtantsur: oh okay.. I mean I saw your name and I went "I'll ask this guy"16:47
EmilienMfultonj: solved, thanks again16:47
dtantsur:)16:47
dmsimardI guess I'll wait for bnemec to come back from PTO16:47
fultonjlooks better16:47
dmsimarddtantsur: thanks anyway :)16:47
*** aufi has joined #tripleo16:49
EmilienMmandre: https://review.openstack.org/#/c/545744/ - will do a separated patch!16:53
openstackgerritEmilien Macchi proposed openstack/tripleo-common master: container-images: support cont. u/c w/ Redis  https://review.openstack.org/54588116:55
EmilienMmandre: ^16:55
holsersshnaidm - Could you push forward https://review.openstack.org/#/c/539525/ ?16:57
holserw+1 is appreciated :)16:57
*** hjensas has quit IRC16:59
mandreEmilienM: sounds good, thanks for the patch17:01
*** yprokule has quit IRC17:02
*** jmelvin has quit IRC17:02
EmilienMmandre: thanks for the help17:03
mandredmsimard: hey! this should indeed be backported17:04
*** moguimar has quit IRC17:05
mandrehave we already branched out stable/queens in tripleo-common? EmilienM17:05
*** moguimar has joined #tripleo17:07
*** moshele has quit IRC17:08
*** links has quit IRC17:09
*** zoli is now known as zoli|gone17:10
*** zoli|gone is now known as zoli17:10
*** ooolpbot has joined #tripleo17:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION17:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/174964117:10
*** ooolpbot has quit IRC17:10
openstackLaunchpad bug 1749641 in tripleo "Overcloud deployment failing in promotion jobs at glance-manage db_sync" [Critical,In progress] - Assigned to Christian Schwede (cschwede)17:10
*** dmellado has joined #tripleo17:10
*** myoung is now known as myoung|bbl17:11
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-quickstart master: New featureset for fast forward upgrade testing.  https://review.openstack.org/52568617:12
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-quickstart master: Set tripleo-upgrade vars in featureset.  https://review.openstack.org/53991017:15
*** trown is now known as trown|lunch17:17
*** itlinux has joined #tripleo17:21
*** jpena|brb is now known as jpena17:22
gfidenteEmilienM about the ptg schedule17:26
gfidenteI noticed we have less "ceph centric" time and more time to discuss ceph integration in the various sessions17:26
gfidente+1 on that, thanks!17:26
*** dsariel has quit IRC17:27
fultonj+117:29
*** Artue has quit IRC17:29
fultonjtonyb: for ptg schedule, there's an integration slot involving ceph and multiarch. are you the multiarch driver?17:30
*** etingof has quit IRC17:32
*** aufi has quit IRC17:37
*** stendulker has quit IRC17:38
mwhahahafultonj: it's not necessarily ceph+multiarch but rather a slot for discussions around each one of those items.17:39
mwhahahaprobably should clarify that in the calendars17:39
mwhahahawe tried to group some of the topics together17:39
*** moguimar has quit IRC17:39
fultonjmwhahaha: makes sense, so you don't need a driver?17:39
mwhahahawell there needs to be a driver of that session17:39
mwhahahaotherwise your stuff won't get talked about :D17:39
*** dsariel has joined #tripleo17:40
fultonji was trying to help satisfy that requirement and suggesting that tony, whom i believe to be the multiarch point of contact, be co-drivers17:40
mwhahahafultonj: sure i just wanted to clarify the intent.17:40
*** shardy has quit IRC17:41
mwhahahafultonj: makes sense for you guys to co-chair. he's on australian time so probably won't be around til later17:41
fultonjmwhahaha: ok, i'll switch the medium to email17:42
EmilienMfultonj, gfidente: hope it works for you! please propose any change if you need :-)17:49
*** jpich has quit IRC17:49
gfidenteEmilienM yeah I was happy to see it organized the way it is now17:50
EmilienMawesome :D17:50
fultonjI see ceph as a topic under the config-download, ci topics, frame work improvements, and the general integration topics. I think it's great as it feels well integrated. Thanks EmilienM mwhahaha17:52
EmilienMawesome17:52
EmilienMI also want to keep time for people to hack and work together in ad hoc if possible17:53
*** sshnaidm is now known as sshnaidm|off17:54
*** ebarrera has quit IRC17:59
*** derekh has quit IRC18:00
*** anilvenkata has quit IRC18:00
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: Default to docker.io instead of trunk.registry.rdoproject.org  https://review.openstack.org/54587318:01
*** tosky_ is now known as tosky18:04
*** assassin has quit IRC18:06
*** assassin has joined #tripleo18:06
*** holser has quit IRC18:07
*** tosky has quit IRC18:07
*** wolverineav has quit IRC18:08
EmilienMsdoran, mwhahaha : would love some early feedback on https://review.openstack.org/#/c/545524/18:09
*** ooolpbot has joined #tripleo18:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION18:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/174964118:10
openstackLaunchpad bug 1749641 in tripleo "Overcloud deployment failing in promotion jobs at glance-manage db_sync" [Critical,In progress] - Assigned to Christian Schwede (cschwede)18:10
*** ooolpbot has quit IRC18:10
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: Run logs collection after job finished in RDO CI  https://review.openstack.org/54594118:10
*** jmelvin has joined #tripleo18:13
*** rwsu has quit IRC18:17
*** openstackgerrit has quit IRC18:18
*** jpena is now known as jpena|away18:19
*** hjensas has joined #tripleo18:23
*** hjensas has joined #tripleo18:23
dsneddonkarthiks, I workflow +1'd the SRIOV network object patch. I think we still need https://review.openstack.org/#/c/541620/418:24
*** openstackgerrit has joined #tripleo18:25
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart-extras master: undercloud: add missing TLS environments when preparing containers  https://review.openstack.org/54544418:25
*** stevebaker has joined #tripleo18:26
* EmilienM waves on stevebaker18:27
*** hjensas has quit IRC18:27
*** jrist is now known as jrist-afk18:28
*** jrist-afk is now known as jrist18:28
*** rwsu has joined #tripleo18:29
*** tesseract has quit IRC18:30
EmilienMmandre: what's the difference between ConfigImage and Image in tripleo docker configs ?18:30
EmilienMor anyone ^18:30
*** vpickard_ is now known as vpickard18:31
*** trown|lunch is now known as trown18:39
*** hjensas has joined #tripleo18:44
*** hjensas has joined #tripleo18:44
*** moshele has joined #tripleo18:45
*** tosky has joined #tripleo18:46
*** rwsu has quit IRC18:46
*** rbrady has quit IRC18:50
*** lucasagomes is now known as lucas-brb18:57
*** rwsu has joined #tripleo19:01
dtrainori'm not having much luck using a single node ceph deployment with three controllers in HA.  afaik, the inclusion of environments/ceph-ansible/ceph-ansible.yaml should be sufficient, correct?  I see images start to be created, but they stay stuck in the "Saving" status, I don't think the deployment is completely set up.  I see many old bugs about similar behavior but they all appear to be quite old19:01
dtrainorI just want to confirm that the inclusion of environments/ceph-ansible/ceph-ansible.yaml is sufficient, or if I'm missing a piece to enable the overcloud to actually use the ceph backend19:02
*** wolverineav has joined #tripleo19:02
*** rbrady has joined #tripleo19:02
*** rbrady has joined #tripleo19:02
*** gfidente is now known as gfidente|afk19:05
fultonjdtrainor: including environments/ceph-ansible/ceph-ansible.yaml should install ceph in your overcloud but you would override params in that file, e.g. your list of OSDs etc19:07
*** myoung|bbl is now known as myoung19:07
fultonjyou said "single node ceph deployment with three controllers" --> 3 controllers = 3 mons on each controller by default. i assume you mean a single ceph storage server to run OSDs19:08
dtrainorright, those defaults appear to be sufficient; CinderEnableRbdBackend and GlanceBackend are about the only ones that I need, afaik19:08
dtrainoryes, a single osd on the dedicated ceph node, 3 mons, one per each controller19:08
fultonjdtrainor: do you have a dedicated block device to use as an OSD?19:08
fultonjit defaults to a device called /dev/sdb but you may need to overrride that19:09
fultonjhttps://docs.openstack.org/tripleo-docs/latest/install/advanced_deployment/ceph_config.html19:09
dtrainori do, and ceph-ansible applied properly, as i was able to see the output from the mistral workflow output in the undercloud update and complete successfully19:09
dtrainorright, i had a deep dive when it didn't work out on account of some weird partition stuff that was on /dev/sdb; i just zeroed out the mbr and it was able to be consumed19:10
*** ooolpbot has joined #tripleo19:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION19:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/174964119:10
*** ooolpbot has quit IRC19:10
openstackLaunchpad bug 1749641 in tripleo "Overcloud deployment failing in promotion jobs at glance-manage db_sync" [Critical,In progress] - Assigned to Christian Schwede (cschwede)19:10
dtrainor(iirc the default is /dev/vdb)19:10
*** dtantsur is now known as dtantsur|afk19:10
fultonjdtrainor: yes, and you'd need to clean the disk (ironic can do it for you too)19:10
fultonjdtrainor: so the install was good and you're now trying to use glance and can't upload an image?19:11
dtrainoryep. i got some good hands-on experience with that too, really nice to learn19:11
fultonj"I see images start to be created, but they stay stuck in the"19:11
*** colonwq has joined #tripleo19:12
dtrainorcorrect, images stay in an "Saving" state indefinitely.  unfortunately i don't have any more information since i tore down the overcloud to re-attempt this, but before doing that i wanted to confirm i was at least configuring my deployment sufficiently for the usage of ceph19:12
fultonjdtrainor: yes, that should be all that was necessary19:12
fultonjdtrainor: using master branch?19:12
*** salmankhan has quit IRC19:12
dtrainorsorry, i should have mentioned that:  pike19:12
fultonjdtrainor: try SSH'ing into a controller and see if 'ceph -s' is happy19:12
dtrainorpike, being that this is also rhosp1219:13
*** salmankhan has joined #tripleo19:13
fultonjyes19:13
dtrainorsure, i'll give it another deployment and poke at it, thank you19:13
fultonjok19:13
fultonjnp19:13
dtrainori'm afraid that my limited ceph troubleshooting experience is hindering the post-deployment investigation19:13
*** ykarel|away has quit IRC19:13
fultonjwhen debugging ceph in pike and newer remember that ceph is running in containers19:13
dtrainorright, ceph-ansible instead of puppet-ceph to facilitate that19:14
fultonjyes19:14
fultonji recommend you try with a small image like cirros19:14
fultonjand use cli to upload19:14
dtrainorthat's what i had been doing, both cli and horizon19:14
*** jpena|away is now known as jpena|off19:14
fultonjdtrainor: did you override defaults for only 1 OSD?19:15
fultonjhttps://review.openstack.org/#/c/544588/4/environments/low-memory-usage.yaml19:16
fultonjdtrainor: you don't have to include this file (it's in master only) but you could use those two lines19:16
fultonjif you only have one OSD (/dev/sdb) then you need to do that or the ceph cluster will not be healthy19:16
dtrainorI see.  You're right, I don't see that parameter in my t-h-t nor anywhere in ceph-ansible19:18
*** jmelvin is now known as jmelvin|bomgar19:19
dtrainorI don't know what I was looking at previously, but I did find CephPoolDefaultSize inside of t-h-t now.  I'll play with that.19:21
fultonjthey default to 128 PGs and 3 replicas19:21
fultonjtoo much for only 1 disk19:21
fultonjdefaults are targetted at minimum prod recomendations (3 storage etc, multiple OSDs, etc)19:22
fultonjeasy enough to override for test env19:22
fultonjtest-only19:22
dtrainorRight, I found 'em.  I see it now, I'm going to deploy again.19:23
dtrainorThank you, this helps immensely.19:23
fultonjdtrainor: also before deployment try converting image to raw directly so ceph doesn't try to do the conversion for you19:23
fultonjqemu-img convert -f qcow2 -O raw $img $raw19:24
fultonjimg=cirros-0.3.4-x86_64-disk.img19:25
dtrainorgood call.  thank you.19:25
fultonjraw=$(echo $img | sed s/img/raw/g)19:25
fultonjopenstack image create cirros --disk-format=raw --container-format=bare < $raw19:25
dtrainorshould be a fairly quick deployment, lots of cores / memory / network / fast disk to work with19:25
dtrainorright19:25
tonybfultonj: I believe I am, for better or worse.  I haven't looked very hard at ceph yet but it's on my plan for this year.19:27
fultonjtonyb: no worries on ceph bit, i can talk about that :)19:27
tonyb:)19:28
fultonjtonyb: so we could be co-drivers on the integration session?19:28
fultonj(hasn't tried multiarch)19:28
*** salmankhan has quit IRC19:28
tonybfultonj: Sure,  If that mostly means come up with a list of challenges and then enlist the smart people in the room to help design solutions \o/19:28
fultonjtonyb: ok, thanks.19:29
tonybfultonj: Wait 'till the mutli-arch bueprint lands it'll be much less painful ;p19:29
fultonjEmilienM mwhahaha if you need to fill a driver slot for integration in the ptg schedule, then tonyb and I volunteer19:30
mwhahahasounds good thanks19:30
EmilienMfultonj: ok, I'll add you, thanks19:31
fultonjthanks19:32
*** dsariel has quit IRC19:33
*** moshele has quit IRC19:36
*** moshele has joined #tripleo19:38
*** moshele has quit IRC19:42
mandreEmilienM: we're using a separate ConfigImage variable for separation of concerns, otherwise it was just too easy to have a config_volume used in separate places where different images were specified19:48
mandreEmilienM: now we have a validation that a config_volume has a XXXConfigImage variable associated19:48
mandreEmilienM: see https://review.openstack.org/#/c/476970/ for the rational19:48
EmilienMmandre: ok, I see19:51
*** Goneri has quit IRC19:57
*** lucas-brb is now known as lucasagomes19:58
EmilienMneed urgent review on https://review.openstack.org/#/c/545710/ please19:59
*** toure is now known as toure|biab20:01
tonybfultonj: ... Oh I misunderstood your question before.  I thought there was a specific session on "MultiArch Ceph", but readeding the schedule it seems like it's more an integration session touching on Ceph and Multiarch as more or less distinct topics.  I'm much more comfortable with the latter20:05
fultonjtonyb: cool20:05
openstackgerritMerged openstack/tripleo-docs master: Update manila deployment doc  https://review.openstack.org/54407720:06
tonybalso "readeding" is now a real word20:06
fultonjfunny, i read what you wrote and my brain told me it was "re-reading"20:06
tonybhehe20:07
fultonjit's like it was symlinked20:07
tonybbrains are good like that :)20:07
*** juriarte has quit IRC20:08
mwhahahaEmilienM: did we already do the version bump for puppet-tripleo for rc1?20:09
*** ooolpbot has joined #tripleo20:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION20:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/174964120:10
*** ooolpbot has quit IRC20:10
openstackLaunchpad bug 1749641 in tripleo "Overcloud deployment failing in promotion jobs at glance-manage db_sync" [Critical,In progress] - Assigned to Christian Schwede (cschwede)20:10
EmilienMmwhahaha: no20:13
EmilienMmwhahaha: in metadata?20:13
EmilienMlet me check20:13
mwhahahayea20:13
EmilienMhttps://github.com/openstack/puppet-tripleo/commit/37205689bd8e0c1a88e3c2bb60a34e35e38497c720:13
EmilienMno20:13
EmilienMwe need to prepare 8.3.020:13
EmilienMand for tripleo-ui too20:13
* EmilienM afk lunch20:13
mwhahahak20:13
*** lucasagomes is now known as lucas-afk20:14
openstackgerritMerged openstack/tripleo-heat-templates master: Disable UseDNS in sshd config  https://review.openstack.org/54495720:15
fultonji'm looking for the output of openstack-infra/tripleo-ci/scripts/bootstrap-overcloud-full-minimal.sh in the results of a run of periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset016-queens20:15
fultonjcan anyone give me an idea of where to look?20:16
fultonjhttps://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset016-queens/8b4282d/20:16
fultonji'm clicking around in ^20:16
fultonjrover?20:16
*** wolverineav has quit IRC20:19
openstackgerritMerged openstack/tripleo-heat-templates master: Add pacemaker upgrade_tasks for P..Q major upgrade  https://review.openstack.org/53924920:21
openstackgerritAlex Schultz proposed openstack/puppet-tripleo master: Prep puppet-tripleo metadata for Queens rc1  https://review.openstack.org/54596820:21
*** moshele has joined #tripleo20:22
*** amoralej is now known as amoralej|off20:23
openstackgerritAlex Schultz proposed openstack/tripleo-ui master: Prepare Queens RC1  https://review.openstack.org/54596920:23
*** janki has quit IRC20:28
openstackgerritAlex Schultz proposed openstack/instack-undercloud master: Ensure FORWARD is ACCEPT by default  https://review.openstack.org/54571020:30
*** etingof has joined #tripleo20:31
*** toure|biab is now known as toure20:33
*** pkovar has quit IRC20:35
*** itlinux has quit IRC20:36
*** dsariel has joined #tripleo20:37
*** moshele has quit IRC20:40
openstackgerritMerged openstack/tripleo-heat-templates stable/pike: Adding new config parameters for Cisco UCSM ML2 driver  https://review.openstack.org/54436720:45
openstackgerritMerged openstack/puppet-tripleo master: Fix galera deployment when stack's name has capital letters  https://review.openstack.org/54438420:45
openstackgerritMerged openstack/tripleo-heat-templates master: Revert "Enable *_use_fqdn in ceph-ansible when EnableInternalTLS"  https://review.openstack.org/54386820:45
openstackgerritMerged openstack/tripleo-heat-templates stable/pike: Fix a typo in docker_puppet_apply.sh  https://review.openstack.org/54575920:45
openstackgerritMerged openstack/tripleo-heat-templates master: undercloud: remove duplicate Neutron Server entry  https://review.openstack.org/54546620:45
openstackgerritMerged openstack/tripleo-heat-templates master: Add TripleOFirewall service to undercloud roles  https://review.openstack.org/54520920:46
openstackgerritMerged openstack/tripleo-puppet-elements master: Create element to purge resolv.conf  https://review.openstack.org/54506720:46
openstackgerritMerged openstack/tripleo-common master: Fix chown command for tripleo validation.  https://review.openstack.org/50963920:46
openstackgerritMerged openstack/tripleo-heat-templates master: Simplify FastForwardRepoArgs structure used in FFU repo selection.  https://review.openstack.org/54526120:46
openstackgerritMerged openstack/tripleo-common master: container-images: support cont. u/c w/ HAproxy & keepalived  https://review.openstack.org/54574420:46
openstackgerritMerged openstack/tripleo-quickstart-extras master: Adding tempest_skip_queens.yml file  https://review.openstack.org/54581020:46
*** sai___ has joined #tripleo20:46
openstackgerritMerged openstack-infra/tripleo-ci master: Allow for multiple playbooks to be run  https://review.openstack.org/52576520:46
openstackgerritMerged openstack/puppet-tripleo master: Add firewall chain support  https://review.openstack.org/54570720:46
openstackgerritMerged openstack/tripleo-heat-templates master: Allow passing custom openshift-ansible playbook  https://review.openstack.org/54340420:46
openstackgerritMerged openstack/tripleo-heat-templates master: Ensure node is rebooted before enabling DPDK  https://review.openstack.org/54262320:46
openstackgerritWojciech Dec proposed openstack/tripleo-heat-templates master: Add cisco VTS ML2 template for a dockerized service and default environment settings  https://review.openstack.org/53335120:46
openstackgerritMerged openstack/os-net-config master: Add network objects sriov_vf and sriov_pf  https://review.openstack.org/52757120:46
openstackgerritMerged openstack/puppet-tripleo master: Change URL for puppet-cassandra  https://review.openstack.org/54540020:46
*** wolverineav has joined #tripleo20:50
openstackgerritDamien Ciabrini proposed openstack/puppet-tripleo stable/pike: Fix galera deployment when stack's name has capital letters  https://review.openstack.org/54597920:50
*** wolverineav has quit IRC20:54
openstackgerritAlan Bishop proposed openstack/puppet-tripleo stable/pike: Avoid hard-coded settings in Cinder HA containers  https://review.openstack.org/54583521:02
*** wolverineav has joined #tripleo21:03
*** rbrady has quit IRC21:06
*** wolverineav has quit IRC21:08
*** ooolpbot has joined #tripleo21:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION21:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/174964121:10
*** ooolpbot has quit IRC21:10
openstackLaunchpad bug 1749641 in tripleo "Overcloud deployment failing in promotion jobs at glance-manage db_sync" [Critical,In progress] - Assigned to Christian Schwede (cschwede)21:10
*** ccamacho has quit IRC21:12
*** pchavva has quit IRC21:15
dtrainorfultonj, ceph++ thank you so much for your help21:15
fultonjdtrainor: cool21:16
*** moshele has joined #tripleo21:18
*** salmankhan has joined #tripleo21:19
*** rbrady has joined #tripleo21:19
*** wolverineav has joined #tripleo21:20
*** dtrainor has quit IRC21:22
*** dtrainor has joined #tripleo21:23
*** wolverineav has quit IRC21:24
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates master: FFU: Introduce fluentd fast-forward-upgrade tasks  https://review.openstack.org/54598621:25
*** moshele has quit IRC21:30
mwhahahardo cloud unhappy?21:31
openstackgerritIhar Hrachyshka proposed openstack/tripleo-quickstart master: Added a troubleshooting tip to clean up ~/.quickstart  https://review.openstack.org/53700821:31
* mwhahaha pokes arxcruz|ruck, rlandy|rover21:31
rlandy|rovermwhahaha: responding to poke21:32
mwhahahaseeing ovb issues getting nodes21:32
mwhahaharlandy|rover: known thing?21:32
rlandy|rovermwhahaha: yeah - looks like the tenant is full - not enough resources21:32
rlandy|roverwe likely need to clean up21:32
mwhahaha ClientException: resources.baremetal_server: Unknown Error (HTTP 504)21:32
rlandy|roverwas just looking at that21:33
*** dprince has quit IRC21:33
rlandy|roverprobably stacks were left undeleted or in create/delete failed during maintenance21:33
mwhahaha:(21:34
rlandy|roverlet me see if I can clean up21:34
*** rcernin has joined #tripleo21:36
rlandy|roveroh yeah - we have stack sitting there from 9 and 10 hours ago21:36
*** threestrands has joined #tripleo21:37
*** pkovar has joined #tripleo21:40
*** pkovar has quit IRC21:40
rlandy|rovermwhahaha: ok - in process of deleting stacks but it takes a few minutes per stack. the good news is that we don't have many stacks in create-failed which are harder to get rid of21:41
mwhahaharlandy|rover: thanks21:42
*** Goneri has joined #tripleo21:42
*** holser has joined #tripleo21:45
openstackgerritMerged openstack/tripleo-ui master: Prepare Queens RC1  https://review.openstack.org/54596921:52
*** aputtur_ has quit IRC21:52
fultonjrlandy|rover: can you comment on https://bugs.launchpad.net/tripleo/+bug/1750311/comments/7 ?21:52
openstackLaunchpad bug 1750311 in tripleo "[queens promotion] - overcloud deployment fails step2: /dev/loop3 missing" [Critical,Triaged] - Assigned to John Fulton (jfulton-org)21:52
rlandy|roverlooking21:53
fultonji'm hoping you can help me find what i'm looking for21:53
*** wolverineav has joined #tripleo21:53
fultonji.e. where's bootstrap-subnodes.log.txt.gz ?21:53
*** jcoufal has quit IRC21:55
*** trown is now known as trown|outtypewww21:56
*** wolverineav has quit IRC21:58
rlandy|roverha - depends if we collect it - looking21:58
rlandy|roverfultonj: have you seen  bootstrap-subnodes.log.txt.gz in logs before?22:04
*** raildo has quit IRC22:04
fultonjrlandy|rover: in the past22:04
rlandy|roverfultonj, under which exact dir? undercloud/var/log/?22:04
fultonjrlandy|rover: example, on July 25th, 2017 i looked at http://logs.openstack.org/63/484963/9/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq-puppet/4a7aabd/logs/undercloud/var/log/bootstrap-subnodes.log.txt.gz22:05
fultonjas per the comment in https://review.openstack.org/#/c/484963/22:05
fultonjthat job has since been garbage collected22:06
fultonjerrr the output of that job i should say22:06
rlandy|roveryeah - the logs rotate22:06
fultonjof course22:06
rlandy|roverso where did it go or did we stop collecting it?22:06
fultonjroot cause: /dev/loop3 is missing ; next logical question, did the job that make it fail? no logs22:06
fultonjexactly22:07
*** hamzy has quit IRC22:07
rlandy|roverwell we can check a passing jon22:07
rlandy|roverjob22:07
fultonjgood idea22:07
* rlandy|rover gets22:07
rlandy|roverhttps://logs.rdoproject.org/openstack-periodic-24hr/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset016-pike/9dd88ec/undercloud/var/log/22:08
rlandy|roverthat is fs016 on pike22:08
rlandy|roverwhich passes22:08
*** florianf has quit IRC22:09
fultonjand doesn't have the same output22:09
*** florianf has joined #tripleo22:09
fultonjidea...22:09
rlandy|roverbut no  bootstrap-subnodes.log.txt.gz22:09
fultonjmake a patch to collect it, depend on it, see what we get22:09
fultonjwhat do you think?22:09
rlandy|roverso we could be collecting logs diff22:09
fultonjit has been 6 months22:10
rlandy|roverlet's check the log collection on that dir22:10
*** ooolpbot has joined #tripleo22:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION22:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/174964122:10
*** ooolpbot has quit IRC22:10
openstackLaunchpad bug 1749641 in tripleo "Overcloud deployment failing in promotion jobs at glance-manage db_sync" [Critical,In progress] - Assigned to Christian Schwede (cschwede)22:10
rlandy|roverso here is the default: https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/collect-logs/defaults/main.yml22:11
rlandy|roverbut ...22:11
fultonjhttps://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/collect-logs/defaults/main.yml#L11922:11
fultonj... but? ...22:12
rlandy|roverwas just checking if the featureset overrides it - but it does not22:13
fultonjgiven what I see in collect-logs/defaults/main.yml#L119 , am i correct that it should be collected?22:13
rlandy|roversorry - was just checking multinode jobs - not there either22:15
rlandy|roverI think we need some .* there22:15
fultonjahhh22:16
fultonjgood call22:16
*** hamzy has joined #tripleo22:16
rlandy|roverlooks like we are picking up top level22:16
rlandy|roverexcept for a few others22:17
openstackgerritAlex Schultz proposed openstack/tripleo-heat-templates master: Add firewall chain configuration  https://review.openstack.org/54599922:18
rlandy|roveridk - we do see some .log files22:18
*** openstackgerrit has quit IRC22:18
rlandy|roverfultonj: I think we should put in a review (not to merge) to collect that file explicitly - that way we can tell if it is there and not being collected or not there to start with22:19
fultonjrlandy|rover: if there's a patch to add the .* on line 119, then can a trivial change to fs016 depend on it and then we can see22:19
rlandy|roverfultonj: yeah - we can add that and run openstack-experimental - just checking of that kicks fs01622:21
rlandy|roversec22:21
rlandy|roverno wait - you have the file listed22:22
*** wolverineav has joined #tripleo22:22
fultonj?22:22
fultonjyou mean on line 119?22:22
fultonji thought you meant, when you suggested * that the path on 119 might be expired22:23
*** morazi has quit IRC22:26
*** wolverineav has quit IRC22:27
*** openstackgerrit has joined #tripleo22:28
openstackgerritAlex Schultz proposed openstack/tripleo-heat-templates master: Add firewall chain configuration  https://review.openstack.org/54599922:28
rlandy|roverhmmm ... no quick path to fs01622:35
rlandy|roverok - openstack-experimental on a  openstack-infra/tripleo-ci change will rrun fs01622:37
rlandy|roverfultonj: ^^22:38
rlandy|roverthe log collection *should* be collecting everything in /var/log22:38
rlandy|roverand you have the file explicitly listed to be stored22:39
EmilienMpradk: if you can review https://review.openstack.org/#/c/544530/ and vote that would help, thanks22:39
fultonjrlandy|rover: so the log wasn't there?22:39
fultonjwasn't there to be collected?22:39
rlandy|roverfultonj: well that would be my guess but to be sure, we can add a DNM change to explicitly collect that file and see22:40
rlandy|roverI was looking to see if we could tell if the script you pointed out actually run22:40
fultonjif it wasn't created, then there'd be no ouput from 'sudo yum -y install git python-heat-agent*'22:40
rlandy|roverreally any multinode jobs should pick it up22:41
fultonjand the deploy would have failed earlier if it hadn't run22:41
rlandy|roveryet we see it nowhere22:41
EmilienMpradk: thanks22:41
pradknp22:41
EmilienMpradk, mwhahaha: and https://review.openstack.org/#/c/544997/22:41
EmilienMso we can test it in CI again22:41
*** abishop has quit IRC22:42
fultonjrlandy|rover: what would the DNM change be? i originally thought it would be line 119 https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/collect-logs/defaults/main.yml#L11922:43
openstackgerritMerged openstack/puppet-tripleo master: Set rgw_keystone_revocation_interval to 0 for ceph::rgw::keystone  https://review.openstack.org/54182622:43
fultonjrlandy|rover: if you know what it is and want to put it up, then i could see that way (may be easier than explaining)22:43
fultonjis it possible that "${TOCI_JOBTYPE:-''}" did not contain "multinode"?22:44
fultonjrlandy|rover: ^ because that may explain22:44
rlandy|roverperiodic-tripleo-ci-centos-7-multinode-1ctlr-featureset016-queens22:45
fultonjso it would be a change to periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset016-queens , e.g. an arebitrary change just to see it run22:46
*** bfournie1 has quit IRC22:46
fultonjarbitrary*22:46
*** bfournie has joined #tripleo22:46
rlandy|roverI was thinking something like an ls on var/log so we could see what is there22:46
fultonjrlandy|rover: oh ok, that makes sense22:47
EmilienMrlandy|rover: do we still have ovb jobs in stable/newton ?22:48
EmilienMI think no, and i wonder why22:48
rlandy|roverafaik, all newton testing has gone to phase 222:49
rlandy|roverpahse 122:50
EmilienMok22:50
EmilienMthanks22:50
rlandy|roverso, no22:50
rlandy|roverEmilienM: do we need a newton ovb job?22:51
EmilienMrlandy|rover: yes for puppet-tripleo22:51
EmilienMtemporary maybe22:51
EmilienMif you can update the ci config for that, it would really help me22:51
EmilienMI have a few backports before we EOL22:52
* EmilienM bbl22:52
rlandy|roverEmilienM: sure but I need a few minutes  - just finishing up with fultonj and cleaning up rdo cloud tenant atm22:52
fultonjrlandy|rover: do you mind putting up that DNM change to ls /var/log ?22:53
*** bfournie has quit IRC22:53
rlandy|roverfultonj: sure - this would be really easy if rdocloud personal tenant were working22:53
chempanda: hey, I've got a successful o->p upgrade using the tripleo-upgrade role there https://review.openstack.org/#/c/543907/22:53
rlandy|roverthen we could just access the undercloud node and review it22:53
chempanda: the simple ping validation at then end, after the converge is successful22:54
fultonjrlandy|rover: ah22:54
rlandy|roverthat is what the reproducer script is for22:54
chempanda: at least we know it's working :)22:54
*** dparkes has joined #tripleo22:54
rlandy|roverfultonj: https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset016-queens/6acce50/reproducer-quickstart.sh22:55
rlandy|rover^^ to run that on personal tenant22:55
chemEmilienM: new tripleo upgrade role is working for o->p with oooq integration ^22:55
chemEmilienM: that will be my good new of the day :)22:55
rlandy|roverfultonj; I can try that tonight on my tenant22:55
fultonjrlandy|rover: thanks22:56
*** jmelvin|bomgar has quit IRC22:58
*** maeca has quit IRC23:03
chemEmilienM: I believe the gate, but I'll test therve's patch for p->q https://review.openstack.org/#/c/545856/ first thing in the morning (I'll trigger the backport if not there)23:05
chemEmilienM: (I believe the gate/upstream ci are overloaded)23:06
*** salmankhan has quit IRC23:06
chemEmilienM: and we would need that patch all the way back to newton for ffu scenario ...23:06
chemand with that I'm gone23:06
*** chem is now known as chem|eod23:06
*** wolverineav has joined #tripleo23:07
*** liverpooler has quit IRC23:10
*** ooolpbot has joined #tripleo23:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION23:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/174964123:10
*** ooolpbot has quit IRC23:10
openstackLaunchpad bug 1749641 in tripleo "Overcloud deployment failing in promotion jobs at glance-manage db_sync" [Critical,In progress] - Assigned to Christian Schwede (cschwede)23:10
mwhahaharlandy|rover: still seeing 504s on ovb from ~30 mins ago. btw are the ovb jobs not on cistatus.tripleo.org?23:10
mwhahahaoh there they are23:11
mwhahahanot sure why they aren't on the check jobs page23:11
rlandy|rovermwhahaha: http://38.145.34.234/- should see them23:11
*** pradk has quit IRC23:12
*** wolverineav has quit IRC23:12
mwhahaharlandy|rover: why aren't they part of http://cistatus.tripleo.org/23:12
mwhahahaor is that two different systems so they can't be joined in views23:12
*** agopi has quit IRC23:15
rlandy|rovermwhahaha: wrt errors, I am still cleaning stacks but we should have enough deleted to be able to run new jobs23:15
mwhahahahmm ok23:16
*** pmannidi has joined #tripleo23:19
*** paramite has quit IRC23:22
*** jbadiapa has quit IRC23:22
*** wolverineav has joined #tripleo23:33
*** holser has quit IRC23:34
*** jbadiapa has joined #tripleo23:35
*** panda is now known as panda|off23:37
*** wolverineav has quit IRC23:39
rlandy|roverEmilienM: is this what you wanted for newton: https://review.rdoproject.org/r/#/c/12470/?23:41
*** bfournie has joined #tripleo23:52
rlandy|rovermwhahaha: there are still some stacks in delete-failed (which I'll try clean up later) but the openstack-nodepool tenant should be functional now23:54
openstackgerritAttila Darazs proposed openstack/tripleo-quickstart-extras master: GATE CHECK for quickstart-extras  https://review.openstack.org/47260723:57

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!