Tuesday, 2018-04-24

*** pcaruana has quit IRC00:02
*** lblanchard has quit IRC00:04
*** zshi has quit IRC00:04
*** ooolpbot has joined #tripleo00:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION00:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/176619500:10
*** ooolpbot has quit IRC00:10
openstackLaunchpad bug 1766195 in tripleo "Error installing fluent-plugin-kubernetes_metadata_filter at pike" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)00:10
openstackgerritEmilien Macchi proposed openstack/python-tripleoclient master: Move test_container_image.py  https://review.openstack.org/56379300:10
*** zshi has joined #tripleo00:16
*** zshi has quit IRC00:21
hjensasEmilienM: I used the button in gerrit, but for some reason it did'nt include the cherry-picked entry in commit message.00:21
hjensasEmilienM: So I hand edited them in.00:22
*** jcoufal_ has quit IRC00:23
openstackgerritEmilien Macchi proposed openstack/python-tripleoclient master: WIP - Implement CloudImagePrepare command  https://review.openstack.org/56379400:24
EmilienMhjensas: because your patch wasn't merged on master00:24
EmilienMhjensas: next time, do the cherry-pick manually with git00:24
EmilienMand -x option00:24
EmilienMhjensas: create a local branch on queens, and find the cherry-pick command in the patch on master, and add -x00:24
EmilienMhjensas: https://prnt.sc/j98xe100:25
* EmilienM out00:25
*** rmascena__ has quit IRC00:25
hjensasEmilienM: ah, cause it was'nt merged. I see, will use -x option or wait till things are merged on master next time.00:26
EmilienMcool00:26
*** dhill_ has joined #tripleo00:27
*** sai_p has quit IRC00:29
*** tonyb has quit IRC00:33
*** tonyb has joined #tripleo00:35
*** dxiri has quit IRC00:40
*** zshi has joined #tripleo00:43
*** rlandy has quit IRC00:52
*** jabadia has joined #tripleo01:06
*** ooolpbot has joined #tripleo01:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION01:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/176619501:10
openstackLaunchpad bug 1766195 in tripleo "Error installing fluent-plugin-kubernetes_metadata_filter at pike" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)01:10
*** ooolpbot has quit IRC01:10
openstackgerritMerged openstack/tripleo-heat-templates stable/queens: ovn: Mount /run instead of /run/openvswitch when starting ovn-controller container  https://review.openstack.org/56253801:12
openstackgerritMerged openstack/tripleo-common stable/queens: Fix error handling in set_provision_state/set_power_state workflows  https://review.openstack.org/55931401:19
*** yamahata has quit IRC01:26
openstackgerritSai Ram Peesapati proposed openstack/tripleo-heat-templates master: Parameterizing Puppet Tags  https://review.openstack.org/55759801:37
*** shardy has quit IRC01:42
*** shardy has joined #tripleo01:43
*** ramishra has joined #tripleo01:52
*** fragatina has quit IRC02:02
*** ohochman has quit IRC02:05
*** ohochman has joined #tripleo02:09
*** ooolpbot has joined #tripleo02:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION02:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/176619502:10
*** ooolpbot has quit IRC02:10
openstackLaunchpad bug 1766195 in tripleo "Error installing fluent-plugin-kubernetes_metadata_filter at pike" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)02:10
*** jabadia has quit IRC02:17
*** fragatina has joined #tripleo02:39
*** dxiri has joined #tripleo02:46
*** psachin has joined #tripleo02:47
*** ooolpbot has joined #tripleo03:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION03:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/176619503:10
*** ooolpbot has quit IRC03:10
openstackLaunchpad bug 1766195 in tripleo "Error installing fluent-plugin-kubernetes_metadata_filter at pike" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)03:10
*** dxiri has quit IRC03:16
*** shreshtha_ has joined #tripleo03:20
*** dxiri has joined #tripleo03:23
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: WIP - heat: align config with instack-undercloud  https://review.openstack.org/55835403:33
EmilienMweshay: tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates is broken it seems03:34
EmilienMit's blocking the gate03:37
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci master: Disable voting on tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates  https://review.openstack.org/56380603:40
EmilienMmwhahaha: ^ in case you're around or I'll self approve03:40
mwhahahaK03:40
EmilienMit looks like a timeout03:41
*** janki has joined #tripleo03:44
*** ykarel|away has joined #tripleo04:05
*** ooolpbot has joined #tripleo04:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION04:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/176619504:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/176640604:10
*** ooolpbot has quit IRC04:10
openstackLaunchpad bug 1766195 in tripleo "Error installing fluent-plugin-kubernetes_metadata_filter at pike" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)04:10
openstackLaunchpad bug 1766406 in tripleo "multinode-oooq-container-updates fails on master" [Critical,Triaged]04:10
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci master: Don't cleanup undercloud  https://review.openstack.org/56377704:13
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Fail validation on new workflow_tasks  https://review.openstack.org/56165704:13
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Suppress warnings with -qq  https://review.openstack.org/56165604:14
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Fail validation on new workflow_tasks  https://review.openstack.org/56165704:14
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci master: Update undercloud install directories  https://review.openstack.org/56376904:14
openstackgerritEmilien Macchi proposed openstack/tripleo-common master: Fix nova-placement healthcheck  https://review.openstack.org/56379104:15
openstackgerritEmilien Macchi proposed openstack/puppet-tripleo stable/queens: Firewall: NOT persist ephemetal ironic-inspector rules  https://review.openstack.org/56358004:15
openstackgerritEmilien Macchi proposed openstack/python-tripleoclient master: Fix ansible config download dir  https://review.openstack.org/56331704:16
openstackgerritEmilien Macchi proposed openstack/python-tripleoclient master: Save temporary working data  https://review.openstack.org/56057804:16
openstackgerritEmilien Macchi proposed openstack/python-tripleoclient master: Enable cleanup by default for undercloud install  https://review.openstack.org/56203204:16
openstackgerritEmilien Macchi proposed openstack/python-tripleoclient master: Move test_container_image.py  https://review.openstack.org/56379304:19
openstackgerritEmilien Macchi proposed openstack/python-tripleoclient master: WIP - Implement CloudImagePrepare command  https://review.openstack.org/56379404:19
*** anilvenkata has joined #tripleo04:19
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates stable/queens: Fix Cinder's default db purge cron settings  https://review.openstack.org/56015404:20
openstackgerritEmilien Macchi proposed openstack/tripleo-common stable/queens: Bump Ceph container image from 3.0.1 to 3.0.3  https://review.openstack.org/56301704:20
*** ratailor has joined #tripleo04:20
*** ratailor has quit IRC04:21
*** tzumainn has quit IRC04:22
*** dxiri has quit IRC04:23
*** anilvenkata has quit IRC04:23
*** ratailor has joined #tripleo04:35
*** ratailor has quit IRC04:35
*** skarthik has joined #tripleo04:37
*** leitan has quit IRC04:37
*** leitan has joined #tripleo04:38
*** slaweq has joined #tripleo04:39
*** janki has quit IRC04:39
*** ykarel|away is now known as ykarel04:40
*** leitan has quit IRC04:42
openstackgerritMerged openstack/tripleo-heat-templates stable/queens: Enable management network in network_data  https://review.openstack.org/56363704:45
Tenguhello there04:45
*** slaweq has quit IRC04:47
Tenguhmmm, what's missing for https://review.openstack.org/#/c/563525/ to be merged now?04:52
Tenguguess the "workflow" line needs something04:52
*** skramaja has joined #tripleo04:56
*** pdeore has joined #tripleo04:56
openstackgerritItzik Brown proposed openstack/tripleo-heat-templates master: Add /var/log/opendaylight directory to ODL  https://review.openstack.org/56381905:05
*** yprokule has joined #tripleo05:09
*** ooolpbot has joined #tripleo05:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION05:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/176619505:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/176640605:10
*** ooolpbot has quit IRC05:10
openstackLaunchpad bug 1766195 in tripleo "Error installing fluent-plugin-kubernetes_metadata_filter at pike" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)05:10
openstackLaunchpad bug 1766406 in tripleo "multinode-oooq-container-updates fails on master" [Critical,Triaged]05:10
*** dhill_ has quit IRC05:15
*** rajinir has quit IRC05:18
*** paramite has joined #tripleo05:21
*** agurenko has joined #tripleo05:22
*** waleedm has joined #tripleo05:23
*** dparkes has quit IRC05:24
*** dparkes has joined #tripleo05:27
*** yolanda_ has joined #tripleo05:27
*** yolanda has quit IRC05:28
*** dhill_ has joined #tripleo05:29
*** paramite has quit IRC05:32
*** dparkes has quit IRC05:32
*** dhill_ has quit IRC05:35
*** dhill_ has joined #tripleo05:36
*** ccamacho has quit IRC05:37
*** marios has joined #tripleo05:39
*** ratailor has joined #tripleo05:40
*** quiquell|off is now known as quiquell|rover05:40
*** dparkes has joined #tripleo05:41
openstackgerritRajesh Tailor proposed openstack/tripleo-common stable/queens: Add healthcheck script for nova-metadata service  https://review.openstack.org/56382805:43
*** dparkes has quit IRC05:46
Tengushardy: hello! I think I'll need some of your time, once again :/. I have a "deploy" running, but it always fails with a timeout for the following task: overcloud-AllNodesDeploySteps-onjhxal6gw5a.WorkflowTasks_Step2_Execution - I've tried to know a bit about it, but I just can't find what host doesn't send back its state, and nothing seems to be staled on the nodes :/. I'm lost.05:47
openstackgerrityatin proposed openstack/tripleo-heat-templates master: Add neutron "segments" plugin to the default list  https://review.openstack.org/56293205:48
*** janki has joined #tripleo05:49
*** dmacpher has joined #tripleo06:09
*** ooolpbot has joined #tripleo06:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION06:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/176619506:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/176640606:10
*** ooolpbot has quit IRC06:10
openstackLaunchpad bug 1766195 in tripleo "Error installing fluent-plugin-kubernetes_metadata_filter at pike" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)06:10
openstackLaunchpad bug 1766406 in tripleo "multinode-oooq-container-updates fails on master" [Critical,Triaged]06:10
*** links has joined #tripleo06:16
*** dparkes has joined #tripleo06:21
*** paramite has joined #tripleo06:21
openstackgerrityolanda.robla proposed openstack/tripleo-heat-templates stable/pike: Change ovs user and fix permissions on ovs upgrade  https://review.openstack.org/56161006:21
*** jtomasek has joined #tripleo06:25
*** anande has joined #tripleo06:28
*** aufi has joined #tripleo06:29
*** artom has quit IRC06:29
*** artom_ has joined #tripleo06:29
*** slaweq has joined #tripleo06:32
*** threestrands has quit IRC06:32
*** kopecmartin has joined #tripleo06:33
*** gfidente has joined #tripleo06:35
*** gfidente has quit IRC06:35
*** gfidente has joined #tripleo06:35
*** moshele has joined #tripleo06:39
openstackgerritYurii Prokulevych proposed openstack/tripleo-upgrade stable/queens: Use --roles in overcloud upgrade run.  https://review.openstack.org/56383706:41
*** slaweq has quit IRC06:42
mosheleskramaja: hi can you +W this https://review.openstack.org/#/c/560261/?06:43
skramajamoshele: done.06:44
mosheleskramaja: thanks :)06:44
mosheleskramaja: and also https://review.openstack.org/#/c/559723/06:45
*** athomas has joined #tripleo06:46
Tenguanyone can w+1 https://review.openstack.org/#/c/563525/ ? :)06:48
*** skarthik is now known as karthiks06:49
skramajamoshele: ci failed for it. i have given a recheck.06:49
*** tesseract has joined #tripleo06:49
*** threestrands has joined #tripleo06:50
*** threestrands has quit IRC06:50
*** threestrands has joined #tripleo06:50
*** yprokule has quit IRC06:52
skramajadone Tengu06:53
Tenguskramaja: thank you :)06:54
Tenguthat will prevent some issues when we link openstack to a freeIPA for authentication, among other issues :)06:54
*** ccamacho has joined #tripleo06:55
*** jfrancoa has joined #tripleo06:56
*** bogdando has joined #tripleo06:56
openstackgerritAndreas Florath proposed openstack/diskimage-builder master: WIP: Docker matrix build  https://review.openstack.org/41434706:57
openstackgerritHamdy Khader proposed openstack/puppet-tripleo master: Add NVMeOF support to configure cinder backend  https://review.openstack.org/55900806:57
*** pcaruana has joined #tripleo06:58
openstackgerritHamdy Khader proposed openstack/tripleo-heat-templates master: Add support for NVMeOF cinder backend  https://review.openstack.org/55957206:58
*** masco has joined #tripleo07:00
*** jfrancoa has quit IRC07:00
*** jfrancoa has joined #tripleo07:00
*** jfrancoa has quit IRC07:04
*** jfrancoa has joined #tripleo07:05
bogdandoPTAL https://review.openstack.org/#/c/556516/07:07
*** jaosorior has joined #tripleo07:08
bogdandohttps://review.openstack.org/#/c/56256607:08
*** jfrancoa1 has joined #tripleo07:08
*** aufi_ has joined #tripleo07:09
*** quiquell|rover is now known as quique|rover|afk07:09
*** gfidente has quit IRC07:09
openstackgerritArx Cruz proposed openstack/tripleo-quickstart-extras master: Install stestr only when release prior to ocata  https://review.openstack.org/56384307:09
*** jfrancoa has quit IRC07:10
*** ooolpbot has joined #tripleo07:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION07:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/176619507:10
openstackLaunchpad bug 1766195 in tripleo "Error installing fluent-plugin-kubernetes_metadata_filter at pike" [Critical,Triaged] - Assigned to Quique Llorente (quiquell)07:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/176640607:10
*** ooolpbot has quit IRC07:10
openstackLaunchpad bug 1766406 in tripleo "multinode-oooq-container-updates fails on master" [Critical,Triaged]07:10
*** threestrands has quit IRC07:10
*** yprokule has joined #tripleo07:10
*** aufi has quit IRC07:11
openstackgerritBogdan Dobrelya proposed openstack/tripleo-quickstart-extras master: Align UC net/env-files override with instack  https://review.openstack.org/56363607:11
*** amoralej|off is now known as amoralej07:11
openstackgerritArx Cruz proposed openstack/tripleo-quickstart-extras master: Install stestr only when release prior to ocata  https://review.openstack.org/56384307:11
*** suuuper has joined #tripleo07:13
*** sshnaidm|off is now known as sshnaidm07:14
*** slaweq has joined #tripleo07:16
*** rcernin has quit IRC07:16
*** gfidente has joined #tripleo07:17
*** gfidente has quit IRC07:17
*** gfidente has joined #tripleo07:17
*** jchhatbar has joined #tripleo07:21
*** janki has quit IRC07:23
*** agopi is now known as agopi|zzz07:24
*** florianf has joined #tripleo07:29
*** avivgt has joined #tripleo07:30
*** leanderthal has joined #tripleo07:33
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates master: Do not pass ceph_release to ceph-ansible  https://review.openstack.org/56363207:34
*** morazi has joined #tripleo07:40
*** holser__ has joined #tripleo07:40
*** ffiore has joined #tripleo07:42
*** derekjhyang has quit IRC07:44
*** jpena|off is now known as jpena07:48
*** quique|rover|afk is now known as quiquell|rover07:49
*** yolanda_ has quit IRC07:49
*** hamzy has quit IRC07:51
*** hamzy has joined #tripleo07:51
*** pblaho has joined #tripleo07:57
*** jpich has joined #tripleo07:58
openstackgerritMehdi Abaakouk (sileht) proposed openstack/tripleo-heat-templates stable/queens: Telemetry services need to access to Storage  https://review.openstack.org/56371508:00
openstackgerritRedHat RDO CI proposed openstack/tripleo-heat-templates stable/queens: GATE CHECK for TripleO  https://review.openstack.org/56045208:00
openstackgerritMehdi Abaakouk (sileht) proposed openstack/tripleo-heat-templates stable/pike: Telemetry services need to access to Storage  https://review.openstack.org/56371608:00
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates master: [DellEMC]Update Manila VNX driver  https://review.openstack.org/53814708:02
*** yolanda_ has joined #tripleo08:02
*** yolanda_ is now known as yolanda08:02
*** ykarel is now known as ykarel|lunch08:03
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates master: Intoduce ceph upgrade environments  https://review.openstack.org/56198008:07
openstackgerritPranali Deore proposed openstack/tripleo-heat-templates master: Disable interoperable default image import method in THT  https://review.openstack.org/56293008:07
*** ooolpbot has joined #tripleo08:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION08:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/176619508:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/176640608:10
*** ooolpbot has quit IRC08:10
openstackLaunchpad bug 1766195 in tripleo "Error installing fluent-plugin-kubernetes_metadata_filter at pike" [Critical,Triaged] - Assigned to Quique Llorente (quiquell)08:10
openstackLaunchpad bug 1766406 in tripleo "multinode-oooq-container-updates fails on master" [Critical,Triaged]08:10
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates master: Intoduce Ceph upgrade environments  https://review.openstack.org/56198008:11
*** tosky has joined #tripleo08:13
*** dsariel has joined #tripleo08:20
shardyTengu: Hi, good morning - re your stalled deployment, have you tried running the deploy steps via config download instead?08:21
*** ssbarnea_ has joined #tripleo08:21
Tengushardy: hello! i.e via the os-config-foo script/daemon? yep.08:22
shardyTengu: Also what is the output from "openstack stack failures list" when you run it via heat?08:22
openstackgerritGiulio Fidente proposed openstack/python-tripleoclient master: Introduce Ceph upgrade command  https://review.openstack.org/56042308:23
Tengushardy: a wonderful "status_reason:| UPDATE aborted"08:23
shardyTengu: No I mean either add the --config-download flag to the deploy, or do it manually e.g openstack overcloud config download; cd <tmpdir>; ansible-playbook -vv -i /usr/bin/tripleo-ansible-inventory deploy_steps_playbook.yaml08:23
shardyTengu: sounds like it timed out perhaps?08:24
Tengushardy: indeed, it actually does timeout. but I just don't know why08:24
shardyTengu: https://hardysteven.blogspot.co.uk/2018/02/debugging-tripleo-revisited-heat.html08:25
shardythat may help - if it's e.g puppet hanging you should see the same when driving it via ansible or directly on the nodes08:25
Tengushardy: stumbled on your post yep. but… in fact, the os-apply-config.log just shows success, in addition.08:25
shardyif that works, it may be a networking problem such that some node can't signal back - I'd check the journalctl -u os-collect-config on all the nodes08:25
Tengushardy: it's like some host just doesn't report back08:25
shardyTengu: Ok that should be visible in the logs08:26
Tenguhmm08:26
shardyTengu: also you can check to see which one, sec08:26
Tengunew run on-going, will check  that.08:26
Tengu(started just before you poped up :))08:26
shardyTengu: yeah sorry starting a bit late today :)08:26
Tenguno problem ;)08:26
TenguI'm starting awfully early on my own08:27
bogdandoPTAL https://review.openstack.org/#/c/562566/08:27
bogdandohttps://review.openstack.org/#/c/556516/08:27
Tengushardy: I also read back our discussions in February in order to debug a bit, but apparently it's NOT related to a software deployment. the stack resources just consist in a huge blob with all the variables08:28
*** jistr is now known as jistr|mtgs08:28
TenguI think it should be generating something on the nodes, and call it later.08:29
shardyTengu: If you think it's one node not signalling leading to a stalled SoftwareDeployment you can see which one e.g http://paste.openstack.org/show/719800/08:30
Tengushardy: will run this one now in order to get an idea of the current output. thanks !08:31
Tenguhmm. | AllNodesDeploySteps                       | 754393c3-a37f-47d0-9389-db8ff99c4bd2                      | OS::TripleO::PostDeploySteps                     | UPDATE_FAILED      | 2018-04-24T03:31:31Z |08:31
*** lucas-afk is now known as lucasagomes08:31
*** moguimar has joined #tripleo08:32
Tenguok, digging.08:33
Tenguhmmm. weird.08:33
Tenguapparently… can't list resources in the failed stack08:33
Tengushardy: I get WorkflowTasks_Step2_Execution   | 29faf594-10af-4858-ac33-ed210684f452                                      | OS::Mistral::ExternalResource                                                                           | UPDATE_FAILED08:34
Tengushardy: meaning I should be able to list the resources in this stack, right ?08:34
Tengushardy: if so, fun part: Stack not found: 29faf594-10af-4858-ac33-ed210684f45208:34
openstackgerritMerged openstack/tripleo-common master: Remove ceph-ansible workflow.  https://review.openstack.org/56304708:34
openstackgerritMerged openstack/tripleo-heat-templates master: Modify libvirt port range for live-migration  https://review.openstack.org/56178408:34
shardyTengu: OS::Mistral::ExternalResource isn't a nested stack like the example I pasted, it's a heat resource plugin that runs mistral workflows08:37
shardyso you may have to check the mistral logs to see why it failed, or check the openstack workflow execution show 29faf594-10af-4858-ac33-ed210684f45208:38
Tengushardy: ok, well… that's the one that fail in fact :/08:38
*** mdnadeem has joined #tripleo08:39
Tenguah, thanks for the workflow.08:39
shardyTengu: Ok, so you found what's failing - does this have Ceph enabled?  Only Ceph and octavia use OS::Mistral::ExternalResource IIRC08:39
Tenguceph, yep.08:39
Tenguwith ceph-ansible, in containers.08:39
shardynormally for mistral issues I end up looking at the mistral logs tbh08:39
Tenguok :]08:39
shardyOk so sounds like that's what's failing08:39
Tenguwhich one?08:39
Tengu(which mistral logs?)08:40
shardygfidente: ^^ is there a log of the ceph-ansible run somewhere ?08:40
*** ykarel|lunch is now known as ykarel08:40
shardyTengu: check /var/log/mistral/executor.log and engine.log08:40
Tengushardy: there's ceph-install-workflow.log08:40
shardyAlright then, that sounds promising ;)08:40
gfidenteTengu yeah08:40
gfidentedid this happen on stack update, after upgrading form 11?08:41
*** gkadam has joined #tripleo08:41
Tenguceph will really kill me…08:41
gfidenteor better ... replacing the non containerized 12 deployment into a containerized deployment?08:41
Tengugfidente: happening now with a "simple" deploy in fact.08:41
*** derekh has joined #tripleo08:41
gfidentelike from scratch/08:41
gfidente?08:41
Tengugfidente: nope, from my migrated env (bm -> containers :D)08:41
openstackgerritMartin Mágr proposed openstack/tripleo-common master: Add Ironic inspector health checks  https://review.openstack.org/56172508:41
openstackgerritMerged openstack/puppet-tripleo master: Fluentd monitoring interface  https://review.openstack.org/56163508:42
openstackgerritMerged openstack/tripleo-common master: Add fluentd health check  https://review.openstack.org/56163808:42
Tengugfidente: and nope, it's not the chcon running on 1.5To of shards ;). already checked that one08:42
gfidenteTengu yeah the env you migrated08:42
Tengugfidente: precisely08:42
gfidenteso on your stack update08:42
gfidentemake sure to pass08:42
gfidenteCephAnsiblePlaybook: /usr/share/ceph-ansible/site-docker.yml.sample08:43
gfidentebecause I think we messed with that parameter to make switch the running cluster08:43
Tenguerffff08:43
gfidenteand it might be sticking to its last value08:43
Tenguseriously?08:43
gfidenteyes08:43
Tenguseeing http://paste.openstack.org/show/719801/ is nice.08:43
Tenguok, so I stop my current run and will add the var.08:43
Tengugfidente: hmm, cluster's already switched to containers using /usr/share/ceph-ansible/infrastructure-playbooks/switch-from-non-containerized-to-containerized-ceph-daemons.yml08:44
gfidenteTengu exactly, that's why08:45
gfidenteI think it's running the switch playbook again08:45
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Bail on common devuser usernames  https://review.openstack.org/56386008:45
gfidentewhen you don't want to08:45
Tengugfidente: ah. ok :D08:45
Tenguso I understood correctly.08:45
Tengudarn.08:45
openstackgerritGiulio Fidente proposed openstack/python-tripleoclient master: Introduce Ceph upgrade command  https://review.openstack.org/56042308:45
openstackgerritGiulio Fidente proposed openstack/python-tripleoclient master: Introduce Ceph upgrade command  https://review.openstack.org/56042308:46
*** radeks has joined #tripleo08:48
*** yolanda has quit IRC08:48
openstackgerritGiulio Fidente proposed openstack/python-tripleoclient master: Introduce Ceph upgrade command  https://review.openstack.org/56042308:49
Tengushardy: is there a way to cancel a running update?08:50
Tengushardy: like, "stop it now and don't mess further, don't rollback, just STOP"08:51
Tenguhmm. ah. remembering stopping mistral is a way to do that.08:51
d0ugalI feel like there should be a better way to do that :)08:53
Tenguso do I08:53
Tenguthere's a cancel.08:53
shardyYeah openstack stack cancel update08:54
*** ianw is now known as ianw_pto08:54
Tengu:)08:54
d0ugalgood point08:54
d0ugalalso, stopping mistral wouldn't cancel the stack update afaik08:55
shardyprobably you also want --no-rollback08:55
shardyanother way is to restart the heat-engine, but that isn't very clean08:55
*** moguimar has quit IRC08:57
openstackgerritCarlos Goncalves proposed openstack/tripleo-quickstart master: Add fs038 for Octavia scenario testing  https://review.openstack.org/53116808:57
openstackgerritGiulio Fidente proposed openstack/python-tripleoclient master: Introduce Ceph upgrade command  https://review.openstack.org/56042308:58
Tengu:P08:59
*** yolanda has joined #tripleo09:01
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates master: Introduce Ceph upgrade environments  https://review.openstack.org/56198009:01
*** shreshtha has joined #tripleo09:03
*** shreshtha_ has quit IRC09:05
openstackgerritJose Luis Franco proposed openstack/tripleo-common stable/queens: Remove version section in overcloud_keystone_url.  https://review.openstack.org/56359809:05
*** akrivoka has joined #tripleo09:05
openstackgerritCarlos Goncalves proposed openstack/tripleo-heat-templates master: Octavia amphora image handling updates  https://review.openstack.org/56208609:07
openstackgerritCarlos Goncalves proposed openstack/tripleo-heat-templates master: Add scenario010 for testing Octavia  https://review.openstack.org/51833109:07
*** gfidente has quit IRC09:09
*** dtantsur|afk is now known as dtantsur09:09
*** ooolpbot has joined #tripleo09:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION09:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/176619509:10
openstackLaunchpad bug 1766195 in tripleo "Error installing fluent-plugin-kubernetes_metadata_filter at pike" [Critical,Triaged] - Assigned to Quique Llorente (quiquell)09:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/176640609:10
*** ooolpbot has quit IRC09:10
openstackLaunchpad bug 1766406 in tripleo "multinode-oooq-container-updates fails on master" [Critical,Triaged]09:10
openstackgerritCarlos Goncalves proposed openstack/tripleo-heat-templates master: Octavia amphora image handling updates  https://review.openstack.org/56208609:10
*** anande has quit IRC09:12
openstackgerritCarlos Goncalves proposed openstack/tripleo-heat-templates master: Add scenario010 for testing Octavia  https://review.openstack.org/51833109:15
*** jaosorior has quit IRC09:18
openstackgerritCarlos Camacho proposed openstack-infra/tripleo-ci master: Source overcloudrc.v3 instead of overcloudrc  https://review.openstack.org/56386909:21
*** panda|bbl is now known as panda|ruck09:25
*** gfidente has joined #tripleo09:27
*** gfidente has quit IRC09:27
*** gfidente has joined #tripleo09:27
*** wolverineav has joined #tripleo09:29
openstackgerritCarlos Camacho proposed openstack/tripleo-quickstart-extras master: Source overcloudrc.v3 instead of overcloudrc  https://review.openstack.org/56387109:31
Tenguhumpf… don't to be pessimistic…09:36
Tengubut I think it will timeout, once again -.-09:36
openstackgerritGiulio Fidente proposed openstack/python-tripleoclient master: Introduce Ceph upgrade command  https://review.openstack.org/56042309:36
radekshow about: heat stack-cancel-update <NAME or ID>09:38
Tenguradeks: for me? heat command is deprecated, replaced by the unified "openstack" one :)09:40
radeksTengu: yeahhh I can't find corresponding "openstack" command :)09:40
Tenguopenstack stack cancel <stack-name> :)09:40
Tenguor something like that. anyway. that won't solve my current issue :/.09:41
radekshad the same kind of problems and I used to wait until stack update fails and run it again, not very smart but it worked09:42
Tenguah, yep. already done that. multiple time. 4 hours timeout. you get crazy ;)09:42
*** jaosorior has joined #tripleo09:44
openstackgerritCarlos Camacho proposed openstack/tripleo-quickstart-extras master: Source overcloudrc.v3 instead of overcloudrc  https://review.openstack.org/56387109:47
*** wolverineav has quit IRC09:49
*** wolverineav has joined #tripleo09:49
*** moguimar has joined #tripleo09:49
openstackgerritCarlos Camacho proposed openstack-infra/tripleo-ci master: Source overcloudrc.v3 instead of overcloudrc  https://review.openstack.org/56386909:51
*** wolverineav has quit IRC09:54
derekhhey gfidente, you were asking yesterday about how to recheck just rdo jobs, did check-rdo work?09:55
gfidentederekh check-rdo it did yes09:56
derekhgfidente: ack, thanks09:56
openstackgerritMartin Mágr proposed openstack/tripleo-common master: Add health checks for Swift services  https://review.openstack.org/55973709:58
*** leitan has joined #tripleo10:01
*** yprokule has quit IRC10:03
*** leitan has quit IRC10:06
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: Remove non used tripleo_root variable  https://review.openstack.org/55250710:07
*** dbecker has joined #tripleo10:07
openstackgerritMerged openstack/tripleo-docs master: fix typos in documentation  https://review.openstack.org/55553210:09
*** ooolpbot has joined #tripleo10:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION10:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/176619510:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/176640610:10
*** ooolpbot has quit IRC10:10
openstackLaunchpad bug 1766195 in tripleo "Error installing fluent-plugin-kubernetes_metadata_filter at pike" [Critical,Triaged] - Assigned to Quique Llorente (quiquell)10:10
openstackLaunchpad bug 1766406 in tripleo "multinode-oooq-container-updates fails on master" [Critical,Triaged]10:10
openstackgerritMartin Mágr proposed openstack/tripleo-common master: [WIP] Activate remaining set of healthchecks  https://review.openstack.org/56387410:11
*** pmannidi has joined #tripleo10:12
openstackgerritSagi Shnaidman proposed openstack/tripleo-quickstart-extras master: Install collectd on undercloud  https://review.openstack.org/56004810:13
openstackgerritSagi Shnaidman proposed openstack/tripleo-quickstart-extras master: Install collectd on undercloud  https://review.openstack.org/56004810:13
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates master: Merge values for same key from multiple services  https://review.openstack.org/56311210:13
*** tvignaud has quit IRC10:14
*** quiquell|rover has quit IRC10:14
*** gfidente has quit IRC10:14
*** quiquell has joined #tripleo10:15
*** zoli is now known as zoli|lunch10:15
ykarelmatbu, ccamacho hi10:18
quiquellHello, adding timestamps at upgrades10:18
*** wolverineav has joined #tripleo10:18
quiquellhttps://review.openstack.org/#/c/561911/10:19
quiquellCan I have some checks there ?10:19
ykarelmatbu, ccamacho can https://review.openstack.org/#/c/562995/ cause https://bugs.launchpad.net/tripleo/+bug/176640610:19
openstackLaunchpad bug 1766406 in tripleo "multinode-oooq-container-updates fails on master" [Critical,Triaged]10:19
ykareli saw the job failed on the patch itself10:20
*** quiquell is now known as quiquell|rover10:20
ykarelEmilienM, ^^10:26
openstackgerritMartin Mágr proposed openstack/tripleo-common master: Add health checks for Swift services  https://review.openstack.org/55973710:26
*** eck` is now known as eck`gone10:27
openstackgerritMartin Mágr proposed openstack/tripleo-heat-templates master: Enable health check scripts for Swift background services  https://review.openstack.org/56387810:31
openstackgerritChandan Kumar proposed openstack/tripleo-quickstart master: Added featureset060 for installing containerized UC  https://review.openstack.org/56387910:33
chandankumarsshnaidm: EmilienM ^^10:33
openstackgerritRadomir Dopieralski proposed openstack/tripleo-heat-templates master: Enable heat-ui plugin for horizon by default  https://review.openstack.org/56388010:34
chandankumarsshnaidm: Any more place i need to add a patch to test f060?10:36
*** dbecker has quit IRC10:36
leanderthalOn 3-4 May NEXT WEEK we're testing the first milestone for Rocky. Join us on #RDO to help. See you NEXT WEEK! https://www.rdoproject.org/testday/rocky/milestone1/10:36
openstackgerritMartin Mágr proposed openstack/tripleo-common master: [WIP] Activate remaining set of healthchecks  https://review.openstack.org/56387410:37
openstackgerritMartin Mágr proposed openstack/tripleo-heat-templates master: Enable fluentd health check  https://review.openstack.org/56388110:40
sshnaidmchandankumar, you need to run this job in different patch, to see if it works10:41
*** jaosorior has quit IRC10:43
*** moguimar has quit IRC10:45
*** dbecker has joined #tripleo10:45
openstackgerrityatin proposed openstack/tripleo-common master: DNM: Try 000 update  https://review.openstack.org/56388410:46
*** pmannidi has quit IRC10:47
*** moguimar has joined #tripleo10:47
weshayEmilienM, mwhahaha this looks suspicious https://review.openstack.org/#/c/562995/ https://review.openstack.org/#/c/563000/10:47
matbuykarel: looking10:48
weshaymatbu, ping re: https://bugs.launchpad.net/tripleo/+bug/176640610:48
openstackLaunchpad bug 1766406 in tripleo "multinode-oooq-container-updates fails on master" [Critical,Triaged]10:48
matbuweshay: only one merged, no?10:49
weshaymatbu, ya.. wondering if you think it broke that updates10:49
*** ssbarnea_ has quit IRC10:50
matbuweshay: hm possible10:51
matbuim looking at the log10:51
openstackgerritwaleed mousa proposed openstack/tripleo-heat-templates master: Removing the deprecated environment files for hw offloading in containers  https://review.openstack.org/55972310:51
weshaymatbu, thanks10:51
matbuweshay: but yes, looking at this log: http://logs.openstack.org/57/561657/2/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/7cda328/logs/undercloud/home/zuul/overcloud_update_run_Controller.log.txt.gz10:52
weshayya..10:53
weshaymatbu, I think we should revert it10:53
matbuweshay: and it seems that the failure has occured here, and without https://review.openstack.org/#/c/563000/, the failure is normal10:53
*** moguimar has quit IRC10:53
*** mdnadeem has quit IRC10:53
matbuweshay: well no, we need the cli patch10:53
*** mdnadeem has joined #tripleo10:53
matbuto land10:53
weshayya.. works with your second patch10:54
weshayhttps://review.openstack.org/#/c/563000/10:54
matbuboth patches is working together, thats the chikken egg issue with ooo-common and cli10:54
matbuand the depends-on doesn't work with those 2 repos10:54
weshaymatbu, hrm10:55
weshaymatbu, review https://review.openstack.org/#/c/563806/10:55
weshaythe patch broke the gate.. so let's get your follow up merged and then revert EmilienM's patch https://review.openstack.org/#/c/563806/10:56
matbuweshay: ok10:56
matbuweshay: but happy to see the upgrade code running in the gate !! :)10:57
matbuweshay: we should think about on how to work both with ooo-common workbook and tripleoclient code10:57
weshayhttps://review.openstack.org/#/q/topic:gate_update+(status:open+OR+status:merged)10:57
weshaymatbu, ya.. we need the same job running on those repos10:57
matbubecause its really classic to fix a bug in both repo10:57
weshaymatbu, so we need to add a review for python-tripleoclient10:58
weshaymatbu, and I think the update job should be voting on those as well10:58
weshayor elese we're going to rehit this10:58
matbuweshay: yeah right10:58
matbuweshay:  and even if it tries to update 'current' to 'current'10:59
weshaymatbu, ya.. I should draw up these steps10:59
weshaymatbu, https://review.rdoproject.org/r/#/c/13476/11:00
matbuweshay: awesome11:01
*** adarazs is now known as adarazs_lunch11:01
weshaymatbu, ya. man.. so let's get this knocked out and really covering for us asap11:02
matbuweshay: yep im hoping it will merge soon (10h25 in the gate)11:03
weshaymatbu, help me push the through the reviews.. I'm in a meetings this week.. but I'll try to get a python-tripleoclient patch up11:03
matbuweshay: yep thx11:05
*** jcoufal has joined #tripleo11:07
*** maufart__ has joined #tripleo11:09
*** ooolpbot has joined #tripleo11:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION11:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/176619511:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/176640611:10
*** ooolpbot has quit IRC11:10
openstackLaunchpad bug 1766195 in tripleo "Error installing fluent-plugin-kubernetes_metadata_filter at pike" [Critical,Triaged] - Assigned to Quique Llorente (quiquell)11:10
openstackLaunchpad bug 1766406 in tripleo "multinode-oooq-container-updates fails on master" [Critical,Triaged]11:10
*** aufi_ has quit IRC11:12
*** dbecker has quit IRC11:13
*** athomas has quit IRC11:13
*** hjensas has quit IRC11:14
*** dprince has joined #tripleo11:18
*** ssbarnea_ has joined #tripleo11:19
*** yprokule has joined #tripleo11:20
openstackgerritMerged openstack/tripleo-heat-templates master: Start using nova_metadata_host  https://review.openstack.org/53189011:23
openstackgerritMerged openstack/puppet-tripleo stable/queens: Split up neutron-lbaas service plugin and agent  https://review.openstack.org/56321411:23
openstackgerritMerged openstack/tripleo-heat-templates stable/queens: Add the service_config_settings from {{service-name}} base  https://review.openstack.org/56355611:23
*** ykarel is now known as ykarel|mtg11:34
*** zoli|lunch is now known as zoli11:35
*** alee has joined #tripleo11:37
openstackgerritChristian Schwede proposed openstack/tripleo-heat-templates stable/pike: Create /var/log/containers on Swift storage-only nodes  https://review.openstack.org/56389511:38
openstackgerritYurii Prokulevych proposed openstack/tripleo-upgrade stable/queens: Fix newline stripping in upgrade template.  https://review.openstack.org/56389611:38
*** alee__ has quit IRC11:39
*** rh-jelabarre has joined #tripleo11:40
*** ansmith has quit IRC11:41
*** lucasagomes is now known as lucas-hungry11:41
openstackgerritMerged openstack/tripleo-heat-templates stable/queens: Adding missing OVN Metadata service to DVR environments  https://review.openstack.org/56184811:42
openstackgerritMerged openstack/tripleo-heat-templates master: Default collectd to overcloud gnocchi  https://review.openstack.org/56237211:44
openstackgerritMerged openstack/tripleo-heat-templates stable/pike: Reduce verbosity during mysql bootstrap  https://review.openstack.org/56297911:44
openstackgerritMerged openstack/tripleo-common stable/queens: Use ansible_queue_name only for upgrade/update  https://review.openstack.org/56370311:44
openstackgerritMerged openstack/tripleo-heat-templates stable/ocata: Resource name correction  https://review.openstack.org/55988511:44
openstackgerritMerged openstack/tripleo-heat-templates stable/ocata: Enable Neutron LBaaS Integration  https://review.openstack.org/56092111:44
openstackgerritMerged openstack/tripleo-heat-templates stable/pike: Resource name correction  https://review.openstack.org/55988411:44
openstackgerritMerged openstack/tripleo-heat-templates stable/newton: Enable Neutron LBaaS Integration  https://review.openstack.org/55674311:44
openstackgerritMerged openstack-infra/tripleo-ci master: Stop using slave_scripts/install-distro-packages.sh  https://review.openstack.org/56324411:44
openstackgerritMerged openstack-infra/tripleo-ci master: Add bindep.txt file  https://review.openstack.org/56324511:44
openstackgerritMerged openstack/instack-undercloud master: Masqeurading: NOT persist ephemeral firewall rules  https://review.openstack.org/56346711:44
openstackgerritMerged openstack/python-tripleoclient master: Use only update/upgrade ansible queue instead  https://review.openstack.org/56300011:44
openstackgerritMerged openstack/instack-undercloud stable/queens: Masquerading, do not persist ephemeral firewall rules  https://review.openstack.org/56358111:44
openstackgerritMerged openstack-infra/tripleo-ci master: Disable voting on tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates  https://review.openstack.org/56380611:44
openstackgerritCarlos Camacho proposed openstack/python-tripleoclient stable/queens: Use only update/upgrade ansible queue instead  https://review.openstack.org/56390511:44
*** athomas has joined #tripleo11:45
*** moguimar has joined #tripleo11:46
openstackgerritOliver Walsh proposed openstack/tripleo-heat-templates stable/newton: Explicitly set nova/neutron/ceilometer host to expected fqdn  https://review.openstack.org/56365011:47
*** jaosorior has joined #tripleo11:48
beaglesshardy: any thoughts on why I might be getting a "permission denied" here: http://logs.openstack.org/23/550823/11/check/tripleo-ci-centos-7-3nodes-multinode/4cabca4/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-04-23_22_42_2811:49
matbuykarel|mtg: weshay EmilienM https://review.openstack.org/563000  is merged on master so the update bug can work again now11:49
*** jpena is now known as jpena|lunch11:50
matbumwhahaha: ^11:50
matbumwhahaha: ^11:50
ykarel|mtgmatbu, nice, but we are not consistent yet, need to fix: https://review.rdoproject.org/r/#/q/topic:rdo-FTBFS+status:open11:50
matbuoops sorry11:50
ykarel|mtgbefore jobs start passing11:50
beaglesshardy: doesn't do it in non-config-download deployments afaik - haven't updated my undercloud in a few weeks11:51
ykarel|mtgmatbu, Okk, no problem for upstream jobs11:51
ykarel|mtgignore my comment11:51
ykarel|mtgas the project is tripleoclient11:51
openstackgerritOliver Walsh proposed openstack/tripleo-heat-templates stable/newton: Explicitly set nova/neutron/ceilometer host to expected fqdn  https://review.openstack.org/56365011:51
matbuykarel|mtg: k no pb, thx anyway :)11:52
ykarel|mtgmatbu, btw i pushed https://review.openstack.org/#/c/563884/ just to check the impact11:52
*** abishop has joined #tripleo11:52
ccamachoykarel|mtg also, this passed correctly al lthe jobs https://review.openstack.org/#/c/563000/11:55
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates stable/queens: Default collectd to overcloud gnocchi  https://review.openstack.org/56390811:56
*** adarazs_lunch is now known as adarazs11:57
*** quiquell|rover is now known as quiquell|rover|l11:57
*** thrash|g0ne is now known as thrash11:57
*** quiquell|rover|l is now known as quique|rover|lch11:57
ykarel|mtgccamacho, yes both tripleo-common and tripleoclient patch together works nicely11:58
ccamachoawesome11:58
matbuykarel|mtg: ack great thx12:00
*** lblanchard has joined #tripleo12:01
* beagles trying in an updated environment12:02
shardybeagles: Hey sorry not seen that before - I'd be looking at the task that runs paunch to see if there's a missing become: true somewhere?12:03
shardythat said, I've tested this fairly recently and IIRC it worked12:03
beaglesshardy: my first thought is that it's a bit of an oddball as we are running docker_puppet_apply directly in a docker_config step, not the puppet_config. There are a few other services that do this (e.g. haproxy) and afaik they are still fine12:05
*** tvignaud has joined #tripleo12:06
beagles(in addition to the normal puppet_config stage that is)12:06
*** dxiri has joined #tripleo12:07
beaglesshardy: I'll see if I can reproduce in a local environment where it'll be a bit easier to see what's going on with that container12:07
*** raildo has joined #tripleo12:08
paramiteGreetings guys, can we please get this backport in: https://review.openstack.org/#/c/563908 ?12:09
*** leitan has joined #tripleo12:09
*** ooolpbot has joined #tripleo12:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION12:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/176619512:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/176640612:10
*** ooolpbot has quit IRC12:10
openstackLaunchpad bug 1766195 in tripleo "Error installing fluent-plugin-kubernetes_metadata_filter at pike" [Critical,Triaged] - Assigned to Quique Llorente (quiquell)12:10
openstackLaunchpad bug 1766406 in tripleo "multinode-oooq-container-updates fails on master" [Critical,Triaged]12:10
*** wolverineav has quit IRC12:12
shardybeagles: ack sounds good, sorry I couldn't be more help!12:12
*** wolverineav has joined #tripleo12:12
beaglesshardy: np. just wondering if it was something obvious :)12:13
shardyjaosorior: Hey I upgraded certmonger and moved undercloud-front.crt, but re-running openstack undercloud install didn't regenerate it12:13
shardyjaosorior: anything else I need to do to make that happen, or can I do it manually?12:13
* beagles has questions for everybody today actually 12:13
*** wolverineav has quit IRC12:13
jaosoriorshardy: it didn't? can you report a bug? (it should re-generate it)12:13
jaosoriorshardy: you can do sudo getcert list12:14
*** wolverineav has joined #tripleo12:14
jaosoriorshardy: that'll get the ID of the certificate12:14
jaosoriorthen12:14
jaosoriorsudo getcert stop-tracking -I <ID>12:14
jaosoriorthat'll stop tracking the cert12:14
beaglesjaosorior: do you remember having a dialog around the apachification of neutron-api. I vaguely recall it being a deliberate decision to not run it under apache but can't find any notes on it and don't remember why we did that12:14
jaosoriorthen you can do openstack undercloud install12:14
*** liverpooler has joined #tripleo12:14
jaosoriorbeagles: no idea, it would have been nice to have it under apache12:14
beaglesjaosorior: my guess it had to do with the fact that neutron-server is both an API and RPC (for agents) server12:15
openstackgerritMarios Andreou proposed openstack/python-tripleoclient master: Q upgrade - fixup completion messages and --help doctext  https://review.openstack.org/56391212:15
jaosoriorcould be12:15
beaglesdamn12:15
*** jaosorior has quit IRC12:15
beaglesI think that was it, but I wish I had made a note of it in the yaml12:15
*** jaosorior has joined #tripleo12:16
beaglesI'll revisit when done with blockers12:16
jaosorioralrighty12:16
shardyjaosorior: thanks, trying again12:17
openstackgerritEric Young proposed openstack/tripleo-heat-templates master: Containerized control plane with ScaleIO storage  https://review.openstack.org/56391412:17
*** wolverineav has quit IRC12:18
openstackgerritMarios Andreou proposed openstack/python-tripleoclient stable/queens: Q upgrade - fixup completion messages and --help doctext  https://review.openstack.org/56391612:18
openstackgerritArx Cruz proposed openstack/tripleo-quickstart-extras master: Install stestr only when release is pike or newer  https://review.openstack.org/56384312:19
*** Dekay has quit IRC12:22
*** amoralej is now known as amoralej|lunch12:22
*** quique|rover|lch is now known as quiquell|rover12:25
*** mwhahaha has quit IRC12:26
*** mwhahaha has joined #tripleo12:27
*** ChanServ sets mode: +v mwhahaha12:27
shardyjaosorior: Ok that did it, will raise a bug and comment on the bz about the steps taken12:27
*** lucas-hungry is now known as lucasagomes12:29
*** tzumainn has joined #tripleo12:30
*** alee has quit IRC12:31
jaosoriorshardy: thanks!12:31
jaosoriorsorry for the bugs dude :/12:31
*** gfidente has joined #tripleo12:33
*** gfidente has quit IRC12:33
*** gfidente has joined #tripleo12:33
shardyjaosorior: np - I now get a CN and SAN which matches the dns name in the cert, but I still get browser warnings after copying /etc/pki/ca-trust/source/anchors/cm-local-ca.pem to my local /etc/pki/ca-trust/source/anchors/ and running update-ca-trust extract12:33
shardyare there additional steps I need to make the browser trust the cert?12:33
*** jaganathan has quit IRC12:34
*** rlandy has joined #tripleo12:35
*** trown|outtypewww is now known as trown12:36
*** portdirect has quit IRC12:36
*** portdirect has joined #tripleo12:37
*** psachin has quit IRC12:37
*** panda|ruck is now known as panda|ruck|lunch12:37
shardyjaosorior: Hmm chrome says SAN is missing again, but the openssl x509 output shows it12:37
*** rbrady has joined #tripleo12:37
shardydo any other certs also need recreating?12:37
*** rbrady is now known as Guest3118912:38
*** Goneri has joined #tripleo12:39
*** tvignaud has quit IRC12:40
openstackgerritPranali Deore proposed openstack/tripleo-heat-templates master: Disable interoperable default image import method in THT  https://review.openstack.org/56293012:41
*** ykarel|mtg is now known as ykarel12:45
*** tvignaud has joined #tripleo12:50
openstackgerritPranali Deore proposed openstack/tripleo-heat-templates master: Disable interoperable default image import method in THT  https://review.openstack.org/56293012:50
*** tvignaud has quit IRC12:51
*** tvignaud has joined #tripleo12:51
*** jpena|lunch is now known as jpena12:54
*** shreshtha_ has joined #tripleo12:56
*** wolverineav has joined #tripleo12:56
openstackgerritDan Prince proposed openstack/puppet-tripleo master: Only run os-net-config if file has content  https://review.openstack.org/56393512:57
*** hjensas has joined #tripleo12:57
*** hjensas has quit IRC12:57
*** hjensas has joined #tripleo12:57
*** shreshtha has quit IRC12:57
*** pchavva has joined #tripleo12:58
*** ansmith has joined #tripleo12:58
fultonjany cores want to give a quick reivew of a one-liner https://review.openstack.org/#/c/563632 which fixes a bug ?12:59
openstackgerritDan Prince proposed openstack/instack-undercloud master: Fix net_config_override validations  https://review.openstack.org/56393713:00
*** hjensas has quit IRC13:01
bogdandodprince: hi, thanks for the patches!13:01
dprincebogdando: hey! np13:02
openstackgerritwes hayutin proposed openstack-infra/tripleo-ci master: DNM TESTING: Revert "Disable voting on tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates"  https://review.openstack.org/56393813:02
bogdandofyi, I created also https://bugs.launchpad.net/tripleo/+bug/1763337 and https://bugs.launchpad.net/tripleo/+bug/176657913:02
openstackLaunchpad bug 1763337 in tripleo "Wire in net_config_override in undercloud.conf for containerized undercloud heat installer" [Medium,In progress] - Assigned to Bogdan Dobrelya (bogdando)13:02
openstackLaunchpad bug 1766579 in tripleo "Wire in hieradata_override in undercloud.conf for containerized undercloud heat installer" [Medium,In progress] - Assigned to Bogdan Dobrelya (bogdando)13:02
bogdandothey seem extremely important for upgrades of customized instack deploys IIUC13:02
bogdandodeployments13:02
ccamachoEmilienM quick question, I have some errors using the keystone endpoint, I have 2 simple fixes, do you mind to check them when having some time? https://review.openstack.org/#/q/topic:use-overcloudrc.v313:03
ccamachoor anyone available please13:03
ccamachothanks!13:03
openstackgerritBrad P. Crochet proposed openstack/tripleo-common master: Add workflow to generate fencing parameters  https://review.openstack.org/54828313:03
bogdandodprince: also added you to https://trello.com/c/7GaMhrhe/81-wire-in-netconfigoverride-from-instack-undercloudconf-feature-parity , the work seems related13:03
dprincebogdando: I'll push the instack-undercloud patch to tripleoclient now...13:04
dprincebogdando: I think that is all that would be required here13:04
bogdandothanks!13:04
bogdandodprince: wrt processing, we need to wire-in it first13:05
bogdandoif you want to catch it up, let me know13:05
bogdandoI mean https://review.openstack.org/#/c/563610/13:05
openstackgerritFlorian Fuchs proposed openstack/tripleo-validations stable/queens: Update ceilometerdb-size to support gnocchi and panko  https://review.openstack.org/56394113:06
openstackgerritBrad P. Crochet proposed openstack/tripleo-validations master: Add validation for checking node counts  https://review.openstack.org/56315213:06
bogdandoI think undercloud_config should just pass it in for undercloud_deploy via --net-config-override, and then process it with some instack-ish context known for the client. WDYT, dprince?13:06
openstackgerritFlorian Fuchs proposed openstack/tripleo-validations stable/pike: Update ceilometerdb-size to support gnocchi and panko  https://review.openstack.org/56394213:06
*** cdearborn has joined #tripleo13:07
bogdandolike translating those http://git.openstack.org/cgit/openstack/instack-undercloud/tree/templates/net-config.json.template into something meaningfull13:07
bogdandodprince: like PARAMETER_MAPPING, prolly13:08
*** jjung has quit IRC13:08
*** jjung has joined #tripleo13:09
bogdandoe.g. UNDERCLOUD_NAMESERVERS templated from  DnsServers joined by undercloud_nameservers. Does it make sense to you? wanna catch it up?13:10
bogdandodprince: ^^13:10
*** ooolpbot has joined #tripleo13:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION13:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/176619513:10
openstackLaunchpad bug 1766195 in tripleo "Error installing fluent-plugin-kubernetes_metadata_filter at pike" [Critical,Fix committed] - Assigned to Quique Llorente (quiquell)13:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/176640613:10
*** ooolpbot has quit IRC13:10
openstackLaunchpad bug 1766406 in tripleo "multinode-oooq-container-updates fails on master" [Critical,Triaged] - Assigned to Carlos Camacho (ccamacho)13:10
*** etingof has quit IRC13:12
*** mcornea has joined #tripleo13:12
bogdandodprince: If you do, I'll switch back to https://trello.com/c/LylrfYDs/83-wire-in-hieradataoverride and  https://trello.com/c/nCDvg0DO/69-document-and-wire-in-undercloudcustomenvfiles-for-the-undercloud-prep-containers et al :)13:13
openstackgerritMartin Mágr proposed openstack/tripleo-common master: [WIP] Activate remaining set of healthchecks  https://review.openstack.org/56387413:15
*** jjung has quit IRC13:15
*** aufi_ has joined #tripleo13:17
openstackgerritMartin Mágr proposed openstack/tripleo-heat-templates master: Enable health check for Ironic inspector services  https://review.openstack.org/56395013:20
*** maufart__ has quit IRC13:20
*** atoth has joined #tripleo13:20
*** alee has joined #tripleo13:22
matbuykarel: so updates job passed successfully: https://review.openstack.org/#/c/563884/13:28
matbuykarel: should we revert mwhahaha non-voting review?13:28
ykarelmatbu, yeah it passed and took 2h 30m 42s13:31
*** eck`gone is now known as eck`13:32
matbuykarel: im happy to see this running :)13:32
matbuweshay: ^ updates job fixed by the merged of tripleoclient13:32
ykarelfor revert, you can propose and wait for mwhahaha to approve13:33
matbuykarel: yep13:33
ykarelmatbu, btw in my patch tripleoclient change was not there13:34
*** vpickard_ is now known as vpickard13:34
ykarelhave u seen any recent passes for that job ?13:34
matbuykarel: this commit wasn't :  https://review.openstack.org/563000  ?13:35
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci master: Revert "Disable voting on tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates"  https://review.openstack.org/56395913:35
ykarelmatbu, yes there it passed13:36
ykarelit would be good to see multiple passes13:36
ykarelto get revert going13:37
matbuykarel: we can kick a recheck on your review13:37
ykarelyes, but that has reverted the tripleo-common change13:38
*** rbowen has joined #tripleo13:39
*** pdeore has quit IRC13:39
ykarelmatbu, i can see multiple -updates check jobs are running currently, we can wait for their result13:40
ykarelsome of them might have tripleoclient change13:40
*** amoralej|lunch is now known as amoralej13:40
*** moshele has quit IRC13:41
ykarelalso job in your revert started13:41
matbuack13:42
ccamachoykarel https://review.openstack.org/#/c/563000/ passed all with the change included13:42
ccamachomatbu ^13:42
*** artom_ is now known as artom13:43
*** atoth has quit IRC13:43
*** panda|ruck|lunch is now known as panda|ruck13:43
openstackgerritOliver Walsh proposed openstack/tripleo-heat-templates stable/newton: Explicitly set nova/neutron/ceilometer host to expected fqdn  https://review.openstack.org/56365013:44
EmilienMccamacho: sure13:46
EmilienMmatbu, weshay : +2 on https://review.openstack.org/#/c/563959/13:47
*** zul has quit IRC13:48
*** zul has joined #tripleo13:49
EmilienMbogdando: can you look at https://review.openstack.org/#/c/563317/ please?13:50
*** skramaja has quit IRC13:50
bogdandoEmilienM: looking13:50
*** ratailor has quit IRC13:50
EmilienMbogdando: and https://review.openstack.org/#/c/563769/13:51
bogdandocommented, EmilienM13:53
*** masco has quit IRC13:54
mwhahahaEmilienM: so what's the deal with the lower-constraints job? https://review.openstack.org/#/c/56057813:54
mwhahahafor that change i think lower-constraints needs tripleo-common 9.0.1 or whatever, but do we need to propose that somewhere else?13:54
openstackgerritFlorian Fuchs proposed openstack/tripleo-validations stable/queens: Fix Switch VLANs validation  https://review.openstack.org/56396913:55
EmilienMmwhahaha: I'll check that out13:55
*** pchavva has quit IRC13:55
EmilienMmwhahaha: maybe some jobs are missing13:55
EmilienMmwhahaha: question on https://review.openstack.org/#/c/560578 - where can we see the generated tarball? (just saw the question of bogdando)13:55
*** jaosorior has quit IRC13:55
*** jaosorior has joined #tripleo13:55
mwhahahaEmilienM: we have to add it to the log collection13:56
mwhahahalet me update my ci patch for that13:56
EmilienMmwhahaha: ok yeah13:56
mwhahahawell wait do we want to capture that?13:56
EmilienMto test the feature at least13:56
EmilienMunless the tarball is too big13:57
*** atoth has joined #tripleo13:58
*** sdoran has quit IRC13:58
*** sdoran has joined #tripleo13:58
openstackgerritAlex Schultz proposed openstack-infra/tripleo-ci master: Update undercloud install directories  https://review.openstack.org/56376913:59
mwhahahaEmilienM: -^13:59
EmilienMmwhahaha: ok thx13:59
mwhahahai'll recheck that patch14:00
mwhahahaand we'll see if it collects14:00
openstackgerritOliver Walsh proposed openstack/tripleo-heat-templates master: Set live_migration_inbound_addr for ssh transport  https://review.openstack.org/56281814:00
mwhahaha#startmeeting tripleo14:00
mwhahaha#topic agenda14:00
mwhahaha* Review past action items14:00
mwhahaha* One off agenda items14:00
mwhahaha* Squad status14:00
mwhahaha* Bugs & Blueprints14:00
mwhahaha* Projects releases or stable backports14:00
mwhahaha* Specs14:00
openstackMeeting started Tue Apr 24 14:00:23 2018 UTC and is due to finish in 60 minutes.  The chair is mwhahaha. Information about MeetBot at http://wiki.debian.org/MeetBot.14:00
mwhahaha* open discussion14:00
mwhahahaAnyone can use the #link, #action and #info commands, not just the moderatorǃ14:00
mwhahahaHi everyone! who is around today?14:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.14:00
*** openstack changes topic to " (Meeting topic: tripleo)"14:00
openstackThe meeting name has been set to 'tripleo'14:00
*** openstack changes topic to "agenda (Meeting topic: tripleo)"14:00
slaglehi14:00
myoungo/14:00
panda|rucko/14:00
openstackgerritOliver Walsh proposed openstack/tripleo-heat-templates master: Set live_migration_inbound_addr for ssh transport  https://review.openstack.org/56281814:01
matbuo/14:01
marioso/14:01
EmilienMo/14:01
mwhahahaa massive turnout today14:01
gfidenteo/14:02
thrasho/14:02
jtomaseko/14:03
owalsho/14:03
mwhahahaalright let's go, should be a quick one14:03
mwhahaha#topic review past action items14:03
mwhahahaNone!14:03
*** openstack changes topic to "review past action items (Meeting topic: tripleo)"14:03
mwhahaha#topic one off agenda items14:03
mwhahaha#link https://etherpad.openstack.org/p/tripleo-meeting-items14:03
*** openstack changes topic to "one off agenda items (Meeting topic: tripleo)"14:03
mwhahaha(myoung) CI Community Meeting starts immediately upon this meeting closing @ https://bluejeans.com/7050859455. All are welcome, ask/discuss anything!  Add to agenda (or just join): https://etherpad.openstack.org/p/tripleo-ci-squad-meeting14:03
*** yamahata has joined #tripleo14:03
mwhahahacome bring your CI issues -^14:04
mwhahahaany questions, thoughts, etc?14:04
*** atoth has quit IRC14:04
mwhahahaok moving on14:05
mwhahaha#topic Squad status14:05
mwhahahaci14:05
mwhahaha#link https://etherpad.openstack.org/p/tripleo-ci-squad-meeting14:05
mwhahahaupgrade14:05
mwhahaha#link https://etherpad.openstack.org/p/tripleo-upgrade-squad-status14:05
mwhahahacontainers14:05
mwhahaha#link https://etherpad.openstack.org/p/tripleo-containers-squad-status14:05
*** openstack changes topic to "Squad status (Meeting topic: tripleo)"14:05
mwhahahaconfig-download14:05
mwhahaha#link https://etherpad.openstack.org/p/tripleo-config-download-squad-status14:05
mwhahahaintegration14:05
mwhahaha#link https://etherpad.openstack.org/p/tripleo-integration-squad-status14:05
mwhahahaui/cli14:05
mwhahaha#link https://etherpad.openstack.org/p/tripleo-ui-cli-squad-status14:05
mwhahahavalidations14:05
mwhahaha#link https://etherpad.openstack.org/p/tripleo-validations-squad-status14:05
mwhahahanetworking14:05
mwhahaha#link https://etherpad.openstack.org/p/tripleo-networking-squad-status14:05
mwhahahaworkflows14:05
mwhahaha#link https://etherpad.openstack.org/p/tripleo-workflows-squad-status14:05
mwhahahasecurity14:05
mwhahaha#link https://etherpad.openstack.org/p/tripleo-security-squad14:05
openstackgerritFlorian Fuchs proposed openstack/tripleo-validations master: Fix the events db size validation  https://review.openstack.org/56397314:06
*** jaosorior has quit IRC14:07
slaglei'll highlight from the config-download squad that it's now the default when using tripleoclient14:07
mwhahahayay new defaults14:07
mwhahahaseems to be working pretty well in CI at least14:07
slagleand i (believe) i've switched over all relevant check/gate ci jobs14:07
mwhahahaexcellent job on that switch14:07
EmilienM++14:08
mwhahahado we have an ETA on the UI bits?14:08
mwhahahai know it what in progress last time we chatted about it14:08
slagle"finalizing" :)14:08
slaglejtomasek: this week maybe? anything you need assistance with?14:08
*** paramite has quit IRC14:09
jtomasekmwhahaha: I am updating deployment success components and UI integration should be finished. We are currently missing undeploy worklfows which we'll need to add too14:09
mwhahahasounds good14:09
mwhahahalet us know if there is anything you need help on14:10
*** ooolpbot has joined #tripleo14:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION14:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/176619514:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/176640614:10
*** ooolpbot has quit IRC14:10
openstackLaunchpad bug 1766195 in tripleo "Error installing fluent-plugin-kubernetes_metadata_filter at pike" [Critical,Fix committed] - Assigned to Quique Llorente (quiquell)14:10
openstackLaunchpad bug 1766406 in tripleo "multinode-oooq-container-updates fails on master" [Critical,Fix committed] - Assigned to Carlos Camacho (ccamacho)14:10
jtomasekmwhahaha: ack14:10
mwhahahaalright moving on14:10
mwhahaha#topic bugs & blueprints14:10
mwhahaha#link https://launchpad.net/tripleo/+milestone/rocky-214:10
mwhahahaFor Rocky we currently have 54 (+3) blueprints and about 656 (+3) open Launchpad bugs.  648 rocky-2, 8 rocky-3.  99 (+2) open Storyboard bugs.14:10
mwhahaha#link https://storyboard.openstack.org/#!/project_group/7614:10
*** openstack changes topic to "bugs & blueprints (Meeting topic: tripleo)"14:10
*** yprokule has quit IRC14:10
mwhahahaSo I moved all the rocky-1 bugs and blueprints over to rocky-214:11
mwhahahai sent an email about this, please check that your items are up to date14:11
mwhahaha#link http://lists.openstack.org/pipermail/openstack-dev/2018-April/129628.html14:11
mwhahahaany questions or issues about a specific bug or blueprint?14:11
openstackgerritJose Luis Franco proposed openstack/tripleo-common stable/queens: Remove version section in overcloud_keystone_url.  https://review.openstack.org/56359814:13
mwhahahasounds like nope14:13
mwhahahamoving on14:13
mwhahaha#topic projects releases or stable backports14:13
*** openstack changes topic to "projects releases or stable backports (Meeting topic: tripleo)"14:13
*** moguimar has quit IRC14:14
mwhahaham1 was last week14:14
EmilienMthis week we released queens/pike/ocata/newton14:14
mwhahahacool14:14
*** rajinir has joined #tripleo14:15
mwhahaha#topic specs14:15
mwhahaha#link https://review.openstack.org/#/q/project:openstack/tripleo-specs+status:open14:15
*** openstack changes topic to "specs (Meeting topic: tripleo)"14:15
mwhahahaonce again, i think we're good on specs. if you have any thing you need reviews on or questions, please send a note to the ML14:15
mwhahaha#topic open discussion14:15
*** openstack changes topic to "open discussion (Meeting topic: tripleo)"14:15
mwhahahaany other business?14:15
*** atoth has joined #tripleo14:16
EmilienMsun is shining14:16
slagleit looks like promotion jobs are failing on featureset02114:16
honzajpich: how did your containerized undercloud install go?14:16
slaglei mention it b/c yesterday I noticed that this job (along with 039) does not use containers14:16
*** atoth has quit IRC14:16
slagleand still uses puppet-pacemaker.yaml14:17
*** atoth has joined #tripleo14:17
mwhahahado we have an open bug for those?14:17
EmilienMmhh I thought all jobs were switched to containers14:17
openstackgerritCarlos Camacho proposed openstack/tripleo-common stable/queens: Allow uploading big files to swift (5GB)  https://review.openstack.org/56397714:17
EmilienMpanda|ruck: ^ can we take care of this one please?14:17
slaglei noticed this b/c i was working on https://review.openstack.org/#/c/560030/14:17
EmilienMwe shouldn't have non-containerized overcloud jobs anywhere, imho14:17
fultonjo/14:17
slagleamong these points, i also wonder why experimental jobs are in the promotion chain?14:18
openstackgerritSaravanan KR proposed openstack/tripleo-heat-templates master: WIP: docker-puppet failure for ovs containers  https://review.openstack.org/56397814:18
EmilienMslagle: correct, it sounds wrong14:18
slaglenot sure if that is intentional, but we promote with way more than we check/gate14:18
slagleseems backwards to me14:18
*** wolverineav has quit IRC14:18
*** waleedm has quit IRC14:18
*** dhill_ has quit IRC14:18
*** wolverineav has joined #tripleo14:18
mwhahahait might just need to be removed from the master/queens promotion criteria14:18
mwhahahait might still be in use for <=pike14:19
panda|ruckI don't see f021 on the promotion pipeline14:19
openstackgerritCarlos Camacho proposed openstack/tripleo-common stable/queens: Allow uploading big files to swift (5GB)  https://review.openstack.org/56397714:19
*** yprokule has joined #tripleo14:19
*** rpioso|afk is now known as rpioso14:19
slaglepanda|ruck: ok. maybe i'm looking in the wrong place14:19
panda|ruckslagle: what experimental job ? not all the jobs that are tested in the pipeline count as promotion14:20
*** pchavva has joined #tripleo14:20
slaglepanda|ruck: i was just looking here: http://cistatus.tripleo.org/promotion/ under RDO cloud OVB14:20
panda|ruckadding them there it's one of the steps, when they are stable , they are added to the criterias14:20
openstackgerritPranali Deore proposed openstack/tripleo-heat-templates master: Disable interoperable default image import method in THT  https://review.openstack.org/56293014:20
*** etingof has joined #tripleo14:21
EmilienMfs021 shouldn't be run on master neither queens anyway14:21
panda|ruckI think that job is in the quickstart gates only14:22
slagleok, i'm just going to remove it from my patch then https://review.openstack.org/#/c/560030/ as we don't know what it's for14:23
*** wolverineav has quit IRC14:24
mwhahahaso it sounds like we should have  a bug opened for the fact that fs021 is running on master14:24
*** wolverineav has joined #tripleo14:24
EmilienM(and queens)14:24
mwhahahawho wants an action item? :D14:25
slaglemaybe it doesn't? i don't know now. my oooq patch is master14:25
*** dhill_ has joined #tripleo14:25
mwhahahawell it's ruinning on 3rd party which is now voting14:25
mwhahahaso it would seem that it's incorrectly configured14:25
panda|ruckI'll check with the tempest squad14:26
quiquell|roverAlso we have this bug on fs021 still in progress14:26
quiquell|roverhttps://bugs.launchpad.net/tripleo/+bug/175987614:26
openstackLaunchpad bug 1759876 in tripleo "ovb 3ctrl 1comp fs021 mismatch_error: test_upload_too_many_objects" [High,Triaged] - Assigned to Arx Cruz (arxcruz)14:26
panda|ruckit seems that job is an old test job for the full tempest run14:26
mwhahaha#action ci team, panda|ruck, quiquell|rover to investigate why fs021 is running on master if it's not properly containerized14:26
mwhahahaalright, anything else?14:27
sshnaidmpanda|ruck, for smoke tempest run, yet before we had tempest in all jobs14:27
panda|rucksshnaidm: ok, I think we can remove it, talking with tempest guys14:28
*** leitan has quit IRC14:28
*** zoli is now known as zoli|brb14:28
*** leitan has joined #tripleo14:28
*** wolverineav has quit IRC14:28
mwhahahaalright thanks everyone14:29
mwhahaha#endmeeting14:29
*** openstack changes topic to "Welcome to Rocky. CI status Yellow/Red, rdo packaging updating yum repos for rocky | http://tripleo.org/ | https://docs.openstack.org/tripleo-docs/latest"14:29
openstackMeeting ended Tue Apr 24 14:29:37 2018 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)14:29
openstackMinutes:        http://eavesdrop.openstack.org/meetings/tripleo/2018/tripleo.2018-04-24-14.00.html14:29
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/tripleo/2018/tripleo.2018-04-24-14.00.txt14:29
openstackLog:            http://eavesdrop.openstack.org/meetings/tripleo/2018/tripleo.2018-04-24-14.00.log.html14:29
openstackgerritMerged openstack/tripleo-heat-templates master: Added missing pki volume for custom CA.  https://review.openstack.org/56352514:30
slaglepanda|ruck: so, when i'm looking here: http://cistatus.tripleo.org/promotion/, i should be lookinag the list of jobs under "Master-promotion" to see what jobs failed that blocked the promotion?14:30
slaglei'm just looking for the right place to check what promotion jobs failed14:31
openstackgerritHonza Pokorny proposed openstack/tripleo-common master: Fix tarball generation when downloading logs  https://review.openstack.org/56398514:31
openstackgerritBogdan Dobrelya proposed openstack/python-tripleoclient master: Wire-in hieradata_override for UC heat-install  https://review.openstack.org/56398614:31
jpichhonza: much better after applying that tip to add an extra networking interface, thanks for the links!14:31
panda|ruckslagle: that is one place, another one is this https://review.rdoproject.org/jenkins/job/periodic-tripleo-centos-7-master-containers-build/838/14:32
*** etingof has quit IRC14:33
*** leitan has quit IRC14:33
*** etingof has joined #tripleo14:33
slaglein which case it's featureset030 that is blocking promotion14:34
slaglewhich is OVN with containerized undercloud14:34
honzajrist: https://review.openstack.org/#/c/563985/14:34
panda|ruckslagle: https://review.rdoproject.org/jenkins/job/periodic-tripleo-centos-7-master-containers-build/836/ here fs030 is the only one failing in master14:34
bogdandoslagle: , dprince: https://review.openstack.org/#/c/563986 may be reused for https://bugs.launchpad.net/tripleo/+bug/1763337 in https://review.openstack.org/#/c/563610/14:34
openstackLaunchpad bug 1763337 in tripleo "Wire in net_config_override in undercloud.conf for containerized undercloud heat installer" [Medium,In progress] - Assigned to Bogdan Dobrelya (bogdando)14:34
myoungslagle: I think (IMHO) we have a bit of work to do to present cleanly which jobs have passed / failed for a particular promotion attempt.  The "source of truth" is really (presently) in the promotion logs.  the Sova pages show all jobs w/ history...while it's useful to drill into job failures, particularly to identify failure causes over time, it's not "per promotion focused"14:34
jristhonza: that's it?14:35
honzajrist: lol yes14:35
jristhonza: link that in the bz?14:35
honzajrist: doing that now14:35
bogdandoexcept that added some templates processing and context mappings ofc14:35
openstackgerritMerged openstack/tripleo-heat-templates stable/queens: Fix Cinder's default db purge cron settings  https://review.openstack.org/56015414:35
honzajrist: do you think we can backport that to queens?14:35
panda|ruckslagle: more recent jobs are failing to build containers for a different reason14:35
myoungslagle: e.g., there's WAY more jobs reported/tracked by sova than are part of actual promotion criteria14:35
slaglemyoung: what's sova14:36
*** quiquell|rover is now known as quiquell|off14:37
myoungslagle: http://cistatus.tripleo.org is and instance of https://github.com/sshnaidm/sova14:37
jristhonza: can/should yes14:37
honzajrist: A+14:38
slaglemyoung: yea, ok. that's where I look14:38
*** leitan has joined #tripleo14:38
bogdandohttps://review.openstack.org/#/c/563562/ what a nice CI status14:38
bogdandoa perfect storm14:39
arxcruzbogdando: in oposite to https://review.openstack.org/#/c/561920/ :D14:40
bogdandoarxcruz: almost, there is one red sneaked in14:41
openstackgerritMerged openstack/tripleo-common stable/queens: Bump Ceph container image from 3.0.1 to 3.0.3  https://review.openstack.org/56301714:41
openstackgerritMerged openstack/tripleo-heat-templates master: Suppress warnings with -qq  https://review.openstack.org/56165614:41
openstackgerritMerged openstack/tripleo-heat-templates master: Fail validation on new workflow_tasks  https://review.openstack.org/56165714:41
*** links has quit IRC14:42
sshnaidmmyoung, what it link to community meeting?14:42
bogdandoPTAL https://review.openstack.org/#/c/556516/14:42
bogdandohttps://review.openstack.org/#/c/562983/14:42
EmilienMsshnaidm: https://bluejeans.com/705085945514:43
EmilienMsshnaidm: https://etherpad.openstack.org/p/tripleo-ci-squad-meeting14:43
sshnaidmEmilienM, thanks14:43
*** moshele has joined #tripleo14:43
*** shreshtha_ has quit IRC14:45
*** shreshtha_ has joined #tripleo14:46
*** hjensas has joined #tripleo14:46
*** hjensas has quit IRC14:46
*** hjensas has joined #tripleo14:46
openstackgerritBogdan Dobrelya proposed openstack/tripleo-quickstart-extras master: Wire-in UC deploy role data file  https://review.openstack.org/56398814:46
*** moshele has quit IRC14:48
*** yprokule has quit IRC14:49
*** ajo has quit IRC14:50
*** ajo has joined #tripleo14:51
openstackgerritDougal Matthews proposed openstack/tripleo-common master: Migrate to the new Mistral context class  https://review.openstack.org/50618614:51
openstackgerritDougal Matthews proposed openstack/tripleo-common master: Fix the passlib deprecation warning  https://review.openstack.org/56225114:51
openstackgerritPranali Deore proposed openstack/tripleo-heat-templates stable/pike: Add support of netapp configuration for glance in THT  https://review.openstack.org/56398914:54
* dtantsur lols at the name of https://github.com/sshnaidm/sova14:55
*** yolanda_ has joined #tripleo14:55
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-heat-templates stable/newton: Expend bash idiom to make it sh compatible in in get_all_bridges.  https://review.openstack.org/56399114:56
*** moguimar has joined #tripleo14:57
openstackgerritPranali Deore proposed openstack/tripleo-heat-templates stable/pike: Add support of netapp configuration for glance in THT  https://review.openstack.org/56398914:57
*** yolanda has quit IRC14:58
openstackgerritJavier Peña proposed openstack-infra/tripleo-ci master: Prepare for DLRN modular build driver structure  https://review.openstack.org/56399315:00
EmilienMpanda|ruck: who is the rover now?15:02
openstackgerritBogdan Dobrelya proposed openstack/tripleo-quickstart-extras master: Fix path and wire-in UC deploy role data file  https://review.openstack.org/56398815:02
panda|ruckEmilienM: quiquell|off15:02
EmilienMok15:03
panda|ruckEmilienM: mwhahaha slagle tempest squad is going to create a new job for master, fs021 will be kept only for older branches. It's the standard gate for tempest related changes.15:04
*** aufi_ has quit IRC15:04
openstackgerritBogdan Dobrelya proposed openstack/tripleo-quickstart-extras master: Fix path and wire-in UC deploy role data file  https://review.openstack.org/56398815:05
*** jaosorior has joined #tripleo15:05
*** dbecker has joined #tripleo15:06
*** tesseract has quit IRC15:06
*** tesseract has joined #tripleo15:06
bogdandoEmilienM: please take another  look https://review.openstack.org/#/c/556522/15:08
openstackgerritMarios Andreou proposed openstack/python-tripleoclient stable/queens: Q upgrade - fixup completion messages and --help doctext  https://review.openstack.org/56391615:09
EmilienMbogdando: ok15:09
*** ooolpbot has joined #tripleo15:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION15:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/176619515:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/176640615:10
*** ooolpbot has quit IRC15:10
openstackLaunchpad bug 1766195 in tripleo "Error installing fluent-plugin-kubernetes_metadata_filter at pike" [Critical,Fix committed] - Assigned to Quique Llorente (quiquell)15:10
openstackLaunchpad bug 1766406 in tripleo "multinode-oooq-container-updates fails on master" [Critical,Fix committed] - Assigned to Carlos Camacho (ccamacho)15:10
EmilienMbogdando: i'll wait for the recheck results15:10
*** ubijtsa has quit IRC15:12
openstackgerritEmilien Macchi proposed openstack/python-tripleoclient master: requirements: bump tripleo-common to 9.0.1  https://review.openstack.org/56400115:16
*** assassin has joined #tripleo15:16
openstackgerritEmilien Macchi proposed openstack/python-tripleoclient master: requirements: bump tripleo-common to 9.0.1  https://review.openstack.org/56400115:16
openstackgerritEmilien Macchi proposed openstack/python-tripleoclient master: Save temporary working data  https://review.openstack.org/56057815:17
openstackgerritEmilien Macchi proposed openstack/python-tripleoclient master: Enable cleanup by default for undercloud install  https://review.openstack.org/56203215:17
*** zoli|brb is now known as zoli15:19
*** lyaaaaaaaaarwood is now known as lyarwood15:19
*** morazi has quit IRC15:20
*** dbecker has quit IRC15:20
*** ccamacho1 has joined #tripleo15:21
openstackgerritAlex Schultz proposed openstack/python-tripleoclient master: Save temporary working data  https://review.openstack.org/56057815:21
*** ccamacho has quit IRC15:21
*** yprokule has joined #tripleo15:22
openstackgerritAlex Schultz proposed openstack/python-tripleoclient master: Save temporary working data  https://review.openstack.org/56057815:23
openstackgerritAlex Schultz proposed openstack/python-tripleoclient master: Enable cleanup by default for undercloud install  https://review.openstack.org/56203215:23
openstackgerritBogdan Dobrelya proposed openstack/python-tripleoclient master: UC: validate paths for tht templates and env files  https://review.openstack.org/56119215:23
*** leanderthal has quit IRC15:24
*** athomas has quit IRC15:31
*** morazi has joined #tripleo15:31
*** spsurya has quit IRC15:31
*** pblaho has quit IRC15:32
*** spsurya has joined #tripleo15:32
bogdandoshardy, matbu: hi. Reworked https://review.openstack.org/#/c/561192 to only contain the undercloud heat part15:33
bogdandoplease consolidate your proposals for the remaining overcloud part15:33
bogdandomatbu: some background:15:34
bogdando4:56:30 PM GMT+2 - shardy: if the file exists in the plan, the file will be j2 rendered15:34
bogdando4:57:26 PM GMT+2 - shardy: bogdando: you could strip the dirname and error (or at least warn) if someone passes a -e foo.yaml where a corresponding foo.j2.yaml exists in the plan15:34
bogdando4:58:19 PM GMT+2 - shardy: bogdando: also when deploying the client downloads all the rendered files (_download_missing_files_from_plan)15:34
openstackgerritJavier Peña proposed openstack/tripleo-quickstart-extras master: Prepare for DLRN modular build driver structure  https://review.openstack.org/56400615:34
shardybogdando: ack thanks will check it out15:34
*** moshele has joined #tripleo15:35
*** moguimar has quit IRC15:35
matbubogdando: ack thx15:37
matbubogdando: i will try to push something later today or tomorow15:37
openstackgerritBogdan Dobrelya proposed openstack/tripleo-heat-templates master: Switch remaining env files to containers defaults  https://review.openstack.org/56055915:39
*** ykarel is now known as ykarel|away15:42
openstackgerritJose Luis Franco proposed openstack/tripleo-common stable/queens: Remove version section in overcloud_keystone_url.  https://review.openstack.org/56359815:43
openstackgerritBogdan Dobrelya proposed openstack/tripleo-heat-templates master: Switch remaining env files to containers defaults  https://review.openstack.org/56055915:43
*** kopecmartin has quit IRC15:48
*** jfrancoa1 has quit IRC15:49
trozetjaosorior: hi15:51
*** pcaruana has quit IRC15:51
*** agurenko has quit IRC15:51
jaosoriortrozet: sup15:51
trozetjaosorior: beagles and I were trying to figure out why neutron was still run as paste web server instead of part of httpd15:51
trozetjaosorior: he thought you may know?15:52
jaosoriorhaha he asked me a bit ago15:53
*** bogdando has quit IRC15:53
jaosoriorI don't know15:53
jaosoriortrozet: so I tried to propose it to run over httpd15:53
*** moshele has quit IRC15:53
jaosoriorand someone was not too keen into the idea15:53
jaosoriornot sure who15:53
jaosoriorand it's been a whlie15:53
*** waleedm has joined #tripleo15:53
jaosorior* while15:53
trozetjaosorior: was that on a patch or?15:53
*** atoth has quit IRC15:53
*** ykarel|away has quit IRC15:54
*** ccamacho1 has quit IRC15:55
*** atoth has joined #tripleo15:55
*** moguimar has joined #tripleo15:55
*** jbadiapa has quit IRC15:55
Tenguhumpf. shardy gfidente : still timeouting for my deploy, still the very same step :'(15:59
openstackgerritRedHat RDO CI proposed openstack/tripleo-heat-templates stable/queens: GATE CHECK for TripleO  https://review.openstack.org/56045216:00
openstackgerritDavid Peacock proposed openstack/python-tripleoclient master: Containerized undercloud completion messages  https://review.openstack.org/56097316:00
Tenguwhat may I show up regarding my deploy command so that we can find the reason of that timeout for WorkflowTasks_Step2_Execution ? :/16:00
gfidenteTengu the ceph-ansible-workflow.log16:00
gfidentecan you paste it somewhere?16:00
Tengugfidente: might be, yep, wait.16:00
gfidentefrom there we see what ceph-ansible task is failing and why16:00
Tenguhm... no failure at the end16:01
Tengulike, ansible returns 0 failed.16:01
Tengubut anyway. lemme cut it in order to get the current run only, and strip the key things if any.16:02
beaglesmwhahaha: what triggers the rebuild of a container image?16:02
beaglesmwhahaha: noticing that the dhcp agent image has docker in it but the l3 agent does not16:02
*** zoli is now known as zoli|gone16:03
beaglesmwhahaha: double checking that the kolla build does hte right thing16:03
*** zoli|gone is now known as zoli16:03
*** avivgt has quit IRC16:04
openstackgerritRyan Brady proposed openstack/tripleo-common master: Add Workflow To Create Overcloudrc Files  https://review.openstack.org/54347716:04
*** ramishra has quit IRC16:05
*** moshele has joined #tripleo16:06
*** lucasagomes is now known as lucas-afk16:06
*** moshele has quit IRC16:07
*** ooolpbot has joined #tripleo16:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION16:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/176619516:10
openstackLaunchpad bug 1766195 in tripleo "Error installing fluent-plugin-kubernetes_metadata_filter at pike" [Critical,Fix committed] - Assigned to Quique Llorente (quiquell)16:10
*** ooolpbot has quit IRC16:10
*** yamahata has quit IRC16:15
*** waleedm has quit IRC16:16
*** gkadam has quit IRC16:18
openstackgerritRicardo Noriega proposed openstack/puppet-tripleo master: WIP: Adding Tacker conductor puppet class  https://review.openstack.org/56402116:19
openstackgerritJames Slagle proposed openstack/python-tripleoclient master: Don't exit on a RUNNING message for deploy workflow  https://review.openstack.org/55735616:22
*** alee_ has joined #tripleo16:22
*** alee has quit IRC16:26
*** moshele has joined #tripleo16:26
mwhahahabeagles: nothing. nothing triggers container rebuilds these days16:26
beaglesmwhahaha: arg16:27
mwhahahabeagles: it's a WIP on getting them auto-updated for CI16:27
mwhahahabeagles: so containers are at the mercy of the promotion process16:27
beaglesmwhahaha: k, I'll double check the container build to make sure it works and then prod whomever to get the l3 agent updated16:28
mwhahahabeagles: sthe l3 agent should just be what we have in the promotion process anyway16:28
*** moshele has quit IRC16:29
beaglesmwhahaha: that's what I figure, which is why I wonder if there is something wrong with the docker file or the template override that is making it not include the docker package16:29
mwhahahabeagles: so that's our tripleo bits right?16:29
mwhahahabeagles: so we'd have to check on the container build as part of promotion16:30
beaglesmwhahaha: yep16:30
mwhahahayea i think there's currently an issue blocking promotion16:30
mwhahahaso that's why it may not have docker yet16:30
*** tvignaud has quit IRC16:30
mwhahahabeagles: so we're 4 days behind for tripleo-ci16:31
*** marios has quit IRC16:31
mwhahahapanda|ruck: what's the current tissue blocking master? is it a specific fs?16:32
*** jchhatbar has quit IRC16:32
*** jpich has quit IRC16:32
Tenguhmm. is it normal for mistral to always output stuff in its "engine.log", even if no task is running?16:33
d0ugalTengu: What sort of stuff?16:33
Tenguit talks about delayed tasks being deleted. always the same amount16:33
*** dparkes has quit IRC16:33
d0ugalTengu: Can you show me?16:33
Tengu1s16:33
*** khyr0n has joined #tripleo16:33
Tengud0ugal: http://paste.openstack.org/show/719844/ (on the undercloud)16:34
*** ffiore has quit IRC16:34
Tenguand it goes on and on and on again.16:34
panda|ruckmwhahaha: https://bugs.launchpad.net/tripleo/+bug/1766493. But sicne it is in the first job in the pipeline, we may have others16:35
openstackLaunchpad bug 1766493 in tripleo "Building kolla openstack-base No package python2-positional available" [Critical,In progress] - Assigned to Quique Llorente (quiquell)16:35
openstackgerritDamien Ciabrini proposed openstack/tripleo-heat-templates master: WIP Instance HA evacuation fix  https://review.openstack.org/56402416:35
d0ugalTengu: that doesn't seem completely normal :)16:35
mwhahahapanda|ruck: thanks, so beagles yea you'll need a promotion before the docker bits show up in your containers16:35
mwhahahabeagles: welcome to containers16:35
d0ugalTengu: about half way along those lines I am seeing some execution id's. "f3e784d5-3eec-4e9d-9a45-45e2965933e3" - that is the first, there are 6 on their own lines16:35
d0ugalTengu: I would run "mistral execution-get $ID" for each of those and see what it shows.16:36
Tengud0ugal: ah :). hmm. do you want me to...16:36
Tenguok :)16:36
beaglesI can barely contain myself16:36
Tenguhmmm. d0ugal State             | RUNNING16:37
d0ugalTengu: I thought they might be - what workflow are they? I wonder if they have somehow gotten stuck.16:37
TenguI suspect this might be the issue with my timeouts. Will try to catch the other IDs.16:37
TenguWorkflow name     | tripleo.storage.v1.ceph-install16:37
Tenguahaaaa16:37
Tenguceph16:37
Tengu:)16:37
d0ugallol16:37
Tenguas usual16:37
Tenguceph will kill me for sure :D16:37
Tenguany way to clean them up?16:38
d0ugalTengu: You could try just deleting them - "mistral execution-delete ID"16:38
mwhahahabeagles: so speaking of containers, did you know the neutron-server health check is bad? it assumes running in apache16:38
Tengud0ugal: trying on the ID you showed earlier.16:39
TenguRequest to delete execution f3e784d5-3eec-4e9d-9a45-45e2965933e3 has been accepted.16:39
Tengusounds good.16:39
beaglesmwhahaha: oh for ...16:39
TenguERROR (app) WorkflowExecution not found [id=f3e784d5-3eec-4e9d-9a45-45e2965933e3]16:39
mwhahahabeagles: i know right?16:39
Tengud0ugal: may I do the same with the now 10 other IDs? :)16:39
Tenguas apparently there were 11 delayed tasks.16:40
d0ugalTengu: sure16:40
d0ugalTengu: we should maybe have tried to track down the root cause... but that'll clean it up.16:40
Tengud0ugal: I suspect my failed deploy.16:40
Tengumight have let some zombies.16:41
d0ugalTengu: right, but even with a failed deploy the workflow should never be left RUNNING.16:41
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: WIP - heat: align config with instack-undercloud  https://review.openstack.org/55835416:41
d0ugalIt should be in an ERROR state.16:41
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: WIP - heat: align config with instack-undercloud  https://review.openstack.org/55835416:41
d0ugalTengu: if you have any left still, I'd be interested to see "mistral task-list ID" for one of the executions16:41
TenguI can, for sure16:41
d0ugalTengu: was Mistral ever restarted?16:42
Tengud0ugal: http://paste.openstack.org/show/719846/16:42
Tengud0ugal: hmmm yep. I confess.16:42
d0ugalTengu: aha, so, restarting Mistral while ansible is running may cause issues.16:43
d0ugalWe need to find a better way to recover from that16:43
* Tengu goes in hiding16:43
d0ugal:)16:43
Tengumay I continue the cleanup?16:43
*** pcaruana has joined #tripleo16:43
d0ugalYeah, go for it16:43
Tengugood :)16:43
d0ugalThanks!16:43
Tenguthank you for the support :).16:44
d0ugalnp16:44
Tengud0ugal: so this output is OK now? http://paste.openstack.org/show/719847/16:45
openstackgerritMatthew Thode proposed openstack/diskimage-builder master: Install sudo on Gentoo images by deault  https://review.openstack.org/56402816:45
*** trown is now known as trown|lunch16:45
d0ugalTengu: seems fine to me16:46
Tengucool16:46
Tengusooo. I'll start over a new deploy :)16:46
d0ugalTengu: it is kinda verbose, but that is DEBUG logging level16:46
d0ugalgood luck!16:47
Tengud0ugal: fine for me. I like debug :).16:47
Tengud0ugal: well, it's only the… errr… hm. I lost count. ;)16:47
openstackgerritAlan Bishop proposed openstack/tripleo-heat-templates stable/pike: Fix Cinder's default db purge cron settings  https://review.openstack.org/56402916:48
d0ugallol16:48
Tengumight be over 1000 since the very beginning. Last year, that is.16:48
Tenguand counting the lab.16:49
*** jabadia has joined #tripleo16:49
*** ccamacho has joined #tripleo16:49
*** etingof has quit IRC16:53
*** rbowen has quit IRC16:55
*** tvignaud has joined #tripleo16:56
*** itlinux has joined #tripleo16:57
*** rbowen has joined #tripleo16:59
*** fragatina has quit IRC16:59
*** moguimar has quit IRC17:01
*** derekh has quit IRC17:01
*** atoth has quit IRC17:02
*** fragatina has joined #tripleo17:03
*** thrash is now known as thrash|g0ne17:03
*** thrash|g0ne is now known as thrash|biab17:04
*** panda|ruck is now known as panda|ruck|off17:04
*** noam__ has joined #tripleo17:06
*** khyr0n has quit IRC17:08
*** fragatina has quit IRC17:08
*** jtomasek has quit IRC17:10
*** ooolpbot has joined #tripleo17:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION17:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/176619517:10
*** ooolpbot has quit IRC17:10
openstackLaunchpad bug 1766195 in tripleo "Error installing fluent-plugin-kubernetes_metadata_filter at pike" [Critical,Fix committed] - Assigned to Quique Llorente (quiquell)17:10
*** khyr0n has joined #tripleo17:11
*** fragatina has joined #tripleo17:11
*** jpena is now known as jpena|off17:12
*** dsariel has quit IRC17:13
*** udesale has joined #tripleo17:15
*** fragatina has quit IRC17:17
*** atoth has joined #tripleo17:17
*** dtantsur is now known as dtantsur|afk17:17
beaglesmwhahaha: doesnt' https://review.openstack.org/#/c/558775/1 fix the api healthcheck thing?17:19
*** artom_ has joined #tripleo17:20
*** artom has quit IRC17:20
Tengud0ugal: crossing fingers - I'm currently in the step that timeouts. IF the zombies were the only issue, it's all good :)17:21
mwhahahabeagles: ah yea, it wasn't in my container17:21
*** jaosorior has quit IRC17:24
*** mdnadeem has quit IRC17:26
Tenguhmmmm.17:26
*** ffiore has joined #tripleo17:27
Tengud0ugal: now I have 2 "delayed tasks", and they are, once again, related to ceph.17:27
Tenguit seems "something" just doesn't report correctly at some point :/ (ping gfidente - sorry, yes, me again)17:27
*** dhill_ has quit IRC17:27
openstackgerritJiri Stransky proposed openstack/tripleo-heat-templates stable/queens: Introduce Ceph upgrade environments  https://review.openstack.org/56403317:27
openstackgerritJiri Stransky proposed openstack/python-tripleoclient stable/queens: Introduce Ceph upgrade command  https://review.openstack.org/56403417:30
Tenguaha!17:32
TenguI have something, digging in mistral.17:32
*** morazi has quit IRC17:32
beaglesgeez.. I can't get kolla-build to build even 'base'17:34
*** dhill_ has joined #tripleo17:34
Tengugfidente fultonj I see this state for an "execution" in mistral : State info        | Failure caused by error in tasks: ceph_base_ansible_workflow - any way to debug that?17:34
TenguDUH17:35
Tengugot it.17:35
TenguCommand: ansible-playbook /usr/share/ceph-ansible/infrastructure-playbooks/switch-from-non-containerized-to-containerized-ceph-daemons.yml   <- wrong playbook. Maybe a typo.17:35
*** thrash|biab is now known as thrash17:35
Tengugfidente fultonj hmmm, apparently, even if I have pushed "CephAnsiblePlaybook: /usr/share/ceph-ansible/site-docker.yml.sample" in my env file, tripleo still wants to use the migration one o__O17:36
*** yprokule has quit IRC17:36
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: Add libvirt option to reproducer script  https://review.openstack.org/56331917:37
Tenguand this, of course, is the root cause of my issues.17:37
*** fragatina has joined #tripleo17:37
*** holser__ has quit IRC17:40
*** dmacpher has quit IRC17:40
*** agopi|zzz has quit IRC17:41
*** agopi|zzz has joined #tripleo17:41
*** ubijtsa has joined #tripleo17:41
*** assassin has quit IRC17:42
Tenguso. now. how am I supposed to enforce the right playbook :/17:43
*** ubijtsa1 has joined #tripleo17:43
*** ubijtsa1 is now known as assassin17:43
*** assassin is now known as Guest6558817:43
Tengud0ugal: if I drop a mistral "execution" part of the "overcloud" heat stack, will it be re-created properly? I suspect something just didn't update the execution properly.17:45
*** gfidente has quit IRC17:46
*** ubijtsa has quit IRC17:46
*** ffiore has quit IRC17:46
jristhonza: can you comment on this plz https://review.openstack.org/#/c/562039/617:49
jristI think I did it wrong17:49
honzajrist: sure thing17:49
openstackgerritMerged openstack/tripleo-heat-templates stable/queens: Fix openstack-nova-consoleauth service upgrade check  https://review.openstack.org/56366917:50
openstackgerritMerged openstack/tripleo-common stable/pike: Bump Ceph container image from 3.0.1 to 3.0.3  https://review.openstack.org/56302717:50
openstackgerritMerged openstack/tripleo-upgrade master: undercloud: workaround for containers  https://review.openstack.org/56377217:50
openstackgerritMerged openstack-infra/tripleo-ci master: Correct branch used in tripleo-upgrade installation.  https://review.openstack.org/56308117:50
openstackgerritMerged openstack/tripleo-heat-templates stable/queens: Telemetry services need to access to Storage  https://review.openstack.org/56371517:50
openstackgerritMerged openstack/tripleo-heat-templates stable/pike: Telemetry services need to access to Storage  https://review.openstack.org/56371617:51
*** ffiore has joined #tripleo17:51
*** trown|lunch is now known as trown17:55
*** udesale has quit IRC17:57
*** bkopilov has quit IRC17:59
*** wolverineav has joined #tripleo18:01
*** moshele has joined #tripleo18:02
*** khyr0n has quit IRC18:04
*** dhill_ has quit IRC18:04
*** ffiore has quit IRC18:04
*** ffiore_ has joined #tripleo18:05
*** alee__ has joined #tripleo18:05
*** wolverineav has quit IRC18:06
openstackgerritMerged openstack/tripleo-quickstart-extras master: Fixing tempest containerized script  https://review.openstack.org/56215518:07
openstackgerritMerged openstack/tripleo-common master: Allow uploading big files to swift (5GB)  https://review.openstack.org/55902418:07
openstackgerritMerged openstack/python-tripleoclient master: Fix ansible config download dir  https://review.openstack.org/56331718:07
openstackgerritMerged openstack/tripleo-upgrade stable/queens: Fix newline stripping in upgrade template.  https://review.openstack.org/56389618:07
openstackgerritMerged openstack/tripleo-heat-templates stable/queens: Default collectd to overcloud gnocchi  https://review.openstack.org/56390818:07
*** ffiore_ has quit IRC18:07
*** alee_ has quit IRC18:08
*** atoth has quit IRC18:08
*** moshele has quit IRC18:10
*** dhill_ has joined #tripleo18:11
*** bkopilov has joined #tripleo18:20
*** amoralej is now known as amoralej|off18:24
openstackgerritHonza Pokorny proposed openstack/tripleo-ui master: Mask Passwords and allow Copy to Clipboard  https://review.openstack.org/56203918:24
*** akrivoka has quit IRC18:25
honzajrist: I have no idea how you got such a huge npm-shrinkwrap diff but I don't blame you.  That thing is full of mysteries.  Did you use npm@5.6.0?  I uploaded a new patchset and it's down from 26k to 17 LOC :)18:26
*** pcaruana has quit IRC18:32
*** moshele has joined #tripleo18:32
*** sshnaidm has quit IRC18:32
openstackgerritJohn Trowbridge proposed openstack/tripleo-quickstart-extras master: WIP Add role to manage snapshots in a libvirt environment  https://review.openstack.org/56329318:33
*** moshele has quit IRC18:36
jristlol18:37
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart master: Disable cloud-init for VMs in libvirt-fake-nodepool  https://review.openstack.org/56376318:38
jristhonza: I'm using a newer version of node, but npm is still behind (using nvm)18:39
jristhow do I update NPM via nvm18:39
jrist3.8.618:39
jristnpm -v ^18:39
jristtrying sudo npm install -g npm18:40
*** dprince has quit IRC18:42
jristhonza: lol wut http://paste.openstack.org/show/719853/18:42
*** shreshtha_ has quit IRC18:42
jristhonza: trying nvm install-latest-npm18:42
jristlol lol lol18:43
jristhttp://paste.openstack.org/show/719854/18:43
jristhonza: ^ utter failure18:43
jristhonza: also this needs an apparent rebase? https://review.openstack.org/#/c/563985/18:48
openstackgerritAlex Schultz proposed openstack-infra/tripleo-ci master: Don't cleanup undercloud  https://review.openstack.org/56377718:49
*** dxiri has quit IRC18:53
beaglesslagle: can you think of any reason why this http://logs.openstack.org/23/550823/11/check/tripleo-ci-centos-7-3nodes-multinode/af1e3ca/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-04-24_17_25_1718:56
beaglesslagle: I tried it in my env and it seemed to work fine18:56
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: Add a libvirt option to the reproducer script  https://review.openstack.org/56331918:56
beaglesslagle: just wonder if there's something deployed-server'ish that I need to account for or something18:57
*** paramite has joined #tripleo18:58
openstackgerritBen Nemec proposed openstack/tripleo-common master: Use Designate Worker container for config  https://review.openstack.org/56404918:59
openstackgerritBen Nemec proposed openstack/tripleo-common master: Use Designate Worker container for config  https://review.openstack.org/56404919:00
bnemecNow in alphabetical order!19:00
bnemecBTW, is the CI status still current?  I think that was fixed?19:00
mwhahahai think it's fixed19:01
mwhahahaor we can just lie and say it's fixed until someone proves otherwise19:01
*** mwhahaha changes topic to "Welcome to Rocky. CI status GREEN? | http://tripleo.org/ | https://docs.openstack.org/tripleo-docs/latest"19:01
mwhahahabeagles: that's an interesting error, not sure why it's getting a permission denied19:02
openstackgerritJohn Trowbridge proposed openstack/tripleo-quickstart-extras master: Setup iptables in nodepool-setup  https://review.openstack.org/56405019:03
*** yamahata has joined #tripleo19:03
beaglesmwhahaha: yeah, it's pretty weird19:04
beaglesmwhahaha: I had been developing/testing with an undercloud and client from before the config-download switchover so I thought that might be it19:04
mwhahahano it's unlikely that is the issue19:05
beaglesmwhahaha: but I fired up a new env with fresh undercloud and it still worked (except I found out that the l3 agent image is missing the docker binaries)19:05
beaglesand you would be correct :)19:06
openstackgerritBen Nemec proposed openstack/tripleo-heat-templates master: Deploy Designate in scenario003  https://review.openstack.org/55500719:06
jristhonza: any bright ideas before I set off a tactical nuke?19:06
* beagles is going to start making voodoo dolls for each thing that "vexes" him - first will be kolla-build19:06
*** agopi|zzz has quit IRC19:07
mwhahahabeagles: oh do those files have a docker_configs_scripts section?19:12
* mwhahaha was trying to remember how we did that19:12
beaglesmwhahaha: it's actually docker_puppet_apply.sh run from docker_config19:13
mwhahahabeagles: https://github.com/openstack/tripleo-heat-templates/commit/6f834f60e619181890c4438d1315ebaa776c1d09#diff-0a502b81270733eae32da319d34f21d7R12719:13
mwhahahabeagles: you have to have the ddocker_config_scripts defined19:13
mwhahahaor it doesn't create the script19:13
*** rbowen has quit IRC19:14
*** radeks has quit IRC19:14
mwhahahaif i remember correctly (which i may not)19:14
mwhahahayea i think you need that19:15
*** yolanda__ has joined #tripleo19:15
*** rbowen has joined #tripleo19:15
beaglesah that would explain it19:15
*** sshnaidm has joined #tripleo19:16
beaglesmwhahaha: okay I'll update and see what happens19:16
*** yolanda_ has quit IRC19:18
slaglebeagles: was in a mtng. did mwhahaha get you squared away?19:18
beaglesslagle: think so maybe :)19:18
openstackgerritBrent Eagles proposed openstack/tripleo-heat-templates master: Generate and mount wrappers for neutron agent processes  https://review.openstack.org/55082319:18
*** tesseract has quit IRC19:26
jristhonza: we are using npm 5 or node 519:27
*** dprince has joined #tripleo19:30
*** agopi|zzz has joined #tripleo19:31
*** paramite has quit IRC19:42
*** dprince has quit IRC19:42
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci master: Introduce tripleo-ci-centos-7-containerized-undercloud-upgrades  https://review.openstack.org/55363319:42
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci master: Introduce tripleo-ci-centos-7-containerized-undercloud-upgrades  https://review.openstack.org/55363319:42
EmilienMcan someone get https://review.openstack.org/#/c/563769/ please?19:43
slagle-119:44
* EmilienM removes -1 gerrit permissions to slagle19:45
*** florianf has quit IRC19:48
*** fragatina has quit IRC20:01
*** fragatina has joined #tripleo20:02
*** rasca has quit IRC20:04
*** weshay has quit IRC20:05
*** aputtur_ has quit IRC20:05
*** aputtur has joined #tripleo20:06
*** leitan has quit IRC20:07
*** weshay has joined #tripleo20:07
*** rasca has joined #tripleo20:07
*** leitan has joined #tripleo20:08
openstackgerritJames Slagle proposed openstack/tripleo-common stable/queens: Don't set deployments as hostvars  https://review.openstack.org/56405620:09
slaglemwhahaha: ^. i don't think any other patches are needed. guess we'll see20:10
mwhahahaslagle: cool thanks20:10
bnemecEmilienM: So you'd rather he -2 everything? ;-)20:11
openstackgerritJames Slagle proposed openstack/tripleo-common stable/queens: Don't set deployments as hostvars  https://review.openstack.org/56405620:11
*** dprince has joined #tripleo20:11
*** leitan has quit IRC20:12
*** itlinux has quit IRC20:13
EmilienMbnemec: hehe20:13
*** dhill_ has quit IRC20:17
*** itlinux has joined #tripleo20:19
*** khyr0n has joined #tripleo20:19
*** fragatina has quit IRC20:22
*** fragatina has joined #tripleo20:22
*** agopi|zzz has quit IRC20:22
*** pblaho has joined #tripleo20:23
*** dhill_ has joined #tripleo20:25
*** slaweq has quit IRC20:25
*** slaweq has joined #tripleo20:26
EmilienMslagle, sdoran: do we have plans to convert https://github.com/openstack/tripleo-common/blob/39ec49ef259cac74ea1e1cf9dc13e37afcb93adc/tripleo_common/actions/ansible.py to use ansible.runner?20:27
*** yamahata has quit IRC20:29
slagleEmilienM: maybe/probably. once ansible runner is actually a released and packaged thing available for the right distros20:29
EmilienMcool20:30
*** dprince has quit IRC20:31
*** agopi|zzz has joined #tripleo20:32
*** suuuper has quit IRC20:32
*** dprince has joined #tripleo20:32
*** raildo has quit IRC20:34
*** raildo has joined #tripleo20:34
*** jcoufal_ has joined #tripleo20:34
*** stevebaker has quit IRC20:37
openstackgerritEmilien Macchi proposed openstack/python-tripleoclient master: WIP - Implement CloudImagePrepare command  https://review.openstack.org/56379420:37
openstackgerritEmilien Macchi proposed openstack/python-tripleoclient master: WIP - Implement CloudImagePrepare command  https://review.openstack.org/56379420:37
*** jcoufal has quit IRC20:37
*** gouthamr has quit IRC20:38
sdoranYes. It should make that Mistral action much simpler.20:38
sdoranAnsible Runner is set to be released this summer, though it's currently open sourced and available for testing.20:38
*** dmellado has quit IRC20:38
sdoranI have a meeting tomorrow with those working on Ansible Runner to talk about how it's emitting output and what we'd like it to do.20:38
openstackgerritBen Nemec proposed openstack/tripleo-heat-templates master: Fix path to deployed-server-environment.yaml  https://review.openstack.org/56405820:39
*** stevebaker has joined #tripleo20:42
*** gouthamr has joined #tripleo20:43
*** ansmith has quit IRC20:43
*** dmellado has joined #tripleo20:43
*** rmascena has joined #tripleo20:44
*** raildo has quit IRC20:46
*** pchavva has quit IRC20:46
*** moshele has joined #tripleo20:47
*** etingof has joined #tripleo20:49
*** rmascena has quit IRC20:50
*** moshele has quit IRC20:50
*** sshnaidm is now known as sshnaidm|off20:52
*** vpickard is now known as vpickard_20:55
*** rbowen has quit IRC20:57
openstackgerritMerged openstack/python-tripleoclient master: Q upgrade - fixup completion messages and --help doctext  https://review.openstack.org/56391220:58
*** trown is now known as trown|outtypewww21:02
openstackgerritMerged openstack/python-tripleoclient master: Alter default http boot path for containerized Ironic  https://review.openstack.org/55651621:03
*** yamahata has joined #tripleo21:09
*** itlinux has quit IRC21:12
*** rajinir has quit IRC21:12
*** slaweq has quit IRC21:14
*** slaweq has joined #tripleo21:15
*** slaweq has quit IRC21:20
*** Goneri has quit IRC21:22
*** tcw has quit IRC21:23
*** tcw has joined #tripleo21:24
*** abishop has quit IRC21:24
*** thrash is now known as thrash|g0ne21:25
*** noam__ has quit IRC21:26
*** ffiore_ has joined #tripleo21:27
*** jcoufal_ has quit IRC21:27
*** jabadia has quit IRC21:31
openstackgerritDamien Ciabrini proposed openstack/tripleo-heat-templates master: WIP Instance HA evacuation fix  https://review.openstack.org/56402421:34
*** ansmith has joined #tripleo21:34
*** wolverineav has joined #tripleo21:35
*** etingof has quit IRC21:37
*** dparkes has joined #tripleo21:38
*** jbadiapa has joined #tripleo21:38
*** dbecker has joined #tripleo21:38
*** dprince has quit IRC21:42
*** lblanchard has quit IRC21:52
*** agopi|zzz has quit IRC21:52
*** dbecker has quit IRC21:53
*** agopi|zzz has joined #tripleo21:57
*** alee__ has quit IRC22:03
*** moshele has joined #tripleo22:06
*** etingof has joined #tripleo22:21
*** ffiore_ has quit IRC22:25
openstackgerritMarius Cornea proposed openstack/tripleo-upgrade master: Add post upgrade Nova operation check  https://review.openstack.org/56273822:39
*** dtrainor has quit IRC22:41
*** moshele has quit IRC22:43
*** ssbarnea_ has quit IRC22:43
*** alee__ has joined #tripleo22:44
*** liverpooler has quit IRC22:47
*** bugzy_ has joined #tripleo22:53
*** etingof has quit IRC22:54
*** bugzy has quit IRC22:56
honzajrist: npm 5.6.0 and node 8.9.423:06
*** etingof has joined #tripleo23:12
*** dhill_ has quit IRC23:18
*** mcornea has quit IRC23:20
*** rlandy has quit IRC23:25
EmilienMsdoran: ok, thanks for the details23:28
openstackgerritAttila Darazs proposed openstack/tripleo-quickstart-extras master: GATE CHECK for quickstart-extras  https://review.openstack.org/47260723:54
*** tosky has quit IRC23:58

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!