Tuesday, 2016-09-13

bnemecbkero: Take a look at the deploy args: http://logs.openstack.org/06/353506/46/check/gate-tripleo-ci-centos-7-nonha-multinode/104e2ba/console.html#_2016-09-12_21_14_47_41197400:00
bnemecMy guess would be that you're missing the validation errors nonfatal one.00:01
bkerobnemec: Ahh, yeah. I had to remove that one because I was targeting mitaka.00:01
bkerobut now I'm trying master -- let's see if that's it.00:02
*** jobewan has joined #tripleo00:09
thrashEmilienM: https://review.openstack.org/#/c/368808/ didn't merge.. I'll backport as soon as that lands on stable/newton00:12
thrashEmilienM: I did a recheck on it.00:12
*** mbound has quit IRC00:16
*** limao has joined #tripleo00:39
EmilienMthrash: ok thx00:40
openstackgerritwes hayutin proposed openstack/tripleo-quickstart: set up quickstart to replace instack-virt-setup  https://review.openstack.org/35808900:53
openstackgerrityuyafei proposed openstack/tripleo-common: Replace raw_input with input to make PY3 compatible  https://review.openstack.org/33712901:01
*** rbrady has quit IRC01:10
*** akshai_ has quit IRC01:10
*** bana_k has quit IRC01:11
*** cwolferh has quit IRC01:12
*** bana_k has joined #tripleo01:12
*** pmannidi has joined #tripleo01:13
*** bana_k has quit IRC01:13
*** chlong_ has joined #tripleo01:13
*** mbound has joined #tripleo01:16
*** mburned is now known as mburned_out01:17
*** fultonj has quit IRC01:19
*** mbound has quit IRC01:22
*** saneax is now known as saneax-_-|AFK01:22
*** rbrady has joined #tripleo01:26
*** rbrady has joined #tripleo01:26
*** jobewan has quit IRC01:32
*** rbrady has quit IRC01:44
*** chlong_ has quit IRC01:49
*** fzdarsky has joined #tripleo01:54
*** egafford has joined #tripleo01:55
*** egafford has quit IRC02:00
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: Expose parameter to enable combination alarms  https://review.openstack.org/36374802:03
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: Add fluentd client service  https://review.openstack.org/35350602:04
*** chlong_ has joined #tripleo02:05
*** dmacpher has joined #tripleo02:07
*** myoung|bbl is now known as myoung02:10
openstackgerritMerged openstack/puppet-tripleo: Fill DNS name for haproxy certificates  https://review.openstack.org/36855902:21
*** bkopilov__ has quit IRC02:24
*** bkopilov_ has quit IRC02:26
*** bkopilov has quit IRC02:27
*** yamahata has quit IRC02:29
*** fzdarsky has quit IRC02:51
*** thrash is now known as thrash|g0ne03:02
*** Ryjedo_ has joined #tripleo03:09
*** Ryjedo has quit IRC03:10
*** Ryjedo_ is now known as Ryjedo03:10
*** tzumainn has quit IRC03:12
*** pmannidi has quit IRC03:19
*** jobewan has joined #tripleo03:26
*** rhefner has joined #tripleo03:29
*** yamahata has joined #tripleo03:29
*** coolsvap has joined #tripleo03:34
*** bana_k has joined #tripleo03:36
*** oshvartz has quit IRC03:43
openstackgerritwes hayutin proposed openstack/tripleo-quickstart: set up quickstart to replace instack-virt-setup  https://review.openstack.org/35808903:45
*** bkopilov has joined #tripleo03:51
*** bkopilov_ has joined #tripleo03:51
*** bkopilov__ has joined #tripleo03:52
*** rbrady has joined #tripleo03:52
*** yamahata has quit IRC03:54
*** nyechiel has joined #tripleo03:55
*** nyechiel has quit IRC03:55
*** oshvartz has joined #tripleo03:58
*** nyechiel has joined #tripleo03:59
*** bana_k has quit IRC04:01
*** pmannidi has joined #tripleo04:04
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Use centos-release-ceph-jewel package  https://review.openstack.org/36913704:04
*** myoung is now known as myoung|afk04:06
openstackgerritGiulio Fidente proposed openstack/tripleo-docs: Use centos-release-ceph-jewel package  https://review.openstack.org/36913904:08
*** karthiks has joined #tripleo04:21
*** oshvartz has quit IRC04:24
*** skramaja has joined #tripleo04:25
*** pgadiya has joined #tripleo04:58
*** limao has quit IRC04:59
*** limao has joined #tripleo05:01
openstackgerritSaravanan KR proposed openstack/os-net-config: Fixed nic numbering issue of DPDK nics after the nic has bound  https://review.openstack.org/36435405:01
*** cwolferh has joined #tripleo05:04
*** jtomasek has quit IRC05:08
*** masco has joined #tripleo05:13
openstackgerritSaravanan KR proposed openstack/os-net-config: Fixed nic numbering issue of DPDK nics after the nic has bound  https://review.openstack.org/36435405:17
*** anshul has joined #tripleo05:20
*** jaosorior has joined #tripleo05:21
*** anshul has quit IRC05:22
*** anshul has joined #tripleo05:23
*** bandini has joined #tripleo05:23
jaosoriord0ugal: let me know when you're online05:28
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Hook internal TLS flag to apache-based services  https://review.openstack.org/36607505:29
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add flag for internal TLS  https://review.openstack.org/36594205:29
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add HAProxy TLS handled by certmonger as composable service  https://review.openstack.org/35643005:29
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add option to specify Certmonger CA  https://review.openstack.org/36864305:29
*** anshul has quit IRC05:31
*** penick has joined #tripleo05:39
*** dsariel has joined #tripleo05:41
*** dbecker has joined #tripleo05:46
*** anshul has joined #tripleo05:47
*** flepied has quit IRC05:47
*** oshvartz has joined #tripleo05:50
*** jobewan has quit IRC05:52
*** saneax-_-|AFK is now known as saneax06:00
*** rasca has joined #tripleo06:03
openstackgerritMerged openstack/diskimage-builder: Fix mellanox element required kernel modules and user space packages  https://review.openstack.org/36531206:07
*** jlinkes has joined #tripleo06:11
*** rcernin has joined #tripleo06:19
*** dmacpher has quit IRC06:22
*** penick has quit IRC06:24
*** fragatina has quit IRC06:25
*** flepied has joined #tripleo06:25
*** fragatina has joined #tripleo06:26
*** jprovazn has joined #tripleo06:28
*** fragatin_ has joined #tripleo06:29
*** fragatina has quit IRC06:30
*** liverpooler has joined #tripleo06:32
*** fragatin_ has quit IRC06:33
*** masco has quit IRC06:34
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Terminate Zaqar websocket endpoint in HAProxy  https://review.openstack.org/36032906:35
*** athomas has joined #tripleo06:40
*** panda|Zz is now known as panda06:40
pandaperiodic jobs did not run today ?06:46
*** masco has joined #tripleo06:48
*** dixiaoli has quit IRC06:48
*** jcoufal has joined #tripleo06:50
*** tremble has joined #tripleo06:56
*** florianf has joined #tripleo06:56
*** dixiaoli has joined #tripleo06:58
*** pcaruana has joined #tripleo07:02
*** mcornea has joined #tripleo07:07
*** tesseract- has joined #tripleo07:08
*** shardy has joined #tripleo07:08
*** jtomasek has joined #tripleo07:10
*** anshul has quit IRC07:10
*** dprince has joined #tripleo07:13
shardyd0ugal: hey, can you check https://bugs.launchpad.net/tripleo/+bug/1622720 please?07:16
openstackLaunchpad bug 1622720 in tripleo "deploying an overcloud a second time after a failure, doesn't find overcloud.yaml" [Undecided,New]07:16
shardylooks like we don't re-run the jinja templating action when updating the plan07:16
jaosoriorshardy: I actually ran into that yesterday07:16
jaosoriorshardy: also have been seeing a bunch of issues with swift not being able to find overcloud-without-mergepy.yaml07:17
*** jbadiapa has joined #tripleo07:18
*** ccamacho has joined #tripleo07:19
shardyjaosorior: Yeah, the problem is we're not always running the ProcessTemplatesAction I think, which means only overcloud.j2.yaml (not overcloud.yaml) exists in the plan07:21
shardythe CreatePlan workflow includes that action, but I suspect the update plan one doesn't07:22
*** shardy is now known as shardy_mtg07:22
*** anshul has joined #tripleo07:23
*** ifarkas_afk is now known as ifarkas07:24
*** ebarrera has joined #tripleo07:26
*** jpena|off is now known as jpena07:37
d0ugalshardy_mtg: Yeah, I opened another bug for that07:37
openstackgerritCarlos Camacho proposed openstack/puppet-tripleo: Add update puppet-lint to check puppet-tripleo manifests  https://review.openstack.org/35284707:38
jaosoriord0ugal: do you know what uses zaqar queues besides mistral?07:39
d0ugaljaosorior: no, nothing that I am aware of.07:40
jaosorioralright07:40
*** dciabrin|away is now known as dciabrin07:41
*** b00tcat has quit IRC07:42
*** zoli_gone-proxy is now known as zoliXXL07:44
*** dsneddon has joined #tripleo07:44
*** dsariel has quit IRC07:46
*** dbecker has quit IRC07:48
*** aufi has joined #tripleo07:48
*** pmannidi has quit IRC07:48
*** openstackgerrit has quit IRC07:48
*** openstackgerrit has joined #tripleo07:49
*** dtantsur|afk is now known as dtantsur07:53
openstackgerritDougal Matthews proposed openstack/tripleo-common: Add process templates to the update_deployment_plan workflow  https://review.openstack.org/36920207:55
*** ohamada has joined #tripleo07:55
openstackgerritDougal Matthews proposed openstack/tripleo-common: Add process templates to the update_deployment_plan workflow  https://review.openstack.org/36920207:56
*** hjensas has joined #tripleo07:56
*** hjensas has joined #tripleo07:56
*** jpich has joined #tripleo08:03
d0ugalshardy_mtg, jaosorior: https://review.openstack.org/#/c/36920208:04
d0ugalJust testing this.08:04
jaosoriord0ugal: dude...08:05
jaosoriorI'm running into a VERY funky issue now08:05
*** mbound has joined #tripleo08:05
jaosoriormistral is failing executions because of this08:05
jaosoriorZaqarAction.queue_post failed: <class 'zaqarclient.transport.errors.MalformedRequest'>: Error response from Zaqar. Code: 400. Title: Invalid API request. Description: Message collection size is too large. Max size 524288.08:05
d0ugaljaosorior: hey, stop finding new issues I am still trying to fix the previous one :P08:06
*** dsariel has joined #tripleo08:07
d0ugaljaosorior: That is weird.08:07
d0ugaljaosorior: Failing all executions?08:07
jaosoriord0ugal: no08:08
jaosoriord0ugal: only the overcloud deploy08:08
jaosoriorintrospection succeeds08:08
d0ugalI wonder why your message is so big.08:08
jaosoriord0ugal: well, it might be cause I have a lot of environments that I'm passing08:08
d0ugalThere was a patch to increase the upper limit, but I forgot what happened to it.08:08
jaosoriorgonna try with a simple one08:08
d0ugaljaosorior: They shouldn't be passed to zaqar tho'08:08
jaosorioralso I'm seeing mongodb errors in zaqar08:08
d0ugaloh, that is weird.08:09
jaosoriorInvalidDocument: key 'https://172.29.172.2:13808/v1/AUTH_91630ce28aae44fdb89cb906e5bac785/overcloud/puppet/services/nova-consoleauth.yaml' must not contain '.'08:09
*** b00tcat has joined #tripleo08:09
d0ugalI have never seen a mongo error - zaqar uses mongo?08:09
jaosoriord0ugal: yes, heavily08:09
d0ugallol08:09
jaosoriorand mistral forces stuff to use the public endpoint haha08:10
jaosoriorbut I'm fixing that one at the moment08:10
d0ugalhrm08:10
d0ugalcool08:10
*** zoliXXL is now known as zoli|trng08:10
d0ugaljaosorior: Actually, nevermind - it wasn't a Zaqar limit I had seen before it was a Mistral execution input08:11
d0ugalbut we changed things08:11
d0ugalso you are in totally new ground again.08:11
jpichThere was a patch merged recently to raise the Zaqar limit -> https://review.openstack.org/#/c/368197/08:11
jaosoriord0ugal: all I wanted was to get TLS everywhere working :(08:11
jaosorior(for haproxy)08:12
jaosoriorjpich: nice!08:12
jaosoriorlet me try that out08:12
d0ugaljaosorior: Based on the error you have, it seems you don't have that new limit08:12
jpichHopefully that helps!08:12
d0ugaljaosorior: so I guess you are hitting whatever slagle hit?08:12
d0ugaljpich: Thanks, it does!08:13
d0ugalor, it should?08:13
jpichd0ugal: Exactly my thoughts ;)08:14
*** limao has quit IRC08:14
*** sshnaidm|pto is now known as sshnaidm08:16
jaosoriorawww yes08:16
jaosoriorthat seems to have worked08:16
d0ugal\o/08:16
jaosoriorthe deploy is on-going08:16
jaosoriorbut08:16
jaosoriorthere is an execution that failed08:16
jaosoriorfor some reason08:16
jaosoriorthough it seems like a non-fatal error08:17
jaosorioryeah08:17
jaosoriorcreate_deployment_plan failed08:17
jaosoriorbut deploy_plan succeeded08:17
jaosoriord0ugal: any idea what's up?08:17
*** limao has joined #tripleo08:18
*** lmiccini_ has joined #tripleo08:19
d0ugaljaosorior: I'd need to see the details08:19
d0ugaljaosorior: mistral execution-get $ID08:19
jaosoriorerror message says Failure caused by error in tasks: notify_zaqar08:19
jaosorioraaand the logs say  ZaqarAction.queue_post failed: <class 'zaqarclient.transport.errors.ServiceUnavailableError'>: Error response from Zaqar. Code: 503. Title: Service temporarily unavailable. Description: Messages could not be en08:20
jaosoriorqueued. Please try again in a few seconds..08:20
jaosoriorwhat the hell08:20
*** lmiccini has quit IRC08:20
jaosorioryeah, and zaqar seems to be failing with the mongodb error08:20
jaosoriorInvalidDocument: key 'http://172.29.172.1:8080/v1/AUTH_91630ce28aae44fdb89cb906e5bac785/overcloud/puppet/services/keystone.yaml' must not contain '.'08:20
d0ugaljaosorior: dang, i feel like something is wrong with your Zaqar/Mongo08:20
jaosoriorwell, looking at the documentation08:21
jaosoriorit seems that it's mongodb's default behavior not to take periods as keys08:21
jaosoriorso in that case, I'm surprised how it worked earlier08:21
d0ugalYeah, that is weird.08:21
*** electrofelix has joined #tripleo08:22
d0ugaljaosorior: I don't understand why zaqar is making a document with that key?08:22
d0ugaljaosorior: That looks like the swift url08:22
d0ugalsomething very strange is going on08:22
d0ugalbrb08:23
*** lucas-dinner is now known as lucasagomes08:23
*** bkopilov__ has quit IRC08:24
*** bkopilov_ has quit IRC08:24
*** lmiccini_ is now known as lmiccini08:25
*** dsneddon has quit IRC08:29
openstackgerritImre Farkas proposed openstack/tripleo-common: Default to Ironic API v1.15  https://review.openstack.org/36431908:30
*** zigo_ is now known as zigo08:34
*** dbecker has joined #tripleo08:37
skramajashardy_mtg: there is an issue in os-net-config for DPDK interface binding. i have raised bug - https://bugs.launchpad.net/os-net-config/+bug/161933008:38
openstackLaunchpad bug 1619330 in os-net-config "os-net-config fails on 2nd run with DPDK net config" [Undecided,In progress] - Assigned to Saravanan KR (skramaja)08:38
skramajashardy_mtg: and the corresponding review is - https://review.openstack.org/#/c/364354/, could you comment on the approach?08:38
*** milan has joined #tripleo08:40
*** dsneddon has joined #tripleo08:40
*** jistr is now known as jistr|mtgs08:43
jaosoriord0ugal: I don't understand either... I honestly don't even know what the relation from swift to zaqar is08:44
thervejaosorior, *presumably*, you're uploading template info into Zaqar. Templates can contain switf temp URLs.08:47
jaosoriorwell that makes sense08:47
jaosoriorand the swift temp URL will most likely contain some path to the file08:48
jaosoriorwhich may contain periods08:48
jaosorioraaand that's not actually supported by zaqar08:48
jaosoriorwell that's a bummer08:48
d0ugaltherve, jaosorior - the only template related messages that should get sent to Zaqar are responses from Heat AFAIK.08:52
d0ugalbut I have no idea why Zaqar would use that contents as a key in Mongo. That makes no sense to me.08:52
*** dtantsur is now known as dtantsur|bbl08:53
openstackgerritDougal Matthews proposed openstack/instack-undercloud: Verify that the Deployment Plan creation was successful  https://review.openstack.org/36924708:53
openstackgerritDougal Matthews proposed openstack/instack-undercloud: Verify that the Deployment Plan creation was successful  https://review.openstack.org/36924708:54
therved0ugal, zaqar directly store messages as mongo document, AFAIK08:57
*** saneax is now known as saneax-_-|AFK08:57
openstackgerritDougal Matthews proposed openstack/tripleo-common: Add template processing to the update plan workflow.  https://review.openstack.org/36876008:58
*** hjensas has quit IRC09:00
openstackgerritDougal Matthews proposed openstack/instack-undercloud: Verify that the Deployment Plan creation was successful  https://review.openstack.org/36924709:00
openstackgerritCarlos Camacho proposed openstack/puppet-tripleo: Add update puppet-lint to check puppet-tripleo manifests  https://review.openstack.org/35284709:04
therved0ugal, jaosorior: I have a workaround if you're blocked09:04
*** skramaja has quit IRC09:04
jaosoriortherve: I am interested in that workaround09:05
jaosoriorwas gonna start working on a zaqar fix09:05
jaosoriortherve: I'm gonna go for lunch, but I'll check it out when I'm back09:05
*** jaosorior is now known as jaosorior_lunch09:05
thervejaosorior_lunch, http://paste.openstack.org/show/573542/09:05
d0ugalI am interested to, it might lead me to a fix :)09:06
d0ugaltherve: thanks09:06
d0ugalhah, that I didn't expect :)09:06
jaosorior_lunchtherve: lol, I was gonna do basically that, but for all of mongodb's operations in the zaqar code-base09:06
jaosorior_lunchwell, adding a config option for that09:06
*** skramaja has joined #tripleo09:06
jaosorior_lunchanyway, that's great09:06
therveKinda :)09:07
therveI think the real fix is to store the message as a blob09:07
therveI don't think we support query operations on it anyway09:07
EmilienMGood morning09:08
d0ugalEmilienM: Morning09:08
d0ugalEmilienM: early start?09:08
EmilienMYeah busy week09:08
d0ugalCan't imagine why :)09:09
EmilienMHA job looks red, I'll look09:09
*** hjensas has joined #tripleo09:11
*** hjensas has quit IRC09:11
*** hjensas has joined #tripleo09:11
*** paramite has joined #tripleo09:15
d0ugaljaosorior_lunch, therve: FWIW, I am now hitting that issue. So I'll try this workaround :)09:18
sshnaidmEmilienM, hi, I see all ha jobs fail from yesterday09:19
EmilienMsshnaidm: I know09:20
EmilienMI thought we fixed it09:20
EmilienMbut I still see the galera problem09:20
*** chem has joined #tripleo09:21
EmilienMwait I think I know why09:21
EmilienMwe have galera-25.3.12-2.el7.x86_64 but we should not since we don't deploy EPEL09:22
EmilienMsshnaidm: look, we had it working here: https://review.openstack.org/#/c/347499/09:22
*** saneax-_-|AFK is now known as saneax09:23
*** fragatina has joined #tripleo09:26
openstackgerritEmilien Macchi proposed openstack/python-tripleoclient: Do not include epel element when building overcloud images  https://review.openstack.org/36927609:26
EmilienMI think we'll need this backport ^09:26
*** limao has quit IRC09:28
*** fragatina has quit IRC09:30
EmilienMd0ugal, therve: sorry for stupid question but are you guys talking about CI failures with mistral/zaqar?09:30
EmilienMhttp://logs.openstack.org/66/364966/2/check/gate-tripleo-ci-centos-7-nonha-multinode/b2226a9/console.html#_2016-09-13_02_46_50_62174409:30
EmilienMis it what you're talking about? ^09:31
therveEmilienM, Possibly09:32
therveEmilienM, Is it tripleo master? I don't see how it could have passed the gate in the fist place09:33
EmilienMI see a high percent of failing CI jobs09:33
EmilienMtherve: yes09:33
therveEmilienM, http://logs.openstack.org/66/364966/2/check/gate-tripleo-ci-centos-7-nonha-multinode/b2226a9/logs/var/log/zaqar/zaqar.txt.gz is the error we're talking about, but not sure it's the root cause09:33
EmilienMstable/newton09:33
*** r-mibu has quit IRC09:34
*** derekh has joined #tripleo09:34
EmilienMbut it's master for most of projects except tripleoclient09:34
*** r-mibu has joined #tripleo09:34
thervestable/newton? Is that a thing already?09:34
EmilienMwe didn't branch others yet09:34
EmilienMtherve: for clients, yes.09:34
therveAh09:34
*** dbecker has quit IRC09:34
therveEmilienM, If I propose a zaqar patch can you check if it fixes the issue?09:34
EmilienMtherve: yes09:35
EmilienMlet me check if zaqar Ci has tripleo jobs otherwise we can use Depends-On09:35
EmilienMso 1) submit your patch in Zaqar 2) send a dummy patch in tripleo-ci and use Depends-On the zaqar patch09:36
EmilienMit should work09:36
therveEmilienM, https://review.openstack.org/36923309:36
therveNo09:36
therveEmilienM, https://review.openstack.org/369285 this one09:37
EmilienMtherve: I can test it, let me do the patch09:37
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: DNM: test https://review.openstack.org/#/c/369285/  https://review.openstack.org/36928909:38
EmilienMtherve: ^ let's see09:38
EmilienMtherve: on another topic, I see puppet CI failed to promote RDO repo last night because of heat timeout09:39
EmilienMhttp://logs.openstack.org/01/369201/1/check/gate-puppet-openstack-integration-3-scenario003-tempest-centos-7/5db05f9/console.html#_2016-09-13_08_37_16_80658609:39
EmilienMhave you already seen that error in CI?09:39
EmilienMlooks like related to this trace: http://logs.openstack.org/01/369201/1/check/gate-puppet-openstack-integration-3-scenario003-tempest-centos-7/5db05f9/logs/heat/heat-engine.txt.gz#_2016-09-13_08_17_09_10009:40
sshnaidmEmilienM, in tripleo ci it passed yesteday: http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-ha-tempest/99df38e/console.html#_2016-09-12_07_54_10_62149009:41
pandaEmilienM: w/r/t epel remove change, should it backported to mitaka and liberty too ? I see liberty HA fail with the same error09:42
pandaEmilienM: oh wait, what are you doing awake ?09:42
EmilienMpanda: yes09:42
d0ugalDo people think we can land https://review.openstack.org/#/c/335460/ with the ha failure?09:42
EmilienMI submitted the patch09:42
* d0ugal really wants to get rid of the warnings on every command09:43
EmilienMsshnaidm: yes I know, we removed EPEL in CI and it was supposed to fix our issue, as job was green09:43
*** saneax is now known as saneax-_-|AFK09:44
pandaEmilienM: ok, filing a bug to keep track of the changes09:44
EmilienMpanda: see https://review.openstack.org/#/c/369276/09:44
therveEmilienM, Sorry I don't know that failure.09:44
EmilienMtherve: np I'll file a bug09:46
*** dbecker has joined #tripleo09:50
derekhEmilienM: looking at the HA problems, do you know yet why the epel removal worked in CI but hasn't worked post merge?09:52
derekhEmilienM: I think I know why09:52
*** tosky has joined #tripleo09:52
EmilienMderekh: whyN09:52
derekhEmilienM: I havn't looked into it much but I'm guessing that EPEL is installed on the cached image09:53
derekhEmilienM: the tripleo-ci patch didn't use the cached image because there was a depends-on for a patch on DIB09:53
EmilienMdciabrin: ^ fyi09:53
EmilienMah09:53
EmilienMderekh: it makes sense09:53
derekhEmilienM: we don't use the cached image in DIB patches09:53
EmilienMhow can we update the cached image?09:53
derekhEmilienM: sshnaidm did the periodic job pass last night? that would update it09:54
sshnaidmderekh, the ones before last passed09:55
derekhsshnaidm: ok, that one still had EPEL09:56
sshnaidm09-1209:56
EmilienMok we need periodic job passing09:56
EmilienMsshnaidm: why did it fail?09:56
sshnaidmEmilienM, last jobs fail because this mysql problem exactly..09:56
EmilienMwhat?09:56
EmilienMat what time09:56
derekhEmilienM: sshnaidm so there are 2 options, 1. get a new cached image with a fake periodic job today   (if we know why it failed) , or 2. disable the image cache until we have another promote09:56
EmilienMderekh: I consider #2 the quickest option09:57
sshnaidmEmilienM, http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-ha/588856a/09:57
EmilienMat this stage of the cycle, we really need CI working09:57
EmilienMsshnaidm: did it have EPEL?09:57
derekhhmmm, this mornings periodic job shouldn't have had EPEL should it?09:58
EmilienMright09:59
pandastupid question, I see the fix is for master and newton, are periodic jobs deploying mitaka and liberty ?09:59
EmilienMderekh: as a backup solution, can you kickoff a patch that disable image caching?09:59
derekhEmilienM: yup09:59
EmilienMthx10:00
EmilienMi'm debugging the periodic job now10:00
sshnaidmas I see they do10:00
*** saneax-_-|AFK is now known as saneax10:00
sshnaidmEmilienM, derekh ^^10:00
sshnaidmEmilienM, derekh overcloud hosts have usual epel repo10:01
sshnaidmthe undercloud has /etc/yum.repos.d/epel.repo.rpmsave10:01
EmilienMsshnaidm: can you run a fake periodic job please?10:02
pandafake ?10:02
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Disable cache  https://review.openstack.org/36930310:02
sshnaidmEmilienM, yes, sure10:02
derekhEmilienM: ^10:03
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: TEST: DONT RECHECK: periodic jobs  https://review.openstack.org/35921510:03
EmilienMthx guys10:03
EmilienMI'm adding more debug in our CI to have repolist10:04
pandaoh, that one10:04
*** jaosorior_lunch is now known as jaosorior10:04
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: get_host_info: get repos list  https://review.openstack.org/36930510:04
EmilienM^ let's see where is EPEL deployed and KILLIT forever :)10:05
pandaare periodic jobs even using cached images ?10:05
jaosoriortherve: so, got any example of storing it as a blob as you mentioned?10:05
jaosoriord0ugal: did it work for you?10:05
derekhpanda: they wouldn't be, thats the confusing thing10:05
pandaderekh: and are they running mitaka and liberty only ?10:06
thervejaosorior, http://paste.openstack.org/show/573555/ is one test I'm making10:06
therveunit tests are blowing up quite spectacularly though10:06
derekhpanda: I'm not sure what you mean?10:06
sshnaidmEmilienM, you have "/etc" folder within hostname.xz file, it's just not flattened as /var/log10:06
EmilienMsshnaidm: right but my patch makes it easy to read in CI10:07
pandaderekh: the periodic jobs, which release are they deploying ?10:07
EmilienMsshnaidm: I haven't looked files yet but will do in a few10:07
pandaderekh: I see that a periodic job ends with -liberty, but the others, are they mitaka ?10:07
derekhpanda: we have peridoc jobs for liberty, mitaka and master10:08
d0ugaljaosorior: no :/ I'm not sure if I am doing something wrong tho'10:08
derekhpanda: only the master job builds an image that we use in the cach though10:08
*** skramaja has quit IRC10:09
EmilienMtherve: re heat issue, it was transient FYI, job passed, we're promoting to trunk.10:12
therveOK10:12
EmilienMtherve: probably timeouting because of running low resources in infra10:13
therveWell presumably a 'real' failure, but we can look at it later10:13
EmilienMtherve: I'll file the bug anyway10:13
EmilienMto keep record10:13
therveThanks10:13
pandaderekh: so, for what I understand, master HA fails because of the cache, mitaka and liberty may fail because the epel removal change was not back ported. Makes sense ?10:14
EmilienMpanda: yes10:14
EmilienMexactly10:14
*** skramaja has joined #tripleo10:15
derekhpanda: yes, but master-ha periodic is the confusing one10:15
pandashould we add a periodic-newton job too now ?10:16
EmilienMpanda: I created it already10:16
derekhwhere did this job come from ? periodic-tripleo-ci-centos-7-promote-ovb-ha10:16
EmilienMhttps://review.openstack.org/#/c/369024/10:16
* derekh is curious10:16
pandaEmilienM: of course, I had project-config 400 commits behind ..10:17
EmilienMpanda: I eat project-config for breakfast10:17
pandadiluted with coffee10:17
EmilienMmore seriously, I had to create the jobs since we're branching very soon10:18
*** thrash|g0ne is now known as thrash10:18
pandaEmilienM: I understand, I asked because I was looking at a very old version of layout10:18
sshnaidmdoes anybody know what is this job for? periodic-tripleo-ci-centos-7-promote-ovb-ha/ - http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-promote-ovb-ha/10:19
sshnaidmpanda, EmilienM derekh ^^10:19
EmilienMno but first run is September 10th10:20
pandasshnaidm: git blams shows slagle put it10:20
derekhsshnaidm: I was wondering the same thing10:20
EmilienMso maybe if we git log project-config we can see what created it?10:20
sshnaidmok, will ask slagle10:20
pandacommit c4f7fcf6f10:20
pandaderekh: I see periodic-tripleo-ci-centos-7-ovb-ha is using rdo-release mitaka, is that normal ?10:23
pandashouldn't this be master ?10:23
EmilienMprobably it use latest GA10:23
pandaok10:23
EmilienMshardy_mtg: you aware about this job?10:23
jaosoriortherve: let me know when you get it working. I can test it out in my deployment10:24
derekhEmilienM: sshnaidm so the reason the periodic job failed is because it used a repository with an old version of DIB , this one http://trunk.rdoproject.org/centos7/4c/93/4c93ad397ef9359797bb80cf930213b0b8028a28_073a7efe/10:24
thervejaosorior, The current patch doesn't work?10:24
derekhEmilienM: sshnaidm panda DIB is from 2016-08-0210:24
EmilienMderekh: ah it's probably infra images are not updated10:24
EmilienMdang10:24
*** lmiccini has quit IRC10:25
EmilienMI go back to sleep10:25
pandaouch10:25
*** ayoung has quit IRC10:25
jaosoriortherve: ah, I haven't tried your current patch10:25
derekhEmilienM: panda sshnaidm so it still has EPEL , thewre is no mention of it being build in the delorean report http://trunk.rdoproject.org/centos7/report.html10:26
jaosoriortherve: you mean the one that gets the keys into blobs?10:26
thervejaosorior, https://review.openstack.org/#/c/369285/ that one10:26
EmilienMso we have no promotion because we haven't latest DIB but we haven't latest DIB because we have no promotion?10:26
*** saneax is now known as saneax-_-|AFK10:27
sshnaidmEmilienM, exactly10:27
EmilienMgreat10:27
derekhEmilienM: I'm not sure why we don't have latest DIB, .... rdo did start only building certain things on releast (instead of on commit), maybe DIB is one of those things10:27
sshnaidmderekh, which version of dib do we need? I'll ask on #rdo10:27
EmilienMmaybe should we add DIB to the list of packages to take from latest consistent10:28
jaosoriortherve: thanks, I'll check that out10:28
EmilienMsshnaidm: no need to ask10:28
sshnaidmEmilienM, it is I think10:28
*** shardy_mtg is now known as shardy10:28
EmilienMyou need the latest patch that remove EPEL10:28
EmilienMmerged yesterday10:28
sshnaidmEmilienM, yeah, but it's not built as I understood from derekh10:29
sshnaidmand this is could be a question to #rdo folks10:30
* derekh has asked in #rdo10:30
*** pkovar has joined #tripleo10:30
EmilienMthere is diskimage-builder-1.18.2-0.20160802165621.e884309.el7.centos.noarch.rpm in trunk10:30
EmilienMJuly 27th !10:31
EmilienMoh dear10:31
*** ramishra has quit IRC10:31
derekhEmilienM: exactly, thats what I'm saying10:31
*** akrivoka has joined #tripleo10:32
EmilienMderekh: cool10:32
EmilienMderekh: /me needs more coffee :P10:32
*** rodrigods has quit IRC10:32
*** rodrigods has joined #tripleo10:32
*** ifarkas has quit IRC10:33
*** ifarkas has joined #tripleo10:33
*** hjensas has quit IRC10:33
pandaso, even if we backport the epel remove change to mitaka and liberty, we'll still need an updated version of DIB there ?10:33
derekhEmilienM: the patch to stop using the cached image should work though, so we can merge that and go from there10:33
*** ramishra has joined #tripleo10:33
EmilienMderekh: i'm not sure, will it pick dib from source?10:34
EmilienMderekh: maybe it worked in tripleo-ci patch because I had depends-on10:34
derekhEmilienM: it will pick DIB from http://trunk.rdoproject.org/centos7/current/delorean.repo10:35
derekhEmilienM: so master, because its on the includepkgs list http://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/scripts/tripleo.sh#n27010:36
derekhEmilienM: yes, the depends-on made it work in CI10:36
EmilienMderekh: current is https://trunk.rdoproject.org/centos7/c1/66/c166e4103d266a446751363e991656ffb9a547da_e74e2f6e/10:36
*** saneax-_-|AFK is now known as saneax10:37
EmilienMand https://trunk.rdoproject.org/centos7/c1/66/c166e4103d266a446751363e991656ffb9a547da_e74e2f6e/ has old DIB10:37
EmilienMiiuc10:37
EmilienMso even with current, we're stucked10:37
jaosoriortherve: that did the trick for me10:37
sshnaidmuntil we have a hack, removing epel mirrors from image10:37
*** ayoung has joined #tripleo10:37
EmilienMsshnaidm: yes!10:38
EmilienMsshnaidm: I'm looking at releasing DIB now10:38
jaosoriortherve: the execution didn't fail, and zaqar didn't crash10:38
jaosoriord0ugal: ^^10:38
EmilienMsshnaidm: can you look at where EPEL is deployed and remove it manually in tripleoCI until we have dib sorted out?10:38
EmilienMif derekh approves this thing10:38
sshnaidmEmilienM, yep, looking10:38
derekhsshnaidm: EmilienM actually we can just make CI allways build DIB10:39
EmilienMderekh: oh right. It would be easier in the meantime10:39
pandafaster10:39
pandaderekh: so, even if we backport the epel remove change to mitaka and liberty, we'll still need an updated version of DIB there ?10:40
*** mburned_out is now known as mburned10:40
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: Remove EPEL mirror manually from image  https://review.openstack.org/36932510:41
derekhpanda: yup I think  so10:41
EmilienMpanda: dib is branchless10:42
sshnaidmEmilienM, derekh  like so? ^^10:42
EmilienMso we only need the tripleoclient patch backported10:42
EmilienMand we're good10:42
EmilienMderekh: DIB release: https://review.openstack.org/36932610:42
pandaEmilienM: and dib released at least10:42
EmilienMderekh: I vote for fastest solution in the meantime, can you submit a patch in tripleo-ci to build dib from source?10:43
derekhsshnaidm: that will remove epel from the cached image, yes but we'll still be using the old DIB10:44
derekhEmilienM: will do10:44
sshnaidmderekh, right10:44
*** dtantsur|bbl is now known as dtantsur10:44
EmilienMtherve: https://bugs.launchpad.net/heat/+bug/162297910:46
openstackLaunchpad bug 1622979 in heat "Stack DELETE IN_PROGRESS: Unhandled error in asynchronous task" [Undecided,New]10:46
EmilienMjaosorior: can you review this backport please? https://review.openstack.org/#/c/369276/10:47
sshnaidmEmilienM, derekh maybe it's time to see which of tripleo related projects are being built regularly, because we download them from last consistent repo that means to provide the latest packages, and in fact it doesn't10:47
*** dixiaoli has quit IRC10:47
d0ugaljaosorior: oh, nice - so it worked for you?10:47
*** flaper87 has joined #tripleo10:48
*** flaper87 has joined #tripleo10:48
d0ugaljaosorior: Great, not sure how I patched it wrong :) I'll try again.10:48
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Always build DIB  https://review.openstack.org/36932910:48
derekhEmilienM: sshnaidm ^10:48
EmilienMsshnaidm: right10:48
sshnaidmEmilienM, derekh I mean those: https://github.com/openstack-infra/tripleo-ci/blob/master/scripts/tripleo.sh#L27010:48
sshnaidmderekh, maybe it's better to combine our patches? if building - use dib, if cached - use my hack10:49
derekhsshnaidm: yes we should, the system we have was put in place back when rdo build all packages per commit, now that they have changed we may need to rethink things10:49
EmilienMderekh: I added more context in a comment, and +210:49
sshnaidmderekh, reenabling image building will increase the time again, and then to disable it again..10:49
EmilienMsshnaidm: I agree. Combining both could be an option.10:50
EmilienMderekh: you ok?10:50
sshnaidmEmilienM, derekh so I'll raise this maybe on the meeting today10:50
derekhsshnaidm: yes it will, you patch might also need to uninstall galera from the cached image and reinstall it after epel is removed ?10:50
*** mhenkel has joined #tripleo10:51
EmilienMsshnaidm: I see no point of waiting for a meeting but if you want10:51
*** saneax is now known as saneax-_-|AFK10:51
sshnaidmderekh, hmm.. is it installed there already? if it has lower version (I'm not sure), it'll just be updated10:52
EmilienMsshnaidm, derekh: feel free to squash your commits, I like the idea10:52
*** hjensas has joined #tripleo10:52
*** hjensas has quit IRC10:52
*** hjensas has joined #tripleo10:52
EmilienMsshnaidm: no, EPEL has higher version10:52
EmilienMand we need a lower version, from RDO10:52
sshnaidmEmilienM, derekh then if it's installed it's better to remove it, of course10:52
EmilienMsshnaidm: go4it10:53
jaosoriord0ugal: yeah, that worked for me10:53
jaosoriord0ugal: now I'm unsure about the workflow though10:53
openstackgerritJulie Pichon proposed openstack/python-tripleoclient: Add missing unit tests for the 'configure' workflows  https://review.openstack.org/36933210:53
sshnaidmEmilienM, maybe it's better to raise it on rdo ML actually.. As I understand we need these packages to be built each commit anyway10:53
jaosoriord0ugal: I guess we don't really do deploy anymore if we want to update10:53
jaosoriorthere is a specific update command in tripleo CLI10:53
d0ugaljaosorior: I don't follow - we don't really do deploy?10:54
sshnaidmderekh, do you update your commit with mine?10:54
jaosoriord0ugal: let me verify10:54
EmilienMsshnaidm: I would rather ask on #rdo if we want to get it solved quickly10:54
jaosoriorwait up10:54
sshnaidmEmilienM, ok10:55
EmilienMjpena, number80 and apevec can help to have a quick reply10:55
derekhsshnaidm: take mine and add it to your, then we'll have two going through CI, if there is a problem with the squash that we havn't thought about we can fall back to the cache disable10:55
jaosoriord0ugal: so initially doing a deploy works10:56
jaosoriorand then if I try to do an overclou deploy again10:57
jaosoriorit fails10:57
sshnaidmderekh, ok10:57
jaosoriorlike this: http://paste.openstack.org/show/573568/10:57
d0ugaljaosorior: Yeah, because there is an issue with updating the plan in swift10:57
*** adarazs is now known as adarazs_lunch10:57
d0ugaldue to the jinja2 changes10:57
d0ugaljaosorior: That is what I am looking into now.10:58
jaosoriorright right10:59
*** ifarkas has quit IRC10:59
openstackgerritDougal Matthews proposed openstack/tripleo-common: Add template processing to the update plan workflow.  https://review.openstack.org/36876010:59
jaosoriord0ugal: sorry about that, got confused10:59
d0ugaljaosorior: no worries - I think ^ will fix it.10:59
d0ugaljaosorior: I am just testing still11:00
d0ugalI also realised that we changed what the workflow for create_plan returned - we need to be careful about that I guess.11:00
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates: Add NetApp Manila driver integration and tidy up generic  https://review.openstack.org/35401911:00
d0ugalrbrady: ^11:00
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: Remove EPEL mirror manually from image  https://review.openstack.org/36932511:02
sshnaidmderekh, EmilienM ^^11:02
*** saneax-_-|AFK is now known as saneax11:02
pandasshnaidm: what reinstalls galera after removing it ?11:03
*** shardy is now known as shardy_mtg11:04
d0ugaljaosorior: yay, okay, I think I have fixed things.11:04
jaosorioryay!11:05
jaosorior:D11:05
EmilienMpanda, sshnaidm: tripleo-puppet-elements should install galera, but are we sure it runs after?11:05
sshnaidmpanda, good question11:05
sshnaidmI'll install it again11:05
derekhsshnaidm: see my comment also11:06
derekhalso is there anything the depends on galera that will also get uninstalled11:07
pandathe removal path is always more problematic11:08
jaosoriord0ugal: so this is what got it working for you, right?11:08
jaosoriorhttps://review.openstack.org/#/c/368760/11:08
sshnaidmderekh, if I install it again, it should be ok I think11:09
sshnaidmit's system package, less problematic than openstack packages11:09
*** dprince has quit IRC11:09
derekhsshnaidm: it depends, try it its the only way we'll find out, uninstalling it might cause other things to be uninstalled, then reinstalling it wont install them back again11:10
pandaI have the same feeling11:10
openstackgerritDougal Matthews proposed openstack/tripleo-common: Add template processing to the update plan workflow.  https://review.openstack.org/36876011:10
*** mhenkel has quit IRC11:13
derekhzoli|trng: sshnaidm: btw, on friday we discovered the bastion was dropping traffic which looks like it was the reason we were weeing all kinds of network problem, we tweaked some conntrack setting and it appears to have fixed the problem11:14
sshnaidmderekh, cool, great news11:15
sshnaidmderekh, do we continue to be after the "bastion"?11:16
derekhsshnaidm: be after the bastion?11:17
jaosorioranybody knows where I can get the logs for the nodes registration?11:18
sshnaidmderekh, sorry, do we still use it?11:18
derekhsshnaidm: yes, all the public traffic goes through it11:18
sshnaidmderekh, so we need also to look there in case of any network problems11:19
derekhsshnaidm: but we have never had to do anything too it11:19
derekhsshnaidm: yup11:19
sshnaidmderekh, is it requirement? can't we just be opened11:20
mariosshardy_mtg: fwiw the jinja roles stuff is pretty cool from https://review.openstack.org/#/c/315679/ I had to rebase https://review.openstack.org/#/c/354019/ to add manila-backends to roles_data.yaml appreciate another pass when you get a chance thanks11:20
derekhsshnaidm: I'm not sure, pabelanger has started a thread on the list about it11:20
pabelangerderekh: sshnaidm: Yes, please comment.  I posted it last week (thursday) I believe.  Personally, I'd like to see it be removed, since it is causing trouble, but need to better understand why it exists.11:23
pabelangerWe should be treating tripleo-test-cloud-rh1 as a public cloud, not private11:23
openstackgerritMerged openstack-infra/tripleo-ci: pingtest: run 'openstack stack failures list' when failure  https://review.openstack.org/36391811:24
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: Remove EPEL mirror manually from image  https://review.openstack.org/36932511:25
*** ccamacho is now known as ccamacho|lunch11:25
sshnaidmderekh, ^^11:25
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: DEBUGGING do not merge  https://review.openstack.org/36935911:26
derekhsshnaidm: +2 hopefully the dependency thing doesn't happen11:27
*** bkopilov has quit IRC11:27
sshnaidmderekh, yeah, I didn't see something special in its deps11:28
derekhsshnaidm: ok11:28
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates: Add NetApp Manila driver integration and tidy up generic  https://review.openstack.org/35401911:29
EmilienMdciabrin: I have a fix coming11:29
akrivokahonza: hey can you comment on rbrady's latest question here? https://review.openstack.org/#/c/33213211:29
sshnaidmEmilienM, derekh now we need to merge both patches?11:29
dciabrinEmilienM, ok. what's the strategy? still removing EPEL or tweaking puppet-tripleo?11:30
*** lucasagomes is now known as lucas-hungry11:30
openstackgerritEmilien Macchi proposed openstack/puppet-tripleo: mysql: only add brackets to mysql_bind_host when needed  https://review.openstack.org/36936911:31
EmilienMdciabrin: ^11:31
EmilienMdciabrin: so is my patch going to work with latest galera?11:31
EmilienMsshnaidm, derekh: I might have a patch that would work to fix the actual root cause (fix galera deployment)11:32
EmilienMhttps://review.openstack.org/#/c/369369/11:32
sshnaidmEmilienM, so we can cached images?11:33
sshnaidms/can/can use/11:33
EmilienMsshnaidm: I'm really unsure if cached images will help, since the version of dib in cache is still old.11:33
EmilienMagain, it passed in my tripleo-ci patch because I had depends-on a DIB patch11:33
sshnaidmEmilienM, yes, but we don't use it if it's cached afaiu11:34
EmilienMso it picked master11:34
*** gfidente|afk is now known as gfidente11:34
*** fultonj has joined #tripleo11:35
dciabrinEmilienM, so I was trying to bootstrap with latest galera-25.3.16-1.el7.x86_64 on a centos 7 (similar to openstack master), and i can confirm that brackets need to be removed. ipv4 and ipv6 bootstrap works iif brackets are gone11:35
EmilienMdciabrin: cool so my patch should help11:36
*** bfournie has quit IRC11:36
EmilienMsshnaidm: let's see if it pass11:36
dciabrinEmilienM, now i want to double check with current version and ensure we don't regress by removing brackets11:37
pandasshnaidm: heh, so in the end, it was the bracket problem ...11:38
EmilienMdciabrin: sure. Let's apevec know11:38
sshnaidmpanda, which one?11:39
*** wfoster has left #tripleo11:40
*** trown|outtypewww is now known as trown11:40
pandaEmilienM: so even if we remove epel, the mysql connection problem will still appear, and has to be fixed by 369369 ?11:42
EmilienMno if we remove EPEL, it won't appear11:43
EmilienMas the version of galera in RDO is old and works for us11:43
EmilienMmy patch in puppet-tripleo helps to move forward with galera and upgrade it11:43
pandaEmilienM: and then we can use EPEL again ?11:43
EmilienMbonus: it will also help tripleo to work if epel is currently enabled11:43
EmilienMpanda: we don't want EPEL11:43
EmilienMEPEL is evil for us11:44
pandabad epel11:44
pandaok11:44
trownya, we have no control over EPEL and no one to complain to if an update in EPEL breaks us11:44
EmilienMpanda: the tl;dr is that RDO provides all we need and we don't need EPEL11:44
EmilienMtrown: good morning sir11:44
trownso even if we fix galera, getting rid of EPEL is still good11:44
EmilienMtrown++11:44
trowngood morning EmilienM11:45
EmilienMtrown: so, tl;dr of this morning, the DIB patch is not taken in account in TripleO CI, because our image have DIB from latest tag11:45
EmilienMtrown: i proposed a new DIB release, but in meantime, sshnaidm and derekh are working on https://review.openstack.org/36932511:46
EmilienMtrown: and https://review.openstack.org/369369 should also help to use latest Galera11:46
*** dprince has joined #tripleo11:47
trownEmilienM: so galera is just upset about brackets in IPv4? they will work for IPv6?11:47
dciabrintrown, i just checked and no they don't work in ipv6 either with new version of galera11:47
EmilienMoh that's sad11:47
trownah, so that is still an issue... we support ipv611:48
EmilienMwe break ipv6 with newest galera11:48
dciabrinEmilienM, trown: but if we remove bracket in ipv6, the cluster bootstraps. so  maybe they aren't needed in the first place?11:48
EmilienMIMHO brackets should always be here for ipv611:49
EmilienMand if it doesn't work with brackets, that's a bug11:49
dciabrinEmilienM, ok so that's an issue then11:49
trownok, so fastest path is still to remove EPEL I think11:49
EmilienMI haven't git blame galera yet to find what commit broke that thing11:49
EmilienMyes11:50
openstackgerritMerged openstack/python-tripleoclient: Replace agent elements with package python-heat-agent-puppet  https://review.openstack.org/36880811:50
honzaakrivoka: yes, will do that now11:51
trownEmilienM: do you know if this is newton only https://review.openstack.org/#/c/366406/11:52
trownEmilienM: we need to change it in tripleo-common yaml too11:52
tbarronmarios: I'll pick up your changes on https://review.openstack.org/354019 and retest today.11:52
tbarronmarios: dumb question about testing: 354019 is THT only, but the rebase is b/c of changes to overcloud.j2 that I need to make sure I pick up, right?11:53
mariostbarron: ack ... just left a comment at https://review.openstack.org/#/c/366760/4 as well which should help11:54
EmilienMtrown: sounds like newton only11:54
mariostbarron: i think i'll also just add that param explicitly puppet-tripleo side too since it seems to be required (see comment there)11:54
jaosoriorthrash: around yet?11:54
mariostbarron: though from what we can see it appears to be set correctly so who knows... but if it lets us progress then so be it11:54
trownEmilienM: k thanks11:55
dciabrinEmilienM, ok so I've tested bootstrapping without brackets in the option and it works for ipv4 and ipv6, both for the old galera-25.3.5-7.el7ost.x86_64 and the new galera-25.3.16-1.el7.x86_64.rpm11:56
mariostbarron: yeah exactly (I think the comment also answers your very valid question about the testing)... you need to pickup the latest version of the templates11:56
mariostbarron: and that review has been rebased today  for the new overcloud. jinja stuff11:56
*** jpich has quit IRC11:57
openstackgerritEmilien Macchi proposed openstack/puppet-tripleo: mysql: never add brackets to mysql_bind_host  https://review.openstack.org/36936911:57
*** pkovar has quit IRC11:58
openstackgerritMerged openstack/python-tripleoclient: Use heatclient poll_for_events in wait_for_stack_ready  https://review.openstack.org/36884411:58
*** fzdarsky has joined #tripleo11:59
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Terminate Zaqar websocket endpoint in HAProxy  https://review.openstack.org/36032912:00
EmilienMgfidente: https://review.openstack.org/369369 FYI12:01
*** jpena is now known as jpena|lunch12:01
thrashjaosorior: yah12:01
EmilienMmcornea: could you test https://review.openstack.org/369369 in ipv6 environment please?12:02
EmilienMmcornea: we're trying to upgrade galera package and we want to make sure we don't break ipv612:02
jaosoriorthrash: dude, so I kept checking the haproxy/websocket issues12:03
mcorneaEmilienM: yep, spinning up a test environment12:03
jaosoriorthrash: and so far I got it working for introspection and deploying the overcloud12:03
jaosoriorthrash: but in CI, for some reason it keeps failing12:03
openstackgerritHonza Pokorny proposed openstack/tripleo-ui: Treat eslint warnings as jenkins failures  https://review.openstack.org/36815212:03
jaosoriorthrash: seems that it has to do with websocket internals... trying to set up a tunnel cause of the proxy and not being allowed to (getting a 403)12:04
*** dsariel has quit IRC12:04
jaosoriorthrash: have you gotten time to check it out? or did you move to working on something else?12:05
thrashjaosorior: moved on...12:05
*** pkovar has joined #tripleo12:06
thrashjaosorior: but I am certainly up to taking another look.12:06
jaosoriorthrash: can you reproduce the bug in CI?12:07
openstackgerritEmilien Macchi proposed openstack/python-tripleoclient: Do not include epel element when building overcloud images  https://review.openstack.org/36927612:07
*** jayg|g0n3 is now known as jayg12:07
thrashjaosorior: I haven't tried..12:09
*** bfournie has joined #tripleo12:11
EmilienMdciabrin: i'm adding more context in the code, i'll reupload a new PS12:11
dciabrinEmilienM, ok super12:11
*** pradk has joined #tripleo12:11
*** pradk has quit IRC12:11
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: Undercloud install with tripleo-quickstart  https://review.openstack.org/35891912:12
*** pradk has joined #tripleo12:12
openstackgerritMerged openstack/python-tripleoclient: Add libffi-dev to bindep.txt  https://review.openstack.org/36880012:13
pandaEmilienM: is there a list of puppet functions that deal with IPv6 somewhere I can use ?12:15
EmilienMpanda: in stdlib12:15
pandaEmilienM: ok12:15
tbarronmarios: just to make sure i understand, are you pushing a new version of https://review.openstack.org/#/c/354014 that I should get before re-testing?12:18
openstackgerritEmilien Macchi proposed openstack/puppet-tripleo: mysql: never add brackets to mysql_bind_host  https://review.openstack.org/36936912:18
EmilienMdciabrin: ^ that's the best I could do now. I'll have to double check with gfidente and mcornea for extended testing12:19
*** chem has quit IRC12:19
dciabrinEmilienM, ok thx12:20
openstackgerritMarios Andreou proposed openstack/puppet-tripleo: Add manila-netapp backend to manila class and tidy up generic  https://review.openstack.org/35401412:20
EmilienMdciabrin: I -2 the patch myself until we actually test it with existing ipv6 deployments12:20
*** saneax is now known as saneax-_-|AFK12:21
dciabrinEmilienM, ok perfect12:21
tbarronmarios: i guess that answered the question :)12:21
EmilienMdciabrin: to sanitize it I would have to write terrible ruby code12:22
EmilienMdciabrin: I prefer good doc so people use valid ipv6 syntax12:22
dciabrinEmilienM, that's fair enough, I agree12:22
EmilienMdciabrin: but I could also write some ruby code that test if the adresse finish by ":" and then I would add brackets12:23
EmilienMdciabrin: I'll think about it today.12:23
*** adarazs_lunch is now known as adarazs12:23
* EmilienM afk 30 min12:23
*** mburned is now known as mburned_out12:23
jaosoriorshadower: hey dude, got time for a review? https://review.openstack.org/#/c/368572/112:23
*** mburned_out is now known as mburned12:23
*** nyechiel has quit IRC12:26
openstackgerritMarios Andreou proposed openstack/puppet-tripleo: Fixup manila-cephfs native backend defaults  https://review.openstack.org/36676012:26
shadowerjaosorior: in a few mins12:26
*** nyechiel has joined #tripleo12:26
openstackgerritAna Krivokapic proposed openstack/tripleo-common: Add node tagging workflow  https://review.openstack.org/33213212:31
*** lucas-hungry is now known as lucasagomes12:34
*** rbrady has quit IRC12:34
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates: IGNORE - testing manila-generic deploy after tidyup  https://review.openstack.org/36286612:34
mariostbarron: k done updating things...please make sure to pickup the puppet-tripleo updates too12:35
mariostbarron: switching for now will revisit let me know how you get on12:35
tbarronmarios: thanks much, i will12:35
*** fzdarsky has quit IRC12:36
*** rbrady has joined #tripleo12:37
*** rbrady has joined #tripleo12:37
ccamacho|lunchHey EmilienM morning :) quick question, do you know if this file (https://github.com/openstack/tripleo-puppet-elements/blob/bb317ab0d37b5048ae9953be7589a2721cabe1e1/elements/puppet-modules/source-repository-puppet-modules)  can work as a reference to check the new upgrade for puppet-lint? Ill check all those repos  "puppet-*"12:37
*** benoit_ has quit IRC12:39
ccamacho|lunchbasically all the ones getting the dependency from https://git.openstack.org/openstack/puppet-openstack_spec_helper12:39
*** ccamacho|lunch is now known as ccamacho12:40
*** egafford has joined #tripleo12:40
*** rlandy has joined #tripleo12:41
jaosoriord0ugal: I pulled in your CR https://review.openstack.org/#/c/368760/ but I still get the same issue :/12:42
jaosoriorObject GET failed: https://172.29.172.2:13808/v1/AUTH_91630ce28aae44fdb89cb906e5bac785/overcloud/overcloud.yaml 404 Not Found  [first 60 chars of response] <html><h1>Not Found</h1><p>The resource could not be found.<12:43
*** fultonj_ has joined #tripleo12:43
*** rhallisey has joined #tripleo12:43
*** jpich has joined #tripleo12:44
*** Goneri has joined #tripleo12:46
*** trown is now known as trown|brb12:47
EmilienMccamacho: not sure that is useful12:47
EmilienMccamacho: you need to run puppet lint for each puppet module12:47
mcornead0ugal: jaosorior I get some similar error when trying to deploy overcloud: Object GET failed: http://192.168.0.1:8080/v1/AUTH_3a4fb51cc9844b4f836f0bb2e41237d4/overcloud/overcloud.yaml 404 Not Found12:48
dtantsurfolks, is it something known:12:48
dtantsurError: /Stage[main]/Rabbitmq/Rabbitmq_plugin[rabbitmq_management]: Could not evaluate: Command is still failing after 180 seconds expired12:48
*** nyechiel has quit IRC12:48
*** pgadiya has quit IRC12:49
*** rhallisey has quit IRC12:49
*** nyechiel has joined #tripleo12:49
openstackgerritDmitry Tantsur proposed openstack/tripleo-docs: [WIP] Document multiple overclouds with shared ctlplane  https://review.openstack.org/36884012:49
dtantsurbfournie, current version ^^^12:49
dtantsurnow fails with rabbitmq error above12:49
ccamachoEmilienM yeahp, this is my list http://paste.openstack.org/show/573600/ need to make a script for that :P12:49
jaosoriormcornea: so currently overcloud update is broken and d0ugal is working on a fix https://review.openstack.org/#/c/368760/512:49
jaosoriorI tried it out, and I had missed initially to update the undercloud with that fix12:50
jaosoriornow I'm trying again12:50
jaosorioraaaaand it failed :(12:50
jaosoriord0ugal: Exception updating plan: The environment is not a valid YAML mapping data type.12:50
*** pkovar has quit IRC12:51
dtantsurbfournie, also, for computes I'm getting Error: Local ip for ovs agent must be set when tunneling is enabled at /etc/puppet/modules/neutron/manifests/agents/ml2/ovs.pp:220 on node stack1-compute1-0.openstackloc12:52
dtantsural12:52
openstackgerritBrad P. Crochet proposed openstack/python-tripleoclient: Remove the rest of a removed path  https://review.openstack.org/36943112:52
thrashEmilienM: ^^^12:52
EmilienMthrash: +212:53
bfourniedtantsur: yes, when you have vxlan tunneling it needs a local IP configured, let me figure out which setting needs to be changed for that. For now we can just disable vxlan tunneling, one sec.12:55
dtantsurbfournie, I'm totally fine with disabling it for now, unless it's easy to fix12:55
*** myoung|afk is now known as myoung12:55
*** saneax-_-|AFK is now known as saneax12:56
*** bkopilov has joined #tripleo12:56
*** lblanchard has joined #tripleo12:57
*** tzumainn has joined #tripleo12:59
*** jaosorior has quit IRC12:59
*** jaosorior has joined #tripleo13:00
*** dbecker has quit IRC13:00
bfourniedtantsur: You can set this under paramater_defaults to disable tunneling while I figure out how to set the local ip in your new config:13:00
bfourniedtantsur: NeutronTunnelTypes: ''13:00
*** trown|brb is now known as trown13:01
openstackgerritBrad P. Crochet proposed openstack-infra/tripleo-ci: debug: Run 'openstack stack failures list' upon overcloud failures  https://review.openstack.org/36943413:01
*** jpena|lunch is now known as jpena13:01
*** rhallisey has joined #tripleo13:02
EmilienMthrash: nice ^13:04
dtantsur++13:05
*** benoit has joined #tripleo13:05
pandaEmilienM: you removed the sanitization step in 369369 ?13:10
*** morazi has joined #tripleo13:10
EmilienMpanda: yes, read my comment in the code, we can't have brackets anyway13:10
EmilienMpanda: because of recent galera version13:10
bfourniedtantsur: You will be fine with disabling vxlan tunneling as above, but if eventually you want to enable it, the local_ip that it needs comes from this setting in the ServiceNetMap:13:14
*** kjw3 has quit IRC13:14
bfourniedtantsur: NeutronTenantNetwork: 'stack1_tenant'13:14
dtantsurhmm, what's wrong with it then?13:14
*** dbecker has joined #tripleo13:15
*** pkovar has joined #tripleo13:15
dtantsurI'd prefer to avoid disabling vxlan eventually, seems too restrictive13:15
bfourniedtantsur: is stack1_tenant set?13:15
dtantsurbfournie, it's TenantNetName13:15
bfourniedtantsur: hmm, not sure, let me look into it more.  That's the value its using for local_ip:13:16
bfourniedtantsur: puppet/services/neutron-ovs-agent.yaml:            neutron::agents::ml2::ovs::local_ip: {get_param: [ServiceNetMap, NeutronTenantNetwork]}13:16
dtantsurthis all is strange...13:17
*** fzdarsky has joined #tripleo13:19
*** anshul has quit IRC13:19
dtantsurbfournie, local ip is fine, but rabbit still fails... I suspect it gets some wrong configuration as well13:24
bfourniedtantsur: yeah, could be related if local_ip wasn't being set correctly13:29
*** saneax is now known as saneax-_-|AFK13:30
dtantsurtripleo networking is pure magic, I have no idea what I am doing...13:30
openstackgerritAna Krivokapic proposed openstack/tripleo-common: Add node tagging workflow  https://review.openstack.org/33213213:30
* social chews on Error: Duplicate declaration: Heat_config[DEFAULT/num_engine_workers] is already declared; cannot redeclare at /etc/puppet/modules/heat/manifests/engine.pp:186 on node overcloud-controller-0.localdomain13:31
bfourniedtantsur: is there any info in logs about rabbit timeouts?13:31
dtantsurlemme dive into local logs13:32
*** cdearborn has joined #tripleo13:32
openstackgerritAttila Darazs proposed openstack/tripleo-quickstart: Set default TERM for tput  https://review.openstack.org/36945513:34
bfourniedtantsur: if you send me machine IP+creds I can also look13:34
mcornead0ugal: jaosorior from what I can see tripleoclient is looking for overcloud.yaml and overcloud-without-mergepy.yaml which are not present in t-h-t anymore: https://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/constants.py#L3413:35
mcorneais there any CR for this?13:35
d0ugalmcornea: I would check tripleoclient is still using that constant, I am not sure it is13:36
*** akshai has joined #tripleo13:36
dtantsurbfournie, http://paste.openstack.org/show/573605/13:36
dtantsurbfournie, creds are a bit hard, this is a quickstart environment13:36
jaosoriormcornea: so this is the CR https://review.openstack.org/#/c/368760/5 but the error seems to be going deeper :/13:36
*** kjw3 has joined #tripleo13:37
mcornead0ugal: https://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/v1/overcloud_deploy.py#L48213:37
dtantsurbfournie, no rabbitmq logs, nothing suspicious in /etc/rabbitmq13:38
*** jobewan has joined #tripleo13:38
mcorneajaosorior: I applied it but get the same swift not found error; should I do anything else except modifying the plan_management.yaml file?13:38
*** benoit has quit IRC13:38
jaosoriormcornea: right, so that's the attempt to fix it, but it still doesn't work on updates13:39
*** nico_auv has joined #tripleo13:39
jaosoriormcornea: and also, one thing is that my environment is pretty modified :/13:39
jaosoriorI have this fix to zaqar too13:39
jaosoriormcornea: https://review.openstack.org/#/c/369285/13:39
jaosoriormcornea: I have that applied manually13:39
bfourniedtantsur: ok, yeah nothing definitive in those logs you sent. Let me look at my quickstart setup13:40
openstackgerritMerged openstack/python-tripleoclient: Update .gitreview for stable/newton  https://review.openstack.org/36496613:40
openstackgerritMerged openstack/tripleo-heat-templates: Enable proxy header parsing for Manila  https://review.openstack.org/36857113:40
dtantsurbfournie, you can reproduce the same thing based on the WIP docs13:40
*** skramaja_ has joined #tripleo13:42
*** benoit has joined #tripleo13:42
pabelangermorning13:42
*** skramaja has quit IRC13:43
pabelangerdprince: slagle: bnemec: derekh: EmilienM: I'd like to loop back to https://review.openstack.org/#/c/357308/ if all possible. Is that something we can land if CI has improved?13:43
*** benoit has quit IRC13:44
*** benoit has joined #tripleo13:44
mcornead0ugal: jaosorior I changed the tripleoclient constants and I get this error so I think that's causing it: Object GET failed: http://192.168.0.1:8080/v1/AUTH_3a4fb51cc9844b4f836f0bb2e41237d4/overcloud/somefile.yaml13:44
slaglepabelanger: i think that's up to dprince13:45
*** flepied has quit IRC13:45
mcornead0ugal: jaosorior  overcloud.yaml → overcloud.j2.yaml https://github.com/openstack/tripleo-heat-templates/commit/0a9b02b60ce366a20d67ff67dfce671f4f46585d13:46
*** skramaja_ is now known as skramaja13:46
d0ugalmcornea: Yeah, I know - but the bit missing is the process templates step13:46
d0ugalI am adding that in, but it is causing another issue13:46
d0ugalmcornea: This is the first pass that adds the process templates action. https://review.openstack.org/#/c/368760/13:46
d0ugalbut that is causing a problem with updating the plan13:47
d0ugalbecause updating a plan and the jinja2 stuff were both merged at the same time - they passed CI individually but broke the world together13:47
d0ugalmcornea: From what I understand, overcloud.yaml will exist, after the process templates step has been done.13:48
d0ugalso the file isn't in github, but should be generated.13:48
*** rhallisey is now known as rhallisey-brb13:48
EmilienMccamacho: you need to loop over all puppet modules and run lint13:50
EmilienMccamacho: there is no other way to test such a change13:50
EmilienMccamacho: so what you can do, is to submit a patch on top of puppet-openstack_spec_helper that loop over all modules and run lint tests, instead of just puppet nova13:51
shardy_mtgEmilienM: Hey, can you run the meeting today please, I'm in another meeting so won't be able to focus completely on it13:52
EmilienMtripleo meeting in 9 minutes, please add off items if needed: https://etherpad.openstack.org/p/tripleo-meeting-items13:52
EmilienMshardy_mtg: yup, will do13:52
shardy_mtgthanks!13:52
*** hjensas has quit IRC13:53
dprincepabelanger: I would suggest we wait until after the release is cut to minimize CI risk for this13:54
dprincepabelanger: so, lets treat https://review.openstack.org/#/c/357308/ as an early Ocata feature13:55
ccamachoEmilienM so if I add a depends-on to every single puppet module it should run lint for all of them, right? Like I did for puppet-nova but a depends-on for each module (Adding an empty line to each module).13:56
EmilienMderekh, sshnaidm, gfidente, trown, dciabrin: pingtest passed for HA job with https://review.openstack.org/#/c/369369/13:56
EmilienMccamacho: no13:56
EmilienMccamacho: it won't work13:56
EmilienMccamacho: because we use bundle to install the gem and bundle doesn't support zuul-cloner13:57
EmilienMso depends-on is useless here13:57
EmilienMthe otherway around work though: you can adds depends-on in spec_helper patch because we clone puppet nova using zuul cloner13:57
EmilienMbut it's useless in our case13:57
ccamachoEmilienM and doing this? https://review.openstack.org/#/c/369445/3/Gemfile13:57
bfourniedtantsur: still working on setting up my quickstart environment which didn't come up cleanly after ram was added to host6…13:58
*** akshai has quit IRC13:58
EmilienMccamacho: I didn't know you could do it13:58
EmilienMccamacho: does it work?13:58
EmilienMtripleo meeting is about to start folks13:58
ccamachowaiting for CI to check if can get the gem reference from gerrit13:58
*** tobias-fiberdata has joined #tripleo13:58
derekhEmilienM: where did it pass, I don;t see it13:58
EmilienMderekh: I'm watching zuul13:59
EmilienMderekh: it will be showed very soon when other jobs pass13:59
*** akshai has joined #tripleo13:59
derekhEmilienM: I'm watching it in zuul too, its only started14:00
dciabrinEmilienM, have you seen my comments on https://review.openstack.org/#/c/369369/3 ? i think we're good with any kind of ipv6 addresses14:00
derekhEmilienM: this one has passed though https://review.openstack.org/#/c/369329/14:00
EmilienMderekh: ok i'll catch up after weekly meeting14:00
pabelangerdprince: I can understand being reluctant about making CI changes, but I don't believe waiting until a future date is the right move in this case. Yes, I know triple CI is pretty touchy these days, but we should be iterating forward in small increments.  But pushing these changes out until Ocata, we're basically blocking CI improvements14:00
derekhEmilienM: ok14:00
EmilienMtripleo meeting is starting14:00
EmilienMdciabrin: it didn't pass HA job yet14:00
*** tobias_fiberdata has quit IRC14:01
*** apetrich has quit IRC14:02
*** oshvartz has quit IRC14:03
*** apetrich has joined #tripleo14:04
*** yamahata has joined #tripleo14:08
*** shardy_mtg is now known as shardy14:16
*** dbecker has quit IRC14:18
*** limao has joined #tripleo14:19
*** jprovazn has quit IRC14:19
honzaI keep getting timeout errors when running "openstack overcloud deploy" even when setting --timeout 30 (minutes) --- what can I look for possible issues?14:19
*** dtantsur is now known as dtantsur|mtg14:20
d0ugalhuh, I'm sure Heat used to allow me to do this: http://paste.openstack.org/show/573620/14:22
*** rhallisey-brb is now known as rhallisey14:26
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Remove the environments from Mistral when removing from Swift  https://review.openstack.org/36948614:26
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Remove the environments from Mistral when removing from Swift  https://review.openstack.org/36948614:27
*** saneax-_-|AFK is now known as saneax14:27
jpichhonza: There's been a couple of issues around timeouts lately, do you get a more specific error message like what component is timing out?14:29
honzajpich: unfortunately not, it simply says "ERROR: Timed out waiting for a reply to message ID <id>"14:29
d0ugalhonza: Are the baremetal machines/vm's powering on?14:29
*** tobias-fiberdata has quit IRC14:30
honzad0ugal: welp, virsh list --all fails to connect, no socket14:31
*** fultonj has quit IRC14:31
d0ugalhonza: as the stack user?14:31
d0ugalhonza: I think you might get that error if you are root14:32
*** fultonj_ is now known as fultonj14:32
honzaah14:32
honzait's empty as the stack user14:32
*** tobias-fiberdata has joined #tripleo14:33
*** nico_auv has quit IRC14:33
honzaugh, ignore me, in the wrong place14:33
honzai have 2 baremetal machines running14:33
d0ugalhonza: I've had timeout's before when waiting for the VM's to start, so that was my only guess14:35
d0ugalhonza: btw --timeout defaults to 60, so with 30 you are reducing it :)14:35
d0ugalSorry, it defaults to 24014:35
*** fzdarsky has quit IRC14:35
honzalolololol so it must be some other thing timing out14:36
d0ugalhonza: Yeah, that timeout is just for the full heat operation.14:36
d0ugalhonza: if you have it in a timedout state, can you list the failures?14:36
*** mhenkel has joined #tripleo14:37
honzad0ugal: not sure how?14:37
d0ugalhonza: openstack stack failures list overcloud14:37
d0ugalI think14:37
d0ugalI have no failures to verify with :P14:37
honzayep that's the one14:38
openstackgerritwes hayutin proposed openstack/tripleo-quickstart: set up quickstart to replace instack-virt-setup  https://review.openstack.org/35808914:38
*** flepied has joined #tripleo14:39
honzahttp://paste.openstack.org/show/573641/ d0ugal14:39
openstackgerritMerged openstack/python-tripleoclient: Do not include epel element when building overcloud images  https://review.openstack.org/36927614:39
d0ugalhonza: I have no idea what it means, but that is your problem :)14:39
honzad0ugal: it was worth a shot :)14:39
d0ugalhonza: You might find people more responsive in here after the tripleo meeting ends14:40
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Remove the environments from Mistral when removing from Swift  https://review.openstack.org/36948614:48
d0ugalrbrady: Thanks for the review, I think we need this workaround for now. tripleoclient already updates the mistral env elsewhere14:49
d0ugalrbrady: Without this everything is broken...14:49
*** shardy is now known as shardy_mtg14:50
*** fzdarsky has joined #tripleo14:51
shadowerhonza: engine went down. Sounds bad. Was that a virtual undercloud? Could possibly be running out of memory14:51
honzashadower: that would explain it14:52
*** florianf has quit IRC14:53
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: WIP: different current repo  https://review.openstack.org/36950614:54
*** dprince has quit IRC14:54
honzashadower: i think i used the defaults, UNDERCLOUD_NODE_MEM=614414:54
d0ugaldprince, shardy_mtg: I sent you both an email ping about a bug if you have time at all this week.14:54
d0ugalhonza: https://github.com/d0ugal/tripleo-util/blob/master/host/restack.sh#L19 :)14:55
sshnaidmslagle, do you know what are these jobs for? periodic-tripleo-ci-centos-7-promote-ovb-ha/  periodic-tripleo-ci-centos-7-promote-ovb-nonha/14:55
shadowerhonza: so I haven't had an issue with the defaults recently but I know folks use 8 or even 12GB just in case. And I did have to do that in the past14:56
openstackgerritJohn Trowbridge proposed openstack-infra/tripleo-ci: Use centos7-master-head DLRN instance for includepkgs repo  https://review.openstack.org/36950914:57
*** florianf has joined #tripleo14:58
*** chem has joined #tripleo14:58
openstackgerritMerged openstack/puppet-tripleo: Enable X-Forwarded-Proto for manila  https://review.openstack.org/36857215:00
*** coolsvap is now known as coolsvap_15:01
slaglesshnaidm: yes, i added them so we could split out the promote jobs from other periodic ha/nonha jobs15:01
slaglesshnaidm: but have not made that change in tripleo-ci yet15:01
slaglesshnaidm: so that only jobs that have "promote" in the name will actually promote15:02
*** jaosorior has quit IRC15:02
openstackgerritJohn Trowbridge proposed openstack/tripleo-docs: Use centos7-master-head DLRN instance for includepkgs repo  https://review.openstack.org/36951715:03
trownderekh: bnemec: EmilienM I think https://review.openstack.org/369509 and https://review.openstack.org/369517 will fix our issue with not getting master of DIB and tripleoclient15:04
*** cdearborn has quit IRC15:04
EmilienMtrown: will look shortly15:05
EmilienMtrown: in puppet meeting now ;)15:05
trownlol, ETOOMANYMEETINGS15:06
EmilienMtrown: well, tripleo and puppet are the 2 meeting I really care by week15:07
EmilienMtrown: the rest is secondary :P15:07
derekhtrown: thanks15:07
*** lblanchard has quit IRC15:08
*** lblanchard has joined #tripleo15:09
*** trown has quit IRC15:10
*** dbecker has joined #tripleo15:11
*** limao has quit IRC15:12
*** rlandy has quit IRC15:12
*** dmanchad has quit IRC15:12
*** jayg has quit IRC15:12
*** trown has joined #tripleo15:12
*** jayg has joined #tripleo15:12
*** tremble has quit IRC15:13
*** dmanchad has joined #tripleo15:13
*** rlandy has joined #tripleo15:14
*** oshvartz has joined #tripleo15:16
bnemecpabelanger: So the tripleo user should remain openstack-nodepool?15:18
*** jlinkes has quit IRC15:19
*** ebarrera has quit IRC15:19
EmilienMtrown: oh wow, nice patch15:19
pabelangerbnemec: Ya, that would be used by tripleo things. Like ovb and private infrastructure.15:20
bnemecpabelanger: Okay, and what would the infra users be creating, respectively?  I'm wondering what quota to give each.15:21
*** aufi has quit IRC15:22
pabelangerbnemec: for openstackci, we'll only be launching a single user. For openstackzuul, that will be the quote for the amount of nodes you want to launch from nodepool15:23
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: Add fluentd client service  https://review.openstack.org/35350615:23
*** skramaja has quit IRC15:23
bnemecpabelanger: Thanks, I'll get that set up.15:23
openstackgerritwes hayutin proposed openstack/tripleo-quickstart: set up quickstart to replace instack-virt-setup  https://review.openstack.org/35808915:24
EmilienMmarios: yo15:26
EmilienMmarios: why do we use pacemaker role in https://review.openstack.org/#/c/354014/ ?15:26
pandaEmilienM: https://bugs.launchpad.net/puppet-tripleo/+bug/162309615:28
openstackLaunchpad bug 1623096 in puppet-tripleo "IPv6 handling for swift add_devices" [Undecided,New]15:28
slaglelucasagomes: dtantsur|mtg : if the ssh power driver is not going to be supported any longer in ironic, do you think we should still be using it in tripleo-ci?15:29
dtantsur|mtgslagle, we should slowly move to virtualbmc and pxe_ipmitool15:30
dtantsur|mtgfeel free to wait until our (ironic) CI moves to them though15:30
slagleok yea that was my next question is what is ironic CI using15:30
bnemecErr, we aren't using the ssh power driver in ci anymore, are we?15:31
bnemecovb doesn't anyway15:31
mariosEmilienM: on advice from manila folks manila-share is the only pcmk managed service and it includes the tripleo::profile::pacemaker::manila profile. The backends are then brought up in this single pacemaker profile. We could move them to new file and include that, say  in  https://review.openstack.org/#/c/354019/17/puppet/services/manila-backend-generic.yaml tht side15:31
mariosEmilienM: EmilienM if you would prefer that should be easy enough to make ... thanks for looking15:32
slaglebnemec: good point. though we should probably switch instack-virt-setup over as well15:32
bnemecslagle: Yeah, that would still need to be changed.15:32
dtantsur|mtgslagle, we're in progress of switching away from ssh drivers to virtualbmc15:32
dtantsur|mtgslagle, I'm not sure if it happens in newton15:32
*** dtantsur|mtg is now known as dtantsur15:33
*** gchamoul is now known as gchamoul|afk15:33
*** gchamoul|afk is now known as gchamoul15:33
EmilienMpanda: thans15:34
EmilienMthanks15:34
lucasagomesslagle, as dtmitry said, we currently are using this: https://github.com/openstack/virtualbmc15:35
lucasagomesslagle, which is very similar to ovb15:35
lucasagomesbut instead of nova commands it converts the ipmi commands to libvirt commands and power on/off the vms15:35
EmilienMmarios: it just doesn't follow other services pattern, where each service has its own yaml, ie: nova-api, etc15:36
EmilienMwhen I read the patch, I see all manila is in manifests/profile/pacemaker/manila.pp15:36
trownlucasagomes: do you know if that still supports qemu://session connections to libvirt instead of using qemu://system?15:36
*** zoli|trng is now known as zoli|gone15:36
EmilienMmarios: moreover, we don't have any CI running on manila, right?15:37
EmilienMhow is this patch tested?15:37
trownlucasagomes: ah I see a libvirt_uri in the code, so seems so15:37
lucasagomestrown, it does yeah yes15:37
sshnaidmtrown, is this repo mirrored on http://buildlogs.centos.org/? We had a lot of performance issues and jobs timing out because slow trunk.rdo15:37
lucasagomestrown, when you create the vbmc you can specify the URI15:37
lucasagomestrown, slagle https://raw.githubusercontent.com/umago/virtualbmc/master/images/demo.gif15:37
lucasagomesit's a demo gif15:38
*** zoli|gone is now known as zoli_gone-proxy15:38
slaglelucasagomes: just watched it :)15:38
trownlucasagomes: dtantsur is there plans to make rpm packaging for that?15:38
lucasagomesjust pass --libvirt-uri in the "add" command15:38
lucasagomestrown, yes we should15:38
lucasagomesthere's a BZ for it already lemme find15:38
*** shardy_mtg has quit IRC15:38
lucasagomestrown, https://bugzilla.redhat.com/show_bug.cgi?id=137346815:39
openstackbugzilla.redhat.com bug 1373468 in rhel-osp-director "[RFE] Switch from SSH drivers to virtualbmc" [Unspecified,New] - Assigned to lmartins15:39
mariosEmilienM: we do have base profiles for manila api/sched/share... this is just for the backends, but np i can revisit to move those out as i said15:39
mariosEmilienM: i mean we have separate files for those15:39
lucasagomestrown, but there's no RPM spec for it yet :-/15:39
*** dsneddon has quit IRC15:40
trownlucasagomes: k, seems pretty simple to package... we probably will have to wait for Ocata if it is not started yet though15:40
EmilienMmarios: no worries15:40
EmilienMmarios: we can iterate later, let me finish the review15:40
mariosEmilienM: yeah no ci for that, we are mainly relying on tbarron testing ... we *can* test manila-generic like https://review.openstack.org/#/c/362866/ but netapp no and i assume neither for manila-cephfs15:41
EmilienMmarios: did you manually tested it?15:41
lucasagomestrown, yeah, I think ocata starts on ~ Sept 2015:41
*** dsneddon has joined #tripleo15:41
*** athomas has quit IRC15:41
mariosEmilienM: tbarron is manually testing - my dev-env is used full time setup for 9..10 upgrades as that has to be a priority right now for me at least...  waiting on ram for my beaker box so i can getn another going15:42
trownlucasagomes: but I am willing to help get it packaged early in ocata cycle (since I want to get tripleo-quickstart switched to it ASAP)15:42
bnemecpabelanger: Okay, the users and projects are created.  Is there a good place for me to drop the credentials for you?15:42
lucasagomestrown, oh that would be great actually15:43
*** mhenkel has quit IRC15:43
lucasagomestrown, you want me to create a initial spec ?15:43
tbarronEmilienM: i have a beaker-box that i'm trying to hang on to and am testing marios' changes15:44
mariosEmilienM: OK, appreciate any thoughts on my comment at https://review.openstack.org/#/c/354014/8/manifests/profile/pacemaker/manila.pp15:44
*** jtomasek has quit IRC15:44
mariostbarron: EmilienM thanks very much I will fixup any issues/comments tomorrow /me approaching eod15:44
*** saneax is now known as saneax-_-|AFK15:45
tbarronEmilienM: it would be great to do CI with cephfs backend down the road as that doesn't require special h/w15:45
trownlucasagomes: sure if you have bandwidth, we will need to create a packaging BZ in RDO for it to get it imported in DLRN, but we won't get much traction on that until after RDO ships newton15:45
lucasagomestrown, added to my todo list, I will give it a go this week or next one tops15:45
*** dsneddon has quit IRC15:45
*** egafford has quit IRC15:45
lucasagomestrown, fair enough15:45
tbarronmarios: yeah, i'm going to test your latest changes again after lunch (my time)15:45
EmilienMtbarron: have you seen our tripleo scenario jobs?15:45
tbarronmarios: thanks, and have a good evening15:45
EmilienMtbarron: we could add the services to one of our scenarios15:46
tbarronEmilienM: only vaguely, i'm honestly kindofa n00b on OOO15:46
tbarronEmilienM: am willing to look and learn, but right now we prob have to do manual testing, like for all the cinder env/backends that aren't in CI15:46
EmilienMyes, manual testing is not the right option for us15:47
tbarronEmilienM: our dirction is going to be to deprecate that "generic" backend, as it's useless in production15:47
tbarronEmilienM: so getting cephfs in there instead is the way to go I think15:47
tbarronEmilienM: I don't disagree :)15:47
tbarronEmilienM: it's all that I can do atm, and for backend that requires h/w15:48
*** ccamacho is now known as ccamacho|afk15:50
*** jcoufal has quit IRC15:51
*** jlinkes has joined #tripleo15:52
EmilienMsshnaidm, derekh: should we abandon https://review.openstack.org/#/c/369329/ and https://review.openstack.org/#/c/369303/ ?15:53
openstackgerritMerged openstack-infra/tripleo-ci: Remove EPEL mirror manually from image  https://review.openstack.org/36932515:53
sshnaidmEmilienM, I hope so15:53
mariosthanks EmilienM !15:53
derekhEmilienM: yup15:53
*** jlinkes_ has joined #tripleo15:54
*** mcornea has quit IRC15:54
openstackgerritJohn Trowbridge proposed openstack-infra/tripleo-ci: Use centos7-master-head DLRN instance for includepkgs repo  https://review.openstack.org/36950915:55
*** akshai has quit IRC15:56
*** jistr|mtgs is now known as jistr15:57
openstackgerritMerged openstack/tripleo-quickstart: Add ability to use overcloud-full.qcow2 for undercloud image  https://review.openstack.org/36881015:57
*** jlinkes has quit IRC15:57
sshnaidmtrown, I think you need to revert my patch to check this repo15:58
*** mhenkel has joined #tripleo15:58
*** akshai has joined #tripleo15:59
rbradyhonza: fwiw I have the same issue atm on an undercloud I built last week.15:59
EmilienMslagle: FYI https://review.openstack.org/#/c/314485/ is re-open16:00
trownsshnaidm: ?16:01
*** dtantsur is now known as dtantsur|afk16:02
openstackgerritBrad P. Crochet proposed openstack/tripleo-common: Replace agent elements with package python-heat-agent-puppet  https://review.openstack.org/36955716:04
thrashEmilienM: this updates the code used for tripleo-build-images ^^^^16:04
thrashEmilienM: which I *think* is what all the build systems use now.16:05
trownthrash: I think tripleo-ci does not yet16:05
thrashtrown: right... except for tripleo-ci16:06
trownthrash: there is a patch for it, but it is not merged16:06
thrashtrown: https://review.openstack.org/#/c/336312/16:06
thrash:)16:06
thrashanyway, my new patch just syncs the two methods16:06
trownthrash: yep, meant to put that up yesterday when I saw the tripleoclient patch16:07
EmilienMthrash: ok will look16:07
trownjust forgot16:07
thrashEmilienM: thanks16:08
*** ohamada has quit IRC16:09
openstackgerritEmilien Macchi proposed openstack/python-tripleoclient: Do not include epel element when building overcloud images  https://review.openstack.org/36955916:12
openstackgerritEmilien Macchi proposed openstack/python-tripleoclient: Do not include epel element when building overcloud images  https://review.openstack.org/36955916:13
slagleEmilienM: liberty is not EOL'd is it?16:14
EmilienMslagle: not yet but soon afik16:15
EmilienMslagle: maybe we can have the tag16:16
*** florianf has quit IRC16:17
* EmilienM afk lunch16:17
*** trown is now known as trown|lunch16:18
*** fzdarsky is now known as fzdarsky|lunch16:19
*** akshai has quit IRC16:19
*** akshai has joined #tripleo16:21
*** akshai has quit IRC16:25
*** akshai has joined #tripleo16:26
*** ebarrera has joined #tripleo16:26
*** fragatina has joined #tripleo16:28
honzarbrady: yay, i'm not doing something super stupid16:29
*** myoung is now known as myoung|lunch16:29
*** milan has quit IRC16:31
*** rasca has quit IRC16:32
*** fragatina has quit IRC16:32
*** derekh has quit IRC16:32
*** fragatina has joined #tripleo16:33
rbradyshardy: in the jinja templating patch series, did you intend for the jinja templating of roles to be optional or required?16:40
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: TEST: DONT RECHECK: periodic jobs  https://review.openstack.org/35921516:40
openstackgerritMerged openstack/tripleo-common: Set Deployment Parameters  https://review.openstack.org/36562516:43
openstackgerritGabriele Cerami proposed openstack/tripleo-quickstart: Revert to tripleo image definition in a separate release file for devmode  https://review.openstack.org/36957116:44
*** myoung|lunch is now known as myoung16:44
*** fragatina has quit IRC16:45
*** fragatina has joined #tripleo16:45
*** tesseract- has quit IRC16:46
*** egafford has joined #tripleo16:47
*** penick has joined #tripleo16:48
*** mhenkel has left #tripleo16:49
*** liverpooler has quit IRC16:50
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-heat-templates: M/N upgrade sahara-api fails to restart.  https://review.openstack.org/35802216:50
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Fix deploy timestamp parsing  https://review.openstack.org/36957716:51
*** kjw3 has quit IRC16:53
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-heat-templates: M/N upgrade sahara-api fails to restart.  https://review.openstack.org/35802216:53
*** electrofelix has quit IRC16:55
*** lucasagomes is now known as lucas-dinner16:56
*** yamahata has quit IRC16:58
*** pcaruana has quit IRC16:59
*** nyechiel has quit IRC17:01
*** dbecker has quit IRC17:01
*** rcernin has quit IRC17:03
*** karthiks has quit IRC17:07
*** gfidente is now known as gfidente|afk17:07
*** panda is now known as panda|dinner17:09
*** fzdarsky|lunch is now known as fzdarsky17:10
*** flepied has quit IRC17:12
*** nyechiel has joined #tripleo17:12
-openstackstatus- NOTICE: The Gerrit service on review.openstack.org is being restarted now to address current performance problems, but should return to a working state within a few minutes17:12
*** fragatina has quit IRC17:13
*** jpena is now known as jpena|off17:15
bnemecAnybody know why my deployment is exploding if I pass a path to --templates: http://paste.openstack.org/show/574431/17:15
*** abregman has joined #tripleo17:18
*** nyechiel has quit IRC17:18
*** bana_k has joined #tripleo17:20
slaglewhat? no one deploys from directories any more17:20
slaglei dont really know :)17:21
*** kjw3 has joined #tripleo17:22
openstackgerritBrad P. Crochet proposed openstack/tripleo-common: Add FS_TYPE xfs to RHEL and CentOS images  https://review.openstack.org/36960017:22
jpichbnemec: It's been reported at https://bugs.launchpad.net/tripleo/+bug/1623086 , not sure if it might be a duplicate of https://bugs.launchpad.net/tripleo/+bug/162268317:27
openstackLaunchpad bug 1623086 in tripleo "'Result' object has no attribute 'copy' - overcloud deployment fails" [Undecided,New]17:27
openstackLaunchpad bug 1622683 in tripleo "Updating plans breaks deployment" [Critical,In progress] - Assigned to Dougal Matthews (d0ugal)17:27
bnemecjpich: Okay, thanks.17:30
*** pkovar has quit IRC17:30
*** ebarrera has quit IRC17:30
bnemecIn the meantime I guess I do the naughty thing and mess with packaged files. :-)17:31
*** trown|lunch is now known as trown17:35
*** weshay is now known as weshay_brb17:37
openstackgerritBrad P. Crochet proposed openstack-infra/tripleo-ci: Use tripleo-build-images for CI  https://review.openstack.org/33631217:38
*** yamahata has joined #tripleo17:38
EmilienMslagle: /me back from lunch. For the tag, yes we didn't follow it during Liberty but maybe we can try again now and show we respected the rules during 2 cycles and we want to continue17:39
*** mbound has quit IRC17:42
*** flepied has joined #tripleo17:55
*** flepied1 has joined #tripleo17:56
*** weshay_brb is now known as weshay17:58
*** jobewan is now known as jobewan_away17:59
*** jobewan_away has quit IRC17:59
*** flepied has quit IRC18:00
*** tosky has quit IRC18:05
*** mcornea has joined #tripleo18:07
bnemecOuch.  8 ci jobs per tripleo-ci patch.18:08
bnemecI wonder if we could try bumping the rh1 job capacity a bit now that we're running stable again.18:08
*** fragatina has joined #tripleo18:10
*** panda|dinner is now known as panda18:11
*** jobewan has joined #tripleo18:12
pabelangerEmilienM: https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/tripleo.yaml#L6418:13
pabelangerEmilienM: the job I was talking about18:13
*** rcernin has joined #tripleo18:13
*** jlinkes_ has quit IRC18:14
*** akshai has quit IRC18:17
openstackgerritJulie Pichon proposed openstack/python-tripleoclient: Stop plan creation when container exists  https://review.openstack.org/36962318:22
*** jpich has quit IRC18:23
*** david-lyle has quit IRC18:23
*** david-lyle has joined #tripleo18:26
*** pkovar has joined #tripleo18:27
*** jlinkes has joined #tripleo18:28
*** dtrainor has quit IRC18:31
*** dtrainor has joined #tripleo18:31
*** weshay is now known as weshay_mtg18:31
*** jobewan is now known as jobewan_away18:32
*** dsariel has joined #tripleo18:32
*** jobewan_away has quit IRC18:32
*** penick has quit IRC18:32
*** pcaruana has joined #tripleo18:36
beaglesEmilienM: bnemec: this https://bugs.launchpad.net/tripleo/+bug/1623155 seems to be true... thoughts?18:38
openstackLaunchpad bug 1623155 in tripleo "Neutron L3 HA isn't apparently being enabled anywere" [High,Confirmed]18:38
beaglesugh.. fixed typo in title18:38
*** mhenkel has joined #tripleo18:39
*** mhenkel has quit IRC18:40
*** akshai has joined #tripleo18:40
*** akshai_ has joined #tripleo18:41
*** penick has joined #tripleo18:42
*** akshai has quit IRC18:45
*** dbecker has joined #tripleo18:48
*** masco has quit IRC18:49
*** abregman has quit IRC18:51
openstackgerritMerged openstack/tripleo-quickstart: Add create-docs payload to minimal config  https://review.openstack.org/36899418:57
*** akrivoka has quit IRC18:57
*** rlandy is now known as rlandy|brb19:11
*** itlinux has joined #tripleo19:12
itlinuxhello all, quick question, when installing with TripleO i see swift services, even though I have ceph configured on the backends. Any suggestions why? Thanks19:13
*** weshay_mtg is now known as weshay19:14
*** david-lyle has quit IRC19:16
*** morazi has quit IRC19:16
*** jschlueter has quit IRC19:16
*** pabelanger has quit IRC19:16
*** jefrite has quit IRC19:16
*** Nakato has quit IRC19:16
*** cschwede has quit IRC19:16
*** thrash has quit IRC19:16
*** lynxman has quit IRC19:16
*** dsariel has quit IRC19:16
*** mcornea has quit IRC19:16
*** flepied1 has quit IRC19:16
*** dmanchad has quit IRC19:16
*** pradk has quit IRC19:16
*** noslzzp has quit IRC19:16
*** jpena|off has quit IRC19:16
*** bnemec has quit IRC19:16
*** fpan has quit IRC19:16
*** rwsu has quit IRC19:16
*** Hazelesque has quit IRC19:16
*** ianw has quit IRC19:16
*** zoli_gone-proxy has quit IRC19:16
*** cmyster has quit IRC19:16
*** jmiu has quit IRC19:16
*** itlinux has quit IRC19:17
*** penick has quit IRC19:17
*** trown has quit IRC19:17
*** rhallisey has quit IRC19:17
*** tzumainn has quit IRC19:17
*** fultonj has quit IRC19:17
*** ayoung has quit IRC19:17
*** Ryjedo has quit IRC19:17
*** gfidente|afk has quit IRC19:17
*** davidlenwell has quit IRC19:17
*** hrybacki has quit IRC19:17
*** hogepodge has quit IRC19:17
*** spredzy has quit IRC19:17
*** patrickeast has quit IRC19:17
*** adam_g has quit IRC19:17
*** pleia2 has quit IRC19:17
*** saneax-_-|AFK has quit IRC19:17
*** vkmc has quit IRC19:17
*** tbarron has quit IRC19:17
*** rickflare has quit IRC19:17
*** SpamapS has quit IRC19:17
*** matbu has quit IRC19:17
*** jlinkes has quit IRC19:17
*** jayg has quit IRC19:17
*** fzdarsky has quit IRC19:17
*** rbrady has quit IRC19:17
*** ccamacho|afk has quit IRC19:17
*** zigo has quit IRC19:17
*** nijaba has quit IRC19:17
*** pblaho has quit IRC19:17
*** zephcom has quit IRC19:17
*** HenryG has quit IRC19:17
*** mrunge has quit IRC19:17
*** andreaf has quit IRC19:17
*** mgagne has quit IRC19:17
*** egafford has quit IRC19:17
*** bkopilov has quit IRC19:17
*** b00tcat has quit IRC19:17
*** d0ugal has quit IRC19:17
*** kbyrne has quit IRC19:17
*** lucas-dinner has quit IRC19:17
*** cinerama has quit IRC19:17
*** stevebaker has quit IRC19:17
*** jidar has quit IRC19:17
*** StevenK has quit IRC19:17
*** onovy has quit IRC19:17
*** larstobi has quit IRC19:17
*** ChanServ has quit IRC19:17
*** dtantsur|afk has quit IRC19:17
*** chem has quit IRC19:17
*** apetrich has quit IRC19:17
*** coolsvap_ has quit IRC19:17
*** colonwq has quit IRC19:17
*** mwhahaha has quit IRC19:17
*** timothyb89 has quit IRC19:17
*** igorbelikov has quit IRC19:17
*** dtrainor has quit IRC19:17
*** yamahata has quit IRC19:17
*** bana_k has quit IRC19:17
*** yolanda has quit IRC19:17
*** tdasilva has quit IRC19:17
*** ipsecguy has quit IRC19:17
*** michchap has quit IRC19:17
*** rcernin has quit IRC19:17
*** flaper87 has quit IRC19:17
*** ramishra has quit IRC19:17
*** jistr has quit IRC19:17
*** etp has quit IRC19:17
*** myoung has quit IRC19:17
*** honza has quit IRC19:17
*** fungi has quit IRC19:17
*** NachoDuck has quit IRC19:17
*** lblanchard has quit IRC19:17
*** jbadiapa has quit IRC19:17
*** bandini has quit IRC19:17
*** rhefner has quit IRC19:17
*** mandre has quit IRC19:17
*** zaneb has quit IRC19:17
*** lazy_prince has quit IRC19:17
*** toure has quit IRC19:17
*** slagle has quit IRC19:17
*** markmc has quit IRC19:18
*** rcrit has quit IRC19:18
*** r2bit has quit IRC19:18
*** Goneri has quit IRC19:18
*** cwolferh has quit IRC19:18
*** chlong_ has quit IRC19:18
*** shadower has quit IRC19:18
*** afazekas has quit IRC19:18
*** tobias-fiberdata has quit IRC19:18
*** rodrigods has quit IRC19:18
*** jbcraig has quit IRC19:18
*** dciabrin has quit IRC19:18
*** bvandenh has quit IRC19:18
*** isq has quit IRC19:18
*** rook has quit IRC19:18
*** trozet has quit IRC19:18
*** panda has quit IRC19:18
*** jrist has quit IRC19:18
*** akrzos has quit IRC19:18
*** hewbrocca-afk has quit IRC19:18
*** weshay has quit IRC19:18
*** dmsimard has quit IRC19:18
*** radez has quit IRC19:18
*** sirushti has quit IRC19:18
*** bkero has quit IRC19:18
*** leifmadsen has quit IRC19:18
*** fabbione has quit IRC19:18
*** pcaruana has quit IRC19:18
*** pkovar has quit IRC19:18
*** r-mibu has quit IRC19:18
*** paramite has quit IRC19:18
*** rbowen has quit IRC19:18
*** bswartz has quit IRC19:18
*** CaptTofu has quit IRC19:18
*** gregwork has quit IRC19:18
*** mburned has quit IRC19:18
*** ansiwen has quit IRC19:18
*** sshnaidm has quit IRC19:18
*** dhill_ has quit IRC19:18
*** Ng has quit IRC19:18
*** dobson` has quit IRC19:18
*** NobodyCam has quit IRC19:18
*** crinkle has quit IRC19:18
*** greghaynes has quit IRC19:18
*** fragatina has quit IRC19:18
*** kjw3 has quit IRC19:18
*** oshvartz has quit IRC19:18
*** openstackgerrit has quit IRC19:18
*** social has quit IRC19:18
*** adarazs has quit IRC19:18
*** lifeless has quit IRC19:18
*** bkreitch has quit IRC19:18
*** therve has quit IRC19:18
*** beagles has quit IRC19:18
*** sanjayu|AFK has quit IRC19:18
*** gchamoul has quit IRC19:18
*** marios has quit IRC19:18
*** jroll has quit IRC19:18
*** EmilienM has quit IRC19:18
*** StevenK has joined #tripleo19:23
*** jmiu has joined #tripleo19:23
*** zoli_gone-proxy has joined #tripleo19:23
*** noslzzp has joined #tripleo19:23
*** trown has joined #tripleo19:23
*** dciabrin has joined #tripleo19:23
*** lynxman has joined #tripleo19:23
*** thrash has joined #tripleo19:23
*** cschwede has joined #tripleo19:23
*** Nakato has joined #tripleo19:23
*** jefrite has joined #tripleo19:23
*** pabelanger has joined #tripleo19:23
*** jschlueter has joined #tripleo19:23
*** morazi has joined #tripleo19:23
*** rodrigod` has joined #tripleo19:23
*** dtantsur has joined #tripleo19:23
*** trozet has joined #tripleo19:23
*** isq has joined #tripleo19:23
*** chlong_ has joined #tripleo19:23
*** flepied has joined #tripleo19:23
*** pkovar has joined #tripleo19:23
*** bnemec has joined #tripleo19:23
*** onovy has joined #tripleo19:23
*** sirushti has joined #tripleo19:23
*** sshnaidm has joined #tripleo19:23
*** pcaruana has joined #tripleo19:23
*** weshay has joined #tripleo19:23
*** paramite has joined #tripleo19:23
*** timothyb89 has joined #tripleo19:23
*** mwhahaha has joined #tripleo19:23
*** colonwq has joined #tripleo19:23
*** apetrich has joined #tripleo19:23
*** chem has joined #tripleo19:23
*** Goneri has joined #tripleo19:23
*** dmanchad has joined #tripleo19:23
*** shadower has joined #tripleo19:23
*** bswartz has joined #tripleo19:23
*** fungi has joined #tripleo19:23
*** honza has joined #tripleo19:23
*** myoung has joined #tripleo19:23
*** etp has joined #tripleo19:23
*** jistr has joined #tripleo19:23
*** ramishra has joined #tripleo19:23
*** flaper87 has joined #tripleo19:23
*** rcernin has joined #tripleo19:23
*** thegodfather has joined #tripleo19:23
*** larstobi has joined #tripleo19:23
*** afazekas has joined #tripleo19:23
*** tobias-fiberdata has joined #tripleo19:23
*** jrist has joined #tripleo19:23
*** dsariel has joined #tripleo19:23
*** hewbrocca-afk has joined #tripleo19:23
*** david-lyle has joined #tripleo19:23
*** bkero has joined #tripleo19:23
*** ChanServ has joined #tripleo19:23
*** ianw has joined #tripleo19:23
*** jbcraig has joined #tripleo19:23
*** mcornea has joined #tripleo19:23
*** fpan has joined #tripleo19:23
*** jlinkes has joined #tripleo19:23
*** jayg has joined #tripleo19:23
*** fzdarsky has joined #tripleo19:23
*** rbrady has joined #tripleo19:23
*** ccamacho|afk has joined #tripleo19:23
*** zigo has joined #tripleo19:23
*** nijaba has joined #tripleo19:23
*** pblaho has joined #tripleo19:23
*** zephcom has joined #tripleo19:23
*** HenryG has joined #tripleo19:23
*** mrunge has joined #tripleo19:23
*** andreaf has joined #tripleo19:23
*** mgagne has joined #tripleo19:23
*** lblanchard has joined #tripleo19:23
*** jbadiapa has joined #tripleo19:23
*** bandini has joined #tripleo19:23
*** rhefner has joined #tripleo19:23
*** mandre has joined #tripleo19:23
*** zaneb has joined #tripleo19:23
*** lazy_prince has joined #tripleo19:23
*** toure has joined #tripleo19:23
*** rcrit has joined #tripleo19:23
*** slagle has joined #tripleo19:23
*** markmc has joined #tripleo19:23
*** r2bit has joined #tripleo19:23
*** ansiwen has joined #tripleo19:23
*** crinkle has joined #tripleo19:23
*** r-mibu has joined #tripleo19:23
*** itlinux has joined #tripleo19:23
*** penick has joined #tripleo19:23
*** rhallisey has joined #tripleo19:23
*** tzumainn has joined #tripleo19:23
*** fultonj has joined #tripleo19:23
*** Ryjedo has joined #tripleo19:23
*** gfidente|afk has joined #tripleo19:23
*** orwell.freenode.net sets mode: +o ChanServ19:23
*** davidlenwell has joined #tripleo19:23
*** hogepodge has joined #tripleo19:23
*** spredzy has joined #tripleo19:23
*** patrickeast has joined #tripleo19:23
*** adam_g has joined #tripleo19:23
*** pleia2 has joined #tripleo19:23
*** saneax-_-|AFK has joined #tripleo19:23
*** tbarron has joined #tripleo19:23
*** vkmc has joined #tripleo19:23
*** rickflare has joined #tripleo19:23
*** SpamapS has joined #tripleo19:23
*** matbu has joined #tripleo19:23
*** orwell.freenode.net sets mode: +v davidlenwell19:23
*** rook has joined #tripleo19:23
*** dmsimard has joined #tripleo19:23
*** leifmadsen has joined #tripleo19:23
*** rbowen has joined #tripleo19:23
*** cmyster has joined #tripleo19:23
*** mburned has joined #tripleo19:23
*** fragatina has joined #tripleo19:23
*** kjw3 has joined #tripleo19:23
*** oshvartz has joined #tripleo19:23
*** openstackgerrit has joined #tripleo19:23
*** social has joined #tripleo19:23
*** adarazs has joined #tripleo19:23
*** lifeless has joined #tripleo19:23
*** beagles has joined #tripleo19:23
*** bkreitch has joined #tripleo19:23
*** therve has joined #tripleo19:23
*** sanjayu|AFK has joined #tripleo19:23
*** gchamoul has joined #tripleo19:23
*** marios has joined #tripleo19:23
*** jroll has joined #tripleo19:23
*** EmilienM has joined #tripleo19:23
*** panda has joined #tripleo19:23
*** bvandenh has joined #tripleo19:23
*** rwsu has joined #tripleo19:23
*** dtrainor has joined #tripleo19:23
*** snecklifter has joined #tripleo19:23
*** pradk has joined #tripleo19:23
*** greghaynes has joined #tripleo19:23
*** cwolferh has joined #tripleo19:23
*** akrzos has joined #tripleo19:23
*** dhill_ has joined #tripleo19:23
*** Hazelesque has joined #tripleo19:23
*** radez has joined #tripleo19:23
*** yamahata has joined #tripleo19:23
*** bana_k has joined #tripleo19:23
*** yolanda has joined #tripleo19:23
*** tdasilva has joined #tripleo19:23
*** ipsecguy has joined #tripleo19:23
*** michchap has joined #tripleo19:23
*** egafford has joined #tripleo19:23
*** bkopilov has joined #tripleo19:23
*** b00tcat has joined #tripleo19:23
*** d0ugal has joined #tripleo19:23
*** kbyrne has joined #tripleo19:23
*** lucas-dinner has joined #tripleo19:23
*** cinerama has joined #tripleo19:23
*** stevebaker has joined #tripleo19:23
*** jidar has joined #tripleo19:23
*** dmanchad has quit IRC19:23
*** dmanchad has joined #tripleo19:23
*** weshay has quit IRC19:24
*** weshay has joined #tripleo19:24
*** Hazelesque has quit IRC19:24
*** Hazelesque has joined #tripleo19:24
*** dtantsur has quit IRC19:24
*** dtantsur has joined #tripleo19:24
*** trown has quit IRC19:24
*** trown has joined #tripleo19:24
*** zoli_gone-proxy has quit IRC19:24
*** zoli_gone-proxy has joined #tripleo19:24
*** jmiu has quit IRC19:24
*** jmiu has joined #tripleo19:24
*** rook has quit IRC19:24
*** rook has joined #tripleo19:24
*** mburned has quit IRC19:24
*** mburned has joined #tripleo19:24
*** cmyster has quit IRC19:24
*** cmyster has joined #tripleo19:24
*** patrickeast has quit IRC19:24
*** rook is now known as Guest4601319:24
*** radez has quit IRC19:24
*** radez has joined #tripleo19:24
*** akrzos has quit IRC19:24
*** akrzos has joined #tripleo19:24
*** leifmadsen has quit IRC19:24
*** leifmadsen has joined #tripleo19:24
*** cmyster is now known as Guest9003119:24
*** panda is now known as Guest4661419:24
*** rbowen has quit IRC19:24
*** rbowen has joined #tripleo19:24
*** jpena|off has joined #tripleo19:24
*** dmsimard has quit IRC19:25
*** rlandy|brb is now known as rlandy19:26
*** ayoung has joined #tripleo19:26
*** dmsimard has joined #tripleo19:26
*** dobson has joined #tripleo19:27
*** CaptTofu has joined #tripleo19:29
*** bfournie has quit IRC19:34
*** Guest46614 is now known as panda19:34
*** bfournie has joined #tripleo19:34
*** bfournie has quit IRC19:34
openstackgerritwes hayutin proposed openstack/tripleo-quickstart: set up quickstart to replace instack-virt-setup  https://review.openstack.org/35808919:36
openstackgerritwes hayutin proposed openstack/tripleo-quickstart: set up quickstart to replace instack-virt-setup  https://review.openstack.org/35808919:37
*** dmsimard is now known as dmsimard|afk19:37
*** chem has quit IRC19:38
*** dobson has quit IRC19:38
*** dobson has joined #tripleo19:39
*** NobodyCam has joined #tripleo19:41
*** jrist has quit IRC19:43
*** jrist has joined #tripleo19:43
*** pkovar has quit IRC19:47
*** akshai_ has quit IRC19:48
openstackgerritGabriele Cerami proposed openstack/tripleo-quickstart: Change position to gate injection  https://review.openstack.org/36964919:49
mwhahahatrown: hey does tripleo-quickstart support multiple environments on a single virt host?19:53
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: Move role ResourceGroups inside the jinja2 loop  https://review.openstack.org/36579319:54
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: Convert allNodesConfig properties to composable jinja2  https://review.openstack.org/36579419:54
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: Convert deploy steps to jinja2 loop  https://review.openstack.org/36579619:54
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: Convert UpdateWorkflow to support composable roles  https://review.openstack.org/36728219:54
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: Convert AllNodesExtraConfig to support composable roles  https://review.openstack.org/36729519:55
EmilienMit shoudl pass19:55
mwhahahanone shall pass \o/|19:55
trownmwhahaha: there is nothing theoretically in the way...19:55
trownmwhahaha: ie if you use different virt names for the two different envs there is nothing blocking that from working19:55
*** dsariel has quit IRC19:56
mwhahahatrown: ok thanks, just wondering19:56
*** mcornea has quit IRC19:57
*** jlinkes has quit IRC20:01
*** jlinkes has joined #tripleo20:01
*** igorbelikov has joined #tripleo20:03
*** jobewan has joined #tripleo20:03
*** NachoDuck has joined #tripleo20:04
*** coolsvap_ has joined #tripleo20:05
*** jobewan has quit IRC20:06
*** gregwork has joined #tripleo20:06
*** patrickeast has joined #tripleo20:06
*** nyechiel has joined #tripleo20:07
*** bfournie has joined #tripleo20:07
*** itlinux has quit IRC20:08
*** tzumainn has quit IRC20:08
*** tzumainn has joined #tripleo20:08
*** mgarciam has joined #tripleo20:10
openstackgerritBen Nemec proposed openstack/python-tripleoclient: Handle openstackclient.tests.utils move  https://review.openstack.org/36965820:11
*** pcaruana has quit IRC20:11
bnemecEmilienM: ^ is a workaround on our side.20:11
EmilienMbnemec: nice, thanks for the quick fix20:11
bnemecbeagles: Sounds like a serious problem.  Do you know how to fix it?20:12
openstackgerritwes hayutin proposed openstack/tripleo-quickstart: [WIP] update to README to explain how to run a full end to end deployment and test  https://review.openstack.org/36965920:13
*** paramite has quit IRC20:17
*** sshnaidm is now known as sshnaidm|afk20:17
*** hrybacki has joined #tripleo20:18
*** jayg is now known as jayg|g0n320:18
*** apetrich has quit IRC20:19
*** akuznetsov has joined #tripleo20:20
*** yolanda has quit IRC20:20
*** yolanda has joined #tripleo20:21
*** itlinux has joined #tripleo20:21
EmilienMlarsks: it sounds like to me https://review.openstack.org/#/c/353506/ has a valid CI failure20:21
EmilienMhttp://logs.openstack.org/06/353506/48/check/gate-tripleo-ci-centos-7-scenario003-multinode/1960dbc/logs/postci.txt.gz#_2016-09-13_15_53_00_00020:21
EmilienMResource CREATE failed: The Referenced Attribute (ControllerServiceChain role_data) is incorrect.20:21
EmilienMI'll dig into logs20:22
*** akuznetsov has quit IRC20:24
*** nyechiel has quit IRC20:24
*** rhallisey has quit IRC20:28
*** rhallisey has joined #tripleo20:30
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Remove Ceph from the ha job  https://review.openstack.org/36299020:30
*** apetrich has joined #tripleo20:33
*** jlinkes_ has joined #tripleo20:35
*** jlinkes has quit IRC20:39
*** trown is now known as trown|outtypewww20:40
openstackgerritwes hayutin proposed openstack/tripleo-quickstart: update the upgrade.sh script to work w/ qs extras  https://review.openstack.org/36967420:42
larsksEmilienM, sigh.  And after passing for two weeks.20:42
larsksI'll take a look this evening.20:43
*** rodrigod` is now known as rodrigods20:45
*** Ng has joined #tripleo20:56
*** ChanServ sets mode: +v Ng20:56
*** rhallisey has quit IRC21:09
*** myoung is now known as myoung|bbl21:11
*** lblanchard has quit IRC21:14
*** apetrich has quit IRC21:16
*** bnemec has quit IRC21:20
*** akuznetsov has joined #tripleo21:20
*** zoli_gone-proxy has quit IRC21:21
*** noslzzp has quit IRC21:21
*** trown|outtypewww has quit IRC21:21
*** trown has joined #tripleo21:23
*** fragatin_ has joined #tripleo21:24
*** nyechiel has joined #tripleo21:24
*** bnemec has joined #tripleo21:24
*** mgarciam has left #tripleo21:26
*** lucas-dinner has quit IRC21:26
*** fragatina has quit IRC21:27
*** fragatin_ has quit IRC21:27
*** jlinkes_ has quit IRC21:27
*** NachoDuck has quit IRC21:27
*** pradk has quit IRC21:27
*** cwolferh has quit IRC21:27
*** akrzos has quit IRC21:27
*** dhill_ has quit IRC21:27
*** greghaynes has quit IRC21:27
*** radez has quit IRC21:27
*** Hazelesque has quit IRC21:27
*** yamahata has quit IRC21:27
*** bana_k has quit IRC21:27
*** tdasilva has quit IRC21:27
*** ipsecguy has quit IRC21:27
*** michchap has quit IRC21:28
*** akuznetsov has quit IRC21:28
*** ipsecguy has joined #tripleo21:28
*** radez has joined #tripleo21:28
*** akrzos has joined #tripleo21:28
*** cwolferh has joined #tripleo21:28
*** greghaynes has joined #tripleo21:28
*** jlinkes_ has joined #tripleo21:28
*** dhill_ has joined #tripleo21:28
*** pradk has joined #tripleo21:28
*** michchap has joined #tripleo21:28
*** bana_k has joined #tripleo21:28
*** Hazelesque has joined #tripleo21:28
*** fragatina has joined #tripleo21:29
*** lucasagomes has joined #tripleo21:29
*** yamahata has joined #tripleo21:30
*** zoli_gone-proxy has joined #tripleo21:33
*** fragatina has quit IRC21:33
rwsuwhy does "openstack undercloud install" ask for the user password?21:34
*** lucasagomes has quit IRC21:34
*** NachoDuck has joined #tripleo21:35
*** lucasagomes has joined #tripleo21:35
bnemecrwsu: There's a bug in the version of osc we have in the dlrn repos right now.21:36
bnemecIt should be fixed in master osc.21:36
bnemecIf you just hit enter on the prompt it will continue normally.21:37
rwsubnemec: ok thanks, i thought I might have misconfigured something21:37
*** fragatina has joined #tripleo21:39
*** fragatina has quit IRC21:39
*** fragatina has joined #tripleo21:40
*** lucasagomes has quit IRC21:40
*** lucasagomes has joined #tripleo21:41
*** Guest46013 has quit IRC21:42
*** rook has joined #tripleo21:42
*** rook is now known as Guest73421:42
*** sai has quit IRC21:43
*** sai has joined #tripleo21:43
*** fragatina has quit IRC21:45
*** lucasagomes has quit IRC21:46
*** fzdarsky is now known as fzdarsky|afk21:46
*** lucasagomes has joined #tripleo21:48
*** nyechiel has quit IRC21:51
*** fzdarsky|afk has quit IRC21:51
*** lucasagomes has quit IRC21:52
*** noslzzp has joined #tripleo21:56
*** fragatina has joined #tripleo21:56
*** lucasagomes has joined #tripleo21:57
*** fragatina has quit IRC21:57
*** lucasagomes has quit IRC22:02
*** lucasagomes has joined #tripleo22:02
*** tdasilva has joined #tripleo22:06
*** lucasagomes has quit IRC22:07
*** morazi has quit IRC22:09
*** lucasagomes has joined #tripleo22:10
*** panda is now known as panda|Zz22:11
*** limao has joined #tripleo22:12
openstackgerritMerged openstack/tripleo-quickstart: set up quickstart to replace instack-virt-setup  https://review.openstack.org/35808922:13
*** itlinux has quit IRC22:13
*** lucasagomes has quit IRC22:15
*** itlinux has joined #tripleo22:16
*** lucasagomes has joined #tripleo22:16
*** fragatina has joined #tripleo22:20
*** lucasagomes has quit IRC22:21
*** lucasagomes has joined #tripleo22:23
*** akuznetsov has joined #tripleo22:23
*** fragatina has quit IRC22:24
*** itlinux has quit IRC22:26
openstackgerritSteve Baker proposed openstack/os-collect-config: WIP ec2 collector attempt config-drive before metadata server  https://review.openstack.org/36973122:26
*** limao has quit IRC22:26
*** limao has joined #tripleo22:26
colonwqEmilienM, or anyone, How do I recheck the OVB tests for a patch?22:26
*** akuznetsov has quit IRC22:27
*** lucasagomes has quit IRC22:28
*** lucasagomes has joined #tripleo22:29
*** limao has quit IRC22:31
*** dsariel has joined #tripleo22:33
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Migrate to using osc-lib  https://review.openstack.org/33546022:35
*** lucasagomes has quit IRC22:36
*** lucasagomes has joined #tripleo22:38
*** itlinux has joined #tripleo22:39
*** sai has quit IRC22:43
*** apetrich has joined #tripleo22:43
*** sai has joined #tripleo22:44
*** lucasagomes has quit IRC22:46
*** itlinux has quit IRC22:49
*** lucasagomes has joined #tripleo22:49
*** gfidente|afk has quit IRC22:51
*** lucasagomes has quit IRC22:54
*** fragatina has joined #tripleo22:55
*** lucasagomes has joined #tripleo22:56
*** sai has quit IRC22:56
*** Goneri has quit IRC22:57
*** itlinux has joined #tripleo22:57
*** fragatina has quit IRC23:00
*** sai has joined #tripleo23:00
*** fragatina has joined #tripleo23:03
*** saneax-_-|AFK is now known as saneax23:03
*** lucasagomes has quit IRC23:05
*** lucasagomes has joined #tripleo23:06
*** fragatina has quit IRC23:07
*** lucasagomes has quit IRC23:10
*** fragatina has joined #tripleo23:10
*** dmsimard|afk is now known as dmsimard23:11
*** yamahata has quit IRC23:15
*** lucasagomes has joined #tripleo23:16
*** ayoung has quit IRC23:20
*** Goneri has joined #tripleo23:22
*** pmannidi has joined #tripleo23:23
*** rcernin has quit IRC23:23
*** lucasagomes has quit IRC23:23
*** akuznetsov has joined #tripleo23:24
*** yamahata has joined #tripleo23:24
*** lucasagomes has joined #tripleo23:27
*** akuznetsov has quit IRC23:29
*** lucasagomes has quit IRC23:31
*** Goneri has quit IRC23:31
*** lucasagomes has joined #tripleo23:33
*** sai has quit IRC23:34
*** dsariel has quit IRC23:36
*** saneax is now known as saneax-_-|AFK23:36
*** lucasagomes has quit IRC23:38
*** sai has joined #tripleo23:38
*** lucasagomes has joined #tripleo23:40
openstackgerritMerged openstack/tripleo-heat-templates: Move role ResourceGroups inside the jinja2 loop  https://review.openstack.org/36579323:51
*** apetrich has quit IRC23:52
EmilienMcolonwq: "recheck"23:55
*** apetrich has joined #tripleo23:55
*** sai has quit IRC23:56
*** sai has joined #tripleo23:57

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!