Wednesday, 2016-01-27

*** penick has joined #tripleo00:01
*** dmacpher has joined #tripleo00:04
*** rlandy has quit IRC00:05
*** rlandy has joined #tripleo00:23
*** derekh has quit IRC00:26
openstackgerritDerek Higgins proposed openstack/tripleo-common: Revert "Pin puppet heat"
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: [NO MERGY] Test a update in trunk repository version
*** jcoufal has quit IRC00:34
openstackgerritDan Sneddon proposed openstack/tripleo-heat-templates: Implement str_replace to unify IPv4/IPv6 ports [DO NOT MERGE]
*** eil397 has quit IRC01:01
*** rhallisey has quit IRC01:02
*** egafford has joined #tripleo01:07
*** dsneddon is now known as dsneddon_biab01:16
*** alop has quit IRC01:28
*** ccrouch has quit IRC01:29
*** thrash is now known as thrash|g0ne01:31
*** egafford has quit IRC01:31
*** david-lyle has quit IRC01:36
*** penick has quit IRC01:37
*** dsneddon_biab is now known as dsneddon01:41
*** tiswanso has joined #tripleo01:47
*** tiswanso has quit IRC01:47
*** tiswanso has joined #tripleo01:48
*** trozet has quit IRC02:08
*** egafford has joined #tripleo02:12
*** trozet has joined #tripleo02:14
*** egafford has quit IRC02:17
*** cwolferh has quit IRC02:19
openstackgerritayoung proposed openstack/tripleo-heat-templates: puppet: run keystone in wsgi
openstackgerritDan Sneddon proposed openstack/tripleo-heat-templates: Implement str_replace to unify IPv4/IPv6 ports [DO NOT MERGE]
*** egafford has joined #tripleo02:35
*** egafford has quit IRC02:40
*** yamahata has quit IRC02:44
*** Marga_ has quit IRC02:53
*** pradk has quit IRC02:59
*** Marga_ has joined #tripleo03:09
*** pradk has joined #tripleo03:12
*** trozet has quit IRC03:13
*** Marga_ has quit IRC03:14
*** yuanying has quit IRC03:21
*** Marga_ has joined #tripleo03:23
*** yuanying has joined #tripleo03:23
*** cwolferh has joined #tripleo03:23
*** Marga_ has quit IRC03:27
*** tzumainn has quit IRC03:27
*** yuanying has quit IRC03:28
*** yuanying has joined #tripleo03:33
openstackgerritEmilien Macchi proposed openstack/instack-undercloud: puppet-stack-config: make sure heat use 'rabbit' rpc_backend
openstackgerritEmilien Macchi proposed openstack/instack-undercloud: puppet-stack-config: make sure heat use 'rabbit' rpc_backend
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: controller: make sure heat use 'rabbit' rpc_backend
*** Marga_ has joined #tripleo03:36
openstackgerritEmilien Macchi proposed openstack/tripleo-common: Revert "Pin puppet heat"
*** yuanying has quit IRC03:40
*** yuanying has joined #tripleo03:40
*** shivrao has quit IRC03:41
*** Marga_ has quit IRC03:41
*** shivrao has joined #tripleo03:43
*** shivrao has quit IRC03:44
*** shivrao has joined #tripleo03:47
*** shivrao has quit IRC03:51
*** Marga_ has joined #tripleo03:52
*** sthillma has quit IRC03:54
openstackgerritMatthew Thode proposed openstack/diskimage-builder: add support for gentoo to a bunch of elements
*** yuanying has quit IRC03:56
*** yuanying has joined #tripleo03:57
*** stendulker has joined #tripleo03:58
*** yuanying_ has joined #tripleo03:58
*** stendulker_ has joined #tripleo04:00
*** yuanying has quit IRC04:01
*** stendulker has quit IRC04:03
*** rlandy has quit IRC04:12
*** stendulker has joined #tripleo04:23
*** stendulker_ has quit IRC04:23
*** david-lyle has joined #tripleo04:25
*** david-lyle has quit IRC04:25
*** coolsvap|away is now known as coolsvap04:31
*** stendulker_ has joined #tripleo04:32
*** stendulker has quit IRC04:34
*** david-lyle has joined #tripleo04:38
*** shivrao has joined #tripleo04:39
*** shivrao has quit IRC04:43
*** cwolferh has quit IRC04:45
*** cwolferh has joined #tripleo04:45
*** shivrao has joined #tripleo04:49
*** masco has joined #tripleo04:54
*** cwolferh has quit IRC05:00
*** yamahata has joined #tripleo05:04
*** dmacpher has quit IRC05:08
*** rbrady has quit IRC05:11
openstackgerritMatthew Thode proposed openstack/diskimage-builder: add support for gentoo to a bunch of elements
*** lazy_prince has joined #tripleo05:51
*** penick has joined #tripleo05:52
*** rbrady has joined #tripleo05:55
openstackgerritMatthew Thode proposed openstack/diskimage-builder: add support for gentoo to a bunch of elements
*** liverpooler has quit IRC06:13
*** jaosorior has joined #tripleo06:27
*** cwolferh has joined #tripleo06:38
*** dshulyak has joined #tripleo06:42
*** larstobi has quit IRC06:51
*** larstobi has joined #tripleo06:55
*** aufi has joined #tripleo06:56
*** shivrao has quit IRC06:58
openstackgerritMatthew Thode proposed openstack/diskimage-builder: add support for gentoo to a bunch of elements
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: [NO MERGY] Test a update in trunk repository version
openstackgerritDerek Higgins proposed openstack/tripleo-common: Revert "Pin puppetlabs-mysql to get CI going"
openstackgerritDerek Higgins proposed openstack/tripleo-common: Revert "Pin puppet heat"
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: [NO MERGY] Test a update in trunk repository version
*** Marga_ has quit IRC07:18
*** stendulker has joined #tripleo07:20
*** stendulker_ has quit IRC07:21
*** AJaeger has joined #tripleo07:22
AJaegerHi tripleo cores, for 4 of you repos, I've removed argparse, a useless import since you're not supporting python 2.6 anymore. Could you review, please?
*** oshvartz has joined #tripleo07:24
*** ukalifon1 has joined #tripleo07:24
*** rcernin has joined #tripleo07:27
*** penick has quit IRC07:27
*** penick has joined #tripleo07:28
*** chlong_zzz is now known as chlong07:31
*** jcoufal has joined #tripleo07:37
*** penick has quit IRC07:49
*** bvandenh has joined #tripleo07:56
openstackgerritEvgeny Bagdasaryan proposed openstack/tripleo-heat-templates: Add BondInterfaceOvsOptions parameter to net-config-bond.yaml
*** liverpooler has joined #tripleo08:02
*** tzumainn has joined #tripleo08:03
*** fgimenez has joined #tripleo08:04
*** fgimenez has quit IRC08:04
*** fgimenez has joined #tripleo08:04
*** aufi has quit IRC08:07
*** aufi has joined #tripleo08:11
*** hjensas has quit IRC08:13
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates: Make enabling of controller services configurable.
*** ifarkas has joined #tripleo08:18
*** tzumainn has quit IRC08:19
*** regebro has joined #tripleo08:22
*** jprovazn has joined #tripleo08:22
*** shardy has joined #tripleo08:29
*** bvandenh has quit IRC08:30
*** mkovacik has quit IRC08:39
*** derekh has joined #tripleo08:56
*** lucas-dinner is now known as lucasagomes08:58
*** hjensas has joined #tripleo08:59
*** hjensas has quit IRC08:59
*** hjensas has joined #tripleo08:59
*** mbound has joined #tripleo09:00
*** hjensas has quit IRC09:00
*** hjensas has joined #tripleo09:01
*** hjensas has quit IRC09:01
*** hjensas has joined #tripleo09:01
*** devvesa has joined #tripleo09:04
*** cmyster has quit IRC09:07
*** cmyster has joined #tripleo09:07
*** cmyster has quit IRC09:07
*** cmyster has joined #tripleo09:07
*** gfidente has joined #tripleo09:10
*** gfidente has quit IRC09:10
*** gfidente has joined #tripleo09:10
*** mkovacik has joined #tripleo09:16
marios:/  gerrit being weird/flaky09:17
marioso/ morning thanks see you already commented on the "Make enabling of controller services configurable."
*** fgimenez has quit IRC09:18
shardymarios: Hey g'morning, yeah I think we need that other backport to land09:18
shardyit failed pingtest on one job, not sure why so I rechecked09:18
*** Marga_ has joined #tripleo09:19
*** jaosorior has quit IRC09:19
mariosshardy: am waiting for gerrit to show me that other change you reference. also, wrt the pingtest, i am still not convinced about cirros. i still have it fail sometimes for me locally (virt env always) when i run it.09:19
mariosshardy: it may be we want to revisit building fedora-user for example09:20
*** jaosorior has joined #tripleo09:20
shardymarios: Ok, that's weird09:20
shardyI thought nova used cirros for nearly all tests, so assumed it'd be solid09:21
shardywe use it for some heat tests too in the gate09:21
shardywe've always had problems using a fedora image because it takes sooo much longer to boot virt-on-virt09:21
mariosshardy: yeah i don't know. i mean it totally makes sense since *that* is precisely what it is for, it is small file size etc etc. just saying when i tested i had issues, but randomly09:21
shardymarios: ack, well it'd be good to figure those out for sure09:22
shardymaybe a local test running in a loop overnight?09:22
shardythen we can at least figure out what the bad state is when it fails09:22
shardyI'm assuming that will be hard to do in the gate09:22
*** fgimenez has joined #tripleo09:22
shardyIs anyone else hitting ?09:23
openstackLaunchpad bug 1538254 in tripleo "Error: Must pass controller_virtual_ip to Class[Tripleo::Loadbalancer]" [Undecided,New]09:23
*** Marga_ has quit IRC09:23
shardyI guess I can pass ControlFixedIPs to work around it, but I didn't have to previously, and I don't think we do in the gate09:23
*** mcornea has joined #tripleo09:23
mariosshardy: yeah i'd like to find out more about why it fails too. Initially i was waiting to see if it was OK once it landed in gate and seemingly it is (seen a couple runs poking at logs) but if it continues to fail then it might point to something in tripleo/env09:24
* shardy regrets updating all-the-things yesterday :(09:24
*** jaosorior has quit IRC09:25
gfidenteshardy, so you continue to get empty fixed_ips in the neutron port?09:25
*** jaosorior has joined #tripleo09:25
shardygfidente: Yeah, I deleted my undercloud, rebuilt everything with, and rebuilt all my images (again)09:25
shardysame problem09:25
jaosorioranybody know if the stable/liberty gate is working yet?09:25
shardyjaosorior: we need before the HA job will pass09:26
jaosoriorshardy: Well, there's my +1 already09:26
gfidenteshardy, and is this only for the controlvirtualip?09:27
shardygfidente: ControlVirtualIP and RedisVirtualIP both have empty fixed_ips09:27
shardyall the other ports have IPs09:27
gfidentebut this makes no sense to me09:28
jaosoriorgfidente: What happened?09:29
gfidenteshardy, all other ports you mean the node ports or have you deployed using network-isolation ?09:30
shardygfidente: I'm not deploying with network isolation09:30
shardyI mean if I do neutron port-list09:30
shardythe redis/controller ports have empty fixed_ips09:30
gfidenteok so control_virtual_ip and redis are the only two ports we create in neutron from the templates in this case09:31
shardythe other unnamed ones do09:31
gfidenteyeah exactly09:31
mariosshardy: i wana +2 this but i also don't want to land it until master does09:31
gfidenteso I'm thinking if any recent change is tricking the neutron port resource into *not* allocating any ip because it gets [] as fixed_ips property?09:31
shardygfidente: Yeah, that's what it looks like, but I don't understand why the gate isn't broken09:32
gfidenteshardy, so I am just randomly guessing, maybe gate is pinning something outside of>09:33
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates: Make enabling of controller services configurable.
mariosshardy: rebased onto /#/c/27219409:40
shardymarios: thanks09:41
*** nico_auv has joined #tripleo09:41
ramishra_shardy/gfidente: hi, passing [] as fixed_ips actually creates port without any ip. This is standard neutron behaviour. Earlier we used clean it up in the neutron port resource in heat, but we fixed that in
*** akrivoka has joined #tripleo09:41
gfidenteramishra_, oh that explains it I think09:42
ramishra_I assume that has created the issue you guys are discussing here.09:42
shardyramishra_: Hmm09:42
shardyramishra_: Isn't that a backwards incompatible change?09:42
shardy(I know, I reviewed it)09:42
*** devvesa has quit IRC09:42
shardyI saw that we removed the default=[] but at the time didn't realize the significance09:43
ramishra_yeah:),  We should not pass anything when we want neutron to allocate the ips.09:43
ramishra_That's the default behaviour. However, I agree it's not backward compatible from the earlier template pov09:43
*** jistr has joined #tripleo09:43
shardyramishra_: I think we'll have to fix it09:44
shardyI agree the new behavior is more technically correct, but we can't break existing templates09:44
shardywhat do you think?09:44
ramishra_Because during update, [] means clear the ips09:44
ramishra_Yeah, If you can raise a bug I'll make the change09:45
ramishra_Does that sound ok?09:45
shardyramishra_: Sure, will do09:45
shardyramishra_: the problem is, we want to have a parameter wired in to fixed_ips, but have it be optional09:46
shardyI can't see any way to do that with this change in place09:46
shardye.g both an empty list or "" will pass the "09:46
shardyif self.FIXED_IPS in props test09:46
ramishra_Yeah, but then we'll be masking neutron behaviour for ever:)09:47
*** bvandenh has joined #tripleo09:47
shardyramishra_: I'll raise a bug and we can discuss it there and in #heat09:47
shardythanks for pointing it out!09:47
ramishra_sure, I'll do a quick fix.09:48
openstackLaunchpad bug 1538473 in heat "Neutron port fixed_ips backwards incompatible change wrt "[]"" [Undecided,New]09:53
shardyAlso this explains why we're not seeing it in the gate, we're pinned to an old heat, I'm running master09:53
*** paramite has joined #tripleo09:56
*** devvesa has joined #tripleo09:57
*** bvandenh has quit IRC09:57
*** olap has joined #tripleo10:07
*** rebrego has joined #tripleo10:22
*** regebro has quit IRC10:22
*** rebrego is now known as regebro10:22
*** mgould has joined #tripleo10:23
*** killer_prince has joined #tripleo10:26
*** lazy_prince has quit IRC10:29
AJaegerHi tripleo cores, for 4 of you repos, I've removed argparse, a useless import since you're not supporting python 2.6 anymore. Could you review, please? - shardy was so kind to +2 all except 27037510:32
*** tosky has joined #tripleo10:34
*** dtantsur|afk is now known as dtantsur10:37
*** electrofelix has joined #tripleo10:38
*** bvandenh has joined #tripleo10:39
*** devvesa has quit IRC10:39
*** killer_prince has quit IRC10:40
*** lazy_prince has joined #tripleo10:41
openstackgerritMerged openstack/os-refresh-config: Remove argparse from requirements
openstackgerritMerged openstack/os-cloud-config: Remove argparse from requirements
AJaegerthanks, derekh10:43
derekhAJaeger: np, thanks10:43
openstackgerritMerged openstack/os-collect-config: Remove argparse from requirements
*** bvandenh has quit IRC10:44
openstackgerritMerged openstack/os-apply-config: Remove argparse from requirements
*** stendulker_ has joined #tripleo10:50
*** gchamoul has left #tripleo10:51
*** stendulker has quit IRC10:53
*** stendulker_ has quit IRC10:56
*** rbrady has quit IRC10:56
*** bvandenh has joined #tripleo10:57
derekhOk people, we can move onto a new delorean repository (tests are passing here ), the recheck has passed the ceph and HA job just hasn't reported back yet11:05
derekhwe just gotta merge a few patches together and update the current-tripleo link, I'd like to do this now as its taken trown a lot of work to get to this point and we need to try and keep it like that (next step)11:06
derekhthese are what needs to merge, I'd say lets ignore the CI, the patch that tests them all together shows them passing11:06
derekhpuppet modules reverts -
derekhRemove empty value for wsrep_notify_cmd
derekhupdates for new heatclient
derekhso how about it, can we go ahead and merge these right now ?11:07
derekh*or as close to now as possible11:07
shardyderekh: ack, looking11:08
*** hjensas has quit IRC11:10
shardyHrm, so we've made a completely backwards incompatible change to heatclient? ugh :(11:13
derekhshardy: ya, trown|outttypeww knows the details, but iirc something that used to be output alone, is now formatted in a table11:15
shardyderekh: Ok, that seems odd and wrong for -F raw11:15
shardybut +1 on landing all-the-things, we can potentially fix that in heatclient - IMO it's a bug11:15
derekhshardy: yup, now that you mention it, it does seem wrong11:16
shardye.g we can potentially fix it later11:16
derekhshardy: yup11:16
shardyI'll ask trown|outttypeww to raise a heatclient bug and we'll investigate11:16
derekhshardy: ok11:16
*** fgimenez has quit IRC11:18
shardyderekh: All looks fine modulo the heatclient thing - how long will it take for CI to report on ?11:18
shardyif approving a bunch of stuff with failing CI it'd be nice to reference that as justification in the comments11:19
*** trown|outttypeww is now known as trown11:19
derekhshardy: its waiting on the containers job to timeout/fail , shouldn't be much longer, I'll ping back when its done11:20
*** fgimenez has joined #tripleo11:20
*** fgimenez has joined #tripleo11:20
shardyderekh: ack, I've got all the reviews open ready to approve ;)11:20
trownshardy: ya I looked into it, and I think it is actually coming from cliff11:20
trownshardy: and heatclient just inherits it11:21
derekhshardy: cool, thanks11:21
shardytrown: ouch, I thought we only used that for the new heat oscplugin11:21
shardytrown: would you be able to please raise a heatclient bug explaining the issue?11:21
trownshardy: sure11:21
shardythen we can make a call as to if a fix is possible11:22
* derekh is tempted to ssh onto the instance running the containers test and kill the deploy command ;-)11:23
shardyderekh: That test was passing a while ago, so I guess that's the next challenge ;)11:24
derekhshardy: ya, not sure if anybody is looking at it / caring about it11:25
shardyderekh: we should chat with rhallisey later - he, jpeeler and Slower have been working hard to get that working11:25
shardySo, I think we do need to care about it after the other stuff gets fixed11:26
derekhshardy: yup, we need to sort it out, otherwise its just a waist of resources11:26
shardyWoot!  Overcloud create - DONE.11:28
shardyfirst time in two days11:28
shardyramishra_: thanks, your patch fixed my latest issue :)11:28
ramishra_shardy: np:) btw what happened to the tripleo gate job for heat?11:30
shardyramishra_: you can run it via check experimental11:30
shardyit's not been running by default for some time11:31
ramishra_shardy:  ok:) I thought we wanted a voting job;)11:31
shardyramishra_: ideally we do, but TripleO CI just isn't reliable enough unfortunately11:32
ramishra_shardy:  yeah11:32
derekhshardy: ya, we shouldn't turn it back on until we get to a place were we arn't broken all the time11:33
derekh Just to reword slightly, most of the time tripleo ci is doing its job perfectly, its tripleo itself that not working.11:34
*** devvesa has joined #tripleo11:34
derekhthe last 2 breakages on master were because or people ignoring CI results11:34
shardyderekh: very true :(11:34
shardyI think folks just see the high recheck/false-negative rate and assume it's OK to ignore11:35
*** gfidente has quit IRC11:36
derekhshardy: yup, this is exactly what happens and it also causes part of the problem11:36
trownderekh: I have also seen where we merged something with 2 week old CI results which is equivalent11:38
trownit is a bit frustrating for a downstream consumer :)11:38
shardytrown: It's frustrating for everyone unfortunately11:39
derekhtrown: yup, that happens also, its a little more forgivable but we should be careful11:39
shardyfolks don't want to recheck after two weeks because it might take another 2 weeks to get a green run11:39
shardyI agree we need to be careful tho11:39
trownat least its not boring11:40
* trown relocating11:41
*** trown is now known as trown|outttypeww11:41
*** gfidente has joined #tripleo11:42
derekhshardy: 9 minutes to timeout11:42
* derekh goes for tes11:42
*** rbrady has joined #tripleo11:43
*** pcaruana has joined #tripleo11:44
*** AJaeger has left #tripleo11:47
*** jkraj has joined #tripleo11:48
*** dprince has joined #tripleo11:52
derekhshardy: ready to update the link when you are11:53
shardy[heat-api]: Could not evaluate: Cannot allocate memory - fork(11:55
shardyI thought we added swap already?11:55
derekhshardy: we did, where did you see that?11:55
shardyIn the nonha job failure for the patch you just linked11:56
shardyIt's odd as that should use the least memory11:56
derekhshardy: and it passed before the recheck, could it have been triggered by a retry or something11:57
shardyIt's hard to say, but we shouldn't have used 1G of swap11:58
derekh+ free -h11:58
derekh              total        used        free      shared  buff/cache   available11:58
derekhMem:           4.8G        4.4G        180M        284K        285M        222M11:58
derekhSwap:          1.0G        452M        571M11:58
derekhshardy: from that  same job11:59
shardyAnyway, I'll land the patches so we can get things running again and investigate further11:59
derekhat the end of the job, something may have been killed by then11:59
derekhshardy: ack11:59
shardyweird, although it's not good that we're swapping so much11:59
shardyI guess that slows things down a lot11:59
*** tzumainn has joined #tripleo12:00
derekhshardy: if its memory that not accessed much it mightn't effect things much, hard to know with the details we have12:00
openstackgerritMerged openstack/tripleo-heat-templates: Remove empty value for wsrep_notify_cmd
shardyYeah it'd be nice to see the vmstat through the run and see how much is getting swapped out12:01
*** egafford has joined #tripleo12:01
openstackgerritMerged openstack/tripleo-common: Update pingtest for newer heatclient
shardyderekh: we need another reviewer for the two reverts, or do you want me to just approve?12:02
* derekh wonders if this would give us the numbers we want
derekhshardy: I think we can approve to move the whole things along, I'll do it12:03
derekhshardy: one more
openstackgerritMerged openstack-infra/tripleo-ci: Switch the overcloud pingtest to use the new heat client
openstackgerritMerged openstack/tripleo-common: Revert "Pin puppetlabs-mysql to get CI going"
openstackgerritMerged openstack/tripleo-common: Revert "Pin puppet heat"
derekhALL we have updated to a repo from monday12:09
*** hjensas has joined #tripleo12:11
*** mgould has quit IRC12:14
shardyNice work derekh and trown|outttypeww12:23
*** thrash|g0ne is now known as thrash12:28
*** mgould has joined #tripleo12:29
*** pcaruana has quit IRC12:37
*** rhallisey has joined #tripleo12:37
*** Goneri has quit IRC12:38
*** weshay_xchat has joined #tripleo12:38
*** Marga_ has joined #tripleo12:38
*** thrash has quit IRC12:40
*** thrash has joined #tripleo12:49
*** thrash has joined #tripleo12:49
jistrhey folks, anybody able to give some heat hints? trying to update from kilo to liberty, i get: Stack failed with status: resources.Controller: ValueError: resources[0]: "u''" is not a list12:59
*** trown|outttypeww is now known as trown12:59
jistri think the trigger here is that we changed NtpServer to be able to process an array
jistrbut a string should still be a valid value for that13:00
trownwoot, thanks derekh, shardy13:01
jistri guess this is something about internal heat representation then13:01
jistrnow the interesting thing is, even if i set it to something completely different via both parameters and parameter_defaults (an array which doesn't mention at all), i still get the error mentioning clock.redhat.com13:02
jistrit's as if it tried to use/validate the old parameter value in the new templates, even though i provided a new different value to the stack-update call13:03
jistrpossibly a heat bug worth reporting?13:03
jistrshardy: could you check please if my conclusion sounds correct, when you have a minute?13:06
*** david-lyle has quit IRC13:08
*** coolsvap is now known as coolsvap|away13:11
*** Marga_ has quit IRC13:11
*** tiswanso has quit IRC13:11
*** Marga_ has joined #tripleo13:12
*** chlong has quit IRC13:15
*** jayg|g0n3 is now known as jayg13:17
jistrreported to catch the info before i try to revert the NtpServer patch locally to move forward13:22
openstackLaunchpad bug 1538551 in heat "Unable to update a parameter from string to comma_delimited_list" [Undecided,New]13:22
*** chlong has joined #tripleo13:28
*** fgimenez has quit IRC13:30
*** fgimenez has joined #tripleo13:32
*** akuznetsov has joined #tripleo13:46
*** julim has joined #tripleo13:48
*** absubram has quit IRC13:51
*** egafford has quit IRC13:52
*** lucasagomes is now known as lucas-hungry13:54
*** jhenner has quit IRC13:56
*** jhenner has joined #tripleo13:56
*** oshvartz has quit IRC13:58
*** jprovazn has quit IRC13:59
*** jkraj has quit IRC13:59
*** julim_ has joined #tripleo14:02
*** tiswanso has joined #tripleo14:03
*** Goneri has joined #tripleo14:03
*** jhenner has quit IRC14:05
*** julim has quit IRC14:05
mariosshardy: /me palmface "2016-01-27 14:04:42.107 | ERROR: <html><body><h1>503 Service Unavailable</h1>" for overcloud heat @ pingtest for ha job of
mariosso it will fail14:07
*** regebro has quit IRC14:08
*** rook-desktop has quit IRC14:08
*** regebro has joined #tripleo14:08
*** masco has quit IRC14:09
*** morazi has quit IRC14:11
shardyjistr: have you tried any minimal templates to reproduce?14:11
jistrshardy: no, just the tripleo ones14:12
shardyIt'd be good to confirm the same behavior is observed via heatclient directly14:12
*** lblanchard has joined #tripleo14:12
shardyjistr: also, can you confirm you see the expected NtpServer in the --debug output getting passed to heat from tripleoclient?14:12
jistrshardy: i'm on a call atm, but will do that next14:13
shardyjistr: ack, I'll also try to reproduce later14:13
*** rpothier has joined #tripleo14:13
*** morazi has joined #tripleo14:13
*** oshvartz has joined #tripleo14:13
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: controller: make sure heat use 'rabbit' rpc_backend
EmilienMderekh: hey, trying to understand the heat bug with rpc_backend14:14
*** rlandy has joined #tripleo14:14
EmilienMthe default value in oslo messaging is 'rabbit' which was the same in puppet-heat before the breakage14:15
EmilienMshardy: ^14:15
*** david-lyle has joined #tripleo14:17
shardyEmilienM: I think it was due to packaging defaults:14:17
EmilienMahah !14:17
EmilienMnice shot14:17
EmilienMshardy: so I can abandon my tripleo patches14:17
EmilienM and
shardyEmilienM: I think so, now that the current-tripleo pin has been moved14:18
EmilienMshardy: thx for that information14:18
EmilienMI did not know it14:18
shardyEmilienM: np, sorry for the inconvenience14:18
*** jhenner has joined #tripleo14:21
EmilienMshardy: why did you dupplicate the patch 272848 ?14:22
EmilienMI was trying to rebase it and I noticed you did
EmilienManyway, it's merged now, but I see CI not passing on, is it expected?14:23
EmilienMshardy, derekh: please abandon I can't do it14:24
shardyEmilienM: I didn't, derekh pushed that review14:25
shardywe landed a series of patches which were checked via another patches earlier, to unblock CI14:26
shardyEmilienM: they were checked via
shardyunfortunately there was no way to get all of the required patches passing CI14:27
EmilienMshardy: ok14:27
*** jhenner has quit IRC14:31
derekhEmilienM: the reason I did a second identical patch doing the revert was so that I could test it without any depends-on in it14:32
EmilienMmost important thing is that is fixed now14:32
derekhEmilienM: will abandon the other one now14:32
derekhEmilienM: yup, we're all good now14:32
ayoungEmilienM, I'm still battling the Keystone/HTTPD issue.  Latest failure is  /home/jenkins/workspace/gate-tripleo-ci-f22-ha/devstack-gate/ line 1088: 15495 Killed                  timeout -s 9 ${REMAINING_TIME}m bash -c "source $WORKSPACE/devstack-gate/ && $cmd"      This    is from
jaosoriorayoung: that timeout is a common hassle. I think shardy was dealing with it at some point14:34
*** ccrouch has joined #tripleo14:34
ayoungjaosorior, so not a direct result of the patch?14:35
shardywell it could mean anything, it just tells you the job timed out and was killed14:35
ayoungshardy, yeah, and I can't find anything that says what the job actually was that was killed14:35
* shardy looks at logs14:35
derekhSo now that we have a recent trunk repository working, lets try and get the periodic job working14:36
derekhHere is the fix for the current reason the periodic job is failing
*** jprovazn has joined #tripleo14:36
ayoungshardy, last success reported before the line was -- Undercloud install - DONE.14:37
ayounggrep for 2016-01-27 04:09:18.29614:37
derekhAnd these will give us support for a report that just shows us the results of a periodic job
shardyayoung: it looks like it timed out trying to build the images14:37
ayoungshardy, OK,  I thought that, but wasn't sure if it was the next task that failed14:38
*** jprovazn has quit IRC14:38
*** jprovazn has joined #tripleo14:39
derekhunfortunately when we get those timouts, the function that collects the logs doesn't run either ;-(14:40
shardyayoung: the ceph job is failing with a different error14:40
*** liverpooler has quit IRC14:40
shardycomplaining about line 179 here:14:40
jaosoriormarios: I remember you +1ing this patch cause master hadn't merged yet, got some time to check it out again? Now that the one proposed for master is in14:40
derekhI think that would be fixed by collecting the logs outside of the devstack-gate runner14:40
shardyayoung: so the ha job failure may be spurious, but I think that one is real14:41
*** ron___ has joined #tripleo14:41
shardynonha has the same issue14:41
mariosjaosorior: ack14:42
*** david-lyle has quit IRC14:44
*** jhenner has joined #tripleo14:47
*** ron___ has quit IRC14:48
ayoungshardy, yeah, I noticed.  I've not been able to get even master tripleo to install on the Dell workstation I just got, although I did manage to get Director to run once (I think)14:49
ayoungIt makes it hard to code14:49
shardyayoung: Yeah, I'm sorry we couldn't figure out your Nova issues yesterday14:50
*** ron___ has joined #tripleo14:50
shardyFWIW I rebuilt my tripleo environment from scratch hoping to reproduce, and while I hit other problems I didn't see that one14:50
ayoungshardy, I'm going to strip down to baremetal and try that one again todya14:51
shardyayoung: ack - if using ensure you've pulled the latest as some changes landed today14:51
ayoungshardy, so the big ticket item is getting Keystone to run in HTTPD14:51
*** akuznetsov has quit IRC14:51
ayoungI don;t care how that happens, If I do it or someone else14:51
ayoungwithout that, Federation can't happen, and that screws over a lot of people14:52
shardyayoung: also, I'd suggest not using --all, instead run each step, and in particular ensure after --register-nodes that nova hypervisor-stats is updated14:52
ayoungshardy, ++  I learned that/14:52
ayoungok,  kernel is upgraded ... time to install instack.14:52
*** egafford has joined #tripleo14:54
ayoungshardy, check me on this, but image building happens before Keystone runs, no?14:55
*** pradk_ has joined #tripleo14:56
shardyayoung: well if you've run --undercloud then it's running on the undercloud, but yeah there's no overcloud at that point14:57
ayoungshardy, so I can't see how my changes could be screwing that up14:58
shardyayoung: they probably aren't, but the other two jobs are failing due to the patch14:58
ayoungshardy, OK, so the first thing I guessed at was this
ayoungI know nothing from Puppet, copied that from other examples15:00
ayoungit was based on a comment in an earlier review15:00
ayoungbnemec, said "I have the unpleasant suspicion that controller_host isn't going to be defined on the overcloud. If that's the case, I'm not sure off the top of my head what the right way to tell it to bind only to the local IP is though. :-/"15:00
ayoungand marios responded with "I believe Ben is right with his comment here on v16 -  we do pass 'controller_node_names' - note however that this is a comma delimited list of all controller host names15:01
ayounglooking at sounds like it expects only one (if so you can split on ',')"15:01
ayoungI really would rather not have gerrit be my debugger, as it has a very slow turn-around15:01
openstackgerritMatthew Thode proposed openstack/diskimage-builder: add support for gentoo to a bunch of elements
*** lucas-hungry is now known as lucasagomes15:02
dprincejtomasek: zaqar is here and we have a puppet-zaqar module as well... so I will see about getting you some patches to have it available in the undercloud15:03
jtomasekdprince: thanks!15:04
*** oshvartz has quit IRC15:06
*** trozet has joined #tripleo15:15
*** oshvartz has joined #tripleo15:18
*** tosky has quit IRC15:21
EmilienMdprince: puppet-zaqar is empty afik15:24
EmilienMit's noop at this time15:24
EmilienMa guy was working on it but no progress since months15:25
EmilienMlet me ask on #openstack-zaqar15:25
dprinceEmilienM: oh no15:25
dprinceEmilienM: sounds like I may be your man then :)15:25
dprinceEmilienM: go easy on me though15:25
*** dmacpher has joined #tripleo15:25
EmilienMdprince: if you could do the same thing you did with puppet-mistral, I'll pay you french wine15:26
dprinceEmilienM: not critical, but it sounds like it is becoming of interest for the UI team for things like websockets integration15:26
EmilienMmakes sense15:26
dprinceEmilienM: sounds like we may have a deal15:26
*** yamahata has quit IRC15:30
*** yamahata has joined #tripleo15:30
*** jistr|mobi has joined #tripleo15:35
*** tosky has joined #tripleo15:35
*** dprince has quit IRC15:37
d0ugalmgould: I am asking about the failure in #openstack-infra15:53
d0ugalmgould: It sounds like new images are needed for the builders, that should be automated but "sometimes fails"15:55
*** egafford has quit IRC15:58 down?15:59
*** egafford has joined #tripleo16:00
mgouldd0ugal, thank goodness, I was starting to lose my faith in determinism :-)16:02
*** masco has joined #tripleo16:06
EmilienMslagle: yes16:08
EmilienMslagle: #rdo is aware16:09
*** rcernin has quit IRC16:09
trownEmilienM: is it self-aware as in the singularity?16:10
gfidenteshardy, so I do get the error logged for a specific resoucr16:13
gfidenteshardy, but it doesn't look like a simple case of the softwaredeployment config being too big16:13
gfidentebecause I can't reproduce it that way with a simple softwaredeployment pushing a big file on a single server16:14
d0ugalmgould: I guess after a number of retires it is best to ask around :)16:14
*** david-lyle has joined #tripleo16:14
*** masco has quit IRC16:15
ayoungwoot! 90bdd865-0b58-4735-add6-554b45cf08f1 | overcloud  | CREATE_COMPLETE16:15
ayoungOK, I have a successful overcloud deployment16:16
shardyayoung: \o/16:16
shardyNow don't change *anything* ;)16:16
ayoungnow...I need to do development, I assume, --overcloud-delete16:17
*** shivrao has joined #tripleo16:17
ayoungand then try the same thing again with git checkout against master16:17
shardyayoung: you can do that, or just "heat stack-delete"16:17
shardyayoung: the difference is polls until the delete is done16:17
shardyvs doing heat stack-list a few times to check16:17
*** paramite has quit IRC16:17
shardy("heat stack-delete overcloud")16:18
prometheanfireis gate-tripleo-ci-f22-nonha still failing for everyone or just me?16:18
ayoungshardy, actually, to be even more cautious....once I do that, how far back do I need to go to redeploy?16:18
ayoungtripleo-common/scripts/ --register-nodes16:18
ayoungor just16:18
shardyayoung: you're making changes to tripleo-heat-templates right?16:18
ayoungshardy, eventually,16:18
shardyayoung: if that's all you're changing, then it's just:16:19
shardyheat stack-delete overcloud (or --overcloud-delete16:19
shardythen openstack overcloud deploy --templates /path/to/git/tripleo-heat-templates16:19
shardyyou can hack on the local tree of t-h-t and just pass the path16:20
ayoungok, let me make sure I can deploy a second time with no changes16:20
mgouldd0ugal, and only now do I notice that check-osc-plugins is non-voting :-(16:20
*** mbound has quit IRC16:21
*** oshvartz has quit IRC16:23
EmilienMslagle: should be back now16:24
*** NobodyCa1 has joined #tripleo16:24
*** NobodyCam has quit IRC16:25
*** mcornea has quit IRC16:25
*** NobodyCa1 is now known as NobodyCam16:27
d0ugalmgould: hah, so it is. However, good to get these things resolved if we can16:27
*** aufi has quit IRC16:28
mgouldyeah, definitely16:28
d0ugalmgould: FWIW, that failure is happening everywhere :)16:37
d0ugalmgould: Just noticed it on one of my other reviews16:37
*** david-lyle has quit IRC16:38
mgouldd0ugal, bizarre16:39
mgouldit looks like a new Jenkins image was cut some time after 1400 UTC16:39
d0ugalmgould: Yeah, they confirmed the image was built correctly - but couldn't confirm it was uploaded :)16:40
*** david-lyle has joined #tripleo16:40
*** bnemec has quit IRC16:42
d0ugalmgould: but anyway, I think we can ignore it since it is non-voting and trust that it will be resolved in time :)16:42
d0ugalI guess being non-voting somebody is working it anyway16:43
d0ugal(to get it in a state to become voting)16:43
d0ugalmgould: but now we need to worry about the other CI failures :(16:43
mgouldprometheanfire, the check-osc-plugins CI has been failing for days despite the fix already being merged16:46
gfidenteshardy, so the message we always timed out16:46
gfidenteINFO oslo_messaging._drivers.amqpdriver [-] No calling threads waiting for msg_id : e86088d2266f4d10984abe5b469cb03216:47
gfidentewhich I think explains the timeout16:47
prometheanfiremgould: then the fix isn'ta fix?16:47
gfidenteit's the only message id printing that16:47
mgouldprometheanfire, nope16:47
shardygfidente: Hmm, that's strange, sounds like either a process got killed or a greenthread handling the request itself died16:48
mgouldthe fix removes the line from the script that errors16:48
shardyno backtrace before that?16:48
mgouldyet it's still being run16:48
mgouldso the CI workers are still running the old version of the script16:48
mgouldprometheanfire, gate-tripleo-ci-f22-nonha passed for me half an hour ago:
mgouldeverything else is failing, though :-(16:49
*** bnemec has joined #tripleo16:49
prometheanfiremgould: odd, I got everything else to pass16:50
mgouldprometheanfire, so if we combine our patches then everything will work? :-)16:50
prometheanfiremgould: you fine with a 1000 line patch? :P16:51
prometheanfiremost of that is in growpart though16:51
mgouldBTW: I seem to spend an awful lot of time in the "patch failed CI; read logs; determine it's not my fault; recheck; goto 10" loop16:51
mgouldam I just doing it wrong?16:51
prometheanfireno, that's my cycle too16:51
mgouldflaky CI is No Fun16:51
mgoulddo we have any stats on how many transitory CI failures we get?16:54
derekhmgould: lots, but the problem isn't CI, the problem is that everybody ignores the intermittent erros and keeps hitting recheck until they get a pass, the intermittent errors go unfixed and pill up on top of each other16:58
derekhand we eventually get into a state where there are so many errors orrcuring that we cant get anything merged16:58
*** sthillma has joined #tripleo16:58
mgouldderekh, sure16:58
*** trown is now known as trown|lunch16:58
* mgould was thinking that having the numbers might convince people to throw resources at fixing the problem16:59
derekhmgould: yup, it would probably help16:59
*** jistr has quit IRC16:59
derekhmgould: this page give you a visual indication of ci jobs that have failed
*** sthillma_ has joined #tripleo17:01
mgouldderekh: awesome, thanks!17:01
*** dprince has joined #tripleo17:01
derekhmgould: but to do it properly, somebody needs to go through all the logs and see which failures were false negatives and with were legitimate, I've done this in the past but its very time consuming17:01
mgouldwow, that's a lotta red17:01
d0ugalmgould: check-osc-plugins passed!17:01
mgouldd0ugal, \o/!17:02
d0ugalso I guess the image finally got where it needed to be.17:02
mgouldthank goodness for that17:02
mgouldnow let's see what happens in gate-tripleo...17:02
derekhmgould: yup, we've had a bad few days, all of yesterday tripleo master was broken17:03
mgouldwe've had a bad few days in ironic too17:03
*** sthillma has quit IRC17:04
*** sthillma_ is now known as sthillma17:04
* derekh is trying now to reproduce some of the intermittent errors17:04
* mgould applauds derekh17:04
prometheanfirealso, lol    2016-01-27 15:08:56.045 | fatal: A branch named 'master' already exists.17:05
mgouldoh dear17:06
mgouldI thought we tested everything in detached HEAD state?17:06
*** yamahata has quit IRC17:07
prometheanfireI'm going to type recheck and see what happens17:07
*** devvesa has quit IRC17:08
*** fgimenez has quit IRC17:11
derekhprometheanfire: it wont pass, the problem with that patch is that it introduces this file
derekhprometheanfire: forget about the master branch error, its a red herring17:13
*** dtantsur is now known as dtantsur|afk17:13
prometheanfirewhy is that an issue?17:13
derekhprometheanfire: rpmbuild sees "#!/sbin/runscript" and adds an autorequires for a package that provides that script17:14
derekhprometheanfire: fails to find one17:14
derekhprometheanfire: in the delorean logs you'll see17:14
derekhDEBUG: Error: Package: diskimage-builder-1.8.1-dev7.el7.centos.noarch (/diskimage-builder-1.8.1-dev7.el7.centos.noarch)17:14
derekhDEBUG:            Requires: /sbin/runscript17:14
prometheanfirebut it's an init script17:14
prometheanfirethat's how our init scripts work...17:15
prometheanfirehow do I work around it?17:16
prometheanfire as an example17:17
derekhprometheanfire: we had a similar issue last week when "/usr/local/bin/dib-python" was added into another script17:17
derekhprometheanfire: if that line is needed in the script then it will need to be excluded from autorequires17:17
derekhprometheanfire: like
mgouldquestion unrelated to CI: should I abandon in favour of ?17:18
*** mkovacik has quit IRC17:18
mgould2683831 is meant to be a hacky version that can be merged without waiting for lots of dependencies17:19
prometheanfirewhat project is that in?17:19
*** jaosorior has quit IRC17:20
mgouldthey add support for the new version of the Ironic state machine17:20
prometheanfiresorry, was asking derekh how what project I need to submit a review for17:20
*** jaosorior has joined #tripleo17:20
mgouldprometheanfire, oh, sorry17:20
*** jaosorior has quit IRC17:21
derekhprometheanfire: so that is a complicated part, the packaging on a different gerrit (openstack-packages/diskimage-builder on gerrithub), I'd love to see it moved onto our gerrit some time soon but for now the best we can do is line up the two changes and try and merge them together17:23
prometheanfireya, I'm depending on that change :(17:23
derekhactually it might be possible to do the packaging change first, we could try that17:23
derekhIf you submit a change to add it, we can get it merged then your patch to DIB shouldn't fail any longer (atleast for the readon it currently is)17:25
derekhif you prefer I can take a look at the packaging part,17:25
prometheanfireso do I need to clone from gerrithub?17:25
prometheanfireit'd help, it is just that one line17:26
*** shivrao has quit IRC17:26
*** shardy has quit IRC17:26
derekhprometheanfire: on it17:27
*** lazy_prince has quit IRC17:27
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Update to accept files or directories
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Add simple parameter test to
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Remove unused parameters
*** tosky has quit IRC17:29
*** jistr|mobi has quit IRC17:30
*** yamahata has joined #tripleo17:31
derekhprometheanfire: , gotta run out, I'll follow up with it when I'm back in a bit17:33
prometheanfirederekh: that redefines it I think17:35
prometheanfirederekh: These macros are not cumulative17:35
ayoungEmilienM, OK, I have a reproducible setup now.  I can create and destroy overclouds at whim.  Care to help me beat the Keystone HTTPD Review into submission?17:40
ayoungI'm running an overcloud-deploy right now, and I should get a better view into where it fails17:41
EmilienMI was about to get lunch17:41
EmilienMcan you run puppet and give me the output?17:41
EmilienMI'll catch-up after my quick lunch break17:42
*** dshulyak has quit IRC17:42
*** mbound has joined #tripleo17:43
*** mbound has quit IRC17:44
*** tiswanso has quit IRC17:46
ayoungEmilienM, aftet lunch is good17:46
ayoungI need to eat too17:46
derekhprometheanfire: doh, I'll have to take a closer look later on,17:50
*** dmsimard has quit IRC17:51
derekhprometheanfire: thanks, I'll try it out later this evening with your patch and make sure there isn't anything else17:51
*** derekh has quit IRC17:52
openstackgerritgreghaynes proposed openstack/diskimage-builder: DO NOT MERGE: unmount debugging
*** mbound has joined #tripleo17:54
*** jcoufal has quit IRC17:54
*** jdob has quit IRC17:57
*** shivrao has joined #tripleo17:58
*** rcernin has joined #tripleo17:59
*** regebro has quit IRC17:59
*** dshulyak has joined #tripleo18:00
*** lucasagomes is now known as lucas-dinner18:04
dprincerbrady: nice on the action executions suggestion, I think that helps the UI quite a bit18:04
ayoungEmilienM,  based on the steps from  leads me to think I have a syntax error.  Trying to map that to the .pp file18:10
ayoungsince it is line 179 I'm guessing  bind_host => split(hiera('controller_node_names'), ',')[0],  is no good18:11
ayoungaccording to the reviews, we need an IP address for the Keystone service to listen on.  Seems to me it should follow the pattern done for Horizon18:12
ayoungHorizon does not specify anything...18:14
ayoungI think I'm going to yank that line and see what happens.  There and in the controller.pp18:14
*** olap has quit IRC18:15
*** jcoufal has joined #tripleo18:18
*** mbound has quit IRC18:18
*** athomas has quit IRC18:22
*** athomas has joined #tripleo18:22
*** ukalifon1 has quit IRC18:26
*** sthillma has quit IRC18:29
*** ifarkas has quit IRC18:30
*** electrofelix has quit IRC18:32
*** regebro has joined #tripleo18:34
EmilienMayoung: back, looking now18:38
ayoungEmilienM, so, I think it has to do with setting the IP address for HTTPD for Keystone18:38
ayoungmy syntax was bad.  I am trying right now with nothing in that line, to see if we actually need to set it18:38
ayoungthe horizon analogue does not set a host to listne on18:39
ayoungEmilienM, it seems to get further, but I don;t know if that constitutes success18:40
ayoungthis one failed on overcloud-ControllerNodesPostDeployment-y2pizpkdnnto-ControllerOvercloudServicesDeployment_Step4-3ypex4d5dutp18:41
*** mgould has quit IRC18:43
ayoungEmilienM, yeah, without that line I get an error message like this18:43
*** tiswanso has joined #tripleo18:48
*** trown|lunch is now known as trown18:49
*** rbrady has quit IRC18:54
*** jdob has joined #tripleo18:58
*** dprince has quit IRC18:59
*** dprince has joined #tripleo19:00
*** trown is now known as trown|brb19:02
*** sthillma has joined #tripleo19:04
*** sthillma has quit IRC19:11
*** trown|brb is now known as trown19:13
*** dshulyak has quit IRC19:15
*** rbrady has joined #tripleo19:21
*** oshvartz has joined #tripleo19:21
*** penick has joined #tripleo19:22
*** nico_auv has quit IRC19:22
prometheanfireand derekh is gone19:27
*** weshay_xchat has quit IRC19:27
EmilienMslagle, dprince: maybe you guys can review
*** weshay_xchat has joined #tripleo19:29
dprinceEmilienM: looks fine, but would it be reasonable to set controller_host via hiera instead?19:31
dprinceEmilienM: stylistically then we could just use 'include keystone::wsgi' right?19:31
*** sthillma has joined #tripleo19:31
EmilienMI can do that!19:31
EmilienMlet me update19:31
dprinceEmilienM: cool. Yeah, We've been gradually moving more and more into heira in the undercloud19:32
openstackgerritEmilien Macchi proposed openstack/instack-undercloud: keystone/wsgi: bind on local IP
EmilienMdprince: let's try ^19:34
*** leanderthal has quit IRC19:38
*** Marga_ has quit IRC19:40
dprinceEmilienM: one more thing, lets just use {{LOCAL_IP}} directly in hiera I think19:40
slagleEmilienM: what causes the db_sync's to happen in the puppet modules?19:50
slagleEmilienM: on stable/liberty, i'm seeing that they are failing with tables already created errors19:51
slaglewhich makes me think a race condition with each controller trying to run them at close to the same time19:51
prometheanfirewell, got this working...
prometheanfiredunno who gets to push the button now that derekh is gone19:53
*** rbrady has quit IRC19:54
EmilienMdprince: ok will do19:55
EmilienMslagle: package upgrade / service restart does19:55
EmilienMdb_sync is (or should be) idempotent19:55
slagleEmilienM: yea, i see what my problem is19:56
EmilienMif it's failing, that's a bug in core projects19:56
dprincebnemec: hey, this would be helpful to some of the tarball -> swift container stuff we'd like to use:
slagleit is idempotent19:56
slagleexcept when you run it at the same time on 3 controllers at once19:56
slaglebut...i forgot to pass the pacemaker environment file19:56
EmilienMslagle: that's a race19:56
slagleovercloud_controller.pp is broken19:56
EmilienMslagle: we fixed it in spinalstack by running it on one controller19:56
EmilienMslagle: let me show you, a sec19:57
EmilienMI'll send a patch after that19:57
slaglebut overcloud_controller_pacemaker.pp only runs it on the bootstrap19:57
slaglethere is a guard already in the pacemaker manifest, so that works19:57
slaglewe should probably retire overcloud_controller.pp since it doesnt actually work anymore19:58
slagleat least for ha it doesn't19:58
EmilienMslagle: wait, overcloud_controller.pp is not used in ha scenario, isn't?19:58
*** rbrady has joined #tripleo19:59
slagleEmilienM: it's not used for ha19:59
prometheanfirewho else is in openstack-packages/diskimage-builder project that can +workflow it?19:59
slagleha requires the pacemaker environment file (which i just forgot to do), but that ought to be encoded somewhere20:00
EmilienMslagle: well our workaround was not really clean anyway20:03
*** Marga_ has joined #tripleo20:03
EmilienMwe should run the db_sync only on the first controller node20:03
*** barra204 has quit IRC20:06
ayoungdprince, can I do {{LOCAL_IP}} for the overcloud, too?20:06
dprinceayoung: in hiera, yeah? patch link?20:07
*** jcoufal has quit IRC20:07
ayoungdprince, in
ayoungdprince, I'm not certain what is correct there, but I assume it should mirror the undercloud20:08
ayoungdprince, sorry, that was the old commit.  Here was my horrible hack
*** oshvartz has quit IRC20:12
*** julim_ has quit IRC20:17
*** akrivoka has quit IRC20:24
*** mbound has joined #tripleo20:31
*** eggmaster has quit IRC20:33
*** olap has joined #tripleo20:35
*** weshay_xchat has quit IRC20:36
ayoungdprince, EmilienM what should be after bind_host =>   in the ::keystone::wsgi::apache'  section?  It looks like the undercloud is using ::keystone::wsgi::apache   so should overcloud do the same?20:43
EmilienMayoung: yeah20:44
ayoungEmilienM, should I do the same thing you did for undercloud, and put that in the template?20:44
ayoungEmilienM, or does  does that for us implicitly?20:45
EmilienMayoung: it should do it20:46
openstackgerritEmilien Macchi proposed openstack/instack-undercloud: keystone/wsgi: bind on local IP
EmilienMayoung: see my patch ^20:46
EmilienMwe should rely on hiera when possible20:47
ayoungEmilienM, right,  I should follow suite on the overcloud patch20:47
ayoungso I change puppet/manifests/overcloud_controller.pp  from class { '::keystone::wsgi::apache':20:47
ayoung+    ssl       => false,20:47
ayoungto class { '::keystone::wsgi::apache':20:47
ayoung+    ssl       => false,20:47
ayoungEmilienM, ^^20:48
EmilienMyou can drop it20:48
EmilienMand put it in hiera20:48
EmilienMin controller.yaml20:48
openstackgerritBen Nemec proposed openstack/instack-undercloud: Add ability to auto-generate self-signed certificates
ayoungEmilienM, I'm not quite sure what that means.  DO you want to make that change?20:49
prometheanfirederekh :(20:49
ayoungbnemec, did you look in to using certmonger first?20:49
bnemecayoung: No20:50
ayoungbnemec, much better to make our Cert story around certmonger. It has a selfsigned CA if necessary, and lets us tie in with a real CA if it is available20:50
*** weshay_xchat has joined #tripleo20:50
ayoungdoes cert reup when they are about to expire as well20:50
bnemecI'll put it on my todo list. :-)20:52
* bnemec wants to stop being responsible for SSL anyway20:52
ayoungships by default on Fedora Centos20:53
ayoungavailable on Ubuntu20:53
EmilienMayoung: I can do it if you want20:54
ayoungI might have a blogpost or two for you. let me look20:54
EmilienMayoung: or we can hack on it together20:54
EmilienMayoung: sorry I was kind of busy by our puppet openstack sprint20:54
ayoungEmilienM, I don;t mind learning20:54
ayoungEmilienM, no problem,  you are much in demand20:54
ayoungbnemec, I have this one
ayoungbnemec, Ah  here
ayoungbnemec, its your call.  I don't want to make things tougher, but certmonger is supposed to offload the cert responsibility20:56
ayoungEmilienM, when I run the overcloud deploy, I run openstack overcloud deploy --template /home/stack/tripleo-heat-templates/20:57
ayoungIn order to pick up  I need a different repo20:57
EmilienM is undercloud, just fyi20:57
EmilienMso my patch in THT is supposed to work but we can improve it to use 100% Hiera20:58
ayoungEmilienM, so the template used in ^^ is not also used in overcloud?20:59
slagleok, finally got my local liberty cloud failed the same way CI is failing. now to see why21:00
EmilienMayoung: no21:00
ayoungEmilienM, OK so do we still need something like21:01
ayoungclass { '::keystone::wsgi::apache':21:01
ayoung+    ssl       => false,21:01
ayoung+    bind_host => split(hiera('controller_node_names'), ',')[0],21:01
ayoung+  }21:01
ayoungor does that go into the equivalent of the template?21:01
*** mbound has quit IRC21:01
*** gfidente has quit IRC21:01
dprinceayoung: commented on your patch21:01
dprinceayoung: on a call, then got ping for something else21:02
dprinceayoung: want me to help with this?21:02
ayoungdprince, I would love some help21:02
ayoungI can wait until you are off the call21:02
dprinceayoung: I'm free now21:03
*** julim has joined #tripleo21:03
*** mbound has joined #tripleo21:04
dprinceayoung: so there is a potential problem here in that previously we allowed the networks for the public (port 5000) and admin (port 35357) networks to be on totally separate networks21:05
dprinceayoung: they had separate bind IPs21:05
dprincedsneddon: are you around?21:05
dprinceayoung: anyways, with WSGI there is only 1 bind host now so if we land your patch we also need to correct the network isolation settings (merge them I think) for keystone. See here:
dprinceEmilienM: can you follow this conversation. I wondering if for backwards compat we need keystone WSGI to support 2 bind hosts, right now it just supports the 121:07
dprincedsneddon: ^^^21:08
EmilienMwe actually need the 2 params21:08
dprinceEmilienM: okay, is this done already (I didn't see it in the module)21:08
dsneddondprince, I'm following21:08
dprinceEmilienM: should I push a patch?21:08
dsneddondprince, I thought we were having Keystone bind on one IP, then using HAProxy to present it on different networks, but I'd need to double-check.21:09
dprincedsneddon: no, we need to locally have WSGI run on the separate networks21:10
ayoungdsneddon, I don't think tthat will work for old V2 stuff.  For V3 we onlty need one port anyway21:10
dprincedsneddon: the HAproxy config for this should support whatever we do21:10
ayoungbut V2 has diffferent stuff exposed on 5000 vs 3535721:10
dprinceEmilienM: sec, and I'll push a sample puppet-keystone patch to add the new setting (bind_host)21:10
dprinceEmilienM: once we have that I can update ayoung's patch and we can move this forwards...21:11
ayoungdprince, awesome.  The driving factor here is Keystone in HTTPD is needed for Federation and SSO.21:11
dprinceEmilienM:, still a WIP but once I fix/add tests do you buy this?21:13
dprinceEmilienM: this gives us parity with the local bind port settings when running keystone under eventlet... which is something that matters to TripleO21:14
*** nkinder has joined #tripleo21:14
EmilienMdprince: wait21:15
EmilienMwhere is used bind_host now?21:15
dprincebind_host would be "public"21:15
dprincewhich I think makes sense21:15
EmilienMwe need to add backward compt21:15
dprinceadmin_bind_host would be the admin network21:16
EmilienMif empty -> take same as public21:16
EmilienMyou'll need to patch puppet-openstacklib21:16
EmilienMerr no21:16
EmilienMnevermind my last comment21:16
dprinceEmilienM: backwards compat is fine21:16
dprinceEmilienM: I will fixup the tests and repost21:16
EmilienMwe need to feed the param is empty21:16
*** rcernin has quit IRC21:16
ayoungThat is right21:17
*** trown is now known as trown|outttypeww21:18
ayoungBTW, I would be totally cool with everything just listening on port 443, but that is too big a change for this release21:21
dsneddonayoung, I'm down with that, but we'll need to move to per-service VIPs if we want multiple services on 44321:22
ayoungdsneddon, nope21:22
dsneddonayoung, Oh?21:22
ayoungwe put em all in HTTP and make the URLS deconflict21:22
ayoungI know that morgan former PTL of Keystone is working on a Proof of concept right now that does that21:23
dsneddonayoung, Ah, yeah, I used to use a similar method with Pound instead of HAProxy doing the HTTP decode and sending requests to the right backend.21:23
ayoungdsneddon, yeah, putting all the services into HTTPD actually makes it simpler.  The different ports were an artifact of running them via different processes21:24
ayoungthe ports 5000 and 35357 are both problematic21:24
ayoung5000 is assigned to a different service (Universal Plug and PLay) and 35357 is in the middle of the ephemeral range21:24
dsneddonayoung, Yeah, and running HTTP servers inside of the Python processes was the original legacy application.21:24
ayoungNext release, though.  For now, I just need Keystone in HTTPD21:25
*** jayg is now known as jayg|g0n321:27
*** jayg|g0n3 is now known as jayg21:27
*** jayg is now known as jayg|g0n321:27
*** weshay_xchat has quit IRC21:32
*** jprovazn has quit IRC21:35
*** eggmaster has joined #tripleo21:38
*** weshay_xchat has joined #tripleo21:39
*** weshay_xchat is now known as weshay21:39
slagleEmilienM: hey again21:40
EmilienMslagle: hey james how are you today21:40
slagleEmilienM: i think there might be some sort of regression in puppetlabs-mysql for stable/liberty21:40
ayoungdprince, so what then will go in
slagleoh i'm great21:40
EmilienMdamn, mysql again21:41
EmilienMslagle: is it the same bug as last time? with wsrep?21:41
slagleEmilienM: so in /etc/my.cnf.d/galera.cnf, i'm seeing just a line of "wsrep_notify_cmd" which actually needs to be "wsrep_notify_cmd ="21:41
slaglewe set it to empty string21:41
slaglebut the equals is missing21:41
ayoungbind_host => hiera('keystone::admin_bind_host')?21:41
slagleEmilienM: this is causing mysqld to fail to start21:41
slagleEmilienM: I noticed the module was rebased recently on the stable/liberty branch of opm,
dprinceayoung: almost finished, I'll update your patch too21:44
ayoungdprince, thanks21:44
*** jcoufal has joined #tripleo21:52
*** lifeless has quit IRC21:53
*** lifeless has joined #tripleo21:55
openstackgerritDan Prince proposed openstack/tripleo-heat-templates: puppet: run keystone in wsgi
dprinceayoung: boom ^^^21:56
dprinceEmilienM: ^^^, depends on the puppet-keystone patch21:57
ayoungdprince, if I grab that branch, can I test it, or do I need a seperate repo as well.  Rigjht now, the only code I have from git is tripleo-heat-templates21:58
*** jhenner1 has joined #tripleo21:58
dprinceayoung: you'd need to get the updated puppet-keystone code into your overcloud-full.qcow image first21:59
dprinceayoung: we've got upstream ways to update puppet modules on-the-fly but sadly that hasn't all landed :/21:59
openstackgerritDan Prince proposed openstack/tripleo-heat-templates: puppet: run keystone in wsgi
EmilienMdprince: no need of depends22:01
*** jhenner has quit IRC22:01
dprinceEmilienM: if we don't do depends on we are missing the 'admin_bind_host' setting22:01
dprinceEmilienM: it functionally would be incomplete...22:01
EmilienMah ok22:02
ayoungdprince, I can regen the images.22:02
dprinceexport DIB_REPOREF_puppet_keystone=refs/changes/41/273241/222:03
ayoungdprince, that for
dprinceayoung: source that first ^^^22:04
dprinceayoung: that is for diskimage-builder22:04
ayoungdprince, will tripleo-common/scripts/ --overcloud-images honor that or should I call it by hand?22:05
dprinceayoung: setting it should (I think) get it propigated down to the right places22:05
dprinceayoung: you probably aren't using network isolation though22:05
dprinceayoung: just test the t-h-t patch, if that works you'll be happy22:05
dprinceayoung: trying to save you some time...22:05
dprinceayoung: CI will test it anyways if you don't want to bother, but I'm guessing you want to be hands on with it22:06
ayoungdprince, yeah, plus it is about time I learned the install proces22:06
dprinceayoung: we would value your expertise on having a go at it. But you may want to wait till CI passes it so as not to wast time22:06
*** jhenner1 has quit IRC22:06
dprinceayoung: our CI is a bit intermittent this week, something slipped in :/22:07
ayoungdprince, so I'm just rerunning overcloud deploy with the template dire pointing at your latest22:07
*** marcusvrn_ has quit IRC22:07
dprinceayoung: the non-ha job may pass sooner than the others
ayoungdprince, there was some concernt that the HA job will get messed up by having Keystone running in HTTPD due to the way things get restarted22:08
*** egafford has quit IRC22:08
*** jdob has quit IRC22:08
ayoungTBH, for now I would be happy with at least a non HA success that I could then test Federation against22:08
dprinceayoung: yeah, I would defer to the pacemaker experts on this. For the non-ha puppet t-h-t integration now I like the patch now22:09
*** ooolpbot has joined #tripleo22:10
openstackLaunchpad bug 1538761 in tripleo "stable/liberty HA: mysqld on overcloud failing to start with /usr/libexec/mysqld: option '--wsrep_notify_cmd' requires an argument" [Critical,In progress] - Assigned to James Slagle (james-slagle)22:10
*** ooolpbot has quit IRC22:10
*** dprince has quit IRC22:11
*** lblanchard has quit IRC22:15
bnemecslagle: I think we probably need to just merge
*** lblanchard has joined #tripleo22:15
*** lblanchard has quit IRC22:15
bnemecIt did pass one CI job, and the other two failed on ping test issues so we could probably just pull the trigger.22:17
slaglebnemec: yea i just saw that22:18
slagleit's had a collective pass of all 3 jobs :)22:19
slaglei merged it22:20
openstackgerritMerged openstack/tripleo-heat-templates: Remove empty value for wsrep_notify_cmd
*** jhenner has joined #tripleo22:21
openstackgerritBen Nemec proposed openstack/instack-undercloud: Add ability to auto-generate self-signed certificates
openstackgerritgreghaynes proposed openstack/diskimage-builder: Resolve sylinks when unmounting
*** penick has quit IRC22:27
*** thrash is now known as thrash|pto22:31
*** rpothier has quit IRC22:36
*** penick has joined #tripleo22:37
*** Goneri has quit IRC22:39
*** olap has quit IRC22:45
openstackgerritgreghaynes proposed openstack/diskimage-builder: Resolve sylinks when unmounting
*** jcoufal has quit IRC22:55
*** jdob_lt has joined #tripleo22:59
*** jdob_lt has left #tripleo23:00
*** tiswanso has quit IRC23:00
*** tiswanso has joined #tripleo23:00
*** yuanying_ has quit IRC23:02
openstackgerritBen Nemec proposed openstack/instack-undercloud: Add ability to auto-generate self-signed certificates
*** tiswanso has quit IRC23:05
*** davidlenwell has quit IRC23:06
*** davidlenwell has joined #tripleo23:08
*** ChanServ sets mode: +v davidlenwell23:08
*** weshay has quit IRC23:11
*** yuanying has joined #tripleo23:12
*** chlong has quit IRC23:18
*** chlong has joined #tripleo23:30
*** dmacpher has quit IRC23:43
*** trozet has quit IRC23:44
openstackgerritBen Nemec proposed openstack/instack-undercloud: Add ability to auto-generate self-signed certificates
*** pradk_ has quit IRC23:59

Generated by 2.14.0 by Marius Gedminas - find it at!