Thursday, 2016-01-14

*** rlandy has joined #tripleo00:01
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Add VIP definitions into -no-tunneling env file  https://review.openstack.org/26723800:03
openstackgerritMerged openstack/diskimage-builder: Prune old branches when updating cache  https://review.openstack.org/26608600:03
*** mcornea has quit IRC00:18
*** shivrao has quit IRC00:22
*** shivrao has joined #tripleo00:23
*** shivrao has quit IRC00:25
*** shivrao has joined #tripleo00:25
*** shivrao has quit IRC00:26
*** morazi has quit IRC00:31
*** shivrao has joined #tripleo00:48
*** trozet has joined #tripleo00:51
*** pino|work_ is now known as pino|work01:01
*** pino|work has quit IRC01:26
*** shivrao has quit IRC01:53
*** tiswanso has joined #tripleo01:55
*** rlandy has quit IRC01:56
*** tiswanso has joined #tripleo01:56
*** yamahata has joined #tripleo01:58
*** rlandy has joined #tripleo01:59
openstackgerritgreghaynes proposed openstack/diskimage-builder: Fill out bootloader pkg-map  https://review.openstack.org/26727102:02
*** cody-somerville- has quit IRC02:06
*** cody-somerville- has joined #tripleo02:07
*** gfidente has quit IRC02:12
*** hanchao has joined #tripleo02:16
*** sthillma has quit IRC02:25
*** rickflare2 has joined #tripleo02:50
*** crinkle_ has joined #tripleo02:51
*** crinkle has quit IRC02:52
*** rickflare has quit IRC02:52
*** rickflare2 is now known as rickflare02:52
*** rhallisey has quit IRC02:55
*** Goneri has quit IRC02:56
*** Goneri has joined #tripleo02:57
*** Goneri has quit IRC02:57
*** Goneri has joined #tripleo02:59
*** crinkle_ is now known as crinkle03:08
*** links has joined #tripleo03:17
openstackgerritIan Wienand proposed openstack/diskimage-builder: Fill out bootloader pkg-map  https://review.openstack.org/26727103:19
*** yuanying_ has quit IRC03:34
*** coolsvap|away is now known as coolsvap03:36
*** thrash is now known as thrash|g0ne03:41
*** yamahata has quit IRC03:43
*** yuanying has joined #tripleo03:46
*** rlandy has quit IRC03:46
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack/tripleo-heat-templates: Fix typos  https://review.openstack.org/26512603:48
*** yuanying_ has joined #tripleo03:56
*** yuanying has quit IRC03:59
*** yuanying_ has quit IRC04:00
*** yuanying has joined #tripleo04:06
*** yuanying has quit IRC04:07
*** yuanying_ has joined #tripleo04:07
*** Marga_ has quit IRC04:28
openstackgerritIan Wienand proposed openstack/diskimage-builder: Fill out bootloader pkg-map  https://review.openstack.org/26727104:32
*** Marga_ has joined #tripleo04:44
*** Marga_ has quit IRC04:48
openstackgerritTzu-Mainn Chen proposed openstack/tripleo-specs: Spec for TripleO overcloud deployment REST API  https://review.openstack.org/23043205:09
*** akuznetsov has joined #tripleo05:13
openstackgerritIan Wienand proposed openstack/diskimage-builder: Print unparsable file in pkg-map  https://review.openstack.org/26730505:24
*** stendulker has joined #tripleo05:34
*** anande has joined #tripleo05:35
*** anande has quit IRC05:39
*** trozet has quit IRC05:40
*** masco has joined #tripleo05:45
*** akuznetsov has quit IRC05:49
openstackgerritdharmendra kushwaha proposed openstack-infra/tripleo-ci: Pass environment variables of proxy to tox  https://review.openstack.org/26731005:50
*** jaosorior has joined #tripleo05:54
openstackgerritdharmendra kushwaha proposed openstack/tripleo-common: Pass environment variables of proxy to tox  https://review.openstack.org/26731405:59
*** tzumainn has quit IRC06:00
openstackgerritdharmendra kushwaha proposed openstack/tripleo-docs: Pass environment variables of proxy to tox  https://review.openstack.org/26731606:04
openstackgerritdharmendra kushwaha proposed openstack/tripleo-heat-templates: Pass environment variables of proxy to tox  https://review.openstack.org/26732506:12
openstackgerritdharmendra kushwaha proposed openstack/tripleo-image-elements:  Pass environment variables of proxy to tox  https://review.openstack.org/26732606:16
*** yamahata has joined #tripleo06:19
*** sthillma has joined #tripleo06:20
*** sthillma_ has joined #tripleo06:22
*** sthillma has quit IRC06:25
*** sthillma_ is now known as sthillma06:25
openstackgerritdharmendra kushwaha proposed openstack/tripleo-incubator: Pass environment variables of proxy to tox  https://review.openstack.org/26732806:28
openstackgerritdharmendra kushwaha proposed openstack/tripleo-puppet-elements:  Pass environment variables of proxy to tox  https://review.openstack.org/26733006:30
openstackgerritdharmendra kushwaha proposed openstack/tripleo-specs:  Pass environment variables of proxy to tox  https://review.openstack.org/26733206:32
*** akuznetsov has joined #tripleo06:33
*** akuznetsov has quit IRC06:37
*** jaosorior has quit IRC06:42
*** jaosorior has joined #tripleo06:43
*** yamahata has quit IRC06:43
*** sthillma has quit IRC06:44
*** jaosorior has quit IRC06:45
*** jaosorior has joined #tripleo06:46
*** akuznetsov has joined #tripleo07:08
*** oshvartz has joined #tripleo07:12
*** akuznetsov has quit IRC07:12
*** akuznetsov has joined #tripleo07:13
*** therve has left #tripleo07:14
*** jtomasek has joined #tripleo07:15
*** ukalifon has joined #tripleo07:24
*** dshulyak_ has joined #tripleo07:27
*** paramite has joined #tripleo07:32
*** jprovazn has joined #tripleo07:40
*** rasca has joined #tripleo07:51
*** jaosorior has quit IRC08:05
*** gfidente has joined #tripleo08:11
*** fgimenez has joined #tripleo08:16
*** fgimenez has quit IRC08:16
*** fgimenez has joined #tripleo08:16
*** fgimenez has quit IRC08:18
*** fgimenez has joined #tripleo08:22
*** fgimenez has joined #tripleo08:22
*** aufi has joined #tripleo08:22
*** jaosorior has joined #tripleo08:24
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Add IPv6 Support to Isolated Networks  https://review.openstack.org/23542308:26
*** mcornea has joined #tripleo08:40
*** devvesa has joined #tripleo08:40
*** coolsvap is now known as coolsvap|away08:44
*** mbound has joined #tripleo08:47
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Enable keystone handling of X-Forwarded-Proto header  https://review.openstack.org/26739908:47
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates: Make enabling of controller services configurable.  https://review.openstack.org/26041308:49
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates: Make enabling of controller services configurable.  https://review.openstack.org/26041308:52
*** pino|work has joined #tripleo08:53
devvesaThanks Marios08:55
*** bvandenh has joined #tripleo08:55
devvesaI looked at it yesterday, but I was afraid to do many changes and end up with more conflicts in the future08:55
*** derekh has joined #tripleo08:57
*** hewbrocca has joined #tripleo08:57
*** olap_ has joined #tripleo08:58
*** ifarkas has joined #tripleo09:00
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable X-Forwarded-Proto header for keystone admin endpoint  https://review.openstack.org/26741709:02
*** jistr has joined #tripleo09:03
*** shardy has joined #tripleo09:04
*** mkovacik has quit IRC09:08
paramitegfidente, Hey, you're there?09:08
gfidenteparamite, morning09:09
paramitegfidente, regarding https://review.openstack.org/#/c/225248/7/ceph-cluster-validation-noop.yaml09:09
paramitegfidente, so IIUC I should also remoe the outputs section right?09:09
openstackgerritJaume Devesa proposed openstack/tripleo-heat-templates: MidoNet heat templates  https://review.openstack.org/26041509:09
gfidenteparamite, yeah the noop thing just doesn't do anything so doesn't need to be updated on stack update09:10
paramiteok09:11
gfidentethanks!09:11
openstackgerritMartin Mágr proposed openstack/tripleo-heat-templates: Ceph cluster validation  https://review.openstack.org/22524809:12
openstackgerritJaume Devesa proposed openstack/tripleo-heat-templates: Make enabling of controller services configurable.  https://review.openstack.org/26041309:13
*** yamahata has joined #tripleo09:15
hewbroccashardy: morning!09:15
*** akrivoka has joined #tripleo09:16
shardyhewbrocca: Hi!09:17
hewbroccashardy: could you take a quick look at https://bugzilla.redhat.com/show_bug.cgi?id=129847609:17
openstackbugzilla.redhat.com bug 1298476 in rhel-osp-director "Deployment failed when using IPv6 templates with HA setup" [Urgent,Assigned] - Assigned to shardy09:17
hewbroccaIt looks like a fairly simple ipv6-related validation failure09:20
* shardy looks for heat definition of ip_addr contraint09:21
shardyhewbrocca: it looks like we use oslo netutils to validate the IP09:29
shardyI'm looking into the status of IPv6 support there09:29
hewbroccahee the plot sickens09:30
openstackgerritJaume Devesa proposed openstack/tripleo-heat-templates: MidoNet heat templates  https://review.openstack.org/26041509:31
*** nico_auv has joined #tripleo09:31
hewbroccajaosorior: This might also be a good place to ask about TLS on the keystone endpoint.09:31
jaosoriorshardy, gfidente: Do you guys remember which network was that the keystone admin endpoint is exposed on?09:32
hewbroccagfidente, shardy: jaosorior is trying to get TLS enabled on any externally accessible overcloud endpoint09:32
hewbroccaquestion is, is the keystone admin endpoint on an external network or on internal API or control plane?09:33
gfidentejaosorior, you can customize that in ServiceNetMap09:33
shardyI really hope it's not on an external network by default09:33
gfidenteshardy, even though the client expects it to be ... but this is another story ... it defaults to ctlplane I think09:33
jaosoriorgfidente, shardy: I was just wondering if it's necessary to enable the X-Forwarded-Proto on the loadbalancer09:34
jaosoriorgfidente, shardy: Which is why I set up this CR https://review.openstack.org/#/c/267417/09:35
*** paramite is now known as paramite|afk09:36
*** paramite|afk is now known as paramite09:38
*** mbound has quit IRC09:39
*** fgimenez has quit IRC09:45
*** mbound has joined #tripleo09:46
*** paramite is now known as paramite|afk09:50
*** paramite|afk is now known as paramite09:52
*** mkovacik has joined #tripleo09:55
*** fgimenez has joined #tripleo10:00
*** fgimenez has quit IRC10:00
*** fgimenez has joined #tripleo10:00
*** rebrego has joined #tripleo10:06
*** regebro has quit IRC10:06
*** rebrego is now known as regebro10:06
*** athomas has quit IRC10:14
*** yamahata has quit IRC10:21
*** athomas has joined #tripleo10:23
*** yamahata has joined #tripleo10:23
openstackgerritDerek Higgins proposed openstack/tripleo-common: Adds a simple overcloud tenant vm ping test to tripleo.sh  https://review.openstack.org/24116710:25
derekhmarios: ^ just rebased that, want to test it now that swift is fixed10:25
mariosderekh: great thanks man10:26
*** yamahata has quit IRC10:28
*** teju has joined #tripleo10:29
teju Hi, I am installing RHEL OS P7. I ran the command to deploy overcloud. The overcloud servers are not getting DHCP IP . How to debug this ?10:30
*** electrofelix has joined #tripleo10:34
*** electrofelix has quit IRC10:38
*** mgould has joined #tripleo10:43
*** regebro has quit IRC10:50
*** mgould has quit IRC11:05
jaosoriorCan someone check this commit out? https://review.openstack.org/#/c/267012/ it's pretty short :D11:07
openstackgerritJaume Devesa proposed openstack/puppet-tripleo: Remove all 'validate_array' statements  https://review.openstack.org/26747411:07
openstackgerritMerged openstack/tripleo-heat-templates: Network Isolation support for containerized compute  https://review.openstack.org/26435011:09
*** fgimenez has quit IRC11:10
*** fgimenez has joined #tripleo11:10
*** fgimenez has quit IRC11:10
*** fgimenez has joined #tripleo11:10
*** electrofelix has joined #tripleo11:15
*** lucas-dinner is now known as lucasagomes11:18
*** mgould has joined #tripleo11:19
*** regebro has joined #tripleo11:22
*** stendulker_ has joined #tripleo11:24
*** masco has quit IRC11:25
*** stendulker has quit IRC11:28
*** Marga_ has joined #tripleo11:29
*** stendulker_ has quit IRC11:29
*** mgould has quit IRC11:32
openstackgerritMerged openstack/tripleo-common: Remove stable/liberty pin  https://review.openstack.org/26585011:33
*** mgould has joined #tripleo11:33
*** Marga_ has quit IRC11:34
*** d0ugal has quit IRC11:40
*** d0ugal has joined #tripleo11:45
*** d0ugal is now known as Guest4395011:45
*** Guest43950 has quit IRC11:46
*** d0ugal_ has joined #tripleo11:46
*** d0ugal_ is now known as d0ugal11:47
*** d0ugal has joined #tripleo11:47
*** d0ugal has quit IRC11:55
*** d0ugal has joined #tripleo11:55
*** d0ugal has joined #tripleo11:55
*** d0ugal has quit IRC11:55
*** d0ugal has joined #tripleo11:58
*** cmyster has quit IRC12:11
*** weshay_xchat has joined #tripleo12:17
jaosoriorjistr: Got time for a quick review? https://review.openstack.org/#/c/267012/ it's exactly the same as other patches that have already been approved, that enable similar headers for haproxy12:19
*** mgould has quit IRC12:19
*** mgould has joined #tripleo12:20
*** mburned is now known as mburned_out12:27
*** mburned_out is now known as mburned12:28
*** trown|outttypeww is now known as trown12:30
*** panda_ has joined #tripleo12:30
jistrjaosorior: lgtm, we should get the HA jobs green on these, triggered a recheck. They seem somewhat unstable lately.12:31
*** rhallisey has joined #tripleo12:32
jaosoriorjistr: thanks dude12:33
*** cmyster has joined #tripleo12:38
*** cmyster has quit IRC12:38
*** cmyster has joined #tripleo12:38
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: Allow to enable IPv6 on Corosync  https://review.openstack.org/26707312:40
EmilienMshardy, slagle: I noticed for some jobs, we don't have overcloud logs anymore. Look at gate-tripleo-ci-f22-nonha for example and gate-tripleo-ci-f22-nonha12:44
EmilienMis it expecte?12:44
EmilienMexpected?12:44
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates: Fixup swift device string to delimit the ipv6 address with []  https://review.openstack.org/26752312:44
shardyEmilienM: No, that sounds like a bug12:45
EmilienMI noticed that for different patches12:45
EmilienMthe job is not failing, it's passing12:45
EmilienMbut no logs12:45
shardyderekh: ^^ any ideas?12:45
EmilienMonly undercloud logs12:45
shardyI know derekh fixed a similar issue sometime ago12:45
derekhshardy: EmilienM looking12:46
shardyEmilienM: one issue is we don't currently test the deployed overcloud12:46
derekhlol12:46
shardymarios has been looking at fixing that (we previously did test it)12:46
derekhfound the problem12:46
shardyderekh: nice, that was quick ;)12:47
derekhwe get the logs off the overcloud after the test has run12:47
derekh2016-01-14 10:44:25.475 | tripleo.sh -- Overcloud delete12:47
shardyhaha! :D12:47
shardyI guess that makes it my fault then12:47
derekh;-)12:47
EmilienMcool12:48
EmilienMhappy to help12:48
derekhshardy: maybe we could manually run postci before the delete and remove the bash EXIT trap12:48
shardyEmilienM: thanks for pointing it out, we'll figure out how to re-order things and fix it12:49
derekhshardy: we would be missing the logs for the delete though12:49
EmilienMshardy: you rocks12:49
shardyderekh: I think that's OK, as any delete error is very likely to be due to undercloud errors12:49
shardywe don't have any DELETE actions on the overcloud except RHEL unregistration, which isn't tested in CI12:49
derekhshardy: yup, the postci function also gets the undercloud logs so we may need to split it out into 2 seperate functions12:50
mariosshardy: o/ if you have a minute, grateful for comments on https://review.openstack.org/#/c/267523/ - seems like a really wasteful way when all i really want is the altenative ip address string - but thought i'd get it out for comments as is12:50
shardymarios: so, the only difference is the []'s around the IP?12:52
mariosshardy: yeah, so far.. i mean i don't know if that object_store_swift_devices will change too, but we can ignore for now12:52
marios(in the config i mean)12:52
*** tiswanso has quit IRC12:54
mariosshardy: didn't mean to corner you right.now just if you get a chance/there is an obvious/harder/better/faster/stronger way please comment12:54
shardymarios: Ok, let me try a couple of things - I think I'd personally go for moving the str_replace out of controller.yaml, and just returning the IP12:54
shardymarios: ack, let me think on it a bit, there may well be :)12:55
mariosshardy: grateful for any help when you get the time, thanks12:55
*** mburned is now known as mburned_out12:59
*** jdob has joined #tripleo13:00
*** mburned_out is now known as mburned13:00
*** hanchao has quit IRC13:06
*** jcoufal has joined #tripleo13:07
*** tzumainn has joined #tripleo13:13
*** dprince has joined #tripleo13:17
*** gchamoul is now known as gchamoul_afk_doc13:17
*** thrash|g0ne is now known as thrash13:18
*** jayg|g0n3 is now known as jayg13:21
*** paramite is now known as paramite|afk13:35
*** trozet has joined #tripleo13:41
*** julim has quit IRC13:42
*** morazi has joined #tripleo13:47
openstackgerritRyan Hallisey proposed openstack/tripleo-heat-templates: Allow container template to recognize an update  https://review.openstack.org/26536813:52
openstackgerritMerged openstack/tripleo-common: Remove stable/liberty pin  https://review.openstack.org/26585113:54
*** paramite|afk is now known as paramite14:04
*** akuznetsov has quit IRC14:12
*** olap_ has quit IRC14:15
gfidentederekh, regarding https://review.openstack.org/#/c/260466/14:16
gfidenteI think I need a little help making that happen for the stable/liberty submissions14:16
gfidentehave a pointer?14:16
jaosoriorgfidente: Got some a sec to review this one? https://review.openstack.org/#/c/267399/ really need it for the TLS work14:18
*** rlandy has joined #tripleo14:20
*** lucasagomes is now known as lucas-hungry14:20
gfidentejaosorior, it's fine but given has to do a recheck wanted to ask14:20
gfidentecan we merge it into the above keystone_config thing?14:21
gfidenteI am not sure if puppet is actually doing a merge there14:22
jaosoriorgfidente: what do you mean?14:22
gfidenteL177 > L17414:22
jaosoriorgfidente: I'm not sure if it's possible. that keystone_config provider inherits from ini_setting, which refers to one setting from the ini file14:23
jaosoriorgfidente: It would be a different story if we would be using ::keystone::config, but I didn't want to introduce the usage of a new resource in that manifest14:24
jaosoriorgfidente: this is the relevant declaration of the resource I'm using in that patch https://github.com/openstack/puppet-keystone/blob/master/lib/puppet/provider/keystone_config/ini_setting.rb14:25
gfidenteyeah I see what you mean, we were indeed using the ::cinder_config class for cinder for example14:26
gfidentecause that stuff is pretty much static and we have it in the hieradata for cinder14:27
openstackgerritBrad P. Crochet proposed openstack/instack-undercloud: Clean out os-refresh-config on every run  https://review.openstack.org/26640414:27
*** teju has quit IRC14:28
jaosoriorgfidente: Seems to me that cinder is using the same type of resource as keystone is using, that cinder_config provider https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/manifests/overcloud_controller.pp#L40814:28
gfidenteso how about moving it into hieradata and use keystone::config ?14:28
gfidenteyeah but we do the ::cinder_config import14:28
gfidentesorry, cinder::confg include14:29
gfidenteso ideally I think that would be nicest for keystone too14:29
gfidentesounds valid?14:29
gfidenteand move the existing param into hieradata too14:29
jaosoriorgfidente: Can you point me to where that's being done for cinder? I can try that out14:29
gfidentemanifests/overcloud_volume.pp14:30
gfidentehieradata/volume.yaml14:30
jaosoriorgfidente: I see now. Will try that out in a bit14:31
*** bvandenh has quit IRC14:31
gfidentejaosorior++14:31
gfidentetks14:31
*** tiswanso has joined #tripleo14:33
derekhgfidente: Do you only want to run the --overcloud-update for the stable jobs?14:37
*** julim has joined #tripleo14:38
shardyderekh: I don't think we can, because we need to catch things landing in master which break upgrades14:39
*** jistr has quit IRC14:40
*** gchamoul_afk_doc is now known as gchamoul14:40
gfidentederekh, I see it now, so at it is now it'll do for both master and stable, did I get it right?14:40
shardyRelatedly, can someone please approve https://review.openstack.org/#/c/260144/14:40
shardythat enables building liberty images on a master undercloud14:40
shardywhich we could then deploy, and upgrade14:40
derekhshardy: so we want to run it for both master and stable jobs? gfidente if thats the case then what you have done will do it14:40
*** jistr has joined #tripleo14:41
gfidentederekh, yeah see it now why, ack14:41
shardyderekh: I think so - we can't land any master commits which break upgrades, and we have to prevent invalid backports breaking upgrades?14:41
shardyderekh: I can't see a way to do that without gating on both14:41
openstackgerritMerged openstack/tripleo-common: Update tripleo.sh to enable repo path override  https://review.openstack.org/26014414:42
*** jaosorior has quit IRC14:42
*** dprince has quit IRC14:42
*** jaosorior has joined #tripleo14:43
*** jaosorior has quit IRC14:44
*** jaosorior has joined #tripleo14:45
*** lblanchard has joined #tripleo14:46
*** paramite is now known as paramite|afk14:50
derekhshardy: ack14:51
*** links has quit IRC14:52
*** paramite|afk is now known as paramite14:52
*** egafford has joined #tripleo14:58
*** egafford has quit IRC14:58
*** dshulyak_ has quit IRC15:00
openstackgerritLeif Madsen proposed openstack/tripleo-docs: Update README files for TripleO documentation  https://review.openstack.org/26719915:09
*** paramite is now known as paramite|afk15:09
*** pradk has joined #tripleo15:11
*** jcoufal has quit IRC15:11
*** bvandenh has joined #tripleo15:12
*** julim_ has joined #tripleo15:13
*** trozet has quit IRC15:13
*** trozet_ has joined #tripleo15:13
*** trozet_ is now known as trozet15:14
*** julim has quit IRC15:15
*** jprovazn has quit IRC15:16
*** dprince has joined #tripleo15:17
leifmadsenhey folks, I'm working on updating the documentation for tripleO on how to generate the documentation :)15:17
leifmadsenthere are both README.md and README.rst files; wonder if there is any reason I shouldn't just nuke the README.md file?15:18
*** gchamoul is now known as gchamoul|afk|brb15:18
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Enable keystone handling of X-Forwarded-Proto header  https://review.openstack.org/26739915:18
leifmadsenboth were there, but basically empty, so I added documentation and generated the other one, but it does just duplicate the information15:18
jaosoriorgfidente: Thanks for the feedback on https://review.openstack.org/#/c/267399 I submitted a patch using the approach you proposed15:20
openstackgerritLeif Madsen proposed openstack/tripleo-docs: Update README files for TripleO documentation  https://review.openstack.org/26719915:20
jaosoriorshardy: got some cycles to review this https://review.openstack.org/#/c/267399/2 ?15:21
*** lucas-hungry is now known as lucasagomes15:24
shardyjaosorior: Yeah, I've seen that and the puppet-tripleo patch, was waiting on CI before reviewing15:25
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: pacemaker/vips: do not set cidr_netmask when ipv6  https://review.openstack.org/26764715:26
*** paramite|afk is now known as paramite15:29
*** paramite is now known as paramite|afk15:30
*** jcoufal has joined #tripleo15:31
*** akuznetsov has joined #tripleo15:31
*** jang has quit IRC15:32
*** jang has joined #tripleo15:33
devvesashardy, marios: the patch https://review.openstack.org/#/c/260413/ has failed15:40
devvesaI see in the logs the following:15:40
devvesahttp://fpaste.org/310768/27860311/15:40
devvesaany idea?15:41
shardydevvesa: it looks like the ceph and HA jobs both took too long and/or got stuck and timed out as a result15:43
devvesaso it is not a problem in tripleo-common?15:44
devvesabecause of this 'timing' call?15:44
shardydevvesa: I'm not sure where that's coming from, but it's not tripleo.sh15:46
*** gchamoul|afk|brb is now known as gchamoul15:46
shardywe probably need to run set -x to see where that actually came from15:46
devvesaok. i'll recheck again then15:46
shardybut the line before says what the problem was - the delete timed out15:46
devvesa(second time today :/)15:46
shardydevvesa: well, it's a real bug in this case, just not with this patch15:46
devvesaoh15:47
shardyrechecking may allow progress, but the root cause is something caused the stack-delete to fail15:47
shardy2016-01-14 14:31:44.207 | | Controller15:49
shardyit looks like we got stuck trying to delete a server with nova15:49
shardydevvesa: I'll raise a bug and we can recheck against that15:49
devvesaok, thanks shardy15:49
shardybut it'd be good to understand the root cause a bit better if possible15:49
* shardy looks at logs15:50
devvesai have a stable/liberty env15:50
devvesai'll try it15:50
*** ayoung has quit IRC15:52
shardydevvesa: thanks - I know it's annoying, but this is actually good, it's the sort of spurious failures we wanted to catch and fix with the delete test in CI15:52
*** akuznetsov has quit IRC15:59
*** akuznetsov has joined #tripleo15:59
*** akuznetsov has quit IRC16:00
*** jprovazn has joined #tripleo16:01
beaglesheh... I was just about to write a note about overcloud delete's timing out... too slow16:05
*** panda_ has quit IRC16:06
beagleserror: engineer beagles report-issue timed out16:06
*** panda_ has joined #tripleo16:06
*** yamahata has joined #tripleo16:06
*** ayoung has joined #tripleo16:09
*** egafford has joined #tripleo16:10
openstackgerritLeif Madsen proposed openstack/tripleo-docs: Update README files for TripleO documentation  https://review.openstack.org/26719916:10
shardydevvesa: https://bugs.launchpad.net/tripleo/+bug/153421316:12
openstackLaunchpad bug 1534213 in tripleo "Timeout trying to delete overcloud stack in CI" [Undecided,New]16:12
shardyAFAICT the root cause is for HA jobs, the stack-delete takes 11 minutes :-O16:12
shardywhich is either something we have to fix, or we have to only do the delete test on the nonha job16:12
shardymy local nonha deployments take ~40 seconds to delete16:13
*** yamahata has quit IRC16:18
*** yamahata has joined #tripleo16:18
*** rasca has quit IRC16:24
*** ukalifon has quit IRC16:27
*** rook-desktio has joined #tripleo16:27
*** ukalifon has joined #tripleo16:28
*** rook-desktio is now known as rook-desktop16:29
*** ukalifon has quit IRC16:32
*** aufi has quit IRC16:36
devvesaThanks shardy16:37
*** mkovacik_ has joined #tripleo16:37
EmilienMshardy: it seems like latest successful HA job was 8h ago16:37
EmilienM(in master)16:37
bnemecYeah, I think we have resource issues with the HA job.16:37
bnemecI suspect we're right up against the memory limit on the undercloud node, which probably is why the delete is taking so long.16:38
shardydid we bump the undercloud CPUs up in the testenv yet?16:38
*** mkovacik has quit IRC16:38
bnemecNot that I know of.  derekh ^ ?16:38
bnemecI think we also likely need more memory.16:38
shardybnemec: Yeah, seems like it, deletes shouldn't take anywhere near that long16:39
jistrwe may need to revisit this then https://review.openstack.org/#/c/243148/16:39
* jistr re-reads the comments16:39
bnemecThere are also a bunch of ha job failures where the stack create times out due to unanswered messages, which I suspect is because a service has OOM'd.16:39
shardydo we configure the CI undercloud with swap?16:39
bnemecI doubt it.16:40
shardyI'd expect to see stuff get killed if we were out of memory otherwise16:40
derekhwe didn't change anything, we just talked about it in the meeting this week, are we all in agreement to try it? If so I'll give it a go16:40
shardyderekh: +116:40
derekhshardy: checking about the swap16:40
*** olap_ has joined #tripleo16:40
bnemecderekh: Yeah, I think we need to.  The ha job is in a bad way right now, and it looks like resource starvation.16:40
shardyderekh: we have a baseline from the logs in bug 1534213, I've got those locally so can compare before/after for the timings16:41
openstackbug 1534213 in tripleo "Timeout trying to delete overcloud stack in CI" [Undecided,New] https://launchpad.net/bugs/153421316:41
shardyI might actually script something to scrape some more data and plot it16:41
derekhshardy: zero swap on the undercloud16:41
shardyderekh: ack16:42
shardya single engine worker would also be painfully slow I expect16:42
shardyderekh: we fixed that in recent heat master, but we're still pinned to some pretty old stuff right?16:42
derekhok, I'll update the CPU's either tonight or over the weekend, things will randomly fail when I do and be down for a few hours probably16:42
derekhshardy: yup our pin is fairly old at the moment16:43
shardyderekh: https://review.openstack.org/#/c/259172/ may help as even with 1 CPU we run more than one worker16:44
derekhshardy: ok16:45
*** dprince has quit IRC16:45
*** dprince has joined #tripleo16:46
*** akuznetsov has joined #tripleo16:46
shardybnemec: the unanswered messages are also caused by a single engine worker, but yeah, OOMing something would also do it ;)16:47
shardyhttps://bugzilla.redhat.com/show_bug.cgi?id=1290949 has further evidence to that effect16:48
openstackbugzilla.redhat.com bug 1290949 in openstack-heat "rhel-osp-director: re-ran the deployment command: "Stack failed with status: resources.Controller: MessagingTimeout: resources[0]: Timed out waiting for a reply to message ID 863d0fbc6ce24cd288074d901d1a6e64 ERROR: openstack Heat Stack update failed."" [Unspecified,Assigned] - Assigned to sbaker16:49
*** dprince has quit IRC16:50
gfidenteshardy, another upgrade attempt in CI ended with same error16:51
*** dprince has joined #tripleo16:51
gfidentehttp://logs.openstack.org/66/260466/3/check-tripleo/gate-tripleo-ci-f22-ha/3eab261/console.html#_2016-01-14_16_43_42_20816:51
*** dshulyak_ has joined #tripleo16:52
gfidentedo you know anything about https://bugzilla.redhat.com/show_bug.cgi?id=1290949 ?16:52
openstackbugzilla.redhat.com bug 1290949 in openstack-heat "rhel-osp-director: re-ran the deployment command: "Stack failed with status: resources.Controller: MessagingTimeout: resources[0]: Timed out waiting for a reply to message ID 863d0fbc6ce24cd288074d901d1a6e64 ERROR: openstack Heat Stack update failed."" [Unspecified,Assigned] - Assigned to sbaker16:52
gfidentethat seems to be the issue we're hitting16:52
shardygfidente: Yeah, I think we have to give the undercloud some more resources, and/or pull in that heat patch to increase the workers16:52
*** paramite|afk is now known as paramite16:52
shardygfidente: see the backscroll, we just discussed this exact issue ;)16:53
gfidenteah sorry :(16:53
*** lblanchard has quit IRC16:53
shardywell, I assume it's the same, log error looks very similar16:53
bnemecI was going to say we could update the ci pin, but I see another issue has crept in since I last looked at https://etherpad.openstack.org/p/delorean_master_current_issues16:54
shardygfidente: no worries - the problem is we're throwing like 100+ stacks at a single python heat-engine process, which isn't smart enough to know not to RPC requests back to itself16:54
shardyso, basically that one process DoSes itself with RPC requests16:54
*** lblanchard has joined #tripleo16:58
*** mbound has quit IRC16:58
*** akuznetsov has quit IRC17:00
trownbnemec: ya... that keeps happening :p17:02
shardytrown: Do you have this fix? https://review.openstack.org/#/c/263643/17:03
trownbnemec: I am a bit stumped on that one too... not sure where to file the bug... is it a puppet-heat issue? a THT issue17:03
trownshardy: let me check that looks relevant17:03
shardytrown: we've fixed a few keystone v3 related things in heat recently, because I updated the default devstack config to stop using v2.0 which flushed out some bugs17:03
shardythere is also an outstanding puppet-heat patch which migrates to non-v2 config, but it should be mandatory, the old config should still work17:04
trownshardy: do you have a link to that review handy? I could at least test if that fixes things17:05
*** oshvartz has quit IRC17:06
shardyhttps://review.openstack.org/#/c/261326/ and https://review.openstack.org/#/c/265865/17:06
shardytrown: ^^17:06
shardybasically (I hope) when those land, the puppet configured heat will look same/similar to the way I configured it via devstack to work with v317:07
shardythat said, none of those should be essential, that error you're seeing is probably due to heat not passing the domain, e.g Rabi's patch I referenced17:07
shardyit you do have that, I'll investigate further17:07
trownshardy: I have that :(17:08
shardytrown: OK, can you paste the heat.conf somewhere, or can I get it from the logs?17:08
trownshardy: https://ci.centos.org/artifacts/rdo/jenkins-trown-poc-rdo_manager-quickstart-mitaka-delorean-717:08
trownhttps://ci.centos.org/artifacts/rdo/jenkins-trown-poc-rdo_manager-quickstart-mitaka-delorean-7/undercloud/etc/heat/heat.conf.gz17:09
*** fgimenez has quit IRC17:10
shardytrown: Ok, so you're still using the legacy mode, which is to steal config from the keystone_authtoken section17:11
shardythat may need https://review.openstack.org/#/c/267275/17:11
* shardy looks at the logs again17:12
shardyhttp://fpaste.org/310838/79160914/17:13
shardytrown: FYI, that's how devstack now configures heat (note no versions except for ec2authtoken)17:14
trownshardy: so looks like we are missing the *domain_id in the keystone_authtoken section17:15
shardytrown: Yeah17:15
shardyit's configured to use v3, but no domains are added17:15
shardyand the implicit "default" domain isn't valid if you move to v317:15
devvesaping marios17:15
shardyso either the auth_uri needs to be switched back to v2.017:15
shardyor, those puppet fixes have to land17:16
shardyI think17:16
trownshardy: ok.. I tried manually switching auth_uri to v2.0 and hit other issues (I do not have any logs because it was just a failed hack attempt)17:16
shardyHmm, looking at my local undercloud, the auth_uri is v317:17
shardyand that works17:17
trownshardy: I will spin up an environment shortly and mess with the heat config to see if I can add more information17:18
trownshardy: btw... tripleo-heat-templates on stable liberty broke instance launching in the overcloud again :(17:18
trownshardy: still need to investigate what happened, but I just tried current delorean with downgraded THT from current-passed-ci and it worked...17:19
shardytrown: your error is when handling a signal, and it's using the legacy config, so https://review.openstack.org/#/c/267275/ may fix it17:19
shardythat patch has other issues, in that it assumes "default", but if it works please let me know and we'll work out how to land something equivalent to it17:20
trownshardy: awesome! I will try that first. thanks17:20
*** akuznetsov has joined #tripleo17:23
shardytrown: thanks, I -1'd that patch, but it should work fine provided the user in the keystone_authtoken section is actually in the default domain17:25
shardyI think maybe that's your issue, because the signal will be coming from the heat_stack domain, which will be wrong17:26
shardytrown: re t-h-t liberty: GAH!17:27
shardy:(17:27
*** jistr has quit IRC17:29
trownping test would catch it :)17:30
shardyyeah, we need to land that asap17:30
*** paramite has quit IRC17:33
*** eil397 has joined #tripleo17:45
*** hewbrocca has quit IRC17:48
*** regebro has quit IRC17:49
*** regebro has joined #tripleo17:49
*** alop has joined #tripleo17:50
openstackgerritLeif Madsen proposed openstack/tripleo-docs: Updated /etc/hosts file definition  https://review.openstack.org/23559917:53
*** mkovacik_ has quit IRC17:56
*** mkovacik_ has joined #tripleo17:56
*** mkovacik_ has quit IRC17:56
*** trown is now known as trown|lunch17:56
*** akuznetsov has quit IRC17:59
openstackgerritDoug Hellmann proposed openstack/python-tripleoclient: Revert "Pass environment variables of proxy to tox"  https://review.openstack.org/26774318:00
openstackgerritRyan Hallisey proposed openstack/tripleo-docs: WIP: Docs for containerized compute node  https://review.openstack.org/25474318:01
*** derekh has quit IRC18:05
*** shivrao has joined #tripleo18:05
*** devvesa has quit IRC18:08
openstackgerritMerged openstack/diskimage-builder: Fill out bootloader pkg-map  https://review.openstack.org/26727118:10
openstackgerritDenis Egorenko proposed openstack/puppet-tripleo: Update to pull in rspec-puppet-facts  https://review.openstack.org/26456118:13
openstackgerritgreghaynes proposed openstack/diskimage-builder: Perform a booting test for our images  https://review.openstack.org/20463918:14
trown|lunchshardy: confirmed https://review.openstack.org/#/c/267275/1 resolves the issue deploying with master, commented there18:17
trown|lunchaww... rats... I dont think it did actually, deploy just timed out while I was gone which tripleoclient exits 0 for18:20
*** mgould has quit IRC18:22
shardytrown|lunch: ack, let me pull latest heat master onto my undercloud18:23
shardythe signalling works fine on devstack, but then the config is different there18:23
*** nico_auv has quit IRC18:25
*** therve has joined #tripleo18:27
*** SlickNik has left #tripleo18:29
*** akrivoka has quit IRC18:30
*** electrofelix has quit IRC18:31
*** jaosorior has quit IRC18:33
*** trown|lunch is now known as trown18:33
trownshardy: actually that patch does fix it if modified to use the domain we are creating for heat: http://fpaste.org/310898/45279638/18:33
trownshardy: I hardcoded the domain id of that heat_stack domain instead of default, and that fixed the issue... any idea why we are creating a special domain for heat?18:34
trownwhich then allows us to fail on configuring Sahara...18:39
trownprogress18:39
shardytrown: http://hardysteven.blogspot.co.uk/2014/04/heat-auth-model-updates-part-2-stack.html18:42
shardytrown: we create a special domain for the credentials we put inside instances18:43
shardye.g the signalling back from all the SoftwareDeployments and the os-collect-config polling18:43
trownoh, nice, I will read both those posts18:44
trownshardy: so hardcoding default, or even falling back to default wont by us much... I guess we need those puppet patches18:45
trowns/by/buy/18:46
shardytrown: Yeah, but this fallback path should still work18:46
shardytrown: I'm a bit confused tbh, that domain should I think be that of the trustee (which is the heat service user in this case)18:46
shardyhttps://review.openstack.org/#/c/263643/18:48
shardytrown: that patch is actually partially reverting that, where, for a long time, we used "default"18:49
*** sthillma has joined #tripleo18:49
shardytrown: I'm dropping off for a while now, if you like please raise a heat bug and/or send me an email, and I'll try to investigate further in the morning18:49
shardyI may also be back in a couple of hours, if I'm awake enough18:50
trownshardy: thanks for all the help, I am slightly less lost now :)18:50
*** ifarkas has quit IRC18:50
shardytrown: np, thanks for the patience, sorry if heat has broken things in our efforts to v3-ify things18:51
shardythose puppet patches probably will fix it, but it'd also be good to understand if/where we've broken the legacy mode18:51
trownif only heat were the only thing to break tripleo :)18:51
shardyhehe18:51
*** Marga_ has joined #tripleo18:52
*** shardy is now known as shardy_afk18:56
*** sthillma has quit IRC19:00
gfidentelet's just say that the project is quite ambitious19:01
trown:)19:01
*** SlickNik has joined #tripleo19:11
*** yamahata has quit IRC19:21
*** sthillma has joined #tripleo19:21
*** bvandenh has quit IRC19:23
*** gfidente is now known as gfidente|afk19:25
*** sthillma has quit IRC19:30
*** sthillma has joined #tripleo19:31
*** shardy_afk has quit IRC19:31
*** shardy_afk has joined #tripleo19:32
*** mcornea has quit IRC19:39
*** rlandy has quit IRC19:46
*** rlandy has joined #tripleo19:49
*** yamahata has joined #tripleo19:50
*** bvandenh has joined #tripleo19:51
*** lucasagomes is now known as lucas-dinner19:55
*** shardy_afk has quit IRC19:55
*** shardy_afk has joined #tripleo19:57
*** eil397 has quit IRC19:58
*** yamahata has quit IRC19:58
*** yamahata has joined #tripleo19:58
*** ayoung has quit IRC19:59
*** shivrao has quit IRC20:00
*** bvandenh has quit IRC20:00
*** shivrao has joined #tripleo20:02
*** panda_ has quit IRC20:05
*** panda_ has joined #tripleo20:06
*** dshulyak_ has quit IRC20:08
openstackgerritJohn Trowbridge proposed openstack/tripleo-heat-templates: Revert "Wire Neutron ML2 plugin and OVS agent settings as arrays" Revert "Enable configuration of Neutron QoS"  https://review.openstack.org/26779620:12
*** ayoung has joined #tripleo20:16
*** eil397 has joined #tripleo20:16
*** penick has joined #tripleo20:18
*** Marga_ has quit IRC20:18
trownslagle: I just noticed that there are already individual reverts posted for ^, you -1 those because the reason to do the reverts was not clear, but now there is a clear reason20:18
trownslagle: should I abandon my squashed revert and comment on the individual ones?20:18
*** Marga_ has joined #tripleo20:18
*** shardy_afk has quit IRC20:18
*** eil397 has quit IRC20:19
*** eil397 has joined #tripleo20:19
*** shardy_afk has joined #tripleo20:19
openstackgerritRyan Hallisey proposed openstack/tripleo-docs: Docs for containerized compute node  https://review.openstack.org/25474320:19
*** bvandenh has joined #tripleo20:20
slagletrown: whichever really. my main reason for -1'ing is b/c i want to know what is failing20:22
*** penick has quit IRC20:22
trownslagle: ya, I do not understand the individual ones reasoning... it doesnt seem like the same as mine20:23
slagletrown: yea, so we can go with yours20:23
*** shardy_afk has quit IRC20:23
trownslagle: ok... I am betting this issue is on master as well20:23
slagletrown: but can you capture what launching instances fails?20:23
trownslagle: ya, neutron port binding fails, I will make a bug for it20:24
*** dshulyak_ has joined #tripleo20:24
slagletrown: ok, thanks20:24
slaglebeagles: fyi ^20:24
slagletrown: we can get beagles to look into it :)20:24
*** shardy_afk has joined #tripleo20:26
*** mbound has joined #tripleo20:28
*** dsneddon is now known as dsneddon_biab20:34
*** morazi has quit IRC20:38
*** penick has joined #tripleo20:38
*** trown has quit IRC20:39
*** thrash has quit IRC20:39
*** dshulyak_ has quit IRC20:41
*** thrash has joined #tripleo20:41
*** thrash has joined #tripleo20:41
*** morazi has joined #tripleo20:49
*** trown_alt has joined #tripleo20:54
*** trown_alt is now known as trown20:54
*** jprovazn has quit IRC20:55
*** mkovacik_ has joined #tripleo20:59
*** dprince has quit IRC21:01
*** gfidente|afk has quit IRC21:01
*** gfidente|afk has joined #tripleo21:02
*** mburned is now known as mburned_out21:05
openstackgerritJohn Trowbridge proposed openstack/tripleo-heat-templates: Configure cinder [keymgr]/encryption_auth_url  https://review.openstack.org/26781521:10
*** gfidente|afk has quit IRC21:11
*** mbound has quit IRC21:14
*** mbound has joined #tripleo21:15
*** pino|work has quit IRC21:16
*** ayoung has quit IRC21:20
*** dsneddon_biab is now known as dsneddon21:28
*** StevenK has quit IRC21:30
*** akuznetsov has joined #tripleo21:30
*** StevenK has joined #tripleo21:31
*** akuznetsov has quit IRC21:36
*** shakamunyi has quit IRC21:40
*** eil397 has quit IRC21:42
*** rickflare2 has joined #tripleo21:49
*** mburned_out is now known as mburned21:52
*** rickflare has quit IRC21:52
*** rickflare2 is now known as rickflare21:52
*** jtomasek has quit IRC21:52
*** cwolferh has quit IRC21:54
openstackgerritJohn Trowbridge proposed openstack/tripleo-heat-templates: Configure cinder [keymgr]/encryption_auth_url  https://review.openstack.org/23910921:57
*** shardy_afk has quit IRC22:00
*** ayoung has joined #tripleo22:01
*** tiswanso has quit IRC22:05
openstackgerritBen Nemec proposed openstack/tripleo-heat-templates: Have Horizon use internal endpoints  https://review.openstack.org/26783622:08
openstackgerritBen Nemec proposed openstack/tripleo-heat-templates: Have Horizon use internal endpoints  https://review.openstack.org/26783722:09
*** cwolferh has joined #tripleo22:13
*** lblanchard has quit IRC22:15
*** julim_ has quit IRC22:20
*** trown has quit IRC22:22
*** bvandenh has quit IRC22:23
*** eil397 has joined #tripleo22:26
*** jcoufal has quit IRC22:28
*** bvandenh has joined #tripleo22:29
*** ayoung has quit IRC22:37
*** bvandenh has quit IRC22:40
*** egafford has quit IRC22:47
*** mburned is now known as mburned_out23:03
*** pradk has quit IRC23:04
*** ayoung has joined #tripleo23:11
*** pradk_ has joined #tripleo23:15
*** olap_ has quit IRC23:16
*** pradk_ has quit IRC23:40
*** mburned_out is now known as mburned23:46

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!