Wednesday, 2016-09-21

*** jamielennox is now known as jamielennox|away00:00
*** Alexey_Abashkin_ has quit IRC00:13
*** jamielennox|away is now known as jamielennox00:14
*** Alexey_Abashkin_ has joined #heat00:14
*** Alexey_Abashkin_ has quit IRC00:18
*** adrian_otto has quit IRC00:21
*** vishwanathj has quit IRC00:22
*** Alexey_Abashkin_ has joined #heat00:32
*** zhurong has joined #heat01:13
*** bana_k has quit IRC01:15
*** sdake_ has joined #heat01:19
*** sdake has quit IRC01:21
openstackgerrithuangtianhua proposed openstack/heat: Return condition functions based on the filter param  https://review.openstack.org/36986001:22
*** yanyanhu has joined #heat01:24
*** zhihui has joined #heat01:30
*** steveg_afk has quit IRC01:54
ramishrastevebaker: hey around?01:54
stevebakerramishra: hai01:54
ramishrastevebaker: I think heatclient stable/newton is missing some more stuff like the env merging01:55
ramishrastevebaker: I think tripleo is using that for their newton release01:55
stevebakeroh, thats important01:55
stevebakerramishra: can you backport what else is needed? and also review https://review.openstack.org/#/c/373531/ ?01:58
stevebakerramishra: then I'll review yours and do a release when they land01:58
ramishrayes, I'll though I can't +2 any of those:)01:58
stevebakerit'll need to be 1.5.001:58
stevebakermeh, I'll approve them. 2 cores +1 is enough01:59
*** vijendar has joined #heat01:59
ramishrabtw, can you review  https://review.openstack.org/#/c/367502/? I think murano team were making noise about it, good if we can get that fix in.01:59
stevebakersure02:00
stevebakerramishra: approved. should it be backported too?02:01
ramishrayeah, I'll do that02:02
ramishraI think it should do without backporting in 1.5.002:02
stevebakerok02:02
openstackgerritzhufl proposed openstack/heat: Remove unnecessary setUp and tearDown  https://review.openstack.org/37358602:03
*** vijendar has quit IRC02:04
*** dimtruck is now known as zz_dimtruck02:05
ramishrastevebaker: done the env merge one. I don't see anything else that needs backporting. should we tag 1.5.0 too?02:11
stevebakerramishra: the stable/newton release will be 1.5.0, because it has a new feature02:11
stevebakerramishra: we could do a 1.6.0 on master when the releases repo opens up ocata02:12
ramishrastevebaker: ok, thanks.02:13
*** zz_dimtruck is now known as dimtruck02:15
openstackgerritMerged openstack/python-heatclient: Don't use endpoint_override with session  https://review.openstack.org/36750202:17
openstackgerritzhufl proposed openstack/heat: Add missing %s in print message  https://review.openstack.org/37359502:29
stevebakerramishra: ugh, I can't approve this either https://review.openstack.org/#/c/373531/02:38
*** caowei has joined #heat02:39
ramishrastevebaker: hehe.. have to wait for skraynev or therve :)02:41
*** jdandrea_ has joined #heat02:44
openstackgerritzhufl proposed openstack/heat: Add missing %s in print message  https://review.openstack.org/37359502:44
*** jdandrea has quit IRC02:47
*** cmyster has quit IRC02:49
*** elynn has joined #heat02:50
*** yuanying has quit IRC02:52
*** priteau has joined #heat02:53
*** elynn has quit IRC02:55
*** elynn has joined #heat02:56
*** priteau has quit IRC02:58
*** david-lyle has quit IRC03:04
*** cmyster has joined #heat03:04
*** sdake_ has quit IRC03:42
openstackgerritzhufl proposed openstack/heat: Remove unnecessary __init__  https://review.openstack.org/37362503:45
*** yuanying has joined #heat03:49
*** dimtruck is now known as zz_dimtruck04:16
*** zz_dimtruck is now known as dimtruck04:16
*** dimtruck is now known as zz_dimtruck04:25
*** KanagarajM has joined #heat04:34
openstackgerritMerged openstack/heat: Add missing %s in print message  https://review.openstack.org/37359504:35
*** elynn has quit IRC04:36
*** jaosorior has joined #heat04:50
*** elynn has joined #heat04:53
*** code-R has joined #heat04:56
*** elynn has quit IRC04:58
*** elynn has joined #heat04:58
*** jaosorior has quit IRC05:10
*** jaosorior has joined #heat05:11
*** bana_k has joined #heat05:15
*** caowei has quit IRC05:32
*** mohankumar has joined #heat05:41
*** bana_k has quit IRC05:45
*** vijendar has joined #heat05:48
*** code-R has quit IRC05:51
*** caowei has joined #heat05:52
*** vijendar has quit IRC05:54
*** akuznetsov has joined #heat05:58
*** code-R has joined #heat05:58
*** rcernin has joined #heat06:07
*** akuznetsov has quit IRC06:08
*** code-R_ has joined #heat06:09
*** akuznetsov has joined #heat06:12
*** code-R has quit IRC06:12
*** shardy has joined #heat06:14
openstackgerritshizhihui proposed openstack/heat: Trivial:Standardize the output format of 'help' in some files  https://review.openstack.org/36356706:20
*** jprovazn has joined #heat06:35
*** akuznetsov has quit IRC06:36
*** sshnaidm|afk is now known as sshnaidm06:44
*** code-R_ has quit IRC06:45
openstackgerritSharat Sharma proposed openstack/python-heatclient: Replace MagicMock with Mock class  https://review.openstack.org/37379206:51
skraynevstevebaker: ramishra: +W06:51
openstackgerrithuangtianhua proposed openstack/heat: Add Ocata version '8.0.0' in doc  https://review.openstack.org/37379706:58
*** ccamacho has joined #heat07:08
*** akuznetsov has joined #heat07:08
*** yprokule has joined #heat07:10
*** Alexey_Abashkin_ has quit IRC07:20
openstackgerritShu Yingya proposed openstack/heat: Remove remaining 'heat/openstack/common' from heat  https://review.openstack.org/37380707:22
*** AlexeyAbashkin has joined #heat07:23
openstackgerritOpenStack Proposal Bot proposed openstack/heat: Updated from global requirements  https://review.openstack.org/37369807:26
openstackgerritRico Lin proposed openstack/heat: Add a new HOT template version for Ocata  https://review.openstack.org/37381407:28
*** ricolin has joined #heat07:28
*** jpich has joined #heat07:36
*** code-R has joined #heat07:37
*** mohankumar has quit IRC07:38
openstackgerritShu Yingya proposed openstack/heat: Remove remaining 'heat/openstack/common' from heat  https://review.openstack.org/37380707:40
*** akuznetsov has quit IRC07:41
*** code-R has quit IRC07:46
stevebakerskraynev: thanks07:48
*** vijendar has joined #heat07:50
skraynevnp :)07:51
*** blues-man has joined #heat07:52
*** akuznetsov has joined #heat07:54
ramishrait seems gate broken again, this time with latest openstacksdk/osc07:54
*** vijendar has quit IRC07:55
stevebakerramishra: can you point to a change?07:55
stevebakerthatis failing?07:56
ramishraI've proposed a work around for the time being https://review.openstack.org/#/c/37385007:56
ramishrastevebaker: https://bugs.launchpad.net/python-openstackclient/+bug/1625954, I've proposed a fix too:)07:56
openstackLaunchpad bug 1625954 in python-openstackclient "router_interface_add/remove broken with openstacksdk" [Undecided,In progress] - Assigned to Rabi Mishra (rabi)07:56
* cmyster looks at the time07:56
cmysterstevebaker: hi07:56
stevebakercmyster: hey07:57
*** zzzeek has quit IRC08:00
*** zzzeek has joined #heat08:00
openstackgerrithuangtianhua proposed openstack/heat: Add networking-sfc port-pair-group resource plug-in  https://review.openstack.org/27784908:02
*** code-R has joined #heat08:02
ramishrado we've a meeting now, therve?08:02
therveWe do!08:02
therveshardy, skraynev, elynn meeting if you can08:05
therveKanagarajM maybe08:05
* shardy joins08:05
*** akuznetsov has quit IRC08:06
*** ricolin_afk has joined #heat08:08
*** yassine has joined #heat08:09
*** yassine is now known as Guest9877408:10
*** ricolin has quit IRC08:11
*** code-R has quit IRC08:11
*** akuznetsov has joined #heat08:12
*** mohankumar has joined #heat08:22
*** jaosorior is now known as jaosorior_brb08:28
*** ricolin has joined #heat08:40
*** ricolin_afk has quit IRC08:40
*** shardy has quit IRC08:42
*** ricolin_afk has joined #heat08:43
*** derekh has joined #heat08:44
*** ricolin_afk has quit IRC08:44
*** ricolin_afk has joined #heat08:45
*** ricolin has quit IRC08:47
*** sorantis has joined #heat08:48
*** shardy has joined #heat08:52
*** akuznetsov has quit IRC08:57
*** sorantis has quit IRC08:59
openstackgerritRico Lin proposed openstack/heat: Add a new HOT template version for Ocata  https://review.openstack.org/37381408:59
*** sorantis has joined #heat09:04
*** nijaba has quit IRC09:05
*** akuznetsov has joined #heat09:06
*** nijaba has joined #heat09:06
*** nijaba has quit IRC09:06
*** nijaba has joined #heat09:06
*** akuznetsov has quit IRC09:08
*** yanyanhu has quit IRC09:10
*** yanyanhu has joined #heat09:11
*** yanyanhu has quit IRC09:11
*** bks has joined #heat09:16
*** akuznetsov has joined #heat09:20
*** mohankumar has quit IRC09:29
*** code-R has joined #heat09:30
*** code-R_ has joined #heat09:31
*** ricolin_afk has quit IRC09:32
*** ricolin has joined #heat09:32
*** code-R has quit IRC09:34
*** vijendar has joined #heat09:51
*** akuznetsov has quit IRC09:51
*** vijendar has quit IRC09:56
openstackgerritRico Lin proposed openstack/heat: Make more clear diff in hot template functions  https://review.openstack.org/37398109:57
*** akuznetsov has joined #heat09:57
*** ricolin has quit IRC09:59
shardytherve: Hey, I raised two bugs describing the nested validation issues we're hitting:10:00
shardyhttps://bugs.launchpad.net/heat/+bug/162602210:00
openstackLaunchpad bug 1626022 in heat "Nested validation issues with non-defaulted nested parameters" [Undecided,New]10:00
shardyhttps://bugs.launchpad.net/heat/+bug/162602510:00
openstackLaunchpad bug 1626025 in heat "Nested validation breaks with ResourceGroup and non-defaulted count parameter" [Undecided,New]10:00
shardyI think the second one we can actually fix in tht by just adding a default value of zero10:00
shardybut it's still kind of unexpected for validation to fail when the template isn't broken10:01
*** zhurong has quit IRC10:02
therveshardy, Thanks, will look at it10:07
*** akuznetsov has quit IRC10:20
*** sshnaidm has quit IRC10:37
*** sshnaidm has joined #heat10:42
*** steveg_afk has joined #heat10:43
*** prazumovsky has joined #heat10:44
*** dfflanders has quit IRC10:47
*** code-R_ has quit IRC10:56
*** code-R has joined #heat10:56
therveshardy, I don't totally get https://bugs.launchpad.net/heat/+bug/162602211:01
openstackLaunchpad bug 1626022 in heat "Nested validation issues with non-defaulted nested parameters" [Undecided,New]11:01
therveshardy, The issue is that if you remove, say, from_parent_attr: {get_attr: [a_resource, value]}, validation will succeed?11:01
shardytherve: No, the problem is that the schema exposed via NestedParameters doesn't convey that a value will be passed from the parent stack11:02
shardye.g there's only a Value field for from_parent_param_default11:03
shardytherve: this makes it impossible to decide which parameters have values passed down from the parent, and which require user input (e.g via parameter_defaults in the environment)11:03
therveAh11:04
shardysome kind of easily identifiable placeholder from the parent in Value would solve that11:04
therveshardy, So you'd like something in the Value field11:04
shardyyeah11:04
therveOk, understood11:04
shardyit's mostly so the Ux can build an interface from this schema11:04
shardywe don't want to ask the user for things we know are passed from the parent11:04
*** shardy is now known as shardy_lunch11:05
therveI'd like to blame parameter_defaults for that :)11:05
*** mewald_ has joined #heat11:05
mewald_What's currently the best way to distribute instances across availability zones with heat? I've browser the internet a bit but haven't found any satisfying solution11:07
*** steveg_afk has quit IRC11:09
*** mohankumar has joined #heat11:17
*** elynn has quit IRC11:17
openstackgerritMerged openstack/heat: Use neutron cli command instead of osc  https://review.openstack.org/37385011:21
openstackgerritMerged openstack/heat: Add Ocata version '8.0.0' in doc  https://review.openstack.org/37379711:22
*** jaosorior_brb is now known as jaosorior11:28
*** shardy_lunch is now known as shardy11:29
*** EricGonczer_ has joined #heat11:34
*** code-R_ has joined #heat11:42
mewald_I feel like the best solution to spreading instances across availbility zones might be to wrap Heat with Ansible and template the HOTs with jinja2. Any experience with that?11:43
*** code-R has quit IRC11:45
*** sshnaidm is now known as sshnaidm|afk11:47
openstackgerritOpenStack Proposal Bot proposed openstack/heat: Updated from global requirements  https://review.openstack.org/37369811:49
openstackgerritPeter Razumovsky proposed openstack/heat: Add parameters and outputs template migrate  https://review.openstack.org/37337311:50
openstackgerritPeter Razumovsky proposed openstack/heat: Add check for unsupported resource to tmpl migrate  https://review.openstack.org/37337711:50
openstackgerritPeter Razumovsky proposed openstack/heat: Add function migration to template migrate  https://review.openstack.org/37337511:50
openstackgerritPeter Razumovsky proposed openstack/heat: Add migrate version to actual HOT template version  https://review.openstack.org/37337611:50
openstackgerritPeter Razumovsky proposed openstack/heat: Add resource keys translation in template migrate  https://review.openstack.org/37337411:51
openstackgerritPeter Razumovsky proposed openstack/heat: Initial implementation template migrate mechanism  https://review.openstack.org/37015311:51
*** vijendar has joined #heat11:52
*** zhurong has joined #heat11:54
*** bks has quit IRC11:57
*** vijendar has quit IRC11:58
therveramishra, There is something really weird going on in tests12:06
therveWe have some pretty strange intermittent failures12:06
ramishratherve: yeah, I could not understand some of the failures. Did you notice anything specific?12:08
therveramishra, Not really. It seems to be around tests with signals though.12:09
therveI'm looking at http://logs.openstack.org/50/373850/1/check/gate-heat-dsvm-functional-orig-mysql-lbaasv2/603074a/ currently12:09
therveSuddenly there is a DELETE in the middle of the test for no apparent reason12:09
ramishrayeah, I agree. It's happening mostly for non-convergence path.12:13
*** dprince has joined #heat12:13
ramishraI think I raised a bug sometime back, where there was nothing after a delete rpc message https://bugs.launchpad.net/heat/+bug/161713012:14
openstackLaunchpad bug 1617130 in heat "test_resource_group functional test times out for delete " [Medium,New] - Assigned to Rabi Mishra (rabi)12:14
therveramishra, http://logs.openstack.org/14/373614/1/gate/gate-heat-dsvm-functional-convg-mysql-lbaasv2/d5c33eb/console.html this convergence one is weird too12:14
*** steveg_afk has joined #heat12:15
*** annegentle has joined #heat12:16
ramishraAnother one http://logs.openstack.org/15/373615/1/gate/gate-heat-dsvm-functional-convg-mysql-lbaasv2/8e33b52/console.html#_2016-09-21_11_58_10_27633512:22
therveYeah I opened one yesterday, though it looks a bit different12:23
ramishrayeah, I saw that.12:25
*** crose has joined #heat12:25
therveI'll open bugs to track them at least12:28
*** dyasny has quit IRC12:28
*** EricGonczer_ has quit IRC12:29
*** sshnaidm|afk is now known as sshnaidm12:32
ramishraI've added SDG test error backtrace with  'heatclient.exc.HTTPNotFound: ERROR: None' to an earlier bug for the same test.12:32
ramishrathough it would have been another bug.12:33
ramishrahttps://bugs.launchpad.net/heat/+bug/1625921/comments/112:34
openstackLaunchpad bug 1625921 in heat "SoftwareDeploymentGroupTest fails at times with TimeoutException" [Medium,New]12:34
*** askb has quit IRC12:35
*** zhihui has quit IRC12:38
*** dyasny has joined #heat12:42
therveOK12:45
therveramishra, https://bugs.launchpad.net/heat/+bug/1626075 is a simple race condition12:46
openstackLaunchpad bug 1626075 in heat "test_hook_pre_update_in_place intermittently fails" [Medium,New]12:46
therveWe store the event, but the hook is not yet in place, so the test doesn't find it12:46
thervePossibly just inverting the operations would be good enough12:46
*** annegentle has quit IRC12:47
openstackgerritThomas Herve proposed openstack/heat: Set hooks before storing events  https://review.openstack.org/37411512:49
openstackgerritPeter Razumovsky proposed openstack/heat: [WIP] Add API call for template migrate  https://review.openstack.org/37411912:53
*** crose has quit IRC12:56
*** david-lyle has joined #heat12:56
*** jcoufal has joined #heat12:57
*** tung_doan has joined #heat13:03
*** zz_dimtruck is now known as dimtruck13:03
openstackgerritMohankumar proposed openstack/heat: Add networking-sfc port-pair-group resource plug-in  https://review.openstack.org/27784913:04
*** vishwanathj has joined #heat13:06
*** dimtruck is now known as zz_dimtruck13:08
*** zz_dimtruck is now known as dimtruck13:08
*** jaosorior has quit IRC13:10
*** jaosorior has joined #heat13:11
*** vijendar has joined #heat13:12
mohankumarprazumovsky : ping13:14
prazumovskyhello13:15
mohankumarprazumovsky : Appened patch with your review comments , Could you pls review while you find time :)  (have little doubt in my understanding)13:15
mohankumarprazumovsky : https://review.openstack.org/#/c/277849/13:15
prazumovskydone13:16
mohankumarprazumovsky : great !13:17
*** dimtruck is now known as zz_dimtruck13:18
*** vijendar has quit IRC13:18
openstackgerritPeter Razumovsky proposed openstack/python-heatclient: [WIP] add cli command for template migrate  https://review.openstack.org/37414013:21
mewald_I feel like the best solution to spreading instances across availbility zones might be to wrap Heat with Ansible and template the HOTs with jinja2. Any experience with that?13:29
mewald_I feel like the best solution to spreading instances across availbility zones might be to wrap Heat with Ansible and template the HOTs with jinja2. Any experience with that?13:29
*** mewald_ has quit IRC13:29
*** blomquisg_gone is now known as blomquisg13:29
*** code-R_ has quit IRC13:34
*** code-R has joined #heat13:36
*** spzala has joined #heat13:47
*** jrist has quit IRC13:52
*** blues-man has quit IRC13:53
*** akshai has joined #heat13:55
*** KanagarajM has quit IRC13:56
*** lifeless has quit IRC13:57
*** sdake has joined #heat13:59
*** lifeless has joined #heat13:59
*** sdake has quit IRC13:59
*** sdake has joined #heat14:00
*** jasond has joined #heat14:00
*** tanvirt has joined #heat14:03
*** tanvirt has quit IRC14:03
*** tanvirt has joined #heat14:03
*** jasond has quit IRC14:05
*** jasond has joined #heat14:06
*** jrist has joined #heat14:07
*** cmartin has joined #heat14:10
*** spedione|AWAY is now known as spedione14:10
*** annegentle has joined #heat14:11
*** jasond has quit IRC14:12
*** pradk_ has joined #heat14:13
*** dyasny has quit IRC14:23
*** pradk has quit IRC14:26
*** pradk_ is now known as pradk14:26
*** dyasny has joined #heat14:26
*** mohankumar has quit IRC14:27
*** mfisch has quit IRC14:28
*** rajpatel has joined #heat14:28
*** rajpatel1 has joined #heat14:30
*** mfisch has joined #heat14:32
*** mfisch has quit IRC14:32
*** mfisch has joined #heat14:32
*** rajpatel has quit IRC14:33
*** zz_dimtruck is now known as dimtruck14:33
*** akshai has quit IRC14:40
*** akshai has joined #heat14:43
*** crose has joined #heat14:44
*** caowei has quit IRC14:48
*** akshai has quit IRC14:49
*** netmanager has joined #heat14:49
*** zhurong has quit IRC14:51
*** akshai has joined #heat14:52
*** jprovazn has quit IRC14:53
*** jprovazn has joined #heat14:55
*** netmanager has quit IRC14:55
*** openstack has joined #heat15:03
*** sorantis has joined #heat15:05
*** vijendar has quit IRC15:18
*** sorantis has quit IRC15:19
*** Drago has joined #heat15:20
*** vijendar has joined #heat15:22
*** vijendar has quit IRC15:27
*** Guest98774 has quit IRC15:28
*** Guest98774 has joined #heat15:32
*** prazumovsky has quit IRC15:38
*** jaosorior has quit IRC15:39
*** bks has joined #heat15:45
*** ccamacho is now known as ccamacho|afk15:47
*** ramishra has quit IRC15:48
*** d0ugal has quit IRC15:48
*** bana_k has joined #heat15:49
*** d0ugal has joined #heat15:49
*** ramishra has joined #heat15:50
*** kumarb has joined #heat15:51
EmilienMhi, Puppet OpenStack reported a bug in heat: https://bugs.launchpad.net/heat/+bug/162617315:52
openstackLaunchpad bug 1626173 in heat "stack failed to reach DELETE_COMPLETE status (timeout)" [Undecided,New]15:52
EmilienMwe have this timeout very often recently15:52
EmilienMI was wondering if you had the same in your gate15:52
EmilienMshardy, therve ^ fyi15:52
*** bks has quit IRC15:52
*** spotz_zzz is now known as spotz15:53
tanvirtzaneb: Continuing our discussion of the bug at: https://bugs.launchpad.net/heat/+bug/1570576. I'm inside the update_task() method inside heat/engine/stack.py. In order for me to steal the lock from the dead engine, I need the engine_id. How I can retreive that from here?15:55
openstackLaunchpad bug 1570576 in heat "After updating a stack stuck IN_PROGRESS, resources will be permanently stuck IN_PROGRESS" [High,Triaged] - Assigned to Tanvir Talukder (tanvirt16)15:55
zanebEmilienM: by happy coincidence I believe I have a fix for that one15:57
zanebtanvirt: looking...15:57
EmilienMzaneb: nice :) please let me know in the bug report or on IRC! thanks15:58
zanebtanvirt: create a StackLock object. try to acquire() it and it'll return the engine ID if something else is holding the lock15:58
tanvirtzaneb: but doesn't the creation of a StackLock object require an engine_id as a parameter?15:59
zanebtanvirt: ah, you mean our local engine_id16:00
*** vijendar has joined #heat16:00
zanebtanvirt: I can't actually think of a way to get back the the EngineService object. The easiest way might be to send an RPC message to do what you need to do and let some other engine handle it in its EngineService object16:02
*** rcernin has quit IRC16:03
zanebEmilienM: do you _always_ see that error message? Just trying to establish if there could be multiple causes16:03
tanvirtzaneb: hmmm... another option may be to somehow utilize the get_engine_id method inside the StackLock object. I'd have to create a variation of it with parameters like: get_engine_id(context stack_id), and make it a class method. Any thoughts?16:05
iurygregoryzaneb, some times this error show up and we do recheck (las time: http://logs.openstack.org/67/373967/2/check/gate-puppet-openstack-integration-3-scenario003-tempest-centos-7/a7960f6/console.html#_2016-09-21_15_01_51_161249, http://logs.openstack.org/67/373967/2/check/gate-puppet-openstack-integration-4-scenario003-tempest-centos-7-nv/dcf98e6/console.html#_2016-09-21_13_54_22_914887)16:06
iurygregoryis just in scenario003 jobs in puppet-openstack =)16:07
*** EricGonczer_ has joined #heat16:07
openstackgerritOleksii Chuprykov proposed openstack/heat: Return error with path in function validate  https://review.openstack.org/37345116:08
tanvirtzaneb: or I can make the engine_id an optional paramter for the StackLock object, with the default value being whats returned by the get_engine_id method16:09
EmilienMzaneb: no, but very often recently16:09
*** Drago1 has joined #heat16:09
zanebtanvirt: StackLock.get_engine_id() returns the ID of the engine that has the lock. StackLock.__init__() takes the ID of the current engine, so that when you acquire the lock it knows to say who has it.16:09
*** Drago has quit IRC16:10
*** Drago1 is now known as Drago16:10
EmilienMzaneb: what iurygregory means is that our "scenario003" is the only one which test Heat.16:10
zanebEmilienM: ok, we'll squash that one first and see what's left behind16:10
tanvirtzaneb: ah, ok16:10
zanebiurygregory, EmilienM: to be clear, I was asking if http://logs.openstack.org/67/373967/2/check/gate-puppet-openstack-integration-3-scenario003-tempest-centos-7/a7960f6/logs/heat/heat-engine.txt.gz#_2016-09-21_14_41_38_553 always shows up when the delete times out16:12
*** jprovazn has quit IRC16:12
EmilienMzaneb: yes16:13
EmilienMzaneb: I saw it everytime I see the timeout16:13
zanebEmilienM: oh, excellent :)16:13
zanebshould be only one bug to squash then16:13
*** crose has quit IRC16:15
*** KanagarajM has joined #heat16:20
*** spzala has quit IRC16:30
*** spzala has joined #heat16:31
*** tung_doan has left #heat16:33
*** adrian_otto1 has joined #heat16:35
*** crose has joined #heat16:35
*** spzala has quit IRC16:37
*** vijendar has quit IRC16:41
zzzeekzaneb: I lied   https://bitbucket.org/zzzeek/sqlalchemy/issues/3258/a-gevent-killed-greenlet-causes-pymysql16:43
zanebzzzeek: there was one! :)16:46
zzzeekzaneb: there were two :)16:46
*** spzala has joined #heat16:46
zzzeekzaneb: but, i think the whole gevent thing only started catching on like 2013/2014 and at this point it's likely on the wane16:46
zzzeekat least the "global monkeypatch" variety16:46
zanebwith any luck16:47
*** rpothier has joined #heat16:47
zzzeekzaneb: so yeah I "knew" this on the openstack-dev list from old muscle memory :)16:48
* zzzeek is old16:48
*** jpich has quit IRC16:48
zzzeekfacts are jumbled and dusty16:48
zzzeekzaneb: from these two issues at least the invalidation mechanics were made to withstand this condition16:49
zanebzzzeek: so IIUC, the issue is all the lines like https://github.com/zzzeek/sqlalchemy/blob/master/lib/sqlalchemy/engine/base.py#L702 are only catching Exception, and if you added another handler to catch BaseException + call _handle_dbapi_exception() + always always always re-raise the original exception, then everything would work?16:49
*** adrian_otto1 has quit IRC16:49
zzzeekzaneb: almost.  I would like to just allow create_engine("mysql+pymysql://...", handle_exceptions=(greenlet.GreenletExit, ))16:50
zzzeeksomething like that16:50
*** rwsu has quit IRC16:51
zzzeekzaneb: I still dont think getting involved with KeyboardInterrupt / SystemExit by default is something I could relax with16:51
zanebzzzeek: ok. I still don't agree that GreenletExit is a special case here16:51
zzzeekzaneb: it's more like, I cannot conceive of what will go wrong if keybaordinterrupt/systemexit suddenly makes tons of new code run for people, only that very disastrous effects have occurred from much, much more innocuous-seeming things16:52
zanebzzzeek: as long as you never ever ever wrap or suppress exit exceptions, then there's no problem handling them, and in fact cleaning up when they happen is the right thing to do16:52
zanebzzzeek: I totally get that you are in a very conservative line of work :)16:53
zzzeekzaneb: messing w/ the boundaries of the interpreter is creepy16:53
zzzeekglad to make it an optoin16:53
*** randallburt has joined #heat16:53
zzzeekzaneb: an old example that recently I revisited is, you connect from the connection pool.  then you just drop the connection, don't close it or return it.   OK let's use __del__ and automatically return it. easy!16:54
zanebI don't feel like it's messing with the boundaries of the interpreter. if you weren't supposed to be able to clean up after getting a signal then Python wouldn't raise an exception, it would just exit16:54
zzzeek8 zillion bug reports and changes later, I ultimately had to use my own Queue.Queue implementation etc. b.c. there were deadlocks in the wild16:55
zzzeekchange __del__ to be weakref callback16:55
zzzeeketc16:55
zaneb__del__ is nasty16:55
*** randallburt1 has joined #heat16:55
zzzeekwell it's weakref callback now.   however, it caused deadlocks.  because gc.collect() runs whenever it feels like, such as inside the mutex in Queue.put()16:55
zzzeekQueuePool is still technically wrong because calling Queue.Queue.put() reentrant-ly does not guarantee its internal state16:56
zzzeekzaneb: also, the variety of "cleanup" we're talking about here is very strange.  I know of no other way to put a pymysql connection in this broken state except for unsynchronized access16:57
*** randallburt has quit IRC16:58
*** akuznetsov has quit IRC16:58
zzzeekonly greenlet's magic breakage of sockets can really make this happen.   the whole "monkeypatch" use case IMO should not be used16:58
zanebzzzeek: say that there's a KeyboardInterrupt. when would the user ever want to not clean up afterwards, and be like "no, I'd rather you return that borked connection to the pool so that some future unrelated transaction can die with a cryptic error message"? that's why I find it strange to make handling it correctly optional16:59
*** derekh has quit IRC16:59
zzzeekzaneb: I'd need to confirm that you can actually break the connection like that.17:00
*** rajpatel1 has quit IRC17:00
zzzeekzaneb: a program might be reading rows on a single connection, press ctrl-C to stop, breaks out, goes back to "READY>", keeps using connection17:01
*** yprokule has quit IRC17:01
zanebzzzeek: I think that'd be worth trying. it should be possible with a variant on your POC from yesterday. just run the function with the loop and hit Ctrl-C on the keyboard a lot17:01
zzzeekzaneb: then also again.  sohuldn't pymysql catch KeyboardInterrupt ?17:02
zzzeekzaneb: i think we might want to look and see that pymysql's "corruption" itself is unusual.  usually we all use native-C DBAPIs which also don't realy have these kinds of problems17:02
zanebzzzeek: arguably, yes. I looked at the code and they assume the only exception that can happen is IOError17:02
zzzeekzaneb: the reason we are using pymysql is....for eventlet compat ! :)17:03
zanebit looks really hard to unwind from the inside though. I think in most cases you'll end up having to kill the connection and start again. that's probably easier to deal with at the sqlalchemy level, even though it leaves other PyMySQL users in the lurch17:04
*** jcoufal has quit IRC17:06
zanebagree that C APIs are atomic w.r.t. async exceptions like this, so they don't have this issue17:06
zanebalthough there are probably (short) sections of sqlalchemy where an exception could cause problems17:07
*** dprince has quit IRC17:07
*** kumarb has quit IRC17:09
*** code-R has quit IRC17:09
zanebzzzeek: that patch from your first comment in that bug report has some interesting context: https://github.com/zzzeek/sqlalchemy/blob/rel_0_9_3/lib/sqlalchemy/pool.py#L541-L54517:13
zzzeekhaha woops17:13
zanebzzzeek: clearly you were intending to catch SystemExit and KeyboardInterrupt there, but you're actually missing them17:14
zzzeekzaneb: i wonder if that's still th4ere b.c. it is obv. wrong17:14
zzzeekzaneb: or perhaps that's from my py2.4 days17:14
zanebnope, not any more afaict17:14
zzzeekbefore BaseException17:14
zanebI did use Python 2.4, but my memory doesn't go that far back ;)17:15
zzzeekzaneb: we're catching BaseException there now17:16
*** vijendar has joined #heat17:16
zanebah, yep. it's 100% correct now17:16
zzzeekzaneb: that's a case where the connection is already being garbage collected17:18
zzzeekzaneb: that's not a normal "userland" use case17:18
zanebgarbage collection is the twilight zone, but userland may not be as normal as we think :D17:19
*** akuznetsov has joined #heat17:20
zzzeekzaneb: but in userland, *the user is there*17:21
zzzeekzaneb: i think the cases of "corrupts the connection" vs. "the SQLAlchemy oepration was interupted" should perhaps be separate17:21
zzzeekzaneb: if i can demontraste to pymysql a ctrl-C corrupting a connection i'll send it to him and see what he thinks17:22
*** Guest98774 has quit IRC17:22
zanebzzzeek: +117:23
*** akuznetsov has quit IRC17:25
*** KanagarajM has quit IRC17:27
*** pradk has quit IRC17:34
*** pradk has joined #heat17:35
*** jasond has joined #heat17:35
*** dsneddon has quit IRC17:40
zzzeekzaneb: there might be an entirely different way to look at this issue17:41
zzzeekzaneb: when heat or whoever has this problem, e.g. not just that the connection was interrupted, but then you get errors afterwards, do you always have the "ResourceClosedError" part of this?17:43
zzzeeke.g. and not just the "commands out of sync" part17:43
*** Dave has quit IRC17:45
*** Dave____ has joined #heat17:51
*** bana_k has quit IRC17:51
*** Zer0Byte__ has joined #heat17:52
*** sshnaidm is now known as sshnaidm|afk17:52
zzzeekeh nevermind17:53
openstackgerritStephen Watson proposed openstack/heat: Brings Magnum resource up to date  https://review.openstack.org/37432118:11
*** jpich has joined #heat18:14
*** bana_k has joined #heat18:17
*** shardy has quit IRC18:18
*** akuznetsov has joined #heat18:21
zanebzzzeek: usually. not sure about 'always'18:25
*** akuznetsov has quit IRC18:26
zzzeekzaneb: nevermind, you win all points https://bitbucket.org/zzzeek/sqlalchemy/issues/3803/dbapi-connections-go-invalid-on18:26
*** rcernin has joined #heat18:30
*** vijendar has quit IRC18:30
*** rajpatel has joined #heat18:31
*** jprovazn has joined #heat18:33
zanebzzzeek: thanks for all your help with this. that patch looks great to me18:34
zanebzzzeek: you might want to add "and not is_base_exception" on the end of the definition of should_wrap, since you never want to wrap an exit exception18:35
*** jpich has quit IRC18:35
zanebzzzeek: maybe something similar on the "if newraise:" line18:36
*** vijendar has joined #heat18:36
*** sdake has quit IRC18:39
*** Dave____ has quit IRC18:48
*** Dave has joined #heat18:50
*** sdake has joined #heat18:56
*** catintheroof has joined #heat19:12
*** sabeen has joined #heat19:13
zanebEmilienM: https://review.openstack.org/#/c/374340/19:14
openstackgerritYosef Hoffman proposed openstack/heat-specs: New Resource Type OS::Nova::Quota  https://review.openstack.org/37354619:16
EmilienMzaneb: :o19:20
*** akshai has quit IRC19:20
EmilienMzaneb: thanks, unfortunatly, I can't test it in puppet CI easily19:21
*** akuznetsov has joined #heat19:23
*** akshai has joined #heat19:25
zanebEmilienM: hmm, actually that might not fix it. that looks to be a different constraint from the one my patch fixes19:27
*** akuznetsov has quit IRC19:28
zanebcwolferh: any clues what might be causing the IntegrityError in http://logs.openstack.org/67/373967/2/check/gate-puppet-openstack-integration-3-scenario003-tempest-centos-7/a7960f6/logs/heat/heat-engine.txt.gz#_2016-09-21_14_41_38_553 ?19:29
cwolferhzaneb, not off the top of my head but i'll dig19:31
*** mfisch has quit IRC19:37
cwolferhit seems odd that a user_cred would disappear. user_cred id is 1 here which is probably an admin. hmm. wrongfully purged?19:38
*** jprovazn has quit IRC19:40
*** jasond has quit IRC19:40
*** bana_k has quit IRC19:43
*** dyasny has quit IRC19:45
*** cmartin has quit IRC19:47
zanebcwolferh: note that this is in the puppet gate, and this is literally the only test they run on heat, so that might be why it gets id=119:47
zanebbut yeah, an incorrect purge somewhere sounds like it could be the culprit19:47
*** tanvirt has quit IRC19:50
swatsonHey everyone, quick question re: heat.egg_info19:51
swatsonI was just updating the Magnum resource for Heat and my tests were failing until I updated heat/heat.egg-info/entrypoints.txt for the Magnum entry19:52
*** rajpatel has quit IRC19:53
swatsonBut I noticed that folder seems to be ignored by git--is there some other file I need to edit that will update the egg-info instead?19:53
*** rajpatel has joined #heat19:56
openstackgerritZane Bitter proposed openstack/heat: Don't delete RawTemplate if it referenced by a stack  https://review.openstack.org/37434019:57
*** rajpatel has quit IRC19:57
*** jasond has joined #heat19:58
*** bana_k has joined #heat19:59
*** dyasny has joined #heat19:59
*** steveg_afk has quit IRC20:00
*** catintheroof has quit IRC20:08
openstackgerritStephen Watson proposed openstack/heat: Brings Magnum resource up to date  https://review.openstack.org/37432120:08
*** blomquisg is now known as blomquisg_gone20:10
*** steveg_afk has joined #heat20:15
*** jcoufal has joined #heat20:15
cwolferhzaneb, so, i think the problem has to do with trying to delete the stack twice, once in the test itself: https://git.openstack.org/cgit/openstack/tempest/tree/tempest/api/orchestration/stacks/test_stacks.py#n58 and again in the cleanup https://git.openstack.org/cgit/openstack/tempest/tree/tempest/api/orchestration/base.py#n84 . the first delete succeeds fine. i'm not sure why the 2nd gets as far as it does before blowing up20:22
zanebcwolferh: hmmm. so with convergence there's no longer a lock at the stack level20:23
zanebwhich is fine as far as starting the update to an empty template goes...20:24
zanebbut when it comes to actually doing the delete of the Stack row in the DB, possibly not as good20:24
cwolferhthe test waiting for DELETE_COMPLETE means the credential will be deleted by the time the 2nd delete in the cleanup is called. but i feel like that 2nd delete should see, "oh there is nothing to do"20:29
cwolferh(but i can imagine an issue if two calls to delete are made without a wait for DELETE_COMPLETE)20:30
*** steveg_afk has quit IRC20:33
zanebyeah, I agree20:36
zanebso if it's DELETE_COMPLETE we should reject any attempts to change it20:36
cwolferhseems fair20:36
zanebI guess that isn't happening?20:36
cwolferhnot sure yet :-)  looking at the issue in my devstack20:37
cwolferhbut clearly in the puppet-ci case the 2nd delete is trying to do too much20:37
zanebactually, now that I think about it, if there's a race it should be OK. only one of them can be the current traversal, and only that one gets to write DELETE_COMPLETE to the DB and then soft-delete the stack20:38
zanebso it's more likely to be that we are starting something after it's already DELETE_COMPLETE that we really shouldn't20:39
cwolferhin my devstack, i'm only seeing one attempt to delete the stack for this test (in h-eng.log). hmm.20:41
*** bana_k has quit IRC20:42
*** crose has quit IRC20:44
*** steveg_afk has joined #heat20:46
openstackgerritStephen Watson proposed openstack/heat: Brings Magnum resource up to date  https://review.openstack.org/37432120:47
cwolferhzaneb, from an api (or heat.engine.service) perspective, i think this is just as simple as checking for DELETE_COMPLETE and treating that as a "not found". in the puppet-ci case, i think the 2nd call just came too fast before the stack was actually deleted after the DELETE_COMPLETE status was set20:54
*** askb has joined #heat20:55
zanebcwolferh: so we're checking for whether the stack has been soft-deleted, but not if it is in DELETE_COMPLETE, and there's a race where a new request can get in between those two?20:55
*** jasond has quit IRC20:57
*** jasond has joined #heat20:57
cwolferhzaneb, yes, between this try: https://git.openstack.org/cgit/openstack/heat/tree/heat/engine/stack.py#n1764 and line 177720:57
zanebcwolferh: that's the legacy path and IIUC this is convergence, but I bet there is something equivalent in convergence20:59
cwolferhah, oops20:59
*** akshai has quit IRC20:59
zanebhttps://git.openstack.org/cgit/openstack/heat/tree/heat/engine/stack.py#n200321:00
*** bana_k has joined #heat21:00
zanebsame thing21:00
zanebI hate that we've added two lots of methods to the same class to do legacy and convergence21:01
zanebI wish we had two separate helper classes :/21:01
cwolferhthat would have been nice21:04
*** akuznetsov has joined #heat21:06
*** rwsu has joined #heat21:06
cwolferhok, so same possible fix is to treat DELETE_COMPLETE by raising exception.EntityNotFound in https://git.openstack.org/cgit/openstack/heat/tree/heat/engine/service.py#n136021:08
*** akuznetsov has quit IRC21:11
zanebcwolferh: +121:11
zanebso pre-convergence we'd have killed the thread doing the first delete, in order to get its lock. by which point it would have either completed (and we'd get NotFound when we tried to reload the stack) or it wouldn't have done the soft-delete yet, and we could try again21:13
zanebtricky21:13
*** vijendar has quit IRC21:17
*** dsneddon has joined #heat21:19
*** vijendar has joined #heat21:24
*** rcernin has quit IRC21:24
*** rajpatel has joined #heat21:25
*** Zer0Byte__ has quit IRC21:28
*** rpothier has quit IRC21:35
*** EricGonczer_ has quit IRC21:48
*** jasond has quit IRC21:52
*** randallburt1 has quit IRC21:55
*** cwolferh has quit IRC21:59
*** ccamacho|afk has quit IRC21:59
*** cwolferh has joined #heat22:01
*** jasond has joined #heat22:05
*** spotz is now known as spotz_zzz22:07
*** akuznetsov has joined #heat22:07
zanebwow, navigating the code base from 2012 is hard22:08
*** spedione is now known as spedione|AWAY22:08
*** annegentle has quit IRC22:11
*** sabeen has quit IRC22:12
*** akuznetsov has quit IRC22:12
*** spzala has quit IRC22:20
*** jcoufal has quit IRC22:20
*** spzala has joined #heat22:20
*** adrian_otto has joined #heat22:20
*** vijendar has quit IRC22:20
*** jasond has quit IRC22:21
openstackgerritYosef Hoffman proposed openstack/heat: Add OS::Nova::Quota resource  https://review.openstack.org/37354822:22
*** steveg_afk has quit IRC22:24
*** spzala has quit IRC22:25
*** adrian_otto has quit IRC22:25
zanebstevebaker: what were you thinking? ;) https://bugs.launchpad.net/heat/+bug/1499669/comments/1722:26
openstackLaunchpad bug 1499669 in heat mitaka "Heat stucks in DELETE_IN_PROGRESS for some input data" [High,Triaged]22:26
*** adrian_otto has joined #heat22:28
stevebakerzaneb: I was thinking I really want a job at Red Hat, whats this OpenStack thing?22:31
zaneblol22:31
stevebakerzaneb: also I probably cargo-culted it from somewhere22:32
zanebyeah, probably nova22:32
stevebakerOpenStack is swift plus things cargo-culted from nova22:34
*** rajpatel has quit IRC22:46
*** adrian_otto has quit IRC22:47
*** rajpatel has joined #heat22:47
*** annegentle has joined #heat22:58
*** jamielennox is now known as jamielennox|away23:04
*** akuznetsov has joined #heat23:10
*** hoonetorg has quit IRC23:10
*** akuznetsov has quit IRC23:14
*** annegentle has quit IRC23:15
openstackgerritCrag Wolfe proposed openstack/heat: Do not attempt deletion of a DELETE_COMPLETE stack in service api  https://review.openstack.org/37444023:20
openstackgerritZane Bitter proposed openstack/heat: Don't use cast() to do StackResource delete  https://review.openstack.org/37444223:27
openstackgerritZane Bitter proposed openstack/heat: Increase the timeout for the stop_stack message  https://review.openstack.org/37444323:27
openstackgerritZane Bitter proposed openstack/heat: RPC Client: don't cast() delete_stack by default  https://review.openstack.org/37444423:27
*** Zer0Byte__ has joined #heat23:28
*** jamielennox|away is now known as jamielennox23:43
*** rwsu has quit IRC23:45
*** zhurong has joined #heat23:46
*** rajpatel has quit IRC23:56
*** zhurong has quit IRC23:57
*** dimtruck is now known as zz_dimtruck23:58
*** EricGonczer_ has joined #heat23:58

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!