| *** rajpatel1 has quit IRC | 00:00 | |
| zaneb | stevebaker: you sure? it looks the same except you stop it before sending the RPC message with template_id=None instead of when you go to write the stack to the DB | 00:03 |
|---|---|---|
| *** zhurong has quit IRC | 00:04 | |
| stevebaker | zaneb: I thought that was a different case, where the template ID is set but the template has already been deleted | 00:04 |
| *** jdandrea has quit IRC | 00:05 | |
| zaneb | stevebaker: ah, I guess that could theoretically happen | 00:08 |
| stevebaker | zaneb: also, this may be ready | 00:09 |
| stevebaker | https://review.openstack.org/#/c/369827/ | 00:09 |
| *** Zer0Byte__ has joined #heat | 00:12 | |
| *** jamielennox is now known as jamielennox|away | 00:14 | |
| *** jdandrea has joined #heat | 00:14 | |
| *** sshnaidm has quit IRC | 00:19 | |
| *** Zer0Byte__ has quit IRC | 00:23 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/heat: Updated from global requirements https://review.openstack.org/371097 | 00:25 |
| *** vijendar has joined #heat | 00:28 | |
| *** vijendar has quit IRC | 00:32 | |
| *** adrian_otto has quit IRC | 00:33 | |
| openstackgerrit | Merged openstack/heat: Raise NotImplementedError instead of NotImplemented https://review.openstack.org/367735 | 00:39 |
| *** spzala has quit IRC | 01:02 | |
| *** caowei has joined #heat | 01:02 | |
| *** zhurong has joined #heat | 01:12 | |
| *** bana_k has quit IRC | 01:28 | |
| *** zz_dimtruck is now known as dimtruck | 01:30 | |
| *** sshnaidm has joined #heat | 01:33 | |
| *** elynn has joined #heat | 01:38 | |
| *** dimtruck is now known as zz_dimtruck | 01:40 | |
| *** elynn has quit IRC | 01:43 | |
| *** elynn has joined #heat | 01:43 | |
| *** steveg_afk has quit IRC | 01:46 | |
| *** yanyanhu has joined #heat | 01:47 | |
| *** EricGonczer_ has joined #heat | 01:58 | |
| *** sdake has quit IRC | 02:06 | |
| *** sdake has joined #heat | 02:06 | |
| *** sdake has quit IRC | 02:18 | |
| *** jamielennox|away is now known as jamielennox | 02:20 | |
| *** EricGonczer_ has quit IRC | 02:22 | |
| *** zz_dimtruck is now known as dimtruck | 02:24 | |
| *** spzala has joined #heat | 03:02 | |
| *** david-lyle has quit IRC | 03:03 | |
| openstackgerrit | Merged openstack/heat: Cancel traversal of nested stack https://review.openstack.org/370123 | 03:03 |
| *** spzala has quit IRC | 03:07 | |
| openstackgerrit | Merged openstack/heat: Updated from global requirements https://review.openstack.org/371097 | 03:11 |
| *** tochi has quit IRC | 03:24 | |
| stevebaker | Cloudformation catches up :) https://aws.amazon.com/blogs/aws/aws-cloudformation-update-yaml-cross-stack-references-simplified-substitution/ | 03:42 |
| *** KanagarajM has joined #heat | 03:43 | |
| *** dimtruck is now known as zz_dimtruck | 03:47 | |
| *** KanagarajM has quit IRC | 03:48 | |
| *** tochi has joined #heat | 03:51 | |
| openstackgerrit | Crag Wolfe proposed openstack/heat: Do not share sessions between threads https://review.openstack.org/372804 | 03:59 |
| openstackgerrit | Crag Wolfe proposed openstack/heat: Trivial refactor of session var in reset_stack_status https://review.openstack.org/372874 | 03:59 |
| *** jcoufal has quit IRC | 04:12 | |
| *** elynn has quit IRC | 04:22 | |
| *** zhurong has quit IRC | 04:43 | |
| *** KanagarajM has joined #heat | 04:44 | |
| *** sdake has joined #heat | 04:46 | |
| *** elynn has joined #heat | 04:52 | |
| *** jaosorior has joined #heat | 04:55 | |
| *** elynn has quit IRC | 04:56 | |
| *** elynn has joined #heat | 04:57 | |
| *** jaosorior has quit IRC | 05:09 | |
| *** jaosorior has joined #heat | 05:10 | |
| *** akshai has joined #heat | 05:11 | |
| *** mohankumar has joined #heat | 05:18 | |
| *** akshai has quit IRC | 05:19 | |
| *** bnemec has joined #heat | 05:27 | |
| *** zhurong has joined #heat | 05:35 | |
| *** zhurong has quit IRC | 05:47 | |
| *** zhurong has joined #heat | 05:47 | |
| *** elynn_ has joined #heat | 05:50 | |
| *** elynn has quit IRC | 05:52 | |
| *** elynn_ has quit IRC | 05:54 | |
| *** elynn_ has joined #heat | 05:55 | |
| *** ccamacho has quit IRC | 06:07 | |
| *** shardy has joined #heat | 06:12 | |
| openstackgerrit | Merged openstack/heat: Legacy delete attempt thread cancel before stop https://review.openstack.org/369827 | 06:13 |
| *** yprokule has joined #heat | 06:20 | |
| *** ccamacho has joined #heat | 06:24 | |
| *** jprovazn has joined #heat | 06:29 | |
| *** elynn_ has quit IRC | 06:33 | |
| *** bana_k has joined #heat | 06:35 | |
| *** elynn has joined #heat | 06:36 | |
| *** elynn has quit IRC | 06:41 | |
| *** elynn has joined #heat | 06:41 | |
| *** prazumovsky has joined #heat | 07:00 | |
| *** bana_k has quit IRC | 07:04 | |
| openstackgerrit | avnish proposed openstack/heat-templates: modify the home-page info with the developer documentation https://review.openstack.org/372962 | 07:07 |
| openstackgerrit | Peter Razumovsky proposed openstack/heat-specs: Fix milestone and new ideas to properties group https://review.openstack.org/333304 | 07:08 |
| *** chlong has quit IRC | 07:14 | |
| openstackgerrit | Merged openstack/heat: Add resource OS::Cinder::QoSAssociation https://review.openstack.org/249713 | 07:15 |
| *** shardy has quit IRC | 07:19 | |
| *** rcernin has joined #heat | 07:19 | |
| *** mohankumar has quit IRC | 07:29 | |
| *** jpich has joined #heat | 07:33 | |
| *** tochi has quit IRC | 07:38 | |
| *** sorantis has joined #heat | 07:48 | |
| *** zhurong has quit IRC | 07:55 | |
| *** zzzeek has quit IRC | 08:00 | |
| *** zzzeek has joined #heat | 08:00 | |
| *** yassine has joined #heat | 08:08 | |
| *** yassine is now known as Guest19246 | 08:09 | |
| *** zhurong has joined #heat | 08:13 | |
| *** derekh has joined #heat | 08:34 | |
| openstackgerrit | huangtianhua proposed openstack/heat: Return condition functions based on the filter param https://review.openstack.org/369860 | 08:37 |
| openstackgerrit | Dmitriy Uvarenkov proposed openstack/heat: Improve validation of res limit for nested templates https://review.openstack.org/362176 | 08:46 |
| *** blues-man has joined #heat | 08:51 | |
| openstackgerrit | huangtianhua proposed openstack/heat: Return condition functions based on the filter param https://review.openstack.org/369860 | 08:52 |
| *** mohankumar has joined #heat | 08:54 | |
| *** yohoffman has quit IRC | 09:03 | |
| *** syjulian has quit IRC | 09:03 | |
| *** xenogear has quit IRC | 09:04 | |
| *** xenogear has joined #heat | 09:04 | |
| *** xenogear has quit IRC | 09:12 | |
| *** code-R has joined #heat | 09:15 | |
| *** yohoffman has joined #heat | 09:18 | |
| *** mohankumar has quit IRC | 09:20 | |
| *** code-R_ has joined #heat | 09:23 | |
| *** code-R has quit IRC | 09:26 | |
| *** xenogear has joined #heat | 09:29 | |
| *** mohankumar has joined #heat | 09:36 | |
| *** syjulian has joined #heat | 09:37 | |
| *** shardy has joined #heat | 09:52 | |
| *** yanyanhu has quit IRC | 09:53 | |
| *** zhurong has quit IRC | 10:03 | |
| *** dave-johnston has joined #heat | 10:30 | |
| *** rwsu has quit IRC | 10:35 | |
| *** mohankumar has quit IRC | 10:36 | |
| *** elynn has quit IRC | 10:37 | |
| *** jaosorior is now known as jaosorior_lunch | 10:37 | |
| *** jaosorior_lunch is now known as jaosorior | 11:03 | |
| *** code-R_ has quit IRC | 11:06 | |
| *** code-R has joined #heat | 11:11 | |
| *** steveg_afk has joined #heat | 11:26 | |
| *** code-R has quit IRC | 11:31 | |
| *** tlashchova has joined #heat | 11:33 | |
| *** spzala has joined #heat | 11:34 | |
| *** caowei has quit IRC | 11:36 | |
| *** code-R has joined #heat | 11:40 | |
| *** dprince has joined #heat | 11:41 | |
| openstackgerrit | Anant Patil proposed openstack/heat: Re-trigger on update-replace https://review.openstack.org/371572 | 11:52 |
| *** zhurong has joined #heat | 11:53 | |
| *** EricGonczer_ has joined #heat | 11:59 | |
| *** yprokule_ has joined #heat | 11:59 | |
| openstackgerrit | Anant Patil proposed openstack/heat: Fix cancel update for nova server with defined port https://review.openstack.org/306490 | 12:00 |
| *** yprokule has quit IRC | 12:01 | |
| *** yprokule_ is now known as yprokule | 12:01 | |
| *** rwsu has joined #heat | 12:03 | |
| *** ccamacho is now known as ccamacho|lunch | 12:04 | |
| *** EricGonc_ has joined #heat | 12:09 | |
| openstackgerrit | Oleksii Chuprykov proposed openstack/heat: Validate outputs of the stack after its create/update https://review.openstack.org/371704 | 12:10 |
| *** EricGonczer_ has quit IRC | 12:11 | |
| *** jcoufal has joined #heat | 12:18 | |
| *** shardy is now known as shardy_lunch | 12:20 | |
| *** yassine has joined #heat | 12:28 | |
| *** spzala has quit IRC | 12:28 | |
| *** yassine is now known as Guest54237 | 12:29 | |
| *** Guest19246 has quit IRC | 12:29 | |
| *** caowei has joined #heat | 12:29 | |
| *** askb has quit IRC | 12:30 | |
| *** ayoung has quit IRC | 12:30 | |
| *** annegentle has joined #heat | 12:35 | |
| *** jprovazn has quit IRC | 12:36 | |
| *** mohankumar has joined #heat | 12:39 | |
| *** rwsu has quit IRC | 12:44 | |
| *** steveg_afk has quit IRC | 12:45 | |
| *** shardy_lunch is now known as shardy | 12:50 | |
| *** steveg_afk has joined #heat | 12:55 | |
| *** ccamacho|lunch is now known as ccamacho | 12:56 | |
| *** david-lyle has joined #heat | 12:56 | |
| *** vishwanathj has joined #heat | 12:57 | |
| zaneb | therve: I think we are in the home stretch :) | 12:59 |
| therve | zaneb, You say that... | 12:59 |
| *** elynn has joined #heat | 12:59 | |
| therve | That's usually when a patch in neutron-lbaas decides to break us | 12:59 |
| * zaneb resumes "speak no evil" pose | 13:01 | |
| therve | On the nice side of things, our own merge rate seems to be pretty nice this last couple of days | 13:01 |
| therve | So we must have merged something among those fixes that did improve the situation | 13:02 |
| *** rwsu has joined #heat | 13:03 | |
| zaneb | therve: I'm pretty sure https://bugs.launchpad.net/heat/+bug/1624538 was responsible for many (most?) of our lock-ups | 13:03 |
| openstack | Launchpad bug 1624538 in heat "Cancelling completed stack retains lock forever" [High,Fix released] - Assigned to Zane Bitter (zaneb) | 13:03 |
| zaneb | although the patch didn't merge until yesterday, so that can't be the whole story | 13:04 |
| therve | Yeah, and that's convergence only, no? | 13:04 |
| zaneb | no, that's non-convergence only | 13:06 |
| therve | Ah | 13:06 |
| zaneb | ananta has made a lot of fixes on the convergence side that have probably helped a lot too | 13:06 |
| *** vijendar has joined #heat | 13:07 | |
| *** markd__ has joined #heat | 13:07 | |
| *** markd__ has quit IRC | 13:08 | |
| *** KanagarajM has quit IRC | 13:08 | |
| *** jaosorior has quit IRC | 13:09 | |
| *** jaosorior has joined #heat | 13:10 | |
| *** spzala has joined #heat | 13:11 | |
| *** spzala has quit IRC | 13:11 | |
| *** spzala has joined #heat | 13:11 | |
| *** sdake has quit IRC | 13:17 | |
| *** cmartin has joined #heat | 13:23 | |
| *** vijendar has quit IRC | 13:24 | |
| *** tanvirt has joined #heat | 13:30 | |
| *** tanvirt has quit IRC | 13:30 | |
| *** tanvirt has joined #heat | 13:30 | |
| *** dyasny has joined #heat | 13:31 | |
| *** openstackstatus has joined #heat | 13:35 | |
| *** ChanServ sets mode: +v openstackstatus | 13:35 | |
| openstackgerrit | Peter Razumovsky proposed openstack/heat-specs: Heat template migrate mechanism https://review.openstack.org/333259 | 13:39 |
| *** rajpatel has joined #heat | 13:40 | |
| -openstackstatus- NOTICE: OpenStack Infra now has a Twitter bot, follow it at https://twitter.com/openstackinfra | 13:42 | |
| *** annegentle has quit IRC | 13:46 | |
| *** sdake has joined #heat | 13:49 | |
| *** jprovazn has joined #heat | 13:50 | |
| *** jasond has joined #heat | 14:01 | |
| *** rajpatel1 has joined #heat | 14:04 | |
| *** rajpatel has quit IRC | 14:06 | |
| *** mohankumar has quit IRC | 14:08 | |
| *** dyasny has quit IRC | 14:09 | |
| *** sdake_ has joined #heat | 14:09 | |
| *** sdake has quit IRC | 14:11 | |
| *** rpothier has joined #heat | 14:12 | |
| openstackgerrit | Oleksii Chuprykov proposed openstack/heat: Validate outputs of the stack after its create/update https://review.openstack.org/371704 | 14:13 |
| *** ayoung has joined #heat | 14:15 | |
| *** annegentle has joined #heat | 14:20 | |
| *** blomquisg_gone is now known as blomquisg | 14:20 | |
| *** dyasny has joined #heat | 14:22 | |
| *** spedione|AWAY is now known as spedione | 14:24 | |
| *** adrian_otto has joined #heat | 14:26 | |
| *** spotz_zzz is now known as spotz | 14:27 | |
| *** zz_dimtruck is now known as dimtruck | 14:28 | |
| *** rajpatel has joined #heat | 14:30 | |
| openstackgerrit | Peter Razumovsky proposed openstack/heat: Initial implementation template migrate mechanism https://review.openstack.org/370153 | 14:31 |
| openstackgerrit | Peter Razumovsky proposed openstack/heat: Add parameters and outputs template migrate https://review.openstack.org/373373 | 14:31 |
| openstackgerrit | Peter Razumovsky proposed openstack/heat: Add resource keys translation in template migrate https://review.openstack.org/373374 | 14:31 |
| openstackgerrit | Peter Razumovsky proposed openstack/heat: Add function migration to template migrate https://review.openstack.org/373375 | 14:31 |
| openstackgerrit | Peter Razumovsky proposed openstack/heat: Add migrate version to actual HOT template version https://review.openstack.org/373376 | 14:31 |
| openstackgerrit | Peter Razumovsky proposed openstack/heat: Add check for unsupported resource to tmpl migrate https://review.openstack.org/373377 | 14:31 |
| openstackgerrit | Peter Razumovsky proposed openstack/heat: Initial implementation template migrate mechanism https://review.openstack.org/370153 | 14:32 |
| *** rajpatel2 has joined #heat | 14:32 | |
| openstackgerrit | Peter Razumovsky proposed openstack/heat: Add parameters and outputs template migrate https://review.openstack.org/373373 | 14:32 |
| openstackgerrit | Peter Razumovsky proposed openstack/heat: Add resource keys translation in template migrate https://review.openstack.org/373374 | 14:32 |
| openstackgerrit | Peter Razumovsky proposed openstack/heat: Add function migration to template migrate https://review.openstack.org/373375 | 14:32 |
| openstackgerrit | Peter Razumovsky proposed openstack/heat: Add migrate version to actual HOT template version https://review.openstack.org/373376 | 14:32 |
| openstackgerrit | Peter Razumovsky proposed openstack/heat: Add check for unsupported resource to tmpl migrate https://review.openstack.org/373377 | 14:32 |
| *** rajpatel1 has quit IRC | 14:33 | |
| *** spedione is now known as spedione|AWAY | 14:34 | |
| *** EricGonc_ has quit IRC | 14:34 | |
| *** rajpatel has quit IRC | 14:35 | |
| *** EricGonczer_ has joined #heat | 14:37 | |
| *** rajpatel2 has quit IRC | 14:38 | |
| *** dyasny has quit IRC | 14:39 | |
| *** rajpatel has joined #heat | 14:41 | |
| *** dyasny has joined #heat | 14:41 | |
| *** KanagarajM has joined #heat | 14:44 | |
| openstackgerrit | Peter Razumovsky proposed openstack/heat: Add check for unsupported resource to tmpl migrate https://review.openstack.org/373377 | 14:47 |
| openstackgerrit | Peter Razumovsky proposed openstack/heat: Add function migration to template migrate https://review.openstack.org/373375 | 14:47 |
| openstackgerrit | Peter Razumovsky proposed openstack/heat: Add migrate version to actual HOT template version https://review.openstack.org/373376 | 14:47 |
| *** adrian_otto has quit IRC | 14:48 | |
| *** prazumovsky has quit IRC | 14:51 | |
| *** huangtianhua has joined #heat | 14:52 | |
| *** blomquisg is now known as blomquisg_gone | 14:54 | |
| *** randallburt has joined #heat | 14:54 | |
| *** tiantian has quit IRC | 14:55 | |
| *** randallburt1 has joined #heat | 14:56 | |
| *** adrian_otto has joined #heat | 14:56 | |
| *** adrian_otto has quit IRC | 14:57 | |
| *** spedione|AWAY is now known as spedione | 14:58 | |
| *** Drago has joined #heat | 14:58 | |
| *** randallburt has quit IRC | 14:59 | |
| *** KanagarajM has quit IRC | 14:59 | |
| *** KanagarajM has joined #heat | 15:00 | |
| *** rajpatel has quit IRC | 15:02 | |
| *** yprokule_ has joined #heat | 15:02 | |
| *** elynn has quit IRC | 15:02 | |
| *** rajpatel has joined #heat | 15:04 | |
| *** yprokule has quit IRC | 15:05 | |
| *** yprokule_ is now known as yprokule | 15:05 | |
| *** rajpatel1 has joined #heat | 15:05 | |
| *** rajpatel has quit IRC | 15:08 | |
| *** caowei has quit IRC | 15:10 | |
| *** zhurong has quit IRC | 15:11 | |
| *** cmartin has quit IRC | 15:12 | |
| *** sshnaidm is now known as sshnaidm|afk | 15:18 | |
| *** sorantis has quit IRC | 15:26 | |
| *** ramishra has quit IRC | 15:31 | |
| *** akshai has joined #heat | 15:33 | |
| *** jaosorior has quit IRC | 15:40 | |
| openstackgerrit | Zane Bitter proposed openstack/heat: Use pydoc for formatting docstrings https://review.openstack.org/373433 | 15:42 |
| *** dprince has quit IRC | 15:42 | |
| *** b00tcat has joined #heat | 15:42 | |
| b00tcat | can a comma_delimited_list be set to '' as a default value to indicate an empty array? | 15:42 |
| *** rajpatel has joined #heat | 15:45 | |
| *** rajpatel1 has quit IRC | 15:45 | |
| *** derekh has quit IRC | 15:49 | |
| *** adrian_otto has joined #heat | 15:53 | |
| *** dprince has joined #heat | 15:58 | |
| *** ramishra has joined #heat | 15:59 | |
| *** yprokule has quit IRC | 16:03 | |
| *** jcoufal_ has joined #heat | 16:04 | |
| *** code-R has quit IRC | 16:05 | |
| *** code-R has joined #heat | 16:06 | |
| *** rajpatel has quit IRC | 16:06 | |
| *** code-R has quit IRC | 16:07 | |
| *** vijendar has joined #heat | 16:08 | |
| *** jcoufal has quit IRC | 16:08 | |
| *** vijendar has quit IRC | 16:08 | |
| openstackgerrit | Oleksii Chuprykov proposed openstack/heat: Return error with path in function validate https://review.openstack.org/373451 | 16:30 |
| *** bana_k has joined #heat | 16:31 | |
| openstackgerrit | Oleksii Chuprykov proposed openstack/heat: Return error with path in function validate https://review.openstack.org/373451 | 16:31 |
| *** blomquisg_gone is now known as blomquisg | 16:34 | |
| *** code-R has joined #heat | 16:35 | |
| *** code-R_ has joined #heat | 16:36 | |
| *** code-R has quit IRC | 16:40 | |
| *** blues-man has quit IRC | 16:43 | |
| *** EricGonc_ has joined #heat | 16:43 | |
| *** EricGonczer_ has quit IRC | 16:45 | |
| *** jpich has quit IRC | 16:48 | |
| *** rcernin has quit IRC | 16:55 | |
| *** adrian_otto has quit IRC | 16:56 | |
| *** dsneddon has joined #heat | 17:00 | |
| *** KanagarajM has quit IRC | 17:02 | |
| *** Zer0Byte__ has joined #heat | 17:06 | |
| *** rajpatel has joined #heat | 17:06 | |
| *** sai is now known as sai|mtg | 17:08 | |
| *** vijendar has joined #heat | 17:10 | |
| *** vijendar has quit IRC | 17:15 | |
| *** bana_k has quit IRC | 17:27 | |
| *** Zer0Byte__ has quit IRC | 17:33 | |
| *** code-R_ has quit IRC | 17:36 | |
| *** bana_k has joined #heat | 17:40 | |
| *** crose has joined #heat | 17:41 | |
| *** akshai has quit IRC | 17:44 | |
| *** akshai has joined #heat | 17:44 | |
| *** cwolferh has quit IRC | 17:44 | |
| *** akshai has quit IRC | 17:45 | |
| *** sai|mtg is now known as sai | 17:57 | |
| *** akshai has joined #heat | 17:57 | |
| *** adrian_otto has joined #heat | 18:04 | |
| *** shardy has quit IRC | 18:05 | |
| *** adrian_otto has quit IRC | 18:08 | |
| *** crose has quit IRC | 18:09 | |
| *** swatson has joined #heat | 18:11 | |
| *** code-R has joined #heat | 18:13 | |
| *** code-R has quit IRC | 18:14 | |
| *** tanvirt has quit IRC | 18:15 | |
| *** akuznetsov has joined #heat | 18:31 | |
| *** rajpatel has quit IRC | 18:32 | |
| *** rajpatel has joined #heat | 18:33 | |
| *** adrian_otto has joined #heat | 18:35 | |
| *** cwolferh has joined #heat | 18:35 | |
| *** sdake_ is now known as sdake | 18:50 | |
| *** akshai has quit IRC | 18:59 | |
| *** akshai has joined #heat | 19:02 | |
| *** rajpatel has quit IRC | 19:10 | |
| *** dave-johnston has quit IRC | 19:11 | |
| *** vijendar has joined #heat | 19:11 | |
| *** vijendar has quit IRC | 19:12 | |
| *** jasond has quit IRC | 19:26 | |
| *** jasond has joined #heat | 19:26 | |
| *** akshai has quit IRC | 19:28 | |
| *** mfisch has quit IRC | 19:32 | |
| *** mfisch has joined #heat | 19:33 | |
| *** mfisch has quit IRC | 19:33 | |
| *** mfisch has joined #heat | 19:33 | |
| *** akshai has joined #heat | 19:34 | |
| *** dbecker has quit IRC | 19:37 | |
| *** dbecker has joined #heat | 19:44 | |
| *** rajpatel has joined #heat | 19:59 | |
| *** akuznetsov has quit IRC | 20:00 | |
| *** adrian_otto has quit IRC | 20:02 | |
| *** jcoufal_ has quit IRC | 20:09 | |
| *** huangtianhua has quit IRC | 20:10 | |
| *** huangtianhua has joined #heat | 20:10 | |
| stevebaker | b00tcat: how about []? | 20:12 |
| stevebaker | therve, zaneb: I've backport this https://review.openstack.org/#/c/373518/ | 20:13 |
| zaneb | yikes | 20:13 |
| *** Zer0Byte__ has joined #heat | 20:15 | |
| *** jprovazn has quit IRC | 20:16 | |
| openstackgerrit | Crag Wolfe proposed openstack/heat: Do not share sessions between threads https://review.openstack.org/372804 | 20:17 |
| *** askb has joined #heat | 20:22 | |
| zaneb | cwolferh: do you have an example of where we *are* sharing sessions between threads? | 20:23 |
| cwolferh | zaneb, i'll try to come up with one. i thought we passed around the requestcontext somewhere | 20:23 |
| openstackgerrit | Merged openstack/heat-templates: modify the home-page info with the developer documentation https://review.openstack.org/372962 | 20:23 |
| zaneb | I know we pass the context when we spawn a new thread, but afaik we never continue to do DB operations with it in the original thread | 20:24 |
| zaneb | cwolferh, sbaker: from my testing plus zzzeek's email http://lists.openstack.org/pipermail/openstack-dev/2016-September/103674.html I'm actually increasingly convinced that it's a sqlalchemy bug | 20:25 |
| zzzeek | zaneb: have you managed to identify an oslo.db version where that issue goes away | 20:26 |
| zaneb | an exception (in our case GreenletExit, but most likely could be almost anything) occurs while talking to the DB. sqlalchemy doesn't invalidate the connection but returns it to the pool. the next session to pick up that connection fails | 20:26 |
| zaneb | zzzeek: no, but our issue has been around since at least Liberty if that helps: https://bugs.launchpad.net/heat/+bug/1499669 | 20:26 |
| openstack | Launchpad bug 1499669 in heat mitaka "Heat stucks in DELETE_IN_PROGRESS for some input data" [High,Triaged] | 20:26 |
| zzzeek | zaneb: b.c. if its been "always", yes, the stack trace I saw w/ mysql connection failing is a mis use of the fixtures.Timeout fixture | 20:26 |
| zzzeek | zaneb: if fixtures.Timeout() is involved, the issue is understood IMO | 20:27 |
| zzzeek | zaneb: if its the traces where Postgresql is just getting tripped up on which database is in use, that's likely soehting different | 20:27 |
| zaneb | zzzeek: it's not, we're just killing a greenthread (which causes GreenletExit to be raised in the thread by eventlet). but it's probably related | 20:27 |
| zzzeek | zaneb: if you kill a greenthread where tests are doing things then you are leaving connections in invalid states. pymysql connections will be cut off at hte socket level, Posgresql connections will have locks remaining opened | 20:28 |
| zzzeek | zaneb: so really both kinds of traces can be explained if a greenlet is being killed | 20:28 |
| zaneb | zzzeek: in our case this isn't a test, we kill threads in real life :) | 20:29 |
| zaneb | zzzeek: so what's the solution? | 20:29 |
| zzzeek | zaneb: ok..the thread above has to do with the test suite | 20:29 |
| zzzeek | zaneb: in real life, you aren't having to DROP DATABASE | 20:30 |
| zzzeek | zaneb: what is the "real life" bug we're talking about ? | 20:30 |
| zaneb | https://bugs.launchpad.net/heat/+bug/1499669 | 20:30 |
| openstack | Launchpad bug 1499669 in heat mitaka "Heat stucks in DELETE_IN_PROGRESS for some input data" [High,Triaged] | 20:30 |
| zzzeek | zaneb: in https://bugs.launchpad.net/heat/+bug/1499669/comments/3 it looks like an async timer is doing somethign that results in work being done on a DB connection is that right | 20:33 |
| openstack | Launchpad bug 1499669 in heat mitaka "Heat stucks in DELETE_IN_PROGRESS for some input data" [High,Triaged] | 20:33 |
| zaneb | zzzeek: here, I just reproduced it, this might be easier to read: http://paste.openstack.org/show/582316/ | 20:33 |
| cwolferh | zaneb, that looks a bit different than the 'Command out of sync' error | 20:35 |
| zaneb | zzzeek: basically another thread is calling kill() on a thread which may be engaged in DB operations. This raises GreenletExit. various handlers may respond to the GreenletExit by doing more DB operations. sometimes those operations fail. sometimes it appears that a future operation in a different session (but presumably using the same connection) fails | 20:35 |
| zzzeek | zaneb: ok in that stack trace, there's a connection that sometihng happened on, the "thing that happened" wasn't intercepted, and the connection was retunred to the connection pool. | 20:35 |
| zzzeek | zaneb: yes, so, if you're in a greenlet and GreenletExit happens, any DB connection you have you shudl call invalidate() on | 20:36 |
| zaneb | cwolferh: that comes next, forgot to paste it | 20:36 |
| zaneb | zzzeek: respectfully, *you* should invalidate it ;) | 20:37 |
| zzzeek | zaneb: a DB connection gets sent back to the pool when its outer wrapper is garbage collected | 20:37 |
| zzzeek | zaneb: OK how would i implement that ? | 20:37 |
| zaneb | zzzeek: catch the exception, invalidate the connection, then re-raise? | 20:37 |
| zzzeek | zaneb: basically this means, "my program digs into the DBAPI driver's mechanics, locates the socket communication, and cuts it off mid-stream. no exception is raised. then the connection is dropped" | 20:38 |
| zzzeek | zaneb: I can't catch a GreenletException | 20:38 |
| zaneb | zzzeek: why not? | 20:38 |
| zzzeek | zaneb: if you want to show me where I'd "catch" that ? | 20:38 |
| zzzeek | zaneb: oslo.db perhaps can | 20:38 |
| zaneb | zzzeek: so I do "with session.begin():" | 20:40 |
| zaneb | zzzeek: sometime during that block, GreenletExit gets raised | 20:40 |
| zzzeek | zaneb: I would look into an oslo.db event handler to implement this | 20:41 |
| *** EricGonc_ has quit IRC | 20:41 | |
| *** dfflanders has joined #heat | 20:41 | |
| zzzeek | zaneb: but the awkwardness is that I'm not sure oslo.db depends on eventlet | 20:41 |
| zaneb | zzzeek: the session's rollback() will fail because the connection is borked. where exactly should I, as the caller, have handled that? | 20:41 |
| zzzeek | zaneb: there's a lot of places you can do that | 20:42 |
| zaneb | zzzeek: GreenletExit is just an exit exception, same as e.g. KeyboardInterrupt. I don't think it needs any special handling | 20:42 |
| zzzeek | zaneb: but oslo.db event might be the simplest since we already have that set up | 20:42 |
| zzzeek | zaneb: im not sure by "I don't think is needs special handling" what you mean on that. There's all kinds of resources a program might establish which require clean teardown | 20:43 |
| zzzeek | zaneb: e.g im not sure if we're still stuck on philosophy or actually solving the problem | 20:44 |
| zzzeek | zaneb: from SQLAlchemy's pov, it relies upon the DBAPI to raise exceptions that are meaningful, and if the exception indicates, "this connection is borked", we kill it. that's in. | 20:44 |
| zzzeek | zaneb: the problem here is that no DBAPI intercepts GreenletException either. | 20:45 |
| zaneb | so to answer your question... "zzzeek> zaneb: OK how would i implement that ?": http://paste.openstack.org/show/582319/ | 20:45 |
| zzzeek | zaneb: SQLAlchemy relies upon the driver to describe errors in the connection's state. it would only be able to respond to the DBAPI level error. | 20:46 |
| zzzeek | zaneb: however in this case, pymysql itself is misbehaving in that regard | 20:46 |
| zzzeek | zaneb: because it too, does not know that it has been killed in the middle of a socket communication | 20:46 |
| zaneb | zzzeek: so it's my contention that all exit exceptions (i.e. anything that inherits from BaseException but not Exception, including GreenletExit, KeyboardInterrupt, SystemExit) should be added to the list of exceptions that cause the connection to be invalidated | 20:47 |
| zaneb | I'm willing to stipulate that this might by pymysql's fault ;) | 20:47 |
| zzzeek | zaneb: OK so, keeping in mind that I can solve this problem for you, likely in oslo.db, how imporatnt is it that you "win" the philosophical argument here vs. fix the problem? | 20:48 |
| zaneb | 0% | 20:48 |
| zzzeek | zaneb: I can illustrate the problem at pymysql's level. we can post it, and I'm pretty sure he will reject it as pymysql's problem also | 20:48 |
| zzzeek | at the pymysql level, it means that you run a SELECT, it succeeds, and then cursor.description is blank | 20:48 |
| zzzeek | which is against spec | 20:49 |
| zzzeek | this is because pymysql was busy sending multiple messages to the mysql server and didn't get to finish them | 20:49 |
| zaneb | how can it be succeeding if there's an exception raised? | 20:49 |
| *** dprince has quit IRC | 20:49 | |
| zaneb | that's the part that I'm confused about | 20:49 |
| zzzeek | zaneb: the exceptin is raised, socket communication is interrupted. pymysql connection stays "open" | 20:50 |
| zzzeek | zaneb: however, messaging to the server was interrupted | 20:50 |
| zzzeek | zaneb: next operation tries to do something w/ the connection. pymysql again communicates w/ the server, but server is like, "hey wait you didn't finish the thing we were doing before" | 20:50 |
| zzzeek | zaneb: you get either "commands out of sync" error, which is easier to catch, or this thing where cursor.description is None | 20:50 |
| zzzeek | sqlalhcemy *should* be invalidating for "commands out of sync", let me check | 20:51 |
| zaneb | zzzeek: so you're saying the exception is not raised in the call to write to the socket? | 20:51 |
| zzzeek | a GreenletException for your timeout or whatever is riased but pymysql does not respond to that nor does SQLalchemy | 20:52 |
| *** spedione is now known as spedione|AWAY | 20:52 | |
| zzzeek | zaneb: now that i see what causes this I can acutllay make tests and confirm all this | 20:52 |
| zzzeek | zaneb: up until now i havent had that opportunitiy | 20:52 |
| zzzeek | zaneb: yeah for 2014 commands out of sync that connection is gone | 20:53 |
| zzzeek | zaneb: let me try to create a proof of concept at he pymysql level, as far as oslo.db patch i'd have to get to that later in the week | 20:56 |
| *** priteau has joined #heat | 20:58 | |
| zaneb | zzzeek: awesome, thank you. what is the magic one-liner to invalidate the connection given a session that you know is questionable? I can try to test out solutions here | 20:58 |
| zzzeek | session.connection().invalidate() | 21:00 |
| *** randallburt1 has quit IRC | 21:00 | |
| zzzeek | but that should be *before* you'd call rollback() | 21:00 |
| zzzeek | zaneb: tehre's better ways to get more directly at the connection if that isn't working | 21:00 |
| zzzeek | there's a "catch error" event | 21:00 |
| zzzeek | wher you can promote any exception you want to be "invalidate" | 21:00 |
| zzzeek | zaneb: it's actually kind of a one liner in oslo.db :) but let me at least confirm this behavior | 21:01 |
| zaneb | zzzeek: thanks, let me try it out and I'll get back to you | 21:04 |
| zzzeek | zaneb: how come in http://paste.openstack.org/show/582316/ i dont see the greenlet exception anywhere? | 21:05 |
| *** ccamacho has quit IRC | 21:06 | |
| *** annegentle has quit IRC | 21:13 | |
| *** rajpatel has quit IRC | 21:15 | |
| zaneb | zzzeek: I'm pretty sure because the ResourceClosedError came up during an exception handler (for the GreenletExit) that was trying to clean up as the thread was exiting | 21:17 |
| zzzeek | zaneb: that is possible | 21:18 |
| zaneb | zzzeek: so we're like "oh, ok, so this failed, I'll write the failed state to the db, WHOOPS" | 21:18 |
| *** rajpatel has joined #heat | 21:22 | |
| *** Zer0Byte__ has quit IRC | 21:24 | |
| *** annegentle has joined #heat | 21:28 | |
| *** rpothier has quit IRC | 21:33 | |
| *** annegentle has quit IRC | 21:33 | |
| zzzeek | zaneb: is this basically greenlet.kill() ? | 21:36 |
| zzzeek | zaneb: if i just make a greenlet talk to a DB and do kill(), I get pretty much one error w/ pymysql and it's a RuntimeError | 21:37 |
| zzzeek | GreenletExit seems to only occur from wait() ? | 21:37 |
| *** cwolferh has quit IRC | 21:38 | |
| *** akshai has quit IRC | 21:39 | |
| zaneb | zzzeek: hmm, I wonder if pymysql is raising RuntimeError in its place somehow? | 21:40 |
| zzzeek | zaneb: BufferedReader is raising it. not sure if my test case is setting things up correctly | 21:41 |
| zaneb | http://eventlet.net/doc/modules/greenthread.html#eventlet.greenthread.kill | 21:41 |
| zzzeek | zaneb: this is the only error i can get pymysql to raise so far http://paste.openstack.org/show/582329/ | 21:41 |
| zaneb | o.O | 21:42 |
| zzzeek | zaneb: let me try SQL that is slower | 21:42 |
| zzzeek | zaneb: these timeouts occur when the DB is hung or soemthing? | 21:42 |
| zaneb | zzzeek: no, but because eventlet it co-operative multitasking, you're pretty much guaranteed to swap into the thread when it's doing I/O | 21:43 |
| *** spzala has quit IRC | 21:43 | |
| zzzeek | zaneb: oh. but that doesn't raise greenlet errors | 21:44 |
| zaneb | (that or something else that eventlet has monkey-patched, like sleep()) | 21:44 |
| zaneb | zzzeek: well, when you call kill() from another thread, that thread is running, and the target thread is in the middle of some blocking, monkey-patched eventlet call | 21:45 |
| zzzeek | zaneb: yup got that | 21:45 |
| zaneb | zzzeek: this bug applies equally to e.g. KeyboardInterrupt, but eventlet acts like a magnet for trouble by forcing the exception to always be raised at the worst possible moment :D | 21:46 |
| * zaneb slow claps for eventlet. again. | 21:46 | |
| *** spotz is now known as spotz_zzz | 21:47 | |
| *** rajpatel has quit IRC | 21:52 | |
| *** rajpatel has joined #heat | 21:57 | |
| zzzeek | zaneb: I can't make any bug happen. been seeing this error for years can never isolate it | 22:00 |
| zzzeek | zaneb: https://gist.github.com/zzzeek/b762218a5f855ec7d1c3a998fe996893 i | 22:01 |
| zzzeek | I'm out of time to work on this today | 22:01 |
| stevebaker | jasond, zaneb: hey, could I get a review on this heatclient backport so I can do a stable release for tripleo? https://review.openstack.org/#/c/373531/ | 22:03 |
| zaneb | zzzeek: "except Exception as err:" won't catch it, since it's an exit exception. don't know if that makes a difference | 22:03 |
| zzzeek | zaneb: let me try | 22:03 |
| jasond | stevebaker: sure | 22:04 |
| zzzeek | zaneb: okey doke i can see the greenlet exit now but still can't get the pymysql command to break https://gist.github.com/zzzeek/b762218a5f855ec7d1c3a998fe996893 | 22:06 |
| zzzeek | zaneb: i have some mor eideas | 22:06 |
| *** Guest54237 has quit IRC | 22:07 | |
| zzzeek | zaneb: OK it's the rollback(), that gets me partway there | 22:08 |
| zaneb | zzzeek: in my testing, it fails 100% of the time when I see "ROLLBACK" in the logs, and 0% of the time when I don't | 22:08 |
| zzzeek | zaneb: yes ROLLBACK is part of it | 22:09 |
| zzzeek | zaneb: still trying to get a blank cursor.descriptoin | 22:09 |
| zzzeek | zaneb: ok this seems to do it https://gist.github.com/zzzeek/b762218a5f855ec7d1c3a998fe996893 | 22:14 |
| zzzeek | zaneb: and if you set ensure_greenlet_exit_handled it goes away | 22:14 |
| *** cwolferh has joined #heat | 22:17 | |
| zaneb | cool | 22:17 |
| *** hoonetorg has quit IRC | 22:24 | |
| openstackgerrit | Yosef Hoffman proposed openstack/heat-specs: New Resource Type OS::Nova::Quota https://review.openstack.org/373546 | 22:25 |
| *** blomquisg is now known as blomquisg_gone | 22:25 | |
| openstackgerrit | Yosef Hoffman proposed openstack/heat: Add OS::Nova::Quota resource https://review.openstack.org/373548 | 22:26 |
| *** annegentle has joined #heat | 22:27 | |
| openstackgerrit | Yosef Hoffman proposed openstack/heat-specs: New Resource Type OS::Nova::Quota https://review.openstack.org/373546 | 22:29 |
| *** jasond has quit IRC | 22:32 | |
| *** Drago has quit IRC | 22:44 | |
| *** vijendar has joined #heat | 22:47 | |
| *** rajpatel has left #heat | 22:51 | |
| *** vijendar has quit IRC | 22:52 | |
| *** priteau has quit IRC | 22:53 | |
| zzzeek | so zaneb, definitely done for now but this can't be handled by SQLA's event beause the exception here is not a subclass of Exception. I can handle it in enginefacade. But heat needs to use enginefacade :) | 22:55 |
| zaneb | zzzeek: I'm still not clear on why SQLA events can't handle BaseException? | 22:55 |
| zzzeek | zaneb: I can look into it for 1.1 as this introduces major surprises for existing apps that use the handle_error event | 22:56 |
| zaneb | that's fair | 22:56 |
| zzzeek | or, enginefacade can do it which just means you're no longer doing "with session.begin()", you're doing "with enginefacade.begin()" (essentially) | 22:56 |
| *** hoonetorg has joined #heat | 22:56 | |
| zaneb | zzzeek: not trying to be difficult, it just seems like there might be lots of sqlalchemy users who expect to be able to e.g. hit Ctrl-C on their program and have it roll back any in-progress transactions gracefully, other than the ones using enginefacade | 22:57 |
| zzzeek | zaneb: 11 years and this is the first :) | 22:58 |
| zzzeek | or, 3800 bug reports and like 50K mailing list emails, this is the first :) | 22:58 |
| zaneb | lol | 22:59 |
| zzzeek | zaneb: when you kill a DB connection, transaction is gone | 22:59 |
| zzzeek | this is a special case | 22:59 |
| zzzeek | there's no analogue in non-greenlet python | 22:59 |
| zzzeek | maybe a signal handler | 22:59 |
| zzzeek | so keyinterrupt is not a problem | 22:59 |
| zaneb | yes, signal handler is the same as far as I can tell | 22:59 |
| zaneb | KeyboardInterrupt _does_ come from a signal handler, doesn't it? | 23:00 |
| zaneb | ditto SystemExit | 23:00 |
| zaneb | the real difference here is that for a signal handler the exception could be raised randomly between any two python bytecodes, but eventlet guarantees that it will happen during your I/O call | 23:01 |
| zaneb | so it makes it *much* easier to hit the bug | 23:01 |
| zaneb | without eventlet or some sort of fault injection you would have 0 chance of reproducing this reliably | 23:02 |
| *** hoonetorg has quit IRC | 23:02 | |
| zaneb | I'd speculate that's why you've never seen a bug report - nobody has ever got it to do this twice before :D | 23:02 |
| zaneb | zzzeek: can't see how this is a different case: https://docs.python.org/2/library/thread.html#thread.exit | 23:05 |
| zaneb | seems like an exact analogue in non-greenlet python to me | 23:05 |
| zaneb | ah, actually, no. that exits the *current* thread, not another one | 23:07 |
| zzzeek | zaneb: systemexit ends the whole program no ? | 23:08 |
| zaneb | only current thread I believe | 23:08 |
| zzzeek | zaneb: the details of how pymysql is interrupted is likely very different in that case also | 23:09 |
| zaneb | ok, there's no sane way to kill another native thread in python (http://stackoverflow.com/questions/323972/is-there-any-way-to-kill-a-thread-in-python#325528) so you're right, it's only signal handlers that are comparable | 23:14 |
| *** adrian_otto has joined #heat | 23:15 | |
| *** hoonetorg has joined #heat | 23:20 | |
| zzzeek | zaneb: i can make the "except Exception:" be "except self.exeption_we_should_catch" and make an option out of it | 23:24 |
| openstackgerrit | Merged openstack/heat: Re-trigger on update-replace https://review.openstack.org/371572 | 23:26 |
| openstackgerrit | Merged openstack/heat: Fix cancel update for nova server with defined port https://review.openstack.org/306490 | 23:26 |
| stevebaker | zaneb: should we backport those 2 to newton? ^^ | 23:28 |
| *** hoonetorg has quit IRC | 23:29 | |
| *** hoonetorg has joined #heat | 23:31 | |
| *** annegentle has quit IRC | 23:41 | |
| *** Alexey_Abashkin_ has joined #heat | 23:45 | |
| *** Alexey_Abashkin has quit IRC | 23:46 | |
| *** om has quit IRC | 23:50 | |
| *** om has joined #heat | 23:52 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!