| openstackgerrit | Deepak proposed openstack/heat master: Heat support for Tap-as-a-Service resources https://review.openstack.org/589238 | 00:01 |
|---|---|---|
| *** sdake has quit IRC | 00:19 | |
| openstackgerrit | Nakul Dahiwade proposed openstack/heat master: Support for shared services in multi region mode https://review.openstack.org/602473 | 00:23 |
| *** sdake has joined #heat | 00:27 | |
| *** sdake has quit IRC | 01:19 | |
| *** sdake has joined #heat | 01:23 | |
| *** _fragatina has quit IRC | 02:05 | |
| *** _fragatina has joined #heat | 02:06 | |
| *** _fragatina has quit IRC | 02:07 | |
| openstackgerrit | johjuhyun proposed openstack/heat master: Heat templates doesn't support AZ for trove cluster https://review.openstack.org/631968 | 02:34 |
| *** sdake has quit IRC | 03:00 | |
| *** maddtux has joined #heat | 03:04 | |
| *** ramishra has joined #heat | 03:23 | |
| *** ricolin has joined #heat | 03:29 | |
| *** skramaja has joined #heat | 03:32 | |
| *** k_mouza has joined #heat | 04:07 | |
| *** k_mouza has quit IRC | 04:11 | |
| *** mikecmpbll has quit IRC | 04:23 | |
| *** nnsingh has joined #heat | 05:11 | |
| *** _fragatina has joined #heat | 05:23 | |
| *** _fragatina has quit IRC | 05:24 | |
| *** _fragatina has joined #heat | 05:24 | |
| *** _fragatina has quit IRC | 05:46 | |
| *** _fragatina has joined #heat | 05:46 | |
| openstackgerrit | Merged openstack/heat-specs master: Change openstack-dev to openstack-discuss https://review.openstack.org/621873 | 06:30 |
| ricolin | zaneb, around? | 06:55 |
| zaneb | yep | 06:55 |
| ricolin | I got a question about the temp file design in remote stack | 06:55 |
| ricolin | so if we really go with create a new tempfile for each remote client call, isn't that means we will keep regenerate file each time when we creat/delete/update/check_complete? | 06:57 |
| ricolin | zaneb, ^^^ | 06:59 |
| *** radeks_ has joined #heat | 07:11 | |
| *** jawad_axd has joined #heat | 07:13 | |
| *** radeks_ has quit IRC | 07:17 | |
| zaneb | ricolin: afraid so, yes | 07:18 |
| *** radeks_ has joined #heat | 07:23 | |
| *** jawad_axd has quit IRC | 07:43 | |
| *** gkadam has joined #heat | 07:51 | |
| *** e0ne has joined #heat | 07:56 | |
| openstackgerrit | Zane Bitter proposed openstack/heat master: Heat support for Tap-as-a-Service resources https://review.openstack.org/589238 | 07:56 |
| *** jtomasek has joined #heat | 08:00 | |
| ricolin | #startmeeting heat | 08:01 |
| openstack | Meeting started Wed Jan 30 08:01:23 2019 UTC and is due to finish in 60 minutes. The chair is ricolin. Information about MeetBot at http://wiki.debian.org/MeetBot. | 08:01 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 08:01 |
| *** openstack changes topic to " (Meeting topic: heat)" | 08:01 | |
| openstack | The meeting name has been set to 'heat' | 08:01 |
| ricolin | #topic roll call | 08:01 |
| *** openstack changes topic to "roll call (Meeting topic: heat)" | 08:01 | |
| zaneb | howdy | 08:02 |
| ricolin | o/ | 08:02 |
| openstackgerrit | Zane Bitter proposed openstack/heat master: Rename the client routines for sfc to more generic name https://review.openstack.org/625210 | 08:02 |
| openstackgerrit | Zane Bitter proposed openstack/heat master: Heat support for Tap-as-a-Service resources https://review.openstack.org/589238 | 08:02 |
| ramishra | hi, I won't be around for long though | 08:02 |
| ricolin | ramishra, got it, let's get start | 08:03 |
| ricolin | #topic Do we have a way of tagging high-priority stories for the release? | 08:03 |
| *** openstack changes topic to "Do we have a way of tagging high-priority stories for the release? (Meeting topic: heat)" | 08:03 | |
| ricolin | zaneb, about this, how about using https://storyboard.openstack.org/#!/board/list | 08:04 |
| zaneb | that works, but iirc there are two ways of creating boards | 08:05 |
| zaneb | either adding stuff manually, or automatically using tags | 08:05 |
| ricolin | zaneb, any idea which might be better fit? | 08:06 |
| zaneb | TBH I think tags is easier | 08:06 |
| ramishra | ricolin: there is a wiki page for it;) https://wiki.openstack.org/wiki/StoryBoard/Priority | 08:06 |
| ricolin | So I remember I once create a board here | 08:06 |
| ricolin | https://storyboard.openstack.org/#!/board/71 | 08:06 |
| zaneb | good ol' board 71 | 08:07 |
| * ricolin try to figure out what is a working list | 08:09 | |
| zaneb | that wiki page illustrates some annoying idealogical biases on the part of the storyboard developers | 08:09 |
| zaneb | "nobody does triage properly so we won't let 'em try" | 08:09 |
| ricolin | so shall we keep using tag `high-priority` and use that board 71? | 08:11 |
| ramishra | yeah, tbh I don't like it a bit. It force changes the way things used to work well | 08:11 |
| zaneb | ricolin: no, I think we should have a board for each release | 08:12 |
| *** asmita has joined #heat | 08:12 | |
| ricolin | that can be done | 08:12 |
| zaneb | like "Heat Stein release" and list the stuff targeted for Stein | 08:12 |
| zaneb | maybe showing status in each lane like https://storyboard.openstack.org/#!/board/8 ? | 08:12 |
| ricolin | yes, that sounds better | 08:14 |
| ricolin | I guess instead say it's mid/hi/low, just list the things we need to do and see how we doing with it | 08:15 |
| ricolin | s/instead say/instead of saying/ | 08:16 |
| ramishra | https://docs.openstack.org/infra/storyboard/gui/theory.html, I think it suggest tracking stuff like trello boards (backlog, in-progess, complete etc) for releases | 08:16 |
| zaneb | yeah, if it doesn't end up getting done then it wasn't High, and we bump it to the next release ;) | 08:16 |
| ramishra | rather than based on priority | 08:16 |
| ricolin | so let's quick brainstorm here, I think we all agree on give a release based Board/working list, so what state we should add in | 08:17 |
| zaneb | a tag like heat-stein-target maybe? | 08:19 |
| *** e0ne has quit IRC | 08:20 | |
| ricolin | how can we move around the state | 08:21 |
| ricolin | zaneb, I think the tag is good | 08:21 |
| ricolin | Infra team use task instead:) | 08:24 |
| zaneb | disadvantage of a tag is that anyone can set it, but I don't think we need to care about that | 08:25 |
| ricolin | zaneb, I think we can always go through the Story and see who added that tag, so it's fine IMO | 08:26 |
| zaneb | yeah | 08:26 |
| ricolin | So I need some more time to learn about how to move things around state(todo->inprogress) in a board | 08:27 |
| ramishra | ricolin: gotta go, btw, heat-agents gate is broken. unit tests are broken with paunch I think, if someone has time to fix before I find time to check it. | 08:27 |
| ricolin | So what state we need? | 08:27 |
| ricolin | ramishra, I will go for that right after meeting and before I need to go too:) | 08:28 |
| ricolin | ramishra, thx! | 08:28 |
| *** shubham_potale_ has joined #heat | 08:28 | |
| zaneb | ricolin: I think you create each lane with a different query, so e.g. under review column is is has tag heat-stein-target and status==Review | 08:28 |
| ricolin | zaneb, I think that's why infra team use task I assume | 08:29 |
| zaneb | ricolin: maybe. that board is waaay old. infra-cloud was decommissioned some time back | 08:30 |
| ricolin | zaneb, I will try to see if tag and task state can be query at the same time | 08:31 |
| *** bnemec has joined #heat | 08:31 | |
| zaneb | you can if I'm reading https://docs.openstack.org/infra/storyboard/gui/theory.html#worklists-and-boards correctly | 08:32 |
| ricolin | TBH, storyboard still slow | 08:32 |
| zaneb | yup :/ | 08:33 |
| ricolin | Is it because the server issue? or it's because the app it self? | 08:33 |
| zaneb | so is bugzilla though | 08:34 |
| zaneb | and launchpad wasn't great | 08:34 |
| zaneb | I think they fixed some stuff on the server, and created some missing indexes | 08:34 |
| zaneb | but at the end of the day, all single-page JS apps that retrieve all their data from an API are slow | 08:36 |
| zaneb | *cough*Gerrit | 08:36 |
| ricolin | appears we can set priority to a task, but I don't know how | 08:36 |
| zaneb | lol | 08:37 |
| zaneb | shall we move on? | 08:37 |
| ricolin | yes we should | 08:37 |
| ricolin | #action rico will go setup Board for Stein release | 08:38 |
| *** mikecmpbll has joined #heat | 08:38 | |
| ricolin | #topic This seems like it would be a good feature for Stein | 08:39 |
| *** openstack changes topic to "This seems like it would be a good feature for Stein (Meeting topic: heat)" | 08:39 | |
| ricolin | #link https://storyboard.openstack.org/#!/story/2003579 | 08:39 |
| zaneb | this is my nomination for the first story to receive the tag for stein ;) | 08:40 |
| zaneb | probably not much more to say about it since ramishra already left :D | 08:41 |
| ricolin | I believe so:) | 08:41 |
| zaneb | but I think this would be a useful achievement for the release, and it's an easy review | 08:41 |
| zaneb | next topic :) | 08:42 |
| ricolin | zaneb, I always think we should improve what we doing with rollback, so agree on that | 08:42 |
| *** gkadam has quit IRC | 08:42 | |
| ricolin | #topic multi-cloud | 08:43 |
| *** openstack changes topic to "multi-cloud (Meeting topic: heat)" | 08:43 | |
| ricolin | So just back around that idea for where to put auth info for another cloud, and what's best for tempfile | 08:43 |
| *** ttsiouts has quit IRC | 08:44 | |
| ricolin | so I know the concern about temp file | 08:45 |
| *** ttsiouts has joined #heat | 08:45 | |
| ricolin | I mean it do leave file behind if we have error else where | 08:45 |
| zaneb | it's worse than that, it leaks a file descriptor | 08:46 |
| ricolin | yeah, so we must do some change | 08:46 |
| *** maddtux has quit IRC | 08:46 | |
| ricolin | I can change it to regenerate file for each remote client request | 08:47 |
| ricolin | but wondering if there's any better way we can targeting here | 08:47 |
| zaneb | I think that's the way to go. writing a tempfile is relatively cheap | 08:47 |
| zaneb | compared to the HTTP request, it's certainly cheap | 08:47 |
| ricolin | it is | 08:48 |
| zaneb | the only alternative i can really is is to regenerate both the tempfile *and* the context for each request. which is obviously worse | 08:49 |
| ricolin | zaneb, NO! | 08:49 |
| *** ttsiouts has quit IRC | 08:49 | |
| * zaneb just thought of a nasty hack | 08:50 | |
| ricolin | I guess we have to wait for python 3.x to be the oldest before get rid of temp file | 08:50 |
| ricolin | zaneb, what hack | 08:50 |
| zaneb | we could override _action_recorder() to create the context in there and clean it up at the end | 08:51 |
| ricolin | yes | 08:52 |
| ricolin | nasty | 08:52 |
| zaneb | I told you so | 08:52 |
| ricolin | a good idea, but nasty | 08:52 |
| zaneb | so in short, a good idea | 08:53 |
| ricolin | I can live with that:) LOL | 08:54 |
| ricolin | zaneb, regarding context with auth and auth_type here https://review.openstack.org/#/c/580550/9/heat/common/context.py | 08:55 |
| zaneb | hmm, actually that idea still sucks for getting the attributes | 08:56 |
| zaneb | maybe we should override node_data() to clear the context | 08:57 |
| ricolin | so the node date have to first check if it needs to be replaced and validate the remote authentication | 08:59 |
| ricolin | Also auth and auth_type can be use in server side too, and it's not necessary be stored in clouds.yaml. | 09:00 |
| *** asmita has quit IRC | 09:02 | |
| zaneb | ricolin: what would that look like? where would the data come from? | 09:03 |
| zaneb | on the server side | 09:04 |
| ricolin | With WSGI, we use HTTP_X_AUTH_KEY to store password when heat/common/auth_password.py, maybe we can use it to store auth too? | 09:04 |
| *** mikecmpbll has quit IRC | 09:06 | |
| zaneb | ricolin: do clients do that? because if they don't it's irrelevant whether we can read it | 09:07 |
| ricolin | It's not in our client design, so that's for Rest API only now | 09:11 |
| *** asmita has joined #heat | 09:14 | |
| asmita | ricolin : Hi | 09:15 |
| ricolin | asmita, Hi back | 09:15 |
| ricolin | :) | 09:15 |
| zaneb | I don't see the point in implementing any auth method that the rest of OpenStack doesn't support | 09:15 |
| *** mikecmpbll has joined #heat | 09:16 | |
| ricolin | zaneb, I guess we can override _auth_plugin in context and move ks_loading.get_plugin_loader to remote stack too | 09:18 |
| zaneb | that would be less risky imho | 09:19 |
| *** gfidente has joined #heat | 09:20 | |
| ricolin | BTW let's end the meeting lol | 09:20 |
| ricolin | #endmeeting | 09:21 |
| *** openstack changes topic to "OpenStack Heat (Logs: http://eavesdrop.openstack.org/irclogs/%23heat/)" | 09:21 | |
| openstack | Meeting ended Wed Jan 30 09:21:25 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 09:21 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/heat/2019/heat.2019-01-30-08.01.html | 09:21 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/heat/2019/heat.2019-01-30-08.01.txt | 09:21 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/heat/2019/heat.2019-01-30-08.01.log.html | 09:21 |
| *** k_mouza has joined #heat | 09:21 | |
| ricolin | I got to go, but will be keep working on those when I'm back | 09:21 |
| ricolin | zaneb, ^^^ | 09:22 |
| zaneb | ok, thanks ricolin! | 09:22 |
| *** k_mouza has quit IRC | 09:22 | |
| *** k_mouza has joined #heat | 09:23 | |
| *** ricolin has quit IRC | 09:28 | |
| *** maddtux has joined #heat | 09:38 | |
| *** ttsiouts has joined #heat | 09:38 | |
| *** ramishra has quit IRC | 09:40 | |
| *** ramishra has joined #heat | 09:40 | |
| * pas-ha hi all, have a strange thing in my Heat as of Queens, wonder if it is a know issue / is there a workaround | 10:00 | |
| pas-ha | hi all, have a strange thing in my Heat as of Queens, wonder if it is a know issue / is there a workaround | 10:00 |
| pas-ha | we have not huge but deep enough stacks running as a base for CI, and at some occasions when the job is aborted, the stack which is in CREATE_IN_PROGRESS is attempted to be DELETEd | 10:01 |
| pas-ha | after that sometimes the stack is being stuck in DELETE_IN_PROGRESS, with various symptoms like the resource of nested stack is DELETE_IN_PROGRESS, but the stack show on it shows CREATE_COMPLETE or CREATE_IN_PROGRESS | 10:02 |
| pas-ha | also heat-engine logs contain warnings like 'Failed to stop all workers for stack ... , stack cancel is not complete' | 10:03 |
| pas-ha | anyone seen such behavior? | 10:03 |
| pas-ha | most of those stuck in CREATE_IN_PROGRESS were as it seems waiting on some WaitCondition inside the nested stack | 10:04 |
| pas-ha | I'm trying to come up with a simple enough repro for now, will keep you posted | 10:05 |
| therve | pas-ha: You probably have rollback enabled. Wait conditions are failing, and heat is trying to delete the stack | 10:09 |
| *** e0ne has joined #heat | 10:18 | |
| pas-ha | AFAIK no | 10:19 |
| *** sshnaidm has quit IRC | 10:19 | |
| pas-ha | there's no rollback in neither create nor delete | 10:19 |
| pas-ha | but I'll doublecheck | 10:20 |
| pas-ha | is there a config option that sets rollback by default? | 10:20 |
| pas-ha | I've also found that the cancel action has hard-coded number of retries and timeout between them (3 and 5s respectively) will try to unhardcode those too | 10:21 |
| *** ramishra has quit IRC | 10:26 | |
| *** sshnaidm has joined #heat | 10:26 | |
| pas-ha | that's the kind of descrepancy between status of nested stack as a resource vs nested stack as a stack http://paste.openstack.org/show/744240/ | 10:35 |
| pas-ha | therve: another question - I could manually delete all the created resources and then abandon the stack, but will "stack abandon" delete all the Heat-specific resources like wait conditions? | 10:37 |
| *** ramishra has joined #heat | 10:39 | |
| therve | pas-ha: If you delete the resources you can delete the stack | 10:39 |
| therve | abandon probably works? No idea | 10:40 |
| pas-ha | I'll try but for now it seems to be hanging on delete of a Heat-specific resource that I can not delete | 10:40 |
| therve | Okay | 10:41 |
| therve | I'd hope abandon wouldn't be different | 10:41 |
| *** ttsiouts has quit IRC | 11:25 | |
| *** k_mouza_ has joined #heat | 11:41 | |
| *** k_mouza has quit IRC | 11:45 | |
| *** k_mouza_ has quit IRC | 11:46 | |
| *** k_mouza has joined #heat | 11:47 | |
| *** k_mouza has quit IRC | 11:47 | |
| *** k_mouza has joined #heat | 11:48 | |
| *** ttsiouts has joined #heat | 11:58 | |
| *** sdake has joined #heat | 12:01 | |
| *** gkadam has joined #heat | 12:08 | |
| *** gkadam is now known as gkadam-bmgr | 12:19 | |
| openstackgerrit | Vishal Manchanda proposed openstack/heat-dashboard master: Fix gate failures by a new pycodestyle https://review.openstack.org/633928 | 12:21 |
| *** k_mouza has quit IRC | 12:37 | |
| *** maddtux has quit IRC | 12:43 | |
| *** sdake has quit IRC | 12:43 | |
| *** sdake has joined #heat | 12:45 | |
| *** e0ne has quit IRC | 12:48 | |
| *** Ashika has joined #heat | 12:58 | |
| Ashika | Hello..When we trying to delete the stack it is going into delete_failed state and these are the errors found in heat-engine.log | 12:59 |
| Ashika | 2019-01-30 21:37:01.086 24056 DEBUG heat.engine.stack_lock | 12:59 |
| Ashika | Can anyone help us in knowing what is stack_lock exactly means? | 13:00 |
| *** _fragatina has quit IRC | 13:38 | |
| *** _fragatina has joined #heat | 13:38 | |
| *** e0ne has joined #heat | 13:46 | |
| *** ekultails has joined #heat | 14:00 | |
| *** jcoufal has joined #heat | 14:02 | |
| *** vabada_ has quit IRC | 14:05 | |
| *** sdake has quit IRC | 14:20 | |
| *** sdake has joined #heat | 14:34 | |
| *** k_mouza has joined #heat | 14:35 | |
| *** skramaja has quit IRC | 14:42 | |
| *** mchlumsky has joined #heat | 14:43 | |
| *** sdake has quit IRC | 14:49 | |
| *** sdake has joined #heat | 14:55 | |
| *** sdake has quit IRC | 15:11 | |
| *** k_mouza has quit IRC | 15:13 | |
| *** sdake has joined #heat | 15:13 | |
| *** gkadam-bmgr has quit IRC | 15:57 | |
| *** _fragatina has quit IRC | 16:01 | |
| *** sdake has quit IRC | 16:14 | |
| *** sdake has joined #heat | 16:22 | |
| pas-ha | Ashika: this is DEBUG statement, what is the actual ERROR in the logs? | 16:23 |
| openstackgerrit | Merged openstack/heat master: Rename the client routines for sfc to more generic name https://review.openstack.org/625210 | 16:23 |
| openstackgerrit | Merged openstack/heat master: Heat support for Tap-as-a-Service resources https://review.openstack.org/589238 | 16:23 |
| *** Ashika has quit IRC | 16:26 | |
| *** e0ne has quit IRC | 16:31 | |
| *** e0ne has joined #heat | 16:31 | |
| *** ramishra has quit IRC | 16:40 | |
| *** k_mouza has joined #heat | 16:47 | |
| *** sdake has quit IRC | 16:51 | |
| *** k_mouza_ has joined #heat | 16:57 | |
| *** k_mouza has quit IRC | 16:59 | |
| *** sdake has joined #heat | 17:00 | |
| *** radeks_ has quit IRC | 17:02 | |
| *** radeks_ has joined #heat | 17:02 | |
| *** _fragatina has joined #heat | 17:06 | |
| *** bnemec has quit IRC | 17:14 | |
| *** k_mouza_ has quit IRC | 17:15 | |
| *** ttsiouts has quit IRC | 17:18 | |
| *** e0ne has quit IRC | 17:31 | |
| *** mikecmpbll has quit IRC | 17:39 | |
| *** mikecmpbll has joined #heat | 18:07 | |
| *** shardy has quit IRC | 18:15 | |
| *** sshnaidm is now known as sshnaidm|afk | 18:25 | |
| *** ttsiouts has joined #heat | 18:28 | |
| *** ekultails has quit IRC | 18:33 | |
| *** e0ne has joined #heat | 19:06 | |
| *** ekultails has joined #heat | 19:11 | |
| *** e0ne has quit IRC | 20:05 | |
| *** sdake has quit IRC | 20:06 | |
| *** sdake has joined #heat | 20:07 | |
| *** ttsiouts has quit IRC | 20:11 | |
| *** e0ne has joined #heat | 20:17 | |
| *** e0ne has quit IRC | 20:19 | |
| *** ttsiouts has joined #heat | 20:33 | |
| *** sdake has quit IRC | 20:39 | |
| *** sdake has joined #heat | 20:42 | |
| *** sdake has quit IRC | 20:44 | |
| *** sdake has joined #heat | 20:46 | |
| *** e0ne has joined #heat | 20:48 | |
| *** gfidente has quit IRC | 20:48 | |
| *** ttsiouts has quit IRC | 20:49 | |
| *** ttsiouts has joined #heat | 20:50 | |
| *** ttsiouts has quit IRC | 20:53 | |
| *** ekultails has quit IRC | 20:57 | |
| *** sdake has quit IRC | 21:01 | |
| *** k_mouza has joined #heat | 21:02 | |
| *** sdake has joined #heat | 21:02 | |
| *** shubham_potale_ has quit IRC | 21:03 | |
| *** ekultails has joined #heat | 21:05 | |
| *** k_mouza has quit IRC | 21:06 | |
| *** e0ne has quit IRC | 21:18 | |
| *** radeks_ has quit IRC | 21:19 | |
| *** higgins has quit IRC | 21:25 | |
| *** higgins has joined #heat | 21:26 | |
| zaneb | pas-ha: so obviously the delete operation involves a traversal of the resource graph. when it gets to a node that has locked a resource, it will stop and wait for the resource to be unlocked again | 21:28 |
| zaneb | in your case the CREATE_IN_PROGRESS wait condition has locked the resource | 21:28 |
| zaneb | this is why we have the cancel thing, it's job is to stop all of the tasks working on previous traversals, which should result in them releasing their locks on the resources | 21:29 |
| zaneb | the log you mentioned indicates that there are locks that still have not been released 3x5s after telling them to stop | 21:31 |
| zaneb | the key to the mystery is figuring out why the worker tasks are not being stopped | 21:32 |
| zaneb | (or if they are, why the lock is not being released) | 21:38 |
| *** sdake has quit IRC | 21:38 | |
| *** e0ne has joined #heat | 21:47 | |
| *** sdake has joined #heat | 21:50 | |
| *** openstackgerrit has quit IRC | 21:50 | |
| zaneb | pas-ha: I can't see any obvious bugs in the code, and nothing is special about WaitCondition | 21:51 |
| *** e0ne has quit IRC | 21:58 | |
| *** ekultails has quit IRC | 22:02 | |
| *** sdake has quit IRC | 23:01 | |
| *** sdake has joined #heat | 23:29 | |
| *** sdake has quit IRC | 23:33 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!