20:00:09 <asalkeld> #startmeeting heat
20:00:11 <openstack> Meeting started Wed Jan 14 20:00:09 2015 UTC and is due to finish in 60 minutes.  The chair is asalkeld. Information about MeetBot at http://wiki.debian.org/MeetBot.
20:00:12 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
20:00:15 <openstack> The meeting name has been set to 'heat'
20:00:18 <ryansb> hello folks
20:00:22 <pas-ha> o/
20:00:27 <tspatzier> hi
20:00:33 <spzala> hi
20:00:35 <asalkeld> hi
20:00:41 <shardy> o/
20:00:48 <asalkeld> #link https://wiki.openstack.org/wiki/Meetings/HeatAgenda
20:00:53 <jpeeler> hi
20:01:58 <asalkeld> #topic Review action items from last meeting
20:02:08 <asalkeld> #link http://eavesdrop.openstack.org/meetings/heat/2014/heat.2014-12-17-20.00.html
20:02:19 <skraynev_> \oo/
20:02:21 <asalkeld> I don't think i did that:-O
20:02:42 <asalkeld> #action asalkeld to write up an email to the ml re: meetup
20:02:52 <asalkeld> oops, I'll get on that
20:03:17 <asalkeld> #topic Adding items to the agenda
20:03:26 <asalkeld> any new topics for today?
20:03:38 <zaneb> o/
20:04:10 <stevebaker> hi
20:04:13 <skraynev_> probably I had one small question about functional tests
20:04:20 <asalkeld> ok
20:04:22 <stevebaker> (slightly distracted)
20:04:50 <asalkeld> maybe zaneb can give us an update on his convergence work
20:04:55 <shardy> Are we still doing critical issues sync as an item?
20:05:04 <asalkeld> we can
20:05:10 <zaneb> that won't take long :/
20:05:20 <shardy> asalkeld: Ok, just wanted to draw attention to https://bugs.launchpad.net/heat/+bug/1384750
20:05:25 <asalkeld> #topic critical issues sync as an item?
20:05:35 <shardy> which has been confirmed today and isn't targetted to k2
20:05:46 <asalkeld> #link https://bugs.launchpad.net/heat/+bug/1384750
20:05:47 <shardy> I reported it but I don't have a fix atm
20:06:10 <asalkeld> ok, maybe i can look at it
20:06:37 <shardy> Ok, thanks, just wanted to get some eyes on it
20:07:04 <asalkeld> #topic functional tests
20:07:07 <asalkeld> skraynev_: ...
20:07:54 <skraynev_> cool. I wonder what we plan to do for functional test, e.i. do we want to have test create/delete for each resource?
20:08:50 <asalkeld> skraynev_: not sure, i am adding more logic tests into functional
20:09:11 <asalkeld> updates, nested with env, files
20:09:27 <asalkeld> deeper tests that should not be in unit tess
20:09:44 <skraynev_> yes, but it covers basic resource structure.
20:09:52 <asalkeld> but you might be thinking of scenario tests
20:10:18 <asalkeld> you can't  really test all resources in isolation
20:10:19 <skraynev_> scenario is not equal functional?
20:10:36 <skraynev_> agree
20:10:38 <shardy> skraynev_: FWIW, I don't think we should duplicate stuff we can test in the unit tests, as there's more overhead in doing the functional testing
20:10:39 <stevebaker> skraynev_: tests which have groups of resources interacting might be more useful
20:10:46 <asalkeld> scenario are linking a bunch of servies together
20:11:03 <skraynev_> stevebaker: sure
20:11:19 <shardy> yeah, and functional are about proving heat behaves in a specific way, testing multiple levels of our implementation
20:12:19 <asalkeld> skraynev_: scenario is for autoscaling
20:12:32 <skraynev_> ok, so you have not objections if it be some bunches f.e. for neutron, sahara, autoscaling
20:12:35 <asalkeld> and other non trivial usecasese
20:12:43 <pas-ha> so we better come up with IRL-like usage scenarios
20:12:48 <asalkeld> skraynev_: that would be great
20:13:19 <asalkeld> pas-ha: "IRL"?
20:13:26 <pas-ha> in real life
20:14:08 <skraynev_> common use-cases which we use constantly on deployments ;)
20:14:10 <asalkeld> dang i am bad at acronym
20:14:28 <asalkeld> ok, lets move on
20:14:43 <asalkeld> #topic zane update on convergence
20:15:09 <asalkeld> zaneb: ..
20:15:13 <zaneb> still working on it
20:15:21 <zaneb> struggling to find time at the moment
20:15:36 <asalkeld> i see
20:15:40 <zaneb> but close to putting something out there for discussion
20:15:56 <zaneb> that resolves the last issues about storing progress
20:16:06 <asalkeld> please priortise as much as possible
20:16:22 <zaneb> unfortunately by throwing away a lot of the efficiency gains I hoped to include :/
20:16:46 <zaneb> but it doesn't look like there is much of a choice
20:17:14 <asalkeld> correct first
20:17:34 <zaneb> indeed
20:17:34 <asalkeld> what efficiency gains are we taking about?
20:17:58 <zaneb> avoiding lots and lots and lots of writes to the database
20:18:26 <asalkeld> so now it's going to be chatty
20:18:32 <asalkeld> :(
20:18:39 <zaneb> basically to ensure we maintain all of the state we're going to be hammering the DB
20:19:20 <asalkeld> ok
20:19:31 <zaneb> but if we don't then it's impossible to correctly hand-off from a previous update that was still in progress on a resource to the current update
20:19:55 <asalkeld> sure
20:20:14 <zaneb> other than polling in a loop, which is (a) unpalatable, and (b) also hammers the DB anyway
20:21:02 <asalkeld> zaneb: keep anat and co. in the loop they are frustated and need some idea of the progress
20:21:27 <zaneb> ok
20:21:32 <shardy> didn't we have to optimze DB access due to scale limitations related to metadata/DB access previously?
20:22:01 <asalkeld> stevebaker: did that
20:22:07 <shardy> e.g if we make the DB access much more high bandwidth, will it break the problem we're trying to solve?
20:22:15 <shardy> e.g the scalability problem
20:22:18 <stevebaker> yes it is much better now
20:22:21 <zaneb> possibly
20:22:31 <asalkeld> i think that was the guest polling
20:22:39 <shardy> I suppose scaling the DB is more of a solved problem, but I'm wary of reintroducing those problems again..
20:22:40 <zaneb> although hopefully itwon't get as O(n^2) as it was before stevebaker's improvements
20:22:49 <stevebaker> it was the overhead of stack parsing
20:22:59 <asalkeld> i see
20:23:13 <shardy> Okay, cool, as long as we're not going back to square one in that regard :)
20:23:15 <stevebaker> joined fetches helped a lot
20:23:43 <asalkeld> lets move on
20:23:51 <asalkeld> #topic kilo-2
20:24:00 <asalkeld> #link https://launchpad.net/heat/+milestone/kilo-2
20:24:18 <asalkeld> that's due on 5 feb
20:24:42 <asalkeld> so if you have a bp that is not likely to make it, please tell me
20:25:09 <asalkeld> i don't know about https://blueprints.launchpad.net/heat/+spec/support-cinderclient-v2
20:25:20 <asalkeld> in "unknown state"
20:26:10 <asalkeld> any questions about kilo-2 and/or specs?
20:26:25 <pas-ha> I think we have some logic around cinder v2 already
20:26:51 <pas-ha> now we use create the client with max version available
20:27:04 <asalkeld> pas-ha: maybe that is already implemented?
20:27:08 <pas-ha> and jump around in couple of places as APIs are not exactly the same
20:27:21 <asalkeld> i'll check the git history
20:27:35 <pas-ha> need to theck with the author what exactly was meant by this bp
20:27:54 <asalkeld> i did send an email, so far no response
20:28:55 <asalkeld> it's implemented
20:29:02 <asalkeld> Adrien Vergé <adrien.verge@numergy.com>
20:29:14 <asalkeld> 2014-09-23
20:29:17 <asalkeld> nice
20:29:59 <asalkeld> ok, we can move to open discussion now
20:30:07 <asalkeld> #topic open discussion
20:30:21 <asalkeld> or close early, I am easy
20:30:50 <asalkeld> blueprint support-cinder-api-v2
20:31:06 <asalkeld> the blueprint was wrong, and didn't update the spec
20:31:25 <pas-ha> cool
20:31:49 <pas-ha> I have a question about removing scheduler logic from handle*/check*complete methods
20:32:16 <asalkeld> pas-ha: i think that's only complex stuff
20:32:24 <asalkeld> zaneb: might remember
20:32:26 <zaneb> pas-ha: do it
20:32:52 <pas-ha> I tried to get rid of them in current architecture, but ended up with simplified TaskRunner :(
20:33:25 <zaneb> pas-ha: FSMs are your friend
20:33:43 <shardy> Does anyone know the status of the Mistral resources?
20:33:54 <zaneb> just pass the state as the parameter to check_*_complete
20:33:58 <shardy> I've seen some updates to the spec lately, but interested how actively the code is being worked on
20:34:03 <zaneb> instead of passing a continuation
20:34:08 <pas-ha> The client is ready,
20:34:17 <pas-ha> resources are being worked on
20:34:33 <asalkeld> shardy: yeah it's active
20:34:39 <pas-ha> for starters there'd be workflow and crontimer
20:35:04 <pas-ha> zaneb, thanks, will dif in than direction
20:35:17 <asalkeld> tho' shardy want to clarify
20:35:26 <asalkeld> not sure what you are asking
20:35:31 <shardy> asalkeld: I'd love it if folks could follow up on my application HA thread on openstack-dev with more concrete examples of how the mistral integration may help solve the problem
20:35:55 <shardy> I know you and zaneb were in favour of that direction, but I'm fuzzy on the details tbh
20:36:08 <shardy> e.g exactly how the interaction between the workflow and orchestration would work
20:36:30 <asalkeld> i have forgotten that thread :-O, need to find it
20:36:45 <zaneb> Mistral calling Heat would be a step in the workflow
20:36:59 <shardy> http://lists.openstack.org/pipermail/openstack-dev/2014-December/053447.html
20:37:03 <shardy> asalkeld: ^^ ;)
20:37:09 <asalkeld> ta
20:37:41 <asalkeld> so instead of  type: OS::Heat::ScalingPolicy
20:37:45 <asalkeld> you have a workflow
20:38:03 <asalkeld> and the workflow, does an update on the stack
20:38:11 <asalkeld> to do what every you want
20:38:24 <zaneb> yeah, exactly
20:38:31 <asalkeld> (more flexible that keep adding new stuff to heat resources)
20:39:13 <asalkeld> i feel like heat autoscaling is too inflexible for many
20:39:24 <shardy> Ok, that sounds OK (although maybe s/update/signal ref my mail)
20:39:30 <skraynev_> shardy: I may ask folks who is working on it to give (may be in g_docs) some examples of use cases. Or may be demo record.
20:39:56 <shardy> but I'd really like more examples of what that workflow would look like, e.g a Heat resource which defines a mistral workflow, which then calls heat
20:39:57 <asalkeld> yeah we need some concrete examples
20:40:30 <asalkeld> in my defense, i did ask for that in the spec
20:40:35 <shardy> Then, when the resources start getting posted, we can experiment with the HA use-case I described and figure out if it'll work :)
20:40:39 <skraynev_> asalkeld: :)
20:40:46 <shardy> asalkeld: Yeah, so did I :)
20:41:26 <asalkeld> https://review.openstack.org/#/c/143989/7/specs/kilo/mistral-resources.rst
20:41:31 <asalkeld> my top comment
20:42:03 <asalkeld> i like my use case 3 ;)
20:42:13 <skraynev_> but you also understand, that it's not so easy and short demonstrate whole information about use case in spec, IMO
20:43:04 <asalkeld> skraynev_: sure, but blindly making resources that match the upstream api is not good either
20:43:19 <asalkeld> we need to make sure that the template looks sensible
20:43:29 <asalkeld> and it actually solves a problem
20:43:37 <skraynev_> asalkeld: fully agree
20:43:58 <shardy> asalkeld: +1
20:44:16 * asalkeld is amazed at how i am doing without coffee
20:44:19 <asalkeld> :)
20:44:24 <skraynev_> so there is reason, why we want to upload first avaliable code of resources
20:44:44 <skraynev_> it will give us a chance to touch it and play with it
20:45:00 <asalkeld> skraynev_: code is great
20:45:21 <asalkeld> we can do actual experimentation
20:45:47 <asalkeld> i'd suggest mistral/heat integartion is esp. important and we need to get it right
20:46:01 <shardy> Yeah, posting the code early would be good, a lot of us are interested in this :)
20:46:23 <asalkeld> shardy: re: decouple-nested
20:46:30 * shardy hides
20:46:35 <asalkeld> i am onto the update-policy tests
20:46:47 <shardy> asalkeld: Sorry about the test nightmare I gave you... :(
20:46:47 <asalkeld> not too many tests to fix after that
20:47:05 <asalkeld> so i am hopeful of landing it in time
20:47:12 <asalkeld> no worries
20:47:14 <shardy> asalkeld: awesome! I owe you many beers at the next summit :)
20:47:32 <asalkeld> hopefully it will be valueble to convergence too
20:47:51 <shardy> I actually think it'll be a really good thing for scalability with big stacks w/lots of provider resources etc
20:48:09 <shardy> I don't have any numbers to back that up tho :)
20:48:11 <pas-ha> btw, functional tests - looks like all instance group tests were silently skipped, and when enabled one of them fails - https://review.openstack.org/#/c/147136/
20:48:37 <asalkeld> ooo, ok thanks pas-ha
20:49:01 <shardy> ouch, nicely spotted pas-ha
20:49:53 <zaneb> I don't understand why our test framework allows us to have slient-skip-on-error
20:49:53 <asalkeld> pas-ha: hopefully the lab in czech is working today
20:50:17 <asalkeld> zaneb: yeah, we need to just fail
20:50:21 <asalkeld> not skip
20:50:24 <pas-ha> well, I use my beefy desktop box
20:50:38 <pas-ha> that was not an error
20:50:44 <pas-ha> of the test
20:50:50 <skraynev_> same for me
20:50:51 <pas-ha> but error of the config
20:50:53 <asalkeld> i need to get one, just have a laptop
20:50:59 <stevebaker> pas-ha: isn't the keypair only for manually poking at failed tests?
20:51:34 <asalkeld> stevebaker: maybe we just need to remove the use of it
20:51:48 <pas-ha> stevebaker, https://github.com/openstack/heat/blob/master/heat_integrationtests/functional/test_instance_group.py#L116
20:51:53 <stevebaker> asalkeld: I like being able to debug failed tests
20:51:59 <pas-ha> and we have None by default in config
20:52:26 <asalkeld> https://github.com/openstack/heat/blob/master/heat_integrationtests/functional/test_instance_group.py#L158
20:52:57 <stevebaker> pas-ha: ah. That test should be creating its own keypair if one is not configured
20:53:12 <pas-ha> yep, that I have pointed in review
20:53:36 <asalkeld> does launchconfig require the keyname
20:54:13 <asalkeld> cool, not required
20:54:33 <asalkeld> we can just remove references to it
20:54:33 <pas-ha> nope, optional
20:54:54 <pas-ha> but stevebaker likes to debug failed stuff
20:55:03 <pas-ha> and me too frankly
20:55:04 <asalkeld> 5mins left
20:55:15 <asalkeld> pas-ha: i am using a random string there
20:55:30 <asalkeld> don't need a key to debug a random string
20:55:43 <pas-ha> aha, ok
20:55:56 <stevebaker> pas-ha: just do this https://github.com/openstack/heat/blob/master/heat_integrationtests/scenario/test_server_cfn_init.py#L31
20:56:23 <stevebaker> pas-ha: that should probably be a common method
20:56:43 <pas-ha> indeed, coud be moved
20:56:44 <asalkeld> cant test.py setup() do that?
20:57:23 <asalkeld> what ever
20:57:40 <pas-ha> some logic would be needed not to create too many keypairs
20:58:56 <asalkeld> the use of it should be removed from instance group test (as it's not needed there)
20:59:16 <asalkeld> i'll add that to the review
20:59:27 <asalkeld> that is all folks..
20:59:31 <asalkeld> #endmeeting