20:01:05 #startmeeting heat 20:01:05 Meeting started Wed Jul 15 20:01:05 2015 UTC and is due to finish in 60 minutes. The chair is zaneb. Information about MeetBot at http://wiki.debian.org/MeetBot. 20:01:07 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 20:01:10 The meeting name has been set to 'heat' 20:01:34 #topic roll call 20:01:40 I am here 20:01:42 o/ 20:01:46 o/ 20:02:18 o/ 20:02:47 fyi stevebaker is away this week, so I drew the short straw 20:03:29 is it rly only 4 of us here? o_O 20:03:31 o/ 20:03:43 I know ryansb is here because I spoke to him like 4 minutes ago 20:04:04 well, let's get started 20:04:11 #topic Adding items to agenda 20:04:17 #link https://wiki.openstack.org/wiki/Meetings/HeatAgenda#Agenda_.282015-07-015_2000_UTC.29 20:04:27 nice of somebody to create an agenda 20:04:43 * zaneb hasn't read scrollback from last week's 3am meeting yet 20:04:54 any other topics? 20:05:15 nope. looks like a short one today... 20:05:28 my favourite kind :) 20:05:29 not from me, thanks everyone who's answered on my ML yesterday 20:05:43 #topic important reviews 20:05:45 agreed 20:05:54 #link https://etherpad.openstack.org/p/heat-reviews 20:06:40 designate stuff merged so I took them off the list 20:06:57 cool 20:07:11 what do we normally do in this part of the meeting? 20:07:25 talk about important reviews 20:07:39 exhort everyone to review harder? 20:07:51 or beg them to 20:07:53 yep 20:08:06 exhort, beg, wheedle, encourage, etc 20:08:39 so many red exes on the convergence patches 20:09:07 I have 2 questions that I wonder this week 20:09:25 randallburt: they probably all need a rebase after the mock 1.1 funness 20:09:27 The first is related to convergence 20:10:02 zaneb: yeah. someone needs a swift kick for that one 20:10:12 On scale we faced withe following problem: when creating a big stack heat is ddosing nova 20:10:49 Kairat: more than it was with non-convergence? 20:10:50 big = a lot of instances in one stack 20:10:50 Kairat: so that we can keep on topic of the patches for the moment: there's a lot of green on https://review.openstack.org/#/c/153235/. Any reason its still not approved? 20:11:05 yep, it is not convergence 20:11:24 I know we are developing some batching for resource groups 20:11:41 randallburt: I personally don't know ;) 20:12:30 looks like most of the priority reviews are specs... 20:12:46 does that mean nobody is having trouble getting their code reviewed? 20:12:56 or everyone is blocked on specs? :D 20:13:04 zaneb: I have added one about deprecation changes (implementation Hidden status) 20:13:19 skraynev: its approved now 20:13:33 zaneb: everyone scares to do something with convergence without you :) 20:13:45 :( 20:14:13 randallburt: thank you ! :) 20:14:25 zaneb: it';s just my guess ;) 20:14:28 skraynev: np 20:14:51 ok, let's move on. Kairat, we'll get to you topic in open discussion 20:15:02 #topic high-priority bugs 20:15:16 zaneb, ok, thanks 20:15:24 #link http://bit.ly/1FnhaIK 20:15:33 zaneb: note, when I copy-pasted agenda - I didn't check link (I hope it works) 20:15:44 worked for me 20:16:07 fuh. it's good news for me :) 20:17:42 https://bugs.launchpad.net/heat/+bug/1301320 should we changes some statuses for this bug ? 20:17:43 Launchpad bug 1301320 in heat "There are no accessibilities for the backup_stack" [High,Triaged] 20:17:46 how many of these bugs are *actually* in progress I wonder 20:18:06 or it's enough to have tag: fixed-by-convergence ? 20:18:22 I think that's probably enough 20:18:30 we could drop it back to Medium 20:18:43 since realistically we will never fix this except with convergence 20:21:10 zaneb: got it 20:21:35 I even wonder why "we do not have API to access backup stack"? access - yes (just append *), manipulation - no :) 20:22:53 looks like the fix for https://bugs.launchpad.net/heat/+bug/1465204 has actually merged 20:22:53 Launchpad bug 1465204 in heat "convergence: resources not using the correct template on update" [High,Fix committed] - Assigned to Anant Patil (ananta) 20:24:07 pas-ha: the backup stack doesn't look much like any stack a user might recognise. I don't think making it visible through the standard API would be a Good Thing 20:24:38 AFAIR it is visible - stack-show * 20:24:50 if I'm not mistaken 20:25:31 just because backup_stack stored in DB 20:25:49 btw re bug https://bugs.launchpad.net/heat/+bug/1409814 - do we care for Icehouse any more (since the branch is no longer there..) 20:25:49 Launchpad bug 1409814 in heat "unit tests fail on stable/icehouse: "AttributeError: 'module' object has no attribute 'neutronclient'"" [High,Triaged] - Assigned to Pradeep Kumar Singh (pradeep-singh-u) 20:25:52 how did https://bugs.launchpad.net/heat/+bug/1468916 get moved to in progress? I don't see any patch for it... 20:25:52 Launchpad bug 1468916 in heat "stack arn is too big for shorter ceilometer resource_id column" [High,In progress] - Assigned to Kairat Kushaev (kkushaev) 20:27:01 zaneb, https://review.openstack.org/#/c/196123/ 20:27:28 Kairat: thanks, added as a comment 20:28:17 pas-ha: in theory we don't support Icehouse any more, so we can deep six that one 20:29:42 yes, we even have nowhere to merge any potential fix already :) 20:30:13 pas-ha: need to restore new branch ? ;) 20:30:24 s/new/old :) 20:30:35 yay, Infra would be super-happy :) 20:31:02 pas-ha: it never affected master anyway, so I deleted Heat from affects (so it's icehouse only now) 20:31:33 does the branch really go away altogether? 20:31:38 * zaneb did not know that 20:32:13 branch - yes, tags do live 20:32:27 interesting 20:32:34 ok, let's move on 20:32:40 we got one off the list at least :) 20:32:51 #topic Show attribute base patch 20:32:58 #link https://review.openstack.org/#/c/195609/ 20:33:06 skraynev: I assume you added this topic :) 20:33:15 zaneb: right :) 20:33:31 just want to ask people to review it. 20:33:32 you have the floor 20:34:16 need more feedback about two base patches on review :) 20:34:28 added it to the etherpad 20:34:38 zaneb: thx 20:35:49 Unfortunately we can not use some auto-generic method now (it's the reason, why I should add _show resource function to each resource) 20:36:06 saw one thing I can comment on :) 20:36:31 maybe I am wrong, but there are a lot of clients with their custom logic (on show) 20:36:38 does that call the whole idea into question? 20:37:22 zaneb: not sure, I want to upload one cross-project spec with changing all client :) 20:37:37 I just afraid, that it takes more time, then current approach 20:37:54 currently it looks expensive only for us :) 20:37:59 ah right, and then after that we could have a generic method? 20:38:22 zaneb: correct, but IMO it may takes more time. 20:38:42 * zaneb is personally ok with that 20:39:19 the resource plugin interface is a public API we need to maintain backwards compat on 20:39:25 however last comment from Divakar give me a good idea about using get... for most part of resources. 20:39:32 so I'd rather be conservative about what we add to it 20:40:17 zaneb: I think, that common _show_resource method is not bad. 20:40:45 I just will re-write it for some "special" resources :) 20:41:01 "special" :) 20:41:25 zaneb: who don't want to make normal utils in clients :) 20:41:26 ok, shall we move on to the next topic? 20:41:30 yes 20:41:39 #topic batching of large templates 20:41:44 Kairat: go 20:41:45 skraynev, seems you would also have to add a class attribute "entity" like client..get() 20:42:33 So we facing with problems on scale when launching large stacks (without convergence) 20:42:53 large = 60 inst and more 20:43:13 * pas-ha notes - as copy-pasted verbatim into template 20:43:14 pas-ha: make sense (suggest to discuss it tomorrow ;) ) 20:43:48 It turns out that we are ddosing nova (and perhaps other services) and it cannot manage a lot of get requests 20:44:08 which is frightening in itself :D 20:44:17 actually what we found is that stack validation is the most troublesome part - it is synchronous 20:44:22 I am interesting how we can manage such cases in convergence 20:44:39 because heat will be faster 20:44:46 with the new architecture 20:45:00 actually, I think Heat will be slower 20:45:10 pas-ha: i did not check it for master honestly (only for juno) 20:45:14 since validation can make "lots" of API calls to e.g. Glance 20:45:29 pas-ha, cashing can help us 20:45:47 Kairat, pas-ha: with validation constraints - yes 20:46:16 zaneb, even if it will be slower the problem stays 20:46:29 Kairat: agreed 20:46:30 skraynev, it is synchronous in master https://github.com/openstack/heat/blob/master/heat/engine/service.py#L707-L710 20:46:55 convergence phase 1 doesn't really make a big difference either way to this 20:46:57 Kairat, no, slower means more delays between API calls, less ddosing 20:47:16 pas-ha: I though, that we skipped it there (for master)... 20:47:38 pas-ha: but may be I mixed it with some patch on review 20:47:42 pas-ha: that might get from 60 to like 65 20:47:42 no, only for non-reslovable right-now references 20:48:42 I wish we had some way of typing parameters 20:49:10 so, as I was saying, for large stack validation may take too long, and rpc call from api to create simply times out 20:49:13 zaneb: and all of these requests will happen in one moment ? or one by one with some delay ? 20:49:15 so that e.g. once we had validated something as a glance.image it would have that type and always be accepted as such 20:49:43 skraynev: depends on the template 20:49:57 pas-ha: correct, but I suppose (caching should solve it) or you want suggest something else ? :) 20:50:29 pas-ha: that's interesting. ryansb - that's probably why we had to increase the timeout in TripleO 20:50:36 btw, in resource-group + environment, how many times the e.g image parameter of the inner template will get validated? 20:50:39 zaneb: in worst case - 100+ OS::Nova::Servers without resource group 20:50:50 pas-ha, we don't know if nova will work when creating stack and polling nova 20:51:20 pas-ha, IMO, even if validation will pass we might face with some problems when polling 20:51:23 skraynev: I was saying that any slow-down in Heat from convergence would have a very marginal impact on the rate of requests to Nova 20:51:44 zaneb, understood 20:52:16 so one thing we have talked about is having batching on create (not just update) for scaling groups 20:52:25 zaneb: hm. so potential issue will be still there :) 20:52:33 though obviously that would only help here if y'all were using a scaling group 20:52:51 we have some specs for resource group 20:53:03 * batching for resource group 20:53:03 IMO everyone creating 100+vms should use at lease resource group :) 20:53:22 Kairat: ramishra is working on a patch for that I know 20:54:09 pas-ha: agree, however I am not sure, that we can forbid use copy-pasted code :( 20:54:21 Kairat: https://review.openstack.org/#/c/194052/15 20:54:57 skraynev: we can, by timing out before it finished validation ;) 20:55:01 zaneb: isn't this just refactoring? 20:55:39 chuprykov: no. I think the refactoring comes later in the series 20:56:23 pas-ha, i am wondering if many users will create stacks in parallel 20:56:25 zaneb: i have spec on batching https://review.openstack.org/#/c/183506/ 20:56:43 zaneb: yes :) small timeout makes deploying huge stack painful :) 20:56:51 Kairat, that's also surely true 20:58:00 zaneb: dont seems that ramishra's patch add batching on create 20:58:11 not on create, no 20:58:35 that would be a separate change required for both AutoscalingGroup and ResourceGroup 20:58:40 just usual rolling_update 20:58:57 So it seems the problem is still here and it is quite complex) 20:59:02 Kairat: we have 2 minutes. did you have another topic? 20:59:09 1 minute 20:59:12 not so short meeting after all :) 20:59:19 Nope 20:59:28 That's all from me 20:59:30 pas-ha: we break expectations :) 20:59:42 ok, let's wrap up and decamp to #heat 20:59:48 thanks everyone! 20:59:51 Thanks 20:59:58 #endmeeting