15:00:17 #startmeeting heat 15:00:18 Meeting started Wed Nov 23 15:00:17 2016 UTC and is due to finish in 60 minutes. The chair is ramishra. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:00:19 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:00:21 The meeting name has been set to 'heat' 15:00:30 #topic roll call 15:01:12 0/ 15:01:45 Yop 15:02:22 o/ 15:02:38 hmm.. US guys probably are in holiday mode already:) 15:02:43 o/ 15:03:10 holiday is so close :-) 15:03:48 ok, let's start, it should be quick one 15:03:53 #topic adding items to agenda 15:04:08 https://wiki.openstack.org/wiki/Meetings/HeatAgenda#Agenda_.282016-11-23_1500_UTC.29 15:04:15 #link https://wiki.openstack.org/wiki/Meetings/HeatAgenda#Agenda_.282016-11-23_1500_UTC.29 15:05:01 Any more items, we only have one bp to discuss I think. 15:05:36 #topic resource type versions 15:05:49 #link https://blueprints.launchpad.net/heat/+spec/heat-resource-type-versions 15:06:04 syjulian: you've added this, right? 15:06:11 yes 15:06:18 o/ 15:06:31 hi zaneb 15:06:45 we discussed that during the summit. 15:06:51 it looks ok to me. 15:06:54 * zaneb blames daylight savings 15:07:01 lol 15:07:22 yup. we're kinda wondering how registering would work 15:07:22 zaneb: you were not present during that discussion, wdyt? 15:08:14 #link https://review.openstack.org/398601 15:08:47 reading now... seems reasonable 15:09:10 I think it might be a good start to support micro version 15:09:21 syjulian, I feel that the problem description is really lacking 15:09:34 It basically says "we don't support version, let's do it" 15:09:55 I want to know why we need resouce versions 15:10:37 what's the advantage over the resource itself just having an arbitrary my_version property and the resource just knows what to do internally to itself? 15:12:09 cwolferh: I think current approch is more like allow V1 and V2 share same resource type 15:12:17 therve: we do lot of hacks err.. translations to be backward compatible. 15:12:19 cwolferh, That's one solution not mentioned too, yeah. But you need to keep the property schema compatible if you do that 15:12:38 therve: it sould probably mention that. 15:12:44 ramishra, Well yeah and that won't solve that? 15:12:44 therve, ah, true 15:13:19 ramishra, Unless we somehow start to enforce versions, you can't break compatibility 15:13:29 therve: yeah 15:14:30 May be should review and put these comments on the spec, however, it would be good to add this. 15:14:40 Yeah I'll do that 15:15:52 yes that will help 15:16:07 syjulian: I think we would review and provide the comments. 15:16:08 syjulian, We're also here to discuss it though :) 15:16:13 just added some thoughts on the spec 15:16:17 Anyway 15:16:18 gotcha 15:16:46 we decided on versions so when a resource has a new version we won't have to register it in a new name 15:17:13 it's way to get attention of the reviewers, by adding it to the meeting agenda:) 15:17:36 it does help :) . thank you guys for looking at it 15:18:48 should we move on now? 15:18:58 #topic gate issues 15:19:29 Urg it's bad 15:19:32 I just thought to give an update on the issues we had in the last few days. 15:19:57 Number of things broken after the jobs changed to xenial 15:19:58 heatclient 1.6 current been add to black list 15:20:32 ricolin: yeah, it had a backward incompatible change, my mistake 15:20:48 https://review.openstack.org/#/c/401096/ 15:21:08 we have a new release now and it should merge anytime. 15:21:21 I mean the global requirement change. 15:21:34 Things seems to be in control now. 15:22:09 therve: most of jobs are passing since a few hours;) 15:23:01 ramishra, There are 50% slower for some reasons though 15:23:16 if someone can help land this https://review.openstack.org/#/c/400689/ stable/newton would be fixed. 15:24:31 +2'd 15:24:52 therve: you can self approve;) 15:25:19 ramishra, I don't have stable permissions for some reasons 15:25:40 skraynev: ^^ 15:25:44 really? 15:25:51 * zaneb will look in to that 15:26:23 ramishra: do you not have stable +A permissions either? 15:26:24 I think Zane can just +workflow and let it land 15:26:41 therve: it seems issues with some clouds, some of the jobs are finishing in 30-35 mins. 15:27:13 therve: like this one finished quick https://review.openstack.org/#/c/400961/ 15:27:28 ramishra, Okay we'll see :) 15:28:00 I thought https://review.openstack.org/#/admin/groups/152,members was the only group needed. 15:28:06 * zaneb goes looking for more 15:28:23 oh ho https://review.openstack.org/#/admin/groups/536,members 15:28:43 I cant edit though :/ 15:29:41 ok, let's move on. 15:29:50 #topic placeholder 15:29:59 #link https://blueprints.launchpad.net/heat/+spec/implement-placeholder 15:30:19 ricolin: stage is yours;) 15:30:23 Need some review on that BP 15:31:10 yeah, I need to find some time to have a look 15:31:36 ricolin: ok just reviews:) 15:31:43 yep 15:32:01 ricolin: there isn't a spec for this, right? 15:32:02 Maybe we can get it done before Pika 15:32:24 btw, there are number of bps for review, we need to see what we can do in this cycle. 15:32:25 we have a spec 15:32:39 we don't have much time though and the holidays. 15:32:46 https://review.openstack.org/#/c/392499/ 15:33:00 ah! 15:33:13 ok, linked it from the bp 15:33:20 ramishra: It's a holiday week for us! 15:33:26 I can review that first, it'll be quicker :) 15:33:28 I mean U.S. 15:34:09 zaneb: yah! don't want you guys go for those 15 patches first 15:34:41 Request to review the specs and then see if/what we can do this cycle. 15:35:02 ok, will do 15:35:05 ramishra: +1 15:36:32 ramishra: btw I have another topic once we have finished the agenda 15:36:47 we have translation, property, template migrate group and more;) 15:36:54 property group 15:37:07 zaneb: sure 15:37:33 ricolin: the next topic is also spec review request I think? 15:37:43 #topic ironic resource 15:38:16 ramishra: kind of, but I would like to know what guys think about it 15:38:48 ricolin: I thought we had agreed to add the ironic resources, no? 15:38:54 zaneb, therve ? 15:38:55 remind me why we're doing this again? 15:39:33 most people will never see it, since it's admin-only, so no big deal... 15:39:59 zaneb: ironic resource should seperate to manage resource and deploy resource 15:39:59 just not sure that it's actually a useful thing 15:40:22 zaneb: I think there was an earlier discussion about it, not able to locate it. 15:40:25 and manage resource will be a good item to add in to heat 15:40:54 ricolin, Are you going to use this? 15:40:55 ok, so it's not actually about going around nova for allocating servers, just about managing the stuff that Ironic knows about 15:40:58 ? 15:41:28 zaneb: Yep 15:41:30 zaneb: yeah, that was also my impression without reading the spec 15:41:59 ok, sounds harmless enough 15:42:00 zaneb: I add all ironic resource in spec 15:42:27 ok, sounds good. 15:42:39 zaneb: would like to let nova do it job and use ironic resource in heat to cover the rest 15:42:56 I'm not actually convinced that any of our admin-only resources are really useful but they're mostly harmless as long as they're hidden 15:43:17 (from non-admins) 15:43:44 yeah, I've seen keystone resources used a lot though. 15:43:54 I see number of bugs reported. 15:43:56 therve: It will be a greate thing if we can use heat to control baremetral 15:44:11 ricolin, That's not my question though :) 15:44:45 ricolin: I'm more thinking about Ironic+magnum case can be done just in heat 15:45:26 ramishra: keystone resources would be _really_ useful if only you didn't have to be an admin to use them :/ 15:46:47 (got to go, bbl) 15:47:03 ok, should we move on, we've few more mins. 15:47:11 #topic open discussion 15:47:33 therve: np, thanks:) 15:47:46 zaneb: you wanted to discuss something? 15:47:48 zaneb: aren't you have some topic? 15:48:03 ramishra: ah, yes. backpressure. 15:48:40 specifically, how, in convergence, do we limit the amount of work that we try to do at one time 15:48:55 so that we don't e.g. run out of DB connections 15:49:48 zaneb: I don't have a clue, you know better:) 15:49:54 right now we do it by limiting the thread pool size... 15:50:25 but as we've found this week that's apparently not an ideal solution 15:50:57 small limit -> everything is slow 15:51:06 large limit -> run out of db connections 15:51:13 it's not an ideal solution in the db connection case. but are there others? 15:51:17 zaneb: Isn't it deployment specific 15:51:19 ? 15:51:42 I mean someone can increase max_connections if they want? 15:51:57 if the concurrency is high. 15:52:06 I guess 15:52:13 Though I may be missing something. 15:52:36 we either need to publish the correct tuning parameters for different sized deployments or make this Just Work imho 15:53:19 zaneb: I saw this patch from therve https://review.openstack.org/#/c/400155/ 15:53:39 the ideal thing would be to enumerate our bottlenecks (like db connections) and apply backpressure, using the message queue as a buffer 15:54:32 iow the exact opposite of therve's slow-start patch ;) 15:54:48 I don't have a solution here 15:54:57 zaneb: yeah:) 15:55:14 just wanted to socialise the idea a little bit 15:55:28 zaneb: Would it be good to start a ML thread for it? 15:55:46 it probably would 15:56:13 therve: can you give a quick update on exactly what we found with the gate and how we fixed it? 15:56:38 zaneb: I'm not sure if therve is still around. 15:56:52 oh, missed that he dropped out 15:56:58 zaneb: we did not know the exact root cause. 15:57:05 ramishra: can *you* give a quick update? ;) 15:57:09 it started when we moved to xenial 15:57:19 but it seems it's fine now. 15:58:00 We changed the executor_thread_pool_size to 8 from default 64 15:58:05 in master 15:58:29 that's actually used as the max_overflow of connections in oslo.db 15:58:44 max_overflow is 40 by default 15:59:02 what does overflow mean in this context? 15:59:22 we have a pool_size of 5 (default) 16:00:01 so if more connections are needed from the pool, it can overflow to the overflow limit 16:00:19 so the max connections from the pool that can be used is 5+40=45 16:00:29 * zaneb scratches head 16:00:45 it sounds like we're saying the pool_size is meaningless? 16:01:19 zaneb: http://docs.openstack.org/developer/oslo.db/api/options.html 16:01:24 << taps into meeting doors >> 16:01:39 time's up 16:01:40 I think we've to end. 16:01:50 ok 16:01:58 thanks all 16:02:02 thank you guys!:) sorry, but I sense our meeting will be packed too 16:02:03 #endmeeting heat