15:00:27 <ramishra> #startmeeting heat
15:00:29 <openstack> Meeting started Wed Sep 28 15:00:27 2016 UTC and is due to finish in 60 minutes.  The chair is ramishra. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:30 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:32 <openstack> The meeting name has been set to 'heat'
15:00:42 <ramishra> #topic roll call
15:01:39 <zaneb> o/
15:01:49 <skraynev> o/
15:01:51 <spzala> o/
15:01:59 <cwolferh> \o
15:02:53 <therve> Hey
15:03:30 <ricolin> 0/
15:03:36 <Drago> o/
15:03:55 <ramishra> #topic adding items to agenda
15:04:06 <ramishra> #link https://wiki.openstack.org/wiki/Meetings/HeatAgenda#Agenda_.282016-09-28_1500_UTC.29
15:05:10 <ramishra> #topic stable:follows-policy tag
15:05:22 <ramishra> #link http://governance.openstack.org/reference/tags/stable_follows-policy.html
15:05:34 <ramishra> not sure who has added this.
15:05:54 <zaneb> me
15:05:54 <ramishra> I assume a patch to governance repo should do it, no?
15:06:01 <zaneb> yes
15:06:08 <ramishra> ok, I'll do that
15:06:26 <zaneb> it will help if we have a link to a meeting decision that this is definitely what we want
15:06:40 <zaneb> I don't think it's controversial though, right?
15:07:10 <therve> Just confirming what we do already, you mean? :)
15:07:21 <zaneb> therve: yeah
15:07:35 <therve> +1
15:08:05 <zaneb> I guess there's a theoretical difference between "following it" and "agreeing to be bound by it"
15:08:37 <ramishra> yeah, we are agreeing to do it from now on:)
15:08:46 <zaneb> I'd also like to know why we have not been rated on "Existence and quality of packages for this project in popular distributions."
15:09:14 <ramishra> zaneb: who can answer that question, tc?
15:09:24 <zaneb> no, it's the ops group
15:09:51 <zaneb> I'd also love to know exactly how many SDKs have to have support before you get a maturity tick there
15:09:58 <zaneb> because honestly 7 feels like a lot
15:10:41 <ramishra> ok, so we can drop a mail to them with the question?
15:10:56 <zaneb> might be worth a try
15:11:11 <ramishra> ok
15:11:13 <zaneb> make sure they give us a fair shake for newton
15:11:38 <zaneb> (we can move on :)
15:11:52 <ramishra> yep
15:11:58 <ramishra> #topic heat summit sessions
15:12:06 <therve> zaneb, https://www.openstack.org/software/releases/mitaka/components/glance 8 is enough apparently :)
15:12:26 <ramishra> I've arranged the topics now on the etherpad
15:12:36 <ramishra> we need more topics
15:12:48 <zaneb> therve: so close!
15:12:53 <ramishra> #link https://etherpad.openstack.org/p/ocata-heat-sessions
15:13:25 <ramishra> therve: I saw your forwarded mail and I also got a mail from magnum PTL yesterday.
15:13:54 <ramishra> They want a joint session to discuss about some of the heat issues they are facing wrt heat scalability
15:14:30 <ramishra> we have one session on performance and scalability, probably add another one?
15:14:35 <therve> Yeah, we should do the usual "large stacks" session
15:15:46 <ramishra> anyway, we don't have enough sessions yet, so I'll add this one too.
15:16:34 <ramishra> btw, we probably have add a little more to the descriptions of the topics:)
15:17:42 <ramishra> anyone has more topics, please propose them. We still have 3 more slots available.
15:19:02 <ramishra> ok, moving on..
15:19:10 <ramishra> #topic gate failures
15:19:29 <ramishra> We are seeing large number of gate failures these few days.
15:19:57 <ramishra> it seems mostly due to some external network connectivity issues.
15:20:22 <ramishra> so we were thinking of disabling some of the tests that use fedora image? any thoughts?
15:21:21 <zaneb> no thoughts, but just a reminder to please tag gate failure bugs with 'gate-failure' so they end up in https://bugs.launchpad.net/heat/+bugs?field.tag=gate-failure
15:22:04 <ramishra> zaneb: sure
15:22:26 <therve> ramishra, There are many tests using image_ref
15:22:29 <therve> More than I thought
15:22:42 <therve> We should switch some of them to minimal first
15:23:24 <ramishra> therve: yeah, that why I was worried to disable all of them.
15:23:53 <therve> ramishra, I don't think most of them need it
15:25:08 <ramishra> zaneb: I've added gate-failure tags to few of them just now.
15:26:32 <ramishra> therve: yeah, probably we can change the tests where it's not needed first.
15:26:49 <ramishra> and then look at disabling the ones left.
15:27:39 <ramishra> I was thinking of waiting for a response from the infra team before doing any test disable.
15:27:55 <therve> We got one, not really encouraging :)
15:28:05 <therve> http://lists.openstack.org/pipermail/openstack-infra/2016-September/004750.html
15:28:28 <therve> Also <fungi> therve: sheer speculation, but if it's ipv4 then maybe we're overloading their pat (since that environment only has ipv6 for direct global connectivity)
15:29:10 <fungi> yeah, we recently increased instance count significantly in that environment
15:29:31 <fungi> so it's possible we've put strain on their networks, and the nat they're using to provide ipv4 egress is a likely weak spot
15:30:10 <therve> fungi, Should we try ipv6 mirror?
15:30:26 <fungi> therve: do they have one for the fedora images?
15:30:34 <therve> No idea
15:30:42 <zaneb> fungi: it seems like sdague's suggestion of mirroring that image locally is a good one, though maybe not a short-term fix. wdyt?
15:31:35 <fungi> zaneb: we agreed during yesterday's infra meeting (in therve's absence) that it makes sense we just need whoever's going to implement the mirroring for arbitrary blobs to write up a very brief infra spec so we can work out the modekl
15:31:38 <fungi> model
15:32:04 <zaneb> fungi++
15:32:10 <fungi> #link http://eavesdrop.openstack.org/meetings/infra/2016/infra.2016-09-27-19.02.log.html#l-119
15:33:03 <fungi> but in the near term, we're working with the operators for osic to see if we can figure out what's happening and then we'll either reduce our utilization there or they'll beef up whatever's overloaded
15:33:20 <fungi> there's been ongoing discussion today in #openstack-infra about it
15:34:52 <therve> http://mirror.speedpartner.de/ seems to be ipv6, it's worth trying
15:35:02 <cloudnull> o/ if there's something i can help out with regarding OSIC issues please let me know.
15:35:20 <fungi> cloudnull: you already have been
15:35:56 <cloudnull> I was just lurking and figured i'd say hi.
15:37:27 <ramishra> fungi, cloudnull, thanks for joining the discussion
15:38:12 <ramishra> therve: will you propose a patch to use the specific mirror?
15:38:23 <therve> ramishra, Yep let me do that
15:38:37 <ramishra> therve: thanks!
15:39:45 <ramishra> #topic open discussion
15:40:06 <ramishra> anything else we want to discuss?
15:41:23 <therve> If anybody wants to look at memory consumption that'd be rad
15:42:29 <zaneb> yeah, I'm looking but not making much progress
15:44:16 <ramishra> do you have an understanding now, when memory consumption started moving up again for tripleo?
15:45:17 <ramishra> from some of the discussions it looked to coincide with the days when conditions patches landed, though we can infer anything from that.
15:45:35 <zaneb> I'm seeing a definite increase between http://logs.openstack.org/58/351358/1/check-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/54cf4ea/logs/ and http://logs.openstack.org/40/359940/2/check-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/f3c1ef5/logs/ which were both on 2016-08-25 and appear to use the same Heat build
15:46:10 <zaneb> so my suspicion is that some change to the templates may have triggered it
15:49:31 <ramishra> yeah, we would need help from the tripleo team to resolve it. shardy does not seem to be online.
15:50:50 <ramishra> if there is nothing else, we can move back to #heat
15:51:43 <ramishra> Thanks all for joining
15:51:55 <ramishra> #endmeeting heat
15:52:09 <ramishra> #endmeeting heat