| *** tochi has quit IRC | 00:02 | |
| *** ayoung has quit IRC | 00:02 | |
| *** asalkeld has joined #heat | 00:07 | |
| *** zhurong has joined #heat | 00:10 | |
| *** catintheroof has joined #heat | 00:11 | |
| *** asalkeld has quit IRC | 00:12 | |
| *** flwang1 has quit IRC | 00:13 | |
| *** flwang1 has joined #heat | 00:14 | |
| *** chris_hultin is now known as chris_hultin|AWA | 00:15 | |
| *** EricGonczer_ has joined #heat | 00:18 | |
| *** zhurong has quit IRC | 00:24 | |
| *** spotz is now known as spotz_zzz | 00:26 | |
| *** asalkeld has joined #heat | 00:27 | |
| *** asalkeld has quit IRC | 00:32 | |
| *** tochi has joined #heat | 00:35 | |
| *** spzala has quit IRC | 00:39 | |
| *** rajpatel has quit IRC | 00:39 | |
| *** Zer0Byte__ has quit IRC | 00:43 | |
| *** ayoung has joined #heat | 00:50 | |
| *** hoangcx has joined #heat | 00:54 | |
| *** catintheroof has quit IRC | 00:54 | |
| *** tkarke has quit IRC | 01:02 | |
| *** tkarke has joined #heat | 01:05 | |
| *** ayoung has quit IRC | 01:06 | |
| *** tkarke has quit IRC | 01:06 | |
| *** tkarke has joined #heat | 01:07 | |
| *** tkarke has quit IRC | 01:08 | |
| *** chlong has quit IRC | 01:11 | |
| *** yanyanhu_ has joined #heat | 01:21 | |
| *** zhurong has joined #heat | 01:36 | |
| *** chetnakh has quit IRC | 01:54 | |
| *** chetnakh has joined #heat | 01:55 | |
| *** elynn has joined #heat | 01:59 | |
| *** zhihui has joined #heat | 01:59 | |
| *** zhihui has quit IRC | 02:00 | |
| *** annegentle has quit IRC | 02:01 | |
| *** elynn_ has joined #heat | 02:05 | |
| *** elynn_ has quit IRC | 02:07 | |
| *** elynn has quit IRC | 02:08 | |
| *** elynn has joined #heat | 02:11 | |
| *** elynn_ has joined #heat | 02:13 | |
| *** EricGonczer_ has quit IRC | 02:17 | |
| *** EricGonc_ has joined #heat | 02:17 | |
| *** elynn has quit IRC | 02:17 | |
| *** adrian_otto has quit IRC | 02:19 | |
| *** tkarke has joined #heat | 02:29 | |
| *** tkarke has quit IRC | 02:30 | |
| *** spzala has joined #heat | 02:39 | |
| *** spzala has quit IRC | 02:44 | |
| *** jdandrea has joined #heat | 02:57 | |
| *** EricGonc_ has quit IRC | 02:58 | |
| *** fragatina has quit IRC | 03:03 | |
| *** fragatina has joined #heat | 03:05 | |
| *** fragatin_ has joined #heat | 03:09 | |
| *** fragatina has quit IRC | 03:09 | |
| *** zhurong_ has joined #heat | 03:10 | |
| *** zhurong has quit IRC | 03:12 | |
| *** steveg_afk has quit IRC | 03:14 | |
| *** fragatin_ has quit IRC | 03:14 | |
| *** jdandrea has quit IRC | 03:31 | |
| *** elynn_ has quit IRC | 03:31 | |
| *** jdandrea has joined #heat | 03:31 | |
| *** wangbo has quit IRC | 03:34 | |
| *** vijendar has joined #heat | 03:38 | |
| *** vijendar1 has joined #heat | 03:38 | |
| *** vijendar1 has quit IRC | 03:38 | |
| *** bks has joined #heat | 03:39 | |
| *** zhurong_ has quit IRC | 03:42 | |
| *** zhurong has joined #heat | 03:43 | |
| *** vijendar has quit IRC | 03:44 | |
| *** chetnakh has quit IRC | 03:46 | |
| *** chetnakh has joined #heat | 03:48 | |
| *** wangbo has joined #heat | 03:51 | |
| ramishra | We've started seeing failures like http://logs.openstack.org/77/397377/1/check/gate-heat-docs-ubuntu-xenial/1b45056/console.html#_2016-11-17_23_35_34_314028, that too in stable branches:/ | 03:58 |
|---|---|---|
| ramishra | we should get these patches reviewed soon https://review.openstack.org/#/q/project:openstack/heat+branch:master+topic:remove_apiclient | 03:59 |
| *** tkarke has joined #heat | 04:01 | |
| *** tkarke has quit IRC | 04:02 | |
| *** flwang1 has quit IRC | 04:04 | |
| *** ctayal has joined #heat | 04:14 | |
| *** ctayal has quit IRC | 04:20 | |
| *** bana_k has joined #heat | 04:26 | |
| *** jdandrea has quit IRC | 04:27 | |
| *** misty-cloud has joined #heat | 04:32 | |
| *** adrian_otto has joined #heat | 04:34 | |
| *** jdandrea has joined #heat | 04:36 | |
| *** jdandrea has quit IRC | 04:40 | |
| *** jdandrea has joined #heat | 04:41 | |
| ramishra | stevebaker: hi around? | 04:43 |
| *** ayoung has joined #heat | 04:44 | |
| *** vishwanathj has joined #heat | 05:05 | |
| *** bana_k has quit IRC | 05:26 | |
| *** bana_k has joined #heat | 05:27 | |
| *** dimtruck is now known as zz_dimtruck | 05:30 | |
| *** spzala has joined #heat | 05:40 | |
| *** spzala has quit IRC | 05:44 | |
| *** elynn_ has joined #heat | 05:54 | |
| *** elynn_ has quit IRC | 06:01 | |
| *** elynn_ has joined #heat | 06:01 | |
| openstackgerrit | Rabi Mishra proposed openstack/heat: Remove oslo-incubator use in other client plugins https://review.openstack.org/394736 | 06:12 |
| *** adrian_otto has quit IRC | 06:18 | |
| *** chetnakh has quit IRC | 06:20 | |
| *** sabeen has quit IRC | 06:35 | |
| *** jamielennox is now known as jamielennox|away | 06:36 | |
| *** chetnakh has joined #heat | 06:36 | |
| *** chetnakh has quit IRC | 06:37 | |
| ramishra | we seem to have more issues after the switch to xenial. Package(openswan) for vpnaas not available on it. | 06:46 |
| ramishra | I think we don't need vpnaas at the gate. Also it's out of neutron governance now. I've pushed a change https://review.openstack.org/#/c/399390/ | 06:47 |
| ramishra | therve, skraynev ^^^ | 06:47 |
| ramishra | we also have another issue where gate-heat-docs-ubuntu-xenial is failing, https://review.openstack.org/#/c/394736/ will fix that. | 06:51 |
| *** sdake has joined #heat | 07:31 | |
| *** sdake_ has joined #heat | 07:32 | |
| *** sdake has quit IRC | 07:36 | |
| *** chetnakh has joined #heat | 07:37 | |
| *** chetnakh has quit IRC | 07:38 | |
| *** jaosorior has joined #heat | 07:39 | |
| *** vijendar has joined #heat | 07:41 | |
| *** vijendar1 has joined #heat | 07:41 | |
| *** vijendar1 has quit IRC | 07:41 | |
| skraynev | ramishra: Also we probably need to remove the same code from our repo here: https://github.com/openstack/heat/blob/master/heat_integrationtests/pre_test_hook.sh#L44 | 07:46 |
| ramishra | skraynev: there is already a patch https://review.openstack.org/#/c/372222/ i had put long time ago | 07:47 |
| *** vijendar has quit IRC | 07:47 | |
| ramishra | skraynev: no one had reviewed it;) | 07:47 |
| therve | ramishra, So, which one do we need to land, in which order? | 07:48 |
| therve | Because I wonder if we may not have to squash some of them | 07:48 |
| therve | Ah we need the infra change first | 07:49 |
| ramishra | therve: yeah | 07:49 |
| skraynev | ramishra: ok. good :) thank you | 07:50 |
| ramishra | therve: yeah, we have to squash the two heat patches, I'll do that | 07:52 |
| therve | ramishra, Do we? I thought https://review.openstack.org/394736 was enough | 07:52 |
| therve | Ah it's possible the local.conf change is necessary too? | 07:53 |
| ramishra | I think we have to squash them, we can check that after the infra change lands. | 07:54 |
| therve | ok | 07:55 |
| *** jamielennox|away is now known as jamielennox | 08:03 | |
| *** openstackgerrit has quit IRC | 08:03 | |
| *** vishwana_ has joined #heat | 08:03 | |
| *** openstackgerrit has joined #heat | 08:04 | |
| *** vishwanathj has quit IRC | 08:05 | |
| *** flwang1 has joined #heat | 08:07 | |
| *** amoralej|off is now known as amoralej | 08:12 | |
| *** jprovazn has joined #heat | 08:28 | |
| *** tochi has quit IRC | 08:34 | |
| *** tkarke has joined #heat | 08:40 | |
| *** tkarke has quit IRC | 08:42 | |
| *** lifeless has quit IRC | 08:45 | |
| *** ishant|away has quit IRC | 08:46 | |
| *** lifeless has joined #heat | 08:47 | |
| *** kirankv has quit IRC | 08:49 | |
| ramishra | therve: it seems to be still using the vpnaas plugin, http://logs.openstack.org/36/394736/3/check/gate-heat-dsvm-functional-orig-mysql-lbaasv2-ubuntu-xenial/aab45ad/console.html#_2016-11-18_08_21_25_646104 | 08:50 |
| ramishra | therve: Wondering why the project-config change did not take effect. | 08:50 |
| *** bana_k has quit IRC | 08:50 | |
| therve | ramishra, Did you wait for a zuul restart? | 08:51 |
| therve | It takes some time for changes to propagate | 08:51 |
| ramishra | therve: ok, how long does it take? | 08:51 |
| *** ishant|away has joined #heat | 08:51 | |
| *** kirankv has joined #heat | 08:51 | |
| therve | ramishra, Depends I think. Updated time is at the bottom of http://status.openstack.org/zuul/ | 08:52 |
| ramishra | therve: oh, ok | 08:52 |
| therve | 10 minutes ago, so the changes ought to be in now | 08:52 |
| ramishra | ok, we can try one more time:) | 08:53 |
| *** shardy has joined #heat | 08:53 | |
| *** jpich has joined #heat | 08:55 | |
| *** chetnakh has joined #heat | 08:55 | |
| *** chetnakh has quit IRC | 08:56 | |
| openstackgerrit | shizhihui proposed openstack/python-heatclient: Add optional arguments in this osc cli: openstack stack snapshot delete https://review.openstack.org/399450 | 08:58 |
| *** hoangcx has quit IRC | 08:59 | |
| *** zzzeek has quit IRC | 09:00 | |
| *** zzzeek has joined #heat | 09:00 | |
| *** zhurong has quit IRC | 09:01 | |
| *** sdake_ is now known as sdake | 09:01 | |
| *** zhurong has joined #heat | 09:02 | |
| *** zz_dimtruck is now known as dimtruck | 09:08 | |
| *** derekh has joined #heat | 09:14 | |
| *** strigazi_AFK is now known as strigazi | 09:15 | |
| *** elynn_ has quit IRC | 09:17 | |
| openstackgerrit | shizhihui proposed openstack/python-heatclient: Add optional arguments in this osc cli: openstack stack snapshot delete https://review.openstack.org/399450 | 09:24 |
| *** yassine has joined #heat | 09:26 | |
| *** yassine is now known as Guest17613 | 09:26 | |
| *** shardy has quit IRC | 09:34 | |
| *** shardy has joined #heat | 09:34 | |
| *** chetnakh has joined #heat | 09:37 | |
| *** chetnakh has quit IRC | 09:38 | |
| *** spzala has joined #heat | 09:40 | |
| *** vijendar has joined #heat | 09:44 | |
| *** spzala has quit IRC | 09:45 | |
| *** vijendar has quit IRC | 09:48 | |
| *** derekjhyang has quit IRC | 09:51 | |
| *** prazumovsky has joined #heat | 09:51 | |
| *** AlexeyAbashkin has joined #heat | 09:53 | |
| ramishra | therve: it did move ahead, but it seems more issues. | 09:54 |
| ramishra | http://logs.openstack.org/36/394736/3/check/gate-heat-dsvm-functional-convg-mysql-lbaasv2-ubuntu-xenial/90f5b5b/logs/screen-h-eng.txt.gz?level=ERROR | 09:55 |
| therve | Urgggggg | 09:55 |
| therve | So that's convergence, hum | 09:56 |
| ramishra | yeah, convergence | 09:56 |
| ramishra | but there are haproxy issues too with non-convergence | 09:57 |
| *** jtomasek has joined #heat | 09:57 | |
| ramishra | the lbaasv2 test | 09:57 |
| therve | Let fix that first.... | 09:57 |
| therve | https://bugs.launchpad.net/oslo.rootwrap/+bug/1555874 | 09:58 |
| openstack | Launchpad bug 1555874 in oslo.rootwrap "IpNetnsExecFilter with missing nested executable returns misleading exception" [Undecided,New] | 09:58 |
| therve | haproxy is not installed I guess? | 09:58 |
| *** zhurong has quit IRC | 09:59 | |
| ramishra | yep, that seems like the same issue | 09:59 |
| therve | ramishra, https://github.com/openstack/neutron-lbaas/blob/master/devstack/plugin.sh#L10 :/ | 10:01 |
| ramishra | huh | 10:03 |
| ramishra | we've to go around all projects to fix this mess:/ | 10:04 |
| *** yanyanhu_ has quit IRC | 10:05 | |
| ramishra | therve: are you planning to propose a patch to fix that? | 10:07 |
| therve | ramishra, Yes | 10:08 |
| ramishra | ok, thanks! | 10:08 |
| therve | ramishra, So, convergence failure... | 10:11 |
| therve | It seems to always fail in the same place, which is interesting | 10:12 |
| ramishra | I was looking at it, may be we increase the engine pool size. | 10:15 |
| ramishra | Did not check what's being used atm. | 10:15 |
| *** AlexeyAbashkin has quit IRC | 10:16 | |
| therve | You mean decrease | 10:16 |
| therve | Or increase mysql connection limit | 10:16 |
| ramishra | I think incrase it. | 10:18 |
| openstackgerrit | shizhihui proposed openstack/python-heatclient: Add optional arguments in this osc cli: openstack stack snapshot delete https://review.openstack.org/399450 | 10:19 |
| ramishra | "If you get a Too many connections error when you try to connect to the mysqld server, this means that all available connections are in use by other clients." | 10:19 |
| therve | Well yeah | 10:20 |
| ramishra | I derive it from that statement | 10:20 |
| therve | Sure, but that means we try to connect too much | 10:20 |
| ramishra | yeah, that's the real problem though. | 10:20 |
| therve | If you increase the engine pool size, we'll connect even more | 10:20 |
| therve | It's annoying because we probably hide the real source of the issue | 10:23 |
| *** ishant|away has quit IRC | 10:23 | |
| *** ishant|away has joined #heat | 10:24 | |
| ramishra | yeah, I agree, It seems max_pool_size is the default https://github.com/openstack/oslo.db/blob/master/oslo_db/sqlalchemy/enginefacade.py#L138 | 10:25 |
| ramishra | Atleast that's what I understand. | 10:25 |
| therve | There is overflow though | 10:26 |
| ramishra | Wondering why moving to a new mysql version should bring this issue. Has some defaults changed? | 10:28 |
| *** fragatina has joined #heat | 10:35 | |
| *** fragatina has quit IRC | 10:36 | |
| *** fragatina has joined #heat | 10:37 | |
| *** sdake has quit IRC | 10:48 | |
| *** misty-cloud has quit IRC | 10:59 | |
| *** jaosorior is now known as jaosorior_lunch | 11:04 | |
| *** tkarke has joined #heat | 11:08 | |
| ramishra | therve: max_connections seems to set to 1024 https://github.com/openstack-dev/devstack/blob/master/lib/databases/mysql#L100 | 11:08 |
| *** tkarke has quit IRC | 11:09 | |
| *** sorantis has joined #heat | 11:10 | |
| *** jdandrea_ has joined #heat | 11:12 | |
| *** jtomasek has quit IRC | 11:13 | |
| *** bks has quit IRC | 11:15 | |
| *** jdandrea has quit IRC | 11:15 | |
| *** sdake has joined #heat | 11:16 | |
| openstackgerrit | Rabi Mishra proposed openstack/heat: Remove oslo-incubator use in other client plugins https://review.openstack.org/394736 | 11:30 |
| *** sorantis has quit IRC | 11:38 | |
| *** sorantis has joined #heat | 11:39 | |
| *** jtomasek has joined #heat | 11:41 | |
| *** jaosorior_lunch is now known as jaosorior | 11:44 | |
| *** vijendar has joined #heat | 11:45 | |
| *** sorantis has quit IRC | 11:47 | |
| *** vijendar has quit IRC | 11:49 | |
| *** sorantis has joined #heat | 12:07 | |
| therve | ramishra, It sounds like it should be plenty enough | 12:14 |
| *** sdake_ has joined #heat | 12:16 | |
| *** sdake has quit IRC | 12:20 | |
| *** derekjhyang has joined #heat | 12:21 | |
| *** ccamacho is now known as ccamacho|lunch | 12:25 | |
| *** steveg_afk has joined #heat | 12:26 | |
| ramishra | yeah, we seem to be opening too many connections and closing them, I tried increase that and check with depends_on and there are still some failures, though a lesser number. | 12:38 |
| ramishra | and *not* closing them. | 12:39 |
| *** spzala has joined #heat | 12:40 | |
| ramishra | therve: ^^^ | 12:40 |
| *** yprokule has joined #heat | 12:43 | |
| *** spzala has quit IRC | 12:44 | |
| ramishra | therve: this is similar to bug 1566845 reported by shardy | 12:49 |
| openstack | bug 1566845 in heat "DB errors for large/nested stacks with convergence enabled" [Undecided,New] https://launchpad.net/bugs/1566845 | 12:49 |
| *** EricGonczer_ has joined #heat | 12:51 | |
| ramishra | I've updated the bug with gate logs. | 12:52 |
| ramishra | zaneb: ^^^ | 12:53 |
| *** tkarke has joined #heat | 13:02 | |
| *** tkarke has quit IRC | 13:03 | |
| *** dsneddon is now known as dsneddon_afk | 13:04 | |
| *** lamt has joined #heat | 13:16 | |
| *** pradk has joined #heat | 13:24 | |
| *** sdake has joined #heat | 13:36 | |
| *** ccamacho|lunch is now known as ccamacho | 13:36 | |
| *** sdake_ has quit IRC | 13:38 | |
| *** sdake_ has joined #heat | 13:40 | |
| *** spzala has joined #heat | 13:41 | |
| *** jaosorior has quit IRC | 13:42 | |
| *** amoralej is now known as amoralej|lunch | 13:42 | |
| *** sdake has quit IRC | 13:43 | |
| *** dmellado is now known as dmellado|lunch | 13:50 | |
| *** sabeen has joined #heat | 13:51 | |
| openstackgerrit | Peter Razumovsky proposed openstack/heat: Move call trove cluster delete to check complete https://review.openstack.org/390828 | 14:03 |
| *** vijendar has joined #heat | 14:06 | |
| *** ctayal has joined #heat | 14:06 | |
| *** vijendar has quit IRC | 14:06 | |
| *** ctayal has quit IRC | 14:07 | |
| *** vijendar has joined #heat | 14:07 | |
| *** vijendar1 has joined #heat | 14:07 | |
| *** ctayal has joined #heat | 14:07 | |
| *** vijendar1 has quit IRC | 14:12 | |
| *** vijendar has quit IRC | 14:12 | |
| *** dmellado|lunch is now known as dmellado | 14:12 | |
| *** zhurong has joined #heat | 14:18 | |
| *** misty-cloud has joined #heat | 14:18 | |
| *** liusheng has quit IRC | 14:20 | |
| *** misty-cloud has quit IRC | 14:21 | |
| *** liusheng has joined #heat | 14:21 | |
| *** ccamacho has quit IRC | 14:29 | |
| *** zhurong has quit IRC | 14:30 | |
| *** ccamacho has joined #heat | 14:30 | |
| *** derekjhyang has quit IRC | 14:31 | |
| *** zhurong has joined #heat | 14:35 | |
| *** misty-cloud has joined #heat | 14:39 | |
| *** sdake has joined #heat | 14:39 | |
| *** amoralej|lunch is now known as amoralej | 14:42 | |
| *** bnemec has joined #heat | 14:42 | |
| *** spzala has quit IRC | 14:42 | |
| *** spzala has joined #heat | 14:43 | |
| *** sdake_ has quit IRC | 14:43 | |
| *** lamt has quit IRC | 14:46 | |
| therve | ramishra, So I tested, and I reached 100 connections with just 2 engines in convergence | 14:46 |
| therve | We should set the overflow a bit lower I guess? | 14:46 |
| *** spzala has quit IRC | 14:48 | |
| *** bnemec is now known as beekneemech-semi | 14:48 | |
| *** beekneemech-semi is now known as bnemec-semi-here | 14:49 | |
| *** lamt has joined #heat | 14:51 | |
| *** rook is now known as rook-baby | 14:51 | |
| *** chlong has joined #heat | 14:53 | |
| *** zhurong has quit IRC | 14:55 | |
| *** adrian_otto has joined #heat | 14:57 | |
| ramishra | therve: Are you suggesting that increased number of engines have resulted in the issue? | 15:06 |
| *** chris_hultin|AWA is now known as chris_hultin | 15:06 | |
| ramishra | I think the overflow default is 10 | 15:06 |
| therve | ramishra, It's 50 | 15:10 |
| ramishra | therve: yeah | 15:10 |
| therve | ramishra, Why do you see number of engines increased? | 15:10 |
| therve | s/see/say | 15:11 |
| therve | Do we have bigger nodes? | 15:11 |
| ramishra | I'm thinking may be more cpu nodes for xenial;) | 15:11 |
| ramishra | don't know. trying to understand what's changed for the issue to manifest. | 15:12 |
| *** vijendar has joined #heat | 15:12 | |
| therve | No looks like it was 8 before as well | 15:13 |
| therve | Well the main thing that changed is mysql version | 15:13 |
| *** ctayal_ has joined #heat | 15:13 | |
| ramishra | yeah, but the max_connections is the same and the oslo.db/sqlalchemy defaults are the same. | 15:14 |
| *** ctayal has quit IRC | 15:15 | |
| *** vijendar has quit IRC | 15:17 | |
| *** prazumovsky has quit IRC | 15:17 | |
| therve | ramishra, I think we should lower the overflow while we investigate | 15:19 |
| *** jasond has joined #heat | 15:20 | |
| ramishra | therve: +1, we can try that witha test patch. | 15:20 |
| therve | Still waiting for the lbaas fix anyway | 15:21 |
| therve | ramishra, Do you want me to try? Must be late for you | 15:21 |
| *** randallburt has joined #heat | 15:21 | |
| ramishra | therve: yeah, please do it. thanks. | 15:22 |
| *** spotz_zzz is now known as spotz | 15:25 | |
| openstackgerrit | Thomas Herve proposed openstack/heat: Remove oslo-incubator use in other client plugins https://review.openstack.org/394736 | 15:26 |
| *** randallburt has quit IRC | 15:26 | |
| openstackgerrit | Thomas Herve proposed openstack/heat: Remove oslo-incubator use in other client plugins https://review.openstack.org/394736 | 15:26 |
| *** adrian_otto has quit IRC | 15:27 | |
| *** rajpatel has joined #heat | 15:31 | |
| *** randallburt has joined #heat | 15:34 | |
| *** jcoufal has joined #heat | 15:34 | |
| *** rook-baby is now known as rook | 15:37 | |
| *** vijendar has joined #heat | 15:39 | |
| *** vijendar1 has joined #heat | 15:39 | |
| *** sdake has quit IRC | 15:44 | |
| *** spzala has joined #heat | 15:47 | |
| *** pradk has quit IRC | 15:47 | |
| *** sdake has joined #heat | 15:51 | |
| *** misty-cloud has quit IRC | 15:52 | |
| *** pradk has joined #heat | 15:54 | |
| *** adrian_otto has joined #heat | 15:54 | |
| *** ctayal_ has quit IRC | 15:54 | |
| *** ctayal has joined #heat | 15:55 | |
| *** jprovazn has quit IRC | 16:01 | |
| *** EricGonc_ has joined #heat | 16:06 | |
| *** EricGonczer_ has quit IRC | 16:08 | |
| *** annegentle has joined #heat | 16:09 | |
| therve | https://bugs.launchpad.net/heat/+bug/1491185 | 16:09 |
| openstack | Launchpad bug 1491185 in heat "convergence: QueuePool limit of size 5 overflow 10 reached" [Medium,Fix released] - Assigned to Anant Patil (ananta) | 16:09 |
| therve | :/ | 16:09 |
| *** rajpatel has quit IRC | 16:11 | |
| *** bana_k has joined #heat | 16:13 | |
| *** bana_k has quit IRC | 16:18 | |
| *** ramishra has quit IRC | 16:19 | |
| *** bana_k has joined #heat | 16:20 | |
| *** ramishra has joined #heat | 16:21 | |
| openstackgerrit | Thomas Herve proposed openstack/heat: Remove oslo-incubator use in other client plugins https://review.openstack.org/394736 | 16:24 |
| *** adrian_otto has quit IRC | 16:26 | |
| *** Drago1 has joined #heat | 16:32 | |
| *** Drago1 has quit IRC | 16:37 | |
| *** sorantis has quit IRC | 16:43 | |
| *** Drago1 has joined #heat | 16:45 | |
| *** sdake has quit IRC | 16:47 | |
| *** vijendar has quit IRC | 16:57 | |
| *** vijendar1 has quit IRC | 16:57 | |
| *** ccamacho has quit IRC | 16:58 | |
| *** EricGonc_ has quit IRC | 16:59 | |
| *** EricGonczer_ has joined #heat | 16:59 | |
| *** adrian_otto has joined #heat | 17:03 | |
| *** ctayal has quit IRC | 17:05 | |
| *** vijendar has joined #heat | 17:12 | |
| *** vijendar1 has joined #heat | 17:12 | |
| *** tkarke has joined #heat | 17:14 | |
| *** sabeen has quit IRC | 17:16 | |
| *** rajpatel has joined #heat | 17:19 | |
| *** sabeen has joined #heat | 17:24 | |
| *** derekh has quit IRC | 17:24 | |
| *** Zer0Byte__ has joined #heat | 17:25 | |
| *** blomquisg_gone is now known as blomquisg | 17:30 | |
| *** sorantis has joined #heat | 17:41 | |
| *** jpich has quit IRC | 17:53 | |
| *** adrian_otto has quit IRC | 17:58 | |
| *** yprokule has quit IRC | 18:03 | |
| *** dbecker has quit IRC | 18:04 | |
| *** rajpatel has quit IRC | 18:04 | |
| *** ccamacho has joined #heat | 18:11 | |
| *** ccamacho has quit IRC | 18:11 | |
| *** adrian_otto has joined #heat | 18:11 | |
| *** sorantis has quit IRC | 18:15 | |
| *** sorantis has joined #heat | 18:16 | |
| *** vijendar1 has quit IRC | 18:19 | |
| *** vijendar has quit IRC | 18:19 | |
| *** Zer0Byte__ has quit IRC | 18:28 | |
| *** chris_hultin is now known as chris_hultin|AWA | 18:45 | |
| *** chris_hultin|AWA is now known as chris_hultin | 18:47 | |
| *** chris_hultin is now known as chris_hultin|AWA | 18:57 | |
| *** lamt has quit IRC | 18:58 | |
| *** sdake has joined #heat | 19:02 | |
| *** cwolferh has quit IRC | 19:03 | |
| *** cwolferh has joined #heat | 19:07 | |
| therve | Gate is still broken, but with the latest lbaas change we may be good to go | 19:12 |
| therve | https://review.openstack.org/#/c/394736/ to monitor | 19:12 |
| *** vijendar has joined #heat | 19:14 | |
| *** vijendar1 has joined #heat | 19:14 | |
| *** rajpatel has joined #heat | 19:15 | |
| *** sorantis has quit IRC | 19:17 | |
| *** rajpatel has quit IRC | 19:19 | |
| *** chris_hultin|AWA is now known as chris_hultin | 19:19 | |
| *** Zer0Byte__ has joined #heat | 19:19 | |
| *** rajpatel has joined #heat | 19:28 | |
| *** ctayal has joined #heat | 19:29 | |
| zaneb | therve: no dice by the looks of it :/ | 19:31 |
| *** dsneddon_afk is now known as dsneddon | 19:37 | |
| *** bnemec-semi-here has quit IRC | 19:39 | |
| *** vijendar1 has quit IRC | 19:39 | |
| *** vijendar has quit IRC | 19:39 | |
| *** chris_hultin is now known as chris_hultin|AWA | 19:47 | |
| *** chris_hultin|AWA is now known as chris_hultin | 19:49 | |
| *** amoralej is now known as amoralej|off | 19:51 | |
| *** Guest17613 has quit IRC | 19:53 | |
| *** ctayal has quit IRC | 20:15 | |
| *** ctayal has joined #heat | 20:16 | |
| therve | zaneb, Nova failure is https://review.openstack.org/#/c/399155/ | 20:27 |
| *** cwolferh has quit IRC | 20:27 | |
| therve | I guess it would have blocked our gate anyway | 20:28 |
| therve | Sigh | 20:28 |
| *** cwolferh has joined #heat | 20:29 | |
| *** bana_k has quit IRC | 20:39 | |
| zaneb | therve: that's the non-convergence one? convergence one looks blocked by the same DB connection stuff you were discussing with rabi earlier | 20:43 |
| therve | zaneb, Yes | 20:43 |
| therve | Which really does look worrying | 20:44 |
| *** rajpatel has left #heat | 20:44 | |
| therve | I think the "fix" for https://bugs.launchpad.net/heat/+bug/1491185 was really half baked | 20:46 |
| openstack | Launchpad bug 1491185 in heat "convergence: QueuePool limit of size 5 overflow 10 reached" [Medium,Fix released] - Assigned to Anant Patil (ananta) | 20:46 |
| therve | And there is an underlying problem we didn't tackle | 20:46 |
| therve | (Also overriding user config options is super bad :/) | 20:47 |
| *** chlong has quit IRC | 20:53 | |
| *** shardy has quit IRC | 20:59 | |
| zaneb | hmm, looks plausible enough to me, so I guess I don't understand what's going on | 21:00 |
| therve | I tested the stack in https://bugs.launchpad.net/heat/+bug/1489548 and there is something going on with concurrency and convergence | 21:05 |
| openstack | Launchpad bug 1489548 in heat "max_resources_per_stack causes scaling issues on large stacks" [High,Fix released] - Assigned to Steve Baker (steve-stevebaker) | 21:05 |
| therve | I was able to create roughly 100 resources on my setup in about 1min | 21:05 |
| therve | But 150 took 2h | 21:05 |
| therve | I think it was something to do with the scheduler, but I wasn't able to find the issue | 21:06 |
| cwolferh | with max_resources_per_stack = -1? | 21:09 |
| therve | No | 21:09 |
| therve | But I don't think it makes a difference | 21:10 |
| *** fragatina has quit IRC | 21:11 | |
| therve | If I had to guess, I'd say https://github.com/openstack/heat/blob/master/heat/engine/stack.py#L1286 is the issue | 21:15 |
| therve | We don't limit it to anything | 21:15 |
| therve | If the resource graph is flat, we're hammering heat at once | 21:15 |
| openstackgerrit | Thomas Herve proposed openstack/heat: Remove oslo-incubator use in other client plugins https://review.openstack.org/394736 | 21:16 |
| cwolferh | yeah, that makes sense. could impose a sleep after N calls to check_resource, i suppose | 21:22 |
| *** adrian_otto has quit IRC | 21:24 | |
| *** EricGonczer_ has quit IRC | 21:29 | |
| *** catintheroof has joined #heat | 21:31 | |
| *** vijendar has joined #heat | 21:38 | |
| *** vijendar1 has joined #heat | 21:38 | |
| *** fragatina has joined #heat | 21:41 | |
| *** ctayal has quit IRC | 21:53 | |
| *** randallburt has quit IRC | 22:04 | |
| *** jcoufal has quit IRC | 22:23 | |
| *** chris_hultin is now known as chris_hultin|AWA | 22:26 | |
| *** adrian_otto has joined #heat | 22:32 | |
| *** spzala has quit IRC | 22:32 | |
| *** dsneddon is now known as dsneddon_afk | 22:43 | |
| *** chetna2 has joined #heat | 22:43 | |
| *** spotz is now known as spotz_zzz | 22:47 | |
| *** spotz_zzz is now known as spotz | 22:55 | |
| *** jasond has quit IRC | 23:00 | |
| *** EricGonczer_ has joined #heat | 23:01 | |
| *** spotz is now known as spotz_zzz | 23:02 | |
| *** bana_k has joined #heat | 23:14 | |
| *** annegentle has quit IRC | 23:36 | |
| *** annegentle has joined #heat | 23:37 | |
| *** annegentle has quit IRC | 23:41 | |
| *** fragatin_ has joined #heat | 23:50 | |
| *** fragatina has quit IRC | 23:53 | |
| *** bana_k has quit IRC | 23:55 | |
| *** bana_k has joined #heat | 23:55 | |
| *** sdake_ has joined #heat | 23:57 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!