Friday, 2017-06-16

*** zz_dimtruck is now known as dimtruck00:22
*** masber has quit IRC00:32
*** masber has joined #heat00:32
*** zhurong has joined #heat00:34
openstackgerritRico Lin proposed openstack/heat master: Log SyncPoint retry statistics  https://review.openstack.org/47470700:36
*** armaan has joined #heat00:44
*** namnh has joined #heat01:01
*** armaan has quit IRC01:22
*** liusheng has joined #heat01:23
*** zhurong has quit IRC01:30
*** Shunli has joined #heat01:30
*** gmoro has quit IRC01:35
*** gmoro has joined #heat01:36
*** zhurong has joined #heat01:49
*** gmoro has quit IRC01:58
*** gomarivera has joined #heat02:07
*** gmoro has joined #heat02:13
*** ramishra has joined #heat02:14
*** hoai has joined #heat02:29
*** hoai has quit IRC02:31
zanebricolin: so https://review.openstack.org/#/c/474004/ is one of the issues that's still blocking the newton gate02:32
zanebhttps://bugs.launchpad.net/heat/+bug/1532445 is another, although the patch doesn't look backportable. not sure if there is anything we can do to make the test itself less racy02:33
openstackLaunchpad bug 1532445 in heat "ParallelDeploymentsTest intermittent failure" [Medium,Fix released] - Assigned to Thomas Herve (therve)02:33
zaneband there is at least (but probably only) one more02:33
ricolinzaneb, for ParallelDeploymentsTest we keep getting update cancel failure, not sure if is the same one as https://bugs.launchpad.net/heat/+bug/153244502:35
openstackLaunchpad bug 1532445 in heat "ParallelDeploymentsTest intermittent failure" [Medium,Fix released] - Assigned to Thomas Herve (therve)02:35
zanebah ok maybe not then. I didn't look too closely tbh02:36
ricolinI put some observe log here02:36
ricolinhttps://bugs.launchpad.net/heat/+bug/168609202:36
openstackLaunchpad bug 1686092 in heat "stack cancel not complete" [Undecided,In progress] - Assigned to Rico Lin (rico-lin)02:36
*** gomarivera has quit IRC03:06
*** namnh has quit IRC03:06
*** namnh has joined #heat03:07
*** gomarivera has joined #heat03:08
*** chlong has quit IRC03:12
*** dbecker has quit IRC03:20
openstackgerrithuangtianhua proposed openstack/heat master: Update defn of resource if its update cancelled  https://review.openstack.org/47483803:29
*** gmoro_ has joined #heat03:33
*** gmoro has quit IRC03:34
*** dbecker has joined #heat03:35
*** ramishra has quit IRC03:41
*** ramishra has joined #heat03:42
*** akuznetsov has joined #heat03:53
*** jaosorior_away is now known as jaosorior03:53
*** gkadam has joined #heat03:55
*** yassine has quit IRC04:02
*** akuznetsov has quit IRC04:04
*** prateek has joined #heat04:09
*** yassine has joined #heat04:09
*** keiichi has joined #heat04:18
*** prateek has quit IRC04:22
*** prateek has joined #heat04:23
*** zhurong has quit IRC04:25
*** keiichi has quit IRC04:28
*** zhurong has joined #heat04:30
*** gkadam is now known as gkadam-brb04:46
*** gmoro_ has quit IRC05:16
*** gomarivera has quit IRC05:20
*** gkadam-brb is now known as gkadam05:20
*** gomarivera has joined #heat05:27
*** ltomasbo|away is now known as ltomasbo05:36
ramishraseems like plenty of issues with nova/neutron at the gate05:43
ramishrahttp://logs.openstack.org/38/474838/1/check/gate-heat-dsvm-functional-convg-mysql-lbaasv2-py35-ubuntu-xenial/0cec39f/logs/screen-q-agt.txt.gz?level=ERROR05:43
ramishrahttp://logs.openstack.org/38/474838/1/check/gate-heat-dsvm-functional-convg-mysql-lbaasv2-py35-ubuntu-xenial/0cec39f/logs/screen-n-cpu.txt.gz?level=ERROR05:44
*** prateek has quit IRC05:56
*** prateek has joined #heat05:56
*** namnh has quit IRC06:14
*** namnh has joined #heat06:15
*** rcernin has joined #heat06:19
*** rcernin has quit IRC06:24
*** jprovazn has joined #heat06:29
*** gmoro_ has joined #heat06:33
*** gmoro_ has quit IRC06:39
*** rcernin has joined #heat06:40
*** dimtruck is now known as zz_dimtruck06:46
therveramishra, http://logs.openstack.org/04/474004/4/check/gate-heat-dsvm-functional-convg-mysql-lbaasv2-py35-ubuntu-xenial/86c8e01/logs/screen-q-lbaasv2.txt.gz#_Jun_16_06_34_06_734721 looks more specific06:49
ramishratherve: Ah, I did not suspect that though py35 job was failing mostly06:50
therveramishra, Nevermind, it's in successful jobs too06:52
*** amoralej|off is now known as amoralej07:01
*** skramaja has joined #heat07:06
*** gmoro_ has joined #heat07:07
*** jtomasek has joined #heat07:10
*** ccamacho has quit IRC07:14
ramishraAll newton approved backports are in now, result of number of rechecks;)07:17
*** ioggstream has joined #heat07:19
ricolinknock the wood07:19
*** fragatina has joined #heat07:20
*** hoonetorg has joined #heat07:31
*** shardy has joined #heat07:32
LanceHaigIt seems the wiki documentation on using heat with devstack and the docs section are out of sync.07:47
LanceHaighttps://docs.openstack.org/developer/heat/getting_started/on_devstack.html > https://wiki.openstack.org/wiki/Heat/GettingStartedUsingDevstack07:47
ramishraLanceHaig: I think you can ignore the wiki, I'm not sure if it's updated at all07:52
*** om2 has joined #heat07:54
*** zzzeek has quit IRC08:00
*** anton has quit IRC08:02
*** anton has joined #heat08:02
*** zzzeek has joined #heat08:02
*** armaan has joined #heat08:06
*** ccamacho has joined #heat08:17
*** armaan has quit IRC08:20
*** rcernin has quit IRC08:21
*** rcernin has joined #heat08:22
*** gomarivera has quit IRC08:22
*** armaan has joined #heat08:23
*** zhurong has quit IRC08:37
*** gmoro_ has quit IRC08:37
*** zhurong has joined #heat08:45
*** armaan has quit IRC09:06
*** armaan has joined #heat09:07
*** gmoro_ has joined #heat09:12
*** ioggstream has quit IRC09:13
*** armaan has quit IRC09:13
*** armaan has joined #heat09:14
*** Shunli has quit IRC09:15
*** gmoro_ has quit IRC09:24
openstackgerrithuangtianhua proposed openstack/heat master: Don't store rsrc_data before zaqar queue created  https://review.openstack.org/47492309:31
openstackgerrithuangtianhua proposed openstack/heat master: Update defn of resource if its update cancelled  https://review.openstack.org/47483809:36
openstackgerrithuangtianhua proposed openstack/heat master: Add 'contains' function  https://review.openstack.org/47159509:39
openstackgerrithuangtianhua proposed openstack/heat master: Add releasenotes for 'contains' function supporting  https://review.openstack.org/47492809:39
openstackgerritMerged openstack/heat-specs master: Add Zun resources  https://review.openstack.org/42621009:44
*** armaan has quit IRC09:48
*** zhurong has quit IRC09:59
*** armaan has joined #heat10:13
therveramishra, Any idea? It's weird that it's happening only on py310:20
therveshardy, So the issue you mentioned on https://review.openstack.org/#/c/461904/ should have been handled :/10:22
therveI'll test again10:22
ramishratherve: No clue, I'm confused with different type of errors. I thought it's probably urllib3 issue but there are some db errors too.10:24
*** armaan has quit IRC10:24
*** ioggstream has joined #heat10:25
shardytherve: actually it is improved, IIRC it was a 500 before, but now we do get a HTTPBadRequest, it just doesn't say why10:28
therveshardy, Ah, ok10:29
shardytherve: thanks for the patch, it's a big improvement, and unblocks using parameter merging for tripleo :)10:29
*** jprovazn has quit IRC10:29
therveshardy, No problem!10:30
therveI'll look at the issue and will try to improve it10:30
ramishratherve: http://logs.openstack.org/65/473765/1/gate/gate-heat-dsvm-functional-convg-mysql-lbaasv2-py35-ubuntu-xenial/bc8ec2f/logs/screen-q-agt.txt.gz?#_Jun_16_09_07_33_80485810:31
ramishra'ovs is dead' LOL10:32
*** gkadam has quit IRC10:40
*** armaan has joined #heat10:44
ramishratherve: it seems there are number of SG RPC/DB related changes that landed in neutron yesterday. https://review.openstack.org/#/q/topic:bp/push-notifications10:53
ramishraMay be it's due to one of those?10:53
*** nicolasbock has joined #heat11:02
*** rubasov has quit IRC11:12
*** rubasov has joined #heat11:12
LanceHaigis there a place we post logs to so that we don't spam the channel? Something like sexy etc?11:28
LanceHaigI have added my log output here11:29
LanceHaighttps://slexy.org/view/s21d4ruUJQ11:29
LanceHaigI added py35 to the tox.ini for the heat-templates and that is the error I get.11:30
LanceHaigWhere do I start looking to debug?11:30
*** namnh has quit IRC11:31
*** gmoro has joined #heat11:32
*** jprovazn has joined #heat11:33
*** ioggstream has quit IRC11:37
*** ioggstream has joined #heat11:38
*** askb has quit IRC11:39
ramishraplease don't recheck your patches. gate is broken, I've raised bug 1698355 for it. Looks like some issue with neutronclient/neutron.11:46
openstackbug 1698355 in heat "py35 dsvm job failing with RemoteDisconnected error" [Critical,New] https://launchpad.net/bugs/169835511:46
*** om2 has quit IRC11:57
openstackgerritLance Haig proposed openstack/heat master: Add tutorial donated by Florin Stingaciu (stingaci)  https://review.openstack.org/47495312:04
*** root4 has joined #heat12:19
*** dprince has joined #heat12:20
therveshardy, So I'm not seeing the same issue with http://paste.openstack.org/show/610947/12:20
therveshardy, Maybe something to do with what you have in env1/env2?12:21
shardytherve: yeah I forgot to add those, sec12:21
shardytherve: http://paste.openstack.org/show/612846/12:22
*** root2 has quit IRC12:23
therveshardy, So yeah, the error is present in master and because you use the list for the group count :)12:25
shardytherve: yeah I expected it to fail, but shouldn't we raise a parameter validation error that's visible via the client?12:26
therveshardy, Oh totally12:26
therveshardy, But the situation is the same before and after that patch12:27
shardytherve: ah, Ok, we can perhaps have a follow-up bug then12:28
shardythanks for confirming12:28
therveYep, I'll open something12:28
*** om2 has joined #heat12:40
openstackgerritThomas Herve proposed openstack/heat master: Catch type errors when casting to int  https://review.openstack.org/47495912:40
therveshardy, ^^^12:40
*** prateek has quit IRC12:42
*** MVenesio has joined #heat12:42
therveramishra, So neutronclient didn't change from a success build, and I don't see anything in openstacksdk12:50
therveSo it has to be neutron, or some binary behind it12:50
ramishratherve: yeah possibly, I added neutron to the bug12:52
*** catintheroof has joined #heat12:54
*** akuznetsov has joined #heat12:55
ramishratherve: something weird happening in q-svc http://logs.openstack.org/95/471595/4/check/gate-heat-dsvm-functional-convg-mysql-lbaasv2-py35-ubuntu-xenial/51c5b1d/logs/screen-q-svc.txt.gz#_Jun_16_10_44_29_23030012:57
*** akuznetsov has quit IRC12:57
shardytherve: thanks, will give it a try13:00
*** jcoufal has joined #heat13:03
*** gomarivera has joined #heat13:11
*** f13o has joined #heat13:13
*** jcoufal_ has joined #heat13:17
*** jcoufal has quit IRC13:19
*** fragatin_ has joined #heat13:21
*** fragatina has quit IRC13:21
*** gomarivera has quit IRC13:25
*** gomarivera has joined #heat13:26
*** jmlowe has quit IRC13:27
*** amoralej is now known as amoralej|lunch13:30
*** gomarivera has quit IRC13:30
*** akuznetsov has joined #heat13:30
*** om2 has quit IRC13:33
zanebramishra: have you told neutron folks that they've borked python3?13:44
ramishrazaneb: no yet other than adding neutron to the bug, was not sure about origin of the issue13:46
ramishramay be we can make some noise in neutron channel now:)13:46
zanebnoise++ :)13:49
therveramishra, OOM would be weird considering how consistent it has been13:50
*** shardy is now known as shardy_afk13:51
*** crose has joined #heat13:57
*** jmlowe has joined #heat14:00
ramishratherve: http://logs.openstack.org/65/473765/1/check/gate-heat-dsvm-functional-convg-mysql-lbaasv2-py35-ubuntu-xenial/e07f32f/logs/syslog.txt.gz#_Jun_16_10_24_2214:07
*** shardy_afk is now known as shardy14:09
therveramishra, 1.3 G of RSS? Yeah that sounds like an issue :)14:09
therveSo there is probably a leak of some sort14:10
therveIt topped at 140M before14:11
*** zz_dimtruck is now known as dimtruck14:12
*** chlong has joined #heat14:15
therveShould we skip py3 tests to unblock the gate? It doesn't sound like an easy fix is in order?14:16
openstackgerritBence Romsics proposed openstack/heat master: New resource: Neutron Trunk  https://review.openstack.org/44249614:16
ramishratherve: probably it's due to the neutron patches that landed yesterday/early this morning.14:16
therveramishra, Yeah which one is going to be fun to find14:17
*** gomarivera has joined #heat14:25
*** amoralej|lunch is now known as amoralej14:26
ramishrashardy: gate broken:)14:33
shardyramishra: Ok, I thought the lbass tests were always flaky14:37
*** jaosorior has quit IRC14:37
*** jmlowe_ has joined #heat14:39
*** jprovazn has quit IRC14:40
ramishrashardy: yep, it's confusing as only py3 job is failing14:40
*** jcoufal has joined #heat14:40
*** jmlowe has quit IRC14:41
*** jcoufal_ has quit IRC14:41
*** hjensas has quit IRC14:54
*** hjensas has joined #heat14:54
*** armaan has quit IRC14:57
*** armaan has joined #heat14:58
*** akuznetsov has quit IRC15:00
openstackgerritThomas Herve proposed openstack/heat master: DNM test neutron  https://review.openstack.org/47499815:01
*** ayoung has quit IRC15:05
openstackgerritAkihiro Motoki proposed openstack/heat master: DNM another test  https://review.openstack.org/47500015:10
*** felipemonteiro has joined #heat15:12
*** gomarivera has quit IRC15:14
*** rcernin has quit IRC15:15
*** jcoufal has quit IRC15:17
*** jcoufal has joined #heat15:18
*** MVenesio has quit IRC15:42
*** bnemec is now known as beekneemech16:01
*** armaan has quit IRC16:04
*** armaan has joined #heat16:05
*** ccamacho has quit IRC16:05
*** ramishra has quit IRC16:08
*** ltomasbo is now known as ltomasbo|away16:10
*** gomarivera has joined #heat16:19
*** felipemonteiro has quit IRC16:24
*** ayoung has joined #heat16:32
*** armaan has quit IRC16:43
*** shardy has quit IRC16:44
*** felipemonteiro has joined #heat16:59
*** felipemonteiro_ has joined #heat17:00
*** jamielennox has quit IRC17:03
*** jamielennox has joined #heat17:03
*** felipemonteiro has quit IRC17:04
*** ioggstream has quit IRC17:12
*** crose has quit IRC17:13
*** jtomasek has quit IRC17:14
*** gomarivera has quit IRC17:15
*** gomarivera has joined #heat17:15
*** gomarivera has quit IRC17:17
*** gomarivera has joined #heat17:17
*** gomarivera has quit IRC17:17
*** gomarivera has joined #heat17:17
*** dprince has quit IRC17:18
*** MVenesio has joined #heat17:32
*** jcoufal_ has joined #heat17:34
*** MVenesio has quit IRC17:35
*** jcoufal has quit IRC17:36
*** jcoufal has joined #heat17:36
*** catintheroof has quit IRC17:37
*** jcoufal_ has quit IRC17:39
*** felipemonteiro_ has quit IRC17:41
*** felipemonteiro_ has joined #heat17:41
*** om2 has joined #heat17:46
*** jcoufal_ has joined #heat17:49
*** jcoufal has quit IRC17:51
*** root has joined #heat17:52
*** root is now known as Guest8533217:52
*** root4 has quit IRC17:55
*** Guest85332 has quit IRC17:56
*** dprince has joined #heat18:01
*** jcoufal has joined #heat18:04
*** jcoufal_ has quit IRC18:06
*** ricolin has quit IRC18:07
*** ihrachys has joined #heat18:09
ihrachystherve, howdy18:10
ihrachysI hear you may be aware of the gate breakage18:10
ihrachysI need to talk to someone about broader context18:10
ihrachysI know the root cause, and I have a fix18:10
ihrachysbut it's in oslo.serialization so will take time to bubble up18:11
*** fragatin_ has quit IRC18:11
ihrachysI don't fully understand yet why it hit just now though. looks like a general issue.18:11
ihrachysunless heat was not gating on py3 before18:11
ihrachysthoughts?18:11
ihrachysthe error comes from the fact that jsonutils.to_primitive for network.IPNetwork('0.0.0.0/0') goes and iterates through all addresses in the range (A LOT!) because the object has __iter__18:12
ihrachysso I wonder where this 0.0.0.0/0 come from18:12
*** root2 has joined #heat18:32
*** nicolasbock has quit IRC18:38
therveihrachys, on and off!18:43
therveihrachys, https://review.openstack.org/#/c/474997/ seems to fix the issue for now18:43
therveWe've been gating on py3 for a while, so it's definitely new18:43
therveihrachys, I don't think the fix should be oslo.serialization, but maybe I miss something18:45
therveWe'll skip the tests, we can't wait for a fix there as it needs to be released and everything18:45
ihrachystherve, there is definitely a bug in oslo.serialization18:46
ihrachysbut also neutron18:47
ihrachysI respinned https://review.openstack.org/#/c/474575/ it had some issues in previous iterations18:47
ihrachysnow rechecking heat depends-on patch again18:47
ihrachysand this is the oslo fix: https://review.openstack.org/47505218:47
*** om2 has quit IRC18:48
*** dontstalkmepls[m has quit IRC18:59
*** jtomasek has joined #heat19:10
*** dimtruck is now known as zz_dimtruck19:14
*** jtomasek has quit IRC19:16
*** liusheng has quit IRC19:21
*** liusheng has joined #heat19:22
*** amoralej is now known as amoralej|off19:25
*** armaan has joined #heat19:45
*** armaan has quit IRC19:49
*** zz_dimtruck is now known as dimtruck19:55
*** jcoufal has quit IRC19:58
*** zzzeek has quit IRC20:07
*** f13o has quit IRC20:11
*** jtomasek has joined #heat20:22
*** f13o has joined #heat20:26
therveihrachys, Great, looks like the last version of the patch worked for us20:27
ihrachystherve, yep :)20:27
ihrachysthere is a small respin still processing but I am hopefu20:28
ihrachys*hopeful20:28
therveThanks a lot20:28
therveihrachys, Do you know why it only happened on py3 btw?20:28
therveSomething to do with the iterator protocol?20:28
ihrachystherve, maybe, not sure. py2 also returns MemoryError but does it a lot quicker20:29
ihrachysmaybe it's that in py2 we request a list for some structure and malloc fails right away20:29
ihrachyswhile in py3 we iterate and appen20:30
ihrachys*append20:30
ihrachysso it does swallow memory by small pieces20:30
ihrachysI would guess that, haven't spent more time to dig that20:30
*** zzzeek has joined #heat20:31
therveOh you're right, we do get the memoryerror in py220:34
therveWeird that there is no failure associated in our tests20:34
therveOh it's just notifications20:35
therveSo it doesn't have a side effect in py2, but it's getting killed in py3, interesting20:35
*** jmlowe_ has quit IRC20:39
*** felipemonteiro_ has quit IRC20:49
*** root2 has quit IRC20:57
*** root2 has joined #heat21:02
*** dprince has quit IRC21:08
*** skramaja has quit IRC21:15
*** skramaja has joined #heat21:15
*** chlong has quit IRC21:30
*** jtomasek has quit IRC21:31
*** askb has joined #heat21:32
*** dontstalkmepls[m has joined #heat21:32
*** fragatina has joined #heat21:35
*** jmlowe has joined #heat21:49
*** jdandrea has quit IRC21:57
*** ihrachys has quit IRC22:00
*** dimtruck is now known as zz_dimtruck22:03
*** fragatina has quit IRC22:10
*** gomarivera has quit IRC22:57
*** masuberu has joined #heat23:16
*** therve_ has joined #heat23:24
*** spotz_ has joined #heat23:24
*** logan_ has joined #heat23:25
*** hoonetorg has quit IRC23:26
*** masber has quit IRC23:26
*** rwsu has quit IRC23:26
*** therve has quit IRC23:26
*** FL1SK has quit IRC23:26
*** ykuo has quit IRC23:26
*** spotz has quit IRC23:26
*** apuimedo has quit IRC23:26
*** logan- has quit IRC23:26
*** weshay has quit IRC23:26
*** mnaser has quit IRC23:26
*** yuval has quit IRC23:26
*** rwsu has joined #heat23:27
*** yuval has joined #heat23:27
*** hoonetorg has joined #heat23:28
*** apuimedo has joined #heat23:28
*** logan_ is now known as logan-23:29
*** weshay has joined #heat23:31
*** ykuo has joined #heat23:31
*** weshay has quit IRC23:31
*** weshay has joined #heat23:31
*** mnaser has joined #heat23:38
*** yassine has quit IRC23:46
*** yassine has joined #heat23:50
*** zz_dimtruck is now known as dimtruck23:52

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!