| *** yangyapeng has quit IRC | 00:25 | |
| *** yassine has joined #heat | 00:26 | |
| *** yassine is now known as Guest92842 | 00:27 | |
| *** hieulq has quit IRC | 00:46 | |
| *** hieulq has joined #heat | 00:47 | |
| *** kiennt26 has joined #heat | 01:08 | |
| *** yangyapeng has joined #heat | 01:10 | |
| *** gouthamr has quit IRC | 01:21 | |
| *** namnh has joined #heat | 01:27 | |
| *** Guest92842 has quit IRC | 01:35 | |
| *** namnh has quit IRC | 01:36 | |
| *** hoonetorg has quit IRC | 01:41 | |
| *** yuanying has joined #heat | 01:42 | |
| -openstackstatus- NOTICE: The freenode network is currently the target of automated spam attacks, we have enabled temporary restrictions on targetted OpenStack channels which requires users to be logged on to NickServ. If you see spam in your channel, please report it in #openstack-infra. Thanks. | 01:46 | |
| *** hoonetorg has joined #heat | 01:50 | |
| *** jmlowe has quit IRC | 01:57 | |
| *** kiennt26 has quit IRC | 02:02 | |
| *** ayoung has quit IRC | 02:16 | |
| *** ricolin_ has joined #heat | 02:45 | |
| *** ChanServ sets mode: +r | 02:51 | |
| openstackgerrit | Rico Lin proposed openstack/heat master: Remove AutoscalingLoadBalancerTest and ReloadOnSighupTest https://review.openstack.org/528570 | 02:59 |
|---|---|---|
| *** gkadam has joined #heat | 03:37 | |
| openstackgerrit | Rico Lin proposed openstack/heat master: Add tempest ids for functional and scenario tests https://review.openstack.org/506525 | 03:54 |
| openstackgerrit | Rico Lin proposed openstack/heat master: Add tempest ids for functional and scenario tests https://review.openstack.org/506525 | 03:56 |
| *** flwang1 has quit IRC | 04:00 | |
| *** ramishra has joined #heat | 04:11 | |
| openstackgerrit | Rico Lin proposed openstack/heat-tempest-plugin master: Add tempest ids for functional and scenario tests https://review.openstack.org/521861 | 04:12 |
| openstackgerrit | Rico Lin proposed openstack/heat-tempest-plugin master: Add tempest ids for functional and scenario tests https://review.openstack.org/521861 | 04:18 |
| *** dbecker has quit IRC | 04:25 | |
| *** gkadam has quit IRC | 04:54 | |
| *** ivve has quit IRC | 04:58 | |
| ricolin_ | ramishra, please help to review if you have time https://review.openstack.org/#/q/topic:ci-test-framework+(status:open+OR+status:merged) | 05:18 |
| *** yangyapeng has quit IRC | 05:31 | |
| *** ramishra has quit IRC | 05:35 | |
| openstackgerrit | Merged openstack/heat master: Enable to specify network for Trove Cluster https://review.openstack.org/505521 | 05:41 |
| openstackgerrit | Merged openstack/heat master: ForceDelete Instance https://review.openstack.org/454941 | 05:46 |
| *** ramishra has joined #heat | 06:01 | |
| *** ramishra has quit IRC | 06:06 | |
| *** ramishra has joined #heat | 06:13 | |
| *** ramishra has quit IRC | 06:21 | |
| *** hjensas has quit IRC | 06:25 | |
| *** hjensas has joined #heat | 06:26 | |
| *** ramishra has joined #heat | 06:27 | |
| *** namnh has joined #heat | 06:37 | |
| *** namnh has quit IRC | 06:37 | |
| *** namnh has joined #heat | 06:38 | |
| *** ricolin__ has joined #heat | 06:43 | |
| *** hoonetorg has quit IRC | 06:45 | |
| *** ricolin_ has quit IRC | 06:45 | |
| *** rcernin has quit IRC | 06:46 | |
| *** hieulq has quit IRC | 06:47 | |
| *** hieulq has joined #heat | 06:48 | |
| *** hoonetorg has joined #heat | 07:02 | |
| *** kiennt26 has joined #heat | 07:16 | |
| *** kei-ichi has quit IRC | 07:24 | |
| *** rcernin has joined #heat | 08:06 | |
| *** shardy has joined #heat | 08:17 | |
| *** gkadam has joined #heat | 08:28 | |
| *** ioggstream has joined #heat | 08:51 | |
| *** ccamacho has joined #heat | 08:51 | |
| *** amoralej|off is now known as amoralej | 08:52 | |
| *** ricolin__ has quit IRC | 08:53 | |
| *** ricolin has joined #heat | 08:53 | |
| *** shardy has quit IRC | 08:59 | |
| *** shardy has joined #heat | 09:00 | |
| *** k_mouza has joined #heat | 09:20 | |
| *** SmearedBeard has joined #heat | 09:21 | |
| *** ricolin has quit IRC | 09:24 | |
| *** andreykurilin_ has quit IRC | 09:36 | |
| *** andreykurilin has joined #heat | 09:36 | |
| *** armaan has joined #heat | 09:44 | |
| *** kiennt26 has quit IRC | 09:46 | |
| *** d0ugal_ has quit IRC | 10:36 | |
| *** namnh has quit IRC | 10:52 | |
| *** SmearedBeard has quit IRC | 10:58 | |
| *** rcernin has quit IRC | 10:58 | |
| *** SmearedBeard has joined #heat | 11:07 | |
| *** d0ugal has joined #heat | 11:17 | |
| *** huangtianhua has quit IRC | 11:24 | |
| *** derekh has joined #heat | 11:29 | |
| *** SmearedBeard has quit IRC | 11:41 | |
| *** pbourke has quit IRC | 11:43 | |
| *** armaan has quit IRC | 12:01 | |
| *** SmearedBeard has joined #heat | 12:07 | |
| *** nicolasbock_ has joined #heat | 12:11 | |
| *** nicolasbock_ has quit IRC | 12:26 | |
| *** SmearedBeard has quit IRC | 13:32 | |
| *** jrist has quit IRC | 13:39 | |
| *** amoralej is now known as amoralej|lunch | 13:45 | |
| *** SmearedBeard has joined #heat | 13:54 | |
| *** jrist has joined #heat | 13:54 | |
| *** ramishra has quit IRC | 14:03 | |
| *** ChanServ sets mode: -r | 14:04 | |
| *** dprince has joined #heat | 14:04 | |
| *** dprince has quit IRC | 14:06 | |
| *** dprince has joined #heat | 14:07 | |
| *** jcoufal has joined #heat | 14:13 | |
| *** zaneb has joined #heat | 14:13 | |
| *** felipemonteiro has joined #heat | 14:20 | |
| *** amoralej|lunch is now known as amoralej | 14:21 | |
| *** jcoufal_ has joined #heat | 14:28 | |
| *** jcoufal has quit IRC | 14:29 | |
| *** stevebaker has quit IRC | 14:35 | |
| *** jmlowe has joined #heat | 14:40 | |
| *** gfidente has joined #heat | 14:43 | |
| *** stevebaker has joined #heat | 14:43 | |
| *** Smeared_Beard has joined #heat | 14:50 | |
| *** SmearedBeard has quit IRC | 14:51 | |
| *** felipemonteiro_ has joined #heat | 14:51 | |
| *** felipemonteiro has quit IRC | 14:54 | |
| *** d0ugal has quit IRC | 14:56 | |
| *** dprince has quit IRC | 15:04 | |
| *** dprince has joined #heat | 15:05 | |
| *** d0ugal has joined #heat | 15:08 | |
| *** d0ugal has quit IRC | 15:11 | |
| *** d0ugal has joined #heat | 15:13 | |
| therve | zaneb, Hi, around? I'm debugging a stuck stack update, wondering if you had any clue | 15:14 |
| zaneb | around, yes :) | 15:14 |
| zaneb | on the other part, opinions differ :D | 15:14 |
| therve | Heh | 15:15 |
| therve | So it seems that I'm stuck on a few resource chains update | 15:16 |
| therve | Nothing seems to be happening | 15:16 |
| therve | Which would tend to put the blame on convergence | 15:16 |
| zaneb | is this a TripleO stack? | 15:17 |
| therve | Yes | 15:17 |
| zaneb | ok | 15:17 |
| therve | They all require the same 4 resources, which sounds suspect | 15:17 |
| zaneb | hmm, yeah that does sound suspect, doesn't it | 15:18 |
| zaneb | like one of those didn't get completed correctly | 15:18 |
| zaneb | no unhandled exceptions in the log I assume? | 15:19 |
| therve | There is another resource with the same requires set, and it completed | 15:19 |
| therve | No, no error | 15:20 |
| zaneb | the plot thickens | 15:20 |
| zaneb | user hooks? | 15:21 |
| therve | No | 15:21 |
| *** ayoung has joined #heat | 15:28 | |
| *** gouthamr has joined #heat | 15:29 | |
| *** gouthamr has quit IRC | 15:29 | |
| zaneb | therve: the only thing I can think of is to check the SyncPoints in the DB and see what they contain | 15:31 |
| therve | zaneb, http://paste.openstack.org/show/629214/ | 15:32 |
| therve | Those are the points for the stacks I think | 15:33 |
| zaneb | trying to wrap my head around what i'm looking at here | 15:34 |
| therve | With the query it'd be better | 15:35 |
| zaneb | those are 4 resources in 4 different stacks | 15:35 |
| therve | http://paste.openstack.org/show/629215/ | 15:35 |
| therve | Yeah those are the 4 stack resources | 15:35 |
| therve | Hum we put both stack id and resource id in entity id, weird | 15:36 |
| zaneb | oh, it's the syncpoint for the stack itself | 15:36 |
| therve | zaneb, http://paste.openstack.org/show/629216/ are the resources? | 15:38 |
| therve | Seems like one is duplicated | 15:38 |
| zaneb | looks like a forward and a reverse node for each resource | 15:39 |
| *** felipemonteiro_ has quit IRC | 15:40 | |
| therve | So yeah when it worked the "is_update false" point is updated | 15:40 |
| zaneb | ? | 15:42 |
| therve | zaneb, http://paste.openstack.org/show/629218/ | 15:44 |
| therve | 65 is the other ServiceChain with the same requires, that completed successfully | 15:45 |
| zaneb | causality is backwards :) when the update runs, the reverse nodes get updated | 15:46 |
| zaneb | so what's in the input_data? | 15:47 |
| zaneb | atomic key of 4 definitely implies that all 4 dependencies updated it though | 15:48 |
| therve | A crapload of stuff | 15:48 |
| therve | s/65/75 | 15:48 |
| therve | It's tripleo, so about 1M of properties each time :D | 15:48 |
| zaneb | exactly 4 craploads of stuff? | 15:49 |
| therve | Can you access it? That'd be easier | 15:49 |
| zaneb | yup | 15:50 |
| therve | I think I've been getting that issue for a while now, where I can't recover from failed state | 15:55 |
| therve | Note that those resources where in failed state previously | 15:56 |
| zaneb | 4 keys in the input_data as expected: [u'tuple:(90, True)', u'tuple:(80, True)', u'tuple:(82, True)', u'tuple:(86, True)'] | 16:01 |
| zaneb | so the SyncPoints are up-to-date | 16:01 |
| *** anton has quit IRC | 16:02 | |
| *** anton has joined #heat | 16:02 | |
| zaneb | oh, and action says UPDATE_IN_PROGRESS | 16:03 |
| zaneb | so we're looking in the wrong place | 16:04 |
| *** felipemonteiro has joined #heat | 16:07 | |
| therve | Why's that? :) | 16:07 |
| therve | The last log we have is Waiting 62: ... | 16:09 |
| therve | still need ConvergenceNode(rsrc_id=64, is_update=True) | 16:09 |
| therve | Which looks like done | 16:10 |
| zaneb | we're in the parent stack. the nested stack is IN_PROGRESS and that's presumably where it has stalled | 16:10 |
| therve | But there are no other resources IN_PROGRESS though? | 16:11 |
| zaneb | indeed | 16:14 |
| therve | zaneb, So that last line looks like a symptom to me. 62 waits on 64 is_update. But 64 is in CREATE_COMPLETE, so shouldn't be updated | 16:18 |
| therve | Unless I miss something which is highly probably | 16:18 |
| *** hjensas has quit IRC | 16:19 | |
| *** hjensas has joined #heat | 16:20 | |
| *** hjensas has quit IRC | 16:20 | |
| *** hjensas has joined #heat | 16:20 | |
| *** jcoufal has joined #heat | 16:30 | |
| *** ricolin has joined #heat | 16:31 | |
| *** jcoufal_ has quit IRC | 16:32 | |
| ricolin | zaneb, therve please help to review if you have time, thx! https://review.openstack.org/#/q/topic:ci-test-framework+(status:open+OR+status:merged) | 16:32 |
| *** zaneb has quit IRC | 16:35 | |
| *** zaneb has joined #heat | 16:36 | |
| therve | zaneb, Ah, we got a timeout :/ | 16:36 |
| zaneb | no, I had to reboot :/ | 16:37 |
| *** catintheroof has joined #heat | 16:38 | |
| therve | zaneb, Me too weirdly. But the stack did timeout | 16:38 |
| zaneb | ah! | 16:39 |
| zaneb | well, that's something | 16:39 |
| therve | I guess that part does work yes :) | 16:39 |
| therve | Any idea of the issue so far? Should I retry? | 16:40 |
| zaneb | gimme a sec to poke at the SyncPoints for 62/64 | 16:43 |
| zaneb | oh, they got deleted by the timeout :/ | 16:43 |
| therve | Yep | 16:44 |
| zaneb | at least timeouts are working well | 16:44 |
| zaneb | yeah, I guess try again, I'm not sure there's much more we can learn atm | 16:45 |
| *** catintheroof has quit IRC | 16:45 | |
| therve | OK | 16:45 |
| *** armaan has joined #heat | 16:47 | |
| therve | zaneb, I think we're here again | 16:50 |
| therve | That was quick at least | 16:50 |
| *** crose has joined #heat | 16:51 | |
| *** AlexeyAbashkin has joined #heat | 16:53 | |
| zaneb | so 62 is waiting on 64 | 16:54 |
| zaneb | but 64 is waiting on 77, 70, 67, 69 | 16:55 |
| *** felipemonteiro has quit IRC | 16:55 | |
| *** AlexeyAbashkin has quit IRC | 16:57 | |
| therve | 77 is waiting on ...? 78? | 16:58 |
| therve | At least it depends on it | 16:58 |
| zaneb | yeah, this all looks pretty standard tbh | 17:00 |
| zaneb | we're still in the top-level stack | 17:00 |
| *** ricolin has quit IRC | 17:00 | |
| therve | 77, 70, 76, 69 all depends on the 4 "stuck" resources | 17:01 |
| *** rwsu has joined #heat | 17:01 | |
| zaneb | problem is buried several nested stacks deep. that's where we have a resource that's in-progress but it's child stack is not AIUI | 17:01 |
| zaneb | *its | 17:01 |
| therve | Do we? | 17:01 |
| therve | Looks like the 5 stacks in progress are the 4 we have plus the main one | 17:02 |
| zaneb | correction, stack is in progress but none of its resources are | 17:03 |
| therve | Ah right | 17:03 |
| zaneb | so the LoggingConfiguration completed | 17:05 |
| zaneb | after that, no activity by the looks of it | 17:05 |
| zaneb | so e.g. look at resource id 126 | 17:08 |
| zaneb | which is LoggingSourcesConfig | 17:08 |
| zaneb | that depends only on LoggingConfiguration afaict | 17:09 |
| zaneb | and that has completed (it's 151), but 126 has not started | 17:10 |
| therve | It depends on 211 too | 17:10 |
| zaneb | oh, my bad, it also depends on ServiceChain | 17:10 |
| *** felipemonteiro has joined #heat | 17:11 | |
| therve | Which... doesn't depend on anything? | 17:12 |
| therve | It's update complete with reason "The Resource ServiceChain requires replacement." | 17:12 |
| therve | Weird | 17:12 |
| zaneb | yeah, interesting that 211 replaced 736 at some point | 17:12 |
| therve | Did it? | 17:13 |
| *** felipemonteiro_ has joined #heat | 17:13 | |
| therve | 211 updated complete ,but none of its sync points did | 17:14 |
| *** felipemonteiro has quit IRC | 17:17 | |
| zaneb | ok, so that's our problem | 17:17 |
| zaneb | crap | 17:20 |
| zaneb | it's sending its updates with resource id 754 | 17:21 |
| zaneb | wait, 754 is not 736, that makes no sense either | 17:22 |
| *** catintheroof has joined #heat | 17:23 | |
| *** armaan has quit IRC | 17:24 | |
| zaneb | oh ignore me, I'm looking at logs from the wrong nested stack I think | 17:24 |
| *** armaan has joined #heat | 17:24 | |
| *** crose has quit IRC | 17:27 | |
| therve | Probably unrelated but found out about https://review.openstack.org/#/c/222562/ and oh god why | 17:29 |
| *** catintheroof has quit IRC | 17:30 | |
| *** ioggstream has quit IRC | 17:30 | |
| therve | https://stackoverflow.com/questions/15721363/preserve-python-tuples-with-json like 5s in google :/ | 17:31 |
| *** pas-ha has left #heat | 17:32 | |
| *** k_mouza has quit IRC | 17:32 | |
| zaneb | therve: afaict the update of 211 was triggered (at 16:48:57.034) but never happened | 17:35 |
| therve | There is a COMPLETE even on ServiceChain at 16:49:23 | 17:37 |
| therve | bbl | 17:37 |
| *** gkadam has quit IRC | 17:38 | |
| *** flwang1 has joined #heat | 17:39 | |
| zaneb | therve: different stack, that's BlockStorage. 211 is in CephStorage | 17:40 |
| *** amoralej is now known as amoralej|off | 17:49 | |
| zaneb | oh. | 17:51 |
| zaneb | I read the replacement thing backwards | 17:51 |
| *** derekh has quit IRC | 17:52 | |
| zaneb | therve: 211 was replaced *by* 736 (which is now DELETE_COMPLETE) so it's bailing out at http://git.openstack.org/cgit/openstack/heat/tree/heat/engine/check_resource.py#n305 | 17:53 |
| zaneb | we should log that. | 17:53 |
| *** harlowja has joined #heat | 18:04 | |
| *** AlexeyAbashkin has joined #heat | 18:14 | |
| *** AlexeyAbashkin has quit IRC | 18:18 | |
| *** shardy has quit IRC | 18:28 | |
| *** gfidente is now known as gfidente|afk | 18:29 | |
| *** catintheroof has joined #heat | 18:36 | |
| *** catintheroof has quit IRC | 18:36 | |
| *** catintheroof has joined #heat | 18:36 | |
| *** catintheroof has quit IRC | 18:36 | |
| *** fragatina has joined #heat | 18:39 | |
| *** jmlowe has quit IRC | 18:47 | |
| *** catintheroof has joined #heat | 18:56 | |
| *** fragatina has quit IRC | 18:59 | |
| *** flwang1 has quit IRC | 19:05 | |
| *** jmlowe has joined #heat | 19:08 | |
| *** jose-phillips has joined #heat | 20:09 | |
| *** jtomasek has joined #heat | 20:20 | |
| *** flwang1 has joined #heat | 20:21 | |
| *** armaan has quit IRC | 20:21 | |
| *** armaan has joined #heat | 20:22 | |
| *** ccamacho has quit IRC | 20:26 | |
| therve | zaneb, I added the logs, and we do get there on 4 resources... | 20:27 |
| *** crose has joined #heat | 20:30 | |
| *** askb has quit IRC | 20:30 | |
| therve | Yeah those are the 4 ServiceChains resource inside the other ones | 20:30 |
| *** jmlowe has quit IRC | 20:36 | |
| *** d0ugal has quit IRC | 20:36 | |
| *** d0ugal has joined #heat | 20:38 | |
| *** fragatina has joined #heat | 20:51 | |
| *** crose has quit IRC | 20:51 | |
| *** ChanServ has quit IRC | 20:52 | |
| zaneb | therve: yeah, I'm guessing something weird has happened to those resources in the past to get it into that state | 20:56 |
| zaneb | like rollback or something? | 20:56 |
| *** dprince has quit IRC | 20:58 | |
| therve | Yeah an update failed | 20:58 |
| therve | But I think we should be able to recover, right? | 20:59 |
| therve | Ah yeah we need to fix ooo to not bail out when things are in progress :) | 21:00 |
| therve | Otherwise there is little point in using convergence | 21:01 |
| *** ChanServ has joined #heat | 21:01 | |
| *** barjavel.freenode.net sets mode: +o ChanServ | 21:01 | |
| therve | We'll see tomorrow :) | 21:05 |
| *** jcoufal_ has joined #heat | 21:13 | |
| *** jcoufal has quit IRC | 21:15 | |
| *** jcoufal_ has quit IRC | 21:17 | |
| *** felipemonteiro_ has quit IRC | 21:19 | |
| *** jcoufal has joined #heat | 21:20 | |
| *** catintheroof has quit IRC | 21:20 | |
| *** jtomasek has quit IRC | 21:24 | |
| *** jcoufal has quit IRC | 21:31 | |
| *** jcoufal has joined #heat | 21:33 | |
| *** openstackgerrit has quit IRC | 21:33 | |
| *** gfidente|afk has quit IRC | 21:36 | |
| *** jose-phillips has quit IRC | 21:38 | |
| *** AlexeyAbashkin has joined #heat | 21:38 | |
| *** AlexeyAbashkin has quit IRC | 21:42 | |
| *** dprince has joined #heat | 21:43 | |
| *** jose-phillips has joined #heat | 21:47 | |
| *** rwsu has quit IRC | 21:54 | |
| *** askb has joined #heat | 22:34 | |
| *** rcernin has joined #heat | 22:37 | |
| *** AlexeyAbashkin has joined #heat | 22:38 | |
| *** AlexeyAbashkin has quit IRC | 22:42 | |
| *** dprince has quit IRC | 23:00 | |
| *** Smeared_Beard has quit IRC | 23:01 | |
| *** catinthe_ has joined #heat | 23:08 | |
| *** gouthamr has joined #heat | 23:13 | |
| *** catinthe_ has quit IRC | 23:13 | |
| *** harlowja has quit IRC | 23:38 | |
| *** d0ugal has quit IRC | 23:51 | |
| *** jcoufal has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!