| openstackgerrit | Harald Jensås proposed openstack/heat master: Exclude updated networks - match actual interface state https://review.openstack.org/471969 | 00:08 |
|---|---|---|
| *** yangyapeng has quit IRC | 00:14 | |
| *** yangyapeng has joined #heat | 00:14 | |
| openstackgerrit | Harald Jensås proposed openstack/heat master: Exclude updated networks - match actual interface state https://review.openstack.org/471969 | 00:16 |
| *** yangyapeng has quit IRC | 00:18 | |
| *** felipemonteiro has quit IRC | 00:28 | |
| *** kiennt has joined #heat | 00:32 | |
| *** kiennt has quit IRC | 00:39 | |
| *** kiennt has joined #heat | 00:51 | |
| *** Shunli has joined #heat | 00:52 | |
| *** dixiaoli has joined #heat | 00:53 | |
| *** dixiaoli has quit IRC | 00:55 | |
| *** fragatin_ has joined #heat | 01:05 | |
| *** fragati__ has joined #heat | 01:07 | |
| *** fragatina has quit IRC | 01:08 | |
| *** fragatin_ has quit IRC | 01:10 | |
| *** fragati__ has quit IRC | 01:12 | |
| *** ricolin has joined #heat | 01:14 | |
| *** elynn has joined #heat | 01:17 | |
| *** yangyapeng has joined #heat | 01:20 | |
| *** yangyapeng has quit IRC | 01:21 | |
| *** yangyapeng has joined #heat | 01:21 | |
| *** ramishra has joined #heat | 02:07 | |
| *** chlong_ has quit IRC | 03:08 | |
| *** chlong_ has joined #heat | 03:09 | |
| openstackgerrit | huangtianhua proposed openstack/heat master: Rollback stack with correct tags https://review.openstack.org/480153 | 03:13 |
| openstackgerrit | Ethan Lynn proposed openstack/heat master: Set ssl options when initialize heat client() https://review.openstack.org/480923 | 03:16 |
| *** chlong_ has quit IRC | 03:17 | |
| *** chlong_ has joined #heat | 03:19 | |
| *** dbecker has quit IRC | 03:22 | |
| *** felipemonteiro has joined #heat | 03:23 | |
| *** Shunli has quit IRC | 03:25 | |
| *** Shunli has joined #heat | 03:26 | |
| *** gkadam has joined #heat | 03:32 | |
| *** dbecker has joined #heat | 03:35 | |
| *** felipemonteiro has quit IRC | 03:43 | |
| *** Shunli has quit IRC | 03:54 | |
| *** Shunli has joined #heat | 03:55 | |
| openstackgerrit | huangtianhua proposed openstack/heat master: Modify the example of networks attribute of server https://review.openstack.org/481408 | 04:12 |
| *** links has joined #heat | 04:30 | |
| openstackgerrit | huangtianhua proposed openstack/heat master: Deprecate 'accessIPv4'/'accessIPv6' attributes https://review.openstack.org/481413 | 04:34 |
| *** faizy has joined #heat | 04:42 | |
| *** zzzeek has joined #heat | 04:55 | |
| *** dixiaoli has joined #heat | 04:56 | |
| openstackgerrit | Rico Lin proposed openstack/heat master: Add converge flag in stack update for observing on reality https://review.openstack.org/461302 | 04:56 |
| *** zzzeek has quit IRC | 04:56 | |
| ricolin | ramishra, could you review on https://review.openstack.org/461302 again | 04:57 |
| ramishra | ricolin: sure, will have a look | 04:58 |
| ricolin | would like to make sure that's what you prefered:) | 04:58 |
| *** zzzeek has joined #heat | 05:02 | |
| *** zzzeek has quit IRC | 05:02 | |
| *** zzzeek has joined #heat | 05:04 | |
| *** zzzeek has quit IRC | 05:04 | |
| *** zzzeek has joined #heat | 05:08 | |
| *** zzzeek has quit IRC | 05:09 | |
| *** zzzeek has joined #heat | 05:10 | |
| *** flg_ has joined #heat | 05:12 | |
| *** zzzeek has quit IRC | 05:16 | |
| *** kiennt has quit IRC | 05:19 | |
| *** zzzeek has joined #heat | 05:22 | |
| *** zzzeek has quit IRC | 05:23 | |
| *** prateek has joined #heat | 05:25 | |
| openstackgerrit | Rico Lin proposed openstack/heat master: Support tenacity exponential backoff retry on resource sync https://review.openstack.org/328613 | 05:26 |
| openstackgerrit | Rico Lin proposed openstack/heat master: Add RabbitMQ expiration policies https://review.openstack.org/357045 | 05:26 |
| openstackgerrit | Rico Lin proposed openstack/heat master: Add function descriptions for tempest plugin https://review.openstack.org/446614 | 05:30 |
| openstackgerrit | Rico Lin proposed openstack/heat master: Add test for basic resources as heat define test https://review.openstack.org/454222 | 05:31 |
| openstackgerrit | Rico Lin proposed openstack/heat-templates master: Add create magnum cluster resources example https://review.openstack.org/455673 | 05:31 |
| openstackgerrit | Rico Lin proposed openstack/heat-templates master: Fix undefined basestring and unicode in py35 https://review.openstack.org/456080 | 05:32 |
| openstackgerrit | Rico Lin proposed openstack/heat master: Remove unrequired check None in bay resource https://review.openstack.org/457864 | 05:32 |
| *** yangyape_ has joined #heat | 05:33 | |
| openstackgerrit | Rico Lin proposed openstack/heat master: Add unit test for translate with no translate value https://review.openstack.org/460405 | 05:34 |
| *** yangyapeng has quit IRC | 05:35 | |
| openstackgerrit | Rabi Mishra proposed openstack/heat master: Add httpd documentation and sample configs https://review.openstack.org/478346 | 05:35 |
| openstackgerrit | Rabi Mishra proposed openstack/heat master: Use keystone session with heatclient in integration tests https://review.openstack.org/435667 | 05:35 |
| openstackgerrit | Rabi Mishra proposed openstack/heat master: Revert "Revert "Enable heat to use uwsgi with devstack"" https://review.openstack.org/478345 | 05:35 |
| *** mdnadeem has joined #heat | 05:37 | |
| *** zzzeek has joined #heat | 05:40 | |
| *** zzzeek has quit IRC | 05:40 | |
| *** rcernin has joined #heat | 05:46 | |
| *** skramaja has joined #heat | 05:47 | |
| openstackgerrit | Ethan Lynn proposed openstack/heat master: Override ssl options for heatclient in RemoteStack https://review.openstack.org/480923 | 05:55 |
| *** kiennt has joined #heat | 05:56 | |
| openstackgerrit | Rico Lin proposed openstack/heat master: Add Heat review dashboard config files https://review.openstack.org/459685 | 05:57 |
| *** jprovazn has joined #heat | 05:59 | |
| *** zzzeek has joined #heat | 06:02 | |
| *** yangyape_ has quit IRC | 06:10 | |
| *** yangyapeng has joined #heat | 06:11 | |
| *** yangyape_ has joined #heat | 06:13 | |
| *** SmearedBeard has joined #heat | 06:14 | |
| *** yangyapeng has quit IRC | 06:16 | |
| *** zzzeek has quit IRC | 06:17 | |
| *** yangyape_ has quit IRC | 06:30 | |
| *** yangyapeng has joined #heat | 06:30 | |
| *** yangyapeng has quit IRC | 06:32 | |
| *** yangyapeng has joined #heat | 06:33 | |
| *** yangyapeng has quit IRC | 06:33 | |
| *** yangyapeng has joined #heat | 06:34 | |
| *** beagles has quit IRC | 06:37 | |
| *** htruta has quit IRC | 06:38 | |
| *** partner has quit IRC | 06:39 | |
| *** partner has joined #heat | 06:40 | |
| *** htruta has joined #heat | 06:42 | |
| *** b3nt_pin has joined #heat | 06:43 | |
| *** b3nt_pin is now known as Guest9640 | 06:43 | |
| *** LanceHaig has joined #heat | 06:55 | |
| *** LanceHaig has joined #heat | 06:55 | |
| *** LanceHaig has quit IRC | 06:56 | |
| *** LanceHaig has joined #heat | 06:57 | |
| *** LanceHaig has joined #heat | 06:57 | |
| *** LanceHaig_ has joined #heat | 06:58 | |
| *** LanceHaig has quit IRC | 06:58 | |
| *** ramishra has quit IRC | 06:58 | |
| *** ramishra has joined #heat | 07:00 | |
| *** jtomasek has joined #heat | 07:04 | |
| *** shardy has joined #heat | 07:06 | |
| *** kiennt has quit IRC | 07:13 | |
| *** kiennt has joined #heat | 07:13 | |
| *** jtomasek has quit IRC | 07:18 | |
| *** SmearedBeard has quit IRC | 07:21 | |
| *** ioggstream has joined #heat | 07:30 | |
| *** gfidente has joined #heat | 07:41 | |
| *** ccamacho has joined #heat | 07:42 | |
| openstackgerrit | Harald Jensås proposed openstack/heat master: Exclude updated networks - match actual interface state https://review.openstack.org/471969 | 07:52 |
| *** jtomasek has joined #heat | 07:55 | |
| *** zzzeek_ has quit IRC | 08:00 | |
| *** zzzeek has joined #heat | 08:00 | |
| *** jtomasek has quit IRC | 08:25 | |
| *** yangyap__ has joined #heat | 08:25 | |
| *** yangyapeng has quit IRC | 08:26 | |
| openstackgerrit | Harald Jensås proposed openstack/heat master: Exclude updated networks - match actual interface state https://review.openstack.org/471969 | 08:31 |
| *** elynn has quit IRC | 08:34 | |
| *** dixiaoli has quit IRC | 08:45 | |
| *** dixiaoli has joined #heat | 08:45 | |
| *** faizy has quit IRC | 09:04 | |
| *** faizy has joined #heat | 09:05 | |
| ricolin | ramishra, about https://review.openstack.org/#/c/461302/7/heat/api/openstack/v1/stacks.py | 09:14 |
| ricolin | if we put it in engine/api.py#30 than it will not be a update only parameter:) | 09:14 |
| *** yangyapeng has joined #heat | 09:19 | |
| *** yangyap__ has quit IRC | 09:22 | |
| *** gkadam_ has joined #heat | 09:25 | |
| *** gkadam is now known as Guest24494 | 09:25 | |
| *** gkadam_ is now known as gkadam | 09:25 | |
| *** Shunli has quit IRC | 09:29 | |
| *** dixiaoli_ has joined #heat | 09:42 | |
| *** dixiaoli_ has quit IRC | 09:42 | |
| *** dixiaoli has quit IRC | 09:46 | |
| *** openstackgerrit has quit IRC | 09:48 | |
| *** gkadam_ has joined #heat | 10:03 | |
| *** gkadam has quit IRC | 10:05 | |
| *** gkadam__ has joined #heat | 10:10 | |
| *** jmlowe_ has joined #heat | 10:10 | |
| *** jmlowe has quit IRC | 10:11 | |
| *** gkadam_ has quit IRC | 10:12 | |
| ramishra | sorry ricolin, was away. I've replied on the review. Hope that makes sense now;) | 10:30 |
| ioggstream | hi @all | 10:31 |
| *** kiennt has quit IRC | 10:38 | |
| ioggstream | I'm trying to replace a failed node in a stack, but heat doesn't do nothing | 11:09 |
| *** ccamacho is now known as ccamacho|lunch | 11:34 | |
| *** zhurong has joined #heat | 11:40 | |
| *** yangyapeng has quit IRC | 11:44 | |
| *** zhurong has quit IRC | 11:46 | |
| *** openstackgerrit has joined #heat | 11:49 | |
| openstackgerrit | James Slagle proposed openstack/heat master: Make deployment_swift_data updateable https://review.openstack.org/481288 | 11:49 |
| *** rwsu has quit IRC | 12:05 | |
| *** rwsu has joined #heat | 12:12 | |
| openstackgerrit | Harald Jensås proposed openstack/heat master: Exclude updated networks - match actual interface state https://review.openstack.org/471969 | 12:13 |
| *** ccamacho|lunch is now known as ccamacho | 12:14 | |
| *** shardy has quit IRC | 12:18 | |
| *** faizy has quit IRC | 12:28 | |
| *** skramaja has quit IRC | 12:29 | |
| *** catintheroof has joined #heat | 12:40 | |
| *** dprince has joined #heat | 12:42 | |
| *** chlong_ has quit IRC | 12:43 | |
| ioggstream | this is the heat issue I got https://pastebin.com/ZLdPuuJE . now I'm try and check if markin the server as unhealthy helps heat to recreate it | 12:44 |
| *** fragatina has joined #heat | 12:52 | |
| *** fragatina has quit IRC | 12:57 | |
| *** gkadam__ has quit IRC | 12:58 | |
| *** links has quit IRC | 13:06 | |
| *** mdnadeem has quit IRC | 13:14 | |
| *** mdnadeem has joined #heat | 13:14 | |
| *** zhurong has joined #heat | 13:23 | |
| ioggstream | I found this interview of zaneb https://www.rdoproject.org/blog/2017/03/zane-bitter-openstack-heat-openstack-ptg-atlanta/ and it worked | 13:28 |
| ioggstream | though there's no mention of "mark unhealthy" on RHOSP10 docs | 13:28 |
| ioggstream | now I'm checking if there are issues in the stack | 13:29 |
| *** blomquisg_gone is now known as blomquisg | 13:38 | |
| openstackgerrit | Harald Jensås proposed openstack/heat master: Exclude updated networks - match actual interface state https://review.openstack.org/471969 | 13:54 |
| *** ebarrera has joined #heat | 14:01 | |
| ebarrera | ping heat, is there any document explaining how resource-mark-unhealty works around ? can someone point it to me ? :) | 14:02 |
| openstackgerrit | Harald Jensås proposed openstack/heat master: Exclude updated networks - match actual interface state https://review.openstack.org/471969 | 14:03 |
| ebarrera | not sure if when marked it triggers an update or the update should be triggered separatelly | 14:04 |
| *** rubasov has quit IRC | 14:04 | |
| *** jcoufal has joined #heat | 14:07 | |
| zaneb | ioggstream: could it be this? https://bugzilla.redhat.com/show_bug.cgi?id=1430753 | 14:07 |
| openstack | bugzilla.redhat.com bug 1430753 in openstack-heat "Heat not able to maintain loadbalancer minimum member count" [Medium,Assigned] - Assigned to zbitter | 14:07 |
| zaneb | ebarrera: trigger the update separately. all mark-unhealthy does is change the state of the resource to CHECK_FAILED | 14:08 |
| *** hongbin has joined #heat | 14:08 | |
| *** chlong_ has joined #heat | 14:09 | |
| ebarrera | zaneb, so the check_failed triggers the update or I should send the signal ? | 14:09 |
| zaneb | CHECK_FAILED is just a state, like UPDATE_COMPLETE | 14:10 |
| zaneb | if you want to update the stack, do a stack update | 14:10 |
| zaneb | when you do the update, any resources in a FAILED state will get replaced | 14:11 |
| zaneb | so mark-unhealthy puts things in a state where they will get replaced on the next update, and stack update actually does the update that replaces them | 14:12 |
| *** prateek has quit IRC | 14:14 | |
| openstackgerrit | Harald Jensås proposed openstack/heat master: Exclude updated networks - match actual interface state https://review.openstack.org/471969 | 14:17 |
| *** felipemonteiro_ has joined #heat | 14:19 | |
| *** askb has quit IRC | 14:21 | |
| *** felipemonteiro__ has joined #heat | 14:21 | |
| *** felipemonteiro_ has quit IRC | 14:24 | |
| *** zhurong has quit IRC | 14:26 | |
| *** zhurong has joined #heat | 14:27 | |
| *** mdnadeem has quit IRC | 14:31 | |
| *** zhurong has quit IRC | 14:31 | |
| *** mdnadeem has joined #heat | 14:32 | |
| *** mdnadeem has quit IRC | 14:32 | |
| *** dyasny has quit IRC | 14:33 | |
| ioggstream | ebarrera: hi, that's me :D | 14:33 |
| ebarrera | ioggstream, sorry ? :D | 14:34 |
| ioggstream | ebarrera: the bz ;) | 14:34 |
| ioggstream | I filed it | 14:34 |
| ebarrera | ioggstream, 01885869 ? | 14:34 |
| ioggstream | ebarrera: y | 14:34 |
| ioggstream | ebarrera: now I'm checking if everything is fine in the stack, it's a complex one | 14:36 |
| ebarrera | :D so you know :) An stack update will trigger a replacement on failed resources :) | 14:36 |
| ioggstream | I expected stack update was able to replace it | 14:36 |
| ioggstream | luckily I found zane.b interview ebarrera with the mark unhealthy stuff | 14:37 |
| ioggstream | and tried it | 14:37 |
| ioggstream | ebarrera: I filed a tkt here too https://github.com/redhat-openstack/openshift-on-openstack/issues/369 as it still says that stack-update -x will suffice | 14:38 |
| ebarrera | zaneb, is one of ours :) | 14:38 |
| ioggstream | zane.b is one that rocks :D | 14:38 |
| ioggstream | OT: anybody at EuroPython next week? | 14:38 |
| ioggstream | more clear: reading docs I thought that stack check + stack update works. | 14:40 |
| *** dyasny has joined #heat | 14:51 | |
| *** catintheroof has quit IRC | 14:52 | |
| *** catintheroof has joined #heat | 14:53 | |
| *** catinthe_ has joined #heat | 14:56 | |
| *** catintheroof has quit IRC | 14:59 | |
| zaneb | ioggstream: it does, but beware of this horrible bug https://bugs.launchpad.net/heat/+bug/1671592 | 15:00 |
| openstack | Launchpad bug 1671592 in heat "Stack check causes entire ResourceGroup to be recreated" [Undecided,Fix released] - Assigned to Jason Dunsmore (jasondunsmore) | 15:00 |
| zaneb | which hasn't been fixed in newton because the gate is broken | 15:01 |
| ioggstream | hi zaneb, we've appliead the patch for that bug | 15:02 |
| zaneb | ok, good | 15:02 |
| ioggstream | zaneb: it have been backported for us :D | 15:02 |
| zaneb | did I do that? | 15:02 |
| ioggstream | but actually stack check + stack update didn't work | 15:02 |
| ioggstream | on newton | 15:02 |
| ioggstream | zaneb: I spoke with therve | 15:02 |
| ioggstream | that pointed me to the bug, then I filed a bz | 15:03 |
| ioggstream | zaneb: running "mark unhealthy" is compulsory on our RHOSP10. If you need some test pls let me kno | 15:04 |
| ioggstream | w and I'll do it | 15:04 |
| zaneb | records show that I did indeed do that backport :D | 15:05 |
| *** gfidente has quit IRC | 15:05 | |
| *** gfidente has joined #heat | 15:05 | |
| *** rcernin has quit IRC | 15:06 | |
| *** catintheroof has joined #heat | 15:06 | |
| ioggstream | zaneb: you mean the resourcegroup blasster backport one, right? | 15:06 |
| zaneb | yeah | 15:06 |
| zaneb | https://bugzilla.redhat.com/show_bug.cgi?id=1459854 | 15:06 |
| openstack | bugzilla.redhat.com bug 1459854 in openstack-heat "request backport of "Only recreate CHECK FAILED resources in ResourceGroup"" [High,Modified] - Assigned to zbitter | 15:06 |
| ioggstream | so no backports for skipping mark unhealthy | 15:06 |
| zaneb | ioggstream: I'm not clear on which patch you're referring to there? | 15:08 |
| ioggstream | zaneb: recap. to let stack update replace a dead node, I have to run mark unhealthy | 15:08 |
| ioggstream | zaneb: is that the expected behavior or stack update should replace it without marking? | 15:09 |
| zaneb | it's expected | 15:09 |
| *** catinthe_ has quit IRC | 15:09 | |
| ioggstream | ok, fine. | 15:09 |
| zaneb | stack update only compares the new vs. old template, it doesn't know about what has really happened | 15:10 |
| ioggstream | even stack check doesn't do the job, right? | 15:10 |
| zaneb | stack check should do it | 15:11 |
| ioggstream | nope | 15:11 |
| zaneb | if the nova server has been deleted then stack check should put it into CHECK_FAILED | 15:11 |
| ioggstream | yes | 15:11 |
| ioggstream | stack-check + stack-update don't replaces | 15:12 |
| ioggstream | I had to do: stack-check + mark unhealthy + stack-update | 15:12 |
| zaneb | and the nova server was deleted already? | 15:13 |
| ioggstream | yes | 15:13 |
| ioggstream | fyi zaneb 01885869 | 15:13 |
| zaneb | that's strange. stack-check + stack-update should have been sufficient | 15:13 |
| ioggstream | zaneb: so the tkt still stands | 15:13 |
| ioggstream | zaneb: ebarrera: is in charge of the ticket | 15:14 |
| ioggstream | I'm going to do some more test on that | 15:14 |
| ioggstream | (eg. deleting node-0 instead of node-1 and check if the new node has the correct number | 15:15 |
| *** zzzeek_ has joined #heat | 15:17 | |
| *** zzzeek_ has quit IRC | 15:17 | |
| *** jprovazn has quit IRC | 15:17 | |
| zaneb | ioggstream, ebarrera: first step to debugging would be to know what state all of the resources were in after stack-check (from e.g. openstack stack resource list -n 5 or something) | 15:18 |
| ebarrera | ioggstream, but dont delete the instance, If you delete it, it can't be in failed state | 15:22 |
| ioggstream | ebarrera: the issue is the following. It's an openshift node with a broken disk | 15:23 |
| ioggstream | ebarrera: the stack doesn't know it's in a failed state | 15:24 |
| ioggstream | ebarrera: I have to delete to let him know it's failed | 15:24 |
| ioggstream | ebarrera: attaching stack event list to the tkt | 15:28 |
| ioggstream | maybe it's useful | 15:29 |
| ebarrera | there is the resource-signal... | 15:29 |
| ebarrera | ioggstream, let me see | 15:29 |
| ioggstream | ebarrera: 2 min, uploading in course | 15:29 |
| ioggstream | ebarrera: done. how can I mark a failed (but working) node with resource signal ? | 15:32 |
| ebarrera | ioggstream, teorically ... but the syntax of the resource-signal is a bit obscure | 15:33 |
| *** catinthe_ has joined #heat | 15:35 | |
| zaneb | so if something is broken at the application level like this (i.e. *you* know it is broken but openstack has no way of knowing that), the easiest course of action is to mark the node unhealthy and then do a stack update | 15:35 |
| zaneb | you can delete the server from nova, and then do a stack-check, and then do a stack update, but I wouldn't recommend that route | 15:36 |
| *** catintheroof has quit IRC | 15:36 | |
| ioggstream | zaneb: I mark unhealthy and stack-update deletes the node? | 15:37 |
| zaneb | correct | 15:37 |
| ioggstream | I can just try it now :D | 15:37 |
| zaneb | well, replaces | 15:38 |
| *** ebarrera has quit IRC | 15:50 | |
| openstackgerrit | Harald Jensås proposed openstack/heat master: Exclude updated networks - match actual interface state https://review.openstack.org/471969 | 15:53 |
| *** yangyapeng has joined #heat | 15:54 | |
| ioggstream | zaneb: mark + update almost works. I'm trying to understand why runs a DELETE after the CREATE. It could be the heat template though | 16:02 |
| *** jmlowe_ has quit IRC | 16:05 | |
| zaneb | ioggstream: when replacing a resource Heat creates the replacement before deleting the original | 16:09 |
| *** ramishra has quit IRC | 16:19 | |
| *** catinthe_ has quit IRC | 16:19 | |
| *** catintheroof has joined #heat | 16:19 | |
| *** ramishra has joined #heat | 16:20 | |
| ioggstream | zaneb: so the issue is that the hostname is the same and that this triggers an rhel-unregister on the replaced node | 16:23 |
| zaneb | and that's bad because the disk is borked and it can't run the unregister? | 16:25 |
| ioggstream | zaneb: that's bad because unregisters the new node ;) | 16:25 |
| ioggstream | zaneb: the unregister is done here https://github.com/redhat-openstack/openshift-on-openstack/blob/master/infra.yaml#L498 | 16:27 |
| zaneb | hmm, that's not how it's supposed to work (it should unregister the _old_ node) | 16:28 |
| zaneb | oh, it's a deployment to a *different* server | 16:29 |
| ioggstream | zaneb: yes, I have to find a way to detect if the server is replaced or actually dropped | 16:30 |
| zaneb | yeah... that's an application-level problem ;) | 16:30 |
| ioggstream | zaneb: is there a way to inject there the deleted node uuid? | 16:31 |
| ioggstream | heat should know it, right? | 16:31 |
| zaneb | sure, {get_resource: host} | 16:31 |
| ioggstream | zaneb: so I could get the hostid from the heat metadata and check if matches or not | 16:32 |
| zaneb | hmm, but that will change unless the SoftwareDeployment is replaced | 16:32 |
| ioggstream | does it sound right? | 16:32 |
| ioggstream | yes | 16:32 |
| zaneb | which it normally is if the server it's being deployed on changes, but in this case you're deploying on a different server | 16:32 |
| ioggstream | the softwaredeployment is triggered with the old hostname, right? | 16:32 |
| ioggstream | sorry, old hostid | 16:32 |
| zaneb | that's where this comes in https://docs.openstack.org/heat/latest/template_guide/openstack.html#OS::Heat::SoftwareConfig-prop-inputs-*-replace_on_change | 16:33 |
| zaneb | if you use an input like ^ to ensure that the deployment is replaced when the server is replaced, then it will use the old hostid. otherwise... it shouldn't run at all actually (because the action is DELETE) | 16:36 |
| *** ramishra has quit IRC | 16:36 | |
| zaneb | ioggstream: are you marking just the server unhealthy, or the whole nested stack (infra.yaml)? | 16:36 |
| ioggstream | only the host | 16:36 |
| ioggstream | otherwise the whole servergroup will be replaced, right? | 16:36 |
| zaneb | you can replace only one member of the servergroup, or only the 'host' resource within that member | 16:39 |
| ioggstream | zaneb: right | 16:40 |
| ioggstream | do you have suggestions? | 16:40 |
| zaneb | if you're only marking the 'host' resource unhealthy, then it seems to me that deployment_bastion_node_cleanup shouldn't be running during the update | 16:41 |
| ioggstream | it's triggered by [host, wait_condition] | 16:41 |
| ioggstream | why shouldn'? | 16:41 |
| zaneb | depends_on doesn't trigger anything, it only controls ordering | 16:42 |
| zaneb | changes to the properties trigger updates | 16:42 |
| zaneb | changes to the properties that require replacement (only | 16:43 |
| zaneb | (only 'server' property in this case) trigger CREATE of new resource followed by DELETE of old one | 16:43 |
| zaneb | this resource has actions: ['DELETE'], so it should only run in that latter case | 16:44 |
| zaneb | yet the server isn't changing (because it's the bastion_node, not 'host') | 16:44 |
| *** dyasny has quit IRC | 16:45 | |
| zaneb | so no reason afacit for deployment_bastion_node_cleanup to be replaced, and therefore no reason for the deployment to be deployed | 16:45 |
| ioggstream | zaneb: I did't get it :( | 16:50 |
| ioggstream | zaneb: let me try to get: I mark the node unhealthy, then stack-update creates a new node and deletes the old one, right? | 16:51 |
| *** jmlowe has joined #heat | 16:52 | |
| zaneb | more precisely, when stack-update reaches that node in the traversal of the dependency graph it will create a replacement, and then it will do a traversal of the dependency graph in reverse and when it reaches that node again it will delete the old one | 16:53 |
| ioggstream | ok, | 16:53 |
| zaneb | https://github.com/zaneb/presentations/blob/heat-workflow/update.svg | 16:53 |
| ioggstream | zaneb: on the old node deletion, how is the deployment_bastion_node_delete triggered? | 16:55 |
| openstackgerrit | Rico Lin proposed openstack/heat master: Add converge flag in stack update for observing on reality https://review.openstack.org/461302 | 16:55 |
| zaneb | ioggstream: afaict it isn't. it _would_ be triggered if the deployment_bastion_node_cleanup were also replaced | 16:56 |
| zaneb | for the same reason, on the reverse traversal it would find the old deployment_bastion_node_cleanup resource and delete it, which causes the software deployment to be deployed (since the action is DELETE) | 16:57 |
| ioggstream | zaneb: ok, now it's clear :D | 17:00 |
| ioggstream | sorry for being sloppy | 17:00 |
| ioggstream | :( | 17:00 |
| zaneb | ideally users should never have to think about this stuff | 17:00 |
| ioggstream | so it's the same. I attach the nova uuid to the old delete resource | 17:01 |
| ioggstream | and run it only if the server uuid is the same | 17:01 |
| ioggstream | the logic is symmetric to the one I proposed | 17:01 |
| ioggstream | before | 17:01 |
| ioggstream | right? | 17:01 |
| ioggstream | zaneb: ideally openstack would work without us. actually there's somebody paying our wages | 17:02 |
| ioggstream | ;) | 17:02 |
| ioggstream | now I'm updating the deployment_... resource | 17:02 |
| zaneb | the same as what? | 17:03 |
| zaneb | (the server uuid is the same) as what? | 17:04 |
| ioggstream | zaneb: the server uuid is different. | 17:04 |
| zaneb | right, but how can you compare it? | 17:05 |
| ioggstream | zaneb: the logic is the same: if the server uuid attached to the deployment_delete is different from the current running host id | 17:05 |
| ioggstream | in the script | 17:05 |
| ioggstream | the script will still be executed on bastion | 17:05 |
| ioggstream | but will check if the target host is actually the same one | 17:05 |
| zaneb | ok, so the bastion knows what the currently running host id is already? | 17:05 |
| ioggstream | or not | 17:05 |
| ioggstream | yes, the script has the nova id attached | 17:06 |
| ioggstream | better, I'm trying to attach the nova id to the deployment_cleanup. it's done at creation | 17:06 |
| ioggstream | right? | 17:06 |
| zaneb | sort-of. but no software deployment happens at creation | 17:09 |
| zaneb | because the only action is DELETE | 17:09 |
| *** jmlowe has quit IRC | 17:10 | |
| ioggstream | zaneb: not the deployment, the SoftwareConfig | 17:10 |
| ioggstream | the uuid is stored there, right? | 17:10 |
| ioggstream | one per node | 17:10 |
| ioggstream | iiuc | 17:10 |
| *** ccamacho has quit IRC | 17:11 | |
| *** ccamacho has joined #heat | 17:11 | |
| zaneb | https://github.com/redhat-openstack/openshift-on-openstack/blob/master/infra.yaml#L483 <- that's the Software Config there? | 17:12 |
| zaneb | I don't see the host referenced in that at all | 17:12 |
| ioggstream | zaneb: I added it | 17:12 |
| ioggstream | ;) | 17:12 |
| ioggstream | and stackupdate | 17:12 |
| ioggstream | so now all nodes have it | 17:12 |
| ioggstream | can I preview that ? | 17:12 |
| zaneb | ok, I wouldn't have done that ;) | 17:12 |
| ioggstream | zaneb: ??? | 17:13 |
| ioggstream | is that wrong? | 17:13 |
| *** fragatina has joined #heat | 17:13 | |
| *** gmoro_ has joined #heat | 17:14 | |
| ioggstream | ok, I can preview add ing in stack output | 17:14 |
| zaneb | well, any change to a SoftwareConfig causes it to be replaced. so it gets a new ID. which will cause the deployment to get replaced. which means the delete deployment really will run, when otherwise it shouldn't have | 17:14 |
| zaneb | of course you are saying that it was running anyway, but I don't know why so it's hard to give good advice | 17:15 |
| *** gmoro has quit IRC | 17:15 | |
| ioggstream | zaneb: no problem, I'm playing with anotehr stack | 17:17 |
| zaneb | what I'm saying is that if you reference the nova ID in the sw *config*, then you'll end up running deployment_bastion_node_cleanup whenever the server is replaced | 17:19 |
| openstackgerrit | Rico Lin proposed openstack/python-heatclient master: Add `--converge` argument for osc stack update https://review.openstack.org/461305 | 17:19 |
| ioggstream | zaneb: it's already that way | 17:19 |
| zaneb | right, and I can't explain why | 17:19 |
| ioggstream | zaneb: if you replace the server you should unregister from rhn | 17:20 |
| ioggstream | and register the new one | 17:20 |
| zaneb | agree, but does deployment_bastion_node_cleanup do that? | 17:20 |
| ioggstream | yes | 17:20 |
| ioggstream | unregister from rhn | 17:20 |
| zaneb | that's... surprising | 17:20 |
| zaneb | I'd have expected a deployment on the node that is being deleted to do that, rather than one on the bastion node | 17:21 |
| ioggstream | the problem is, as the old and new server have the same hostname, the old cleanup connects to the new one | 17:21 |
| ioggstream | and unregister | 17:21 |
| ioggstream | and run ansible to remove it from openshfit | 17:21 |
| ioggstream | cluster | 17:21 |
| ioggstream | there's ansible taking care of node removal | 17:22 |
| zaneb | oh, ok, so you're running everything through ansible on the bastion node | 17:22 |
| zaneb | that makes sense | 17:22 |
| ioggstream | zaneb: it's how the templates are made | 17:23 |
| ioggstream | I didn't wrote it, but I think they are quite clever | 17:23 |
| zaneb | yeah, I wasn't familiar with the architecture | 17:23 |
| ioggstream | I'm just modifying them and filing fixing bugs | 17:23 |
| zaneb | it makes sense given that there is an ansible installer, but there will be impedance mismatches like this | 17:24 |
| ioggstream | yes, I'm fixing those corner cases | 17:24 |
| zaneb | so yeah, hostnames are not a good way of uniquely identifying a server | 17:24 |
| zaneb | IP address would be better | 17:24 |
| ioggstream | openstack/dhcp | 17:24 |
| ioggstream | that would be an issue with openshift | 17:25 |
| ioggstream | kube idendifies resource via nova hostnames | 17:25 |
| ioggstream | to be dhcp resilient | 17:25 |
| *** jmlowe has joined #heat | 17:26 | |
| *** links has joined #heat | 17:26 | |
| zaneb | then you should let heat choose the hostnames :) | 17:26 |
| zaneb | kube isn't the issue here though, it's ansible right? | 17:27 |
| *** LanceHaig has joined #heat | 17:28 | |
| *** LanceHaig_ has quit IRC | 17:28 | |
| ioggstream | zaneb: it's the script. It should only be run if the actual node uuid matches the resource one | 17:29 |
| *** links has quit IRC | 17:29 | |
| zaneb | well, I'd think that you'd still want to deregister the old node from RHN even if it has been replaced by a new one, no? | 17:30 |
| zaneb | but using the hostname will give you the wrong one | 17:30 |
| ioggstream | zaneb: they have the same hostname | 17:30 |
| ioggstream | iiuc rhn will just understand that and deregister | 17:31 |
| ioggstream | on rhn I can see only one active node with the given hostname | 17:31 |
| *** links has joined #heat | 17:31 | |
| zaneb | right, so use the IP address instead to find the one for ansible to deregister | 17:31 |
| zaneb | ah, ok. I'm not sure how RHN handles it tbh | 17:32 |
| ioggstream | still I have to attach the ipaddress to the softwareconfig | 17:32 |
| ioggstream | zaneb: if I rhn two nodes with the same ip and one can't contact rhn it becomes "red" | 17:32 |
| ioggstream | :D | 17:32 |
| ioggstream | can't cheat on rhn :D you need a proxy | 17:33 |
| *** links has quit IRC | 17:36 | |
| zaneb | you'd add an input to the software config but pass the actual IP address value to the deployment | 17:38 |
| *** fragatina has quit IRC | 17:42 | |
| *** fragatina has joined #heat | 17:43 | |
| ioggstream | zaneb: yes, that's fine too. replacing softwareconfig shouldn't trigger softwaredeployment replacement iiuc | 17:43 |
| *** gfidente has quit IRC | 17:43 | |
| ioggstream | https://docs.openstack.org/heat/latest/template_guide/openstack.html#OS::Heat::SoftwareDeployment-prop-config | 17:44 |
| *** bnemec is now known as beekneemech | 17:45 | |
| zaneb | oh, you're right | 17:45 |
| zaneb | I totally misremembered that | 17:45 |
| ioggstream | zaneb: np ;) you were great to let me understand all of that | 17:46 |
| ioggstream | working on that templates I learnt a lot thx to you, therve and all the great heat staff | 17:46 |
| *** fragatina has quit IRC | 17:47 | |
| ioggstream | I'm going to EP next week, hope to meet some of you | 17:47 |
| *** fragatina has joined #heat | 17:51 | |
| *** fragatina has quit IRC | 17:51 | |
| zaneb | if therve is going he didn't mention it | 17:51 |
| zaneb | so you will probably be disappointed | 17:52 |
| *** fragatina has joined #heat | 17:52 | |
| openstackgerrit | Rico Lin proposed openstack/heat master: Allow only validate external resource template https://review.openstack.org/481713 | 18:01 |
| openstackgerrit | Rico Lin proposed openstack/heat-templates master: [WIP]Add example for using external resources https://review.openstack.org/417594 | 18:04 |
| openstackgerrit | Rico Lin proposed openstack/heat-templates master: Add example for using external resources https://review.openstack.org/417594 | 18:04 |
| *** bit_lySLH2uSZHed has joined #heat | 18:07 | |
| *** bit_lySLH2uSZHed has left #heat | 18:08 | |
| ioggstream | zaneb: therve is not going ;) | 18:09 |
| ioggstream | zaneb: gotta go now. for now thx1e+3 | 18:09 |
| ioggstream | really | 18:09 |
| zaneb | np | 18:09 |
| *** anton has quit IRC | 18:10 | |
| *** dyasny has joined #heat | 18:11 | |
| *** anton has joined #heat | 18:12 | |
| *** ioggstream has quit IRC | 18:14 | |
| ricolin | zaneb, I reply your comment in https://review.openstack.org/#/c/464032/2 :) | 18:16 |
| openstackgerrit | Zane Bitter proposed openstack/heat master: Add a functional test for nested get_attr functions https://review.openstack.org/480820 | 18:17 |
| openstackgerrit | Zane Bitter proposed openstack/heat master: Cache dep_attrs for all resources in definitions https://review.openstack.org/477046 | 18:17 |
| openstackgerrit | Zane Bitter proposed openstack/heat master: Implement all_dep_attrs() efficiently for get_attr https://review.openstack.org/477047 | 18:17 |
| openstackgerrit | Zane Bitter proposed openstack/heat master: Fix legacy updates with nested dep_attr functions https://review.openstack.org/480819 | 18:17 |
| openstackgerrit | Zane Bitter proposed openstack/heat master: Allow StackDefinition to calculate explicit dependencies https://review.openstack.org/481719 | 18:17 |
| openstackgerrit | Zane Bitter proposed openstack/heat master: Add a way to get the resources depended on by an output https://review.openstack.org/481720 | 18:17 |
| zaneb | ricolin: ok, +2 | 18:17 |
| ricolin | zaneb, thx | 18:19 |
| ricolin | zaneb, Also a very small fix https://review.openstack.org/#/c/481713/ | 18:21 |
| ricolin | zaneb, see you tomorrow! | 18:22 |
| *** ricolin has quit IRC | 18:22 | |
| *** chlong_ has quit IRC | 18:40 | |
| *** ayoung has quit IRC | 19:19 | |
| *** ioggstream has joined #heat | 19:37 | |
| *** anton has quit IRC | 19:44 | |
| *** ioggstream has quit IRC | 19:46 | |
| *** anton has joined #heat | 19:48 | |
| *** anton has quit IRC | 19:53 | |
| *** anton has joined #heat | 19:53 | |
| *** ayoung has joined #heat | 19:55 | |
| *** catinthe_ has joined #heat | 19:56 | |
| *** anton has quit IRC | 19:58 | |
| *** anton has joined #heat | 19:58 | |
| *** catintheroof has quit IRC | 19:59 | |
| *** jmlowe has quit IRC | 20:00 | |
| *** ioggstream has joined #heat | 20:15 | |
| *** catinthe_ has quit IRC | 20:39 | |
| *** dprince has quit IRC | 20:49 | |
| openstackgerrit | Zane Bitter proposed openstack/heat master: Add a functional test for nested get_attr functions https://review.openstack.org/480820 | 20:57 |
| openstackgerrit | Zane Bitter proposed openstack/heat master: Log unhandled exceptions in worker https://review.openstack.org/481757 | 20:57 |
| *** ioggstream has quit IRC | 21:10 | |
| *** catintheroof has joined #heat | 21:33 | |
| *** iurygregory has quit IRC | 21:41 | |
| *** iurygregory has joined #heat | 21:44 | |
| *** openstackgerrit has quit IRC | 21:47 | |
| *** openstackstatus has quit IRC | 21:56 | |
| *** openstack has joined #heat | 22:01 | |
| *** felipemonteiro__ has quit IRC | 22:07 | |
| *** openstackgerrit has joined #heat | 22:08 | |
| openstackgerrit | Zane Bitter proposed openstack/heat master: Log unhandled exceptions in worker https://review.openstack.org/481757 | 22:08 |
| *** zzzeek_ has joined #heat | 22:08 | |
| *** zzzeek_ has quit IRC | 22:09 | |
| *** zzzeek_ has joined #heat | 22:14 | |
| *** zzzeek_ has quit IRC | 22:14 | |
| *** zzzeek_ has joined #heat | 22:14 | |
| *** zzzeek_ has quit IRC | 22:15 | |
| *** zzzeek_ has joined #heat | 22:19 | |
| *** askb has joined #heat | 22:21 | |
| *** zzzeek_ has quit IRC | 22:24 | |
| *** zzzeek_ has joined #heat | 22:29 | |
| *** zzzeek_ has quit IRC | 22:30 | |
| *** zzzeek_ has joined #heat | 22:31 | |
| *** zzzeek- has joined #heat | 22:33 | |
| *** zzzeek_ has quit IRC | 22:35 | |
| *** zzzeek- has quit IRC | 22:41 | |
| *** catintheroof has quit IRC | 22:42 | |
| *** flg_ has quit IRC | 22:47 | |
| *** askb has quit IRC | 22:49 | |
| openstackgerrit | Zane Bitter proposed openstack/heat master: Fix ResourceActionNotSupported errors in log https://review.openstack.org/481775 | 22:54 |
| *** jcoufal has quit IRC | 22:57 | |
| *** jmlowe has joined #heat | 23:07 | |
| *** zzzeek_ has joined #heat | 23:17 | |
| *** zzzeek_ has quit IRC | 23:21 | |
| *** zzzeek_ has joined #heat | 23:21 | |
| *** dbecker has quit IRC | 23:28 | |
| *** jmlowe has quit IRC | 23:48 | |
| *** jmlowe has joined #heat | 23:51 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
