| opendevreview | Michael Johnson proposed openstack/octavia master: Add Barbican secrets consumers https://review.opendev.org/c/openstack/octavia/+/864308 | 00:19 |
|---|---|---|
| johnsom | That is all set now. We have the package version for the requirements. | 00:20 |
| johnsom | heat is broken and holding up the upper-constraints update for python-barbicanclient: | 00:20 |
| johnsom | https://review.opendev.org/c/openstack/requirements/+/873906 | 00:20 |
| opendevreview | Gregory Thiemonge proposed openstack/octavia-tempest-plugin master: WIP Testing spliting centos scenario job into 2 jobs https://review.opendev.org/c/openstack/octavia-tempest-plugin/+/874054 | 07:18 |
| opendevreview | Gregory Thiemonge proposed openstack/octavia-tempest-plugin master: WIP Add a testing job for Rocky Linux https://review.opendev.org/c/openstack/octavia-tempest-plugin/+/873490 | 12:03 |
| opendevreview | Omer Schwartz proposed openstack/octavia master: Fix hm operating status to ONLINE in single lb call https://review.opendev.org/c/openstack/octavia/+/868363 | 12:33 |
| opendevreview | Omer Schwartz proposed openstack/octavia master: Fix hm operating status to ONLINE in single lb call https://review.opendev.org/c/openstack/octavia/+/868363 | 13:10 |
| opendevreview | Omer Schwartz proposed openstack/octavia master: Fix hm operating status to ONLINE in single lb call https://review.opendev.org/c/openstack/octavia/+/868363 | 13:12 |
| opendevreview | Omer Schwartz proposed openstack/octavia master: Fix h2 listener and pool creation with any protocol https://review.opendev.org/c/openstack/octavia/+/857676 | 13:34 |
| skraynev | hi folks! could someone help with understanding the right way to debug the following issue: All resources except HM have provisioning state: PENDING_UPDATE state and also pool, memeber, LB have operating status ERROR states. | 14:13 |
| skraynev | also I could not find in logs any information about incoming API requests, which changed something for this LB or its inner resources. | 14:14 |
| skraynev | I met one old bug about similar behaviour after restart VMs (connected via members)... but looks like it's already fixed in Yoga release | 14:15 |
| skraynev | So where and what can I check to find a root cause? | 14:16 |
| gthiemonge | skraynev: hi, for the PENDING_UPDATE provisioning_status, check the logs to see if one service is not updating the LB | 14:23 |
| gthiemonge | the timeout before switching to ERROR in case of problems is sometimes really long, the LB looks stuck but it will eventually be in ERROR | 14:24 |
| gthiemonge | if you don't see any activity for this LB in the logs, search for an ERROR message in the same logs, or a potential restart of the service while the LB was being updated | 14:25 |
| gthiemonge | there's 2 reasons for a LB to be stuck in PENDING_UPDATE: a bug (with ERROR-level message+exception) or the octavia worker/hm/hk was not gracefully shutdown | 14:26 |
| skraynev | gthiemonge: thank you for explanation. do I understand right, that state PENDING_UPDATE could set only api or worker? | 14:44 |
| skraynev | and HM/HK could not do it? | 14:44 |
| gthiemonge | you missed on message: "gthiemonge> there's 2 reasons for a LB to be stuck in PENDING_UPDATE: a bug (with ERROR-level message+exception) or the octavia worker/hm/hk was not gracefully shutdown" | 14:47 |
| skraynev | and what do you mean under "the octavia worker/hm/hk was not gracefully shutdown"? I mean the full story of the case, like: LB/member/pool/listener was in update state and octavia service was restarted "unexpectedly" without waiting finishing processes (as you said not graceful shutdown)? | 14:48 |
| gthiemonge | in most of the cases, PENDING_UPDATE is set by the API when an action is performed on a LB, and it is unset by the worker when it's completed | 14:48 |
| gthiemonge | operators cannot change it, except by updating directly the DB | 14:48 |
| gthiemonge | 'octavia service was restarted "unexpectedly" without waiting finishing processes' yeah this is a potential root cause, when restarting octavia, we need to ensure that it finishes the tasks that it is running | 14:49 |
| skraynev | aha. I have seen this part of the code, where API "lock" resources by pending_update state. I am wondering, could it be some other place (except tasks an API). maybe in HM or somewhere else? | 14:49 |
| gthiemonge | during a failover triggered by the HM, the LB is locked in PENDING_UPDATE too | 14:50 |
| skraynev | 'octavia service was restarted "unexpectedly" without waiting finishing processes' --> got it. thx. | 14:50 |
| skraynev | during a failover triggered by the HM -> interesting. will it be rpc or some http request? and will it be logged? do I understand right, that it happens, when amphora does not respond the long period of time? | 14:52 |
| *** dkehn_ is now known as dkehn | 14:59 | |
| opendevreview | Gregory Thiemonge proposed openstack/octavia-tempest-plugin master: WIP Add a testing job for Rocky Linux https://review.opendev.org/c/openstack/octavia-tempest-plugin/+/873490 | 15:04 |
| gthiemonge | skraynev: it is logged in the health-manager logs, if an amphora does not send heartbeat messages, you see: https://opendev.org/openstack/octavia/src/commit/93a2b0a0c3c94f555de1b89aaf16aebb30a836fc/octavia/controller/healthmanager/health_manager.py#L136 | 15:06 |
| skraynev | gthiemonge: you are rock! big thank you :) | 15:07 |
| opendevreview | Omer Schwartz proposed openstack/octavia master: Fix hm operating status to ONLINE in single lb call https://review.opendev.org/c/openstack/octavia/+/868363 | 16:28 |
| opendevreview | Gregory Thiemonge proposed openstack/octavia-tempest-plugin master: WIP Add a testing job for Rocky Linux https://review.opendev.org/c/openstack/octavia-tempest-plugin/+/873490 | 18:17 |
| *** luksky_w9 is now known as luksky_w | 23:11 | |
| opendevreview | Michael Johnson proposed openstack/octavia master: Fix Octavia API HTTP Accept header handling https://review.opendev.org/c/openstack/octavia/+/865475 | 23:23 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!