Wednesday, 2021-08-25

*** tamas_erdei is now known as terdei06:57
opendevreviewGregory Thiemonge proposed openstack/octavia master: Preserve haproxy server states during reloads
opendevreviewGregory Thiemonge proposed openstack/octavia-tempest-plugin master: Add new scenario test to create LB in specific AZ
mnaserjohnsom: the current issue that i'm dealing with is that it seems my octavia workers are getting stuck for some reason.14:29
mnaserlike, absolutely no movement even though i have a bunch of PENDING_DELETE14:30
johnsommnaser Did they pop off the rabbit queue or did rabbit lose them?14:33
mnaserrabbit is fine, it's running all other openstack services with no issues, let me see if they did pop off though14:33
johnsomI assume you are running durable queues14:34
mnaser36 PENDING_DELETE lbs14:34
mnaserso they are in PENDING_DELETE state14:35
mnaserso i cant re-run a delete?14:35
johnsomThere are pretty much just two possibilities for that. 1. Rabbit lost the message.  2. The controller that was working on the delete got kill -9'd.14:35
mnasernone of these were up :(14:35
mnaseris there anyway without db to get out of this state now14:35
johnsomPENDING is a lock state meaning one of the controllers has "ownership" of the object. If none of the controllers are actively working (retrying), then I would set it to ERROR and retry the delete call.14:36
mnaserbut there's no nova equiv of reset-state ?14:36
johnsomNope. One of the controllers owns that object, so it is invalid to "reset" the state without making sure it's not actively being worked on.14:38
johnsomWe have made a lot of progress towards eliminating the #2 possibility though. It's just not quite ready yet.14:38
opendevreviewMerged openstack/octavia master: Fix pylint checks
rm_workYeah, there have been some requests to allow an admin call to reset the state to ERROR. Most of us have been united in the opinion that we should not do that. I've wavered a bit, and would probably cave if more pressure was applied, but I don't know if johnsom would. It really should be solved by the jobboard work.15:58
rm_workThe problem is how long that is taking. I go into the DB to do a state reset about once a month.15:58
gthiemonge#startmeeting Octavia16:01
opendevmeetMeeting started Wed Aug 25 16:01:08 2021 UTC and is due to finish in 60 minutes.  The chair is gthiemonge. Information about MeetBot at
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:01
opendevmeetThe meeting name has been set to 'octavia'16:01
gthiemongeHi everyone16:01
gthiemonge#topic Announcements16:02
gthiemongeXena-3 milestone16:03
gthiemongenext week is Xena-3 milestone (Feature Freeze, Final release for client libraries)16:03
gthiemongethe priority review etherpad is up to date:16:03
gthiemongewe have to focus on [feature] reviews16:03
gthiemongefor instance "Generic Network Interface Management" (wink wink)16:04
gthiemongeit is important for Centos/RHEL users, this commit is required for Centos/RHEL 9 amphora images.16:04
gthiemongejohnsom: rm_work: ^16:04
gthiemongeCentos Stream as well16:05
rm_workwhy does time keep moving16:07
gthiemongeI think Centos 9 Stream will be released... in 202116:07
rm_workanyway ok, will take a look16:07
johnsomIt is an issue on the debian side too16:07
gthiemongeamphorav2 is now the default driver!16:09
gthiemonge(amphora is now an alias for amphorav2)16:09
gthiemonge(amphorav1 still exists)16:09
gthiemongeThanks to all the people involved in this work!16:10
gthiemongenote that persistence is not enabled by default16:10
gthiemongeAny other announcements?16:12
gthiemonge#topic Brief progress reports / bugs needing review16:14
johnsomI have mostly been working on reviews16:15
gthiemongeI've been working on a issue with the status of members. It is not kept after reloading haproxy configuration16:15
gthiemongejohnsom: +116:15
gthiemongea member with an ERROR operating_status can be switched to ONLINE after reconfiguring a LB (ex: adding another member)16:15
gthiemongeI have a WIP patch that uses the server-state feature in haproxy:16:16
rm_workmmm yeah because it resets haproxy and it remakes the member table, so healthchecks needs to rerun16:16
rm_workthat seems expected16:16
rm_workoh so there is a way to solve that? neat16:16
gthiemongeIf you have some concerns about using server-state, please comment in the review16:16
gthiemongeYeah but I read that using this file can be time-consuming with older haproxy proxy releases, someone with 100k backends complained that it took 1h to launch haproxy :D16:17
gthiemongethis feature has been optimized since 2.116:18
gthiemongeso if there's a way to improve the user experience with hm status, that's great, but if it introduces some issues, ...16:19
gthiemonge#topic Open Discussion16:22
gthiemongeAny other topics today?16:22
gthiemongeThanks everyone!16:25
opendevmeetMeeting ended Wed Aug 25 16:25:13 2021 UTC.  Information about MeetBot at . (v 0.1.4)16:25
opendevmeetMinutes (text):
johnsomThanks Greg16:25
mnaserrm_work, johnsom: i mean we have that in nova and, nova's been around for a while17:11
johnsombugs and all17:11
mnaseryeah like reset state will always be a helpful tool to avoid going into the db17:12
mnaserin an ideal world yes it should never exist17:12
johnsomThe challenge is you have to check every controller first.17:13
mnaserjohnsom: same challenge as nova :)17:15
mnaserwell, setting them to ERROR and deleting them again fixed it17:27
mnaserbut the fact i have to login to db to do that is really not ideal :\17:27
rm_workyes :/18:09
rm_worksee above18:09

Generated by 2.17.2 by Marius Gedminas - find it at!