Wednesday, 2022-11-09

*** yadnesh|away is now known as yadnesh05:28
*** yadnesh is now known as yadnesh|afk08:09
*** yadnesh|afk is now known as yadnesh08:53
QGHello all !15:23
QGDid you already have a member staying in pending_update due to 'taskflow.conductors.backends.impl_executor taskflow.exceptions.Duplicate: Atoms with duplicate names found: ['octavia-mark-member-active-indb-0129470e-40f8-4bce-9ed9-b5d4badc3f26']' ?15:23
johnsomOh! That is a coding error. What version of Octavia?15:25
QGWe are running Yoga version 15:25
johnsomCan you provide the log snippet at Logs back to the start of the call that raised this would be helpful.15:25
johnsomReally curious how that could happen given the UUID there. Is jobboard enabled in this deployment?15:26
QGyes jobboard is enabled 15:27
QGand we have also this patch :
gthiemongemaybe a task that is executed on 2 controllers at the same time?15:29
gthiemonge^ this commit should be backported15:30
johnsomMaybe..... That would be odd though. Typically that error means the flow compile failed, like the flow was built wrong. But yeah, I am wondering if this isn't a jobboard side effect we haven't seen yet.15:30
johnsomQG getting those logs would be super helpful so we can link them to a storyboard.15:31
QGyep i'm trying to anonymize the log15:32
johnsomAck, thanks!15:33
QGthis is the logs from the worker-0 15:42
QGand we have two workers15:42
gthiemonge"old='[]', new='[None]', updated='[]'" this is a bit weird15:43
QGthe other one at the same time was logging this :
gthiemongeupdated='['0129470e-40f8-4bce-9ed9-b5d4badc3f26', '0129470e-40f8-4bce-9ed9-b5d4badc3f26']15:46
gthiemongesame id15:46
QGso Octavia tried to update twice a member 15:47
gthiemongeQG: you don't have the content of the API call?15:49
johnsomgthiemonge Yeah, and that is the ID that is the problem. So, I think we just found the RCA on that bug.15:49
johnsomThat should have been de-duplicated15:50
QGgthiemonge: let me find it and upload it to paste.openstack.org15:50
gthiemongejohnsom: maybe the API call has updated twice the same member?15:51
johnsomyeah, but we should either de-dup that or throw them back an error.15:51
gthiemongeyes of course15:51
johnsomI vaguely remember a bug like this in the past. I'm searching to see if it was fixed, but not backported15:51
johnsomNope, I didn't find any patches, maybe it is just a story15:53
gthiemongebased on those logs, I'm going to try to reproduce it15:56
gthiemonge#startmeeting Octavia16:01
opendevmeetMeeting started Wed Nov  9 16:01:04 2022 UTC and is due to finish in 60 minutes.  The chair is gthiemonge. Information about MeetBot at
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:01
opendevmeetThe meeting name has been set to 'octavia'16:01
gthiemongeHi Folks16:01
gthiemonge#topic Announcements16:03
gthiemonge** Antelope-1 milestone16:03
gthiemongenext week is Antelope-1, we planned to get some RFEs merged by this milestone (amphora vertical scaling)16:04
gthiemongeso please review/test these commits16:05
tweiningif you need assistance wrt. testing feel free to ping me16:06
gthiemongetweining: thanks16:07
gthiemongeany other announcements folks?16:07
gthiemonge#topic CI Status16:08
gthiemongeno update this week16:09
gthiemonge#topic Brief progress reports / bugs needing review16:09
gthiemongeI haven't spent a lot of time on upstream stuff, I'm working on Octavia tests downstream, I hope I will focus more on upstream tasks next week16:10
johnsomI am still working on secrets consumers. It was a bigger job than I expected.16:10
gthiemonge#topic Open Discussion16:14
gthiemongeanything folks?16:15
tweiningno really16:16
johnsomNothing here16:16
gthiemongeok, that was a quick one16:17
gthiemongethank you!16:17
opendevmeetMeeting ended Wed Nov  9 16:17:04 2022 UTC.  Information about MeetBot at . (v 0.1.4)16:17
opendevmeetMinutes (text):
QGgthiemonge:  this is the call on api side : 16:18
gthiemongeQG: ok, we don't have the json params. I'm trying to forge an API call that triggers those duplicate member IDs16:21
tweiningbtw. did we miss to backport ?16:36
gthiemongewe didn't miss it :D we still have a lot of fixes from Z to backport, but there's also a long list of backports to review, I didn't want to create too many bp at the same time16:37
tweiningyeah, I see your comment above now16:37
*** yadnesh is now known as yadnesh|away16:42
gthiemongetaskflow.exceptions.Duplicate: Atoms with duplicate names found: ['octavia-mark-member-active-indb-039c16a3-5b2f-4385-aad2-a541288c10ca']16:43
gthiemongeOk I reproduce it16:43
gthiemongeQG: johnsom: so I think the only way to trigger it is to update the same member twice in a PUT call (like
johnsomyeah, so maybe missing validation that the call isn't updating the same member twice in the same call?17:02
QGgthiemonge: GG !17:06
gthiemongeQG: do you know which client/application sent this request?17:07
gthiemongebecause if we fix it, the API will return a 400 (ValidationException error)17:07
QGgthiemonge: Yep it's 17:08
QGHashiCorp Terraform/1.3.4 (+ Terraform Plugin SDK/2.10.1 gophercloud/2.0.0"17:08
QGuhhhh there is something i don't understand17:10
QGin the i put two lines,  one for the apache log and the other one for octavia-api but they are in fact one call17:11
QGlet me upload  more logs17:12
johnsomYeah, the request flows through those17:12
QGso i added the field wsgi for apache log17:18
johnsomWhat is your question?17:20
johnsomThis looks like a pretty normal terraform log17:20
QGfrom the logs there is no call for updating the same member twice in a PUT17:21
johnsomSo, this call: PUT /v2.0/lbaas/pools/a7fecc48-c125-4b9a-bff4-cbef08b6326e/members HTTP/1.1" status: 202 len: - time: 775(ms)17:22
QGI thought you deduced that from the fact that there was a duplicate log line when there was one for octavia-api and one for apache17:22
johnsomThis is a batch member update call. The JSON body of this PUT had the duplicate members in it17:22
johnsomOh, nope, it was this:17:22
johnsomupdated='['0129470e-40f8-4bce-9ed9-b5d4badc3f26', '0129470e-40f8-4bce-9ed9-b5d4badc3f26']17:23
QGoh ok thanks17:23
johnsomFrom this snippet:
QGdo you advice us to log the body also ?17:23
johnsomIt was the worker log17:23
gthiemongewhich is also the id of the duplicate atom17:23
johnsomGenerally you don't want to, it's a lot of data.17:24
johnsomThe worker will generally log what you need to know out of the request17:24
gthiemongebut terraform or gophercloud probably sends a invalid request17:24
johnsomYeah, it has had a history of issues. Like originally it didn't honor the 409's17:25
gthiemongeQG: do you have the logs (DEBUG) from terraform?
QGno unfortunately a customer did that17:28
gthiemongebecause here, it triggers a bug in octavia, and when we will have a fix, they will have a bug in terraform17:31
QGi will try to ask customer if he have the DEBUG logs from terraform17:35
johnsomI would argue updating the same member twice in one API call is a bug in terraform17:35

Generated by 2.17.3 by Marius Gedminas - find it at!