Wednesday, 2021-01-13

« Tuesday, 2021-01-12 Index Thursday, 2021-01-14 »
*** ianychoi has joined #openstack-lbaas00:14
*** ramishra has quit IRC00:38
*** ramishra has joined #openstack-lbaas00:40
*** kevinz has joined #openstack-lbaas01:03
*** icey has quit IRC01:15
*** icey has joined #openstack-lbaas01:16
*** tkajinam has quit IRC01:41
*** tkajinam has joined #openstack-lbaas01:42
*** armax has quit IRC01:47
*** armax has joined #openstack-lbaas02:08
*** tkajinam has quit IRC02:09
*** tkajinam has joined #openstack-lbaas02:10
*** armax has quit IRC02:10
openstackgerritLingxian Kong proposed openstack/octavia stable/victoria: Use 'bash' in the keepalived check script  https://review.opendev.org/c/openstack/octavia/+/76778702:11
*** zzzeek has quit IRC02:30
*** spatel has joined #openstack-lbaas02:31
*** zzzeek has joined #openstack-lbaas02:31
*** zzzeek has quit IRC02:36
*** zzzeek has joined #openstack-lbaas02:40
*** devfaz has quit IRC02:41
*** armax has joined #openstack-lbaas02:48
*** devfaz has joined #openstack-lbaas02:54
*** armax has quit IRC02:54
*** rcernin has quit IRC02:57
*** sapd1 has joined #openstack-lbaas02:58
*** sapd1 has quit IRC03:03
*** rcernin has joined #openstack-lbaas03:18
*** rcernin has quit IRC03:21
*** rcernin has joined #openstack-lbaas03:21
*** psachin has joined #openstack-lbaas03:34
*** sapd1 has joined #openstack-lbaas03:36
*** sapd1 has quit IRC04:00
*** sapd1 has joined #openstack-lbaas04:09
*** rcernin has quit IRC04:35
*** rcernin has joined #openstack-lbaas04:35
*** sapd1 has quit IRC05:01
*** vishalmanchanda has joined #openstack-lbaas05:11
*** rcernin_ has joined #openstack-lbaas05:42
*** rcernin has quit IRC05:42
*** sapd1 has joined #openstack-lbaas05:55
*** spatel has quit IRC05:58
*** gcheresh has joined #openstack-lbaas06:10
*** zzzeek has quit IRC07:28
*** rcernin_ has quit IRC07:28
*** ccamposr has joined #openstack-lbaas07:29
*** zzzeek has joined #openstack-lbaas07:31
*** ccamposr__ has quit IRC07:31
*** openstackgerrit has quit IRC07:47
*** devfaz_ has joined #openstack-lbaas07:51
*** devfaz has quit IRC07:52
*** devfaz has joined #openstack-lbaas07:59
*** devfaz_ has quit IRC08:00
*** rcernin_ has joined #openstack-lbaas08:06
*** openstackgerrit has joined #openstack-lbaas08:11
openstackgerritArieh Maron proposed openstack/octavia-tempest-plugin master: Updating _test_pool_CRUD to enable testing of updates to the  load balancer algorithm:  https://review.opendev.org/c/openstack/octavia-tempest-plugin/+/77016308:11
*** rpittau|afk is now known as rpittau08:25
*** rcernin_ has quit IRC08:26
*** dasp_ has quit IRC09:16
*** dasp has joined #openstack-lbaas09:17
openstackgerritArieh Maron proposed openstack/octavia-tempest-plugin master: Updating _test_pool_CRUD to enable testing of updates to the  load balancer algorithm:  https://review.opendev.org/c/openstack/octavia-tempest-plugin/+/77016311:13
*** sapd1 has quit IRC12:01
*** ramishra has quit IRC12:19
*** ramishra has joined #openstack-lbaas12:23
*** armax has joined #openstack-lbaas12:24
*** devfaz_ has joined #openstack-lbaas12:41
*** devfaz has quit IRC12:42
*** spatel has joined #openstack-lbaas12:47
openstackgerritArkady Shtempler proposed openstack/octavia-tempest-plugin master: Adding failover test. Send HTTP traffic while MASTER Amphorae is rebooted. BACKUP Amphorae should serve the traffic.  https://review.opendev.org/c/openstack/octavia-tempest-plugin/+/75161712:50
*** spatel has quit IRC12:52
*** vishalmanchanda has quit IRC13:01
openstackgerritArieh Maron proposed openstack/octavia-tempest-plugin master: Updating _test_pool_CRUD to enable testing of updates to the  load balancer algorithm:  https://review.opendev.org/c/openstack/octavia-tempest-plugin/+/77016313:47
*** vishalmanchanda has joined #openstack-lbaas14:35
*** TrevorV has joined #openstack-lbaas14:41
*** armax has quit IRC14:48
*** armax has joined #openstack-lbaas14:50
*** psachin has quit IRC15:28
johnsomhaleyb FYI, I put you on the agenda: https://wiki.openstack.org/wiki/Octavia/Weekly_Meeting_Agenda#Meeting_2021-01-1315:46
haleybjohnsom: thanks15:48
*** sapd1 has joined #openstack-lbaas15:59
johnsom#startmeeting Octavia16:00
openstackMeeting started Wed Jan 13 16:00:26 2021 UTC and is due to finish in 60 minutes.  The chair is johnsom. Information about MeetBot at http://wiki.debian.org/MeetBot.16:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:00
*** openstack changes topic to " (Meeting topic: Octavia)"16:00
openstackThe meeting name has been set to 'octavia'16:00
gthiemongehi16:00
johnsomHi everyone16:01
johnsomanyone?16:01
johnsomlol16:01
gthiemongeo/16:01
haleybo/16:01
johnsom#topic Announcements16:01
*** openstack changes topic to "Announcements (Meeting topic: Octavia)"16:01
johnsomAs contributing members of OpenStack, you should have received an e-mail for:16:02
johnsom2021 Individual Director Election and Bylaws Amendments16:02
johnsomPlease vote. They need 10% of the eligible people to vote to pass the amendments.16:02
johnsomAlso, this week is milestone 2 for Wallaby.16:04
johnsomOh, next week.16:04
johnsom#link https://releases.openstack.org/wallaby/schedule.html16:04
johnsomAnyway, it is close.16:04
johnsomRemember that MS3 is feature freeze, so if you have any features you want to get into Octavia, you need those patches posted super soon.16:05
johnsomAny other announcements today?16:05
johnsom#topic Brief progress reports / bugs needing review16:06
*** openstack changes topic to "Brief progress reports / bugs needing review (Meeting topic: Octavia)"16:06
johnsomI have been doing a few reviews and answering questions via the discuss e-mail list and in the channel. Otherwise my focus is on Designate things.16:06
johnsomNo other updates this week?16:08
gthiemongeWe still have 2 open reviews for the gate issues:16:08
*** ccamposr__ has joined #openstack-lbaas16:09
gthiemongehttps://review.opendev.org/c/openstack/octavia/+/770444 https://review.opendev.org/c/openstack/octavia/+/76934816:09
johnsom#link https://review.opendev.org/c/openstack/octavia/+/77044416:10
johnsom#link https://review.opendev.org/c/openstack/octavia/+/76934816:10
johnsomJust so it highlights in the meeting minutes summary16:10
johnsom#topic Removing lower constraints jobs (haleyb)16:10
*** openstack changes topic to "Removing lower constraints jobs (haleyb) (Meeting topic: Octavia)"16:10
johnsomhaleyb You have the floor16:11
haleyback16:11
haleybother projects have been proposing changes to their stable branches to remove the lower-constraints jobs16:11
haleybespecially after the fiasco of the past couple weeks16:12
*** ccamposr has quit IRC16:12
haleybi know that we have found the job useful, but given the amount of time we've spent fixing stable, should we disable it there?16:12
haleybor on just really old stable branches?  looking for opinions16:13
haleyb#link https://review.opendev.org/c/openstack/octavia/+/770476 - a victoria WIP16:13
johnsomYeah, I know we put a lot of effort into setting those up and getting them working correctly. It was triggered by issues with CentOS 7 shipping OLD python packages.16:14
johnsomSo, IMO they have value.... That said, no one is managing that list effectively.16:14
johnsomAs for stable branches. I agree that the requirements should be fixed on stable, so they should not be needed in theory.16:15
haleybright.  and i don't think we should remove on master, as it could find an issue16:15
johnsomThough recent events have messed with some stable branch requirements.16:16
haleybjohnsom: the alternative would have been to disable the l-c job i guess16:17
johnsomnon-voting?16:18
haleybyeah, could do that too, and after your earlier comment we can leave the files and tox.ini changes if you like, although most repos i looked at removed everything16:19
johnsomLol, most patches I saw left the l-c.txt file, which didn't make any sense to me.16:20
johnsomWell, I think the risk is low on stable branch, so I guess I am ok with that.16:20
johnsomReally, this just shifts the burden to downstream testing.16:21
* haleyb wonders if there are other opinions16:22
* johnsom wonders as well16:22
johnsomWell, I guess patches talk louder than IRC meeting chat....16:25
johnsom#topic Open Discussion16:25
haleybthe crickets win :)16:25
*** openstack changes topic to "Open Discussion (Meeting topic: Octavia)"16:25
johnsomAny other topics today?16:25
haleybi had one16:26
haleyb#link https://review.opendev.org/c/openstack/octavia/+/77017616:26
haleybbackport chain of an empty UDP pool fix16:26
johnsomZuul does not like you16:27
*** ccamposr has joined #openstack-lbaas16:27
haleybi have a dependent patch that can't merge until that gets to stein16:27
haleybthe grenade failure there should be fixed soon i hope with greg's change16:27
haleyband a somewhat related change is the one enabling the stable/victoria tempest tests16:28
haleyb#link https://review.opendev.org/c/openstack/octavia-tempest-plugin/+/75521016:28
johnsomOk, sounds good. Thanks for working on backports.16:28
johnsomIt would be good if someone can drive some stable branch releases as well. I think we are way behind on some of those.16:29
haleyb^^ that one will add more jobs but not having stable/victoria tested seems bad16:29
*** ccamposr__ has quit IRC16:29
johnsomWe were trying to get a bunch of stuff merged for the releases. I'm just not sure who is driving / tracking that now16:29
gthiemongeI'll try to rebase/recheck the backports when the gate fixes are merged16:30
haleybyes, i think we were waiting for the gates to get green16:30
johnsomOk, any other topics today?16:32
johnsomOk then, thanks everyone!16:34
johnsom#endmeeting16:34
*** openstack changes topic to "Discussions for OpenStack Octavia | Priority bug review list: https://etherpad.openstack.org/p/octavia-priority-reviews"16:34
openstackMeeting ended Wed Jan 13 16:34:14 2021 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:34
openstackMinutes:        http://eavesdrop.openstack.org/meetings/octavia/2021/octavia.2021-01-13-16.00.html16:34
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/octavia/2021/octavia.2021-01-13-16.00.txt16:34
gthiemongethanks johnsom16:34
openstackLog:            http://eavesdrop.openstack.org/meetings/octavia/2021/octavia.2021-01-13-16.00.log.html16:34
*** mchlumsky8 has joined #openstack-lbaas16:41
*** mchlumsky has quit IRC16:42
*** mchlumsky8 is now known as mchlumsky16:42
openstackgerritMerged openstack/octavia stable/ussuri: Fix Ussuri requirements for Victoria grenade  https://review.opendev.org/c/openstack/octavia/+/77044416:53
*** rpittau is now known as rpittau|afk18:05
*** xgerman has joined #openstack-lbaas18:08
*** gcheresh has quit IRC18:56
*** gcheresh has joined #openstack-lbaas19:37
*** ccamposr__ has joined #openstack-lbaas20:11
*** ccamposr has quit IRC20:14
openstackgerritLingxian Kong proposed openstack/octavia stable/victoria: Use 'bash' in the keepalived check script  https://review.opendev.org/c/openstack/octavia/+/76778720:30
*** adeberg has quit IRC20:53
openstackgerritMerged openstack/octavia stable/train: Fix lower-constraints & requirements  https://review.opendev.org/c/openstack/octavia/+/76934821:06
*** vishalmanchanda has quit IRC21:41
*** AlexStaf has quit IRC21:50
*** TrevorV has quit IRC21:50
openstackgerritBrian Haley proposed openstack/octavia master: Add SCTP support in Amphora  https://review.opendev.org/c/openstack/octavia/+/75324721:52
*** gcheresh has quit IRC21:57
*** rcernin has joined #openstack-lbaas21:59
*** AlexStaf has joined #openstack-lbaas22:01
*** spatel has joined #openstack-lbaas22:02
spateljohnsom: hey!22:02
johnsomspatel Hi22:02
spatelmy LB is in PENDING_CREATE (since long time)22:02
spateltrying to delete but not letting me do that22:03
spatelany workaround ?22:03
johnsomYeah, that means a controller has it locked and owns it.22:03
johnsomDid you kill -9 a controller while it was working on that load balancer?22:03
spatelkill -9 a controller?22:04
spateli am not following you22:04
johnsomYeah, like the Octavia worker process, or non-graceful shutdown the container, or pull the power on the compute host?22:04
spateli have 3 controller node, do you think restart service would be fine?22:05
johnsomAlso, that could mean an issue with your rabbitmq22:05
spateli can build VM etc.. fine..22:05
johnsomGraceful restarts are fine, non-graceful can lead to things in PENDING_*22:05
spateli think its networking issue22:05
spatelwhy don't octavia has some kind of timeout if it stall then ERROR out instead trying hard22:06
johnsomOk, so first thing to do is tail all three Octavia worker logs. If it's retrying actions on that load balancer, it should be writing out to the log.22:06
spatellet me find out22:06
johnsomspatel It does have that timeout. The only way it doesn't finish is a kill -9 of the process22:07
spateljohnsom: one of the controller i am seeing - http://paste.openstack.org/show/801611/22:10
spateljohnsom: finally it error out itself and i am able to delete it22:11
johnsomYeah, so that is probably it. Look at the openstack loadbalancer amphora list output, find the load balancer, see if one of the amphora associated with your PENDING_CREATE loadbalancer is using 10.62.5.83 as the lb-mgmt-net IP22:11
johnsomYeah, ok, so a controller was still retrying to create the load balancer, thus it was in PENDING_CREATE.22:12
johnsomThere are configuration settings to turn how long it retries before giving up and marking it ERROR, should you want it to not try so hard.22:12
spateljohnsom: i like to have short time :) let me try to see what went wrong22:13
spatelthank you22:13
johnsomSome people argue it should try forever, so... that is why it's configurable. grin22:14
mchlumskyOur users complain that they can't do anything with the LBs while they are PENDING_* while they can at least delete and re-create them when they are in ERROR. You have to find the right timeout balance between trying hard enough and empowering users.22:46
johnsomYep22:53
johnsomPersonally, I lean towards shorter timeouts. Typically if you get into one of these retry loops, nova or neutron aren't going to get fixed any time soon.22:54
openstackgerritBrian Haley proposed openstack/octavia stable/train: Fix operating status for empty UDP pools  https://review.opendev.org/c/openstack/octavia/+/77017522:55
*** spatel has quit IRC22:56
openstackgerritBrian Haley proposed openstack/octavia stable/stein: Fix operating status for empty UDP pools  https://review.opendev.org/c/openstack/octavia/+/77019622:58
rm_workYeah, unfortunately letting users do Delete or Update operations on a LB that's in PENDING is a recipe for getting orphaned resources... It's hard to say what point in the workflow the controller-worker process has gotten to, and I'm not sure if there's a reasonable way to cancel a workflow from the outside when it's midway through (or if that would actually help the situation).23:01
johnsomI don't think they are asking for a way to cancel the workflow, just how soon we should give up and go to ERROR state23:02
johnsomThe above issue was the controller retrying to connect to the amp via neutron, but getting "no route to host".23:03
rm_worksounded like mchlumsky was advocating for possibly allowing changes in PENDING, and I was speculating about what would be required to make that feasible (since we would need to interrupt existing workflows if we were going to do further changes).23:03
rm_workIt was more of a response to him than to the specific issue23:03
johnsomWhile it was in PENDING_* it was retrying, over and over in hopes neutron would get fixed.23:03
johnsomAh, yeah, overriding the resource lock of PENDING is a bad thing23:04
rm_workwas just expanding on why the choice was made to have PENDING be immutable23:04
lxkongrm_work, johnsom, can I have your review for https://review.opendev.org/c/openstack/octavia/+/767787?23:18
johnsomlxkong, yep, that is as easy one. +223:19
lxkongthanks, we are waiting for this one to support UDP23:20
mchlumskyrm_work Sorry, I should have been more clear. I was talking about how soon to give up and go to ERROR. We can totally live with immutable LBs in PENDING_*. Octavia appears to be a complex state machine as it is without adding arbitrary cancellation points in the workflow.23:20
rm_work:thumbsup:23:22
rm_worklet me look lxkong23:31
rm_workkk23:32
*** yamamoto has joined #openstack-lbaas23:32
rm_work+W23:32
« Tuesday, 2021-01-12 Index Thursday, 2021-01-14 »

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!