Wednesday, 2020-01-15

openstackgerritVishal Manchanda proposed openstack/octavia-dashboard master: Remove six usage
openstackgerritMerged openstack/octavia-dashboard stable/train: Fix Django version in lower-constraints
*** hongbin has joined #openstack-lbaas02:02
*** hongbin has joined #openstack-lbaas02:03
*** hongbin has joined #openstack-lbaas02:04
*** hongbin has quit IRC02:04
*** hongbin has joined #openstack-lbaas02:06
*** mithilarun has joined #openstack-lbaas02:29
*** mithilar_ has joined #openstack-lbaas02:36
*** mithilarun has quit IRC02:39
*** mithilar_ has quit IRC02:41
*** tkajinam has quit IRC03:48
*** hongbin_ has joined #openstack-lbaas04:12
*** hongbin has quit IRC04:15
*** tkajinam has joined #openstack-lbaas04:28
*** hongbin_ has quit IRC04:37
*** sapd1_x has joined #openstack-lbaas04:40
openstackgerritMerged openstack/octavia-dashboard master: Remove six usage
*** sapd1_x has quit IRC05:44
*** mugsie has quit IRC06:11
*** mugsie has joined #openstack-lbaas06:14
openstackgerritAkihiro Motoki proposed openstack/octavia-dashboard stable/stein: Fix Django version in lower-constraints
*** gcheresh has joined #openstack-lbaas06:32
*** KeithMnemonic1 has joined #openstack-lbaas07:15
*** KeithMnemonic has quit IRC07:19
*** gcheresh has quit IRC07:38
*** gcheresh has joined #openstack-lbaas07:40
*** tesseract has joined #openstack-lbaas07:58
*** tesseract has quit IRC07:58
*** tesseract has joined #openstack-lbaas07:59
*** maciejjozefczyk has joined #openstack-lbaas08:06
*** lxkong has quit IRC08:06
*** rpittau|afk is now known as rpittau08:22
openstackgerritOpenStack Proposal Bot proposed openstack/octavia-dashboard stable/train: Imported Translations from Zanata
*** ataraday_ has joined #openstack-lbaas08:40
*** tkajinam has quit IRC08:48
*** dmellado has quit IRC08:51
*** dmellado has joined #openstack-lbaas08:55
*** ccamposr has joined #openstack-lbaas09:01
openstackgerritAnn Taraday proposed openstack/octavia master: Add an ability to retry failover process automatically
*** ccamposr__ has joined #openstack-lbaas09:42
*** pcaruana has joined #openstack-lbaas09:42
*** ccamposr has quit IRC09:44
openstackgerritCarlos Goncalves proposed openstack/octavia-tempest-plugin master: Add tests for allowed CIDRs in listeners
openstackgerritCarlos Goncalves proposed openstack/octavia-tempest-plugin master: Add tests for allowed CIDRs in listeners
*** ccamposr__ has quit IRC10:55
openstackgerritAdam Harwell proposed openstack/octavia master: Conf option to use VIP ip as source ip for backend
rm_workdoubt i will make the meeting in 5 hours, as it will fall right in the middle of my 8 hours of sleep <_<11:12
*** pcaruana has quit IRC11:12
rm_workstill need to get around to making another meeting time poll11:13
openstackgerritAnn Taraday proposed openstack/octavia master: Jobboard based controller
openstackgerritCarlos Goncalves proposed openstack/octavia-tempest-plugin master: Add scenario test for IPv6 VIP and IPv6-only members
*** ccamposr has joined #openstack-lbaas11:27
*** AlexStaf has joined #openstack-lbaas11:30
*** AlexStaf has quit IRC11:34
*** ccamposr has quit IRC11:52
*** ccamposr has joined #openstack-lbaas11:53
*** ccamposr has quit IRC12:31
*** pcaruana has joined #openstack-lbaas12:49
*** cgoncalves has quit IRC12:58
*** cgoncalves has joined #openstack-lbaas13:00
*** cgoncalves has quit IRC13:04
*** cgoncalves has joined #openstack-lbaas13:10
*** ccamposr has joined #openstack-lbaas13:24
*** ppeere_os has joined #openstack-lbaas13:28
ppeere_osHi All! Does any core reviewer has time to review: ?13:28
*** servagem has joined #openstack-lbaas13:33
ppeere_osAh i see it's already in the ussuri priority reviews nvm13:47
*** mnaser has quit IRC13:54
*** mnaser has joined #openstack-lbaas13:55
*** rcernin has quit IRC14:04
*** ccamposr has quit IRC14:09
*** TrevorV has joined #openstack-lbaas15:08
*** ccamposr has joined #openstack-lbaas15:13
*** ataraday_ has quit IRC15:22
*** ataraday_ has joined #openstack-lbaas15:26
johnsomgthiemonge So, if you have time and the environment up, hack the systemd unit for keepalived and add the -D and maybe even a --debug level and try to reproduce it. To my knowledge, the keepalived config should never know about the "ens9" interface name, so I'm not sure that is related.15:56
johnsomI mean obviously it knows from the netlink stream, but our config file should never know about that name15:57
gthiemongejohnsom: I tried the "debug 4" in the conf, it didn't add many useful information15:57
johnsomYeah, keepalived is conservative on what it puts in each debug level and then you hit a certain number and it's a flood15:58
gthiemongejohnsom: I think this is the issue with keepalived 2.0.10, it does some stuff when moving/updating unrelated interfaces15:58
gthiemongejohnsom: I need to check with 2.0.19 or master15:59
johnsomOk. Yeah. It also could be faulting prior to us configuring it for our interfaces, in which case we don't care.15:59
johnsom#startmeeting Octavia16:00
openstackMeeting started Wed Jan 15 16:00:38 2020 UTC and is due to finish in 60 minutes.  The chair is johnsom. Information about MeetBot at
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:00
*** openstack changes topic to " (Meeting topic: Octavia)"16:00
openstackThe meeting name has been set to 'octavia'16:00
johnsomHi everyone16:00
johnsomPretty light agenda today as I didn't have a lot of time to prepare in the absence of our PTL....16:02
*** gcheresh has quit IRC16:02
johnsom#topic Announcements16:02
*** openstack changes topic to "Announcements (Meeting topic: Octavia)"16:02
johnsomThe only item I have here is the North Dakota State University (NDSU) students are selecting projects this week. We will hear in the next week or two if there is a team that will work on Octavia.16:03
johnsomBackground is that OpenStack has worked with NDSU in the past for student "capstone" projects. Adam was offered for us to put in a project idea. We put in the TLS cipher and protocol stories.16:04
johnsomIf we get students, they will work on completing those two stories for us.16:04
johnsomKendall, rm_work, and I will act as mentors16:05
cgoncalvesvery good news :)16:05
cgoncalvesThe "V" release will be named Victoria16:05
johnsomMore info on the capstone projects past/present is here:16:06
johnsomAh yeah, Victoria!16:06
*** armax has joined #openstack-lbaas16:07
cgoncalvesback to pronounceable names :)16:07
johnsomHa, yes16:07
johnsomAny other announcements today?16:07
johnsom#topic Brief progress reports / bugs needing review16:08
*** openstack changes topic to "Brief progress reports / bugs needing review (Meeting topic: Octavia)"16:08
johnsomI have been working on internal planning activities, so a bit distracted.16:08
johnsomI did get some time in on failover, which is good, but could use some more....16:08
johnsomI have also carved out some time doing reviews as I can.16:09
ataraday_Jobboard patches are waiting for reviews :) All discovered issues seems to be resolved, I proposed to mask retry connecntion traces16:10
johnsomAlso, I started work on tags support in our CLI a long time ago. That is finally moving forward in OSC/OSC-lib so that might be ready to work on again in the next few weeks. Waiting on the osc-lib Upper Constraints (UC) update to merge.16:10
cgoncalvesstable branch gates got broken with the release of hacking 2.0.0. this has been fixed, thanks for the quick reviews16:10
johnsomataraday_ I saw that patch go by, thank you!16:10
johnsomataraday_ With my current work, it will likely be next week before I can start reviewing that chain again.16:11
ataraday_cgoncalves, I want to ask you to review devstack part of if you will some time16:11
cgoncalvesalso spent time doing reviews16:12
cgoncalvessadly I did not review jobboard patches yet, sorry16:12
ataraday_johnsom, sure! thanks!16:12
cgoncalvesataraday_, I'm stacking with jobboard now :)16:13
johnsomAny other updates today?16:14
cgoncalvesany chance more people could help review would be appreciated16:15
johnsomAh, yes, I can take a look later.16:16
johnsomI guess it was November the last time I reviewed that... sigh16:17
johnsom#topic Open Discussion16:17
*** openstack changes topic to "Open Discussion (Meeting topic: Octavia)"16:17
johnsomAny other topics for today?16:18
ataraday_I've customer request about having retry for amphora failovers and pushed on the review, was this discussed in past days?16:20
johnsomNo, I haven't seen this. However, my failover re-work code does enable retries. The current code is not idempotent and will blow up. This is part of what is taking so long, I'm going through the tasks and fixing them.16:21
*** ppeere_os has quit IRC16:22
johnsomI hope I can post an early version in the next week, but it won't be complete by then.16:22
ataraday_I pushed this today as I feel having changes on review is the best way to start discussions :) Will mark it WIP for now then16:24
johnsomYeah, I agree, I'm a bit embarrassed I haven't been able to push anything yet. I'm just getting pulled in too many directions.16:24
johnsomRight now 100% of the tests will fail, but a standalone failover will succeed. I'm working on cleaning up the standalone code and creating subflows that can be used for the act/stdby. Then I need to fix all of the tests I broke and then port the code over to the v2 (jobboard) paths. Somewhere in there a rebase is necessary as well.16:26
johnsomKey fixes in:16:26
johnsomThe tasks are idempotent, so retries now work.16:27
johnsomThere are retries in the code.16:27
johnsomIt handles missing amphora correctly. I.e. an LB with no associated amphora now repairs itself.16:27
johnsomIt cleans up extra resources and amphora that may be attached to an LB16:28
johnsomIt checks and repairs the VIP if it is corrupted somehow16:28
johnsomIt will succeed in failing over even if Nova will not release resources.16:28
johnsomIt slices, dices, and juliennes16:29
ataraday_wow, sounds great :)16:30
cgoncalveshah :)16:30
johnsomAnyway, that is the beast I have been working on.16:31
johnsomOh, and it parallelizes more of the flow, so should be slightly faster16:31
cgoncalves1-800-EXPLODE-MY-HEAD xD16:32
johnsomYeah, some days that is how I feel...16:32
johnsomAnyway, it's coming. Work is being done to make failover function as it was originally intended16:33
*** gcheresh has joined #openstack-lbaas16:33
johnsomAny other topics?16:34
johnsomOk then, have a good week everyone!16:37
*** openstack changes topic to "Discussions for OpenStack Octavia | Priority bug review list:"16:37
openstackMeeting ended Wed Jan 15 16:37:31 2020 UTC.  Information about MeetBot at . (v 0.1.4)16:37
openstackMinutes (text):
cgoncalvesthanks for running the meeting :)16:37
gthiemongejohnsom: FYI here is a reproducer for the keepalived bug with simple commands:
johnsomgthiemonge Hmmm, interesting for sure16:41
johnsomgthiemonge Do we have a story open for this?16:41
*** openstackgerrit has quit IRC16:43
johnsomgthiemonge FYI, the Ubuntu bionic we have running the gates is using 1.3.9. So, currently this is limited to centos 8, though it looks like the next LTS for Ubuntu will also have 2.0.1016:43
gthiemongenop, I'm opening one16:44
*** gcheresh has quit IRC16:44
johnsomCool, thanks16:44
*** pcaruana has quit IRC16:45
*** beisner has quit IRC16:50
*** beisner has joined #openstack-lbaas16:50
cgoncalvescentos just can't pick the right versions, uhm? :P17:00
johnsomMaybe they jumped too fast on that one. (can't believe I'm saying that)17:01
*** maciejjozefczyk has quit IRC17:04
*** maciejjozefczyk has joined #openstack-lbaas17:04
johnsomgthiemonge That looks great, thanks!17:08
*** gmann has quit IRC17:09
johnsomWow, that is a sizable changelog between 2.0.10 and 2.0.19 keepalived:
*** gmann has joined #openstack-lbaas17:10
gthiemongejohnsom: adding no_track options in keepalived conf is a possible workaround (see
*** rpittau is now known as rpittau|afk17:15
johnsomgthiemonge Doesn't that mean that if the VIP base port goes down it won't transition to the standby amp?17:16
johnsomWe would have to no_track a specific interface, which we don't know I think.17:17
gthiemongeI haven't verified if no_track affects the VIP port :/17:18
gthiemongebut you're probably right17:18
gthiemongeI may have another workaround that I will try when centos mirrors will be fixed :D17:19
johnsomlol, ok17:19
cgoncalvesgthiemonge, does keepalived crash/get respawn?17:24
johnsomFault state would not be a crash, so I doubt it17:25
gthiemongecgoncalves: no, it stays in FAULT state17:25
gthiemongeand if I restart keepalived service, everything works as expected17:26
*** tesseract has quit IRC17:26
johnsomFAULT just means something it is monitoring failed, so this instance cannot be in MASTER or BACKUP state17:26
cgoncalvesok. asked because I was looking at, also 2.0.10 and not reproducible in 2.0.717:27 bug 1692788 in keepalived "keepalived crashes in a loop when the vrrp interface does not exist" [Urgent,Closed: duplicate] - Assigned to rohara17:27
johnsomLooks like Ryan says that is due to smtp config, which we don't use17:29
*** guilhermesp has quit IRC17:31
*** guilhermesp has joined #openstack-lbaas17:32
xgermanRyan? The guy who kept us out of all Redhat parties?17:41
johnsomYes, one and the same17:42
*** ataraday_ has quit IRC17:56
*** mithilarun has joined #openstack-lbaas17:58
*** servagem has quit IRC17:59
*** born2bake has joined #openstack-lbaas18:02
*** rpittau|afk has quit IRC18:30
*** andrein has quit IRC18:30
*** coreycb has quit IRC18:30
*** mnaser has quit IRC18:31
*** openstackstatus has quit IRC18:32
*** coreycb has joined #openstack-lbaas18:33
*** andrein has joined #openstack-lbaas18:33
*** johnsom has quit IRC18:33
*** mnaser has joined #openstack-lbaas18:33
*** johnsom has joined #openstack-lbaas18:35
*** rcernin has joined #openstack-lbaas18:40
*** sorrison has quit IRC18:42
*** sorrison has joined #openstack-lbaas18:46
*** ccamposr has quit IRC18:52
*** gcheresh has joined #openstack-lbaas18:57
*** openstackstatus has joined #openstack-lbaas19:22
*** ChanServ sets mode: +v openstackstatus19:22
*** gcheresh has quit IRC19:31
*** mithilarun has quit IRC19:36
*** mithilarun has joined #openstack-lbaas19:46
*** mithilarun has quit IRC20:25
*** mithilarun has joined #openstack-lbaas20:27
*** mithilarun has quit IRC20:31
*** mithilarun has joined #openstack-lbaas20:43
*** vishalmanchanda has quit IRC20:44
*** mithilarun has quit IRC20:49
*** mithilarun has joined #openstack-lbaas21:09
*** mithilarun has quit IRC21:13
*** gcheresh has joined #openstack-lbaas21:16
*** mithilarun has joined #openstack-lbaas21:32
*** rcernin has quit IRC21:42
*** gcheresh has quit IRC22:00
*** ccamposr has joined #openstack-lbaas22:19
*** maciejjozefczyk has quit IRC22:35
*** dosaboy has quit IRC22:36
*** TrevorV has quit IRC22:52
*** dosaboy has joined #openstack-lbaas22:53
*** rcernin has joined #openstack-lbaas22:55
*** tkajinam has joined #openstack-lbaas23:03
*** born2bake has quit IRC23:23
*** ccamposr has quit IRC23:34

Generated by 2.15.3 by Marius Gedminas - find it at!