Wednesday, 2019-10-16

*** goldyfruit_ has joined #openstack-lbaas00:05
*** ricolin has joined #openstack-lbaas01:57
*** psachin has joined #openstack-lbaas04:02
*** ramishra has joined #openstack-lbaas04:06
*** ajay33 has joined #openstack-lbaas04:35
*** pcaruana has joined #openstack-lbaas04:37
*** tkajinam has quit IRC05:04
*** gcheresh_ has joined #openstack-lbaas05:11
*** pcaruana has quit IRC05:14
*** tkajinam has joined #openstack-lbaas05:42
*** pcaruana has joined #openstack-lbaas06:13
*** gcheresh_ has quit IRC06:44
*** maciejjozefczyk has joined #openstack-lbaas07:03
*** takamatsu has joined #openstack-lbaas07:08
*** tesseract has joined #openstack-lbaas07:12
*** tesseract has quit IRC07:14
*** tesseract has joined #openstack-lbaas07:14
*** ivve has joined #openstack-lbaas07:19
*** trident has quit IRC07:27
*** trident has joined #openstack-lbaas07:31
*** rpittau|afk is now known as rpittau08:12
*** gcheresh_ has joined #openstack-lbaas08:13
*** tkajinam has quit IRC08:16
*** takamatsu has quit IRC08:21
*** takamatsu has joined #openstack-lbaas08:36
*** gcheresh_ has quit IRC08:38
*** ramishra has quit IRC08:43
*** ramishra has joined #openstack-lbaas09:00
f0ois train coming out today or will it be postponned?09:07
cgoncalvesf0o, it is scheduled to be released this week:
f0ojust referring to together with - the Schedule advertises today as release day but the release summary lists a lot of RCs  - just curious I guess09:14
cgoncalvesmy understanding is that is the go/no-go for final release09:18
f0othanks :)09:19
cgoncalvesreading backlog of #openstack-release, it all seems the train is on track to leave on schedule and today09:20
*** gcheresh_ has joined #openstack-lbaas09:25
*** rcernin has quit IRC09:54
fricklernlbaas cores please have a look at this fix, mainly to support py3, but also resolving a long standing bug for py2
cgoncalvesreviewed. thank you for the patch10:10
*** gcheresh_ has quit IRC10:40
openstackgerritJens Harbott (frickler) proposed openstack/neutron-lbaas stable/stein: Fix lb stats model
fricklercgoncalves: ^^ added a reno, but IIUC with the master branch being dead, release notes aren't updated/published anymore. at least there is no job for it I can find11:06
*** maciejjozefczyk has quit IRC11:07
cgoncalvesfrickler, thanks. hmm, now that you mention it I think we were sort of forced to disable release notes jobs due to some CI related issues11:20
*** maciejjozefczyk has joined #openstack-lbaas11:21
*** rcernin has joined #openstack-lbaas11:30
*** sapd1 has joined #openstack-lbaas12:00
*** ivve has quit IRC12:03
openstackgerritAnn Taraday proposed openstack/octavia master: Convert Lb flows to use provider dicts
openstackgerritAnn Taraday proposed openstack/octavia master: Jobboard based controller
*** sapd1 has quit IRC12:12
*** AlexStaf has quit IRC12:18
*** ivve has joined #openstack-lbaas12:18
*** AlexStaf has joined #openstack-lbaas12:18
*** AlexStaf has quit IRC12:28
*** AlexStaf has joined #openstack-lbaas12:28
*** AustinR has joined #openstack-lbaas12:32
*** gcheresh_ has joined #openstack-lbaas13:04
*** lxkong has joined #openstack-lbaas13:07
*** lemko has joined #openstack-lbaas13:25
lemkoHi, which database does octavia directly use? Only octavia? or also neutron, nova?13:26
cgoncalveslemko, only the octavia database13:27
cgoncalvesall interactions with neutron happen via Neutron API13:28
lemkobecause I've some problem with some slow databases running on slow ceph : health message was processed too slowly: 14.0596139431s! The system may be overloaded or otherwise malfunctioning. This heartbeat has been ignored and no update was made to the amphora health entry13:33
lemkoThis means the update to Octavia db only was too slow?13:33
lemkoI don't want to migrate all my database to some other storage13:34
lemkobut if it's only Octavia, I would not be too sad.13:34
*** gcheresh_ has quit IRC13:37
cgoncalveslemko, correct. I'd recommend you to fix the DB slowdown rather than moving the octavia database elsewhere, especially if that is a production environment13:40
johnsomlemko: that db transaction that is taking your system 14+ seconds normally only takes hundredths of a second to complete.13:55
*** rcernin has quit IRC13:57
*** ajay33 has quit IRC13:58
*** spatel has joined #openstack-lbaas14:30
*** spatel has quit IRC14:45
*** tobias-urdin has joined #openstack-lbaas14:58
tobias-urdingot an amphora that failed when reprovisioning a new image, so the active_standby load balancer is degraded now15:01
tobias-urdinonly one amphora instance exists, the backup one is status ERROR in the database and octavia doesn't seem to want to try rebuilding it again15:01
tobias-urdinhow can i force it to try to spawn a new backup amphora?15:02
*** ataraday_ has joined #openstack-lbaas15:02
johnsomThe amphora failover API is not working for you?15:03
tobias-urdinhm what do you mean? one amphora is gone but octavia doesn't seem to want to spawn a new one15:05
*** ivve has quit IRC15:05
tobias-urdinthe master is alive so traffic is flowing, just that there is no standby because it failed the last spawn and now it has stopped trying15:05
johnsomIf the failed amphora is still associated with the load balancer, but listed in ERROR you can try:
johnsomIf not, there is the LB failover that may work:
lemkocgoncalves and johnsom, the DB's performance are not possible to improve. It can happen that under some specific circumstances (that I cannot affect ) that the storage gets really slow. It doesn't matter too much for other openstack components, but only for Octavia and its health checks.15:07
tobias-urdinjohnsom: just want it to spawn a the standby and not be a single amphora15:09
tobias-urdinopenstack loadbalancer failover <id>15:09
tobias-urdinInvalid state ERROR of loadbalancer resource 1f40457d-059c-4ce6-bd91-5c1b26d2853915:09
tobias-urdinprovisioning_status is ERROR on the loadbalancer15:10
tobias-urdinthe master amphora is ALLOCATED, the standby is ERROR (and does not exist)15:10
tobias-urdinthe master is alive and traffic is flowing15:10
*** maciejjozefczyk has quit IRC15:10
tobias-urdinrocky btw so not sure if i have the amphora failover api, i dont have the openstackclient command atleast15:12
johnsomI am wondering if you have an old version of Rocky too, as failover should work with LB in ERROR.15:13
*** trident has quit IRC15:13
*** trident has joined #openstack-lbaas15:17
openstackgerritJens Harbott (frickler) proposed openstack/neutron-lbaas stable/rocky: Fix lb stats model
openstackgerritJens Harbott (frickler) proposed openstack/neutron-lbaas stable/rocky: Fix lb stats model
tobias-urdinjohnsom: running 3.0.1 and latest is 3.2.0 can see the _test_lb_status or where it's called in load_balancer controller that would allow ERROR state to failover15:25
tobias-urdin*can't see15:25
*** lxkong has quit IRC15:39
*** AustinR has quit IRC15:40
*** maciejjozefczyk has joined #openstack-lbaas15:46
*** maciejjozefczyk is now known as mjozefcz|afk15:47
*** AustinR has joined #openstack-lbaas15:56
johnsomblah, late, just a sec16:02
johnsom#startmeeting Octavia16:02
openstackMeeting started Wed Oct 16 16:02:54 2019 UTC and is due to finish in 60 minutes.  The chair is johnsom. Information about MeetBot at
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:02
*** openstack changes topic to " (Meeting topic: Octavia)"16:02
openstackThe meeting name has been set to 'octavia'16:03
johnsomHi folks, sorry for the delay....16:03
colin-hello octavians16:03
johnsomcgoncalves Thanks for the poke.16:03
johnsom#topic Announcements16:03
*** openstack changes topic to "Announcements (Meeting topic: Octavia)"16:03
johnsomTrain released today!16:03
johnsomThank you to everyone for your contributions. Code, reviews, otherwise!16:04
johnsomIn our patch review push we merged 106 patches.16:05
cgoncalvesmany more patches were merged. those 106 patches were the ones we at some point prioritized :)16:06
johnsomRight, absolutely correct. There was more than just the 106, but the 106 was our priority list.16:07
amotokijohnsom: congrats! how did you calculate it? just from my curiosity.16:07
johnsomamotoki We created a "priority review list" around MS2:16:08
*** tesseract has quit IRC16:08
johnsomIt helps us stay on top of patch dependency ordering and priority for the release.16:08
openstackgerritAnn Taraday proposed openstack/octavia master: Convert Lb flows to use provider dicts
johnsomLooking forward, the PTG is coming up in a few weeks.16:09
johnsomWe have an etherpad up to gather topics:16:09
cgoncalves$ git rev-list --count 4.0.0..5.0.0 -> 258 patches merged between Stein GA and Train GA16:10
johnsomWe are a small team, but we can get stuff done. lol16:10
johnsomPlease add any topics to the list that you think the team should discuss at the PTG.16:11
johnsomThere will be three Octavia cores attending, so a good quorum.16:11
johnsomIf you are interested in other project team etherpads for the PTG, the list is being managed here:16:12
johnsomOctavia is booked to have a room/table for two and a half days.16:13
johnsom(it sounds like there may not be rooms at this PTG)16:13
johnsomAny other announcements today?16:14
johnsom#topic Brief progress reports / bugs needing review16:15
*** openstack changes topic to "Brief progress reports / bugs needing review (Meeting topic: Octavia)"16:15
johnsomI have been working on a bug around barbican outages/secrets being deleted. I have a patch up for review on that. There is one more part of the bug I'm trying to reproduce. I should have that wrapped up today/tomorrow-ish.16:16
*** sapd1 has joined #openstack-lbaas16:16
ataraday_I created this  #link!/story/2006627 some time ago, and pushed one patch, look forward some thoughts on this16:17
johnsomI also took a mental break and put up some docs patches updating the cookbook for some of the new TLS capabilities we added.16:17
johnsomI had the devstack already setup for barbican, so it seemed like a good time to check those off the list.16:17
ataraday_Jobboard is in progress, convert patches are ready, working on the main change #link
cgoncalvesataraday_, thanks for working on that! I have a question/ask for feedback if that is okay16:18
johnsomataraday_ Ah, yes! Adding ciphers and protocols was on my short-term wish list. I will take a look and give feedback. I have put some thought into this as well.16:18
cgoncalvesshould it be set in the configuration file or made available to LB owners via API?16:18
johnsomI assumed we were adding it to the listener API16:19
ataraday_I proposed adding default cipher setting via config, and one change will add setting specific cipher on each listener16:20
ataraday_via listener API16:20
johnsomOk, so a default in config and then optional listener API setting. Yeah, that aligns to my idea as well.16:20
cgoncalveswhat if the admin wants to limit ciphers? e.g. not permit SSL v1 and v2?16:21
*** mjozefcz|afk has quit IRC16:21
johnsomI also think our default should follow:16:22
johnsomI was leaning towards suite B16:23
johnsomYeah, it probably is valid to have a way for an operator to set a required "minimum" protocol level. Maybe even a blacklist for ciphers16:23
ataraday_via API or via config?16:24
johnsomThose are all separate requests/patches however. They don't all need to be in this initial patch.16:24
johnsomI think the minimum and blacklist would be config file settings that the API input is validated against.16:25
colin-for our use cases a minimum protocol level would satisfy most concerns, fwiw16:25
ataraday_sounds good!16:25
cgoncalvesnice, we are all in agreement \o/16:26
johnsomataraday_ If you don't mind, after the meeting I can capture my thoughts in the storyboard story.16:26
johnsomWe can break it down into different tasks16:27
ataraday_johnsom, It is highly appreciated!16:27
johnsomAfter I wrap up this barbican story, I'm back to working on the failover flow I started before my vacation.16:28
johnsomStill a lot of work to do there.16:28
cgoncalvesnothing to report from my side. reviewing patches and working on tripleo16:28
*** psachin has quit IRC16:29
johnsomAny other updates? If not I will move on to open discussion16:30
colin-mostly sharing for visibility but did open this and may take a stab at an implementation for you folks to review if i can get it working!/story/200665316:30
johnsomcolin- Cool, thanks for working on that. Let us know if we can answer questions, etc....16:31
colin-appreciate it, the links to the existing bodies of work where other timeouts were implented was super helpful16:31
johnsom#topic Open Discussion16:32
*** openstack changes topic to "Open Discussion (Meeting topic: Octavia)"16:32
johnsomOne thing I would like us to start thinking about is how we currently handle TLS offload listeners.16:32
johnsomThe existing implementation is protocol "TERMINATED_HTTPS"16:33
johnsomHowever, we can support other protocols wrapped in TLS.16:33
*** ramishra has quit IRC16:34
johnsomI have been thinking about should we add a "tls_enabled" boolean and move away from long lists of protocols, or just add more "TERMINATED_*" protocols.16:34
johnsomNot something I can work on any time soon, but something to think about for a future discussion.16:35
johnsomFor example, we could support "TERMINATED_TLS_TCP" now if we added it....16:35
johnsomAnyway, any other topics for today?16:36
colin-fair point, not sure which i prefer but certainly worth noodling on16:36
colin-i do kind of like the idea of a boolean16:36
cgoncalveswhat would be the benefit of adding a boolean? protocols would still need to be added to the list16:36
colin-abstracting that characteristic away from the listener protocol i guess16:37
johnsomTrue, but we wouldn't be adding two for each. I.e. SMTP and TERMINATED_TLS_SMTP (bogus example, but...)16:37
colin-so then an HTTPS type listener would either terminate or pass-through their HTTPS traffic based on that, if i am conceptualizing it correctly16:37
johnsomRight, we would need to come up with a "how to do the right thing with the legacy protocol list"16:38
cgoncalveswould users be allowed to update "tls_enabled"? I think there would be some amount of implications we would need to take care of in the server side16:38
colin-hm yeah that would be difficult to make mutable16:39
cgoncalvesright now to be it seems easier to both users and devs to keep adding TERMINATED_* protocols to the list16:39
johnsomI think it could be possible to update to a tls_enabled state. As long as the validation passes, i.e. certs exist, etc.16:39
cgoncalvesmutating that boolean flag would incur anyway in short downtime and breakage of open connections, no?16:40
johnsomYeah, changing protocols underneath a connection is going to break it. for sure....16:41
*** AustinR has quit IRC16:42
cgoncalvesright. so I am not certain of the value it would bring. I'd need to understand this better, I guess16:42
johnsomI just wanted to raise the discussion. Like I said, not on my short term roadmap at the moment16:43
colin-yeah it's a good thing to start considering16:44
johnsomAny other topics today?16:45
johnsomOk then, thank you!  Nice work on Train. We did some great work there.16:46
openstackgerritMerged openstack/octavia master: Add client authentication to the LB cookbook
*** openstack changes topic to "Discussions for OpenStack Octavia | Priority bug review list:"16:47
openstackMeeting ended Wed Oct 16 16:47:06 2019 UTC.  Information about MeetBot at . (v 0.1.4)16:47
openstackMinutes (text):
cgoncalvesthanks johnsom16:47
openstackgerritMerged openstack/octavia master: Add backend re-encryption to the LB cookbook
gthiemongejohnsom: I have script that reads worker's logs (from stdin) and outputs a csv that contains flows and tasks duration:
gthiemonge(not sure about the date parser, but it works for me on devstack and tripleo)16:50
*** rpittau is now known as rpittau|afk16:50
cgoncalvesreally good stuff that script!16:54
cgoncalvescolin-, would you be interested in testing ^? we were wondering last week's meeting what would be a sane graceful shutdown period16:56
cgoncalvessame goes to any other Octavia operator out there :)16:56
gthiemongeit would be nice to get feedback on the duration of octavia-create-loadbalancer-flow16:56
johnsomMaybe we should remove the name field to protect privacy16:59
gthiemongejohnsom: the name field is the name of the flow/task17:01
johnsomAh, ok17:01
colin-cgoncalves: testing gthiemonge's script you mean, right? sure will make some time for that later on and share any feedback17:02
johnsommnaser Maybe you could also help us out with some real-world stats by running the above script.17:02
colin-do i ineed to induce a certain condition while  it's watching?17:02
*** goldyfruit has joined #openstack-lbaas17:03
johnsomIt should be just scraping the times from the log for normal LB creates, etc.17:03
colin-sounds good17:03
colin-looks handy, thanks gthiemonge17:04
*** goldyfruit_ has quit IRC17:05
gthiemongecolin-: thank you, ping me if you have any issues17:06
*** AustinR has joined #openstack-lbaas17:09
*** ataraday_ has quit IRC17:23
*** sapd1 has quit IRC17:27
cgoncalvescolin-, thank you!!17:33
*** ricolin has quit IRC17:36
*** lemko has quit IRC17:55
*** gcheresh_ has joined #openstack-lbaas17:58
*** TrevorV has joined #openstack-lbaas18:05
openstackgerritBrian Haley proposed openstack/octavia master: Stop testing python 2
openstackgerritMerged openstack/neutron-lbaas stable/stein: Fix lb stats model
*** gcheresh_ has quit IRC18:58
openstackgerritMerged openstack/octavia-dashboard master: Update master for stable/train
*** gcheresh_ has joined #openstack-lbaas20:04
rm_workgood meeting today :D20:26
rm_workthanks for running it again johnsom20:27
rm_workI'm still debating whether I should propose a different time again20:27
rm_workcould torture carlos's girlfriend with Octavia Date Night again ;)20:27
*** pcaruana has quit IRC20:28
*** gcheresh_ has quit IRC20:30
johnsomYeah, found an old centos7 image that had devstack on it, figured I could just install queens there. Nope. Devstack used to pull in some bogus version of mariadb so keystone blows up doing a db migration.  blah, harder than it should be to run this old version....20:55
johnsomFresh VM here I come20:55
openstackLaunchpad bug 1833696 in Manila "Devstack installs version-less rdo-release package" [High,Fix released] - Assigned to Goutham Pacha Ravi (gouthamr)20:59
colin-cgoncalves: worked with gthiemonge to produce these in support of the change we discussed earlier. the scenario was a two-controller setup of octavia workers and a terraform job i have that sets up two ordinary web backends and an octavia LB, HTTP/S listeners, pools, members, and HMs. ran it three times: create, destroy // create, destroy // create, failover, destroy21:03
colin-results can be found here: lmk if anything is out of order21:03
johnsomcolin- Thank you for helping us out with that21:08
colin-always happy to run a strange script with little scrutiny on my controllers :o)21:08
johnsomlol, noted...21:13
*** TrevorV has quit IRC21:24
rm_worknow we need something to turn that into useful graphs22:01
rm_workI could try to whip out some R....22:01
johnsomIt's not that many data points really. All of the boots are under 1:3022:03
*** AustinR has quit IRC22:17
*** AustinR has joined #openstack-lbaas22:30
johnsomrm_work You know a bit about kolla ansible for Octavia. Any chance you can go figure out why it's installing old versions of Octavia? Someone on the mailing list is again having trouble with an old version of stable/stein getting installed.22:37
rm_workyeah i can take a peek in a minute22:37
johnsomIt's like the third or fourth person having the trouble with kolla22:37
rm_workk-a should just be grabbing the tag for whatever version the user specifies22:38
rm_workbut i'll check it out22:38
johnsomCool, thanks22:38
rm_workcurrently reviewing internal doc proposals >_>22:38
johnsomFun... I'm fighting with installing old versions of stuff....22:38
rm_workalso i've yet to stand up today since I woke up, so i should do that really quick22:39
johnsomInternalError: Nova requires QEMU version 2.1.0 or greater.22:39
rm_workyeah bitrot makes some of that a crapshoot22:39
johnsomSeems I need to pull in hacky repos for centos 722:39
johnsomqemu is provided by two different repos qemu-ev and EPEL. libvert only sees qemu version installed by epel repository. so removing epel repo and restarting libvirt service should fix the issue22:43
johnsomsudo yum remove qemu-system-x8622:43
cgoncalvesjohnsom, I know that keystone bug! I think I fixed it in devstack, check
cgoncalvesoh, you found the launchpad # I created :)23:30
johnsomYeah, it is "fixed" but not if you are already on the newer mariadb....23:30
johnsomI finally got a queens devstack to boot on centos7.23:31
johnsomStill can't reproduce your "PENDING_UPDATE" state other than when it should be in that state. I'm going to try act/standby next as I see you had that enabled.23:31
cgoncalvescolin-, thank you!23:31
*** tkajinam has joined #openstack-lbaas23:31
cgoncalvesjohnsom, shame on me. I stacked today but haven't touched it23:32
johnsomFunny, even the second TCP listener came up fine. It threw errors in the logs for sure, but it completed.23:33
johnsomWell, something to look at tomorrow. I'm about to call it for the day.23:35
rm_workjust finished doc reviews, taking a glance at k-a23:42
rm_worki'll let you know23:42
rm_workso k-a basically just uses a tag, and it's set up for octavia just like everything else23:47
rm_workbut all it's doing is looking up that tag in your configured docker-repository, so it depends on what docker image you built and uploaded using "kolla" (k-a is in the kolla umbrella but that's a different project)23:48
rm_workand there's binary mode and source mode for that -- so if it's using binaries, that's up to whatever centos/debian/etc have released, so if the distros don't keep up, then stuff will be old23:49
rm_workif it's source mode, it's again just pulling from a tag, and it's all consistent with the way other projects are built23:50
openstackgerritMerged openstack/python-octaviaclient master: Drop netifaces from requirements.txt
rm_worki wonder if they're using an old version of kolla-ansible?23:56

Generated by 2.15.3 by Marius Gedminas - find it at!