16:00:02 <rm_work> #startmeeting Octavia
16:00:02 <openstack> Meeting started Wed Jun 19 16:00:02 2019 UTC and is due to finish in 60 minutes.  The chair is rm_work. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:04 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:06 <openstack> The meeting name has been set to 'octavia'
16:00:30 <cgoncalves> o/
16:00:33 <rm_work> o/
16:00:40 <gthiemonge> o/
16:00:41 <johnsom> o/
16:01:12 <rm_work> #topic Announcements
16:02:13 <rm_work> So, the TC has decided to stop tracking the health of project teams
16:02:23 <rm_work> #link http://lists.openstack.org/pipermail/openstack-discuss/2019-June/007085.html
16:02:37 <rm_work> So... I guess no more worrying about that
16:02:40 <johnsom> Yeah, probably for the best. It was a strange process anyway.
16:02:49 <cgoncalves> this is what it looked like before for Octavia
16:02:52 <cgoncalves> #link https://wiki.openstack.org/w/index.php?title=OpenStack_health_tracker&direction=prev&oldid=170660#Octavia
16:03:23 <johnsom> Yeah, the paraphrasing was, interesting
16:03:57 <johnsom> I think they only did that once anyway.
16:05:12 <rm_work> Any other announcements?
16:05:51 <johnsom> Shanghai call for papers is open and end in less than a month.
16:06:17 <johnsom> deadline: July 2, 2019 at 11:59pm PT
16:06:39 <rm_work> Ah yes. I'm not sure what we want to submit this time.
16:06:42 <johnsom> #link http://cfp.openstack.org/?_ga=2.124047753.2032053596.1560783728-1706076231.1557509450
16:06:50 <rm_work> I'm aiming to go -- who else is?
16:07:15 <cgoncalves> dulek and I submitted a proposal today on kuryr and octavia
16:07:16 <johnsom> I am planning to not attend in person, but could be available virtual.
16:07:46 <rm_work> ok, so cgoncalves you will be there?]
16:07:47 <johnsom> Happy to help folks with slides too
16:08:01 <cgoncalves> rm_work, I have no idea
16:08:04 <rm_work> Then we could maybe do one of the normal presentations... if we think it's useful
16:08:08 <rm_work> ah, hmmm
16:08:26 <rm_work> well, ok
16:08:33 <cgoncalves> having a session helps but does not guarantee a lottery ticket
16:08:41 <johnsom> It would be great if we can do the project update at least.
16:08:42 <rm_work> heh
16:08:52 <rm_work> yeah but that isn't part of the CFP
16:09:06 <rm_work> I am planning to do that
16:09:06 <johnsom> Correct, you as PTL should get an e-mail about the project update sessions
16:09:46 <rm_work> so maybe we're good enough with just that and onboarding, which they said will be part of the PTG side this time
16:10:12 <rm_work> alright, is that it for announcements?
16:11:20 <rm_work> ok
16:11:22 <rm_work> #topic Brief progress reports / bugs needing review
16:12:22 <ataraday_> I started a couple of new transition to dicts changes, and review needed for #link https://review.opendev.org/#/c/662791/ and #link https://review.opendev.org/#/c/659538/
16:12:22 <johnsom> Log offloading is done and merged. I still want to see if I can get creative with a tempest test for that.
16:12:44 <rm_work> I've been working on multiple things recently, the biggest of which is MultiVIP support, which could use reviews: https://review.opendev.org/#/c/660239/
16:12:50 <rm_work> #link https://review.opendev.org/#/c/660239/
16:12:53 <johnsom> Currently I'm working on some octavia-lib enhancements and a functional test for the driver-agent.
16:13:34 <gthiemonge> I have some changes related to UDP LB that need reviews: https://review.opendev.org/#/q/status:open+project:openstack/octavia+branch:master+topic:udp_states
16:13:53 <rm_work> Also working on some changes to the member subnet plugging calculations/handling, to resolve issues plugging additional subnets on the same network
16:13:55 <rm_work> #link https://review.opendev.org/#/c/665402/
16:14:15 <johnsom> I have also put up some patches removing references to neutron-lbaas from the neutron and neutron-lib repos.
16:15:16 <cgoncalves> I resumed work on the VIP ACL RFE side but progresses slowly. the octavia-lib patch is ready for review
16:15:17 <johnsom> And finally, I did a PoC switching ubutnu over to the -kvm kernel for the image buids. It saves ~200MB in space for the image by removing a bunch of kernel modules we don't need. I have some cleanup to do in DIB for that, but look for that soon.
16:15:19 <cgoncalves> #link https://review.opendev.org/#/q/topic:vip-acl
16:16:22 <johnsom> So comparing the old kernel to the new, size: 605397504 vs 393244160 according to glance
16:16:27 <cgoncalves> the active-standby tempest scenario patch merged. I have to go now enable the jobs also in octavia
16:16:42 <johnsom> -rw-r--r--  1 stack stack 376M Jun 17 18:59 amphora-x64-haproxy.qcow2.kvm
16:16:43 <johnsom> -rw-r--r--  1 stack stack 578M Jun 11 17:18 amphora-x64-haproxy.qcow2.orig
16:17:31 <rm_work> noice, now just need to make centos not huge :D
16:17:53 * johnsom hold my coffee
16:18:20 <rm_work> Ok, cool, lots of work going on
16:18:53 <cgoncalves> ah, I also propose an octavia-tempest-plugin tag release: https://review.opendev.org/#/c/666037/
16:19:40 <rm_work> Ah, before Open Discussion, I think gthiemonge did have a topic? Guess it wasn't added to the agenda page
16:19:56 <rm_work> gthiemonge: i forgot what exactly it was, hopefully you remember :D
16:20:06 <gthiemonge> oh yes, we were talking about UDB LB that mixes IPv4 and IPv6
16:20:07 <gthiemonge> UDP
16:20:43 <gthiemonge> currently, we can create a such LB with members, but keepalived keeps crashing because it doesn't support mixing IPv4/IPV6
16:20:56 <johnsom> Ah, yeah, so LVS doesn't/didn't support mixing the VIP and member protocol versions.
16:21:00 <gthiemonge> so we want to find a good way to handle this
16:21:31 <johnsom> I thought there was a check in the API that blocked it....  Maybe that was missed.
16:21:41 <rm_work> Yeah... so... do we try to validate the members that are added?
16:21:47 <rm_work> Is that the right approach?
16:22:39 <rm_work> and how does that work with multivip? if you have both ipv4 and ipv6... do you allow both kinds of members, but only add the ones that match the address-family for each individual vip?
16:22:57 <rm_work> that could be confusing
16:23:35 <rm_work> like if you add three members, 2x IPv4 and 1x IPv6, the IPv4 VIP would balance between two of them, and the IPv6 VIP would go directly to one
16:24:09 <rm_work> seems like it's very non-intuitive
16:24:19 <johnsom> There might be a way to make it work. Someone should spend some quality time with the keepalived bug list on github and see if there is a fix or workaround.
16:24:45 <rm_work> well, I don't know if it is literally possible to route UDP cross-family
16:25:36 <johnsom> Sure, it's just the IP wrapper that needs NAT really
16:25:40 <rm_work> can LVS do the necessary packet work?
16:26:24 <johnsom> I don't know. Like I said, you may be able to work around it with some iptables NAT rules.
16:26:34 <rm_work> hmm
16:27:01 <rm_work> k, need some help probably from someone who understands the low level networking aspects of this better than I do :)
16:27:06 <johnsom> I know at the time of the UDP work it was identified as a problem, so we did a release note about it. But I don't think it was investigated at all
16:28:10 <johnsom> I'm seeing some comments that this was fixed in the kernel. So, maybe needs a re-test  or test for that matter.
16:28:19 <johnsom> #link https://github.com/acassen/keepalived/issues/876
16:28:21 <rm_work> k
16:31:29 <rm_work> yep, seems like it should work
16:31:39 <gthiemonge> I will test it
16:31:49 <rm_work> so, ok. just need to fix that. I wonder if moving to a new enough keepalived will be difficult
16:31:58 <rm_work> or kernel... is cent7 still on 2.6.x?
16:32:01 <johnsom> Thank you gthiemonge. We should be able to solve it one way or another.
16:32:52 <rm_work> or did they get to 3.x yet
16:33:13 <johnsom> Pretty sure it's 3.x
16:33:36 <johnsom> Let's check the log offload.... grin
16:33:42 <cgoncalves> fully loaded with feature backports, I must add
16:34:06 <rm_work> need at least 3.18
16:34:11 <johnsom> Linux version 3.10.0-957.21.2.el7.x86_64
16:34:18 <rm_work> also i was joking but i guess maybe it still is that old, rofl
16:34:26 * rm_work dies
16:34:28 <johnsom> Yeah, but it's hard to say if that feature was backported
16:34:44 <johnsom> #link http://logs.openstack.org/29/665029/3/check/octavia-v2-dsvm-py2-scenario-centos-7/735918f/controller/logs/octavia-amphora_log.txt.gz#_Jun_19_00_28_26
16:35:08 <rm_work> rofl ok, so yes, actually ancient
16:35:14 <cgoncalves> if not, we could try to check with the kernel team if it's possible
16:35:29 <rm_work> can we immediately drop cent7 support once cent8 is ready? >_>
16:35:52 <cgoncalves> yes IMO
16:35:56 <rm_work> i guess we can check for HAVE_DECL_IPVS_DEST_ATTR_ADDR_FAMILY
16:36:22 <rm_work> in keepalived/check/libipvs.c
16:36:39 <cgoncalves> our commercial Stein-based product will be fully on RHEL 8
16:36:48 <rm_work> err or is it in the kernel's configure.ac
16:36:55 <rm_work> well anyway yeah, we can check
16:37:03 <rm_work> ubuntu should be on 4.x so no issues right
16:37:16 <rm_work> and keepalived version 1.4.5+ probably
16:37:39 <johnsom> Linux version 4.4.0-151-generic
16:37:49 <rm_work> hmm no
16:37:57 <rm_work> bionic even still has 1.3.9
16:38:13 <rm_work> disco as 2.0 ....
16:38:22 <rm_work> even cosmic is still 1.3.9
16:38:29 <rm_work> that's problematic
16:38:59 <rm_work> i wonder if it can be backported
16:39:19 <johnsom> keepalived amd64 1:1.3.9-1ubuntu0.18.04.2
16:39:49 <johnsom> Again, hard to say on what they pulled back, etc.
16:39:49 <rm_work> yeah gross
16:39:56 <rm_work> i mean
16:40:11 <johnsom> We are just too cutting edge... lol
16:40:18 <rm_work> i don't think they would have backported a ton of features from 1.4.x right? wouldn't they just... RUN 1.4.x in that case?
16:40:36 <rm_work> anywho, we can work this out
16:41:05 <rm_work> #topic Open Discussion
16:41:22 <johnsom> I gave up trying to guess that stuff a long time ago
16:43:01 <johnsom> Just a quick qeustion. For the driver-agent functional tests, I need to create real files on the filesystem. I'm currently generating unique files in /tmp for that. Is that the right approach?
16:43:25 <colin-> are their contents meaningful?
16:43:29 <johnsom> I need to open the Unix domain sockets and create a DB file for sqlite.
16:43:43 <johnsom> The test has a cleanup hook to remove them
16:44:32 <johnsom> Basically they are the live driver agent sockets the tests will use. We will be firing up a driver-agent, without the full devstack for the functional tests.
16:45:24 <cgoncalves> I think that is fine
16:45:55 <johnsom> They are all uuid'd so they won't conflict with others running the same tests, etc.
16:46:06 <colin-> i would do it in /tmp, too
16:46:31 <johnsom> Cool, I thought so, just thought I would ask. I should have something posted for review today on that.
16:49:56 <colin-> do you folks think there is anything in the 2.0 release of haproxy that we should especially be looking forward to? most of what i'm interested in is outside the scope of octavia (prometheus scraping, for example)
16:50:28 <johnsom> I'm excited about the HTTP/2 work personally.
16:50:35 <rm_work> fyi looks like newer keepalived is needed for other reasons in bionic: https://bugs.launchpad.net/ubuntu/+source/keepalived/+bug/1819074
16:50:36 <openstack> Launchpad bug 1819074 in systemd (Ubuntu) "Keepalived < 2.0.x in Ubuntu 18.04 LTS not compatible with systemd-networkd" [Undecided,Confirmed]
16:50:47 <rm_work> so... it could happen
16:51:07 <colin-> good point johnsom more streaming and fewer conn brokering is always a good thing
16:51:32 <johnsom> Yeah, they have backported newer versions of haproxy for us in the past. Just need to request it on launchpad and reference the version in a newer release.
16:51:41 <rm_work> yes, 2.0 looks quite good actually
16:51:48 <rm_work> also the dataplane api will be interesting
16:52:18 <rm_work> I made this "question" but not sure if it should just be a bug: https://answers.launchpad.net/ubuntu/+source/keepalived/+question/681490
16:52:27 <johnsom> colin- The challenge we have is that the distros won't have 2.0 for a while, so it's a decision if we want to do things that require custom built images...
16:52:50 <johnsom> rm_work In the past I have just opened a backport request bug.
16:52:51 <rm_work> 2.1 will bring UDP loadbalancing to haproxy :D
16:52:53 <colin-> ah, i see yeah
16:52:55 <johnsom> I think they have a process
16:54:24 <rm_work> but yeah, a couple new protocols will be nice, http2 especially
16:54:27 <johnsom> We had a topic about that at the last PTG. Even 1.9 brings some really useful stuff like fully functional threading.
16:54:59 <rm_work> i wonder if we could just maintain an octavia packages repo <_<
16:55:04 <rm_work> for haproxy and keepalived
16:55:06 <rm_work> lol
16:55:24 <johnsom> And the kernel... Oh, wait.
16:56:07 <johnsom> I think we just need to come up with a strategy of how we handle such things and document it.
16:56:12 <rm_work> i mean it'd be possible with periodics
16:56:31 <rm_work> we could build them in-gate and upload them to the openstack artifact store
16:56:35 <rm_work> or our own PPA
16:56:38 <johnsom> For example, if someone asks for HTTP/2 via the API but the amp we get doesn't have a compatible version of HAproxy.
16:57:34 <johnsom> We can detect that, but do we just ERROR out the object? Ignore the setting and fall back to HTTP 1.1, etc.
16:57:51 <rm_work> probably safe-fallback strategies
16:58:12 <rm_work> or the admin configures which protocols are available
16:58:20 <rm_work> based on the amp version they run
16:58:36 <cgoncalves> ERROR out at API with a suggestion to an alternative (fall back to HTTP 1.1)
16:58:39 <johnsom> How would they know?
16:58:52 <rm_work> cgoncalves: API can't know
16:58:56 <rm_work> unless admin configures it
16:58:59 <cgoncalves> ah, right
16:59:05 <johnsom> Right, things to think about. We can continue the discussion next week.
16:59:14 <rm_work> yep, good meeting everyone
16:59:30 <cgoncalves> +1
16:59:31 <rm_work> #endmeeting