16:00:27 <johnsom> #startmeeting Octavia
16:00:27 <openstack> Meeting started Wed Nov 13 16:00:27 2019 UTC and is due to finish in 60 minutes.  The chair is johnsom. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:28 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:30 <openstack> The meeting name has been set to 'octavia'
16:01:02 <rm_work> ohai
16:01:12 <rm_work> was reading logs and missed the time :D
16:01:14 <rm_work> but go ahead
16:01:31 <ataraday_> hi
16:01:39 <johnsom> rm_work Nice, you made it. I put an agenda together:
16:01:44 <johnsom> #link https://wiki.openstack.org/wiki/Octavia/Weekly_Meeting_Agenda#Meeting_2019-11-13
16:02:17 <johnsom> #topic Announcements
16:02:41 <johnsom> In case you missed it, our fearless PTL put together a PTG summary e-mail
16:02:48 <johnsom> #link http://lists.openstack.org/pipermail/openstack-discuss/2019-November/010682.html
16:03:19 <johnsom> BTW, I know a few people are travelling today, so attendance might be light today.
16:04:01 <johnsom> That is about all I have for announcements today.
16:04:20 <johnsom> I don't know when the video recordings will be available, so no update on that.
16:04:56 <johnsom> Any other announcements today?
16:05:32 <johnsom> #topic Brief progress reports / bugs needing review
16:06:28 <johnsom> I have been on a bug fix spree recently.  A lot of updates around barbican secrets and TLS offload.
16:06:42 <ataraday_> Small octaviaclient change become not so small https://review.opendev.org/#/c/693144/ :)
16:07:02 <redrobot> 👀
16:07:05 <johnsom> One set are important fixes for LBs with multi-listeners that are using TLS offload.
16:07:23 <openstackgerrit> Adam Harwell proposed openstack/octavia master: Availability Zone admin API  https://review.opendev.org/693765
16:07:24 <rm_work> We're making quick progress on the AZ work -- please to be reviewing :D ^^
16:07:52 <johnsom> redrobot Hi.  Just mentioning some of the patches I posted to handle pkcs12 secrets disappearing (since they don't support registration at the moment).
16:07:54 <ataraday_> I researched ciphers and ciphersuits a bit - maybe can discuss this in open discussion section
16:08:50 <johnsom> Ok, sounds good
16:09:26 <johnsom> I am now shifting focus back to the failover flow re-factor.
16:10:40 <johnsom> As part of the TLS work, I have proposed tempest tests for all of the listener/frontend TLS paths. TLS, SNI, client auth.
16:10:47 <johnsom> Super happy to have good coverage on those.
16:11:24 <johnsom> The backend is going to be a bit more work as we need to enhance our testing web server to have TLS support enough to cover the test cases.
16:11:58 <johnsom> Any other updates today?
16:13:21 <johnsom> #topic Helping Kolla-Ansible eitherpad
16:13:29 <johnsom> #link https://etherpad.openstack.org/p/kolla-ansible-octavia
16:13:57 <johnsom> We have recently had a number of folks come into the channel struggling to use Octavia with the kolla-ansible deployment project.
16:14:46 <johnsom> I don't think any of the core team regularly contribute to kolla-ansible, but we have offered to help resolve some of the issues.
16:15:00 <johnsom> In support of that I setup the above etherpad to answer questions, etc.
16:15:10 <johnsom> Please feel free to contribute, etc.
16:15:51 <johnsom> I think we are close to over ten ways to deploy Octavia, so it is to be expected that the Octavia team may not be directly involved in all of them.
16:16:46 <johnsom> Any other questions/comments on helping kolla-ansible?
16:17:20 <rm_work> I am probably the most experienced of core members, but have been stretched very thin recently, so not sure how much I can help directly
16:17:28 <rm_work> but I will see if I can answer some questions at least
16:17:53 <johnsom> Yep, all I can commit to at this point is helping to answer questions.
16:18:33 <johnsom> #topic Open Discussion
16:18:40 <johnsom> Ok, any other topics today?
16:19:18 <rm_work> Ah, I have one
16:19:45 <johnsom> Ok. I think Ann has one too
16:20:09 <rm_work> I've recently been testing to see if traffic that goes to members via the vip-net is marked as originating from the vrrp_ip (unknown to users), or the ha_ip (VIP) -- and it looks like it's coming from the vrrp_ip
16:20:24 <rm_work> Has anyone else noticed this?
16:21:00 <rm_work> I thought it was set up to route "from" the VIP... We use CentOS amps, it might be working properly in Ubuntu but not in CentOS, or maybe it's not working anywhere...
16:21:18 <rm_work> Anyway, if anyone has a moment to test that or happens to have noticed it is an issue, let me know
16:21:29 <johnsom> Ah, now that I think about this more, yes, I think that is the case. It is the same as for members going out member subnets. It's an arbitrary source IP
16:21:53 <johnsom> I should have a stack in about half an hour I can try it on
16:22:02 <rm_work> can't we have it originate from the VIP? since haproxy IS bound to that address?
16:22:45 <johnsom> Well, it cuts into your capacity if I remember correctly
16:23:30 <johnsom> We might have to make some jinja changes for that too.
16:23:42 <rm_work> yeah, I was looking in the port templates
16:23:59 <rm_work> but I can't read that stuff very well (the port config, not the jinja -- i speak jinja :D)
16:24:08 <johnsom> There is also a RFE to add multiple source IPs, which would conflict with forcing it all to the VIP
16:24:18 <rm_work> hmmm k
16:24:27 <rm_work> well i mean... yeah... multivip :D
16:24:33 <rm_work> also
16:24:54 <johnsom> I am guessing you just want this for "easy security group rules"?
16:25:04 <rm_work> well, not "easy" but "any at all"
16:25:19 <rm_work> it's otherwise impossible for a user to open members to the LB
16:25:24 <rm_work> without opening it to the whole world
16:25:33 <rm_work> since they can't predict the vrrp_ip
16:25:34 <johnsom> One-armed load balancers are less efficient anyway. I would hope that is a rare usecase
16:25:47 <rm_work> it's the only use-case
16:25:49 <rm_work> we have no SDN
16:25:56 <rm_work> there is only one network
16:26:02 <johnsom> Right, or the member subnet source IP, both of which change on failovers.
16:26:04 <ataraday_> sorry, I got disconncted, bad hotel wifi
16:26:24 <rm_work> we need a solution to this
16:26:26 <johnsom> Right now you have two options: Put the members on private networks with no router.
16:26:36 <johnsom> Use TLS client auth for the members
16:27:02 <rm_work> yeah neither of those are possible/viable with our setup
16:27:23 <rm_work> per PCI compliance we just can't open firewall ports that widely apparently
16:27:24 <johnsom> Do you use FWaaS?
16:27:58 <johnsom> Wait, what? Neither of those options provided require opening ports
16:28:08 <rm_work> no, there are physical (and somewhat manually managed for compliance) firewalls for our PCI environments
16:28:32 <rm_work> using TLS client auth for members "secures them" but still requires opening up the SGs to the world
16:28:43 <rm_work> ie exposing the serving port
16:28:47 <rm_work> is what i meant
16:28:58 <rm_work> which is non-viable
16:29:07 <johnsom> Not the world, just the range you are using for you VIP addresses
16:29:22 <johnsom> (the base ports are on the same range as the VIP)
16:29:26 <rm_work> yeah, but that means "any LB" and was rejected
16:29:39 <rm_work> so, no-go
16:29:45 <johnsom> Right. Did you answer my question, do you have FWaaS?
16:29:49 <rm_work> no we do not
16:30:04 <rm_work> we also do not have the ability to let users create private networks T_T
16:30:42 <rm_work> and humorously I'm 2 for 2, this is exactly the same issues we had at GD lol
16:30:45 <johnsom> Yeah, then at this point, there is no solution for that setup.
16:31:01 <rm_work> so either it's not that rare, or I manage to pick exactly the two companies with deployments like this
16:31:16 <johnsom> Ha, that later
16:31:27 <rm_work> i don't believe in those odds :D
16:32:27 <johnsom> Yeah, so you would need an RFE for any of the other changes I can think of. All of which really suck.
16:32:36 <rm_work> this issue was actually a decently relevant factor for the death of octavia in the GD deployment, and it worries me a lot here
16:32:48 <rm_work> and I really doubt it is just us
16:32:59 <rm_work> as funny as that would be :D
16:33:22 <johnsom> The one-armed solution would require a config option, then change the haproxy jinja to force the source to be VIP. This will have a negative impact on the performance and capacity of the LB.
16:33:48 <rm_work> one solution is to allow the user to pass us a SG just to attach it to the port -- then they can use that to allow traffic in
16:33:53 <rm_work> though it's a little janky
16:34:35 <rm_work> or i wonder if it is possible for a user to "allow" traffic via SG that they don't own -- if we exposed one for them
16:35:27 <rm_work> you can say "allow traffic from any port with security_group <ID>", right?
16:35:39 <rm_work> as a SG rule
16:35:48 <johnsom> The other would be (possibly, haven't tried it) to pass an SG ID to each member API create, then have the source ports added to that SG. In theory the neutron transitive-trust would magically make traffic flow. The downside is it would also open those arbitrary ports on our amphora source IPs.
16:36:35 <johnsom> Your second idea is what we can do with FWaaS if I remember, but I'm not sure neutron proper can do it.
16:39:00 <johnsom> I wish SGs had better support for "AND" than it does. Really that would solve a bunch of our problems.
16:39:01 <rm_work> hmm, i thought that was just part of the core SG stuff
16:39:17 <rm_work> that definitely worked at GD and I didn't think we deployed FWaaS
16:39:32 <rm_work> I'll check with Miguel and we can discuss it another time
16:39:39 <johnsom> Ok
16:39:50 <rm_work> You answered me question about drawbacks of the one-armed solution
16:40:08 <johnsom> ataraday_ You had a question about ciphers/suites?
16:40:14 <rm_work> though ... in reality, there's really only one physical NIC in use across all of these virtual NICs anyway, so I don't know how much it really affects throughput
16:40:43 <rm_work> (besides I guess the number of concurrent connections? though i don't think the member side will ever be the limiting factor there)
16:41:13 <johnsom> Well, that is a deployment flaw, but It's not just the nic. Don't forget you have TCP ports in use, queues in the kernel, etc. that have nothing to do with your NIC topology (though you should have more than one)
16:41:53 <ataraday_> johnsom, yes, I put comment on https://review.opendev.org/#/c/685337/ -  I look closer and we can use ssl-default-server-ciphersuites and ssl-default-server-ciphers in one config file. But ciphersuites available only since haproxy 1.8.20
16:42:14 <ataraday_> and for now we have 1.8.8
16:43:09 <rm_work> right, but given that there is one VIP, and it splits those connections to many members, I think it is impossible for that to be an issue on the backend side
16:43:37 <johnsom> Ah, interesting. We should probably request Ubuntu to update the available package. That said, we have code that can detect the version of HAProxy and make changes. This seems like a good candidate for that.
16:43:44 <rm_work> eugh, another thing going to 2.0 would resolve :D
16:44:36 <johnsom> ataraday_ https://github.com/openstack/octavia/blob/master/octavia/amphorae/backends/agent/api_server/haproxy_compatibility.py
16:45:14 <ataraday_> johnsom, thanks for pointing this!
16:45:21 <johnsom> We could expand that to remove the ciphersuite configuration line based on the available version. If it's not greater than 1.8.20 it probably doesn't support TLS 1.3 anyway.
16:46:05 <ataraday_> yeah, seems the way to do that
16:46:56 <johnsom> It looks like my code there only does major minor, so it may need to be expanded to look at the patch number too
16:48:28 <johnsom> Kind of lame they added that in a patch release reallly
16:49:13 <rm_work> yeah
16:49:21 <rm_work> alternatively, we had some plans to pre-cache this info
16:49:35 <rm_work> as a possible way to deal upfront with option validation for providers
16:50:00 <rm_work> we discussed this a bit at the summit, I believe it was in my summary, and there are more notes on the etherpad
16:50:01 <johnsom> Yeah, there is code for round tripping for the version too, but this seems straight forward for this case
16:51:29 <johnsom> #link https://github.com/openstack/octavia/blob/master/octavia/amphorae/drivers/haproxy/rest_api_driver.py#L78
16:51:35 <johnsom> #link https://github.com/openstack/octavia/blob/master/octavia/amphorae/backends/agent/api_server/haproxy_compatibility.py
16:51:42 <johnsom> Forgot we were in a meeting. lol
16:52:20 <johnsom> That L78 is the controller side query for version, but again, it might be more straight forward for this issue to just add it to the agent side adjustments.
16:53:40 <johnsom> Any other topics today?
16:56:11 <rm_work> reviews! reviews reviews! all reviews are useful!
16:56:46 <rm_work> only cores can +2, but the real power is in -1 and that's the same for everyone! reviews!
16:56:47 <johnsom> Yes please. I did a review-day recently, but we still have a bunch of patches that can use some reviews. +1's matter!
16:57:07 <rm_work> I guess I'm just thinking more negatively today :D
16:57:12 <johnsom> True, -1 matters more. lol
16:58:10 <johnsom> Ok, thanks for a great meeting!
16:58:15 <rm_work> I expect most code I write is going to have a bug or two that I didn't catch, and I'm counting on you guys to find them! it's a treasure hunt for bugs! the reward is internet brownie points! <3
16:58:30 <johnsom> #endmeeting