20:00:12 <johnsom> #startmeeting Octavia
20:00:13 <openstack> Meeting started Wed Sep 26 20:00:12 2018 UTC and is due to finish in 60 minutes.  The chair is johnsom. Information about MeetBot at http://wiki.debian.org/MeetBot.
20:00:14 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
20:00:16 <openstack> The meeting name has been set to 'octavia'
20:00:54 <johnsom> Hi folks
20:00:56 <ltomasbo> o/
20:00:59 <xgerman_> o/
20:01:18 <nmagnezi> o/
20:01:25 <johnsom> Pretty light agenda today
20:01:38 <johnsom> #topic Announcements
20:02:13 <johnsom> We are in the last few days to vote for the TC, so if you have not already please check your e-mail and vote for our future TC members.
20:03:09 <johnsom> I have also started the process to create octavia-lib as we discussed at the PTG.
20:03:27 <johnsom> The intent of octavia-lib is to support the provider drivers with a "library" style release.
20:03:46 <johnsom> Other than that, I don't think I have any announcements.  Anyone else?
20:03:48 <evgenyf> hi
20:03:49 <rm_work> o/
20:04:32 <johnsom> #topic Brief progress reports / bugs needing review
20:05:27 <johnsom> I have mostly been busy with work from the PTG. I have created stories for the backend re-encryption work, and started a bunch of the gate test cleanup I started at the PTG.
20:05:27 <evgenyf> I pushed 2 changes for provider driver framework, fixing to bugs
20:05:54 <xgerman_> #link https://review.openstack.org/#/c/587505/
20:06:11 <evgenyf> #link https://review.openstack.org/#/c/605382/
20:06:15 <xgerman_> I rewrote it to appease my critics so review…
20:06:31 <evgenyf> #link https://review.openstack.org/#/c/605376/
20:06:32 <ltomasbo> I reported on bug on a (seems to be) race on DB access: https://storyboard.openstack.org/#!/story/2003833
20:06:48 <johnsom> Right now I am working on getting multi-node working again after the zuul v3 changes. This is a bit of a challenge as most teams are still using the legacy multi-node code.  Sorry for the channel noise as I work through getting this stuff to work.
20:06:50 <xgerman_> also playing with some refactor of the AAP driver: https://review.openstack.org/#/c/604479/
20:07:31 <johnsom> I have also started to backport the HM performance patch. Rocky is posted, Queens and Pike will be later today as those are not clean backports and need some careful work.
20:08:28 <johnsom> Next up is fixing the Active/Standby topology with IPv6 VIPs, then I plan to focus on flavors.
20:08:53 <johnsom> Whenever the octavia-lib repo is ready I will also work on moving the driver code out to the octavia-lib.
20:09:27 <johnsom> Oh, and I hope to announce the neutron-lbaas end of life this week as we agreed at the PTG.
20:10:35 <johnsom> Any other updates this week?  (Thanks to you all that provided them!)
20:11:01 <evgenyf> I created a story related to provider driver loading function driver_factory.get_driver(), please review and share your feedback
20:11:07 <evgenyf> #link https://storyboard.openstack.org/#!/story/2003875
20:11:12 <nmagnezi> Both cgoncalves and myself are on a leave this week, so no updates from our side
20:11:51 <xgerman_> yeah, about cgoncalves he owes me a +@
20:12:17 <evgenyf> johnsom: is there something to review for octavia_lib?
20:12:23 <johnsom> evgenyf Yeah, the statement that a driver has "state" that isn't stored outside the API process (driver DB or in an appliance) is very worrisome
20:12:46 <johnsom> It is waiting on governance and infra reviews:
20:12:56 <johnsom> #link https://review.openstack.org/604890
20:13:03 <johnsom> #link https://review.openstack.org/605273
20:13:16 <johnsom> #link https://review.openstack.org/604889
20:13:45 <johnsom> Those are the first steps, once the repo is created I will setup the cookiecutter, pypi, etc. for it.
20:14:12 <johnsom> I'm hoping it will be ready to start moving the driver code out by next week.
20:15:09 <johnsom> #topic Talk about VIP security groups
20:15:32 <johnsom> Ok, next on the agenda (sorry it was late this week) is the SG discussion we held over from last week.
20:15:55 <ltomasbo> did anyone investigate other possible solutions for it?
20:15:56 <johnsom> Does anyone have any research info to share?
20:16:23 <johnsom> Yeah, that is my question to the team
20:16:51 <ltomasbo> I did (a week ago) test applying the SG to the VIP, and that is not working on ml2/ovs driver, as the VIP port is just attached to the amphora as an allowed_address_pair
20:17:22 <ltomasbo> and the SG applied is the one on the amphora port instead
20:17:44 <xgerman_> mmh, so we would need to handle the applying ourselves
20:18:03 <ltomasbo> I, for instance, tested with ovn-octavia driver, and as they used the VIP in a different way, there it is possible to apply it there
20:18:27 <xgerman_> well, we like to be uniform among all drivers
20:19:14 <ltomasbo> yep, actually in ovn-octavia the SG on the VIP port belongs to the tenant actually
20:19:45 <ltomasbo> as they do not create the VIP port, and therefore it just gets the default one (which seems to be wrong and we opened a bug related to it)
20:19:54 <johnsom> Yeah, the VIP belongs to the tenant in the reference driver too.
20:20:02 <xgerman_> +1
20:20:03 <ltomasbo> https://bugs.launchpad.net/networking-ovn/+bug/1794020
20:20:03 <openstack> Launchpad bug 1794020 in networking-ovn "Security Group Creation in octavia's OVN Driver" [Undecided,New] - Assigned to Reedip (reedip-banerjee)
20:20:07 <johnsom> Which was a compromise that is really a problem now
20:20:50 <xgerman_> nah, it let the user assign a FIP so good times
20:22:05 <johnsom> Right, but now users can break that VIP port as they own it.
20:22:45 <ltomasbo> johnsom, break it as in what sense?
20:23:02 <xgerman_> they delete it out of band with their purge script
20:23:06 <ltomasbo> johnsom, you mean deleting it?
20:23:10 <xgerman_> yep
20:23:45 <xgerman_> or whatever else comes to their mind. I just had a user who deleted the Octavia VM flavor and was surprised
20:24:07 <ltomasbo> xD
20:24:19 <ltomasbo> I didn't know the flavor was visible to the tenant...
20:24:37 <xgerman_> it is not but some are admin and do silly things
20:24:41 <ltomasbo> usually tenants cannot create/modify flavors
20:24:53 <ltomasbo> ahh, sure, but there is nothing you can do about that!
20:25:09 <xgerman_> yep, but they called me in ayway :-(
20:25:14 <ltomasbo> xD
20:25:21 <johnsom> Yeah, it is mostly users (non-admin) that are running "apply to all" scripts that end up causing trouble.
20:25:54 <ltomasbo> johnsom, but there is not much you can do with that port besides deleting it
20:26:09 <ltomasbo> you can change the SG, but it has no effect whatsoever
20:26:16 <johnsom> Really we want to remove their visibility to it at all
20:26:40 <ltomasbo> yep, it will be nice to not have it listed... but it is still consuming an IP from tenant network
20:26:51 <xgerman_> I used to be a big fan to give users lot’s of freedom until I met them…
20:27:24 <xgerman_> ok, for the VIP we have a plan to create a shadow VIP holding the IP…
20:27:39 <rm_work> yeah, that was the one i was in favor of
20:27:50 <rm_work> assuming it works technically
20:28:08 <rm_work> we already have a VIP table, just include another column for the shadow_vip_id
20:28:10 <xgerman_> pretty sure if we make it an AAP ip it *should* be fine
20:28:15 <rm_work> and create an extra thing, and done
20:28:20 <xgerman_> boo,
20:28:25 <xgerman_> boom
20:28:36 <rm_work> well, we need to account for non-AAP drivers I suppose
20:28:47 <xgerman_> yep
20:28:58 <xgerman_> revert to the old model ;-)
20:29:04 <johnsom> Yeah, not sure it should be an AAP port as it's not bound, so not sure the win on that
20:29:14 <rm_work> ah does it not have t be
20:29:19 <rm_work> to share the IP
20:29:39 <rm_work> can we create a second port with the same IP that's not AAP enabled, as long as we never bind it?
20:29:52 <rm_work> i thought just creating it required AAP
20:29:52 <johnsom> Hmm, not sure if we don't actually use the port
20:29:58 <rm_work> but maybe not?
20:29:58 <ltomasbo> yep, it will not always be a AAP port, in fact, in ovn-driver it is not an AAP
20:30:11 <rm_work> but i mean, we can leave that up to the driver
20:30:18 <rm_work> we'd be creating it in our normal vip_allocate method
20:30:21 <rm_work> in the AAP driver
20:30:29 <rm_work> so really we're just talking about an AAP driver enhancement anyway
20:30:35 <rm_work> because this doesn't even necessarily apply to others
20:30:37 <xgerman_> +1
20:30:38 <rm_work> so i think it's not a huge problem
20:31:26 <xgerman_> but back to the problem at hand: don’t think we can have a shadow SG
20:31:53 <ltomasbo> if different drivers can implement octavia resources in different ways, how do you ensure they belong to the admin? that will be driver-dependant, right?
20:32:18 <xgerman_> yes, indeed
20:32:21 <ltomasbo> could we make the driver to select to whom the resources belong?
20:32:42 <ltomasbo> for instance, VIP could also belong to the admin (on octavia.conf) if the customer does not need to assign floating ips
20:32:50 <ltomasbo> similarly for the SG
20:33:11 <xgerman_> yes, that sounds interesting — might even be a flavor thing
20:33:21 <johnsom> Well, there should be some consistency. Especially around the API.
20:33:40 <rm_work> yes, in my FLIP driver, i do not give the VIP to the tenant
20:33:42 <johnsom> For example, we are talking about adding the ability for a user to specify a security group via the API
20:33:43 <rm_work> for example :)
20:34:01 <ltomasbo> johnsom, I agree that is the perfect solution!
20:34:27 <ltomasbo> johnsom, but it is not there yet... :/
20:35:09 <ltomasbo> johnsom, do you think applying the sg via API could be completed in this cycle?
20:35:25 <johnsom> What we were supposed to be researching is how to handle that security group. Ideally we would add that to the port, it would AND with our existing SG and we are in good shape. But the question is, in neutron ports, is that how multiple security groups work?
20:35:33 <ltomasbo> johnsom, and could we do something about previous releases (as API changes are not really backportable)
20:36:18 <ltomasbo> johnsom, if there is a rule accepting it, it goes through
20:36:43 <ltomasbo> johnsom, everything is denied by default, and then if some rule matches, it goes through
20:37:03 <johnsom> So, if we have two SGs on a port, A and B.   B says "only port 80" and A says "open everything", the resultant is "open everything"?
20:37:11 <ltomasbo> yep
20:37:23 <johnsom> Yeah, so that is exactly what we don't want
20:37:59 <johnsom> What we want is an AND, where the resultant is "only port 80"
20:38:31 <xgerman_> we can always strip the group the user provides of all rules… but that’s invasive
20:38:41 <johnsom> And they can just change it
20:38:43 <xgerman_> and not enforcable
20:39:05 <ltomasbo> another option is update the listener API, with a more similar syntax to SG rule creation
20:39:08 <johnsom> The idea with the AND is it would allow the user to narrow the scope, but not expand it
20:39:16 <ltomasbo> so that it allows you to create more fine grain rules
20:39:31 <ltomasbo> and then, ther will be only one SG, but with several rules
20:39:45 <johnsom> Yeah, that was the other proposal that was previously agreed to. Add an ACL API to the LB
20:40:53 <johnsom> The third was using FWaaS, which was planning to support the "AND" option, but I don't think it does yet. Is that correct xgerman_?
20:41:15 <ltomasbo> plus it adds another dependency
20:41:22 <xgerman_> it supports it but it’s not enabled by default
20:42:25 <johnsom> Yeah, the additional dependency is not ideal either
20:43:13 <ltomasbo> and that will be T release most probably?
20:43:13 <xgerman_> we won’t have an ideal outcome…
20:43:36 <xgerman_> well, the peices are there you just need to enable it
20:43:48 <ltomasbo> yep, that is why I wanted to have this small revert-able fix for steain and previous releases until ACL or similar is in place
20:44:11 <ltomasbo> *stein
20:44:55 <xgerman_> it’s hard to add something and then take it away
20:45:22 <ltomasbo> but if it is disable by default... and just enable upon need...
20:45:41 <ltomasbo> you can just disable it once the right solution is enabled
20:46:15 <ltomasbo> but yep, another bad things is that if there is a solution in place (even if it is not good) there is less urgent for the proper fix
20:49:00 <johnsom> Does anyone know how "remote_group_id" works?
20:49:08 <johnsom> In a security group rule?
20:49:15 <ltomasbo> yep
20:49:28 <ltomasbo> that basically enables traffic from ports that has that SG id
20:49:46 <ltomasbo> so if the rules says remote_group_id=SG_A
20:49:59 <ltomasbo> all the ports that has SG_A in their security group can access it
20:50:56 <johnsom> Ah, ok.
20:51:16 <ltomasbo> what I don't know is if, for instance, if you create a SG on the tenant, but add a dummy rule inside as an admin
20:51:25 <xgerman_> yeah, all the SG build a transitive group and allow all traffic between each other
20:51:29 <ltomasbo> that will make the user imposible to remove it
20:51:40 <ltomasbo> ahh, well, the amphora is using it, so it cannot remove it anyway
20:51:42 <johnsom> Yeah, so unless we are willing to give up SG control all together, I think the only option (short of changing neutron SGs) is to add the ACLs to the API.
20:53:39 <johnsom> The other topic discussed was letting the SG go, making it tenant facing, and relying on iptables in the amphora kernel or the provider.
20:54:24 <ltomasbo> johnsom, how do you ensure that at the moment?
20:54:43 <ltomasbo> user could still open any port on the SG by creating the right listener
20:54:53 <ltomasbo> so giving away the SG will not change any of that
20:55:08 <johnsom> We don't. It's just the "open" port at the moment. This was because of the performance overhead of adding conntrack to the amphora...
20:55:17 <ltomasbo> rather the opossite, limiting the access to the amphora to the ports on the same subnet or form the same remote_group_id
20:57:06 <xgerman_> we can’t assume users do what we ant
20:58:00 <ltomasbo> sure! I just meant that they can already open any port they want by creating a listener
20:58:06 <xgerman_> I know people were reading out our SG. manipulating it, and using it for access restrictions
20:58:20 <johnsom> Currently, they can restrict the source by setting up tenant networks, but I get that it is less than ideal
20:58:57 <johnsom> Right, but with the listener we know the right thing is there and on that port.
20:59:13 <xgerman_> yep, we will listen ;-)
20:59:40 <ltomasbo> :)
20:59:48 <johnsom> Darn, we are out of time today....
21:00:09 <ltomasbo> thanks for discussing it!
21:00:09 <johnsom> Thanks folks
21:00:16 <johnsom> #endmeeting