Wednesday, 2018-09-26

*** celebdor has quit IRC00:16
*** abaindur has joined #openstack-lbaas00:17
*** JudeCross has joined #openstack-lbaas00:23
openstackgerritMichael Johnson proposed openstack/octavia-tempest-plugin master: Add v2 two-node scenario test
*** JudeCross has quit IRC01:05
*** hongbin has joined #openstack-lbaas01:39
*** ramishra has joined #openstack-lbaas03:26
*** hongbin has quit IRC03:59
*** irenab has quit IRC04:01
openstackgerritMerged openstack/octavia master: Pass through DIB_LOCAL_ELEMENTS from localrc
*** pcaruana has joined #openstack-lbaas04:14
*** yamamoto has quit IRC04:31
*** yamamoto has joined #openstack-lbaas04:31
*** pcaruana has quit IRC04:38
*** irenab has joined #openstack-lbaas04:39
openstackgerritMerged openstack/octavia-tempest-plugin master: Use the infra mirrors for DIB
*** reedipb has joined #openstack-lbaas05:35
*** pcaruana has joined #openstack-lbaas05:43
*** JudeCross has joined #openstack-lbaas06:03
*** JudeCross has quit IRC06:07
*** blake has joined #openstack-lbaas06:08
*** blake has quit IRC06:39
*** blake has joined #openstack-lbaas06:40
*** blake has quit IRC06:45
*** rcernin has quit IRC07:02
*** velizarx has joined #openstack-lbaas07:12
*** velizarx has quit IRC07:31
*** celebdor has joined #openstack-lbaas07:46
*** JudeCross has joined #openstack-lbaas07:55
*** rcernin has joined #openstack-lbaas07:56
*** velizarx has joined #openstack-lbaas08:07
*** celebdor has quit IRC08:12
*** Emine has joined #openstack-lbaas08:17
*** velizarx has quit IRC08:24
*** velizarx has joined #openstack-lbaas08:28
*** abaindur has quit IRC08:30
openstackgerritMerged openstack/octavia master: Fix the API list performance regression
*** abaindur has joined #openstack-lbaas08:48
*** rcernin has quit IRC09:16
*** salmankhan has joined #openstack-lbaas09:17
*** JudeCross has quit IRC09:27
openstackgerritEvgeny Fedoruk proposed openstack/octavia master: Fix fot utils LB DM transaction function
openstackgerritEvgeny Fedoruk proposed openstack/octavia master: Fix for utils LB DM transformation function
openstackgerritEvgeny Fedoruk proposed openstack/octavia master: Fix for utils LB DM transformation function
openstackgerritEvgeny Fedoruk proposed openstack/octavia master: Fixing data model to_dict() recursive function
*** celebdor has joined #openstack-lbaas09:56
velizarxHi folks! I have some strange question. I'm creating the table of relations ( between listener.protocol and pool.protocol to understand which protocols work with. And I see what listener.protocol = HTTPS it's "mode tcp" in haproxy configuration as well as pool.protocol = HTTPS it's "mode tcp". What the difference between protocol HTTPS and protocol TCP in this case? Why Octavia provides HTTPS like alias for TCP?09:59
*** velizarx has quit IRC10:01
*** velizarx has joined #openstack-lbaas10:02
sapd1velizarx: I think to clear with end user. Somebody do not know use TCP if they want to setup HTTPS Load Balancer ( I mean frontend tcp, backend tcp and server backend use HTTPS).10:09
*** reedipb has quit IRC10:11
velizarxsapd1, Hm, I think for the end user it adds confusion. Let's wait johnsom10:15
*** abaindur has quit IRC10:26
*** abaindur has joined #openstack-lbaas10:27
rm_workThey should use HTTPS if they want HTTPS backend members?10:27
*** yamamoto has quit IRC10:35
velizarxrm_work, In the haproxy configuration HTTPS looks the same TCP (see configs examples:, and pool.protocol == TCP works fine. And it's the main reason why I asked question:)10:42
sapd1velizarx: To clear for end user :D10:43
*** yamamoto has joined #openstack-lbaas11:00
*** pcaruana has quit IRC11:15
*** irenab has quit IRC11:34
*** oanson has quit IRC11:35
*** savvas has joined #openstack-lbaas11:55
*** velizarx has quit IRC12:12
*** velizarx has joined #openstack-lbaas12:28
*** aojea_ has joined #openstack-lbaas12:29
*** yamamoto has quit IRC12:35
*** blake has joined #openstack-lbaas12:42
*** blake has quit IRC12:47
*** yamamoto has joined #openstack-lbaas13:13
*** yamamoto has quit IRC13:21
*** yamamoto has joined #openstack-lbaas13:21
*** yamamoto has quit IRC14:03
*** yamamoto has joined #openstack-lbaas14:04
*** yamamoto has quit IRC14:09
*** yamamoto has joined #openstack-lbaas14:52
*** pcaruana has joined #openstack-lbaas15:01
*** velizarx has quit IRC15:01
*** velizarx has joined #openstack-lbaas15:06
johnsomThe HTTPS protocol is for HTTPS pass-through use.15:16
velizarxjohnsom, Thx for your answer. AFAIU difference between TCP and HTTPS pass-through it's monitoring options, but LB doesn't see L7 traffic. And why HTTP pass-through the best?15:22
*** celebdor has quit IRC15:31
johnsomHTTPS pass through is not necessary best as we can’t make good load balancing decisions with it.15:32
johnsomThat said, we envisioned using the TLS session ID in HTTPS pass through to do session persistence. Though I am not sure that ever got implemented.15:33
johnsomThat differentiates HTTPS from just TCP15:34
*** savvas has quit IRC15:37
*** celebdor has joined #openstack-lbaas15:39
*** numans has joined #openstack-lbaas15:46
openstackgerritboden proposed openstack/neutron-lbaas master: add local tox targets for pep8 and py3
*** aojea_ has quit IRC15:55
*** amuller has joined #openstack-lbaas15:56
*** aojea has joined #openstack-lbaas15:56
openstackgerritboden proposed openstack/neutron-lbaas master: use common rpc and exceptions from neutron-lib
*** ramishra has quit IRC16:09
*** blake has joined #openstack-lbaas16:09
*** celebdor has quit IRC16:12
*** Emine has quit IRC16:16
openstackgerritMichael Johnson proposed openstack/octavia-tempest-plugin master: DNM: Testing bionic nodes
*** blake has quit IRC16:28
*** blake has joined #openstack-lbaas16:28
*** blake has quit IRC16:29
*** velizarx has quit IRC16:32
openstackgerritMichael Johnson proposed openstack/octavia-tempest-plugin master: Add v2 two-node scenario test
*** velizarx has joined #openstack-lbaas16:35
*** velizarx has quit IRC16:55
*** salmankhan has quit IRC17:06
KeithMnemonicQuestion on act/stdby in pike. I deploy an LB, and if I delete one of the amphoras, it creates a new amphor expected. However when I go to delete the LB,  I get an message about immutable and the LB goes to ERROR and cannot be removed by the CLI. Has anyone seen this before?17:35
xgerman_Yes, I have seen that. Mosy of my problems stem from ports not being cleaned up properly - there is a patch in a later versions to aggressively delete ports when lb delete.17:36
KeithMnemonicis that an easy backport to pike?17:37
johnsomI have not seen that. Can you give us more information?17:38
KeithMnemonicsure, what do  you want17:38
johnsomDetails from the controller worker log file17:38
KeithMnemonicok let me go repoduce it again now17:38
johnsomThere should be some kind of error message of why it happened.17:39
KeithMnemonicso my steps, create an lb, once it is active, do a nova delete on one amp, wait for the new amp to spawn,connect,,,, and then delete the lb17:39
johnsomYeah, that should work just fine17:40
*** abaindur has quit IRC17:49
KeithMnemonicok almost ready to test the delete, i noticed that even though i have an ssh amp key, for some reason it is not working on the recovered amphora17:49
KeithMnemoniconly on the originals17:49
johnsomThat should not happen either17:50
KeithMnemonicthe console show diff keys17:53
johnsomHost keys, but it should be the same nova keypair key. Are you sure all of your controllers have the same configuration files?17:53
KeithMnemonicso it may not happen, but it does. let me go verfiy they should17:54
KeithMnemonici have the key name in worker.conf under controller_worker17:54
KeithMnemonicthat setting is identical17:56
KeithMnemonicsince this is being spawned by the health manager, does the key name need to be other places as well?17:56
johnsomNo it shares that config setting and the key is stored in nova, so should pull the same key. We don't actually load that, nova does.17:58
KeithMnemonicok one fish at a time ;-)18:00
KeithMnemoniclet me do the delete18:00
KeithMnemonicthis time it worked, maybe i was doing the delete too fast18:01
johnsomWell, it locks the objects to you can't really do it too fast, it should stop you.18:02
*** celebdor has joined #openstack-lbaas18:05
KeithMnemonicok so on the ssh thing, it does not find the key18:10
KeithMnemonicthat means i need that entry in another conf file or section?18:11
johnsomIt is the exact same configuration setting, but it needs to be in the config files for CW, HM, and HK processes. I don't know if you are running separate config files for each process or not, but that is one of the shared settings.18:14
KeithMnemonici bet that is my issue, sep conf files, fixing that now18:15
*** evgenyf has joined #openstack-lbaas18:18
*** pcaruana has quit IRC18:24
*** yamamoto has quit IRC18:26
*** yamamoto has joined #openstack-lbaas18:26
*** maciejjozefczyk has joined #openstack-lbaas18:33
*** celebdor has quit IRC18:36
KeithMnemonicok conf file fixed the ssh issue and now i cannot get the imuutable message18:39
KeithMnemoniclet me try one mor test18:39
*** abaindur has joined #openstack-lbaas18:40
*** rtjure has quit IRC19:05
baffleRunning octavia:rocky with liberty nova and kilo neutron without native allowed-address-pairs support suprisingly works after some ugly modifications...19:07
*** JudeCross has joined #openstack-lbaas19:07
*** rtjure has joined #openstack-lbaas19:08
*** salmankhan has joined #openstack-lbaas19:17
*** salmankhan has quit IRC19:22
*** Emine has joined #openstack-lbaas19:26
xgerman_baffle: tell us more… maybe a patch?19:30
*** yboaron_ has joined #openstack-lbaas19:35
rm_workhah yes I run octavia:master with liberty openstack and it works great :P though I have AAP19:38
rm_workbuuuut i also use a totally different network driver, so :P19:38
abainduris there any reason the flavor ID is fixed via config?19:50
abaindurrather than being allowed to be specified in the loadbalancer create call?19:50
abaindurA one size fits all loadbalancer solution really isnt ideal... we'd like to scale them up or down as needed individually19:50
*** maciejjozefczyk has quit IRC19:54
johnsomabaindur That is coming with flavors20:00
johnsom#startmeeting Octavia20:00
openstackMeeting started Wed Sep 26 20:00:12 2018 UTC and is due to finish in 60 minutes.  The chair is johnsom. Information about MeetBot at
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.20:00
*** openstack changes topic to " (Meeting topic: Octavia)"20:00
openstackThe meeting name has been set to 'octavia'20:00
johnsomHi folks20:00
johnsomPretty light agenda today20:01
johnsom#topic Announcements20:01
*** openstack changes topic to "Announcements (Meeting topic: Octavia)"20:01
*** abaindur has quit IRC20:01
johnsomWe are in the last few days to vote for the TC, so if you have not already please check your e-mail and vote for our future TC members.20:02
johnsomI have also started the process to create octavia-lib as we discussed at the PTG.20:03
johnsomThe intent of octavia-lib is to support the provider drivers with a "library" style release.20:03
johnsomOther than that, I don't think I have any announcements.  Anyone else?20:03
*** abaindur has joined #openstack-lbaas20:03
johnsom#topic Brief progress reports / bugs needing review20:04
*** openstack changes topic to "Brief progress reports / bugs needing review (Meeting topic: Octavia)"20:04
johnsomI have mostly been busy with work from the PTG. I have created stories for the backend re-encryption work, and started a bunch of the gate test cleanup I started at the PTG.20:05
evgenyfI pushed 2 changes for provider driver framework, fixing to bugs20:05
xgerman_I rewrote it to appease my critics so review…20:06
ltomasboI reported on bug on a (seems to be) race on DB access:!/story/200383320:06
johnsomRight now I am working on getting multi-node working again after the zuul v3 changes. This is a bit of a challenge as most teams are still using the legacy multi-node code.  Sorry for the channel noise as I work through getting this stuff to work.20:06
xgerman_also playing with some refactor of the AAP driver:
johnsomI have also started to backport the HM performance patch. Rocky is posted, Queens and Pike will be later today as those are not clean backports and need some careful work.20:07
johnsomNext up is fixing the Active/Standby topology with IPv6 VIPs, then I plan to focus on flavors.20:08
johnsomWhenever the octavia-lib repo is ready I will also work on moving the driver code out to the octavia-lib.20:08
johnsomOh, and I hope to announce the neutron-lbaas end of life this week as we agreed at the PTG.20:09
*** nmagnezi is now known as nmagnezi_meeting20:09
*** nmagnezi_meeting is now known as nmagnezi20:10
johnsomAny other updates this week?  (Thanks to you all that provided them!)20:10
evgenyfI created a story related to provider driver loading function driver_factory.get_driver(), please review and share your feedback20:11
nmagneziBoth cgoncalves and myself are on a leave this week, so no updates from our side20:11
xgerman_yeah, about cgoncalves he owes me a +@20:11
evgenyfjohnsom: is there something to review for octavia_lib?20:12
johnsomevgenyf Yeah, the statement that a driver has "state" that isn't stored outside the API process (driver DB or in an appliance) is very worrisome20:12
johnsomIt is waiting on governance and infra reviews:20:12
johnsomThose are the first steps, once the repo is created I will setup the cookiecutter, pypi, etc. for it.20:13
johnsomI'm hoping it will be ready to start moving the driver code out by next week.20:14
johnsom#topic Talk about VIP security groups20:15
*** openstack changes topic to "Talk about VIP security groups (Meeting topic: Octavia)"20:15
johnsomOk, next on the agenda (sorry it was late this week) is the SG discussion we held over from last week.20:15
ltomasbodid anyone investigate other possible solutions for it?20:15
johnsomDoes anyone have any research info to share?20:15
*** abaindur has quit IRC20:16
johnsomYeah, that is my question to the team20:16
ltomasboI did (a week ago) test applying the SG to the VIP, and that is not working on ml2/ovs driver, as the VIP port is just attached to the amphora as an allowed_address_pair20:16
*** abaindur has joined #openstack-lbaas20:17
ltomasboand the SG applied is the one on the amphora port instead20:17
xgerman_mmh, so we would need to handle the applying ourselves20:17
ltomasboI, for instance, tested with ovn-octavia driver, and as they used the VIP in a different way, there it is possible to apply it there20:18
xgerman_well, we like to be uniform among all drivers20:18
ltomasboyep, actually in ovn-octavia the SG on the VIP port belongs to the tenant actually20:19
ltomasboas they do not create the VIP port, and therefore it just gets the default one (which seems to be wrong and we opened a bug related to it)20:19
johnsomYeah, the VIP belongs to the tenant in the reference driver too.20:19
openstackLaunchpad bug 1794020 in networking-ovn "Security Group Creation in octavia's OVN Driver" [Undecided,New] - Assigned to Reedip (reedip-banerjee)20:20
johnsomWhich was a compromise that is really a problem now20:20
xgerman_nah, it let the user assign a FIP so good times20:20
johnsomRight, but now users can break that VIP port as they own it.20:22
ltomasbojohnsom, break it as in what sense?20:22
xgerman_they delete it out of band with their purge script20:23
ltomasbojohnsom, you mean deleting it?20:23
xgerman_or whatever else comes to their mind. I just had a user who deleted the Octavia VM flavor and was surprised20:23
ltomasboI didn't know the flavor was visible to the tenant...20:24
xgerman_it is not but some are admin and do silly things20:24
ltomasbousually tenants cannot create/modify flavors20:24
ltomasboahh, sure, but there is nothing you can do about that!20:24
xgerman_yep, but they called me in ayway :-(20:25
johnsomYeah, it is mostly users (non-admin) that are running "apply to all" scripts that end up causing trouble.20:25
ltomasbojohnsom, but there is not much you can do with that port besides deleting it20:25
ltomasboyou can change the SG, but it has no effect whatsoever20:26
johnsomReally we want to remove their visibility to it at all20:26
ltomasboyep, it will be nice to not have it listed... but it is still consuming an IP from tenant network20:26
xgerman_I used to be a big fan to give users lot’s of freedom until I met them…20:26
xgerman_ok, for the VIP we have a plan to create a shadow VIP holding the IP…20:27
rm_workyeah, that was the one i was in favor of20:27
rm_workassuming it works technically20:27
rm_workwe already have a VIP table, just include another column for the shadow_vip_id20:28
xgerman_pretty sure if we make it an AAP ip it *should* be fine20:28
rm_workand create an extra thing, and done20:28
rm_workwell, we need to account for non-AAP drivers I suppose20:28
xgerman_revert to the old model ;-)20:28
johnsomYeah, not sure it should be an AAP port as it's not bound, so not sure the win on that20:29
rm_workah does it not have t be20:29
rm_workto share the IP20:29
rm_workcan we create a second port with the same IP that's not AAP enabled, as long as we never bind it?20:29
rm_worki thought just creating it required AAP20:29
johnsomHmm, not sure if we don't actually use the port20:29
rm_workbut maybe not?20:29
ltomasboyep, it will not always be a AAP port, in fact, in ovn-driver it is not an AAP20:29
rm_workbut i mean, we can leave that up to the driver20:30
rm_workwe'd be creating it in our normal vip_allocate method20:30
rm_workin the AAP driver20:30
rm_workso really we're just talking about an AAP driver enhancement anyway20:30
rm_workbecause this doesn't even necessarily apply to others20:30
rm_workso i think it's not a huge problem20:30
xgerman_but back to the problem at hand: don’t think we can have a shadow SG20:31
ltomasboif different drivers can implement octavia resources in different ways, how do you ensure they belong to the admin? that will be driver-dependant, right?20:31
xgerman_yes, indeed20:32
ltomasbocould we make the driver to select to whom the resources belong?20:32
ltomasbofor instance, VIP could also belong to the admin (on octavia.conf) if the customer does not need to assign floating ips20:32
ltomasbosimilarly for the SG20:32
xgerman_yes, that sounds interesting — might even be a flavor thing20:33
johnsomWell, there should be some consistency. Especially around the API.20:33
rm_workyes, in my FLIP driver, i do not give the VIP to the tenant20:33
johnsomFor example, we are talking about adding the ability for a user to specify a security group via the API20:33
rm_workfor example :)20:33
ltomasbojohnsom, I agree that is the perfect solution!20:34
ltomasbojohnsom, but it is not there yet... :/20:34
ltomasbojohnsom, do you think applying the sg via API could be completed in this cycle?20:35
johnsomWhat we were supposed to be researching is how to handle that security group. Ideally we would add that to the port, it would AND with our existing SG and we are in good shape. But the question is, in neutron ports, is that how multiple security groups work?20:35
ltomasbojohnsom, and could we do something about previous releases (as API changes are not really backportable)20:35
ltomasbojohnsom, if there is a rule accepting it, it goes through20:36
ltomasbojohnsom, everything is denied by default, and then if some rule matches, it goes through20:36
johnsomSo, if we have two SGs on a port, A and B.   B says "only port 80" and A says "open everything", the resultant is "open everything"?20:37
johnsomYeah, so that is exactly what we don't want20:37
johnsomWhat we want is an AND, where the resultant is "only port 80"20:37
xgerman_we can always strip the group the user provides of all rules… but that’s invasive20:38
johnsomAnd they can just change it20:38
xgerman_and not enforcable20:38
ltomasboanother option is update the listener API, with a more similar syntax to SG rule creation20:39
johnsomThe idea with the AND is it would allow the user to narrow the scope, but not expand it20:39
ltomasboso that it allows you to create more fine grain rules20:39
ltomasboand then, ther will be only one SG, but with several rules20:39
johnsomYeah, that was the other proposal that was previously agreed to. Add an ACL API to the LB20:39
johnsomThe third was using FWaaS, which was planning to support the "AND" option, but I don't think it does yet. Is that correct xgerman_?20:40
ltomasboplus it adds another dependency20:41
xgerman_it supports it but it’s not enabled by default20:41
johnsomYeah, the additional dependency is not ideal either20:42
ltomasboand that will be T release most probably?20:43
xgerman_we won’t have an ideal outcome…20:43
xgerman_well, the peices are there you just need to enable it20:43
ltomasboyep, that is why I wanted to have this small revert-able fix for steain and previous releases until ACL or similar is in place20:43
*** amuller has quit IRC20:44
xgerman_it’s hard to add something and then take it away20:44
ltomasbobut if it is disable by default... and just enable upon need...20:45
ltomasboyou can just disable it once the right solution is enabled20:45
ltomasbobut yep, another bad things is that if there is a solution in place (even if it is not good) there is less urgent for the proper fix20:46
johnsomDoes anyone know how "remote_group_id" works?20:49
johnsomIn a security group rule?20:49
ltomasbothat basically enables traffic from ports that has that SG id20:49
ltomasboso if the rules says remote_group_id=SG_A20:49
ltomasboall the ports that has SG_A in their security group can access it20:49
johnsomAh, ok.20:50
ltomasbowhat I don't know is if, for instance, if you create a SG on the tenant, but add a dummy rule inside as an admin20:51
xgerman_yeah, all the SG build a transitive group and allow all traffic between each other20:51
ltomasbothat will make the user imposible to remove it20:51
ltomasboahh, well, the amphora is using it, so it cannot remove it anyway20:51
johnsomYeah, so unless we are willing to give up SG control all together, I think the only option (short of changing neutron SGs) is to add the ACLs to the API.20:51
johnsomThe other topic discussed was letting the SG go, making it tenant facing, and relying on iptables in the amphora kernel or the provider.20:53
ltomasbojohnsom, how do you ensure that at the moment?20:54
ltomasbouser could still open any port on the SG by creating the right listener20:54
ltomasboso giving away the SG will not change any of that20:54
johnsomWe don't. It's just the "open" port at the moment. This was because of the performance overhead of adding conntrack to the amphora...20:55
ltomasborather the opossite, limiting the access to the amphora to the ports on the same subnet or form the same remote_group_id20:55
xgerman_we can’t assume users do what we ant20:57
ltomasbosure! I just meant that they can already open any port they want by creating a listener20:58
xgerman_I know people were reading out our SG. manipulating it, and using it for access restrictions20:58
johnsomCurrently, they can restrict the source by setting up tenant networks, but I get that it is less than ideal20:58
johnsomRight, but with the listener we know the right thing is there and on that port.20:58
xgerman_yep, we will listen ;-)20:59
johnsomDarn, we are out of time today....20:59
*** evgenyf has quit IRC21:00
ltomasbothanks for discussing it!21:00
johnsomThanks folks21:00
*** openstack changes topic to "OpenStack PTG etherpad:"21:00
openstackMeeting ended Wed Sep 26 21:00:16 2018 UTC.  Information about MeetBot at . (v 0.1.4)21:00
openstackMinutes (text):
ltomasboappreciated the time!21:00
*** aojea has quit IRC21:01
johnsomI will put it on next weeks agenda as well. I think we are really down to two options we should discuss more.21:01
rm_workwait, the remote_group_id thing sounded like it'd work perfectly?21:12
rm_worksorry i had to go afk for a minute right when we started discussing that T_T21:12
xgerman_mmh, there was one snag people hit. Not sure if it can admin owned to work…21:14
johnsomI thought it didn't solve the "I want to limit access to my VIP to only be from"21:15
*** yboaron_ has quit IRC21:28
*** blake has joined #openstack-lbaas21:29
xgerman_well, you will need to assign that SG to the nodes you want to allow access from22:00
*** rcernin has joined #openstack-lbaas22:35
*** rcernin has quit IRC22:53
*** rcernin has joined #openstack-lbaas22:55
*** blake_ has joined #openstack-lbaas23:02
*** blake has quit IRC23:05
*** colby_ has quit IRC23:16
openstackgerritGerman Eichberger proposed openstack/octavia master: [WIP] Refactor the pluggin of the VIP
*** blake_ has quit IRC23:26
*** rcernin_ has joined #openstack-lbaas23:41
*** blake has joined #openstack-lbaas23:41
*** rcernin has quit IRC23:43
*** blake has quit IRC23:45
*** openstackgerrit has quit IRC23:49
*** blake has joined #openstack-lbaas23:55

Generated by 2.15.3 by Marius Gedminas - find it at!