Monday, 2019-03-04

*** luksky has quit IRC00:16
johnsommloza, the amphora are service instances, they are always owned by the Octavia service account.00:56
johnsomkosa777777 Some database work, deleting an instance in nova, and a failover call. But I have never done this, so there are likely more details to workout.00:58
johnsomMaybe cleaning up a neutron port too00:58
*** henriqueof2 has quit IRC01:20
*** henriqueof2 has joined #openstack-lbaas01:21
*** henriqueof2 has quit IRC01:52
*** Dinesh_Bhor has joined #openstack-lbaas02:31
*** Dinesh_Bhor has quit IRC02:31
*** Dinesh_Bhor has joined #openstack-lbaas02:41
*** psachin has joined #openstack-lbaas02:52
*** yamamoto has joined #openstack-lbaas03:16
*** yamamoto has quit IRC04:18
*** ramishra has joined #openstack-lbaas04:26
*** yamamoto has joined #openstack-lbaas04:36
*** mkuf has quit IRC06:54
*** ccamposr has joined #openstack-lbaas07:00
*** mkuf has joined #openstack-lbaas07:06
*** gcheresh_ has joined #openstack-lbaas07:27
openstackgerritCarlos Goncalves proposed openstack/octavia-dashboard master: Fix auth url for Barbican client
*** yamamoto has quit IRC07:40
*** yamamoto has joined #openstack-lbaas07:41
*** sapd1 has joined #openstack-lbaas07:57
*** luksky has joined #openstack-lbaas08:07
*** rpittau|sardegna is now known as rpittau08:13
*** pcaruana has joined #openstack-lbaas08:25
nmagnezikosa777777, any chance you have a floating ip attached to your vip?08:30
openstackgerritCarlos Goncalves proposed openstack/octavia-tempest-plugin master: Add iptables-based active/standby scenario test
*** celebdor has joined #openstack-lbaas09:15
*** sapd1 has quit IRC09:41
*** yamamoto has quit IRC10:18
*** yamamoto has joined #openstack-lbaas10:36
*** yamamoto has quit IRC10:36
*** yamamoto has joined #openstack-lbaas10:37
*** yamamoto has quit IRC10:41
*** andrein has joined #openstack-lbaas10:59
*** luksky has quit IRC11:00
*** salmankhan has joined #openstack-lbaas11:04
andreinHello everyone. I'm trying to set up octavia on rocky, deployed via kolla-ansible. I'm getting the following error in octavia-worker when I create a load balancer:
*** yamamoto has joined #openstack-lbaas11:14
andreinLooks like my client cert was missing the key... need to follow the PKI docs again11:41
*** luksky has joined #openstack-lbaas11:50
*** sapd1 has joined #openstack-lbaas12:45
*** andrein has quit IRC12:46
*** yamamoto has quit IRC12:48
*** andrein has joined #openstack-lbaas12:50
*** andrein has quit IRC12:52
*** andrein has joined #openstack-lbaas12:57
*** yamamoto has joined #openstack-lbaas13:16
*** yamamoto has quit IRC13:37
*** yamamoto has joined #openstack-lbaas13:39
*** pcaruana has quit IRC14:31
*** henriqueof2 has joined #openstack-lbaas14:56
mlozawhat seems to be a problem if the lb is working and provisioning status is active but the operating status is offline?15:03
*** sapd1 has quit IRC15:09
*** sapd1 has joined #openstack-lbaas15:10
*** fnaval has joined #openstack-lbaas15:19
*** pcaruana has joined #openstack-lbaas15:25
johnsommloza: operating status is the observed status, so your health check is failing or your health heartbeat packets from the amphora are not reaching the controller endpoint on 555515:27
*** andrein has quit IRC15:32
*** luksky has quit IRC15:40
dulekI'm getting "TemplateSyntaxError: expected token 'end of statement block', got '.'" in o-cw. What can be wrong?15:40
dulekI mean I understand it's template. :D15:40
*** andrein has joined #openstack-lbaas15:42
johnsomWhat action are you taking and with which driver?15:48
*** gcheresh_ has quit IRC16:00
mloza<johnsom> mloza, the amphora are service instances, they are always owned by the Octavia service account. -- Does this mean if I add octavia user in a project the service instance will get launch in their own space?16:01
mloza<johnsom>  mloza: operating status is the observed status, so your health check is failing or your health heartbeat packets from the amphora are not reaching the controller endpoint on 5555 -- Thanks, will take a look.16:02
johnsommloza: yes, you can setup a service project for Octavia. Most of us do.16:07
johnsomJust make sure you have good quotas and that things like the image and optional ssh keys are owned by that project.16:07
*** henriqueof3 has joined #openstack-lbaas16:10
*** luksky has joined #openstack-lbaas16:11
*** henriqueof2 has quit IRC16:12
mlozajohnsom: I added octavia as admin in a tenant project but the service instance still gets launched in the admin project16:15
*** henriqueof2 has joined #openstack-lbaas16:17
*** henriqueof3 has quit IRC16:20
*** henriqueof2 has quit IRC16:29
*** ramishra has quit IRC16:31
johnsommloza You need to set the account in the [service_auth] section of the octavia.conf16:33
*** xgerman has joined #openstack-lbaas16:35
mlozajohnsom: its the project_name that I have to set right? What if I want the service instance to be created under the project of the user who created it not just one project?16:38
*** yamamoto has quit IRC16:41
*** rpittau is now known as rpittau|afk16:42
johnsommloza You cannot have the amphroa created under the user project. They are a managed service instance and should not be visible or accessible to the user.16:42
dulekjohnsom: Whoops, sorry, missed your answer. That was Amphorae and seems to be during listener creation.16:47
dulekMight be listeners update as well.16:48
johnsomYes, probably. So that likely means a jinja2 template got corrupted somehow.  There is a template override in the octavia.conf :
johnsomCheck that it is not set and pointing to some local templates. Then I would check the standard templates in and see if those have been changed somehow.  Then after that it's the VRRP or UDP/LVS templates potentially if you are using those features.16:50
*** yamamoto has joined #openstack-lbaas16:52
dulekjohnsom: That's DevStack, so I strongly doubt any corruption. Data point - the setup also has networking-ovn provider enabled.16:56
johnsomHmm, well, then it shouldn't be using any templates....  Hmmm, is there a stack trace?16:57
*** yamamoto has quit IRC16:57
dulekjohnsom: No, no, sorry for confusing you, it's using both Octavia and networking-ovn and the error is while Octavia LB is getting created.17:03
dulekjohnsom: I can try putting the logs somewhere, just a sec.17:04
johnsomYeah, most interested in the worker log17:04
dulekjohnsom: I'm not overriding any template, in DevStack we just set the listeners timeouts.17:07
*** psachin has quit IRC17:09
johnsomdulek Oh, ok, I know what this is. You have an old version of jinja2 installed17:10
dulekjohnsom: I was updating that system recently, it's possible I've some pip packages got overwritten…17:11
dulekjohnsom: Although I would expect global requirements would handle that.17:12
johnsomdulek Yeah, it needs to be jinja 2.10 or newer17:12
cgoncalvesqueens, right? we didn't get to fix in queens as it required bumping version17:13
dulekjohnsom: Yep, looks like system packages overwritten this package.17:13
dulekcgoncalves: It's actually more-or-less master17:13
dulek>>> jinja2.__version__17:14
cgoncalvesqueens is more-or-less still master for me xD17:14
cgoncalvesit will be for the next 5 years of support lol17:14
johnsomSounds like you need help with that....17:14
dulekMe? I'll handle it from here, no problem. ;)17:15
cgoncalveswe got to fix that in RDO/OSP via .spec17:15
openstackgerritCarlos Goncalves proposed openstack/octavia-lib master: Sync data models and import new constants from Octavia
openstackgerritCarlos Goncalves proposed openstack/octavia master: Updates Octavia to support octavia-lib
cgoncalvesjohnsom, I hope you don't mind me messing with your changes17:27
johnsomcgoncalves Always nice to ask first. I was just about to do the octavia-lib patch17:27
johnsomBut, since you beat me too it....17:28
openstackgerritMerged openstack/python-octaviaclient master: Add 'tls_container_ref' option into Pool for backend re-encrption
openstackgerritMerged openstack/python-octaviaclient master: Add 2 new options to Pool for support backend certificates validation
openstackgerritCarlos Goncalves proposed openstack/octavia-lib master: Sync data models and import new constants from Octavia
*** andrein has quit IRC17:47
openstackgerritMerged openstack/python-octaviaclient master: Add enable_tls option into Pool CLI
johnsomWelcome to feature freeze week folks! I have updated the priority review sheet.18:10
johnsomnote: It is not feature freeze yet, but winter is coming....18:11
*** andrein has joined #openstack-lbaas18:14
*** ccamposr has quit IRC18:21
*** ccamposr has joined #openstack-lbaas18:21
*** ccamposr has quit IRC18:22
*** andrein has quit IRC18:29
*** HD|Laptop has joined #openstack-lbaas18:45
HD|LaptopHello all!18:45
HD|LaptopI am trying to use Octavia for magnum/k8s loadbalancing18:45
HD|Laptopafter a long learning period I have managed to get everything connected and running - except that haproxy on the amphora (used the image from )seems  to be messed ip18:46
HD|Laptopon the amphora I can connect to the backend just fine18:53
HD|Laptopbut haproxy cannot18:53
HD|Laptop <= here is all my config18:53
*** pcaruana has quit IRC18:56
johnsomHD|Laptop By default the netns does not have the loopback up, so you can't curl the VIP local. Just bring up the loopback interface and it will work.  (loopback is not needed for normal operations)19:00
HD|Laptopjohnsom: lo is up in the amphora netns, see line 3519:02
johnsom1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 119:03
johnsomstate DOWN19:03
*** andrein has joined #openstack-lbaas19:04
HD|Laptopjohnsom: ok, now at least curl'ing the frontend inside the amphora works19:05
HD|Laptopjohnsom: but curl'ing either the frontend ip ( or the floating IP ( still doesn't work19:09
johnsomFrom outside the amphora?19:10
HD|Laptopfrom outside, on the host machine of the amphora19:11
HD|Laptopwhich has an interface on the network19:11
HD|Laptopneutron security groups, port security etc. are all disabled19:12
*** salmankhan has quit IRC19:12
HD|Laptopinterestingly, on the host: => timeout, => connection refused19:13
mlozahow do i get rid of amphora instances in status  BOOTING and ERROR that are in `openstack loadbalancer amphora list`. Probably, these were leftovers when i updated provisioning_status to ERROR in the database because the LB was in PENDING_UPDATE19:15
HD|Laptopmloza: I had ones stuck in PENDING_CREATE during deployment, ended up wiping them in the mysql db by hand19:20
*** openstackgerrit has quit IRC19:23
eanderssonWhat happens if the health_manager update_db takes too long?19:32
*** andrein has quit IRC19:34
johnsomeandersson We drop that update packet.19:38
eanderssonkk so it has no direct impact on the lb itself?19:38
johnsomAnd print some big error messages in the logs19:38
eanderssonYea - the error message might be too big :p19:38
eandersson"THIS IS NOT GOOD"19:39
johnsomNo, assuming one of the HMs is processing ok in the next cycles. Basically it needs to get one in inside the overall time limit. It's six tries by default19:39
eanderssonWhat is the highest delay we could hit before impact?19:43
johnsomDepends on your settings. The default heartbeat interval is 10 seconds, with a 60 second limit before a failover may start.  So, the HM needs to process the heartbeat inside 10 seconds by default, and one out of six heartbeats must be successful.19:44
*** blake has joined #openstack-lbaas19:46
xgermanalso keep in mind if your DB is “slow" we will pile up update requests in our queues which we resolve quickly if timestamp is old — but if you limit memory for the HM too tightly tat might trigger issues19:50
HD|Laptopjohnsom: any idea what else could be wrong with my amphora?19:50
xgermanHaven’t seen that after we optimized our queries...19:50
johnsomHD|Laptop Well, if you can query the VIP from inside the amphora, I suspect it is not the amphora that is the issue. Here are some things to look at:19:52
xgermanyou made sure that the ports i neutron are ok? openstack port list ?19:52
johnsom1. Check the subnet you created the VIP on, does it have any host routes that may be conflicting? We honor the neutron host routes inside the amphora.19:52
johnsom2. Check the ports, as xgerman mentioned. One will be down, one up. This is normal as the "down" port is an "allowed address pairs" neutron port and a "fake" port.19:53
johnsom3. Check the security groups applied to those ports. They should be octavia managed SGs with just a few ports open, including your listener port.19:54
johnsom4. Check the routes on your "client" instance, make sure there isn't a bad route on your test system that is blocking access.19:54
johnsom5. You can install tcpdump inside the amphora and see if your request is making it to the amp. If it is, but you don't see a reply, you have a routing or L2 problem.19:55
HD|Laptopjohnsom: as for 0.
HD|Laptopfor 1. I cannot remember that I set up anything routing related except a neutron SNAT router from the to the network19:57
cgoncalveshow's best (if possible at all) to set lower requirement constraints for an unreleased lib version?19:58
johnsomHD|Laptop That port list looks odd, there should be an octavia vrrp port listed19:58
cgoncalves requires
HD|Laptop| dd37cd45-84c5-4c04-bdab-86e6eb260066 |                                                                                                        | fa:16:3e:57:5f:3a | ip_address='', subnet_id='483f4d33-2fff-4315-a30d-57114e0e5262'  | ACTIVE |19:58
cgoncalves>=1.1.0 wouldn't pass lower-constraints job19:59
HD|Laptopthis is the port entry for the vrrp_ip showing in "openstack loadbalancer amphora show": | vrrp_ip         |                        |19:59
*** andrein has joined #openstack-lbaas19:59
johnsomcgoncalves I don't think you can, but you can ask in the #openstack-requirements channel19:59
HD|Laptopjohnsom: here's amphora show in full20:00
kosa777777nmagnezi no floating IPs, and yes mysql change was necessary, it worked but it would be great to have some API way of changing topology for existing loadbalancers, in case you change octavia config.20:00
johnsomHD|Laptop that is odd, it should have a name. Did you pass in an existing neutron port at LB create?20:00
HD|Laptopjohnsom: the lb was created by magnum/heat from the kubernetes-cloud-controller.20:00
cgoncalvesjohnsom, right.  I image the answer is merge octavia-lib patch and tag octavia-lib first20:01
HD|Laptopas for port security groups, I didn't enable them in neutron and iptables (host + amphora)/ebtables (host) doesn't show anything weird either20:01
*** abaindur has joined #openstack-lbaas20:01
HD|Laptopi'll go the tcpdump route, though.20:02
HD|LaptopI know whats happening!20:03
johnsomcgoncalves The depends on should at least help keep some order.20:03
HD|LaptopI forgot to set the subnet DHCP allocation range...20:03
HD|Laptopwhich means that .12 is colliding with an existing device on the network.20:03
johnsomThere might also be an issue with the allowed-address-pairs function if port security is disabled.20:04
HD|Laptopjohnsom: I was wondering why I didn't see a single packet inside the amphora, and then it hit me, what the hell was I seeing in the ARP cache of the compute host for that IP20:13
HD|Laptopunfortunately I had to tear down the entire k8s cluster as it used colliding IPs also... so it will be ~10min to tell if this is working now or not.20:13
HD|Laptopjohnsom: yep, it works now! only thing missing now would be that octavia also creates a DNS record in designate for the loadbalancer floating IP20:32
rm_workthat would probably be up to heat/k8s, because octavia itself isn't floating-ip aware20:34
johnsomHD|Laptop Since Octavia doesn't manage floating IPs, you will need to set that up with neutron.20:34
johnsomlol, yeah, what he said.20:34
HD|Laptopweird. when I create a server with "openstack server create xyz", it creates a dns entry
HD|Laptopbut for the amphora, it says in the neutron-api "neutron_lib.exceptions.dns.DNSDomainNotFound: Domain not found in the external DNS service"20:35
HD|Laptophmm. when I run "openstack port set <amphora_floatingip_port> --dns-name=mylb", it works. Can I somehow coerce octavia to set the name & dns-name?20:44
johnsomNo, we had to explicitly disable the designate integration on our ports due to bugs. Neutron would fail to migrate ports during a failover with the dns integration enabled.20:45
johnsomOn failover you will either get an error if the bugs are still present, or it will remove the DNS information.20:45
johnsomBasically, designate integration into Octavia is a ToDo item and not implemented yet.20:46
HD|Laptopah okay, thanks for the info :)20:47
HD|Laptophmm. I just created a second exposed deployment via kubectl ("kubectl expose deployment hello-node2 --type=LoadBalancer --port=8082") and it spins up a new lb/amphora20:47
HD|Laptopcan I tell octavia to simply add a listener on the existing amphora/port 8082 instead?20:47
johnsomYes, absolutely20:48
johnsomAs long as they have the same VIP, you can add up to around 65,000 ports to it20:48
HD|Laptopso basically I have to ask this question again with the magnum team? ;) but good to know it is possible in theory20:50
HD|Laptopthanks for your time, I now have a decent working OpenStack with everything needed to run k8s!20:50
johnsomExcellent, good luck!20:50
*** openstackgerrit has joined #openstack-lbaas21:09
openstackgerritCarlos Goncalves proposed openstack/octavia-tempest-plugin master: Add octavia-lib to the base job
openstackgerritCarlos Goncalves proposed openstack/octavia master: Updates Octavia to support octavia-lib
openstackgerritCarlos Goncalves proposed openstack/octavia-lib master: Sync data models and import new constants from Octavia
*** abaindur has quit IRC21:25
*** blake has quit IRC21:32
*** yamamoto has joined #openstack-lbaas21:33
*** yamamoto has quit IRC21:38
*** salmankhan has joined #openstack-lbaas21:50
*** abaindur has joined #openstack-lbaas22:00
*** luksky has quit IRC23:02
*** abaindur has quit IRC23:04
openstackgerritMichael Johnson proposed openstack/octavia master: Support L7policy redirect http code
*** abaindur has joined #openstack-lbaas23:06
*** openstackgerrit has quit IRC23:28
*** andrein has quit IRC23:35
*** salmankhan has quit IRC23:37
*** andrein has joined #openstack-lbaas23:43

Generated by 2.15.3 by Marius Gedminas - find it at!