| *** sapd1 has quit IRC | 00:44 | |
| *** mchlumsky has joined #openstack-lbaas | 01:19 | |
| *** zzzeek has quit IRC | 02:48 | |
| *** zzzeek has joined #openstack-lbaas | 02:50 | |
| *** sapd1 has joined #openstack-lbaas | 03:16 | |
| *** vishalmanchanda has joined #openstack-lbaas | 04:52 | |
| *** gcheresh has joined #openstack-lbaas | 05:04 | |
| *** yamamoto has quit IRC | 05:07 | |
| *** strigazi has quit IRC | 05:37 | |
| *** strigazi has joined #openstack-lbaas | 05:39 | |
| *** luksky has joined #openstack-lbaas | 06:09 | |
| *** luksky has quit IRC | 06:17 | |
| *** luksky has joined #openstack-lbaas | 06:18 | |
| *** rpittau|afk is now known as rpittau | 07:12 | |
| *** openstack has joined #openstack-lbaas | 07:53 | |
| *** ChanServ sets mode: +o openstack | 07:53 | |
| *** openstackgerrit has quit IRC | 08:11 | |
| marlinc | johnsom: https://gist.github.com/Marlinc/f60efb44df457b8131404be0daae853c | 08:18 |
|---|---|---|
| zigo | Hi. Could I get a 2nd view on that one? | 08:21 |
| zigo | https://review.opendev.org/c/openstack/octavia/+/656491 | 08:21 |
| zigo | I don't think the -1 is relevant, and I've been carying the patch in the Debian package for more than a year now ... | 08:22 |
| *** lxkong has quit IRC | 08:24 | |
| *** emccormick has quit IRC | 08:25 | |
| gthiemonge | zigo: yeah I agree. LGTM | 08:27 |
| *** emccormick has joined #openstack-lbaas | 08:29 | |
| *** lxkong has joined #openstack-lbaas | 08:30 | |
| gthiemonge | marlinc: it's weird: the octavia management network is missing from the 'addresses' attribute in server list | 08:40 |
| *** parallax has joined #openstack-lbaas | 08:57 | |
| marlinc | Yea gthiemonge that's the issue I'm running into. When I check the 'action logs' in Horizon then I can see the octavia user creating the instance, attaching the tenant network and then detaching the management network | 09:24 |
| marlinc | I hope enabling debugging will provide more information on what's causing that | 09:28 |
| *** strigazi has quit IRC | 11:05 | |
| *** sapd1 has quit IRC | 11:20 | |
| *** servagem has joined #openstack-lbaas | 12:12 | |
| *** marlinc has quit IRC | 12:48 | |
| *** marlinc has joined #openstack-lbaas | 12:59 | |
| *** gregraka has joined #openstack-lbaas | 13:05 | |
| *** marlinc has quit IRC | 13:23 | |
| *** marlinc has joined #openstack-lbaas | 13:23 | |
| *** sapd1 has joined #openstack-lbaas | 13:25 | |
| *** vishalmanchanda has quit IRC | 13:41 | |
| *** openstackgerrit has joined #openstack-lbaas | 13:49 | |
| openstackgerrit | Gregory Thiemonge proposed openstack/octavia master: Spare pool removal https://review.opendev.org/c/openstack/octavia/+/787722 | 13:49 |
| openstackgerrit | Gregory Thiemonge proposed openstack/octavia master: Spare pool removal https://review.opendev.org/c/openstack/octavia/+/787722 | 13:50 |
| *** vishalmanchanda has joined #openstack-lbaas | 14:22 | |
| *** openstackgerrit has quit IRC | 14:23 | |
| *** openstackgerrit has joined #openstack-lbaas | 14:43 | |
| openstackgerrit | Mohammed Naser proposed openstack/octavia stable/ussuri: Fix empty Batch Member Update to unlock objects https://review.opendev.org/c/openstack/octavia/+/787190 | 14:43 |
| openstackgerrit | Mohammed Naser proposed openstack/octavia stable/train: Fix empty Batch Member Update to unlock objects https://review.opendev.org/c/openstack/octavia/+/788015 | 14:47 |
| openstackgerrit | Carlos Gonçalves proposed openstack/octavia stable/victoria: Fix empty Batch Member Update to unlock objects https://review.opendev.org/c/openstack/octavia/+/788018 | 14:55 |
| marlinc | About the issue where the management interface gets detached, I haven't figured out why its happening yet but I have gathered some debug information: https://gist.github.com/Marlinc/70a3b38b0dc9bc84c6af25d8a324d9ca | 15:23 |
| marlinc | Not sure what "Disabling health monitoring on amphora" is, going to look into that | 15:27 |
| *** spatel has joined #openstack-lbaas | 15:28 | |
| spatel | johnsom quick question does octavia support TLS based load-balancing ? | 15:29 |
| johnsom | marlinc Just a tip, if you are asking for help, please stay around 20-30 minutes. It was my weekend too, but jumped online to offer help. | 15:30 |
| johnsom | spatel Yes | 15:30 |
| spatel | perfect! thanks | 15:30 |
| johnsom | spatel Pass through and via offloading on the LB | 15:30 |
| marlinc | johnsom, yea normally I do, I just installed ZNC again so I can use multiple IRC clients to make that easier. I had irssi open on a remote machine so I didn't get a notification | 15:31 |
| spatel | johnsom do i need to tell octavia to perform TLS load-balancing ? | 15:31 |
| johnsom | spatel You specify it when creating your listeners. This section and down in the docs: https://docs.openstack.org/octavia/latest/user/guides/basic-cookbook.html#deploy-a-non-terminated-https-load-balancer | 15:31 |
| spatel | +1 | 15:33 |
| johnsom | marlinc Can you provide the settings in octavia.conf [controller_worker] amp_boot_network_list , if you have any Octavia availability zones configured, and a subnet show for "some-test-network" | 15:34 |
| marlinc | johnsom, wait a minute I did not add the management network to amp_boot_network_list because its the only network in the project so I expected it to work because its the default. Is Octavia removing it because it didn't explicitly add that interface? | 15:35 |
| johnsom | Are you saying you are trying to use the same network for your lb-mgmt-net and your VIP network? | 15:36 |
| marlinc | No what I meant to say is that the service project in which Octavia is creating instances only has one network (the lb-mgmt-net network) so Nova is auto attaching that network to the Octavia Amphora instances | 15:38 |
| johnsom | VIP and member networks can be any project. lb-mgmt-net needs to be owned by the octavia account. Nova isn't auto-attaching it, we are telling nova to attache the lb-mgmt-net to the instance. | 15:40 |
| marlinc | Yea that's I think what's going wrong, I did not tell Octavia to add the lb-mgmt-net network (by setting it in amp_boot_network_list) | 15:41 |
| marlinc | amp_boot_network_list is currently not set at all in [controller_worker] | 15:41 |
| johnsom | Yeah, that is a required setting | 15:42 |
| johnsom | It must match your lb-mgmt-net | 15:42 |
| openstackgerrit | Greg Rakauskas proposed openstack/octavia master: Edits for "Basic Load Balancing Cookbook" https://review.opendev.org/c/openstack/octavia/+/787643 | 15:51 |
| *** rpittau is now known as rpittau|afk | 15:55 | |
| *** mnaser has joined #openstack-lbaas | 15:55 | |
| marlinc | Found the logic that is was removing the network because it didn't expect it: https://github.com/openstack/octavia/blob/master/octavia/controller/worker/v2/tasks/network_tasks.py#L94 | 15:58 |
| marlinc | I think I'm going to open a review for a small change that introduces some debug logs indicating that Octavia ia removing interfaces it doesn't expect | 15:59 |
| marlinc | So as a operator you know why Octavia is removing them and you can be pointed to amp_boot_network_list | 15:59 |
| johnsom | Well, since you probably used the same for the VIP, when you manually triggered that failover, it also probably removed it. | 15:59 |
| johnsom | That is probably not helpful in this situation. A better approach is probably adding an error and exit if the lb-mgmt-net is not defined. | 16:01 |
| johnsom | Networks in that list are already excluded from that delta calculation | 16:01 |
| *** xgerman has joined #openstack-lbaas | 16:01 | |
| marlinc | What I think happened is that Octavia was sending a empty network list to Nova when creating the instance, this causes Nova to assume it has to add the default network (which was lb-mgmt-net). Then the Octavia reconciliation loop comes along (the delta calculation) and sees a unexpected network interface and thus removes it | 16:03 |
| johnsom | Yeah, because the lb-mgmt-net wasn't on the boot network list | 16:04 |
| johnsom | Arguably nova should have just failed to boot the instance. | 16:04 |
| marlinc | Yea so giving an error when amp_boot_network_list isn't set is probably a good idea | 16:04 |
| johnsom | Or booted it with no network at all, which might have happened and the VIP network came along | 16:04 |
| marlinc | And maybe some debugging information in that reconciliation loop so you know what its trying to change the state to | 16:05 |
| johnsom | I would vote against that as it will be fairly noisy for a very rare configuration issue. That delta calculation runs often with member add/remove, etc. | 16:06 |
| johnsom | What deployment tool did you use that it didn't set that for you? | 16:07 |
| marlinc | I'm using openstack-helm together with Terraform to deployed the OpenStack tooling so I have to set that manually based on the network that Terraform creates | 16:08 |
| marlinc | Was just thinking of a log message when it actually needs to remove a network, not some log message that always shows that there's nothing to removed | 16:09 |
| *** nicolasbock has joined #openstack-lbaas | 16:09 | |
| johnsom | Yep | 16:09 |
| marlinc | I have had multiple times with OpenStack components where they didn't really log anything even with the highest level of debugging so I had to go through the code to figure out what was wrong | 16:10 |
| marlinc | So having some log message that indicates that indicates that something's wrong would be very useful | 16:10 |
| johnsom | If you used the helm chart for Octavia, it might be worth opening a bug for the helm team to make sure the lb-mgmt-net gets configured | 16:10 |
| marlinc | The issue is not with the Helm chart but rather with the Terraform logic I wrote around it so that's my fault | 16:11 |
| marlinc | The Helm chart itself doesn't set anything, it just sets up the deployments etc in Kubernetes. In the official openstack-helm tooling there's tons of bash scripts around Helm to provide that information to the Helm chart | 16:12 |
| johnsom | I am just saying I won't vote for it as I don't think it is addressing the real problem. It's completely unrelated to the root cause and will lead people in the wrong direction of fixing the problem. | 16:12 |
| johnsom | We get a bit of feedback that we log too much already, lol. | 16:13 |
| marlinc | Yea I understand I'll leave it, also I wasn't talking INFO or WARNING logging but DEBUG level, I assume people don't run with DEBUG in production | 16:14 |
| johnsom | Many do. I even know of one distro that runs with TRACE on, which is bizarre to me. | 16:15 |
| marlinc | Okay well, haha | 16:16 |
| *** QG has joined #openstack-lbaas | 16:17 | |
| QG | Hello | 16:17 |
| marlinc | Hey there | 16:18 |
| QG | I have a question about the octavia plugin for Horizon, I made a small patch to be able to check if the Octavia endpoint is present in the region, do you think I should make a blueprint? | 16:19 |
| johnsom | QG No, a blueprint (we have been migrated to Storyboard, so don't use launchpad) should not be necessary for that. It wouldn't need an RFE either. At most a story (bug) in my opinion. | 16:21 |
| QG | johnsom: ok i will create a story bug thank you | 16:22 |
| johnsom | NP | 16:24 |
| marlinc | johnsom, anyway thank you for the pointed in the right direction | 16:28 |
| johnsom | Sure, NP | 16:29 |
| johnsom | Oh joy, the concrete grinding outside has started. This should be a productive day | 16:30 |
| gthiemonge | FYI openstack-tox-pep8 fails on master | 16:34 |
| johnsom | gregraka Was just asking me about that. New linter package I assume | 16:38 |
| johnsom | pylint... | 16:40 |
| johnsom | gthiemonge Are you on it, or should I work on a patch? | 16:41 |
| gthiemonge | johnsom: I'll work on it ;-) | 16:42 |
| johnsom | Thanks | 16:42 |
| gregraka | Thanks, gthiemonge++ !! and Johnsom++ !! | 16:43 |
| *** QG is now known as INFO | 16:52 | |
| *** INFO is now known as QG | 16:57 | |
| openstackgerrit | Gregory Thiemonge proposed openstack/octavia master: Fix pylint errors https://review.opendev.org/c/openstack/octavia/+/788047 | 17:11 |
| gthiemonge | ^ I choose to not disable the new checker | 17:12 |
| johnsom | Cool. Interesting, those cases really don't make sense to use "with" IMO, so excluding them is the right answer IMO. | 17:14 |
| openstackgerrit | Quentin GROLLEAU proposed openstack/octavia-dashboard master: Look for load-balancer endpoints in catalog https://review.opendev.org/c/openstack/octavia-dashboard/+/788050 | 17:28 |
| QG | gthiemonge thank you so much for the respond on storyboard, i was on stable/rocky branch so i didn't have the patch | 17:35 |
| QG | However, I looked in the existing patches but I did not find one | 17:36 |
| johnsom | Rocky is end of life, so maybe that is why it didn't get the patch | 17:37 |
| QG | but I can try to backport it ? | 17:37 |
| johnsom | Sure, just note that there will not be any more rocky releases created | 17:38 |
| johnsom | The patch will also likely not merge | 17:38 |
| QG | ohhhhh | 17:38 |
| QG | OK thank you johnsom | 17:39 |
| johnsom | https://github.com/openstack/octavia/tree/rocky-eol | 17:39 |
| johnsom | It was marked EOL in July last year | 17:40 |
| QG | k, so i put the gerrit patch as abandoned, how can i close the story as it is no longer relevant ? | 17:43 |
| johnsom | QC You can mark it as "Invalid" | 17:47 |
| QG | Thank you johnsom and sorry about it | 18:01 |
| johnsom | No worries | 18:01 |
| *** QG has quit IRC | 18:06 | |
| *** vishalmanchanda has quit IRC | 18:21 | |
| *** stand has joined #openstack-lbaas | 18:42 | |
| marlinc | How can I delete or failover a Amphora instance when I get the the load balancer gets marked as immutable: Load Balancer b83bcf0a-a36d-4751-91c1-6acd677c2c30 is immutable and cannot be updated. (HTTP 409) (Request-ID: req-23cfe4e2-a066-40ea-8c46-d3dcbe7ae535) | 19:30 |
| johnsom | You need to wait for it to finish and become mutable | 19:31 |
| johnsom | How long depends on your settings. | 19:31 |
| marlinc | Does it also become mutable again when the load balancer is in an ERROR state? After some timeout for example? | 19:34 |
| johnsom | Yes | 19:37 |
| marlinc | Cool then I'll wait a bit longer! | 19:38 |
| johnsom | When the load balancer is in a PENDING_ state, one of the controllers has ownership and is working to finish your requested action. It will retry actions if nova or neutron are failing up to the retry settings in the config. It will either move to ONLINE or ERROR. | 19:39 |
| marlinc | Is this retrying supposed to be visible in the worker? I did see an error that it got from Nova but after that it went completely quiet since about half an hour ago now | 19:42 |
| johnsom | With debug you will see repeated retry messages. Make sure to check all of the controller logs. | 19:44 |
| *** zzzeek has quit IRC | 19:44 | |
| *** zzzeek has joined #openstack-lbaas | 19:47 | |
| *** gregraka has quit IRC | 20:04 | |
| *** zzzeek has quit IRC | 20:09 | |
| marlinc | Definitely have some more testing to do tomorrow, also with Nova | 20:16 |
| *** zzzeek has joined #openstack-lbaas | 20:16 | |
| marlinc | I see Octavia nicely adding a interface for the tenant network to the Amphora instance, you can see it when viewing info about the server but the VM doesn't actually see a new interface | 20:18 |
| marlinc | But that's probably something I have to check in Nova not Octavia | 20:18 |
| *** gcheresh has quit IRC | 20:18 | |
| johnsom | It does, it's in a network namespace. | 20:18 |
| johnsom | We isolate tenant traffic for security reasons | 20:18 |
| marlinc | Oh interesting, I did see the interface quite a few times but that was probably was while it was still configuring things | 20:19 |
| marlinc | Ah yes I see it | 20:19 |
| marlinc | Very nice engineering, like it! | 20:20 |
| marlinc | Thanks again | 20:36 |
| *** zzzeek has quit IRC | 20:40 | |
| *** zzzeek has joined #openstack-lbaas | 20:41 | |
| *** zzzeek has quit IRC | 20:46 | |
| *** zzzeek has joined #openstack-lbaas | 20:48 | |
| *** gregraka has joined #openstack-lbaas | 21:27 | |
| *** spatel has quit IRC | 22:01 | |
| *** zzzeek has quit IRC | 22:42 | |
| *** zzzeek has joined #openstack-lbaas | 22:43 | |
| *** rcernin has joined #openstack-lbaas | 23:09 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!