| *** gthiemon1e is now known as gthiemonge | 07:12 | |
| opendevreview | Tom Weininger proposed openstack/octavia master: Add pytest testenv to tox.ini https://review.opendev.org/c/openstack/octavia/+/881739 | 08:35 |
|---|---|---|
| opendevreview | Tom Weininger proposed openstack/octavia master: DNM: Test CI with pytest running https://review.opendev.org/c/openstack/octavia/+/881740 | 08:35 |
| opendevreview | Tom Weininger proposed openstack/octavia master: Make tests work with pytest runner https://review.opendev.org/c/openstack/octavia/+/881805 | 08:35 |
| opendevreview | Tom Weininger proposed openstack/octavia master: DNM: Test CI with pytest running https://review.opendev.org/c/openstack/octavia/+/881740 | 08:40 |
| opendevreview | Tom Weininger proposed openstack/octavia master: Make tests work with pytest runner https://review.opendev.org/c/openstack/octavia/+/881805 | 08:40 |
| opendevreview | Tom Weininger proposed openstack/octavia master: Add pytest testenv to tox.ini https://review.opendev.org/c/openstack/octavia/+/881739 | 09:00 |
| opendevreview | Tom Weininger proposed openstack/octavia master: Make tests work with pytest runner https://review.opendev.org/c/openstack/octavia/+/881805 | 09:00 |
| opendevreview | Tom Weininger proposed openstack/octavia master: DNM: Test CI with pytest running https://review.opendev.org/c/openstack/octavia/+/881740 | 09:00 |
| opendevreview | Tom Weininger proposed openstack/octavia master: Add support for HTTP Strict Transport Security https://review.opendev.org/c/openstack/octavia/+/880806 | 09:01 |
| opendevreview | Tom Weininger proposed openstack/octavia-lib master: Add support for HTTP Strict Transport Security https://review.opendev.org/c/openstack/octavia-lib/+/880821 | 09:02 |
| opendevreview | Tom Weininger proposed openstack/python-octaviaclient master: Add support for HTTP Strict Transport Security https://review.opendev.org/c/openstack/python-octaviaclient/+/880808 | 09:04 |
| opendevreview | Tom Weininger proposed openstack/octavia master: Integrate mypy type checker https://review.opendev.org/c/openstack/octavia/+/879749 | 09:16 |
| opendevreview | Tom Weininger proposed openstack/octavia master: Make tests work with pytest runner https://review.opendev.org/c/openstack/octavia/+/881805 | 09:33 |
| opendevreview | Tom Weininger proposed openstack/octavia master: DNM: Test CI with pytest running https://review.opendev.org/c/openstack/octavia/+/881740 | 09:33 |
| gthiemonge | FYI https://review.opendev.org/c/openstack/project-config/+/881810 | 10:53 |
| crab | hi all im having some difficulty with octavia on victoria. ive managed to create a loadbalancer which is in PENDING_CREATE and ONLINE. | 13:34 |
| crab | when i ssh into the amphora and tcpdump the interface looking for traffic on port 9443, i can see traffic from the controller. | 13:34 |
| crab | the worker.log on the controller warns: octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect to instance. Retrying.: requests.exceptions.ReadTimeout: | 13:35 |
| crab | the amphora definitely appears to want to listen on 9443 but something isnt working. | 13:36 |
| gthiemonge | crab: hi | 13:36 |
| crab | hi | 13:36 |
| gthiemonge | the octavia worker process should establish a connection with the amp, through an interface that is created on the controller (what is your deployment method?) | 13:38 |
| gthiemonge | I would check that this interface is correctly set up | 13:38 |
| crab | we used our own puppet stuff to deploy, and that connection uses neutron-linuxbridge-agent for the vxlan part and we used network manager for the veth part. | 13:40 |
| crab | i can ssh from the controller and ping so there seems to be some connectivity. | 13:40 |
| crab | also as i say, netstat and tcpdump show that there is a connection to port 9443. | 13:41 |
| crab | on the amphora i see worker process constantly dying and respawning | 13:41 |
| johnsom | It might be TLS related. | 13:41 |
| johnsom | Check this document: https://docs.openstack.org/octavia/latest/admin/guides/certificates.html | 13:42 |
| crab | will do | 13:43 |
| crab | i keep seeing a lot of python errors which culiminate in: | 13:43 |
| crab | amphora-agent[772]: AssertionError: can only join a child process | 13:43 |
| johnsom | Hmmm, haven’t seen that. Maybe give us a longer amphora agent log snippet in paste.openstack.org | 13:44 |
| crab | ok... | 13:44 |
| crab | lots of this: https://paste.opendev.org/show/br7dLaU5j7j7r3AENTxU/ | 13:45 |
| gthiemonge | I saw this when amphora-agent is restarting | 13:47 |
| gthiemonge | AssertionError should not be a problem, the "WORKER TIMEOUT" is | 13:48 |
| crab | ok | 13:49 |
| gthiemonge | crab: can you paste the logs of the worker? (the lines around "octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect to instance." | 13:51 |
| crab | from the controller side? | 13:51 |
| gthiemonge | yep, worker.log | 13:51 |
| crab | just lots and lots of this: | 13:54 |
| crab | https://paste.opendev.org/show/bIFSdMtnsskbvpEKDapB/ | 13:54 |
| gthiemonge | ok, only timeouts | 13:54 |
| gthiemonge | I would check the octavia interface connectivity on the controller (with tcpdump) and maybe also on the amphora port when it is created | 13:55 |
| crab | yeah ive done that. | 13:56 |
| crab | it looks good as far as i can tell. | 13:56 |
| crab | if i do "netstat | grep 9443" on the amphora side i can see connections to / from the controller ip | 13:56 |
| crab | and tcpdump ens3 port 9443 on the amphora also shows the traffic | 13:56 |
| crab | hmmm. | 13:57 |
| gthiemonge | ok, at least this part is working | 13:57 |
| crab | we did see something slightly unusual which im now thinking may be significant. | 13:57 |
| gthiemonge | as johnsom mentioned, might be TLS related then | 13:57 |
| crab | in testing, when we'd got the interface up on the controller, we created an instance and sort of manually added an interface | 13:58 |
| crab | iirc we could ping the instance from controller but not the other way around which we thought was very odd... | 13:58 |
| crab | but at that point we sort of ignored it and soldiered on. | 13:58 |
| johnsom | We typically do not enable ping on the amphora instances | 14:02 |
| crab | aah so thats nothing to be worried about per se? | 14:03 |
| johnsom | Right | 14:04 |
| crab | i can ping amphora from controller fine, but not visa versa. | 14:04 |
| crab | ok.. looks like its ssl fun time then. | 14:04 |
| * crab grits teeth | 14:04 | |
| crab | thanks for your help both of you. | 14:04 |
| johnsom | Sure, the document is pretty detailed and should step you through it. | 14:05 |
| johnsom | I am guessing some of the certificate settings are not configured correctly | 14:06 |
| crab | hmmm. | 14:48 |
| crab | new certs same error. | 14:48 |
| crab | if i hop on to the amphora and kill off the agent, and start netcat listening on 9443, | 14:55 |
| crab | i can see constant chatter from the controller | 14:55 |
| johnsom | After you update the certs, you need to create a new load balancer or do a load balancer failover to have it pick up your changes (after you have restarted the controllers) | 16:12 |
| crab | johnsom: yeah none of that seems to have helped. | 16:50 |
| crab | i think our amphora images might be bad. | 16:50 |
| johnsom | How did you build them? Using the script we provide? | 16:57 |
| crab | yeah i used the script in the git repo | 16:58 |
| crab | i checked out stable/victoria | 16:58 |
| crab | and followed directions. | 16:58 |
| crab | however, we struggled to get any centos images built (or fedora) | 16:58 |
| crab | and i ended up installing debootstrap on a rocky9 host, | 16:59 |
| crab | and using that | 16:59 |
| crab | its very odd. the error looks *kind of* like this: | 17:01 |
| crab | https://storyboard.openstack.org/#!/story/2008226 | 17:01 |
| crab | but without the socket error processing request bit | 17:01 |
| johnsom | Yeah, centos has had many issues over the last few years. The packages have been a bit unstable. It's nothing in the Octavia code, but fundamental centos package issues. | 17:20 |
| johnsom | I seriously doubt this is a bug or issue inside the amphora. I think it's a configuration issue. If you want to run a test to prove that, grab the focal test image from here:https://tarballs.opendev.org/openstack/octavia/test-images/test-only-amphora-x64-haproxy-ubuntu-focal.qcow2 | 17:22 |
| johnsom | Those are not for production use, but would remove the image as a potential issue. | 17:22 |
| crab | thanks... i'll give that a go. | 17:22 |
| crab | our deployment is on rocky8 - its been pretty solid we've upgraded it from centos 7 iirc. | 17:29 |
| crab | and then we've upgraded openstack itself a couple of times too. | 17:29 |
| johnsom | Yeah, I think someone here was working with the DIB team to get Rocky going. I'm not sure where that is in the process. | 17:31 |
| crab | well we dont really need to worry about using ubuntu for amphora / building amphora (unless im missing something) | 17:31 |
| crab | but moving all of openstack to ubuntu would be more of a wrench | 17:32 |
| crab | johnsom: well that image doesn't seem to behave any differently so i guess it is a config issue. | 17:58 |
| crab | i cant spot anything obvious to investigate though. | 17:58 |
| johnsom | I will have limited availability for a few hours. Can you paste.openstack.org the amphora agent config file? | 18:00 |
| crab | sure | 18:00 |
| johnsom | I will look at it later and see if I see something | 18:01 |
| johnsom | Mark it private and DM me if you are concerned about the content | 18:01 |
| crab | done | 18:03 |
| crab | i dont think there is anything contentious in there really. remember thats from the test image you just suggested. | 18:03 |
| crab | i suspect that the problem is more likely to be in my octavia.conf! ;) | 18:03 |
| crab | i have tried switching Debug = False to True a few times, but that doesnt seem to be very revealing. | 18:04 |
| crab | anyway thanks very much for your help - its appreciated. but don't stress it. im gonna have a rest and see im sure it'll sort itself at some stage. we've been going quite hard on this over the last few days and sometimes a break works wonders. :) | 18:05 |
| *** JayF is now known as Guest12444 | 18:27 | |
| *** JasonF is now known as JayF | 18:27 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!