| opendevreview | Gregory Thiemonge proposed openstack/octavia stable/zed: Pass config to castellan https://review.opendev.org/c/openstack/octavia/+/880435 | 05:45 |
|---|---|---|
| opendevreview | Gregory Thiemonge proposed openstack/octavia stable/yoga: Pass config to castellan https://review.opendev.org/c/openstack/octavia/+/880436 | 05:45 |
| opendevreview | Gregory Thiemonge proposed openstack/octavia stable/xena: Pass config to castellan https://review.opendev.org/c/openstack/octavia/+/880437 | 05:46 |
| opendevreview | Gregory Thiemonge proposed openstack/octavia stable/wallaby: Pass config to castellan https://review.opendev.org/c/openstack/octavia/+/880438 | 05:46 |
| opendevreview | Gregory Thiemonge proposed openstack/octavia stable/zed: Rename Context to RequestContext https://review.opendev.org/c/openstack/octavia/+/880439 | 05:48 |
| opendevreview | Gregory Thiemonge proposed openstack/octavia stable/yoga: Rename Context to RequestContext https://review.opendev.org/c/openstack/octavia/+/880440 | 05:49 |
| opendevreview | Gregory Thiemonge proposed openstack/octavia stable/xena: Rename Context to RequestContext https://review.opendev.org/c/openstack/octavia/+/880445 | 06:01 |
| opendevreview | Gregory Thiemonge proposed openstack/octavia stable/wallaby: Rename Context to RequestContext https://review.opendev.org/c/openstack/octavia/+/880446 | 06:01 |
| opendevreview | Gregory Thiemonge proposed openstack/octavia stable/zed: Fix SQLAlchemy warning about conflict relationship with Tags https://review.opendev.org/c/openstack/octavia/+/880447 | 06:18 |
| opendevreview | Gregory Thiemonge proposed openstack/octavia stable/yoga: Fix SQLAlchemy warning about conflict relationship with Tags https://review.opendev.org/c/openstack/octavia/+/880448 | 06:18 |
| opendevreview | Gregory Thiemonge proposed openstack/octavia stable/xena: Fix SQLAlchemy warning about conflict relationship with Tags https://review.opendev.org/c/openstack/octavia/+/880449 | 06:18 |
| opendevreview | Gregory Thiemonge proposed openstack/octavia stable/wallaby: Fix SQLAlchemy warning about conflict relationship with Tags https://review.opendev.org/c/openstack/octavia/+/880450 | 06:19 |
| opendevreview | Gregory Thiemonge proposed openstack/octavia stable/2023.1: Fix failover when the last listener is deleted https://review.opendev.org/c/openstack/octavia/+/880451 | 06:46 |
| opendevreview | Gregory Thiemonge proposed openstack/octavia stable/zed: Fix failover when the last listener is deleted https://review.opendev.org/c/openstack/octavia/+/880452 | 06:47 |
| opendevreview | Gregory Thiemonge proposed openstack/octavia stable/yoga: Fix failover when the last listener is deleted https://review.opendev.org/c/openstack/octavia/+/880453 | 06:48 |
| opendevreview | Gregory Thiemonge proposed openstack/octavia stable/xena: Fix failover when the last listener is deleted https://review.opendev.org/c/openstack/octavia/+/880454 | 06:48 |
| opendevreview | Gregory Thiemonge proposed openstack/octavia stable/wallaby: Fix failover when the last listener is deleted https://review.opendev.org/c/openstack/octavia/+/880455 | 06:49 |
| opendevreview | Tom Weininger proposed openstack/octavia master: todo mypy https://review.opendev.org/c/openstack/octavia/+/879749 | 09:31 |
| opendevreview | Merged openstack/octavia master: allowed_cidr validation for additional_vips https://review.opendev.org/c/openstack/octavia/+/876042 | 10:24 |
| opendevreview | Julian DA CUNHA proposed openstack/octavia master: Add new spec Let's Encrypt support https://review.opendev.org/c/openstack/octavia/+/877281 | 10:41 |
| opendevreview | Julian DA CUNHA proposed openstack/octavia master: Add new spec Let's Encrypt support https://review.opendev.org/c/openstack/octavia/+/877281 | 10:45 |
| opendevreview | Tom Weininger proposed openstack/octavia master: Remove python-neutronclient https://review.opendev.org/c/openstack/octavia/+/866327 | 11:06 |
| opendevreview | Julian DA CUNHA proposed openstack/octavia master: Add new spec Let's Encrypt support https://review.opendev.org/c/openstack/octavia/+/877281 | 11:07 |
| opendevreview | Tom Weininger proposed openstack/octavia master: Remove python-neutronclient https://review.opendev.org/c/openstack/octavia/+/866327 | 11:11 |
| opendevreview | Edward Hope-Morley proposed openstack/octavia master: Fix hm operating status to ONLINE in single lb call https://review.opendev.org/c/openstack/octavia/+/868363 | 12:58 |
| noonedeadpunk | folks, are you aware of any issues with amphora image from tarballs? | 14:32 |
| noonedeadpunk | I'm not sure of reasons yet, but we're having issues that ssh is basically down on it. Spawning cirros on the same network and with same security groups working nicely at the same time | 14:34 |
| noonedeadpunk | in console log I've spotted no ipv4 network in cloud-init output, but it's from config drive, so IP can be retrieved on some later step... | 14:35 |
| noonedeadpunk | But I was wondering if amphora relies on DHCP? Or if IP is not present on config-drive it won't be fetched from dhcp? | 14:36 |
| noonedeadpunk | As then it could be configuration issue... | 14:37 |
| tweining | noonedeadpunk: I am not aware of issues with those tarballs, but I haven't tested them either recently. | 14:58 |
| noonedeadpunk | sooo... security group doesn't contain any single egress rule. So DHCP can't be reached | 15:30 |
| noonedeadpunk | aha... but it looks like it's operator who must ensure security groups | 15:31 |
| tweining | DHCP should work in principle or else the "no-resolvconf" element for the image would be pointless. IDK it it relies on it for IP config though. I'd say probably not. | 15:34 |
| noonedeadpunk | or not? | 15:34 |
| noonedeadpunk | Does octavia create a security group for itself? | 15:34 |
| tweining | AFAIK the deployment tool is responsible to do that. I know that TripleO does it. | 15:35 |
| noonedeadpunk | ah, yes, you're right | 15:36 |
| noonedeadpunk | thanks and sorry for bothering! | 15:41 |
| johnsom | noonedeadpunk For the lb-mgmt-net interface, it typically gets it's address from the config drive, but will fall back to DHCP | 15:41 |
| noonedeadpunk | Well, looks it's not present on config drive now somehow | 15:42 |
| johnsom | That is odd | 15:43 |
| johnsom | noonedeadpunk Example from a recent check job: https://zuul.opendev.org/t/openstack/build/16febfbe27684287bc31b7be8deb6fc7/log/controller/logs/octavia-amphora_log.txt#1082 | 15:45 |
| noonedeadpunk | johnsom: console log in my case shows this somehow https://paste.openstack.org/show/bUhFZrANHWWH6jgVPZcL/ | 15:46 |
| johnsom | Yeah, so no IP address provided to cloud-init | 15:47 |
| noonedeadpunk | it's really interesting what could be the reason | 15:47 |
| noonedeadpunk | though if fetch more output it would be like that https://paste.openstack.org/show/bFH8dciiY4NoDj8V0hPh/ | 15:49 |
| noonedeadpunk | so it waits for DHCP I assume anyway | 15:50 |
| noonedeadpunk | and then just fails | 15:50 |
| johnsom | Yeah, the amphora are configured to fall back to DHCP | 15:50 |
| johnsom | Can you provide the output of a "openstack server show" on the amphora instance? | 15:51 |
| noonedeadpunk | well, dhcp is "legitly"not allowed by security groups... | 15:52 |
| johnsom | Yeah, that is the responsibility of the deployment tool to setup the lb-mgmt-net | 15:52 |
| noonedeadpunk | I know. But it used to work like that for years. And last time I know it was passing CI is march 3 | 15:53 |
| johnsom | Hmmm, somebody changed something.... grin | 15:54 |
| noonedeadpunk | Yeah and trying to understand was it us or not. At least security groups were always like that... And eventually I'd assume IP to be contained on config drive indeed | 15:55 |
| johnsom | I am pretty sure OSA always got the IP via config drive | 15:55 |
| noonedeadpunk | yeah, true | 15:55 |
| johnsom | Can you provide a server show output? | 15:55 |
| noonedeadpunk | https://paste.openstack.org/show/bwLWf9LMH8r4zicLDyro/ | 15:56 |
| tweining | could it be caused by https://review.opendev.org/c/openstack/octavia/+/855441? | 15:56 |
| johnsom | Nope, it never worked | 15:57 |
| johnsom | Which is very odd that user_data is being passed to that amphora. | 15:57 |
| johnsom | Did someone change the user_data_config_drive setting in you octavia conf. It should always be False | 15:57 |
| noonedeadpunk | here's octavia.conf https://paste.openstack.org/show/bT5zrTehrFOYb5yCZon8/ | 15:59 |
| noonedeadpunk | (i don't care about secrets - it's sandbox) | 15:59 |
| johnsom | Hmm, can you do a show on the nova flavor? ad6028fb-af69-4c86-886a-83047c7e0d92 | 16:00 |
| noonedeadpunk | https://paste.openstack.org/show/bdaqYPuA1VUPz6KWCsbO/ | 16:02 |
| noonedeadpunk | I can place an SSH key there if you're up to :) | 16:03 |
| johnsom | Well, if it doesn't get an IP.... lol | 16:03 |
| noonedeadpunk | On the deployment I meant - it's a VM. | 16:03 |
| noonedeadpunk | AND, I changed image to include arbitrary user with sudo :) | 16:04 |
| johnsom | I'm really puzzled by where that "user_data" would be coming from | 16:04 |
| noonedeadpunk | So with ssh port forwarding VNC console is available | 16:04 |
| johnsom | Yeah, you can attach with virsh too | 16:05 |
| noonedeadpunk | well, yeah | 16:05 |
| johnsom | Oh, ok, I think that was a workaround Greg added to a rsyslog/systemd issue: | 16:06 |
| johnsom | runcmd: | 16:06 |
| johnsom | - systemctl restart rsyslog | 16:06 |
| johnsom | I guess next step is to find the lb-mgmt-net port in neutron and see if it has an IP | 16:07 |
| noonedeadpunk | it does | 16:07 |
| noonedeadpunk | As once I assign IP inside VM LB is marked as active | 16:08 |
| johnsom | Oh: networks | None | 16:08 |
| johnsom | That is odd, since the conf has: amp_boot_network_list = 1e2fa36c-12b6-4fc9-bc1e-48668cda1523 | 16:09 |
| johnsom | Nova never attached the network | 16:09 |
| noonedeadpunk | also - https://paste.openstack.org/show/bTep3gFgHrAcHW55cnfR/ | 16:09 |
| noonedeadpunk | well, I do see an interface inside | 16:09 |
| noonedeadpunk | https://pasteboard.co/SPAZVEjbbWjW.png | 16:11 |
| johnsom | Yeah, that just seemed odd in the server show output. I'm restacking so I can compare. | 16:11 |
| johnsom | FYI, you can mount the config drive by mounting the sr0 device if you want to look at the metadata | 16:12 |
| noonedeadpunk | I think it's somewhere on 2023.1 - at least octavia version is exactly 12.0.0 | 16:14 |
| noonedeadpunk | it kinda aligned with having no IP in it https://paste.openstack.org/show/bNGH0dceR991yU0OthqO/ | 16:15 |
| noonedeadpunk | which might be kinda nova thing... | 16:15 |
| noonedeadpunk | `"type": "ipv4_dhcp"` | 16:16 |
| noonedeadpunk | sorry, this is output from another amphora - the one I've pasted before got deleted as LB went to ERROR | 16:16 |
| johnsom | Yeah, the port output and the server show had an IP, so I would expect it to be in the metadata. I can confirm in just a few minutes, just about done with the stack | 16:17 |
| noonedeadpunk | Nova is 27.0.1.dev2 (so 2 commits from 27.0.0) | 16:18 |
| johnsom | https://www.irccloud.com/pastebin/fEVIIOCM/ | 16:22 |
| noonedeadpunk | huh | 16:22 |
| johnsom | So my "networks" is also None, but the config drive does have an IP in it | 16:22 |
| noonedeadpunk | for what version is that? | 16:22 |
| johnsom | master branches across the board | 16:23 |
| noonedeadpunk | meaning - metadata version | 16:23 |
| noonedeadpunk | I looked at `openstack/2020-10-14/network_data.json ` | 16:23 |
| johnsom | Oh, I opened "latest", let me check | 16:23 |
| noonedeadpunk | yeah latest is the same | 16:24 |
| johnsom | Yeah, 2020-10-14 is the same | 16:24 |
| noonedeadpunk | hm | 16:24 |
| noonedeadpunk | I probably should bug nova folks then... | 16:24 |
| johnsom | Yeah, that is an odd one. No idea why it wouldn't populate that correctly. It's on the port and server instance info | 16:25 |
| noonedeadpunk | do you have dhcp enabled for the network? | 16:25 |
| gthiemonge | I also have type: "ipv4" and an ip address in this file | 16:25 |
| johnsom | | enable_dhcp | True | 16:25 |
| johnsom | But I'm 99% sure cloud-init is overriding that | 16:26 |
| noonedeadpunk | Weird part is that somehow upgrade job from yoga is passing | 16:27 |
| noonedeadpunk | Well, the biggest change we did - updated version of ansible collections. So smth might be off with network creation | 16:27 |
| johnsom | Yeah, I am pretty sure all of our jobs are passing as we have been merging a bunch of backports | 16:27 |
| johnsom | We haven't cut in stable branch releases yet though | 16:28 |
| noonedeadpunk | yeah, smth is obviously off in metadata | 16:28 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
