| mathgonzlez | Hi! anyone knows why 2023.2 repo isnt available to clone anymore? | 01:24 |
|---|---|---|
| shallowred07 | my guess is non-slurp releases aren't maintaining an eom branch, I'd pull the 17.9.0 tag instead | 03:31 |
| shallowred07 | https://docs.openstack.org/releasenotes/kolla-ansible/2023.2.html#relnotes-17-9-0-stable-2023-2 | 03:31 |
| mathgonzlez | yeah, i pulled 17.9.0 but its not possible to deploy because try to run prechecks and bootstrap with 2023.2 | 03:31 |
| shallowred07 | ah yeah, I've had similar with eol branches and ended up having to sed a little in our builds | 03:32 |
| mathgonzlez | the main purposes to use 2023.2 and no new release is because after this release OVN cannot reach +18gbps on tenant networks (but this is an issue that i have in pending to report) | 03:32 |
| mathgonzlez | im going to try the workaroung you recomend | 03:34 |
| shallowred07 | yeah, my ansible controller containers usually have something like: sed -i 's/unmaintained\/yoga/yoga-eol/' ~/.local/share/kolla-ansible/requirements.yml | 03:40 |
| shallowred07 | for older releases | 03:40 |
| mathgonzlez | thanks shallow! thats a good one | 03:43 |
| mathgonzlez | and regarding to the performance impact in new release you have some information? is really werid because before 2024.1 i can reach +300gbps in same nodes tenant traffic, after the update from 2024.1 to 2025.1 the speed is capped at 18gbps | 03:44 |
| shallowred07 | I don't actually, I'm just working on 2024 migration now personally, 18gbps still sounds fast enough that the switch would be hardware offloading, but not sure what could give you such a dramatic loss of speed | 03:49 |
| mathgonzlez | i test differents baremetal node with OVN+geneve and if the deployment is made with 2023.2 the performance is blazing fast, after that any VM can reach 18gbps, that is an issue for provider VMs too. i cannot use mellanox6 full speed because is capped at 12gbps for example | 03:51 |
| shallowred07 | how strange! if nobody else has any ideas I'll at least let you know how our lab deploy on 2024 series goes, similar setup just with intel e810s instead of mellanox | 03:55 |
| mathgonzlez | that would be great, i do some test with hpe dl380g10 and g9, and we have a new environment with a amd epyc (i dont have the model rith now but have 200cores + mellanox 6) and in all the environments the behavior is the same, good speed transfer in 2023.2 and 12 to 20 gbps in 2024+ | 03:58 |
| mathgonzlez | Hi all, I'm investigating a confirmed speed degradation on tenant networks between the 2023 and 2024/2025 releases. Any hints or starting points are very welcome. | 05:26 |
| mathgonzlez | If any want to check what is the behavior heres the bug reported: | 06:13 |
| mathgonzlez | https://bugs.launchpad.net/kolla-ansible/+bug/2114813 | 06:13 |
| frickler | we should really change the wording there, these aren't "official" images, just testing images. you really need to build your own images if you are going towards production | 07:04 |
| frickler | that said, the most likely option would be to compare OVN versions, it should be easily possibly to use newer OVN on your 2023.x installation and compare | 07:05 |
| mathgonzlez | is this more acurate : testing images (kolla provided)? | 07:05 |
| frickler | yes, we need to update the bug template | 07:06 |
| mathgonzlez | ok, updated now | 07:06 |
| mathgonzlez | how i can test new OVN usign my actual environment? | 07:10 |
| frickler | well you can build ovn containers with a different version and deploy these. maybe even running ovn containers from 2024.1 would work | 07:22 |
| bbezak | we've hit recently very slow performance with ovn hardware offloading on mellanox cards, after upgrading ovn/ovs - https://mail.openvswitch.org/pipermail/ovs-dev/2025-April/422797.html | 10:19 |
| bbezak | https://issues.redhat.com/browse/OSPRH-15393 | 10:19 |
| bbezak | it is now fixed in c9s nfv repo - https://git.centos.org/rpms/ovn/blob/c9s-sig-nfv-ovn24.03/f/SPECS/ovn24.03.spec#_647 | 10:20 |
| bbezak | please verify ovs/ovn versions in your container | 10:21 |
| mathgonzlez | Greaat! im going to check and come with updates | 10:42 |
| mathgonzlez | thank you!!! | 10:42 |
| bbezak | np ;) | 11:05 |
| mathgonzlez | the version installed in centos container image is more newer than patched in the link you share: | 11:23 |
| mathgonzlez | ovn-controller 24.03.6, but still have bad permonace, likely 10 to 17 gbps, same compute node, using OVN in tenant networks | 11:24 |
| mathgonzlez | Something weird is happening: if I run iperf3 pointing to localhost inside a VM, the performance is awesome—almost 1:1 with bare metal. But when I try between two different VMs, the performance is really bad. | 11:26 |
| mathgonzlez | Another point is whether all processors have the same performance. For example, right now a Xeon v3 16-core , dual Xeon Gold 6138, and dual AMD EPYC 9534 show similar throughput, It does not matter if iperf3 is running with 2, 4, 16, or 32 parallels. | 11:35 |
| bbezak | I assume this is for regular vcpu (not pinned PCPU) - from my experience to achieve line rate with mellanox cx5 and above one needs to use sriov/vf-lag with cpu-pinning. However with vpcu and "hw_vif_multiqueue_enabled" set in the image + good amount of vcpus, performance should be pretty good - highly dependent on the hardware though - https://docs.openstack.org/glance/latest/admin/useful-image-properties.html. | 12:14 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Add way how to change api-paste file https://review.opendev.org/c/openstack/kolla-ansible/+/952700 | 13:55 |
| kevko | anybody ? https://review.opendev.org/c/openstack/kolla/+/952444 | 14:03 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Add way how to change api-paste file https://review.opendev.org/c/openstack/kolla-ansible/+/952700 | 14:11 |
| bbezak | let me see | 14:11 |
| kevko | bbezak: thanks | 14:41 |
| WJeffs49 | Anyone else had issues with nova-api commands to hypervisor info since 2023.1->2024.1 upgrade? I can't seem to work out why apart from "Timeout when reading response headers from daemon process 'nova-api'" in the logs | 14:45 |
| bbezak | kevko: I'll need to do it tomorrow, as I need to go now | 14:52 |
| kevko | bbezak: ok | 15:13 |
| opendevreview | Merged openstack/kayobe stable/2025.1: Use latest CentOS Stream 9 GenericCloud image https://review.opendev.org/c/openstack/kayobe/+/952441 | 15:18 |
| opendevreview | Merged openstack/kayobe stable/2024.1: Use latest CentOS Stream 9 GenericCloud image https://review.opendev.org/c/openstack/kayobe/+/952443 | 15:18 |
| opendevreview | Merged openstack/kayobe stable/2024.2: Use latest CentOS Stream 9 GenericCloud image https://review.opendev.org/c/openstack/kayobe/+/952442 | 15:18 |
| -opendevstatus- NOTICE: Zuul jobs reporting POST_FAILURE were due to an incident with one of our cloud providers; this provider has been temporarily disabled and changes can be rechecked | 22:37 | |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!