| *** ralonsoh_ is now known as ralonsoh | 07:04 | |
| EugenMayer440180 | When deploying a new instance, i get a port binding issues since "no ovn chassis": https://gist.github.com/EugenMayer/de98984752dd4fba1ad33b83e65a751f and https://gist.github.com/EugenMayer/15e6438d7276a3a290c92a2bac059832 - i'am running 2024.2 (via kolla deployment) | 07:09 |
|---|---|---|
| ralonsoh | EugenMayer440180, the compute node Neutron is looking for is compute3 | 07:31 |
| ralonsoh | but most probably the hostname defined in the local OVS database is compute3.cluster.kontextwork.net | 07:31 |
| ralonsoh | check what is in the local ovs | 07:32 |
| ralonsoh | ovs-vsctl list open . | grep external_ids | 07:32 |
| ralonsoh | and the value of "hostname" | 07:32 |
| EugenMayer440180 | ralonsoh interesting - i did not change anything here, but i upgraded to 2024.1 / 2024.2 without spawning any new VMs ... and it might be related to those upgrades | 07:32 |
| ralonsoh | EugenMayer440180, then you need to check the deployment tool | 07:33 |
| EugenMayer440180 | (this is kolla in my case) | 07:33 |
| EugenMayer440180 | ralonsoh: ovs-vsctl list open . | grep external_ids | 07:33 |
| EugenMayer440180 | external_ids : {hostname=controller1, ovn-bridge-mappings="physnet1:br-wan", ovn-cms-options=enable-chassis-as-gw, ovn-encap-ip="10.0.0.3", ovn-encap-type=geneve, ovn-monitor-all="false", ovn-openflow-probe-interval="60", ovn-remote="tcp:10.0.0.3:6642", ovn-remote-probe-interval="60000", system-id=controller1} | 07:33 |
| ralonsoh | not in the controller | 07:33 |
| EugenMayer440180 | understood | 07:33 |
| ralonsoh | EugenMayer440180, also check the agents | 07:34 |
| ralonsoh | openstack network agent list | 07:34 |
| ralonsoh | there should be an ovn-controller agent located in the compute4 | 07:35 |
| ralonsoh | compute3 | 07:35 |
| EugenMayer440180 | ovs-vsctl list open . | grep external_ids | 07:36 |
| EugenMayer440180 | external_ids : {hostname=compute3.cluster.kontextwork.net, ovn-chassis-mac-mappings="physnet1:52:54:00:4c:a6:1f", ovn-encap-ip="10.0.0.6", ovn-encap-type=geneve, ovn-monitor-all="false", ovn-openflow-probe-interval="60", ovn-remote="tcp:10.0.0.3:6642", ovn-remote-probe-interval="60000", system-id=compute3} | 07:36 |
| ralonsoh | and the agent? | 07:36 |
| EugenMayer440180 | --- | 07:36 |
| EugenMayer440180 | That you know, all computes are affected, it is no matter where the payload is scheduled on | 07:36 |
| ralonsoh | the hostname doesn't match what Neutron is looking for | 07:36 |
| EugenMayer440180 | ralonsoh: those are the agent logs: https://gist.github.com/EugenMayer/d050c787cc28851230aacf117c4a6ef5 | 07:37 |
| ralonsoh | EugenMayer440180, no no | 07:37 |
| ralonsoh | the agent list | 07:37 |
| ralonsoh | $ openstack network agent list | 07:38 |
| EugenMayer440180 | sorry, trying my best to follow | 07:38 |
| EugenMayer440180 | ralonsoh: i have ran the upgrade steps once again to try to fix the issue and now i seem to have issues connecting via openstack cli | 07:40 |
| EugenMayer440180 | fixed, ok network agent list: https://gist.github.com/EugenMayer/8293975fdbe9a21295f0e785d435ce4e | 07:41 |
| ralonsoh | Nova will try to bind to port to a compute name with the list provided in | 07:43 |
| ralonsoh | $ openstack hypervisor list | 07:43 |
| EugenMayer440180 | > the hostname doesn't match what Neutron is looking for | 07:43 |
| EugenMayer440180 | i assume this is what you identified as a possible cause from the beginning right? | 07:43 |
| ralonsoh | if these names (that I think are compute1, compute2, etc) | 07:43 |
| ralonsoh | doesn't match with the info in the local OVS, it won't be possible | 07:43 |
| EugenMayer440180 | ralonsoh: https://gist.github.com/EugenMayer/8293975fdbe9a21295f0e785d435ce4e#file-gistfile2-txt | 07:43 |
| ralonsoh | well, not a possible cause but the actual cause | 07:44 |
| EugenMayer440180 | understood | 07:44 |
| ralonsoh | then I don't know what nova is sending, why they are requesting host: compute3 | 07:45 |
| EugenMayer440180 | so the issue is that nova requests host "computeX", while the expectation would we the FQDN "computeX.cluster.kontextwork.net"? | 07:46 |
| ralonsoh | yes | 07:46 |
| ralonsoh | because this is how Neutron is identifying the compute nodes | 07:46 |
| EugenMayer440180 | i assume this is what you can not help with. Could you point me at where you found out what nova is requesting - so which was the actual line that was the offender? | 07:49 |
| ralonsoh | This is the port binding request, not sure exactly in Nova where this is done | 07:50 |
| EugenMayer440180 | i have found according logs in the nova-conducutor and nova-compute.log logs | 07:51 |
| ralonsoh | this is the nova conductor who makes the request | 07:52 |
| EugenMayer440180 | nova-conductor logs are "jsonified", but the compute log reads better in terms https://gist.github.com/EugenMayer/45f5b1c050d7411d5b121b668fc94caf | 07:53 |
| opendevreview | yatin proposed openstack/neutron stable/2024.2: [OVN] Use the MySQL backend for the ``TestOvnNbSync`` tests (2) https://review.opendev.org/c/openstack/neutron/+/943875 | 08:18 |
| opendevreview | yatin proposed openstack/neutron stable/2024.1: [OVN] Use the MySQL backend for the ``TestOvnNbSync`` tests (2) https://review.opendev.org/c/openstack/neutron/+/943876 | 08:18 |
| opendevreview | yatin proposed openstack/neutron stable/2023.2: [OVN] Use the MySQL backend for the ``TestOvnNbSync`` tests (2) https://review.opendev.org/c/openstack/neutron/+/943877 | 08:19 |
| EugenMayer440180 | ralonsoh looking at the sb chassi defintions, the hostname is FQDN, but the chassi name is only the hostname https://gist.github.com/EugenMayer/15e6438d7276a3a290c92a2bac059832#file-controller-ovn-sbctl-show-L18 | 08:25 |
| EugenMayer440180 | but when neutron tries to find the chassi by name, it uses 'compute3' - and looking at the sb output beforehand, it is the right name? https://gist.github.com/EugenMayer/de98984752dd4fba1ad33b83e65a751f#file-neutron-server-log-L1 | 08:26 |
| ralonsoh | EugenMayer440180, Neutron uses what Nova passes in the port binding request, nothing else | 08:28 |
| ralonsoh | check where is the API call in the Neutron logs (in debug mode) | 08:29 |
| EugenMayer440180 | ralonsoh i'am currently trying to understand where openstack is expected to have the hostname and where it expects the host and then trying to understand where the nova request did the wrong thing. Nova tells neutron to use 'compute3' - now this alone is not enough to understand the issues, since either this is now 'the Host' (which would be | 08:34 |
| EugenMayer440180 | correct', or the hostname (in terms of the hypervisor), where we have a FQDN. On my compute3, hostname returns 'compute3' and hostname -f returns 'compute3.cluster.kontextwork.net' - so the basic resolution here is correct. Looking at https://gist.github.com/EugenMayer/8293975fdbe9a21295f0e785d435ce4e#file-gistfile2-txt they hypervisor hostname is | 08:34 |
| EugenMayer440180 | the FQDN, but when using the UI under /admin/hypervisors/ tab 'Compute hosts' the host is 'compute3' - what is nova telling neutron, the 'hypevisor hostname' or the 'compute host' | 08:34 |
| EugenMayer440180 | (sorry if i sound confused - i 'am :) ) trying to sort things out | 08:36 |
| ralonsoh | EugenMayer440180, I would make this questions in the nova channel. I don't know when Nova uses the fqdn or not | 08:39 |
| ralonsoh | it should use it always | 08:39 |
| EugenMayer440180 | ralonsoh the reason i try to circle it in is to formulate a proper question there. But I totally agree, it might belong there. | 08:40 |
| EugenMayer440180 | ralonsoh the point is, the chassis names are 'compute3', not 'compute3.cluster.kontextwork.net' - so looking at what neutron created, there are a couple of places (already) where the host is used | 08:40 |
| ralonsoh | EugenMayer440180, the chassis.hostname must match the local OVS external_ids:hostname value | 08:44 |
| ralonsoh | if that is not the case, then your upgrade did something wrong | 08:45 |
| EugenMayer440180 | and it does AFAICS, right? | 08:46 |
| EugenMayer440180 | external_ids : {hostname=compute3.cluster.kontextwork.net, | 08:46 |
| EugenMayer440180 | which matches: https://gist.github.com/EugenMayer/15e6438d7276a3a290c92a2bac059832#file-controller-ovn-sbctl-show-L19 | 08:46 |
| ralonsoh | the OVN SB chassis.name is the system-id | 08:49 |
| ralonsoh | that you manually defined per host | 08:49 |
| ralonsoh | this value, by default, is a uuid OVS creates | 08:49 |
| ralonsoh | so you also modified the system-id | 08:50 |
| EugenMayer440180 | just to be sure, i did not modify anything manually (I did not fiddle around) | 08:53 |
| EugenMayer440180 | the system-id in my case is system-id=compute3 and this fits the chassis name compute3 | 08:54 |
| EugenMayer440180 | currently trying to follow where the convergation / definition failed to I can look backward and find the cause | 08:54 |
| opendevreview | Renjing Xiao proposed x/whitebox-neutron-tempest-plugin master: Add nested snat test https://review.opendev.org/c/x/whitebox-neutron-tempest-plugin/+/940908 | 09:23 |
| opendevreview | Merged openstack/neutron stable/2024.2: [QoS] QoS rule check also considers the direction https://review.opendev.org/c/openstack/neutron/+/943644 | 10:15 |
| opendevreview | Merged openstack/neutron stable/2024.1: [QoS] QoS rule check also considers the direction https://review.opendev.org/c/openstack/neutron/+/943645 | 10:15 |
| opendevreview | Merged openstack/neutron stable/2023.2: [QoS] QoS rule check also considers the direction https://review.opendev.org/c/openstack/neutron/+/943646 | 10:15 |
| opendevreview | Merged openstack/neutron stable/2024.2: [OVN] Use the MySQL backend for the ``TestOvnNbSync`` tests (2) https://review.opendev.org/c/openstack/neutron/+/943875 | 10:37 |
| opendevreview | Merged openstack/neutron stable/2024.1: [OVN] Use the MySQL backend for the ``TestOvnNbSync`` tests (2) https://review.opendev.org/c/openstack/neutron/+/943876 | 10:37 |
| opendevreview | Renjing Xiao proposed x/whitebox-neutron-tempest-plugin master: Add nested snat test https://review.opendev.org/c/x/whitebox-neutron-tempest-plugin/+/940908 | 11:14 |
| EugenMayer440180 | ralonsoh we found the issue. i think it should be attributed to kolla. The point is, if a compute has 'hostname !== hostname -f', like in our case, compute3 and compute3.cluster.kontextwork.net, the network agent will be registered using the FQDN under host as you have seen in | 11:23 |
| EugenMayer440180 | https://gist.github.com/EugenMayer/8293975fdbe9a21295f0e785d435ce4e#file-gistfile1-txt-L5 under the column host. Later, nova will ask neutron to bind the port using the host 'compute3', and neutron will search in the network agents using the 'Host' column, which will not match | 11:23 |
| EugenMayer440180 | If we change our computes to resolve to compute3 for hostname -f, and run deploy, then network agents will be redeployed using 'compute3' in Host columen, and scheduling instances/servers will work again | 11:24 |
| ralonsoh | EugenMayer440180, perfect then | 11:39 |
| opendevreview | Maor Blaustein proposed x/whitebox-neutron-tempest-plugin master: Wait in both log start track/retrieve https://review.opendev.org/c/x/whitebox-neutron-tempest-plugin/+/943922 | 12:03 |
| opendevreview | Maor Blaustein proposed x/whitebox-neutron-tempest-plugin master: Wait in both log start track/retrieve https://review.opendev.org/c/x/whitebox-neutron-tempest-plugin/+/943922 | 12:09 |
| opendevreview | Merged openstack/neutron stable/2023.2: [OVN] Use the MySQL backend for the ``TestOvnNbSync`` tests (2) https://review.opendev.org/c/openstack/neutron/+/943877 | 12:13 |
| opendevreview | Lajos Katona proposed openstack/neutron master: [eventlet-removal] Remove the usage of eventlet in the SR-IOV agent https://review.opendev.org/c/openstack/neutron/+/942884 | 12:36 |
| EugenMayer440180 | ralonsoh thank you for helping | 12:39 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: [OVN] Retrieve the HA_Chassis_Group in ``_sync_ha_chassis_group`` https://review.opendev.org/c/openstack/neutron/+/943689 | 12:43 |
| opendevreview | Dmitriy Chubinidze proposed openstack/neutron master: Update CIDR notation for loopback and multicast address https://review.opendev.org/c/openstack/neutron/+/942763 | 12:58 |
| opendevreview | Terry Wilson proposed openstack/neutron master: Avoid race condition by using persist_uuid https://review.opendev.org/c/openstack/neutron/+/923971 | 13:09 |
| ralonsoh | ykarel, just to be sure, CI meeting is referred to UTC time, right? | 13:30 |
| ralonsoh | so is in 1,5 hours | 13:30 |
| ykarel | ralonsoh, yes right | 13:30 |
| ralonsoh | hehehe thanks! | 13:30 |
| mlavalle | ykarel: ++ I was about to ask the same question | 13:34 |
| opendevreview | Merged openstack/neutron-fwaas master: Drop v1 to v2 migration https://review.opendev.org/c/openstack/neutron-fwaas/+/929501 | 13:35 |
| opendevreview | yatin proposed openstack/neutron master: [eventlet-removal] Remove logger mechanism from functional tests https://review.opendev.org/c/openstack/neutron/+/943937 | 13:42 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: [fullstack] Re-implement ``wait_until_true`` with eventlet https://review.opendev.org/c/openstack/neutron/+/943942 | 14:37 |
| ykarel | #startmeeting neutron_ci | 15:00 |
| opendevmeet | Meeting started Mon Mar 10 15:00:40 2025 UTC and is due to finish in 60 minutes. The chair is ykarel. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| opendevmeet | The meeting name has been set to 'neutron_ci' | 15:00 |
| ykarel | Ping list: bcafarel, lajoskatona, slawek, mlavalle, mtomaska, ralonsoh, ykarel, jlibosva, elvira | 15:00 |
| mlavalle | \o | 15:00 |
| ralonsoh | hello, video or IRC today? | 15:00 |
| ralonsoh | I always forget | 15:01 |
| ykarel | IRC | 15:01 |
| ralonsoh | cool | 15:01 |
| mlavalle | ++ | 15:02 |
| slaweq | hi | 15:02 |
| lajoskatona | Hi | 15:02 |
| opendevreview | Merged openstack/neutron stable/2024.2: [ovn][trivial] Add 'empty_string_filtering' extension to OVN https://review.opendev.org/c/openstack/neutron/+/943417 | 15:02 |
| opendevreview | Merged openstack/neutron stable/2024.1: [ovn][trivial] Add 'empty_string_filtering' extension to OVN https://review.opendev.org/c/openstack/neutron/+/943418 | 15:02 |
| opendevreview | Merged openstack/neutron stable/2023.2: [ovn][trivial] Add 'empty_string_filtering' extension to OVN https://review.opendev.org/c/openstack/neutron/+/943419 | 15:02 |
| bcafarel | late o/ | 15:03 |
| ykarel | k let's start with topics, others can join in meanwhile | 15:03 |
| ykarel | #topic Actions from previous meetings | 15:03 |
| ykarel | ralonsoh to check and open bug for privsep related failure in functional job | 15:03 |
| ralonsoh | Sorry, I didn't open it yet | 15:04 |
| ralonsoh | I was investigating it | 15:04 |
| ykarel | ack thx for checking, will readd | 15:05 |
| ykarel | #action ralonsoh to check and open bug for privsep related failure in functional job | 15:05 |
| ykarel | ralonsoh to check and open bug for fullstack failures failing at wait_until_true from quite sometime, original suspect was https://review.opendev.org/c/openstack/neutron/+/937843/ | 15:05 |
| ralonsoh | yes | 15:05 |
| ralonsoh | #link https://bugs.launchpad.net/neutron/+bug/2101839 | 15:05 |
| ralonsoh | And I proposed https://review.opendev.org/c/openstack/neutron/+/943942 | 15:05 |
| ralonsoh | temporarily | 15:05 |
| ykarel | thx ralonsoh | 15:06 |
| ykarel | ohkk so eventlet reintroduced in the test | 15:07 |
| ralonsoh | yes only for fullstack tempest | 15:07 |
| ralonsoh | until we remove it from this FW | 15:07 |
| ykarel | ok | 15:08 |
| lajoskatona | I hope we will have time and compony support in the comming cycles for this eventlet work otherwise Neutron will be like the gordian knot | 15:08 |
| ykarel | ++ | 15:08 |
| ralonsoh | I finshed last week the metadata and I'm working now in the L3 agent | 15:08 |
| ralonsoh | but yes, more hands are welcome | 15:08 |
| ykarel | moving to next | 15:09 |
| ykarel | ykarel to check and open bug for ovs tempest failures | 15:09 |
| ykarel | opened https://bugs.launchpad.net/neutron/+bug/2101840 | 15:09 |
| ykarel | #link https://bugs.launchpad.net/neutron/+bug/2101840 | 15:09 |
| ykarel | it's kind of critical now as the ovs jobs failing quite frequently now | 15:10 |
| ykarel | seems related to the eventlet cleanups we have ? | 15:10 |
| ykarel | like https://review.opendev.org/c/openstack/neutron/+/942393 | 15:10 |
| lajoskatona | thanks fro checking, I just read your comment in lp, I check it | 15:10 |
| ralonsoh | but the dhcp agent moved out of eventlet some months ago | 15:10 |
| ykarel | from logs i see dhcp agents have high number of events to process like 200+ | 15:11 |
| ralonsoh | right... it received some adaptations | 15:11 |
| ralonsoh | but the complete removal was finished 2 weeks ago | 15:11 |
| lajoskatona | I check if the eventlet removal is related or not, and check this bug | 15:13 |
| ykarel | thx lajoskatona | 15:13 |
| ykarel | #action lajoskatona to check https://bugs.launchpad.net/neutron/+bug/2101840 | 15:13 |
| ykarel | ykarel to open bug for ping failure post upgrade in grenade | 15:14 |
| ykarel | #link https://bugs.launchpad.net/neutron/+bug/2101166 | 15:14 |
| ykarel | this also seen in ovs jobs, dvr multinode | 15:14 |
| ykarel | not much frequent though | 15:14 |
| ykarel | ykarel to open bug for tempest failure in test_established_tcp_session_after_re_attachinging_sg | 15:15 |
| ykarel | #link https://bugs.launchpad.net/neutron/+bug/2101165 | 15:15 |
| ykarel | this also seen mostly in ovs jobs | 15:15 |
| ykarel | and i recall seeing similar traces from long | 15:16 |
| ykarel | didn't find an old bug so reported this to track | 15:16 |
| ykarel | that's it for action items from previous week | 15:16 |
| ykarel | #topic Stable branches | 15:16 |
| ykarel | bcafarel, any update ^ | 15:16 |
| bcafarel | a few backports last week (and merging), nothing looking bad in the CI runs there | 15:17 |
| ykarel | couple of patches merged in stable | 15:17 |
| ykarel | yeap merges in both stable and unmaintained went fine | 15:17 |
| ykarel | in periodic i saw some failures but none of them consistent this week | 15:17 |
| ykarel | last was openstacksdk but that got resolved already | 15:18 |
| ykarel | thx bcafarel for the updates | 15:18 |
| ykarel | #topic Stadium projects | 15:18 |
| ykarel | all green in periodic | 15:18 |
| ykarel | lajoskatona, anything to add | 15:18 |
| lajoskatona | nothing from my side for stadiums | 15:18 |
| ykarel | k thx | 15:19 |
| ykarel | #topic Rechecks | 15:19 |
| ykarel | we still have couple of rechecks to get patches merged due to known issues | 15:19 |
| ykarel | bare recheck wise we had 4/39 | 15:20 |
| ykarel | out of those 3 in same patch, test patch for pyroute2 master | 15:20 |
| ykarel | so good, let's keep avoiding bare rechecks | 15:20 |
| ralonsoh | yes, sorry for that | 15:20 |
| ralonsoh | Peter is testing pyroute2 with a branch created only for neutron | 15:20 |
| ykarel | np those were from Petr | 15:20 |
| ykarel | ++ thx for working on that | 15:21 |
| ykarel | Now let's check failures | 15:21 |
| ykarel | test_ovn_nb_sync_repair_delete_ovn_nb_db in stable | 15:21 |
| ykarel | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_811/periodic/opendev.org/openstack/neutron/stable/2024.2/neutron-functional-with-uwsgi-fips/811c11f/testr_results.html | 15:21 |
| ykarel | pushed backports https://review.opendev.org/q/Ibaeaa36dbbb9def10a163b4ee071cb432db5a383 | 15:21 |
| ykarel | all merged to stable | 15:21 |
| ykarel | some randomly failures in functional | 15:22 |
| ykarel | - https://27d33baddc3887bf0580-122b756d505eee79094990db0a134990.ssl.cf1.rackcdn.com/periodic/opendev.org/openstack/neutron/master/neutron-functional-with-oslo-master/77dd0c1/testr_results.html | 15:22 |
| ykarel | - https://6047f309b8432f0272cd-3f475577843c6519bda0c8c12c28da61.ssl.cf2.rackcdn.com/periodic/opendev.org/openstack/neutron/master/neutron-functional-with-sqlalchemy-master/1d78049/testr_results.html | 15:22 |
| ykarel | from failures could see traces related to sqlite but as seeing mainly in master and logger driver involved | 15:22 |
| ralonsoh | ^^ I'll check them | 15:22 |
| ykarel | proposed https://review.opendev.org/c/openstack/neutron/+/943937 | 15:22 |
| ykarel | we can check again once this merges | 15:23 |
| ralonsoh | cool | 15:23 |
| lajoskatona | +1 | 15:23 |
| ykarel | test_migration failed twice in last periodic runs in 2024.1 | 15:23 |
| ykarel | - https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d55/periodic/opendev.org/openstack/neutron/stable/2024.1/neutron-fullstack-with-uwsgi-fips/d55b927/testr_results.html | 15:23 |
| ykarel | - https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_b18/periodic/opendev.org/openstack/neutron/stable/2024.1/neutron-fullstack-with-uwsgi-fips/b18873b/testr_results.html | 15:23 |
| ykarel | have to confirm if next run fails similarly or it was intermittent failure | 15:24 |
| ykarel | i can check that tomorrow and report an issue for this in any case if consistent or not | 15:25 |
| ralonsoh | thanks | 15:25 |
| ykarel | #action ykarel to check next runs for fullstack fips and report issue with fullstack | 15:25 |
| ykarel | test_configurations_are_synced_towards_placement(NIC Switch agent) | 15:26 |
| ykarel | https://70ea00e7f434a8c178c6-dd4a29c0fe6beace2356eed9dd1f7a86.ssl.cf2.rackcdn.com/928586/18/gate/neutron-fullstack/cd00a4b/testr_results.html | 15:26 |
| ykarel | lajoskatona, may be you recall ^ | 15:26 |
| ykarel | i recall you looking into this in past | 15:26 |
| ykarel | from perspective of timeout increase is needed or not | 15:27 |
| lajoskatona | yes, occasionally this failed | 15:27 |
| lajoskatona | I check this one also | 15:27 |
| ykarel | thx lajoskatona | 15:28 |
| ykarel | #action lajoskatona to check fullstack test failure test_configurations_are_synced_towards_placement(NIC Switch agent) | 15:28 |
| ykarel | #topic Tempest/Scenario | 15:28 |
| ykarel | ubuntu jammy job fails randomly https://zuul.openstack.org/builds?job_name=neutron-tempest-plugin-ovn-ubuntu-jammy&project=openstack/neutron | 15:28 |
| ykarel | i noticed couple of failures where tests fail during ssh for missing keys | 15:29 |
| ykarel | i recall some recent discussion but couldn't find it | 15:29 |
| ralonsoh | I talked about this to haleyb|out last week | 15:29 |
| ralonsoh | I just proposed to mark it as non-voting (the easy path...) | 15:29 |
| ykarel | ohkk do we already have bug for this? | 15:29 |
| ralonsoh | nope sorry | 15:30 |
| ralonsoh | there was no pattern | 15:30 |
| ykarel | ohkk let's start with bug and see if we need non voting or any fix for it | 15:30 |
| ralonsoh | perfect | 15:30 |
| ralonsoh | I'll open it | 15:30 |
| ykarel | i recall one issue with ovn metadata in jammy, not sure if that's related but can check | 15:30 |
| ykarel | https://bugzilla.redhat.com/show_bug.cgi?id=2172036 and https://bugs.launchpad.net/neutron/+bug/2007166 | 15:31 |
| ykarel | but let's open a new one and we can check if it's same ^ or different | 15:32 |
| ykarel | #action ralonsoh to open an issue for random issues in ovn ubuntu job | 15:32 |
| ykarel | thx ralonsoh | 15:32 |
| ykarel | #topic Periodic | 15:32 |
| ykarel | - pyroute2 functional, already being tracked in https://bugs.launchpad.net/neutron/+bug/2100261 | 15:32 |
| ykarel | #topic Grafana | 15:33 |
| ykarel | https://grafana.opendev.org/d/f913631585/neutron-failure-rate | 15:33 |
| ykarel | let's have a quick look here | 15:33 |
| ykarel | can see some spike in ovs jobs, which we already discussed | 15:35 |
| ykarel | and same in fullstack and functional | 15:35 |
| ykarel | and other check pipeline failures likely patches specific | 15:35 |
| ykarel | anything to add? | 15:35 |
| ralonsoh | no thanks | 15:36 |
| ykarel | k let's move | 15:37 |
| ykarel | #topic On Demand | 15:37 |
| ykarel | anything else you would like to raise? | 15:37 |
| lajoskatona | nothing from me | 15:37 |
| ykarel | k if nothing else, let's close early and have everyone 19 minutes back | 15:41 |
| ykarel | #endmeeting | 15:41 |
| opendevmeet | Meeting ended Mon Mar 10 15:41:25 2025 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:41 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/neutron_ci/2025/neutron_ci.2025-03-10-15.00.html | 15:41 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/neutron_ci/2025/neutron_ci.2025-03-10-15.00.txt | 15:41 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/neutron_ci/2025/neutron_ci.2025-03-10-15.00.log.html | 15:41 |
| mlavalle | \o | 15:41 |
| ykarel | thx all for joining | 15:41 |
| lajoskatona | o/ | 15:41 |
| ralonsoh | bye | 15:41 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: [fullstack] Re-implement ``wait_until_true`` with eventlet https://review.opendev.org/c/openstack/neutron/+/943942 | 15:49 |
| opendevreview | Merged openstack/ovn-bgp-agent stable/2024.2: functional: Use shutil to find privsep-helper executable https://review.opendev.org/c/openstack/ovn-bgp-agent/+/943802 | 15:59 |
| opendevreview | Merged openstack/ovn-bgp-agent stable/2024.1: functional: Use shutil to find privsep-helper executable https://review.opendev.org/c/openstack/ovn-bgp-agent/+/943803 | 16:02 |
| opendevreview | Merged openstack/ovn-bgp-agent stable/2023.2: functional: Use shutil to find privsep-helper executable https://review.opendev.org/c/openstack/ovn-bgp-agent/+/943804 | 16:02 |
| opendevreview | Merged openstack/ovn-octavia-provider master: Validate ovn_nb/sb_connection in config parser https://review.opendev.org/c/openstack/ovn-octavia-provider/+/942591 | 18:06 |
| opendevreview | Fernando Royo proposed openstack/ovn-octavia-provider master: Add sync floating IP support https://review.opendev.org/c/openstack/ovn-octavia-provider/+/929039 | 18:08 |
| opendevreview | Merged openstack/neutron master: Enable ovsdbapp logging in tests https://review.opendev.org/c/openstack/neutron/+/942157 | 20:12 |
| opendevreview | Merged openstack/neutron master: [eventlet-removal] Remove logger mechanism from functional tests https://review.opendev.org/c/openstack/neutron/+/943937 | 20:12 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!