| opendevreview | yatin proposed openstack/neutron master: [CI] Bump OVS_BRANCH in ovs/ovn source deploy jobs https://review.opendev.org/c/openstack/neutron/+/893700 | 05:01 |
|---|---|---|
| opendevreview | Lajos Katona proposed openstack/networking-bagpipe master: CI: Change focal nodeset to jammy https://review.opendev.org/c/openstack/networking-bagpipe/+/893709 | 08:40 |
| *** ralonsoh_ is now known as ralonsoh | 08:41 | |
| ralonsoh | lajoskatona, I've reviewed (and approved) your patches for bagpipe and sfc | 08:44 |
| ralonsoh | is there any other one missing? | 08:44 |
| ralonsoh | https://review.opendev.org/c/openstack/networking-bagpipe/+/879463 | 08:45 |
| ralonsoh | ^ if I'm not wrong, this one depends on the sfc patch | 08:45 |
| ralonsoh | so we'll wait until the other one merges | 08:45 |
| *** dmitriis is now known as Guest1871 | 09:12 | |
| lajoskatona | ralonsoh: I still strugling with finding the root cause of the failures of the bagpipe sfc driver with sqlalchemy 2 | 09:20 |
| lajoskatona | ralonsoh: there's another one for the tempest job of bagpipe: https://review.opendev.org/c/openstack/networking-bagpipe/+/893709 | 09:21 |
| lajoskatona | ralonsoh: I just realized that this job uses focal, I hope it is replacable with jammy :-) | 09:22 |
| ralonsoh | I'll follow this patch | 09:27 |
| opendevreview | Merged openstack/networking-sfc master: [sqlalchemy-20] Add context wrapper to ppg operations https://review.opendev.org/c/openstack/networking-sfc/+/893660 | 09:54 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: Revert "[OVN][Trunk] Set the subports correct host during live migration" https://review.opendev.org/c/openstack/neutron/+/893552 | 10:50 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron stable/2023.1: Revert "[OVN][Trunk] Set the subports correct host during live migration" https://review.opendev.org/c/openstack/neutron/+/893553 | 10:52 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron stable/zed: Revert "[OVN][Trunk] Set the subports correct host during live migration" https://review.opendev.org/c/openstack/neutron/+/893554 | 10:52 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron stable/xena: Revert "[OVN][Trunk] Set the subports correct host during live migration" https://review.opendev.org/c/openstack/neutron/+/893555 | 10:52 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron stable/yoga: Revert "[OVN][Trunk] Set the subports correct host during live migration" https://review.opendev.org/c/openstack/neutron/+/893556 | 10:52 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron stable/wallaby: Revert "[OVN][Trunk] Set the subports correct host during live migration" https://review.opendev.org/c/openstack/neutron/+/893557 | 10:52 |
| ralonsoh | lajoskatona, slaweq ^^ sorry, that patch (and https://review.opendev.org/c/openstack/neutron/+/893447), introduced an error in the cold migration with OVN and trunk ports | 10:53 |
| ralonsoh | we need to revert them | 10:53 |
| lajoskatona | ralonsoh: ack | 10:58 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron-dynamic-routing master: Replace "tenant_id" with "project_id" https://review.opendev.org/c/openstack/neutron-dynamic-routing/+/882940 | 11:27 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: Add a new extension "security-groups-rules-belongs-to-default-sg" https://review.opendev.org/c/openstack/neutron/+/883907 | 11:33 |
| ralonsoh | lajoskatona, slaweq, https://review.opendev.org/c/openstack/neutron/+/892564 if you have some mins. thanks! (last petition today) | 11:35 |
| Continuity_ | Hello. all. I have a Zed based OVN+DVR environment which I'm having some "fun" with. When using vlan provider networks for Ironic. If an instance is given a FIP, it becomes uncontactable from the internet, or at least, a couple of pings will get through, and then nothing. Same from inside the network, i can ping outbound sometimes, a couple of pings work then nothing. | 13:15 |
| Continuity_ | I know there has been some work around VLAN + OVN + DVR | 13:15 |
| Continuity_ | but i would love to try and get to the bottom of this. | 13:15 |
| ralonsoh | Continuity_, please check that your neutron server has https://review.opendev.org/c/openstack/neutron/+/879296 and https://review.opendev.org/c/openstack/neutron/+/875673 | 13:19 |
| Continuity_ | ralonsoh: checking.. | 13:20 |
| Continuity_ | ralonsoh: the first fix is there in full, the second. We seem to be missing the utils.py check (line 629) and the corresponding part in ovn_client.py line 1577-1579 | 13:47 |
| Continuity_ | our reads as | 13:47 |
| Continuity_ | https://pastebin.com/LL6zcvwH | 13:48 |
| ralonsoh | Continuity_, qq, when you said instance, that was an ironic instance? | 13:49 |
| ralonsoh | Ping list: bcafarel, elvira, frickler, mlavalle, mtomaska, obondarev, slawek, tobias-urdin, ykarel, lajoskatona, jlibosva, averdagu, amotoki | 13:59 |
| ralonsoh | #startmeeting networking | 14:00 |
| opendevmeet | Meeting started Tue Sep 5 14:00:07 2023 UTC and is due to finish in 60 minutes. The chair is ralonsoh. Information about MeetBot at http://wiki.debian.org/MeetBot. | 14:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 14:00 |
| opendevmeet | The meeting name has been set to 'networking' | 14:00 |
| mlavalle | o/ | 14:00 |
| slaweq | o/ | 14:00 |
| mtomaska | o/ | 14:00 |
| obondarev | hi | 14:00 |
| haleyb | o/ | 14:00 |
| ykarel | o/ | 14:00 |
| ralonsoh | hello all | 14:00 |
| frickler | \o | 14:00 |
| ralonsoh | we have quorum today, I'll wait 30 more seconds | 14:00 |
| elodilles | o/ | 14:01 |
| lajoskatona | o/ | 14:01 |
| ralonsoh | #topic announcements | 14:01 |
| ralonsoh | the schedule | 14:01 |
| ralonsoh | #link https://releases.openstack.org/bobcat/schedule.html | 14:01 |
| ralonsoh | this week is the | 14:01 |
| ralonsoh | * Election Email Deadline | 14:01 |
| ralonsoh | * Election Campaigning Begins | 14:01 |
| opendevreview | Merged openstack/neutron master: Check the device ID and host ID during virtual port binding https://review.opendev.org/c/openstack/neutron/+/892564 | 14:01 |
| ralonsoh | but, of course, we know who is going to be our future PTL | 14:02 |
| rubasov | late o/ | 14:02 |
| ralonsoh | so, in advance, thanks haleyb for proposing yourself | 14:02 |
| lajoskatona | +1 | 14:02 |
| haleyb | o/ | 14:02 |
| mlavalle | ++ | 14:03 |
| ralonsoh | you'll have my support during the next cycle, for sure | 14:03 |
| haleyb | looking forward to leading the project | 14:03 |
| ralonsoh | we had the library releases last week | 14:03 |
| ralonsoh | n-lib created some problems in n-d-r | 14:04 |
| ralonsoh | and a new Bobcat beta version of Neutron was created | 14:04 |
| ralonsoh | the 3rd one | 14:04 |
| ralonsoh | for now, so far so good | 14:04 |
| ralonsoh | now you need to make a last review of the highlights | 14:04 |
| ralonsoh | #link https://review.opendev.org/c/openstack/releases/+/893174 | 14:04 |
| ralonsoh | that should be approved during this week | 14:04 |
| ralonsoh | and the last topic I have for this section is a new episode in the openinfra web | 14:05 |
| ralonsoh | #link https://openinfra.dev/live/#all-episodes | 14:05 |
| ralonsoh | "Cyber Resilience Act: What Now?" | 14:05 |
| ralonsoh | any other topic in this section?? | 14:05 |
| ralonsoh | ok, let's jump to the next topic | 14:06 |
| ralonsoh | #topic bugs | 14:06 |
| ralonsoh | last week report is from elvira | 14:06 |
| ralonsoh | #link https://lists.openstack.org/pipermail/openstack-discuss/2023-September/034958.html | 14:06 |
| ralonsoh | there are still some pending bugs not assigned/to be discussed | 14:07 |
| ralonsoh | #link https://bugs.launchpad.net/neutron/+bug/2033651 | 14:07 |
| ralonsoh | [fullstack] Reduce the CI job time | 14:07 |
| ralonsoh | I opened this bug, I think the goal is clear | 14:07 |
| ralonsoh | fullstack job is taking between 2 and 3 hours, depending on the node | 14:07 |
| ralonsoh | and sometimes it times out | 14:08 |
| ralonsoh | so we need to find a way to reduce the tests, combine them or improve the code | 14:08 |
| ralonsoh | but I think that is a long term goal | 14:08 |
| lajoskatona | good goal | 14:08 |
| ralonsoh | and not easy, for sure, but we can do small steps | 14:08 |
| mtomaska | right. seems more like tech depth than a bug. | 14:09 |
| ralonsoh | anyone can help on this, so you are welcome | 14:09 |
| ralonsoh | the next one is | 14:09 |
| ralonsoh | #link https://bugs.launchpad.net/neutron/+bug/2033293 | 14:09 |
| ralonsoh | "dns integration saying plugin does not match requirements" | 14:09 |
| ralonsoh | but frickler couldn't reproduce it | 14:10 |
| ralonsoh | so we are waiting for more logs | 14:10 |
| ralonsoh | frickler, any comment on this one? | 14:10 |
| ralonsoh | ok, we can keep the "incomplete" tag for now until new logs (neutron server) are provided | 14:11 |
| ralonsoh | the last one is | 14:11 |
| ralonsoh | #link https://bugs.launchpad.net/neutron/+bug/2033683 | 14:11 |
| ralonsoh | openvswitch.agent.ovs_neutron_agent fails to Cmd: ['iptables-restore', '-n']" | 14:11 |
| ralonsoh | ykarel commented on this one | 14:12 |
| ralonsoh | this job has been working fine for months | 14:12 |
| ralonsoh | but now it seems that a binary is missing | 14:12 |
| ykarel | but yes nothing for neutron in this, i asked for more details and suggested what needs to be fixed in tripleo side | 14:13 |
| ralonsoh | as you mentioned, in https://review.opendev.org/c/openstack/kolla/+/761182 "iptables-restore" is missing | 14:13 |
| ralonsoh | to be honest, I don't know why that worked before | 14:14 |
| ralonsoh | but they maybe bumped the iptables library, that also affects these tools | 14:14 |
| ykarel | not iptables-restore but /usr/bin/update-alternatives | 14:14 |
| ralonsoh | sorry, not iptables but nftables | 14:14 |
| ralonsoh | yes, because of nftables | 14:14 |
| ralonsoh | I'll mark this bug as "invalid" from the Neutron point of view | 14:15 |
| ykarel | ok /me not aware about the relation b/w update-alternatives and nftables | 14:15 |
| ykarel | +1 | 14:15 |
| ralonsoh | nftables provide all the iptables legacy support | 14:15 |
| ralonsoh | so thanks for taking care of this one | 14:16 |
| frickler | in kolla iirc we needed to fix a mismatch between what is happening inside the container and outside and what docker does | 14:16 |
| frickler | maybe the situation in tripleo is similar | 14:17 |
| frickler | but not a neutron issue I agree | 14:17 |
| ralonsoh | I think any command is executed from inside the containers, so we don't use any external binary | 14:18 |
| ralonsoh | I only remember one issue related to "ip-netns" but due to some missing permissions | 14:18 |
| ralonsoh | anyway, we can skip this one | 14:19 |
| frickler | but docker does that, and then nftables and legacy iptables may collide | 14:19 |
| frickler | or rather the nftables rule do not get used or something similar | 14:19 |
| ralonsoh | sorry, I used kolla many years ago and I'm not familiar with this tool, to be honest | 14:20 |
| frickler | nevermind, let's go on | 14:20 |
| ralonsoh | I don't have any other bug in the list | 14:20 |
| ralonsoh | do you? | 14:21 |
| frickler | regarding 2033293 I suspect a misconfiguration, but need more data as mentioned in order to confirm | 14:21 |
| ralonsoh | yeah, we can wait for this information if you couldn't reproduce it | 14:21 |
| ralonsoh | this week ykarel is the deputy, next week will be mtomaska | 14:21 |
| ralonsoh | ack? | 14:21 |
| mtomaska | ack | 14:22 |
| ykarel | ack | 14:22 |
| ralonsoh | cool, thanks! | 14:22 |
| ralonsoh | I'm jumping to the next section | 14:22 |
| ralonsoh | #topic community_goals | 14:22 |
| frickler | any progress on the OVN MTU issue? | 14:22 |
| ralonsoh | no, I had some internal issues and I coudbn't spend a single moment on this | 14:23 |
| ralonsoh | we are also in the release weeks | 14:23 |
| ralonsoh | and everything seems to fail! | 14:23 |
| ralonsoh | ok, let's continue | 14:24 |
| ralonsoh | 1) Add support for the service role in neutron API policies | 14:24 |
| ralonsoh | #link https://review.opendev.org/c/openstack/neutron/+/886724 | 14:24 |
| ralonsoh | (I don't think you spend much time on this one last week, right?) | 14:24 |
| ralonsoh | slaweq, | 14:24 |
| slaweq | no, nothing new this week | 14:25 |
| lajoskatona | it is still in merge conflict by gerrit | 14:25 |
| ralonsoh | ok, this will be a C release feature but you are free to review the patch ^^ | 14:25 |
| ralonsoh | the next one is | 14:26 |
| ralonsoh | 2) Neutron client deprecation | 14:26 |
| ralonsoh | lajoskatona, please | 14:26 |
| lajoskatona | There's 3 open patche that can fit to Bobcat: | 14:26 |
| lajoskatona | https://review.opendev.org/q/topic:bug/1999774+project:openstack/python-neutronclient+status:open | 14:26 |
| lajoskatona | these are all for neutronclient, as SDK release happened with the SDK side of these | 14:27 |
| lajoskatona | that's it for this topic from me | 14:27 |
| ralonsoh | qq: you are not bumping the openstacksdk library in any of them | 14:27 |
| ralonsoh | that should be necessary to receive the dependant sdk patch | 14:28 |
| lajoskatona | As I know we got the highest from upper-constraints and that was bumped | 14:28 |
| ralonsoh | perfect | 14:28 |
| ralonsoh | I said that because we had an issue last week | 14:28 |
| ralonsoh | related to this | 14:28 |
| ralonsoh | #link https://review.opendev.org/c/openstack/python-neutronclient/+/893346 | 14:28 |
| lajoskatona | this was the bump: https://review.opendev.org/c/openstack/requirements/+/893351 | 14:29 |
| lajoskatona | yes last week there was some central issue with requ bumps | 14:29 |
| ralonsoh | yeah, but I was talking about the neutronclient min reqs | 14:29 |
| lajoskatona | ah, true, that is necessary | 14:30 |
| ralonsoh | please check what sdk version has each patch | 14:30 |
| ralonsoh | and bump it correspondingly | 14:30 |
| lajoskatona | ack | 14:30 |
| ralonsoh | thanks a lot | 14:30 |
| ralonsoh | and that's all | 14:31 |
| ralonsoh | #topic on_demand | 14:31 |
| ralonsoh | any topic you want to bring here? | 14:31 |
| mlavalle | not me | 14:31 |
| lajoskatona | I added one | 14:32 |
| lajoskatona | it is a heads up: Nova EOL-ed Train branch | 14:32 |
| ralonsoh | yes, good topic | 14:32 |
| ralonsoh | what should we do? | 14:32 |
| lajoskatona | perhaps elodilles has more background | 14:32 |
| ralonsoh | we still accept Train patches | 14:32 |
| elodilles | well, nova had some CVE fix that did not land on train, hence the team decided to EOL train | 14:33 |
| elodilles | so i'm not insisting anymore to keep train open for neutron either o:) | 14:33 |
| lajoskatona | yes something like that, so it can be that we do the same or keep the branch open | 14:33 |
| ralonsoh | we still don't have this problem. Some users asked us to keep it open, months ago | 14:33 |
| ralonsoh | but I didn't see any activity on this branch | 14:34 |
| lajoskatona | but anyway as things going most projects close most of these branches soon I suppose | 14:34 |
| ralonsoh | ok, I can send a mail (again) to propose the EOL of Train in Neutron | 14:34 |
| ralonsoh | and we can receive feedback of the community | 14:34 |
| elodilles | ralonsoh: ++ | 14:34 |
| lajoskatona | +1 | 14:35 |
| elodilles | (it still can be kept open, but then we have to create a patch for devstack to consume train-eol from nova) | 14:35 |
| ralonsoh | IMO, this branch is old enough at this point | 14:35 |
| ralonsoh | ok then, I'll send the mail today. Thanks! | 14:35 |
| ralonsoh | any other topic? | 14:35 |
| elodilles | thanks too | 14:35 |
| ralonsoh | elodilles, thanks! | 14:35 |
| elodilles | maybe this: since we have talked about python-neutronclient: https://review.opendev.org/c/openstack/releases/+/893615 | 14:36 |
| ralonsoh | so please remember the CI meeting is in 25 mins in this channel | 14:36 |
| mlavalle | video or irc? | 14:36 |
| ralonsoh | elodilles, ups, I missed this patch | 14:36 |
| ykarel | irc | 14:36 |
| ralonsoh | we can delay that some days | 14:36 |
| ralonsoh | until we have lajoskatona's patches | 14:36 |
| mlavalle | ack ykarel | 14:36 |
| ralonsoh | lajoskatona, so please, check your nclient patches asap | 14:37 |
| elodilles | ralonsoh: no problem, you are not late :) | 14:37 |
| ralonsoh | and then I'll update the release hash | 14:37 |
| Continuity_ | ralonsoh: no this is currently a virtual instance, although we do see the same issue on BareMetal as well.. | 14:37 |
| elodilles | note that *client libs freeze was last week | 14:37 |
| elodilles | this is just a stable/2023.2 branch cut patch | 14:38 |
| ralonsoh | right | 14:38 |
| lajoskatona | ok, so the above 3 patches are anyway to C | 14:38 |
| ralonsoh | actually for nclient we'll need to release a new stable version to have them in Bobcat | 14:38 |
| ralonsoh | yes | 14:39 |
| ralonsoh | lajoskatona, is that a problem or we need to have them in Bobcat? | 14:39 |
| lajoskatona | I dont think so | 14:39 |
| ralonsoh | ok then | 14:39 |
| lajoskatona | nothing will break if don't have these, and we don't have a deadline for it as I know | 14:40 |
| ralonsoh | right | 14:40 |
| ralonsoh | so please folks check the non-client libs freeze https://review.opendev.org/c/openstack/releases/+/893615 | 14:40 |
| ralonsoh | and that's all | 14:40 |
| ralonsoh | thank you all for attending | 14:40 |
| ralonsoh | #endmeeting | 14:41 |
| opendevmeet | Meeting ended Tue Sep 5 14:41:06 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 14:41 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/networking/2023/networking.2023-09-05-14.00.html | 14:41 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/networking/2023/networking.2023-09-05-14.00.txt | 14:41 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/networking/2023/networking.2023-09-05-14.00.log.html | 14:41 |
| mlavalle | o/ | 14:41 |
| lajoskatona | o/ | 14:41 |
| elodilles | thanks o/ | 14:41 |
| ykarel | o/ | 14:41 |
| obondarev | o/ | 14:42 |
| Continuity_ | ralonsoh: sorry didnt mean to comment in the middle of the meeting.. bad form : | 14:47 |
| ralonsoh | no problem | 14:48 |
| ralonsoh | the issue in OVN with ironic nodes (and SRIOV ports) could be the scheduler | 14:49 |
| ralonsoh | these ports don't fall in the same chassis as the LRPs | 14:49 |
| ralonsoh | but this could be a performance issue | 14:49 |
| ralonsoh | what you are describing is a communication problem | 14:50 |
| ralonsoh | please check the missing patch and apply it | 14:50 |
| Continuity_ | so the odd thing is we build containers a few weeks ago (kolla-ansible) so I would have thought that we would have that second patch in our code... | 14:53 |
| Continuity_ | looking at that second patch it looks to say. if the network type is vlan, and DVR is enabled, set reside on redirect chassis to false | 14:54 |
| ralonsoh | yes | 14:55 |
| ralonsoh | and the bridge-type | 14:55 |
| Continuity_ | so redirect is set to false, and the bridge type is set to bridged | 14:55 |
| ralonsoh | yes | 14:56 |
| Continuity_ | https://pastebin.com/az7aYS67 | 14:57 |
| Continuity_ | the current settings, which looks to match what should be set by the patch | 14:57 |
| Continuity_ | *note I havent changed anything, this is how the openstack network create command deployed it | 14:57 |
| *** dasm is now known as Guest1929 | 14:57 | |
| ralonsoh | do you have port forwarding? | 14:58 |
| Continuity_ | not sure what you mean. sorry | 15:00 |
| ralonsoh | floating IP port forwarding | 15:00 |
| ykarel | k meeting time | 15:00 |
| ykarel | #startmeeting neutron_ci | 15:01 |
| opendevmeet | Meeting started Tue Sep 5 15:01:04 2023 UTC and is due to finish in 60 minutes. The chair is ykarel. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:01 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:01 |
| opendevmeet | The meeting name has been set to 'neutron_ci' | 15:01 |
| ralonsoh | hello | 15:01 |
| lajoskatona | o/ | 15:01 |
| ykarel | ping bcafarel, lajoskatona, mlavalle, mtomaska, ralonsoh, ykarel, jlibosva, elvira | 15:01 |
| ykarel | Grafana dashboard: https://grafana.opendev.org/d/f913631585/neutron-failure-rate?orgId=1 | 15:01 |
| mtomaska | o/ | 15:01 |
| mlavalle | o/ | 15:02 |
| *** Guest1929 is now known as dasm | 15:02 | |
| ykarel | k let's start with the topic as some of the folks are on PTO | 15:02 |
| ykarel | #topic Actions from previous meetings | 15:02 |
| ykarel | lajoskatona to check failures for bagpipe and check use of master neutron in stadium projects | 15:02 |
| lajoskatona | I still strugle with bagpipe | 15:03 |
| lajoskatona | it is only with sqlalchemy2 | 15:03 |
| ykarel | yes that's what i recall | 15:04 |
| lajoskatona | In the meantime I also found that the bagpipe tempest job is running with focal, so I changed the nodeset to jammy: https://review.opendev.org/c/openstack/networking-bagpipe/+/893709 | 15:04 |
| ykarel | looking at periodic i saw some issue in other job, but that can checked in later section | 15:04 |
| ykarel | yeap i noticed that failure too, thx for fixing it | 15:04 |
| ykarel | other was https://zuul.openstack.org/builds?job_name=networking-bagpipe-openstack-tox-py310-with-sqlalchemy-main&project=openstack/networking-bagpipe | 15:05 |
| lajoskatona | the issue is only with the sfc driver of bagpipe, so quite complex for me at least to understand the root cause | 15:05 |
| lajoskatona | yes that is the sqlalchemy2 issue | 15:06 |
| Continuity_ | ralonsoh: yes i have allowed all ICMP and SSH for testing. | 15:06 |
| lajoskatona | ralonsoh: what is the timeline for the sqlalchemy2 introduction? | 15:06 |
| slaweq | o/ | 15:06 |
| slaweq | sorry I was on different meeting | 15:06 |
| ykarel | lajoskatona, do we have bug already for this issue? | 15:06 |
| ralonsoh | it will be introduced, most probably, at the begining of C | 15:06 |
| lajoskatona | ok | 15:07 |
| ralonsoh | there is a requirements patch proposed already | 15:07 |
| ralonsoh | Continuity_, we can talk after the meeting | 15:07 |
| lajoskatona | ralonsoh: not for bagpipe, I used as reference the big sqlalchemy2 bug | 15:07 |
| lajoskatona | I can open a bagpipe bug to track it | 15:08 |
| ralonsoh | yeah, much better | 15:08 |
| ykarel | +1 | 15:08 |
| lajoskatona | ack | 15:08 |
| ralonsoh | to be honest, I don't know what is failing in this CI | 15:08 |
| ykarel | let's check this offline | 15:09 |
| ykarel | #topic Stable branches | 15:09 |
| ykarel | Bernard is out this week, but stable branches looks good considering patches merge | 15:09 |
| ykarel | i din't noticed any consistent failure on stable branches | 15:10 |
| ykarel | anything to add for stable branches? | 15:10 |
| ykarel | sounds all good then, moving to next topic | 15:12 |
| ykarel | #topic Stadium projects | 15:12 |
| ykarel | lajoskatona, anything else apart from that sqlalchemy and focal issues for stadium projects? | 15:12 |
| lajoskatona | except bagpipe things seems to be green | 15:12 |
| ykarel | k good | 15:12 |
| ykarel | #link https://zuul.openstack.org/builds?job_name=networking-bagpipe-openstack-tox-py310-with-sqlalchemy-main&project=openstack/networking-bagpipe | 15:13 |
| ykarel | #link https://zuul.openstack.org/builds?job_name=networking-bagpipe-tempest&project=openstack%2Fnetworking-bagpipe&branch=master&skip=0 | 15:13 |
| ykarel | #topic Grafana | 15:13 |
| ykarel | https://grafana.opendev.org/d/f913631585/neutron-failure-rate | 15:13 |
| ykarel | let's give a minute to it if we observe anything abnormal there | 15:13 |
| ykarel | i see a spike at tempest job, but that was a known issue already fixed | 15:14 |
| slaweq | today morning there was some "spike" but on all jobs | 15:14 |
| slaweq | so it's probably not an issue on our side really | 15:14 |
| ralonsoh | neutron-ovn-tempest-ipv6-only-ovs-release had a 66% of failures | 15:15 |
| ykarel | yes i noticed there was quite of failures today morning but most of them were related to series of patches pushed together for l3-ovn iirc | 15:15 |
| ralonsoh | right, perfect then | 15:15 |
| ralonsoh | yes, I see the same spikes in other jobs | 15:16 |
| ykarel | yeap, let's move to next section, will keep monitoring it if something new comes in | 15:18 |
| ykarel | #topic Rechecks | 15:18 |
| ykarel | it was better last week | 15:19 |
| ykarel | there were some known issues last week which might have resulted into those rechecks | 15:19 |
| ykarel | bare rechecks were also not much 3/17, so good | 15:19 |
| ykarel | let's keep avoiding bare rechecks | 15:20 |
| ykarel | #topic Unit tests | 15:20 |
| mlavalle | what's 3/17? | 15:20 |
| ralonsoh | 3 out of 17 | 15:20 |
| ykarel | 17 total rechecks, 3 out of them were bare | 15:20 |
| ykarel | yeap | 15:20 |
| mlavalle | ahhh! | 15:20 |
| mlavalle | LOL | 15:20 |
| ykarel | #info There was issue with unit test job running with sqlalchemy/alembic main branches | 15:21 |
| ykarel | It's already fixed with https://review.opendev.org/c/openstack/neutron/+/893602 | 15:21 |
| ralonsoh | +1 | 15:21 |
| ykarel | #topic fullstack/functional | 15:22 |
| ykarel | neutron.tests.functional.services.trunk.drivers.ovn.test_trunk_driver.TestOVNTrunkDriver.test_subport_delete | 15:22 |
| ykarel | AttributeError: 'NoneType' object has no attribute 'status' | 15:22 |
| ykarel | Seen twice in a master/wallaby and failure looks related to the patch itself already merged | 15:23 |
| ykarel | so some race in the test as not happening always | 15:23 |
| ykarel | #link https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_b18/892890/2/gate/neutron-functional-with-uwsgi/b18c02c/testr_results.html | 15:23 |
| ykarel | #link https://e872331dabdf974ff450-5a66e2fcfa24aae6b75c2058251d7e58.ssl.cf5.rackcdn.com/periodic/opendev.org/openstack/neutron/master/neutron-functional-with-uwsgi-fips/c431c66/testr_results.html | 15:24 |
| ykarel | https://review.opendev.org/#/q/I2370ea2f96e2e31dbd43bf232a63394388e6945f | 15:25 |
| ralonsoh | I'll ping Arnau to check these errors, look related to ^ | 15:25 |
| ykarel | i see this being reverted, so likely the failures would also go away | 15:25 |
| ralonsoh | ah no | 15:25 |
| ralonsoh | this could be another issue | 15:25 |
| ralonsoh | in any case, we are reverting the patch you mentioned | 15:25 |
| ralonsoh | so please wait until the next week | 15:25 |
| ykarel | k +1, will keep an eye | 15:26 |
| ykarel | if it still happens will open a bug | 15:26 |
| ykarel | next one is neutron.tests.functional.agent.l3.test_keepalived_state_change.TestMonitorDaemon.test_read_queue_change_state | 15:26 |
| ykarel | AssertionError: Text not found in file /tmp/tmpuh6gesvz/tmp50ei9qfp/log_file: "Initial status of router". | 15:26 |
| ykarel | https://157ba513c840b85e5d0e-e65fbda5c4a8fc14eb81d398bd7b0a80.ssl.cf5.rackcdn.com/892896/1/gate/neutron-functional-with-uwsgi/0acdecd/testr_results.html | 15:26 |
| ralonsoh | this is something recurrent, the monitor doesn't start in some tests | 15:27 |
| ykarel | k so it's already a known issue? | 15:28 |
| ralonsoh | I would need to investigate how to make these tests more stable | 15:28 |
| ralonsoh | yes, that has been happening for years | 15:28 |
| ralonsoh | not very often | 15:28 |
| ykarel | ok i see 3 failures in last 15 days across branches | 15:28 |
| ykarel | k thanks, will keep an eye on this and if it's start happening more frequently will open a bug for it | 15:29 |
| ralonsoh | sure | 15:29 |
| ykarel | neutron.tests.functional.services.ovn_l3.test_plugin.TestRouter.test_router_gateway_port_binding_host_id | 15:30 |
| ykarel | Timeout exception with self.mech_driver.nb_ovn.ovsdb_connection.stop() | 15:30 |
| ykarel | #link https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_2a0/892897/1/gate/neutron-functional-with-uwsgi/2a03caa/testr_results.html | 15:30 |
| ykarel | any idea for this? | 15:31 |
| ralonsoh | no, just a random timeout during the cleanup phase | 15:31 |
| ykarel | seen 4 hits as per opensearch across stable/zed and yoga | 15:31 |
| ralonsoh | in the same test? | 15:32 |
| ykarel | no different test also hit it, like test_gateway_chassis_rebalance_max_chassis | 15:34 |
| ykarel | but that's also in same class neutron.tests.functional.services.ovn_l3.test_plugin.TestRouter | 15:34 |
| ralonsoh | let me open a LP bug for this one. We can try, maybe, checking if the ovsdb_connection is still open during the cleanup | 15:36 |
| ykarel | k thanks | 15:36 |
| ralonsoh | if the connection is not active, then we don't need to stop it | 15:36 |
| ykarel | #action ralonsoh to open bug for Timeout exception with self.mech_driver.nb_ovn.ovsdb_connection.stop() | 15:36 |
| ykarel | neutron.tests.fullstack.test_connectivity.TestUninterruptedConnectivityOnL2AgentRestart.test_l2_agent_restart(LB,VLANs) | 15:37 |
| ykarel | #link https://2df199e43476e3c732e7-3130556d487e5cec46a1ba3d1eaa7fda.ssl.cf5.rackcdn.com/892890/2/gate/neutron-fullstack-with-uwsgi/209726b/testr_results.html | 15:37 |
| ykarel | #link https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_da8/892869/3/check/neutron-fullstack-with-uwsgi/da88e54/testr_results.html | 15:37 |
| ykarel | anyone recall this failure, from meeting history i see it was seen in past too | 15:38 |
| ralonsoh | no, sorry | 15:38 |
| slaweq | this is related to LB so we can simply mark this test as unstable or simply skip it if it's not stable | 15:39 |
| slaweq | as LB is marked as "experimental" since some time | 15:39 |
| lajoskatona | +1 | 15:39 |
| slaweq | I can propose patch for that | 15:39 |
| ykarel | k thanks, i think i saw it in stable branches | 15:40 |
| ykarel | #action slaweq to check failures with fullstack test test_l2_agent_restart | 15:40 |
| ykarel | thx slaweq | 15:40 |
| ykarel | neutron.tests.fullstack.test_agent_bandwidth_report.TestPlacementBandwidthReport.test_configurations_are_synced_towards_placement(Open vSwitch agent) | 15:41 |
| lajoskatona | do you have link for this one? I can check it | 15:41 |
| ykarel | lajoskatona, may be you recall something for ^? | 15:41 |
| ykarel | #link https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_4fb/893543/1/check/neutron-fullstack-with-uwsgi/4fba5fb/testr_results.html | 15:41 |
| ykarel | #link https://c7cd1f2001ec5f5b729f-4854ed941a7816d1225c43ae9b456d0e.ssl.cf2.rackcdn.com/893143/1/check/neutron-fullstack-with-uwsgi/9137417/testr_results.html | 15:42 |
| lajoskatona | I will check this one | 15:42 |
| ykarel | thx lajoskatona | 15:42 |
| ykarel | #action lajoskatona to check failures with fullstack test test_configurations_are_synced_towards_placement | 15:42 |
| ykarel | also generic one | 15:42 |
| ykarel | we recently seeing quite frequent timeouts in fullstack job https://zuul.openstack.org/builds?job_name=neutron-fullstack-with-uwsgi&project=openstack%2Fneutron&result=TIMED_OUT&skip=0 | 15:43 |
| ykarel | even after timeout increase to 3 hours as part of isolated db per test patch | 15:43 |
| ralonsoh | we can start removing the LB tests, for example | 15:44 |
| lajoskatona | for this ralonsoh opend the bug: https://bugs.launchpad.net/neutron/+bug/2033651 | 15:44 |
| lajoskatona | so we can track the patches under the lp | 15:44 |
| ykarel | yeap +1 | 15:44 |
| ykarel | ok let's move to next topic | 15:45 |
| ykarel | #topic Tempest/Scenario | 15:45 |
| ykarel | there was an issue but already fixed with https://review.opendev.org/c/openstack/nova/+/893502 | 15:45 |
| ykarel | #topic Periodic | 15:45 |
| ykarel | #link https://zuul.openstack.org/builds?job_name=devstack-tobiko-neutron&branch=master&skip=0 | 15:46 |
| ykarel | the job was running with ubuntu focal | 15:46 |
| ykarel | there is already a patch to move it to jammy with https://review.opendev.org/c/x/devstack-plugin-tobiko/+/893662 | 15:46 |
| ykarel | #link https://zuul.openstack.org/builds?job_name=neutron-ovn-tempest-ipv6-only-ovs-master&job_name=neutron-ovn-tempest-ovs-master-centos-9-stream&project=openstack%2Fneutron&skip=0 | 15:46 |
| ykarel | ovs/ovn source deploy jobs with OVN_BRANCH=main are broken https://bugs.launchpad.net/neutron/+bug/2034096 | 15:47 |
| ykarel | #link https://review.opendev.org/c/openstack/neutron/+/893700 | 15:47 |
| ykarel | that's it for periodic, please review the above fixes | 15:48 |
| ralonsoh | +2 (high priority one0 | 15:48 |
| ykarel | #topic On Demand | 15:48 |
| ykarel | anything else to discuss? | 15:49 |
| lajoskatona | nothing from me | 15:51 |
| slaweq | nope | 15:51 |
| ykarel | k thanks everyone, let's close then and have everyone few minutes back | 15:52 |
| mlavalle | nothing | 15:52 |
| ykarel | #endmeeting | 15:52 |
| opendevmeet | Meeting ended Tue Sep 5 15:52:15 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:52 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/neutron_ci/2023/neutron_ci.2023-09-05-15.01.html | 15:52 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/neutron_ci/2023/neutron_ci.2023-09-05-15.01.txt | 15:52 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/neutron_ci/2023/neutron_ci.2023-09-05-15.01.log.html | 15:52 |
| mlavalle | o/ | 15:52 |
| slaweq | o/ | 15:52 |
| lajoskatona | o/ | 15:52 |
| ralonsoh | bye | 15:52 |
| mtomaska | o/ | 15:52 |
| Continuity_ | ralonsoh: sorry about that I should pay more attention... | 15:55 |
| Continuity_ | one thing that does strike me as odd, we dont have an ha-chassis group for this new provider network. | 16:02 |
| ralonsoh | that is created when the network is | 16:03 |
| ralonsoh | please, open a LP bug with the information and upload the logs | 16:03 |
| ralonsoh | that will help to find the issue, most probably | 16:03 |
| Continuity_ | ok ill open a bug. as currently we dont see a ha-chassis group for newly created provider vlan networks. | 16:10 |
| Continuity_ | to confirm - ovn-nbctl ha-chassis-group-list | 16:10 |
| Continuity_ | does not return an item for the newly created network | 16:11 |
| Continuity_ | any ideas on how we could force a sync or creation of that | 16:11 |
| ralonsoh | hold on, this is only for external ports | 16:14 |
| ralonsoh | ha-chassis-group | 16:14 |
| ralonsoh | what version are you using? | 16:15 |
| ralonsoh | the default ha chasiss group is no longer used | 16:16 |
| Continuity_ | we are running ZED, containers built 2023-08-23 | 16:18 |
| Continuity_ | we have some vlan provider networks which are working. They have a mix of ironic and virtaul intances on them. They have ha-chassis groups show in ovn | 16:18 |
| Continuity_ | a newly created vlan provider network, with virtual instances on them, exhibit the issue where outgoing pings from the machine work twice then stop, (with or without a fip) when a fip is attached, incoming pings do the same, two then nothing. | 16:19 |
| Continuity_ | occasaionly it just starts working. then it stops again | 16:19 |
| Continuity_ | if "feels" like a traffic centralisation problem, or a pathing problem | 16:20 |
| Continuity_ | but we are at a loss. | 16:20 |
| ralonsoh | ok, at this point we are still providing support for non tunnelled networks and DVR in OVN | 16:21 |
| ralonsoh | so please, don't use DVR with VLAN for now | 16:21 |
| Continuity_ | ok..... how do i switch back? | 16:21 |
| Continuity_ | i assume i need to disable DVR? | 16:22 |
| ralonsoh | unset the enable_distributed_floating_ip flag in the config and restart the neutron servers | 16:23 |
| Continuity_ | how damaging will that be to currently running workloads? | 16:25 |
| Continuity_ | will we need to rebuild anything or will it just work? | 16:25 |
| Continuity_ | will it affect north south traffic during the restart? | 16:25 |
| ralonsoh | but you said the FIPs are currently not working | 16:25 |
| ralonsoh | yes, it will affect N/S traffic | 16:26 |
| Continuity_ | only on instances connected to these provider vlan networks | 16:26 |
| Continuity_ | other geneve tenant networks are working fine | 16:26 |
| ralonsoh | so that will be a problem | 16:26 |
| ralonsoh | we don't have a way to unset DVR only for VLAN | 16:26 |
| Continuity_ | this bug https://review.opendev.org/c/openstack/neutron/+/875673 which you mentioned earlier talks about "ovn-chassis-mac-mappings", which are not configured in our environment. should they be? | 16:28 |
| ralonsoh | but this is something working now | 16:29 |
| ralonsoh | you should have the mac-mappins for VLAN | 16:29 |
| Continuity_ | ok. | 16:29 |
| Continuity_ | i *think* it was removed as part of a kolla-ansible change a while back | 16:30 |
| ralonsoh | that is a configuration step for the phsycail bridges | 16:30 |
| Continuity_ | maybe it hasnt been readded | 16:30 |
| ralonsoh | actually this is not controlled by Neutron, but by puppet (TripleO), for example | 16:31 |
| Continuity_ | yeah, so i *think* we are missing the ovn-chassis-mac-mappings on the controllers. | 16:36 |
| Continuity_ | we shall try adding that tomorrow | 16:40 |
| Continuity_ | ralonsoh: thanks for your assistance. ill let you know how we get on. and i may be back with more questions :D | 16:40 |
| ralonsoh | sure | 16:40 |
| opendevreview | Merged openstack/neutron master: Revert "[OVN][Trunk] Set the subports correct host during live migration" https://review.opendev.org/c/openstack/neutron/+/893552 | 16:50 |
| opendevreview | Merged openstack/neutron stable/xena: Update dns_assignment attribute documentation https://review.opendev.org/c/openstack/neutron/+/893544 | 16:50 |
| opendevreview | Merged openstack/neutron stable/wallaby: Update dns_assignment attribute documentation https://review.opendev.org/c/openstack/neutron/+/893545 | 16:50 |
| opendevreview | Merged openstack/neutron master: [CI] Bump OVS_BRANCH in ovs/ovn source deploy jobs https://review.opendev.org/c/openstack/neutron/+/893700 | 16:50 |
| TheJulia | ralonsoh: any answer on SNAT + tftp from ovn folks? | 17:25 |
| opendevreview | Brian Haley proposed openstack/neutron stable/zed: DNM: Testing Zed gate https://review.opendev.org/c/openstack/neutron/+/893807 | 22:00 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!