| *** kopecmartin|off is now known as kopecmartin | 07:04 | |
| opendevreview | Lajos Katona proposed openstack/tap-as-a-service stable/stein: [stable-only]: Fix reno on stein https://review.opendev.org/c/openstack/tap-as-a-service/+/897147 | 07:17 |
|---|---|---|
| opendevreview | Lajos Katona proposed openstack/tap-as-a-service stable/stein: [stable-only]: Fix reno on stein https://review.opendev.org/c/openstack/tap-as-a-service/+/897147 | 07:18 |
| opendevreview | Slawek Kaplonski proposed openstack/neutron-tempest-plugin master: New basic API tests for the default SG rules templates CRUDs https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/883553 | 08:05 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: [OVN] Add the default condition check in ``PortBindingChassisEvent`` https://review.opendev.org/c/openstack/neutron/+/896926 | 08:25 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: [OVN] Remove backwards compatibility with OVN < v20.09 https://review.opendev.org/c/openstack/neutron/+/884898 | 08:25 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: Missing SG rule default group extension in ``Ml2Plugin`` https://review.opendev.org/c/openstack/neutron/+/897211 | 10:33 |
| opendevreview | Slawek Kaplonski proposed openstack/neutron-tempest-plugin master: New basic API tests for the default SG rules templates CRUDs https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/883553 | 11:02 |
| ralonsoh | haleyb, hey, just to be sure: you'll chair the Neutron meeting now, right? | 11:34 |
| ralonsoh | I think we should update the chair list | 11:34 |
| haleyb | ralonsoh: yes, i can chair it | 12:27 |
| haleyb | ralonsoh: regarding your patch for security_groups_rules_belongs_to_default_sg, i think there are typos in the neutron-lib patch from quickly looking, seems the name security_group_rules_belongs_to_default_sg (group not groups) was used in some places like the api ref, but the files have the 's' - which one is correct? | 12:29 |
| * haleyb guesses it's groups not group | 12:30 | |
| ralonsoh | haleyb, let me check that | 12:38 |
| haleyb | ralonsoh: i could also be confused because the api has security_group_rule-belongs-to-default-sg as a parameter but the extension has the 's' | 12:40 |
| ralonsoh | sorry but I don't get that | 12:41 |
| haleyb | ralonsoh: the parameter to a POST is security_group_rule-belongs-to-default-sg in api-ref, but the extension is named security-groups-rules-belongs-to-default-sg | 12:45 |
| haleyb | one uses group the other groups | 12:46 |
| haleyb | it's early here so might be seeing it wrong, but your patch also failed to find the extension | 12:47 |
| ralonsoh | yes, the extension is named correctly | 12:47 |
| ralonsoh | the patch is not failing, I'm testing it right now | 12:47 |
| ralonsoh | the extension is loaded correctly | 12:47 |
| ralonsoh | the parameter name is only "belongs_to_default_sg" | 12:47 |
| haleyb | neutron_lib.exceptions.ExtensionsNotFound: Extensions not found: ['security-groups-rules-belongs-to-default-sg'] | 12:48 |
| haleyb | from the unit test in your patch | 12:48 |
| ralonsoh | I'll check the UTs | 12:50 |
| ralonsoh | the Neutron API, running in a server, is not failing | 12:50 |
| haleyb | ping bcafarel, elvira, frickler, mlavalle, mtomaska, obondarev, slawek, tobias-urdin, ykarel, lajoskatona, jlibosva, averdagu, amotoki | 13:00 |
| haleyb | #startmeeting networking | 13:00 |
| opendevmeet | Meeting started Tue Oct 3 13:00:43 2023 UTC and is due to finish in 60 minutes. The chair is haleyb. Information about MeetBot at http://wiki.debian.org/MeetBot. | 13:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 13:00 |
| opendevmeet | The meeting name has been set to 'networking' | 13:00 |
| ralonsoh | ? I though the meeting was 1 hour later | 13:01 |
| ralonsoh | at 14UTC | 13:01 |
| ralonsoh | https://meetings.opendev.org/#Neutron_Team_Meeting | 13:01 |
| obondarev | +1 | 13:01 |
| haleyb | doh, did time move in europe? we don't move here for a month | 13:01 |
| ralonsoh | no, we didn't yet | 13:01 |
| ralonsoh | but the time is in UTC | 13:01 |
| lajoskatona | :-) | 13:02 |
| haleyb | oh, you're right, i got in early today and no coffee | 13:02 |
| haleyb | #endmeeting | 13:02 |
| opendevmeet | Meeting ended Tue Oct 3 13:02:20 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 13:02 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/networking/2023/networking.2023-10-03-13.00.html | 13:02 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/networking/2023/networking.2023-10-03-13.00.txt | 13:02 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/networking/2023/networking.2023-10-03-13.00.log.html | 13:02 |
| ralonsoh | see you later! | 13:02 |
| lajoskatona | I just checked and we have one more month for the time change | 13:02 |
| haleyb | i'm so excited to be ptl i guess :) | 13:02 |
| lajoskatona | life would be much easier without the change, but local governments had no time to do the homework and forget this whole change when I have to sleep thing (jetlag without the excitement of travelling) | 13:04 |
| haleyb | lajoskatona: our government talks about it every year, then they complain moving it will hurt school kids/farmers/bigfoot/global warming and nothing happens | 13:05 |
| lajoskatona | haleyb: better than here, there's just silence.... | 13:06 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: Missing SG rule default group extension in ``Ml2Plugin`` https://review.opendev.org/c/openstack/neutron/+/897211 | 13:54 |
| haleyb | ping bcafarel, elvira, frickler, mlavalle, mtomaska, obondarev, slawek, tobias-urdin, ykarel, lajoskatona, jlibosva, averdagu, amotoki | 14:00 |
| jlibosva | ko/ | 14:00 |
| lajoskatona | o/ | 14:00 |
| haleyb | #startmeeting networking | 14:00 |
| opendevmeet | Meeting started Tue Oct 3 14:00:25 2023 UTC and is due to finish in 60 minutes. The chair is haleyb. Information about MeetBot at http://wiki.debian.org/MeetBot. | 14:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 14:00 |
| opendevmeet | The meeting name has been set to 'networking' | 14:00 |
| obondarev | o/ | 14:00 |
| mtomaska | o/ | 14:00 |
| ralonsoh | hello | 14:00 |
| rubasov | o/ | 14:00 |
| haleyb | i got the right time now | 14:00 |
| haleyb | #topic announcements | 14:01 |
| ykarel | o/ | 14:02 |
| haleyb | we tagged what should be the final RC of neutron for Bobcat, https://review.opendev.org/c/openstack/releases/+/896971 | 14:02 |
| bcafarel | o/ | 14:02 |
| haleyb | i'm guessing that will merge soon as tomorrow is the expected release date | 14:02 |
| ralonsoh | we missed https://review.opendev.org/c/openstack/neutron/+/896803, but we couldn't change the RC hash | 14:03 |
| ralonsoh | so we'll need to propose it for the next bobcat release | 14:03 |
| slaweq | o/ | 14:03 |
| haleyb | ralonsoh: i thought you updated the review? | 14:04 |
| ralonsoh | yes but I needed to revert it | 14:04 |
| ralonsoh | we could only approve the RC version | 14:04 |
| ralonsoh | it was not possible to add this hash without creating a new RC | 14:05 |
| ralonsoh | and we were out of time | 14:05 |
| haleyb | ralonsoh: ah, i somehow never got the update | 14:05 |
| haleyb | we can just propose a new tag in a short while | 14:06 |
| ralonsoh | yes | 14:06 |
| haleyb | I would like to thank ralonsoh for all his work and guidance over the many cycles he was PTL! | 14:07 |
| slaweq | ralonsoh++ thx for all Your work as PTL :) | 14:08 |
| ralonsoh | thanks | 14:08 |
| lajoskatona | +1 | 14:08 |
| ykarel | +1 | 14:08 |
| mtomaska | +1 | 14:09 |
| haleyb | the etherpad for Caracal vPTG is up, please add your name if you are attending | 14:10 |
| haleyb | https://etherpad.opendev.org/p/oct2023-ptg-neutron | 14:10 |
| haleyb | https://openinfra-ptg.eventbrite.com/ is signup | 14:10 |
| haleyb | I will send out a doodle to vote on times people are available | 14:11 |
| lajoskatona | thanks, I missed the registration mail | 14:11 |
| haleyb | there is also a cycle highlights presentation this thursday, ralonsoh will give an update for neutron | 14:12 |
| ralonsoh | at 13:30UTC | 14:13 |
| ralonsoh | https://www.google.com/url?q=https://streamyard.com/ydfc48bmf2&sa=D&source=calendar&ust=1696751231543261&usg=AOvVaw11KOUAfAr1u2DAMbp5Lqcv | 14:13 |
| haleyb | openinfra.dev/live is the place if you want to watch i believe | 14:13 |
| haleyb | ah, that's a better link | 14:13 |
| ralonsoh | shorter link: https://streamyard.com/ydfc48bmf2 | 14:14 |
| slaweq | 13:30 UTC? On the webpage it says 1400 | 14:14 |
| ralonsoh | I have the invitation starting at 1330 | 14:15 |
| ralonsoh | maybe this is just for the people presenting | 14:15 |
| slaweq | ahh, but You are speaker and need to be earlier probably | 14:15 |
| ralonsoh | yes, my bad | 14:15 |
| slaweq | event will start 1400 UTC | 14:15 |
| slaweq | no problem, I just wanted to make sure :) | 14:15 |
| haleyb | any other announcements? | 14:17 |
| haleyb | #topic bugs | 14:17 |
| haleyb | last week's report is from lajoskatona | 14:17 |
| haleyb | https://lists.openstack.org/pipermail/openstack-discuss/2023-October/035254.html | 14:17 |
| haleyb | most bugs have owners and/or patches which is great, just a doc one is without | 14:18 |
| haleyb | https://bugs.launchpad.net/neutron/+bug/2037263 (low hanging fruit if submittor does not propose change) | 14:19 |
| lajoskatona | in comment the opener said they can push the fix to the doc | 14:19 |
| haleyb | lajoskatona: ack | 14:19 |
| haleyb | there were some high priority bugs | 14:20 |
| haleyb | #link https://bugs.launchpad.net/neutron/+bug/2037596 | 14:20 |
| haleyb | Description Failed to invoke the API interface to obtain the address group list | 14:20 |
| haleyb | #link https://review.opendev.org/c/openstack/neutron/+/896649 proposed | 14:20 |
| haleyb | i will review after meeting see it has another +2 already | 14:21 |
| haleyb | next one | 14:21 |
| haleyb | #link https://bugs.launchpad.net/neutron/+bug/2037717 | 14:21 |
| haleyb | [OVN] ``PortBindingChassisEvent`` event is not executing the conditions check | 14:21 |
| haleyb | #link https://review.opendev.org/c/openstack/neutron/+/896926 | 14:21 |
| haleyb | thanks for the test on that one ralonsoh | 14:22 |
| ralonsoh | yw | 14:22 |
| haleyb | there was also 3 ovsdbapp bugs filed, along with patches | 14:23 |
| haleyb | #link https://bugs.launchpad.net/ovsdbapp/+bug/2037536 | 14:23 |
| haleyb | #link https://bugs.launchpad.net/ovsdbapp/+bug/2037573 | 14:23 |
| haleyb | #link https://bugs.launchpad.net/ovsdbapp/+bug/2037652 | 14:24 |
| lajoskatona | yes all ovsdb bugs are addressed | 14:24 |
| haleyb | otherwiseguy seems to have already reviewed :) | 14:24 |
| * otherwiseguy jumps | 14:25 | |
| haleyb | sorry to scare you terry :) | 14:25 |
| otherwiseguy | haleyb: yeah, i still have a bit more review to do. I'll get to it today. | 14:26 |
| haleyb | next bug is a gate failure from last week that is still unsolved | 14:26 |
| haleyb | #link https://bugs.launchpad.net/neutron/+bug/2037239 | 14:26 |
| haleyb | neutron-tempest-plugin-openvswitch-* jobs randomly failing in gate | 14:26 |
| slaweq | I saw many failures in those scenario jobs this week :/ | 14:27 |
| ralonsoh | this is the most serious issue we have right now | 14:27 |
| lajoskatona | yes, many rechecks | 14:27 |
| ralonsoh | and is blocking the CI almost 100% of the times | 14:27 |
| slaweq | probably as a workaround temporary we can disable "l3_ha" in those jobs | 14:27 |
| slaweq | so routers will be always active on the L3 agent as it will be legacy routers | 14:27 |
| lajoskatona | last week I tried to find something in the logs but nothing which could be suspiciuo | 14:27 |
| haleyb | i will spend a little time today looking, i can't tell if the OVS never plugs the port, or if it's just a little late and causes the l3-agent to fail | 14:27 |
| slaweq | but we will lost coverage for L3ha then so I don't think it's good permanent solution | 14:27 |
| ykarel | i too digged there but couldn't trace what exactly causing it, but seems the thread processing those events are just stuck | 14:28 |
| ralonsoh | in the l3 agent? | 14:29 |
| ralonsoh | or the API? | 14:29 |
| ykarel | yes l3-agent basically keepalived-state-change | 14:29 |
| ralonsoh | pffff ok | 14:29 |
| haleyb | slaweq: is it easy enough to disable l3_ha in the yaml? as opposed to marking failed tests as unstable? | 14:30 |
| haleyb | ykarel: so keepalived-state-change just sat there? | 14:30 |
| ykarel | locally it reproduces in many runs in the vm, but in CI it's reproduces quite frequently | 14:30 |
| slaweq | https://github.com/openstack/neutron-tempest-plugin/blob/master/zuul.d/master_jobs.yaml#L258 | 14:31 |
| slaweq | just change this to false and routers will be created as non-ha | 14:31 |
| slaweq | so L3 agent will go through different path configuring them | 14:31 |
| ykarel | haleyb, so it stuck differently sometimes in handle_initial_state, sometimes in ip_monitor thread around the calls to privileged methods | 14:32 |
| slaweq | maybe that will workaround the issue and if so, it will also be clear pointer where the issue may be :) | 14:32 |
| haleyb | slaweq: ack, i can send a change out after meeting | 14:32 |
| haleyb | ykarel: can you add any notes to the bug? | 14:33 |
| ykarel | haleyb, ack will do | 14:33 |
| haleyb | ykarel: thanks | 14:34 |
| haleyb | any other bugs to talk about? | 14:34 |
| haleyb | #topic specs | 14:34 |
| haleyb | #link https://review.opendev.org/c/openstack/neutron-specs/+/891204 is the only spec in review at the moment, looks like it needs an update based on reviews | 14:35 |
| haleyb | ralonsoh: was there anything new from the drivers meeting last week? i was unable to attend and didn't look at notes yet | 14:36 |
| ralonsoh | yes, I have one question | 14:36 |
| ralonsoh | one sec | 14:36 |
| ralonsoh | about https://bugs.launchpad.net/neutron/+bug/2037294 | 14:37 |
| ralonsoh | I presented this RFE in behalf of lucasagomes | 14:37 |
| ralonsoh | but I thought a change in the oVN L3 scheduler was needed | 14:37 |
| lucasagomes | o/ | 14:37 |
| ralonsoh | lucasagomes, confirmed that the change is much more smaller and the scope | 14:37 |
| ralonsoh | so I would propose not to request a spec for this rfe | 14:38 |
| ralonsoh | just a patch | 14:38 |
| lucasagomes | yeah, haleyb ralonsoh no L3 changes needed. The RFE is only about external ports | 14:38 |
| lucasagomes | GW ports will continue to be scheduled as they are today (using the enable-chassis-as-gw flag) | 14:38 |
| ralonsoh | so if you agree, now the scope is smaller, we can just accept a patch (with the needed documentation and reno) | 14:39 |
| slaweq | ++ | 14:39 |
| haleyb | lucasagomes: ack, will remove the rfe-approved tag then and look for the change | 14:39 |
| lajoskatona | ack | 14:39 |
| haleyb | or guess can leave since it is a slight change | 14:39 |
| lucasagomes | thanks folks! | 14:39 |
| haleyb | ralonsoh: regarding specs, we can merge this now - https://review.opendev.org/c/openstack/neutron-specs/+/891204 | 14:40 |
| ralonsoh | I would need to read this spec again | 14:40 |
| haleyb | it's just creating the folder | 14:40 |
| ralonsoh | but there are some questions there by dalvarez | 14:41 |
| haleyb | ah, bad copy/paste | 14:41 |
| haleyb | #link https://review.opendev.org/c/openstack/neutron-specs/+/891342 | 14:41 |
| ralonsoh | ah yes, sure | 14:41 |
| ralonsoh | done | 14:41 |
| opendevreview | yatin proposed openstack/neutron-tempest-plugin master: [DNM] Test l3_ha=False https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/897233 | 14:41 |
| haleyb | moving on | 14:42 |
| haleyb | #topic community_goals | 14:42 |
| haleyb | there are still just two open goals | 14:43 |
| haleyb | #link https://bugs.launchpad.net/neutron/+bug/2026182 | 14:43 |
| haleyb | S-RBAC, https://review.opendev.org/c/openstack/neutron/+/886724 | 14:43 |
| slaweq | for that I still have patch https://review.opendev.org/c/openstack/neutron/+/886724 | 14:43 |
| slaweq | I will try to rebase it and address commnets there in next days | 14:44 |
| slaweq | so we hopefully be able to merge it finally | 14:44 |
| haleyb | slaweq: ack thanks | 14:44 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: WIP - Create a keepalived-state-change script in bash https://review.opendev.org/c/openstack/neutron/+/836140 | 14:44 |
| haleyb | second is https://bugs.launchpad.net/neutron/+bug/1999774 | 14:45 |
| haleyb | neutronclient deprecation | 14:45 |
| haleyb | lajoskatona: is there anything that needs review there? | 14:45 |
| lajoskatona | the usual etherpad: https://etherpad.opendev.org/p/python-neutronclient_deprecation | 14:45 |
| lajoskatona | but no fresh reviews, I still have no time for this topic | 14:45 |
| haleyb | lajoskatona: ack, i'll update the list in the meeting notes as some have merged | 14:46 |
| haleyb | #topic on_demand | 14:47 |
| lajoskatona | haleyb: thanks | 14:47 |
| haleyb | is there anything else people want to discuss? | 14:47 |
| ralonsoh | no thanks | 14:48 |
| haleyb | oh, and i forgot during 'bugs' that slaweq is the deputy this week, i am next week | 14:48 |
| slaweq | yeap, I'm on it :) | 14:48 |
| slaweq | but thx for reminder | 14:48 |
| haleyb | and CI meeting in :11 minutes for those interested | 14:49 |
| ykarel | Also CI meeting will be in next 11 minutes, IRC this time | 14:49 |
| lajoskatona | +1 | 14:49 |
| haleyb | ok, thanks everyone for attending, have a good week! | 14:49 |
| haleyb | #endmeeting | 14:49 |
| opendevmeet | Meeting ended Tue Oct 3 14:49:48 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 14:49 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/networking/2023/networking.2023-10-03-14.00.html | 14:49 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/networking/2023/networking.2023-10-03-14.00.txt | 14:49 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/networking/2023/networking.2023-10-03-14.00.log.html | 14:49 |
| lajoskatona | o/ | 14:49 |
| opendevreview | Merged openstack/neutron-specs master: Spec folder for 2024.1 cycle https://review.opendev.org/c/openstack/neutron-specs/+/891342 | 14:58 |
| ykarel | #startmeeting neutron_ci | 15:00 |
| opendevmeet | Meeting started Tue Oct 3 15:00:12 2023 UTC and is due to finish in 60 minutes. The chair is ykarel. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| opendevmeet | The meeting name has been set to 'neutron_ci' | 15:00 |
| ralonsoh | hello | 15:00 |
| lajoskatona | o/ | 15:00 |
| ykarel | ping bcafarel, lajoskatona, mlavalle, mtomaska, ralonsoh, ykarel, jlibosva, elvira | 15:00 |
| ykarel | Grafana dashboard: https://grafana.opendev.org/d/f913631585/neutron-failure-rate?orgId=1 | 15:00 |
| ykarel | Please open now :) | 15:00 |
| haleyb | o/ | 15:00 |
| slaweq | o/ | 15:00 |
| bcafarel | o/ | 15:01 |
| ykarel | Ok let's start with the topics | 15:01 |
| ykarel | #topic Actions from previous meetings | 15:01 |
| ykarel | ralonsoh to check failure with ha functional test | 15:01 |
| ralonsoh | I opened a bug but I would need to find it | 15:02 |
| ralonsoh | but I didn't push any patch yet | 15:02 |
| ykarel | Thanks ralonsoh for checking, yes can share the bug later | 15:02 |
| ykarel | lajoskatona to check consistency with bgpvpn related to https://zuul.openstack.org/build/8a624b4d29ea44589c9c83b0ec1da446 | 15:02 |
| lajoskatona | yes I pushed a dnm patch an it passed, and by the logs it looks like the ssh timeout issue we have in other jobs | 15:03 |
| ykarel | yes looks a random one https://zuul.openstack.org/builds?job_name=neutron-tempest-plugin-bgpvpn-bagpipe&result=TIMED_OUT&skip=0 | 15:03 |
| ykarel | and it happend again in weekly run | 15:04 |
| lajoskatona | yes, this week it was bgpvpn, and as the 2 run together with tempest, I suppose the same pattern | 15:04 |
| ykarel | overall jobs looks healthy https://zuul.openstack.org/builds?job_name=neutron-tempest-plugin-bgpvpn-bagpipe&skip=0 | 15:05 |
| lajoskatona | yes, it is not frequent | 15:06 |
| ykarel | Ok thanks lajoskatona, so we can keep monitoring it | 15:06 |
| ykarel | ykarel to check https://bugs.launchpad.net/neutron/+bug/2036603 | 15:06 |
| ykarel | This was on me, but i couldn't check so will check it this week | 15:07 |
| ykarel | #action ykarel to check https://bugs.launchpad.net/neutron/+bug/2036603 | 15:07 |
| ykarel | #topic Stable branches | 15:07 |
| ykarel | bcafarel anything to share for stable branches? | 15:07 |
| ykarel | atleast stable/2023.2 is impacted with l3 router issue | 15:08 |
| bcafarel | indeed | 15:09 |
| ykarel | rest all branches looks good | 15:10 |
| ykarel | #topic Stadium projects | 15:10 |
| bcafarel | I am a bit behind on the rest but older branches look good as far I could check | 15:10 |
| ykarel | thx bcafarel | 15:10 |
| ykarel | it was again bgpvpn job timeout https://zuul.openstack.org/buildset/5f604ee2caaf4d5883496d63087aa0dc | 15:10 |
| ykarel | which we already discussed | 15:11 |
| lajoskatona | except the issue in bagpipe/bgpvpn tempest job nothing else | 15:11 |
| ykarel | thx | 15:11 |
| lajoskatona | I pusehd a few patches to updte the weekly jobs to run with py311 | 15:11 |
| ykarel | #topic Grafana | 15:11 |
| lajoskatona | and update small things in zull.yaml | 15:11 |
| ykarel | #undo | 15:11 |
| opendevmeet | Removing item from minutes: #topic Grafana | 15:11 |
| lajoskatona | this is the topic for them: https://review.opendev.org/q/topic:py311_neutron | 15:12 |
| ykarel | #info lajoskatona pushed patches to stadium to include py311 jobs in weekly pipeline | 15:12 |
| ykarel | #link https://review.opendev.org/q/topic:py311_neutron | 15:12 |
| lajoskatona | as I see I have to go back to the fwaas one (the last week I had no time to care them) | 15:12 |
| lajoskatona | thats it from me | 15:12 |
| ykarel | thx lajoskatona | 15:12 |
| ykarel | #topic Grafana | 15:13 |
| ykarel | https://grafana.opendev.org/d/f913631585/neutron-failure-rate?orgId=1 | 15:13 |
| ykarel | So we can see some significant failures for that l3 agent tempest failures in scenario jobs | 15:14 |
| slaweq | IMHO grafana looks good, except those neutron-tempest-plugin jobs which are failing A LOT | 15:14 |
| ykarel | yeap right | 15:15 |
| ykarel | ok moving to next topic | 15:15 |
| ykarel | #topic Rechecks | 15:15 |
| ykarel | stat looks good, but there was not much activity this week | 15:16 |
| ykarel | and just 1 bare recheck, so that's good too | 15:16 |
| slaweq | yeah, I guess it's also because of those broken neutrn-tempest-plugin jobs as not many patches are really merged recently | 15:16 |
| slaweq | and my script is checking only patches already merged | 15:16 |
| ykarel | yes right | 15:17 |
| ykarel | #topic fullstack/functional | 15:17 |
| ykarel | AssertionError: Text not found in file /tmp/tmp_kg_b71e/tmpgbxs4_ju/log_file: "Initial status of router". | 15:17 |
| ykarel | https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_fe2/periodic/opendev.org/openstack/neutron/master/neutron-functional-with-sqlalchemy-master/fe21035/testr_results.html | 15:17 |
| ykarel | https://81a66b9ba1b7a73d7079-cf2e6d1d128a778849f5c48f17b8a34a.ssl.cf1.rackcdn.com/896926/2/check/neutron-functional-with-uwsgi/ea8b000/testr_results.html | 15:17 |
| ykarel | we discussed this few weeks back and ralonsoh said it's something know issue and happens rarely | 15:18 |
| ralonsoh | yes and this could be also the case of the tempest problems | 15:18 |
| ykarel | but it's seems to be happening quite often now | 15:18 |
| ykarel | seen 6 failures in a week | 15:18 |
| slaweq | maybe it's somehow related to the issue with scenario jobs? | 15:18 |
| ralonsoh | yes | 15:19 |
| ralonsoh | it could be | 15:19 |
| ykarel | slaweq, ralonsoh yes symptoms looks quite similar | 15:19 |
| ralonsoh | I don't know if to refactor the IP monitor | 15:19 |
| ralonsoh | or to continue the implementation of the bash script | 15:19 |
| ralonsoh | one sec | 15:19 |
| ralonsoh | --> https://review.opendev.org/c/openstack/neutron/+/836140?usp=dashboard | 15:19 |
| ralonsoh | but somehow the IP monitor is not working fine now | 15:20 |
| slaweq | maybe we can use something like https://raymii.org/s/tutorials/Keepalived_notify_script_execute_action_on_failover.html instead ? | 15:21 |
| slaweq | I didn't read it fully, just google about somethig like that now | 15:21 |
| slaweq | AFAIR ip_monitor is there only to monitor IP address to see if keepalived did failover | 15:21 |
| ralonsoh | yes, exactly | 15:21 |
| slaweq | maybe instead keepalived can notify neutron-l3-agent by itself | 15:21 |
| slaweq | I would like to help with it but I don't know if I will have time in next few weeks | 15:22 |
| slaweq | so I will not volunteer for it, at least not for now | 15:22 |
| ykarel | ralonsoh, and what's left in move to bash script? | 15:23 |
| ykarel | just if we can see if with it these current issue reproduces | 15:24 |
| ralonsoh | this change needs to handle the migration to this new script, in order to stop and replace the running keepalived-state-change scripts | 15:24 |
| ralonsoh | what slaweq commented in the review | 15:24 |
| ykarel | ohkk and without those migration changes in place, it can be validated with current issues, right? | 15:26 |
| ralonsoh | yes, the current script should work fine now | 15:26 |
| ralonsoh | I've pushed a new PS 30 mins ago | 15:26 |
| ralonsoh | I'll change the zuul definitions to check multiple times the ovs jobs | 15:27 |
| ykarel | okk thanks, just noticed the update | 15:27 |
| ykarel | Also similar failures seen in other tests | 15:27 |
| ykarel | test_dvr_lifecycle_ha_with_snat_with_fips_with_cent_fips_no_gw and test_dvr_ha_router_interface_mtu_update | 15:27 |
| ykarel | https://53d9a8858ad69ec7c4a3-c555fae2d8c498523cc4b2c363541725.ssl.cf5.rackcdn.com/periodic/opendev.org/openstack/neutron/master/neutron-functional/6b7fe58/testr_results.html | 15:28 |
| ykarel | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_1ad/periodic/opendev.org/openstack/neutron/master/neutron-functional/1ad8b06/testr_results.html | 15:28 |
| opendevreview | Anton Vazhnetsov proposed openstack/ovsdbapp master: nb: add 'nexthop' argument to 'lr_route_del' https://review.opendev.org/c/openstack/ovsdbapp/+/896645 | 15:28 |
| ykarel | #topic Tempest/Scenario | 15:28 |
| ykarel | This we already discussed | 15:28 |
| ykarel | master/stable2023.2 Tests in linuxbridge/openvswitch scenario jobs fails randomly bug https://bugs.launchpad.net/neutron/+bug/2037239 since 22nd Septmeber with | 15:28 |
| ykarel | fails like Details: Router 411b39c1-b9fd-4fa1-a28b-d7976858a4d4 is not active on any of the L3 agents | 15:29 |
| ykarel | I tried to reproduce with linuxbridge setup locally, reproduces rarely and saw below observations with adding some debug statements, in CI reproducibility is quite high | 15:29 |
| ykarel | Fails differently at different threads within keepalived-state-change process | 15:29 |
| ykarel | handle_initial_state stuck, nothing written to state file, ha_state is set to "unknown" | 15:29 |
| ykarel | Timeout reading the initial status of router, write state backup in state file, and state remains in backup | 15:29 |
| ykarel | ip_monitor stuck, leading to not start of read_ip_updates stuck | 15:29 |
| ykarel | but i couldn't find any recent change that could change the behavior in master/stable2023.2 | 15:30 |
| lajoskatona | even dependency thins seem to be the same, like no keepalived change or such | 15:31 |
| ralonsoh | could be a change in the eventlet library and now something is being blocked in the ip_monitor | 15:31 |
| ralonsoh | to be honest (and I implemented it) I don't like how it works | 15:32 |
| ykarel | yes right, so something should be with openstack, as external packages should be same in some stable branches as running jammy | 15:32 |
| ykarel | ralonsoh, i check eventlet was not bumped from quite few months | 15:32 |
| ykarel | update 6+ months back | 15:32 |
| ykarel | haleyb btw i had pushed https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/897233 with le_ha=False on top of your 2023.2 jobs patch | 15:35 |
| lajoskatona | +1 | 15:35 |
| lajoskatona | for short term | 15:35 |
| haleyb | ykarel: ack, will review and see if it helps | 15:35 |
| ykarel | i just pushed test patch, but we could change and take it for short term if it works fine | 15:35 |
| ykarel | thx haleyb | 15:36 |
| ykarel | #topic Periodic | 15:36 |
| ykarel | periodic also had same tempest failures in linuxbridge job, and test failures in functional jobs | 15:36 |
| ykarel | which we already discussed | 15:36 |
| ykarel | #topic On Demand | 15:37 |
| ykarel | anything else to raise here? | 15:37 |
| ralonsoh | no | 15:37 |
| lajoskatona | nothing from me | 15:38 |
| slaweq | nope | 15:38 |
| ykarel | thx everyone | 15:38 |
| ykarel | #endmeeting | 15:38 |
| opendevmeet | Meeting ended Tue Oct 3 15:38:52 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:38 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/neutron_ci/2023/neutron_ci.2023-10-03-15.00.html | 15:38 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/neutron_ci/2023/neutron_ci.2023-10-03-15.00.txt | 15:38 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/neutron_ci/2023/neutron_ci.2023-10-03-15.00.log.html | 15:38 |
| slaweq | thx | 15:38 |
| slaweq | o/ | 15:38 |
| ralonsoh | bye | 15:38 |
| lajoskatona | o/ | 15:39 |
| lajoskatona | otherwiseguy: Hi, I have a patch: https://review.opendev.org/c/openstack/ovsdbapp/+/890328 , it is for tap-as-a-service to have erspan/gre mirroring support with OVN | 15:52 |
| * otherwiseguy waves | 15:53 | |
| otherwiseguy | lajoskatona: it's definitely on my list to look at today/tomorrow. trying to get a bunch of things done before PTO Thurs-Mon. | 15:53 |
| lajoskatona | otherwiseguy: would be so kind to check it if I am on the right track with it? if I understand well technically this code could be in taas repo also, so not sure if we want to have more stadium things in ovsdbapp, or keep it as a clear lib | 15:53 |
| lajoskatona | otherwiseguy: thanks, it is not so urgent (this will be the consumer for it: https://review.opendev.org/c/openstack/tap-as-a-service/+/893509 and I just went back to it to remove the wip tag) | 15:54 |
| otherwiseguy | lajoskatona: For ovsdbapp, if it's basic operations on OVS/OVN objects, seems like a good fit. If I could go back in time, the ovsdbapp built-in stuff would essentially only be "things that would be in ovs-vsctl/ovn-nbctl/ovn-sbctl, etc." as a reference architecture, then encourage people to actually write more highly optimized versions in their individual projects where it made sense. | 15:56 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: Remove unused ``get_security_groups_count`` method https://review.opendev.org/c/openstack/neutron/+/897243 | 15:57 |
| opendevreview | Jakub Libosvar proposed openstack/neutron master: ovn-metadata: Refactor events https://review.opendev.org/c/openstack/neutron/+/896163 | 16:03 |
| opendevreview | Roberto Acosta proposed openstack/neutron-specs master: Add spec for coexistence support for OVN Interconnect RFE https://review.opendev.org/c/openstack/neutron-specs/+/891204 | 16:03 |
| lajoskatona | otherwiseguy: thanks, what is strange for me that there's a lot of duplication in neutron that is also in ovsdbapp | 16:06 |
| otherwiseguy | lajoskatona: Yeah. We moved some stuff over to the ovsdbapp versions of things. And then it just seemed like there wasn't a huge benefit to it. Potential behavior changes, etc. The versions in ovsdbapp (at least the ones I added) tried to mimic exactly the interface ovs the ctl commands, and what the C code in those utilities executed. It isn't always the most efficient way to do something (there maybe be name lookups where the neutron | 16:09 |
| otherwiseguy | code may already have a reference to the row, etc.). | 16:09 |
| otherwiseguy | lajoskatona: so in the end, I just decided not to spend the time to get neutron to use the ovsdbapp versions. And it really does make more sense to use versions specific to your environment. | 16:11 |
| otherwiseguy | lajoskatona: one issue I see a lot in the neutron code is "Do a local read of a bunch of rows" which is basically its own transaction, then "take a bunch of actions based on those rows" in another transaction. This isn't really very safe. | 16:12 |
| otherwiseguy | lajoskatona: really, the reads and writes should be in their own Command object so they are executed in the txn together. | 16:13 |
| otherwiseguy | lajoskatona: 9-years-ago me didn't know as much about OVSDB/python-ovs as today-me. I wish I could go back and tell him to do some things differently. ;) | 16:15 |
| ralonsoh | "ovsdbapp2, the return" | 16:15 |
| otherwiseguy | ralonsoh: indeed. Also, it was originally written for ovs agent. It's a bit different having one copy of the Open_vSwitch DB in memory on a compute node than have 32 copies of the OVN NB/SB DBs running because of API workers... | 16:17 |
| otherwiseguy | So maybe using python-ovs IDL was one of those choices I would reconsider (though ovsdb-server having to do more than just send out updates to clients would also probably be an issue, it doesn't really have a query optimizer and is single threaded, so...) | 16:17 |
| ralonsoh | yeah, that part is loading the DB a lot | 16:17 |
| lajoskatona | otherwiseguy, ralonsoh: :-) | 16:25 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: WIP == SG rules default SG https://review.opendev.org/c/openstack/neutron/+/896222 | 16:26 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: WIP - Create a keepalived-state-change script in bash https://review.opendev.org/c/openstack/neutron/+/836140 | 16:30 |
| ralonsoh | ykarel, ^ ovs jobs were passing, I've updated the zuul definition to run more | 16:31 |
| otherwiseguy | With ovsdbapp usage, the general rule is "the more you can shove in a single Command, the more efficient/safe things will be. You won't have to do unnecessary lookups by name, you're free to access to the actual row objects in the Command, you'll have a consistent view of the DB data because it is all in the same txn, etc. | 16:32 |
| otherwiseguy | Basically the opposite of what ovn_client.py does. ;) | 16:33 |
| grishatsu | Hey is there like a simple onboarding guide for new contributors? I opened an issue and already have a solution but I'm unfamiliar with the brz and neutron CI at all | 18:08 |
| mnaser | grishatsu: o/ https://docs.openstack.org/contributors/common/git.html and https://docs.openstack.org/contributors/common/setup-gerrit.html should get you started | 18:52 |
| mnaser | you can go up and see the start but if you've filed an issue you probably already did a bunch of it ;) | 18:52 |
| grishatsu | tnx :) | 19:02 |
| opendevreview | Brian Haley proposed openstack/neutron master: Remove a print statement from the iptables unit test https://review.opendev.org/c/openstack/neutron/+/897251 | 19:21 |
| opendevreview | Jakub Libosvar proposed openstack/neutron master: ovn: Spawn metadata resources on additional chassis event https://review.opendev.org/c/openstack/neutron/+/895402 | 19:25 |
| opendevreview | Brian Haley proposed openstack/neutron master: TestSegmentHostMappingNoStore class is missing config https://review.opendev.org/c/openstack/neutron/+/897252 | 19:27 |
| opendevreview | Terry Wilson proposed openstack/ovsdbapp master: Handle events with conditions and match_fn https://review.opendev.org/c/openstack/ovsdbapp/+/897090 | 19:43 |
| opendevreview | Jakub Libosvar proposed openstack/neutron master: ovn: Spawn metadata resources on additional chassis event https://review.opendev.org/c/openstack/neutron/+/895402 | 21:39 |
| opendevreview | Jakub Libosvar proposed openstack/neutron master: ovn-metadata: Refactor events https://review.opendev.org/c/openstack/neutron/+/896163 | 21:41 |
| opendevreview | Jakub Libosvar proposed openstack/neutron master: ovn: Spawn metadata resources on additional chassis event https://review.opendev.org/c/openstack/neutron/+/895402 | 21:41 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!