| opendevreview | Maor Blaustein proposed openstack/neutron master: Fix LRP.external_ids router name prefix https://review.opendev.org/c/openstack/neutron/+/940445 | 05:57 |
|---|---|---|
| opendevreview | Maor Blaustein proposed openstack/neutron master: Fix LRP.external_ids router name prefix https://review.opendev.org/c/openstack/neutron/+/940445 | 07:37 |
| opendevreview | Vladimir Prokofev proposed openstack/neutron-specs master: Add spec for random-fully per-FIP feature RFE. https://review.opendev.org/c/openstack/neutron-specs/+/932650 | 09:53 |
| opendevreview | Konstantin Eremin proposed openstack/neutron stable/2024.2: OVS: Set log level to INFO for unconfigured ofport cases https://review.opendev.org/c/openstack/neutron/+/940949 | 10:03 |
| opendevreview | Konstantin Eremin proposed openstack/neutron stable/2024.1: OVS: Set log level to INFO for unconfigured ofport cases https://review.opendev.org/c/openstack/neutron/+/940950 | 10:06 |
| opendevreview | Konstantin Eremin proposed openstack/neutron stable/2023.2: OVS: Set log level to INFO for unconfigured ofport cases https://review.opendev.org/c/openstack/neutron/+/940951 | 10:07 |
| opendevreview | Renjing Xiao proposed x/whitebox-neutron-tempest-plugin master: Add new nested snat test to fetch ovn config dynamically https://review.opendev.org/c/x/whitebox-neutron-tempest-plugin/+/940908 | 10:38 |
| opendevreview | Renjing Xiao proposed x/whitebox-neutron-tempest-plugin master: Add new nested snat test to fetch ovn config dynamically https://review.opendev.org/c/x/whitebox-neutron-tempest-plugin/+/940908 | 11:01 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: [OVN][FT] Use MySQL backend for ``Test*Maintenance`` classes https://review.opendev.org/c/openstack/neutron/+/940959 | 11:34 |
| sahid | lajoskatona: "all works well" on devstack with os-ken patches | 12:20 |
| sahid | I had to redeploy devstack in totally I don't know if you where like me but mine one a bit old | 12:21 |
| sahid | perhaps you have to fetch last change from devstack, clean it, remove repo or reclone and stack again | 12:21 |
| sahid | from the CI point of view, the job with os-ken-master, only 3 tests are failing | 12:25 |
| sahid | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_053/937765/53/check/neutron-ovs-tempest-with-os-ken-master/0538d30/testr_results.html | 12:25 |
| sahid | I have triggered it a second time, just to check | 12:25 |
| opendevreview | Maor Blaustein proposed openstack/neutron master: Fix LRP.external_ids router name prefix https://review.opendev.org/c/openstack/neutron/+/940445 | 12:32 |
| opendevreview | Maor Blaustein proposed openstack/neutron master: Fix LRP.external_ids router name prefix https://review.opendev.org/c/openstack/neutron/+/940445 | 12:41 |
| lajoskatona | sahid: cool, I try to check it this afternoon | 12:59 |
| priteau | Hello ralonsoh. I have just seen this issue happen in CI again: https://bugs.launchpad.net/neutron/+bug/2093258. We are using Kolla images built from neutron master, but since your fix was a revert in neutron-lib, we are not running it yet because we use the latest pip release: 3.18.0. I know you just released neutron-lib last week, but would you consider making another | 14:19 |
| priteau | release soon so we can verify that the revert does indeed fix the issue? | 14:19 |
| ralonsoh | priteau, I don't think we have any other patch apart from this one | 14:22 |
| ralonsoh | but I think it could worth a minor release | 14:22 |
| priteau | Maybe haleyb can give his input too. | 14:23 |
| ralonsoh | haleyb, https://review.opendev.org/c/openstack/releases/+/940972 | 14:26 |
| ralonsoh | as PTL, must approve it | 14:26 |
| priteau | Thanks | 14:27 |
| haleyb | yes, either me or the release liaison (slaweq), i'm looking now | 14:38 |
| opendevreview | Michel Nederlof proposed openstack/ovn-bgp-agent master: Fix running sync method for every external_ids update. https://review.opendev.org/c/openstack/ovn-bgp-agent/+/940129 | 14:41 |
| otherwiseguy | ralonsoh: on the hash ring stuff, on my system all of the extra ovn_hash_ring nodes 'updated_at' columns are being refreshed, not just the ones that are still "up". | 14:43 |
| ralonsoh | otherwiseguy, yes, that is the goal | 14:44 |
| ralonsoh | we assume al hash ring workers of the same node are running at the same time | 14:44 |
| otherwiseguy | but I have 2 workers and 10 hash ring entries, all saying that they are up and refreshed? | 14:45 |
| ralonsoh | otherwiseguy, the use case you commented (change the config once started with an error) is a non recoverable error | 14:45 |
| otherwiseguy | it happens w/o changing the config. | 14:45 |
| ralonsoh | otherwiseguy, how that happened? | 14:45 |
| otherwiseguy | kill -9 $worker_pid | 14:45 |
| otherwiseguy | (there is a script in the comments that does it) | 14:45 |
| ralonsoh | but wsgi will restart this worker | 14:46 |
| otherwiseguy | yes. and leave the ovn_hash_ring entry in there. and it keeps getting updated_at set. | 14:46 |
| otherwiseguy | so now we have more hash ring nodes than workers. | 14:46 |
| ralonsoh | no no, the hash ring register should be reused by the restarted worker | 14:46 |
| otherwiseguy | it is not | 14:47 |
| ralonsoh | that's a problem I think I solved | 14:47 |
| ralonsoh | let me check | 14:47 |
| otherwiseguy | this is what I have been trying to say | 14:47 |
| ralonsoh | I was not understanding your point | 14:47 |
| ralonsoh | is a valid issue, of course | 14:47 |
| ralonsoh | and I think I fixed that, let me check | 14:47 |
| ralonsoh | otherwiseguy, I found it: https://review.opendev.org/c/openstack/neutron/+/940140/3/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py#372 | 14:49 |
| ralonsoh | that's the problem of the many POCs I've been implemented | 14:49 |
| ralonsoh | I had this somewhere but not in the correct place | 14:49 |
| ralonsoh | let me push a patch under https://review.opendev.org/c/openstack/neutron/+/940256/ | 14:50 |
| otherwiseguy | as far as the "non-recoverable error" that was just a simple example I could easily come up with for "one worker dies and keeps getting restarted, while the others are fine" so assuming that they are all running/up would be incorrect. I'm sure there are others. The world is a complex place. :) | 14:53 |
| otherwiseguy | In any case, I'll try to make sure I fully understand the entire patchset before bothering with too many design comments. | 14:53 |
| otherwiseguy | Just saw a +2 last night and was seeing this particular bug and so stayed up to make a script to make it easy to show the issue. | 14:54 |
| ralonsoh | otherwiseguy, let me first push something to fix your comment | 14:54 |
| otherwiseguy | +1 | 14:54 |
| opendevreview | Lajos Katona proposed openstack/tap-as-a-service master: GRE/ERSPAN mirroring for taas https://review.opendev.org/c/openstack/tap-as-a-service/+/885357 | 14:55 |
| opendevreview | Lajos Katona proposed openstack/tap-as-a-service master: Tap Mirror OVS driver https://review.opendev.org/c/openstack/tap-as-a-service/+/893086 | 14:55 |
| opendevreview | Lajos Katona proposed openstack/tap-as-a-service master: TAAS tap-mirror OVN driver https://review.opendev.org/c/openstack/tap-as-a-service/+/893509 | 14:55 |
| opendevreview | Lajos Katona proposed openstack/tap-as-a-service master: CLI for Tap Mirrors https://review.opendev.org/c/openstack/tap-as-a-service/+/886085 | 14:55 |
| opendevreview | Lajos Katona proposed openstack/tap-as-a-service master: Documentation for tap-mirrors https://review.opendev.org/c/openstack/tap-as-a-service/+/907220 | 14:55 |
| opendevreview | Merged openstack/ovn-octavia-provider stable/2024.1: Fix pep8 with pylint 3.3.0 https://review.opendev.org/c/openstack/ovn-octavia-provider/+/940867 | 15:02 |
| opendevreview | Merged openstack/ovn-octavia-provider stable/2023.2: Fix pep8 with pylint 3.3.0 https://review.opendev.org/c/openstack/ovn-octavia-provider/+/940868 | 15:04 |
| otherwiseguy | ralonsoh: re: non-recoverable error--my main problem is that assuming all nodes are up on the host means that a failure that takes out one worker (broken config example where you don't restart neutron-api, or maximum db connections reached, but existing workers are fine because they have connections, file descriptors, whatever) means that instead of it just affecting that one worker that is down, it affects everything that is | 15:24 |
| otherwiseguy | still up because the number of "up" hash ring nodes is different than the number of "up" workers, so events end up randomly not processed. | 15:24 |
| otherwiseguy | absolutely, it's an edge case. and working most of the time > not working at all. Given how many edge cases I know I miss, I worry when I can find them early. :) | 15:29 |
| opendevreview | Lajos Katona proposed openstack/tap-as-a-service master: GRE/ERSPAN mirroring for taas https://review.opendev.org/c/openstack/tap-as-a-service/+/885357 | 15:29 |
| opendevreview | Lajos Katona proposed openstack/tap-as-a-service master: Tap Mirror OVS driver https://review.opendev.org/c/openstack/tap-as-a-service/+/893086 | 15:29 |
| opendevreview | Lajos Katona proposed openstack/tap-as-a-service master: TAAS tap-mirror OVN driver https://review.opendev.org/c/openstack/tap-as-a-service/+/893509 | 15:29 |
| opendevreview | Lajos Katona proposed openstack/tap-as-a-service master: CLI for Tap Mirrors https://review.opendev.org/c/openstack/tap-as-a-service/+/886085 | 15:29 |
| opendevreview | Lajos Katona proposed openstack/tap-as-a-service master: Documentation for tap-mirrors https://review.opendev.org/c/openstack/tap-as-a-service/+/907220 | 15:29 |
| ralonsoh | otherwiseguy, but that already happens, for example, if another controller dies | 15:30 |
| ralonsoh | for a period of time, the hash ring leftovers are in the DB until they rot | 15:30 |
| otherwiseguy | Yes, but nothing would be refreshing their 'updated_at' | 15:31 |
| ralonsoh | yeah... | 15:31 |
| ralonsoh | I think this is a corner case, IMO | 15:32 |
| ralonsoh | if we have a zombie worker, the API in general (in this node) should not be working | 15:32 |
| otherwiseguy | It is definitely a corner case. But it's one the previous design handled (at least with eventlet) | 15:32 |
| ralonsoh | is it possible to have some kind of event from the OVN DB? | 15:33 |
| ralonsoh | we can update something (I don't know what) and all workers in all nodes will receive this event | 15:33 |
| ralonsoh | that will update the local hash ring manager | 15:33 |
| ralonsoh | that will be faster than having a refresh time (15 secs is now) | 15:34 |
| ralonsoh | then we need to handle the case of having leftovers (if the worker dies) | 15:34 |
| ralonsoh | can we monitor the DB connections? | 15:35 |
| otherwiseguy | We have to be careful w/ sending extra ovsdb events--it can get kind of messy since they get broadcast everywhere and can trigger work, etc. | 15:35 |
| otherwiseguy | like when we try using them for metadata agent API keepalives, etc. | 15:36 |
| ralonsoh | yes but pubsub is ideal for this job | 15:36 |
| otherwiseguy | and basically just disable that feature downstream with a huge timeout | 15:36 |
| ralonsoh | right | 15:36 |
| otherwiseguy | I'll spend some time while looking at the patchset today to look at what's going on with the API and that initial blockage on the hash ring periodic as well, just because I want to really understand what's happening. | 15:38 |
| otherwiseguy | I feel like I'm throwing darts in the dark on some of this. | 15:38 |
| ralonsoh | if we can figure out why the periodic thread is now fired if the API is idle, that will be perfect | 15:38 |
| opendevreview | Lajos Katona proposed openstack/tap-as-a-service master: CLI for Tap Mirrors https://review.opendev.org/c/openstack/tap-as-a-service/+/940979 | 15:47 |
| opendevreview | Takashi Kajinami proposed openstack/neutron master: Use native interface to explain choices https://review.opendev.org/c/openstack/neutron/+/940980 | 15:52 |
| opendevreview | Merged openstack/ovn-bgp-agent master: Add BGP NIC IP cleanup logic for the OVN driver https://review.opendev.org/c/openstack/ovn-bgp-agent/+/940224 | 16:01 |
| opendevreview | Sahid Orentino Ferdjaoui proposed openstack/neutron master: agent/linux: remove eventlet usage from of_monitor https://review.opendev.org/c/openstack/neutron/+/940684 | 16:03 |
| opendevreview | Sahid Orentino Ferdjaoui proposed openstack/neutron master: agent/linux: remove eventlet usage from ip_conntrack https://review.opendev.org/c/openstack/neutron/+/940685 | 16:03 |
| opendevreview | Sahid Orentino Ferdjaoui proposed openstack/neutron master: agent/linux: remove eventlet usage from ip_lib https://review.opendev.org/c/openstack/neutron/+/940686 | 16:03 |
| opendevreview | Sahid Orentino Ferdjaoui proposed openstack/neutron master: ovs: reimplement signals handling https://review.opendev.org/c/openstack/neutron/+/939321 | 16:03 |
| opendevreview | Sahid Orentino Ferdjaoui proposed openstack/neutron master: ovs: remove the usage of eventlet in the OVS agent https://review.opendev.org/c/openstack/neutron/+/937765 | 16:03 |
| opendevreview | Sahid Orentino Ferdjaoui proposed openstack/neutron master: ofswitch: remove usage of eventlet timeout https://review.opendev.org/c/openstack/neutron/+/940982 | 16:03 |
| opendevreview | Sahid Orentino Ferdjaoui proposed openstack/neutron master: osdb: remove usage of eventlet in dbhandler https://review.opendev.org/c/openstack/neutron/+/940983 | 16:03 |
| opendevreview | Fernando Royo proposed openstack/ovn-octavia-provider master: Add LB sync logic https://review.opendev.org/c/openstack/ovn-octavia-provider/+/925324 | 16:11 |
| opendevreview | Fernando Royo proposed openstack/ovn-octavia-provider master: Add Listener sync logic https://review.opendev.org/c/openstack/ovn-octavia-provider/+/931250 | 16:11 |
| opendevreview | Fernando Royo proposed openstack/ovn-octavia-provider master: Add Pool sync logic https://review.opendev.org/c/openstack/ovn-octavia-provider/+/931266 | 16:11 |
| opendevreview | Fernando Royo proposed openstack/ovn-octavia-provider master: Add Member sync logic https://review.opendev.org/c/openstack/ovn-octavia-provider/+/931267 | 16:12 |
| opendevreview | Fernando Royo proposed openstack/ovn-octavia-provider master: Add Health monitor sync logic https://review.opendev.org/c/openstack/ovn-octavia-provider/+/931288 | 16:12 |
| opendevreview | Fernando Royo proposed openstack/ovn-octavia-provider master: Add sync floating IP support https://review.opendev.org/c/openstack/ovn-octavia-provider/+/929039 | 16:12 |
| opendevreview | Brian Haley proposed openstack/neutron-specs master: Update to use openstack-python3-jobs template https://review.opendev.org/c/openstack/neutron-specs/+/940986 | 16:16 |
| opendevreview | Renjing Xiao proposed x/whitebox-neutron-tempest-plugin master: Add new nested snat test to fetch ovn config dynamically https://review.opendev.org/c/x/whitebox-neutron-tempest-plugin/+/940908 | 16:18 |
| opendevreview | Renjing Xiao proposed x/whitebox-neutron-tempest-plugin master: Add new nested snat test to fetch ovn config dynamically https://review.opendev.org/c/x/whitebox-neutron-tempest-plugin/+/940908 | 16:21 |
| opendevreview | Fernando Royo proposed openstack/ovn-octavia-provider master: Add LB sync logic https://review.opendev.org/c/openstack/ovn-octavia-provider/+/925324 | 16:22 |
| opendevreview | Fernando Royo proposed openstack/ovn-octavia-provider master: Add Listener sync logic https://review.opendev.org/c/openstack/ovn-octavia-provider/+/931250 | 16:22 |
| opendevreview | Fernando Royo proposed openstack/ovn-octavia-provider master: Add Pool sync logic https://review.opendev.org/c/openstack/ovn-octavia-provider/+/931266 | 16:22 |
| opendevreview | Fernando Royo proposed openstack/ovn-octavia-provider master: Add Member sync logic https://review.opendev.org/c/openstack/ovn-octavia-provider/+/931267 | 16:22 |
| opendevreview | Fernando Royo proposed openstack/ovn-octavia-provider master: Add Health monitor sync logic https://review.opendev.org/c/openstack/ovn-octavia-provider/+/931288 | 16:22 |
| opendevreview | Fernando Royo proposed openstack/ovn-octavia-provider master: Add sync floating IP support https://review.opendev.org/c/openstack/ovn-octavia-provider/+/929039 | 16:22 |
| opendevreview | Fernando Royo proposed openstack/ovn-octavia-provider stable/2023.2: Remove join on helper request daemon thread https://review.opendev.org/c/openstack/ovn-octavia-provider/+/940794 | 16:23 |
| opendevreview | Fernando Royo proposed openstack/ovn-octavia-provider stable/2024.1: Remove join on helper request daemon thread https://review.opendev.org/c/openstack/ovn-octavia-provider/+/940793 | 16:23 |
| opendevreview | Brian Haley proposed openstack/neutron-specs master: Update to use openstack-python3-jobs template https://review.opendev.org/c/openstack/neutron-specs/+/940986 | 16:28 |
| opendevreview | Fernando Royo proposed openstack/ovn-octavia-provider master: Add Listener sync logic https://review.opendev.org/c/openstack/ovn-octavia-provider/+/931250 | 16:41 |
| opendevreview | Fernando Royo proposed openstack/ovn-octavia-provider master: Add Pool sync logic https://review.opendev.org/c/openstack/ovn-octavia-provider/+/931266 | 16:41 |
| opendevreview | Fernando Royo proposed openstack/ovn-octavia-provider master: Add Member sync logic https://review.opendev.org/c/openstack/ovn-octavia-provider/+/931267 | 16:41 |
| opendevreview | Fernando Royo proposed openstack/ovn-octavia-provider master: Add Health monitor sync logic https://review.opendev.org/c/openstack/ovn-octavia-provider/+/931288 | 16:41 |
| opendevreview | Fernando Royo proposed openstack/ovn-octavia-provider master: Add sync floating IP support https://review.opendev.org/c/openstack/ovn-octavia-provider/+/929039 | 16:41 |
| opendevreview | Takashi Kajinami proposed openstack/neutron master: Use native interface to explain choices https://review.opendev.org/c/openstack/neutron/+/940980 | 16:58 |
| opendevreview | Takashi Kajinami proposed openstack/neutron master: Use native interface to explain choices https://review.opendev.org/c/openstack/neutron/+/940980 | 17:01 |
| lajoskatona | sahid: Hi, I checked again (os-ken series: https://review.opendev.org/c/openstack/neutron/+/937765 & neutron: https://review.opendev.org/c/openstack/os-ken/+/938337 )and it is still not working for me (no ovs-agent in the agent list as 1st simptom) | 17:15 |
| sahid | lajoskatona: it's strange | 17:15 |
| sahid | i have mine well registered with metaga and dhcp | 17:16 |
| sahid | in all cases, in a cases there are still some issue with neutron-ovs-tempest-with-os-ken-master | 17:18 |
| sahid | https://zuul.opendev.org/t/openstack/build/0538d30960b8465e9cf7b0bb2f05d60f/logs | 17:18 |
| sahid | I have split my last patch in 2 smaller patches to try to identify what is making the tests failing | 17:19 |
| sahid | s/2/3 | 17:19 |
| sahid | btw thanks a lot for your tests :-) | 17:19 |
| sahid | in the logs of q-agt, do you see any error? | 17:20 |
| lajoskatona | sahid: no error logs, all seem quiet | 17:25 |
| sahid | but you see the log message form the event loop? | 17:26 |
| sahid | log messages | 17:26 |
| sahid | Feb 07 18:27:10 devstack01 neutron-openvswitch-agent[1134689]: DEBUG neutron.agent.common.ovsdb_monitor [None req-cc5087cf-5540-4259-acc7-4aa4b0cbf12f None None] Current size of new_events: added=0 modified=0 removed=0. New events: {'added': [], 'removed': [], 'modified': []} {{(pid=1134689) process_events /opt/stack/neutron/neutron/agent/common/ovsdb_monitor.py:147}} | 17:27 |
| sahid | no.. | 17:27 |
| sahid | Feb 07 18:27:10 devstack01 neutron-openvswitch-agent[1134689]: INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [None req-cc5087cf-5540-4259-acc7-4aa4b0cbf12f None None] Agent rpc_loop - iteration:9229 completed. Processed ports statistics: {'regular': {'added': 0, 'updated': 0, 'removed': 0}}. Elapsed:0.008 | 17:27 |
| opendevreview | Merged openstack/ovn-octavia-provider stable/2024.2: Fix pep8 with pylint 3.3.0 https://review.opendev.org/c/openstack/ovn-octavia-provider/+/940866 | 18:28 |
| opendevreview | Merged openstack/neutron stable/2024.2: OVS: Set log level to INFO for unconfigured ofport cases https://review.opendev.org/c/openstack/neutron/+/940949 | 18:31 |
| opendevreview | Merged openstack/neutron stable/2024.1: OVS: Set log level to INFO for unconfigured ofport cases https://review.opendev.org/c/openstack/neutron/+/940950 | 18:31 |
| opendevreview | Merged openstack/neutron unmaintained/yoga: Set trunk parent port as access port in ovs to avoid loop https://review.opendev.org/c/openstack/neutron/+/913505 | 20:07 |
| opendevreview | Merged openstack/neutron stable/2023.2: OVS: Set log level to INFO for unconfigured ofport cases https://review.opendev.org/c/openstack/neutron/+/940951 | 20:31 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: WIP == [OVN] Create a deterministic hash ring node UUID generator https://review.opendev.org/c/openstack/neutron/+/941008 | 21:02 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: [OVN] Create a deterministic hash ring node UUID generator https://review.opendev.org/c/openstack/neutron/+/941008 | 21:29 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: [OVN] Refresh host nodes before notifying https://review.opendev.org/c/openstack/neutron/+/940256 | 21:33 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: [OVN] Refactor ``HashRingManager`` sync method https://review.opendev.org/c/openstack/neutron/+/940342 | 21:33 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: [OVN] Remove the hash ring maintenance task https://review.opendev.org/c/openstack/neutron/+/940427 | 21:33 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: DNM - Test "neutron-ovn-tempest-ipv6-only-ovs*" with WSGI https://review.opendev.org/c/openstack/neutron/+/939977 | 21:33 |
| opendevreview | Brian Haley proposed openstack/neutron master: Add basical functionalities for metadata path extension https://review.opendev.org/c/openstack/neutron/+/881535 | 21:36 |
| opendevreview | Merged openstack/neutron master: Add meter bandwidth limit support https://review.opendev.org/c/openstack/neutron/+/860767 | 23:40 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!