| opendevreview | likui proposed openstack/neutron-vpnaas master: Update UPPER_CONSTRAINTS_FILE URL https://review.opendev.org/c/openstack/neutron-vpnaas/+/895990 | 02:04 |
|---|---|---|
| opendevreview | likui proposed openstack/neutron-vpnaas master: Update TOX_CONSTRAINTS_FILE URL https://review.opendev.org/c/openstack/neutron-vpnaas/+/895990 | 02:05 |
| *** ravlew is now known as Guest808 | 04:18 | |
| frickler | haleyb: ralonsoh: please check https://review.opendev.org/c/openstack/os-ken/+/894134 (usually I think bot patches can be single-approved, not sure how neutron handles this in general?) | 07:00 |
| ralonsoh | let me check | 07:25 |
| opendevreview | Merged openstack/os-ken master: Update master for stable/2023.2 https://review.opendev.org/c/openstack/os-ken/+/894134 | 07:38 |
| sahid | lajoskatona: o/ | 08:05 |
| sahid | regarding the rpc workers change, is that only related to neutron server or should we also update that on agents aswell? | 08:06 |
| lajoskatona | sahid: Hi, the workers' settings are for the server(s), so I would change it only on the server side | 08:14 |
| sahid | for a given cluster of 4 neutron-server, we are agree we don't have any notion of a master which could get more traffics or doing more work? | 08:29 |
| sahid | one of how neutron server is doing twince of the work compared to the others | 08:29 |
| sahid | s/how/our | 08:29 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron-lib master: Add the "cancellable" flag to the ``CallbacksManager`` events https://review.opendev.org/c/openstack/neutron-lib/+/895940 | 08:31 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: Make ``OVNMechanismDriver.post_fork_initialize`` callback cancellable https://review.opendev.org/c/openstack/neutron/+/895946 | 08:32 |
| ralonsoh | slaweq, hi! if you have 1 minute: https://review.opendev.org/c/openstack/neutron/+/893447 | 08:32 |
| ralonsoh | ah, and https://review.opendev.org/c/openstack/neutron/+/887977 (if you have time) | 08:42 |
| sahid | i'm suspecting something when a agent is reconnecting to the server | 08:46 |
| sahid | it looks like it's always the same that read the rpc | 08:46 |
| *** elodilles_pto is now known as elodilles | 08:47 | |
| opendevreview | Lucas Alvares Gomes proposed openstack/neutron master: [OVN] External ports scheduling (WIP) https://review.opendev.org/c/openstack/neutron/+/894767 | 09:02 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: [OVN] Add a log message after the "post_fork_initialize" method https://review.opendev.org/c/openstack/neutron/+/896009 | 09:11 |
| mohan | hi | 10:12 |
| mohan | neutron-bug-deputy: could you have a look at : https://bugs.launchpad.net/neutron/+bug/2028338 | 10:37 |
| frickler | mohan: which version of neutron are you seeing this on? | 12:09 |
| mohan | frickler: it's been a while I've raised this bug , maybe ussuri not sure !! | 12:42 |
| racosta | Hello folks, has anyone who uses OVN seen or has any idea why neutron-ovn-metadata-agent uses a high CPU load? (~10% - OpenStack Yoga). With approximately 60 networks/processes on a chassis I'm seeing that the processes are receiving port updates every 3 seconds. | 12:43 |
| racosta | recvfrom(13, "{\"id\":null,\"method\":\"update3\",\"params\":[\".............",{\"Port_Binding\":{\"........... | 12:43 |
| frickler | mohan: well if the issue isn't reproducible, there's not much chance of getting it resolved. the "incomplete" status calls for more information to make it reproducible, being able to state at least one affected version is a pretty important part of that | 12:46 |
| racosta | I'm talking about "idle" operation, without creating/removing VMs. It seems to me like unusual CPU usage in these conditions. Have you seen this felixhuettner[m]? | 12:49 |
| mohan | frickler: I'm sure it's reproducable . Let me setup devstack and test it | 12:49 |
| ralonsoh | mohan, hi, please check in the logs the message "Agent has just been revived. Doing a full sync." or any other message in L3NATAgentWithStateReport._report_state, in the L3 agent logs | 12:55 |
| ralonsoh | you can also check the "report_state" RPC call in the Neutron API | 12:55 |
| ralonsoh | racosta, what do you see in the logs? | 12:55 |
| ralonsoh | are the VMs doing http requests for metadata? | 12:55 |
| ralonsoh | and how many metadata workers do you have? | 12:56 |
| racosta | ralonsoh, the neutron-ovn-metadata-agent log for the last 5 minutes is empty (I'm running without debug in production). | 12:58 |
| racosta | 60 worker (per network) | 12:58 |
| ralonsoh | no no, metadata workers | 12:58 |
| ralonsoh | not neutron api workers | 12:58 |
| racosta | let me check | 12:59 |
| ralonsoh | if the system is not very active, you can reduce the number of "metadata_workers" to 0 | 13:00 |
| ralonsoh | that will 1) reduce the number of OVN DB connections and 2) will spawn the HTTP server on the same process as the metadata agent | 13:00 |
| ralonsoh | if the number of http request is low (not too much activity in the compute node, regardless of the number of VMs) that will reduce the metadata cpu load | 13:01 |
| racosta | I'm not configuring metadata_workers, it's default. | 13:03 |
| ralonsoh | what version are you using? that value depends on the version | 13:04 |
| racosta | 16.x (Ussuri) and 20.x (Yoga). | 13:09 |
| racosta | It seems that in cluster mode (OVN) it increased... in this case with more OVN DB connections | 13:11 |
| ralonsoh | racosta, yoga can have metadata_workers=0, spawning the http server in the same process | 13:19 |
| ralonsoh | ussuri doesn't | 13:19 |
| ralonsoh | check how many metadata processes you have per compute node | 13:20 |
| racosta | sorry, I confirmed that on Yoga, even with frequent port update events, CPU usage is very low.(with more than 40 metadata processes on a compute node). | 13:23 |
| ralonsoh | what does it means 40 metadata processes? | 13:24 |
| ralonsoh | didn't you say that you didn't overwrite the "metadata_workers" variable? | 13:25 |
| ralonsoh | if you have 40 metadata workers, you are killing the OVN DB, in particular the SB | 13:25 |
| ralonsoh | please, check that, 40 workers is unnecessary | 13:25 |
| racosta | Exactly, that seems to be the problem. | 13:26 |
| racosta | The Yoga deploy uses one metadata and N haproxy processes... In Ussuri it is N to N (no way)... | 13:27 |
| ralonsoh | no, we spawn one haproxy per metadata worker | 13:28 |
| ralonsoh | that didn't change | 13:28 |
| *** jlibosva is now known as Guest844 | 13:33 | |
| racosta | ok, but when haproxy is spawed in the same process it replaces the old nneutron-ovn-metadata-agent process, right? | 13:34 |
| ralonsoh | no, the ovn metadata agent is running in one process. This process will start as many metadata_workers as configured. Each worker is a haproxy that receives the http requests and proxies them | 13:38 |
| ralonsoh | each worker has a OVN SB DB connection too | 13:38 |
| racosta | Thanks, it makes sense. I see a lot of metadata_workers in ussuri because the default is very high (default=host.cpu_count() // 2) | 13:43 |
| racosta | I'll set this metadata_workers=0 in Ussuri and test asap. Thanks a lot for the help ralonsoh :) | 13:47 |
| opendevreview | Jakub Skunda proposed openstack/neutron-tempest-plugin master: Remove test duplication between tempest and n-t-p RoutersDVRTest https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/896122 | 14:05 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: Add a "port" child table "porthardwareoffloadtype" https://review.opendev.org/c/openstack/neutron/+/882832 | 14:16 |
| ralonsoh | slaweq, please check https://review.opendev.org/c/openstack/neutron/+/887977. Trivial stuff, no rush | 14:16 |
| opendevreview | Brian Haley proposed openstack/neutron master: Fix pointless-string-statement pylint warning https://review.opendev.org/c/openstack/neutron/+/895975 | 14:44 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: [OVN] Populate the "router.distributed" flag in ML2/OVN https://review.opendev.org/c/openstack/neutron/+/886992 | 14:57 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: Replace "tenant_id" with "project_id" in IPAM engine https://review.opendev.org/c/openstack/neutron/+/877533 | 15:01 |
| opendevreview | Luis Tomas Bolivar proposed openstack/ovn-bgp-agent master: Add initial support for local OVN cluster instead of kernel-networking https://review.opendev.org/c/openstack/ovn-bgp-agent/+/881779 | 15:01 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: [OVN] Remove backwards compatibility with OVN < v20.09 https://review.opendev.org/c/openstack/neutron/+/884898 | 15:02 |
| ralonsoh | bcafarel, hi! if you have 2 mins: https://review.opendev.org/c/openstack/neutron/+/895433 | 15:03 |
| ralonsoh | thanks! | 15:03 |
| jlibosva | haleyb: hey Brian :) thanks for commenting on https://review.opendev.org/c/openstack/neutron/+/895849/1/tools/configure_for_func_testing.sh - it was just to test provide testing results for the depending patch but perhaps we can bump it and merge. Is there a plan which versions we intend to use, do they have to be LTS? | 15:10 |
| ralonsoh | jlibosva, for functional doesn't need to be the LTS one, this is why we are compiling it | 15:12 |
| ralonsoh | we use the distributed version in the OS in the integrated networking job (and other n-t-p ones) | 15:13 |
| jlibosva | ralonsoh: I think it would be good to have those aligned but the distribution one is quite "old" for master branch. Thinking about scenario where we work on a feature that requires OVN bits we may need fairly new OVN | 15:17 |
| ralonsoh | jlibosva, but this is why we need to override the distributed version. We need to keep compatibility with 22.03 (during this release will remove compatibility wiht 20.03) | 15:17 |
| ralonsoh | but FT will need to check newer features implemented in newer versions | 15:18 |
| jlibosva | why would we want FT to be different? it's just a different level of testing. imho we should claim "This current stable release works with this OVN version" | 15:19 |
| jlibosva | if we use 2 different versions of OVN in different testing suites then we may be missing coverage for the version we claim works | 15:19 |
| ralonsoh | to test new features, as commented | 15:19 |
| jlibosva | but then we won't test them with other jobs | 15:20 |
| ralonsoh | we can't be stuck in 20.03 or 22.03 during 2 years | 15:20 |
| jlibosva | I agree with that | 15:20 |
| haleyb | jlibosva: hey Kuba. I just noticed the LTS comment there, and v22.03.x is Ubuntu LTS for cloud, which is the underlying OS for most things here. If the goal is new features it doesn't fit | 15:20 |
| ralonsoh | no, in the other jobs will check using the released OVN version, checking that there are no regressions | 15:20 |
| haleyb | the previous version was v23.03.3 which is stable | 15:20 |
| haleyb | err, maybe to me 23.03 == LTS, habit | 15:21 |
| jlibosva | so the idea is to stay on LTS Ubuntu until the next LTS, with all the packages that come with it (e.g. OVN being 22.03) and then have FT with some new compiled stable OVN? | 15:24 |
| jlibosva | and we won't test a Neutron feature, if it relies on some late OVN code, until the next Ubuntu LTS release? | 15:25 |
| jlibosva | test end-to-end that is | 15:25 |
| ralonsoh | For example: chassis_private support. It was implemented after 20.03, but we started supporting and testing it in FT | 15:27 |
| ralonsoh | using newer versions | 15:27 |
| jlibosva | but with tempest we stayed with chassis? | 15:27 |
| ralonsoh | yes when using Focal (ovn 20.03) | 15:27 |
| ralonsoh | now we test it with Jammy (ovn 22.03) | 15:27 |
| ralonsoh | in any case, we have "two" tempests: we have "tempest-integrated-networking", that uses always the released OVN in the OS | 15:28 |
| jlibosva | thanks for explanation - so now with the FT, I understand it would be ok to bump it to stable non-LTS version? | 15:28 |
| ralonsoh | and n-t-p, where we alse override the OVN version | 15:28 |
| haleyb | jlibosva: well, in this case we're building OVN by default, and looking at the changelog it went from main/20.03/20.06/22.03... so i guess in this case 22.06.1 is ok. | 15:29 |
| jlibosva | got it | 15:29 |
| ralonsoh | yes for FT if we need to test new features | 15:29 |
| haleyb | what rodolfo said :) | 15:29 |
| jlibosva | ok, so now what would be a good version? the latest stable or the earliest that is required for a feature? | 15:30 |
| haleyb | maybe it was just 22.06 seemed random, if it was 23.0x that would be newer, not sure how bleeding edge we want to go | 15:31 |
| -opendevstatus- NOTICE: The lists.openinfra.dev and lists.starlingx.io sites will be offline briefly for migration to a new server | 15:31 | |
| jlibosva | I put 22.06 as this was the first release with additional_chassis column that I needed | 15:32 |
| jlibosva | but still, this is more than a year old release | 15:33 |
| jlibosva | ancient :) | 15:33 |
| haleyb | ah, makes sense. v23.09.0 just dropped, as did v23.06.2 | 15:33 |
| jlibosva | so maybe we want newer. I don't know. I'm asking | 15:33 |
| ralonsoh | in FT, I would use the newer one, testing that this version is not breaking anything in neutron | 15:34 |
| jlibosva | my thinking is if we use 23.09 then maybe we won't need to change it for another year, perhaps just the minor version for bugfixes if any | 15:34 |
| ralonsoh | we have tempest to avoid regressions in the Neutron code | 15:34 |
| haleyb | i don't know either, anything newer should work, if not we change when we find a bug | 15:34 |
| jlibosva | but we have conditions in Neutron that depends on OVN schema version, so the regressed code might not be hit until we have the updated schema | 15:34 |
| jlibosva | depend* | 15:35 |
| jlibosva | ok, so I'm gonna make that patch more "official" and will use the v23.09.0 tag, how about that? | 15:35 |
| ralonsoh | +1 | 15:35 |
| haleyb | +1, and that requires 1d78a3f3164a6bf651b34f52812f38655b28a9ce ovs (v3.2.0-20-g1d78a3f31) | 15:36 |
| haleyb | from 'git submodule' | 15:36 |
| jlibosva | perfect, thanks a lot! :) | 15:36 |
| haleyb | i'm assuming 'git submodule update' did the right thing | 15:36 |
| ralonsoh | yes | 15:37 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: Add a new extension "security-groups-rules-belongs-to-default-sg" https://review.opendev.org/c/openstack/neutron/+/883907 | 15:50 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: Add a new extension "security-groups-rules-belongs-to-default-sg" https://review.opendev.org/c/openstack/neutron/+/883907 | 16:37 |
| opendevreview | Merged openstack/neutron master: Create a single method to set the quota usage dirty bit https://review.opendev.org/c/openstack/neutron/+/887977 | 17:52 |
| opendevreview | Jakub Libosvar proposed openstack/neutron master: ovn-metadata: Refactor events https://review.opendev.org/c/openstack/neutron/+/896163 | 20:32 |
| opendevreview | Jakub Libosvar proposed openstack/neutron master: ovn-metadata: Refactor events https://review.opendev.org/c/openstack/neutron/+/896163 | 20:47 |
| *** damian___921832 is now known as damian___92183 | 20:56 | |
| opendevreview | Jakub Libosvar proposed openstack/neutron master: ovn-metadata: Refactor events https://review.opendev.org/c/openstack/neutron/+/896163 | 21:00 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!