| *** snapiri has quit IRC | 00:18 | |
| *** snapiri has joined #openstack-neutron | 00:18 | |
| *** ksambor has quit IRC | 00:20 | |
| openstackgerrit | Terry Wilson proposed openstack/neutron master: [OVN] Allow schema checks to happen before starting connection https://review.opendev.org/c/openstack/neutron/+/754485 | 00:25 |
|---|---|---|
| openstackgerrit | Merged openstack/neutron stable/ussuri: Improve DHCP agent's debug messages https://review.opendev.org/c/openstack/neutron/+/772105 | 00:36 |
| *** rm_work has quit IRC | 00:37 | |
| *** logan- has quit IRC | 00:38 | |
| *** inflatador has quit IRC | 00:40 | |
| *** logan- has joined #openstack-neutron | 00:40 | |
| *** yamamoto_ has joined #openstack-neutron | 00:45 | |
| *** yamamoto has quit IRC | 00:46 | |
| *** gregoiremahe has quit IRC | 00:46 | |
| *** gregoiremahe has joined #openstack-neutron | 00:47 | |
| *** tbachman has quit IRC | 00:49 | |
| *** tbachman has joined #openstack-neutron | 00:49 | |
| *** dviroel has quit IRC | 00:52 | |
| openstackgerrit | yangjianfeng proposed openstack/neutron-specs master: L3 router support ndp proxy https://review.opendev.org/c/openstack/neutron-specs/+/728628 | 00:53 |
| *** sapd__x has quit IRC | 00:58 | |
| *** sri_ has quit IRC | 01:11 | |
| *** mgagne has quit IRC | 01:11 | |
| *** sri_ has joined #openstack-neutron | 01:12 | |
| *** mgagne has joined #openstack-neutron | 01:12 | |
| *** mnaser has quit IRC | 01:13 | |
| *** thedac has quit IRC | 01:13 | |
| *** fyx has quit IRC | 01:14 | |
| *** masterpe has quit IRC | 01:15 | |
| *** thedac has joined #openstack-neutron | 01:15 | |
| *** mnaser has joined #openstack-neutron | 01:16 | |
| *** macz_ has joined #openstack-neutron | 01:24 | |
| *** macz_ has quit IRC | 01:29 | |
| *** rm_work has joined #openstack-neutron | 01:29 | |
| *** fyx has joined #openstack-neutron | 01:33 | |
| *** raildo has quit IRC | 01:35 | |
| *** yan0s has quit IRC | 01:57 | |
| *** rm_work has joined #openstack-neutron | 02:01 | |
| *** yamamoto_ has quit IRC | 02:15 | |
| *** yamamoto has joined #openstack-neutron | 02:16 | |
| *** liuyulong has joined #openstack-neutron | 02:18 | |
| openstackgerrit | liuyulong proposed openstack/neutron master: Add neutron_tempest_plugin.scenario.* cases to multinode jobs https://review.opendev.org/c/openstack/neutron/+/772965 | 02:22 |
| *** sapd1_x has quit IRC | 02:49 | |
| *** zzzeek has quit IRC | 03:01 | |
| *** zzzeek has joined #openstack-neutron | 03:02 | |
| *** sapd__x has joined #openstack-neutron | 03:21 | |
| *** johanssone has quit IRC | 03:24 | |
| *** rcernin has quit IRC | 03:27 | |
| *** nicolasbock_ has joined #openstack-neutron | 03:30 | |
| *** lyr6 has joined #openstack-neutron | 03:31 | |
| *** ianw_ has joined #openstack-neutron | 03:32 | |
| *** cp-- has joined #openstack-neutron | 03:37 | |
| *** mgariepy has quit IRC | 03:38 | |
| *** nicolasbock has quit IRC | 03:38 | |
| *** cp- has quit IRC | 03:38 | |
| *** lyr has quit IRC | 03:38 | |
| *** ebbex has quit IRC | 03:38 | |
| *** jpena|off has quit IRC | 03:38 | |
| *** ianw has quit IRC | 03:38 | |
| *** beagles has quit IRC | 03:38 | |
| *** ianw_ is now known as ianw | 03:38 | |
| *** nicolasbock_ is now known as nicolasbock | 03:38 | |
| *** lyr6 is now known as lyr | 03:39 | |
| *** jpena|off has joined #openstack-neutron | 03:39 | |
| *** ebbex has joined #openstack-neutron | 03:41 | |
| *** rcernin has joined #openstack-neutron | 03:41 | |
| *** gouthamr has quit IRC | 03:43 | |
| *** beagles has joined #openstack-neutron | 03:43 | |
| *** mgariepy has joined #openstack-neutron | 03:45 | |
| *** gouthamr has joined #openstack-neutron | 04:00 | |
| *** gouthamr has quit IRC | 04:00 | |
| *** ratailor has joined #openstack-neutron | 04:14 | |
| *** gouthamr has joined #openstack-neutron | 04:18 | |
| *** gouthamr has quit IRC | 04:18 | |
| *** lajoskatona has joined #openstack-neutron | 05:01 | |
| openstackgerrit | Terry Wilson proposed openstack/neutron master: [OVN] Allow schema checks to happen before starting connection https://review.opendev.org/c/openstack/neutron/+/754485 | 05:02 |
| *** hjensas|afk is now known as hjensas | 05:18 | |
| openstackgerrit | liuyulong proposed openstack/neutron master: Add neutron_tempest_plugin.scenario.* cases to multinode jobs https://review.opendev.org/c/openstack/neutron/+/772965 | 05:28 |
| *** sapd__x has quit IRC | 05:42 | |
| *** calbers_ has joined #openstack-neutron | 05:45 | |
| *** calbers has quit IRC | 05:46 | |
| *** calbers_ is now known as calbers | 05:46 | |
| *** sapd__x has joined #openstack-neutron | 05:47 | |
| *** ianw has quit IRC | 05:59 | |
| *** ianw has joined #openstack-neutron | 06:01 | |
| *** lbragstad has quit IRC | 06:11 | |
| openstackgerrit | Merged openstack/neutron stable/ussuri: ovn: Support live migration to DPDK nodes https://review.opendev.org/c/openstack/neutron/+/771124 | 06:16 |
| *** lbragstad has joined #openstack-neutron | 06:16 | |
| *** gouthamr has joined #openstack-neutron | 06:22 | |
| *** gouthamr has quit IRC | 06:22 | |
| *** ygk_12345 has joined #openstack-neutron | 06:55 | |
| *** gouthamr has joined #openstack-neutron | 06:56 | |
| *** gouthamr has quit IRC | 06:56 | |
| *** gcheresh has joined #openstack-neutron | 06:57 | |
| *** lbragstad has quit IRC | 07:14 | |
| *** lbragstad has joined #openstack-neutron | 07:14 | |
| *** viks____ has joined #openstack-neutron | 07:19 | |
| *** gouthamr has joined #openstack-neutron | 07:22 | |
| *** openstackgerrit has quit IRC | 07:23 | |
| *** lpetrut has joined #openstack-neutron | 07:30 | |
| *** ralonsoh has joined #openstack-neutron | 07:39 | |
| *** lajoskatona has quit IRC | 07:40 | |
| *** lajoskatona has joined #openstack-neutron | 07:48 | |
| *** slaweq has joined #openstack-neutron | 07:57 | |
| *** gcheresh has quit IRC | 07:59 | |
| *** johanssone has joined #openstack-neutron | 07:59 | |
| *** tesseract has joined #openstack-neutron | 08:00 | |
| *** tmorin has joined #openstack-neutron | 08:01 | |
| *** slaweq has quit IRC | 08:10 | |
| *** slaweq has joined #openstack-neutron | 08:14 | |
| *** tmorin has quit IRC | 08:15 | |
| lajoskatona | Good morning, shall I ask review for some stadium projects, these are to make their master free of recent issues with dependencies: | 08:19 |
| lajoskatona | https://review.opendev.org/c/openstack/networking-bgpvpn/+/771219 & https://review.opendev.org/c/openstack/networking-odl/+/769877 | 08:19 |
| lajoskatona | thanks in advance | 08:19 |
| *** benj_ has quit IRC | 08:22 | |
| *** ygk_12345 has quit IRC | 08:24 | |
| *** tmorin has joined #openstack-neutron | 08:32 | |
| *** macz_ has joined #openstack-neutron | 08:33 | |
| *** bengates has joined #openstack-neutron | 08:33 | |
| *** macz_ has quit IRC | 08:37 | |
| *** jlibosva has joined #openstack-neutron | 08:39 | |
| *** bengates has quit IRC | 08:46 | |
| *** bengates has joined #openstack-neutron | 08:46 | |
| *** rpittau|afk is now known as rpittau | 08:52 | |
| ralonsoh | lajoskatona, done | 08:53 |
| lajoskatona | ralonsoh: thanks, I check the ones below these, to make the stable branches clean as well :-) | 08:53 |
| *** jpena|off is now known as jpena | 08:57 | |
| *** elvira has joined #openstack-neutron | 08:58 | |
| *** ccstone has quit IRC | 08:59 | |
| *** ccstone has joined #openstack-neutron | 08:59 | |
| *** zzzeek has quit IRC | 09:01 | |
| *** rcernin has quit IRC | 09:02 | |
| *** yamamoto has quit IRC | 09:02 | |
| *** zzzeek has joined #openstack-neutron | 09:03 | |
| *** bengates_ has joined #openstack-neutron | 09:06 | |
| *** lucasagomes has joined #openstack-neutron | 09:07 | |
| *** gcheresh has joined #openstack-neutron | 09:07 | |
| *** bengates has quit IRC | 09:10 | |
| *** openstackgerrit has joined #openstack-neutron | 09:16 | |
| openstackgerrit | Michal Nasiadka proposed openstack/neutron master: ovn: set allowed_address_pairs to addresses also https://review.opendev.org/c/openstack/neutron/+/772983 | 09:16 |
| openstackgerrit | Michal Nasiadka proposed openstack/neutron master: ovn: set allowed_address_pairs to addresses also https://review.opendev.org/c/openstack/neutron/+/772983 | 09:18 |
| *** gcheresh has quit IRC | 09:21 | |
| *** rcernin has joined #openstack-neutron | 09:21 | |
| *** bengates_ has quit IRC | 09:29 | |
| *** bengates has joined #openstack-neutron | 09:30 | |
| *** obondarev has joined #openstack-neutron | 09:33 | |
| *** bengates has quit IRC | 09:33 | |
| *** bengates has joined #openstack-neutron | 09:34 | |
| *** slaweq has quit IRC | 09:35 | |
| *** bengates_ has joined #openstack-neutron | 09:36 | |
| *** yamamoto has joined #openstack-neutron | 09:38 | |
| *** bengates has quit IRC | 09:39 | |
| *** slaweq has joined #openstack-neutron | 09:43 | |
| obondarev | hi all, seems rally job in check queue is broken on master | 09:45 |
| obondarev | one of the failures are list_port requests taking too much time | 09:45 |
| obondarev | a brief look at logs shows most of the time is spent on "neutron.pecan_wsgi.hooks.policy_enforcement [None req-9c144574-6ccf-4e89-b517-f76e830193d0 c_rally_111c37a9_Sq0DRK8S c_rally_111c37a9_eeSkdtpE] Attributes excluded by policy engine: ['standard_attr_id', 'binding:profile', 'binding:host_id', 'binding:vif_type', 'binding:vif_details'] {{(pid=66366) _exclude_attributes_by_policy /opt/stack/neutron/neutron/pecan_wsgi/hooks | 09:46 |
| obondarev | /policy_enforcement.py:255}}" | 09:46 |
| obondarev | a log line per each port it seems | 09:46 |
| slaweq | obondarev: maybe it's related to patches https://review.opendev.org/q/project:openstack/neutron+topic:secure-rbac | 09:47 |
| obondarev | @slaweq, yeah I was thinking the same | 09:47 |
| obondarev | but secure RBAC for ports is not merged yet | 09:48 |
| slaweq | nope | 09:48 |
| slaweq | obondarev: let me check when it started failing | 09:48 |
| *** yamamoto has quit IRC | 09:48 | |
| obondarev | @slaweq, on Jan 27: https://zuul.opendev.org/t/openstack/builds?job_name=neutron-rally-task&project=openstack%2Fneutron&branch=master | 09:49 |
| slaweq | according to https://grafana.opendev.org/d/PfjNuthGz/neutron-failure-rate?viewPanel=30&orgId=1 it started around 27.01 | 09:49 |
| obondarev | there was a bunch of secure RBAC patches merged this day | 09:50 |
| slaweq | and only them | 09:50 |
| slaweq | obondarev: will You check that? | 10:01 |
| slaweq | or should I? | 10:01 |
| obondarev | @slaweq, unfortunately have little time for it now, like half a hour, not sure it'll be enough.. | 10:02 |
| slaweq | ok, I will open LP bug and will try to look at it more deeply today | 10:02 |
| obondarev | great, thanks! | 10:04 |
| slaweq | https://bugs.launchpad.net/neutron/+bug/1913718 | 10:05 |
| openstack | Launchpad bug 1913718 in neutron "rally ci job is unstable - port list takes very long time" [Critical,Confirmed] - Assigned to Slawek Kaplonski (slaweq) | 10:05 |
| slaweq | obondarev: ^^ | 10:05 |
| *** ociuhandu has joined #openstack-neutron | 10:06 | |
| slaweq | obondarev: I think I have same issue on my devstack | 10:09 |
| slaweq | so hopefully I will be able to debug it now | 10:09 |
| slaweq | if I will not find it today, I will make rally job non-voting temporary to unblock the gate | 10:10 |
| obondarev | @slaweq, nice, and I still scratching my head on how could those patches be related to ports policy rules.. | 10:10 |
| slaweq | maybe they are not, idk :) | 10:11 |
| *** yamamoto has joined #openstack-neutron | 10:19 | |
| *** rcernin has quit IRC | 10:23 | |
| slaweq | obondarev: I don't think those logs are problem really | 10:24 |
| slaweq | I see them also when listing ports is faster | 10:25 |
| obondarev | yep | 10:28 |
| obondarev | but anyway seems all this time we're in PolicyHook.after() func | 10:29 |
| obondarev | so probably some policy checks become slower | 10:29 |
| slaweq | obondarev: yes | 10:30 |
| slaweq | and I also think that it may not be one single commit responsible for that | 10:30 |
| *** zzzeek has quit IRC | 10:30 | |
| *** zzzeek has joined #openstack-neutron | 10:32 | |
| obondarev | @slaweq, indeed, because rally job would stop stop that commit | 10:32 |
| obondarev | also those rally failures include 2 fails for list networks | 10:33 |
| *** macz_ has joined #openstack-neutron | 10:33 | |
| obondarev | @slaweq, https://38c22b82d2b68d3a8475-a78f202f1ef95696b139bf8dd6cecb5d.ssl.cf5.rackcdn.com/772817/6/check/neutron-rally-task/0d49aa0/results/report.html#/NeutronNetworks.create_and_list_networks/failures | 10:34 |
| *** spatel has joined #openstack-neutron | 10:36 | |
| slaweq | obondarev: yes, I saw it | 10:37 |
| slaweq | thx | 10:38 |
| slaweq | I reverted all "secure rbac" patches locally | 10:38 |
| slaweq | and list of ports tooks my about 10 seconds (couple of tousands of ports) | 10:38 |
| slaweq | with all patches it was about 1 minute or more | 10:38 |
| slaweq | :/ | 10:38 |
| *** hemanth_n has joined #openstack-neutron | 10:38 | |
| slaweq | seems like this secure rbac causes some perf degradation | 10:38 |
| *** macz_ has quit IRC | 10:39 | |
| *** spatel has quit IRC | 10:40 | |
| obondarev | @slaweq, ah, so each secure-rbac patch adds to processing time, right? | 10:45 |
| slaweq | yes | 10:46 |
| obondarev | @slaweq, that would be interesting to debug :) | 10:46 |
| slaweq | obondarev: yes | 10:46 |
| slaweq | I will also ask lbragstad if he maybe saw something like that already in other projects | 10:46 |
| openstackgerrit | Lucas Alvares Gomes proposed openstack/neutron master: DO NOT REVIEW: OVN check slow job failures https://review.opendev.org/c/openstack/neutron/+/772341 | 10:47 |
| obondarev | @slaweq, cool to see rally job doing its job | 10:48 |
| *** hemanth_n has quit IRC | 10:49 | |
| slaweq | yes, it is now | 10:49 |
| slaweq | :) | 10:49 |
| obondarev | at least some good news | 10:49 |
| *** baojg has joined #openstack-neutron | 10:51 | |
| *** zzzeek has quit IRC | 10:54 | |
| *** zzzeek has joined #openstack-neutron | 10:55 | |
| *** baojg has quit IRC | 10:58 | |
| *** benj_ has joined #openstack-neutron | 11:00 | |
| *** xarlos has quit IRC | 11:00 | |
| openstackgerrit | Merged openstack/neutron stable/victoria: [OVN] Ensure metadata checksum https://review.opendev.org/c/openstack/neutron/+/772867 | 11:01 |
| *** gcheresh has joined #openstack-neutron | 11:01 | |
| *** xarlos has joined #openstack-neutron | 11:06 | |
| *** raildo has joined #openstack-neutron | 11:13 | |
| *** bengates_ has quit IRC | 11:28 | |
| *** bengates has joined #openstack-neutron | 11:29 | |
| *** rcernin has joined #openstack-neutron | 11:33 | |
| *** bengates_ has joined #openstack-neutron | 11:37 | |
| *** bengates_ has quit IRC | 11:39 | |
| *** bengates_ has joined #openstack-neutron | 11:39 | |
| *** bengates has quit IRC | 11:40 | |
| *** gcheresh has quit IRC | 11:49 | |
| openstackgerrit | Rodolfo Alonso proposed openstack/neutron master: Fix typo in "DhcpAgent.enable_isolated_metadata_proxy" https://review.opendev.org/c/openstack/neutron/+/773017 | 11:52 |
| *** bengates_ has quit IRC | 11:53 | |
| *** bengates has joined #openstack-neutron | 11:53 | |
| openstackgerrit | Lucas Alvares Gomes proposed openstack/neutron master: DO NOT REVIEW: OVN check slow job failures https://review.opendev.org/c/openstack/neutron/+/772341 | 11:54 |
| *** dviroel has joined #openstack-neutron | 11:54 | |
| *** sapd__x has quit IRC | 11:55 | |
| *** ociuhandu has quit IRC | 12:22 | |
| *** ociuhandu has joined #openstack-neutron | 12:23 | |
| *** ociuhandu has quit IRC | 12:27 | |
| *** mgariepy has quit IRC | 12:28 | |
| *** ociuhandu has joined #openstack-neutron | 12:30 | |
| *** elvira has quit IRC | 12:31 | |
| *** rcernin has quit IRC | 12:32 | |
| *** jpena is now known as jpena|lunch | 12:33 | |
| *** macz_ has joined #openstack-neutron | 12:34 | |
| *** ociuhandu has quit IRC | 12:35 | |
| *** bengates has quit IRC | 12:37 | |
| *** spatel has joined #openstack-neutron | 12:37 | |
| *** bengates has joined #openstack-neutron | 12:38 | |
| *** macz_ has quit IRC | 12:39 | |
| *** ccamposr__ has joined #openstack-neutron | 12:40 | |
| *** ratailor has quit IRC | 12:40 | |
| *** spatel has quit IRC | 12:41 | |
| *** ccamposr has quit IRC | 12:43 | |
| *** ociuhandu has joined #openstack-neutron | 12:44 | |
| *** ociuhandu has quit IRC | 12:49 | |
| *** elvira has joined #openstack-neutron | 12:52 | |
| *** yamamoto has quit IRC | 12:53 | |
| *** rm_work has quit IRC | 13:00 | |
| *** jamesdenton has quit IRC | 13:00 | |
| *** thedac has quit IRC | 13:00 | |
| *** mnaser has quit IRC | 13:00 | |
| *** jrosser has quit IRC | 13:00 | |
| *** Jeffrey4l has quit IRC | 13:00 | |
| *** rm_work has joined #openstack-neutron | 13:01 | |
| *** jamesdenton has joined #openstack-neutron | 13:01 | |
| *** mnaser has joined #openstack-neutron | 13:01 | |
| *** thedac has joined #openstack-neutron | 13:01 | |
| *** jrosser has joined #openstack-neutron | 13:01 | |
| *** Jeffrey4l has joined #openstack-neutron | 13:01 | |
| *** ociuhandu has joined #openstack-neutron | 13:01 | |
| *** benj_ has quit IRC | 13:02 | |
| *** liuyulong has quit IRC | 13:03 | |
| *** fyx has quit IRC | 13:03 | |
| *** f0o has quit IRC | 13:03 | |
| *** andy_ has quit IRC | 13:03 | |
| *** janno has quit IRC | 13:03 | |
| *** zigo has quit IRC | 13:03 | |
| *** sorrison has quit IRC | 13:03 | |
| *** liuyulong has joined #openstack-neutron | 13:03 | |
| *** fyx has joined #openstack-neutron | 13:03 | |
| *** f0o has joined #openstack-neutron | 13:03 | |
| *** andy_ has joined #openstack-neutron | 13:03 | |
| *** janno has joined #openstack-neutron | 13:03 | |
| *** zigo has joined #openstack-neutron | 13:03 | |
| *** sorrison has joined #openstack-neutron | 13:03 | |
| *** benj_ has joined #openstack-neutron | 13:04 | |
| *** johanssone has quit IRC | 13:04 | |
| *** johanssone has joined #openstack-neutron | 13:05 | |
| *** benj_ has quit IRC | 13:05 | |
| *** benj_ has joined #openstack-neutron | 13:06 | |
| *** ociuhandu has quit IRC | 13:08 | |
| *** ociuhandu has joined #openstack-neutron | 13:08 | |
| *** ociuhandu has quit IRC | 13:08 | |
| *** ociuhandu has joined #openstack-neutron | 13:09 | |
| *** ociuhandu has quit IRC | 13:14 | |
| *** inflatador has joined #openstack-neutron | 13:19 | |
| *** yamamoto has joined #openstack-neutron | 13:21 | |
| *** sapd__x has joined #openstack-neutron | 13:25 | |
| *** yamamoto has quit IRC | 13:26 | |
| *** mgariepy has joined #openstack-neutron | 13:27 | |
| *** zzzeek has quit IRC | 13:29 | |
| *** jpena|lunch is now known as jpena | 13:30 | |
| *** zzzeek has joined #openstack-neutron | 13:31 | |
| *** ociuhandu has joined #openstack-neutron | 13:33 | |
| *** tmorin has quit IRC | 13:34 | |
| openstackgerrit | Lucas Alvares Gomes proposed openstack/neutron master: [OVN][Zuul] Copy OVN logs and databases https://review.opendev.org/c/openstack/neutron/+/773032 | 13:35 |
| *** yamamoto has joined #openstack-neutron | 13:37 | |
| *** ociuhandu has quit IRC | 13:38 | |
| openstackgerrit | Lucas Alvares Gomes proposed openstack/neutron master: [OVN][Zuul] Copy OVN logs and databases https://review.opendev.org/c/openstack/neutron/+/773032 | 13:39 |
| *** spatel has joined #openstack-neutron | 13:51 | |
| *** ociuhandu has joined #openstack-neutron | 13:52 | |
| *** zzzeek has quit IRC | 13:52 | |
| *** spatel has quit IRC | 13:53 | |
| *** zzzeek has joined #openstack-neutron | 13:56 | |
| *** ociuhandu has quit IRC | 13:58 | |
| *** mlavalle has joined #openstack-neutron | 13:58 | |
| *** ociuhandu has joined #openstack-neutron | 13:58 | |
| *** zzzeek has quit IRC | 14:02 | |
| *** zzzeek has joined #openstack-neutron | 14:03 | |
| *** masterpe has joined #openstack-neutron | 14:13 | |
| *** bengates_ has joined #openstack-neutron | 14:15 | |
| *** bengates_ has quit IRC | 14:15 | |
| *** bengates has quit IRC | 14:15 | |
| *** bengates has joined #openstack-neutron | 14:15 | |
| *** nweinber has joined #openstack-neutron | 14:21 | |
| *** gcheresh has joined #openstack-neutron | 14:24 | |
| mgariepy | ralonsoh, do you have a minute ? | 14:45 |
| mgariepy | i found where the issue is for the lost ovs flow but i'm not quite sure how to fix it. | 14:46 |
| ralonsoh | mgariepy, sure, what is happening with those flows? | 14:46 |
| ralonsoh | did you see, in the code, how those flows are being lost? | 14:47 |
| mgariepy | when neutron agent starts it does init the ConfIPFlowManager https://github.com/openstack/neutron/blob/master/neutron/agent/linux/openvswitch_firewall/firewall.py#L480 | 14:48 |
| mgariepy | when ovs gets disconnected, all the flows are lost. the _initialize_firewall does set all the flows under the OVSFirewallDriver class. but since the ConfIPFlowManager is not re-initialiez the object still contains the flows so they are not re-added since it thinks they are already there. | 14:50 |
| ralonsoh | ahhhhhh | 14:50 |
| ralonsoh | ok, perfect | 14:50 |
| mgariepy | so you know how to fix it ? | 14:50 |
| mgariepy | ;) haha | 14:50 |
| ralonsoh | mgariepy, in the ovs agent we know if the OVS service is down | 14:51 |
| ralonsoh | https://github.com/openstack/neutron/blob/master/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py#L377 | 14:52 |
| ralonsoh | in this case, we can consider resetting "conj_ip_manager" | 14:52 |
| ralonsoh | of course, this is just a quick idea I had | 14:52 |
| ralonsoh | when OVS is restarted, we re-write all flows again | 14:53 |
| ralonsoh | like when we start the OVS agent | 14:53 |
| ralonsoh | that could be an option | 14:53 |
| ralonsoh | of course, you should also check other "OVSFirewallDriver" class variables that should be re-initializated | 14:53 |
| mgariepy | another one would be to look at ovsflows and see if we need to add or modify something. | 14:54 |
| ralonsoh | if needed | 14:54 |
| mgariepy | currently it rely on the tought that the obect and the ovs data are the same. but it might not be always true. | 14:55 |
| ralonsoh | mgariepy, take a look at the code, in the OVS agent, when the OVS is restarted | 14:55 |
| ralonsoh | https://github.com/openstack/neutron/blob/master/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py#L2607 | 14:55 |
| ralonsoh | https://github.com/openstack/neutron/blob/0fb63f729751d7e62ac107d3275d440632936978/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py#L2523 | 14:56 |
| ralonsoh | we re-write everything | 14:56 |
| *** yamamoto has quit IRC | 14:57 | |
| *** ociuhandu has quit IRC | 15:01 | |
| *** ociuhandu has joined #openstack-neutron | 15:02 | |
| *** ociuhandu has quit IRC | 15:08 | |
| *** ociuhandu has joined #openstack-neutron | 15:11 | |
| *** ociuhandu has quit IRC | 15:16 | |
| *** ociuhandu has joined #openstack-neutron | 15:20 | |
| *** gcheresh has quit IRC | 15:20 | |
| *** obondarev has quit IRC | 15:21 | |
| *** ociuhandu has quit IRC | 15:21 | |
| *** ociuhandu has joined #openstack-neutron | 15:22 | |
| openstackgerrit | Flavio Fernandes proposed openstack/neutron master: [DNM] [WIP] Please ignore -- [OVN] security group logging support https://review.opendev.org/c/openstack/neutron/+/768129 | 15:23 |
| *** rpittau is now known as rpittau|afk | 15:27 | |
| *** sapd__x has quit IRC | 15:34 | |
| slaweq | lbragstad: hi, are You around? | 15:48 |
| slaweq | lbragstad: we found out that we have problem with rally job and listing of ports and networks | 15:49 |
| slaweq | lbragstad: https://bugs.launchpad.net/neutron/+bug/1913718 | 15:49 |
| openstack | Launchpad bug 1913718 in neutron "rally ci job is unstable - port list takes very long time" [Critical,Confirmed] - Assigned to Slawek Kaplonski (slaweq) | 15:49 |
| slaweq | and it is related to the secure-rbac | 15:49 |
| slaweq | I found out that most of the slow down happens in https://github.com/openstack/oslo.policy/blob/master/oslo_policy/policy.py#L641 | 15:50 |
| slaweq | did You saw something like that before? | 15:50 |
| lbragstad | slaweq interesting, i haven't seen a regression like that with the policy engine when we updated the scopes in keystone | 15:54 |
| lbragstad | i don't think gmann saw anything like that with nova? | 15:54 |
| lbragstad | or at least i didn't hear about it | 15:54 |
| *** macz_ has joined #openstack-neutron | 15:55 | |
| *** macz_ has quit IRC | 15:55 | |
| slaweq | lbragstad: it's clearly in that place in code | 15:55 |
| slaweq | but I don't know yet why it happens like that | 15:55 |
| slaweq | I was hoping that maybe You will know what we are doing wrong | 15:55 |
| lbragstad | interesting... i assume i can recreate this on devstack, yeah? | 15:55 |
| slaweq | but tbh regression in our case is huge | 15:55 |
| slaweq | yes, I have in on devstack | 15:55 |
| lbragstad | cool - i just stood up a fresh deployment yesterday, so i'll use it to recreate this | 15:56 |
| slaweq | but I have many ports created | 15:56 |
| lbragstad | which policy is it? | 15:56 |
| slaweq | so, I'm using latest neutron code which has about 12 patches related to policies merged | 15:56 |
| lbragstad | ok | 15:57 |
| slaweq | and I noticed that more we have migrated, slower api is | 15:57 |
| lbragstad | i have a meeting coming up, but i'll focus on neutron this afternoon | 15:58 |
| zigo | ralonsoh: slaweq: Hi there! Would any of you know how to fix eventlet with Py 3.9, as per the thread I started? | 15:58 |
| slaweq | lbragstad: thx a lot | 15:58 |
| ralonsoh | zigo, not yet | 15:58 |
| lbragstad | slaweq if you have a paste or anything like that describing the slow down, feel free to send them my way | 15:58 |
| slaweq | zigo: sorry I didn't check Your thread at all, I didn't had time for it | 15:58 |
| zigo | ralonsoh: You're on it? | 15:58 |
| slaweq | lbragstad: I will send email to ML, is that ok for You? | 15:58 |
| ralonsoh | zigo not really, I just spent some time | 15:58 |
| lbragstad | slaweq yep - that works for me | 15:59 |
| slaweq | lbragstad++ thx a lot | 15:59 |
| slaweq | I will sum up my findings and will send email shortly | 15:59 |
| zigo | ralonsoh: Any hint I could investigate too? | 16:00 |
| ralonsoh | no sorry | 16:01 |
| zigo | ralonsoh: I tried a diff with ssl.py in 3.8 and 3.9, and there's almost no diff ... | 16:03 |
| zigo | -from socket import socket, AF_INET, SOCK_STREAM, create_connection | 16:04 |
| zigo | +from socket import socket, SOCK_STREAM, create_connection | 16:04 |
| gmann | slaweq: lbragstad oh, i have not seen that for nova yet. | 16:04 |
| zigo | no more ... | 16:04 |
| gmann | slaweq lbragstad _handle_deprecated_rule() does not do much just some if condition and append rule's check | 16:05 |
| gmann | slaweq: lbragstad and i think warning are disabled now in neutron right? | 16:05 |
| slaweq | gmann: let me summarize all and send email first | 16:08 |
| gmann | ok | 16:09 |
| *** mgariepy has quit IRC | 16:14 | |
| slaweq | gmann: ok, I sent email | 16:14 |
| gmann | slaweq: thanks i will check | 16:15 |
| *** lpetrut has quit IRC | 16:15 | |
| slaweq | I just tested once again, just commenting call to this one method in https://github.com/openstack/oslo.policy/blob/e103baa002e54303b08630c436dfc7b0b8a013de/oslo_policy/policy.py#L641 made api as fast as without those secure-rbac policies | 16:15 |
| slaweq | so it's clearly something there | 16:15 |
| slaweq | but I don't know what | 16:15 |
| slaweq | gmann: lbragstad: I think I found something | 16:26 |
| slaweq | e.g. in nova, when enforcer is initialized, it loads rule https://github.com/openstack/nova/blob/master/nova/policy.py#L93 | 16:26 |
| slaweq | it is not setting force_reload=True | 16:27 |
| slaweq | so it's set to False | 16:27 |
| slaweq | and in oslo_policy this sets "self.use_conf = False" https://github.com/openstack/oslo.policy/blob/e103baa002e54303b08630c436dfc7b0b8a013de/oslo_policy/policy.py#L597 | 16:27 |
| slaweq | so later use_conf = False | 16:27 |
| slaweq | in Neutron we are loading rules with force_reload=True | 16:28 |
| slaweq | so enforcer.use_conf is set to True all the time | 16:28 |
| slaweq | and due to that it loads all rules all the time | 16:28 |
| slaweq | and that's why it's so slow | 16:28 |
| gmann | slaweq: use_conf is true in nova case too https://github.com/openstack/nova/blob/master/nova/policy.py#L58 | 16:38 |
| openstackgerrit | Lajos Katona proposed openstack/neutron-specs master: BFD support for Neutron https://review.opendev.org/c/openstack/neutron-specs/+/767337 | 16:39 |
| slaweq | gmann: but here https://github.com/openstack/nova/blob/master/nova/policy.py#L93 You call enforcer without force_reload | 16:39 |
| slaweq | so force_reload=False (default) | 16:39 |
| slaweq | and that will be set for use_conf in oslo_policy Enforcer | 16:39 |
| *** mgariepy has joined #openstack-neutron | 16:40 | |
| slaweq | ahh, ok | 16:41 |
| slaweq | wait | 16:41 |
| slaweq | I missed "if" there | 16:41 |
| slaweq | gmann: so, I didn't found anything :/ | 16:42 |
| gmann | slaweq: yeah only difference force_reload makes if you load_rule more than once with same policy enforcer | 16:43 |
| slaweq | gmann: correct | 16:43 |
| slaweq | use_conf in nova is also True :/ | 16:43 |
| slaweq | so I didn't found anything really | 16:43 |
| gmann | force_reload=True then always ready from file and set in cache otherwise get from cache if availaible | 16:43 |
| slaweq | I think I need to take a break today | 16:44 |
| gmann | :) its friday and much needed :) | 16:44 |
| slaweq | gmann: yeah, and this week wasn't easy :) | 16:44 |
| slaweq | I will get back to this on Monday morning | 16:44 |
| gmann | I will try to check it sometime sooner but cannot commit for today. | 16:44 |
| slaweq | or maybe You or lbragstad will find something there in meantime | 16:44 |
| slaweq | gmann: thx a lot for any help | 16:45 |
| gmann | slaweq: sure, take rest and have nice weekend | 16:45 |
| slaweq | thx, You too gmann | 16:45 |
| *** lajoskatona has quit IRC | 16:57 | |
| *** lucasagomes has quit IRC | 16:57 | |
| *** jlibosva has quit IRC | 16:58 | |
| *** tesseract has quit IRC | 17:10 | |
| *** tesseract has joined #openstack-neutron | 17:21 | |
| openstackgerrit | Rodolfo Alonso proposed openstack/neutron master: [WIP] Process DHCP events in order if related https://review.opendev.org/c/openstack/neutron/+/773160 | 17:22 |
| openstackgerrit | Rodolfo Alonso proposed openstack/neutron stable/ussuri: [OVN] Ensure metadata checksum https://review.opendev.org/c/openstack/neutron/+/772868 | 17:24 |
| *** ociuhandu has quit IRC | 17:48 | |
| *** ociuhandu has joined #openstack-neutron | 17:48 | |
| *** ociuhandu has quit IRC | 17:53 | |
| *** jpena is now known as jpena|off | 17:57 | |
| *** elvira has quit IRC | 18:04 | |
| openstackgerrit | Rodolfo Alonso proposed openstack/neutron master: Remove rootwrap execution (2) https://review.opendev.org/c/openstack/neutron/+/772068 | 18:05 |
| *** ociuhandu has joined #openstack-neutron | 18:06 | |
| *** ociuhandu has quit IRC | 18:12 | |
| *** ralonsoh has quit IRC | 18:19 | |
| *** ociuhandu has joined #openstack-neutron | 18:19 | |
| *** ociuhandu has quit IRC | 18:20 | |
| *** ociuhandu has joined #openstack-neutron | 18:20 | |
| *** bengates has quit IRC | 18:25 | |
| *** ociuhandu has quit IRC | 18:25 | |
| *** bengates has joined #openstack-neutron | 18:25 | |
| openstackgerrit | Marc Gariépy proposed openstack/neutron master: Fix losses of ovs flows when ovs is restarted https://review.opendev.org/c/openstack/neutron/+/773165 | 18:26 |
| mgariepy | ralonsoh, can you check this quickly ?^^ | 18:26 |
| *** ociuhandu has joined #openstack-neutron | 18:59 | |
| *** ociuhandu has quit IRC | 19:03 | |
| openstackgerrit | Merged openstack/neutron stable/victoria: Limit usage of resources in the fullstack tests job https://review.opendev.org/c/openstack/neutron/+/770079 | 19:10 |
| *** tesseract has quit IRC | 19:18 | |
| *** bnemec has quit IRC | 19:22 | |
| openstackgerrit | Brian Haley proposed openstack/ovn-octavia-provider master: Delay string interpolations at logging calls https://review.opendev.org/c/openstack/ovn-octavia-provider/+/773175 | 19:26 |
| *** bnemec has joined #openstack-neutron | 19:30 | |
| *** bengates has quit IRC | 19:38 | |
| openstackgerrit | Brian Haley proposed openstack/ovn-octavia-provider master: Correctly set member operating status https://review.opendev.org/c/openstack/ovn-octavia-provider/+/765213 | 20:03 |
| openstackgerrit | Brian Haley proposed openstack/ovn-octavia-provider master: Start running the tempest API tests https://review.opendev.org/c/openstack/ovn-octavia-provider/+/752558 | 20:03 |
| *** rcernin has joined #openstack-neutron | 20:09 | |
| *** tbachman has quit IRC | 20:13 | |
| *** tbachman has joined #openstack-neutron | 20:14 | |
| *** macz_ has joined #openstack-neutron | 20:27 | |
| *** rcernin has quit IRC | 20:44 | |
| *** slaweq has quit IRC | 20:59 | |
| *** rcernin has joined #openstack-neutron | 21:10 | |
| *** raildo has quit IRC | 21:11 | |
| *** rcernin has quit IRC | 21:15 | |
| *** sean-k-mooney has quit IRC | 21:24 | |
| *** sean-k-mooney has joined #openstack-neutron | 21:26 | |
| *** rcernin has joined #openstack-neutron | 21:27 | |
| *** sean-k-mooney has quit IRC | 21:36 | |
| *** sean-k-mooney has joined #openstack-neutron | 21:36 | |
| *** nweinber has quit IRC | 22:07 | |
| *** ociuhandu has joined #openstack-neutron | 22:09 | |
| *** yamamoto has joined #openstack-neutron | 22:11 | |
| *** ociuhandu has quit IRC | 22:15 | |
| *** yamamoto has quit IRC | 22:21 | |
| *** yamamoto has joined #openstack-neutron | 22:25 | |
| *** stand has quit IRC | 22:26 | |
| *** armax has quit IRC | 22:43 | |
| *** rcernin has quit IRC | 22:47 | |
| *** ccamposr has joined #openstack-neutron | 22:47 | |
| *** ccamposr__ has quit IRC | 22:49 | |
| *** rcernin has joined #openstack-neutron | 23:04 | |
| *** rcernin has quit IRC | 23:19 | |
| *** zzzeek has quit IRC | 23:48 | |
| *** zzzeek has joined #openstack-neutron | 23:50 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!