| opendevreview | Miguel Lavalle proposed openstack/neutron master: Router flavors and service type for OVN https://review.opendev.org/c/openstack/neutron/+/883988 | 01:12 |
|---|---|---|
| opendevreview | Merged openstack/neutron master: Support rarp protocol for egress table https://review.opendev.org/c/openstack/neutron/+/661921 | 04:38 |
| opendevreview | Frode Nordahl proposed openstack/neutron-tempest-plugin master: Add Active Active L3 GW scenario test cases https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/898810 | 06:02 |
| *** elodilles_pto is now known as elodilles | 07:34 | |
| opendevreview | Slawek Kaplonski proposed openstack/neutron stable/2023.2: [Fullstack] Double check that agent is dead when it should be dead https://review.opendev.org/c/openstack/neutron/+/902881 | 07:46 |
| opendevreview | Slawek Kaplonski proposed openstack/neutron stable/2023.1: [Fullstack] Double check that agent is dead when it should be dead https://review.opendev.org/c/openstack/neutron/+/902882 | 07:47 |
| opendevreview | Slawek Kaplonski proposed openstack/neutron stable/zed: [Fullstack] Double check that agent is dead when it should be dead https://review.opendev.org/c/openstack/neutron/+/902883 | 07:47 |
| opendevreview | Merged openstack/neutron-tempest-plugin master: Test job for VPNaaS on OVN https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/847007 | 10:16 |
| opendevreview | Max proposed openstack/ovsdbapp master: add Port_Binding to lookup_table https://review.opendev.org/c/openstack/ovsdbapp/+/902878 | 10:47 |
| *** sfinucan is now known as stephenfin | 10:58 | |
| opendevreview | Felipe Figueroa Vergara proposed openstack/tap-as-a-service master: Enables external network mirroring https://review.opendev.org/c/openstack/tap-as-a-service/+/896515 | 12:06 |
| opendevreview | Felipe Figueroa Vergara proposed openstack/tap-as-a-service master: Enables external network mirroring https://review.opendev.org/c/openstack/tap-as-a-service/+/896515 | 12:07 |
| opendevreview | Felipe Figueroa Vergara proposed openstack/tap-as-a-service master: Enables external network mirroring https://review.opendev.org/c/openstack/tap-as-a-service/+/896515 | 12:08 |
| opendevreview | Felipe Figueroa Vergara proposed openstack/tap-as-a-service master: Enables external network mirroring https://review.opendev.org/c/openstack/tap-as-a-service/+/896515 | 12:12 |
| opendevreview | Felipe Figueroa Vergara proposed openstack/tap-as-a-service master: Enables external network mirroring https://review.opendev.org/c/openstack/tap-as-a-service/+/896515 | 12:12 |
| opendevreview | Felipe Figueroa Vergara proposed openstack/tap-as-a-service master: Enables external network mirroring https://review.opendev.org/c/openstack/tap-as-a-service/+/896515 | 12:14 |
| opendevreview | Felipe Figueroa Vergara proposed openstack/tap-as-a-service master: Enables external network mirroring https://review.opendev.org/c/openstack/tap-as-a-service/+/896515 | 12:17 |
| opendevreview | Felipe Figueroa Vergara proposed openstack/tap-as-a-service master: Enables external network mirroring https://review.opendev.org/c/openstack/tap-as-a-service/+/896515 | 12:18 |
| *** tobias-urdin9 is now known as tobias-urdin | 13:03 | |
| opendevreview | Michel Nederlof proposed openstack/neutron master: [OVN] Update lsp host id when virtual parent moves https://review.opendev.org/c/openstack/neutron/+/896883 | 14:07 |
| opendevreview | Felipe Figueroa Vergara proposed openstack/tap-as-a-service master: Enables external network mirroring https://review.opendev.org/c/openstack/tap-as-a-service/+/896515 | 14:17 |
| opendevreview | Felipe Figueroa Vergara proposed openstack/tap-as-a-service master: Enables external network mirroring https://review.opendev.org/c/openstack/tap-as-a-service/+/896515 | 14:18 |
| opendevreview | Felipe Figueroa Vergara proposed openstack/tap-as-a-service master: Enables external network mirroring https://review.opendev.org/c/openstack/tap-as-a-service/+/896515 | 14:19 |
| opendevreview | Felipe Figueroa Vergara proposed openstack/tap-as-a-service master: Enables external network mirroring https://review.opendev.org/c/openstack/tap-as-a-service/+/896515 | 14:19 |
| opendevreview | Felipe Figueroa Vergara proposed openstack/tap-as-a-service master: Enables external network mirroring https://review.opendev.org/c/openstack/tap-as-a-service/+/896515 | 14:20 |
| opendevreview | Felipe Figueroa Vergara proposed openstack/tap-as-a-service master: Enables external network mirroring https://review.opendev.org/c/openstack/tap-as-a-service/+/896515 | 14:21 |
| opendevreview | Merged openstack/neutron stable/2023.1: [Fullstack] Double check that agent is dead when it should be dead https://review.opendev.org/c/openstack/neutron/+/902882 | 16:47 |
| *** tobias-urdin34 is now known as tobias-urdin | 17:27 | |
| haleyb | otherwiseguy: question on ovsdb transations... sometimes running ovn-db-sync-util we see timeouts due to too many items in a transaction, for example 10K. Is it safe to split them into groups? I'm thinking some are maybe related and need to be together? | 17:38 |
| otherwiseguy | Yeah, it's one of those things where it's hard to really know what the unit of work should be. We need to group into transactions otherwise it takes *forever* to run. Doing everything in one can mean you blow your timeout, or if there is something wrong with 1/10k, it fails, etc. | 17:40 |
| otherwiseguy | If it were me, I'd just make the timeout for the connections in the db-sync util higher than normal runtime. | 17:41 |
| otherwiseguy | atomicity of the txn isn't super important since we're syncing to another source of truth anyway, so I don't think there's anything inherently wrong with breaking up txns. We don't normally do a lot of transaction ops that *have* to be in the same txn to work, but I'd have to audit to make sure. | 17:44 |
| haleyb | otherwiseguy: i've seen even high timeouts fail, like 600+ | 17:45 |
| otherwiseguy | Is that because a lot of other work is going on on the ovsdb-server at the same time or something? | 17:46 |
| otherwiseguy | haleyb: in general, we usually recommend people disable access to neutron API entirely when running db sync. | 17:46 |
| haleyb | otherwiseguy: i only have a black box sometimes, and the api is supposedly off | 17:47 |
| otherwiseguy | Mostly because adding ports, etc. while syncing isn't particularly safe. But performance could also play a part. | 17:47 |
| otherwiseguy | Hmm. | 17:47 |
| otherwiseguy | Where does it usually time out? ACLs? | 17:47 |
| haleyb | port dhcp options strangely | 17:48 |
| * otherwiseguy looks | 17:48 | |
| * otherwiseguy notes that vim brought him right to `_sync_port_dhcp_options()` | 17:49 | |
| haleyb | i was going to play with batching them like we did for agent sync code | 17:50 |
| opendevreview | Frode Nordahl proposed openstack/neutron-tempest-plugin master: Add Active Active L3 GW API test cases https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/897823 | 17:55 |
| opendevreview | Frode Nordahl proposed openstack/neutron-tempest-plugin master: Add Active Active L3 GW scenario test cases https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/898810 | 17:55 |
| otherwiseguy | haleyb: is this for something like "there is no OVN db, and we're syncing the whole neutron db to OVN" ? | 18:07 |
| haleyb | otherwiseguy: no, it's more "OVN has gone to shit and there are inconsistencies, let's run a sync" | 18:10 |
| otherwiseguy | haleyb: it just seems weird that it would be out of sync enough in normal operation to have 600+s worth of dhcp options to sync. | 18:11 |
| haleyb | otherwiseguy: is sure does, and these are all SetLSwitchPortCommand(). it could just another false positive | 18:12 |
| otherwiseguy | Part of me thinks we should just write a SyncDhcpOptionsCommand that just does all reads and writes in the command. | 18:14 |
| otherwiseguy | In general having Command objects that do everything you want to do is better than a bunch of separate commands that do a bunch of lookups by name, etc. that you probably don't need. | 18:15 |
| otherwiseguy | But there's kind of a lot to trace in this code to figure out everything that's going on. | 18:15 |
| otherwiseguy | but if it's timing out because the commit() is taking to long, that doesn't really matter. | 18:19 |
| otherwiseguy | s/to/too/ | 18:19 |
| otherwiseguy | As that would really just be taking too long *generating* the txn in python-ovs (either or both could be the culprit). Is there a bug w/ logs? | 18:20 |
| otherwiseguy | haleyb: ^ | 18:20 |
| haleyb | otherwiseguy: there is no bug at the moment, is there something i can collect to point fingers at what might be happening? | 18:23 |
| haleyb | otherwiseguy: the best i can say is commit() is timing out, so TimeoutException, sometimes see a weird _queue.Empty | 18:24 |
| haleyb | i can create one | 18:24 |
| otherwiseguy | I'm assuming there are a bunch of Running txn n=(x) command(idx=y)" log messages? Basically the 1st one of those and last one of those w/ timestamps, and then any errors or related logs around them. | 18:26 |
| otherwiseguy | it occurs to me that changing that to printing out idx=y/len(commands) might be useful info. | 18:27 |
| haleyb | otherwiseguy: i would have to get the ovsdb log, i only have the neutron side of things | 18:27 |
| otherwiseguy | that should be in the neutron debug logs. | 18:28 |
| haleyb | otherwiseguy: hmm, the sync tool seems to generate it's own log (or dumps to stdout) | 18:31 |
| otherwiseguy | oh, yeah, i'm dumb. :D | 18:37 |
| otherwiseguy | Maintenance task is when the sync stuff would show up in neutron. | 18:38 |
| otherwiseguy | haleyb: I cannot express the number of times I've cursed us from changing from OVN plugin to ML2 driver. | 18:47 |
| otherwiseguy | Dealing with syncing multiple databases is just terrible. | 18:48 |
| otherwiseguy | (I am going to ignore that that is basically what northd is also doing.) | 18:48 |
| opendevreview | Merged openstack/neutron master: [Trivial] speed up l3_db floating IP port janitor DB query https://review.opendev.org/c/openstack/neutron/+/888200 | 18:52 |
| opendevreview | Merged openstack/neutron stable/zed: [Fullstack] Double check that agent is dead when it should be dead https://review.opendev.org/c/openstack/neutron/+/902883 | 18:52 |
| haleyb | otherwiseguy: who designed this thing? :-p | 19:02 |
| otherwiseguy | Russell initially designed it as a plugin and was pressured to change it to an ml2 driver. I seem to remember him being like "ok, you all are the neutron experts, but ..." :) | 19:03 |
| haleyb | and thanks for looking, i'm almost done with my hack at splitting into smaller chunks | 19:04 |
| otherwiseguy | Granted, it's nice to be able to use other drivers w/o having to reimplement them, etc. but man I hate db syncing. | 19:04 |
| otherwiseguy | haleyb: if we can get logs from a run of the sync util, it'd be interesting to see the various "out of sync" messages etc. as well. | 19:20 |
| haleyb | otherwiseguy: i'll add what i can to the bug, but i don't see any messages regarding what it's syncing in that method, just a traceback | 19:35 |
| otherwiseguy | is debug enabled? | 19:35 |
| otherwiseguy | If so, I wonder if somehow the ovsdbapp vlog.use_python_logging() isn't called or something when done from the sync util? | 19:36 |
| haleyb | hah, maybe not, but can ask the next time we run to add --debug=True or whatever it is | 19:37 |
| otherwiseguy | s/logging/logger/ | 19:37 |
| * haleyb shrugs | 19:37 | |
| * haleyb takes off for a bit | 19:37 | |
| otherwiseguy | looks like it should be called as get_ovn_idls() calls it. | 19:38 |
| opendevreview | zhaobo proposed openstack/neutron stable/zed: [OVN] Prevent binding a virtual type port https://review.opendev.org/c/openstack/neutron/+/903104 | 20:26 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!