| *** jmasud has joined #openstack-meeting | 00:14 | |
| *** jmasud has quit IRC | 00:23 | |
| *** jmasud has joined #openstack-meeting | 00:29 | |
| *** jmasud has quit IRC | 00:41 | |
| *** rcernin has quit IRC | 00:42 | |
| *** rcernin has joined #openstack-meeting | 00:43 | |
| *** jmasud has joined #openstack-meeting | 00:48 | |
| *** jmasud has quit IRC | 01:23 | |
| *** jmasud has joined #openstack-meeting | 01:26 | |
| *** _mlavalle_1 has quit IRC | 01:31 | |
| *** jmasud has quit IRC | 01:49 | |
| *** rcernin has quit IRC | 01:58 | |
| *** macz_ has quit IRC | 02:09 | |
| *** tinwood has quit IRC | 02:10 | |
| *** tinwood has joined #openstack-meeting | 02:13 | |
| *** rh-jelabarre has quit IRC | 02:33 | |
| *** macz_ has joined #openstack-meeting | 02:39 | |
| *** macz_ has quit IRC | 02:45 | |
| *** jmasud has joined #openstack-meeting | 02:45 | |
| *** rcernin has joined #openstack-meeting | 02:57 | |
| *** jmasud has quit IRC | 02:58 | |
| *** jmasud has joined #openstack-meeting | 03:05 | |
| *** jmasud has quit IRC | 03:09 | |
| *** rfolco has quit IRC | 03:14 | |
| *** jmasud has joined #openstack-meeting | 03:16 | |
| *** jmasud has quit IRC | 03:20 | |
| *** jmasud has joined #openstack-meeting | 03:21 | |
| *** jmasud has quit IRC | 03:25 | |
| *** jmasud has joined #openstack-meeting | 03:27 | |
| *** jmasud has quit IRC | 03:29 | |
| *** rcernin has quit IRC | 03:38 | |
| *** jmasud has joined #openstack-meeting | 03:39 | |
| *** rbudden has joined #openstack-meeting | 03:46 | |
| *** armax has quit IRC | 03:47 | |
| *** jmasud has quit IRC | 03:50 | |
| *** manpreet has quit IRC | 03:58 | |
| *** jmasud has joined #openstack-meeting | 04:03 | |
| *** rcernin has joined #openstack-meeting | 04:05 | |
| *** jmasud has quit IRC | 04:13 | |
| *** hyunsikyang__ has joined #openstack-meeting | 04:20 | |
| *** hyunsikyang has quit IRC | 04:23 | |
| *** jmasud has joined #openstack-meeting | 04:27 | |
| *** jmasud has quit IRC | 04:30 | |
| *** jmasud has joined #openstack-meeting | 04:33 | |
| *** jmasud has quit IRC | 04:35 | |
| *** jmasud has joined #openstack-meeting | 04:39 | |
| *** jmasud has quit IRC | 04:43 | |
| *** psachin has joined #openstack-meeting | 04:52 | |
| *** jmasud has joined #openstack-meeting | 04:53 | |
| *** jmasud has quit IRC | 05:02 | |
| *** gyee has quit IRC | 05:24 | |
| *** evrardjp has quit IRC | 05:33 | |
| *** evrardjp has joined #openstack-meeting | 05:33 | |
| *** rbudden has quit IRC | 05:33 | |
| *** baojg has quit IRC | 05:34 | |
| *** baojg has joined #openstack-meeting | 05:35 | |
| *** rcernin has quit IRC | 05:39 | |
| *** ociuhandu has joined #openstack-meeting | 05:40 | |
| *** rcernin has joined #openstack-meeting | 05:42 | |
| *** ociuhandu has quit IRC | 06:00 | |
| *** manpreet has joined #openstack-meeting | 06:15 | |
| *** tosky has joined #openstack-meeting | 06:24 | |
| *** jmasud has joined #openstack-meeting | 06:35 | |
| *** jmasud has quit IRC | 06:36 | |
| *** jmasud has joined #openstack-meeting | 06:37 | |
| *** rcernin_ has joined #openstack-meeting | 06:53 | |
| *** rcernin has quit IRC | 06:54 | |
| *** rcernin_ has quit IRC | 06:59 | |
| *** hyunsikyang has joined #openstack-meeting | 07:06 | |
| *** hyunsikyang__ has quit IRC | 07:09 | |
| *** jmasud has quit IRC | 07:12 | |
| *** ociuhandu has joined #openstack-meeting | 07:13 | |
| *** ralonsoh has joined #openstack-meeting | 07:14 | |
| *** ociuhandu has joined #openstack-meeting | 07:14 | |
| *** imcsk8 has quit IRC | 07:14 | |
| *** imcsk8 has joined #openstack-meeting | 07:16 | |
| *** ociuhandu has quit IRC | 07:17 | |
| *** rpittau|afk is now known as rpittau | 07:20 | |
| *** slaweq has joined #openstack-meeting | 07:26 | |
| *** david-lyle has joined #openstack-meeting | 08:47 | |
| *** dklyle has quit IRC | 08:47 | |
| *** ociuhandu has joined #openstack-meeting | 08:58 | |
| *** david-lyle has quit IRC | 08:58 | |
| *** ociuhandu has quit IRC | 09:08 | |
| *** jawad_axd has joined #openstack-meeting | 09:22 | |
| *** ociuhandu has joined #openstack-meeting | 09:48 | |
| *** ociuhandu has quit IRC | 09:58 | |
| *** baojg has quit IRC | 10:15 | |
| *** dsariel has quit IRC | 10:28 | |
| *** icey has quit IRC | 10:30 | |
| *** icey has joined #openstack-meeting | 10:36 | |
| *** jawad_axd has quit IRC | 11:01 | |
| *** ociuhandu has joined #openstack-meeting | 11:03 | |
| *** ociuhandu has quit IRC | 11:27 | |
| *** takamatsu has quit IRC | 11:33 | |
| *** takamatsu has joined #openstack-meeting | 11:33 | |
| *** e0ne has joined #openstack-meeting | 11:35 | |
| *** dsariel has joined #openstack-meeting | 11:36 | |
| *** ociuhandu has joined #openstack-meeting | 11:44 | |
| *** ykatabam has joined #openstack-meeting | 11:48 | |
| *** dsariel has quit IRC | 11:48 | |
| *** ykatabam has quit IRC | 11:49 | |
| *** rfolco has joined #openstack-meeting | 11:59 | |
| *** baojg has joined #openstack-meeting | 12:03 | |
| *** jmasud has joined #openstack-meeting | 12:23 | |
| *** jmasud has quit IRC | 12:36 | |
| *** rh-jelabarre has joined #openstack-meeting | 13:09 | |
| *** rh-jelabarre has quit IRC | 13:13 | |
| *** rh-jelabarre has joined #openstack-meeting | 13:21 | |
| *** ociuhandu_ has joined #openstack-meeting | 13:21 | |
| *** raildo has joined #openstack-meeting | 13:24 | |
| *** ociuhandu has quit IRC | 13:25 | |
| *** mahatic has quit IRC | 13:39 | |
| *** obondarev has joined #openstack-meeting | 13:54 | |
| *** mlavalle has joined #openstack-meeting | 13:58 | |
| *** baojg has quit IRC | 13:58 | |
| *** baojg has joined #openstack-meeting | 13:59 | |
| slaweq | #startmeeting neutron_drivers | 14:00 |
|---|---|---|
| openstack | Meeting started Fri Nov 13 14:00:07 2020 UTC and is due to finish in 60 minutes. The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot. | 14:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 14:00 |
| *** openstack changes topic to " (Meeting topic: neutron_drivers)" | 14:00 | |
| *** lajoskatona has joined #openstack-meeting | 14:00 | |
| openstack | The meeting name has been set to 'neutron_drivers' | 14:00 |
| mlavalle | o/ | 14:00 |
| lajoskatona | Hi | 14:00 |
| slaweq | welcome everyone! | 14:00 |
| obondarev | hi | 14:00 |
| slaweq | lets wait few more minutes for other people to join | 14:01 |
| ralonsoh | hi | 14:01 |
| slaweq | ping haleyb: njohnston amotoki yamamoto | 14:02 |
| slaweq | maybe they will join us soon | 14:02 |
| mlavalle | don't pay them today | 14:04 |
| obondarev | :D | 14:04 |
| slaweq | LOL | 14:05 |
| haleyb | hi, sorry i'm late | 14:05 |
| slaweq | ok, lets start | 14:07 |
| slaweq | we quorum already | 14:07 |
| slaweq | and we have only one topic in the on demand agenda for today | 14:07 |
| slaweq | no new RFEs to discuss | 14:08 |
| slaweq | #topi On Demand | 14:08 |
| slaweq | #topic On Demand | 14:08 |
| *** openstack changes topic to "On Demand (Meeting topic: neutron_drivers)" | 14:08 | |
| slaweq | obondarev: topic was added by You so please go on :) | 14:08 |
| obondarev | yep, it's about https://bugs.launchpad.net/neutron/+bug/1887523 | 14:09 |
| openstack | Launchpad bug 1887523 in neutron "Deadlock detection code can be stale" [High,In progress] | 14:09 |
| obondarev | so let me briefly describe my point | 14:09 |
| obondarev | I think the bug with it's current title, description and suggested approach is not quite correct | 14:09 |
| obondarev | "Deadlock detection code can be stale" - don't think so | 14:10 |
| obondarev | "neutron has it's own implementation of it which is missing a bunch of deadlocks" - not True | 14:10 |
| obondarev | so indeed neutron has a few decorators on top of oslo.db one | 14:11 |
| lajoskatona | yeah perhaps "neutron has its own extra decorators around oslo decorator" | 14:11 |
| obondarev | but they were added for a reason | 14:11 |
| obondarev | for many reasons in fact | 14:11 |
| obondarev | neutron had a long history of fighting against DB errors in concurrent scenarios | 14:12 |
| obondarev | git blame may shed some light on it: https://github.com/openstack/neutron/blame/stable/rocky/neutron/db/api.py | 14:12 |
| obondarev | just a few examples | 14:12 |
| obondarev | https://bugs.launchpad.net/neutron/+bug/1596075 | 14:12 |
| openstack | Launchpad bug 1596075 in neutron "Neutron confused about overlapping subnet creation" [High,Fix released] - Assigned to Kevin Benton (kevinbenton) | 14:12 |
| obondarev | https://bugs.launchpad.net/neutron/+bug/1612798 | 14:12 |
| openstack | Launchpad bug 1612798 in neutron "Move db retry logic closer to where DB error occur" [Critical,Fix released] - Assigned to Kevin Benton (kevinbenton) | 14:12 |
| obondarev | https://github.com/openstack/neutron/commit/948461c8b2fbeb30e4fa3a43cc523cff76327d4e | 14:13 |
| obondarev | most were quite a tricky ones | 14:13 |
| obondarev | so I don't think moving back to oslo.db decorator is the right thing to do | 14:14 |
| obondarev | there were not much work done in oslo.db: https://github.com/openstack/oslo.db/commits/master/oslo_db/api.py | 14:14 |
| mlavalle | you mean lately? | 14:14 |
| obondarev | right | 14:14 |
| obondarev | thanks | 14:14 |
| obondarev | in fact oslo.db retries were started by neutron folks) | 14:15 |
| mlavalle | LOL | 14:15 |
| obondarev | IIRC | 14:15 |
| obondarev | and in fact neutron retry logic now handles more cases | 14:15 |
| slaweq | but from what is written in the bug description we are missing some of deadlocks and not handling them properly | 14:15 |
| obondarev | some more info could be found here: https://github.com/openstack/neutron/blob/master/doc/source/contributor/internals/retries.rst | 14:15 |
| obondarev | slaweq, correct, so I think the bug should be about quota deadlocks | 14:16 |
| slaweq | maybe we should change this LP to something like "compare our implementation with oslo db and update our where it's needed" | 14:16 |
| obondarev | not about bad neutron retry logic | 14:16 |
| mlavalle | mhhhhh.... | 14:17 |
| obondarev | at least the bug should clearly show where neutron retries are bad | 14:17 |
| *** macz_ has joined #openstack-meeting | 14:17 | |
| lajoskatona | yeah that 's perhaps missing | 14:17 |
| obondarev | as if we just start replacing retry_if_session_inactive - we may got regressions | 14:17 |
| mlavalle | in principle I might agree wioth you obondarev. However, I want to point out that Mohammed is the CEO of a big operator | 14:17 |
| obondarev | wjich might be hard to spot | 14:17 |
| lajoskatona | if I undrstand well from summit generally there are scaling issues with neutron and one thing was that db issues can be behind that | 14:18 |
| obondarev | even with Loki service plugin | 14:18 |
| mlavalle | so at the very least I would like to get more input from mnaser and hear more about his point of view | 14:18 |
| obondarev | so in the bug comment and proposed patch I agree that quota retries have issues | 14:18 |
| ralonsoh | let me remember you that we are still migrating to the new DB engine facade | 14:19 |
| ralonsoh | that will remove subtransactions | 14:19 |
| mlavalle | that is also true | 14:19 |
| obondarev | but please let's not just replace retry_if_session_inactive for oslo-db.wrap_db_retries all over neutron | 14:19 |
| ralonsoh | and other issues related to mixing both facades | 14:19 |
| ralonsoh | so I recommend start investigating this after completing the migration | 14:20 |
| obondarev | ralonsoh, so there is no evidence what's exactly wrong with quota deadlock issue, right? | 14:20 |
| ralonsoh | I can't tell, sorry | 14:20 |
| ralonsoh | but | 14:20 |
| ralonsoh | with the new facade at least we know each thread has one single transaction without subtransactions | 14:21 |
| ralonsoh | and the context is unique | 14:21 |
| ralonsoh | (per thread) | 14:21 |
| ralonsoh | e.g.: https://review.opendev.org/#/c/715315/20/neutron/tests/functional/services/portforwarding/test_port_forwarding.py | 14:21 |
| patchbot | patch 715315 - neutron - Finish the new DB engine facade migration - 20 patch sets | 14:21 |
| *** macz_ has quit IRC | 14:22 | |
| obondarev | great, so I'd suggest we put this bug on hold for now, and after new facade is there - try to reproduce and investigate | 14:22 |
| ralonsoh | +1 to this | 14:22 |
| mlavalle | I propose three steps: | 14:22 |
| mlavalle | 1) finish the migration to the new db engine facade, as indicated by ralonsoh and obondarev | 14:23 |
| mlavalle | 2) Limit this bug, for the time being, to the quota issue | 14:23 |
| mlavalle | 3) Seek more input from mnaser. He may giove us some good insights, given his operational experience | 14:23 |
| obondarev | +1 | 14:24 |
| slaweq | mlavalle++ | 14:24 |
| lajoskatona | sounds good | 14:24 |
| ralonsoh | +1 to this proposal | 14:24 |
| slaweq | and also if there will be another similar issues to what we have now with quota, lets treat them separately as regular bugs | 14:24 |
| * mnaser is happy to chime in -- just ping me via email/ml (cc'd directly) or any other way :) | 14:24 | |
| ralonsoh | we can always increase the scope of 2) to other code places | 14:25 |
| mlavalle | thanks mnaser ! | 14:25 |
| njohnston | makes sense to me | 14:25 |
| slaweq | ok, so I think we have agreement on that for now | 14:26 |
| slaweq | I will sum it up in the LP comment after the meeting | 14:26 |
| lajoskatona | thanks slaweq | 14:26 |
| obondarev | thanks slaweq | 14:26 |
| slaweq | so I think this topic is done for today, right? | 14:27 |
| obondarev | + | 14:27 |
| mlavalle | I think so | 14:27 |
| slaweq | do You have anything else You want to discuss today? if not, I will give You 30 minutes back :) | 14:28 |
| mlavalle | yaaay, weekened is now a bit closer! | 14:28 |
| slaweq | LOL | 14:28 |
| slaweq | for me it's almost there :) | 14:28 |
| lajoskatona | it's time :-) | 14:28 |
| obondarev | same here) | 14:28 |
| slaweq | ok, thx for attending the meeting | 14:28 |
| ralonsoh | bye! enjoy the weekend | 14:29 |
| slaweq | and have a great weekend! | 14:29 |
| njohnston | o/ | 14:29 |
| obondarev | bye! | 14:29 |
| lajoskatona | Bye | 14:29 |
| mlavalle | o/ | 14:29 |
| slaweq | #endmeeting | 14:29 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 14:29 | |
| openstack | Meeting ended Fri Nov 13 14:29:09 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 14:29 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/neutron_drivers/2020/neutron_drivers.2020-11-13-14.00.html | 14:29 |
| *** lajoskatona has left #openstack-meeting | 14:29 | |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/neutron_drivers/2020/neutron_drivers.2020-11-13-14.00.txt | 14:29 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/neutron_drivers/2020/neutron_drivers.2020-11-13-14.00.log.html | 14:29 |
| haleyb | o/ | 14:29 |
| *** dmacpher has joined #openstack-meeting | 14:33 | |
| *** ociuhandu_ has quit IRC | 14:42 | |
| *** dsariel has joined #openstack-meeting | 14:42 | |
| *** ociuhandu has joined #openstack-meeting | 14:43 | |
| *** lpetrut has joined #openstack-meeting | 14:49 | |
| *** rpittau is now known as rpittau|afk | 14:49 | |
| *** hemna has quit IRC | 14:50 | |
| *** hemna has joined #openstack-meeting | 14:51 | |
| *** ociuhandu has quit IRC | 15:06 | |
| *** ociuhandu has joined #openstack-meeting | 15:08 | |
| *** obondarev has quit IRC | 15:10 | |
| *** lpetrut has quit IRC | 15:12 | |
| *** ociuhandu has quit IRC | 15:24 | |
| *** ianychoi has joined #openstack-meeting | 15:30 | |
| *** TrevorV has joined #openstack-meeting | 15:31 | |
| *** bcafarel has quit IRC | 15:32 | |
| *** manpreet has quit IRC | 15:34 | |
| *** bcafarel has joined #openstack-meeting | 15:36 | |
| *** dklyle has joined #openstack-meeting | 15:48 | |
| *** mlavalle has quit IRC | 15:53 | |
| *** mlavalle has joined #openstack-meeting | 15:55 | |
| *** armax has joined #openstack-meeting | 15:57 | |
| *** ociuhandu has joined #openstack-meeting | 16:02 | |
| *** ociuhandu has quit IRC | 16:09 | |
| *** takamatsu has quit IRC | 16:15 | |
| *** jmasud has joined #openstack-meeting | 16:26 | |
| *** baojg has quit IRC | 16:28 | |
| *** baojg has joined #openstack-meeting | 16:30 | |
| *** macz_ has joined #openstack-meeting | 16:36 | |
| *** jmasud has quit IRC | 16:42 | |
| *** takamatsu has joined #openstack-meeting | 16:46 | |
| *** ociuhandu has joined #openstack-meeting | 16:51 | |
| *** jmasud has joined #openstack-meeting | 16:58 | |
| *** hemna has quit IRC | 16:58 | |
| *** TrevorV has quit IRC | 16:58 | |
| *** hemna has joined #openstack-meeting | 16:59 | |
| *** hemna has quit IRC | 17:08 | |
| *** hemna has joined #openstack-meeting | 17:09 | |
| *** ociuhandu_ has joined #openstack-meeting | 17:15 | |
| *** ociuhandu has quit IRC | 17:17 | |
| *** ociuhandu_ has quit IRC | 17:19 | |
| *** jmasud has quit IRC | 17:26 | |
| *** ociuhandu has joined #openstack-meeting | 17:26 | |
| *** ociuhandu has quit IRC | 17:31 | |
| *** e0ne has quit IRC | 17:36 | |
| *** bnemec is now known as beekneemech | 17:37 | |
| *** psachin has quit IRC | 17:44 | |
| *** jmasud has joined #openstack-meeting | 17:46 | |
| *** mlavalle has quit IRC | 17:58 | |
| *** mlavalle has joined #openstack-meeting | 18:05 | |
| *** ralonsoh has quit IRC | 18:34 | |
| *** gyee has joined #openstack-meeting | 18:54 | |
| *** jmasud has quit IRC | 19:13 | |
| *** lifeless has quit IRC | 19:31 | |
| *** lifeless has joined #openstack-meeting | 19:47 | |
| *** slaweq has quit IRC | 20:03 | |
| *** rfolco has quit IRC | 21:10 | |
| *** ociuhandu has joined #openstack-meeting | 21:19 | |
| *** rcernin has joined #openstack-meeting | 21:19 | |
| *** jmasud has joined #openstack-meeting | 21:32 | |
| *** jmasud has quit IRC | 21:39 | |
| *** jmasud has joined #openstack-meeting | 21:42 | |
| *** ociuhandu has quit IRC | 21:45 | |
| *** jmasud has quit IRC | 21:48 | |
| *** jmasud has joined #openstack-meeting | 21:49 | |
| *** jmasud has quit IRC | 21:54 | |
| *** jmasud has joined #openstack-meeting | 22:15 | |
| *** jmasud has quit IRC | 22:32 | |
| *** jmasud has joined #openstack-meeting | 22:33 | |
| *** jmasud has quit IRC | 22:33 | |
| *** jmasud has joined #openstack-meeting | 22:40 | |
| *** rh-jelabarre has quit IRC | 22:43 | |
| *** jmasud has quit IRC | 22:58 | |
| *** jmasud has joined #openstack-meeting | 22:59 | |
| *** jmasud has quit IRC | 22:59 | |
| *** jmasud has joined #openstack-meeting | 23:09 | |
| *** armax has quit IRC | 23:10 | |
| *** armax has joined #openstack-meeting | 23:13 | |
| *** hemna has quit IRC | 23:19 | |
| *** hemna has joined #openstack-meeting | 23:20 | |
| *** corvus has quit IRC | 23:22 | |
| *** rfolco has joined #openstack-meeting | 23:22 | |
| *** jeblair has joined #openstack-meeting | 23:24 | |
| *** jeblair is now known as corvus | 23:26 | |
| *** rfolco has quit IRC | 23:26 | |
| *** tosky has quit IRC | 23:43 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!