| *** dsneddon_ has quit IRC | 00:04 | |
| *** ivve has quit IRC | 00:33 | |
| openstackgerrit | zhanghao proposed openstack/neutron master: Make network support read and write separation https://review.opendev.org/677166 | 01:24 |
|---|---|---|
| *** ociuhandu has joined #openstack-neutron | 01:31 | |
| *** chenhaw has joined #openstack-neutron | 01:35 | |
| *** ociuhandu has quit IRC | 01:41 | |
| *** goldyfruit_ has quit IRC | 01:53 | |
| *** dsneddon_ has joined #openstack-neutron | 02:00 | |
| *** macz has joined #openstack-neutron | 02:05 | |
| *** dsneddon_ has quit IRC | 02:05 | |
| *** zhanglong has joined #openstack-neutron | 02:24 | |
| *** ociuhandu has joined #openstack-neutron | 02:26 | |
| *** ociuhandu has quit IRC | 02:30 | |
| *** ramishra has joined #openstack-neutron | 02:39 | |
| *** macz has quit IRC | 02:51 | |
| *** ociuhandu has joined #openstack-neutron | 03:00 | |
| *** ociuhandu has quit IRC | 03:07 | |
| *** dsneddon_ has joined #openstack-neutron | 04:01 | |
| *** dsneddon_ has quit IRC | 04:05 | |
| *** baojg has quit IRC | 04:22 | |
| *** ociuhandu has joined #openstack-neutron | 04:27 | |
| *** ociuhandu has quit IRC | 04:29 | |
| *** ociuhandu has joined #openstack-neutron | 04:30 | |
| *** ociuhandu has quit IRC | 04:36 | |
| *** ociuhandu has joined #openstack-neutron | 04:53 | |
| *** ociuhandu has quit IRC | 04:58 | |
| *** ociuhandu has joined #openstack-neutron | 04:58 | |
| *** ociuhandu has quit IRC | 05:03 | |
| *** gcheresh_ has joined #openstack-neutron | 05:12 | |
| *** gcheresh_ has quit IRC | 05:20 | |
| *** ratailor has joined #openstack-neutron | 05:23 | |
| *** ileixe has quit IRC | 05:29 | |
| *** gcheresh_ has joined #openstack-neutron | 05:30 | |
| *** ociuhandu has joined #openstack-neutron | 05:30 | |
| *** ileixe has joined #openstack-neutron | 05:32 | |
| *** ociuhandu has quit IRC | 05:35 | |
| *** ircuser-1 has quit IRC | 05:35 | |
| *** slaweq has joined #openstack-neutron | 05:47 | |
| *** gcheresh_ has quit IRC | 05:53 | |
| *** slaweq has quit IRC | 05:57 | |
| *** abaindur has joined #openstack-neutron | 05:58 | |
| *** dsneddon_ has joined #openstack-neutron | 06:02 | |
| *** abaindur has quit IRC | 06:05 | |
| *** dsneddon_ has quit IRC | 06:06 | |
| *** Luzi has joined #openstack-neutron | 06:11 | |
| *** awalende has joined #openstack-neutron | 06:16 | |
| *** awalende has quit IRC | 06:20 | |
| *** numans_ has joined #openstack-neutron | 06:44 | |
| *** ksambor has joined #openstack-neutron | 06:51 | |
| *** ileixe has quit IRC | 06:51 | |
| *** ileixe has joined #openstack-neutron | 06:52 | |
| *** abaindur has joined #openstack-neutron | 07:01 | |
| *** ociuhandu has joined #openstack-neutron | 07:02 | |
| *** rcernin has quit IRC | 07:04 | |
| *** ociuhandu has quit IRC | 07:06 | |
| *** abaindur has quit IRC | 07:07 | |
| *** ltomasbo has joined #openstack-neutron | 07:13 | |
| openstackgerrit | Oleg Bondarev proposed openstack/neutron master: L3 agent graceful shutdown https://review.opendev.org/693323 | 07:19 |
| *** rpittau|afk is now known as rpittau | 07:28 | |
| *** maciejjozefczyk has joined #openstack-neutron | 07:36 | |
| *** slaweq has joined #openstack-neutron | 07:38 | |
| *** igordc has joined #openstack-neutron | 07:58 | |
| *** tkajinam has quit IRC | 08:02 | |
| *** igordc has quit IRC | 08:02 | |
| *** dsneddon_ has joined #openstack-neutron | 08:03 | |
| *** dsneddon_ has quit IRC | 08:08 | |
| *** luksky has joined #openstack-neutron | 08:08 | |
| *** gcheresh_ has joined #openstack-neutron | 08:09 | |
| *** lajoskatona has joined #openstack-neutron | 08:11 | |
| *** tesseract has joined #openstack-neutron | 08:15 | |
| *** jlibosva has joined #openstack-neutron | 08:16 | |
| *** ociuhandu has joined #openstack-neutron | 08:22 | |
| *** ociuhandu has quit IRC | 08:29 | |
| *** ratailor_ has joined #openstack-neutron | 08:40 | |
| *** ratailor has quit IRC | 08:42 | |
| *** lucasagomes has joined #openstack-neutron | 08:45 | |
| *** ivve has joined #openstack-neutron | 08:46 | |
| *** jpena|off is now known as jpena | 08:49 | |
| *** ralonsoh has joined #openstack-neutron | 08:49 | |
| *** ociuhandu has joined #openstack-neutron | 09:04 | |
| openstackgerrit | Merged openstack/networking-ovn master: [vagrants] Move to Ubuntu 18.04 by default https://review.opendev.org/692790 | 09:06 |
| *** nanzha has joined #openstack-neutron | 09:10 | |
| *** yankcrime has left #openstack-neutron | 09:12 | |
| *** awalende has joined #openstack-neutron | 09:16 | |
| *** awalende has quit IRC | 09:18 | |
| openstackgerrit | Daniel Bengtsson proposed openstack/neutron master: Stop configuring install_command in tox. https://review.opendev.org/694568 | 09:21 |
| *** awalende has joined #openstack-neutron | 09:26 | |
| *** awalende has quit IRC | 09:30 | |
| openstackgerrit | Slawek Kaplonski proposed openstack/networking-bagpipe stable/train: bagpipe-bgp: cleanly ignore RTC route of unsupported type https://review.opendev.org/690395 | 09:36 |
| *** awalende has joined #openstack-neutron | 09:36 | |
| openstackgerrit | Slawek Kaplonski proposed openstack/networking-bagpipe stable/stein: bagpipe-bgp: cleanly ignore RTC route of unsupported type https://review.opendev.org/690396 | 09:37 |
| *** awalende has quit IRC | 09:37 | |
| *** ileixe has quit IRC | 09:40 | |
| *** ratailor__ has joined #openstack-neutron | 09:41 | |
| *** ileixe has joined #openstack-neutron | 09:41 | |
| *** ratailor_ has quit IRC | 09:43 | |
| openstackgerrit | Merged openstack/neutron-fwaas stable/stein: Fix list_entries for netlink_lib when running on py3 https://review.opendev.org/693834 | 09:45 |
| *** bobmel has joined #openstack-neutron | 09:45 | |
| *** ociuhandu has quit IRC | 09:57 | |
| *** ociuhandu has joined #openstack-neutron | 09:58 | |
| *** ociuhandu has quit IRC | 10:03 | |
| *** dsneddon_ has joined #openstack-neutron | 10:04 | |
| *** lennyb has quit IRC | 10:06 | |
| *** zhanglong has quit IRC | 10:06 | |
| *** dsneddon_ has quit IRC | 10:08 | |
| *** ileixe has quit IRC | 10:10 | |
| *** pcaruana has joined #openstack-neutron | 10:10 | |
| *** nanzha has quit IRC | 10:11 | |
| *** ileixe has joined #openstack-neutron | 10:11 | |
| *** nanzha has joined #openstack-neutron | 10:12 | |
| openstackgerrit | Lucas Alvares Gomes proposed openstack/networking-ovn master: Add support for virtual port type https://review.opendev.org/676223 | 10:23 |
| openstackgerrit | Merged openstack/neutron-fwaas stable/rocky: Fix list_entries for netlink_lib when running on py3 https://review.opendev.org/693835 | 10:32 |
| *** CeeMac has joined #openstack-neutron | 10:37 | |
| *** davidsha has joined #openstack-neutron | 10:42 | |
| openstackgerrit | Daniel Alvarez proposed openstack/networking-ovn stable/train: [metadata-agent] Fix issue with TLS/SSL connections https://review.opendev.org/694742 | 10:42 |
| openstackgerrit | Aditya Reddy Nagaram proposed openstack/neutron master: [WIP] Support for stateless security groups https://review.opendev.org/572767 | 10:44 |
| *** rcernin has joined #openstack-neutron | 10:52 | |
| *** chenhaw has quit IRC | 10:59 | |
| *** ociuhandu has joined #openstack-neutron | 11:02 | |
| openstackgerrit | Merged openstack/neutron stable/stein: Add extra unit test for get_cmdline_from_pid function https://review.opendev.org/694316 | 11:03 |
| *** luksky has quit IRC | 11:06 | |
| *** awalende has joined #openstack-neutron | 11:14 | |
| *** ramishra has quit IRC | 11:14 | |
| *** awalende has quit IRC | 11:14 | |
| *** awalende has joined #openstack-neutron | 11:20 | |
| *** awalende has quit IRC | 11:21 | |
| *** luksky has joined #openstack-neutron | 11:22 | |
| openstackgerrit | Aditya Reddy Nagaram proposed openstack/neutron master: [WIP] Support for stateless security groups https://review.opendev.org/572767 | 11:29 |
| *** bobmel has quit IRC | 11:52 | |
| openstackgerrit | Merged openstack/neutron-lib stable/train: install neutron_lib international messages https://review.opendev.org/689619 | 12:02 |
| *** zhanglong has joined #openstack-neutron | 12:04 | |
| *** dsneddon_ has joined #openstack-neutron | 12:05 | |
| *** dsneddon_ has quit IRC | 12:09 | |
| *** ociuhandu has quit IRC | 12:13 | |
| *** jpena is now known as jpena|lunch | 12:26 | |
| *** ramishra has joined #openstack-neutron | 12:26 | |
| *** rcernin has quit IRC | 12:31 | |
| openstackgerrit | Adrian Chiris proposed openstack/neutron master: Add upgrade check for NIC Switch agent https://review.opendev.org/694757 | 12:32 |
| *** lpetrut has joined #openstack-neutron | 12:42 | |
| *** ociuhandu has joined #openstack-neutron | 12:43 | |
| *** luksky has quit IRC | 12:47 | |
| *** ratailor__ has quit IRC | 12:51 | |
| *** dsneddon_ has joined #openstack-neutron | 12:53 | |
| *** zhanglong has quit IRC | 13:02 | |
| *** ociuhandu has quit IRC | 13:04 | |
| *** zhanglong has joined #openstack-neutron | 13:07 | |
| *** lennyb has joined #openstack-neutron | 13:09 | |
| *** zhanglong has quit IRC | 13:13 | |
| *** sapd1 has quit IRC | 13:16 | |
| *** lseki has joined #openstack-neutron | 13:17 | |
| openstackgerrit | Lajos Katona proposed openstack/neutron master: HA race condition test for DHCP scheduling https://review.opendev.org/683987 | 13:18 |
| *** dtantsur is now known as dtantsur|bbl | 13:21 | |
| *** nanzha has quit IRC | 13:24 | |
| *** nanzha has joined #openstack-neutron | 13:25 | |
| *** jpena|lunch is now known as jpena | 13:25 | |
| *** zhanglong has joined #openstack-neutron | 13:29 | |
| *** damiandabrowski2 has joined #openstack-neutron | 13:33 | |
| *** nweinber has joined #openstack-neutron | 13:37 | |
| *** luksky has joined #openstack-neutron | 13:39 | |
| openstackgerrit | Slawek Kaplonski proposed openstack/neutron master: Switch neutron-tempest-with-os-ken-master job to zuul v3 https://review.opendev.org/694770 | 13:40 |
| damiandabrowski2 | Hello, is it possible for neutron to provide full multihomed BGP routing for the cloud (not only advertise routes, but also direct traffic to on of the external/provider BGP peers? The documentation (neutron-dynamic-routing) shows route advertisment, but that would leave the outgoing traffic statically routed through one of the external networks. | 13:44 |
| slaweq | damiandabrowski2: afaict we don't have anything like what You're asking for | 13:49 |
| slaweq | You can only advertise prefixes from nodes using neutron-dynamic-routing | 13:50 |
| slaweq | but maybe tidwellr will know more about it as he is expert in neutron-dynamic-routing | 13:50 |
| openstackgerrit | Slawek Kaplonski proposed openstack/neutron-lib master: Revert "'interconnection' API extension definition (neutron-interconnection)" https://review.opendev.org/694466 | 13:53 |
| *** awalende has joined #openstack-neutron | 13:55 | |
| *** zhanglong has quit IRC | 13:55 | |
| damiandabrowski2 | slaweq: thanks for Your answer! tidwellr I would be very grateful if You could confirm that it's not possible ATM. | 13:56 |
| *** zhanglong has joined #openstack-neutron | 13:57 | |
| openstackgerrit | Merged openstack/networking-ovn stable/train: Add missing unittests to OVN provider driver https://review.opendev.org/694004 | 13:58 |
| *** ramishra has quit IRC | 13:59 | |
| *** slaweq has quit IRC | 14:02 | |
| *** slaweq has joined #openstack-neutron | 14:04 | |
| openstackgerrit | Merged openstack/networking-ovn stable/stein: Add missing unittests to OVN provider driver https://review.opendev.org/694005 | 14:07 |
| *** haleyb has joined #openstack-neutron | 14:12 | |
| *** ramishra has joined #openstack-neutron | 14:13 | |
| *** lennyb has quit IRC | 14:14 | |
| openstackgerrit | Lajos Katona proposed openstack/networking-odl master: Change function.func_doc to function.__doc__ https://review.opendev.org/683152 | 14:15 |
| openstackgerrit | Lajos Katona proposed openstack/networking-odl master: Try deinit odl_features in TestOdlFeaturesNoFixture setUpClass https://review.opendev.org/668904 | 14:17 |
| *** awalende has quit IRC | 14:26 | |
| *** beekneemech is now known as bnemec | 14:29 | |
| *** zhanglong has quit IRC | 14:34 | |
| *** goldyfruit has joined #openstack-neutron | 14:40 | |
| openstackgerrit | Merged openstack/networking-odl master: Remove the remaining neutron-lbaas related constants https://review.opendev.org/668161 | 14:44 |
| *** goldyfruit_ has joined #openstack-neutron | 14:51 | |
| *** goldyfruit has quit IRC | 14:53 | |
| *** baha has joined #openstack-neutron | 14:57 | |
| *** Luzi has quit IRC | 14:59 | |
| *** tesseract has quit IRC | 15:01 | |
| *** tesseract has joined #openstack-neutron | 15:01 | |
| *** dtantsur|bbl is now known as dtantsur | 15:01 | |
| tidwellr | damiandabrowski2: neutron-dynamic-routing will only announce the appropriate next-hops for floating IP's, subnets, and when using DVR the fixed IP. At the moment it doesn't steer egress traffic originating from VM's, the BGP announcements will only steer ingress traffic | 15:05 |
| *** tidwellr has quit IRC | 15:06 | |
| damiandabrowski2 | ok Thank You! | 15:09 |
| *** dsneddon_ has quit IRC | 15:18 | |
| *** dsneddon_ has joined #openstack-neutron | 15:23 | |
| *** ociuhandu has joined #openstack-neutron | 15:26 | |
| *** ociuhandu has quit IRC | 15:28 | |
| *** dsneddon_ has quit IRC | 15:28 | |
| frickler | damiandabrowski2: the bgp speakers in neutron are also not directly attached to the datapath, so what you want would be very difficult to achieve. most likely you rather want to setup a (pair of) router(s) in front of your openstack cloud that does this | 15:30 |
| *** lajoskatona has quit IRC | 15:30 | |
| *** dsneddon_ has joined #openstack-neutron | 15:49 | |
| *** dsneddon_ has quit IRC | 15:55 | |
| *** dklyle has quit IRC | 15:57 | |
| *** macz has joined #openstack-neutron | 15:58 | |
| *** dklyle has joined #openstack-neutron | 15:58 | |
| *** luksky has quit IRC | 15:59 | |
| *** dsneddon_ has joined #openstack-neutron | 16:00 | |
| zigo | I'm getting a huge amount of logs from openvswitch-agent, things like this: http://paste.openstack.org/show/786284/ | 16:04 |
| *** mlavalle has joined #openstack-neutron | 16:04 | |
| zigo | This looks like a real bug in Neutron that's been there for a long time already. :( | 16:04 |
| *** jmlowe has joined #openstack-neutron | 16:07 | |
| *** gcheresh has joined #openstack-neutron | 16:08 | |
| *** gcheresh_ has quit IRC | 16:09 | |
| *** ociuhandu has joined #openstack-neutron | 16:14 | |
| *** gcheresh has quit IRC | 16:14 | |
| *** ociuhandu has quit IRC | 16:19 | |
| zigo | I'm having this issue often, and the only way I know to fix is: 1/ stop neutron-l3 and ovs-agent 2/ iptables -F ; iptables -X 3/ restart the agents. | 16:21 |
| zigo | This is *very* annoying ... | 16:21 |
| zigo | Any clue on what's going on? | 16:22 |
| zigo | slaweq: mlavalle: ^ | 16:22 |
| zigo | I'm also getting this in the l3-agent logs: http://paste.openstack.org/show/786286/ | 16:28 |
| *** ociuhandu has joined #openstack-neutron | 16:30 | |
| *** gcheresh has joined #openstack-neutron | 16:33 | |
| openstackgerrit | Merged openstack/networking-ovn stable/train: [metadata-agent] Fix issue with TLS/SSL connections https://review.opendev.org/694742 | 16:34 |
| openstackgerrit | Slawek Kaplonski proposed openstack/neutron master: Switch neutron-tempest-with-os-ken-master job to zuul v3 https://review.opendev.org/694770 | 16:42 |
| *** gcheresh has quit IRC | 16:44 | |
| frickler | zigo: is that also on rocky or newer? | 16:52 |
| njohnston | slaweq: So we have the new review-priority field, is it applied to neutron-lib as well? Have we written down the rules on when to set it so we have a common understanding? | 16:53 |
| zigo | frickler: Rocky. | 16:54 |
| zigo | 13.0.4... | 16:54 |
| zigo | frickler: Is this fixed in the point release ? | 16:54 |
| zigo | 13.0.5 ? | 16:54 |
| zigo | I saw related commits on the tip of the branch. | 16:55 |
| frickler | njohnston: neutron-lib doesn't have rp yet | 16:55 |
| frickler | zigo: I don't know anything about your issue in general, but it seems that py3 related testing in rocky was a bit thin, so I'd not be surprised if this was another py3 issue. I've seen some already, though they were more obvious | 16:56 |
| zigo | :/ | 16:57 |
| *** ralonsoh has quit IRC | 16:58 | |
| *** aedc has joined #openstack-neutron | 16:59 | |
| *** ralonsoh has joined #openstack-neutron | 17:01 | |
| *** luksky has joined #openstack-neutron | 17:02 | |
| *** lucasagomes has quit IRC | 17:03 | |
| *** jmlowe has quit IRC | 17:04 | |
| *** ociuhandu has quit IRC | 17:17 | |
| *** rpittau is now known as rpittau|afk | 17:18 | |
| *** nanzha has quit IRC | 17:20 | |
| *** jlibosva has quit IRC | 17:23 | |
| *** nweinber has quit IRC | 17:26 | |
| *** ociuhandu has joined #openstack-neutron | 17:28 | |
| *** dsneddon_ has quit IRC | 17:32 | |
| *** dsneddon_ has joined #openstack-neutron | 17:34 | |
| openstackgerrit | Merged openstack/networking-ovn master: Devstack: Install six via pip https://review.opendev.org/692096 | 17:36 |
| *** dsneddon_ has quit IRC | 17:39 | |
| *** davidsha has quit IRC | 17:39 | |
| *** ircuser-1 has joined #openstack-neutron | 17:40 | |
| *** ociuhandu has quit IRC | 17:40 | |
| *** jpena is now known as jpena|off | 17:47 | |
| *** jlibosva has joined #openstack-neutron | 17:51 | |
| openstackgerrit | Merged openstack/networking-ovn stable/stein: Support for Router Scheduling on addition/removal of chassis https://review.opendev.org/694362 | 17:51 |
| *** bobmel has joined #openstack-neutron | 17:51 | |
| *** jlibosva has quit IRC | 18:01 | |
| *** tbachman has joined #openstack-neutron | 18:02 | |
| *** dtantsur is now known as dtantsur|afk | 18:03 | |
| *** dsneddon_ has joined #openstack-neutron | 18:11 | |
| openstackgerrit | Adrian Chiris proposed openstack/neutron master: Add upgrade check for NIC Switch agent https://review.opendev.org/694757 | 18:18 |
| *** manjeets has joined #openstack-neutron | 18:31 | |
| *** mvkr has quit IRC | 18:32 | |
| *** tbachman has quit IRC | 18:36 | |
| *** hjensas has quit IRC | 18:36 | |
| *** jlibosva has joined #openstack-neutron | 18:44 | |
| *** igordc has joined #openstack-neutron | 18:46 | |
| *** gouthamr_ is now known as gouthamr | 18:51 | |
| *** tesseract has quit IRC | 18:52 | |
| *** tbachman has joined #openstack-neutron | 18:52 | |
| *** ralonsoh has quit IRC | 18:56 | |
| *** jlibosva has quit IRC | 19:04 | |
| *** hjensas has joined #openstack-neutron | 19:11 | |
| *** aedc has quit IRC | 19:29 | |
| *** abaindur has joined #openstack-neutron | 19:34 | |
| *** dsneddon_ has quit IRC | 19:35 | |
| *** manjeets has quit IRC | 19:35 | |
| *** abaindur has quit IRC | 19:35 | |
| *** abaindur has joined #openstack-neutron | 19:36 | |
| *** manjeets has joined #openstack-neutron | 19:39 | |
| *** jlibosva has joined #openstack-neutron | 19:40 | |
| *** dsneddon_ has joined #openstack-neutron | 19:40 | |
| *** lajoskatona has joined #openstack-neutron | 19:48 | |
| *** dsneddon_ has quit IRC | 19:48 | |
| *** abaindur has quit IRC | 19:50 | |
| *** dsneddon_ has joined #openstack-neutron | 19:50 | |
| *** abaindur has joined #openstack-neutron | 19:51 | |
| *** dsneddon_ has quit IRC | 19:55 | |
| *** gcheresh has joined #openstack-neutron | 19:55 | |
| openstackgerrit | Brian Haley proposed openstack/neutron master: Add accepted egress direct flow https://review.opendev.org/666991 | 19:56 |
| openstackgerrit | Terry Wilson proposed openstack/networking-ovn master: Fix agent extension support after hashring merge https://review.opendev.org/694840 | 19:58 |
| *** lajoskatona has quit IRC | 19:59 | |
| *** dsneddon_ has joined #openstack-neutron | 20:06 | |
| *** dsneddon_ has quit IRC | 20:11 | |
| *** jmlowe has joined #openstack-neutron | 20:16 | |
| *** bobmel has quit IRC | 20:17 | |
| *** gcheresh has quit IRC | 20:47 | |
| *** dsneddon_ has joined #openstack-neutron | 20:50 | |
| *** dsneddon_ has quit IRC | 20:58 | |
| *** goldyfruit_ has quit IRC | 21:02 | |
| *** dsneddon_ has joined #openstack-neutron | 21:04 | |
| *** dsneddon_ has quit IRC | 21:08 | |
| *** jlibosva has quit IRC | 21:10 | |
| *** awalende has joined #openstack-neutron | 21:16 | |
| *** goldyfruit has joined #openstack-neutron | 21:17 | |
| *** awalende has quit IRC | 21:21 | |
| openstackgerrit | Brian Haley proposed openstack/networking-ovn master: Correctly initialize HashRingIsEmpty class https://review.opendev.org/694847 | 21:22 |
| *** gcheresh has joined #openstack-neutron | 21:22 | |
| *** awalende has joined #openstack-neutron | 21:26 | |
| *** awalende has quit IRC | 21:31 | |
| *** rkukura has joined #openstack-neutron | 21:32 | |
| *** awalende has joined #openstack-neutron | 21:36 | |
| *** ociuhandu has joined #openstack-neutron | 21:38 | |
| *** dsneddon_ has joined #openstack-neutron | 21:40 | |
| *** awalende has quit IRC | 21:40 | |
| *** abaindur has quit IRC | 21:42 | |
| *** ociuhandu has quit IRC | 21:43 | |
| *** abaindur has joined #openstack-neutron | 21:43 | |
| *** awalende has joined #openstack-neutron | 21:46 | |
| *** abaindur has quit IRC | 21:50 | |
| *** gcheresh has quit IRC | 21:52 | |
| *** awalende has quit IRC | 21:56 | |
| *** awalende has joined #openstack-neutron | 21:57 | |
| *** awalende has quit IRC | 22:07 | |
| *** awalende has joined #openstack-neutron | 22:07 | |
| *** awalende_ has joined #openstack-neutron | 22:08 | |
| *** awalende has quit IRC | 22:12 | |
| *** abaindur has joined #openstack-neutron | 22:20 | |
| *** abaindur has quit IRC | 22:24 | |
| *** pcaruana has quit IRC | 22:26 | |
| *** maciejjozefczyk has quit IRC | 22:39 | |
| *** maciejjozefczyk has joined #openstack-neutron | 22:39 | |
| *** mvkr has joined #openstack-neutron | 22:40 | |
| *** abaindur has joined #openstack-neutron | 22:44 | |
| *** abaindur has quit IRC | 22:44 | |
| *** abaindur has joined #openstack-neutron | 22:45 | |
| abaindur | any idea what might be causing this DBDeadlock when updating agent timestamps? | 22:45 |
| abaindur | ERROR oslo_db.api [req-231004a2-d988-47b3-9730-d6b5276fdcf8 - - - - -] DB exceeded retry limit.: DBDeadlock: (_mysql_exceptions.OperationalError) (1205, 'Lock wait timeout exceeded; try restarting transaction') [SQL: u'UPDATE agents SET heartbeat_timestamp=%s WHERE agents.id = %s'] [parameters: (datetime.datetime(2019, 11, 18, 8, 50, 23, 804716), '223c754e-9d7f-4df3-b5a5-9be4eb8692b0')] (Background on this error at: http://sqlalch | 22:46 |
| abaindur | e.me/e/e3q8) | 22:46 |
| *** awalende_ has quit IRC | 22:46 | |
| *** rcernin has joined #openstack-neutron | 22:46 | |
| abaindur | Since upgrading to Rocky, seeing repeatedly, AMQP loses connection (see various errors like missed heartbeats, Socket closed, Broken pipes, etc...) and report state RPCs from agents are timing out. Our rabbit-server and neutron-server are on the same node, all localhost communication | 22:47 |
| eandersson | abaindur we had this as well | 22:47 |
| abaindur | after some time, the q-reports-plugin rabbitmq queue grows large | 22:47 |
| abaindur | and we see those DBDeadlock stack traces in logs | 22:47 |
| eandersson | We had to scale up neutron massively and after about an hour the load went down and these problems went away | 22:48 |
| abaindur | scale up... what? | 22:48 |
| abaindur | the state report workers #? | 22:48 |
| abaindur | or rpc_workers? | 22:48 |
| eandersson | rpc workers | 22:48 |
| eandersson | The problem we had was that the rpc workers would take up a ton of memory | 22:48 |
| eandersson | How long ago was it that you upgraded? | 22:49 |
| abaindur | per atop, not seeing memory that high... although neutron and rabbit are at top of list | 22:49 |
| eandersson | How many rpc workers do you have, and how many computes did you upgrade? | 22:49 |
| abaindur | maybe a month ago or something. but we branched off stable/rocky sometime back in June or July | 22:50 |
| *** slaweq has quit IRC | 22:50 | |
| eandersson | We ended up setting out agent down time to 150 for agents | 22:50 |
| eandersson | but another key thing we had to do was tweak nova to contact neutron less often | 22:50 |
| abaindur | this isnt really a scaled setup either - only 22 hypervisors | 22:51 |
| abaindur | 345 instances | 22:51 |
| abaindur | I think we have 4 or 6 rpc_workers | 22:52 |
| eandersson | You can try to raise heal_instance_info_cache_interval on the compute side | 22:52 |
| abaindur | we had rpc_state_report_workers = 1, but scaled that up to 4 since it was the agent heartbeats that was getting DBDeadlock | 22:52 |
| abaindur | How would that help? Just trying to understand problem here - so first, what might cause neutron to miss AMQP heartbeats and get all kinds of Socket CLosed and broken pipe errors? | 22:53 |
| eandersson | So the issue we were having was that computes was hitting neutron too hard, causing deadlocks. | 22:54 |
| abaindur | and then, what would be causing a DBDeadlock? That seems like some kind of bug with syncrhonization... if we insrease timeouts/add workers, worried might just be delaying the problem...? | 22:54 |
| eandersson | It's odd as we had the exact same issue and tweaking the workers, agent timeout and the interval on computes fixed it for us | 22:55 |
| eandersson | and we have 1k+ computes | 22:55 |
| abaindur | were you hitting some random AMQP connection errors and broken pipes prior to dbdeadlocks? | 22:55 |
| abaindur | AMQP server on 127.0.0.1:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. | 22:55 |
| eandersson | I don't think so. | 22:55 |
| abaindur | WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: ConnectionForced: Too many heartbeats missed | 22:56 |
| eandersson | Haven't seen that. | 22:56 |
| abaindur | WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 32] Broken pipe | 22:56 |
| eandersson | We have heavily tweaked RabbitMQ for our deploys. | 22:56 |
| abaindur | [65c381f9-6766-4d88-815d-e13b74a7c46e] AMQP server 127.0.0.1:5672 closed the connection. Check login credentials: Socket closed: IOError: Socket closed | 22:56 |
| abaindur | All kinds of various errors like that ^^ | 22:57 |
| eandersson | I only see the above in scenarios where something has gone wrong and everything is trying to reconnect too fast. | 22:57 |
| abaindur | yea first few times it happened after a network/DB outage. | 22:57 |
| abaindur | but most recently, didnt have any control plane outage - all of a sudden AMQP errors, agents start being reported as down | 22:58 |
| eandersson | So one of the things we noticed with this issue is that the report queue was growing exponentially. | 22:58 |
| abaindur | Takes about 1 hr for the DBDeadlock errors to pop up | 22:58 |
| eandersson | q-reports- | 22:58 |
| abaindur | yea we see that queue growing | 22:58 |
| eandersson | or something like that | 22:58 |
| abaindur | rabbitmqctl list_queues | grep -vw "0$" | 22:59 |
| abaindur | Timeout: 60.0 seconds ... | 22:59 |
| abaindur | Listing queues for vhost / ... | 22:59 |
| abaindur | name messages | 22:59 |
| abaindur | q-reports-plugin 9239 | 22:59 |
| abaindur | Yes, thats the only queue that grows | 22:59 |
| abaindur | did you tweak any sql params>? | 23:00 |
| abaindur | like max_pool_size or max_overflow ? | 23:00 |
| eandersson | We did at some point, but for Rocky I think we went back to the defaults | 23:01 |
| abaindur | our agent_down_time is actually at 360 sec | 23:02 |
| *** luksky has quit IRC | 23:02 | |
| abaindur | report_interval is default, i think 30 sec | 23:02 |
| eandersson | We have it set to 60 | 23:06 |
| eandersson | But we also have 1k computes | 23:06 |
| abaindur | how many rpc and state report workers do you havve btw? | 23:07 |
| eandersson | 5x 20 rpc | 23:07 |
| eandersson | maybe 5x 10 state | 23:07 |
| eandersson | btw the problem we found with heartbeats was that one of those agent deadlocks locked one process | 23:08 |
| eandersson | So with your 9000 queued heartbeats stuck in deadlock | 23:08 |
| eandersson | each one of those would lock up one neutron-rpc worker | 23:09 |
| *** tkajinam has joined #openstack-neutron | 23:09 | |
| abaindur | hmm how does it get deadlocked in first place? | 23:09 |
| eandersson | I think it's retrying too fast | 23:10 |
| abaindur | yea in mysql, we saw some stuck sql queries for quite old timestamps that were still trying to be executed | 23:10 |
| eandersson | Let me see if I can find the code | 23:10 |
| *** slaweq has joined #openstack-neutron | 23:11 | |
| abaindur | create_or_update_agent in neutron/db/agents_db.py | 23:12 |
| abaindur | Seens this on some very basic, small setups so i definitely think something is wrong here. seems like its using some new neutron_lib code to wrap the DB updates since we moved to rocky | 23:14 |
| *** slaweq has quit IRC | 23:17 | |
| eandersson | Yea - we assumed that this was just due to our scale. | 23:17 |
| eandersson | I cant find my notes, but we found that the agent stuff was hitting a db retry (which retried like 6-10 times and once every 0.5s) | 23:22 |
| eandersson | When multiple were done at the same time it would cause them to race condition with eachother | 23:23 |
| eandersson | So each worker would be locked up for the duration of the db retry | 23:23 |
| eandersson | And I am pretty sure that in rocky they introduced that retry. | 23:23 |
| *** mlavalle has quit IRC | 23:24 | |
| *** ivve has quit IRC | 23:26 | |
| *** ociuhandu has joined #openstack-neutron | 23:31 | |
| *** ociuhandu has quit IRC | 23:36 | |
| *** goldyfruit has quit IRC | 23:36 | |
| eandersson | btw I would send an email to the mailinglist or open a bug abaindur | 23:37 |
| abaindur | i am filing one right now :) | 23:39 |
| *** zhanglong has joined #openstack-neutron | 23:47 | |
| abaindur | i filed https://bugs.launchpad.net/neutron/+bug/1853071 | 23:59 |
| openstack | Launchpad bug 1853071 in neutron "AMQP disconnects, q-reports-plugin queue grows, leading to DBDeadlocks while trying to update agent heartbeats" [Undecided,New] | 23:59 |
| *** dsneddon_ has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!