| *** abaindur has quit IRC | 00:05 | |
| *** macz has quit IRC | 00:08 | |
| openstackgerrit | Mithil Arun proposed openstack/neutron master: Don't advertise reserved DHCP ports as DNS servers https://review.opendev.org/694859 | 00:09 |
|---|---|---|
| *** dsneddon_ has joined #openstack-neutron | 00:09 | |
| *** dsneddon_ has quit IRC | 00:14 | |
| *** abaindur has joined #openstack-neutron | 00:14 | |
| *** abaindur has quit IRC | 00:14 | |
| *** zhanglong has quit IRC | 00:14 | |
| *** abaindur has joined #openstack-neutron | 00:14 | |
| *** zhanglong has joined #openstack-neutron | 00:17 | |
| *** chenhaw has joined #openstack-neutron | 00:39 | |
| *** chenhaw has quit IRC | 00:39 | |
| *** goldyfruit has joined #openstack-neutron | 00:39 | |
| *** abaindur has quit IRC | 00:43 | |
| *** abaindur has joined #openstack-neutron | 00:44 | |
| *** dsneddon_ has joined #openstack-neutron | 00:46 | |
| *** sapd1 has joined #openstack-neutron | 00:54 | |
| *** nanzha has joined #openstack-neutron | 01:04 | |
| *** nanzha has quit IRC | 01:10 | |
| *** slaweq has joined #openstack-neutron | 01:11 | |
| *** nanzha has joined #openstack-neutron | 01:13 | |
| *** bobmel has joined #openstack-neutron | 01:13 | |
| *** haleyb has quit IRC | 01:15 | |
| *** slaweq has quit IRC | 01:15 | |
| *** haleyb has joined #openstack-neutron | 01:16 | |
| *** mvkr has quit IRC | 01:18 | |
| *** bobmel has quit IRC | 01:19 | |
| *** ociuhandu has joined #openstack-neutron | 01:31 | |
| openstackgerrit | Terry Wilson proposed openstack/networking-ovn master: Fix agent extension support after hashring merge https://review.opendev.org/694840 | 01:33 |
| *** zhanglong has quit IRC | 01:33 | |
| *** ociuhandu has quit IRC | 01:36 | |
| *** goldyfruit has quit IRC | 01:43 | |
| *** ociuhandu has joined #openstack-neutron | 01:44 | |
| *** ociuhandu has quit IRC | 01:48 | |
| *** ociuhandu has joined #openstack-neutron | 01:49 | |
| *** ociuhandu has quit IRC | 01:57 | |
| openstackgerrit | pangliye proposed openstack/neutron-specs master: Fix misspell word https://review.opendev.org/694888 | 02:07 |
| *** slaweq has joined #openstack-neutron | 02:11 | |
| *** slaweq has quit IRC | 02:16 | |
| *** igordc has quit IRC | 02:21 | |
| *** zhanglong has joined #openstack-neutron | 02:25 | |
| *** zhanglong has quit IRC | 02:34 | |
| *** goldyfruit has joined #openstack-neutron | 02:41 | |
| *** hjensas has quit IRC | 02:45 | |
| *** baojg has joined #openstack-neutron | 02:46 | |
| *** ociuhandu has joined #openstack-neutron | 02:56 | |
| *** ociuhandu has quit IRC | 03:01 | |
| *** ociuhandu has joined #openstack-neutron | 03:03 | |
| *** abaindur has quit IRC | 03:05 | |
| *** abaindur has joined #openstack-neutron | 03:06 | |
| *** ociuhandu has quit IRC | 03:07 | |
| *** ociuhandu has joined #openstack-neutron | 03:10 | |
| *** slaweq has joined #openstack-neutron | 03:11 | |
| *** mvkr has joined #openstack-neutron | 03:12 | |
| *** bobmel has joined #openstack-neutron | 03:14 | |
| *** slaweq has quit IRC | 03:16 | |
| *** ociuhandu has quit IRC | 03:18 | |
| *** bobmel has quit IRC | 03:19 | |
| *** abaindur has quit IRC | 03:24 | |
| *** ociuhandu has joined #openstack-neutron | 03:25 | |
| *** ociuhandu has quit IRC | 03:30 | |
| openstackgerrit | ZhouHeng proposed openstack/neutron master: Parameter 'fileds' value is not used in _get_subnets https://review.opendev.org/691544 | 03:31 |
| *** ociuhandu has joined #openstack-neutron | 03:32 | |
| *** ociuhandu has quit IRC | 03:36 | |
| *** dsneddon_ has quit IRC | 03:38 | |
| openstackgerrit | Merged openstack/networking-ovn master: Add support for virtual port type https://review.opendev.org/676223 | 03:46 |
| openstackgerrit | Merged openstack/neutron-lib stable/stein: install neutron_lib international messages https://review.opendev.org/689620 | 03:49 |
| openstackgerrit | Reedip proposed openstack/neutron master: Dont schedule Network, respecting network_auto_schedule config https://review.opendev.org/693645 | 03:54 |
| *** goldyfruit has quit IRC | 04:00 | |
| *** macz has joined #openstack-neutron | 04:04 | |
| *** ociuhandu has joined #openstack-neutron | 04:06 | |
| *** macz has quit IRC | 04:09 | |
| *** dsneddon_ has joined #openstack-neutron | 04:10 | |
| *** slaweq has joined #openstack-neutron | 04:11 | |
| *** ociuhandu has quit IRC | 04:14 | |
| *** slaweq has quit IRC | 04:15 | |
| *** lajoskatona has joined #openstack-neutron | 04:19 | |
| *** abaindur has joined #openstack-neutron | 04:32 | |
| *** jmlowe has quit IRC | 04:42 | |
| *** ratailor has joined #openstack-neutron | 04:53 | |
| *** dsneddon_ has quit IRC | 05:06 | |
| *** dsneddon_ has joined #openstack-neutron | 05:14 | |
| *** bobmel has joined #openstack-neutron | 05:15 | |
| *** ociuhandu has joined #openstack-neutron | 05:18 | |
| *** nanzha has quit IRC | 05:18 | |
| *** nanzha has joined #openstack-neutron | 05:19 | |
| *** bobmel has quit IRC | 05:20 | |
| *** dsneddon_ has quit IRC | 05:20 | |
| *** abaindur has quit IRC | 05:22 | |
| *** ociuhandu has quit IRC | 05:22 | |
| *** abaindur has joined #openstack-neutron | 05:23 | |
| *** zhanglong has joined #openstack-neutron | 05:24 | |
| *** elod is now known as elod_off | 05:28 | |
| *** ociuhandu has joined #openstack-neutron | 05:30 | |
| *** baojg has quit IRC | 05:31 | |
| *** baojg has joined #openstack-neutron | 05:34 | |
| *** ociuhandu has quit IRC | 05:35 | |
| *** pcaruana has joined #openstack-neutron | 05:43 | |
| *** dsneddon_ has joined #openstack-neutron | 05:48 | |
| *** lennyb has joined #openstack-neutron | 05:54 | |
| *** Luzi has joined #openstack-neutron | 06:00 | |
| *** openstackstatus has joined #openstack-neutron | 06:09 | |
| *** ChanServ sets mode: +v openstackstatus | 06:09 | |
| *** slaweq has joined #openstack-neutron | 06:11 | |
| *** slaweq has quit IRC | 06:15 | |
| *** lpetrut has quit IRC | 06:25 | |
| openstackgerrit | pangliye proposed openstack/neutron-specs master: Replace git.openstack.org with opendev.org https://review.opendev.org/694909 | 06:35 |
| *** jlibosva has joined #openstack-neutron | 06:37 | |
| *** abaindur has quit IRC | 06:44 | |
| *** abaindur has joined #openstack-neutron | 06:44 | |
| *** pcaruana has quit IRC | 06:56 | |
| *** rcernin has quit IRC | 06:58 | |
| *** dsneddon_ has quit IRC | 07:03 | |
| *** nanzha has quit IRC | 07:04 | |
| *** nanzha has joined #openstack-neutron | 07:04 | |
| *** dsneddon_ has joined #openstack-neutron | 07:13 | |
| *** bobmel has joined #openstack-neutron | 07:16 | |
| *** bobmel has quit IRC | 07:20 | |
| openstackgerrit | Taoyunxiang proposed openstack/networking-ovn master: Support QoS by meter action https://review.opendev.org/692084 | 07:52 |
| *** slaweq has joined #openstack-neutron | 08:00 | |
| *** gcheresh has joined #openstack-neutron | 08:09 | |
| *** ivve has joined #openstack-neutron | 08:17 | |
| *** gcheresh_ has joined #openstack-neutron | 08:18 | |
| *** ygk_12345 has joined #openstack-neutron | 08:18 | |
| *** gcheresh has quit IRC | 08:18 | |
| ygk_12345 | hi all | 08:18 |
| ygk_12345 | i am seeing messaging timeouts in the neutron-dhcp logs and hence the vif creation is timing out | 08:18 |
| *** tesseract has joined #openstack-neutron | 08:18 | |
| ygk_12345 | is this a bug in neutron rocky 18.1.9 branch ? | 08:19 |
| ygk_12345 | it used to work well earlier | 08:19 |
| slaweq | ygk_12345: hi, can You show me this error from logs, maybe I saw it before already | 08:19 |
| ygk_12345 | slaweq http://paste.openstack.org/show/786324/ | 08:21 |
| ygk_12345 | even though I restarted the services no luck | 08:21 |
| slaweq | ygk_12345: give me a sec, I think I know what can be the reason | 08:26 |
| ygk_12345 | ok sure | 08:27 |
| slaweq | ygk_12345: You probably hit this bug https://bugs.launchpad.net/neutron/+bug/1834257 | 08:28 |
| openstack | Launchpad bug 1834257 in neutron "dhcp-agent can overwhelm neutron server with dhcp_ready_on_ports RPC calls" [High,Fix released] - Assigned to Sebastian Lohff (sebageek) | 08:28 |
| slaweq | it's fixed in https://github.com/openstack/neutron/commit/76ccdb35d4b09106aa8adc97528b02f0fd8acbcc | 08:28 |
| slaweq | but it's not in rocky probably | 08:28 |
| ygk_12345 | suddenly it showed up in rocky. we have other setups and they r running fine | 08:29 |
| slaweq | ygk_12345: oh, it should be fixed in rocky https://review.opendev.org/#/c/669943/ | 08:29 |
| ygk_12345 | but why we r seeing it now ? | 08:29 |
| slaweq | idk | 08:29 |
| eandersson | Yea can confirm that it was fixed by that patch | 08:29 |
| ygk_12345 | it is 18.1.9 rocky openstack ansible | 08:29 |
| openstackgerrit | inspurericzhang proposed openstack/neutron-specs master: Replace git.openstack.org with opendev.org https://review.opendev.org/694938 | 08:30 |
| eandersson | The more ports you have the more likely you are to see it | 08:30 |
| ygk_12345 | so shall I applyt the patch now ? | 08:30 |
| ygk_12345 | ahhhhh | 08:30 |
| ygk_12345 | thats the case maybe | 08:30 |
| eandersson | Because all ports were handled in a single RPC call. | 08:30 |
| slaweq | probably You have many ports on networks hosted by dhcp agent and agent was restarted or something else happend and triggered "full sync" of agent | 08:30 |
| ygk_12345 | increase of ports | 08:30 |
| eandersson | So for us with 10k+ ports it would take like 20 minutes. | 08:30 |
| ygk_12345 | let me check the rough value of no of ports | 08:30 |
| slaweq | than it starts sending information about many ports and it is timing out because neutron-server is processing those ports for long time | 08:31 |
| slaweq | ygk_12345: there was also another issue with this reporting of ready ports in chunks | 08:32 |
| slaweq | ygk_12345: it was fixed by https://review.opendev.org/#/c/692459/ but this isn't merged to rocky yet | 08:32 |
| slaweq | if You don't have those patches in Your env, You can apply them manually - it should probably help to recover Your agent | 08:32 |
| ygk_12345 | so once again can you please post the patches to be applied ? | 08:33 |
| ygk_12345 | it is roughly 6.6. k ports intotal | 08:33 |
| slaweq | ygk_12345: first, this one: https://review.opendev.org/#/c/669943/ | 08:34 |
| ygk_12345 | ok then ? | 08:34 |
| slaweq | and than also https://review.opendev.org/#/c/694560/ | 08:34 |
| ygk_12345 | ok thanks a lot guys. i will test it | 08:35 |
| ygk_12345 | slaweq i PMed you. pls respond | 08:35 |
| openstackgerrit | Merged openstack/neutron stable/stein: Check dnsmasq process is active when spawned https://review.opendev.org/694250 | 08:35 |
| *** pcaruana has joined #openstack-neutron | 08:36 | |
| eandersson | I would recommend rebasing against stable/rocky tbh | 08:37 |
| *** abaindur has quit IRC | 08:38 | |
| *** luksky has joined #openstack-neutron | 08:42 | |
| *** jpena|off is now known as jpena | 08:42 | |
| *** root has joined #openstack-neutron | 08:43 | |
| *** root is now known as Guest98380 | 08:43 | |
| *** ralonsoh has joined #openstack-neutron | 08:50 | |
| *** priteau has joined #openstack-neutron | 08:55 | |
| *** tkajinam has quit IRC | 08:56 | |
| *** FlorianFa has joined #openstack-neutron | 08:58 | |
| *** ociuhandu has joined #openstack-neutron | 09:00 | |
| *** baojg has quit IRC | 09:01 | |
| *** lucasagomes has joined #openstack-neutron | 09:02 | |
| *** Luzi has quit IRC | 09:04 | |
| *** dsneddon_ has quit IRC | 09:06 | |
| *** rpittau|afk is now known as rpittau | 09:08 | |
| openstackgerrit | Aditya Reddy Nagaram proposed openstack/neutron master: [WIP] Support for stateless security groups https://review.opendev.org/572767 | 09:10 |
| openstackgerrit | Oleg Bondarev proposed openstack/neutron master: L3 agent graceful shutdown https://review.opendev.org/693323 | 09:13 |
| *** Luzi has joined #openstack-neutron | 09:19 | |
| openstackgerrit | Lajos Katona proposed openstack/networking-odl master: Remove networking-odl python2 jobs https://review.opendev.org/687858 | 09:19 |
| *** Luzi has quit IRC | 09:21 | |
| *** dsneddon_ has joined #openstack-neutron | 09:30 | |
| *** ratailor_ has joined #openstack-neutron | 09:30 | |
| openstackgerrit | Lajos Katona proposed openstack/networking-odl master: Remove networking-odl python2 jobs https://review.opendev.org/687858 | 09:31 |
| *** ratailor has quit IRC | 09:32 | |
| *** dsneddon_ has quit IRC | 09:36 | |
| *** ociuhandu has quit IRC | 09:43 | |
| openstackgerrit | Slawek Kaplonski proposed openstack/neutron master: Remove neutron-grenade job from Neutron CI queues https://review.opendev.org/694039 | 09:52 |
| *** njohnston_ has joined #openstack-neutron | 09:53 | |
| ygk_12345 | slaweq even after applying the patches I see the message timeouts in dhcp agent logs | 09:54 |
| ygk_12345 | vm launch is timing out | 09:54 |
| *** njohnston has quit IRC | 09:54 | |
| ygk_12345 | slaweq http://paste.openstack.org/show/786328/ | 09:57 |
| *** davidsha has joined #openstack-neutron | 09:58 | |
| *** zhanglong has quit IRC | 10:03 | |
| *** dsneddon_ has joined #openstack-neutron | 10:05 | |
| ygk_12345 | MessagingTimeout: Timed out waiting for a reply to message ID 037fb977709a4e348539c806fbf3ab17 | 10:06 |
| slaweq | ygk_12345: this is timeout on different message | 10:10 |
| openstackgerrit | Federico Ressi proposed openstack/networking-ovn master: Remove xtrace alteration in devstack/plugin.sh https://review.opendev.org/694968 | 10:10 |
| slaweq | ygk_12345: maybe Your neutron server is overloaded or You have problem with rabbitmq? | 10:10 |
| slaweq | I don't know tbh | 10:10 |
| ygk_12345 | how t o check to confirm rabbitmq issues ? | 10:11 |
| ralonsoh | ygk_12345, you have a quick guide for MQ troubleshooting https://wiki.openstack.org/wiki/OpsGuide/RabbitMQ_troubleshooting | 10:14 |
| ralonsoh | if this can help you | 10:14 |
| ygk_12345 | i see these | 10:16 |
| ygk_12345 | operation basic.publish caused a channel exception not_found: no exchange 'reply_8ac6b4a82df242db94a2581aa68a7d06' in vhost '/nova'2019-11-19 05:16:32.505 [error] <0.19767.6> Channel error on connection <0.11199.6> (172.29.239.217:35020 -> 172.29.239.237:5671, vhost: '/nova', user: 'nova'), channel | 10:16 |
| zigo | It really looks like ovs-agent doesn't like /usr/sbin/iptables-nft at all. | 10:17 |
| zigo | Is this is known problem ? | 10:17 |
| ygk_12345 | [error] <0.26986.6> Channel error on connection <0.23176.6> (172.29.239.162:57866 -> 172.29.239.237:5671, vhost: '/neutron', user: 'neutron'), channel 1 | 10:17 |
| zigo | I had to switch /usr/sbin/iptables alternative to /usr/sbin/iptables-legacy to have ovs-agent to correctly apply its rules. | 10:17 |
| ralonsoh | ygk_12345, check manually the borker is online | 10:19 |
| ralonsoh | https://www.rabbitmq.com/troubleshooting-ssl.html | 10:19 |
| ygk_12345 | eventhough i restarted the rabbitmq still I see errors in the rabbitmq log | 10:23 |
| ygk_12345 | 2019-11-19 05:23:13.547 [error] <0.13702.0> Channel error on connection <0.13489.0> (172.29.239.162:44502 -> 172.29.239.237:5671, vhost: '/neutron', user: 'neutron'), channel 1: | 10:23 |
| trident | ygk_12345: Do you have TLS enabled? Could you have hit this one? https://bugs.launchpad.net/oslo.messaging/+bug/1800957 | 10:26 |
| openstack | Launchpad bug 1800957 in oslo.messaging (Ubuntu) "Upgrading to pike version causes rabbit timeouts with ssl" [Undecided,Confirmed] | 10:26 |
| ygk_12345 | but it was working fine earlier and in other setups we haveit is working fine | 10:27 |
| *** ociuhandu has joined #openstack-neutron | 10:31 | |
| openstackgerrit | Maciej Józefczyk proposed openstack/networking-ovn master: WIP Add support for double listeners on the same LB https://review.opendev.org/671293 | 10:39 |
| *** lpetrut has joined #openstack-neutron | 10:40 | |
| lajoskatona | slaweq: Hi, question regarding tap-as-a-service, where shall I put new spec? | 10:48 |
| ygk_12345 | how to cleanout a rabbitmq queue ? | 10:48 |
| lajoskatona | slaweq: I've seen spec in neutron-spec repo and in taas repo, and Yamamoto was not sure where to put a new one | 10:48 |
| slaweq | lajoskatona: In neutron-spec I think we have only one "spec" to track progress of inclusion of tap-as-a-service into stadium | 10:49 |
| slaweq | but as there was no progress on that during Train, we decided that for now it's not candidate to the stadium | 10:50 |
| slaweq | so IMO as it's 3rd party project still, spec for that shouldn't go to neutron-spec repo | 10:51 |
| lajoskatona | slaweq: ok, thanks, I add then to taas spec folder | 10:51 |
| slaweq | lajoskatona: ok, that will be good way to do it IMO, thx | 10:52 |
| *** ratailor_ has quit IRC | 10:58 | |
| *** ratailor has joined #openstack-neutron | 10:59 | |
| *** dsneddon_ has quit IRC | 11:08 | |
| *** root3 has joined #openstack-neutron | 11:08 | |
| *** Guest98380 has quit IRC | 11:10 | |
| *** awalende has joined #openstack-neutron | 11:28 | |
| *** hjensas has joined #openstack-neutron | 11:31 | |
| openstackgerrit | Federico Ressi proposed openstack/networking-ovn master: [WIP] Use pip_install because pip3 command could be unavailable https://review.opendev.org/694987 | 11:32 |
| *** tbachman has quit IRC | 11:38 | |
| *** bobmel has joined #openstack-neutron | 11:49 | |
| *** dtantsur|afk is now known as dtantsur | 11:49 | |
| *** ociuhandu has quit IRC | 11:52 | |
| *** ociuhandu has joined #openstack-neutron | 11:53 | |
| *** ociuhandu has quit IRC | 11:53 | |
| *** bobmel has quit IRC | 11:53 | |
| *** ociuhandu has joined #openstack-neutron | 11:54 | |
| *** root4 has joined #openstack-neutron | 11:55 | |
| *** root3 has quit IRC | 11:57 | |
| ygk_12345 | slaweq we r rerunning the neutron playbook and the rabbit error messages seems to have stopped :) | 11:59 |
| *** ociuhandu has quit IRC | 11:59 | |
| slaweq | ygk_12345: hmm, interesting, maybe agent was connected to some queue which didn't had consumer? | 12:03 |
| slaweq | idk tbh | 12:03 |
| ygk_12345 | slaweq weird things | 12:04 |
| ygk_12345 | :) | 12:04 |
| *** root has joined #openstack-neutron | 12:05 | |
| *** root is now known as Guest3144 | 12:05 | |
| *** root4 has quit IRC | 12:07 | |
| *** ratailor has quit IRC | 12:15 | |
| openstackgerrit | Federico Ressi proposed openstack/networking-ovn master: [WIP] Use pip_install because pip3 command could be unavailable https://review.opendev.org/694987 | 12:15 |
| *** goldyfruit has joined #openstack-neutron | 12:18 | |
| openstackgerrit | Maciej Józefczyk proposed openstack/networking-ovn master: WIP Add support for double listeners on the same LB https://review.opendev.org/671293 | 12:22 |
| *** root has joined #openstack-neutron | 12:29 | |
| *** root is now known as Guest14170 | 12:30 | |
| *** Guest3144 has quit IRC | 12:30 | |
| *** ygk_12345 has quit IRC | 12:30 | |
| *** jpena is now known as jpena|lunch | 12:31 | |
| openstackgerrit | Taoyunxiang proposed openstack/networking-ovn master: Support QoS by meter action https://review.opendev.org/692084 | 12:34 |
| *** dave-mccowan has joined #openstack-neutron | 12:34 | |
| *** ociuhandu has joined #openstack-neutron | 12:36 | |
| *** root3 has joined #openstack-neutron | 12:44 | |
| *** Guest14170 has quit IRC | 12:47 | |
| *** ociuhandu has quit IRC | 13:00 | |
| *** ociuhandu has joined #openstack-neutron | 13:01 | |
| *** ygk_12345 has joined #openstack-neutron | 13:04 | |
| ygk_12345 | slaweq still the vif creation is faling | 13:04 |
| *** dsneddon_ has joined #openstack-neutron | 13:05 | |
| ygk_12345 | Failed to allocate the network(s), not rescheduling.: BuildAbortException: Build of instance 2bd9b475-c1e6-4f14-b8ef-796604f17520 aborted: Failed to allocate the network(s), not rescheduling | 13:05 |
| ygk_12345 | ERROR neutron.agent.rpc MessagingTimeout: Timed out waiting for a reply to message ID 36254fb80b2a4638b18b34d87903ada1 | 13:06 |
| *** goldyfruit has quit IRC | 13:08 | |
| ygk_12345 | slaweq does this DHCP port limit apply to neutron openvswitch agent as well ? | 13:09 |
| *** dsneddon_ has quit IRC | 13:09 | |
| ygk_12345 | i see messaging timeouts in neutron ovs as well | 13:09 |
| *** ociuhandu has quit IRC | 13:17 | |
| *** ociuhandu has joined #openstack-neutron | 13:18 | |
| openstackgerrit | Lucas Alvares Gomes proposed openstack/networking-ovn master: Add support for external ports https://review.opendev.org/678261 | 13:20 |
| *** ociuhandu_ has joined #openstack-neutron | 13:22 | |
| *** awalende has quit IRC | 13:26 | |
| *** ociuhandu has quit IRC | 13:26 | |
| *** tbachman has joined #openstack-neutron | 13:27 | |
| *** root4 has joined #openstack-neutron | 13:30 | |
| *** jpena|lunch is now known as jpena | 13:31 | |
| *** root3 has quit IRC | 13:32 | |
| CobHead | Hi everyone - I seem to encounter https://review.opendev.org/#/c/629912/ in Rocky Neutron. Any chances of a backport from master? Was thinking of trying to patch from master, but I am unsure how well that would work on Rocky.. | 13:33 |
| CobHead | https://bugs.launchpad.net/neutron/+bug/1804274 also mentions someone in Ocata that has the same issue. | 13:33 |
| openstack | Launchpad bug 1804274 in neutron "Neutron error agreggation interfaces" [Low,Fix released] | 13:33 |
| CobHead | This bug is kinda breaking, as it slows down all of the controller nodes in the setup, causing Neutron to drop a lot of packages from tenant networks. | 13:35 |
| *** liuyulong has joined #openstack-neutron | 13:36 | |
| *** dsneddon_ has joined #openstack-neutron | 13:36 | |
| *** dsneddon_ has quit IRC | 13:41 | |
| *** awalende has joined #openstack-neutron | 13:41 | |
| *** awalende has quit IRC | 13:42 | |
| *** awalende has joined #openstack-neutron | 13:42 | |
| lajoskatona | slaweq: Hi again :-) Regarding networking-bgpvn and networking-bagpipe maintenance, Ericsson is willing to work on that | 13:43 |
| lajoskatona | slaweq: we use bgpvpn with ODL, but we can at least help the maintenance of bagpipe as well. | 13:44 |
| *** ociuhandu_ has quit IRC | 13:52 | |
| slaweq | lajoskatona: that would be great | 13:52 |
| slaweq | lajoskatona: can You reply on the ML for that too, that it would be visible for everyone | 13:52 |
| *** ociuhandu has joined #openstack-neutron | 13:53 | |
| *** tkajinam has joined #openstack-neutron | 13:56 | |
| *** tkajinam has quit IRC | 13:56 | |
| *** tkajinam has joined #openstack-neutron | 13:57 | |
| ralonsoh | CobHead, I don't see any problem if having this in Rocky but I don't know if this could be backported so far | 13:57 |
| ralonsoh | CobHead, you can try and propose it | 13:57 |
| CobHead | ralonsoh, I will get the commit and try it in a test lab first. | 13:58 |
| *** ociuhandu has quit IRC | 13:58 | |
| *** njohnston_ is now known as njohnston | 14:00 | |
| *** davidsha has quit IRC | 14:01 | |
| lajoskatona | slaweq: I check the list | 14:02 |
| *** luksky has quit IRC | 14:02 | |
| *** ociuhandu has joined #openstack-neutron | 14:02 | |
| openstackgerrit | Merged openstack/neutron stable/stein: Increase timeout when waiting for dnsmasq enablement https://review.opendev.org/692377 | 14:07 |
| *** ociuhandu has quit IRC | 14:08 | |
| *** baha has left #openstack-neutron | 14:10 | |
| *** nweinber has joined #openstack-neutron | 14:15 | |
| *** davidsha has joined #openstack-neutron | 14:16 | |
| *** goldyfruit has joined #openstack-neutron | 14:19 | |
| *** awalende has quit IRC | 14:25 | |
| *** awalende has joined #openstack-neutron | 14:26 | |
| *** goldyfruit_ has joined #openstack-neutron | 14:28 | |
| ygk_12345 | I have updated this https://bugs.launchpad.net/neutron/+bug/1852175 | 14:30 |
| openstack | Launchpad bug 1852175 in oslo.messaging "MessagingTimeout" [Undecided,Incomplete] | 14:30 |
| *** goldyfruit has quit IRC | 14:30 | |
| ygk_12345 | can someone look into this issue ? | 14:31 |
| *** ociuhandu has joined #openstack-neutron | 14:49 | |
| openstackgerrit | Maciej Józefczyk proposed openstack/networking-ovn master: WIP Add support for double listeners on the same LB https://review.opendev.org/671293 | 14:58 |
| *** tkajinam has quit IRC | 15:00 | |
| *** lajoskatona has quit IRC | 15:13 | |
| openstackgerrit | Slawek Kaplonski proposed openstack/neutron-tempest-plugin master: Use python 3 on all nodes in multinode job https://review.opendev.org/695013 | 15:20 |
| *** markmcclain has quit IRC | 15:21 | |
| *** markmcclain has joined #openstack-neutron | 15:23 | |
| openstackgerrit | Nate Johnston proposed openstack/neutron-dynamic-routing master: Remove tempest tests that have moved to neutron-tempest-plugin https://review.opendev.org/695014 | 15:24 |
| *** nweinber_ has joined #openstack-neutron | 15:35 | |
| *** dsneddon_ has joined #openstack-neutron | 15:37 | |
| *** nweinber has quit IRC | 15:37 | |
| *** ivve has quit IRC | 15:37 | |
| *** dsneddon_ has quit IRC | 15:42 | |
| *** gcheresh_ has quit IRC | 15:43 | |
| *** lajoskatona has joined #openstack-neutron | 15:48 | |
| *** pcaruana has quit IRC | 15:50 | |
| *** pcaruana has joined #openstack-neutron | 15:51 | |
| openstackgerrit | Maciej Józefczyk proposed openstack/networking-ovn master: WIP Add support for double listeners on the same LB https://review.opendev.org/671293 | 15:51 |
| *** armax has quit IRC | 15:54 | |
| *** armax has joined #openstack-neutron | 15:56 | |
| *** jlibosva has quit IRC | 15:59 | |
| *** ygk_12345 has quit IRC | 16:01 | |
| *** awalende has quit IRC | 16:01 | |
| *** awalende has joined #openstack-neutron | 16:08 | |
| *** mlavalle has joined #openstack-neutron | 16:09 | |
| *** lucasagomes has quit IRC | 16:11 | |
| *** awalende has quit IRC | 16:12 | |
| *** awalende has joined #openstack-neutron | 16:12 | |
| *** awalende has quit IRC | 16:13 | |
| *** lucasagomes has joined #openstack-neutron | 16:13 | |
| *** awalende has joined #openstack-neutron | 16:14 | |
| *** awalende has quit IRC | 16:14 | |
| *** ociuhandu has quit IRC | 16:17 | |
| *** nanzha has quit IRC | 16:17 | |
| *** awalende has joined #openstack-neutron | 16:19 | |
| *** jmlowe has joined #openstack-neutron | 16:33 | |
| *** abaindur has joined #openstack-neutron | 16:39 | |
| openstackgerrit | Thierry Carrez proposed openstack/neutron master: Start README.rst with a better title https://review.opendev.org/695031 | 16:42 |
| *** lajoskatona has quit IRC | 16:46 | |
| *** jpena is now known as jpena|brb | 16:46 | |
| *** lucasagomes has quit IRC | 16:47 | |
| *** tesseract has quit IRC | 16:51 | |
| *** nweinber__ has joined #openstack-neutron | 17:04 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/neutron master: Explicit mention py2 for neutron-grenade job https://review.opendev.org/695036 | 17:05 |
| *** nweinber_ has quit IRC | 17:06 | |
| *** rpittau is now known as rpittau|afk | 17:06 | |
| openstackgerrit | Merged openstack/networking-odl master: Try deinit odl_features in TestOdlFeaturesNoFixture setUpClass https://review.opendev.org/668904 | 17:07 |
| *** awalende has quit IRC | 17:09 | |
| *** awalende has joined #openstack-neutron | 17:17 | |
| *** dtantsur is now known as dtantsur|afk | 17:20 | |
| *** awalende has quit IRC | 17:20 | |
| *** awalende_ has joined #openstack-neutron | 17:20 | |
| *** ivve has joined #openstack-neutron | 17:21 | |
| *** ociuhandu has joined #openstack-neutron | 17:22 | |
| openstackgerrit | Merged openstack/neutron master: Start README.rst with a better title https://review.opendev.org/695031 | 17:23 |
| *** awalende_ has quit IRC | 17:24 | |
| *** jpena|brb is now known as jpena | 17:26 | |
| *** ociuhandu has quit IRC | 17:27 | |
| openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/neutron master: NetcatTester.stop_processes skip "No such process" exception https://review.opendev.org/695047 | 17:30 |
| openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/neutron master: NetcatTester.stop_processes skip "No such process" exception https://review.opendev.org/695047 | 17:32 |
| *** dsneddon_ has joined #openstack-neutron | 17:38 | |
| *** dsneddon_ has quit IRC | 17:43 | |
| *** munimeha1 has joined #openstack-neutron | 17:49 | |
| *** igordc has joined #openstack-neutron | 17:52 | |
| *** ociuhandu has joined #openstack-neutron | 18:04 | |
| *** davidsha has quit IRC | 18:04 | |
| *** ociuhandu has quit IRC | 18:08 | |
| *** armax has quit IRC | 18:11 | |
| *** ociuhandu has joined #openstack-neutron | 18:17 | |
| *** ociuhandu has quit IRC | 18:21 | |
| *** dsneddon_ has joined #openstack-neutron | 18:22 | |
| *** dsneddon_ has quit IRC | 18:26 | |
| openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/neutron master: [WIP] Serialize subnet creating depending on the network ID https://review.opendev.org/695060 | 18:32 |
| *** jpena is now known as jpena|off | 18:32 | |
| *** abaindur has quit IRC | 18:46 | |
| *** armax has joined #openstack-neutron | 18:51 | |
| openstackgerrit | Merged openstack/neutron-fwaas master: Remove fwaas_v1 extensions https://review.opendev.org/692094 | 18:55 |
| *** ralonsoh has quit IRC | 19:03 | |
| *** dsneddon_ has joined #openstack-neutron | 19:18 | |
| *** abaindur has joined #openstack-neutron | 19:23 | |
| *** abaindur has quit IRC | 19:28 | |
| *** abaindur has joined #openstack-neutron | 19:28 | |
| *** abaindur has quit IRC | 19:29 | |
| *** abaindur has joined #openstack-neutron | 19:30 | |
| gmann | slaweq: amotoki mlavalle we need to get this in before devstack patch to default py3 is merged - https://review.opendev.org/#/c/695036/1 | 19:32 |
| *** ociuhandu has joined #openstack-neutron | 19:37 | |
| *** rkukura has quit IRC | 19:39 | |
| *** ociuhandu has quit IRC | 19:47 | |
| *** goldyfruit___ has joined #openstack-neutron | 19:55 | |
| *** dsneddon_ has quit IRC | 19:56 | |
| *** gmann is now known as gmann_afk | 19:57 | |
| *** goldyfruit_ has quit IRC | 19:57 | |
| *** dsneddon_ has joined #openstack-neutron | 19:58 | |
| *** dsneddon_ has quit IRC | 20:03 | |
| *** dsneddon_ has joined #openstack-neutron | 20:06 | |
| *** dsneddon_ has quit IRC | 20:11 | |
| *** dsneddon_ has joined #openstack-neutron | 20:12 | |
| *** root3 has joined #openstack-neutron | 20:17 | |
| *** root4 has quit IRC | 20:20 | |
| *** openstackgerrit has quit IRC | 20:35 | |
| *** gcheresh_ has joined #openstack-neutron | 20:37 | |
| *** ociuhandu has joined #openstack-neutron | 20:38 | |
| *** ociuhandu has quit IRC | 20:42 | |
| *** ociuhandu has joined #openstack-neutron | 20:43 | |
| *** igordc has quit IRC | 20:50 | |
| *** goldyfruit_ has joined #openstack-neutron | 20:58 | |
| *** ociuhandu has quit IRC | 20:58 | |
| *** goldyfruit___ has quit IRC | 21:00 | |
| *** gmann_afk is now known as gmann | 21:03 | |
| *** igordc has joined #openstack-neutron | 21:15 | |
| *** awalende has joined #openstack-neutron | 21:15 | |
| abaindur | eandersson: I am able to reproduce issue discussed yesterday and one i filed for https://bugs.launchpad.net/neutron/+bug/1853071 | 21:23 |
| openstack | Launchpad bug 1853071 in neutron "AMQP disconnects, q-reports-plugin queue grows, leading to DBDeadlocks while trying to update agent heartbeats" [Undecided,New] | 21:23 |
| *** rkukura has joined #openstack-neutron | 21:23 | |
| abaindur | in fact was able to repro it on a setup with just 3 hosts aand no user activity, in a stable state | 21:24 |
| abaindur | so basically add this iptables rule to block MYSQL traffic from neutron, where 988 is neutron user's ID: | 21:25 |
| abaindur | iptables -I OUTPUT -o lo -p tcp --dport 3306 --match owner --uid-owner 988 -j DROP | 21:26 |
| abaindur | let it stay like that for 5 minutes or so. you'll notice q-reports-plugin grow a little, and neutron-server logs will be blocked | 21:26 |
| eandersson | Out of interest is your MySQL servers behind a vip or connecting directly/ | 21:26 |
| abaindur | then remove that iptables rule, so SQL traffic resumes | 21:26 |
| eandersson | ah lo | 21:27 |
| abaindur | and bam. you will see AMQP broken pipes and missed heartbeats, and q-reports-plugin kesps growing | 21:27 |
| eandersson | Yea - I wonder if the cause is the same for us in our very different enviornment. | 21:27 |
| abaindur | mySQL is on the same env as neutron here. It's all in one controller VM | 21:27 |
| eandersson | I think we tracked it down to the revision table. | 21:27 |
| abaindur | this is just a test env with mysql on localhost:3306 and one neutron-server with 1 api worker, 1 rpc worker, 1 state report worker, etc... | 21:28 |
| abaindur | I can confirm that while we are seeing the AMQP errors in neutron, mysql shows neutron is stuck making repeated queries for outdates agent heartbeat timestamsps | 21:29 |
| *** ociuhandu has joined #openstack-neutron | 21:29 | |
| abaindur | mysql> SELECT host,time, INFO FROM INFORMATION_SCHEMA.PROCESSLIST where INFO is NOT NULL and DB ='neutron' ORDER BY time DESC LIMIT 30; | 21:29 |
| abaindur | +-----------------+------+----------------------------------------------------------------------------------------------------------------------------------+ | 21:29 |
| abaindur | | host | time | INFO | | 21:29 |
| abaindur | +-----------------+------+----------------------------------------------------------------------------------------------------------------------------------+ | 21:29 |
| abaindur | | localhost:51840 | 47 | UPDATE agents SET heartbeat_timestamp='2019-11-19 21:12:11.037474' WHERE agents.id = '5e0f0692-4f55-4a35-9efa-12aa866153a2' | | 21:29 |
| abaindur | | localhost:50636 | 20 | UPDATE agents SET heartbeat_timestamp='2019-11-19 20:54:45.458495' WHERE agents.id = '1d7c5726-a336-4156-ace5-952d5363ad50' | | 21:29 |
| abaindur | | localhost | 0 | SELECT host,time, INFO FROM INFORMATION_SCHEMA.PROCESSLIST where INFO is NOT NULL and DB ='neutron' ORDER BY time DESC LIMIT 30 | | 21:29 |
| abaindur | +-----------------+------+----------------------------------------------------------------------------------------------------------------------------------+ | 21:29 |
| *** awalende has quit IRC | 21:30 | |
| abaindur | looks like after time hits 60 seconds, it starts a brand new query (the localhost:<port_number> changes) | 21:30 |
| abaindur | Only thing is, not seeing the DBDeadlock ERRORs in logs yet. but maybe we will start seeing it in an hour or so | 21:31 |
| *** gcheresh_ has quit IRC | 21:36 | |
| abaindur | http://paste.openstack.org/show/786369/ | 21:41 |
| abaindur | not a sql expert but thats output of "SHOW ENGINE INNODB STATUS" which indicates some kind of deadlock | 21:43 |
| eandersson | Yea - it's an odd one. We tracked it down to it retrying too fast. | 21:44 |
| eandersson | But of course our issue could be very different. | 21:44 |
| *** awalende has joined #openstack-neutron | 21:45 | |
| *** ociuhandu has quit IRC | 21:48 | |
| *** lpetrut has quit IRC | 21:50 | |
| slaweq | mlavalle: hi, just short reminder as You wasn't on today's team meeting - it's Your week on bug deputy now. I hope You are aware of it :) | 21:51 |
| eandersson | > > mysql> UPDATE standardattributes SET revision_number=ZZZ, updated_at='<date>' WHERE standardattributes.id = YYYY AND standardattributes.revision_number = XXX; | 21:53 |
| eandersson | I was seeing this causing deadlocks for us after upgrade. We assumed the agent heartbeats deadlocks were caused by those. | 21:54 |
| *** rcernin has joined #openstack-neutron | 21:59 | |
| *** awalende has quit IRC | 22:01 | |
| *** rcernin has quit IRC | 22:01 | |
| *** rcernin has joined #openstack-neutron | 22:01 | |
| *** rcernin has quit IRC | 22:01 | |
| *** rcernin has joined #openstack-neutron | 22:02 | |
| *** nweinber__ has quit IRC | 22:05 | |
| *** openstackgerrit has joined #openstack-neutron | 22:12 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/networking-midonet master: Migrate grenade jobs to py3 https://review.opendev.org/695094 | 22:12 |
| *** awalende has joined #openstack-neutron | 22:14 | |
| *** priteau has quit IRC | 22:15 | |
| *** awalende has quit IRC | 22:19 | |
| *** awalende has joined #openstack-neutron | 22:21 | |
| mlavalle | slaweq: I am | 22:23 |
| abaindur | eandersson: we have ALSO been seeing occasional errors on standardattributes table. its on a update_floatingip_status RPC. but we couldn't tie with with the DB Deadlock/ rabbit queue issue we've been seeing now | 22:29 |
| *** awalende has quit IRC | 22:29 | |
| *** awalende has joined #openstack-neutron | 22:29 | |
| abaindur | DB exceeded retry limit.: StaleDataError: UPDATE statement on table 'standardattributes' expected to update 1 row(s); 0 were matched. | 22:29 |
| eandersson | Yep exactly that error | 22:33 |
| *** pcaruana has quit IRC | 22:33 | |
| eandersson | It looked like one of those was filling up the pipes for us. Whenever we restarted neutron it would drain the queues for 1-10 minutes before it started backing up again. | 22:34 |
| *** awalende has quit IRC | 22:35 | |
| eandersson | I remember one of the queries aggressively retrying. | 22:35 |
| *** slaweq has quit IRC | 22:39 | |
| *** ociuhandu has joined #openstack-neutron | 22:41 | |
| *** ociuhandu has quit IRC | 22:46 | |
| *** slaweq has joined #openstack-neutron | 22:48 | |
| *** munimeha1 has quit IRC | 22:49 | |
| *** slaweq has quit IRC | 22:53 | |
| *** slaweq has joined #openstack-neutron | 23:03 | |
| *** ociuhandu has joined #openstack-neutron | 23:07 | |
| *** tkajinam has joined #openstack-neutron | 23:07 | |
| *** abaindur has quit IRC | 23:07 | |
| *** ociuhandu has quit IRC | 23:09 | |
| *** abaindur has joined #openstack-neutron | 23:09 | |
| *** ociuhandu has joined #openstack-neutron | 23:11 | |
| *** armax has quit IRC | 23:13 | |
| *** armax has joined #openstack-neutron | 23:14 | |
| *** ociuhandu has quit IRC | 23:16 | |
| *** slaweq has quit IRC | 23:16 | |
| *** armax has quit IRC | 23:16 | |
| *** priteau has joined #openstack-neutron | 23:19 | |
| *** slaweq has joined #openstack-neutron | 23:25 | |
| *** slaweq has quit IRC | 23:30 | |
| *** ociuhandu has joined #openstack-neutron | 23:30 | |
| *** abaindur has quit IRC | 23:34 | |
| *** abaindur has joined #openstack-neutron | 23:36 | |
| *** ociuhandu has quit IRC | 23:40 | |
| *** slaweq has joined #openstack-neutron | 23:42 | |
| *** slaweq has quit IRC | 23:47 | |
| *** ivve has quit IRC | 23:54 | |
| *** slaweq has joined #openstack-neutron | 23:56 | |
| *** ociuhandu has joined #openstack-neutron | 23:58 | |
| *** goldyfruit_ has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!