| eandersson | > Checking segment: NetworkSegment | 00:01 |
|---|---|---|
| eandersson | This call is extremly noisy with debug on | 00:02 |
| eandersson | and we believe is extremly expensive | 00:03 |
| *** tosky has quit IRC | 00:17 | |
| *** wolverineav has joined #openstack-neutron | 00:17 | |
| *** wolverineav has quit IRC | 00:21 | |
| *** wolverineav has joined #openstack-neutron | 00:22 | |
| *** wolverineav has quit IRC | 00:27 | |
| openstackgerrit | sean mooney proposed openstack/neutron master: do not skip ports with unset or invalid ofport https://review.openstack.org/640258 | 00:39 |
| *** luksky has quit IRC | 00:45 | |
| *** ijw_ has quit IRC | 01:23 | |
| *** lujinluo has joined #openstack-neutron | 01:31 | |
| *** wolverineav has joined #openstack-neutron | 01:37 | |
| *** lujinluo has quit IRC | 01:38 | |
| *** wolverineav has quit IRC | 01:44 | |
| dougwig | eandersson: from what release to what? | 02:11 |
| eandersson | mitaka to rocky | 02:11 |
| dougwig | we had massive cpu spikes from liberty -> pike. | 02:11 |
| eandersson | Did you figure out the cause? | 02:12 |
| dougwig | most of ours was around security groups; i've submitted a few patches for that, and have a few more to go. the switch from sqlalchemy at the api layer to neutron objects is more cpu time. we use calico, so the other plugin operations weren't affected as much. | 02:13 |
| dougwig | which plugins are you using? | 02:13 |
| dougwig | that's also why this change was submitted, so it's quicker to see which task is chewing the cpu: https://review.openstack.org/#/c/637019/ | 02:14 |
| eandersson | ml2 / linuxbridge / plus l3 (cisco)/lbaas | 02:16 |
| eandersson | We are seeing some really expensive calls > Checking segment: NetworkSegment | 02:16 |
| eandersson | from the ml2 driver | 02:16 |
| dougwig | linuxbridge with flat, vlan, vxlan ? | 02:16 |
| *** yamamoto has joined #openstack-neutron | 02:17 | |
| dougwig | if you've got a way to run cProfile on the suspect calls, we can see what's taking all the time. | 02:17 |
| eandersson | vlan | 02:18 |
| eandersson | We are seeing a couple of computes with | 02:19 |
| eandersson | > ; Stdout: ; Stderr: iptables-restore v1.4.21: unknown option "--dport" | 02:19 |
| dougwig | i've never seen that. i wonder if that could be causing spinning, if the expected state never matches up. | 02:20 |
| *** yamamoto has quit IRC | 02:22 | |
| eandersson | Yea - thinking that it could cause that | 02:24 |
| *** yamamoto has joined #openstack-neutron | 02:26 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack/networking-midonet master: Pin astroid versions https://review.openstack.org/640614 | 02:36 |
| eandersson | dougwig, it looks to be failing at something like this > -s <ip>/32 -p vrrp --dport 112 -j RETURN | 02:36 |
| eandersson | *iptables -s <ip>/32 -p vrrp --dport 112 -j RETURN | 02:36 |
| openstackgerrit | Merged openstack/neutron master: Cleanup DVR SNAT port list creation code https://review.openstack.org/636994 | 02:39 |
| eandersson | So looking at Mitaka it was not adding the dport or the source ip | 02:46 |
| eandersson | > iptables -s <ip>/32 -p vrrp --dport 112 -j RETURN | 02:46 |
| eandersson | > iptables -p vrrp -j RETURN | 02:46 |
| eandersson | So looks like neutron is generating bad iptables | 02:47 |
| openstackgerrit | YAMAMOTO Takashi proposed openstack/networking-midonet master: Pin astroid versions https://review.openstack.org/640614 | 02:53 |
| *** yamamoto has quit IRC | 02:53 | |
| *** yamamoto has joined #openstack-neutron | 02:57 | |
| *** yamamoto has quit IRC | 03:09 | |
| *** cheng1 has joined #openstack-neutron | 03:10 | |
| *** yamamoto has joined #openstack-neutron | 03:13 | |
| *** yamamoto has quit IRC | 03:16 | |
| *** yamamoto has joined #openstack-neutron | 03:16 | |
| *** yamamoto has quit IRC | 03:20 | |
| *** yamamoto has joined #openstack-neutron | 03:32 | |
| *** yamamoto has quit IRC | 03:32 | |
| *** yamamoto has joined #openstack-neutron | 03:33 | |
| *** yamamoto has quit IRC | 03:37 | |
| eandersson | So still not sure why, but something or someone created a VRRP rule like this | 03:38 |
| eandersson | openstack security group rule create xxx --protocol vrrp --ingress --remote-ip <ip> --dst-port 112 | 03:38 |
| eandersson | This breaks the compute, as it can no longer apply the iptables | 03:38 |
| eandersson | Probably a pretty critical bug, as it allows you to break computes | 03:39 |
| *** tbachman has quit IRC | 03:54 | |
| *** lujinluo has joined #openstack-neutron | 04:02 | |
| *** lujinluo has quit IRC | 04:06 | |
| *** wolverineav has joined #openstack-neutron | 04:24 | |
| *** wolverineav has quit IRC | 04:32 | |
| dougwig | Yes, did you file it? | 04:36 |
| *** yamamoto has joined #openstack-neutron | 04:44 | |
| eandersson | dougwig, https://bugs.launchpad.net/neutron/+bug/1818385 | 04:44 |
| openstack | Launchpad bug 1818385 in neutron "It's possible to add a security group rule for VRRP with a dport" [Undecided,New] | 04:44 |
| *** yamamoto has quit IRC | 04:48 | |
| dougwig | Awesome, thanks | 05:00 |
| *** sapd1 has quit IRC | 05:09 | |
| *** sapd1 has joined #openstack-neutron | 05:10 | |
| *** cbrumm_ has joined #openstack-neutron | 05:11 | |
| *** yamamoto has joined #openstack-neutron | 05:18 | |
| *** yamamoto has quit IRC | 05:23 | |
| openstackgerrit | Doug Wiegley proposed openstack/neutron master: For vrrp sg rules, do not emit dport option https://review.openstack.org/640619 | 05:38 |
| *** colin- has joined #openstack-neutron | 06:07 | |
| *** wolverineav has joined #openstack-neutron | 06:13 | |
| *** colin- has quit IRC | 06:27 | |
| *** gcheresh_ has joined #openstack-neutron | 06:30 | |
| *** yamamoto has joined #openstack-neutron | 06:35 | |
| openstackgerrit | Doug Wiegley proposed openstack/neutron master: When converting sg rules to iptables, do not emit dport if not supported https://review.openstack.org/640619 | 06:36 |
| *** yamamoto has quit IRC | 06:40 | |
| *** colin- has joined #openstack-neutron | 06:41 | |
| *** yamamoto has joined #openstack-neutron | 06:45 | |
| *** yamamoto has quit IRC | 06:58 | |
| *** yamamoto has joined #openstack-neutron | 06:58 | |
| *** cheng1 has quit IRC | 07:19 | |
| *** cheng1 has joined #openstack-neutron | 07:20 | |
| *** abaindur has joined #openstack-neutron | 07:27 | |
| *** wolverineav has quit IRC | 07:27 | |
| *** pck has quit IRC | 07:29 | |
| *** slaweq has joined #openstack-neutron | 07:37 | |
| *** yamamoto has quit IRC | 07:45 | |
| *** yamamoto has joined #openstack-neutron | 07:51 | |
| *** yamamoto has quit IRC | 07:51 | |
| *** yamamoto has joined #openstack-neutron | 07:52 | |
| *** yamamoto has quit IRC | 07:57 | |
| *** wolverineav has joined #openstack-neutron | 07:59 | |
| *** colin- has quit IRC | 08:07 | |
| *** lujinluo has joined #openstack-neutron | 08:15 | |
| *** lujinluo has quit IRC | 08:20 | |
| *** cheng1 has quit IRC | 08:25 | |
| *** luksky has joined #openstack-neutron | 08:25 | |
| *** abaindur has quit IRC | 08:25 | |
| *** yamamoto has joined #openstack-neutron | 08:36 | |
| *** yamamoto has quit IRC | 08:41 | |
| *** wolverineav has quit IRC | 08:47 | |
| openstackgerrit | Kailun Qin proposed openstack/neutron master: Support Network Segment Range CRUD as extensions https://review.openstack.org/624708 | 08:59 |
| openstackgerrit | Kailun Qin proposed openstack/neutron master: Use network segment ranges for segment allocation https://review.openstack.org/624709 | 08:59 |
| openstackgerrit | Kailun Qin proposed openstack/neutron master: Devstack plugin for network-segment-range api https://review.openstack.org/638386 | 08:59 |
| *** wolverineav has joined #openstack-neutron | 09:17 | |
| *** wolverineav has quit IRC | 09:18 | |
| *** wolverineav has joined #openstack-neutron | 09:19 | |
| *** abaindur has joined #openstack-neutron | 09:23 | |
| *** abaindur has quit IRC | 09:27 | |
| *** hjensas has quit IRC | 09:35 | |
| *** hjensas has joined #openstack-neutron | 09:37 | |
| *** wolverineav has quit IRC | 09:39 | |
| *** yamamoto has joined #openstack-neutron | 10:24 | |
| *** yamamoto has quit IRC | 10:29 | |
| openstackgerrit | Adrian Chiris proposed openstack/neutron master: Delete port binding level for deleted bindings https://review.openstack.org/634276 | 10:30 |
| *** yamamoto has joined #openstack-neutron | 10:58 | |
| *** yamamoto has quit IRC | 11:02 | |
| *** wolverineav has joined #openstack-neutron | 11:03 | |
| *** wolverineav has quit IRC | 11:08 | |
| openstackgerrit | Kailun Qin proposed openstack/neutron master: Support Network Segment Range CRUD as extensions https://review.openstack.org/624708 | 11:10 |
| openstackgerrit | Kailun Qin proposed openstack/neutron master: Use network segment ranges for segment allocation https://review.openstack.org/624709 | 11:10 |
| openstackgerrit | Kailun Qin proposed openstack/neutron master: Devstack plugin for network-segment-range api https://review.openstack.org/638386 | 11:10 |
| *** yamamoto has joined #openstack-neutron | 11:21 | |
| *** yamamoto has quit IRC | 11:28 | |
| *** yamamoto has joined #openstack-neutron | 11:30 | |
| *** yamamoto has quit IRC | 11:30 | |
| *** yamamoto has joined #openstack-neutron | 11:30 | |
| *** tosky has joined #openstack-neutron | 11:35 | |
| openstackgerrit | Kailun Qin proposed openstack/neutron master: Support Network Segment Range CRUD as extensions https://review.openstack.org/624708 | 11:40 |
| openstackgerrit | Kailun Qin proposed openstack/neutron master: Use network segment ranges for segment allocation https://review.openstack.org/624709 | 11:40 |
| openstackgerrit | Kailun Qin proposed openstack/neutron master: Devstack plugin for network-segment-range api https://review.openstack.org/638386 | 11:40 |
| *** salmankhan has joined #openstack-neutron | 11:48 | |
| *** yamamoto has quit IRC | 11:53 | |
| *** yamamoto has joined #openstack-neutron | 12:00 | |
| *** lujinluo has joined #openstack-neutron | 12:12 | |
| *** lujinluo has quit IRC | 12:17 | |
| openstackgerrit | Kailun Qin proposed openstack/neutron-lib master: standard-attr for network-segment-range api-ref https://review.openstack.org/640633 | 12:18 |
| *** yamamoto_ has joined #openstack-neutron | 12:19 | |
| *** yamamoto has quit IRC | 12:22 | |
| *** salmankhan has quit IRC | 12:27 | |
| *** dayou has quit IRC | 12:48 | |
| *** dayou has joined #openstack-neutron | 12:50 | |
| openstackgerrit | Kailun Qin proposed openstack/neutron-lib master: standard-attr for network-segment-range api-ref https://review.openstack.org/640633 | 13:03 |
| *** elod has quit IRC | 13:04 | |
| *** elod has joined #openstack-neutron | 13:04 | |
| *** elod has quit IRC | 13:06 | |
| *** elod has joined #openstack-neutron | 13:11 | |
| *** yamamoto_ has quit IRC | 13:27 | |
| openstackgerrit | Slawek Kaplonski proposed openstack/neutron master: Revert "Restart all L3 agents in test that check no packet loss in HA routers" https://review.openstack.org/640634 | 13:31 |
| *** cheng1 has joined #openstack-neutron | 13:42 | |
| *** cheng1 has quit IRC | 13:48 | |
| *** yamamoto has joined #openstack-neutron | 13:54 | |
| *** dayou has quit IRC | 14:02 | |
| *** dayou has joined #openstack-neutron | 14:03 | |
| openstackgerrit | Merged openstack/neutron master: DVR edge router: avoid accidental centralized floating IP remove https://review.openstack.org/638648 | 14:21 |
| *** cshen has joined #openstack-neutron | 14:35 | |
| *** cshen has quit IRC | 14:57 | |
| openstackgerrit | Merged openstack/neutron master: Placement reporting service plugin https://review.openstack.org/580672 | 14:57 |
| *** cshen has joined #openstack-neutron | 15:05 | |
| *** snapiri has quit IRC | 15:09 | |
| *** cshen has quit IRC | 15:10 | |
| *** cshen has joined #openstack-neutron | 15:20 | |
| *** cshen has quit IRC | 15:25 | |
| *** gcheresh_ has quit IRC | 15:57 | |
| openstackgerrit | Ryan Tidwell proposed openstack/neutron-dynamic-routing master: Implement DVR-aware fixed IP lookups https://review.openstack.org/581098 | 16:00 |
| *** lujinluo has joined #openstack-neutron | 16:14 | |
| *** lujinluo has quit IRC | 16:18 | |
| *** yamamoto has quit IRC | 16:33 | |
| openstackgerrit | Merged openstack/neutron master: [Functional tests] Change way how conntrack entries are checked https://review.openstack.org/638635 | 16:40 |
| *** yamamoto has joined #openstack-neutron | 17:13 | |
| *** jmlowe has quit IRC | 17:15 | |
| *** yamamoto has quit IRC | 17:18 | |
| *** cshen has joined #openstack-neutron | 17:21 | |
| *** cshen has quit IRC | 17:25 | |
| *** jmlowe has joined #openstack-neutron | 17:36 | |
| *** njohnston_ has joined #openstack-neutron | 17:42 | |
| *** njohnston_ has quit IRC | 17:44 | |
| *** vishwanathj has joined #openstack-neutron | 18:15 | |
| *** lujinluo has joined #openstack-neutron | 18:18 | |
| *** lujinluo has quit IRC | 18:22 | |
| *** colin- has joined #openstack-neutron | 18:28 | |
| *** armax has quit IRC | 18:34 | |
| *** lujinluo has joined #openstack-neutron | 18:50 | |
| *** tidwellr has quit IRC | 18:54 | |
| *** lujinluo has quit IRC | 18:55 | |
| *** rkukura has quit IRC | 18:55 | |
| *** cshen has joined #openstack-neutron | 19:00 | |
| *** cshen has quit IRC | 19:04 | |
| *** wolverineav has joined #openstack-neutron | 19:07 | |
| *** wwriverrat has joined #openstack-neutron | 19:09 | |
| *** cshen has joined #openstack-neutron | 19:13 | |
| *** cshen has quit IRC | 19:18 | |
| *** igordc has joined #openstack-neutron | 19:23 | |
| *** dsneddon has quit IRC | 19:24 | |
| *** wolverineav has quit IRC | 19:41 | |
| *** hongbin has joined #openstack-neutron | 20:09 | |
| *** dsneddon has joined #openstack-neutron | 20:12 | |
| *** dsneddon has quit IRC | 20:17 | |
| *** lujinluo has joined #openstack-neutron | 20:34 | |
| *** hongbin has quit IRC | 20:39 | |
| *** hongbin_ has joined #openstack-neutron | 20:39 | |
| *** dave-mccowan has joined #openstack-neutron | 20:54 | |
| *** wolverineav has joined #openstack-neutron | 21:02 | |
| *** wolverineav has quit IRC | 21:07 | |
| *** ijw has joined #openstack-neutron | 21:07 | |
| *** ijw_ has joined #openstack-neutron | 21:10 | |
| *** ijw has quit IRC | 21:12 | |
| *** cshen has joined #openstack-neutron | 21:14 | |
| *** cshen has quit IRC | 21:18 | |
| *** tbachman_ has joined #openstack-neutron | 21:25 | |
| *** lujinluo has quit IRC | 21:33 | |
| *** ijw_ has quit IRC | 21:37 | |
| *** jmlowe has quit IRC | 21:37 | |
| *** ijw has joined #openstack-neutron | 21:38 | |
| *** ijw has quit IRC | 21:49 | |
| *** ijw_ has joined #openstack-neutron | 21:49 | |
| *** jmlowe has joined #openstack-neutron | 21:50 | |
| *** jmlowe has quit IRC | 21:50 | |
| *** ijw has joined #openstack-neutron | 21:51 | |
| *** abaindur has joined #openstack-neutron | 21:51 | |
| *** ijw_ has quit IRC | 21:54 | |
| *** ijw_ has joined #openstack-neutron | 21:58 | |
| *** ijw has quit IRC | 22:01 | |
| *** dsneddon has joined #openstack-neutron | 22:13 | |
| *** wolverineav has joined #openstack-neutron | 22:14 | |
| *** dsneddon has quit IRC | 22:17 | |
| *** wolverineav has quit IRC | 22:19 | |
| *** mlavalle has joined #openstack-neutron | 22:27 | |
| openstackgerrit | Slawek Kaplonski proposed openstack/neutron master: Spawn metadata proxy on dvr ha standby routers https://review.openstack.org/639979 | 22:37 |
| *** abaindur has quit IRC | 22:39 | |
| *** jmlowe has joined #openstack-neutron | 22:39 | |
| *** dave-mccowan has quit IRC | 22:45 | |
| *** jmlowe has quit IRC | 22:47 | |
| *** wolverineav has joined #openstack-neutron | 22:54 | |
| *** tkajinam has joined #openstack-neutron | 22:55 | |
| *** hongbin_ has quit IRC | 22:55 | |
| *** wolverineav has quit IRC | 22:58 | |
| *** slaweq has quit IRC | 23:00 | |
| *** rkukura has joined #openstack-neutron | 23:03 | |
| *** slaweq has joined #openstack-neutron | 23:08 | |
| *** slaweq has quit IRC | 23:13 | |
| *** cshen has joined #openstack-neutron | 23:14 | |
| *** cshen has quit IRC | 23:19 | |
| *** ijw_ has quit IRC | 23:29 | |
| *** ijw has joined #openstack-neutron | 23:29 | |
| *** tosky has quit IRC | 23:30 | |
| *** tbachman_ has quit IRC | 23:32 | |
| *** ijw has quit IRC | 23:34 | |
| mlavalle | dougwig: please see what you think of my comments to https://review.openstack.org/#/c/640619 | 23:50 |
| mlavalle | dougwig: thanks for proposing a fix so quickly for that bug | 23:50 |
| mlavalle | dougwig: I think we can simplify a bit | 23:50 |
| *** wolverineav has joined #openstack-neutron | 23:55 | |
| *** sapd1 has quit IRC | 23:56 | |
| *** vishwanathj has quit IRC | 23:57 | |
| *** wolverineav has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!