14:00:04 <slaweq> #startmeeting neutron_drivers
14:00:05 <openstack> Meeting started Fri Jan 17 14:00:04 2020 UTC and is due to finish in 60 minutes.  The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:06 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:00:08 <openstack> The meeting name has been set to 'neutron_drivers'
14:00:10 <slaweq> hi
14:00:12 <ralonsoh> hi
14:00:14 <yamamoto> hi
14:00:14 <mlavalle> o/
14:01:11 <haleyb> hi
14:02:12 <slaweq> ok, I think we can start
14:02:21 <slaweq> amotoki: told me that he may not be here today
14:02:28 <slaweq> #topic RFEs
14:02:28 <njohnston> o/
14:02:35 <slaweq> we have 2 RFEs for today
14:03:00 <slaweq> first one is old and I think we were discuss about it some time ago:
14:03:03 <slaweq> https://bugs.launchpad.net/neutron/+bug/1828494
14:03:03 <openstack> Launchpad bug 1828494 in neutron "[RFE][L3] l3-agent should have its capacity" [Wishlist,In progress] - Assigned to LIU Yulong (dragon889)
14:05:27 <liuyulong> hi
14:05:35 <liuyulong> I'm here
14:06:09 <liuyulong> #link https://review.opendev.org/#/q/topic:bug/1828494+(status:open+OR+status:merged)
14:06:16 <liuyulong> we have the spec and the code already.
14:06:30 <slaweq> liuyulong: did You read my last comment there?
14:06:43 <slaweq> would it solve Your problem?
14:08:23 <liuyulong> Hmm, one problem, one single router will have unlimited bandwidth.
14:08:54 <liuyulong> I mean for centralized floating IPs.
14:09:37 <slaweq> sorry but I don't understand now, is this RFE about node "capacity" or bandwidth limitations (QoS)?
14:10:24 <liuyulong> But the limit of router quantity is indeed a simple way.
14:11:08 <liuyulong> slaweq, no, not qos, amount of bandwidth for one single router can serve.
14:11:32 <slaweq> but You can limit that with QoS already, no?
14:12:12 <liuyulong> Sum of all floating IPs' bandwidth for one single router.
14:13:15 <ralonsoh> is this a read-only parameter or can you enforce this limitation when adding a port to a router?
14:13:37 <ralonsoh> or changing the qos parameter of a port belonging to a router?
14:13:42 <liuyulong> There are two levels sum of bandwidth for this spec.
14:14:18 <liuyulong> ralonsoh, https://review.opendev.org/#/c/661492/7/neutron/extensions/_router_total_bandwidth.py@45
14:14:25 <liuyulong> ralonsoh, it is configurable.
14:15:09 <slaweq> liuyulong: and how this bandwidth should be enforced on backend?
14:15:13 <ralonsoh> so this is another max BW enforcing level
14:15:45 <liuyulong> slaweq, One single router has max limitation of bandwidth. One single host has limitation as weel.
14:15:47 <haleyb> are you enforcing bandwidth via some mechanism?  otherwise router A (10Gps) and router B (1Gps) are equal, right?
14:16:19 <liuyulong> haleyb, no, just a simple number.
14:17:36 <ralonsoh> hmm now I don't get it: this is going to be just an informative number, no enforcement at all
14:17:41 <ralonsoh> ?
14:18:25 <liuyulong> Ohh, sorry, it will.
14:19:05 <mlavalle> who keeps the counting of the actual bandwidth usage in a L3 agent?
14:19:14 <slaweq> ok, so You want to set e.g. max_router_bandwidth to let's say 1G, this will be limit for all router's ports or per port?
14:19:24 <mlavalle> and I mean ***actual***
14:19:54 <liuyulong> mlavalle, L3 related DB could result such amount.
14:20:44 <mlavalle> mhhh, so all of this is declarative, right? no actual tracking of how much total bandwidth is being used at the NIC, right?
14:22:19 <liuyulong> One host has 10G bandwidth so it can host 10 1G routers.
14:22:31 <mlavalle> I get that
14:22:35 <slaweq> liuyulong: this is very simple example what You said
14:22:44 <liuyulong> One router has max 1G bandwidth, it can allow 10 100Mbps floating IPs.
14:22:47 <slaweq> liuyulong: one host can be connected to more than one physical networks
14:22:57 <mlavalle> what I don't get is whether actual bandwidth usage is tracked or not?
14:23:03 <slaweq> and e.g. external traffic will go with different NIC than tenant's traffic
14:23:19 <slaweq> how You want to know how much bandwidth for each of those NICs is available?
14:23:48 <liuyulong> mlavalle, for the physical host level, it is based on the router's bandwidth.
14:24:35 <liuyulong> mlavalle, for router level, it will be tracked by its all floating IPs.
14:25:30 <mlavalle> but in this approach I can create a router declaring 1GB, say, and then actually use 10GB, right? or how do you actually enforce the 1GB I declared?
14:25:41 <haleyb> liuyulong: but isn't the bandwidth based on a physical nic property?  wouldn't the l3-agent need to find this and report it?
14:27:20 <liuyulong> slaweq, good question, for multiple external networks, lazy loading router should be used (it will be another story). Otherwise, just use the number as you suggest.
14:27:44 <liuyulong> haleyb, it will be reported, https://review.opendev.org/#/c/658451/7/specs/ussuri/l3-agent-capacity.rst@96
14:27:50 <slaweq> liuyulong: so should this be only for external networks (Floating IPs)?
14:27:57 <slaweq> or for all router's ports?
14:28:45 <liuyulong> mlavalle, no, all floating IPs under that 1G router will not exceed that limitation.
14:29:03 <liuyulong> mlavalle, I mean all floating IPs bandwidth sum.
14:29:22 <liuyulong> slaweq, external networks only
14:29:46 <slaweq> for floating IPs why You can't limit bandwidth with gateway_ip_qos
14:29:47 <slaweq> ?
14:30:45 <liuyulong> slaweq, disable SNAT, or just a very low bandwidth for user for free.
14:32:00 <liuyulong> Allow me quota some background here:
14:32:02 <liuyulong> Neutron now assumes that L3 agent has unlimited capacity, so it will
14:32:02 <liuyulong> continuously schedule new router to agents. For a large scale public
14:32:02 <liuyulong> cloud it meets scale issue about L3-agent. And mostly, cloud service
14:32:02 <liuyulong> provider does not charge for the neutron virtual router. This can become
14:32:02 <liuyulong> a headache for the operators. Neutron will create many resource for
14:32:03 <liuyulong> routers, especially the HA scenario, there will be namespaces, keepalived
14:32:04 <slaweq> and another problem with this "bandwidth based scheduling" - what if router only with internal ports is already scheduled to some node and than user will connect it to external network too? will it be migrated to other node if there is no bandwidth on existing node? Or will it be oversubscribed?
14:32:05 <liuyulong> processes, and monitor processes and so on. It will absolutely
14:32:07 <liuyulong> increase the failure risk, especially for agent restart. If one agent
14:32:09 <liuyulong> goes down, scheduling more routers, causing more users to be affected.
14:33:21 <slaweq> liuyulong: I ready Your spec and this problem description, but I still don't understand how using resources like memory, cpu by e.g. keepalived can be related to bandwidth
14:33:31 <slaweq> IMO those are 2 different problems
14:34:25 <slaweq> 1. problem which You described - too many "not really" used routers created which may lead to lack of networker node resources, like cpu and ram
14:35:07 <slaweq> 2. guarantee bandwidth for routers which are actually used and have traffic
14:35:23 <slaweq> or am I missunderstood something?
14:37:17 <liuyulong> slaweq, lazy load, for centralized network node, everything can be created until the router gateway is created, yes for dvr+ha router only. For HA/legacy router, could just use the max number of router as you suggest.
14:38:37 <liuyulong> slaweq, when you try to add gateway, the schedule may get failed, it should be raised to API, or set router to ERROR state.
14:40:29 <liuyulong> Actually for most public cloud, they abstract a concept of VPC which by default includes one router with external network, one network, and one subnet connected to router.
14:40:57 <slaweq> liuyulong: but we can't assume that it's the only possible use case :)
14:41:24 <haleyb> and setting a router to ERROR without an API failure leads to a bad user experience :(
14:41:26 <liuyulong> In the very beginning of router creating, the request will include the external_gateway attribute.
14:41:57 <liuyulong> haleyb, better than unlimited scheduling, I guess.
14:43:35 <liuyulong> So if router created failed, the 'VPC' will not be created. Failed at API level, better than cluster crush.
14:44:24 <slaweq> and what about HA routers? will it "consume" bandwidth on all 3 (by default) nodes? or only on Master?
14:44:50 <liuyulong> Especially in large-scale public cloud, network nodes always face high probability of failure
14:47:01 <liuyulong> slaweq, for simple scheduling and implementation we consider the all nodes since we have ratio of host bandwidth.
14:47:15 <slaweq> another question: why not use placement-api to which we are already reporting bandwidth for minumum bw qos?
14:48:44 <ralonsoh> ^ exactly, this is a perfect example to track a resource (and nested ones) with placement
14:49:10 <njohnston> ^ that is what I was wondering as well
14:50:04 <liuyulong> slaweq, I'm not familiar with placement, and personally I'm not a fan of involving new component for neutron.
14:50:31 <liuyulong> nova has pain on placement for a long time.
14:51:10 <slaweq> liuyulong: but that  doesn't mean that we should reinvent everything in Neutron, right? At least that's my opinion :)
14:53:15 <liuyulong> It's not totally unacceptable, just need someone to help me. : )
14:53:56 <liuyulong> For now the approach is adding a new attribute for router, and add something in agent configuration dict.
14:54:27 <liuyulong> Easy for implementation, easy use.
14:55:46 <slaweq> IMHO for simple solution, scheduling based on number of routers simply could also works and would be easier and for bandwidth driven scheduling we should try to find good architecture of such solution, maybe using bits from placement and QoS which we already have
14:55:58 <slaweq> but that's only my opinion
14:56:43 <slaweq> we are almost on top of the hour so now my question is: what we should do with this RFE now? Do we want some more data to be added to bug/spec? what it should be then?
14:58:09 <liuyulong> And from my understanding of min bandwidth of VM scheduling, it is also mainly for compute things. Router and floating IP bandwidth are networking resources.
14:58:11 <mlavalle> continue the conversation in the spec and the code
14:59:03 <slaweq> ok, so please find some time to review the spec and lets continue discussion there
14:59:10 <liuyulong> OK, we can discuss this again.
14:59:18 <slaweq> patch with spec is here:
14:59:20 <slaweq> #link https://review.opendev.org/#/c/658451/7
14:59:42 <slaweq> ok, thx for attending and for good discussion all
14:59:47 <slaweq> have a great weekend
14:59:52 <mlavalle> o/
14:59:56 <slaweq> and see You online next week
14:59:57 <liuyulong> bye
14:59:59 <slaweq> #endmeeting