14:02:23 <ajo> #startmeeting neutron_qos
14:02:24 <openstack> Meeting started Wed Sep 16 14:02:23 2015 UTC and is due to finish in 60 minutes.  The chair is ajo. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:02:25 <Irenab_> Hi
14:02:26 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:02:28 <openstack> The meeting name has been set to 'neutron_qos'
14:02:43 <ajo> Thanks everybody for joining, specially thanks to the nova folks for joining us: ndipanov , vladikr and johnthetubaguy
14:03:00 <johnthetubaguy> hi all
14:03:04 <ajo> We need to make some progress in a couple of topics regarding nova-neutron integration for QoS
14:03:45 <ajo> #topic nova-neutron integration : flavours?, or how do we do it
14:04:19 <ajo> First one, we need some way to make sure we can have some sort of association from the type of instance tenant picks to the QoS level he gets,
14:04:38 <ajo> or something equivalent, I know johnthetubaguy commented something about vif flavours? (could be?) but I'm unaware of what did you mean with that
14:04:50 <ajo> is that something gestating on the nova side?
14:05:00 <ihrachys> we may need it for stuff outside qos btw, like jumbo
14:05:08 <johnthetubaguy> no, I was thinking on the neutron side
14:05:16 <johnthetubaguy> really, I am thinking about volumes
14:05:27 <johnthetubaguy> with volumes you have your SSD vs spinning disk volumes
14:05:40 <johnthetubaguy> and they have have different QoS setup
14:05:53 <johnthetubaguy> and you pick what happens by selecting a volume type owned by cinder
14:06:06 <johnthetubaguy> I have always liked the idea of something similar in Neutron
14:06:10 <ajo> johnthetubaguy, ack, may be we could setup something like port flavors speciying the level of Qos + the type of nic, for example, and nova could reference that from it's own flavors in a key?
14:06:21 <johnthetubaguy> ideally no
14:06:30 <ajo> QoS policies are quite like flavors, but, dont' specify the type of nic, for example
14:06:30 <johnthetubaguy> users are meant to create ports, then pass them into nova
14:06:43 <johnthetubaguy> when they create the port, they need to set QoS
14:06:54 <ajo> johnthetubaguy, doesn't nova create the ports now?
14:07:04 <ajo> I mean, you can specify a port id, but you can specify the network
14:07:07 <johnthetubaguy> ajo: in the happy case, it does not
14:07:28 <johnthetubaguy> we have some code we want to deprecate that can create ports for you, mostly for backwards compatiblitiy
14:07:35 <johnthetubaguy> now thats an extreme view of the world
14:07:39 <ajo> johnthetubaguy, unless we enforce nova not to be able to do that, it's out of our control a bit, or nova may need to receive the "flavor" of port it will create
14:07:50 <ajo> net-id=xxxx,port-flavor=xxxx (or qos-policy=xxxxx)
14:07:56 <johnthetubaguy> but in theory, we build for that world, then add any sugar that you need to make that true
14:08:09 <ihrachys> johnthetubaguy: so basically you suggest to have no integration, and allow users to assign QoS policies to ports prior to passing them to nova, basically smth that we already have
14:08:10 <johnthetubaguy> Nova would just use a default port flavor, I would assume
14:08:42 <johnthetubaguy> ihrachys: so if that works already today, thats interesting
14:08:43 <ajo> ihrachys, in that case, we miss a way in QoS to enforce default policies to ports per tenant (via Role based access control, I guess)
14:08:47 <ajo> we need that anyway
14:09:04 <ajo> johnthetubaguy, or have one per tenant,
14:09:19 <ihrachys> ajo: it can be similar to default sec groups
14:09:23 <ajo> ok, at worst, we could just pass the policy id / or flavor, along with the net-id to nova, if that works...
14:09:36 <ajo> or we don't , encouraging the people to create the port first
14:09:44 <johnthetubaguy> well ideally neutron knows about the default volume type, and just deals with it when we don't specify one
14:09:57 <ihrachys> ajo: I actually think that preparing port for nova is the ideal situation, but some people complain about complexity behind neutron flow
14:10:00 <ajo> johnthetubaguy, ok, I guess that makes sense,
14:10:10 <johnthetubaguy> but if you want something other than the default, yeah, I would prefere them to create things up front really
14:10:10 <ndipanov> and when you do need to specify? you need to create a port first?
14:10:14 <ihrachys> not sure adding required explicit port allocation step will help
14:10:34 <ajo> ndipanov, yes, that's what johnthetubaguy means
14:10:46 <ajo> and I agree with ihrachys, we grow complexity of the user, but...
14:10:56 <ajo> only if they want to move to a non-default policy
14:11:09 <ihrachys> that said, I believe all complexity concerns should be hidden on client layer, not API
14:11:25 <johnthetubaguy> so if you want to do security groups, today, you create your ports upfront, else crazy things happen, as I understand it
14:11:29 <ihrachys> and if we can live with decoupled nova and neutron, then I am good to stay lazy
14:11:34 <johnthetubaguy> so I don't see any change here with qos
14:11:43 <ajo> decoupling is better
14:11:54 <johnthetubaguy> so my thing is more, we assume decoupled
14:11:57 <ihrachys> then why do you suggest integration in the first place? :)
14:12:02 <ajo> may be we need a third project to provide "flavors" ;)
14:12:17 <ajo> since an instance is comprised by volume, net, compute, etc...
14:12:18 <moshele> maybe it should be part of the openstack client project
14:12:19 <ajo> ;)
14:12:24 * ajo adds more complexity ;D
14:12:30 <johnthetubaguy> ajo: so thats how glance started, but lets not go there
14:12:54 <ajo> johnthetubaguy, I'm partly joking, but worried about usability
14:13:05 <johnthetubaguy> so the idea, originally was that heat would know about ports and servers, and help tie those together, its not really turned out that way
14:13:13 <ajo> johnthetubaguy, may be it's, as ihar said, something to be left to the client (horizon, cmdline..)
14:13:24 <johnthetubaguy> so here is the thing...
14:13:34 <johnthetubaguy> my view is that we should make it work with creating ports up front
14:13:43 <ihrachys> yeah, heat did not prove itself successful :(
14:13:45 <johnthetubaguy> if we need something else, then maybe, but that comes second
14:13:59 <johnthetubaguy> ihrachys: not sure, I think its doing some good stuff, just not really that
14:14:09 <ihrachys> johnthetubaguy: with ports prepared, it works now
14:14:20 <ajo> johnthetubaguy, that has to be assumed, it's on our agenda for M
14:14:31 <ihrachys> johnthetubaguy: re heat - more about expectations of universal adoption
14:14:42 <ajo> johnthetubaguy, but we're looking for the plan forward in coordination with nova. Will that work for users, do they need something more?
14:14:42 <johnthetubaguy> ihrachys: but it sounds like its two granular, and there are no flavors, quotas and policy?
14:15:11 <johnthetubaguy> ihrachys: ah, thats fair
14:15:14 <ihrachys> johnthetubaguy: there are qos policies that are attachable to ports
14:15:24 <johnthetubaguy> ihrachys: OK, so maybe we are there then
14:15:36 <johnthetubaguy> so what happens by default if I don't set any QoS policy?
14:15:39 <ihrachys> johnthetubaguy: if you mean neutron should have a global flavor that would be superset of qos policies and something else, then yes, it's not there
14:15:40 <johnthetubaguy> its just open?
14:15:41 <ajo> johnthetubaguy, we're only missing: when you create a port, what's the default policy for a given tenant
14:15:58 <ihrachys> johnthetubaguy: if no policy is set, no policy is applied
14:16:02 <ajo> or.. a flavor if we make a superset containing more details about ports
14:16:08 <ajo> (vnic type, qos policy, etc..)
14:16:09 <ihrachys> johnthetubaguy: there is no modifiable default policy
14:16:12 <johnthetubaguy> right, well, surely it might be per network? in the provider network case?
14:16:37 <johnthetubaguy> I think we should consider how default policies work when you create your ports before starting your VM, basically
14:16:43 <ihrachys> we can attach policies to networks already
14:16:50 <ihrachys> and ports inherit it
14:17:09 <ajo> true
14:17:20 <johnthetubaguy> ihrachys: cool, so maybe what we need is a default policy that gets attached to user created networks?
14:17:26 <ihrachys> I will also side note that neutron still does not have modifiable sec groups
14:17:31 <johnthetubaguy> although that sounds a little useless
14:17:38 <ihrachys> so here, we are consistent in not supporting it :)
14:17:40 <ajo> ok
14:17:45 <ajo> here we have two points of view
14:17:51 <ajo> 1) all left to port creation and neutron control
14:18:11 <ajo> and so controlled by the client (in regards of flavor relations if user wants that)
14:18:35 <ajo> 2) the port flavor/qos_policy key to flavors, so nova can tell neutron
14:18:42 <johnthetubaguy> but for 1) you do have nice controlled defaults, which should just work when nova creates the ports right?
14:18:52 <johnthetubaguy> so here is the problem...
14:18:58 <johnthetubaguy> you pick flavor 7
14:19:06 <johnthetubaguy> it says QoS policy A
14:19:18 <johnthetubaguy> but you pass in your own port with QoS policy B
14:19:21 <johnthetubaguy> now what does that mean?
14:19:47 <johnthetubaguy> leaving this all in Nova really side steps that issue
14:19:50 <johnthetubaguy> oops
14:19:54 <ihrachys> that's an interesting note
14:19:59 <johnthetubaguy> leaving this all in Neutron side steps that
14:20:10 <ajo> yes,
14:20:20 <ajo> that's a fair point, I was just thinking of that
14:20:33 <ajo> even if nova changed the QoS, user could move it to a different one if he wanted / had access to it
14:20:55 <ajo> so may be, quotas, enforcement and defaults should be left to neutron,
14:21:01 <ajo> and make horizon aware of it
14:21:26 <ajo> with some mechanism to be able to specify access & default policies for tenant network / port creation
14:21:46 <ajo> does that sound reasonable?
14:22:21 <ajo> I know irenab_ was rising the this topic of flavours initially, so irenab_ irenab , you may want to read the log later and ping us via mail if you see any concern
14:22:40 <ihrachys> yes, it seems to me no integration is actually needed
14:22:51 <Irenab_> Sure, will follow up later
14:23:02 <ajo> +1 johnthetubaguy , thanks for discussing this with us, the feedback has been very valuable
14:23:10 <ajo> so, let's move on
14:23:33 <ajo> #topic nova-neutron integration: bandwidth guarantees, and overcommitments...
14:23:47 <ajo> vladikr, ndipanov ^
14:23:52 <ajo> how do we handle that?
14:23:57 <ndipanov> well
14:24:05 <ajo> we can set a policy that says "guarantee 1Gbps on this port"
14:24:18 <ndipanov> it's tricky really
14:24:24 <ajo> but if it's plugged to the wrong compute node, or the wrong PF, it won't work
14:24:46 <ajo> ndipanov: nothing is easy lately :)
14:24:48 <ndipanov> so we are only assuming this when using physical ports (VFs)
14:25:01 <vladikr> PFs
14:25:02 <ajo> ndipanov: our goal is to do it also for virtio ports
14:25:10 <vladikr> ah
14:25:14 <ajo> but I understand that'd require neutron coordination
14:25:15 <ndipanov> ajo I thought so
14:25:25 <Irenab_> Are you talking about sriov only?
14:25:26 <ajo> as neutron knows what's the upstream port/route/tunnel for a network
14:25:48 <ajo> Irenab_: we may solve both, may be not both at the same time, as requisites and interactions could be different
14:26:02 <ndipanov> ok so the way nova scheduling currently works is the compute host publishes it's data about available resources
14:26:13 <ndipanov> so available bandwith would have to be published as well
14:26:14 <ajo> ndipanov, aha
14:26:36 <ajo> ndipanov, and also, be known internally on nova-compute for the PF/VFs relationship
14:26:40 <Irenab_> So nova will do bw bookkeeping?
14:26:43 <ndipanov> johnthetubaguy, similar things keep popping up with cinder (like proximity to volumes etc)
14:26:49 <ajo> may be you have 20G total in the compute, but 10G is to one PF, 10G is to other PF
14:26:53 <ndipanov> do you remember anything about that?
14:27:11 <ndipanov> ajo, right but that's a problem of structuring the data
14:27:22 <ajo> ndipanov, right, that's just internal to nova-compute
14:27:33 <ndipanov> what I am getting at is that we have no way of letting nova-compute know any of this in the first place right now
14:28:01 <ajo> ndipanov, what do you mean?, letting it know the available BW ?
14:28:06 <johnthetubaguy> so what ndipanov, I see it as something our scheduler refactoring efforts are heading towards
14:28:06 <ndipanov> yeah
14:28:30 <johnthetubaguy> oops, s/what ndipanov/what ndipanov said/
14:28:36 <ndipanov> johnthetubaguy, well yes :) eventually
14:29:15 <ajo> ok, so we may need to provide something to nova compute to discover the available bandwidth on the PFs, it's a matter of inspecting the link, or... otherwise get it given in config
14:29:27 <johnthetubaguy> ajo: once we add network segments, and I particularly interested in IP address availability, for example, but thats an asside
14:29:30 <ajo> I suspect for virtio's it's going to be harder
14:29:32 <ndipanov> ajo, but doesn't that get complicated with qos?
14:29:37 <ajo> since neutron may need to publish that info back to nova somehow
14:30:00 <ndipanov> ajo, currently there is no such an interface at all
14:30:07 <ajo> ndipanov, I know
14:30:13 <ajo> ndipanov, yes, then we put qos in place, and...
14:30:20 <johnthetubaguy> yeah, XenAPI has some basic BW reporting stuff, but I wouldn't say its a great pattern to copy for libvirt
14:30:20 <ndipanov> there are several ways to think about it
14:30:21 <ajo> ndipanov, nova may need to be aware of what the policy says
14:30:34 <ajo> ndipanov, or neutron needs to publish the policy to nova in a way nova understands
14:30:36 <johnthetubaguy> that is more usage focused, rather than scheduling
14:30:44 <ajo> or just the constrains out of the policy...
14:30:47 <ndipanov> johnthetubaguy, but it affects it no?
14:31:20 <johnthetubaguy> ndipanov: not the way we have things setup at least
14:31:49 <ndipanov> ajo, so in an ideal world - (global?) scheduler runs a neutron plugin that neutron agents publish data to, and none of this code lives in nova
14:31:57 <ndipanov> we are far away from that world atm
14:32:10 <ajo> ndipanov, ok, that sounds like a good ideal
14:32:12 <johnthetubaguy> ndipanov: +1 to both of those comments
14:32:33 <ajo> ndipanov, so neutron publishes a plugin, via stevedore or whatever... and the scheduler just takes it, and uses it
14:32:36 <johnthetubaguy> I always like to keep one eye to the future and one to the present, so its worth keeping that all in mind
14:32:46 <ndipanov> yes that would be ideal
14:33:04 <ndipanov> but that is blocked by a lot of work that needs to be done to split out the scheduler
14:33:04 <ajo> may be, it's better to tackle the SR-IOV thing first
14:33:09 <ajo> as it's the simplest case
14:33:19 <ajo> "simplest"
14:33:26 <ndipanov> yes - sr-iov is simpler because it is already handled by nova-compute
14:33:47 <ajo> ndipanov, you may only need to be aware of what "min bandwidth constraint" do we have on a certain port
14:34:18 <ndipanov> as in - the least amount of bandwith we can provide but we may be able to do more?
14:34:32 <ajo> ndipanov, isn't sbauza working on those scheduler improvements? or was he?
14:34:46 <ndipanov> ajo, he is but they are not easy with the current constraints nova has
14:34:57 <ndipanov> and they are not easy in general
14:35:01 <ajo> ndipanov: let me re-read your phrase
14:35:23 <ajo> ndipanov: correct
14:35:52 <ndipanov> so yeah that would be possible in some fashion for sriov ports but only for them without the scheduler work
14:35:57 <ajo> said that, neutron may need to somehow know that other ports on the same PF are min: restricted, and cap a max: on the unrestricted ports <- moshele
14:36:01 <ajo> Irenab_ ^
14:36:10 <ndipanov> there is one more way to do it
14:36:15 <ndipanov> and johnthetubaguy may disagree
14:36:30 <ajo> ndipanov: which one?
14:36:35 * ajo listens :)
14:36:59 <ndipanov> we had a notion of an extensible resource tracker, where you would be able to add your own resource plugins to nova
14:37:00 <Irenab_> In general i think neutron should be responsible for bookkeeping
14:37:19 <ajo> Irenab_: yes, that's why I like the point of a neutron plugin inside the nova scheduler :)
14:37:21 <ndipanov> the idea was interesting but the implementation was lacking because the general code has very loose apis
14:37:41 <ndipanov> but there is some work now to make those APIs stable and ... well useful
14:37:46 <Irenab_> +1 on this ajo
14:37:54 <ajo> ndipanov, can you link to those APIs?
14:37:56 <johnthetubaguy> ndipanov: I guess the summary of my position is: we need to make progress in the short term, if we agree the longer term plan, I am fine with a bit of compromise in the short term
14:38:08 <ajo> johnthetubaguy: +1
14:38:15 <ndipanov> well this is a compromise because we'd wait for jaypipes_ 's work
14:38:25 * ndipanov looks for the patches
14:38:50 <ajo> thanks ndipanov
14:39:11 <ndipanov> https://review.openstack.org/#/c/128992
14:39:26 <ndipanov> so this is the first patch
14:39:46 <ndipanov> this is still not ideal because nova has to own a lot of code that is basically a neutron implementation detail
14:39:56 <ndipanov> but at least it will have a well defined boundary within nova
14:40:02 <ajo> #link https://review.openstack.org/#/c/128992
14:40:03 <ndipanov> (I think)
14:40:07 <ndipanov> johnthetubaguy, ^
14:40:10 <ndipanov> ?
14:40:20 <ndipanov> does that sound not entirely false to you?
14:41:04 <ajo> Ok, I have 772 LOCs to read :)
14:41:30 <Irenab_> Same here 😊
14:41:35 <ajo> or at least ~300 not counting tests :)
14:41:51 <ndipanov> ajo read the related BP instead maybe
14:41:56 * johnthetubaguy looks
14:42:04 <ajo> yeah, ndipanov , that's a better idea in fact
14:42:42 <ajo> #link https://blueprints.launchpad.net/nova/+spec/resource-objects
14:43:04 <Irenab_> Ajo: what about overcommitment ?
14:43:23 <ajo> Irenab_, that's what we were talking about, preventing it
14:43:28 <johnthetubaguy> ndipanov: yeah, thats stuff is close now, seems worth following that
14:43:59 <ndipanov> ajo, so that would give you the place to do it in nova that nova folks won't hate
14:44:20 <Irenab_> I wonder if it always desired to be precented or should ve some provider policy
14:44:24 <ndipanov> ajo, but than we would want to have a unified view for both virt and phys ports
14:44:52 <ajo> ndipanov, yes,
14:44:58 <ajo> that's the longer term plan
14:45:02 <ajo> irenab_ not sure I follow
14:45:27 <ajo> ndipanov: for virt ports, neutron needs to feed data
14:45:49 <Irenab_> VM can be created and paused, other can be active
14:45:57 <ajo> ndipanov, another extra option (on the compute low level side), could be letting neutron decide the exact pci address/based on PF capabilities
14:46:14 <ajo> and there's where the new os-libvif thing could come in
14:46:27 * ajo looks for the link johnthetubaguy provided him this morning
14:46:46 <ajo> #link http://lists.openstack.org/pipermail/openstack-dev/2015-June/067783.html
14:47:36 <johnthetubaguy> maybe, I see the vif lib first making it easier to add support for driver XYZ vif
14:48:54 <ajo> ndipanov, is it possible/reasonable to have some kind of rfe/spec for M in regards of the basic min-bandwidth guarantees/not overcommitment in nova?
14:49:10 <ajo> johnthetubaguy ^  ?
14:49:28 <ajo> talking of SR-IOV only
14:50:09 <ajo> not asking for commitments, just trying to understand where do we stand
14:50:18 <ndipanov> ajo, yes - I actually proposed a session for SR-IOV
14:50:31 <ajo> ah, true, that could be in the same session
14:50:32 <ndipanov> for the summit
14:50:50 <ndipanov> well maybe - this touches on several more things like scheduling
14:50:55 <ajo> yeah,
14:51:26 <ajo> may be needs a separate one, yikes, too bad I can't go this year :/
14:51:47 <ajo> ndipanov: I proposed a cross project session for neutron
14:51:47 <ndipanov> and I am not too keen on sriov bandwith being a different thing than other bandwidth but such is life
14:51:48 <ajo> regarding this
14:51:56 <ndipanov> ajo, ok cool that makes sense
14:52:05 <ajo> let me find the link
14:52:56 <ajo> #link https://etherpad.openstack.org/p/neutron-mitaka-designsummit
14:53:04 <ajo> 4.
14:53:33 <Irenab_> ndipanov: i agree, it should be general approach to handle different vif types
14:54:01 <ajo> I just removed the "port defaults per tenant / RBAC" part from the etherpad
14:54:11 <ajo> makes no sense after our discussion and the other proposed solution
14:54:32 <ndipanov> yeah so it may make sense to be part of the vif lib
14:54:35 <ajo> Irenab_: ping me back if you believe the agreement we made misses something important when you read the minutes
14:55:01 <ndipanov> and then nova compute would just call it's reporting method when reporting it's own resources
14:55:15 <Irenab_> Ajo: will do
14:55:31 <ndipanov> but that seems like it can get arbitrarily complicated ...
14:55:49 <ajo> ndipanov, yep
14:55:53 <ndipanov> anyway - let's conclude - did you get some info to go on ajo ?
14:56:03 <ajo> ndipanov, we either add realtime interactions, or reporting so the other side can act quickly
14:56:25 <ajo> ndipanov: lots of information to digest
14:56:45 <ndipanov> ok ping me or johnthetubaguy on -nova if something is unclear I guess
14:56:46 <ajo> ndipanov, let us digest, and then let's organize more meetings
14:56:52 <ndipanov> yay meetings!
14:57:02 <ajo> ndipanov ':D let's minimize them
14:57:23 <ajo> ndipanov: if we had powerpoints this would be solved already.. (in the powerpoint)
14:57:36 <ndipanov> I love powerpoints
14:57:37 <ajo> s/powerpoint/presentations/g
14:57:38 <Irenab_> We can collaborate on etherpad
14:57:44 <ndipanov> 0 bugs reported to date
14:57:48 <ajo> ndipanov: lol
14:58:03 <ajo> Irenab_, what was the etherpad link you proposed initially?
14:58:07 <ajo> and let's close the meeting, 2 minutes to go
14:58:24 <ajo> I guess we can update that etherpad with the conclussions...
14:58:55 <Irenab_> Will send you once get tocomputer
14:59:00 <ajo> ok :)
14:59:04 <ajo> thanks everybody for joining
14:59:05 <Irenab_> Impossible from phone ....
14:59:14 <ajo> and providing feedback on the topic
14:59:19 <ajo> #endmeeting