19:01:07 <mestery> #startmeeting networking_policy
19:01:08 <openstack> Meeting started Thu Feb 20 19:01:07 2014 UTC and is due to finish in 60 minutes.  The chair is mestery. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:10 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:12 <openstack> The meeting name has been set to 'networking_policy'
19:01:26 <mestery> banix: Hi there!
19:01:41 <mestery> SumitNaiksatam rkukura: ping
19:01:51 <SumitNaiksatam> mestery, all: Hi!
19:01:53 <banix> mestery: hi
19:02:02 <mestery> #link https://wiki.openstack.org/wiki/Meetings/Neutron_Group_Policy Agenda
19:02:07 <SumitNaiksatam> sorry, was having trouble with my IRC client
19:02:09 <rkukura> mestery: hi
19:02:23 <mestery> So, lets get started with covering action items from last week.
19:02:51 <mestery> Per the fact we're all here, I've updated the wiki with the new meeting time slot. :)
19:03:11 <mestery> SumitNaiksatam: Thanks for updating hte meeting wiki with temporary github access instructions!
19:03:20 <mestery> And mandeep, thanks for the collaboration instructions as well!
19:03:20 <SumitNaiksatam> mestery: sure, np
19:03:40 <mestery> If anyone has questions on those, please feel free to ask now.
19:03:58 <banix> Is code already added wrt API
19:04:10 <mestery> banix: That's the next topic :)
19:04:15 <mestery> But w can move there.
19:04:20 <mestery> #topic API
19:04:21 <s3wong> banix: I haven't push anything out yet, sorry
19:04:26 <banix> mestery: sorry for jumping the gun
19:04:30 <mestery> #link https://github.com/noironetworks/neutron-group-policy/tree/sumit/api API Branch
19:04:44 <mestery> s3wong: SumitNaiksatam has taken your patch and pushed it to a branch I believe.
19:04:54 <SumitNaiksatam> mestery: thanks, i have started work on this
19:05:01 <s3wong> Oh, OK - Thanks SumitNaiksatam
19:05:07 <SumitNaiksatam> s3wong: i incorporated your diff
19:05:27 <SumitNaiksatam> s3wong: i am going resource by resource, and making changes/enhancements as required
19:05:33 <SumitNaiksatam> and adding UTs
19:05:34 <mestery> Thanks to s3wong and SumitNaiksatam for getting this pushed out there!
19:05:45 <banix> s3wong, SumitNaiksatam: thanks
19:05:55 <SumitNaiksatam> was hoping to get much further along on this, hopefully soon
19:06:33 <mestery> This is really good progress though, having the API out there will be great!
19:06:36 <mdhami> (this is mandeep, rejoined as mdhami, client problems)
19:07:01 <mestery> So, any questions on API related items?
19:07:04 <banix> I have to push the database code so you guys can review
19:07:09 <mestery> Maybe it's best for people to review the branch.
19:07:10 <SumitNaiksatam> mestery: i am hoping that i can push out a WIP patch to gerrit soon
19:07:13 <mestery> banix: Awesome!
19:07:14 <banix> Again, jumping the gun! :)
19:07:27 <mestery> SumitNaiksatam: That would be super awesome!
19:07:33 <mestery> banix: You get an AI for that. :)
19:07:44 <mestery> #action banix to push DB code to shared github for review
19:07:45 <banix> :)
19:07:52 <mdhami> SumitNaiksatam: cool!
19:08:11 <mestery> OK, any other API related items to discuss now?
19:08:18 <SumitNaiksatam> gerrit will be eventually easier to review for everyone
19:08:52 <s3wong> nice
19:09:09 <marun> uh
19:09:14 <mestery> Agreed s3wong.
19:09:15 <banix> SumitNaiksatam, so that would include individual patches? sorry for asking the same questions many times
19:09:18 <marun> I think I'll have some commentary on the chosen testing strategy
19:09:29 <marun> I know it's consistent with how things are done in neutron, but that's not a good thing.
19:09:36 <SumitNaiksatam> banix: we can decide how we can structure the patches
19:09:42 <mestery> marun: Please, feedback is very much welcome!
19:09:43 <s3wong> marun: please elaborate
19:09:48 <SumitNaiksatam> banix: lets first have the code in the temp repo
19:09:56 <SumitNaiksatam> banix: does that sound okay?
19:10:13 <banix> SumitNaiksatam: yes, sure.
19:10:51 <mestery> marun: Are you providing feedback now or once you've looked stuff in more detail?
19:10:52 <marun> s3wong: these are functional tests that exercise lots of boilerplate unnecessarily.
19:11:16 <marun> s3wong: ideally there should be a separation between testing of the api and the lower-level stuff, keeping each test as focused as possible.
19:12:11 <banix> marun: What do you suggest to do in this respect?
19:12:33 <s3wong> marun: actually that sounds reasonable. I think we do normally write separate API test (from the plugin test) in Neutron
19:12:38 <markmcclain> marun: +1 for new code we need to make sure we get away from the full stack test pattern we have in much of the existing code
19:13:25 <mestery> #info Group Policy unit test code needs to separate out api testing from lower-level stuff
19:14:07 <marun> my larger goal is to ensure that api tests are reusable by tempest
19:14:08 <SumitNaiksatam> mestery: i believe that's the path i have taken in my branch
19:14:31 <mestery> Awesome, good to hear SumitNaiksatam.
19:14:42 <SumitNaiksatam> happy to be correct :-)
19:14:48 <SumitNaiksatam> *corrected
19:15:06 <SumitNaiksatam> but i have only a few tests, more coming
19:15:32 <mestery> OK, cool.
19:15:34 <s3wong> marun: that said, specifically for the PoC, we do have a goal to see the whole network policy work working end to end (from Heat template to ovs driver) - though I don't think we are deviating from separating API test from lower level functional tests
19:16:11 <marun> s3wong: well, I think the intent might be there but there is room for improvement in the implementation.
19:16:41 <marun> s3wong: it's likely I'm going to have to rewrite some of the tests to demonstrate what I mean[.
19:16:55 <mdhami> marun: Can you give an example from this case?
19:17:08 <marun> mdhami: all the tests are full-stack
19:17:12 <s3wong> marun: absolutely. Please do so. We welcome any feedback
19:17:15 <marun> mdhami: and going against the rest api manually
19:17:36 <marun> mdhami: lots of common infrastructure is in play that doesn't need to be.
19:17:43 <mdhami> marun: Look forward to your example. That should help
19:18:16 <marun> anyway moving on :)
19:18:30 <mestery> marun: Thanks for your inputs here. :)
19:18:30 <s3wong> marun: Thanks!
19:18:46 <mestery> #topic Model
19:19:04 <mestery> There has some talk of where the policy code is implemented.
19:19:27 <mestery> I'd at least like to have that discussion with the team now given the alternatives, especially since markmcclain is lurking here and can provide some input as well. :P
19:19:48 <mestery> One option would be as a service plugin. Another option could be as a new core plugin.
19:20:12 <mestery> SumitNaiksatam had some concerns about the service plugin approach which we should discuss here.
19:20:22 <mestery> Because there are some issues with each approach I think.
19:20:24 <mdhami> The issue is that many policies are naturally cross cutting (across layers/functions) and putting the policy as an extension to say ML2 would not be appropriate. Hence the recommendation for a new policy plugin
19:20:24 <s3wong> mestery: so far we are assuming core plugin - what is the benefit of doing it as service plugin?
19:20:47 <mdhami> I agree a core plugin makes more sense
19:21:15 <mestery> So, with a new core plugin, you only get the APIs if you use that plugin, and a new abstraction layer is needed to plug in old plugins.
19:21:19 <mestery> Thoughts from people on that?
19:21:20 <SumitNaiksatam> mestery: thanks for the cue, yeah it does not seem like the service plugin will serve our purpose
19:21:28 <banix> we have been talking about implementing this as part of ML2
19:21:56 <rkukura> has the combination of a service plugin and an ML2 MechanismDriver been considered?
19:22:12 <SumitNaiksatam> rkukura: i don't think that works
19:22:28 <mdhami> I suspect that you do not want policies that handle L3 or services (say needs Firewall inspection) to go thru' ML2
19:22:55 <s3wong> banix: I would expect other monolithic plugins can implement group-policy if they choose to
19:23:05 <rkukura> So is the requirement to intercept the entire API, not just the L2 API?
19:23:13 <banix> s3wong: yes sure; for the PoC.
19:23:13 <SumitNaiksatam> rkukura: thats correct
19:23:35 <SumitNaiksatam> rkukura: policy spans across L2, L3, serives, IPAM
19:23:37 <markmcclain> a service plugin should allow for the policy API to be developed alongside of the core plugin
19:23:43 <mestery> And we're intercepting the entire API to head off using both the policy APIs and the old APIs at the same time?
19:23:44 <rkukura> But lots of those APIs (even L3 in ML2) are service plugins
19:24:04 <mestery> rkukura markmcclain: Agreed
19:24:11 <rkukura> So it seems like there will be a need to intercept the service plugin APIs even with a new core plugin
19:24:14 <SumitNaiksatam> markmcclain: its not just a matter of API endpoints
19:25:16 <SumitNaiksatam> markmcclain: if we want to support current neutron constructs as well, we will have two entities managing them - the core plugin and the service plugin
19:25:26 <SumitNaiksatam> markmcclain: and that will lead to conflict
19:25:48 <mdhami> markmcclain: there can be a conflict between policy intent and direct L2 configuration. The policy plugin should resolve that.
19:26:03 <SumitNaiksatam> this issue manifests even if we want to service chaining across multiple services
19:26:36 <banix> We have been saying the policy will complement the core API
19:26:40 <SumitNaiksatam> we don't have a footprint in neutron which orchestrate across L2, L3, services, IPAM
19:27:05 <banix> and to that extent we define the endpoints as Neutron objects like Networks...
19:27:10 <SumitNaiksatam> banix: yes for that to happen we needs a plugin which serves as the endpoint for both, core abstractions and the policy abstractions
19:27:19 <SumitNaiksatam> *need
19:28:06 <mdhami> banix, SumitNaiksatam: I agree.
19:28:27 <rkukura> SumitNaiksatam: So would this new core plugin not use the existing service plugins, or somehow intermediate?
19:28:39 <SumitNaiksatam> rkukura: yes
19:28:52 <mdhami> rkukura: intermediate
19:28:53 <SumitNaiksatam> rkukura: we can leverage all the existing plugins as libraries
19:28:58 <rkukura> So if that can be done in a new core plugin, why not in ml2?
19:29:00 <SumitNaiksatam> rkukura: we don't rewrite any code
19:29:24 <kanzhe> I guess the existing L2, L3 and service plugins becomes libraries to the new core plugin.
19:29:24 <SumitNaiksatam> rkukura: doesn't ml2 only do L2?
19:29:34 <rkukura> I'm a bit concerned if this starts looking like the metaplugin
19:29:39 <marun> hmmm
19:29:39 <SumitNaiksatam> kanzhe: yes, that would be the natural way to do it
19:29:57 <mdhami> rkukura: Are proposing implementaing all policy API in ML2?
19:30:08 <markmcclain> yeah I'm not a big fan of a meta plugin design
19:30:13 <SumitNaiksatam> rkukura: this will be different from the metaplugin
19:30:17 <marun> did i mishear? that supporting group policy in a given deployment means not supporting the normal neutron api?
19:30:21 <SumitNaiksatam> its not a metaplugin
19:30:35 <rkukura> mdhami: I'd hope it could be a service plugin that could be intergrated with ml2 or other monolithic core plugins
19:30:50 <mdhami> marun: No that is not correct.
19:31:02 <banix> SumitNaiksatam: Why not be part of ML2 and reuse other service plugins
19:31:11 <kanzhe> No, the existing Neutron APIs will still be supported, just added new group poilcy APIs.
19:31:13 <SumitNaiksatam> banix: how?
19:31:14 <mdhami> marun: The current APIs need to be validated to be consistent with the policies applied (if applied)
19:31:53 <banix> mdahmi: the other way around
19:31:58 <SumitNaiksatam> banix: there is no notion of "reuse" service plugins
19:32:00 <s3wong> banix: wouldn't that preclude all the other monolithic plugins (there are still a lot of those) to implement group-policy?
19:32:03 <mdhami> marun: Without policies, there is no impact
19:32:14 <marun> mdhami: so existing plugins will need to be instrumented for validation
19:32:40 <mdhami> marun: instrumented as?
19:32:51 <mestery> One concern I have here is that to use the policy APIs you have to use an entirally new core plugin, and side-load normal plugins (core and service) under this. Is that right?
19:33:23 <marun> mdhami: a call to the core api occurs, the group policy plugin needs to know what is being attempted to validate
19:33:27 <SumitNaiksatam> mestery: you would not "side-load" other plugins
19:33:27 <mdhami> marun: In ideal design current plugin have no impact at all
19:33:29 <banix> s3wong: well, they are on the way out and this exercise is for a PoC. Isn't it?
19:34:10 <marun> mdhami: not a bad assumption initially, but I don't think it holds for realworld
19:34:15 <SumitNaiksatam> mestery: the policy plugin would leverage those other plugins as libraries
19:34:17 <s3wong> banix: Oh? are we talking about just PoC / reference implementation here?
19:34:26 <SumitNaiksatam> mestery: it would policy plugin configuration
19:35:04 <SumitNaiksatam> s3wong banix: i think we are talking about the path we want to take going forward
19:35:14 <SumitNaiksatam> PoC might be an early implementation of that
19:35:35 <markmcclain> SumitNaiksatam: leveraging other plugins as libraries… how is that different from the meta plugin?
19:35:39 <mdhami> Yes, but it would be quite a basic design assumption
19:35:40 <kanzhe> I guess one way to look at it is that policy API are extensions. The existing monolithic plugins needs to implement the policy API if they decide to support policies. A new plugin will be developed as a core plugin to implement L2, L3, service, and the new policy APIs.
19:35:41 <s3wong> SumitNaiksatam: yes - that's what I thought. This will have impact on the entire group-policy moving forward
19:36:13 <SumitNaiksatam> markmcclain: yes, i wanted to ask what people's understanding is of a meta plugin
19:36:45 <SumitNaiksatam> markmcclain: so we are not referring to the meta plugin which is currently in the neutron plugins, right?
19:37:05 <mestery> That's what I thought the meta plugin was SumitNaiksatam. :()
19:37:07 <SumitNaiksatam> kanzhe: yes, that's the proposed approach
19:37:08 <mestery> :)
19:37:43 <SumitNaiksatam> mestery: that metaplugin chooses between different plugins to implement a particular neutron resource
19:37:56 <markmcclain> yeah when I say meta plugin… I'm talking more in concept
19:38:01 <markmcclain> than the one we currently have now
19:38:18 <SumitNaiksatam> markmcclain: correct, i guessed as much
19:38:44 <mdhami> markmcclain: I understand your concern for meta plugin. Since we will use existing published core/service APIs, this is more an intermediation than a loader of distinct plugin.
19:38:45 <banix> So are we saying that a given plugin (already out there) cannot simply add support for the pGroup Policy extension without modifying what it does for the core API?
19:39:02 <rkukura> I'm still not seeing how a new core plugin intercepts the APIs of service plugins?
19:39:39 <mdhami> banix: You can do that. Or you can use this reference implementation - like any other plugin usage
19:40:22 <SumitNaiksatam> markmcclain: what you are probably referring to metaplugin I am seeing that as code reuse
19:40:25 <marun> rkukura: we're handwaving that ;)
19:40:29 <mdhami> rkukura: It registers as the core plugin
19:40:41 <mdhami> marun: ;-)
19:40:48 <rkukura> But the core plugin only implements the core (L2) API
19:41:00 <SumitNaiksatam> rkukura: if the policy plugin supports the relevant extensions then it will get those API calls, right?
19:41:37 <SumitNaiksatam> rkukura: for implementation it will rely existing plugin implementation as lib modules
19:41:49 <rkukura> So this is a "combo plugin" that implements the L2 and all service APIs, and may do that by calling out into other core or service plugins?
19:42:31 <mdhami> rkukura: (and of course also for the policy API that we define)
19:43:56 <mdhami> rkukura: The question is how do you not end up with an inconsistent system when both the policy and the current API can bot address that same entity without any co-ordination?
19:44:40 <SumitNaiksatam> mdhami: yeah, the issue is that currently there is no place in Neutron where you can do that coordination
19:44:43 <rkukura> Seems to me some simple mechanism in the neutron core that allows code to intercept API calls before they get to the core or service plugin may be a cleaner approach.
19:44:49 <mdhami> This design just intermediates to ensure a correct system based on both policy expectations and API calls,
19:44:50 <markmcclain> ok.. I'm going to run the risk of derailing this discussion for a bit :)
19:45:42 <markmcclain> so it seems a lot of the patterns we are discussing is because the current v2 API and plugin def were setup synchronous (even though most aren't)
19:45:44 <mdhami> rkukura: I agree. The hope was to prove this first as a plugin in
19:45:52 <mdhami> rkukura: and once that has been stabalized, we can promote that code to core (as appropriate)
19:46:08 <markmcclain> if we were to move to a flow based async plugin layer
19:46:12 <s3wong> ruukura: that probably will avoid the conflict
19:46:15 <rkukura> mdhami: That might be a reasonable way forward.
19:46:21 <markmcclain> with a defined pipeline would it make things easier to implement?
19:47:07 <mdhami> markmcclain: I suspect so
19:47:22 <SumitNaiksatam> markmcclain: can you elaborate a little more, sorry for being dense
19:47:44 <mdhami> markmcclain: We could use this deisgn as a proving ground for that design goal, and if that works out we can promote to to Neutron API layer
19:47:48 <markmcclain> one of the items that several in the community have been kicking around
19:48:04 <SumitNaiksatam> markmcclain: are you referring to a orchestration layer within the API framework?
19:48:15 <mdhami> markmcclain: This allows us a non-invasive aproach to validate that deisgn goal
19:49:38 <markmcclain> so our API current presents everything as synchronous which is why you're talking about wrapping calls and writing a mono plugin
19:50:05 <SumitNaiksatam> markmcclain: i am not sure that is so much of an issue here
19:51:19 <SumitNaiksatam> markmcclain: but please finish if you have more to add
19:51:28 <SumitNaiksatam> markmcclain: perhaps i am not understanding
19:51:33 <markmcclain> how so? you're have to hook into the L2, L3 and other extension apis
19:53:34 <markmcclain> if the pipeline for those resources had reasonable point of integration I think the implementation would be cleaner
19:53:57 <mdhami> markmcclain: I agree.
19:54:22 <SumitNaiksatam> markmcclain: so in this case it would mean that the pipeline would need to be constructed in such a way that all the calls go to the policy plugin first
19:54:24 <markmcclain> from a big picture perspective, I'm bit concerned on the amount of duplicative code we generate because the minor variations in workflow each plugin/extension/driver needs
19:54:53 <mdhami> markmcclain: Should we use this plugin as a first step to work out that design in context of a plugin? That was my question.
19:55:02 <rkukura> markmcclain: That's kind of what I was trying to get at. But I do see the approach SumitNaiksatam is describing as maybe being appropriate for PoC at least.
19:55:09 <marun> SumitNaiksatam, markmcclain: when we start talking about pipeline I start thinking about wsgi applications as used by swift
19:55:35 <marun> it's actually pretty clever - just like pipes, really, but with request/response instead of stdin/stdout
19:55:38 <markmcclain> marun: no much different than wsgi
19:55:58 <marun> markmcclain: I know that you're proposing something different
19:56:08 <marun> markmcclain: I'm wondering if a simpler way is possible, though.
19:56:22 <marun> markmcclain: one without all the requirement on static architecture
19:56:33 <marun> markmcclain: anyway, I'll shutup for now
19:56:55 * mestery thinks this has actually been a pretty good discussion.
19:57:19 <markmcclain> marun: I'm not necessarily proposing anything concrete just trying to distill a few of the ideas that have been floating around in the community
19:57:26 <markmcclain> into something might help everyone
19:57:59 <SumitNaiksatam> markmcclain: so based on mdhami and rkukura's suggestion - how about we use this policy plugin approach as a first step towards that?
19:58:22 <SumitNaiksatam> mestery: thanks for the time btw, for this :-)
19:58:26 <mdhami> markmcclain: I agree that we need that API. Since it can quite disruptive, the proposal here was to try it out in the context of a plugin
19:58:37 <mestery> This has been a good discussion SumitNaiksatam.
19:58:41 <kevinbenton> markmcclain: in this async flow, do you envision plugins registering callback functions for events?
19:58:49 <mdhami> mestery: I agree]
19:59:05 <markmcclain> kevinbenton: not sure on the mechanisms
19:59:21 <markmcclain> so for now figuring out the rest API is something we can do
19:59:29 <rkukura> SumitNaiksatam: Maybe I'm not fully clued in, but the concern I have with the policy plugin approach is that it has to implement all the APIs, so it kind of prevents further extensibility
19:59:35 <markmcclain> even if the implementation under the hood is more PoC
19:59:56 <mestery> That seems like a good starting point markmcclain.
20:00:13 <mestery> I think we may have to continue this discussion over email and in next week's meeting as well.
20:00:18 <mestery> Since we're out of time in 1 minute :)
20:00:20 <markmcclain> my original thought in suggesting a service plugin is that vehicle allows this work and any changes to the core plugin to proceed in parallel
20:00:45 <SumitNaiksatam> markmcclain: doing the policy plugin will exactly achieve that goal
20:00:59 <marun> I know SumitNaiksatam expressed concern about the service plugin approach, but I'm not sure of the specifics.  Are they documented anyway?
20:01:02 <marun> anywhere?
20:01:31 <mestery> marun: We need to start a document perhaps to share things, may be easier to comment on that.
20:01:34 <mestery> OK folks, time is up for today.
20:01:36 <SumitNaiksatam> marun: how about follow up in email, since we are out of time
20:01:43 <mestery> Lets continue this discussion on email and in next weeks meeting.
20:01:48 <mestery> Thanks all!
20:01:50 <mestery> #endmeeting