17:03:53 <thinrichs> #startmeeting CongressTeamMeeting
17:03:54 <openstack> Meeting started Tue Mar 25 17:03:53 2014 UTC and is due to finish in 60 minutes.  The chair is thinrichs. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:03:55 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:03:57 <openstack> The meeting name has been set to 'congressteammeeting'
17:04:09 <thinrichs> david-lyle: no worries
17:04:31 <thinrichs> rajdeep: do you want to give us a status update on neutron/etc. drivers?
17:04:41 <rajdeep> yes
17:05:08 <rajdeep> i just sent a couple a change requests
17:05:21 <rajdeep> finishing the integration of Neutron with congress
17:05:56 <thinrichs> rajdeep: yeah--it's looking good.  Didn't get a chance to look at your changes yet this morning.
17:06:02 <rajdeep> you can see the CR here https://review.openstack.org/#/c/82293/3
17:06:48 <rajdeep> also added another variable to store timestamp of syncup
17:07:26 <thinrichs> Sounds good.
17:07:33 <rajdeep> this finished networks and ports conversion from json to tuple
17:07:51 <thinrichs> I'd ask about the next component you'll be working on, but there's another action item that will impact that.
17:08:21 <rajdeep> i was thinking of doing nova driver
17:08:31 <thinrichs> rajdeep: any lessons learned that you want to share about converting JSON to tables/tuples?
17:08:57 <rajdeep> yes - it gets complicated so spent some time creating detailed mappings
17:08:58 <rajdeep> https://docs.google.com/document/d/1K9RkQuBSPN7Z2TmKfok7mw3E24otEGo8Pnsemxd5544/edit
17:09:06 <rajdeep> refer to the design doc
17:09:30 <rajdeep> also one json maps to n tuples
17:09:44 <thinrichs> It would be nice to have some sort of writeup so that new people have an easier time understanding what needs to be done when writing a new driver.
17:10:00 <rajdeep> thats a good idea
17:10:25 <thinrichs> rajdeep: want to tackle that?  You're the expert
17:10:29 <rajdeep> will do that
17:10:33 <rajdeep> - yes
17:10:45 <rajdeep> what should be the format - rst?
17:10:59 <rajdeep> and where should i place it
17:11:21 <thinrichs> There's a section in the current RST docs on "Data Integration".  Let's put it there.
17:11:31 <rajdeep> ok
17:11:50 <thinrichs> #action rajdeep will work on the Data Integration portion of the docs, focusing on adding a new driver
17:12:06 <thinrichs> Let's move on then.
17:12:37 <thinrichs> kudva: did you get enough feedback from us to work on the policy engine/Datalog builtins?
17:12:51 <kudva> Would like to have a discussion on it.
17:13:25 <kudva> I have some ideas to implement, but want to make sure that the team agrees
17:13:32 <thinrichs> Want to briefly layout the problem you're working on and what you're planning?
17:14:05 <kudva> Okay, so the idea is the we have a congress_builtin that has the following components
17:14:22 <kudva> 1. It of course has the python builtins
17:15:19 <kudva> 2. It provides (manipulation of datalog primitives), particularly string manipulation (as you mentioned) to extract elements
17:15:40 <kudva> 3. It provides mappign from datalog function to data source tables (is this part of the builtin?)
17:15:55 <kudva> Once we agree, I can go ahead and implement
17:16:19 <thinrichs> By (1) you mean basic functions like addition, subtraction, string manipulation, right?
17:16:36 <thinrichs> Or are you also thinking of set(), list(), dict(), and so on?
17:16:42 <kudva> By 1, I mean everything that is part of python builtin
17:16:56 <kudva> 2. would be specific to datalog and congress
17:17:19 <kudva> 1 = http://docs.python.org/2.7/library/functions.html
17:17:52 <thinrichs> Thanks for the link.
17:18:19 <kudva> 2. = datalog specific addition, subtraction, string etc.
17:19:01 <kudva> Python already provides many of the functions for add, sub etc, so what I am adding is datalog primitive support correct?
17:19:12 <thinrichs> My main concern about including all of those Python builtins is that they can have side effects.
17:19:31 <kudva> Can you give an example?
17:19:51 <thinrichs> From the policy language's point of view, there shouldn't be side effects b/c then Datalog will need to be reasoning about those side effects to get the right answer.
17:20:34 <thinrichs> You create a dict() and then later set a value--you *modify* that dict, you don't create a new one.
17:21:00 <thinrichs> Datalog sort of assumes that every time you change a dict you would create a new one.  Sort of like Haskell.
17:21:09 <kudva> I see
17:21:18 <thinrichs> Also, eval() is super worrisome because it could do anything.
17:22:12 <thinrichs> Typically a policy language will allow any builtin without side effects.  Though there are downsides to having even those builtins.
17:22:30 <thinrichs> The more builtins we have, the harder it is to do *analysis* of the policy.
17:23:10 <thinrichs> Say we wanted to check if there were any cloud service tables (input tables to the policy) that would make some table empty.  Without any builtins, there are reasonably straightforward algorithms to use.
17:23:28 <kudva> Got it. Unless all the builtins can be reduced to some fixed set of basic primitives...
17:23:32 <thinrichs> But with builtins, we'd need to modify those algorithms so they know (for example) how to add 2 numbers, etc.
17:23:39 <thinrichs> kudva: Right.
17:24:09 <thinrichs> That was why I would start with a small set of builtins (arithmetic, string manipulation) and build up from there.
17:24:24 <thinrichs> Once we get the mechanism for adding builtins nailed down, it should be easy to add new ones.
17:24:53 <kudva> Okay, I can start that. So to be clear. The builtins will be used in the policy language. They will be reduced to a fixed set up primitives for analysis.
17:25:23 <kudva> That is not to complicate the analysis
17:26:00 <kudva> So what we need is a fixed set of primitives that keeps the analysis simple. All builtins or macros sitting on top should be reducible to those primitives
17:26:20 <kudva> Let me start a draft to make it more concrete
17:26:35 <kudva> I will take a few arithmetic examples and send out some examples for discussoin
17:26:38 <kudva> discussion
17:26:40 <thinrichs> Great!  Here's the question I was about to ask you...
17:26:41 <thinrichs> kudva: sounds good.  Want to take a stab at a (brief) proposal to nail down (i) syntax within the policy language (see my email), (ii) how one would add additional builtins, and (iii) a word or two about what code would need modifications?
17:26:57 <kudva> okay, will do that
17:27:15 <thinrichs> #action kudva will put a proposal together for adding builtins to the Datalog policy language.
17:27:41 <thinrichs> Moving on...
17:28:04 <thinrichs> pballand: you had 2 action items.
17:28:34 <thinrichs> (1) Hooking up the API to the policy engine and (2) looking into the data integration glue (for helping the policy engine communicate with the component drivers)
17:28:44 <thinrichs> Any progress?
17:29:22 <pballand> some, on (1), I've begun refactoring the existing API code to make it more tightly coupled with policy
17:29:33 <pballand> I don't have code to post yet, but hope to by the end of the week.
17:29:56 <thinrichs> Great!  It'll help others, especially, to see the end-to-end flow of information from API to policy engine.
17:30:29 <pballand> on (2), I haven't gotten very far, but I believe there is some other work underway
17:31:22 <pballand> it seems like an AMQP framework would work week for the data integration, and we are looking to leverage some existing code
17:32:02 <pballand> I'll post to the ML when we have more on this
17:33:09 <pballand> that's all I have to report today :(
17:34:11 <thinrichs> Sounds good.  Thanks for the update
17:34:19 <pballand> actually, just saw in email that it is okay to announce that Plexxi is going to contribute some code for the data integration
17:34:37 <pballand> we are super excited to have them on board
17:34:48 <thinrichs> Awesome!
17:35:08 <rajdeep> this is great news!
17:35:32 <pballand> yes!
17:35:56 <thinrichs> The cool thing about the Plexxi code is that they have a Python impl of AMQP for doing data integration.
17:36:08 <pballand> we will be working to merge in that code, and will report more on the ML and next meeting
17:37:02 <thinrichs> So this will enable the policy engine to ask for data from our cloud service drivers and eventually tell those cloud service drivers to execute actions.
17:37:43 <rajdeep> i am assuming this will sit between the policy engine and the datasource drivers
17:38:09 <thinrichs> That's the idea, at least conceptually.  It'll take some work to figure out how the integration will actually work.
17:38:26 <thinrichs> Any other questions about Plexxi before we move on?
17:38:41 <pballand> rajdeep: yes - we will probably support other mechanisms for POC work, but I expect this would be the preferred integration pattern
17:39:13 <rajdeep> great
17:39:37 <thinrichs> The last item on the agenda before opening it up is an update on my action items.
17:39:39 <rajdeep> this will make it scalable so that drivers could run on a separate node as compared the engine
17:40:01 <thinrichs> I was supposed to investigate the idea that we would replace the Classificatin+Action policies with a Classification, Access Control, and Condition action policy.
17:40:20 <thinrichs> I sent a note to the ML, and no one seemed to have a strong opinion one way or the other.
17:40:39 <thinrichs> After waiting a while, I went ahead and edited the design doc to reflect the new ideas.
17:40:59 <thinrichs> It's much simpler to understand and I think it should help us focus better on making this first alpha release a good one.
17:41:28 <thinrichs> I also pitched Congress to 3 different groups, and our newer simplified version went over well.  People seemed to understand it pretty much immediately.
17:41:44 <thinrichs> Does anyone have questions/comments about this?
17:42:38 <rajdeep> it is easier to understand it this way
17:42:40 <rajdeep> :)
17:43:09 <thinrichs> Then I'll take an action item to make the changes in the code.  Once that's done I'll update the docs as well.
17:43:43 <thinrichs> #action thinrichs will remove the Action description policy and add the Access Control policy from the prototype.
17:44:18 <thinrichs> The last thing we have is Open Discussion.
17:45:09 <rajdeep> had a question on how will congress work with neutron policy related work
17:45:56 <thinrichs> It's not clear how they will interoperate right now.
17:46:14 <thinrichs> Conceptually, we'd like it to be the case that they can exchange policy fragments.
17:47:13 <thinrichs> If, for example, Congress could push policy down to Neutron, then that policy could be used to prevent violations before they happen more efficiently than having Neutron consult with Congress directly.
17:47:23 <rajdeep> they also have classifier, rules and actions
17:47:41 <thinrichs> Syntactically they're similar languages (as most policy languages are).
17:48:11 <thinrichs> But we'd need to make sure that we are enforcing the same policy.
17:48:35 <thinrichs> One challenge is that our policies may require information that Neutron doesn't itself have.
17:48:43 <thinrichs> (That's why Data Integration is so important).
17:49:08 <thinrichs> For example, our policy might refer to Nova info, Neutron info, and AD info.  We can't push that policy down to Neutron unless it has access to that same info.
17:49:27 <rajdeep> that was my thought too
17:49:40 <thinrichs> Similarly, Neutron policies can refer to L4-L7 data in a network packet--info we may or may not have.
17:49:45 <rajdeep> neutron needs more info to have a comprehensive policy
17:50:03 <thinrichs> So just from the perspective of information, it's non-trivial to have them exchange policies.
17:51:49 <thinrichs> Anything else?
17:52:02 <rajdeep> no
17:52:40 <thinrichs> Great discussion, all.  Let's follow up on the ML, as usual.
17:52:41 <thinrichs> Thanks!
17:52:59 <rajdeep> thanks
17:53:25 <pballand> #endmeeting
17:53:57 <thinrichs> #endmeeting