17:01:57 <thinrichs> #startmeeting CongressTeamMeeting
17:01:58 <openstack> Meeting started Tue Jul 29 17:01:57 2014 UTC and is due to finish in 60 minutes.  The chair is thinrichs. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:01:59 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:02:01 <openstack> The meeting name has been set to 'congressteammeeting'
17:02:02 <thinrichs> Hi all
17:02:05 <sarob> morning
17:02:07 <cloudtoa_> Hello.
17:03:07 <skn_> Hi guys
17:03:20 <kudva> Hi
17:03:57 <pballand> Hi
17:04:11 <thinrichs> Looks like we have enough to get started.
17:04:41 <thinrichs> We definitely want to hear from the Tetris folks, since they were cutoff last time.
17:04:52 <thinrichs> But I don't see gokul online yet.
17:05:08 <thinrichs> So let's start with status updates.
17:05:16 <thinrichs> kudva: want to start?
17:05:25 <kudva> Sure, couple of things
17:06:20 <kudva> Completed the integration by adding more tests for builtin-runtime integration
17:06:32 <kudva> Addressed comments from arosen and thinrichs
17:06:35 <kudva> https://review.openstack.org/109099
17:06:45 <arosen> Hi
17:06:54 <arosen> kudva:  sounds good I"ll take another look.
17:07:16 <kudva> Also, sent a blueprint for Congress Ceilometer integration just this morning. Very, very early first draft
17:07:17 <arosen> kudva:  seems like it's still failing jenkins right now. Do you know why?
17:08:03 <kudva> So, the pep8 errors I can fix, they are minor (white spaces, import alphabetical order)
17:08:15 <kudva> But the congress unit tests don't even run
17:08:31 <kudva> so, not sure why they don't run. The call to runtests itself is failing it seems
17:08:48 <kudva> https://docs.google.com/document/d/1NV8NbQTNyqWZnyhtfoKt6LQElcG6HAVPuyUHHRYE5nM/edit?usp=drive_web
17:08:57 <kudva> ceilometer/congress integration start document
17:09:36 <thinrichs> Is that world-readable?  It says I need permission.
17:09:59 <arosen> kudva:  actually i think you might need to just rebase it and it should work again. I think it's failing because of broken requirements.txt that we had at one point.
17:10:21 <kudva> thinrichs: okay, will do on tests
17:10:35 <kudva> thinrichs: will set the permissions,
17:11:44 <kudva> thinrichs: now anyone can read
17:11:51 <thinrichs> kudva: I did a quick scan over the ceilometer doc.
17:12:32 <thinrichs> kudva: I think the key thing we need for an integration with Ceilometer is a description of what their data model is and how it will look in Congress.
17:13:00 <kudva> thinrichs: they store their metrics and samples in mongodb, and have apis
17:13:03 <thinrichs> kudva: the rest of the architecture you're describing is already in place (though we don't sometimes cache data and sometimes not)
17:13:06 <sarob> #info ceilometer integration proposal #link https://docs.google.com/document/d/1NV8NbQTNyqWZnyhtfoKt6LQElcG6HAVPuyUHHRYE5nM/edit
17:13:44 <thinrichs> gokul: glad you could make it!  We're in the midst of getting a status update from kudva.
17:14:14 <gokul> hello all.   thinrichs:  thanks.  there as a network outage last week here.
17:14:20 <kudva> thinrichs: so we could access the ceilometer apis directly, but the support in Congress we need is to determine 'when' we go to local store versus Ceilometer based on variable in predicates
17:15:13 <thinrichs> kudva: if the policy mentions ceilometer:cpu_util, the 'ceilometer' prefix tells Congress that the table comes from the ceilometer datasource driver.
17:15:13 <kudva> thinrichs: so I will add more details and outline a full example, just wanted to run initial thoughts by the team
17:15:23 <thinrichs> kudva: we do that with Nova/Neutron already today.
17:15:47 <kudva> thinrichs: Okay, will look at the dse code a bit more
17:16:26 <kudva> thinrichs: regarding my earlier commit, rebasing is good enough to get the requirements.txt, and run jenkins tests?
17:16:42 <cloudtoad> kudva:  any questions re: DSE, I will gladly assist.
17:16:50 <thinrichs> kudva: check out dsepolicy.py in particular.  It's the glue the connects the policy engine to DSE.
17:17:03 <thinrichs> Maybe this is a good time to give my status update then.
17:17:11 <kudva> cloudtoad: okay, thanks! Would like to see the specific python code which chooses between data stores
17:17:31 <thinrichs> We almost have a fully integrated system.
17:18:09 <thinrichs> I'm planning on putting a demo script together around the private/public networking use case, mainly b/c it's small and we have the necessary data sources.
17:18:25 <sarob> thinrichs: makes sense
17:18:26 <thinrichs> If all goes well I'll send out the script in the next few days.
17:18:49 <sarob> thinrichs: this is what we will demo at ops summit?
17:18:55 <thinrichs> Then everyone will be able to write policies against Nova/Neutron and start tracing how the code works.
17:19:24 <thinrichs> sarob: we should be able to demo any policy we want over Nova/Neutron sources.
17:20:11 <sarob> thinrichs: sounds good.
17:20:22 <cloudtoad> Do we have a table class yet, Tim?
17:20:28 <gokul> so, does a fully integrated system mean:  a server running that can take some action based on some conditions [right now, nova/neutron] and policies [datalog]?
17:21:12 <thinrichs> cloudtoad: DSE itself doesn't have tables, but all the instances of DSE that we're using happen to send collections of tuples around (i.e. tables and table-deltas).
17:21:47 <cloudtoad> Right, but is that collection defined in a class somewhere?
17:22:01 <thinrichs> gokul: just monitoring—not actions.  So that means that people can write a policy that describes the desired state of the datacenter (over Nova/Neutron datasources), and we can check if the current state of the system matches that desired state or not.
17:22:31 <thinrichs> cloudtoad: not within DSE but there's a Database class within the policy runtime.
17:22:45 <gokul> i see.  ok.   thanks.
17:23:19 <skn_> thinrichs: Are we planning to demo Congress API support too?
17:23:31 <thinrichs> cloudtoad: when we talked about adding a table class to DSE, I tried it out and find a simpler change that gave us what we needed.
17:23:47 <thinrichs> skn_: the API is working, so yes the demo script will have Curl commands.
17:24:04 <cloudtoad> Ok, I'll look at the database class...
17:24:11 <skn_> Cool. Thanks.
17:24:31 <thinrichs> arosen is working on keystone integration, which will make Horizon integration straightforward, as I understand it.  That'll be helpful for the demo as well.
17:24:35 <thinrichs> arosen: want to report?
17:24:40 <arosen> thinrichs:  sure.
17:25:10 <arosen> I'm in the mist of rebasing my keystone and policy.json integration patches. Hopefully once i get those up we'll be able to merge those later today.
17:25:34 <arosen> I'm also working on devstack integration which should help us build some kind of CI system so we can automate the setup of congress.
17:25:47 <arosen> That's all I got for now to report.
17:26:32 <thinrichs> arosen: thanks!
17:27:25 <thinrichs> I think that's most of the coding progress I know about.
17:27:51 <thinrichs> Before we miss out on it again, I think we should chat with gokul about Tetris.
17:28:14 <thinrichs> gokul: want to say a little about the Tetris project and why you're interested in Congress?
17:28:54 <gokul> <thinrichs> and all:   thanks.    so, we had started this initiative called Tetris, where the goal was on policy automation encompassing compliance and optimization policies.
17:29:40 <gokul> so, in summary, after looking at congress and its framework etc. we decided to merge with congress and see if we can have other policies such as
17:30:08 <gokul> runtime optimization etc.
17:30:41 <gokul> however, for congress itself, i believe it is a nice initiative and i look forward to contribute.
17:30:57 <gokul> Jay and others (from China) were all part of the Tetris team
17:31:06 <gokul> all have now joined and will be contributing to Congress.  :)
17:31:17 <skn_> I believe we have had some example runtime policies in Congress too
17:31:25 <sarob> gokul: super awesome
17:31:28 <gokul> thats the brief summary.  <thinrichs>:  want me to add anything else?
17:31:28 <thinrichs> gokul: It's great to have you on board!
17:31:34 <gokul> thanks
17:31:43 <thinrichs> gokul: that's a good summary.
17:32:09 <thinrichs> I think the cool thing is that one of the next big issues to address is how we do enforcement of policies.
17:32:28 <thinrichs> And from what I understand that's what the Tetris team is primarily interested in.
17:32:55 <skn_> That falls pretty much in line
17:32:56 <thinrichs> So it'll be great having you all help direct that aspect of the project.
17:33:16 <skn_> Let me give a brief update on the IDS use case for Congress
17:33:27 <sarob> #info gokul and the tetris team has joined the congress project
17:33:28 <gokul> thats correct.  conditions --> actions based on policies.   so, for this week, i'll be looking at the code and just exercise congress itself.   i'll have to find a place to focus on.
17:33:41 <gokul> as I move forward.
17:33:58 <thinrichs> gokul: as soon as I get this demo script sent around to everyone, that'll be a good way to start.
17:34:12 <gokul> thinrichs:  awesome!  -- look forward to it.
17:34:26 <thinrichs> gokul: we have the #congress IRC channel for non-meeting time chats.
17:34:37 <thinrichs> gokul: there are usually several people on it, in case you have questions.
17:34:53 <thinrichs> skn_: how's the IDS use case going?
17:34:54 <gokul> thinrichs: great.  will be there.
17:35:13 <skn_> Yes, I was waiting until you guys are done with Tetris
17:35:48 <skn_> I had a chat with banix last week
17:35:59 <skn_> on the Advanced services in Neutron proposals
17:36:27 <skn_> Although there were a number of blueprints, code wise I did not see a lot
17:37:13 <skn_> So, I have some support within Neutron for tapping and sending the traffic to IDS traffic
17:37:43 <skn_> I have started with Bro as the open source IDS
17:38:03 <skn_> I am current working on a plan for the IDS agent for Congress
17:38:35 <thinrichs> skn_: The hope is that all you'll need to do to integrate the IDS is write a datasource driver, like the ones we have for Nova/Neutron.
17:38:38 <skn_> That would interact with Bro (bro scripts, etc) and then we can have an action with Neutron/Nova
17:39:03 <thinrichs> skn_: we shouldn't need to have an IDS-specific agent running on Congress.
17:39:30 <skn_> Data source driver for IDS, you mean?
17:39:31 <thinrichs> skn_: have you looked at how we integrated Neutron/Nova?
17:39:46 <thinrichs> skn_: yes—a datasource driver for Bro in particular.
17:40:12 <skn_> Got it, I think I used the term IDS "agent", by mistake
17:40:20 <skn_> I meant a data source driver
17:40:34 <thinrichs> skn_: good—just wanted to make sure.
17:40:46 <skn_> Yeah, thanks for the clarification
17:41:06 <gokul> just to clarify:  IDS = intrusion detection system -- correct?
17:41:19 <skn_> So, once I have bro up and running, I'll start on the datasource driver
17:41:29 <skn_> gokul: correct
17:41:33 <thinrichs> skn_: great!
17:41:40 <gokul> skn_: thanks.
17:42:00 <skn_> The eventual goal is to integrate with Neutron and Nova action
17:42:13 <thinrichs> One thing cloudtoad mentioned in a review is that we should figure out what to do with services that are either unavailable or that crash, etc.
17:42:43 <thinrichs> So if on some install Bro isn't available, we should continue functioning to the extent we can.
17:42:54 <sarob> #info skn_: started with Bro as the open source IDS and working on a plan for the IDS data source driver for Congress
17:43:15 <thinrichs> There would need to be some coordination with the policy engine, so it knows what services are available and which aren't.
17:43:23 <skn_> Got it, thats something to keep in mind
17:43:41 <cloudtoad> @thinrichs From the DSE perspective, I'd suggest, ultimately, that we simply capture those exceptions and log them.
17:44:21 <pballand> at the api, we can expose the plugin status using the /v1/data-sources/<id>/status resource
17:44:26 <cloudtoad> If there is a policy that requires a data source that is not available...  would it "do nothing" or is there a tighter coupling there?
17:44:26 <thinrichs> cloudtoad: agreed from the DSE perspective, but the policy engine needs to know that if IDS is unavailable that we shouldn't treat the IDS tables as empty; rather we should treat them as unknown.
17:44:34 <skn_> are we logging stuffs for failures in data sources?
17:45:24 <cloudtoad> skn_ No.
17:45:32 <thinrichs> skn_: right now we're throwing exceptions, partly so I didn't go crazy debugging, but also so that we remember to address this issue
17:46:00 <skn_> thinrichs: one idea is to write robes policies to capture these scenarios, e.g. ids:available(), ids:xyz()
17:46:55 <thinrichs> skn_: My guess is that having the policy writer add all those xxx:availabe() checks will be hard.
17:47:26 <thinrichs> I think instead the policy engine can figuree that out itself: if a datasource isn't available, we shouldn't be evaluating any error conditions based on that service.
17:47:44 <thinrichs> skn_: but I like the idea of perhaps automatically adding and populating xxx:available() checks as an implementation approach.
17:47:59 <skn_> thinrichs: yes, that's what I was thinking, just automate these checks
17:48:03 <pballand> thinrichs: if we use skn_’s suggestion, the policy writer could leverage existing alerting/remediation functionality to address datasource issues
17:48:34 <pballand> +1 to adding the checks automatically
17:49:04 <thinrichs> pballand: I like the idea of giving people the ability to check a datasource's status within policy.  Then they can choose to do it or not.
17:49:50 <cloudtoad> Well, d6cage can iterate over the list of known eventlets, checking their status... then publish this information to a well known DSE bus address.
17:49:57 <pballand> any volunteers to write this up as a spec?
17:50:22 <skn_> Anyways, that's all I wanted to update on IDS
17:50:32 <thinrichs> I'd be happy to help, but my plate is pretty full right now.  If someone else takes the lead, I'll pitch in.
17:50:38 <pballand> I guess I’ll volunteer
17:50:58 <pballand> #action pballand will spec exposing datasource status in policy language
17:51:25 <thinrichs> skn_: sorry the conversation meandered away from IDS.  Let me know if there's anything I can help with getting the IDS spec in place.
17:52:01 <skn_> Let me have a stab on the IDS spec, and then I'll update you guys
17:52:14 <thinrichs> skn_: sounds good.  Thanks!
17:52:26 <skn_> We should link it with the IDS use case bp, or something
17:52:49 <sarob> btw, i will continue working on getting the congress-spec repo working
17:52:57 <skn_> the "compromised VM" bp has some references to IDS, I think
17:53:11 <sarob> I will update the ML and channel when its ready
17:54:02 <thinrichs> sarob: thanks for the update — I was just getting ready to ask about the spec repo. :)
17:54:48 <sarob> thinrichs: finally got the acls file merged, but its not working so another issue to work through
17:55:09 <thinrichs> sarob: let us know if there's anything we can do to help.
17:55:14 <thinrichs> 5 minutes remaining.  Let's open it up for discussion.
17:55:17 <sarob> thinrichs: yup
17:55:20 <thinrichs> #topic Open Discussion
17:55:57 <sarob> regarding policy summit
17:56:10 <skn_> did we submit the design summit request for Congress for Paris?
17:57:05 <thinrichs> skn_: we submitted a talk request for the Paris summit.
17:57:09 <sarob> im working on a space at vmware palo alto, then i will start up a more official meeting request
17:57:16 <thinrichs> skn_: sarob was talking about the policy summit, which is in Sept.
17:57:27 <sarob> skn_: right
17:57:47 <cloudtoad> Is there a link for the policy summit?
17:57:54 <skn_> thinrichs: Got it.  I know this one is Sep 18-19, but I was asking about the Paris thing
17:58:25 <sarob> cloudtoad: not until i get the place locked now, then eventbrite will be set
17:58:31 <cloudtoad> Sweet
17:58:41 <sarob> cloudtoad: with all the info and invites
17:58:50 <skn_> sarob: Great
17:58:58 <pballand> we will also be attending the OpenStack silicon valley event on Sept 16th
17:59:37 <sarob> pballand: is martin talking up policy or just attending
17:59:54 <skn_> Are you guys attending OpenStack meet up tomorrow night?
18:00:15 <sarob> #info sarob working on the policy summit location then eventbrite will be set up
18:00:31 <thinrichs> We're out of time for today.  We can continue on #congress.
18:00:40 <thinrichs> Thanks all!
18:00:44 <sarob> cheers
18:01:02 <thinrichs> #endmeeting