17:01:38 <thinrichs> #startmeeting CongressTeamMeeting
17:01:39 <openstack> Meeting started Tue Jun 10 17:01:38 2014 UTC and is due to finish in 60 minutes.  The chair is thinrichs. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:01:40 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:01:42 <openstack> The meeting name has been set to 'congressteammeeting'
17:01:59 <thinrichs> So let’s start with action items from last week.
17:02:12 <thinrichs> kudva: how are the builtins progressing?
17:03:05 <kudva> I completed the basic builtin with many rounds of comments from Tim
17:03:34 <thinrichs> And it was merged!
17:03:38 <kudva> sending  link in a minute
17:03:57 <kudva> Thanks a lot!
17:04:11 <banix> great
17:04:16 <kudva> https://review.openstack.org/#/c/97673/
17:04:23 <kudva> So there were more comments, especially
17:04:55 <kudva> one cool thing I learned about python (storing a function in a class), and then calling it via code(*arglist): thanks Tim :)
17:05:10 <kudva> But I messed up some rebasing and so we are missing some commits in the full patch set
17:05:15 <thinrichs> Python can be fun!
17:05:18 <kudva> Now the next step is to
17:05:34 <kudva> take a real use case where I can apply builtin
17:05:48 <rajdeep> hi guys
17:05:59 <kudva> I was hoping sjcazzoll would be here. the host evacuation example would be a good one (but I have others in mind as well)
17:06:04 <thinrichs> hi rajdeep
17:06:10 <thinrichs> sjcazzoll sends his regrets.
17:06:38 <kudva> where one can combine predicates such as x > 10 (where 10 may be the number of alerts from a host)
17:06:43 <thinrichs> rajdeep: we just started hearing about kudva’s progress on builtins.
17:06:45 <kudva> with other predicates
17:06:51 <rajdeep> ok..
17:06:54 <kudva> hi rajdeep
17:07:08 <rajdeep> hi kudva
17:07:12 <kudva> So my next items:
17:07:23 <kudva> 1. hook builtin to runtime.py
17:07:30 <thinrichs> kudva: sjcazzoll’s use case may very well need the builtins.  Builtins are definitely important in practice.
17:08:00 <kudva> 2. an example such as host evactuation with builtins (have to think if we need ceilometer integration). will craft an example
17:09:34 <thinrichs> Sounds good.  I was looking into (1) today, since it requires knowledge deep into the policy engine.
17:09:45 <thinrichs> I’ll let you know what I figure out.
17:09:50 <kudva> I need a bit of tutorail on runtime.py
17:09:53 <kudva> if that is possible
17:10:04 <thinrichs> May require a phone call.
17:10:08 <kudva> Mainly want a quick overview of the four classes there.
17:10:09 <kudva> okay
17:10:18 <thinrichs> Let me know if that’s useful for you.
17:10:28 <thinrichs> Let’s move on to rajdeep.
17:10:30 <kudva> Yes, it would be. Perhaps the whole team may be interested
17:10:45 <thinrichs> rajdeep: how did the data source docs go?
17:11:00 <rajdeep> i have added details for nova
17:11:12 <banix> a hangout meeting may be useful
17:11:27 <rajdeep> and got feedback from sergio
17:11:30 <thinrichs> kudva: I’m guessing that’s something people will only want to understand if they want to modify the policy engine.
17:11:50 <kudva> okay, I am interested for sure
17:12:20 <thinrichs> Anyone who wants an overview: email me: thinrichs@vmware.com
17:12:29 <thinrichs> rajdeep: sorry for the interruption.  please continue
17:12:54 <rajdeep> sergio's use case needs host status which is not available in Nova
17:13:17 <rajdeep> looks like we will need to build driver for ceilometer to get that data
17:13:20 <thinrichs> It just doesn’t exist in Nova, or the driver we have doesn’t extract it?
17:13:37 <rajdeep> it doesn't exist in nova
17:13:39 <kudva> We can either get host status from ceilometer or monitor nova notifications
17:13:41 <kudva> for host
17:13:46 <rajdeep> i checked the json returned
17:14:02 <kudva> Ceilometer actually gets the host status from nova "compute-metrics-update" notifications
17:14:05 <pballand> so does nova expose it on the message bus but not in REST?
17:14:15 <rajdeep> looks like
17:14:19 <skn_> Oh, finally I got into this room
17:14:22 <rajdeep> it exposes in message bus
17:14:41 <rajdeep> should driver be looking for message bus notifications as well?
17:15:12 <kudva> ceilometer just picks up the nova notifications for host. We can get this from the notification bus or ceilometer api
17:15:13 <pballand> rajdeep: that is reasonable, but how would it bootstrap initial state?
17:15:14 <banix> ideally the monitoring engine should do that
17:15:44 <thinrichs> cloudtoad has been working on adding the ability to listen on the usual OS message bus.
17:15:46 <pballand> banix: what do you mean by monitoring engine?  ceilometer?
17:15:56 <banix> pballand: yes
17:15:57 <kudva> pballand: what do you mean by bootstrap initial state?
17:16:25 <rajdeep> i guess writing ceilometer driver would be cleaner design
17:16:27 <pballand> kudva: if all you get are updates, what is the initial state (when congress starts after nova has been running)
17:16:39 <rajdeep> so that we stick to REST
17:16:47 <thinrichs> rajdeep: agreed that having a driver for ceilometer would be cleaner.
17:16:49 <rajdeep> what do you guys think
17:17:04 <kudva> pballand: ceilometer will give you a snapshot of the current situation of the metric, which it gets from notifications usually
17:17:17 <banix> rajdeep: agree
17:17:25 <kudva> rajdeep: agree
17:17:48 <rajdeep> another problem is the ceilometer metrics are very closely tied to kvm
17:17:48 <pballand> long term, the message bus is a more efficient means of receiving updates - but sticking with REST initially simplifies the design
17:17:51 <skn_> Does ceilometer maintain the current state of the hosts?
17:18:08 <pballand> rajdeep: that sounds like a blueprint for ceilometer to handle :)
17:18:15 <rajdeep> e.g host status doesn't make sense if we use vCenter behind nova
17:18:44 <rajdeep> skn_ : i think so it does
17:18:46 <pballand> rajdeep: wouldn’t ceilometer handle that?
17:19:30 <skn_> rajdeep: I meant, all vhosts, not just kvm
17:19:56 <thinrichs> rajdeep: the policy engine needs to trust the data it’s given from components.  If the user doesn’t like that data, it should be reflected in the policy.
17:20:15 <skn_> banix: hi, did you see my reply to your email?
17:20:29 <banix> skn_: hi; yes
17:20:49 <thinrichs> Perhaps this is a good transition to the use case action item.
17:20:51 <kudva> So take an example: if I have a ceilometer plugin that monitors number of fan alerts (or some other hardware alerts), and ceilometer puts it onto notification for the collector to pick up.  How does vcenter fit into this?
17:20:59 <banix> skn_: will respond; sorry for delay
17:21:03 <skn_> banix: great, may be we can catch up on some of that at a later time this week
17:21:15 <banix> skn_: sounds good
17:21:18 <rajdeep-1> sorry got disconnected
17:21:33 <thinrichs> We had talked about adding a list of data sources for each of the use cases.
17:21:44 <thinrichs> We had also talked about solicting feedback on them.
17:21:49 <thinrichs> #link https://docs.google.com/document/d/1ExDmT06vDZjzOPePYBqojMRfXodvsk0R8nRkX-zrkSw/edit#
17:22:12 <thinrichs> I see there’s been a bunch of editing.
17:22:23 <thinrichs> sarob, skn_: want to report?
17:22:54 <skn_> I was mostly looking at the specs and other docs
17:23:06 <skn_> I have some comments, not sure how to put them in
17:23:20 <thinrichs> How about the usual gdoc comments?
17:23:21 <skn_> thinrichs: specs that you wrote up
17:23:51 <skn_> Is there a gdoc for comments?
17:23:57 <sarob> i gots very typy
17:24:21 <thinrichs> sarob: :)
17:24:31 <sarob> hmm, looking for the gdoc link
17:24:49 <sarob> #link https://docs.google.com/document/d/1ExDmT06vDZjzOPePYBqojMRfXodvsk0R8nRkX-zrkSw/edit#heading=h.fbota2qx0jad
17:24:59 <thinrichs> skn_: I meant go into the google doc, highlight some text you want to comment on, and (right?) click to choose to leave a comment.
17:25:11 <thinrichs> sarob has entered several comments on the use case doc linked above.
17:25:26 <sarob> i pointed msteel and mestery at it
17:25:39 * mestery got it, it's on his reading list. :)
17:25:57 <sarob> ;)
17:26:01 <banix> mestery: always alert :)
17:26:17 <mestery> banix: It's easy when it calls my name out :)
17:26:19 <sarob> both have postively replied that they are looking at
17:26:29 <thinrichs> sarob: great!
17:26:31 <skn_> thinrichs: I meant the congress’ nova-like specs
17:26:53 <thinrichs> skn_: oh—I’d leave a comment in gerritt.
17:27:03 <sarob> isnt the user stories to turn into specs into blueprints?
17:27:28 <sarob> just sayin
17:27:35 <skn_> sarob: are we planning to merge the specs that tthinrichs wrote up with the user stories
17:28:00 <sarob> roger roger
17:28:07 <banix> blueprint if need to track with milestones, etc
17:28:28 <thinrichs> Can someone explain the diff between a nova-spec and a blueprint?
17:29:00 <thinrichs> Other than they live in different places w/ diff formats.
17:29:03 <kudva> thinrichs: can we take the host evacuation example or one use case and deep dive for that, so we can see the difference
17:29:08 <banix> i think the spec is better for keeping track of reviews and updating while blueprint has other features such as milestones
17:29:37 <banix> so the need to kinf of use both until something better with all features is available
17:31:03 <pballand> kudva: +1
17:31:16 <pballand> I agree it would be helpful to take one story all the way thorugh
17:31:21 <pballand> through
17:31:48 <skn_> thinrichs: for data sources, can we also get some from other policy project, e.g. neutro GBP?
17:31:52 <pballand> shall we start with host evacuation?
17:32:06 <kudva> rajdeep, pballand: since have a few people interested in host evacuation (including sjcazzoll), I second that.
17:32:36 <kudva> for one deep dive. We can have more than one (say neutron GBP example)
17:32:36 <thinrichs> skn_: get some what?
17:33:15 <skn_> data sources from other “policy” projects, in addition to the core OS components like nova?
17:33:41 <rajdeep-1> skn_ : we could write a driver for neutro GBP
17:33:49 <thinrichs> skn_: I don’t think the other policy projects are writing the same kinds of drivers we are to import data.
17:33:56 <rajdeep-1> but it is not prioritized
17:34:18 <rajdeep-1> looks like exploring a driver for ceilometer would make sense
17:34:41 <thinrichs> But as rajdeep-1 says we could write drivers for any collection of APIs.
17:34:57 <skn_> rajdeep-1: one idea we discussed earlier was to work with the other policy projects too, to enforce things.
17:35:08 <thinrichs> In terms of writing new drivers we want to prioritize based on use-case prioritization.  Which is why it’s important to know which use cases utilize which data sources.
17:35:19 <rajdeep-1> sure thinrichs : what do you suggest
17:35:55 <kudva> rajdeep-1: I can work with you on the ceilometer driver if we decide to pursue it. Would like to understand better how the drivers are implemented.
17:35:57 <skn_> For use case prioritization, we should probably take one from each categories of use cases
17:36:15 <rajdeep-1> kudva : that is great
17:36:46 <pballand> I think thinrichs is saying that we need more clarity of the required data for each use case
17:37:12 <pballand> having that would help drive data source development
17:37:55 <skn_> Let’s start a deep dive on something then
17:38:04 <skn_> it’ll help us understand better
17:38:04 <thinrichs> Once we have data source drivers needed for a use case, we just write the policy for that use case.
17:38:05 <thinrichs> So there’s (hopefully) no development to do once we have the data source drivers in place.
17:38:43 <thinrichs> Perhaps we should add our basic example to the use case doc, along with the data sources needed, and the policy so that we can all see what technical info we need.
17:38:54 <thinrichs> I’ll volunteer to do that.
17:39:11 <banix> thinrichs: that’s a good idea
17:39:30 <pballand> so for the evacuation use case, what data sources do people envision the policy referencing?
17:39:36 <thinrichs> #action thinrichs will add private/public network example to use case doc
17:39:37 <skn_> thinrichs: how does the action happen, if its needed? Data sources is one way from the drivers to the Congress, sounds like
17:40:44 <thinrichs> skn_: we’re working on monitoring first.  Then we’ll work on enforcement (adding actions).  I imagine we can talk about just monitoring for almost all of our use cases.
17:41:02 <kudva> pballand: not sure what sjcazzoll had in mind. But in general, the host will be evacuated if there are a high number of failures on the host. So a ceilometer plugin on the host or nova notification from the hosts are picked up by ceilometer.
17:41:21 <kudva> So, we need to get some data from ceilometer on this to say if it's time to evacuate.
17:41:52 <kudva> we may have a threshold on a policy: host_owner, and faults> x and any other conditions
17:41:53 <skn_> thinrichs: but most of the use cases make sense only if we have some action, right?
17:41:54 <pballand> possible evacuation sources that come to mind: nova:host->vm bindings, ceilometer: metrics on hosts and networks
17:42:27 <thinrichs> skn_: Let’s put it this way.  The action mechanism we’ll use is independent of the use cases.  So once we get it in, we can execute actions.
17:42:51 <skn_> thinrichs: Agreed
17:42:52 <pballand> monitoring is a precursor to actions - the first way to “execution” actions is “tell the admin to fix it"
17:43:00 <kudva> pballand: what do you mean by>? host->vm bindings, and metrics on hosts and networks?
17:43:01 <thinrichs> But until we have monitoring in place, we can’t be executing actinos.  So the use cases aren’t driving those features.
17:43:10 <thinrichs> The use cases are only driving the development of data sources.
17:44:04 <pballand> kudva: sorry I’m not being clear - I mean we need host:vm bindings, and that would come from nova, and we also need additional metrics that would likely come form ceilometer
17:44:17 <skn_> thinrichs: OK, got iit.  The priority now is to have data source developments, e.g. via drivers
17:44:19 <kudva> pballand: got it
17:45:03 <thinrichs> skn_: Use cases/stories are important for 2 reasons: to drive technical development and to help explain why the project is useful to people.
17:45:19 <skn_> I can work on the data sources in the security use cases that I’ve added
17:45:20 <thinrichs> For technical development, the use cases are only helping prioritize data source development.
17:45:34 <thinrichs> But independently we need to push on policy engine/action execution development
17:45:50 <thinrichs> skn_: great!
17:46:10 <thinrichs> skn_: rajdeep is the expert on datasource driver development.
17:46:39 <thinrichs> Speaking of which I’m making progress on integration of datasources, policy engine, and DSE.
17:46:49 <thinrichs> And there are a number of small changes to the datasource drivers we have.
17:47:00 <skn_> I meant, I can start with putting the info on doc and then take it from there
17:47:17 <banix> thinrichs: link? if the code is out there
17:47:19 <thinrichs> skn_: got it.
17:47:34 <thinrichs> Starting at:
17:47:35 <thinrichs> https://review.openstack.org/#/c/95965/7
17:47:36 <thinrichs> Ending at:
17:47:37 <thinrichs> https://review.openstack.org/#/c/96978/
17:47:48 <skn_> Rajdeep: I can take some help in understanding what all data sources do we already have (or in plan) for Congress
17:48:07 <thinrichs> The very last change in the series is unfinished (and the commit msg says so).  Don’t bother reviewing that one yet.
17:48:12 <banix> thinrichs: thanks
17:48:19 <thinrichs> But the others are ready for review.
17:48:27 <rajdeep-1> sure skn_
17:48:42 <rajdeep-1> currently nova and neutron drivers are implemented
17:48:56 <thinrichs> I’ll go through and add reviewers for the ones who I know.
17:49:03 <rajdeep-1> https://github.com/stackforge/congress/tree/master/congress/datasources
17:49:19 <skn_> rajdeep: oh ok. thanks.
17:49:22 <kudva> rajdeep-1: how do you plan to start ceiometer integration
17:49:36 <thinrichs> I know pballand is swamped.  So if everybody could pitch in and do some reviewing, that would help.
17:49:42 <skn_> isn’t dereck@plexxi doing something along this line?
17:50:12 <banix> thinrichs: will do
17:50:27 <skn_> thinrichs: sure
17:50:30 <thinrichs> skn_: cloudtoad isn’t doing this kind of integration.  I talked to him yesterday.
17:50:49 <rajdeep-1> kudva : i will look at what ceilometer has and come up with a design doc
17:50:52 <skn_> thinrichs: I see
17:51:05 <kudva> rajdeep-1: okay let me know, thanks
17:52:14 <skn_> OK, so looks like two actions on me, 1. review data source code, and 2. add some data sources needed for the security use cases
17:52:35 <thinrichs> skn_: yes—thankyou!
17:52:56 <skn_> thinrichs: feel free to put these in actions
17:53:14 <thinrichs> #action skn_ will add data sources to the use case doc
17:53:19 <pballand> skn_: just to be clear, by 2) - do you mean update the use case doc, or or actually work on the data sources?
17:53:32 <skn_> update the doc
17:53:42 <skn_> we have to first get that done
17:53:45 <thinrichs> #action skn_, banix, (and others who have time) will review thinrichs code changes
17:54:38 <thinrichs> Time check: 5 mins left.
17:54:42 <thinrichs> Open discussion
17:54:56 <skn_> any plans on the blueprint?
17:55:30 <thinrichs> We had talked about working thru host evac from Story to spec and blueprint.
17:55:36 <thinrichs> Anyone want to work on that?
17:56:19 <skn_> thinrichs: Is it per use cases? I thought we are going to put in multiple of them together
17:57:12 <thinrichs> Looking into whether we can post blueprints.  Anyone know how we can add Congress as a blueprint project?
17:57:41 <skn_> sarob: are you there?
17:57:50 <sarob> here
17:58:11 <thinrichs> sarob: do you know how we get Congress added as a project when creating a new blueprint?
17:58:17 <skn_> sarob: Any ideas on how to add congress as a blueprint project?
17:58:26 <sarob> very
17:58:50 <thinrichs> sarob:???
17:58:52 <sarob> you mean incubated right?
17:59:11 <sarob> im working on incubating training-guides as well
17:59:17 <sarob> so i gots the skinny
17:59:19 <pballand> does Congress need to be inclubated before it can be listed as a project for blueprints?
17:59:25 <sarob> nope
17:59:52 <skn_> ok, then we start with blueprints and then work on getting it incubated, right?
17:59:58 <sarob> general practice now is repo spec patches then blueprints
18:00:06 <pballand> I’ll volunteer to research how to get congress listed as a project when creating blueproints
18:00:07 <sarob> we following that so good and good
18:00:22 <sarob> ah, create a launchpad project?
18:00:28 <skn_> pballand: thanks
18:00:35 <sarob> i can do that for you
18:00:40 <sarob> with you
18:00:40 <pballand> sarob: ah - sweet!
18:00:53 <thinrichs> #action sarob will add Congress as a launchpad project
18:00:59 <thinrichs> And we’re out of time.
18:01:06 <thinrichs> Thanks all!  Follow up on the ML if need be.
18:01:07 <sarob> done
18:01:13 <skn_> Bye guys
18:01:14 <thinrichs> sarob: sweet!
18:01:15 <banix> bye
18:01:21 <thinrichs> #endmeeting