17:04:55 <thinrichs> #startmeeting CongressTeamMeeting
17:04:56 <openstack> Meeting started Tue Apr 22 17:04:55 2014 UTC and is due to finish in 60 minutes.  The chair is thinrichs. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:04:57 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:04:57 <thinrichs> Hi rajdeep
17:05:00 <openstack> The meeting name has been set to 'congressteammeeting'
17:05:28 <thinrichs> Usual agenda today: recap on action items, look at todos for alpha release, open discussion
17:06:01 <pballand> I’d like to chat a bit about the conference at the end if there is time
17:06:03 <thinrichs> Let's start with rajdeep.  Can you report on data drivers?
17:06:16 <rajdeep> yes..
17:06:35 <rajdeep> i completed some key integration points for nova driver
17:06:51 <rajdeep> getting the server and flavor data in and converted to tuples
17:07:02 <rajdeep> the code has been checked in
17:07:10 <rajdeep> and tim approved today morning
17:07:35 <rajdeep> #link https://review.openstack.org/#/c/88228/
17:07:59 <thinrichs> So for our running example (network connections to VMs with group constraints), do we have all the data sources that we need from Nova and Neutron?
17:08:12 <rajdeep> i guess so
17:08:32 <rajdeep> one challenge which i came across is how much API coverage should driver have
17:08:52 <rajdeep> POST and DELETE APIs are not needed
17:09:19 <thinrichs> Right--for data source drivers we want only API calls that do not CHANGE anything
17:09:29 <rajdeep> so focussing on GET requests and prioritize
17:09:48 <pballand> rajdeep: I’m working through those details now - I expect that some tables will just “appear” when the data source plugin is enabled, and others can be created (for users to manually provision data through the API)
17:09:56 <thinrichs> But as we discussed during review, ideally we'll support the core API for each component.
17:10:17 <rajdeep> yes we should strive for that
17:10:25 <rajdeep> cover as much possible and applicable
17:10:55 <rajdeep> now for the use case i had few questions we can discuss during the end
17:10:56 <thinrichs> Eventually we'll want libraries where users say "I have Neutron v 1.2.1", and we'll use the data drivers for that version of the codebase.
17:11:16 <thinrichs> But for POC, I think the coverage we have is good.
17:11:24 <thinrichs> rajdeep: sounds good.
17:11:44 <rajdeep> our drivers will also evolve with the openstack and their component versions
17:11:55 <rajdeep> which is not designed in the driver itself
17:12:11 <rajdeep> i am assuming we will have separate driver for each component and version
17:12:17 <thinrichs> rajdeep: agreed
17:12:19 <rajdeep> perhaps a driver factory on the top
17:12:33 <rajdeep> to load the right driver depending on the version
17:13:04 <thinrichs> Sounds good.
17:13:14 <pballand> qustion: do we expect drivers to be registered at startup?
17:13:15 <thinrichs> kudva: how are the builtins going?
17:13:26 <kudva> Okay, I got the feedback from Tim on my document.
17:13:32 <kudva> I have started implementation in my sandbox.
17:13:40 <thinrichs> pballand: not sure.  Ideally we could hot-swap at runtime
17:13:54 <thinrichs> kudva: great!
17:13:56 <kudva> In a week or so, I can put the code in for review
17:14:18 <rajdeep> pballand : drivers can be instantiated but you need to call def update_from_datasource(self)
17:14:20 <kudva> Right now I am assuming my own builtin directory in congress/policy
17:14:28 <kudva> but we can decide where to put it.
17:14:34 <rajdeep> to pull data
17:14:37 <thinrichs> kudva: that sounds about right.
17:15:12 <thinrichs> I'm looking forward to doing that review--builtins have been on the list for a long time.
17:15:32 <thinrichs> kudva: any issues come up that need discussing with the group?
17:17:48 <thinrichs> perhaps kudva had to step away.
17:18:11 <thinrichs> pballand: how's the API going?
17:18:19 <pballand> can I ask more questions about data source plugins?
17:18:23 <thinrichs> sure
17:18:44 <kudva> sorry, someone came into my office
17:18:44 <pballand> do we want validation between the policy and data sources?
17:19:15 <pballand> e.g. if a data source can dynamically start publishing data, can the policy be declared over it before it joins?
17:19:17 <thinrichs> I could imagine telling the user that there are data sources referenced in policy that we don't have data for.  Is that what you mean?
17:19:50 <pballand> yeah, for the API, I’m wondering how strict we should be with schemas and validation
17:19:58 <thinrichs> pballand: I'd say yes.  I'm thinking of upgrades.  Suppose user decides to upgrade Neutron to the next version.
17:20:25 <thinrichs> Either they upgrade the data source first or the policy first.  Either way there's a window where we have a mismatch.
17:20:51 <rajdeep> or we have policy meta data which ties it to a version of the data source
17:20:54 <pballand> so are tables versioned? ('nova-diablo:vm’)
17:21:48 <thinrichs> I could imagine versioning tables temporarily while doing an upgrade.  I think we need to think through what the right behavior should be during upgrade in terms of policy.
17:21:59 <thinrichs> policy violations that is.
17:22:26 <thinrichs> I think for alpha release we're just assuming upgrades don't happen.
17:22:41 <pballand> I propose we allow adding items to tuples, but force a table name change if columns are removed or the meaning of a column changes
17:23:12 <thinrichs> If we think this is important right now, does anyone want to volunteer to think through what happens during upgrade and data-source changes?
17:23:18 <pballand> so we may wind up with something like nova-vm-version1, nova-vm-version2
17:23:24 <thinrichs> pballand: forcing a table-name change would require changing policy.
17:23:47 <pballand> yup - so we’d like to avoid table semantics if possible
17:23:58 <pballand> avoid *changing* table semantics
17:24:24 <thinrichs> Policy is hard to get right--I worry about telling people they need to change it every time they upgrade.
17:24:40 <thinrichs> Without knowing there are no other options.
17:24:53 <rajdeep> perhaps we could allow schema changes
17:24:59 <pballand> I think that’s good enough for now - no need to go through a full upgrade modeling - we just need to be clear that once a table is defined for a data source, we should avoid chanign the schema
17:25:01 <thinrichs> Sorry--got to step away for a minute.  pballand, can you take over?
17:25:01 <rajdeep> and ignore errors because of missing data
17:25:11 <pballand> thinrichs: yup
17:25:58 <pballand> what I was going to propose is that for a given datasource::table name, we can add columns between releases, but not remove any
17:26:24 <rajdeep> well it will be easier to add new colums at the end of tuple
17:26:37 <pballand> rajdeep: exactly - that would be allowed
17:26:44 <rajdeep> i have a new method to get metadata for a tuple
17:26:57 <rajdeep> def get_tuple_metadata(self, type): 56	        if type == self.SERVERS: 57	            return ("id", "name", "host_id", "status", "tenant_id", "progress", 58	                    "user_id", "image_id", "flavor_id") 59	        elif type == self.FLAVORS: 60	            return ("id", "name", "vcpus", "ram", "disk", "ephemeral", 61	                    "rxtx_factor")
17:27:05 <rajdeep> sorry
17:27:09 <pballand> if a version change calls for changing the type of a column (or removing it), we can either leave the column empty and add a new column, or create a new table
17:27:28 <pballand> ok, thanks for the input
17:27:32 <pballand> lets let kudva finish :)
17:27:42 <rajdeep> ok
17:27:49 <kudva> I think many of my questions I can ask around the code
17:28:23 <pballand> many are probably best answered by thinrichs
17:28:24 <kudva> So, I would like to have the code out to you
17:28:54 <pballand> do you want to discuss in IRC, or after you work on the code?
17:28:56 <kudva> and then have a discussion around the code to make sure the implementations, concepts I have captured properly. This way we will have a concrete strawman
17:29:05 <pballand> perfect
17:29:21 <kudva> after I have a version ready
17:29:47 <kudva> we can do a code/concept review together. Once I get feedback, I'll go back rework it
17:30:07 <pballand> that sounds good - anything else to discuss in this forum?
17:30:08 <kudva> < 1 week is the time I have set for myself
17:31:25 <pballand> ok, I’ll give a summary of where I am at with the API
17:32:02 <pballand> I’m starting to put together a design doc just on the api/data model
17:32:30 <pballand> I will have something to publish by the end of the week
17:32:59 <rajdeep> ok, i am assuming data model will reflect the driver data structures
17:33:15 <pballand> rajdeep: yes, that’s the idea
17:33:26 <pballand> 1/2 will be data-source oriented
17:33:32 <pballand> and the other half will be based on policy
17:33:40 <rajdeep> ok
17:34:14 <pballand> one addition is the ability to have API-sourced tables, which act like the driver data, except the user can create/update them through the API
17:34:33 <pballand> I expect this to be very useful for testing, demoing, and training
17:34:56 <rajdeep> you mean REST APIs?
17:35:11 <pballand> yes
17:35:41 <pballand> unlike datasource-plugins, the framework would manage data storage for these ‘internal’ tables
17:35:44 <rajdeep> that will be great
17:35:46 <pballand> (better name welcome)
17:35:58 <rajdeep> and perhaps a client library :)
17:36:38 <pballand> so there are 3 types of data - ephemeral data from a datasource-plugin, static data managed through the API and stored by the framework, and derived data computed and exposed by the policy
17:37:16 <pballand> (really the static/internal data is just a datasource-plugin that is always available)
17:37:26 <pballand> does that sound reasonable?
17:37:36 <thinrichs> Sounds good to me.
17:37:47 <rajdeep> yes it does where will static data be stored?
17:37:51 <pballand> thinrichs: you’re back :)
17:38:09 <pballand> rajdeep: my thought was to just use sqlite for now
17:38:14 <thinrichs> Been following but repeatedly interrupted.  Should be back for good now.
17:38:31 <thinrichs> Why not use our normal datastore?
17:38:48 <pballand> what is our “normal” datastore?
17:38:50 <thinrichs> Are we worried about persistence?  That's something we need to worry about for policy as well.
17:39:07 <pballand> the ‘internal’ tables would need to be persisted just like policy
17:39:17 <pballand> I am considering putting both in sqlite for now
17:39:38 <thinrichs> Our in-memory database that we use to cache data sources, reason about policy, etc.
17:39:41 <pballand> (it’s not a scalable solution, but should be adequate for the use case)
17:40:01 <pballand> ah, for the internal tables, I think it would be useful for the data to survive restart
17:40:12 <pballand> (and policy should definitely survive restart)
17:40:23 <thinrichs> We can always look at the internal tables as extra policy statements.
17:40:39 <thinrichs> And use the same mechanism for persisting them that we do for persisting policy.
17:40:50 <pballand> thinrichs: I’d rather model them closer to datasource plugins where we manage the persistance
17:41:30 <pballand> in fact, maybe there are just 2 types of data (datasource and policy) and the ‘internal’ tables are just a datasource plugin that we ship with
17:41:33 <thinrichs> I had thought we'd address persistence at the same time we address the need to store history.
17:41:46 <thinrichs> For auditing, that is.
17:42:10 <thinrichs> So the most recent policy/data is not the only thing we want to persist--we want the history of those things.
17:42:18 <thinrichs> Or at least some of the history.
17:42:35 <pballand> hmm, I don’t think we want to tackle history yet
17:43:15 <pballand> how about I write up a proposal and send it out
17:43:36 <thinrichs> That's fine if we want to delay.  But if we use the usual database approach to atomicity we get both history and persistence solved at the same time.
17:43:59 <thinrichs> pballand: proposal sounds good
17:44:13 <pballand> except the scale requirements for the different things are entirely different orders of magnatude
17:44:45 <pballand> ok, other than the modeling (document end of week), I am working on the runtime
17:45:25 <pballand> my thought was to take the existing eventlet runtime and make it more pluggable so it can run API, policy, and message bus components
17:45:46 <pballand> optionally, data source plugins can run in the loop too
17:46:01 <pballand> any thoughts?
17:46:27 <thinrichs> Can you give me a brief description of the eventlet runtime?  It's the scheduler for the policy/message bus/API, etc.?  ???
17:47:04 <rajdeep> integration with the eventlet engine will be good
17:47:11 <pballand> it is basically a select loop - as each component blocks, something else gets scheduled
17:47:33 <thinrichs> So this is cooperative threading?
17:48:13 <pballand> basically yes
17:48:34 <pballand> python sucks at threads, so the greenthread approach is popular
17:49:26 <pballand> if we want to go multi-processor, we can use multiple processes (orchastrated over message bus) or rewrite in a different language
17:49:33 <thinrichs> So we make our classes inherit from some special threading class and insert yield() calls at appropriate places?
17:49:51 <pballand> thinrichs: exactly
17:50:09 <pballand> (except you probalby don’t need many/any yields)
17:50:49 <thinrichs> Let's keep everything in Python for now.
17:51:06 <thinrichs> If we need to we can think about multi-processor approaches later.
17:51:24 <pballand> agreed - do we think we can all live in a single event loop for now?
17:51:46 <thinrichs> yes
17:51:51 <pballand> (I don’t know how well the plexxi DSE code will cooperate
17:51:59 <pballand> anyone from plexxi on?
17:52:23 <thinrichs> BTW, I'll try to get someone from Plexxi to start attending these meetings.
17:52:40 <thinrichs> Oops--running short on time.
17:52:48 <thinrichs> Are we finished with API?
17:53:01 <pballand> yup, I’ll stop rambling :)
17:53:04 <thinrichs> :)
17:53:16 <thinrichs> Last things from last week's action items...
17:53:24 <thinrichs> I was supposed to get the Plexxi code merged.
17:53:32 <thinrichs> Done (though I didn't do much to make it happen).
17:53:53 <thinrichs> I'll volunteer to work with pballand to see if we can get everything integrated.
17:53:58 <thinrichs> message bus, policy, data drivers
17:54:06 <pballand> sounds good :)
17:54:26 <rajdeep> plexxi code check in is good news
17:54:38 <thinrichs> agreed
17:55:12 <thinrichs> rajdeep: do you have tasks for next week that you're working on?
17:55:22 <thinrichs> The rest of us do.
17:55:37 <rajdeep> i am planning to add some more usecases to nova
17:55:43 <rajdeep> driver
17:55:51 <thinrichs> Sounds good.
17:55:59 <thinrichs> So 5 minutes of open discussion.
17:56:12 <thinrichs> rajdeep: you had some questions earlier.  Were they answered?
17:56:20 <pballand> I was wondering if anyone was heading to Atlanta?
17:56:36 <rajdeep> i had a question on groups and users construct in our use case
17:56:45 <rajdeep> groups are not used in keystone..
17:56:52 <thinrichs> pballand: I'm headed to Atlanta, though late.  Arriving Wed night.
17:57:13 <thinrichs> rajdeep: for groups we could use the internal tables pballand was mentioning.
17:57:29 <thinrichs> Or we could use the AD integration that's there.  We'd need to port it over to your framework.
17:57:37 <rajdeep> yeah ..
17:57:54 <rajdeep> that would be better
17:57:59 <thinrichs> The downside to AD is that you need it running, which is inconvenient for demos/etc.
17:58:14 <rajdeep> or use a light weight ldap
17:59:30 <thinrichs> Users know they need to install OS to do a reasonable demo.  But the more things we make them install to see how use case number 1 works, the more likely they'll get hung up on problems that are irrelevant.
18:00:47 <thinrichs> I'd lean toward using only OS components for our first release, just to minimize that problem.  But include AD drivers/LDAP drivers/whatever, so they know it's not just about OS.
18:00:59 <rajdeep> +1 to keep it simple
18:01:20 <thinrichs> And we're over time.
18:01:25 <pballand> :)
18:01:29 <pballand> thanks everyone
18:01:34 <rajdeep> thanks bye
18:01:37 <thinrichs> So thanks everyone, and let's keep up discussions on ML and through google docs!
18:01:45 <thinrichs> #endmeeting