17:01:56 <thinrichs> #startmeeting CongressTeamMeeting
17:01:57 <openstack> Meeting started Tue Dec 16 17:01:56 2014 UTC and is due to finish in 60 minutes.  The chair is thinrichs. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:01:58 <sarob> Nice
17:01:59 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:02:01 <openstack> The meeting name has been set to 'congressteammeeting'
17:02:03 <sarob> Welcome people
17:02:27 <arosen1> hi
17:02:34 <thinrichs> Before I forget, I'm thinking we should cancel the next 2 weeks of meetings, for the holidays.
17:02:41 <arosen1> +1  :)
17:02:58 <thinrichs> Any objections?
17:03:28 <sarob> Not here
17:03:29 <sarob> Merry Christmas
17:03:31 <cloudtoa_> I object!  Actually, I don't. I don't object.
17:03:41 <thinrichs> I'll take that as a no.  So no meetings the next 2 weeks.  We'll resume Tuesday, Jan 6.
17:03:45 <thinrichs> cloudtoa_: :)
17:03:50 <sarob> Conflicted
17:04:13 <thinrichs> Sorry—sarob are you considering objecting?
17:04:28 <cloudtoa_> We can  meet up in #Congress for specific items with ppl... no?
17:04:34 <cloudtoa_> At any time..
17:04:44 <thinrichs> cloudtoa_: absolutely we can meet up in #Congress
17:05:10 <thinrichs> This week people will probably generally be in #congress all day.
17:05:25 <thinrichs> But I'm guessing the 2 weeks after that it'll be spotty at best.
17:05:42 <thinrichs> But there's always the mailing list.  I think most people at least check email over the holidays.
17:06:08 <thinrichs> Just remember to include [congress] in the subject line.
17:06:29 <thinrichs> For those who might not know, we use the standard openstack developer's mailing list.
17:06:31 <thinrichs> openstack-dev <openstack-dev@lists.openstack.org>
17:06:50 <thinrichs> If you're not subscribed, please do so.  Just google for it.
17:07:06 <thinrichs> And I'd recommend setting up filters since it's a high-volume list.
17:07:29 <thinrichs> I think mine deliver anything with 'congress' or 'policy' in either subject or body to my inbox.
17:08:02 <thinrichs> The first milestone is upon us.  Dec 18.
17:08:33 <thinrichs> Tried to glance at which blueprints are due then, but that's harder than I expected.
17:09:04 <thinrichs> I know having multiple policy support (mine) is due Dec 18.  I expect it should happen.  Just adding finishing touches.
17:09:12 <thinrichs> Anyone else signed up for Dec 18?
17:09:28 <cloudtoa_> Yes... I'll have the control-bus blueprint submitted today.
17:09:58 <thinrichs> cloudtoa_: great!
17:10:02 <arosen1> thinrichs they all show up in a list here: https://blueprints.launchpad.net/congress unfortunately  this page doesnt' say kilo-1,2,or 3 unless you click on the blueprints individually.
17:10:08 <thinrichs> That's a good reminder actually.
17:10:09 <samta> Hi, i am able to get the count working like this: error(x,c):-p(x),count(x,c)
17:10:33 <thinrichs> arosen1: just figured out: if you click on kilo it shows the blueprints with kilo-1, kilo-2, etc.
17:10:38 <thinrichs> #link https://blueprints.launchpad.net/congress/kilo
17:11:01 <arosen1> ah okay nice
17:11:17 <thinrichs> samta: great!  I see that the datalog-aggregates are assigned kilo-1.
17:11:31 <thinrichs> Do you think that'll get done this week?
17:11:39 <samta> Yeah..if looks good to u..will push my patch..
17:11:41 <thinrichs> Or should we move the deadline back to kil-2
17:11:43 <samta> yes this week
17:12:00 <thinrichs> samta: I'll take a look, and we can discuss offline.
17:12:09 <samta> If i am nt stuck with any test..
17:12:14 <samta> Sure
17:12:51 <thinrichs> arosen1: have you finished the refactor-datasource-driver-framework blueprint?
17:12:58 <thinrichs> That's the only other one slotted for kilo-1
17:13:07 <thinrichs> that isn't finished.
17:13:32 <arosen1> thinrichs:  i would say more or less though this was kinda of a blanket blueprint for anything i find along the way.
17:14:14 <thinrichs> arosen1: okay—so when you've finished refactoring, can you mark that as Implemented?
17:14:16 <arosen1> We can probably set it as implemented though
17:14:25 <arosen1> k will do .
17:15:10 <thinrichs> cloudtoa_: so you're planning to push a revised spec for ...
17:15:11 <thinrichs> #link https://review.openstack.org/#/c/134686/
17:15:37 <cloudtoa_> Yes
17:16:19 <thinrichs> pballand: think you'll have time to leave comments on that sometime this week?
17:16:42 <pballand> yes
17:16:54 <thinrichs> The more design we can get settled before the holidays the easier it will be for people to hit the ground running in the new year.
17:17:55 <thinrichs> While we're on the topic of specs, if you've volunteered for a spec it's good to at least leave comments on it to flush out the design.
17:18:21 <cloudtoa_> We will leave comments on framework related specs in the next week or so.
17:18:53 <thinrichs> I'm looking for a spec that has a round of comments so we can all see what's supposed to happen.
17:19:17 <thinrichs> Try this.  Just 2 comments.  But you get the idea.
17:19:18 <thinrichs> #link https://review.openstack.org/#/c/134340/2/specs/kilo/publish-policy-results-to-bus.rst
17:19:49 <thinrichs> Let's go ahead and get through some status updates.
17:20:03 <thinrichs> Volunteers?
17:20:15 <arosen1> sure I can go first.
17:21:27 <arosen1> so I've continued working on the datasource refactoring mostly to refactor the sub tables out of the current neutron driver.
17:22:27 <thinrichs> So if you've written policies using neutron's tables, you may need to go back and change those.
17:22:44 <arosen1> from this work we added a few new features to to the datasource driver base class to make things easier. Now for HDICTS a datasource driver can set in-list =True which will help unroll data that is in this format dict(list(dict()). Previously, one needed to create an extra table to unroll this data.
17:22:55 <cloudtoa_> This is the way by you can specify how an arbitrary/nested data structure is converted to a table.. so peope don't have to write nested loops to do this?
17:22:56 <thinrichs> We're hoping to quickly get to the point where the tables for datasource drivers don't change, but we're not there quite yet.
17:23:14 <thinrichs> cloudtoa_: right
17:23:16 <arosen1> thinrichs:  unfortinately
17:23:29 <arosen1> cloudtoa_: yup exactly.
17:24:03 <arosen1> here's the blueprint for it: https://review.openstack.org/#/c/140550/ and implemenation that has already merged https://review.openstack.org/#/c/140560/
17:25:15 <thinrichs> One thing we realized this week that's especially cool about this line of work...
17:25:41 <arosen1> The other thing is 'parent-key' columns can now specify their name. So previously the schema for their column would return their name as parent_key. Now they can specify their name to be something like port_id. This is done via setting 'parent-col-name: <name> . In addition, the parent-key can now be used from one subtable to another which wasn't possible before.
17:25:41 <thinrichs> if people use this language alexsyip and arosen are working on to build their datasource driver
17:25:47 <kudva> Hi
17:26:08 <thinrichs> the schema for that driver is fixed and will not change.
17:26:30 <arosen1> bluepinrt: https://review.openstack.org/#/c/140875/  implementation: https://review.openstack.org/#/c/140855/ https://review.openstack.org/#/c/140876/
17:26:38 <thinrichs> There's no way for the datasource driver writer to make a mistake and for example sometimes leave out certain columns in the tables if the data coming back from the service doesn't have that column.
17:26:50 <thinrichs> kudva: hi
17:26:57 <cloudtoa_> What do you mean "the schema?"  We were thinking that it might actually be a "sub-schema" meaning for certain types of drivers, certain columns must be present... but additional columns may optionally be presented.
17:27:26 <thinrichs> cloudtoa_: the schema = the columns present in each table.
17:27:26 <arosen1> cloudtoa_:  i think that is still somewhat of an open question.
17:27:28 <cloudtoa_> So, for instance, drivers for different hypervisors have certain columns in common.
17:27:35 <sarob_> Thinrichs that's a big improvement
17:27:58 <arosen1> cloudtoa_:  if possible i think we'd want to move the differences into subtables
17:28:08 <cloudtoa_> Ok...
17:28:13 <thinrichs> The reason it's important to fix the schema (having optional columns is still somewhat open) is that we want to ensure that once someone writes policy, that policy is insulated from changes in the datasource driver to the extent possible.
17:28:16 <cloudtoa_> I'll buy that.
17:28:21 <arosen1> the neutron driver is a good example of this though since different plugins have different extensions
17:28:57 <arosen1> so i'm working on figuring out how this should work soon
17:29:02 <cloudtoa_> So, are we sending row updates to the policy engine or are we sending whole tables and diffing them in the policy engine?
17:29:23 <thinrichs> cloudtoa_: datasource driver (base class) computes deltas.
17:29:25 <cloudtoa_> Sorry, that's a different topic...
17:29:29 <thinrichs> policy engine receives deltas on the bus.
17:29:33 <thinrichs> But I agree—a different topic.
17:29:49 <thinrichs> Let's continue on with statuses.
17:30:01 <arosen1> that's it from me.
17:30:14 <thinrichs> Anyone else have questions for the group or making changes the group needs to know about?
17:30:47 <alexsyip> I’m working on performance improvements for the runtime.
17:30:56 <alexsyip> Starting with indexing for table lookups.
17:31:51 <thinrichs> alexsyip: that's some really important stuff, given our experiences running Congress in our own cloud.
17:32:00 <thinrichs> alexsyip: want to tell everyone what prompted this?
17:32:20 <alexsyip> Pierre wrote a policy for demo purposes in our vmware cloud
17:32:34 <alexsyip> and one of the queries would take 45min to complete.
17:32:56 <cloudtoa_> Seems reasonable.  You can get lunch during that time.
17:33:00 <alexsyip> We believe it’s because one of the tables has about 10,000 entries, and the runtime iterates through each entry to lookup a single email addr.
17:33:33 <alexsyip> So if we index the table, the lookup should be faster (not needing to iterate through all the entries).
17:33:43 <cloudtoa_> Are you using something like a bloom filter?
17:34:04 <cloudtoa_> Or thinking about it?
17:34:15 <alexsyip> I was planning to use a hash table.  I’m not sure how i’d use a bloom filter.
17:34:45 <cloudtoa_> It's a hash table.  Hash compute per entry, constant look up time.
17:36:12 <cloudtoa_> Is this an optimization in the policy engine or in the table code in the drivers?  Or is it both?
17:36:17 <thinrichs> So if we turn the linear walk over 10,000 entries into a hash lookup, it seems we'll get a speedup of 4 orders of magnitude (since the linear walk is in the inner loop).
17:36:20 <alexsyip> From what I remember a bloom filter only tells you if an item exists in a set.
17:36:55 <alexsyip> Let’s move this discussion to another room.
17:36:57 <cloudtoa_> ok.. I was just looking at a vswitch implementation that did mac-lookups with a bloom filter.
17:37:01 <cloudtoa_> ok
17:37:33 <thinrichs> I'm looking forward to having that indexing in place!
17:37:45 <thinrichs> Let's hear from the people signed up for blueprints due in kilo2.
17:37:52 <thinrichs> madhumohan: how are things going?
17:39:17 <thinrichs> Maybe he stepped away.
17:39:25 <thinrichs> zhipeng: how are things going?
17:39:46 <zhipeng> still in the beginning phase
17:39:47 <madhumohan> Got hold of antlrworks and now I am able to get the literals for execute[] call...
17:39:55 <zhipeng> have some questions about spec
17:40:19 <thinrichs> Sorry—zhipeng.  Let's have madhumohan go.  I'll look at the spec in the meantime.
17:40:33 <zhipeng> that's alright :)
17:41:14 <thinrichs> madhumohan: great!
17:41:44 <madhumohan> thinrichs: formula generated seems fine with Modals in place of Literals.. Need to get started with runtime implementation.
17:41:50 <thinrichs> Let me know if you want tips for antlrworks.  There are a few things that aren't so intuitive, but once you get the hang of it, it speeds up development quite a bit.
17:43:00 <thinrichs> madhumohan: sounds good.
17:43:20 <madhumohan> yes... there are some tricky bits... I was not able to generate Python code... I ended up using the command line version....
17:44:35 <thinrichs> Whatever works.
17:44:45 <thinrichs> zhipeng: I've got a link to the spec.
17:44:47 <thinrichs> #link https://github.com/stackforge/congress-specs/blob/master/specs/kilo/policy-engine-trigger.rst
17:45:13 <thinrichs> Do you want to ask questions of the group?
17:45:15 <zhipeng> ya sorry our tempo is pretty slow, we will catch up during the Xmas days :P
17:45:21 <thinrichs> It looks like the spec was merged.
17:45:31 <zhipeng> yep, but still got one question
17:45:34 <thinrichs> We can resubmit it so that we can discuss via comments in Gerrit, if that would be easier.
17:45:34 <zhipeng> about iii
17:46:18 <thinrichs> What about iii?
17:47:01 <zhipeng> so for triggers, is it necessory to know what exactly should be done for the other engines
17:47:17 <thinrichs> No.  i, ii, iii were applications of triggers.
17:47:58 <thinrichs> I see that the spec is pretty light in terms of spelling out details.
17:47:59 <zhipeng> ok, was confused
17:48:13 <thinrichs> Seems like it needs to be resubmitted so that we can flush out details via Gerrit.
17:48:25 <zhipeng> that would be a good idea
17:48:28 <thinrichs> I'll volunteer to resubmit and try to spell things out more clearly.
17:48:45 <zhipeng> thx man
17:48:47 <thinrichs> zhipeng: I'll try to do that today.  Think you'll have time this week for a couple of rounds of comments.
17:48:48 <thinrichs> ?
17:48:55 <zhipeng> definitely
17:49:24 <thinrichs> I'll add you as a reviewer, so you should get an email whenever something changes.
17:49:45 <zhipeng> i think i'm already one
17:49:50 <thinrichs> Great.
17:49:59 <zhipeng> :)
17:50:02 <thinrichs> Anyone else assigned a blueprint that they have questions about?
17:50:08 <thinrichs> (Time check: 10 min left)
17:50:48 <thinrichs> I understand we have a couple of new people today.
17:51:21 <thinrichs> If this is your first time at the meeting, could you say a few words about yourself?
17:52:04 <cloudtoa_> Networkn3rd: ping
17:52:17 <Networkn3rd> Sure! - My name is Ed Henry, I work at Plexxi with Derick and Andrew whom should also be on the call.
17:52:36 <regana> I'm new...my name is Andrew Regan. Just switched jobs from EMC --> Plexxi. Working with cloudtoad on DSE.
17:52:39 <Networkn3rd> Call...Ha, meeting*
17:52:56 <regana> and Ed
17:52:57 <thinrichs> Networkn3rd: :)
17:53:19 <arosen1> cool, well welcome :)
17:53:40 <thinrichs> Welcome!  Glad to have you.
17:53:43 <Networkn3rd> Thanks :)
17:54:08 <samta> Hi. this is samta from Montavista..One query..during holidays patch review will happen?
17:54:40 <thinrichs> samta: I'd imagine there will be many fewer reviews done over the holidays.  I wouldn't plan on getting anything merged.
17:54:45 <arosen1> We'll try though I expect things to slow down a little.
17:54:48 <thinrichs> (It could always happen—I just wouldn't plan on it.)
17:54:58 <thinrichs> That reminds me.
17:55:05 <thinrichs> We need more people to review code.
17:55:12 <sarob_> indeed
17:55:22 <thinrichs> I think I saw stackalytics say that over 90% of reviews are done by VMware folks.
17:55:44 <thinrichs> We could always institute a rule that before you get a review, you have to do a review for someone else.
17:55:49 <thinrichs> But I'd rather not do that.
17:56:18 <sarob> reviews should be a big part of how we communicate as a team
17:56:45 <thinrichs> I'd suggest finding patches in a piece of the code you're familiar with and trying to understand the code and what the change is doing.
17:56:56 <thinrichs> Style suggestions are a fine way to start...
17:57:23 <thinrichs> but I'd really like us to have at least one deeper kind of comment for each review, e.g. questioning the design/algorithm/etc.
17:57:44 <thinrichs> 3 minutes left.
17:57:50 <thinrichs> Open discussion.
17:57:53 <thinrichs> #topic open discussion
17:58:26 <sarob> it would be great if each person worked towards a review a day
17:58:33 <sarob> that would make a huge difference
17:59:14 <thinrichs> Sorry we didn't get status updates from everyone.
17:59:42 <thinrichs> Seems there's enough people and work that we don't have enough time.
17:59:46 <thinrichs> A good problem to have, for sure.  :)
17:59:55 <sarob> :0
18:00:04 <jwy> just want to quickly mention that some changes were merged in congress-specs to make unit test errors clearer
18:00:18 <jwy> please run tox there also before submitting specs for review to make sure all the sections are included, etc.
18:00:37 <arosen1> jwy: yea we need to figure out why jenkins isn't checking those correctly.
18:00:45 <thinrichs> Good to know.
18:00:48 <thinrichs> We're out of time.
18:00:52 <sarob> cheers!
18:00:57 <arosen1> later
18:01:06 <thinrichs> Thanks all!  I'll be in #congress for 30 min if anyone wants to continue the discussion.
18:01:09 <thinrichs> And there's always the ML.
18:01:11 <thinrichs> #endmeeting