00:03:11 #startmeeting CongressTeamMeeting 00:03:12 Meeting started Thu Jan 14 00:03:11 2016 UTC and is due to finish in 60 minutes. The chair is thinrichs. Information about MeetBot at http://wiki.debian.org/MeetBot. 00:03:13 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 00:03:15 hi 00:03:16 The meeting name has been set to 'congressteammeeting' 00:03:17 hi 00:03:24 hi 00:03:30 pballand, ekcs, ramineni: hi! 00:03:54 masahito: you here? 00:04:05 yes 00:04:23 Here's my agenda for today… 00:04:39 1. distributed arch 00:04:42 2. mid-cycle meet-up 00:04:49 3. Austin 00:04:57 4. mitaka-2 next week 00:05:04 5. push driver 00:05:09 Anything else? 00:05:43 6. stable branches 00:06:15 Let's get started then. 00:06:18 #topic distributed arch 00:06:28 pballand: I saw you pushing code to review. How's it going? 00:06:55 yeah - I abandoned the old set and pushed a new one: https://review.openstack.org/#/q/status:open+project:openstack/congress+branch:master+topic:dse2 00:07:03 here’s: where it’s at... 00:07:36 the code is completely standalone still, pending enough functionality (I’ll get to that) to merge existing services 00:07:57 it supports: launching multiple DseNodes, that discover eachother and share status 00:08:17 DseNodes can expose RPCs and invoke them on other nodes 00:08:38 each node runs zero or more services, which also support exposing and invoking RPCs 00:09:14 the big missing piece is the DseService base class support for table subscription - I’m working on that right now 00:09:37 the pieces pushed so far should give a good indication of where I am going, however, and should be ready for review 00:10:01 that’s the summary - any questions? 00:10:08 That's tremendous progress! 00:11:40 thanks - I’ll try to respond to review comments quickly, so we can iterate on the design 00:11:43 I'm trying to figure out what we can do with the code that's already there. 00:12:08 We could (i) hook up the API models to the datasources and policy engines… 00:13:03 yes, and we could link the main driver to launch DseNode with a command-line option to run those services that are hooked up 00:13:06 I guess the only other thing we use the bus for right now having the policy engine/datasources pub/sub tables, which we can't do. 00:13:26 thinrichs: right - that is the big missing piece, sorry :-( 00:13:43 on the plus side, all of that relies on RPC, which has quite a bit of test coverage 00:13:53 pballand: not calling out problems—just trying to enumerate the main pieces of functionality that we use the bus for. 00:14:26 Is it fair to say that we have 2 uses for the bus: (1) RPC from API and (2) table pub/sub for everything else? 00:14:45 that, and policy delegation 00:14:55 policy delegation is rpc, right? 00:15:04 action execution is also rpc 00:15:25 I guess so, that’s fair 00:15:37 How's the unit test framework looking? 00:16:18 if anyone is looking for something to do, integrating the dse2 unit tests with the framework would be very helpful - it’s not something I’ve put _any_ time in 00:16:23 on 00:17:50 pballand: how's the test coverage looking (independent if those tests are running in the current framework). 00:18:42 I haven’t run any coverage checks, but there are tests for all of the main methods 00:19:00 mostly functional tests (little/nothing mocked) rather than unit tests 00:19:20 https://review.openstack.org/#/c/267228/2/congress/dse2/test_dse_node.py 00:19:36 for example 00:20:31 All right, well that's some good background I'll use when I go to review. 00:20:48 Any other questions/comments on the distributed arch? 00:21:30 that test is an example of simple RPC calls, and the DseControlBus extends DataService in non-trivial ways 00:23:02 thinrichs: none from me. 00:23:06 I'll take a look and make a review pass either tonight or tomorrow hopefully. 00:23:45 Let's all get pballand reviews as soon as possible, so we all have a chance to pull down the code before the mid-cycle meetup. 00:23:52 #action Everyone will review the DseNode code 00:24:01 #topic Push driver 00:24:07 masahito: how is the push driver going? 00:25:19 I push patches which receive data with existing API server. 00:25:35 https://review.openstack.org/#/c/256303/ 00:25:40 https://review.openstack.org/#/c/266347/ 00:26:33 I'm wondering if I remove http_driver.py or not in next patch set. 00:27:22 because if Congress don't have it there is not reference implementation for other developers. 00:27:35 That's my update. 00:27:50 The http_driver.py shows how to set up a datasource to accept http requests for pushing data. Is that right? 00:28:55 right. and shows what developers have to implement like get_datasource_info() 00:29:28 Gerrit saves everything forever… 00:29:46 so what I've done in the past is abandon the change that has the code that might be useful later but that we decided to not use immediately... 00:30:05 and then create a new changeset that has just the code that gets merged. 00:30:31 That way, we always have the alternate code available, but we don't pollute master with code that isn't currently being used. 00:30:46 Just my idea is keeping http_driver.py there or add a new driver that works for my OPNFV use-case. 00:31:19 I understand your concern. 00:31:37 Does your OPNFV use case not work if the http push is routed through the API model? 00:32:34 no. it works when http push is routed via the API. 00:32:52 What if the reference code is moved into an rst? Would that be a good solution? 00:33:36 but the table the driver should have would be different among the user. 00:34:14 masahito: not sure I understand. Can you give us some more context? 00:36:17 example are following. 00:38:33 user A want to use a table which has schema, event_id, host and status, but user B want to use a table which has 3 schema, name, status and disk.∫ 00:39:03 In this case, 00:41:02 I don't think it's not good to extend the driver everytime when the number of user increases and they use different schema. 00:42:40 So you're saying that if we have 2 users who want to push different schema, we need 2 different datasources in congress that define the translators for that schema. 00:43:01 And then if we have N different schema, we'd need N different datasources in congress. 00:43:11 right. 00:43:32 If we could tell people that they are only allowed to push tables, then we don't really need a translator. 00:43:45 And 1 datasource would be enough for everyone. 00:44:08 The only time we need translators is when people are pushing data that is not a list of tuples of strings/numbers. 00:44:59 So maybe we would want a TablePushDriver that anyone can use who wants to push relational data in any schema. 00:45:48 I suppose we could also create a JsonPushDriver that people can use who want to push json data, and recursively walk that data to turn it into tables. 00:46:19 I guess the downside to that idea is that the schema isn't fixed in the code. The schema is dynamically computed at runtime. 00:46:47 I think the policy engine should be flexible enough to handle that now. 00:47:18 question: is this a problem that arises only with push not poll? 00:47:36 masahito: does that make sense? (We're running short on time, so maybe we should continue this discussion offline.) 00:47:49 yes. 00:47:52 my idea for it is we implement create table API for definition of the driver. 00:47:52 ekcs: right now all the datasources have static schemas. 00:48:15 masahito: that makes sense 00:48:16 I check if it works in our use-case. 00:49:08 masahito: if not, we can always compute the schema dynamically based on the data that has been pushed. (The schema is being set dynamically in either case.) 00:49:21 We're short on time, so let's move on. 00:49:43 #topic stable branches 00:50:07 ramineni: I saw your change for project-config finally got merged. Right? 00:50:33 thinrichs: yes :) , so now stable branches run only congress tests without plugin mechanism 00:50:48 https://review.openstack.org/#/c/266801/ 00:50:51 ramineni: congrats! That one took a while. 00:50:56 this patch should turn the gate green 00:51:22 ramineni: great! I'll take a look once we finish up the meeting. 00:51:26 thinrichs: yep :D 00:51:38 Hopefully then we can get the backlog of stable/liberty fixes in. 00:52:04 ramineni: Do you remember if there's some stable/kilo work to do? 00:52:04 thinrichs: tempet tests are still disabled though , i will enable in one patch after it merges 00:52:31 ramineni: are you on the stable-maint team? (Seems like you should be.) 00:52:32 thinrichs: no, stable kilo we have disabled this job i guess right? 00:53:01 Maybe so…I'm losing track. 00:53:06 thinrichs: ya, think so 00:53:11 I'll try to look into it. 00:53:38 thinrichs: i think its disabled , but do we want to support stable/kilo also? 00:54:09 ramineni: I don't remember if it was working at all. I seem to think not. Let's focus on stable/liberty for now. 00:54:32 ramineni: let us know if there's any help you need getting stable/liberty back on track. 00:54:47 thinrichs: sure 00:54:55 If you're in stable/maint, you should be able to +2 things and merge them. 00:55:24 thinrichs: ok 00:55:27 5 minutes left. 00:55:37 #topic open discussion 00:55:44 ekcs: anything to report? 00:56:05 I'm working on this bp for pub-sub communication between policy engine and datasources: https://blueprints.launchpad.net/congress/+spec/dist-datasources-on-bus 00:56:05 Almost ready to push reviewable WIP. 00:56:10 anyone else have anything? (We can interleave the discussions to finish on time). 00:56:55 ekcs: nice! Having that in the pipeline will be good once we start getting pballand's changes integrated. 00:57:16 Couple of reminders… 00:57:37 1. Calll for speakers at Austin closes Feb 3 00:57:56 2. Our mid-cycle meetup is Jan 26-28. 00:58:01 3. Mitaka-2 is next week 00:58:15 Think that's it. 00:58:18 masahito: also, on this patch (https://review.openstack.org/#/c/263464/), I’m not sure I understand your comment. Could we talk about it either here or later? 00:58:36 I'd suggest taking that to #congress, since we've got 2 min left. 00:59:09 ekcs: in #congrss later :) 00:59:11 thinrichs: k 00:59:24 That does it for this week. Thanks all! 00:59:35 Let's make sure to get those reviews in. 00:59:52 #endmeeting