00:01:01 <thinrichs> #startmeeting CongressTeamMeeting
00:01:02 <openstack> Meeting started Thu Feb 25 00:01:01 2016 UTC and is due to finish in 60 minutes.  The chair is thinrichs. Information about MeetBot at http://wiki.debian.org/MeetBot.
00:01:03 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
00:01:05 <openstack> The meeting name has been set to 'congressteammeeting'
00:01:57 <thinrichs> ekcs, ramineni, pballand: courtesy ping
00:02:47 <thinrichs> Just a reminder that next week we release the python-client.
00:03:04 <thinrichs> Quick discussion about that and then status updates
00:03:06 <ramineni1> hi
00:03:12 <thinrichs> ramineni1: hi
00:03:30 <thinrichs> Has anyone tested our python client recently?
00:04:51 <ekcs> I haven't.
00:05:08 <thinrichs> ramineni1: do our tempest tests run the python client?
00:05:53 <ramineni1> thinrichs: it does i guess
00:06:06 * ramineni1 checks qucikly
00:06:38 <thinrichs> If not, I'd say we should do some manual testing before next week.
00:06:47 <masahito> no. and I think I should update it for push driver.
00:08:04 <thinrichs> masahito: do you think the push driver is sufficiently well-tested to expose it through the client?
00:08:54 <thinrichs> We could always delay the inclusion of the push driver and release a new version of the client in the middle of the cycle
00:09:47 <masahito> thinrichs: just question. Has python-congressclient a CLI only for well-tested API?
00:09:54 <ramineni1> thinrichs: it does https://github.com/openstack/congress/blob/master/congress_tempest_tests/services/policy/policy_client.py
00:10:49 <thinrichs> masahito: maybe not.
00:11:10 <thinrichs> masahito: I haven't done ANY manual testing recently, which makes me more nervous about expanding what we expose through the client.
00:11:46 <thinrichs> ramineni1: that's separate code from the python client (it's basically a copy)
00:12:07 <masahito> thinrichs: I understand your concern.
00:12:09 <thinrichs> ramineni1: we used to use the python client code directly, but that didn't work with the devstack/tempest plugin
00:12:30 <ramineni1> oh, ya, it tests api basically
00:12:45 <thinrichs> masahito: if you wanted to add the push driver to the client and do some manual testing, that would work for me.
00:13:21 <thinrichs> masahito: maybe you could test the existing CLI functionality first to make sure there's nothing broken
00:14:27 <thinrichs> In fact, if we have confidence that the basic system is working, we could branch soon, release Mitaka, and get back to the distributed architecture.
00:15:07 <masahito> thinrichs: yeah. I should existing CLI test first. and then I try to add it.
00:15:19 <thinrichs> masahito: sounds good.
00:16:02 <thinrichs> The thing we want to do is make sure that once we do testing, we don't get a bunch of changes afterward that would make us retest.
00:16:14 <thinrichs> So we want master to be fairly stable during testing.
00:17:24 <thinrichs> What we don't want is to do a bunch of testing, find a bug, and realize that we need to patch it completely differently on master and on mitaka because a bunch of changes went into master since the branch.
00:17:57 <thinrichs> We want to branch at a point where we don't expect a bunch of changes to come in immediately afterward.
00:18:34 <thinrichs> So let's branch once we expect there to be a lull in development on master.
00:19:33 <thinrichs> As long as the client is functioning as we expect, we release it next week.  Then we pick a slow period to branch master.
00:19:39 <thinrichs> Yes?  (Sorry for the rambling.)
00:19:58 <ekcs> this is for full mitaka release right?
00:20:06 <ekcs> makes sense.
00:20:49 <thinrichs> Yes the branch is for the mitaka release
00:21:01 <ramineni1> sounds good
00:21:04 <masahito> it makes sense.
00:22:02 <thinrichs> #action masahito will update the client to include push functionality and test basic functionality
00:22:26 <thinrichs> masahito: let us know if you need help with the testing or run into anything that needs fixing before next week's client release.
00:23:03 <thinrichs> #topic status
00:23:11 <masahito> thinrichs: I'll report a bug in launchpad if I find something strange.
00:23:21 <masahito> thinrichs: to easy share.
00:23:47 <thinrichs> masahito: great.  And then ping us if it needs to be fixed before next week.
00:24:26 <thinrichs> ramineni1: want to give a status update?
00:25:02 <ramineni1> thinrichs: worked on datasource api model on top of your change
00:25:45 <ramineni1> thinrichs: migration test_congress is pending , i think i need your harness change for doing that
00:25:58 <thinrichs> How does it look?  Does the add/delete datasource functionality in the dse_node do what we need?
00:26:19 <thinrichs> ramineni1: yeah—I think the harness work and the test_congress work are tied together.
00:26:45 <thinrichs> ramineni1: Originally I thought the harness couldn't be tested at all until we did the datasource_model.
00:27:05 <ramineni1> thinrichs: ya ,
00:27:09 <thinrichs> but then I realized we could test the harness by having the API talk to the policy engine (create policy, create rules, etc.)
00:27:48 <thinrichs> ramineni1: I haven't gotten back to it.  But if you want, you could take over that harness work
00:27:51 <ramineni1> thinrichs: for add datasource policy related changes i included in the api itself , as suggested by you in TODO
00:28:24 <thinrichs> ramineni1: could you describe more for ekcs and masahito?  I think that's something worth discussing.
00:28:38 <ramineni1> thinrichs: ok your change seems to be in pretty good shape right
00:29:16 <thinrichs> ramineni1: I think my create/delete datasource patch is good.  The harness patch might be okay, but it's pretty much untested.
00:29:29 <thinrichs> (The create/delete datasource will be merged today.)
00:29:34 <ramineni1> thinrichs: ok
00:30:31 <thinrichs> ramineni1: could you describe for ekcs and masahito the logic that you put into the datasource API that ties the datasources and poicy engine together?
00:30:43 <ramineni1> thinrichs: ya,ok
00:30:49 <thinrichs> #link https://review.openstack.org/#/c/280509/
00:33:07 <ramineni1> now, after adding datasource to db , we do couple of things like creating a policy and setting the schema for it. Now, we have moved the policy engine part seperately to API instead of adding it in dse_node
00:34:14 <ramineni1> the change is part of above review, please do review and let me know any commnts
00:35:24 <thinrichs> It used to be that several things happened all at once: (i) create datasource on DSE, (ii) record datasource in DB, ...
00:35:40 <thinrichs> (iii) create policy for that datasource in policy engine, (iv) set schema for that policy
00:36:03 <thinrichs> Since everything was running in a single process, this was all fine.
00:36:41 <thinrichs> The tricky bit was making sure that if any of those failed that all of them failed.
00:36:54 <thinrichs> That tricky bit is now quite a bit harder since all the components are distributed.
00:37:27 <thinrichs> I think the right first thing to do is in your change ramineni1: put all of that logic into the API.
00:38:00 <thinrichs> Longer term we need to figure out if we can avoid all that synchronization between different distributed components in the system.
00:38:35 <ekcs> Got it.
00:39:02 <thinrichs> Definitely post-Mitaka since for Mitaka we'll only support running in a single process.
00:39:48 <thinrichs> ramineni1: I'll try to review your code after the meeting
00:40:05 <thinrichs> ekcs: want to give us an update?
00:40:28 <ekcs> sure.
00:40:29 <masahito> ramineni1: ok, I'll check it.
00:41:05 <ekcs> On this migrating subhandler, (https://bugs.launchpad.net/congress/+bug/1544126) This patch: https://review.openstack.org/#/c/281586/ uses heartbeat to give each node and service knowledge of which of its tables have subscribers.
00:41:05 <openstack> Launchpad bug 1544126 in congress "Migrate subhandler to DSE2" [High,In progress] - Assigned to Eric K (ekcs)
00:41:20 <ekcs> Last step to finish the task is integrate with the (un)subhandler hooks for policy_engine. Should be done tmr or Friday.
00:41:42 <ekcs> Once that's in, migrating vm_placement should be straightforward. https://bugs.launchpad.net/congress/+bug/1541665
00:41:42 <openstack> Launchpad bug 1541665 in congress "Migrate the vm_placement policy engine to DSE2" [High,New] - Assigned to Eric K (ekcs)
00:42:09 <ekcs> On the delta-update task (https://bugs.launchpad.net/congress/+bug/1544139), I'm not sure yet how best to proceed. It's related to the update-sequencing task (https://blueprints.launchpad.net/congress/+spec/dist-datasources-on-bus) and there are some subtle decisions which would take a few tries to get right in distributed mode.  Different ways to proceed: 1. First get the delta-update done, delay update sequencing until after
00:42:09 <openstack> Launchpad bug 1544139 in congress "Migrate prepush processor to dse2" [High,New] - Assigned to Eric K (ekcs)
00:42:39 <ekcs> Different ways to proceed:
00:42:44 <ekcs> 1. First get the delta-update done, delay update sequencing until after mitaka. In single process case, sequencing shouldn't be an issue. Drawback is that it's extra work compared to doing the two together. And I'm not positive that sequencing isn't needed for single process.
00:42:48 <ekcs> 2. Delay everything until after mitaka, but then the mitaka release would send only full tables all the time. Could affect deployability depending on use case.
00:42:52 <ekcs> 3. Get both delta-update and update-sequencing in before mitaka. Could affect testing and release effort.
00:43:56 <thinrichs> It's not clear to me at this point whether we'll switch over to the new DSE by Mitaka.
00:44:51 <thinrichs> If we don't switch over by Mitaka, all 3 are the same, right?
00:45:20 <ekcs> I think so.
00:46:10 <thinrichs> Nevermind.  Next week is Mitaka3.  We're not ready to switch over to DSE2.
00:47:01 <thinrichs> So for Mitaka, we'll ship with the original DSE.
00:47:38 <thinrichs> This week and early next let's get our existing changes merged so we can branch middle of next week.
00:47:50 <thinrichs> We'll do all our testing and release with distributed_arch=False
00:48:14 <thinrichs> Once we branch, we should all spend some time hammering to make sure we haven't introduced problems with all the code changes
00:48:18 <thinrichs> we introduced recently.
00:48:32 <thinrichs> By hammering I mean testing
00:48:44 <thinrichs> spinning up a devstack, and trying to break us
00:49:39 <thinrichs> (Earlier I was thinking that the client library release date was several weeks before the server release date.  That was for the other libraries.)
00:49:50 <ekcs> Got it. and hold off on merging new dse2 changes during testing?
00:50:32 <thinrichs> Let's get the couple of changes that are ready to go in merged by this week or maybe Mon/Tue of next week.
00:51:07 <ekcs> got it.
00:51:10 <thinrichs> Then hold off on merging til we're pretty sure everything is mostly solid.
00:51:41 <thinrichs> Before we run out of time, we need to hear from masahito.
00:51:44 <thinrichs> masahito: status update?
00:52:35 <masahito> I don't have enough time to review codes, so I just push it for launch Congress with distributed_arch=True
00:52:46 <masahito> https://review.openstack.org/#/c/280793/
00:52:55 <masahito> I did some minor fix.
00:53:17 <masahito> That's from my side
00:53:59 <masahito> sorry, I couldn't join last 2 meeting because I was out of town.
00:54:23 <thinrichs> masahito: Are you saying you think that patch is ready to merge?
00:54:36 <thinrichs> masahito: how risky do you think that change is?
00:54:53 <thinrichs> masahito: I ask only b/c it is touching a part of the codebase we don't have many tests for.
00:54:53 <masahito> I think it's not ready.
00:55:15 <masahito> not ready for merge.
00:55:26 <thinrichs> Ok.  So hold off on that until after Mitaka?
00:55:59 <thinrichs> masahito: Shall we wait til after Mitaka to merge it?
00:56:00 <masahito> because it depends on the patch about migrating harness.
00:56:18 <masahito> thinrichs: of course, no problem :)
00:56:45 <thinrichs> Maybe we should wait on merging the new harness too.  Again, few tests there.
00:58:37 <thinrichs> Anything else in our remaining 2 minutes?
00:59:24 <thinrichs> I think we're nearing an end-to-end migration to DSE2, which is really exciting.
01:00:06 <ekcs> yup!
01:00:33 <thinrichs> I'm looking forward to getting the release done so we can finish it off!
01:00:36 <thinrichs> Thanks all!
01:00:58 <thinrichs> #endmeeting