17:01:01 <LouisF> #startmeeting service_chaining
17:01:01 <openstack> Meeting started Thu Sep  1 17:01:01 2016 UTC and is due to finish in 60 minutes.  The chair is LouisF. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:01:02 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:01:05 <openstack> The meeting name has been set to 'service_chaining'
17:01:11 <LouisF> hi all
17:01:17 <pcarver> hi LouisF
17:01:25 <LouisF> hi paul
17:01:57 <LouisF> hi doonhammer
17:02:08 <doonhammer> Hi Louis
17:02:55 <yamahata> hello
17:03:03 <LouisF> hi yamahata
17:04:18 <LouisF> yamahata: can we discuss your precommit/postcommit needed for the SFC-ODL driver
17:04:21 <LouisF> https://review.openstack.org/#/c/363893
17:04:29 <yamahata> LouisF: yes.
17:04:46 <yamahata> The patch status is to show the idea in order to reach consensus.
17:04:48 <LouisF> yamahata: can you explain why it is needed?
17:05:09 <yamahata> ODL driver needs a hook in during db transaction and a hook after that.
17:05:42 <yamahata> Basically ODL doesn't send requests to ODL directly on behalf on crud_xxx
17:05:53 <yamahata> but it creates opereration log in DB
17:06:09 <yamahata> and later background thread sync it by sending actual request to ODL.
17:06:41 <yamahata> The important point is that the journal logging should happen in the same transaction that manipulates neutron resource.
17:07:14 <yamahata> Does it explain? or more details are needed?
17:07:50 <yamahata> The motivation is to avoid out-of-sync state between neutron db and ODL controller.
17:08:02 <LouisF> why a background thread?
17:08:34 <yamahata> Actually with postcommit, we sends requests to ODL as optimization.
17:09:47 <LouisF> why is a background thread needed?
17:10:14 <yamahata> We can't assume that postcommit can always handle operation log.
17:10:21 <yamahata> Neutron can crash at anytime.
17:10:36 <yamahata> for example neutron can crash after db transaction, but before postcommit.
17:11:04 <yamahata> In that case, background thread monitors journal db and odl state, it processes operation log.
17:11:25 <yamahata> In normal cases, background thread only processes maintenance.
17:12:51 <LouisF> so the odl driver design is to ensure sync between the neutron db and odl is neutron crashes?
17:13:08 <shasha_tavil> :q:aQ
17:13:14 <LouisF> if neutron crashes
17:13:15 <yamahata> Half yes.
17:13:28 <yamahata> Other half goal is to keep the order of operation.
17:13:36 <yamahata> For example there are two update.
17:13:47 <yamahata> In that case, we have to send those two update in order.
17:14:06 <yamahata> Without operation log, the order of sending request to ODL can be reversed.
17:14:18 <yamahata> with operation log, we can maintain the order.
17:14:43 <LouisF> how would the order of requests sent to the odl controller be reversed?
17:14:57 <yamahata> Neutron server is multi threaded with eventlet.
17:15:07 <yamahata> the thread switch can happend at any io.
17:15:34 <yamahata> When neutorn tries to send request to ODL, thread switch can happend.
17:15:39 <yamahata> happen
17:16:00 <yamahata> Then, the second request can be sent to ODL first.
17:16:55 <yamahata> With operation log, we add sequence number. such reorder is checked.
17:17:06 <LouisF> yamahata: is this request reversal something that occurs frequently?
17:17:07 <yamahata> If such reorder is found, pass it to background thread
17:17:18 <yamahata> Not so often.
17:17:22 <yamahata> with single neutron server.
17:17:33 <yamahata> But with multiple neutron server deployment, it can easily happen.
17:18:09 <yamahata> But it happends, it's quite difficult to fix.
17:18:25 <yamahata> but clear all state in ODL, then push all the state toODL.
17:18:46 <yamahata> We don't want to do such, blow up and populate from scratch.
17:19:30 <yamahata> I think this problem is quite common. e.g. OVN, ONO have similar issue, I believe.
17:19:40 <yamahata> ONOS
17:19:54 <LouisF> for others on the call - Isaku had requested we discuss this in this meeting
17:20:24 <yamahata> For ODL support, driver interface needs to be enhanced.
17:20:34 <yamahata> The patch is at https://review.openstack.org/#/c/363893
17:20:36 <LouisF> please chime if you have questions
17:21:05 <Murali_> LouisF: I have question on cross subnet check
17:21:53 <Murali_> LouisF: I seeing the error while creating the port chain
17:21:59 <LouisF> Murali_: can we take that a bit later i'd like to finish isaku's discussion
17:22:22 <Murali_> LouisF:Sure Thank you
17:22:57 <LouisF> yamahata: you are propsing the pre/postcommit for each crud
17:23:05 <yamahata> Yes, exactly.
17:23:23 <pcarver> I don't have any experience troubleshooting race conditions in multi-threaded systems that need order to be synchronized, but everything yamahata wrote seems reasonable
17:23:53 <yamahata> the patch keeps backward compatibility to the existing driver
17:25:15 <LouisF> yamahata: thats good
17:25:45 <LouisF> yamahata: can you add unit tests
17:25:52 <yamahata> Yes, will do.
17:26:01 <yamahata> I wanted to get consensus before going futher.
17:26:27 <LouisF> yamahata: +1
17:26:44 <yamahata> Seems like reached consensus.
17:26:53 <yamahata> I'm done.
17:27:16 <LouisF> i would like vikram choudry  to comment as this may affect the onos driver
17:27:41 <LouisF> i will add him as reviewer
17:28:22 <yamahata> makes sense. that's quite fair.
17:28:24 <LouisF> he is not on the call
17:28:54 <LouisF> i want to make sure that the existing ovs and onos drivers work ok
17:29:14 <LouisF> with the changes for pre/postcommit
17:29:53 <LouisF> yamahata: if this is done how long before the odl driver will be complete?
17:30:27 <yamahata> Not sure. It's up to Anil. My call would be 1 week or so.
17:31:14 <fsunaval> ...
17:31:15 <LouisF> this patch https://review.openstack.org/#/c/337948
17:31:54 <LouisF> yamahata: i'd like to progress this as fast as possible
17:32:11 <yamahata> I see. Do you have any deadline or something?
17:32:29 <LouisF> yamahata: i'd like this work complete for newton
17:32:51 <yamahata> Wow. gotcha.
17:33:10 <LouisF> yamahata: do you see that as a problem?
17:33:31 <yamahata> I can take care of driver interface part.
17:33:38 <yamahata> ODL driver patch is out of my control.
17:33:53 <yamahata> We need to nudge anil.
17:34:13 <LouisF> yamahata: great - i will contact anil on the driver itself
17:34:38 <LouisF> yamahata: could you also speak to anil
17:34:43 <yamahata> Sure, off course.
17:35:34 <LouisF> question of the journal logging - i am not familiar with that can you explain
17:36:27 <yamahata> For detailed design, there is a wiki page
17:36:37 <yamahata> https://wiki.opendaylight.org/view/NeutronNorthbound:NeutronDriverOverhaul
17:36:49 <yamahata> Basically there is a table which is called opendaylightjournal
17:37:12 <yamahata> it contains seqnum(autoincrement), uuid, operation, data(json to be sent)
17:37:22 <yamahata> with other maintenance columns.
17:37:30 <yamahata> like timestamp
17:37:43 <LouisF> yamahata: thanks i will look at that
17:37:50 <yamahata> with timestamp and seqnum, the entries are ordered in the order that operation is done.
17:38:23 <fsunaval_> ...
17:38:59 <LouisF> yamahata: so this is an issue in a larger context of all neutron requests and odl
17:39:08 <yamahata> Yes.
17:39:26 <yamahata> It applies to all drivers, plugins. ML2, l3 etc...
17:39:41 <LouisF> the other neutron crud requests all have pre/postcommit support?
17:40:01 <yamahata> Now, I'm requesting it.
17:40:06 <yamahata> Right now ML2 and L3.
17:40:18 <yamahata> L3 case it's plugin, so it's okay.
17:40:31 <yamahata> We are starting to cover other apis.
17:40:52 <LouisF> yamahata: we definitely want to help with this
17:42:03 <yamahata> thanks, it definitively helps.
17:42:45 <LouisF> yamahata: thanks for putting that patch together
17:42:57 <LouisF> all - please review
17:43:27 <LouisF> ok moving on
17:43:38 <LouisF> Murali_: you had a question
17:44:09 <Murali_> LouisF: Shall we discuss off-line after the meeting?
17:44:41 <LouisF> Murali_: ok thats fine
17:45:51 <LouisF> i have to get to another meeting so i am going to cut this short today
17:46:55 <LouisF> bye all
17:47:06 <LouisF> #endmeeting