17:01:01 <LouisF> #startmeeting service_chaining 17:01:01 <openstack> Meeting started Thu Sep 1 17:01:01 2016 UTC and is due to finish in 60 minutes. The chair is LouisF. Information about MeetBot at http://wiki.debian.org/MeetBot. 17:01:02 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 17:01:05 <openstack> The meeting name has been set to 'service_chaining' 17:01:11 <LouisF> hi all 17:01:17 <pcarver> hi LouisF 17:01:25 <LouisF> hi paul 17:01:57 <LouisF> hi doonhammer 17:02:08 <doonhammer> Hi Louis 17:02:55 <yamahata> hello 17:03:03 <LouisF> hi yamahata 17:04:18 <LouisF> yamahata: can we discuss your precommit/postcommit needed for the SFC-ODL driver 17:04:21 <LouisF> https://review.openstack.org/#/c/363893 17:04:29 <yamahata> LouisF: yes. 17:04:46 <yamahata> The patch status is to show the idea in order to reach consensus. 17:04:48 <LouisF> yamahata: can you explain why it is needed? 17:05:09 <yamahata> ODL driver needs a hook in during db transaction and a hook after that. 17:05:42 <yamahata> Basically ODL doesn't send requests to ODL directly on behalf on crud_xxx 17:05:53 <yamahata> but it creates opereration log in DB 17:06:09 <yamahata> and later background thread sync it by sending actual request to ODL. 17:06:41 <yamahata> The important point is that the journal logging should happen in the same transaction that manipulates neutron resource. 17:07:14 <yamahata> Does it explain? or more details are needed? 17:07:50 <yamahata> The motivation is to avoid out-of-sync state between neutron db and ODL controller. 17:08:02 <LouisF> why a background thread? 17:08:34 <yamahata> Actually with postcommit, we sends requests to ODL as optimization. 17:09:47 <LouisF> why is a background thread needed? 17:10:14 <yamahata> We can't assume that postcommit can always handle operation log. 17:10:21 <yamahata> Neutron can crash at anytime. 17:10:36 <yamahata> for example neutron can crash after db transaction, but before postcommit. 17:11:04 <yamahata> In that case, background thread monitors journal db and odl state, it processes operation log. 17:11:25 <yamahata> In normal cases, background thread only processes maintenance. 17:12:51 <LouisF> so the odl driver design is to ensure sync between the neutron db and odl is neutron crashes? 17:13:08 <shasha_tavil> :q:aQ 17:13:14 <LouisF> if neutron crashes 17:13:15 <yamahata> Half yes. 17:13:28 <yamahata> Other half goal is to keep the order of operation. 17:13:36 <yamahata> For example there are two update. 17:13:47 <yamahata> In that case, we have to send those two update in order. 17:14:06 <yamahata> Without operation log, the order of sending request to ODL can be reversed. 17:14:18 <yamahata> with operation log, we can maintain the order. 17:14:43 <LouisF> how would the order of requests sent to the odl controller be reversed? 17:14:57 <yamahata> Neutron server is multi threaded with eventlet. 17:15:07 <yamahata> the thread switch can happend at any io. 17:15:34 <yamahata> When neutorn tries to send request to ODL, thread switch can happend. 17:15:39 <yamahata> happen 17:16:00 <yamahata> Then, the second request can be sent to ODL first. 17:16:55 <yamahata> With operation log, we add sequence number. such reorder is checked. 17:17:06 <LouisF> yamahata: is this request reversal something that occurs frequently? 17:17:07 <yamahata> If such reorder is found, pass it to background thread 17:17:18 <yamahata> Not so often. 17:17:22 <yamahata> with single neutron server. 17:17:33 <yamahata> But with multiple neutron server deployment, it can easily happen. 17:18:09 <yamahata> But it happends, it's quite difficult to fix. 17:18:25 <yamahata> but clear all state in ODL, then push all the state toODL. 17:18:46 <yamahata> We don't want to do such, blow up and populate from scratch. 17:19:30 <yamahata> I think this problem is quite common. e.g. OVN, ONO have similar issue, I believe. 17:19:40 <yamahata> ONOS 17:19:54 <LouisF> for others on the call - Isaku had requested we discuss this in this meeting 17:20:24 <yamahata> For ODL support, driver interface needs to be enhanced. 17:20:34 <yamahata> The patch is at https://review.openstack.org/#/c/363893 17:20:36 <LouisF> please chime if you have questions 17:21:05 <Murali_> LouisF: I have question on cross subnet check 17:21:53 <Murali_> LouisF: I seeing the error while creating the port chain 17:21:59 <LouisF> Murali_: can we take that a bit later i'd like to finish isaku's discussion 17:22:22 <Murali_> LouisF:Sure Thank you 17:22:57 <LouisF> yamahata: you are propsing the pre/postcommit for each crud 17:23:05 <yamahata> Yes, exactly. 17:23:23 <pcarver> I don't have any experience troubleshooting race conditions in multi-threaded systems that need order to be synchronized, but everything yamahata wrote seems reasonable 17:23:53 <yamahata> the patch keeps backward compatibility to the existing driver 17:25:15 <LouisF> yamahata: thats good 17:25:45 <LouisF> yamahata: can you add unit tests 17:25:52 <yamahata> Yes, will do. 17:26:01 <yamahata> I wanted to get consensus before going futher. 17:26:27 <LouisF> yamahata: +1 17:26:44 <yamahata> Seems like reached consensus. 17:26:53 <yamahata> I'm done. 17:27:16 <LouisF> i would like vikram choudry to comment as this may affect the onos driver 17:27:41 <LouisF> i will add him as reviewer 17:28:22 <yamahata> makes sense. that's quite fair. 17:28:24 <LouisF> he is not on the call 17:28:54 <LouisF> i want to make sure that the existing ovs and onos drivers work ok 17:29:14 <LouisF> with the changes for pre/postcommit 17:29:53 <LouisF> yamahata: if this is done how long before the odl driver will be complete? 17:30:27 <yamahata> Not sure. It's up to Anil. My call would be 1 week or so. 17:31:14 <fsunaval> ... 17:31:15 <LouisF> this patch https://review.openstack.org/#/c/337948 17:31:54 <LouisF> yamahata: i'd like to progress this as fast as possible 17:32:11 <yamahata> I see. Do you have any deadline or something? 17:32:29 <LouisF> yamahata: i'd like this work complete for newton 17:32:51 <yamahata> Wow. gotcha. 17:33:10 <LouisF> yamahata: do you see that as a problem? 17:33:31 <yamahata> I can take care of driver interface part. 17:33:38 <yamahata> ODL driver patch is out of my control. 17:33:53 <yamahata> We need to nudge anil. 17:34:13 <LouisF> yamahata: great - i will contact anil on the driver itself 17:34:38 <LouisF> yamahata: could you also speak to anil 17:34:43 <yamahata> Sure, off course. 17:35:34 <LouisF> question of the journal logging - i am not familiar with that can you explain 17:36:27 <yamahata> For detailed design, there is a wiki page 17:36:37 <yamahata> https://wiki.opendaylight.org/view/NeutronNorthbound:NeutronDriverOverhaul 17:36:49 <yamahata> Basically there is a table which is called opendaylightjournal 17:37:12 <yamahata> it contains seqnum(autoincrement), uuid, operation, data(json to be sent) 17:37:22 <yamahata> with other maintenance columns. 17:37:30 <yamahata> like timestamp 17:37:43 <LouisF> yamahata: thanks i will look at that 17:37:50 <yamahata> with timestamp and seqnum, the entries are ordered in the order that operation is done. 17:38:23 <fsunaval_> ... 17:38:59 <LouisF> yamahata: so this is an issue in a larger context of all neutron requests and odl 17:39:08 <yamahata> Yes. 17:39:26 <yamahata> It applies to all drivers, plugins. ML2, l3 etc... 17:39:41 <LouisF> the other neutron crud requests all have pre/postcommit support? 17:40:01 <yamahata> Now, I'm requesting it. 17:40:06 <yamahata> Right now ML2 and L3. 17:40:18 <yamahata> L3 case it's plugin, so it's okay. 17:40:31 <yamahata> We are starting to cover other apis. 17:40:52 <LouisF> yamahata: we definitely want to help with this 17:42:03 <yamahata> thanks, it definitively helps. 17:42:45 <LouisF> yamahata: thanks for putting that patch together 17:42:57 <LouisF> all - please review 17:43:27 <LouisF> ok moving on 17:43:38 <LouisF> Murali_: you had a question 17:44:09 <Murali_> LouisF: Shall we discuss off-line after the meeting? 17:44:41 <LouisF> Murali_: ok thats fine 17:45:51 <LouisF> i have to get to another meeting so i am going to cut this short today 17:46:55 <LouisF> bye all 17:47:06 <LouisF> #endmeeting