17:00:04 <cathy_> #startmeeting service_chaining
17:00:06 <openstack> Meeting started Thu Dec  3 17:00:04 2015 UTC and is due to finish in 60 minutes.  The chair is cathy_. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:00:07 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:00:10 <openstack> The meeting name has been set to 'service_chaining'
17:00:19 <cathy_> hi everyone
17:00:23 <vikram> hi
17:00:28 <mohankumar> hi
17:00:32 <cathy_> vikram: welcom back!
17:00:38 <johnsom> o/
17:00:45 <vikram> cathy_: thanx
17:00:46 <cathy_> hi mohankumar johnsom
17:01:21 <pcarver> hi
17:01:23 <cathy_> hi s3wong
17:01:25 <cathy_> hi pcarver
17:01:31 <s3wong> hello
17:01:56 <cathy_> any topic you would like to discuss today?
17:02:13 <Prithiv> sfc manager
17:02:18 <igordcard> hi all
17:02:27 <vikram> cathy_: I have raised a bug for that
17:02:32 <cathy_> hi igordcard Prithiv
17:02:42 <vikram> https://bugs.launchpad.net/networking-sfc/+bug/1513368
17:02:42 <openstack> Launchpad bug 1513368 in networking-sfc "Introducing SFC manager" [Undecided,New] - Assigned to vikram.choudhary (vikschw)
17:03:05 <vikram> cathy_: It will be great to have it
17:03:14 <s3wong> what is SFC manager?
17:03:38 <Prithiv> hi cathy... also, about the issue that i have posted in networking-sfc page
17:03:43 <cathy_> vikram: yes, but we can deliver that in the second phase. Ok with you?
17:03:54 <Prithiv> https://review.openstack.org/#/c/233758/
17:04:06 <cathy_> Prithiv: let me take a look at the link
17:04:19 <vikram> cathy_: when is the second phase planned?
17:04:56 <prithivm> thank you cathy
17:05:07 <cathy_> vikram: could you change the bug name to something like "SFC OAM operation" since we already have a "SFC manager" terminology in our design to refer to management of the chain resources
17:05:24 <vikram> cathy_: Sure I will
17:05:31 <cathy_> vikram: thanks.
17:05:49 <vikram> cathy_: Done
17:05:54 <vikram> cathy_: when is the second phase planned?
17:06:15 <vikram> cathy_: Can you approve the bug so that we can start some initial write-up?
17:06:41 <mohankumar> s3wong , it monitor and helps debugging faults in SFC chain.
17:07:08 <vikram> s3wong: Just changed the bug heading
17:07:23 <cathy_> vikram: yes, I can do that
17:07:23 <s3wong> vikram, mohankumar: OK... taking a look now. Thanks
17:08:17 <cathy_> Prithiv: the link in the comment is not valid. Could you check that and update the links? But anyway, we will take a look at the issue off line and get back to you.
17:08:37 <vikram> cathy_: Thanks..
17:08:38 <Prithiv> sure. I will send you a detailed email about the error
17:08:54 <vikram> Cathy_: One help
17:09:12 <vikram> Cathy_: I feel most of the people are struggling testing our changes
17:09:12 <cathy_> Prithiv: thanks. please, we will try it to see the problem, investigate that , and get back to you
17:09:24 <Prithiv> thank you cathy... no problem
17:09:37 <cathy_> vikram: yes, we are doing the testing too.
17:09:41 <vikram> Cathy_: It would be better if can document the test procedure
17:09:51 <cathy_> here are some update of our recent work
17:09:53 <pcarver> vikram: +1
17:10:07 <vikram> Cathy_: test + installation process
17:10:18 <cathy_> I think most of the issues are related to the installation process.
17:10:32 <Prithiv> cathy: yes, i agree
17:10:44 <cathy_> we spent quite some time getting it installed.
17:10:46 <pcarver> vikram: I've been splitting my time between code review and trying to actually get the code working. I'm reluctant to spend too much time reviewing code when I can't get it to even install/work.
17:11:31 <pcarver> I'd spend more time reviewing code if I were spending less time just trying to get it installed/working.
17:11:38 <igordcard> pcarver, +1
17:11:50 <cathy_> Now we have got it installed on multiple machines and start doing conprehensive end-to-end SFC setup/update/deletion functionality testing
17:11:51 <vikram> pcarver: me either ;(
17:11:52 <Prithiv> pcarver: with you on that
17:11:57 <igordcard> also I can't seem to actually test it because the flow classifier is WIP
17:11:59 <LouisF> igordcard: your email indicated that you have been able to install using your procedures
17:12:22 <igordcard> LouisF, yes, it is running at the moment - however no flow classifier right?
17:12:26 <cathy_> so looks like the devstack patch has some glitch integrating with the code
17:12:46 <LouisF> igordcard: classifier should be there
17:12:48 <pcarver> igordcard's latest procedure did work for me as far as installation, but I'm not sure I've actually reached functionally working.
17:13:09 <vikram> cathy_: IMO, we must start getting the code in at the earliest
17:13:15 <Prithiv> Louis, i am not able to install with the latest commit of neutron (770624b13bf57fdab549bfb66ee9cc6781ae8d65) but am able to get the q-agt runnign with neutron older commit (commit 9b531b9e4eb6edfc1ad0d5ebfed6cd9b90e89422)
17:13:17 <LouisF> igordcard: are you still having a problem starting the agent?
17:13:22 <igordcard> LouisF, can you send some recommended API calls to deploy a simple working chain amongst 2 VMs, after the meeting?
17:13:38 <LouisF> igordcard: ok
17:13:43 <igordcard> LouisF, no, the agent is fine if I rollback neutron to some revisions earlier
17:14:16 <Prithiv> but with the newer revision of neutron, the agent doesnt start
17:14:23 <LouisF> igordcard: neutron or networking-sfc code?
17:14:24 <cathy_> vikram: after we get the end-to-end functionality testing cases all passed, I will approve the codes. But before we get to that stage, I would rather not get the codes merged.
17:14:31 <Prithiv> it looks for the module create_agent_config_map
17:14:33 <Prithiv> and fails
17:14:38 <igordcard> everything seems to work from the outside, with the ovs driver enabled, but when I create a chain, I can only see MPLS decapsulation flows - I don't seen any MPLS encapsulation flows
17:14:44 <igordcard> so traffic never gets steered
17:14:53 <mohankumar> Prithiv ,+1
17:14:55 <vikram> cathy_: When we can start the review?
17:15:02 <igordcard> LouisF, neutron code
17:15:04 <cathy_> I understand that once we have the first patches of code in, the painful dependency issues are gone. So we are working hard everyday on the testing and bug fixes
17:15:06 <vikram> cathy_: Is the patches ready for review?
17:15:23 <igordcard> LouisF, recent neutron commits have made the error create_agent_config_map appear
17:15:26 <cathy_> yes, they have been open for review for over one month.
17:15:49 <LouisF> igordcard: so you are using master not stable/liberty?
17:15:57 <igordcard> LouisF, yes, master
17:16:10 <pcarver> Do we have a prioritized list of changes for merging?
17:16:31 <igordcard> LouisF, will stick to stable/liberty then, no reason for master in this scenario
17:16:32 <cathy_> vikram: I understand you have been on leave for several weeks.
17:16:35 <LouisF> igordcard: yes using master we will be affected by neutron commits
17:16:41 <pcarver> I mean, if we were to merge changes one at a time, do we know what order we would like to tackle them?
17:16:52 <cathy_> pcarver: yes, we have the dependency order
17:17:23 <vikram> cathy_: plz let the team know when the patches are ready for review..
17:17:33 <vikram> cathy_: we can help ;)
17:17:39 <igordcard> personally I was going to review everything at once after I had successfully deployed a chain, but so far haven't been able
17:17:53 <cathy_> pcarver: I think we will merge the patches that run on the Neutron server, then the patches running on the Compute node, then the patches of Horizon, CLI, Heat. make sense to you?
17:18:09 <igordcard> cathy_, about Horizon, where are the patches?
17:18:28 <Prithiv> the horizon patches by mohan seems to work when i tried
17:18:30 <vikram> igordcard: it's there.. mohankumar can share
17:18:33 <cathy_> mohankumar: could you post the link of the horizon patch to igordcard ?
17:18:39 <igordcard> vikram, cathy_ thank you
17:18:44 <Prithiv> i have it igor
17:18:54 <igordcard> Prithiv, I want to see it upstream or up for review
17:18:59 <pcarver> cathy_: I think so. I just don't know if I've got the right list. My thought based on igordcard's install instructions would be to focus on 238428 and 233758
17:19:24 <vikram> https://review.openstack.org/#/c/231868/
17:19:28 <pcarver> because those are the two changes that his instructions pull in in order to get a working install
17:19:34 <vikram> igordcad: https://review.openstack.org/#/c/231868/
17:19:55 <pcarver> I don't want to give +1 or +2 to anything that I haven't pulled in locally to a working installation
17:20:10 <igordcard> vikram, oh.. so horizon is the only one where support can go to the upstream project (Horizon)?
17:20:30 <cathy_> pcarver: so you mean that the devstack patch that mohan posted does not work?
17:20:36 <mohankumar> igordcard , https://review.openstack.org/#/c/231868/
17:20:36 <igordcard> pcarver, same thought here
17:20:57 <vikram> igordcard: mohankumar can update on this
17:21:09 <pcarver> cathy_: I haven't checked it. I've been focussing on igordcard's instructions and getting them to work.
17:21:23 <igordcard> vikram, mohankumar okay thanks, we can talk offline
17:21:46 <mohankumar> igordcard , will ping you once meeting finished
17:22:01 <igordcard> cathy_, I believe mohan's patch would work if the code was already merged
17:22:21 <cathy_> pcarver: I would like us to have one destack patch that works and that everyone can use  to install the functionality.
17:22:37 <igordcard> cathy_, I have left some comments on mohankumar devstack patch
17:22:40 <cathy_> igordcard: OK, thanks for the info
17:22:42 <pcarver> cathy_: agreed. I don't want redundant effort
17:22:49 <vikram> pcarver: can you mention about the issue you are facing for the devstack patch
17:23:51 <cathy_> pcarver: so let's use mohan's devstack patch to install it and post comments if something does not work or any modification suggestion. This will benefit other people having the same issue.
17:24:08 <pcarver> vikram: I've been following instructions that igordcard sent in email and got to the point that the services are running and the database tables are present.
17:24:27 <mohankumar> cathy_ :  devstack changes will work , once all the code gets into sfc repo
17:24:34 <igordcard> cathy_, not really possible until the code is merged - it's a chicken and egg problem - what we need now is cumbersome, but working, instructions, so we can test and try, then review and merge - and then a single devstack plugin enabling will be enough
17:24:59 <pcarver> vikram: but I can't create VMs due to some other problem that may or may not be sfc related. I haven't dug into it yet, but it prevented me from setting up a service chain if I can't create a VM.
17:25:10 <mohankumar> till then some manual efforts needs based on patch updates we having
17:25:18 <cathy_> mohankumar: thanks. SO I guess the key issue is that we need to get all the codes merged into the repo, then the installation process pain will be gone.
17:25:26 <igordcard> cathy_, oh yeah
17:25:45 <vikram> cathy_:+++1000
17:26:13 <igordcard> pcarver, we can talk here on IRC afterwards to bring the VMs up as well
17:26:31 <igordcard> pcarver, but I can't help about the chaining itself, as I don't see the encapsulation flows on ovs-ofctl :(
17:27:05 <pcarver> igordcard: Thanks, I'll try. Meetings are a persistent issue as well, today may be mostly a loss.
17:27:09 <igordcard> which is why I was asking about the flow classifier readiness, even more because it is marked as WIP right now
17:27:24 <cathy_> igordcard: vikram OK, I will approve the merge once all the end-to-end functionality test cases are all passed. Maybe another week or shorter since now we have got several testbeds up and the SFC installed and have progressed on the testing well.
17:28:12 <cathy_> pcarver: we have got VM created and chained in on several of our testbeds.
17:28:51 <vikram> cathy_: plz notify the team when to start the review before the merge
17:29:43 <cathy_> pcarver: besides getting a VM installed, you also need to either have a SF running on the VM or do some config to fake the traffic hair-pin through the VM.
17:29:48 <igordcard> vikram, +1
17:30:27 <cathy_> vikram: you can start the review now. As I said before, all the patches have been open for review for over one month and we have incorporated most if not all comments
17:30:53 <vikram> cathy_: thanks
17:30:58 <igordcard> cathy_, what do you mean by that having a SF running on the VM?
17:30:58 <pcarver> cathy_: yep, understood. My latest issue is just not being able to boot a VM and a bunch of errors on the q-agt screen. I haven't had time between meetings to get to the bottom of it.
17:31:31 <cathy_> pcarver: that seems like a non-SFC issue.
17:31:43 <Prithiv> cathy: same question as igor
17:32:02 <pcarver> igordcard, cathy_: I'm assuming something as simple as a Linux VM with IP forwarding enabled will suffice as an SF to demonstrate that the chaining works.
17:32:21 <Prithiv> pcarver: +1
17:32:27 <Prithiv> from the demo vi, it sounded like that
17:32:31 <Prithiv> demo vid
17:32:32 <igordcard> pcarver, understood
17:32:38 <cathy_> igordcard: Prithiv yes, pcarver stated what I meant:-)
17:32:51 <igordcard> so, if I set up tcpdump in my VM I should see packets coming in right? even if I don't enable anything
17:32:56 <LouisF> pcarver: correct
17:33:41 <pcarver> cathy_: maybe, maybe not. One of the files in the traceback is /opt/stack/networking-sfc/networking_sfc/services/sfc/agent/agent.py but it's not the top of bottom of the traceback, so I'm not sure if it's the cause.
17:33:52 <pcarver> but we don't need to discuss that as part of this meeting
17:33:58 <cathy_> igordcard: yes if you enable some "Ip forwarding" so that the traffic going into the VM will be fowarded out of the VM and can go to next SF VM per the API specification
17:34:48 <cathy_> pcarver: let's discuss that offline after you did a little bit more investigation on the VM creation issue
17:35:01 <pcarver> cathy_: agreed
17:35:05 <igordcard> cathy_, this was my scenario: client VM - SF1 VM - Internet ; traffic was sent from the client VM, but was never captured in SF1 VM (in its ingress interface)
17:35:42 <igordcard> I will share the flow list, at least for reference: http://paste.openstack.org/show/cIjekDRVyMxkyOQCZdYc/
17:35:48 <cathy_> what chain API did you specify?
17:36:04 <igordcard> I see pop_mpls flows, but I don't see any push_mpls flows
17:36:23 <cathy_> did you set up the SF1 VM to do the "hairpin"?
17:36:29 <igordcard> cathy_, so, SF1 was a port-pair-group with a single port-pair inside
17:37:03 <igordcard> cathy_, and I matched all udp traffic going to 8.8.8.8, so I did a UDP iperf to 8.8.8.8 at my client VM and expected traffic to reach SF1, but it didn't
17:37:55 <igordcard> cathy_, no, it should at least receive traffic right? being capable of forwarding it would be my second test
17:38:17 <cathy_> igordcard: maybe we can discuss and debug this offline? Shoot us an email? OK with you?
17:38:24 <igordcard> cathy_, sure!
17:38:27 <vikram> igordcard, cathy_: I feel a detailed documentation about installation and testing would be useful
17:38:48 <igordcard> vikram, +1 I can post my current cumbersome instructions to the wiki if you want
17:39:01 <vikram> igordcard: Great..
17:39:11 <vikram> igordcard: We can update the devrefs as well
17:39:13 <LouisF> igordcard: please do that
17:39:43 <igordcard> thanks to pcarver for all the feedback
17:39:58 <vikram> igordcard: IMO, updating existing devref doc for this would be useful
17:41:01 <vikram> igordcard: or we can add a separate one illustrating test/installation nits
17:41:04 <igordcard> vikram, maybe, but the instructions I have are only relevant before merge...
17:41:22 <pcarver> vikram: I think we need to update the installation devref, but I'm not sure we want to do that with the intermediate procedure that will go away once changes merge
17:41:45 <cathy_> igordcard: thanks. could you post it on the SFC wik link and we will add ours?
17:41:59 <mohankumar> pcarver: IMO  After merge will be good
17:42:02 <vikram> pcarver, cathy_: i am okay with wiki as well
17:42:03 <igordcard> after merge it should literally be: git clone devstack; append networking-sfc plugin line to local.conf
17:42:06 <cathy_> let me find the link and post it here
17:42:24 <LouisF> igordcard: +100
17:42:25 <igordcard> cathy_, yeah
17:42:48 <vikram> igordcard: I was mostly interested on the test process
17:42:49 <cathy_> https://wiki.openstack.org/wiki/Neutron/ServiceInsertionAndChaining
17:43:16 <cathy_> vikram: we will post the testing process
17:43:18 <vikram> igordcard: For diff scenarios's, may be single / multiple node deployment
17:43:30 <vikram> cathy_: Great!
17:43:33 <igordcard> vikram, yes, I too am interested, very much, on the test process, as so far I haven't been able to have the right OF flows
17:43:58 <cathy_> igordcard: we will post the testing process
17:44:14 <igordcard> cathy_, LouisF, thank you
17:44:24 <vikram> igordcard: don't worry till cathy_ is there :-)
17:44:59 <cathy_> now I would like to discuss the Neutron branch the SFC codes base on.
17:47:15 <cathy_> currently our codes are based on the master branch and we found that the functionality gets broken due to new changes on the master branch. This is slowed down our testing work. So we think we will rebase our code to a stable master branch label. Neutron M-1 cut is end of this week. So if there is a label for that cut, we will rebase our codes to that M-1 label
17:48:32 <mohankumar> cathy_ : sounds good !
17:49:05 <cathy_> does anyone know whether there is such a M-1 label?
17:49:46 <johnsom> https://github.com/openstack/neutron/releases
17:49:53 <cathy_> I have sent email to Kyle on that and hopefully there is such a label
17:49:57 <johnsom> 8.0.0.0b1 is M1 I think
17:50:34 <cathy_> johnsom: great. Thanks for the info!
17:50:36 <Prithiv> yes
17:50:44 <Prithiv> https://launchpad.net/neutron/+milestone/mitaka-1
17:50:44 <cathy_> cool
17:51:14 <cathy_> Prithiv: thanks!
17:51:58 <mohankumar> cathy_ : so we not incorporate any  master changes after M-1 ?
17:52:23 <cathy_> no, we will incorporate master changes later
17:53:08 <mohankumar> cathy_ : okay
17:53:10 <cathy_> after we get the testing completed without much interference and then get our first patches merged to remove the dependency pain and the installation pain
17:53:56 <igordcard> cathy_, so, will the code be merged after or before M1 rebase?
17:54:11 <cathy_> after M1 rebase
17:55:42 <cathy_> After rebase we will continue our testing and if all test cases passed, I will approve the merge and then we can start the bug filing process for any more issues we found, then we will rebase to incorporate changes on master and get ready for release. make sense to everyone?
17:56:50 <mohankumar> cathy_ +1
17:57:10 <pcarver> cathy_ +1
17:57:39 <igordcard> cathy_, yes
17:57:46 <LouisF> cathy_: +1
17:59:28 <vikram> cathy_:+1
17:59:49 <cathy_> Ok, time is up. Let's work hard together on the testing and "bug fixes"(we will not officially file bugs, but will keep a copy of the problems found and fix them), and get the codes ready. Also post comments if you have more.
17:59:55 <LouisF> bye
18:00:05 <s3wong> bye
18:00:06 <cathy_> bye now.
18:00:09 <vikram> bye
18:00:12 <igordcard> bye
18:00:21 <cathy_> #endmeeting