15:00:37 <mlavalle> #startmeeting neutron_l3
15:00:43 <openstack> Meeting started Thu Oct 18 15:00:37 2018 UTC and is due to finish in 60 minutes.  The chair is mlavalle. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:44 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:46 <openstack> The meeting name has been set to 'neutron_l3'
15:00:49 <manjeets_> o/
15:00:54 <mlavalle> o/
15:00:55 <haleyb> hi
15:00:59 <davidsha_> o/
15:01:07 <mlavalle> haleyb: go Sox. almost there
15:01:25 <haleyb> even the bad calls have been going our way :)
15:01:44 * haleyb is very tired this morning
15:01:50 <mlavalle> is next game at Fainway?
15:01:58 <haleyb> no, tonight in houston
15:02:05 <mlavalle> good luck
15:02:06 <Swami> hi
15:02:29 <mlavalle> #topic Announcements
15:02:47 <mlavalle> Just a reminder that next week we reach Stein-1
15:02:59 <njohnston> o/
15:03:26 <mlavalle> and the Summit in Berlin is three weeks away, November 13 - 15
15:03:38 <mlavalle> any other annoucements?
15:04:15 <mlavalle> ok, let's move on
15:04:24 <mlavalle> #topic Bugs
15:04:47 <mlavalle> Swami: fire away
15:04:50 <Swami> mlavalle: thanks, will do
15:05:11 <Swami> #link https://bugs.launchpad.net/neutron/+bug/1794991
15:05:12 <openstack> Launchpad bug 1794991 in neutron "Inconsistent flows with DVR l2pop VxLAN on br-tun" [Undecided,New]
15:05:43 <Swami> I was trying to reproduce this pike, but unfortunately I am not successful in reproducing this bug.
15:06:03 <Swami> We have also internally seen such problems in newton.
15:06:41 <Swami> Has anyone else seen similar problems mentioned in this bug, where there is a race between the vxlan tunnel creation and the l2pop adding the flows for the vxlan.
15:06:54 <mlavalle> no I haven't
15:07:28 <mlavalle> so, according to the bug report, what is the effect when it happens?
15:07:29 <Swami> mlavalle: OK I will try couple of more tests to see if I can reproduce this. There is definitely a race.
15:08:02 <mlavalle> The reason I ask is that we have other bugs related to DVR
15:08:05 <Swami> When it happens the new VMs that come in on that compute cannot reach the DHCP server on the network node and also VM to VM communication will be blocked.
15:08:39 <Swami> mlavalle: So just to refresh, the vxlan tunnels are created when the agent first registers with the neutron-server right?
15:09:04 <Swami> Is there a possibility that the vxlan tunnel interfaces might change down the line.
15:09:05 <mlavalle> yes, that should be it
15:09:37 <mlavalle> and no, I don't think so
15:09:51 <mlavalle> The reason I am asking is that I am debugging a couple of DVR related bugs
15:10:12 <Swami> mlavalle: sure
15:10:14 <mlavalle> I will keep this in mind as a possible cause for the bugs I am working on
15:10:24 <mlavalle> If we see this problem, it should be in the gate
15:10:35 <mlavalle> where we have concurrency
15:10:39 <Swami> mlavalle: Yes basically you should have all the flows in place for the communication to be error free.
15:10:47 <Swami> mlavalle: ok thanks
15:11:04 <mlavalle> many of these races happen only with many concurrrent thinks happening
15:12:03 <Swami> mlavalle:yes that's why reproducing it is a nightmare, unless we have the hardware resources and a script to trigger it.
15:12:09 <Swami> #link https://bugs.launchpad.net/neutron/+bug/1793529
15:12:09 <openstack> Launchpad bug 1793529 in neutron "[dvr][ha][dataplane down] router_gateway port binding host goes wrong after the 'master' host down/up" [Undecided,In progress] - Assigned to LIU Yulong (dragon889)
15:12:28 <mlavalle> now, if the it hist DHCP, it seems to me that the problem goes beyond DVR, right?
15:12:31 <Swami> There is a patch up for review. #link https://review.openstack.org/#/c/606384/
15:13:42 <Swami> mlavalle: no the reason is there should be a flow rule in there with the MAC address of the dhcp server pointing to the right vxlan tunnel. Otherwise the dhcp request is not sent
15:14:03 <mlavalle> ah, that's right
15:14:03 <mlavalle> ok?
15:14:04 <mlavalle> ok
15:14:42 <Swami> mlavalle: It is directly a problem with DHCP, but somehow when the l2pop populates the flows, it is either not populated properly when the VM on that network pops up or it is getting removed on restarts etc.,
15:15:00 <mlavalle> ack
15:15:02 <Swami> s/it is directly/it is not directly
15:15:06 <Swami> mlavalle: thanks
15:15:21 <Swami> If you have not review this, please review this patch. #link https://review.openstack.org/#/c/606384/
15:15:53 <Swami> The next one in the list is
15:16:31 <Swami> #link https://bugs.launchpad.net/neutron/+bug/1796491
15:16:31 <openstack> Launchpad bug 1796491 in neutron "DVR Floating IP setup in the SNAT namespace of the network node and also in the qrouter namespace in the compute node" [Medium,In progress] - Assigned to Swaminathan Vasudevan (swaminathan-vasudevan)
15:16:59 <Swami> There is a patch up for review. #link https://review.openstack.org/609924
15:17:20 <Swami> haleyb: liuyulong: thanks for your reviews on the patch.
15:17:47 <Swami> haleyb: I will address your comment, I think it is a minor one. So I hope this patch is in a good condition.
15:17:56 <haleyb> thanks
15:18:44 <Swami> The next one is
15:18:49 <Swami> #link https://bugs.launchpad.net/neutron/+bug/1796824
15:18:49 <openstack> Launchpad bug 1796824 in neutron "Port in some type of device_owner should not allow update IP address" [Medium,In progress] - Assigned to LIU Yulong (dragon889)
15:19:11 <Swami> There is a patch up for review #link https://review.openstack.org/#/c/608909/
15:19:22 <Swami> If not review please review it.
15:20:04 <Swami> I think this bug #link https://bugs.launchpad.net/neutron/+bug/1785227 is a duplicate of the above one. So I am going to mark it as Duplicate.
15:20:04 <openstack> Launchpad bug 1785227 in neutron "Router port: no dataplane update on change" [Medium,Confirmed]
15:20:13 <Swami> Does anyone have any concerns on this.
15:20:37 <mlavalle> go ahead
15:20:50 <Swami> mlavalle: ok
15:21:11 <Swami> #link https://bugs.launchpad.net/neutron/+bug/1797037
15:21:11 <openstack> Launchpad bug 1797037 in neutron "Extra routes configured on routers are not set in the router namespace and snat namespace with DVR-HA routers" [Medium,In progress] - Assigned to Brian Haley (brian-haley)
15:21:53 <Swami> #link https://review.openstack.org/609273 - This patch fails zuul.
15:22:30 <Swami> It seems to be happy now, but let us see if it will merge.
15:22:56 <Swami> #link https://bugs.launchpad.net/neutron/+bug/1786272
15:22:56 <openstack> Launchpad bug 1786272 in neutron "Connection between two virtual routers does not work with DVR" [Medium,In progress] - Assigned to Slawek Kaplonski (slaweq)
15:23:15 <slaweq> I just pushed new PS for this one: https://review.openstack.org/#/c/597567/27
15:23:21 <Swami> #link https://review.openstack.org/#/c/597567/
15:23:50 <slaweq> liuyulong is doing great review and testing manually different scenarios on this patch
15:23:55 <Swami> slaweq: yes just noticed.  I will review it. I hope everything is fine with this patch. But liuyulong had some questions still.
15:23:59 <slaweq> I hope that now it will be fine
15:24:10 <Swami> slaweq: Ok thanks.
15:24:31 <slaweq> Swami: if You can, please try it also on Your env - it's really tricky patch IMO and I don't want to break something :)
15:24:37 <liuyulong> I'm testing this locally too.
15:24:47 <Swami> slaweq: Yes I will do one more round of testing today.
15:24:48 <slaweq> You have much more experience with dvr than me
15:24:55 <slaweq> thx liuyulong and Swami
15:25:13 <Swami> slaweq:I also wanted to test the shared-network scenario with this patch between two tenants. Let me do it.
15:25:29 <slaweq> Swami: ok
15:25:31 <slaweq> thx
15:25:44 <haleyb> thanks everyone, at PS27 this has taken a long time but it's good work :)
15:26:04 <Swami> haleyb: Yep it is a complex one.
15:26:20 <slaweq> yes, it is
15:26:24 <Swami> The next one is
15:26:29 <Swami> #link https://bugs.launchpad.net/neutron/+bug/1774459
15:26:29 <openstack> Launchpad bug 1774459 in neutron "Update permanent ARP entries for allowed_address_pair IPs in DVR Routers" [High,Confirmed] - Assigned to Swaminathan Vasudevan (swaminathan-vasudevan)
15:27:17 <Swami> mlavalle: haleyb: can you take a look at this patch . #link https://review.openstack.org/601336
15:27:49 <mlavalle> Swami: yes, I promised last week to take a look and couldn't find the time. sorry
15:27:50 <Swami> This is just a WIP based on our conversation at the PTG, but I am waiting for your inputs on this. I have a question. I also mentioned it in the last meeting.
15:27:55 <mlavalle> I'll try again this week
15:27:59 <Swami> mlavalle: thanks
15:28:18 <haleyb> i'll try to look by tomorrow too
15:28:28 <Swami> mlavalle: haleyb : thank you
15:28:36 <Swami> mlavalle: that's all I have for today.
15:28:41 <Swami> mlavalle: Back to you.
15:29:01 <mlavalle> I want to mention two bugs
15:29:19 <mlavalle> First one is https://bugs.launchpad.net/neutron/+bug/1795870
15:29:19 <openstack> Launchpad bug 1795870 in neutron "Trunk scenario test test_trunk_subport_lifecycle fails from time to time" [High,Confirmed] - Assigned to Miguel Lavalle (minsel)
15:29:36 <mlavalle> it was reported as a trunk failure (it is a trunk test failure)
15:30:10 <mlavalle> however, based on the debugging I've done so far, it seems to be a DVR problem
15:30:22 <mlavalle> I left debugging notes a couple of days ago
15:30:37 <mlavalle> so for the time being I'm adding it to the DVR backlog
15:30:44 <mlavalle> and I'l, continue debugging
15:31:23 <Swami> mlavalle: thanks
15:31:41 <mlavalle> the problem really occurs when trying to ssh to a test instance using a floating ip
15:33:05 <mlavalle> the second bug is https://bugs.launchpad.net/neutron/+bug/1787919
15:33:05 <openstack> Launchpad bug 1787919 in neutron "Upgrade router to L3 HA broke IPv6" [High,Confirmed] - Assigned to Miguel Lavalle (minsel)
15:33:25 <mlavalle> At this point I have a deployment reproducing the conditions and am testing
15:33:41 <mlavalle> I'll keep pushing forward with this one as well
15:34:16 <mlavalle> finally, we have https://bugs.launchpad.net/neutron/+bug/1789434
15:34:16 <openstack> Launchpad bug 1789434 in neutron "neutron_tempest_plugin.scenario.test_migration.NetworkMigrationFromHA failing 100% times" [High,Confirmed] - Assigned to Manjeet Singh Bhatia (manjeet-s-bhatia)
15:34:27 <manjeets_> https://review.openstack.org/#/c/611461/
15:34:57 <manjeets_> I posted a solution ideally notification part should be in l3_hamodedb code
15:35:25 <manjeets_> Please reviews it if solution is okay I’ll update and add tests
15:36:18 <Swami> manjeets_: Yes i did a first pass on it yesterday
15:36:36 <manjeets_> Thanks swami totally agree to your comments
15:37:26 <manjeets_> I’ll revise it today
15:38:18 <Swami> manjeets_: thanks
15:38:41 <mlavalle> manjeets_: I will also look at it soon
15:40:50 <mlavalle> any other bugs we should discuss today?
15:42:01 <mlavalle> ok, moving on
15:42:19 <mlavalle> #topic On demand agenda
15:43:01 <davidsha_> mlavalle, Thank you for looking at the agent refactor, We're working through it atm to make sure the tests are right.
15:43:19 <mlavalle> davidsha_: last evening I left Xubo some guidnace as to how to debug the failures
15:43:29 <davidsha_> It's mostly mocks pointing to the wrong objects atm.
15:43:50 <mlavalle> yeah, I know. I just didn't want to fix it for him
15:44:04 <mlavalle> I wanted to have him work through the process
15:44:28 <mlavalle> it's part of my job to help new devs
15:45:24 <davidsha_> kk, We're having calls as well so I'll try to keep things moving!
15:45:56 <mlavalle> tell him that if he has questions or needs guidance, he can leave a comment in the patch and I'll follow up
15:46:37 <mlavalle> any other topics to discuss?
15:46:58 <mlavalle> ok, thanks for attending
15:47:06 <davidsha_> cool, think he's online if you have any questions
15:47:08 <davidsha_> thanks!
15:47:11 <mlavalle> have a great weekend!
15:47:33 <mlavalle> haleyb: go finish those Astros off tonight!
15:47:34 <mlavalle> #endmeeting