16:00:55 <slaweq> #startmeeting neutron_ci
16:00:56 <openstack> Meeting started Tue Aug  6 16:00:55 2019 UTC and is due to finish in 60 minutes.  The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:57 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:01:01 <openstack> The meeting name has been set to 'neutron_ci'
16:01:01 <ralonsoh> hello
16:01:03 <slaweq> hi
16:01:12 <mlavalle> LO, I thought I had missed the meeting by 1 hour
16:01:15 <mlavalle> LOL
16:01:35 <slaweq> :D
16:01:48 <slaweq> mlavalle: You're just in time
16:02:20 <mlavalle> yes, I realized that right a way, but for a split second I got confused
16:02:21 <slaweq> Nate is no PTO this week
16:02:28 <mlavalle> on
16:02:39 <slaweq> but lets wait 2 more minutes for haleyb and others
16:02:46 <haleyb> hi
16:03:06 <slaweq> hi haleyb
16:03:26 <haleyb> hi, will need to leave about :15 early
16:03:46 <slaweq> haleyb: ok
16:03:53 <slaweq> so lets start this meeting
16:03:55 <slaweq> Grafana dashboard: http://grafana.openstack.org/dashboard/db/neutron-failure-rate
16:04:01 <slaweq> #topic Actions from previous meetings
16:04:10 <slaweq> mlavalle to report bug with router migrations
16:04:22 <mlavalle> I did report the bug
16:04:33 <mlavalle> https://bugs.launchpad.net/neutron/+bug/1838449
16:04:35 <openstack> Launchpad bug 1838449 in neutron "Router migrations failing in the gate" [Medium,Confirmed] - Assigned to Miguel Lavalle (minsel)
16:04:42 <mlavalle> and I spent time debugging it
16:04:59 <mlavalle> to make a long story short
16:06:08 <mlavalle> the neutron server never receives notification that the router_interface_distributed is down in the controller ovs agent
16:06:21 <mlavalle> I can see that L3 agent removed the interface
16:06:41 <mlavalle> I can also see that the ovs agent removes the port
16:06:51 <mlavalle> but the notification never gets to the server
16:07:20 <mlavalle> and therefore the port remains in status UP from the point of the neutron server and the API
16:07:22 <slaweq> and do You see that notification was sent by ovs-agent?
16:07:47 <mlavalle> checking that is my next step
16:08:08 <mlavalle> I kight need to add some LOG statements in a DNM patch
16:08:50 <slaweq> yes, some additional logs might be useful for that
16:09:05 <mlavalle> but definitely the problem is somewhere in the notifcation path
16:09:41 <slaweq> so sounds like real bug, not testonly issue
16:09:50 <mlavalle> yes it is
16:10:02 <slaweq> thx mlavalle for update and for working on this
16:10:09 <mlavalle> in my dev system the test always succeeds
16:10:18 <mlavalle> but I am executing it without load
16:10:32 <mlavalle> so it must be related with the system under load
16:10:46 <slaweq> yes, probably
16:10:51 <mlavalle> this type is tough to debug
16:10:54 <slaweq> as many of our gate failures
16:11:11 <slaweq> I think we already fixed easy ones ;)
16:11:43 <slaweq> ok, lets move on to the next action
16:11:45 <slaweq> ralonsoh to report a bug about failed test neutron.tests.fullstack.test_qos.TestMinBwQoSOvs.test_bw_limit_qos_port_removed
16:12:06 <ralonsoh> slaweq, I talked to you last week
16:12:26 <ralonsoh> The logs are gone and I didn't see this error reproduced
16:12:26 <mlavalle> #action mlavalle will continue debugging router migration bug
16:12:42 <slaweq> ralonsoh: ok, I probably forgot about it
16:12:44 <ralonsoh> so I didn't open a bug because we don't have a log to report
16:12:48 <slaweq> now I remember
16:12:57 <slaweq> ok, sounds good for me
16:13:01 <ralonsoh> (but I'm still checking the CI)
16:13:08 <slaweq> mlavalle: thx for adding action for Yourself
16:13:21 <slaweq> ralonsoh: thx :)
16:13:33 <slaweq> ok, next one than
16:13:35 <slaweq> slaweq to check midonet job and report bug(s) related to it
16:13:54 <slaweq> I checked that there was issue with expired SSL cert for some midonet domain
16:14:13 <slaweq> yamamoto workarounded that with https://review.opendev.org/#/c/674538/
16:14:40 <slaweq> but today I found out that there is still one test failing in this job, so I reported bug https://bugs.launchpad.net/networking-midonet/+bug/1839169
16:14:41 <openstack> Launchpad bug 1839169 in networking-midonet "Tempest test tempest.api.compute.admin.test_migrations.MigrationsAdminTest is failing 100% times" [Undecided,New]
16:15:10 <slaweq> and I just sent patch to skip this test in this midonet job running on neutron repo
16:15:26 <slaweq> https://review.opendev.org/674858
16:16:56 <slaweq> and the last one from last week was
16:16:58 <slaweq> slaweq to report and try to fix bug in neutron_tempest_plugin.api.test_port_forwardings.PortForwardingTest
16:17:24 <slaweq> I reported this bug, started looking into it and than found out that this was releated to the patch on which it was running, so no bug at all :P
16:17:43 <slaweq> I just lost about 30minutes of my life on that one :D
16:18:12 <mlavalle> LOL
16:18:40 <slaweq> mlavalle: yeah, that was my reaction when I noticed that :)
16:18:47 <slaweq> ok, any questions/comments?
16:19:58 <slaweq> ok, lets move on than
16:20:02 <slaweq> #topic Stadium projects
16:20:12 <slaweq> python 3 migration
16:20:14 <slaweq> Stadium projects etherpad: https://etherpad.openstack.org/p/neutron_stadium_python3_status
16:20:48 <slaweq> I think there is nothing to update here today
16:21:54 <slaweq> ok, next one
16:21:56 <slaweq> tempest-plugins migration
16:21:59 <slaweq> Etherpad: https://etherpad.openstack.org/p/neutron_stadium_move_to_tempest_plugin_repo
16:22:23 <slaweq> I know that tidwellr is working on patch for neutron-dynamic-routing
16:22:36 <slaweq> and still have some problems with some tests there
16:22:45 <mlavalle> I did spend some time with the vpn one
16:22:47 <slaweq> any update about vpnaas mlavalle ?
16:22:52 <mlavalle> didn't make much progress though
16:23:00 <mlavalle> I'll continue this week
16:23:19 <slaweq> ok, thx for taking care of it
16:24:23 <slaweq> any questions/comments about stadium projects?
16:26:20 <slaweq> ok, lets move on then
16:26:23 <slaweq> next topic
16:26:25 <slaweq> #topic Grafana
16:26:37 <slaweq> link was given earlier: http://grafana.openstack.org/dashboard/db/neutron-failure-rate
16:27:04 <slaweq> in overall it looks quite good IMO
16:27:52 <mlavalle> and summer vacation has hit us
16:28:00 <mlavalle> it seems there is not a lot of activity
16:28:27 <slaweq> we had problem with functional/fullstack jobs at the end of last week but it's fixed now by frickler https://review.opendev.org/#/c/674426/
16:28:42 <slaweq> mlavalle: right, we have much less patches, especially in gate queue
16:28:46 <slaweq> that is my impression too
16:30:12 <slaweq> any other comments related to grafana in overall?
16:30:24 <mlavalle> none from me
16:30:41 <slaweq> ok, so lets move on
16:30:56 <slaweq> I don't have anything related to functional/fullstack jobs for today
16:31:01 <slaweq> so lets skip to
16:31:03 <slaweq> #topic Tempest/Scenario
16:31:15 <slaweq> here I have only one thing which I want to mention
16:31:45 <slaweq> recently I realized that we are not running neutron-tempest-plugin tests on any non-dvr openvswitch environment
16:31:51 <slaweq> so I proposed 2 new jobs:
16:31:56 <slaweq> neutron-tempest-plugin-scenario-openvswitch - https://review.opendev.org/#/c/670738/ - merged already
16:32:01 <slaweq> neutron-tempest-plugin-scenario-openvswitch-iptables-hybrid https://review.opendev.org/#/c/674274/
16:32:29 <slaweq> both are proposed as voting and gating because this is in fact our most common and most supported configuration
16:32:34 <slaweq> so it should works fine
16:32:46 <slaweq> Description of those jobs is proposed in https://review.opendev.org/#/c/674272/
16:33:08 <slaweq> please review those patches and tell me what You think about them :)
16:34:24 <slaweq> anything else You want to talk about, related to tempest/scenario jobs?
16:34:38 <mlavalle> the description is already merged
16:34:52 <slaweq> no, it's not as it depends-on on other patches
16:34:59 <slaweq> so it's approved but not merged yet
16:36:11 <mlavalle> ok
16:36:19 <mlavalle> approved the other one
16:36:59 <slaweq> mlavalle: thx a lot
16:37:26 <slaweq> I will push also patch to update grafana and add those 2 new jobs
16:37:46 <mlavalle> thanks
16:38:57 <slaweq> Last one thing which I have for today is new neutron-tempest-plugin release
16:39:08 <slaweq> I proposed it yesterday: https://review.opendev.org/#/c/674573/
16:39:35 <slaweq> the reason why I did it so fast after the last one is that it has some quite important new test cases, like multicast traffic test
16:39:54 <slaweq> so mlavalle if You can look at it too, that would be nice
16:40:03 <slaweq> amotoki already gave +1 for it :)
16:40:18 <mlavalle> ok
16:40:57 <mlavalle> done
16:41:03 <slaweq> mlavalle: thx
16:41:10 <slaweq> ok, that's all from my side for today
16:41:23 <mlavalle> I don't have anything else
16:41:24 <slaweq> (faster than ever)
16:41:36 <mlavalle> it must be the summer vacation spirit
16:41:47 <slaweq> if You don't have anything else to talk about, we can finish this meeting earlier today :)
16:41:53 <slaweq> mlavalle: that's right :)
16:41:56 <mlavalle> at least for us in the Northern hemisphere
16:42:13 <slaweq> yeah :)
16:42:19 <mlavalle> The Aussies are in the middle of their winter
16:42:32 <ralonsoh> (I'm going to the pool right now)
16:42:33 <slaweq> but do we have anyone from Southern hemispere here?
16:42:39 <mlavalle> no
16:42:44 <slaweq> :)
16:42:50 <slaweq> ralonsoh: have fun
16:42:52 <mlavalle> I am probably the one farthest south
16:43:09 <slaweq> ok, thx for attending
16:43:13 <mlavalle> both now and where I was born, whihch is even further south
16:43:34 <mlavalle> o/
16:43:40 <ralonsoh> bye!
16:43:41 <slaweq> have a great day and week :)
16:43:45 <slaweq> see You online
16:43:47 <slaweq> #endmeeting