16:01:27 <mlavalle> #startmeeting neutron_ci
16:01:28 <openstack> Meeting started Tue Aug 13 16:01:27 2019 UTC and is due to finish in 60 minutes.  The chair is mlavalle. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:01:29 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:01:31 <openstack> The meeting name has been set to 'neutron_ci'
16:02:48 <bcafarel> hey again
16:03:05 <mlavalle> hey how are you
16:03:15 <mlavalle> most likely, it will you and me today
16:03:40 <mlavalle> ralonsoh is on pto and slaweq has something personal to do
16:03:48 <mlavalle> so let's get going
16:04:19 <bcafarel> sounds good
16:04:27 <mlavalle> so please open the grafana dashboard: http://grafana.openstack.org/dashboard/db/neutron-failure-rate
16:05:21 <mlavalle> #topic Actions from previous meetings
16:05:50 <mlavalle> mlavalle will continue debugging https://bugs.launchpad.net/neutron/+bug/1838449
16:05:51 <openstack> Launchpad bug 1838449 in neutron "Router migrations failing in the gate" [Medium,Confirmed] - Assigned to Miguel Lavalle (minsel)
16:05:57 <mlavalle> which I did
16:06:28 <mlavalle> my findings are:
16:07:12 <mlavalle> 1) Failure occurs becuase, when senting the router to admin_state_up to false, the distributed router interface port status is never set to DOWN
16:08:30 <mlavalle> 2) That in turn is caused because on of the L3 agents involved takes 5 minutes to process the router update, so the port is not removed from the corrsponding node (controller) fast enough
16:08:49 * mlavalle chacking how long it takes for the test to timeout
16:09:26 <bcafarel> digging through my recent mails but was there a recent stable backport related to 1? it sounds familiar to me
16:10:54 <bcafarel> oh nevermind it was a fix for ports already stuck in DOWN ( https://review.opendev.org/#/c/606827/ )
16:10:56 <mlavalle> yeah, I can see that the test timeouts in 300 seconds (5 minutes): https://github.com/openstack/neutron-tempest-plugin/blob/99684ac97eb1ad1e0be6fd824ba00ee707421e0f/neutron_tempest_plugin/scenario/test_migration.py#L81
16:12:05 <mlavalle> so I am going to investigate why the l3 agent takes so long in one of the nodes (controller). The port is removed very quickly in the other node (compute1)
16:12:33 <mlavalle> so action is digging why L3 agent might take so long to update a router update
16:13:18 <mlavalle> #action mlavalle will continue debugging https://bugs.launchpad.net/neutron/+bug/1838449. specifically, why l3 agent is taking so long in one of the nodes to process a router update
16:13:19 <openstack> Launchpad bug 1838449 in neutron "Router migrations failing in the gate" [Medium,Confirmed] - Assigned to Miguel Lavalle (minsel)
16:13:55 <mlavalle> #topic Stadium projects
16:15:19 <bcafarel> I think most topics (and any progress in this quiet week) were already covered in team meeting
16:15:19 <mlavalle> pyhton3 migration Stadium projects etherpad: https://etherpad.openstack.org/p/neutron_stadium_python3_status
16:15:40 <mlavalle> yeah, just mentioning the etherpad
16:16:03 <mlavalle> lajoskatona made progress with networking-odl
16:16:39 <mlavalle> tempest-plugins migrations Etherpad: https://etherpad.openstack.org/p/neutron_stadium_move_to_tempest_plugin_repo
16:17:01 * mlavalle hoping to make progress with vpnaas this wekk
16:17:04 <mlavalle> week^^^
16:17:53 <mlavalle> any other updates on other tempest-plugins
16:18:13 <bcafarel> https://review.opendev.org/#/c/643668/ looks good with slaweq's latest change, but I did not have time to review yet
16:18:28 <bcafarel> that one will get fwaas in the completed section too
16:19:24 <mlavalle> yeah, and I will probably cheat and see if that applies to what I'm doing on the vpnaas side
16:19:52 <mlavalle> :-)
16:19:58 <bcafarel> :)
16:20:18 <mlavalle> ok, let's move on
16:20:21 <mlavalle> #topic Grafana
16:22:55 <mlavalle> I don't see much to worry about. the gate and check queues seem to be having a summer vacation as well. They might be in the South of France going to the beach
16:23:22 <mlavalle> or Ibiza, maybe
16:24:00 <bcafarel> a warmer place at least :)
16:24:23 <bcafarel> grenade jobs took their turn on a grumpy day apparently, but that was short
16:25:00 <mlavalle> ok, moving on
16:25:08 <mlavalle> #topic Tempest/Scenario
16:25:29 <mlavalle> neutron-tempest-plugin-scenario-openvswitch - https://review.opendev.org/#/c/670738/ - merged already
16:26:07 <mlavalle> we added a neutron-tempest-plugin-scenario-openvswitch job
16:27:36 <mlavalle> neutron-tempest-plugin-scenario-openvswitch-iptables-hybrid https://review.opendev.org/#/c/674274/
16:28:17 <mlavalle> scenario tests with openvswitch agent and
16:28:19 <mlavalle> iptables_hybrid firewall driver.
16:28:46 <bcafarel> nice, I know a few users that will like having this one around!
16:29:10 <mlavalle> and we updated the docs with their description: https://review.opendev.org/#/c/674272/2/doc/source/contributor/testing/ci_scenario_jobs.rst
16:29:50 <mlavalle> anything else on this topic?
16:31:14 <bcafarel> just seeing the grafana, https://bugs.launchpad.net/neutron/+bug/1837552 is still valid
16:31:15 <openstack> Launchpad bug 1837552 in neutron "neutron-tempest-with-uwsgi job finish with timeout very often" [Medium,Confirmed]
16:31:37 <bcafarel> I wanted to take a look at this one this week, but it will probably have to wait for next week (public holiday on Thursday)
16:31:58 <mlavalle> enjoy!
16:32:06 <mlavalle> what's the ocassion?
16:33:03 <mlavalle> #topic Periodic
16:33:04 <bcafarel> just it being the middle of August I guess :)
16:33:19 <mlavalle> nice!
16:33:57 <mlavalle> slaweq left a note in the etherpad pointing to the logs: Logs from jobs are on: http://logs.openstack.org/periodic/opendev.org/openstack/neutron/master/
16:34:20 <mlavalle> I don't think there are any action itmers here
16:34:29 <mlavalle> #topic On demand agenda
16:34:42 <mlavalle> anything else we should discuss today?
16:35:24 <bcafarel> none from me, recent stable jobs had some failures, but not out of the ordinary rate
16:35:34 <mlavalle> cool
16:35:39 <mlavalle> thanks for attending
16:35:45 <mlavalle> #endmeeting