16:00:14 <slaweq> #startmeeting neutron_ci
16:00:18 <openstack> Meeting started Tue Jun 25 16:00:14 2019 UTC and is due to finish in 60 minutes.  The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:19 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:21 <openstack> The meeting name has been set to 'neutron_ci'
16:00:24 <slaweq> hi
16:00:29 <ralonsoh> hi
16:00:54 <slaweq> I know that mlavalle and bcafarel will not be there today but lets wait couple of minutes for others
16:02:56 <ralonsoh> this is going to be a 1vs1 match
16:03:53 <slaweq> ralonsoh: yeah :)
16:04:05 <slaweq> maybe we can just cancel it then, what do You think?
16:04:09 <ralonsoh> let's go
16:04:14 <ralonsoh> ok ok
16:04:16 <ralonsoh> no
16:04:25 <ralonsoh> is there something important?
16:04:27 <ralonsoh> critical?
16:04:51 <slaweq> TBH I was  today involved in one internal issue almost whole day
16:04:58 <slaweq> and I didn't have time to check it
16:05:23 <slaweq> lets maybe check now grafana how it looks like
16:05:25 <slaweq> ok?
16:05:29 <ralonsoh> sure
16:05:32 <slaweq> without strict agenda :)
16:05:40 <ralonsoh> no problem, at least you solved yesterday the CI problem with the nova patch
16:05:43 <slaweq> http://grafana.openstack.org/d/Hj5IHcSmz/neutron-failure-rate?orgId=1
16:05:51 <slaweq> yes, that one should be fine now :)
16:05:56 <ralonsoh> it is
16:06:04 <slaweq> that is good :)
16:07:01 <ralonsoh> do you see something with high failure rate?
16:07:16 <ralonsoh> neutron-tempeste-with_uwsgi
16:07:28 <slaweq> yes, but this one is like that for long time
16:07:35 <slaweq> and I didn't have time to look into it yet
16:07:37 <ralonsoh> I see
16:08:01 <slaweq> other than that there is also neutron-tempest-plugin-dvr-multinode-scenario (non-voting) failing a lot
16:08:05 <slaweq> but that is also "normal"
16:08:12 <ralonsoh> you are right
16:08:35 <slaweq> and good news is that I think I found why test_connectivity_through_2_routers is failing
16:08:49 <ralonsoh> did you? what's happening?
16:09:24 <slaweq> so it's IMO race
16:09:30 <slaweq> instances are spawned here: https://github.com/openstack/neutron-tempest-plugin/blob/master/neutron_tempest_plugin/scenario/test_connectivity.py#L86
16:09:33 <njohnston> sorry I am late
16:09:45 <slaweq> and after that in next lines networks are connected to the routers
16:09:53 <ralonsoh> yes
16:10:13 <ralonsoh> and do you think there is a race condition there?
16:10:15 <slaweq> and if L3 agent is overloaded and is connecting everything slowly, than instance boot and can't configure public keys as metadata is not reachable yet
16:10:22 <ralonsoh> ahhhh
16:10:27 <slaweq> I was able to reproduce this issue locally
16:10:36 <njohnston> makes sense, good find
16:10:51 <slaweq> and when I moved spawning servers to L103, it passed few times for me
16:11:06 <ralonsoh> so, adding more time to reach the metadata is not possible
16:11:23 <slaweq> and also evey log which I checked from this failed test got those errors with reaching metadata IP
16:11:32 <slaweq> njohnston: and hi :)
16:11:50 <ralonsoh> oook so good change, add me to the patch reviews
16:11:56 <slaweq> sure
16:12:03 <slaweq> but, I will push patch tomorrow probably
16:12:07 <ralonsoh> ok
16:12:13 <ralonsoh> I have another issue, related to CI
16:12:20 <slaweq> and the same issue can be in our d/s CI with this test
16:12:23 <ralonsoh> and maybe real deployments
16:12:48 <ralonsoh> #link https://bugs.launchpad.net/neutron/+bug/1833721
16:12:49 <openstack> Launchpad bug 1833721 in neutron "ip_lib synchronized decorator should wrap the privileged one" [Medium,In progress] - Assigned to Rodolfo Alonso (rodolfo-alonso-hernandez)
16:12:58 <ralonsoh> the patch proposed is not working in py27
16:13:30 <ralonsoh> and maybe, if pyroute2 fixes the problem in the next days, we can remove the sync decorators
16:13:46 <ralonsoh> so I'll keep this bug just for reference
16:13:51 <slaweq> I hope we can remove this decorator soon :)
16:13:57 <ralonsoh> sure
16:14:04 <ralonsoh> that's all from my side
16:14:47 <slaweq> njohnston: just FYI, as we were only 2 with ralonsoh we decided today to quickly check current CI status and don't follow usual agenda today
16:15:00 <slaweq> njohnston: do You have anything You would like to discuss?
16:15:14 <njohnston> sounds good; I have been hip deep in regressions but nothing that pertains to CI
16:16:18 <njohnston> so nothing here
16:16:31 <slaweq> ok, so I think we can finish this meeting earlier today as I didn't prepare anything else
16:16:38 <ralonsoh> perfect for me
16:16:48 <slaweq> CI looks "as usual" today so we should be good
16:16:55 <ralonsoh> hehehe "usual"
16:17:00 <slaweq> ;)
16:17:10 <njohnston> better than usual!
16:17:24 <slaweq> maybe even better :)
16:17:28 <ralonsoh> hehehe
16:17:58 <slaweq> ok thx for attending
16:18:04 <slaweq> and see You online :)
16:18:06 <slaweq> o/
16:18:07 <njohnston> \o
16:18:09 <slaweq> #endmeeting