15:00:25 <slaweq> #startmeeting neutron_ci
15:00:26 <openstack> Meeting started Wed Feb 26 15:00:25 2020 UTC and is due to finish in 60 minutes.  The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:27 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:27 <slaweq> hi
15:00:29 <openstack> The meeting name has been set to 'neutron_ci'
15:00:54 <njohnston> o/
15:01:12 <bcafarel> o/
15:02:03 <slaweq> ping ralonsoh: ci meeting
15:02:10 <ralonsoh> hi
15:02:24 <slaweq> ok, lets start
15:02:36 <slaweq> Grafana dashboard: http://grafana.openstack.org/dashboard/db/neutron-failure-rate
15:02:38 <slaweq> Please open now :)
15:03:07 <slaweq> #topic Actions from previous meetings
15:03:14 <slaweq> first one
15:03:16 <slaweq> ralonsoh to talk with oslo people about our functional tests rootwrap issue
15:03:28 <ralonsoh> one secx
15:03:44 <ralonsoh> https://review.opendev.org/#/c/708936/
15:03:52 <ralonsoh> patch merged and released 6.0.1
15:03:56 <ralonsoh> oslo rootwrap
15:04:21 <slaweq> ralonsoh++ thx a lot
15:04:33 <slaweq> and indeed I didn't saw such issues in last few days
15:04:50 <slaweq> ok, next one
15:04:52 <slaweq> slaweq to try to reproduce and debug neutron-tempest-dvr ssh issue
15:05:04 <slaweq> I didn't have time for that yet
15:05:13 <slaweq> but I have some idea about this dvr job
15:05:34 <slaweq> it is single node dvr job which IMO don't makes too much sense
15:05:40 <slaweq> so my proposal is to
15:05:47 <slaweq> a) remove this job from our ci
15:05:59 <slaweq> b) change a bit neutron-tempest-dvr-ha-multinode-full to run networking related tests only
15:06:09 <slaweq> c) make neutron-tempest-dvr-ha-multinode-full voting in few weeks if it will be stable enough
15:06:24 <slaweq> so finally we will have one less job in our CI
15:06:29 <bcafarel> hmm did we not plan something like that in Shanghai PTG? (dropping single node dvr)
15:06:33 <slaweq> what do You think about it?
15:06:44 <bcafarel> (and +1 on the plan of course)
15:06:56 <slaweq> bcafarel: nope, we agreed to drop some grenade single node jobs
15:07:04 <bcafarel> aah ok
15:07:25 <haleyb> i'm fine with it as long as multinode job is stable
15:07:49 <bcafarel> anyway yes that sounds good, it will be add more visibility (and stability) to multinode dvr job
15:08:02 <slaweq> haleyb: multinode job seems to be at least as stable as single node one
15:08:19 <slaweq> and in fact this single node job is testing almost only API
15:08:43 <haleyb> it wasn't always that way, but you and miguel have fixed some hard bugs :)
15:09:33 <slaweq> haleyb: but I'm not talking about neutron-tempest-plugin-dvr-multinode-scenario job which is still pretty not stable
15:09:53 <slaweq> I'm talking about promoting neutron-tempest-dvr-ha-multinode-full to be voting and replace neutron-tempest-dvr
15:10:11 <slaweq> the differences between neutron-tempest-plugin-dvr-multinode-scenario and neutron-tempest-dvr-ha-multinode-full are:
15:10:27 <slaweq> 1. one is DVR and the other is DVR HA
15:10:51 <slaweq> 2. one runs neutron-tempest-plugin tests and the other runs tempest API + some basic scenario tests
15:11:47 <haleyb> ack
15:12:22 <slaweq> so, can I assume that I have "green light" from the team to do that? :)
15:12:34 <ralonsoh> +1
15:12:41 <njohnston> +1
15:12:58 <slaweq> thx, so I will propose patch(es) for that asap :)
15:12:58 <haleyb> +1
15:13:28 <slaweq> #action slaweq to drop neutron-tempest-dvr job and finally replace it with neutron-tempest-dvr-ha-multinode-full
15:13:46 <slaweq> ok, next one
15:13:48 <slaweq> ralonsoh to check periodic neutron-ovn-tempest-ovs-master-fedora job's failures
15:14:01 <ralonsoh> I spent 20 mins on this
15:14:19 <ralonsoh> and then I had other high priority tasks
15:14:21 <ralonsoh> sorry again
15:14:29 <slaweq> ralonsoh: no problem :)
15:14:31 <ralonsoh> (this is the 2nd time I miss this task(
15:14:47 <slaweq> do You want to assign it to yourself for next week?
15:15:01 <slaweq> or should we simply report a bug and maybe someone will check it?
15:15:52 <ralonsoh> next week
15:15:57 <slaweq> ralonsoh: ok :)
15:15:59 <slaweq> thx
15:16:04 <slaweq> #action ralonsoh to check periodic neutron-ovn-tempest-ovs-master-fedora job's failures
15:16:15 <slaweq> ok, next one
15:16:17 <slaweq> slaweq to report tempest bug with routers cleanup
15:16:29 <slaweq> I checked this error and it wasn't problem with cleaning in tempest in fact
15:16:55 <slaweq> the root cause of this was that API call to add_router_interface was processed for more than 30 seconds by Neutron
15:17:14 <slaweq> so urllib got timeout waiting for response and it retried
15:17:44 <slaweq> but in the retry this subnet was already plugged to the router as first call was already finished
15:18:15 <slaweq> so it failed on adding interface to the router and because of that it wasn't added to cleanup list and failed to be cleaned too
15:18:25 <slaweq> so I didn't report tempest bug for that :)
15:18:54 <slaweq> and that's all actions from last week
15:19:03 <slaweq> anything else You want to ask/add in this section?
15:19:18 <ralonsoh> no
15:19:26 <slaweq> if no, lets move on
15:19:28 <slaweq> #topic Stadium projects
15:19:34 <slaweq> standardize on zuul v3
15:19:36 <slaweq> Etherpad: https://etherpad.openstack.org/p/neutron-train-zuulv3-py27drop
15:19:38 <slaweq> we are almost done there
15:20:02 <slaweq> I need to fix my patches for neutron-vpnaas
15:20:37 <bcafarel> nice
15:20:40 <slaweq> networking-bgpvpn is almost there - patch waits for some other fix for tempest job
15:20:57 <slaweq> we merged neutron-dynamic-routing patch recently
15:20:57 <njohnston> I need to check to see if we have any changes in the field yet for networking-odl or networking-midonet
15:21:22 <slaweq> njohnston: yes, those 2 projects are probably not touched yet
15:21:44 <bcafarel> hopefully current recheck will be the good one this time (bagpipe is back up and running thanks to amotoki lajoskatona
15:22:46 <slaweq> and according to py2 leftovers, we need to merge https://review.opendev.org/#/c/699213/ then release last version with py2 support and then merge https://review.opendev.org/#/c/704257/
15:23:00 <slaweq> so please review https://review.opendev.org/#/c/699213/
15:23:03 <slaweq> :)
15:23:20 <slaweq> thx bcafarel and ralonsoh for review of it :)
15:23:24 <ralonsoh> sure
15:23:32 <bcafarel> :) njohnston looks like you are next on it then
15:23:41 <slaweq> or haleyb :)
15:23:51 <slaweq> as he is here too :)
15:24:01 <bcafarel> oh true
15:24:07 <njohnston> https://review.opendev.org/699213 +2+W
15:24:16 <haleyb> nate beat me to it
15:24:30 <slaweq> thx :)
15:24:31 <bcafarel> haleyb: today you are in grey in IRC colors, so quite stealthy
15:24:37 <haleyb> oh
15:25:03 <slaweq> no, in my weechat haleyb is red :)
15:25:25 * haleyb can't hide
15:25:33 <slaweq> anything else You want to ask regarding stadium projects?
15:25:38 * njohnston uses limechat today so you are all golden
15:26:26 <bcafarel> :)
15:26:55 <bcafarel> final rocky releases will be out for neutron et all (so including stadium projects)
15:27:45 <slaweq> bcafarel: yes, it should be done this week, today or tomorrow
15:27:45 <njohnston> \o/
15:28:16 <slaweq> ok, lets move on
15:28:19 <slaweq> next topic
15:28:24 <slaweq> #topic Grafana
15:28:30 <slaweq> #link http://grafana.openstack.org/dashboard/db/neutron-failure-rate
15:28:46 <slaweq> we don't have almost anything in gate recently
15:29:04 <slaweq> and it's mostly because of ovn tempest jobs failing almost all the time
15:29:14 <ralonsoh> we are on it now
15:29:37 <ralonsoh> https://review.opendev.org/#/c/710019/
15:29:41 <slaweq> this patch should fix it probably https://review.opendev.org/#/c/709691/
15:30:05 <ralonsoh> this one for tempest-plugin
15:30:12 <ralonsoh> and then we need to recheck the FTs
15:30:22 <ralonsoh> yesterday we detected an error in OVN core
15:30:27 <ralonsoh> (the bug is opened)
15:30:33 <slaweq> ralonsoh: ok, thx for that too
15:30:42 <ralonsoh> and we have this https://review.opendev.org/#/c/709762/
15:30:43 <slaweq> but for now most important is this tempest-plugin issue
15:30:53 <ralonsoh> yes, I now
15:31:14 <maciejjozefczyk> yes, the bugs is about missing cherry-pick of one patch to ovn 2.12
15:31:23 <maciejjozefczyk> its already fixed but not cherry-picked
15:31:46 <maciejjozefczyk> I asked ovn folks to do it
15:32:01 <maciejjozefczyk> but anyways blacklisting it for now makes perfect sense
15:32:04 <slaweq> thx maciejjozefczyk and ralonsoh for taking care of it
15:32:25 <slaweq> njohnston: haleyb please check https://review.opendev.org/#/c/709691/ if You have a minute
15:32:40 <slaweq> that should allow us to make zuul happy at least from time to time ;)
15:32:46 <haleyb> slaweq: i did a second ago, +2+W
15:32:50 <njohnston> haleyb beat me to it
15:32:53 <slaweq> LOL
15:32:56 <maciejjozefczyk> :)
15:33:02 <slaweq> you have many races today ;P
15:33:36 <haleyb> and meetings
15:33:37 <njohnston> We need to have one more - tiebreaker!
15:33:47 <slaweq> njohnston: :D
15:34:13 <slaweq> if I will have one more I will send it to You as a tiebreaker :)
15:34:33 <slaweq> from other things in grafana, still functional tests are failing a lot
15:34:52 <slaweq> and I have one new "star" there
15:35:16 <slaweq> anythig else regarding grafana You want to add?
15:36:08 <ralonsoh> no
15:36:19 <slaweq> if no, lets talk about functional tests now
15:36:26 <slaweq> #topic Fullstack/Functional
15:36:43 <slaweq> I saw that during last week test neutron.tests.functional.services.trunk.drivers.openvswitch.agent.test_trunk_manager.TrunkManagerTestCase.test_connectivity failed many times
15:36:53 <slaweq> I have now opened LP to report this bug there
15:37:18 <ralonsoh> but this is because the problem in rootwrap
15:37:24 <slaweq> ralonsoh: yes, now I see
15:37:28 <ralonsoh> ok ok
15:37:41 <slaweq> I just saw some "FileNotFound" error and I though that it's something different :)
15:38:09 <slaweq> and this looks like issue with Your rootwrap patch already
15:39:55 <slaweq> ralonsoh: do You think we should report new bug for that?
15:40:25 <ralonsoh> but I have this one
15:40:26 <ralonsoh> https://bugs.launchpad.net/oslo.rootwrap/+bug/1864090
15:40:27 <openstack> Launchpad bug 1864090 in oslo.rootwrap ""PathFilter" can raise exception when executing "os.path.realpath()"" [Undecided,Fix released] - Assigned to Rodolfo Alonso (rodolfo-alonso-hernandez)
15:43:18 <slaweq> it's not with Your fix
15:43:40 <ralonsoh> ?
15:43:50 <slaweq> ralonsoh: so should Your fix https://review.opendev.org/#/c/708936/ solve this problem with functional tests?
15:44:30 <ralonsoh> yes, I think so
15:44:43 <slaweq> ok, so lets keep fingers crossed for that :)
15:45:33 <slaweq> and that was my main concern for today in fact
15:46:01 <slaweq> other than that I saw some single failures so lets just keep an eye on it for now
15:46:34 <slaweq> regarding scenario jobs we already discussed about ovn jobs and neutron-tempest-dvr so we are also good there
15:46:44 <slaweq> so that's basically all from my for today
15:46:55 <slaweq> do You have anything else You want to discuss today?
15:47:01 <slaweq> #topic On demand agenda
15:48:00 <maciejjozefczyk> I thinnk in near future we would be able to enable ovn multinode job
15:48:07 <maciejjozefczyk> with fixes:
15:48:19 <maciejjozefczyk> #link https://review.opendev.org/#/c/709111/
15:48:28 <maciejjozefczyk> #link https://review.opendev.org/#/c/709110/
15:49:04 <maciejjozefczyk> it will be green (was already, but failed on test_two_sec_groups - quota issue)
15:49:27 <maciejjozefczyk> thats all from me
15:50:06 <slaweq> ok, thx maciejjozefczyk :)
15:50:49 <slaweq> if there is nothing else for today, I think I can You back few minutes
15:50:52 <slaweq> thx for attending
15:50:54 <njohnston> o/
15:50:56 <slaweq> and see You next week
15:50:57 <ralonsoh> bye!
15:50:58 <slaweq> o/
15:51:00 <slaweq> #endmeeting