15:00:04 <slaweq> #startmeeting neutron_ci
15:00:04 <opendevmeet> Meeting started Tue Aug  3 15:00:04 2021 UTC and is due to finish in 60 minutes.  The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:04 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:04 <opendevmeet> The meeting name has been set to 'neutron_ci'
15:00:25 <bcafarel> hey again
15:00:56 <slaweq> hi
15:02:01 <slaweq> let's wait for lajoskatona, obondarev and maybe others to join
15:03:12 <lajoskatona> Hi, sorry
15:03:24 <slaweq> np lajoskatona :)
15:04:02 <slaweq> Grafana dashboard: http://grafana.openstack.org/dashboard/db/neutron-failure-rate
15:04:09 <slaweq> please open and we can start
15:04:16 <slaweq> #topic Actions from previous meetings
15:04:23 <slaweq> slaweq to check dns nameserver issues in cirros
15:04:35 <slaweq> I was checking that and I proposed patch https://review.opendev.org/c/openstack/tempest/+/802005
15:04:38 <slaweq> it's now merged
15:04:50 <slaweq> I didn't saw any such issue in last few days
15:04:58 <slaweq> so hopefully this patch solves that problem
15:05:02 <slaweq> *solved
15:05:20 <slaweq> but if You would see same issue again, please let me know
15:05:25 <bcafarel> crossing fingers
15:05:42 <slaweq> next one
15:05:44 <slaweq> slaweq to blacklist failing test_established_tcp_session_after_re_attachinging_sg in LB job
15:05:49 <slaweq> done https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/801602
15:06:01 <slaweq> and last one
15:06:03 <slaweq> slaweq to check FIP set to DOWN issue
15:06:18 <slaweq> there was old bug reported already https://bugs.launchpad.net/neutron/+bug/1815585
15:06:30 <slaweq> but I didn't had time to look at it deeper
15:06:57 <obondarev> hi, here, sorry
15:07:09 <slaweq> obondarev: no worries :) hi
15:07:15 <slaweq> #topic Stadium projects
15:07:24 <slaweq> lajoskatona: any new issues with ci in stadium?
15:07:38 <slaweq> or anything related stadium's ci to discuss?
15:08:02 <lajoskatona> just the payload changes, but at least happens something during the summer :-)
15:08:13 <slaweq> LOL
15:08:26 <bcafarel> :)
15:08:40 <bcafarel> elod also reported some periodic job failing in stadium, 1 sec
15:09:03 <bcafarel> I pasted mail in https://etherpad.opendev.org/p/neutron-periodic-stable-202107
15:09:15 <lajoskatona> +1
15:09:30 <lajoskatona> He just mentioned it on our internal chat :-)
15:10:11 <slaweq> thx bcafarel
15:10:27 <slaweq> it seems we have issues with old branches on bgpvpn
15:10:33 <slaweq> and vpnaas
15:10:59 <slaweq> in case of bgpvpn maybe we can EOL pike and queens branches if there will be no volunteers to check those issues
15:11:01 <slaweq> wdyt?
15:11:48 <lajoskatona> yeah, can eol them
15:11:59 <bcafarel> that was my initial thinking too, vpnaas most can probably be fixed (and failures are in newer branches)
15:12:40 <slaweq> ok, I will check failures in bgpvpn and will start process of EOLing it if needed
15:13:01 <slaweq> #action slaweq to check networking-bgpvpn stable branches failures
15:13:07 <slaweq> any volunteer for vpnaas?
15:13:47 <bcafarel> o/ (if I do not have time until PTO I will tell you)
15:13:48 <lajoskatona> I can check
15:14:01 <lajoskatona> ok bcafarel wins :P
15:14:06 <slaweq> thank You both :)
15:14:09 <bcafarel> yes \o/
15:14:19 <slaweq> #action bcafarel to check vpnaas failures in stable branches
15:15:26 <slaweq> ok, so I think stadium projects are covered already
15:15:29 <slaweq> now next topic
15:15:34 <slaweq> #topic Stable branches
15:15:44 <slaweq> bcafarel: any new issues in stable branches of neutron?
15:16:00 <bcafarel> mostly quiet with the summer, and the active backports went OK
15:16:04 <bcafarel> except ussuri with https://bugs.launchpad.net/neutron/+bug/1938262
15:16:34 <bcafarel> I posted more details in bug, but overall we seem to hit a kernel regression with IPv6+GRE
15:16:45 <bcafarel> (though I still do not understand why train is fine)
15:16:50 <slaweq> do we need to wait for new kernel in Ubuntu to unblock it?
15:17:06 <bcafarel> I have https://review.opendev.org/c/openstack/neutron/+/802552 as workaround in the meantime (2 tests to skip)
15:17:36 <slaweq> bcafarel: isn't Train running on older Ubuntu?
15:18:33 <bcafarel> Train and ussuri are both 18.04 (Bionic), we switched with Victoria
15:18:38 <slaweq> ok
15:18:46 <slaweq> I thought that maybe train was still on 16.04
15:19:19 <slaweq> so we need Your skip patch only on the ussuri, right?
15:19:31 <bcafarel> for now yes, maybe it will appear later in train?
15:19:56 <bcafarel> but until then yes only ussuri has issue
15:20:11 <slaweq> ok, I think we will need mlavalle's +2 power to approve https://review.opendev.org/c/openstack/neutron/+/802552 :)
15:20:23 <slaweq> as ralonsoh is off this week
15:22:26 <slaweq> ok, let's move on
15:22:27 <slaweq> #topic Grafana
15:23:30 <slaweq> http://grafana.openstack.org/dashboard/db/neutron-failure-rate
15:23:43 <slaweq> I see there that dvr grenade job is failing pretty often recently
15:24:17 <slaweq> and also scenario dvr ha job is failing almost 100% of runs (again)
15:24:39 <slaweq> obondarev: do You know about any issues which could cause that many failures recently?
15:25:23 <obondarev> I do
15:25:26 <bcafarel> I think we had a few bugs reported recently on DVR
15:25:49 <obondarev> please check https://bugs.launchpad.net/neutron/+bug/1938685
15:26:01 <obondarev> and fix https://review.opendev.org/c/openstack/neutron/+/803211
15:26:16 <obondarev> so far 3 successful ddvr-ha jobs in a row
15:27:27 <slaweq> obondarev: good catch
15:27:36 <obondarev> thanks bcafarel for adding Sean and Rodolfo
15:27:45 <slaweq> qq is this about all ports?
15:27:54 <obondarev> they are best candidates for review this one
15:28:01 <bcafarel> :) np they will know better about it indeed
15:28:07 <slaweq> I think that recently Sean and Rodolfo added binding for such ports on purpose in ovs agent
15:28:31 <obondarev> slaweq: ah, I missed that probably
15:28:45 <slaweq> obondarev: I'm not sure now if they did that in dvr too
15:28:55 <slaweq> but I remember that there was some work on that recently
15:29:14 <slaweq> so, I agree with bcafarel that they should review it before we will move on with that patch :)
15:29:21 <obondarev> but for DVR it's clearly an issue: OFPActionOutput with ofport == -1 will fail
15:29:45 <mlavalle> Done
15:29:57 <slaweq> obondarev: I will take a look at that patch more deeply too
15:29:59 <slaweq> :)
15:30:01 <slaweq> mlavalle: thx a lot
15:30:09 <obondarev> slaweq: thanks!
15:30:29 <slaweq> obondarev: added to my todo list for tomorrow morning :)
15:30:37 <obondarev> cool!
15:30:53 <slaweq> obondarev: thx for working on that
15:31:00 <obondarev> sure
15:31:06 <slaweq> anything else regarding grafana? or can we move on?
15:32:34 <slaweq> ok, so let's move on
15:32:37 <slaweq> next topic
15:32:45 <slaweq> #topic fullstack/functional
15:32:58 <slaweq> I found out that functional tests job is failing pretty often recently
15:33:08 <slaweq> and in most cases it was due to same error
15:33:15 <slaweq> I opened bug https://bugs.launchpad.net/neutron/+bug/1938766
15:33:41 <slaweq> I already pinged jlibosva and lucasagomes to take a look into that one
15:33:57 <slaweq> and I think jlibosva already did some investigation
15:34:56 <bcafarel> so it will probably be good by next CI meeting
15:35:00 <jlibosva> I think it's a regression in python-ovs
15:35:12 <jlibosva> after this commit - https://github.com/openvswitch/ovs/commit/c39751e44539a014e642bcd930cb9e3a33af1805
15:36:23 <slaweq> jlibosva: I just checked that in functional job we are using ovs from commit 0047ca3a0290f1ef954f2c76b31477cf4b9755f5
15:36:37 <jlibosva> it's been there for a while tho and also https://review.opendev.org/c/openstack/neutron/+/794892 was supposed to workaround that
15:38:19 <slaweq> so it seems that this patch didn't workaround that issue :/
15:39:14 <jlibosva> yeah, I will have to investigate further, maybe it's a combination of things since we started to see it recently but the OVS patch has been in for over a year
15:39:40 <slaweq> jlibosva++ ok, thx for working on that
15:40:50 <slaweq> that was the only issue which I had to discuss today
15:41:01 <slaweq> maybe I still didn't catch up enough after week off :)
15:41:18 <slaweq> do You have any other issues/things to discuss today?
15:42:46 <bcafarel> not that I know of
15:43:43 <slaweq> if there are no other topics for today, I think we can finish few minutes earlier today
15:43:55 <slaweq> thx for attending the meeting and for keeping our ci green :)
15:44:02 <lajoskatona> o/
15:44:03 <slaweq> have a great week and see You all online
15:44:05 <slaweq> o/
15:44:07 <bcafarel> o/
15:44:08 <slaweq> #endmeeting