15:00:04 <slaweq> #startmeeting neutron_ci 15:00:04 <opendevmeet> Meeting started Tue Aug 3 15:00:04 2021 UTC and is due to finish in 60 minutes. The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:00:04 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:00:04 <opendevmeet> The meeting name has been set to 'neutron_ci' 15:00:25 <bcafarel> hey again 15:00:56 <slaweq> hi 15:02:01 <slaweq> let's wait for lajoskatona, obondarev and maybe others to join 15:03:12 <lajoskatona> Hi, sorry 15:03:24 <slaweq> np lajoskatona :) 15:04:02 <slaweq> Grafana dashboard: http://grafana.openstack.org/dashboard/db/neutron-failure-rate 15:04:09 <slaweq> please open and we can start 15:04:16 <slaweq> #topic Actions from previous meetings 15:04:23 <slaweq> slaweq to check dns nameserver issues in cirros 15:04:35 <slaweq> I was checking that and I proposed patch https://review.opendev.org/c/openstack/tempest/+/802005 15:04:38 <slaweq> it's now merged 15:04:50 <slaweq> I didn't saw any such issue in last few days 15:04:58 <slaweq> so hopefully this patch solves that problem 15:05:02 <slaweq> *solved 15:05:20 <slaweq> but if You would see same issue again, please let me know 15:05:25 <bcafarel> crossing fingers 15:05:42 <slaweq> next one 15:05:44 <slaweq> slaweq to blacklist failing test_established_tcp_session_after_re_attachinging_sg in LB job 15:05:49 <slaweq> done https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/801602 15:06:01 <slaweq> and last one 15:06:03 <slaweq> slaweq to check FIP set to DOWN issue 15:06:18 <slaweq> there was old bug reported already https://bugs.launchpad.net/neutron/+bug/1815585 15:06:30 <slaweq> but I didn't had time to look at it deeper 15:06:57 <obondarev> hi, here, sorry 15:07:09 <slaweq> obondarev: no worries :) hi 15:07:15 <slaweq> #topic Stadium projects 15:07:24 <slaweq> lajoskatona: any new issues with ci in stadium? 15:07:38 <slaweq> or anything related stadium's ci to discuss? 15:08:02 <lajoskatona> just the payload changes, but at least happens something during the summer :-) 15:08:13 <slaweq> LOL 15:08:26 <bcafarel> :) 15:08:40 <bcafarel> elod also reported some periodic job failing in stadium, 1 sec 15:09:03 <bcafarel> I pasted mail in https://etherpad.opendev.org/p/neutron-periodic-stable-202107 15:09:15 <lajoskatona> +1 15:09:30 <lajoskatona> He just mentioned it on our internal chat :-) 15:10:11 <slaweq> thx bcafarel 15:10:27 <slaweq> it seems we have issues with old branches on bgpvpn 15:10:33 <slaweq> and vpnaas 15:10:59 <slaweq> in case of bgpvpn maybe we can EOL pike and queens branches if there will be no volunteers to check those issues 15:11:01 <slaweq> wdyt? 15:11:48 <lajoskatona> yeah, can eol them 15:11:59 <bcafarel> that was my initial thinking too, vpnaas most can probably be fixed (and failures are in newer branches) 15:12:40 <slaweq> ok, I will check failures in bgpvpn and will start process of EOLing it if needed 15:13:01 <slaweq> #action slaweq to check networking-bgpvpn stable branches failures 15:13:07 <slaweq> any volunteer for vpnaas? 15:13:47 <bcafarel> o/ (if I do not have time until PTO I will tell you) 15:13:48 <lajoskatona> I can check 15:14:01 <lajoskatona> ok bcafarel wins :P 15:14:06 <slaweq> thank You both :) 15:14:09 <bcafarel> yes \o/ 15:14:19 <slaweq> #action bcafarel to check vpnaas failures in stable branches 15:15:26 <slaweq> ok, so I think stadium projects are covered already 15:15:29 <slaweq> now next topic 15:15:34 <slaweq> #topic Stable branches 15:15:44 <slaweq> bcafarel: any new issues in stable branches of neutron? 15:16:00 <bcafarel> mostly quiet with the summer, and the active backports went OK 15:16:04 <bcafarel> except ussuri with https://bugs.launchpad.net/neutron/+bug/1938262 15:16:34 <bcafarel> I posted more details in bug, but overall we seem to hit a kernel regression with IPv6+GRE 15:16:45 <bcafarel> (though I still do not understand why train is fine) 15:16:50 <slaweq> do we need to wait for new kernel in Ubuntu to unblock it? 15:17:06 <bcafarel> I have https://review.opendev.org/c/openstack/neutron/+/802552 as workaround in the meantime (2 tests to skip) 15:17:36 <slaweq> bcafarel: isn't Train running on older Ubuntu? 15:18:33 <bcafarel> Train and ussuri are both 18.04 (Bionic), we switched with Victoria 15:18:38 <slaweq> ok 15:18:46 <slaweq> I thought that maybe train was still on 16.04 15:19:19 <slaweq> so we need Your skip patch only on the ussuri, right? 15:19:31 <bcafarel> for now yes, maybe it will appear later in train? 15:19:56 <bcafarel> but until then yes only ussuri has issue 15:20:11 <slaweq> ok, I think we will need mlavalle's +2 power to approve https://review.opendev.org/c/openstack/neutron/+/802552 :) 15:20:23 <slaweq> as ralonsoh is off this week 15:22:26 <slaweq> ok, let's move on 15:22:27 <slaweq> #topic Grafana 15:23:30 <slaweq> http://grafana.openstack.org/dashboard/db/neutron-failure-rate 15:23:43 <slaweq> I see there that dvr grenade job is failing pretty often recently 15:24:17 <slaweq> and also scenario dvr ha job is failing almost 100% of runs (again) 15:24:39 <slaweq> obondarev: do You know about any issues which could cause that many failures recently? 15:25:23 <obondarev> I do 15:25:26 <bcafarel> I think we had a few bugs reported recently on DVR 15:25:49 <obondarev> please check https://bugs.launchpad.net/neutron/+bug/1938685 15:26:01 <obondarev> and fix https://review.opendev.org/c/openstack/neutron/+/803211 15:26:16 <obondarev> so far 3 successful ddvr-ha jobs in a row 15:27:27 <slaweq> obondarev: good catch 15:27:36 <obondarev> thanks bcafarel for adding Sean and Rodolfo 15:27:45 <slaweq> qq is this about all ports? 15:27:54 <obondarev> they are best candidates for review this one 15:28:01 <bcafarel> :) np they will know better about it indeed 15:28:07 <slaweq> I think that recently Sean and Rodolfo added binding for such ports on purpose in ovs agent 15:28:31 <obondarev> slaweq: ah, I missed that probably 15:28:45 <slaweq> obondarev: I'm not sure now if they did that in dvr too 15:28:55 <slaweq> but I remember that there was some work on that recently 15:29:14 <slaweq> so, I agree with bcafarel that they should review it before we will move on with that patch :) 15:29:21 <obondarev> but for DVR it's clearly an issue: OFPActionOutput with ofport == -1 will fail 15:29:45 <mlavalle> Done 15:29:57 <slaweq> obondarev: I will take a look at that patch more deeply too 15:29:59 <slaweq> :) 15:30:01 <slaweq> mlavalle: thx a lot 15:30:09 <obondarev> slaweq: thanks! 15:30:29 <slaweq> obondarev: added to my todo list for tomorrow morning :) 15:30:37 <obondarev> cool! 15:30:53 <slaweq> obondarev: thx for working on that 15:31:00 <obondarev> sure 15:31:06 <slaweq> anything else regarding grafana? or can we move on? 15:32:34 <slaweq> ok, so let's move on 15:32:37 <slaweq> next topic 15:32:45 <slaweq> #topic fullstack/functional 15:32:58 <slaweq> I found out that functional tests job is failing pretty often recently 15:33:08 <slaweq> and in most cases it was due to same error 15:33:15 <slaweq> I opened bug https://bugs.launchpad.net/neutron/+bug/1938766 15:33:41 <slaweq> I already pinged jlibosva and lucasagomes to take a look into that one 15:33:57 <slaweq> and I think jlibosva already did some investigation 15:34:56 <bcafarel> so it will probably be good by next CI meeting 15:35:00 <jlibosva> I think it's a regression in python-ovs 15:35:12 <jlibosva> after this commit - https://github.com/openvswitch/ovs/commit/c39751e44539a014e642bcd930cb9e3a33af1805 15:36:23 <slaweq> jlibosva: I just checked that in functional job we are using ovs from commit 0047ca3a0290f1ef954f2c76b31477cf4b9755f5 15:36:37 <jlibosva> it's been there for a while tho and also https://review.opendev.org/c/openstack/neutron/+/794892 was supposed to workaround that 15:38:19 <slaweq> so it seems that this patch didn't workaround that issue :/ 15:39:14 <jlibosva> yeah, I will have to investigate further, maybe it's a combination of things since we started to see it recently but the OVS patch has been in for over a year 15:39:40 <slaweq> jlibosva++ ok, thx for working on that 15:40:50 <slaweq> that was the only issue which I had to discuss today 15:41:01 <slaweq> maybe I still didn't catch up enough after week off :) 15:41:18 <slaweq> do You have any other issues/things to discuss today? 15:42:46 <bcafarel> not that I know of 15:43:43 <slaweq> if there are no other topics for today, I think we can finish few minutes earlier today 15:43:55 <slaweq> thx for attending the meeting and for keeping our ci green :) 15:44:02 <lajoskatona> o/ 15:44:03 <slaweq> have a great week and see You all online 15:44:05 <slaweq> o/ 15:44:07 <bcafarel> o/ 15:44:08 <slaweq> #endmeeting