15:00:10 <slaweq> #startmeeting neutron_ci
15:00:10 <opendevmeet> Meeting started Tue Feb 28 15:00:10 2023 UTC and is due to finish in 60 minutes.  The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:10 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:10 <opendevmeet> The meeting name has been set to 'neutron_ci'
15:00:19 <slaweq> ping bcafarel, lajoskatona, mlavalle, mtomaska, ralonsoh, ykarel, jlibosva for CI meeting
15:00:23 <ralonsoh> hi
15:00:29 <slaweq> o/
15:00:36 <slaweq> Grafana dashboard: https://grafana.opendev.org/d/f913631585/neutron-failure-rate?orgId=1
15:00:57 <mtomaska> o/
15:01:05 <bcafarel> o/
15:01:35 <opendevreview> Dmitrii Shcherbakov proposed openstack/neutron master: Allow Multiple External Gateways  https://review.opendev.org/c/openstack/neutron/+/873593
15:01:35 <opendevreview> Dmitrii Shcherbakov proposed openstack/neutron master: Add extra router attributes for ECMP and BFD  https://review.opendev.org/c/openstack/neutron/+/874797
15:01:39 <slaweq> ok, lets start
15:02:03 <slaweq> it needs to be a bit shorter meeting than usual as I need to finish around 16:45 today
15:02:07 <slaweq> sorry for that
15:02:12 <slaweq> #topic Actions from previous meetings
15:02:16 <lajoskatona> o/
15:02:18 <slaweq> lajoskatona to continue checking dvr functional tests issues
15:03:00 <lajoskatona> not much progress, but we discussed today that some wait/sleep or busy-wait should be added to plug_new
15:03:29 <lajoskatona> so I will check it, thanks for the comment slaweq
15:03:38 <slaweq> yeah, I was checking other issue, with interface not found and my analysis is here: https://paste.opendev.org/show/b2a6RB41JeeEt5Rv51Of/
15:03:57 <slaweq> maybe the same would fix that other DVR failures too
15:04:03 <lajoskatona> and the patch I used recently to try different aproaches: https://review.opendev.org/c/openstack/neutron/+/873111
15:04:42 <slaweq> lajoskatona I will propose simple patch with sleep and will rebase Your test patch on top of it to test if that will help
15:04:44 <lajoskatona> yes most probably as I found the same root method plug_new, so I hope we can catch it there
15:04:45 <slaweq> ok for You?
15:04:55 <lajoskatona> yes let's do it
15:04:59 <slaweq> thx
15:05:26 <slaweq> #action slaweq to propose "sleep patch" to hopefully fix interface not found and dvr tests issues
15:05:36 <lajoskatona> but in recent runs I havent seen the issue, only other failures
15:06:42 <slaweq> slaweq to check https://64e9807acd80d4cab1ab-851182d93d98b3728f798963c90c7371.ssl.cf5.rackcdn.com/periodic/opendev.org/openstack/neutron/master/neutron-functional-with-uwsgi-fips/1914083/testr_results.html
15:06:44 <slaweq> it's the one which I already mentioned above
15:06:44 <slaweq> so lets move on to the next one
15:06:45 <slaweq> slaweq to report bug about macspoofing_port test in ovn-tempest-slow job
15:06:59 <slaweq> ok, next one
15:07:15 <slaweq> bug reported https://bugs.launchpad.net/neutron/+bug/2008062
15:07:43 <slaweq> it's the only one not assigned to anyone
15:07:51 <slaweq> as You mentioned in the previous meeting ralonsoh
15:08:10 <ralonsoh> if I have time this week, I'll check it and assign it to myself
15:08:18 <ralonsoh> but I can't promise that
15:08:18 <mtomaska> I can take a look next week
15:08:19 <slaweq> thx
15:08:23 <mtomaska> so let me take it
15:08:23 <slaweq> I didn't saw it this week
15:08:31 <slaweq> so it's not that often
15:09:10 <slaweq> thx mtomaska
15:09:13 <slaweq> I will assign AI for You for next week
15:09:16 <slaweq> #action mtomaska to check https://bugs.launchpad.net/neutron/+bug/2008062
15:09:19 <mtomaska> ok
15:09:20 <slaweq> and last one:
15:09:21 <slaweq> ralonsoh to check neutron-functional-with-sqlalchemy-master failures
15:09:31 <ralonsoh> yes, I've opened a bug for oslo.db
15:09:35 <ralonsoh> https://bugs.launchpad.net/oslo.db/+bug/2008209
15:09:41 <ralonsoh> there is a patch under review
15:09:50 <ralonsoh> and I've pushed this for Neutron
15:10:03 <ralonsoh> https://review.opendev.org/c/openstack/neutron/+/874778
15:10:20 <ralonsoh> this patch should NOT be merged, but I'll rebase it frequently
15:10:29 <ralonsoh> to check the warnings still present in the code
15:10:42 <ralonsoh> that's all
15:11:15 <slaweq> thx for taking care of it ralonsoh
15:11:16 <bcafarel> great work there :)
15:11:31 <slaweq> it's really important to make all those fixes and be ready for sqlalchemy 2.0 :)
15:11:49 <slaweq> and also thx zzzeek for huge help with this issue
15:11:55 <ralonsoh> for sure!
15:12:12 <slaweq> I think we can now move on to the next topic
15:12:14 <slaweq> #topic Stable branches
15:12:25 <slaweq> bcafarel the floor is Yours :)
15:13:18 <opendevreview> Christian Rohmann proposed openstack/neutron-vpnaas master: Improve the reconciling for VPNaaS  https://review.opendev.org/c/openstack/neutron-vpnaas/+/875745
15:13:21 <opendevreview> Rodolfo Alonso proposed openstack/neutron master: Add Slawomir Kaplonski as Lieutenant in API, Client and Testing areas  https://review.opendev.org/c/openstack/neutron/+/875746
15:13:36 <bcafarel> I was away a few days last week so I may have missed a few things - but overall stable branches are in good shape
15:14:11 <bcafarel> I wonder about ussuri 2 recent backports had a lot of failures https://review.opendev.org/c/openstack/neutron/+/873629 https://review.opendev.org/c/openstack/neutron/+/874979 but I did not have time to look into yet
15:15:17 <slaweq> yeah, worth to check
15:15:48 <haleyb> bcafarel: i am interested as well so can probably look as one of those is from someone here, let me know
15:16:13 <bcafarel> haleyb: that would be great! I don't think I will have lots of time this week
15:17:34 <slaweq> thx haleyb and bcafarel - if You will need any help, please let me know
15:18:13 <slaweq> ok, next topic
15:18:15 <slaweq> #topic Stadium projects
15:18:23 <slaweq> lajoskatona how things are going?
15:18:35 <slaweq> I saw that networking-sfc and networking-odl were red this week
15:18:52 <lajoskatona> yes, that is due to sqlalchemy2 with the new job, I will check it
15:18:58 <lajoskatona> I mean for sfc
15:18:58 <slaweq> ok
15:18:59 <opendevreview> Christian Rohmann proposed openstack/neutron-vpnaas master: Improve the reconciling for VPNaaS  https://review.opendev.org/c/openstack/neutron-vpnaas/+/875745
15:19:14 <lajoskatona> for ODL I am not sure, I started to check
15:19:14 <opendevreview> Brian Haley proposed openstack/neutron stable/ussuri: DNM: Testing Ussuri gate  https://review.opendev.org/c/openstack/neutron/+/875747
15:20:08 <slaweq> thx lajoskatona
15:20:17 <lajoskatona> that;s it from me
15:20:19 <opendevreview> Rodolfo Alonso proposed openstack/neutron master: Add Jens Harbott as Lieutenants in Infra area  https://review.opendev.org/c/openstack/neutron/+/875748
15:20:29 <slaweq> so lets move on
15:20:35 <slaweq> #topic Grafana
15:21:01 <slaweq> #link https://grafana.opendev.org/d/f913631585/neutron-failure-rate
15:21:38 <slaweq> IMO things looks better this week
15:22:22 <slaweq> there are some failures, especially during weekend there were some spikes but I saw many patches marked as "WIP" this week and those were very red
15:22:29 <slaweq> so I don't see there any specific issue this week
15:22:35 <ralonsoh> much better since last week fixes
15:22:45 <slaweq> yeah
15:23:02 <slaweq> I think we can smoothly go to the next topic which is
15:23:04 <slaweq> #topic Rechecks
15:23:13 <slaweq> and here it also seems that it's better this week
15:23:32 <slaweq> we are again below 1 recheck in average to merge patches
15:23:57 <ralonsoh> cool
15:24:04 <lajoskatona> \o/
15:24:09 <slaweq> thx all for hard work with the gate failures
15:24:16 <mtomaska> \o\`
15:24:29 <slaweq> I hope that if we will finally fix those functional tests issues it will be really good :)
15:25:20 <slaweq> so now lets talk about particular failures :)
15:25:35 <slaweq> #topic fullstack/functional
15:25:39 <slaweq> here I found few issues which we already discussed today, related to dvr lifecycle and interface not found issues
15:25:40 <slaweq> but except that I saw also:
15:25:43 <slaweq> Initial state of ha router failed:
15:25:47 <slaweq> https://45c66512638ca6be811a-ed2c0d9273209e4be8351fb65446e61d.ssl.cf1.rackcdn.com/periodic/opendev.org/openstack/neutron/master/neutron-functional-with-oslo-master/90c9e81/testr_results.html
15:25:59 <slaweq> I think we saw that one few weeks ago too
15:26:33 <slaweq> ralonsoh do You maybe remember it?
15:26:41 <ralonsoh> yes but I didn't check it
15:27:00 <ralonsoh> this code, to be honest, is not very stable
15:27:11 <ralonsoh> (not the agent but the test itself)
15:27:28 <ralonsoh> I'll check it tomorrow
15:27:43 <slaweq> thx a lot
15:28:01 <opendevreview> Fernando Royo proposed openstack/ovn-octavia-provider master: Add new FTs for health monitoring basic operations  https://review.opendev.org/c/openstack/ovn-octavia-provider/+/875749
15:28:06 <slaweq> #action ralonsoh to check failed initail state of ha router test: https://45c66512638ca6be811a-ed2c0d9273209e4be8351fb65446e61d.ssl.cf1.rackcdn.com/periodic/opendev.org/openstack/neutron/master/neutron-functional-with-oslo-master/90c9e81/testr_results.html
15:28:12 <slaweq> and next one
15:28:26 <slaweq> neutron.tests.functional.agent.test_dhcp_agent.DHCPAgentOVSTestCase.test_good_address_allocation
15:28:27 <slaweq> https://359f4eae1c9be4d72e49-0343763e072aec8a44b36b196f4e6ba5.ssl.cf2.rackcdn.com/periodic/opendev.org/openstack/neutron/master/neutron-functional-with-uwsgi-fips/ff7e0fd/testr_results.html
15:28:33 <slaweq> also I think we have seen it already before
15:29:04 <slaweq> yeah, many times actually
15:29:09 <slaweq> I see it in etherpad
15:29:09 <ralonsoh> but I pushed a patch last week (I think so) related to this test
15:29:25 <opendevreview> Merged openstack/networking-sfc master: Update master for stable/zed  https://review.opendev.org/c/openstack/networking-sfc/+/857683
15:29:26 <slaweq> so hopefully it will be fixed
15:29:31 <slaweq> is Your patch merged?
15:29:42 <ralonsoh> checking now
15:29:58 <slaweq> maybe we spot it before Your patch was merged
15:30:31 <ralonsoh> https://review.opendev.org/c/openstack/neutron/+/874658
15:30:39 <ralonsoh> and https://review.opendev.org/c/openstack/neutron/+/874654/1 (merged)
15:31:14 <slaweq> ok, but failure was on 26.02
15:31:19 <slaweq> so before this patch was merged
15:31:30 <slaweq> hopefully with those fixes we will not see that issue again :)
15:31:34 <ralonsoh> perfect
15:31:56 <slaweq> even 24.02
15:32:05 <slaweq> but still before patch was merged :)
15:32:09 <slaweq> thx ralonsoh
15:32:28 <slaweq> and that's all what I have for today in the agenda
15:32:38 <slaweq> not many new failures this week, which is good
15:32:55 <slaweq> anything else You want to discuss today as we have few more minutes left
15:33:04 <slaweq> #topic On Demand
15:34:00 <opendevreview> Merged openstack/neutron master: [OVN] Ensure traffic for provider vlan networks is not tunneled  https://review.opendev.org/c/openstack/neutron/+/871252
15:34:06 <opendevreview> Merged openstack/neutron master: Change neutron-ovs-tempest-dvr-ha-multinode-full job's config  https://review.opendev.org/c/openstack/neutron/+/874536
15:34:27 <slaweq> if not, I will give You some time back today
15:34:44 <mtomaska> o/
15:34:45 <slaweq> thx for attending the meeting and helping with CI stability :)
15:34:50 <ralonsoh> bye!
15:34:52 <slaweq> have a great day!
15:34:52 <lajoskatona> o/
15:34:53 <slaweq> o/
15:34:56 <slaweq> #endmeeting