15:02:14 <slaweq> #startmeeting neutron_ci
15:02:15 <opendevmeet> Meeting started Tue Jun  8 15:02:14 2021 UTC and is due to finish in 60 minutes.  The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:02:16 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:02:18 <opendevmeet> The meeting name has been set to 'neutron_ci'
15:02:20 <slaweq> hi (again)
15:04:09 <opendevreview> Slawek Kaplonski proposed openstack/neutron-tempest-plugin master: Add new API tests for active field in the allowed_address_pairs  https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/795358
15:04:15 <slaweq> anyone still here? :)
15:04:24 <lajoskatona> Hi,
15:04:54 <ralonsoh> hi
15:05:06 <slaweq> ok, let's start :)
15:05:13 <obondarev> hi
15:05:19 <slaweq> Grafana dashboard: http://grafana.openstack.org/dashboard/db/neutron-failure-rate
15:05:27 <slaweq> please open to have it ready
15:05:30 <slaweq> and we can move on
15:05:36 <slaweq> #topic Actions from previous meetings
15:06:01 <slaweq> slaweq to prepare plan of switch ovn <-> ovs jobs in neutron CI
15:06:12 <slaweq> I did such table: https://etherpad.opendev.org/p/neutron-ci-ovs-to-ovn
15:06:25 <slaweq> please check it and let me know what do You think about it
15:06:47 <ralonsoh> good stuff
15:06:48 <ralonsoh> thanks
15:07:03 <ralonsoh> what are we going to do with the naming?
15:07:55 <lajoskatona> we said last time to have the backend in the job name as I remember
15:08:07 <obondarev> neutron-rally-task should be  ovs-ovs
15:08:35 <obondarev> ovn multinode seems missing
15:09:07 <slaweq> for neutron-rally-task that's good point - it can be renamed to neutron-ovs-rally-task
15:09:13 <lajoskatona> +1
15:09:16 <slaweq> that will be more consistent
15:09:45 <slaweq> and good news is that I think that we can remove completly neutron-ovn-tempest-ovs-release  :)
15:09:52 <slaweq> so one job less in our CI
15:09:56 <slaweq> wdyt?
15:10:19 <ralonsoh> but this one is using master ovs, right?
15:10:30 <slaweq> no, released ovs
15:10:45 <ralonsoh> so what's the different then?
15:11:00 <lajoskatona> By the way haleyb missed jobs last time: http://lists.openstack.org/pipermail/openstack-discuss/2021-June/022904.html
15:11:45 <slaweq> ralonsoh I think we have same coverage in other jobs which by default runs ovn now
15:11:53 <slaweq> but I will check that to be sure
15:12:15 <slaweq> lajoskatona I think that this should be now fixed, right?
15:12:30 <haleyb> yes, that was reverted and i see the jobs again
15:12:37 <lajoskatona> ok
15:13:03 <slaweq> haleyb++ thx for catching that
15:13:37 <haleyb> zuul passing in 30 minutes was suspicious :-o
15:13:42 <lajoskatona> neutron-ovn-tempest-slow   & neutron-tempest-slow-py3 can't we keep them and rename to show which one is ovn and ovs?
15:14:00 <ralonsoh> yeah
15:14:21 <slaweq> lajoskatona yes, good point
15:16:08 <slaweq> so basically we will have pairs like *-ovs and *ovn jobs
15:16:23 <slaweq> and it will be for rally, tempest-slow, and ipv6-only jobs
15:17:37 <slaweq> anyone wants to push patches for that?
15:18:11 <ralonsoh> I can do it
15:19:15 <slaweq> thx ralonsoh
15:20:16 <slaweq> #action ralonsoh to update our ci job's definitions
15:20:22 <slaweq> ok, let's move on
15:20:30 <slaweq> ralonsoh to propose tempest patch to use resolvectl to address https://bugs.launchpad.net/neutron/+bug/1929523
15:20:31 <opendevmeet> Launchpad bug 1929523 in neutron "Test tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_subnet_details is failing from time to time" [High,Confirmed]
15:20:43 <ralonsoh> one sec
15:20:54 <ralonsoh> https://review.opendev.org/c/openstack/tempest/+/794776
15:21:40 <slaweq> thx ralonsoh
15:21:54 <slaweq> I will review it later
15:22:06 <slaweq> ok, next one
15:22:07 <slaweq> slaweq to reach out to boden about payload patches and x/vmware-nsx
15:22:25 <slaweq> I tried to contact Adit from Vmware but he didn't reply to me
15:22:48 <slaweq> but I see that Salvadore approved that patch
15:22:51 <slaweq> so it's merged now
15:23:02 <slaweq> lajoskatona I think we are good with neutron patch now, right?
15:23:22 <lajoskatona> cool, than we didn't break their ci
15:23:45 <slaweq> yeah :)
15:23:53 <slaweq> ok, next one
15:23:56 <slaweq> ralonsoh to check failing neutron-lib-from-master periodic job
15:24:09 <ralonsoh> I didn't check anything
15:24:23 <obondarev> I think I saw a patch
15:24:29 <ralonsoh> really?
15:24:32 <slaweq> yeah, I saw it too :)
15:24:36 <obondarev> yes :)
15:25:54 <slaweq> https://review.opendev.org/c/openstack/neutron/+/794236
15:25:58 <slaweq> that one I think
15:25:59 <slaweq> no?
15:26:25 <ralonsoh> ahhh yes
15:26:34 <slaweq> LOL
15:26:35 <ralonsoh> pffff my brain if melting
15:26:58 <slaweq> ralonsoh TooManyPatchesException() :P
15:27:26 <obondarev> :D
15:27:46 <slaweq> so thx for fixing that issue ralonsoh :)
15:27:49 <slaweq> and let's move on
15:27:51 <slaweq> lajoskatona to check fullstack failures https://bugs.launchpad.net/neutron/+bug/1930401
15:27:53 <opendevmeet> Launchpad bug 1930401 in neutron "Fullstack l3 agent tests failing due to timeout waiting until port is active" [Critical,Confirmed] - Assigned to Lajos Katona (lajos-katona)
15:28:14 <lajoskatona> that is some ugly privsep issue
15:28:35 <slaweq> ahh, that one
15:28:36 <lajoskatona> I pushed one patch, but with ralonsoh and obondarev we agreed that is not enough
15:28:41 <slaweq> the famous one :)
15:28:50 <ralonsoh> I agree with the idea of having two decorators
15:28:55 <ralonsoh> one with timeout
15:29:12 <ralonsoh> and we can also add a default timeout in the context definition
15:29:28 <obondarev> +1
15:29:32 <lajoskatona> yeah I started to check that and can produce terrible failures in my locan env already
15:29:49 <lajoskatona> so it's ongoing
15:30:12 <obondarev> meanwhile workaround is almost there: https://review.opendev.org/c/openstack/neutron/+/795300
15:30:28 <lajoskatona> this privsep looks like if I touch somewhere I have a break somewhere else and can't see the connection :-)
15:30:29 <obondarev> lajoskatona: please add workaround revert in your patch
15:30:38 <lajoskatona> obondarev: sure
15:31:25 <slaweq> thx lajoskatona and obondarev for working on that issue
15:31:32 <slaweq> it is really killing our CI now
15:31:47 <obondarev> to ralonsoh as well
15:31:54 <slaweq> ahh, right :)
15:32:05 <slaweq> sorry ralonsoh that I missed You
15:32:12 <ralonsoh> hehehe
15:32:30 <slaweq> ok, last one from previous week
15:32:33 <slaweq> lajoskatona to mark failing TestUninterruptedConnectivityOnL2AgentRestart fullstack tests as unstable temporary
15:32:59 <lajoskatona> https://review.opendev.org/c/openstack/neutron/+/794228
15:33:24 <lajoskatona> ohhh, it's more red than green now....
15:34:30 <lajoskatona> looks like some glitch in pip mirroring perhaps
15:34:35 <slaweq> thx lajoskatona
15:34:50 <slaweq> and thx obondarev and ralonsoh for reviewing it :)
15:34:56 <ralonsoh> yw
15:35:05 <slaweq> ok, let's move on to the other topics
15:35:05 <obondarev> sure
15:35:09 <slaweq> #topic Stadium projects
15:35:16 <slaweq> lajoskatona any issues with stadium's ci?
15:35:33 <lajoskatona> nothing as I know
15:35:48 <slaweq> no news is good news in this case :)
15:36:15 <lajoskatona> exactly
15:36:16 <slaweq> I will skip today stable branches topic as there is no bcafarel here
15:36:34 <slaweq> and I don't think we have any new issues there, I saw bunch of patches merged in stable branches recently
15:36:46 <slaweq> finally even rocky and queens are running more or less fine :)
15:36:55 <slaweq> #topic Grafana
15:37:06 <slaweq> #link http://grafana.openstack.org/dashboard/db/neutron-failure-rate
15:37:31 <slaweq> the biggest issue for now is with fullstack
15:38:00 <slaweq> and either we will fix that privsep thing or we will need to mark some tests as unstable or make job non-voting temporary
15:38:03 <slaweq> wdyt?
15:38:17 <obondarev> slaweq: https://review.opendev.org/c/openstack/neutron/+/795300
15:38:22 <obondarev> no?
15:38:36 <slaweq> right
15:38:43 <slaweq> we talked about it few minutes ago :)
15:38:46 <ralonsoh> yes, I prefer not to mark all the job as non-voting
15:38:49 <slaweq> I hope this will make it better
15:39:01 <slaweq> ralonsoh me neighter
15:39:35 <slaweq> ok, let's see how it will be with that patch
15:39:59 <slaweq> anything else regarding grafana for today?
15:41:56 <slaweq> ok, I guess this means "no"
15:42:11 <slaweq> regarding specific jobs' failures, I only found today fullstack issue
15:42:34 <slaweq> I even reported new bug  https://bugs.launchpad.net/neutron/+bug/1931217 but now I see that this is dup of https://bugs.launchpad.net/neutron/+bug/1930401
15:42:38 <opendevmeet> Launchpad bug 1931220 in neutron "duplicate for #1931217 _ObjectChangeHandler.handle_event failing on port after_create event" [Critical,In progress] - Assigned to Mamatisa Nurmatov (isabek)
15:42:38 <ralonsoh> yes, this is the main issue
15:42:39 <opendevmeet> Launchpad bug 1930401 in neutron "Fullstack l3 agent tests failing due to timeout waiting until port is active" [Critical,Confirmed] - Assigned to Lajos Katona (lajos-katona)
15:43:15 <slaweq> so good news is that we don't have new ci issues, at least no that I would be aware of :)
15:43:39 <obondarev> only pip glitch :)
15:44:53 <slaweq> do You have anything else related to the CI You would like to discuss today?
15:45:02 <ralonsoh> no
15:45:03 <slaweq> or if not, I think we can finish earlier today
15:45:09 <opendevreview> Rodolfo Alonso proposed openstack/neutron-lib master: API "subnet-segmentid-writable" should inherit field definition  https://review.opendev.org/c/openstack/neutron-lib/+/795340
15:45:16 <ralonsoh> (sorry)
15:45:22 <slaweq> np :)
15:46:33 <slaweq> ok, thx for attending the meeting
15:46:45 <ralonsoh> bye!
15:46:48 <slaweq> have a great rest of the day and see You online
15:46:49 <slaweq> o/
15:46:52 <obondarev> o/
15:46:53 <slaweq> #endmeeting