15:02:14 <slaweq> #startmeeting neutron_ci 15:02:15 <opendevmeet> Meeting started Tue Jun 8 15:02:14 2021 UTC and is due to finish in 60 minutes. The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:02:16 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:02:18 <opendevmeet> The meeting name has been set to 'neutron_ci' 15:02:20 <slaweq> hi (again) 15:04:09 <opendevreview> Slawek Kaplonski proposed openstack/neutron-tempest-plugin master: Add new API tests for active field in the allowed_address_pairs https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/795358 15:04:15 <slaweq> anyone still here? :) 15:04:24 <lajoskatona> Hi, 15:04:54 <ralonsoh> hi 15:05:06 <slaweq> ok, let's start :) 15:05:13 <obondarev> hi 15:05:19 <slaweq> Grafana dashboard: http://grafana.openstack.org/dashboard/db/neutron-failure-rate 15:05:27 <slaweq> please open to have it ready 15:05:30 <slaweq> and we can move on 15:05:36 <slaweq> #topic Actions from previous meetings 15:06:01 <slaweq> slaweq to prepare plan of switch ovn <-> ovs jobs in neutron CI 15:06:12 <slaweq> I did such table: https://etherpad.opendev.org/p/neutron-ci-ovs-to-ovn 15:06:25 <slaweq> please check it and let me know what do You think about it 15:06:47 <ralonsoh> good stuff 15:06:48 <ralonsoh> thanks 15:07:03 <ralonsoh> what are we going to do with the naming? 15:07:55 <lajoskatona> we said last time to have the backend in the job name as I remember 15:08:07 <obondarev> neutron-rally-task should be ovs-ovs 15:08:35 <obondarev> ovn multinode seems missing 15:09:07 <slaweq> for neutron-rally-task that's good point - it can be renamed to neutron-ovs-rally-task 15:09:13 <lajoskatona> +1 15:09:16 <slaweq> that will be more consistent 15:09:45 <slaweq> and good news is that I think that we can remove completly neutron-ovn-tempest-ovs-release :) 15:09:52 <slaweq> so one job less in our CI 15:09:56 <slaweq> wdyt? 15:10:19 <ralonsoh> but this one is using master ovs, right? 15:10:30 <slaweq> no, released ovs 15:10:45 <ralonsoh> so what's the different then? 15:11:00 <lajoskatona> By the way haleyb missed jobs last time: http://lists.openstack.org/pipermail/openstack-discuss/2021-June/022904.html 15:11:45 <slaweq> ralonsoh I think we have same coverage in other jobs which by default runs ovn now 15:11:53 <slaweq> but I will check that to be sure 15:12:15 <slaweq> lajoskatona I think that this should be now fixed, right? 15:12:30 <haleyb> yes, that was reverted and i see the jobs again 15:12:37 <lajoskatona> ok 15:13:03 <slaweq> haleyb++ thx for catching that 15:13:37 <haleyb> zuul passing in 30 minutes was suspicious :-o 15:13:42 <lajoskatona> neutron-ovn-tempest-slow & neutron-tempest-slow-py3 can't we keep them and rename to show which one is ovn and ovs? 15:14:00 <ralonsoh> yeah 15:14:21 <slaweq> lajoskatona yes, good point 15:16:08 <slaweq> so basically we will have pairs like *-ovs and *ovn jobs 15:16:23 <slaweq> and it will be for rally, tempest-slow, and ipv6-only jobs 15:17:37 <slaweq> anyone wants to push patches for that? 15:18:11 <ralonsoh> I can do it 15:19:15 <slaweq> thx ralonsoh 15:20:16 <slaweq> #action ralonsoh to update our ci job's definitions 15:20:22 <slaweq> ok, let's move on 15:20:30 <slaweq> ralonsoh to propose tempest patch to use resolvectl to address https://bugs.launchpad.net/neutron/+bug/1929523 15:20:31 <opendevmeet> Launchpad bug 1929523 in neutron "Test tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_subnet_details is failing from time to time" [High,Confirmed] 15:20:43 <ralonsoh> one sec 15:20:54 <ralonsoh> https://review.opendev.org/c/openstack/tempest/+/794776 15:21:40 <slaweq> thx ralonsoh 15:21:54 <slaweq> I will review it later 15:22:06 <slaweq> ok, next one 15:22:07 <slaweq> slaweq to reach out to boden about payload patches and x/vmware-nsx 15:22:25 <slaweq> I tried to contact Adit from Vmware but he didn't reply to me 15:22:48 <slaweq> but I see that Salvadore approved that patch 15:22:51 <slaweq> so it's merged now 15:23:02 <slaweq> lajoskatona I think we are good with neutron patch now, right? 15:23:22 <lajoskatona> cool, than we didn't break their ci 15:23:45 <slaweq> yeah :) 15:23:53 <slaweq> ok, next one 15:23:56 <slaweq> ralonsoh to check failing neutron-lib-from-master periodic job 15:24:09 <ralonsoh> I didn't check anything 15:24:23 <obondarev> I think I saw a patch 15:24:29 <ralonsoh> really? 15:24:32 <slaweq> yeah, I saw it too :) 15:24:36 <obondarev> yes :) 15:25:54 <slaweq> https://review.opendev.org/c/openstack/neutron/+/794236 15:25:58 <slaweq> that one I think 15:25:59 <slaweq> no? 15:26:25 <ralonsoh> ahhh yes 15:26:34 <slaweq> LOL 15:26:35 <ralonsoh> pffff my brain if melting 15:26:58 <slaweq> ralonsoh TooManyPatchesException() :P 15:27:26 <obondarev> :D 15:27:46 <slaweq> so thx for fixing that issue ralonsoh :) 15:27:49 <slaweq> and let's move on 15:27:51 <slaweq> lajoskatona to check fullstack failures https://bugs.launchpad.net/neutron/+bug/1930401 15:27:53 <opendevmeet> Launchpad bug 1930401 in neutron "Fullstack l3 agent tests failing due to timeout waiting until port is active" [Critical,Confirmed] - Assigned to Lajos Katona (lajos-katona) 15:28:14 <lajoskatona> that is some ugly privsep issue 15:28:35 <slaweq> ahh, that one 15:28:36 <lajoskatona> I pushed one patch, but with ralonsoh and obondarev we agreed that is not enough 15:28:41 <slaweq> the famous one :) 15:28:50 <ralonsoh> I agree with the idea of having two decorators 15:28:55 <ralonsoh> one with timeout 15:29:12 <ralonsoh> and we can also add a default timeout in the context definition 15:29:28 <obondarev> +1 15:29:32 <lajoskatona> yeah I started to check that and can produce terrible failures in my locan env already 15:29:49 <lajoskatona> so it's ongoing 15:30:12 <obondarev> meanwhile workaround is almost there: https://review.opendev.org/c/openstack/neutron/+/795300 15:30:28 <lajoskatona> this privsep looks like if I touch somewhere I have a break somewhere else and can't see the connection :-) 15:30:29 <obondarev> lajoskatona: please add workaround revert in your patch 15:30:38 <lajoskatona> obondarev: sure 15:31:25 <slaweq> thx lajoskatona and obondarev for working on that issue 15:31:32 <slaweq> it is really killing our CI now 15:31:47 <obondarev> to ralonsoh as well 15:31:54 <slaweq> ahh, right :) 15:32:05 <slaweq> sorry ralonsoh that I missed You 15:32:12 <ralonsoh> hehehe 15:32:30 <slaweq> ok, last one from previous week 15:32:33 <slaweq> lajoskatona to mark failing TestUninterruptedConnectivityOnL2AgentRestart fullstack tests as unstable temporary 15:32:59 <lajoskatona> https://review.opendev.org/c/openstack/neutron/+/794228 15:33:24 <lajoskatona> ohhh, it's more red than green now.... 15:34:30 <lajoskatona> looks like some glitch in pip mirroring perhaps 15:34:35 <slaweq> thx lajoskatona 15:34:50 <slaweq> and thx obondarev and ralonsoh for reviewing it :) 15:34:56 <ralonsoh> yw 15:35:05 <slaweq> ok, let's move on to the other topics 15:35:05 <obondarev> sure 15:35:09 <slaweq> #topic Stadium projects 15:35:16 <slaweq> lajoskatona any issues with stadium's ci? 15:35:33 <lajoskatona> nothing as I know 15:35:48 <slaweq> no news is good news in this case :) 15:36:15 <lajoskatona> exactly 15:36:16 <slaweq> I will skip today stable branches topic as there is no bcafarel here 15:36:34 <slaweq> and I don't think we have any new issues there, I saw bunch of patches merged in stable branches recently 15:36:46 <slaweq> finally even rocky and queens are running more or less fine :) 15:36:55 <slaweq> #topic Grafana 15:37:06 <slaweq> #link http://grafana.openstack.org/dashboard/db/neutron-failure-rate 15:37:31 <slaweq> the biggest issue for now is with fullstack 15:38:00 <slaweq> and either we will fix that privsep thing or we will need to mark some tests as unstable or make job non-voting temporary 15:38:03 <slaweq> wdyt? 15:38:17 <obondarev> slaweq: https://review.opendev.org/c/openstack/neutron/+/795300 15:38:22 <obondarev> no? 15:38:36 <slaweq> right 15:38:43 <slaweq> we talked about it few minutes ago :) 15:38:46 <ralonsoh> yes, I prefer not to mark all the job as non-voting 15:38:49 <slaweq> I hope this will make it better 15:39:01 <slaweq> ralonsoh me neighter 15:39:35 <slaweq> ok, let's see how it will be with that patch 15:39:59 <slaweq> anything else regarding grafana for today? 15:41:56 <slaweq> ok, I guess this means "no" 15:42:11 <slaweq> regarding specific jobs' failures, I only found today fullstack issue 15:42:34 <slaweq> I even reported new bug https://bugs.launchpad.net/neutron/+bug/1931217 but now I see that this is dup of https://bugs.launchpad.net/neutron/+bug/1930401 15:42:38 <opendevmeet> Launchpad bug 1931220 in neutron "duplicate for #1931217 _ObjectChangeHandler.handle_event failing on port after_create event" [Critical,In progress] - Assigned to Mamatisa Nurmatov (isabek) 15:42:38 <ralonsoh> yes, this is the main issue 15:42:39 <opendevmeet> Launchpad bug 1930401 in neutron "Fullstack l3 agent tests failing due to timeout waiting until port is active" [Critical,Confirmed] - Assigned to Lajos Katona (lajos-katona) 15:43:15 <slaweq> so good news is that we don't have new ci issues, at least no that I would be aware of :) 15:43:39 <obondarev> only pip glitch :) 15:44:53 <slaweq> do You have anything else related to the CI You would like to discuss today? 15:45:02 <ralonsoh> no 15:45:03 <slaweq> or if not, I think we can finish earlier today 15:45:09 <opendevreview> Rodolfo Alonso proposed openstack/neutron-lib master: API "subnet-segmentid-writable" should inherit field definition https://review.opendev.org/c/openstack/neutron-lib/+/795340 15:45:16 <ralonsoh> (sorry) 15:45:22 <slaweq> np :) 15:46:33 <slaweq> ok, thx for attending the meeting 15:46:45 <ralonsoh> bye! 15:46:48 <slaweq> have a great rest of the day and see You online 15:46:49 <slaweq> o/ 15:46:52 <obondarev> o/ 15:46:53 <slaweq> #endmeeting