15:04:04 <slaweq> #startmeeting neutron_ci
15:04:04 <opendevmeet> Meeting started Tue Oct 12 15:04:04 2021 UTC and is due to finish in 60 minutes.  The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:04:04 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:04:04 <opendevmeet> The meeting name has been set to 'neutron_ci'
15:04:06 <slaweq> hi
15:04:15 <slaweq> and sorry for being a bit late with start :)
15:04:54 <lajoskatona> Hi
15:04:58 <slaweq> Grafana dashboard: http://grafana.openstack.org/dashboard/db/neutron-failure-rate
15:05:50 <slaweq> #topic Actions from previous meetings
15:05:57 <slaweq> slaweq to report fullstack issue with neutron.tests.fullstack.test_l3_agent.TestHAL3Agent.test_router_fip_qos_after_admin_state_down_up test
15:06:04 <slaweq> I did: https://bugs.launchpad.net/neutron/+bug/1946186
15:06:15 <slaweq> I also checked it
15:06:21 <bcafarel> o/
15:06:27 <slaweq> and it seems for me that this is still same privsep issue in the dhcp agent
15:06:52 <slaweq> so I proposed patch https://review.opendev.org/c/openstack/neutron/+/813128/
15:07:00 <slaweq> which I hope will make it more stable
15:07:09 <slaweq> please check that patch when You will have some time
15:07:36 <lajoskatona> slaweq: good idea to avoid dhcp agent when not needed
15:07:45 <slaweq> thx
15:07:53 <slaweq> ok, next one
15:07:57 <slaweq> slaweq to report bug regarding ha routers not going to be primary never
15:08:06 <slaweq> https://bugs.launchpad.net/neutron/+bug/1946187
15:08:15 <slaweq> I will try to check it this week
15:08:54 <slaweq> this issue happend at least couple of times in the CI already
15:09:02 <slaweq> so it's pretty urgent thing
15:09:17 <slaweq> #action slaweq to check bug https://bugs.launchpad.net/neutron/+bug/1946187
15:09:28 <slaweq> next one
15:09:31 <slaweq> ralonsoh will come up with list of ryu patches which we should backport to os-ken
15:09:42 <slaweq> ralonsoh is off today but lajoskatona did the list :)
15:09:44 <slaweq> thx lajoskatona
15:10:18 <lajoskatona> I collected things and you started to propose things :-)
15:10:33 <slaweq> I just did 2 small patches for now
15:10:54 <slaweq> which I hope will help us with the u/s ci issue when ovs-agent is crashing
15:11:14 <slaweq> but I will try to propose some of the patches from Your list to os-ken
15:11:32 <lajoskatona> slaweq: thanks
15:11:39 <slaweq> yw :)
15:11:47 <slaweq> ok, last one from previous week
15:11:49 <slaweq> lajoskatona will add ryu - os-ken sync topic to the neutron weekly meeting's agenda
15:12:01 <slaweq> which You did :)
15:12:43 <slaweq> I think we can move on to the next topic then
15:12:48 <slaweq> #topic Stadium projects
15:13:09 <lajoskatona> it's quiet
15:13:32 <lajoskatona> no issues what I know about
15:13:48 <slaweq> that's good :)
15:13:55 <slaweq> thx for taking care of it lajoskatona
15:14:03 <slaweq> #topic Stable branches
15:14:38 <bcafarel> overall good, train is back in working order after last week's failures
15:14:52 <bcafarel> stein fails on n-t-p scenario though now I filled https://bugs.launchpad.net/neutron/+bug/1946748
15:15:40 <slaweq> maybe we should use pinned version of tempest in the Stein jobs?
15:16:38 <bcafarel> I still have to dig into logs, but apparently we may have switched back to an older version of tempest?
15:17:14 <bcafarel> but this is recent failure so I am not sure yet
15:17:26 <slaweq> hmm, but should we even run test_metadata module in the Stein?
15:17:41 <slaweq> I'm not sure now but I think we introduced that later
15:17:55 <slaweq> maybe we simply started using newer n-t-p there?
15:18:41 <bcafarel> aah possible I had not considered that other way - I *think* we have override-checkout in place but worth checking
15:18:55 <slaweq> will You check it bcafarel ?
15:19:31 <bcafarel> slaweq: I hope so yes :) at least before nxet week meeting
15:19:37 <slaweq> thx
15:19:49 <slaweq> #action bcafarel to check n-t-p issue in Stein branch
15:20:30 <slaweq> speaking about stable branches, please also review https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/811003
15:20:37 <slaweq> we need jobs for xena now :)
15:21:05 <bcafarel> oh true, it is somewhere in my tabs but I forgot about it
15:21:16 <slaweq> :)
15:21:16 <bcafarel> I will take a look tomorrow
15:21:19 <slaweq> thx
15:21:26 <lajoskatona> me also
15:22:13 <slaweq> ok, I think we can move on now
15:22:22 <slaweq> #topic Grafana
15:22:53 <slaweq> https://grafana.opendev.org/d/BmiopeEMz/neutron-failure-rate?orgId=1
15:23:12 <slaweq> in general I think that it looks ok this week
15:23:17 <slaweq> at least so far
15:24:25 <slaweq> I don't see anything very dramatic on the dashboard :)
15:25:28 <lajoskatona> good start for the PTG week :-)
15:25:35 <slaweq> yeah :)
15:25:47 <slaweq> lets discuss some specific jobs' issues then
15:25:53 <bcafarel> either that, or failures are waiting for the PTG week indeed
15:25:59 <slaweq> #topic fullstack/functional
15:26:04 <slaweq> bcafarel: I hope not :)
15:26:52 <slaweq> in functional jobs I found one new failure: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_fe0/811242/7/check/neutron-functional-with-uwsgi/fe00ce9/testr_results.html
15:28:11 <slaweq> but I saw it only once so far
15:28:20 <slaweq> so IMO it can be just "on our radar" for now
15:28:32 <slaweq> and lets see if that will be failing more often
15:28:35 <slaweq> wdyt?
15:29:00 <bcafarel> sounds good yes
15:29:20 <bcafarel> let's see if it creeps up again (a few times)
15:29:26 <lajoskatona> +1
15:29:39 <slaweq> ok, next one
15:29:46 <slaweq> I mean, next topic :)
15:29:48 <slaweq> #topic Tempest/Scenario
15:29:59 <slaweq> I'm trying to optimize scenario jobs a bit https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/813195
15:30:31 <opendevreview> Lucas Alvares Gomes proposed openstack/neutron master: [OVN] Allow for tuning the use of MAC_Binding  https://review.opendev.org/c/openstack/neutron/+/813610
15:30:41 <slaweq> unfortunatelly it fails in the ovn scenario jobs
15:31:00 <slaweq> there is different kernel used in that minimal image and it has multicast support disabled
15:31:19 <slaweq> and because of that igmp snooping multicast test fails with that ubuntu minimal image
15:31:31 <slaweq> but for openvswitch and linuxbridge jobs it works fine
15:31:43 <slaweq> and with smaller flavor (256M of memory only)
15:32:01 <slaweq> so I hope it will help us a bit saving resources
15:32:11 <slaweq> please check it and tell me there wdyt about it
15:32:47 <lajoskatona> will check it
15:32:51 <slaweq> thx
15:33:07 <bcafarel> it is strange that ovs/lb would pass if the test image does not have multicast no?
15:33:30 <slaweq> bcafarel: it's like that because in ovs/lb cases there is no igmp snooping enabled
15:33:35 <bcafarel> aaah
15:33:37 <slaweq> so multicast in fact works like broadcast
15:33:48 <slaweq> and for ovn igmp snooping is enabled
15:34:07 <slaweq> and because of this different kernel, receiver vm don't subscribes to the igmp group and that's why test fails
15:34:17 <slaweq> I spent whole day yesterday trying to understand it :)
15:35:22 <bcafarel> nice job, having advanced images without gobbling all the memory in CI will be great
15:35:43 <slaweq> thx bcafarel :)
15:36:00 <slaweq> next one regarding scenario jobs
15:36:13 <slaweq> I found 2 issues with start of pmlogger.service:
15:36:18 <slaweq> https://zuul.opendev.org/t/openstack/build/869fce45f5eb421eb513023780c55a81
15:36:20 <slaweq> https://58a7b6787d8cddbd1804-9a0ca07f110a4e59baefffd341058310.ssl.cf2.rackcdn.com/802037/4/check/neutron-ovs-tempest-multinode-full/23eea89/job-output.txt
15:36:27 <slaweq> did You saw such issues too?
15:36:30 <opendevreview> Lucas Alvares Gomes proposed openstack/neutron master: [OVN] Allow for tuning the use of MAC_Binding  https://review.opendev.org/c/openstack/neutron/+/813610
15:37:19 <slaweq> maybe it is already known issue?
15:37:55 <bcafarel> I am still at the "what is pmlogger" step
15:38:09 <bcafarel> that sounds like more a generic base image issue no?
15:38:25 <slaweq> yes, it don't seems like neutron related thing
15:39:11 <opendevreview> Lucas Alvares Gomes proposed openstack/neutron master: [OVN] Allow for tuning the use of MAC_Binding  https://review.opendev.org/c/openstack/neutron/+/813610
15:39:35 <slaweq> if nobody saw it before, let's keep an eye on it and we will see next week
15:39:59 <bcafarel> +1
15:40:20 <lajoskatona> we can ask around infra if it will become more disturbing
15:40:39 <slaweq> lajoskatona++
15:41:18 <slaweq> ok, lets move on
15:41:21 <slaweq> #topic Periodic
15:41:35 <slaweq> in the periodic jobs I see only fedora job failing (again)
15:41:46 <slaweq> but that's caused by the ovn's branch rename
15:42:10 <bcafarel> ok, so faiing for a good reason then
15:43:59 <lajoskatona> for ovsdb it caused xena to fail as well, so perhaps on other nranches it will be visible
15:44:03 <slaweq> I think it should be fixed with https://review.opendev.org/c/openstack/neutron/+/812674
15:44:22 <slaweq> actually, it is
15:44:25 <slaweq> today it was green :)
15:44:40 <slaweq> so periodic jobs are ok for now
15:45:04 <slaweq> and that are all things which I had for today
15:45:12 <slaweq> do You have anything else You want to discuss today?
15:45:49 <bcafarel> nothing from me
15:46:06 <lajoskatona> neither from me
15:46:17 <bcafarel> ah though maybe just the usual questions, do we keep meetings during the PTG week?
15:46:28 <slaweq> bcafarel: good question
15:46:33 <slaweq> I will cancel it next week
15:46:40 <slaweq> thx for the reminder :)
15:47:00 <bcafarel> np, I was looking at the calendar and saw this "PTG" item :)
15:47:15 <slaweq> :)
15:47:21 <slaweq> ok, thx for the attending the meeting today
15:47:28 <slaweq> have a great week and see You online
15:47:30 <slaweq> o/
15:47:32 <slaweq> #endmeeting