15:01:49 <slaweq> #startmeeting neutron_ci
15:01:49 <opendevmeet> Meeting started Tue Jan 25 15:01:49 2022 UTC and is due to finish in 60 minutes.  The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:49 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:01:49 <opendevmeet> The meeting name has been set to 'neutron_ci'
15:01:53 <mlavalle> o/
15:01:54 <ralonsoh> hi
15:01:57 <slaweq> o/
15:02:10 <mlavalle> LOL, it was a one week long CI meeting
15:02:39 <slaweq> yeah :)
15:03:07 <slaweq> so lets do this one quick as we had last one very long :D
15:03:09 <slaweq> Grafana dashboard: http://grafana.openstack.org/dashboard/db/neutron-failure-rate
15:03:16 <mlavalle> yeap
15:03:19 <slaweq> #topic Actions from previous meetings
15:03:24 <slaweq> lajoskatona to check if we can use neutron from master in the networking-bgpvpn master branch jobs
15:03:29 <mlavalle> you must be exhausted
15:03:38 <slaweq> LOL
15:03:40 <slaweq> yeah
15:03:40 <bcafarel> o /
15:04:01 <slaweq> I think lajoskatona is not available now, so I will keep this action item for him for next week
15:04:06 <ykarel> o/
15:04:08 <mlavalle> he is not
15:04:10 <slaweq> #action lajoskatona to check if we can use neutron from master in the networking-bgpvpn master branch jobs
15:04:29 <slaweq> and that was the only AI from last week for today
15:04:34 <slaweq> #topic Stable branches
15:04:56 <slaweq> bcafarel: any updates about stable branches CI?
15:05:37 <bcafarel> with xena fix merged a few hours ago (looking for link) we are good from my last checks
15:05:58 <bcafarel> ah https://review.opendev.org/c/openstack/neutron/+/817529
15:06:39 <ykarel> for neutron-lib stable branches i see ci jobs have issues
15:06:47 <ykarel> so pushed https://review.opendev.org/c/openstack/neutron-lib/+/826266 for wallaby
15:07:16 <slaweq> thx ykarel
15:07:17 <ykarel> for victoria/ussuri also pushed patches but those need some updates
15:07:32 <slaweq> yes, we usually don't push patches to stable branches in neutron-lib so we could not know about it
15:08:04 <ykarel> yeap i too noticed when i pushed a backport :)
15:08:13 <bcafarel> :)
15:08:25 <slaweq> but why this is only for wallaby?
15:08:36 <slaweq> isn't something similar needed also for newer stable branches?
15:08:49 <slaweq> like change to use proper neutron-tempest-plugin-api job there
15:08:50 <ykarel> actually changes are wallaby specific
15:08:55 <ykarel> like job name contains -wallaby
15:09:13 <slaweq> ok, but will You push similar changes to other branches too?
15:09:17 <ykarel> yes
15:09:37 <slaweq> ok, thx a lot
15:10:58 <slaweq> so we can move on to the next topic
15:11:03 <slaweq> #topic Stadium projects
15:11:18 <slaweq> lajoskatona: is not here so probably we will not have a lot of updates here
15:11:28 <slaweq> we already know about neutron-lib issues in stable branch
15:11:37 <slaweq> do You know about any other issues with stadium CI?
15:12:11 <bcafarel> not that I saw at least
15:12:20 <slaweq> ok, good
15:12:24 <slaweq> so I think we can move on :)
15:12:29 <slaweq> #topic Grafana
15:12:36 <slaweq> #link http://grafana.openstack.org/dashboard/db/neutron-failure-rate
15:13:06 <slaweq> IMO numbers looks pretty good this week
15:13:10 <ralonsoh> +1
15:13:21 <slaweq> the only job on a bit high failure rate is functional tests job
15:13:28 <gmann> sorry missed the topic for stable branch CI. can i just update a quick think for stable/train testing ?
15:13:29 <slaweq> but it's also not very bad
15:13:39 <slaweq> gmann: sure
15:13:54 <gmann> as stable/train is in EM, I am capping it with older tempest (26.1.0).
15:14:24 <gmann> with some workaround I am able to run it successfully. I have tested neutron (master, stable/train, neutron-tempest-plugin) along with other projects
15:14:27 <gmann> #link https://review.opendev.org/q/topic:%22train-tempest-cap%22+(status:open%20OR%20status:merged)
15:14:51 <gmann> we might be merging devstack and tempest change soon, so if you encounter any issue please ping me anytime
15:15:06 <gmann> also tested neutron stable/ussuri grenade job
15:15:33 <slaweq> https://0735506f8023839b161e-58818f8d6a88f31ae0841c158a7f2252.ssl.cf2.rackcdn.com/816597/6/check/networking-ovn-tempest-dsvm-ovs-release/beb12c3/job-output.txt
15:15:36 <gmann> or anything more to be tested before we merge the change
15:15:44 <slaweq> I think we may need to update also networking-ovn job there
15:16:32 <gmann> slaweq: ah, sorry missed that. I will check after meeting
15:16:40 <slaweq> thx gmann
15:16:49 <slaweq> if You will need anything, please ping me
15:17:07 <gmann> sure
15:17:22 <gmann> I will debug today and update you on this
15:17:23 <gmann> that is all for me
15:17:29 <gmann> from me for meeting
15:17:37 <slaweq> now, number of rechecks to get patch merged
15:17:39 <slaweq> +---------+----------+... (full message at https://matrix.org/_matrix/media/r0/download/matrix.org/hXiNdGcZOfaENfXPQmyTmtJw)
15:17:46 <slaweq> it looks really good in last weeks
15:17:48 <slaweq> :)
15:20:04 <slaweq> if there is nothing else regarding grafana, I think we can move on
15:20:10 <slaweq> #topic fullstack/functional
15:20:25 <slaweq> I found some issues in functional jobs
15:20:31 <slaweq> neutron.tests.functional.plugins.ml2.drivers.ovn.mech_driver.test_mech_driver.TestAgentApi.test_agent_list
15:20:36 <slaweq> https://12d9c9014a31cd38106f-d99046410a2db92aeb18e96327c94fc0.ssl.cf1.rackcdn.com/825513/1/gate/neutron-functional-with-uwsgi/8857dcc/testr_results.html
15:20:44 <ralonsoh> this is still under development
15:20:51 <ralonsoh> there is a patch to use the Neutron DB
15:20:55 <ralonsoh> instead of a local cache
15:21:11 <slaweq> so it's known issue, right?
15:21:16 <ralonsoh> yes
15:21:26 <slaweq> great, one less :)
15:21:28 <slaweq> thx ralonsoh
15:21:58 <slaweq> next one is something what I saw most often this week
15:22:06 <slaweq> assertion errors in nb_gobal_event.wait()
15:22:10 <slaweq> https://zuul.opendev.org/t/openstack/build/bece91989f6b4436be91557509e396b2... (full message at https://matrix.org/_matrix/media/r0/download/matrix.org/QMzBsFrbvDRHnipkWXYFAviC)
15:22:22 <slaweq> did You saw it already maybe?
15:22:42 <ralonsoh> yes and lajoskatona pushed a patch for this
15:22:43 <ralonsoh> one sec
15:23:03 <ralonsoh> https://review.opendev.org/c/openstack/neutron/+/825530
15:23:13 <slaweq> ahh, ok
15:23:16 <slaweq> I saw that patch today
15:23:38 <slaweq> but it is also failing on functional tests and it seems that it's related
15:24:01 <slaweq> next one
15:24:04 <slaweq> network interface not found in namespace
15:24:08 <slaweq> https://zuul.opendev.org/t/openstack/build/002e6bbc4a5a4cd698db8beaa77e971a
15:24:08 <slaweq> https://zuul.opendev.org/t/openstack/build/de9d16f06d2a477ab2a355268c0fcf3d
15:24:31 <slaweq> this one happens from time to time
15:24:37 <slaweq> I think I saw it already earlier
15:24:51 <ralonsoh> https://bcf5e84ce85dfd740e48-c3fbbb652718002c010964532c238f5b.ssl.cf2.rackcdn.com/825428/5/check/neutron-functional-with-uwsgi/de9d16f/testr_results.html
15:25:00 <ralonsoh> is this one related to the problem of the "shy" port?
15:25:17 <ralonsoh> no, that was a problem in Train
15:25:24 <ralonsoh> and you pushed the fix
15:25:40 <mlavalle> lol, "el puerto penoso"
15:25:52 <ralonsoh> timido
15:26:06 <mlavalle> +1
15:27:10 <slaweq> but I think this may be something similar to what we saw in one of the d/s issues
15:27:15 <slaweq> with dhcp port
15:27:29 <slaweq> that ovs transaction was aborted and port wasn't created
15:27:35 <ralonsoh> yes, that's the bug I was talking about
15:27:35 <slaweq> I will check that deeper
15:27:41 <ralonsoh> ++
15:27:59 <slaweq> #action slaweq to check missing devices in the namespace, like in https://11881b08dc5457b2ffc3-34db3bac40a43d535a8bfaf3904b1b2c.ssl.cf5.rackcdn.com/821208/8/check/neutron-functional-with-uwsgi/002e6bb/testr_results.html
15:28:16 <slaweq> and the last one
15:28:18 <slaweq> neutron.tests.functional.agent.test_dhcp_agent.DHCPAgentOVSTestCaseOwnBridge.test_good_address_allocation
15:28:22 <slaweq> https://49a71c27f1891b6ef61e-c80779e78a1a20b721125eb9333742c5.ssl.cf2.rackcdn.com/825428/4/check/neutron-functional-with-uwsgi/7ca0cf1/testr_results.html
15:28:41 <slaweq> this one I saw only once
15:29:07 <mlavalle> so just keep an eye on it?
15:29:11 <slaweq> but I really don't think it could be related to the patch on which it was run
15:29:32 <slaweq> mlavalle: yes, if nobody saw it before, I think it will be good idea
15:29:41 <mlavalle> +1
15:30:31 <opendevreview> Frode Nordahl proposed openstack/neutron master: [OVN] Add unit test for binding profile validation  https://review.opendev.org/c/openstack/neutron/+/826099
15:30:31 <opendevreview> Frode Nordahl proposed openstack/neutron master: [OVN] Extend port binding parameter validation  https://review.opendev.org/c/openstack/neutron/+/818420
15:30:32 <opendevreview> Frode Nordahl proposed openstack/neutron master: WIP ovn: Off-path SmartNIC DPU Port Binding with OVN  https://review.opendev.org/c/openstack/neutron/+/808961
15:31:32 <slaweq> ok, so that are all functional jobs issues from me for today
15:31:39 <obondarev> https://bugs.launchpad.net/neutron/+bug/1955008 - saw it couple of times
15:31:47 <obondarev> sorry for being late
15:32:07 <slaweq> obondarev: ahh, yes
15:32:07 <obondarev> https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f76/825521/3/gate/neutron-functional-with-uwsgi/f76cef6/testr_results.html
15:32:11 <slaweq> that one I saw too but didn't mention it here as it is reported already :)
15:32:19 <obondarev> ah
15:32:21 <obondarev> right
15:32:37 <slaweq> I have it on my todo list already
15:32:44 <obondarev> just raised here as it has no assignee
15:32:46 <slaweq> so I will try to investigate it
15:32:55 <obondarev> cool :)
15:32:58 <slaweq> obondarev: thx for reminding about it
15:33:11 <slaweq> #action slaweq to check bug https://bugs.launchpad.net/neutron/+bug/1955008
15:33:31 <slaweq> ok, next topic
15:33:33 <slaweq> #topic Tempest/Scenario
15:34:01 <slaweq> I think ykarel added one issue there:
15:34:03 <slaweq> https://cfaa2d1e4f6a936642aa-ae5561c9d080274a217713c4553af257.ssl.cf5.rackcdn.com/824022/2/check/neutron-tempest-plugin-scenario-openvswitch-wallaby/a7c128e/testr_results.html
15:34:07 <slaweq> https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_803/824022/2/check/neutron-tempest-plugin-scenario-openvswitch-iptables_hybrid-wallaby/803c276/testr_results.html
15:34:17 <ykarel> yes i noticed ^ in a backport patch
15:34:45 <ykarel> not sure if similar issue is known
15:34:52 <ykarel> also i couldn't trace why those failed
15:35:43 <slaweq> hmm
15:35:56 <slaweq> in both cases it seems that metadata was configured properly on instances
15:36:32 <slaweq> so connection from vm to router worked fine
15:37:19 <ykarel> yes
15:39:27 <slaweq> I'm looking at https://cfaa2d1e4f6a936642aa-ae5561c9d080274a217713c4553af257.ssl.cf5.rackcdn.com/824022/2/check/neutron-tempest-plugin-scenario-openvswitch-wallaby/a7c128e/testr_results.html
15:39:49 <slaweq> and if You look for the IP address 172.24.5.205 You will find IMO strange entry in the arp entries:
15:39:52 <slaweq> 172.24.5.205     0x1         0x0         00:00:00:00:00:00     *        br-ex
15:40:09 <slaweq> for all other FIPs it is properly set to MAC of some qg- interface
15:40:20 <ykarel> yes correct, what can cause that?
15:40:21 <slaweq> do You know why it's 00:00:00:00:00:00
15:40:25 <ykarel> nope
15:40:27 <slaweq> maybe that causes problem?
15:40:51 <ykarel> possibly yes
15:42:37 <slaweq> ralonsoh: can You take a look at it maybe?
15:42:56 <ralonsoh> I can, yes, probably at the end of this week
15:43:04 <slaweq> thx a lot
15:43:24 <slaweq> #action ralonsoh to check ssh failures and arp entry with 00:00:00:00:00:00
15:43:46 <slaweq> thx ykarel for reporting this here
15:43:49 <slaweq> last one for today
15:43:55 <slaweq> https://278f4cf4e040bdd4e722-74c6967b9c388ca77dd0a83c1c662326.ssl.cf5.rackcdn.com/824378/3/check/neutron-tempest-plugin-scenario-linuxbridge/3b878c4/testr_results.html
15:44:03 <slaweq> it's again bug https://bugs.launchpad.net/neutron/+bug/1945283
15:44:23 <slaweq> someone will need to take a look at it
15:44:26 <slaweq> any volunteer?
15:44:28 <slaweq> :)
15:44:34 <mlavalle> chiI'll take it
15:44:55 <slaweq> chil'll ? :D
15:45:02 <mlavalle> I'll take it
15:45:07 <mlavalle> :-)
15:45:08 <slaweq> :)
15:45:10 <slaweq> thx mlavalle
15:45:21 <slaweq> #action mlavalle to investigate https://bugs.launchpad.net/neutron/+bug/1945283
15:45:43 <slaweq> and that's all what I had for today
15:45:51 <slaweq> periodic jobs are all green last few days
15:45:54 <slaweq> so it's very good
15:46:21 <slaweq> do You have anything else related to CI to discuss today?
15:47:02 <ralonsoh> nothing from me
15:47:05 <slaweq> if not, I have one last thing
15:47:21 <slaweq> next week I will be busy with some internal meetings so I would like to cancel CI meeting
15:47:38 <slaweq> and in 2 weeks I should be on holidays so also I will not be able to chair it
15:47:52 <ralonsoh> we can skip next week meeting
15:47:58 <ralonsoh> I can chair it in 2 weeks
15:48:05 <mlavalle> +1
15:48:19 <slaweq> will that be ok for You if we will cancel next 2 CI meetings and get back it it on 15.02 ?
15:48:28 <slaweq> ralonsoh: ok, thx a lot
15:48:41 <slaweq> so I will cancel meeting next week and You will do it in 2 weeks
15:48:42 <slaweq> thx a lot
15:48:43 <mlavalle> I think it is better if ralonsoh CHAIRS IN 2 WEEKS
15:48:51 <ralonsoh> perfect then
15:49:04 <slaweq> thx for attending the meeting today
15:49:12 <slaweq> have a great day and see You online
15:49:18 <mlavalle> CI is mischivious. Likes to give us surprises when we don't mind it
15:49:19 <slaweq> #endmeeting