15:00:27 <slaweq> #startmeeting neutron_ci
15:00:27 <opendevmeet> Meeting started Tue Oct  5 15:00:27 2021 UTC and is due to finish in 60 minutes.  The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:27 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:27 <opendevmeet> The meeting name has been set to 'neutron_ci'
15:00:35 <ralonsoh> hi
15:00:42 <slaweq> hi
15:02:15 <slaweq> let's wait few more minutes for others to join and we will start
15:03:01 <lajoskatona> Hi
15:03:51 <slaweq> Grafana dashboard: http://grafana.openstack.org/dashboard/db/neutron-failure-rate
15:03:56 <bcafarel> o/ (sorry was wrapping up a downstream meeting)
15:04:14 <slaweq> #topic Actions from previous meetings
15:04:24 <slaweq> slaweq to check vpnaas stable/train patch https://review.opendev.org/c/openstack/neutron-vpnaas/+/805969/10
15:04:37 <slaweq> this is done, with huge help from gmann :)
15:04:51 <bcafarel> yes gmann++ indeed
15:05:08 <bcafarel> that passing CI job made me smile :)
15:05:17 <gmann> np!
15:05:22 <slaweq> we missed to define tempest_plugins there :)
15:05:35 <slaweq> next one
15:05:37 <slaweq> alonsoh to check functional tests issue with router's state transition
15:06:27 <ralonsoh> yes, I have a patch, one sec
15:06:46 <ralonsoh> #link https://review.opendev.org/c/openstack/neutron/+/811751
15:08:12 <slaweq> thx ralonsoh
15:08:25 <slaweq> ok, next one
15:08:27 <slaweq> slaweq to check api extensions list in ovs based jobs, how it was generated
15:08:35 <slaweq> I didn't check it but I think that gibi and gmann found the solution for that issue already
15:09:11 <lajoskatona> I added the link to that checklist to our checklist for relase
15:10:12 <lajoskatona> https://review.opendev.org/c/openstack/neutron/+/812112
15:10:47 <slaweq> thx lajoskatona
15:11:01 <slaweq> and that are all actions from last week
15:11:07 <slaweq> #topic Stadium projects
15:11:11 <slaweq> any updates?
15:11:24 <lajoskatona> all is green, except vpnaas
15:11:41 <lajoskatona> fix is here: https://review.opendev.org/c/openstack/neutron-vpnaas/+/811731
15:12:33 <bcafarel> autumn cleanup time there I guess
15:12:35 <lajoskatona> some news for vpnaas (half stable topic): p, q, r, s branches are deleted so no more failures from periodic jobs
15:12:58 <slaweq> sorry that I missed that one
15:13:01 <bcafarel> +train and newer fixed :)
15:13:08 <ralonsoh> cool!
15:13:32 <lajoskatona> and I have a question: for bgpvpn theres patches for centos8 : https://review.opendev.org/q/owner:bshewale%2540redhat.com+project:%2522%255Eopenstack/networking.*%2522+intopic:%2522%255Ec7-to-c8.*%2522
15:14:24 <lajoskatona> with redhat hat (:P) could you help me if those are ok?
15:14:51 <slaweq> LOL
15:14:53 <slaweq> sure
15:15:24 <bcafarel> interesting, added to the list
15:15:33 <slaweq> I added it for my tomorrow's todo list
15:15:46 <lajoskatona> thanks
15:16:26 <lajoskatona> that's it for stadium from me
15:16:34 <slaweq> thx lajoskatona
15:16:41 <slaweq> I think we can move on
15:16:46 <slaweq> #topic Stable branches
15:17:06 <bcafarel> I have a few pending backports to check but overall all good
15:17:15 <bcafarel> we got the needed fixes in xena just in time for rc2
15:18:31 <ralonsoh> +1 to this
15:19:35 <slaweq> yeah, that were "last minute" patches
15:19:49 <ralonsoh> I don't know what you are talking about...
15:19:55 <bcafarel> :)
15:19:58 <slaweq> btw. speaking about xena
15:19:58 <bcafarel> ralonsoh++
15:20:04 <slaweq> I proposed https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/811003 some time ago
15:20:09 <slaweq> please review it
15:20:13 <ralonsoh> suer
15:20:15 <ralonsoh> sure
15:21:16 <bcafarel> thanks yes time to add xena to the list
15:22:15 <slaweq> ok, I think we can move on to the next topic then
15:22:17 <slaweq> #topic Grafana
15:24:33 <slaweq> I don't see anything critically wrong there
15:24:38 <lajoskatona> it seems that after the release week the gods of CI are more merciful
15:24:45 <slaweq> :D
15:24:51 <slaweq> well said lajoskatona :)
15:25:28 <slaweq> there was some short spike yesterday but it was on most of the jobs (or all even) so I think it was something outside neutron
15:25:33 <slaweq> and it is gone already
15:26:47 <slaweq> lets talk about some specific jobs and issues there :)
15:27:00 <slaweq> #topic fullstack/functional
15:27:21 <slaweq> I found one fullstack failure:
15:27:26 <slaweq> https://8d5ef598bba78b1573a4-7dfe055f87ad090ed1b50745545f409a.ssl.cf1.rackcdn.com/805391/10/check/neutron-fullstack-with-uwsgi/6e03086/testr_results.html
15:27:38 <slaweq> it was failed neutron.tests.fullstack.test_l3_agent.TestHAL3Agent.test_router_fip_qos_after_admin_state_down_up test
15:27:51 <slaweq> and I saw in logs "Network unreachable" errors
15:28:04 <slaweq> so IMO this may be some issue with test itself, not in the Neutron code really
15:28:17 <slaweq> I will report bug for that
15:28:29 <slaweq> but I don't think I will have time to check it really this week
15:28:55 <slaweq> #action slaweq to report fullstack issue with neutron.tests.fullstack.test_l3_agent.TestHAL3Agent.test_router_fip_qos_after_admin_state_down_up test
15:30:26 <slaweq> #topic Tempest/Scenario
15:30:31 <slaweq> here I saw something interesting
15:30:57 <slaweq> at least 3 times scenario jobs failed with errors like e.g. in: https://6599da62140c9583e14a-cd7f53ffbb0b86c69deae453da021fe8.ssl.cf5.rackcdn.com/811746/4/check/neutron-tempest-plugin-scenario-openvswitch/3cafcd7/testr_results.html
15:31:13 <slaweq> basically all HA routers were in backup state but non of them were transitioned to master
15:31:38 <slaweq> I wonder if that could be the same issue as the one which ralonsoh fixed with https://review.opendev.org/c/openstack/neutron/+/811751
15:31:42 <ralonsoh> I think so
15:31:47 <ralonsoh> I was checking that
15:32:02 <ralonsoh> because are those routers created only for those tests?
15:32:14 <slaweq> yes, each test is creating router
15:32:25 <slaweq> and deleting it on cleanup
15:32:29 <ralonsoh> if so, they initial "primary" state could be delayed unnecessarily
15:33:38 <slaweq> but can this delay actually cause the issue that it will never later be switched?
15:33:42 <slaweq> to be primary
15:34:05 <ralonsoh> no, this delay is by design
15:34:18 <ralonsoh> to avoid histeresis during the transitions to master
15:34:24 <ralonsoh> *primary
15:34:54 <slaweq> so that is likely different issue probably
15:35:01 <ralonsoh> could be, yes
15:35:10 <slaweq> as in this case routers aren't switched to primary at all
15:35:25 <slaweq> I will report bug and will try to investigate in the logs
15:35:39 <slaweq> but help is welcome with that one :)
15:35:54 <ralonsoh> sure, I'll check it once we have the other patch merged
15:36:01 <slaweq> #action slaweq to report bug regarding ha routers not going to be primary never
15:36:06 <slaweq> ralonsoh++ thx
15:36:31 <slaweq> ok, next one
15:36:38 <slaweq> we still have some issues with ovs agent crashing, like https://04824dc10f811bf71cc7-f60cbd2bdbb8b5648c0b0982a5f4272f.ssl.cf1.rackcdn.com/805391/10/check/neutron-ovs-tempest-slow/cd391db/compute1/logs/screen-q-agt.txt
15:36:49 <slaweq> but I hope that recent backports to os-ken will solve that issue
15:37:11 <slaweq> the bad thing is that we need to have new os-ken release for that
15:37:25 <ralonsoh> yes but we need the new branches in "releases"
15:37:29 <slaweq> and releases repo isn't ready for yoga yet
15:37:31 <ralonsoh> and then create a new tag there
15:37:32 <ralonsoh> nope
15:38:35 <lajoskatona> side note: I tried to reach directly some ryu developers (most active ones in last months) to have some discussion with them
15:40:07 <ralonsoh> can we add a topic in Neutron meetings? to track weekly any change in ryu that should be backported to os-ken
15:40:20 <ralonsoh> that could take 5 secs (if nothing is merged)
15:40:31 <ralonsoh> or a couple of links to be reviewed
15:40:35 <ralonsoh> I can take care of it
15:40:43 <slaweq> sounds good
15:41:01 <lajoskatona> sure, that;s a good idea, to follow ehat happens there
15:42:54 <slaweq> maybe we can also propose some simple script in https://github.com/openstack/neutron/tree/master/tools to list such changes
15:43:09 <slaweq> and maybe even to backport such patches to os-ken
15:43:11 <slaweq> wdyt?
15:43:14 <lajoskatona> I can check it before the meeting and we can triage if there's anything during the meeting to include or not
15:43:40 <lajoskatona> yeah, that's a good idea
15:43:41 <ralonsoh> the first triage will be the worst, I'll check on friday what we should migrate
15:43:52 <ralonsoh> and I'll propose the needed patches
15:43:56 <lajoskatona> as I see not much activity (weekly 1-2 patches) in ryu
15:44:16 <slaweq> ok, thx ralonsoh for that
15:44:34 <slaweq> #action ralonsoh will come up with list of ryu patches which we should backport to os-ken
15:44:53 <slaweq> #action lajoskatona will add ryu - os-ken sync topic to the neutron weekly meeting's agenda
15:45:01 <lajoskatona> +1
15:45:01 <slaweq> is that ok for You ^^ ?
15:45:42 <lajoskatona> yes it's ok for me
15:45:49 <ralonsoh> perfect for me too
15:45:56 <slaweq> thx
15:46:13 <slaweq> that was the last topic from me for today
15:46:21 <slaweq> btw. periodic jobs are fine - even fedora one :)
15:46:25 <slaweq> thx ralonsoh
15:46:44 <bcafarel> nice CI status for release week then
15:46:55 <lajoskatona> yeah suprisingly few failures this week, and we can get rid of old vpnaas jobs now
15:47:05 <slaweq> yeah, I just need to check tobiko job there but that's not urgent
15:47:17 <ralonsoh> I don't know how the error in the compute test is passing now in fedora
15:47:30 <slaweq> me neighter but it's green :)
15:47:37 <ralonsoh> I'll close the bug (half of it)
15:47:43 <slaweq> so I don't want to worry about it
15:47:46 <slaweq> thx ralonsoh :)
15:47:56 <slaweq> ok, anything else You want to discuss today?
15:48:07 <bcafarel> all good here
15:49:02 <lajoskatona> nothing from me
15:49:12 <ralonsoh> I'm ok
15:49:45 <slaweq> so thx for attending the meeting
15:49:49 <slaweq> and have a great evening
15:49:51 <slaweq> o/
15:49:53 <slaweq> #endmeeting