15:00:20 <slaweq> #startmeeting neutron_ci
15:00:20 <opendevmeet> Meeting started Tue Jun 15 15:00:20 2021 UTC and is due to finish in 60 minutes.  The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:20 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:20 <opendevmeet> The meeting name has been set to 'neutron_ci'
15:00:33 <slaweq> hi
15:00:39 <lajoskatona> Hi
15:00:43 <ralonsoh> hi
15:01:20 <bcafarel> o/
15:02:41 <slaweq> Grafana dashboard: http://grafana.openstack.org/dashboard/db/neutron-failure-rate
15:02:41 <slaweq> Please open now :)
15:02:42 <slaweq> #topic Actions from previous meetings
15:02:42 <slaweq> ralonsoh to update our ci job's definitions
15:02:42 <slaweq> and I think we can start
15:02:54 <ralonsoh> yes, I have some questions about this
15:03:05 <ralonsoh> are we going to duplicate all OVS jobs?
15:03:15 <ralonsoh> are we going to have OVS and OVN jobs?
15:03:38 <slaweq> personally I would be happy if we could end up with the same number of jobs which we had before switch default backend
15:04:15 <slaweq> so IMO we should have now "default" jobs which I those without backend in the name to be running ovn
15:04:25 <slaweq> and change *-ovn" jobs to be "-ovs" jobs
15:04:31 <obondarev> hi
15:04:33 <ralonsoh> but then, as in https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/791255, we are mixing OVS and OVN
15:04:45 <ralonsoh> I mean, we don't have a consistent CI
15:04:55 <ralonsoh> we have mixed OVS and OVN
15:05:29 <ralonsoh> "and change *-ovn" jobs to be "-ovs" jobs" --> didn't we decided to explicitly name all jobs?
15:05:33 <ralonsoh> with the name of the backend?
15:05:58 <slaweq> yes, we talked about that
15:06:17 <ralonsoh> ok, initially I'll explicitly rename all jobs
15:06:35 <ralonsoh> then we'll see if we need to have a duplicated CI running in "periodic"
15:06:37 <slaweq> we should always remember that jobs which e.g. inherits from some tempest jobs should use default devstack's config
15:07:27 <ralonsoh> ok, I'll push a patch
15:07:38 <slaweq> so e.g. job neutron-tempest-slow-py3 which inherits from tempest-slow-py3 directly but only limits number of tests run should use ovn as it's what is used in tempest-slow-py3
15:08:03 <slaweq> and similar thing is also for some other jobs like e.g. neutron-tempest-ipv6-only
15:08:19 <ralonsoh> and should be named with -ovn suffix
15:08:20 <bcafarel> ralonsoh: so with "-ovn" for default jobs?
15:08:26 <slaweq> we can name those jobs "-ovn" but we should use default settings, no set explicitly backend in the job's definition
15:08:36 <ralonsoh> yes
15:08:40 <slaweq> does it makes sense?
15:08:43 <ralonsoh> perfect
15:08:48 <bcafarel> or -default-backend (or shorter one if possible)?
15:09:10 <ralonsoh> changing the default backend is not usual
15:09:14 <ralonsoh> I prefer -ovn
15:09:54 <slaweq> I'm ok with "-ovn"
15:10:04 <bcafarel> hmm true next default change is not for tomorrow :)
15:10:19 <slaweq> bcafarel I hope so :P
15:11:24 <slaweq> so ralonsoh You will continue that next week, right?
15:11:29 <ralonsoh> yes
15:11:35 <slaweq> great, thx
15:11:43 <slaweq> #action ralonsoh to update our ci job's definitions
15:11:54 <slaweq> ok, that was only AI from last week
15:12:00 <slaweq> we can move on to the next topic
15:13:26 <slaweq> #topic Stadium projects
15:13:38 <slaweq> anything to discuss regarding stadium projects' CI today?
15:13:51 <lajoskatona> nothing for today
15:14:01 <lajoskatona> I haven't seen any issue
15:14:09 <slaweq> good news, thx :)
15:14:19 <slaweq> #topic Stable branches
15:14:24 <slaweq> same question for stable branches
15:14:52 <bcafarel> and similar answer, all good from what I saw
15:14:59 <slaweq> great
15:15:11 <slaweq> I just today sent some patches for EM branches:
15:15:16 <slaweq> Train: https://review.opendev.org/q/topic:%22clean-non-voting-jobs%22+(status:open%20OR%20status:merged)
15:15:16 <slaweq> Stein: https://review.opendev.org/q/topic:%22clean-non-voting-jobs-stable%252Fstein%22+(status:open%20OR%20status:merged)
15:15:23 <slaweq> please review them
15:15:48 <ralonsoh> sure
15:15:56 <slaweq> I noticed last week that we are still running non-voting jobs on those branches and we agreed some time ago that we should move them to the experimental queue
15:16:03 <slaweq> so those patches are doing exactly that
15:16:26 <bcafarel> for stadium projects it will also be a nice spot check on CI status in those branches
15:16:46 <slaweq> bcafarel yes, I saw already some pep8/py27 failures
15:16:59 <slaweq> but I didn't had time to check them closely
15:18:30 <slaweq> ok, let's move on
15:18:34 <slaweq> #topic Grafana
15:19:30 <slaweq> http://grafana.openstack.org/dashboard/db/neutron-failure-rate#link
15:19:59 <slaweq> I don't see anything critical on the grafana this week
15:20:11 <slaweq> we need to update it as not all jobs are there
15:20:28 <slaweq> but I want to update it after ovs->ovn migration cleaning
15:22:13 <slaweq> except that, I think that our biggest problem still is fullstack job
15:22:20 <slaweq> and those failing L3 related tests
15:22:45 <obondarev> is it same issue?
15:23:29 <obondarev> I saw 2nd dhcp agent mostly workarounded that
15:23:39 <slaweq> obondarev let me check
15:24:07 <slaweq> https://614c13582752de139914-134fee3611f99557a1466ec5867d2d52.ssl.cf5.rackcdn.com/795781/3/check/neutron-fullstack-with-uwsgi/bf03ae2/testr_results.html
15:24:12 <slaweq> first example
15:24:20 <slaweq> it's from today morning
15:24:30 <slaweq> I think it's the same issue, or at least looks similar
15:25:35 <obondarev> yeah, looks the same..
15:25:47 <slaweq> https://zuul.opendev.org/t/openstack/builds?job_name=neutron-fullstack-with-uwsgi
15:26:02 <slaweq> obondarev here You have more exmples
15:26:16 <slaweq> I think that I saw similar failures in most of those cases
15:27:57 <obondarev> hmm, but I see dhcp agents not hanging
15:28:03 <obondarev> same for ovs agents
15:28:14 <obondarev> so that might be not related to privsep
15:28:39 <obondarev> I'll look into this when have some time (tomorrow I hope)
15:29:00 <slaweq> obondarev thx
15:29:08 <lajoskatona> For the hanging issue privsep change is on its way: https://review.opendev.org/c/openstack/oslo.privsep/+/794993
15:29:16 <slaweq> #action obondarev to check fullstack timeouts in L3 agent tests
15:29:46 <ralonsoh> ^^ please review this patch
15:29:46 <slaweq> lajoskatona++ thx
15:30:56 <slaweq> I added it to my review list for tomorrow morning
15:31:04 <ralonsoh> thx
15:32:04 <slaweq> ok, let's move on
15:32:19 <slaweq> I just have one question about periodic jobs left for today
15:32:38 <slaweq> ralonsoh how is Your patch for fix neutron-lib-master periodic job?
15:32:42 <ralonsoh> done
15:32:44 <ralonsoh> one sec
15:32:46 <slaweq> is it merged?
15:32:49 <ralonsoh> no
15:32:57 <slaweq> ahh, ok
15:33:11 <ralonsoh> and this is blocking requirements patch for n-lib 2.12.0
15:33:16 <ralonsoh> https://review.opendev.org/c/openstack/neutron/+/794236
15:33:36 <ralonsoh> ok, I have a +2 from liuyulong
15:33:54 <slaweq> it's in the gate now
15:33:56 <slaweq> thx
15:34:01 <ralonsoh> thanks
15:35:12 <slaweq> I didn't found today any new issues worth to discuss here
15:35:28 <slaweq> so if You don't have anything else, we can finish meeting earlier today
15:35:43 <bcafarel> nothing from me
15:35:50 <ralonsoh> nothing here
15:37:34 <slaweq> ok, so thx for attending the meeting
15:37:42 <ralonsoh> bye!
15:37:42 <slaweq> have a great evening and see You online
15:37:43 <slaweq> o/
15:37:46 <slaweq> #endmeeting