15:00:49 <slaweq> #startmeeting neutron_ci
15:00:49 <opendevmeet> Meeting started Tue Feb 22 15:00:49 2022 UTC and is due to finish in 60 minutes.  The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:49 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:49 <opendevmeet> The meeting name has been set to 'neutron_ci'
15:00:56 <slaweq> hi all
15:00:59 <slaweq> Grafana dashboard: https://grafana.opendev.org/d/f913631585/neutron-failure-rate?orgId=1
15:00:59 <slaweq> Please open now :)
15:01:04 <mlavalle> o/
15:01:15 <mlavalle> no video today, right?
15:01:41 <slaweq> mlavalle video is every other week
15:01:44 <slaweq> and it was last time
15:01:49 <slaweq> so today only irc :)
15:01:55 <ykarel> o/
15:01:59 <bcafarel> o/
15:02:37 <ralonsoh> hi
15:02:48 <elvira> o/
15:03:10 <slaweq> Grafana dashboard: https://grafana.opendev.org/d/f913631585/neutron-failure-rate?orgId=1
15:03:10 <slaweq> Please open now :)
15:03:22 <lajoskatona> Hi
15:03:42 <slaweq> ok, lets start
15:03:44 <slaweq> #topic Actions from previous meetings
15:03:54 <slaweq> mlavalle https://bugs.launchpad.net/neutron/+bug/1945283
15:04:09 <obondarev> hi
15:04:17 <mlavalle> I've started looking at it. Haven't come up with a root cause yet
15:04:29 <mlavalle> will continue working on it this week
15:04:35 <slaweq> ok, let me assign it to You for next week too
15:04:44 <mlavalle> yeah, please
15:04:49 <slaweq> #action mlavalle will continue work on https://bugs.launchpad.net/neutron/+bug/1945283
15:04:51 <slaweq> thx
15:04:55 <mlavalle> :-)
15:04:59 <slaweq> next one
15:05:01 <slaweq> slaweq to get data about rechecks in stable branches
15:05:15 <slaweq> I fetched that data
15:05:25 <slaweq> let me paste it here
15:06:09 <slaweq> https://paste.opendev.org/show/bWpvJRHMisJAvcenIY1Y/
15:06:44 <slaweq> in general there were weeks where numbers for e.g. wallaby or victoria were higher
15:06:53 <ralonsoh> we had less patches
15:06:59 <slaweq> but it's not very bad
15:07:05 <ralonsoh> right, quite good
15:07:09 <bcafarel> and there were some recent breakages (like pyroute bump in wallaby)
15:07:20 <slaweq> ralonsoh but those are "average rechecks per week"
15:07:27 <ralonsoh> yes, I know
15:07:41 <bcafarel> but overall less than 1.xx nice
15:08:01 <ralonsoh> but the higher the number of patches is, usually the higher the number of rechecks
15:08:05 <slaweq> generally, as with master could be better, but it's not the nightmare
15:09:10 <bcafarel> and overall does not get too bad as we go to older branches
15:09:27 <lajoskatona> good news before the release :-)
15:09:27 <slaweq> yes
15:10:13 <slaweq> ok, so speaking about stable branches :)
15:10:15 <slaweq> #topic Stable branches
15:10:23 <slaweq> any new issues to discuss there?
15:11:37 <bcafarel> none that I am aware of
15:11:55 <slaweq> that's great news :)
15:12:07 <bcafarel> indeed I will never complain about "no news" :)
15:12:40 <lajoskatona> LoL
15:12:50 <slaweq> ok, so I think we can move on
15:12:54 <slaweq> next topic
15:12:57 <slaweq> #topic Stadium projects
15:13:04 <lajoskatona> no news :-)
15:13:14 <slaweq> ++
15:13:25 <lajoskatona> I checked this morning and the ones I checked are green
15:13:29 <slaweq> I like that :)
15:13:59 <mlavalle> +1
15:13:59 <slaweq> thx lajoskatona
15:14:12 <slaweq> ok, so lets move on to the next topic
15:14:12 <slaweq> #topic Grafana
15:14:15 <slaweq> https://grafana.opendev.org/d/f913631585/neutron-failure-rate
15:15:31 <slaweq> I think ykarel did some update to our dashboard today
15:15:38 <slaweq> ykarel do You have link to Your patch?
15:15:52 <ykarel> slaweq, just sent https://review.opendev.org/c/openstack/project-config/+/830440
15:16:04 <slaweq> thx
15:16:20 <ykarel> also periodic jobs are not running since 15th on master branch
15:16:39 <ykarel> for that sent https://review.opendev.org/c/openstack/neutron/+/830371
15:17:02 <ralonsoh> nice
15:17:04 <slaweq> I just +2 this patch
15:17:09 <slaweq> thx for it
15:17:40 <ykarel> ack Thanks
15:18:07 <slaweq> from other things related to grafana, it seems that there are missing some data from last days
15:18:20 <slaweq> but I don't think we can do anything with it
15:18:26 <slaweq> I hope it will be better in next days
15:18:48 <slaweq> as for rechecks in master branch, here are the stats:
15:18:50 <slaweq> +---------+----------+... (full message at https://matrix.org/_matrix/media/r0/download/matrix.org/opdMcATWObwJnLwbGuBZmVPE)
15:19:09 <slaweq> we are going a bit higher last 2 or 3 weeks
15:19:20 <ralonsoh> now is 0??
15:19:29 <slaweq> but I know there were e.g. issue with some fullstack local_ip test which is now fixed by obondarev
15:19:52 <slaweq> ralonsoh it's this week, maybe there wasn't many patches in gate this week :)
15:21:12 <slaweq> any other questions/comments regarding grafana?
15:22:18 <slaweq> ok, I guess that this silence means "no" :)
15:22:22 <bcafarel> :)
15:22:23 <slaweq> #topic fullstack/functional
15:22:44 <slaweq> I looked at the patches with failed tests today and I found only 2 issues
15:22:47 <slaweq> one in functional
15:22:56 <slaweq> Network interface not found in namespace (again)
15:23:07 <slaweq> I opened bug https://bugs.launchpad.net/neutron/+bug/1961740
15:23:12 <slaweq> https://89b1b88fa362b409cfb1-2a70ac574f4ba34d12afc72df211f1b3.ssl.cf5.rackcdn.com/828687/1/gate/neutron-functional-with-uwsgi/a5b844b/testr_results.html
15:23:22 <slaweq> I have it on my todo list to check
15:23:35 <slaweq> but if there is anyone who would like to check it, feel free to take it
15:28:07 <slaweq> ralonsoh I wonder if it's not the same issue as we have seen in one d/s bug
15:28:15 <ralonsoh> which one?
15:28:19 <slaweq> I added https://review.opendev.org/c/openstack/ovsdbapp/+/825727 to hopefully understand it better
15:28:36 <slaweq> but in logs of the functional test there is no such logs so it's hard to say really
15:29:33 <lajoskatona> we just discussed it but there was no bug for it, am I right?
15:29:33 <slaweq> but looking at timestamps, it's probably something different
15:29:52 <slaweq> lajoskatona yes
15:33:13 <slaweq> I will check journal log from that test to see if there is maybe something interesting there
15:33:46 <slaweq> #action slaweq to check missing devices in namespace, see https://bugs.launchpad.net/neutron/+bug/1961740 for details
15:33:56 <slaweq> now fullstack
15:34:01 <slaweq> https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_99b/828022/3/gate/neutron-fullstack-with-uwsgi/99b5482/testr_results.html
15:34:06 <slaweq> this is issue with local_ip test
15:34:18 <slaweq> but IIUC it should be fixed with https://review.opendev.org/c/openstack/neutron/+/829659
15:34:22 <slaweq> thx obondarev for the fix
15:34:56 <lajoskatona> and it is merged so we are on the safe side :-)
15:35:07 <slaweq> we should be :)
15:36:07 <obondarev> hopefully :) but not 100%
15:36:25 <slaweq> anything else regarding functional/fullstack jobs for today?
15:36:32 <slaweq> or can we move on?
15:37:57 <slaweq> ok, so lets move on
15:38:02 <slaweq> #topic Periodic
15:38:09 <slaweq> generally periodic jobs looks ok
15:38:19 <slaweq> thx ykarel for fix of the tripleo jobs there
15:38:39 <slaweq> from other things I have one patch to review https://review.opendev.org/c/openstack/neutron/+/827302
15:39:10 <slaweq> and I have one more question regarding periodic jobs
15:39:33 <slaweq> wdyt if we would add fips related jobs, especially functional and scenario to the periodic queue?
15:39:39 <slaweq> those jobs are so far in the experimental queue
15:39:50 <ralonsoh> I'm ok with this
15:40:08 <slaweq> but I though to maybe have them in perdiodic to have more data about how stable they are
15:40:10 <ralonsoh> will you remove them from experimental queue?
15:40:16 <slaweq> and if we will see any issues with them
15:40:35 <slaweq> ralonsoh yes, I though to simply move them from experimental to periodic now
15:40:42 <ralonsoh> perfect for me
15:40:45 <mlavalle> +1
15:40:46 <lajoskatona> +1
15:40:48 <ykarel> +1
15:40:51 <obondarev> +1
15:40:58 <slaweq> thx, so I will do it
15:41:00 <bcafarel> joining the +1 club :)
15:41:00 <lajoskatona> the plan is to later run it for every patch in check and gate queue?
15:41:08 <slaweq> #action slaweq to move fips jobs to periodic queue
15:41:39 <slaweq> lajoskatona that would be ideal at some point but giving number of jobs which we already have, I don't think it's topic for now
15:41:51 <slaweq> maybe if that will become official community goal at some point
15:41:59 <slaweq> and then it may be required to have such job
15:42:40 <lajoskatona> ok, agree
15:42:51 <slaweq> that's basically all what I had for today
15:43:03 <slaweq> anything else related to our CI what You want to discuss today?
15:44:12 <slaweq> if not, I think I can give You 15 minutes back :)
15:44:21 <slaweq> thx for attending the meeting and have a great week
15:44:22 <bcafarel> yay \o/
15:44:25 <slaweq> see You online
15:44:26 <slaweq> o/
15:44:26 <lajoskatona> \o/
15:44:26 <ralonsoh> bye
15:44:29 <lajoskatona> bye
15:44:31 <obondarev> o/
15:44:31 <slaweq> #endmeeting