15:00:23 <slaweq> #startmeeting neutron_ci
15:00:23 <opendevmeet> Meeting started Tue Apr 26 15:00:23 2022 UTC and is due to finish in 60 minutes.  The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:23 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:23 <opendevmeet> The meeting name has been set to 'neutron_ci'
15:00:24 <slaweq> hi
15:00:26 <mlavalle> o/
15:00:30 <ralonsoh> hi
15:01:43 <slaweq> Grafana dashboard: https://grafana.opendev.org/d/f913631585/neutron-failure-rate?orgId=1
15:01:47 <bcafarel> o/
15:02:05 <opendevreview> Merged openstack/neutron-vpnaas master: Fix failover with L3 HA  https://review.opendev.org/c/openstack/neutron-vpnaas/+/823904
15:02:38 <lajoskatona> o/
15:02:57 <slaweq> #topic Actions from previous meetings
15:03:04 <slaweq> mlavalle to add note about rechecking with reason
15:03:10 <mlavalle> I  did
15:03:29 <mlavalle> it turns out we already had  anote
15:03:31 <obondarev> hi
15:03:37 <ykarel> hi
15:03:40 <mlavalle> buried in the Neutron policies
15:03:56 <mlavalle> so I moved it one lever higher to make it more visible
15:04:14 <lajoskatona> good idea
15:04:15 <slaweq> thx mlavalle
15:04:19 <mlavalle> fixed some wording and added some examples (shamelessly stolen from Cinder)
15:04:39 <slaweq> now harder part - we need to enforce ourselves not to recheck without giving reasons
15:04:49 <slaweq> and also educate others during reviews
15:04:52 <slaweq> :)
15:05:01 <bcafarel> do what we say, not what we usually do :)
15:05:18 <lajoskatona> https://docs.openstack.org/neutron/latest/contributor/index.html#gerrit-rechecks
15:05:18 <slaweq> bcafarel: exactly :)
15:05:18 <ralonsoh> link please?
15:05:57 <mlavalle> ralonsoh: https://docs.openstack.org/neutron/latest/contributor/gerrit-recheck.html
15:06:02 <ralonsoh> tahnks
15:06:20 <mlavalle> ralonsoh: https://review.opendev.org/c/openstack/neutron/+/839107
15:06:24 <mlavalle> patch^^^
15:07:01 <slaweq> ok, I think we can move on
15:07:04 <slaweq> next one
15:07:06 <slaweq> lajoskatona to check with QA team if neutron-tempest-plugin tag for victoria EM will be done together with tempest
15:07:24 <lajoskatona> yes I asked gmann about it
15:07:58 <lajoskatona> I have to search for the logs, but summary: they will prepare the tag after victoria EM-ed, and they can do it for neutron-tempest-plugin
15:08:11 <slaweq> great
15:08:41 <slaweq> so as last step we will update our victoria jobs in neutron-tempest-plugin to use those tagged versions of tempest and neutron-tempest-plugin
15:08:43 <lajoskatona> "we can do tempest first and then plugins. but I am waiting for victoria to be EM which should be in coming week or so" (https://meetings.opendev.org/irclogs/%23openstack-qa/%23openstack-qa.2022-04-25.log.html )
15:08:55 <lajoskatona> yes, exactly
15:08:56 <slaweq> and remove those jobs from the check/gate queue in neutron-tempest-plugin repo
15:09:03 <slaweq> thx lajoskatona
15:09:30 <slaweq> next one
15:09:32 <slaweq> lajoskatona to contiune checking failing py39 job in networking-odl
15:10:16 <lajoskatona> it's on the way, but many small things to fix there (https://review.opendev.org/c/openstack/networking-odl/+/838451 )
15:11:05 <lajoskatona> functional is now failing due to a recent change in devstack to collect stats from jobs and they import pymysql and that is not present for functional jobs
15:12:03 <slaweq> lajoskatona: for that one there is fix proposed already: https://review.opendev.org/c/openstack/devstack/+/839217
15:12:42 <lajoskatona> slaweq: cool, thanks
15:13:23 <slaweq> I pinged gmann and other devstack cores to review it ASAP
15:13:54 <slaweq> ok, next one
15:13:55 <slaweq> mlavalle to check neutron-ovn-tempest-ovs-master-fedora periodic job
15:14:06 <mlavalle> I checked it
15:14:32 <mlavalle> the original problem was with ovs compiling / building
15:14:46 <mlavalle> so the it failed building
15:14:52 <mlavalle> that was fixed last week
15:15:18 <slaweq> yeah, I saw that this job was green few times this week :)
15:15:33 <mlavalle> it now succeeds building, but is intermitently failing a test: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_15b/periodic/opendev.org/openstack/neutron/master/neutron-ovn-tempest-ovs-master-fedora/15b5850/testr_results.html
15:15:55 <mlavalle> so I need to investigate this latter failure
15:15:58 <slaweq> but this test isn't really related to neutron
15:16:05 <mlavalle> I know
15:16:32 <mlavalle> I'll just keep an eye on it
15:17:28 <slaweq> yes, thx a lot
15:17:40 <mlavalle> :-)
15:17:50 <slaweq> ok, last one from previous week
15:17:51 <slaweq> mlavalle to check failed fips periodic job
15:18:01 <mlavalle> with that one I got lucky
15:18:20 <mlavalle> it has been succeeding every single time since we talked about it last week
15:18:31 <mlavalle> https://zuul.openstack.org/builds?job_name=neutron-ovn-tempest-ovs-release-fips
15:18:43 <mlavalle> I think that failure was a one of
15:19:08 <slaweq> great then :)
15:19:13 <mlavalle> because even the few failures during the preceding days were build failures
15:20:45 <opendevreview> Merged openstack/networking-ovn stable/train: Do not announce any DNS resolver if "0.0.0.0" or "::" provided  https://review.opendev.org/c/openstack/networking-ovn/+/838987
15:21:22 <opendevreview> Merged openstack/tap-as-a-service master: Add weekly jobs  https://review.opendev.org/c/openstack/tap-as-a-service/+/834505
15:21:33 <slaweq> ok, that were all actions from last meeting
15:21:33 <ykarel> ahkk https://github.com/ovn-org/ovn/commit/996ed75db776c59098b6539d0c03d43977202885 fixed that fedora ovn job
15:21:47 <ykarel> compilation issue
15:21:47 <slaweq> I think we can move on
15:21:51 <slaweq> #topic Stable branches
15:21:56 <slaweq> thx ykarel for link
15:22:37 <bcafarel> not much this week (aka no news is good news!)
15:22:53 <bcafarel> main point is victoria switching to EM we already talked about
15:24:01 <slaweq> ok, thx bcafarel :)
15:24:10 <slaweq> so next topic then
15:24:12 <slaweq> #topic Stadium projects
15:25:35 <slaweq> lajoskatona: any new issues in stadium's ci, except those in networking-odl?
15:26:05 <lajoskatona> yes for weekly jobs: https://review.opendev.org/q/topic:weekly_job
15:26:27 <lajoskatona> I just realized that for sfc and bgpvpn we have to add neutron to required projects list
15:27:05 <lajoskatona> and I have 2 more for taas to make py310 green: py310 for taas: https://review.opendev.org/q/topic:p310_for_taas
15:27:25 <lajoskatona> but otherwise the other stadiums seems to be green
15:27:32 <opendevreview> Merged openstack/neutron stable/xena: Fix setting table monitoring conditions  https://review.opendev.org/c/openstack/neutron/+/838783
15:28:17 <slaweq> I think You need to rebase those 2 taas patches
15:29:06 <lajoskatona> slaweq: true, I will do, 2 hours ago it was green....
15:29:18 <slaweq> :)
15:30:10 <slaweq> and for those weekly jobs in sfc and bgpvpn - do You have patches ready?
15:30:22 <lajoskatona> https://review.opendev.org/q/topic:weekly_job
15:30:39 <slaweq> ahh, sorry
15:30:42 <slaweq> thx
15:30:49 <lajoskatona> thanks for the attention
15:31:26 <slaweq> +2 on both
15:32:09 <slaweq> ok, I think we can move on
15:32:11 <slaweq> #topic Grafana
15:32:30 <slaweq> #link https://grafana.opendev.org/d/f913631585/neutron-failure-rate
15:32:39 <slaweq> I noticed there is a lot of gaps there recently
15:32:59 <slaweq> I'm not sure if that is going to be shutdown together with services like logstash too
15:33:15 <slaweq> I hope grafana will still be up :)
15:33:50 <lajoskatona> I haven't heard about it, so I hope not
15:34:15 <bcafarel> no logs, no bugs? :)
15:34:32 <bcafarel> but yeah hopefully it still works after that change
15:34:35 <slaweq> bcafarel: I wish :P
15:35:23 <slaweq> regarding dashboard I saw some spike to 100% of failures during the weekend
15:35:44 <slaweq> but there was not many runs so it could be some kind of red hering
15:35:59 <slaweq> now it seems to be ok(ish) again
15:36:26 <slaweq> even functional jobs is pretty low with failures this week
15:36:56 <lajoskatona> a little strange that all tempest job failure rates are 0
15:37:16 <slaweq> lajoskatona: I'm not worried about it :)
15:37:44 <lajoskatona> slaweq: if it is true I like it
15:37:54 <slaweq> :)
15:38:13 <slaweq> anything else regarding grafana?
15:38:17 <slaweq> or can we move on?
15:38:26 <mlavalle> move on
15:38:38 <lajoskatona> +1
15:38:45 <slaweq> #topic Rechecks
15:39:03 <slaweq> I added it as separate topic as we discuss it every week basically
15:39:11 <slaweq> +---------+----------+... (full message at https://matrix.org/_matrix/media/r0/download/matrix.org/fFJABtYBrfTVRoIFUycCRQHt)
15:39:25 <slaweq> we are pretty ok with the number of rechecks
15:39:44 <slaweq> above 1 in average this week but I checked patches which had most of the recheck
15:39:51 <slaweq> and there were only 3 above that average:
15:40:12 <slaweq> https://review.opendev.org/c/openstack/neutron/+/836863 - 8 rechecks - mostly hitting https://bugs.launchpad.net/neutron/+bug/1956958
15:40:20 <lajoskatona> these numbers for master or all branches?
15:40:30 <slaweq> https://review.opendev.org/c/openstack/neutron/+/834952 - 5 rechecks - mostly hitting
15:40:34 <slaweq> https://bugs.launchpad.net/neutron/+bug/1956958
15:40:56 <ralonsoh> ^^ https://review.opendev.org/c/openstack/neutron/+/836140
15:40:59 <slaweq> and https://review.opendev.org/c/openstack/neutron/+/837143 - 2 rechecks, pretty old ones
15:41:03 <lajoskatona> the doc one was from me, It was boring at the end but I had the opportunity to force myself to do not recheck blindly
15:41:05 <ralonsoh> I would need help with this patch
15:41:12 <slaweq> lajoskatona: yes, those are numbers from master branch only
15:41:46 <slaweq> lajoskatona: no worries, it's not to blame anyone here
15:42:06 <slaweq> more to understand where and why we have those rechecks :)
15:42:13 <lajoskatona> slaweq: no offense but that small patch for doc I keep in my memories :-)
15:42:17 <slaweq> and it seems that https://bugs.launchpad.net/neutron/+bug/1956958 is our main problem currently :)
15:42:43 <lajoskatona> and ralonsoh has the above patch for it :  https://review.opendev.org/c/openstack/neutron/+/836140 ?
15:42:49 <ralonsoh> yes
15:43:01 <slaweq> ralonsoh: I remember about Your patch https://review.opendev.org/c/openstack/neutron/+/836140 and looking forward for it :)
15:43:02 <ralonsoh> I've tested it manually
15:43:20 <ralonsoh> but I don't know why the script is not running when called from the L3 agent
15:43:31 <slaweq> do You need help with it?
15:43:37 <slaweq> I can take a look
15:43:45 <ralonsoh> that will be perfect
15:44:30 <slaweq> ok
15:44:36 <slaweq> I will try, but probably on Thursday or Friday
15:45:08 <ralonsoh> no rush
15:45:18 <slaweq> #action slaweq to check patch https://review.opendev.org/c/openstack/neutron/+/836140
15:45:42 <slaweq> anything else regarding rechecks today?
15:45:49 <slaweq> or are we moving on?
15:46:09 <lajoskatona> we can move
15:46:24 <slaweq> #topic fullstack/functional
15:46:44 <slaweq> here ykarel (probably) gave some examples of failures:
15:46:49 <slaweq> https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_955/834952/1/gate/neutron-functional-with-uwsgi/955ffd5/testr_results.html
15:46:54 <slaweq> https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_b61/834952/1/gate/neutron-functional-with-uwsgi/b617dc3/testr_results.html
15:47:00 <slaweq> https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_211/837681/2/check/neutron-functional-with-uwsgi/2117823/testr_results.html
15:47:21 <ykarel> yes i think all those are related to keeplived thing?
15:47:22 <slaweq> but IMO all of them are related to the bug     Probably https://bugs.launchpad.net/neutron/+bug/1956958 and timeout while waiting for router transition
15:48:16 <ykarel> slaweq, is it different then what ralonsoh is trying to fix with https://review.opendev.org/c/openstack/neutron/+/836140?
15:48:17 <slaweq> and that's basically all what I have for today for functional/fullstack jobs
15:48:40 <slaweq> ykarel: I think it's exactly the same
15:48:44 <ralonsoh> yeah
15:48:54 <ykarel> okk Thanks for confirmation
15:49:43 <ykarel> also i noticed failure related to functional test stuck in stable/yoga and pushed backport https://review.opendev.org/c/openstack/neutron/+/839189
15:50:13 <slaweq> thx
15:50:40 <ralonsoh> (nice patch!)
15:50:42 <ralonsoh> hehehe
15:51:00 <slaweq> :)
15:51:02 <lajoskatona> LoL
15:51:06 <slaweq> ok, lets move on
15:51:55 <slaweq> for periodic jobs all looks mostly good
15:52:04 <slaweq> we have failing neutron-ovn-tempest-postgres-full job but it's because of the same bug as networking-odl functional jobs
15:52:07 <slaweq> and patch https://review.opendev.org/c/openstack/devstack/+/839217 should fix it
15:52:09 <slaweq> so one last item for today
15:52:16 <slaweq> #topic On Demand agenda
15:52:35 <slaweq> I have one quick thing here
15:52:49 <slaweq> next Tuesday will be public holiday in Poland
15:52:53 <slaweq> so I will be off
15:53:08 <slaweq> do we want to cancel ci meeting or someone else will chair it?
15:53:25 <bcafarel> not public holiday here but I will be off too
15:53:32 <ralonsoh> if we have a critical number of bugs, we can have it
15:53:37 <ralonsoh> if not, for now, we can cancel
15:53:39 <lajoskatona> +1
15:54:02 <slaweq> ok, I will check on Monday how it looks and will ping You ralonsoh to chair it or will cancel it
15:54:07 <ralonsoh> perfect
15:54:12 <mlavalle> +1
15:54:12 <slaweq> thx
15:54:19 <slaweq> that's all from me for today
15:54:28 <ykarel> +1
15:54:28 <slaweq> anything else You want to discuss, regarding our ci?
15:54:36 <mlavalle> nothing from me
15:55:02 <lajoskatona> nothing from me
15:55:09 <bcafarel> nothing here
15:55:20 <slaweq> ok, so thx for attending the meeting
15:55:24 <slaweq> and have a great week
15:55:25 <slaweq> o/
15:55:26 <obondarev> o/
15:55:26 <slaweq> #endmeeting