15:00:23 <slaweq> #startmeeting neutron_ci 15:00:23 <opendevmeet> Meeting started Tue Apr 26 15:00:23 2022 UTC and is due to finish in 60 minutes. The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:00:23 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:00:23 <opendevmeet> The meeting name has been set to 'neutron_ci' 15:00:24 <slaweq> hi 15:00:26 <mlavalle> o/ 15:00:30 <ralonsoh> hi 15:01:43 <slaweq> Grafana dashboard: https://grafana.opendev.org/d/f913631585/neutron-failure-rate?orgId=1 15:01:47 <bcafarel> o/ 15:02:05 <opendevreview> Merged openstack/neutron-vpnaas master: Fix failover with L3 HA https://review.opendev.org/c/openstack/neutron-vpnaas/+/823904 15:02:38 <lajoskatona> o/ 15:02:57 <slaweq> #topic Actions from previous meetings 15:03:04 <slaweq> mlavalle to add note about rechecking with reason 15:03:10 <mlavalle> I did 15:03:29 <mlavalle> it turns out we already had anote 15:03:31 <obondarev> hi 15:03:37 <ykarel> hi 15:03:40 <mlavalle> buried in the Neutron policies 15:03:56 <mlavalle> so I moved it one lever higher to make it more visible 15:04:14 <lajoskatona> good idea 15:04:15 <slaweq> thx mlavalle 15:04:19 <mlavalle> fixed some wording and added some examples (shamelessly stolen from Cinder) 15:04:39 <slaweq> now harder part - we need to enforce ourselves not to recheck without giving reasons 15:04:49 <slaweq> and also educate others during reviews 15:04:52 <slaweq> :) 15:05:01 <bcafarel> do what we say, not what we usually do :) 15:05:18 <lajoskatona> https://docs.openstack.org/neutron/latest/contributor/index.html#gerrit-rechecks 15:05:18 <slaweq> bcafarel: exactly :) 15:05:18 <ralonsoh> link please? 15:05:57 <mlavalle> ralonsoh: https://docs.openstack.org/neutron/latest/contributor/gerrit-recheck.html 15:06:02 <ralonsoh> tahnks 15:06:20 <mlavalle> ralonsoh: https://review.opendev.org/c/openstack/neutron/+/839107 15:06:24 <mlavalle> patch^^^ 15:07:01 <slaweq> ok, I think we can move on 15:07:04 <slaweq> next one 15:07:06 <slaweq> lajoskatona to check with QA team if neutron-tempest-plugin tag for victoria EM will be done together with tempest 15:07:24 <lajoskatona> yes I asked gmann about it 15:07:58 <lajoskatona> I have to search for the logs, but summary: they will prepare the tag after victoria EM-ed, and they can do it for neutron-tempest-plugin 15:08:11 <slaweq> great 15:08:41 <slaweq> so as last step we will update our victoria jobs in neutron-tempest-plugin to use those tagged versions of tempest and neutron-tempest-plugin 15:08:43 <lajoskatona> "we can do tempest first and then plugins. but I am waiting for victoria to be EM which should be in coming week or so" (https://meetings.opendev.org/irclogs/%23openstack-qa/%23openstack-qa.2022-04-25.log.html ) 15:08:55 <lajoskatona> yes, exactly 15:08:56 <slaweq> and remove those jobs from the check/gate queue in neutron-tempest-plugin repo 15:09:03 <slaweq> thx lajoskatona 15:09:30 <slaweq> next one 15:09:32 <slaweq> lajoskatona to contiune checking failing py39 job in networking-odl 15:10:16 <lajoskatona> it's on the way, but many small things to fix there (https://review.opendev.org/c/openstack/networking-odl/+/838451 ) 15:11:05 <lajoskatona> functional is now failing due to a recent change in devstack to collect stats from jobs and they import pymysql and that is not present for functional jobs 15:12:03 <slaweq> lajoskatona: for that one there is fix proposed already: https://review.opendev.org/c/openstack/devstack/+/839217 15:12:42 <lajoskatona> slaweq: cool, thanks 15:13:23 <slaweq> I pinged gmann and other devstack cores to review it ASAP 15:13:54 <slaweq> ok, next one 15:13:55 <slaweq> mlavalle to check neutron-ovn-tempest-ovs-master-fedora periodic job 15:14:06 <mlavalle> I checked it 15:14:32 <mlavalle> the original problem was with ovs compiling / building 15:14:46 <mlavalle> so the it failed building 15:14:52 <mlavalle> that was fixed last week 15:15:18 <slaweq> yeah, I saw that this job was green few times this week :) 15:15:33 <mlavalle> it now succeeds building, but is intermitently failing a test: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_15b/periodic/opendev.org/openstack/neutron/master/neutron-ovn-tempest-ovs-master-fedora/15b5850/testr_results.html 15:15:55 <mlavalle> so I need to investigate this latter failure 15:15:58 <slaweq> but this test isn't really related to neutron 15:16:05 <mlavalle> I know 15:16:32 <mlavalle> I'll just keep an eye on it 15:17:28 <slaweq> yes, thx a lot 15:17:40 <mlavalle> :-) 15:17:50 <slaweq> ok, last one from previous week 15:17:51 <slaweq> mlavalle to check failed fips periodic job 15:18:01 <mlavalle> with that one I got lucky 15:18:20 <mlavalle> it has been succeeding every single time since we talked about it last week 15:18:31 <mlavalle> https://zuul.openstack.org/builds?job_name=neutron-ovn-tempest-ovs-release-fips 15:18:43 <mlavalle> I think that failure was a one of 15:19:08 <slaweq> great then :) 15:19:13 <mlavalle> because even the few failures during the preceding days were build failures 15:20:45 <opendevreview> Merged openstack/networking-ovn stable/train: Do not announce any DNS resolver if "0.0.0.0" or "::" provided https://review.opendev.org/c/openstack/networking-ovn/+/838987 15:21:22 <opendevreview> Merged openstack/tap-as-a-service master: Add weekly jobs https://review.opendev.org/c/openstack/tap-as-a-service/+/834505 15:21:33 <slaweq> ok, that were all actions from last meeting 15:21:33 <ykarel> ahkk https://github.com/ovn-org/ovn/commit/996ed75db776c59098b6539d0c03d43977202885 fixed that fedora ovn job 15:21:47 <ykarel> compilation issue 15:21:47 <slaweq> I think we can move on 15:21:51 <slaweq> #topic Stable branches 15:21:56 <slaweq> thx ykarel for link 15:22:37 <bcafarel> not much this week (aka no news is good news!) 15:22:53 <bcafarel> main point is victoria switching to EM we already talked about 15:24:01 <slaweq> ok, thx bcafarel :) 15:24:10 <slaweq> so next topic then 15:24:12 <slaweq> #topic Stadium projects 15:25:35 <slaweq> lajoskatona: any new issues in stadium's ci, except those in networking-odl? 15:26:05 <lajoskatona> yes for weekly jobs: https://review.opendev.org/q/topic:weekly_job 15:26:27 <lajoskatona> I just realized that for sfc and bgpvpn we have to add neutron to required projects list 15:27:05 <lajoskatona> and I have 2 more for taas to make py310 green: py310 for taas: https://review.opendev.org/q/topic:p310_for_taas 15:27:25 <lajoskatona> but otherwise the other stadiums seems to be green 15:27:32 <opendevreview> Merged openstack/neutron stable/xena: Fix setting table monitoring conditions https://review.opendev.org/c/openstack/neutron/+/838783 15:28:17 <slaweq> I think You need to rebase those 2 taas patches 15:29:06 <lajoskatona> slaweq: true, I will do, 2 hours ago it was green.... 15:29:18 <slaweq> :) 15:30:10 <slaweq> and for those weekly jobs in sfc and bgpvpn - do You have patches ready? 15:30:22 <lajoskatona> https://review.opendev.org/q/topic:weekly_job 15:30:39 <slaweq> ahh, sorry 15:30:42 <slaweq> thx 15:30:49 <lajoskatona> thanks for the attention 15:31:26 <slaweq> +2 on both 15:32:09 <slaweq> ok, I think we can move on 15:32:11 <slaweq> #topic Grafana 15:32:30 <slaweq> #link https://grafana.opendev.org/d/f913631585/neutron-failure-rate 15:32:39 <slaweq> I noticed there is a lot of gaps there recently 15:32:59 <slaweq> I'm not sure if that is going to be shutdown together with services like logstash too 15:33:15 <slaweq> I hope grafana will still be up :) 15:33:50 <lajoskatona> I haven't heard about it, so I hope not 15:34:15 <bcafarel> no logs, no bugs? :) 15:34:32 <bcafarel> but yeah hopefully it still works after that change 15:34:35 <slaweq> bcafarel: I wish :P 15:35:23 <slaweq> regarding dashboard I saw some spike to 100% of failures during the weekend 15:35:44 <slaweq> but there was not many runs so it could be some kind of red hering 15:35:59 <slaweq> now it seems to be ok(ish) again 15:36:26 <slaweq> even functional jobs is pretty low with failures this week 15:36:56 <lajoskatona> a little strange that all tempest job failure rates are 0 15:37:16 <slaweq> lajoskatona: I'm not worried about it :) 15:37:44 <lajoskatona> slaweq: if it is true I like it 15:37:54 <slaweq> :) 15:38:13 <slaweq> anything else regarding grafana? 15:38:17 <slaweq> or can we move on? 15:38:26 <mlavalle> move on 15:38:38 <lajoskatona> +1 15:38:45 <slaweq> #topic Rechecks 15:39:03 <slaweq> I added it as separate topic as we discuss it every week basically 15:39:11 <slaweq> +---------+----------+... (full message at https://matrix.org/_matrix/media/r0/download/matrix.org/fFJABtYBrfTVRoIFUycCRQHt) 15:39:25 <slaweq> we are pretty ok with the number of rechecks 15:39:44 <slaweq> above 1 in average this week but I checked patches which had most of the recheck 15:39:51 <slaweq> and there were only 3 above that average: 15:40:12 <slaweq> https://review.opendev.org/c/openstack/neutron/+/836863 - 8 rechecks - mostly hitting https://bugs.launchpad.net/neutron/+bug/1956958 15:40:20 <lajoskatona> these numbers for master or all branches? 15:40:30 <slaweq> https://review.opendev.org/c/openstack/neutron/+/834952 - 5 rechecks - mostly hitting 15:40:34 <slaweq> https://bugs.launchpad.net/neutron/+bug/1956958 15:40:56 <ralonsoh> ^^ https://review.opendev.org/c/openstack/neutron/+/836140 15:40:59 <slaweq> and https://review.opendev.org/c/openstack/neutron/+/837143 - 2 rechecks, pretty old ones 15:41:03 <lajoskatona> the doc one was from me, It was boring at the end but I had the opportunity to force myself to do not recheck blindly 15:41:05 <ralonsoh> I would need help with this patch 15:41:12 <slaweq> lajoskatona: yes, those are numbers from master branch only 15:41:46 <slaweq> lajoskatona: no worries, it's not to blame anyone here 15:42:06 <slaweq> more to understand where and why we have those rechecks :) 15:42:13 <lajoskatona> slaweq: no offense but that small patch for doc I keep in my memories :-) 15:42:17 <slaweq> and it seems that https://bugs.launchpad.net/neutron/+bug/1956958 is our main problem currently :) 15:42:43 <lajoskatona> and ralonsoh has the above patch for it : https://review.opendev.org/c/openstack/neutron/+/836140 ? 15:42:49 <ralonsoh> yes 15:43:01 <slaweq> ralonsoh: I remember about Your patch https://review.opendev.org/c/openstack/neutron/+/836140 and looking forward for it :) 15:43:02 <ralonsoh> I've tested it manually 15:43:20 <ralonsoh> but I don't know why the script is not running when called from the L3 agent 15:43:31 <slaweq> do You need help with it? 15:43:37 <slaweq> I can take a look 15:43:45 <ralonsoh> that will be perfect 15:44:30 <slaweq> ok 15:44:36 <slaweq> I will try, but probably on Thursday or Friday 15:45:08 <ralonsoh> no rush 15:45:18 <slaweq> #action slaweq to check patch https://review.opendev.org/c/openstack/neutron/+/836140 15:45:42 <slaweq> anything else regarding rechecks today? 15:45:49 <slaweq> or are we moving on? 15:46:09 <lajoskatona> we can move 15:46:24 <slaweq> #topic fullstack/functional 15:46:44 <slaweq> here ykarel (probably) gave some examples of failures: 15:46:49 <slaweq> https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_955/834952/1/gate/neutron-functional-with-uwsgi/955ffd5/testr_results.html 15:46:54 <slaweq> https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_b61/834952/1/gate/neutron-functional-with-uwsgi/b617dc3/testr_results.html 15:47:00 <slaweq> https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_211/837681/2/check/neutron-functional-with-uwsgi/2117823/testr_results.html 15:47:21 <ykarel> yes i think all those are related to keeplived thing? 15:47:22 <slaweq> but IMO all of them are related to the bug Probably https://bugs.launchpad.net/neutron/+bug/1956958 and timeout while waiting for router transition 15:48:16 <ykarel> slaweq, is it different then what ralonsoh is trying to fix with https://review.opendev.org/c/openstack/neutron/+/836140? 15:48:17 <slaweq> and that's basically all what I have for today for functional/fullstack jobs 15:48:40 <slaweq> ykarel: I think it's exactly the same 15:48:44 <ralonsoh> yeah 15:48:54 <ykarel> okk Thanks for confirmation 15:49:43 <ykarel> also i noticed failure related to functional test stuck in stable/yoga and pushed backport https://review.opendev.org/c/openstack/neutron/+/839189 15:50:13 <slaweq> thx 15:50:40 <ralonsoh> (nice patch!) 15:50:42 <ralonsoh> hehehe 15:51:00 <slaweq> :) 15:51:02 <lajoskatona> LoL 15:51:06 <slaweq> ok, lets move on 15:51:55 <slaweq> for periodic jobs all looks mostly good 15:52:04 <slaweq> we have failing neutron-ovn-tempest-postgres-full job but it's because of the same bug as networking-odl functional jobs 15:52:07 <slaweq> and patch https://review.opendev.org/c/openstack/devstack/+/839217 should fix it 15:52:09 <slaweq> so one last item for today 15:52:16 <slaweq> #topic On Demand agenda 15:52:35 <slaweq> I have one quick thing here 15:52:49 <slaweq> next Tuesday will be public holiday in Poland 15:52:53 <slaweq> so I will be off 15:53:08 <slaweq> do we want to cancel ci meeting or someone else will chair it? 15:53:25 <bcafarel> not public holiday here but I will be off too 15:53:32 <ralonsoh> if we have a critical number of bugs, we can have it 15:53:37 <ralonsoh> if not, for now, we can cancel 15:53:39 <lajoskatona> +1 15:54:02 <slaweq> ok, I will check on Monday how it looks and will ping You ralonsoh to chair it or will cancel it 15:54:07 <ralonsoh> perfect 15:54:12 <mlavalle> +1 15:54:12 <slaweq> thx 15:54:19 <slaweq> that's all from me for today 15:54:28 <ykarel> +1 15:54:28 <slaweq> anything else You want to discuss, regarding our ci? 15:54:36 <mlavalle> nothing from me 15:55:02 <lajoskatona> nothing from me 15:55:09 <bcafarel> nothing here 15:55:20 <slaweq> ok, so thx for attending the meeting 15:55:24 <slaweq> and have a great week 15:55:25 <slaweq> o/ 15:55:26 <obondarev> o/ 15:55:26 <slaweq> #endmeeting