15:00:26 <slaweq> #startmeeting neutron_ci
15:00:26 <opendevmeet> Meeting started Tue Apr 12 15:00:26 2022 UTC and is due to finish in 60 minutes.  The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:26 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:26 <opendevmeet> The meeting name has been set to 'neutron_ci'
15:00:28 <slaweq> hi
15:00:32 <ralonsoh> hi
15:00:41 <slaweq> Grafana dashboard: https://grafana.opendev.org/d/f913631585/neutron-failure-rate?orgId=1
15:01:03 <mlavalle> o/
15:01:20 <lajoskatona> o/
15:02:44 <bcafarel> o/
15:02:59 <slaweq> ok, I think we can start
15:03:01 <slaweq> #topic Actions from previous meetings
15:03:04 <ykarel> o/
15:03:14 <slaweq> we have pretty long list of action items from last time
15:03:22 <slaweq> mlavalle will continue work on https://bugs.launchpad.net/neutron/+bug/1945283
15:03:41 <mlavalle> I haven't found instances of that in openserach
15:03:47 <mlavalle> opensearch
15:04:02 <mlavalle> maybe I don't know how to use it well, yet
15:04:13 <mlavalle> I'll keep searching and querying
15:05:18 <slaweq> ok
15:06:20 <slaweq> next one
15:06:22 <slaweq> slaweq to add extra logs to fullstack tests to investigate https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_366/830623/1/check/neutron-fullstack-with-uwsgi/366febd/testr_results.html
15:06:47 <slaweq> Patch https://review.opendev.org/c/openstack/neutron/+/834070
15:07:10 <slaweq> next one
15:07:11 <slaweq> slaweq to continue investigation on nova-neutron notifications issue https://bugs.launchpad.net/neutron/+bug/1963899
15:07:13 <slaweq> Fix merged  https://review.opendev.org/c/openstack/neutron/+/834852
15:07:27 <slaweq> next one
15:07:29 <slaweq> lajoskatona to create yoga templates in neutron-tempest-plugin jobs
15:08:28 <lajoskatona> that is merged
15:08:37 <slaweq> great, thx
15:08:48 <slaweq> next one
15:08:49 <slaweq> mlavalle to add note about rechecking with reason
15:08:59 <mlavalle> working on this
15:09:05 <mlavalle> I'll finish this week
15:09:16 <slaweq> ok
15:09:17 <lajoskatona> cool
15:09:24 <slaweq> #action mlavalle to add note about rechecking with reason
15:09:50 <slaweq> next one
15:09:52 <slaweq> ralonsoh to propose patch to run mysql functional tests in serial
15:09:56 <ralonsoh> merged
15:10:39 <slaweq> great
15:10:50 <slaweq> next one
15:10:57 <slaweq> slaweq to investigate "inprogress" tests which are interrupted somehow https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_a75/833620/4/check/neutron-functional-with-uwsgi/a75e59b/testr_results.html
15:11:04 <slaweq> Bug reported https://bugs.launchpad.net/neutron/+bug/1966394
15:11:13 <slaweq> Patch https://review.opendev.org/c/openstack/neutron/+/836426 merged, but we also have now https://review.opendev.org/c/openstack/neutron/+/836120/ so we can investigate memory usage a bit more
15:11:37 <slaweq> and the last one from my list
15:11:39 <slaweq> slaweq to report bug about https://12ef4cf37bb4f9cf8615-49968699300828e6c9b78fd54dff75ef.ssl.cf5.rackcdn.com/828687/8/gate/neutron-functional-with-uwsgi/3688faf/testr_results.html
15:11:44 <slaweq> Bug https://bugs.launchpad.net/neutron/+bug/1966035
15:12:13 <slaweq> obondarev is investigating that (at least trying to)
15:12:26 <slaweq> and that's all actions from last meeting
15:12:36 <slaweq> I think we can move on to the next topic now
15:12:37 <slaweq> #topic Stable branches
15:12:43 <slaweq> any updates bcafarel ?
15:13:23 <bcafarel> lajoskatona++ quickly fixed yoga branch last week, the rest is in good shape
15:13:38 <bcafarel> victoria is going EM soon I have to check the pending backports to get a last release
15:13:46 <lajoskatona> ++
15:13:53 <slaweq> ok
15:14:08 <slaweq> so we should also tag neutron-tempest-plugin for victoria when it will go EM
15:14:23 <bcafarel> great point yes indeed!
15:14:31 <lajoskatona> isn't that done together with tempest?
15:14:39 <slaweq> lajoskatona: I'm not sure
15:14:58 <lajoskatona> I can check with kopecmartin or gmann
15:15:08 <slaweq> thx lajoskatona
15:15:36 <slaweq> #action lajoskatona to check with QA team if neutron-tempest-plugin tag for victoria EM will be done together with tempest
15:15:50 <lajoskatona> +1
15:16:08 <gmann> ack, I will plan that for tempest this week sometime.
15:16:24 <slaweq> thx gmann
15:16:50 <lajoskatona> thanks
15:16:59 <slaweq> ok, I think we can move on now
15:17:06 <slaweq> #topic Stadium projects
15:17:10 <slaweq> lajoskatona: any updates here?
15:17:21 <slaweq> I checked weekly jobs today
15:17:26 <slaweq> https://zuul.openstack.org/buildsets?project=openstack%2Fnetworking-sfc&project=openstack%2Fnetworking-bagpipe&project=openstack%2Fnetworking-bgpvpn&project=openstack%2Fnetworking-odl&project=%09openstack%2Ftap-as-a-service&project=openstack%2Fneutron-vpnaas&project=openstack%2Fneutron-dynamic-routing&branch=master&pipeline=periodic-weekly&skip=0
15:17:41 <slaweq> and it seems that py39 job in networking-odl is failing
15:17:47 <slaweq> https://zuul.openstack.org/build/0412f7ed782d400598fe51a08949b08e
15:17:52 <slaweq> do You know about it?
15:17:56 <lajoskatona> I have tocheck that
15:18:16 <lajoskatona> another thing is that mnaser would like to work on vpnaas issues
15:18:31 <lajoskatona> I have to check with him what does this mean
15:18:42 <slaweq> that's good news
15:18:49 <mnaser> yeah, i can try and have a look at it, we use it in prod and we have a few atches we're running with locally (that are proposed upstream but would like to have them landed)
15:18:50 <slaweq> #action lajoskatona to check failing py39 job in networking-odl
15:19:28 <lajoskatona> mnaser: thanks
15:19:35 <bcafarel> that's good news
15:20:04 <slaweq> anything else regarding stadium projects' CI?
15:20:07 <lajoskatona> mnaser: would be good to have somebody who can check time-to-time if the jobs are passing or if there's new bugs
15:20:08 <slaweq> or can we move on?
15:20:15 <lajoskatona> I think we can
15:20:25 <mnaser> lajoskatona: i will try and make that effort, as long as i can get +2's and +W's in return :)
15:20:50 <slaweq> mnaser: sure :)
15:20:52 <slaweq> thx
15:20:55 <mlavalle> mnaser: I'll help with that
15:21:07 <mnaser> +1
15:21:12 <mlavalle> will it be you submitting the patches?
15:21:27 <mlavalle> mnaser: ^^^
15:21:41 <mnaser> mlavalle: probably yeah, at least to fix the func jobs
15:21:46 <lajoskatona> mnaser: we can add you I suppose to the core group for vpnaas also
15:22:20 <mnaser> i'd be okay with that too, i think it would be also good to clear up the backlog of stuff there
15:22:42 <mlavalle> still, he cannot +2 +W himself
15:23:15 <lajoskatona> mlavalle: that's true, but more eyes on hanging patches
15:23:25 <mlavalle> +1
15:23:42 <lajoskatona> we can discuss that offline, and continue the ci meeting I think, sorry
15:23:45 <mnaser> +1
15:23:58 <slaweq> sure, no problem at all :)
15:24:06 <slaweq> ok, lets move one with CI meeting then
15:24:11 <slaweq> #topic Grafana
15:24:15 <slaweq> https://grafana.opendev.org/d/f913631585/neutron-failure-rate
15:24:57 <slaweq> it seems for me that still our biggest problem is functional tests job in check queue :/
15:25:32 <slaweq> and in gate queue too
15:25:47 <ralonsoh> most of these failures are due to pyroute2
15:26:23 <lajoskatona> ralonsoh: do we have to use new version?
15:26:25 <slaweq> I saw many failures related to the timeout while waiting on the transition of the router
15:26:41 <ralonsoh> we should merge your retry patch
15:27:12 <ralonsoh> https://review.opendev.org/c/openstack/neutron/+/833015
15:28:08 <lajoskatona> ahh, ok, I forgot that, and thought it is merged :-)
15:28:16 <slaweq> approved :)
15:28:17 <ralonsoh> slaweq, btw, I'm working on a patch to replace the keepalived-state-change monitor
15:28:29 <ralonsoh> instead of using a python binary
15:28:32 <ralonsoh> to use a shell script
15:28:33 <slaweq> ralonsoh: great, I hope that will help too
15:28:47 <ralonsoh> (but I didn't have time to continue wiht that)
15:28:53 <slaweq> because that job is basically killing us currently
15:29:17 <slaweq> should we maybe mark those l3 ha tests as unstable temporary?
15:29:25 <slaweq> to make job more stable
15:29:27 <slaweq> wdyt?
15:29:31 <ralonsoh> I wouldn't
15:29:39 <ralonsoh> it is not blocking completely the CI
15:29:58 <slaweq> completly no, but number of rechecks is going high
15:30:03 <ralonsoh> ok then
15:30:42 <slaweq> ok, maybe it's not that bad yet
15:30:51 <slaweq> so lets wait for next week
15:32:38 <mlavalle> +1
15:33:40 <slaweq> ok, so for functional job failures, list of examples is in the etherpad https://etherpad.opendev.org/p/neutron-ci-meetings#L67
15:34:07 <slaweq> but most of them (or even all) are related to those pyroute2 issues, like e.g.:
15:34:12 <slaweq> https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_170/836120/5/gate/neutron-functional-with-uwsgi/1701f9a/testr_results.html
15:34:27 <slaweq> https://b5e0246d2a61241093f5-52f9a55e019565a745f2bb8d45b0d6fa.ssl.cf1.rackcdn.com/836725/2/check/neutron-functional-with-uwsgi/ad2bdce/testr_results.html
15:34:36 <slaweq> or to the issue with router transition to primary state:
15:34:43 <slaweq> https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_868/835786/1/check/neutron-functional-with-uwsgi/8688fff/testr_results.html
15:34:43 <slaweq> https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_0e9/837146/1/check/neutron-functional-with-uwsgi/0e9f5a8/testr_results.html
15:35:53 <slaweq> so I think we can move on to the next topic now
15:35:57 <slaweq> #topic Tempest/Scenario
15:36:11 <slaweq> here I (again) found one old issue with linuxbridge job:
15:36:15 <slaweq> https://4de9725805f5e105f3a0-621008116cc26c58a3190713e2eda213.ssl.cf1.rackcdn.com/836863/3/check/neutron-tempest-plugin-scenario-linuxbridge/95dd563/testr_results.html
15:36:22 <slaweq> failed to transition device to DOWN state
15:36:29 <slaweq> I think lajoskatona was trying to fix it some time ago
15:36:37 <slaweq> right lajoskatona ?
15:37:18 <lajoskatona> slaweq: yeah, I think I remember
15:37:47 <slaweq> now, giving what we discussed during ptg, should we simply skip that test in the linuxbridge job?
15:37:51 <slaweq> wdyt?
15:38:02 <lajoskatona> +1
15:38:12 <slaweq> ok, I will propose patch for that
15:38:13 <ralonsoh> with a NOTE
15:38:26 <ralonsoh> maybe we should add this to the docs first
15:38:26 <lajoskatona> this is the patch for the above issue I think: https://review.opendev.org/c/openstack/neutron/+/827728 I have to check it again when I have some time
15:38:30 <slaweq> #action slaweq to skip failing linuxbridge scenario test
15:38:31 <ralonsoh> just the warning
15:38:38 <slaweq> ralonsoh: sure
15:38:41 <ralonsoh> perfect
15:38:46 <lajoskatona> +1
15:39:17 <slaweq> I also started patch https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/836912
15:39:32 <slaweq> to remote neutron-tempest-plugin-api job and run those tests in the "scenario" jobs
15:39:39 <slaweq> but it's not ready yet
15:39:47 <slaweq> I need to check why some tests are failing there
15:40:01 <slaweq> but please be ready for review of it soon :)
15:40:05 <ralonsoh> sure
15:40:21 <slaweq> ok, and last topic from me today
15:40:27 <slaweq> #topic Periodic
15:40:37 <slaweq> Jobs results: http://zuul.openstack.org/buildsets?project=openstack%2Fneutron&pipeline=periodic&branch=master
15:40:41 <mlavalle> speaking of tempest, if you have a chabce, review https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/834411
15:40:42 <slaweq> we do have couple of issues to investigate there
15:40:49 <mlavalle> it's an easy one
15:40:56 <slaweq> sure mlavalle
15:41:17 <slaweq> getting back to periodic
15:41:23 <slaweq> neutron-ovn-tempest-ovs-master-fedora - broken again
15:41:27 <slaweq> anyone wants to check it?
15:41:28 <slaweq> :)
15:41:37 <mlavalle> again
15:41:38 <slaweq> and the same for neutron-ovs-tempest-slow
15:41:42 <mlavalle> I'll look at it
15:41:48 <slaweq> mlavalle: thx
15:42:04 <slaweq> #action mlavalle to check neutron-ovn-tempest-ovs-master-fedora periodic job
15:42:11 <slaweq> so I will check neutron-ovs-tempest-slow
15:42:20 <slaweq> #action slaweq to check neutron-ovs-tempest-slow periodic job
15:42:40 <slaweq> there is also "propose-translation-update" which has some problem with pyroute2 version
15:42:57 <slaweq> https://zuul.openstack.org/build/8a44d67db5474acdb1cfce78d4ba04cd
15:43:06 <slaweq> anyone have cycles to check it this week?
15:43:13 <ykarel> i can check
15:43:18 <slaweq> thx ykarel
15:43:40 <slaweq> #action ykarel to check failing propose-translation-update periodic job
15:44:04 <slaweq> ok, and that's all what I had for today
15:44:12 <slaweq> do You have any other topics to discuss today?
15:44:20 <ralonsoh> ah, this is the doc/requirements.txt file
15:44:21 <slaweq> if not, I will give You few minutes back
15:44:32 <mlavalle> well, not a short one but we didn't get to the top of the hour
15:44:35 <mlavalle> not bad
15:44:38 <ralonsoh> just one little topic
15:44:40 <ralonsoh> https://review.opendev.org/c/openstack/neutron-lib/+/828738/18/.zuul.yaml
15:44:45 <ralonsoh> this is an example
15:44:59 <ralonsoh> I know we are always talking about reducing CI jobs
15:45:14 <ralonsoh> but having tempest jobs in n-lib is very useful
15:45:24 <slaweq> I agree
15:45:30 <ralonsoh> do you agree with adding them to the n-lib zuul deifnition?
15:45:34 <slaweq> that's why we added neutron-tempest-plugin-api job there some time ago
15:45:51 <slaweq> but if You think that adding scenario jobs too also make sense, I'm ok with that
15:45:59 <mlavalle> me too
15:46:05 <ralonsoh> I can add only OVS native and OVN
15:46:17 <lajoskatona> agree
15:46:18 <ralonsoh> skipping hybrid and Linux Bridge
15:46:20 <ralonsoh> perfect
15:46:21 <ykarel> +1
15:46:22 <ralonsoh> thanks a lot
15:46:22 <slaweq> actually if You will remove neutron-tempest-plugin-api from check queue in that Your patch, it will help me with my "consolidation" patch :)
15:46:26 <mlavalle> we should try to reduce the number of jobs but not to the point where we are opening holes
15:46:42 <mlavalle> in our testing
15:46:44 <ralonsoh> perfect
15:47:09 <slaweq> #action ralonsoh to add neutron-tempest-plugin jobs to the neutron-lib CI
15:47:18 <slaweq> thx ralonsoh for bringing it up
15:47:37 <ykarel> small one: https://review.opendev.org/c/openstack/neutron/+/837552 to fix arm64 unit test jobs
15:48:20 <lajoskatona> ykarel: I try to check if the pkg is needed for the API tests for NDP proxy
15:48:38 <lajoskatona> but let's remove it from bindep.txt now
15:49:01 <ykarel> lajoskatona, yes bindep.txt is not getting used in devstack jobs, so i think it's fine
15:49:53 <ralonsoh> hold on
15:50:09 <ralonsoh> ok, but it is better to have an explanation from the commiter
15:50:37 * mlavalle just W+ it
15:50:50 <lajoskatona> ralonsoh: you mean from NDP proxy dvelopers?
15:51:14 <ralonsoh> I think this is a leftover, this is not needed for NDP, if I'm not wrong
15:51:24 <ralonsoh> in any case, this is a feature in development now
15:51:29 <ralonsoh> we can read it if needed
15:51:44 <slaweq> ++
15:52:07 <lajoskatona> ok, ack
15:52:52 <slaweq> ok, I think that with this we can conclude the meeting for today :)
15:52:57 <slaweq> thx for attending
15:52:59 <ralonsoh> bye!
15:53:07 <mlavalle> o/
15:53:08 <slaweq> have a great week and see You online
15:53:10 <lajoskatona> bye
15:53:13 <slaweq> #endmeeting