15:01:07 <slaweq> #startmeeting neutron_ci
15:01:08 <openstack> Meeting started Tue Dec 15 15:01:07 2020 UTC and is due to finish in 60 minutes.  The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:09 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:01:12 <slaweq> hi again :)
15:01:12 <openstack> The meeting name has been set to 'neutron_ci'
15:01:31 <ralonsoh> hi
15:01:34 <bcafarel> not holidays yet, hi again :)
15:01:45 <slaweq> bcafarel: unfortunatelly not :/
15:01:52 <slaweq> but it's close
15:01:53 <slaweq> :)
15:02:15 <bcafarel> just a few rechecks away
15:02:18 <slaweq> lajoskatona will not be here today
15:02:25 <slaweq> so lets start
15:02:33 <slaweq> we don't have any actions from last week
15:02:39 <slaweq> ahh, sorry
15:02:44 <slaweq> Grafana dashboard: http://grafana.openstack.org/dashboard/db/neutron-failure-rate
15:02:55 <slaweq> #topic Stadium projects
15:03:12 <slaweq> anything related to stadium projects for today?
15:03:28 <ralonsoh> no
15:03:49 <bcafarel> nothing from me
15:04:06 <slaweq> usually lajos got some updates but when I was reviewing some patches I didn't saw anything urgent there
15:04:20 <slaweq> #topic Stable branches
15:04:31 <slaweq> bcafarel: we already spoke about pip and l-c issues
15:04:44 <bcafarel> slaweq: yes, busy week, and not in a good way :(
15:05:02 <slaweq> IIUC You have patches which should unblock our stable branches, right?
15:05:19 <bcafarel> yes, overall most branches are probably not passing CI right now but hopefully this will be better in 2021
15:05:29 <slaweq> LOL
15:05:55 <bcafarel> I will move them under same gerrit topic so we can track them more easily
15:06:20 <bcafarel> also there was a separate bandit issue (workarounded in devstack), and apparently rocky broken https://review.opendev.org/c/openstack/stackviz/+/767063
15:07:31 <slaweq> k
15:08:05 <bcafarel> https://review.opendev.org/q/topic:fix-pip+project:openstack/neutron for pip fixes
15:08:08 <slaweq> so I think we will be good with those as bcafarel is on it already :)
15:08:41 <slaweq> thx bcafarel
15:08:52 <slaweq> anything else or we can move on?
15:10:00 <bcafarel> nothing else, hopefully we can merge this "Upgrade RPC version of SecurityGroup*Rpc" soon after CI is green again
15:10:13 <slaweq> yes, I proposed backports of that patch already
15:11:04 <slaweq> ok, lets move on
15:11:06 <slaweq> #topic Grafana
15:11:12 <slaweq> http://grafana.openstack.org/dashboard/db/neutron-failure-rate
15:12:07 <slaweq> As many of my recent changes to our jobs are now merged, I think I need to update grafana dashboard config
15:12:20 <slaweq> #action slaweq to update grafana dashboard
15:13:16 <bcafarel> well removed jobs show at 0% failure right? so it makes nicer dashboards :)
15:13:39 <slaweq> bcafarel: true
15:13:50 <slaweq> but I don't think that it's what we really want :P
15:15:10 <slaweq> in general I think that it looks better this week in master branch
15:15:29 <slaweq> we recently merged some fixes for functional and fullstack tests
15:15:37 <slaweq> so I hope it will be better with those jobs too
15:15:45 <slaweq> scenario jobs are pretty ok now
15:15:56 <bcafarel> nice
15:17:17 <slaweq> ok, lets talk about specific jobs
15:17:19 <slaweq> #topic fullstack/functional
15:17:40 <slaweq> we have opened bug     https://bugs.launchpad.net/neutron/+bug/1903008
15:17:41 <openstack> Launchpad bug 1903008 in neutron "Create network failed during functional test" [Critical,Confirmed]
15:17:54 <slaweq> and I saw at list 2 or 3 times this week that it happened again
15:18:02 <slaweq> https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_78e/764913/2/check/neutron-functional-with-uwsgi/78eb5fc/testr_results.html
15:18:03 <slaweq> https://zuul.opendev.org/t/openstack/build/a24070f28271446090761135b83812b7
15:18:20 <slaweq> we need to investigate that issue as it happens on various random tests
15:18:33 <ralonsoh> -->
15:18:33 <ralonsoh> https://review.opendev.org/c/openstack/neutron/+/766920
15:18:35 <slaweq> so there is no way to easy mark some test(s) as unstable
15:18:40 <ralonsoh> I though that was the problem
15:19:02 <ralonsoh> but I really don't know what is not in this trasaction: the port register or the standard attrs register
15:19:30 <slaweq> this patch didn't solve it - still same failure 3 times :/
15:19:36 <ralonsoh> yes...
15:20:05 <slaweq> ok, I will try to look at that issue and maybe I will add some logs or things like that to investigate it more
15:20:15 <slaweq> I will use Your patch ralonsoh as a base
15:20:21 <ralonsoh> perfect
15:20:27 <slaweq> to run many functional tests jobs in the same time
15:21:00 <slaweq> I also saw once some db migration test failure https://81b3bd201171c89feee7-a605dbf95134d478b50bec2dfa092555.ssl.cf2.rackcdn.com/766875/2/check/neutron-functional-with-uwsgi/146dc99/testr_results.html
15:22:24 <slaweq> at first glance it looks for me like maybe mysql was killed by oom
15:24:51 <slaweq> but it isn't hitting us very often so I will just report it to not forget
15:25:02 <slaweq> and we can get back to it when we will fix more urgent things
15:25:22 <slaweq> are You ok with that?
15:25:48 <ralonsoh> perfect
15:26:00 <slaweq> ok, lets move on
15:26:03 <slaweq> #topic Tempest/Scenario
15:26:21 <slaweq> here I still see those failures in neutron-ovn-tempest-ovs-release-ipv6-only
15:26:31 <slaweq> for now I proposed patch to skip those tests in that job
15:26:36 <slaweq> https://review.opendev.org/c/openstack/neutron/+/765070
15:26:48 <slaweq> and I'm also working on reproduce of that issue locally
15:27:01 <slaweq> so hopefully I will find something soon
15:27:41 <slaweq> and that's all from my side for scenario jobs
15:27:45 <slaweq> anything else You want to add?
15:28:52 <ralonsoh> no
15:29:10 <slaweq> ok, so I have one more thing only
15:29:13 <slaweq> #topic grenade
15:29:20 <bcafarel> small topic
15:29:29 <slaweq> I recently pushed patch https://review.opendev.org/c/openstack/neutron/+/766760
15:29:40 <slaweq> it is some short guide about how to debug grenade jobs
15:29:46 <slaweq> thx haleyb  and njohnston for review of it
15:29:54 <slaweq> I will address those comments soon
15:30:02 <slaweq> and please take a look then once again
15:30:11 <slaweq> that's all from my side
15:30:18 <slaweq> bcafarel: go on
15:30:23 <slaweq> with Your topic :)
15:30:52 <bcafarel> slaweq: no, nothing from me , I was just commenting that grenade usually is "small" topic :)
15:30:59 <slaweq> ahh :)
15:31:00 <slaweq> ok
15:31:04 <slaweq> so that's all from my side
15:31:23 <slaweq> I will cancel next 2 meetings - are You ok with that?
15:31:42 <bcafarel> same +1 as team meeting yes
15:31:48 <slaweq> :)
15:32:33 <slaweq> so, once again, have a great holidays and see You all on the next ci meeting in 2021 :)
15:32:38 <ralonsoh> you too!
15:32:38 <slaweq> thx for attending today
15:32:42 <slaweq> #endmeeting