15:00:21 <slaweq> #startmeeting neutron_ci
15:00:25 <slaweq> hi
15:00:26 <openstack> Meeting started Wed Aug 26 15:00:21 2020 UTC and is due to finish in 60 minutes.  The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:27 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:29 <openstack> The meeting name has been set to 'neutron_ci'
15:00:35 <bcafarel> o/
15:00:37 <slaweq> and please give me 3 minutes before we start
15:01:34 <ralonsoh> hi
15:02:35 <slaweq> ok, I'm back
15:02:39 <slaweq> and I think we can start
15:02:46 <slaweq> Grafana dashboard: http://grafana.openstack.org/dashboard/db/neutron-failure-rate
15:02:58 <slaweq> #topic Actions from previous meetings
15:03:06 <slaweq> ralonsoh to check timing out neutron-ovn-tempest-full-multinode-ovs-master jobs
15:03:31 <ralonsoh> slaweq, sorry but I didn't spend time on this
15:03:49 <ralonsoh> (and at this point I barely remember it)
15:04:25 <slaweq> ralonsoh: bug is here https://bugs.launchpad.net/neutron/+bug/1886807
15:04:26 <openstack> Launchpad bug 1886807 in neutron "neutron-ovn-tempest-full-multinode-ovs-master job is failing 100% times" [High,Confirmed] - Assigned to Maciej Jozefczyk (maciejjozefczyk)
15:04:33 <slaweq> I think it is failing like that still
15:04:40 <lajoskatona> o/
15:04:54 <ralonsoh> slaweq, I'll check it this week
15:04:59 <slaweq> thx ralonsoh
15:05:17 <slaweq> #action ralonsoh to check timing out neutron-ovn-tempest-full-multinode-ovs-master jobs - bug https://bugs.launchpad.net/neutron/+bug/1886807
15:05:26 <slaweq> next one
15:05:28 <slaweq> slaweq to move uwsgi jobs to periodic queue and promote -uwsgi ones to be gating
15:05:32 <slaweq> Patch https://review.opendev.org/745822
15:05:47 <slaweq> please review it :)
15:05:53 <slaweq> and the last one on my list:
15:05:57 <slaweq> maciejjozefczyk to check https://bugs.launchpad.net/neutron/+bug/1890445
15:05:58 <openstack> Launchpad bug 1890445 in neutron "[ovn] Tempest test test_update_router_admin_state failing very often" [Critical,Confirmed]
15:06:20 <slaweq> I don't think maciek will look into this so we will need new volunteer
15:06:36 <slaweq> I will ask jlibosva and lucasgomes if they can take a look into that one
15:07:07 <ralonsoh> (thanks)
15:07:16 <slaweq> #action slaweq to ask jlibosva and lucasgomes if they can check https://bugs.launchpad.net/neutron/+bug/1890445
15:07:17 <openstack> Launchpad bug 1890445 in neutron "[ovn] Tempest test test_update_router_admin_state failing very often" [Critical,Confirmed]
15:07:38 <slaweq> that are all actions from last meeting
15:07:48 <slaweq> so lets move to the next topic
15:07:49 <slaweq> #topic Switch to Ubuntu Focal
15:07:55 <slaweq> any updates about that one?
15:08:34 <ralonsoh> bcafarel, pushed a change to update the lower contrains
15:08:44 <ralonsoh> https://review.opendev.org/#/c/748168/
15:09:17 <bcafarel> sorry it took some trials to get this one passing, it is OK locally now (and testing in progress in https://review.opendev.org/#/c/734304/ )
15:10:18 <slaweq> ok, so with that we should be good with functional/fullstack jobs to be moved to focal
15:10:20 <slaweq> right?
15:10:39 <ralonsoh> I think so
15:10:45 <bcafarel> yes they work fine with ncat installed, just need to pass that pesky cI
15:10:49 <slaweq> good
15:11:03 <bcafarel> although we could just drop depends-on in https://review.opendev.org/#/c/734304/ and merge the change, we would be ready then
15:11:22 <slaweq> for other jobs we need this https://review.opendev.org/#/c/731207/ to be merged, right?
15:12:00 <lajoskatona> not much activity on it recently
15:12:03 <slaweq> or can we simply change nodeset in e.g. neutron-tempest-plugin jobs now and move on with that?
15:12:07 <slaweq> wdyt?
15:12:41 <ralonsoh> I don't think this will work
15:12:46 <ralonsoh> we need this devstack change
15:13:05 <slaweq> why?
15:13:17 <bcafarel> https://review.opendev.org/#/c/738163/ still has a few red jobs (in addition to ft/functional/lower constraints)
15:13:23 <ralonsoh> hmmmm you are right
15:13:30 <ralonsoh> just changing the node set could work
15:13:33 <ralonsoh> yes, you are right
15:13:42 <ralonsoh> we can try it in a DNM patch
15:13:48 <lajoskatona> +1
15:13:52 <slaweq> let me try to propose patch for neutron-tempest-plugin to change that and see
15:13:55 <ralonsoh> sure!
15:14:09 <slaweq> #action slaweq to propose neutron-tempest-plugin switch to focal nodes
15:14:19 <lajoskatona> if devstack change wil be merged we can remove the nodesets
15:14:27 <slaweq> lajoskatona: right
15:14:31 <ralonsoh> yeah, correct
15:15:16 <slaweq> lajoskatona: what about https://review.opendev.org/#/c/736703/ - is that ready to go?
15:16:17 <lajoskatona> I check again rally
15:16:41 <lajoskatona> but as I see it is not worse than with bionic
15:16:52 <slaweq> lajoskatona: great
15:17:00 <slaweq> so lets review this patch and move on with it
15:17:02 <slaweq> thx
15:17:18 <slaweq> I think that's all regarding migration to Focal
15:17:24 <slaweq> so lets move on to the next topic
15:17:26 <slaweq> #topic Stadium projects
15:17:45 <slaweq> anything You want to discuss regarding stadium's ci?
15:18:00 <ralonsoh> no
15:18:06 <lajoskatona> not really
15:18:12 <bcafarel> nothign here
15:18:25 <slaweq> ok, that's fast :)
15:18:33 <slaweq> lets move to the next topic then
15:18:39 <slaweq> #topic Stable branches
15:18:42 <slaweq> anything here?
15:18:53 <slaweq> IMO ci of stable branches is pretty ok still
15:18:54 <bcafarel> not much either :) (still catching up from backlog)
15:19:06 <bcafarel> yep from what I saw everything seems to be in good shape
15:19:29 <slaweq> great, at least ci of stable branches is stable :)
15:20:34 <slaweq> ok, so lets move on
15:20:36 <slaweq> #topic Grafana
15:20:50 <slaweq> today I added neutron-tempest-plugin-scenario-ovn job to the grafana https://review.opendev.org/748223
15:20:57 <slaweq> as we are missing it there
15:21:59 <slaweq> except that, it looks pretty good now
15:22:09 <slaweq> I don't see any major problems with any of the jobs
15:22:40 <slaweq> maybe except pep8 failures like:
15:22:42 <slaweq> https://zuul.opendev.org/t/openstack/build/6c8fbf9b97b44139bf1d70b9c85455bb
15:22:51 <slaweq> did You saw something similar recently?
15:23:42 <bcafarel> it may be caused by recent pylint bump (so existing reviews can start to fail pep8)
15:24:20 <ralonsoh> slaweq, if I'm not wrong
15:24:32 <ralonsoh> this is when you implement a DictModel
15:24:50 <ralonsoh> you need to define the parameters passed to this dict
15:25:01 <ralonsoh> (very bad description)
15:25:41 <slaweq> ok, so we will need to fix that issue
15:25:47 <ralonsoh> let me check this patch and I'll try to find what is happening
15:25:58 <ralonsoh> do you have a patch?
15:26:02 <slaweq> https://review.opendev.org/#/c/715482/21/neutron/agent/linux/dhcp.py
15:26:08 <slaweq> it failed here
15:26:30 <slaweq> those lines were not touched by the patch but file was changed
15:26:53 <ralonsoh> slaweq, ok, I'll debug this today or tomorrow morning
15:27:16 <slaweq> thx ralonsoh
15:27:33 <slaweq> #action ralonsoh to check issue with pep8 failures like https://zuul.opendev.org/t/openstack/build/6c8fbf9b97b44139bf1d70b9c85455bb
15:27:50 <slaweq> that's all what I have regarding grafana
15:28:12 <slaweq> do You have anything else or can we move on?
15:28:19 <ralonsoh> I'm ok
15:28:26 <bcafarel> all good
15:28:56 <slaweq> ok, lets move on
15:28:58 <slaweq> #topic Tempest/Scenario
15:29:09 <slaweq> Due to recent issue https://bugs.launchpad.net/neutron/+bug/1890842 I would like to add neutron-tempest-plugin-api to be running on neutron-lib patches,
15:29:10 <openstack> Launchpad bug 1890842 in neutron "API test test_create_port_without_propagate_uplink_status fails with neutron-lib 2.5.0" [Critical,Fix released] - Assigned to Slawek Kaplonski (slaweq)
15:29:10 <slaweq> Patch proposed https://review.opendev.org/745830
15:29:17 <slaweq> this patch has already 3x +2
15:29:22 <slaweq> but needs +W :)
15:29:34 <slaweq> can one of You +W it?
15:29:36 <ralonsoh> I'll do it
15:29:39 <slaweq> thx a lot
15:29:46 <ralonsoh> and thanks for this patch
15:29:53 <slaweq> the other issue which we have is
15:29:55 <slaweq> https://bugs.launchpad.net/neutron/+bug/1892017
15:29:55 <openstack> Launchpad bug 1892017 in neutron "Neutron server logs are too big in the gate jobs" [High,Fix released] - Assigned to Slawek Kaplonski (slaweq)
15:30:10 <slaweq> melwitt reported us that q-svc logs are very big
15:30:23 <slaweq> I already proposed https://review.opendev.org/#/c/746714/ and it's merged
15:30:36 <slaweq> this reduces log file for about 50%
15:30:43 <slaweq> but it's still around 40MB
15:31:00 <slaweq> so infra guys for now disabled indexing of q-svc logs in logstash
15:31:07 <ralonsoh> yeah, the resource extend logging was excesive in this case
15:31:20 <slaweq> I will try to check there what else we can reduce there
15:31:35 <slaweq> but if You have any ideas, feel free to propose patches
15:31:47 <ralonsoh> (remove debug logs!!)
15:32:09 <slaweq> ralonsoh: yeah
15:32:25 <slaweq> but that may make our life harder :P
15:32:54 <slaweq> next thing
15:33:07 <slaweq> ralonsoh found and reported today issue https://bugs.launchpad.net/neutron/+bug/1893031
15:33:08 <openstack> Launchpad bug 1893031 in neutron "[tempest] "test_dns_domain_and_name" is failing 100% of times" [Undecided,New]
15:33:22 <slaweq> it should be fixed with https://review.opendev.org/#/c/748140/ which is now in the gate
15:33:27 <ralonsoh> correct
15:34:20 <slaweq> from other issues I found today one failure of qos test in linuxbridge job
15:34:22 <slaweq> https://fb1e48505d4c8944956b-dc45e57a445c28f373928001a4323ec4.ssl.cf5.rackcdn.com/696926/23/check/neutron-tempest-plugin-scenario-linuxbridge/086d79c/testr_results.html
15:34:56 <slaweq> did You saw something similar before?
15:35:18 <ralonsoh> let me check
15:35:46 <ralonsoh> bytes_per_second = 385961, expected_bw = 192000
15:35:55 <ralonsoh> hmmmm twice the speed
15:36:14 <slaweq> yes, that's why I raised it here
15:36:23 <ralonsoh> no sorry, I never saw this
15:36:24 <slaweq> because it seems "interesting" for me :)
15:36:32 <ralonsoh> and LB is the most stable backend
15:36:36 <ralonsoh> (I never said that)
15:36:38 <ralonsoh> in QoS
15:37:13 <slaweq> xD
15:37:17 <ralonsoh> is this the first time you see it?
15:37:25 <ralonsoh> or is this something recurrent?
15:37:26 <slaweq> yes
15:37:29 <ralonsoh> ok
15:37:54 <ralonsoh> I'll save it, just in case
15:38:51 <slaweq> ok
15:39:08 <slaweq> I will try to look deeper into logs, maybe I will find something interesting there
15:39:24 <slaweq> #slaweq to check logs of qos failure in LB job https://fb1e48505d4c8944956b-dc45e57a445c28f373928001a4323ec4.ssl.cf5.rackcdn.com/696926/23/check/neutron-tempest-plugin-scenario-linuxbridge/086d79c/testr_results.html
15:40:11 <slaweq> speaking about qos and scenario tests
15:40:18 <slaweq> long time ago I proposed patch https://review.opendev.org/#/c/721235/
15:40:27 <slaweq> please review it when You will have some time
15:40:42 <slaweq> and that's all from me for today
15:40:49 <slaweq> periodic jobs seems to be working fine
15:41:04 <slaweq> anything else You want to talk about today?
15:41:14 <ralonsoh> one quick question
15:41:21 <ralonsoh> during those two weeks
15:41:39 <ralonsoh> with the oslo.privsep change (un-monkey patching the daemon(
15:41:52 <ralonsoh> and the latests pyroute2 lib version
15:42:06 <ralonsoh> did you notice the functional/fullstack tests more stable?
15:42:13 <ralonsoh> in other works
15:42:14 <ralonsoh> words
15:42:19 <ralonsoh> did you see timeouts?
15:42:54 <slaweq> looking at http://grafana.openstack.org/d/Hj5IHcSmz/neutron-failure-rate?viewPanel=24&orgId=1&from=now-30d&to=now I don't think it is more stable than it was
15:43:04 <slaweq> maybe slightly better
15:43:41 <slaweq> fullstack seems to be pretty good recently
15:43:42 <ralonsoh> yes, a 0.001% better
15:43:43 <ralonsoh> hehehehe
15:43:46 <slaweq> :)
15:43:47 <bcafarel> :)
15:44:00 <ralonsoh> once we have George
15:44:08 <slaweq> but even if it's 0.001% better, it's step in good direction
15:44:11 <ralonsoh> we can think about moving some fullstack tests to George
15:44:23 <slaweq> yes, I was reviewing George patches today
15:44:25 <ralonsoh> using containers will help to isolate some noisy tests
15:44:28 <slaweq> looks good and very interesting
15:44:32 <ralonsoh> a lot!!
15:44:45 <ralonsoh> that's what fullstack test should have been implemented
15:44:51 <ralonsoh> but this is easy to say this now
15:44:57 <slaweq> yes
15:46:34 <slaweq> ok, if You don't have anything else for today, lets finish this meeting earlier
15:46:39 <slaweq> thx for attending
15:46:44 <ralonsoh> bye!
15:46:44 <slaweq> #endmeeting