15:00:21 #startmeeting neutron_ci 15:00:25 hi 15:00:26 Meeting started Wed Aug 26 15:00:21 2020 UTC and is due to finish in 60 minutes. The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:00:27 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:00:29 The meeting name has been set to 'neutron_ci' 15:00:35 o/ 15:00:37 and please give me 3 minutes before we start 15:01:34 hi 15:02:35 ok, I'm back 15:02:39 and I think we can start 15:02:46 Grafana dashboard: http://grafana.openstack.org/dashboard/db/neutron-failure-rate 15:02:58 #topic Actions from previous meetings 15:03:06 ralonsoh to check timing out neutron-ovn-tempest-full-multinode-ovs-master jobs 15:03:31 slaweq, sorry but I didn't spend time on this 15:03:49 (and at this point I barely remember it) 15:04:25 ralonsoh: bug is here https://bugs.launchpad.net/neutron/+bug/1886807 15:04:26 Launchpad bug 1886807 in neutron "neutron-ovn-tempest-full-multinode-ovs-master job is failing 100% times" [High,Confirmed] - Assigned to Maciej Jozefczyk (maciejjozefczyk) 15:04:33 I think it is failing like that still 15:04:40 o/ 15:04:54 slaweq, I'll check it this week 15:04:59 thx ralonsoh 15:05:17 #action ralonsoh to check timing out neutron-ovn-tempest-full-multinode-ovs-master jobs - bug https://bugs.launchpad.net/neutron/+bug/1886807 15:05:26 next one 15:05:28 slaweq to move uwsgi jobs to periodic queue and promote -uwsgi ones to be gating 15:05:32 Patch https://review.opendev.org/745822 15:05:47 please review it :) 15:05:53 and the last one on my list: 15:05:57 maciejjozefczyk to check https://bugs.launchpad.net/neutron/+bug/1890445 15:05:58 Launchpad bug 1890445 in neutron "[ovn] Tempest test test_update_router_admin_state failing very often" [Critical,Confirmed] 15:06:20 I don't think maciek will look into this so we will need new volunteer 15:06:36 I will ask jlibosva and lucasgomes if they can take a look into that one 15:07:07 (thanks) 15:07:16 #action slaweq to ask jlibosva and lucasgomes if they can check https://bugs.launchpad.net/neutron/+bug/1890445 15:07:17 Launchpad bug 1890445 in neutron "[ovn] Tempest test test_update_router_admin_state failing very often" [Critical,Confirmed] 15:07:38 that are all actions from last meeting 15:07:48 so lets move to the next topic 15:07:49 #topic Switch to Ubuntu Focal 15:07:55 any updates about that one? 15:08:34 bcafarel, pushed a change to update the lower contrains 15:08:44 https://review.opendev.org/#/c/748168/ 15:09:17 sorry it took some trials to get this one passing, it is OK locally now (and testing in progress in https://review.opendev.org/#/c/734304/ ) 15:10:18 ok, so with that we should be good with functional/fullstack jobs to be moved to focal 15:10:20 right? 15:10:39 I think so 15:10:45 yes they work fine with ncat installed, just need to pass that pesky cI 15:10:49 good 15:11:03 although we could just drop depends-on in https://review.opendev.org/#/c/734304/ and merge the change, we would be ready then 15:11:22 for other jobs we need this https://review.opendev.org/#/c/731207/ to be merged, right? 15:12:00 not much activity on it recently 15:12:03 or can we simply change nodeset in e.g. neutron-tempest-plugin jobs now and move on with that? 15:12:07 wdyt? 15:12:41 I don't think this will work 15:12:46 we need this devstack change 15:13:05 why? 15:13:17 https://review.opendev.org/#/c/738163/ still has a few red jobs (in addition to ft/functional/lower constraints) 15:13:23 hmmmm you are right 15:13:30 just changing the node set could work 15:13:33 yes, you are right 15:13:42 we can try it in a DNM patch 15:13:48 +1 15:13:52 let me try to propose patch for neutron-tempest-plugin to change that and see 15:13:55 sure! 15:14:09 #action slaweq to propose neutron-tempest-plugin switch to focal nodes 15:14:19 if devstack change wil be merged we can remove the nodesets 15:14:27 lajoskatona: right 15:14:31 yeah, correct 15:15:16 lajoskatona: what about https://review.opendev.org/#/c/736703/ - is that ready to go? 15:16:17 I check again rally 15:16:41 but as I see it is not worse than with bionic 15:16:52 lajoskatona: great 15:17:00 so lets review this patch and move on with it 15:17:02 thx 15:17:18 I think that's all regarding migration to Focal 15:17:24 so lets move on to the next topic 15:17:26 #topic Stadium projects 15:17:45 anything You want to discuss regarding stadium's ci? 15:18:00 no 15:18:06 not really 15:18:12 nothign here 15:18:25 ok, that's fast :) 15:18:33 lets move to the next topic then 15:18:39 #topic Stable branches 15:18:42 anything here? 15:18:53 IMO ci of stable branches is pretty ok still 15:18:54 not much either :) (still catching up from backlog) 15:19:06 yep from what I saw everything seems to be in good shape 15:19:29 great, at least ci of stable branches is stable :) 15:20:34 ok, so lets move on 15:20:36 #topic Grafana 15:20:50 today I added neutron-tempest-plugin-scenario-ovn job to the grafana https://review.opendev.org/748223 15:20:57 as we are missing it there 15:21:59 except that, it looks pretty good now 15:22:09 I don't see any major problems with any of the jobs 15:22:40 maybe except pep8 failures like: 15:22:42 https://zuul.opendev.org/t/openstack/build/6c8fbf9b97b44139bf1d70b9c85455bb 15:22:51 did You saw something similar recently? 15:23:42 it may be caused by recent pylint bump (so existing reviews can start to fail pep8) 15:24:20 slaweq, if I'm not wrong 15:24:32 this is when you implement a DictModel 15:24:50 you need to define the parameters passed to this dict 15:25:01 (very bad description) 15:25:41 ok, so we will need to fix that issue 15:25:47 let me check this patch and I'll try to find what is happening 15:25:58 do you have a patch? 15:26:02 https://review.opendev.org/#/c/715482/21/neutron/agent/linux/dhcp.py 15:26:08 it failed here 15:26:30 those lines were not touched by the patch but file was changed 15:26:53 slaweq, ok, I'll debug this today or tomorrow morning 15:27:16 thx ralonsoh 15:27:33 #action ralonsoh to check issue with pep8 failures like https://zuul.opendev.org/t/openstack/build/6c8fbf9b97b44139bf1d70b9c85455bb 15:27:50 that's all what I have regarding grafana 15:28:12 do You have anything else or can we move on? 15:28:19 I'm ok 15:28:26 all good 15:28:56 ok, lets move on 15:28:58 #topic Tempest/Scenario 15:29:09 Due to recent issue https://bugs.launchpad.net/neutron/+bug/1890842 I would like to add neutron-tempest-plugin-api to be running on neutron-lib patches, 15:29:10 Launchpad bug 1890842 in neutron "API test test_create_port_without_propagate_uplink_status fails with neutron-lib 2.5.0" [Critical,Fix released] - Assigned to Slawek Kaplonski (slaweq) 15:29:10 Patch proposed https://review.opendev.org/745830 15:29:17 this patch has already 3x +2 15:29:22 but needs +W :) 15:29:34 can one of You +W it? 15:29:36 I'll do it 15:29:39 thx a lot 15:29:46 and thanks for this patch 15:29:53 the other issue which we have is 15:29:55 https://bugs.launchpad.net/neutron/+bug/1892017 15:29:55 Launchpad bug 1892017 in neutron "Neutron server logs are too big in the gate jobs" [High,Fix released] - Assigned to Slawek Kaplonski (slaweq) 15:30:10 melwitt reported us that q-svc logs are very big 15:30:23 I already proposed https://review.opendev.org/#/c/746714/ and it's merged 15:30:36 this reduces log file for about 50% 15:30:43 but it's still around 40MB 15:31:00 so infra guys for now disabled indexing of q-svc logs in logstash 15:31:07 yeah, the resource extend logging was excesive in this case 15:31:20 I will try to check there what else we can reduce there 15:31:35 but if You have any ideas, feel free to propose patches 15:31:47 (remove debug logs!!) 15:32:09 ralonsoh: yeah 15:32:25 but that may make our life harder :P 15:32:54 next thing 15:33:07 ralonsoh found and reported today issue https://bugs.launchpad.net/neutron/+bug/1893031 15:33:08 Launchpad bug 1893031 in neutron "[tempest] "test_dns_domain_and_name" is failing 100% of times" [Undecided,New] 15:33:22 it should be fixed with https://review.opendev.org/#/c/748140/ which is now in the gate 15:33:27 correct 15:34:20 from other issues I found today one failure of qos test in linuxbridge job 15:34:22 https://fb1e48505d4c8944956b-dc45e57a445c28f373928001a4323ec4.ssl.cf5.rackcdn.com/696926/23/check/neutron-tempest-plugin-scenario-linuxbridge/086d79c/testr_results.html 15:34:56 did You saw something similar before? 15:35:18 let me check 15:35:46 bytes_per_second = 385961, expected_bw = 192000 15:35:55 hmmmm twice the speed 15:36:14 yes, that's why I raised it here 15:36:23 no sorry, I never saw this 15:36:24 because it seems "interesting" for me :) 15:36:32 and LB is the most stable backend 15:36:36 (I never said that) 15:36:38 in QoS 15:37:13 xD 15:37:17 is this the first time you see it? 15:37:25 or is this something recurrent? 15:37:26 yes 15:37:29 ok 15:37:54 I'll save it, just in case 15:38:51 ok 15:39:08 I will try to look deeper into logs, maybe I will find something interesting there 15:39:24 #slaweq to check logs of qos failure in LB job https://fb1e48505d4c8944956b-dc45e57a445c28f373928001a4323ec4.ssl.cf5.rackcdn.com/696926/23/check/neutron-tempest-plugin-scenario-linuxbridge/086d79c/testr_results.html 15:40:11 speaking about qos and scenario tests 15:40:18 long time ago I proposed patch https://review.opendev.org/#/c/721235/ 15:40:27 please review it when You will have some time 15:40:42 and that's all from me for today 15:40:49 periodic jobs seems to be working fine 15:41:04 anything else You want to talk about today? 15:41:14 one quick question 15:41:21 during those two weeks 15:41:39 with the oslo.privsep change (un-monkey patching the daemon( 15:41:52 and the latests pyroute2 lib version 15:42:06 did you notice the functional/fullstack tests more stable? 15:42:13 in other works 15:42:14 words 15:42:19 did you see timeouts? 15:42:54 looking at http://grafana.openstack.org/d/Hj5IHcSmz/neutron-failure-rate?viewPanel=24&orgId=1&from=now-30d&to=now I don't think it is more stable than it was 15:43:04 maybe slightly better 15:43:41 fullstack seems to be pretty good recently 15:43:42 yes, a 0.001% better 15:43:43 hehehehe 15:43:46 :) 15:43:47 :) 15:44:00 once we have George 15:44:08 but even if it's 0.001% better, it's step in good direction 15:44:11 we can think about moving some fullstack tests to George 15:44:23 yes, I was reviewing George patches today 15:44:25 using containers will help to isolate some noisy tests 15:44:28 looks good and very interesting 15:44:32 a lot!! 15:44:45 that's what fullstack test should have been implemented 15:44:51 but this is easy to say this now 15:44:57 yes 15:46:34 ok, if You don't have anything else for today, lets finish this meeting earlier 15:46:39 thx for attending 15:46:44 bye! 15:46:44 #endmeeting