16:00:03 #startmeeting neutron_ci 16:00:05 Meeting started Tue Dec 3 16:00:03 2019 UTC and is due to finish in 60 minutes. The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:00:06 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:00:09 The meeting name has been set to 'neutron_ci' 16:00:10 welcome (again) :) 16:00:19 o/ (again too :) ) 16:01:38 hi 16:01:55 njohnston liuyulong CI meeting, do You have time to attend? 16:02:30 ok, lets start 16:02:31 #topic Actions from previous meetings 16:02:38 njohnston to check failing NetworkMigrationFromHA in multinode dvr job 16:03:44 ok, I think njohnston is not here now 16:03:47 so lets move on 16:03:56 slaweq to continue investigating issue https://bugs.launchpad.net/neutron/+bug/1850557 16:03:56 Launchpad bug 1850557 in neutron "DHCP connectivity after migration/resize not working" [High,Fix released] - Assigned to Slawek Kaplonski (slaweq) 16:04:04 Patch is now merged already https://review.opendev.org/696794 16:04:19 I hope we will be good now with those migration/shelve tests on multinode jobs 16:05:00 and next one: 16:05:01 yep I have seen this one in stable backports queue 16:05:02 slaweq to move job definitions to zuul.d directory 16:05:38 bcafarel: yes, I proposed backport as this issue is also valid for stable braches and backport was easy 16:06:01 according to zuul jobs definitions, patch is proposed https://review.opendev.org/#/c/696286/ 16:06:10 please review it if You have some time 16:06:37 any questions/comments to actions from last week? 16:07:32 just big +1 on that zuul jobs definitions split :) 16:07:38 for sure 16:07:59 thx 16:08:19 I hope it will be easier to look for jobs' definitions now 16:08:30 ok, lets move on 16:08:35 #action njohnston to check failing NetworkMigrationFromHA in multinode dvr job 16:08:42 ^^ just a reminder for next week 16:08:51 and I think we can move on to the next topic now 16:08:59 #topic Actions from previous meetings 16:09:02 #undo 16:09:03 Removing item from minutes: #topic Actions from previous meetings 16:09:09 #topic Stadium projects 16:09:21 tempest-plugins migration 16:09:23 Etherpad: https://etherpad.openstack.org/p/neutron_stadium_move_to_tempest_plugin_repo 16:09:52 we finally merged step 2 for neutron-dynamic-routing 16:09:56 thx njohnston for that 16:10:03 so the last project on this list is vpnaas 16:10:16 and mlavalle told me yesterday that patches for that are ready for review 16:10:28 I wanted to review them today but didn't had time 16:10:43 Patches are here: 16:10:45 Step 1: https://review.openstack.org/#/c/649373 16:10:47 Step 2: https://review.opendev.org/#/c/695834 16:10:58 so please review them if You will have some time :) 16:11:16 same I started to review step 1 but got sidetracked 16:11:17 and we will be good with whole this migration finally 16:11:35 one question I had there, vpnaas should only be migrated from ussuri? 16:11:46 or from train to be "in sync" with others 16:12:30 no, I think that ussuri is enough 16:12:44 ok, off to -1 then :) 16:13:27 :) 16:14:21 thx bcafarel for taking look into that 16:14:38 and next topic related to stadium projects is: 16:14:40 Neutron Train - Drop py27 and standardize on zuul v3 16:14:42 Etherpad: https://etherpad.openstack.org/p/neutron-train-zuulv3-py27drop 16:14:50 but this was already discussed today on team meeting 16:15:00 so I don't think we need to talk about it here too 16:15:25 do You have anything else related to stadium projects for today? 16:16:23 ok, I guess that this means "no" :) 16:16:27 so lets move on 16:16:29 no 16:16:33 #topic Grafana 16:16:35 http://grafana.openstack.org/dashboard/db/neutron-failure-rate 16:18:04 We need to clean it a bit from some recently removed jobs. But I want to do it after grenade-py3 will be removed and after we will add ovn jobs to neutron CI too. 16:18:19 other than that I don't see anything really wrong in grafana 16:19:31 do You have anything related to grafana? 16:19:38 no 16:19:53 me neither 16:20:19 so lets talk about some specific issues now 16:20:21 #topic fullstack/functional 16:20:39 regarding functional tests I found new (IMO) bug: 16:20:40 o/ sorry I am late 16:20:42 https://bugs.launchpad.net/neutron/+bug/1854462 16:20:42 Launchpad bug 1854462 in neutron "[Functional tests] Timeout exception in list_namespace_pids" [High,Confirmed] 16:20:59 I know that we had something similar in the past but I was sure that ralonsoh fixed it already 16:21:09 slaweq, I need to check that 16:21:10 njohnston: o/ no problem :) 16:21:34 that sounds really familiar indeed 16:21:35 thx ralonsoh - I saw it at least couple of times this last week so I marked it as High for now 16:21:35 yes, the fix was merged, isn't it? 16:22:11 I mean: we implement, in Neutron, this part of the pyroute2 code 16:22:12 ralonsoh: I don't remember fix exactly so I can't find it now 16:22:25 and we implemented a retry catch in the testcase class 16:22:30 but I'm pretty sure we merged Your fix for this 16:22:35 for timeouts 16:22:48 slaweq, put this in my TODO list 16:22:54 ralonsoh: thx 16:23:11 #action ralonsoh to check functional tests timeouts https://bugs.launchpad.net/neutron/+bug/1854462 16:23:12 Launchpad bug 1854462 in neutron "[Functional tests] Timeout exception in list_namespace_pids" [High,Confirmed] 16:23:23 ralonsoh: just to remember to check it next week :) 16:23:26 sure 16:23:52 for fullstack tests I noticed one failed test https://0050cb9fd8118437e3e0-3c2a18acb5109e625907972e3aa6a592.ssl.cf5.rackcdn.com/696600/1/check/neutron-fullstack/4966bce/testr_results.html.gz 16:24:09 but as I checked logs from it, it seems that there was problem with rabbitmq during this test 16:24:20 all agents were dead in neutron db 16:24:35 so maybe it was some host slowdown or something like that 16:24:54 I will simply check if that will happen more times or not 16:24:57 did you open a bug for this one? 16:25:05 not necessary 16:25:08 ralonsoh: no 16:25:10 ok 16:25:21 I found it today and wanted to take a look for few days first 16:25:29 to check if that will happen more times 16:25:57 and that's all related to functional/fullstack tests from my side 16:26:04 anything else You want to add/ask? 16:27:03 ok, if not, lets move on 16:27:07 go ahead 16:27:12 #topic Tempest/Scenario 16:27:37 here, after merging my fix for resize/shelve failure I think we are quite good now 16:27:45 but we have problem with grenade jobs 16:27:58 those jobs are failing quite often recently 16:28:32 so first of all, as we talked some time ago, I proposed to remove grenade-py3 from our gate: https://review.opendev.org/#/c/695172/ 16:28:43 please review this patch if You have few minutes 16:29:03 less grenade jobs, smaller chance to hit its failures :) 16:29:11 hahahaha 16:29:16 :) 16:29:28 +100 16:29:37 and than second part is worts, as in multinode grenade jobs we are hitting some issue quite often 16:29:46 examples of such failures are e.g.: 16:29:47 https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_ad0/696592/3/check/neutron-grenade-multinode/ad0df97/logs/grenade.sh.txt.gz 16:29:49 https://6e84b50c364d7e277563-65c8cd20428a10135cd2762abf51d9a7.ssl.cf2.rackcdn.com/697035/1/check/grenade-py3/78b1764/logs/grenade.sh.txt.gz 16:29:51 https://819efd42b5c79a55763b-90a63ad77a0414e858bcf634436e4dc8.ssl.cf5.rackcdn.com/697035/1/check/neutron-grenade-multinode/4d68281/logs/testr_results.html.gz 16:29:53 https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_725/696103/8/check/neutron-grenade-dvr-multinode/7255438/logs/grenade.sh.txt.gz 16:30:01 in each of those cases it seems for me as some error on nova's side 16:30:12 but I didn't dig a lot into logs so I'm not 100% sure 16:31:20 so I will try to dig into those failures a bit more and try to find solution for it or report nova bug at least :) 16:31:34 #action slaweq to check reason of grenade jobs failures 16:32:05 thanks! 16:32:34 from other things related to scenario jobs I have 2 patches ready for review: 16:32:55 https://review.opendev.org/#/c/694049/ - this one switches queens jobs to run on tagged version of tempest plugin 16:33:19 and removes those jobs from check and gate queue 16:33:36 and second one: 16:33:38 https://review.opendev.org/#/c/695013/ 16:33:53 this one switches to use py3 on all nodes in multinode jobs 16:34:08 so please review those patches if You will have some time :) 16:34:12 +2 to boith 16:34:14 both 16:34:25 ralonsoh: thx 16:34:25 +1 to both ;) 16:34:32 bcafarel: thx :) 16:34:53 and that's all what I have for today 16:35:00 +2+W x 2 16:35:06 thx njohnston :) 16:35:10 that was fast 16:35:15 team++ 16:35:19 I have one bug to talk about 16:35:25 njohnston: go on 16:35:32 "py36 unit test cases fails" https://bugs.launchpad.net/neutron/+bug/1854051 16:35:32 Launchpad bug 1854051 in neutron "py36 unit test cases fails" [Critical,New] 16:35:37 from last week as bug deputy 16:36:01 I have not seen that in the gate or personally but I wanted to see if anyone had any experience with this sort of thing 16:36:29 not me, I didn't noticed that bug in gate 16:36:53 maybe we can block this specific "typing" version 16:37:00 and based on last comment from liuyulong it seems that he is using rpm to install deps 16:37:08 yeah 16:37:21 so maybe we are installing some other version of typing from pypi 16:37:26 and that's why we are fine? 16:38:25 I think that is possible 16:38:56 anyhow, just wanted to raise it here and see if anyone had seen it. thanks! 16:39:26 and also typing package is not in neutron requirements 16:39:46 so I'm not sure how this may cause problem in neutron 16:40:28 this is part of the standard library 16:41:05 ok, but than it shouldn't cause any problems for us, right? 16:41:13 right 16:41:33 tbh I would close this bug for now as incomplete as it not happens on gate 16:42:35 done! 16:42:43 njohnston: thx :) 16:42:55 ok, anything else You want to discuss today? 16:43:05 if not, I will give You 15 minutes back :) 16:43:16 I like option 2 16:43:21 hahaha 16:43:29 ok 16:43:32 +2 16:43:33 so thx for attending 16:43:35 bye! 16:43:37 and see You tomorrow :) 16:43:39 o/ 16:43:39 o/ 16:43:41 #endmeeting