15:00:27 #startmeeting neutron_ci 15:00:28 Meeting started Wed Jun 24 15:00:27 2020 UTC and is due to finish in 60 minutes. The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:00:30 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:00:31 hi 15:00:32 The meeting name has been set to 'neutron_ci' 15:00:36 hi 15:01:37 hello 15:01:47 first of all 15:01:55 #link http://grafana.openstack.org/dashboard/db/neutron-failure-rate 15:02:05 please open now to have it ready for later :) 15:02:08 o/ 15:03:33 ok, lets start 15:03:39 \o 15:03:40 #topic Actions from previous meetings 15:03:56 we have only one action from last week 15:03:58 slaweq to add additional logging for fullstack's firewall tests 15:04:08 I found the issue again in https://zuul.opendev.org/t/openstack/build/c5451e9e66fe4c14b2a09339a77fc449 15:04:10 After checking it seems for me that failure was at the beginning of the test, when connectivity with ncat was checked 15:04:12 I proposed patch https://review.opendev.org/#/c/737741/ for now. 15:04:14 After that lets see what will happen more 15:04:43 o/ 15:05:00 that's all update from me about this 15:05:24 #topic Stadium projects 15:05:34 I don't have any updates about stadium projects today 15:05:39 but maybe You have something 15:06:02 nothing special from me, but if you have time please look there for things to review :-) 15:06:17 not from me, we had enough CI trouble in neutron itself to keep busy 15:06:21 nothing for me either 15:06:40 ok, sure lajoskatona I will check list of opened patches in stadium projects 15:06:58 ok, so next topic 15:07:00 #topic Stable branches 15:07:05 Ussuri dashboard: http://grafana.openstack.org/d/pM54U-Kiz/neutron-failure-rate-previous-stable-release?orgId=1 15:07:06 Train dashboard: http://grafana.openstack.org/d/dCFVU-Kik/neutron-failure-rate-older-stable-release?orgId=1 15:07:15 for ussuri and train it seems that it works fine now 15:07:16 slaweq: thanks 15:07:36 but for older releases, like rocky and queens, I think we still have uwsgi issue 15:07:43 bcafarel: are You aware of it? 15:07:59 I got many failures e.g. on https://review.opendev.org/#/c/737703/ today 15:08:27 yes for EM branches we have lingering issue with grenade jobs ( gmann mentioned them in recent mail update) 15:08:39 https://review.opendev.org/737414 should workaround it for the time being 15:08:52 once it works, and backported to older branches 15:10:03 bcafarel: but it's only for grenade 15:10:14 and in my patch I saw failures in all tempest jobs too 15:10:29 basicaly everything except UT and pep8 was red 15:10:46 oh sigh 15:11:20 I did not check rocky thoroughly yet, still on stein :/ 15:11:22 I see error like: "ls: cannot access 'uwsgi*': No such file or directory" in devstack log 15:13:00 ack there was mention on uwsgi + rocky and older still WIP http://lists.openstack.org/pipermail/openstack-discuss/2020-June/015558.html 15:13:01 bcafarel: do You have cycles to check that this week? 15:13:28 slaweq: yes, at least to get some up-to-date status on it! 15:13:36 thx 15:13:59 #action bcafarel to check gate status on rocky and queens (uwsgi problem) 15:14:36 ok 15:14:39 lets move on 15:14:41 #topic Grafana 15:14:48 #link http://grafana.openstack.org/dashboard/db/neutron-failure-rate 15:15:28 in master branch it looks ok this week IMO 15:17:55 +1 15:18:17 so lets talk about some issues in specific jobs 15:18:19 #topic fullstack/functional 15:18:30 first functional tests 15:18:37 I found today some db migration errors (again), like https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_ec3/736269/3/check/neutron-functional-with-uwsgi/ec30046/testr_results.html 15:18:55 ralonsoh: is it something what You want to address with Your db migration script changes? 15:19:11 yes 15:19:19 those errors in test_walk_versions 15:19:41 and test_has_offline_migrations_all_heads_upgraded 15:19:41 ok 15:20:01 in general I think that those failures are due to slow node where it was run 15:20:18 but will be good if we can get rid of at least some of them 15:21:27 I also have other issue 15:21:31 https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_63a/711425/11/check/neutron-functional/63ac4ca/testr_results.html 15:21:53 this seems for me like somethig related to our privileged ip_lib or pyroute2 15:22:18 as there are issues that interfaces are not found in namespace None 15:22:50 and there are also issue with _check_bridge_datapath_id() method in tests there 15:22:52 well, maybe that's correct and the interface was not created 15:23:59 ralonsoh: can be 15:24:18 so at least we should probably fix exception message IMO 15:24:19 there is also this failure test_ovsdb_monitor.TestNBDbMonitorOverTcp.test_floatingip_mac_bindings (IndexError: list index out of range)... I can take a look on that one 15:24:29 or not, nvm :) 15:24:35 thx maciejjozefczyk 15:25:06 I can take a look at the bridge.get_datapath_id problem 15:25:13 if I have some time this week 15:25:25 thx ralonsoh 15:25:49 #action maciejjozefczyk will check test_ovsdb_monitor.TestNBDbMonitorOverTcp.test_floatingip_mac_bindings failiure in https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_63a/711425/11/check/neutron-functional/63ac4ca/testr_results.html 15:26:07 #action ralonsoh will check get_datapath_id issues in https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_63a/711425/11/check/neutron-functional/63ac4ca/testr_results.htm 15:26:16 #undo 15:26:17 Removing item from minutes: #action ralonsoh will check get_datapath_id issues in https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_63a/711425/11/check/neutron-functional/63ac4ca/testr_results.htm 15:26:18 #action ralonsoh will check get_datapath_id issues in https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_63a/711425/11/check/neutron-functional/63ac4ca/testr_results.html 15:27:48 I will check this errors with non existing devices 15:28:21 #action slaweq will check errors with non existing interface in https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_63a/711425/11/check/neutron-functional/63ac4ca/testr_results.html 15:28:47 ok, lets move on 15:28:49 #topic Tempest/Scenario 15:29:02 in scenario tests I found one interesting issue this week 15:29:17 it happend only once I think but maybe worth to deeper look 15:29:27 https://11b28b714aaa0f2eaa01-115c1089095738e3e088969e8724f0ca.ssl.cf1.rackcdn.com/712640/9/check/neutron-tempest-plugin-scenario-openvswitch/2ad316a/testr_results.html 15:29:35 error with address already allocated in subnet 15:29:40 maybe some bug in tests? 15:30:00 didn't I sent a patch for this? 15:30:10 send* 15:30:22 ralonsoh: I don't remember 15:30:29 (checking) 15:30:34 ahh, right 15:30:40 I think that now I remember 15:30:56 https://review.opendev.org/#/c/731267/ 15:31:24 ok, this failure was before Your patch was merged 15:31:28 so we should be good now 15:31:31 thx ralonsoh 15:31:39 ahhh ok perfect 15:32:16 ralonsoh: fixing CI issues before slaweq complains about them, nice! 15:32:22 hahaha 15:32:28 haha 15:32:31 true 15:32:35 ralonsoh++ :D 15:32:51 ok, so one last topic for today 15:32:53 #topic Periodic 15:33:03 all except one job looks fine there 15:33:13 but neutron-ovn-tempest-ovs-master-fedora is broken again 15:33:31 sigh 15:33:46 and it's broken on compilation of ovs 15:33:50 https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_b03/periodic/opendev.org/openstack/neutron/master/neutron-ovn-tempest-ovs-master-fedora/b039158/job-output.txt 15:34:48 maciejjozefczyk: can You take a look into that one? 15:35:12 slaweq, yes 15:35:17 thx 15:35:26 slaweq, I fixed that failure some time ago and it is failing again.. strange 15:35:32 I'll take a closer look on that, thanks 15:35:32 #action maciejjozefczyk to check failing neutron-ovn-tempest-ovs-master-fedora periodic job 15:35:42 thx a lot 15:35:48 ok, that's all from me today 15:35:53 (that was fast ;)) 15:36:09 anything else You want to discuss today? 15:36:21 no thanks 15:37:34 if not, I think I can give You back about 20 minutes today :) 15:37:36 I just realized "stadium projects" section we had also community goals mixed in 15:37:53 short update on that I started to fill https://etherpad.opendev.org/p/neutron-victoria-switch_to_focal 15:38:18 hopefully it will be more complete by next week :) 15:38:22 thx bcafarel 15:38:24 bcafarel: good point, 15:38:48 btw, speaking about focal, it seems from https://review.opendev.org/#/c/737370/ that neutron is running fine there 15:38:54 I checked with odl but I got weird failures, and thought to wait some time till the waves go down :-) 15:39:02 there are some problems realated to cinder in tempest jobs 15:40:40 bcafarel, slaweq: for odl I had this: https://review.opendev.org/736703 15:40:45 lajoskatona: indeed, main patches in devstack and tempest should clear out most of issues - and most of our jobs will inherit directly 15:40:53 and it fails with nodeset not found or similar 15:41:15 bcafarel: that was my feeling as well :-) 15:41:39 lajoskatona: or depending on https://review.opendev.org/734700 should help to get the nodes and job definitions 15:41:54 thx bcafarel and lajoskatona 15:42:32 bcafarel: thanks, I do a try with that, the wf -1 though a little frightening 15:42:50 but that's perhaps just some timing from gmann 15:44:07 yes it is in "heavy progress" there 15:44:24 though just for inheriting the nodeset it should be safe enough for testing 15:44:36 yeah, 734700 is right to test. i was waiting for gate result there and i will announce the same on ML also. 15:44:53 gmann, bcafarel: thanks, I check it 15:45:05 gmann: in our tests in https://review.opendev.org/#/c/737370/ I saw some cinder related failures 15:45:18 gmann: do You know if cinder team is aware of them? 15:45:21 bcafarel: if your job override nodeset then you need to change otherwise 734700 can take care of devstack base job nodeset switch to focal 15:45:25 or maybe I should open LP for them? 15:45:59 slaweq: current known bug is this https://bugs.launchpad.net/nova/+bug/1882521 15:45:59 Launchpad bug 1882521 in OpenStack Compute (nova) "Failing device detachments on Focal" [Undecided,New] 15:46:13 gmann: ack I am making a list of jobs that will need some action (more than just 734700) 15:46:20 which is volume detach issue but i have not checked your patch failure 15:46:28 bcafarel: +1 15:47:09 gmann: seems like the same one 15:47:11 thx 15:49:46 ok, I think we can finish meeting now 15:49:49 thx for attending 15:49:53 \o 15:49:56 and see You o/ 15:49:59 #endmeeting