15:01:37 <slaweq> #startmeeting neutron_ci 15:01:37 <openstack> Meeting started Tue Apr 13 15:01:37 2021 UTC and is due to finish in 60 minutes. The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:01:38 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:01:40 <openstack> The meeting name has been set to 'neutron_ci' 15:01:40 <slaweq> hi 15:01:41 <ralonsoh> hi 15:02:18 <bcafarel> o/ 15:02:59 <slaweq> ping lajoskatona 15:03:48 <slaweq> ok, lets start 15:03:55 <slaweq> Grafana dashboard: http://grafana.openstack.org/dashboard/db/neutron-failure-rate 15:04:06 <slaweq> #topic Actions from previous meetings 15:04:13 <slaweq> ralonsoh to check failed qos scenario test 15:04:18 <lajoskatona> Hi, I am here 15:04:30 <ralonsoh> yes, I didn't replicate the error 15:04:36 <ralonsoh> and I didn't see it again 15:04:54 <ralonsoh> and reviewing the logs, I didn't see why the QoS was not applied 15:05:05 <ralonsoh> sorry, no solution for this one 15:05:07 <slaweq> ralonsoh: ok, so let's not bother with this for now 15:05:27 <slaweq> we will get back to it when it will be happening more often 15:05:32 <ralonsoh> ok 15:05:37 <slaweq> ralonsoh: thx for checking it 15:05:39 <slaweq> next one 15:05:41 <slaweq> bcafarel to update grafana dashboards with stable/wallaby 15:05:58 <bcafarel> sent and merged (I think), looking for the link 15:06:42 <bcafarel> yep https://review.opendev.org/c/openstack/project-config/+/786008 merged, also good time to ping other for release-related changes 15:07:06 <bcafarel> like https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/785092 (switch to -wallaby variant jobs) 15:07:08 <slaweq> thx bcafarel 15:08:02 <slaweq> ok, next one 15:08:04 <slaweq> ralonsoh to check failed test_get_egress_min_bw_for_port functional test 15:08:18 <ralonsoh> yes, one sec 15:08:33 <ralonsoh> #link https://review.opendev.org/c/openstack/neutron/+/785158 15:08:47 <ralonsoh> this actually a pre-check that must be always true 15:09:00 <ralonsoh> this is not solving the error 15:10:03 <lajoskatona> If I understand well we will have clean feedback if there are policies on port 15:10:31 <ralonsoh> sorry, someone else pinged me 15:10:52 <ralonsoh> this patch will provide info if any other qos is set before the test 15:11:38 <slaweq> so this may help to understand what the issue is there 15:11:42 <ralonsoh> sure 15:11:49 <slaweq> which is good step forward IMO :) 15:11:51 <slaweq> thx ralonsoh 15:12:07 <slaweq> ok, and the last one 15:12:08 <slaweq> ralonsoh to update LB installation guide with info about legacy ebtables 15:12:12 <ralonsoh> done 15:12:15 <ralonsoh> one sec 15:12:49 <ralonsoh> I don't find it 15:13:02 <ralonsoh> https://review.opendev.org/c/openstack/neutron/+/785137 15:13:17 <slaweq> thx 15:13:21 <slaweq> it's merged already 15:13:51 <slaweq> and that were all action items from last week 15:13:57 <slaweq> let's move on 15:13:59 <slaweq> #topic Stadium projects 15:14:18 <slaweq> lajoskatona: any updates? 15:14:23 <slaweq> or issues 15:14:24 <lajoskatona> Nothing what I have seen, so seems to be quiet everywhere 15:14:38 <slaweq> ++ 15:14:40 <slaweq> thx 15:15:58 <slaweq> #topic Stable branches 15:16:17 <slaweq> as bcafarel already mentioned there are some patches https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/785092 and https://review.opendev.org/c/openstack/neutron/+/785093 15:16:23 <slaweq> which are for wallaby's ci 15:16:34 <slaweq> and I will need to do similar patches for stadium projects 15:16:45 <slaweq> I will do them this week 15:17:02 <slaweq> #action slaweq to update wallaby's scenario jobs in neutron-tempest-plugin 15:17:21 <slaweq> bcafarel: anything else regarding stable branches ci? 15:18:03 <bcafarel> recent branches look OK overall 15:18:14 <bcafarel> stein has https://bugs.launchpad.net/neutron/+bug/1923412 small fix needed 15:18:15 <openstack> Launchpad bug 1923412 in neutron "[stable/stein] Tempest fails with unrecognized arguments: --exclude-regex" [Critical,Triaged] 15:18:40 <bcafarel> rocky I wonder, it seems iptables_hybrid job is consistently failing (like on https://review.opendev.org/c/openstack/neutron/+/770794 ) 15:18:54 <bcafarel> but I did not have time to check this week sorry :( 15:20:00 <ralonsoh> I'll try to check it 15:20:13 <slaweq> there are some failing tests there 15:20:24 <slaweq> but also it fails on collecting logs after job's failure 15:21:18 <slaweq> there is some error in subunit2html.py 15:21:37 <slaweq> I wonder when we will be able to EOL those branches which runs on python 2.7 still 15:21:50 <slaweq> and on Ubuntu Xenial 15:22:34 <slaweq> hmm 15:22:37 <slaweq> interesting 15:22:51 <slaweq> Ubuntu Xenial support ends in April 2021 according to https://lists.ubuntu.com/archives/ubuntu-announce/2016-April/000207.html 15:23:05 <slaweq> and we are still using it for testing Queens and Rocky 15:23:19 <ralonsoh> I think we can try to bump this library to 1.1.0 15:23:31 <ralonsoh> we are still using 1.0.0 15:23:37 <slaweq> sorry, wrong link, good one is https://wiki.ubuntu.com/Releases 15:24:04 <slaweq> ok, bcafarel can You report an issue regarding that rocky jobs? 15:24:28 <bcafarel> slaweq: sure will do later tonight (I will check on other backports too to confirm) 15:24:30 <slaweq> and I will send an email to the community to ask what we should do with that 15:24:37 <slaweq> thx bcafarel 15:24:50 <slaweq> #action bcafarel to report stable/rocky ci failures on LP 15:25:05 <slaweq> #action slaweq to send email about testing older branches on Ubuntu Xenial 15:25:23 <slaweq> ahh, wait 15:25:37 <slaweq> Xenial's standard support ends this month 15:25:42 <slaweq> but EOL will be in 2024 15:25:46 <slaweq> so we still can use it :) 15:25:52 <slaweq> I will not send any email about it 15:25:55 <slaweq> sorry for the noise 15:26:29 <slaweq> ok, I think we can move on 15:26:32 <slaweq> #topic Grafana 15:26:36 <slaweq> http://grafana.openstack.org/dashboard/db/neutron-failure-rate 15:27:12 <slaweq> it seems we are better with functional job finally 15:27:23 <slaweq> but we have some issue with neutron-tempest-plugin-scenario jobs 15:27:43 <slaweq> as most of them failing at around 20-30 % of times 15:29:12 <slaweq> except that I think that things are pretty ok 15:30:09 <slaweq> let's talk about some specific issues 15:30:11 <slaweq> #topic fullstack/functional 15:30:24 <slaweq> I found (again) this week failure of test_keepalived_spawns_conflicting_pid_vrrp_subprocess 15:30:28 <slaweq> https://7c4af2962e649c78a0b8-8dbbf393fa4fd2548b6eb8c45bf2a7e6.ssl.cf2.rackcdn.com/785011/1/check/neutron-functional-with-uwsgi/dc727d9/testr_results.html 15:30:38 <slaweq> but I think we marked those tests as unstable, didn't we? 15:30:45 <ralonsoh> yes, same problem as before 15:30:59 <ralonsoh> the sigterm signal do not reply 15:31:08 <ralonsoh> and the thread never ends 15:31:27 <slaweq> yes, but is the patch which markes that as unstable merged already? 15:31:32 <slaweq> do You remember that patch? 15:31:39 <ralonsoh> nope, I think we removed it 15:31:44 <ralonsoh> we'll need to propose it again 15:31:51 <slaweq> ahh, ok 15:32:08 <slaweq> maybe we should then mark them as unstable until we will fix that issue completly 15:32:31 <slaweq> ralonsoh: will You propose it? 15:32:32 <ralonsoh> perfect 15:32:34 <ralonsoh> yes 15:32:37 <slaweq> thx 15:32:54 <slaweq> #action ralonsoh to mark test_keepalived_spawns_conflicting_pid_vrrp_subprocess functional test as unstable 15:33:07 <slaweq> ok, regarding fullstack 15:33:19 <slaweq> I just want to ask You about review of the patch https://review.opendev.org/c/openstack/neutron/+/783748 15:33:24 <slaweq> it's new fullstack test to add 15:33:29 <slaweq> for segmentation_id updates 15:33:41 <slaweq> no rush but if You will have few minutes, please take a look 15:33:53 <slaweq> next topic 15:33:55 <slaweq> #topic Tempest/Scenario 15:34:07 <slaweq> here there is most "hot" issue for this week 15:34:34 <slaweq> I noticed that in all our neutron-tempest-plugin scenario jobs random tests are failing due to ssh failure 15:34:48 <slaweq> every time there is problem with accessing metadata service 15:35:00 <slaweq> https://f237087178b608be7cbc-a3abe88e6ab2911093ccea19e77537d0.ssl.cf5.rackcdn.com/785749/1/check/neutron-tempest-plugin-scenario-linuxbridge/f746672/testr_results.html 15:35:02 <slaweq> https://5e09181bcc1a50499619-17764b56a5c622705c872e3c7dca2597.ssl.cf2.rackcdn.com/769788/6/check/neutron-tempest-plugin-scenario-openvswitch/32b4182/testr_results.html 15:35:04 <slaweq> https://f3719601e38156e4ec26-626e176c8fffa963d6c301ab5a223935.ssl.cf1.rackcdn.com/785633/2/check/neutron-tempest-plugin-scenario-linuxbridge/4fcd101/testr_results.html 15:35:06 <slaweq> https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_da7/785895/1/check/neutron-tempest-plugin-scenario-linuxbridge/da7ea5e/testr_results.html 15:35:09 <slaweq> https://5b963027deeaf11430aa-de753611fe18d17817cd3cc57c427456.ssl.cf1.rackcdn.com/785895/1/check/neutron-tempest-plugin-scenario-openvswitch/ce74a5b/testr_results.html 15:35:10 <slaweq> https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_785/785889/3/check/neutron-tempest-plugin-scenario-openvswitch/785f8bb/testr_results.html 15:35:12 <slaweq> https://3acd25be4b1698636631-3e445c1a5b8a8c84f71b5a61cff5a09b.ssl.cf5.rackcdn.com/785950/1/check/neutron-tempest-plugin-scenario-linuxbridge/5c9ea3c/testr_results.html 15:35:14 <slaweq> those are examples of the failures 15:35:27 <slaweq> all those jobs runs with "ha=True" so keepalived is involved in the router's namespace 15:35:41 <slaweq> any volunteer to investigate it? 15:35:51 <ralonsoh> I could do it 15:36:24 <slaweq> I think that maybe https://review.opendev.org/c/openstack/neutron/+/782679 may be useful for debugging that issue 15:36:44 <slaweq> ahh, no 15:36:45 <slaweq> sorry 15:36:49 <slaweq> it's only for grenade jobs 15:37:01 <slaweq> we should do similar change for neutron-tempest-plugin jobs also 15:37:06 <ralonsoh> ok 15:37:15 <slaweq> ralonsoh: I will open LP for that issue 15:37:22 <ralonsoh> so you think this is a problem in the iptables rule? 15:37:23 <slaweq> I forgot to do it before the meeting 15:37:32 <slaweq> idk but maybe can be 15:37:39 <ralonsoh> ping me with the LP link 15:37:59 <slaweq> sure, I will 15:39:02 <slaweq> #action slaweq to report LP with metadata issue in scenario jobs 15:39:17 <slaweq> #ralonsoh to check metadata issue in scenario jobs 15:41:29 <slaweq> ok, that's all regarding scenario jobs for today 15:41:35 <slaweq> #topic Periodic 15:41:46 <slaweq> periodic jobs looks pretty ok recently 15:42:01 <slaweq> one thing I want to ask You is to review https://review.opendev.org/c/openstack/neutron/+/785660 15:42:07 <ralonsoh> sure 15:42:21 <slaweq> as we recently discovered that periodic jobs based on tripleo are testing neutron Victoria every day 15:42:28 <slaweq> which basically doesn't makes any sense :) 15:42:30 <ralonsoh> ups... 15:42:47 <slaweq> so this patch ensures that neutron is from master branch in that jobs always 15:43:14 <slaweq> ok, that's all from my side 15:43:15 <bcafarel> he, nice catch 15:43:28 <slaweq> do You have anything else to discuss today? 15:43:49 <ralonsoh> no, thanks 15:44:18 <slaweq> ahh, one last thing - next week's meeting will be cancelled as we have ptg in the same time 15:44:38 <slaweq> ok, so if there are no other topics, we can finish earlier today 15:44:43 <slaweq> thx for attending the meeting 15:44:49 <slaweq> #endmeeting