| *** jamesmcarthur has joined #openstack-meeting-3 | 00:15 | |
| *** jamesmcarthur has quit IRC | 00:57 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 00:57 | |
| *** jamesmcarthur has quit IRC | 02:02 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 02:02 | |
| *** diablo_rojo has quit IRC | 02:03 | |
| *** yamamoto has joined #openstack-meeting-3 | 02:16 | |
| *** jamesmcarthur has quit IRC | 02:54 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 02:57 | |
| *** ricolin_ has joined #openstack-meeting-3 | 03:00 | |
| *** apetrich has quit IRC | 03:09 | |
| *** jamesmcarthur has quit IRC | 03:16 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 03:31 | |
| *** negronjl has quit IRC | 04:29 | |
| *** negronjl has joined #openstack-meeting-3 | 04:32 | |
| *** ricolin_ has quit IRC | 05:05 | |
| *** ricolin_ has joined #openstack-meeting-3 | 05:05 | |
| *** ricolin_ has quit IRC | 05:05 | |
| *** jamesmcarthur has quit IRC | 05:11 | |
| *** links has joined #openstack-meeting-3 | 05:40 | |
| *** tsmith2 has quit IRC | 05:51 | |
| *** tsmith2 has joined #openstack-meeting-3 | 05:51 | |
| *** ralonsoh has joined #openstack-meeting-3 | 07:31 | |
| *** e0ne has joined #openstack-meeting-3 | 07:59 | |
| *** slaweq has joined #openstack-meeting-3 | 08:04 | |
| *** slaweq has quit IRC | 08:28 | |
| *** slaweq has joined #openstack-meeting-3 | 08:29 | |
| *** apetrich has joined #openstack-meeting-3 | 08:40 | |
| *** yamamoto has quit IRC | 09:28 | |
| *** yamamoto has joined #openstack-meeting-3 | 10:15 | |
| *** yamamoto has quit IRC | 11:07 | |
| *** yamamoto has joined #openstack-meeting-3 | 11:10 | |
| *** raildo has joined #openstack-meeting-3 | 11:56 | |
| *** ralonsoh has quit IRC | 12:05 | |
| *** ralonsoh has joined #openstack-meeting-3 | 12:09 | |
| *** yamamoto has quit IRC | 12:14 | |
| *** yamamoto has joined #openstack-meeting-3 | 12:16 | |
| *** yamamoto has quit IRC | 12:35 | |
| *** yamamoto has joined #openstack-meeting-3 | 13:12 | |
| *** yamamoto has quit IRC | 13:14 | |
| *** jraju__ has joined #openstack-meeting-3 | 13:43 | |
| *** links has quit IRC | 13:44 | |
| *** liuyulong has joined #openstack-meeting-3 | 13:46 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 14:44 | |
| *** macz_ has joined #openstack-meeting-3 | 14:55 | |
| slaweq | #startmeeting neutron_ci | 15:00 |
|---|---|---|
| openstack | Meeting started Wed Mar 18 15:00:11 2020 UTC and is due to finish in 60 minutes. The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| *** openstack changes topic to " (Meeting topic: neutron_ci)" | 15:00 | |
| openstack | The meeting name has been set to 'neutron_ci' | 15:00 |
| njohnston | o/ | 15:00 |
| bcafarel | o/ | 15:00 |
| ralonsoh | hi | 15:00 |
| slaweq | ok, lets go :) | 15:00 |
| slaweq | Grafana dashboard: http://grafana.openstack.org/dashboard/db/neutron-failure-rate | 15:00 |
| slaweq | Please open now :) | 15:00 |
| slaweq | #topic Actions from previous meetings | 15:01 |
| *** openstack changes topic to "Actions from previous meetings (Meeting topic: neutron_ci)" | 15:01 | |
| slaweq | maciejjozefczyk to take a look at https://bugs.launchpad.net/neutron/+bug/1865453 | 15:01 |
| openstack | Launchpad bug 1865453 in neutron "neutron.tests.functional.plugins.ml2.drivers.ovn.mech_driver.test_mech_driver.TestVirtualPorts.test_virtual_port_created_before fails randomly" [High,In progress] - Assigned to Maciej Jozefczyk (maciej.jozefczyk) | 15:01 |
| slaweq | I just ping maciej to join here | 15:03 |
| slaweq | maybe he has some update | 15:03 |
| slaweq | ok, lets move on for now | 15:04 |
| slaweq | maybe he will join later | 15:04 |
| slaweq | ralonsoh to check "neutron_tempest_plugin.scenario.test_multicast.MulticastTestIPv4 failing often on deletion of server" | 15:04 |
| ralonsoh | I tried but I didn't find anything | 15:04 |
| *** maciejjozefczyk has joined #openstack-meeting-3 | 15:05 | |
| ralonsoh | sorry, I can't find the cause of this error | 15:05 |
| maciejjozefczyk | sorry, Im here | 15:05 |
| ralonsoh | (let's go back to the previous one) | 15:05 |
| maciejjozefczyk | forgot about '-3' suffix:) | 15:05 |
| ralonsoh | #link https://bugs.launchpad.net/neutron/+bug/1865453 | 15:05 |
| openstack | Launchpad bug 1865453 in neutron "neutron.tests.functional.plugins.ml2.drivers.ovn.mech_driver.test_mech_driver.TestVirtualPorts.test_virtual_port_created_before fails randomly" [High,In progress] - Assigned to Maciej Jozefczyk (maciej.jozefczyk) | 15:05 |
| ralonsoh | maciejjozefczyk, ^^ | 15:05 |
| slaweq | thx ralonsoh | 15:06 |
| maciejjozefczyk | ok, so. I tried to debug whats going on there | 15:07 |
| maciejjozefczyk | proposed a change to retry on failed asserts | 15:07 |
| maciejjozefczyk | #link https://review.opendev.org/#/c/712888/ | 15:07 |
| maciejjozefczyk | but anyways, even with those, we still have random failures of tests in that class, like 1 test per 20 runs of functional tests (based the experiment from the link) | 15:08 |
| maciejjozefczyk | and actually I don't really know what causes that | 15:08 |
| slaweq | maciejjozefczyk++ for hack with multiple functional jobs :) | 15:08 |
| maciejjozefczyk | Does it hurt as very much? I mean on gates? | 15:08 |
| maciejjozefczyk | if so, i vote for setting those tests as unstable for now | 15:09 |
| maciejjozefczyk | cause we could spend hours on debugging it... | 15:09 |
| maciejjozefczyk | and better solution would be to do virtual ports association in some more clever way, like setting a proper device_type | 15:09 |
| maciejjozefczyk | in neutron API by octavia | 15:09 |
| maciejjozefczyk | that will solve those failures instantly | 15:09 |
| maciejjozefczyk | thats what I discussed with Lucas | 15:10 |
| slaweq | maciejjozefczyk: so I'm fine with marking those tests as unstable for now, let's make other's life easier :) | 15:10 |
| njohnston | maciejjozefczyk: That is pretty darn cool what you did in that change | 15:10 |
| ralonsoh | +1 to make then unstable, for now | 15:11 |
| ralonsoh | mark* | 15:11 |
| maciejjozefczyk | njohnston, about multiplexing functionals? | 15:11 |
| slaweq | maciejjozefczyk: will You propose patch to mark it as unstable? | 15:11 |
| maciejjozefczyk | slaweq, yes | 15:11 |
| slaweq | thx a lot | 15:11 |
| njohnston | maciejjozefczyk: yes | 15:11 |
| maciejjozefczyk | njohnston, creds for Jakub Libosvar :) | 15:11 |
| slaweq | #action maciejjozefczyk to mark TestVirtualPorts functional tests as unstable for now | 15:11 |
| slaweq | ok, so lets back to second action item | 15:12 |
| slaweq | ralonsoh to check "neutron_tempest_plugin.scenario.test_multicast.MulticastTestIPv4 failing often on deletion of server" | 15:12 |
| slaweq | You said that You couldn't find root cause of this | 15:12 |
| ralonsoh | again, sorry for not finding abything | 15:12 |
| slaweq | but is it still happening? | 15:12 |
| slaweq | I didn't saw it this week | 15:12 |
| ralonsoh | maybe (maybe) this is cause by another test executed in parallel | 15:13 |
| ralonsoh | yes, I didn't see this test failing in the last 10 days | 15:13 |
| slaweq | so lets hope it will not hit as again :D | 15:14 |
| slaweq | and forget about it for now | 15:14 |
| slaweq | do You agree with that? | 15:14 |
| slaweq | :) | 15:14 |
| ralonsoh | I'll freeze it for now | 15:14 |
| slaweq | ++ | 15:14 |
| slaweq | ok, last one from last week | 15:15 |
| slaweq | slaweq to prepare etherpad to track progress with ipv6-only testing goal | 15:15 |
| slaweq | Etherpad https://etherpad.openstack.org/p/neutron-stadium-ipv6-testing | 15:15 |
| slaweq | but I didn't had time to work on any of those patches so far | 15:16 |
| slaweq | if anyone wants to help, would be great | 15:16 |
| slaweq | and also I will add this etherpad to the stadium projects topic for next weeks | 15:16 |
| slaweq | to not forget about it and to track what is still todo | 15:16 |
| slaweq | are You ok with that? | 15:16 |
| ralonsoh | yes | 15:17 |
| bcafarel | sounds good | 15:17 |
| njohnston | +1 | 15:17 |
| slaweq | ok | 15:18 |
| njohnston | I'll try to work on it as well | 15:18 |
| slaweq | thx njohnston | 15:18 |
| slaweq | ok, lets move on | 15:18 |
| slaweq | #topic Stadium projects | 15:18 |
| *** openstack changes topic to "Stadium projects (Meeting topic: neutron_ci)" | 15:18 | |
| slaweq | standardize on zuul v3 | 15:19 |
| slaweq | Etherpad: https://etherpad.openstack.org/p/neutron-train-zuulv3-py27drop | 15:19 |
| slaweq | I don't think there was any progress on that | 15:19 |
| slaweq | but maybe I missed something | 15:19 |
| njohnston | I did not see any progress either | 15:20 |
| slaweq | ok, anything else regarding stadium projects and ci for today? | 15:20 |
| maciejjozefczyk | yes | 15:21 |
| maciejjozefczyk | In OVN octavia provider driver we're setting up devstack plugin and CI (for now non voting) | 15:21 |
| maciejjozefczyk | #link https://review.opendev.org/#/c/708870/ | 15:21 |
| maciejjozefczyk | If you have a minute to check if that makes sense, would be great :) | 15:22 |
| slaweq | maciejjozefczyk: noted, I will check it tomorrow morning | 15:23 |
| slaweq | ok, lets move on | 15:24 |
| slaweq | #topic Grafana | 15:24 |
| *** openstack changes topic to "Grafana (Meeting topic: neutron_ci)" | 15:24 | |
| slaweq | #link http://grafana.openstack.org/dashboard/db/neutron-failure-rate | 15:24 |
| maciejjozefczyk | slaweq, thanks | 15:25 |
| slaweq | I checked today average number of rechecks from last days | 15:26 |
| slaweq | Average number of rechecks in last weeks: | 15:26 |
| slaweq | week 11 of 2020: 5.4 | 15:26 |
| slaweq | week 12 of 2020: 0.67 | 15:26 |
| slaweq | week 12 is now | 15:26 |
| njohnston | wow nice | 15:26 |
| slaweq | but it looks generally better than it was | 15:26 |
| maciejjozefczyk | That means its better or we don't produce code :)? | 15:26 |
| slaweq | and if we will fix some functional tests failures, should be even better | 15:27 |
| slaweq | maciejjozefczyk: my script don't reports how many patches was merged during the week but I think that we do still produce code | 15:27 |
| maciejjozefczyk | slaweq, ok :) | 15:28 |
| slaweq | looking at grafana, tempest and neutron-tempest-plugin jobs looks pretty good | 15:28 |
| bcafarel | I hope we still do :) | 15:28 |
| slaweq | specially voting ones | 15:29 |
| slaweq | the highest failure rates are still on functional job and grenade jobs | 15:29 |
| slaweq | but I know that people from nova are working on this failure which causes many grenade failures so hopefully it will be better soon | 15:30 |
| slaweq | and our functional job failures are mostly related to those ovn tests which maciejjozefczyk will mark as unstable for now | 15:31 |
| slaweq | anything else You want to add regarding grafana? | 15:31 |
| bcafarel | nothing here | 15:32 |
| slaweq | ok, lets move on | 15:32 |
| slaweq | #topic fullstack/functional | 15:32 |
| *** openstack changes topic to "fullstack/functional (Meeting topic: neutron_ci)" | 15:32 | |
| slaweq | first I wanted to ask njohnston How it's going with Marking security group fullstack tests as stable - https://review.opendev.org/710782 | 15:33 |
| slaweq | should we try to merge this? | 15:33 |
| njohnston | We have had 8 successes and 2 failures due to 'broken pipe' on ssh | 15:34 |
| njohnston | and 1 failure due to an apt cache issue that was before the tests were even started | 15:34 |
| slaweq | hmm, I'm affraid about this broken pipe failures | 15:34 |
| slaweq | do You have link to such failure? | 15:35 |
| njohnston | broken pipe example: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f5c/710782/3/check/neutron-fullstack/f5c09cb/testr_results.html | 15:35 |
| njohnston | broken pipe 2: https://82f990e55ca70e41f871-36409e2f060733ae498eef8cd27f40f4.ssl.cf5.rackcdn.com/710782/3/check/neutron-fullstack/126cd0c/testr_results.html | 15:36 |
| bcafarel | is it borken pipe, or _StringException before? | 15:36 |
| bcafarel | (_StringException reminding me of py3 fixes some time ago) | 15:36 |
| slaweq | IMHO broken pipe raised in netcat.test_connectivity() | 15:36 |
| bcafarel | ok | 15:37 |
| slaweq | so looks for me like we still have some issue there | 15:37 |
| slaweq | lets not merge it yet, I will try to take a look into that | 15:37 |
| slaweq | ok? | 15:37 |
| bcafarel | +1 | 15:37 |
| njohnston | I am of the same mind | 15:37 |
| slaweq | #action slaweq to investigate fullstack SG test broken pipe failures | 15:38 |
| slaweq | ok | 15:39 |
| slaweq | now functional tests | 15:39 |
| slaweq | I found this week 2 ovn related failures which seems new for me | 15:39 |
| slaweq | neutron.tests.functional.plugins.ml2.drivers.ovn.mech_driver.test_mech_driver.TestVirtualPorts.test_virtual_port_delete_parents | 15:39 |
| slaweq | https://1ad6f2e4d61e2bf5ff0b-20b98b64cfa6ea87451df6eaddafb782.ssl.cf5.rackcdn.com/712640/1/check/neutron-functional/17dbbf8/testr_results.html | 15:39 |
| slaweq | neutron.tests.functional.plugins.ml2.drivers.ovn.mech_driver.ovsdb.test_ovn_db_sync.TestOvnNbSyncOverTcp.test_ovn_nb_sync_log | 15:39 |
| slaweq | https://4232a5777fe8521e933a-68bc071a5cbea1ed39a41226592204b6.ssl.cf5.rackcdn.com/712474/3/check/neutron-functional/3e06670/testr_results.html | 15:39 |
| slaweq | can You take a look and check if that is something You already know? | 15:39 |
| slaweq | this first one may be related to same issue like we discussed earlier with maciejjozefczyk | 15:40 |
| slaweq | but assertion failure seems different here | 15:40 |
| maciejjozefczyk | the test_ovn_db_sync is a different problem | 15:40 |
| maciejjozefczyk | I can take a look | 15:41 |
| slaweq | maciejjozefczyk: thx a lot | 15:41 |
| slaweq | please open LP bug for it too, ok? | 15:41 |
| maciejjozefczyk | slaweq, ok | 15:41 |
| slaweq | ok | 15:42 |
| slaweq | #action maciejjozefczyk to take a look and report LP for failures in neutron.tests.functional.plugins.ml2.drivers.ovn.mech_driver.ovsdb.test_ovn_db_sync.TestOvnNbSyncOverTcp.test_ovn_nb_sync_log | 15:42 |
| slaweq | thx maciejjozefczyk | 15:42 |
| slaweq | and one last thing regarding this topic from me | 15:43 |
| maciejjozefczyk | slaweq, sure | 15:43 |
| slaweq | I was looking into grafana dashboards and I think that fullstack-with-uwsgi job is as stable as fullstack job | 15:44 |
| slaweq | should we maybe promote it to be voting? | 15:44 |
| slaweq | I don't want to do the same with functional job for now as it's failing much more often (the same as non uwsgi functional job but both are failing too often) | 15:44 |
| njohnston | +1 for making fullstack-uwsgi voting | 15:45 |
| slaweq | dashboard is here: http://grafana.openstack.org/d/Hj5IHcSmz/neutron-failure-rate?orgId=1&fullscreen&panelId=22 | 15:45 |
| slaweq | to check | 15:45 |
| bcafarel | yes it looks good compared to fullstack | 15:45 |
| slaweq | ok, I will propose patch for that | 15:46 |
| slaweq | #action slaweq to change fullstack-with-uwsgi job to be voting | 15:47 |
| slaweq | what about gating? should we add it to gate queue also, or not yet? | 15:47 |
| slaweq | I think that we can but want to know Your opinion :) | 15:47 |
| njohnston | Let's give it a couple of weeks voting and make sure we don't have buyer's remorse | 15:48 |
| njohnston | no reason to hit the accelerator | 15:49 |
| slaweq | ok | 15:49 |
| slaweq | sounds reasonable | 15:49 |
| bcafarel | also checking we do not have any uwsgi jobs in gate for now? | 15:49 |
| slaweq | bcafarel: nope | 15:49 |
| slaweq | we have only non-voting ones in check queue | 15:49 |
| slaweq | ok, lets move on | 15:50 |
| slaweq | #topic Tempest/Scenario | 15:50 |
| *** openstack changes topic to "Tempest/Scenario (Meeting topic: neutron_ci)" | 15:50 | |
| slaweq | here I don't have any new issues | 15:51 |
| slaweq | only one question | 15:51 |
| slaweq | similar to the previous one | 15:51 |
| slaweq | what do You think about making neutron-tempest-with-uwsgi job to be voting? | 15:51 |
| slaweq | http://grafana.openstack.org/d/Hj5IHcSmz/neutron-failure-rate?orgId=1&fullscreen&panelId=16 | 15:51 |
| slaweq | it also looks pretty stable | 15:52 |
| slaweq | in last 30 days it was following other jobs basically and most of the time was belowe 10% of failures | 15:52 |
| njohnston | +1 for this too | 15:52 |
| bcafarel | more voting uwsgi jobs ++ | 15:52 |
| maciejjozefczyk | ++ | 15:52 |
| slaweq | :) | 15:52 |
| slaweq | #action slaweq to make neutron-tempest-with-uwsgi voting too | 15:53 |
| slaweq | ok, that was fast | 15:53 |
| slaweq | so one last topic or today | 15:53 |
| slaweq | #topic Periodic | 15:53 |
| *** openstack changes topic to "Periodic (Meeting topic: neutron_ci)" | 15:53 | |
| slaweq | I noticed that since few days our neutron-ovn-tempest-ovs-master-fedora job is failing with RETRY_LIMIT | 15:53 |
| slaweq | if may be something related to fedora/devstack/infra | 15:54 |
| slaweq | but we should check that | 15:54 |
| slaweq | is there any volunteer? | 15:54 |
| bcafarel | I can take a look (and grab ovn folks if help needed) | 15:54 |
| slaweq | bcafarel: thx a lot | 15:54 |
| slaweq | #action bcafarel to check neutron-ovn-tempest-ovs-master-fedora RETRY_LIMIT failures | 15:55 |
| slaweq | ok, that's all from my side for today | 15:55 |
| slaweq | is there anything else You want to talk about? | 15:55 |
| maciejjozefczyk | bcafarel, ping me if you need any help about ovn | 15:56 |
| bcafarel | :) maciejjozefczyk thanks will do (if it is not a generic fedora/infra issue)à | 15:57 |
| slaweq | thx maciejjozefczyk :) | 15:57 |
| slaweq | ok, I think we are done for today | 15:57 |
| njohnston | o/ | 15:58 |
| slaweq | thx for atttending | 15:58 |
| slaweq | #endmeeting | 15:58 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 15:58 | |
| openstack | Meeting ended Wed Mar 18 15:58:06 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:58 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-03-18-15.00.html | 15:58 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-03-18-15.00.txt | 15:58 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-03-18-15.00.log.html | 15:58 |
| *** yamamoto has joined #openstack-meeting-3 | 15:58 | |
| *** slaweq has quit IRC | 16:28 | |
| *** baojg has joined #openstack-meeting-3 | 16:37 | |
| *** jraju__ has quit IRC | 16:54 | |
| *** diablo_rojo has joined #openstack-meeting-3 | 17:05 | |
| *** rmcall has joined #openstack-meeting-3 | 17:26 | |
| *** yamamoto has quit IRC | 17:44 | |
| *** diablo_rojo has quit IRC | 18:43 | |
| *** diablo_rojo has joined #openstack-meeting-3 | 18:45 | |
| *** e0ne has quit IRC | 18:46 | |
| *** maciejjozefczyk has quit IRC | 18:56 | |
| *** diablo_rojo has quit IRC | 19:52 | |
| *** gmann is now known as gmann_afk | 19:57 | |
| *** diablo_rojo has joined #openstack-meeting-3 | 20:00 | |
| *** ralonsoh has quit IRC | 20:01 | |
| *** jamesmcarthur has quit IRC | 20:09 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 20:28 | |
| *** gmann_afk is now known as gmann | 20:33 | |
| *** rmcall has quit IRC | 20:38 | |
| *** jamesmcarthur has quit IRC | 20:52 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 20:53 | |
| *** jamesmcarthur has quit IRC | 20:58 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 20:58 | |
| *** jamesmcarthur has quit IRC | 21:00 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 21:03 | |
| *** jamesmcarthur has quit IRC | 21:08 | |
| *** slaweq has joined #openstack-meeting-3 | 21:15 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 21:18 | |
| *** jamesmcarthur has quit IRC | 21:40 | |
| *** slaweq has quit IRC | 22:00 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 22:20 | |
| *** jamesmcarthur has quit IRC | 22:37 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 22:39 | |
| *** macz_ has quit IRC | 22:56 | |
| *** jamesmcarthur has quit IRC | 23:16 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 23:20 | |
| *** jamesmcarthur has quit IRC | 23:21 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 23:23 | |
| *** jamesmcarthur has quit IRC | 23:23 | |
| *** diablo_rojo has quit IRC | 23:29 | |
| *** diablo_rojo has joined #openstack-meeting-3 | 23:31 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 23:53 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!