| *** jamesmcarthur has joined #openstack-meeting-3 | 00:01 | |
| *** jamesmcarthur has quit IRC | 00:04 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 00:04 | |
| *** bbowen has joined #openstack-meeting-3 | 00:27 | |
| *** jamesmcarthur has quit IRC | 00:29 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 00:32 | |
| *** jamesmcarthur has quit IRC | 01:02 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 01:06 | |
| *** macz_ has joined #openstack-meeting-3 | 01:24 | |
| *** liuyulong has quit IRC | 01:28 | |
| *** macz_ has quit IRC | 01:28 | |
| *** jamesmcarthur has quit IRC | 02:01 | |
| *** macz_ has joined #openstack-meeting-3 | 02:17 | |
| *** macz_ has quit IRC | 02:21 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 02:39 | |
| *** jamesmcarthur has quit IRC | 02:46 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 03:02 | |
| *** jamesmcarthur has quit IRC | 03:06 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 03:08 | |
| *** apetrich has quit IRC | 03:09 | |
| *** jamesmcarthur has quit IRC | 03:14 | |
| *** psachin has joined #openstack-meeting-3 | 03:32 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 03:38 | |
| *** jamesmcarthur has quit IRC | 03:47 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 04:41 | |
| *** jamesmcarthur has quit IRC | 04:54 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 05:05 | |
| *** jamesmcarthur has quit IRC | 05:07 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 05:11 | |
| *** links has joined #openstack-meeting-3 | 05:29 | |
| *** jamesmcarthur has quit IRC | 05:33 | |
| *** macz_ has joined #openstack-meeting-3 | 05:53 | |
| *** macz_ has quit IRC | 05:58 | |
| *** vesper11 has quit IRC | 07:16 | |
| *** vesper has joined #openstack-meeting-3 | 07:16 | |
| *** apetrich has joined #openstack-meeting-3 | 07:27 | |
| *** njohnston has quit IRC | 07:45 | |
| *** belmoreira has joined #openstack-meeting-3 | 07:46 | |
| *** njohnston has joined #openstack-meeting-3 | 07:48 | |
| *** e0ne has joined #openstack-meeting-3 | 07:56 | |
| *** slaweq has joined #openstack-meeting-3 | 07:59 | |
| *** ralonsoh has joined #openstack-meeting-3 | 08:01 | |
| *** stephenfin has quit IRC | 08:19 | |
| *** stephenfin has joined #openstack-meeting-3 | 08:30 | |
| *** slaweq has quit IRC | 08:41 | |
| *** slaweq has joined #openstack-meeting-3 | 09:18 | |
| *** e0ne has quit IRC | 10:03 | |
| *** e0ne has joined #openstack-meeting-3 | 10:04 | |
| *** trident has quit IRC | 10:29 | |
| *** trident has joined #openstack-meeting-3 | 10:31 | |
| *** trident has quit IRC | 10:33 | |
| *** trident has joined #openstack-meeting-3 | 10:37 | |
| *** raildo has joined #openstack-meeting-3 | 11:31 | |
| *** links has quit IRC | 12:27 | |
| *** links has joined #openstack-meeting-3 | 12:28 | |
| *** Adri2000 has quit IRC | 13:39 | |
| *** liuyulong has joined #openstack-meeting-3 | 13:59 | |
| *** maciejjozefczyk has joined #openstack-meeting-3 | 14:19 | |
| *** psachin has quit IRC | 14:27 | |
| *** Adri2000 has joined #openstack-meeting-3 | 14:29 | |
| *** jraju__ has joined #openstack-meeting-3 | 14:49 | |
| *** links has quit IRC | 14:49 | |
| *** macz_ has joined #openstack-meeting-3 | 14:52 | |
| slaweq | #startmeeting neutron_ci | 15:00 |
|---|---|---|
| openstack | Meeting started Wed Mar 25 15:00:57 2020 UTC and is due to finish in 60 minutes. The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| slaweq | hi | 15:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:01 |
| *** openstack changes topic to " (Meeting topic: neutron_ci)" | 15:01 | |
| openstack | The meeting name has been set to 'neutron_ci' | 15:01 |
| ralonsoh | hi | 15:01 |
| bcafarel | hello | 15:01 |
| maciejjozefczyk | hey | 15:01 |
| slaweq | ok, lets start | 15:01 |
| slaweq | #topic Actions from previous meetings | 15:01 |
| *** openstack changes topic to "Actions from previous meetings (Meeting topic: neutron_ci)" | 15:01 | |
| slaweq | first one | 15:02 |
| slaweq | maciejjozefczyk to mark TestVirtualPorts functional tests as unstable for now | 15:02 |
| njohnston | o/ | 15:02 |
| maciejjozefczyk | slaweq, thats done | 15:02 |
| slaweq | thx maciejjozefczyk :) | 15:02 |
| maciejjozefczyk | #link https://review.opendev.org/#/c/713860/ | 15:02 |
| slaweq | ok, next one | 15:03 |
| slaweq | slaweq to investigate fullstack SG test broken pipe failures | 15:03 |
| slaweq | and unfortunatelly I didn't had time to check it | 15:03 |
| slaweq | sorry | 15:03 |
| slaweq | #action slaweq to investigate fullstack SG test broken pipe failures | 15:03 |
| slaweq | I will try this week | 15:03 |
| slaweq | next one | 15:04 |
| slaweq | maciejjozefczyk to take a look and report LP for failures in neutron.tests.functional.plugins.ml2.drivers.ovn.mech_driver.ovsdb.test_ovn_db_sync.TestOvnNbSyncOverTcp.test_ovn_nb_sync_log | 15:04 |
| maciejjozefczyk | #link https://bugs.launchpad.net/neutron/+bug/1868110 | 15:05 |
| openstack | Launchpad bug 1868110 in neutron "[OVN] neutron.tests.functional.plugins.ml2.drivers.ovn.mech_driver.ovsdb.test_ovn_db_sync.TestOvnNbSyncOverTcp.test_ovn_nb_sync_log randomly fails" [High,Confirmed] - Assigned to Maciej Jozefczyk (maciej.jozefczyk) | 15:05 |
| maciejjozefczyk | I tried to figure it out, but for now I wasn't able to reproduce it, but I didn't spend too much time on it. | 15:05 |
| maciejjozefczyk | This week I'm gonna spend more time on it. | 15:05 |
| slaweq | today I saw yet another similar issue https://ab00d9261534a206496f-e4dcf05f554cd2b2192f6b35230c9943.ssl.cf1.rackcdn.com/708985/8/gate/neutron-functional/2132176/testr_results.html | 15:06 |
| maciejjozefczyk | slaweq, that looks like different test in the same clas | 15:07 |
| maciejjozefczyk | class* | 15:07 |
| maciejjozefczyk | thanks for the link | 15:07 |
| slaweq | #action maciejjozefczyk to take a look and report LP for failures in neutron.tests.functional.plugins.ml2.drivers.ovn.mech_driver.ovsdb.test_ovn_db_sync.TestOvnNbSyncOverTcp.test_ovn_nb_sync_log | 15:09 |
| slaweq | next one | 15:09 |
| slaweq | even next 2 :) | 15:09 |
| slaweq | slaweq to change fullstack-with-uwsgi job to be voting | 15:09 |
| slaweq | and | 15:09 |
| slaweq | slaweq to make neutron-tempest-with-uwsgi voting too | 15:09 |
| slaweq | both done in same patch | 15:10 |
| slaweq | Patch https://review.opendev.org/714917 | 15:10 |
| bcafarel | and both passing (luckily) | 15:10 |
| slaweq | yes :) | 15:11 |
| slaweq | so please review it :) | 15:11 |
| slaweq | and the last one from previous week | 15:11 |
| slaweq | bcafarel to check neutron-ovn-tempest-ovs-master-fedora RETRY_LIMIT failures | 15:11 |
| * bcafarel looks for LP link | 15:12 | |
| bcafarel | root cause was new fedora image did not have directory created for cache, infra++ folks quickly fixed it | 15:12 |
| bcafarel | #link https://bugs.launchpad.net/devstack/+bug/1868076 | 15:12 |
| openstack | Launchpad bug 1868076 in devstack "Fedora jobs fail in setup-devstack-cache: find: ‘/opt/cache/files’: No such file or directory" [Undecided,Fix released] | 15:12 |
| slaweq | thx bcafarel and infra-root :) | 15:12 |
| slaweq | ok, lets move on | 15:13 |
| slaweq | #topic Stadium projects | 15:13 |
| *** openstack changes topic to "Stadium projects (Meeting topic: neutron_ci)" | 15:13 | |
| slaweq | njohnston: any updates about migration to zuulv3? | 15:14 |
| njohnston | I think it's the same as yesterday's team meeting - midonet, odl, and one change in bagpipe | 15:14 |
| njohnston | but I have been in meetings continuously to this point so I have not had a chance to check today, sorry | 15:15 |
| slaweq | ok | 15:15 |
| slaweq | no problem | 15:15 |
| slaweq | that isn't urgent for now :) | 15:15 |
| slaweq | about IPv6-only deployments, I don't have any updates too | 15:15 |
| slaweq | do You have anything else regarding stadium projects and ci for today? | 15:16 |
| njohnston | nope! Not aware of any pernicious issues. | 15:16 |
| slaweq | ok, so lets move on | 15:17 |
| slaweq | #topic Grafana | 15:17 |
| *** openstack changes topic to "Grafana (Meeting topic: neutron_ci)" | 15:17 | |
| slaweq | #link http://grafana.openstack.org/dashboard/db/neutron-failure-rate | 15:17 |
| slaweq | (sorry that I forgot it at the beginning) | 15:17 |
| slaweq | Average number of rechecks in last weeks: | 15:18 |
| slaweq | week 12 of 2020: 0.74 | 15:18 |
| slaweq | week 13 of 2020: 0.67 | 15:18 |
| slaweq | those numners from this and previous week looks really good :) | 15:18 |
| slaweq | and when looking at grafana, it looks for me that we are in pretty good shape currently | 15:19 |
| maciejjozefczyk | \o/ | 15:19 |
| slaweq | still same issues which we had, but fortunatelly mostly with non-voting jobs :) | 15:19 |
| *** macz_ has quit IRC | 15:20 | |
| *** macz_ has joined #openstack-meeting-3 | 15:21 | |
| bcafarel | one step at a time, stabler voting jobs is already great! | 15:21 |
| slaweq | bcafarel: indeed :) | 15:22 |
| njohnston | +1 | 15:22 |
| slaweq | and, as our gate is working pretty good, I don't have many examples of new failures to discuss today | 15:23 |
| slaweq | #topic fullstack/functional | 15:23 |
| *** openstack changes topic to "fullstack/functional (Meeting topic: neutron_ci)" | 15:23 | |
| slaweq | I saw again issue in neutron.tests.functional.agent.linux.test_keepalived.KeepalivedManagerTestCase | 15:24 |
| slaweq | https://116f9c4d7ae86f7f062e-eb80c0e4aee14229f3b21e03b2a44f1c.ssl.cf2.rackcdn.com/712446/1/check/neutron-functional-with-uwsgi/cdb2795/testr_results.html | 15:24 |
| slaweq | ralonsoh: maybe You will want to take a look ^^ :) | 15:24 |
| ralonsoh | yes | 15:24 |
| ralonsoh | but this problem is, again, in the ns deletion | 15:24 |
| ralonsoh | pfffff | 15:24 |
| slaweq | yep :/ | 15:24 |
| slaweq | IIRC it's always in ns deletion | 15:25 |
| ralonsoh | ok, I'll take care of it | 15:25 |
| slaweq | maybe we should add some "safe_cleanup" method in test | 15:25 |
| slaweq | catch timeout there and retry | 15:25 |
| ralonsoh | the problem is in the privsep method and the eventlet library | 15:26 |
| ralonsoh | if, for any reason, we give the GIL, the timeout will happen | 15:26 |
| ralonsoh | (I'll check this later) | 15:26 |
| slaweq | ok, thx | 15:27 |
| slaweq | may I add action with this for You? | 15:27 |
| ralonsoh | sure | 15:28 |
| slaweq | #action ralonsoh to check (again) issue with ns deletion in neutron.tests.functional.agent.linux.test_keepalived.KeepalivedManagerTestCase | 15:29 |
| slaweq | ok, and that's all from me regarding functional/fullstack jobs | 15:29 |
| slaweq | do You have anything else You want to discuss, related to those jobs? | 15:29 |
| njohnston | The security group tests for fullstack marked as stable - https://review.opendev.org/#/c/710782/ | 15:30 |
| njohnston | We have rechecked it 5 times since I removed the iptables-hybrid scenario and the fullstack job has not failed | 15:30 |
| ralonsoh | so we drop LB testing | 15:31 |
| njohnston | sorry I said iptables hybrid, I meant LB | 15:31 |
| slaweq | but should we really drop this LB scenario from it? | 15:31 |
| njohnston | yes, I think it's better to test the two other scenarios than to not test any | 15:31 |
| slaweq | can You maybe just comment this scenario and add TODO to bring it back when it will be fixed? | 15:32 |
| njohnston | sure thing | 15:32 |
| ralonsoh | +1 | 15:32 |
| slaweq | thx | 15:32 |
| slaweq | and I will +2 it :) | 15:33 |
| slaweq | anything else related to fullstack/functional or we can move on? | 15:34 |
| njohnston | go ahead | 15:36 |
| slaweq | ok, lets move on | 15:36 |
| slaweq | #topic Tempest/Scenario | 15:36 |
| *** openstack changes topic to "Tempest/Scenario (Meeting topic: neutron_ci)" | 15:36 | |
| slaweq | here I also have only 1 issue to mention | 15:36 |
| slaweq | I spotted again issue with server termination on multicast test | 15:36 |
| slaweq | https://103900b4a03cdd60217d-625a0eb0440aa527fbdb216e8991f5a6.ssl.cf5.rackcdn.com/714726/1/check/neutron-ovn-tempest-ovs-release/2b52c20/testr_results.html | 15:36 |
| *** jamesmcarthur has joined #openstack-meeting-3 | 15:37 | |
| slaweq | ralonsoh: do You want to take a look at it or do You want me to check it this week? | 15:37 |
| ralonsoh | I don't know if I'll have time this week | 15:38 |
| ralonsoh | sorry | 15:38 |
| bcafarel | I think I saw it pop on few stable backports too (at least I remember typing "recheck test_multicast_between_vms_on_same_network") | 15:38 |
| slaweq | ok, I will check that one | 15:38 |
| slaweq | #action slaweq to check server termination on multicast test | 15:38 |
| slaweq | bcafarel: good to know that it's not only on master branch :) | 15:39 |
| slaweq | and that's all from me regarding tempest jobs | 15:39 |
| slaweq | do You have anything else to discuss? | 15:39 |
| slaweq | ok, so lets move on | 15:41 |
| slaweq | one last thing from me or today | 15:41 |
| slaweq | #topic Periodic | 15:41 |
| *** openstack changes topic to "Periodic (Meeting topic: neutron_ci)" | 15:41 | |
| slaweq | neutron-tempest-postgres-full - failing since 18.03 constantly | 15:41 |
| slaweq | Errors like http://zuul.openstack.org/build/95cb0e8c4c664727a2235bf5ee8d76ca/log/controller/logs/screen-q-svc.txt?severity=4 every day | 15:41 |
| slaweq | it's failing only on postgresql | 15:42 |
| ralonsoh | another postgresql incompatibility? in our ORM definition | 15:43 |
| slaweq | and IMO some db expert should take a look | 15:43 |
| ralonsoh | that happened before | 15:43 |
| slaweq | it's either one of our patches merged around 17.03 or some pgsql update | 15:44 |
| slaweq | I will open LP for that | 15:44 |
| slaweq | and I will check if PG versions didn't changed recently in jobs | 15:44 |
| slaweq | but I don't think I will have time to check something more with this issue | 15:45 |
| slaweq | #action slaweq to report LP about PGSQL periodic job failures | 15:46 |
| slaweq | if You have any experience with postgresql, and some cycles, You are more than welcome to check it :) | 15:46 |
| slaweq | ok, and that's all on my side for today | 15:47 |
| *** jraju__ has quit IRC | 15:47 | |
| slaweq | #topic Open discussion | 15:47 |
| *** openstack changes topic to "Open discussion (Meeting topic: neutron_ci)" | 15:47 | |
| slaweq | do You have anything else related to our ci to discuss today? | 15:47 |
| ralonsoh | no | 15:47 |
| bcafarel | all good here | 15:48 |
| slaweq | ok, so I will give You few minutes back :) | 15:49 |
| slaweq | thx for attending | 15:49 |
| bcafarel | yay | 15:49 |
| bcafarel | o/ | 15:49 |
| slaweq | and have a great week | 15:49 |
| slaweq | o/ | 15:49 |
| slaweq | #endmeeting | 15:50 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 15:50 | |
| openstack | Meeting ended Wed Mar 25 15:50:01 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:50 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-03-25-15.00.html | 15:50 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-03-25-15.00.txt | 15:50 |
| ralonsoh | bye | 15:50 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-03-25-15.00.log.html | 15:50 |
| *** liuyulong has quit IRC | 16:42 | |
| *** belmoreira has quit IRC | 17:01 | |
| *** jamesmcarthur has quit IRC | 17:12 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 17:13 | |
| *** jamesmcarthur has quit IRC | 17:24 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 17:24 | |
| *** e0ne has quit IRC | 17:36 | |
| *** e0ne has joined #openstack-meeting-3 | 17:37 | |
| *** e0ne has quit IRC | 17:37 | |
| *** e0ne has joined #openstack-meeting-3 | 17:38 | |
| *** e0ne has quit IRC | 17:39 | |
| *** jamesmcarthur has quit IRC | 18:01 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 18:03 | |
| *** jamesmcarthur has quit IRC | 18:08 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 18:15 | |
| *** jamesmcarthur has quit IRC | 18:22 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 18:23 | |
| *** jamesmcarthur has quit IRC | 18:25 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 18:25 | |
| *** irclogbot_3 has quit IRC | 18:37 | |
| *** jamesmcarthur has quit IRC | 18:58 | |
| *** maciejjozefczyk has quit IRC | 19:01 | |
| *** irclogbot_1 has joined #openstack-meeting-3 | 19:02 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 19:06 | |
| *** macz_ has quit IRC | 19:17 | |
| *** ralonsoh has quit IRC | 19:26 | |
| *** irclogbot_1 has quit IRC | 19:37 | |
| *** irclogbot_3 has joined #openstack-meeting-3 | 19:40 | |
| *** irclogbot_3 has quit IRC | 19:42 | |
| *** irclogbot_2 has joined #openstack-meeting-3 | 19:45 | |
| *** macz_ has joined #openstack-meeting-3 | 19:50 | |
| *** irclogbot_2 has quit IRC | 20:00 | |
| *** jamesmcarthur has quit IRC | 20:01 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 20:03 | |
| *** irclogbot_0 has joined #openstack-meeting-3 | 20:03 | |
| *** jamesmcarthur has quit IRC | 20:08 | |
| *** irclogbot_0 has quit IRC | 20:12 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 20:15 | |
| *** irclogbot_2 has joined #openstack-meeting-3 | 20:15 | |
| *** diablo_rojo has quit IRC | 20:16 | |
| *** irclogbot_2 has quit IRC | 20:16 | |
| *** irclogbot_3 has joined #openstack-meeting-3 | 20:21 | |
| *** raildo_ has joined #openstack-meeting-3 | 20:58 | |
| *** diablo_rojo has joined #openstack-meeting-3 | 21:00 | |
| *** raildo has quit IRC | 21:00 | |
| *** jamesmcarthur has quit IRC | 21:21 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 21:21 | |
| *** jamesmcarthur has quit IRC | 21:26 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 21:41 | |
| *** raildo_ has quit IRC | 22:34 | |
| *** slaweq has quit IRC | 22:34 | |
| *** slaweq has joined #openstack-meeting-3 | 22:46 | |
| *** slaweq has quit IRC | 22:51 | |
| *** macz_ has quit IRC | 23:24 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!