Wednesday, 2020-02-12

*** slaweq has joined #openstack-meeting-300:11
*** slaweq has quit IRC00:16
*** negronjl has quit IRC01:02
*** negronjl has joined #openstack-meeting-301:08
*** slaweq has joined #openstack-meeting-301:11
*** jamesmcarthur has joined #openstack-meeting-301:12
*** slaweq has quit IRC01:16
*** jamesmcarthur has quit IRC01:23
*** jamesmcarthur has joined #openstack-meeting-301:37
*** yamamoto has joined #openstack-meeting-301:57
*** slaweq has joined #openstack-meeting-302:11
*** yamamoto has quit IRC02:12
*** yamamoto has joined #openstack-meeting-302:13
*** slaweq has quit IRC02:16
*** yamamoto has quit IRC02:17
*** ricolin has joined #openstack-meeting-302:19
*** jamesmcarthur has quit IRC02:38
*** macz has joined #openstack-meeting-302:40
*** yamamoto has joined #openstack-meeting-302:40
*** macz has quit IRC02:45
*** jamesmcarthur has joined #openstack-meeting-302:53
*** yamamoto has quit IRC03:01
*** yamamoto has joined #openstack-meeting-303:03
*** slaweq has joined #openstack-meeting-303:11
*** slaweq has quit IRC03:16
*** psachin has joined #openstack-meeting-303:22
*** jamesmcarthur has quit IRC03:55
*** slaweq has joined #openstack-meeting-304:11
*** slaweq has quit IRC04:16
*** psachin has quit IRC04:33
*** macz has joined #openstack-meeting-304:41
*** macz has quit IRC04:46
*** slaweq has joined #openstack-meeting-305:11
*** slaweq has quit IRC05:16
*** macz has joined #openstack-meeting-305:41
*** macz has quit IRC05:45
*** jamesmcarthur has joined #openstack-meeting-305:57
*** macz has joined #openstack-meeting-305:58
*** jamesmcarthur has quit IRC06:01
*** macz has quit IRC06:03
*** bobmel has joined #openstack-meeting-306:03
*** bobmel has quit IRC06:08
*** slaweq has joined #openstack-meeting-306:11
*** slaweq has quit IRC06:16
*** slaweq has joined #openstack-meeting-307:08
*** slaweq has quit IRC07:15
*** macz has joined #openstack-meeting-307:24
*** macz has quit IRC07:29
*** lpetrut has joined #openstack-meeting-307:31
*** ianychoi has joined #openstack-meeting-307:46
*** slaweq has joined #openstack-meeting-307:51
*** slaweq has quit IRC07:57
*** slaweq has joined #openstack-meeting-307:59
*** yamamoto has quit IRC08:30
*** ralonsoh has joined #openstack-meeting-308:38
*** yamamoto has joined #openstack-meeting-308:44
*** e0ne has joined #openstack-meeting-309:54
*** raildo has joined #openstack-meeting-311:47
*** ianychoi has quit IRC12:25
*** jamesmcarthur has joined #openstack-meeting-313:09
*** jamesmcarthur has quit IRC13:35
*** jamesmcarthur has joined #openstack-meeting-313:45
*** liuyulong has joined #openstack-meeting-313:51
*** jamesmcarthur has quit IRC14:11
*** jamesmcarthur has joined #openstack-meeting-314:12
*** jamesmcarthur has quit IRC14:17
*** jamesmcarthur has joined #openstack-meeting-314:42
*** jamesmcarthur has quit IRC14:47
slaweq#startmeeting neutron_ci15:00
openstackMeeting started Wed Feb 12 15:00:10 2020 UTC and is due to finish in 60 minutes.  The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot.15:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:00
slaweqhi15:00
*** openstack changes topic to " (Meeting topic: neutron_ci)"15:00
openstackThe meeting name has been set to 'neutron_ci'15:00
ralonsohhi15:00
*** jamesmcarthur has joined #openstack-meeting-315:00
slaweqbcafarel: is on pto but maybe njohnston will join soon15:01
njohnstonI am here15:01
slaweqhi :)15:01
njohnstonsorry, also on a videoconference with the Octavia folks15:01
slaweqso lets start15:01
slaweqnjohnston: no problem15:01
slaweq#topic Actions from previous meetings15:01
*** openstack changes topic to "Actions from previous meetings (Meeting topic: neutron_ci)"15:01
slaweqslaweq to backport https://review.opendev.org/#/c/695834/ to stable branches in neutron-vpnaas15:02
slaweqI was checking that and I had some doubts about it15:02
slaweqfinally gmann and amotoki fixed stable/rocky branch in other way so this wasn't needed15:02
slaweqnext one15:03
slaweqslaweq to update grafana dashboard with missing jobs15:03
slaweqpatch https://review.opendev.org/70627115:03
slaweqwaiting for second +2 now15:03
slaweqnext one15:03
slaweqslaweq to open LP related to fullstack placement issue15:03
slaweqBug reported: https://bugs.launchpad.net/neutron/+bug/186217715:03
openstackLaunchpad bug 1862177 in neutron "Fullstack tests failing due to problem with connection to the fake placement service" [High,In progress] - Assigned to Lajos Katona (lajos-katona)15:03
slaweqlajoskatona and rubasov are on it already15:04
slaweqso we are in good hands :)15:04
slaweqnext one15:04
slaweqslaweq to open LP related to "hang" neutron-server15:04
slaweqBug reported: https://bugs.launchpad.net/neutron/+bug/186217815:05
openstackLaunchpad bug 1862178 in neutron "Fullstack tests failing due to "hang" neutron-server process" [High,In progress] - Assigned to Rodolfo Alonso (rodolfo-alonso-hernandez)15:05
ralonsohtwo patches15:05
ralonsoh#link https://review.opendev.org/#/c/707151/15:05
ralonsoh#link https://review.opendev.org/#/c/707222/15:05
ralonsoh(second one is good stuff)15:05
slaweqand do You think this will solve this issue?15:06
ralonsohI've tested the second one with my dev env15:06
ralonsohand I see a big performance improvement15:06
ralonsoha) the delete operation (bulk), removing unneded physnets15:06
ralonsohb) adding new VLAN tags in bulk mode15:07
ralonsohand of course, reducing the number of tags15:07
ralonsohthat;s all15:07
slaweqok, thx ralonsoh for those patches - I hope this will help with this issue in fullstack tests too15:08
slaweqok, next one15:08
slaweqralonsoh  to check missing project_id issue15:08
ralonsohno luck sorry....15:08
slaweqnp15:08
ralonsohI can't find were/why/how this is happening...15:08
ralonsoh*where15:08
slaweqshould we simply maybe send some patch to log in such case both expected and actual dict15:09
slaweqmaybe we will at least will know in which one it's missing15:10
ralonsohwe can see that in the response15:10
ralonsohthe problem is to find where this project_id is filtered and deleted from this returned dict15:11
ralonsohand, why is this happening 1 out of 500 tests?15:11
slaweqok, I see now15:12
slaweqit's filtered out from the "actual" dict than15:12
slaweq*then15:12
ralonsohexactly, but no one (SDK, client or server) should do this15:13
ralonsohI tried to find something related to the project_id/tenant_id migration15:13
ralonsohbecause in some places, some black magic is done to convert one into the other15:13
ralonsohbut project_id should be always there15:13
ralonsoh(not tenant_id, that should be removed)15:14
slaweqbut afaik tempest has got own implementation of clients15:15
ralonsohI know15:15
slaweqit don't use OpenStack SDK or neutronclient15:15
slaweqmaybe there is some bug there?15:15
ralonsohlet me check again this15:16
slaweqralonsoh: ok, thx15:16
ralonsohbut this is the nth time I try to find this bug15:16
slaweqor maybe we should add some debug log of every response which is going to be send from neutron-server15:16
*** jamesmcarthur has quit IRC15:16
slaweqso we than can confirm if that wasn't send from server or was filtered out on client's side15:16
ralonsohwe could15:17
slaweqok, so You will take another look into it this week, right?15:18
ralonsohsure!15:18
slaweqthx a lot15:18
slaweq#ralonsoh to check again mystery of vanishing project_id15:19
ralonsohhahahaha15:19
slaweq:)15:19
slaweqok, that's all actions from last week15:19
slaweqdo You have anything else to add here or can we move on?15:20
ralonsohno thanks15:20
slaweqok, so lets move on15:20
slaweq#topic Stadium projects15:20
*** openstack changes topic to "Stadium projects (Meeting topic: neutron_ci)"15:20
slaweqstandardize on zuul v315:20
slaweqEtherpad: https://etherpad.openstack.org/p/neutron-train-zuulv3-py27drop15:20
slaweqthere was slow progress this week15:20
njohnstonSo I added a summary section at the top of that etherpad15:20
njohnstonso it's easy to find the pending updates15:21
slaweqthx njohnston - that's really helpful15:21
njohnstonfor zuulv3 we have 3 projects with pending changes, 2 that have no activity15:21
slaweqand we are really close to finish it15:21
njohnstonyep!15:21
njohnstonI also noted a few remaining py27 things, including the neutron-tempest-plugin change15:21
slaweqyes, neutron-tempest-plugin is something what I'm aware of15:22
slaweqpatch is ready15:22
njohnstonyep!15:22
slaweqbut I would like to first merge few fixes, release last version with support for py2715:22
slaweqand than drop this support15:22
slaweqnjohnston: and I will check this midonet patch today15:24
*** lpetrut has quit IRC15:24
slaweqanything else regarding stadium projects?15:24
njohnstonnope, things are looking good.15:24
slaweqI just commented in https://review.opendev.org/#/c/695094/15:26
slaweqcan You check it and tell me what do You think about it?15:26
ralonsohsure15:26
njohnstonwill do15:27
slaweqthx15:27
slaweqok, so lets move on15:28
slaweq#topic Grafana15:28
*** openstack changes topic to "Grafana (Meeting topic: neutron_ci)"15:28
slaweq#link http://grafana.openstack.org/dashboard/db/neutron-failure-rate15:28
slaweqfunctional jobs were recently failing pretty often15:30
slaweqbut as I checked many of those failures were in patches related to ovn15:30
*** jamesmcarthur has joined #openstack-meeting-315:31
slaweqother than that I think that graphs are pretty good15:32
ralonsohI think so15:33
slaweqok, lets talk about few issues which I found recently15:34
slaweq#topic fullstack/functional15:34
*** openstack changes topic to "fullstack/functional (Meeting topic: neutron_ci)"15:34
slaweqhere I have only one thing to mention15:34
slaweq    Unauthorized commands like "ping" or "ncat":15:34
slaweq    https://2a0154cb9a3e47bde3ed-4a9629bf7847ad9c8b03c9755148c549.ssl.cf1.rackcdn.com/705660/4/check/neutron-functional/2e5030b/testr_results.html15:34
slaweq    https://656129f4adff35088518-c39e8636195a8a58924c560773952ce4.ssl.cf1.rackcdn.com/705480/3/check/neutron-functional/e49a784/testr_results.html15:34
ralonsohsorry for that15:34
slaweqand also probably same issue https://ad181adc6d8db459c7ce-fbb316944f0ca23c676e132d61555672.ssl.cf1.rackcdn.com/705237/4/check/neutron-functional/ed9cfb5/testr_results.html15:34
ralonsohbug opened and patch submitted15:34
slaweqI wonder why it not happens all the time15:34
ralonsoh#link https://review.opendev.org/#/c/707368/15:34
ralonsohyes, that was my question15:35
ralonsohand I have no answer15:35
ralonsohbut this patch should solve this problem15:35
slaweqand also in https://656129f4adff35088518-c39e8636195a8a58924c560773952ce4.ssl.cf1.rackcdn.com/705480/3/check/neutron-functional/e49a784/testr_results.html error was because of unauthorized "ping" command15:35
slaweqbut ping is in this rootwrap filters file15:36
ralonsoh??15:36
slaweqralonsoh: first failed test in the link above15:37
slaweqit failed with error:15:37
ralonsohyes yes15:37
ralonsohbut why??15:37
slaweqUnauthorized command: ip netns exec test-19869f4e-e878-47fe-8cf8-90b60f6269e1 ping 192.178.0.2 -W 1 -c 3 (no filter matched)15:37
slaweqralonsoh: yeah, that's the question15:37
ralonsohoook ok15:37
ralonsohmaybe I229e926341c5e6c8b06f59950e3ae09864d0f1f615:37
ralonsohis the problem15:38
ralonsohlet me review this patch15:38
ralonsohhttps://review.opendev.org/#/c/705065/15:38
slaweqok15:38
slaweqso I will assign it as an action to You, ok?15:38
ralonsohok15:39
slaweq#action ralonsoh to check issues with unauthorized ping and ncat commands in functional tests15:40
slaweqok, so lets talk about scenario jobs15:41
slaweq#topic Tempest/Scenario15:41
*** openstack changes topic to "Tempest/Scenario (Meeting topic: neutron_ci)"15:41
slaweqfirst, I proposed patch to increase timeout for tempest-ipv6-only job: https://review.opendev.org/70735615:41
slaweqI will respond to haleyb's comment there in a minute15:41
haleyb:)15:41
ralonsohOVN slow is taking almost 3 hours and sometimes I've seen timeouts15:43
slaweqhaleyb: answered :)15:43
slaweqralonsoh: yes, but that's slow tests15:43
slaweqtempest-slow-py3 also takes almost 3 hours15:43
slaweqok, lets move on quickly to other issues15:44
slaweqI saw couple of times ssh problem due to "socket.timeout: timed out" error in various tests, like:15:44
slaweq    https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_9aa/703376/5/check/neutron-tempest-dvr/9aa046a/testr_results.html15:44
slaweq    https://db85fb7d3af8e26f7154-0d96b608ecbdac6c8248619c1dff0910.ssl.cf5.rackcdn.com/704833/4/check/neutron-tempest-dvr/67fd708/testr_results.html15:44
slaweq    https://19574e4665a40f62095e-6b9500683e6a67d31c1bad572acf67ba.ssl.cf1.rackcdn.com/705982/6/check/neutron-tempest-dvr/8f3fbd0/testr_results.html15:44
slaweq    https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_37a/705452/5/check/neutron-tempest-dvr-ha-multinode-full/37a65a9/testr_results.html15:44
slaweqmostly in neutron-tempest-dvr job15:44
slaweqbut I saw it also in multinode dvr job15:44
slaweqI think it's new issue and we should investigate it15:45
slaweqany volunteers?15:45
slaweqok, I will report new bug and will try to look into logs15:46
slaweq#action slaweq to report issue with ssh timeout on dvr jobs and check logs there15:46
slaweqother than that, I saw one issue with nova's revert_resize, like e.g. in15:47
slaweq    https://3e447a3e4caf9c486a4d-b84d447537aa444ce20bcf5414a5ef0e.ssl.cf5.rackcdn.com/707248/1/check/neutron-tempest-dvr-ha-multinode-full/057233d/testr_results.html15:47
slaweq    https://3ceba9124358a5c9851b-33ba471340a760763569a038d91ca6b5.ssl.cf2.rackcdn.com/706875/2/check/neutron-tempest-dvr-ha-multinode-full/6d77e6d/testr_results.html15:47
slaweqso I will report bug for nova about that15:47
slaweqand that's all from my side regarding scenario jobs15:47
slaweqanything else You want to add?15:47
ralonsohno thanks15:47
njohnstonnope15:47
slaweqok, thx15:48
slaweqso (almost) last thing for today15:48
slaweq#topic Periodic15:48
*** openstack changes topic to "Periodic (Meeting topic: neutron_ci)"15:48
slaweqneutron-ovn-tempest-ovs-master-fedora is failing everyday15:48
slaweqwe should check this job15:48
slaweqit's failing on deploying devstack15:49
ralonsohdo you have the link for the periodic jobs?15:49
slaweqhttps://9fc08b4308a330f341b7-ee326d1edc43244c4c522686856ef03c.ssl.cf2.rackcdn.com/periodic/opendev.org/openstack/neutron/master/neutron-ovn-tempest-ovs-master-fedora/07d0c6f/job-output.txt15:49
slaweqerror is like:15:49
slaweq2020-02-11 07:05:25.363036 | controller | + lib/infra:install_infra:32               :   virtualenv -p python3 /opt/stack/requirements/.venv15:49
slaweq2020-02-11 07:05:25.441404 | controller | ERROR:root:ImportError: cannot import name ensure_text15:49
ralonsohI can check it15:50
slaweqI think it's some issue related to fedora 2915:50
slaweqralonsoh: thx15:50
slaweq#action ralonsoh to check periodic neutron-ovn-tempest-ovs-master-fedora job's failures15:50
slaweqother periodic jobs looks fine15:51
slaweqok, and I have one more topic for tody15:51
slaweq*today15:51
slaweq#topic Open discussion15:51
*** openstack changes topic to "Open discussion (Meeting topic: neutron_ci)"15:51
slaweqsome time ago I asked on ML about future of lib/neutron and lib/neutron-legacy in devstack15:52
slaweqI think haleyb said that there shouldn't be much additional work needed to finally make lib/neutron usable15:52
slaweqso I would like to ask if there is any volunteer to work on that or maybe we should simply remove lib/neutron and rename lib/neutron-legacy to not be "legacy" anymore?15:53
slaweqas now things can be confusing for users :)15:53
slaweqany thoughts?15:53
ralonsohI'm ok with this but not a priority now15:54
ralonsohI can take it but not now15:54
slaweqralonsoh: sure, it's not top priority but IMO we should at least decide what to do with it finally15:54
slaweqas now we are "in the middle" of 2 solutions since few years15:55
ralonsohI'm ok with finishing the migration15:55
slaweqgreat15:55
slaweqso lets keep this in mind and maybe move it slowly forward15:56
slaweqok, that's all from my side for today15:56
slaweqanything else You want to talk today?15:56
ralonsohno15:56
* njohnston just finished the other meeting, is catching up15:57
njohnstonnothing from me15:57
slaweqok, thx for attending15:57
slaweqsee You online15:57
slaweqo/15:57
slaweq#endmeeting15:57
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"15:57
openstackMeeting ended Wed Feb 12 15:57:27 2020 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)15:57
openstackMinutes:        http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-02-12-15.00.html15:57
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-02-12-15.00.txt15:57
openstackLog:            http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-02-12-15.00.log.html15:57
ralonsohbye15:57
njohnstono/15:58
*** jamesmcarthur_ has joined #openstack-meeting-316:03
*** jamesmcarthur has quit IRC16:06
*** macz has joined #openstack-meeting-316:43
*** macz has quit IRC16:43
*** macz has joined #openstack-meeting-316:44
*** macz has quit IRC16:50
*** igordc has joined #openstack-meeting-317:17
*** jamesmcarthur_ has quit IRC17:47
*** igordc has quit IRC18:08
*** jamesmcarthur has joined #openstack-meeting-318:18
*** igordc has joined #openstack-meeting-318:23
*** e0ne has quit IRC18:26
*** jamesmcarthur has quit IRC18:31
*** jamesmcarthur has joined #openstack-meeting-318:31
*** jamesmcarthur has quit IRC18:42
*** jamesmcarthur has joined #openstack-meeting-318:43
*** jamesmcarthur has quit IRC18:48
*** jamesmcarthur has joined #openstack-meeting-318:54
*** jamesmcarthur has quit IRC18:57
*** jamesmcarthur has joined #openstack-meeting-318:58
*** yamamoto has quit IRC19:03
*** ralonsoh has quit IRC19:04
*** negronjl has quit IRC19:08
*** jamesmcarthur has quit IRC19:08
*** jamesmcarthur has joined #openstack-meeting-319:09
*** negronjl has joined #openstack-meeting-319:13
*** jamesmcarthur has quit IRC19:13
*** jamesmcarthur has joined #openstack-meeting-319:30
*** jamesmcarthur_ has joined #openstack-meeting-319:31
*** jamesmcarthur has quit IRC19:35
*** yamamoto has joined #openstack-meeting-319:39
*** e0ne has joined #openstack-meeting-319:40
*** e0ne has quit IRC19:51
*** yamamoto has quit IRC19:52
*** jamesmcarthur_ has quit IRC19:54
*** igordc has quit IRC19:58
*** raildo_ has joined #openstack-meeting-322:05
*** raildo has quit IRC22:08
*** jamesmcarthur has joined #openstack-meeting-322:20
*** jamesmcarthur has quit IRC22:24
*** jamesmcarthur has joined #openstack-meeting-322:25
*** slaweq has quit IRC22:26
*** jamesmcarthur has quit IRC23:02
*** Adri2000 has quit IRC23:19
*** yamamoto has joined #openstack-meeting-323:49
*** yamamoto has quit IRC23:54
*** bobmel has joined #openstack-meeting-323:55
*** bobmel has quit IRC23:59

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!