Wednesday, 2020-08-05

*** jamesmcarthur has quit IRC00:09
*** jamesmcarthur has joined #openstack-meeting-300:11
*** jamesmcarthur has quit IRC00:12
*** jamesmcarthur has joined #openstack-meeting-300:12
*** jamesmcarthur has quit IRC00:13
*** _erlon_ has quit IRC00:14
*** jamesmcarthur has joined #openstack-meeting-300:18
*** jamesmcarthur has quit IRC00:20
*** jamesmcarthur has joined #openstack-meeting-300:30
*** jamesmcarthur has quit IRC00:32
*** jamesmcarthur has joined #openstack-meeting-300:37
*** jamesmcarthur has quit IRC00:38
*** jamesmcarthur has joined #openstack-meeting-300:44
*** jamesmcarthur has quit IRC01:59
*** apetrich has quit IRC02:14
*** aarents has quit IRC02:22
*** aarents has joined #openstack-meeting-302:23
*** psahoo has joined #openstack-meeting-302:39
*** markvoelker has joined #openstack-meeting-303:10
*** markvoelker has quit IRC03:15
*** psachin has joined #openstack-meeting-303:33
*** jamesmcarthur has joined #openstack-meeting-303:37
*** jamesmcarthur has quit IRC03:42
*** markvoelker has joined #openstack-meeting-303:46
*** markvoelker has quit IRC03:51
*** markvoelker has joined #openstack-meeting-303:53
*** yamamoto has quit IRC04:02
*** markvoelker has quit IRC04:05
*** yamamoto has joined #openstack-meeting-304:19
*** jamesmcarthur has joined #openstack-meeting-304:28
*** jamesmcarthur has quit IRC04:55
*** jamesmcarthur has joined #openstack-meeting-305:15
*** psahoo has quit IRC05:23
*** psahoo has joined #openstack-meeting-305:50
*** e0ne has joined #openstack-meeting-306:47
*** jamesmcarthur has quit IRC06:58
*** jamesmcarthur has joined #openstack-meeting-306:59
*** jamesmcarthur has quit IRC07:03
*** apetrich has joined #openstack-meeting-307:06
*** slaweq has joined #openstack-meeting-307:07
*** psahoo has quit IRC07:21
*** psahoo has joined #openstack-meeting-307:34
*** tosky has joined #openstack-meeting-307:38
*** psahoo has quit IRC07:52
*** e0ne_ has joined #openstack-meeting-307:54
*** e0ne has quit IRC07:54
*** psahoo has joined #openstack-meeting-308:08
*** jamesmcarthur has joined #openstack-meeting-308:17
*** tosky has quit IRC08:26
*** jamesmcarthur has quit IRC08:27
*** tosky has joined #openstack-meeting-308:27
*** ralonsoh has joined #openstack-meeting-308:30
*** e0ne_ has quit IRC09:42
*** e0ne has joined #openstack-meeting-309:43
*** e0ne has quit IRC09:46
*** e0ne has joined #openstack-meeting-309:46
*** ralonsoh has quit IRC09:50
*** e0ne has quit IRC09:53
*** e0ne has joined #openstack-meeting-309:54
*** e0ne has quit IRC10:04
*** e0ne has joined #openstack-meeting-310:04
*** e0ne_ has joined #openstack-meeting-310:06
*** e0ne has quit IRC10:09
*** e0ne has joined #openstack-meeting-310:16
*** e0ne_ has quit IRC10:16
*** jamesmcarthur has joined #openstack-meeting-310:24
*** jamesmcarthur has quit IRC10:31
*** yamamoto has quit IRC10:56
*** yamamoto has joined #openstack-meeting-311:01
*** ralonsoh has joined #openstack-meeting-311:08
*** markvoelker has joined #openstack-meeting-311:16
*** raildo has joined #openstack-meeting-311:21
*** markvoelker has quit IRC11:25
*** psahoo has quit IRC11:27
*** psahoo has joined #openstack-meeting-311:42
*** yamamoto has quit IRC11:58
*** psahoo has quit IRC12:22
*** tsmith_ has joined #openstack-meeting-312:24
*** yamamoto has joined #openstack-meeting-312:26
*** tsmith2 has quit IRC12:27
*** tsmith_ is now known as tsmith212:27
*** psahoo has joined #openstack-meeting-312:33
*** yamamoto has quit IRC12:36
*** Adri2000 has quit IRC13:04
*** yamamoto has joined #openstack-meeting-313:07
*** Adri2000 has joined #openstack-meeting-313:11
*** yamamoto has quit IRC13:15
*** jamesmcarthur has joined #openstack-meeting-313:16
*** raildo has quit IRC13:35
*** raildo has joined #openstack-meeting-313:38
*** raildo has quit IRC13:43
*** raildo has joined #openstack-meeting-313:43
*** raildo has quit IRC13:44
*** raildo has joined #openstack-meeting-313:44
*** raildo_ has joined #openstack-meeting-313:46
*** raildo_ has quit IRC13:50
*** rambo_li has joined #openstack-meeting-313:54
*** jamesmcarthur has quit IRC14:18
*** mlavalle has joined #openstack-meeting-314:37
*** psachin has quit IRC14:57
slaweq#startmeeting neutron_ci15:00
openstackMeeting started Wed Aug  5 15:00:19 2020 UTC and is due to finish in 60 minutes.  The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot.15:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:00
*** openstack changes topic to " (Meeting topic: neutron_ci)"15:00
openstackThe meeting name has been set to 'neutron_ci'15:00
ralonsohhi15:00
ralonsoh(lot of fun last weeks in the CI)15:00
slaweqhi15:01
slaweqralonsoh: yes :/15:01
bcafarelo/15:01
slaweqok, lets start, maybe others will join15:02
slaweqGrafana dashboard: http://grafana.openstack.org/dashboard/db/neutron-failure-rate15:02
slaweqplease open and lets go with the meeting :)15:02
slaweq#topic Actions from previous meetings15:03
*** openstack changes topic to "Actions from previous meetings (Meeting topic: neutron_ci)"15:03
slaweqwe have only one for today15:03
slaweqbcafarel to clean stable branches jobs from *-master jobs15:03
bcafarelhttps://review.opendev.org/#/c/743795/ is up for ussuri, I will push older ones when it looks good15:03
slaweqthx bcafarel15:04
slaweqI will review it today15:04
slaweqok, so lets move on15:04
slaweqnext topic15:04
slaweq#topic Switch to Ubuntu Focal15:04
*** openstack changes topic to "Switch to Ubuntu Focal (Meeting topic: neutron_ci)"15:04
slaweqEtherpad: https://etherpad.opendev.org/p/neutron-victoria-switch_to_focal15:05
*** maciejjozefczyk has joined #openstack-meeting-315:05
slaweqI made progress with functional and fullstack jobs15:05
maciejjozefczyk\o15:05
slaweqpatch  https://review.opendev.org/#/c/734304/: should be ok with https://review.opendev.org/#/c/744500/15:05
ralonsoh+115:05
slaweqbut we should be aware that linuxbridge driver can't work now with ebtables-nft15:06
slaweqwe have to use legacy implementation to make it working15:06
slaweqmaybe some additional docs update will be needed too15:06
ralonsoh(are we going to deprecate LB soon?)15:08
slaweqralonsoh: nope15:08
ralonsohok15:08
slaweqfor other, tempest related jobs I will try to check this week15:08
slaweqthat's all from me about Ubuntu 20.0415:09
slaweqdo You have anything else to add?15:09
ralonsohno15:09
ralonsohwell, the wsgi jobs15:09
ralonsohbut maybe in following sections15:10
slaweqralonsoh: I added Your topic to the functional/fullstack section15:10
slaweq:)15:10
ralonsohthanks15:10
bcafarelother tempest jobs looked OK in https://review.opendev.org/#/c/738163/ so we should be OK there15:10
bcafarel(at least the voting ones)15:10
slaweqbcafarel: that's good15:11
slaweqthx15:11
slaweqso now it seems we need to ask gmann about his patch https://review.opendev.org/#/c/734700/15:11
slaweqas our depends on it15:11
bcafarelit will probably get in when more projects look ready to go15:12
bcafarelhopefully neutron will be in that list soon!15:13
slaweqahh, ok15:13
slaweqok, lets move on to the next topic then15:14
slaweq#topic Stadium projects15:14
*** openstack changes topic to "Stadium projects (Meeting topic: neutron_ci)"15:14
slaweqhere we still have this one goal:15:14
slaweqstandardize on zuul v315:14
slaweqEtherpad: https://etherpad.openstack.org/p/neutron-train-zuulv3-py27drop15:14
slaweqbut I didn't had time to look into those old jobs yet15:14
slaweqif anyone has any cycles please take a look into those too :)15:15
slaweqand that's all from my side about stadium projects15:15
slaweqanything else You want to add?15:15
ralonsohno15:15
bcafarelno15:16
slaweqok, so next topic15:16
slaweq#topic Stable branches15:16
*** openstack changes topic to "Stable branches (Meeting topic: neutron_ci)"15:16
slaweqUssuri dashboard: http://grafana.openstack.org/d/pM54U-Kiz/neutron-failure-rate-previous-stable-release?orgId=115:16
slaweqTrain dashboard: http://grafana.openstack.org/d/dCFVU-Kik/neutron-failure-rate-older-stable-release?orgId=115:16
slaweqI don't know about any new issues there, except the one with pip which was fixed yesterday I believe15:17
slaweqbcafarel: do You have any new issues from stable branches ci?15:18
bcafarelyes I sent a few rechecks this morning, so far it seems we are back in order (though most are still in queue)15:18
slaweqok, good that ci for stable branches works fine15:19
bcafarel+115:19
slaweqok15:19
slaweqso next topic15:19
slaweq#topic Grafana15:19
*** openstack changes topic to "Grafana (Meeting topic: neutron_ci)"15:19
slaweqhttp://grafana.openstack.org/dashboard/db/neutron-failure-rate15:20
*** rambo_li has quit IRC15:20
slaweqafter fix of the lower-constraints job and grenade jobs (thx ralonsoh) it seems for me that it's pretty ok15:21
ralonsohI think so15:21
gmannslaweq: ack, in internal meeting, will look after that15:22
slaweqthe only thing which worries me is neutron-ovn-tempest-full-multinode-ovs-master  failing 100% of time since few weeks15:22
slaweqgmann: np, we just talked about Ubuntu 20.0415:22
ralonsohslaweq, I think this is just a timeout problem15:23
slaweqralonsoh: it is timeout mostly15:23
slaweqbut it's I think related to many test failures15:23
slaweqand due to that finally we are hitting job's timeout15:23
ralonsohslaweq, I'll try to take a look this week15:24
slaweqsee for example https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_6a5/738163/12/check/neutron-ovn-tempest-full-multinode-ovs-master/6a5ca86/job-output.txt15:24
slaweqmany tests failing to ssh to the instance15:24
slaweqso that may be some problem with job's configuration IMO15:25
slaweqralonsoh: thx a lot15:25
slaweq#action ralonsoh to check timing out neutron-ovn-tempest-full-multinode-ovs-master jobs15:25
slaweqdo You have anything else regarding grafana?15:26
ralonsohno15:26
slaweqok, so lets move on15:27
slaweq#topic fullstack/functional15:27
*** openstack changes topic to "fullstack/functional (Meeting topic: neutron_ci)"15:27
slaweqso ralonsoh have an idea to make "uwsgi" jobs default (neutron-fullstack-with-uwsgi, neutron-functional-with-uwsgi) and remove the non-uwsgi ones (to save CI time)15:27
ralonsohexactly15:27
ralonsohto save time in the CI15:27
slaweqand I basically agree with that15:27
slaweqwe even discussed some time ago to switch uwsgi to be default in devstack15:28
slaweqbut there is some problem with grenade jobs and I didn't had time to investigate it then15:28
slaweqso it's still pending15:28
slaweqbut the other thing is that even if we will switch uwsgi to be default, we will need IMO keep e.g. functional job without uwsgi as this deployment method isn't deprecated or unsupported so far15:29
slaweqso I think we need to have jobs for both15:29
ralonsohok, let's wait then15:29
slaweqand also so far -uwsgi jobs are still not gating, only voting in check queue15:30
*** tosky has quit IRC15:31
slaweqmaybe, if we want to move forward with uwsgi we can move non-uwsgi functional/fullstack jobs to the periodic queue and promote -uwsgi ones to be gating15:31
slaweqwdyt?15:31
ralonsohthat will save ci time15:31
ralonsohok for me15:31
*** jamesmcarthur has joined #openstack-meeting-315:31
bcafarelthat sounds good, I suppose the -uwsgi jobs are stable?15:31
ralonsohpretty much15:32
slaweqbcafarel: as stable as non-uwsgi ones15:32
ralonsohsame as non-uwsgi15:32
bcafarelack, sounds good to me then!15:32
slaweqso I will propose such patch then15:32
slaweq#action slaweq to move uwsgi jobs to periodic queue and promote -uwsgi ones to be gating15:32
slaweqand that's all regarding functional/fullstack jobs for today from me15:33
slaweqlets move on15:33
slaweq#topic Tempest/Scenario15:33
*** openstack changes topic to "Tempest/Scenario (Meeting topic: neutron_ci)"15:33
slaweqhere I found one new (for me) issue in neutron-ovn-tempest-slow15:33
slaweqtest tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_update_router_admin_state is failing very often15:34
slaweqlike e.g. https://2e5a48e004c25d40a540-8aab0ff599dd9fcfda722d0dee6871dd.ssl.cf1.rackcdn.com/743832/1/check/neutron-ovn-tempest-slow/268b7cf/testr_results.html15:34
slaweqI reported bug https://bugs.launchpad.net/neutron/+bug/189044515:34
openstackLaunchpad bug 1890445 in neutron "[ovn] Tempest test test_update_router_admin_state failing very often" [Critical,Confirmed]15:34
slaweqmaciejjozefczyk: ralonsoh are You aware of such issue maybe?15:34
ralonsohno15:34
maciejjozefczykhmm15:35
maciejjozefczykperhaps the same as https://bugs.launchpad.net/neutron/+bug/188589815:35
openstackLaunchpad bug 1885898 in neutron "test connectivity through 2 routers fails in neutron-ovn-tempest-full-multinode-ovs-master job" [High,Confirmed] - Assigned to Maciej Jozefczyk (maciej.jozefczyk)15:35
maciejjozefczykI noticed that we have a race condition in updating router ports15:36
maciejjozefczykI was trying to fix/debug it here: https://review.opendev.org/#/c/740491/2015:36
maciejjozefczykfor now without any luck :(15:36
slaweqmaciejjozefczyk: can You check if this is the same issue and maybe mark 1890445 as duplicate of 1885898?15:37
maciejjozefczykslaweq, yes sure15:37
slaweqand maybe also blacklist this one test until this issue will be fixed, to make this job green15:38
maciejjozefczykok, please assign it to me15:38
slaweqthx maciejjozefczyk15:38
slaweq#action maciejjozefczyk to check https://bugs.launchpad.net/neutron/+bug/189044515:38
openstackLaunchpad bug 1890445 in neutron "[ovn] Tempest test test_update_router_admin_state failing very often" [Critical,Confirmed]15:38
maciejjozefczykkk15:38
slaweqand last one thing for today15:40
bcafarel^ laptop issue our PTL will hulk smash it into reboot and come back15:43
ralonsohhahahaha15:43
ralonsohso much suspense... what is this last thing?15:44
*** slaweq_ has joined #openstack-meeting-315:45
slaweq_I'm back15:45
*** slaweq has quit IRC15:45
slaweq_sorry15:45
*** slaweq_ is now known as slaweq15:45
ralonsohnp15:45
slaweqso one last thing for today is periodic job based on fedora15:45
slaweqwhich is again failing every day15:45
maciejjozefczykwoot15:45
bcafarel"periodic failure job" :(15:46
maciejjozefczykI just fixed that one :(15:46
slaweqit's failing since 30.0715:46
slaweq15 tests fails every day15:47
slaweqlike e.g. https://zuul.openstack.org/build/64b62afdb4ce4f2fa8b1b2d97bb36c3915:47
slaweqI will open LP for that one after the meeting15:47
maciejjozefczykok, thanks slaweq15:48
ralonsohsimilar problem to the ovn tempest multinode15:48
ralonsohssh access timeout15:48
maciejjozefczyksomethign bad recently happened in ovn I think15:48
slaweqralonsoh: yes, but this is singlenode job AFAIR15:48
slaweqin neutron logs I see errors like:15:51
slaweqkeystoneauth1.exceptions.auth_plugins.MissingAuthPlugin: An auth plugin is required to determine endpoint URL15:51
ralonsohthis is because of the placement15:52
slaweqbut I don't know if that may be related15:52
ralonsohbut this is not important15:52
slaweqok15:52
ralonsohif we can't update the placement, that's ok15:52
slaweqso nothing else obviously wrong15:52
ralonsohno...15:52
slaweqanyway, I will open bug and we can check later15:53
slaweqthat's not top priority as it's "only" fedora based periodic job15:53
slaweqok, that's all from me for today15:53
slaweqdo You have anything else You want to discuss today?15:53
ralonsohno15:53
bcafarelall good from me15:54
slaweqok, so thx for attending and see You online15:54
slaweq#endmeeting15:54
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"15:54
openstackMeeting ended Wed Aug  5 15:54:41 2020 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)15:54
openstackMinutes:        http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-08-05-15.00.html15:54
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-08-05-15.00.txt15:54
openstackLog:            http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-08-05-15.00.log.html15:54
ralonsohbye15:55
*** e0ne has quit IRC15:56
*** maciejjozefczyk has quit IRC15:59
*** psahoo has quit IRC16:00
*** markvoelker has joined #openstack-meeting-316:27
*** tosky has joined #openstack-meeting-316:29
*** maciejjozefczyk has joined #openstack-meeting-316:42
*** e0ne has joined #openstack-meeting-317:09
*** yamamoto has joined #openstack-meeting-317:14
*** yamamoto has quit IRC17:18
*** raildo has quit IRC18:03
*** raildo has joined #openstack-meeting-318:08
*** e0ne has quit IRC18:10
*** raildo_ has joined #openstack-meeting-318:11
*** ralonsoh has quit IRC18:12
*** raildo_ has quit IRC18:13
*** e0ne has joined #openstack-meeting-318:26
*** raildo has quit IRC18:46
*** raildo has joined #openstack-meeting-318:48
*** e0ne has quit IRC18:51
*** tosky has quit IRC19:39
*** jamesmcarthur has quit IRC19:49
*** jamesmcarthur has joined #openstack-meeting-319:49
*** jamesmcarthur has quit IRC19:54
*** jamesmcarthur has joined #openstack-meeting-320:21
*** raildo has quit IRC20:40
*** raildo has joined #openstack-meeting-320:40
*** raildo has quit IRC20:57
*** maciejjozefczyk has quit IRC21:00
*** slaweq has quit IRC21:18
*** tosky has joined #openstack-meeting-321:25
*** yamamoto has joined #openstack-meeting-321:45
*** markmcclain has quit IRC21:53
*** yamamoto has quit IRC22:11
*** yangyapeng has quit IRC22:30
*** yangyapeng has joined #openstack-meeting-322:34
*** artom has quit IRC22:45
*** artom has joined #openstack-meeting-322:46
*** jamesmcarthur has quit IRC22:59
*** mlavalle has quit IRC23:09
*** markmcclain has joined #openstack-meeting-323:12
*** tosky has quit IRC23:14
*** irclogbot_1 has quit IRC23:34
*** irclogbot_3 has joined #openstack-meeting-323:37

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!