Wednesday, 2020-06-17

*** artom has quit IRC00:08
*** yamamoto has quit IRC00:35
*** yamamoto has joined #openstack-meeting-300:35
*** cheng1 has quit IRC01:14
*** cheng1 has joined #openstack-meeting-301:17
*** ricolin has joined #openstack-meeting-302:01
*** apetrich has quit IRC02:42
*** psachin has joined #openstack-meeting-302:48
*** psachin has quit IRC03:31
*** psachin has joined #openstack-meeting-303:33
*** yamamoto has quit IRC04:19
*** markvoelker has joined #openstack-meeting-304:27
*** markvoelker has quit IRC04:33
*** yamamoto has joined #openstack-meeting-304:35
*** diablo_rojo has quit IRC04:53
*** slaweq has joined #openstack-meeting-307:00
*** apetrich has joined #openstack-meeting-307:11
*** psachin has quit IRC07:15
*** psachin has joined #openstack-meeting-307:17
*** ralonsoh has joined #openstack-meeting-307:57
*** e0ne has joined #openstack-meeting-308:15
*** cheng1 has quit IRC08:34
*** cheng1 has joined #openstack-meeting-308:36
*** cheng1 has quit IRC08:44
*** cheng1 has joined #openstack-meeting-308:50
*** links has joined #openstack-meeting-308:55
*** e0ne has quit IRC09:02
*** e0ne has joined #openstack-meeting-309:08
*** apetrich has quit IRC09:17
*** cheng1 has quit IRC09:24
*** apetrich has joined #openstack-meeting-309:26
*** cheng1 has joined #openstack-meeting-309:26
*** e0ne has quit IRC09:47
*** e0ne_ has joined #openstack-meeting-309:47
*** e0ne_ has quit IRC09:51
*** e0ne has joined #openstack-meeting-309:52
*** yamamoto has quit IRC10:01
*** yamamoto has joined #openstack-meeting-310:02
*** yamamoto has quit IRC10:02
*** psachin has quit IRC10:31
*** yamamoto has joined #openstack-meeting-310:38
*** yamamoto has quit IRC10:39
*** yamamoto has joined #openstack-meeting-310:40
*** e0ne has quit IRC10:42
*** e0ne has joined #openstack-meeting-310:43
*** e0ne has quit IRC10:44
*** e0ne has joined #openstack-meeting-310:45
*** e0ne has quit IRC10:46
*** e0ne has joined #openstack-meeting-310:46
*** yamamoto has quit IRC10:50
*** yamamoto has joined #openstack-meeting-311:00
*** purplerbot has joined #openstack-meeting-311:00
*** psachin has joined #openstack-meeting-311:03
*** cheng1 has quit IRC11:04
*** cheng1 has joined #openstack-meeting-311:05
*** artom has joined #openstack-meeting-311:20
*** cheng1 has quit IRC11:23
*** cheng1 has joined #openstack-meeting-311:29
*** raildo has joined #openstack-meeting-311:44
*** psachin has quit IRC11:58
*** psachin has joined #openstack-meeting-311:59
*** lkoranda has joined #openstack-meeting-312:19
*** cheng1 has quit IRC12:20
*** lkoranda has quit IRC12:21
*** lkoranda has joined #openstack-meeting-312:21
*** lkoranda has quit IRC12:21
*** cheng1 has joined #openstack-meeting-312:22
*** slaweq_ has joined #openstack-meeting-312:47
*** slaweq has quit IRC12:48
*** cheng1 has quit IRC13:01
*** slaweq_ is now known as slaweq13:03
*** cheng1 has joined #openstack-meeting-313:06
*** cheng1 has quit IRC13:22
*** e0ne has quit IRC13:37
*** e0ne has joined #openstack-meeting-313:37
*** ricolin has quit IRC13:40
*** e0ne has quit IRC13:48
*** e0ne_ has joined #openstack-meeting-313:48
*** lpetrut has joined #openstack-meeting-313:56
*** yamamoto has quit IRC14:06
*** liuyulong has joined #openstack-meeting-314:07
*** yamamoto has joined #openstack-meeting-314:34
*** yamamoto has quit IRC14:34
*** yamamoto has joined #openstack-meeting-314:34
*** psachin has quit IRC14:34
*** links has quit IRC14:37
*** yamamoto has quit IRC14:41
*** lajoskatona has joined #openstack-meeting-315:01
slaweq#startmeeting neutron_ci15:01
openstackMeeting started Wed Jun 17 15:01:22 2020 UTC and is due to finish in 60 minutes.  The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot.15:01
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:01
slaweqhi15:01
*** openstack changes topic to " (Meeting topic: neutron_ci)"15:01
openstackThe meeting name has been set to 'neutron_ci'15:01
lajoskatonao/15:01
ralonsohhi15:01
bcafarelyeah o/15:01
slaweqok, let's start15:02
slaweqGrafana dashboard: http://grafana.openstack.org/dashboard/db/neutron-failure-rate15:03
slaweqplease open link and we can move on15:03
slaweq#topic Actions from previous meetings15:03
*** openstack changes topic to "Actions from previous meetings (Meeting topic: neutron_ci)"15:03
njohnston0o/15:03
*** maciejjozefczyk has joined #openstack-meeting-315:03
maciejjozefczyk\o15:03
slaweqfirst action from last meeting15:03
slaweqslaweq to check failure in test_ha_router_failover: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_6d0/726168/2/check/neutron-functional/6d0b174/testr_results.html15:04
slaweqI didn't saw this issue any time later so I didn't check it. I will if it will start failing more often15:04
slaweqor maybe You saw it more often and it's just me who missed that15:04
slaweqso please tell me :)15:04
slaweq64159015:04
slaweq^^ sorry :)15:04
bcafarelnice score, I will not try to compete :)15:05
slaweq:)15:06
slaweqok, so next one15:07
slaweqslaweq to add additional logging for fullstack's firewall tests15:07
slaweq    For now I just did DNM patch https://review.opendev.org/#/c/736034/1 to see exactly when this test is failing and then I will add some additional logs to know more what's going on there.15:07
slaweqso I will assign it to myself for next week too15:07
slaweq#action slaweq to add additional logging for fullstack's firewall tests15:07
slaweqand the last one15:07
slaweqmaciejjozefczyk to check failing test_port_security_macspoofing_port test15:07
* maciejjozefczyk looking for the link :)15:09
maciejjozefczyk#link https://bugs.launchpad.net/tempest/+bug/172888615:10
openstackLaunchpad bug 1728886 in tripleo "test_port_security_macspoofing_port doesn't work when l2pop and arpresponding are enabled" [Critical,Fix released]15:10
slaweqmaciejjozefczyk: so it's fixed and test is passing now, right?15:11
maciejjozefczykslaweq, the problem was in core-ovn and it was fixed there15:12
maciejjozefczykin some stable releases I needed to blacklist it, because those are not using the newest ovn15:12
slaweqok15:12
slaweqthx maciejjozefczyk15:12
slaweqok, lets move on15:13
slaweq#topic Stadium projects15:13
*** openstack changes topic to "Stadium projects (Meeting topic: neutron_ci)"15:13
slaweqstandardize on zuul v315:14
slaweqEtherpad: https://etherpad.openstack.org/p/neutron-train-zuulv3-py27drop15:14
slaweqthere are only 2 missing things there15:14
slaweqnetworking-midonet15:14
slaweqand ovn grenade job15:14
slaweqfor ovn grenade job I have it in my backlog15:14
slaweqand for networking-midonet, I asked project core reviewers this week, and they sadly agreed that we should probably deprecate this project as we did with fwaas15:15
slaweqso IMO we can cross it out from this list here15:15
njohnston+115:15
bcafareltime has come then15:16
ralonsoh+115:16
slaweqok15:16
bcafarelfor ovn grenade job is it still working? neutron grenade has issues in stable branches (where it is still legacy atm)15:16
bcafarelwith virtualenv/pip removal from base image (and also our friend uwsgi)15:16
slaweqbcafarel: currently we are not running this ovn grenade job so I don't know if it's really working or not15:17
bcafarel:)15:17
slaweqI will need to make it working and migrate to zuulv315:17
slaweqand we have new topic which also needs to be done in stadium projects15:18
slaweq"Switch to Ubuntu Focal"15:18
slaweqis there any volunteer to drive this?15:19
njohnstonI started it but I think with my other focuses someone else might be a good candidate to push forward15:19
njohnstonwith the zuul v3 job almost done I think our job inheritance will do almost all of this15:20
njohnstonas the base jobs are modified15:20
njohnstonbut the process of debugging any changes is unknown15:20
njohnstons/changes/issues/15:20
slaweqnjohnston: yes, I think it shouldn't be a lot of work really15:21
bcafarelI can help it will be a change from fixing xenial jobs :)15:21
slaweqbcafarel: :D15:21
slaweqthx a lot15:21
njohnstonlooks like the change I did for fullstack/functional is breaking on the step of compiling OVN15:22
njohnstonhttps://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_12b/734304/3/check/neutron-functional/12b22c0/job-output.txt15:22
njohnstonthat change is https://review.opendev.org/#/c/734304/15:22
maciejjozefczyknjohnston, with moving to Focal we can stop compiling the modules15:22
njohnstonmaciejjozefczyk: Perfect, that was my hope15:23
maciejjozefczyknjohnston, cause the kernel in Focal has OVS metric actions that we need :)15:23
slaweqmaciejjozefczyk: unless we will have another feature which we will need there :P15:23
maciejjozefczykmeter*15:23
maciejjozefczykslaweq, unless :D15:23
slaweqbut for now it should be good way to go probably15:23
bcafarelone step at a time yep15:23
njohnstonSo probably the first step is to disable the module compilation if focal is detected?15:24
maciejjozefczykwait, the error in njohnston link is about "ipv6_dst_lookup"15:24
maciejjozefczykso15:24
maciejjozefczyk#link https://review.opendev.org/#/c/735799/15:24
maciejjozefczykshould fix it15:24
slaweqmaciejjozefczyk: I see other error in https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_12b/734304/3/check/neutron-functional/12b22c0/job-output.txt15:24
slaweqRuntimeError: dictionary keys changed during iteration15:24
bcafarelRuntimeError: dictionary keys changed during iteration15:24
slaweqfirst! \o/ :P15:25
njohnstonright, that is in the compile_ovs action15:25
njohnstonhttp://paste.openstack.org/show/794904/15:25
bcafarelslaweq: that's why you are PTL :)15:25
njohnstonthe dictionary keys error, I mean15:25
maciejjozefczykyhmmm, allright15:25
slaweqbcafarel: I hope it's not the only reason ;P15:25
ralonsohand njohnston's question about disabling module compilation?15:26
ralonsohis there a patch for this?15:26
bcafarelnot yet I think15:26
njohnstonno, I think that would be a new action item15:26
ralonsohok, I'll propose one15:26
maciejjozefczykwe need to start discussion again if we want to comiple ovs/ovn at all, that seems to be outside module (error is related to ovsdb)15:26
bcafarelfirst focus was on https://review.opendev.org/#/c/735799/15:26
ralonsohwe can also add a knob for this, apart from checking the version15:26
njohnston+115:27
ralonsoh(ok we have OVN_BUILD_MODULES)15:28
slaweqnjohnston: I think You can simply add new variable to https://github.com/openstack/neutron/blob/master/roles/configure_functional_tests/tasks/main.yaml#L1315:28
slaweqand then set it in the job's definition to not compile ovs from source15:29
slaweqthe variable in script is here https://github.com/openstack/neutron/blob/master/tools/configure_for_func_testing.sh#L6515:29
lajoskatona+1 if possible get rid of depenency to out of tree error sources15:29
maciejjozefczykslaweq, I can work on that part... cause I started doing something similar in: https://review.opendev.org/#/c/727193/15:29
njohnstonslaweq: sounds good, I will iterate on that after the meeting15:29
* njohnston yields to maciejjozefczyk15:30
slaweqthx maciejjozefczyk and njohnston15:30
maciejjozefczykactually we need to make a cleanup around compile_ovs functions, because we have this one both in devstack ovn lib and devstack ovs lib ...15:30
slaweqmaciejjozefczyk: yes, so we need cleanup15:31
slaweqwill You take care of this?15:31
maciejjozefczykyes15:31
slaweqthx a lot15:32
slaweqok, and bcafarel will You prepare some list of the other jobs which we need to switch to Focal?15:32
bcafarelslaweq: that's the plan yep!15:33
slaweqthx a lot15:33
slaweqso we should be good with that topic :)15:33
slaweqso next one15:33
slaweq#topic Stable branches15:33
*** openstack changes topic to "Stable branches (Meeting topic: neutron_ci)"15:33
slaweqUssuri dashboard: http://grafana.openstack.org/d/pM54U-Kiz/neutron-failure-rate-previous-stable-release?orgId=115:33
slaweqTrain dashboard: http://grafana.openstack.org/d/dCFVU-Kik/neutron-failure-rate-older-stable-release?orgId=115:33
slaweqwe are still broken with this uwsgi issue in stable branches, right?15:33
* bcafarel looks for links15:34
bcafareluwsgi should be good in recent branches now though there are other issues15:34
slaweqouch15:34
bcafarel#link https://review.opendev.org/#/c/735523/15:35
bcafareluwsgi should be fixed up to stein15:35
slaweqgood :)15:36
maciejjozefczyk++15:36
bcafarelthen we need https://review.opendev.org/#/c/735948/1 (grenade job) + ovs_version fix to backport15:36
slaweqby ovs_version You mean the patch from jlibosva https://review.opendev.org/#/c/735799/ right?15:39
bcafarelsorry, laptop issue15:39
bcafarelslaweq: that's the one yes15:39
*** e0ne_ has quit IRC15:40
slaweqhmm, but will we be able to merge them? as one requires another15:40
slaweqso we are in deadlock in ussuri, aren't we?15:40
ralonsohI think so15:40
ralonsohwe need both15:40
bcafarelgood point, yes they will most probably need to be squashed15:40
slaweqso we can combine them too in ussuri, or (better IMO) propose patch to set all those jobs non-voting in ussuri15:41
slaweqthen merge both patches and revert first one15:41
slaweqwhat do You think?15:42
ralonsoh+1 if we only merge the ovs patch with those jobs stopped15:43
ralonsoh(well, and the zuul migration one)15:44
bcafarelsounds good, each fix patch can (and should) re-enable relevant jobs15:44
slaweqok, bcafarel will You propose patch to make those jobs non-voting temporary or do You want me to do that?15:44
bcafarelslaweq: I have the relevant tab already open I can send it (neutron-grenade-multinode* and ovn jobs)15:46
slaweqthx bcafarel15:46
*** e0ne has joined #openstack-meeting-315:46
slaweqI think that should be all related to the stable branches, right?15:46
slaweqor do You have anything else?15:46
bcafarelolder branches will need some fixes too, but one branch at a time!15:47
slaweqbcafarel: ok :)15:47
slaweqlets move on to the next topic as we are late today :)15:48
slaweq#topic Grafana15:48
*** openstack changes topic to "Grafana (Meeting topic: neutron_ci)"15:48
slaweqas we already discussed, there are 2 main issues which are causing many jobs failing: uwsgi problem and ovs compilation problem15:48
slawequwsgi should be fixed in master and fix for  ovs issue is in the gate already15:49
slaweqthere was also problem with designate job but it is also going down with failure rate todayu15:49
slaweqso seems like it's fixed already15:50
bcafarelyes frickler was commenting about it earlier in #openstack-neutron it should be good15:50
slaweqdo You have anything else regarding grafana and our dashboard?15:51
slaweqok, so lets move on15:52
slaweqnext topic15:52
slaweq#topic fullstack/functional15:52
*** openstack changes topic to "fullstack/functional (Meeting topic: neutron_ci)"15:52
slaweqI found only one new issue in fullstack job for today15:52
slaweq    Ovs compilation error: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_410/735799/1/check/neutron-fullstack/410977b/job-output.txt15:52
slaweqbut this one seems different than those which we discussed earlier already15:53
slaweqmaciejjozefczyk: can You take a look at it?15:53
slaweqit's not happening all the time for sure15:53
maciejjozefczykslaweq, yes15:53
slaweqahh, wait15:54
slaweqit was on https://review.opendev.org/#/c/735799/15:54
maciejjozefczykbcafarel, found the same in https://review.opendev.org/#/c/735799/15:54
maciejjozefczykyeah... I'll take a look tomorrow morning on it15:54
slaweqthx a lot15:54
bcafarelsigh so this is reproducible then15:55
maciejjozefczyk;/15:55
slaweqbcafarel: no, it's on the same patch15:55
slaweq:)15:55
bcafareloh true15:55
slaweqI found it also in same job run15:55
slaweqok, lets move on quickly to the last topic for today15:55
slaweq#topic Tempest/Scenario15:56
*** openstack changes topic to "Tempest/Scenario (Meeting topic: neutron_ci)"15:56
slaweqfirst of all, I sent today 2 patches:15:56
slaweq    https://review.opendev.org/73617715:56
slaweq    https://review.opendev.org/73618615:56
slaweqboth are follow-ups from the PTG discussion15:56
slaweqfirst one is move of not stable dvr multinode job to experimental queue15:56
maciejjozefczykthe problem should be solved by https://github.com/openvswitch/ovs/commit/0c4d144a989a444d038d58272d8571e97e00e86f15:56
slaweqand the second one is to remove tempest-integrated-networking job with multinode job15:57
maciejjozefczykso maybe we can use this tag instead the one proposed by Kuba15:57
slaweqmaciejjozefczyk++15:57
maciejjozefczyk(oh sorry, too late) :D15:57
maciejjozefczykslaweq, yeah, sorry for disturbing, I'll update the patch15:57
slaweqmaciejjozefczyk: no problem at all15:57
slaweqthx for looking into that15:57
ralonsoh+1 to those two patches15:58
slaweqok, and that's all what I have for this week15:58
slaweqthx ralonsoh15:58
slaweqanything else You want to raise here quickly in last minute?15:58
slaweqok, if not then thx for attending the meeting15:59
slaweqhave a great evening15:59
slaweqo/15:59
ralonsohbye15:59
slaweq#endmeeting15:59
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"15:59
openstackMeeting ended Wed Jun 17 15:59:46 2020 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)15:59
openstackMinutes:        http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-06-17-15.01.html15:59
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-06-17-15.01.txt15:59
openstackLog:            http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-06-17-15.01.log.html15:59
lajoskatonabye15:59
maciejjozefczyk\o15:59
*** lajoskatona has left #openstack-meeting-315:59
bcafarelo/16:00
njohnstono/16:00
*** e0ne has quit IRC16:01
*** liuyulong has quit IRC16:56
*** maciejjozefczyk has quit IRC16:56
*** maciejjozefczyk has joined #openstack-meeting-316:57
*** diablo_rojo has joined #openstack-meeting-316:59
*** lpetrut has quit IRC17:02
*** e0ne has joined #openstack-meeting-317:03
*** e0ne has quit IRC17:43
*** jamesmcarthur has joined #openstack-meeting-318:47
*** jamesmcarthur has quit IRC18:54
*** jamesmcarthur has joined #openstack-meeting-318:55
*** jamesmcarthur_ has joined #openstack-meeting-318:59
*** jamesmcarthur has quit IRC19:01
*** ralonsoh has quit IRC19:22
*** jamesmcarthur_ has quit IRC19:45
*** jamesmcarthur has joined #openstack-meeting-319:46
*** tsmith_ has joined #openstack-meeting-320:09
*** tobberydberg_ has joined #openstack-meeting-320:12
*** irclogbot_3 has quit IRC20:14
*** purplerbot has quit IRC20:18
*** tobberydberg has quit IRC20:18
*** amorin has quit IRC20:18
*** tsmith2 has quit IRC20:18
*** tsmith_ is now known as tsmith220:19
*** amorin has joined #openstack-meeting-320:20
*** irclogbot_0 has joined #openstack-meeting-320:21
*** purplerbot has joined #openstack-meeting-320:23
*** jamesmcarthur has quit IRC20:31
*** jamesmcarthur has joined #openstack-meeting-320:32
*** srwilkers has quit IRC20:35
*** jamesmcarthur has quit IRC20:37
*** csatari has quit IRC20:39
*** knikolla has quit IRC20:40
*** knikolla has joined #openstack-meeting-320:42
*** knikolla has quit IRC20:52
*** maciejjozefczyk has quit IRC21:00
*** e0ne has joined #openstack-meeting-321:03
*** jamesmcarthur has joined #openstack-meeting-321:04
*** jamesmcarthur has quit IRC21:13
*** srwilkers has joined #openstack-meeting-321:34
*** csatari has joined #openstack-meeting-321:38
*** knikolla has joined #openstack-meeting-321:40
*** slaweq has quit IRC21:41
*** raildo has quit IRC21:42
*** jamesmcarthur has joined #openstack-meeting-321:50
*** jamesmcarthur has quit IRC21:54
*** e0ne has quit IRC22:19

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!