Tuesday, 2021-03-02

*** macz_ has quit IRC00:06
*** gibi has joined #openstack-meeting-300:17
*** tosky has quit IRC00:17
*** mlavalle has quit IRC01:14
*** hemanth_n has joined #openstack-meeting-301:25
*** jkulik has quit IRC03:03
*** jkulik has joined #openstack-meeting-303:14
*** psachin has joined #openstack-meeting-303:47
*** psahoo has joined #openstack-meeting-304:44
*** ralonsoh has joined #openstack-meeting-306:55
*** yamamoto has quit IRC06:55
*** slaweq has joined #openstack-meeting-307:08
*** lpetrut has joined #openstack-meeting-307:27
*** eolivare has joined #openstack-meeting-307:44
*** belmoreira has joined #openstack-meeting-307:55
*** tosky has joined #openstack-meeting-308:35
*** psahoo_ has joined #openstack-meeting-309:05
*** psahoo has quit IRC09:09
*** e0ne has joined #openstack-meeting-310:02
*** e0ne has quit IRC10:02
*** psachin has quit IRC11:22
*** Luzi has joined #openstack-meeting-311:58
*** obondarev has joined #openstack-meeting-313:57
*** mlavalle has joined #openstack-meeting-313:59
*** lajoskatona has joined #openstack-meeting-314:00
slaweq#startmeeting networking14:00
openstackMeeting started Tue Mar  2 14:00:41 2021 UTC and is due to finish in 60 minutes.  The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot.14:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.14:00
slaweqhi14:00
*** openstack changes topic to " (Meeting topic: networking)"14:00
openstackThe meeting name has been set to 'networking'14:00
mlavalleo/14:00
ralonsohhi14:00
obondarevhi14:00
lajoskatonaHi14:00
rubasovo/14:00
bcafarelo/14:01
slaweq#topic announcements14:01
*** openstack changes topic to "announcements (Meeting topic: networking)"14:01
slaweqWallaby cycle calendar https://releases.openstack.org/wallaby/schedule.html14:01
slaweqwe are getting close to the end of the Wallaby cycle14:01
slaweqthis week we need to release final versions of neutron-lib, ovsdbapp and os-ken14:01
slaweqrelease patches are ready:14:01
slaweq     neutron-lib release patch https://review.opendev.org/c/openstack/releases/+/77794914:02
slaweq    ovsdbapp release patch https://review.opendev.org/c/openstack/releases/+/77795914:02
slaweq    os-ken release patch https://review.opendev.org/c/openstack/releases/+/77814414:02
slaweqI think we are good to go with them14:02
slaweqbut maybe You have something what You really want to include in the final release, so please tell me that ASAP14:02
slaweqok, next one14:03
slaweqPTG will be 19-23th April: http://lists.openstack.org/pipermail/openstack-discuss/2021-March/020778.html14:03
slaweqYou can register already14:03
slaweqit is free14:04
*** hemanth_n has quit IRC14:04
slaweqnext one14:05
slaweqPTG and TC election, nominations starts today: http://lists.openstack.org/pipermail/openstack-discuss/2021-March/020797.html14:05
slaweqso if You want to nominate your self, now is the time :)14:05
slaweqand last one reminder from my side14:06
slaweqWe have bunch of security bugs opened http://lists.openstack.org/pipermail/openstack-discuss/2021-February/020579.html - we need to revisit them, if You have some time, please take a look14:06
* mlavalle registered for PTG14:07
slaweqthat's all from me for today14:07
slaweqdo You have any announcements to the team today?14:08
slaweqok, so let14:09
slaweqlet's move on14:09
slaweq#topic Blueprints14:10
*** openstack changes topic to "Blueprints (Meeting topic: networking)"14:10
slaweqWallaby-3 https://bugs.launchpad.net/neutron/+milestone/wallaby-314:10
slaweqany updates?14:10
mlavallefor address groups, the RBAC patch needs one finel nudge: https://review.opendev.org/c/openstack/neutron/+/77246014:10
mlavalleralonsoh: in your last review you said it looked good. you only had a few nits, which have been addressed14:11
ralonsohI'll check it again14:11
mlavalleTHanks!14:11
mlavalleand this is the accompanying Tempest API tests: https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/77327414:12
mlavalleagain, it only needs one final +214:12
lajoskatonaI check it14:12
mlavalleThanks!14:13
slaweqthx mlavalle for the update and thx ralonsoh and lajoskatona for reviewing them :)14:14
slaweqregarding https://blueprints.launchpad.net/neutron/+spec/secure-rbac-roles we still have some patches opened14:14
slaweq https://review.opendev.org/q/topic:%2522secure-rbac%2522+(status:open+OR+status:merged)+project:openstack/neutron14:14
mlavalleI'll review today14:14
slaweqin one of them bcafarel raised some doubt which was discussed with lance14:15
slaweqI will try to ping him today and clarify that with him14:15
slaweqthx mlavalle14:15
slaweqregarding  https://blueprints.launchpad.net/neutron/+spec/distributed-dhcp-for-ml2-ovs14:16
slaweqwe have bunch of patches opened: https://review.opendev.org/q/topic:%22bp%252Fdistributed-dhcp-for-ml2-ovs%22+(status:open%20OR%20status:merged)14:16
slaweqplease review them too :)14:16
haleybi will try and look later as i had some comments in earlier versions14:17
slaweqhaleyb: thx14:17
slaweqthat's basically all regarding BPs for today14:17
slaweqI think we can move on to the next topics14:17
slaweq#topic Community Goals14:18
*** openstack changes topic to "Community Goals (Meeting topic: networking)"14:18
slaweqralonsoh: any updates/comments about privsep?14:18
ralonsohslaweq, not today, still testing alternatives to remove last piece of rootwrap14:18
ralonsohbut not in this cycle...14:18
ralonsohsorry for that14:18
slaweqnp14:18
ralonsoh(I didn't have time)14:18
slaweqI think we did great progress in that cycle14:19
slaweqactually, You did great progress :)14:19
ralonsohthanks!14:19
slaweqlet's move on to the next topic14:20
slaweq#topic Bugs14:20
*** openstack changes topic to "Bugs (Meeting topic: networking)"14:20
slaweqbcafarel was bug deputy, report: http://lists.openstack.org/pipermail/openstack-discuss/2021-March/020767.html14:20
slaweqbcafarel: any bugs You want to bring here up14:20
bcafareland slaweq was link-corrector for the bug deputy :)14:20
bcafareloverall as I said it was quiet week, https://bugs.launchpad.net/neutron/+bug/1916761 may benefit from DVR eyes (lengthy discusison in it already)14:21
openstackLaunchpad bug 1916761 in neutron "[dvr] bound port permanent arp entries never deleted" [High,In progress] - Assigned to Edward Hope-Morley (hopem)14:21
slaweq:)14:22
bcafareland ralonsoh replied already in opnion bug I wondered about https://bugs.launchpad.net/neutron/+bug/191691814:22
openstackLaunchpad bug 1916918 in neutron "It's better to use deepcopy in function rpc_loop of ovs-agent" [Undecided,New]14:22
bcafarelthe rest is either fixed or good progress14:22
slaweqregarding dvr issue, liuyulong seems to be already on it so we should be in good hands :)14:23
bcafarelindeed!14:24
slaweqthx bcafarel for taking care of bugs last week14:24
slaweqthis week I'm bug deputy14:24
slaweqand I already reminded it to myself :P14:24
bcafarelaka everyone, clear up that bug backlog you have been keeping around14:25
slaweqany other bugs You want to discuss today?14:25
slaweqnow it's the time for that ;)14:25
ralonsohhehehe14:25
slaweqbcafarel: indeed! :D14:25
slaweqok, I assume that we don't have any other bugs to discuss today then14:27
slaweqI think it's all from me for today14:28
slaweqdo You have anything else You would like to discuss/share with the team today?14:28
ralonsohslaweq, you had another list of patches to be reviewed this week, right?14:29
slaweqralonsoh: nope14:29
ralonsohok, sorry14:30
slaweqthere was 2 patches in neutron-lib which I though would be good to include in last release14:30
slaweqbut actually after more thinking of it, both can easily wait a bit longer14:30
slaweqralonsoh: most important patches from my pov to review are those related to BPs, like secure-rbac and address-groups14:31
ralonsohsure14:31
slaweqthx14:31
slaweqok, if there are no other topics to discuss, lets finish meeting earlier today14:32
slaweqthx for attending14:32
slaweqsee You online14:32
ralonsohbye14:32
slaweqo/14:32
bcafarelo/14:32
slaweq#endmeeting14:32
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"14:32
openstackMeeting ended Tue Mar  2 14:32:57 2021 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)14:32
openstackMinutes:        http://eavesdrop.openstack.org/meetings/networking/2021/networking.2021-03-02-14.00.html14:32
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/networking/2021/networking.2021-03-02-14.00.txt14:33
openstackLog:            http://eavesdrop.openstack.org/meetings/networking/2021/networking.2021-03-02-14.00.log.html14:33
lajoskatonaBye14:33
amotokibye14:33
rubasovbye14:35
*** Luzi has quit IRC14:36
*** jlibosva has joined #openstack-meeting-314:55
slaweq#startmeeting neutron_ci15:00
openstackMeeting started Tue Mar  2 15:00:33 2021 UTC and is due to finish in 60 minutes.  The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot.15:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:00
*** openstack changes topic to " (Meeting topic: neutron_ci)"15:00
openstackThe meeting name has been set to 'neutron_ci'15:00
slaweqhi15:00
bcafarelhey again15:00
slaweqping ralonsoh: lajoskatona15:02
slaweqci meeting, are You going to attend?15:02
ralonsohhi15:02
lajoskatonaHi15:02
slaweqI think we can start15:03
slaweqGrafana dashboard: http://grafana.openstack.org/dashboard/db/neutron-failure-rate15:03
slaweq#topic Actions from previous meetings15:03
*** openstack changes topic to "Actions from previous meetings (Meeting topic: neutron_ci)"15:03
slaweqwe have only one15:03
slaweqslaweq to check failing periodic task in functional test15:03
slaweqI reported bug  https://bugs.launchpad.net/neutron/+bug/191676115:03
openstackLaunchpad bug 1916761 in neutron "[dvr] bound port permanent arp entries never deleted" [High,In progress] - Assigned to Edward Hope-Morley (hopem)15:03
slaweqand proposed patch https://review.opendev.org/c/openstack/neutron/+/77808015:03
slaweqit seems that it works, at least there are no errors related to the maintenance task in the logs15:04
slaweqjob-output.txt file is about 10x smaller with that change15:04
slaweqno, sorry15:05
slaweqit's not so much smaller15:05
slaweqbut it is smaller significantly15:05
slaweqplease review that patch if You will have some time15:05
slaweqand let's move on15:06
slaweq#topic Stadium projects15:06
*** openstack changes topic to "Stadium projects (Meeting topic: neutron_ci)"15:06
slaweqanything regarding stadium projects' ci?15:06
lajoskatonanot much from me15:06
lajoskatonastill strugling with old branches fixes15:06
bcafarel:)15:06
lajoskatonaI am on the point to ask around infra or QA channel15:06
lajoskatonaI run into stupid no pip2.7 available issues and similar15:07
lajoskatonabut it's on older (before train??) branches so15:07
lajoskatonathats it as I remember15:08
slaweqI think that all branches before Train are already EM15:09
bcafarelyep15:09
slaweqso we can mark them as EOL for stadium projects probably15:09
slaweqno?15:09
lajoskatonayes, we can check all and decide based on that15:10
lajoskatonaI mean based on the alive backports or similar15:10
slaweqlajoskatona: so if there is no any interest in community to maintain them, and there are big issues, I would say - don't spent too much time on it :)15:10
lajoskatonaagree15:11
slaweqthx, please keep me updated if You will want to EOL some branches in some projects15:11
bcafarelespecially if they have been broken for some time and no one complained15:12
lajoskatonaok, I check where we are with those branches15:12
bcafareland I think we can easily announce them as Unmaintained (then if people show up to fix it, it can go back to EM)15:12
slaweq++15:13
lajoskatona+115:13
slaweqthx lajoskatona for taking care of it15:13
slaweqlet's move on15:13
slaweq#topic Stable branches15:13
*** openstack changes topic to "Stable branches (Meeting topic: neutron_ci)"15:13
slaweqVictoria dashboard: https://grafana.opendev.org/d/HUCHup2Gz/neutron-failure-rate-previous-stable-release?orgId=115:13
slaweqUssuri dashboard: https://grafana.opendev.org/d/smqHXphMk/neutron-failure-rate-older-stable-release?orgId=115:13
slaweqbcafarel: any updates about stable branches?15:13
ralonsohtrain is failing, "neutron-tempest-dvr-ha-multinode-full" job, qos migration tests15:15
ralonsoheg: https://0c345762207dc13e339e-d1e090fdf1a39e65d2b0ba37cbdce0a4.ssl.cf2.rackcdn.com/777781/1/check/neutron-tempest-dvr-ha-multinode-full/463e963/testr_results.html15:15
ralonsohin all patches15:15
slaweqralonsoh: but this job is non-voting, right?15:15
ralonsohit is, yes15:16
ralonsohjust a heads-up15:16
slaweqbut it seems like nova issue really15:16
slaweqNo valid host found for cold migrate15:16
slaweqand15:16
slaweq No valid host found for resize15:16
slaweqis it only in train?15:16
bcafarelsorry Murphy's law, mailman ringing just before the ping15:17
bcafareldid it work before? I remember this job being mostly unstable15:17
*** macz_ has joined #openstack-meeting-315:19
slaweqTBH I wasn't checking it for pretty long time15:19
slaweqI can investigate and reach out to nova ppl if needed15:19
slaweq#action slaweq to check failing qos migration tests in train neutron-tempest-dvr-ha-multinode-full job15:20
slaweqanything else related to stable branches?15:21
bcafarelhopefully https://review.opendev.org/c/openstack/neutron/+/777389 will be done soon for stein :)15:22
bcafarelrest looked OK, I have a few ones to review in my backlog, but CI looked good overall15:22
slaweqthx15:22
slaweqso we can move on15:23
slaweqnext topic15:23
slaweq#topic Grafana15:23
*** openstack changes topic to "Grafana (Meeting topic: neutron_ci)"15:23
slaweqhttp://grafana.openstack.org/dashboard/db/neutron-failure-rate15:23
slaweqI see just one, but pretty serious problem there15:23
slaweqfullstack/functional jobs are failing way too much still15:23
slaweqexcept that it seems that we are good15:24
slaweqdo You see any other issues there?15:25
ralonsohwe need to commit ourselves to, if we find an error in those jobs, report it in LP15:25
ralonsohjust to track it15:25
slaweqyes, I agree15:26
slaweqI spent some time today on them15:26
slaweqand I came with few small patches https://review.opendev.org/q/topic:%22improve-neutron-ci%22+(status:merged)15:26
slaweqsorry15:26
slaweqhttps://review.opendev.org/q/topic:%2522improve-neutron-ci%2522+status:open15:26
slaweqthat is correct link15:27
slaweqplease take a look at them15:27
slaweqI think that most often failures are due to oom-killer kills mysql server15:28
ralonsohright15:28
slaweqso I proposed to lower number of test workers in both jobs15:28
slaweqin fullstack I already did that some time ago15:28
ralonsohlet's reduce FT to 4 and fullstack to 315:28
slaweqbut I forgot about dsvm-fullstack-gate tox env which is used in gate really15:28
slaweqralonsoh: that's exactly what my patches proposed :)15:29
slaweqFT to 4 and fullstack to 315:29
ralonsohyeah hehehe15:29
slaweq:)15:29
slaweqso that should be covered :)15:29
slaweqanother issue are timeouted jobs15:29
slaweqand I think that those are mostly due to stestr and "too much output"15:30
slaweqlike we had already in the past in UT IIRC15:30
slaweqso I proposed https://review.opendev.org/c/openstack/neutron/+/77819615:30
slaweqand also https://review.opendev.org/c/openstack/neutron/+/778080 should helps for that15:30
slaweqbut in FT job there is still a lot of things logged15:30
slaweqif You check in https://23965cc52ad55df824a3-476d86922c45abb704c82e069ca48dea.ssl.cf1.rackcdn.com/778080/2/check/neutron-functional-with-uwsgi/875857e/job-output.txt15:31
slaweqthere is a lot of errors like:15:31
slaweqoslo_db.exception.CantStartEngineError: No sql_connection parameter is established15:31
slaweqand a lot of lines like:15:31
slaweqRunning command: ...15:31
slaweqI was trying to somehow get rid of them but I really don't know how15:31
slaweqif You would have any ideas, help is more than welcome :)15:32
slaweqanyone wants to check that?15:33
ralonsohsure15:33
*** lpetrut has quit IRC15:34
slaweqthx ralonsoh15:34
slaweq#action ralonsoh to try to check how to limit number of logged lines in FT output15:35
* slaweq will be back in 2 minutes15:35
ralonsohduring this waiting time, I think this is because "DBInconsistenciesPeriodics", in FTs15:36
ralonsohbut I need to check it15:36
* slaweq is back15:36
slaweqralonsoh: but in patch https://review.opendev.org/c/openstack/neutron/+/778080 I mocked this maintenance worker thread15:37
slaweqand still there are those lines logged there15:37
slaweqmaybe I missed something there, idk15:37
ralonsohright, you are stopping the thread there15:38
slaweqthere is one more issue which I found couple of times in FT recently15:39
slaweqTimeouts while doing some ip operations, like in15:39
slaweq    https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_018/772460/13/check/neutron-functional-with-uwsgi/0181e4f/testr_results.html15:39
slaweq    https://3d423a08ba57e3349bef-667e59a55d2239af414b0984e42f005a.ssl.cf5.rackcdn.com/771621/7/check/neutron-functional-with-uwsgi/c8d3396/testr_results.html15:39
slaweq    https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_ac4/768129/37/check/neutron-functional-with-uwsgi/ac480c2/testr_results.html15:39
slaweqdid You saw them already?15:39
slaweqmaybe You know what can cause such problems?15:40
ralonsohI did but I still can't find the root cause15:40
slaweqI saw it always in those test_linuxbridge_arp_protect tests module15:41
slaweqdid You saw it in different modules too?15:41
ralonsohI can't remember15:41
slaweqmaybe we could mark those tests are unstable for now and that would give us a breath15:41
ralonsohI'll record all appearances I find and I'll report a LP15:42
slaweqralonsoh: thx15:42
slaweq#action ralonsoh to report bug with ip operations timeout in FT15:43
slaweqso that's basically all what I had for today regarding those jobs15:43
slaweqlong story short, lets merge patches which we have now and hopefully it will be bit better15:43
slaweqand then lets focus on debugging issues which we already mentioned here15:44
slaweqlast topic for today15:44
slaweq#topic Periodic15:44
*** openstack changes topic to "Periodic (Meeting topic: neutron_ci)"15:44
slaweqJobs results: http://zuul.openstack.org/buildsets?project=openstack%2Fneutron&pipeline=periodic&branch=master15:44
slaweqin overall periodic jobs seems good15:44
slaweqbut fedora based job is all the time red15:44
slaweqis there any volunteer to check it?15:44
*** obondarev has quit IRC15:45
ralonsohnot this week, sorry15:45
bcafarelagain? sigh15:45
slaweqno need to sorry ralonsoh :)15:45
bcafarelI can try to take a look15:45
slaweqbcafarel: I think it is failing still, not again :)15:45
slaweqbcafarel: thx a lot15:45
slaweq#action bcafarel to check failing fedora based periodic job15:46
bcafarelseems not so long ago we had to push a few fixes for it :)15:46
slaweqyes, but then it start failing again15:47
slaweqand we never fixed it I think :/15:47
slaweqand that's basically all what I had for today15:48
slaweqdo You have anything else You want to discuss today, regarding our ci?15:48
ralonsohno15:48
bcafarelnothing from me15:48
slaweqif no, I will give You few minutes back today15:48
slaweqthx for attending the meeting15:48
slaweqand have a great week15:48
slaweqo/15:48
ralonsohbye!15:48
slaweq#endmeeting15:49
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"15:49
openstackMeeting ended Tue Mar  2 15:48:59 2021 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)15:49
openstackMinutes:        http://eavesdrop.openstack.org/meetings/neutron_ci/2021/neutron_ci.2021-03-02-15.00.html15:49
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/neutron_ci/2021/neutron_ci.2021-03-02-15.00.txt15:49
openstackLog:            http://eavesdrop.openstack.org/meetings/neutron_ci/2021/neutron_ci.2021-03-02-15.00.log.html15:49
lajoskatonaBye15:49
*** psahoo_ has quit IRC16:02
*** lajoskatona has left #openstack-meeting-316:05
*** e0ne has joined #openstack-meeting-316:37
*** lpetrut has joined #openstack-meeting-317:28
*** bnemec has quit IRC17:49
*** lpetrut has quit IRC18:07
*** bnemec has joined #openstack-meeting-318:16
*** ralonsoh has quit IRC18:29
*** eolivare has quit IRC19:33
*** e0ne has quit IRC20:55
*** jlibosva has quit IRC21:16
*** belmoreira has quit IRC21:24
*** belmoreira has joined #openstack-meeting-321:37
*** jlibosva has joined #openstack-meeting-321:52
*** jlibosva has quit IRC21:57
*** SWDevAngel has quit IRC22:02
*** belmoreira has quit IRC22:23
*** jlibosva has joined #openstack-meeting-323:53
*** jlibosva has quit IRC23:58

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!