Tuesday, 2021-04-27

*** mlavalle has quit IRC00:01
*** ricolin has quit IRC01:37
*** hemanth_n has joined #openstack-meeting-302:57
*** slaweq has joined #openstack-meeting-305:52
*** ralonsoh has joined #openstack-meeting-306:04
*** lpetrut has joined #openstack-meeting-306:11
*** eolivare has joined #openstack-meeting-306:59
*** e0ne has joined #openstack-meeting-307:24
*** tosky has joined #openstack-meeting-307:41
*** ricolin has joined #openstack-meeting-307:43
*** isabek has joined #openstack-meeting-307:55
*** masaki-ueno has joined #openstack-meeting-308:02
*** masaki-ueno has left #openstack-meeting-308:02
*** e0ne has quit IRC10:46
*** macz_ has joined #openstack-meeting-311:24
*** eolivare_ has joined #openstack-meeting-311:27
*** macz_ has quit IRC11:28
*** eolivare has quit IRC11:29
*** eolivare_ has quit IRC11:49
*** hemanth_n has quit IRC12:12
*** eolivare_ has joined #openstack-meeting-312:28
*** macz_ has joined #openstack-meeting-312:37
*** macz_ has quit IRC12:42
*** macz_ has joined #openstack-meeting-312:58
*** e0ne has joined #openstack-meeting-313:01
*** macz_ has quit IRC13:02
*** e0ne has quit IRC13:03
*** e0ne has joined #openstack-meeting-313:45
*** lpetrut has quit IRC14:00
ralonsohslaweq, do we have meeting today?14:01
slaweq#startmeeting networking14:01
openstackMeeting started Tue Apr 27 14:01:47 2021 UTC and is due to finish in 60 minutes.  The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot.14:01
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.14:01
*** openstack changes topic to " (Meeting topic: networking)"14:01
slaweqralonsoh: yes14:01
openstackThe meeting name has been set to 'networking'14:01
slaweqsorry for being late :)14:01
ralonsohhehehe hi14:01
bcafarel:)14:01
bcafarelo/14:01
ralonsohnp at all14:01
rubasov_o/14:02
haleybhi14:02
slaweq#topic announcements14:02
*** openstack changes topic to "announcements (Meeting topic: networking)"14:02
slaweqwelcome on the first meeting after the PTG :)14:02
slaweqXena cycle calendar https://releases.openstack.org/xena/schedule.html14:03
slaweqnext milestone is X-1 and it's in the week of May 24th14:03
slaweqyesterday I sent summary of our PTG sessions, it's available at http://lists.openstack.org/pipermail/openstack-discuss/2021-April/022044.html14:04
slaweqand also at  http://kaplonski.pl/blog/virtual_ptg_april_2021_summary/14:04
slaweqand basically that's all announcements from me today14:04
slaweqdo You have anything else?14:04
slaweqok, so let's move on14:06
slaweq#topic Blueprints14:06
*** openstack changes topic to "Blueprints (Meeting topic: networking)"14:06
slaweqI updated list for X-1 https://bugs.launchpad.net/neutron/+milestone/xena-114:06
slaweqand I created BPs for specs proposed by rubasov_ and lajoskatona, which we discussed during the PTG14:06
rubasov_thank you14:07
slaweqall are scheduled there so we will track them here every week14:07
slaweqdo You have any updates about any of the BPs today?14:07
slaweqif not, I think we can move on, but please spent some time reviewing patches and specs related to those BPs14:09
slaweqok, next topic14:09
slaweq#topic Community goals14:09
*** openstack changes topic to "Community goals (Meeting topic: networking)"14:09
slaweqAFAIK there are no new goals for Xena cycle, right?14:09
slaweqif yes, I propose to suspend that topic for now14:09
slaweqwdyt?14:09
ralonsohok for me14:10
*** njohnston has joined #openstack-meeting-314:11
*** obondarev has joined #openstack-meeting-314:11
bcafareland if Xena goal is "stabilize and clean" it probably does not need specific section either14:12
slaweqyeah14:12
ralonsohright14:12
slaweqok, so let's move on14:12
slaweq#topic bugs14:12
*** openstack changes topic to "bugs (Meeting topic: networking)"14:12
slaweqralonsoh was our bug deputy last week14:12
slaweqreport: http://lists.openstack.org/pipermail/openstack-discuss/2021-April/022029.html14:12
slaweqralonsoh: any bugs You want to discuss now?14:12
ralonsohjust a heads-up, related to those three frist bugs, not assigned14:13
ralonsohtwo critical, one high14:13
ralonsohI didn't have time to check https://bugs.launchpad.net/neutron/+bug/192549814:13
openstackLaunchpad bug 1925498 in neutron "SNAT is not working" [High,New]14:13
ralonsohif this is true, that's a big issue. But I don't know how this bug could be possible, this is also tested in the CI14:14
ralonsohwell, not exactly14:14
slaweqralonsoh: tbh I'm not sure if we are testing snat really14:14
ralonsohno, not SNAT14:14
slaweqmaybe we have (yet another) testing gap there14:14
ralonsohand I don't know if this is related to the backend (linux bridges)14:14
ralonsohanyway, if you have time, please try to check it14:15
slaweqI will add it to my backlog, but I don't think I will have time this week14:15
slaweqso I will not assign it to myself for now14:15
slaweqif someone have time, feel free to take it14:16
slaweqralonsoh: I was checking also https://bugs.launchpad.net/neutron/+bug/1925368 and IMO it is a bug14:17
openstackLaunchpad bug 1925368 in neutron "[L3] Router GW can be removed with routes defined" [Low,New]14:17
ralonsohslaweq, thanks for checking it14:17
slaweqwe should probably forbid removal of the gw if we have routes which are using it14:17
ralonsohok, I'll push a patch to limit that14:17
slaweqthx14:18
rubasov_if I may call attention to one more bug from the week before ralonsoh's (since we did not have a meeting last week)14:18
rubasov_this is an unassigned ovn bug: https://bugs.launchpad.net/neutron/+bug/192208914:18
openstackLaunchpad bug 1922089 in neutron "[ovn] enable_snat cannot be disabled once enabled" [High,Triaged]14:18
slaweqrubasov_: sure14:18
slaweqmaybe lucasgomes or jlibosva can take a look at that one?14:19
slaweqI will ask them14:19
rubasov_thanks14:19
slaweqthx rubasov_ for raising it here14:19
slaweqralonsoh: is that all what You wanted to highlight?14:20
ralonsohyes, thanks14:20
slaweqthx14:20
slaweqI also wanted to talk about one bug14:20
slaweqhttps://bugs.launchpad.net/neutron/+bug/192265314:20
openstackLaunchpad bug 1922653 in neutron "[L3][Port forwarding] multiple floating_ip:port to same internal fixed_ip:port (N-to-1 rule support)" [Medium,In progress] - Assigned to LIU Yulong (dragon889)14:20
slaweqand fix for it https://review.opendev.org/c/openstack/neutron/+/78730414:20
slaweqI'm not sure if we should have shim api extension for that14:20
slaweqas it in fact changes api behaviour14:20
slaweqsomething what was returing 4xx now is accepted14:21
slaweqwdyt?14:21
ralonsohI'll need to check it first14:21
ralonsohbut yes, if this is the case, this is changing the API14:21
slaweqok, please take a look at that patch when You will have some time14:22
slaweqand that's basically all from me regarding bugs14:22
*** obondarev has quit IRC14:22
slaweqthis week lucasgomes is bug deputy14:22
slaweqand he is aware of it already14:22
slaweqnext week it will be jlibosva14:23
slaweqand after him, obondarev14:23
slaweqand we will have new round14:23
slaweqso bcafarel is it ok for You to cover week of the May 17th?14:23
* bcafarel checks the calendar14:24
bcafarelslaweq: should be OK yes14:25
slaweqthx14:25
slaweqok, any other bugs You want to discuss today?14:25
slaweqI guess this means "no" :)14:26
slaweqso let's move on14:26
slaweq#topic CLI/SDK14:27
*** openstack changes topic to "CLI/SDK (Meeting topic: networking)"14:27
slaweqWe recently merged https://review.opendev.org/c/openstack/openstacksdk/+/768208 in Openstack SDK14:27
slaweqthat patch should allows us to pass unknown attributes from OSC to the Neutron14:27
slaweqI will now work on that change in OSC14:27
slaweqwhen we will have it, it should be last missing part to finally deprecate neutronclient in this cycle14:28
bcafarel\o/14:28
slaweqthat's just a heads up for now :)14:29
ralonsoh+114:29
slaweqok, if there are no other topics I will give You 30 minutes back today :)14:29
slaweqdo You have anything else You want to discuss?14:29
slaweqif not, then thx for attending14:30
ralonsohthanks, see you in 30 mins14:30
slaweqand please remember about ci meeting in 30 minutes :)14:30
bcafarelI think we talked a lot last week :)14:30
slaweq#endmeeting14:30
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"14:31
openstackMeeting ended Tue Apr 27 14:30:59 2021 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)14:31
openstackMinutes:        http://eavesdrop.openstack.org/meetings/networking/2021/networking.2021-04-27-14.01.html14:31
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/networking/2021/networking.2021-04-27-14.01.txt14:31
openstackLog:            http://eavesdrop.openstack.org/meetings/networking/2021/networking.2021-04-27-14.01.log.html14:31
rubasov_o/14:31
*** lajoskatona has joined #openstack-meeting-314:46
*** e0ne has quit IRC14:47
slaweq#startmeeting neutron_ci15:01
openstackMeeting started Tue Apr 27 15:01:21 2021 UTC and is due to finish in 60 minutes.  The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot.15:01
slaweqhi15:01
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:01
*** openstack changes topic to " (Meeting topic: neutron_ci)"15:01
openstackThe meeting name has been set to 'neutron_ci'15:01
ralonsohhi15:01
lajoskatonaHi15:01
bcafarelo/15:01
slaweqGrafana dashboard: http://grafana.openstack.org/dashboard/db/neutron-failure-rate15:01
slaweqand now we can start15:02
slaweq#topic Actions from previous meetings15:02
*** openstack changes topic to "Actions from previous meetings (Meeting topic: neutron_ci)"15:02
slaweqfirst one15:03
slaweqslaweq to update wallaby's scenario jobs in neutron-tempest-plugin15:03
slaweqI did, all patches are merged but I don't have links now15:03
slaweqnext one15:03
slaweqbcafarel to report stable/rocky ci failures on LP15:03
bcafarelhttps://bugs.launchpad.net/neutron/+bug/1924315 and our fearless PTL close to fixing it (when CI is happy)15:05
openstackLaunchpad bug 1924315 in neutron "[stable/rocky] neutron-tempest-plugin-scenario-openvswitch-iptables_hybrid-rocky job fails" [Critical,In progress] - Assigned to Slawek Kaplonski (slaweq)15:05
slaweq"fearless PTL" :D15:05
bcafarelalthough I was looking in https://bugs.launchpad.net/neutron/+bug/1925451 - grenade seems to fail about 50% of the time on that DistutilsError15:05
openstackLaunchpad bug 1925451 in neutron "[stable/rocky] grenade job is broken" [Critical,New]15:05
slaweqyou made my day now15:05
bcafarel:)15:06
slaweqpatch https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/786657 should fix that original issue15:06
slaweqregarding grenade one, did You check if that is e.g. failing only one some of the cloud providers?15:07
bcafarelno, good point I will check that15:08
slaweqthx15:08
slaweqso lets continue this discussion later/tomorrow15:08
*** mlavalle has joined #openstack-meeting-315:08
slaweqwe need to fix it finally and unblock rocky's gate15:09
bcafarel+115:09
slaweqralonsoh: please check https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/786657 :)15:09
slaweqthis is also needed for rocky gate15:09
ralonsohdone15:09
slaweqthx15:09
slaweqok, next one15:10
slaweqralonsoh to mark test_keepalived_spawns_conflicting_pid_vrrp_subprocess functional test as unstable15:10
ralonsohno progress last week, but related to the kill signal15:10
ralonsohno progress, sorry15:10
slaweqI will set it for You for this week, ok?15:10
ralonsohsure15:10
slaweq#action ralonsoh to mark test_keepalived_spawns_conflicting_pid_vrrp_subprocess functional test as unstable15:10
slaweqthx15:10
slaweqnext one15:11
slaweqslaweq to report LP with metadata issue in scenario jobs15:11
slaweq    Bug reported: https://bugs.launchpad.net/neutron/+bug/192363315:11
openstackLaunchpad bug 1923633 in neutron "Neutron-tempest-plugin scenario jobs failing due to metadata issues" [Critical,Confirmed] - Assigned to Slawek Kaplonski (slaweq)15:11
slaweqthis is currently IMO most hurting us bug in ci15:11
slaweqwe investigated that with ralonsoh last week15:11
ralonsohhttps://review.opendev.org/c/openstack/neutron/+/78777715:11
slaweqand we think we know what is going on there15:11
ralonsohdoesn't help too much15:11
slaweqstill same issues?15:12
ralonsohyes but less frequent15:12
slaweq:/15:12
slaweqso maybe we don't know exactly what is the problem there15:12
ralonsohthe socket receiving the messages is not responsive15:12
slaweqfor very long time, or forever?15:13
ralonsohlong time15:13
ralonsohanother option could be not to mock socket module15:13
ralonsohin the L3 agent15:13
slaweqI wonder what have been changed recently there15:13
ralonsohI'll try it in this patch15:13
slaweqas it wasn't that bad few weeks back15:13
ralonsohs/mock/monkey_patch15:13
slaweqok, lets try that15:14
ralonsohok15:14
slaweqI also made patch https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/787324 which should mittigate the issue a bit at least15:14
slaweqit's merged now, so hopefully jobs will be a bit more stable now15:15
slaweqand also, if You will see in job error that "router wasn't become active on any L3 agent" than it means for sure that You hit the same bug15:15
slaweqso it will be easier to identify that specific issue now15:15
ralonsohright15:15
slaweqok, next topic15:16
slaweq#topic Stadium projects15:16
*** openstack changes topic to "Stadium projects (Meeting topic: neutron_ci)"15:16
slaweqany updates?15:16
lajoskatonano specific thing15:17
lajoskatonaCI is working (at elast where I see new patches :-)15:17
bcafarelthat section will probably heat up with OVN switch15:17
slaweqok, that's good news :)15:17
slaweqthx15:17
slaweqbcafarel: true15:17
slaweqmaybe we should start changing jobs definitions where it's needed?15:18
slaweqany volunteer to do that?15:18
lajoskatonayeah perhaps, to make it happen paralelly15:18
lajoskatonaI can check15:18
slaweqthx lajoskatona15:18
*** isabek has quit IRC15:19
slaweq#action lajoskatona to check stadium job's and what needs to be switched to ovs explicity15:19
slaweqok, next topic15:19
slaweq#topic Stable branches15:19
*** openstack changes topic to "Stable branches (Meeting topic: neutron_ci)"15:19
slaweqanything to discuss?15:19
slaweqexcept rocky15:19
bcafarelthere is still https://bugs.launchpad.net/neutron/+bug/1923412 for stein, I hope to finally take a look this week15:20
openstackLaunchpad bug 1923412 in neutron "[stable/stein] Tempest fails with unrecognized arguments: --exclude-regex" [Critical,Triaged]15:20
slaweqbcafarel: ouch, I missed that one15:21
slaweqit's the same issue like for rocky15:21
*** macz_ has joined #openstack-meeting-315:21
slaweqor very similar15:21
toskybcafarel: oh, there is a devstack change which may solve that (but you can still fix it by refactoring the jobs)15:21
toskynamely https://review.opendev.org/c/openstack/tempest/+/78745515:21
toskytempest, not devstack15:21
bcafareltosky: oh nice! I will test it as depends-on on one of our stein pending backports15:22
slaweqnice, thx tosky15:22
toskyor you can do what I did for cinder-tempest-plugin15:23
toskyhttps://review.opendev.org/c/openstack/cinder-tempest-plugin/+/78675515:23
toskybut that requires branch-specific job variants and maybe a bit of refactoring (or it may be easy, depending on your job structure)15:23
lajoskatonaHave you read the TC pad (https://etherpad.opendev.org/p/tc-xena-ptg ~l360) about EOLing old branches (ocata....) ?15:23
slaweqyes, I did something similar for our rocky jobs already15:24
toskyocata hopefully will be finally EOLed15:24
toskyand pike, I guess it depends on more projects abandoning it (we did it in cinder)15:24
tosky(so if you think about abandoning pike, please do it :)15:24
lajoskatonaok, so perhaps the lavina will start :-)15:25
bcafarel:) I don't recall recent backport requests on pike15:25
slaweqme neighter15:25
slaweqonly queens and newer15:25
slaweqbut still, even queens and rocky are starting to be pain15:25
bcafarelno open pike backport, last merge in July 202015:26
toskyyeah, in a non far future (in cinder, again) we are thinking about abandoning those too15:26
slaweq++15:26
slaweqwe can think about it also15:26
slaweqor just do it15:26
slaweqI will take a look15:26
toskyit seems one of those things where, if no one starts, it's never going to happen15:26
lajoskatonayeah but silently we anyway skip those branches15:27
toskyso better give that message to the community in an official way: this is gone15:28
slaweqtrue, it's just not officially marked as EOL15:28
slaweqI will check how to do it in next week15:28
slaweqthx for bringing that topic up15:29
bcafarel+115:30
slaweqok, lets move on15:30
slaweq#topic Grafana15:30
*** openstack changes topic to "Grafana (Meeting topic: neutron_ci)"15:30
slaweqlooking at dashboard, the only big problem which I see is that one with neutron-tempest-plugin jobs15:31
slaweqand that is mostly cause by the bug with L3 HA which we already discussed earlier15:32
slaweqdo You see anything else You want to discuss?15:34
slaweqor can we move on?15:34
bcafarellet's go to next topic yes15:35
slaweqok, let's go15:35
slaweq#topic fullstack/functional15:35
*** openstack changes topic to "fullstack/functional (Meeting topic: neutron_ci)"15:35
slaweqHere there is just one quick thing15:35
slaweqplease review new test https://review.opendev.org/c/openstack/neutron/+/783748 :)15:36
ralonsohsure15:36
slaweqthx15:36
slaweqI don't have any new issues from those jobs for today15:36
slaweq#topic Tempest/Scenario15:36
*** openstack changes topic to "Tempest/Scenario (Meeting topic: neutron_ci)"15:36
slaweqhere there is couple of new issues15:36
slaweqfirst one, there is bug reported by Liu:15:36
slaweqhttps://bugs.launchpad.net/neutron/+bug/192610915:37
openstackLaunchpad bug 1926109 in neutron "SSH timeout (wait timeout) due to potential paramiko issue" [Critical,New]15:37
slaweqbut tbh, I'm not sure if that isn't the same issue with L3 HA like we discussed already15:37
ralonsohis this one related to the ha router?15:37
slaweqthe problem is that in that case there is no console log logged15:37
ralonsohyes, same concern15:37
slaweqI think we should first add log of the vm's consolelog first15:38
slaweqand then we will see if that's not duplicate15:38
slaweqany volunteer to do that?15:38
ralonsohexactly, to check the metadata update15:38
ralonsohI can (at the end of the week)15:38
slaweqor, even better15:38
lajoskatonaI can add, will se if can do before ralonsoh15:38
slaweqwe should be able to know it without console log now, when https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/787324 is merged15:38
slaweqbut console log could be useful always15:39
slaweqso thx lajoskatona and ralonsoh for taking care of it15:39
ralonsohyeah, console output will help15:39
slaweq#action ralonsoh or lajoskatona will add logging of the console log, related to the https://bugs.launchpad.net/neutron/+bug/192610915:39
openstackLaunchpad bug 1926109 in neutron "SSH timeout (wait timeout) due to potential paramiko issue" [Critical,New]15:39
slaweq:)15:39
slaweqI assigned it to both of You :P15:39
slaweqI also found one issue with multicast test in the ovn job:15:40
slaweq    https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_b66/712474/7/check/neutron-tempest-plugin-scenario-ovn/b661cd4/testr_results.html15:40
slaweqbut I need to check if that is something what happens more often and report LP with it15:40
slaweq#action slaweq to check frequency of the multicast issue in the ovn job and report a LP bug for that15:41
slaweqand last one topic for today15:42
slaweq#topic Periodic15:42
*** openstack changes topic to "Periodic (Meeting topic: neutron_ci)"15:42
slaweqI just noticed that nftables jobs are failing every day15:42
slaweqlike e.g. https://619cfb3845a212f70f8d-f88cc2e228aea8b2c74f92ce7ecb609d.ssl.cf2.rackcdn.com/periodic/opendev.org/openstack/neutron/master/neutron-tempest-plugin-scenario-linuxbridge-nftables/1d9785e/job-output.txt15:42
slaweqand it's like that for both of them15:43
slaweqthey are failing on "[nftables : Restore saved IPv4 iptables rules, stored by iptables-persistent]"15:43
ralonsohyeah... ok, I'll check it15:43
slaweqthx15:43
slaweqralonsoh: to check periodic nftables jobs15:43
slaweq#action ralonsoh: to check periodic nftables jobs15:43
slaweqand that's basically all what I have for today15:44
slaweqdo You have anything else to discuss now?15:44
slaweqor if not, I'm closing the meeting and calling it a day finally :)15:44
bcafarelin that case, nothing to add for me :)15:45
slaweqok, so thx for attending the meeting15:45
ralonsohbye!15:45
lajoskatonao/15:46
bcafarelo/15:46
slaweqhave a nice day, and see You online15:46
slaweq#endmeeting15:46
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"15:46
openstackMeeting ended Tue Apr 27 15:46:06 2021 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)15:46
slaweqo/15:46
openstackMinutes:        http://eavesdrop.openstack.org/meetings/neutron_ci/2021/neutron_ci.2021-04-27-15.01.html15:46
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/neutron_ci/2021/neutron_ci.2021-04-27-15.01.txt15:46
openstackLog:            http://eavesdrop.openstack.org/meetings/neutron_ci/2021/neutron_ci.2021-04-27-15.01.log.html15:46
*** mlavalle has quit IRC15:53
*** lajoskatona has quit IRC16:40
*** eolivare_ has quit IRC17:07
*** mlavalle has joined #openstack-meeting-317:27
*** lajoskatona has joined #openstack-meeting-318:06
*** Adri2000 has quit IRC18:28
*** lajoskatona has quit IRC18:42
*** njohnston has quit IRC19:03
*** ralonsoh has quit IRC19:37
*** slaweq has quit IRC21:22
*** macz_ has quit IRC22:03
*** macz_ has joined #openstack-meeting-322:28
*** macz_ has quit IRC22:33
*** tosky has quit IRC23:08
*** mlavalle has quit IRC23:51

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!