Tuesday, 2020-12-15

*** tosky has quit IRC00:10
*** hemanth_n has joined #openstack-meeting-300:55
*** macz_ has quit IRC01:37
*** macz_ has joined #openstack-meeting-301:38
*** mlavalle has quit IRC02:16
*** macz_ has quit IRC06:38
*** slaweq has quit IRC07:56
*** slaweq has joined #openstack-meeting-307:58
*** macz_ has joined #openstack-meeting-308:35
*** macz_ has quit IRC08:39
*** tosky has joined #openstack-meeting-308:41
*** lpetrut has joined #openstack-meeting-308:57
*** e0ne has joined #openstack-meeting-309:15
*** yamamoto has joined #openstack-meeting-310:34
*** ricolin has quit IRC12:09
*** raildo has joined #openstack-meeting-312:25
*** yamamoto has quit IRC12:42
*** yamamoto has joined #openstack-meeting-312:43
*** yamamoto has quit IRC12:48
*** yamamoto has joined #openstack-meeting-312:48
*** yamamoto has quit IRC13:03
*** yamamoto has joined #openstack-meeting-313:03
*** macz_ has joined #openstack-meeting-313:07
*** macz_ has quit IRC13:07
*** yamamoto has quit IRC13:08
*** yamamoto has joined #openstack-meeting-313:09
*** yamamoto has quit IRC13:18
*** yamamoto has joined #openstack-meeting-313:55
*** mlavalle has joined #openstack-meeting-313:58
slaweq#startmeeting networking14:00
openstackMeeting started Tue Dec 15 14:00:17 2020 UTC and is due to finish in 60 minutes.  The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot.14:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.14:00
*** openstack changes topic to " (Meeting topic: networking)"14:00
openstackThe meeting name has been set to 'networking'14:00
ralonsohhi14:00
mlavalleo/14:00
slaweqhi14:00
hemanth_no/14:00
rubasovo/14:00
slaweq#topic Announcements14:01
*** openstack changes topic to "Announcements (Meeting topic: networking)"14:01
bcafarelo/14:01
amotokio/14:01
slaweqWallaby cycle calendar https://releases.openstack.org/wallaby/schedule.html14:02
slaweqwe are still getting to the milestone-2 of Wallaby14:02
slaweqplease focus on review of the specs listed in https://wiki.openstack.org/wiki/Network/Meetings#Announcements_.2F_Reminders14:02
slaweq     https://review.opendev.org/c/openstack/neutron-specs/+/72953214:02
slaweq    https://review.opendev.org/c/openstack/neutron-specs/+/65845114:03
slaweq    https://review.opendev.org/c/openstack/neutron-specs/+/72862814:03
slaweq    https://review.opendev.org/c/openstack/neutron-specs/+/73954914:03
slaweqI also have to read them again14:03
slaweqnext one14:03
slaweqwe are now getting into holidays season14:03
slaweqso I propose to cancel this meeting in next 2 weeks14:04
slaweqwdyt?14:04
ralonsoh+1 from me14:04
mlavalle++14:04
haleyb+214:04
slaweqok, I will send an email about that after that meeting14:04
amotoki+114:04
slaweqthx :)14:04
slaweqand that's all announcements from me for today14:04
bcafarelyay for holidays :)14:04
slaweqdo You have anything else You want to share?14:05
bcafarel"pip new resolver is fun"?14:05
* haleyb swears at pip14:05
slaweqbcafarel: yes I wanted to ask about summary of that issue :)14:06
*** yamamoto has quit IRC14:06
bcafarelthough I was thinking mostly about stable http://lists.openstack.org/pipermail/openstack-discuss/2020-December/019362.html thread has some nice answers14:06
bcafarelfrom what I got, l-c in its current state is not really helpful so the big bumps we do there are OK-ish14:07
bcafareland requirements ones are actually real issues in our deps so better to have them sorted out finally14:07
slaweqbcafarel: shouldn't we remove l-c job then?14:07
slaweqI know that some other projects are doing that, no14:07
slaweq?14:07
bcafareltest-requirements we may limit damage by moving deps in tox.ini (I sent https://review.opendev.org/c/openstack/neutron/+/767125 to see how it goes on master)14:07
amotokino installation of test-reqs in devstack perhaps makes sense to me14:08
bcafarelslaweq: there is a separate thread that started on that, I guess we can wait to see the conclusion there (if it should be dropped everywhere)14:08
amotokiI am not sure about removing l-c job14:08
bcafarelhttps://review.opendev.org/q/I8f24b839bf42e2fb9803dc7df3a30ae20cf264eb will probably help for some failures (it helped with that bandit 1.6.3 stuff)14:09
bcafarelso far victoria has backport pending https://review.opendev.org/c/openstack/neutron/+/766940 I am trying ussuri atm (I have hope a 1-line req change may be enough there let's see)14:10
haleybbcafarel: i'm having trouble with ussuri for ovn provider, will be looking to see if there's something i'm missing14:11
slaweqok, so lets see how it will go14:12
slaweqand thx bcafarel for taking care of it14:12
slaweqI think we can move on then14:13
slaweq#topic Blueprints14:13
*** openstack changes topic to "Blueprints (Meeting topic: networking)"14:13
slaweqWallaby-2 https://bugs.launchpad.net/neutron/+milestone/wallaby-214:13
slaweqany updates about BPs?14:13
slaweqregarding engine facade BP there are still some opened patches14:15
slaweqplease check https://review.opendev.org/q/status:open+topic:bp/enginefacade-switch and review them14:15
slaweqthose are some small leftovers which we missed to migrate earlier14:15
slaweqand I need to check stadium projects as well14:16
slaweqregarding https://blueprints.launchpad.net/neutron/+spec/secure-bac-roles there is a lot of patches ready to review: https://review.opendev.org/q/topic:%2522secure-rbac%2522+(status:open+OR+status:merged)+project:openstack/neutron14:16
slaweqI'm currently updating patches proposed by lbragstad to the format which we wanted, like define deprecated rule close to the rule definition, not on the top of file14:17
slaweqetc14:17
slaweqbut many of them should be already ok to review14:17
slaweqplease do so14:17
slaweqand that's all update which I have for today regarding BPs14:18
slaweqok, if there is no other update, lets move on14:20
amotokiI have one question on testing. In the policy pop-up meeting, how to test new policies was dsicussed.14:20
amotokido you have any discussion on this?14:20
amotokiI haven't followed up the discussion well though...14:21
slaweqnope, for now I was just focused on patches to change roles definitions14:21
slaweqbut IIRC keystone (and maybe nova) got some examples of UT to test that14:21
slaweqand we will probably need to add such UT also14:21
amotokithanks. some RBAC policies are now covered in our UT, but we may explore more systematic way. it would be the next step.14:22
slaweqamotoki: yes, it will be14:23
amotokii have nothing on this topic. I think we can move on14:24
slaweqthx14:24
slaweqso lets move on14:24
slaweq#topic Community Goals14:25
*** openstack changes topic to "Community Goals (Meeting topic: networking)"14:25
slaweqamotoki: ralonsoh any updates?14:25
ralonsohno14:25
amotokino updates on the policy.yaml migration. we are waiting for l-c fix in neutron-lib.14:25
slaweqok14:26
slaweqso lets move on14:26
slaweq#topic Bugs14:26
*** openstack changes topic to "Bugs (Meeting topic: networking)"14:26
slaweqI was bug deputy last week14:26
slaweqfull summary is at http://lists.openstack.org/pipermail/openstack-discuss/2020-December/019414.html14:26
slaweqthere is couple of bugs which needs some attention14:27
slaweqhttps://bugs.launchpad.net/neutron/+bug/1907232 - neutron-dynamic-routing issue14:27
openstackLaunchpad bug 1907232 in neutron "dynamic-routing: "dragent add speaker" is buggy" [High,New]14:27
slaweqwould be good if someone familiar with this project could take a look14:27
slaweqhttps://bugs.launchpad.net/neutron/+bug/1908057 - l3 ha issue with killing keepalived processes14:28
openstackLaunchpad bug 1908057 in neutron "Ensure "keepalived" is correctly disabled (the process does not exist)" [Medium,Confirmed] - Assigned to Rodolfo Alonso (rodolfo-alonso-hernandez)14:28
slaweqralonsoh: will You fix it?14:28
ralonsohI'm on this one14:28
ralonsohone sec14:28
ralonsohhttps://review.opendev.org/c/openstack/neutron/+/76693914:28
slaweqthx14:28
slaweqhttps://bugs.launchpad.net/neutron/+bug/1907175 - for me seems more like ovs bug, not neutron but maybe You can also take a look14:29
openstackLaunchpad bug 1907175 in neutron " intermittently ALL VM's floating IP connection is disconnected, and can be reconnected after 5-6 minutes " [Undecided,New]14:29
slaweqso if You will have some time, please take a look - maybe I missed something there14:29
slaweqand the last one14:30
slaweqhttps://bugs.launchpad.net/horizon/+bug/1907843 - I moved it to Horizon but maybe worth to check if we can do something in Neutron to help with tha14:30
openstackLaunchpad bug 1907843 in OpenStack Dashboard (Horizon) "security groups filtered with tenant_id does not accept rbac" [Undecided,New]14:30
slaweqamotoki: maybe You could take a look?14:30
amotokislaweq: sure14:30
ralonsoh(we need to remove tenant_id references)14:30
slaweqralonsoh: I think it's the same for project_id in that specific case :)14:30
amotokiit is a feature gap in horizon. neutron RBAC is not implemented yet.14:31
slaweqamotoki: ok, so that's what I thought - it's more for horizon then Neutron :)14:32
amotokiyes. If neutron API behavior in admin is like nova, the horizon implemention would be much simpler :-(14:32
amotokianyway it is a horizon bug.14:33
slaweqamotoki: thx for confirmation14:33
slaweqthis week our bug deputy is hongbin14:33
slaweqI sent him an email with reminder but I don't have any reply yet14:34
slaweqnext week it's haleyb turn14:34
slaweqare You ok with that?14:34
slaweqit should be quiet week :)14:34
bcafarelI can promise I will not login that week and fill bugs at least14:34
haleybslaweq: i'm on PTO next week, can maybe check email a couple of times14:34
haleybno bugs please everyone!14:35
slaweqhaleyb: I hope there will be no many bugs next week14:35
slaweqmonday and tuesday I will be still here so I will help You with that14:35
slaweqany other bugs You want to discuss today?14:36
mlavalleslaweq: I can help triaging bugs next week14:36
slaweqthx mlavalle14:36
ralonsohif we have time at the end of this meeting, I would like to talk about the live-migration issue14:36
ralonsoh(at the end)14:36
mlavalleslaweq: or this week if hongbin doesn't respond. let me know14:37
slaweqmlavalle: sure, thx a lot14:37
mlavalleor both14:37
slaweq:)14:37
*** elvira has joined #openstack-meeting-314:39
*** ralonsoh has quit IRC14:40
slaweqok, it seems that there are no other bugs to discuss today14:40
*** ralonsoh has joined #openstack-meeting-314:41
slaweqso ralonsoh lets go with live-migration issue14:41
ralonsohhttps://bugs.launchpad.net/neutron/+bug/1901707/comments/314:41
openstackLaunchpad bug 1901707 in neutron "race condition on port binding vs instance being resumed for live-migrations" [Undecided,In progress] - Assigned to Rodolfo Alonso (rodolfo-alonso-hernandez)14:41
ralonsohlive-migration is working now by mistake14:41
ralonsohwe are sending too many vif-plugged events to Nova during the live-migration14:42
ralonsohNova is expecting only one: when the destination host port is bound14:42
ralonsohthe problem in Nova is that this port is created by libvirt when the VM is unpaused14:42
ralonsohso Sean Mooney and me are pushing some changes to14:43
ralonsoh1) avoid unneeded vif-plugged events during live migration14:43
ralonsoh2) allow libvirt to create the port before unpausing the VM14:43
ralonsoh3) allow to bind the port in OVS with ofport -1 (merged)14:43
ralonsohwe found today the problem with libvirt in the CI jobs14:44
ralonsohthe main problem with this LP bug is that, if thee OVS agent is loaded, the OF rules will be set too late and we'll have packets lost14:44
ralonsoh(this is the main summary)14:44
ralonsohand finally, this is the Neutron patch: https://review.opendev.org/c/openstack/neutron/+/76627714:45
ralonsohI'll comment in the LP the Nova ones14:45
ralonsoh(that's all)14:45
slaweqralonsoh: first of all thx for finally cleaning this mess with notifications between nova and neutron :)14:45
ralonsohhehehhhe no easy!14:45
slaweqand please also be sure that those changes will not break ovn - I remember that maciek was doing some hacks to make it working with ML2/ovn some time ago14:46
ralonsohfor sure, I'm aware of this14:46
ralonsohand checking that in a testing env14:46
slaweqralonsoh: please remember that tempest-slow-ovn job is broken now14:46
slaweqand I think that live-migration tests are run in that job14:47
slaweqso You may miss problems with ML2/ovn14:47
ralonsohI'll make the test manually14:47
ralonsoh(make... execute)14:47
slaweqthx14:48
slaweqok, so I think we are good for today14:49
slaweqthx for attending the meeting14:49
hemanth_nhi, i'm looking for reviews on https://review.opendev.org/c/openstack/neutron/+/75224814:49
slaweqhemanth_n: You have my +2 already14:50
hemanth_nslaweq: yes, some more final reviews and WF +114:50
slaweqok, I hope ralonsoh  or mlavalle will find some time to check it :)14:50
ralonsohsure14:50
slaweqor haleyb :)14:50
hemanth_nack, thank you14:51
slaweqok, have a great holidays everyone14:51
njohnstono/14:51
slaweqand see You all on the meetings in 2021 :)14:51
slaweqo/14:52
hemanth_no/14:52
slaweq#endmeeting14:52
bcafarelhave great holidays folks14:52
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"14:52
rubasovo/14:52
openstackMeeting ended Tue Dec 15 14:52:06 2020 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)14:52
openstackMinutes:        http://eavesdrop.openstack.org/meetings/networking/2020/networking.2020-12-15-14.00.html14:52
ralonsohbye14:52
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/networking/2020/networking.2020-12-15-14.00.txt14:52
openstackLog:            http://eavesdrop.openstack.org/meetings/networking/2020/networking.2020-12-15-14.00.log.html14:52
mlavalleo/14:52
amotokio/14:53
*** hemanth_n has quit IRC14:58
slaweq#startmeeting neutron_ci15:01
openstackMeeting started Tue Dec 15 15:01:07 2020 UTC and is due to finish in 60 minutes.  The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot.15:01
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:01
*** openstack changes topic to " (Meeting topic: neutron_ci)"15:01
slaweqhi again :)15:01
openstackThe meeting name has been set to 'neutron_ci'15:01
ralonsohhi15:01
bcafarelnot holidays yet, hi again :)15:01
slaweqbcafarel: unfortunatelly not :/15:01
slaweqbut it's close15:01
slaweq:)15:01
bcafareljust a few rechecks away15:02
slaweqlajoskatona will not be here today15:02
slaweqso lets start15:02
slaweqwe don't have any actions from last week15:02
slaweqahh, sorry15:02
slaweqGrafana dashboard: http://grafana.openstack.org/dashboard/db/neutron-failure-rate15:02
slaweq#topic Stadium projects15:02
*** openstack changes topic to "Stadium projects (Meeting topic: neutron_ci)"15:02
slaweqanything related to stadium projects for today?15:03
ralonsohno15:03
bcafarelnothing from me15:03
slawequsually lajos got some updates but when I was reviewing some patches I didn't saw anything urgent there15:04
slaweq#topic Stable branches15:04
*** openstack changes topic to "Stable branches (Meeting topic: neutron_ci)"15:04
slaweqbcafarel: we already spoke about pip and l-c issues15:04
bcafarelslaweq: yes, busy week, and not in a good way :(15:04
slaweqIIUC You have patches which should unblock our stable branches, right?15:05
bcafarelyes, overall most branches are probably not passing CI right now but hopefully this will be better in 202115:05
slaweqLOL15:05
bcafarelI will move them under same gerrit topic so we can track them more easily15:05
bcafarelalso there was a separate bandit issue (workarounded in devstack), and apparently rocky broken https://review.opendev.org/c/openstack/stackviz/+/76706315:06
*** lpetrut has quit IRC15:07
slaweqk15:07
bcafarelhttps://review.opendev.org/q/topic:fix-pip+project:openstack/neutron for pip fixes15:08
slaweqso I think we will be good with those as bcafarel is on it already :)15:08
slaweqthx bcafarel15:08
slaweqanything else or we can move on?15:08
bcafarelnothing else, hopefully we can merge this "Upgrade RPC version of SecurityGroup*Rpc" soon after CI is green again15:10
slaweqyes, I proposed backports of that patch already15:10
slaweqok, lets move on15:11
slaweq#topic Grafana15:11
*** openstack changes topic to "Grafana (Meeting topic: neutron_ci)"15:11
slaweqhttp://grafana.openstack.org/dashboard/db/neutron-failure-rate15:11
slaweqAs many of my recent changes to our jobs are now merged, I think I need to update grafana dashboard config15:12
slaweq#action slaweq to update grafana dashboard15:12
*** eolivare has joined #openstack-meeting-315:12
bcafarelwell removed jobs show at 0% failure right? so it makes nicer dashboards :)15:13
slaweqbcafarel: true15:13
slaweqbut I don't think that it's what we really want :P15:13
slaweqin general I think that it looks better this week in master branch15:15
slaweqwe recently merged some fixes for functional and fullstack tests15:15
slaweqso I hope it will be better with those jobs too15:15
slaweqscenario jobs are pretty ok now15:15
bcafarelnice15:15
slaweqok, lets talk about specific jobs15:17
slaweq#topic fullstack/functional15:17
*** openstack changes topic to "fullstack/functional (Meeting topic: neutron_ci)"15:17
slaweqwe have opened bug     https://bugs.launchpad.net/neutron/+bug/190300815:17
openstackLaunchpad bug 1903008 in neutron "Create network failed during functional test" [Critical,Confirmed]15:17
slaweqand I saw at list 2 or 3 times this week that it happened again15:17
slaweq    https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_78e/764913/2/check/neutron-functional-with-uwsgi/78eb5fc/testr_results.html15:18
slaweq    https://zuul.opendev.org/t/openstack/build/a24070f28271446090761135b83812b715:18
slaweqwe need to investigate that issue as it happens on various random tests15:18
ralonsoh-->15:18
ralonsohhttps://review.opendev.org/c/openstack/neutron/+/76692015:18
slaweqso there is no way to easy mark some test(s) as unstable15:18
ralonsohI though that was the problem15:18
ralonsohbut I really don't know what is not in this trasaction: the port register or the standard attrs register15:19
slaweqthis patch didn't solve it - still same failure 3 times :/15:19
ralonsohyes...15:19
slaweqok, I will try to look at that issue and maybe I will add some logs or things like that to investigate it more15:20
slaweqI will use Your patch ralonsoh as a base15:20
ralonsohperfect15:20
slaweqto run many functional tests jobs in the same time15:20
slaweqI also saw once some db migration test failure https://81b3bd201171c89feee7-a605dbf95134d478b50bec2dfa092555.ssl.cf2.rackcdn.com/766875/2/check/neutron-functional-with-uwsgi/146dc99/testr_results.html15:21
slaweqat first glance it looks for me like maybe mysql was killed by oom15:22
slaweqbut it isn't hitting us very often so I will just report it to not forget15:24
slaweqand we can get back to it when we will fix more urgent things15:25
slaweqare You ok with that?15:25
ralonsohperfect15:25
slaweqok, lets move on15:26
slaweq#topic Tempest/Scenario15:26
*** openstack changes topic to "Tempest/Scenario (Meeting topic: neutron_ci)"15:26
slaweqhere I still see those failures in neutron-ovn-tempest-ovs-release-ipv6-only15:26
slaweqfor now I proposed patch to skip those tests in that job15:26
slaweqhttps://review.opendev.org/c/openstack/neutron/+/76507015:26
slaweqand I'm also working on reproduce of that issue locally15:26
slaweqso hopefully I will find something soon15:27
slaweqand that's all from my side for scenario jobs15:27
slaweqanything else You want to add?15:27
ralonsohno15:28
slaweqok, so I have one more thing only15:29
slaweq#topic grenade15:29
*** openstack changes topic to "grenade (Meeting topic: neutron_ci)"15:29
bcafarelsmall topic15:29
slaweqI recently pushed patch https://review.opendev.org/c/openstack/neutron/+/76676015:29
slaweqit is some short guide about how to debug grenade jobs15:29
slaweqthx haleyb  and njohnston for review of it15:29
slaweqI will address those comments soon15:29
slaweqand please take a look then once again15:30
slaweqthat's all from my side15:30
slaweqbcafarel: go on15:30
slaweqwith Your topic :)15:30
bcafarelslaweq: no, nothing from me , I was just commenting that grenade usually is "small" topic :)15:30
slaweqahh :)15:30
slaweqok15:31
slaweqso that's all from my side15:31
slaweqI will cancel next 2 meetings - are You ok with that?15:31
bcafarelsame +1 as team meeting yes15:31
slaweq:)15:31
slaweqso, once again, have a great holidays and see You all on the next ci meeting in 2021 :)15:32
ralonsohyou too!15:32
slaweqthx for attending today15:32
slaweq#endmeeting15:32
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"15:32
slaweqo/15:32
openstackMeeting ended Tue Dec 15 15:32:42 2020 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)15:32
openstackMinutes:        http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-12-15-15.01.html15:32
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-12-15-15.01.txt15:32
ralonsohbye15:32
openstackLog:            http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-12-15-15.01.log.html15:32
bcafarelo/15:32
*** markmcclain has joined #openstack-meeting-315:38
*** elvira has left #openstack-meeting-315:49
*** ralonsoh has quit IRC15:58
*** ralonsoh has joined #openstack-meeting-315:59
*** yamamoto has joined #openstack-meeting-316:05
*** ralonsoh has quit IRC16:15
*** ralonsoh has joined #openstack-meeting-316:15
*** ralonsoh has quit IRC16:23
*** yamamoto has quit IRC16:24
*** ralonsoh has joined #openstack-meeting-316:29
*** e0ne has quit IRC17:11
*** yamamoto has joined #openstack-meeting-318:24
*** yamamoto has quit IRC18:37
*** e0ne has joined #openstack-meeting-318:48
*** artom has quit IRC19:25
*** imtiazc has joined #openstack-meeting-319:42
*** e0ne has quit IRC20:22
*** yamamoto has joined #openstack-meeting-320:36
*** artom has joined #openstack-meeting-320:42
*** e0ne has joined #openstack-meeting-321:02
*** raildo has quit IRC21:03
*** yamamoto has quit IRC21:03
*** e0ne has quit IRC21:03
*** trident has quit IRC21:56
*** trident has joined #openstack-meeting-322:00
*** yamamoto has joined #openstack-meeting-322:01
*** yamamoto has quit IRC22:28
*** yamamoto has joined #openstack-meeting-322:28
*** eolivare has quit IRC22:42
*** macz_ has joined #openstack-meeting-323:09
*** slaweq has quit IRC23:32
*** artom has quit IRC23:47
*** artom has joined #openstack-meeting-323:48

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!