Friday, 2020-10-09

« Thursday, 2020-10-08 Index Saturday, 2020-10-10 »
*** spatel has joined #openstack-neutron00:23
*** spatel has quit IRC00:28
*** hamalq has quit IRC00:32
*** JamesBenson has quit IRC00:42
*** JamesBenson has joined #openstack-neutron00:43
*** JamesBenson has quit IRC00:47
*** zzzeek has quit IRC01:10
*** zzzeek has joined #openstack-neutron01:11
*** JamesBenson has joined #openstack-neutron01:20
openstackgerritMerged openstack/neutron master: Fix formatting error in agent/linux/external_process.py  https://review.opendev.org/75661001:23
openstackgerritMerged openstack/neutron master: [L3] Let agent extension do delete router first  https://review.opendev.org/73625801:23
*** JamesBenson has quit IRC01:24
*** armax has joined #openstack-neutron01:29
*** armax has quit IRC02:00
*** JamesBenson has joined #openstack-neutron02:30
*** JamesBenson has quit IRC02:34
*** JamesBenson has joined #openstack-neutron03:05
*** armax has joined #openstack-neutron03:07
*** JamesBenson has quit IRC03:10
*** rcernin has quit IRC03:17
*** macz_ has joined #openstack-neutron03:22
*** macz_ has quit IRC03:26
*** armax has quit IRC03:40
*** rcernin has joined #openstack-neutron03:55
*** rcernin has quit IRC03:56
*** lajoskatona has joined #openstack-neutron03:58
*** bengates has joined #openstack-neutron04:01
*** rcernin has joined #openstack-neutron04:03
*** bengates has quit IRC04:05
*** pmannidi has quit IRC04:16
*** mpsairam has joined #openstack-neutron04:16
*** zzzeek has quit IRC05:02
*** zzzeek has joined #openstack-neutron05:05
*** ratailor has joined #openstack-neutron05:07
*** zzzeek has quit IRC05:42
*** zzzeek has joined #openstack-neutron05:43
*** rpittau|afk is now known as rpittau05:43
*** zzzeek has quit IRC06:11
*** zzzeek has joined #openstack-neutron06:12
*** gcheresh has joined #openstack-neutron06:31
*** tkajinam has quit IRC06:42
*** tkajinam has joined #openstack-neutron06:42
*** jobewan has quit IRC06:46
*** jlibosva has joined #openstack-neutron06:46
*** jobewan has joined #openstack-neutron06:48
*** tesseract has joined #openstack-neutron06:59
*** ralonsoh has joined #openstack-neutron06:59
*** tmorin has joined #openstack-neutron07:10
lajoskatonabcafarel: Hi/ Good morning! I'am struggling with networking-odl train jobs: https://review.opendev.org/74580607:15
lajoskatonabcafarel: perhaps you have better eye for what else to do with functional tests:07:16
lajoskatonabcafarel: example failure: https://6ce17512e47061b26988-2e52c217801cfd672d11b663a7c290aa.ssl.cf5.rackcdn.com/745806/16/check/networking-odl-functional-neon/684c0b1/job-output.txt07:17
lajoskatonabcafarel: [ERROR] /opt/stack/devstack/inc/python:41 Unable to find pip2.7; cannot continue07:17
bcafarellajoskatona: that looks similar to something I fixed in older neutron branches, looking for the patch07:21
lajoskatonabcafarel: thanks, would be good :-)07:22
bcafarellajoskatona: "ensure_pip_from_packages_with_python2" may help here07:24
bcafarelhttps://opendev.org/openstack/neutron/commit/57f527a748f292597ee548e8d386720b72346d17 was for rocky but from the variable name it looks similar :)07:24
*** armax has joined #openstack-neutron07:24
lajoskatonabcafarel: I check this magic word, thanks :-)07:25
*** bengates has joined #openstack-neutron07:26
lajoskatonabcafarel: on train, do we support python2 or 3?07:27
*** Inflatad_ has quit IRC07:30
openstackgerritLajos Katona proposed openstack/networking-odl stable/train: This change is a squash of 2 different patches to pass the gate  https://review.opendev.org/74580607:31
bcafarellajoskatona: both :) for functional I guess that depends on what was the status in train07:31
lajoskatonabcafarel: ok I go with the original USE_PYTHON3: True and see now if it works with ensure_pip_from_packages_with_python2 , 🤞07:32
bcafarelyes neutron has both functional and functional-python27 (jobs and tox targets) maybe networking-odl was still py2 by then?07:36
*** slaweq has joined #openstack-neutron07:37
*** slaweq has quit IRC07:37
bcafarelI'll keep an eye on the ci results :)07:37
*** slaweq has joined #openstack-neutron07:38
*** rcernin has quit IRC07:45
*** sapd1 has joined #openstack-neutron07:50
lajoskatonabcafarel: thanks, to tell the truth I don't know, but if it fails I have to change to my archeologist hat....07:50
bcafarellajoskatona: watch out, try not to get crushed by the giant python2.7 boulder!07:54
bcafarelbut yes I hope ensure-pip and some of its options should do the trick (may need some attemps though)07:55
*** jpena|off is now known as jpena08:01
*** gcheresh has quit IRC08:04
*** Adri2000 has joined #openstack-neutron08:07
*** lucasagomes has joined #openstack-neutron08:10
openstackgerritRodolfo Alonso Hernandez proposed openstack/neutron master: [WIP] Register a known agent in TestAgentApi  https://review.opendev.org/75666808:14
*** gcheresh has joined #openstack-neutron08:14
Adri2000hello08:22
Adri2000I've got a "NetlinkError: (13, 'Permission denied')" error in neutron-linuxbridge-agent - actually the problem itself seems to happen in oslo.privsep.daemon (called by neutron-linuxbridge-agent). this is running Stein, and the error started appearing in the logs just after upgrading from Rocky; it's not just once, the agent keeps reporting that in the logs ("Error in agent loop").08:24
Adri2000disabling IPv6 on the host seems to workaround the problem, but I'd rather find a proper fix. any idea?08:24
slaweqAdri2000: hi, can You try if same issue happens on e.g. master branch? and also can You report Launchpad bug for that, and attach Your logs from agent there08:31
openstackgerritOpenStack Proposal Bot proposed openstack/neutron-vpnaas-dashboard stable/victoria: Imported Translations from Zanata  https://review.opendev.org/75600408:37
openstackgerritLajos Katona proposed openstack/neutron master: fullstack: Fetch created subnet in loop  https://review.opendev.org/74904108:38
*** ccamposr has joined #openstack-neutron08:40
*** ccamposr__ has quit IRC08:43
*** dtantsur|afk is now known as dtantsur08:44
*** ociuhandu has joined #openstack-neutron08:51
openstackgerritRodolfo Alonso Hernandez proposed openstack/neutron master: Add port NUMA affinity policy  https://review.opendev.org/74006708:52
*** tmorin has quit IRC08:59
Adri2000slaweq: not sure about testing master, it's an openstack-ansible pre-prod deployment, it might be hard to upgrade only neutron and keep the rest of the deployment in stein :/ I can open a bug report though and attach logs, if that helps09:02
jlibosvaslaweq: hi, are we still allowed to request new tags for stable Queens after it's been marked as EM ?09:08
slaweqjlibosva: nope, we are not creating any new tags for EM branches09:13
slaweqwe can backport things there but no new releases09:13
Adri2000slaweq: just filed a bug a report, with log attached: https://bugs.launchpad.net/neutron/+bug/189914109:13
openstackLaunchpad bug 1899141 in neutron "Linuxbridge agent NetlinkError: (13, 'Permission denied') after Stein upgrade" [Undecided,New]09:13
jlibosvaslaweq: gotcha09:13
openstackgerritSlawek Kaplonski proposed openstack/neutron stable/ussuri: Support gateway which is not in subnet CIDR in ha_router  https://review.opendev.org/75704409:35
openstackgerritSlawek Kaplonski proposed openstack/neutron stable/train: Support gateway which is not in subnet CIDR in ha_router  https://review.opendev.org/75704509:36
openstackgerritSlawek Kaplonski proposed openstack/neutron stable/stein: Support gateway which is not in subnet CIDR in ha_router  https://review.opendev.org/75704609:36
*** takamatsu has joined #openstack-neutron09:37
openstackgerritSlawek Kaplonski proposed openstack/neutron stable/rocky: Support gateway which is not in subnet CIDR in ha_router  https://review.opendev.org/75704809:38
openstackgerritSlawek Kaplonski proposed openstack/neutron stable/queens: Support gateway which is not in subnet CIDR in ha_router  https://review.opendev.org/75704909:39
openstackgerritSlawek Kaplonski proposed openstack/neutron stable/ussuri: windows: fix terminating processes  https://review.opendev.org/75705009:40
openstackgerritSlawek Kaplonski proposed openstack/neutron stable/train: windows: fix terminating processes  https://review.opendev.org/75705109:40
openstackgerritSlawek Kaplonski proposed openstack/neutron stable/stein: windows: fix terminating processes  https://review.opendev.org/75705209:41
openstackgerritSlawek Kaplonski proposed openstack/neutron stable/rocky: windows: fix terminating processes  https://review.opendev.org/75705409:41
openstackgerritSlawek Kaplonski proposed openstack/neutron stable/queens: windows: fix terminating processes  https://review.opendev.org/75705509:41
openstackgerritSlawek Kaplonski proposed openstack/neutron stable/ussuri: Import "oslo_config.cfg" before "eventlet"  https://review.opendev.org/75705909:53
openstackgerritSlawek Kaplonski proposed openstack/neutron stable/train: Import "oslo_config.cfg" before "eventlet"  https://review.opendev.org/75706009:54
openstackgerritSlawek Kaplonski proposed openstack/neutron stable/stein: Import "oslo_config.cfg" before "eventlet"  https://review.opendev.org/75706109:54
openstackgerritSlawek Kaplonski proposed openstack/neutron stable/rocky: Import "oslo_config.cfg" before "eventlet"  https://review.opendev.org/75706209:56
Adri2000ralonsoh: hello, thanks for your reply on my bug report, it helped a lot. I'll put all the infos in there but thought it'd be faster to discuss on IRC first. indeed after adding some debug I managed to reproduce the problem using ip addr dev ... - what neutron is trying to do is to add an ipv6 address on one of the br bridges. I believe the reason it fails is because the sysctl09:56
Adri2000net.ipv6.conf.<bridge>.disable_ipv6 is set to 1...09:56
openstackgerritSlawek Kaplonski proposed openstack/neutron stable/queens: Import "oslo_config.cfg" before "eventlet"  https://review.opendev.org/75706309:56
*** lucasagomes has quit IRC09:56
ralonsohAdri2000, what bridge is that?09:56
ralonsohAdri2000, https://github.com/openstack/os-vif/blob/master/vif_plug_linux_bridge/linux_net.py#L10009:57
*** lucasagomes has joined #openstack-neutron09:57
ralonsohos-vif is in charge of creating the L1 infrastructure09:58
*** lucasagomes has joined #openstack-neutron09:58
Adri2000it's the bridge created by neutron for our external/provider network09:58
Adri2000yes so... I'm a bit lost why neutron disables ipv6 on a bridge, then why it'd try to add an ipv6 on that same bridge?09:59
ralonsohcheck this https://github.com/openstack/os-vif/blob/master/vif_plug_linux_bridge/linux_net.py#L100-L11209:59
Adri2000ralonsoh: it looks like the ipv6 address it's trying to add comes from the physical NIC - like it's moving the IP from the physical NIC to the bridge when adding the physical NIC into the bridge, I guess10:05
Adri2000it's an ipv6 link local address. we don't make use of ipv6 actually10:05
ralonsohAdri2000, can you point me to the code in the LB agent when this is happening?10:06
Adri2000but I imagine that's why, when we disable ipv6 at the system level (via sysctl), that ipv6 goes away, and therefore neutron doesn't try to move it and add it to the bridge (bridge where neutron had disabled ipv6 in the past)10:06
Adri2000exactly what you linked in the bug report: https://github.com/openstack/neutron/blob/b63d1d5a5125f1110fb5b257aa9f1f14d0787864/neutron/privileged/agent/linux/ip_lib.py#L302-L317 I added debug log just before the try:10:07
ralonsohAdri2000, no, the code from the LB that is calling this method10:07
ralonsohthis is the exception in the privsep daemon10:08
ralonsohbut I need to know where this is called from the LB agent10:08
Adri2000https://launchpadlibrarian.net/501314638/neutron-linuxbridge-agent.log what about the second stack trace there? does it show the code path inside the LB agent?10:09
Adri2000if not, let me know how I can get the info you need pease :)10:10
Adri2000please*10:10
ralonsohI think there is a clash between the code in os-vif and Ic236de04c0203633df49967a9a4528fda13c51df10:12
ralonsohI'll that10:12
ralonsohI'll check that10:12
ralonsohAdri2000, you said the NIC IPv6 address is link-local10:22
ralonsohright?10:22
ralonsohbecause in "update_interface_ip_details", "get_interface_details" only gets scope='global'10:23
Adri2000ralonsoh: yes, example: inet6 fe80::f080:38ff:febc:645c/64 scope link10:28
ralonsohAdri2000, and is this the IP that LB agent tries to set??10:29
Adri2000yes. here is the infos I get with the LOG I added in the source: add device brqe59f512b-b9; namespace None; ip fe80::f080:38ff:febc:645c; prefixlen 64; family 10; broadcast None add_ip_address10:32
Adri2000in this case, the bridge also corresponds to an external/provider network, and that ipv6 local ip address is currently on the physical interface that neutron is trying to put inside the bridge10:33
ralonsohAdri2000, it doesn't make sense...10:33
ralonsohget_interface_details should not return this IP10:33
Adri2000I can try to put some LOG debug in that method?10:33
ralonsohI'm trying to execute this manually on a interface with IPv6 link-local address10:34
ralonsohAdri2000, can you open a python terminal?10:37
Adri2000ralonsoh: yes I can, just let me share what I've just got10:37
Adri2000I put a LOG.debug in get_interface_details(), to output the value of ips10:38
ralonsohAdri2000, http://paste.openstack.org/show/798903/10:38
ralonsohcan you execute this?10:38
Adri2000sometimes it's an empty list10:38
Adri2000sometimes it's ips [{'dadfailed': False, 'name': 'eth14', 'tentative': False, 'dynamic': False, 'broadcast': None, 'scope': 'link', 'cidr': 'fe80::f080:38ff:febc:645c/64'}] get_interface_details /openstack/venvs/neutron-19.1.5/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py:27710:38
Adri2000yes give me one min I'll run that10:39
Adri2000ralonsoh: http://paste.openstack.org/show/798904/10:41
Adri2000indeed it's as if scope is ignored or ...?10:42
ralonsohAdri2000, ok, I think this could be an error in pyroute10:42
ralonsohwhat version do you have?10:43
*** tbachman has quit IRC10:43
ralonsohpython -m pip freeze | grep pyroute10:43
Adri2000ralonsoh:pyroute2==0.5.410:44
ralonsohAdri2000, please, try a higher version, like 0.5.710:46
ralonsohI think this is a problem in pyroute10:46
ralonsohI have 0.5.14 and the filter works fine10:46
Adri2000ralonsoh: sure, I'll try that in a moment and let you know today. thank you very much for your help so far10:47
ralonsohyw10:47
*** priteau has joined #openstack-neutron10:50
*** rcernin has joined #openstack-neutron10:50
*** zzzeek has quit IRC10:52
*** zzzeek has joined #openstack-neutron10:54
*** tamas_erdei has quit IRC11:05
*** terdei has joined #openstack-neutron11:06
*** zzzeek has quit IRC11:17
*** zzzeek has joined #openstack-neutron11:18
*** rcernin has quit IRC11:31
*** ccamposr__ has joined #openstack-neutron11:33
*** ratailor_ has joined #openstack-neutron11:33
*** ccamposr has quit IRC11:34
*** ratailor has quit IRC11:34
*** jamesdenton has quit IRC11:34
*** jpena is now known as jpena|lunch11:34
*** jamesdenton has joined #openstack-neutron11:40
openstackgerritRoman Safronov proposed openstack/neutron master: Set proper bridge mappings before migration to OVN  https://review.opendev.org/75671411:48
*** jlibosva has quit IRC11:58
*** sapd1 has quit IRC12:01
*** slaweq has quit IRC12:03
*** slaweq has joined #openstack-neutron12:17
*** jpena|lunch is now known as jpena12:32
openstackgerritLajos Katona proposed openstack/networking-odl stable/train: This change is a squash of 2 different patches to pass the gate  https://review.opendev.org/74580612:34
Adri2000ralonsoh: unfortunately the pyroute2 version doesn't change anything, I tried all of them up until 0.5.14... but! the good news if after some more digging I found the culprit12:51
Adri2000ralonsoh: this patch https://review.opendev.org/#/c/747628/12:51
Adri2000ralonsoh: reverting this patch: 1) your test case in a python shell works as expected and 2) my neutron-linuxbridge-agent doesn't complain anymore in the logs12:52
*** jlibosva has joined #openstack-neutron12:53
ralonsohAdri2000, I need to review the code but I don't know where this method is called form the LB agent12:53
Adri2000ralonsoh: the list() method used in your test case calls get_devices_with_ip() here https://github.com/openstack/neutron/blob/stable/stein/neutron/agent/linux/ip_lib.py#L543-L54412:58
Adri2000I'll let you have a look :)12:59
Adri2000with the problematic patch, we end ip with that same code here https://github.com/openstack/neutron/blob/stable/stein/neutron/agent/linux/ip_lib.py#L537-L538 and there https://github.com/openstack/neutron/blob/stable/stein/neutron/agent/linux/ip_lib.py#L1426-L142713:00
Adri2000and looks like there is a code path that goes through both of them, which may cause a problem?13:01
Adri2000actually slaweq you might be interested in this as well, as I see you are the author of the commit13:02
*** rpittau is now known as rpittau|afk13:03
ralonsohAdri2000, the problem of this patch is that we are translating in the other way around13:03
ralonsohhttps://review.opendev.org/#/c/747628/1/neutron/agent/linux/ip_lib.py13:03
*** ratailor_ has quit IRC13:03
ralonsohno no13:03
*** ratailor has joined #openstack-neutron13:03
ralonsohI'm wrong13:03
ralonsohit's ok this patch13:04
*** nweinber has joined #openstack-neutron13:11
ralonsohcould no be the culprit13:12
ralonsohwhen I call "get_interface_details" I get the correct values13:12
ralonsoheven with the filter13:12
ralonsohOk, now I see the error (my dev env was a bit old and didn't have this patch)13:14
*** ratailor has quit IRC13:15
*** mmethot_ is now known as mmethot13:15
ralonsohAdri2000, I'll push a patch today13:16
Adri2000ralonsoh: great! thanks13:25
openstackgerritSlawek Kaplonski proposed openstack/neutron master: Add locks for methods which sets nat rules in router  https://review.opendev.org/75610713:49
openstackgerritSlawek Kaplonski proposed openstack/neutron stable/rocky: Drop invalid rootwrap filters  https://review.opendev.org/75710513:53
openstackgerritSlawek Kaplonski proposed openstack/neutron stable/queens: Drop invalid rootwrap filters  https://review.opendev.org/75710613:55
openstackgerritRodolfo Alonso Hernandez proposed openstack/neutron master: "scope" conversion to pyroute2 parameter only in last method  https://review.opendev.org/75710713:57
*** ociuhandu has quit IRC14:11
*** ociuhandu has joined #openstack-neutron14:12
openstackgerritMerged openstack/neutron master: [Doc] Add section about diffs between ovs and iptables fw drivers  https://review.opendev.org/75576614:14
*** slaweq has quit IRC14:42
*** jlibosva has quit IRC14:55
*** mlavalle has joined #openstack-neutron14:57
*** lpetrut has joined #openstack-neutron15:19
*** blkmanta has joined #openstack-neutron15:26
openstackgerritFlavio Fernandes proposed openstack/neutron stable/train: [DNM] [ovn] port forwarding support  https://review.opendev.org/75713415:31
*** jangutter_ has joined #openstack-neutron15:35
*** bengates has quit IRC15:35
*** lpetrut has quit IRC15:35
*** bengates has joined #openstack-neutron15:36
*** jangutter has quit IRC15:39
openstackgerritFlavio Fernandes proposed openstack/networking-ovn stable/train: [DNM] [ovn] port forwarding support  https://review.opendev.org/75713915:40
*** bengates has quit IRC15:41
*** macz_ has joined #openstack-neutron15:41
*** sapd1 has joined #openstack-neutron15:42
*** lucasagomes has quit IRC15:56
*** tesseract has quit IRC16:01
*** blkmanta has quit IRC16:03
*** priteau has quit IRC16:06
*** ociuhandu has quit IRC16:21
*** dtantsur is now known as dtantsur|afk16:26
*** raildo has quit IRC16:30
*** raildo has joined #openstack-neutron16:30
openstackgerritRodolfo Alonso Hernandez proposed openstack/neutron master: Use new engine facade in L3 DB code  https://review.opendev.org/71531516:42
*** priteau has joined #openstack-neutron16:42
*** hamalq has joined #openstack-neutron16:45
*** ViniciusCoelho has joined #openstack-neutron16:46
*** jpena is now known as jpena|off16:55
*** ccamposr has joined #openstack-neutron17:05
*** ccamposr__ has quit IRC17:07
*** sapd1 has quit IRC17:13
*** hack-char has quit IRC17:50
*** hack-char has joined #openstack-neutron17:50
*** priteau has quit IRC17:58
*** lajoskatona has left #openstack-neutron18:01
*** ralonsoh has quit IRC18:10
openstackgerritLajos Katona proposed openstack/networking-bagpipe master: Make ovs evpn driver an os-ken app  https://review.opendev.org/73593418:17
*** tkajinam has quit IRC18:25
*** armax has quit IRC18:25
openstackgerritLajos Katona proposed openstack/networking-bagpipe master: Make ovs evpn driver an os-ken app  https://review.opendev.org/73593418:28
*** armax has joined #openstack-neutron18:30
openstackgerritBrian Haley proposed openstack/ovn-octavia-provider stable/ussuri: Fix the check for allowed_cidrs in listeners  https://review.opendev.org/75717018:36
*** priteau has joined #openstack-neutron18:42
*** armax has quit IRC18:45
*** priteau has quit IRC18:48
*** priteau has joined #openstack-neutron19:14
*** ociuhandu has joined #openstack-neutron19:24
openstackgerritNate Johnston proposed openstack/neutron-dynamic-routing master: Bump SQL Alchemy version to catch ambiguous join fix  https://review.opendev.org/75717819:25
*** priteau has quit IRC19:27
*** armax has joined #openstack-neutron19:42
*** zzzeek has quit IRC19:49
*** ociuhandu has quit IRC19:50
*** zzzeek has joined #openstack-neutron19:51
*** SmearedBeard has joined #openstack-neutron20:26
*** nweinber has quit IRC20:37
*** ociuhandu has joined #openstack-neutron20:51
*** SmearedBeard has quit IRC21:06
*** ociuhandu has quit IRC21:13
*** ViniciusCoelho has quit IRC21:18
*** ociuhandu has joined #openstack-neutron21:18
*** ociuhandu has quit IRC21:25
*** raildo has quit IRC21:30
*** hamalq has quit IRC21:37
*** hamalq has joined #openstack-neutron22:00
*** gcheresh has quit IRC22:01
*** armax has quit IRC22:16
*** takamatsu has quit IRC22:26
*** macz_ has quit IRC23:06
*** mlavalle has quit IRC23:19
*** hamalq has quit IRC23:26
« Thursday, 2020-10-08 Index Saturday, 2020-10-10 »

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!