Wednesday, 2020-02-26

« Tuesday, 2020-02-25 Index Thursday, 2020-02-27 »
*** mattw4 has quit IRC00:03
*** ociuhandu has joined #openstack-neutron00:04
*** ccamposr has joined #openstack-neutron00:05
*** ociuhandu has quit IRC00:09
*** nicolasbock has quit IRC00:18
*** nweinber has quit IRC00:33
*** takamatsu has quit IRC00:35
*** macz_ has quit IRC00:36
*** sean-k-mooney has joined #openstack-neutron00:39
*** tbachman has joined #openstack-neutron00:43
*** takamatsu has joined #openstack-neutron00:52
*** sean-k-mooney has quit IRC00:59
*** ileixe has joined #openstack-neutron01:09
openstackgerritMerged openstack/neutron stable/rocky: ovs agent: signal to plugin if tunnel refresh needed  https://review.opendev.org/70944501:35
openstackgerritMerged openstack/neutron stable/queens: ovs agent: signal to plugin if tunnel refresh needed  https://review.opendev.org/70944601:35
*** yamamoto has joined #openstack-neutron01:47
*** ociuhandu has joined #openstack-neutron02:06
*** yamamoto has quit IRC02:07
*** yamamoto has joined #openstack-neutron02:10
*** ociuhandu has quit IRC02:11
*** ijw has quit IRC02:23
*** ileixe has quit IRC02:46
openstackgerritMerged openstack/neutron-vpnaas master: Deprecate ``ovs_integration_bridge``  https://review.opendev.org/69923302:47
openstackgerritDongcan Ye proposed openstack/neutron-vpnaas master: Run neutron-vpn-netns-wrapper in venv  https://review.opendev.org/68753802:52
*** Nel1x has quit IRC03:04
*** ijw has joined #openstack-neutron03:07
*** nweinber has joined #openstack-neutron03:08
*** zhanglong has joined #openstack-neutron03:13
*** nweinber has quit IRC03:17
*** Nel1x has joined #openstack-neutron03:22
*** dsneddon has quit IRC03:27
*** zhanglong has quit IRC03:36
*** zhanglong has joined #openstack-neutron03:37
*** ijw has quit IRC03:42
*** hongbin has joined #openstack-neutron03:49
*** ileixe has joined #openstack-neutron03:53
*** chenhaw has joined #openstack-neutron03:55
*** chenhaw has quit IRC03:55
*** chenhaw has joined #openstack-neutron03:57
*** chenhaw has quit IRC03:58
*** dsneddon has joined #openstack-neutron03:58
*** nweinber has joined #openstack-neutron03:59
*** dsneddon has quit IRC04:08
*** nweinber has quit IRC04:09
*** yedongcan has quit IRC04:10
openstackgerritMerged openstack/neutron-vpnaas master: Run neutron-vpn-netns-wrapper in venv  https://review.opendev.org/68753804:22
openstackgerritMerged openstack/neutron-tempest-plugin master: Increase default security group quota up to 150  https://review.opendev.org/70947804:31
openstackgerritMerged openstack/neutron stable/queens: Do not initialize snat-ns twice  https://review.opendev.org/70940604:31
openstackgerritMerged openstack/neutron stable/rocky: Do not initialize snat-ns twice  https://review.opendev.org/70965004:36
*** dsneddon has joined #openstack-neutron04:37
*** hongbin has quit IRC04:39
*** dsneddon has quit IRC04:42
*** dsneddon has joined #openstack-neutron04:45
*** imacdonn has quit IRC04:47
*** imacdonn has joined #openstack-neutron04:47
*** Nel1x has quit IRC04:50
*** dsneddon has quit IRC04:50
*** zhanglong has quit IRC04:53
*** zhanglong has joined #openstack-neutron04:55
openstackgerritMerged openstack/neutron stable/stein: Do not initialize snat-ns twice  https://review.opendev.org/70964905:02
openstackgerritzhangyuhe proposed openstack/neutron master: create reserved dhcp port when subnet without gateway and dhcp is enabled  https://review.opendev.org/70801105:18
*** dsneddon has joined #openstack-neutron05:25
*** dsneddon has quit IRC05:31
*** ramishra has quit IRC05:42
*** ramishra has joined #openstack-neutron05:50
*** ramishra has quit IRC05:55
*** ramishra has joined #openstack-neutron05:56
*** zhanglong has quit IRC05:58
*** dsneddon has joined #openstack-neutron05:59
*** zhanglong has joined #openstack-neutron06:01
*** gcheresh has joined #openstack-neutron06:01
*** dsneddon has quit IRC06:04
*** ratailor has joined #openstack-neutron06:06
*** kozhukalov has joined #openstack-neutron06:08
*** ociuhandu has joined #openstack-neutron06:08
*** ociuhandu has quit IRC06:13
*** ratailor has quit IRC06:18
*** zhanglong has quit IRC06:20
*** ratailor has joined #openstack-neutron06:21
*** zhanglong has joined #openstack-neutron06:22
*** dsneddon has joined #openstack-neutron06:37
*** dsneddon has quit IRC06:42
*** rcernin has quit IRC06:47
*** ociuhandu has joined #openstack-neutron07:02
*** ociuhandu has quit IRC07:07
*** dsneddon has joined #openstack-neutron07:17
*** dsneddon has quit IRC07:22
*** ccamposr__ has joined #openstack-neutron07:26
*** ccamposr has quit IRC07:28
*** gcheresh_ has joined #openstack-neutron07:31
*** gcheresh has quit IRC07:33
*** lajoskatona has joined #openstack-neutron07:33
*** slaweq_ has joined #openstack-neutron07:42
*** ociuhandu has joined #openstack-neutron07:42
*** slaweq_ is now known as slaweq07:45
slaweqdansmith: hi, sorry for late answer but if You still need help with this issue, it seems for me that You don't have any interface with cfg.CONF.OVS.local_ip IP address configured07:48
*** ociuhandu has quit IRC07:48
slaweqI think we should handle that better and log something more nice rather than such traceback07:48
*** dsneddon has joined #openstack-neutron07:51
*** dsneddon has quit IRC07:56
*** ociuhandu has joined #openstack-neutron08:05
*** tesseract has joined #openstack-neutron08:06
*** tmorin has joined #openstack-neutron08:06
*** jawad_axd has joined #openstack-neutron08:07
*** ociuhandu has quit IRC08:11
*** gcheresh_ has quit IRC08:15
*** iurygregory has joined #openstack-neutron08:21
*** rcernin has joined #openstack-neutron08:22
*** maciejjozefczyk has joined #openstack-neutron08:22
*** ccamposr__ has quit IRC08:24
*** ccamposr__ has joined #openstack-neutron08:24
*** dsneddon has joined #openstack-neutron08:30
*** dsneddon has quit IRC08:34
*** jpena|off is now known as jpena08:46
*** ralonsoh has joined #openstack-neutron08:53
*** tkajinam has quit IRC08:57
*** dsneddon has joined #openstack-neutron09:00
ChosSimbaOne_Hi. anyone familiar with ERROR neutron     assert hub.greenlet is not current, 'do not call blocking functions from the mainloop'#0122020-02-26 09:29:52.624 15125 ERROR neutron AssertionError: do not call blocking functions from the mainloop#0122020-02-26 09:29:52.624 15125 ERROR neutron09:04
*** dsneddon has quit IRC09:05
*** lennyb has quit IRC09:06
openstackgerritmanjunathpayala proposed openstack/networking-odl master: BGPVPN missing networks in ODL  https://review.opendev.org/70998909:06
*** lennyb has joined #openstack-neutron09:07
*** ociuhandu has joined #openstack-neutron09:08
*** jlibosva has joined #openstack-neutron09:13
*** lucasagomes has joined #openstack-neutron09:15
*** ociuhandu has quit IRC09:22
openstackgerritTaoyunxiang proposed openstack/neutron master: [OVN] Make QoS policy which has multi-rules work  https://review.opendev.org/70858609:32
*** tesseract has quit IRC09:35
*** ccamposr has joined #openstack-neutron09:36
*** tesseract has joined #openstack-neutron09:36
*** dsneddon has joined #openstack-neutron09:37
*** ccamposr__ has quit IRC09:38
*** dsneddon has quit IRC09:42
*** ociuhandu has joined #openstack-neutron09:42
*** gcheresh_ has joined #openstack-neutron09:46
*** davidsha has joined #openstack-neutron09:50
*** jangutter has joined #openstack-neutron09:57
*** jangutter has joined #openstack-neutron09:57
*** jangutter has quit IRC09:58
*** jangutter has joined #openstack-neutron09:59
*** xakaitetoia1 has joined #openstack-neutron10:07
*** davidsha has quit IRC10:08
*** ociuhandu has quit IRC10:11
*** dsneddon has joined #openstack-neutron10:15
*** hjensas has quit IRC10:18
*** dsneddon has quit IRC10:20
*** ociuhandu has joined #openstack-neutron10:21
*** hjensas has joined #openstack-neutron10:21
*** ociuhandu has quit IRC10:31
*** sean-k-mooney has joined #openstack-neutron10:45
*** dsneddon has joined #openstack-neutron10:51
*** dsneddon has quit IRC10:56
*** priteau has joined #openstack-neutron11:01
*** ociuhandu has joined #openstack-neutron11:07
*** tesseract has quit IRC11:08
*** tesseract has joined #openstack-neutron11:11
*** ociuhandu has quit IRC11:12
*** dsneddon has joined #openstack-neutron11:23
*** dsneddon has quit IRC11:28
*** tmorin has quit IRC11:35
*** gcheresh_ has quit IRC11:45
*** gcheresh has joined #openstack-neutron11:45
*** yamamoto has quit IRC11:49
*** ociuhandu has joined #openstack-neutron11:51
CeeMacon a rocky setup, with OVS. Should one be able to see the 'qbr' bridges on the Compute Node that the instance tap port connects to before patching in to the intergration bridge?11:53
CeeMacI'm trying to track a packet journey to see where it is dropping but am not able to see any qbr bridges or qvb/qvo interfaces on the compute node ip link / bridge link or within the ovs-vsctl11:54
*** ociuhandu has quit IRC11:55
CeeMacbased on this: https://docs.openstack.org/neutron/rocky/admin/deploy-ovs-selfservice.html11:57
*** dsneddon has joined #openstack-neutron12:01
CeeMacalso, is the connection still patched into OVS via lxb if 'port security' is disabled on the port?12:01
*** nicolasbock has joined #openstack-neutron12:03
*** TristanSullivan has quit IRC12:03
*** dsneddon has quit IRC12:06
*** yamamoto has joined #openstack-neutron12:09
openstackgerritMaciej Józefczyk proposed openstack/neutron master: [DNM] Verify returned LSP value in functional tests  https://review.opendev.org/71001912:13
*** ratailor has quit IRC12:14
openstackgerritMaciej Józefczyk proposed openstack/neutron master: [DNM] Verify returned LSP value in functional tests  https://review.opendev.org/71001912:16
openstackgerritMaciej Józefczyk proposed openstack/neutron master: [DNM] Verify returned LSP value in functional tests  https://review.opendev.org/71001912:17
*** maciejjozefczyk has quit IRC12:23
*** ociuhandu has joined #openstack-neutron12:30
*** jpena is now known as jpena|lunch12:34
slaweqralonsoh: njohnston: hi, can You take a look at https://review.opendev.org/#/c/699213/ if You will have few minutes?12:35
*** dsneddon has joined #openstack-neutron12:35
slaweqI think it's the last one needed to finally release last version of neutron-tempest-plugin for rocky (with py2 support)12:35
*** dsneddon has quit IRC12:40
*** maciejjozefczyk has joined #openstack-neutron12:40
*** ociuhandu has quit IRC12:40
CeeMacslaweq: I don't suppose you have a few minutes free to assist with my query? ^^12:43
CeeMacI've hit a brick wall and I'm struggling how to proceed :(12:43
slaweqCeeMac: please ping me in about 45 minutes, I can try to help You12:51
*** yamamoto has quit IRC12:51
CeeMacperfect, thanks12:51
openstackgerritMerged openstack/networking-bagpipe master: Make releasenotes generation working again  https://review.opendev.org/70864712:51
openstackgerritMerged openstack/networking-bagpipe master: Remove bagpipe-bgp's deprecated action CLI option  https://review.opendev.org/70864812:59
*** jangutter has quit IRC13:01
*** jangutter_ has joined #openstack-neutron13:01
*** ociuhandu has joined #openstack-neutron13:06
ralonsohslaweq, yes, let me check13:08
*** dsneddon has joined #openstack-neutron13:08
*** tmorin has joined #openstack-neutron13:09
openstackgerritMaciej Józefczyk proposed openstack/neutron master: [DNM] Verify returned LSP value in functional tests  https://review.opendev.org/71001913:13
*** dsneddon has quit IRC13:13
openstackgerritMaciej Józefczyk proposed openstack/neutron master: [DNM] Verify returned LSP value in functional tests  https://review.opendev.org/71001913:14
openstackgerritMaciej Józefczyk proposed openstack/neutron master: [DNM] Verify returned LSP value in functional tests  https://review.opendev.org/71001913:22
*** jpena|lunch is now known as jpena13:26
*** nweinber has joined #openstack-neutron13:27
CeeMacslaweq: ping :)13:37
*** nicolasbock has quit IRC13:40
*** nicolasbock has joined #openstack-neutron13:40
*** tbachman has quit IRC13:42
*** nweinber has quit IRC13:46
*** dsneddon has joined #openstack-neutron13:46
*** nweinber has joined #openstack-neutron13:47
openstackgerritwaleed mousa proposed openstack/neutron master: Fix: Removing SmartNIC OVS representor port on instance tear down and resync  https://review.opendev.org/70280613:47
slaweqCeeMac: ok, I'm back now13:47
slaweqhow can I help You?13:47
CeeMachi13:47
CeeMacso, I have an instance which I am setting up to use as a FW appliance13:48
CeeMacas it stands currently, I can route out from instance on the LAN network of the FW and reach the internet fine and it picks up the floating IP attached to the WAN13:48
CeeMacI can ping out from the FW itself and again it picks up the floating IP13:49
openstackgerritwaleed mousa proposed openstack/neutron master: Fix: Removing SmartNIC OVS representor port on instance tear down and resync  https://review.opendev.org/70280613:49
CeeMacHowever, I'm trying to access a VM from the internet that is NATd behind the FW and replies are getting lost13:49
CeeMacthe request packet is traversing the FW and reaching the instance behind, the reply packet comes in the LAN interface, leaves the FW WAN interface then disappears13:50
CeeMacso, i've been trying to tcpdump the various interfaces along the way and the last place I see the reply packet is on the TAP interface of the VM on the compute node13:50
slaweqYou probably need to disable port_security on port which connects Your FW instance to the internet13:50
CeeMacalread y done that13:50
CeeMacon all of the FW interfaces13:50
CeeMacsame issue13:51
slaweqwhat fw driver are You using? openvswitch or iptables_hybrid?13:51
CeeMacgood question, let me check13:51
*** dsneddon has quit IRC13:52
*** jlibosva has quit IRC13:52
*** ileixe has quit IRC13:52
*** ileixe has joined #openstack-neutron13:53
CeeMacopenvswitch13:53
slaweqI'm not sure but that rings a bell for me that there was some issue with that13:53
slaweqcan You try to add IP/MAC of Your nat'ed VM to the allowed address pairs in FW's port?13:54
slaweqand try then13:54
CeeMacjamesdenton mentioned a bug you were working on related to address pairs and garp resolution13:54
CeeMacthought it might be linked13:54
CeeMaccan you add address pairs if port security is disabled?13:54
slaweqI'm not sure, but if not, please enable port_security and add it then13:55
slaweqjust to test if it will work13:55
CeeMacfyi the NAT address is the default port IP of the FW WAN interface, not a VIP13:55
CeeMacyou can add address-pairs through horizon or just through cli?13:55
CeeMacis there an openstack cli command for that now or still the neutron client one?13:56
slaweqI'm always using neutron client13:56
CeeMacok13:56
CeeMacso is the MAC of the VM behind the FW I add, or the MAC of the FW LAN interface?13:56
CeeMacfrom what I could see in the tcpdump, the source mac when it leaves the WAN interface is correct for the WAN interface13:57
slaweqNAT is done where? on FW vm?13:57
CeeMacyes13:57
slaweqso adding only IP address should be enough13:57
CeeMacso, neutron router does public > private NAT to WAN interface of FW VM, then FW VM does private > private NAT to instance on LAN network13:57
*** ileixe has quit IRC13:58
slaweqif that will not help, I think You will need to investigate where exactly packets are dropped and then probably report a bug agains neutron for that14:01
*** slaweq has quit IRC14:05
CeeMacok, will do14:05
CeeMacthanks14:05
CeeMacalthough thats probably going to require some help too :)14:05
*** zhanglong has quit IRC14:06
*** zhanglong has joined #openstack-neutron14:07
*** jlibosva has joined #openstack-neutron14:08
openstackgerritMerged openstack/neutron stable/rocky: Fix bug: AttributeError arises while sorting with standard attributes  https://review.opendev.org/70519914:08
openstackgerritLucas Alvares Gomes proposed openstack/neutron master: [OVN] DO NOT REVIEW: PoC Schema aware periodics  https://review.opendev.org/70977014:08
*** dasp has quit IRC14:08
*** jmlowe has quit IRC14:09
*** slaweq has joined #openstack-neutron14:09
*** jmlowe has joined #openstack-neutron14:09
*** zhanglong has quit IRC14:14
*** zhanglong has joined #openstack-neutron14:16
xakaitetoia1slaweq, you can add address pair through dashboard as well14:20
*** dsneddon has joined #openstack-neutron14:20
*** yamamoto has joined #openstack-neutron14:22
slaweqxakaitetoia1: thx, good to know that :)14:22
openstackgerritRoman Safronov proposed openstack/neutron-tempest-plugin master: Add VRRP failover test  https://review.opendev.org/70766014:24
*** dsneddon has quit IRC14:25
*** abdysn has joined #openstack-neutron14:28
*** abdysn has quit IRC14:28
*** jmlowe has quit IRC14:29
bcafarelslaweq: amotoki all rocky backports in, I'll send new releases request for those mentioned in https://review.opendev.org/#/c/709901/14:29
slaweqbcafarel: I just updated this release patch14:29
slaweqplease check if I didn't missed anything :)14:29
bcafarel:) we have the Flash PTL!14:30
slaweqbcafarel: LOL14:30
*** jmlowe has joined #openstack-neutron14:32
openstackgerritLucas Alvares Gomes proposed openstack/neutron master: [OVN] DO NOT REVIEW: PoC Schema aware periodics  https://review.opendev.org/70977014:35
*** ociuhandu has quit IRC14:37
*** zhanglong has quit IRC14:38
*** sean-k-mooney has quit IRC14:38
dansmithslaweq: okay, but... I've done a clean and re-stack, so I'm not sure why that's not just working14:39
dansmithslaweq: can you tell me more about what to look for?14:40
slaweqdansmith: what IP address You have set in ovs agent's config as local_ip?14:40
dansmithslaweq: I'm booting back up, so I'll look in a sec.. do you mean in local.conf or /etc/neutron/something?14:41
slaweqdansmith: in /etc/neutron/plugins/ml2/ml2_conf.ini14:41
slaweqor /etc/neutron/plugins/ml2/openvswitch_agent.ini14:42
dansmithslaweq: the correct ip for my enp3s0 interface14:45
slaweqdansmith: and this interface is up, right?14:45
dansmithslaweq: I'm ssh'd over it right now :)14:45
slaweqdansmith: ok14:45
slaweqok, so that's weird14:46
slaweqdansmith: can You send me link to this log once again?14:46
dansmithslaweq: https://pastebin.com/zC1Qim8q14:47
dansmithslaweq: the same trace gets logged at least twice, but that's the last copy14:47
*** yamamoto has quit IRC14:48
amotokislaweq: bcafarel: I am checking every projects in the release review now.14:49
openstackgerritLucas Alvares Gomes proposed openstack/neutron master: [OVN] Add support for external ports  https://review.opendev.org/70337614:49
amotokislaweq: networking-odl might be worth having another release.14:49
*** iurygregory has quit IRC14:49
*** yamamoto has joined #openstack-neutron14:50
*** imalinovskiy has joined #openstack-neutron14:52
slaweqdansmith: that's interesting, it seems that for some reason pyroute2 returned some interface without "index" key14:52
slaweqralonsoh: did You saw something like that before with pyroute2 ^^?14:52
dansmithslaweq: can I confirm that by running something directly to dump the output?14:53
slaweqdansmith: can You also send us output from "ip a" command?14:53
ralonsohslaweq, that was supposed to be solved14:53
ralonsohcan I see the logs?14:53
slaweqralonsoh: https://pastebin.com/zC1Qim8q14:53
dansmithslaweq: can I paste it to you in a /query? I'd rather not dump my addresses here :)14:53
slaweqdansmith: sure14:54
slaweqYou can even hide real IPs, that's not most important part ;)14:54
dansmithokay14:54
*** dsneddon has joined #openstack-neutron14:55
*** yamamoto has quit IRC14:55
bcafarelamotoki: thanks!14:56
amotokibcafarel: looking at the irc log, it looks like you are preparing a new release review. networking-odl has several backports since the last release, so it looks worth having another release.14:56
slaweqamotoki: thx a lot14:57
dansmithslaweq: https://pastebin.com/mZAk22kx14:57
amotokislaweq: bcafarel: other than this, slaweq's update on the rocky-em patch looks fine to me (except Sean's point)14:58
ralonsohslaweq, dansmith do you have the logs from pyroute2? when pyroute2 is called, we usually print the returned blob14:58
ralonsohdo you have it?14:58
bcafarelamotoki: thanks, yes the versions and hashes looked good to me too14:58
bcafarelso separate releases patch with mentioned changes + odl, and rebase rock-em on top after that :)14:59
dansmithralonsoh: what does it look like? I see "privsep reply..*" and then a python dict structure14:59
ralonsohdansmith, exactly, this long structure14:59
*** dsneddon has quit IRC15:00
ralonsohif there are sensitive IP addresses15:00
slaweqamotoki: bcafarel ok, I can push separate patch with those releases before going EM15:00
ralonsohwe can handle this in a private chat15:00
slaweqor You can do that if You want, as I have now CI meeting15:00
dansmithralonsoh: it's ... large :)15:00
ralonsohdansmith, I know, I know15:00
slaweqdansmith: please follow up with ralonsoh as he is our "pyroute2" expert15:00
slaweq:)15:01
dansmithslaweq: okay thanks for your help15:01
bcafarelooh true CI meeting15:01
amotokislaweq: bcafarel: I have a horizon meeting in the same slot. let's check who can do it later.15:01
slaweqdansmith: I didn't help too much :)15:01
slaweqamotoki: ok :)15:01
bcafarelamotoki: slaweq: sounds good, as long as we do not end up sending 3 separate requests :)15:02
slaweqbcafarel: yep15:02
ralonsohdansmith, do you have this patch https://review.opendev.org/#/c/686206/8/neutron/privileged/agent/linux/ip_lib.py?15:04
ralonsohdansmith, https://bugs.launchpad.net/neutron/+bug/184636015:05
openstackLaunchpad bug 1846360 in neutron "Pyroute2 can return dictionary keys in bytes instead of strings" [High,Fix released] - Assigned to Rodolfo Alonso (rodolfo-alonso-hernandez)15:05
ralonsoherror: http://paste.openstack.org/show/780851/15:05
dansmithralonsoh: I'm on 6b9765c991da8731fe39f7e7eed1ed6e2bca231a15:05
dansmithralonsoh: and yes, I have that patch content15:06
ralonsohdansmith, do you see in the long pyroute2 structure returned the "index" parameter?15:06
dansmithralonsoh: this large structure doesn't seem to have my ip4 addresses in it15:07
*** tbachman has joined #openstack-neutron15:07
dansmithyes, it has indexes15:07
*** yamamoto has joined #openstack-neutron15:10
dansmithralonsoh: am I missing where it has my ip4 and ip6 addresses? if not I'll just pastebin it15:10
*** ociuhandu has joined #openstack-neutron15:11
ralonsohdansmith, this method      "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 200, in get_device_by_ip15:11
ralonsohshould retrieve the devices filtering by "address"15:11
dansmithokay I'm not sure if that's an answer to my question or not.. I'm just trying to make sure this blob doesn't contain anything sensitive so I can pastebin it to you :)15:14
*** sean-k-mooney has joined #openstack-neutron15:14
ralonsohdansmith, sure15:14
*** ociuhandu has quit IRC15:16
dansmithralonsoh: sent the dump15:17
dansmithI don't see my ip4 or ip6 addresses in there, so I'm guessing maybe that's related?15:17
ralonsohdansmith, 1) yes, this is a problem if you don't find the IP address there, because those are all interfaces you have in root namespace15:18
ralonsoh2) the "index" problem... I don't know why it's happening15:19
dansmithokay15:19
dansmiththis stack worked on my machine earlier this week, which is very confusing15:19
*** mugsie has quit IRC15:20
*** mugsie has joined #openstack-neutron15:22
ralonsohdansmith, did you update the NIC drivers? or the NIC itself?15:24
dansmithralonsoh: no15:24
dansmith'ip a' works so... :)15:25
ralonsohand you are using the same pyroute2 version15:25
ralonsoh(no new releases last week)15:25
dansmithsame as what? earlier in the week? yes, this is just devstack15:25
ralonsohdansmith, this is very strange: no IFA_ADDRESS in any device15:26
*** dsneddon has joined #openstack-neutron15:26
slaweqralonsoh: can it be that e.g. oslo.privsep was updated and has some changes? (just guessing)15:27
ralonsohslaweq, let me check15:27
ralonsohnot relevant updates (apart from drop py2 support)15:28
*** dsneddon has quit IRC15:31
openstackgerritBrian Haley proposed openstack/neutron master: Improve log message on exception for invalid tunnel type  https://review.opendev.org/70863415:34
*** iurygregory has joined #openstack-neutron15:34
openstackgerritBrian Haley proposed openstack/neutron-tempest-plugin master: [ussuri][goal] Drop python 2.7 support and testing  https://review.opendev.org/70425715:35
dansmithralonsoh: can I run the pyroute command as root myself to see if the addrs are there?15:40
ralonsohdansmith, yes, you can import the neutron.privileged.agent.linux.ip_lib module15:41
ralonsohand execute those commands15:41
*** ociuhandu has joined #openstack-neutron15:41
ralonsohdansmith, BTW, reviewing the code, the IP addresses should NOT be there15:41
dansmithoh, heh15:42
ralonsohbecause privileged.get_link_devices is like "ip link"15:42
ralonsohthen, with the index retrieved (and the index is there) we call like "ip address show dev **"15:42
dansmithokay15:43
dansmithwell, I'll stand by for your next suggestion then :)\15:43
*** ijw has joined #openstack-neutron15:44
*** dsneddon has joined #openstack-neutron15:45
ralonsohdansmith,  can you execute manually, as normal user,   neutron.agent.linux.ip_lib.get_devices_with_ip(None)15:46
dansmithralonsoh:15:47
dansmith  File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 1347, in _parse_link_device15:47
dansmith    index=device['index'],15:47
dansmithKeyError: 'index'15:47
dansmith(that's the last line of the trace)15:47
ralonsohdansmith, ok, there is a problem in the structure returned in L135815:48
ralonsohdevices = privileged.get_link_devices(namespace, **link_args)15:48
ralonsohcan you please execute this command15:48
ralonsoha, for example, review what you have in devices[0]15:48
*** ijw has quit IRC15:48
ralonsohand print the type of 'index'15:48
ralonsohbecause, IMO, there is something weird in what you have there (this is not string)15:49
dansmithhow can I run that manually? I don't know what namespace or link_args should be15:50
ralonsohnamespace None15:50
ralonsohno args15:50
dansmithdon't I need some setup for neutron.privileged to work?15:51
ralonsohno no15:51
ralonsohnot neutron.privileged.....15:51
ralonsohthis is neutron.agent.linux.ip_lib15:52
*** dasp has joined #openstack-neutron15:52
ralonsohah, ok, you are calling privileged.get_link_devices(namespace, **link_args)15:52
ralonsohbut no, no root user15:52
dansmithmy point is:15:53
dansmith>>> from neutron import privileged15:53
dansmith>>> privileged.get_link_devices(None)15:53
dansmithTraceback (most recent call last):15:53
dansmith  File "<stdin>", line 1, in <module>15:53
dansmithAttributeError: module 'neutron.privileged' has no attribute 'get_link_devices'15:53
dansmithneutron.privileged is setup by privsep, IIRC15:53
ralonsohfrom neutron.privileged.agent.linux import ip_lib15:54
ralonsohz=ip_lib.get_link_devices(None)15:54
ralonsoh>>> z[0]15:54
ralonsoh{'family': 0, '__align': (), 'ifi_type': 772,..............15:54
dansmithah, okay, you said neutron.agent.linux above15:54
amotokislaweq: bcafarel: if none of you is preparing a release patch, I can do it now.15:54
slaweqamotoki: thx, I didn't had time to do it yet15:55
ralonsohdansmith, then get the first element15:55
ralonsohelement = z[0]15:56
ralonsoh_list = list(element.keys())15:56
ralonsohdansmith, and then print the type of those elements in the list15:56
ralonsohall of them should be string15:56
dansmithdict_keys([b'index', b'family', b'__align', b'header', b'state', b'flags', b'ifi_type', b'event', b'change', b'attrs'])15:56
*** jawad_axd has quit IRC15:56
ralonsoharggggggggg15:57
ralonsohwhy byte?????15:57
ralonsohthere we have the problem15:57
dansmith>>> pprint.pprint(ip_lib.get_link_devices(None)[1]['index'])15:57
dansmithTraceback (most recent call last):15:57
dansmith  File "<stdin>", line 1, in <module>15:57
dansmithKeyError: 'index'15:57
dansmith>>> pprint.pprint(ip_lib.get_link_devices(None)[1][b'index'])15:57
dansmith215:57
ralonsohdansmith, thanks for spotting out a problem15:57
ralonsohI'll send a patch to fix this now15:57
dansmithI dunno why this worked before, but.. thanks :)15:58
bcafarelamotoki: not on it right now, you can go ahead (and thx)15:58
*** lajoskatona has quit IRC15:58
amotokibcafarel: slaweq: okay. I will prepare it and let you know.15:58
slaweqamotoki: thx a lot16:00
slaweqralonsoh: dansmith thx for identyfing this problem fast16:01
openstackgerritMaciej Józefczyk proposed openstack/neutron master: [DNM] Verify returned LSP value in functional tests  https://review.opendev.org/71001916:01
dansmithslaweq: ralonsoh was this a regression recently?16:04
ralonsohdansmith, nope, we didn't see that16:05
*** iurygregory has quit IRC16:05
openstackgerritSlawek Kaplonski proposed openstack/neutron master: Remove neutron-tempest-dvr job from CI  https://review.opendev.org/71007116:11
openstackgerritzhangyuhe proposed openstack/neutron master: create reserved dhcp port when subnet without gateway and dhcp is enabled  https://review.opendev.org/70801116:11
ralonsohdansmith, I don't understand. The current code should convert byte to string16:14
ralonsohwhat is happening to you should not happen16:14
ralonsohhttps://review.opendev.org/#/c/686206/16:14
dansmithdan@guaranine:~$ grep 'def _ensure_string' /opt/stack/neutron/neutron/privileged/agent/linux/ip_lib.py16:16
dansmith    def _ensure_string(value):16:16
dansmithralonsoh: ^16:16
ralonsohdansmith, exactly, this should convert from byte to string16:16
openstackgerritSlawek Kaplonski proposed openstack/neutron master: Remove neutron-tempest-dvr job from CI  https://review.opendev.org/71007116:16
dansmithralonsoh: maybe six.binary_type isn't matching up?16:17
ralonsohdansmith, but I have those tests16:17
ralonsohhttps://review.opendev.org/#/c/686206/8/neutron/tests/unit/privileged/agent/linux/test_ip_lib.py16:17
ralonsohusing byte keys16:17
dansmithralonsoh: yeah, I just tested with a byte key and it works, so something must not be happening like you think16:18
ralonsohdansmith, can you change the code?16:20
ralonsoh    def _ensure_string(value):16:20
ralonsoh        return value.decode() if isinstance(value, bytes) else value16:20
ralonsohand then run again the manual retrieval16:20
ralonsoh(I can reproduce this error locally)16:20
*** gcheresh has quit IRC16:20
*** maciejjozefczyk has quit IRC16:21
dansmithFeb 26 08:21:22 guaranine neutron-openvswitch-agent[15678]: ERROR neutron   File "/usr/local/lib/python3.6/dist-packages/netaddr/ip/__init__.py", line 823, in parse_ip_network16:22
dansmithFeb 26 08:21:22 guaranine neutron-openvswitch-agent[15678]: ERROR neutron     raise TypeError('unexpected type %s for addr arg' % type(addr))16:22
dansmithFeb 26 08:21:22 guaranine neutron-openvswitch-agent[15678]: ERROR neutron TypeError: unexpected type <class 'NoneType'> for addr arg16:22
dansmiththat's with: https://pastebin.com/PTVnTgrH16:23
ralonsohdansmith, ok, and this is another error...16:24
*** imalinovskiy has quit IRC16:28
dansmithralonsoh: okay....surely seems like it'd be good to figure out why I'm seeing these issues and nobody else is..16:30
ralonsohdansmith, ok, so the problem with the conversion is solved16:30
dansmithkernel version? pyroute2 stuck somehow?16:30
ralonsohnow we have another problem16:30
dansmithpyroute2==0.5.916:30
dansmithLinux guaranine 4.15.0-48-generic #51-Ubuntu SMP Wed Apr 3 08:28:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux16:30
ralonsohdansmith, this is not the problem. The problem now is why ip_lib.get_ip_addresses(None, index=???)16:32
ralonsohis returning data WITHOUT IFA_ADDRESS16:32
*** mattw4 has joined #openstack-neutron16:33
dansmithokay16:33
ralonsohdansmith, can you check, per interface index (same index as in "ip link")16:35
ralonsohthe result of ip_lib.get_ip_addresses(None, index=???)16:35
dansmithralonsoh: the addresses are in there16:37
dansmithip4 and ip616:37
*** ijw has joined #openstack-neutron16:38
*** yamamoto has quit IRC16:39
dansmith>>> list(k[0] for k in ip_lib.get_ip_addresses(None, index=2)[0]['attrs'])16:40
dansmith[b'IFA_ADDRESS', b'IFA_LOCAL', b'IFA_BROADCAST', b'IFA_LABEL', b'IFA_FLAGS', b'IFA_CACHEINFO']16:40
ralonsohdansmith, yes, I think I'm missing a conversion16:40
ralonsohactually this should be16:40
ralonsoh    elif isinstance(value, netlink.nla_slot):16:40
ralonsoh        return [_ensure_string(value[0]), make_serializable(value[1])]16:40
dansmithso it's failing to match 'IFA_ADDRESS' to b'IFA_ADDRESS' ?16:40
ralonsohyes16:41
dansmithokay16:41
ralonsohthe problem is in the netlink.nla_slot tuple16:41
ralonsohcan you change the previous code?16:41
*** gcheresh has joined #openstack-neutron16:42
dansmithralonsoh: https://pastebin.com/kTGcpU2116:43
dansmithappears to be running now :)16:43
ralonsohdansmith, thanks a lot!!16:43
ralonsohI'll push a patch right now16:44
ralonsohand I'll cherry-pick it to stable versions16:44
dansmithralonsoh: I really want to know why I'm seeing this and nobody else is, especially if it's not a recent regression16:44
dansmithralonsoh: let me know when you have something up and I will configure my devstack to pull it so I can start fresh and make sure everything is good16:44
ralonsohthis is what pyroute2 is reading directly from the iproute libraries16:44
ralonsohdansmith, sure thanks!16:44
*** rubasov has quit IRC16:47
*** tesseract has quit IRC16:50
*** dsneddon has quit IRC16:50
amotokislaweq: bcafarel: I proposed a release patch https://review.opendev.org/710080 and updated rocky EM patch https://review.opendev.org/70990116:51
openstackgerritRodolfo Alonso Hernandez proposed openstack/neutron master: Ensure netlink.nla_slot tuple key is a string  https://review.opendev.org/71008216:51
amotokislaweq: bcafarel: I incremented minor version (13.1.0) of neutron-dynamic-routing per my comment in https://review.opendev.org/70990116:52
amotokislaweq: bcafarel: I haven't checked the detail on networking-ovn minor version bump to 5.1.0 and followed slaweq's change. There are many changes from 5.0.1.....16:55
bcafarelamotoki: thanks, both look OK to me!17:00
*** lucasagomes has quit IRC17:01
*** ociuhandu has quit IRC17:03
*** mattw4 has quit IRC17:03
*** mattw4 has joined #openstack-neutron17:04
*** dsneddon has joined #openstack-neutron17:09
openstackgerritTerry Wilson proposed openstack/networking-ovn stable/train: Ensure we don't pass empty addresses to lsp_set_addresses  https://review.opendev.org/70502817:10
*** yamamoto has joined #openstack-neutron17:11
*** TristanSullivan has joined #openstack-neutron17:15
*** yamamoto has quit IRC17:17
openstackgerritRodolfo Alonso Hernandez proposed openstack/neutron master: [WIP] Fix query for segment ranges  https://review.opendev.org/71009017:19
*** tmorin has quit IRC17:26
*** ociuhandu has joined #openstack-neutron17:30
*** mattw4 has quit IRC17:34
*** jpena is now known as jpena|off17:40
*** mattw4 has joined #openstack-neutron17:44
*** jangutter_ has quit IRC17:47
*** jangutter has joined #openstack-neutron17:48
*** jangutter has quit IRC17:50
*** tbachman has quit IRC17:51
openstackgerritMerged openstack/neutron stable/queens: Improve "OVSFirewallDriver.process_trusted_ports"  https://review.opendev.org/70518817:58
*** ociuhandu has quit IRC18:05
*** mattw4 has quit IRC18:05
*** ociuhandu has joined #openstack-neutron18:05
*** jlibosva has quit IRC18:09
*** jlibosva1 has joined #openstack-neutron18:09
*** jlibosva1 is now known as jlibosva18:09
*** ociuhandu has quit IRC18:11
dansmithralonsoh: using that patch with a fresh devstack seems to be good.. thanks!18:21
*** yamamoto has joined #openstack-neutron18:22
openstackgerritMerged openstack/neutron-tempest-plugin master: Fix intermittent port_forwarding test failures  https://review.opendev.org/69921318:24
openstackgerritMerged openstack/neutron-tempest-plugin master: Disable security group quotas in NetworkSecGroupTest  https://review.opendev.org/70969118:24
*** ijw has quit IRC18:27
*** nweinber_ has joined #openstack-neutron18:29
*** nweinber has quit IRC18:31
*** priteau has quit IRC18:32
*** ramishra has quit IRC18:34
*** mattw4 has joined #openstack-neutron18:36
*** igordc has joined #openstack-neutron18:37
ralonsohdansmith, you are welcome18:42
*** maciejjozefczyk has joined #openstack-neutron18:42
openstackgerritMerged openstack/networking-bgpvpn master: Switch functional/install jobs to Zuulv3 syntax  https://review.opendev.org/70360118:43
*** igordc has quit IRC18:43
*** ociuhandu has joined #openstack-neutron18:46
*** N3l1x has joined #openstack-neutron18:48
*** ociuhandu has quit IRC18:51
*** maciejjozefczyk has quit IRC18:57
*** mattw4 has quit IRC19:09
*** mattw4 has joined #openstack-neutron19:17
*** gcheresh has quit IRC19:26
*** gcheresh has joined #openstack-neutron19:28
*** TristanSullivan has quit IRC19:31
*** yamamoto has quit IRC19:31
*** gcheresh has quit IRC19:32
*** ociuhandu has joined #openstack-neutron19:34
*** nweinber__ has joined #openstack-neutron19:35
*** nweinber_ has quit IRC19:38
*** jlibosva has quit IRC19:39
*** ociuhandu has quit IRC19:40
*** TristanSullivan has joined #openstack-neutron19:49
*** gcheresh has joined #openstack-neutron19:59
*** yamamoto has joined #openstack-neutron20:02
*** tbachman has joined #openstack-neutron20:05
*** yamamoto has quit IRC20:08
*** ociuhandu has joined #openstack-neutron20:08
*** gcheresh has quit IRC20:11
*** ociuhandu has quit IRC20:12
*** ociuhandu has joined #openstack-neutron20:28
openstackgerritSlawek Kaplonski proposed openstack/neutron-tempest-plugin master: Add list security group rules API test  https://review.opendev.org/68191220:33
*** kozhukalov has quit IRC20:33
*** kozhukalov has joined #openstack-neutron20:36
*** ileixe has joined #openstack-neutron20:36
*** sean-k-mooney has quit IRC20:41
*** mattw4 has quit IRC20:42
*** tbachman has quit IRC20:42
*** mattw4 has joined #openstack-neutron20:42
*** tbachman has joined #openstack-neutron20:56
*** nweinber__ has quit IRC20:59
*** ccamposr__ has joined #openstack-neutron21:06
*** ccamposr has quit IRC21:06
*** ccamposr__ has quit IRC21:10
*** ccamposr__ has joined #openstack-neutron21:10
*** ccamposr__ has quit IRC21:19
*** ccamposr has joined #openstack-neutron21:19
*** rcernin has quit IRC21:21
*** ralonsoh has quit IRC21:22
*** bnemec has quit IRC21:23
*** priteau has joined #openstack-neutron21:28
*** kozhukalov has quit IRC21:29
*** slaweq has quit IRC21:32
*** ociuhandu has quit IRC21:36
*** ociuhandu has joined #openstack-neutron21:37
*** ccamposr has quit IRC21:37
*** ociuhandu has quit IRC21:41
*** ccamposr has joined #openstack-neutron21:46
openstackgerritBrian Haley proposed openstack/neutron master: [OVN] Call OVNClient methods with context when possible  https://review.opendev.org/70615221:48
*** ociuhandu has joined #openstack-neutron21:50
*** ociuhandu has quit IRC22:02
*** ociuhandu has joined #openstack-neutron22:02
*** ociuhandu has quit IRC22:07
*** tbachman has quit IRC22:11
*** priteau has quit IRC22:11
openstackgerritBrian Haley proposed openstack/neutron master: [OVN] Call OVNClient methods with context when possible  https://review.opendev.org/70615222:13
*** ccamposr__ has joined #openstack-neutron22:22
*** ccamposr has quit IRC22:25
*** tbachman has joined #openstack-neutron22:27
*** ccamposr has joined #openstack-neutron22:33
*** tbachman has quit IRC22:34
*** ccamposr__ has quit IRC22:35
*** ijw has joined #openstack-neutron22:37
*** ociuhandu has joined #openstack-neutron22:40
*** nweinber__ has joined #openstack-neutron22:45
*** ociuhandu has quit IRC22:45
*** nweinber__ has quit IRC22:47
*** rcernin has joined #openstack-neutron22:49
*** tkajinam has joined #openstack-neutron22:53
openstackgerritBrian Haley proposed openstack/neutron master: Ensure netlink.nla_slot tuple key is a string  https://review.opendev.org/71008222:55
openstackgerritwaleed mousa proposed openstack/neutron master: Fix: Removing SmartNIC OVS representor port on instance tear down and resync  https://review.opendev.org/70280622:59
*** N3l1x has quit IRC23:02
openstackgerritBrian Haley proposed openstack/neutron master: [OVN] Call OVNClient methods with context when possible  https://review.opendev.org/70615223:08
*** bnemec has joined #openstack-neutron23:18
*** tbachman has joined #openstack-neutron23:31
*** TristanSullivan has quit IRC23:44
« Tuesday, 2020-02-25 Index Thursday, 2020-02-27 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!