Monday, 2021-10-04

*** sshnaidm|off is now known as sshnaidm07:26
odyssey4mehey folks - not sure who may be interested in this, but there is an ovn/bgp agent in the works... https://opendev.org/x/ovn-bgp-agent11:30
jamesdenton_alti'm interested.11:42
*** jamesdenton_alt is now known as jamesdenton11:42
jamesdentonthanks odyssey4me :)11:42
odyssey4meawesome, good to know jamesdenton :) I'm sure the team would love feedback... they're driving this as a way of doing multi-cloud or inter-cloud connectivity11:55
jrosserodyssey4me: is that what we need for ipv6 prefix advertisment too?11:58
jrosseripv6 is would be our biggest gap for OVN right now i think her11:58
jrossere11:58
odyssey4mejrosser no idea, lemme try and find out where they hang out and we can put y'all in touch with them directly ... no need for me to play middle-man :)12:02
noonedeadpunkodyssey4me: oh, that _is_ really interesting and what we were looking for for quite a while12:23
noonedeadpunkas indeed I believe this is smth that should give ipv612:24
noonedeadpunknot sure how stable is that considering initial commit 1 month ago :)12:25
noonedeadpunkbut still awesome that it has landed12:25
odyssey4mejrosser noonedeadpunk jamesdenton I'm told that #openstack-neutron would be a good place to ask questions about it and make contact with the team if you'd like to know more about stability, plans, etc.12:34
noonedeadpunksure! thanks!12:35
odyssey4mespatel Not sure if OVN is something on your radar, but https://opendev.org/x/ovn-bgp-agent might be of interest if so.12:35
mgariepyhmm why does the doc points to 19.0.6  instead of the latest ? https://docs.openstack.org/project-deploy-guide/openstack-ansible/stein/deploymenthost.html#install-the-source-and-dependencies12:42
spatelodyssey4me sounds interesting, thanks i will sure take a look 12:56
spatelhttps://opendev.org/x/ovn-bgp-agent/src/branch/master/doc/images/evpn_traffic_flow.png 13:02
spatelmay need some more document to understand whole process 13:02
spateljamesdenton do you have Intel X510T2 ? 13:06
spatelsorry X550T213:06
jamesdentonX550? I only have X540 here, but know some environments have them13:34
cardiff_danhey folks, I've got a near standard install of Openstack Wallaby on Ubuntu 18.04 and experiencing some odd behaviour: instances with an internal IP in the 10.0.3.0/24 range are failing to get internet connectivity when the iptables rule `-A POSTROUTING -s 10.0.3.0/24 ! -d 10.0.3.0/24 -j MASQUERADE` is in place on the controller nat iptable. As far as I can tell this rule is added automatically when lxc is installed13:39
jrossernoonedeadpunk: didnt we look at something similar to this before ^ ?13:40
spateljamesdenton this is the error i am getting when i attach dpdk port - https://paste.opendev.org/show/809766/14:14
spatelwhat driver version are you running? 14:14
jamesdentonsame driver14:18
jrossercardiff_dan: it's interesting that traffic seems to be escaping the neutron router namespace and getting mixed up with stuff on the host? can you see how thats happening?14:19
jamesdentonspatel i had the same issue and never figured it out14:19
spatelreally :O14:20
spateleven after upgrade of firmware ?14:20
jamesdentonthis was a couple of weeks ago, and you mentioned the shared memory thing fixed yours. Didn't fix mine14:20
spatelofficial document saying, its supported 14:20
spatelthat was in HP server, now i have bunch of DELL servers i am dealing with 14:21
jamesdentonyes, it's supported, i've done it before (with these same hosts) but it's been a couple of years14:21
spatellook like we have to make habit of documenting everything single thing :)14:21
spatelits hard to remember what we did 2 month ago.. 14:22
jamesdentonfor sure14:22
jamesdentonhow's your OVN deploy working out>14:22
spateli am planning to create blog of each server and card i have and process of DPDK / SRIOV etc.. 14:22
spatelMy OVN is running great, not a single issue14:22
spateli am running DPDK / SRIOV and standard virtio all kind of workload 14:22
spatelvery stable so far.. 14:23
spatelonly issue is hard to know where and how to troubleshoot when it comes to OpenFlow related issue. (so far i didn't see any issue) 14:23
opendevreviewAndrew Bonney proposed openstack/openstack-ansible master: Remove unnecessary pki step in haproxy install  https://review.opendev.org/c/openstack/openstack-ansible/+/81236114:23
cardiff_danjrosser: I think it is yeah, certainly removing the rule fixes the problem, but wasn't viable to leave that way. Can you suggest a way of identifying that's happening? 14:24
cardiff_dan*what's 14:24
spatelone downside i don't like about OVN is when you reload ovs switch it disturb traffic for few seconds 14:24
jamesdentoncardiff_dan is your tenant network behind a neutron router?14:24
cardiff_danjamesdenton: it is yeah14:28
jamesdentonis SNAT disabled on that router?14:28
cardiff_danjamesdenton: SNAT is enabled14:33
spateljrosser help me to understand this output - https://paste.opendev.org/show/809768/15:09
spatelwhy VmallocTotal is so high ? on 64GB memory box15:09
jrosserwell, vmalloctotal is the amount of virtual rather than physical memory that is allocated15:14
spatelthat output saying 34TB am i wrong15:17
foutatoroHi, I deployed openstack + ceph on 2 servers, after deployment it seems that objects storage service is up but I can't create containers nor make a "openstack container list". In my understanding when using ceph we don't need to set swift conf in openstack_user_conf.yml 16:19
foutatoroyou can find my conf and output erreor here https://paste.opendev.org/show/809771/16:20
spatelfoutatoro i don't think you need two storage system, ceph is all you need 16:25
foutatorospatel: yes I know but I'm wondering if I miss something in my conf 16:26
foutatoroI deploy from the current master16:27
spatelyou should look at swift error logs, may not able to authenticate keystone 16:29
spatelI am not expert in swift so don't know.. mostly i stay away from swift since ceph covers everything 16:30
jrosserfoutatoro: you have a 503 response, so start by looking at the haproxy status16:31
jrossersee if the frontend/backend are up or down16:31
foutatorojrosser: thanks for your suggestion haproxy call ceph-rgw containers on port 8080 for object storage services but the port 8080 is not in used into those containers16:43
foutatoroI have to check why ceph is not running properly16:44
jrosserthere is a status interface to haproxy, or you can use hatop to look at the internal status of it17:05
jrosserif the backends are down you need to investigate the rgw containers17:05
jamesdentoncardiff_dan was able to sorta-replicate your issue locally. Added a second subnet, 10.0.4.0/24, behind the same router and it worked fine. 10.0.3.0/24 gets lost in the nether17:52
cardiff_danjamesdenton: great, glad it isn't just me!18:03
cardiff_danjamesdenton: crazier question though, any ideas how to remove the problem?18:06
jamesdentonnot quite. i feel like we've seen something similar before, and it was related to the way linux routes packets. lxcbr0 likely owns 10.0.3.1, and when traffic is traverses the brq bridge it might hit there first rather than up into the namespace18:14
jawad-axdHi folks, trying to run cloudkitty in my openstack-asnible aio environment with genocchi and ceilometer (Wallaby and Ubuntu). All telemetry services seem to be running fine, but cant see any reports on horizon. In rating summary, cant see tenants to which I did "openstack role add --user cloudkitty --project demo rating". Added hashmap service 'instance' with flavor_id (followed the docs). Some pointer would give18:17
jawad-axda push on it..thanks18:17
jawad-axdhttps://paste.openstack.org/show/809777/ Journal cloudkitty.18:23
mgariepyso it's time to get tcpdump out @jamesdenton  ?18:32
jamesdentonyep. this post rings a bell: https://vincent.bernat.ch/en/blog/2017-linux-bridge-isolation19:05
mgariepysooo.. we should enable bridge vlan filtering by default on osa >?19:28
jamesdentoni don't really know19:42
spateljamesdenton ty21:43
spatelyt21:43
spateli have no idea what i did but now dpdk working with X550T21:43
jamesdentonyou'll be happy to know i've converted my LXB to OVN. now time to fix everything21:43
jamesdentonhey, cool!21:44
spatelnow trying to revert my changes to find out what i did :)21:44
spatelLXB to OVS ?21:44
spatelLXB to OVN.. oh! 21:44
spateldid you find out how to migrate existing setup to ovn?21:44
jamesdentonwell, sorta? i deleted all of my VMs but left everything else21:45
jamesdentonleaving the VMs would require some DB work that i'm just not prepared to tackle21:46
spatelThat is awesome! 21:47

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!