| *** sarob has quit IRC | 01:33 | |
| *** davidlenwell has quit IRC | 01:33 | |
| *** t0mb0 has quit IRC | 01:34 | |
| *** davidlenwell has joined #akanda | 01:34 | |
| *** ChanServ sets mode: +o davidlenwell | 01:34 | |
| *** adam_g has quit IRC | 01:35 | |
| *** mugu has quit IRC | 01:35 | |
| *** sarob has joined #akanda | 01:38 | |
| *** adam_g has joined #akanda | 01:38 | |
| *** adam_g has quit IRC | 01:38 | |
| *** adam_g has joined #akanda | 01:38 | |
| *** mugu has joined #akanda | 01:38 | |
| *** t0mb0 has joined #akanda | 01:40 | |
| *** praveendp526 has joined #akanda | 05:13 | |
| praveendp526 | hai | 05:13 |
|---|---|---|
| praveendp526 | network related issues | 05:14 |
| * praveendp526 slaps davidlenwell around a bit with a large fishbot | 05:15 | |
| praveendp526 | help | 05:15 |
| *** praveendp526 has quit IRC | 05:38 | |
| *** dhellmann has quit IRC | 06:28 | |
| *** dhellmann has joined #akanda | 06:28 | |
| *** dhellmann has quit IRC | 06:34 | |
| *** dhellmann has joined #akanda | 06:34 | |
| *** davidlenwell_ has joined #akanda | 07:10 | |
| *** davidlenwell has quit IRC | 07:15 | |
| *** ronis has joined #akanda | 08:57 | |
| *** openstack has joined #akanda | 12:50 | |
| stupidnic | adam_g: is the akanda appliance supposed to be running dhcp and metadata agents? | 15:51 |
| *** ronis has quit IRC | 16:58 | |
| *** elo1 has quit IRC | 17:10 | |
| *** elo has joined #akanda | 17:10 | |
| adam_g | stupidnic, it runs a dhcp server and a metadata proxy, but not agents | 17:12 |
| stupidnic | adam_g: hmmm okay... I am not seeing them running in the ps list | 17:12 |
| adam_g | stupidnic, dnsmasq + /usr/local/bin/akanda-metadata-proxy | 17:13 |
| stupidnic | let me spin up an instance and look | 17:13 |
| *** davidlenwell_ is now known as davidlenwell | 17:15 | |
| *** ChanServ sets mode: +o davidlenwell | 17:15 | |
| *** cleverdevil has joined #akanda | 17:15 | |
| stupidnic | okay... I see dnsmasq running, but not the akanda-metadata-proxy | 17:16 |
| stupidnic | having said that, my instances aren't getting IPs anyways | 17:17 |
| adam_g | stupidnic, can you confirm that you've also got gunicorn running, serving /usr/local/share/akanda ? | 17:19 |
| stupidnic | affirmative | 17:19 |
| adam_g | stupidnic, in /etc/dnsmasq.d/ there hould .conf's for each NIC. can you see the addresses you're expecting to be assigned configured there? | 17:20 |
| stupidnic | adam_g: let me verify that I have the router setup correctly. | 17:21 |
| adam_g | stupidnic, neutron router-show should report it as ACTIVE if so | 17:21 |
| stupidnic | I think I am missing a port from the internal network to the router. | 17:22 |
| stupidnic | so that's my bad | 17:22 |
| davidlenwell | /mode * +o adam_g | 17:37 |
| davidlenwell | wait that didn't work | 17:37 |
| *** davidlenwell sets mode: +o adam_g | 17:38 | |
| davidlenwell | there you go adam_g | 17:38 |
| adam_g | davidlenwell, ty | 17:38 |
| stupidnic | adam_g: okay... so I added the interface on the router to attach to the internal network but I am still not getting DHCP traffic | 17:38 |
| stupidnic | I can confirm that the network interface does have dnsmasq conf | 17:39 |
| stupidnic | I have two cirros instances setup | 17:39 |
| stupidnic | if I assign IPs manually they can talk to each other but not the router's internal network interface | 17:39 |
| adam_g | stupidnic, so starting from the bottom, you should have their router interface brought up on eth2 in the appliance? | 17:39 |
| stupidnic | confirmed. It is up with an IP address of 10.0.0.1 | 17:40 |
| stupidnic | I show RX and TX on that interface | 17:40 |
| adam_g | stupidnic, and then /etc/dnsmasq.d/eth2.conf should contain the addresses bound for the cirros's? | 17:41 |
| stupidnic | yes | 17:42 |
| stupidnic | dhcp-host..... | 17:42 |
| stupidnic | I have run a dump on the interface and I am not seeing traffic | 17:42 |
| adam_g | stupidnic, whats neutron_port_security_extension_enabled set to in /etc/astara/orchestrator.ini ? | 17:44 |
| stupidnic | True | 17:44 |
| adam_g | markmcclain, any ideas ^ no dhcp /w linuxbridge. | 17:45 |
| stupidnic | I looked at the compute nodes themselves and confirmed that the bridges are using the same vxlan ids | 17:47 |
| stupidnic | so compute04 is running the astara instance and is bridged to vxlan 10 (management) and vxlan 50 (tenant network) | 17:47 |
| markmcclain | any traffic being dropped by (eb|ip)tables | 17:47 |
| stupidnic | on the router? or compute node? | 17:48 |
| markmcclain | compute node | 17:48 |
| markmcclain | also if you can... force a guest to the same node at the router | 17:48 |
| stupidnic | alright, hold on | 17:48 |
| markmcclain | this will help to bisect the problem between replication and local hypervisor issues | 17:49 |
| stupidnic | Okay. my live migration doesn't seem to be working, so let me try and spawn an instance directly on that node | 17:50 |
| stupidnic | alright... I have a cirros instance on the same tenant network and on the same compute node as the router, and it isn't pulling an IP either | 17:52 |
| stupidnic | should I be using things like l2_population on my neutron config? | 17:53 |
| markmcclain | no... l2 pop should be off if you're using hpb | 17:54 |
| stupidnic | hpb? | 17:54 |
| stupidnic | Okay. Removed it from ml2_conf and linuxbridge_agent | 17:55 |
| adam_g | ok. meeting time | 18:01 |
| *** stanchan has joined #akanda | 18:06 | |
| markmcclain | stupidnic: hierarchical port binding | 18:11 |
| stupidnic | markmcclain: okay I am not doing that, I don't think. my current setup is only one LY9 running cumulus. Sort of a collpased network. | 18:12 |
| stupidnic | later one if we add spines then we would certainly do HPB | 18:13 |
| stupidnic | but for the moment we are only using vxlans contained within the leaf | 18:13 |
| markmcclain | k | 18:14 |
| drwahl | markmcclain: i think jordan left some stuff on your plate thursday/friday regarding the vxfld/neutron stuff. do you have any updates? | 18:21 |
| *** cleverdevil- has joined #akanda | 18:58 | |
| *** cleverdevil has quit IRC | 19:00 | |
| *** cleverdevil- is now known as cleverdevil | 19:33 | |
| stupidnic | adam_g and markmcclain: I can confirm that I am seeing the BOOTP/DHCP requests on the router | 19:41 |
| stupidnic | I can also confirm that the astara instance is responding to the requests for DHCP, so something else is blocking the requests | 19:45 |
| adam_g | stupidnic, hmm this sounds a lot like port security stuff that (i thought) we fixed | 19:57 |
| elo | is (eb|ip)tables dropping the response from astara router to the VM on the host? | 19:58 |
| adam_g | stupidnic, you should have a neutron port for the router thats named with /w a AKANDA:VRRP: prefix | 19:58 |
| adam_g | stupidnic, if you do a 'neutron port-show' on that, port_security_enabled should be False | 19:58 |
| adam_g | and ya, adding iptables logging might be a good idea on the compute host | 19:59 |
| stupidnic | let me see what I get | 19:59 |
| adam_g | just to see if its being dropped there | 19:59 |
| stupidnic | I can see it is hitting the fallback rule | 19:59 |
| stupidnic | neutron-linuxbri-sg-fallback | 19:59 |
| adam_g | stupidnic, that chain is just dropping it then, ya? | 20:00 |
| stupidnic | yeah | 20:00 |
| stupidnic | it's the final fall through taget | 20:00 |
| adam_g | stupidnic, you might try adding a secgroup rule for the tenant, ala https://git.openstack.org/cgit/openstack/astara/tree/devstack/plugin.sh#n319 | 20:01 |
| adam_g | ... to test | 20:01 |
| stupidnic | adam_g: I remember seeing the VRRP port, but now I can't seem to find it | 20:03 |
| stupidnic | where would that be? | 20:03 |
| adam_g | stupidnic, it owned by the servic tenant, so if you're looking at neutron ports from POV of a tenant its not visible | 20:04 |
| stupidnic | Okay. I found it. It is set to be False | 20:05 |
| stupidnic | I am guessing that the problem is the security groups are blocking the traffic because as far as it is concerned this is two separate tenants, is that correct? | 20:08 |
| adam_g | stupidnic, if that security group rule fixes the issue, yes. tho i thought this wasn't any issue anymore. if it is, we need to find a more elegant solution. requiring that secgroup rule isn't going to cut it | 20:09 |
| stupidnic | I am floundering a little bit here... when I try to execute that command I am getting an error "Multiple security_group matches found for name 'default'" | 20:12 |
| stupidnic | neutron security-group-rule-create --direction ingress --remote-ip-prefix 0.0.0.0/0 default | 20:12 |
| adam_g | stupidnic, are you running that with the admin credentials or the tenant's? | 20:15 |
| stupidnic | admin | 20:15 |
| adam_g | stupidnic, run it as the tenant's | 20:16 |
| adam_g | or reference the desired group by ID | 20:16 |
| *** cleverdevil has quit IRC | 20:17 | |
| stupidnic | Okay. What I did was delete all the security groups and add them all back | 20:18 |
| stupidnic | That seems to have sorted it out I think | 20:19 |
| *** cleverdevil has joined #akanda | 20:19 | |
| elo | I was just checking - there is a global default sec group rule and if you create one for a tenant there will be two reference to the same name | 20:20 |
| stupidnic | Yeah, deleted all the security groups and recreated them manually | 20:22 |
| stupidnic | works perfect. rebooted all the instances and they immediately got DHCP assigned addresses | 20:22 |
| stupidnic | our current cloud also had issues with security groups, the default just up and disappeared one day... coupled with OVS, that was a treat to debug | 20:23 |
| adam_g | stupidnic, nicenice! | 20:25 |
| *** cleverdevil has quit IRC | 20:25 | |
| stupidnic | I also have astara running under a user astara with an upstart config file | 20:25 |
| adam_g | stupidnic, would you mind creating a bug at http://bugs.launchpad.net/astara RE the need to have that security group? we need to come up with a better story there, requiring tenants to explicitly add that in is bogus | 20:25 |
| stupidnic | is there any sort of rootwrap available? | 20:25 |
| adam_g | stupidnic, at the moment, no. oslo.rootwrap support is coming in M | 20:26 |
| stupidnic | adam_g: alright. | 20:26 |
| stupidnic | adam_g: I am not 100% sure if that security group rule was the actual issue, or maybe I don't understand the full implications of what me adding the security groups manually did | 20:28 |
| stupidnic | oh and one other minor issue... and this really might be a configuration issue with my networks | 20:30 |
| stupidnic | when astara starts it is creating an interface called ns-<uuid> that has the gateway IP address assigned to it | 20:30 |
| stupidnic | for the external network | 20:30 |
| *** cleverdevil has joined #akanda | 20:30 | |
| stupidnic | It shouldn't be doing that I don't think | 20:31 |
| stupidnic | at least it is wrecking my external network connectivity | 20:31 |
| stupidnic | on my controller | 20:31 |
| adam_g | hmm not sure about that one, markmcclain may have some insight, he's more familiar with the lower-level plumbing than me | 20:32 |
| elo | yeah… doesn't sound right… | 20:33 |
| stupidnic | I can easily delete it and it has no impact on astara that I can determine | 20:34 |
| stupidnic | and I am certain that it is astara that is creating it | 20:34 |
| stupidnic | 2015-11-16 14:06:28.798 DEBUG akanda.rug.common.linux.utils:20622:pmain:tmain Running command: ['sudo', 'ip', '-4', 'addr', 'add', '192.168.1.1/24', 'brd', '192.168.1.255', 'scope', 'global', 'dev', 'ns-7d3ec6ac-3d'] execute /usr/local/lib/python2.7/dist-packages/akanda/rug/common/linux/utils.py:74 | 20:35 |
| stupidnic | taking a look at the interface code... it would seem that it is creating a bridge interface for management network on the controller (which it should), but also creating a bridge interface for the external network (which it should not). | 20:44 |
| stupidnic | Okay. Found it. There is a default in the rug.ini file called "plug_external_port=True" | 20:53 |
| stupidnic | For a flat network that should be false. | 20:54 |
| stupidnic | I am guessing the default settings are set to work with devstack | 20:54 |
| *** sarob_ has joined #akanda | 20:59 | |
| *** sarob_ has quit IRC | 22:05 | |
| *** clett has quit IRC | 22:54 | |
| *** rods has quit IRC | 22:54 | |
| *** rods has joined #akanda | 22:59 | |
| *** X-Istence has joined #akanda | 23:01 | |
| *** x58 has quit IRC | 23:06 | |
| *** shashank_hegde has joined #akanda | 23:17 | |
| *** elo has quit IRC | 23:32 | |
| *** clett has joined #akanda | 23:54 | |
| *** adam_g changes topic to "OpenStack Astara (formely Akanda) ** CHANNEL MOVED ** Join #openstack-astara" | 23:55 | |
| *** adam_g sets mode: +m | 23:56 | |
| adam_g | OK! #openstack-astara is officially registered and setup with openstack bots. lets all migrate over there now and be done with this place! | 23:57 |
| adam_g | davidlenwell, clett cleverdevil dhellmann drwahl mandoonandy markmcclain mugu rods sarob shashank_hegde stanchan stupidnic t0mb0 X-Istence | 23:57 |
| adam_g | thanks! | 23:57 |
| *** cleverdevil has left #akanda | 23:58 | |
| *** mugu has left #akanda | 23:58 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!