Friday, 2015-06-05

*** markvoelker has joined #openstack-ansible00:13
*** markvoelker has quit IRC00:18
*** JRobinson__ has quit IRC00:19
*** dkalleg has quit IRC00:45
*** galstrom_zzz is now known as galstrom00:47
*** dkalleg has joined #openstack-ansible01:45
*** dkalleg has quit IRC01:50
*** galstrom is now known as galstrom_zzz01:57
*** markvoelker has joined #openstack-ansible02:02
*** radek_ has joined #openstack-ansible02:05
*** markvoelker has quit IRC02:07
*** radek_ has quit IRC02:17
*** daneyon_ has quit IRC02:29
*** javeriak has quit IRC02:36
*** stevemar has quit IRC02:40
*** galstrom_zzz is now known as galstrom02:45
*** JRobinson__ has joined #openstack-ansible03:23
*** markvoelker has joined #openstack-ansible03:50
*** markvoelker has quit IRC03:55
*** galstrom is now known as galstrom_zzz04:32
*** markvoelker has joined #openstack-ansible05:39
*** markvoelker has quit IRC05:44
evrardjpmorning everyone06:56
svgHi all07:15
*** javeriak has joined #openstack-ansible07:18
*** markvoelker has joined #openstack-ansible07:28
*** JRobinson__ has quit IRC07:28
*** markvoelker has quit IRC07:33
evrardjphello svg07:34
* svg still does death dances around network interfaces07:36
odyssey4melol, good morning07:37
*** javeriak has quit IRC07:37
evrardjpsvg, for that part, I don't know if it could be related, but did you try load testing with something like iperf on your hardware?07:39
evrardjpjust to see if everything is fine07:39
evrardjpeven on containers07:39
evrardjpand in both ways07:39
svgactuall couple of colleagues are testing the hell out of it07:43
svgso far no cigar07:43
svgThe thing we are wondering now, is something show in the exampple interfaces config in osad08:05
svgit show bond1.1998 being member of br-vxlan, and bond1 being member of br-vlan08:06
svgbut there are lots of references that explain this cannot work: combining a vlan tagged interface in a bridge and that same interface untagged in another bridge08:08
svgsee https://wiki.debian.org/NetworkConfiguration#Caveats_when_using_bridging_and_vlan08:09
svgwe have our intrfaces setup as per default in that example, except we use only 1 bond008:09
odyssey4mesvg it's possible that the example file has some mistakes08:14
svgperhaps, but then I'm not sure how we would be supposed to configure br-vlan08:15
svgexcept for using a physicaly separate interface/bond08:16
svg(if this is true)08:16
evrardjpis this what the example shows?08:19
evrardjpI thought the bond was only for physical layer, and they used vlans on top to configure br-vlan and br-vxlan08:19
svgno, as per ^^ bond1 does bioth08:19
evrardjpCould you resend the link with the (possibly faulty) documentation?08:20
evrardjphttp://docs.rackspace.com/rpc/api/v10/bk-rpc-installation/content/sec_overview_neutron.html08:21
evrardjpbecause when you look at that08:23
evrardjphttp://docs.rackspace.com/rpc/api/v10/bk-rpc-installation/content/sec-hosts-target-network-refarch.html08:23
evrardjpit seems that it's not wrong08:23
svghttps://github.com/stackforge/os-ansible-deployment/blob/kilo/etc/network/interfaces.d/openstack_interface.cfg.example08:23
evrardjpthe vlan are defined on top of the bond08:23
evrardjpthen bridged08:24
svgthat is consistent with those docs yes08:24
svgI'm not saying iut is wrong, only that it seems wrong If that Debian doc is right08:24
svghttps://wiki.debian.org/NetworkConfiguration#Caveats_when_using_bridging_and_vlan08:25
svgand it sounds like this might be the cause of our troubles08:25
evrardjpfirst, I'd like to say that sometimes ubuntu and debian are inconsistent in terms of documentation vs the actual implementation08:27
evrardjpdid you try without the bonding? just to make sure it works?08:28
evrardjpI mean just the bridging the vifs with the NICs08:30
evrardjpit's maybe only bonding that's causing you the pain08:33
*** sdake has quit IRC08:34
svgwe did try to keeponly one interface in the bond08:36
svgthis rackspace document also cocnfirms this http://www.rackspace.com/blog/vms-vlans-and-bridges-oh-my-part-2/08:39
evrardjpI meant not configuring the bond at all08:49
evrardjpthe link you're giving is from 2012... I guess they were also changes in the ubuntu 14.04 release08:52
*** markvoelker has joined #openstack-ansible09:17
*** markvoelker has quit IRC09:22
openstackgerritMerged stackforge/os-ansible-deployment: Allow the Horizon SESSION_TIMEOUT to be configured  https://review.openstack.org/18814009:24
*** markvoelker has joined #openstack-ansible10:17
*** markvoelker has quit IRC10:23
*** markvoelker has joined #openstack-ansible11:18
*** markvoelker has quit IRC11:22
openstackgerritMerged stackforge/os-ansible-deployment: Allow the Horizon SESSION_TIMEOUT to be configured  https://review.openstack.org/18814411:54
cloudnullMorning.11:54
cloudnullSvg the file https://github.com/stackforge/os-ansible-deployment/blob/master/etc/network/interfaces.d/openstack_interface.cfg.example is the interface file that has been made generic but is very very close to what we have running in production, last I was aware of.11:58
evrardjpcloudnull: morning11:59
odyssey4memorning cloudnull :)11:59
vincent_vdkthis is what i've always seen : evrardjp> I thought the bond was only for physical layer, and they used vlans on top to configure br-vlan and br-vxlan11:59
cloudnullIn this case bond1 is serving vxlan and vlan using a tagged and untagged interface.11:59
cloudnullMorning gents.11:59
vincent_vdkgood afternoon :)12:00
cloudnullThat too ;)12:00
cloudnullWith bond1 the untagged bridge is given directly to neutron for provider networks which it uses with its set vlan tags.12:01
evrardjpcloudnull: very close is not equal12:01
evrardjpand apparently ubuntu is kinda difficult with its /etc/network/interfaces12:02
cloudnullIts about 98% there.12:02
cloudnullWe setup an additional storage network for backups.12:03
evrardjpsvg: the last time I had an issue with my bonding interface, I had to upgrade the firmware of the NICs12:03
evrardjpit fixed everything12:03
evrardjpmaybe it could help12:03
evrardjpmaybe not12:03
cloudnullAnd if the customer wants flat networks we do a little something like  https://github.com/stackforge/os-ansible-deployment/blob/master/etc/network/interfaces.d/aio_interfaces.cfg12:03
cloudnullFor the for the br-vlan12:04
cloudnullTo float a beth pair off the bridge for the flat network to consume.12:04
evrardjpinteresting12:04
cloudnullEvrardjp we had that issue to when trying to use melnox NICs.12:04
cloudnullAnd so.e Intel models.12:05
cloudnull*some12:05
evrardjptoo bad I didn't have your feedback at that time :p12:05
cloudnullNext time for sure ;)12:05
evrardjpanyway I was just trying to help12:05
cloudnullThat's awesome.12:06
cloudnullI just came work from my slumber logged in and thought it good to jiberjaber about network interfaces in Ubuntu first thing. :)12:07
cloudnullOn an unrelated note.12:08
cloudnullHow's everyone doing on this fine friday ?12:08
svgwell according to that rackspace site I linked ^^ one should not have a bond0 and a bond0.X in bridges12:09
cloudnullApsu wrote that blog post. When he awakes from his slumber maybe he can chime in on it.12:09
svgbut that is the case in the .example interfaces file12:10
cloudnullAlso I remember that to be true for 12.04 , it required some ebtables mungering to make it go.12:11
cloudnullBut in 14.04 we've not had issue with it.12:11
cloudnullEvrardjp did klevinstine or sam-i-am get the doc issues sorted ?12:14
cloudnullI failed to check back on that yesterday.12:14
evrardjpcloudnull: didn't check that atm12:14
evrardjpI'll do the followup12:14
evrardjpI may not do that immediately though12:15
cloudnullNo worries. Much appreciated.12:15
svganyway, our testing seems to not indicate we have issues with specificaaly this12:16
svga coworker noticed the fan getrs hit when stuff happens on the management bridge12:16
svghe now split out the bond in bond 0 and 1 with each 1 interface, and put the management bridge on a separate bond, and at first (now on one machine) that seems to fix stuff12:17
svg(.. ongoing..)12:17
evrardjpit's (I think) the best practice to split tenant traffic and management traffic12:18
odyssey4mesvg in a former life, we used to run everything on a bonded pair of 10GB NIC's... we still had eth0 as a 1GB NIC which was just used for automated builds and management12:19
evrardjprolls royce would be to split management and storage, but yes, interfaces12:19
svgwe have two 10gb nics in one bond0, so yes, same thing12:20
svgevrardjp: you mean split physically on separate interfaces?12:20
evrardjpBonding is always better because it brings resiliency. But if you can afford separate interfaces for management and tenants traffic, it's really better12:22
evrardjp-but12:23
*** sc has quit IRC12:23
*** sc has joined #openstack-ansible12:26
*** markvoelker has joined #openstack-ansible12:34
*** markvoelker has quit IRC12:38
svgyes, but why?12:41
svgnm, I see why obviosuly12:41
*** KLevenstein has joined #openstack-ansible13:33
*** jaypipes is now known as leakypipes13:43
openstackgerritMerged stackforge/os-ansible-deployment: Allow the Horizon SESSION_TIMEOUT to be configured  https://review.openstack.org/18814313:47
*** Mudpuppy has joined #openstack-ansible13:54
*** Mudpuppy has quit IRC13:55
*** Mudpuppy_ has joined #openstack-ansible13:55
*** Mudpuppy_ is now known as Mudpuppy13:58
*** markvoelker has joined #openstack-ansible14:05
*** markvoelker has quit IRC14:09
*** sigmavirus24_awa is now known as sigmavirus2414:12
openstackgerritgit-harry proposed stackforge/os-ansible-deployment: Enable udev for lvm in cinder-volume container  https://review.openstack.org/18879714:33
Apsusvg: Yeah, that blog post applied to the kernels at the time. About a year later or so, they fixed the issue.14:35
ApsuI think they fixed it in... 3.11 or so.14:35
svgthx, good to know14:35
cloudnullboom the man the myth the legend Apsu in da house !14:35
ApsuI was pretty surprised when it happened, lol14:35
Apsucloudnull: ohai2uthar!14:36
cloudnullhows it brother ?14:36
svgeither way, not sure if the part where mixing bond0 and bond0.x is bad still applies14:36
ApsuNot too bad. Clawing my way out of my melatonin-induced zombieism14:36
openstackgerritMerged stackforge/os-ansible-deployment: Moved all of group_vars into a variable file  https://review.openstack.org/18807414:36
svgour tsesting didnt show that14:36
Apsusvg: It doesn't.14:36
svgso far it seems we have a problem having the mgmt bridge on the same bond as other things, which is very weird14:37
svgApsu: do you have some 'official' reference to that?14:37
ApsuWhat do you mean exactly?14:37
Apsusvg: I can try to find a patch reference14:38
svgso originally we have a single bond0, which is used for all the rbdiges14:38
svgand we experience packet loss14:38
svgwhich was seen at deploy time a bit, but becomes very obvious once we start e.g. launching 100 vm instances14:39
ApsuOk.14:39
svgwhen removing all the container veth interfaces from the mgmt bridge, the packet loss disappears14:39
ApsuWhat kernel? What NICs?14:40
ApsuWhat bonding mode?14:40
svg3.13.0-46-generic, bond mode 1 (active passive), and nics are Intel 10GB (AOC-XEH-iN2, http://supermicro.com/products/superblade/networking/AOC-XEH-iN2.cfm)14:42
svgthe current testing-workaround was to take one nic out of the bond, create a second bond1 with it, and put that (vlantagged) into the mgmt bridge interface instead of its bond0 counterpart14:43
svgwith that setup we have no packet loss whatsovere14:43
*** stevemar has joined #openstack-ansible14:44
ApsuHrm.14:44
svgYes.14:45
ApsuDid you check the bond stats and individual link stats to see if it was one particular NIC misbehaving?14:47
ApsuY'know, cat /proc/net/bonding/bond0, and ip -s -s link show $nic1, ... $nic214:48
svgnot sure (didnt troublkeshoot this myself) but I believe we had this when any of the nics was primary14:50
andymccronce cinder 1.2.2 client is released we need to update the openstack_clients.yml :)14:51
svgWhat we did see, is lots of dropped packets on the passive nic14:53
Apsusvg: That's particularly strange, since it shouldn't be passing any :P14:53
svgyes14:53
ApsuAssuming it's actually passive14:53
Apsulol14:53
svgthat's what /proc/net/bonding/bond0 says :)14:54
ApsuPretty good sign then14:57
ApsuHow are you detecting packet loss on the passive link?14:57
svgdropped packets, on the ifconfig output15:02
ApsuCan you show me the output of ip -s -s link show $link?15:08
ApsuAlso, in a network arch meeting, so afkish15:08
*** cbaesema has quit IRC15:17
evrardjpsvg: another tool you could use is mii_tool15:29
evrardjpto see the current mii status15:29
evrardjpanyway I'm off for today, see you next week!15:30
*** markvoelker has joined #openstack-ansible15:30
*** markvoelker has quit IRC15:34
*** daneyon has joined #openstack-ansible15:36
*** daneyon has quit IRC15:38
cloudnulllater evrardjp, have a good weekend15:38
*** daneyon has joined #openstack-ansible15:38
openstackgerritgit-harry proposed stackforge/os-ansible-deployment: Enable udev for lvm in cinder-volume container  https://review.openstack.org/18839415:41
svgApsu: http://sprunge.us/PjPX15:41
*** faman has joined #openstack-ansible15:51
*** faman has quit IRC15:53
*** famanfortune has joined #openstack-ansible15:55
Apsusvg: So, those drops on your p1p2 bond slave are all RX15:59
ApsuAnd it's basically the same # of total RX packets16:00
ApsuWhich means something is sending traffic into it from the switch.16:00
ApsuPresumably because it's got the same MAC address... now I thought that mode 1 bonding didn't team the MACs up to be the same between active and passive slaves16:01
Apsusvg: Can you paste the output of cat /proc/net/bonding/bond0? :D16:01
ApsuAlso, are these links hooked into the same switch...?16:02
svghttp://paste.ubuntu.com/11590869/16:04
svgthese are blades, interfaces are connected to two separate internal chassis swicthes16:05
svgAFAIK those don;t support LACP which is why we do active passive16:05
svgbut I don;t know all detaiuls there16:06
svg(atm on train, flacky connection)16:06
*** dkalleg has joined #openstack-ansible16:08
Apsukk16:08
Apsusvg: Ok, so the bonding driver knows the links have different MACs, it's just displaying them as teamed because of the bond. Cool.16:09
ApsuSo in theory the switches shouldn't be sending traffic on both links for active/backup mode...16:10
ApsuBut that's where those drops are coming in16:10
ApsuWhen using mode 1 bonding on Linux, you generally don't tell the switches anything at all about it16:10
*** dkalleg has quit IRC16:12
svgso basically the lost packets would be caused by the switch sending packets to the wrong port, which would mean it somehow got the wrong ip address?16:15
svgI have been wondering where those Link Failure Counts come from also16:16
ApsuNo it's not the wrong IP16:16
ApsuThis is purely a layer 2 thing16:16
svgsorry I meant mac address16:16
ApsuEssentially it means the switch is sending frames to the MAC, because it saw that MAC on frames entering that switch port, for the backup link16:17
ApsuMaybe your blade switches aren't communicating their MAC tables with each other very well.16:17
ApsuOr at all16:17
ApsuSo they're operating as independent switches, and incoming traffic to your bond MAC is being sent to both bond slaves always :P16:17
svgI didn't think they were independent; that would be #$*&@16:18
ApsuThe number of RX frames on both slaves is almost the same, with a bit more on the active link16:18
ApsuSo it's nearly what I said, that all incoming traffic comes to both links16:19
ApsuVery odd.16:19
svgso the so called workaround of having the interfaces split out in separate bonds just happens to solve this as there is only one interface in the bonds16:20
svgwhich as nothing to do with manage,ment bridge16:20
svgmanagement bridge caries the most traffic when we start to deploy and configure things, so that might just be more obvious there16:21
ApsuYep16:21
ApsuYou're probably getting actual communication issues -- which wouldn't happen if it were just mirroring all traffic to both links, and the passive slave was dropping it all -- because it's not mirroring16:22
ApsuIt's probably alternating or something similar, and the passive drops everything16:22
ApsuSo the active has to constantly request TCP retransmits and shrink window sizes, slowing traffic to a crawl16:23
ApsuWhich fails when you have a ton of deployment traffic :D16:23
Apsuyay networking16:23
*** javeriak has joined #openstack-ansible16:24
ApsuI bet the blade switches are somewhat independent, and setup active-active16:24
ApsuWhich is either actively balancing traffic by MAC or or the MAC tables are flapping thinking the same MAC is on both switches16:25
ApsuAre you certain they don't support MLAG LACP?16:25
svgthats what the network guy told me16:27
Apsusvg: You could always verify by tcpdump/wireshark/tshark and check for a bunch of duplicates/retransmits/window shrinking events16:27
svgI think I should have a look at it myself16:27
ApsuThere are also other bonding modes that may be appropriate if it doesn't do MLAG16:27
*** javeriak_ has joined #openstack-ansible16:29
*** javeriak has quit IRC16:29
svgyou mean better appropriate in this case than mode1?16:32
*** javeriak has joined #openstack-ansible16:35
ApsuRight16:37
ApsuDepending on how your switches are actually behaving, mode0 or mode6 may be better.16:37
Apsumode6 (ALB) has some other considerations to take into account16:37
*** sdake has joined #openstack-ansible16:39
*** javeriak_ has quit IRC16:40
*** sdake_ has joined #openstack-ansible16:44
*** sdake has quit IRC16:48
*** famanfortune has quit IRC17:03
*** javeriak_ has joined #openstack-ansible17:09
*** javeriak has quit IRC17:13
*** dkalleg has joined #openstack-ansible17:18
*** markvoelker has joined #openstack-ansible17:19
*** markvoelker has quit IRC17:24
*** famanfortune has joined #openstack-ansible17:25
*** galstrom_zzz is now known as galstrom17:33
*** javeriak_ has quit IRC17:35
*** galstrom is now known as galstrom_zzz17:38
*** galstrom_zzz is now known as galstrom17:41
*** galstrom is now known as galstrom_zzz17:44
*** sdake has joined #openstack-ansible17:50
*** sdake_ has quit IRC17:54
*** sdake_ has joined #openstack-ansible17:56
*** jwagner_away is now known as jwagner17:59
*** sdake has quit IRC17:59
*** javeriak has joined #openstack-ansible18:04
*** sdake_ is now known as sdake18:10
*** galstrom_zzz is now known as galstrom18:11
*** serverascode has quit IRC18:54
*** metral is now known as metral_zzz18:55
*** cbaesema has joined #openstack-ansible19:07
*** sdake has quit IRC19:08
*** markvoelker has joined #openstack-ansible19:08
*** metral_zzz is now known as metral19:09
*** markvoelker has quit IRC19:12
*** abitha has joined #openstack-ansible19:28
abithahi, I am using os_image module to upload an image to glance and having issues to figure out the right path for the file to exist. should it exist in some specific path in ansible project , or should it be copied manually to the target machine and ran from there?19:32
*** leakypipes has quit IRC19:34
cloudnullif the image was uploaded to glance and the task succeeded and your using the glance file backend you should be able to find the image at /var/lib/glance/images19:35
cloudnullor something similar19:35
abithathis is when using create_image, while specifying the image 'filename', its not able to access an image that is in the ansible project structure19:37
abithabut if I copy the image to the target machine and give the absolute path it works fine. is this how its supposed to be?19:38
cloudnullto be quite honest idk , this is channel is geared at the os-ansible-deployment project19:39
abithaoh ok. thanks19:39
cloudnullmaybe give http://docs.ansible.com/glance_image_module.html a go19:40
cloudnullthat module has the "file" attribute which should be able to follow your local directory structure and upload the miage.19:40
cloudnull*image19:41
abithathanks @cloudnull19:41
*** moravec has quit IRC19:46
*** moravec has joined #openstack-ansible19:47
*** abitha has left #openstack-ansible19:55
openstackgerritMerged stackforge/os-ansible-deployment: Enable udev for lvm in cinder-volume container  https://review.openstack.org/18879720:05
*** javeriak has quit IRC20:18
cloudnullstevelle: https://github.com/cloudnull/osad-basic-test-node/blob/master/osad-basic-test-aio-playbook.yml#L9420:19
*** metral is now known as metral_zzz20:22
cloudnullstevelle: sigmavirus24: https://gist.github.com/cloudnull/732534820:23
*** markvoelker has joined #openstack-ansible20:24
*** metral_zzz is now known as metral20:27
*** markvoelker has quit IRC20:28
*** annashen has joined #openstack-ansible20:34
*** famanfortune has left #openstack-ansible20:36
*** famanfortune has joined #openstack-ansible20:43
*** serverascode has joined #openstack-ansible20:50
openstackgerritSteve Lewis proposed stackforge/os-ansible-deployment: Configure DB addresses for each service  https://review.openstack.org/18892621:13
openstackgerritKevin Carter proposed stackforge/os-ansible-deployment: Prep for 10.1.8 tag  https://review.openstack.org/18817321:15
*** toddnni has joined #openstack-ansible21:16
*** Mudpuppy_ has joined #openstack-ansible21:29
*** Mudpuppy has quit IRC21:32
*** Mudpuppy_ has quit IRC21:33
openstackgerritgit-harry proposed stackforge/os-ansible-deployment: Enable udev for lvm in cinder-volume container  https://review.openstack.org/18839421:34
*** openstack has joined #openstack-ansible21:43
openstackgerritKevin Carter proposed stackforge/os-ansible-deployment: Updated all of master to the latest SHAs  https://review.openstack.org/18893721:51
*** javeriak has joined #openstack-ansible21:55
*** KLevenstein has quit IRC22:06
*** stevemar has quit IRC22:12
*** markvoelker has joined #openstack-ansible22:12
*** markvoelker has quit IRC22:17
*** annashen has quit IRC22:19
openstackgerritKevin Carter proposed stackforge/os-ansible-deployment: Updated kilo for new dev work  https://review.openstack.org/18895422:22
* svg proposes clouds without networking22:28
stevelle+122:28
stevellemajor boon for security there22:29
cloudnullaccording to defcore you dont have to have networking to be an OpenStack cloud .22:29
* svg proposes ethernet switches with one single, simple, standardized configuration system22:31
svgthose should have neat html5 interfaces22:32
*** daneyon has quit IRC22:33
*** daneyon has joined #openstack-ansible22:33
*** daneyon has quit IRC22:33
cloudnulllol22:34
*** daneyon has joined #openstack-ansible22:34
*** openstackgerrit has quit IRC22:37
*** openstackgerrit has joined #openstack-ansible22:37
openstackgerritSteve Lewis proposed stackforge/os-ansible-deployment: Configure DB addresses for each service  https://review.openstack.org/18892622:46
*** famanfortune has quit IRC22:46
openstackgerritJaveria Khan proposed stackforge/os-ansible-deployment: Adding PLUMgrid plugin option to neutron  https://review.openstack.org/18896422:54
javeriakguys, i went ahead and commited the changes ^, for the plumgrid BP, hope this gives you more clarity on what it was asking for22:55
*** annashen has joined #openstack-ansible23:20
openstackgerritMerged stackforge/os-ansible-deployment: Update for 11.0.2 release  https://review.openstack.org/18893423:21
openstackgerritMerged stackforge/os-ansible-deployment: Prep for 10.1.8 tag  https://review.openstack.org/18817323:21
*** sdake has joined #openstack-ansible23:26
*** annashen has quit IRC23:26
openstackgerritKevin Carter proposed stackforge/os-ansible-deployment: Updated kilo for new dev work  https://review.openstack.org/18895423:31
*** daneyon has quit IRC23:34
*** dkalleg has quit IRC23:52
*** sdake_ has joined #openstack-ansible23:57
*** galstrom is now known as galstrom_zzz23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!