Monday, 2016-04-25

*** sacharya has quit IRC00:13
*** oneswig has joined #openstack-ansible00:13
*** itlinux has joined #openstack-ansible00:16
*** sacharya has joined #openstack-ansible00:18
*** oneswig has quit IRC00:18
*** itlinux has quit IRC00:37
*** sacharya has quit IRC00:49
*** sacharya has joined #openstack-ansible00:56
*** iceyao has joined #openstack-ansible01:02
*** iceyao_ has joined #openstack-ansible01:03
*** sacharya has quit IRC01:03
*** iceyao has quit IRC01:07
*** sacharya has joined #openstack-ansible01:07
*** itlinux has joined #openstack-ansible01:12
*** itlinux has quit IRC01:23
*** galstrom_zzz is now known as galstrom01:35
*** Mudpuppy has joined #openstack-ansible01:44
*** markvoelker has joined #openstack-ansible01:57
*** Mudpuppy has quit IRC01:59
*** Mudpuppy has joined #openstack-ansible02:00
*** markvoelker_ has joined #openstack-ansible02:00
*** markvoelker has quit IRC02:02
*** Mudpuppy has quit IRC02:04
*** itlinux has joined #openstack-ansible02:07
*** johnmilton has quit IRC02:08
*** sacharya has quit IRC02:10
*** markvoelker has joined #openstack-ansible02:10
*** sacharya has joined #openstack-ansible02:13
*** markvoelker_ has quit IRC02:14
*** oneswig has joined #openstack-ansible02:15
*** itlinux has quit IRC02:16
*** galstrom is now known as galstrom_zzz02:19
*** oneswig has quit IRC02:21
*** markvoelker_ has joined #openstack-ansible02:25
*** markvoelker has quit IRC02:27
*** markvoel_ has joined #openstack-ansible02:32
*** markvoelker_ has quit IRC02:34
*** markvoelker has joined #openstack-ansible02:34
*** markvoel_ has quit IRC02:38
*** sacharya has quit IRC02:41
*** sacharya has joined #openstack-ansible02:43
*** persia has quit IRC02:52
*** persia has joined #openstack-ansible02:58
*** ChrisBenson has quit IRC03:24
*** ChrisBenson has joined #openstack-ansible03:24
*** weezS has joined #openstack-ansible03:24
*** oneswig has joined #openstack-ansible03:25
*** aludwar has quit IRC03:26
*** sacharya has quit IRC03:26
*** sacharya has joined #openstack-ansible03:31
*** ChrisBenson has quit IRC03:33
*** ChrisBenson has joined #openstack-ansible03:35
*** sacharya has quit IRC03:36
*** sacharya has joined #openstack-ansible03:36
*** markvoelker_ has joined #openstack-ansible03:51
*** markvoelker has quit IRC03:54
*** julian1 has quit IRC03:55
*** julian1 has joined #openstack-ansible03:56
*** sacharya has quit IRC04:12
*** sacharya has joined #openstack-ansible04:17
*** weezS has quit IRC04:25
*** oneswig has quit IRC04:27
pellaeonwhen I run nova client it reports     from oslo.utils import encodeutils04:29
pellaeonImportError: No module named utils04:29
pellaeonjust checked, my python-novaclient is still at 2.23.304:30
pellaeonany idea why this happened?04:31
*** sacharya has quit IRC04:36
*** sacharya has joined #openstack-ansible04:38
*** daneyon_ has quit IRC04:48
*** kiranv_ has joined #openstack-ansible04:53
*** aludwar has joined #openstack-ansible04:55
*** sacharya has quit IRC05:05
*** sacharya has joined #openstack-ansible05:06
*** saneax_AFK is now known as saneax05:23
*** sacharya has quit IRC05:26
*** sacharya has joined #openstack-ansible05:29
pellaeonjust re-run repo-build and pip install -I python-novaclient , everything works now :-/05:36
*** markvoelker_ has quit IRC05:43
*** sacharya has quit IRC05:43
*** sacharya has joined #openstack-ansible05:43
*** sacharya has quit IRC05:57
*** sacharya has joined #openstack-ansible06:01
*** javeriak has joined #openstack-ansible06:06
*** maestro has joined #openstack-ansible06:11
*** markvoelker has joined #openstack-ansible06:12
*** oneswig has joined #openstack-ansible06:13
*** sacharya has quit IRC06:14
*** markvoelker has quit IRC06:17
*** oneswig has quit IRC06:17
*** dalees has quit IRC06:17
*** javeriak has quit IRC06:18
*** sacharya has joined #openstack-ansible06:18
*** dalees has joined #openstack-ansible06:24
*** javeriak has joined #openstack-ansible06:24
*** jiteka has joined #openstack-ansible06:26
*** sdake has joined #openstack-ansible06:36
*** joker_ has joined #openstack-ansible06:40
*** saneax is now known as saneax_AFK06:44
*** sacharya has quit IRC06:50
*** saneax_AFK is now known as saneax06:53
*** sacharya has joined #openstack-ansible06:53
*** iceyao has joined #openstack-ansible07:02
*** iceyao_ has quit IRC07:05
*** javeriak has quit IRC07:07
*** sdake_ has joined #openstack-ansible07:09
*** sdake has quit IRC07:12
*** jiteka has quit IRC07:14
*** jiteka has joined #openstack-ansible07:15
*** sacharya has quit IRC07:16
*** saneax is now known as saneax_AFK07:25
*** kiranv_ has quit IRC07:30
*** javeriak has joined #openstack-ansible07:30
*** kiranv_ has joined #openstack-ansible07:32
*** d9k has quit IRC07:33
*** maestro has quit IRC07:34
*** saneax_AFK is now known as saneax07:35
*** sdake_ has quit IRC07:35
*** d9k has joined #openstack-ansible07:37
*** sdake has joined #openstack-ansible07:39
*** Oku_OS-away is now known as Oku_OS07:39
*** admin0 has joined #openstack-ansible07:40
*** mikelk has joined #openstack-ansible07:42
*** kiranv_ has quit IRC07:45
*** sdake_ has joined #openstack-ansible07:47
*** sdake has quit IRC07:51
*** maestro has joined #openstack-ansible07:54
admin0good morning07:56
matttmorning admin008:00
admin0@ the summit ?08:00
*** sdake_ is now known as sdake08:01
*** iceyao_ has joined #openstack-ansible08:05
*** javeriak has quit IRC08:06
*** iceyao has quit IRC08:08
evrardjpgood morning everyone08:09
*** javeriak has joined #openstack-ansible08:10
*** javeriak has quit IRC08:10
*** fxpester has joined #openstack-ansible08:10
matttadmin0: not i08:12
admin0morning evrardjp , mattt08:13
*** persia has quit IRC08:19
*** sacharya has joined #openstack-ansible08:21
admin0evrardjp: i cherry picked the lastest from the SSL patch by cloudnull .. could not get horizon to work @ all . is there a way i can tell horizon not to use external ssl but use its own08:24
admin0in that way, i think i can have SSL of all endpoints via this patch, and still have SSL on horizon on its own08:24
evrardjpadmin0 latest change of cloudnull works08:24
evrardjpyou mean passthrough?08:25
evrardjpI think it's gonna take you some time to make it work08:25
admin0either way is fine with me :)08:25
evrardjpbut if you remove the variable08:25
evrardjpand try again08:25
evrardjpit's gonna be fine08:25
*** sacharya has quit IRC08:25
admin0evrardjp: this morning, i started in a completely new environment,  master branch .. cherry pick of the latest SSL patch and this user_var file: https://gist.github.com/a1git/8c9ad6cd8f5e62319ee0b1774e6a133508:26
*** iceyao_ has quit IRC08:27
*** iceyao_ has joined #openstack-ansible08:27
evrardjpthis should work08:27
evrardjpwant me to rebuild one for further testing?08:28
admin0well, let me see if mine works08:29
admin0if it does, then I will rebuild with 1 more ..08:29
admin0if it does not, then either. 1. i add your key to my test env and you can play here :D08:30
admin0or build yours08:30
*** persia has joined #openstack-ansible08:32
*** sdake has quit IRC08:40
*** persia has quit IRC08:53
*** gparaskevas has joined #openstack-ansible08:53
evrardjphello gparaskevas08:54
gparaskevasHello evrard :)08:54
gparaskevasnnot in the summit?08:54
admin0evrardjp: stil 503  ..08:55
evrardjpgparaskevas not this time08:55
*** markvoelker has joined #openstack-ansible08:55
evrardjpadmin0 I'm building mine08:55
gparaskevasme neither :(08:55
evrardjpgparaskevas I've seen this assigned to you: https://bugs.launchpad.net/openstack-ansible/+bug/144643408:55
openstackLaunchpad bug 1446434 in openstack-ansible trunk "Provide RabbitMQ management UI through HAProxy" [Wishlist,In progress] - Assigned to George Paraskevas (gparaskevas)08:55
evrardjpis that still in your wishlist/stuff to do?08:55
gparaskevasYes it is but havent find time to crate a blueprint and work on it08:56
admin0evrardjp: are you doing an AIO ?08:56
*** persia has joined #openstack-ansible08:57
*** markvoelker has quit IRC09:00
*** electrofelix has joined #openstack-ansible09:01
evrardjpadmin0 yup09:03
admin0if it works, it works :)09:03
admin0waiting09:03
evrardjpgparaskevas no problem I'm just wondering if we have lost that from our scope or not09:04
gparaskevasit also depends on that09:07
gparaskevashttps://review.openstack.org/#/c/276672/09:07
pellaeonHi, just upgraded from kilo to liberty, my internal networks seems to be broken, the router can't seem to get ARP reply from the vm09:08
pellaeonbut the vm did get reply from qrouter09:09
evrardjphey pellaeon could this be a problem with l2pop ?09:09
evrardjpgparaskevas I see09:10
pellaeonprobably, I'm not familiar with l2pop, any hints where I can look further ?09:10
evrardjpyeah you're opening pandoras box09:10
evrardjppellaeon It's not necessarily that09:10
evrardjpI just heard there was a switch in default variables09:11
evrardjpdue to l2pop mechanism being not really stable09:11
evrardjplet me find that review09:11
evrardjppellaeon could you check what's the value of your neutron_l2_population variable?09:13
evrardjpjust to know09:13
evrardjpbefore and after upgrade09:13
pellaeonafter upgrade it's default "False"09:14
pellaeonbefore upgrade I'm not sure, but I had never changed that variable09:14
evrardjpok let me check if we have upgrade notes for that09:15
admin0i think before it was true, and after upgrade, it is set to false09:17
*** oneswig has joined #openstack-ansible09:18
pellaeonadmin0: yep, just checked, default value for neutron_l2_population in kilo is True09:20
admin0yep ..09:21
admin0i tried to address this bug and took in the request, but after talking with the neutron guys, decided that it should be set to false09:21
admin0and then have the thing in notes09:21
admin0so by default, we must not set it to true09:21
pellaeonso... I set it to True and re-deploy ? will this break anything ?09:22
admin0i do not know the upgrade/change path from it set to true -> false .. i recommend set it to true, run the playbooks .. get your env back online .. and then talk in the neutron channel on how to migrate or change procedure of it from true to false, so that you can be compatible with future releases09:22
admin0 your env is already broken due to l2pop true => false .. you are doing it true => true now and fixing that09:23
admin0but best confirm with the neutron guys09:24
evrardjppellaeon did you check this ? http://docs.openstack.org/releasenotes/openstack-ansible/liberty.html09:24
evrardjpfor other possible deprecations ?09:24
pellaeonredeploying...09:25
pellaeonevrardjp: nope, checking it right now09:25
pellaeonredeployed, same situation09:41
*** markvoelker has joined #openstack-ansible09:49
*** tiagogomes has joined #openstack-ansible09:51
*** markvoelker has quit IRC09:53
*** tiagogomes has quit IRC09:56
openstackgerritMatt Thompson proposed openstack/openstack-ansible-os_cinder: Use tempest for functional testing  https://review.openstack.org/30991709:58
pellaeonevrardjp: vm's arp reply is blocked between compute host's vxlan-62 and eth1.30, I can see vm arp replies with tcpdump -i vxlan-62 but not eth1.3010:01
openstackgerritMatt Thompson proposed openstack/openstack-ansible-os_cinder: Use tempest for functional testing  https://review.openstack.org/30991710:01
evrardjpwhat's your eth1.30 ?10:02
evrardjpno bonding etc?10:03
evrardjpI don't know your env10:03
openstackgerritMatt Thompson proposed openstack/openstack-ansible-os_cinder: Use tempest for functional testing  https://review.openstack.org/30991710:04
pellaeonno bonding, eth1.30 implements vxlan, hosts the tunnel cidr_network, 172.29.240.0/22, as in example configuration10:05
*** jiteka has quit IRC10:09
evrardjpinteresting10:09
*** jiteka has joined #openstack-ansible10:11
evrardjppellaeon did you remove your interface before changing the variable neutron_l2_pop ?10:13
evrardjpip link delete vxlan-6210:13
pellaeonevrardjp: no i didn't, i'll try and redeploy10:13
evrardjppellaeon wait a sec10:15
pellaeonah, just did10:15
evrardjpI'd migrate away of l2pop10:15
evrardjpif you do that10:15
evrardjpoh10:15
evrardjpso I'd delete the vxlan link10:16
evrardjpchange the neutron_l2_population to False10:16
evrardjprun the os10:16
evrardjp-neutron-install10:16
pellaeoni noticed in compute node tcpdump -i br-vxlan there are a lot of packets like this:10:17
pellaeonIP 172.29.240.21.38582 > 172.29.240.31.8472: OTV, flags [I] (0x08), overlay 0, instance 6210:17
pellaeon240.21 is the compute node, 240.31 is the neutron agents container10:17
pellaeonim looking into the packets now10:18
evrardjpwhy do you think it's abnormal?10:19
evrardjp(not a neutron expert but I understand the basics, so feel free to enlighten me)10:21
pellaeonno, that's totally normal, I misunterstood tcpdump's output10:21
pellaeon18:14:54.937756 IP 172.29.240.21.38582 > 172.29.240.31.8472: OTV, flags [I] (0x08), overlay 0, instance 6210:21
admin0evrardjp:  build finished ?10:21
pellaeonIP 10.110.0.70 > 10.110.0.1: ICMP echo request, id 36099, seq 1091, length 6410:21
pellaeonwas the output10:21
evrardjpadmin0 nope still running10:22
admin0ok10:22
pellaeonthat's actually just one packet , outer layer VXLAN 172.29.240.21.38582 > 172.29.240.31.8472 , inside vxlan 10.110.0.70 > 10.110.0.1: ICMP echo request10:22
evrardjppellaeon you mean one ping generated plenty of vxlan packets? Not sure to understand your issue here10:23
evrardjpbut anyway, you should migrate away of l2pop if you have the opportunity to do it10:24
pellaeonno no, I meant I misunderstood the output of tcpdump, the two line output above i misunderstood it as 2 packets, but it's actually just normal ICMP echo req encapsulated in vxlan10:25
pellaeonso totally normal10:25
pellaeonredeploy done, vxlan-62 on compute node never came back10:25
pellaeonI'll just try to move away from l2pop as you suggested10:26
evrardjpmove away from it, it's better for your sanity :D10:31
evrardjpit's also the suggestion from upstream10:31
evrardjpneutron team10:32
evrardjpyou'll probably need help from them at some point if my help is not enough10:32
pellaeonredeployed, on compute node vxlan-62 was not recreated10:32
evrardjpredeployed neutron ?10:32
pellaeonyes, os-neutron-install10:33
evrardjpcould you restart your linux bridge agent?10:33
*** iceyao_ has quit IRC10:33
pellaeonyay it is created now10:34
pellaeonYAY everything works now10:35
pellaeonevrardjp: admin0 really thanks for the help :-D10:36
evrardjpyw10:37
*** markvoelker has joined #openstack-ansible10:43
admin0evrardjp: so we verified that a  run will take as long as it takes from flying from UK to NL :D10:47
admin0run here, take the flight, go there, demo :D or the other way around10:47
evrardjplol10:47
evrardjpit depends10:47
evrardjphere it's my repo build taking AGES10:48
evrardjpit's sometimes faster to just redelete and recreate10:48
evrardjpI'll do that10:48
*** markvoelker has quit IRC10:48
evrardjpbut I'm also busy doing other stuff you know :p10:48
evrardjpadmin0 hitting pip issues I am completely rebuilding10:57
*** Bofu2U has joined #openstack-ansible11:00
*** joker_ has quit IRC11:02
*** agireud has quit IRC11:03
*** swati has quit IRC11:04
*** joker_ has joined #openstack-ansible11:06
*** agireud has joined #openstack-ansible11:07
*** permalac has joined #openstack-ansible11:10
*** johnmilton has joined #openstack-ansible11:17
*** karimb has joined #openstack-ansible11:20
*** retreved has quit IRC11:24
*** javeriak has joined #openstack-ansible11:25
*** markvoelker has joined #openstack-ansible11:37
*** markvoelker has quit IRC11:42
*** markvoelker has joined #openstack-ansible11:50
*** Oku_OS is now known as Oku_OS-away11:51
*** markvoelker has quit IRC12:01
*** karimb has quit IRC12:05
*** Oku_OS-away is now known as Oku_OS12:15
openstackgerritJaveria Khan proposed openstack/openstack-ansible: Update neutron services dict override  https://review.openstack.org/30995212:17
*** psilvad has joined #openstack-ansible12:18
*** spotz_zzz is now known as spotz12:20
admin0evrardjp: still running ?12:21
evrardjpadmin0 failed at keystone12:22
admin0oh12:22
admin0cloudnull is @ summit i guess :)12:22
*** sacharya has joined #openstack-ansible12:22
evrardjpit's probably the beginning of the summit for them12:23
evrardjpso don't expect anything12:24
matttsomething broken?12:24
admin0mattt: in my case horizon12:25
*** sacharya has quit IRC12:27
matttadmin0: more specifically ?12:28
matttdoesn't build?  can't log in?12:28
admin0503 service unavailable12:28
evrardjpmattt rebuilding with latest cloudnull change doesn't work yet12:28
evrardjphaproxy does check at l7 and returns down12:28
evrardjp50312:28
evrardjpI think this is an acl issue12:29
evrardjpI'll check12:29
matttok so is the issue localized to cloudnull's review ?12:30
*** woodard has joined #openstack-ansible12:30
evrardjpyes12:33
evrardjpfor this one12:33
evrardjpdon't worry I'll handle this12:33
*** iceyao has joined #openstack-ansible12:35
admin0so is it an acl issue ? or a config issue ?12:38
evrardjpI dont' know yet12:42
evrardjpwhat I know12:42
evrardjpArgsAlreadyParsedError: arguments already parsed: cannot register CLI option12:42
evrardjpI'll check if everything is alright on standard master12:43
spotzyeah it's people:)12:43
evrardjphello spotz12:43
spotzevrardjp: Wish you were here! Had fun meeting everyone yesterday12:46
*** javeriak has quit IRC12:47
evrardjp:D12:48
*** git-harry has quit IRC12:48
evrardjpall the good comes to those who wait? (or something like that)12:48
*** git-harry has joined #openstack-ansible12:48
spotzcloudnull prometheanfire d34dh0r53 odyssey4me automagically andymccr and me are all here for sure. There might be others I didn't get handles for. Oh and mudpuppy and b3rnard012:49
evrardjpyes, the place to be12:49
evrardjp:D12:49
*** git-harry has quit IRC12:49
*** git-harry has joined #openstack-ansible12:50
*** toan has joined #openstack-ansible12:50
evrardjpspotz we can meet at barcelona if you go there :D12:51
spotzoh and how could I forget mhayden12:51
evrardjpand mancdaz too12:51
*** javeriak has joined #openstack-ansible12:51
spotzevrardjp: I'm going to try, trouble is I'm really a systems engineer and not a deveoper12:51
evrardjpis jmccrory there too?12:51
*** permalac_ has joined #openstack-ansible12:56
*** mgodeck has joined #openstack-ansible12:57
*** brad[] has quit IRC12:59
*** mgodeck has quit IRC12:59
*** permalac has quit IRC12:59
spotzhe might be. Just saw I'm down to 30% and not near a wall. Going to shu down in case I need to fix something. I'll log back in if I find somewhere to charge12:59
*** dalees has quit IRC13:02
evrardjpfine13:02
evrardjpenjoy the summit :D13:02
*** brad[] has joined #openstack-ansible13:02
*** dweaver has quit IRC13:03
*** woodard has quit IRC13:04
*** woodard has joined #openstack-ansible13:04
*** dweaver has joined #openstack-ansible13:06
*** spotz is now known as spotz_zzz13:09
*** dalees has joined #openstack-ansible13:14
*** woodard has quit IRC13:15
*** LanceHaig has quit IRC13:18
*** LanceHaig has joined #openstack-ansible13:18
*** spotz_zzz is now known as spotz13:19
*** schwicht has joined #openstack-ansible13:21
*** saneax is now known as saneax_AFK13:28
*** oneswig has quit IRC13:31
*** oneswig has joined #openstack-ansible13:31
*** weshay has joined #openstack-ansible13:34
openstackgerritMatt Thompson proposed openstack/openstack-ansible-os_tempest: Test os_tempest role deploys correctly  https://review.openstack.org/30997613:34
*** weshay has quit IRC13:35
openstackgerritMatt Thompson proposed openstack/openstack-ansible-os_tempest: Test os_tempest role deploys correctly  https://review.openstack.org/30997613:37
Bofu2Ugah, is there not a direct qcow download for windows 2012 eval? looks like cloudbase pulled it all behind the EULA agreement stuff13:40
*** oneswig has joined #openstack-ansible13:40
openstackgerritMatt Thompson proposed openstack/openstack-ansible-os_tempest: Install tempest plugins  https://review.openstack.org/30903413:41
evrardjpBofu2U wans't it like that before?13:41
evrardjpI think it must be that way in all cases13:41
Bofu2Uyeah but there was also a direct link that worked13:41
Bofu2Usucks.13:41
Bofu2Uas I download from my machine here over comcast, just to upload to my servers from comcast :P13:42
*** oneswig_ has joined #openstack-ansible13:42
*** karimb has joined #openstack-ansible13:43
*** sdake has joined #openstack-ansible13:43
*** thorst_ has joined #openstack-ansible13:44
matttis the gate hosed again?13:44
matttgit.openstack.org[0: 104.130.246.128]: errno=Connection timed out13:44
matttat least it can resolve it :)13:44
*** oneswig has quit IRC13:44
*** oneswig_ has quit IRC13:47
evrardjpmattt lol13:47
prometheanfirespotz: hi :D13:48
*** busterswt has joined #openstack-ansible13:54
*** spotz is now known as spotz_zzz13:55
*** oneswig has joined #openstack-ansible13:57
mhaydenprometheanfire: ohai13:58
prometheanfiremhayden: hi14:00
*** javeriak has quit IRC14:00
pellaeonevrardjp: it broke again :-(14:02
evrardjp?14:02
evrardjpwhat happened?14:02
pellaeonneither the compute node nor the agents are receiving broadcast packets14:03
pellaeon239.1.1.114:03
pellaeonbut normal unicasts works fine14:03
evrardjpthis is a multicast address right?14:04
pellaeonyes14:04
pellaeonas specified in vxlan_group, ml2_conf.ini14:04
pellaeonI didn't do anything and it broke on itself, restarting dhcp, l3, linuxbridge agents on every node didn't help14:05
evrardjpso now that you moved out of l2pop you got issues14:06
pellaeonyeah, ha14:06
pellaeontcpdump -i br-vxlan shows no packets coming from 239.1.1.114:09
evrardjppellaeon should you see it that way?14:09
pellaeonhmm I'm not sure how multicast packets should look like14:10
evrardjpI think they should be sent as unicast to the instances14:11
evrardjpin tcpdump check the EII address14:11
evrardjpit should be broadcast14:12
evrardjplet me check14:12
evrardjpmaybe you should ask neutron ppl here14:15
pellaeonOK I will14:16
evrardjpI don't have enough hardware nodes here to play with that right now14:17
*** elopez has joined #openstack-ansible14:18
pellaeonevrardjp: ok, thanks anyway14:19
evrardjppellaeon ensure you're in the right namespace when doing tcpdump14:21
evrardjpbut I guess you knew that14:22
pellaeonbr-vxlan is not in any namespace14:23
pellaeonon infra nodes it's outside the container14:23
*** sacharya has joined #openstack-ansible14:23
evrardjpyes ofc14:24
evrardjpbut if you want to see your broadcast you want to test it on different levels14:24
evrardjpI guess14:24
evrardjpthe easier to see what's flowing would be to connect on your qrouter namespace14:25
*** phalmos has joined #openstack-ansible14:25
*** sigmavirus24_awa is now known as sigmavirus2414:26
*** sacharya has quit IRC14:28
pellaeonfrom what I understand about neutron, when received on physical interface (in my case eth1.30), it is de-encapsulated and put in vxlan-6214:29
*** Brew has joined #openstack-ansible14:29
pellaeonthis is done by the kernel14:29
pellaeonbut on my physical interface eth1.30 doesn't see multicast packets coming in at the first place14:30
pellaeon(br-vxlan contains eth1.30)14:30
*** jayc_ has joined #openstack-ansible14:34
openstackgerritMatt Thompson proposed openstack/openstack-ansible-os_nova: Run specific tempest tests  https://review.openstack.org/30999114:35
evrardjpoh you meant the other way14:35
evrardjpyou mean general broadcast on your interface eth1.3014:36
evrardjpthis is by design14:36
evrardjpI'm not sure what you really mean14:36
busterswtWhen you switched from l2pop to multicast, how did you do that? Just change the config and restart services?14:37
pellaeonhttps://kimizhang.wordpress.com/2014/04/01/how-ml2vxlan-works/ i read this article, so from what I understand, if there's a broadcast packet coming from a VM, then it will be encapsulated into a vxlan multicast packet14:39
pellaeonthen physical hosts that are in the multicast group should receive that vxlan multicast packet and decapsulate it14:40
*** Brew has quit IRC14:41
pellaeonbusterswt: I removed the variable neutron_l2_population from my user_variables.yml, so it would take the default value of False, and re-run os-neutron-install, restart services14:42
busterswtpellaeon and you had existing vxlan networks in place?14:42
busterswtprior to rerunning?14:42
pellaeonbusterswt: yes14:42
busterswtok14:42
*** jwagner is now known as jwagner_away14:43
busterswtI suspect you're running into this: https://bugs.launchpad.net/openstack-ansible/liberty/+bug/156344814:43
openstackLaunchpad bug 1563448 in openstack-ansible liberty "Change to linuxbridge agent defaults affects upgrades" [Medium,Triaged]14:43
busterswtand you can confirm by checking the flooding entries in the FDB table to see if they've been reconfigured for multicast or are still set for unicast14:44
busterswtwhen you switch from l2pop to multicast, you'll want to delete the vxlan interfaces on all the hosts and restart the linuxbridge agent. It should recreate those interfaces with multicast and arp proxy OFF14:45
*** Brew has joined #openstack-ansible14:45
pellaeonbusterswt: looks identical to my case14:47
*** gparaskevas has quit IRC14:47
pellaeonpreviously evrardjp also suggested deleting vxlan interface first then run os-neutron-install, at first it worked, but after a while the multicast doesn't work again14:48
busterswthmm, interesting. Well now that all the bits are installed you should have to rerun playbooks to get this one thing working14:49
busterswt*should not14:49
evrardjpbusterswt rerunning playbooks should not be an issue14:50
busterswtevrardjp of course14:50
pellaeoni'm trying deleting vxlan interfaces then rerun and restart services again14:50
busterswtevrardjp i meant interesting that it worked and then stopped working :)14:50
busterswtpellaeon ok cool. lemme know14:50
evrardjpmy point precisely14:50
evrardjpbut I don't understand the issue pellaeon is hitting right now14:51
busterswtevrardjp i can't completely confirm the issue without seeing the fdb table but it sounds like the bug i posted14:52
evrardjppellaeon broadcast from a tenant vm will be tunneled in tenant vxlan and "broadcasted" to other tenant vms with the help of vxlan, using vtep id14:52
pellaeondone, same issue, checking the fdb table14:54
pellaeonby the way, I should see a multicast group in my switch, right?14:54
pellaeonright now I don't see any14:56
*** Brew has quit IRC14:56
busterswtehhhh, i dunno. Maybe only if you have snooping enabled - otherwise it will probably just broadcast. Not super familiar with it14:57
busterswtwhat brand switch are you using?14:57
*** Brew has joined #openstack-ansible14:57
pellaeonyes igmp snooping is enabled14:57
pellaeonJuniper EX330014:57
evrardjpohoh14:57
evrardjpit rings me a bell14:57
busterswtpellaeon do you mind sharing the fdb table and the output of 'ip -d link vxlan-X' where X is the ID14:57
evrardjpcheck with your vendor if everything is alright14:57
*** jwagner_away is now known as jwagner14:58
evrardjpI remember having heard an issue with that14:58
evrardjplet me ask14:58
*** larsks has left #openstack-ansible15:00
evrardjpI think neutron team will probably help you better as advised earlier15:00
pellaeonArrrrghhh, just found that linuxbridge agent on compute node was not started15:01
pellaeonuh, no15:02
busterswtpellaeon is it starting properly?15:02
pellaeonnow it is, and floating ip worked, which means VM unicast works15:04
busterswtpellaeon glad to hear it!15:04
busterswtmay want to spin up a new vm and make sure that works too15:04
pellaeonbut I just tried dhclient (VM network broadcast), it doesn't work15:04
busterswtDid you delete the vxlan interfaces inside the agent containers as well? And restart the agent there?15:05
pellaeonis ip -d link vxlan-X a valid command ? did you mean ip -d link show ?15:05
busterswtoh sorry, ip -d link show vxlan-x15:06
pellaeonbusterswt: yes i did15:06
*** sigmavirus24 is now known as sigmavirus24_awa15:06
*** sigmavirus24_awa is now known as sigmavirus2415:07
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible: Improved logging for memcached (OSA calling part)  https://review.openstack.org/30516515:08
pellaeonbusterswt: https://gist.github.com/pellaeon/fcd93b18288d48ddf2d102b02ee8f3ef15:09
*** Brew has quit IRC15:09
busterswtpellaeon what does fa:16:3e:6b:ed:bb belong to?15:13
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible: Docs: Cleanup page to configure to docs standards  https://review.openstack.org/30931615:13
busterswtpellaeon and do you know the MAC addrs of the DHCP server?15:13
pellaeonI updated the gist to contain dhcp server information15:18
pellaeonfa:16:3e:9f:d1:ba , fa:16:3e:c8:e2:82, fa:16:3e:9b:83:3915:18
pellaeonstill looking for fa:16:3e:6b:ed:bb15:18
busterswtNothing returned in the port-list with that MAC?15:19
*** admin0 has quit IRC15:20
pellaeonah, i forgot there was that command, fa:16:3e:6b:ed:bb is that subnet's neutron-l3-agent15:20
*** galstrom_zzz is now known as galstrom15:20
busterswtk15:20
busterswtIn those agents containers, can you check the bridge and see if the dhcp servers are connected?15:22
*** oneswig has quit IRC15:22
pellaeonchecked, their tap interfaces are all bridged with vxlan-6215:24
*** sacharya has joined #openstack-ansible15:24
busterswtok. You may want to run a packet capture inside those containers when you run dhclient or spin up a new VM to see if the broadcasts are making it.15:25
busterswtAlso - try pinging from one dhcp namespace to another. The ARP request will be a broadcast15:26
*** sacharya has quit IRC15:29
pellaeonhmmmm, tcpdump -i vxlan-62 on one of the agent containers did not capture dhcp request from that vm, but it did capture another VM's ARP request to l3 agent, and l3 agent's arp request15:30
pellaeon* l3 agent's arp response15:30
pellaeonping from one dhcp namespace to another works15:31
busterswtTry a pcap on the compute side to see where the dhcp request gets lost - if it does.15:32
*** Brew has joined #openstack-ansible15:33
pellaeonbusterswt: no, on the compute node I can see dhcp reqs from br-vxlan, to physical interface eth1.3015:35
pellaeoncheching the infra nodes15:35
*** mikelk has quit IRC15:39
pellaeonbusterswt: no, tcpdump on outmost physical interface eth1.30 on infra nodes doesn't capture dhcp reqs15:39
pellaeonso this probably means something wrong is with the switch ?...15:40
busterswtthey would be encapsulated - you're peeking inside the vxlan packet on eth1.30?15:41
*** iceyao has quit IRC15:42
*** jayc_ has quit IRC15:42
busterswtpellaeon i can't think of anything wrong there, especially since you've verified traffic from compute to infra over vxlan network. What is the MAC address of the instance? And can you update the FDB tables in your gist?15:43
pellaeonbusterswt: tcpdump will show the contents, I can see unicast packets encapsulated in vxlan, like this:15:43
*** galstrom is now known as galstrom_zzz15:43
pellaeon19:50:35.544396 IP 172.29.240.21.44695 > 172.29.240.31.8472: OTV, flags [I] (0x08), overlay 0, instance 6215:44
pellaeonIP 10.110.0.43.60001 > xxxxxxxxx.10806: UDP, length 7715:44
busterswtpellaeon so maybe the switch config is an issue, then. Found this - different model but may apply or may be worth a shot: http://movingpackets.net/2013/06/04/multicast-problems-juniper-ex/15:46
busterswtpellaeon i have to jet - best of luck to you. Sorry we couldn't get it resolved15:46
pellaeonbusterswt: thanks a lot!15:47
busterswtif you get it working, ping me on twitter @jimmdenton. Would love to know15:47
pellaeonThe mac addr of the vm sending dhcp reqs is only seen in compute node's fdb15:48
pellaeonbusterswt: sure, thanks again15:48
*** busterswt has quit IRC15:48
*** maestro has quit IRC15:53
pellaeonupdate: it was really the switch's problem , disabling igmp snooping made it work15:54
*** itlinux has joined #openstack-ansible15:55
*** thorst_ has quit IRC15:55
*** thorst_ has joined #openstack-ansible15:56
*** jayc_ has joined #openstack-ansible15:56
*** thorst__ has joined #openstack-ansible15:59
*** elopez has quit IRC15:59
*** thorst_ has quit IRC16:00
*** galstrom_zzz is now known as galstrom16:01
*** sacharya has joined #openstack-ansible16:01
*** thorst__ has quit IRC16:04
*** galstrom is now known as galstrom_zzz16:05
*** thorst_ has joined #openstack-ansible16:06
*** galstrom_zzz is now known as galstrom16:06
*** thorst__ has joined #openstack-ansible16:07
*** galstrom is now known as galstrom_zzz16:10
*** thorst_ has quit IRC16:10
*** sacharya has quit IRC16:11
*** thorst__ has quit IRC16:12
*** itlinux has quit IRC16:14
*** sdake has quit IRC16:16
*** sacharya has joined #openstack-ansible16:16
*** ccesario has quit IRC16:16
*** michaelgugino has joined #openstack-ansible16:18
*** thorst_ has joined #openstack-ansible16:19
*** jayc_ is now known as jayc16:22
*** michaelgugino has quit IRC16:22
*** thorst_ has quit IRC16:23
*** ccesario has joined #openstack-ansible16:28
*** karimb has quit IRC16:28
*** mageshgv has joined #openstack-ansible16:33
palendaelol...."If you don't use packages you're not running OpenStack"16:33
evrardjptalk?16:36
*** javeriak has joined #openstack-ansible16:36
evrardjpor link?16:37
palendaeevrardjp, Design summit work session16:37
evrardjpk16:37
*** thorst_ has joined #openstack-ansible16:38
*** sacharya has quit IRC16:41
*** thorst_ has quit IRC16:43
*** sacharya has joined #openstack-ansible16:46
*** elopez has joined #openstack-ansible16:46
*** thorst_ has joined #openstack-ansible16:49
*** thorst_ has quit IRC16:54
*** Oku_OS is now known as Oku_OS-away16:56
*** itlinux has joined #openstack-ansible16:57
*** markvoelker has joined #openstack-ansible16:59
*** thorst_ has joined #openstack-ansible17:01
*** markvoelker_ has joined #openstack-ansible17:02
*** markvoelker has quit IRC17:03
*** itlinux has quit IRC17:04
*** oneswig has joined #openstack-ansible17:05
*** thorst_ has quit IRC17:06
*** raddaoui has joined #openstack-ansible17:07
*** markvoelker has joined #openstack-ansible17:08
*** schwicht has quit IRC17:09
*** sacharya has quit IRC17:09
*** markvoelker_ has quit IRC17:09
*** itlinux has joined #openstack-ansible17:13
*** sacharya has joined #openstack-ansible17:14
*** thorst_ has joined #openstack-ansible17:15
*** jayc has quit IRC17:18
*** markvoelker_ has joined #openstack-ansible17:19
*** markvoelker has quit IRC17:19
*** itlinux has quit IRC17:19
*** KLevenstein has joined #openstack-ansible17:21
*** javeriak has quit IRC17:22
*** thorst_ has quit IRC17:23
*** winggundamth_ has joined #openstack-ansible17:23
*** thorst_ has joined #openstack-ansible17:23
*** KLevenstein has quit IRC17:24
*** jamielennox|away is now known as jamielennox17:24
*** markvoelker_ has quit IRC17:25
*** kiranv has joined #openstack-ansible17:26
*** sigmavirus24 is now known as sigmavirus24_awa17:27
evrardjpI'm off for today17:27
*** sigmavirus24_awa is now known as sigmavirus2417:28
*** jayc has joined #openstack-ansible17:29
*** sacharya has quit IRC17:32
*** schwicht has joined #openstack-ansible17:34
palendaeThese sessions need way more mics17:35
palendae1 isn't enough17:35
*** sacharya has joined #openstack-ansible17:36
*** thorst_ has quit IRC17:37
*** schwicht has quit IRC17:39
*** elopez has quit IRC17:47
*** winggundamth_ has quit IRC17:48
*** jamielennox is now known as jamielennox|away17:49
*** sacharya has quit IRC17:53
*** sacharya has joined #openstack-ansible17:55
*** oneswig has quit IRC17:56
*** itlinux has joined #openstack-ansible17:59
*** schwicht has joined #openstack-ansible18:01
*** oneswig has joined #openstack-ansible18:05
*** itlinux has quit IRC18:05
*** busterswt has joined #openstack-ansible18:13
*** jamielennox|away is now known as jamielennox18:14
*** sacharya has quit IRC18:21
*** itlinux has joined #openstack-ansible18:22
*** thorst_ has joined #openstack-ansible18:22
*** oneswig has quit IRC18:23
*** busterswt has quit IRC18:25
*** oneswig has joined #openstack-ansible18:25
*** sacharya has joined #openstack-ansible18:26
*** saneax_AFK is now known as saneax18:27
*** oneswig has quit IRC18:29
*** sacharya has quit IRC18:31
*** sacharya has joined #openstack-ansible18:31
*** electrofelix has quit IRC18:32
*** oneswig has joined #openstack-ansible18:32
*** oneswig has quit IRC18:33
*** oneswig has joined #openstack-ansible18:33
*** elopez has joined #openstack-ansible18:37
*** sacharya has quit IRC18:37
*** sacharya has joined #openstack-ansible18:40
*** weezS has joined #openstack-ansible18:41
*** jayc has quit IRC18:43
*** phiche has joined #openstack-ansible18:43
*** kiranv has quit IRC18:43
*** oneswig has quit IRC18:44
*** kiranv_ has joined #openstack-ansible18:46
*** phiche1 has joined #openstack-ansible18:48
*** phiche has quit IRC18:48
*** woodard has joined #openstack-ansible18:51
*** saneax is now known as saneax_AFK18:51
*** retreved has joined #openstack-ansible18:51
*** retreved has quit IRC18:52
*** woodard has quit IRC18:53
*** elopez has quit IRC18:56
*** jayc has joined #openstack-ansible18:57
*** sacharya has quit IRC19:06
*** KLevenstein has joined #openstack-ansible19:06
*** sacharya has joined #openstack-ansible19:07
*** aslaen has joined #openstack-ansible19:11
*** aslaen has quit IRC19:14
*** weezS has quit IRC19:23
*** schwicht has quit IRC19:25
*** schwicht has joined #openstack-ansible19:28
*** galstrom_zzz is now known as galstrom19:29
*** ccesario has quit IRC19:34
*** KLevenstein has quit IRC19:36
*** ccesario has joined #openstack-ansible19:36
*** KLevenstein has joined #openstack-ansible19:38
*** weezS has joined #openstack-ansible19:39
*** sacharya has quit IRC19:42
*** thorst_ has quit IRC19:42
*** KLevenstein has quit IRC19:43
*** sacharya has joined #openstack-ansible19:47
*** thorst_ has joined #openstack-ansible19:49
*** galstrom is now known as galstrom_zzz19:53
*** thorst_ has quit IRC19:54
*** KLevenstein has joined #openstack-ansible19:56
sigmavirus24mhayden: you around?19:57
prometheanfiresigmavirus24: I think he's doing the cantina thing20:00
sigmavirus24ah20:01
ApsuEntirely possible20:01
prometheanfireiirc it was starting at 3 for him20:01
*** elopez has joined #openstack-ansible20:02
mhaydensigmavirus24: here now20:05
sigmavirus24mhayden: I can't quite tell if any of these kernel tasks in osa-security require restarts. Ideas?20:06
mhaydenit would probably require rolling through each one by one :/20:06
sigmavirus24mhayden: right, wondering if you know of any of the tasks in that task file needing a restart specifically20:07
sigmavirus24None of them are jumping out to me20:09
*** egonzalez has joined #openstack-ansible20:11
*** johnmilton has quit IRC20:12
mhaydenthere might not be -- i tried to apply stuff immediately if at all possible20:15
openstackgerritMerged openstack/openstack-ansible-os_swift: Add fucntional tests for swift  https://review.openstack.org/30913120:22
sigmavirus24mhayden: figured as much. I did find a couple already but haven't found many20:22
*** KLevenstein has quit IRC20:26
*** sacharya has quit IRC20:26
*** KLevenstein has joined #openstack-ansible20:34
*** sacharya has joined #openstack-ansible20:35
*** thorst_ has joined #openstack-ansible20:36
*** jamielennox is now known as jamielennox|away20:37
*** egonzalez has quit IRC20:39
*** thorst_ has quit IRC20:40
*** sdake has joined #openstack-ansible20:41
*** thorst_ has joined #openstack-ansible20:42
*** thorst__ has joined #openstack-ansible20:44
*** egonzalez has joined #openstack-ansible20:46
*** tiagogomes has joined #openstack-ansible20:46
*** thorst_ has quit IRC20:47
*** jamielennox|away is now known as jamielennox20:47
*** Rodrigo_BR has joined #openstack-ansible20:47
*** thorst__ has quit IRC20:49
Rodrigo_BRHello ! someone can share with me the openstack_user_config.yml ?20:49
*** thorst_ has joined #openstack-ansible20:50
*** thorst_ has quit IRC21:04
*** dalees has quit IRC21:05
*** weezS has quit IRC21:08
*** egonzalez has quit IRC21:09
*** thorst_ has joined #openstack-ansible21:10
*** egonzalez has joined #openstack-ansible21:11
*** thorst__ has joined #openstack-ansible21:13
*** psilvad has quit IRC21:14
*** thorst_ has quit IRC21:15
*** thorst__ has quit IRC21:17
*** KLevenstein has quit IRC21:18
*** thorst_ has joined #openstack-ansible21:19
*** thorst_ has quit IRC21:23
*** dalees has joined #openstack-ansible21:23
*** Brew has quit IRC21:25
*** thorst_ has joined #openstack-ansible21:26
*** elopez has quit IRC21:26
*** phiche has joined #openstack-ansible21:28
*** can8dnSix has joined #openstack-ansible21:28
*** phiche1 has quit IRC21:29
*** sacharya has quit IRC21:29
*** thorst_ has quit IRC21:30
*** phiche has quit IRC21:30
*** agireud has quit IRC21:30
*** KLevenstein has joined #openstack-ansible21:30
*** KLevenstein has quit IRC21:32
*** agireud has joined #openstack-ansible21:32
andymccrodyssey4me cloudnull mattt - which role is setup properly for testing? e.g. a good template to copy when cleaning up the swift tests.21:33
*** KLevenstein has joined #openstack-ansible21:33
*** thorst_ has joined #openstack-ansible21:34
*** thorst_ has quit IRC21:34
*** sdake has quit IRC21:37
*** sacharya has joined #openstack-ansible21:37
meteorfoxis there an example of a conf.d/ceph.yml  documenting the available vars ?21:43
andymccrhey meteorfox - ceph isn't apart of the Openstack-ansible work - you can use the ceph repos https://github.com/ceph/ceph-ansible though. but for ease, RAX did an implementation to integrate with OSA21:45
andymccryou can check that out: https://github.com/rcbops/rpc-openstack/tree/master/rpcd/etc/openstack_deploy/conf.d21:45
andymccryou'll also need the env.d files from that repo to create the appropriate containers/target hosts21:46
meteorfoxandymccr: thanks, I already saw that one, but I don't know what 'affinity' is and what other variables are there21:46
andymccrahh21:47
andymccrlemme check21:47
andymccrah so we only have an aio file and not a sample file21:47
*** elo has joined #openstack-ansible21:47
meteorfoxyeah21:47
andymccrlemme check there are some docs for how to deploy ceph21:48
andymccrgonna try hunt those down for you.21:48
*** aludwar has quit IRC21:48
meteorfoxandymccr:  I'm looking at these http://docs.rackspace.com/rpc/api/v11/bk-rpc-installation/content/ch-ceph.html21:49
andymccrahh sweet21:49
andymccryou should just have to go through that then21:49
meteorfoxok so staging docs has more info21:50
*** aludwar has joined #openstack-ansible21:50
meteorfoxpublic one doesn't have it yet21:50
andymccrdrop me a msg if you run into issues i'll try help out on that21:51
meteorfoxthanks man, I appreciate your help21:52
*** mageshgv has quit IRC21:52
*** schwicht has quit IRC21:54
*** KLevenstein has quit IRC22:01
*** alextricity25 has quit IRC22:01
openstackgerritIan Cordasco proposed openstack/openstack-ansible-security: Notify deployer of hosts that need a restart  https://review.openstack.org/31006722:02
*** evrardjp has quit IRC22:02
*** evrardjp has joined #openstack-ansible22:03
*** charz has quit IRC22:03
*** deadnull has joined #openstack-ansible22:03
*** alextricity25 has joined #openstack-ansible22:04
*** sigmavirus24 is now known as sigmavirus24_awa22:04
*** deadnull has quit IRC22:05
*** deadnull has joined #openstack-ansible22:06
*** schwicht has joined #openstack-ansible22:06
*** charz has joined #openstack-ansible22:07
*** can8dnSix has quit IRC22:08
*** weezS has joined #openstack-ansible22:13
*** galstrom_zzz is now known as galstrom22:22
*** galstrom is now known as galstrom_zzz22:27
*** jamielennox is now known as jamielennox|away22:27
*** egonzalez has quit IRC22:29
*** jamielennox|away is now known as jamielennox22:31
*** sdake has joined #openstack-ansible22:34
*** sacharya has quit IRC22:34
*** jayc has quit IRC22:41
*** sacharya has joined #openstack-ansible22:43
*** schwicht has quit IRC22:45
*** elo has quit IRC22:47
*** michaelgugino has joined #openstack-ansible22:50
*** weezS has quit IRC22:53
*** jayc has joined #openstack-ansible22:54
*** tiagogomes has quit IRC22:57
*** deadnull has quit IRC23:00
*** asettle has joined #openstack-ansible23:05
*** michaelgugino has quit IRC23:05
*** jamielennox is now known as jamielennox|away23:07
*** saneax_AFK is now known as saneax23:13
*** sdake has quit IRC23:22
*** thorst has joined #openstack-ansible23:33
*** thorst has quit IRC23:38
*** sacharya has quit IRC23:38
*** thorst_ has joined #openstack-ansible23:42
*** sacharya has joined #openstack-ansible23:42
*** thorst_ has quit IRC23:46
*** raddaoui has quit IRC23:47
*** deadnull has joined #openstack-ansible23:52
*** thorst has joined #openstack-ansible23:55

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!