Monday, 2015-11-23

*** sacharya has joined #openstack-ansible00:33
*** alkari has quit IRC01:04
*** sacharya has quit IRC01:13
openstackgerritMajor Hayden proposed openstack/openstack-ansible: Use paramiko transport for gate check jobs  https://review.openstack.org/24836101:24
*** karimb has quit IRC01:36
*** cfarquhar_ has quit IRC01:43
*** cfarquhar has joined #openstack-ansible01:45
*** cfarquhar has quit IRC01:45
*** cfarquhar has joined #openstack-ansible01:45
*** galstrom_zzz is now known as galstrom02:04
*** galstrom is now known as galstrom_zzz02:05
*** baker has joined #openstack-ansible02:16
*** baker has quit IRC02:41
*** sdake has joined #openstack-ansible02:41
*** sdake_ has joined #openstack-ansible02:47
*** sdake has quit IRC02:48
*** sdake_ has quit IRC03:05
*** phschwartz has quit IRC03:16
*** phschwartz has joined #openstack-ansible03:16
*** baker has joined #openstack-ansible03:46
*** gouthamr has quit IRC04:11
*** agireud has quit IRC04:18
*** agireud has joined #openstack-ansible04:20
*** sacharya has joined #openstack-ansible04:24
*** agireud has quit IRC04:25
*** galstrom_zzz is now known as galstrom04:33
*** agireud has joined #openstack-ansible04:46
*** baker has quit IRC04:55
*** baker has joined #openstack-ansible05:02
*** alkari has joined #openstack-ansible05:10
*** galstrom is now known as galstrom_zzz05:13
*** baker has quit IRC05:27
*** galstrom_zzz is now known as galstrom05:33
openstackgerritMatthew Oliver proposed openstack/openstack-ansible: Add statsd configuration for swift  https://review.openstack.org/24618205:35
openstackgerritMatthew Oliver proposed openstack/openstack-ansible: Add container-reconciler and object-expirer to os-swift  https://review.openstack.org/24729005:35
openstackgerritMerged openstack/openstack-ansible: Update neutron service handler to restart only enabled services  https://review.openstack.org/24582505:39
*** galstrom is now known as galstrom_zzz05:46
*** eil397 has quit IRC05:51
*** mss has joined #openstack-ansible06:05
*** sacharya has quit IRC06:12
*** sacharya has joined #openstack-ansible06:13
*** jhesketh has quit IRC06:31
*** phiche has joined #openstack-ansible06:32
*** jhesketh has joined #openstack-ansible06:34
*** sacharya has quit IRC06:36
*** fawadkhaliq has joined #openstack-ansible06:41
openstackgerritXiaBing Yao proposed openstack/openstack-ansible: autodetect neutron service_en  https://review.openstack.org/24854506:46
*** eil397 has joined #openstack-ansible06:48
*** vCloudernBeer has joined #openstack-ansible06:51
vCloudernBeerTrying to setup an AIO environment06:52
vCloudernBeerLooking at the document, I have git cloned optnstack-ansible06:53
vCloudernBeerIn the document, the next step is to do a "git checkout kilo"06:53
vCloudernBeerThis it switch to a new branch? namely Kilo?06:54
*** vCloudernBeer has quit IRC06:59
*** shausy has joined #openstack-ansible07:01
*** phiche has quit IRC07:02
mcardenvCloudernBeer... ask a question then immediately disappear? Tough to get an answer to you.07:02
*** shausy has quit IRC07:02
*** shausy has joined #openstack-ansible07:08
*** openstack has joined #openstack-ansible07:21
*** phiche has joined #openstack-ansible07:23
*** mss has quit IRC07:26
*** javeriak has joined #openstack-ansible07:31
*** javeriak has quit IRC07:35
*** javeriak has joined #openstack-ansible07:35
*** sacharya has joined #openstack-ansible07:37
*** sdake has joined #openstack-ansible07:38
*** javeriak_ has joined #openstack-ansible07:40
*** javeriak has quit IRC07:41
*** sacharya has quit IRC07:41
*** dolpher1 has joined #openstack-ansible07:48
*** karimb has joined #openstack-ansible07:49
*** dolpher1 has quit IRC07:51
*** phiche1 has joined #openstack-ansible07:58
openstackgerritJaveria Khan proposed openstack/openstack-ansible: Update neutron service handler to restart only enabled services  https://review.openstack.org/24855707:59
*** phiche has quit IRC08:00
openstackgerritJaveria Khan proposed openstack/openstack-ansible: Update neutron service handler to restart only enabled services  https://review.openstack.org/24855808:01
evrardjphello everyone08:04
evrardjpjust to jump on the conversation of SSH connection issues, we moved to paramiko for a while08:04
evrardjpfor openstack-ansible08:04
evrardjpand it's far more reliable08:04
evrardjpit's still not 100% reliable, and is really a little slower08:05
evrardjpbut we felt it was better08:05
evrardjpwe didn't move back to another connection plugin08:06
*** fawadkhaliq has quit IRC08:11
*** fawadkhaliq has joined #openstack-ansible08:12
*** mss has joined #openstack-ansible08:22
iceyaoevrardjp: good job.08:25
evrardjp?08:26
openstackgerritXiaBing Yao proposed openstack/openstack-ansible: Update some steps in quickstart-aio.rst  https://review.openstack.org/24856708:26
evrardjpit's not changed upstream, so if one want to change it's connection plugin, he needs to edit his ansible.cfg08:27
evrardjpI'd like to summarize what has been changed for the repo-* stuff, could you tell me if I'm wrong?08:35
evrardjpso for master/liberty/kilo, the work to simplify repo-build has been (or will be) merged08:36
evrardjpthis creates (for people who want it) your own repository with the software you want08:36
*** mpavone has joined #openstack-ansible08:36
evrardjpthe repo-clone thingy, is not a role, only a sync of the upstream repos, and its rework has been introduced in master/liberty08:37
evrardjpand will be reintroduced in kilo too08:38
evrardjpit's purpose is only to mirror an already built mirror (with the previous repo-build playbook)08:38
evrardjpam I right?08:38
*** dolpher1 has joined #openstack-ansible08:44
*** xek has joined #openstack-ansible08:46
evrardjpI'm planning to do a cleaning of the docs08:48
*** gparaskevas has joined #openstack-ansible08:49
evrardjpis there someone from the doc team here, I'd like to know if I can edit some png08:50
evrardjpso I'd be happy to have the initial svgs08:50
*** dolpher1 is now known as dolpher08:51
*** tiagogomes has joined #openstack-ansible08:54
*** fawadkhaliq has quit IRC09:00
*** fawadkhaliq has joined #openstack-ansible09:01
odyssey4memorning all09:03
odyssey4mewelcome iceyao :) good to see you join us in IRC!09:03
evrardjpwelcome :)09:07
odyssey4mejaveriak_ if you have a chance, I'd love your input on https://review.openstack.org/24827109:09
javeriak_odyssey4me sure leme see09:09
odyssey4mejaveriak_ also https://review.openstack.org/248545 which iceyao added09:10
javeriak_wow we're getting lots of new additions lately, this is nice :)09:11
evrardjp248271 is nice :)09:14
*** fawadkhaliq has quit IRC09:21
*** fawadkhaliq has joined #openstack-ansible09:22
*** mgoddard has joined #openstack-ansible09:23
evrardjp248545 is also a nice addition09:24
odyssey4meevrardjp if you can put feedback into the reviews, that'd be best :)09:28
*** mpavone has quit IRC09:29
evrardjpoh I have started :p09:33
evrardjpit was a subtle way of saaying, maybe you should check on these :p09:33
evrardjp-a09:33
evrardjpnow that I have fixed my networking issue for my testing, I'll have a more flexible testing system, so I'm gonna try this09:35
javeriak_odyssey4me reviewed both, evrardjp had a comment on your comment :). btw don't we have a preference of bug/bp ref for all commits?09:36
evrardjpwhat do you mean javeriak_?09:37
evrardjplarge changes often requires blueprints09:37
javeriak_evrardjp, about the comment or the bp/bug question?09:37
evrardjpyup09:38
javeriak_oh yes that, i just thought openstack communities had a rule/preference of having every commit tied to one of these09:38
odyssey4meah javeriak_ well, I'm taking a more relaxed position on the submission of reviews - blueprints/specs are for large feature changes, bugs can be registered for bugs found... but if someone finds a bug or implements a small improvement then a bug is not a hard requirement09:39
evrardjpbugs should be linked to launchpad09:39
odyssey4meI don't like people registering bugs for the sake of submitting a patch - that creates a silliness in the bug management process and results in orphaned bugs09:40
javeriak_okay cool, i wont have to file new things for stuff i mean to push shortly09:40
odyssey4meit's more important for me to have the patches move along than to stick to a bureaucratic process09:41
javeriak_yea that makes perfect sense09:41
*** mpavone has joined #openstack-ansible09:41
odyssey4mejaveriak_ sure - the general position is that if the core team feels that the patch warrants a bug or spec, then they can ask for it09:41
javeriak_but i remember a point in this discussion being that it would be easier to track backports for stuff other people push in one branch and then forget09:42
odyssey4mejaveriak_ yup, but that doesn't work when people register bugs for the purpose of implementing a patch because the bug doesn't get properly triaged09:43
evrardjpodyssey4me: I thought that registering a bug helps tracking the issues accross branches09:43
odyssey4meso I'm rather facilitating proactive and almost immediate backports once patches have merged09:43
*** alkari has quit IRC09:44
evrardjpbut this makes sense, more action than paperwork. I like it09:44
odyssey4meevrardjp that was an original intent, but it has not been my experience so I've taken a different position... if someone wants to take up a release management role for the project and define a process we should all stick to, that's fine by me... but right now it's me, so I'm working the way that makes sense for me :)09:44
evrardjpyou're right09:45
*** mpavone has quit IRC09:45
javeriak_so i like what https://review.openstack.org/#/c/248545/1 is doing, and i mean to add something similar that adds a condition on what core plugin is selected, ie if plumgrid then you set all these agents to false, ill push something and then you guys can tell me what you think09:46
javeriak_it'll make things easier to setup on our side09:47
evrardjpthis makes sense09:50
evrardjpwouldn't it be better to completely override the service_en: for each agent you want to disable?09:51
evrardjpmy point would be: if each agent dict has a specific configuration that works for only one plugin, then it shouldn't be globally set.09:53
evrardjpor we should maybe introduce a way to have the service_en automatically set based on a fact09:54
javeriak_evrardjp yes thats sorta what i have in mind, havent started coding yet so lets see what the optimal solution looks like, i would however like to leave these service_en's open to other conditions as well instead of assiging them all a global value dependednt on one plugin09:55
evrardjpbut yeah, it's not that important we can keep the auxiliary variable to have an easier override09:55
javeriak_yea something like that09:55
odyssey4mejaveriak_ evrardjp agreed - I'd like to steer clear of complex jinja logic in the variables if possible too - the simpler the logic is to follow, the better09:56
*** sacharya has joined #openstack-ansible09:56
odyssey4mebut yes, if there are other variables we can base decisions on to automate decisions, then that's good - like the enabling of lbaas09:56
evrardjpjaveriak_: your input will be important, because I have no clue how we deploy this differently...09:57
odyssey4meit makes it much, much simpler for the deployer when less things need to be changed to implement something properly09:57
javeriak_evrardjp ofcourse, thats how we're doing it in our labs all the time :)09:57
evrardjpodyssey4me: +109:57
*** mpavone has joined #openstack-ansible09:58
evrardjpfor me, moving to plumgrid should be as simple as flipping a switch09:58
evrardjp:p09:58
javeriak_btw i dont think ive expressed my appreciation at how there is someone almost all the time on this commnity to talk to and answer questions. It can get so frustrating with a few others :), so great job guys!09:59
*** sacharya has quit IRC10:01
odyssey4mejaveriak_ your help evolving the neutron role into something more pluggable has been invaluable - thank you!10:06
evrardjpindeed10:06
evrardjpjaveriak_: so what you're doing at plumgrid is editing these neutron_l3, neutron_metering... variables to edit the conditions, or to set them as false, right?10:07
javeriak_glad to be a part of this guys :),10:07
evrardjpI'm wondering if the |combine filter to merge dicts would be a good way to do it10:08
javeriak_evrardjp yes we just set all of them to false, as plumgrid provides the whole networking layer, see here http://docs.openstack.org/developer/openstack-ansible/install-guide/app-plumgrid.html#neutron-configurations10:08
evrardjpok10:11
evrardjpI'll have to check more deeply in the neutron role10:11
javeriak_evrardjp hmm havnt really played with the combine filter yet, so cant say10:11
evrardjpit's not in ansible 1 :)10:12
evrardjpexcept if you bring it yourself, which is a few lines of python10:12
evrardjpbut ansible2 is close10:12
javeriak_yea i am very anxiously waiting for thst release, theres a lot of stuff we can clean up in our roles once that arrives, and hopefully with the supporting features that we're expecting10:14
evrardjpdid you try the rc1?10:26
evrardjpit's out for 4 days10:26
javeriak_nope not yet, have you?10:26
evrardjpfor small fry stuff yes10:26
evrardjpI didn't try with all my plugins, etc.10:27
javeriak_any idea on the main release eta?10:27
evrardjpthis month I guess10:27
javeriak_i havnt really been following10:27
javeriak_ok great10:27
evrardjpI thought they were planning on delivering it during ansible fest10:27
evrardjpbut they delivered an RC10:27
evrardjpso I guess it won't take much more time anymore10:28
evrardjpI'll give more news when I'll now :p10:28
evrardjpknow*10:30
javeriak_great thanks evrardjp10:30
javeriak_gtg meetings, catch you guys later10:30
openstackgerritJesse Pretorius proposed openstack/openstack-ansible: Fixed haproxy backend config issue  https://review.openstack.org/24588310:34
*** javeriak_ has quit IRC10:34
odyssey4memattt hughsaunders we need another vote for https://review.openstack.org/246392 to move it along and get the focus for new patches into the right repositories, reviews would be appreciated10:42
odyssey4metiagogomes ping?10:48
tiagogomesodyssey4me hello10:49
odyssey4metiagogomes :) did you see cloudnull's message in https://review.openstack.org/238509 ?10:49
odyssey4meit'd be great if your patch could be updated based on the referred review10:49
* tiagogomes reads10:52
odyssey4meevrardjp what differences are you seeing? are they due to recently landed patches? if so, we need to add reviews ASAP to get those changes into master in the new repositories10:54
evrardjpI'll redo a diff, but it was cosmetic changes IIRC10:55
evrardjpmy point was that we should have a clean diff10:56
evrardjpand improve afterwards10:56
evrardjpbut let me completely check and I'll give you my +1 if it's ok or not10:56
evrardjpit's in my pipeline10:56
hughsaundersodyssey4me: I'll build it and check10:58
tiagogomesodyssey4me so you want me to rebase my patchsets, is that right?10:59
odyssey4metiagogomes as far as I saw, your patch wasn't quite functional - it never passed the gate check. cloudnull's patch was... but you did implement some changes where he didn't... so if you could compare the changes you made to those he made, implement any differences and get it fully functional, that'd be great11:15
*** javeriak has joined #openstack-ansible11:17
evrardjpodyssey4me: comment added on https://review.openstack.org/#/c/246392/. TL;DR: No concern to have11:18
odyssey4meevrardjp great, thanks - the .DS_Store thing should be added to .gitignore - can you prep reviews to add that to each repo?11:19
*** javeriak has quit IRC11:19
odyssey4meand whichever repo has it should have it removed also :)11:19
evrardjpyou told me to not add review on the IRR right now, am I allowed now?11:20
evrardjp:p11:20
odyssey4meevrardjp no, reviews for the other repositories are open - there is only a freeze for those roles in the main repo11:20
odyssey4meie we don't want changes in the playbook repo for any roles which are split into seperate repo's, changes should only go into the new role repositories11:21
evrardjpbefore doing these changes on these new roles, we should define how to tag them to have a long term support (branching them + tagging them), but that's a topic for Thursday :D11:23
evrardjp(just to keep it in mind)11:23
evrardjpmay I also add a .gitreview file?11:26
odyssey4meevrardjp do the changes to the roles - for now the way we do things won't change much - when we release mitaka the role repositories will get branches, but what will be different is that each role can tag independently11:26
odyssey4meevrardjp yeah, good idea11:26
evrardjpI'll put them in their appropriate project11:27
*** fawadkhaliq has quit IRC11:41
*** fawadkhaliq has joined #openstack-ansible11:41
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-apt_package_pinning: New git dotfiles for independant repository  https://review.openstack.org/24863211:50
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-lxc_container_create: New git dotfiles for independant repository  https://review.openstack.org/24863411:50
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-lxc_hosts: New git dotfiles for independant repository  https://review.openstack.org/24863511:51
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-openstack_hosts: New git dotfiles for independant repository  https://review.openstack.org/24863611:51
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-pip_install: New git dotfiles for independant repository  https://review.openstack.org/24863711:51
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-py_from_git: New git dotfiles for independant repository  https://review.openstack.org/24863811:51
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-rsyslog_client: New git dotfiles for independant repository  https://review.openstack.org/24863911:51
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-lxc_hosts: Removed undesired .DS_Store file  https://review.openstack.org/24864311:54
*** sacharya has joined #openstack-ansible12:05
*** jaypipes has joined #openstack-ansible12:08
*** sacharya has quit IRC12:10
openstackgerritMerged openstack/openstack-ansible: Add "httpchk /versions" for glance-api haproxy.  https://review.openstack.org/24821312:10
iceyaohello everyone12:10
tiagogomesodyssey4me I can't see why one passed and other failed in the gate. Pherpahs it would be better to abandon mine and use the other one instead12:16
odyssey4meo/ iceyao12:16
odyssey4metiagogomes I'd prefer you to have a commit. cloudnull has enough of his own. :)12:16
openstackgerritMerged openstack/openstack-ansible: Added logging for haproxy to rsyslog  https://review.openstack.org/24504612:21
evrardjpI dont' know anything about tox  or its integration with jenkins (yet), what's best approach to learn? read tox docs first? or have someone to explain me the current status of integration of the IRR?12:27
evrardjpok got it12:29
tiagogomescloudnull, re https://review.openstack.org/#/c/238509/, I suggest using your patch instead :)12:36
evrardjpcloudnull: when you'll be back, I'd like to have a look at tox tests of the IRR with you, especially the one that effectively starts the playbook.12:38
*** cristicalin has joined #openstack-ansible12:48
iceyaotiagogomes: may be domain_id = 'default' cause jenkins failed , you should use 'Default'.12:51
iceyaoI guess12:52
*** woodard has joined #openstack-ansible12:53
tiagogomesiceyao maybe, but I'd had thought that if I used 'Default', the openstack client would try to translate the name to the ID12:54
odyssey4metiagogomes The openstack client has that functionality, but the keystone library does not. With the keystone library you have to be clear about whether you're passing a domain name or a domain ID.12:58
tiagogomesodyssey4me good point12:58
mgariepygood morning everyone !13:03
evrardjpgood morning mgariepy13:03
mgariepyhow are you guys doing ?13:05
*** baker has joined #openstack-ansible13:08
odyssey4memgariepy can you please backport https://review.openstack.org/248213 to the liberty & kilo branches? if you need some help getting that done, let me know13:09
*** shausy has quit IRC13:10
mgariepyodyssey4me, i guess I need to click the cherry-pick to button ?13:11
openstackgerritMarc Gariépy proposed openstack/openstack-ansible: Add "httpchk /versions" for glance-api haproxy.  https://review.openstack.org/24868813:11
odyssey4memgariepy yep - do the cherry-pick button from the master patch for each branch13:12
openstackgerritMarc Gariépy proposed openstack/openstack-ansible: Add "httpchk /versions" for glance-api haproxy.  https://review.openstack.org/24868913:12
mgariepysimple enough hehe :)13:12
odyssey4meany volunteers to back port https://review.openstack.org/248318 ?13:13
odyssey4methanks mgariepy :)13:13
odyssey4meevrardjp odd, if I do the tox test on a fresh cloud server it works fine13:14
odyssey4mealthough I did have to figure out how to do it13:14
evrardjpI have a few questions on it :)13:15
evrardjpfirst permissions, and second the envlist13:15
evrardjpor the other way around13:15
evrardjplike I said, I don't know much about it, and I'm learning13:16
odyssey4meevrardjp so to run a tox test I needed to clone the repo, change into the root of the dir, execute https://github.com/openstack/openstack-ansible/blob/master/scripts/bootstrap-aio.sh#L102-L115 (although I expect that only some of those packages are needed), install pip, then run 'tox -e ansible-functional'13:17
evrardjpyeah, that's what I guess13:18
evrardjpbut is it normal that envlist doesn't mention ansible-functional ?13:18
evrardjpwhile in the same file, there is an ansible-functional environment13:19
evrardjpI'll try this on a server13:19
*** baker has quit IRC13:20
odyssey4meevrardjp no, that's a mistake :)13:24
openstackgerritMarc Gariépy proposed openstack/openstack-ansible: Remove duplicated Aodh host var  https://review.openstack.org/24869613:25
evrardjpI think this is the cause, the env is not created so the variables are incorrectly set... etc13:26
evrardjpit's maybe intentionnal to not create many environments, but then, we need to do some wiring I guess13:26
evrardjpanyway I'll test on my side first13:27
odyssey4memgariepy thanks for the liberty backport on that one - the patch doesn't apply for kilo, so no more needed13:28
mgariepyyeah i saw that it doesn't apply to kilo ;)13:28
*** gouthamr has joined #openstack-ansible13:28
*** jaypipes has quit IRC13:29
*** gparaskevas_ has joined #openstack-ansible13:30
*** gparaskevas has quit IRC13:32
*** jaypipes has joined #openstack-ansible13:32
*** dalees has quit IRC13:37
openstackgerritXiaBing Yao proposed openstack/openstack-ansible: Update some steps in quickstart-aio.rst  https://review.openstack.org/24856713:43
*** dalees has joined #openstack-ansible13:49
*** antonym has joined #openstack-ansible13:51
openstackgerritJesse Pretorius proposed openstack/openstack-ansible-apt_package_pinning: Update run_tests to be more complete  https://review.openstack.org/24870713:51
*** elgertam1 has quit IRC13:52
*** fawadkhaliq has quit IRC13:55
*** fawadkhaliq has joined #openstack-ansible13:55
odyssey4meevrardjp I suspect that the actual issue is that the role presumes root access, but openstack-ci is not running the role with sudo access13:56
evrardjpthat was my second point :p14:00
evrardjpwe could use remote_user: root I think14:01
evrardjpoh you mean openstack-cli doesn't even have sudo?14:01
evrardjpci*14:01
*** greg_a has joined #openstack-ansible14:11
odyssey4meevrardjp very possibly, looking into that14:13
odyssey4memay need to have a chat to pabelanger_ about whether we should rather not assume root, and how we should handle that14:13
odyssey4meFYI 12.0.1 has been tagged14:14
odyssey4mehttps://launchpad.net/openstack-ansible/liberty/12.0.114:14
*** mpavone has quit IRC14:26
*** HimanshuGarg has joined #openstack-ansible14:27
*** mpavone has joined #openstack-ansible14:30
*** mgoddard_ has joined #openstack-ansible14:30
*** mgoddard has quit IRC14:33
*** jasondotstar_afk is now known as jasondotstar14:33
*** gouthamr has left #openstack-ansible14:36
evrardjpnice!14:36
evrardjpcongrats everyone14:36
evrardjpwhen I'll be able to move to work (cf. what's in the news), I'll be happy to deploy that in my pre-prod lab!14:37
evrardjpbtw I had a weird issue with liberty AIO, galera was behaving weirdly: by default only one mysqld (1/3) was running, and service mysql start was failing. starting the mysql daemons with only 'mysql' was working perfectly fine 'though14:40
*** alextricity_h has joined #openstack-ansible14:41
odyssey4meevrardjp liberty has mariadb10, so things are a little different14:43
odyssey4mebut that shouldn't happen, although I have seen that behaviour with both sometimes14:43
*** spotz_zzz is now known as spotz14:44
evrardjpodyssey4me: I know about mariadb change, I got the chance to follow that part14:44
*** javeriak has joined #openstack-ansible14:45
*** javeriak_ has joined #openstack-ansible14:46
*** javeriak has quit IRC14:46
*** alextricity_h has quit IRC14:49
*** javeriak has joined #openstack-ansible14:50
openstackgerritJesse Pretorius proposed openstack/openstack-ansible: Disable ssh pipelining for gate check jobs  https://review.openstack.org/24873914:50
*** javeriak_ has quit IRC14:52
*** KLevenstein has joined #openstack-ansible14:57
openstackgerritJesse Pretorius proposed openstack/openstack-ansible: Disable ssh pipelining for gate check jobs  https://review.openstack.org/24873914:57
*** subscope has joined #openstack-ansible14:58
*** greg_a has quit IRC15:00
evrardjpodyssey4me: I think mhayden is trying to fix that by moving to paramiko15:01
*** openstackgerrit has quit IRC15:02
odyssey4meevrardjp yep, I'm doing this as a comparison15:02
evrardjpok15:02
odyssey4meparamiko apparently is the most reliable, but also the slowest15:02
*** openstackgerrit has joined #openstack-ansible15:02
odyssey4meso I'd just like to see the comparison15:02
evrardjpit's not that slow anymore15:02
evrardjphttp://www.ansible.com/blog/2014/01/15/ssh-connection-upgrades-coming-in-ansible-1-515:04
pabelanger_morning15:04
*** pabelanger_ is now known as pabelanger15:05
evrardjpgood morning pabelanger15:05
pabelangerodyssey4me: just reading backsroll15:05
pabelangerso, some questions about gate I see :)15:06
evrardjpapparently15:06
odyssey4meo/ pabelanger short and sweet - the role functional tests are hitting http://logs.openstack.org/07/248707/1/check/gate-openstack-ansible-apt_package_pinning-tox-ansible-functional/087b326/console.html#_2015-11-23_13_53_40_53215:06
evrardjpwe didn't define to run the playbooks as root, atm15:07
evrardjpbut can we?15:07
evrardjpor should we use sudo?15:07
odyssey4memy suspicion is that the role tests are running without sudo access, or we need to specify the use of sudo?15:07
pabelangerodyssey4me: Ah. Ya, I know the issue15:07
evrardjpodyssey4me: atm they are running as user15:07
pabelangerthe tox gate drops sudo support15:07
evrardjppabelanger: so, what's the solution?15:08
evrardjpforce using sudo or, force using root user?15:08
pabelangerso we should port you to15:08
odyssey4mepabelanger hmm, so do we add sudo support back - or do we change the functional tests to run without sudo?15:08
pabelangeruse: https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/ansible-role-jobs.yaml#L4115:08
pabelangerand make voting jobs15:08
pabelangerI created gate-ansible-role-nodepool-dsvm-functional-centos7 gate15:09
pabelangerand trusty game15:09
pabelangergate15:09
pabelangerso I could support 2 OSs15:09
odyssey4mepabelanger we do intend to include other OSs later on (and are looking for contributors) :)15:10
pabelangerevrardjp: odyssey4me: we'd update your projects to use: https://github.com/openstack-infra/project-config/blob/master/zuul/layout.yaml#L360515:10
odyssey4mebut for now we obviously don't have that15:10
pabelangerso, what OS are you supporting ATM?15:10
odyssey4mepabelanger just ubuntu trusty15:10
pabelangerokay15:11
pabelangergreat15:11
evrardjpit's just a rename of jobs then15:11
evrardjpor am I wrong?15:11
pabelangerso, we can setup gate-{name}-dsvm-functional-trusty for you15:11
pabelangerya, need to add proper voting job, but I can do that if you like15:11
evrardjplike I said I'm like John Snow on this zuul thingy.15:11
*** HimanshuGarg has quit IRC15:12
evrardjpmaybe we should also wait for cloudnull to be here15:13
pabelangerWFM15:13
odyssey4mepabelanger so simply a change like 'gate-openstack-ansible-apt_package_pinning-tox-ansible-functional' -> 'gate-openstack-ansible-apt_package_pinning-dsvm-functional-trusty' in zuul/layout.yml ?15:13
*** HimanshuGarg has joined #openstack-ansible15:14
pabelangerodyssey4me: Ya, all your modules would need to change but that should work15:14
evrardjpthat's what I thought15:14
odyssey4meok cool - let me work up a patch quickly then15:14
pabelangerhttp://logs.openstack.org/98/246998/7/check/gate-ansible-dsvm-functional-trusty-nv/bc521c0/console.html15:14
pabelangerfor an example15:14
pabelangermind you I am using connection_local, but it does access sudo15:15
evrardjpwe are using connection_local too15:15
pabelangererr15:15
pabelangeransible_connection=local15:15
evrardjpwe didn't set the use_sudo: True in the playbooks15:15
pabelangerbut I want to change that so SSH shortly15:15
pabelangerI have sudo: True for mine15:16
odyssey4mepabelanger we're hitting all sorts of pain using ssh at the moment15:16
pabelangershould revisit that15:16
evrardjpodyssey4me: do you mind sharing the commit to make everything work? this way I understand more the relationship between tox runs and this gate jobs15:16
evrardjpthese*15:16
*** galstrom_zzz is now known as galstrom15:17
odyssey4meevrardjp sure, will do - although this isn't really tox15:17
evrardjpno, this is gate jobs, right?15:17
odyssey4metox creates a python venv to run all the tests so that the host isn't sullied with all the python bits15:17
odyssey4metox can be used locally - so, for instance, you can run it on a mac15:17
evrardjpyeah I understood that, but these tox runs are started from somewhere15:17
evrardjpthat's the glue missing for me15:17
*** gouthamr has joined #openstack-ansible15:18
evrardjpI started to use it today, it doesn't seem difficult to understand15:18
cloudnullmorning15:18
evrardjpCI of openstack and on what all of these jobs are running is another story... that I'd like to understand too15:19
evrardjpmorning cloudnull15:19
*** grumpycatt has quit IRC15:19
odyssey4mepabelanger evrardjp https://review.openstack.org/248757 <- is that it?15:22
pabelangerodyssey4me: It is right, but will actually fail.  I need to patch something first15:22
pabelangerright now they are only -nv jobs15:22
odyssey4mepabelanger ok, thanks - let me know when you have that in and I'll add the review as a dependency15:23
pabelangerodyssey4me: this is needed too: http://paste.openstack.org/show/479737/15:24
pabelangerodyssey4me: if you want to just append it15:24
*** mss has quit IRC15:24
odyssey4mepabelanger sure, I can do that15:25
evrardjpI'm editing the become thingy15:25
pabelangerodyssey4me: what sort of SSH issues are you having?15:26
pabelangercurious to hear the problems15:26
*** alkari has joined #openstack-ansible15:26
odyssey4mereview updated pabelanger - please check15:27
odyssey4methanks evrardjp15:27
pabelangerodyssey4me: that should be right. Waiting for gate to confirm15:28
odyssey4mepabelanger as an example, look at the check results for all the jobs in https://review.openstack.org/24588315:28
odyssey4mehmm, let me find a more specific one15:29
*** mgoddard has joined #openstack-ansible15:29
*** mgoddard_ has quit IRC15:29
openstackgerritBjoern Teipel proposed openstack/openstack-ansible: Pass domain to some calls in the keystone library  https://review.openstack.org/23850915:29
pabelangerAlso, forgive me if I get this wrong. Is the majority of the people in openstack-ansible at rackspace?15:30
*** eil397 has quit IRC15:31
*** javeriak has quit IRC15:32
odyssey4mepabelanger for the moment, yes15:32
evrardjpI'm not15:32
odyssey4methere are several contributors and even more deployers who are not, but we're still working hard at getting more development contributors15:34
pabelanger /me nods15:34
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-apt_package_pinning: [WIP] Tentative to use sudo for role testing  https://review.openstack.org/24877415:35
evrardjppabelanger: does this looks right to you ^15:36
*** gouthamr has left #openstack-ansible15:36
evrardjpI wrote using the become method instead of sudo: http://docs.ansible.com/ansible/become.html15:36
openstackgerritBjoern Teipel proposed openstack/openstack-ansible: Pass domain to some calls in the keystone library  https://review.openstack.org/23850915:36
pabelangerevrardjp: not sure, never used that code before15:36
pabelangershould read up on it and test15:37
evrardjpwe should first wait on this: https://review.openstack.org/#/c/248757/ to pass15:37
pabelangerStill using the old way :)15:37
*** elgertam1 has joined #openstack-ansible15:38
pabelangerevrardjp: ya, I'll try for an -infra push15:38
evrardjpme too, but it's time to evolve :D15:38
*** elgertam1 has quit IRC15:38
evrardjpthanks pabelanger :)15:38
*** galstrom is now known as galstrom_zzz15:39
evrardjpcloudnull: what do you think of editing the tox file and adding ansible-functional to the envlist?15:39
evrardjpI could do that for you if you prefer15:40
*** galstrom_zzz is now known as galstrom15:42
cloudnullevrardjp: that sounds sensible15:44
cloudnullevrardjp:  i added the ansible-functional bits to the break our roles i've done thus far https://github.com/openstack/openstack-ansible-lxc_hosts/blob/master/tox.ini15:46
evrardjpyeah I've seen that15:48
evrardjpI'll help you on these ones15:48
evrardjpmoreover, when I'll understand the complete process, I'll move my roles to openstack-ansible stadion15:48
evrardjpbig tent15:48
evrardjpI don't know the name15:49
evrardjpopenstack-ansible herd?15:49
cloudnullansible gaggle15:50
cloudnull:)15:50
evrardjpI just google translated it. Nice term15:51
evrardjp"a disorderly or noisy group of people."15:51
cloudnullyea15:52
*** alkari has quit IRC15:52
evrardjpdid some of you mentionned that you were doing openstack horizon branding internally for RPC?15:56
evrardjpcause I have an embryonic role that could be improved, and (maybe better) be moved upstream15:56
evrardjpfor the branding15:57
evrardjpif I check at your role, maybe I can find a decent base that will be helpful for everyone15:57
evrardjpthe only thing I see is that: https://github.com/rcbops/rpc-openstack/blob/master/rpcd/playbooks/roles/horizon_extensions/tasks/main.yml#L1615:59
cloudnullevrardjp:  rpc is doing branding as well as a new tab , that role https://github.com/rcbops/rpc-openstack/tree/master/rpcd/playbooks/roles/horizon_extensions is essentially all we're doing on that front16:00
evrardjpok16:01
odyssey4meevrardjp cloudnull a clowder?16:01
*** jmvalente has joined #openstack-ansible16:02
*** galstrom is now known as galstrom_zzz16:04
cloudnullIm +1 for herd or gaggle :)16:04
*** fawadkhaliq has quit IRC16:04
*** javeriak has joined #openstack-ansible16:05
*** elo has joined #openstack-ansible16:07
*** karimb_ has joined #openstack-ansible16:10
evrardjpansible logo is bull right? so it's maybe better to have gaggle, just for the sake of being different :p16:11
evrardjpor not, whatever16:11
evrardjpI'll check on clowder, it looks like clown and sound like cloud16:12
*** karimb has quit IRC16:13
evrardjpmaybe we should put a vote in the last part of Thursday meeting if there is one minute remaining16:13
evrardjp:p16:13
*** Pablo|off| has joined #openstack-ansible16:14
*** karimb_ has quit IRC16:14
*** skamithi has joined #openstack-ansible16:15
*** Pablo|off| is now known as pcaruana16:15
odyssey4meevrardjp as I recall we last discussed that it should be simple enough to add to the horizon role... the changes are so small that it doesn't feel like it needs another role16:16
evrardjpit's really two tasks16:16
odyssey4meyup16:17
evrardjpI have another role, but I find it really overkill16:17
evrardjpI'll update horizon it's really nothing16:17
*** woodard has quit IRC16:19
*** galstrom_zzz is now known as galstrom16:20
*** cristicalin has quit IRC16:24
*** daneyon_ has joined #openstack-ansible16:24
*** mpavone has quit IRC16:24
*** daneyon has quit IRC16:26
*** alkari has joined #openstack-ansible16:30
*** mss has joined #openstack-ansible16:31
*** javeriak_ has joined #openstack-ansible16:35
*** javeriak has quit IRC16:37
*** javeriak has joined #openstack-ansible16:37
*** greg_a has joined #openstack-ansible16:38
*** greg_a has left #openstack-ansible16:39
*** javeriak_ has quit IRC16:40
*** fawadkhaliq has joined #openstack-ansible16:41
*** gparaskevas_ has quit IRC16:43
odyssey4meevrardjp :)16:44
*** baker has joined #openstack-ansible16:45
*** subscope has quit IRC16:50
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible: Branding of OpenStack  https://review.openstack.org/24881916:54
logan-i have a few things regarding horizon customization too, not sure if it would be useful to anyone: https://github.com/Logan2211/openstack-ansible/commits/horizon-customization16:55
*** sacharya has joined #openstack-ansible16:55
*** woodard has joined #openstack-ansible16:59
*** mgoddard_ has joined #openstack-ansible17:00
*** sacharya has quit IRC17:00
evrardjpahah copying policies!17:00
evrardjpI have another thing for that17:00
evrardjpcustom module too!17:01
evrardjpthat's what I have done in the past...17:01
evrardjpit proves it's needed17:01
evrardjpI'll add documentation on this one: https://review.openstack.org/#/c/24881917:02
*** mgoddard has quit IRC17:03
logan-yeah that is funny, site branding was next on the list for me :P17:04
evrardjpnow it's done17:04
evrardjpfor the policies, I'd rather create another playbook for that17:05
logan-yes thanks for that. a couple pieces of these would be easier to implement for operators if we had config_template style overrides on local_settings.py in the horizon role17:05
evrardjpand another role17:05
evrardjpyup, but I was lazy17:05
evrardjp:p17:05
*** phiche1 has quit IRC17:07
odyssey4meevrardjp logan- it would seem that copying policies should just use the config_template?17:10
evrardjppolicies should be handled differently17:10
logan-yep the policy copy does, im not sure the implementation there is the correct way17:10
logan-it is symlinking to all of the separate roles policy templates and using their overrides if set17:11
evrardjpthe problem of policies: they aren't the same within the component and horizon17:11
logan-yup17:11
evrardjpwhat I've implemented on my side, is a way to download the policy files for each component17:11
evrardjpand then a way to upload on horizon, about the same way as yours17:12
evrardjpthis way you have time to adjust the mapping17:12
logan-ah17:12
evrardjpit's maybe not required for you17:13
*** galstrom is now known as galstrom_zzz17:13
evrardjpbut with config_template is maybe possible to take the override dict from each component, and fill it with other values to override the override, and then send it to horizon17:14
logan-just curious.. what is the reason you would have differing policies on horizon vs. the service container?17:15
evrardjpbecause sometimes horizon just doesn't map with the same name (different cycles of deprecations)17:16
logan-ah ok17:16
evrardjpthe other way would be to just use config_template for each component, and let the deployer edit its own dicts for that17:17
evrardjpand not care about the automatic way17:17
evrardjpit's maybe the best solution, because we could (later) combine override from a component into these variables17:18
*** baker has quit IRC17:20
*** phiche has joined #openstack-ansible17:20
*** galstrom_zzz is now known as galstrom17:33
*** baker has joined #openstack-ansible17:34
*** rebase has joined #openstack-ansible17:35
*** tiagogomes has quit IRC17:43
*** baker has quit IRC17:50
*** egonzalez has joined #openstack-ansible17:54
odyssey4mestevelle sigmavirus24_awa here's an odd one for you - if I set the glance {api,registry} workers to n, the number of processes running always seems to end up being n+117:55
*** egonzalez is now known as egonzalez9017:55
*** baker has joined #openstack-ansible17:55
palendaeodyssey4me: I think they're both out all week, fwiw17:55
*** egonzalez90 has quit IRC17:55
*** galstrom is now known as galstrom_zzz17:56
jmccroryodyssey4me: think the +1 is the parent, workers sets the number of child processes17:58
odyssey4mejmccrory oh? nova works differently though17:59
odyssey4mejmccrory but you're right - one is a parent of the other17:59
jmccrorypossibly, remember running into the n+1 before with something but can't remember exactly which service it was17:59
*** mgoddard_ has quit IRC18:01
*** mgoddard has joined #openstack-ansible18:01
logan-http://paste.gentoolinux.info/oricidojax.hs18:01
logan-is what im seeing for nova api18:01
*** egonzalez has joined #openstack-ansible18:02
*** daneyon has joined #openstack-ansible18:03
odyssey4melogan- jmccrory quite right - I see now that nova has the same behaviour :)18:05
odyssey4memhayden cloudnull fyi - I'm working on some tweaks to gate configs to reduce resource usage again... it seems that a lot of the gate fails relate to low resource availability again18:07
*** daneyon_ has quit IRC18:07
*** baker has quit IRC18:10
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible: Allow Horizon Custom Branding  https://review.openstack.org/24881918:10
*** sacharya has joined #openstack-ansible18:13
*** egonzalez has quit IRC18:16
*** eil397 has joined #openstack-ansible18:16
evrardjpis it possible to merge stuff even with Jenkins giving a -1?18:17
evrardjpAll these "New git dotfiles for independant repository" should be merged as soon as possible IMO18:17
eil397as i know it possible18:17
odyssey4meevrardjp yes, but only if we merge something into the gate checks to change the checks done18:18
evrardjpI see :p18:19
odyssey4meevrardjp the fix for the gate is waiting for one more +218:19
evrardjpyeah sorry18:19
evrardjpanyway I shouldn't continue to work for today :p18:19
*** rromans_ is now known as rromans18:22
*** fawadkhaliq has quit IRC18:24
*** sacharya has quit IRC18:24
*** fawadkhaliq has joined #openstack-ansible18:24
cloudnullhave a good one evrardjp18:25
evrardjpI'll follow these gate changes, this way I can adapt the (I hope soon) working solution for all the independant roles18:26
evrardjpbye18:26
cloudnullodyssey4me: do we have an OSA dashboard setup for the new repos ?18:27
odyssey4mecloudnull see the link list on https://launchpad.net/openstack-ansible18:27
odyssey4methe bottom two links18:27
*** fawadkhaliq has quit IRC18:28
*** fawadkhaliq has joined #openstack-ansible18:29
*** fawadkhaliq has quit IRC18:29
kmARCHi all again! I have problems with the repo container deployment. The error message says that one or two containers are unreachable.18:31
kmARCThe problem stems from - I guess - that there are for example DUP answers from ping18:32
kmARCnow I use different networks for mgmt, vxlan and storage18:32
kmARCon a /23 network18:32
odyssey4mekmARC oh? we're hitting the same problem fairly frequently in the gate checks so we're trying to figure out why18:32
kmARCand blacklisted the hosts18:32
kmARCOh thank God18:33
kmARCSoooo.... any workarounds? :)18:33
odyssey4mekmARC well, retry is one - but that doesn't work for the gate :/18:33
odyssey4mekmARC another attempt is to use 'transport = paramiko' in ansible.cfg18:33
odyssey4methat switches how connections are setup and torn down, and seems to be reliable18:34
cloudnullkmARC odyssey4me: https://review.openstack.org/#/c/248361/18:34
odyssey4meyep, I'm trying this too: https://review.openstack.org/24873918:34
odyssey4mebut hitting resource issues causing failures - so busy working on that now18:34
cloudnulljust as a note, the nv job seems to be passing with more regularity than dsvm18:35
* cloudnull lunching18:35
*** _hanhart has joined #openstack-ansible18:36
odyssey4mecloudnull oh nice... if that's the case then there may be a package diff... although it could also be something to do with the hosts they run on :/18:36
kmARCso which section should I define the paramiko transport?18:36
kmARC[ssh_connection]? (I guess)18:36
odyssey4mekmARC in defaults18:37
kmARCOK now I see in the git review18:44
kmARCI teared down, now let's see18:44
*** _hanhart has quit IRC18:49
*** alkari has quit IRC18:52
*** cemmason1 has joined #openstack-ansible18:53
*** baker has joined #openstack-ansible19:00
mgariepyquick question for git experts ;), i'm on tag 11.2.3 which i checkedout with : git checkout -b kilo-11.2.3 11.2.3, on top of that i pull some patches (cherry-pick), is there a simple way to rebase to update to tag 11.2.5 ?19:14
*** baker has quit IRC19:15
mgariepywithout loosing my patches19:15
*** ashishb has joined #openstack-ansible19:23
*** gouthamr has joined #openstack-ansible19:35
*** Bjoern_ has joined #openstack-ansible19:35
openstackgerritJesse Pretorius proposed openstack/openstack-ansible: Optimised the workers used for all services for the AIO  https://review.openstack.org/24888719:36
*** jmvalente has quit IRC19:36
*** Bjoern_ is now known as BjoernT19:36
cloudnullmgariepy:  you should be able to do a `git rebase -i gerrit/11.2.5` which will take you step by step through the rebase.19:38
*** baker has joined #openstack-ansible19:38
cloudnullI'd recommend commiting your patches and making a backup19:38
cloudnullalso if your patches have been picked clean you can rebase the kilo branch and then apply your patch branch on top19:39
*** skamithi has quit IRC19:39
*** baker has quit IRC19:43
mgariepycloudnull, my patches are all commited and most of them are cherry-pick from master or kilo branches.19:45
cloudnullso it should cleanly apply19:46
cloudnullyou may have a conflict but rebasing your feature branch with the tag you want to go to should be all thats needed.19:47
kmARCodyssey4me: I still get the DUP pings with transport = paramiko19:47
cloudnullkmARC:  is it always to the first container in the build process ?19:48
cloudnulland does a rerun make it happy ?19:48
*** greg_a has joined #openstack-ansible19:48
cloudnullor is it intermitant ?19:48
cloudnullkmARC: sorry if im asking questions that have already been aswered .19:48
openstackgerritKevin Carter proposed openstack/openstack-ansible: Keystone domain fix  https://review.openstack.org/23850919:52
openstackgerritJesse Pretorius proposed openstack/openstack-ansible: Optimise the workers used for all services for the AIO  https://review.openstack.org/24888719:53
*** neillc_ is now known as neillc19:56
*** mss_ has joined #openstack-ansible20:04
*** mss has quit IRC20:07
kmARCoh sorry was out20:11
kmARCyes, the first dies immediately20:11
kmARCin setup-infrastructure20:11
kmARCif I retry with the suggested --limit, I get the same. Authentication Failed20:12
kmARCand still DUP ping20:12
cloudnullthats an ssh "authentication failed"?20:15
*** egonzalez has joined #openstack-ansible20:17
*** ashishb has quit IRC20:18
kmARCyes20:18
kmARCI guess20:18
*** egonzalez has quit IRC20:18
kmARCyes it is20:18
odyssey4mekmARC and can you ssh to the container?20:19
kmARCof course :)20:19
odyssey4mekmARC are you running as a non root user?20:19
kmARCnope, everything is root20:19
kmARCthis worked when I wasn't using paramiko, tho20:20
*** tlian has joined #openstack-ansible20:21
*** tlian has quit IRC20:23
openstackgerritJesse Pretorius proposed openstack/openstack-ansible: Update Liberty SHA's - 23.11.2015  https://review.openstack.org/24596420:25
openstackgerritJesse Pretorius proposed openstack/openstack-ansible: Update Liberty SHA's - 23.11.2015  https://review.openstack.org/24596420:26
odyssey4mekmARC try removing the paramico line, but setting ssh_pipelining to False?20:27
kmARCok, now it didn't stop after the first container20:29
kmARCbut I still have the DUPs20:30
kmARCshould I tear down first?20:30
odyssey4meI'm not sure if the DUPs are bad or what may cause them. Perhaps rackertom or Apsu can help you figure that out?20:31
*** tlian has joined #openstack-ansible20:31
* Apsu perks up20:32
ApsuWhat's up?20:32
rackertomHello20:32
rackertomDUPE ping responses?20:32
ApsuNetwork Rangers, deploy20:32
*** greg_a has quit IRC20:32
odyssey4meyeah, kmARC appears to be getting duplicate ping responses - from containers, I believe20:33
ApsukmARC: If you're getting duplicate ping responses when pinging a container IP, you probably have (wait for it)... duplicate IPs. :D20:33
ApsuEasy way to tell is run a tcpdump and check the MAC addresses sending the responses.20:33
Apsutcpdump -i br-mgmt (or whatever) -npevl icmp20:33
odyssey4meargh, with all the teardowns, etc that might actually be possible20:34
ApsuBackground that, do your ping, and look at the MACs in the ICMP Echo Reply20:34
ApsuThen, if you're feeling enterprising... you can figure out which host veth the offending MACs are coming in on with brctl showmacs br-mgmt | grep $MAC20:35
ApsuIt'll show the port20:35
ApsuIf you're using recent enough OSA that your veths are named after the containers they're hooked to... yay20:35
ApsuIf not, life gets harder.20:35
rackertomApsu: Could possibly see the dupe entries in `ip neigh show`, no?20:35
*** KLevenstein has quit IRC20:35
Apsurackertom: It would rotate, after each ARP20:36
ApsuFlapping like crazy, another good sign you've got duplicate IPs20:36
rackertomYeah, I meant as you were performing a ping, you should see the MAC change in the output. Not as positive a detection as tcpdump, but people may have varying skill levels and comfort with tcpdump :)20:37
ApsuSure20:37
*** greg_a has joined #openstack-ansible20:38
openstackgerritKevin Carter proposed openstack/openstack-ansible: Fixes playbook runtime issues with ldap  https://review.openstack.org/24890020:40
kmARCThanks Apsu, I really wanted not to tuch tcpdump :) But my problem is that I now have a /23 space. All the IP's assigned by me are blacklisted in the user.yaml. And after teardown and removing ansible_facts my networks are fine. Running setup-hosts screws it20:40
ApsukmARC: After you tear everything down can you still ping the IP(s) in question? :P20:40
kmARCwhatever happens, the playbooks should NOT set up any containters with clashing IPs20:40
kmARCApsu, no, of cors not20:41
ApsuWell that's a good sign. Means the duplicate is not somewhere else on the network.20:41
ApsuOk... next question...20:41
kmARCThe fact is, I have a perfectly set up network. With big enough address space. Then I run the ansible scripts, and my network is screwed20:41
SamYaplebonding with an old kernel causes DUPs20:41
ApsuYou setup your own bridges on your host(s), and are any of those bridges set to promiscuous mode?20:42
Apsuip -o l | grep PROMISC20:42
ApsuIf your bridges are in promiscuous mode, you can get duplicate ping responses without duplicate IPs20:42
kmARCno outpu20:42
kmARCI gues that means no they arent20:42
ApsuIndeed. Also a good sign.20:42
ApsuTo be really clear, when you say "of course not", did you actually try pinging the IP to verify? :P20:43
kmARC... yes? :)20:43
ApsuJust making sure. Because if the IP existed elsewhere on the network then no troubleshooting of containers would ever find it20:43
ApsuSo we have to confirm that we can't ping it before deploying the containers, is all.20:44
kmARCno it didn't. Actually I configured a separate vlan and tried it too20:44
ApsuCool.20:44
ApsuSo definitely hiding in containers somehow.20:44
ApsuEither through broken bridging creating a loop or actual duplicate IPs.20:44
kmARCI guess so. Is there a convenient way to squeeze this info out from one of the lxc-* commands?20:44
odyssey4meok, that's me out for the night - catch y'all tomorrow20:45
ApsuWell the correct way to assess the situation is to tcpdump in the same location you're pinging from, to see if you get different MAC addresses on replies.20:45
ApsuIt's really not that scary. Just use the flags and parameters I gave above20:45
kmARCah okay. let's learn how to do this then :)20:45
kmARCokay20:45
Apsu-i for interface, which is the bridge for the subnet you're pinging on. I assume br-mgmt20:46
Apsu-npevl for the other flags, and lastly 'icmp' for the protocol to listen for20:46
kmARCso basically when I ping from the infra-2-memcahced the deploy machine (and vice versa) I have the DUP syndrome20:46
kmARCokay20:46
ApsuAre you pinging on the management subnet? br-mgmt bridge IPs?20:47
kmARCyes20:47
openstackgerritBjoern Teipel proposed openstack/openstack-ansible: Creating nova-secret with safe file permissions  https://review.openstack.org/24890420:47
ApsuOk. Well, lxc-ls -f will show all the IPs assigned to all the containers20:47
ApsuSo you can grep the output for the IP in question and see if it shows up twice20:48
*** gouthamr has quit IRC20:49
ApsuBut as for tcpdump, use those flags, do your ping, and look at the ICMP Echo Reply output20:49
ApsuNear the beginning of each line will be the MAC addresses for source and destination20:49
ApsuIn the replies, we care about the source MAC20:49
Apsu(the first one)20:50
rackertomApsu: Can confirm, finding the error with ip neigh would be very difficult20:50
Apsurackertom: Seems legit, but yeah, it's rough to pin down without logging of state changes20:50
*** gouthamr has joined #openstack-ansible20:50
*** elo has quit IRC20:51
kmARChm. without pinging I can see this message in tcpdump's output: br-mgmt: received packet on eth1 with own add20:51
kmARCress as source address20:51
ApsuInteresting20:51
ApsuRun: dmesg | tail20:51
openstackgerritMiguel Alex Cantu proposed openstack/openstack-ansible: Add documentation for HA ceilometer  https://review.openstack.org/24890520:51
ApsuSee if the last few lines from the kernel log are complaining loudly about network loops20:51
kmARCyeah, full of it20:51
*** skamithi has joined #openstack-ansible20:52
ApsuSo you've got a loop! Neat. Now to figure out if it's software or hardware.20:52
ApsuFirst should find out what interfaces are in the br-mgmt bridge20:52
*** javeriak has quit IRC20:52
rackertomkmARC Apsu I think we found an issue a few months ago where if the address space was small enough, then we would get duplicate addresses anyway20:52
rackertomI think cloudnull saw it, too?20:52
ApsuYou can either run ip -o | grep br-mgmt, or brctl show br-mgmt20:53
kmARCrackertom: I'm using a /23 is that small?20:53
ApsuNah, that's fine20:53
ApsuAlso that's not what's happening here20:53
Apsutcpdump/the kernel complaining about "same address" isn't IP address20:53
ApsuIt's MAC20:53
ApsuAnd that's a network loop :)20:53
rackertoma /23 should be just over 500 addresses20:53
kmARCwell, 600 :)20:54
Apsu51020:54
kmARCOh, sure20:54
rackertomyeah, 512-220:54
Apsu2^n-220:54
ApsukmARC: So, aside from a bunch of veth pairs, what else is in br-mgmt?20:54
kmARCnope20:54
ApsuAny physical interfaces?20:54
kmARCyes20:54
kmARCet120:54
kmARCeth120:54
ApsuLets take eth1 out of br-mgmt for a moment and see if the dupes go away20:55
ApsuYou can easily do so with ip link20:55
Apsuip link set eth1 nomaster20:55
kmARCokay, I pullet that out20:55
ApsuPing some of the containers local to this box20:56
kmARCyeah this works fine20:56
ApsuCool. So the loop is external somehow20:56
kmARCoh wait20:56
kmARCI just looked at dmesg20:57
kmARCit says "dev eth1 left promisc mode"20:57
kmARCis this okay?20:57
ApsuYes, that's ok20:57
kmARCokay that makes sense yes. It actually needs to be in promisc if my guess is right20:57
*** KLevenstein has joined #openstack-ansible20:58
ApsuYeah it's fine. Links in bridges behave promiscuously20:58
ApsuSo they can pass traffic to MACs not their own20:58
ApsuYou just don't usually want the bridge Itself to behave promiscuously20:59
ApsuSo, with eth1 out of br-mgmt, no more dupes?20:59
kmARCwell, now I can't ping the deploy machine20:59
kmARCso can't tell. However in dmesg I cant see any suspicous messages either20:59
ApsuOk20:59
ApsuWell we can put eth1 back in and take a look on the deploy machine21:00
ApsuMaybe the issue is there21:00
Apsuip link set eth1 master br-mgmt21:00
*** gouthamr has quit IRC21:00
ApsuIs there a similar bridge on the deploy machine?21:00
kmARCyes21:00
ApsuI assume no containers there?21:00
kmARCnope21:00
ApsuCool. Mind pasting a link to the output of "ip -o l"?21:01
kmARCso now I don't have the DUPs21:01
ApsuIt won't include IPs so shouldn't have to sanitize21:01
ApsuEr... ok21:01
kmARCoh wait21:01
kmARCI was pinging from outside the container21:01
kmARCcool21:01
kmARCnow I tried from inside the container21:02
kmARCnow _ALL_ three "memcahced" containers have DUPs21:02
ApsuLet's try something21:02
ApsuTake eth1 back out, and ping one container from another21:02
ApsuThat are on the same box21:02
ApsuMake sure that works without dupes21:02
kmARCok21:03
ApsuWhich will tend to rule out the bridge/veth/container setup21:03
ApsuI suspect it's on the deploy box or the switching inbetween21:03
kmARCSo now I'm pinging from the memcahced container the utility container, and don't have dupes. eth1 is out21:04
ApsuCool.21:04
ApsuLet's see the deploy host links :D21:07
*** sdake_ has joined #openstack-ansible21:08
kmARChttp://paste.openstack.org/show/VCNGaPBsZnqxbxJ9ksoZ/21:08
*** sdake has quit IRC21:09
ApsuNo :)21:09
ApsuThat's ifconfig. I need to see "ip -o link"21:09
kmARChttp://paste.openstack.org/show/B3hh89S4PEHSJpI9Vtza/21:09
ApsuThere it is!21:10
ApsuOk... so... good news is it looks good.21:10
ApsuBad news is... it looks good.21:10
ApsuMight be a switch loop. Do you see errors in dmesg on this box too?21:11
kmARCbr-mgmt: received packet on eth1 with own address as source address21:11
kmARCfull of these21:11
ApsuYep21:11
ApsuPretty sure it's external to the boxes21:12
*** jasondotstar has quit IRC21:12
kmARCSo these boxes are vm's on a vmware single hypervisor. No distributed switch, no nothing, only a vlan21:12
kmARCbut now I'm thinking of putting these guys on a completely different, new vlan created by me21:13
ApsuThat sounds like a good plan21:13
ApsuThe VMware virtual switch the VMs are on can definitely have a loop created21:13
ApsuAssuming there's not a physical switch somewhere with a patch cable no one noticed ;P21:13
kmARCthe problem is that I already tried putting the management interfaces on different vlan21:13
*** rromans has quit IRC21:14
*** rromans has joined #openstack-ansible21:15
kmARCYeah, so now I double checked we have only one other machine on this hypervisor, and it's IP is not the dupe :)21:16
openstackgerritMajor Hayden proposed openstack/openstack-ansible: Use paramiko transport for gate check jobs  https://review.openstack.org/24836121:16
openstackgerritMajor Hayden proposed openstack/openstack-ansible: AIO bootstrap in Ansible  https://review.openstack.org/23952521:18
*** rromans has quit IRC21:18
*** jasondotstar has joined #openstack-ansible21:18
openstackgerritMajor Hayden proposed openstack/openstack-ansible: Use Ansible AIO playbook for run-aio-build.sh  https://review.openstack.org/24470821:19
openstackgerritMajor Hayden proposed openstack/openstack-ansible: Updating AIO docs for Ansible playbook  https://review.openstack.org/24472021:19
ApsukmARC: Another good sign!21:19
ApsuBut I don't think it's duplicate IPs21:20
ApsuI think it's duplicate MACs :)21:20
kmARCoh shit21:20
mhaydenApsu: just don't use a Mac21:21
*** elgertam1 has joined #openstack-ansible21:21
Apsumhayden: GENIUS!21:22
mhaydenApsu: had to come in on my day off to heckle you at least once21:22
Apsuhaha21:22
mhaydenand it can't be a monday without one additional patch set for the AIO bootstrap review21:22
ApsuDay = complete21:22
kmARChm. So for a second I thought when I cloned the machine's then the MAC's weren't replaced but this is not the case21:23
ApsukmARC: Maybe the vmware MAC assignment overlaps our LXC container MAC assignment21:23
*** skamithi has quit IRC21:23
kmARCchances are...? :-)21:23
ApsuHehe, that would explain a lot!21:23
kmARCI mean I retried like 5 times, with different settings, always removing the containers.21:24
spotzmhayden don't change it again I reviewed it:)21:24
*** sdake_ has quit IRC21:42
*** sdake has joined #openstack-ansible21:42
*** tlian has quit IRC21:46
*** gouthamr has joined #openstack-ansible21:48
*** elo has joined #openstack-ansible21:48
*** woodard_ has joined #openstack-ansible21:51
*** elgertam1 has quit IRC21:51
*** alkari has joined #openstack-ansible21:52
*** elgertam1 has joined #openstack-ansible21:52
*** elgertam1 has quit IRC21:52
*** woodard has quit IRC21:55
*** jhesketh has quit IRC21:55
*** jhesketh has joined #openstack-ansible21:55
*** woodard_ has quit IRC21:56
*** rromans has joined #openstack-ansible21:57
*** rromans has quit IRC21:57
*** rromans has joined #openstack-ansible21:57
*** harlowja_ has joined #openstack-ansible21:59
*** harlowja has quit IRC21:59
*** cemmason1 has quit IRC22:00
*** cemmason has joined #openstack-ansible22:00
*** greg_a has quit IRC22:29
*** daneyon_ has joined #openstack-ansible22:36
*** phiche has quit IRC22:36
*** elo has quit IRC22:37
*** sdake has quit IRC22:37
*** daneyon has quit IRC22:39
*** sdake has joined #openstack-ansible22:39
*** rromans has quit IRC22:56
*** rromans has joined #openstack-ansible22:56
*** BjoernT has quit IRC22:57
*** HimanshuGarg has quit IRC23:07
openstackgerritBjoern Teipel proposed openstack/openstack-ansible: Creating nova-secret with safe file permissions  https://review.openstack.org/24890423:15
*** jwagner is now known as jwagner_away23:16
*** galstrom_zzz is now known as galstrom23:27
*** alkari has quit IRC23:28
*** spotz is now known as spotz_zzz23:30
*** jhesketh has quit IRC23:31
*** josh6627 has joined #openstack-ansible23:31
*** cemmason has quit IRC23:31
*** skamithi13 has joined #openstack-ansible23:35
*** skamithi13 has quit IRC23:35
*** josh6627 has quit IRC23:35
*** skamithi13 has joined #openstack-ansible23:36
*** KLevenstein has quit IRC23:38
*** neillc is now known as neillc_away23:51
*** sacharya has joined #openstack-ansible23:55

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!