Tuesday, 2016-03-15

*** saneax is now known as saneax_AFK00:23
*** vinsh has quit IRC00:23
*** dims has joined #openstack-operators00:23
*** vinsh has joined #openstack-operators00:23
*** vinsh_ has joined #openstack-operators00:27
*** vinsh has quit IRC00:27
*** Marga__ has quit IRC00:27
*** Marga_ has joined #openstack-operators00:28
*** lhcheng has quit IRC00:30
*** mnaser has quit IRC00:31
*** dims has quit IRC00:32
*** furlongm has joined #openstack-operators00:35
*** markvoelker has joined #openstack-operators00:40
*** mnaser has joined #openstack-operators00:41
*** SimonChung has joined #openstack-operators00:41
*** SimonChung has quit IRC00:42
*** SimonChung2 has joined #openstack-operators00:42
*** SimonChung has joined #openstack-operators00:43
*** SimonChung2 has quit IRC00:43
*** SimonChung1 has quit IRC00:44
*** SimonChung has quit IRC00:47
*** dims has joined #openstack-operators00:56
*** ducttape_ has joined #openstack-operators01:12
*** ducttape_ has quit IRC01:12
*** ducttape_ has joined #openstack-operators01:14
*** markvoelker has quit IRC01:14
*** SimonChung has joined #openstack-operators01:21
*** kstev has quit IRC01:22
*** SimonChung1 has joined #openstack-operators01:23
*** SimonChung has quit IRC01:25
*** chlong has quit IRC01:26
*** ducttape_ has quit IRC01:36
*** ducttape_ has joined #openstack-operators01:38
*** harshs has quit IRC01:42
*** ducttape_ has quit IRC01:54
*** rockyg has joined #openstack-operators01:54
*** apoorvad has quit IRC01:58
*** piet has joined #openstack-operators02:01
*** ducttape_ has joined #openstack-operators02:02
*** piet has quit IRC02:06
*** piet has joined #openstack-operators02:06
*** fawadkhaliq has quit IRC02:12
*** ducttape_ has quit IRC02:23
*** rockyg has quit IRC02:31
*** dims has quit IRC02:38
*** armax has quit IRC02:38
*** lhcheng has joined #openstack-operators02:44
*** armax has joined #openstack-operators02:58
*** fawadkhaliq has joined #openstack-operators03:06
*** markvoelker has joined #openstack-operators03:11
*** esker has quit IRC03:13
*** lhcheng_ has joined #openstack-operators03:26
*** lhcheng has quit IRC03:29
*** chlong has joined #openstack-operators03:30
*** haukebruno has quit IRC03:38
*** haukebruno has joined #openstack-operators03:40
*** markvoelker has quit IRC03:44
*** VW has joined #openstack-operators03:55
*** haukebruno has quit IRC03:57
*** Marga_ has quit IRC03:59
*** Marga_ has joined #openstack-operators04:00
*** Marga_ has quit IRC04:00
*** piet has quit IRC04:19
*** piet has joined #openstack-operators04:19
*** armax has quit IRC04:20
*** vsedelnik has joined #openstack-operators04:20
*** Marga_ has joined #openstack-operators04:21
*** vsedelnik has quit IRC04:23
*** armax has joined #openstack-operators04:25
*** Marga_ has quit IRC04:25
*** piet has quit IRC04:30
*** piet has joined #openstack-operators04:31
*** Marga_ has joined #openstack-operators04:33
*** Marga_ has quit IRC04:34
*** Marga_ has joined #openstack-operators04:34
*** piet has quit IRC04:44
*** piet has joined #openstack-operators04:53
*** VW has quit IRC05:26
*** armax has quit IRC05:29
*** lhcheng has joined #openstack-operators05:40
*** markvoelker has joined #openstack-operators05:41
*** piet has quit IRC05:44
*** lhcheng_ has quit IRC05:44
*** lhcheng has quit IRC05:45
*** armax has joined #openstack-operators05:48
*** saneax_AFK is now known as saneax06:00
*** furlongm has quit IRC06:00
*** furlongm has joined #openstack-operators06:05
*** vsedelnik has joined #openstack-operators06:07
*** vsedelnik has quit IRC06:13
*** markvoelker has quit IRC06:14
*** simon-AS559 has joined #openstack-operators06:17
*** lhcheng has joined #openstack-operators06:43
*** lhcheng_ has joined #openstack-operators06:59
*** lhcheng has quit IRC07:03
*** admin0 has joined #openstack-operators07:11
*** tesseract has joined #openstack-operators07:13
*** tesseract is now known as Guest9751907:13
*** vsedelnik has joined #openstack-operators07:19
*** vsedelnik has quit IRC07:19
*** vsedelnik has joined #openstack-operators07:19
*** liverpooler has joined #openstack-operators07:24
*** simon-AS559 has quit IRC07:25
*** VW has joined #openstack-operators07:26
*** maishsk has joined #openstack-operators07:29
*** fawadkhaliq has quit IRC07:30
*** VW has quit IRC07:30
*** liverpooler has quit IRC07:33
*** liverpooler has joined #openstack-operators07:34
*** rarcea has joined #openstack-operators07:35
*** lhcheng_ has quit IRC07:50
*** belmoreira has joined #openstack-operators07:58
*** Guest97519 has quit IRC08:00
*** rarcea has quit IRC08:03
*** matrohon has joined #openstack-operators08:04
*** simon-AS559 has joined #openstack-operators08:05
*** simon-AS5591 has joined #openstack-operators08:05
*** tesseract has joined #openstack-operators08:05
*** tesseract is now known as Guest9179408:06
*** admin0 has quit IRC08:07
*** bvandenh has joined #openstack-operators08:08
*** simon-AS559 has quit IRC08:10
*** markvoelker has joined #openstack-operators08:12
*** haukebruno has joined #openstack-operators08:29
*** maishsk_ has joined #openstack-operators08:31
*** fawadkhaliq has joined #openstack-operators08:31
*** maishsk has quit IRC08:32
*** maishsk_ is now known as maishsk08:32
*** chlong has quit IRC08:32
*** markvoelker has quit IRC08:44
*** Guest91794 has quit IRC08:45
*** zeih has joined #openstack-operators08:50
*** admin0 has joined #openstack-operators08:58
*** Guest91794 has joined #openstack-operators09:00
*** admin0 has quit IRC09:01
*** luisfdez1 has joined #openstack-operators09:07
*** luisfdez has quit IRC09:07
*** admin0 has joined #openstack-operators09:10
*** zeih has quit IRC09:18
*** admin0 has quit IRC09:19
*** admin0 has joined #openstack-operators09:22
*** rcernin has joined #openstack-operators09:31
*** admin0 has quit IRC09:32
*** admin0 has joined #openstack-operators09:35
*** lhcheng has joined #openstack-operators09:38
*** mathlin has joined #openstack-operators09:40
*** maishsk has quit IRC09:40
*** lhcheng has quit IRC09:43
*** derekh has joined #openstack-operators09:47
*** luisfdez has joined #openstack-operators10:11
*** liverpooler has quit IRC10:12
*** liverpooler has joined #openstack-operators10:12
*** luisfdez1 has quit IRC10:14
*** luisfdez has quit IRC10:18
*** electrofelix has joined #openstack-operators10:21
*** admin0 has quit IRC10:28
*** fwdit has joined #openstack-operators10:37
*** admin0 has joined #openstack-operators10:42
*** markvoelker has joined #openstack-operators10:42
*** luisfdez has joined #openstack-operators10:47
*** admin0 has quit IRC10:53
*** maishsk has joined #openstack-operators11:02
*** dims has joined #openstack-operators11:12
*** markvoelker has quit IRC11:15
*** dims has quit IRC11:20
*** dims has joined #openstack-operators11:21
*** dims has quit IRC11:21
*** dims has joined #openstack-operators11:21
claytonThe OVS restart bug got pushed back to newton: https://bugs.launchpad.net/neutron/+bug/151405611:38
openstackLaunchpad bug 1514056 in neutron "Restarting OVS agent drops VMs traffic when using VLAN provider bridges" [High,In progress] - Assigned to Hynek Mlnarik (hmlnarik-s)11:38
*** admin0 has joined #openstack-operators11:49
*** lhcheng has joined #openstack-operators11:51
*** lhcheng has quit IRC11:55
*** eddima has joined #openstack-operators12:03
*** admin0_ has joined #openstack-operators12:05
*** bvandenh has quit IRC12:06
*** belmoreira has quit IRC12:07
*** admin0 has quit IRC12:08
*** admin0_ is now known as admin012:08
*** ducttape_ has joined #openstack-operators12:10
raginbajinclayton:  Any reason why they pushed it back?12:11
claytonmy guess was just that it was taking too long12:12
*** eddima has quit IRC12:17
*** saneax is now known as saneax_AFK12:17
*** liverpooler has quit IRC12:18
*** liverpooler has joined #openstack-operators12:19
*** eddima has joined #openstack-operators12:19
*** ducttape_ has quit IRC12:23
*** dims has quit IRC12:25
*** dims has joined #openstack-operators12:26
*** piet has joined #openstack-operators12:29
*** vsedelnik has quit IRC12:29
*** vsedelnik has joined #openstack-operators12:30
*** armax has quit IRC12:33
*** markvoelker has joined #openstack-operators12:53
*** vsedelnik has quit IRC12:57
*** armax has joined #openstack-operators13:05
*** maishsk has quit IRC13:08
*** maishsk has joined #openstack-operators13:08
*** ctrath has joined #openstack-operators13:14
*** ig0r__ has joined #openstack-operators13:19
*** piet has quit IRC13:20
*** armax has quit IRC13:21
*** ig0r_ has quit IRC13:22
*** chlong has joined #openstack-operators13:25
*** mriedem_away is now known as mriedem13:30
*** piet has joined #openstack-operators13:34
*** luisfdez has quit IRC13:34
*** piet has quit IRC13:34
*** luisfdez has joined #openstack-operators13:35
krobzaurmjrichardson: Hey, thanks for the pointers guys. I just rebooted the box that was misbehaving and corosync/pacemaker services restarted properly during system init.13:37
*** krobzaur has quit IRC13:37
*** ducttape_ has joined #openstack-operators13:38
*** dminer has joined #openstack-operators13:50
*** VW has joined #openstack-operators13:54
*** VW has quit IRC13:55
*** VW has joined #openstack-operators13:56
*** saneax_AFK is now known as saneax14:01
*** krobzaur has joined #openstack-operators14:02
*** Guest41345 has quit IRC14:03
*** admin0 has quit IRC14:04
*** piet has joined #openstack-operators14:08
*** Marga_ has quit IRC14:08
*** armax has joined #openstack-operators14:10
*** armax has left #openstack-operators14:11
*** vsedelnik has joined #openstack-operators14:11
yankcrimeclayton: :(14:14
*** bvandenh has joined #openstack-operators14:14
*** VW has quit IRC14:15
*** VW has joined #openstack-operators14:16
*** krobzaur has quit IRC14:16
*** krobzaur has joined #openstack-operators14:18
*** kstev has joined #openstack-operators14:20
*** VW has quit IRC14:22
*** Guest91794 has quit IRC14:31
*** vsedelnik has quit IRC14:32
*** VW has joined #openstack-operators14:32
*** bvandenh has quit IRC14:33
*** vsedelnik has joined #openstack-operators14:33
*** VW has quit IRC14:34
*** VW has joined #openstack-operators14:34
*** regXboi has joined #openstack-operators14:47
*** bvandenh has joined #openstack-operators14:49
*** maishsk has quit IRC14:54
*** maishsk has joined #openstack-operators14:56
*** markvoelker has quit IRC15:00
*** kstev has quit IRC15:00
*** bvandenh has quit IRC15:00
*** SimonChung1 has quit IRC15:01
klindgrenkrobzaur, what all are you running under pacemaker?  We are running everything active-active-active15:04
*** kstev has joined #openstack-operators15:04
*** tesseract has joined #openstack-operators15:06
notmynamecross-post from the mailing list...15:07
*** harshs has joined #openstack-operators15:07
*** tesseract is now known as Guest7146715:07
notmynameSwift recently added an ops runbook upstream: http://docs.openstack.org/developer/swift/ops_runbook/index.html15:07
notmynameI hope it will be helpful to anyone running Swift clusters15:07
krobzaurklindgren: A whole bunch of stuff. I'm using Fuel to deploy and so almost everything is configured and installed automatically. But I have 3 controllers. Most of my pacemaker resources are active-active-active (neutron vpn,metadata, and dhcp agents. mysql. heat-engine and ceilometer agent-central and alarm-evaluator)15:11
krobzaurThen rabbitmq and conntrackd are in a master-slave setup15:12
krobzaurLike I said, Fuel did all this configuration for me so I take absolutely no credit for making all this work. I'm just now starting to pick the HA aspect of things apart and try to understand how Fuel is making it all work15:13
wasmumok, think HA.  I have keystone behind lb (f5), all endpoint urls are the vips configured in the LB.  I have two "controllers" running keystone - mixed mode ad for ident sql for assignment.  I also have two neutron nodes the configs on those and the neutron-server service on the controller use the keystone vips for auth.15:13
krobzaurI also got 2 VIPs for my virtual routers on my public and openstack management network, and the controllers themselves share a vip for their interface on the public and management networks15:14
wasmumwhen one neutron-server auths, sees the agent list quiers the agents all ok15:14
wasmumsecond neutron-server node does same thing, list the agents in down state15:15
wasmumkeystone is reissueing tokens each request, i suspect15:15
wasmumsuggestions....?15:15
*** admin0 has joined #openstack-operators15:15
klindgrenWe run a clustered rabbitmq setup, with each service configured for list of rabbitmq hosts, seems to work pretty well.  Older versions of openstack did not handle rabbitmq behind a vip well at all15:16
*** mdorman has joined #openstack-operators15:16
wasmumwe also run rabbitmq cluster with all queues mirrored.  then specify the nodes with rabbit_hosts vs. rabbit_host15:16
klindgrenwasmum, are you running memcache?15:16
wasmumna, token going into sql15:17
krobzaurklindgren: Gotcha. So the config files for all your openstack services all you to specify a list of rabbit hosts to puul messages from?15:17
krobzaurallow*15:17
wasmumshould I switch to memcache and throw that port into a pool behind a vip?15:17
wasmum@krobzaur rabbit_hosts = host:a5672, hostb:567215:18
*** admin0 has quit IRC15:19
*** admin0 has joined #openstack-operators15:19
*** piet has quit IRC15:20
krobzaurwasmum: Sweet, thanks. I've done very little configuring by hand so my understanding of the config files for each service and the multitude options they contain is also pretty limited. I guess my lack of understanding stands as a testament to Fuel as a deployment method15:20
klindgrenso we run bascially the same setup as you wasmum tokens going to db - never had an issue that you were having15:20
krobzaurJust curious, when you guys deploy openstack how do you go about things?15:20
klindgrena while back neutron added support for memcache and we turned that on to address some issues we were seeing15:21
wasmumklindgren: thanks I have something configed incorrectly on a node then.  I'll comb through this second controller15:21
*** fwdit has quit IRC15:21
krobzaurMy boss put me on this and told me he wanted to use Mirantis/Fuel so thats what I did but using it seriously limits your ability to adapt/customize your deployment for different use cases15:21
klindgrenwasmum, but we do have memcache for nova/neutron/heatwahtever else will use it.  But keystone tokens still go to dob.15:22
wasmumkrobzaur: first openstack install by hand, taught me a lot about openstack.  Then we sent it to our puppet team to puppetize15:22
klindgrendb*15:22
klindgrenWe started with packstack then quickly moved to a non-packstack config (within a week).  The puppetized our configis that way15:23
klindgrenWe run masterless puppet and use ansible to performance actions (such as kicking off a puppet deployment on compute nodes)15:23
krobzaurwasmum: That makes sense. I know Fuel uses puppet behind the scenes to do all its configuration and installation. I've had to poke through some puppet manifests a few times to troubleshoot some deployment errors. I've yet to make my journey through the manual installation process though15:24
krobzaurklindgren: Gotcha. How do you guys handle deploying the OS to hardware. What are you running for a base OS? Ubuntu? CentOS?15:25
haukebruno1 year ago we hit a limit in token generation vs. galera commit latency where we used to switch to memcache as token backend. worked super well in icehouse, but I don't have any idea how the newer releases look like15:25
haukebrunofor now we are fine running galera as token backend for our kilo platforms + looking forward to use fernet tokens after upgrading to liberty15:26
haukebrunojust my 2 cents ;)15:26
krobzaurklindgren: I know Fuel use Cobbler for bare-metal booting15:26
*** harshs has quit IRC15:29
klindgrenWe have an internal system for on boarding servers into the DC.  But we are currently working on doing up an ironic install.  We Use CentOS.15:29
*** armax has joined #openstack-operators15:29
krobzaurGotcha. So what are you guys using your openstack deployments for? I'm just an intern right now doing some research/testing for the company I work for, but I studied physics in school and I am hugely interested in how Openstack and be leveraged for HPC use cases15:33
*** chlong has quit IRC15:33
krobzaurI've done some work using heat to deploy TORQUE clusters "on-demand", I think it would be really sweet to provide pre-packaged infrastructure for scientists to use in a "one-click" fashion15:34
haukebrunokrobzaur, just virtual private or private cloud here.15:35
*** eddima has quit IRC15:35
*** eddima has joined #openstack-operators15:36
haukebrunokrobzaur, I am curious what you HPC guys would expect from an openstack cloud. would you just need "big" and new CPUs?15:36
krobzaurIt really depends entirely on the application. My physics sims would just need really high clock speed CPU's (the sim is single threaded). Protein folding sims need huge amounts of memory. Astrophysics guys need tons of parallel processing (high core count)15:37
krobzaurGenome sequencing guys need to process enormous amounts of data (object storage would be absolutely perfect for them)15:37
haukebrunokrobzaur, thanks for pointing out. really interesting15:38
*** admin0 has quit IRC15:39
krobzaurSo I'm envisioning building tons of heat "stacks" that provide the virtual infrastructure for all these use cases and scientists and just use Horizon to deploy a virtual cluster catered to their use case15:39
krobzaurscientists can*15:39
krobzaurThe one thing I'm a little bummed about it the apparent lack of GPU virtualization. But for special GPU accelerated apps you could just use Ironic to deploy an environment on a special subset of bare metal servers equipped with the proper GPUs15:40
*** fawadkhaliq has quit IRC15:40
*** saneax is now known as saneax_AFK15:40
*** fawadkhaliq has joined #openstack-operators15:40
krobzaurAnyway I'm kinda rambling at this point because I get excited about these things =). But thanks for the insight guys, I really appreciate the opportunity to talk to guys who are using openstack in production environments. Nobody at my company seems to think openstack works or is worth anything (thats why they put the intern on it I suppose)15:42
*** maishsk has quit IRC15:43
wasmumhaukebruno: thanks!15:44
wasmumkrobzaur: keep excited about openstack - there are many more na sayers than you think.  i find that the na sayers are those that really need a CD to install things and a support agreement to coddle them15:45
*** fawadkhaliq has quit IRC15:45
haukebrunolol15:45
haukebrunofullack.15:45
haukebrunoof course, running an openstack platform needs a lot of learning but imho it is still worth it15:46
wasmumhell's yes.  that knowledge translates into a ton of areas as well.15:46
wasmumto forsake that is just fraking dumb15:46
wasmum*freaking15:46
wasmumI guess you can tell where I am at:)15:47
klindgrenGPU virt is more of a function of the chosen HV than openstack.  But I thought you could enable pass through and have openstack support that.  I also though I saw some purposals to make that more configurable inside openstack15:48
haukebrunowasmum, fully agree. it is also the community thing that is pretty awesome for all the different projects and stuff. I never felt that comfortable or familiar inside a community  /o/15:51
*** gyee has joined #openstack-operators15:51
*** maishsk has joined #openstack-operators15:53
wasmumhaukebruno: agree 100%15:54
*** eddima1 has joined #openstack-operators15:55
*** kstev has quit IRC15:57
*** eddima has quit IRC15:57
*** eddima1 is now known as eddima15:57
*** kstev has joined #openstack-operators16:00
*** kstev1 has joined #openstack-operators16:01
*** maishsk has quit IRC16:03
*** kstev has quit IRC16:04
*** armax has quit IRC16:16
*** kazuIchikawa has joined #openstack-operators16:20
*** permalac has joined #openstack-operators16:22
*** permalac has quit IRC16:22
*** permalac has joined #openstack-operators16:23
*** permalac has quit IRC16:23
*** markvoelker has joined #openstack-operators16:25
*** permalac_ has joined #openstack-operators16:31
*** permalac_ has quit IRC16:31
*** Marga_ has joined #openstack-operators16:35
*** rcernin has quit IRC16:35
*** eil397 has joined #openstack-operators16:38
*** SimonChung has joined #openstack-operators16:39
*** eddima has quit IRC16:41
*** Guest71467 has quit IRC16:42
*** harshs has joined #openstack-operators16:43
*** piet has joined #openstack-operators16:43
*** apoorvad has joined #openstack-operators16:51
*** fawadkhaliq has joined #openstack-operators17:00
*** SimonChung1 has joined #openstack-operators17:01
*** SimonChung has quit IRC17:01
*** fawadkhaliq has quit IRC17:10
*** admin0 has joined #openstack-operators17:14
*** openstackgerrit has quit IRC17:14
*** openstackgerrit_ is now known as openstackgerrit17:14
*** openstackgerrit has quit IRC17:15
*** openstackgerrit_ has joined #openstack-operators17:15
*** openstackgerrit_ is now known as openstackgerrit17:16
*** openstackgerrit_ has joined #openstack-operators17:16
*** fawadkhaliq has joined #openstack-operators17:18
*** pcaruana has quit IRC17:24
*** fawadkhaliq has quit IRC17:25
*** fawadkhaliq has joined #openstack-operators17:25
*** ctrath has quit IRC17:28
*** matrohon has quit IRC17:31
*** ctrath has joined #openstack-operators17:34
*** SimonChung1 has quit IRC17:36
*** SimonChung has joined #openstack-operators17:36
*** pcaruana has joined #openstack-operators17:36
*** simon-AS5591 has quit IRC17:37
*** lhcheng has joined #openstack-operators17:39
krobzaurYeah. I will lend them some credit and say that openstack does require a fair bit of expertise and a broad skill set to deploy, manage, and monitor effectively. But the platform itself is absolutely mind-blowing in its current capability and that doesn't even account for the untapped potential of the newer projects out there17:41
krobzaurA lot of the guys I talk to that are anti-openstack are pretty hardcore VMware guys (also a great product), which seems a bit easier to manage from an ops perspective17:42
*** mriedem1 has joined #openstack-operators17:43
*** mriedem has quit IRC17:44
krobzaurklindgren: I'll definitely have to look into that. I didn't find anything that really stood out to me during some casual research but perhaps I'll have to dig a little deeper17:45
*** mriedem1 has quit IRC17:46
*** ducttape_ has quit IRC17:49
*** ducttape_ has joined #openstack-operators17:49
*** derekh has quit IRC17:49
*** mriedem has joined #openstack-operators17:53
haukebrunokrobzaur, lol. I also saw that vmware guys17:53
haukebrunoluckily my vmware guy is highly interested in openstack too + play around with VIO and stuff17:53
*** vsedelnik has quit IRC17:54
*** kazuIchikawa has quit IRC17:57
*** ducttape_ has quit IRC18:02
krobzaurI wish the VMware pro at my company wasn't so anti openstack because I think we could play around with a lot of really cool integrations between the two cloud platforms18:02
krobzaurI know its possible to use esxi as the hypervisor in openstack, that might be interesting to try18:03
krobzaurVMware also has some excellent automation tools that handle hybrid cloud environments18:03
*** dims_ has joined #openstack-operators18:03
*** dims has quit IRC18:03
wasmumklindgren: caching tokens - how long.  Seems like one neutron node will auth for a call, the second neutron node will auth for a call - new tokens for each, then the following calls fail.  Say `openstack server create` one portion of that network call works with original token, but continuing on down the chain, the next network api call gets auth with the other ks server in the lb pool = new token.  I have a18:06
wasmum suspicion that these back and forths are breaking it.18:06
klindgreniirc we cache for 12 hours18:06
haukebruno1 hour here18:07
*** bsilverman has joined #openstack-operators18:08
klindgrenso in my xp trying to trace timeouts and the like, neutron should use whatever token was passed into it for all further calls related to the original request.  Unless you are talking about nova -> neutron interactions for call backs and stuff?18:08
klindgrenalso new tokens should not invalidate old tokens18:09
mgagneusing a common memcache server for keystonemiddleware?18:13
*** flwang has quit IRC18:16
klindgrenIIRC we aren't specifically configuring anything in the paste files.  I believe it will pull from [DEFAULT}/memcache_servers18:16
mgagneklindgren: [keystone_authtoken]/memcached_servers does the job18:17
klindgrenthen nopes18:18
mgagneand works fine for us. no more request to KS for API calls proxied from Nova to Neutron.18:18
klindgrenand I assume glance as well?18:18
klindgrenthat will reduce load on keystone a bit18:18
mgagneyes, glance too18:18
mgagnein fact, we are looking into disabling catalog retrieval across the board. it's overloading our firewall18:19
mgagne[keystone_authtoken]/include_service_catalog = False18:19
mgagnewe added this config to Ceilometer and it dropped ~20% of trafic going to keystone across all our regions.18:20
mgagnewe have system constantly poking Ceilometer and this single config helped a lot18:20
*** haukebruno has quit IRC18:21
wasmummgagne: using mysql backend (galera cluster)18:22
*** ducttape_ has joined #openstack-operators18:22
mgagnewasmum: not yet, will be 2nd part of our fernet token migration18:22
mgagnewasmum: time being the lacking resource atm =)18:23
wasmumunderstand18:23
mgagnebut yes, we are looking into it18:23
mgagneI'm sure it will greatly reduce our trafic across regions18:23
mgagneplan is to have galera nodes for keystone in each region18:24
wasmumI like the disabled catalog, good find18:24
wasmumthats a huge increase in performance18:24
mgagneyea, really I don't know why it's not disabled by default18:24
mgagneI don't worry much about disabling it for Ceilometer, Neutron and Glance since AFAIK they don't contact other services18:25
mgagnenot sure the impact if I disable it for Nova, Heat or Cinder.18:25
mgagnebecause service could rely on catalog being available in auth context for some operations18:26
wasmumbe nice to see that flow chart somewhere on the interwez18:27
klindgrenmgagne, under your cells config, do you run a large memcache setup for all the cells or do you run a memcache setup per cell?18:27
*** piet has quit IRC18:28
mgagneklindgren: one memcache per region18:28
mgagneklindgren: but problem is: if memcache is down, API doesn't work anymore18:28
mgagneklindgren: cache code looks to not like unreachable memcache18:28
klindgrenwe are api cell (also handles glance/nova-api/neutron/keystone) which has a memcache and nova-cell also has its own memcache18:28
*** piet has joined #openstack-operators18:28
klindgrenre: authtokens or in general?18:28
mgagneauthtokens18:29
*** dims_ has quit IRC18:29
mgagnewe don't have memcache for the rest, only authtoken18:29
*** flwang has joined #openstack-operators18:29
klindgrenunsing memache to handle metadata lookups under nova-metadata helped reduce conductor workload a bit18:29
*** SimonChung has quit IRC18:31
*** SimonChung1 has joined #openstack-operators18:31
klindgrensince metadata lookups from services (puppet) are typically ~8 calls in a few seconds.  It means that we fetch metadata once fro mthe db/conductor vs's 8 times18:31
*** eil397 has left #openstack-operators18:34
*** simon-AS559 has joined #openstack-operators18:37
*** admin0 has quit IRC18:39
*** simon-AS5591 has joined #openstack-operators18:41
*** simon-AS559 has quit IRC18:44
*** electrofelix has quit IRC18:53
*** admin0 has joined #openstack-operators18:56
mgagneklindgren: we don't have metadata service yet =)19:04
*** rarcea has joined #openstack-operators19:05
*** dminer has quit IRC19:09
*** dims has joined #openstack-operators19:12
*** ctrath has quit IRC19:13
*** krotscheck is now known as kro_death_by_sin19:14
*** kro_death_by_sin is now known as kro_sinus_hell19:14
*** ctrath has joined #openstack-operators19:15
*** rarcea has quit IRC19:17
*** dims has quit IRC19:18
klindgrenso under the api services you also disabled the metadata api? or was that more of a you dont run a seperate nova-metadata service?19:21
*** rarcea has joined #openstack-operators19:23
*** dims has joined #openstack-operators19:26
*** vsedelnik has joined #openstack-operators19:27
*** harshs has quit IRC19:32
*** fawadkhaliq has quit IRC19:35
*** ctina has joined #openstack-operators19:36
*** busterswt has joined #openstack-operators19:42
*** dims has quit IRC19:45
*** vsedelnik has quit IRC19:46
*** vsedelnik has joined #openstack-operators19:50
*** amitry has quit IRC19:53
*** amitry has joined #openstack-operators19:54
*** dminer has joined #openstack-operators19:54
*** rockyg has joined #openstack-operators19:58
*** kazuIchikawa has joined #openstack-operators19:58
*** kazuIchikawa has quit IRC20:03
*** ig0r_ has joined #openstack-operators20:04
*** delattec has quit IRC20:05
*** ig0r__ has quit IRC20:06
*** dims has joined #openstack-operators20:11
*** rarcea has quit IRC20:12
*** esker has joined #openstack-operators20:14
*** rarcea has joined #openstack-operators20:16
*** Marga_ has quit IRC20:19
*** ctina has quit IRC20:20
*** VW has quit IRC20:22
*** matrohon has joined #openstack-operators20:23
*** piet has quit IRC20:30
*** piet has joined #openstack-operators20:30
*** regXboi has quit IRC20:34
*** cdelatte has joined #openstack-operators20:35
*** maishsk has joined #openstack-operators20:36
*** rockyg has quit IRC20:37
*** esker has quit IRC20:38
*** rockyg has joined #openstack-operators20:38
*** esker has joined #openstack-operators20:38
*** fawadkhaliq has joined #openstack-operators20:42
*** VW has joined #openstack-operators20:43
*** VW has quit IRC20:48
*** VW has joined #openstack-operators20:50
*** harshs has joined #openstack-operators20:55
*** f1ller is now known as filler21:03
*** maishsk has quit IRC21:15
*** maishsk_ has joined #openstack-operators21:15
*** Guest93664 is now known as med_21:17
*** med_ has joined #openstack-operators21:17
*** esker has quit IRC21:20
*** esker has joined #openstack-operators21:20
*** vsedelnik has quit IRC21:25
*** esker has quit IRC21:26
*** busterswt has quit IRC21:30
*** kstev has joined #openstack-operators21:31
*** kstev1 has quit IRC21:32
*** pcaruana has quit IRC21:38
*** cdelatte has quit IRC21:44
*** maishsk_ has quit IRC21:45
*** cdelatte has joined #openstack-operators21:45
*** matrohon has quit IRC21:47
*** VW has quit IRC21:48
*** fawadkhaliq has quit IRC21:51
*** fawadkhaliq has joined #openstack-operators21:51
*** pcaruana has joined #openstack-operators21:52
*** krobzaur has quit IRC21:53
*** mriedem has quit IRC21:55
*** Marga_ has joined #openstack-operators21:56
*** ctrath has quit IRC21:57
*** kstev has quit IRC22:00
*** kazuIchikawa has joined #openstack-operators22:00
*** fawadkhaliq has quit IRC22:00
*** fawadkhaliq has joined #openstack-operators22:01
*** harlowja has joined #openstack-operators22:01
*** kstev has joined #openstack-operators22:03
*** kazuIchikawa has quit IRC22:05
*** rockyg has quit IRC22:10
*** VW has joined #openstack-operators22:11
*** dims_ has joined #openstack-operators22:11
*** david-lyle has quit IRC22:13
*** dims has quit IRC22:14
*** VW has quit IRC22:15
*** piet has quit IRC22:16
*** dims has joined #openstack-operators22:16
*** dims_ has quit IRC22:17
*** ducttape_ has quit IRC22:20
*** vsedelnik has joined #openstack-operators22:26
*** piet has joined #openstack-operators22:29
*** vsedelnik has quit IRC22:30
*** shakamunyi has joined #openstack-operators22:32
*** mdorman has quit IRC22:53
*** VW has joined #openstack-operators22:56
*** saneax_AFK is now known as saneax22:56
*** dminer has quit IRC22:56
*** piet has quit IRC22:56
*** VW has quit IRC23:04
*** dims has quit IRC23:06
*** saneax is now known as saneax_AFK23:08
*** VW has joined #openstack-operators23:09
*** admin0 has quit IRC23:09
*** rarcea has quit IRC23:25
*** vsedelnik has joined #openstack-operators23:27
*** chlong has joined #openstack-operators23:28
*** VW has quit IRC23:32
*** vsedelnik has quit IRC23:33
*** unicell has joined #openstack-operators23:46
*** harlowja has quit IRC23:47
*** kstev has quit IRC23:47

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!