| *** dminer has quit IRC | 00:06 | |
| *** catintheroof has joined #openstack-operators | 00:11 | |
| *** ducttape_ has joined #openstack-operators | 00:17 | |
| *** ducttape_ has quit IRC | 00:17 | |
| *** ducttape_ has joined #openstack-operators | 00:17 | |
| *** ducttape_ has quit IRC | 00:23 | |
| *** ducttape_ has joined #openstack-operators | 00:33 | |
| *** kstev has quit IRC | 00:51 | |
| *** catintheroof has quit IRC | 00:54 | |
| *** ducttape_ has quit IRC | 00:57 | |
| *** kstev has joined #openstack-operators | 01:03 | |
| *** ducttape_ has joined #openstack-operators | 01:05 | |
| *** kstev has quit IRC | 01:07 | |
| *** chlong has quit IRC | 01:11 | |
| *** kstev has joined #openstack-operators | 01:19 | |
| *** ducttape_ has quit IRC | 02:14 | |
| *** markvoelker has quit IRC | 02:18 | |
| *** kstev has quit IRC | 02:28 | |
| *** kstev has joined #openstack-operators | 02:41 | |
| *** ducttape_ has joined #openstack-operators | 03:01 | |
| *** fragatina has quit IRC | 03:03 | |
| *** fragatina has joined #openstack-operators | 03:05 | |
| *** fragatin_ has joined #openstack-operators | 03:09 | |
| *** fragatina has quit IRC | 03:09 | |
| *** fragatin_ has quit IRC | 03:14 | |
| *** Apoorva has quit IRC | 03:15 | |
| *** markvoelker has joined #openstack-operators | 03:19 | |
| *** mriedem has quit IRC | 03:23 | |
| *** markvoelker has quit IRC | 03:25 | |
| *** ducttape_ has quit IRC | 03:49 | |
| *** ducttape_ has joined #openstack-operators | 03:51 | |
| *** ducttape_ has quit IRC | 03:51 | |
| *** ducttape_ has joined #openstack-operators | 03:52 | |
| *** kstev has quit IRC | 03:54 | |
| *** ducttape_ has quit IRC | 03:56 | |
| *** haplo37_ has quit IRC | 04:00 | |
| *** armax has quit IRC | 04:26 | |
| *** udesale has joined #openstack-operators | 05:18 | |
| *** markvoelker has joined #openstack-operators | 05:20 | |
| *** I has joined #openstack-operators | 05:24 | |
| *** I is now known as Guest66254 | 05:24 | |
| *** markvoelker has quit IRC | 05:25 | |
| *** Guest66254 has quit IRC | 05:29 | |
| *** ducttape_ has joined #openstack-operators | 05:30 | |
| *** ducttape_ has quit IRC | 05:34 | |
| *** twiggy has joined #openstack-operators | 05:36 | |
| *** haplo37 has joined #openstack-operators | 06:04 | |
| *** pilgrimstack has quit IRC | 06:06 | |
| *** pilgrimstack has joined #openstack-operators | 06:06 | |
| *** haplo37 has quit IRC | 06:13 | |
| *** mgagne has quit IRC | 06:18 | |
| *** hughsaunders has quit IRC | 06:20 | |
| *** timburke has quit IRC | 06:20 | |
| *** mgagne has joined #openstack-operators | 06:21 | |
| *** mgagne is now known as Guest52285 | 06:21 | |
| *** timburke has joined #openstack-operators | 06:23 | |
| *** hughsaunders has joined #openstack-operators | 06:23 | |
| *** haplo37 has joined #openstack-operators | 06:26 | |
| *** tesseract has joined #openstack-operators | 07:18 | |
| *** tesseract is now known as Guest90313 | 07:18 | |
| *** Miouge has joined #openstack-operators | 07:20 | |
| *** simon-AS5591 has quit IRC | 07:20 | |
| *** jsheeren has joined #openstack-operators | 07:44 | |
| *** pcaruana has joined #openstack-operators | 07:45 | |
| *** lukl has quit IRC | 07:49 | |
| *** belmoreira has joined #openstack-operators | 07:51 | |
| *** simon-AS559 has joined #openstack-operators | 07:52 | |
| *** sticker_ has joined #openstack-operators | 07:57 | |
| *** sticker has quit IRC | 08:00 | |
| *** openstackgerrit has quit IRC | 08:03 | |
| *** openstackgerrit has joined #openstack-operators | 08:03 | |
| *** matrohon has joined #openstack-operators | 08:11 | |
| *** belmoreira has quit IRC | 09:03 | |
| *** zz9pzza has left #openstack-operators | 09:04 | |
| *** paramite has joined #openstack-operators | 09:07 | |
| *** rmart04 has joined #openstack-operators | 09:12 | |
| *** derekh has joined #openstack-operators | 09:14 | |
| yankcrime | ah, this is interesting - we're currently limiting at ~260k | 09:31 |
|---|---|---|
| yankcrime | klindgren__: care to share your other conntrack-related tunables? | 09:31 |
| yankcrime | if you haven't already? | 09:31 |
| yankcrime | med_: we've had issues where shady (former) customers have been doing stuff like sending out spam | 09:32 |
| yankcrime | we'd see behaviour like 1+ million DNS requests and then hundreds of thousands of outbound tcp connections to port 25 | 09:33 |
| yankcrime | anyway, because of that i'm reluctant to whack the limit right up - it's usually a symptom of someone Doing It Wrong ™ (i.e hacked) or generally up to no good, so early sight of that is useful | 09:35 |
| *** derekjhyang has quit IRC | 09:51 | |
| *** AlexeyAbashkin has joined #openstack-operators | 09:53 | |
| *** AlexeyAbashkin has quit IRC | 10:16 | |
| *** belmoreira has joined #openstack-operators | 10:17 | |
| *** electrofelix has joined #openstack-operators | 10:33 | |
| *** fragatina has joined #openstack-operators | 10:35 | |
| *** fragatina has quit IRC | 10:36 | |
| *** fragatina has joined #openstack-operators | 10:37 | |
| *** simon-AS559 has quit IRC | 10:41 | |
| *** udesale has quit IRC | 10:50 | |
| *** chaology has quit IRC | 11:31 | |
| *** chaology has joined #openstack-operators | 11:33 | |
| *** ducttape_ has joined #openstack-operators | 11:35 | |
| *** chaology has quit IRC | 11:36 | |
| *** chaology has joined #openstack-operators | 11:37 | |
| *** ducttape_ has quit IRC | 11:39 | |
| *** chaology has quit IRC | 11:45 | |
| *** chaology has joined #openstack-operators | 11:45 | |
| *** mjrichardson has quit IRC | 11:47 | |
| *** mjrichardson has joined #openstack-operators | 11:47 | |
| *** derekjhyang has joined #openstack-operators | 12:21 | |
| logan- | yankcrime: yep @ tcp/25. we do hashlimit policing on tcp/25 per /32 and also 2mil conntrack_max. +1 interested in seeing what other conntrack/net tunables folks have implemented. I can think of a few setup/teardown ones I'd like to get added that we use to harden other infra (ie. ddos scrubbing boxes) | 12:29 |
| yankcrime | thanks logan- - good info | 12:29 |
| *** ducttape_ has joined #openstack-operators | 12:36 | |
| *** vijaykc4 has joined #openstack-operators | 12:38 | |
| *** vijaykc4 has quit IRC | 12:40 | |
| *** ducttape_ has quit IRC | 12:40 | |
| *** simon-AS559 has joined #openstack-operators | 13:01 | |
| *** stupidnic has quit IRC | 13:02 | |
| *** stupidnic has joined #openstack-operators | 13:02 | |
| *** ducttape_ has joined #openstack-operators | 13:08 | |
| *** simon-AS559 has quit IRC | 13:13 | |
| mnaser | yankcrime do you use provider networks for public internet (directly connected) or via nats? | 13:15 |
| mnaser | logan- thats interesting, we never thought about doing something like that, do you implement it on the compute/hv side? | 13:17 |
| logan- | yes. my env is calico so internet traffic is routed straight to/from the computes. the hashlimit runs on the compute iptables and polices down outbound syn on tcp/25 to some threshold per /32. | 13:18 |
| logan- | then logs exceeded hits, aggregate and counts them, and posts to #abuse on our slack periodically :P | 13:20 |
| mnaser | logan- thats pretty badass | 13:22 |
| mnaser | since looking at calico i've really wanted to try it but.. | 13:22 |
| mnaser | there's no way we're going to even think about migrating our env to that | 13:22 |
| yankcrime | mnaser: the latter | 13:22 |
| mnaser | ah, i think that's a bigger challenge with iptables | 13:23 |
| mnaser | err | 13:23 |
| mnaser | conntrack tables | 13:23 |
| yankcrime | yeah, interesting to hear what people's approaches are though for mitigating this kind of problem | 13:24 |
| *** simon-AS559 has joined #openstack-operators | 13:29 | |
| *** simon-AS559 has quit IRC | 13:39 | |
| *** mriedem has joined #openstack-operators | 13:40 | |
| *** ducttape_ has quit IRC | 13:48 | |
| *** dminer has joined #openstack-operators | 13:48 | |
| *** markvoelker has joined #openstack-operators | 13:49 | |
| *** baffle has quit IRC | 13:51 | |
| *** baffle has joined #openstack-operators | 13:52 | |
| *** ducttape_ has joined #openstack-operators | 14:24 | |
| *** mriedem has quit IRC | 14:30 | |
| *** derekjhyang has quit IRC | 14:31 | |
| *** jsheeren has quit IRC | 14:35 | |
| *** jsheeren has joined #openstack-operators | 14:36 | |
| *** jsheeren has quit IRC | 14:37 | |
| *** jsheeren has joined #openstack-operators | 14:37 | |
| *** jsheeren has quit IRC | 14:39 | |
| *** ducttape_ has quit IRC | 14:40 | |
| *** jsheeren has joined #openstack-operators | 14:41 | |
| *** jsheeren has quit IRC | 14:42 | |
| *** dansmith is now known as superdan | 14:42 | |
| *** jsheeren has joined #openstack-operators | 14:42 | |
| *** jsheeren has quit IRC | 14:42 | |
| *** jsheeren has joined #openstack-operators | 14:43 | |
| *** chlong has joined #openstack-operators | 14:53 | |
| *** simon-AS559 has joined #openstack-operators | 15:02 | |
| *** simon-AS559 has quit IRC | 15:08 | |
| *** mriedem has joined #openstack-operators | 15:12 | |
| *** kstev has joined #openstack-operators | 15:12 | |
| *** ducttape_ has joined #openstack-operators | 15:14 | |
| *** simon-AS559 has joined #openstack-operators | 15:22 | |
| *** dminer has quit IRC | 15:24 | |
| *** slaweq has quit IRC | 15:26 | |
| *** simon-AS559 has quit IRC | 15:28 | |
| *** simon-AS559 has joined #openstack-operators | 15:31 | |
| klindgren__ | we do that in other ways, like we have TC running on all of our public vm's. So that we can ratelimit specific things that we have learned over time through our VPS products | 15:37 |
| klindgren__ | we also setup mgmt ports into the hv's as notrack rules so we can always ssh into the servers, same with RMQ connectivity and our monitoring stuff | 15:38 |
| klindgren__ | we monitor conntrack and have an auto-rememdiation to clear cruft out of the tables if they are full | 15:39 |
| *** simon-AS559 has quit IRC | 15:40 | |
| *** dminer has joined #openstack-operators | 15:41 | |
| *** cheetah has quit IRC | 15:43 | |
| *** pcaruana has quit IRC | 15:50 | |
| *** HenryG has quit IRC | 15:50 | |
| *** HenryG has joined #openstack-operators | 15:51 | |
| *** simon-AS559 has joined #openstack-operators | 15:52 | |
| *** armax has joined #openstack-operators | 15:55 | |
| *** kstev has quit IRC | 15:57 | |
| *** belmoreira has quit IRC | 15:57 | |
| *** simon-AS559 has quit IRC | 16:01 | |
| *** jamesdenton has joined #openstack-operators | 16:02 | |
| logan- | thats good stuff re: the notrack klindgren__ | 16:02 |
| *** klindgren__ is now known as klindgren | 16:02 | |
| *** jsheeren has quit IRC | 16:03 | |
| logan- | i remember the tc stuff you did.. still been wanting to implement some of that fair queueing stuff on our hvs but no time :( | 16:03 |
| klindgren | I think the team that did that was looking at moving it to some other thing so that we dont blow stuff up. | 16:04 |
| klindgren | I should say that we have to create an ifb device per vm | 16:05 |
| klindgren | and we have to have a program come along and nuke the ifb devices on occasion. | 16:05 |
| klindgren | If we get too many ifb devices legacy monitoring systems start to have issues | 16:06 |
| klindgren | like snmp polling finds a few thousand network devices | 16:06 |
| klindgren | or it takes a long time for some other actions to run | 16:06 |
| klindgren | eitherway IIRC the code and the cronjob script our on our github | 16:07 |
| *** kstev has joined #openstack-operators | 16:07 | |
| klindgren | https://github.com/godaddy/openstack-traffic-shaping | 16:08 |
| *** Guest90313 has quit IRC | 16:09 | |
| *** Oku_OS is now known as Oku_OS-away | 16:10 | |
| *** kstev has quit IRC | 16:11 | |
| *** kstev has joined #openstack-operators | 16:22 | |
| *** uxdanielle has joined #openstack-operators | 16:33 | |
| mnaser | klindgren cant you shape traffic directly with a libvirt feature? | 16:35 |
| *** simon-AS559 has joined #openstack-operators | 16:35 | |
| mnaser | let me try to remember the flavor settings | 16:35 |
| *** rmart04 has quit IRC | 16:36 | |
| klindgren | thats not shaping | 16:36 |
| klindgren | thats policing | 16:36 |
| mnaser | ooo | 16:37 |
| mnaser | you're right | 16:37 |
| klindgren | their is a subtle but very important difference between policing and shaping. | 16:37 |
| mnaser | i thought there was extra spec keys for policing too | 16:37 |
| *** cheetah has joined #openstack-operators | 16:39 | |
| *** makowals has quit IRC | 16:49 | |
| *** makowals has joined #openstack-operators | 16:50 | |
| mnaser | while we're on the topic of networks | 16:53 |
| mnaser | how has everyone dealt with large l2 domain if you arent using nat'd setups / more of a provider network setup | 16:53 |
| *** makowals has quit IRC | 16:53 | |
| *** makowals has joined #openstack-operators | 16:54 | |
| *** makowals has quit IRC | 16:58 | |
| *** Miouge has quit IRC | 17:01 | |
| *** Miouge has joined #openstack-operators | 17:02 | |
| *** Miouge has quit IRC | 17:02 | |
| *** pilgrimstack has quit IRC | 17:04 | |
| *** matrohon has quit IRC | 17:13 | |
| *** Miouge has joined #openstack-operators | 17:15 | |
| *** paramite has quit IRC | 17:18 | |
| *** derekh has quit IRC | 17:24 | |
| *** mriedem has quit IRC | 17:29 | |
| Guest52285 | anyone running rundeck, ansible tower or any similar tool so one can trigger a task without having direct access to secrets or production network? | 17:33 |
| *** Guest52285 is now known as mgagne | 17:33 | |
| *** mgagne has quit IRC | 17:33 | |
| *** mgagne has joined #openstack-operators | 17:33 | |
| mgagne | was me ^ | 17:37 |
| *** rmart04 has joined #openstack-operators | 17:43 | |
| logan- | i've looked at rundeck a little bit but seemed like it was going to take quite a bit of work to get that all going | 17:44 |
| logan- | ansible-semaphore is another one i've heard of people using | 17:44 |
| *** rmart04_ has joined #openstack-operators | 17:46 | |
| *** rmart04_ has quit IRC | 17:48 | |
| *** rmart04 has quit IRC | 17:48 | |
| *** Miouge has quit IRC | 18:00 | |
| *** dbecker has quit IRC | 18:04 | |
| *** Apoorva has joined #openstack-operators | 18:09 | |
| *** Miouge has joined #openstack-operators | 18:10 | |
| mgagne | I tried ansible-semaphore once and I found it to be complex to setup and lacking "finishing". clearly not user friendly tbh. | 18:13 |
| mgagne | we are currently using Jenkins to trigger those tasks, not the best tool and it comes (in our case) with a lot of legacy settings | 18:13 |
| *** Miouge has quit IRC | 18:14 | |
| *** Miouge has joined #openstack-operators | 18:16 | |
| *** simon-AS559 has quit IRC | 18:21 | |
| *** slaweq_ has quit IRC | 18:22 | |
| *** slaweq has joined #openstack-operators | 18:24 | |
| *** slaweq has quit IRC | 18:31 | |
| *** uxdanielle has quit IRC | 18:31 | |
| *** slaweq has joined #openstack-operators | 18:33 | |
| mnaser | mgagne what sort of tasks would these be? why not just tap into openstack and implement api extensions if they are service oriented | 18:38 |
| mnaser | thats what we did for a while | 18:38 |
| *** kstev1 has joined #openstack-operators | 18:38 | |
| mgagne | mnaser: ansible playbook and bash scripts. I think I will go with a new Jenkins install without legacy junk and lock it down. | 18:39 |
| mnaser | i wonder if RH got around open sourcing tower | 18:40 |
| mgagne | not yet afaik. we have been waiting for it for months. for me, it's time to move on for now and will revisit later. | 18:40 |
| jlk | soon... | 18:55 |
| *** mriedem has joined #openstack-operators | 18:55 | |
| jlk | but not soon enough | 18:55 |
| *** Miouge has quit IRC | 19:00 | |
| *** Miouge has joined #openstack-operators | 19:01 | |
| mgagne | yea, I don't mind waiting if I know I can refactor a temp solution later | 19:07 |
| *** Miouge has quit IRC | 19:08 | |
| *** Miouge has joined #openstack-operators | 19:09 | |
| *** Miouge has quit IRC | 19:13 | |
| *** Miouge has joined #openstack-operators | 19:22 | |
| jlk | the intent is definitely to opensource it. | 19:25 |
| jlk | at least from what I gather both public and private conversations | 19:25 |
| *** Miouge has quit IRC | 19:27 | |
| *** zul has quit IRC | 19:30 | |
| *** Miouge has joined #openstack-operators | 19:38 | |
| *** Miouge has quit IRC | 19:42 | |
| *** Miouge has joined #openstack-operators | 19:47 | |
| *** zul has joined #openstack-operators | 19:49 | |
| *** cheetah has quit IRC | 19:53 | |
| *** twiggy has quit IRC | 19:58 | |
| *** electrofelix has quit IRC | 20:05 | |
| *** Miouge has quit IRC | 20:19 | |
| *** piet has joined #openstack-operators | 20:19 | |
| *** Miouge has joined #openstack-operators | 20:33 | |
| *** chlong has quit IRC | 20:53 | |
| dmsimard | mgagne: I've recently used git-crypt and was pleasantly surprised about it -- perhaps that with a combination of the ansible no_log parameter ? | 20:55 |
| *** dminer has quit IRC | 20:56 | |
| *** twiggy has joined #openstack-operators | 20:57 | |
| dmsimard | ex: https://github.com/CentOS/centos-cloud/commit/ad7a646a6489bead9d4dcfc48f05b21d596ae3cc && https://github.com/CentOS/centos-cloud/commit/c0ece0c5b58f0780023a138d545268a61e629e34 | 20:57 |
| mgagne | dmsimard: the end user would still have to get network access to the deployed infra (if using ansible) | 20:59 |
| dmsimard | can wrap it inside jenkins, using a gpg deploy key stored as a jenkins credential binding (for example) | 21:00 |
| dmsimard | probably needs some hacking (i.e, passing the gpg key passphrase) but maybe it'd work, I don't know. | 21:02 |
| *** fragatina has quit IRC | 21:11 | |
| *** Miouge has quit IRC | 21:18 | |
| *** Miouge has joined #openstack-operators | 21:23 | |
| *** catintheroof has joined #openstack-operators | 21:31 | |
| *** Miouge has quit IRC | 21:32 | |
| *** fragatina has joined #openstack-operators | 21:41 | |
| catintheroof | Hi ! quick question, talking about how to do ha on openstack with pacemaker, if i have database on host1, host2 & host 3 but i have keystone on host4, host5 & host6, what is the right way on a single cluster of pacemaker, to tell that mysql resource should only happen on db hosts and keystone resource only occurs on keystone servers ? so that i can then define the orders ? i mean, what is the right way because when i clo | 21:41 |
| catintheroof | ne a resource, it happens everywhere and then i have to "ban" hosts for those resources not to happen there. | 21:41 |
| *** twiggy has quit IRC | 21:51 | |
| klindgren | mgagne, I looked at rundeck | 22:05 |
| klindgren | but quickly scrapped it when I found I couldnt do something as simple as pass variables between 2 work flow actions | 22:05 |
| klindgren | We are actively looking at stackstorm | 22:06 |
| *** kstev has quit IRC | 22:08 | |
| *** jamesdenton has quit IRC | 22:26 | |
| *** ckonstanski has quit IRC | 22:38 | |
| *** slaweq has quit IRC | 22:43 | |
| *** ducttape_ has quit IRC | 22:43 | |
| *** ducttape_ has joined #openstack-operators | 23:45 | |
| *** fragatin_ has joined #openstack-operators | 23:50 | |
| *** ducttape_ has quit IRC | 23:53 | |
| *** fragatina has quit IRC | 23:53 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!