| *** ducttape_ has quit IRC | 00:10 | |
| *** Apoorva has quit IRC | 00:13 | |
| *** mrhillsman is now known as mrhillsman_ | 00:24 | |
| *** mrhillsman_ is now known as mrhillsman_Away | 00:24 | |
| *** aprice has joined #openstack-operators | 00:33 | |
| *** aprice has quit IRC | 00:39 | |
| *** ducttape_ has joined #openstack-operators | 01:11 | |
| *** fragatin_ has quit IRC | 01:12 | |
| *** fragatina has joined #openstack-operators | 01:12 | |
| *** ducttape_ has quit IRC | 01:16 | |
| *** Apoorva has joined #openstack-operators | 01:19 | |
| *** ducttape_ has joined #openstack-operators | 01:26 | |
| *** ducttape_ has quit IRC | 01:28 | |
| *** vinsh has joined #openstack-operators | 01:33 | |
| *** mrhillsman_Away is now known as mrhillsman | 01:50 | |
| *** ducttape_ has joined #openstack-operators | 02:28 | |
| *** Apoorva has quit IRC | 02:29 | |
| *** ducttape_ has quit IRC | 02:33 | |
| *** ducttape_ has joined #openstack-operators | 02:38 | |
| *** ducttape_ has quit IRC | 03:04 | |
| *** britthouser has joined #openstack-operators | 03:20 | |
| *** britthouser has quit IRC | 03:25 | |
| *** julim has quit IRC | 03:30 | |
| *** armax has quit IRC | 03:48 | |
| *** jlk has quit IRC | 04:04 | |
| *** jlk has joined #openstack-operators | 04:04 | |
| *** jlk has quit IRC | 04:04 | |
| *** jlk has joined #openstack-operators | 04:04 | |
| *** permalac has quit IRC | 04:25 | |
| *** permalac has joined #openstack-operators | 04:26 | |
| *** ducttape_ has joined #openstack-operators | 04:35 | |
| *** fragatin_ has joined #openstack-operators | 04:36 | |
| *** fragatina has quit IRC | 04:39 | |
| *** ducttape_ has quit IRC | 04:40 | |
| *** fragatin_ has quit IRC | 04:40 | |
| *** jamesdenton has quit IRC | 05:01 | |
| *** pcaruana has quit IRC | 05:01 | |
| *** emccormickva has joined #openstack-operators | 05:01 | |
| *** emccormick has quit IRC | 05:04 | |
| *** Trident has quit IRC | 05:25 | |
| *** negronjl_ has quit IRC | 05:32 | |
| *** ducttape_ has joined #openstack-operators | 05:47 | |
| *** VW has joined #openstack-operators | 05:47 | |
| *** negronjl has joined #openstack-operators | 05:51 | |
| *** VW has quit IRC | 05:51 | |
| *** ducttape_ has quit IRC | 05:52 | |
| *** rarcea_ has joined #openstack-operators | 05:59 | |
| *** rcernin has joined #openstack-operators | 06:04 | |
| *** jsheeren has joined #openstack-operators | 06:06 | |
| *** saneax_AFK is now known as saneax | 06:19 | |
| *** saneax is now known as Guest38498 | 06:19 | |
| *** maishsk has joined #openstack-operators | 06:24 | |
| *** maishsk has quit IRC | 06:32 | |
| *** Hosam has joined #openstack-operators | 06:32 | |
| *** maishsk has joined #openstack-operators | 06:34 | |
| *** fragatina has joined #openstack-operators | 06:37 | |
| *** rarcea_ has quit IRC | 06:40 | |
| *** fragatina has quit IRC | 06:42 | |
| *** tesseract- has joined #openstack-operators | 06:44 | |
| *** ducttape_ has joined #openstack-operators | 06:48 | |
| *** zhangjn has quit IRC | 06:49 | |
| *** zhangjn has joined #openstack-operators | 06:52 | |
| *** ducttape_ has quit IRC | 06:53 | |
| *** Hosam_ has joined #openstack-operators | 06:55 | |
| *** Hosam has quit IRC | 06:56 | |
| *** liverpooler has joined #openstack-operators | 06:56 | |
| *** simon-AS559 has joined #openstack-operators | 07:04 | |
| *** crinkle has quit IRC | 07:19 | |
| *** crinkle has joined #openstack-operators | 07:21 | |
| *** pcaruana has joined #openstack-operators | 07:24 | |
| *** maishsk has quit IRC | 07:25 | |
| *** Guest38498 is now known as saneax_AFK | 07:27 | |
| *** Hosam has joined #openstack-operators | 07:32 | |
| *** matrohon has joined #openstack-operators | 07:33 | |
| *** Hosam_ has quit IRC | 07:36 | |
| *** maishsk has joined #openstack-operators | 07:37 | |
| *** matrohon has quit IRC | 07:41 | |
| *** ducttape_ has joined #openstack-operators | 07:49 | |
| *** maishsk has quit IRC | 07:50 | |
| *** maishsk has joined #openstack-operators | 07:51 | |
| *** Trident has joined #openstack-operators | 07:52 | |
| *** ducttape_ has quit IRC | 07:54 | |
| *** Trident has quit IRC | 07:54 | |
| *** Trident has joined #openstack-operators | 07:55 | |
| *** matrohon has joined #openstack-operators | 07:57 | |
| *** simonmcc has quit IRC | 08:08 | |
| *** simonmcc has joined #openstack-operators | 08:09 | |
| *** jsheeren has quit IRC | 08:10 | |
| *** jsheeren has joined #openstack-operators | 08:10 | |
| *** Trident has quit IRC | 08:11 | |
| *** matrohon has quit IRC | 08:15 | |
| *** furlongm has joined #openstack-operators | 08:18 | |
| *** openstackgerrit has quit IRC | 08:18 | |
| *** openstackgerrit has joined #openstack-operators | 08:18 | |
| *** Trident has joined #openstack-operators | 08:23 | |
| *** saneax_AFK is now known as saneax | 08:27 | |
| *** saneax is now known as Guest25838 | 08:28 | |
| *** matrohon has joined #openstack-operators | 08:31 | |
| *** electrofelix has joined #openstack-operators | 08:36 | |
| *** dfflanders has quit IRC | 08:38 | |
| *** permalac has quit IRC | 08:40 | |
| *** ducttape_ has joined #openstack-operators | 08:50 | |
| *** admin0 has joined #openstack-operators | 08:54 | |
| *** ducttape_ has quit IRC | 08:55 | |
| *** matrohon has quit IRC | 09:22 | |
| *** paramite has joined #openstack-operators | 09:28 | |
| *** matrohon has joined #openstack-operators | 09:38 | |
| *** admin0 has quit IRC | 09:46 | |
| *** Trident has quit IRC | 09:46 | |
| *** Trident has joined #openstack-operators | 09:47 | |
| *** admin0 has joined #openstack-operators | 09:50 | |
| *** admin0 has quit IRC | 09:50 | |
| *** ducttape_ has joined #openstack-operators | 09:50 | |
| *** ducttape_ has quit IRC | 09:56 | |
| *** Hosam_ has joined #openstack-operators | 10:04 | |
| *** matrohon has quit IRC | 10:04 | |
| *** Hosam has quit IRC | 10:07 | |
| *** matrohon has joined #openstack-operators | 10:20 | |
| *** Hosam has joined #openstack-operators | 10:20 | |
| *** snecklifter has joined #openstack-operators | 10:23 | |
| *** Hosam_ has quit IRC | 10:24 | |
| *** matrohon has quit IRC | 10:29 | |
| *** Trident has quit IRC | 10:30 | |
| *** admin0 has joined #openstack-operators | 10:31 | |
| *** admin0 has quit IRC | 10:33 | |
| beddari | hm so what do people use for automation via Python, openstacksdk or e.g novaclient directly? | 10:38 |
|---|---|---|
| beddari | driving the API through Python is kind of new to me, I found openstacksdk through this stackoverflow post http://stackoverflow.com/questions/33698861/openstack-novaclient-python-api-not-working | 10:40 |
| *** matrohon has joined #openstack-operators | 10:45 | |
| beddari | then apparently there is openstack-infra/shade too | 10:47 |
| *** VW has joined #openstack-operators | 10:48 | |
| *** ducttape_ has joined #openstack-operators | 10:51 | |
| beddari | shade seems to make the most sense, actually https://specs.openstack.org/openstack-infra/infra-specs/specs/shade.html | 10:52 |
| *** VW has quit IRC | 10:52 | |
| *** permalac has joined #openstack-operators | 10:53 | |
| *** matrohon has quit IRC | 10:53 | |
| *** cvstealth has joined #openstack-operators | 10:53 | |
| beddari | hmm ;) "We could improve the client libraries or python-openstacksdk. We’ve tried to include richer logic in the client libraries and have been told it’s not what they are for. The python-openstacksdk is still young and we’ve been told it’s not ready for production use yet. We need some of the logic for shade now, so the timescale for getting it done in python-openstacksdk isn’t very | 10:55 |
| beddari | workable." | 10:55 |
| *** cvstealt1 has quit IRC | 10:55 | |
| *** ducttape_ has quit IRC | 10:56 | |
| *** matrohon has joined #openstack-operators | 11:09 | |
| *** dbecker has quit IRC | 11:09 | |
| *** Anticime1 is now known as Anticimex | 11:10 | |
| *** matrohon has quit IRC | 11:12 | |
| *** Hosam has quit IRC | 11:12 | |
| *** cdelatte has joined #openstack-operators | 11:23 | |
| *** dbecker has joined #openstack-operators | 11:24 | |
| *** matrohon has joined #openstack-operators | 11:28 | |
| yankcrime | beddari: shade seems to be getting a lot of attention lately | 11:31 |
| yankcrime | for bigger projects it's probably the one to go for | 11:31 |
| yankcrime | for small scripts and basic tooling we've stuck to using {neutron,nova,keystone}client directly though | 11:31 |
| beddari | yankcrime: we did that too, but then ... things happening, keystone v3 etc, more involved workflows ... | 11:32 |
| beddari | actually we wrapped the os provided client package binaries haha | 11:33 |
| *** matrohon has quit IRC | 11:36 | |
| yankcrime | beddari: yeah, we're probably going to have to rework some of our stuff in the not-too-distant future | 11:36 |
| *** fragatina has joined #openstack-operators | 11:36 | |
| yankcrime | our most complicated api interactions are actually done via ruby though (using fog) | 11:37 |
| beddari | onboarding / provisioning ... this is the code we got that could be made a lot "better" using shade it seems https://github.com/norcams/himlar-dp-prep/blob/master/himlar_dp_prep/dp_provisioner.py | 11:39 |
| *** fragatina has quit IRC | 11:41 | |
| *** dbecker has quit IRC | 11:47 | |
| *** dbecker has joined #openstack-operators | 11:48 | |
| *** dbecker has quit IRC | 11:49 | |
| *** ducttape_ has joined #openstack-operators | 11:52 | |
| *** matrohon has joined #openstack-operators | 11:52 | |
| *** dbecker has joined #openstack-operators | 11:53 | |
| *** ducttape_ has quit IRC | 11:57 | |
| *** maishsk has quit IRC | 11:59 | |
| *** maishsk has joined #openstack-operators | 12:04 | |
| *** admin0 has joined #openstack-operators | 12:04 | |
| *** VW has joined #openstack-operators | 12:06 | |
| *** ducttape_ has joined #openstack-operators | 12:07 | |
| *** Hosam has joined #openstack-operators | 12:07 | |
| *** maishsk has quit IRC | 12:19 | |
| *** snecklifter has quit IRC | 12:19 | |
| *** maishsk has joined #openstack-operators | 12:21 | |
| *** maishsk has quit IRC | 12:21 | |
| *** matrohon has quit IRC | 12:21 | |
| *** VW has quit IRC | 12:28 | |
| *** ducttape_ has quit IRC | 12:31 | |
| *** jsheeren_ has joined #openstack-operators | 12:32 | |
| *** jsheeren has quit IRC | 12:32 | |
| *** matrohon has joined #openstack-operators | 12:38 | |
| *** matrohon has quit IRC | 12:49 | |
| *** VW has joined #openstack-operators | 12:49 | |
| *** VW has quit IRC | 12:56 | |
| *** cdelatte has quit IRC | 13:01 | |
| *** priteau has joined #openstack-operators | 13:02 | |
| *** Hosam_ has joined #openstack-operators | 13:03 | |
| *** matrohon has joined #openstack-operators | 13:05 | |
| *** Hosam has quit IRC | 13:06 | |
| *** dminer has joined #openstack-operators | 13:15 | |
| *** matrohon has quit IRC | 13:17 | |
| *** mriedem has joined #openstack-operators | 13:21 | |
| *** harlowja has quit IRC | 13:23 | |
| *** julim has joined #openstack-operators | 13:24 | |
| *** dbecker has quit IRC | 13:25 | |
| *** Zucan has joined #openstack-operators | 13:26 | |
| *** ducttape_ has joined #openstack-operators | 13:27 | |
| *** ducttape_ has quit IRC | 13:31 | |
| *** matrohon has joined #openstack-operators | 13:34 | |
| *** admin0 has quit IRC | 13:34 | |
| *** jsheeren_ has quit IRC | 13:34 | |
| *** jsheeren has joined #openstack-operators | 13:35 | |
| *** VW has joined #openstack-operators | 13:38 | |
| *** VW_ has joined #openstack-operators | 13:40 | |
| *** matrohon has quit IRC | 13:41 | |
| *** VW_ has quit IRC | 13:41 | |
| *** VW_ has joined #openstack-operators | 13:42 | |
| *** VW has quit IRC | 13:42 | |
| *** jamesdenton has joined #openstack-operators | 13:49 | |
| *** hjanssen-hpe has joined #openstack-operators | 13:51 | |
| *** hj-hpe has joined #openstack-operators | 13:51 | |
| *** admin0 has joined #openstack-operators | 13:56 | |
| *** matrohon has joined #openstack-operators | 13:57 | |
| *** ducttape_ has joined #openstack-operators | 13:59 | |
| *** Hosam_ has quit IRC | 14:02 | |
| *** rlrossit has joined #openstack-operators | 14:05 | |
| *** catintheroof has joined #openstack-operators | 14:09 | |
| mnaser | beddari, fyi, just be careful to make sure you have no leftover resources when you delete projects | 14:11 |
| mnaser | deleting keystone project/domain does not remove assigned resources in nova/etc (this part stinks tbh) | 14:12 |
| *** matrohon has quit IRC | 14:13 | |
| *** hieulq_ has joined #openstack-operators | 14:14 | |
| *** jsheeren has quit IRC | 14:18 | |
| *** jsheeren has joined #openstack-operators | 14:18 | |
| *** zhangjn has quit IRC | 14:27 | |
| *** Hosam has joined #openstack-operators | 14:28 | |
| *** zhangjn has joined #openstack-operators | 14:29 | |
| *** matrohon has joined #openstack-operators | 14:29 | |
| *** Zucan has quit IRC | 14:31 | |
| beddari | mnaser: thanks for the warning! | 14:41 |
| *** aprice has joined #openstack-operators | 14:42 | |
| *** matrohon has quit IRC | 14:44 | |
| *** VW has joined #openstack-operators | 14:46 | |
| *** VW__ has joined #openstack-operators | 14:47 | |
| *** VW__ has quit IRC | 14:48 | |
| *** VW__ has joined #openstack-operators | 14:48 | |
| *** VW__ has quit IRC | 14:49 | |
| *** VW_ has quit IRC | 14:49 | |
| *** VW_ has joined #openstack-operators | 14:49 | |
| *** VW has quit IRC | 14:50 | |
| *** Guest25838 is now known as saneax_AFK | 14:51 | |
| *** aprice has quit IRC | 14:52 | |
| *** VW_ has quit IRC | 14:53 | |
| *** aprice has joined #openstack-operators | 14:54 | |
| *** jsheeren has quit IRC | 14:57 | |
| *** matrohon has joined #openstack-operators | 14:58 | |
| *** aprice has quit IRC | 15:00 | |
| *** hieulq_ has quit IRC | 15:01 | |
| *** d0ugal has quit IRC | 15:01 | |
| *** hieulq_ has joined #openstack-operators | 15:02 | |
| *** VW has joined #openstack-operators | 15:02 | |
| *** aprice has joined #openstack-operators | 15:06 | |
| *** david-lyle has quit IRC | 15:12 | |
| *** david-lyle has joined #openstack-operators | 15:12 | |
| *** armax has joined #openstack-operators | 15:13 | |
| *** fragatina has joined #openstack-operators | 15:16 | |
| *** fragatina has quit IRC | 15:16 | |
| *** fragatina has joined #openstack-operators | 15:17 | |
| *** VW has quit IRC | 15:18 | |
| *** VW has joined #openstack-operators | 15:18 | |
| rlrossit | Does anyone have pointers to Sensu checks for openstack? I want to do a service up check for Nova, and all I've been able to find is https://github.com/sensu-plugins/sensu-plugins-openstack which isn't very helpful | 15:36 |
| keekz | rlrossit: we're using all custom plugins, and we have a ton of different nova checks. we check to make sure the processes are running (like nova-api, nova-cells, nova-compute), that api works / responds successfully, things like that. then we also check that builds succeed, alert on build failure rates, etc. some of this stuff might be highly specific to your situation | 15:40 |
| keekz | we use nagios though, not sensu, but the ideas will be the same | 15:41 |
| rlrossit | keekz: thanks for the help. We are trying to figure out a way to make sure the services are pulling messages off of rabbit, and we figured a good initial check to be sure it's still listening is to see if it's still labelled as "up" by Nova | 15:44 |
| rlrossit | because that handles both the "is it running" and the "is it connected to rabbit" questions | 15:45 |
| keekz | ah yeah we also check rabbit | 15:47 |
| keekz | one of the rabbit checks we have is to dump queue size in to graphite, then we make sure queue size doesn't grow above some number | 15:48 |
| keekz | when you say nova, that's unclear to me, since nova is like 50 different services | 15:48 |
| keekz | like a nova-compute might be "running" but the config might be bad and it might not even be connected to the right rabbitmq | 15:49 |
| keekz | same for a nova-cells | 15:49 |
| *** matrohon has quit IRC | 15:50 | |
| *** VW has quit IRC | 15:52 | |
| rlrossit | keekz: by service I mean everything that comes back from nova service-list (the service group api) | 15:52 |
| rlrossit | which uses rabbit to report if it's up | 15:52 |
| rlrossit | (as far as I know) | 15:52 |
| *** VW has joined #openstack-operators | 15:53 | |
| rlrossit | because I've had bad configs too that when the service says it is up, but the config is bad, or if it hasn't been able to connect to rabbit, it's listed as down | 15:53 |
| keekz | interesting.. we're not doing anything withs service-list | 15:53 |
| rlrossit | keekz: this makes me think no one has done this before :). I'll see what I can write up and I'll let you know how it works if you're interested | 15:55 |
| keekz | tbh i don't think i've ever used nova service-list before | 15:56 |
| *** piet has joined #openstack-operators | 15:56 | |
| keekz | we have a lot of things and i see a bunch of down stuff... stuff that's probably old/dead/should be removed | 15:56 |
| keekz | even more interesting that service-list doesn't report everything lol | 15:57 |
| keekz | i'm guessing it's hitting some kind of internal limits | 15:58 |
| rlrossit | I'm trying to fake having 200 n-cpu's across 2 nodes with the fake driver, and on multiple occasions I had a bad config and I was wondering why nothing would schedule, that's where I learned service-list tells me if I'm an idiot | 15:58 |
| openfly | faking cores in the linux kernel was pretty easy | 15:58 |
| keekz | ah i had a bad grep with my service-list.. it might actually be returning everything.. | 15:59 |
| openfly | just for 0..whatever replay the cpu in proc/cpuinfo | 15:59 |
| keekz | it's pretty slow if you have more than a few hundred things | 15:59 |
| rlrossit | well I'm using some beefy hosts, so I haven't hit issues yet :) | 16:00 |
| rlrossit | well, with n-cpu at least... | 16:00 |
| *** rcernin has quit IRC | 16:01 | |
| *** shamail has joined #openstack-operators | 16:02 | |
| *** shamail has quit IRC | 16:07 | |
| *** britthouser has joined #openstack-operators | 16:08 | |
| *** hieulq_ has quit IRC | 16:19 | |
| *** Apoorva has joined #openstack-operators | 16:22 | |
| simon-AS559 | The USENIX Security conference will start in a few hours. The first technical presentation will be on "Flip Feng Shui" - https://www.vusec.net/projects/flip-feng-shui/ | 16:25 |
| simon-AS559 | Have people here heard about this? | 16:25 |
| simon-AS559 | The TL;DR is that if you have "memory deduplication" enabled on your hypervisor hosts, a malicious cloud user may be able to change memory contents in another user's VM running on the same hypervisor host. | 16:26 |
| simon-AS559 | The method is for the attacker to obtain (in their VM) a memory page that is identical to one in a victim's VM (which must run on the same hypervisor) | 16:27 |
| simon-AS559 | Then the attacker can assume that (because of memory deduplication) the two pages use the same physical memory (DRAM)... | 16:27 |
| simon-AS559 | …and use the "Rowhammer" attack to change bits in that memory using excessive READING | 16:28 |
| simon-AS559 | (actual writes would cause the page to be separated into its own copy) | 16:28 |
| *** pcaruana has quit IRC | 16:29 | |
| simon-AS559 | An easy way for a cloud provider to prevent this is to not do memory deduplication… so we're thinking about deactivating KSM on our (Linux/QEMU/KVM-based) hypervisor hosts. | 16:29 |
| simon-AS559 | The question is how to do that efficiently on a fleet of hypervisors. | 16:29 |
| simon-AS559 | KSM is active by default on modern Linux kernels, at least in the distributions that I checked (Ubuntu 14.04 and 16.04). | 16:30 |
| simon-AS559 | QEMU/KVM has a configuration option that allows you to override the system default (KSM_ENABLED). | 16:31 |
| simon-AS559 | Run-time status and stats about KSM can be found in /sys/kernel/mm/ksm/ | 16:31 |
| simon-AS559 | If /sys/kernel/mm/ksm/run is 1 (or non-zero), you'll have a daemon that scans pages in memory and tries to deduplicate them. | 16:32 |
| *** admin0 has quit IRC | 16:32 | |
| simon-AS559 | So you can disable KSM by setting that to 0 (echo 0 | sudo tee /sys/kernel/mm/ksm/run). | 16:33 |
| simon-AS559 | Then no new pages will be merged, but currently merged pages will stay merged. | 16:33 |
| simon-AS559 | Well… hopefully they will still be unmerged automatically when one of the users WRITES to the page. | 16:34 |
| simon-AS559 | What worries me a bit is that the statistics (e.g. /sys/kernel/mm/ksm/pages_shared) don't change anymore once you stop the background task (run -> 0). | 16:34 |
| simon-AS559 | But hopefully that's just because updating of the statistics stops… can anyone confirm? | 16:35 |
| *** paramite has quit IRC | 16:57 | |
| *** aprice has quit IRC | 16:58 | |
| *** simon-AS559 has quit IRC | 17:10 | |
| *** aprice has joined #openstack-operators | 17:16 | |
| *** mrhillsman is now known as mrhillsman_away | 17:22 | |
| *** harlowja has joined #openstack-operators | 17:27 | |
| *** VW has quit IRC | 17:29 | |
| *** ducttape_ has quit IRC | 17:30 | |
| *** aprice has quit IRC | 17:38 | |
| *** priteau has quit IRC | 17:41 | |
| *** Apoorva has quit IRC | 17:44 | |
| *** Apoorva has joined #openstack-operators | 17:45 | |
| *** priteau has joined #openstack-operators | 17:45 | |
| *** ducttape_ has joined #openstack-operators | 17:48 | |
| *** electrofelix has quit IRC | 17:49 | |
| *** dbecker has joined #openstack-operators | 17:49 | |
| *** simon-AS559 has joined #openstack-operators | 17:55 | |
| *** admin0 has joined #openstack-operators | 18:00 | |
| *** Apoorva_ has joined #openstack-operators | 18:00 | |
| *** VW has joined #openstack-operators | 18:02 | |
| *** Apoorva_ has quit IRC | 18:02 | |
| *** Apoorva_ has joined #openstack-operators | 18:02 | |
| *** Apoorva has quit IRC | 18:04 | |
| *** elo has joined #openstack-operators | 18:05 | |
| *** ducttape_ has quit IRC | 18:07 | |
| *** fragatina has quit IRC | 18:07 | |
| *** ducttape_ has joined #openstack-operators | 18:07 | |
| *** tesseract- has quit IRC | 18:13 | |
| *** Trident has joined #openstack-operators | 18:14 | |
| *** elo has quit IRC | 18:14 | |
| *** aprice has joined #openstack-operators | 18:19 | |
| *** elo has joined #openstack-operators | 18:22 | |
| *** dbecker has quit IRC | 18:26 | |
| *** dbecker has joined #openstack-operators | 18:28 | |
| *** aprice has quit IRC | 18:28 | |
| *** aprice has joined #openstack-operators | 18:28 | |
| *** VW has quit IRC | 18:31 | |
| *** Apoorva_ has quit IRC | 18:33 | |
| *** catintheroof has quit IRC | 18:34 | |
| *** Apoorva has joined #openstack-operators | 18:34 | |
| *** VW has joined #openstack-operators | 18:37 | |
| *** ducttape_ has quit IRC | 18:38 | |
| *** fragatina has joined #openstack-operators | 18:44 | |
| *** dbecker has quit IRC | 18:44 | |
| *** zul has quit IRC | 18:44 | |
| *** ducttape_ has joined #openstack-operators | 18:49 | |
| *** zul has joined #openstack-operators | 18:49 | |
| *** stanchan has joined #openstack-operators | 18:51 | |
| *** liverpooler has quit IRC | 18:52 | |
| *** fifieldt has quit IRC | 19:02 | |
| *** zul has quit IRC | 19:06 | |
| *** stanchan has quit IRC | 19:08 | |
| *** stanchan has joined #openstack-operators | 19:08 | |
| *** Trident has quit IRC | 19:14 | |
| *** fifieldt has joined #openstack-operators | 19:18 | |
| *** stanchan has quit IRC | 19:24 | |
| *** elo has quit IRC | 19:24 | |
| *** jsheeren has joined #openstack-operators | 19:26 | |
| *** jsheeren has quit IRC | 19:27 | |
| *** zul has joined #openstack-operators | 19:30 | |
| *** piet has quit IRC | 19:35 | |
| *** mrhillsman_away is now known as mrhillsman | 19:43 | |
| *** stanchan has joined #openstack-operators | 19:46 | |
| *** stanchan has quit IRC | 19:47 | |
| *** dminer has quit IRC | 19:49 | |
| *** Zucan has joined #openstack-operators | 19:50 | |
| *** d0ugal has joined #openstack-operators | 20:02 | |
| *** simon-AS5591 has joined #openstack-operators | 20:04 | |
| *** simon-AS559 has quit IRC | 20:05 | |
| *** elo has joined #openstack-operators | 20:16 | |
| *** VW has quit IRC | 20:25 | |
| *** julim has quit IRC | 20:30 | |
| *** elo has quit IRC | 20:33 | |
| *** elo has joined #openstack-operators | 20:36 | |
| *** elo has quit IRC | 20:40 | |
| *** aprice has quit IRC | 20:43 | |
| *** elo has joined #openstack-operators | 20:44 | |
| *** Zucan has quit IRC | 20:44 | |
| *** elo has quit IRC | 20:55 | |
| *** aprice has joined #openstack-operators | 20:56 | |
| *** VW has joined #openstack-operators | 21:02 | |
| *** Apoorva_ has joined #openstack-operators | 21:04 | |
| *** stanchan has joined #openstack-operators | 21:05 | |
| *** elo has joined #openstack-operators | 21:05 | |
| *** Apoorva has quit IRC | 21:07 | |
| *** fragatin_ has joined #openstack-operators | 21:08 | |
| mnaser | simon-AS5591, echo 2 | sudo tee /sys/kernel/mm/ksm/run will flush out merged pages afaik :) | 21:11 |
| mnaser | we've had ksm off for a while | 21:11 |
| mnaser | performance issues and honestly we don't oversubscribe memory so what's the point | 21:11 |
| mnaser | and you really shouldn't be oversubscribing memory ;) | 21:11 |
| *** fragatina has quit IRC | 21:12 | |
| *** aprice has quit IRC | 21:12 | |
| *** elo has quit IRC | 21:13 | |
| mnaser | anyways, also in regards to the discussion above, this might be useful to someone .. rlrossit / keekz: sharing is caring right? https://gist.github.com/mnaser/65d3dff4c751bc3e9ace288d9e080cdf | 21:13 |
| mnaser | thats a sensu check to ensure all nova services are running, we typically have that check run on the primary controller | 21:14 |
| keekz | cool, thanks mnaser | 21:14 |
| mnaser | https://gist.github.com/mnaser/5802d7b3cefc47103c3166b7199637d4 | 21:15 |
| mnaser | and that is a nice helper to make api calls to endpoints | 21:15 |
| mnaser | it takes care of getting a token automagically so its useful for small health checks like list volumes, list servers, etc | 21:15 |
| mnaser | i'll get around merging it to osops-monitoring-tools at some point.. | 21:16 |
| *** ducttape_ has quit IRC | 21:18 | |
| *** ducttape_ has joined #openstack-operators | 21:20 | |
| jlk | We moved away from doing API calls and instead just implemented the oslo healthcheck middleware across all the services | 21:30 |
| jlk | and we just hit that instead. | 21:30 |
| *** ducttape_ has quit IRC | 21:36 | |
| *** ducttape_ has joined #openstack-operators | 21:38 | |
| mnaser | i didnt know about the healthcheck api | 21:39 |
| mnaser | thank you so much jlk :> | 21:39 |
| *** rlrossit has quit IRC | 21:41 | |
| *** jamesdenton has quit IRC | 21:44 | |
| *** dbecker has joined #openstack-operators | 21:45 | |
| *** priteau has quit IRC | 21:54 | |
| *** stanchan has quit IRC | 21:56 | |
| dmsimard | KSM has been mostly worry free in my experience -- except with Windows guests. | 21:56 |
| dmsimard | mnaser: curious, why use ruby for your sensu checks ? | 21:58 |
| dmsimard | I mean, it's not like there are python clients or anything :) | 21:58 |
| jlk | inertia I would imagine | 22:04 |
| jlk | sensu being ruby | 22:04 |
| *** stanchan has joined #openstack-operators | 22:05 | |
| *** mriedem has quit IRC | 22:05 | |
| dmsimard | jlk: sensu just runs a check script, though.. it can run nagios checks that are written in perl, bash or whatever. What matters to sensu is just the exit code being 0/1/2 and the check output. | 22:05 |
| jlk | I know, we have a pile of our own | 22:06 |
| jlk | some are ruby, many are python | 22:06 |
| jlk | new ones often come from copying an existing check to a new name | 22:06 |
| dmsimard | It's fine if the answer is "I prefer ruby to python", just curious because the openstack python clients already take care of abstracting all the authentication and json parsing stuff which is re-written in ruby in that snippet | 22:07 |
| dmsimard | Genuinely interested, perhaps there are missing things in the clients and whatnot. | 22:08 |
| *** elo has joined #openstack-operators | 22:09 | |
| mnaser | dmsimard, the reason is from what i understand, the framework for writing sensu checks is largely based around ruby | 22:12 |
| mnaser | so kinda wanted to leverage that first | 22:12 |
| dmsimard | mnaser: yeah, I'm not familiar with writing /sensu/ checks -- but I know it can run anything you punt at it. Like for example we have this bash script https://github.com/rdo-infra/ansible-role-rdomonitoring/blob/master/files/check-http-file-age.sh and the sensu check definition that uses it here: http://paste.openstack.org/raw/553676/ | 22:15 |
| dmsimard | I need to read into that actual check framework :) | 22:16 |
| *** stanchan has quit IRC | 22:17 | |
| *** ducttape_ has quit IRC | 22:19 | |
| *** ducttape_ has joined #openstack-operators | 22:20 | |
| *** stanchan has joined #openstack-operators | 22:20 | |
| *** eric_lopez has joined #openstack-operators | 22:20 | |
| *** elo has quit IRC | 22:21 | |
| *** VW has quit IRC | 22:23 | |
| xavpaice | jlk: is there somewhere I can read more about the oslo healthcheck middleware? All we have it working with is Swift, and the others don't look quite so easy | 22:24 |
| *** ducttape_ has quit IRC | 22:24 | |
| jlk | sure | 22:25 |
| notmyname | xavpaice: what does the oslo healthcheck middleware do? | 22:25 |
| jlk | xavpaice: here's what we did https://github.com/blueboxgroup/ursula/pull/1953/files | 22:25 |
| jlk | notmyname: it short-circuits the application and provides a quick pass/fail on the application. | 22:26 |
| *** fragatin_ has quit IRC | 22:26 | |
| xavpaice | many thanks! | 22:26 |
| jlk | http://docs.openstack.org/developer/oslo.middleware/healthcheck_plugins.html | 22:26 |
| *** fragatina has joined #openstack-operators | 22:27 | |
| jlk | Those are mitaka based paste.inis | 22:28 |
| xavpaice | awesome, will have a play - was going to ask if it's mitaka | 22:28 |
| *** admin0 has quit IRC | 22:39 | |
| xavpaice | jlk: does https://review.openstack.org/#/c/148595/ look familiar? | 22:44 |
| jlk | xavpaice: I think I used that as a starting point | 22:45 |
| xavpaice | ta | 22:45 |
| jlk | I saw glance had done it, and I got curious if I could do it for the rest of the projects | 22:45 |
| *** stanchan has quit IRC | 22:46 | |
| *** stanchan has joined #openstack-operators | 22:47 | |
| xavpaice | I am failing to understand what creates the, e.g., /var/run/nova/healthcheck_disable file | 22:47 |
| jlk | oh, a human would do that | 22:47 |
| jlk | it's a way to turn on/off the middleware without having to edit config files and restart serices | 22:48 |
| *** Hosam has quit IRC | 22:50 | |
| xavpaice | clearly I'm mis-reading https://specs.openstack.org/openstack/oslo-specs/specs/kilo/oslo-middleware-healthcheck.html | 22:52 |
| * jlk looks | 22:52 | |
| jlk | so that was the spec | 22:52 |
| xavpaice | I thought that we were looking at a way for the service itself to report if it's able to function correctly (e.g. database working OK) | 22:52 |
| jlk | oh, no | 22:52 |
| xavpaice | still useful, but in a different way | 22:53 |
| jlk | I thikn what this gives you is "is the API process running enough to push things through the wsgi pipeline" | 22:53 |
| jlk | it's not a functional test of the service per se, it's a "is the API responding" test. | 22:53 |
| xavpaice | yeah, and that relies on some external $thing to check and report the result of the test if we want it to go any further than that | 22:54 |
| xavpaice | guess I was looking for a cheat's way out of maintinaing API status tests for all the APIs | 22:54 |
| jlk | I think it depends on how depe into the 'status' you care to go. | 22:55 |
| xavpaice | :) | 22:55 |
| *** dbecker has quit IRC | 22:56 | |
| xavpaice | dead or alive is a bunch better than checking the port is open | 22:56 |
| jlk | yup, and this gives you a "mostly alive" | 22:57 |
| *** mriedem has joined #openstack-operators | 23:01 | |
| xavpaice | many thanks! | 23:02 |
| *** bryan_att has quit IRC | 23:03 | |
| *** eric_lopez has quit IRC | 23:10 | |
| *** saneax_AFK is now known as saneax | 23:13 | |
| *** saneax is now known as Guest7941 | 23:14 | |
| *** elo has joined #openstack-operators | 23:20 | |
| *** eric_lopez has joined #openstack-operators | 23:25 | |
| *** elo has quit IRC | 23:25 | |
| *** stanchan has quit IRC | 23:27 | |
| *** chlong|mtg has quit IRC | 23:31 | |
| *** stanchan has joined #openstack-operators | 23:42 | |
| *** simon-AS5591 has quit IRC | 23:48 | |
| *** stanchan has quit IRC | 23:53 | |
| *** stanchan has joined #openstack-operators | 23:56 | |
| *** jamesdenton has joined #openstack-operators | 23:58 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!