| *** Marga_ has joined #openstack-operators | 00:05 | |
| *** Marga_ has quit IRC | 00:05 | |
| *** Marga_ has joined #openstack-operators | 00:06 | |
| *** alop has quit IRC | 00:10 | |
| *** Marga_ has quit IRC | 00:24 | |
| *** Marga_ has joined #openstack-operators | 00:28 | |
| *** blair has joined #openstack-operators | 00:30 | |
| *** VW_ has quit IRC | 00:33 | |
| *** blair has quit IRC | 00:35 | |
| *** zz_avozza is now known as avozza | 00:49 | |
| *** mdorman has quit IRC | 00:51 | |
| *** signed8bit has quit IRC | 00:59 | |
| *** Marga_ has quit IRC | 01:02 | |
| *** dmsimard_away is now known as dmsimard | 01:04 | |
| *** Marga__ has joined #openstack-operators | 01:05 | |
| *** avozza is now known as zz_avozza | 01:11 | |
| *** jaypipes has quit IRC | 01:14 | |
| *** Rockyg has quit IRC | 01:27 | |
| *** SimonChung1 has quit IRC | 01:31 | |
| *** blair has joined #openstack-operators | 01:31 | |
| *** blair has quit IRC | 01:36 | |
| *** Marga__ has quit IRC | 01:37 | |
| *** Marga_ has joined #openstack-operators | 01:41 | |
| *** david-lyle has joined #openstack-operators | 01:43 | |
| *** csoukup has quit IRC | 01:50 | |
| *** dmsimard is now known as dmsimard_away | 01:53 | |
| *** david-lyle has quit IRC | 02:06 | |
| *** david-lyle has joined #openstack-operators | 02:16 | |
| *** david-lyle has quit IRC | 02:19 | |
| *** Rockyg has joined #openstack-operators | 02:21 | |
| *** david-lyle has joined #openstack-operators | 02:23 | |
| *** zz_avozza is now known as avozza | 02:30 | |
| *** blair has joined #openstack-operators | 02:32 | |
| *** Marga_ has quit IRC | 02:33 | |
| *** Marga_ has joined #openstack-operators | 02:33 | |
| *** blair has quit IRC | 02:36 | |
| *** Rockyg has quit IRC | 02:38 | |
| *** Rockyg has joined #openstack-operators | 02:40 | |
| *** avozza is now known as zz_avozza | 02:40 | |
| *** pcaruana has quit IRC | 02:41 | |
| *** david-lyle has quit IRC | 02:42 | |
| *** david-lyle has joined #openstack-operators | 02:43 | |
| *** Rockyg has quit IRC | 02:45 | |
| *** david-lyle has quit IRC | 02:49 | |
| *** david-lyle has joined #openstack-operators | 02:56 | |
| *** Marga_ has quit IRC | 03:15 | |
| *** VW_ has joined #openstack-operators | 03:28 | |
| *** david-lyle has quit IRC | 03:32 | |
| *** blair has joined #openstack-operators | 03:33 | |
| *** blair has quit IRC | 03:37 | |
| *** harlowja is now known as harlowja_away | 03:38 | |
| *** harlowja_away is now known as harlowja | 03:47 | |
| *** ggillies has quit IRC | 04:03 | |
| *** VW_ has quit IRC | 04:04 | |
| *** ggillies has joined #openstack-operators | 04:04 | |
| *** VW_ has joined #openstack-operators | 04:04 | |
| *** reed has joined #openstack-operators | 04:06 | |
| *** ggillies_ has joined #openstack-operators | 04:07 | |
| *** ggillies1 has joined #openstack-operators | 04:08 | |
| *** ggillies_ has quit IRC | 04:08 | |
| *** ggillies has quit IRC | 04:09 | |
| *** ggillies has joined #openstack-operators | 04:09 | |
| *** ggillies_ has joined #openstack-operators | 04:11 | |
| *** ggillies2 has joined #openstack-operators | 04:13 | |
| *** ggillies1 has quit IRC | 04:13 | |
| *** ggillies has quit IRC | 04:14 | |
| *** ggillies_ has quit IRC | 04:16 | |
| *** ggillies has joined #openstack-operators | 04:18 | |
| *** zz_avozza is now known as avozza | 04:19 | |
| *** ggillies2 has quit IRC | 04:21 | |
| *** ggillies_ has joined #openstack-operators | 04:21 | |
| *** ggillies has quit IRC | 04:22 | |
| *** ggillies has joined #openstack-operators | 04:25 | |
| *** ggillies1 has joined #openstack-operators | 04:26 | |
| *** ggillies_ has quit IRC | 04:26 | |
| *** ggillies_ has joined #openstack-operators | 04:28 | |
| *** ggillies2 has joined #openstack-operators | 04:29 | |
| *** avozza is now known as zz_avozza | 04:29 | |
| *** ggillies has quit IRC | 04:30 | |
| *** ggillies1 has quit IRC | 04:31 | |
| *** ggillies has joined #openstack-operators | 04:31 | |
| *** blair has joined #openstack-operators | 04:33 | |
| *** ggillies_ has quit IRC | 04:33 | |
| *** ggillies_ has joined #openstack-operators | 04:34 | |
| *** ggillies1 has joined #openstack-operators | 04:35 | |
| *** ggillies2 has quit IRC | 04:35 | |
| *** ggillies has quit IRC | 04:36 | |
| *** ggillies has joined #openstack-operators | 04:36 | |
| *** ggillies2 has joined #openstack-operators | 04:37 | |
| *** blair has quit IRC | 04:38 | |
| *** ggillies_ has quit IRC | 04:39 | |
| *** ggillies1 has quit IRC | 04:40 | |
| *** ggillies has quit IRC | 04:40 | |
| *** reed has quit IRC | 04:41 | |
| *** ggillies has joined #openstack-operators | 04:42 | |
| *** ggillies2 has quit IRC | 04:42 | |
| *** ggillies_ has joined #openstack-operators | 04:45 | |
| *** ggillies has quit IRC | 04:46 | |
| *** ggillies_ has quit IRC | 04:50 | |
| *** ggillies has joined #openstack-operators | 04:51 | |
| *** VW_ has quit IRC | 04:53 | |
| *** ggillies has quit IRC | 04:57 | |
| *** ggillies has joined #openstack-operators | 04:58 | |
| *** ggillies_ has joined #openstack-operators | 05:05 | |
| *** SimonChung has joined #openstack-operators | 05:08 | |
| *** ggillies1 has joined #openstack-operators | 05:08 | |
| *** ggillies has quit IRC | 05:09 | |
| *** SimonChung1 has joined #openstack-operators | 05:10 | |
| *** ggillies1 has quit IRC | 05:12 | |
| *** SimonChung has quit IRC | 05:12 | |
| *** ggillies_ has quit IRC | 05:13 | |
| *** ggillies has joined #openstack-operators | 05:18 | |
| *** ggillies_ has joined #openstack-operators | 05:22 | |
| *** ggillies1 has joined #openstack-operators | 05:23 | |
| *** ggillies2 has joined #openstack-operators | 05:25 | |
| *** ggillies has quit IRC | 05:26 | |
| *** ggillies_ has quit IRC | 05:27 | |
| *** Marga_ has joined #openstack-operators | 05:27 | |
| *** Miouge has quit IRC | 05:28 | |
| *** ggillies1 has quit IRC | 05:29 | |
| *** ggillies has joined #openstack-operators | 05:29 | |
| *** sanjayu has joined #openstack-operators | 05:29 | |
| *** ggillies has quit IRC | 05:30 | |
| *** ggillies2 has quit IRC | 05:30 | |
| *** ggillies has joined #openstack-operators | 05:33 | |
| *** blair has joined #openstack-operators | 05:34 | |
| *** ggillies_ has joined #openstack-operators | 05:35 | |
| *** ggillies has quit IRC | 05:38 | |
| *** blair has quit IRC | 05:39 | |
| *** ggillies has joined #openstack-operators | 05:39 | |
| *** ggillies_ has quit IRC | 05:40 | |
| *** ggillies_ has joined #openstack-operators | 05:41 | |
| *** ggillies has quit IRC | 05:45 | |
| *** Marga_ has quit IRC | 05:53 | |
| *** zz_avozza is now known as avozza | 06:05 | |
| *** harlowja is now known as harlowja_away | 06:08 | |
| *** avozza is now known as zz_avozza | 06:15 | |
| *** blair has joined #openstack-operators | 06:35 | |
| *** blair has quit IRC | 06:40 | |
| *** Marga_ has joined #openstack-operators | 06:53 | |
| *** Marga_ has quit IRC | 06:58 | |
| *** belmoreira has joined #openstack-operators | 07:29 | |
| *** blair has joined #openstack-operators | 07:36 | |
| *** zz_avozza is now known as avozza | 07:36 | |
| *** Miouge has joined #openstack-operators | 07:38 | |
| *** blair has quit IRC | 07:40 | |
| *** avozza is now known as zz_avozza | 07:46 | |
| *** Marga_ has joined #openstack-operators | 07:54 | |
| *** Marga_ has quit IRC | 07:59 | |
| *** zz_avozza is now known as avozza | 08:19 | |
| *** Marga_ has joined #openstack-operators | 08:27 | |
| *** Marga_ has quit IRC | 08:31 | |
| *** blair has joined #openstack-operators | 08:33 | |
| *** bvandenh has joined #openstack-operators | 09:18 | |
| *** Marga_ has joined #openstack-operators | 09:28 | |
| *** Marga_ has quit IRC | 09:32 | |
| *** subscope has quit IRC | 09:33 | |
| *** subscope has joined #openstack-operators | 09:38 | |
| *** Miouge_ has joined #openstack-operators | 09:47 | |
| *** Miouge has quit IRC | 09:48 | |
| *** Miouge_ is now known as Miouge | 09:51 | |
| *** derekh has joined #openstack-operators | 10:13 | |
| *** Miouge has quit IRC | 10:22 | |
| *** Marga_ has joined #openstack-operators | 10:25 | |
| *** Miouge has joined #openstack-operators | 11:13 | |
| *** blair has quit IRC | 11:24 | |
| *** Marga_ has quit IRC | 11:37 | |
| *** Marga_ has joined #openstack-operators | 12:00 | |
| *** Marga_ has quit IRC | 12:16 | |
| *** Marga_ has joined #openstack-operators | 12:17 | |
| *** avozza is now known as zz_avozza | 12:36 | |
| *** Marga_ has quit IRC | 12:38 | |
| *** bilco105 has quit IRC | 12:47 | |
| *** bilco105 has joined #openstack-operators | 12:48 | |
| *** zz_avozza is now known as avozza | 12:49 | |
| *** pboros has joined #openstack-operators | 12:50 | |
| *** blair has joined #openstack-operators | 12:55 | |
| *** blair has quit IRC | 13:00 | |
| *** sanjayu has quit IRC | 13:21 | |
| *** jaypipes has joined #openstack-operators | 13:56 | |
| *** blair has joined #openstack-operators | 13:56 | |
| *** blair has quit IRC | 14:00 | |
| *** csoukup has joined #openstack-operators | 14:30 | |
| *** VW_ has joined #openstack-operators | 14:31 | |
| *** VW__ has joined #openstack-operators | 14:35 | |
| *** VW__ has quit IRC | 14:35 | |
| *** VW_ has quit IRC | 14:35 | |
| *** VW_ has joined #openstack-operators | 14:36 | |
| *** avozza is now known as zz_avozza | 14:53 | |
| *** blair has joined #openstack-operators | 14:57 | |
| *** blair has quit IRC | 15:01 | |
| *** dmsimard_away is now known as dmsimard | 15:06 | |
| *** radez_g0n3 is now known as radez | 15:07 | |
| *** VW_ has quit IRC | 15:41 | |
| *** VW_ has joined #openstack-operators | 15:44 | |
| *** mdorman has joined #openstack-operators | 15:45 | |
| *** Ctina__ has joined #openstack-operators | 15:46 | |
| *** signed8bit has joined #openstack-operators | 15:51 | |
| *** blair has joined #openstack-operators | 15:57 | |
| *** reed has joined #openstack-operators | 15:58 | |
| *** bvandenh has quit IRC | 16:00 | |
| *** Marga_ has joined #openstack-operators | 16:01 | |
| *** blair has quit IRC | 16:02 | |
| *** Marga_ has quit IRC | 16:21 | |
| *** j05h has joined #openstack-operators | 16:25 | |
| *** belmoreira has quit IRC | 16:33 | |
| *** mdorman_ has joined #openstack-operators | 16:35 | |
| *** mdorman has quit IRC | 16:35 | |
| *** mdorman_ is now known as mdorman | 16:35 | |
| *** signed8bit is now known as signed8bit_ZZZzz | 16:37 | |
| *** signed8bit_ZZZzz is now known as signed8bit | 16:41 | |
| *** signed8bit has quit IRC | 16:43 | |
| *** signed8bit has joined #openstack-operators | 16:44 | |
| *** j05h has quit IRC | 16:44 | |
| *** alop has joined #openstack-operators | 16:48 | |
| klindgren | So - is anyone here happy with their rabbitmq configuration? Do you have random issues (servers looking like their connected to rabbitmq - but actually not doing anything?) when restarting rabbitmq, or rebooting a hosting that rabbitmq is on or in general anything that would cause a failover? | 16:48 |
|---|---|---|
| *** j05h has joined #openstack-operators | 16:49 | |
| mdorman | ain’t nobody happy with their rmq config :) | 16:51 |
| *** _nick has joined #openstack-operators | 16:52 | |
| *** SimonChung1 has quit IRC | 16:55 | |
| jlk | haha | 16:55 |
| jlk | we're moving from rabbits behind HA proxy / ucarp to all rabbits listed in the openstack configs | 16:56 |
| jlk | to allow oslo to handle the clustering. No real data on how happy we are with that yet, it's rolled into our juno upgrade | 16:56 |
| *** blair has joined #openstack-operators | 16:58 | |
| *** blair has quit IRC | 17:03 | |
| mdorman | jlk: i’ve been thinking along the same lines. it doesn’t seem like running rmq behind a vip really buys you anything. | 17:04 |
| mdorman | plus i think there is some rmq keep alive stuff in oslo now, or at least there was a push for that at some point | 17:05 |
| _nick | we had no end of trouble with rabbitmq behind haproxy - it's a lot less problematic letting oslo handle it and with ha queues | 17:05 |
| jlk | yeah, unfortunately like others we have non-openstack services wanting ot use rabbit | 17:07 |
| jlk | for those I'm not sure we can list multiple servers for them or if we'll have to continue down the VIP route | 17:07 |
| *** emagana has joined #openstack-operators | 17:13 | |
| mdorman | well these are some good data points | 17:15 |
| mdorman | i feel like every discussion around rmq at summit/meetups mostly just devolves to “rmq and/or oslo’s handling of it sucks!” and we never get to enough specifics to be useful. | 17:16 |
| jlk | I'm curious if anybody has given rabbit the finger and is using some other AMQP provider | 17:17 |
| klindgren | you mean qpid or 0mq? | 17:17 |
| klindgren | I read some stuff about qpid not scaling a long time ago and most of those people said - we switched to rabbitmq and the issues went away | 17:18 |
| klindgren | pretty sure omq is getting pulled out everywhere due to lack of use/maintenace | 17:18 |
| klindgren | 0mq* | 17:18 |
| jlk | yeah, not sure trading erlang for java is we'd want to do | 17:18 |
| *** Piet has joined #openstack-operators | 17:20 | |
| Ctina__ | We tried using qpid but had scaling issues so we quickly went back to rabbitmq | 17:27 |
| *** Ctina__ is now known as Ctina | 17:28 | |
| *** derekh has quit IRC | 17:42 | |
| *** SimonChung has joined #openstack-operators | 17:46 | |
| *** blair has joined #openstack-operators | 17:59 | |
| *** SimonChung1 has joined #openstack-operators | 18:00 | |
| *** SimonChung has quit IRC | 18:00 | |
| *** blair has quit IRC | 18:03 | |
| *** Marga_ has joined #openstack-operators | 18:05 | |
| *** Marga__ has joined #openstack-operators | 18:06 | |
| *** Marga_ has quit IRC | 18:06 | |
| *** Marga__ has quit IRC | 18:08 | |
| *** Marga_ has joined #openstack-operators | 18:08 | |
| *** harlowja_away is now known as harlowja | 18:16 | |
| *** Marga__ has joined #openstack-operators | 18:17 | |
| *** Marga_ has quit IRC | 18:18 | |
| *** Marga__ has quit IRC | 18:20 | |
| *** Marga_ has joined #openstack-operators | 18:21 | |
| *** jaypipes is now known as jpipes-car-repai | 18:26 | |
| *** Marga_ has quit IRC | 18:35 | |
| *** Marga_ has joined #openstack-operators | 18:36 | |
| *** Marga_ has quit IRC | 18:37 | |
| *** Marga_ has joined #openstack-operators | 18:38 | |
| *** Marga_ has quit IRC | 18:39 | |
| *** Marga_ has joined #openstack-operators | 18:40 | |
| *** Marga_ has quit IRC | 18:41 | |
| *** Marga_ has joined #openstack-operators | 18:42 | |
| *** signed8bit is now known as signed8bit_ZZZzz | 18:45 | |
| *** emagana has quit IRC | 18:46 | |
| *** Marga_ has quit IRC | 18:49 | |
| *** Marga_ has joined #openstack-operators | 18:50 | |
| *** signed8bit_ZZZzz has quit IRC | 18:51 | |
| *** Marga_ has quit IRC | 18:52 | |
| *** Marga_ has joined #openstack-operators | 18:53 | |
| *** Marga_ has quit IRC | 18:53 | |
| *** Marga_ has joined #openstack-operators | 18:54 | |
| *** signed8bit has joined #openstack-operators | 18:55 | |
| *** esker has joined #openstack-operators | 18:55 | |
| *** Marga_ has quit IRC | 18:56 | |
| *** Marga_ has joined #openstack-operators | 18:56 | |
| *** Marga_ has quit IRC | 18:57 | |
| *** Marga_ has joined #openstack-operators | 18:58 | |
| *** Marga_ has quit IRC | 18:59 | |
| *** Marga_ has joined #openstack-operators | 18:59 | |
| *** signed8bit has quit IRC | 19:00 | |
| *** blair has joined #openstack-operators | 19:00 | |
| *** signed8bit has joined #openstack-operators | 19:00 | |
| *** blair has quit IRC | 19:04 | |
| *** emagana has joined #openstack-operators | 19:08 | |
| *** reed has quit IRC | 19:26 | |
| *** reed_ has joined #openstack-operators | 19:27 | |
| *** lodgenbd has joined #openstack-operators | 19:41 | |
| *** Miouge has quit IRC | 19:45 | |
| *** david-lyle has joined #openstack-operators | 19:49 | |
| *** emagana has quit IRC | 19:51 | |
| *** emagana has joined #openstack-operators | 19:52 | |
| *** emagana has quit IRC | 19:56 | |
| *** emagana has joined #openstack-operators | 19:57 | |
| *** Marga_ has quit IRC | 19:59 | |
| *** Marga_ has joined #openstack-operators | 19:59 | |
| *** blair has joined #openstack-operators | 20:00 | |
| *** Marga_ has quit IRC | 20:03 | |
| *** Marga_ has joined #openstack-operators | 20:04 | |
| *** Marga_ has quit IRC | 20:05 | |
| *** blair has quit IRC | 20:05 | |
| *** Marga_ has joined #openstack-operators | 20:06 | |
| *** Marga_ has quit IRC | 20:08 | |
| *** Marga_ has joined #openstack-operators | 20:09 | |
| *** Marga_ has quit IRC | 20:14 | |
| *** Marga_ has joined #openstack-operators | 20:15 | |
| *** david-lyle has quit IRC | 20:15 | |
| *** signed8bit has quit IRC | 20:15 | |
| *** j05h has quit IRC | 20:16 | |
| *** Marga_ has quit IRC | 20:17 | |
| *** Marga_ has joined #openstack-operators | 20:18 | |
| *** j05h has joined #openstack-operators | 20:18 | |
| *** fifieldt_ has joined #openstack-operators | 20:22 | |
| *** fifieldt__ has quit IRC | 20:25 | |
| *** VW_ has quit IRC | 20:34 | |
| *** SimonChung1 has quit IRC | 20:34 | |
| *** SimonChung has joined #openstack-operators | 20:34 | |
| *** j05h has quit IRC | 20:34 | |
| *** VW_ has joined #openstack-operators | 20:35 | |
| *** radez is now known as radez_g0n3 | 20:35 | |
| *** signed8bit has joined #openstack-operators | 20:50 | |
| *** signed8bit has quit IRC | 20:51 | |
| *** signed8bit has joined #openstack-operators | 20:51 | |
| *** j05h has joined #openstack-operators | 20:56 | |
| *** blair has joined #openstack-operators | 21:01 | |
| *** SimonChung1 has joined #openstack-operators | 21:03 | |
| *** SimonChung has quit IRC | 21:03 | |
| *** SimonChung has joined #openstack-operators | 21:04 | |
| *** SimonChung1 has quit IRC | 21:04 | |
| *** SimonChung1 has joined #openstack-operators | 21:04 | |
| *** SimonChung has quit IRC | 21:04 | |
| *** blair has quit IRC | 21:06 | |
| mgagne | mdorman: thanks for sharing your experience with nova-cells ;) | 21:07 |
| mdorman | yeah, np, i hope it’s useful for people | 21:08 |
| mdorman | working on moving our prod environment over this week and next, so hope to have some tidbits to add after that | 21:08 |
| mgagne | mdorman: I'm in the *exact* same boat and didn't know you were paddling next to me ;) | 21:11 |
| *** j05h has quit IRC | 21:12 | |
| mdorman | nice | 21:16 |
| mdorman | so far i’ve spent all my time getting the rmq cluster expanded | 21:16 |
| mdorman | the new server i have are cent7, existing ones cent6. they had different versions of erlang so i couldn’t add the new ones into the existing rmq cluster | 21:17 |
| mdorman | so i had to uplift the cent6 boxes to cent7 first | 21:17 |
| mdorman | not a real fun exercise :) | 21:17 |
| mgagne | haha, yes, rmq isn't always part of a success story | 21:17 |
| *** j05h has joined #openstack-operators | 21:18 | |
| mdorman | :) | 21:19 |
| mgagne | btw, why can't you let nova-cells sync "up" the instances to the api cell? will it result in missing pieces? | 21:20 |
| *** esker has quit IRC | 21:26 | |
| mdorman | that’s what my experience was. the only field it actually updated in the api cell was the instances.cell field. everything else was just NULL | 21:27 |
| mdorman | which broke a bunch of stuff in the api cell. | 21:27 |
| mdorman | that’s a good point, i should add that reasoning to the post | 21:27 |
| mdorman | maybe there some some config or something i was missing, but that was the behavior i experienced | 21:28 |
| *** jpipes-car-repai is now known as jaypipes | 21:30 | |
| mgagne | mdorman: alright, will have do to more experimentation because I was under the impression that you could spawn nova-api at the compute cell level, create instance in there and let nova-cells sync them up. (ok, you might have discrepancy in quota usage) | 21:33 |
| mdorman | yeah let me know if you figure out a way to make that work. that’d simplify it quite a bit | 21:35 |
| *** pboros has quit IRC | 21:40 | |
| *** david-lyle has joined #openstack-operators | 21:46 | |
| *** signed8bit has quit IRC | 21:51 | |
| *** Marga_ has quit IRC | 21:53 | |
| *** Marga_ has joined #openstack-operators | 21:53 | |
| *** Marga_ has quit IRC | 21:58 | |
| *** Marga_ has joined #openstack-operators | 21:58 | |
| *** Ctina_ has joined #openstack-operators | 21:58 | |
| *** Marga_ has quit IRC | 21:59 | |
| *** Marga_ has joined #openstack-operators | 21:59 | |
| *** Marga_ has quit IRC | 22:00 | |
| *** Marga_ has joined #openstack-operators | 22:00 | |
| *** blair has joined #openstack-operators | 22:02 | |
| *** Ctina has quit IRC | 22:02 | |
| *** Ctina_ has quit IRC | 22:03 | |
| *** SimonChung has joined #openstack-operators | 22:04 | |
| *** SimonChung1 has quit IRC | 22:04 | |
| *** SimonChung1 has joined #openstack-operators | 22:05 | |
| *** SimonChung has quit IRC | 22:05 | |
| *** SimonChung has joined #openstack-operators | 22:06 | |
| *** SimonChung1 has quit IRC | 22:06 | |
| *** blair has quit IRC | 22:07 | |
| *** SimonChung has quit IRC | 22:07 | |
| *** SimonChung has joined #openstack-operators | 22:07 | |
| *** Marga_ has quit IRC | 22:08 | |
| *** Marga_ has joined #openstack-operators | 22:08 | |
| *** Marga_ has quit IRC | 22:09 | |
| *** Marga_ has joined #openstack-operators | 22:09 | |
| *** Marga_ has quit IRC | 22:10 | |
| *** Marga_ has joined #openstack-operators | 22:10 | |
| *** Marga_ has quit IRC | 22:11 | |
| *** Marga_ has joined #openstack-operators | 22:11 | |
| *** Marga_ has quit IRC | 22:12 | |
| *** Marga_ has joined #openstack-operators | 22:12 | |
| *** mdorman has quit IRC | 22:14 | |
| *** Marga_ has quit IRC | 22:19 | |
| *** Marga_ has joined #openstack-operators | 22:19 | |
| *** j05h has quit IRC | 22:21 | |
| *** Marga_ has quit IRC | 22:22 | |
| *** Marga_ has joined #openstack-operators | 22:22 | |
| *** j05h has joined #openstack-operators | 22:23 | |
| *** Marga_ has quit IRC | 22:23 | |
| *** Marga_ has joined #openstack-operators | 22:24 | |
| *** david-lyle has quit IRC | 22:27 | |
| *** Marga_ has quit IRC | 22:27 | |
| *** Marga_ has joined #openstack-operators | 22:27 | |
| *** fifieldt__ has joined #openstack-operators | 22:32 | |
| *** fifieldt_ has quit IRC | 22:34 | |
| *** david-lyle has joined #openstack-operators | 22:37 | |
| *** openstack has joined #openstack-operators | 23:02 | |
| *** ruagair has joined #openstack-operators | 23:05 | |
| *** med_ has joined #openstack-operators | 23:05 | |
| *** furlongm has joined #openstack-operators | 23:05 | |
| *** bradm has joined #openstack-operators | 23:05 | |
| *** j^2 has joined #openstack-operators | 23:05 | |
| *** avozza has joined #openstack-operators | 23:05 | |
| *** jemangs has joined #openstack-operators | 23:05 | |
| *** crinkle_ has joined #openstack-operators | 23:05 | |
| *** david-lyle has joined #openstack-operators | 23:05 | |
| *** michchap_ has joined #openstack-operators | 23:05 | |
| *** Nebula__ has joined #openstack-operators | 23:05 | |
| *** VW__ has joined #openstack-operators | 23:05 | |
| *** retr0h has joined #openstack-operators | 23:05 | |
| *** alop has joined #openstack-operators | 23:05 | |
| *** Guest97735 has joined #openstack-operators | 23:05 | |
| *** dvorak has joined #openstack-operators | 23:05 | |
| *** radez_g0` has joined #openstack-operators | 23:05 | |
| *** j05h has joined #openstack-operators | 23:05 | |
| *** csoukup has joined #openstack-operators | 23:05 | |
| *** subscope has joined #openstack-operators | 23:05 | |
| *** ggillies_ has joined #openstack-operators | 23:05 | |
| *** serverascode has joined #openstack-operators | 23:05 | |
| *** jraim has joined #openstack-operators | 23:05 | |
| *** michchap has joined #openstack-operators | 23:05 | |
| *** jlk has joined #openstack-operators | 23:05 | |
| *** verdurin has joined #openstack-operators | 23:05 | |
| *** tacticus has joined #openstack-operators | 23:05 | |
| *** xavpaice has joined #openstack-operators | 23:05 | |
| *** B_Smith has joined #openstack-operators | 23:05 | |
| *** xnw has joined #openstack-operators | 23:05 | |
| *** beddari has joined #openstack-operators | 23:05 | |
| *** notmyname has joined #openstack-operators | 23:05 | |
| *** Ti-mo has joined #openstack-operators | 23:05 | |
| *** ChanServ has joined #openstack-operators | 23:05 | |
| *** dmsimard has joined #openstack-operators | 23:05 | |
| *** sendak.freenode.net sets mode: +o ChanServ | 23:05 | |
| *** Marga_ has joined #openstack-operators | 23:06 | |
| *** Marga_ has quit IRC | 23:06 | |
| *** j05h has quit IRC | 23:07 | |
| *** mdavidson has joined #openstack-operators | 23:07 | |
| *** dvorak has quit IRC | 23:07 | |
| *** subscope has quit IRC | 23:07 | |
| *** michchap has quit IRC | 23:07 | |
| *** bilco105 has joined #openstack-operators | 23:08 | |
| *** Marga_ has joined #openstack-operators | 23:10 | |
| *** Marga_ has quit IRC | 23:10 | |
| *** Marga_ has joined #openstack-operators | 23:11 | |
| *** pasquier-s has joined #openstack-operators | 23:12 | |
| *** simonmcc has joined #openstack-operators | 23:12 | |
| *** emagana has joined #openstack-operators | 23:12 | |
| *** klindgren_ has joined #openstack-operators | 23:12 | |
| *** blair has joined #openstack-operators | 23:12 | |
| *** _nick has joined #openstack-operators | 23:12 | |
| *** fsaad has joined #openstack-operators | 23:12 | |
| *** SimonChung has joined #openstack-operators | 23:12 | |
| *** jamespd has joined #openstack-operators | 23:12 | |
| *** Marga_ has quit IRC | 23:13 | |
| *** subscope has joined #openstack-operators | 23:13 | |
| *** dvorak has joined #openstack-operators | 23:13 | |
| *** Marga_ has joined #openstack-operators | 23:13 | |
| *** puck` has joined #openstack-operators | 23:13 | |
| *** wabu has joined #openstack-operators | 23:14 | |
| *** Marga_ has quit IRC | 23:14 | |
| *** Marga_ has joined #openstack-operators | 23:15 | |
| *** harlowja has joined #openstack-operators | 23:15 | |
| *** snk has joined #openstack-operators | 23:16 | |
| *** Marga_ has quit IRC | 23:16 | |
| *** Marga_ has joined #openstack-operators | 23:16 | |
| *** mdorman has joined #openstack-operators | 23:21 | |
| *** david-lyle has quit IRC | 23:24 | |
| *** alop has quit IRC | 23:26 | |
| *** alop has joined #openstack-operators | 23:26 | |
| *** VW__ has quit IRC | 23:26 | |
| *** VW__ has joined #openstack-operators | 23:26 | |
| *** j^2 has quit IRC | 23:27 | |
| *** j^2 has joined #openstack-operators | 23:27 | |
| *** SimonChung has quit IRC | 23:30 | |
| *** SimonChung has joined #openstack-operators | 23:30 | |
| *** harlowja has quit IRC | 23:31 | |
| *** harlowja has joined #openstack-operators | 23:31 | |
| *** Marga_ has quit IRC | 23:31 | |
| *** Marga_ has joined #openstack-operators | 23:31 | |
| *** Marga_ has quit IRC | 23:34 | |
| *** Marga_ has joined #openstack-operators | 23:34 | |
| *** Marga_ has quit IRC | 23:35 | |
| *** Marga_ has joined #openstack-operators | 23:35 | |
| *** Marga_ has quit IRC | 23:37 | |
| *** Marga_ has joined #openstack-operators | 23:37 | |
| *** Marga_ has quit IRC | 23:39 | |
| *** Marga_ has joined #openstack-operators | 23:39 | |
| *** j05h has joined #openstack-operators | 23:40 | |
| *** Marga_ has quit IRC | 23:41 | |
| dvorak | klindgren_: we had pivotal in for 2 days recently to talk about rabbitmq, they strongly recommended running it through a load balancer | 23:41 |
| *** Marga_ has joined #openstack-operators | 23:41 | |
| dvorak | one thing we've talked about doing when doing rabbit restarts is to disable rabbit traffic into a specific node, then doing graceful restarts of everything that might be using it | 23:42 |
| *** Marga_ has quit IRC | 23:47 | |
| *** Marga_ has joined #openstack-operators | 23:47 | |
| mdorman | curious why they highly recommended going through a LB? | 23:48 |
| dvorak | part of it was that they said very few clients correctly handle connectiing to multiple hosts, or reconnecting to when connections fail with multiple hosts, or properly distributing traffic | 23:51 |
| *** crinkle_ is now known as crinkle | 23:51 | |
| dvorak | but a big part of it was that it makes it a lot easier to take one node offline before taking it down | 23:51 |
| dvorak | so you can disable new connections to the node and bleed them off over time | 23:51 |
| mdorman | i see | 23:52 |
| mdorman | that makes sense | 23:52 |
| dvorak | our experience with rabbit has been about the same as everyone elses | 23:52 |
| mdorman | def. seems like you hear about the same story from everybody | 23:52 |
| dvorak | a lot of our problems have been with unsynchronized queues, because we weren't manually syncing them after bringing a server back up | 23:52 |
| dvorak | and we don't have auto sync turned on (it's not the default) | 23:52 |
| *** Marga_ has quit IRC | 23:53 | |
| dvorak | and they didn't recommend turning on auto sync queues either | 23:53 |
| *** Marga_ has joined #openstack-operators | 23:53 | |
| dvorak | what I've proposed we do, and I'm not sure it's a good idea, is that when we need to restart a rabbit node, we disable it in the LB to disable any new connections to it | 23:54 |
| dvorak | then for each node that has connections talking to it, take *that* out of the LB, restart all it's services, then move to the next | 23:54 |
| dvorak | which should allow us to bleed off all the rabbitmq connections without causing issues with dead connections that the clients dont' know about | 23:55 |
| dvorak | which is pretty horrible, but probably not actually too bad in practice if we automate it | 23:55 |
| dvorak | I've been doing a bunch of LB work this week as prep for our Juno upgrade that should make it easier to do that sort of thing later | 23:56 |
| *** reed_ has joined #openstack-operators | 23:56 | |
| *** csoukup has quit IRC | 23:57 | |
| dvorak | how far have you guys gotten on your juno upgrade? Are you done? | 23:57 |
| *** SimonChung1 has joined #openstack-operators | 23:58 | |
| *** SimonChung has quit IRC | 23:58 | |
| *** SimonChung has joined #openstack-operators | 23:58 | |
| *** SimonChung1 has quit IRC | 23:58 | |
| *** Marga_ has quit IRC | 23:59 | |
| *** Marga__ has joined #openstack-operators | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!