| *** mahito has joined #openstack-operators | 00:03 | |
| *** saneax has quit IRC | 00:06 | |
| *** alop has quit IRC | 00:44 | |
| *** alop has joined #openstack-operators | 00:44 | |
| *** jmckind has quit IRC | 00:45 | |
| *** mahito has quit IRC | 01:02 | |
| *** mahito has joined #openstack-operators | 01:04 | |
| *** Marga__ has quit IRC | 01:12 | |
| *** saneax has joined #openstack-operators | 01:38 | |
| *** signed8bit has joined #openstack-operators | 01:40 | |
| *** ggillies_ has quit IRC | 01:42 | |
| *** ggillies has joined #openstack-operators | 01:45 | |
| *** bhunter71 has quit IRC | 01:55 | |
| *** alop has quit IRC | 02:00 | |
| *** bhunter71 has joined #openstack-operators | 02:02 | |
| *** mahito has quit IRC | 02:18 | |
| *** markvoelker has joined #openstack-operators | 02:19 | |
| *** mahito has joined #openstack-operators | 02:20 | |
| *** markvoelker_ has joined #openstack-operators | 02:21 | |
| *** markvoelker has quit IRC | 02:23 | |
| *** saneax has quit IRC | 02:31 | |
| *** hakimo has joined #openstack-operators | 02:52 | |
| *** hakimo_ has quit IRC | 02:54 | |
| *** signed8bit has quit IRC | 03:12 | |
| *** markvoelker_ has quit IRC | 03:23 | |
| *** markvoelker_ has joined #openstack-operators | 03:26 | |
| *** bhunter71 has quit IRC | 03:43 | |
| *** mahito has quit IRC | 04:06 | |
| *** mahito has joined #openstack-operators | 04:07 | |
| *** saneax has joined #openstack-operators | 04:21 | |
| *** fawadkhaliq has joined #openstack-operators | 04:55 | |
| *** mahito has quit IRC | 04:59 | |
| *** mahito has joined #openstack-operators | 05:01 | |
| *** ig0r_ has joined #openstack-operators | 05:24 | |
| *** yapeng has joined #openstack-operators | 05:38 | |
| *** markvoelker_ has quit IRC | 05:48 | |
| *** ig0r__ has joined #openstack-operators | 06:11 | |
| *** ig0r_ has quit IRC | 06:12 | |
| *** Marga_ has joined #openstack-operators | 06:16 | |
| *** ig0r_ has joined #openstack-operators | 06:24 | |
| *** ig0r_ has quit IRC | 06:24 | |
| *** yapeng has quit IRC | 06:31 | |
| *** markvoelker has joined #openstack-operators | 06:48 | |
| *** belmoreira has joined #openstack-operators | 06:52 | |
| *** markvoelker has quit IRC | 06:53 | |
| *** melwitt has quit IRC | 07:27 | |
| *** melwitt has joined #openstack-operators | 07:33 | |
| *** ToMiles has quit IRC | 07:47 | |
| *** saneax has quit IRC | 08:07 | |
| *** mahito has quit IRC | 08:35 | |
| beddari | on that road myself so would love to know the answer of the above ^^ :) | 08:40 |
|---|---|---|
| *** markvoelker has joined #openstack-operators | 08:49 | |
| *** markvoelker has quit IRC | 08:54 | |
| beddari | omg the "help us build openstack talent" survey ... | 08:58 |
| beddari | I was expecting to have a hard time but not this :) | 08:58 |
| *** Marga_ has quit IRC | 09:44 | |
| *** _nick has quit IRC | 10:22 | |
| *** bilco105 has quit IRC | 10:22 | |
| *** _nick has joined #openstack-operators | 10:26 | |
| *** bilco105 has joined #openstack-operators | 10:27 | |
| *** ig0r__ has quit IRC | 11:15 | |
| *** signed8bit has joined #openstack-operators | 11:17 | |
| *** ig0r_ has joined #openstack-operators | 11:19 | |
| *** ig0r__ has joined #openstack-operators | 11:20 | |
| *** signed8b_ has joined #openstack-operators | 11:20 | |
| *** signed8bit has quit IRC | 11:23 | |
| *** Marga_ has joined #openstack-operators | 12:03 | |
| *** subscope has quit IRC | 12:16 | |
| clayton | klindgren or mdorman, are you guys running rabbit behind a load balancer now that you have heartbeats turned on? | 12:19 |
| clayton | I'd like to run it behind a load balancer to make some migrations easier we'll be doing soon, and if so, I'd like to just do that when we move to kilo at the same time | 12:21 |
| *** ig0r__ has quit IRC | 12:22 | |
| *** ig0r__ has joined #openstack-operators | 12:27 | |
| *** subscope has joined #openstack-operators | 12:31 | |
| clayton | also, what do you guys have heartbeat_timeout_threshold set to? I see in 1.8.3 of oslo.messaging it's disabled, but I know it used to be 60 seconds | 12:34 |
| clayton | that seems kind of long | 12:35 |
| *** bhunter71 has joined #openstack-operators | 12:52 | |
| *** VW_ has joined #openstack-operators | 13:00 | |
| *** arcimboldo has joined #openstack-operators | 13:24 | |
| *** markvoelker has joined #openstack-operators | 13:26 | |
| *** jmckind has joined #openstack-operators | 13:27 | |
| *** markvoelker_ has joined #openstack-operators | 13:30 | |
| *** markvoelker has quit IRC | 13:31 | |
| *** signed8b_ is now known as signed8bit_ZZZzz | 13:32 | |
| *** alaski is now known as lascii | 13:32 | |
| *** signed8bit_ZZZzz is now known as signed8b_ | 13:32 | |
| *** ig0r__ has quit IRC | 13:34 | |
| *** jmckind has quit IRC | 13:36 | |
| *** jmckind has joined #openstack-operators | 13:37 | |
| *** ferest has joined #openstack-operators | 13:39 | |
| *** ferest has quit IRC | 13:40 | |
| *** mdorman has joined #openstack-operators | 13:49 | |
| *** balajin has quit IRC | 13:50 | |
| *** yapeng has joined #openstack-operators | 13:56 | |
| klindgren | clayton, 60 seconds, 2 something, it also heartbeats every time it receives a message as well. So it usually ends up throwing out a fair number of heartbeats. We moved to listing all the rabbitmq hosts. Behind the LB still had some issues when I was testing | 14:25 |
| clayton | heartbeating every time it sends a message is pretty broken | 14:26 |
| clayton | any message it sends is supposed to effectively count as a heartbeat according to the AMQP spec | 14:26 |
| klindgren | it heartbeats first to make sure its still connected | 14:26 |
| klindgren | then send the message | 14:26 |
| clayton | so you're only supposed to send heartbeats when the connection is idle | 14:26 |
| clayton | I'd really like to move rabbit behind the LB because it'd make taking a node out of service a lot easier and we're going to be moving a pretty much all of our gear to another part of the datacenter relatively soon | 14:27 |
| clayton | what kind of issues did you see behind the LB? | 14:27 |
| clayton | I did some packet captures this morning and it actually sends the heartbeats every 25 seconds if you have it set to 60 and 2 | 14:29 |
| clayton | which is fine, I'd rather it send them early rather than have a race condition | 14:29 |
| klindgren | when I failed out rabbitmq server | 14:29 |
| klindgren | either by iptables or buy stopping the broken | 14:30 |
| klindgren | I would start logging ssl errors on some of the clients | 14:30 |
| klindgren | I know a guy from rabbitmq was going to actively work with the oslo.messaging guys to make it better | 14:30 |
| klindgren | but thts what I remember as of 1.8.1 | 14:31 |
| clayton | we were planning to take the node out the LB before restarting clients | 14:31 |
| klindgren | from my xp - with a list - you just dont need to worry about it | 14:31 |
| klindgren | but agreed I would like to get to ^^ that state | 14:32 |
| clayton | so force them to move over before taking down a specific node | 14:32 |
| klindgren | in my xp everythign detected and reconnected pretty fast with the list | 14:32 |
| clayton | well, we're hoping to upgrade prod to kilo in the next week or so, so I think we're going to try to test it going through the LB a lot during that time and see if we have issues | 14:32 |
| klindgren | to the point that I didn't have to restart clients | 14:32 |
| clayton | I'd like to avoid service restarts in the future to move to the LB if we're going to end up at that point anyway | 14:32 |
| klindgren | +100 | 14:33 |
| klindgren | I am jsut syaing - thought behind a LB seems like (is a better) config with the state of oslo.messaging and rabbitmq | 14:33 |
| klindgren | for us - it was nothing but pain | 14:33 |
| clayton | nod, we'll see, I'll let you know how it goes | 14:33 |
| clayton | do you guys have everything on kilo now? I saw you asking about conductor cpu load | 14:34 |
| clayton | We've had so many more problems going to kilo than we did with juno, and juno was a giant pain | 14:34 |
| *** markvoelker_ has quit IRC | 14:40 | |
| *** fawadkhaliq has quit IRC | 14:41 | |
| klindgren | that was my own fault | 14:42 |
| klindgren | we have a random issue with our staticroute api | 14:42 |
| klindgren | thats fails to remove routes for floating ip's | 14:42 |
| klindgren | so we created a deletion task in nova-computes | 14:43 |
| klindgren | to find vm's in deleting state and retry the deltion | 14:43 |
| klindgren | apparently somehtign made the change that actually deletes the vm's but they dont get the deleting state removed from them | 14:43 |
| klindgren | so we hada number of those vm's in that state and every2 minutes were tryign to get info about the vm's and nuke them | 14:44 |
| clayton | ugh | 14:44 |
| *** klindgren has quit IRC | 14:49 | |
| *** fawadkhaliq has joined #openstack-operators | 14:54 | |
| *** chlong has quit IRC | 14:57 | |
| *** fawadkhaliq has quit IRC | 14:59 | |
| *** fawadkhaliq has joined #openstack-operators | 15:06 | |
| *** subscope has quit IRC | 15:07 | |
| *** chlong has joined #openstack-operators | 15:10 | |
| mdorman | clayton: re: kilo. we have one of our prod environments up on kilo now, the other one upgrades on monday | 15:19 |
| *** belmoreira has quit IRC | 15:24 | |
| *** yapeng has quit IRC | 15:30 | |
| *** Piet has joined #openstack-operators | 15:44 | |
| *** markvoelker has joined #openstack-operators | 15:59 | |
| *** markvoelker has quit IRC | 16:02 | |
| *** Marga_ has quit IRC | 16:05 | |
| *** SimonChung1 has quit IRC | 16:13 | |
| *** SimonChung has joined #openstack-operators | 16:13 | |
| *** ig0r__ has joined #openstack-operators | 16:21 | |
| *** signed8bit has joined #openstack-operators | 16:21 | |
| *** signed8b_ has quit IRC | 16:24 | |
| *** alop has joined #openstack-operators | 16:33 | |
| *** amit213 has quit IRC | 16:50 | |
| *** amit213 has joined #openstack-operators | 16:50 | |
| *** markvoelker has joined #openstack-operators | 17:03 | |
| *** chlong has quit IRC | 17:09 | |
| *** Marga_ has joined #openstack-operators | 17:10 | |
| *** signed8bit is now known as signed8bit_ZZZzz | 17:20 | |
| *** signed8bit_ZZZzz is now known as signed8bit | 17:22 | |
| *** fawadkhaliq has quit IRC | 17:23 | |
| *** fawadkhaliq has joined #openstack-operators | 17:23 | |
| *** Marga_ has quit IRC | 17:24 | |
| *** ig0r__ has quit IRC | 17:24 | |
| *** arcimboldo has quit IRC | 17:25 | |
| *** signed8b_ has joined #openstack-operators | 17:25 | |
| *** Marga_ has joined #openstack-operators | 17:25 | |
| *** markvoelker has quit IRC | 17:26 | |
| *** signed8bit has quit IRC | 17:28 | |
| *** Piet has quit IRC | 17:50 | |
| *** Piet has joined #openstack-operators | 17:52 | |
| *** Piet has quit IRC | 18:09 | |
| *** Piet has joined #openstack-operators | 18:14 | |
| *** markvoelker has joined #openstack-operators | 18:28 | |
| *** VW__ has joined #openstack-operators | 18:28 | |
| *** Piet has quit IRC | 18:30 | |
| *** VW_ has quit IRC | 18:32 | |
| *** VW__ has quit IRC | 18:32 | |
| *** markvoelker has quit IRC | 18:37 | |
| *** fawadkhaliq has quit IRC | 18:40 | |
| *** Marga_ has quit IRC | 18:49 | |
| *** Marga__ has joined #openstack-operators | 18:49 | |
| *** fawadkhaliq has joined #openstack-operators | 19:05 | |
| *** fawadkhaliq has quit IRC | 19:09 | |
| *** ig0r__ has joined #openstack-operators | 19:12 | |
| *** ig0r_ has quit IRC | 19:26 | |
| *** openstack has joined #openstack-operators | 19:34 | |
| *** ig0r__ has quit IRC | 19:35 | |
| *** ig0r_ has joined #openstack-operators | 19:39 | |
| *** ig0r_ has quit IRC | 19:46 | |
| *** jmckind has quit IRC | 19:57 | |
| *** jmckind has joined #openstack-operators | 19:58 | |
| *** jmckind has quit IRC | 19:58 | |
| *** jmckind has joined #openstack-operators | 19:59 | |
| *** Piet has joined #openstack-operators | 20:28 | |
| *** ToMiles has joined #openstack-operators | 20:43 | |
| *** fawadkhaliq has joined #openstack-operators | 21:06 | |
| *** fawadkhaliq has quit IRC | 21:11 | |
| *** hiddentoken has joined #openstack-operators | 21:38 | |
| *** hiddentoken has quit IRC | 21:39 | |
| *** hiddentoken has joined #openstack-operators | 21:40 | |
| *** jmckind has quit IRC | 21:42 | |
| *** signed8b_ is now known as signed8bit_ZZZzz | 21:44 | |
| *** ToMiles has quit IRC | 21:51 | |
| *** ToMiles has joined #openstack-operators | 21:53 | |
| *** Marga__ has quit IRC | 23:05 | |
| *** mdorman has quit IRC | 23:42 | |
| *** alop has quit IRC | 23:52 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!