| *** mjrichardson has quit IRC | 00:02 | |
| *** xavpaice has quit IRC | 00:02 | |
| *** beddari has joined #openstack-operators | 00:02 | |
| *** xavpaice has joined #openstack-operators | 00:03 | |
| *** piet has quit IRC | 00:03 | |
| *** saneax is now known as saneax_AFK | 00:05 | |
| *** beddari has quit IRC | 00:07 | |
| *** ducttape_ has quit IRC | 00:07 | |
| *** mjrichardson has joined #openstack-operators | 00:07 | |
| *** ducttape_ has joined #openstack-operators | 00:08 | |
| *** piet has joined #openstack-operators | 00:08 | |
| *** ducttape_ has quit IRC | 00:09 | |
| *** beddari has joined #openstack-operators | 00:12 | |
| *** piet has quit IRC | 00:21 | |
| *** beddari has quit IRC | 00:22 | |
| *** beddari_ has joined #openstack-operators | 00:22 | |
| *** beddari_ is now known as beddari | 00:22 | |
| *** fragatina has joined #openstack-operators | 00:23 | |
| *** saneax_AFK is now known as saneax | 00:25 | |
| *** piet has joined #openstack-operators | 00:25 | |
| *** fragatina has quit IRC | 00:27 | |
| *** beddari has quit IRC | 00:27 | |
| *** beddari has joined #openstack-operators | 00:32 | |
| *** beddari has quit IRC | 00:37 | |
| *** beddari has joined #openstack-operators | 00:40 | |
| *** beddari has quit IRC | 00:48 | |
| *** beddari has joined #openstack-operators | 00:52 | |
| *** beddari has quit IRC | 00:57 | |
| *** piet has quit IRC | 00:59 | |
| *** piet has joined #openstack-operators | 00:59 | |
| *** beddari has joined #openstack-operators | 01:01 | |
| *** beddari has quit IRC | 01:08 | |
| *** beddari has joined #openstack-operators | 01:12 | |
| *** piet has quit IRC | 01:14 | |
| *** piet has joined #openstack-operators | 01:14 | |
| *** beddari has quit IRC | 01:17 | |
| *** beddari has joined #openstack-operators | 01:22 | |
| *** fragatina has joined #openstack-operators | 01:24 | |
| *** fragatina has quit IRC | 01:29 | |
| *** piet has quit IRC | 01:29 | |
| *** beddari has quit IRC | 01:32 | |
| *** beddari has joined #openstack-operators | 01:32 | |
| *** beddari has quit IRC | 01:37 | |
| *** ducttape_ has joined #openstack-operators | 01:40 | |
| *** beddari has joined #openstack-operators | 01:42 | |
| *** ducttape_ has quit IRC | 01:45 | |
| *** beddari has quit IRC | 01:47 | |
| *** beddari has joined #openstack-operators | 01:51 | |
| *** beddari has quit IRC | 02:01 | |
| *** beddari has joined #openstack-operators | 02:01 | |
| *** beddari has quit IRC | 02:07 | |
| *** beddari has joined #openstack-operators | 02:11 | |
| *** beddari has quit IRC | 02:18 | |
| *** furlongm has joined #openstack-operators | 02:20 | |
| *** beddari has joined #openstack-operators | 02:22 | |
| *** fragatina has joined #openstack-operators | 02:24 | |
| *** fragatina has quit IRC | 02:29 | |
| *** beddari has quit IRC | 02:32 | |
| *** beddari has joined #openstack-operators | 02:32 | |
| *** beddari has quit IRC | 02:38 | |
| *** beddari has joined #openstack-operators | 02:40 | |
| *** ducttape_ has joined #openstack-operators | 02:42 | |
| *** britthouser has joined #openstack-operators | 02:47 | |
| *** beddari has quit IRC | 02:47 | |
| *** ducttape_ has quit IRC | 02:48 | |
| *** beddari has joined #openstack-operators | 02:52 | |
| *** beddari has quit IRC | 02:58 | |
| *** beddari has joined #openstack-operators | 03:02 | |
| *** rcernin has joined #openstack-operators | 03:09 | |
| *** piet_ has joined #openstack-operators | 03:10 | |
| *** beddari has quit IRC | 03:12 | |
| *** beddari has joined #openstack-operators | 03:12 | |
| *** beddari has quit IRC | 03:17 | |
| *** beddari has joined #openstack-operators | 03:22 | |
| *** fragatina has joined #openstack-operators | 03:25 | |
| *** piet_ has quit IRC | 03:26 | |
| *** fragatina has quit IRC | 03:30 | |
| *** beddari has quit IRC | 03:32 | |
| *** beddari has joined #openstack-operators | 03:32 | |
| *** beddari has quit IRC | 03:38 | |
| *** beddari has joined #openstack-operators | 03:40 | |
| *** ducttape_ has joined #openstack-operators | 03:44 | |
| *** beddari has quit IRC | 03:48 | |
| *** ducttape_ has quit IRC | 03:48 | |
| *** beddari has joined #openstack-operators | 03:51 | |
| *** M00nr41n has joined #openstack-operators | 03:56 | |
| *** beddari has quit IRC | 03:57 | |
| *** beddari has joined #openstack-operators | 04:02 | |
| *** saneax is now known as saneax_AFK | 04:07 | |
| *** beddari has quit IRC | 04:07 | |
| *** beddari has joined #openstack-operators | 04:12 | |
| *** beddari_ has joined #openstack-operators | 04:22 | |
| *** beddari has quit IRC | 04:22 | |
| *** beddari_ is now known as beddari | 04:22 | |
| *** fragatina has joined #openstack-operators | 04:26 | |
| *** M00nr41n has quit IRC | 04:28 | |
| *** beddari has quit IRC | 04:28 | |
| *** fragatina has quit IRC | 04:32 | |
| *** beddari has joined #openstack-operators | 04:32 | |
| *** beddari has quit IRC | 04:42 | |
| *** beddari_ has joined #openstack-operators | 04:43 | |
| *** beddari_ is now known as beddari | 04:43 | |
| *** ducttape_ has joined #openstack-operators | 04:44 | |
| *** beddari has quit IRC | 04:47 | |
| *** ducttape_ has quit IRC | 04:49 | |
| *** beddari has joined #openstack-operators | 04:53 | |
| *** beddari has quit IRC | 04:57 | |
| *** beddari has joined #openstack-operators | 05:00 | |
| *** beddari has quit IRC | 05:12 | |
| *** beddari has joined #openstack-operators | 05:13 | |
| *** xavpaice has quit IRC | 05:13 | |
| *** xavpaice has joined #openstack-operators | 05:13 | |
| *** mjrichardson has quit IRC | 05:14 | |
| *** mjrichardson has joined #openstack-operators | 05:14 | |
| *** beddari has quit IRC | 05:18 | |
| *** beddari has joined #openstack-operators | 05:21 | |
| *** M00nr41n has joined #openstack-operators | 05:24 | |
| *** fragatina has joined #openstack-operators | 05:28 | |
| *** beddari has quit IRC | 05:28 | |
| *** fragatina has quit IRC | 05:32 | |
| *** beddari has joined #openstack-operators | 05:33 | |
| *** beddari has quit IRC | 05:37 | |
| *** beddari has joined #openstack-operators | 05:41 | |
| *** ducttape_ has joined #openstack-operators | 05:45 | |
| *** ducttape_ has quit IRC | 05:50 | |
| *** beddari has quit IRC | 05:51 | |
| *** beddari has joined #openstack-operators | 05:52 | |
| *** rcernin has quit IRC | 06:01 | |
| *** beddari has quit IRC | 06:01 | |
| *** beddari_ has joined #openstack-operators | 06:02 | |
| *** beddari_ is now known as beddari | 06:02 | |
| *** saneax_AFK is now known as saneax | 06:10 | |
| *** beddari has quit IRC | 06:11 | |
| *** beddari_ has joined #openstack-operators | 06:12 | |
| *** beddari_ is now known as beddari | 06:12 | |
| *** beddari has quit IRC | 06:22 | |
| *** beddari has joined #openstack-operators | 06:22 | |
| *** beddari has quit IRC | 06:28 | |
| *** fragatina has joined #openstack-operators | 06:28 | |
| *** beddari has joined #openstack-operators | 06:32 | |
| *** rcernin has joined #openstack-operators | 06:32 | |
| *** fragatina has quit IRC | 06:33 | |
| *** toMeloos has joined #openstack-operators | 06:40 | |
| *** beddari has quit IRC | 06:42 | |
| *** beddari_ has joined #openstack-operators | 06:42 | |
| *** beddari_ is now known as beddari | 06:42 | |
| *** pcaruana has joined #openstack-operators | 06:46 | |
| *** ducttape_ has joined #openstack-operators | 06:46 | |
| *** toMeloos has quit IRC | 06:46 | |
| *** ducttape_ has quit IRC | 06:50 | |
| *** simon-AS559 has joined #openstack-operators | 06:52 | |
| *** beddari has quit IRC | 06:52 | |
| *** beddari has joined #openstack-operators | 06:53 | |
| *** beddari has quit IRC | 06:58 | |
| *** simon-AS5591 has joined #openstack-operators | 07:00 | |
| *** beddari has joined #openstack-operators | 07:01 | |
| *** simon-AS559 has quit IRC | 07:02 | |
| *** simon-AS5591 has quit IRC | 07:05 | |
| *** beddari has quit IRC | 07:08 | |
| *** tesseract- has joined #openstack-operators | 07:09 | |
| *** fragatina has joined #openstack-operators | 07:10 | |
| *** beddari has joined #openstack-operators | 07:11 | |
| *** liverpooler has joined #openstack-operators | 07:15 | |
| *** beddari has quit IRC | 07:18 | |
| *** nibalizer has quit IRC | 07:18 | |
| *** fragatina has quit IRC | 07:18 | |
| *** fragatina has joined #openstack-operators | 07:19 | |
| *** simon-AS559 has joined #openstack-operators | 07:19 | |
| *** beddari has joined #openstack-operators | 07:22 | |
| *** nibalizer has joined #openstack-operators | 07:26 | |
| *** beddari has quit IRC | 07:38 | |
| *** beddari has joined #openstack-operators | 07:42 | |
| *** zeih has joined #openstack-operators | 07:47 | |
| *** ducttape_ has joined #openstack-operators | 07:47 | |
| *** beddari has quit IRC | 07:48 | |
| *** zeih has quit IRC | 07:49 | |
| *** ducttape_ has quit IRC | 07:51 | |
| *** admin0 has joined #openstack-operators | 07:51 | |
| *** admin0 has quit IRC | 07:52 | |
| *** chlong has quit IRC | 07:52 | |
| *** beddari has joined #openstack-operators | 07:52 | |
| *** zeih has joined #openstack-operators | 07:55 | |
| *** admin0 has joined #openstack-operators | 07:56 | |
| *** zeih has quit IRC | 08:01 | |
| *** beddari has quit IRC | 08:02 | |
| *** beddari has joined #openstack-operators | 08:02 | |
| *** zeih has joined #openstack-operators | 08:04 | |
| *** chlong has joined #openstack-operators | 08:04 | |
| *** beddari has quit IRC | 08:13 | |
| *** beddari_ has joined #openstack-operators | 08:13 | |
| *** beddari_ is now known as beddari | 08:13 | |
| *** dalees has quit IRC | 08:13 | |
| *** derekh has joined #openstack-operators | 08:15 | |
| *** beddari has quit IRC | 08:18 | |
| *** rarcea has joined #openstack-operators | 08:21 | |
| *** beddari has joined #openstack-operators | 08:22 | |
| *** toMeloos has joined #openstack-operators | 08:25 | |
| *** beddari has quit IRC | 08:28 | |
| *** beddari has joined #openstack-operators | 08:32 | |
| *** beddari has quit IRC | 08:38 | |
| *** dalees has joined #openstack-operators | 08:39 | |
| *** makowals has joined #openstack-operators | 08:40 | |
| *** makowals has quit IRC | 08:40 | |
| *** makowals has joined #openstack-operators | 08:42 | |
| *** beddari has joined #openstack-operators | 08:42 | |
| *** makowals has quit IRC | 08:43 | |
| *** makowals has joined #openstack-operators | 08:43 | |
| *** electrofelix has joined #openstack-operators | 08:44 | |
| *** makowals has quit IRC | 08:46 | |
| *** ducttape_ has joined #openstack-operators | 08:48 | |
| *** beddari has quit IRC | 08:48 | |
| *** ducttape_ has quit IRC | 08:52 | |
| *** beddari has joined #openstack-operators | 08:52 | |
| *** beddari has quit IRC | 08:58 | |
| *** beddari has joined #openstack-operators | 09:01 | |
| *** beddari has quit IRC | 09:11 | |
| *** beddari has joined #openstack-operators | 09:13 | |
| *** beddari has quit IRC | 09:18 | |
| *** makowals has joined #openstack-operators | 09:22 | |
| *** beddari has joined #openstack-operators | 09:22 | |
| *** beddari has quit IRC | 09:28 | |
| *** beddari has joined #openstack-operators | 09:32 | |
| *** beddari has quit IRC | 09:42 | |
| *** beddari has joined #openstack-operators | 09:42 | |
| *** admin0 has quit IRC | 09:45 | |
| *** arif-ali has quit IRC | 09:48 | |
| *** beddari has quit IRC | 09:48 | |
| *** ducttape_ has joined #openstack-operators | 09:48 | |
| *** arif-ali has joined #openstack-operators | 09:50 | |
| *** ducttape_ has quit IRC | 09:53 | |
| *** beddari has joined #openstack-operators | 09:53 | |
| *** beddari has quit IRC | 10:02 | |
| *** beddari has joined #openstack-operators | 10:03 | |
| *** beddari has quit IRC | 10:08 | |
| *** beddari has joined #openstack-operators | 10:11 | |
| *** admin0 has joined #openstack-operators | 10:20 | |
| *** beddari has quit IRC | 10:21 | |
| *** beddari has joined #openstack-operators | 10:22 | |
| *** beddari has quit IRC | 10:28 | |
| *** beddari has joined #openstack-operators | 10:33 | |
| *** zeih has quit IRC | 10:34 | |
| *** piet has joined #openstack-operators | 10:36 | |
| *** dims has joined #openstack-operators | 10:38 | |
| *** beddari has quit IRC | 10:38 | |
| *** beddari has joined #openstack-operators | 10:43 | |
| *** beddari has quit IRC | 10:48 | |
| *** ducttape_ has joined #openstack-operators | 10:49 | |
| *** zeih has joined #openstack-operators | 10:50 | |
| *** beddari has joined #openstack-operators | 10:53 | |
| *** ducttape_ has quit IRC | 10:54 | |
| *** jsheeren has joined #openstack-operators | 10:56 | |
| *** jsheeren has quit IRC | 10:58 | |
| *** beddari has quit IRC | 10:58 | |
| *** beddari has joined #openstack-operators | 11:03 | |
| *** admin0 has quit IRC | 11:03 | |
| *** jsheeren has joined #openstack-operators | 11:05 | |
| *** beddari has quit IRC | 11:12 | |
| *** beddari_ has joined #openstack-operators | 11:13 | |
| *** beddari_ is now known as beddari | 11:13 | |
| *** admin0 has joined #openstack-operators | 11:15 | |
| *** admin0 has quit IRC | 11:16 | |
| *** cdelatte has joined #openstack-operators | 11:17 | |
| *** beddari has quit IRC | 11:18 | |
| *** piet has quit IRC | 11:18 | |
| *** zeih has quit IRC | 11:23 | |
| *** beddari has joined #openstack-operators | 11:23 | |
| *** zeih has joined #openstack-operators | 11:27 | |
| *** beddari has quit IRC | 11:28 | |
| *** zeih has quit IRC | 11:32 | |
| *** beddari has joined #openstack-operators | 11:33 | |
| *** beddari has quit IRC | 11:38 | |
| *** beddari has joined #openstack-operators | 11:43 | |
| *** admin0 has joined #openstack-operators | 11:43 | |
| *** beddari has quit IRC | 11:48 | |
| *** ducttape_ has joined #openstack-operators | 11:50 | |
| *** beddari has joined #openstack-operators | 11:53 | |
| *** zeih has joined #openstack-operators | 11:54 | |
| *** ducttape_ has quit IRC | 11:55 | |
| *** admin0 has quit IRC | 11:56 | |
| *** beddari has quit IRC | 11:58 | |
| *** zeih has quit IRC | 12:01 | |
| *** beddari has joined #openstack-operators | 12:03 | |
| *** admin0 has joined #openstack-operators | 12:03 | |
| *** zeih has joined #openstack-operators | 12:08 | |
| *** beddari has quit IRC | 12:08 | |
| *** ducttape_ has joined #openstack-operators | 12:09 | |
| *** chlong has quit IRC | 12:12 | |
| *** beddari has joined #openstack-operators | 12:13 | |
| *** openstackgerrit has quit IRC | 12:19 | |
| *** openstackgerrit has joined #openstack-operators | 12:19 | |
| *** beddari has quit IRC | 12:22 | |
| *** beddari has joined #openstack-operators | 12:23 | |
| *** ducttape_ has quit IRC | 12:27 | |
| *** zeih has quit IRC | 12:28 | |
| *** beddari has quit IRC | 12:28 | |
| *** zeih has joined #openstack-operators | 12:32 | |
| *** beddari has joined #openstack-operators | 12:33 | |
| dewanee_ | hello everyone | 12:34 |
|---|---|---|
| dewanee_ | anyone running liberty with rbd backed volumes? | 12:34 |
| dewanee_ | having issues with 13:46 -!- dmsimard|pto is now known as dmsimard | 12:35 |
| dewanee_ | 15:34 -!- beddari_ is now known as beddari | 12:35 |
| dewanee_ | 20:03 -!- beddari_ is now known as beddari | 12:35 |
| dewanee_ | 21:04 -!- beddari_ is now known as beddari | 12:35 |
| *** zengine has joined #openstack-operators | 12:35 | |
| dewanee_ | oops | 12:35 |
| dewanee_ | having issues with live_migration_progress_timeout | 12:35 |
| mnaser | whats the actual isue dewanee_ | 12:36 |
| *** beddari has quit IRC | 12:38 | |
| *** jsheeren has left #openstack-operators | 12:41 | |
| *** beddari has joined #openstack-operators | 12:43 | |
| *** chlong has joined #openstack-operators | 12:43 | |
| *** admin0_ has joined #openstack-operators | 12:45 | |
| *** admin0 has quit IRC | 12:47 | |
| *** admin0_ is now known as admin0 | 12:47 | |
| *** beddari has quit IRC | 12:48 | |
| *** beddari has joined #openstack-operators | 12:52 | |
| *** mdavidson has quit IRC | 12:55 | |
| dewanee_ | apparently my migration attempts get aborted | 12:56 |
| dewanee_ | because of this timeout | 12:56 |
| dewanee_ | despite the migration actually progressing | 12:56 |
| *** beddari has quit IRC | 12:58 | |
| *** admin0 has quit IRC | 13:00 | |
| *** zengine has quit IRC | 13:01 | |
| *** beddari has joined #openstack-operators | 13:03 | |
| *** julim has joined #openstack-operators | 13:05 | |
| *** admin0 has joined #openstack-operators | 13:09 | |
| *** zengine has joined #openstack-operators | 13:09 | |
| *** ducttape_ has joined #openstack-operators | 13:14 | |
| *** ducttape_ has quit IRC | 13:15 | |
| *** beddari has quit IRC | 13:18 | |
| *** VW has joined #openstack-operators | 13:21 | |
| *** VW has quit IRC | 13:21 | |
| *** VW has joined #openstack-operators | 13:21 | |
| *** beddari has joined #openstack-operators | 13:23 | |
| *** beddari has quit IRC | 13:28 | |
| *** armax has joined #openstack-operators | 13:28 | |
| *** admin0 has quit IRC | 13:32 | |
| *** beddari has joined #openstack-operators | 13:33 | |
| *** admin0 has joined #openstack-operators | 13:36 | |
| *** beddari has quit IRC | 13:38 | |
| *** toMeloos has quit IRC | 13:38 | |
| *** snecklifter has joined #openstack-operators | 13:39 | |
| *** beddari has joined #openstack-operators | 13:43 | |
| *** zeih has quit IRC | 13:44 | |
| *** zeih has joined #openstack-operators | 13:46 | |
| *** beddari has quit IRC | 13:48 | |
| *** ducttape_ has joined #openstack-operators | 13:50 | |
| *** dminer has joined #openstack-operators | 13:50 | |
| *** zeih has quit IRC | 13:51 | |
| *** VW has quit IRC | 13:51 | |
| *** zeih has joined #openstack-operators | 13:51 | |
| *** beddari has joined #openstack-operators | 13:53 | |
| *** beddari has quit IRC | 13:58 | |
| *** britthou_ has joined #openstack-operators | 13:59 | |
| *** britthouser has quit IRC | 13:59 | |
| *** dminer has quit IRC | 14:02 | |
| *** beddari has joined #openstack-operators | 14:03 | |
| *** beddari has quit IRC | 14:08 | |
| *** admin0 has quit IRC | 14:12 | |
| *** beddari has joined #openstack-operators | 14:13 | |
| *** permalac has quit IRC | 14:14 | |
| *** admin0 has joined #openstack-operators | 14:17 | |
| *** beddari has quit IRC | 14:18 | |
| *** admin0 has quit IRC | 14:18 | |
| *** mriedem has joined #openstack-operators | 14:20 | |
| *** piet has joined #openstack-operators | 14:21 | |
| *** dminer has joined #openstack-operators | 14:21 | |
| *** pilgrimstack1 has joined #openstack-operators | 14:23 | |
| *** beddari has joined #openstack-operators | 14:23 | |
| *** pilgrimstack has quit IRC | 14:24 | |
| *** admin0 has joined #openstack-operators | 14:26 | |
| *** beddari has quit IRC | 14:29 | |
| *** emccormick has joined #openstack-operators | 14:33 | |
| *** beddari has joined #openstack-operators | 14:33 | |
| *** zhangjn has quit IRC | 14:36 | |
| *** zhangjn has joined #openstack-operators | 14:36 | |
| *** rarcea has quit IRC | 14:37 | |
| *** beddari has quit IRC | 14:38 | |
| *** beddari has joined #openstack-operators | 14:43 | |
| *** jamesdenton has joined #openstack-operators | 14:46 | |
| *** beddari has quit IRC | 14:48 | |
| *** VW has joined #openstack-operators | 14:50 | |
| *** piet has quit IRC | 14:51 | |
| *** pilgrimstack1 has quit IRC | 14:53 | |
| *** beddari has joined #openstack-operators | 14:53 | |
| *** pilgrimstack has joined #openstack-operators | 14:54 | |
| *** beddari has quit IRC | 14:58 | |
| *** piet has joined #openstack-operators | 15:00 | |
| *** kstev has joined #openstack-operators | 15:01 | |
| *** beddari has joined #openstack-operators | 15:03 | |
| *** rarcea has joined #openstack-operators | 15:05 | |
| *** makowals has quit IRC | 15:07 | |
| *** beddari has quit IRC | 15:08 | |
| *** beddari has joined #openstack-operators | 15:13 | |
| *** zengine has quit IRC | 15:14 | |
| *** hieulq_ has joined #openstack-operators | 15:17 | |
| *** beddari has quit IRC | 15:18 | |
| *** beddari has joined #openstack-operators | 15:23 | |
| clayton | jlk: sadly I got ops meetup covered, but not barcelona | 15:23 |
| *** pilgrimstack has quit IRC | 15:26 | |
| *** pilgrimstack has joined #openstack-operators | 15:26 | |
| *** beddari has quit IRC | 15:28 | |
| *** saneax is now known as saneax_AFK | 15:30 | |
| *** _ducttape_ has joined #openstack-operators | 15:30 | |
| *** ducttape_ has quit IRC | 15:31 | |
| *** beddari has joined #openstack-operators | 15:33 | |
| *** openstackgerrit has quit IRC | 15:33 | |
| *** openstackgerrit has joined #openstack-operators | 15:34 | |
| *** admin0 has quit IRC | 15:35 | |
| *** zeih has quit IRC | 15:35 | |
| *** harlowja has joined #openstack-operators | 15:40 | |
| *** beddari has quit IRC | 15:43 | |
| *** beddari_ has joined #openstack-operators | 15:43 | |
| *** beddari_ is now known as beddari | 15:43 | |
| *** gyee has joined #openstack-operators | 15:45 | |
| *** _ducttape_ has quit IRC | 15:47 | |
| *** ducttape_ has joined #openstack-operators | 15:48 | |
| *** beddari has quit IRC | 15:48 | |
| *** beddari has joined #openstack-operators | 15:52 | |
| *** simon-AS559 has quit IRC | 15:53 | |
| *** beddari has quit IRC | 15:58 | |
| *** beddari has joined #openstack-operators | 16:02 | |
| *** beddari has quit IRC | 16:08 | |
| *** beddari has joined #openstack-operators | 16:13 | |
| *** doffm_ has joined #openstack-operators | 16:15 | |
| *** beddari has quit IRC | 16:18 | |
| *** beddari has joined #openstack-operators | 16:23 | |
| *** Apoorva has joined #openstack-operators | 16:25 | |
| *** beddari has quit IRC | 16:28 | |
| *** Apoorva_ has joined #openstack-operators | 16:29 | |
| *** rcernin has quit IRC | 16:31 | |
| *** Apoorva has quit IRC | 16:33 | |
| *** tesseract- has quit IRC | 16:33 | |
| *** beddari has joined #openstack-operators | 16:33 | |
| *** pcaruana has quit IRC | 16:33 | |
| *** Apoorva_ has quit IRC | 16:38 | |
| *** beddari has quit IRC | 16:38 | |
| *** Apoorva has joined #openstack-operators | 16:38 | |
| *** beddari has joined #openstack-operators | 16:43 | |
| *** beddari has quit IRC | 16:48 | |
| *** beddari has joined #openstack-operators | 16:51 | |
| *** admin0 has joined #openstack-operators | 16:55 | |
| *** beddari has quit IRC | 16:58 | |
| dansmith | klindgren__: a backport request from compute to conductor is just like any other call | 16:59 |
| dansmith | klindgren__: so the compute sets up a temporary reply queue, then sends a message to the conductor topic, | 16:59 |
| dansmith | which is picked up by any conductor, handled, and replied | 16:59 |
| klindgren__ | for each message | 16:59 |
| klindgren__ | ? | 16:59 |
| dansmith | the above happens for any rpc call | 17:00 |
| dansmith | the automatic backporting operation happens for objects incidental to a call coming into a compute | 17:00 |
| klindgren__ | I thought the fanout queues for normal operation were pre-established. | 17:00 |
| dansmith | so when a compute receives a blob of data too new for it, it makes a call to conductor first to backport it | 17:00 |
| klindgren__ | anyway - reason for asking. We are running in a partial upgrade mode | 17:00 |
| klindgren__ | with control plane at one level and compute at another | 17:01 |
| klindgren__ | n-1 | 17:01 |
| klindgren__ | version | 17:01 |
| dansmith | not sure what fanout has to do with any of this | 17:01 |
| * dansmith nods | 17:01 | |
| klindgren__ | this works fine | 17:01 |
| klindgren__ | BUT | 17:01 |
| klindgren__ | rabbitmq has been dieing about every 24-48hours | 17:01 |
| klindgren__ | sometimes every 2 hours | 17:01 |
| klindgren__ | because the mgmt_db is exploding | 17:01 |
| klindgren__ | some bugs around rabbitmq seem to indicate tht if you have a farily high message volume when an exchangecreates lots of temporrary queues and removes them | 17:02 |
| klindgren__ | that this can cause the statistcis tracking in the mgmtdb to grow out of control | 17:02 |
| *** beddari has joined #openstack-operators | 17:03 | |
| *** Apoorva_ has joined #openstack-operators | 17:03 | |
| dansmith | okay, so like I said, this is not really different under upgrade vs. normal.. rpc calls all behave like this. But, there are more rpc calls happening when you're in that state because of the backports | 17:03 |
| dansmith | so you could hit that issue with just "more traffic" or "more traffic because you're in the middle of an upgrade" | 17:03 |
| *** piet has quit IRC | 17:03 | |
| klindgren__ | we upgraded everything to the same level and so far the rabbitmq mgmt_db hasn't been growing like a weed | 17:03 |
| dansmith | so you're asking for post-mortem/next-time yes? | 17:04 |
| *** piet has joined #openstack-operators | 17:04 | |
| klindgren__ | kk - so thats was the thought behind if a temp queue was created per message, in which case we aresignifcantly increasing the number of temp queues being created | 17:04 |
| klindgren__ | working on multiple front | 17:04 |
| dansmith | yep | 17:04 |
| klindgren__ | we have a few more clouds to upgrade | 17:05 |
| *** maishsk has joined #openstack-operators | 17:05 | |
| dansmith | so definitely a bunch of extra calling going on, therefore more temporary queues | 17:05 |
| klindgren__ | thanks - https://github.com/rabbitmq/rabbitmq-management/issues/185 | 17:05 |
| klindgren__ | we are actually running an earlier version (3.4.0) to that bug, however - very similar symptoms | 17:06 |
| dansmith | okay | 17:06 |
| dansmith | yeah, interesting | 17:06 |
| *** Apoorva has quit IRC | 17:07 | |
| klindgren__ | so yea - code works. Just uncovering issues in rabbitmq :-/ | 17:07 |
| dansmith | yeah, so partial-yay :) | 17:07 |
| dansmith | is this the first time you've tried to do that sort of mixed-version upgrade? | 17:07 |
| klindgren__ | kinda of | 17:08 |
| *** beddari has quit IRC | 17:08 | |
| klindgren__ | longest duration | 17:08 |
| dansmith | cool | 17:09 |
| *** permalac has joined #openstack-operators | 17:09 | |
| klindgren__ | It was going fine for about a week or two - then went terribly | 17:09 |
| klindgren__ | with last night columanting in rabbitmq chewing 40+GB of memory in 2 hours | 17:09 |
| dansmith | ouch | 17:09 |
| klindgren__ | and when a node in a rabbitmq cluster hits the high mem mark - the whole cluster stopps accepting messages and new connections | 17:10 |
| *** derekh has quit IRC | 17:10 | |
| klindgren__ | So i blew though the cell that it was happening on like a huricane last night | 17:11 |
| klindgren__ | seems stable since | 17:11 |
| klindgren__ | IE 4+ hours onward, mgmt_db is at 250MB | 17:11 |
| dansmith | heh, yeah | 17:11 |
| klindgren__ | thanks for the info's | 17:12 |
| dansmith | so any chance of upgrading rabbit before doing another cloud to see if that fix addresses the problem? | 17:12 |
| dansmith | no problem | 17:12 |
| *** beddari has joined #openstack-operators | 17:13 | |
| dansmith | klindgren__: ah, looks like harlowja just posted about this to the ML | 17:13 |
| klindgren__ | sadly we have a puppet dependecy chain issue going on. So its going to be painful to seperate that out | 17:14 |
| dansmith | ah, bummer | 17:14 |
| klindgren__ | because new rabbitmq deprecates some ssl options that we were using | 17:14 |
| klindgren__ | so we need the new options - othwise agent cant connect to the upgraded nodes | 17:14 |
| dansmith | ah, yeah | 17:14 |
| harlowja | dansmith yup yup | 17:16 |
| harlowja | some kind of weirdo-ness | 17:16 |
| *** rcernin has joined #openstack-operators | 17:16 | |
| harlowja | i just didn't quite understand how backporting would cause this | 17:17 |
| harlowja | its at most 2x more messages? | 17:17 |
| dansmith | harlowja: yeah, not even 2x most likely | 17:17 |
| dansmith | just at most | 17:17 |
| harlowja | right | 17:17 |
| *** VW has quit IRC | 17:18 | |
| *** beddari has quit IRC | 17:18 | |
| harlowja | there is also a way to disable the stats db; i'd like for that to be tried also, but i think klindgren__ might have upgraded all the things (so now we aren't running cross-release) | 17:18 |
| dansmith | which is why I said you probably are close to hitting this normally if that is what pushed you over the edg4e | 17:18 |
| dansmith | yeah | 17:18 |
| harlowja | right | 17:18 |
| *** VW has joined #openstack-operators | 17:18 | |
| * harlowja wishes rabbitmq had a replay ability | 17:19 | |
| harlowja | cause we also ran the oslo messaging simulator tool, but that didn't trigger much of anything | 17:20 |
| harlowja | http://imgur.com/FGxVbgI (an gui capture of the crash viewer tool) | 17:22 |
| harlowja | damn thing keeps on garbing, lol | 17:23 |
| *** beddari has joined #openstack-operators | 17:23 | |
| *** mdavidson has joined #openstack-operators | 17:23 | |
| *** simon-AS559 has joined #openstack-operators | 17:25 | |
| *** beddari has quit IRC | 17:28 | |
| *** simon-AS559 has quit IRC | 17:31 | |
| *** electrofelix has quit IRC | 17:32 | |
| *** beddari has joined #openstack-operators | 17:33 | |
| *** fragatina has quit IRC | 17:34 | |
| harlowja | thing i don't get is what changed to cause this | 17:35 |
| harlowja | one thing i can think of | 17:36 |
| harlowja | is `Kombu: make reply and fanout queues expire instead of auto-delete` change in oslo.messaging | 17:36 |
| harlowja | and maybe these queues now using expiry get tracked by the stats db vs auto-delete ones didn't? | 17:36 |
| klindgren__ | why does it seem to not be terrible now that we finished the upgrade? | 17:37 |
| harlowja | right | 17:37 |
| harlowja | that's also weird | 17:37 |
| *** maishsk has quit IRC | 17:37 | |
| harlowja | so a few things are off here IMHO :-P | 17:37 |
| klindgren__ | mix of auto-delete and expiry? | 17:37 |
| harlowja | possibly | 17:37 |
| *** beddari has quit IRC | 17:38 | |
| *** ducttape_ has quit IRC | 17:42 | |
| *** beddari has joined #openstack-operators | 17:43 | |
| harlowja | 'Part of the problem is that we can go from a warning to a zomg out of memory in under 2 minutes. ' is the weird part | 17:44 |
| harlowja | why did that get shorter than it was previously | 17:44 |
| harlowja | weird | 17:44 |
| *** beddari has quit IRC | 17:49 | |
| klindgren__ | well we notcied that before. We an get a high memory alarm | 17:49 |
| klindgren__ | and then death | 17:49 |
| klindgren__ | or go from no alarm to death | 17:49 |
| *** beddari has joined #openstack-operators | 17:53 | |
| *** zhangjn has quit IRC | 17:54 | |
| *** zhangjn has joined #openstack-operators | 17:54 | |
| *** ducttape_ has joined #openstack-operators | 17:55 | |
| *** beddari has quit IRC | 17:58 | |
| *** makowals has joined #openstack-operators | 18:01 | |
| *** beddari has joined #openstack-operators | 18:01 | |
| *** makowals has quit IRC | 18:02 | |
| clayton | klindgren__: we've also rarely seen problems where the mgmt db ballons and kills rabbit | 18:02 |
| * klindgren__ had thought we got out from under these issues a long time ago | 18:03 | |
| * klindgren__ sighs | 18:03 | |
| clayton | Running 2.5.6 | 18:03 |
| clayton | err, 3.5.6 | 18:03 |
| harlowja | klindgren__ version X+1 will always fix it | 18:03 |
| harlowja | lol | 18:03 |
| *** simon-AS559 has joined #openstack-operators | 18:03 | |
| harlowja | its a things developers swear by | 18:03 |
| clayton | we'd actually like to upgrade to 3.6, iirc there are some nice features in it | 18:03 |
| *** Apoorva_ has quit IRC | 18:05 | |
| *** meghead has joined #openstack-operators | 18:05 | |
| *** Apoorva has joined #openstack-operators | 18:05 | |
| *** meghead has quit IRC | 18:05 | |
| klindgren__ | I remember where 3.3.x was terrible (actual clustering issues) and 3.3.5? finally fixed things. Then came the rabbitmq keepalives and suddenly it was like rabbitmq was not longer problem number one. | 18:07 |
| *** parch has joined #openstack-operators | 18:07 | |
| *** parch is now known as p_arch | 18:08 | |
| *** beddari has quit IRC | 18:08 | |
| harlowja | why is rabbit all over the place in openstack :-P | 18:08 |
| harlowja | lol | 18:08 |
| *** p_arch has quit IRC | 18:08 | |
| harlowja | (tears off the bandaid, lol) | 18:08 |
| *** maishsk has joined #openstack-operators | 18:09 | |
| *** p_arch has joined #openstack-operators | 18:09 | |
| *** beddari has joined #openstack-operators | 18:13 | |
| *** beddari has quit IRC | 18:18 | |
| *** hieulq_ has quit IRC | 18:19 | |
| *** beddari has joined #openstack-operators | 18:23 | |
| *** beddari has quit IRC | 18:28 | |
| *** beddari has joined #openstack-operators | 18:32 | |
| *** beddari has quit IRC | 18:42 | |
| *** beddari_ has joined #openstack-operators | 18:42 | |
| *** beddari_ is now known as beddari | 18:42 | |
| *** beddari has quit IRC | 18:48 | |
| *** beddari has joined #openstack-operators | 18:53 | |
| *** dtrainor has quit IRC | 18:56 | |
| *** dtrainor has joined #openstack-operators | 18:56 | |
| *** piet has quit IRC | 18:56 | |
| *** M00nr41n has quit IRC | 18:58 | |
| *** beddari has quit IRC | 18:58 | |
| *** M00nr41n has joined #openstack-operators | 19:00 | |
| *** piet has joined #openstack-operators | 19:03 | |
| *** beddari has joined #openstack-operators | 19:03 | |
| openstackgerrit | Tyler Britten proposed openstack/osops-tools-contrib: fixes refrence to wrong file https://review.openstack.org/337825 | 19:15 |
| *** beddari has quit IRC | 19:15 | |
| *** julim has quit IRC | 19:19 | |
| *** beddari has joined #openstack-operators | 19:20 | |
| *** piet has quit IRC | 19:23 | |
| *** beddari has quit IRC | 19:25 | |
| *** admin0 has quit IRC | 19:28 | |
| *** beddari has joined #openstack-operators | 19:30 | |
| *** gyee has quit IRC | 19:31 | |
| *** toMeloos has joined #openstack-operators | 19:34 | |
| *** beddari has quit IRC | 19:35 | |
| *** beddari has joined #openstack-operators | 19:40 | |
| *** Zucan has joined #openstack-operators | 19:42 | |
| *** beddari has quit IRC | 19:45 | |
| *** julim has joined #openstack-operators | 19:45 | |
| *** Zucan has quit IRC | 19:47 | |
| *** admin0 has joined #openstack-operators | 19:47 | |
| *** beddari has joined #openstack-operators | 19:50 | |
| *** Zucan has joined #openstack-operators | 19:52 | |
| *** beddari has quit IRC | 19:55 | |
| *** toddnni has joined #openstack-operators | 19:58 | |
| *** beddari has joined #openstack-operators | 20:00 | |
| *** stanchan has joined #openstack-operators | 20:01 | |
| openstackgerrit | Mike Dorman proposed openstack/osops-tools-logging: Improve folding of multiline messages https://review.openstack.org/337844 | 20:01 |
| *** toMeloos has quit IRC | 20:02 | |
| *** beddari has quit IRC | 20:05 | |
| *** Apoorva has quit IRC | 20:07 | |
| *** admin0 has quit IRC | 20:08 | |
| *** beddari has joined #openstack-operators | 20:09 | |
| *** Apoorva has joined #openstack-operators | 20:09 | |
| *** piet has joined #openstack-operators | 20:18 | |
| *** beddari has quit IRC | 20:19 | |
| *** beddari has joined #openstack-operators | 20:19 | |
| *** Zucan has quit IRC | 20:23 | |
| *** fragatina has joined #openstack-operators | 20:25 | |
| *** beddari has quit IRC | 20:25 | |
| *** beddari has joined #openstack-operators | 20:29 | |
| *** beddari has quit IRC | 20:35 | |
| *** beddari has joined #openstack-operators | 20:40 | |
| *** beddari has quit IRC | 20:45 | |
| *** beddari has joined #openstack-operators | 20:50 | |
| *** beddari has quit IRC | 20:56 | |
| *** admin0 has joined #openstack-operators | 21:00 | |
| *** beddari has joined #openstack-operators | 21:00 | |
| *** ducttape_ has quit IRC | 21:03 | |
| *** maishsk has quit IRC | 21:04 | |
| *** beddari has quit IRC | 21:05 | |
| *** admin0 has quit IRC | 21:07 | |
| *** beddari has joined #openstack-operators | 21:10 | |
| *** beddari has quit IRC | 21:15 | |
| *** simon-AS559 has quit IRC | 21:15 | |
| *** ducttape_ has joined #openstack-operators | 21:16 | |
| *** fragatina has quit IRC | 21:19 | |
| *** beddari has joined #openstack-operators | 21:20 | |
| *** beddari has quit IRC | 21:25 | |
| *** beddari has joined #openstack-operators | 21:30 | |
| *** beddari has quit IRC | 21:40 | |
| *** beddari_ has joined #openstack-operators | 21:40 | |
| *** beddari_ is now known as beddari | 21:40 | |
| *** mriedem is now known as mriedem_away | 21:43 | |
| *** beddari has quit IRC | 21:45 | |
| *** beddari has joined #openstack-operators | 21:50 | |
| *** simon-AS559 has joined #openstack-operators | 21:52 | |
| *** beddari has quit IRC | 21:55 | |
| *** britthou_ has quit IRC | 21:56 | |
| *** beddari has joined #openstack-operators | 22:00 | |
| *** Apoorva_ has joined #openstack-operators | 22:03 | |
| *** beddari has quit IRC | 22:05 | |
| *** Apoorva has quit IRC | 22:06 | |
| *** simon-AS559 has quit IRC | 22:10 | |
| *** beddari has joined #openstack-operators | 22:10 | |
| *** jamesdenton has quit IRC | 22:13 | |
| *** beddari has quit IRC | 22:15 | |
| *** Apoorva has joined #openstack-operators | 22:16 | |
| *** Apoorva_ has quit IRC | 22:16 | |
| *** beddari has joined #openstack-operators | 22:20 | |
| *** VW has quit IRC | 22:20 | |
| *** piet has quit IRC | 22:20 | |
| *** piet has joined #openstack-operators | 22:22 | |
| *** beddari has quit IRC | 22:25 | |
| *** purp_too is now known as purp | 22:27 | |
| *** beddari has joined #openstack-operators | 22:27 | |
| *** ducttape_ has quit IRC | 22:30 | |
| *** dminer has quit IRC | 22:33 | |
| *** beddari has quit IRC | 22:40 | |
| *** beddari has joined #openstack-operators | 22:40 | |
| *** piet has quit IRC | 22:43 | |
| *** beddari has quit IRC | 22:45 | |
| *** ggillies_ is now known as ggillies | 22:47 | |
| *** simon-AS559 has joined #openstack-operators | 22:47 | |
| *** Apoorva_ has joined #openstack-operators | 22:47 | |
| *** Apoorva has quit IRC | 22:47 | |
| *** piet has joined #openstack-operators | 22:48 | |
| *** simon-AS5591 has joined #openstack-operators | 22:49 | |
| *** beddari has joined #openstack-operators | 22:50 | |
| *** simon-AS559 has quit IRC | 22:52 | |
| *** beddari has quit IRC | 22:55 | |
| *** rarcea has quit IRC | 22:56 | |
| *** beddari has joined #openstack-operators | 23:00 | |
| *** beddari has quit IRC | 23:05 | |
| *** beddari has joined #openstack-operators | 23:10 | |
| *** beddari has quit IRC | 23:15 | |
| *** saneax_AFK is now known as saneax | 23:16 | |
| *** piet has quit IRC | 23:16 | |
| *** rcernin has quit IRC | 23:18 | |
| *** kelv has quit IRC | 23:20 | |
| *** beddari has joined #openstack-operators | 23:20 | |
| *** hieulq_ has joined #openstack-operators | 23:20 | |
| *** Apoorva has joined #openstack-operators | 23:24 | |
| *** beddari has quit IRC | 23:25 | |
| *** Apoorva_ has quit IRC | 23:26 | |
| *** simon-AS5591 has quit IRC | 23:28 | |
| *** Apoorva has quit IRC | 23:29 | |
| *** Apoorva has joined #openstack-operators | 23:29 | |
| *** beddari has joined #openstack-operators | 23:29 | |
| *** breton has quit IRC | 23:32 | |
| *** permalac has quit IRC | 23:35 | |
| *** beddari has quit IRC | 23:35 | |
| *** beddari has joined #openstack-operators | 23:40 | |
| *** breton has joined #openstack-operators | 23:45 | |
| *** beddari has quit IRC | 23:45 | |
| *** ducttape_ has joined #openstack-operators | 23:46 | |
| *** beddari has joined #openstack-operators | 23:50 | |
| *** beddari has quit IRC | 23:55 | |
| *** beddari has joined #openstack-operators | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!