| *** vijaykc4 has quit IRC | 00:17 | |
| *** vijaykc4 has joined #openstack-operators | 00:17 | |
| *** vijaykc4 has quit IRC | 00:17 | |
| *** blake has joined #openstack-operators | 00:44 | |
| *** iranzo has joined #openstack-operators | 00:46 | |
| *** jamesmcarthur has joined #openstack-operators | 00:49 | |
| *** jamesmcarthur has quit IRC | 00:57 | |
| *** jamesmcarthur has joined #openstack-operators | 00:58 | |
| *** chyka has joined #openstack-operators | 01:01 | |
| *** chyka has quit IRC | 01:05 | |
| *** fragatina has quit IRC | 01:08 | |
| *** jamesmcarthur has quit IRC | 01:08 | |
| *** AlexeyAbashkin has joined #openstack-operators | 01:08 | |
| *** jamesmcarthur has joined #openstack-operators | 01:09 | |
| *** fragatina has joined #openstack-operators | 01:10 | |
| *** blake has quit IRC | 01:12 | |
| *** AlexeyAbashkin has quit IRC | 01:13 | |
| *** jamesmcarthur has quit IRC | 01:13 | |
| *** fragatina has quit IRC | 01:14 | |
| *** jamesmcarthur has joined #openstack-operators | 01:18 | |
| *** fragatina has joined #openstack-operators | 01:20 | |
| *** fragatina has quit IRC | 01:25 | |
| *** jamesmcarthur has quit IRC | 01:27 | |
| *** jamesmcarthur has joined #openstack-operators | 01:35 | |
| *** blake has joined #openstack-operators | 01:36 | |
| *** jamesmcarthur has quit IRC | 01:38 | |
| *** jamesmcarthur has joined #openstack-operators | 01:43 | |
| *** jamesmcarthur has quit IRC | 01:54 | |
| *** jamesmcarthur has joined #openstack-operators | 01:56 | |
| *** jamesmcarthur has quit IRC | 02:06 | |
| *** fragatina has joined #openstack-operators | 02:06 | |
| *** liverpooler has joined #openstack-operators | 02:08 | |
| *** blake has quit IRC | 02:18 | |
| *** fragatina has quit IRC | 02:25 | |
| *** VW has joined #openstack-operators | 02:27 | |
| *** fragatina has joined #openstack-operators | 03:20 | |
| *** VW_ has joined #openstack-operators | 03:21 | |
| *** VW has quit IRC | 03:23 | |
| *** harlowja has quit IRC | 03:24 | |
| *** purp has quit IRC | 03:49 | |
| *** zeus has quit IRC | 03:49 | |
| *** bollig_ has joined #openstack-operators | 03:49 | |
| *** bollig has quit IRC | 03:49 | |
| *** Mantorok has quit IRC | 03:49 | |
| *** bollig_ is now known as bollig | 03:49 | |
| *** Mantorok has joined #openstack-operators | 03:50 | |
| *** purp has joined #openstack-operators | 03:50 | |
| *** zeus has joined #openstack-operators | 03:51 | |
| *** zeus is now known as Guest88902 | 03:51 | |
| *** harlowja has joined #openstack-operators | 03:59 | |
| *** liverpooler has quit IRC | 04:00 | |
| *** germs has quit IRC | 04:04 | |
| *** germs has joined #openstack-operators | 04:04 | |
| *** germs has quit IRC | 04:04 | |
| *** fragatina has quit IRC | 04:05 | |
| *** germs has joined #openstack-operators | 04:05 | |
| *** germs has quit IRC | 04:05 | |
| *** germs has joined #openstack-operators | 04:05 | |
| *** jamesmcarthur has joined #openstack-operators | 04:07 | |
| *** germs has quit IRC | 04:09 | |
| *** AlexeyAbashkin has joined #openstack-operators | 04:09 | |
| *** jamesmcarthur has quit IRC | 04:11 | |
| *** sticker has quit IRC | 04:12 | |
| *** AlexeyAbashkin has quit IRC | 04:14 | |
| *** VW_ has quit IRC | 04:16 | |
| *** negronjl has quit IRC | 04:26 | |
| *** negronjl has joined #openstack-operators | 04:29 | |
| *** chyka has joined #openstack-operators | 04:37 | |
| *** chyka has quit IRC | 04:41 | |
| *** rcernin_ has joined #openstack-operators | 04:59 | |
| *** rcernin has quit IRC | 05:00 | |
| *** vijaykc4 has joined #openstack-operators | 05:21 | |
| *** vijaykc4 has quit IRC | 05:24 | |
| *** belmoreira has joined #openstack-operators | 05:30 | |
| *** vijaykc4 has joined #openstack-operators | 05:33 | |
| *** admin0 has joined #openstack-operators | 05:42 | |
| *** dmibrid_ has quit IRC | 05:45 | |
| *** simon-AS559 has joined #openstack-operators | 05:56 | |
| *** germs has joined #openstack-operators | 06:05 | |
| *** germs has quit IRC | 06:05 | |
| *** germs has joined #openstack-operators | 06:05 | |
| *** harlowja has quit IRC | 06:06 | |
| *** germs has quit IRC | 06:10 | |
| *** vijaykc4 has quit IRC | 06:10 | |
| *** vijaykc4 has joined #openstack-operators | 06:14 | |
| *** vijaykc4 has quit IRC | 06:14 | |
| *** rcernin_ has quit IRC | 06:14 | |
| *** rcernin has joined #openstack-operators | 06:17 | |
| *** dbecker has quit IRC | 06:30 | |
| *** arnewiebalck_ has quit IRC | 06:32 | |
| *** arnewiebalck has joined #openstack-operators | 06:38 | |
| *** d0ugal_ has joined #openstack-operators | 06:39 | |
| *** vijaykc4 has joined #openstack-operators | 06:42 | |
| *** d0ugal has quit IRC | 06:42 | |
| *** dbecker has joined #openstack-operators | 06:45 | |
| *** simon-AS559 has quit IRC | 06:54 | |
| *** rcernin has quit IRC | 07:03 | |
| *** blake has joined #openstack-operators | 07:04 | |
| *** tesseract has joined #openstack-operators | 07:22 | |
| *** vijaykc4 has quit IRC | 07:31 | |
| *** vijaykc4 has joined #openstack-operators | 07:34 | |
| *** paken has joined #openstack-operators | 07:39 | |
| *** blake has quit IRC | 07:40 | |
| *** vijaykc4 has quit IRC | 07:46 | |
| *** electrofelix has joined #openstack-operators | 07:54 | |
| *** AlexeyAbashkin has joined #openstack-operators | 08:00 | |
| *** germs has joined #openstack-operators | 08:06 | |
| *** germs has quit IRC | 08:06 | |
| *** germs has joined #openstack-operators | 08:06 | |
| *** AlexeyAbashkin has quit IRC | 08:07 | |
| *** AlexeyAbashkin has joined #openstack-operators | 08:10 | |
| *** germs has quit IRC | 08:11 | |
| *** chyka has joined #openstack-operators | 08:12 | |
| *** chyka has quit IRC | 08:17 | |
| *** paken_ has joined #openstack-operators | 08:17 | |
| *** paken has quit IRC | 08:19 | |
| *** paken_ is now known as paken | 08:19 | |
| *** vijaykc4 has joined #openstack-operators | 08:23 | |
| *** admin0 has quit IRC | 08:26 | |
| *** derekh has joined #openstack-operators | 08:33 | |
| *** rmart04 has joined #openstack-operators | 08:38 | |
| *** rcernin has joined #openstack-operators | 08:48 | |
| *** paken_ has joined #openstack-operators | 08:49 | |
| *** paken has quit IRC | 08:52 | |
| *** paken_ is now known as paken | 08:52 | |
| *** Alexey_Abashkin has joined #openstack-operators | 08:52 | |
| *** vijaykc4 has quit IRC | 08:52 | |
| *** AlexeyAbashkin has quit IRC | 08:54 | |
| *** Alexey_Abashkin is now known as AlexeyAbashkin | 08:54 | |
| *** AlexeyAbashkin has quit IRC | 08:55 | |
| *** AlexeyAbashkin has joined #openstack-operators | 08:58 | |
| *** gkadam has joined #openstack-operators | 09:09 | |
| *** kukacz_ has joined #openstack-operators | 09:10 | |
| *** gkadam_ has joined #openstack-operators | 09:10 | |
| *** paken has quit IRC | 09:12 | |
| *** gkadam has quit IRC | 09:13 | |
| *** paken has joined #openstack-operators | 09:13 | |
| *** vijaykc4 has joined #openstack-operators | 09:14 | |
| *** vijaykc4 has quit IRC | 09:25 | |
| *** rmart04 has quit IRC | 09:35 | |
| *** tuanla____ has joined #openstack-operators | 09:35 | |
| *** rmart04 has joined #openstack-operators | 09:35 | |
| *** kukacz_ has quit IRC | 09:41 | |
| *** kukacz_ has joined #openstack-operators | 09:42 | |
| *** rmart04_ has joined #openstack-operators | 09:52 | |
| *** rmart04 has quit IRC | 09:52 | |
| *** rmart04_ is now known as rmart04 | 09:52 | |
| *** kukacz_ is now known as kukacz | 09:52 | |
| *** germs has joined #openstack-operators | 10:07 | |
| *** germs has quit IRC | 10:07 | |
| *** germs has joined #openstack-operators | 10:07 | |
| *** aojea has joined #openstack-operators | 10:09 | |
| *** germs has quit IRC | 10:12 | |
| *** aojea has quit IRC | 10:23 | |
| *** vijaykc4 has joined #openstack-operators | 10:27 | |
| kukacz | hi, using Swift as cinder-backup backend. when doing a large full backup (1TB volume), it always fails after exactly 12 hour (43200 sec). any ideas whether that can be a timeout at some level or whatever else? | 10:31 |
|---|---|---|
| *** vijaykc4 has quit IRC | 10:35 | |
| *** rmart04 has quit IRC | 10:36 | |
| *** AlexeyAbashkin has quit IRC | 10:53 | |
| *** rmart04 has joined #openstack-operators | 10:54 | |
| rmart04 | expired token kukacz? | 10:59 |
| kukacz | rmart04: yes, that can be, we've got them set exactly at 12 hours. is extending the global keystone token expiration value a correct approach? | 11:02 |
| *** aojea has joined #openstack-operators | 11:05 | |
| *** paken has quit IRC | 11:08 | |
| *** aojea has quit IRC | 11:10 | |
| *** AlexeyAbashkin has joined #openstack-operators | 11:18 | |
| *** tuanla____ has quit IRC | 11:30 | |
| *** chyka has joined #openstack-operators | 11:34 | |
| *** aojea has joined #openstack-operators | 11:34 | |
| *** belmoreira has quit IRC | 11:37 | |
| *** vijaykc4 has joined #openstack-operators | 11:38 | |
| *** chyka has quit IRC | 11:39 | |
| *** belmoreira has joined #openstack-operators | 11:41 | |
| *** gkadam_ has quit IRC | 11:46 | |
| *** gkadam_ has joined #openstack-operators | 11:47 | |
| *** rcernin has quit IRC | 12:05 | |
| *** germs has joined #openstack-operators | 12:08 | |
| *** germs has quit IRC | 12:08 | |
| *** germs has joined #openstack-operators | 12:08 | |
| *** germs has quit IRC | 12:12 | |
| *** aojea has quit IRC | 12:32 | |
| *** aojea has joined #openstack-operators | 12:33 | |
| *** liverpooler has joined #openstack-operators | 12:38 | |
| rmart04 | kukacz: So there a two approaches, increase the token time out, but with caution over the security rammifications, or use a tool that can re-authenticate, rclone possibly, but I’m not sure if this will work if it is one large file, maybe split the object into chunks and create some type of manifest to re-build when retrieving. Just some thoughts anyway | 12:46 |
| *** electrofelix has quit IRC | 12:46 | |
| rmart04 | two approaches that spring to mind at least! | 12:46 |
| kukacz | thanks. the latter one is not applicable perhaps, as the "client" in this case is cinder itself (cinder-backup) | 12:50 |
| *** aojea has quit IRC | 12:50 | |
| kukacz | maybe there's way how to approach the issue - it's to attempt to speed up the backup process itself - now it runs some 20MB/s, which is very slow | 12:52 |
| kukacz | network is not the constraint here. seems more to be related to compression, hash calculations, chunk splitting, ... on the cinder part | 12:53 |
| *** jamesmcarthur has joined #openstack-operators | 13:08 | |
| *** kstev has joined #openstack-operators | 13:09 | |
| *** fragatina has joined #openstack-operators | 13:09 | |
| *** mriedem has joined #openstack-operators | 13:10 | |
| *** damien_r has quit IRC | 13:12 | |
| *** kstev1 has joined #openstack-operators | 13:12 | |
| rmart04 | kukacz: Ah yes, cinder, sorry. More bandwidth needed :) | 13:12 |
| *** jamesmcarthur has quit IRC | 13:13 | |
| *** kstev has quit IRC | 13:13 | |
| *** belmoreira has quit IRC | 13:13 | |
| *** vijaykc4 has quit IRC | 13:13 | |
| *** aojea has joined #openstack-operators | 13:20 | |
| *** vijaykc4 has joined #openstack-operators | 13:21 | |
| *** vijaykc4 has quit IRC | 13:23 | |
| *** vijaykc4 has joined #openstack-operators | 13:23 | |
| *** vijaykc4 has quit IRC | 13:23 | |
| kukacz | rmart04: it does not seem to be network constrained - it's a 10GbE network with negligible load, have checked that during backup run | 13:29 |
| rmart04 | 20MB/s seems low then! | 13:30 |
| kukacz | yes, it is very low, but not because of the network perhaps | 13:30 |
| *** VW has joined #openstack-operators | 13:31 | |
| kukacz | the CPU executing cinder-backup process is peaking 100% most the time when backup runs | 13:31 |
| rmart04 | You could do some iperf tests between your storage node and the swift proxy to check high level network connectivity. Oh right could be CPU bound then | 13:31 |
| rmart04 | Are you doing any type of encryption in transit? That could load the CPU up a bit | 13:32 |
| kukacz | no, but there's the cinder zlib compression enabled. it seemed to me reasonable to have it, both to save the backup space and network utilization | 13:34 |
| *** belmoreira has joined #openstack-operators | 13:35 | |
| kukacz | maybe I should disable it experimentally, to see and compare the effect | 13:36 |
| *** fragatina has quit IRC | 13:45 | |
| *** fragatina has joined #openstack-operators | 13:46 | |
| *** germs has joined #openstack-operators | 14:09 | |
| *** germs has quit IRC | 14:09 | |
| *** germs has joined #openstack-operators | 14:09 | |
| *** germs has quit IRC | 14:13 | |
| *** damien_r has joined #openstack-operators | 14:17 | |
| *** vijaykc4 has joined #openstack-operators | 14:27 | |
| *** VW has quit IRC | 14:30 | |
| *** fragatina has quit IRC | 14:40 | |
| *** jproulx1 has quit IRC | 14:57 | |
| *** chyka has joined #openstack-operators | 14:58 | |
| *** rmart04 has quit IRC | 15:02 | |
| *** chyka has quit IRC | 15:02 | |
| *** aojea has quit IRC | 15:02 | |
| *** aojea has joined #openstack-operators | 15:05 | |
| *** jproulx has joined #openstack-operators | 15:06 | |
| *** aojea has quit IRC | 15:09 | |
| *** VW has joined #openstack-operators | 15:17 | |
| *** VW_ has joined #openstack-operators | 15:17 | |
| *** vijaykc4 has quit IRC | 15:17 | |
| *** VW_ has quit IRC | 15:20 | |
| *** VW_ has joined #openstack-operators | 15:20 | |
| *** VW has quit IRC | 15:21 | |
| *** VW_ has quit IRC | 15:23 | |
| *** chyka has joined #openstack-operators | 15:33 | |
| *** iranzo has quit IRC | 15:40 | |
| *** beddari has quit IRC | 15:41 | |
| *** damien_r has quit IRC | 15:44 | |
| *** aojea has joined #openstack-operators | 15:46 | |
| *** aojea has quit IRC | 15:51 | |
| notmyname | kukacz: nothing springs to mind for me other than what rmart04 said. check the net connectivity in the swift cluster and make sure it's good | 15:52 |
| kukacz | notmyname: thanks! it is visibly constrained on cinder controller CPU though | 15:53 |
| notmyname | ah ok | 15:53 |
| kukacz | seems there's no way of parallel processing, which could help theoretically | 15:53 |
| smcginnis | kukacz: Which version are you running? | 15:55 |
| kukacz | I've just disabled the compression and started a big job - will see the effect | 15:55 |
| kukacz | smcginnis: ocata | 15:56 |
| smcginnis | kukacz: This may be of interest then: https://review.openstack.org/#/c/518306/ | 15:56 |
| smcginnis | kukacz: I'm not sure, but it sounds like you may be running into the condition that attempts to address. | 15:57 |
| kukacz | smcginnis: hmm, had to read it couple of times. partly it sounds like it might be my case | 15:59 |
| kukacz | I'm just not seeing those database connection drops | 16:00 |
| smcginnis | kukacz: I think that may be one of a few potential side effects of the thread starvation. | 16:00 |
| kukacz | my original (here reported) issue of backup jobs terminating after (exactly) 12 hours seems to match my token expiration time | 16:00 |
| kukacz | on the background though, there is some contention causing the slow progress ... | 16:01 |
| smcginnis | kukacz: Ah, that does sound like it could be it then. (the token expiration) | 16:01 |
| kukacz | smcginnis: aha | 16:01 |
| kukacz | yes, that sounds reasonably | 16:01 |
| *** belmoreira has quit IRC | 16:02 | |
| kukacz | as the server is quite a modern one with fast CPU and the zlib library I believe is fitting the usecase of relatively fast stream compression, I can think there's yet something else which would cause the slow progress and 100% CPU in cinder-backup process | 16:04 |
| smcginnis | That part is very odd. | 16:05 |
| smcginnis | How big is the volume being backed up? | 16:05 |
| kukacz | 1TB | 16:05 |
| smcginnis | Big, but I wouldn't think 12+ hours big. | 16:06 |
| kukacz | I should mention though the volume is full of random data - I did it to test the incremental test effectiveness later | 16:08 |
| kukacz | it might be challenging for the compression library therefore | 16:08 |
| kukacz | random = I've filled it with stream of bytes from openssl | 16:09 |
| *** germs has joined #openstack-operators | 16:09 | |
| *** germs has quit IRC | 16:09 | |
| *** germs has joined #openstack-operators | 16:09 | |
| *** germs has quit IRC | 16:14 | |
| kukacz | smcginnis: thanks for pointing me at that cinder fix. I'll try to apply it and see | 16:18 |
| smcginnis | kukacz: No problem. Not sure if that will address your problem, but would love to hear if it does. | 16:18 |
| kukacz | ok, I'll tell you. next week hopefully | 16:19 |
| kukacz | also, now I'm running the same volume backup with compression disabled. throughput seems to be about 2x compared to the compressed one, CPU utilized by cinder-backup process oscillating around 50% | 16:22 |
| smcginnis | I'm surprised the random data would have that much of an impact on compression, but then again, really random data is hard to compress so the lib might be working a lot harder than normal. | 16:23 |
| smcginnis | Might also be interesting to see if one of the other compression algorithms would make a difference. | 16:24 |
| smcginnis | And by "other", I guess I mean bz2. :) | 16:24 |
| *** Caterpillar has quit IRC | 16:31 | |
| *** jamesmcarthur has joined #openstack-operators | 16:35 | |
| kukacz | smcginnis: I will try bz2 too | 16:36 |
| *** gyee has joined #openstack-operators | 16:49 | |
| *** AlexeyAbashkin has quit IRC | 16:53 | |
| *** derekh has quit IRC | 16:55 | |
| *** immune2iocane has joined #openstack-operators | 16:57 | |
| *** gkadam__ has joined #openstack-operators | 17:09 | |
| *** gkadam has joined #openstack-operators | 17:11 | |
| *** gkadam_ has quit IRC | 17:12 | |
| *** gkadam__ has quit IRC | 17:15 | |
| *** fragatina has joined #openstack-operators | 17:28 | |
| *** aojea has joined #openstack-operators | 17:35 | |
| *** aojea has quit IRC | 17:39 | |
| *** chyka_ has joined #openstack-operators | 17:41 | |
| *** chyka has quit IRC | 17:43 | |
| *** germs has joined #openstack-operators | 17:45 | |
| *** germs has quit IRC | 17:45 | |
| *** germs has joined #openstack-operators | 17:45 | |
| *** germs has quit IRC | 17:45 | |
| *** germs has joined #openstack-operators | 17:46 | |
| *** germs has quit IRC | 17:46 | |
| *** germs has joined #openstack-operators | 17:46 | |
| *** jamesmcarthur has quit IRC | 17:50 | |
| *** gkadam has quit IRC | 18:01 | |
| *** harlowja has joined #openstack-operators | 18:01 | |
| *** AlexeyAbashkin has joined #openstack-operators | 18:18 | |
| *** AlexeyAbashkin has quit IRC | 18:23 | |
| *** tesseract has quit IRC | 18:31 | |
| *** aojea has joined #openstack-operators | 18:33 | |
| *** jamesmcarthur has joined #openstack-operators | 18:49 | |
| *** aojea has quit IRC | 19:01 | |
| *** iranzo has joined #openstack-operators | 19:07 | |
| *** AlexeyAbashkin has joined #openstack-operators | 19:08 | |
| *** aojea has joined #openstack-operators | 19:09 | |
| *** jamesmca_ has joined #openstack-operators | 19:11 | |
| *** AlexeyAbashkin has quit IRC | 19:13 | |
| *** jamesmcarthur has quit IRC | 19:14 | |
| *** iranzo has quit IRC | 19:26 | |
| *** dbecker has quit IRC | 19:31 | |
| *** dbecker has joined #openstack-operators | 19:32 | |
| *** jamesmca_ has quit IRC | 19:52 | |
| *** jamesmcarthur has joined #openstack-operators | 19:54 | |
| *** jamesmcarthur has quit IRC | 19:59 | |
| *** jamesmcarthur has joined #openstack-operators | 20:00 | |
| *** jamesmcarthur has quit IRC | 20:04 | |
| *** v12aml has quit IRC | 20:10 | |
| *** liverpooler has quit IRC | 20:16 | |
| *** aojea has quit IRC | 20:20 | |
| *** aojea has joined #openstack-operators | 20:23 | |
| *** paken has joined #openstack-operators | 20:37 | |
| *** VW has joined #openstack-operators | 20:46 | |
| *** VW has quit IRC | 20:54 | |
| *** jamesmcarthur has joined #openstack-operators | 21:32 | |
| *** paken has quit IRC | 21:39 | |
| *** jamesmcarthur has quit IRC | 21:58 | |
| *** immune2iocane has quit IRC | 22:14 | |
| *** kstev1 has quit IRC | 22:39 | |
| *** chyka has joined #openstack-operators | 22:45 | |
| *** chyka_ has quit IRC | 22:48 | |
| *** aojea has quit IRC | 22:52 | |
| *** aojea has joined #openstack-operators | 22:55 | |
| *** jamesmcarthur has joined #openstack-operators | 23:00 | |
| *** jamesmcarthur has quit IRC | 23:04 | |
| *** AlexeyAbashkin has joined #openstack-operators | 23:08 | |
| *** AlexeyAbashkin has quit IRC | 23:12 | |
| *** aojea has quit IRC | 23:12 | |
| *** jamesmcarthur has joined #openstack-operators | 23:16 | |
| *** chyka_ has joined #openstack-operators | 23:19 | |
| *** jamesmcarthur has quit IRC | 23:23 | |
| *** chyka has quit IRC | 23:23 | |
| *** chyka_ has quit IRC | 23:23 | |
| *** jamesmca_ has joined #openstack-operators | 23:24 | |
| *** mihalis68 has quit IRC | 23:52 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!