Tuesday, 2015-10-13

*** zhill has joined #openstack-swift00:01
openstackgerritMerged openstack/swift: Fix 'swift-ring-builder write_builder' after you remove a device  https://review.openstack.org/23348800:04
*** blmartin_ has joined #openstack-swift00:08
*** blmartin_ has quit IRC00:08
*** hrou has joined #openstack-swift00:09
*** _hrou_ has quit IRC00:11
*** opal has joined #openstack-swift00:12
*** _hrou_ has joined #openstack-swift00:13
*** hrou has quit IRC00:16
*** opal has left #openstack-swift00:18
*** fbo has quit IRC00:18
*** fbo has joined #openstack-swift00:22
*** minwoob has quit IRC00:28
*** DericHorn-HP has joined #openstack-swift00:28
*** garthb has quit IRC00:29
*** gyee has quit IRC00:31
*** m_kazuhiro has joined #openstack-swift00:41
kota_good morning00:45
mattoliveraukota_: morning00:54
kota_mattoliverau: :-)00:55
*** zhill has quit IRC01:00
*** klrmn1 has quit IRC01:01
*** zhill has joined #openstack-swift01:05
*** DericHorn-HP has quit IRC01:08
*** hrou has joined #openstack-swift01:10
*** bkeller has joined #openstack-swift01:13
*** _hrou_ has quit IRC01:13
*** zhill has quit IRC01:33
*** DericHorn-HP has joined #openstack-swift01:36
openstackgerritjanonymous proposed openstack/swift: Python3: do not use im_self/im_func/func_closure  https://review.openstack.org/20306601:45
*** haomaiwang has joined #openstack-swift01:55
*** haomaiwang has quit IRC02:01
*** haomaiwang has joined #openstack-swift02:01
*** DericHorn-HP has quit IRC02:05
*** DericHorn-HP has joined #openstack-swift02:10
*** klrmn1 has joined #openstack-swift02:15
*** links has joined #openstack-swift02:28
*** DericHorn-HP has quit IRC02:34
*** wbhuber has joined #openstack-swift02:37
*** DericHorn-HP has joined #openstack-swift02:39
*** sanchitmalhotra has joined #openstack-swift02:44
*** bill_az has quit IRC02:56
*** haomaiwang has quit IRC03:01
*** wbhuber has quit IRC03:01
*** haomaiwang has joined #openstack-swift03:01
*** hrou has quit IRC03:03
*** sanchitmalhotra1 has joined #openstack-swift03:07
*** silor has joined #openstack-swift03:07
*** trifon has joined #openstack-swift03:08
*** sanchitmalhotra has quit IRC03:09
*** lpabon has joined #openstack-swift03:24
*** Venkatesh has joined #openstack-swift03:28
janonymousRebased merged conflict in jenkins:https://review.openstack.org/#/c/203066/ please review.03:32
janonymousand https://review.openstack.org/#/c/203414/03:33
*** klrmn2 has joined #openstack-swift03:37
*** klrmn1 has quit IRC03:37
*** sasik has joined #openstack-swift03:37
openstackgerritMatthew Oliver proposed openstack/swift: Possible pipeline auditor - WIP  https://review.openstack.org/23392203:38
*** haomaiwang has quit IRC03:41
*** sasik has left #openstack-swift03:42
*** lpabon has quit IRC03:44
*** haomaiwang has joined #openstack-swift03:45
*** klrmn2 has quit IRC03:48
*** 32NAAJR1K has quit IRC03:54
*** trifon has quit IRC03:54
*** links has quit IRC03:58
*** sanchitmalhotra has joined #openstack-swift03:58
openstackgerritMerged openstack/swift: Autodetect systemctl in SAIO and use it on systemd distros  https://review.openstack.org/23338803:59
*** haomaiwang has quit IRC04:01
*** sanchitmalhotra1 has quit IRC04:01
*** 20WAB9N4Y has joined #openstack-swift04:01
*** blmartin_ has joined #openstack-swift04:02
*** blmartin has quit IRC04:03
*** bkeller has quit IRC04:09
*** blmartin_ has quit IRC04:09
*** ppai has joined #openstack-swift04:12
*** sanchitmalhotra1 has joined #openstack-swift04:17
*** sanchitmalhotra has quit IRC04:19
openstackgerritKota Tsuyuzaki proposed openstack/swift: Refactor versioned_writes middleware  https://review.openstack.org/20931504:22
*** early` has quit IRC04:30
*** opal has joined #openstack-swift04:33
*** early has joined #openstack-swift04:36
*** silor1 has joined #openstack-swift04:38
*** silor has quit IRC04:40
*** silor1 is now known as silor04:40
*** opal has left #openstack-swift04:41
*** zhill has joined #openstack-swift04:41
*** haigang has joined #openstack-swift04:41
*** mmedvede has quit IRC04:41
*** mmedvede has joined #openstack-swift04:41
*** dosaboy has quit IRC04:42
*** haigang has quit IRC04:42
*** dosaboy has joined #openstack-swift04:43
*** zhill has quit IRC04:46
*** zhill has joined #openstack-swift04:48
*** trifon has joined #openstack-swift05:00
*** 20WAB9N4Y has quit IRC05:01
*** haomaiwang has joined #openstack-swift05:01
*** sanchitmalhotra has joined #openstack-swift05:04
*** links has joined #openstack-swift05:05
*** sanchitmalhotra1 has quit IRC05:06
*** breitz has quit IRC05:13
*** breitz has joined #openstack-swift05:14
*** DericHorn-HP has quit IRC05:17
*** mahatic has joined #openstack-swift05:19
openstackgerritOpenStack Proposal Bot proposed openstack/swift: Updated from global requirements  https://review.openstack.org/8873605:32
*** breitz has quit IRC05:35
*** breitz has joined #openstack-swift05:36
*** sanchitmalhotra1 has joined #openstack-swift05:45
openstackgerritPrashanth Pai proposed openstack/swift: Make object creation more atomic in Linux  https://review.openstack.org/16224305:47
*** sanchitmalhotra has quit IRC05:47
openstackgerritChristian Schwede proposed openstack/swift: Do not use pre-authenticated requests in staticweb  https://review.openstack.org/22720405:49
*** sanchitmalhotra has joined #openstack-swift05:53
*** sanchitmalhotra1 has quit IRC05:54
*** haomaiwang has quit IRC06:01
*** haomaiwang has joined #openstack-swift06:01
*** silor1 has joined #openstack-swift06:08
openstackgerritChristian Schwede proposed openstack/swift: Do not use pre-authenticated requests in staticweb  https://review.openstack.org/22720406:10
*** silor has quit IRC06:10
*** silor1 is now known as silor06:10
*** haomaiwang has quit IRC06:10
*** haomaiwang has joined #openstack-swift06:13
*** links has quit IRC06:20
*** openstack has joined #openstack-swift06:29
*** Venkatesh has quit IRC06:30
*** m_kazuhiro has quit IRC06:32
*** m_kazuhiro has joined #openstack-swift06:34
*** sanchitmalhotra1 has joined #openstack-swift06:39
*** sanchitmalhotra has quit IRC06:41
*** haomaiwang has quit IRC06:47
*** 18VAAUGO0 has joined #openstack-swift06:48
*** 18VAAUGO0 has quit IRC06:58
*** haomaiwang has joined #openstack-swift07:02
openstackgerritOndřej Nový proposed openstack/swift: swift-init: SIGKILL daemon after kill_wait period.  https://review.openstack.org/22953207:07
*** jmccarthy has quit IRC07:07
*** jmccarthy has joined #openstack-swift07:07
*** rledisez has joined #openstack-swift07:09
openstackgerritOndřej Nový proposed openstack/swift: Script for running unit, func and probe tests at once  https://review.openstack.org/23336807:10
openstackgerritMerged openstack/swift: swift-recon fails with socket.timeout exception  https://review.openstack.org/23361607:12
*** links has joined #openstack-swift07:13
*** peterlisak has joined #openstack-swift07:14
openstackgerritChristian Schwede proposed openstack/swift: Do not use pre-authenticated requests in staticweb  https://review.openstack.org/22720407:17
*** mac_ified has quit IRC07:21
*** Guokang has joined #openstack-swift07:24
*** rminmin has joined #openstack-swift07:26
*** haypo has joined #openstack-swift07:26
*** rminmin has left #openstack-swift07:27
*** rminmin has joined #openstack-swift07:29
* rminmin slaps rminmin around a bit with a large fishbot07:30
* Guokang slaps rminmin around a bit with a large fishbot07:30
rminmin@Guokang hi07:32
openstackgerritKota Tsuyuzaki proposed openstack/swift: Add backend subrequest tests for staticweb  https://review.openstack.org/23401207:32
openstackgerritvenkatesh proposed openstack/swift: remove unused parameters  https://review.openstack.org/23134707:32
*** Guokang has quit IRC07:34
*** rminmin has quit IRC07:37
*** zhill has quit IRC07:38
*** mahatic_ has joined #openstack-swift07:39
*** mahatic has quit IRC07:39
*** rminmin has joined #openstack-swift07:39
*** geaaru has joined #openstack-swift07:47
*** eranrom has joined #openstack-swift07:56
*** haomaiwang has quit IRC08:01
*** haomaiwang has joined #openstack-swift08:01
*** jordanP has joined #openstack-swift08:02
*** jistr has joined #openstack-swift08:03
openstackgerritPeter Lisák proposed openstack/python-swiftclient: swiftclient content-type header  https://review.openstack.org/23135408:05
*** jmccarthy has quit IRC08:07
*** jmccarthy has joined #openstack-swift08:07
*** links has quit IRC08:08
*** ppai has quit IRC08:15
*** silor has quit IRC08:16
*** links has joined #openstack-swift08:17
*** silor has joined #openstack-swift08:19
*** rminmin has quit IRC08:23
*** links has quit IRC08:24
*** ppai has joined #openstack-swift08:29
*** links has joined #openstack-swift08:31
*** openstack has joined #openstack-swift08:48
*** links has quit IRC08:54
*** links has joined #openstack-swift08:54
*** marzif has joined #openstack-swift08:57
*** Venkatesh has joined #openstack-swift08:59
*** haomaiwang has quit IRC09:01
*** haomaiwang has joined #openstack-swift09:01
*** acoles_ is now known as acoles09:11
*** links has quit IRC09:15
openstackgerritEran Rom proposed openstack/swift: Container-Sync to iterate only over synced containers  https://review.openstack.org/20580309:15
*** sanchitmalhotra has joined #openstack-swift09:28
*** sanchitmalhotra1 has quit IRC09:30
openstackgerritAlistair Coles proposed openstack/swift: Moving DLO functionality doc to the middleware code  https://review.openstack.org/21999109:37
*** openstackstatus has joined #openstack-swift09:39
*** ChanServ sets mode: +v openstackstatus09:39
kota_hi, acoles09:43
acoleshi kota_09:43
-openstackstatus- NOTICE: gerrit is undergoing an emergency restart to investigate load issues09:44
*** ChanServ changes topic to "gerrit is undergoing an emergency restart to investigate load issues"09:44
kota_I'm now writing some tests for ResponseBucket on non durable data patch.09:44
kota_and reading your code for that.09:44
kota_and then,09:44
acoleskota_: cool. i think my patch may need a rebase09:44
kota_I'm becoming I'd like to move some code around generating response from controller into ResponseBucket to be readable.09:45
kota_but not sure it would be good for others.09:45
kota_yeah, I know you are still working for addressing clayg's comment.09:46
acolesok so what would you want to move?09:46
kota_that's no matter right now because I'm just in making tests.09:46
kota_I'll point out...09:46
openstackgerritAlistair Coles proposed openstack/swift: Merge branch 'master' into merge-master  https://review.openstack.org/23406509:47
kota_araound from L2271 in current patch set.09:47
kota_I'm seeing at https://review.openstack.org/#/c/215276/23/swift/proxy/controllers/obj.py09:48
openstackgerritAlistair Coles proposed openstack/swift: Merge branch 'master' into feature/crypto  https://review.openstack.org/23406509:48
kota_if we have enough getters in the bucket and it's best09:48
kota_I guess, it would be good if we could get just the response like as "resp = best_bucket.get_response()"09:49
kota_the get_response will wrap backend getters into Response with ECAppIter as well as current controller does.09:50
*** marzif has quit IRC09:50
*** marzif has joined #openstack-swift09:51
kota_Absolutely, I didn't get all about the impact of moving them but it seems good encapsulation09:51
acolesso lines 2271 - 2291 would move to the ECResponseBucket?09:51
kota_almost yes.09:51
kota_It might be ok that only L2291is remaining at there.09:52
kota_if we employ the syntax "resp = best_bucket.get_response()"09:52
kota_actually, I'm not sure, tho :/09:53
acolesbut for bucket to construct the ECAppIter and Response it would need to be passed the eq, fa_length, range_specs etc09:53
acoleskota_: is a good thought but I wonder if we'd end up with passing all those args into bucket^^09:54
kota_acoles: agreed09:55
kota_probably I need more time to think "everything in bucket" or "control on controller" and to write some poc code.09:58
acoleskota_: thanks for working on it. i am adding this topic to etherpad as a suggestion for summit.09:58
kota_acoles: nice :-)09:58
kota_acoles: actually, I think, the response bucket encapsulation is really good because I'd like to use it for my global ec implementation.09:59
kota_acoles: thanks for advice :)09:59
acoleskota_: np.09:59
*** haomaiwang has quit IRC10:01
*** openstackgerrit has quit IRC10:01
*** aix has joined #openstack-swift10:01
*** haomaiwang has joined #openstack-swift10:01
*** openstackgerrit has joined #openstack-swift10:01
*** marzif has quit IRC10:10
*** ntt has quit IRC10:11
*** marzif has joined #openstack-swift10:11
*** eranrom has quit IRC10:14
*** haomaiwang has quit IRC10:29
*** haomaiwang has joined #openstack-swift10:30
*** aix has quit IRC10:30
openstackgerritAlistair Coles proposed openstack/swift: Re-organise ssync tests  https://review.openstack.org/22019810:38
*** aix has joined #openstack-swift10:42
*** mahatic_ has quit IRC10:48
*** mahatic_ has joined #openstack-swift10:51
*** wasmum has quit IRC10:52
openstackgerritMerged openstack/swift: Replace itertools.ifilter with six.moves.filter  https://review.openstack.org/23253910:53
openstackgerritMerged openstack/swift: py3: Replace basestring with six.string_types  https://review.openstack.org/23254010:56
*** haomaiwang has quit IRC11:01
*** haomaiwang has joined #openstack-swift11:01
*** marzif has quit IRC11:01
*** marzif has joined #openstack-swift11:02
*** SkyRocknRoll has joined #openstack-swift11:04
haypowow, my patch serie of the 6 giant python 3 patches was merged. thanks for the reviews!11:09
*** diogogmt has quit IRC11:16
*** ChanServ changes topic to "Priorities: https://wiki.openstack.org/wiki/Swift/PriorityReviews | Review Dashboard: https://goo.gl/eqeGwE | Summary Dashboard: https://goo.gl/jL0byl | Summit planning: https://etherpad.openstack.org/p/tokyo-summit-swift | Logs: http://eavesdrop.openstack.org/irclogs/%23openstack-swift/"11:19
-openstackstatus- NOTICE: Gerrit has been restarted and is responding to normal load again.11:19
*** m_kazuhiro has quit IRC11:23
*** marzif has quit IRC11:29
*** jistr is now known as jistr|biab11:32
*** sanchitmalhotra has quit IRC11:38
acoleshaypo: :)11:42
vinshHey all.  When I change worker settings for a swift service.. I notice more workers running under ps.  When I change concurrency for example with container-replicator I only ever see one running in ps... I take it that the concurrency is threads per process vs MORE processes like the workers setting provides?11:51
vinshI'm trying to tune my EC cluster which is finally up. :)11:51
vinshI notice container-replicator pegged above 100% most of the time.11:51
*** haomaiwang has quit IRC11:52
*** haomaiwang has joined #openstack-swift11:52
vinshShould I see multiple container-replicator processes?11:56
*** silor has joined #openstack-swift11:56
*** cdelatte has quit IRC11:59
*** haomaiwang has quit IRC12:01
*** haomaiwa_ has joined #openstack-swift12:01
*** Venkatesh_ has joined #openstack-swift12:08
*** Venkatesh_ has quit IRC12:11
*** Venkatesh has quit IRC12:11
*** haomaiwa_ has quit IRC12:17
*** haomaiwang has joined #openstack-swift12:17
*** Guest93107 has quit IRC12:18
*** cdelatte has joined #openstack-swift12:18
*** delattec has joined #openstack-swift12:18
*** kei_yama has quit IRC12:20
*** dencaval has joined #openstack-swift12:21
*** ppai has quit IRC12:22
*** delatte has joined #openstack-swift12:22
*** cdelatte has quit IRC12:24
*** haomaiwang has quit IRC12:29
*** mahatic has joined #openstack-swift12:33
*** mahatic_ has quit IRC12:35
*** ppai has joined #openstack-swift12:35
*** delattec has quit IRC12:38
*** hrou has joined #openstack-swift12:46
*** Venkatesh has joined #openstack-swift12:48
openstackgerritAlistair Coles proposed openstack/swift: EC: Avoid conflicts when ssync'ing fragments  https://review.openstack.org/18140712:59
openstackgerritAlistair Coles proposed openstack/swift: Make ECDiskFile report all fragments found on disk  https://review.openstack.org/23112112:59
*** dustins has joined #openstack-swift13:01
*** haomaiwang has joined #openstack-swift13:03
*** jistr|biab is now known as jistr13:03
*** alejandrito has joined #openstack-swift13:09
openstackgerritAlistair Coles proposed openstack/swift: Merge branch 'master' into feature/crypto  https://review.openstack.org/23406513:16
openstackgerritAlistair Coles proposed openstack/swift: Don't ssync data when only a durable is missing  https://review.openstack.org/23268413:22
*** Venkatesh has quit IRC13:25
*** mahatic has quit IRC13:35
*** SkyRocknRoll has quit IRC13:35
*** mahatic has joined #openstack-swift13:38
*** ppai has quit IRC13:39
*** DericHorn-HP has joined #openstack-swift13:42
*** jrichli has joined #openstack-swift13:42
*** petertr7_away is now known as petertr713:43
*** geaaru has quit IRC13:53
openstackgerritRobert Francis proposed openstack/swift: WIP: Symlink implementation. (work in progress)  https://review.openstack.org/23216213:58
*** DericHorn-HP has quit IRC13:59
*** haomaiwang has quit IRC14:01
*** haomaiwa_ has joined #openstack-swift14:01
*** silor1 has joined #openstack-swift14:01
*** tongli has joined #openstack-swift14:02
*** DericHorn-HP has joined #openstack-swift14:03
*** silor has quit IRC14:03
*** silor1 is now known as silor14:03
*** jlhinson has joined #openstack-swift14:05
*** marekkaleta has joined #openstack-swift14:06
*** eranrom has joined #openstack-swift14:07
*** CaioBrentano has joined #openstack-swift14:07
*** jmccarthy has quit IRC14:08
*** jmccarthy has joined #openstack-swift14:08
haypohi. my huge patch serie fixing most obvious Python 3 issues has been merged, but these two minor changes to fix "tox -e py34" were not reviewed yet :-(14:13
haypohttps://review.openstack.org/#/c/217423/ & https://review.openstack.org/#/c/199034/14:13
haypothe second change hacks PyEClib to "port it" to Python 3, just to be able to run unit tests14:13
haypowhat's the plan to upgrade PyEClib to 1.0.8 or 1.0.9?14:13
*** haomaiwa_ has quit IRC14:21
*** haomaiwang has joined #openstack-swift14:21
*** wbhuber has joined #openstack-swift14:26
*** wbhuber has quit IRC14:26
*** wbhuber has joined #openstack-swift14:27
openstackgerritRobert Francis proposed openstack/swift: WIP: Symlink implementation. (work in progress)  https://review.openstack.org/23216214:34
*** diogogmt has joined #openstack-swift14:34
vinshSeeing some impressive write performance to my ec cluster with ssbench.  also seeing really awful read performance14:40
vinshnot sure where i'm going wrong on that part14:40
*** Ailus has joined #openstack-swift14:46
*** Ailus has quit IRC14:46
*** okdas has quit IRC14:47
*** okdas has joined #openstack-swift14:47
*** okdas has quit IRC14:47
*** okdas has joined #openstack-swift14:47
*** Ailus has joined #openstack-swift14:50
*** siva_krishnan has joined #openstack-swift14:50
*** SkyRocknRoll has joined #openstack-swift14:54
*** janonymous_ has joined #openstack-swift14:54
*** minwoob has joined #openstack-swift14:57
*** trifon has quit IRC14:59
*** petertr7 is now known as petertr7_away15:01
*** haomaiwang has quit IRC15:01
*** haomaiwang has joined #openstack-swift15:01
*** marekkaleta has quit IRC15:01
*** petertr7_away is now known as petertr715:02
*** peterlisak has quit IRC15:17
openstackgerritPeter Lisák proposed openstack/swift: node_timeout as float in configs  https://review.openstack.org/23428715:17
*** pchng has joined #openstack-swift15:40
*** nadeem has joined #openstack-swift15:40
*** nadeem has quit IRC15:40
*** nadeem has joined #openstack-swift15:41
*** gyee has joined #openstack-swift15:55
*** haomaiwang has quit IRC16:01
*** haomaiwang has joined #openstack-swift16:01
*** petertr7 is now known as petertr7_away16:01
*** janonymous_ has quit IRC16:06
vinshnotmyname: Those initial EC results I promised: http://paste.openstack.org/show/476160/16:20
vinshRead pref makes me sad. Write is pretty OK.16:20
vinsh2 proxies with account/container on NVME drives. 10gig networking16:20
vinsh4 objects, 36 drives each, 10gig16:20
vinsh10:3 intel ec16:20
*** trifon has joined #openstack-swift16:21
*** garthb has joined #openstack-swift16:26
*** SkyRocknRoll has quit IRC16:27
*** jistr has quit IRC16:31
notmynamegood morning16:36
vinshG'mornin :)16:37
notmynamevinsh: thanks for pasting those. looking now16:37
*** rledisez has quit IRC16:43
notmynamevinsh: looks like your max time is killing your averages. might be some timeouts causing that16:43
*** eariasn has joined #openstack-swift16:44
vinshHow does one combat that?16:45
vinshmore object drives?16:45
eariasnsharding_split or sharding_trie ?16:45
*** jordanP has quit IRC16:49
notmynamevinsh: it depends on where your bottleneck is. a first step is looking in the logs for the reported slow transaction ids16:50
*** petertr7_away is now known as petertr716:53
tdasilvanotmyname: hi, just came across this link, might give you some ideas about community data: http://community.redhat.com/dashboard/16:56
notmynametdasilva: cool. thanks16:57
*** DericHorn-HP has quit IRC17:00
*** haomaiwang has quit IRC17:01
*** haomaiwang has joined #openstack-swift17:01
*** hezhiqiang has joined #openstack-swift17:04
*** zhill has joined #openstack-swift17:06
*** petertr7 is now known as petertr7_away17:08
*** hezhiqiang has quit IRC17:09
*** hezhiqiang has joined #openstack-swift17:12
*** siva_krishnan has quit IRC17:17
*** aix has quit IRC17:22
*** mahatic has quit IRC17:22
*** klrmn1 has joined #openstack-swift17:25
openstackgerritGreg Lange proposed openstack/swift: Adds read only middleware.  https://review.openstack.org/23374917:28
*** klrmn1 has quit IRC17:30
notmynameglange: it looks like you pushed up a rebase, but no actual changes between patch sets 1 and 217:33
claygtorgomatic: you gotta watch out braggin' too much about swob - or it'll get copied into oslo and then someone will tell us we should use the "standard" one17:38
*** petertr7_away is now known as petertr717:41
glangenotmyname: yeah, I messed up17:42
openstackgerritCarlos Cavanna proposed openstack/swift: bug/1270114. Missing documentation for Bulk Middleware.  https://review.openstack.org/23268817:42
*** bill_az has joined #openstack-swift17:46
openstackgerritGreg Lange proposed openstack/swift: Adds read only middleware.  https://review.openstack.org/23436417:46
claygglange: I think *gerrit* messed up17:47
openstackgerritGreg Lange proposed openstack/swift: Adds read only middleware.  https://review.openstack.org/23374917:49
claygglange: just keep going man!17:50
glangeI think patch 3 is right ?17:50
openstackgerritRobert Francis proposed openstack/swift: WIP: Symlink implementation. (work in progress)  https://review.openstack.org/23216217:52
claygglange: push two more to be sure17:53
*** wbhuber has quit IRC17:56
*** wbhuber has joined #openstack-swift17:57
*** haomaiwang has quit IRC18:01
*** haomaiwang has joined #openstack-swift18:01
*** wbhuber has quit IRC18:01
*** eariasn has quit IRC18:02
*** wbhuber has joined #openstack-swift18:03
*** siva_krishnan has joined #openstack-swift18:06
*** jmccarthy has quit IRC18:07
*** jkugel has joined #openstack-swift18:08
*** hezhiqiang has quit IRC18:11
*** acoles is now known as acoles_18:14
*** gustavo_ has quit IRC18:14
ctennisvinsh my hunch is that 1000 workers and 10000 concurrency is way too high for your cluster18:22
vinshits actually 1000 workers 1 job each18:23
*** klrmn1 has joined #openstack-swift18:23
vinshSpawning the workers across 2 benchmark hosts with a script. setting their concurrency at 118:23
vinsh500 workers on each host18:24
ctennisyou are best to tie worker count to # of CPUs18:24
vinshssbench seems to just report whatever is in the scenario file vs what ran.18:24
ctennisyou'll avoid the context switching18:24
vinshok. so reduce workers.. increase concurrency per worker?18:25
ctennisI think the overall concurrency is still too high18:25
ctennisnot necessary for the worker count, btu for the cluster..it probably can't support 10,000 simultaneous users18:25
vinshagain its not 10,00018:25
ctennisah that's right18:25
vinshthat would inface melt it18:25
ctennisso with 144 disks, figure 20 or so streams per disk, so 3000 streams, divide by 13 for EC, that's something like ~250 concurrency18:26
*** subscope has joined #openstack-swift18:26
ctennistotal, not per worker18:26
ctennisI'd start there as a basis point, then ratchet it up and see if things get better or fall apart18:27
vinsh32 core hosts18:28
vinshabout 4 per worker then.18:28
ctennisthat sounds good18:28
ctennisin particular I think you shoudln't have the timeouts/errors18:28
ctennismaybe a few, but not 40%18:29
vinsha boat load of "INFO:root:ConnectionPool: re-creating connection" succeeded after 1 retries message when benchmarking still18:31
ctennisdo you kill and restart workers between test runs?18:32
ctennisare you using haproxy as a LB?18:33
vinshI'm going direct to the proxies from ssbench.. using two "-S" endpoints18:34
claygvinsh: -S option is the bomb18:34
ctennisah good call18:34
vinsheven with 4 workers and 4 threads each I get a constant stream of retries succeeded after 1 tries messages18:35
vinshconnection pool recreating18:35
ctennisyeah sounds strange18:36
ctennismight want to amke sure oyu don't have stray workers18:36
vinshps -ef confirms only the 418:36
ctenniswhat's your ssbench-master command line you're using?18:37
vinshssbench-master run-scenario --zmq-bind-ip -f cdvr_combined.scenario -S -S -T $TOKEN18:37
ctennisan ok, what's your scenario file then?18:37
vinshadded to: http://paste.openstack.org/show/476174/18:38
vinshI tried increasing number of initial files thinking maybe it would make them more available18:40
vinshhelps a tiny bit18:40
vinshwrite only tests don't have the same retry issues..even with many workers18:41
vinshI wonder if read affinity is not right18:43
vinshi have set sorting_method to shuffle. got rid of any affinity settings. testing18:47
vinshnope. no change.18:50
vinsh"_INFO:root:ConnectionPool: re-creating connection..."18:52
ctennisdoes that translate to any errors in the run statistics?18:52
vinshmore retries18:53
vinsh       Count:   835 (    0 error;   349 retries: 41.80%)18:53
*** haomaiwang has quit IRC18:53
vinshwould more/less object/proxy/any workers help/hurt reads?18:54
*** portante_ has quit IRC18:55
*** ndk has quit IRC18:55
claygvinsh: try setting --network-timeout to 120 or 20018:56
claygvinsh: just as a test...18:56
vinshSomething else I am doing is dropping the TOKEN part..18:56
vinshwill test one at a time here18:57
claygvinsh: yeah totally - sorry - didn't see you already had something you were running with...18:57
vinshits ok!!18:57
vinshAny advice will be tested :)  Need to sort out the reads here.18:58
CaioBrentanohi all... is there any work in progress related to exporting/indexing metadata that I could read about it?19:01
*** DericHorn-HP has joined #openstack-swift19:01
ctennisvinsh I've seen similar before with ssbench testing I've done, my hunch is there's an ssbench bug being hit19:01
vinshSomething interesting.. is that this is the second swift cluster in a single openstack cluster19:04
vinshI registered it as a new service with its own endpoint19:04
vinshto leverage keystone19:05
vinshit just uses port 8081 instead of 8080 which main swift uses.19:05
openstackgerritRobert Francis proposed openstack/swift: WIP: Symlink implementation. (work in progress)  https://review.openstack.org/23216219:05
vinshI set --network-timeout 200 but still see timeouts19:06
vinshoh maybe connection-timeout19:06
claygCaioBrentano: http://sched.co/49so maybe?19:09
*** CR7 has joined #openstack-swift19:14
openstackgerritTim Burke proposed openstack/swift: Have versioned_writes use reverse listings  https://review.openstack.org/23439119:15
*** nadeem has quit IRC19:15
claygtimburke: oh wow, did reverse listings land?19:15
timburkeclayg: nope, just trying to sweeten the pot :)19:16
timburkeso, y'know...if you *wanted* to review it...19:16
claygtimburke: nope, rings19:16
CaioBrentanothanks clayg! I'll check this speak! ;)19:17
*** ndk has joined #openstack-swift19:17
*** petertr7 is now known as petertr7_away19:19
*** portante has joined #openstack-swift19:19
*** alejandrito has quit IRC19:20
*** alejandrito has joined #openstack-swift19:22
*** petertr7_away is now known as petertr719:22
*** eranrom has quit IRC19:28
*** alejandrito has quit IRC19:32
*** silor has quit IRC19:33
*** petertr7 is now known as petertr7_away19:36
*** delatte has quit IRC19:36
*** gyee has quit IRC19:37
*** haomaiwang has joined #openstack-swift19:37
*** marzif has joined #openstack-swift19:39
notmynameplease keep updating https://etherpad.openstack.org/p/tokyo-summit-swift so that tomorrow during the meeting we can weight the topics and start working on a schedule19:41
*** blmartin has joined #openstack-swift19:42
*** petertr7_away is now known as petertr719:43
*** haomaiwang has quit IRC19:43
*** hezhiqiang has joined #openstack-swift19:46
*** marzif has quit IRC19:54
*** hezhiqiang has quit IRC20:07
*** haypo has left #openstack-swift20:10
*** blmartin_ has joined #openstack-swift20:15
*** blmartin has quit IRC20:15
*** blmartin_ has quit IRC20:16
vinshctennis: if you have any magic moments and remember how you got around the ssbench connection bug I hit.. id be so happy :)20:17
vinshI'm stumped.20:17
*** dencaval has quit IRC20:22
vinshfixed it.20:25
vinshIts magic, you whine at the right time and things work.20:26
claygvinsh: that was less than 10 mins from "I'm stumped" to "fixed it"20:26
ctennisvinsh I seem to recall having issues where the test runs were relatively short20:26
vinshhere I spawned workers manually with a script20:26
ctennisvinsh like it's trying to allocate a lot of jobs across a short time span and throwing errors as a result20:26
vinshand it doesn't work well for reads20:26
vinshwhen I let ssbench-master spawn them. all is well20:27
vinshssbench will spawn like so: INFO:Spawning local ssbench-worker (logging to /tmp/ssbench-worker-local-3.log) with ssbench-worker --zmq-host --zmq-work-port 13579 --zmq-results-port 13580 --concurrency 1 --batch-size 1 320:27
vinshmine were just: ssbench-worker -c 4 --zmq-host 120:28
vinshbut increasing the id at the end20:28
ctennisyou can make the -c option for the worker higher20:29
ctennisthere's like 3 concurrency settings at play with ssbench, it's a bit confusing20:30
vinshI know!20:30
*** marzif has joined #openstack-swift20:30
vinshand some override others it seems20:30
ctennisso you want workers to be something similar to your core count (or less)20:30
ctennisconcurrency with those is just how many jobs they can keep queued20:31
ctennisso that can be something like 50 or 10020:31
ctennis(just ball parking)20:31
ctennisand then the user_count in the scenario is how many simultaenous connections to throw at the cluster20:31
ctennisso workers * concurrency needs to be more than your user_count20:31
vinshwhere does user_count get used?20:32
vinshwhat does that dictate?20:32
ctennishow many simultaneous connections ssbench will make to the cluster while running20:32
ctennisso let's say you have 8 workers, and concurrency of 10 each.20:32
ctennisbut your user count is 120:32
ctennisthen you will have one request at a time hitting the cluster20:33
vinshAh, I follow ya. Thats a good explanation.20:33
*** bill_az has quit IRC20:33
ctennisuser_count is in the scenario file or you can override it on the command line with -u20:33
*** hezhiqiang has joined #openstack-swift20:35
*** then3rd has joined #openstack-swift20:37
*** jmccarthy has joined #openstack-swift20:37
vinshHuh. spawning workers with ssbench-master silences the reconnection warnings20:38
vinshbut retries are still at about 40%20:38
*** jmccarthy has quit IRC20:39
vinshuser count is 4.  I have one worker with a concurrency of 4.20:40
vinshsame as 100020:40
vinsh40% retries.20:41
*** haomaiwang has joined #openstack-swift20:41
ctennisyeah, my hunch is the retries aren't "real", they're just ssbench workers in a funky state and exceptioning then retrying20:41
*** nadeem has joined #openstack-swift20:45
*** DericHorn-HP has quit IRC20:47
*** haomaiwang has quit IRC20:47
*** _hrou_ has joined #openstack-swift20:48
*** hrou has quit IRC20:49
*** hezhiqiang has quit IRC20:52
*** trifon has quit IRC20:56
*** subscope has quit IRC21:02
*** petertr7 is now known as petertr7_away21:04
openstackgerritTim Burke proposed openstack/swift: Add delete markers to versioned_writes middleware  https://review.openstack.org/21492221:10
timburkegood morning mattoliverau!21:12
mattoliverauHey timburke :)21:13
*** dustins has quit IRC21:14
*** hezhiqiang has joined #openstack-swift21:21
claygwbhuber: try going into your /srv/nodeX and moving the symlink to sdbX.old - then validate you get the 507 with a curl directly to the object server - maybe `curl http://localhost:60X0/sdbX -XREPLICATE -v`21:23
claygin that state if you run your reconstructors (`swift-init object-reconstructor once -nv`) you *should* be able to spot the "unmounted" log warnings/errors21:24
wbhuberclayg: ok, thanks.  i'll give it a shot.21:25
claygwbhuber: this is RE lp bug #149156721:25
openstackLaunchpad bug 1491567 in OpenStack Object Storage (swift) "Excessive checks of unmounted disk in reconstructor" [Low,Confirmed] https://launchpad.net/bugs/149156721:25
claygwbhuber: np - thanks for looking at it!21:25
*** bill_az has joined #openstack-swift21:28
timburkehmm. mattoliverau, fyi: looks like patch 120709 will need a rebase to s/assertEquals/assertEqual/ -- wouldn't have noticed except i did a follow-on (patch 234391) to highlight the utility of your change21:28
patchbottimburke: https://review.openstack.org/#/c/120709/ - Add container and account reverse listings21:28
patchbottimburke: https://review.openstack.org/#/c/234391/ - Have versioned_writes use reverse listings21:29
vinshctennis: clayg:  I used my same benchmark hosts to test our swift 2.3.0 cluster.  ZERO retries.  This read issue appears to be related only to 2.4.0 swift with EC in my setup.21:29
notmynamevinsh: do you have the logs for one transaction id that you can pastebin?21:30
mattoliverautimburke: k, thanks will rebase this morning :)21:30
*** wbhuber_ has joined #openstack-swift21:31
claygvinsh: do you have a replicated policy in your 2.4.0 cluster?21:31
vinshnotmyname: added to bottom of: http://paste.openstack.org/show/476191/21:32
vinshclayg: no, only one policy which is EC21:32
*** _hrou_ has quit IRC21:33
*** hrou has joined #openstack-swift21:33
*** wbhuber_ has quit IRC21:33
vinshhttp://paste.openstack.org/show/476192/ has some object node logs added to bottom21:34
*** wbhuber has quit IRC21:35
*** wbhuber has joined #openstack-swift21:36
*** hrou has quit IRC21:37
*** hrou has joined #openstack-swift21:38
claygvinsh: you should seutp a replicated one too!21:38
clayga/b and all that21:38
vinshmight have to21:39
*** hezhiqiang has quit IRC21:40
*** hrou has quit IRC21:42
*** hrou has joined #openstack-swift21:42
*** pgbridge has quit IRC21:43
*** hrou has quit IRC21:44
*** hrou has joined #openstack-swift21:44
claygvinsh: https://gist.github.com/clayg/f38d364a418f2c95998421:45
claygmight at least get to see whats failing or being retried... maybe21:45
gmmahahi, could someone please advise me on how to reproduce this bug? https://bugs.launchpad.net/swift/+bug/149167521:46
openstackLaunchpad bug 1491675 in OpenStack Object Storage (swift) "Different errors for REPLICATE vs. DELETE for same issue" [Undecided,New] - Assigned to Ganesh Maharaj Mahalingam (ganesh-mahalingam)21:46
claygor even maybe just see if there's a theme?21:46
gmmahai am fairly new to openstack and with my limited knowledge i am not able to reproduce this21:46
clayggmmaha: you need an unmounted drive - or something that will raise DiskFileDeviceUnavailable - then you can curl on the object server and see the log lines21:47
clayggmmaha: there's probably some unittests in test.unit.obj.test_server that deal with requests that hit a 507 response - you can look at the debug loggers captured log lines21:47
*** jkugel has quit IRC21:48
gmmahaclayg: thanks. will try running unittests to see if i can capture this error.21:48
wbhuberclayg: reproduced.  thanks!21:51
*** pgbridge has joined #openstack-swift21:57
claygwbhuber: effin' A21:58
*** david-lyle has quit IRC22:10
*** david-lyle has joined #openstack-swift22:12
*** hezhiqiang has joined #openstack-swift22:13
*** jlhinson has quit IRC22:14
*** tongli has quit IRC22:15
*** pgbridge has quit IRC22:20
*** pgbridge has joined #openstack-swift22:21
*** mac_ified has joined #openstack-swift22:22
*** lcurtis has quit IRC22:23
*** hrou has quit IRC22:23
*** hezhiqiang has quit IRC22:30
jrichliclayg: you can express another congratulatory expletive for the fact that I have a version of footers support that works with replication + crypto!22:33
*** blmartin has joined #openstack-swift22:33
jrichliit still has a long way to go ... but things work with it - using replication.  gotta move on to EC now22:34
*** darrenc is now known as darrenc_afk22:35
openstackgerritMatthew Oliver proposed openstack/swift: Add container and account reverse listings  https://review.openstack.org/12070922:38
mattoliveraujrichli:  \o/22:40
mattoliverautimburke: ^^ thanks22:40
*** marzif has quit IRC22:41
*** marzif has joined #openstack-swift22:41
*** willsama has joined #openstack-swift22:43
notmynamejrichli: :-)22:44
*** siva_krishnan has quit IRC22:44
*** haomaiwang has joined #openstack-swift22:44
wbhuberjrichli: woot!22:45
notmynamemattoliverau: timburke: maybe we'll finally get reverse listings someday soon! ;-)22:48
*** haomaiwang has quit IRC22:50
jrichliall: thx, I am pretty excited.  It was a big challenge to get this far.  Again, I still have a long way to go :-)  but mini-milestone22:51
notmyname#success mini-milestone for encryption in swift: proxy->object communication is working for replication.22:53
openstackstatusnotmyname: Added success to Success page22:53
jrichlinotmyname: thx! i wasnt sure if i should use that yet or not :-p22:54
minwoobjrichli: Congrats!!22:54
openstackgerritTim Burke proposed openstack/swift: Have versioned_writes use reverse listings  https://review.openstack.org/23439122:56
timburkenotmyname: and when we do, we can make versioned_writes better at the same time!22:56
notmynametimburke: well, yeah. we certainly don't want to make it worse!22:57
mattoliveraunotmyname: reverse listings in swift, are you crazy! :P22:58
*** xnox has quit IRC22:58
notmynamemattoliverau: that's like the backwards way of it22:58
*** Ailus1 has joined #openstack-swift22:59
*** chlong has quit IRC22:59
*** hezhiqiang has joined #openstack-swift22:59
*** xnox has joined #openstack-swift23:00
*** Ailus has quit IRC23:01
*** xnox has quit IRC23:01
*** xnox has joined #openstack-swift23:02
*** km has joined #openstack-swift23:03
*** km is now known as Guest9237223:03
*** wbhuber has quit IRC23:04
*** wbhuber has joined #openstack-swift23:04
*** xnox has quit IRC23:07
*** gyee has joined #openstack-swift23:07
*** willsama has quit IRC23:08
*** wbhuber has quit IRC23:08
*** nadeem has quit IRC23:12
*** jrichli has quit IRC23:12
*** willsama has joined #openstack-swift23:13
*** hezhiqiang has quit IRC23:18
*** I has joined #openstack-swift23:25
*** I is now known as Guest9019423:25
*** kei_yama has joined #openstack-swift23:29
*** Guest90194 has quit IRC23:33
*** cdelatte has joined #openstack-swift23:34
*** delattec has joined #openstack-swift23:34
*** delatte has joined #openstack-swift23:39
*** delattec has quit IRC23:41
*** cdelatte has quit IRC23:42
*** proteusguy_ has quit IRC23:42
*** haomaiwang has joined #openstack-swift23:46
*** delattec has joined #openstack-swift23:46
*** delatte has quit IRC23:50
*** hezhiqiang has joined #openstack-swift23:50
*** haomaiwang has quit IRC23:50
*** delattec has quit IRC23:55
*** proteusguy_ has joined #openstack-swift23:55
*** willsama has quit IRC23:56
*** hrou has joined #openstack-swift23:57

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!