| *** jistr has quit IRC | 00:00 | |
| *** jistr has joined #openstack-swift | 00:01 | |
| *** mat128 has quit IRC | 00:05 | |
| timburke | we should go land https://review.openstack.org/#/c/441520/ so i can stop cherry-picking it every time i want to work with a modern s3 client... | 00:09 |
|---|---|---|
| patchbot | patch 441520 - swift - Use swift3's check_signature function | 00:09 |
| *** lnxnut has joined #openstack-swift | 00:11 | |
| *** stewie925 has quit IRC | 00:12 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: Clean up SLO tests and docs https://review.openstack.org/509305 | 00:17 |
| openstackgerrit | Tim Burke proposed openstack/swift master: Let clients request heartbeats during SLO PUTs https://review.openstack.org/509306 | 00:17 |
| kota_ | good morning | 00:20 |
| *** lnxnut has quit IRC | 00:21 | |
| *** gyee has joined #openstack-swift | 00:36 | |
| *** itlinux has joined #openstack-swift | 00:36 | |
| mattoliverau | kota_: morning | 00:39 |
| kota_ | mattoliverau: o/ | 00:50 |
| *** itlinux has quit IRC | 00:58 | |
| *** cshastri has joined #openstack-swift | 01:05 | |
| *** vint_bra has quit IRC | 01:07 | |
| *** vint_bra has joined #openstack-swift | 01:07 | |
| *** mingyu has joined #openstack-swift | 01:07 | |
| *** lnxnut has joined #openstack-swift | 01:18 | |
| *** mingyu has quit IRC | 01:28 | |
| *** lnxnut has quit IRC | 01:29 | |
| *** mingyu has joined #openstack-swift | 01:31 | |
| *** itlinux has joined #openstack-swift | 01:35 | |
| *** vint_bra has quit IRC | 01:40 | |
| *** itlinux has quit IRC | 01:42 | |
| *** gyee has quit IRC | 01:47 | |
| *** tovin07_ has joined #openstack-swift | 01:51 | |
| openstackgerrit | Merged openstack/swift feature/deep: make broker get_shard_range_rows private https://review.openstack.org/509111 | 01:52 |
| openstackgerrit | Merged openstack/swift feature/deep: minor follow up on cleave unit tests https://review.openstack.org/509109 | 01:52 |
| openstackgerrit | Merged openstack/swift feature/deep: Use temp db and a rename for set_sharding_state https://review.openstack.org/506002 | 01:56 |
| *** catintheroof has joined #openstack-swift | 02:03 | |
| *** catintheroof has quit IRC | 02:19 | |
| *** mingyu has quit IRC | 02:22 | |
| *** mingyu has joined #openstack-swift | 02:22 | |
| *** lnxnut has joined #openstack-swift | 02:26 | |
| *** catintheroof has joined #openstack-swift | 02:29 | |
| *** lnxnut has quit IRC | 02:36 | |
| zaitcev | torgomatic poke | 03:00 |
| zaitcev | torgomatic: Where can I find the patch that allows to use linkat with in-memory tracking? Is there a review number? | 03:01 |
| *** catintheroof has quit IRC | 03:09 | |
| *** itlinux has joined #openstack-swift | 03:23 | |
| *** abhitechie has quit IRC | 03:26 | |
| *** itlinux has quit IRC | 03:31 | |
| *** itlinux has joined #openstack-swift | 03:32 | |
| *** lnxnut has joined #openstack-swift | 03:34 | |
| *** gkadam has joined #openstack-swift | 03:39 | |
| *** itlinux has quit IRC | 03:41 | |
| *** lnxnut has quit IRC | 03:44 | |
| *** zaitcev has quit IRC | 03:44 | |
| *** kei_yama has quit IRC | 03:49 | |
| *** kei_yama has joined #openstack-swift | 04:03 | |
| *** alenavolk has joined #openstack-swift | 04:05 | |
| *** SkyRocknRoll has joined #openstack-swift | 04:12 | |
| *** SkyRocknRoll has quit IRC | 04:12 | |
| *** tristanC has quit IRC | 04:16 | |
| *** tristanC has joined #openstack-swift | 04:23 | |
| *** alenavolk has quit IRC | 04:28 | |
| *** alenavolk has joined #openstack-swift | 04:32 | |
| *** alenavolk has quit IRC | 04:33 | |
| *** alenavolk has joined #openstack-swift | 04:34 | |
| *** alenavolk has quit IRC | 04:34 | |
| *** alenavolk has joined #openstack-swift | 04:35 | |
| *** alenavolk has quit IRC | 04:35 | |
| *** alenavolk has joined #openstack-swift | 04:36 | |
| *** alenavolk has joined #openstack-swift | 04:37 | |
| *** alenavolk has quit IRC | 04:38 | |
| *** kei_yama has quit IRC | 04:39 | |
| *** wes_dillingham has quit IRC | 04:39 | |
| *** wes_dillingham has joined #openstack-swift | 04:40 | |
| *** lnxnut has joined #openstack-swift | 04:40 | |
| *** wes_dillingham has quit IRC | 04:41 | |
| *** wes_dillingham has joined #openstack-swift | 04:41 | |
| *** wes_dillingham has quit IRC | 04:41 | |
| *** wes_dillingham has joined #openstack-swift | 04:42 | |
| *** wes_dillingham has quit IRC | 04:42 | |
| *** wes_dillingham has joined #openstack-swift | 04:43 | |
| *** wes_dillingham has quit IRC | 04:43 | |
| *** wes_dillingham has joined #openstack-swift | 04:43 | |
| *** wes_dillingham has quit IRC | 04:44 | |
| *** wes_dillingham has joined #openstack-swift | 04:44 | |
| *** wes_dillingham has quit IRC | 04:45 | |
| *** wes_dillingham has joined #openstack-swift | 04:45 | |
| *** wes_dillingham has quit IRC | 04:45 | |
| *** wes_dillingham has joined #openstack-swift | 04:46 | |
| *** wes_dillingham has quit IRC | 04:46 | |
| *** wes_dillingham has joined #openstack-swift | 04:47 | |
| *** wes_dillingham has quit IRC | 04:47 | |
| *** wes_dillingham has joined #openstack-swift | 04:47 | |
| *** wes_dillingham has quit IRC | 04:48 | |
| *** wes_dillingham has joined #openstack-swift | 04:49 | |
| *** kei_yama has joined #openstack-swift | 04:57 | |
| *** lnxnut has quit IRC | 05:06 | |
| *** klrmn has joined #openstack-swift | 05:07 | |
| *** alenavolk has joined #openstack-swift | 05:11 | |
| *** psachin has joined #openstack-swift | 05:15 | |
| *** early has quit IRC | 05:16 | |
| *** early has joined #openstack-swift | 05:16 | |
| *** lnxnut has joined #openstack-swift | 05:33 | |
| *** abhitechie has joined #openstack-swift | 05:36 | |
| *** lnxnut has quit IRC | 05:37 | |
| openstackgerrit | HCLTech-SSW proposed openstack/swift master: Any inputs on the proposed fix. https://review.openstack.org/509338 | 05:41 |
| *** wes_dillingham has quit IRC | 05:42 | |
| *** ChubYann has quit IRC | 05:42 | |
| *** f0o has quit IRC | 05:48 | |
| *** portante has quit IRC | 05:53 | |
| *** ndk- has quit IRC | 05:55 | |
| *** kei_yama has quit IRC | 06:03 | |
| *** oshritf has joined #openstack-swift | 06:10 | |
| *** abhitechie has quit IRC | 06:14 | |
| *** alenavolk has quit IRC | 06:20 | |
| *** portante has joined #openstack-swift | 06:21 | |
| *** ndk_ has joined #openstack-swift | 06:21 | |
| *** rcernin has joined #openstack-swift | 06:30 | |
| *** lnxnut has joined #openstack-swift | 06:34 | |
| *** pcaruana has joined #openstack-swift | 06:48 | |
| openstackgerrit | Matthew Oliver proposed openstack/swift feature/deep: unit tests for _misplaced_objects https://review.openstack.org/509256 | 06:52 |
| mattoliverau | acoles: ^^ thats just a rebase. I left some comments on the patchset before.. nothing really as a blocker. | 06:53 |
| *** spectr has quit IRC | 06:54 | |
| *** spectr has joined #openstack-swift | 06:55 | |
| *** lnxnut has quit IRC | 06:57 | |
| mattoliverau | So whose around for the 0700 swift meeting? | 06:58 |
| mahatic | mattoliverau: o/ | 07:00 |
| mattoliverau | acoles said he might make it. I wonder if kota_ is around | 07:00 |
| psachin | mattoliverau: o/ | 07:00 |
| kota_ | o/ | 07:01 |
| mattoliverau | cool, well lets wait in the actual meeting channel then :) | 07:01 |
| *** skudlik has joined #openstack-swift | 07:01 | |
| acoles | good morning | 07:01 |
| mahatic | acoles: morning! | 07:02 |
| *** cbartz has joined #openstack-swift | 07:03 | |
| *** m_kazuhiro has joined #openstack-swift | 07:05 | |
| *** skudlik has quit IRC | 07:08 | |
| *** skudlik has joined #openstack-swift | 07:08 | |
| *** skudlik has left #openstack-swift | 07:09 | |
| *** cbartz has quit IRC | 07:10 | |
| *** geaaru has joined #openstack-swift | 07:12 | |
| *** tesseract has joined #openstack-swift | 07:16 | |
| *** kei_yama has joined #openstack-swift | 07:16 | |
| *** cbartz has joined #openstack-swift | 07:24 | |
| *** cbartz has quit IRC | 07:39 | |
| *** cbartz has joined #openstack-swift | 07:39 | |
| acoles | mattoliverau: I have one quick question for you on sharding | 07:47 |
| acoles | mattoliverau: whenever we cleave or move misplaced objects, we should be moving deleted object rows, correct? we'll need the history of deletion in shard containers until reclaim age passes | 07:49 |
| mattoliverau | oh yeah, yes we should. I'm not sure it's currently doing that. We don't want to loose a potential delete. | 07:50 |
| mattoliverau | but we don't need to propergate the deletes we make after deleting them after they moved. | 07:50 |
| acoles | mattoliverau: right, I thought so and have a bunch of TODOs in tests to confirm deleted rows are moved | 07:51 |
| mattoliverau | but that could make misplaced objects harder, and to timburke | 07:51 |
| mattoliverau | s point | 07:51 |
| acoles | you mean, when we delete the 'source' row in root container after that row has been moved to shard? | 07:51 |
| mattoliverau | if we mark misplaced object rows as deleted, will be pick them up the next cycle | 07:52 |
| mattoliverau | yeah | 07:52 |
| acoles | oic, yeah in the misplaced objects table that could be tricky | 07:53 |
| acoles | the read only root object table should be ok though | 07:53 |
| acoles | we only cleave once | 07:53 |
| mattoliverau | currently we only move objects where deleted=0 I think.. but then we may potentually miss a delete | 07:54 |
| acoles | I think that is correct based on my tests ^^ | 07:54 |
| mattoliverau | yeah, readonly, not an issue, just misplaced in the holding table | 07:54 |
| *** f0o|away has joined #openstack-swift | 07:54 | |
| acoles | or misplaced outside of shard range in shard container | 07:54 |
| mattoliverau | yeah | 07:54 |
| mattoliverau | do we need to add some mark to indicate moved objects, marking as deleted might not be enough. | 07:56 |
| acoles | so should we just remove the misplaced row once it has been relocated? | 07:56 |
| mattoliverau | true | 07:56 |
| mattoliverau | so long as we're happy they've moved.. but in the current code, thats the assumption | 07:56 |
| mattoliverau | so yeah, that works :) | 07:56 |
| acoles | yep we need to be careful to know when rows have successfully moved, but once they have been they can be dropped I think | 07:57 |
| acoles | ok thanks mattoliverau, have a good evening | 07:58 |
| *** m_kazuhiro has quit IRC | 07:58 | |
| *** f0o|away is now known as f0o | 08:00 | |
| mattoliverau | acoles: nps, thank you | 08:01 |
| *** lnxnut has joined #openstack-swift | 08:28 | |
| *** lnxnut has quit IRC | 08:47 | |
| *** kei_yama has quit IRC | 08:50 | |
| *** lnxnut has joined #openstack-swift | 09:45 | |
| *** lnxnut has quit IRC | 09:56 | |
| openstackgerrit | Alistair Coles proposed openstack/swift feature/deep: unit tests for _misplaced_objects https://review.openstack.org/509256 | 10:01 |
| *** tovin07_ has quit IRC | 10:06 | |
| *** klrmn has quit IRC | 10:32 | |
| *** mingyu has quit IRC | 10:34 | |
| *** tesseract has quit IRC | 10:40 | |
| *** abhitechie has joined #openstack-swift | 10:44 | |
| *** tesseract has joined #openstack-swift | 10:44 | |
| *** lnxnut has joined #openstack-swift | 10:53 | |
| *** spectr-RH has joined #openstack-swift | 10:59 | |
| *** mat128 has joined #openstack-swift | 11:00 | |
| *** abhitechie has quit IRC | 11:01 | |
| *** spectr has quit IRC | 11:02 | |
| *** lnxnut has quit IRC | 11:03 | |
| *** mjseger has joined #openstack-swift | 11:06 | |
| mjseger | mattoliverau: are you around? | 11:06 |
| *** cshastri has quit IRC | 11:11 | |
| mattoliverau | mjseger: kinda, bit busy tonight so not really near a computer | 11:18 |
| mjseger | ok, np. we can talk tomorrow. looks like with 150M objects spread out over Ks of containers I'm seeing 1/2 the PUT rates. drilling deeper now. | 11:21 |
| *** oshritf has quit IRC | 11:30 | |
| *** lnxnut has joined #openstack-swift | 12:00 | |
| *** lnxnut has quit IRC | 12:11 | |
| *** mjseger has quit IRC | 12:12 | |
| *** ianychoi has quit IRC | 12:25 | |
| *** ianychoi has joined #openstack-swift | 12:26 | |
| *** gkadam has quit IRC | 12:50 | |
| *** spectr-RH has quit IRC | 13:00 | |
| *** pcaruana has quit IRC | 13:02 | |
| *** lnxnut has joined #openstack-swift | 13:08 | |
| *** mat128 has quit IRC | 13:11 | |
| *** spectr-RH has joined #openstack-swift | 13:12 | |
| *** pcaruana has joined #openstack-swift | 13:15 | |
| *** spectr-RH has quit IRC | 13:16 | |
| *** spectr-RH has joined #openstack-swift | 13:16 | |
| *** lnxnut has quit IRC | 13:17 | |
| *** chlong_ has joined #openstack-swift | 13:22 | |
| *** lnxnut has joined #openstack-swift | 13:23 | |
| *** catintheroof has joined #openstack-swift | 13:24 | |
| *** mat128 has joined #openstack-swift | 13:27 | |
| *** mingyu has joined #openstack-swift | 13:34 | |
| *** mingyu has quit IRC | 13:39 | |
| *** wes_dillingham has joined #openstack-swift | 13:41 | |
| *** vint_bra has joined #openstack-swift | 14:09 | |
| *** d0ugal has quit IRC | 14:10 | |
| *** d0ugal has joined #openstack-swift | 14:13 | |
| *** d0ugal has quit IRC | 14:13 | |
| *** d0ugal has joined #openstack-swift | 14:13 | |
| *** pcaruana has quit IRC | 14:33 | |
| *** wes_dillingham has quit IRC | 14:34 | |
| openstackgerrit | Alistair Coles proposed openstack/swift feature/deep: Include deleted objects when moving misplaced objects https://review.openstack.org/509496 | 14:36 |
| *** spectr-RH has quit IRC | 14:41 | |
| *** spectr has joined #openstack-swift | 14:42 | |
| *** pcaruana has joined #openstack-swift | 14:45 | |
| *** wes_dillingham has joined #openstack-swift | 14:49 | |
| *** cbartz has quit IRC | 14:54 | |
| *** lnxnut_ has joined #openstack-swift | 14:56 | |
| *** lnxnut has quit IRC | 14:58 | |
| *** jrichli has quit IRC | 14:59 | |
| *** jrichli has joined #openstack-swift | 15:03 | |
| *** ChanServ sets mode: +v jrichli | 15:03 | |
| *** rcernin has quit IRC | 15:13 | |
| *** silor has joined #openstack-swift | 15:18 | |
| *** silor has quit IRC | 15:20 | |
| timburke | good morning | 15:21 |
| *** thurloat has quit IRC | 15:22 | |
| *** silor has joined #openstack-swift | 15:24 | |
| *** wes_dillingham has quit IRC | 15:25 | |
| *** psachin has quit IRC | 15:25 | |
| openstackgerrit | Alistair Coles proposed openstack/swift feature/deep: Eliminate _generate_object_list method https://review.openstack.org/509508 | 15:26 |
| *** pcaruana has quit IRC | 15:28 | |
| *** chsc has joined #openstack-swift | 15:28 | |
| *** chsc has joined #openstack-swift | 15:28 | |
| *** tesseract has quit IRC | 15:29 | |
| *** gyee has joined #openstack-swift | 15:29 | |
| *** chsc has quit IRC | 15:34 | |
| *** guimaluf has joined #openstack-swift | 15:41 | |
| *** SkyRocknRoll has joined #openstack-swift | 16:26 | |
| openstackgerrit | Tim Burke proposed openstack/swift feature/deep: Add shard_ranges table on failed writes, not reads https://review.openstack.org/508355 | 16:36 |
| *** chsc has joined #openstack-swift | 16:39 | |
| *** chsc has joined #openstack-swift | 16:39 | |
| *** thurloat has joined #openstack-swift | 16:44 | |
| *** zaitcev has joined #openstack-swift | 16:45 | |
| *** ChanServ sets mode: +v zaitcev | 16:45 | |
| *** alenavolk has joined #openstack-swift | 16:48 | |
| *** thurloat has quit IRC | 16:49 | |
| notmyname | good morning | 16:56 |
| guimaluf | hi guys, I'm checking my swift-recon output and there is nothing in the updater times metric... Is this normal? Acctualy most of my metrics are 0 | 17:02 |
| notmyname | (I haven't validated this, so only based on old memories...) are you running the swift-recon-cron? | 17:03 |
| guimaluf | notmyname, probably not... :( | 17:06 |
| guimaluf | I run in my dev storage nodes but no metrics so far | 17:06 |
| *** klrmn has joined #openstack-swift | 17:09 | |
| notmyname | ah. the cron is only for async pendings. https://docs.openstack.org/swift/latest/admin_guide.html#cluster-telemetry-and-monitoring | 17:09 |
| guimaluf | [updater_last_sweep] low: 0, high: 0, avg: 0.0, total: 0, Failed: 0.0%, no_result: 0, reported: 2 | 17:14 |
| guimaluf | this is even happening with 8 nodes | 17:15 |
| guimaluf | http://10.1.1.139:6000/recon/updater/object: {'object_updater_sweep': 0.020959138870239258} | 17:20 |
| guimaluf | seems like the value is just out of display range..? | 17:20 |
| *** wes_dillingham has joined #openstack-swift | 17:20 | |
| guimaluf | hmmm I've data for containers! | 17:23 |
| guimaluf | so what this metric does really means? | 17:24 |
| *** SkyRocknRoll has quit IRC | 17:31 | |
| *** thurloat has joined #openstack-swift | 17:37 | |
| *** itlinux has joined #openstack-swift | 17:49 | |
| *** itlinux has quit IRC | 17:53 | |
| *** itlinux has joined #openstack-swift | 18:04 | |
| *** itlinux has quit IRC | 18:12 | |
| clayg | so.. the weird follow on patch to draw attention to the patch that has a -1 review and failed zuul? | 18:15 |
| clayg | we should probably have some sort of plan/delegation for who can review that patch - if they author is committed to finishing a solution that meets the design requirements of the maintainers it *would* be great to have something for delayed account un-delete that works! | 18:16 |
| clayg | this is patch 507808 | 18:16 |
| patchbot | https://review.openstack.org/#/c/507808/ - swift - Add ability to undelete an account. | 18:16 |
| clayg | is... *today* bug triage day again? do we do that *every* wednesday? | 18:17 |
| clayg | ... i didn't put it on my calendar | 18:17 |
| clayg | notmyname: wait, this still says 9/27 - is there 2100 meeting today? | 18:17 |
| clayg | i was going to put the undelete patch on there... maybe pete's PUT+POST patch? | 18:18 |
| guimaluf | hi guys, what updater_sweep value really means on swift-recon output? | 18:25 |
| *** chsc has quit IRC | 18:27 | |
| *** chsc has joined #openstack-swift | 18:29 | |
| notmyname | clayg: yes! please update the meeting agenda | 18:38 |
| clayg | guimaluf: cycle time of the last finished swift-*-updater daemon? | 18:40 |
| clayg | this is container or object? container-updater is the thing that tells account db's how many objects/bytes - object-updater is the thing that makes async updates of object put/delete make their way into the container layer for sure (eventually) | 18:41 |
| guimaluf | clayg, is it in seconds? | 18:41 |
| clayg | oh... yeah units there are pretty stupid annoying - sometimes they're in minutes - i'd just have to look through the docs/code | 18:42 |
| clayg | where have you been looking? | 18:42 |
| guimaluf | couldn't find any meaningful doc explaining swift-recon output | 18:42 |
| guimaluf | looking just at recon output | 18:43 |
| guimaluf | http://192.168.55.109:6001/recon/updater/container: {'container_updater_sweep': 20.479094982147217} | 18:43 |
| clayg | might be in docs for "telemetry"? | 18:43 |
| clayg | code is here-ish -> https://github.com/openstack/swift/blob/master/swift/container/updater.py#L184 | 18:44 |
| clayg | maybe something... https://docs.openstack.org/swift/latest/admin_guide.html#cluster-telemetry-and-monitoring | 18:45 |
| guimaluf | I was following this telemetry guide | 18:46 |
| guimaluf | but no clear information inside :( | 18:46 |
| clayg | if you can make a specific burn down of information that you had to track down a documention bug report would be super helpful - https://bugs.launchpad.net/swift (super double bonus points if you include any answers you were able to find along the way) | 18:46 |
| clayg | ok, well the code seems to be using the return value of time.time directly with no / 60 conversation - so you can safely assume it's in seconds | 18:47 |
| openstackgerrit | Alistair Coles proposed openstack/swift feature/deep: Don't lose timestamp info when moving misplaced objects https://review.openstack.org/509589 | 18:48 |
| acoles | timburke: mattoliverau I've got a little chain of patches going on misplaced objects, this is the last ^^ | 18:49 |
| guimaluf | clayg, ty! | 18:49 |
| *** ChubYann has joined #openstack-swift | 18:49 | |
| acoles | timburke: mattoliverau we probably need a similar fix for cleave, but one step at a time ;) there is a TODO already in the tests to remind us | 18:50 |
| *** geaaru has quit IRC | 18:50 | |
| timburke | thanks acoles! thoughts on the comments for remove_objects? looks like we'll need to respin that patch anyway to fix the doc build | 18:51 |
| clayg | guimaluf: np, pay me back by taking whatever notes you're gathering and thinking about how they could have been conveniently presented on that doc link I posted earlier - then put your thoughts/notes in a bug report. I'd really appreciate your contribution to OpenStack swift! Let me know if you bug into any other stumbling blocks - good luck. | 18:51 |
| *** alenavolk has quit IRC | 18:58 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: Stop clearing params for account_autocreate responses https://review.openstack.org/509594 | 19:01 |
| timburke | ^^^ looks like i tried too hard to not change tests in patch 449394 | 19:02 |
| patchbot | https://review.openstack.org/#/c/449394/ - swift - Move listing formatting out to proxy middleware (MERGED) | 19:02 |
| guimaluf | clayg I'll take some time to do this! | 19:15 |
| *** chlong_ has quit IRC | 19:16 | |
| openstackgerrit | Jan Zerebecki proposed openstack/swift master: Fix swift-ring-builder set_weight with >1 device https://review.openstack.org/509606 | 19:32 |
| openstackgerrit | Samuel Merritt proposed openstack/swift master: Step 1: make DiskFileWriters able to hang around https://review.openstack.org/509607 | 19:34 |
| torgomatic | zaitcev: https://review.openstack.org/509607 | 19:34 |
| patchbot | patch 509607 - swift - Step 1: make DiskFileWriters able to hang around | 19:34 |
| zaitcev | torgomatic: thank you | 19:36 |
| *** Sukhdev has joined #openstack-swift | 19:39 | |
| *** Sukhdev has quit IRC | 19:51 | |
| guimaluf | clayg, I hope to be able to fix this: https://bugs.launchpad.net/swift/+bug/1721380 :) | 20:01 |
| openstack | Launchpad bug 1721380 in OpenStack Object Storage (swift) "Admin guide have unclear swift-recon output details" [Undecided,New] | 20:01 |
| clayg | guimaluf: oh wow, that'd be even *better* - if you've never built swift docs before you might find it's easier to get things up in going in a vm instead of trying to find somewhere you can install sphinx... https://docs.openstack.org/swift/latest/associated_projects.html#developer-tools | 20:31 |
| *** mat128 has quit IRC | 20:53 | |
| kota_ | good morning | 20:59 |
| notmyname | good morning | 20:59 |
| notmyname | meeting time in #openstack-meeting | 21:00 |
| *** m_kazuhiro has joined #openstack-swift | 21:00 | |
| notmyname | clayg: torgomatic: acoles: ping | 21:01 |
| acoles | timburke: re remove_objects - great points, just replied on patch 509496 | 21:15 |
| patchbot | https://review.openstack.org/#/c/509496/ - swift (feature/deep) - Include deleted objects when moving misplaced objects | 21:15 |
| acoles | timburke: I will spin it tomorrow but more than happy if you want to fix my bads before then | 21:16 |
| timburke | acoles: thanks. fwiw, a sqlite repl and `explain query plan` go a long way for me :-) | 21:16 |
| *** silor has quit IRC | 21:25 | |
| *** wes_dillingham has quit IRC | 21:27 | |
| *** wes_dillingham has joined #openstack-swift | 21:27 | |
| *** wes_dillingham has quit IRC | 21:27 | |
| *** wes_dillingham has joined #openstack-swift | 21:28 | |
| *** wes_dillingham has quit IRC | 21:28 | |
| *** wes_dillingham has joined #openstack-swift | 21:30 | |
| *** wes_dillingham has quit IRC | 21:30 | |
| *** wes_dillingham has joined #openstack-swift | 21:30 | |
| *** wes_dillingham has quit IRC | 21:30 | |
| *** wes_dillingham has joined #openstack-swift | 21:32 | |
| *** wes_dillingham has quit IRC | 21:32 | |
| *** wes_dillingham has joined #openstack-swift | 21:33 | |
| *** wes_dillingham has quit IRC | 21:34 | |
| *** wes_dillingham has joined #openstack-swift | 21:35 | |
| *** wes_dillingham has quit IRC | 21:35 | |
| *** wes_dillingham has joined #openstack-swift | 21:37 | |
| *** wes_dillingham has quit IRC | 21:37 | |
| *** wes_dillingham has joined #openstack-swift | 21:37 | |
| *** wes_dillingham has quit IRC | 21:38 | |
| *** wes_dillingham has joined #openstack-swift | 21:39 | |
| *** wes_dillingham has quit IRC | 21:39 | |
| *** wes_dillingham has joined #openstack-swift | 21:40 | |
| *** wes_dillingham has quit IRC | 21:40 | |
| *** catintheroof has quit IRC | 21:41 | |
| * notmyname will be back online later today | 21:49 | |
| mattoliverau | notmyname: o/ hope all is well | 21:50 |
| * kota_ is too, going to find his breakfast. | 21:50 | |
| notmyname | yeah, nobody in the hospital or anything like that ... | 21:50 |
| * rledisez will go to bed now… | 21:51 | |
| rledisez | have a good day or night all | 21:51 |
| kota_ | good night rledisez! | 21:51 |
| acoles | rledisez: good night | 21:51 |
| clayg | timburke: I guess I should just out and say I think you should always return 202 with heartbeat=on | 21:52 |
| clayg | you can't turn on heartbeat=on w/o a code change - so make the code change more straight forward | 21:52 |
| clayg | clients don't want no if statements | 21:52 |
| acoles | timburke: lol @ 'coward' | 21:53 |
| clayg | ^ IMHO obvs. | 21:53 |
| timburke | clayg: does the client know that it's talking to a new-enough cluster? | 21:54 |
| *** m_kazuhiro has quit IRC | 21:54 | |
| torgomatic | timburke: a client smart enough to check /info does | 21:55 |
| timburke | i know it *can* (i expose the yield_frequency in /info), but swiftclient (for example) generally avoids that extra request | 21:55 |
| clayg | timburke: that's an interesting point... but I don't think we should assume that a client *has* to support fallback | 21:55 |
| clayg | like level 0 client would be do the best right thing for the current version of swift - we want to make that as easy as possible to find out what is the best practices (i.e. use heartbeat=on, it's fun and easy!) | 21:56 |
| clayg | level 1 client might try to understand that some old deployments of swift are crappy and try to make useful error messages "umm... idk if this old ass swift is going to work, someone said one time it didn't, you should tell your ops to upgrade" | 21:57 |
| clayg | level 2 client maybe knows enought about swift internals to start to reason about a path to fallback and maintain compatibility - but YOU more than anyone know that is not for the feint of heart - there be dragons | 21:58 |
| clayg | i'd love to just make the heartbeat 202 with status in the body be "what you get" but agree for backwards compat that's be probably pretty impossible :'( | 21:58 |
| clayg | timburke: I could be wrong!? | 21:59 |
| timburke | yeah, see, the one client that i want to give support for this like *right now* -- swift3 -- is that level 2 guy, so that's mostly where i'm thinking :P | 21:59 |
| not_me | Hi! I'm trying to understand why object-replication on one of geo replicated ring is slow. | 22:00 |
| timburke | i'd be OK with always flipping to 202. but the client will still need to look for 4XXs both as the primary status *and* in the body | 22:00 |
| not_me | object-server 5436/5964 (91.15%) partitions replicated in 3900.03s (1.39/sec, 6m remaining) | 22:00 |
| clayg | oic, ok - well if you already have the fallback code *written* it probably doesn't have to change even if you make the heartbeat behavior less unpredictable | 22:00 |
| *** wes_dillingham has joined #openstack-swift | 22:00 | |
| not_me | Our other rings replication takes around 100s to complete. object-server 32340/32340 (100.00%) partitions replicated in 102.91s (314.26/sec, 0s remaining) | 22:01 |
| clayg | timburke: oic... does it *have* to be that way? maybe we should say heartbeat=on returns 202 with the status in the body? I'm not sure that's better... but at least they could assume resp != 202 ==> ERROR TIME | 22:01 |
| timburke | since auth comes before slo, i'ma say yes, it has to be that way | 22:02 |
| not_me | I'm unsure where to look at this point. 1 partition/second seems extremely slow. | 22:02 |
| clayg | not_me: sounds like that one server is doing more actual work (moving bytes) todo? or it's gummed up on a slow disk... | 22:02 |
| clayg | timburke: good point about auth! | 22:03 |
| clayg | timburke: errors can come from *anywhere* - they must check the status | 22:03 |
| timburke | maybe it's worth it to have status != 202 -> error, but i'm also not real happy with the idea that one-and-only-one status code can be success | 22:04 |
| not_me | All the nodes in this ring takes between 5000 and 9000s to complete their object-replication pass. There are only 18 nodes in this ring and 100 devices. | 22:04 |
| timburke | i could go either way. maybe it'd be simpler to always 202 | 22:05 |
| clayg | i just think people writing clients are going to tend towards making it work the one time they tested it - so keeping behavior consistent leads to less client bugs (and less client authors made an un-deterministic behavior in the swift api) | 22:05 |
| not_me | object.builder, build version 234 | 22:05 |
| not_me | 16384 partitions, 9.000000 replicas, 9 regions, 18 zones, 100 devices, 0.07 balance, 100.00 dispersion | 22:05 |
| clayg | timburke: maybe i'm looking for you to give a good reason it *should* be variable? I had assumed it had something to do with "looks more like resp w/o heartbeat=on" - which to me sounded like a recipe for trouble - what's the info I'm missing? | 22:06 |
| clayg | not_me: have you read this section on counting handoff parititons on geo clusters? https://docs.openstack.org/swift/latest/admin_guide.html#checking-handoff-partition-distribution | 22:07 |
| not_me | The weird part is that I can't see any actual data being moved. The replication pass seems to be wasting its time checking very slowly, without having to transfer anything | 22:07 |
| not_me | I haven't, reading now! | 22:08 |
| clayg | are the concurrency settings consistent across the configs? can you turn on debug logging and check timing info on REPLICATE requests? | 22:08 |
| clayg | not_me: something like this might help too -> https://gist.github.com/clayg/5d3b3ca13dea15f6a48efaa3cea92d7d | 22:09 |
| clayg | but... idk... iostat might be your best bet - are the disks on that node under heavy utilization? or worse lots of await for far to few iops? maybe check smartctl | 22:10 |
| not_me | What do you mean by concurrency settings? | 22:25 |
| not_me | [object-replicator] | 22:27 |
| not_me | concurrency = x | 22:27 |
| not_me | right? | 22:27 |
| clayg | yeah I think that's the big one... it can have a big impact on the number of disk/parts that are "working" | 22:35 |
| *** wes_dillingham has quit IRC | 23:05 | |
| *** wes_dillingham has joined #openstack-swift | 23:12 | |
| *** mingyu has joined #openstack-swift | 23:16 | |
| *** vint_bra has quit IRC | 23:17 | |
| *** vint_bra has joined #openstack-swift | 23:17 | |
| *** vint_bra has quit IRC | 23:21 | |
| *** kei_yama has joined #openstack-swift | 23:26 | |
| *** chsc has quit IRC | 23:32 | |
| * notmyname back | 23:51 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!