| *** gyee has quit IRC | 00:11 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: sharding: Add probe test that exercises swift-manage-shard-ranges https://review.opendev.org/721376 | 00:45 |
|---|---|---|
| openstackgerrit | Tim Burke proposed openstack/swift master: WIP: sharding: Add some more shard-related output to swift-container-info https://review.opendev.org/721377 | 00:45 |
| openstackgerrit | Tim Burke proposed openstack/swift master: sharding: Add probe test that exercises swift-manage-shard-ranges https://review.opendev.org/721376 | 01:46 |
| openstackgerrit | Tim Burke proposed openstack/swift master: WIP: sharding: Add some more shard-related output to swift-container-info https://review.opendev.org/721377 | 01:46 |
| *** manuvakery has joined #openstack-swift | 02:24 | |
| *** psachin has joined #openstack-swift | 03:25 | |
| openstackgerrit | Merged openstack/python-swiftclient master: Add py38 classifier https://review.opendev.org/721313 | 03:26 |
| openstackgerrit | Matthew Oliver proposed openstack/swift master: Auto-sharding: Initial steps https://review.opendev.org/667030 | 04:24 |
| openstackgerrit | Matthew Oliver proposed openstack/swift master: sharding: first attempt at _elect_leader https://review.opendev.org/667579 | 04:24 |
| timburke | i think we only have one func test job that uses the saio config we updated in p 704920 -- and it's the rolling upgrade one :-/ | 04:29 |
| patchbot | https://review.opendev.org/#/c/704920/ - swift - Add etag quoter to saio proxy config (MERGED) - 2 patch sets | 04:29 |
| openstackgerrit | Tim Burke proposed openstack/swift master: func tests: Allow test_etag_quoter to be run multiple times https://review.opendev.org/721518 | 04:33 |
| *** manuvakery has quit IRC | 04:34 | |
| *** evrardjp has quit IRC | 04:35 | |
| *** evrardjp has joined #openstack-swift | 04:35 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: Make rolling-upgrade job non-voting https://review.opendev.org/721519 | 04:37 |
| timburke | still at least two items of follow-up needed: (1) have an in-process func test that enables the etag-quoter and (2) re-spin p 695131 as *just* the func test cleanup so they won't care if the etag-quoter is on by default | 04:40 |
| patchbot | https://review.opendev.org/#/c/695131/ - swift - Add proxy-server option to quote-wrap all ETags - 5 patch sets | 04:40 |
| *** manuvakery has joined #openstack-swift | 06:53 | |
| *** rpittau|afk is now known as rpittau | 07:03 | |
| *** ccamacho has joined #openstack-swift | 07:47 | |
| *** threestrands has quit IRC | 07:55 | |
| *** godog has quit IRC | 08:31 | |
| *** godog has joined #openstack-swift | 08:35 | |
| *** dtantsur|afk is now known as dtantsur | 09:00 | |
| *** rdejoux has quit IRC | 09:58 | |
| mattoliverau | Starting to take a look back at auto-sharding, especially so we can, hopefully, talk about it at the virtual PTG. Just whipped up this: https://docs.google.com/document/d/17NllKQmH6tfTsKm5nAx3KCKUvs0zs_qamXtkreOQDWg/edit?usp=sharing will make sure it's in the etherpad. But thought I'd paste it here first :) | 09:59 |
| *** rpittau is now known as rpittau|bbl | 10:31 | |
| *** rdejoux has joined #openstack-swift | 11:35 | |
| *** dtantsur is now known as dtantsur|brb | 12:30 | |
| *** rpittau|bbl is now known as rpittau | 12:31 | |
| *** psachin has quit IRC | 12:53 | |
| *** rcernin has quit IRC | 13:14 | |
| *** dtantsur|brb is now known as dtantsur | 14:47 | |
| *** viks____ has quit IRC | 15:07 | |
| *** abelur has quit IRC | 15:24 | |
| *** abelur has joined #openstack-swift | 15:25 | |
| timburke | mattoliverau, i've been thinking about sharding lately, too -- on my list of things to look at soon :-) | 15:30 |
| *** tkajinam has quit IRC | 15:40 | |
| *** gyee has joined #openstack-swift | 15:52 | |
| *** evrardjp has quit IRC | 16:35 | |
| *** evrardjp has joined #openstack-swift | 16:35 | |
| *** rpittau is now known as rpittau|afk | 16:39 | |
| *** abelur has quit IRC | 16:39 | |
| *** jrosser has quit IRC | 16:41 | |
| *** abelur has joined #openstack-swift | 16:42 | |
| *** manuvakery has quit IRC | 16:43 | |
| *** jrosser has joined #openstack-swift | 16:44 | |
| *** manuvakery has joined #openstack-swift | 16:46 | |
| *** dtantsur is now known as dtantsur|afk | 16:59 | |
| timburke | tdasilva, thanks for the rechecks, but we can hold off until p 721519 merges | 17:46 |
| patchbot | https://review.opendev.org/#/c/721519/ - swift - Make rolling-upgrade job non-voting - 1 patch set | 17:46 |
| tdasilva | timburke: thanks for the heads up! I had missed that issue | 17:48 |
| timburke | no worries! the whole thing reminds me that i still want to do something like p 691747 -- i really need to think harder about what kind of test matrix we want 🤔 | 17:51 |
| patchbot | https://review.opendev.org/#/c/691747/ - swift - Enable s3api and staticweb tests across all func t... - 8 patch sets | 17:51 |
| timburke | i feel like a bit of an idiot for not having done the etag-quoter func test like p 721518 from the get-go | 17:53 |
| patchbot | https://review.opendev.org/#/c/721518/ - swift - func tests: Allow test_etag_quoter to be run multi... - 1 patch set | 17:53 |
| DHE | if a DELETE request for an object comes up but the proxy server returns 404, did it still check with the container DB and delete any possible matching row? | 17:57 |
| DHE | using bulk-delete if that's applicable | 17:58 |
| openstackgerrit | Tim Burke proposed openstack/swift master: func tests: work with etag-quoter on by default https://review.opendev.org/695131 | 18:00 |
| timburke | DHE, as long as the container exists, yes. bulk-delete doesn't really matter -- it just has the proxy fan out a bunch of backend DELETEs similar to if the client did them individually -- the main benefit is in reducing WAN traffic | 18:05 |
| timburke | on older swifts, the delete may 404 thinking that the container doesn't exist when it's actually just temporarily unavailable (say, because it's overwhelmed trying to eat a bunch of deletes), but newer swifts should 503 in that case | 18:07 |
| DHE | timburke: hmm.. I'm getting back from bulk-delete "Number Deleted: 4720 Number Not Found: 280" and concerned about those not-found objects | 18:07 |
| timburke | might be worth looking at proxy logs to confirm that those 404s actually resulted in tombstones getting laid down -- what version of swift is this? | 18:08 |
| timburke | as long as the delete makes it to the object servers, it should write down a tombstone and make sure the container finds out about the delete | 18:10 |
| DHE | 2.23.1 | 18:10 |
| DHE | error 499 logged by proxy server.. aw hell... | 18:11 |
| timburke | version's good -- the change i was looking for was https://github.com/openstack/swift/commit/563e167 | 18:13 |
| DHE | I'm mainly concerned about objects still on disk despite not being in the container db any more... | 18:15 |
| timburke | what's the swift.source for the 499? i know i cleaned up a bunch of those later in https://github.com/openstack/swift/commit/2a8d47f but that was pretty recent -- and i maybe should have touched bulk as part of that :-/ | 18:16 |
| DHE | it's the 'BD' individualized bulk delete requests | 18:16 |
| timburke | container DB should be a lagging indicator -- if the object *was* in listings but now *isn't*, it should have gotten cleaned up | 18:17 |
| timburke | yeah, i probably need to have it drain app_iters for 404s, too :-( | 18:17 |
| timburke | i'm fairly certain those 499s were 404s, though -- now that you know which object tripped it, you can use swift-get-nodes to find where it should have been written down and confirm that the data's gone (and a .ts took its place) | 18:18 |
| DHE | I suspect I know the problem. One object server machine is a big bogged down due a rebalance gone not so well and it's probably moving everything stored on it... | 18:19 |
| DHE | -rwxr-xr-x 1 swift swift 0 Apr 21 14:04 1587492297.65613.ts | 18:19 |
| DHE | Well that's a relief. I can live with that | 18:20 |
| DHE | thanks | 18:20 |
| timburke | fwiw, the way i usually see 404s in bulk deletes is that a client takes a container listing and pipes it to a bulk delete. then a little while later, client sees that there's still data in the listing and does it again. but, since the listing can (and often does) lag, some of the data's already gone | 18:21 |
| timburke | that's why we treat 404s as basically successful | 18:22 |
| DHE | my program sorta does that, but there's a postgresql database in between. container listing -> postgresql -> bulk-delete | 18:22 |
| DHE | although objects should sit in the postgres DB for an extended period, not just deleted... | 18:22 |
| *** gmann is now known as gmann_lunch | 18:31 | |
| *** gmann_lunch is now known as gmann | 18:51 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: func tests: work with etag-quoter on by default https://review.opendev.org/695131 | 19:27 |
| openstackgerrit | Tim Burke proposed openstack/swift master: py3: Make etag-quoter work https://review.opendev.org/721714 | 19:27 |
| *** ccamacho has quit IRC | 19:34 | |
| zaitcev | I'm trying to use both keystone and tempauth. An HEAD authorized with keystone token makes tempauth to traceback with "AttributeError: 'tuple' object has no attribute 'split'". | 21:16 |
| timburke | REMOTE_USER :-/ i should dust off p 630415 | 21:45 |
| patchbot | https://review.opendev.org/#/c/630415/ - swift - Fix the handling of keystone groups in tempauth. - 5 patch sets | 21:45 |
| zaitcev | Thanks a lot | 21:46 |
| rledisez | did anybody already try to increase the max_manifest_segments of SLO to something like 10,000? except increasing concurrency (and maybe max_manifest_size), anything I should be worried of? | 22:10 |
| *** rcernin has joined #openstack-swift | 22:17 | |
| timburke | zaitcev, i know the dsvm jobs run with both keystone and tempauth though -- is this something with cross-reseller_prefix access, or is this happening all the time? if the latter, it might be a pipeline ordering issue... | 22:20 |
| timburke | (one which we should fix, of course. but at least we know `pipeline = ... authtoken keystoneauth tempauth ...` should work, since that's what the gate uses) | 22:22 |
| openstackgerrit | Tim Burke proposed openstack/swift master: WIP: swift-dsvm: Create more Keystone users so we skip fewer tests https://review.opendev.org/620189 | 22:29 |
| timburke | ugh, right -- p 613681 :-( | 22:32 |
| patchbot | https://review.opendev.org/#/c/613681/ - keystonemiddleware - [WIP] Correct auth_token headers to be WSGI compliant (ABANDONED) - 5 patch sets | 22:32 |
| *** tkajinam has joined #openstack-swift | 22:53 | |
| zaitcev | I already arrived at the "authtoken keystone tempauth" order, because nothing else worked. | 23:33 |
| zaitcev | With this, at least authentication passes, the traceback happens at the access to account itself. | 23:33 |
| openstackgerrit | Tim Burke proposed openstack/swift master: WIP: swift-dsvm: Create more Keystone users so we skip fewer tests https://review.opendev.org/620189 | 23:34 |
| timburke | i wonder if it'd be worth us working with env['keystone.token_info'] directly and side-stepping the WSGI string issue... | 23:48 |
| timburke | i mean, s3token already knows about most of the structure: https://github.com/openstack/swift/blob/2.25.0/swift/common/middleware/s3api/s3token.py#L101-L134 | 23:48 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!