| mattoliverau | morning | 00:20 |
|---|---|---|
| notmyname | good morning mattoliverau | 00:40 |
| *** ianychoi has quit IRC | 01:20 | |
| *** mahatic has quit IRC | 01:59 | |
| *** jistr has quit IRC | 02:42 | |
| *** jistr has joined #openstack-swift | 02:50 | |
| *** psachin has joined #openstack-swift | 02:52 | |
| *** mahatic has joined #openstack-swift | 03:48 | |
| *** ChanServ sets mode: +v mahatic | 03:48 | |
| *** gyee has quit IRC | 03:53 | |
| kota_ | hello world | 04:19 |
| *** rcernin has quit IRC | 06:43 | |
| *** e0ne has joined #openstack-swift | 07:10 | |
| *** e0ne has quit IRC | 07:12 | |
| *** e0ne has joined #openstack-swift | 07:13 | |
| *** e0ne has quit IRC | 07:14 | |
| *** e0ne has joined #openstack-swift | 07:15 | |
| *** e0ne has quit IRC | 07:17 | |
| *** ccamacho has joined #openstack-swift | 08:04 | |
| *** hseipp has joined #openstack-swift | 08:13 | |
| *** persia has left #openstack-swift | 09:31 | |
| *** mcape has joined #openstack-swift | 10:07 | |
| *** e0ne has joined #openstack-swift | 10:12 | |
| mcape | hello All! I tried to enable sharding on container with 2.5mil objects, it looks like the process is stalled. After 12 hours of wait, "cleaved" number stays the same it was. The only error from container-sharder I see in logs is this: "Since Tue Dec 11 10:03:50 2018 audit_root - attempted:1 success:0 failure:1" | 10:13 |
| mcape | I try to look at the stats in container.recon to get some understanding what is going on, with mixed success so far | 10:14 |
| mcape | the load on the swift is small, ~20k puts in 24 hours, I thought that sharding process should finish in 12 hours... but it don't | 10:20 |
| mcape | Found an extended audit error message: http://paste.openstack.org/raw/736968/ | 10:25 |
| mcape | looks like i recovered the sequence of action of this botched sharding attempt | 10:57 |
| mcape | i did followed the documentation at https://docs.openstack.org/swift/latest/overview_container_sharding.html | 10:57 |
| mcape | did swift-manage-shard-ranges <path_to_db> find 500000 > my_shard_ranges | 10:58 |
| mcape | than I copy-pasted from doc the line "swift-manage-shard-ranges <path_to_db> find_and_replace 500000 --enable > --force" | 10:58 |
| mcape | which redirected output to file "--force", and resulted in no output on console | 10:59 |
| mcape | when I saw no expected output, i did swift-manage-shard-ranges <path_to_db> replace my_shard_ranges | 11:00 |
| mcape | id said that sharding is already underway, and i pressed "q", but it injected ranges anyway | 11:00 |
| mcape | looks like that between find_and_replace and replace a few files added to container, and ranges shifted a bit | 11:00 |
| mcape | maybe sharding doc needs an update? what is the idea behind output redirection in there? | 11:01 |
| mcape | it's not a production container, but what can be done to complete sharding process? loss of files are acceptable | 11:03 |
| *** NM has joined #openstack-swift | 11:06 | |
| *** psachin has quit IRC | 12:35 | |
| *** e0ne has quit IRC | 12:47 | |
| *** hseipp has quit IRC | 13:04 | |
| mcape | another log of stuck sharding process | 13:09 |
| mcape | https://snag.gy/Br3elG.jpg | 13:09 |
| *** hseipp has joined #openstack-swift | 14:03 | |
| *** mvkr has quit IRC | 14:07 | |
| *** psachin has joined #openstack-swift | 14:11 | |
| *** mpasserini has joined #openstack-swift | 14:14 | |
| *** coreycb has joined #openstack-swift | 14:22 | |
| coreycb | notmyname: hi, I wanted to check and see what the status of py3 support is for swift these days. we're switching everything in ubuntu to py3-only for stein. in the next ubuntu LTS anything that remains py2 will be moved to universe which basically means they won't get security support. | 14:29 |
| coreycb | notmyname: so I want to make sure we have swift covered correctly. currently we don't have py3 packages. | 14:30 |
| *** mvkr has joined #openstack-swift | 14:44 | |
| *** e0ne has joined #openstack-swift | 14:45 | |
| mcape | so I got 12 shards instead of six shards | 14:55 |
| mcape | https://paste.fedoraproject.org/paste/PSCYYgSKyX3rhy9Nq18E5w | 14:55 |
| mcape | now I guess I need to remove 6 extra shards | 14:56 |
| *** psachin has quit IRC | 15:05 | |
| *** itlinux has quit IRC | 15:06 | |
| *** ccamacho has quit IRC | 16:09 | |
| openstackgerrit | Thiago da Silva proposed openstack/swift master: Fix container sharding doc https://review.openstack.org/624432 | 16:18 |
| *** itlinux has joined #openstack-swift | 16:22 | |
| tdasilva | mcape: just sent a patch to review the doc typo... | 16:23 |
| tdasilva | last week a patch was merged to fix an issue in sharding that would cause the process to get stuck: https://github.com/openstack/swift/commit/c26d67efcf44c813034f59b1df78d59fdb4a50e7 | 16:24 |
| tdasilva | but it only happens when a drive us unmounted | 16:24 |
| mcape | i did recreated the shards, and now the process is done successfully. I guess i lost only updates since the first sharding attempt, which is okay | 16:25 |
| *** hseipp has quit IRC | 16:25 | |
| mcape | but there is another possible bug, how did I end up with two copies of shards? | 16:27 |
| mcape | swift-manage-shard-ranges warned me that sharding is already underway | 16:27 |
| mcape | and i did press 'q', but shards got injected anyway | 16:27 |
| *** e0ne has quit IRC | 16:36 | |
| tdasilva | mcape: yeah, sounds like there's multiple issues there. one is the number of shards doesn't seem right and second is the cli tool. can you file a bug so we can investigate? | 16:42 |
| notmyname | coreycb: our py3 support is still in progress. are you implying that the stein releases for openstack projects is what's in the next LTS? | 16:43 |
| tdasilva | mpasserini: any luck with 1space? | 16:44 |
| mcape | yeah i'll try to recreate the problem on some test container | 16:44 |
| tdasilva | mcape: it may be that the number of shards issue is the intended behavior due to the fact that you ran "find and replace" twice, but still doesn't answer the 'q' issue where the second attempt should have been aborted | 16:45 |
| tdasilva | mcape: if you can reproduce and add the steps to the bug would be great | 16:46 |
| coreycb | notmyname: I just confirmed with our distro maintainer and py2.7 will be removed from main (and therefore all py2.7 packages) in 19.04 if possible (stein) or 19.10 latest (train) | 16:55 |
| *** itlinux_ has joined #openstack-swift | 16:59 | |
| *** itlinux has quit IRC | 17:03 | |
| mcape | tdasivla: behavior is reproducable, can you confirm that it is a bug? http://paste.openstack.org/show/737042/ | 17:14 |
| mcape | after I did press 'q' i got two sets of shards | 17:14 |
| mcape | https://bugs.launchpad.net/swift/+bug/1808001 | 17:19 |
| openstack | Launchpad bug 1808001 in OpenStack Object Storage (swift) "swift-manage-shard-ranges inject shards into the container that is in process of sharding" [Undecided,New] | 17:19 |
| *** gyee has joined #openstack-swift | 17:22 | |
| *** gyee has quit IRC | 17:56 | |
| notmyname | coreycb: we've got some people from red hat contributing to the py3 port for their own distro needs and timetables. if there are people from canonical who can also help, it's much more likely we can run under py3 by your own deadlines | 18:00 |
| *** mvkr has quit IRC | 18:04 | |
| *** e0ne has joined #openstack-swift | 18:10 | |
| coreycb | notmyname: well it's not our deadline. it's https://pythonclock.org/ really. we're contributing as much as we can upstream at the moment. | 18:16 |
| coreycb | notmyname: we've been fixing a lot of issues we're hitting in py3 across all the openstack projects. | 18:16 |
| notmyname | sounded like you were saying the next lts release deadline is different than either the train release cycle or the python 2020 clock. sounded like it was sooner than both | 18:17 |
| coreycb | notmyname: it is and that's because distros will be releasing prior to python 2.7 EOL but will support beyond the EOL | 18:19 |
| notmyname | yeah. TBH the biggest py3 driver from corporate sponsors (ie people paying people to work on swift) is from distros. so it's prioritization and enough people to help out (both writing and reviewing). so anything canonical can do in swift to help out with py3 would be great. | 18:26 |
| clayg | rledisez: how'd you even notice p 611614 ??? | 18:39 |
| patchbot | https://review.openstack.org/#/c/611614/ - swift - Fix SSYNC concurrency on partition - 5 patch sets | 18:39 |
| clayg | debug logging and trying to line up the REPLICATE requests like in ELK or something? Do you have some out-of-tree instrumented metrics telling you something special about what's going on with partition re-hashing? | 18:41 |
| openstackgerrit | Tim Burke proposed openstack/swift master: s3 secret caching https://review.openstack.org/603529 | 18:44 |
| openstackgerrit | Tim Burke proposed openstack/swift master: s3token: When caching is enabled, default auth_type to password https://review.openstack.org/624462 | 18:44 |
| *** sorrison has quit IRC | 18:47 | |
| *** sorrison has joined #openstack-swift | 18:48 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: s3token: Raise error on negative secret_cache_duration config https://review.openstack.org/624464 | 18:48 |
| notmyname | timburke: those are rebases right? | 18:53 |
| timburke | notmyname: the base patch is a rebase. the other two are follow-ups | 18:53 |
| notmyname | ah I see. thanks | 18:53 |
| notmyname | kota_: mattoliverau: zaitcev: clayg: timburke: tdasilva: now that the s3api patches have landed, I want to get a release tagged (2.20.0). I update https://wiki.openstack.org/wiki/Swift/PriorityReviews and I'll be working on the authors/changelog | 18:54 |
| timburke | might still want a follow-up for https://review.openstack.org/#/c/603529/10/swift/common/middleware/s3api/s3token.py@289 before a release, but i stand by the idea that swift is better with the patch as-is than without | 18:55 |
| patchbot | patch 603529 - swift - s3 secret caching - 11 patch sets | 18:55 |
| notmyname | it will be a couple of days at least, so a few more patches can land. I would like to see it tagged early next week at the latest (eg one week from right now) | 18:55 |
| *** NM has quit IRC | 19:11 | |
| *** NM has joined #openstack-swift | 19:14 | |
| openstackgerrit | Merged openstack/swift master: Fix container sharding doc https://review.openstack.org/624432 | 19:47 |
| *** NM has quit IRC | 20:01 | |
| *** NM has joined #openstack-swift | 20:04 | |
| timburke | clayg: got anything else to add on https://review.openstack.org/#/c/606192/ ? seems like you'd at least thought about it a *little*... | 20:29 |
| patchbot | patch 606192 - swift - Ignore ENOENT and ENOTEMPTY errors in delete_parti... - 1 patch set | 20:29 |
| clayg | timburke: i probably spent a bunch of time trying to convince myself with a test that the tpool will re-raise the exception cleanly and gave up | 20:31 |
| clayg | but i guess if it was logging it's probably fine | 20:32 |
| *** NM has quit IRC | 20:32 | |
| clayg | why didn't pete merge it | 20:32 |
| timburke | clayg: fwiw, torgomatic did some nice digging on the eventlet history in the comments on https://review.openstack.org/#/c/518783/ | 20:35 |
| patchbot | patch 518783 - swift - Get rid of tpool_reraise (MERGED) - 4 patch sets | 20:35 |
| clayg | great | 20:37 |
| openstackgerrit | Merged openstack/swift master: s3 secret caching https://review.openstack.org/603529 | 20:55 |
| notmyname | tdasilva: ugh. https://review.openstack.org/#/c/611614 | 20:55 |
| patchbot | patch 611614 - swift - Fix SSYNC concurrency on partition - 5 patch sets | 20:56 |
| *** mvkr has joined #openstack-swift | 21:18 | |
| timburke | notmyname: if we're thinking about releases for stable branches, too, https://review.openstack.org/#/c/604937/ might be worth bringing back to rocky. and https://review.openstack.org/#/c/607723/. maybe https://review.openstack.org/#/c/619502/ (cc rledisez)? probably https://review.openstack.org/#/c/621727/ | 21:22 |
| patchbot | patch 604937 - swift - Allow kmip_keymaster to be configured in proxy-ser... (MERGED) - 1 patch set | 21:22 |
| patchbot | patch 607723 - swift - Prevent kmip_protocol logger from logging at DEBUG (MERGED) - 3 patch sets | 21:22 |
| patchbot | patch 619502 - swift - Use eventlet.patcher.original to get Python select... (MERGED) - 2 patch sets | 21:22 |
| patchbot | patch 621727 - swift - fixed _check_node() in the container sharder (MERGED) - 5 patch sets | 21:22 |
| notmyname | yeah, it would be a good idea to tag stable releases too | 21:22 |
| timburke | how far back do we want to go? rocky, yes; queens, probably; pike, maybe; ocata...? | 21:23 |
| sorrison | timburke: we upgraded to swift to 2.17.0 and all our issues with the 2.15.1 release are gone | 21:24 |
| sorrison | the pike release in the ubuntu cloud archive is pretty buggy and shouldn't be used | 21:24 |
| timburke | yay! newer is always better ;-) | 21:24 |
| notmyname | timburke: we're only required to backport to pike at this point. but we could do ocata. newton is right out. (https://releases.openstack.org) | 21:24 |
| timburke | so, yeah -- might be good for us to tag something better for pike | 21:25 |
| notmyname | timburke: updated https://wiki.openstack.org/wiki/Swift/PriorityReviews for tracking | 21:26 |
| timburke | idk -- certainly the sharder guy's not going back to ocata | 21:27 |
| notmyname | sure. I just copy/pasted. I figure having a "N/A" as needed there will be good for answering if it's needed at a glance | 21:28 |
| timburke | i feel like a better way to track is to (1) say yes, this thing is worth backporting, (2) propose backports to affected branches, and (3) track in gerrit. but what do i know... | 21:28 |
| timburke | if you don't already, you should really have a link like https://review.openstack.org/#/dashboard/?title=Open+Backports&foreach=is:open+branch:%255Estable/.*&Swift=project:openstack/swift&Swift+Client=project:openstack/python-swiftclient off your top gerrit menu ;-) | 21:30 |
| notmyname | there's one in the review dashboard. hmm... doesn't seem to include swiftclient? that's odd | 21:30 |
| timburke | maybe ignores bot-proposed patches? | 21:31 |
| notmyname | ah. there's an "is:mergable" filter | 21:32 |
| *** e0ne has quit IRC | 21:47 | |
| *** mcape has quit IRC | 21:52 | |
| *** itlinux_ has quit IRC | 21:57 | |
| *** rcernin has joined #openstack-swift | 21:59 | |
| timburke | might be worth backporting https://review.openstack.org/#/c/571903/ for pre-rocky... | 22:15 |
| patchbot | patch 571903 - swift - copy: Stop hanging per-request data on middleware ... (MERGED) - 2 patch sets | 22:15 |
| timburke | otoh, i don't know that we've ever actually *observed* the bad behavior -- pretty sure i just spotted it while reading code | 22:17 |
| openstackgerrit | Merged openstack/swift master: Ignore ENOENT and ENOTEMPTY errors in delete_partition https://review.openstack.org/606192 | 22:20 |
| *** itlinux has joined #openstack-swift | 22:26 | |
| *** itlinux has quit IRC | 22:45 | |
| openstackgerrit | Merged openstack/swift master: Fix SSYNC concurrency on partition https://review.openstack.org/611614 | 23:34 |
| openstackgerrit | Tim Burke proposed openstack/swift master: Add slo_manifest_hook callback https://review.openstack.org/609860 | 23:50 |
| openstackgerrit | Tim Burke proposed openstack/swift master: s3api: Stop relying on container listings during multipart uploads https://review.openstack.org/610747 | 23:50 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
