Friday, 2020-05-08

*** renich has quit IRC00:24
donnydwhat would be the best way to disable public reading of large array of objects00:32
donnydnevermind00:39
donnydthat was easier than I though00:39
seongsoocho hi, I'm in emergency ..... about a few hours ago, All container server's disk are suddenly full. I add more node but the disk usage does not getting down.... Is there any solution?01:44
timburke*all* of them? what's the ratio like between container disk space vs object disk space? the usual sort of failure we see with containers and full drives is that a handful get *very* full, but most of them are still fairly empty...02:12
timburkeseparately (but maybe kinda related?) i wonder if there should be a skip_commits=True around https://github.com/openstack/swift/blob/2.25.0/swift/container/sharder.py#L1628-L1629 ...02:13
seongsoochotimburke:  yes all of them. container server running alone.02:16
seongsoochothe total container disk's size is 800GB and object disk are 1PB02:18
timburkeso replication pretty much *always* leads to some increased disk usage during an expansion -- gotta make sure the data's been replicated everywhere before we unlink. there are some emergency replication options like handoffs_only that were born out of some cluster-full incidents and needing a way to quickly process handoffs (and *only* handoffs) so we can unlink faster02:21
timburkehow full are the new disks?02:21
seongsoochotimburke:  ok then..  Set handoffs_only = true in all container node and restart replicator?02:22
seongsoochohandoffs_only = yes02:23
timburkeyeah -- you'll want to watch your logs for when the replicators start complaining that they didn't find any work to do and reassess the situation02:24
seongsoochook thanks I will try02:24
timburkeif disks are still full and even the new guys are looking bad, we've gotta worry. if things are coming down, flip it back off and life goes on :-)02:25
seongsoochotimburke:  the new guys disk are increasing now.. but  the original one still full. I wil wait02:26
timburkefwiw, i've got ~470T in SSDs for ~25P in spinners (both raw, not usable) -- but those SSDs are pretty empty. lemme get an estimate on what the actual used numbers are...02:28
*** manuvakery has joined #openstack-swift02:35
timburkeit's kinda fuzzy -- there's an object policy on most of them that brings the usage up a good bit, like 18T of the 470T is filled probably? 3-4% or so. there are a couple disks *not* in that object policy; they're sitting at like 0.6%, but i'm not sure that's actually representative; feels like it might be low02:36
timburkefwiw, i'd treat container disks kinda like object in that you don't really want them to get more than 75-80% full if you can avoid it02:54
*** psachin has joined #openstack-swift03:00
seongsoochotimburke:  the situation are getting better, the disk usage is decreasing....03:07
seongsoocho;;;;;; what a terrible friday morning..03:07
seongsoochotimburke:  thanks for your help. I really appreciate you03:08
timburke\o/ glad it's working03:55
timburkeand hey, could be worse -- could've been a late friday night ;-)03:55
seongsoocho:-)   hope to happy friday nigh.04:17
*** evrardjp has quit IRC04:36
*** evrardjp has joined #openstack-swift04:36
*** psachin has quit IRC04:51
timburkealecuyer, rledisez: i'm about to go to bed, but something to think about with the updaters -- what's your container-replicator cycle time like? we just intentionally *lengthened* ours, which seems to be having good effect on our updaters' success/failure ratio05:40
timburketheory is that we used to have it tuned too fast, causing the replicator and/or sharder to flush the db.pending file more often than was helpful05:40
timburke(side note: we should probably make https://github.com/openstack/swift/blob/2.25.0/swift/common/db.py#L52-L54 configurable. especially since that value predates encryption and updates have gotten bigger)05:42
seongsoochoAfter recovery all container-server, the container replicate log some db fail to replicate....  Should I delete that db file and wait for re-replicate?06:08
seongsoochoMay  8 15:06:43 cnfswf-sto003.svr.toastmaker.net container-replicator: Synchronization for /srv/node/sdb/containers/2124/776/84cb2e263a504ce3f48a81135e53d776/84cb2e263a504ce3f48a81135e53d776.db has fallen more than 100000 rows behind; moving on and will try again next pass06:08
timburkeno, that's fine -- it should catch up eventually, it just may take some time. probably worth checking to see how many rows are in the DB, and what sync points are recorded06:35
seongsoochotimburke:  thanks . and good night :-)06:36
timburkeit made progress on those 100k; it just doesn't want to be stuck on this one db for too long. if it seems like it's really holding you up, you could consider upping max_diffs: https://github.com/openstack/swift/blob/master/etc/container-server.conf-sample#L154-L15806:37
timburkeidk that i've ever bothered to do that, though06:38
timburkeand you're right, i should get to bed ;-)06:38
timburkegood night! enjoy the rest of your day06:38
*** dtantsur|afk is now known as dtantsur07:39
*** manuvakery has quit IRC08:05
*** tkajinam has quit IRC08:13
*** takamatsu has joined #openstack-swift08:29
*** ccamacho has joined #openstack-swift09:31
*** rcernin has quit IRC09:39
*** threestrands has quit IRC09:41
*** rcernin has joined #openstack-swift10:22
viks____I see lot of errors like :10:22
viks____```10:22
viks____container-server: ERROR __call__ error with DELETE /disk2/1552/5cc5358c50e34a3196eca65ea9039633/backup/Test/Archive/test/9bf82551-aa58-4efd-b045-263c5bd20f50/01f410c3-25fe-ec74-2491-88db97686035/blocks/42a1b2043d580213243c531152cfe1e0/34457.8b7f06a499d03d21dfe659ccc6d9e75d.028a8da294d30174d43667c98d9586c8.blk : LockTimeout (10s) /srv/node/disk2/containers/1552/e51/610717f4ac1d0c3ca1212deb49299e51/.lock (txn:10:22
viks____tx3c75433f0a284425bbb69-005eb52e70)10:22
viks____```10:22
viks____when s3api bulk delete is in action with a container with 2 million objects and the container db file size is also around 3GB10:22
viks____I also see the below in container server configuration under `[container-sharder]`:10:22
viks____```10:22
viks____# Large databases tend to take a while to work with, but we want to make sure10:22
viks____# we write down our progress. Use a larger-than-normal broker timeout to make10:22
viks____# us less likely to bomb out on a LockTimeout.10:22
viks____# broker_timeout = 6010:22
viks____```10:22
viks____So do can i set similar timeout for container-server as well? or how do i overcome this error?10:22
*** tkajinam has joined #openstack-swift12:27
*** mpasserini has joined #openstack-swift12:43
mpasseriniHi there. Is there a way to trace in the log who changed a Swift ACL on a given container, and how he changed it? I only see logs about "POST" but no details of contents.12:43
*** jv_ has quit IRC13:02
*** jv_ has joined #openstack-swift13:05
DHEA POST or a PUT are really about the ways to do it.13:30
claygmpasserini: there's no audit trail of the contents of the requests, there are some options in some of the logging middlewares to capture headers out to logs you could investigate15:11
claygviks____: Anyone that's ever run into contention on container databases has seen the LockTimeout - a longer timeout isn't really a good solution for the container server (10s is already a long time)15:12
claygviks____: we gave the sharder a pass for the longer timeout mainly for legacy support of containers that were 100's of gigabytes before sharding existed and the sharder has some unique long running operations15:13
claygviks____:  3GB for a container database is not that "big" IME - I think it would be difficult to hit the timeout unless... are the containers maybe on spinning disks?15:15
claygI got a DM that looked kinda interesting.  Is anyone else working on s3 acl's right now?  https://github.com/openstack/swift/compare/master...idrive-online-backup:feature/bucket-policy15:18
*** tkajinam has quit IRC15:22
viks____clayg: Thanks for the reply... So what action should i take for such error? my drives are SAS drives, but i used 1 nvme ssd drive to check if this goes away... in that also i see this error..15:25
claygyou get a TEN SECOND lock timeout with a 3 GB database on an nvme drive?  In 10s an nvme could read the entire DB into memory and write it back out ... 100 times15:30
viks____clayg: Hmmm... i'm not sure what else could be causing this.. 🤔15:32
claygso it's hard to run dbs on HDDs - keeping them small helps keep them fast15:33
claygwhat's the io utilization of the disk under the database?15:34
viks____container db disk utilization is not going beyond 5%...  also  if you check the error above it's basically timeout happening on object path...15:39
*** renich has joined #openstack-swift15:40
*** mpasserini has quit IRC15:40
*** gyee has joined #openstack-swift15:41
viks____sorry... its's some path in container disk itself...15:42
viks____what is the file with `.blk` extension it  is trying to lock?15:47
timburkeviks____, so the container DB is trying to process an object DELETE -- that is, an object-server or object-updater is trying to tell the DB that a delete occurred so the object stops showing up in listings16:10
*** jv_ has quit IRC16:15
timburkehow big does /srv/node/disk2/containers/1552/e51/610717f4ac1d0c3ca1212deb49299e51/610717f4ac1d0c3ca1212deb49299e51.db.pending get? it should keep growing and shrinking, growing and shrinking as the container server scribbles down updates then batch-loads them16:15
timburkethere's a cap around 128k, but some operations will cause it to flush early16:16
claygviks____: `.blk` is part of the object name , not a fs path16:26
viks____timburke: Thanks... i see it growing and shrinking not necessarily reaching 12k limit...16:28
viks____clayg: ok16:28
viks____sorry ... 128K limit..16:31
viks____is that error messages can be safely ignored?16:34
*** evrardjp has quit IRC16:36
*** evrardjp has joined #openstack-swift16:36
*** dtantsur is now known as dtantsur|afk16:42
timburkeviks____, so if it's never even getting close to the limit, your container replicator/sharder/auditor might be too aggressive. just last night i dropped the dbs_per_second for my replicators and sharders so i could make some headway on async pendings; seems to have greatly reduced the number of updater failures17:29
zaitcevBTW, I gave up on https://github.com/eventlet/eventlet/issues/52617:42
zaitcevI configured an additional region where identity has no SSL, and told Swift to use that region. That avoids the crash.17:43
zaitcevFortunately, regions mean nothing for tokens. So, the client obtains token using RegionOne, passes it to Swift, keystonemiddleware talks to Keystone using RegionX and passes the token for verification, and it works. No region scoping.17:44
*** manuvakery has joined #openstack-swift17:56
*** renich has quit IRC18:05
timburkei really ought to learn how to setup/use wireguard -- i feel like that'll be a great idea for all intra-cluster traffic, including swift <-> keystone18:57
claygviks____: I don't think you should ignore it - it's telling you something important: "a DELETE request to this database took >10s so I gave up and returned 500" - realistically there's not a lot of good reasons for anything to take that long19:11
claygviks____: you say the database isn't big, you say the disk isn't busy - those are the reasons WE'VE ALL SEEN for things to slow down (and sharding or more iops solves them) - but if NOT those it's something more sinister which is even MORE interesting19:12
claygviks____: but you know... it could just be this -> https://bugs.launchpad.net/swift/+bug/187765119:28
openstackLaunchpad bug 1877651 in OpenStack Object Storage (swift) "Reclaim of tombstone rows is unbounded and causes LockTimeout (10s)" [Medium,New] - Assigned to clayg (clay-gerrard)19:28
*** benj_ has quit IRC19:55
*** benj_ has joined #openstack-swift19:56
*** manuvakery has quit IRC20:06
*** renich has joined #openstack-swift20:55
*** ccamacho has quit IRC21:02
*** mikecmpbll has joined #openstack-swift22:46
*** jv_ has joined #openstack-swift23:19
*** jv_ has quit IRC23:25

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!