| *** sri_ has quit IRC | 01:50 | |
| *** sri_ has joined #openstack-swift | 01:51 | |
| *** gyee has quit IRC | 02:10 | |
| *** rcernin has quit IRC | 02:43 | |
| *** rcernin has joined #openstack-swift | 03:12 | |
| *** psachin has joined #openstack-swift | 03:19 | |
| *** rcernin has quit IRC | 03:37 | |
| *** rcernin has joined #openstack-swift | 04:02 | |
| *** rcernin has quit IRC | 04:10 | |
| *** rcernin has joined #openstack-swift | 04:10 | |
| *** irclogbot_0 has quit IRC | 04:27 | |
| *** irclogbot_2 has joined #openstack-swift | 04:33 | |
| *** evrardjp has quit IRC | 05:33 | |
| *** evrardjp has joined #openstack-swift | 05:33 | |
| *** gregwork has quit IRC | 06:05 | |
| *** rcernin has quit IRC | 06:39 | |
| *** dsariel has joined #openstack-swift | 06:50 | |
| openstackgerrit | Merged openstack/swift master: Add tasks_per_second option to expirer https://review.opendev.org/755894 | 06:55 |
|---|---|---|
| *** rpittau|afk is now known as rpittau | 08:15 | |
| *** rcernin has joined #openstack-swift | 08:28 | |
| *** mcape2 has joined #openstack-swift | 08:43 | |
| *** rcernin has quit IRC | 08:45 | |
| *** mcape2 has quit IRC | 08:58 | |
| *** mcape2 has joined #openstack-swift | 09:01 | |
| *** rcernin has joined #openstack-swift | 09:03 | |
| *** rcernin has quit IRC | 09:17 | |
| zigo | timburke: DHE: I can now confirm that tweaking logging/__init__.py in Python 3.7 standard lib to disable logthread fixed the stuck proxy for me. No gateway timeout for the last 12 hours ... | 10:25 |
| DHE | I'm on py 3.6 but same deal | 10:45 |
| mcape2 | Hello guys! First of all, thanks for the Swift - it is a great software (as we figured when 1/3 of our disks went down, but no one user noticed). | 10:59 |
| mcape2 | I have two questions that I can't find the answers in google. | 10:59 |
| mcape2 | We run a ~200Tb cluster of ~150 million files, ~120 disks, 2 regions, 3 zones. | 10:59 |
| mcape2 | 1) I'm trying to monitor cluster errors, and most commonly occuring are timeouts, of following three types: | 11:00 |
| mcape2 | ERROR with Object server 172.31.3.18:6000/device15 re: Trying to HEAD /v1/AUTH_xxxxxxxx/some_file.gz: Timeout (10.0s) | 11:00 |
| mcape2 | ERROR with Object server 172.31.3.15:6000/device2 re: Trying to GET /v1/AUTH_xxxxxxxx/another_file.gz: ConnectionTimeout (0.5s) | 11:00 |
| mcape2 | ERROR with Object server 172.31.3.13:6000/device24 re: Trying to get final status of PUT to /v1/AUTH_xxxxxxxx/openprocurement-storage/some_other_file.gz: Timeout (10.0s) | 11:00 |
| mcape2 | they are spread more or less evenly on all disk/nodes and in time... | 11:00 |
| mcape2 | errors occuring at rate ~100-200 in a hour. Load to the cluster is not big, disks are utilised less than 20% (apart from disks that are under replication, they are 100% utilised ) | 11:00 |
| mcape2 | we are seeing no user-facing errors, but its nice to know what is going on... | 11:01 |
| mcape2 | and second question is 2) What is normal/recommended replication pass total time? | 11:01 |
| *** alecuyer has quit IRC | 12:00 | |
| *** rledisez has quit IRC | 12:00 | |
| *** alecuyer has joined #openstack-swift | 12:03 | |
| *** rledisez has joined #openstack-swift | 12:04 | |
| *** ChanServ sets mode: +v rledisez | 12:04 | |
| *** djhankb has quit IRC | 12:37 | |
| *** djhankb has joined #openstack-swift | 12:38 | |
| zigo | timburke: DHE: It looks like uwsgi works kind of well, so I'll try it in production. I'm currently setting the Debian package to use that by default, at least for the proxy-server for now. | 12:53 |
| zigo | What would be the recommended number of threads ? Just a single one ? Or what ? | 13:06 |
| zigo | Will having multiple threads make me fall in the same trap with eventlet and threads logs ? | 13:07 |
| *** cschwede has joined #openstack-swift | 13:46 | |
| *** ChanServ sets mode: +v cschwede | 13:46 | |
| *** mcape2 has quit IRC | 14:04 | |
| *** JohnBeales has joined #openstack-swift | 15:28 | |
| *** JohnBeales has quit IRC | 15:51 | |
| *** rpittau is now known as rpittau|afk | 15:54 | |
| *** JohnBeales has joined #openstack-swift | 15:59 | |
| kota_ | that was great PTG! | 16:00 |
| seongsoocho | 👍 | 16:00 |
| seongsoocho | see you next PTG in a person! | 16:00 |
| *** tdasilva has quit IRC | 16:20 | |
| *** tdasilva has joined #openstack-swift | 16:20 | |
| *** ChanServ sets mode: +v tdasilva | 16:20 | |
| *** tdasilva has quit IRC | 16:23 | |
| *** tdasilva_ has joined #openstack-swift | 16:23 | |
| *** ChanServ sets mode: +v tdasilva_ | 16:23 | |
| *** thiago__ has joined #openstack-swift | 16:26 | |
| *** ChanServ sets mode: +v thiago__ | 16:26 | |
| *** tdasilva_ has quit IRC | 16:29 | |
| *** JohnBeales has quit IRC | 16:37 | |
| *** cschwede has quit IRC | 16:53 | |
| *** JohnBeales has joined #openstack-swift | 17:37 | |
| *** JohnBeales has quit IRC | 17:52 | |
| *** JohnBeales has joined #openstack-swift | 17:56 | |
| *** JohnBeales has quit IRC | 18:01 | |
| *** psachin has quit IRC | 18:23 | |
| *** JohnBeales has joined #openstack-swift | 18:40 | |
| *** JohnBeales has quit IRC | 18:45 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: Add option to REPLICATE to just invalidate hashes https://review.opendev.org/758636 | 18:48 |
| *** JohnBeales has joined #openstack-swift | 19:20 | |
| *** JohnBeales has quit IRC | 19:34 | |
| ormandj | if an object is not on disk, but it is in the container listing, issuing a DELETE shoudl sort it, right? because we're getting 404s on DELETE requests | 19:56 |
| ormandj | but it's plainly in _one_ of the three container dbs | 19:56 |
| ormandj | which, btw, none have the same object count | 19:56 |
| ormandj | even after a replication pass :) | 19:57 |
| timburke | yep, the 404 is expected and the object server will still update the container | 20:19 |
| ormandj | awesome | 20:19 |
| ormandj | thanks tim | 20:19 |
| ormandj | we see object counts dropping | 20:19 |
| ormandj | we added functionality to swift-account-audit to output audit info in json for found/notfound, any tombstone dates if they exist, and then output a batch of swift commands to nuke the stuff from the container that isn't on disk | 20:20 |
| ormandj | then we're just using parallel to run it all in parallel | 20:21 |
| *** dasp_ has joined #openstack-swift | 22:26 | |
| *** dasp has quit IRC | 22:28 | |
| zigo | timburke: I got the proxy-server working under uwsgi, however, I wonder, should I activate threading? How many threads? | 23:54 |
| zigo | How will it perform compared to eventlet? | 23:55 |
| zigo | I'd like to put sensible defaults in the Debian package, you know... | 23:56 |
| zigo | I currently have this: http://paste.openstack.org/show/799574/ | 23:58 |
| zigo | and "processes" is controled by what's in /etc/swift/proxy-server.conf (ie: "workers" control what the init script does for the --processes argument). | 23:58 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!