*** mvkr has joined #openstack-swift | 00:12 | |
*** gyee has quit IRC | 00:16 | |
openstackgerrit | Merged openstack/swift master: s3api: Add basic support for ?versions bucket listings https://review.openstack.org/575838 | 00:19 |
---|---|---|
*** pcaruana has quit IRC | 00:25 | |
kota_ | good morning | 01:06 |
kota_ | thanks timburke and tdasilva for addressing issues on p 575838 | 01:07 |
patchbot | https://review.openstack.org/#/c/575838/ - swift - s3api: Add basic support for ?versions bucket list... (MERGED) - 4 patch sets | 01:07 |
timburke | kota_: thanks for looking at it! it definitely improved from where i started with it | 01:07 |
mattoliverau | kota_: morning | 01:30 |
openstackgerrit | Matthew Oliver proposed openstack/python-swiftclient master: Clean up work dir before running func tests https://review.openstack.org/611471 | 01:58 |
openstackgerrit | Matthew Oliver proposed openstack/python-swiftclient master: Stop leaking quite so many connections https://review.openstack.org/577584 | 01:58 |
mattoliverau | ^ Just a rebase | 01:58 |
*** nguyenhai_ has joined #openstack-swift | 04:16 | |
*** nguyenhai has quit IRC | 04:19 | |
*** openstackstatus has quit IRC | 04:59 | |
*** openstack has joined #openstack-swift | 07:09 | |
*** ChanServ sets mode: +o openstack | 07:09 | |
*** pcaruana has joined #openstack-swift | 07:21 | |
*** e0ne has quit IRC | 07:31 | |
*** admin6 has joined #openstack-swift | 07:33 | |
*** admin6 has quit IRC | 07:36 | |
*** e0ne has joined #openstack-swift | 09:39 | |
*** e0ne has quit IRC | 10:56 | |
*** e0ne has joined #openstack-swift | 11:44 | |
openstackgerrit | chenxiangui proposed openstack/swift master: Fix typo https://review.openstack.org/616890 | 12:10 |
openstackgerrit | chenxiangui proposed openstack/swift master: Fix typo https://review.openstack.org/616895 | 12:32 |
*** mikecmpbll has joined #openstack-swift | 14:53 | |
*** jistr is now known as jistr|call | 16:00 | |
clayg | hi folks | 16:11 |
rledisez | good morning clayg | 16:12 |
*** mikecmpbll has quit IRC | 16:15 | |
*** mikecmpb_ has joined #openstack-swift | 16:15 | |
*** jistr|call is now known as jistr | 16:17 | |
*** gyee has joined #openstack-swift | 16:22 | |
*** mikecmpb_ has quit IRC | 16:32 | |
*** mikecmpbll has joined #openstack-swift | 16:34 | |
*** e0ne has quit IRC | 16:35 | |
*** portante has joined #openstack-swift | 16:38 | |
portante | cschwede, notmyname, tdasilva, clayg: hi folks | 16:39 |
portante | perhaps a quick question ... do you know of a web server technology that will serve the contents of a tar file just like a file system hierarchy could be served? | 16:40 |
portante | so say I had a tar ball in an object store, but wanted to browse it and look at a file or two in it? | 16:40 |
DHE | the problem with a tarball is there's no index. building a file list would require fetching the whole file. that's rather suboptimal | 16:55 |
*** mikecmpbll has quit IRC | 16:57 | |
timburke | might be able to do something with range requests... but it'll be slow; each range will depend on having read the previous header... | 17:41 |
timburke | could be an interesting further application for joeljwright's https://review.openstack.org/#/c/499260/ though... | 17:42 |
patchbot | patch 499260 - swift - WIP: Add TLO middleware - 7 patch sets | 17:42 |
timburke | would still require have the constituent files already splayed out, though :-/ | 17:43 |
*** d34dh0r53 has joined #openstack-swift | 17:46 | |
openstackgerrit | Tim Burke proposed openstack/python-swiftclient master: Stop leaking quite so many connections https://review.openstack.org/577584 | 17:55 |
*** e0ne has joined #openstack-swift | 18:09 | |
openstackgerrit | Tim Burke proposed openstack/python-swiftclient master: Update reno for stable/rocky https://review.openstack.org/585823 | 18:25 |
*** e0ne has quit IRC | 18:33 | |
portante | DHE, timburke, thanks! | 18:33 |
*** e0ne has joined #openstack-swift | 18:36 | |
*** bigdogstl has joined #openstack-swift | 18:50 | |
*** MooingLe1ur is now known as MooingLemur | 18:53 | |
*** e0ne has quit IRC | 18:58 | |
*** bigdogstl has quit IRC | 19:08 | |
*** bigdogstl has joined #openstack-swift | 19:12 | |
*** zigo has quit IRC | 19:25 | |
*** bigdogstl has quit IRC | 19:26 | |
*** bigdogstl has joined #openstack-swift | 19:30 | |
*** bigdogstl has quit IRC | 19:35 | |
openstackgerrit | Merged openstack/swift master: py3: port account/container replicators https://review.openstack.org/614656 | 19:43 |
*** bigdogstl has joined #openstack-swift | 19:53 | |
*** bigdogstl has quit IRC | 19:57 | |
*** e0ne has joined #openstack-swift | 20:01 | |
*** e0ne has quit IRC | 20:04 | |
*** anovaes has joined #openstack-swift | 20:11 | |
anovaes | Hi people, I'm having a problem using swift with swift3 on the openstack pike. | 20:16 |
anovaes | I can use swift with swift api normaly, but with s3 api a get problem. | 20:16 |
anovaes | The short message is: SignatureDoesNotMatch | 20:16 |
anovaes | Can anybody help me? | 20:16 |
*** edmilsonssa has joined #openstack-swift | 20:17 | |
notmyname | anovaes: what s3 client are you using? | 20:26 |
*** edmilsonssa has quit IRC | 20:26 | |
anovaes | notmyname: using aws, s3curl and s3cmd | 20:27 |
anovaes | awscli* | 20:27 |
timburke | do you know what region your client is using? does it match the `location` configured for swift3? | 20:28 |
timburke | adding a --debug flag to the command line is probably going to be helpful, too | 20:30 |
notmyname | ah, good call. didn't older versions of swift3 default to a non-standard aws region name? | 20:30 |
timburke | notmyname: still do: https://github.com/openstack/swift/blob/2.19.0/swift/common/middleware/s3api/s3api.py#L109 | 20:30 |
notmyname | bah! I thought we changed that | 20:31 |
notmyname | we only changed it in our product, didn't we? | 20:31 |
timburke | iirc we had some reason to think it would work OK for s3cmd, though? fwiw, one of the things i pretty much always do is switch that to us-east-1 | 20:31 |
*** edmilsonssa_ has joined #openstack-swift | 20:32 | |
anovaes | timburke: my region is RegionOne, I have keystone and swift, only. The region was configured on filter s3token in proxy-server.conf | 20:33 |
timburke | swift3 has its own idea of region, configured with the option `location` :-/ | 20:34 |
timburke | it gets used as part of the v4 signing process | 20:34 |
edmilsonssa_ | @timburke I thought Keystone region and Swift location should match, doesn't? | 20:35 |
timburke | fwiw, after the transition to s3api we landed some patches to improve SignatureDoesNotMatch messages, and provide separate errors if the region doesn't match -- but that's newer swift | 20:35 |
timburke | edmilsonssa_: maybe? they certainly don't have to, though. if you don't explicitly configure swift3's location, it defaults to "US" | 20:37 |
edmilsonssa_ | timburke: ok, and what else could lead to SignatureDoesNotMatch ? | 20:39 |
timburke | swift in general (as opposed to the swift3/s3api middlewares) doesn't really expose "region" to the client... it's just a layer in the failure-domain hierarchy, and not necessarily tied to keystone's notions of regions | 20:40 |
edmilsonssa_ | ok, got it. but we are geting SignatureDoesNotMatch and we've have checked the config many times and didn't find anything suspicious. we're going nuts :) | 20:41 |
timburke | basically, either the user can't be found (so we don't have a secret to use in signing), the secrets don't match (so of course the signatures don't), or the string that's getting signed doesn't match (which can be a bug in our code, a bug in the client's code, or a misconfiguration like the S3-notion of region not matching) | 20:42 |
edmilsonssa_ | we've seen some people configuring the client with a combined string of project:user_id. does it make sense? | 20:42 |
timburke | makes sense for something like tempauth, but as i recall keystone's ec2 credentials (which we piggy-back off of) don't usually follow that form... | 20:43 |
timburke | what do your swift logs say? maybe turn on debug logging and restart the proxy server first. we should be able to see what we're sending to keystone and what response we're getting back at debug level | 20:44 |
edmilsonssa_ | we are passing access and secret, we don't use tempauth in our pipeline config | 20:45 |
edmilsonssa_ | good, how do we enable debug for this? | 20:46 |
timburke | in the [DEFAULT] section of proxy-server.conf, set log_level=DEBUG | 20:46 |
anovaes | OK | 20:47 |
edmilsonssa_ | let's try | 20:47 |
anovaes | we go configure | 20:47 |
anovaes | 2018-11-09 17:51:25.732 4903 INFO keystone.common.wsgi [req-85897ae5-1b74-4478-b701-4d78a3efe505 f35a85b996a54e17af1a305695340fa3 f8b8cba91c654277832b432ddf55ddcb - default default] GET http://keystone.ip:35357/v3/users/f35a85b996a54e17af1a305695340fa3/credentials/OS-EC2 | 20:55 |
anovaes | 2018-11-09 17:52:01.160 4904 INFO keystone.common.wsgi [req-1372fcb3-5e66-426d-b3ed-d37741b1c97b - - - - -] POST http://my.keystone.ip::35357/v2.0/s3tokens | 20:55 |
anovaes | 2018-11-09 17:52:01.169 4904 WARNING keystone.common.wsgi [req-1372fcb3-5e66-426d-b3ed-d37741b1c97b - - - - -] Authorization failed. Credential signature mismatch (Disable insecure_debug mode to suppress these details.) from my.swift-proxy.ip: Unauthorized: Credential signature mismatch (Disable insecure_debug mode to suppress these details.) | 20:55 |
*** bigdogstl has joined #openstack-swift | 20:59 | |
edmilsonssa_ | timburke: Nov 9 18:05:56 swfpxd001 proxy-server: {'access_key': 'c564e802a5d849b99b180593e5520ea0', 'account': None, '_timestamp': 1541797556.00000_0000000000000000, 'user_id': None, 'slo_enabled': True, 'container_name': '', 'string_to_sign': 'GET\n\napplication/x-www-form-urlencoded\nSex, 09 Nov 2018 21:05:56 +0000\n/', '_params_cache': {}, 'headers': <swift.common.swob.HeaderEnvironProxy object at 0x7f64f9d62e90>, | 21:08 |
*** bigdogstl has quit IRC | 21:09 | |
*** bigdogstl has joined #openstack-swift | 21:13 | |
anovaes | Can charset is a problem? | 21:14 |
timburke | hmm... access key looks like a keystone ec2 cred... string_to_sign looks sane for a v2 request, which wouldn't need to think much about regions... | 21:18 |
*** bigdogstl has quit IRC | 21:18 | |
timburke | the secret key the clent's using is the one you got back when creating the credential, yeah? not the keystone user's password? | 21:19 |
edmilsonssa_ | yes, it's the ec2 secret | 21:20 |
edmilsonssa_ | I can see it using "openstack ec2 credentials list" | 21:20 |
timburke | do the swift logs have something like https://github.com/openstack/swift3/blob/master/swift3/s3_token_middleware.py#L270-L271 ? "Connecting to Keystone sending ..." | 21:22 |
edmilsonssa_ | yes | 21:24 |
edmilsonssa_ | Nov 9 18:05:56 swfpxd001 proxy-server: Connecting to Keystone sending this JSON: {"credentials": {"access": "c564e802a5d849b99b180593e5520ea0", "token": "R0VUCgphcHBsaWNhdGlvbi94LXd3dy1mb3JtLXVybGVuY29kZWQKU2V4LCAwOSBOb3YgMjAxOCAyMTowNTo1NiArMDAwMAov", "signature": "thJHlKHic/wKX4ECBxYp9/AE51I="}} (txn: tx3e2a6c83f2604eccabb72-005be5f6b4) | 21:24 |
timburke | so you can try manually doing what keystone would be doing, with something like `import base64, hashlib, hmac; print base64.b64encode(hmac.new('your key', base64.b64decode('R0VUCgphcHBsaWNhdGlvbi94LXd3dy1mb3JtLXVybGVuY29kZWQKU2V4LCAwOSBOb3YgMjAxOCAyMTowNTo1NiArMDAwMAov'), hashlib.sha1).digest())` | 21:28 |
timburke | (substituting "your key" for your actual key, of course) | 21:28 |
timburke | and you can also try to get the debug output for the client, and checking that the client's notion of the "string to sign" matches what's being received | 21:29 |
timburke | one thing that seems odd to me is the date: "Sex, 09 Nov 2018 21:05:56 +0000" | 21:30 |
timburke | but if that's what the client's using too, it shouldn't be a problem... | 21:30 |
edmilsonssa_ | date match between proxy, keystone and client | 21:32 |
edmilsonssa_ | let me test manualy | 21:32 |
timburke | cool; i was just expecting something more like "Fri, 09 Nov 2018 21:05:56 +0000" | 21:33 |
edmilsonssa_ | that's why our server is installed with brazilian language settings ... | 21:35 |
edmilsonssa_ | may this be a problem? | 21:35 |
timburke | could be... though i thought we just took what the client provided when building the string to sign... so if the client sent a portuguese date header, i'd expect it to have signed with a portuguese date header in the string-to-sign... | 21:37 |
edmilsonssa_ | I've made it manually and the signatures doesn't match .. | 21:38 |
timburke | might be worth trying with something like `base64.b64encode(hmac.new('your key', base64.b64decode('R0VUCgphcHBsaWNhdGlvbi94LXd3dy1mb3JtLXVybGVuY29kZWQKU2V4LCAwOSBOb3YgMjAxOCAyMTowNTo1NiArMDAwMAov').replace('Sex,', 'Fri,'), hashlib.sha1).digest())` | 21:39 |
edmilsonssa_ | yes, client sends: string_to_sign': 'GET\n\n\nSex, 09Nov201820: 55: 01+0000\n/ | 21:39 |
timburke | note the replace(...) | 21:39 |
edmilsonssa_ | replace(...) where? | 21:40 |
edmilsonssa_ | got it.... | 21:40 |
edmilsonssa_ | desn't match either ... | 21:42 |
edmilsonssa_ | :( | 21:42 |
edmilsonssa_ | signature sent by client is: "signature": "sMVKbDrZckblWs9l6CUQMYDJ3EE=" | 21:42 |
edmilsonssa_ | calculated is wPys9vuUdv25tS6IcqkrMc2z0J0= | 21:42 |
edmilsonssa_ | or RPxckYM7Xb1oTbKUNeCeCSMd05M= w/o replace | 21:43 |
edmilsonssa_ | keystone log says "CredentialNotFound: Could not find credential" | 21:45 |
timburke | fwiw, when i use awscli on my machine with a --debug flag, i get output like http://paste.openstack.org/show/734501/ | 21:45 |
timburke | buried in there is the StringToSign, toward the end | 21:46 |
timburke | you could compare the sts you see in the cli against the sts you see in the log, try to figure out what was different (if anything) | 21:47 |
timburke | i forget whether keystone honestly can't find the credential at all or just can't find a credential *that can produce a valid signature* when it logs that... | 21:49 |
edmilsonssa_ | keystone log is: | 21:50 |
edmilsonssa_ | 2018-11-09 18:50:04.196 4905 WARNING keystone.common.wsgi [req-bc6a8b94-b7a5-476b-be30-7f0852f265c4 - - - - -] Authorization failed. Credential signature mismatch (Disable insecure_debug mode to suppress these details.) from 10.10.1.158: Unauthorized: Credential signature mismatch (Disable insecure_debug mode to suppress these details.) | 21:50 |
timburke | yeah, that makes it seem more like it could find the credential, but got a mismatch (which we manually verified is in some sense "expected") | 21:52 |
edmilsonssa_ | yeah, but can't figure out why they doesn't match. do the time in both servers need to be equal? | 21:55 |
edmilsonssa_ | we have 8 sec of diff | 21:56 |
timburke | they should be similar, but don't have to be exact. 8s should be fine; iirc if it hits 5min you'll start seeing RequestTimeTooSkewed errors | 21:56 |
timburke | can we get a client debug log, along with the proxy-server logs for the same time? i think the next step is to compare the string-to-sign from each side manually | 21:58 |
edmilsonssa_ | yes | 21:59 |
edmilsonssa_ | tim, anovaes sent to you in pvt... | 22:09 |
edmilsonssa_ | we are going to put in the paste.... | 22:10 |
edmilsonssa_ | better | 22:10 |
timburke | edmilsonssa_: i was just going to recommend that :-) | 22:10 |
anovaes | Hi | 22:11 |
anovaes | http://paste.openstack.org/show/734502/ | 22:11 |
anovaes | log client and server proxy | 22:11 |
timburke | httpt? should just be http, most likely? | 22:12 |
anovaes | sorry, one minute | 22:14 |
timburke | and yeah, looks like swift3 either doesn't have location set, or has it set to it's default of "US" -- you could try changing the client to use that too, or configure location=RegionOne in the [filter:swift3] section of proxy-server.conf | 22:14 |
timburke | (based on the 403 that showed up in the proxy log) | 22:15 |
timburke | just after "Calling Swift3 Middleware" there's a long line that includes `'string_to_sign': 'AWS4-HMAC-SHA256\n20181109T220825Z\n20181109/US/s3/aws4_request\n641722ee6c367ce8eb5af0ad3501e581ad7a0e0bb997bd3719384ec426f6e279'` | 22:17 |
timburke | that 20181109/US/s3/aws4_request is the giveaway | 22:17 |
timburke | still not sure why the v2 signature wasn't working though :-/ | 22:18 |
timburke | fwiw, you can force different signature versions via config, as at https://github.com/tipabu/swift3-demo/blob/master/.aws/config#L10 | 22:19 |
timburke | and with awscli-plugin-endpoint, you can avoid needing to put --endpoint-url on the command line all the time | 22:20 |
anovaes | sorry, we need change de desk | 22:25 |
timburke | no worries :-) | 22:26 |
anovaes | this new log http://paste.openstack.org/show/734503/ | 22:30 |
*** edmilsonssa has joined #openstack-swift | 22:31 | |
timburke | looks like we need a bit more of the server logs -- maybe try grepping for txf2d005b1812442e0b88f3 ? | 22:32 |
timburke | server log got cut off at 19:22:03, while the client request was ~19:26:25 | 22:32 |
anovaes | grep: http://paste.openstack.org/show/734505/ | 22:39 |
timburke | :-/ StringToSign looks right... | 22:41 |
timburke | oh, hey, yeah! "Keystone Reply: Status: 200" | 22:42 |
timburke | so why the 403...? | 22:42 |
timburke | is s3_acl enabled? what's your [filter:keystoneauth] section look like? | 22:44 |
edmilsonssa_ | maybe because we've put location in right place ... | 22:46 |
edmilsonssa_ | it was in s3token before ... | 22:46 |
anovaes | endpoint do swift on the keystone is http://swif-proxy.ip:8080/v1 | 22:47 |
anovaes | have problem? | 22:47 |
timburke | is there a reseller prefix configured for s3token? does it match the one (if any) for keystoneauth? | 22:47 |
timburke | nope, that should be fine -- at this point, i think we've eliminated the true mismatch problem (keystone responds 200, yay!) but we have some other authorization problem that still returns a SignatureDoesNotMatch error to the client | 22:48 |
timburke | oh, and it looks like reseller_prefix in s3token may need to have a trailing underscore... i think we've got some niceties in keystoneauth and tempauth to take things like AUTH and automatically add an underscore, but not so with s3token... | 22:51 |
*** bigdogstl has joined #openstack-swift | 22:51 | |
edmilsonssa_ | [filter:keystoneauth] has no reseller, but [filter:s3token] has reseller = AUTH | 22:54 |
timburke | try adding an underscore to the end | 22:58 |
openstackgerrit | Tim Burke proposed openstack/swift master: s3token: Add an underscore to the end of reseller_prefix if not present https://review.openstack.org/617043 | 23:00 |
*** bigdogstl has quit IRC | 23:03 | |
edmilsonssa_ | we've added. anovaes gonna send paste | 23:03 |
anovaes | Log client: http://paste.openstack.org/show/734507/ | 23:04 |
*** bigdogstl has joined #openstack-swift | 23:05 | |
timburke | next bit of debugging i'd add: in [filter:swift3], set `force_swift_request_proxy_log = true` and we can dig into what backend requests we're making a bit, make sure we're going to the account that we were expecting | 23:07 |
*** mikecmpbll has joined #openstack-swift | 23:08 | |
*** bigdogstl has quit IRC | 23:10 | |
*** bigdogstl has joined #openstack-swift | 23:11 | |
edmilsonssa_ | set it and got http://paste.openstack.org/show/734509/ | 23:16 |
openstackgerrit | Merged openstack/python-swiftclient master: Update reno for stable/rocky https://review.openstack.org/585823 | 23:18 |
openstackgerrit | Merged openstack/python-swiftclient master: Stop leaking quite so many connections https://review.openstack.org/577584 | 23:18 |
timburke | :-/ so why the 401 on GET /v1/AUTH_f8b8cba91c654277832b432ddf55ddcb ? hmm... | 23:19 |
edmilsonssa_ | f8b8cba91c654277832b432ddf55ddcb is Project ID, is it right? | 23:20 |
timburke | yeah | 23:20 |
edmilsonssa_ | shoudn't be Access Key? | 23:21 |
timburke | nope | 23:21 |
edmilsonssa_ | ok | 23:21 |
timburke | does the user work via the Swift API? | 23:21 |
edmilsonssa_ | yes | 23:23 |
timburke | ie, keystoneauth includes "admin" in the list of operator_roles, and knows that user with id f35a85b996a54e17af1a305695340fa3 can access AUTH_f8b8cba91c654277832b432ddf55ddcb? | 23:23 |
timburke | good... | 23:23 |
timburke | hmm | 23:23 |
*** bigdogstl has quit IRC | 23:24 | |
edmilsonssa_ | [filter:keystoneauth] has operator_roles = admin, user | 23:24 |
timburke | ah! proxy-server pipeline needs adjusting | 23:24 |
timburke | i saw New pipeline is "catch_errors gatekeeper healthcheck proxy-logging cache container_sync bulk ratelimit authtoken keystoneauth copy swift3 s3token container-quotas account-quotas slo dlo versioned_writes proxy-logging proxy-server" in one of the pastes | 23:25 |
edmilsonssa_ | it is set as pipeline = catch_errors gatekeeper healthcheck proxy-logging cache container_sync bulk ratelimit authtoken keystoneauth swift3 s3token container-quotas account-quotas slo dlo versioned_writes proxy-logging proxy-server | 23:25 |
timburke | we want keystoneauth after (to right of) s3token | 23:25 |
edmilsonssa_ | done, testing | 23:26 |
*** bigdogstl has joined #openstack-swift | 23:27 | |
edmilsonssa_ | seens like it's working http://paste.openstack.org/show/734510/ | 23:30 |
timburke | \o/ | 23:30 |
edmilsonssa_ | and I own you a botle of beer ... | 23:30 |
edmilsonssa_ | \o/ | 23:30 |
timburke | :-) | 23:30 |
edmilsonssa_ | Thkank you a lot!!! | 23:30 |
timburke | happy to help! | 23:31 |
edmilsonssa_ | we where about to reinstall everything ... | 23:31 |
timburke | edmilsonssa_: anovaes: out of curiosity, what are you guys using swift for? | 23:31 |
edmilsonssa_ | thank you in name of ma colegue anovaes (who had an emergency with is dog broken leg). | 23:32 |
*** bigdogstl has quit IRC | 23:32 | |
timburke | eep! hope his dog's doing better | 23:32 |
edmilsonssa_ | we are starting a poc for Bahia Court House here in Brazil ... | 23:32 |
edmilsonssa_ | if it goes right, we will use it for one of our systems ... | 23:33 |
timburke | cool! good luck with Swift, let us know if you need more help! | 23:34 |
edmilsonssa_ | and if it performs well maybe other court houses in Brazil will start unsing something similar or even Swift .... | 23:34 |
edmilsonssa_ | Ok, thank you, We'll let you know ... | 23:35 |
edmilsonssa_ | bye, it's late here ... | 23:35 |
timburke | good night | 23:36 |
*** mikecmpbll has quit IRC | 23:36 | |
*** edmilsonssa has quit IRC | 23:39 | |
*** edmilsonssa_ has quit IRC | 23:40 | |
*** bigdogstl has joined #openstack-swift | 23:43 | |
openstackgerrit | Tim Burke proposed openstack/swift master: s3api: Change default location to us-east-1 https://review.openstack.org/617046 | 23:52 |
*** bigdogstl has quit IRC | 23:54 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!