| *** cschwede has quit IRC | 00:46 | |
| *** dsariel has quit IRC | 00:56 | |
| *** gyee has quit IRC | 02:57 | |
| *** tkajinam has quit IRC | 03:10 | |
| openstackgerrit | likui proposed openstack/swift master: fix ci error test-requirements.txt https://review.opendev.org/759836 | 03:24 |
|---|---|---|
| *** psachin has joined #openstack-swift | 03:38 | |
| *** rcernin has quit IRC | 03:48 | |
| *** rcernin has joined #openstack-swift | 03:53 | |
| *** mvalsecc has quit IRC | 04:47 | |
| *** mvalsecc has joined #openstack-swift | 05:11 | |
| *** evrardjp has quit IRC | 05:33 | |
| *** evrardjp has joined #openstack-swift | 05:33 | |
| *** openstackgerrit has quit IRC | 05:46 | |
| *** tkajinam has joined #openstack-swift | 06:53 | |
| *** dsariel has joined #openstack-swift | 06:55 | |
| *** m75abrams has joined #openstack-swift | 07:27 | |
| *** rcernin has quit IRC | 07:30 | |
| *** patchbot has quit IRC | 07:41 | |
| *** patchbot has joined #openstack-swift | 07:41 | |
| *** rpittau|afk is now known as rpittau | 07:51 | |
| *** rcernin has joined #openstack-swift | 07:54 | |
| *** rcernin has quit IRC | 08:15 | |
| *** mvalsecc has quit IRC | 09:21 | |
| *** openstack has quit IRC | 09:21 | |
| *** openstack has joined #openstack-swift | 09:23 | |
| *** ChanServ sets mode: +o openstack | 09:23 | |
| *** rcernin has joined #openstack-swift | 09:36 | |
| *** cschwede has joined #openstack-swift | 09:47 | |
| *** ChanServ sets mode: +v cschwede | 09:47 | |
| *** rcernin has quit IRC | 09:56 | |
| *** psachin has quit IRC | 10:18 | |
| *** psachin has joined #openstack-swift | 10:27 | |
| *** psachin has quit IRC | 10:45 | |
| *** psachin has joined #openstack-swift | 10:48 | |
| *** psachin has quit IRC | 10:49 | |
| *** psachin has joined #openstack-swift | 10:51 | |
| *** rcernin has joined #openstack-swift | 11:16 | |
| *** rcernin has quit IRC | 12:43 | |
| *** thiago__ has quit IRC | 13:12 | |
| *** tdasilva_ has joined #openstack-swift | 13:13 | |
| *** ChanServ sets mode: +v tdasilva_ | 13:13 | |
| *** tdasilva_ has quit IRC | 13:14 | |
| *** tdasilva has joined #openstack-swift | 13:14 | |
| *** ChanServ sets mode: +v tdasilva | 13:14 | |
| *** psachin has quit IRC | 13:40 | |
| *** psachin has joined #openstack-swift | 14:54 | |
| *** gyee has joined #openstack-swift | 15:24 | |
| *** takamatsu has quit IRC | 15:45 | |
| *** tdasilva_ has joined #openstack-swift | 15:46 | |
| *** ChanServ sets mode: +v tdasilva_ | 15:46 | |
| *** tdasilva has quit IRC | 15:49 | |
| *** takamatsu has joined #openstack-swift | 15:56 | |
| *** rpittau is now known as rpittau|afk | 17:09 | |
| *** m75abrams has quit IRC | 17:14 | |
| ormandj | is there a good way to find out the cause of 401/403s issued via s3 requests by swift proxy? we're seeing a request fail, but keystone has no failures logged | 17:22 |
| ormandj | with insecure logging enabled in keystone, we only see 200s | 17:22 |
| ormandj | only the proxy is returning a 401/403 to client | 17:22 |
| *** renich has joined #openstack-swift | 17:34 | |
| *** renich has quit IRC | 17:43 | |
| *** renich has joined #openstack-swift | 17:44 | |
| *** psachin has quit IRC | 18:22 | |
| *** renich has quit IRC | 18:46 | |
| *** dsariel has quit IRC | 19:12 | |
| *** ianychoi__ has joined #openstack-swift | 19:17 | |
| *** ianychoi_ has quit IRC | 19:21 | |
| *** openstackgerrit has joined #openstack-swift | 19:43 | |
| openstackgerrit | Pete Zaitcev proposed openstack/swift master: Let developers/operators add watchers to object audit (simplified) https://review.opendev.org/706653 | 19:43 |
| *** camelCaser has quit IRC | 20:00 | |
| *** camelCaser has joined #openstack-swift | 20:03 | |
| *** camelCaser has quit IRC | 20:08 | |
| timburke | ormandj, it's tricky without client logs -- might be able to sniff for the 401/403 response with tcpdump though? | 20:12 |
| ormandj | timburke: it's intermittent | 20:16 |
| ormandj | is there no way to have the proxy log the reason for the 401/403 | 20:16 |
| ormandj | we know it's a signature mismatch, but we can see the client logs | 20:16 |
| ormandj | same signature on both requests, we did the manual signature AWS authorization calculation | 20:17 |
| ormandj | the value sent is correct | 20:17 |
| ormandj | but for some reason on one of the requests (both have same date to the second, both GETs for the same object, only different is the Range: header) returns a 403 to the client (401 from swift) | 20:17 |
| ormandj | same date/get/container/path/etc is why the authorization header stays the same | 20:18 |
| ormandj | we don't see any failures in keystone at these times | 20:21 |
| *** camelCaser has joined #openstack-swift | 20:22 | |
| *** camelCaser has quit IRC | 20:25 | |
| *** camelCaser has joined #openstack-swift | 20:26 | |
| timburke | strange -- to my knowledge, range headers aren't used at all in signature calculations... | 20:43 |
| ormandj | they aren't | 20:44 |
| ormandj | and the AWS authorization header is the exact same on both requests | 20:44 |
| ormandj | so we don't know why one is returning a 403 through s3/401 via swift | 20:44 |
| ormandj | it's breaking the customer :/ | 20:45 |
| *** camelCaser has quit IRC | 20:45 | |
| ormandj | we can't find anywhere the reason for the 401/403 is logged, just that it's a 401/403 | 20:45 |
| tdasilva_ | timburke: IIRC range headers are used in signature calculations | 20:45 |
| ormandj | tdasilva_: https://docs.aws.amazon.com/AmazonS3/latest/dev/RESTAuthentication.html | 20:45 |
| ormandj | it's http verb, content md4, content type, date, canonicalizedamzheaders + canonicalizedresource, but they aren't adding those | 20:46 |
| tdasilva_ | https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.html | 20:46 |
| ormandj | same request is successful with a 2xx and fails with a 403 (via s3), with the only difference being the range: being slightly different | 20:47 |
| ormandj | that's v4 sigs | 20:47 |
| *** camelCaser has joined #openstack-swift | 20:47 | |
| ormandj | we did the Authorization: build ourselves using the data, and get the same string passed in, which again - works on one of the requests in question | 20:49 |
| ormandj | just intermittently we see 403s | 20:49 |
| ormandj | we think it might be a timeout with something but there's no logging of the cause/source of the 401 in swift/403 in s3api | 20:49 |
| ormandj | so to construct a authorization, you can do: echo -n "GET\n\n\nWed, 28 Oct 2020 16:17:12 GMT\n/container/object" | openssl sha1 -hmac "EC2SECRETHERE" -binary | base64 <-- that, for example, then your Authorization header is Authorization = "AWS" + " " + AWSKEY + ":" + signaturecalculatedpreviously | 20:51 |
| ormandj | range isn't included, and that will work for requests with that date/container/object/creds | 20:52 |
| ormandj | but _intermittently_ we see failures, even though a request at that same second with the same Authorization header 2xxd | 20:52 |
| ormandj | we don't know if it's a timeout somewhere that's just not being logged for a lookup to memcache/keystone/something else or something else entirely | 20:53 |
| ormandj | we don't see any failures in the keystone logs even with insecure debug on | 20:53 |
| ormandj | but unfortunately the proxy service doesn't log any detail beyond just returning a 401/403 | 20:54 |
| timburke | how far apart are the requests? i wonder if it might be hitting a clock-skew error... i think we set our limit more aggressively than AWS, should probably make that configurable... | 20:54 |
| ormandj | literally the same second with the same date in the request header | 20:54 |
| ormandj | in this case, it was same second down to the thousandth of a millisecon | 20:54 |
| ormandj | d | 20:54 |
| timburke | but what about the timestamp on the *log line*? | 20:54 |
| ormandj | i'll go check, but i think we're configurd to allow a 15 minute delta on time just like AWS | 20:55 |
| ormandj | i doubt it's outside that, one sec, looking | 20:55 |
| *** camelCaser has quit IRC | 20:55 | |
| timburke | the delta at https://github.com/openstack/swift/blob/2.26.0/swift/common/middleware/s3api/s3request.py#L740-L744 should totally be a config option :-/ | 20:56 |
| *** camelCaser has joined #openstack-swift | 20:57 | |
| ormandj | one hit proxy01 at 13:24:34, one hit proxy04 13:24:35 | 21:00 |
| ormandj | the one hitting 01 failed | 21:01 |
| ormandj | the one hitting 04 succeeded | 21:01 |
| ormandj | both had the header date set as 13:24:34 | 21:01 |
| ormandj | i am noticing something i missed before, on the request that failed, the S3 request looks the same, but the swift request is not (we started enabling the swift logging too) | 21:04 |
| ormandj | on the working, we see /v1/AUTH_customerid/blahblahblah for request | 21:04 |
| ormandj | on the non-working, we see /v1/somehashofsomething/bucket/object/etc | 21:04 |
| ormandj | AUTH_ is our prefix | 21:04 |
| ormandj | and no idea what that second hash is of | 21:05 |
| ormandj | s3 request is the exact same in the log | 21:05 |
| *** ccamel has joined #openstack-swift | 21:07 | |
| *** camelCaser has quit IRC | 21:07 | |
| timburke | somehashofsomething is probably the access_key_id -- s3token *should* have swapped it out for the proper swift account tho | 21:09 |
| timburke | clayg, rledisez (and anyone else who wants to join) ops feedback session starting! | 21:10 |
| ormandj | yeah, can check that, and apologies, the s3 request is the one that's different, the swift request looks the same but spits a 401 probably related to the s3 thing | 21:11 |
| ormandj | so some reason s3token isn't doing the nice thing on some requests, sometimes | 21:12 |
| ormandj | and yep, that's the access key | 21:17 |
| *** dsariel has joined #openstack-swift | 21:32 | |
| seongsoocho | ormandj: In my case, If the Host value of http header is different with swift's domain, the s3token always return 403 . | 21:49 |
| *** jrosser has quit IRC | 21:59 | |
| *** yuxin_ has quit IRC | 21:59 | |
| *** yuxin_ has joined #openstack-swift | 21:59 | |
| *** jrosser has joined #openstack-swift | 21:59 | |
| *** rcernin has joined #openstack-swift | 22:46 | |
| *** cschwede has quit IRC | 23:20 | |
| *** dsariel has quit IRC | 23:23 | |
| ormandj | seongsoocho: thank you, i'll look | 23:28 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!