| *** Socket_0x03 has joined #swift3 | 05:42 | |
| *** Socket_0x03 has joined #swift3 | 05:42 | |
| *** openstackgerrit has quit IRC | 05:48 | |
| *** openstackgerrit has joined #swift3 | 05:49 | |
| *** Socket_0x03 has quit IRC | 06:03 | |
| *** openstackgerrit has quit IRC | 07:48 | |
| *** openstackgerrit has joined #swift3 | 07:48 | |
| *** acoles_ is now known as acoles | 09:56 | |
| *** tingo has joined #swift3 | 13:31 | |
| *** chsc has joined #swift3 | 15:56 | |
| *** chsc has quit IRC | 15:56 | |
| *** chsc has joined #swift3 | 15:56 | |
| *** chsc has quit IRC | 16:03 | |
| *** vquicksilver has joined #swift3 | 16:14 | |
| vquicksilver | Hi | 16:15 |
|---|---|---|
| vquicksilver | I'm trying to configure swift3 on centos 7 with openstack mitaka, I created the credentials in keystone without any trouble but I'm unable to access any resource using s3cmd or s3curl | 16:16 |
| vquicksilver | I always get the following error: <html><h1>Not Found</h1><p>The resource could not be found.</p></html> | 16:17 |
| vquicksilver | how can I debug this? | 16:17 |
| vquicksilver | I have the following pipeline: pipeline = catch_errors gatekeeper healthcheck cache swift3 s3token authtoken keystoneauth container-quotas account-quotas slo dlo versioned_writes proxy-logging proxy-server | 16:18 |
| vquicksilver | any help would be apreciated :) | 16:19 |
| *** bill_az has joined #swift3 | 16:29 | |
| *** bill_az has quit IRC | 16:38 | |
| cnf | vquicksilver check the logs, are you hitting v2 or v3? | 16:40 |
| vquicksilver | cnf: I don't see any related info in the logs | 16:41 |
| vquicksilver | cnf: I'm using wsgi to run the swift proxy, and I enabled debug in apache | 16:41 |
| vquicksilver | cnf: anyway think I'm using v3 | 16:42 |
| cnf | swift3 is hardcoded to v2 on keystone, I ran into trouble with that | 16:43 |
| vquicksilver | cnf: how can I check what version is using? on how can I enable v2 in keystone? | 16:43 |
| cnf | does it work in swift? | 16:43 |
| vquicksilver | cnf: sorry I'm a noob with this kind of things | 16:44 |
| vquicksilver | cnf: swift is working fine with the swift client | 16:44 |
| vquicksilver | AFAIK | 16:44 |
| cnf | well, do make sure :P | 16:44 |
| vquicksilver | cnf: hehe, you know I can make swift stat, quotas are working etc... | 16:45 |
| vquicksilver | cnf: so seems to be ok, how can I make a petition to the v2 api? | 16:45 |
| cnf | put keystone in debug, and watch the logs | 16:45 |
| vquicksilver | cnf: should I change OS_IDENTITY_API_VERSION=3 | 16:45 |
| vquicksilver | ? | 16:45 |
| cnf | when you hit the s3 endpoint, it'll query keystone | 16:45 |
| cnf | see what url it hits | 16:46 |
| vquicksilver | cnf: ok | 16:46 |
| vquicksilver | cnf: so I should launch a petition to the proxy-server, and see a petition in the keystone log | 16:46 |
| vquicksilver | let me check it | 16:46 |
| vquicksilver | cnf: uhm I can't see anything in keystone | 16:49 |
| cnf | well, that's not good, is it | 16:50 |
| cnf | do you see anything when you manually query it? | 16:50 |
| vquicksilver | cnf yes, for example if I do a swift list I can see lines in the log | 16:50 |
| vquicksilver | cnf: I'm using fernet tokens by the way | 16:51 |
| cnf | then check your swift3 config | 16:51 |
| vquicksilver | cnf: I don't have any special config for it | 16:52 |
| vquicksilver | for s3token I have my keystone url and the user for querying it | 16:52 |
| cnf | you have at least 2 sections in your proxy.conf, rght? | 16:53 |
| cnf | swift3, and s3token ? | 16:53 |
| vquicksilver | [filter:swift3] | 16:53 |
| vquicksilver | use = egg:swift3#swift3 | 16:53 |
| vquicksilver | s3_acl = false | 16:53 |
| vquicksilver | yes | 16:53 |
| vquicksilver | [filter:s3token] | 16:53 |
| vquicksilver | paste.filter_factory = keystonemiddleware.s3_token:filter_factory | 16:53 |
| vquicksilver | auth_host = keystone.iaas.es | 16:53 |
| vquicksilver | auth_port = 35357 | 16:53 |
| vquicksilver | auth_protocol = https | 16:53 |
| vquicksilver | auth_uri = https://keystone.iaas.es:5000/ | 16:53 |
| vquicksilver | admin_tenant_name = service | 16:53 |
| vquicksilver | admin_user = swift | 16:53 |
| vquicksilver | I omit the admin_password line | 16:53 |
| vquicksilver | and I think I have it correctly placed in the pipeline | 16:53 |
| cnf | don't even need those | 16:55 |
| vquicksilver | cnf: I got it from the config in the test folder | 16:55 |
| vquicksilver | cnf: should I left the auth_uri only? | 16:56 |
| cnf | [filter:s3token] | 16:56 |
| cnf | use = egg:keystonemiddleware#s3_token | 16:56 |
| cnf | auth_host = <ip> | 16:56 |
| cnf | auth_port = 35357 | 16:56 |
| cnf | auth_protocol = http | 16:56 |
| cnf | is all I have | 16:56 |
| vquicksilver | let me check it | 16:56 |
| vquicksilver | still the same | 16:59 |
| vquicksilver | no lines appear in the keystone.log | 16:59 |
| vquicksilver | swift list is still working | 16:59 |
| vquicksilver | but when I use s3cmd ls I can't see any lines | 17:00 |
| *** chsc has joined #swift3 | 17:00 | |
| vquicksilver | cnf: in the client I should point it to the proxy right? | 17:01 |
| cnf | yeah | 17:02 |
| vquicksilver | I'm out of ideas | 17:02 |
| cnf | and my workday is almost done | 17:02 |
| cnf | wait for the US people to come online :P | 17:02 |
| vquicksilver | cnf: hehe ok | 17:02 |
| vquicksilver | cnf: thanks for your time anyway :) | 17:04 |
| vquicksilver | cnf: so in my endpoints I only have keystone with /v3 | 17:20 |
| timburke | good morning | 17:26 |
| timburke | (catching up on scrollback) | 17:26 |
| vquicksilver | morning timburke | 17:27 |
| timburke | vquicksilver: can the same user access things through the Swift API? i wonder if perhaps the account doesn't exist yet (and account_autocreate is still set to its default of false) | 17:29 |
| vquicksilver | timburke: yes, the user can access three containers | 17:29 |
| vquicksilver | and put files etc... | 17:30 |
| vquicksilver | let me check the account_autocreate value | 17:30 |
| vquicksilver | [app:proxy-server] | 17:30 |
| vquicksilver | use = egg:swift#proxy | 17:30 |
| vquicksilver | account_autocreate = true | 17:30 |
| timburke | got some proxy logs that include an S3-style attempt? | 17:30 |
| vquicksilver | timburke: this is what I see in the proxy log | 17:31 |
| vquicksilver | Nov 3 18:31:28 rack32u10 proxy-server: Starting Keystone auth_token middleware | 17:31 |
| vquicksilver | Nov 3 18:31:28 rack32u10 proxy-server: Deferring reject downstream | 17:31 |
| vquicksilver | Nov 3 18:31:28 rack32u10 proxy-server: 109.70.132.20 109.70.132.20 03/Nov/2016/17/31/28 GET / HTTP/1.1 404 - - - - 70 - tx013a3b2a29ae412295014-00581b7470 - 0.0003 - - 1478194288.110852957 1478194288.111125946 - | 17:31 |
| vquicksilver | timburke: after doing s3cmd ls | 17:32 |
| vquicksilver | timburke: maybe I'm not using the client correctly? I followed the guide for swiftstack, I did s3cmd --configure and changed the urls to point my proxy | 17:33 |
| vquicksilver | by the way I get the same result with s3curl and also with cyberduck | 17:33 |
| timburke | yeah, that should work... just before those lines, was there anything about swift3/s3token? | 17:34 |
| vquicksilver | timburke: nothing, that's why I was asking if there exist some way of enabling debug or something like that | 17:34 |
| timburke | it doesn't make sense to me that it should 404. 403, maybe, but 404 is weird | 17:34 |
| timburke | i've got three settings i like to turn on for hunting down weird issues: in the DEFAULT section, set log_headers = true and log_level = DEBUG, and in the swift3 section set force_swift_request_proxy_log = true | 17:36 |
| vquicksilver | timburke: ok, the system is not in production yet, so I can change anything | 17:37 |
| vquicksilver | timburke: let me apply those changes | 17:37 |
| vquicksilver | Nov 3 18:39:16 rack32u10 proxy-server: Pipeline is "catch_errors gatekeeper healthcheck cache swift3 s3token authtoken keystoneauth container-quotas account-quotas slo dlo versioned_writes proxy-logging proxy-server" | 17:39 |
| vquicksilver | Nov 3 18:39:16 rack32u10 proxy-server: Starting Keystone auth_token middleware | 17:39 |
| vquicksilver | Nov 3 18:39:16 rack32u10 proxy-server: Starting the S3 Token Authentication component | 17:40 |
| vquicksilver | Nov 3 18:39:16 rack32u10 proxy-server: Use keystone middleware. | 17:40 |
| vquicksilver | Nov 3 18:39:16 rack32u10 proxy-server: Calling S3Token middleware. | 17:40 |
| vquicksilver | Nov 3 18:39:16 rack32u10 proxy-server: Not a path query, skipping. | 17:40 |
| vquicksilver | Nov 3 18:39:16 rack32u10 proxy-server: Deferring reject downstream | 17:40 |
| vquicksilver | Nov 3 18:39:16 rack32u10 proxy-server: Received request from | 17:40 |
| vquicksilver | Nov 3 18:39:16 rack32u10 proxy-server: Authorizing as anonymous (txn: tx16cf9f34da394d5ab4ab7-00581b7644) | 17:40 |
| vquicksilver | Nov 3 18:39:16 rack32u10 proxy-server: 109.70.132.20 109.70.132.20 03/Nov/2016/17/39/16 GET / HTTP/1.1 404 - curl/7.43.0 - - 70 - tx16cf9f34da394d5ab4ab7-00581b7644 X-Identity-Status:%20Invalid%0AUser-Agent:%20curl/7.43.0%0ADate:%20jue%2C%2003%20nov%202016%2017:39:14%20%2B0000%0AHost:%20swift.iaas.es%0AAccept:%20%2A/%2A 0.0003 - - 1478194756.576819897 1478194756.577096939 - | 17:40 |
| vquicksilver | timburke: Authorizing as anonymous? | 17:40 |
| vquicksilver | this is with s3cmd instead | 17:41 |
| vquicksilver | Nov 3 18:40:38 rack32u10 proxy-server: Pipeline is "catch_errors gatekeeper healthcheck cache swift3 s3token authtoken keystoneauth container-quotas account-quotas slo dlo versioned_writes proxy-logging proxy-server" | 17:41 |
| vquicksilver | Nov 3 18:40:38 rack32u10 proxy-server: Starting Keystone auth_token middleware | 17:41 |
| vquicksilver | Nov 3 18:40:38 rack32u10 proxy-server: Starting the S3 Token Authentication component | 17:41 |
| vquicksilver | Nov 3 18:40:38 rack32u10 proxy-server: Use keystone middleware. | 17:41 |
| vquicksilver | Nov 3 18:40:38 rack32u10 proxy-server: Calling S3Token middleware. | 17:41 |
| *** tingo has quit IRC | 17:41 | |
| vquicksilver | Nov 3 18:40:38 rack32u10 proxy-server: Not a path query, skipping. | 17:41 |
| vquicksilver | Nov 3 18:40:38 rack32u10 proxy-server: Deferring reject downstream | 17:41 |
| vquicksilver | Nov 3 18:40:38 rack32u10 proxy-server: Received request from | 17:41 |
| vquicksilver | Nov 3 18:40:38 rack32u10 proxy-server: Authorizing as anonymous (txn: tx03afc9eaccb84edf99d4e-00581b7696) | 17:41 |
| vquicksilver | Nov 3 18:40:38 rack32u10 proxy-server: 109.70.132.20 109.70.132.20 03/Nov/2016/17/40/38 GET / HTTP/1.1 404 - - - - 70 - tx03afc9eaccb84edf99d4e-00581b7696 X-Amz-Date:%20Thu%2C%2003%20Nov%202016%2017:40:37%20%2B0000%0AX-Identity-Status:%20Invalid%0AHost:%20swift.iaas.es%0AAccept-Encoding:%20identity%0AContent-Length:%200 0.0003 - - 1478194838.733294010 1478194838.733561993 - | 17:41 |
| timburke | hmm. looks like swift3 doesn't realize it's an S3 request? https://github.com/openstack/swift3/blob/467e5db/swift3/s3_token_middleware.py#L151 | 17:41 |
| timburke | and wait, i don't see anything about an Authorization header... | 17:42 |
| vquicksilver | timburke: so what I'm doing wrong? | 17:45 |
| vquicksilver | or is it some bug with the swift3 version in centos? | 17:45 |
| timburke | you've set up access_key and secret_key entries in the s3cmd config, right? | 17:46 |
| vquicksilver | timburke: yes, let me check anyway | 17:46 |
| vquicksilver | timburke: yeah | 17:46 |
| vquicksilver | timburke: I can provide the credentials if you want to try | 17:49 |
| vquicksilver | maybe is my .s3cfg file | 17:49 |
| vquicksilver | but I get the same result with s3curl --id id --key key | 17:49 |
| vquicksilver | why it says invalid host? | 17:52 |
| timburke | where did it say that? i saw something about X-Identity-Status: Invalid... | 17:57 |
| vquicksilver | X-Identity-Status:%20Invalid%0AHo │ | 17:58 |
| vquicksilver | | st:%20swift.iaas.es% | 17:58 |
| timburke | ah, yeah; it's logging multiple headers lines there. "X-Identity-Status: Invalid" is one, "Host: swift.iaas.es" is another | 18:00 |
| vquicksilver | timburke: ah ok, sorry | 18:00 |
| vquicksilver | timburke: I ve been trying to make this work for a few hours, I need coffee | 18:01 |
| timburke | tends to help :-) | 18:01 |
| cnf | ohai | 18:02 |
| cnf | did it get fixed? | 18:02 |
| vquicksilver | cnf: still not working | 18:02 |
| cnf | well, if timburke is on the case, things are sure to move along | 18:03 |
| cnf | he the man! | 18:03 |
| vquicksilver | cnf: thanks to timburke we got more info | 18:03 |
| cnf | (well, i'm assuming tim is a man, I might be wrong) | 18:03 |
| vquicksilver | cnf: do you have any endpoint in your keystone explictly enabling the v2 api or something like that? | 18:04 |
| vquicksilver | cnf: I only have /v3 endpoints | 18:04 |
| cnf | no, it's always enabled | 18:04 |
| vquicksilver | ok | 18:04 |
| cnf | but on v3 ALL endpoints work on 5000 | 18:05 |
| cnf | on v2 this is not the case | 18:05 |
| cnf | which was what bit me | 18:05 |
| vquicksilver | cnf: ok | 18:05 |
| cnf | because I had keystone behind an ssl proxy, and it was forwarding everyhting to port 5000 | 18:06 |
| vquicksilver | cnf here is on the same machine, so it should work | 18:06 |
| cnf | oh, but you set an FQDN for the keystone host | 18:06 |
| cnf | are you sure it is resolving to an address that works? | 18:06 |
| cnf | on that host? | 18:07 |
| vquicksilver | cnf: I have keystone.iaas.es like 127.0.0.1 in my /etc/hosts | 18:07 |
| vquicksilver | it should work | 18:07 |
| cnf | ok | 18:07 |
| cnf | and keystone is listening to 127.0.0.1? | 18:07 |
| vquicksilver | yeah | 18:07 |
| cnf | it didn't bind to the ip of eth0 or something? | 18:07 |
| cnf | k | 18:07 |
| timburke | vquicksilver: with s3curl, mind tacking on a --debug? i'm hoping to see that it actually *did* calculate a signature and drop it in an Authorization header | 18:08 |
| cnf | with timburke on the case, i'll stop adding random stuff ^^; | 18:08 |
| vquicksilver | timburke: sure | 18:09 |
| timburke | assuming that it does, we'll then need to sort out why that wasn't showing up in the logs, and why swift3 didn't translate it to a /v1/some-swift-account wort of request | 18:09 |
| vquicksilver | Will sleep and continue despite this problem. | 18:09 |
| vquicksilver | Please set up /home/ghost/.s3curl for future requests. | 18:09 |
| vquicksilver | s3curl: Found the url: host=swift.iaas.es; port=; uri=; query=; | 18:10 |
| vquicksilver | s3curl: cname endpoint signing case | 18:10 |
| vquicksilver | s3curl: StringToSign='GET\n\n\njue, 03 nov 2016 18:09:23 +0000\n/swift.iaas.es/' | 18:10 |
| vquicksilver | s3curl: exec curl -H Date: jue, 03 nov 2016 18:09:23 +0000 -H Authorization: AWS 81a6e1a6d6d24a75a41cacd8c68d753f:OsTfxFfeH2RX111b2N/61bO3/fM= -L -H content-type: https://swift.iaas.es | 18:10 |
| vquicksilver | by the way if you want to try secret key is 4a4783c0e3fe40d88164b8fc87ad06c3 and id is 81a6e1a6d6d24a75a41cacd8c68d753f | 18:10 |
| vquicksilver | thinks I can pass -s -v to curl if I do: s3curl --id 81a6e1a6d6d24a75a41cacd8c68d753f --key -- -s -v https://swift.iaas.es | 18:13 |
| vquicksilver | timburke: maybe I should remove the rpm, and install a more recent version? | 18:16 |
| timburke | vquicksilver: i wonder if there's any chance Apache could be trying to handle the Authorization header, then strips it out before it reaches Swift? | 18:20 |
| vquicksilver | timburke: maybe you are right | 18:21 |
| timburke | i know i've had a bit of trouble lately with apache mangling requests in unexpected ways (like https://bugs.launchpad.net/python-swiftclient/+bug/1621581) | 18:23 |
| openstack | Launchpad bug 1621581 in python-swiftclient "swiftclient returns response headers without 'Content-Length' param, thus causing upload object to fail" [Undecided,In progress] - Assigned to Arun Mani (arun-mani) | 18:23 |
| vquicksilver | timburke: can I prevent apache from doing that? | 18:24 |
| vquicksilver | timburke: I have a very basic configuration for it, just defined some virtualhosts | 18:24 |
| vquicksilver | disable mod_proxy maybe? | 18:26 |
| vquicksilver | timburke: just disabled it | 18:29 |
| timburke | unfortunately, i'm not very familiar with using apache to reverse-proxy. you might be able to try connecting directly to port that swift is running on? i know that various libraries have difficulty with computing signatures for that, though | 18:30 |
| vquicksilver | timburke: have to go home know, but I will try to use swift out of apache to diagnose this | 18:31 |
| vquicksilver | I'll let you know the results | 18:31 |
| vquicksilver | thanks for your help | 18:31 |
| timburke | good luck! | 18:31 |
| *** acoles is now known as acoles_ | 18:45 | |
| *** bill_az has joined #swift3 | 19:14 | |
| *** bill_az has quit IRC | 19:25 | |
| *** bill_az has joined #swift3 | 21:24 | |
| *** bill_az has quit IRC | 21:39 | |
| openstackgerrit | Tim Burke proposed openstack/swift3: Add more validation for auth_uri https://review.openstack.org/365166 | 21:57 |
| openstackgerrit | Tim Burke proposed openstack/swift3: Make s3token work in a Keystone-V3-only world https://review.openstack.org/384659 | 22:16 |
| *** chsc has quit IRC | 23:32 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!