| *** _CrustY has quit IRC | 01:26 | |
| *** _CrustY has joined #swift3 | 01:26 | |
| *** onovy has quit IRC | 02:08 | |
| *** onovy has joined #swift3 | 02:10 | |
| *** _CrustY has quit IRC | 06:00 | |
| *** _CrustY has joined #swift3 | 06:03 | |
| *** [-mugatu-] has quit IRC | 06:45 | |
| *** [-mugatu-] has joined #swift3 | 06:48 | |
| *** [-mugatu-] has quit IRC | 07:42 | |
| *** [-mugatu-] has joined #swift3 | 14:36 | |
| *** tgk has joined #swift3 | 14:45 | |
| tgk | Hi. Is this a good place to get help troubleshooting a swift3 install? | 14:54 |
|---|---|---|
| *** chsc has joined #swift3 | 15:19 | |
| timburke | tgk: yeah! what sort of issues are you running into? | 15:43 |
| tgk | Thanks timburke | 15:43 |
| tgk | I've been trying to setup swift3 in my openstack/swift servers on CentOS 7 and got some errors that I need some help troubleshooting. My EC2 key authentication goes through ok according to keystone logs, but all my requests using s3curl end with Internal Error - Invalid Bucket - Could not parse the specified URI. I'm running s3curl from the same box that runs swift-proxy, trying to retrieve http://127.0.0.1:8080/ | 15:43 |
| tgk | I'm running Pike btw | 15:44 |
| timburke | oh good; i know there was an issue with ocata keystone | 15:45 |
| tgk | yeah, that was the main reason I upgraded to pike | 15:45 |
| tgk | and the old keystone errors from ocata when trying to authenticate with EC2 keys are no longer there, so that's good | 15:46 |
| timburke | what's the name of the bucket you're trying to hit? we try to enforce the same naming restrictions as AWS, but they aren't consistent -- you might want to try switching dns_compliant_bucket_names to False in the swift3 config | 15:47 |
| tgk | I created a bucket called "test" to keep it simple | 15:48 |
| tgk | Let me try the dns_compliant_bucket_names option... that's in the proxy-server.conf, right? | 15:48 |
| timburke | yup. but "test" should be fine on both... i wonder if s3curl's trying to use virtual-host style requests? try throwing a --verbose on there to see what gets plumbed down to curl | 15:49 |
| timburke | i'm thinking it might be trying to contact http://test.127.0.0.1:8080/ | 15:50 |
| timburke | which won't make sense :P but you could add an entry for test.localhost to your /etc/hosts and try that way | 15:51 |
| timburke | though... no... it usually takes the full url, doesn't it? let me dig around a little... | 15:52 |
| tgk | I pasted some logs and output here: https://pastebin.com/ehcCQUKu | 15:54 |
| timburke | well...that's a special kind of stupid on our part. sorry there | 15:57 |
| timburke | looks like it's coming out of either https://github.com/openstack/swift3/blob/master/swift3/s3_token_middleware.py#L202-L204 or https://github.com/openstack/swift3/blob/master/swift3/s3_token_middleware.py#L321-L325 | 15:57 |
| timburke | you said that keystone seemed happy, though? makes me think it's the latter, in which case i wonder what the reply looks like from keystone... | 15:58 |
| tgk | yes, I see a 200 reply in keystone logs on authentication, and it's something else when my keys are incorrect | 15:58 |
| timburke | try turning on debug logging for s3token -- you don't want to run with that in production because of https://github.com/openstack/swift3/blob/master/swift3/s3_token_middleware.py#L294-L295 but it's really handy to see the complete keystone response | 16:00 |
| tgk | ok. How do I enable debug? Under the [filter:swift3] section of proxy-server.conf? | 16:02 |
| tgk | sorry, I meant, filter:s3token | 16:02 |
| timburke | yep; log_level = DEBUG iirc | 16:06 |
| timburke | ...which i should really call out in the sample config... | 16:07 |
| tgk | I tried both "log_level = DEBUG" and "set log_level = DEBUG" under [filter:s3token], but the output in /var/log/swift/swift.log seems to be the same when trying s3curl | 16:12 |
| timburke | maybe check /var/log/syslog? i really ought to invest some time into learning more about how to configure swift's logging... | 16:16 |
| timburke | fwiw, i typically set log_level = DEBUG in the [DEFAULT] section for development, then get everything out of /var/log/syslog | 16:17 |
| timburke | and i don't have any other logging config to speak of | 16:18 |
| tgk | sounds simpler that way. I'll give it a try. | 16:18 |
| timburke | biggest downside is that it's also noisier :-) | 16:19 |
| tgk | ah, ok, I got some logs... let me paste them | 16:21 |
| tgk | https://pastebin.com/K9eLN7kE | 16:23 |
| tgk | relevant part is most likely: | 16:24 |
| tgk | 7.Oct 6 12:20:53 openstack-host01 proxy-server: Error on keystone reply: 200 | 16:24 |
| tgk | but I only see a long output with all the endpoints... I don't see anything pointing at an error | 16:27 |
| timburke | what version of swift3 is this? is it a tagged release, or coming off master? | 16:29 |
| tgk | openstack-swift-plugin-swift3-1.11.0-1.el7.noarch | 16:29 |
| tgk | out of the official repo | 16:30 |
| timburke | i really need to go review https://review.openstack.org/#/c/504479/ so we actually have a release that includes https://github.com/openstack/swift3/commit/807ed38 :-( | 16:30 |
| patchbot | patch 504479 - swift3 - Change log updates for version 1.12 | 16:30 |
| tgk | Is 1.12 in any public repo? | 16:33 |
| timburke | no, it hasn't been tagged yet. that's part of the problem | 16:34 |
| tgk | I guess I'll give it another try once that version is released :) | 16:34 |
| timburke | fwiw, we work hard to keep tip of master stable -- the company i work for often releases from master (sometimes even with some in-review patches) to get fixes that haven't been officially released yet | 16:35 |
| tgk | I'm not testing in prod, so if there's a release candidate with the patch, I could try it too | 16:38 |
| tgk | but I think the version I'm running matches the latest in github | 16:40 |
| timburke | if it were me, i'd just do a `pip install git+https://github.com/openstack/swift3.git` ;-) | 16:40 |
| tgk | I'll give it a shot. I really appreciate you taking the time to look at this with me. | 16:43 |
| tgk | Thanks!! | 16:43 |
| timburke | sure! thanks for prodding me to look at swift3 more today :-) i've been owing kota_ some reviews/patchsets | 16:45 |
| *** openstack has joined #swift3 | 17:09 | |
| *** ChanServ sets mode: +o openstack | 17:09 | |
| *** tgk has quit IRC | 21:37 | |
| *** chsc has quit IRC | 23:21 | |
| *** chsc has joined #swift3 | 23:41 | |
| *** chsc has joined #swift3 | 23:41 | |
| *** chsc has quit IRC | 23:45 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!