| *** DCWillia_ has quit IRC | 00:02 | |
| *** annegentle has joined #openstack-swift | 00:03 | |
| *** minwoob_ has quit IRC | 00:06 | |
| *** annegentle has quit IRC | 00:08 | |
| openstackgerrit | Denis Cavalcante proposed openstack/swift-specs: Internal communication measurement for swift spec https://review.openstack.org/180801 | 00:19 |
|---|---|---|
| peluse | torgomatic, tricky | 00:23 |
| notmyname | my favorite thing about EC: that it's "beta" | 00:27 |
| *** dmorita has joined #openstack-swift | 00:28 | |
| mattoliverau | notmyname: lets make everything beta!! | 00:30 |
| notmyname | lol | 00:30 |
| * mattoliverau has landed and found a coffee btw | 00:30 | |
| peluse | the only thing better than a beta is "a prototype" :) | 00:32 |
| notmyname | summary for today is tracing down some bugs in both read and write. clayg / torgomatic have a patch for some PUT stuff | 00:35 |
| notmyname | still tracking the GET stuff | 00:35 |
| notmyname | 6+4 scheme, proxy finds 6 but still returns 503 to the client | 00:36 |
| notmyname | and other times there's a traceback from liberasure | 00:36 |
| notmyname | I don't know if they are the same or related | 00:36 |
| *** annegentle has joined #openstack-swift | 00:39 | |
| *** DCWilliams_VA has joined #openstack-swift | 00:40 | |
| peluse | how to repro the GET issue? | 00:41 |
| *** annegentle has quit IRC | 00:56 | |
| *** kota_ has joined #openstack-swift | 00:57 | |
| kota_ | good morning | 00:58 |
| *** shakamunyi has quit IRC | 01:02 | |
| *** thurloat is now known as thurloat_isgone | 01:04 | |
| mattoliverau | kota_: Morning!! How's the holidays? | 01:09 |
| clayg | peluse: dunno how to repro - but it is reproduable | 01:09 |
| clayg | peluse: notmyname had this weird object with the fragments all mixed up the wrong nodes (no idea why, still need to go look for the PUT log) and some of the primaries 404'd (the frags were on handoffs) and the nodes that do come back seems like they should be enough but the proxy barfs with no log message | 01:10 |
| clayg | peluse: i was just troubleshooting with swifterdarrell how come ssync doesn't work with replication networks! | 01:11 |
| clayg | peluse: turns out that @replication decorator doesn't make any allowances for the pipelined ssync requests - so replication_server = true means you can't use ssync | 01:11 |
| *** DCWilliams_VA has quit IRC | 01:12 | |
| clayg | for now we'll probably just make our replication servers allow all verbs - that works around the 405's in the subrequests anyway | 01:12 |
| clayg | neat trick there I could have used the replication_ip for the fragment rebuild GET requests after all! | 01:12 |
| clayg | also, I really need to update probetests to work with seperate replication networks so I can run my dev setup like that sometimes | 01:13 |
| clayg | but last time i tried it looked like it was going to be a ton of work :'( | 01:13 |
| kota_ | mattoliverau: was good ;) | 01:26 |
| *** jrichli has joined #openstack-swift | 01:26 | |
| *** annegentle has joined #openstack-swift | 01:56 | |
| *** erlon has quit IRC | 02:01 | |
| *** annegentle has quit IRC | 02:02 | |
| *** jkugel has joined #openstack-swift | 02:04 | |
| *** wbhuber has joined #openstack-swift | 02:09 | |
| *** shakamunyi has joined #openstack-swift | 02:09 | |
| *** kota_ has quit IRC | 02:11 | |
| *** bkopilov has quit IRC | 02:35 | |
| *** thumpba has joined #openstack-swift | 02:43 | |
| *** jkugel has quit IRC | 02:49 | |
| *** panbalag has quit IRC | 02:54 | |
| *** annegentle has joined #openstack-swift | 02:57 | |
| *** silor has joined #openstack-swift | 03:01 | |
| *** wbhuber has quit IRC | 03:09 | |
| clayg | indeed python y u no ordered dict literal | 03:21 |
| *** annegentle has quit IRC | 03:25 | |
| clayg | ok, so segfault issue has been emailed - there's reconstructor bug in there too I think - i guess I'll file that, and the replication network ssync thing, and then maybe back to the GET issue | 03:41 |
| *** links has joined #openstack-swift | 03:45 | |
| *** tamizh_geek has joined #openstack-swift | 03:49 | |
| *** bkopilov has joined #openstack-swift | 04:01 | |
| *** asettle has joined #openstack-swift | 04:02 | |
| *** kota_ has joined #openstack-swift | 04:17 | |
| *** jrichli has quit IRC | 04:18 | |
| *** tamizh_geek has quit IRC | 04:23 | |
| *** zaitcev has quit IRC | 04:38 | |
| *** SkyRocknRoll has joined #openstack-swift | 04:58 | |
| *** SkyRocknRoll has joined #openstack-swift | 04:58 | |
| *** tamizh_geek has joined #openstack-swift | 05:11 | |
| openstackgerrit | Hisashi Osanai proposed openstack/swift: WIP: Enable Role-based access control using oslo.policy in Swift https://review.openstack.org/149930 | 05:19 |
| clayg | bah why is zaitcev never around when you need him - roflmao @ object-supertransmogrificator | 05:23 |
| mattoliverau | Yay, killed my cloud server SAIO, turns out upscaling testing is usefult to find problems, who whould have thought! Back to debugging :) | 05:28 |
| kota_ | clayg: what's object-supertransmogrificator? | 05:29 |
| kota_ | clayg: sounds interesting | 05:29 |
| clayg | idk, I *think* zaitcev made it up - it *does* sound interesting, right! :D | 05:30 |
| kota_ | hah | 05:30 |
| clayg | notmyname: charz: are you running benchmarks on the qa cluster | 05:33 |
| clayg | notmyname: charz: I'm finally getting back around to looking at the GET issue | 05:33 |
| charz | clayg: no, did you know where is logs? | 05:35 |
| clayg | charz: yeah I think i found them, I'm confused about ssbench on a read scenario | 05:49 |
| clayg | if you do 100% reads - it has to get the objects from somewhere - so it will pre-seed data if it needs to? | 05:50 |
| clayg | but what about if you have a previous run - is there any chance we're not wiping the cluster between benchmarks? | 05:50 |
| charz | clayg: yes, it | 05:50 |
| clayg | I don't know how else my greps could be missing this objects PUT unless they were in logs that got rolled out or it was PUT in a different run | 05:51 |
| charz | it will upload objects and get it back | 05:51 |
| clayg | so i have this object who's fragments are over the wrong nodes - just out of order mostly - so i want to understand why the PUT went so bad - but I don't see a PUT in the current log set for this object name - just GETs | 05:52 |
| clayg | ... and a HEAD | 05:52 |
| charz | clayg: which scenario/log you are looking? | 05:53 |
| charz | clayg: ec_small_test_read_only? | 05:54 |
| clayg | yeah i'm pretty sure the pre-seed option is doing a HEAD check before the PUT and these objects are all on the wrong nodes from a previous run | 06:01 |
| ho | I would like to modify https://github.com/openstack-dev/devstack/blob/master/lib/swift to pass check-* for #149930 in gerrit. Is there any rule to modify it? | 06:04 |
| charz | clayg: I think I can do a small change(object name) in scenario file, that'll avoid this problem. And ssbench will put the object to new path. | 06:08 |
| clayg | and there you go - Returning 503 for %s; found too many etags (%s) <- debug log level! | 06:13 |
| clayg | so there's two fragments with index 0 and one of them is an older timestamp - when it get's picked up in the list of six there's a debug message about hey I'm totally going to throw this request on the floor and barf for this very specific reason - but shhhhhh - don't tell anyone | 06:14 |
| kota_ | ho: you mean you want to change the gate? | 06:14 |
| clayg | fwiw this acctually *would* get better if we had a working reconstrcutor config on these nodes | 06:14 |
| kota_ | ho: if you want to make any changes the gerrit gate, you should push the changes to openstack-infra project. | 06:15 |
| ho | kota_: yes. I would like to add a new file to /etc/swift/ | 06:15 |
| ho | kota_: thanks! but my patch is not approved so is it ok to push the modification? | 06:16 |
| kota_ | I didn't see the patch yet | 06:16 |
| clayg | patch 149930 | 06:16 |
| patchbot | clayg: https://review.openstack.org/#/c/149930/ | 06:16 |
| kota_ | ho: maybe... no? | 06:16 |
| clayg | isn't that the cart before the horse tho? should the gate pass with your change without requiring new configs - this is called "backwards compatible" | 06:17 |
| ho | clayg: it is! | 06:17 |
| openstackgerrit | Prashanth Pai proposed openstack/swift: Refactor server side copy as middleware https://review.openstack.org/156923 | 06:17 |
| openstackgerrit | Prashanth Pai proposed openstack/swift: versioned writes middleware https://review.openstack.org/134347 | 06:17 |
| kota_ | clayg: right | 06:17 |
| kota_ | ho: if the change for the gate has backward compatibility, we can push it to gerrit always. | 06:18 |
| kota_ | ho: i don't know patch 149930 depends on whatever but if the gate change needs 149930, we should push it after 149930 merged, IMO. | 06:21 |
| patchbot | kota_: https://review.openstack.org/#/c/149930/ | 06:21 |
| clayg | g'night | 06:21 |
| kota_ | clayg: good night, have a good sleep ;) | 06:22 |
| clayg | notmyname: I'm pretty sure I know the 503's on GET are from - they are related but subtley different - there's no "fix" except make the GET smarter, which we might have to do, but it would probably have never been noticed except the qa cluster's reconstructor isn't running | 06:22 |
| clayg | notmyname: charz is going to fix the configs and I'll chat with torgomatic more about how we can maybe think about imporving the "throw out frags with crap old etags and look for more before you 503" situation in the proxy | 06:23 |
| ho | kota_: clayg: thanks :-) now i'm thinking... | 06:23 |
| ho | clayg: good night! | 06:23 |
| ho | kota_: I would like to add a file /etc/swift but this file is not used without my patch. means just push a patch to devstack has backward compatibility right? | 06:25 |
| kota_ | ho: sounds it has a compatibility but | 06:27 |
| kota_ | ho: it looks unnecessary to devstack users until your pach merged, right? | 06:28 |
| ho | kota_: right. | 06:31 |
| kota_ | ho: imho, the better way could be make tests w/o the file you want to add (something like using mock) and then | 06:31 |
| kota_ | s/make/to make/ | 06:31 |
| kota_ | ho: add the file the stuff needs to devstack (or infra-ish) after your change landed on master. | 06:33 |
| kota_ | ho: and then, create more tests using the real files you added. | 06:33 |
| openstackgerrit | Shri Javadekar proposed openstack/swift: Create object files directly in datadir. https://review.openstack.org/180883 | 06:34 |
| ho | kota_: i see. i will have the content of the file in the code now. thanks for your help/ :-) | 06:36 |
| kota_ | ho: nop, we want to keep the tracking files clean as possible and we can add files *when we need it* :P | 06:38 |
| *** kota_ has quit IRC | 06:46 | |
| *** joeljwright has joined #openstack-swift | 06:55 | |
| *** thumpba has quit IRC | 07:08 | |
| openstackgerrit | Christian Schwede proposed openstack/swift: Increase replicator log level if nothing got replicated https://review.openstack.org/180893 | 07:17 |
| *** joeljwright has quit IRC | 07:23 | |
| openstackgerrit | Daisuke Morita proposed openstack/swift-specs: Changing Policies spec https://review.openstack.org/168761 | 07:29 |
| *** krykowski has joined #openstack-swift | 07:31 | |
| cschwede | notmyname: I added you as a reviewer for the spec on adding cors in openstack (patch 179866) | 07:36 |
| patchbot | cschwede: https://review.openstack.org/#/c/179866/ | 07:36 |
| cschwede | i commented on the spec earlier, and there was a change regarding swift+global cors middleware. pls have a look if you find some time | 07:37 |
| *** jordanP has joined #openstack-swift | 07:41 | |
| openstackgerrit | Hisashi Osanai proposed openstack/swift: WIP: Enable Role-based access control using oslo.policy in Swift https://review.openstack.org/149930 | 07:41 |
| *** acoles_away is now known as acoles | 07:53 | |
| *** jistr has joined #openstack-swift | 07:54 | |
| *** dmorita has quit IRC | 07:59 | |
| *** joeljwright has joined #openstack-swift | 08:00 | |
| openstackgerrit | Christian Schwede proposed openstack/swift-specs: Prevent accidental object deletion spec https://review.openstack.org/155421 | 08:03 |
| openstackgerrit | Christian Schwede proposed openstack/swift: Increase replicator log level if nothing got replicated https://review.openstack.org/180893 | 08:07 |
| *** geaaru has joined #openstack-swift | 08:18 | |
| *** chlong has quit IRC | 08:23 | |
| openstackgerrit | Hisashi Osanai proposed openstack/swift: WIP: Enable Role-based access control using oslo.policy in Swift https://review.openstack.org/149930 | 08:34 |
| *** winggundamth has joined #openstack-swift | 09:06 | |
| winggundamth | hi. I follow http://docs.openstack.org/kilo/install-guide/install/apt/content/swift-install-controller-node.html with keystone but after I try doing swift list I got "Invalid user token - deferring reject downstream" error | 09:07 |
| winggundamth | I can do keystone user-list but not swift-list. This is full log http://paste.openstack.org/show/214954/ | 09:07 |
| *** silor has quit IRC | 09:11 | |
| *** aix has joined #openstack-swift | 09:14 | |
| openstackgerrit | Christian Schwede proposed openstack/swift-specs: Add event notification spec https://review.openstack.org/180914 | 09:46 |
| acoles | winggundamth: it looks like the keystone authtoken middleware is failing to validate your token - double check your config options for the authtoken middleware, and (if you can) verify that it is communicating with your keystone service (from its logs) | 09:49 |
| winggundamth | acoles: let me recheck my configuration. I think I already follow everything as docs said | 09:49 |
| winggundamth | acoles: just wonder if the docs right http://docs.openstack.org/kilo/install-guide/install/apt/content/swift-install-controller-node.html in [filter:authtoken] it used auth_uri and auth_url for var name. but on juno it used auth_uri and identity_uri with admin_ as prefix for user/pass/tenant name | 09:52 |
| acoles | winggundamth: i use the config described here http://docs.openstack.org/developer/swift/overview_auth.html#configuring-swift-to-use-keystone | 09:57 |
| acoles | ie identity_uri | 09:58 |
| acoles | winggundamth: i'm not familiar with the password plugin described in the doc you linked | 09:58 |
| acoles | winggundamth: btw auth_uri is only ever used to return to a client, not used to validate tokens, so that is not cause of problem | 09:59 |
| acoles | winggundamth: i'm just going to try the password plugin options... | 10:00 |
| winggundamth | acoles: http://paste.openstack.org/show/216005/ This is my proxy-server configuration | 10:00 |
| winggundamth | acoles: I used self-signed for https | 10:00 |
| winggundamth | acoles: and let me try the one you sent also | 10:03 |
| winggundamth | acoles: when used config the one you've sent. I still got the same error | 10:07 |
| openstackgerrit | Christian Schwede proposed openstack/swift-specs: Add metadata search spec https://review.openstack.org/180918 | 10:08 |
| winggundamth | acoles: I see this error in account-server log "HEAD /swift01/53488/AUTH_bebf7840e5914834968571d068ceff24" 404 - "HEAD http://storage.example.com/v1/AUTH_bebf7840e5914834968571d068ceff24" "tx463e99d6057d413aa8abb-00554b3956" "proxy-server 137" 0.0002 "-" 227 | 10:08 |
| acoles | winggundamth: ok, i cannot get the password plugin method to work from doc you linked :( | 10:13 |
| acoles | winggundamth: so is the account-server log from after you changed your auth token options? | 10:13 |
| winggundamth | acoles: yes after I change to your configuration | 10:14 |
| winggundamth | acoles: just wonder I couldn't make your config working either. Do you think this is Kilo bug? | 10:14 |
| acoles | winggundamth: so with 'my' config method do you still get a 401 or are you now getting a 404? | 10:15 |
| acoles | btw i'm running kilo | 10:15 |
| winggundamth | acoles: I still got 401 with your config | 10:16 |
| winggundamth | acoles: http://paste.openstack.org/show/216012/ this is config in authtoken with your config | 10:16 |
| winggundamth | let me test that user/pass is correct | 10:17 |
| winggundamth | yes confirm that swift user/pass is correct since I can do keystone token-get command | 10:18 |
| acoles | winggundamth: you have no port on the identity_uri | 10:18 |
| winggundamth | acoles: yes. this is expected since I'm using nginx to forward port to real identity port | 10:19 |
| winggundamth | acoles: and I can confirm that I can accessing to that domain and identity_url correct in endpoint | 10:21 |
| winggundamth | acoles: but I wonder in account-server log that I got 404 "HEAD /swift01/53488/AUTH_bebf7840e5914834968571d068ceff24" 404 - "HEAD http://storage.example.com/v1/AUTH_bebf7840e5914834968571d068ceff24" "tx925d24a0d7274fd68a4d5-00554b3947" "proxy-server 135" 0.0002 "-" 229 | 10:22 |
| winggundamth | the url of storage.example.com is http not https | 10:22 |
| winggundamth | acoles: checking with juno one that on our production right now and it's correct that url will be http | 10:26 |
| acoles | winggundamth: i think the 404 from the account server is normal until you actually create a container in the account - with account_autocreate=true the account doesn't actually exist until you create a container | 10:26 |
| winggundamth | acoles: i see | 10:26 |
| winggundamth | acoles: so do you think the problem is from proxy-server configuration right? | 10:27 |
| openstackgerrit | Christian Schwede proposed openstack/swift-specs: Add metadata search spec https://review.openstack.org/180918 | 10:27 |
| acoles | i can see from your paste you do have account_autocreate=true which is correct | 10:27 |
| *** silor has joined #openstack-swift | 10:28 | |
| acoles | winggundamth: can you paste your proxy log with current auth token config? | 10:28 |
| winggundamth | sure | 10:28 |
| acoles | winggundamth: and maybe bump your log level to debug | 10:30 |
| winggundamth | acoles: http://paste.openstack.org/show/216013/ this normal one first | 10:30 |
| winggundamth | acoles: this is debug one http://paste.openstack.org/show/216014/ | 10:32 |
| winggundamth | acoles: I see it that it authorized as anonymous | 10:33 |
| *** ho has quit IRC | 10:36 | |
| acoles | winggundamth: if you scroll right on your last paste you can see the token validation failure is due to SSLError: SSL exception connecting to https://admin-identity.example.com | 10:38 |
| winggundamth | acoles: so is it a self-signed bug problem? | 10:38 |
| winggundamth | acoles: should I try to upgrade keystoneclient to latest? | 10:42 |
| acoles | winggundamth: you can try - auth token is from keystonemiddleware not keystoneclient | 10:44 |
| acoles | winggundamth: sorry i'm out of ideas to help - other than it appearing to be auth_token not being able to connect to your keystone service | 10:45 |
| acoles | winggundamth: i'm stepping away for a bit | 10:46 |
| *** acoles is now known as acoles_away | 10:47 | |
| winggundamth | acoles: ok I'm out too. thank you so much for your help | 10:49 |
| *** cdelatte has quit IRC | 10:51 | |
| *** acoles_away is now known as acoles | 11:03 | |
| *** panbalag has joined #openstack-swift | 11:07 | |
| *** links has quit IRC | 11:31 | |
| *** links has joined #openstack-swift | 11:32 | |
| *** kota_ has joined #openstack-swift | 11:41 | |
| *** ppai has joined #openstack-swift | 11:50 | |
| *** openstackgerrit has quit IRC | 11:52 | |
| *** openstackgerrit has joined #openstack-swift | 11:52 | |
| *** lpabon has joined #openstack-swift | 12:01 | |
| *** thurloat_isgone is now known as thurloat | 12:03 | |
| *** ppai has quit IRC | 12:11 | |
| *** waterkinfe has joined #openstack-swift | 12:13 | |
| *** waterkinfe has quit IRC | 12:15 | |
| *** fthiagogv has joined #openstack-swift | 12:25 | |
| *** cdelatte has joined #openstack-swift | 12:28 | |
| *** proteusguy has joined #openstack-swift | 12:33 | |
| *** proteusguy has quit IRC | 12:34 | |
| *** tamizh_geek has quit IRC | 12:34 | |
| *** tamizh_geek has joined #openstack-swift | 12:35 | |
| *** bkopilov has quit IRC | 12:39 | |
| *** tamizh_geek has quit IRC | 12:39 | |
| *** fthiagogv has quit IRC | 12:43 | |
| *** silor has quit IRC | 12:44 | |
| *** dencaval has joined #openstack-swift | 12:45 | |
| *** fthiagogv has joined #openstack-swift | 12:46 | |
| *** ppai has joined #openstack-swift | 12:48 | |
| winggundamth | acoles_away: I got it working :) you have to specify insecure = True in authtoken | 12:51 |
| *** jkugel has joined #openstack-swift | 12:54 | |
| *** ppai has quit IRC | 12:55 | |
| winggundamth | acoles_away: I also test with config as in kilo installation and it works too | 12:57 |
| openstackgerrit | Christian Schwede proposed openstack/swift-specs: Add event notification spec https://review.openstack.org/180914 | 12:58 |
| *** kei_yama has quit IRC | 13:02 | |
| *** NM has joined #openstack-swift | 13:15 | |
| *** jkugel has left #openstack-swift | 13:16 | |
| acoles | winggundamth: great, glad you have it working, i've not used ssl in my SAIO so never set that option | 13:19 |
| *** david-lyle has quit IRC | 13:19 | |
| acoles | winggundamth: now i am wondering why i could not get the kilo instructions to work for me :/ | 13:19 |
| winggundamth | acoles: haha. it's your job now. I'll post mine configuration in case of you want it | 13:21 |
| acoles | heh | 13:21 |
| winggundamth | acoles: http://paste.openstack.org/show/216163/ here you go | 13:22 |
| acoles | winggundamth: thx | 13:23 |
| *** openstackstatus has quit IRC | 13:25 | |
| *** openstackstatus has joined #openstack-swift | 13:26 | |
| *** ChanServ sets mode: +v openstackstatus | 13:26 | |
| *** kota_ has quit IRC | 13:38 | |
| *** chlong has joined #openstack-swift | 13:39 | |
| *** SkyRocknRoll has quit IRC | 13:39 | |
| openstackgerrit | Emmanuel Cazenave proposed openstack/swift: X-Auth-Token should be a byestring. https://review.openstack.org/180098 | 13:44 |
| *** esker has joined #openstack-swift | 13:50 | |
| *** jamielennox is now known as jamielennox|away | 13:51 | |
| *** fanyaohong has joined #openstack-swift | 13:54 | |
| *** links has quit IRC | 13:54 | |
| *** zul has quit IRC | 13:57 | |
| *** delattec has joined #openstack-swift | 13:59 | |
| *** zul has joined #openstack-swift | 14:02 | |
| *** cdelatte has quit IRC | 14:02 | |
| *** jrichli has joined #openstack-swift | 14:06 | |
| *** Nadeem has joined #openstack-swift | 14:08 | |
| openstackgerrit | Alistair Coles proposed openstack/swift: Filter Etag key from ssync replication-headers https://review.openstack.org/173973 | 14:12 |
| acoles | clayg: i made this ^^ how i think you'd prefer it | 14:13 |
| *** atan8 has joined #openstack-swift | 14:14 | |
| openstackgerrit | Alistair Coles proposed openstack/swift: Filter Etag key from ssync replication-headers https://review.openstack.org/173973 | 14:18 |
| *** bkopilov has joined #openstack-swift | 14:20 | |
| *** wbhuber has joined #openstack-swift | 14:21 | |
| *** rookie has joined #openstack-swift | 14:26 | |
| *** rookie has quit IRC | 14:26 | |
| *** hdworker has joined #openstack-swift | 14:27 | |
| openstackgerrit | Alistair Coles proposed openstack/swift: X-Auth-Token should be a bytestring. https://review.openstack.org/180098 | 14:28 |
| *** hdworker has quit IRC | 14:35 | |
| *** Nadeem has quit IRC | 14:47 | |
| *** minwoob has joined #openstack-swift | 14:47 | |
| *** tamizh_geek has joined #openstack-swift | 14:48 | |
| *** nadeem has joined #openstack-swift | 15:08 | |
| notmyname | good morning | 15:09 |
| notmyname | clayg: thanks for looking more at the read errors | 15:09 |
| *** annegentle has joined #openstack-swift | 15:09 | |
| notmyname | cschwede: thanks | 15:10 |
| notmyname | cschwede: although I think it's _really_ strange to say "hey let's just add CORS support to eg a service that associates block devices with VMs". ie CORS is for browsers. it doesn't make much sense to implement it for most projects | 15:12 |
| *** annegentle has quit IRC | 15:14 | |
| *** annegentle has joined #openstack-swift | 15:14 | |
| openstackgerrit | Merged openstack/swift: Add missing statsd metrics section for object-reconstructor https://review.openstack.org/180655 | 15:33 |
| *** krykowski has quit IRC | 15:38 | |
| *** fthiagogv_ has joined #openstack-swift | 15:46 | |
| *** fthiagogv has quit IRC | 15:48 | |
| *** fthiagogv_ has quit IRC | 15:50 | |
| *** fthiagogv has joined #openstack-swift | 15:50 | |
| *** zaitcev has joined #openstack-swift | 15:54 | |
| *** ChanServ sets mode: +v zaitcev | 15:54 | |
| *** fanyaohong has quit IRC | 15:59 | |
| notmyname | jrichli: good morning | 16:03 |
| *** bkopilov has quit IRC | 16:04 | |
| *** nadeem has quit IRC | 16:04 | |
| *** hdworker has joined #openstack-swift | 16:05 | |
| jrichli | good morning. what's up? | 16:05 |
| notmyname | jrichli: pm sent | 16:06 |
| *** bkopilov has joined #openstack-swift | 16:09 | |
| *** winggundamth has quit IRC | 16:15 | |
| *** madgriff has joined #openstack-swift | 16:28 | |
| *** shakamunyi has quit IRC | 16:29 | |
| notmyname | I have to plug in to my big external monitor to do summit calendar stuff. my laptop screen is too big to track it all | 16:37 |
| *** atan8 has quit IRC | 16:38 | |
| *** hdworker has quit IRC | 16:38 | |
| *** annegentle has quit IRC | 16:39 | |
| *** cazino has left #openstack-swift | 16:48 | |
| *** jordanP has quit IRC | 17:02 | |
| *** tamizh_geek has quit IRC | 17:04 | |
| *** setmason has joined #openstack-swift | 17:09 | |
| setmason | Anybody know of a swift version of apt-transport-s3 (https://github.com/kyleshank/apt-transport-s3)? Which would allow you to store your apt repo directly in swift? | 17:12 |
| *** zhill has joined #openstack-swift | 17:16 | |
| *** jistr has quit IRC | 17:17 | |
| *** thumpba has joined #openstack-swift | 17:21 | |
| *** thumpba_ has joined #openstack-swift | 17:22 | |
| *** thumpba has quit IRC | 17:26 | |
| *** tamizh_geek has joined #openstack-swift | 17:38 | |
| *** tacotuesday has joined #openstack-swift | 17:39 | |
| *** annegentle has joined #openstack-swift | 17:40 | |
| *** madgriff has quit IRC | 17:42 | |
| *** erlon has joined #openstack-swift | 17:43 | |
| *** MVenesio has joined #openstack-swift | 17:45 | |
| *** anteaya has quit IRC | 17:45 | |
| *** bejorg has joined #openstack-swift | 17:45 | |
| *** annegentle has quit IRC | 17:46 | |
| bejorg | Hi guys, question regarding Swift quarantine zone, is there is way to see content ? Do we have to clear this quarantine zone ? | 17:46 |
| bejorg | I have some objects inside as displayed by swift-recon | 17:46 |
| *** vinsh has joined #openstack-swift | 17:47 | |
| *** jistr has joined #openstack-swift | 17:49 | |
| *** nadeem has joined #openstack-swift | 17:50 | |
| *** theanalyst has quit IRC | 17:52 | |
| *** nadeem has quit IRC | 17:54 | |
| *** geaaru has quit IRC | 17:55 | |
| *** theanalyst has joined #openstack-swift | 17:55 | |
| *** acoles is now known as acoles_away | 17:56 | |
| *** SkyRocknRoll has joined #openstack-swift | 18:02 | |
| *** SkyRocknRoll has quit IRC | 18:03 | |
| *** SkyRocknRoll has joined #openstack-swift | 18:03 | |
| *** SkyRocknRoll has quit IRC | 18:03 | |
| *** SkyRocknRoll has joined #openstack-swift | 18:03 | |
| *** tellesnobrega has quit IRC | 18:05 | |
| *** annegentle has joined #openstack-swift | 18:06 | |
| *** nadeem has joined #openstack-swift | 18:07 | |
| *** dencaval has quit IRC | 18:10 | |
| *** jrichli has quit IRC | 18:12 | |
| *** jordanP has joined #openstack-swift | 18:12 | |
| *** jordanP has quit IRC | 18:13 | |
| *** tacotuesday has quit IRC | 18:14 | |
| *** jrichli has joined #openstack-swift | 18:18 | |
| *** tellesnobrega has joined #openstack-swift | 18:22 | |
| *** SkyRocknRoll has quit IRC | 18:32 | |
| zaitcev | bejorg: As far as I know, it is necessary to clean up the quarantine area manually. In practice, however, Swift rarely exhibits bugs that quarantine improperly. So, if it starts filling up, your filesystem and/or device is toast and the node needs to be replaced. | 18:36 |
| zaitcev | bejorg: I am not an operator, however. Maybe a good question to ask at the openstack-operators list. | 18:36 |
| bejorg | I will, thanks for informations | 18:36 |
| *** tellesnobrega has quit IRC | 18:41 | |
| *** nadeem has quit IRC | 18:43 | |
| *** tamizh_g_ has joined #openstack-swift | 18:47 | |
| *** tamizh_geek has quit IRC | 18:49 | |
| *** SkyRocknRoll has joined #openstack-swift | 18:49 | |
| *** jistr has quit IRC | 18:51 | |
| *** nadeem has joined #openstack-swift | 18:53 | |
| *** dencaval has joined #openstack-swift | 18:54 | |
| clayg | morning | 18:58 |
| *** nadeem has quit IRC | 19:05 | |
| *** nadeem has joined #openstack-swift | 19:06 | |
| *** nadeem has quit IRC | 19:07 | |
| *** fthiagogv has quit IRC | 19:09 | |
| *** zul has quit IRC | 19:16 | |
| openstackgerrit | Pete Zaitcev proposed openstack/swift: Spell "rebalance" right in swift-ring-builder.1 https://review.openstack.org/181122 | 19:17 |
| zaitcev | I'm happy when people file bugs. I really am. | 19:18 |
| zaitcev | Heh. | 19:18 |
| notmyname | zaitcev: I like how gerrit auto-links that bug to launchpad and shows something for ubuntu | 19:20 |
| *** tellesnobrega has joined #openstack-swift | 19:20 | |
| notmyname | zaitcev: however, if you're tracked by how many tickets you close, this is great ;-) https://bugzilla.redhat.com/show_bug.cgi?id=1218269 | 19:21 |
| openstack | bugzilla.redhat.com bug 1218269 in openstack-swift "Action rebalence should be rebalance for command swift-ring-builder" [Unspecified,New] - Assigned to zaitcev | 19:21 |
| zaitcev | notmyname: It's worse. When I checked out, I did git checkout -b rhbz/1218269 (because I knew not to use bug/1218269), but it renamed the topic into "bug/1218269". | 19:21 |
| clayg | wat? | 19:24 |
| *** tamizh_geek has joined #openstack-swift | 19:24 | |
| clayg | heh, nice | 19:24 |
| notmyname | zaitcev: you should follow torgomatic's example and make branches named with funny puns | 19:25 |
| clayg | bejorg: that's intresting! What's in the objects? Are they fully replicated and durable in the correct parts? How old are they? Can you read metadata off them!? | 19:25 |
| *** tamizh_g_ has quit IRC | 19:26 | |
| zaitcev | clayg: Before bejorg can answer that, he has to find where the objects are placed and how to read the metadata (well, I can imagine running pyton, import pickle, p=pickle.load(open(xxxxx)) etc.) | 19:27 |
| clayg | setmason: hey that looks sweet! You could try it out with swift + s3 - it looks like it uses the domain/bucketname syntax instead of bucketname.domain - so it might work - or you could file an issue with the maintainer to add swift support | 19:27 |
| clayg | zaitcev: yeah swift-object-info will tell you most of that if the metadata is still there | 19:28 |
| zaitcev | oh, it takes a full filename | 19:28 |
| clayg | if not it's more fun! you have the hash (or you might, sometimes I think other things will get quarantined - depends what happened!) very exciting | 19:28 |
| *** zul has joined #openstack-swift | 19:31 | |
| clayg | notmyname: so, are we doing better in the qa cluster post-fix-the-reconstructor | 19:32 |
| notmyname | clayg: yay | 19:33 |
| clayg | yay or yeah? | 19:33 |
| notmyname | thanks for looking in to that. I've been swamped with summit talk prep this morning | 19:33 |
| clayg | oh - bummer | 19:33 |
| notmyname | oh, I saw "we are doing better..." | 19:33 |
| *** bkopilov has quit IRC | 19:33 | |
| notmyname | I haven't looked | 19:33 |
| clayg | heh - k | 19:33 |
| *** SkyRocknRoll has quit IRC | 19:33 | |
| notmyname | charz has some results, they looked ok-ish, but he cranked up concurrency again | 19:34 |
| clayg | crank it up! | 19:34 |
| clayg | torgomatic: so - broken unittest one the PUT fix for quorum? | 19:34 |
| clayg | or... broken code? | 19:34 |
| *** bkopilov has joined #openstack-swift | 19:43 | |
| bejorg | clayg: zaitcev: no idea what's inside objects. I just saw quarantine from swift-recon. From my understanding, auditor process moved them into quarantine zone (where is it ? do we have to flush it ?), and them replicator process has copied from health replica | 19:46 |
| bejorg | just 17 quarantine objects... curious to know if the swift-recon stats will go back to 0 or not... | 19:47 |
| notmyname | bejorg: there's a top-level directory in each device named "quarantine". that's where it's stored | 19:47 |
| bejorg | ok ! checking | 19:48 |
| notmyname | bejorg: and you can use swift-object-info to inspect the individual .data files there | 19:48 |
| notmyname | bejorg: and, no, that's not automatically cleaned up | 19:49 |
| bejorg | ok but how do I know "failed" objects ? if you have 100000 objects... | 19:49 |
| bejorg | nice, I got them from the quarantined folder | 19:50 |
| *** bkopilov has quit IRC | 19:53 | |
| *** bkopilov has joined #openstack-swift | 19:53 | |
| zaitcev | Objects are all self-identified, so swift-object-info may be used to parse the xattrs if available. However, it's usually not needed to try and "restore" them. The files may have bad sectors or whatnot. So what you need is track if excessive quarantining going on at the node, like you did. | 19:54 |
| *** tamizh_geek has quit IRC | 19:55 | |
| bejorg | then, it is safe to delete quarantined folder content ? rm -f /path/to/quarantined/folder/* ? | 19:55 |
| *** thumpba_ has quit IRC | 19:56 | |
| zaitcev | no | 19:57 |
| zaitcev | Well, at least "not without testing". Like I said, what if those files cover bad blocks? If you delete them, you release those blocks into free pool. | 19:58 |
| bejorg | ok, then how can I clear each quarantined folder ? right now, that pretty small on my case. 17 objects on quarantined = ~100MB | 19:58 |
| bejorg | good point! | 19:58 |
| zaitcev | I'd cat them into /dev/null at least, see if it works | 19:59 |
| *** bkopilov has quit IRC | 19:59 | |
| zaitcev | only then rm per your message above | 19:59 |
| *** bkopilov has joined #openstack-swift | 19:59 | |
| *** delattec has quit IRC | 19:59 | |
| *** annegentle has quit IRC | 20:02 | |
| *** annegentle has joined #openstack-swift | 20:02 | |
| bejorg | cat object > /dev/null all passed. what should I see in case of bad block ? | 20:03 |
| *** dencaval has quit IRC | 20:03 | |
| zaitcev | EIO from cat | 20:05 |
| zaitcev | I think you should be fine to delete them, but it's curious why they fell into quarantine. Maybe we have a bug that makes back-ends corrupt checksums, forget to store checksums, or some other misbehavior and your cluster makes it show up. | 20:06 |
| zaitcev | Excessive crashes could do that | 20:06 |
| bejorg | a swift-object-info on any quarantined objects says : Etag: 2af48fde1100d2cbc3d621c5ff84898f doesn't match file hash of 5ab66c6cd4eb3f876774473831787002! | 20:09 |
| bejorg | checksum were not equal | 20:09 |
| *** jkremer has joined #openstack-swift | 20:14 | |
| *** breitz has quit IRC | 20:17 | |
| *** breitz has joined #openstack-swift | 20:17 | |
| *** thurloat is now known as thurloat_isgone | 20:31 | |
| clayg | oh interesting - so bit rot maybe | 20:32 |
| clayg | bejorg: you can turn off the automatic checksum in swift-object-info (there's some flag, try --help, or calling it with no args) - might go ahead and print out the object info if it's not already | 20:32 |
| bejorg | swift-object-info -n | 20:47 |
| *** lpabon has quit IRC | 20:51 | |
| notmyname | torgomatic: you just really needed to get your review stats up? ;-) https://review.openstack.org/#/c/181122/ | 20:51 |
| torgomatic | notmyname: the exchange rate between Gerrit review stats and Reddit karma is looking especially good these days ;) | 20:52 |
| notmyname | I have you know that I once submitted an article, as an Anonymous Coward, to Slashdot and it made it to the front page. that's worth at least *some* reddit karma. | 20:53 |
| torgomatic | it's the broker fees that get you | 20:57 |
| *** annegentle has quit IRC | 21:04 | |
| *** Anticime1 is now known as Anticimex | 21:14 | |
| *** EmilienM is now known as EmilienM|afk | 21:16 | |
| openstackgerrit | Janie Richling proposed openstack/swift: WIP - working on the encryption feature. https://review.openstack.org/157907 | 21:29 |
| *** jrichli has quit IRC | 21:30 | |
| *** MVenesio has quit IRC | 21:37 | |
| *** jkremer has quit IRC | 21:37 | |
| openstackgerrit | Merged openstack/swift: Spell "rebalance" right in swift-ring-builder.1 https://review.openstack.org/181122 | 21:40 |
| *** NM has quit IRC | 21:45 | |
| *** bill_az has joined #openstack-swift | 21:47 | |
| *** david-lyle has joined #openstack-swift | 21:56 | |
| *** chlong has quit IRC | 21:59 | |
| *** david-lyle has quit IRC | 22:38 | |
| *** esker has quit IRC | 22:42 | |
| *** ujjain has quit IRC | 22:48 | |
| *** km has joined #openstack-swift | 22:51 | |
| *** ujjain has joined #openstack-swift | 22:52 | |
| *** ujjain has joined #openstack-swift | 22:52 | |
| *** vinsh has quit IRC | 23:00 | |
| mattoliverau | Morning all | 23:02 |
| torgomatic | hi | 23:02 |
| *** wbhuber has quit IRC | 23:13 | |
| notmyname | clayg: I'm going to look at the qa cluster again. have you done anything on it today? | 23:17 |
| clayg | notmyname: NOPE | 23:19 |
| notmyname | ok | 23:19 |
| notmyname | very few errors. still a lot of retires. didn't charz say those were IOError things? | 23:20 |
| notmyname | I'd like to rerun with a low concurrency | 23:21 |
| notmyname | see if those retries can be eliminated | 23:21 |
| notmyname | clayg: what should I be looking at about the weird placement of the FAs? did you find anything there last night? | 23:23 |
| clayg | notmyname: they were pre-existing objects | 23:23 |
| clayg | charz said that he fixed the ssbench configs so every run will use new object names (even if they're read only) | 23:23 |
| notmyname | ah | 23:24 |
| *** chlong has joined #openstack-swift | 23:24 | |
| notmyname | ok, I'll verify that. thanks | 23:24 |
| clayg | notmyname: anyway the pre-existing objects were in the wrong place because god only knows what you guys did to those rings - and there was no rebalancer/reconstructor running | 23:24 |
| notmyname | yup | 23:24 |
| notmyname | I thought each run was using a new container name | 23:24 |
| notmyname | that's what I get for thinkin' | 23:24 |
| clayg | notmyname: either way it'd be boss to verify the reconstructor is working now - and that if there is some data in the cluster it mostly seems in the right place | 23:24 |
| clayg | i could check that | 23:24 |
| clayg | notmyname: well I'd look at a benchmark run for a read job - and wehre you'd expect a bunch of PUTs in the logs all I would find is a HEAD | 23:25 |
| clayg | i should have spot checked the timestamps to see if it jived with my assersion they existed before the benchmark started - but the logs seemed to support it - I never tracked down the PUTs | 23:25 |
| clayg | ... and knowing that reconstruction couldn't rebalance - i guess I considered my theory validated :\ | 23:26 |
| notmyname | ok | 23:27 |
| notmyname | I don't see a reconstructer running on 01 | 23:27 |
| notmyname | hmm.. there is a config section in 2.conf | 23:28 |
| clayg | notmyname: bummer | 23:28 |
| *** kei_yama has joined #openstack-swift | 23:34 | |
| *** vinsh has joined #openstack-swift | 23:38 | |
| *** ho has joined #openstack-swift | 23:46 | |
| *** kota_ has joined #openstack-swift | 23:51 | |
| *** rmcall has joined #openstack-swift | 23:53 | |
| *** minwoob has quit IRC | 23:55 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!