| *** klamath has quit IRC | 00:05 | |
| *** _JZ_ has quit IRC | 00:07 | |
| JimCheung | Morning Kota_: | 00:09 |
|---|---|---|
| kota_ | morning! | 00:09 |
| kota_ | JimCheung: ^^ | 00:09 |
| JimCheung | Kota_: We're getting ready to put the 3rd party testing/verification online. In the mean time, do you have any additional comments on the code review? https://review.openstack.org/#/c/424353/ | 00:14 |
| patchbot | patch 424353 - liberasurecode - Add Phazr.IO libphazr backend to liberasurecode | 00:14 |
| *** catintheroof has quit IRC | 00:25 | |
| *** dmorita has joined #openstack-swift | 00:26 | |
| *** oshritf has quit IRC | 00:28 | |
| mattoliverau | kota_: morning | 00:35 |
| kota_ | mattoliverau: o/ | 00:35 |
| *** dmorita has quit IRC | 00:42 | |
| *** dmorita has joined #openstack-swift | 00:47 | |
| *** dmorita has quit IRC | 00:54 | |
| *** gatuus has quit IRC | 01:02 | |
| *** jamielennox is now known as jamielennox|away | 01:09 | |
| *** jamielennox|away is now known as jamielennox | 01:17 | |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/swift master: EC Fragment Duplication - Foundational Global EC Cluster Support https://review.openstack.org/219165 | 01:35 |
| *** sams-gleb has joined #openstack-swift | 01:54 | |
| *** sams-gleb has quit IRC | 01:59 | |
| *** cdelatte has quit IRC | 01:59 | |
| *** tqtran has quit IRC | 02:02 | |
| *** dmorita has joined #openstack-swift | 02:14 | |
| *** klrmn has quit IRC | 02:19 | |
| *** dmorita has quit IRC | 02:21 | |
| *** tanee_away is now known as tanee | 02:22 | |
| *** JimCheung has quit IRC | 02:34 | |
| *** adu has joined #openstack-swift | 02:48 | |
| *** Jeffrey4l_ has quit IRC | 03:08 | |
| *** Jeffrey4l_ has joined #openstack-swift | 03:09 | |
| *** Jeffrey4l_ has quit IRC | 03:13 | |
| *** Jeffrey4l_ has joined #openstack-swift | 03:14 | |
| *** Jeffrey4l_ has quit IRC | 03:16 | |
| *** Jeffrey4l_ has joined #openstack-swift | 03:16 | |
| *** adu has left #openstack-swift | 03:16 | |
| *** dmorita has joined #openstack-swift | 03:17 | |
| *** klrmn has joined #openstack-swift | 03:20 | |
| *** dmorita has quit IRC | 03:26 | |
| *** JimCheung has joined #openstack-swift | 03:27 | |
| *** JimCheung has quit IRC | 03:31 | |
| mahatic | good morning | 03:34 |
| mattoliverau | mahatic: morning | 03:39 |
| mahatic | mattoliverau: o/ | 03:39 |
| *** SkyRocknRoll has joined #openstack-swift | 03:43 | |
| *** david-lyle_ has joined #openstack-swift | 03:56 | |
| *** sams-gleb has joined #openstack-swift | 03:57 | |
| *** david-lyle has quit IRC | 03:58 | |
| *** takashi has joined #openstack-swift | 03:59 | |
| *** links has joined #openstack-swift | 03:59 | |
| *** sams-gleb has quit IRC | 04:02 | |
| *** dmorita has joined #openstack-swift | 04:08 | |
| *** SkyRocknRoll has quit IRC | 04:12 | |
| *** dmorita has quit IRC | 04:12 | |
| *** psachin has joined #openstack-swift | 04:27 | |
| *** zaitcev has quit IRC | 04:32 | |
| *** tanee is now known as tanee_away | 04:34 | |
| *** tanee_away is now known as tanee | 04:37 | |
| *** SkyRocknRoll has joined #openstack-swift | 04:40 | |
| *** sams-gleb has joined #openstack-swift | 04:59 | |
| *** tqtran has joined #openstack-swift | 05:01 | |
| *** zul has quit IRC | 05:02 | |
| *** sams-gleb has quit IRC | 05:03 | |
| *** tqtran has quit IRC | 05:06 | |
| *** klrmn has quit IRC | 05:10 | |
| *** dja has quit IRC | 05:13 | |
| *** cshastri has joined #openstack-swift | 05:13 | |
| openstackgerrit | Clay Gerrard proposed openstack/swift master: Deprecate broken handoffs_first in favor of handoffs_only https://review.openstack.org/425493 | 05:13 |
| *** zul has joined #openstack-swift | 05:25 | |
| *** SkyRocknRoll has quit IRC | 05:37 | |
| *** Jeffrey4l_ has quit IRC | 05:37 | |
| *** Jeffrey4l_ has joined #openstack-swift | 05:40 | |
| jrichli | clayg: what is our guarantee that probetests will pass against a particular configuration? I only ask because 'handoffs_only' mode breaks all tests that assert reconstruction. | 05:49 |
| jrichli | but 'handoffs_only' is not a normal setting, so maybe it is ok for them to just fail (no need to put in a check for mode and skip test) | 05:50 |
| *** links has quit IRC | 05:55 | |
| clayg | jrichli: I don't think that's strongly defined - there are definately some options/configs that cause probe tests to fail | 06:06 |
| jrichli | clayg: ok. in a way, i sort of like that they fail in that mode. it serves to emphasize that it is an invalid way to run things wrt to reconstruction. | 06:08 |
| clayg | To the extend we believe developers can extrat useful information from the probetest suite when running it against a given configuration - we should (probably not must?) try to support that | 06:09 |
| jrichli | true ... i can see it that way as well | 06:09 |
| clayg | support can either be raising a skiptest that includes the tests requirements; or even possibly altering the expected behavior | 06:10 |
| clayg | like the required_replicas tests just skiptest | 06:11 |
| *** links has joined #openstack-swift | 06:11 | |
| clayg | test expirer raises skiptest | 06:12 |
| clayg | container sync raises skiptest | 06:12 |
| *** jtomasek has quit IRC | 06:12 | |
| clayg | test_merge_policy_index hits /info | 06:14 |
| clayg | jrichli: which tests fail? | 06:15 |
| jrichli | clayg: http://paste.openstack.org/show/598745/ | 06:17 |
| jrichli | which I think are all failing in a way that makes sense | 06:18 |
| clayg | i like that all of them say "rebuild" or "reconstruct" - it's telling me the change works! :P | 06:18 |
| jrichli | exactly - that is what i was expressing about the failures serving a sort of purpose | 06:19 |
| clayg | heh | 06:19 |
| clayg | i can't make a judgement w/o trying to fix it | 06:21 |
| jrichli | about which way to go for support? yeah, maybe that would be a good follow-up. or are you wanting to add that support now? | 06:22 |
| clayg | ... and even fixing the tests once doesn't mean the next rebuild/reconstruct probetest we *add* won't introduce a similar failure | 06:22 |
| clayg | we have a similar problem with encryption - probetests worked with encryption at one time - do they *still* work? dunno! | 06:23 |
| jrichli | yes, they do. i typically run with crypto on in prob half my environments. | 06:24 |
| clayg | nice! | 06:24 |
| jrichli | you have to have it setup for container-sync for that to work though | 06:24 |
| clayg | ... right - the way those skip is pretty nice | 06:24 |
| jrichli | heh - actually, maybe the container-sync tests are the only ones that are really testing the crypto at a probetest level. | 06:26 |
| jrichli | cause the body doesn't match if crypto isn't in the internal-client pipeline. so it is going through get/put and verifying the body contents. | 06:26 |
| clayg | so I'm pretty sure in this case it would *not* be appropriate to change the assertion - so at best we're talking about skipping - it could delay that until the last moment - and the either fail "frag rebuilt in handoffs_first" or skip "test requires handoffs_only False to assert rebuild" | 06:26 |
| clayg | but if it just "passed" you dev could be in handoffs_only and get a false sense of security (could happen with skips too to a lesser degree) | 06:27 |
| jrichli | that sounds good - wait until the last moment | 06:29 |
| clayg | perhaps the most correct thing to do is fail - but with better annotatin "failed because handoffs_only is on" | 06:29 |
| jrichli | even better! +1 | 06:30 |
| clayg | idk, maybe skip is better | 06:31 |
| jrichli | then devs can still essentially assert the expected handoffs_only failures, and it still reveals an invalid way to run | 06:31 |
| clayg | ... does that have long term value? | 06:32 |
| clayg | like someone staring at broken probests only to realize they've had handoffs_only turned is worth avoiding | 06:32 |
| jrichli | I dont know how ops uses or doesn't use the probetests for verification. I am pretty sure that SL uses them at some point. | 06:33 |
| clayg | but the cheapest solution to that is to pre-check in probe.common? | 06:33 |
| clayg | the probetests can only run in a development environment - the audiance might have different titles/roles at their org - but if you're running development tests in a development environment - I hope I can treat you like a developer :\ | 06:35 |
| jrichli | ok, right ... | 06:35 |
| jrichli | so one big pre-check with a warning - prob what i'd do | 06:37 |
| jrichli | and skips | 06:38 |
| clayg | maybe ony in the ECProbeTest case? | 06:38 |
| jrichli | yep | 06:39 |
| clayg | ... same sort of "probetests pass I must not have broken rebuild" risk :\ | 06:39 |
| jrichli | well, i really liked the "failed because handoffs_only is on" in exactly the right spots. its just more work. | 06:40 |
| clayg | jrichli: ok, so ECProbeTest could parse the reconstructor config for each node on start up and just keep a map | 06:41 |
| clayg | then anytime when you fail in those existing tests you say "... and also probably you left handoffs_only on" | 06:42 |
| clayg | then if we forget to add that same decoration to another test in the future that also fails with handoffs_only on - at least you still have the message on the others? | 06:42 |
| clayg | ... but you can't "run tests" with handoffs_only on - that's not a supported config? | 06:42 |
| clayg | it all feels rather arbitrary | 06:43 |
| mahatic | ..but how likely is it that devs would turn on handoffs_only and then run probe tests? | 06:44 |
| jrichli | its weird cause we have a defined/valid behavior for an invalid mode of operation :-) | 06:44 |
| mahatic | unless we're testing a feature, in which case we'd already some context | 06:45 |
| mahatic | s/already/already have | 06:45 |
| jrichli | the map sounds like a plan. | 06:45 |
| jrichli | mahatic: true, it would be rare. I am not probably the normal dev case, but I know that I have left weird configs like this in my environments and didn't realize it later. :/ | 06:46 |
| jrichli | because I don | 06:47 |
| jrichli | 't use vagrant - and I reuse my VMs - I am more vulnerable to that sort of thing | 06:47 |
| *** links has quit IRC | 06:48 | |
| mahatic | right. so maybe leaving a meaningful message on the failure "fails because of handoffs_only mode is on" would do, although we may face inconsistencies in future if we forget to add the msg to newer tests | 06:49 |
| mahatic | but maybe it's better than leaving as is? | 06:49 |
| clayg | jrichli: +1 to that - probe.common's checks exist because of the time I lost only to discover my config was no longer "supported" because I changed it while reviewing some unrelated thing | 06:49 |
| jrichli | its my bed time. you all have a good night / day :-) | 06:52 |
| *** oshritf has joined #openstack-swift | 06:53 | |
| *** rcernin has joined #openstack-swift | 06:54 | |
| mahatic | jrichli: good night! | 06:56 |
| *** oshritf has quit IRC | 06:59 | |
| *** sams-gleb has joined #openstack-swift | 07:01 | |
| *** links has joined #openstack-swift | 07:02 | |
| *** tqtran has joined #openstack-swift | 07:04 | |
| *** oshritf has joined #openstack-swift | 07:05 | |
| *** sams-gleb has quit IRC | 07:05 | |
| *** tqtran has quit IRC | 07:08 | |
| *** ChubYann has quit IRC | 07:08 | |
| *** takashi has quit IRC | 07:14 | |
| *** tesseract has joined #openstack-swift | 07:15 | |
| *** SkyRocknRoll has joined #openstack-swift | 07:16 | |
| *** sams-gleb has joined #openstack-swift | 07:25 | |
| *** bkopilov has quit IRC | 07:26 | |
| *** bkopilov_ has quit IRC | 07:27 | |
| *** reedip has left #openstack-swift | 07:29 | |
| *** oshritf has quit IRC | 07:40 | |
| *** zul has quit IRC | 07:50 | |
| *** links has quit IRC | 07:50 | |
| *** links has joined #openstack-swift | 07:52 | |
| openstackgerrit | Merged openstack/swift master: Hisashi Osanai's responsibility has been changed https://review.openstack.org/432747 | 07:56 |
| openstackgerrit | Merged openstack/swift master: Add Apache License 2.0 description in ../conf.py https://review.openstack.org/432718 | 07:57 |
| openstackgerrit | Merged openstack/swift master: [Trivial]Fix some type errors in ../sample.conf https://review.openstack.org/432740 | 07:57 |
| openstackgerrit | Merged openstack/swift master: Fix swift-get-nodes arg parsing for missing ring https://review.openstack.org/406012 | 07:58 |
| *** takashi has joined #openstack-swift | 08:01 | |
| *** pcaruana has joined #openstack-swift | 08:11 | |
| *** amoralej|off is now known as amoralej | 08:15 | |
| *** rledisez has joined #openstack-swift | 08:19 | |
| *** geaaru has joined #openstack-swift | 08:34 | |
| *** zul has joined #openstack-swift | 08:54 | |
| *** joeljwright has joined #openstack-swift | 09:02 | |
| *** ChanServ sets mode: +v joeljwright | 09:02 | |
| *** tqtran has joined #openstack-swift | 09:05 | |
| *** tqtran has quit IRC | 09:09 | |
| *** psachin has quit IRC | 09:11 | |
| *** psachin has joined #openstack-swift | 09:11 | |
| *** psachin has quit IRC | 09:17 | |
| *** Dw_Sn has joined #openstack-swift | 09:19 | |
| Dw_Sn | I have n00b question, I have private buckets " S3 " interface , I am using keystone for auth, can I allow HTTP apps like wget to be able to download the files with authentication tokens ? instead of going into dashboard or using s3cmd or aws cli ? | 09:19 |
| openstackgerrit | Christian Schwede proposed openstack/swift master: Add support to increase object ring partition power https://review.openstack.org/337297 | 09:24 |
| openstackgerrit | Christian Schwede proposed openstack/swift master: Add support to increase object ring partition power https://review.openstack.org/337297 | 09:26 |
| *** bkopilov has joined #openstack-swift | 09:26 | |
| *** bkopilov_ has joined #openstack-swift | 09:27 | |
| *** jordanP has joined #openstack-swift | 09:29 | |
| cschwede_ | notmyname: ^^ Next patchset. It would be nice to merge it soon, but I don't want to rush it - we might want to merge it shortly *after* the release? | 09:32 |
| *** psachin has joined #openstack-swift | 09:34 | |
| *** psachin has quit IRC | 09:36 | |
| *** psachin has joined #openstack-swift | 09:36 | |
| *** dmorita has joined #openstack-swift | 09:41 | |
| *** oshritf has joined #openstack-swift | 09:51 | |
| *** cbartz has joined #openstack-swift | 09:52 | |
| *** cbartz has quit IRC | 09:53 | |
| *** dmorita has quit IRC | 09:54 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/swift master: Updated from global requirements https://review.openstack.org/88736 | 09:55 |
| *** tesseract-RH has joined #openstack-swift | 09:56 | |
| *** psachin has quit IRC | 10:00 | |
| *** silor has joined #openstack-swift | 10:00 | |
| *** psachin has joined #openstack-swift | 10:01 | |
| *** tesseract has quit IRC | 10:04 | |
| *** dmorita has joined #openstack-swift | 10:10 | |
| *** silor1 has joined #openstack-swift | 10:10 | |
| *** silor has quit IRC | 10:12 | |
| *** silor1 is now known as silor | 10:12 | |
| *** dmorita has quit IRC | 10:14 | |
| *** mvk has quit IRC | 10:26 | |
| onovy | notmyname: hi. i send few reviews for newton stable branch. can you look into it please? | 10:26 |
| *** mvk has joined #openstack-swift | 10:52 | |
| *** tqtran has joined #openstack-swift | 11:07 | |
| *** sams-gleb has quit IRC | 11:11 | |
| *** tqtran has quit IRC | 11:11 | |
| *** sams-gleb has joined #openstack-swift | 11:11 | |
| *** kei_yama has quit IRC | 11:11 | |
| *** dmorita has joined #openstack-swift | 11:12 | |
| *** sams-gleb has quit IRC | 11:15 | |
| *** dmorita has quit IRC | 11:17 | |
| *** JimCheung has joined #openstack-swift | 11:35 | |
| *** dmorita has joined #openstack-swift | 11:39 | |
| *** JimCheung has quit IRC | 11:40 | |
| *** sams-gleb has joined #openstack-swift | 11:53 | |
| *** cbartz has joined #openstack-swift | 11:59 | |
| *** bkopilov has quit IRC | 12:43 | |
| *** bkopilov_ has quit IRC | 12:43 | |
| *** SkyRocknRoll has quit IRC | 13:03 | |
| *** cshastri has quit IRC | 13:04 | |
| *** catintheroof has joined #openstack-swift | 13:07 | |
| *** tqtran has joined #openstack-swift | 13:08 | |
| *** tqtran has quit IRC | 13:13 | |
| *** jamielennox is now known as jamielennox|away | 13:21 | |
| *** links has quit IRC | 13:28 | |
| *** mvk has quit IRC | 13:32 | |
| *** amoralej is now known as amoralej|lunch | 13:50 | |
| *** cbartz has quit IRC | 13:59 | |
| *** cbartz has joined #openstack-swift | 14:00 | |
| *** takashi has quit IRC | 14:03 | |
| *** klamath has joined #openstack-swift | 14:09 | |
| Dw_Sn | I am getting this error "proxy-server: Deferring reject downstream" I have S3 enabled on swift, the buckets are public so I am not using any auth.. | 14:11 |
| *** bkopilov has joined #openstack-swift | 14:24 | |
| *** bkopilov_ has joined #openstack-swift | 14:24 | |
| *** geaaru has quit IRC | 14:25 | |
| *** mvk has joined #openstack-swift | 14:26 | |
| *** links has joined #openstack-swift | 14:29 | |
| *** rsFF- has quit IRC | 14:45 | |
| *** dmorita has quit IRC | 14:46 | |
| *** bkopilov_ has quit IRC | 14:47 | |
| *** bkopilov has quit IRC | 14:47 | |
| *** caiobrentano has joined #openstack-swift | 14:48 | |
| *** rsFF has joined #openstack-swift | 14:48 | |
| *** amoralej|lunch is now known as amoralej | 14:49 | |
| *** neonpastor has joined #openstack-swift | 14:49 | |
| *** briancline has quit IRC | 14:52 | |
| *** briancline has joined #openstack-swift | 14:52 | |
| *** dmorita has joined #openstack-swift | 15:02 | |
| onovy | notmyname: ah. jenkins for stable branch is broken. who will fix it? :) | 15:02 |
| *** dmorita has quit IRC | 15:06 | |
| *** sams-gleb has quit IRC | 15:10 | |
| *** sams-gleb has joined #openstack-swift | 15:11 | |
| *** Dw_Sn has quit IRC | 15:14 | |
| *** bkopilov has joined #openstack-swift | 15:15 | |
| *** sams-gleb has quit IRC | 15:15 | |
| *** bkopilov_ has joined #openstack-swift | 15:15 | |
| *** sams-gleb has joined #openstack-swift | 15:28 | |
| *** dmorita has joined #openstack-swift | 15:34 | |
| *** _JZ_ has joined #openstack-swift | 15:38 | |
| *** dmorita has quit IRC | 15:39 | |
| *** tonanhngo has joined #openstack-swift | 15:43 | |
| *** links has quit IRC | 15:47 | |
| *** dmorita has joined #openstack-swift | 15:50 | |
| *** psachin has quit IRC | 15:52 | |
| *** gatuus has joined #openstack-swift | 15:53 | |
| *** mvk has quit IRC | 15:54 | |
| *** dmorita has quit IRC | 15:54 | |
| *** rcernin has quit IRC | 16:11 | |
| notmyname | good morning | 16:26 |
| *** david-lyle has joined #openstack-swift | 16:26 | |
| notmyname | onovy: I'll be getting on the bus in about 10 minutes. I'll check when I get to the office. thanks for bringing it up | 16:26 |
| *** david-lyle_ has quit IRC | 16:29 | |
| *** dmorita has joined #openstack-swift | 16:36 | |
| *** chsc has joined #openstack-swift | 16:38 | |
| *** chsc has joined #openstack-swift | 16:38 | |
| *** joeljwright has quit IRC | 16:38 | |
| *** joeljwright has joined #openstack-swift | 16:38 | |
| *** ChanServ sets mode: +v joeljwright | 16:38 | |
| *** dmorita has quit IRC | 16:40 | |
| *** rcernin has joined #openstack-swift | 16:49 | |
| *** cbartz has quit IRC | 16:49 | |
| *** mvk has joined #openstack-swift | 16:55 | |
| *** tqtran has joined #openstack-swift | 16:57 | |
| *** catintheroof has quit IRC | 17:07 | |
| *** catintheroof has joined #openstack-swift | 17:07 | |
| *** dmorita has joined #openstack-swift | 17:07 | |
| *** dmorita has quit IRC | 17:12 | |
| notmyname | good morning (for realz) | 17:19 |
| *** pcaruana has quit IRC | 17:31 | |
| *** rledisez has quit IRC | 17:34 | |
| *** JimCheung has joined #openstack-swift | 17:35 | |
| *** nicodemus_ has joined #openstack-swift | 17:39 | |
| nicodemus_ | hello | 17:43 |
| nicodemus_ | I'm preparing to deploy a swift cluster, my datanodes have 20 2TB SATA disks, two 240GB SSD disks and two 16GB SATADOM modules | 17:44 |
| nicodemus_ | I was wondering what would be the ideal disk layout for this datanode (obviously the OS will be installed in the SATADOM modules in RAID 1) | 17:45 |
| *** gatuus has quit IRC | 17:47 | |
| *** gatuus has joined #openstack-swift | 17:48 | |
| *** JimCheung has quit IRC | 17:49 | |
| *** JimCheung has joined #openstack-swift | 17:50 | |
| *** dmorita has joined #openstack-swift | 17:53 | |
| clayg | heyoh! | 17:57 |
| clayg | happy valentines day | 17:57 |
| gatuus | ^.^ | 18:05 |
| clayg | I want to point lp bug #1664474 at http://docs.openstack.org/developer/swift/overview_large_objects.html but I'm embarrassed it still mentions DLO first? | 18:14 |
| openstack | Launchpad bug 1664474 in OpenStack Object Storage (swift) "Uploading large video data to swift storage" [Undecided,New] https://launchpad.net/bugs/1664474 | 18:14 |
| clayg | I thought there was a change to fix that order?! | 18:14 |
| *** dmorita has quit IRC | 18:15 | |
| *** Renich has joined #openstack-swift | 18:15 | |
| Renich | hello | 18:16 |
| clayg | idk, i couldn't find the change :'( | 18:16 |
| *** oshritf has quit IRC | 18:16 | |
| Renich | I am getting many of these: https://paste.fedoraproject.org/558239/09623014/ | 18:17 |
| Renich | any ideas? | 18:17 |
| clayg | io conention on the storage nodes | 18:18 |
| clayg | you can stop background daemons in the short term | 18:18 |
| clayg | if the problem goes away you can try to tune those | 18:18 |
| clayg | if not - you can try to tune your object-servers | 18:18 |
| clayg | ... or both | 18:18 |
| clayg | more complex deployment topologies like servers-per-port or seperate-replication-networks are targeted at mitigating and isolating io conention | 18:19 |
| *** tesseract-RH has quit IRC | 18:26 | |
| notmyname | nicodemus_: nothing fancy. put the HDDs into the object ring, but the SSDs into the account and container rings, and put the OS on the fast flash (like you said). | 18:27 |
| *** aluria has joined #openstack-swift | 18:27 | |
| *** klrmn has joined #openstack-swift | 18:29 | |
| *** Renich has quit IRC | 18:30 | |
| *** Renich has joined #openstack-swift | 18:30 | |
| *** dmorita has joined #openstack-swift | 18:31 | |
| *** dmorita has quit IRC | 18:35 | |
| *** jordanP has quit IRC | 18:38 | |
| aluria | hi o/ -- I was looking for info on account-replicator.failures counter, but I can't find it in http://docs.openstack.org/developer/swift/admin_guide.html#recon-replication | 18:41 |
| *** DuncanT has quit IRC | 18:41 | |
| aluria | does this counter only get reset when restarting the swift-account-replicator process? | 18:41 |
| aluria | how can I find the reason of those failures? | 18:43 |
| notmyname | aluria: it gets reset on every cycle | 18:46 |
| aluria | notmyname: by cycle, you mean everytime replication finishes? | 18:47 |
| notmyname | aluria: yep. normally you run the replicators in a "forever" mode (technically, there's a "once" mode and this is the not-once mode, ie forever) | 18:48 |
| notmyname | aluria: ...and it will process data. wait. and process again | 18:48 |
| notmyname | aluria: stats a reset at the start of each process cycle | 18:48 |
| aluria | notmyname: I see -- thanks for the clarification | 18:48 |
| notmyname | aluria: you can see it at https://github.com/openstack/swift/blob/master/swift/common/db_replicator.py#L628 inside of run_once(). and the run_forever() method just repeatedly calls run_once() | 18:49 |
| nicodemus_ | notmyname, thanks! | 18:50 |
| notmyname | aluria: that code is shared by both the account and the container replicator | 18:50 |
| Renich | clayg: was that for me? | 18:50 |
| aluria | notmyname: ack -- and apart from looking the return codes for the http methods, is there any other source of info on the reason of a failure? ie. I see that localhost:6000/recon/object shows failures per storage | 18:51 |
| *** tnovacik has joined #openstack-swift | 18:57 | |
| *** dmorita has joined #openstack-swift | 19:17 | |
| *** ChubYann has joined #openstack-swift | 19:18 | |
| *** zaitcev has joined #openstack-swift | 19:19 | |
| *** ChanServ sets mode: +v zaitcev | 19:19 | |
| *** dmorita has quit IRC | 19:22 | |
| *** amoralej is now known as amoralej|off | 19:23 | |
| *** _JZ_ has quit IRC | 19:28 | |
| *** Renich has quit IRC | 20:01 | |
| *** dmorita has joined #openstack-swift | 20:04 | |
| *** dmorita has quit IRC | 20:09 | |
| clayg | Renich: yes, good luck | 20:14 |
| *** jamielennox|away is now known as jamielennox | 20:17 | |
| *** openstackgerrit has quit IRC | 20:18 | |
| *** CowboyPride has quit IRC | 20:22 | |
| *** abqkawi1000 has joined #openstack-swift | 20:37 | |
| *** JimCheung has quit IRC | 20:41 | |
| *** JimCheung has joined #openstack-swift | 20:45 | |
| *** dmorita has joined #openstack-swift | 20:51 | |
| abqkawi1000 | Hoping someone can point me in the right direction. I am unable to get my swift-proxy to cooperate with keystone. I have double checked that the swift user has admin on the service tenant with: openstack role list --project service --user swift. | 20:51 |
| abqkawi1000 | My output from client:swift -V 2 -A http://10.203.2.1:5000/v2.0 -U admin:myadminname -K myadminpass stat --debug DEBUG:keystoneclient.auth.identity.v2:Making authentication request to http://10.203.2.1:5000/v2.0/tokens INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): 10.203.2.1 DEBUG:requests.packages.urllib3.connectionpool:"POST /v2.0/tokens HTTP/1.1" 200 3689 INFO:requests.packages.urll | 20:52 |
| abqkawi1000 | Ack--- ok pate buffer | 20:52 |
| abqkawi1000 | From my client --swift -V 2 -A http://10.203.2.1:5000/v2.0 -U admin:myadminname -K myadminpass stat --debug DEBUG:keystoneclient.auth.identity.v2:Making authentication request to http://10.203.2.1:5000/v2.0/tokens INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): 10.203.2.1 DEBUG:requests.packages.urllib3.connectionpool:"POST /v2.0/tokens HTTP/1.1" 200 3689 | 20:53 |
| abqkawi1000 | INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): 10.203.0.101 DEBUG:requests.packages.urllib3.connectionpool:"HEAD /v1/AUTH_175ea570020a4fe8a8bd1ed2f7070d84 HTTP/1.1" 503 0 INFO:swiftclient:REQ: curl -i http://10.203.0.101:8080/v1/AUTH_175ea570020a4fe8a8bd1ed2f7070d84 -I -H "X-Auth-Token: gAAAAABYo2quDtDS8GTAIKoziw5--ujlbdSXFs_sFVmSKwSPzsy1ixSARZ2HV4QZ0lMVg6UG1z8ZjlIt2_ac1GDrq3an4H7cY_NLb | 20:53 |
| patchbot | Error: No closing quotation | 20:53 |
| abqkawi1000 | Then | 20:53 |
| abqkawi1000 | INFO:swiftclient:RESP STATUS: 503 Service Unavailable INFO:swiftclient:RESP HEADERS: {u'Date': u'Tue, 14 Feb 2017 20:38:06 GMT', u'Content-Length': u'0', u'Content-Type': u'text/html; charset=UTF-8', u'X-Trans-Id': u'txb205b4ec511e48f18eb1e-0058a36aae'} | 20:53 |
| abqkawi1000 | EBUG:requests.packages.urllib3.connectionpool:"HEAD /v1/AUTH_175ea570020a4fe8a8bd1ed2f7070d84 HTTP/1.1" 503 0 INFO:swiftclient:REQ: curl -i http://10.203.0.101:8080/v1/AUTH_175ea570020a4fe8a8bd1ed2f7070d84 -I -H "X-Auth-Token: gAAAAABYo2quDtDS8GTAIKoziw5--ujlbdSXFs_sFVmSKwSPzsy1ixSARZ2HV4QZ0lMVg6UG1z8ZjlIt2_ac1GDrq3an4H7cY_NLbBuwlCi86wxs9u0qhrtY9Kf_JAYcZP6_k7iNzAV276TzYGA0BnuSPL1a3h4vfA" | 20:53 |
| abqkawi1000 | From proxy server | 20:54 |
| abqkawi1000 | Feb 14 13:35:44 cee-cldswift proxy-server: Rejecting request Feb 14 13:35:44 cee-cldswift proxy-server: Identity server rejected authorization Feb 14 13:35:44 cee-cldswift proxy-server: Identity response: {"error": {"message": "The request you have made requires authentication.", "code": 401, "title": "Unauthorized"}} | 20:54 |
| abqkawi1000 | From controller | 20:54 |
| abqkawi1000 | 2017-02-14 20:38:07.103 21283 INFO keystone.common.wsgi [req-07f2cdeb-6109-4c6d-8c5b-7d65df45b68a - - - - -] POST http://10.203.2.1:35357/v3/auth/tokens 2017-02-14 20:38:07.116 21283 WARNING keystone.common.wsgi [req-07f2cdeb-6109-4c6d-8c5b-7d65df45b68a - - - - -] Authorization failed. The request you have made requires authentication. from 10.203.2.9 | 20:54 |
| abqkawi1000 | 2017-02-14 20:38:07.120 21283 INFO eventlet.wsgi.server [req-07f2cdeb-6109-4c6d-8c5b-7d65df45b68a - - - - -] 10.203.0.101,10.203.2.9 - - [14/Feb/2017 20:38:07] "POST /v3/auth/tokens HTTP/1.1" 401 419 0.020417 2017-02-14 20:38:09.155 21279 DEBUG keystone.middleware.auth [req-28fb8ba2-3b99-4e68-a57a-9c3038f771f4 - - - - -] There is either no auth token in the request or the certificate issuer is not trusted. | 20:55 |
| abqkawi1000 | No auth context will be set. _build_auth_context /usr/lib/python2.7/dist-packages/keystone/middleware/auth.py:71 | 20:55 |
| *** dmorita has quit IRC | 20:55 | |
| *** caiobrentano_ has joined #openstack-swift | 21:02 | |
| jrichli | abqkawi1000: I am not very experienced with keystone, but I'd say either your token was rejected by keystone, or swift has no authority to talk to keystone. | 21:03 |
| *** caiobrentano has quit IRC | 21:04 | |
| abqkawi1000 | jrichli: Yes that is exactly what it feels like to me | 21:04 |
| jrichli | abqkawi1000: you can use http://paste.openstack.org/ when you want to share some logs such as this, btw | 21:04 |
| abqkawi1000 | jrichli: ah thanks for that log paste url | 21:04 |
| jrichli | np | 21:05 |
| abqkawi1000 | so how do I make keystone trust my swift proxy box? I setup user swift that is admin for the service project | 21:05 |
| abqkawi1000 | jrichli: my proxy-server.conf users the swift user and password in authtoken middleware | 21:07 |
| jrichli | abqkawi1000: since i am not very experienced with keystone, I am going to first point you here: http://docs.openstack.org/developer/swift/overview_auth.html#troubleshooting-tips-for-keystoneauth-deployment | 21:07 |
| jrichli | abqkawi1000: but i will be poking around myself to see if i can help more | 21:07 |
| abqkawi1000 | jrichli: terrific! I will be reading myself. Thanks a ton! | 21:08 |
| zaitcev | Man, now that portante is no more, coverage keeps slipping. | 21:08 |
| zaitcev | Or, actually, it's very good in numbers - 93%, but also a huge number of small gaps. | 21:09 |
| mattoliverau | Morning | 21:11 |
| *** abqkawi1000 has quit IRC | 21:12 | |
| *** abqkawi1000 has joined #openstack-swift | 21:13 | |
| *** tnovacik has quit IRC | 21:15 | |
| notmyname | hello mattoliverau | 21:15 |
| *** abqkawi1000 has quit IRC | 21:15 | |
| *** abqkawi1000 has joined #openstack-swift | 21:16 | |
| mattoliverau | notmyname: hey man, how was the field trip yesterday? | 21:16 |
| jrichli | abqkawi1000: I think swift has to be setup as a 'service' in keystone rather than a user. at least, that is what i have seen done | 21:16 |
| notmyname | mattoliverau: yesterday was good. thanks | 21:16 |
| abqkawi1000 | rjichli: correct. The swift service ID shows up with a openstack service list | 21:17 |
| abqkawi1000 | jrichli: my endpoints are setup pointing to my swift proxy | 21:18 |
| *** silor has quit IRC | 21:18 | |
| abqkawi1000 | jrichli: This is what puzzles me "There is either no auth token in the request or the certificate issuer is not trusted. No auth context will be set." | 21:20 |
| abqkawi1000 | jrlichli: I am hitting http so I cannot imagine why a cert is an issue. Why a token would have been stripped boggles me. From my proxy it is clear a /v1/AUTH_*************** was sent to keystone | 21:21 |
| tdasilva | zaitcev: maybe we can convince portante to come back ;) | 21:27 |
| notmyname | tdasilva: I think you'd be better able to do that than anyone else in here ;-) | 21:27 |
| *** gatuus has quit IRC | 21:27 | |
| tdasilva | notmyname: believe me, i've tried... | 21:28 |
| notmyname | heh. believe me, me too ;-) | 21:29 |
| tdasilva | heh | 21:30 |
| *** dmorita has joined #openstack-swift | 21:32 | |
| notmyname | in another episode of "FYI in OpenStack..." | 21:34 |
| notmyname | the TC decided, in their meeting that ended 30 min ago, that there is now a "only infra-approved bots are allowed on Gerrit" | 21:34 |
| notmyname | when I asked about more detail about how that would be addressed.. | 21:35 |
| *** Jeffrey4l_ has quit IRC | 21:35 | |
| notmyname | I was told "it's done not by active policing in any automated fashion - but by referring to it as an agreement when a misbehaving bot surfaces" | 21:35 |
| *** Jeffrey4l_ has joined #openstack-swift | 21:35 | |
| notmyname | so there's that. if you use a bot to post on gerrit, I guess it's not allowed any more | 21:35 |
| *** dmorita has quit IRC | 21:36 | |
| tdasilva | does that mean clayg can't do reviews anymore? | 21:39 |
| *** rcernin has quit IRC | 21:44 | |
| notmyname | that's hilarious | 21:44 |
| jrichli | tdasilva: lol | 21:45 |
| *** dmorita has joined #openstack-swift | 21:46 | |
| *** dmorita has quit IRC | 21:53 | |
| mattoliverau | lol | 21:59 |
| *** dmorita has joined #openstack-swift | 22:09 | |
| *** dmorita has quit IRC | 22:13 | |
| *** gatuus has joined #openstack-swift | 22:23 | |
| pdardeau | is the eviction of object data from page cache controlled by "keep_cache_size" in diskfile.py? | 22:24 |
| pdardeau | (default of 5MB)? | 22:25 |
| *** dmorita has joined #openstack-swift | 22:25 | |
| notmyname | pdardeau: yes. but also the other config var in there. keep_cache_private = T|F or something like that | 22:27 |
| notmyname | IIRC it effectively comes down to publicly accessible content that is small is kept in the cache | 22:27 |
| notmyname | but also, IIRC it's a little confusing at first glance because it's phrased negatively | 22:28 |
| pdardeau | notmyname: thx. this one? keep_cache: should resulting reads be kept in the buffer cache | 22:29 |
| notmyname | yeah that sounds right | 22:29 |
| notmyname | pdardeau: in practice, I'm not sure it actually helps anything. and I'm pretty sure it's never actually been measured. and I worry that it might actually hurt things in some cases | 22:31 |
| notmyname | I'm much *much* more interested in the OVH work around small file optimization instead of this | 22:31 |
| *** tonanhngo_ has joined #openstack-swift | 22:34 | |
| *** sanchitmalhotra1 has joined #openstack-swift | 22:34 | |
| *** klrmn1 has joined #openstack-swift | 22:36 | |
| notmyname | I just got an email with logistics info about the PTG. check your email and you'll likely have one too (sent individually, so no link) | 22:38 |
| *** tonyb_ has joined #openstack-swift | 22:38 | |
| *** kozhukalov_ has joined #openstack-swift | 22:40 | |
| *** dims has joined #openstack-swift | 22:41 | |
| *** mtreinish_ has joined #openstack-swift | 22:41 | |
| notmyname | changelog entries: "some pretty core parts of swift that you just assumed were working now work even better." | 22:41 |
| *** klrmn has quit IRC | 22:42 | |
| *** tonanhngo has quit IRC | 22:42 | |
| *** dims_ has quit IRC | 22:42 | |
| *** mtreinish has quit IRC | 22:42 | |
| *** sanchitmalhotra has quit IRC | 22:42 | |
| *** kozhukalov has quit IRC | 22:42 | |
| *** tonyb has quit IRC | 22:42 | |
| *** wasmum has quit IRC | 22:42 | |
| *** jamielennox has quit IRC | 22:42 | |
| *** fungi has quit IRC | 22:42 | |
| *** mtreinish_ is now known as mtreinish | 22:42 | |
| *** sanchitmalhotra1 is now known as sanchitmalhotra | 22:43 | |
| jrichli | notmyname: sounds good! better than i could have come up with | 22:44 |
| *** kozhukalov_ is now known as kozhukalov | 22:44 | |
| notmyname | mostly I just consolidate it down to "Swift 1.13.0: the best one yet, please upgrade" | 22:44 |
| notmyname | oh. yeah. 1.13.0 was the best one at that time. | 22:45 |
| notmyname | turns out 2.13.0 is the best one, now | 22:45 |
| *** tonanhngo_ has quit IRC | 22:46 | |
| *** zigo has quit IRC | 22:46 | |
| *** chlong has quit IRC | 22:46 | |
| *** early has quit IRC | 22:46 | |
| *** swifterdarrell has quit IRC | 22:46 | |
| *** mmotiani has quit IRC | 22:46 | |
| *** AbyssOne has quit IRC | 22:46 | |
| *** CrackerJackMack has quit IRC | 22:46 | |
| *** mcarden has quit IRC | 22:46 | |
| *** darrenc has quit IRC | 22:46 | |
| *** mattoliverau has quit IRC | 22:46 | |
| *** jarbod_ has quit IRC | 22:46 | |
| *** cppforlife_ has quit IRC | 22:46 | |
| *** dcourtoi has quit IRC | 22:46 | |
| *** tonyb_ is now known as tonyb | 22:46 | |
| *** swifterdarrell has joined #openstack-swift | 22:47 | |
| *** ChanServ sets mode: +v swifterdarrell | 22:47 | |
| *** early has joined #openstack-swift | 22:48 | |
| *** acorwin has quit IRC | 22:48 | |
| *** fungi has joined #openstack-swift | 22:50 | |
| *** zigo has joined #openstack-swift | 22:52 | |
| *** chlong has joined #openstack-swift | 22:52 | |
| *** AbyssOne has joined #openstack-swift | 22:52 | |
| *** mcarden has joined #openstack-swift | 22:52 | |
| *** darrenc has joined #openstack-swift | 22:52 | |
| *** mattoliverau has joined #openstack-swift | 22:52 | |
| *** jarbod_ has joined #openstack-swift | 22:52 | |
| *** dcourtoi has joined #openstack-swift | 22:52 | |
| *** adams.freenode.net sets mode: +v mattoliverau | 22:52 | |
| *** acorwin has joined #openstack-swift | 22:52 | |
| *** tonanhngo has joined #openstack-swift | 22:52 | |
| *** sams-gleb has quit IRC | 22:53 | |
| *** cppforlife_ has joined #openstack-swift | 22:53 | |
| *** zigo is now known as Guest27057 | 22:54 | |
| *** dmorita has quit IRC | 23:00 | |
| *** abqkawi1000_ has joined #openstack-swift | 23:01 | |
| *** CrackerJackMack has joined #openstack-swift | 23:01 | |
| *** abqkawi1000 has quit IRC | 23:01 | |
| *** dmorita has joined #openstack-swift | 23:03 | |
| *** nicodemus_ has quit IRC | 23:13 | |
| MooingLemur | What could cause the condition that while all devices have weight 0 and no partitions assigned, but the node has 23 container dbs and container-replicator doesn't remove them? | 23:16 |
| MooingLemur | https://bpaste.net/show/3702d8f1e36d | 23:17 |
| MooingLemur | 2.7.0 from centos-openstack-mitaka | 23:18 |
| *** dja has joined #openstack-swift | 23:28 | |
| *** kei_yama has joined #openstack-swift | 23:30 | |
| MooingLemur | "attempted: 23" attempted to do what? I don't even see any connect()s when stracing :) | 23:35 |
| notmyname | dude. it tried soooooo hard to do it. you don't even know. not once. not twice. 23 TIMES! that's pretty hard work. ;-) | 23:37 |
| *** chsc has quit IRC | 23:38 | |
| notmyname | hmm...looking at the code to refresh my memory on that | 23:39 |
| notmyname | looks like maybe there's 23 DBs that are in the deleted state | 23:39 |
| notmyname | specifically deleted but not yet past the reclaim age | 23:41 |
| notmyname | at least that seems like a code path that would increment the attemted count but not do anything else | 23:42 |
| MooingLemur | oh, maybe.. I haven't tried running sql on those yet. | 23:42 |
| notmyname | can you check the disks to see if there are .db files there? how many? what happens if you do swift-container-info on them? | 23:42 |
| MooingLemur | oh, I could do that too | 23:42 |
| MooingLemur | there are *.db files | 23:42 |
| notmyname | it would be really cool if there are 23 .db files ;-) | 23:43 |
| *** dmorita has quit IRC | 23:44 | |
| MooingLemur | here's one https://bpaste.net/show/c25b4dd072ed | 23:44 |
| MooingLemur | it's a 2.5GB .db file :) | 23:45 |
| MooingLemur | -[/srv/node:#]- find . -name \*.db | wc -l | 23:45 |
| MooingLemur | 23 | 23:45 |
| *** dmorita has joined #openstack-swift | 23:46 | |
| notmyname | nice! | 23:46 |
| MooingLemur | if it's supposed to be deleted, I'm satisfied, and I'll be happy to proceed knowing that I'm not going to lose anything real by wiping this data | 23:46 |
| *** dmorita has quit IRC | 23:47 | |
| notmyname | yep. so that DB is deleted (see the delete timestamp as the latest) | 23:47 |
| notmyname | the size is just from the fact that you could have had a lot of tombstone rows in it (and we never vacuum the DBs) | 23:47 |
| MooingLemur | yeah | 23:48 |
| notmyname | so as long as you have a healthy cluster and you've had a container replication cycle complete successfully, you're totally ok by yanking the drive | 23:48 |
| MooingLemur | I'm just wondering why it's been there since May of last year though (May 13 is the mtime for that db) | 23:48 |
| MooingLemur | could it be because the account doesn't exist anymore? | 23:48 |
| notmyname | you said this is on a drive with zero weight? | 23:49 |
| MooingLemur | yeah | 23:49 |
| notmyname | could be the fact the account doesn't exist | 23:49 |
| notmyname | could be the fact it's zero weight | 23:49 |
| notmyname | that sounds like the sort of corner case we'd have in swift ;-) | 23:49 |
| MooingLemur | it's zero weight as of this week. It drained all the dbs except for these 23 | 23:49 |
| notmyname | (read: "bug") | 23:49 |
| notmyname | since it's just 23 of them, a quick by-hand check for the delete timestamp would be what you need to check. and a check in the logs that replication cycle is good (but I think that is shown in the first pastebin you had) | 23:50 |
| MooingLemur | yeah, I'm satisfied. Thanks for the explanation :D | 23:51 |
| notmyname | MooingLemur: might be worth opening a bug in LP about it. "empty container db on zero-weight drive doesn't get deleted" | 23:51 |
| *** sams-gleb has joined #openstack-swift | 23:54 | |
| *** wasmum has joined #openstack-swift | 23:57 | |
| *** dmorita has joined #openstack-swift | 23:58 | |
| *** sams-gleb has quit IRC | 23:58 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!