| notmyname | zaitcev: go go gadget notmyname http://d.not.mn/EC_swift_proxy_design.pdf | 00:00 |
|---|---|---|
| zaitcev | notmyname: thanks | 00:00 |
| zaitcev | and right in the root? Nice. | 00:00 |
| notmyname | zaitcev: d.not.mn is my dropbox container in cloud files. http://programmerthoughts.com/programming/simple-script-to-upload-content-to-cloud-files/ | 00:01 |
| notmyname | jsut a cname to the CDN url | 00:01 |
| zaitcev | It's like the whole new world. You don't program Swift on your iPad by any chance | 00:02 |
| notmyname | use some ssh ios app to do remote coding on my server somewhere. now that would be a neat trick | 00:03 |
| zaitcev | I thought about it, but I cannot bring myself to trust iPad with any passphrases... I imagine Apple can keylog anything there. | 00:04 |
| zaitcev | "Why trello? Mostly because Launchpad isn't sufficient to have discussions around the design, and gerrit only supports discussion around a particular piece of code." -- Ironically written in an e-mail message to a mailing list. | 00:06 |
| *** occupant has quit IRC | 00:07 | |
| *** occupant has joined #openstack-swift | 00:08 | |
| *** tzumainn has left #openstack-swift | 00:12 | |
| zaitcev | I ordered my hotel for Hackathon today. Decided to stay in a motel by freeway. $90/night. On-site hotels are $150-200. | 00:13 |
| notmyname | oh yeah. I need to book hotel/travel too | 00:14 |
| zaitcev | notmyname: I see "Given 10+4 RS coding parameters, EC Quorum is 11 volumes". How did you arrive to the value of 11? | 00:43 |
| zaitcev | RS(10,4) can tolerate 2 failures at the most, isn't that right? | 00:44 |
| zaitcev | Or not? | 00:44 |
| *** shri1 has quit IRC | 00:44 | |
| *** Honghui_ has joined #openstack-swift | 00:52 | |
| *** Honghui_ has quit IRC | 01:03 | |
| *** gyee has quit IRC | 01:13 | |
| *** Honghui has joined #openstack-swift | 01:22 | |
| *** gyee has joined #openstack-swift | 01:42 | |
| *** bach has joined #openstack-swift | 01:42 | |
| *** zhiyan_ is now known as zhiyan | 01:52 | |
| *** Honghui has quit IRC | 01:55 | |
| *** bach has quit IRC | 01:57 | |
| *** Honghui has joined #openstack-swift | 02:05 | |
| *** shakamunyi has quit IRC | 02:11 | |
| *** Honghui has quit IRC | 02:45 | |
| *** bach has joined #openstack-swift | 02:45 | |
| *** Honghui has joined #openstack-swift | 02:45 | |
| *** praveenkumar has joined #openstack-swift | 02:51 | |
| *** Honghui__ has joined #openstack-swift | 02:58 | |
| *** Honghui has quit IRC | 03:00 | |
| *** haoma____ has quit IRC | 03:07 | |
| *** bach has quit IRC | 03:09 | |
| *** haomaiwang has joined #openstack-swift | 03:09 | |
| *** bach has joined #openstack-swift | 03:15 | |
| *** bach has quit IRC | 03:17 | |
| *** Honghui__ has quit IRC | 03:48 | |
| *** Honghui has joined #openstack-swift | 03:49 | |
| notmyname | zaitcev: AFAIK RS(10,4) can tolerate 4 failures. but I'm no EC expert. the idea of "one plus minimum" is what I meant there. eg with 3 replicas, quorum is 2. | 03:56 |
| *** gyee has quit IRC | 04:04 | |
| *** chandan_kumar has joined #openstack-swift | 04:05 | |
| *** haomaiwang has quit IRC | 04:08 | |
| *** haomaiwang has joined #openstack-swift | 04:10 | |
| *** Honghui has quit IRC | 04:12 | |
| *** Honghui has joined #openstack-swift | 04:15 | |
| *** ppai has joined #openstack-swift | 04:17 | |
| *** haomai___ has joined #openstack-swift | 04:22 | |
| *** haomaiwang has quit IRC | 04:25 | |
| *** Honghui has quit IRC | 04:29 | |
| *** Honghui has joined #openstack-swift | 04:52 | |
| *** ppai has quit IRC | 05:03 | |
| zaitcev | notmyname: thanks, I see. I thought that for a regular EC tolerance is half, e.g. 2t = n - k. | 05:07 |
| notmyname | zaitcev: whatever the number is, the proposal in that original pdf is basically just saying "one more than that" so that there must be 2 drive failures (ie >1) before data is lost | 05:09 |
| zaitcev | notmyname: makes sense | 05:09 |
| *** ppai has joined #openstack-swift | 05:16 | |
| *** cheri has joined #openstack-swift | 05:20 | |
| *** zaitcev has quit IRC | 05:24 | |
| *** ppai has quit IRC | 05:24 | |
| *** nshaikh has joined #openstack-swift | 05:26 | |
| *** ppai has joined #openstack-swift | 05:36 | |
| *** chandan_kumar has quit IRC | 06:01 | |
| *** gvernik has joined #openstack-swift | 06:03 | |
| *** ppai has quit IRC | 06:14 | |
| *** madhuri has joined #openstack-swift | 06:15 | |
| *** chandan_kumar has joined #openstack-swift | 06:16 | |
| *** Honghui has quit IRC | 06:21 | |
| *** chandan_kumar has quit IRC | 06:22 | |
| *** cheri has quit IRC | 06:26 | |
| *** bvandenh has joined #openstack-swift | 06:31 | |
| *** bvandenh has quit IRC | 06:31 | |
| madhuri | How all services are being tracked in Swift? What happens when a service lets say an object server goes down? | 06:34 |
| *** psharma has joined #openstack-swift | 06:35 | |
| *** haomai___ has quit IRC | 06:36 | |
| *** haomaiwang has joined #openstack-swift | 06:36 | |
| clayg | that kind of monitoring is implemented outside of swift, either with some custom tooling, something like zenos, or a proprietary product. | 06:36 |
| madhuri | Yeah I have looked for zenos and nagios too | 06:38 |
| madhuri | That provides support for swift | 06:38 |
| madhuri | The healthcheck is also kind of monitoring tool for proxy server only?? | 06:39 |
| clayg | it's really there for load balancers | 06:39 |
| *** ppai has joined #openstack-swift | 06:39 | |
| *** chandan_kumar has joined #openstack-swift | 06:40 | |
| madhuri | So health check is just for load balancers. Right? | 06:40 |
| clayg | well... it's there... it does what it does... you can use it however you like? But the orignal usecase I'm pretty sure had loadbalancers in mind - that's how we use it as a tool to control rolling swift upgrades. | 06:41 |
| *** Manish_ has joined #openstack-swift | 06:41 | |
| madhuri | Ok. | 06:42 |
| madhuri | Can't it be used at storage service level? | 06:42 |
| madhuri | means account, container, object server?? | 06:43 |
| clayg | well the proxy implements it's own notion of error limiting, but as a piece of middleware I would imagine you could stick in the pipeline if you were keen on that? | 06:43 |
| madhuri | error limiting is also kinda monitoring at proxy server level? | 06:47 |
| clayg | well you're not really alerted to it - which would be a defining factor of something I called "monitoring" - it's just a built in property of the proxy that it will work around failures associated with a individual storage node that is not responding in a timely fashion | 06:48 |
| madhuri | Thanks clayg | 06:52 |
| *** Honghui has joined #openstack-swift | 06:53 | |
| madhuri | Is there any documentation available for nagios or zenos integration with swift? | 06:54 |
| *** haomaiw__ has joined #openstack-swift | 07:02 | |
| hugokuo | Did anyone ever produce a document about the compatibility between S3 and Swift(swift3) ? Am working on it hope that's not a re-do works in the community | 07:04 |
| *** haomaiwang has quit IRC | 07:05 | |
| clayg | hugokuo: you've seen https://github.com/fujita/swift3/issues/3 | 07:07 |
| hugokuo | clayg: That's awesome !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! | 07:11 |
| hugokuo | clayg: You save my time .. appreciate | 07:12 |
| hugokuo | thanks fifieldt... | 07:12 |
| *** mmcardle has joined #openstack-swift | 07:15 | |
| openstackgerrit | Clay Gerrard proposed a change to openstack/swift: Keep status_changed_at up-to-date with status changes. https://review.openstack.org/86714 | 07:27 |
| openstackgerrit | Clay Gerrard proposed a change to openstack/swift: Extend direct_client https://review.openstack.org/88144 | 07:27 |
| openstackgerrit | Clay Gerrard proposed a change to openstack/swift: Add write_fake_ring helper to test.unit https://review.openstack.org/88151 | 07:27 |
| openstackgerrit | Clay Gerrard proposed a change to openstack/swift: Add container-reconciler daemon https://review.openstack.org/85908 | 07:27 |
| openstackgerrit | Clay Gerrard proposed a change to openstack/swift: Convert POLICIES to list for random.choice https://review.openstack.org/88150 | 07:27 |
| openstackgerrit | Clay Gerrard proposed a change to openstack/swift: Extend interface on InternalClient https://review.openstack.org/88152 | 07:27 |
| openstackgerrit | Clay Gerrard proposed a change to openstack/swift: Merge container storage_policy_index values. https://review.openstack.org/86720 | 07:27 |
| openstackgerrit | Clay Gerrard proposed a change to openstack/swift: Expose container info on deleted containers. https://review.openstack.org/86708 | 07:27 |
| openstackgerrit | Clay Gerrard proposed a change to openstack/swift: Fix X-Backend-Storage-Policy-Index requests https://review.openstack.org/90929 | 07:27 |
| openstackgerrit | Clay Gerrard proposed a change to openstack/swift: Fix recursive Policy patching https://review.openstack.org/90930 | 07:27 |
| openstackgerrit | Clay Gerrard proposed a change to openstack/swift: Cleanup container update headers in obj server tests https://review.openstack.org/90931 | 07:27 |
| openstackgerrit | Clay Gerrard proposed a change to openstack/swift: Update FakeLogger with better log level handling https://review.openstack.org/90932 | 07:27 |
| openstackgerrit | Clay Gerrard proposed a change to openstack/swift: Make FakeRing a use more real code https://review.openstack.org/90933 | 07:28 |
| openstackgerrit | Clay Gerrard proposed a change to openstack/swift: Add reconciler probetest outline https://review.openstack.org/90934 | 07:28 |
| openstackgerrit | Clay Gerrard proposed a change to openstack/swift: Add Storage Policies to container.backend https://review.openstack.org/90935 | 07:28 |
| openstackgerrit | Clay Gerrard proposed a change to openstack/swift: Add Storage Policy to container put/delete object https://review.openstack.org/90936 | 07:28 |
| openstackgerrit | Clay Gerrard proposed a change to openstack/swift: Enqueue misplaced objects during container replication https://review.openstack.org/90937 | 07:28 |
| openstackgerrit | Clay Gerrard proposed a change to openstack/swift: Add LRUCache to common.utils https://review.openstack.org/90938 | 07:28 |
| *** chalcedony has quit IRC | 07:28 | |
| omame | clayg: go to sleep ;) | 07:28 |
| *** cheri has joined #openstack-swift | 07:32 | |
| fifieldt | lol hugokuo | 07:43 |
| fifieldt | sadly I have not had time to work on this recently | 07:43 |
| * hugokuo am going to conduct the tests on 1.13 here | 07:44 | |
| fifieldt | that would be very interesting :) | 07:51 |
| *** foexle has joined #openstack-swift | 07:53 | |
| *** Honghui has quit IRC | 08:05 | |
| *** joeljwright has joined #openstack-swift | 08:12 | |
| *** Honghui has joined #openstack-swift | 08:14 | |
| *** psharma has quit IRC | 08:17 | |
| *** jamie_h has joined #openstack-swift | 08:23 | |
| *** psharma has joined #openstack-swift | 08:23 | |
| *** chalcedony has joined #openstack-swift | 08:28 | |
| *** mmcardle has quit IRC | 08:30 | |
| openstackgerrit | Joel Wright proposed a change to openstack/python-swiftclient: Add importable SwiftService incorporating shell.py logic https://review.openstack.org/85453 | 09:03 |
| *** chandan_kumar has quit IRC | 09:14 | |
| *** chandan_kumar has joined #openstack-swift | 09:15 | |
| *** mmcardle has joined #openstack-swift | 09:34 | |
| *** haomaiw__ has quit IRC | 09:34 | |
| *** haomaiwa_ has joined #openstack-swift | 09:35 | |
| *** haomaiwa_ has quit IRC | 09:55 | |
| *** haomaiwang has joined #openstack-swift | 09:56 | |
| *** [1]gvernik has joined #openstack-swift | 10:31 | |
| *** gvernik has quit IRC | 10:33 | |
| *** [1]gvernik is now known as gvernik | 10:33 | |
| *** Honghui has quit IRC | 10:47 | |
| *** [1]gvernik has joined #openstack-swift | 11:01 | |
| *** gvernik has quit IRC | 11:03 | |
| *** [1]gvernik is now known as gvernik | 11:03 | |
| *** fifieldt_ has joined #openstack-swift | 11:16 | |
| *** fifieldt has quit IRC | 11:17 | |
| *** Manish_ has quit IRC | 11:35 | |
| *** haomaiwang has quit IRC | 11:42 | |
| *** haomaiwang has joined #openstack-swift | 11:43 | |
| *** cheri has quit IRC | 11:58 | |
| *** mkollaro has quit IRC | 12:03 | |
| *** mkollaro has joined #openstack-swift | 12:03 | |
| *** mkollaro has quit IRC | 12:05 | |
| *** mkollaro has joined #openstack-swift | 12:05 | |
| *** mkollaro has quit IRC | 12:06 | |
| *** mkollaro has joined #openstack-swift | 12:06 | |
| *** mkollaro has quit IRC | 12:11 | |
| *** cheri has joined #openstack-swift | 12:16 | |
| *** mkollaro has joined #openstack-swift | 12:17 | |
| *** zhiyan is now known as zhiyan_ | 12:23 | |
| *** cheri has quit IRC | 12:32 | |
| openstackgerrit | Eamonn O'Toole proposed a change to openstack/swift: Parallel object auditor https://review.openstack.org/59778 | 12:33 |
| *** mmcardle has quit IRC | 12:35 | |
| *** ppai has quit IRC | 12:36 | |
| openstackgerrit | Peter Portante proposed a change to openstack/swift: In-process swift server for functional tests https://review.openstack.org/66108 | 12:45 |
| *** cheri has joined #openstack-swift | 12:45 | |
| *** mkollaro1 has joined #openstack-swift | 12:45 | |
| *** mkollaro has quit IRC | 12:45 | |
| *** Trixboxer has joined #openstack-swift | 12:46 | |
| portante | dfg: interested in taking https://review.openstack.org/66108 for a spin? | 12:46 |
| *** tongli has joined #openstack-swift | 12:55 | |
| *** jamie_h has quit IRC | 13:01 | |
| *** jamie_h has joined #openstack-swift | 13:01 | |
| *** fifieldt_ is now known as fifieldt | 13:03 | |
| *** mmcardle has joined #openstack-swift | 13:05 | |
| *** gvernik has quit IRC | 13:11 | |
| *** psharma has quit IRC | 13:31 | |
| *** lpabon has joined #openstack-swift | 13:46 | |
| *** lpabon has quit IRC | 13:50 | |
| *** cheri has quit IRC | 13:59 | |
| *** tdasilva has joined #openstack-swift | 14:02 | |
| *** pconstantine has joined #openstack-swift | 14:03 | |
| openstackgerrit | Eamonn O'Toole proposed a change to openstack/swift: Sleep between object ZBF process forks https://review.openstack.org/91037 | 14:11 |
| luisbg | morning | 14:15 |
| *** zhiyan_ is now known as zhiyan | 14:23 | |
| *** changbl has quit IRC | 14:27 | |
| *** praveenkumar_ has joined #openstack-swift | 14:47 | |
| *** praveenkumar has quit IRC | 14:49 | |
| *** praveenkumar_ is now known as praveenkumar | 14:51 | |
| *** Longgeek_ has joined #openstack-swift | 15:01 | |
| *** tongli has quit IRC | 15:14 | |
| *** jamie_h has quit IRC | 15:17 | |
| *** jamie_h has joined #openstack-swift | 15:18 | |
| *** kevinc_ has joined #openstack-swift | 15:19 | |
| *** shakamunyi has joined #openstack-swift | 15:21 | |
| *** haomaiw__ has joined #openstack-swift | 15:26 | |
| *** haomaiwang has quit IRC | 15:28 | |
| *** zhiyan is now known as zhiyan_ | 15:31 | |
| openstackgerrit | Eamonn O'Toole proposed a change to openstack/swift: Sleep between object ZBF process forks https://review.openstack.org/91037 | 15:38 |
| *** foexle has quit IRC | 15:38 | |
| otoolee | gholt: portante: Thanks for the rapid review of the ZBF sleep change. I've posted a revised patch set with unit tests that actually pass this time | 15:40 |
| *** mwstorer has joined #openstack-swift | 15:40 | |
| *** zhiyan_ is now known as zhiyan | 15:42 | |
| *** Longgeek_ has quit IRC | 15:49 | |
| *** kevinc_ has quit IRC | 15:57 | |
| *** kevinc_ has joined #openstack-swift | 15:59 | |
| *** mwstorer has quit IRC | 16:02 | |
| *** joeljwright has quit IRC | 16:05 | |
| *** zhiyan is now known as zhiyan_ | 16:11 | |
| notmyname | good morning | 16:13 |
| physcx | good afternoon | 16:14 |
| notmyname | looks like San Francisco has one of the latest average workday start times. http://fivethirtyeight.com/datalab/which-cities-sleep-in-and-which-get-to-work-early/ :-) | 16:16 |
| physcx | notmyname: we did track down the cause of the problem with chunked transfer-encoding and how to duplicate it on vagrant-- will post a write up about it later | 16:17 |
| physcx | also a fix | 16:17 |
| notmyname | ah, nice. thanks | 16:17 |
| *** chandan_kumar has quit IRC | 16:19 | |
| openstackgerrit | Eamonn O'Toole proposed a change to openstack/swift: Sleep between object ZBF process forks https://review.openstack.org/91037 | 16:22 |
| *** joeljwright has joined #openstack-swift | 16:36 | |
| *** nshaikh has quit IRC | 16:36 | |
| *** joeljwright has quit IRC | 16:48 | |
| peluse_ | FYI, a few storage policy patches to wrap up support for modules that we missed previously. Available for review now... especially for cores who have not looked too much into Policies: | 16:56 |
| peluse_ | Add Storage Policy Support to Container Sync https://review.openstack.org/#/c/86469/ | 16:56 |
| peluse_ | Add Storage Policy Support to Recon Middleware https://review.openstack.org/#/c/87387/ | 16:56 |
| peluse_ | Add Storage Policy support to the Account Reaper https://review.openstack.org/#/c/88098/ | 16:56 |
| *** shri has joined #openstack-swift | 16:56 | |
| *** krtaylor has quit IRC | 16:56 | |
| peluse_ | notmyname: on the reaper, clayg and I talked last week and feel like its OK to add policy support as it is in this patch and deal with the internal client comment later... hope that's cool w/you | 16:57 |
| *** gvernik has joined #openstack-swift | 16:59 | |
| notmyname | peluse_: ah, cool. I haven't had a chance to catch up on that since last week, but it's on my todo list for today and tomorrow | 17:00 |
| peluse_ | notmyname: no prob - just trying to lobby for some of these misc patches to get merged so we can focus on reviee of clayg's reconciler here very soon | 17:01 |
| *** zaitcev has joined #openstack-swift | 17:06 | |
| *** ChanServ sets mode: +v zaitcev | 17:06 | |
| *** rcleere has joined #openstack-swift | 17:18 | |
| *** chandan_kumar has joined #openstack-swift | 17:20 | |
| *** gyee has joined #openstack-swift | 17:23 | |
| *** changbl has joined #openstack-swift | 17:23 | |
| *** shri1 has joined #openstack-swift | 17:29 | |
| *** shri1 has quit IRC | 17:29 | |
| portante | otoolee: you are welcome, and I will take a look in a bit | 17:30 |
| *** shri has quit IRC | 17:31 | |
| *** openstackgerrit has quit IRC | 17:32 | |
| *** openstackgerrit has joined #openstack-swift | 17:32 | |
| *** gvernik has quit IRC | 17:33 | |
| *** mmcardle has quit IRC | 17:35 | |
| *** kevinc_ has quit IRC | 17:38 | |
| *** kevinc_ has joined #openstack-swift | 17:38 | |
| *** gvernik has joined #openstack-swift | 17:39 | |
| *** joeljwright has joined #openstack-swift | 17:44 | |
| *** lpabon has joined #openstack-swift | 17:47 | |
| *** chandan_kumar has quit IRC | 17:49 | |
| *** joeljwright has quit IRC | 17:49 | |
| luisbg | zaitcev, you reviewed the previous iteration of this positively: https://review.openstack.org/#/c/90103/ | 18:00 |
| luisbg | have a minute to see the new patch (based on a suggestion from Christian) | 18:00 |
| zaitcev | luisbg: I might | 18:01 |
| *** shri has joined #openstack-swift | 18:01 | |
| luisbg | zaitcev, thank you :) | 18:01 |
| *** gvernik has quit IRC | 18:02 | |
| * notmyname is figuring out the new gerrit stuff | 18:17 | |
| clayg | peluse_: notmyname: reconciler is done, i'm cleaning up some tests/pep8 in the patch chain them i'm going to be doing reviews | 18:17 |
| notmyname | woot | 18:18 |
| clayg | peluse_: notmyname: I'm going to try and get through all of the pending ec changes and review some stuff on master too - clear out the queue ahead of the sp-rebase chain | 18:18 |
| *** gvernik has joined #openstack-swift | 18:21 | |
| openstackgerrit | Clay Gerrard proposed a change to openstack/swift: Keep status_changed_at up-to-date with status changes. https://review.openstack.org/86714 | 18:27 |
| openstackgerrit | Clay Gerrard proposed a change to openstack/swift: Extend direct_client https://review.openstack.org/88144 | 18:27 |
| openstackgerrit | Clay Gerrard proposed a change to openstack/swift: Add reconciler probetest outline https://review.openstack.org/90934 | 18:27 |
| openstackgerrit | Clay Gerrard proposed a change to openstack/swift: Add Storage Policies to container.backend https://review.openstack.org/90935 | 18:27 |
| openstackgerrit | Clay Gerrard proposed a change to openstack/swift: Update FakeLogger with better log level handling https://review.openstack.org/90932 | 18:27 |
| openstackgerrit | Clay Gerrard proposed a change to openstack/swift: Add container-reconciler daemon https://review.openstack.org/85908 | 18:27 |
| openstackgerrit | Clay Gerrard proposed a change to openstack/swift: Make FakeRing a use more real code https://review.openstack.org/90933 | 18:27 |
| openstackgerrit | Clay Gerrard proposed a change to openstack/swift: Add LRUCache to common.utils https://review.openstack.org/90938 | 18:27 |
| openstackgerrit | Clay Gerrard proposed a change to openstack/swift: Add Storage Policy to container put/delete object https://review.openstack.org/90936 | 18:27 |
| openstackgerrit | Clay Gerrard proposed a change to openstack/swift: Enqueue misplaced objects during container replication https://review.openstack.org/90937 | 18:27 |
| openstackgerrit | Clay Gerrard proposed a change to openstack/swift: Merge container storage_policy_index values. https://review.openstack.org/86720 | 18:27 |
| dfg | portante: ya i'll check out that patch | 18:30 |
| portante | dfg: thanks | 18:36 |
| luisbg | notmyname, some of the new gerrit stuff is cool | 18:39 |
| luisbg | I like it | 18:39 |
| luisbg | clayg, whoa. busy? :P | 18:40 |
| cschwede | cschwede | 18:41 |
| clayg | it was ust a rebase | 18:41 |
| luisbg | clayg, yeah :) | 18:43 |
| physcx | Question about submitting to gerrit (doing my first one): I have a change to DiskfileWriter's write method and a rather lengthy write-up explaining the reason for it as well as some performance metrics of before and after. The write-up is roughly 250 lines at 80 col width -- is that too large to use within the commit description? Is there a preferred way to attach the lengthy description to... | 18:44 |
| physcx | ...the patch? | 18:44 |
| notmyname | physcx: as long as you have a summary at the top of the commit message, I don't mind long commit messages. but if it's info that's good for ongoing cluster operations, then it could go in the docs | 18:45 |
| *** joeljwright has joined #openstack-swift | 18:45 | |
| clayg | well... there's bound to be *some* limit - and 250 lines has got to be close? | 18:46 |
| portante | physcx: interested in that patch, can you add me? | 18:46 |
| portante | You can always post it work-in-progress as well | 18:46 |
| clayg | hell dfg's cors commit was only 110 | 18:47 |
| notmyname | heh, that's the one I was thinking of :-) | 18:47 |
| physcx | portante: do I add you after i push with git review? | 18:48 |
| notmyname | physcx: alternatives are to provide a good summary in the commit message and then give details in the review comments | 18:48 |
| clayg | physcx: you might see if you can cut it down to ~100 lines and post additional information in the comments (just review +0 your change with some notes or a link to a gist/paste or something) | 18:49 |
| dfg | i made the commit msg that long so people would just get bored and hit Approved without looking at the code. | 18:49 |
| physcx | alright I'll go that route, ty | 18:49 |
| portante | physcx: that will work | 18:49 |
| *** joeljwright has quit IRC | 18:50 | |
| notmyname | dfg: you shouldn't have made the message so entertaining then "...since the purpose of CORS is to annoy people..." | 18:51 |
| dfg | :) | 18:51 |
| creiht | lol | 18:55 |
| *** bsdkurt has joined #openstack-swift | 18:56 | |
| openstackgerrit | Shaun McDowell proposed a change to openstack/swift: Buffer DiskfileWriter writes https://review.openstack.org/91133 | 18:59 |
| *** jamie_h has quit IRC | 18:59 | |
| cschwede | Look at that patch - now this is quite cool for new contributors ^^ | 19:00 |
| physcx | well thats no good... the comment section had no respect for spacing what-so-ever | 19:04 |
| creiht | heh | 19:04 |
| creiht | physcx: yeah it is annoying | 19:04 |
| physcx | is there even a way to delete it? | 19:04 |
| creiht | physcx: it uses some sort of markup but doesn't really indicate that to you | 19:04 |
| portante | physcx: that is why I put that stuff in the commit message, since it pertains to the nature of the commit | 19:04 |
| zaitcev | Is anyone here familiar with our EC? I have a simple question: do we have a segment size for reconstruction and if yes, how big is it? | 19:04 |
| zaitcev | Not sure if I use our terminology right but basically what we call "chunk" works like a very long codeword. | 19:05 |
| peluse_ | zaitcev: so we have lots of terms... | 19:06 |
| peluse_ | zaitcev: I'll send you an email as its too much to type in here | 19:07 |
| zaitcev | So I'm wondering, is it possible to stream inside the chunk (in which case chunk can be as long as we need, but its length has to be recorded in metadata somewhere). | 19:07 |
| zaitcev | peluse_: thanks a lot, although I'm going to have a meeting with Intel in 53 minutes where I should know it | 19:07 |
| peluse_ | zaitcev: so things still stream through the proxy and we buffer up what we call a 'segment'. then we EC that segment and send those 'frgaments' out to the nodes. The default segment size is 1MB | 19:08 |
| zaitcev | peluse_: makes sense, thanks. What's a chunk, then? | 19:08 |
| *** gvernik has quit IRC | 19:09 | |
| peluse_ | zaitcev: we don't use chunk on purpose to avoid confusing with http chukning. You'll hear it used in other EC implementations though and the equivalent for us is fragment (the output of the encoder) | 19:10 |
| zaitcev | peluse_: got it | 19:10 |
| peluse_ | zaitcev: there are some slides in your inbox, feel free to hit us up w/questions or if you need something real time for your upcoming call ping me and we can chat on the phone if needed | 19:11 |
| clayg | physcx: so what was the speed the difference of using the 64k disk write vs 2-5MB write? It's not clear to me why the larger is faster - it's buffered fileio - we only sync every 512MB by default? | 19:11 |
| zaitcev | peluse_: do you happen to know the Subject: by any chance | 19:12 |
| peluse_ | clayg: thanks for taking a look at the recon thing - I'll go check out the cli that you mention since right now I don't understand your comment :) | 19:12 |
| peluse_ | zaitcev:RE: Sync up on Swift | 19:12 |
| clayg | well... it's just that in cli.recon it does this thing where it set's like a "ring_file" var - and then it keys into the response from that | 19:13 |
| peluse_ | zaitcev: wait.. sorry... "RE: Swift EC tree" | 19:13 |
| clayg | peluse_: it needs to like look at all the rings in the response and then decide if it shold check them or not or something? | 19:13 |
| zaitcev | peluse_: ah, I got that. It's 2 images that Tushar sent me. Thanks. | 19:13 |
| zaitcev | peluse_: admittedly one of them has Segment_Size mentioned. | 19:14 |
| peluse_ | zaitcev: yup | 19:14 |
| peluse_ | zaitcev: and as you can see we store a bunch of info in swift visible meta and there's also opaque meta embedded in the segments for use by the EC library itself to aid in all sorta of operations | 19:14 |
| zaitcev | ttyl | 19:15 |
| peluse_ | clayg: hell, I didn't even know that was there :) | 19:15 |
| portante | physcx: I like the concept, how many concurrent writes to multiple objects were performed? | 19:16 |
| peluse_ | clayg: as you saw, I updated the middleware to look at all of the ring files and then added what was previously missing test code for that capabiilties (ring md5 check) which was just a barrel of monkeys with the different gzip header styles between py 26 and 27 | 19:18 |
| physcx | clayg: we use a modified version of diskfile that uses DirectIO. We skip the buffered fileio and write 4k aligned straight to disk and under heavy load were getting around a 5x speed up over the os buffered fileio. | 19:18 |
| clayg | oic | 19:19 |
| clayg | well then you better buffer in userland hadn't you! | 19:19 |
| portante | physcx: is that diskfile open source? | 19:19 |
| physcx | yes but we also felt that it was nice to have disk_chunk_size affect the disk read/writes and the network_chunk_size affect the network | 19:20 |
| physcx | portante: yep its submitted to gerrit somewhere i'll dig up a link - we'll probably turn it into it's own diskfile similar to the way mem_diskfile was done but for direct-io | 19:20 |
| portante | physcx: cool | 19:21 |
| creiht | notmyname: the other thing that I miss now that we have pbr, is being able to just look at the code and know what version it is | 19:21 |
| creiht | notmyname: is there an easy way to know what the current version of swift is? | 19:21 |
| portante | creiht: its the latest verision ,right? | 19:21 |
| portante | ;) | 19:21 |
| creiht | heh | 19:21 |
| physcx | portante: are you asking about concurrent writes to multiple objects in our SAIO testing? | 19:22 |
| portante | yes | 19:22 |
| physcx | just the default swift-bench --saio settings which i believe is 10 concurrent | 19:22 |
| *** gholt has quit IRC | 19:22 | |
| portante | so in the past I have measured effects of eventlet concurrency on these operations | 19:23 |
| portante | one thing I found, which seems to hold still today, is that by lowering the number of concurrent connections per worker and raising the number of worker processes, response times start to smooth out | 19:24 |
| portante | max_clients & workers parameters | 19:24 |
| physcx | well we saw the problem with a single connection even | 19:24 |
| portante | was that with or without the direct io? | 19:25 |
| physcx | without | 19:25 |
| physcx | its in that mangled comment | 19:25 |
| portante | ;) | 19:25 |
| portante | yes, that was kinda hard to follow entirely | 19:25 |
| portante | :) | 19:25 |
| portante | with the single connection that was with chunked transfer? | 19:25 |
| physcx | want me to email you the unmangled version of that? | 19:25 |
| portante | yes, please | 19:26 |
| portante | thanks | 19:26 |
| zaitcev | creiht: does this work PYTHONPATH=$(pwd) python -c "import swift; print swift.__version__" | 19:29 |
| zaitcev | peluse_: Do you remember how the mode of storage is called where you do not have the original object stored in the clear? Kevin mentioned it on Trello, but... Like synchronous vs. asynchronous, but different. | 19:32 |
| creiht | zaitcev: well yeah I can do something like that, but I'm not certain it is showing the right version | 19:32 |
| peluse_ | zaitcev: symmetrical I think... one sec I'll double check | 19:33 |
| zaitcev | creiht: why not if path is correct, my shows 1.13.1.rc1.346.g8dd0aab or 1.12.0-1.z6.el6 in my tests, looks correct | 19:34 |
| creiht | I guess the rc1 thing is throwing me off | 19:34 |
| creiht | because that doesn't seem right | 19:35 |
| creiht | notmyname: what version of swift was the icehouse relese? | 19:35 |
| peluse_ | zaitcev: sorry, systematic is in the clear (versus non-systematic) | 19:36 |
| zaitcev | peluse_: I'm sorry I sent you away, it's called "systematic" and "non-systematic". He mentioned it in EC Auditor card. | 19:36 |
| notmyname | creiht: 1.13.1 | 19:36 |
| zaitcev | peluse_: got that | 19:36 |
| notmyname | creiht: checking on something | 19:36 |
| creiht | notmyname: ok | 19:36 |
| creiht | thx | 19:36 |
| notmyname | bah!!! | 19:36 |
| clayg | fucking pbr | 19:37 |
| notmyname | no, not that | 19:37 |
| clayg | it's sorta fucked up | 19:37 |
| creiht | lol | 19:37 |
| peluse_ | zaitcev: if you're looking for a good general EC overview: http://web.eecs.utk.edu/~plank/plank/papers/2013-02-11-FAST-Tutorial.pdf | 19:37 |
| creiht | I was trying hard to not badmouth pbr this time :) | 19:37 |
| notmyname | the -infra job that is supposed to automatically merge the milestone-proposed branch into master (ie so the tags work ie so the versions work) never got ran | 19:37 |
| clayg | like you get the version of the last thing you installed - unless it's not installed - then just changing the checkout (if you have pbr installed) can sorta figure it out for tags and shas | 19:37 |
| * notmyname goes to talk in -infra | 19:37 | |
| peluse_ | zaitcev: Kevin's done lots of work with him... (Plank, authro of the slides I just sent a link to) | 19:38 |
| clayg | and why *is* it rc1? I have an rc2 tag? of all complicated things we have to work on - why does version numbers have to be one of them? | 19:38 |
| notmyname | clayg: because the rc1 is an ancestor of the tip of master. sans rc and rc2 both are on the milestone-proposed branch, which is not an ancestor of master | 19:39 |
| *** gholt has joined #openstack-swift | 19:46 | |
| *** ChanServ sets mode: +v gholt | 19:46 | |
| *** joeljwright has joined #openstack-swift | 19:47 | |
| *** joeljwright has quit IRC | 19:51 | |
| *** krtaylor has joined #openstack-swift | 19:58 | |
| openstackgerrit | Shaun McDowell proposed a change to openstack/swift: Buffer DiskfileWriter writes https://review.openstack.org/91133 | 20:07 |
| *** fifieldt has quit IRC | 20:08 | |
| *** fifieldt has joined #openstack-swift | 20:09 | |
| shri | In devstack, if I specify the branch for swift as stable/icehouse, shouldn't it install swift version 1.13.1? | 20:09 |
| shri | I did that but inspite of it, I see the version in /info as 1.11.0 | 20:10 |
| *** bach has joined #openstack-swift | 20:12 | |
| shri | no one uses devstack here? | 20:13 |
| *** bach has quit IRC | 20:15 | |
| glange | most of us do swift on "hard mode" :) | 20:16 |
| notmyname | shri: ya it should. I'm not sure how it picks the version | 20:17 |
| clayg | shri: i've run swift on devstack, but I haven't tried the branch specifier? | 20:17 |
| notmyname | ditto | 20:17 |
| shri | I see.. let me try without specifying a branch. | 20:17 |
| physcx | Is there a specific swift wiki for posting long formatted write-ups for submitted patches to gerrit? | 20:20 |
| physcx | nvmd got it | 20:21 |
| glange | physcx: https://blueprints.launchpad.net/swift <-- maybe ? | 20:22 |
| notmyname | https://wiki.openstack.org/wiki/Swift | 20:23 |
| notmyname | and you can proabably add pages under that | 20:23 |
| glange | http://paste.openstack.org/ <-- maybe ? | 20:23 |
| notmyname | http://paste.ronin.io | 20:24 |
| notmyname | 'cause it uses swift itself | 20:24 |
| * notmyname is trying to build pandemicsyn's user base | 20:25 | |
| glange | I think he just hopes somebody pastes some valuable secrets | 20:29 |
| *** tsg has joined #openstack-swift | 20:44 | |
| luisbg | is it possible Jenkins missed a +2 core review? or is it blocking pushes to Zuul for some other reason? | 20:46 |
| *** joeljwright has joined #openstack-swift | 20:48 | |
| *** tdasilva has quit IRC | 20:51 | |
| *** joeljwright has quit IRC | 20:52 | |
| *** tdasilva has joined #openstack-swift | 20:53 | |
| notmyname | luisbg: what are you seeing? | 21:16 |
| luisbg | notmyname, https://review.openstack.org/#/c/90103/ | 21:16 |
| luisbg | has a +2 from Pete, but it wasn't pushed to Zuul or isn't listed in patches waiting for one core review | 21:17 |
| luisbg | so in a bit of limbo | 21:17 |
| notmyname | luisbg: probably a side effect of the gerrit upgrade (for it not being listed) | 21:18 |
| luisbg | notmyname, it fell through the cracks then hehehee | 21:18 |
| *** krtaylor has quit IRC | 21:18 | |
| *** joeljwright has joined #openstack-swift | 21:18 | |
| notmyname | ah, I bet it's because of the "Verified" "CodeReview" and "Approved" filters | 21:19 |
| notmyname | probably one of those is different now | 21:19 |
| notmyname | probably Approved vs the new workflow thing | 21:19 |
| luisbg | notmyname, yes, I thought of that but I don't know what the differences are now | 21:19 |
| notmyname | me neither | 21:20 |
| luisbg | what is Workflow? and probably the list filters in https://wiki.openstack.org/wiki/Swift/PriorityReviews need to be updated | 21:20 |
| notmyname | luisbg: ya, I updated them so that they don't error out, but obviously didn't get all the changes | 21:22 |
| luisbg | notmyname, if you find some resource where the new changes are explained I can help you update the filters | 21:25 |
| *** bach has joined #openstack-swift | 21:27 | |
| openstackgerrit | Clay Gerrard proposed a change to openstack/swift: Fixes probe tests with non-zero default storage policy https://review.openstack.org/87492 | 21:32 |
| *** bach has quit IRC | 21:35 | |
| notmyname | luisbg: that would be great! all the "docs" I've seen are https://wiki.openstack.org/wiki/GerritUpgrade. you may need to look at upstream gerrit to find the right incantation | 21:41 |
| openstackgerrit | A change was merged to openstack/python-swiftclient: Mock auth_end_time in test_shell.test_download https://review.openstack.org/90026 | 21:45 |
| Alex_Gaynor | woo, one more patch until all swiftclient tests pass on py3k! | 21:49 |
| notmyname | luisbg: looks like maybe the -infra team is making some dashboards? https://etherpad.openstack.org/p/gerrit-dashboards | 21:58 |
| *** dosaboy has quit IRC | 22:03 | |
| *** dosaboy has joined #openstack-swift | 22:04 | |
| *** rcleere has quit IRC | 22:13 | |
| *** saschpe has quit IRC | 22:23 | |
| *** joeljwright has quit IRC | 22:24 | |
| *** saschpe has joined #openstack-swift | 22:26 | |
| *** bach has joined #openstack-swift | 22:28 | |
| *** zigo_ has joined #openstack-swift | 22:35 | |
| *** zigo has quit IRC | 22:37 | |
| *** wer has quit IRC | 22:37 | |
| *** wer has joined #openstack-swift | 22:37 | |
| *** changbl has quit IRC | 22:37 | |
| *** bach has quit IRC | 22:40 | |
| *** kevinc_ has quit IRC | 22:50 | |
| *** bach has joined #openstack-swift | 23:08 | |
| *** Trixboxer has quit IRC | 23:08 | |
| *** bach has quit IRC | 23:14 | |
| *** shakamunyi has quit IRC | 23:14 | |
| *** joeljwright has joined #openstack-swift | 23:15 | |
| *** shakamunyi has joined #openstack-swift | 23:18 | |
| *** joeljwright has quit IRC | 23:19 | |
| *** joeljwright has joined #openstack-swift | 23:24 | |
| *** joeljwright has quit IRC | 23:37 | |
| *** matsuhashi has joined #openstack-swift | 23:38 | |
| openstackgerrit | Shaun McDowell proposed a change to openstack/swift: Buffer DiskfileWriter writes https://review.openstack.org/91133 | 23:43 |
| openstackgerrit | Clay Gerrard proposed a change to openstack/swift: Add Storage Policy support to the Account Reaper https://review.openstack.org/88098 | 23:43 |
| physcx | I've added a much less bad version of the write-up on buffered writes and the problem with chunked transfers for large read sizes at https://wiki.openstack.org/wiki/User:Physcx | 23:47 |
| notmyname | physcx: great writeup. thanks | 23:50 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!