| *** otoolee has quit IRC | 00:29 | |
| *** dmorita has joined #openstack-swift | 00:30 | |
| classicsnail | does anyone know of a swift setup using facebook's mcrouter as the memcache server? | 00:32 |
|---|---|---|
| classicsnail | seems to be a decent idea on the face of it | 00:32 |
| *** kyles_ne has quit IRC | 00:36 | |
| *** dmsimard is now known as dmsimard_away | 00:43 | |
| *** kopparam has joined #openstack-swift | 00:47 | |
| *** kopparam has quit IRC | 00:51 | |
| *** exploreshaifali has quit IRC | 00:53 | |
| *** X019 has quit IRC | 01:06 | |
| *** otoolee has joined #openstack-swift | 01:11 | |
| *** kyles_ne has joined #openstack-swift | 01:12 | |
| *** jasondotstar has quit IRC | 01:17 | |
| *** X019 has joined #openstack-swift | 01:19 | |
| *** addnull has joined #openstack-swift | 01:24 | |
| *** X019 has quit IRC | 01:30 | |
| *** nosnos has joined #openstack-swift | 01:32 | |
| *** haomaiw__ has quit IRC | 01:47 | |
| *** kopparam has joined #openstack-swift | 01:47 | |
| *** haomaiwang has joined #openstack-swift | 01:48 | |
| *** kopparam has quit IRC | 01:52 | |
| *** kyles_ne has quit IRC | 01:52 | |
| *** kyles_ne has joined #openstack-swift | 01:53 | |
| *** kyles_ne has quit IRC | 01:57 | |
| *** haomaiw__ has joined #openstack-swift | 02:04 | |
| *** haomaiwang has quit IRC | 02:04 | |
| *** kyles_ne has joined #openstack-swift | 02:10 | |
| *** haomaiwa_ has joined #openstack-swift | 02:16 | |
| *** haomaiw__ has quit IRC | 02:19 | |
| *** kyles_ne has quit IRC | 02:27 | |
| *** kyles_ne has joined #openstack-swift | 02:28 | |
| *** kyles_ne has quit IRC | 02:30 | |
| *** kyles_ne has joined #openstack-swift | 02:30 | |
| *** madhuri has quit IRC | 02:37 | |
| *** madhuri_ has quit IRC | 02:38 | |
| *** kopparam has joined #openstack-swift | 02:48 | |
| *** kopparam has quit IRC | 02:53 | |
| *** kyles_ne has quit IRC | 03:06 | |
| *** kyles_ne has joined #openstack-swift | 03:06 | |
| *** kyles_ne has quit IRC | 03:10 | |
| *** dave-mccowan has quit IRC | 03:15 | |
| *** diablos has joined #openstack-swift | 03:15 | |
| *** mahatic has quit IRC | 03:16 | |
| *** mrsnivvel has quit IRC | 03:28 | |
| *** acoles has quit IRC | 03:32 | |
| *** acoles_away has joined #openstack-swift | 03:37 | |
| *** acoles_away is now known as acoles | 03:37 | |
| *** ChanServ sets mode: +v acoles | 03:37 | |
| *** mrsnivvel has joined #openstack-swift | 03:46 | |
| *** kopparam has joined #openstack-swift | 03:49 | |
| *** kopparam has quit IRC | 03:54 | |
| *** diablos has quit IRC | 03:54 | |
| *** addnull has quit IRC | 04:01 | |
| *** adu has quit IRC | 04:04 | |
| *** hhuang has joined #openstack-swift | 04:12 | |
| *** adu has joined #openstack-swift | 04:16 | |
| *** SkyRocknRoll has joined #openstack-swift | 04:19 | |
| *** SkyRocknRoll has joined #openstack-swift | 04:19 | |
| *** addnull has joined #openstack-swift | 04:33 | |
| *** addnull has quit IRC | 04:45 | |
| *** addnull has joined #openstack-swift | 04:46 | |
| *** kyles_ne has joined #openstack-swift | 04:48 | |
| *** kopparam has joined #openstack-swift | 04:50 | |
| *** addnull has quit IRC | 04:50 | |
| *** ppai has joined #openstack-swift | 04:52 | |
| *** kopparam has quit IRC | 04:54 | |
| *** dmsimard_away is now known as dmsimard | 04:56 | |
| *** acoles has quit IRC | 05:00 | |
| *** addnull has joined #openstack-swift | 05:03 | |
| *** zaitcev has quit IRC | 05:08 | |
| *** acoles_away has joined #openstack-swift | 05:12 | |
| *** acoles_away is now known as acoles | 05:14 | |
| *** ChanServ sets mode: +v acoles | 05:14 | |
| *** otoolee has quit IRC | 05:15 | |
| *** otoolee has joined #openstack-swift | 05:27 | |
| *** kopparam has joined #openstack-swift | 05:36 | |
| mattoliverau | Good night swift world, have a great night and weekend. | 05:38 |
| *** gyee has quit IRC | 05:42 | |
| *** hhuang has quit IRC | 05:49 | |
| *** k4n0 has joined #openstack-swift | 06:09 | |
| *** kyles_ne has quit IRC | 06:14 | |
| *** kopparam has quit IRC | 06:30 | |
| *** addnull has quit IRC | 06:32 | |
| *** addnull has joined #openstack-swift | 06:33 | |
| *** adu has quit IRC | 06:34 | |
| *** ttrumm has joined #openstack-swift | 06:34 | |
| *** echevemaster has quit IRC | 06:36 | |
| *** addnull has quit IRC | 06:37 | |
| *** addnull has joined #openstack-swift | 06:40 | |
| *** echevemaster has joined #openstack-swift | 06:41 | |
| *** echevemaster has quit IRC | 06:43 | |
| *** acoles has quit IRC | 06:44 | |
| *** addnull has quit IRC | 06:49 | |
| *** acoles_away has joined #openstack-swift | 06:56 | |
| *** acoles_away is now known as acoles | 06:57 | |
| *** ChanServ sets mode: +v acoles | 06:57 | |
| *** addnull has joined #openstack-swift | 07:14 | |
| *** dmsimard is now known as dmsimard_away | 07:15 | |
| *** jamiehannaford has joined #openstack-swift | 07:21 | |
| *** otoolee has quit IRC | 07:28 | |
| *** nshaikh has joined #openstack-swift | 07:29 | |
| *** tushar has quit IRC | 07:35 | |
| *** otoolee has joined #openstack-swift | 07:42 | |
| *** jordanP has joined #openstack-swift | 07:51 | |
| *** addnull has quit IRC | 07:55 | |
| *** jamiehan_ has joined #openstack-swift | 07:58 | |
| *** addnull has joined #openstack-swift | 08:00 | |
| *** jamiehannaford has quit IRC | 08:00 | |
| *** jamiehan_ has quit IRC | 08:12 | |
| *** jamiehannaford has joined #openstack-swift | 08:16 | |
| *** otoolee has quit IRC | 08:19 | |
| *** aix has joined #openstack-swift | 08:19 | |
| *** jistr has joined #openstack-swift | 08:24 | |
| *** otoolee has joined #openstack-swift | 08:33 | |
| *** CybergeekDK has quit IRC | 08:35 | |
| *** sileht has quit IRC | 08:41 | |
| *** CybergeekDK has joined #openstack-swift | 08:42 | |
| *** sileht has joined #openstack-swift | 08:42 | |
| *** CybergeekDK has quit IRC | 08:44 | |
| *** CybergeekDK has joined #openstack-swift | 08:48 | |
| *** addnull has quit IRC | 08:50 | |
| *** nellysmitt has joined #openstack-swift | 08:57 | |
| *** addnull has joined #openstack-swift | 09:10 | |
| *** ttrumm has quit IRC | 09:11 | |
| *** astellwag has joined #openstack-swift | 09:14 | |
| *** SkyRocknRoll has quit IRC | 09:32 | |
| *** mkollaro has joined #openstack-swift | 09:33 | |
| *** addnull has quit IRC | 09:38 | |
| *** addnull has joined #openstack-swift | 09:40 | |
| *** addnull has left #openstack-swift | 09:41 | |
| *** addnull has joined #openstack-swift | 09:41 | |
| *** addnull has quit IRC | 09:45 | |
| *** addnull has joined #openstack-swift | 09:45 | |
| *** haomaiwa_ has quit IRC | 09:59 | |
| *** haomaiwa_ has joined #openstack-swift | 10:00 | |
| *** CybergeekDK has quit IRC | 10:00 | |
| *** astellwag has quit IRC | 10:11 | |
| *** astellwag has joined #openstack-swift | 10:15 | |
| *** haomai___ has joined #openstack-swift | 10:16 | |
| *** haomaiwa_ has quit IRC | 10:16 | |
| *** haomai___ has quit IRC | 10:21 | |
| *** haomaiwang has joined #openstack-swift | 10:21 | |
| *** CybergeekDK has joined #openstack-swift | 10:26 | |
| *** mkollaro has quit IRC | 10:31 | |
| *** haomaiwang has quit IRC | 10:39 | |
| *** jamiehan_ has joined #openstack-swift | 10:45 | |
| *** jamiehannaford has quit IRC | 10:48 | |
| *** jamiehan_ has quit IRC | 10:50 | |
| *** jamiehannaford has joined #openstack-swift | 10:53 | |
| *** jamiehan_ has joined #openstack-swift | 10:55 | |
| *** addnull has quit IRC | 10:58 | |
| *** jamiehannaford has quit IRC | 10:58 | |
| *** ppai has quit IRC | 11:03 | |
| *** addnull has joined #openstack-swift | 11:04 | |
| *** jamiehan_ has quit IRC | 11:04 | |
| *** dave-mccowan has joined #openstack-swift | 11:10 | |
| *** X019 has joined #openstack-swift | 11:10 | |
| *** jamiehannaford has joined #openstack-swift | 11:10 | |
| *** nellysmitt has quit IRC | 11:13 | |
| *** dave-mccowan has quit IRC | 11:14 | |
| *** nellysmitt has joined #openstack-swift | 11:14 | |
| *** dave-mccowan has joined #openstack-swift | 11:15 | |
| *** ppai has joined #openstack-swift | 11:17 | |
| *** nellysmitt has quit IRC | 11:19 | |
| *** exploreshaifali has joined #openstack-swift | 11:21 | |
| *** CybergeekDK has quit IRC | 11:29 | |
| *** CybergeekDK has joined #openstack-swift | 11:31 | |
| *** ppai has quit IRC | 11:35 | |
| *** exploreshaifali has quit IRC | 11:44 | |
| *** ppai has joined #openstack-swift | 11:49 | |
| *** jordan__ has joined #openstack-swift | 11:55 | |
| *** ppai has quit IRC | 11:56 | |
| *** jordan__ has quit IRC | 11:56 | |
| *** mkollaro has joined #openstack-swift | 12:04 | |
| *** ppai has joined #openstack-swift | 12:10 | |
| *** dmorita has quit IRC | 12:25 | |
| *** davidhadas has quit IRC | 12:30 | |
| *** g4rg4m3|_ has joined #openstack-swift | 12:31 | |
| *** nosnos has quit IRC | 12:41 | |
| *** nosnos has joined #openstack-swift | 12:42 | |
| *** nosnos has quit IRC | 12:46 | |
| *** k4n0 has quit IRC | 12:55 | |
| *** mrsnivvel has quit IRC | 13:09 | |
| *** haomaiwa_ has joined #openstack-swift | 13:10 | |
| *** openstackgerrit has quit IRC | 13:19 | |
| *** openstackgerrit has joined #openstack-swift | 13:20 | |
| *** exploreshaifali has joined #openstack-swift | 13:21 | |
| *** ppai has quit IRC | 13:23 | |
| *** nshaikh has quit IRC | 13:27 | |
| *** CaioBrentano has joined #openstack-swift | 13:39 | |
| *** addnull has quit IRC | 13:45 | |
| openstackgerrit | Daniel Wakefield proposed a change to openstack/python-swiftclient: Add MD5 sum that can calculate partial files https://review.openstack.org/129254 | 13:45 |
| *** adu has joined #openstack-swift | 13:46 | |
| *** dave-mccowan has quit IRC | 13:47 | |
| *** davidhadas_ has joined #openstack-swift | 13:51 | |
| *** adu has quit IRC | 13:53 | |
| *** adu has joined #openstack-swift | 13:54 | |
| *** mahatic has joined #openstack-swift | 14:04 | |
| *** infotection has quit IRC | 14:04 | |
| *** dave-mccowan has joined #openstack-swift | 14:07 | |
| *** adu has quit IRC | 14:07 | |
| *** infotection has joined #openstack-swift | 14:14 | |
| *** hemanthm has joined #openstack-swift | 14:17 | |
| *** infotection has quit IRC | 14:20 | |
| *** infotection has joined #openstack-swift | 14:22 | |
| *** otoolee has quit IRC | 14:30 | |
| *** morganfainberg has quit IRC | 14:32 | |
| *** silor has joined #openstack-swift | 14:32 | |
| *** nellysmitt has joined #openstack-swift | 14:33 | |
| *** infotection has quit IRC | 14:33 | |
| *** morganfainberg has joined #openstack-swift | 14:33 | |
| *** infotection has joined #openstack-swift | 14:36 | |
| *** mkollaro has quit IRC | 14:38 | |
| *** mkollaro has joined #openstack-swift | 14:40 | |
| *** infotection has quit IRC | 14:40 | |
| *** infotection has joined #openstack-swift | 14:41 | |
| *** mahatic has quit IRC | 14:41 | |
| *** infotection has quit IRC | 14:49 | |
| *** infotection has joined #openstack-swift | 14:50 | |
| *** annegent_ has joined #openstack-swift | 14:50 | |
| *** otoolee has joined #openstack-swift | 14:50 | |
| *** infotection has quit IRC | 14:57 | |
| *** CaioBrentano has quit IRC | 14:58 | |
| *** mahatic has joined #openstack-swift | 14:59 | |
| *** infotection has joined #openstack-swift | 15:01 | |
| *** erlon_ has joined #openstack-swift | 15:03 | |
| *** jamiehan_ has joined #openstack-swift | 15:05 | |
| *** jergerber has joined #openstack-swift | 15:07 | |
| *** jamiehannaford has quit IRC | 15:08 | |
| *** infotection has quit IRC | 15:14 | |
| *** infotection has joined #openstack-swift | 15:15 | |
| *** infotection has quit IRC | 15:19 | |
| *** bill_az has joined #openstack-swift | 15:20 | |
| *** lpabon has joined #openstack-swift | 15:22 | |
| *** infotection has joined #openstack-swift | 15:25 | |
| openstackgerrit | A change was merged to openstack/swift: use replication_ip in ssync https://review.openstack.org/128808 | 15:29 |
| *** otoolee has quit IRC | 15:29 | |
| notmyname | good morning | 15:35 |
| *** infotection has quit IRC | 15:37 | |
| peluse | good morning | 15:37 |
| *** infotection has joined #openstack-swift | 15:37 | |
| *** otoolee has joined #openstack-swift | 15:37 | |
| *** mkollaro has quit IRC | 15:37 | |
| notmyname | peluse: (clayg) I think I'm missing something on the proposed GET path with .durable files | 15:42 |
| peluse | wanna chat on the phone for a few secs? | 15:42 |
| notmyname | sure | 15:43 |
| peluse | I'm at 480 554 3688 | 15:43 |
| notmyname | FWIW, virtualbox with my SAIO works on yosemite | 15:43 |
| notmyname | calling | 15:43 |
| *** exploreshaifali has quit IRC | 15:45 | |
| notmyname | that was easy :-) | 15:50 |
| notmyname | clayg: ok so I had a mistaken assumption, but I'm good now | 15:51 |
| notmyname | 1) there is always a .data and .durable pair. the .durable file is what tracks the EC object | 15:51 |
| notmyname | 2) on reads, the proxy doesn't hear about anything fromt he object server (body or headers) for any .data file that doesn't have an associated .durable | 15:51 |
| notmyname | 3) on write, the object can remove older (.data, .durable) pairs after the current timestamp.durable is written | 15:52 |
| notmyname | peluse: right? | 15:52 |
| *** infotection has quit IRC | 15:54 | |
| *** infotection has joined #openstack-swift | 15:55 | |
| hemanthm | Hey, Swift folks! In Openstack Glance world we need some Swift expertise on a spec that is proposing to use multiple containers to store images | 15:57 |
| hemanthm | Glance uses a single container to store all its images at the moment | 15:58 |
| hemanthm | Here is the spec, https://review.openstack.org/#/c/124522/ | 15:58 |
| notmyname | hemanthm: glance is storing info on every image in its own DB, right? so if glance is storing a billion images, there are a billion rows in a glance DB somewhere | 16:01 |
| hemanthm | hemanthm: yes | 16:02 |
| hemanthm | notmyname: yes | 16:02 |
| notmyname | hemanthm: other than the location of the image, how much data is stored about the image in each row? | 16:03 |
| notmyname | hemanthm: well, let me get to my end question... | 16:03 |
| notmyname | hemanthm: do you expect that glance will continue with this model or that glance will offload that bookkeeping problem to the storage backend (like swift)? alternatively, will glance ever move to some DHT/hash ring sort of placement of images? | 16:04 |
| notmyname | is anyone talking about that? is it realistic that glance will do that in the next year? | 16:05 |
| hemanthm | notmyname: there is no talk about that at this point | 16:05 |
| notmyname | hemanthm: then the spec looks fine from a swift perspective :-) | 16:06 |
| notmyname | I'll add a note in gerrit | 16:06 |
| notmyname | hemanthm: but first.. | 16:06 |
| *** infotection has quit IRC | 16:06 | |
| notmyname | hemanthm: jsut to confirm that I understand what's happening. | 16:06 |
| hemanthm | sure | 16:06 |
| notmyname | hemanthm: you're storing images by UUID, and you're proposing that instead of putting them all in one container you put them in different containers based on the first N bytes of the UUID | 16:07 |
| notmyname | that's pretty much all I see as the swift-impacting parts of that spec | 16:07 |
| notmyname | and that's a normal pattern for swift usage. and recommended | 16:07 |
| hemanthm | notmyname: yes, it's actually just following that recommendation | 16:08 |
| notmyname | hemanthm: did I miss or misunderstand anything? | 16:08 |
| *** davidhadas_ has quit IRC | 16:09 | |
| hemanthm | notmyname: there is a suggestion to use Swift 'account_whitelist' to whitelist glance so that we don't hit the rate-limits, if any | 16:09 |
| notmyname | no, I wouldn't do that | 16:09 |
| hemanthm | notmyname: another suggestion was to store each image in its own container, so 1 container/image | 16:09 |
| notmyname | ratelimits aren't because swift depoyers are mean. it's to protect the system overall | 16:10 |
| hemanthm | +1 | 16:10 |
| hemanthm | :) | 16:10 |
| hemanthm | I did mention that in one of my comments, but I think it makes more sense coming from a Swift guy | 16:10 |
| hemanthm | notmyname: from a swift perspective, how does storing each image in its own container sound? | 16:11 |
| hemanthm | is it better or inefficient or .. ? | 16:12 |
| notmyname | sounds like you better not sit next to the swift ops guys or they might one day do things to you that you'd find unpleasant. | 16:13 |
| notmyname | mostly it moves the container issues to the account level. it doesn't avoid any issues overall | 16:14 |
| *** kyles_ne has joined #openstack-swift | 16:14 | |
| notmyname | and there are some deployers that do limit the number of swift containers that you can have. | 16:14 |
| hemanthm | yes, sir! makes so much sense, you'll find my comment mentioning that as well | 16:16 |
| hemanthm | notmyname: it'd be really helpful if you could mention that on the spec | 16:16 |
| openstackgerrit | paul luse proposed a change to openstack/swift-specs: Updates to EC design spec https://review.openstack.org/125190 | 16:16 |
| peluse | notmyname, right (to your prior EC related comments) | 16:17 |
| notmyname | hemanthm: done | 16:19 |
| notmyname | peluse: thanks | 16:19 |
| *** infotection has joined #openstack-swift | 16:20 | |
| *** dmsimard_away is now known as dmsimard | 16:20 | |
| *** dmsimard is now known as dmsimard_away | 16:22 | |
| hemanthm | notmyname: Glance says thanks! :) | 16:22 |
| peluse | notmyname, maybe you can take a look at https://review.openstack.org/#/c/128491/ I just re-reviwed and tested and I think we're ready to rock | 16:24 |
| *** infotection has quit IRC | 16:46 | |
| *** jamiehan_ has quit IRC | 16:46 | |
| *** infotection has joined #openstack-swift | 16:47 | |
| *** gyee has joined #openstack-swift | 16:50 | |
| *** exploreshaifali has joined #openstack-swift | 16:51 | |
| *** annegent_ has quit IRC | 16:52 | |
| *** Rynfar has joined #openstack-swift | 16:53 | |
| Rynfar | I am looking for some assistance solving an issue when performing segmented uploads with swift. I am having difficulty finding a good way to validate the upload. | 16:54 |
| *** acoles is now known as acoles_away | 16:56 | |
| *** infotection has quit IRC | 16:56 | |
| Rynfar | Never used IRC... am I just supposed to ask questions here? | 16:57 |
| notmyname | Rynfar: yup. and welcome! | 16:58 |
| Rynfar | Thanks! | 16:59 |
| *** infotection has joined #openstack-swift | 16:59 | |
| *** jyoti-ranjan has joined #openstack-swift | 17:00 | |
| Rynfar | We are trying to find a way to upload content via swift to our cloud storage server, but we are running into problems with content periodically failing hashes. We manually hash the files before uploading and then after downloading verify against that same hash. | 17:00 |
| notmyname | that's good (that you send and check the etag | 17:01 |
| Rynfar | Part of the problem is i don't understand the hashing done by the cloud storage or how to take advantage of it. | 17:03 |
| notmyname | Rynfar: what do you not understand? | 17:03 |
| *** jordanP has quit IRC | 17:04 | |
| *** dmsimard_away is now known as dmsimard | 17:04 | |
| Rynfar | When I upload content, i need generate a hash of the file and compare it to the etag. Anytime i have done this it has not matched. So i started relying on my own hashes. | 17:05 |
| Rynfar | What is the proper way to generate a hash that will match the tag? | 17:05 |
| Rynfar | etag | 17:05 |
| notmyname | Rynfar: this is a normal object right? you aren't using large object manifests, right? | 17:05 |
| Rynfar | Im not sure if i can answer that. | 17:06 |
| Rynfar | is that based on the file size? | 17:06 |
| notmyname | yes. objects in swift are limited to 5GB (by default). you can logically tie together objects so that you have a "large object" of arbitrary size | 17:07 |
| Rynfar | ok so most of our files are less than 5GB but i am segmenting them in 50 MB chunks in order to speed up our upload process. | 17:07 |
| notmyname | ok. and you're getting an error (a hash mismatch) on those 50MB chunks? | 17:08 |
| Rynfar | We are not using the built in hashing of the object storage as we don't know how basically. | 17:10 |
| Rynfar | I am using a script to hash the file before uploading it. Then after i download it i use the previous hash to compare. | 17:10 |
| Rynfar | I would like to use the built in hashing of the object storage, but cannot find documentation that i can understand. I am very new to this process. | 17:11 |
| notmyname | Rynfar: https://gist.github.com/notmyname/f2873b8370c680bb67bd <<-- here's an example using curl | 17:11 |
| notmyname | Rynfar: so let's walk through it | 17:11 |
| notmyname | first is an account GET. just so you see that it's there | 17:11 |
| notmyname | second, I create the container "c" | 17:12 |
| notmyname | then `ls -lh 1MB` just to show that it exists locally | 17:12 |
| notmyname | next I upload it to "obj1", and don't set an etag. swift calculates that and stores it for later auditing | 17:12 |
| notmyname | then I md5 the local 1MB file | 17:13 |
| notmyname | and then I upload 1MB again, this time to "obj2" and also pass in the etag. note that it all works just fine | 17:13 |
| Rynfar | this is exactly what i was looking for. | 17:13 |
| notmyname | and finally, I try again to "obj3" but this time I send a bad etag (I changed the last 4 bytes to ffff). note the error | 17:14 |
| Rynfar | fantastic, thank you so much! | 17:14 |
| notmyname | Rynfar: np | 17:15 |
| *** infotection has quit IRC | 17:15 | |
| notmyname | Rynfar: just out of curiosity, can you share what you're using swift for? I'm always interested to find out how people are using it | 17:15 |
| Rynfar | We are using it for a simple automated file delivery system. We allow users to drop content into a hot folder and then expose it to them via a webpage. They can select the files they need and retrieve them via a download manager we have built. | 17:17 |
| notmyname | nice! | 17:18 |
| Rynfar | We previously used ftp to upload to the cloud so our manual hash process made more sense and we were able to trap failures easier...(because we understood it) | 17:18 |
| Rynfar | I just recently started using swift to do this so there is a lot of new stuff! | 17:18 |
| notmyname | Rynfar: please let me know how I can help! | 17:18 |
| notmyname | Rynfar: does your download manager do auth and proxy the serving of the files? | 17:19 |
| notmyname | Rynfar: one thing you should look in to is swift's tempurl support http://docs.openstack.org/developer/swift/middleware.html#tempurl | 17:19 |
| Rynfar | Awesome thank you, that is actually right in line with what we would be using this for. | 17:20 |
| Rynfar | I am sure i will be back here, your answers were "swift" :P | 17:20 |
| Rynfar | and most helpful | 17:21 |
| notmyname | Rynfar: are you using your own swift deployment or are you using a service provider? | 17:21 |
| Rynfar | We are using a service provider | 17:21 |
| notmyname | ok | 17:22 |
| Rynfar | Out of curiosity if a file is segmented with swift, will the ETAG still match? | 17:25 |
| Rynfar | or doe the local file have to be segmented and then hashed to obtain a matching hash? | 17:25 |
| Rynfar | The ETAG is concatenated of the ETAG of each segment correct? | 17:26 |
| notmyname | segmented with swift? | 17:26 |
| notmyname | so if you eg use `split` and break up a file locally and then upload those segments, those are the things that are hashed. think of it from the perspective of swift. swift checks the hashes on a per-object basis | 17:27 |
| Rynfar | Ok so we don't split them up locally, but when i call swift i use the split command | 17:27 |
| notmyname | however, if you are using large object manifests (which you may want to at some point), then the etag is calculated differently | 17:27 |
| notmyname | swift doesn't have anything to automatically split files | 17:28 |
| Rynfar | hang on let me grab the command i am using | 17:28 |
| *** infotection has joined #openstack-swift | 17:28 | |
| Rynfar | -S 52428800 is the flag i am adding when calling swift on our files. | 17:29 |
| *** morganfainberg is now known as captainmorgan | 17:29 | |
| notmyname | ah ok. the `swift` CLI tool | 17:29 |
| Rynfar | Oh yes, my apologies, i am probably not using terminology correctly. | 17:29 |
| notmyname | no, you are. it's just confusing because "swift" is both the name of the client-side CLI tool and the whole project | 17:30 |
| Rynfar | right, adds to the confusion! | 17:30 |
| Rynfar | some of our files are bigger than 5GB, but par to the reason we start using -S was so we could speed up the uploads and add --object-threads 20 | 17:31 |
| notmyname | right. | 17:31 |
| Rynfar | we noticed a huge increase in speeds when we split the files and threaded it. | 17:32 |
| Rynfar | But then we got lost in properly validating our uploads, without doing the silly process we are doing now. | 17:32 |
| notmyname | also, I'd recommend you using the --use-slo option for static manifests rather than dynamic ones. | 17:32 |
| Rynfar | perfect, i will add that | 17:33 |
| notmyname | Rynfar: so when you are using manifests (you are with the -S option), then etags are created differently. it's not just an md5 | 17:33 |
| Rynfar | ah, that explains why it never matched. | 17:34 |
| notmyname | Rynfar: reason being, large objects can be very large. and they aren't stored in just one place in the cluster. | 17:34 |
| Rynfar | So each piece must have its own hash? | 17:34 |
| notmyname | ie swift (the cluster) can't check a local object against it's md5 if it doesn't have all the data on that particular hard drive | 17:35 |
| notmyname | Rynfar: see http://docs.openstack.org/developer/swift/overview_large_objects.html | 17:36 |
| notmyname | Rynfar: look for etag there for how to do large object etags | 17:37 |
| Rynfar | Wow i wish i had found that doc a while back. That has lot of information i have stumbled upon in it. | 17:38 |
| notmyname | Rynfar: http://swift.openstack.org/ is where all of those things I've linked are rooted. those are the dev docs that we the devs write for swift | 17:38 |
| *** infotection has quit IRC | 17:39 | |
| Rynfar | Ok so i understand now that it is a concatenated md5sum of the segments, which will return a different md5 than that of a single file. How do you compare this to the single file you originally uploaded? | 17:41 |
| notmyname | Rynfar: well, if you're using the CLI, then you don't worry about it. if you are using the API directly (eg either with a launguage SDK or otherwise with HTTP requests), then you do | 17:44 |
| notmyname | Rynfar: basically, each chunk can be validated by the md5sum, but the manifest object etag is done differently | 17:45 |
| Rynfar | Right now i am using the CLI tool. But i recently had an upload that didn't match our manually generated hash when it was downloaded. That is why i am trying to figure out how to validate the upload matches the local file before registering the content in out database as "good" | 17:47 |
| notmyname | Rynfar: yeah, when downloading large object manifests, the etag will be the alternate way | 17:50 |
| Rynfar | This is where i get lost. If the ETAG is a concatenated md5sum of the segments and i uploaded a single file. How do i get the same md5sum locally to compare against what was uploaded? | 17:51 |
| Rynfar | My appologies if i am running in a circle. | 17:52 |
| Rynfar | the single file has many segments that are summed up to make the md5sum hash which is the ETAG. But locally i only have a single file to generate a hash from, so this is where i don't understand how to verify. | 17:53 |
| *** infotection has joined #openstack-swift | 17:53 | |
| Rynfar | when i request the ETAG and try to match it to my local md5hash it doesn't match. | 17:53 |
| Rynfar | even though the content is technically good when i download it because the hash matches once its pieced back together. | 17:53 |
| *** acoles_away has quit IRC | 17:55 | |
| Rynfar | right now our process is - Hash File > Send MD5 Hash to Database > Upload via Swift > Download File > Get MD5 Hash from Database > Hash File | 17:55 |
| Rynfar | I would like to Hash File > Send Hash to Database > Upload File > Verify Hash that was uploaded is good > Download File > Get MD5 Hash from Database > Hash File and verify it is good. | 17:55 |
| notmyname | Rynfar: you're local DB? | 17:56 |
| Rynfar | yes | 17:56 |
| notmyname | s/you're/your/ | 17:56 |
| notmyname | ok, so here's how that would have to work for the manifest objects | 17:56 |
| *** acoles_away has joined #openstack-swift | 17:57 | |
| Rynfar | part of the reason for the redundant need to hash file when uploading and when downloading is we don't trust on the fly hashing when files are downloaded. We require it to be on disk before performing a final hash. | 17:58 |
| notmyname | you can store the hash of the whole thing in your DB. and you can calculate the hash that you send on the PUT ("md5 of cat md5s of chunks"). when you GET the object, you calculate the md5 of the bytes you receive from swift and compare that to your stored md5. the etag header that you get from swift won't help you there | 17:58 |
| *** acoles_away is now known as acoles | 17:58 | |
| *** ChanServ sets mode: +v acoles | 17:58 | |
| Rynfar | oh wow, i didn't even think about doing that. | 17:59 |
| notmyname | Rynfar: you're hashing only what's on disk? that means you're spooling and would be a bottleneck for scale and throughput right? | 17:59 |
| Rynfar | right, the reason we do this is we don't trust what happens in memory. We've had more than once hashes pass in memory but then fail when written to disk. | 18:00 |
| Rynfar | Sometimes we use content as large as 700 GB | 18:01 |
| Rynfar | So we have a post processing time that we plan for | 18:01 |
| *** CaioBrentano has joined #openstack-swift | 18:01 | |
| notmyname | is your download app or server something that is a service or is it a local app that every client is using? | 18:03 |
| Rynfar | every client is using it. | 18:04 |
| Rynfar | locally | 18:04 |
| notmyname | ah ok | 18:04 |
| Rynfar | we host the device for them, so we are in control of everything end to end. | 18:04 |
| *** davidhadas has joined #openstack-swift | 18:06 | |
| notmyname | after your app downloads it and validates it, is the data going on the network again? or is it download to the same box where it's used? | 18:06 |
| Rynfar | once it is validated it is only moved locally to a new location on the same hard drive. | 18:06 |
| notmyname | ok | 18:06 |
| Rynfar | We are moving from a satellite system that does most of this for us, so we are attempting to develop our own electronic solution. | 18:07 |
| notmyname | just to be clear, because it's important, the etag checking weirdness is only necessary for the large object manifests (ie when you're using the -S option). when you aren't, md5sum == etag | 18:07 |
| Rynfar | I am a fairly amateur python programmer taking over this project so there is a lot to learn. Thus my lack of understanding data. | 18:08 |
| Rynfar | ok so that is super helpful to know. | 18:08 |
| Rynfar | the particular file that failed us was actually only 250mb, but i had it split... no reason to split it at 250 mb in this case. | 18:08 |
| *** infotection has quit IRC | 18:09 | |
| Rynfar | Do you happen to have an example of how you md5 a local file in byte segments? | 18:09 |
| notmyname | true, but it sounds like you have to solve for that case anyway | 18:09 |
| Rynfar | i believe you were md5 cat the file? | 18:09 |
| notmyname | what do you mean "in byte segments"? | 18:09 |
| Rynfar | i meant chunks sorry | 18:10 |
| Rynfar | hang on I'm rereading what you wrote, i think i mis understood it. | 18:10 |
| Rynfar | ("md5 of cat md5s of chunks") this is what i am unsure how to do. | 18:11 |
| Rynfar | you are referring to cat in linux correct? | 18:12 |
| Rynfar | using cat to read the file and md5 the chunks? | 18:12 |
| notmyname | no, not specifically. I'm referring to the concatenation of the md5 sums of the chunks | 18:12 |
| Rynfar | ok heres a better question. how do you "calculate the hash that you send on the PUT" | 18:13 |
| notmyname | so in python(-ish): h = md5(''.join(md5(x) for x in list_of_all_chunk_bodies)) | 18:14 |
| notmyname | clearly that doesn't scale | 18:14 |
| notmyname | but the idea is there | 18:14 |
| Rynfar | yup that makes sense, that is what i was looking for. | 18:15 |
| Rynfar | this will give me the redundancy i need but allow me to validate my uploads to the cloud | 18:16 |
| notmyname | redundancy is the wrong word ;-) | 18:16 |
| notmyname | validation might be better :-) | 18:16 |
| notmyname | ie you aren't storing multiple copies | 18:16 |
| Rynfar | Agree'd! | 18:17 |
| Rynfar | You have been extremely helpful today, far more than that of my service provider. | 18:17 |
| Rynfar | I think i have everything i need to proceed to a solution. | 18:17 |
| Rynfar | Many thanks! | 18:19 |
| notmyname | ouch. now I'm really curious about who your service provider is ;-) | 18:20 |
| *** infotection has joined #openstack-swift | 18:20 | |
| notmyname | Rynfar: I just approved https://review.openstack.org/#/c/126310/ for merge. which should make your life easier on specifying segment sizes | 18:25 |
| Rynfar | *cough* high winds, don't tell them i said anything! | 18:26 |
| Rynfar | Human readable! That is a fantastic change! | 18:26 |
| *** infotection has quit IRC | 18:37 | |
| *** jyoti-ranjan has quit IRC | 18:37 | |
| *** IRTermite has quit IRC | 18:39 | |
| notmyname | Rynfar: isn't that the CDN for games? or at least that's what they we're advertising a while back | 18:40 |
| Rynfar | Its possible, though we use them for another purpose. | 18:43 |
| Rynfar | They are fairly new to open stack. | 18:44 |
| notmyname | peluse: torgomatic: https://review.openstack.org/#/c/128491/ | 18:46 |
| notmyname | +2/+1 | 18:46 |
| torgomatic | sweet | 18:46 |
| notmyname | peluse: so I think tsg needs to rebase his PUT path patch on that now? | 18:49 |
| peluse | notmyname, excellent | 18:49 |
| peluse | I'll ping him in case he's not IRC | 18:50 |
| notmyname | (he's not) | 18:51 |
| peluse | pinged :) | 18:51 |
| peluse | FYI the other priority item for EC that we covered in SC is ready as well (for yours and togomatics review) | 18:52 |
| torgomatic | peluse: link? | 18:52 |
| peluse | https://review.openstack.org/#/c/112449/ | 18:53 |
| peluse | just the diskfile delete thing where we cleaned up some test code - not time critical but might as well knock it out while fresh in the brain if you have a minute | 18:53 |
| *** tkay has joined #openstack-swift | 18:54 | |
| torgomatic | peluse: sure, I'll take a look at that once I've gotten some lunch :) | 18:54 |
| peluse | gracias | 18:54 |
| *** zaitcev has joined #openstack-swift | 18:54 | |
| *** ChanServ sets mode: +v zaitcev | 18:54 | |
| peluse | thanks for the reminder too - should go eat something too :) | 18:54 |
| *** infotection has joined #openstack-swift | 18:57 | |
| peluse | notmyname, tsg is on it... | 19:00 |
| *** kyles_ne has quit IRC | 19:02 | |
| *** kyles_ne has joined #openstack-swift | 19:03 | |
| *** infotection has quit IRC | 19:06 | |
| *** kyles_ne has quit IRC | 19:07 | |
| *** infotection has joined #openstack-swift | 19:10 | |
| openstackgerrit | Alan Erwin proposed a change to openstack/swift: Ssync_sender bug with creating a delete request https://review.openstack.org/129358 | 19:12 |
| *** hemanthm has left #openstack-swift | 19:14 | |
| *** annegent_ has joined #openstack-swift | 19:15 | |
| *** IRTermite has joined #openstack-swift | 19:16 | |
| openstackgerrit | paul luse proposed a change to openstack/swift: Build up reconstructor with correct node selection https://review.openstack.org/129361 | 19:18 |
| *** nshaikh has joined #openstack-swift | 19:18 | |
| *** dmsimard is now known as dmsimard_away | 19:19 | |
| openstackgerrit | Alan Erwin proposed a change to openstack/swift: Ssync_sender bug with creating a delete request https://review.openstack.org/129358 | 19:19 |
| *** nshaikh has quit IRC | 19:20 | |
| *** bill_az has quit IRC | 19:22 | |
| *** annegent_ has quit IRC | 19:25 | |
| *** infotection has quit IRC | 19:28 | |
| *** infotection has joined #openstack-swift | 19:29 | |
| openstackgerrit | A change was merged to openstack/python-swiftclient: Allow segment size to be specified in a human readable way. https://review.openstack.org/126310 | 19:33 |
| *** nshaikh has joined #openstack-swift | 19:33 | |
| *** kyles_ne has joined #openstack-swift | 19:34 | |
| openstackgerrit | Pete Zaitcev proposed a change to openstack/python-swiftclient: Fix the info command with --insecure https://review.openstack.org/129364 | 19:39 |
| *** kyles_ne has quit IRC | 19:41 | |
| *** infotection has quit IRC | 19:42 | |
| openstackgerrit | paul luse proposed a change to openstack/swift: Add another device to SAIO to support the default EC development environment. https://review.openstack.org/128154 | 19:42 |
| *** astellwag has quit IRC | 19:51 | |
| *** kyles_ne has joined #openstack-swift | 19:54 | |
| *** infotection has joined #openstack-swift | 19:59 | |
| *** aerwin has joined #openstack-swift | 20:07 | |
| *** nshaikh has quit IRC | 20:08 | |
| openstackgerrit | A change was merged to openstack/swift: Allow sending object metadata after data https://review.openstack.org/128491 | 20:14 |
| *** infotection has quit IRC | 20:18 | |
| -openstackstatus- NOTICE: Gerrit will be offline from 2100-2130 for project renames | 20:34 | |
| *** ChanServ changes topic to "Gerrit will be offline from 2100-2130 for project renames" | 20:34 | |
| *** silor has quit IRC | 20:35 | |
| *** dave-mccowan has quit IRC | 20:35 | |
| *** infotection has joined #openstack-swift | 20:48 | |
| *** lpabon has quit IRC | 20:49 | |
| *** miqui has quit IRC | 20:50 | |
| *** kyles_ne has quit IRC | 20:52 | |
| *** kyles_ne has joined #openstack-swift | 20:52 | |
| peluse | nice placement for Swift: http://www.openstack.org/software/juno/press-release/ | 20:55 |
| peluse | right there near the top of the list! | 20:55 |
| *** infotection has quit IRC | 20:56 | |
| *** jistr has quit IRC | 20:58 | |
| openstackgerrit | Anne Gentle proposed a change to openstack/swift-specs: Adds Object Storage v1 API specification information https://review.openstack.org/129384 | 21:00 |
| *** bill_az has joined #openstack-swift | 21:03 | |
| -openstackstatus- NOTICE: Gerrit is offline from 2100-2130 for project renames | 21:04 | |
| *** ChanServ changes topic to "Gerrit is offline from 2100-2130 for project renames" | 21:04 | |
| clayg | whoa just got a 503 on review | 21:06 |
| *** gyee has quit IRC | 21:14 | |
| notmyname | clayg: see the topic | 21:14 |
| openstackgerrit | Anne Gentle proposed a change to openstack/swift-specs: Adds Object Storage v1 API specification information https://review.openstack.org/129384 | 21:15 |
| notmyname | annegentle: specs now equals api docs? | 21:15 |
| *** acoles has quit IRC | 21:16 | |
| notmyname | peluse: yup. pretty cool. swift's storage policies are pretty much getting top billing in all the articles I've seen | 21:17 |
| *** infotection has joined #openstack-swift | 21:18 | |
| *** acoles_away has joined #openstack-swift | 21:19 | |
| *** acoles_away is now known as acoles | 21:19 | |
| *** ChanServ sets mode: +v acoles | 21:19 | |
| *** tdasilva has quit IRC | 21:20 | |
| *** infotection has quit IRC | 21:25 | |
| *** ChanServ changes topic to "Swift 2.2.0 https://launchpad.net/swift/juno/2.2.0 | Priority Reviews: https://wiki.openstack.org/wiki/Swift/PriorityReviews | Ideas: https://wiki.openstack.org/wiki/Swift/ideas | Review Dashboard: http://bit.ly/1iVBigF | Kilo summit topics: https://etherpad.openstack.org/p/kilo-swift-summit-topics" | 21:26 | |
| -openstackstatus- NOTICE: Gerrit is back online | 21:26 | |
| *** infotection has joined #openstack-swift | 21:27 | |
| *** gyee has joined #openstack-swift | 21:27 | |
| *** captainmorgan is now known as needscoffee | 21:29 | |
| *** needscoffee is now known as captainmorgan | 21:29 | |
| *** kyles_ne has quit IRC | 21:31 | |
| *** kyles_ne has joined #openstack-swift | 21:33 | |
| annegentle | notmyname: only the long form doc | 21:42 |
| annegentle | notmyname: I have a collection of articles from the object storage long form doc that I'm going to propose for the User Guide | 21:42 |
| notmyname | annegentle: what's the "long form doc"? | 21:43 |
| annegentle | notmyname: remember these? Sourced from object-api, image-api, compute-api repos? http://docs.openstack.org/api/api-specs.html | 21:43 |
| notmyname | ah, yes. I was looking for that earlier today, in fact | 21:43 |
| annegentle | notmyname: ah yes. Emails went out Aug 20 and Oct 7th letting first PTLs know then the -dev and -docs lists | 21:44 |
| notmyname | annegentle: so you want to move the content of those docs into the respective specs repo? | 21:44 |
| * notmyname goes to search email | 21:45 | |
| annegentle | http://lists.openstack.org/pipermail/openstack-dev/2014-October/048001.html | 21:47 |
| annegentle | notmyname: not all the contents, only spec-ish content | 21:47 |
| *** acoles has quit IRC | 21:47 | |
| annegentle | notmyname: for object storage, the non-spec-ish content can go in the User Guide - http://docs.openstack.org/user-guide/content/ | 21:48 |
| *** exploreshaifali has quit IRC | 21:48 | |
| notmyname | how do you define "spec-ish"? | 21:48 |
| annegentle | notmyname: not "how to" -- content that specifies | 21:48 |
| *** acoles_away has joined #openstack-swift | 21:49 | |
| *** acoles_away is now known as acoles | 21:49 | |
| *** ChanServ sets mode: +v acoles | 21:49 | |
| *** marcusvrn_ has quit IRC | 21:52 | |
| *** CaioBrentano has quit IRC | 21:56 | |
| notmyname | annegentle: I've got to go offline now. I want to ask more questions later | 21:56 |
| annegentle | notmyname: sure thing, post on the review ideally | 21:57 |
| openstackgerrit | Anne Gentle proposed a change to openstack/swift-specs: Adds Object Storage v1 API specification information https://review.openstack.org/129384 | 21:58 |
| *** infotection has quit IRC | 22:02 | |
| *** aerwin has quit IRC | 22:02 | |
| *** infotection has joined #openstack-swift | 22:02 | |
| *** openstackgerrit has quit IRC | 22:03 | |
| *** openstackgerrit has joined #openstack-swift | 22:05 | |
| *** erlon_ has quit IRC | 22:09 | |
| *** infotection has quit IRC | 22:20 | |
| *** infotection has joined #openstack-swift | 22:32 | |
| *** mahatic has quit IRC | 22:42 | |
| openstackgerrit | Anne Gentle proposed a change to openstack/swift-specs: Adds Object Storage v1 API specification information https://review.openstack.org/129384 | 22:43 |
| *** mahatic has joined #openstack-swift | 22:54 | |
| *** infotection has quit IRC | 22:59 | |
| *** infotection has joined #openstack-swift | 22:59 | |
| openstackgerrit | Anne Gentle proposed a change to openstack/swift-specs: Adds Object Storage v1 API specification information https://review.openstack.org/129384 | 23:05 |
| *** kyles_ne has quit IRC | 23:12 | |
| *** kyles_ne has joined #openstack-swift | 23:13 | |
| *** infotection has quit IRC | 23:14 | |
| *** infotection has joined #openstack-swift | 23:15 | |
| *** infotection has quit IRC | 23:22 | |
| *** tkay has quit IRC | 23:27 | |
| *** kyles_ne has quit IRC | 23:29 | |
| *** kyles_ne has joined #openstack-swift | 23:29 | |
| *** infotection has joined #openstack-swift | 23:30 | |
| *** gyee has quit IRC | 23:30 | |
| *** infotection has quit IRC | 23:39 | |
| *** infotection has joined #openstack-swift | 23:39 | |
| *** gyee has joined #openstack-swift | 23:44 | |
| *** nellysmitt has quit IRC | 23:45 | |
| *** elambert has quit IRC | 23:45 | |
| *** nellysmitt has joined #openstack-swift | 23:45 | |
| *** davidhadas has quit IRC | 23:46 | |
| *** davidhadas has joined #openstack-swift | 23:46 | |
| *** kyles_ne has quit IRC | 23:50 | |
| *** nellysmitt has quit IRC | 23:50 | |
| *** kyles_ne has joined #openstack-swift | 23:50 | |
| *** infotection has quit IRC | 23:53 | |
| *** infotection has joined #openstack-swift | 23:53 | |
| *** kyles_ne has quit IRC | 23:55 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!