| *** blmartin1 has quit IRC | 00:07 | |
| *** km__ has joined #openstack-swift | 00:13 | |
| *** mattoliverau has quit IRC | 00:24 | |
| *** chlong has quit IRC | 00:32 | |
| *** chlong has joined #openstack-swift | 00:49 | |
| *** rminmin has joined #openstack-swift | 00:57 | |
| *** nadeem has joined #openstack-swift | 01:03 | |
| *** mattoliverau has joined #openstack-swift | 01:06 | |
| *** ChanServ sets mode: +v mattoliverau | 01:07 | |
| *** chlong has quit IRC | 01:09 | |
| *** matt6434 has joined #openstack-swift | 01:13 | |
| *** mattoliverau has quit IRC | 01:19 | |
| *** matt6434 is now known as mattoliverau | 01:19 | |
| *** ChanServ sets mode: +v mattoliverau | 01:19 | |
| *** chlong has joined #openstack-swift | 01:22 | |
| *** dmorita has joined #openstack-swift | 01:26 | |
| *** dmorita has quit IRC | 01:45 | |
| *** itlinux has joined #openstack-swift | 02:24 | |
| *** itlinux has quit IRC | 02:30 | |
| *** csmart has quit IRC | 02:36 | |
| *** nadeem has quit IRC | 02:41 | |
| *** asettle has quit IRC | 02:44 | |
| *** nadeem has joined #openstack-swift | 02:49 | |
| *** csmart has joined #openstack-swift | 02:52 | |
| *** asettle has joined #openstack-swift | 02:52 | |
| *** sanchitmalhotra has joined #openstack-swift | 03:00 | |
| *** janonymous has joined #openstack-swift | 03:04 | |
| *** venkatesh has joined #openstack-swift | 03:10 | |
| *** rminmin has quit IRC | 03:11 | |
| *** haomaiwang has joined #openstack-swift | 03:16 | |
| *** janonymous has quit IRC | 03:17 | |
| *** janonymous_ has joined #openstack-swift | 03:17 | |
| *** mragupat has joined #openstack-swift | 03:19 | |
| *** janonymous_ has quit IRC | 03:21 | |
| *** mragupat has quit IRC | 03:22 | |
| *** mragupat has joined #openstack-swift | 03:23 | |
| openstackgerrit | Janie Richling proposed openstack/swift: WIP: Change POST-as-COPY to preserve the DLO manifest contents https://review.openstack.org/252791 | 03:37 |
|---|---|---|
| *** janonymous has joined #openstack-swift | 03:46 | |
| *** links has joined #openstack-swift | 03:48 | |
| *** trifon has joined #openstack-swift | 04:00 | |
| *** haomaiwang has quit IRC | 04:01 | |
| *** haomaiwang has joined #openstack-swift | 04:01 | |
| *** km__ has quit IRC | 04:05 | |
| *** trifon has quit IRC | 04:06 | |
| *** csmart has quit IRC | 04:17 | |
| *** csmart has joined #openstack-swift | 04:19 | |
| *** SkyRocknRoll has joined #openstack-swift | 04:21 | |
| *** SkyRocknRoll has quit IRC | 04:36 | |
| *** haomaiwang has quit IRC | 04:37 | |
| *** klrmn has quit IRC | 05:00 | |
| *** david-lyle has quit IRC | 05:00 | |
| *** david-lyle has joined #openstack-swift | 05:01 | |
| *** mragupat has quit IRC | 05:03 | |
| *** mragupat has joined #openstack-swift | 05:04 | |
| *** lcurtis has quit IRC | 05:16 | |
| *** trifon has joined #openstack-swift | 06:00 | |
| *** mingdang1 has joined #openstack-swift | 06:01 | |
| mingdang1 | . | 06:01 |
| *** janonymous_ has joined #openstack-swift | 06:20 | |
| *** janonymous has quit IRC | 06:22 | |
| *** janonymous_ has quit IRC | 06:25 | |
| *** mragupat has quit IRC | 06:27 | |
| *** venkatesh has quit IRC | 06:33 | |
| *** ChubYann has quit IRC | 06:37 | |
| *** dmorita has joined #openstack-swift | 06:46 | |
| *** venkatesh has joined #openstack-swift | 06:47 | |
| *** bkumar has joined #openstack-swift | 06:48 | |
| *** dmorita has quit IRC | 06:51 | |
| *** silor has joined #openstack-swift | 07:07 | |
| *** asettle has quit IRC | 07:08 | |
| *** silor has quit IRC | 07:24 | |
| *** ntt has quit IRC | 07:33 | |
| *** silor has joined #openstack-swift | 07:33 | |
| openstackgerrit | Jude Augustine Job proposed openstack/python-swiftclient: Error with uploading large object includes unicode path https://review.openstack.org/265544 | 07:36 |
| *** ppai has joined #openstack-swift | 07:37 | |
| *** chlong has quit IRC | 07:39 | |
| openstackgerrit | Jude Augustine Job proposed openstack/python-swiftclient: Error with uploading large object includes unicode path https://review.openstack.org/265544 | 07:43 |
| *** arnox has joined #openstack-swift | 08:01 | |
| *** rledisez has joined #openstack-swift | 08:09 | |
| *** haomaiwang has joined #openstack-swift | 08:12 | |
| *** ntt has joined #openstack-swift | 08:22 | |
| *** haomaiwang has quit IRC | 08:27 | |
| *** openstackgerrit has quit IRC | 08:32 | |
| *** openstackgerrit has joined #openstack-swift | 08:33 | |
| *** nadeem has quit IRC | 08:37 | |
| *** geaaru has joined #openstack-swift | 08:39 | |
| *** jmccarthy has quit IRC | 08:45 | |
| *** jmccarthy has joined #openstack-swift | 08:49 | |
| *** venkatesh has quit IRC | 08:55 | |
| *** jordanP has joined #openstack-swift | 09:02 | |
| *** silor has quit IRC | 09:06 | |
| *** venkatesh has joined #openstack-swift | 09:07 | |
| *** daemontool has joined #openstack-swift | 09:14 | |
| *** Jeffrey4l__ has quit IRC | 09:19 | |
| *** daemontool has quit IRC | 09:25 | |
| *** jistr has joined #openstack-swift | 09:25 | |
| *** daemontool has joined #openstack-swift | 09:25 | |
| *** daemontool has quit IRC | 09:27 | |
| *** daemontool has joined #openstack-swift | 09:29 | |
| *** joeljwright has joined #openstack-swift | 09:32 | |
| *** ChanServ sets mode: +v joeljwright | 09:32 | |
| *** proteusguy has quit IRC | 09:39 | |
| *** itlinux has joined #openstack-swift | 09:39 | |
| *** itlinux has quit IRC | 09:44 | |
| *** proteusguy has joined #openstack-swift | 09:52 | |
| *** daemontool has quit IRC | 09:53 | |
| *** daemontool has joined #openstack-swift | 09:53 | |
| *** daemontool has quit IRC | 09:55 | |
| *** daemontool has joined #openstack-swift | 09:56 | |
| *** daemontool has quit IRC | 09:57 | |
| *** daemontool has joined #openstack-swift | 09:59 | |
| *** saltsa_ has joined #openstack-swift | 10:02 | |
| *** Sushma has joined #openstack-swift | 10:02 | |
| *** saltsa has quit IRC | 10:02 | |
| *** Sushma has quit IRC | 10:03 | |
| *** hseipp has joined #openstack-swift | 10:03 | |
| *** hseipp has quit IRC | 10:03 | |
| *** hseipp has joined #openstack-swift | 10:04 | |
| *** Sushma has joined #openstack-swift | 10:04 | |
| Sushma | swauth middleware depends on swift version, how swift version is formed? | 10:05 |
| *** daemontool_ has joined #openstack-swift | 10:10 | |
| *** daemontool has quit IRC | 10:13 | |
| *** daemontool_ has quit IRC | 10:17 | |
| *** daemontool_ has joined #openstack-swift | 10:17 | |
| openstackgerrit | Jude Augustine Job proposed openstack/python-swiftclient: Error with uploading large object includes unicode path https://review.openstack.org/265544 | 10:22 |
| *** aix has joined #openstack-swift | 10:35 | |
| *** silor has joined #openstack-swift | 10:36 | |
| *** Jeffrey4l__ has joined #openstack-swift | 10:37 | |
| *** mingdang1 has quit IRC | 10:45 | |
| *** silor has quit IRC | 11:00 | |
| *** bkumar has quit IRC | 11:12 | |
| *** silor has joined #openstack-swift | 11:18 | |
| *** natarej has joined #openstack-swift | 11:22 | |
| *** daemontool_ is now known as daemontool | 11:32 | |
| *** sanchitmalhotra has quit IRC | 11:33 | |
| *** aix has quit IRC | 11:47 | |
| *** venkatesh has quit IRC | 11:52 | |
| *** daemontool_ has joined #openstack-swift | 11:53 | |
| *** daemontool has quit IRC | 11:55 | |
| *** lcurtis has joined #openstack-swift | 11:55 | |
| *** geaaru has quit IRC | 11:56 | |
| *** chlong has joined #openstack-swift | 11:57 | |
| *** mingdang1 has joined #openstack-swift | 11:57 | |
| *** Sushma has quit IRC | 11:59 | |
| *** geaaru has joined #openstack-swift | 12:00 | |
| *** haomaiwang has joined #openstack-swift | 12:04 | |
| *** daemontool has joined #openstack-swift | 12:12 | |
| *** daemontool_ has quit IRC | 12:14 | |
| *** lcurtis has quit IRC | 12:17 | |
| *** acoles has quit IRC | 12:20 | |
| *** ppai has quit IRC | 12:22 | |
| *** aix has joined #openstack-swift | 12:22 | |
| *** acoles has joined #openstack-swift | 12:27 | |
| *** ChanServ sets mode: +v acoles | 12:27 | |
| *** HenryG has quit IRC | 12:27 | |
| *** HenryG has joined #openstack-swift | 12:27 | |
| *** acoles has quit IRC | 12:36 | |
| *** ppai has joined #openstack-swift | 12:36 | |
| *** acoles has joined #openstack-swift | 12:38 | |
| *** ChanServ sets mode: +v acoles | 12:38 | |
| *** alejandrito has joined #openstack-swift | 12:45 | |
| *** alejandrito has quit IRC | 12:46 | |
| *** alejandrito has joined #openstack-swift | 12:46 | |
| *** haomaiwang has quit IRC | 13:01 | |
| *** Jeffrey4l__ has quit IRC | 13:01 | |
| *** haomaiwang has joined #openstack-swift | 13:01 | |
| *** Jeffrey4l__ has joined #openstack-swift | 13:06 | |
| *** daemontool has quit IRC | 13:07 | |
| *** Jeffrey4l__ is now known as Jeffrey4l | 13:08 | |
| *** daemontool has joined #openstack-swift | 13:10 | |
| *** haomaiwang has quit IRC | 13:11 | |
| *** haomaiwang has joined #openstack-swift | 13:12 | |
| *** links has quit IRC | 13:13 | |
| *** MVenesio has joined #openstack-swift | 13:21 | |
| MVenesio | Hi guys, quick question : there's any way to measure outbound traffic in swift, always with ceilometer agents ? | 13:23 |
| *** daemontool_ has joined #openstack-swift | 13:23 | |
| *** Jeffrey4l has quit IRC | 13:24 | |
| *** daemontool has quit IRC | 13:26 | |
| *** Jeffrey4l has joined #openstack-swift | 13:28 | |
| *** joeljwright1 has joined #openstack-swift | 13:33 | |
| *** joeljwright has quit IRC | 13:34 | |
| openstackgerrit | Mahati Chamarthy proposed openstack/swift: Decrypt multipart mime doc https://review.openstack.org/248377 | 13:44 |
| *** dmorita has joined #openstack-swift | 13:47 | |
| *** dmorita has quit IRC | 13:52 | |
| *** Jeffrey4l has quit IRC | 13:55 | |
| *** haomaiwang has quit IRC | 14:01 | |
| *** haomaiwang has joined #openstack-swift | 14:01 | |
| *** esker has quit IRC | 14:05 | |
| *** Jeffrey4l has joined #openstack-swift | 14:07 | |
| *** ppai has quit IRC | 14:14 | |
| *** dslevin has quit IRC | 14:15 | |
| *** joeljwright1 has quit IRC | 14:19 | |
| *** joeljwright has joined #openstack-swift | 14:21 | |
| *** ChanServ sets mode: +v joeljwright | 14:21 | |
| *** venkatesh has joined #openstack-swift | 14:22 | |
| glange | you could parse the proxy logs | 14:24 |
| *** mingdang1 has quit IRC | 14:29 | |
| *** petertr7_away is now known as petertr7 | 14:30 | |
| *** dslevin has joined #openstack-swift | 14:30 | |
| *** blmartin has joined #openstack-swift | 14:33 | |
| *** bkumar has joined #openstack-swift | 14:36 | |
| *** jordanP has quit IRC | 14:38 | |
| *** haomaiwang has quit IRC | 15:01 | |
| *** haomaiwang has joined #openstack-swift | 15:01 | |
| *** haomaiwang has quit IRC | 15:01 | |
| *** haomaiwang has joined #openstack-swift | 15:02 | |
| *** Jeffrey4l has quit IRC | 15:06 | |
| *** bkumar has quit IRC | 15:17 | |
| *** Jeffrey4l has joined #openstack-swift | 15:18 | |
| *** petertr7 is now known as petertr7_away | 15:18 | |
| *** venkatesh has quit IRC | 15:25 | |
| *** jordanP has joined #openstack-swift | 15:27 | |
| *** mragupat has joined #openstack-swift | 15:27 | |
| *** breitz has quit IRC | 15:32 | |
| *** breitz has joined #openstack-swift | 15:32 | |
| *** Jeffrey4l has quit IRC | 15:40 | |
| *** petertr7_away is now known as petertr7 | 15:49 | |
| *** Jeffrey4l has joined #openstack-swift | 15:57 | |
| *** esker has joined #openstack-swift | 16:01 | |
| *** esker has quit IRC | 16:01 | |
| *** resker has joined #openstack-swift | 16:02 | |
| *** klrmn has joined #openstack-swift | 16:10 | |
| *** Jeffrey4l has quit IRC | 16:16 | |
| *** klrmn has quit IRC | 16:19 | |
| *** daemontool_ has quit IRC | 16:21 | |
| *** diazjf has joined #openstack-swift | 16:28 | |
| *** lcurtis has joined #openstack-swift | 16:29 | |
| *** Jeffrey4l has joined #openstack-swift | 16:29 | |
| *** john_b has joined #openstack-swift | 16:30 | |
| *** tongli has joined #openstack-swift | 16:31 | |
| *** trifon has quit IRC | 16:34 | |
| diazjf | notmyname, jrichli, I'm at the barbican midcycle sprint today, we will discuss https://review.openstack.org/#/c/241068/ at sometime. | 16:34 |
| diazjf | notmyname, jrichli, take a look and let me know what you think :) | 16:35 |
| *** ntt has quit IRC | 16:35 | |
| *** ntt has joined #openstack-swift | 16:36 | |
| ntt | Hi, is there a way to migrate objects between 2 swift clusters? I have the same keystone database and I want to preserve metadata | 16:37 |
| portante | do folks here do anything with aggregating swift logs with other OS level logs? | 16:45 |
| *** bjkeller has joined #openstack-swift | 16:47 | |
| *** natarej has quit IRC | 16:52 | |
| *** natarej has joined #openstack-swift | 16:56 | |
| openstackgerrit | Ben Martin proposed openstack/swift: Print min_part_hours lockout time remaining https://review.openstack.org/257577 | 16:59 |
| *** tongli_ has joined #openstack-swift | 16:59 | |
| *** tongli has quit IRC | 17:02 | |
| jrichli | diazjf: thanks | 17:06 |
| *** tongli_ has quit IRC | 17:06 | |
| blmartin | mattoliverau: Thanks for the review. I changed all the lambdas in the min_part_hours fix and created a pull request for the bounds fix for sharding_split | 17:07 |
| *** gyee has joined #openstack-swift | 17:08 | |
| *** arnox has quit IRC | 17:08 | |
| *** petertr7 is now known as petertr7_away | 17:09 | |
| diazjf | jrichli, no problemo | 17:10 |
| *** daemontool has joined #openstack-swift | 17:10 | |
| *** daemontool_ has joined #openstack-swift | 17:12 | |
| *** petertr7_away is now known as petertr7 | 17:15 | |
| *** daemontool has quit IRC | 17:15 | |
| openstackgerrit | Bill Huber proposed openstack/swift: Re-format the SLO manifest file on new multipart-manifest GET call https://review.openstack.org/263902 | 17:19 |
| *** rledisez has quit IRC | 17:35 | |
| *** joeljwright has quit IRC | 17:44 | |
| notmyname | good morning | 17:49 |
| *** hseipp has quit IRC | 17:53 | |
| *** dmorita has joined #openstack-swift | 17:57 | |
| *** dmorita has quit IRC | 17:57 | |
| *** klrmn has joined #openstack-swift | 18:02 | |
| *** dmorita has joined #openstack-swift | 18:04 | |
| *** ChanServ changes topic to "Review Dashboard: https://goo.gl/mtEv1C | Hackathon: https://www.eventbrite.com/e/swift-hackathon-bristol-sponsored-by-hpe-tickets-19994495073 | Logs: http://eavesdrop.openstack.org/irclogs/%23openstack-swift/ | Meetings: https://wiki.openstack.org/wiki/Meetings/Swift" | 18:08 | |
| notmyname | swift mid-cycle hackathon registration link is in the channel topic | 18:09 |
| notmyname | thanks acoles for hosting and handling logistics! | 18:09 |
| notmyname | https://www.eventbrite.com/e/swift-hackathon-bristol-sponsored-by-hpe-tickets-19994495073 | 18:09 |
| diazjf | Good morning notmyname, I updated https://review.openstack.org/#/c/241068/ I'm at the Barbican mid-cycle now and we will discuss it at some point. | 18:11 |
| diazjf | let me know what you think | 18:11 |
| diazjf | thanks!! | 18:11 |
| *** jistr has quit IRC | 18:15 | |
| *** sileht has quit IRC | 18:17 | |
| *** zaitcev has joined #openstack-swift | 18:17 | |
| *** ChanServ sets mode: +v zaitcev | 18:17 | |
| *** diazjf has quit IRC | 18:29 | |
| *** daemontool_ has quit IRC | 18:30 | |
| *** xzvk has joined #openstack-swift | 18:32 | |
| *** sileht has joined #openstack-swift | 18:32 | |
| *** aix has quit IRC | 18:35 | |
| *** ChubYann has joined #openstack-swift | 18:36 | |
| *** jordanP has quit IRC | 18:39 | |
| *** diazjf has joined #openstack-swift | 18:40 | |
| *** petertr7 is now known as petertr7_away | 18:41 | |
| *** dmorita has quit IRC | 18:48 | |
| openstackgerrit | Alan Erwin proposed openstack/swift: Refactoring the expiring objects feature https://review.openstack.org/252085 | 18:48 |
| *** dmorita has joined #openstack-swift | 18:51 | |
| *** dmorita has quit IRC | 18:51 | |
| *** dmorita has joined #openstack-swift | 18:52 | |
| *** geaaru has quit IRC | 18:53 | |
| *** barra204 has joined #openstack-swift | 18:55 | |
| *** shakamunyi has quit IRC | 18:56 | |
| *** tongli has joined #openstack-swift | 18:57 | |
| *** diazjf has quit IRC | 18:59 | |
| *** diogogmt has joined #openstack-swift | 19:01 | |
| *** shakamunyi has joined #openstack-swift | 19:12 | |
| *** barra204 has quit IRC | 19:12 | |
| clayg | heyoh! | 19:14 |
| openstackgerrit | Tim Burke proposed openstack/python-swiftclient: Retry file uploads via SwiftService https://review.openstack.org/226897 | 19:16 |
| *** garthb has joined #openstack-swift | 19:20 | |
| *** diazjf has joined #openstack-swift | 19:25 | |
| *** petertr7_away is now known as petertr7 | 19:33 | |
| *** acoles is now known as acoles_ | 19:33 | |
| *** Jeffrey4l has quit IRC | 19:33 | |
| *** bjkeller1 has joined #openstack-swift | 19:37 | |
| *** bjkeller has quit IRC | 19:38 | |
| *** badari has joined #openstack-swift | 19:39 | |
| *** lpabon has joined #openstack-swift | 19:45 | |
| *** Jeffrey4l has joined #openstack-swift | 19:47 | |
| *** petertr7 is now known as petertr7_away | 19:48 | |
| *** minwoob has joined #openstack-swift | 19:49 | |
| *** petertr7_away is now known as petertr7 | 19:53 | |
| *** natarej has quit IRC | 19:55 | |
| *** natarej has joined #openstack-swift | 19:56 | |
| *** barker has joined #openstack-swift | 19:59 | |
| Zyric | Good morning | 20:06 |
| *** dimasot has joined #openstack-swift | 20:10 | |
| dimasot | hi | 20:13 |
| dimasot | are clay or sam here? | 20:15 |
| dimasot | I have an object ring rebalancing related question? hwo is the right person to ask about? | 20:17 |
| clayg | dimasot: ohia | 20:22 |
| *** dmorita has quit IRC | 20:24 | |
| clayg | ahale: do you think new devices getting interleaved into device_id holes is going to mess with anyone terribly enough that we can't merge pdardeau's fix and clean up display ordering later? re: patch 265461 | 20:25 |
| patchbot | clayg: https://review.openstack.org/#/c/265461/ - Look for device holes that can be reused when addi... | 20:25 |
| *** dslevin has quit IRC | 20:25 | |
| clayg | ahale: although I'd imagine if some script was expecting new devices to be added to the end of the device list output it'd probably be hozed pretty good by reusing device id's anyway :\ Hopefully no one is doing that and the humans wouldn't mind an ordering that groups by device location instead of id? | 20:26 |
| *** barker has quit IRC | 20:26 | |
| clayg | ... unless anyone was using device id as a proxy for when a device was created or some such. | 20:26 |
| mattoliverau | blmartin: you da man | 20:27 |
| mattoliverau | oh and morning | 20:27 |
| openstackgerrit | ting wang proposed openstack/swift: Add debug testenv in tox https://review.openstack.org/266025 | 20:27 |
| dimasot | clayg: hi | 20:28 |
| *** barker has joined #openstack-swift | 20:29 | |
| dimasot | the issue that I have that after I make ring rebalance it takes a lot of time till the data is moved by the replicator to the right place | 20:30 |
| dimasot | for 1.5 TB it took >20 hours | 20:30 |
| dimasot | I obseved that the speed of the data movement digrades over time | 20:31 |
| clayg | dimasot: how many workers do you have on your rsync configs? | 20:31 |
| *** natarej_ has joined #openstack-swift | 20:32 | |
| dimasot | I digged a little and looks like that many time is spent at repliactor at phase that compares the hashes | 20:32 |
| dimasot | will check in a second | 20:32 |
| clayg | dimasot: how do you configure vm_cache_pressure - do you track xfs inode/dentry reads? | 20:32 |
| dimasot | I did some PoC script that gots the new and old ring, and creats a data movement mapping between them | 20:33 |
| clayg | dimasot: was this a capacity increase or where you doing a topology migration? | 20:33 |
| dimasot | and then I run a script that moves the data based on this mapping by usync rsync - 1 process per node | 20:33 |
| dimasot | and got 3 times improvement - the data was moved in 6 hours | 20:34 |
| dimasot | how I can track xfs inode/dentry reads? | 20:35 |
| *** dmorita has joined #openstack-swift | 20:35 | |
| dimasot | is it collected by collectd? | 20:35 |
| *** natarej has quit IRC | 20:35 | |
| clayg | dimasot: yeah collectd would have it - probably in a plugin | 20:35 |
| dimasot | topology migration | 20:35 |
| dimasot | ok I will check the xfs inode/dentry reads and number of rsync workers | 20:36 |
| clayg | dimasot: yeah, object-replicator concurrency is one - but the rsync max connections option is also a factor | 20:38 |
| clayg | dimasot: you can also turn on handoffs_first in the obj-replicator config to prioritize rebalanced partitions over normal suffix syncing | 20:39 |
| *** dmorita has quit IRC | 20:40 | |
| dimasot | my main question was if it will be interesting to the comunity to get a tool that makes the rebalancing much faster then waiting to teh replication | 20:40 |
| clayg | dimasot: *I'm* curious | 20:41 |
| dimasot | from the tests that I did I got x3-x10 speedup compared to the replication | 20:41 |
| jlhinson | clayg: I have a question about how memcache works in Swift | 20:42 |
| clayg | dimasot: i wonder how it handles some situations with hash-invalidation that might make it unsafe, or if the default replicator tunings might not be directly comparable? | 20:43 |
| dimasot | I plan to run more tests to observe how it will effect the client workload | 20:43 |
| *** lpabon_ has joined #openstack-swift | 20:43 | |
| *** lpabon_ has quit IRC | 20:43 | |
| jlhinson | My understanding is that the memcache middleware creates a client that connects to some memcache service that's already running | 20:44 |
| dimasot | my main observation was that at rbalancing case you start from the point that at A you have all the data and at B nothing | 20:44 |
| clayg | dimasot: there's also https://review.openstack.org/#/c/215867/ which has a similar goal of "make handoff/rebalance replication go full speed ignoring that replication can consume available i/o that's *generally* bad for access to the system during rebalance | 20:44 |
| dimasot | it is a different from the regular case where replicator catsh some object that exists at one place but not at another | 20:44 |
| dimasot | at rebalancig case it is cleare that all teh data that belog to partition should be moved from A to B | 20:45 |
| dimasot | and you don't need to run all the hash comparisions | 20:45 |
| clayg | dimasot: yeah a partition fork lift during a rebalance is slightly different than a handoff object - but the replicator sees them as basically equivilant (everything in this part should be somewhere else) | 20:45 |
| dimasot | so I created a maping from old to new ring | 20:46 |
| clayg | dimasot: the handling of *primary* partitions is entirely different | 20:46 |
| dimasot | then run a script that creates a tmp dir at destination disk | 20:46 |
| dimasot | and rsync the data from source to destination based on the mapping | 20:47 |
| jlhinson | clayg: so when we want to make changes to the memcache configuration, do we update that primary memcache service before updating the proxy's connection to that service? | 20:47 |
| dimasot | and after it finishes I distribute the new ring, and run "mv" comand to move the partitions from "tmp" to "objects" dirs | 20:48 |
| clayg | dimasot: the replicator does not make a pre-flight REPLICATION request for handoffs already - it makes one at the end so that the new primary can update it's indexes for the partition it's now responsible for - with aggressive vm_cache_pressure tuning and lots of ram it can do this with a reasonable amount of i/o | 20:48 |
| dimasot | and wait till repliactor delete the redundant data at source | 20:48 |
| clayg | jlhinson: in general yeah, you wouldn't tell your proxy to connect to a memcache server that doesn't exist yet? | 20:49 |
| clayg | jlhinson: so, bring some new memcache online, then update proxy configs and HUP the proxies - done | 20:49 |
| *** barker has quit IRC | 20:50 | |
| xzvk | is this a good place to ask a generic swift question? | 20:50 |
| dimasot | clayg: regard https://review.openstack.org/#/c/215867/ - I wil ltake a look but from what I observed at my tests that for same concarency level I got X3 times speedup - and lower netweok usage | 20:50 |
| *** dmorita has joined #openstack-swift | 20:50 | |
| jlhinson | clayg: cool, so now I have a question related to patch 218490 | 20:50 |
| patchbot | jlhinson: https://review.openstack.org/#/c/218490/ - Automatic refresh of memcache config settings | 20:50 |
| clayg | dimasot: preseed the new ring with over replication - interesting | 20:50 |
| *** barker has joined #openstack-swift | 20:51 | |
| dimasot | clayg: what do you mean by "*primary* partitions "? | 20:51 |
| jlhinson | clayg: onovy left a comment saying that we could reduce the number of cache misses when updating config settings by ensuring that all proxy's update their settings at the same time | 20:52 |
| clayg | dimasot: it's a novel approach as far as i know - dfg had mentioned ... in japan maybe .. that he was kicking around an idea to push our updated rings to storage nodes ahead of the proxy to keep primary lookups more likely to hit where the object is *currently* instead of where it should be going but probably isn't there yet | 20:52 |
| xzvk | what is the "swift_hash_path_suffix = HASH_PATH_SUFFIX" I'm not sure what the documentation is referring to? | 20:52 |
| clayg | jlhinson: cache miss is going to happen - not sure how to make a set of distributed proxies atomically start using the new memcache servers - not to mention the new set of servers's will cause some percentage of your memcache lookups to hit new servers anyway | 20:53 |
| clayg | dimasot: i just ment the replicas of the partitions that are on their primary servers - how did I use it? | 20:54 |
| clayg | dimasot: oh, yeah so when a part *belongs* on this node then it'll do the pre-flight REPLICATION requests | 20:54 |
| jlhinson | clayg: right. I was thinking it doesn't matter that the proxies might be out of sync with one another. That's not what's causing the cache misses. | 20:54 |
| jlhinson | clayg: so do I just ignore the cache miss problem for now? | 20:55 |
| *** dmorita has quit IRC | 20:55 | |
| clayg | dimasot: it's not useful for partition fork-lifting - so if your concern was getting rebalanced parts moved as quickly as possible - your using not-useful work - but in general replications focus is not to "rebalance as quickly as possible" or else handoffs_first would be on by default | 20:55 |
| jlhinson | clayg: I'd also like to get more feedback from onovy. Do you know what time he's usually on? | 20:56 |
| clayg | it's primary job is ensuring the health and durability of the system without imposing thundering herd i/o stampeeds that trample all the i/o we want to spend servicing client requests | 20:56 |
| clayg | jlhinson: no idea for onovy - sorry | 20:56 |
| *** diazjf has quit IRC | 20:56 | |
| clayg | jlhinson: yeah afaik it's not huge problem causing folks pain | 20:56 |
| dimasot | yep 100% this is why I thought that maybe a separate "process" should be exists that is "rebalancing" optimized | 20:57 |
| dimasot | this is the reason why I started to look on this problem | 20:58 |
| clayg | dimasot: i agree the existing object-replicator isn't "optimized" for rebalancing - but it already has some knobs to turn it that way - my initial reaction is that we would continue that work over having some ops heavy out-of-band process | 20:58 |
| clayg | dimasot: and others are interest as well (but not terribly interested apparently - no one has reviewed my fix for handoffs_first :P) | 20:59 |
| clayg | dimasot: post up whatever you've got - i'm interested to take a look - I really do think the pre-seed idea is novel | 20:59 |
| dimasot | I thought that handoffs_first is inside since notmyname once suggested to use it | 21:00 |
| dimasot | ok so wha tshould be my next steps | 21:01 |
| *** dmorita has joined #openstack-swift | 21:01 | |
| dimasot | should I create a blue print or something ? | 21:01 |
| dimasot | I still not have my code at any public repo | 21:02 |
| dimasot | where should I put it to share with you? | 21:03 |
| *** dmorita has quit IRC | 21:03 | |
| *** dmorita has joined #openstack-swift | 21:04 | |
| dimasot | clayg? | 21:04 |
| *** dmorita has quit IRC | 21:05 | |
| *** dmorita has joined #openstack-swift | 21:06 | |
| *** barker has quit IRC | 21:07 | |
| *** diazjf has joined #openstack-swift | 21:08 | |
| *** saltsa_ is now known as saltsa | 21:09 | |
| clayg | dimasot: any place on the internet is good for me | 21:15 |
| clayg | dimasot: stick it in a personal github repo | 21:15 |
| clayg | handoffs_first is an option that exists on the swift-object-replicator since forever | 21:16 |
| *** silor has quit IRC | 21:16 | |
| clayg | there's been a few situations where people need rebalance replication to move faster than it would normally want to - it's an active area of interest - I think you'll find you're in good company looking into that system | 21:16 |
| dimasot | ok, thanks | 21:20 |
| dimasot | will ping you when my source will be publicly avaliable - hope it will happen later today | 21:20 |
| clayg | dimasot: neato! | 21:21 |
| openstackgerrit | Tim Burke proposed openstack/swift: Validate X-Timestamps https://review.openstack.org/265488 | 21:22 |
| *** badari has quit IRC | 21:23 | |
| clayg | I keep getting 502's form gerrit | 21:26 |
| * clayg grrrs | 21:26 | |
| *** garthb_ has joined #openstack-swift | 21:27 | |
| *** garthb has quit IRC | 21:28 | |
| *** andrew__ has joined #openstack-swift | 21:33 | |
| *** dmorita has quit IRC | 21:33 | |
| *** dmorita has joined #openstack-swift | 21:34 | |
| *** nadeem has joined #openstack-swift | 21:35 | |
| andrew__ | Hi, I'm following the SAIO steps and on the step to GET account I'm running into "ClientException: Account HEAD failed: http://127.0.0.1:8080/v1/AUTH_test 401 Unauthorized". Does anyone know how to fix this? | 21:35 |
| *** ntt has quit IRC | 21:35 | |
| clayg | torgomatic_: I thought "id est" was reasonable when giving a *single* sepecific additional description | 21:35 |
| torgomatic_ | clayg: maybe | 21:36 |
| torgomatic_ | okay, let's try that again... | 21:36 |
| *** MVenesio has quit IRC | 21:36 | |
| clayg | torgomatic_: either way that bad boy is in a patch chain with mine the middle and timur is out at jury duty | 21:36 |
| torgomatic_ | maybe; I'm pretty medicine-headed today. If I'm wrong, just post a comment to that effect and I'll go change my vote | 21:36 |
| torgomatic_ | eh, or I could supply a follow-on | 21:37 |
| clayg | torgomatic_: I'm sure I could figure it out but I'm not sure the docs as written are confusing if not gramatically correct :\ sorry! | 21:37 |
| *** dmorita has quit IRC | 21:38 | |
| ahale | uh yeah clayg i dunno - those things might be things.. or not. i know if i want to compare rings i just 'import ring' and look at them that way not parsing ring-builder.. maybe others do. i kinda liked the idea of having a "ring-builder defrag" command or something but thats not whats proposed.. | 21:45 |
| clayg | timburke: at first glance it looks like patch 265488 is going to close/fix lp bug #1532471 ? | 21:45 |
| openstack | Launchpad bug 1532471 in OpenStack Object Storage (swift) "invalid x-timestamp causes 500" [Undecided,Confirmed] https://launchpad.net/bugs/1532471 | 21:45 |
| patchbot | clayg: https://review.openstack.org/#/c/265488/ - Validate X-Timestamps | 21:45 |
| timburke | clayg: bah, I knew I forgot something. will add the closes-bug: | 21:46 |
| clayg | timburke: I'm told I can fix these sorts of things in gerrit but it cases some carzy rebasy magic I don't unerstand :\ | 21:46 |
| andrew__ | Sorry my SAIO issue was due to memcache not running, its resolved now | 21:46 |
| clayg | ahale: ah yeah, I also would have been ok with defrag - keep ordering pack down update replica2part2dev :\ | 21:46 |
| clayg | andrew__: sorry bro didn't see your message! good on you getting it fixed! | 21:47 |
| clayg | LOL @ torgomatic_ | 21:48 |
| clayg | torgomatic_: thank you :P | 21:49 |
| torgomatic_ | clayg: you run that back through google translate? ;) | 21:49 |
| clayg | torgomatic_: of course :) couldn't wait. i knew i was in for a laugh | 21:49 |
| *** diazjf has quit IRC | 21:51 | |
| *** dmorita has joined #openstack-swift | 21:52 | |
| openstackgerrit | Alan Erwin proposed openstack/swift: Refactoring the expiring objects feature https://review.openstack.org/252085 | 21:54 |
| *** garthb__ has joined #openstack-swift | 22:01 | |
| *** garthb_ has quit IRC | 22:03 | |
| -openstackstatus- NOTICE: Gerrit is restarting to resolve java memory issues | 22:06 | |
| *** petertr7 is now known as petertr7_away | 22:14 | |
| *** diazjf has joined #openstack-swift | 22:14 | |
| timur | does that mean clayg went shopping? :) | 22:15 |
| clayg | timur: I might go buy you a beer if you get ipv6 support fixed in memcache! | 22:22 |
| timur | hahaha | 22:23 |
| *** dmorita has quit IRC | 22:24 | |
| *** mingdang1 has joined #openstack-swift | 22:28 | |
| jrichli | andrew__: are you using tempauth? did you supply the token (X-Auth-Token) on the request? If you did, check the logs for errors. | 22:29 |
| clayg | timburke: accidently looked at patch 263259 - I hadn't previously understood how psudeo-folder "support" in container names "worked" :\ | 22:29 |
| patchbot | clayg: https://review.openstack.org/#/c/263259/ - Fix upload to pseudo-dir passed by <container> arg | 22:29 |
| *** tongli has quit IRC | 22:31 | |
| *** tongli has joined #openstack-swift | 22:31 | |
| clayg | timburke: i'm sorta freaking out that the service is sending the "container" name with a '/' down to the low level client :\ | 22:32 |
| *** diazjf has quit IRC | 22:32 | |
| clayg | like put_object(container='some/crazy", obj='thing') ends up just being AUTH_XXX/some/crazy/thing like it's no big deal? | 22:33 |
| clayg | is it no big deal? the path is correct in the end - the container is 'some' not 'some/crazy'? | 22:33 |
| *** dmorita has joined #openstack-swift | 22:35 | |
| *** dmorita has quit IRC | 22:35 | |
| timburke | clayg: yeah, that whole thing's weird. i'm not sure i like it either, but (1) things are certainly less broken with that patch and (2) i have no idea whether there are people relying on the existing behavior. if it weren't for (2) i'd consider just ripping it out, which was the other possibility James put forward in bug 1478210 | 22:35 |
| openstack | bug 1478210 in python-swiftclient "swift client psudeo-folder upload syntax can't create the container" [Undecided,In progress] https://launchpad.net/bugs/1478210 - Assigned to James Nzomo (moje) | 22:35 |
| *** tongli has quit IRC | 22:35 | |
| clayg | timburke: sheet | 22:36 |
| *** dmorita has joined #openstack-swift | 22:39 | |
| clayg | timburke: ok, I think after merging patch 263259 it would be trivial to fix the segment container naming to use the container_name value; and that's the only new/different brokeness I'm aware of so far | 22:40 |
| patchbot | clayg: https://review.openstack.org/#/c/263259/ - Fix upload to pseudo-dir passed by <container> arg | 22:40 |
| clayg | timburke: I'll probably also file a bug that the low level client put_object should require the container argument have no slashes? Which is probably debatable; but I think in at least our usage of it we'd be better off with the more disciplined interface forcing to fix psudeo-folders on the command line before taking the arguments into the lower level interfaces? | 22:42 |
| timburke | ...probably. still something where we'll need to worry about potentially breaking glance or horizon or shade or os-loganalyze or whoever. maybe better left for an eventual 3.0 | 22:44 |
| mattoliverau | I've been looking at some new swift screen designs for Horizon folk, and psuedo folders is definitely confusing those folks (but why can't you just use folders, why can't we just change psudo folders name, etc.), tho I have told them about how timburke has added more smarts to swiftclient by looking at the /info of the cluster it points at. So they might do something similar.. or steal some ideas :) | 22:44 |
| *** dmorita has quit IRC | 22:45 | |
| notmyname | mattoliverau: heh. I asked them about what they were working on, and I got added to 60 or 70 different projects. that wasn't fun. (also, I haven't looked at it) | 22:46 |
| andrew__ | jrichli: I resolved the issue, it was due to memcache not running. Now im facing "ImportError: No module named pbr.version" when trying to run the functional tests. Im going through the logs to try to find any clues | 22:46 |
| mattoliverau | notmyname: I have been dragged in, so I have a little.. your welcome | 22:46 |
| jrichli | andrew__: sounds like you are using an older version of pbr than what is required | 22:47 |
| mattoliverau | notmyname: richardjones is now a horizon core, so I'm now voluntold to look | 22:47 |
| notmyname | yay | 22:47 |
| mattoliverau | They are thinking about what kind of options would people like in a UI, and they may want a rename/move. So asked breifly about adding a move (COPY + DELETE) to the bulk middleware. But I haven't thought about it enough. | 22:49 |
| andrew__ | jrichli: is there a way to check the version or update pbr? | 22:49 |
| jlhinson | andrew__: you can use "pip freeze | grep pbr" to check your version, assuming it's installed through pip | 22:53 |
| andrew__ | jlhinson: Its not returning a version and I can't find it on pip list so I'm assuming its not installed through pip | 22:57 |
| mattoliverau | andrew__: how are you running the tests? | 23:00 |
| * mattoliverau is getting pulled into another meeting bbs | 23:01 | |
| andrew__ | mattoliverau: per SAIO instructions: $HOME/swift/.functests | 23:01 |
| *** km_ has joined #openstack-swift | 23:03 | |
| *** blmartin has quit IRC | 23:04 | |
| *** bjkeller1 has left #openstack-swift | 23:05 | |
| clayg | timburke: that was fun -> lp bug #1532991 (but good point on ecosystem projects, at least for those we have the gate - could log a warning and run tests with that weird MAKE_WARNINGS_ERRORS python env var?) | 23:06 |
| openstack | Launchpad bug 1532991 in python-swiftclient "low level client functions should validate against pseudo-folder in container argument" [Undecided,New] https://launchpad.net/bugs/1532991 | 23:06 |
| *** asettle has joined #openstack-swift | 23:07 | |
| *** badari has joined #openstack-swift | 23:09 | |
| clayg | andrew__: pip install --upgrade pip; apt-get purge python-setuptools python-pip; cd ~/swift; pip install --upgrade -e . | 23:10 |
| clayg | andrew__: stick a few sudo's in there and hope you don't hoze your entire machine and you should be good to go! | 23:10 |
| *** sgundur has left #openstack-swift | 23:14 | |
| *** dmorita has joined #openstack-swift | 23:14 | |
| *** dmorita has quit IRC | 23:16 | |
| *** mingdang1 has quit IRC | 23:17 | |
| *** minwoob has quit IRC | 23:19 | |
| *** nadeem has quit IRC | 23:20 | |
| andrew__ | clayg: ill try this, thanks | 23:23 |
| clayg | andrew__: nice - good luck! | 23:25 |
| *** kei_yama has joined #openstack-swift | 23:27 | |
| *** andrew__ has quit IRC | 23:30 | |
| *** xzvk has quit IRC | 23:31 | |
| *** MVenesio has joined #openstack-swift | 23:37 | |
| *** dmorita has joined #openstack-swift | 23:37 | |
| *** dmorita has quit IRC | 23:38 | |
| *** mragupat has quit IRC | 23:38 | |
| *** resker has quit IRC | 23:38 | |
| openstackgerrit | Tim Burke proposed openstack/python-swiftclient: _RetryBody doesn't need to take explicit etag/content-length https://review.openstack.org/265417 | 23:38 |
| *** dmorita has joined #openstack-swift | 23:38 | |
| *** dmorita has quit IRC | 23:39 | |
| clayg | timburke: jesus - you're so prolific! | 23:39 |
| clayg | timburke: I thought I'd try to beat on the review queue "a little" today and it's all timburke all the time! | 23:39 |
| *** alejandrito has quit IRC | 23:40 | |
| timburke | bugs... bugs everywhere... | 23:40 |
| clayg | for sure :'( | 23:40 |
| clayg | timburke: you're doing a great job trying to squash them tho | 23:41 |
| timburke | gotta earn that title on my business card :) | 23:41 |
| clayg | timburke: you're cirtainly making a dent in the incoming bug stream by keeping me doing reviews instead of writing new ones! | 23:41 |
| timburke | fun fact: i don't think object-auditor works with EC | 23:41 |
| clayg | heh | 23:41 |
| timburke | (speaking of bugs) | 23:41 |
| clayg | timburke: EAT THAT AUDITOR! | 23:41 |
| *** MVenesio has quit IRC | 23:42 | |
| timburke | Invalid Timestamp value in filename '1452548640.28136#2.data' -- it doesn't know how :( | 23:42 |
| clayg | timburke: *hopefully* it's checking the objects real (stored) etag and not the X-Object-Sysmeta-EC-Etag or whatever it's called? | 23:42 |
| clayg | timburke: oh how interesting... | 23:42 |
| timburke | https://github.com/openstack/swift/blob/master/swift/obj/auditor.py#L42 -- no DiskFileRouter | 23:43 |
| clayg | timburke: I guess I would have hoped some "from_audit_location" method somewhere on the ECDiskFileManager knows wtf it's doing? | 23:43 |
| clayg | timburke: so it's just a world of ERROR trying to audit with a traceback pointing at DiskFileManager.get_diskfile_from_audit_location? | 23:45 |
| clayg | timburke: cause that'd be *awesome* | 23:45 |
| *** dimasot has quit IRC | 23:46 | |
| *** xzvk has joined #openstack-swift | 23:47 | |
| timburke | clayg: no traceback, but it came from https://github.com/openstack/swift/blob/master/swift/obj/diskfile.py#L2080 (DiskFileManager.parse_on_disk_filename) | 23:47 |
| openstackgerrit | Tim Burke proposed openstack/swift: Validate X-Timestamps https://review.openstack.org/265488 | 23:50 |
| *** ho has joined #openstack-swift | 23:54 | |
| *** ho is now known as Guest62293 | 23:54 | |
| Guest62293 | good morning! | 23:56 |
| *** Guest62293 is now known as ho_ | 23:56 | |
| *** dmorita has joined #openstack-swift | 23:58 | |
| openstackgerrit | Merged openstack/python-swiftclient: Fix upload to pseudo-dir passed by <container> arg https://review.openstack.org/263259 | 23:58 |
| *** mragupat has joined #openstack-swift | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!