| *** rcernin has quit IRC | 00:42 | |
| *** catinthe_ has quit IRC | 01:16 | |
| *** catintheroof has joined #openstack-swift | 01:17 | |
| *** catintheroof has quit IRC | 01:21 | |
| *** davideagle has quit IRC | 01:25 | |
| *** jamielennox is now known as jamielennox|away | 01:26 | |
| kota_ | good morning | 01:26 |
|---|---|---|
| kota_ | it looks like, that was really quiet weekend due to, maybe, thanks giving. | 01:27 |
| *** siva_krish has joined #openstack-swift | 01:33 | |
| mattoliverau | kota_: morning, yeah pretty quiet | 01:48 |
| kota_ | mattolvierau: o/ | 01:49 |
| kota_ | mattoliverau:^^ | 01:49 |
| *** btorch has left #openstack-swift | 02:00 | |
| *** siva_krish has quit IRC | 02:11 | |
| *** jamielennox|away is now known as jamielennox | 02:15 | |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/pyeclib: Add soft warning log line when using liberasurecode <1.3.1 https://review.openstack.org/398744 | 02:22 |
| kota_ | yeah, quiet environment makes our gate jobs to run so quick | 02:24 |
| *** Jeffrey4l has joined #openstack-swift | 02:28 | |
| *** cmd-t has joined #openstack-swift | 02:59 | |
| cmd-t | Hi all | 02:59 |
| cmd-t | Can anyone help with swift query option | 03:00 |
| *** cmd-t has quit IRC | 03:03 | |
| *** kei_yama has quit IRC | 03:12 | |
| *** psachin has joined #openstack-swift | 04:10 | |
| mattoliverau | cschwede: we can try, whats up? | 04:10 |
| mattoliverau | cschwede: sorry, tab complete error, seems like cmd-t has gone :) | 04:10 |
| *** links has joined #openstack-swift | 04:13 | |
| *** SkyRocknRoll has joined #openstack-swift | 04:51 | |
| *** ppai has joined #openstack-swift | 05:26 | |
| *** PavelK_ has joined #openstack-swift | 06:10 | |
| *** PavelK_ has quit IRC | 06:10 | |
| *** PavelK_ has joined #openstack-swift | 06:11 | |
| *** tdasilva has quit IRC | 06:35 | |
| *** sams-gleb has joined #openstack-swift | 06:46 | |
| *** SkyRocknRoll has quit IRC | 07:01 | |
| *** SkyRocknRoll has joined #openstack-swift | 07:05 | |
| *** silor has joined #openstack-swift | 07:14 | |
| *** tdasilva has joined #openstack-swift | 07:14 | |
| *** ChubYann has quit IRC | 07:19 | |
| *** silor1 has joined #openstack-swift | 07:23 | |
| *** silor has quit IRC | 07:25 | |
| *** silor1 is now known as silor | 07:25 | |
| *** manous has quit IRC | 07:29 | |
| *** Jeffrey4l has quit IRC | 07:35 | |
| *** Jeffrey4l has joined #openstack-swift | 07:43 | |
| *** rcernin has joined #openstack-swift | 07:44 | |
| *** pcaruana has joined #openstack-swift | 07:45 | |
| *** manous has joined #openstack-swift | 07:46 | |
| *** sams-gleb has quit IRC | 07:49 | |
| *** oshritf has joined #openstack-swift | 08:05 | |
| *** geaaru has joined #openstack-swift | 08:25 | |
| *** hseipp has joined #openstack-swift | 08:26 | |
| *** rledisez has joined #openstack-swift | 08:27 | |
| *** bkopilov has quit IRC | 08:30 | |
| *** bkopilov has joined #openstack-swift | 08:32 | |
| *** manous has quit IRC | 08:48 | |
| *** manous has joined #openstack-swift | 08:49 | |
| *** amoralej|off is now known as amoralej | 08:52 | |
| *** ppai has quit IRC | 08:53 | |
| *** oshritf has quit IRC | 08:58 | |
| *** oshritf has joined #openstack-swift | 08:59 | |
| *** ppai has joined #openstack-swift | 09:06 | |
| *** acoles_ is now known as acoles | 09:16 | |
| *** PetrKubena has left #openstack-swift | 09:25 | |
| *** PetrKubena has joined #openstack-swift | 09:29 | |
| *** oshritf has quit IRC | 09:38 | |
| onovy | cschwede: hi :) | 09:41 |
| onovy | i'm not saying your patch is bad, but i think there is better solution without list | 09:41 |
| *** Worry has joined #openstack-swift | 09:42 | |
| onovy | but i don't want to waste Pavel's time. so if you say: Pavel solution is wrong, let's abandon it and have your solution | 09:42 |
| *** donagh has joined #openstack-swift | 09:45 | |
| *** Worry has quit IRC | 09:48 | |
| *** Worry has joined #openstack-swift | 09:49 | |
| *** NeverWorry has joined #openstack-swift | 09:54 | |
| *** EverNine has joined #openstack-swift | 09:55 | |
| *** Worry has quit IRC | 09:56 | |
| *** asettle has joined #openstack-swift | 09:56 | |
| *** NeverWorry has quit IRC | 09:58 | |
| *** abalfour has quit IRC | 09:58 | |
| *** abalfour has joined #openstack-swift | 09:59 | |
| *** cbartz has joined #openstack-swift | 10:01 | |
| *** gabor_antal has joined #openstack-swift | 10:02 | |
| *** oshritf has joined #openstack-swift | 10:02 | |
| *** SkyRocknRoll has quit IRC | 10:03 | |
| *** SkyRocknRoll has joined #openstack-swift | 10:03 | |
| *** EverNine has quit IRC | 10:04 | |
| *** SkyRocknRoll has quit IRC | 10:05 | |
| *** gabor_antal_ has quit IRC | 10:05 | |
| *** SkyRocknRoll has joined #openstack-swift | 10:05 | |
| *** manous has quit IRC | 10:06 | |
| *** stevemar has quit IRC | 10:19 | |
| *** sams-gleb has joined #openstack-swift | 10:22 | |
| cschwede | onovy: Hi! No, I’m not saying PavelK_ patch is bad, actually it’s a nice idea! But there is an error in it, and I think it’s even more simple. Also tests are missing. Let me try something, if that works i update my patch and include PavelK_ as Co-Author, if that’s ok for you | 10:31 |
| *** Worry has joined #openstack-swift | 10:33 | |
| onovy | cschwede: or Pavel can fix his patch. But we are fine with update of your patch too :)) | 10:33 |
| openstackgerrit | Charles Hsu proposed openstack/python-swiftclient: Apply headers to SLO/DLO segments. https://review.openstack.org/403610 | 10:38 |
| *** Worry has quit IRC | 10:51 | |
| *** kragniz has quit IRC | 10:54 | |
| *** kragniz has joined #openstack-swift | 10:54 | |
| *** Worry has joined #openstack-swift | 11:10 | |
| *** acoles is now known as acoles_ | 11:10 | |
| cschwede | onovy: PavelK_: added my comments to https://review.openstack.org/#/c/402324/2 | 11:13 |
| patchbot | patch 402324 - swift - Fix missing suffixes during replication | 11:13 |
| cschwede | onovy: PavelK_ I have a local patch that I could squash into your patch, if you’re ok with it | 11:14 |
| *** ppai has quit IRC | 11:14 | |
| openstackgerrit | Donagh McCabe proposed openstack/swift: Add python3-dev to bindep and use py27for some envs https://review.openstack.org/395049 | 11:14 |
| *** ppai has joined #openstack-swift | 11:17 | |
| *** Worry has quit IRC | 11:21 | |
| PavelK_ | cschwede: Sure, I'm OK with it | 11:31 |
| *** ppai has quit IRC | 11:31 | |
| *** acoles_ is now known as acoles | 11:33 | |
| PavelK_ | Yesterday I saw that you work on a patch so I cleaned some unrelated code from my first commit and left my solution because yours looks fine... | 11:34 |
| PavelK_ | So if you have patch prepared, please update 402043. Or leave me a comment what should be better and I can complete tests. | 11:36 |
| *** davideagle has joined #openstack-swift | 11:41 | |
| openstackgerrit | Charles Hsu proposed openstack/python-swiftclient: Apply headers to SLO/DLO segments. https://review.openstack.org/403610 | 11:42 |
| *** derekjhyang has joined #openstack-swift | 11:43 | |
| *** ppai has joined #openstack-swift | 11:44 | |
| *** SkyRocknRoll has quit IRC | 12:01 | |
| openstackgerrit | Christian Schwede proposed openstack/swift: Fix non-deterministic suffix updates in hashes.pkl https://review.openstack.org/402324 | 12:05 |
| cschwede | PavelK_: onovy ^^ | 12:05 |
| onovy | cschwede: self.replication_phase = 0 // this will not work for run_once=True | 12:10 |
| *** Worry has joined #openstack-swift | 12:29 | |
| *** Worry has quit IRC | 12:41 | |
| *** amoralej is now known as amoralej|off | 12:46 | |
| *** amoralej|off is now known as amoralej|lunch | 12:46 | |
| *** silor has quit IRC | 13:01 | |
| *** vint_bra has joined #openstack-swift | 13:06 | |
| asettle | Hey guys - this is a bit of a stab in the dark, but is there an Alexis Lee in this channel? | 13:16 |
| asettle | Also, cschwede just wanted to see if you had time to check this one again: https://review.openstack.org/#/c/398814/ | 13:17 |
| patchbot | patch 398814 - openstack-manuals - Fix Swift statsd sample rate notes | 13:17 |
| *** _JZ_ has joined #openstack-swift | 13:18 | |
| *** ppai has quit IRC | 13:24 | |
| *** xionchen_ has joined #openstack-swift | 13:43 | |
| openstackgerrit | Pavel Kvasnička proposed openstack/swift: Fix non-deterministic suffix updates in hashes.pkl https://review.openstack.org/402324 | 13:45 |
| openstackgerrit | Pavel Kvasnička proposed openstack/swift: Fix non-deterministic suffix updates in hashes.pkl https://review.openstack.org/402324 | 13:46 |
| cschwede | PavelK_: your last changes are not explicitly, they might pass even if there is a failure | 13:49 |
| *** amoralej|lunch is now known as amoralej | 13:50 | |
| PavelK_ | cschwede: I know, please set replication_phase = 0 before test and ignore my "patch" | 13:53 |
| cschwede | asettle: Hey Alexandra! Sorry, I just had a quick look but no time yet to submit another patch - will do that soon. Thanks a lot for your help, truely appreciated! | 13:53 |
| cschwede | PavelK_: ok | 13:53 |
| asettle | cschwede: no problem! I'm doing a bit of mass cleanup through the patches so I'm being a bit pushy lately :P hopefully my responses inline helped in someway, it is a pretty weird situation. | 13:53 |
| openstackgerrit | Christian Schwede proposed openstack/swift: Fix non-deterministic suffix updates in hashes.pkl https://review.openstack.org/402324 | 13:53 |
| cschwede | asettle: yeah; honestly i should have read the note at the top of that file… | 13:54 |
| asettle | Haha cschwede I wouldn't sweat that, nobody does ;) | 13:57 |
| asettle | I am very good at typing "this is an autogen..." :p | 13:57 |
| StevenK | Note to self: Next time asettle sends out an automated message, question it | 13:58 |
| * StevenK hides from asettle | 13:58 | |
| asettle | StevenK: hahaha I CHASE YOU DOWN | 13:58 |
| asettle | I've become awkwardly good at finding people in channel | 13:58 |
| StevenK | There's this whole /names list and everything | 13:59 |
| *** Worry has joined #openstack-swift | 14:00 | |
| asettle | I know! | 14:00 |
| asettle | Crazy stuff ;) | 14:00 |
| *** links has quit IRC | 14:04 | |
| *** PavelK_ has quit IRC | 14:08 | |
| acoles | asettle: Alexis == lxsli | 14:15 |
| *** Worry has quit IRC | 14:15 | |
| asettle | acoles: you know all | 14:16 |
| asettle | Is lxsli around today? :) | 14:16 |
| acoles | that I do not know ;) | 14:16 |
| *** Worry has joined #openstack-swift | 14:16 | |
| asettle | disppoint! :p | 14:16 |
| lxsli | Yo | 14:18 |
| lxsli | asettle: o/ | 14:18 |
| asettle | HELLO! | 14:20 |
| asettle | lxsli: I've been poking around manuals stuff all week removing technical debt etc, and I happened along yours: https://review.openstack.org/#/c/398424/ | 14:21 |
| patchbot | patch 398424 - openstack-manuals - WIP first stab at generating Swift config tables | 14:21 |
| lxsli | asettle: for future reference, googling "launchpad <name>" often works :) | 14:21 |
| asettle | Good to know ;) | 14:21 |
| asettle | It's along a similar vain to the patch that cschwede is working on (at least, won't cause a merge conflict, but is also an auto-gen run for config tables) | 14:22 |
| lxsli | 398814? | 14:22 |
| asettle | Yis | 14:23 |
| asettle | That's the ticket | 14:23 |
| lxsli | asettle: the hitch is that autohelp.py doesn't work the same for Swift as it does the other projects | 14:23 |
| asettle | Yeah I noticed | 14:23 |
| asettle | *head desk* | 14:23 |
| lxsli | I've a good start on converting autohelp.py to use Stevedore + oslo.config but it's not really part of my job so it might take a little while | 14:24 |
| lxsli | plus it's quite a major change which is always scary from a new-to-the-project contributor | 14:24 |
| asettle | Understandable :) well, if you need anything, let us know. But we'll do our best to support any big changes. | 14:25 |
| asettle | Promise nothing bad will happen :p | 14:25 |
| lxsli | That's on behalf of the openstack-doc-tools team? | 14:25 |
| lxsli | Thanks :) | 14:25 |
| asettle | My overarching statement? | 14:26 |
| lxsli | "if you need anything, let us know" - we're in the swift channel atm so just checking | 14:27 |
| asettle | Oh, ha. #openstack-doc | 14:27 |
| asettle | :) | 14:27 |
| *** rcernin has quit IRC | 14:27 | |
| asettle | Head in the clouds. | 14:27 |
| lxsli | badum tish | 14:27 |
| asettle | Thank you everyone, here all week. | 14:27 |
| *** openstackstatus has quit IRC | 14:28 | |
| lxsli | asettle: so re: 398424, it's not dead only sleeping, can I leave it open for now? it has -1 jenkins so no one should be looking at it anyway | 14:28 |
| asettle | Except for this nosy ass, hahahah yeah, if you could just put a note that it's chilling that would be awesome :) | 14:29 |
| lxsli | Sure thing | 14:29 |
| asettle | We get a lot of 'sleeping' patches in docs that never quite wake up, so we tend to do a massive cleanup a few times a year to remove things that people will never touch again. | 14:29 |
| asettle | Thanks for getting back in touch :) | 14:30 |
| *** openstackstatus has joined #openstack-swift | 14:30 | |
| *** ChanServ sets mode: +v openstackstatus | 14:30 | |
| lxsli | No worries | 14:30 |
| *** rcernin has joined #openstack-swift | 14:36 | |
| *** sams-gleb has quit IRC | 15:00 | |
| *** sams-gleb has joined #openstack-swift | 15:00 | |
| *** psachin has quit IRC | 15:05 | |
| *** sams-gleb has quit IRC | 15:05 | |
| *** Worry has quit IRC | 15:09 | |
| *** daemontool has joined #openstack-swift | 15:10 | |
| *** sams-gleb has joined #openstack-swift | 15:17 | |
| *** cbartz has left #openstack-swift | 15:40 | |
| openstackgerrit | Mathias Bjoerkqvist proposed openstack/swift: Storing encryption root secret in Barbican https://review.openstack.org/364878 | 15:45 |
| *** arch-nemesis has joined #openstack-swift | 15:54 | |
| *** derekjhyang has quit IRC | 16:01 | |
| *** rledisez has quit IRC | 16:01 | |
| *** mvk has quit IRC | 16:01 | |
| *** PetrKubena has quit IRC | 16:01 | |
| *** tmoreira has quit IRC | 16:01 | |
| *** david-lyle has quit IRC | 16:01 | |
| *** janonymous has quit IRC | 16:01 | |
| *** pdardeau has quit IRC | 16:01 | |
| *** notmyname has quit IRC | 16:01 | |
| *** cebreidian has quit IRC | 16:01 | |
| *** zigo has quit IRC | 16:01 | |
| *** charz_ has quit IRC | 16:01 | |
| *** zigo has joined #openstack-swift | 16:01 | |
| *** PetrKubena has joined #openstack-swift | 16:01 | |
| *** pdardeau has joined #openstack-swift | 16:02 | |
| *** tmoreira has joined #openstack-swift | 16:02 | |
| *** mvk has joined #openstack-swift | 16:02 | |
| *** david-lyle has joined #openstack-swift | 16:02 | |
| *** charz has joined #openstack-swift | 16:02 | |
| *** rledisez has joined #openstack-swift | 16:02 | |
| *** notmyname has joined #openstack-swift | 16:03 | |
| *** ChanServ sets mode: +v notmyname | 16:03 | |
| *** cebreidian has joined #openstack-swift | 16:03 | |
| *** janonymous has joined #openstack-swift | 16:06 | |
| *** mvk has quit IRC | 16:17 | |
| openstackgerrit | Christian Schwede proposed openstack/swift: Fix non-deterministic suffix updates in hashes.pkl https://review.openstack.org/402324 | 16:23 |
| *** rcernin has quit IRC | 16:27 | |
| notmyname | good morning. time to catch up on a lot of scrollback and a whole lot of emails | 16:41 |
| acoles | notmyname: welcome back | 16:43 |
| notmyname | my holiday break: https://www.youtube.com/watch?v=LIln7Z1iInA | 16:44 |
| clayg | good morning | 16:44 |
| notmyname | clayg: welcome back! (did you see peluse last week?) | 16:44 |
| acoles | notmyname: sounds perfect | 16:46 |
| clayg | notmyname: no he was in florida | 16:47 |
| notmyname | it was everything I thought it could be | 16:47 |
| notmyname | clayg: AZ wasn't warm/sunny enough? | 16:48 |
| clayg | notmyname: who knows | 16:49 |
| notmyname | clayg: I hope your trip was good | 16:49 |
| acoles | clayg: welcome back too | 16:50 |
| *** cschwede is now known as cschwede|mtg | 16:52 | |
| notmyname | acoles: I was about to make a joke about what landed in swift, but I checked the dashboard and I see that there's a lot of stuff waiting on a final +2. clearly you were busy last week | 16:52 |
| acoles | notmyname: IDK about +2's, there was plenty of activity on https://bugs.launchpad.net/swift/+bug/1634967 | 16:53 |
| openstack | Launchpad bug 1634967 in OpenStack Object Storage (swift) "object-replicator in 2.7.0 and later doesn't replicate all suffixes" [Critical,Fix released] | 16:53 |
| acoles | fix merged on master, backports proposed and cschwede|mtg is working on a follow up | 16:54 |
| clayg | it was good - my cousins on my dad's side are older, so their kids are almost grown now (most in late-highschool/college) - it was great to catch up and drink bud light around the fire on the chicken farm | 16:54 |
| clayg | the funniest part was coming home - and my friends are like "what it's *LIKE* there!?" - "umm... still the same" | 16:55 |
| *** tqtran has joined #openstack-swift | 16:59 | |
| notmyname | acoles: cschwede|mtg: what's the follow-up do? | 17:01 |
| cschwede|mtg | notmyname: it’s fixing another serious issue: patch 402324 | 17:01 |
| patchbot | https://review.openstack.org/#/c/402324/ - swift - Fix non-deterministic suffix updates in hashes.pkl | 17:02 |
| cschwede|mtg | notmyname: launchpad 1644807 | 17:02 |
| openstack | Launchpad bug 1644807 in OpenStack Object Storage (swift) "hashes.pkl suffix update is not deterministic and might be never triggered" [High,In progress] https://launchpad.net/bugs/1644807 - Assigned to Christian Schwede (cschwede) | 17:02 |
| acoles | notmyname: makes the replicator do a listdir on suffixes more predictably | 17:02 |
| cschwede|mtg | https://bugs.launchpad.net/swift/+bug/1644807 | 17:02 |
| *** cschwede|mtg is now known as cschwede | 17:02 | |
| *** PetrKubena has left #openstack-swift | 17:02 | |
| cschwede | notmyname: if you’re unlucky, you have suffixes that won’t replicate for a long time - only after the 50-100 replication cycle this gets fixed | 17:03 |
| *** daemontool has quit IRC | 17:04 | |
| notmyname | TIL https://en.wikipedia.org/wiki/Coupon_collector's_problem | 17:05 |
| notmyname | I'll prioritize looking at that today | 17:05 |
| *** winggundamth_ has joined #openstack-swift | 17:15 | |
| notmyname | https://review.openstack.org/#/c/401225/ and https://review.openstack.org/#/c/401226/ are TC patches that we should pay attention to | 17:18 |
| patchbot | patch 401225 - governance - Add a draft TC vision structure | 17:18 |
| patchbot | patch 401226 - governance - Add a draft OpenStack technical vision | 17:18 |
| *** pcaruana has quit IRC | 17:19 | |
| *** oshritf has quit IRC | 17:20 | |
| *** klrmn has joined #openstack-swift | 17:25 | |
| *** hseipp has quit IRC | 17:26 | |
| notmyname | emails successfully marked as "read" | 17:28 |
| sweeper | notmyname: that 'select all unread in inbox' feature gmail added is really handy isn't it? >.> | 17:33 |
| *** xionchen_ has quit IRC | 17:34 | |
| *** rledisez has quit IRC | 17:38 | |
| onovy | clayg: hi, https://launchpadlibrarian.net/295278975/rsync-after-upgrade.png | 17:41 |
| onovy | fyi :) | 17:41 |
| onovy | and morning to everyone else | 17:42 |
| clayg | onovy: congras! | 17:43 |
| clayg | quite an endeavour - well earned | 17:43 |
| onovy | Pavel found it and fixed it, cschwede and acoles helped a lot with patch. so congrats to them :) | 17:44 |
| clayg | yes, too all - go go team swift! | 17:44 |
| onovy | :) | 17:45 |
| sweeper | so I'm a noob here....this patch makes replication faster? | 17:46 |
| onovy | sweeper: this patch prevents data loss | 17:52 |
| onovy | there is bug which can prevent replicate some suffixes from partitions | 17:52 |
| onovy | bug is there from mitaka release | 17:53 |
| clayg | onovy: i'm still having problems following the relation to the rsync metric that tipped you off - did you get the entire cluster upgraded to liberty before applying pavel's fix? | 18:00 |
| sweeper | onovy: ah, that's a good thing :) | 18:01 |
| *** dmorita has joined #openstack-swift | 18:02 | |
| *** davideagle has quit IRC | 18:03 | |
| *** catintheroof has joined #openstack-swift | 18:04 | |
| *** mweshi has joined #openstack-swift | 18:06 | |
| *** winggundamth_ has quit IRC | 18:16 | |
| *** mweshi has quit IRC | 18:16 | |
| *** ChubYann has joined #openstack-swift | 18:31 | |
| *** chlong has joined #openstack-swift | 18:42 | |
| notmyname | http://lists.openstack.org/pipermail/openstack-dev/2016-November/108074.html <-- changes proposed related to driver code in openstack projects. this doesn't affect swift too much today, but nearly any of the proposed options (all trying to answer "what do to about vendor-specific drivers?") will affect most openstack projects. it's likely good to have a cursory understanding of what's going | 18:43 |
| notmyname | on | 18:43 |
| jrichli | notmyname: thanks for highlighting the TC and mailing list items | 18:52 |
| onovy | clayg: yep | 18:52 |
| onovy | to mitaka | 18:53 |
| onovy | liberty was fine | 18:53 |
| notmyname | jrichli: I'm your secondary mail filter | 18:53 |
| onovy | clayg: i think this is explanation. Have suffix inside partition, which is not inside hashes.pkl | 18:54 |
| onovy | clayg: replicator every 10th run do "listdir". He found this new suffix, because this suffix is not at destination, call rsycn | 18:55 |
| onovy | clayg: cluster is synced -> nothing happens | 18:55 |
| onovy | clayg: then destition recounts that suffix -> fixed | 18:55 |
| onovy | this happens every write to new suffix | 18:56 |
| *** TheShaman has joined #openstack-swift | 19:01 | |
| TheShaman | hello, i cant seem to get the s3 swift interface installed, is this the right channel? | 19:01 |
| jrichli | TheShaman: #swift3 is the best channel for that, although some of those folks are here too | 19:04 |
| glange | you won't be able to install that on this channel | 19:04 |
| *** TheShaman has quit IRC | 19:06 | |
| *** geaaru has quit IRC | 19:07 | |
| *** amoralej is now known as amoralej|off | 19:11 | |
| *** clu_ has joined #openstack-swift | 19:12 | |
| notmyname | glange: ? | 19:17 |
| glange | it was a joke - you can't install software on an irc channel | 19:17 |
| glange | sorry :) | 19:17 |
| notmyname | no worries. that's how I read it :-) | 19:18 |
| tdasilva | hehe | 19:18 |
| glange | hah | 19:18 |
| *** nikivi has joined #openstack-swift | 19:45 | |
| *** nikivi has quit IRC | 19:53 | |
| openstackgerrit | Paul Dardeau proposed openstack/swift: CompositeRing Support https://review.openstack.org/271920 | 19:57 |
| clayg | i don't think we ever had a good test for invalidation of a new suffix in part with an existing hashes pickle :\ | 19:58 |
| *** chlong has quit IRC | 20:02 | |
| *** nikivi has joined #openstack-swift | 20:14 | |
| *** chlong has joined #openstack-swift | 20:17 | |
| *** nikivi has joined #openstack-swift | 20:19 | |
| *** acoles is now known as acoles_ | 20:38 | |
| cschwede | clayg: i’m trying to fix that. already working on that, but any ideas welcome! | 20:48 |
| clayg | cschwede: well, i'm mostly trying to see how any test in patch 286618 could have caught this - and mostly happy with what went in from patch 401330 | 20:53 |
| patchbot | https://review.openstack.org/#/c/286618/ - swift - Faster suffix invalidations on object PUT/DELETE (MERGED) | 20:54 |
| patchbot | https://review.openstack.org/#/c/401330/ - swift - Fixed regression in consolidate_hashes (MERGED) | 20:54 |
| onovy | clayg: maybe we should invalidate random part of hashes every replicator run to mitigate bugs like this | 21:01 |
| clayg | i was sorta thinking i might should have noticed this when reviewing patch 346865 - but hash invalidation from a reclaimed tombstone never creates a new suffix :'( | 21:01 |
| patchbot | https://review.openstack.org/#/c/346865/ - swift - Delete old tombstones (MERGED) | 21:01 |
| onovy | For example how can I be sure if my whole cluster is in sync, if any of my hashes can be desynced | 21:02 |
| clayg | onovy: i'm not quite sure I follow - you mean like if change something under a suffix and don't invalidate that suffix? | 21:03 |
| onovy | Yep | 21:03 |
| onovy | I think this can occur | 21:04 |
| onovy | Code bug, xfs bug, power outage | 21:04 |
| onovy | Maybe we should invalidate random hashes | 21:04 |
| onovy | Just for sure | 21:04 |
| onovy | We are doing same for new hashes now | 21:05 |
| onovy | Hashes=suffix :-) | 21:05 |
| MooingLemur | Has anyone ever proposed creating a sort of handoff "pool"? The use case I'm thinking is to have a handful of very fast machines with SSD that are dedicated handoff nodes. This would provide for low latency for write affinity, and the nodes would have nothing other than handoff data so that they can replicate very quickly. | 21:07 |
| MooingLemur | replicate/revert | 21:07 |
| clayg | onovy: i'm not sure what we be the appropriate amount of re-hash-calc to introduce to a stable cluster - I think someone at one time had the good idea of adding some timing information to the hashes data file format so that a trip a hash recalc in inacive suffixes after "a while" | 21:08 |
| onovy | clayg: yep, Pavel is thinking about this solution | 21:08 |
| onovy | Recount staled hashes | 21:09 |
| onovy | And Christian had similar idea | 21:09 |
| notmyname | MooingLemur: that's an interesting idea, and it wouldn't be too difficult for you to test it out and get some numbers | 21:10 |
| clayg | onovy: there may be bugs/wishlist/ideas open related along those lines - might be a useful to go searching for them - if for no other reason to mark them done if Pavel ends up doing something | 21:10 |
| onovy | clayg: yep. I will talk about it with Pavel tomorrow | 21:10 |
| clayg | notmyname: it sounds difficult to me? how would you force get_more_nodes to produce those specific devices in the ring? | 21:10 |
| notmyname | MooingLemur: hack up the get_more_nodes() to return a list of your SSDs, then run some benchmarks comparing the two. | 21:10 |
| clayg | notmyname: oh | 21:11 |
| MooingLemur | notmyname: it might be something I'm capable of doing with my current understanding if I understand the code path :) | 21:11 |
| onovy | We will create simple script which will invalidate all hashes for now | 21:11 |
| onovy | Just to check if we have all data... | 21:11 |
| clayg | onovy: just run rm -f /srv/node*/*/object*/hashes.pkl on all nodes? | 21:11 |
| notmyname | yeah, nothing too sophisticated. just enough to get a comparison of numbers. then, of course, there will need to be more testing to figure out where the particular differences come from | 21:11 |
| onovy | clayg: I am bit worried about this | 21:12 |
| clayg | onovy: it's the same thing as invalidating all the suffixes? | 21:12 |
| notmyname | MooingLemur: as in, there's no reason to start changing stuff to make it faster if we don't know exactly what's making it slow today. (and what is fast or slow and what's good enough?) | 21:12 |
| clayg | onovy: you could do a few parts a time - something like 'for phase range in 10; for part in self.parts; if part % phase; continue ; else delete; sleep' or .. whatever | 21:14 |
| clayg | ^ N.B. that's not valid or correct :P | 21:14 |
| *** sams-gleb has quit IRC | 21:14 | |
| onovy | clayg: it's not completly same | 21:14 |
| onovy | if i remove hashes, i force replicator/o-server to count it | 21:14 |
| MooingLemur | notmyname: My observed problem right now isn't necessarily ingest speed, it's the delay between upload and when the object is geographically consistent due to the multi-hour replication sweep. | 21:14 |
| onovy | i can't fine tune speed of recount | 21:15 |
| onovy | if i call recount, i can fine tune speed of recouting | 21:15 |
| onovy | so no invalidating, force-recounting :) | 21:15 |
| *** sams-gleb has joined #openstack-swift | 21:15 | |
| MooingLemur | s/geographically/globally/ | 21:15 |
| clayg | MooingLemur: turn off write affinity - you'll get geographically consistent much quicker | 21:16 |
| *** chlong has quit IRC | 21:16 | |
| notmyname | MooingLemur: ah, ok. | 21:16 |
| onovy | MooingLemur: we are using read affinity only inside geo-cluster. fast read, slower write, but geo-consitent :) | 21:17 |
| onovy | + what is "multi-hour replication sweep"? | 21:18 |
| onovy | i think you can fine tune replicator concurrency to get "multi-minutes replication sweep" | 21:18 |
| clayg | onovy: i don't think i follow beyond you want to control the rate of hash recalculation - I don't think there's much benifit to doing a subset of suffixes w/i a subset of parts rather than doing just a subset of parts - i also do think the the management overhead of trying to spread this out over too long is worth the congnative effort to try and get to clever - unless you're planning on doing this ongoing | 21:18 |
| MooingLemur | onovy: object-replicator on the storage nodes is taking multiple hours to complete a single pass, and right now it's only made worse because we're adding storage in region 2, and not very much is being written directly in region 2 yet. | 21:19 |
| onovy | clayg: so you recommends to just remove hashes.pkl? | 21:19 |
| clayg | ... in which case I think just randomly recalculating a subset of suffixes on an ongoing basis is a bad idea - and effort would be spent to recalculate suffixes that are inactive | 21:19 |
| onovy | MooingLemur: add concurency to replicator? | 21:19 |
| *** sams-gleb has quit IRC | 21:20 | |
| clayg | onovy: you can either remove hashes.pkl in the whole cluster (rip the bandaid off, it doesn't hurt that bad IME) - or you can remove hashes.pkl in subset of parts in a sweep | 21:20 |
| onovy | "in subset" = same partitions on all replicas? | 21:20 |
| clayg | yeah so a script that does like `rm /srv/node*/*/object*/[0-100]/hashes.pkl` then `rm /srv/node*/*/object*/[100-200]/hashes.pkl` etc | 21:22 |
| clayg | i'm sure that'd be good enough | 21:22 |
| MooingLemur | onovy: I tried, it doesn't seem to help beyond a certain point. I think I might be running into some bottleneck with the Python GIL but I haven't proved it except I did try running concurrent one-off replications confined to individual devices which increased throughput. | 21:22 |
| onovy | MooingLemur: ah, right replicator per-device // this is what we need :) | 21:22 |
| clayg | onovy: again, unless your just trying to operationalize some sort of continuous suffix recalculation - rahter than a one time "i'm scared recalc everything" | 21:23 |
| onovy | but we have concurrency=2 and ~30 minutes to check whole cluster, if there is nothing to transfer | 21:23 |
| onovy | clayg: i'm scared, recalc everything :) | 21:23 |
| onovy | MooingLemur: ^ but we have... was to you :) | 21:24 |
| MooingLemur | I've also selected a part power of 22 which may not be helping things. I didn't want to come up short if the cluster expanded to a 5-figure number of disks. We are at mid 4 figure now. | 21:25 |
| clayg | continuous recalculation of inactive suffixes seems like a good and safe practical step in improving the consistence engine - but it should be something like "if a hash has been valid for a suffix for a reclaim age w/o any of the nodes doing a recalc - don't trust it" - ideally it would also log something scary if this "i don't trust this hash" operation resulted in a different result! | 21:25 |
| onovy | MooingLemur: ah, how many partitions do you have per server? | 21:25 |
| clayg | zomg 22!@##@ | 21:25 |
| onovy | bambilion? :) | 21:25 |
| *** chlong has joined #openstack-swift | 21:29 | |
| MooingLemur | clayg: would be nice to rewind a few years and pick a lower number now that I know multiple storage policies would be a thing. :) | 21:30 |
| onovy | timemachine... i will buy two. thanks | 21:30 |
| mattoliverau | lol | 21:30 |
| mattoliverau | oh and morning | 21:31 |
| onovy | it's 10pm... | 21:31 |
| MooingLemur | onovy: two? wouldn't that create a paradox? :D | 21:31 |
| mattoliverau | pfft, you mean 8:30am | 21:31 |
| onovy | MooingLemur: i will put one timemachine into second | 21:31 |
| onovy | and timetravel into past | 21:31 |
| onovy | and sell timemachine to myself | 21:31 |
| MooingLemur | causal loop complete | 21:31 |
| MooingLemur | But anyway, in-place storage policy migration would probably allow me to fix picking too high a part power. Definitely looking forward to being able to use that :D | 21:32 |
| onovy | that's right | 21:33 |
| onovy | ovh can do it already | 21:33 |
| onovy | if i remember correctly | 21:33 |
| MooingLemur | onovy: re two-time machines: http://www.myextralife.com/wp-content/uploads/2009/02/2009-02-26.jpg | 21:33 |
| onovy | MooingLemur: hmm, maybe i'm stupid, but i don't get it :/ | 21:34 |
| onovy | a portal! | 21:34 |
| onovy | got it. | 21:34 |
| *** Jeffrey4l has quit IRC | 21:35 | |
| onovy | MooingLemur: how big geo-cluster do you have? | 21:37 |
| *** Jeffrey4l has joined #openstack-swift | 21:47 | |
| *** dmorita has quit IRC | 22:03 | |
| *** davideagle has joined #openstack-swift | 22:04 | |
| *** mmotiani_ has joined #openstack-swift | 22:06 | |
| *** chlong has quit IRC | 22:09 | |
| *** klrmn has quit IRC | 22:13 | |
| *** vint_bra has quit IRC | 22:21 | |
| *** vint_bra has joined #openstack-swift | 22:24 | |
| *** dmorita has joined #openstack-swift | 22:24 | |
| *** mvk has joined #openstack-swift | 22:25 | |
| *** catintheroof has quit IRC | 22:28 | |
| *** zigo has quit IRC | 22:29 | |
| *** vint_bra has quit IRC | 22:31 | |
| *** vint_bra has joined #openstack-swift | 22:33 | |
| *** zigo has joined #openstack-swift | 22:36 | |
| *** vint_bra has quit IRC | 22:37 | |
| *** vint_bra has joined #openstack-swift | 22:39 | |
| *** vint_bra has quit IRC | 22:44 | |
| *** mmotiani_ has quit IRC | 22:47 | |
| *** klrmn has joined #openstack-swift | 22:48 | |
| *** _JZ_ has quit IRC | 23:00 | |
| *** klrmn has quit IRC | 23:15 | |
| *** kei_yama has joined #openstack-swift | 23:37 | |
| *** davideagle has quit IRC | 23:47 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
