| JimCheung | Morning kota_: | 00:01 |
|---|---|---|
| kota_ | morning JimCheung | 00:01 |
| JimCheung | Kota_: Sorry I missed you yesterday. | 00:02 |
| kota_ | JimCheung: np | 00:02 |
| JimCheung | Kota_: We submitted a new patch for review. Still need some help with how to add the test to gate | 00:03 |
| kota_ | JimCheung: k, thanks. will look at | 00:03 |
| JimCheung | Kota_: Thanks! | 00:04 |
| kota_ | JimCheung: for the gate testing, I think there is a couple of way to enable it | 00:05 |
| kota_ | JimCheung: 1. add custom script to setup your backend in the liberasurecode repo but it requires that your backend should be available at online | 00:07 |
| kota_ | JimCheung: it may be difficult because of patented | 00:07 |
| notmyname | FYI the (few) PTL elections have concluded, and the list of PTLs has been posted http://lists.openstack.org/pipermail/openstack-dev/2017-February/111769.html | 00:07 |
| kota_ | JimCheung: 2. Another solution could be adding third party testing. | 00:07 |
| kota_ | I'm not familir with the third party setting but someone knows about more detail... | 00:08 |
| kota_ | maybe swiftstackers? | 00:08 |
| JimCheung | Kota_: For option 1 but create a debian package to add on the server. This will be a minimal function test only | 00:08 |
| notmyname | here's docs on the third party CI. this is what's done with the swiftstack community qa cluster http://docs.openstack.org/infra/system-config/third_party.html | 00:09 |
| JimCheung | Kota_: Option 2 maybe tricky to implement | 00:09 |
| clayg | kota_: just waiting on me I'm sorry :'( | 00:09 |
| notmyname | JimCheung: it's not too tricky. there's plenty of docs and people to ask | 00:10 |
| kota_ | notmyname: nice | 00:10 |
| notmyname | JimCheung: however, it does require you to commit compute resources, so that part may or may not be tricky for you | 00:10 |
| notmyname | JimCheung: also, note that 3rd party ci can only be advisory. it will never be able to actually gate a commit | 00:10 |
| kota_ | clayg: no worries, I'll grub my coffee and chocolate while waiting :P | 00:10 |
| *** stradling has quit IRC | 00:11 | |
| JimCheung | Kota_: and notmyname: Okay, let me review the document. We maybe able to implement a compute resource w/o too much issue | 00:11 |
| *** mvk has joined #openstack-swift | 00:11 | |
| kota_ | JimCheung: sounds nice | 00:12 |
| JimCheung | Kota_: and notmyname: Cool! Will get back with you. Thanks very much! | 00:12 |
| kota_ | looking at the ML for PTL election, PTLs are as I expected :P | 00:13 |
| clayg | kota_: notmyname: wouldn't it be *better* to have a built package to test so we can help troubleshoot issues - vs 3rd part CI (e.g. probetests on community cluster broken since... post as copy something someting?) | 00:13 |
| -kota_- * Storlets : Eran Rom | 00:13 | |
| -kota_- * Swift : John Dickinson | 00:13 | |
| notmyname | JimCheung: also, please note clayg's comments earlier here in IRC. | 00:13 |
| kota_ | on my related projects. | 00:13 |
| notmyname | JimCheung: there's a discussion to be had (but maybe not at the moment) | 00:13 |
| kota_ | clayg: yeah, currently it seems it has been broken | 00:14 |
| notmyname | JimCheung: but I will state that whatever is agreed to, if it involves phazr.io support, it absolutely *must* be tested in an automated way. so the 3rd party ci is a good start | 00:14 |
| JimCheung | notmyname: Understood. I'll rope in GaryG into the conversation as well | 00:14 |
| kota_ | (definitely, I should add shss 3rd party :/) | 00:15 |
| notmyname | JimCheung: honestly, i'm not sure when/where to have that conversation (I know clayg's in the middle of internal stuff at the moment). probably the best place to start is in gerrit | 00:15 |
| notmyname | kota_: yes! ;-) | 00:15 |
| clayg | kota_: so like I can't test shss on my dev machines right? | 00:15 |
| clayg | kota_: or I *could* but I don't - because i'm lazy? | 00:15 |
| kota_ | only on your machine (not online available), it could be tested, IIRC, you got testing binary of shss at the past tokyo summit? | 00:16 |
| clayg | notmyname: I think kota_ timburke and tdasilva are the main people driving maintainership of liberasurecode - I think *those* people need to have a conversation about what they can realistically hope to "support" with an increased plugin matrix | 00:16 |
| clayg | I would expect it would include defining some sort of scope/requirements - and then hopefully the can get some feedback if thats reasonable | 00:17 |
| JimCheung | Clayg: Makes sense | 00:17 |
| notmyname | kota_: yeah, you gave us each one. on a cd. I don't have a cd drive ;-) | 00:17 |
| clayg | however - I think there's lots of examples opensource and openstack projects that have walked down this path - I think there's lots of good insight to be gained from talking to people that have been there done that | 00:17 |
| clayg | notmyname: maybe I can find a torrent somewhere? | 00:18 |
| notmyname | lol | 00:18 |
| kota_ | notmyname: that's too bad :/ you need to google where you can buy a portable cd drive from. | 00:19 |
| clayg | lol | 00:19 |
| notmyname | lol | 00:19 |
| *** stradling has joined #openstack-swift | 00:23 | |
| *** stradling has quit IRC | 00:36 | |
| *** delatte has joined #openstack-swift | 00:42 | |
| *** delattec has quit IRC | 00:43 | |
| *** tqtran has quit IRC | 00:51 | |
| *** tonanhngo has quit IRC | 00:57 | |
| *** sams-gleb has joined #openstack-swift | 00:59 | |
| *** ukaynar has quit IRC | 01:03 | |
| *** sams-gleb has quit IRC | 01:04 | |
| *** _JZ_ has quit IRC | 01:10 | |
| *** yarkot has quit IRC | 01:10 | |
| *** tonanhngo has joined #openstack-swift | 01:17 | |
| *** adu has joined #openstack-swift | 01:19 | |
| *** adu has quit IRC | 01:21 | |
| *** tonanhngo has quit IRC | 01:22 | |
| *** tongli has joined #openstack-swift | 01:29 | |
| *** winggundamth_ has joined #openstack-swift | 02:16 | |
| clayg | wweeee git review -d 219165 | 02:19 |
| *** ukaynar has joined #openstack-swift | 02:19 | |
| kota_ | clayg: that shows patch set 53 | 02:28 |
| clayg | i'm just laughing that i'm only now getting to checking it out finally | 02:28 |
| clayg | what's the short version of the tests that got updated? | 02:29 |
| *** adu has joined #openstack-swift | 02:30 | |
| kota_ | the changes in the tests are related to some points of view with retry counts for read node and sort key on proxy. | 02:30 |
| clayg | ... the diff still ~2K lines in proxy.test_server that changed - but I think maybe only one or two tests in that module are *new* | 02:30 |
| *** winggundamth has quit IRC | 02:31 | |
| kota_ | as you know, I removed the sor key on the proxy read which ignores affinity, the proxy is now going to access the object-servers (basically shuffled in the tests) | 02:31 |
| clayg | is that good? (for the tests) | 02:32 |
| kota_ | so I reviewed some assertions | 02:32 |
| kota_ | some of them are changed from assertEqual to assertGreaterEqual and assertLessEqual. | 02:32 |
| clayg | oh wow, so somewhere deep in that 2K line diff is a few very small subtle changes to the tests - but since they moved to an entirely different part of the file it's very hard to find them? | 02:33 |
| kota_ | clayg: ah, you're right | 02:34 |
| clayg | k | 02:34 |
| kota_ | however, almost of 2K lines are tested both ECPolicy w/ and w/o ECDuplication | 02:35 |
| clayg | right - still need to wrap my head around that | 02:35 |
| kota_ | the new test should be https://review.openstack.org/#/c/219165/53/test/unit/proxy/test_server.py@5650, if i remember correctly | 02:36 |
| patchbot | patch 219165 - swift - EC Fragment Duplication - Foundational Global EC C... | 02:36 |
| kota_ | and other changes are from making Mixin for https://review.openstack.org/#/c/219165/53/test/unit/proxy/test_server.py@5549 | 02:37 |
| patchbot | patch 219165 - swift - EC Fragment Duplication - Foundational Global EC C... | 02:37 |
| clayg | k | 02:37 |
| *** tongli has quit IRC | 02:38 | |
| kota_ | ah one more, adding test setup (maybe you don't like) for policy 4 (i.e. ec_duplication storage policy) | 02:38 |
| clayg | kota_: right i saw that - it's ok - i'll get over it | 02:39 |
| *** winggundamth_ has quit IRC | 02:45 | |
| *** ukaynar has quit IRC | 02:45 | |
| clayg | timburke: kota_: I bump into this while testing the reconstructor a lot -> 425495 | 02:47 |
| clayg | patch 425495 | 02:47 |
| patchbot | https://review.openstack.org/#/c/425495/ - swift - Give spawn a cycle | 02:47 |
| *** winggundamth_ has joined #openstack-swift | 02:47 | |
| clayg | I think the associated bug report has a lot of repro details - i think the risk is pretty small overall | 02:47 |
| clayg | tfw ctrl-r matches the wrong command but you hit enter too fast | 02:51 |
| kota_ | looking | 02:55 |
| kota_ | ah, got it, it's in *once* mode | 02:55 |
| kota_ | and never re-call reconstruct to tranpoline the event hub via sleep | 02:56 |
| clayg | is *that* what's going on!? | 02:56 |
| kota_ | is it? | 02:56 |
| clayg | I really never understood why the changes fixes it - much less enough to write a test - i just notice the traceback pretty consistently except when I have that sleep in there | 02:56 |
| clayg | i found "the spot" by trial and error | 02:57 |
| kota_ | in my quick view | 02:57 |
| clayg | i have no idea | 02:57 |
| kota_ | and will look at more deeply after lunch | 02:57 |
| clayg | have a good lunch! thanks | 02:57 |
| *** sams-gleb has joined #openstack-swift | 03:01 | |
| *** sams-gleb has quit IRC | 03:06 | |
| clayg | TIL git fetch origin master:master to update local tracking branch w/o changing branches! | 03:06 |
| *** hurricanerix has quit IRC | 03:07 | |
| *** klrmn has quit IRC | 03:09 | |
| mattoliverau | you can also use the 'git fetch origin master' 'git merge origin master' to do a 2 phase git pull (actually what a pull is), except the merge can then be aborted if required. | 03:09 |
| *** dmorita has quit IRC | 03:09 | |
| mattoliverau | ^ speaking of sometimes useful git tips ;) | 03:10 |
| *** dmorita has joined #openstack-swift | 03:11 | |
| clayg | and I don't have to be *on* master to merge origin/master into master? | 03:11 |
| mattoliverau | oh yeah, you may still have to change branches.. but ^ has got me out some acidental merges on saios I have floating around and not updated in a while :P | 03:13 |
| clayg | mattoliverau: oh! sorry! yes! :protip: | 03:13 |
| mattoliverau | clayg: yeah sorry, not trying extend yours :P | 03:14 |
| * mattoliverau its still recovering from vaction | 03:14 | |
| mattoliverau | *vacation | 03:15 |
| clayg | rofl | 03:15 |
| *** dmorita has quit IRC | 03:18 | |
| *** adu has quit IRC | 03:41 | |
| *** dmorita has joined #openstack-swift | 03:43 | |
| *** links has joined #openstack-swift | 03:50 | |
| *** JimCheung has quit IRC | 03:55 | |
| *** mkaminski has joined #openstack-swift | 03:55 | |
| *** mkaminski has quit IRC | 04:04 | |
| *** psachin has joined #openstack-swift | 04:49 | |
| *** dmorita has quit IRC | 04:59 | |
| *** sams-gleb has joined #openstack-swift | 05:03 | |
| clayg | kota_: I don't understand lack list | 05:06 |
| kota_ | clayg: ok, let me describe | 05:07 |
| *** sams-gleb has quit IRC | 05:08 | |
| kota_ | clayg: parhaps, it could be said as priority list | 05:09 |
| kota_ | in the case using duplication factor | 05:09 |
| clayg | kota_: it's trying to make error handling better right? | 05:09 |
| kota_ | we would have diffrent number of missing hole with indexes | 05:09 |
| kota_ | i'm not sure to get correctly for *the error handling* you said. | 05:10 |
| clayg | like if a node['index'] % uniq_frags == 4 - then it tries to connect another node then it tries to pick that node maybe? | 05:10 |
| kota_ | it may be called as error handling but... | 05:10 |
| clayg | er... tries to pick another node that might have same frag_index? maybe? | 05:10 |
| kota_ | the test for the process is https://review.openstack.org/#/c/219165/53/test/unit/proxy/controllers/test_obj.py@4305 | 05:12 |
| patchbot | patch 219165 - swift - EC Fragment Duplication - Foundational Global EC C... | 05:12 |
| kota_ | er | 05:12 |
| clayg | ok, let me read those | 05:12 |
| clayg | thanks! | 05:12 |
| kota_ | yeah | 05:13 |
| kota_ | maybe you're closing to the answer | 05:13 |
| kota_ | it's a kind of technique when we need another node (i.e. handoffs) to pick for missing primaries | 05:13 |
| kota_ | when choosing handoffs and iif the case # of handoff nodes is less than replicas | 05:14 |
| kota_ | # of replicas | 05:14 |
| clayg | yeah the test was good a tip - thanks | 05:15 |
| kota_ | e.g. 4-2 * 2 case, we need 12 nodes to store whole fragments | 05:15 |
| kota_ | and if only 6 ~ 11 nodes available, how we could choose the index which should be stored. | 05:15 |
| kota_ | and the lack list will be effective to choose the indexes as possible as unique | 05:16 |
| kota_ | please feel free to ask me any time you need more description | 05:17 |
| *** tonanhngo has joined #openstack-swift | 05:25 | |
| timburke | hmmm.... once we land https://review.openstack.org/#/c/425493/, i guess https://review.openstack.org/#/c/340584/ can be abandoned? | 05:43 |
| patchbot | patch 425493 - swift - Make the reconstructor handoffs_first work (and us... | 05:43 |
| patchbot | patch 340584 - swift - Make handoffs_first per partition in reconstructor | 05:43 |
| clayg | i've never seen patch 340584 before? | 05:44 |
| patchbot | https://review.openstack.org/#/c/340584/ - swift - Make handoffs_first per partition in reconstructor | 05:44 |
| clayg | lol - that's *very* similar to patch 428408 | 05:46 |
| patchbot | https://review.openstack.org/#/c/428408/ - swift - Optimize reconstructor handoffs_first | 05:46 |
| clayg | at one point I even had witten the is_handoff in a closure like that | 05:47 |
| clayg | too funny | 05:47 |
| clayg | onovy: is in the future! | 05:47 |
| clayg | I think onovy's patch would have fixed lp bug #1491605 too | 05:52 |
| openstack | Launchpad bug 1491605 in OpenStack Object Storage (swift) "Reconstructor jobs are ordered by disk instead of randomized" [High,Fix released] https://launchpad.net/bugs/1491605 | 05:52 |
| clayg | onovy: when you put up changes you should associate them with bugs - help prioritize reviews "objective observable undesirable behavior" goes away after you review this (hopefully samll and well tested) change | 05:53 |
| clayg | ^ MY FAVORITE REVIEW | 05:53 |
| *** sams-gleb has joined #openstack-swift | 06:06 | |
| *** sams-gleb has quit IRC | 06:10 | |
| *** SkyRocknRoll has joined #openstack-swift | 06:20 | |
| mahatic | swift-ring-builder doesn't seem to have a -h or --help command option, but does have good docs inside code! | 06:25 |
| clayg | OMG lack_list is for PUT :'( | 06:27 |
| timburke | mahatic: yeah, i've been disappointed by that too :-( | 06:27 |
| timburke | clayg: ...yes? of course. what *else* would it be for?? | 06:27 |
| timburke | i mean, we don't want 2 of frag 0 and none of frag 1, right? | 06:28 |
| clayg | I thought it had something to do with picking handoffs for GET | 06:28 |
| timburke | :P | 06:28 |
| clayg | ... is that what would happen? | 06:28 |
| mahatic | so what's a good way to help myself get ramped up on patch 219165 other than reading code :P | 06:28 |
| patchbot | https://review.openstack.org/#/c/219165/ - swift - EC Fragment Duplication - Foundational Global EC C... | 06:28 |
| clayg | mahatic: I always ran ith w/o any args - seems to print a bunch of text? | 06:29 |
| timburke | it could. the idea behind lack_list is to avoid that sort of situation, anyway | 06:29 |
| mahatic | clayg: works like a charm \o/ thanks! | 06:30 |
| timburke | ok, so -- EC: great! high durability, lower storage requirements! global clusters: great! high durability (because things are in multiple data centers) *and* nice response times (because you can totally prefer to service requests locally) | 06:31 |
| mahatic | timburke: ^ | 06:32 |
| timburke | EC + global clusters: sad :-( | 06:32 |
| timburke | currently. | 06:32 |
| timburke | you could *try* to make it less sad by upping parity frags past data frags. but you'll still be sad, because now you *always* have to reconstruct on GET (in at least one region) | 06:33 |
| timburke | kota's patch (assuming your ring is *just right*) -> not so sad. lots of data frags, some of parity frags, ~same storage requirements compared to replicated, better durability | 06:34 |
| timburke | kota_: i think i'm representing it right? yeah? | 06:34 |
| timburke | but now i should sleep, not review. because margaritas. | 06:35 |
| timburke | g'night! | 06:35 |
| mahatic | timburke: :D good night! | 06:36 |
| mahatic | that looks like a gist, but I didn't know that we *did not always* reconstruct on GET prior to kota_ 's patch | 06:36 |
| kota_ | timburke: right and good night! | 06:37 |
| mahatic | timburke: thanks! | 06:38 |
| clayg | mahatic: there used to be some presentation slides linked from the commit message | 06:38 |
| clayg | mahatic: yes, they're still there -> http://www.slideshare.net/tsuyuzaki/global-ec-cluster-updates-openstack-mitaka-swift-design-summit | 06:39 |
| mahatic | clayg: looking, thanks | 06:39 |
| kota_ | mahatic: note that, it could be a bit stale info (espically about frag index design at Page 7) | 06:40 |
| kota_ | mahatic: but the anlysis about performance should be always true. | 06:40 |
| mahatic | kota_: oic | 06:41 |
| clayg | I personally have a *different* reason for not liking a 10+14 schema sort of solution to global EC | 06:41 |
| clayg | with a 10x14 I have a fixed width - I can support rebuilding from either side of even split (any 10 will do, so two regions should get 12 frags each, so i'm ok) | 06:42 |
| clayg | but if I want to increase to 3 regions - can't | 06:42 |
| clayg | with a 10+2x2 I can always migrate to a 10+2x3 | 06:42 |
| kota_ | clayg: definately | 06:42 |
| clayg | and it's *also* faster | 06:42 |
| * mahatic is off to lunch | 06:47 | |
| *** tqtran has joined #openstack-swift | 06:50 | |
| *** sams-gleb has joined #openstack-swift | 06:50 | |
| mahatic | kota_: clayg: "(object ring must have 6 replicas)" commit message says that after swift.conf example. Does it mean EC global clusters will need 6 replicas? Did I read it wrong? | 06:51 |
| kota_ | mahatic: it's an example | 06:52 |
| clayg | mahatic: yes, you'll need to make some ring chnages to test this change | 06:52 |
| kota_ | mahatic: actually, (ec_k + ec_m) * duplication_factor -> # of replicas | 06:52 |
| clayg | are you using vsaio or something fancy like that - or just plain vanilla? | 06:52 |
| kota_ | mahatic: in the case, ec_k = 2, ec_m = 1, and duplication_factor = 2 -> (2 + 1) * 2 == 6 you need. | 06:53 |
| mahatic | clayg: plain vanilla. My current testing env has swift + keystone (installed via devstack). But I also have just saio vms | 06:53 |
| clayg | what's the default # of devices for the ec policy? | 06:54 |
| clayg | how many sdb* looking things do you have in /srv/node*/ | 06:54 |
| *** tqtran has quit IRC | 06:54 | |
| mahatic | kota_: can it go lower than that? I don't think so? | 06:54 |
| kota_ | mahatic: on ec setting? or just # of replicas? | 06:55 |
| mahatic | kota_: ec_k and ec_m | 06:55 |
| mahatic | kota_: in effect the number of replicas | 06:55 |
| kota_ | mahatic: it could be on the setting constraint but... | 06:56 |
| kota_ | mahatic: i think the setting with lower values is nothing effective | 06:56 |
| kota_ | ec_k = 1, ec_m = 1, it looks just 2 replica? | 06:56 |
| clayg | heh | 06:57 |
| clayg | mahatic: how many *devices* do you have? | 06:57 |
| mahatic | clayg: sorry | 06:57 |
| *** abqkawi1000 has quit IRC | 06:57 | |
| kota_ | with ec_k = 2, and ec_m = 0... hmm, it could be effective as striping? | 06:57 |
| mahatic | clayg: in my current vm saio, i have what I configured via loopback | 06:57 |
| mahatic | so 4 | 06:57 |
| clayg | mahatic: yeah that's not enough :'( | 06:57 |
| mahatic | clayg: what do I need? I just fetched a brand new server! | 06:58 |
| clayg | I think the saio docs have instructions for setting up ec - wants at least 8 devices for a 4+2 | 06:58 |
| kota_ | not sure if current swift ec works as striping. I don't think we have big problem on that except durability and reliability. | 06:58 |
| clayg | kota_: ? | 06:59 |
| kota_ | what? | 06:59 |
| *** rcernin has joined #openstack-swift | 06:59 | |
| clayg | kota_: I didn't follow what you were saying about striping/durability/reliablity? | 06:59 |
| clayg | kota_: but it sounded important! | 07:00 |
| clayg | mahatic: http://docs.openstack.org/developer/swift/development_saio.html#using-a-loopback-device-for-storage | 07:00 |
| kota_ | ah | 07:00 |
| clayg | says you need 8 devices | 07:00 |
| *** dmorita has joined #openstack-swift | 07:00 | |
| clayg | mahatic: you should try https://github.com/ntata/swift-setup-scripts | 07:00 |
| kota_ | e.g. if you set ec_k = 5 and ec_m =0 (which should work likely RAID 0 striping), we don't have any redundancy with the stored data. | 07:01 |
| kota_ | clayg:^^ | 07:01 |
| clayg | mahatic: vsaio makes it easy to build saio with different configs - for my global-ec test I'm using 4 nodes x 4 devices (sdb1-sdb16) 4+2x2 | 07:01 |
| kota_ | I'm not sure if such a parameter could be available in current Swift master | 07:01 |
| mahatic | clayg: I do have that setup too in a vm. this one - http://docs.openstack.org/developer/swift/development_saio.html#using-a-loopback-device-for-storage | 07:02 |
| clayg | kota_: OH! | 07:02 |
| -kota_- >>> driver = ECDriver(ec_type='liberasurecode_rs_vand', k=5, m=0) | 07:02 | |
| -kota_- Traceback (most recent call last): | 07:02 | |
| -kota_- File "<stdin>", line 1, in <module> | 07:02 | |
| -kota_- File "/usr/local/lib/python2.7/dist-packages/pyeclib/ec_iface.py", line 169, in __init__ | 07:02 | |
| -kota_- "Invalid number of data fragments (m)") | 07:02 | |
| -kota_- pyeclib.ec_iface.ECDriverError: Invalid number of data fragments (m) | 07:02 | |
| mahatic | clayg: oic | 07:02 |
| kota_ | unfortunately, we cannot | 07:02 |
| kota_ | ec_m must be >0 | 07:02 |
| kota_ | it looks like weird error message though | 07:03 |
| clayg | kota_: that'd be cool tho! 3+0x3 sounds really interesting all the sudden :P | 07:03 |
| kota_ | it should say "Invalid number of *parity* fragments (m)", right? | 07:03 |
| clayg | data frags? m? | 07:03 |
| clayg | lol | 07:03 |
| mahatic | :D | 07:03 |
| clayg | kota_: you *had* to go turn over the rock | 07:03 |
| mahatic | yes parity | 07:03 |
| clayg | kota_: and I didn't even know I wanted 3+0x3 - i totally do now tho | 07:04 |
| clayg | i want that | 07:04 |
| *** dmorita has quit IRC | 07:04 | |
| clayg | or like maybe 5+0x3 no sure | 07:04 |
| kota_ | lol | 07:04 |
| * mahatic needs be afk. Be back in a bit | 07:05 | |
| kota_ | it sound like raid 0 + 1 (and +1)? | 07:05 |
| clayg | mahatic: but with 8 devices (I think ntata's script will setup 8 devices) - I think you can do a 2+1x2 and be ok | 07:05 |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/pyeclib master: Fix error message with invalid parity number https://review.openstack.org/430599 | 07:12 |
| kota_ | done for fixing the error message at pyeclib | 07:12 |
| clayg | kota_: I think my 16 devices are maybe to constrained to really understand lack list with 12 total frags | 07:15 |
| clayg | https://gist.github.com/clayg/26c804954ffa1032228bff8649b830eb | 07:15 |
| clayg | so without lack list my handoff devices are r1sdb11#3, r2sdb6#0, r2sdb12#4 - with lack list my handoffs are r1sdb11#0, r2sdb6#3, r2sdb12#4 | 07:16 |
| clayg | so 0 & 3 flip? is that... good? | 07:16 |
| clayg | like it's the same handoff nodes in the iter (obvs.) so what are the principles we even use to decide which handoff node should get which fragment? | 07:17 |
| clayg | how would I even say one list is better than another? I think the holes enumeration on master is determistic - same nodes down gives you same indexes on same handoffs - but we don't go out of our way to assign any specific handoff any specific node index | 07:18 |
| clayg | I think we get it on disk and let the proxy sort it out on read | 07:18 |
| clayg | proxy is good at sorting it out on read | 07:18 |
| kota_ | looking | 07:19 |
| kota_ | it looks like the down node 1 assignged a primary half of ec unique_fragments? | 07:21 |
| kota_ | i think if node 2 (it looks to have secondary half of fragments) down, it will be different behavior between w/ and w/o lack_list | 07:22 |
| clayg | kota_: cool I try it! thanks! | 07:23 |
| openstackgerrit | Christian Schwede proposed openstack/swift master: Add support to increase object ring partition power https://review.openstack.org/337297 | 07:24 |
| kota_ | oooh | 07:24 |
| kota_ | it may need more specific situation | 07:24 |
| kota_ | i think 1 node down behaves only 1 missing for 3 indexes | 07:25 |
| *** tonanhngo has quit IRC | 07:25 | |
| kota_ | the lack_list will be effective when mixture case with 2 missing and 1 missing for each index | 07:26 |
| clayg | why? why is missing index 0, 6 any different in 4+2x2 than missing 0, 5 - all that matters it that the missing index gets assigned to a handoff - and that the backend conversion changes the assigned index to correct frag index? | 07:28 |
| clayg | how do I say which node in my handoff iter is better to have frag #X vs frag #Y | 07:28 |
| clayg | what is the rules used to govern such a decision? I don't under stand why assinging on or another is better as long as one get assigned. | 07:29 |
| *** pcaruana has joined #openstack-swift | 07:29 | |
| clayg | so what if fist handoff gets 0 and second handoff gets 6 (also 0 frag index) - what choice do you have? | 07:29 |
| kota_ | wait a bit, I'm trying to understand the situation on your gist. | 07:29 |
| *** ChubYann has quit IRC | 07:30 | |
| clayg | i'm trying to understand the motivation of changing the holes assignment | 07:30 |
| clayg | you have primaries missing - they have indexes - you replace them with handoffs - you give the primary indexes you don't have assigned to handoffs - simplicity itself | 07:31 |
| clayg | how would I evaluate if assigning the missing set (0, 5, 6) is better (0, 6, 5) or (5, 6, 0) | 07:31 |
| clayg | what is the model by which I would say one is better than another? | 07:31 |
| clayg | only requirement I have is "deterministic" | 07:32 |
| kota_ | ah, not intend to the order | 07:32 |
| clayg | which is why I like first handoff gets first hole, second handoff gets second hole | 07:32 |
| kota_ | set (0, 5, 6) and 0 == unique_idnex(6), right? | 07:32 |
| kota_ | you mean? | 07:33 |
| clayg | sure that's the idea | 07:33 |
| kota_ | and if we have only one handoff | 07:33 |
| clayg | but... I mean also like how is assigning the set (0, 5, 6) have different requirements from assigning the set (0, 5, 7) | 07:33 |
| kota_ | should we choose 0 or 6 rather than 5 | 07:33 |
| kota_ | right? | 07:33 |
| clayg | only *one* handoff!? | 07:33 |
| * clayg mind blown | 07:33 | |
| kota_ | when get_more_nodes gave up to pop | 07:34 |
| clayg | lazy ass get_more_nodes - someone needs a bigger cluster | 07:34 |
| clayg | lol | 07:34 |
| kota_ | clayg: | 07:34 |
| kota_ | yes | 07:34 |
| clayg | so my 16 devices had *too many* devices to show the issue | 07:34 |
| clayg | :D | 07:34 |
| kota_ | if we have enough devices for handoffs | 07:35 |
| kota_ | nothing intended in the lack list in my mind | 07:35 |
| clayg | and here I thought I better have at least enough handoffs that I can kill a whole node | 07:35 |
| clayg | ok - that is super helpful - thanks | 07:35 |
| *** tesseract has joined #openstack-swift | 07:35 | |
| *** david-lyle_ has joined #openstack-swift | 07:37 | |
| kota_ | clayg: great, exactly it may be a corner case but i could not stop to think the case the handoff nodes filled out only duplicated one dropping unique ones | 07:37 |
| clayg | what's the total number of responses I need for success in a 4+2x2? | 07:39 |
| * mahatic is back | 07:40 | |
| mahatic | clayg: ack | 07:40 |
| kota_ | clayg: depends on # of node down? but thinking of failure domain, at least 2 nodes should be down | 07:40 |
| kota_ | with more than or equal to one same frag index and there is a few frag index with single missing | 07:41 |
| kota_ | looking at the gist above... | 07:42 |
| kota_ | down node 1 and node 2 will be missing index 0, 4 (2 missing) and 5 (1 missing) | 07:43 |
| kota_ | so if you would have 2 available devices in node 3, 4 (sounds like 1 device for each) | 07:44 |
| kota_ | could be nice to reproduce | 07:44 |
| kota_ | the handoff devices in node 3, 4 will have index 0, 4 fragments | 07:45 |
| clayg | ok, so total number of successful backend responses required to 201 on PUT from a 4+2x2 depends on which frags are in response? (in the ideal? or what's in the patch?) | 07:45 |
| clayg | what's the *bounds* | 07:46 |
| clayg | 4 < x < 12 - can we get any tighter? | 07:46 |
| clayg | I think 4x2 is 5 (k + 1) | 07:46 |
| clayg | s/4x2/standard ec 4+2/ | 07:47 |
| kota_ | clayg: sorry, I may not get the question | 07:48 |
| kota_ | ah | 07:48 |
| kota_ | you means # of quorum? | 07:49 |
| kota_ | ah, 5, ec_k + 1 IIRC | 07:50 |
| clayg | yeah - my 4+2x2 also returns 5 | 07:50 |
| kota_ | but the 5 should be unique | 07:50 |
| clayg | *interesting* | 07:50 |
| kota_ | oops!??? | 07:55 |
| kota_ | _check_failure_put_connections doesn't care about frag_index??? | 07:56 |
| clayg | kota_: https://gist.github.com/clayg/36c87fc8fdb07bff70f7160828725e95 | 08:00 |
| clayg | with lack list I get a #4 :D | 08:01 |
| kota_ | clayg: great :D | 08:03 |
| kota_ | and you may point out good catch | 08:03 |
| *** tonanhngo has joined #openstack-swift | 08:03 | |
| kota_ | i think it could happen, iif we have no handoffs and get connected with 3 indexes and 3 duplicate indexes (e.g. 0, 1, 2, 0, 1, 2) > 5 could result in success??? | 08:03 |
| kota_ | even if it's compltely no durable. | 08:04 |
| *** tonanhngo has quit IRC | 08:04 | |
| clayg | idk, up until an hour ago I wasn't even considering the case where you might connect to < total frags nodes | 08:05 |
| kota_ | ok | 08:05 |
| kota_ | it could be, need to try to make sure. | 08:06 |
| clayg | for all the time I spend worrying about how long it takes to 404 a HEAD request on EC - thinking about *less* disks isn't really on my radar | 08:06 |
| kota_ | GET? | 08:08 |
| clayg | sure 404 on GET is just as bad | 08:08 |
| kota_ | got it | 08:08 |
| clayg | because node_iter tends to not run out | 08:08 |
| clayg | in practice | 08:08 |
| clayg | unlike in our dev environments | 08:08 |
| clayg | I think quorum for ec should be * duplication_factor | 08:10 |
| clayg | dodge that bullet | 08:10 |
| *** rledisez has joined #openstack-swift | 08:11 | |
| kota_ | clayg: might be | 08:12 |
| kota_ | for now, it's just ec_k I'm making sure now. | 08:12 |
| kota_ | make sure the code | 08:13 |
| *** david-lyle_ has quit IRC | 08:28 | |
| *** chlong has quit IRC | 08:30 | |
| *** chlong has joined #openstack-swift | 08:31 | |
| *** winggundamth__ has joined #openstack-swift | 08:32 | |
| *** stevemar has quit IRC | 08:33 | |
| *** winggundamth_ has quit IRC | 08:35 | |
| *** stevemar has joined #openstack-swift | 08:35 | |
| clayg | kota_: I don't think it was such a great idea to bring in those reconstructor changes from my WIP | 08:36 |
| clayg | there's a lot of untested branches in that change :\ | 08:36 |
| clayg | like I think it might be better than what's on master - but it might have bugs and be hard to maintain if it's not tested? | 08:36 |
| *** oshritf has joined #openstack-swift | 08:49 | |
| *** tqtran has joined #openstack-swift | 08:52 | |
| *** tqtran has quit IRC | 08:56 | |
| *** kei_yama has quit IRC | 08:58 | |
| openstackgerrit | Christian Schwede proposed openstack/swift master: Fix timing test error when rebalancing https://review.openstack.org/430660 | 09:01 |
| *** xlucas has joined #openstack-swift | 09:01 | |
| *** Shashikant86 has joined #openstack-swift | 09:17 | |
| *** jordanP has joined #openstack-swift | 09:21 | |
| *** oshritf has quit IRC | 09:29 | |
| *** oshritf has joined #openstack-swift | 09:29 | |
| *** openstackgerrit has quit IRC | 09:34 | |
| *** links has quit IRC | 09:35 | |
| *** gabor_antal_km has joined #openstack-swift | 09:35 | |
| *** Shashikant86 has quit IRC | 09:37 | |
| *** sanchitmalhotra1 has joined #openstack-swift | 09:37 | |
| *** JimCheung has joined #openstack-swift | 09:39 | |
| *** clayg_ has joined #openstack-swift | 09:40 | |
| *** ChanServ sets mode: +v clayg_ | 09:40 | |
| *** Guest66676 has joined #openstack-swift | 09:41 | |
| *** JimCheung has quit IRC | 09:43 | |
| *** bobby2_ has joined #openstack-swift | 09:43 | |
| *** topol_ has joined #openstack-swift | 09:44 | |
| *** dims_ has joined #openstack-swift | 09:44 | |
| *** cschwede_ has joined #openstack-swift | 09:44 | |
| *** mlanner_ has joined #openstack-swift | 09:44 | |
| *** hugokuo_ has joined #openstack-swift | 09:45 | |
| *** timur has joined #openstack-swift | 09:45 | |
| *** jordanP has quit IRC | 09:45 | |
| *** SkyRocknRoll has quit IRC | 09:45 | |
| *** dims has quit IRC | 09:45 | |
| *** gabor_antal_ has quit IRC | 09:45 | |
| *** jtomasek has quit IRC | 09:45 | |
| *** sanchitmalhotra has quit IRC | 09:45 | |
| *** pdardeau has quit IRC | 09:45 | |
| *** topol has quit IRC | 09:45 | |
| *** aj701_ has quit IRC | 09:45 | |
| *** hoonetorg has quit IRC | 09:45 | |
| *** timss has quit IRC | 09:45 | |
| *** cschwede has quit IRC | 09:45 | |
| *** mmmucky has quit IRC | 09:45 | |
| *** timur_ has quit IRC | 09:45 | |
| *** bobby2 has quit IRC | 09:45 | |
| *** ntata has quit IRC | 09:45 | |
| *** clayg has quit IRC | 09:45 | |
| *** hugokuo has quit IRC | 09:45 | |
| *** balajir has quit IRC | 09:45 | |
| *** Guest66666 has quit IRC | 09:45 | |
| *** mlanner has quit IRC | 09:45 | |
| *** mlanner_ is now known as mlanner | 09:45 | |
| *** hugokuo_ is now known as hugokuo | 09:45 | |
| *** balajir_ has joined #openstack-swift | 09:45 | |
| *** oshritf has quit IRC | 09:46 | |
| *** mmmucky has joined #openstack-swift | 09:46 | |
| *** ntata has joined #openstack-swift | 09:46 | |
| *** sanchitmalhotra1 is now known as sanchitmalhotra | 09:46 | |
| *** aj701 has joined #openstack-swift | 09:46 | |
| *** Shashikant86 has joined #openstack-swift | 09:50 | |
| *** links has joined #openstack-swift | 09:51 | |
| *** pdardeau has joined #openstack-swift | 09:51 | |
| *** SkyRocknRoll has joined #openstack-swift | 09:52 | |
| *** timss has joined #openstack-swift | 09:52 | |
| *** jordanP has joined #openstack-swift | 09:52 | |
| *** jtomasek has joined #openstack-swift | 09:52 | |
| *** oshritf has joined #openstack-swift | 09:54 | |
| *** hoonetorg has joined #openstack-swift | 09:54 | |
| *** Shashikant86 has quit IRC | 09:55 | |
| *** Shashikant86 has joined #openstack-swift | 09:56 | |
| *** oshritf has quit IRC | 10:03 | |
| clayg_ | kota_: it's looking good! | 10:06 |
| kota_ | clayg_: :D | 10:07 |
| kota_ | clayg_: are you still around!? | 10:07 |
| clayg_ | kota_: I think you should consider increasing the quorum size | 10:07 |
| kota_ | clayg_: yeah, I was thinking on that, and just trying to make it (k + 1) * duplication | 10:08 |
| clayg_ | kota_: yeah i like that - only hit a couple of tests - just needed to dial back some ... it's in the a gist in the review somewhere | 10:08 |
| kota_ | it looks like only 2 tests at proxy/controllers/test_obj.py failed with the setting, but no failures on proxy/test_server.py | 10:08 |
| kota_ | clayg_: it seems we're doing same thing :\ | 10:09 |
| clayg_ | is it wierd that you can make that kind of change to a "fundemental constraint" and only a couple of minor spurious unrelated tests fail? | 10:09 |
| clayg_ | idk, maybe not - i wonder how many tests fail if you jerk around quorum in replicated | 10:09 |
| kota_ | maybe we *want* to add more tests for *really* strict case less than quorum but more than ec_k. | 10:12 |
| *** dmorita has joined #openstack-swift | 10:13 | |
| kota_ | i think almost of tests addresses less than ec_k which should not fail because they're consistent the new quorum | 10:13 |
| kota_ | so it should fail anyway | 10:13 |
| kota_ | s/it/they | 10:13 |
| clayg_ | yeah making replicated quorum +1 fails a lot - making it -1 much fewer tests notice | 10:14 |
| *** Shashikant86 has quit IRC | 10:15 | |
| clayg_ | I like proxy.controllers.test_obj - stuff like test_PUT_with_just_enough_durable_responses | 10:15 |
| *** dmorita has quit IRC | 10:17 | |
| clayg_ | I think (k + 1) * duplication prevents a lot of worry | 10:17 |
| clayg_ | i think maybe you depend on the duplication like replication - just because [201, 503, 503] in triple replica *can* fully rebuild - doesn't mean it's durable enough to trust | 10:18 |
| *** Shashikant86 has joined #openstack-swift | 10:18 | |
| kota_ | clayg_: i agree with increasing quorum rather than worrying with current one for data loss after 201. we could have more efficiency in the future (check the unique indexes count) but set it as in the future. | 10:21 |
| clayg_ | werd | 10:21 |
| *** clayg_ is now known as clayg | 10:22 | |
| kota_ | so I'll try to read/address on your comments and push a new version maybe either today or tommorow. | 10:23 |
| *** dmorita has joined #openstack-swift | 10:29 | |
| kota_ | ah, you already found the way to fix the failed tests (e.g. test_DELETE_mostly_not_found) with increasing duplication factor | 10:29 |
| kota_ | nice | 10:29 |
| kota_ | what i will do is just merging and testing in local | 10:30 |
| *** dmorita has quit IRC | 10:33 | |
| clayg | kota_: other thing is either pull out the fixes in reconstructor or add tests for them | 10:40 |
| *** winggundamth__ has quit IRC | 10:49 | |
| *** links has quit IRC | 10:50 | |
| kota_ | clayg: ok | 10:50 |
| *** sams-gleb has quit IRC | 10:56 | |
| *** sams-gleb has joined #openstack-swift | 10:57 | |
| *** tanee is now known as tanee_away | 11:00 | |
| *** sams-gleb has quit IRC | 11:01 | |
| *** acoles_ is now known as acoles | 11:02 | |
| kota_ | hah? It looks like I hit the bug cschwede_ is trying to solve at https://review.openstack.org/430660? | 11:05 |
| patchbot | patch 430660 - swift - Fix timing test error when rebalancing | 11:05 |
| clayg | +A it! | 11:05 |
| kota_ | the report said | 11:06 |
| -kota_- AssertionError: '1 hours (1:00:00 remaining)' not found in 'No partitions could be reassigned.\nThe t | 11:06 | |
| -kota_- ime between rebalances must be at least min_part_hours: 1 hours (0:59:59 remaining)\n' | 11:06 | |
| *** links has joined #openstack-swift | 11:06 | |
| kota_ | clayg: is it same bug? | 11:06 |
| clayg | yeah | 11:06 |
| clayg | well - same error output | 11:06 |
| clayg | we could easily make that same bug in more than one test | 11:07 |
| kota_ | k, I should look at it first to add my +A | 11:07 |
| *** saltsa has quit IRC | 11:11 | |
| *** silor has joined #openstack-swift | 11:14 | |
| *** dmorita has joined #openstack-swift | 11:18 | |
| *** sams-gleb has joined #openstack-swift | 11:18 | |
| *** ganders has joined #openstack-swift | 11:22 | |
| *** dmorita has quit IRC | 11:23 | |
| kota_ | done! | 11:24 |
| *** silor1 has joined #openstack-swift | 11:31 | |
| *** silor has quit IRC | 11:32 | |
| *** silor1 is now known as silor | 11:32 | |
| *** mvk has quit IRC | 11:53 | |
| *** Shashikant86 has quit IRC | 11:56 | |
| *** Shashikant86 has joined #openstack-swift | 12:01 | |
| *** Shashikant86 has quit IRC | 12:01 | |
| kota_ | hmm, i got a weird report when adding logger assertion, 'Unable to get enough responses (27/10) :/ | 12:02 |
| kota_ | probably, it's from handoff retry to get in reconstruct | 12:03 |
| *** silor has quit IRC | 12:03 | |
| *** silor has joined #openstack-swift | 12:04 | |
| *** silor1 has joined #openstack-swift | 12:07 | |
| kota_ | ah... no? | 12:08 |
| *** silor has quit IRC | 12:09 | |
| *** silor1 is now known as silor | 12:09 | |
| *** oshritf has joined #openstack-swift | 12:12 | |
| *** dmorita has joined #openstack-swift | 12:18 | |
| *** mvk has joined #openstack-swift | 12:21 | |
| *** Shashikant86 has joined #openstack-swift | 12:21 | |
| *** dmorita has quit IRC | 12:22 | |
| *** SkyRocknRoll has quit IRC | 12:35 | |
| *** MVenesio has joined #openstack-swift | 12:38 | |
| *** acoles is now known as acoles_ | 12:47 | |
| *** acoles_ is now known as acoles | 12:53 | |
| kota_ | hmm... it's time I should leave office... | 12:56 |
| *** mvk has quit IRC | 13:04 | |
| *** dmorita has joined #openstack-swift | 13:06 | |
| *** psachin has quit IRC | 13:07 | |
| *** catintheroof has joined #openstack-swift | 13:08 | |
| *** dmorita has quit IRC | 13:10 | |
| *** klamath has joined #openstack-swift | 13:13 | |
| *** mvk has joined #openstack-swift | 13:17 | |
| *** openstackgerrit has joined #openstack-swift | 13:35 | |
| openstackgerrit | Merged openstack/swift master: Fix timing test error when rebalancing https://review.openstack.org/430660 | 13:35 |
| *** links has quit IRC | 13:39 | |
| *** silor has quit IRC | 13:42 | |
| *** dmorita has joined #openstack-swift | 13:42 | |
| *** oshritf has quit IRC | 13:46 | |
| *** dmorita has quit IRC | 13:47 | |
| eranrom | greetings: does anybody know if a container's read-acl can be set from the openstack cli? | 13:55 |
| *** oshritf has joined #openstack-swift | 13:55 | |
| *** tongli has joined #openstack-swift | 13:59 | |
| eranrom | Looking at the code, it seems like the answer is no. | 14:00 |
| *** stradling has joined #openstack-swift | 14:02 | |
| *** dmorita has joined #openstack-swift | 14:07 | |
| *** Shashikant86 has quit IRC | 14:10 | |
| *** Shashikant86 has joined #openstack-swift | 14:11 | |
| *** dmorita has quit IRC | 14:11 | |
| *** oshritf has quit IRC | 14:23 | |
| *** oshritf has joined #openstack-swift | 14:26 | |
| *** oshritf has quit IRC | 14:29 | |
| *** Shashikant86 has quit IRC | 14:29 | |
| *** oshritf has joined #openstack-swift | 14:32 | |
| *** Shashikant86 has joined #openstack-swift | 14:34 | |
| *** ganders has quit IRC | 14:34 | |
| *** Shashikant86 has quit IRC | 14:37 | |
| *** JimCheung has joined #openstack-swift | 14:38 | |
| *** ganders has joined #openstack-swift | 14:43 | |
| *** JimCheung has quit IRC | 14:43 | |
| *** Shashikant86 has joined #openstack-swift | 14:43 | |
| *** dmorita has joined #openstack-swift | 14:43 | |
| *** dmorita has quit IRC | 14:47 | |
| *** dosaboy_ is now known as dosaboy | 14:50 | |
| *** oshritf has quit IRC | 14:51 | |
| *** oshritf has joined #openstack-swift | 14:53 | |
| *** tonanhngo has joined #openstack-swift | 15:01 | |
| *** Shashikant86 has quit IRC | 15:01 | |
| *** tonanhngo has quit IRC | 15:02 | |
| *** dmorita has joined #openstack-swift | 15:07 | |
| *** sams-gleb has quit IRC | 15:08 | |
| *** sams-gleb has joined #openstack-swift | 15:08 | |
| *** oshritf has quit IRC | 15:09 | |
| *** dmorita has quit IRC | 15:12 | |
| *** stradling has quit IRC | 15:12 | |
| *** sams-gleb has quit IRC | 15:13 | |
| *** mvk has quit IRC | 15:14 | |
| *** caiobrentano has joined #openstack-swift | 15:15 | |
| *** tongli has quit IRC | 15:16 | |
| *** gabor_antal_km is now known as gabor_antal | 15:19 | |
| *** chsc has joined #openstack-swift | 15:19 | |
| *** chsc has joined #openstack-swift | 15:19 | |
| *** sams-gleb has joined #openstack-swift | 15:25 | |
| *** chsc has quit IRC | 15:30 | |
| *** tongli has joined #openstack-swift | 15:31 | |
| *** philipw has quit IRC | 15:32 | |
| *** hurricanerix has joined #openstack-swift | 15:38 | |
| *** stradling has joined #openstack-swift | 15:39 | |
| *** _JZ_ has joined #openstack-swift | 15:40 | |
| *** mvk has joined #openstack-swift | 15:42 | |
| *** dmorita has joined #openstack-swift | 15:56 | |
| *** dmorita has quit IRC | 16:00 | |
| *** Shashikant86 has joined #openstack-swift | 16:03 | |
| *** dmorita has joined #openstack-swift | 16:10 | |
| *** dmorita has quit IRC | 16:14 | |
| *** SkyRocknRoll has joined #openstack-swift | 16:26 | |
| *** ganders has quit IRC | 16:32 | |
| *** dmorita has joined #openstack-swift | 16:34 | |
| *** silor has joined #openstack-swift | 16:37 | |
| *** dmorita has quit IRC | 16:38 | |
| *** rcernin has quit IRC | 16:39 | |
| *** chsc has joined #openstack-swift | 16:41 | |
| *** chsc has joined #openstack-swift | 16:41 | |
| *** nickchase has joined #openstack-swift | 16:42 | |
| nickchase | Hey, all, can anybody tell me if there's anything new for Swift between the Newton and Ocata releases? | 16:43 |
| tdasilva | nickchase: hi, there has been two swift releases since the newton release. The changelog details what went in: https://github.com/openstack/swift/blob/master/CHANGELOG | 16:46 |
| tdasilva | nickchase: there will also be another release (2.13) before the ocata release | 16:46 |
| *** silor has quit IRC | 16:46 | |
| *** silor has joined #openstack-swift | 16:46 | |
| tdasilva | coincidentally i believe notmyname said the deadline for that is 2/13 | 16:47 |
| nickchase | tdasilva: Awesome. Thanks so much! | 16:52 |
| *** Guest66676 is now known as Guest6666 | 16:53 | |
| *** nickchase has left #openstack-swift | 16:54 | |
| *** stradling has quit IRC | 16:55 | |
| *** tonanhngo has joined #openstack-swift | 16:56 | |
| *** tonanhngo has quit IRC | 16:56 | |
| *** ganders has joined #openstack-swift | 16:57 | |
| notmyname | good morning | 16:57 |
| notmyname | hmm...my IRC bouncer is dropping messages. what I see in my client is not complete when compared to the eavesdrop bot | 16:58 |
| *** Shashikant86 has quit IRC | 16:59 | |
| *** stradling has joined #openstack-swift | 16:59 | |
| *** stradling has quit IRC | 17:05 | |
| *** tesseract has quit IRC | 17:05 | |
| *** tqtran has joined #openstack-swift | 17:06 | |
| *** silor1 has joined #openstack-swift | 17:06 | |
| *** delattec has joined #openstack-swift | 17:07 | |
| *** delatte has quit IRC | 17:08 | |
| *** silor has quit IRC | 17:08 | |
| *** silor1 is now known as silor | 17:08 | |
| *** tqtran has quit IRC | 17:10 | |
| notmyname | tdasilva: thanks for responding to nick. my bouncer completely missed that conversation | 17:11 |
| notmyname | are there any cores who are *not* going to be at the PTG? | 17:19 |
| notmyname | here's why I ask... | 17:19 |
| notmyname | the foundation is inviting PTLs to give a project update in boston. as part of that, they are giving me the chance to put down 2 cores who will get free registration codes for boston | 17:20 |
| notmyname | however, if you're going to the PTG, you'll already get a free code for boston anyway | 17:21 |
| notmyname | so if someone isn't going to the PTG but likely going to the boston summit, I'd like to know | 17:21 |
| *** MVenesio has quit IRC | 17:24 | |
| tdasilva | notmyname: did mattoliverau say if he was going? i remember he was looking into it | 17:25 |
| notmyname | yeah, he'll be at the PTG for wed-fri | 17:25 |
| notmyname | honestly, if I recommended only one of the events for a dev to attend, it would be the PTG instead of the summit (since AFAIK there won't be any rooms for projects in boston) | 17:26 |
| *** JimCheung has joined #openstack-swift | 17:32 | |
| *** topol_ is now known as topol | 17:33 | |
| openstackgerrit | Joel Wright proposed openstack/swift master: Add Preamble and Postamble to SLO and SegmentedIterable https://review.openstack.org/365371 | 17:35 |
| *** dmorita has joined #openstack-swift | 17:36 | |
| *** dmorita has quit IRC | 17:36 | |
| *** dmorita has joined #openstack-swift | 17:36 | |
| acoles | notmyname: is joel going? | 17:37 |
| *** dmorita has quit IRC | 17:37 | |
| *** rledisez has quit IRC | 17:37 | |
| notmyname | acoles: that is a great question. I don't know | 17:37 |
| acoles | he's not here right now | 17:38 |
| * tdasilva wonders if OVH guys are going... | 17:38 | |
| *** dmorita has joined #openstack-swift | 17:41 | |
| *** joeljwright has joined #openstack-swift | 17:41 | |
| *** ChanServ sets mode: +v joeljwright | 17:41 | |
| *** klrmn has joined #openstack-swift | 17:44 | |
| notmyname | joeljwright: we were just talking about you! | 17:48 |
| notmyname | joeljwright: are you going to the PTG? or the boston summit? | 17:48 |
| joeljwright | I'm not going to the PTG, Boston is undecided | 17:48 |
| joeljwright | It all depends on whether we'll have any chance for developer stuff in Boston | 17:49 |
| notmyname | joeljwright: ok. mind if I put you down for a free ticket to boston? (PTG attendees will get one, I'm told, but I've been given the opportunity to put down a couple of names for boston tickets) | 17:49 |
| joeljwright | that would be great! | 17:50 |
| notmyname | joeljwright: I don't know about dev stuff. I don't think there will be any room for swift (or any project), but I think there will be ops sessions. | 17:50 |
| notmyname | joeljwright: ok, I'll add you | 17:51 |
| joeljwright | I'm sure people will still find time to discuss things :) | 17:51 |
| notmyname | joeljwright: as clayg's said before, get a few swift devs in the same city and see if a hackathon doesn't break out | 17:51 |
| notmyname | maybe if it all gets too expo-hall-ish, we can just catch a train up to tdasilva's house and get some work done ;-) | 17:52 |
| joeljwright | :) | 17:52 |
| joeljwright | notmyname: is there a swift meeting today? | 17:56 |
| notmyname | yes. clay has an ops question to raise, and we need to be winding down on the ocata release. also we'll emphasize the ptg planning etherpad | 17:57 |
| notmyname | oops. I didn't update the date on the wiki | 17:57 |
| notmyname | ok, done | 17:57 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/swift master: Updated from global requirements https://review.openstack.org/88736 | 17:58 |
| joeljwright | notmyname: that's why I asked :) | 17:58 |
| *** jamielennox is now known as jamielennox|away | 18:00 | |
| *** tqtran has joined #openstack-swift | 18:04 | |
| openstackgerrit | Joel Wright proposed openstack/swift master: Add Preamble and Postamble to SLO and SegmentedIterable https://review.openstack.org/365371 | 18:05 |
| tdasilva | notmyname: sounds like a great deal! | 18:15 |
| joeljwright | tdasilva: I'm sold on the plan already | 18:15 |
| timburke | eranrom: yeah, my understanding (when last i looked at it) was that OSC doesn't support sending arbitrary headers | 18:15 |
| *** dmorita has quit IRC | 18:16 | |
| tdasilva | joeljwright: i'll fire up the grill and we'll have some great food with that :) | 18:16 |
| joeljwright | :) | 18:17 |
| *** acoles is now known as acoles_ | 18:18 | |
| *** fbo has quit IRC | 18:21 | |
| *** dmorita_ has joined #openstack-swift | 18:25 | |
| timburke | i... what? i go to add a reviewer on a patch, get the error message "Invalid authentication method. In order to authenticate, prefix the REST endpoint URL with /a/ (e.g. http://example.com/a/projects/)." | 18:30 |
| *** stradling has joined #openstack-swift | 18:35 | |
| *** robcresswell has quit IRC | 18:35 | |
| *** kmARC has quit IRC | 18:35 | |
| *** kozhukalov has quit IRC | 18:35 | |
| *** amit213 has quit IRC | 18:35 | |
| *** ogelbukh has quit IRC | 18:35 | |
| *** janonymous has quit IRC | 18:35 | |
| *** ediardo has quit IRC | 18:37 | |
| *** DuncanT has quit IRC | 18:37 | |
| *** cargonza has quit IRC | 18:37 | |
| *** AndyWojo has quit IRC | 18:37 | |
| *** nottrobin has quit IRC | 18:45 | |
| *** serverascode has quit IRC | 18:45 | |
| *** tonanhngo has joined #openstack-swift | 18:45 | |
| *** tonanhngo has quit IRC | 18:47 | |
| *** tonanhngo has joined #openstack-swift | 18:48 | |
| *** stradling has quit IRC | 18:52 | |
| *** stradling has joined #openstack-swift | 18:53 | |
| notmyname | timburke: have you considered using a valid authentication method? | 19:00 |
| *** ChubYann has joined #openstack-swift | 19:05 | |
| eranrom | timburke: yep. that is what I saw. Thanks. | 19:21 |
| *** nottrobin has joined #openstack-swift | 19:29 | |
| clayg | is there a meeting today!? | 19:31 |
| *** DuncanT has joined #openstack-swift | 19:31 | |
| notmyname | there is! hooray! | 19:31 |
| clayg | jrichli: those probetests failures are with handoffs_first enabled in your saio configs maybe? | 19:31 |
| *** serverascode has joined #openstack-swift | 19:32 | |
| clayg | i mean maybe not - they may just be flaky/broken? | 19:32 |
| jrichli | I didn't change my config for that patch yet. so, not enabled. | 19:32 |
| *** nikivi has joined #openstack-swift | 19:32 | |
| clayg | jrichli: k, i'll try to confirm - do they work for you on master currently? | 19:33 |
| jrichli | so they dont fail for you? i will look more closely | 19:33 |
| clayg | we land stuff that breaks probetests not infrequently | 19:33 |
| jrichli | clayg: lol, yes. i will check | 19:33 |
| clayg | jrichli: no i'm sorry - i haven't confirmed that - i will | 19:33 |
| *** kozhukalov has joined #openstack-swift | 19:33 | |
| *** jordanP has quit IRC | 19:34 | |
| *** DuncanT has quit IRC | 19:35 | |
| jrichli | clayg: i just ran one on a different vm, with the patch, and it succeeded. looking farther ... | 19:41 |
| *** ogelbukh has joined #openstack-swift | 19:47 | |
| jrichli | clayg: one difference is it is failing with replication as default, but succeeding for EC. but your changes wouldnt have affected repl path ... so, doubtful that is the thing | 19:47 |
| *** DuncanT has joined #openstack-swift | 19:51 | |
| *** SkyRocknRoll has quit IRC | 19:54 | |
| *** ganders has quit IRC | 19:54 | |
| *** tonanhngo has quit IRC | 19:54 | |
| *** robcresswell has joined #openstack-swift | 19:55 | |
| clayg | jrichli: probetests are the most fun thing evar! | 19:58 |
| jrichli | clayg: that they are! so ... repl vs. ec as default wasn't the thing. so far, test_replication_servers_working.py:TestReplicatorFunctions.test_main fails on both the vms i have tried | 20:02 |
| jrichli | clayg: doh! that one does fail against master! | 20:07 |
| *** stradling has quit IRC | 20:07 | |
| *** tonanhngo has joined #openstack-swift | 20:08 | |
| *** mrda has left #openstack-swift | 20:09 | |
| *** cargonza has joined #openstack-swift | 20:10 | |
| *** amit213 has joined #openstack-swift | 20:11 | |
| jrichli | clayg: yes, i can get both the failures against master. I will remove my -l and keep looking at how to fix those. | 20:12 |
| *** ediardo has joined #openstack-swift | 20:12 | |
| *** AndyWojo has joined #openstack-swift | 20:13 | |
| *** jordanP has joined #openstack-swift | 20:13 | |
| *** janonymous has joined #openstack-swift | 20:14 | |
| clayg | jrichli: you can leave it -1 - i never got around to updating it to deprecate support for handoffs_first in favor of handoffs_only | 20:16 |
| *** kmARC has joined #openstack-swift | 20:19 | |
| *** jordanP has quit IRC | 20:19 | |
| jrichli | clayg: ok. I haven't proved it yet, but I am thinking it was the "Fix race in new partitions detecting new/invalid suffixes." that introduced one of the failures. | 20:20 |
| jrichli | it complains about a behavior of a hashes.invalid file | 20:20 |
| clayg | rofl - that'd be awesome | 20:20 |
| clayg | lp bug #1663016 annoyed me this morning - be kind of a soft introduction to db_replicator if someone was so inclined | 20:25 |
| openstack | Launchpad bug 1663016 in OpenStack Object Storage (swift) "db replicator missing override partitions/devices options" [Medium,Confirmed] https://launchpad.net/bugs/1663016 | 20:25 |
| *** foexle has joined #openstack-swift | 20:27 | |
| *** stradling has joined #openstack-swift | 20:36 | |
| *** pcaruana has quit IRC | 20:43 | |
| tdasilva | jrichli, clayg: i'm seeing two failures on probetests too | 20:44 |
| tdasilva | FAIL: test_delete_propagate (test.probe.test_reconstructor_revert.TestReconstructorRevert and test_main (test.probe.test_replication_servers_working.TestReplicatorFunctions | 20:45 |
| tdasilva | ran into this by chance while running probe test for patch 337297 | 20:46 |
| patchbot | https://review.openstack.org/#/c/337297/ - swift - Add support to increase object ring partition power | 20:46 |
| jrichli | tdasilva: oh, thanks. I just wrote https://bugs.launchpad.net/swift/+bug/1663021 so you can add to that if you like | 20:46 |
| openstack | Launchpad bug 1663021 in OpenStack Object Storage (swift) "TestReplicatorFunctions probetest fails" [Undecided,New] | 20:46 |
| notmyname | meeting in5 minutes in #openstack-meeting | 20:54 |
| clayg | omg omg omg | 20:54 |
| clayg | tdasilva: jrichli: nice work on lp bug #1663021 | 20:57 |
| openstack | Launchpad bug 1663021 in OpenStack Object Storage (swift) "TestReplicatorFunctions probetest fails" [Undecided,Confirmed] https://launchpad.net/bugs/1663021 | 20:57 |
| *** m_kazuhiro has joined #openstack-swift | 20:57 | |
| mattoliverau | morning | 20:58 |
| kota_ | good mornin | 20:58 |
| mattoliverau | kota_: morning | 20:58 |
| notmyname | hello | 20:59 |
| kota_ | mattoliverau: o/ | 20:59 |
| notmyname | meeting started in #openstack-meeting | 21:00 |
| *** acoles_ is now known as acoles | 21:02 | |
| *** stradling has quit IRC | 21:02 | |
| *** joeljwright1 has joined #openstack-swift | 21:02 | |
| *** jamielennox|away is now known as jamielennox | 21:03 | |
| *** stradling has joined #openstack-swift | 21:05 | |
| timburke | i've not had good experiences submitting patches to meetbot. notmyname, if *you* want to make #link work inline, godspeed, but i'm not touching it | 21:05 |
| notmyname | timburke: nope | 21:06 |
| *** silor has quit IRC | 21:10 | |
| *** mmotiani_ has joined #openstack-swift | 21:22 | |
| *** caiobrentano has quit IRC | 21:30 | |
| *** caiobrentano has joined #openstack-swift | 21:31 | |
| *** Jeffrey4l has quit IRC | 21:34 | |
| *** Jeffrey4l has joined #openstack-swift | 21:34 | |
| *** caiobrentano has quit IRC | 21:35 | |
| *** foexle has quit IRC | 21:39 | |
| *** jordanP has joined #openstack-swift | 21:40 | |
| *** mmotian__ has joined #openstack-swift | 21:41 | |
| clayg | tdasilva: should put a soft -1 or something on the part power until you get the probetest question where you want it | 21:42 |
| clayg | I see no reason not to +A this with +2 from timburke mattoliverau and yourself (and anyone else!) | 21:42 |
| tdasilva | clayg: yeah, trying to figure out right now, but i have a hard stop in 15min today | 21:43 |
| tdasilva | so if people want to +A that's fine by me, i'll try to add a comment with the failures i'm seeing | 21:43 |
| mattoliverau | tdasilva: if you could leave a comment about the probe test, when I get round to the patch today, I'll have a play | 21:44 |
| tdasilva | ok | 21:44 |
| *** mmotiani_ has quit IRC | 21:44 | |
| timburke | in case we ever need a logo for hummingbird, let's use https://en.wikipedia.org/wiki/File:Emu-wild.jpg as a starting point | 21:59 |
| joeljwright1 | :D | 21:59 |
| *** stradling has quit IRC | 21:59 | |
| acoles | timburke: nah, https://en.wikipedia.org/wiki/Gibson_Hummingbird#/media/File:Gibson_Hummingbird_2008_-_Vintage_Sunburst.jpeg | 22:01 |
| *** m_kazuhiro has quit IRC | 22:01 | |
| kota_ | acoles, timburke: humming bird? https://youtu.be/Jfat_FReZIE?t=922 | 22:03 |
| jrichli | i guess these are the proposals? https://www.youtube.com/watch?v=JmMTCWyY8Y4 | 22:06 |
| notmyname | jrichli: yeah, those are the rev 1 of them | 22:06 |
| jrichli | i still couldnt find actual links (and attachments were scrubbed for ml history) | 22:06 |
| clayg | jrichli: how reliably does test_replication_servers_working fail for you on master? | 22:07 |
| * kota_ is off for breakfast and will be back at office | 22:07 | |
| clayg | is test_delete_propagate the one that fails because of ssync and expired objects tombstones | 22:07 |
| jrichli | good question. i was so focused on testing to find out if it was a particular patch or config option, i haven't done that bit. will do now | 22:07 |
| jrichli | i just learned of the test_delete_propagate before the meeting from tdasilva | 22:08 |
| clayg | it just passed for me on master with ssync - testing again with rsync | 22:08 |
| clayg | ok, that one failed for me twice in a row | 22:08 |
| clayg | but wasn't one of the two referenced on patch 425493 | 22:09 |
| patchbot | https://review.openstack.org/#/c/425493/ - swift - Make the reconstructor handoffs_first work (and us... | 22:09 |
| jrichli | correct, because that one succeeds for me against that patch | 22:09 |
| clayg | notmyname: you might just keep it in your mind that we *do* want to get probetests working before the release in case we do need to backport something | 22:09 |
| jrichli | i updated the bug with some info on that | 22:09 |
| clayg | oh right - the bug | 22:09 |
| mattoliverau | timburke: yay emu | 22:13 |
| notmyname | mattoliverau: didn't you lose a war to those? | 22:14 |
| jrichli | clayg: so far, I think i hit the test_replication_servers_working every time against master for both the vms i have been using for this. | 22:15 |
| *** JimCheung has quit IRC | 22:15 | |
| clayg | jrichli: try it with ssync - it only started failing for me when I switched to rsync | 22:17 |
| jrichli | clayg: ok, will do | 22:18 |
| *** acoles is now known as acoles_ | 22:19 | |
| jrichli | clayg: success! | 22:22 |
| jrichli | with ssync, that is | 22:22 |
| *** jordanP has quit IRC | 22:27 | |
| *** sams-gleb has quit IRC | 22:28 | |
| jrichli | clayg: rebase of your patch does cause it to fail test_delete_propagate, so i at least have a set of suspect merges | 22:29 |
| clayg | interesting :\ | 22:29 |
| openstackgerrit | Joel Wright proposed openstack/swift master: Add Preamble and Postamble to SLO and SegmentedIterable https://review.openstack.org/365371 | 22:33 |
| *** JimCheung has joined #openstack-swift | 22:37 | |
| mattoliverau | notmyname: we actually did, humans lost the emu war ;) | 22:38 |
| jrichli | clayg, tdasilva: I updated the bug with the latest results i know about. I gotta go now, but will be back on it later. keep me posted. | 22:42 |
| clayg | jrichli: thanks! | 22:43 |
| *** dmorita_ has quit IRC | 22:49 | |
| *** dmorita has joined #openstack-swift | 22:49 | |
| *** JimCheung has quit IRC | 23:02 | |
| *** joeljwright1 has quit IRC | 23:02 | |
| *** JimCheung has joined #openstack-swift | 23:02 | |
| *** dja has joined #openstack-swift | 23:09 | |
| *** lespaul has joined #openstack-swift | 23:09 | |
| *** tonanhngo has quit IRC | 23:12 | |
| *** joeljwright1 has joined #openstack-swift | 23:21 | |
| *** joeljwright1 has quit IRC | 23:22 | |
| *** catintheroof has quit IRC | 23:25 | |
| *** sams-gleb has joined #openstack-swift | 23:28 | |
| *** mmotian__ has quit IRC | 23:29 | |
| *** sams-gleb has quit IRC | 23:33 | |
| *** chsc has quit IRC | 23:36 | |
| *** kei_yama has joined #openstack-swift | 23:36 | |
| *** JimCheung has quit IRC | 23:44 | |
| *** JimCheung has joined #openstack-swift | 23:45 | |
| *** dja has quit IRC | 23:46 | |
| *** dja has joined #openstack-swift | 23:48 | |
| *** klamath has quit IRC | 23:52 | |
| *** lespaul has quit IRC | 23:55 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
