| notmyname | and all the unscheduled stuff is below the lines at the bottom ater the friday section | 00:00 |
|---|---|---|
| *** minwoob has quit IRC | 00:02 | |
| mattoliverau | notmyname: i see it.. my ehterpad tab was out of date :P | 00:04 |
| notmyname | or rather, "eventually consistent"? | 00:05 |
| mattoliverau | notmyname: where could combine fast-post and updateable object sysmeta spec maybe | 00:07 |
| mattoliverau | or do we want to talk about testing | 00:08 |
| notmyname | yeah, I was thinking of combining those two | 00:10 |
| notmyname | the testing one was intended to be more as an intro to what/how to test swift, especially for newer contributors. not something that's good for a working session | 00:12 |
| notmyname | ok, I'll check back tomorrow. i'm going home now | 00:12 |
| *** NM has quit IRC | 00:16 | |
| mattoliverau | notmyname: k, have a great evening :) | 00:28 |
| *** ho has joined #openstack-swift | 00:29 | |
| *** zhill_ has quit IRC | 00:43 | |
| -openstackstatus- NOTICE: Restarted gerrit due to stuck stream-events connections. Events since 23:49 were missed and changes uploaded since then will need to be rechecked. | 00:43 | |
| *** mgagne has joined #openstack-swift | 00:53 | |
| mgagne | I'm currently trying to setup tempest to test swift and I'm getting this error: "Insufficient number of users provided. ObjectExpiryTest,PublicObjectTest have allocated all the credentials for this allocation request" Anyone with tempest experience who could give me a hand? | 00:55 |
| mgagne | I have accounts.yaml setup with 2 accounts with the operator_role role assigned to them. Do I need more accounts? | 00:56 |
| mgagne | ok, somehow I had garbage left in my lock_path folder. I cleaned it and it worked. thanks | 01:01 |
| zaitcev | odd, but all is well that ends well | 01:12 |
| mgagne | zaitcev: in fact, it looks like tempest leaves lock behind. If I run tempest a second time, tests start to fail again | 01:15 |
| *** haigang has joined #openstack-swift | 01:53 | |
| *** jrichli has quit IRC | 01:56 | |
| *** panbalag has quit IRC | 01:57 | |
| *** haigang has quit IRC | 02:05 | |
| *** wbhuber has joined #openstack-swift | 02:32 | |
| *** bill_az has quit IRC | 02:39 | |
| *** vinsh has joined #openstack-swift | 02:53 | |
| *** fifieldt has joined #openstack-swift | 03:11 | |
| *** NM has joined #openstack-swift | 03:13 | |
| *** wbhuber has quit IRC | 03:19 | |
| *** fifieldt has quit IRC | 03:20 | |
| *** NM has quit IRC | 03:28 | |
| *** vinsh has quit IRC | 03:31 | |
| *** vinsh has joined #openstack-swift | 03:32 | |
| *** vinsh has quit IRC | 03:36 | |
| *** tdasilva has quit IRC | 03:41 | |
| *** links has joined #openstack-swift | 03:41 | |
| *** aix has joined #openstack-swift | 03:44 | |
| *** zhill_ has joined #openstack-swift | 03:50 | |
| *** aix has quit IRC | 03:50 | |
| *** zhill_ has quit IRC | 03:51 | |
| *** tdasilva has joined #openstack-swift | 04:00 | |
| *** aix has joined #openstack-swift | 04:17 | |
| *** proteusguy has quit IRC | 04:24 | |
| *** proteusguy has joined #openstack-swift | 04:29 | |
| *** portante has quit IRC | 04:40 | |
| *** portante has joined #openstack-swift | 04:45 | |
| peluse | wbhuber, so I looked and don't have a quick answer for you - will mess with it some more in the morn and we may have ot grab someone else for some more advanced mocking | 04:46 |
| *** silor has joined #openstack-swift | 04:58 | |
| *** bkopilov has quit IRC | 05:04 | |
| *** tamizh_geek has joined #openstack-swift | 05:05 | |
| *** SkyRocknRoll has joined #openstack-swift | 05:10 | |
| *** proteusguy has quit IRC | 05:11 | |
| *** bkopilov has joined #openstack-swift | 05:13 | |
| *** bkopilov has quit IRC | 05:18 | |
| *** bkopilov has joined #openstack-swift | 05:28 | |
| *** bkopilov has quit IRC | 05:33 | |
| *** bkopilov has joined #openstack-swift | 05:34 | |
| *** ppai has joined #openstack-swift | 05:40 | |
| *** early has quit IRC | 05:49 | |
| *** early has joined #openstack-swift | 05:59 | |
| charz | notmyname: 20mins(run_seconds) for ec_huge_test scenarios are too long, it almost took 3+ hours to finish a run. I dropped the tests. | 06:32 |
| *** zaitcev has quit IRC | 06:35 | |
| *** yuanz has joined #openstack-swift | 06:36 | |
| *** bkopilov has quit IRC | 06:36 | |
| charz | notmyname: I'm going to change run_seconds to operation_count in ec_huge_test. And also reduce the number of run_secondes, I'll start from 120s, 240s, 480s to 20mins. | 06:37 |
| charz | notmyname: And see which one is close to result(20mins). And I'll use the number for run_seconds. | 06:39 |
| *** yuan has quit IRC | 06:39 | |
| *** bkopilov has joined #openstack-swift | 06:57 | |
| *** joeljwright has quit IRC | 06:58 | |
| *** geaaru has joined #openstack-swift | 07:31 | |
| openstackgerrit | Hisashi Osanai proposed openstack/swift: WIP: Enable Role-based access control using oslo.policy in Swift https://review.openstack.org/149930 | 07:48 |
| *** tamizh_geek has quit IRC | 07:49 | |
| *** tamizh_geek has joined #openstack-swift | 07:50 | |
| openstackgerrit | Hisashi Osanai proposed openstack/swift: WIP: Enable Role-based access control using oslo.policy in Swift https://review.openstack.org/149930 | 07:52 |
| *** jistr has joined #openstack-swift | 07:59 | |
| *** acoles_away is now known as acoles | 08:13 | |
| *** jordanP has joined #openstack-swift | 08:14 | |
| acoles | notmyname: ack no meeting today | 08:15 |
| *** haomaiw__ has quit IRC | 08:21 | |
| *** chlong has quit IRC | 08:22 | |
| *** ppai has quit IRC | 08:41 | |
| *** proteusguy has joined #openstack-swift | 08:43 | |
| cschwede | notmyname: just one question regarding summit schedule: the test framework has been moved to Friday, so it will be more like a working session then? | 08:47 |
| *** kota_ has joined #openstack-swift | 08:47 | |
| kota_ | notmyname: ack for no meeting today. Thanks for the information :) | 08:48 |
| acoles | kota_: you get more sleep ;) | 08:49 |
| kota_ | acoles: thanks, today is the last day of the long holidays in Japan, so I'll enjoy it :D | 08:49 |
| acoles | yay! | 08:50 |
| *** haomaiwa_ has joined #openstack-swift | 08:50 | |
| kota_ | k, leaving for dinner, see you tomorrow | 08:51 |
| *** kota_ has quit IRC | 08:51 | |
| *** ppai has joined #openstack-swift | 08:55 | |
| *** clduser_ has quit IRC | 08:58 | |
| *** swifterdarrell has quit IRC | 08:58 | |
| *** torgomatic has quit IRC | 08:58 | |
| *** rsFF has quit IRC | 08:58 | |
| *** clduser_ has joined #openstack-swift | 09:00 | |
| *** swifterdarrell has joined #openstack-swift | 09:00 | |
| *** torgomatic has joined #openstack-swift | 09:00 | |
| *** rsFF has joined #openstack-swift | 09:00 | |
| *** sendak.freenode.net sets mode: +vv swifterdarrell torgomatic | 09:00 | |
| *** bkopilov has quit IRC | 09:07 | |
| *** bkopilov has joined #openstack-swift | 09:08 | |
| *** joeljwright has joined #openstack-swift | 09:13 | |
| *** bkopilov has quit IRC | 09:14 | |
| *** ho has quit IRC | 09:14 | |
| *** bkopilov has joined #openstack-swift | 09:15 | |
| *** bkopilov has quit IRC | 09:21 | |
| *** bkopilov has joined #openstack-swift | 09:21 | |
| *** bkopilov has quit IRC | 09:29 | |
| *** bkopilov has joined #openstack-swift | 09:30 | |
| *** bkopilov has quit IRC | 09:35 | |
| *** joeljwright has quit IRC | 10:03 | |
| openstackgerrit | Merged openstack/swift: Fix account replication during pre-storage-policy upgrade https://review.openstack.org/177964 | 10:17 |
| *** panbalag has joined #openstack-swift | 10:22 | |
| *** joeljwright has joined #openstack-swift | 10:35 | |
| *** ppai has quit IRC | 11:16 | |
| *** ppai has joined #openstack-swift | 11:33 | |
| *** aix has quit IRC | 11:38 | |
| *** bkopilov has joined #openstack-swift | 11:38 | |
| *** mherdad has joined #openstack-swift | 12:01 | |
| mherdad | swift client check number of replication deploy ? | 12:02 |
| *** joeljwright has quit IRC | 12:06 | |
| *** SkyRocknRoll has quit IRC | 12:10 | |
| *** ppai has quit IRC | 12:13 | |
| *** erlon has joined #openstack-swift | 12:15 | |
| *** thurloat_isgone is now known as thurloat | 12:20 | |
| *** gyee has joined #openstack-swift | 12:22 | |
| *** SkyRocknRoll has joined #openstack-swift | 12:23 | |
| *** proteusguy has quit IRC | 12:23 | |
| *** mherdad has quit IRC | 12:26 | |
| *** mherdad has joined #openstack-swift | 12:27 | |
| *** aix has joined #openstack-swift | 12:27 | |
| *** NM has joined #openstack-swift | 12:40 | |
| *** tamizh_geek has quit IRC | 12:46 | |
| *** tamizh_geek has joined #openstack-swift | 12:46 | |
| *** proteusguy has joined #openstack-swift | 12:50 | |
| *** tamizh_geek has quit IRC | 12:51 | |
| *** openstackgerrit has quit IRC | 12:53 | |
| *** openstackgerrit has joined #openstack-swift | 12:53 | |
| *** links has quit IRC | 12:54 | |
| *** ppai has joined #openstack-swift | 12:58 | |
| *** tab__ has joined #openstack-swift | 12:59 | |
| *** bill_az has joined #openstack-swift | 13:00 | |
| *** joeljwright has joined #openstack-swift | 13:10 | |
| *** mherdad has quit IRC | 13:17 | |
| *** esker has joined #openstack-swift | 13:26 | |
| *** joeljwright has quit IRC | 13:34 | |
| *** openstackgerrit has quit IRC | 13:38 | |
| *** openstackgerrit has joined #openstack-swift | 13:38 | |
| *** gyee has quit IRC | 13:45 | |
| *** gyee has joined #openstack-swift | 13:46 | |
| *** zaitcev has joined #openstack-swift | 14:02 | |
| *** ChanServ sets mode: +v zaitcev | 14:02 | |
| *** shakamunyi has joined #openstack-swift | 14:09 | |
| *** Nadeem has joined #openstack-swift | 14:12 | |
| *** shakamunyi has quit IRC | 14:13 | |
| *** wbhuber has joined #openstack-swift | 14:16 | |
| *** minwoob has joined #openstack-swift | 14:35 | |
| *** shakamunyi has joined #openstack-swift | 14:35 | |
| *** shakamunyi has quit IRC | 14:36 | |
| *** shakamunyi has joined #openstack-swift | 14:36 | |
| *** vinsh has joined #openstack-swift | 14:40 | |
| *** joeljwright has joined #openstack-swift | 14:42 | |
| *** jrichli has joined #openstack-swift | 14:50 | |
| *** ppai has quit IRC | 14:50 | |
| *** gyee has quit IRC | 15:03 | |
| *** gyee has joined #openstack-swift | 15:06 | |
| *** marcusvrn_ has joined #openstack-swift | 15:07 | |
| *** aix has quit IRC | 15:09 | |
| *** joeljwright has quit IRC | 15:30 | |
| *** Nadeem has quit IRC | 15:33 | |
| *** openstackgerrit has quit IRC | 15:37 | |
| *** openstackgerrit has joined #openstack-swift | 15:37 | |
| *** joeljwright has joined #openstack-swift | 15:39 | |
| *** silor has quit IRC | 15:50 | |
| *** SkyRocknRoll has quit IRC | 15:56 | |
| *** silor has joined #openstack-swift | 15:57 | |
| *** shakamunyi has quit IRC | 15:58 | |
| *** jordanP has quit IRC | 15:59 | |
| *** tamizh_geek has joined #openstack-swift | 16:08 | |
| *** silor has quit IRC | 16:10 | |
| *** proteusguy has quit IRC | 16:13 | |
| *** gyee has quit IRC | 16:13 | |
| *** tab_ has joined #openstack-swift | 16:20 | |
| *** Fin1te has joined #openstack-swift | 16:21 | |
| *** tab_ has quit IRC | 16:22 | |
| *** proteusguy has joined #openstack-swift | 16:25 | |
| *** cdelatte has quit IRC | 16:26 | |
| *** jistr has quit IRC | 16:29 | |
| *** cdelatte has joined #openstack-swift | 16:33 | |
| notmyname | good morning | 16:36 |
| notmyname | cschwede: no, for now that fishbowl session is in the "overflow" bucket. if we talk about it in friday, great. but it's not scheduled | 16:37 |
| notmyname | reminder that there is no meeting today | 16:37 |
| *** tamizh_g_ has joined #openstack-swift | 16:41 | |
| *** joeljwright has quit IRC | 16:41 | |
| *** tab_ has joined #openstack-swift | 16:42 | |
| *** marcusvrn_ is now known as marcusvrn | 16:42 | |
| *** tamizh_geek has quit IRC | 16:43 | |
| *** jordanP has joined #openstack-swift | 16:46 | |
| *** wolsen_ is now known as wolsen | 16:47 | |
| *** shakamunyi has joined #openstack-swift | 16:49 | |
| *** Nadeem has joined #openstack-swift | 16:50 | |
| *** zhill_ has joined #openstack-swift | 16:52 | |
| *** fthiagogv has joined #openstack-swift | 16:56 | |
| notmyname | acoles: http://xkcd.com/1521/ | 17:04 |
| *** Fin1te has quit IRC | 17:04 | |
| *** shakamunyi has quit IRC | 17:05 | |
| *** gyee has joined #openstack-swift | 17:08 | |
| acoles | notmyname: lol | 17:08 |
| *** tamizh_geek has joined #openstack-swift | 17:09 | |
| hurricanerix | morning | 17:09 |
| *** tamizh_g_ has quit IRC | 17:10 | |
| *** Nadeem has quit IRC | 17:16 | |
| *** shakamunyi has joined #openstack-swift | 17:19 | |
| *** gyee has quit IRC | 17:22 | |
| openstackgerrit | Merged openstack/swift: Don't quarantine on missing data files https://review.openstack.org/180317 | 17:27 |
| *** bill_az has quit IRC | 17:27 | |
| *** marcusvrn has quit IRC | 17:29 | |
| *** tamizh_g_ has joined #openstack-swift | 17:32 | |
| *** tamizh_geek has quit IRC | 17:33 | |
| openstackgerrit | Alistair Coles proposed openstack/swift: Add POST capability to ssync for .meta files https://review.openstack.org/138498 | 17:42 |
| *** acoles is now known as acoles_away | 17:43 | |
| *** gyee has joined #openstack-swift | 17:46 | |
| openstackgerrit | Christian Schwede proposed openstack/swift: Add missing statsd metrics section for object-reconstructor https://review.openstack.org/180655 | 17:54 |
| *** jkugel has joined #openstack-swift | 17:57 | |
| *** Nadeem_ has joined #openstack-swift | 17:57 | |
| *** Nadeem_ has quit IRC | 17:58 | |
| *** tamizh_geek has joined #openstack-swift | 18:05 | |
| *** tamizh_g_ has quit IRC | 18:05 | |
| *** Nadeem_ has joined #openstack-swift | 18:07 | |
| *** Nadeem_ has quit IRC | 18:15 | |
| *** Fin1te has joined #openstack-swift | 18:23 | |
| *** jordanP has quit IRC | 18:34 | |
| *** gyee has quit IRC | 18:37 | |
| *** geaaru has quit IRC | 18:39 | |
| *** fthiagogv has quit IRC | 18:45 | |
| *** jkugel has quit IRC | 18:47 | |
| *** zhill_ has quit IRC | 18:47 | |
| *** openstackgerrit has quit IRC | 18:47 | |
| *** vinsh has quit IRC | 18:47 | |
| *** clduser_ has quit IRC | 18:48 | |
| *** swifterdarrell has quit IRC | 18:48 | |
| *** torgomatic has quit IRC | 18:48 | |
| *** rsFF has quit IRC | 18:48 | |
| *** bsdkurt1 has quit IRC | 18:48 | |
| *** morganfainberg has quit IRC | 18:48 | |
| *** alpha_ori has quit IRC | 18:48 | |
| *** redbo has quit IRC | 18:48 | |
| *** tanee has quit IRC | 18:48 | |
| *** early has quit IRC | 18:48 | |
| *** wer has quit IRC | 18:48 | |
| *** raginbajin has quit IRC | 18:48 | |
| *** briancline has quit IRC | 18:48 | |
| *** InAnimaTe has quit IRC | 18:48 | |
| *** early has joined #openstack-swift | 18:49 | |
| *** bsdkurt1 has joined #openstack-swift | 18:49 | |
| *** wer has joined #openstack-swift | 18:49 | |
| *** tanee has joined #openstack-swift | 18:49 | |
| *** redbo has joined #openstack-swift | 18:49 | |
| *** alpha_ori has joined #openstack-swift | 18:49 | |
| *** morganfainberg has joined #openstack-swift | 18:49 | |
| *** raginbajin has joined #openstack-swift | 18:49 | |
| *** briancline has joined #openstack-swift | 18:49 | |
| *** InAnimaTe has joined #openstack-swift | 18:49 | |
| *** sendak.freenode.net sets mode: +v redbo | 18:49 | |
| *** jkugel has joined #openstack-swift | 18:49 | |
| *** zhill_ has joined #openstack-swift | 18:49 | |
| *** openstackgerrit has joined #openstack-swift | 18:49 | |
| *** vinsh has joined #openstack-swift | 18:49 | |
| *** clduser_ has joined #openstack-swift | 18:49 | |
| *** swifterdarrell has joined #openstack-swift | 18:49 | |
| *** torgomatic has joined #openstack-swift | 18:49 | |
| *** rsFF has joined #openstack-swift | 18:49 | |
| *** sendak.freenode.net sets mode: +vv swifterdarrell torgomatic | 18:49 | |
| dencaval | Guys, Isn't possible to create bp for swift-bench at launchpad? | 18:51 |
| torgomatic | dencaval: blueprints are terrible; if you want people to actually see your words and discuss them, submit a spec to the swift-specs repository | 18:52 |
| dencaval | torgomatic Hm, I'll do it. Thanks. | 18:54 |
| *** minwoob_ has joined #openstack-swift | 19:02 | |
| *** minwoob has quit IRC | 19:03 | |
| *** wbhuber_ has joined #openstack-swift | 19:04 | |
| *** annegentle has joined #openstack-swift | 19:06 | |
| *** wbhuber has quit IRC | 19:06 | |
| *** gyee has joined #openstack-swift | 19:07 | |
| *** zul has quit IRC | 19:14 | |
| *** silor has joined #openstack-swift | 19:16 | |
| *** zul has joined #openstack-swift | 19:16 | |
| *** wbhuber_ has quit IRC | 19:18 | |
| *** bsdkurt1 has quit IRC | 19:18 | |
| *** morganfainberg has quit IRC | 19:18 | |
| *** alpha_ori has quit IRC | 19:18 | |
| *** redbo has quit IRC | 19:18 | |
| *** tanee has quit IRC | 19:18 | |
| *** annegentle has quit IRC | 19:19 | |
| *** wbhuber_ has joined #openstack-swift | 19:19 | |
| *** bsdkurt1 has joined #openstack-swift | 19:19 | |
| *** tanee has joined #openstack-swift | 19:19 | |
| *** redbo has joined #openstack-swift | 19:19 | |
| *** alpha_ori has joined #openstack-swift | 19:19 | |
| *** morganfainberg has joined #openstack-swift | 19:19 | |
| *** sendak.freenode.net sets mode: +v redbo | 19:19 | |
| *** annegentle has joined #openstack-swift | 19:19 | |
| *** Nadeem has joined #openstack-swift | 19:19 | |
| zaitcev | no meeting today? | 19:20 |
| *** morganfainberg has quit IRC | 19:20 | |
| *** morganfainberg has joined #openstack-swift | 19:21 | |
| notmyname | no meeting today | 19:22 |
| *** Fin1te has quit IRC | 19:25 | |
| ahale | hmm, so im failing to install a working saio today :( | 19:32 |
| redbo | liberasurecode? | 19:33 |
| ahale | nope, recon middleware import error weirdness | 19:34 |
| ahale | 'ImportError: No module named middleware.recon' | 19:34 |
| *** silor has quit IRC | 19:35 | |
| ahale | well i didnt need recon anyway, I only wanted a single object-sever and its fine without that in pipeline | 19:35 |
| openstackgerrit | Janie Richling proposed openstack/swift: WIP - working on the encryption feature. https://review.openstack.org/157907 | 19:45 |
| *** dencaval has quit IRC | 19:55 | |
| clayg | notmyname: torgomatic: I went ahead and opened lp bug # 1452431 for the ring issue | 19:59 |
| clayg | notmyname: torgomatic: although I also re-validated that the qa cluster isn't suffereing from lp bug #1452431 | 20:00 |
| openstack | Launchpad bug 1452431 in OpenStack Object Storage (swift) "some parts replicas assigned to duplicate devices" [Undecided,New] https://launchpad.net/bugs/1452431 | 20:00 |
| notmyname | clayg: thanks | 20:04 |
| *** wbhuber__ has joined #openstack-swift | 20:09 | |
| *** tamizh_geek has quit IRC | 20:10 | |
| *** thurloat is now known as thurloat_isgone | 20:12 | |
| *** wbhuber_ has quit IRC | 20:12 | |
| *** gyee has quit IRC | 20:13 | |
| *** breitz has quit IRC | 20:16 | |
| *** breitz has joined #openstack-swift | 20:16 | |
| *** tamizh_geek has joined #openstack-swift | 20:17 | |
| *** zhill has quit IRC | 20:18 | |
| *** Nadeem has quit IRC | 20:28 | |
| *** tamizh_geek has quit IRC | 20:36 | |
| *** bkopilov has quit IRC | 20:46 | |
| mattoliverau | Morning all, I'm up early cause I get to wait at the airport for a flight. Another day, another city :p | 20:57 |
| jrichli | morning! where are you going? | 20:58 |
| *** bkopilov has joined #openstack-swift | 21:00 | |
| mattoliverau | Just up to Canberra, the city I was born and raised in. Have a house tenanted out, tenants are moving out so I need to go and inspect the property. I lead an exciting life :p | 21:01 |
| mattoliverau | I'll stay up for most the weekend and see family while I'm there (and hack from cafes of course) | 21:01 |
| jrichli | I hope you find good things at the house :-) Have fun with fam | 21:04 |
| notmyname | mattoliverau: hope you have a safe flight | 21:05 |
| mattoliverau | Thanks y'all | 21:11 |
| *** esker has quit IRC | 21:28 | |
| *** david-lyle has quit IRC | 21:29 | |
| *** InAnimaTe has quit IRC | 21:34 | |
| *** DCWillia_ has joined #openstack-swift | 21:37 | |
| *** NM has quit IRC | 21:46 | |
| clayg | notmyname: lp bug #1452468 is the thing for the PUT with the post_quorum_timeout setting - hopefully torgomatic can pick it up | 21:52 |
| openstack | Launchpad bug 1452468 in OpenStack Object Storage (swift) "ec PUT post commit response is to quick" [Undecided,New] https://launchpad.net/bugs/1452468 | 21:52 |
| clayg | notmyname: I need to bounce off and look at swifterdarrell's segfault thing | 21:52 |
| notmyname | ok | 21:52 |
| notmyname | thanks for the test | 21:53 |
| notmyname | I hope torgomatic and acoles_away can think on it | 21:53 |
| clayg | I think this is another one of those bugs acoles_away already told us about :) | 21:54 |
| notmyname | torgomatic: acoles_away: peluse: also, here are some logs for a GET that show some weird behavior (independent of the PUT issue in the bug above: https://gist.github.com/notmyname/3179e59c4a01b7bdc871 | 21:55 |
| clayg | there was one with timestamps or etags or something in the reconstructor and peluse and I were like - nah it's fine - I think we did the same thing to him with "if final_phase and need_quorum" and he was like - I think final_phase always means not need_quorum? | 21:55 |
| notmyname | clayg: raising the post put timeout has really helped, but not completely solved it: https://gist.github.com/notmyname/8ab7131d3e948a7dfecd | 21:56 |
| *** jrichli has quit IRC | 21:57 | |
| clayg | notmyname: well the way youhave it broken down there i can't tell if those errors are maybe on GET's right? | 21:58 |
| notmyname | yeah, they could be | 21:58 |
| clayg | the GET is a seperate issue - I think there's some overwrites in there and we know the ECObjectController is sorta per-snickity about mixed etags? | 21:58 |
| *** jkugel has quit IRC | 21:58 | |
| peluse | just reading the PUT issue now... | 22:07 |
| notmyname | in-office conversation has speculated that its "overwrites" that are causing the errors, so I'm going to try to prove that with a new run | 22:07 |
| notmyname | my plan, for those who are curious, is to have all this info available to anyone who wants it (and maybe even a pretty summary) at the summit | 22:07 |
| notmyname | hmm..but there were a few errors on a read-only test | 22:07 |
| *** thurloat_isgone is now known as thurloat | 22:09 | |
| peluse | clayg, yeah the final phase never needs quorum. those two flags were just added to reuse _get_put_responses for sending data and commit | 22:09 |
| peluse | which makes me question my memory as to why we needed both :) | 22:10 |
| mattoliverau | I'm turning the phone off, be back online later | 22:15 |
| *** tab_ has quit IRC | 22:17 | |
| clayg | peluse: I think when it was all smushed together with the replicated PUT path it made sense - now it's all crazy town | 22:20 |
| clayg | I think I broke it when I merged to master cause I tried to get cleaver and make the policy.quorum a property instead of a function that takes a number - idk | 22:21 |
| peluse | clayg, yeah, I'm looking at a few things I don't quite "get" anymore. I do recall the quorum confusion there towards the end... | 22:24 |
| *** DCWillia_ has quit IRC | 22:27 | |
| peluse | hmmm, I guess post_quorum_timeout was a pre-EC thing | 22:27 |
| *** DCWillia_ has joined #openstack-swift | 22:28 | |
| peluse | clayg, have already traced through the PUT path line by line to make sure none of these flags got fliped and/or quorom related variables got misused during all the PUT path refactoring? | 22:30 |
| notmyname | here's more on the GET error. has a traceback in it: https://gist.github.com/notmyname/0d6874edf9c2b8650312 | 22:30 |
| clayg | notmyname: that may yet a different issue? it looks sorta like a chunk read timeout? | 22:33 |
| *** DCWillia_ has quit IRC | 22:33 | |
| peluse | notmyname, hmmm, somehow we call decode before having all our frags? that's interesting too | 22:33 |
| notmyname | I found it when I was trying to find the 6 errors in the read only test (from above) | 22:33 |
| clayg | peluse: yeah I thought there was an explicit test for len(fragments) > num_needed_to_decode that will fail the request (i.e. stop sending segments) before calling decode. | 22:34 |
| clayg | notmyname: maybe it was a duplicated fragment index in the payload - do you have the reconstructor running while you're benchmarking? | 22:35 |
| notmyname | "No object-reconstructor running" | 22:36 |
| notmyname | actually that surprised me. I expected it to be running | 22:36 |
| *** annegentle has quit IRC | 22:38 | |
| peluse | I've got to take off for a few hrs. Keep me posted, I have some time tonight if needed and again on Fri but Thu is totally shot to hell for me | 22:38 |
| peluse | is this only repro'able so far on a real cluster under load? | 22:39 |
| notmyname | peluse: it's the only place I've tried. | 22:39 |
| peluse | cool, back later... | 22:41 |
| notmyname | ok, updated the scenarios to remove updates and restarted the tests | 22:43 |
| *** DCWillia_ has joined #openstack-swift | 22:44 | |
| *** annegentle has joined #openstack-swift | 22:45 | |
| *** wbhuber__ has quit IRC | 22:46 | |
| *** DCWillia_ has quit IRC | 22:47 | |
| *** DCWillia_ has joined #openstack-swift | 22:48 | |
| *** DCWilliams_VA has joined #openstack-swift | 22:51 | |
| *** DCWillia_ has quit IRC | 22:52 | |
| *** DCWilliams_VA has quit IRC | 22:56 | |
| *** macredcape has quit IRC | 22:56 | |
| torgomatic | peluse: doesn't the EC commit phase need a quorum of 2? as in, if we only manage to drop 0 or 1 .durable files, it's not enough? | 23:00 |
| *** annegentle has quit IRC | 23:02 | |
| notmyname | clayg: nope :-( | 23:03 |
| clayg | torgomatic: well i mean - yeah that's how the code wants to work - but I think I broke it - sorry | 23:03 |
| clayg | oh... no i misread you | 23:04 |
| clayg | I mean I think we need more than 0 | 23:04 |
| torgomatic | notmyname: clayg: yeah, that'd have to be it... there's no way to pass fewer than ec_ndta items through the GET code, so it'd have to be a duplicated index or something that makes pyeclib throw data away | 23:04 |
| clayg | and in a system that might fail - I think more than 1 is good | 23:04 |
| notmyname | clayg: my "nope" was wrt the Update phase doing the errors | 23:04 |
| torgomatic | clayg: yeah, which is why we picked 2, right? or something like ec_ndata // 2 + 1 ? | 23:04 |
| clayg | I guess someone picked 2 - but it turns out there was some plumbing that got horked or maybe it only always looked liked it worked because commits were quick and post_quorum_timeout was long enough | 23:05 |
| torgomatic | clayg: so your assertion is that we're landing ec_ndata + 1 FAs on disk okay, but then we're looking to land ec_ndata + 1 .durable files on disk, but they don't complete quickly enough? | 23:05 |
| torgomatic | (just so I'm clear) | 23:05 |
| notmyname | https://gist.github.com/notmyname/3738929f75b25dd06c71 | 23:05 |
| clayg | torgomatic: well... I think the test attached to the bug is pretty clear on the *failure* - I'm less sure about how we go from "this doesn't work" -> "how it should work is this" -> "here this works like we want" | 23:07 |
| clayg | because - well I don't know how it should work :\ | 23:07 |
| clayg | it should not 503 | 23:07 |
| notmyname | ok, same read issue. gets 6 200s and some 404s and returns 503 | 23:07 |
| notmyname | no traceback | 23:08 |
| clayg | notmyname: oh ok, so no object-reconstructor running - no overwrites - still getting 503's | 23:09 |
| clayg | notmyname: but you're not seeing the liberasure "not enough fragments" trackebacks? It's just 6 200's from the object servers and then a 503 | 23:09 |
| notmyname | in this case, no liberasure string in the entire logs | 23:10 |
| notmyname | well, wait | 23:11 |
| notmyname | maybe I can lurn to grep | 23:11 |
| torgomatic | clayg: yeah, I'm having trouble understanding the bug from that unit test :( | 23:11 |
| notmyname | I see the liberasure error some. not many times. definitely way less than the number of errors I have | 23:12 |
| *** ondergetekende has quit IRC | 23:13 | |
| notmyname | https://gist.github.com/notmyname/b0bd85c0b068e0eec895 <-- 404s 200s, resulting in a 503 (with the liberasure traceback) | 23:14 |
| *** asettle is now known as asettle-bfast | 23:14 | |
| *** asettle-bfast has quit IRC | 23:14 | |
| notmyname | oh, interesting | 23:15 |
| *** ondergetekende has joined #openstack-swift | 23:15 | |
| *** chlong has joined #openstack-swift | 23:16 | |
| notmyname | the requests with the tracebacks don't have a proxy log line. the lines that had the proxy log a 503 don't have a traceback | 23:18 |
| * notmyname isn't sure if that's interesting or not | 23:18 | |
| clayg | torgomatic: so if you PUT and some of the commits are slower than post_quorum timeout - you get a 503 | 23:19 |
| *** zhill_ has quit IRC | 23:20 | |
| clayg | so in that unittest you can either make post_quorum_timeout longer - or make more of the requests not be slow to commit - and it will pass | 23:21 |
| clayg | but it'd be cool like if it worked like for realzy on jd's cluster | 23:21 |
| torgomatic | clayg: maybe something like this? https://gist.github.com/smerritt/d8ca1e378c23d66c65e0 | 23:21 |
| torgomatic | instead of needing $quorum, then 2, then $quorum again | 23:21 |
| * torgomatic goes to try it with that unit test | 23:21 | |
| notmyname | not just my cluster. it's everyone's cluster ;-) /music plays. and credits | 23:21 |
| *** proteusguy has quit IRC | 23:22 | |
| *** vinsh has quit IRC | 23:23 | |
| torgomatic | notmyname: does that mean I get to go home now? | 23:25 |
| clayg | torgomatic: maybe - how'd your patch go? | 23:25 |
| notmyname | torgomatic: you're already home! | 23:25 |
| clayg | torgomatic: wait - aren :D | 23:25 |
| clayg | beat me | 23:25 |
| *** kei_yama has joined #openstack-swift | 23:26 | |
| torgomatic | heh, my house is full of kids. I'm at a coffee shop for my own sanity. | 23:26 |
| torgomatic | clayg: so that patch makes your test pass | 23:26 |
| torgomatic | I'm gonna refine it a tiny bit here; it could really use a comment or two | 23:27 |
| clayg | torgomatic: for sure! | 23:27 |
| clayg | torgomatic: i wasn't even sure what I was doing really besides just making a unittest that did like the qa cluster and watching it 503 | 23:27 |
| torgomatic | well, I was gonna update my gist, but I'll just chuck it into Gerrit and see what happens | 23:29 |
| openstackgerrit | Samuel Merritt proposed openstack/swift: EC: don't 503 on marginally-successful PUT https://review.openstack.org/180795 | 23:36 |
| torgomatic | clayg: alright, try that on for size | 23:36 |
| notmyname | what am I missing? | 23:37 |
| notmyname | that renames a variable and adds a comment | 23:37 |
| torgomatic | notmyname: isn't there a unit test in there? | 23:37 |
| torgomatic | or did I screw that up? | 23:37 |
| torgomatic | no, looks like it's there | 23:37 |
| notmyname | yeah, the test file changes are there | 23:38 |
| notmyname | I was just looking at the non-test first | 23:38 |
| torgomatic | SUBTLETY! | 23:38 |
| clayg | torgomatic: wait wait? | 23:38 |
| torgomatic | clayg: what'd I do? | 23:38 |
| clayg | so you didn't need to *fix* anything to make the test pass? | 23:38 |
| notmyname | yeah, that's what I'm saying | 23:39 |
| torgomatic | heh, min_conns is defined before that try: and is used both before and after it | 23:39 |
| torgomatic | I'm updating it with a new value | 23:39 |
| torgomatic | the minimum goes down once it's time for .durable files | 23:39 |
| * clayg 's head explodes | 23:40 | |
| *** ho has joined #openstack-swift | 23:44 | |
| ho | good morning! | 23:44 |
| notmyname | torgomatic: clayg: I'm going to apply that patch live and see what happens | 23:44 |
| clayg | notmyname: is doing it live! | 23:45 |
| clayg | notmyname: only think it will help with is you can lower your post_quroum_timeout back down | 23:45 |
| torgomatic | notmyname: http://www.strengthleader.com/wp-content/uploads/2014/07/Jenga.jpg | 23:46 |
| *** david-lyle has joined #openstack-swift | 23:47 | |
| clayg | torgomatic: don't breath | 23:50 |
| notmyname | patched and tests have restarted | 23:50 |
| notmyname | also lowered the post put timeout back to 0.5 | 23:51 |
| clayg | notmyname: so torgomatic and I need more info on the GET failures - like we need to expect the available and requested fragments from the GET that failed | 23:55 |
| clayg | notmyname: can you run a scenario that will generate the 503 on GET (pure write, not overwrite) and halt then try to see if those objects will continue to 503 if you GET with curl? | 23:55 |
| clayg | notmyname: i'm not even clear if there's two bugs or one - the one with the liberasure decode thing I think torgomatic and I have good idea how that might happen - is that the only one? | 23:56 |
| clayg | I mean I know there's the overwrite bug - but the remaining failures are from non-overwrite workloads correct? | 23:56 |
| notmyname | yeah, no object is deleted so fetching the object stat failed on PUT should still be available | 23:57 |
| *** DCWillia_ has joined #openstack-swift | 23:57 | |
| notmyname | correct. no overwrites right now | 23:57 |
| clayg | notmyname: oh I thought we were done with PUT bugs :P | 23:57 |
| notmyname | the increased proxy timeout is for that | 23:57 |
| clayg | notmyname: but yeah if you can run a scenario that will trigger a the 503 on GET (no overwrite) and then we can try to GET it again - that'd be *greataaaaat* | 23:57 |
| notmyname | so that's what i'm testing | 23:58 |
| clayg | fine | 23:58 |
| openstackgerrit | Samuel Merritt proposed openstack/swift: EC: don't 503 on marginally-successful PUT https://review.openstack.org/180795 | 23:58 |
| * clayg has so moved on past "someone should probably check if the passing unittest acctually fixes the issue" | 23:58 | |
| clayg | I have so much confidence in torgomatic it's not even funny | 23:58 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
