| *** Viswanath has joined #openstack-swift | 00:25 | |
| *** dmorita has joined #openstack-swift | 00:27 | |
| *** Viswanath has quit IRC | 00:29 | |
| *** ircolle has quit IRC | 00:31 | |
| *** mkollaro has quit IRC | 00:41 | |
| clayg | w00t wtg torgomatic - encryption spec looks solid! | 00:44 |
|---|---|---|
| *** addnull has joined #openstack-swift | 00:51 | |
| *** addnull has quit IRC | 00:51 | |
| *** addnull has joined #openstack-swift | 00:53 | |
| *** kyles_ne_ has quit IRC | 00:57 | |
| *** kyles_ne has joined #openstack-swift | 00:58 | |
| openstackgerrit | Clay Gerrard proposed a change to openstack/swift: Prefer X-Backend-Timestamp for X-Newest https://review.openstack.org/133869 | 01:05 |
| *** CaioBrentano has quit IRC | 01:05 | |
| *** rmcall has quit IRC | 01:14 | |
| *** sungju has joined #openstack-swift | 01:16 | |
| openstackgerrit | Clay Gerrard proposed a change to openstack/swift-specs: Updating containers on object fast-POST https://review.openstack.org/102592 | 01:20 |
| *** kopparam has joined #openstack-swift | 01:20 | |
| *** kyles_ne has quit IRC | 01:23 | |
| *** kyles_ne has joined #openstack-swift | 01:23 | |
| *** kopparam has quit IRC | 01:25 | |
| *** gyee has quit IRC | 01:27 | |
| openstackgerrit | Clay Gerrard proposed a change to openstack/python-swiftclient: Make preauth params work https://review.openstack.org/130711 | 01:29 |
| *** infotection has quit IRC | 01:44 | |
| *** shri has quit IRC | 01:45 | |
| *** Viswanath has joined #openstack-swift | 01:47 | |
| *** infotection has joined #openstack-swift | 01:48 | |
| *** Viswanath has quit IRC | 01:51 | |
| *** addnull has quit IRC | 02:05 | |
| *** nosnos has joined #openstack-swift | 02:06 | |
| *** addnull has joined #openstack-swift | 02:18 | |
| *** openstackgerrit has quit IRC | 02:34 | |
| *** sluo_laptop has joined #openstack-swift | 02:37 | |
| *** haomai___ has quit IRC | 02:38 | |
| *** tellesnobrega_ has joined #openstack-swift | 02:46 | |
| *** rmcall has joined #openstack-swift | 02:47 | |
| *** haomaiwang has joined #openstack-swift | 02:57 | |
| *** haomai___ has joined #openstack-swift | 02:59 | |
| *** haomaiwang has quit IRC | 02:59 | |
| *** kyles_ne has quit IRC | 03:00 | |
| *** kyles_ne_ has joined #openstack-swift | 03:01 | |
| *** haomai___ has quit IRC | 03:01 | |
| *** haomaiwang has joined #openstack-swift | 03:05 | |
| *** abhirc has quit IRC | 03:07 | |
| *** haomaiwang has quit IRC | 03:08 | |
| *** haomaiwa_ has joined #openstack-swift | 03:08 | |
| *** kopparam has joined #openstack-swift | 03:09 | |
| *** kopparam has quit IRC | 03:14 | |
| *** rmcall has quit IRC | 03:16 | |
| *** rmcall has joined #openstack-swift | 03:17 | |
| *** kopparam has joined #openstack-swift | 03:18 | |
| *** Viswanath has joined #openstack-swift | 03:22 | |
| *** kopparam has quit IRC | 03:23 | |
| *** Viswanath has quit IRC | 03:25 | |
| *** haomaiw__ has joined #openstack-swift | 03:29 | |
| *** haomaiwa_ has quit IRC | 03:33 | |
| *** kyles_ne_ has quit IRC | 03:35 | |
| *** addnull has quit IRC | 03:44 | |
| *** jyoti-ranjan has joined #openstack-swift | 03:56 | |
| *** rmcall_ has joined #openstack-swift | 04:05 | |
| *** rmcall has quit IRC | 04:05 | |
| *** rmcall_ is now known as rmcall | 04:05 | |
| *** kyles_ne has joined #openstack-swift | 04:55 | |
| *** kyles_ne_ has joined #openstack-swift | 04:57 | |
| *** miqui has quit IRC | 04:58 | |
| *** kopparam has joined #openstack-swift | 04:59 | |
| *** kyles_ne has quit IRC | 05:00 | |
| *** addnull has joined #openstack-swift | 05:01 | |
| *** ppai has joined #openstack-swift | 05:15 | |
| *** tellesnobrega_ has quit IRC | 05:23 | |
| *** kyles_ne has joined #openstack-swift | 05:27 | |
| *** kyles_ne_ has quit IRC | 05:30 | |
| *** nshaikh has joined #openstack-swift | 05:32 | |
| *** kyles_ne has quit IRC | 05:49 | |
| *** zaitcev has quit IRC | 06:04 | |
| *** rmcall has quit IRC | 06:11 | |
| *** SkyRocknRoll has joined #openstack-swift | 06:17 | |
| *** SkyRocknRoll has joined #openstack-swift | 06:17 | |
| *** k4n0 has joined #openstack-swift | 06:37 | |
| *** rmcall has joined #openstack-swift | 06:46 | |
| *** ppai has quit IRC | 06:50 | |
| *** sungju has quit IRC | 06:54 | |
| *** nellysmitt has joined #openstack-swift | 06:56 | |
| *** ppai has joined #openstack-swift | 07:05 | |
| *** addnull has quit IRC | 07:07 | |
| *** kopparam has quit IRC | 07:08 | |
| *** nellysmitt has quit IRC | 07:14 | |
| *** chmouel has quit IRC | 07:17 | |
| *** chmouel has joined #openstack-swift | 07:19 | |
| *** nellysmitt has joined #openstack-swift | 07:36 | |
| *** addnull has joined #openstack-swift | 07:38 | |
| *** ppai has quit IRC | 07:40 | |
| *** chmouel has quit IRC | 07:43 | |
| *** kopparam has joined #openstack-swift | 07:44 | |
| *** ppai has joined #openstack-swift | 07:54 | |
| *** chmouel has joined #openstack-swift | 07:55 | |
| *** jistr has joined #openstack-swift | 08:08 | |
| *** haomaiw__ has quit IRC | 08:13 | |
| *** nellysmitt has quit IRC | 08:13 | |
| *** rmcall has quit IRC | 08:14 | |
| *** haomaiwa_ has joined #openstack-swift | 08:14 | |
| *** joeljwright has joined #openstack-swift | 08:15 | |
| *** rmcall has joined #openstack-swift | 08:20 | |
| *** HONDA has joined #openstack-swift | 08:26 | |
| *** rmcall has quit IRC | 08:30 | |
| *** nellysmitt has joined #openstack-swift | 08:32 | |
| *** jordanP has joined #openstack-swift | 08:36 | |
| *** mmcardle has joined #openstack-swift | 08:46 | |
| *** kyles_ne has joined #openstack-swift | 08:50 | |
| *** kyles_ne has quit IRC | 08:54 | |
| *** mkollaro has joined #openstack-swift | 09:01 | |
| *** evanjfraser_ has joined #openstack-swift | 09:02 | |
| *** ppai has quit IRC | 09:02 | |
| *** evanjfraser has quit IRC | 09:02 | |
| *** jistr has quit IRC | 09:06 | |
| *** foexle has joined #openstack-swift | 09:08 | |
| *** ppai has joined #openstack-swift | 09:16 | |
| *** nellysmitt has quit IRC | 09:17 | |
| *** nellysmitt has joined #openstack-swift | 09:19 | |
| *** nellysmitt has quit IRC | 09:22 | |
| *** jistr has joined #openstack-swift | 09:26 | |
| *** haomaiw__ has joined #openstack-swift | 09:26 | |
| *** haomaiwa_ has quit IRC | 09:30 | |
| *** mkollaro has quit IRC | 09:56 | |
| *** jordanP has quit IRC | 09:58 | |
| *** mkollaro has joined #openstack-swift | 10:02 | |
| *** jordanP has joined #openstack-swift | 10:05 | |
| *** tellesnobrega_ has joined #openstack-swift | 10:11 | |
| *** kopparam has quit IRC | 10:11 | |
| *** aix has joined #openstack-swift | 10:18 | |
| *** addnull has quit IRC | 10:22 | |
| *** ppai has quit IRC | 10:23 | |
| *** tellesnobrega_ has quit IRC | 10:23 | |
| *** tellesnobrega_ has joined #openstack-swift | 10:29 | |
| *** ppai has joined #openstack-swift | 10:35 | |
| *** haomaiw__ has quit IRC | 10:36 | |
| *** nosnos has quit IRC | 10:39 | |
| *** addnull has joined #openstack-swift | 10:48 | |
| *** tellesnobrega_ has quit IRC | 10:50 | |
| *** kopparam has joined #openstack-swift | 11:00 | |
| *** nosnos has joined #openstack-swift | 11:02 | |
| *** Dafna has joined #openstack-swift | 11:03 | |
| *** HONDA has quit IRC | 11:07 | |
| *** aix has quit IRC | 11:14 | |
| *** tellesnobrega_ has joined #openstack-swift | 11:20 | |
| *** aix has joined #openstack-swift | 11:28 | |
| *** acoles_away is now known as acoles | 11:28 | |
| *** ppai has quit IRC | 11:39 | |
| *** ppai has joined #openstack-swift | 11:52 | |
| *** nellysmitt has joined #openstack-swift | 11:53 | |
| *** mkollaro has quit IRC | 11:53 | |
| *** dmsimard_away is now known as dmsimard | 11:59 | |
| *** kopparam has quit IRC | 11:59 | |
| *** tdasilva has joined #openstack-swift | 11:59 | |
| *** kopparam has joined #openstack-swift | 12:04 | |
| *** dmorita has quit IRC | 12:27 | |
| *** kopparam has quit IRC | 12:27 | |
| *** ppai has quit IRC | 12:32 | |
| *** addnull has quit IRC | 12:32 | |
| *** exploreshaifali has joined #openstack-swift | 12:37 | |
| *** nshaikh has left #openstack-swift | 12:47 | |
| *** rawat has quit IRC | 12:56 | |
| *** tellesnobrega_ has quit IRC | 12:56 | |
| *** tellesnobrega_ has joined #openstack-swift | 13:03 | |
| *** nosnos has quit IRC | 13:10 | |
| *** SkyRocknRoll has quit IRC | 13:23 | |
| *** nellysmitt has quit IRC | 13:41 | |
| *** k4n0 has quit IRC | 13:46 | |
| *** tellesnobrega_ has quit IRC | 13:47 | |
| *** haomaiwa_ has joined #openstack-swift | 14:16 | |
| *** jyoti-ranjan has quit IRC | 14:23 | |
| *** haomaiwa_ has quit IRC | 14:28 | |
| *** kopparam has joined #openstack-swift | 14:30 | |
| *** haomaiwa_ has joined #openstack-swift | 14:37 | |
| *** miqui has joined #openstack-swift | 14:39 | |
| *** exploreshaifali has quit IRC | 14:39 | |
| *** openstackgerrit has joined #openstack-swift | 14:41 | |
| *** rmcall has joined #openstack-swift | 14:41 | |
| *** geaaru has joined #openstack-swift | 14:43 | |
| openstackgerrit | Thomas Bechtold proposed openstack/swift: Raise max header size to accommodate large tokens https://review.openstack.org/133967 | 14:44 |
| *** rmcall has quit IRC | 14:46 | |
| *** kopparam has quit IRC | 14:50 | |
| *** rmcall has joined #openstack-swift | 14:50 | |
| *** rmcall has quit IRC | 14:55 | |
| *** zaitcev has joined #openstack-swift | 14:55 | |
| *** ChanServ sets mode: +v zaitcev | 14:55 | |
| *** haomaiwa_ has quit IRC | 15:17 | |
| cschwede | acoles: thanks for your comment :) | 15:27 |
| acoles | cschwede: np. btw, i'm all set to +2 this https://review.openstack.org/#/c/121893/ but had a (probably dumb) question if you have chance to answer | 15:29 |
| acoles | cschwede: yes/no is all i need :) no change needed to patch | 15:29 |
| cschwede | acoles: oh yes, sure, let me have a look | 15:29 |
| *** joe__ has joined #openstack-swift | 15:31 | |
| *** mahatic has joined #openstack-swift | 15:34 | |
| joe__ | I noticed something, that an object that has been quarantine ( which is moved to the quarantine folder) gets replicated back to the same wrong hash directory name | 15:34 |
| *** ircolle has joined #openstack-swift | 15:36 | |
| joe__ | that means it will never will get remove and will kip filling up the storage with quarantine items | 15:36 |
| cschwede | joe__: sounds like there is an old ringfile on some node? | 15:40 |
| joe__ | when I do swift-recon --md5 it says it matches in all nodes | 15:42 |
| swifterdarrell | joe__: cschwede: differing swift.conf prefix/suffix on some node(s)? | 15:43 |
| cschwede | joe__: is there any (old) node (maybe not even in the ring anymore) with running replicators? | 15:43 |
| swifterdarrell | joe__: is there a log line of some sort that gives a reason for the quarantine? | 15:44 |
| *** ircolle has quit IRC | 15:45 | |
| joe__ | I while back I had a node that had the wrong hash in the swift.conf that is how all this started, that was corrected, but I have been unable to get rid of the quarantine | 15:45 |
| joe__ | this is the error: ERROR Object /srv/node/sdc1/objects/99974/d0e/73a28f953e60275850f161be8d7d7d0e failed audit and was quarantined: Hash of name in metadata does not match directory name | 15:45 |
| *** ircolle has joined #openstack-swift | 15:46 | |
| joe__ | so the auditor moves it to quarantine, but then after a while it gets replicated back | 15:46 |
| swifterdarrell | joe__: ah, so I guess the object's name (in metadata) reflects the old/bad has from the incorrect swift.conf values... and that'll never get updated/fixed :-/ | 15:46 |
| joe__ | I just checked manually md5 in all rings in all nodes and they match | 15:46 |
| swifterdarrell | joe__: cschwede: sounds like a legit undesirable result for objects PUT while some nodes have mismatched swift.conf settings, and I don't know that there's any existing tooling to repair this specific resulting state... | 15:48 |
| swifterdarrell | joe__: what you'd need to do is re-write the object's metadata hash data to reflect its correct location so there's no more mismatch | 15:48 |
| joe__ | here is the info for the file http://paste.openstack.org/show/132411/ | 15:48 |
| swifterdarrell | joe__: how many objects are affected? | 15:48 |
| joe__ | there are a lot of files affected, as it was one node and that node had a lot of storage space | 15:50 |
| openstackgerrit | Donagh McCabe proposed openstack/swift-specs: Composite Autorization and Service Accounts https://review.openstack.org/105228 | 15:50 |
| joe__ | the meta data shows the correct location, if I go to the location in the metadata the file exists too | 15:50 |
| swifterdarrell | joe__: just to make sure i'm reading that correctly, 697c4c6e6442e9b5e9f27146d886fb5f is the new/correct hash and ebcd36d71eaded8cc10b8def205ee650 is the busted one that's living inside the object and/or xattrs? | 15:51 |
| cschwede | swifterdarrell: yes, sounds reasonable. hmm, we should fix this in swift | 15:51 |
| *** annegent_ has joined #openstack-swift | 15:51 | |
| joe__ | the other way 697c4c6e6442e9b5e9f27146d886fb5f is the incorrect hash and ebcd36d71eaded8cc10b8def205ee650 is the correct one | 15:52 |
| swifterdarrell | joe__: cschwede: anything we do will have to be carefully done to correct legit problems but not activate in other situations where changing the object may not be 100% correct to do | 15:52 |
| cschwede | swifterdarrell: yes, fully agree | 15:52 |
| swifterdarrell | joe__: cschwede: that's why I'm thinking of some kind of run-on-demand repair tool vs. automatic activation in normal Swift activities | 15:52 |
| swifterdarrell | joe__: cschwede: it's unfortunate, but running with mismatched swift.conf values is pretty pathological | 15:53 |
| swifterdarrell | joe__: k (re: i had the good/bad hashes backward) | 15:53 |
| swifterdarrell | joe__: wait, so 697c4c6e6442e9b5e9f27146d886fb5f is the incorrect hash, but that's where the replicator keeps trying to put the object? | 15:54 |
| swifterdarrell | joe__: since fixing the swift.conf are you 100% sure all object-replicators on affected servers have been restarted? | 15:56 |
| joe__ | swifterdarrell: yes it puts it back there, I don't understand how, as the replicator in theory should be looking into the right place only | 15:56 |
| swifterdarrell | joe__: iirc, the swift.conf stuff is imported and never updated, so changes require process-restart | 15:56 |
| joe__ | swifterdarrell: yes since then for sure they got restarted, I upgraded the version 2.1 for example | 15:57 |
| swifterdarrell | joe__: so maybe double-check that there are no lingering object-replicator processes (they may not have pidfile entries, etc.) | 15:57 |
| swifterdarrell | joe__: like, compare process listing and process start times vs. when you know the services should have restarted | 15:57 |
| swifterdarrell | joe__: I think my earlier analysis is wrong and it's not a matter of repairing the object/xattrs, but it really sounds like a rogue replicator is still running with the wrong swift.conf vals in memory | 15:58 |
| joe__ | swifterdarrell: I will check to be safe, but I am pretty sure they have been restarted at some point since then | 15:58 |
| swifterdarrell | joe__: knowing that new replicators are running ok is insufficient...you're after old replicators that, for whatever reason, escaped the "stop" part of "restart" | 15:59 |
| *** Nadeem has joined #openstack-swift | 16:00 | |
| swifterdarrell | joe__: if no stale/rogue object-replicators can be found, another thing to do would be to check the replication logs and for a single affected file, see if you can tell which object-replicator is sending that flie to the wrong place | 16:00 |
| swifterdarrell | joe__: then try to "zoom in" on that object-replicator and see if you can tell how it's making the wrong hash determination | 16:00 |
| joe__ | swifterdarrell: okay I will dig in | 16:01 |
| swifterdarrell | joe__: i'm in the middle of something else, so i can't check for sure, but maybe a lingering object-server could mis-handle the REPLICATE request if *it* still had old swift.conf values loaded in memory? | 16:01 |
| swifterdarrell | joe__: so you may need to look for any stale swift-related processes on all servers | 16:02 |
| joe__ | swifterdarrell: but would this proper clean up be part of swift in the future? | 16:02 |
| swifterdarrell | joe__: if the cause of the not-fixing is operational, like stale processes were running, I don't how Swift could fix that, really | 16:03 |
| *** david-lyle has joined #openstack-swift | 16:03 | |
| swifterdarrell | joe__: it sounds like everything is fine now except for whatever replicator(s) is/are putting the object back into the wrong hash path (causing it to get quarantined)... which smells like one or more stale daemons which have the bad swift.conf loaded | 16:04 |
| *** rmcall has joined #openstack-swift | 16:05 | |
| swifterdarrell | joe__: so i guess the answer to "would swift clean this up automatically in the future?" is "it depends" | 16:05 |
| cschwede | swifterdarrell: i’m just looking into the swift code, i might be wrong but i think the replicators fix this (as long as they have the correct swift.conf and ring). so no need for a tool? | 16:06 |
| swifterdarrell | cschwede: ya, I think we'll know more once we know for sure there are or are not rogue replicators or other swift backend daemons still running | 16:07 |
| swifterdarrell | cschwede: my current theory is that the issue is operational and not a bug in swift per se; it's tricky for a swift cluster to "automatically" verify they all have the same swift.conf prefix/suffix values, since those are sensitive data | 16:11 |
| joe__ | swifterdarrell: cschwede: okay I found the node that replicated back the bad one, so I am rebooting the whole node to be 100% sure, | 16:21 |
| *** ircolle has left #openstack-swift | 16:23 | |
| cschwede | acoles: to give you a short answer to your question: yes :) | 16:29 |
| acoles | cschwede: ok :) | 16:29 |
| cschwede | acoles: but i need to look into the bug report again, i’m wondering why there was a partition stored two times on one disc. this should not happen at all? (at least not if there are more discs than replicas) | 16:29 |
| cschwede | joe__: ah, great that you found the node! let us know if this solved your problem | 16:30 |
| acoles | cschwede: are you still happy with your patch though? | 16:31 |
| cschwede | acoles: yes, the patch is ok from my point of view. but there might be another issue (not related to my patch, only to the bugreport). looking | 16:32 |
| cschwede | :) | 16:33 |
| *** Viswanath has joined #openstack-swift | 16:36 | |
| openstackgerrit | Donagh McCabe proposed openstack/swift-specs: Adds Object Storage v1 API specification information https://review.openstack.org/129384 | 16:38 |
| *** Viswanath has quit IRC | 16:39 | |
| *** gyee has joined #openstack-swift | 16:45 | |
| *** abhirc has joined #openstack-swift | 16:47 | |
| *** dANO has joined #openstack-swift | 16:59 | |
| dANO | Hi, How i can add an ssl certificat in my openstack swift ? | 17:01 |
| *** abhirc has quit IRC | 17:07 | |
| *** exploreshaifali has joined #openstack-swift | 17:08 | |
| zaitcev | Add it to load balancer instead. | 17:17 |
| *** nellysmitt has joined #openstack-swift | 17:17 | |
| *** kyles_ne has joined #openstack-swift | 17:22 | |
| openstackgerrit | Donagh McCabe proposed openstack/swift-specs: Composite Autorization and Service Accounts https://review.openstack.org/105228 | 17:23 |
| *** nshaikh has joined #openstack-swift | 17:38 | |
| *** jistr has quit IRC | 17:38 | |
| *** acoles is now known as acoles_away | 17:52 | |
| openstackgerrit | Merged openstack/swift-specs: updated docs on specs workflow https://review.openstack.org/133174 | 17:53 |
| *** annegent_ has quit IRC | 17:58 | |
| *** bkopilov has quit IRC | 17:59 | |
| *** annegent_ has joined #openstack-swift | 18:00 | |
| *** rmcall has quit IRC | 18:00 | |
| *** dANO has quit IRC | 18:00 | |
| *** rmcall has joined #openstack-swift | 18:01 | |
| *** abhirc has joined #openstack-swift | 18:03 | |
| *** abhirc has quit IRC | 18:04 | |
| *** abhirc has joined #openstack-swift | 18:05 | |
| openstackgerrit | Merged openstack/swift: Fix bug in swift-ring-builder list_parts https://review.openstack.org/121893 | 18:06 |
| *** abhirc has quit IRC | 18:13 | |
| *** kyles_ne has quit IRC | 18:15 | |
| *** abhirc has joined #openstack-swift | 18:16 | |
| *** aix has quit IRC | 18:19 | |
| *** kyles_ne has joined #openstack-swift | 18:19 | |
| swifterdarrell | joe__: please let us know if the reboot solved the issue (hoping so) | 18:49 |
| *** shri has joined #openstack-swift | 18:53 | |
| *** elambert has joined #openstack-swift | 18:58 | |
| *** jordanP has quit IRC | 19:03 | |
| *** tsg_ has joined #openstack-swift | 19:06 | |
| *** tsg_ has quit IRC | 19:10 | |
| *** rmcall has quit IRC | 19:10 | |
| *** rmcall has joined #openstack-swift | 19:12 | |
| joe__ | swifterdarrell: I rebooted one node, and then saw replicating back from another node, then I restarted that other node, now I am waiting to see if it replicates the wrong one again from any other node | 19:12 |
| openstackgerrit | Clay Gerrard proposed openstack/swift-specs: Updating containers on object fast-POST https://review.openstack.org/102592 | 19:13 |
| swifterdarrell | joe__: k | 19:13 |
| *** kyles_ne has quit IRC | 19:13 | |
| *** cbaesema has joined #openstack-swift | 19:13 | |
| *** kyles_ne has joined #openstack-swift | 19:13 | |
| clayg | morning | 19:16 |
| *** mahatic has quit IRC | 19:26 | |
| *** kyles_ne has quit IRC | 19:26 | |
| openstackgerrit | Clay Gerrard proposed openstack/python-swiftclient: Make preauth params work https://review.openstack.org/130711 | 19:30 |
| *** mahatic has joined #openstack-swift | 19:39 | |
| clayg | how can we make changes to the dashboard? Would anyone be against making it filter wip patches? | 19:42 |
| openstackgerrit | Clay Gerrard proposed openstack/swift-specs: Composite Autorization and Service Accounts https://review.openstack.org/105228 | 19:50 |
| *** abhirc has quit IRC | 19:50 | |
| *** abhirc has joined #openstack-swift | 19:51 | |
| openstackgerrit | Clay Gerrard proposed openstack/swift-specs: Composite Authorization and Service Accounts https://review.openstack.org/105228 | 19:51 |
| *** foexle has quit IRC | 19:59 | |
| elambert | notmyname: did the meeting time get moved? | 20:06 |
| mahatic | notmyname, hello! around? | 20:06 |
| *** annegent_ has quit IRC | 20:07 | |
| *** annegent_ has joined #openstack-swift | 20:08 | |
| *** jordanP has joined #openstack-swift | 20:14 | |
| *** icksa has joined #openstack-swift | 20:14 | |
| openstackgerrit | Clay Gerrard proposed openstack/swift: Prefer X-Backend-Timestamp for X-Newest https://review.openstack.org/133869 | 20:18 |
| clayg | elambert: mahatic: notmyname is on vacation! | 20:19 |
| clayg | swifterdarrell is acting PTL - be careful | 20:19 |
| elambert | clayg: thanks .. so .. no meeting this week? | 20:19 |
| clayg | we could have a meeting i suppose - peluse torgomatic, is EC done yet? | 20:20 |
| torgomatic | nope! | 20:20 |
| clayg | good meeting | 20:20 |
| * torgomatic drops the mic | 20:20 | |
| swifterdarrell | good meeting everyone | 20:20 |
| elambert | good talk russ :-) | 20:20 |
| clayg | oh, national lampoon reference, nice | 20:21 |
| clayg | briancline: hey - can you elaborate on your comment on https://review.openstack.org/#/c/133869/ ? | 20:23 |
| *** infotection has quit IRC | 20:24 | |
| *** exploreshaifali has quit IRC | 20:24 | |
| *** abhirc has quit IRC | 20:24 | |
| clayg | briancline: it sounded like you said that if X-newest is trucking along and it hits a 404 it will stop digging into handoffs - but I think it will acctually go until it hits the end of the request_node_count config option (or runs out of nodes) | 20:24 |
| clayg | briancline: maybe you can help me find a test that will fail like you're observing - or i'm mis understanding your observation entirely? | 20:24 |
| * clayg hopes briancline was just wrong; but seems unlikely | 20:25 | |
| *** infotection has joined #openstack-swift | 20:27 | |
| peluse | heh | 20:27 |
| *** abhirc has joined #openstack-swift | 20:30 | |
| clayg | peluse: don't laugh; briancline is very teddy roosevelt about shit - when you can hear what he's saying it's normally very frightening | 20:31 |
| clayg | dfg: you gunna look at that error I saw with the node limiting patch - https://review.openstack.org/#/c/112424/ | 20:32 |
| clayg | also, why do all the needs approval patches on the dashboard for swiftclient already have my +2 on them | 20:33 |
| clayg | also, does anyone have any good ideas what to do about the sysmeta probetests for ssync? maybe I can fix up ssync to work with .meta files | 20:33 |
| peluse | clayg, I'm actually behind a few minutes, was responding to your sarcastic question about EC status :) | 20:33 |
| clayg | I mean I'm going to have to anyway; I was just thinking i would do it after fast-POST is a thing again | 20:33 |
| clayg | peluse: yeah I'm a real joker, I knew you were refering to that but I thought I'd take another shot at BC since he doesn't seem to be in here atm | 20:34 |
| peluse | nice... extra cup of coffee right after lunch maybe? | 20:35 |
| clayg | DID SOMEONE SAY COFFEE!?!!!!1 | 20:35 |
| peluse | you know... there are a lot of decaffeinated brands on the market today that are just as tasty as the real thing... | 20:37 |
| * peluse wonders if clayg can name *that* flick | 20:37 | |
| clayg | oh i had to google the national lampoon thing too | 20:38 |
| peluse | hah! Real Genius, Val Kilmer from a really long time ago... awesome flick though | 20:38 |
| elambert | https://www.youtube.com/watch?v=iYTdILs7-1Q .. for those who want the real thing | 20:39 |
| * clayg was hoping for Val Kilmer; reading http://en.wikiquote.org/wiki/Real_Genius | 20:39 | |
| elambert | http://www.hark.com/clips/nfxfkgdhbf-decaffeinated | 20:42 |
| elambert | clayg: ^^ not a video clip sadly | 20:43 |
| clayg | not bad | 20:43 |
| peluse | VK is one funny mofo | 20:45 |
| elambert | peluse: indeed .. top secret one of my favorite movies | 20:46 |
| clayg | there was some movie recently where it went to mars - it wasn't a funny movie as i recall - but he was funny in it | 20:46 |
| peluse | for sure! top secret, forgot about that one | 20:47 |
| *** shakamunyi_ has joined #openstack-swift | 20:49 | |
| *** kyles_ne has joined #openstack-swift | 20:57 | |
| *** gyee has quit IRC | 20:58 | |
| *** shakamunyi_ has quit IRC | 21:03 | |
| *** MasterPiece has joined #openstack-swift | 21:11 | |
| *** gyee has joined #openstack-swift | 21:14 | |
| *** shakamunyi_ has joined #openstack-swift | 21:17 | |
| *** abhirc has quit IRC | 21:18 | |
| *** abhirc has joined #openstack-swift | 21:18 | |
| *** shakamunyi_ has quit IRC | 21:27 | |
| *** Viswanath has joined #openstack-swift | 21:36 | |
| *** Viswanath has quit IRC | 21:40 | |
| *** Viswanath has joined #openstack-swift | 21:41 | |
| *** annegent_ has quit IRC | 21:44 | |
| *** Viswanath has quit IRC | 21:44 | |
| *** icksa has quit IRC | 21:47 | |
| *** abhirc has quit IRC | 21:50 | |
| *** abhirc has joined #openstack-swift | 21:51 | |
| *** nshaikh has quit IRC | 21:53 | |
| *** annegent_ has joined #openstack-swift | 21:55 | |
| *** annegent_ has quit IRC | 22:03 | |
| peluse | FYI, Swift native vs Ceph w/RADOSGW comparison. http://pushpeshsharma.blogspot.in/2014/11/openstack-swift-vs-ceph-rgw-read.html | 22:04 |
| peluse | sent to me by one of our benchmakring guys, I've haven't digested it yet... | 22:05 |
| clayg | peluse: can you just tell us who wins so we don't have to use our heads? always better to base your world-view off what someone told you about something they read on the internet than to do my own evaluation... | 22:05 |
| peluse | Swift, so I'm sure many won't bother reading now :) | 22:07 |
| clayg | peluse: am i missing the part where he benchmarked write thorughput? | 22:07 |
| peluse | oddly missing | 22:08 |
| clayg | yeah i guess in the 8 worker count 1GB reads we're not gunna do well because we'll only have 8 disks working | 22:12 |
| clayg | not sure what'd I change there, it makes sense, wonder what a mixed read workload would look like | 22:13 |
| tdasilva | clayg, peluse: pushpesh is a former red hatter, I believe now working at SanDisk, he was our QE engineer working on swiftonfile. He contacted us with these results earlier and you will see some of those emails on the comments below | 22:16 |
| peluse | yup, saw your name in there :) | 22:17 |
| clayg | yay redhat! | 22:17 |
| peluse | clayg, maybe I'm still jetlagged, why does the 8 concurrent clients w/1GB reads being low seem normal? | 22:17 |
| tdasilva | I think Yehuda and Mark from rgw is helping him out with some changes he could make to rgw | 22:17 |
| briancline | clayg: just saw your question from earlier here. clearly I was paying attention | 22:21 |
| *** nellysmitt has quit IRC | 22:22 | |
| briancline | clayg: I'd done a few tests around this in the past, but the last one I did was removing an entire ring partition from disk so that it'd be re-replicated, but during that time, requesting objects that live in that partition with x-newest would yield a 404 to the client and stop trying, even though the other two replica holders still had good copies | 22:24 |
| openstackgerrit | paul luse proposed openstack/swift: Add node/pair index patch back into feature/EC https://review.openstack.org/134065 | 22:25 |
| briancline | clayg: I haven't tested the same under 2.x, I just wasn't sure if anything around that had changed quasi-recently | 22:26 |
| *** miqui has quit IRC | 22:27 | |
| briancline | lol @ teddy roosevelt | 22:27 |
| peluse | torgomatic, See patch ^ needs your eyes for sure | 22:28 |
| torgomatic | peluse: cool, will look shortly | 22:28 |
| peluse | gracias | 22:28 |
| clayg | peluse: can i look at it too? (admiting my eyes are only half as good as torgomatic's) | 22:28 |
| peluse | clayg, you can look with just one eye :) | 22:29 |
| clayg | briancline: that's fucked up - I'll do a quick functional test but I think I know how it's gunna work out - don't expect anything changed there recently so more likely we have no idea what was going on | 22:29 |
| clayg | peluse: maybe if torgomatic used only one eye? | 22:29 |
| peluse | pair reviewing, rock n roll | 22:30 |
| torgomatic | arr matey | 22:30 |
| * peluse is outta here... for a few hrs anyway | 22:30 | |
| clayg | peluse: i tried that once and it turned out to not be dramatically better - there was some notion that you could avoid two people doing the same thing - but sometimes reproducibility of results is a reasonable part of validation | 22:31 |
| briancline | clayg: that's what I was thinking. if not, the small test cluster I saw it on is still alive, just outdated (1.12), so I could probably reproduce. if it fails again, I could move up the version chain to satisfy my curiosity as to when it changed (assuming nothing else was super crazy wrong with my test cluster) | 22:32 |
| *** jordanP has quit IRC | 22:32 | |
| clayg | briancline: yeah idk, need more info, the 404 thing seems solid | 22:35 |
| *** geaaru has quit IRC | 22:42 | |
| *** nomad411 has joined #openstack-swift | 22:44 | |
| *** tellesnobrega_ has joined #openstack-swift | 22:44 | |
| clayg | peluse: torgomatic: idk, my gut is that rather than change the signature we should just decorate the node dicts with an index than will just be there for when it's needed | 22:51 |
| clayg | it'd be easy to do in _get_part_nodes - I'm not really sure from our conversations if it's required for handoffs? | 22:51 |
| clayg | peluse: torgomatic: any idea why the FA index order doesn't match the order returned from swift-get-nodes? | 22:58 |
| clayg | peluse: torgomatic: neverind totally matches swift-get-nodes - i was calling get-nodes with a /v1/ in the path bah | 23:00 |
| *** jordanP has joined #openstack-swift | 23:02 | |
| *** david-lyle is now known as david-lyle_afk | 23:12 | |
| torgomatic | clayg: the node dicts are currently shared | 23:12 |
| briancline | sanity check: keystone is absolutely required in order to run its func tests against swift, right? no tempauth, that is | 23:12 |
| briancline | er, run tempest's func tests | 23:12 |
| *** abhirc has quit IRC | 23:13 | |
| torgomatic | the ring (class Ring) deserializes stuff to an array of (mutable) dicts, then hands out references to them for folks to go use | 23:13 |
| clayg | torgomatic: by who? you mean in the ring? yeah but we could return a copy | 23:13 |
| torgomatic | so yeah, we'd have to copy them or not mutate them | 23:13 |
| clayg | torgomatic: ok | 23:13 |
| torgomatic | I went with not mutating them, but ymmv | 23:13 |
| * torgomatic doesn't have a particularly strong opinion here | 23:13 | |
| clayg | torgomatic never seems to have strong opinions when it comes to the interface of the implementation or style of code that is functionally equivilant... | 23:15 |
| * clayg thinks torgomatic may be a cylon | 23:15 | |
| torgomatic | hehe | 23:15 |
| torgomatic | now if you'll pardon me, I'm going to go make some toast | 23:15 |
| *** jordanP has quit IRC | 23:16 | |
| *** tellesnobrega_ has quit IRC | 23:20 | |
| *** rmcall has quit IRC | 23:29 | |
| *** mmcardle1 has joined #openstack-swift | 23:31 | |
| *** mmcardle has quit IRC | 23:33 | |
| *** ahonda has joined #openstack-swift | 23:34 | |
| *** tellesnobrega_ has joined #openstack-swift | 23:35 | |
| *** kyles_ne has quit IRC | 23:40 | |
| *** dmsimard is now known as dmsimard_away | 23:44 | |
| *** tellesnobrega_ has quit IRC | 23:45 | |
| *** nomad411 has quit IRC | 23:45 | |
| *** exploreshaifali has joined #openstack-swift | 23:48 | |
| *** joeljwright has quit IRC | 23:50 | |
| *** abhirc has joined #openstack-swift | 23:51 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!