| *** jakeyip has quit IRC | 00:37 | |
| *** threestrands has joined #openstack-swift | 00:43 | |
| *** tovin07_ has joined #openstack-swift | 00:49 | |
| *** andybotting has quit IRC | 01:01 | |
| *** mtreinish has quit IRC | 01:58 | |
| *** mtreinish has joined #openstack-swift | 01:59 | |
| *** threestrands has quit IRC | 02:01 | |
| *** itlinux has joined #openstack-swift | 02:06 | |
| kota_ | good morning | 02:07 |
|---|---|---|
| *** threestrands has joined #openstack-swift | 02:16 | |
| *** threestrands has quit IRC | 02:16 | |
| *** threestrands has joined #openstack-swift | 02:16 | |
| mattoliverau | kota_: morning | 02:17 |
| kota_ | mattoliverau: o/ | 02:17 |
| kota_ | mattoliverau: in this morning I found the next Vancouver summit hotel information is available at https://www.openstack.org/summit/vancouver-2018/travel/ . | 02:18 |
| kota_ | even i just finished to get accepted about my hotel at Dublin PTG :/ | 02:18 |
| *** itlinux has quit IRC | 02:37 | |
| mattoliverau | they're keen. At least giving information as soon as possible is good. Though I think it'll be a long while before I hear from my employer letting me know if I can go or not. | 02:39 |
| kota_ | mattoliverau: yeah, I'll try it soon as possible | 02:40 |
| *** cshastri has joined #openstack-swift | 03:39 | |
| *** itlinux has joined #openstack-swift | 03:47 | |
| *** psachin has joined #openstack-swift | 04:06 | |
| *** threestrands has quit IRC | 04:07 | |
| *** itlinux has quit IRC | 04:12 | |
| *** threestrands has joined #openstack-swift | 04:19 | |
| *** threestrands has quit IRC | 04:19 | |
| *** threestrands has joined #openstack-swift | 04:19 | |
| *** threestrands has quit IRC | 04:36 | |
| *** threestrands has joined #openstack-swift | 04:37 | |
| *** ukaynar has joined #openstack-swift | 04:55 | |
| *** kei_yama has quit IRC | 05:03 | |
| *** kei_yama has joined #openstack-swift | 05:06 | |
| *** threestrands has quit IRC | 05:07 | |
| *** threestrands has joined #openstack-swift | 05:42 | |
| *** links has joined #openstack-swift | 05:46 | |
| *** threestrands has quit IRC | 05:47 | |
| *** threestrands has joined #openstack-swift | 05:47 | |
| *** links has quit IRC | 05:49 | |
| *** links has joined #openstack-swift | 05:51 | |
| *** gkadam has joined #openstack-swift | 05:52 | |
| *** links has quit IRC | 05:59 | |
| *** links has joined #openstack-swift | 06:00 | |
| *** threestrands_ has joined #openstack-swift | 06:04 | |
| *** threestrands_ has quit IRC | 06:04 | |
| *** threestrands_ has joined #openstack-swift | 06:04 | |
| *** threestrands has quit IRC | 06:06 | |
| *** links has quit IRC | 06:14 | |
| *** links has joined #openstack-swift | 06:16 | |
| *** m_kazuhiro has joined #openstack-swift | 06:35 | |
| m_kazuhiro | good morning | 06:37 |
| *** pcaruana has joined #openstack-swift | 07:04 | |
| openstackgerrit | Kazuhiro MIYAHARA proposed openstack/swift master: Fix inconsistency of account info in expirer's unit tests https://review.openstack.org/540735 | 07:07 |
| *** m_kazuhiro_ has joined #openstack-swift | 07:15 | |
| *** rcernin has quit IRC | 07:18 | |
| *** m_kazuhiro has quit IRC | 07:19 | |
| *** d0ugal has quit IRC | 07:42 | |
| *** armaan has joined #openstack-swift | 07:44 | |
| *** d0ugal has joined #openstack-swift | 08:07 | |
| *** ukaynar has quit IRC | 08:09 | |
| *** hoonetorg has quit IRC | 08:15 | |
| *** silor has joined #openstack-swift | 08:18 | |
| *** cbartz has joined #openstack-swift | 08:23 | |
| *** d0ugal has quit IRC | 08:26 | |
| *** hoonetorg has joined #openstack-swift | 08:28 | |
| *** tesseract has joined #openstack-swift | 08:28 | |
| *** kei_yama has quit IRC | 08:32 | |
| *** HCLTech-SSW has joined #openstack-swift | 08:38 | |
| *** d0ugal has joined #openstack-swift | 08:48 | |
| *** threestrands_ has quit IRC | 08:54 | |
| *** frti has joined #openstack-swift | 09:40 | |
| acoles | good morning | 09:52 |
| *** silor has quit IRC | 10:13 | |
| *** tovin07_ has quit IRC | 10:15 | |
| *** m_kazuhiro_ has quit IRC | 10:19 | |
| *** kallenp has joined #openstack-swift | 10:38 | |
| *** kallenp has left #openstack-swift | 10:39 | |
| *** Key has joined #openstack-swift | 10:47 | |
| Key | Having a s3 api issue | 10:50 |
| Key | Exact same problem as https://bugs.launchpad.net/swift/+bug/1695262 | 10:50 |
| openstack | Launchpad bug 1695262 in OpenStack Object Storage (swift) "swift3 middleware not working " [Undecided,New] | 10:50 |
| *** hseipp has joined #openstack-swift | 10:51 | |
| Key | none of the solutions works | 10:51 |
| *** HCLTech-SSW has quit IRC | 11:19 | |
| *** armaan has quit IRC | 11:32 | |
| *** armaan has joined #openstack-swift | 11:51 | |
| *** pcaruana has quit IRC | 12:02 | |
| *** hseipp has quit IRC | 12:42 | |
| *** hseipp has joined #openstack-swift | 12:51 | |
| *** psachin has quit IRC | 12:56 | |
| *** cshastri has quit IRC | 13:21 | |
| *** armaan has quit IRC | 13:48 | |
| *** armaan has joined #openstack-swift | 13:48 | |
| *** d0ugal has quit IRC | 13:55 | |
| *** hseipp has quit IRC | 14:09 | |
| *** hseipp has joined #openstack-swift | 14:14 | |
| *** d0ugal has joined #openstack-swift | 14:30 | |
| *** hseipp has quit IRC | 14:54 | |
| *** armaan has quit IRC | 14:58 | |
| *** armaan has joined #openstack-swift | 14:59 | |
| Key | Having a s3 api issue | 15:17 |
| Key | Exact same problem as https://bugs.launchpad.net/swift/+bug/1695262 | 15:17 |
| openstack | Launchpad bug 1695262 in OpenStack Object Storage (swift) "swift3 middleware not working " [Undecided,New] | 15:17 |
| Key | none of the solutions works | 15:18 |
| *** bob_cheesey has joined #openstack-swift | 15:18 | |
| *** Supun has joined #openstack-swift | 15:25 | |
| Key | <Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><RequestId>txf468d08f436a49c9a8149-005932a3fc</RequestId></Error> | 15:45 |
| Key | None of the ec3 credentials created from cli on pike works | 15:46 |
| Key | *ec2 | 15:47 |
| *** armaan has quit IRC | 15:49 | |
| *** Key has quit IRC | 16:01 | |
| *** Supun has quit IRC | 16:06 | |
| *** Supun has joined #openstack-swift | 16:06 | |
| openstackgerrit | James E. Blair proposed openstack/swift master: Zuul: Remove project name https://review.openstack.org/540915 | 16:15 |
| *** itlinux has joined #openstack-swift | 16:37 | |
| *** links has quit IRC | 16:37 | |
| notmyname | good morning. | 16:40 |
| notmyname | looks like the 2.17.0 has landed! | 16:40 |
| *** itlinux has quit IRC | 16:40 | |
| *** gkadam has quit IRC | 16:47 | |
| *** itlinux has joined #openstack-swift | 16:48 | |
| *** gyee has joined #openstack-swift | 16:55 | |
| *** frti has quit IRC | 17:04 | |
| *** cbartz has quit IRC | 17:05 | |
| *** d0ugal has quit IRC | 17:15 | |
| *** d0ugal has joined #openstack-swift | 17:23 | |
| *** Supun has quit IRC | 17:24 | |
| *** ukaynar has joined #openstack-swift | 17:38 | |
| *** d0ugal has quit IRC | 18:13 | |
| *** d0ugal has joined #openstack-swift | 18:27 | |
| *** itlinux has quit IRC | 18:31 | |
| *** armaan has joined #openstack-swift | 18:42 | |
| *** d0ugal has quit IRC | 18:59 | |
| *** itlinux has joined #openstack-swift | 19:07 | |
| openstackgerrit | John Dickinson proposed openstack/swift master: made changelog section breaks more consistent https://review.openstack.org/540978 | 19:18 |
| *** d0ugal has joined #openstack-swift | 19:23 | |
| *** tesseract has quit IRC | 19:36 | |
| *** frti has joined #openstack-swift | 20:01 | |
| *** armaan_ has joined #openstack-swift | 20:04 | |
| *** armaan has quit IRC | 20:04 | |
| *** ukaynar has quit IRC | 20:04 | |
| *** armaan_ has quit IRC | 20:04 | |
| *** armaan has joined #openstack-swift | 20:05 | |
| *** itlinux has quit IRC | 20:06 | |
| *** mrjk_ has joined #openstack-swift | 20:18 | |
| *** itlinux has joined #openstack-swift | 20:23 | |
| mrjk_ | Hi! | 20:31 |
| mrjk_ | We have an issue with our swift cluster, and I come here to get some advice. We have a policy with 2.5 replicats and few nodes associated to this policy (5). It happens replicator failed few days ago on node4 and we had a config issue on node5 (hash prefix and hash suffix was not set, so everything was copied, and directly pushed into quarentine. The quarentine folders has been unfortunately emptied since). We rebalanced 3 times before discovering the | 20:31 |
| mrjk_ | problem. Everything is fine for object with 3 replicats, but not for objects with two replicats which would have been moved from node4 to node5 during the three rebalances. So now, we basically hit some 404 for the missing objects. | 20:31 |
| mrjk_ | As the replicator was broken on node4, we strongly think all data are still on node4 (has they are deleted only once they have been replicated), but we are wondering what we could do to recover these unreplicated objects, and place them correctly on the good servers (not only node05 has there were 3 rebalance). Right know, we didn't restarted the replicator services, but we are thinking about two solutions: | 20:32 |
| mrjk_ | 1) Restart the replicator on node4, by hopping everything will get back on place (but we have strong doubts as we rebalanced 3 times) | 20:32 |
| mrjk_ | 2) Redeploy the setp by step the last three rebalanced rings on node04 (but we don't know if that will success either), and restart replicator service. | 20:32 |
| mrjk_ | Do you have any advice to give me? (I would be very very gratefull) | 20:32 |
| *** onovy has quit IRC | 20:33 | |
| mrjk_ | s/replicats/replicas/g | 20:38 |
| *** onovy has joined #openstack-swift | 20:39 | |
| *** spiette has joined #openstack-swift | 20:40 | |
| notmyname | mrjk_: replication will still continue to move stuff to the right place. at first guess, your first option sounds better | 20:59 |
| mrjk_ | @notmyname, Even if 3 ring rebalances has passed since the replicator on node4 hung ? | 21:01 |
| notmyname | the reason you want to replicate after every rebalance is to limit the data movement per replication cycle. the replicator can still move the data in the right place after multiple rebalanaces; it's just more likely that more data has to move | 21:02 |
| mrjk_ | notmyname, How does the replicator know where to find the source of the object to replicate if the object is orphaned? The handoff partitions? | 21:04 |
| notmyname | replication is push-based. so it finds whatever it has on its local drives and looks up where it's supposed to be in the cluster. if it doesn't match remotely, it will push local data to the other node. replication never pulls data from other places | 21:05 |
| mrjk_ | We did find references to the orphaned object deep in the list of handoffs (swift-get-nodes -a ) but not in primary handoffs. How does it know where to find where to find the object? | 21:06 |
| notmyname | what do you mean by "orphaned"? | 21:07 |
| mrjk_ | We had lost access to some objects (404) due to a chain of events on two nodes at the same time. This affected objects in the ring that only had 2 replicas and were on both of those nodes. | 21:08 |
| mrjk_ | storage policy replica count is 2.5 | 21:08 |
| notmyname | so you're saying orphaned to refer to objects that are stored somewhere in the cluster but currently are returning 404s? | 21:09 |
| mrjk_ | Correct. Node5 swift.conf was deployed w/ a bad hash and node4 replicator hung. We quickly noticed for node5, but node4 had a hung replicator for about 7 days and we rebalanced 3 times since. | 21:09 |
| notmyname | ok, those are two separate issues with different resolutions, but let's focus on the latter (since you said you figured out the former already) | 21:10 |
| mrjk_ | So objects that got onto Node5 while it's swift.conf was bad *and* whose second copy happened to be on Node4 are orphaned | 21:10 |
| mrjk_ | Happened to be on Node4 at that time I mean. | 21:11 |
| notmyname | in this case, you'd make sure every node has the most up-to-date ring and make sure replication is running everywhere. monitor the replication process and tune to make sure it goes as quickly as possible | 21:11 |
| notmyname | ah | 21:11 |
| notmyname | for the other problem, the one where you had the bad hash, that's more tricky | 21:12 |
| mrjk_ | We should also note that we deleted the quarantine directories on Node5 since we didn't think we would need them (we didn't know about the replicator problem on Node4 at the time) | 21:12 |
| notmyname | ok | 21:12 |
| notmyname | quarantine should have only been holding stuff that had detected bit rot. not misplaced data | 21:12 |
| mrjk_ | No no, it was directory hash not matching ERROR due to bad hash prefix in swift.conf | 21:13 |
| mrjk_ | A sample log from the time: an 29 23:08:21 msr-iaas-obj05 object-auditor: Quarantined object /srv/node/D22/objects/68286/1bb/855f2a9dee3d92c8b3b2abfd998c31bb/1481798668.55028.data: Hash of name in metadata does not match directory name | 21:15 |
| mrjk_ | I guess the question is since these orphaned objects have correct references deep in the list of handoffs, will the replicator know to use the full list of handoffs? | 21:17 |
| *** ukaynar has joined #openstack-swift | 21:21 | |
| *** rcernin has joined #openstack-swift | 21:26 | |
| *** itlinux has quit IRC | 21:45 | |
| *** itlinux has joined #openstack-swift | 21:54 | |
| openstackgerrit | Merged openstack/swift master: Fix inconsistency of account info in expirer's unit tests https://review.openstack.org/540735 | 22:25 |
| openstackgerrit | Merged openstack/swift master: Zuul: Remove project name https://review.openstack.org/540915 | 22:26 |
| *** armaan has quit IRC | 22:26 | |
| *** armaan has joined #openstack-swift | 22:27 | |
| *** early has quit IRC | 22:31 | |
| *** frti has quit IRC | 22:32 | |
| mattoliverau | morning | 22:33 |
| *** early has joined #openstack-swift | 22:33 | |
| *** ukaynar has quit IRC | 22:42 | |
| mrjk_ | Ok, after some analysis, we found that restarting the service solve our problem. Thank for your help notmyname :) | 22:45 |
| *** saint_ has joined #openstack-swift | 22:47 | |
| *** itlinux has quit IRC | 22:48 | |
| *** ukaynar has joined #openstack-swift | 22:53 | |
| openstackgerrit | Samuel Merritt proposed openstack/swift master: Add fallocate_reserve to account and container servers. https://review.openstack.org/541058 | 22:56 |
| *** saint_ has quit IRC | 23:14 | |
| *** ukaynar has quit IRC | 23:16 | |
| *** threestrands has joined #openstack-swift | 23:19 | |
| *** ukaynar has joined #openstack-swift | 23:24 | |
| *** ukaynar has quit IRC | 23:36 | |
| *** ukaynar has joined #openstack-swift | 23:37 | |
| *** ukaynar has quit IRC | 23:47 | |
| *** ukaynar has joined #openstack-swift | 23:49 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!