| zaitcev | Tim's on the roll. While I was tinkering with DLO, he's basically crushed object server completely. | 00:08 |
|---|---|---|
| DHE | py3 porting? | 00:12 |
| zaitcev | YEs. | 00:12 |
| mattoliverau | morning. | 00:23 |
| mattoliverau | nice. I guess it's back to py3 reviews then :) | 00:23 |
| openstackgerrit | John Dickinson proposed openstack/swift master: added changelog entries from stable releases https://review.openstack.org/639233 | 00:44 |
| notmyname | timburke: mattoliverau: if you wanna sanity check ^^. should just be an exact match of the stable releases that just were tagged | 00:46 |
| notmyname | (no I didn't go back and check all the older stable releases. I just did the ones we just tagged) | 00:47 |
| notmyname | interesting. in the past month, swift has used 0.7% of the total resources used (node time) in gate jobs | 00:56 |
| notmyname | http://paste.openstack.org/show/746083/ | 00:56 |
| *** mrjk_ has quit IRC | 01:33 | |
| *** mrjk_ has joined #openstack-swift | 01:34 | |
| * DHE FINALLY has a meeting this week about getting a production swift cluster.. :/ | 01:47 | |
| notmyname | yay! | 01:48 |
| DHE | ikr | 01:48 |
| tdasilva | notmyname: re: resources used in the gate jobs, that's really good. i've been concerned about the multi-node jobs, but i guess that ṕretty ok | 02:02 |
| notmyname | yeah, seems like not much to worry about there :-) | 02:08 |
| *** rexor has quit IRC | 02:36 | |
| *** psachin has joined #openstack-swift | 02:55 | |
| *** gyee has quit IRC | 03:58 | |
| mattoliverau | DHE: nice! | 04:39 |
| openstackgerrit | Merged openstack/swift master: added changelog entries from stable releases https://review.openstack.org/639233 | 06:01 |
| kota_ | DHE: congratulations!!! | 06:22 |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/swift feature/losf: Fix requirments to be correct. https://review.openstack.org/639260 | 06:35 |
| *** DHE has quit IRC | 07:06 | |
| *** DHE has joined #openstack-swift | 07:10 | |
| *** hseipp has joined #openstack-swift | 07:43 | |
| openstackgerrit | Pete Zaitcev proposed openstack/swift master: py3: port dlo https://review.openstack.org/639263 | 07:49 |
| *** rcernin has quit IRC | 07:56 | |
| *** ccamacho has joined #openstack-swift | 08:02 | |
| *** tkajinam has quit IRC | 08:16 | |
| *** pcaruana has joined #openstack-swift | 08:35 | |
| *** mikecmpbll has joined #openstack-swift | 08:57 | |
| *** ccamacho has quit IRC | 08:58 | |
| *** e0ne has joined #openstack-swift | 09:25 | |
| *** mcape has joined #openstack-swift | 09:26 | |
| *** FlorianFa has quit IRC | 09:30 | |
| *** FlorianFa has joined #openstack-swift | 09:35 | |
| *** ccamacho has joined #openstack-swift | 09:39 | |
| mcape | Hi guys! I need help with understanding of situation. I was asked to fix a poorly run cluster, and removal of one of six object nodes was needed. I mostly did only adding maxconn values in haproxy to limit the load going to cluster, which allowed it to breathe normally. After that, a decision to shrink it was made ( to create space for new cluster with proper defaults, and container sharding, which they need as they have 20mi | 09:39 |
| mcape | t by new admin, which removed one node from the ring, did rebalance but - oh yeah - forgot to sync rings to the all cluster nodes. | 09:40 |
| mcape | Now they started to get errors. | 09:40 |
| mcape | My understanding is that this situation is foreseen (as ring sync can't happen atomically) and if I distribute new ring (with one node less) to all nodes of cluster, the replicators will eventually fix the thing. Can you confirm that? | 09:40 |
| mcape | i just trying to understand what happened, the cluster was run for maybe 12 hours this way, one proxy sent files according to 5-node-ring, other two - according to 6-node-ring | 09:41 |
| mcape | from proxy with 5 nodes, the view is staying the same. but what will happen with files that were written by proxies with 6-node-ring? | 09:43 |
| alecuyer | mcape: hello, that's right, if all machines now have the correct copy of the ring, the replicator (or reconstructor if running EC) will eventually store data on the 5 nodes. The data that was stored according to the old ring topology will be moved | 09:47 |
| kota_ | mcape: are you meaning proxy has 5-node-ring but object-servers still keeps 6-node-ring? | 09:48 |
| kota_ | alecuyer: o/ | 09:48 |
| mcape | only one proxy had 6-node-ring for 12 hours | 09:49 |
| alecuyer | hello kota_ | 09:49 |
| mcape | nope, sorry | 09:49 |
| mcape | only one proxy had 5-node-ring, all other nodes had 6-node-ring | 09:49 |
| kota_ | oic, got it. | 09:50 |
| mcape | i do not grasp the concept of handoff nodes, so now some of the files landed on the handoffs from the 5-node-ring perspective? | 09:51 |
| kota_ | generally speaking, swift will distribute replicas to different disks/nodes/zones/regions so it will be not much big issue except rebalance overhead. | 09:51 |
| kota_ | the correct 5-nodes will attempts to locate replicas to correct location according to the ring. | 09:52 |
| mcape | but how it will find object that landed via 6-node-rings? | 09:53 |
| mcape | given it has different ring now | 09:53 |
| kota_ | and an object-server which has 5-nodes but no-assignment to itself do nothing anymore. | 09:53 |
| kota_ | strictly speaking, object-replicator. | 09:53 |
| kota_ | i think you removed a node from the ring, swift compute the new ring to avoid device change as possible. | 09:54 |
| mcape | so, until the moment cluster finishes replication, some objects will be unaccessible? | 09:54 |
| kota_ | so generally, 1 replica of 3 replica can be changed at once. | 09:54 |
| mcape | a-ha, got it | 09:54 |
| mcape | so we have 2 replicas which is intact | 09:55 |
| kota_ | mcape: true but proxy will find another one remaining in the available node. | 09:55 |
| mcape | got it, thank you very much | 09:55 |
| kota_ | :-) | 09:56 |
| *** mikecmpb_ has joined #openstack-swift | 10:35 | |
| *** mikecmpbll has quit IRC | 10:35 | |
| *** psachin has quit IRC | 11:40 | |
| *** ccamacho has quit IRC | 11:50 | |
| *** ccamacho has joined #openstack-swift | 12:12 | |
| *** mrjk__ has joined #openstack-swift | 12:26 | |
| *** mikecmpbll has joined #openstack-swift | 12:26 | |
| *** thurloat7 has joined #openstack-swift | 12:27 | |
| *** [diablo]9 has joined #openstack-swift | 12:28 | |
| *** [diablo] has quit IRC | 12:30 | |
| *** mikecmpb_ has quit IRC | 12:30 | |
| *** mrjk_ has quit IRC | 12:30 | |
| *** mvkr has quit IRC | 12:30 | |
| *** cschwede has quit IRC | 12:30 | |
| *** thurloat has quit IRC | 12:30 | |
| *** dosaboy has quit IRC | 12:30 | |
| *** openstackgerrit has quit IRC | 12:30 | |
| *** mvkr has joined #openstack-swift | 12:31 | |
| *** cschwede_ has joined #openstack-swift | 12:32 | |
| *** ChanServ sets mode: +v cschwede_ | 12:32 | |
| *** mikecmpbll has quit IRC | 12:40 | |
| *** henriqueof has joined #openstack-swift | 12:47 | |
| *** thurloat7 is now known as thurloat | 12:51 | |
| *** ccamacho has quit IRC | 12:52 | |
| *** ccamacho has joined #openstack-swift | 13:05 | |
| *** mikecmpbll has joined #openstack-swift | 13:09 | |
| *** dosaboy has joined #openstack-swift | 13:51 | |
| *** guimaluf has quit IRC | 14:09 | |
| *** guimaluf has joined #openstack-swift | 14:23 | |
| *** mrjk__ has quit IRC | 14:41 | |
| *** mrjk has joined #openstack-swift | 14:41 | |
| *** e0ne has quit IRC | 15:16 | |
| *** ccamacho has quit IRC | 15:51 | |
| *** e0ne has joined #openstack-swift | 15:53 | |
| *** ccamacho has joined #openstack-swift | 15:55 | |
| *** itlinux has joined #openstack-swift | 16:25 | |
| *** pcaruana has quit IRC | 16:32 | |
| *** ccamacho has quit IRC | 17:01 | |
| *** itlinux_ has joined #openstack-swift | 17:09 | |
| *** gyee has joined #openstack-swift | 17:13 | |
| *** itlinux has quit IRC | 17:13 | |
| *** hseipp has quit IRC | 17:22 | |
| *** e0ne has quit IRC | 17:24 | |
| *** itlinux_ has quit IRC | 18:13 | |
| *** itlinux has joined #openstack-swift | 18:16 | |
| *** guimaluf has quit IRC | 18:32 | |
| *** henriqueof has quit IRC | 19:04 | |
| *** itlinux_ has joined #openstack-swift | 19:19 | |
| *** itlinux has quit IRC | 19:21 | |
| *** mvkr has quit IRC | 19:21 | |
| *** mvkr has joined #openstack-swift | 19:21 | |
| *** e0ne has joined #openstack-swift | 19:26 | |
| *** henriqueof has joined #openstack-swift | 19:43 | |
| *** e0ne has quit IRC | 19:46 | |
| *** e0ne has joined #openstack-swift | 19:46 | |
| *** itlinux_ has quit IRC | 19:52 | |
| *** itlinux has joined #openstack-swift | 19:56 | |
| *** itlinux has quit IRC | 20:16 | |
| *** itlinux has joined #openstack-swift | 20:20 | |
| *** mikecmpbll has quit IRC | 20:26 | |
| *** mikecmpbll has joined #openstack-swift | 20:27 | |
| *** mathiasb has quit IRC | 20:28 | |
| *** e0ne has quit IRC | 20:31 | |
| *** takamatsu_ has quit IRC | 20:48 | |
| *** takamatsu_ has joined #openstack-swift | 20:50 | |
| *** itlinux has quit IRC | 20:59 | |
| *** spsurya has quit IRC | 21:03 | |
| *** itlinux has joined #openstack-swift | 21:07 | |
| *** e0ne has joined #openstack-swift | 21:28 | |
| *** openstackgerrit has joined #openstack-swift | 21:41 | |
| openstackgerrit | Tim Burke proposed openstack/pyeclib master: Update tox default env list; add py37 job https://review.openstack.org/639449 | 21:41 |
| *** itlinux has quit IRC | 21:53 | |
| *** henriqueof has quit IRC | 21:59 | |
| *** e0ne has quit IRC | 22:03 | |
| *** itlinux has joined #openstack-swift | 22:04 | |
| *** rcernin has joined #openstack-swift | 22:05 | |
| *** itlinux has quit IRC | 22:08 | |
| *** itlinux has joined #openstack-swift | 22:16 | |
| *** itlinux has quit IRC | 22:24 | |
| *** itlinux has joined #openstack-swift | 22:26 | |
| *** itlinux has quit IRC | 22:31 | |
| *** henriqueof has joined #openstack-swift | 22:49 | |
| *** itlinux has joined #openstack-swift | 22:50 | |
| *** itlinux has quit IRC | 22:57 | |
| mattoliverau | morning | 23:00 |
| *** tkajinam has joined #openstack-swift | 23:02 | |
| *** henriqueof has quit IRC | 23:28 | |
| *** henriqueof has joined #openstack-swift | 23:43 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!