| openstackgerrit | Alex Pecoraro proposed a change to openstack/swift: Allow hostname for nodes in Ring https://review.openstack.org/74542 | 00:01 |
|---|---|---|
| *** sungju has joined #openstack-swift | 00:17 | |
| *** openstackgerrit has quit IRC | 00:35 | |
| *** miurahr has joined #openstack-swift | 00:35 | |
| *** openstackgerrit has joined #openstack-swift | 00:35 | |
| *** miurahr has left #openstack-swift | 00:35 | |
| zaitcev | class Writr(object): | 00:41 |
| zaitcev | Oh Solly you magnificent | 00:41 |
| *** hurricanerix has quit IRC | 00:44 | |
| torgomatic | dfg: ping | 00:45 |
| *** sungju has quit IRC | 00:55 | |
| *** NM has joined #openstack-swift | 00:56 | |
| *** dmsimard has joined #openstack-swift | 00:56 | |
| *** dmsimard has quit IRC | 01:01 | |
| *** Dharmit has joined #openstack-swift | 01:01 | |
| *** kazdorsnab has joined #openstack-swift | 01:03 | |
| *** dmsimard has joined #openstack-swift | 01:04 | |
| *** dmsimard has quit IRC | 01:10 | |
| *** dmsimard1 has joined #openstack-swift | 01:10 | |
| *** mkollaro has quit IRC | 01:12 | |
| *** kazdorsnab has quit IRC | 01:16 | |
| *** csd has quit IRC | 01:38 | |
| *** csd has joined #openstack-swift | 01:43 | |
| *** shri has quit IRC | 01:43 | |
| *** csd has quit IRC | 01:49 | |
| *** nosnos has joined #openstack-swift | 02:00 | |
| *** taras___ has joined #openstack-swift | 02:05 | |
| *** taneez has joined #openstack-swift | 02:05 | |
| *** taneez is now known as tanee | 02:05 | |
| *** changbl has joined #openstack-swift | 02:05 | |
| *** taras__ has quit IRC | 02:06 | |
| *** tanee-away has quit IRC | 02:06 | |
| *** haomaiw__ has quit IRC | 02:17 | |
| *** haomaiwang has joined #openstack-swift | 02:17 | |
| *** j_king_ has joined #openstack-swift | 02:21 | |
| *** ondergetekende_ has joined #openstack-swift | 02:24 | |
| *** otherjon_ has joined #openstack-swift | 02:24 | |
| *** openstackgerrit has quit IRC | 02:35 | |
| *** mtreinish has quit IRC | 02:35 | |
| *** gholt has quit IRC | 02:35 | |
| *** j_king_ has quit IRC | 02:35 | |
| *** changbl has quit IRC | 02:35 | |
| *** Dharmit has quit IRC | 02:35 | |
| *** gyee has quit IRC | 02:35 | |
| *** PradeepChandani has quit IRC | 02:35 | |
| *** Slidey has quit IRC | 02:35 | |
| *** clarkb has quit IRC | 02:35 | |
| *** haomaiwang has quit IRC | 02:35 | |
| *** mandarine has quit IRC | 02:35 | |
| *** grapsus__ has quit IRC | 02:35 | |
| *** joearnold has quit IRC | 02:35 | |
| *** Kim-Chi-San has quit IRC | 02:35 | |
| *** NM has quit IRC | 02:35 | |
| *** occupant has quit IRC | 02:35 | |
| *** j_king has quit IRC | 02:35 | |
| *** mkerrin has quit IRC | 02:35 | |
| *** wer has quit IRC | 02:35 | |
| *** ondergetekende has quit IRC | 02:35 | |
| *** briancline has quit IRC | 02:35 | |
| *** gdrudy has quit IRC | 02:35 | |
| *** swifterdarrell has quit IRC | 02:35 | |
| *** sileht has quit IRC | 02:35 | |
| *** sudorandom has quit IRC | 02:35 | |
| *** otherjon has quit IRC | 02:35 | |
| *** CrackerJackMack has quit IRC | 02:35 | |
| *** otherjon_ has quit IRC | 02:35 | |
| *** ondergetekende_ has quit IRC | 02:35 | |
| *** tanee has quit IRC | 02:35 | |
| *** taras___ has quit IRC | 02:35 | |
| *** nosnos has quit IRC | 02:35 | |
| *** chandan_kumar has quit IRC | 02:35 | |
| *** zul has quit IRC | 02:35 | |
| *** peluse has quit IRC | 02:35 | |
| *** krtaylor has quit IRC | 02:35 | |
| *** creiht has quit IRC | 02:35 | |
| *** wkelly has quit IRC | 02:35 | |
| *** saschpe has quit IRC | 02:35 | |
| *** annegentle has quit IRC | 02:35 | |
| *** acoles has quit IRC | 02:35 | |
| *** sfineberg has quit IRC | 02:35 | |
| *** tristanC has quit IRC | 02:35 | |
| *** cropalato has quit IRC | 02:35 | |
| *** wayneeseguin has quit IRC | 02:35 | |
| *** acorwin has quit IRC | 02:35 | |
| *** jokke_ has quit IRC | 02:35 | |
| *** zackmdavis has quit IRC | 02:35 | |
| *** jeblair has quit IRC | 02:35 | |
| *** greghaynes has quit IRC | 02:35 | |
| *** pberis has quit IRC | 02:35 | |
| *** yuan has quit IRC | 02:35 | |
| *** early has quit IRC | 02:35 | |
| *** redbo has quit IRC | 02:35 | |
| *** notmyname has quit IRC | 02:35 | |
| *** MooingLemur has quit IRC | 02:35 | |
| *** rahmu has quit IRC | 02:35 | |
| *** dosaboy has quit IRC | 02:35 | |
| *** bsdkurt has quit IRC | 02:35 | |
| *** torgomatic has quit IRC | 02:35 | |
| *** minnear has quit IRC | 02:35 | |
| *** glange has quit IRC | 02:35 | |
| *** jogo has quit IRC | 02:35 | |
| *** hugokuo has quit IRC | 02:35 | |
| *** ctennis has quit IRC | 02:35 | |
| *** anderstj has quit IRC | 02:35 | |
| *** alpha_ori has quit IRC | 02:35 | |
| *** ryao has quit IRC | 02:35 | |
| *** anticw has quit IRC | 02:35 | |
| *** pconstantine has quit IRC | 02:35 | |
| *** dfg has quit IRC | 02:35 | |
| *** mordred has quit IRC | 02:35 | |
| *** kragniz has quit IRC | 02:35 | |
| *** Alex_Gaynor has quit IRC | 02:35 | |
| *** StevenK has quit IRC | 02:35 | |
| *** EmilienM has quit IRC | 02:35 | |
| *** pandemicsyn has quit IRC | 02:35 | |
| *** chmouel has quit IRC | 02:35 | |
| *** Diddi has quit IRC | 02:35 | |
| *** omame has quit IRC | 02:35 | |
| *** zigo has quit IRC | 02:35 | |
| *** mhu has quit IRC | 02:35 | |
| *** russellb has quit IRC | 02:35 | |
| *** Anticimex has quit IRC | 02:35 | |
| *** zanc has quit IRC | 02:35 | |
| *** fbo_away has quit IRC | 02:35 | |
| *** ChanServ has quit IRC | 02:35 | |
| *** madhuri has quit IRC | 02:35 | |
| *** mjseger has quit IRC | 02:35 | |
| *** ekarlso has quit IRC | 02:35 | |
| *** Dieterbe has quit IRC | 02:35 | |
| *** physcx has quit IRC | 02:35 | |
| *** JelleB has quit IRC | 02:35 | |
| *** mlanner has quit IRC | 02:35 | |
| *** amandap has quit IRC | 02:35 | |
| *** therve has quit IRC | 02:35 | |
| *** portante has quit IRC | 02:35 | |
| *** luisbg has quit IRC | 02:35 | |
| *** ahale has quit IRC | 02:35 | |
| *** rpedde has quit IRC | 02:35 | |
| *** clayg has quit IRC | 02:35 | |
| *** mtreinish has joined #openstack-swift | 02:40 | |
| *** dmorita has joined #openstack-swift | 03:02 | |
| *** mkerrin has joined #openstack-swift | 03:02 | |
| *** NM has joined #openstack-swift | 03:02 | |
| *** briancline has joined #openstack-swift | 03:02 | |
| *** CrackerJackMack has joined #openstack-swift | 03:02 | |
| *** sudorandom has joined #openstack-swift | 03:02 | |
| *** gdrudy has joined #openstack-swift | 03:02 | |
| *** wer has joined #openstack-swift | 03:02 | |
| *** dmorita has quit IRC | 03:06 | |
| *** wer has quit IRC | 03:06 | |
| *** sudorandom has quit IRC | 03:06 | |
| *** gdrudy has quit IRC | 03:06 | |
| *** CrackerJackMack has quit IRC | 03:06 | |
| *** briancline has quit IRC | 03:06 | |
| *** NM has quit IRC | 03:06 | |
| *** mkerrin has quit IRC | 03:06 | |
| *** dmorita has joined #openstack-swift | 03:11 | |
| *** mkerrin has joined #openstack-swift | 03:11 | |
| *** NM has joined #openstack-swift | 03:11 | |
| *** briancline has joined #openstack-swift | 03:11 | |
| *** CrackerJackMack has joined #openstack-swift | 03:11 | |
| *** sudorandom has joined #openstack-swift | 03:11 | |
| *** gdrudy has joined #openstack-swift | 03:11 | |
| *** wer has joined #openstack-swift | 03:11 | |
| *** ondergetekende has joined #openstack-swift | 03:22 | |
| *** creiht has joined #openstack-swift | 03:23 | |
| *** otherjon has joined #openstack-swift | 03:30 | |
| *** annegentle has joined #openstack-swift | 03:31 | |
| *** occup4nt has joined #openstack-swift | 03:31 | |
| *** taras___ has joined #openstack-swift | 03:31 | |
| *** zul has joined #openstack-swift | 03:31 | |
| *** peluse has joined #openstack-swift | 03:31 | |
| *** krtaylor has joined #openstack-swift | 03:31 | |
| *** wkelly has joined #openstack-swift | 03:31 | |
| *** tristanC has joined #openstack-swift | 03:31 | |
| *** wayneeseguin has joined #openstack-swift | 03:31 | |
| *** annegentle has quit IRC | 03:31 | |
| *** zul has quit IRC | 03:31 | |
| *** zul has joined #openstack-swift | 03:32 | |
| *** annegentle has joined #openstack-swift | 03:33 | |
| *** basha has joined #openstack-swift | 03:56 | |
| *** erlon has joined #openstack-swift | 04:04 | |
| *** basha has quit IRC | 04:12 | |
| *** NM has quit IRC | 04:19 | |
| *** fifieldt has joined #openstack-swift | 04:48 | |
| *** zaitcev has quit IRC | 04:55 | |
| *** nosnos has joined #openstack-swift | 05:00 | |
| *** Dharmit has joined #openstack-swift | 05:20 | |
| *** fifieldt has quit IRC | 05:30 | |
| *** erlon has quit IRC | 06:12 | |
| *** nshaikh has joined #openstack-swift | 06:17 | |
| *** Dharmit has quit IRC | 06:21 | |
| *** Dharmit has joined #openstack-swift | 06:21 | |
| *** sungju has joined #openstack-swift | 06:28 | |
| *** miurahr has joined #openstack-swift | 06:41 | |
| *** miurahr has quit IRC | 06:44 | |
| *** saju_m has joined #openstack-swift | 07:01 | |
| *** nosnos_ has joined #openstack-swift | 07:27 | |
| *** nosnos has quit IRC | 07:27 | |
| *** psharma has joined #openstack-swift | 07:54 | |
| *** Midnightmyth has joined #openstack-swift | 07:54 | |
| *** bvandenh has joined #openstack-swift | 07:54 | |
| *** madhuri has joined #openstack-swift | 07:54 | |
| *** ppai has joined #openstack-swift | 07:54 | |
| *** haomaiw__ has joined #openstack-swift | 07:54 | |
| *** sfineberg has joined #openstack-swift | 07:54 | |
| *** acoles has joined #openstack-swift | 07:54 | |
| *** acorwin_ has joined #openstack-swift | 07:54 | |
| *** tanee has joined #openstack-swift | 07:54 | |
| *** chandan_kumar has joined #openstack-swift | 07:54 | |
| *** saschpe has joined #openstack-swift | 07:54 | |
| *** sileht has joined #openstack-swift | 07:54 | |
| *** swifterdarrell has joined #openstack-swift | 07:54 | |
| *** j_king_ has joined #openstack-swift | 07:54 | |
| *** changbl has joined #openstack-swift | 07:54 | |
| *** mandarine has joined #openstack-swift | 07:54 | |
| *** jogo has joined #openstack-swift | 07:54 | |
| *** PradeepChandani has joined #openstack-swift | 07:54 | |
| *** Slidey has joined #openstack-swift | 07:54 | |
| *** pberis has joined #openstack-swift | 07:54 | |
| *** yuan has joined #openstack-swift | 07:54 | |
| *** early has joined #openstack-swift | 07:54 | |
| *** Alex_Gaynor has joined #openstack-swift | 07:54 | |
| *** grapsus__ has joined #openstack-swift | 07:54 | |
| *** gholt has joined #openstack-swift | 07:54 | |
| *** redbo has joined #openstack-swift | 07:54 | |
| *** ekarlso has joined #openstack-swift | 07:54 | |
| *** mjseger has joined #openstack-swift | 07:54 | |
| *** clarkb has joined #openstack-swift | 07:54 | |
| *** Dieterbe has joined #openstack-swift | 07:54 | |
| *** physcx has joined #openstack-swift | 07:54 | |
| *** JelleB has joined #openstack-swift | 07:54 | |
| *** greghaynes has joined #openstack-swift | 07:54 | |
| *** notmyname has joined #openstack-swift | 07:54 | |
| *** MooingLemur has joined #openstack-swift | 07:54 | |
| *** mlanner has joined #openstack-swift | 07:54 | |
| *** amandap has joined #openstack-swift | 07:54 | |
| *** therve has joined #openstack-swift | 07:54 | |
| *** jokke_ has joined #openstack-swift | 07:54 | |
| *** zackmdavis has joined #openstack-swift | 07:54 | |
| *** jeblair has joined #openstack-swift | 07:54 | |
| *** joearnold has joined #openstack-swift | 07:54 | |
| *** Kim-Chi-San has joined #openstack-swift | 07:54 | |
| *** rahmu has joined #openstack-swift | 07:54 | |
| *** dickson.freenode.net sets mode: +vvv gholt redbo notmyname | 07:54 | |
| *** StevenK has joined #openstack-swift | 07:54 | |
| *** dosaboy has joined #openstack-swift | 07:54 | |
| *** bsdkurt has joined #openstack-swift | 07:54 | |
| *** EmilienM has joined #openstack-swift | 07:54 | |
| *** pandemicsyn has joined #openstack-swift | 07:54 | |
| *** torgomatic has joined #openstack-swift | 07:54 | |
| *** minnear has joined #openstack-swift | 07:54 | |
| *** glange has joined #openstack-swift | 07:54 | |
| *** hugokuo has joined #openstack-swift | 07:54 | |
| *** ctennis has joined #openstack-swift | 07:54 | |
| *** anderstj has joined #openstack-swift | 07:54 | |
| *** alpha_ori has joined #openstack-swift | 07:54 | |
| *** portante has joined #openstack-swift | 07:54 | |
| *** ryao has joined #openstack-swift | 07:54 | |
| *** anticw has joined #openstack-swift | 07:54 | |
| *** pconstantine has joined #openstack-swift | 07:54 | |
| *** dfg has joined #openstack-swift | 07:54 | |
| *** dickson.freenode.net sets mode: +vvvv torgomatic glange portante dfg | 07:54 | |
| *** mordred has joined #openstack-swift | 07:54 | |
| *** kragniz has joined #openstack-swift | 07:54 | |
| *** chmouel has joined #openstack-swift | 07:54 | |
| *** Diddi has joined #openstack-swift | 07:54 | |
| *** omame has joined #openstack-swift | 07:54 | |
| *** zigo has joined #openstack-swift | 07:54 | |
| *** mhu has joined #openstack-swift | 07:54 | |
| *** russellb has joined #openstack-swift | 07:54 | |
| *** Anticimex has joined #openstack-swift | 07:54 | |
| *** luisbg has joined #openstack-swift | 07:54 | |
| *** ahale has joined #openstack-swift | 07:54 | |
| *** zanc has joined #openstack-swift | 07:54 | |
| *** rpedde has joined #openstack-swift | 07:54 | |
| *** fbo_away has joined #openstack-swift | 07:54 | |
| *** ChanServ has joined #openstack-swift | 07:54 | |
| *** clayg has joined #openstack-swift | 07:54 | |
| *** dickson.freenode.net sets mode: +ov ChanServ clayg | 07:54 | |
| *** nshaikh has quit IRC | 07:56 | |
| *** nshaikh has joined #openstack-swift | 07:56 | |
| *** Dharmit has quit IRC | 07:56 | |
| *** Dharmit has joined #openstack-swift | 07:56 | |
| *** sungju is now known as Guest59280 | 07:56 | |
| *** Alex_Gaynor has quit IRC | 07:56 | |
| *** Alex_Gaynor has joined #openstack-swift | 07:57 | |
| *** Guest59280 has quit IRC | 08:09 | |
| *** joeljwright has joined #openstack-swift | 08:11 | |
| *** kun_huang has joined #openstack-swift | 08:24 | |
| *** nacim has joined #openstack-swift | 08:45 | |
| *** nosnos has joined #openstack-swift | 08:45 | |
| *** Dharmit has quit IRC | 08:46 | |
| *** nosnos_ has quit IRC | 08:46 | |
| *** Dharmit has joined #openstack-swift | 08:49 | |
| *** nacim has quit IRC | 08:49 | |
| *** nacim has joined #openstack-swift | 08:50 | |
| *** Dharmit has quit IRC | 08:54 | |
| *** Dharmit has joined #openstack-swift | 08:54 | |
| *** Midnightmyth has quit IRC | 08:56 | |
| *** dmorita has quit IRC | 09:01 | |
| *** basha has joined #openstack-swift | 09:11 | |
| *** basha has quit IRC | 09:13 | |
| *** basha has joined #openstack-swift | 09:14 | |
| *** basha_ has joined #openstack-swift | 09:18 | |
| *** basha has quit IRC | 09:18 | |
| *** basha_ is now known as basha | 09:18 | |
| *** Dharmit has quit IRC | 09:21 | |
| *** foexle has joined #openstack-swift | 09:24 | |
| *** jamieh has joined #openstack-swift | 09:24 | |
| *** haomaiw__ has quit IRC | 09:28 | |
| *** haomaiwang has joined #openstack-swift | 09:29 | |
| *** mkerrin has quit IRC | 09:30 | |
| *** gdrudy has quit IRC | 09:30 | |
| *** tanee is now known as tanee-away | 09:39 | |
| *** fbo_away is now known as fbo | 09:44 | |
| *** gdrudy has joined #openstack-swift | 09:45 | |
| *** mkerrin has joined #openstack-swift | 09:55 | |
| *** Kim-Chi-San has quit IRC | 10:06 | |
| *** tanee-away is now known as tanee | 10:06 | |
| *** otoolee has quit IRC | 10:19 | |
| *** gdrudy has quit IRC | 10:19 | |
| *** saju_m has quit IRC | 10:19 | |
| *** tanee is now known as tanee-away | 10:20 | |
| *** mkollaro has joined #openstack-swift | 10:20 | |
| *** kun_huang has quit IRC | 10:24 | |
| *** kun_huang has joined #openstack-swift | 10:26 | |
| *** PradeepChandani has left #openstack-swift | 10:41 | |
| *** tanee-away is now known as tanee | 10:56 | |
| *** ccorrigan has quit IRC | 11:14 | |
| *** mkollaro has quit IRC | 11:16 | |
| *** Trixboxer has joined #openstack-swift | 11:29 | |
| *** saju_m has joined #openstack-swift | 11:30 | |
| *** psharma has quit IRC | 11:41 | |
| *** _bluev has joined #openstack-swift | 11:49 | |
| *** nosnos has quit IRC | 11:51 | |
| _bluev | Hi. What do people use for concurrency for the object-replicator ? We have 6 back-end servers. 36 SATA disks in each and find that with worked 1, replicator concurrency 8 we have 200+ hours for replicaton..... | 11:52 |
| *** PradeepChandani has joined #openstack-swift | 11:53 | |
| *** Midnightmyth has joined #openstack-swift | 11:54 | |
| *** basha has quit IRC | 12:02 | |
| _bluev | I looks to me like the 'workers' config value for the object-* daemons has the affect of running multiple *wsgi* processes - is that correct ? If that is true increasing the workers count in object-server.conf has no impact on swift-obect-replicator as that's not a wgsi user | 12:16 |
| *** PradeepChandani has left #openstack-swift | 12:16 | |
| *** bvandenh has quit IRC | 12:19 | |
| *** ppai has quit IRC | 12:21 | |
| *** mkollaro has joined #openstack-swift | 12:25 | |
| *** kun_huang has quit IRC | 12:26 | |
| *** NM has joined #openstack-swift | 12:28 | |
| *** erlon has joined #openstack-swift | 12:30 | |
| *** PradeepChandani has joined #openstack-swift | 12:30 | |
| *** PradeepChandani has left #openstack-swift | 12:30 | |
| *** PradeepChandani has joined #openstack-swift | 12:31 | |
| *** PradeepChandani has left #openstack-swift | 12:31 | |
| *** kun_huang has joined #openstack-swift | 12:37 | |
| *** nshaikh has quit IRC | 12:42 | |
| *** miurahr has joined #openstack-swift | 12:47 | |
| *** saju_m has quit IRC | 12:49 | |
| *** miurahr has quit IRC | 12:54 | |
| *** kun_huang has quit IRC | 12:55 | |
| *** miurahr has joined #openstack-swift | 12:58 | |
| *** foexle has quit IRC | 13:02 | |
| *** foexle has joined #openstack-swift | 13:02 | |
| *** miurahr has quit IRC | 13:02 | |
| *** miurahr has joined #openstack-swift | 13:08 | |
| *** miurahr has quit IRC | 13:16 | |
| *** Kim-Chi-San has joined #openstack-swift | 13:31 | |
| *** bowdengl has joined #openstack-swift | 14:05 | |
| *** kun_huang has joined #openstack-swift | 14:12 | |
| *** hurricanerix has joined #openstack-swift | 14:21 | |
| *** j_king_ is now known as j_king | 14:33 | |
| *** Diddi has quit IRC | 14:45 | |
| *** dmsimard has joined #openstack-swift | 14:46 | |
| *** Diddi has joined #openstack-swift | 14:52 | |
| creiht | _bluev: correct | 15:03 |
| creiht | http://docs.openstack.org/developer/swift/deployment_guide.html#object-server-configuration | 15:03 |
| creiht | you can sett concurrency under the [object-replicator] section of the conf | 15:04 |
| dfg | torgomatic: did gholt answe your question? of course i guess there's other problems with that patch. clayg thanks you looking at it | 15:07 |
| _bluev | creiht: we have very slow replication. e.g 3579/3734658 (0.10%) partitions replicated in 1800.01s (1.99/sec, 521h remaining) | 15:14 |
| _bluev | creiht: we set concurrency for the replicator to 8, 12 and 24 but we don't see any difference. We think the REPLICATE object-serve request could be slowing everything down. | 15:15 |
| creiht | _bluev: has this happened suddenly? | 15:16 |
| _bluev | creiht: I think it's got worse as we've got busier and fuller to be honest. | 15:18 |
| creiht | _bluev: well there can be a lot of reasons | 15:18 |
| creiht | first things to look at are double check your max connections options in rsyncd.conf | 15:19 |
| *** jeblair is now known as jegerritbot | 15:19 | |
| *** jegerritbot is now known as jeblair | 15:19 | |
| creiht | that limits how many incomming replication rsyncs are allowed | 15:19 |
| creiht | you also don't want to set the concurrencies too high, as that will just case everything to overload | 15:20 |
| creiht | look at the logs for obvious types of errors | 15:20 |
| creiht | look for things that prevent replication passes from completing | 15:21 |
| creiht | as that can easily baloon how long replication passes can take | 15:21 |
| _bluev | creiht: we have rsyncd max connection set to 30. We increased from 10 about 4 weeks ago | 15:21 |
| gholt | dfg: clayg: torgomatic: Updated https://review.openstack.org/#/c/78766/ to show tempurl needs to be before dlo and slo. A tempurl test would be good. dfg and I are working on that. | 15:21 |
| creiht | _bluev: look at the logs to make sure the rsyncs are completing successfully | 15:21 |
| creiht | look at your network throughput to see if you are using all of that | 15:21 |
| *** openstackgerrit has joined #openstack-swift | 15:22 | |
| creiht | there is likely something going on somewhere that is preventing replication passes from completing, which causes stuff to back up | 15:22 |
| *** tanee is now known as tanee-away | 15:23 | |
| creiht | _bluev: also make sure your system overall is operating fine | 15:24 |
| creiht | if swift is working around issues, then there will be a lot of extra handoffs going on | 15:25 |
| creiht | which can also make replication take a lot longer because your disks start getting extra partitions | 15:25 |
| _bluev | creiht: thanks for that. rsync logs look good to move from the client and server perspective . The main log is full of messages like "object-replicator Successful rsync of /srv/node/sdo1/objects/7630784/219 at 10.32.37.85::object/sdg1/objects/7630784 (0.827)" | 15:25 |
| creiht | _bluev: a quick thing you can do is calculate how many partitions you should have on disk on average, then see how many you have on some disks for real | 15:28 |
| creiht | most of these are shots in the dark, since i don't have any insight into your cluster | 15:30 |
| creiht | _bluev: sorry that I can't help more right this instance, but hopefully that will help lead you to the source of the issue | 15:31 |
| luisbg | morning | 15:31 |
| _bluev | Our statd data shows the REPLICATE mean is often over a second - we have WORKERS on object-server.conf set to 2 , we're thinking we may need to increase that as REPLICATE takes an age sometimes. | 15:34 |
| *** judd7 has joined #openstack-swift | 15:36 | |
| gholt | _bluev: creiht: With an example partition number of 7630784 and only 216 disks it sounds like you have way too high a partition power. | 15:37 |
| gholt | Are you running at 2**23 or something? | 15:38 |
| *** bowdengl has quit IRC | 15:38 | |
| gholt | We've got tens of thousands of disks in clusters and only run at 2**20. | 15:38 |
| _bluev | gholt: Yes. The original cluster was 6*36 but we added an extra 9 systems each with 32 disks. The replication of the data is whats killing us | 15:39 |
| gholt | Well yeah, if you have it uselessly walking a tons of directory structures that aren't really needed, that'd slow things down. | 15:40 |
| gholt | You might look at http://rackerlabs.github.io/swift-ppc/ | 15:41 |
| *** tanee-away is now known as tanee | 15:42 | |
| *** piyush has joined #openstack-swift | 15:44 | |
| *** kun_huang has quit IRC | 15:45 | |
| _bluev | gholt: I'm in production with a loads of customers. The PP was chosen based on the storage growth our management predicted. I have added capacity as I was becoming full. I started by adding in small weigh increments as per best practice. Replication is very slow -eg 200h, 400h etc. Do I have any options or am I just totally screwed. | 15:45 |
| _bluev | gholt: I read all your consistent hashing ring blog posts and the impression given was the a large PP means you burn lots of memory which was the only guidance I could find. We have 48G per object server. | 15:46 |
| gholt | Hmm. Well, I assume you're running at part power of 23? 2**23/(6*36+9*32) = ~16644 partitions per disk? | 15:47 |
| *** Trixboxer has quit IRC | 15:48 | |
| _bluev | gholt: Yes PP set to 23 | 15:49 |
| gholt | Er, with 3 replicas that'd be ~49932 parts per disk. In one example you pasted it said 3579/3734658 which is still pretty high for 36 disk system but I guess maybe that's because it hasn't re-replicated yet. | 15:51 |
| *** judd7_ has joined #openstack-swift | 15:52 | |
| gholt | Unfortunately we still don't have a way to re-part-power a running cluster. :/ We want that badly, but a lot of other stuff has to fall into place first (ssync, index.db, etc.) We'll get there, but that doesn't help you now. | 15:52 |
| *** judd7 has quit IRC | 15:53 | |
| _bluev | ok, thanks gholt. I think I need to focus on decreasing the amount time the REPLICATE verb takes | 15:54 |
| gholt | I'm thinking, but honestly the only thing I can think to do is to make a new cluster and migrate data from the old to the new. Not an easy endeavor I can imagine. | 15:54 |
| gholt | Upping the replicator concurrency might help to a point, but at a certain point you'd just be thrashing the disk and slowing things down. So you might have to experiment with that setting a bit to find its happiest point. | 15:56 |
| _bluev | gholt: Ok, thanks. | 15:57 |
| *** tanee is now known as tanee-away | 16:06 | |
| *** russellb is now known as rustlebee | 16:06 | |
| *** csd has joined #openstack-swift | 16:07 | |
| *** otherjon has quit IRC | 16:08 | |
| *** otherjon_ has joined #openstack-swift | 16:09 | |
| *** otherjon_ is now known as otherjon | 16:09 | |
| *** ChanServ sets mode: +v creiht | 16:14 | |
| _bluev | gholt: Regarding the low number of partitions-per-second as reported here: object-replicator 82831/3734658 (2.22%) partitions replicated in 7200.78s (11.50/sec, 88h remaining) . If you increase replicator concurrency , should that not increase the partitions-per-second, regardless of what you PP is set to ? | 16:14 |
| gholt | _bluev: Theoretically yes, but disk trashing will come into play. | 16:17 |
| gholt | I assume from your earlier comments when you add capacity you're doing it with as many disks as you can at pretty low weight changes on each which is good. You want to get as much breadth as you can as fast as you can. | 16:18 |
| _bluev | Thanks gholt - we'll check collect data and see if the disks are the bottlenecks | 16:20 |
| gholt | I'd probably keep the incoming rsync setting the same as the outgoing replicator concurrency, so you always have as many receivers as senders collectively across the cluster. As you get to higher concurrency and because everything is randomized you'll run the chance of stampeding a single disk from time to time. But you've got run that risk to let the cluster get rebalanced again. | 16:20 |
| gholt | You can also consider shutting off all auditors for a while to give you just that bit of extra i/o. | 16:21 |
| _bluev | gholt: makes sense. | 16:22 |
| _bluev | if only the users would stop PUT'ing new data :-) | 16:22 |
| gholt | Heheh, yeah, you're kind of at the point where "Hey everybody, quit using the cluster just for a while, please?" | 16:22 |
| *** basha has joined #openstack-swift | 16:25 | |
| gholt | I'm sorry for this, it's a fine mess you're in. :( I wish I could code faster so re-partitioning was a real thing already. | 16:25 |
| *** tanee-away is now known as tanee | 16:27 | |
| *** tanee is now known as tanee-away | 16:27 | |
| _bluev | gholt: I think if we tweak the settings down to give us a cycle in 100hours, that hopefully will be sustainable until we grow the cluster again to ease the per-disk partition count | 16:28 |
| notmyname | hello world | 16:30 |
| notmyname | atlanta design summit session proposals are now open at http://summit.openstack.org | 16:31 |
| notmyname | details on the process are at http://lists.openstack.org/pipermail/openstack-dev/2014-March/029319.html | 16:31 |
| *** mkollaro1 has joined #openstack-swift | 16:32 | |
| *** mkollaro has quit IRC | 16:32 | |
| peluse | notmyname: these are for the technical tracks like the 'Swift day" in HK where you ran the room the whole day right? | 16:32 |
| notmyname | correct | 16:33 |
| notmyname | I don't know how many slots swift will get, but in the past we've normally had about 8-9 slots (one day) | 16:33 |
| peluse | notmyname: Cool, do we need to submit something for EC or just ask you to put it on the agenda and we'll figure out who talks about when later? | 16:33 |
| notmyname | peluse: ya, if you are going to be in atlanta, then I think you should submit at least a placeholder for it :-) | 16:34 |
| peluse | notmyname: will do then. and I assume we're good on policies since that's an icehouse thing right? | 16:34 |
| creiht | peluse: I wouldn't mind a session going over the policies work | 16:35 |
| notmyname | well, that's the plan. progress is slow, though ;-/ | 16:35 |
| *** mkollaro2 has joined #openstack-swift | 16:35 | |
| notmyname | ya, it's still good to talk about it, especially as it relates to the future work | 16:35 |
| peluse | notmyname: maybe we can get torgotmatic and extra coffee subscription this month :) | 16:35 |
| *** tanee-away is now known as tanee | 16:36 | |
| *** mkollaro1 has quit IRC | 16:36 | |
| *** tanee is now known as tanee-away | 16:37 | |
| notmyname | wow. I just looked at gerrit. jenkins is angry this morning | 16:38 |
| *** miurahr has joined #openstack-swift | 16:39 | |
| notmyname | yup. http://not.mn/all_gate_status.html | 16:40 |
| *** mkollaro has joined #openstack-swift | 16:41 | |
| *** gyee has joined #openstack-swift | 16:41 | |
| *** mkollaro2 has quit IRC | 16:42 | |
| *** tanee-away is now known as tanee | 16:42 | |
| *** tanee is now known as tanee-away | 16:43 | |
| *** hurricanerix has quit IRC | 16:47 | |
| peluse | notmyname: OK, EC and policies are both up there as sugegsted topics | 16:48 |
| notmyname | thanks :-) | 16:48 |
| peluse | np | 16:48 |
| peluse | FYI I'm just firing up a cluster now that I'm going to install w/o policies and then get some IO going while I roll it over to the EC branch to try and ID upgrade issues that we may not have thought of... | 16:48 |
| *** basha has quit IRC | 16:50 | |
| *** joeljwright has left #openstack-swift | 16:50 | |
| *** chandan_kumar has quit IRC | 16:51 | |
| notmyname | good | 16:53 |
| *** tanee-away is now known as tanee | 16:56 | |
| *** tanee is now known as tanee-away | 16:56 | |
| openstackgerrit | Samuel Merritt proposed a change to openstack/swift: Functional tests for tempurl https://review.openstack.org/79008 | 16:58 |
| *** tanee-away is now known as tanee | 17:00 | |
| *** tanee is now known as tanee-away | 17:00 | |
| gholt | torgomatic: Sweet https://review.openstack.org/#/c/79008/ | 17:01 |
| torgomatic | gholt: :) | 17:01 |
| torgomatic | I had those laying around mostly passing a while ago, but the UTF8 stuff was busted | 17:01 |
| torgomatic | and then you mentioned writing some, and I figured I should go polish and submit those things | 17:02 |
| mandarine | Hello there | 17:02 |
| torgomatic | next step is a separate test class for tempurl + SLO together | 17:02 |
| mandarine | I have a terrible problem using swift, today : it seems I cannot have consistency | 17:03 |
| notmyname | mandarine: how do you mean? | 17:03 |
| mandarine | I explain : I have 2 proxies and 2 storages. I create a container on proxy1 and I cannot access it on proxy2 | 17:04 |
| notmyname | what error do you get on proxy2? | 17:04 |
| mandarine | accounts and container are rightly repartited on the two storage but I did not activate replication | 17:04 |
| mandarine | A 404 ;) | 17:05 |
| mandarine | It's like the account is not the same as the one in proxy1 | 17:05 |
| notmyname | do you have the same ring files on both machines? check the md5 sums | 17:05 |
| NM | mandarine: can you see the replication between the SN? | 17:05 |
| mandarine | Shall I put every device of proxy1 in a zone and every device of proxy2 in another zone and put "2" in the replication digit during the ring creation ?? | 17:06 |
| mandarine | notmyname: I scp'ed them ;) | 17:06 |
| mandarine | NM: Is there replication if I put '1' as the number of replicas during my ring creation ? | 17:07 |
| *** nacim has quit IRC | 17:07 | |
| mandarine | (e.g. : swift-ring-builder container.builder create 11 1 1 ) | 17:07 |
| NM | notmyname: can correct me but I think you have to replicate at least the accounts and containers. | 17:08 |
| mandarine | Oh. | 17:08 |
| mandarine | I shall try this immediately, then | 17:09 |
| mandarine | Thank you very much in advance :) | 17:09 |
| notmyname | using one replica should work. you're missing out on a lot of the reason swift exists, though | 17:09 |
| *** jogo is now known as flashgordon | 17:10 | |
| notmyname | mandarine: what do the proxy logs on proxy2 tell you? a 404 on a container could be because either the container isn't found or the account isn't found | 17:10 |
| mandarine | well, as the account is not replicated, its just that the container is not in the account | 17:12 |
| mandarine | I agree that swift is a total overkill solution for what I actually intend to do ... | 17:13 |
| mandarine | I shall replicate accounts and containers | 17:13 |
| notmyname | both proxies should be able to access the proxy, even if there is a single replica. assuming you have net connectivity etc | 17:14 |
| mandarine | Yeah, i've tried this | 17:14 |
| mandarine | And accounts and containers are actually created on both storage nodes | 17:15 |
| mandarine | I mean : not replicated. I would describe it like a raid0 : a little bit on storage1 and a little bit on storage2 | 17:15 |
| *** tanee-away is now known as tanee | 17:22 | |
| *** tanee is now known as tanee-away | 17:22 | |
| *** tanee-away is now known as tanee | 17:23 | |
| *** tanee is now known as tanee-away | 17:24 | |
| mandarine | Alright, I have now remade my rings and I have a failure in the container replication | 17:28 |
| *** hurricanerix has joined #openstack-swift | 17:29 | |
| *** tanee-away is now known as tanee | 17:30 | |
| *** tanee is now known as tanee-away | 17:30 | |
| *** fbo is now known as fbo_away | 17:31 | |
| *** tanee-away is now known as tanee | 17:32 | |
| *** tanee is now known as tanee-away | 17:32 | |
| *** tanee-away is now known as tanee | 17:35 | |
| *** tanee is now known as tanee-away | 17:36 | |
| *** tanee-away is now known as tanee | 17:36 | |
| *** piyush has quit IRC | 17:37 | |
| *** ChanServ sets mode: +v swifterdarrell | 17:39 | |
| *** shri has joined #openstack-swift | 17:39 | |
| notmyname | managing different pipelines is a pain: (from my proxy-server.conf on my SAIO) https://gist.github.com/notmyname/733211b5f26473433e5a | 17:43 |
| shri | notmyname: what is mempeek? | 17:44 |
| openstackgerrit | Alex Holden proposed a change to openstack/swift: fixed mispelling in function call name - accross to across https://review.openstack.org/79026 | 17:44 |
| notmyname | shri: magic :-) | 17:44 |
| shri | I'm intrigued :-) | 17:45 |
| notmyname | shri: https://github.com/gholt/mempeek makes finding memory leaks (somewhat) easier (thanks gholt!) | 17:45 |
| shri | oh.. cool | 17:46 |
| mandarine | Okay, now that I set the total number of replica to '2', it seems to work but the replication takes about 40secs. | 17:48 |
| notmyname | torgomatic: wait. you _don't_ want zero byte files named after dictionary words? and especially not ones with apostrophes in them? I'm shocked! | 17:51 |
| *** tanee is now known as tanee-away | 17:51 | |
| torgomatic | notmyname: I know, it's weird, but you'll just have to live with it. :) | 17:52 |
| openstackgerrit | Alex Holden proposed a change to openstack/swift: Fixed mispelling in function name - accross to across https://review.openstack.org/79031 | 17:56 |
| *** piyush has joined #openstack-swift | 18:02 | |
| *** nshaikh has joined #openstack-swift | 18:03 | |
| *** nshaikh has left #openstack-swift | 18:03 | |
| *** piyush1 has joined #openstack-swift | 18:05 | |
| *** piyush has quit IRC | 18:07 | |
| mandarine | Logs tell me : account-replicator 0 successes, 2 failures, but it still seems to work :( | 18:13 |
| *** jamieh has quit IRC | 18:21 | |
| *** piyush has joined #openstack-swift | 18:42 | |
| *** piyush2 has joined #openstack-swift | 18:44 | |
| *** piyush1 has quit IRC | 18:44 | |
| *** piyush has quit IRC | 18:47 | |
| *** occup4nt has quit IRC | 18:51 | |
| *** jamieh has joined #openstack-swift | 18:51 | |
| *** occupant has joined #openstack-swift | 18:51 | |
| *** zaitcev has joined #openstack-swift | 18:58 | |
| *** ChanServ sets mode: +v zaitcev | 18:58 | |
| *** _bluev has quit IRC | 19:02 | |
| *** gholt has quit IRC | 19:02 | |
| *** bada has joined #openstack-swift | 19:02 | |
| *** gholt has joined #openstack-swift | 19:03 | |
| *** ChanServ sets mode: +v gholt | 19:03 | |
| notmyname | gholt: bouncer issues? | 19:03 |
| openstackgerrit | A change was merged to openstack/swift: Functional tests for tempurl https://review.openstack.org/79008 | 19:04 |
| gholt | Nah, security update on my server needing a reboot. | 19:04 |
| gholt | Been a lot of Ubuntu security updates lately. | 19:04 |
| notmyname | gholt: torgomatic's patch landed. will you or dfg add the funtests to https://review.openstack.org/#/c/78766/ ? or should I tackle that next? :-) | 19:07 |
| *** haomaiwang has quit IRC | 19:07 | |
| gholt | It's torgomatic's work; I'll let him take the credit for his own sweat. | 19:07 |
| notmyname | :-) | 19:07 |
| gholt | And tears, probably. | 19:08 |
| torgomatic | This is way more fun than storage policy stuff; I can start *and* finish in a single day! :) | 19:09 |
| * torgomatic has been working on storage policies for a while | 19:09 | |
| *** haomaiwang has joined #openstack-swift | 19:11 | |
| *** mjseger has quit IRC | 19:18 | |
| *** judd7 has joined #openstack-swift | 19:19 | |
| *** _bluev has joined #openstack-swift | 19:20 | |
| *** judd7_ has quit IRC | 19:20 | |
| *** _bluev has quit IRC | 19:28 | |
| *** bagleybter has joined #openstack-swift | 19:29 | |
| openstackgerrit | Samuel Merritt proposed a change to openstack/swift: Functional tests for TempURL and SLO together https://review.openstack.org/79061 | 19:32 |
| openstackgerrit | Samuel Merritt proposed a change to openstack/swift: copy over swift.authorize stuff into subrequests https://review.openstack.org/78766 | 19:32 |
| torgomatic | ^^ rebased on top of master for the functest changes, plus dependent patch with new functest that only passes if your pipeline is in the right order | 19:33 |
| openstackgerrit | John Dickinson proposed a change to openstack/swift: Make PBR based setup completely optional https://review.openstack.org/77091 | 19:34 |
| notmyname | torgomatic: thanks | 19:34 |
| torgomatic | notmyname: np | 19:34 |
| notmyname | I added some color commentary to the PBR patch in the comments of setup.py ("Why aren't we using PBR for every install?"). I'd love feedback if that's not phrased well or not complete | 19:35 |
| portante | torgomatic: any further thoughts on the in-process functional tests? | 19:43 |
| torgomatic | portante: I haven't had a chance to look | 19:43 |
| portante | k thanks | 19:43 |
| *** bsdkurt has quit IRC | 19:46 | |
| notmyname | gholt: torgomatic: thanks for the tests. the previous pipeline asplodes with the new tests (that's good). I'll merge the first one and +2 the tests | 19:49 |
| notmyname | something to highlight in release notes | 19:49 |
| torgomatic | indeed | 19:49 |
| gholt | Oh sweet, was just getting to running all that myself | 19:49 |
| notmyname | why do my computer's speakers seem to crash the computer? | 19:54 |
| torgomatic | magnets? | 19:54 |
| peluse | gremlins? | 19:54 |
| *** zul has quit IRC | 19:55 | |
| openstackgerrit | John Dickinson proposed a change to openstack/swift: Make PBR based setup completely optional https://review.openstack.org/77091 | 19:55 |
| torgomatic | always seemed a little odd to me; you have a metric crapton of teeny-tiny transistors operating at fantastic speed, and then the laptop manufacturer is like "you know what this needs? a big moving magnet right next to it" | 19:56 |
| notmyname | ah, in this case, it's the USB speakers I just plugged in. old Harmon Kardon speakers. never really given me too much trouble before | 19:57 |
| *** Midnightmyth has quit IRC | 20:04 | |
| *** bsdkurt has joined #openstack-swift | 20:08 | |
| *** jamieh has quit IRC | 20:11 | |
| openstackgerrit | A change was merged to openstack/swift: Fixed mispelling in function name - accross to across https://review.openstack.org/79031 | 20:19 |
| *** _bluev has joined #openstack-swift | 20:20 | |
| *** erlon has quit IRC | 20:22 | |
| *** csd has quit IRC | 20:25 | |
| *** _bluev has quit IRC | 20:31 | |
| *** _bluev has joined #openstack-swift | 20:32 | |
| *** judd7_ has joined #openstack-swift | 20:33 | |
| *** judd7 has quit IRC | 20:34 | |
| *** judd7_ has quit IRC | 20:36 | |
| *** _bluev has quit IRC | 20:41 | |
| *** physcx has quit IRC | 20:46 | |
| *** bsdkurt has quit IRC | 20:47 | |
| *** bsdkurt has joined #openstack-swift | 20:49 | |
| * portante wonders what a color visual of the magnetic fields lines around our laptops would look like | 20:49 | |
| *** csd has joined #openstack-swift | 20:50 | |
| *** tdasilva has joined #openstack-swift | 20:52 | |
| *** bagleybter has quit IRC | 21:09 | |
| *** csd has quit IRC | 21:25 | |
| openstackgerrit | A change was merged to openstack/swift: Speed up failing InternalClient requests https://review.openstack.org/78070 | 21:27 |
| notmyname | looks like devstack is set up with tempurl so therefore the new test isn't passing | 21:29 |
| torgomatic | notmyname: yeah, I'm cloning devstack to try to figure out its pipeline ordering stuff | 21:30 |
| notmyname | torgomatic: so far, I think the problem is in the neighborhood of https://github.com/openstack-dev/devstack/blob/master/lib/swift#L71 | 21:31 |
| *** acorwin_ is now known as acorwin | 21:32 | |
| portante | tempurl needs to go in the NOAUTH one, right? | 21:33 |
| notmyname | maybe? but also the test should be skipped if slo isn't enabled, and I don't see any references to that | 21:35 |
| *** csd has joined #openstack-swift | 21:36 | |
| *** NM has quit IRC | 21:38 | |
| peluse | notmyname: is there a PPA that I can add to get 1.13? | 21:46 |
| notmyname | peluse: no, but you can download a tarball | 21:46 |
| peluse | notmyname: OK, stupid question. I need to upgrade from 1.10 to 1.13, there's more to it than just copying over the old install isn't there? | 21:47 |
| notmyname | https://launchpad.net/swift/icehouse/1.13.0 or https://github.com/openstack/swift/archive/1.13.0.zip | 21:47 |
| *** tsg has joined #openstack-swift | 21:48 | |
| notmyname | peluse: handwaving about the different ways you can have it installed? ya, just copying the code over /should/ work | 21:48 |
| peluse | hmm.. OK. getting an error... one sec | 21:49 |
| notmyname | peluse: a couple of config things are important to point out, like the gatekeeper middleware. as always, read the release notes ;-) | 21:49 |
| peluse | RTFRN? | 21:50 |
| peluse | :) | 21:50 |
| peluse | BTW the release notes on that webpage are "This is Swift 1.13.0 release." | 21:53 |
| notmyname | peluse: https://github.com/openstack/swift/blob/master/CHANGELOG | 21:53 |
| peluse | notmyname: thanks, was just being a smart a&* | 21:54 |
| torgomatic | Mar 7 19:55:02 localhost proxy-server: Pipeline is "catch_errors gatekeeper healthcheck proxy_logging memcache container_sync bulk slo dlo ratelimit crossdomain keystoneclient.middleware.auth_token:filter_factory keystoneauth tempauth tempurl formpost staticweb container_quotas account_quotas proxy_logging swift proxy" | 22:13 |
| torgomatic | go go gadget logging | 22:13 |
| torgomatic | ^^ that's the devstack config BTW | 22:15 |
| torgomatic | you can probably tell by the presence of the keystone middleware :) | 22:16 |
| notmyname | do you know where/how that's set? | 22:17 |
| torgomatic | I think it's that stuff in lib/swift that you brought up earlier | 22:17 |
| notmyname | eg a search for "proxy_logging" in the entirety of the devstack repo has 0 results | 22:18 |
| torgomatic | notmyname: looks like they start with the sample config, and then goof it up^W^W^Wcustomize it | 22:18 |
| notmyname | oh, good. I'm sure that has no side-effects. what could go wrong? tests failing? | 22:18 |
| torgomatic | seriously | 22:18 |
| notmyname | you know what would be cool? we should have a solver for middleware to dynamically create a pipeline based on the functionality available ;-) | 22:19 |
| * creiht sighs | 22:20 | |
| notmyname | I wonder if anyone has ever written that | 22:20 |
| notmyname | lol | 22:20 |
| creiht | notmyname: I agree there is a problem with the current state of middleware/wsgi pipeline | 22:20 |
| creiht | a magic solver is not the answer though | 22:20 |
| creiht | our pipeline has like 22 pieces of middleware in it | 22:21 |
| creiht | so what's the problem with the devstack pipeline? (sorry missed the context) | 22:21 |
| torgomatic | creiht: needs to be tempurl [blahblah] slo, but is slo [blahblah] tempurl | 22:22 |
| creiht | ahh | 22:22 |
| creiht | yeah | 22:22 |
| notmyname | and to be realistic, it's not fair to provide a sample config file and not expect people (ie non-contributors, ie normal deployers) to use sed to automatically configure it | 22:23 |
| gholt | Hmm. Didn't we fix the sample conf? | 22:23 |
| creiht | tempurl isn't in the sample | 22:24 |
| gholt | Ah | 22:24 |
| torgomatic | let me try adding it and see if that fixes anything | 22:24 |
| creiht | and to be realistic, we can't expect a solver to magically fix things without there being hidden edge cases and side effects | 22:25 |
| notmyname | hmm..I thought we had added everything to the sample pipeline | 22:25 |
| creiht | :) | 22:25 |
| creiht | [pipeline:main] | 22:25 |
| creiht | pipeline = catch_errors gatekeeper healthcheck proxy-logging cache container_sync bulk slo dlo ratelimit tempauth container-quotas account-quotas proxy-logging proxy-server | 22:25 |
| notmyname | well that's why we have code reviews, right? | 22:25 |
| gholt | It's in the saio pipeline, just not the -sample one | 22:26 |
| creiht | yeah I was looking at the sample one | 22:26 |
| notmyname | so after you install devstack (yes, after nearly 5 years on this project I just installed devstack for the first time) where is the code installed? | 22:26 |
| openstackgerrit | Samuel Merritt proposed a change to openstack/swift: Add tempurl to the example proxy config's pipeline https://review.openstack.org/79094 | 22:26 |
| notmyname | gholt: ah, ok | 22:26 |
| creiht | I want to make a merge prop that uses the sample configs for the saio so that all stays in sync | 22:26 |
| torgomatic | ^^ let's see if that blows up the functests too | 22:27 |
| creiht | notmyname: lol... so when is the first time you are going to use swift? >;) | 22:27 |
| torgomatic | it even already had the config stanza (the [filter:tempurl] stuff) | 22:27 |
| notmyname | creiht: one of these days. I hear it's pretty cool | 22:27 |
| notmyname | torgomatic: but that patch doesn't have your test in it. you're just looking for the setup to break? | 22:28 |
| gholt | There are several things that have conf sections that aren't in the pipeline; mostly because they're optional, but I think we moved away from that at some point. | 22:28 |
| gholt | staticweb, cname thingies, .. | 22:28 |
| torgomatic | notmyname: yup | 22:28 |
| gholt | formpost, etc. :) | 22:29 |
| torgomatic | notmyname: if nothing else, the logging will tell me what the pipeline ends up looking like | 22:29 |
| torgomatic | gholt: yeah, but if devstack is going to use the sample conf as a starting point, we'll have to add the world so we can functionally test it | 22:29 |
| *** NM has joined #openstack-swift | 22:29 | |
| torgomatic | which is slightly annoying, but I'm not gonna lose any sleep over it | 22:30 |
| gholt | That's what I was hinting at. Does your prop add them all I guess. | 22:30 |
| notmyname | creiht's plan is a good one. build saio from the sample (and put all* into the sample config) | 22:30 |
| notmyname | * for some value of all | 22:31 |
| creiht | yeah I would want them all added, and also on saio so that we can be sure that functests are running on a full pipeline | 22:31 |
| gholt | Heh, okay... | 22:31 |
| torgomatic | gholt: no, just tempurl... this one's really more of an experiment than anything, so I only changed one variable | 22:31 |
| notmyname | creiht: https://gist.github.com/notmyname/733211b5f26473433e5a the "basic" one (ie everything but cname_lookup and domain_remap) | 22:31 |
| gholt | domain_remap, cname_lookup, list-endpoints (things I don't intend to maintain. ;) | 22:32 |
| creiht | lol | 22:32 |
| gholt | Sorry, bad humor | 22:32 |
| *** NM has quit IRC | 22:32 | |
| * notmyname suspects gholt will care a lot more about list_endpoints in the future since it's a requirement for zerovm ;-) | 22:32 | |
| gholt | I don't even remember what that was tbh | 22:34 |
| creiht | notmyname: http://summit.openstack.org/cfp/details/15 | 22:34 |
| creiht | :) | 22:34 |
| gholt | Oh yeah, that ringlookupservice thing. | 22:35 |
| notmyname | creiht: actually, that would be a good topic. I may edit the body though. no need for too much snark ;-) | 22:36 |
| creiht | notmyname: it is just for fun | 22:37 |
| gholt | Hmm. That summit thing is weird. Wants me to log in just to view the top level domain? And then it does some insecure redirect thing? | 22:37 |
| creiht | heh | 22:38 |
| gholt | Maybe if I log out of everything and then try to view it... | 22:38 |
| notmyname | creiht: but actually a good topic. the reality is that we're adding more middleware, and there is even more available in the broader ecosystem. and keeping it straight isn't straightforward, especially to those looking to run a storage system and not write one | 22:39 |
| gholt | Eh, nope. Apparently that's a only-registered-club-members site. Good 'nuff | 22:39 |
| creiht | notmyname: also note that I equally poked fun at myself in that description | 22:40 |
| gholt | We're going to shorten our pipeline here by renaming all the middleware: New pipeline = a b c d e f h i j k l m n o p | 22:41 |
| gholt | Don't ask about g -- it made us angry | 22:41 |
| creiht | nah, this is openstack, each item will have a uuid | 22:41 |
| notmyname | creiht: no, a service registry for "core" functionality | 22:42 |
| torgomatic | that's even funnier because you can actually do it :) | 22:42 |
| creiht | notmyname: we should stop now, somebody might get a bright idea | 22:42 |
| creiht | notmyname: oh and of course it is a good topic, I came up with it :) | 22:43 |
| notmyname | ya, I was thinking the same thing. cause the concept of core features (ie independent of a particular project) being what defines things is actually a topic of conversation in some circles | 22:43 |
| notmyname | creiht: a literal and figurative big head ;-) | 22:44 |
| creiht | notmyname: that's Principal Big Head to you ;) | 22:44 |
| notmyname | heh | 22:44 |
| notmyname | now with a flag! | 22:45 |
| notmyname | torgomatic: so I found the swift repo in the devstack install. I didn't install keystone with swift, and I'm running the .functests from the repo. it's _very_ slow (and 3 failures so far) | 22:45 |
| torgomatic | notmyname: woo, fun times; this is on master? | 22:46 |
| notmyname | torgomatic: yes? I cloned devstack and ran stack.sh | 22:46 |
| *** NM has joined #openstack-swift | 22:48 | |
| torgomatic | well, let it finish and let's see what blows up | 22:49 |
| notmyname | torgomatic: yes. it's master | 22:49 |
| torgomatic | er, blew up | 22:49 |
| torgomatic | I tried doing that same thing with devstack once; it took a really long time and tons of tests failed. I don't have that VM any more. | 22:49 |
| notmyname | tons of retries based on connection refused errors | 22:51 |
| torgomatic | so are all the swift-*-server even running? | 22:51 |
| notmyname | ya. | 22:52 |
| creiht | is it possible it could do with how they set up things with just one replica? | 22:52 |
| *** hurricanerix has quit IRC | 22:52 | |
| torgomatic | and is rabbitmq using the utf8mb4 collation for mysql, or just utf8? | 22:53 |
| torgomatic | wait, no, never mind. don't know what came over me. | 22:53 |
| notmyname | torgomatic: I disabled keystone and mysql both | 22:54 |
| torgomatic | :) | 22:54 |
| torgomatic | maybe the connections to keystone are what are getting refused | 22:55 |
| notmyname | could be | 22:55 |
| notmyname | keystoneauth is in the pipeline, but not authtoken (since keystone isn't installed). but keystone auth doesn't talk to any external services | 22:56 |
| *** krtaylor has quit IRC | 22:57 | |
| notmyname | well, if the only perception of swift a "normal" openstack contributor has is what they see in devstack, this is horrible. well and truly aweful | 22:57 |
| torgomatic | notmyname: darn... I'm out of ideas, then. | 22:58 |
| notmyname | who's trying to talk to port 35357 | 22:58 |
| notmyname | is that a keystone thing? | 22:58 |
| torgomatic | I think that's a keystone thing | 22:58 |
| torgomatic | it's got two ports | 22:58 |
| notmyname | hmm...I am seeing other keystone errors. so ya, that's it | 22:59 |
| notmyname | but why? | 22:59 |
| gholt | Well, (and I'm to blame here too), if some of us swift devs actually participated in making devstack the stack it is.... | 23:00 |
| notmyname | gholt: by lack of participation? | 23:00 |
| gholt | I'm just trying to get better about only complaining about things I've actually tried to influence. :) | 23:01 |
| *** NM has quit IRC | 23:01 | |
| notmyname | argh! there is an authtoken middleware in the pipeline. I suppose that's just put there by default | 23:03 |
| *** dmsimard has quit IRC | 23:04 | |
| notmyname | anyone know the devstack commands to restart a process? it's not swift-init | 23:04 |
| *** _bluev has joined #openstack-swift | 23:10 | |
| *** piyush2 has quit IRC | 23:15 | |
| shri | notmyname: just run stack.sh | 23:15 |
| notmyname | that actually rebuilds the whole environment. I just wanted to restart processes | 23:16 |
| shri | oh.. sorry, don't know about that. | 23:16 |
| *** zigo has quit IRC | 23:17 | |
| notmyname | got it. I now have functests working on devstack without keystone. (1) take out the keystone stuff in the pipeline (2) actually use the values in the sample test conf file (3) it works! | 23:20 |
| notmyname | anyone know what the default reseller prefix for keystone accounts is? | 23:28 |
| notmyname | ah. it's AUTH | 23:29 |
| notmyname | also, why do we have 2 auth systems in our codebase with the same default reseller prefix? ;-) | 23:29 |
| openstackgerrit | Alex Pecoraro proposed a change to openstack/swift: Allow hostname for nodes in Ring https://review.openstack.org/74542 | 23:30 |
| notmyname | oh. *sigh* (to borrow from creiht) | 23:32 |
| notmyname | keystone's authtoken middleware makes a call to keystone for every request, no matter what the reseller prefix is. | 23:33 |
| *** booi has joined #openstack-swift | 23:35 | |
| * notmyname sees the scary jungle of "fixing" devstack to work without keystone and just goes to install keystone | 23:41 | |
| openstackgerrit | A change was merged to openstack/swift: copy over swift.authorize stuff into subrequests https://review.openstack.org/78766 | 23:51 |
| openstackgerrit | Samuel Merritt proposed a change to openstack/swift: Functional tests for TempURL and SLO together https://review.openstack.org/79061 | 23:54 |
| torgomatic | alright, now that's on top of adding tempurl to the sample pipeline in the right place; let's see if it works | 23:54 |
| notmyname | torgomatic: I approved them all | 23:55 |
| torgomatic | notmyname: thanks | 23:55 |
| notmyname | and a task to ensure that the saio matches the sample. or vice versa. | 23:57 |
| notmyname | FWIW, 78 seconds for func tests on devstack without keystone, 92 seconds with keystone | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!