Tuesday, 2015-06-09

notmynamemattoliverau: I wanted to ask you about container sharding. how's it going?00:02
notmynamemattoliverau: also, have you talked to alex about docs at all? I'm going to try to send an email to her and the hp docs person tomorrow00:03
mattoliveraunotmyname: I have talked briefly to Alex, told her would introduce her and get her more information when they came to hand. She's keen00:07
mattoliveraunotmyname: sharing is going well, I've come up with a new "find best candidate subtrie" that doesn't use much ram. Tested it on some really large containers I've made, and the algorithm seems to work. So spend most of Friday last week plumbing it into the POC, plan on testing it this morning. Once that's done, I'm going to start running some better bench marks and start working on shrinking.00:09
notmynamewhat can we do to get more eyes on it?00:09
mattoliverauNow that is a good question :) If the benchmarks go well, maybe get it out of my github and into something more visable.00:11
swifterdarrellmattoliverau: yay benchmarks!00:12
notmynameI added a line on it to the meeting agenda00:12
*** asettle-afk is now known as asettl00:14
*** asettl is now known as asettle00:14
*** dmorita has joined #openstack-swift00:15
notmynamemattoliverau: thanks for talking to alex00:15
mattoliveraunotmyname: cools, I'll see what I can get by then :) I have a team member of jrichli helping me on sharinding now, cause they were interested, so that's awesome.00:16
notmynamein irc?00:16
mattoliverauHis IRC nick is blmartin,  getting him up to speed. He's new to swift, so given him some brain dumps etc, and got him working on some unit tests until he feels he is ready to bite into something more meaty ;)00:18
notmynamemattoliverau: also, you were one of the last people to comment on https://bugs.launchpad.net/swift/+bug/897451. If you don't leave a comment on there by the meeting, I'm going to ask you about it in the meeting :-)00:18
openstackLaunchpad bug 897451 in OpenStack Object Storage (swift) "memcached exception when trying to connect to offline node" [Medium,Confirmed] - Assigned to Matthew Oliver (matt-0)00:18
*** ho has joined #openstack-swift00:19
notmynamemattoliverau: also, same story with https://bugs.launchpad.net/swift/+bug/104088300:19
openstackLaunchpad bug 1040883 in OpenStack Object Storage (swift) "object-server partital write prevention not tested" [Low,In progress] - Assigned to Matthew Oliver (matt-0)00:19
notmynamemattoliverau: I'm not trying to dump it all on you, but you were the last person on these (and you assigned them to yourself)00:20
notmynamesee what happens when i start looking at launchpad? ;-)00:20
mattoliveraunotmyname: lol00:23
notmynameand with that, I'm headed home00:23
mattoliveraunotmyname: no probs, I'll go and reset my brain on them, sorry for letthing them drop out of my mind not prioritising them00:24
notmynameno worries at all00:24
hogood morning!00:28
mattoliverauho: morning00:29
*** wbhuber has joined #openstack-swift00:31
*** gyee_ has quit IRC00:32
*** kutija|away has quit IRC00:33
homattoliverau: morning!00:33
*** kutija has joined #openstack-swift01:16
*** lastops has quit IRC01:20
*** lastops has joined #openstack-swift01:21
*** Kudos2me2 has joined #openstack-swift01:30
*** jamielennox|away is now known as jamielennox01:32
*** Kudos2me2 has quit IRC01:33
*** haigang has joined #openstack-swift01:34
*** asettle is now known as asettle-afk01:35
wbhubernotmyname: you mentioned that you had 6+4 scheme across 5 servers, meaning 5 storage nodes? 20 drives each node?  i guess that cluster was configured to handle 2 each server (5 * 2 = 6 + 4 scheme).01:43
*** kota_ has joined #openstack-swift01:49
*** ChanServ sets mode: +v kota_01:49
wbhubernotmyname: disregard my question.01:53
*** jkugel has left #openstack-swift02:37
*** wbhuber has quit IRC02:57
*** asettle-afk is now known as asettle03:02
*** nottrobin has quit IRC03:06
*** briancurtin has quit IRC03:06
*** zhiyan has quit IRC03:06
*** serverascode has quit IRC03:06
*** jeblair has quit IRC03:08
*** cppforlife_ has quit IRC03:09
*** lastops has quit IRC03:09
*** harlowja has quit IRC03:22
*** harlowja_ has joined #openstack-swift03:22
*** blmartin has joined #openstack-swift03:35
*** ajiang has left #openstack-swift04:08
*** haigang has quit IRC04:30
*** zhill has joined #openstack-swift04:38
*** ppai has joined #openstack-swift04:51
*** blmartin has quit IRC04:54
*** ktsuyuzaki has joined #openstack-swift04:56
*** kota_ has quit IRC04:57
*** kota_ has joined #openstack-swift05:00
*** ChanServ sets mode: +v kota_05:00
*** SkyRocknRoll has joined #openstack-swift05:01
*** ktsuyuzaki has quit IRC05:02
*** ktsuyuzaki has joined #openstack-swift05:05
*** Triveni has joined #openstack-swift05:06
*** kota_ has quit IRC05:07
*** proteusguy has joined #openstack-swift05:08
*** zhill_desktop has quit IRC05:10
*** zhill has quit IRC05:11
*** kota_ has joined #openstack-swift05:13
*** ChanServ sets mode: +v kota_05:13
*** haigang has joined #openstack-swift05:15
*** ktsuyuzaki has quit IRC05:16
*** ktsuyuza_ has joined #openstack-swift05:16
*** kota_ has quit IRC05:18
*** kota_ has joined #openstack-swift05:19
*** ChanServ sets mode: +v kota_05:19
*** ktsuyuza_ has quit IRC05:21
*** ktsuyuzaki has joined #openstack-swift05:21
*** SkyRocknRoll has quit IRC05:23
*** kota_ has quit IRC05:23
*** kota_ has joined #openstack-swift05:26
*** ChanServ sets mode: +v kota_05:26
*** ktsuyuzaki has quit IRC05:28
*** ktsuyuzaki has joined #openstack-swift05:28
*** kota_ has quit IRC05:30
*** kota_ has joined #openstack-swift05:31
*** ChanServ sets mode: +v kota_05:31
*** ktsuyuzaki has quit IRC05:33
*** ktsuyuzaki has joined #openstack-swift05:34
*** kota_ has quit IRC05:36
*** haomaiwa_ has quit IRC06:08
*** silor has joined #openstack-swift06:22
*** ktsuyuzaki has quit IRC06:27
*** nottrobin has joined #openstack-swift06:30
*** haomaiwang has joined #openstack-swift06:35
*** serverascode has joined #openstack-swift06:39
*** mmcardle has joined #openstack-swift06:41
*** zhiyan has joined #openstack-swift06:46
*** ahonda has joined #openstack-swift06:47
*** cppforlife_ has joined #openstack-swift06:53
*** briancurtin has joined #openstack-swift07:09
*** mfalatic has joined #openstack-swift07:25
mfalaticFor SAIO, how can I force object reclamation to start after adjusting the object-replicator values in the configs? Trying to get tombstones to be collected quickly (for the purpose of experimentation)07:26
mattoliveraushorten the reclaim age, and shorten the run_pause between replication jobs07:35
*** geaaru has joined #openstack-swift07:49
mfalaticrun_pause is only 30 seconds by default... I've set reclaim_age to 60 and it's still not happening - could something else be triggering it?07:52
*** haomaiwang has quit IRC07:55
*** haomaiwang has joined #openstack-swift07:55
*** SkyRocknRoll has joined #openstack-swift07:57
*** haomaiwang has quit IRC08:05
*** jordanP has joined #openstack-swift08:20
*** haomaiwang has joined #openstack-swift08:20
*** mandarine has joined #openstack-swift08:21
*** mfalatic has quit IRC08:22
*** mfalatic has joined #openstack-swift08:22
mandarineHello Swifties (sorry, this was too easy and I couldn't not do this)08:34
mandarineI appear to have two more questions :)08:34
mandarine1) Do any of you know a way to change the zone of a device?08:35
mandarine2) Is there an internal mechanism in swift to make empty container expire?08:35
mandarineIf you have a lead on any of these two topics, I'll be eternally thankful (well, relative eternity is a thing, ain't it?)08:36
*** leopoldj has joined #openstack-swift08:42
*** chlong has quit IRC08:52
*** joeljwright has joined #openstack-swift08:56
*** proteusguy has quit IRC08:57
*** aix has joined #openstack-swift09:05
openstackgerritChristian Schwede proposed openstack/swift: WIP - Empty test commit - do not merge  https://review.openstack.org/18960909:20
homandarine: (1) Use swift-ring-builder <builder_file> set_info (2) I think swift doesn't have container expire.09:27
*** tellesnobrega has quit IRC09:28
*** dmorita has quit IRC09:34
*** silor has quit IRC09:41
*** Triveni has quit IRC09:44
*** aix has quit IRC09:53
*** aix has joined #openstack-swift10:05
*** km has quit IRC10:16
openstackgerritJiri Suchomel proposed openstack/swift: Let swift-object-info print information about files in given directory, if directory is provided instead of data file together with --recursive option.  https://review.openstack.org/18925810:45
*** silor has joined #openstack-swift11:20
*** distroguy has joined #openstack-swift11:28
*** csmart has quit IRC11:30
*** distroguy has quit IRC11:32
*** SkyRocknRoll has quit IRC11:55
*** kei_yama has quit IRC11:56
*** haigang has quit IRC12:02
*** joeljwright has quit IRC12:04
*** tellesnobrega has joined #openstack-swift12:11
*** XiOViC has joined #openstack-swift12:21
*** XiOViC has quit IRC12:22
*** ppai has quit IRC12:23
openstackgerritJiri Suchomel proposed openstack/swift: Let swift-object-info print information about files in given directory, if directory is provided instead of data file together with --recursive option.  https://review.openstack.org/18925812:29
*** thurloat_isgone is now known as thurloat12:33
*** chlong has joined #openstack-swift12:37
mandarineho: The set_info thingy doesn't seem to be okay if I modify zones12:39
*** joeljwright has joined #openstack-swift12:40
*** acoles_away is now known as acoles12:44
acolesnotmyname: oh dear (lost pin), does that mean we will lose all our good communication and harmony ;)12:51
dmsimardApple's Open sourcing Swift -- should definitely abandon the Go rewrite and go with Swift so you can say Swift is written in Swift /s12:57
*** wbhuber has joined #openstack-swift13:00
*** wbhuber has quit IRC13:07
*** acampbell has joined #openstack-swift13:11
*** janonymous_ has joined #openstack-swift13:12
janonymous_Hi, I have a Question: Just like every package comes with uninstall procedure , it there any uninstall procedure for swift also ?13:14
*** petertr7_away is now known as petertr713:14
janonymous_+ **along with dependencies.13:16
petertr7Hello! I was wondering if anyone was familiar with form post. In particular, my cluster has trouble handling large file uploads (without segmenting roughly 4-5gb) through formposts. Specifically I get a 500 Internal Error.13:19
petertr7In addition, if I run a script to continuously upload files onto swift, formpost is unable to handle even 2gb files.13:19
*** proteusguy has joined #openstack-swift13:19
petertr7I was reading through the formpost code and thinking perhaps the way the cluster handles form post and other uploads causes conflicts of resources13:20
*** lastops has joined #openstack-swift13:21
*** jamielennox is now known as jamielennox|away13:23
cschwedepetertr7: the 2gb fompost limitation sounds familiar, which swift version are you running?13:27
cschwedepetertr7: https://github.com/openstack/swift/commit/7056ec6a16fd8707564ec4b0a05cab461ee2a80d13:27
petertr7Good morning/afternoon Christian! I'm running Kilo13:27
cschwedepetertr7: Hello Peter! Well, that patch is included in Kilo, so there must be another reason. Do you have access to the proxy server logs?13:29
petertr7I should. Let me check13:30
petertr7They should be in /etct/swift/logs?13:30
petertr7Sorry, my mistake, we're still on Juno13:32
cschwedeMore likely /var/logs/swift13:32
petertr7Strangely my /var/logs/swift/proxy.log is empty and not logging any errors as I attempt to upload large files13:34
*** janonymous__ has joined #openstack-swift13:36
*** janonymous_ has quit IRC13:36
jordanPpetertr7, do you have a webserver in front of the proxy ? like apache ? if so check its logs maybe13:36
petertr7That's right, apache atm. I'm pulling up the logs now13:37
petertr7My /var/log/apache2/error.log doesn't get updated on failed uploads either13:38
*** acampbell has quit IRC13:39
*** acampbell has joined #openstack-swift13:40
*** ppai has joined #openstack-swift13:40
*** jeblair has joined #openstack-swift13:43
janonymous__http://docs.openstack.org/developer/swift/development_saio.html#setting-up-rsync-----> sudo apt-get install policycoreutils (is also needed which doesn't come in ubuntu packages by default, should be added in doc)  for -- >setsebool-->sudo setsebool -P rsync_full_access 113:50
*** ajiang has joined #openstack-swift13:53
*** breitz has quit IRC13:56
*** ptb has joined #openstack-swift13:59
*** wbhuber has joined #openstack-swift14:00
*** ptb has quit IRC14:01
*** breitz has joined #openstack-swift14:02
*** blmartin has joined #openstack-swift14:03
openstackgerritCharles Hsu proposed openstack/python-swiftclient: Add ability to download objects to particular folder.  https://review.openstack.org/16028314:06
*** ppai has quit IRC14:07
*** proteusguy has quit IRC14:11
*** proteusguy has joined #openstack-swift14:12
*** jrichli has joined #openstack-swift14:16
*** ptb has joined #openstack-swift14:16
*** lpabon has joined #openstack-swift14:32
*** minwoob has joined #openstack-swift14:46
*** lpabon has quit IRC14:48
*** ptb has quit IRC14:49
*** leopoldj has quit IRC14:50
janonymous__Hi ,14:52
janonymous__Jun  9 20:22:01 janonymous- proxy-server: STDERR: (10485) accepted ('', 48084) Jun  9 20:22:01 janonymous- account-server: STDERR: (10534) accepted ('', 43394)14:53
swifterdarrellmfalatic: still unable to get tombstones reaped?14:53
janonymous__could somebody point out the problem ..14:53
mfalaticHey there14:54
mfalaticswifterdarrell - they do eventually get reaped (after a day or more it seems) but I had no luck getting them to reap in the 60 second timeframe by adjusting reclaim_age.14:56
mfalaticnote that this is SAIO based on kilo stable, nothing unusual.14:57
mfalaticI added "reclaim_age 60" to object_expirer.conf as well as the four confs under object_server14:59
mfalatic"reclaim_age = 60"14:59
swifterdarrellmfalatic: perhaps double-check that the reclaim_age setting is under [object-replicator] for all object server confs14:59
mfalaticIt is14:59
swifterdarrellmfalatic: and then restart them? or use the --run-one or whatever, if that deamon has that flag?15:00
mfalaticIt's not urgent BTW, just peculiar. I actually did resetswift, remakerings and swift-init main stop followed by start.15:01
mfalaticThen I uploaded a test file, deleted it, and watched the /srv filesystem. The tombstones didn't go away for a many minutes I spent checking in on it periodically.15:02
swifterdarrellmfalatic: what kind of start?   startmain or startall?15:02
mfalaticstart-init main15:02
mfalatic(which is what saio has you do)15:03
swifterdarrellmfalatic: object-replicator isn't part of main, I don't think; so you'll need to run the replicator by hand or startall15:03
mfalaticstartmain = start-init main of course15:03
mfalatichmm. Replicator is separate? Let me look.15:03
swifterdarrellmfalatic: looks like swift-object-replicator has this option:    -o, --once            only run one pass of daemon15:04
swifterdarrellmfalatic: ya, "main" is just the swift-*-server procs15:04
mfalaticI've rebooted the vm this is all in.15:05
mfalaticNot seeing any swift processes at this point (before manually starting things)15:06
mfalaticI've reset swift and remade the rings. I normally just run startmain at this point.15:08
mfalaticI see startrest there... but I've never actually run it. Wouldn't it be quite broken if replication isn't running at that point?15:09
*** chlong is now known as chlong-zzz15:10
mfalaticswifterdarrell: that's interesting... I've never run "rest" but clearly it kicks off many more processes.15:11
mfalaticyet up to now it seemed like things worked normally despite that, for what I've tested.15:12
swifterdarrellmfalatic: it's an SAIO... there's little definition of "broken"15:12
swifterdarrellmfalatic: if by "broken" you mean that the background consistency daemons (auditor, replicator, reapers, etc.) aren't running then yes, "startmain" leaves your SAIO "broken"15:12
swifterdarrellmfalatic: really, for lots of testing you only need the main daemons and all the daemons consume more RAM and whatnot, and often SAIO vms are small15:13
swifterdarrellmfalatic: do the SAIO docs not mention startrest and how background daemons run/get started and how they're separate from the "main" servers?15:13
swifterdarrellmfalatic: (I haven't looked at SAIO docs in a whlie)15:13
swifterdarrellmfalatic: while we're on the topic, clayg made a super-sweet Vagrant-based SAIO that I've been using for a bit now:  https://github.com/swiftstack/vagrant-swift-all-in-one15:14
mfalaticWill make a note of that, thanks!15:16
swifterdarrellmfalatic: np15:16
*** annegentle has joined #openstack-swift15:16
mfalaticswifterdarrell: As a note, even with "rest" running, tombstones aren't getting removed any more quickly. My main question is, which config should have the reclaim_age set for this to work?15:17
*** annegentle has quit IRC15:18
swifterdarrellmfalatic: what'ever the swift-object-replicator is using; and inside that, under the [object-replicator] section15:19
swifterdarrellmfalatic: process command-line ought to indicate the config file used by swift-object-replicator15:19
thurloathas anyone had experience with the seagate HMR drives in a swift deployment?15:21
thurloatsorry, SMR15:21
mfalaticswifterdarrell: Yep - (It's clearer now that all the processes are running) it uses the object-server configs... which definitely have the reclaim_age set in them.15:21
*** adutta has joined #openstack-swift15:22
aduttaThe accoun auditor throws the following error for quite a few accounts: "The total object_count for the container (5) does not match the sum of object_count across policies (0)" . Probably it is thrown at this line https://github.com/openstack/swift/blob/master/swift/account/auditor.py#L123 . Any reason on why it may be happening. It is after upgrade from 1.12 to 2.2.215:25
*** aix has quit IRC15:29
swifterdarrellmfalatic: maybe there's something else affecting actual reclaiming in the code that we're forgetting; I don't have time to dig atm15:31
mfalaticThat's ok, at least now I know to run "rest" (which with "main" == "all")15:33
mfalaticThanks for the vagrant tip!15:33
*** gyee_ has joined #openstack-swift15:34
*** lpabon has joined #openstack-swift15:38
*** lpabon has quit IRC15:40
*** aix has joined #openstack-swift15:41
*** Fin1te has joined #openstack-swift15:51
*** haomaiwang has quit IRC15:55
openstackgerritAlex Khivin proposed openstack/swift: Make umask customizable for Swift daemons  https://review.openstack.org/18978515:55
*** haomaiwang has joined #openstack-swift15:55
openstackgerritAlex Khivin proposed openstack/swift: Make umask customizable for Swift daemons  https://review.openstack.org/18978916:00
hugokuomorning !16:01
mfalaticIndeed it is16:03
hugokuoerr.. actually it's midnight here :)16:04
hugokuois there a way to disallow any access to an Swift account temporarily ?16:05
hugokuoaccount rate limit to 0 ?16:05
*** petertr7 is now known as petertr7_away16:07
openstackgerritAlex Khivin proposed openstack/swift: Make umask customizable for Swift daemons  https://review.openstack.org/18978516:09
*** gyee_ has quit IRC16:10
peluseclayg, around?16:14
openstackgerritMichael Barton proposed openstack/swift: go: ring self-reloading  https://review.openstack.org/18703016:18
*** annegentle has joined #openstack-swift16:25
*** lcurtis has joined #openstack-swift16:37
*** jordanP has quit IRC16:38
*** aix has quit IRC16:39
*** Fin1te has quit IRC16:42
hurricanerix_hugokuo: I think the ratelimit middleware allows you to black/white list accounts: http://docs.openstack.org/developer/swift/ratelimit.html#black-white-listing16:45
*** ptb has joined #openstack-swift16:45
*** ptb has quit IRC16:46
*** ptb has joined #openstack-swift16:47
*** ptb has quit IRC16:48
*** acampbell has quit IRC16:49
hugokuohurricanerix_: thx. This way may still allow GET request to the account ?16:50
hurricanerix_hugokuo: no, if you blacklist an account, it should prevent all access.16:51
hugokuohurricanerix_: nice. I'll check the code tmr. Thanks for this information. pretty useful.16:51
hugokuoX-Account-Sysmeta-Global-Write-Ratelimit <--- make me a bit confused due to the string "Write" in it :)16:52
hurricanerix_hugokuo: Yeah, I just tested it on my SAIO, and it did prevent GETs.16:58
hugokuohurricanerix_: appreciate for a quick try . You rock16:59
*** petertr7_away is now known as petertr717:00
*** Sripadwallabha has joined #openstack-swift17:00
SripadwallabhaHello @all17:00
SripadwallabhaI am trying to use python-swiftclient and trying to authenticate using user_id instead of user_name17:01
Sripadwallabhais it possible?17:01
SripadwallabhaI am getting following excption17:01
Sripadwallabha  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1209, in get_auth     insecure=self.insecure)   File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 406, in get_auth     auth_version=auth_version)   File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 341, in get_auth_keystone     raise ClientException('Authorization Failure. %s' % err)17:01
*** dstanek has left #openstack-swift17:02
Sripadwallabhaif I dont specify user_name but just mention user_id inside os_options dict17:02
hurricanerix_hugokuo: So the first request after blacklisting was allowed, but after that is blocks access.  http://pastebin.com/Rw9dE9su17:03
hugokuohurricanerix_: perfect. I like it. I'm going to update and add more stuff into Swift community document.17:05
hugokuoctennis: mlanner : Check this out http://pastebin.com/Rw9dE9su   .... This could be the best choice to disallow access to the account :)17:06
openstackgerritLouis Taylor proposed openstack/python-swiftclient: Rename `obj` to `name` in *_object wrappers  https://review.openstack.org/18981517:07
*** kevinc_ has joined #openstack-swift17:07
*** acoles is now known as acoles_away17:10
notmynamegood morning17:11
*** mmcardle has quit IRC17:12
*** remix_tj has quit IRC17:12
kevinc_If I have admin access on keystone, is there any way to do a head on every tenant with out having the admin account added to every tenant?17:13
*** remix_tj has joined #openstack-swift17:14
notmynamekevinc_: if that user has the "reseller admin" role/group, then it will work. IIRC there's some keystone config to map the swift reseller admin to a role in keystone17:23
notmynamekevinc_: but the general answer is that a "reseller admin" as full read/write access for a given auth system (ie root-like)17:23
*** zhill_desktop has joined #openstack-swift17:29
*** SkyRocknRoll has joined #openstack-swift17:30
*** Sripadwallabha has left #openstack-swift17:44
*** janonymous__ is now known as new17:47
*** new is now known as zzzzz17:47
*** mmcardle has joined #openstack-swift17:47
*** otoolee has quit IRC17:47
*** zzzzz has left #openstack-swift17:48
*** zzzzz has joined #openstack-swift17:48
*** zzzzz is now known as janonymous_17:48
kevinc_notmyname: perfect, thank you that solved my issue17:49
*** torgomatic has quit IRC17:52
*** torgomatic has joined #openstack-swift17:53
*** ChanServ sets mode: +v torgomatic17:53
*** otoolee has joined #openstack-swift17:55
kevinc_One more question, when I add a tenant to Keystone when does the account/tenant get added to Swift? Does the account/tenant get added when the tenant is added to keystone, when the user first tries to access Swift or something else?17:56
*** mmcardle has quit IRC18:00
*** gyee_ has joined #openstack-swift18:02
*** geaaru has quit IRC18:02
*** harlowja_ has quit IRC18:18
hugokuokevinc_: 1) If account_auto_create=true , the swift account for corresponding keystone tenant will be created while the user try to access it first time18:22
hugokuokevinc_: 2) if auto create = false, the reseller admin must creates the swift account for the tenant as initialization18:23
kevinc_hugokuo: thanks!18:24
hugokuokevinc_: np18:24
*** harlowja has joined #openstack-swift18:25
openstackgerritJiri Suchomel proposed openstack/swift: Let swift-object-info print information about files in given directory, if directory is provided instead of data file together with --recursive option.  https://review.openstack.org/18925818:26
*** annegentle has quit IRC18:30
*** si1v3r has joined #openstack-swift18:35
si1v3rHey all. We have 20ish object storage nodes and one is running a much higher load than it's peers with about 100 more running rsync processes.18:36
si1v3rWhere would be a good place to find out why one node would be running so many more than peers?18:37
*** blmartin has quit IRC18:39
ctennisI'd look in the swift logs and see if replication is running, perhaps due to hardware issues18:39
si1v3robject-replicator logs show a similar number of events between the bad node and good ones.18:42
*** acampbell has joined #openstack-swift18:42
*** petertr7 is now known as petertr7_away18:47
*** openstackgerrit has quit IRC18:48
*** openstackgerrit has joined #openstack-swift18:48
*** petertr7_away is now known as petertr718:49
*** silor1 has joined #openstack-swift18:52
*** silor has quit IRC18:52
si1v3rLooks like the old rsyncs are hanging onto one specific disk, according to lsof18:53
notmynamemaybe an old replicatin process is running and firing those off18:58
si1v3rThere is this monster running: http://pastebin.com/NmzUqS5m19:01
si1v3rother rsync processes seem to be replicating objects from more than one disk at a time.19:02
ctennisdid you change your storage?  or do you have a failed disk elsewhere?19:02
si1v3rNo changes have been made recently (all users say that...), the disk is working and has no errors in dmesh.19:03
ctennisreplication wouldn't be running unless something is failed or something changed19:03
ctennisrsync replication that is19:03
MooingLemurpeluse, clayg: any chance either of you might be available today to take a look at a reconstruction issue that's been happening?  I've kinda left the cluster alone so that the issue will actually persist, so there might be a smoking gun pointing to an actual bug that can be found :)19:04
ctennisor things were out of whack for a while19:04
*** acampbell has quit IRC19:04
*** annegentle has joined #openstack-swift19:06
si1v3rctennis: Is there a way to see what triggered it?19:07
ctennisyou could find a partition that is replicating to another node and see where it's supposed to be using swift-get-nodes, and check the primary locations.  if it just happened all of a sudden it smells like you have a bad disk somewhere in the system.19:09
*** annegentle has quit IRC19:14
*** kevinc_ has quit IRC19:22
*** marcusvrn has joined #openstack-swift19:24
*** annegentle has joined #openstack-swift19:30
*** Syed_A has joined #openstack-swift19:34
Syed_AHello folks, I created a SwiftOperator role, added it in proxy-server.conf but  when a user with SwiftOperator roles tries to use swift it  gets 403(forbidden), however a user with admin role can successfully use the commands.19:36
Syed_AHow do i make sure that SwiftOperator also has admin like privileges to use swift19:37
*** gyee_ has quit IRC19:38
occupantYou may want a cyanide pill in case things go bad.19:38
Syed_ADo i need to make any change in keystone policy.json file19:42
*** InAnimaTe has left #openstack-swift19:50
notmynameSyed_A: today swift doesn't use the policy.json file. so I don't htink that would help anything19:52
*** mmcardle has joined #openstack-swift19:54
*** silor1 has quit IRC19:57
Syed_Anotmyname: I am slightly confused as to why a user with operator_role i.e. SwiftOperator is forbidden from using swift19:59
*** jamielennox|away is now known as jamielennox20:01
Syed_Anotmyname: operator_roles = Member, admin, SwiftOperator, ResellerAdmin, _member_20:02
notmynameSyed_A: there are 2 config vars that can be set in swift (in proxy-server.conf in the [filter:keystoneauth] section) that relate to what you're doing20:04
notmynameoperator_roles and reseller_admin_role20:04
notmynamethe first is for what you would consider an "owner" of the swift account. eg you get an account on a swift cluster and you can do what you want with it, but you can't mess with stuff in other accounts20:05
*** SkyRocknRoll has quit IRC20:06
notmynamethe second is what you'd consider the owner of every account that keystone manages in swift. ie it's the owner of the entire reseller (keystone in your case) and can do anything in any account for that auth system20:06
notmynameit's a bad idea to have end-users that have a reseller admin role. that role is intended for operators20:07
notmynameI'll admin that the first config value (operator_roles) is poorly named. I think that's because of early miscommunication between swift and keystone about what is what, and it's stuck because of compatibility reasons20:08
Syed_Anotmyname: Roger that. I did the first one. Created a user and then assigned it the SwiftOperator role.20:09
notmynameok. I think that just means that the user will have access to their own account20:10
Syed_Anotmyname: this user is able to get authenticated by keystone but somehow it is getting rejected by swift, thus getting a 40320:10
notmynameSyed_A: in the [app:proxy-server] section, do you have account_autocreate set to true?20:11
Syed_Anotmyname: Yes i do.20:12
Syed_Anotmyname: I can pastebin the proxy-server.conf for you ?20:12
*** mmcardle has quit IRC20:13
*** lpabon has joined #openstack-swift20:14
Syed_Anotmyname: Here it is: http://pastebin.com/z4Aedt2220:15
*** thurloat is now known as thurloat_isgone20:20
notmynameSyed_A: looking20:24
Syed_Anotmyname: keystone endpoint-list : http://pastebin.com/LAkpcjTw20:25
*** zaitcev has joined #openstack-swift20:28
*** ChanServ sets mode: +v zaitcev20:28
notmynameSyed_A: do you see anything in the logs?20:32
notmynameI see one place (only one) that can return a 403 and also has some extra logging20:33
Syed_Anotmyname: Nope, i don't see anything unsual.20:36
Syed_Anotmyname: What is it, please do tell :)20:36
*** lastops has quit IRC20:37
Syed_Anotmyname: when i use --debug flag, i can see this : INFO:urllib3.connectionpool:Starting new HTTP connection (1): proxy.cloud.comDEBUG:urllib3.connectionpool:"HEAD /v1/AUTH_s HTTP/1.1" 403 020:46
Syed_Anotmyname: http://paste.openstack.org/show/278851/20:47
Syed_Anotmyname: proxy-server logs http://paste.openstack.org/show/278853/20:49
openstackgerritJoel Wright proposed openstack/python-swiftclient: Reduce memory usage for download/delete and add --no-shuffle option to st_download  https://review.openstack.org/13033920:49
zaitcevSyed_A: You typoed something in the endpoint descriptor in Keystone. That AUTH_s is the remnant of what should be interpolated from AUTH_%(tenant_id)s.20:55
*** petertr7 is now known as petertr7_away20:56
*** thurloat_isgone is now known as thurloat20:59
*** annegentle has quit IRC21:00
Syed_Anotmyname: Ahhh, i feel so stupid.21:04
Syed_Anotmyname: Thanks a lot for pointing this out :)21:04
Syed_Anotmyname: I was struggling from past 2 days21:04
notmynameSyed_A: was it what zaitcev said?21:04
Syed_Anotmyname: Ahh yeas, looks like it. I missed it in the endpoint. I have to check it.21:06
Syed_Azaitcev: Thanks man :)21:06
notmynamethanks zaitcev!21:06
*** annegentle has joined #openstack-swift21:10
*** wbhuber_ has joined #openstack-swift21:14
*** thurloat is now known as thurloat_isgone21:14
*** wbhuber has quit IRC21:17
*** csmart has joined #openstack-swift21:18
zaitcevSyed_A: That syntax catches people all the time, I wish we had a simple special case... Like, AUTH_# and voila.21:20
*** annegentle has quit IRC21:21
*** annegentle has joined #openstack-swift21:22
*** csmart has quit IRC21:22
*** lpabon has quit IRC21:26
*** evanjfraser has quit IRC21:27
*** evanjfraser has joined #openstack-swift21:27
Syed_Azaitcev: Yes it does,  i spent a lot of time trying to figure it out. Thanks to you now it is fixed. :)21:30
*** csmart has joined #openstack-swift21:32
*** lpabon has joined #openstack-swift21:34
*** lpabon has quit IRC21:36
*** csmart has quit IRC21:36
*** Syed_A has quit IRC21:38
*** thurloat_isgone is now known as thurloat21:45
*** thurloat is now known as thurloat_isgone21:47
*** csmart has joined #openstack-swift21:53
*** joeljwright has quit IRC21:54
*** csmart has quit IRC21:57
*** csmart has joined #openstack-swift22:03
si1v3rOk, I cannot figure out the difference between this slot drive and the rest of the cluster. How can I make sure it's safe to reformat the thing?22:06
*** wbhuber_ has quit IRC22:06
*** wbhuber_ has joined #openstack-swift22:06
*** wbhuber__ has joined #openstack-swift22:07
*** csmart has quit IRC22:08
*** annegentle has quit IRC22:08
*** jrichli has quit IRC22:09
*** wbhuber_ has quit IRC22:11
*** csmart has joined #openstack-swift22:12
*** minwoob has quit IRC22:13
si1v3rWhat does "container-updater account is not mounted" in container.error.log mean?22:15
*** csmart has quit IRC22:16
*** csmart has joined #openstack-swift22:20
*** csmart has quit IRC22:24
*** wbhuber__ has quit IRC22:35
*** adutta has quit IRC22:37
*** csmart has joined #openstack-swift22:46
*** chlong-zzz has quit IRC22:58
zaitcevsi1v3r: The only similar message that I see is  swift/container/updater.py: self.logger.warn(_('%s is not mounted'), device). Did you actually call one of your devices "account"?23:00
*** kei_yama has joined #openstack-swift23:18
*** lcurtis has quit IRC23:19
*** zhill_desktop has quit IRC23:21
*** asettle has quit IRC23:23
notmynamepatch 189015 is rechecked. -infra says that the pyeclib/jerasure issue should be fixed23:29
patchbotnotmyname: https://review.openstack.org/#/c/189015/23:29
notmynameI'm watching zuul status23:30
notmynameif it passes, then torgomatic can you get out the plunger?23:30
*** lpabon has joined #openstack-swift23:38
*** lpabon has quit IRC23:38
*** harlowja has quit IRC23:42
*** harlowja has joined #openstack-swift23:43
notmynameseems to have passed23:58

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!