Friday, 2014-05-23

*** kevinc___ has quit IRC00:04
*** matsuhashi has joined #openstack-swift00:23
*** igor_ has joined #openstack-swift00:30
*** igor_ has quit IRC00:34
*** dmorita has joined #openstack-swift00:38
*** shri has quit IRC00:39
*** pberis has quit IRC00:46
*** r-daneel has joined #openstack-swift01:17
*** igor_ has joined #openstack-swift01:31
*** igor_ has quit IRC01:35
*** nosnos has joined #openstack-swift01:35
*** eglynn has quit IRC01:54
*** ZBhatti_ has quit IRC01:55
*** saschpe- has quit IRC01:57
*** dmorita has quit IRC01:58
*** gholt has quit IRC01:58
*** r-daneel has quit IRC01:59
*** dmorita has joined #openstack-swift02:01
*** saschpe has joined #openstack-swift02:03
*** gholt has joined #openstack-swift02:04
*** ChanServ sets mode: +v gholt02:05
*** igor_ has joined #openstack-swift02:32
*** haomaiwa_ has joined #openstack-swift02:32
*** igor_ has quit IRC02:36
*** ZBhatti has joined #openstack-swift02:37
*** haomaiwa_ has quit IRC02:38
*** gyee has quit IRC02:40
*** blazesurfer has joined #openstack-swift02:48
blazesurferhi all02:48
blazesurfergot a questions around swift rings and changing the ip address of a node in the ring? any body done this before ?02:48
*** nosnos has quit IRC03:03
occupantcan't say i have. this channel has mostly gone to sleep, might wanna try tomorrow.03:03
*** matsuhashi has quit IRC03:29
*** igor_ has joined #openstack-swift03:32
*** haomaiwa_ has joined #openstack-swift03:34
*** matsuhashi has joined #openstack-swift03:36
*** igor_ has quit IRC03:37
*** haomaiwa_ has quit IRC03:39
*** matsuhashi has quit IRC03:43
*** matsuhashi has joined #openstack-swift03:43
*** matsuhas_ has joined #openstack-swift03:45
*** matsuhashi has quit IRC03:45
*** nosnos has joined #openstack-swift03:45
openstackgerritPete Zaitcev proposed a change to openstack/swift: Separate Backend from Broker  https://review.openstack.org/8590904:00
*** ppai has joined #openstack-swift04:17
*** matsuhas_ has quit IRC04:34
*** matsuhashi has joined #openstack-swift04:43
anticw_blazesurfer: there isn't really an easy way to do this ... that is edit a device04:46
anticw_though you could certainly hack something up04:46
*** ppai has quit IRC04:52
*** psharma has joined #openstack-swift05:04
*** ppai has joined #openstack-swift05:10
*** zaitcev has quit IRC05:19
*** shakayumi has joined #openstack-swift05:30
*** Guest75729 has joined #openstack-swift05:30
*** Guest75729 has quit IRC05:31
*** sagar_ has joined #openstack-swift05:32
*** sagar_ has quit IRC05:34
*** igor_ has joined #openstack-swift05:34
*** igor_ has quit IRC05:38
*** blazesurfer has quit IRC05:47
*** nmap911 has quit IRC06:02
*** nshaikh has joined #openstack-swift06:23
*** jamie_h has joined #openstack-swift07:05
*** foexle has joined #openstack-swift07:20
*** igor has joined #openstack-swift07:42
*** haomaiwang has joined #openstack-swift07:42
*** haomaiwang has quit IRC07:47
*** acoles_away is now known as acoles07:51
*** nacim has joined #openstack-swift08:00
*** matsuhashi has quit IRC08:48
*** shakayumi has quit IRC09:01
*** matsuhashi has joined #openstack-swift09:16
openstackgerritChristian Berendt proposed a change to openstack/swift: use /usr/bin/env python instead of /usr/bin/python  https://review.openstack.org/9511509:27
*** dmorita has quit IRC09:39
*** nosnos has quit IRC09:45
*** ppai has quit IRC09:55
*** ccorrigan has quit IRC09:55
*** mkollaro has joined #openstack-swift10:02
*** ppai has joined #openstack-swift10:08
*** matsuhashi has quit IRC10:12
*** matsuhas_ has joined #openstack-swift10:15
*** openstackstatus has quit IRC10:20
*** openstack has joined #openstack-swift10:21
*** openstackstatus has joined #openstack-swift10:22
*** ChanServ sets mode: +v openstackstatus10:22
*** erlon has quit IRC10:22
*** erlon has joined #openstack-swift10:23
*** fifieldt has quit IRC10:31
*** ppai has quit IRC10:33
*** ccorrigan has joined #openstack-swift10:34
*** acoles is now known as acoles_away10:47
*** matsuhas_ has quit IRC10:55
*** matsuhashi has joined #openstack-swift10:57
*** matsuhashi has quit IRC10:58
*** matsuhashi has joined #openstack-swift10:58
*** matsuhashi has quit IRC10:58
*** matsuhashi has joined #openstack-swift10:59
*** igor_ has joined #openstack-swift11:05
*** igor has quit IRC11:08
*** ccorrigan has quit IRC11:23
*** acoles_away is now known as acoles11:34
*** nshaikh has quit IRC11:42
*** haomaiwang has joined #openstack-swift11:49
*** psharma has quit IRC11:52
*** gpocentek has quit IRC11:52
*** haomaiwang has quit IRC11:59
*** matsuhashi has quit IRC12:20
*** matsuhashi has joined #openstack-swift12:21
*** shakayumi has joined #openstack-swift12:23
*** prassee has joined #openstack-swift12:25
*** matsuhashi has quit IRC12:26
*** prassee has left #openstack-swift12:27
*** foexle has quit IRC12:28
*** matsuhashi has joined #openstack-swift12:37
*** matsuhashi has quit IRC12:38
*** matsuhashi has joined #openstack-swift12:38
*** matsuhashi has quit IRC12:38
*** matsuhashi has joined #openstack-swift12:39
*** matsuhashi has quit IRC12:43
*** ophuk has quit IRC12:51
*** miqui has joined #openstack-swift12:54
*** haomaiwang has joined #openstack-swift12:56
*** kenhui has joined #openstack-swift13:06
*** Ju has quit IRC13:07
*** roock has joined #openstack-swift13:10
*** haomaiwang has quit IRC13:13
roockhello. i'm currently evaluating openstack swift and i have the problem with "Quarantined object(s)" which are not deleted and filling up my disk. see https://gist.github.com/roock/e3bf6ba2e58166bdd3a4 I couldn't find any documentation. Any help appreciated. Thank you in advanceD!13:14
acolesdfg: clayg: i read playback of the xlo stuff you were discussing here yesterday. looks like you found a solution?13:20
*** tdasilva has joined #openstack-swift13:29
*** mrsnivvel has quit IRC13:33
*** r-daneel has joined #openstack-swift13:44
creihtroock: That indicates there is an issue with your configuration13:46
creihtmake sure your hash prefix/suffixes are the same in swift.conf on all of your boxes13:47
*** bill_az has joined #openstack-swift13:47
roockcreiht: ok. thanks i'll check it. can i just remove the whole quarantined/ on the discs which got full?13:48
*** ujjain has joined #openstack-swift13:50
creihtroock: hrm13:50
ujjainso objects are files?13:50
creihtujjain: objects are stored as files on a filesystem13:50
creihtroock: that kinda depends13:50
creihtif you don't care about your data, the easy answer is fix the configs, and delete it13:50
roockit's still a test setup, so i don't care about the data (yet)13:51
creiht(since it is an evaluation)13:51
creihtyeah that's what I was curious about13:51
creihtthen deleting should be fine13:51
creihtif this were a production system, you might want to keep it around a bit to make sure you don't lose data13:52
roockand you were right, i had indeed a typo in the swift.conf - thought i had replaced them all13:54
roockthank you very much13:57
creihtnp13:58
roockare there any ways to improve performance by using a small cache storge on the proxy/storage nodes (e.g. small ssd)?14:01
*** bill_az has quit IRC14:02
*** bill_az has joined #openstack-swift14:02
*** shakayumi has quit IRC14:07
dfgacoles: ya i think so- i still need to try it out a bit14:07
acolesdfg: i played with clayg's patch a bit and it worked ok with func tests.14:08
*** haomaiwa_ has joined #openstack-swift14:09
acolesdfg: the way clayg copies keystone.identity in make_env is probably better idea than copying all the keystone headers, cos that header set will expand with v3 keystone. kesytone.identity will encapsulate the whole lot.14:10
*** pberis has joined #openstack-swift14:13
dfgacoles: ok cool.14:13
*** LIYUEZHO_ has joined #openstack-swift14:15
*** LIYUEZHO_ has quit IRC14:23
*** kenhui has quit IRC14:26
notmynamecschwede_: to confirm, as far as you know, python-swiftclient is good for a release saying py33 is fully supported. is this correct?14:26
notmynamecschwede_: if so, I'll work on cutting a release today14:27
*** haomaiwa_ has quit IRC14:31
cschwede_notmyname: Yes, I think so - I run the functional and unit tests as well as some manual command line actions inside a Python 3.3 virtualenv, and everything worked as expected14:33
cschwede_notmyname: looking forward to a py33 release :)14:33
*** yuanz has joined #openstack-swift14:36
notmynamecschwede_: cool. thanks14:38
* notmyname is going to attempt the office today14:39
notmynameI'll be back online later14:39
*** yuanzz has quit IRC14:40
*** igor_ has quit IRC15:09
*** kevinc___ has joined #openstack-swift15:20
*** gyee has joined #openstack-swift15:23
*** kenhui has joined #openstack-swift15:26
*** haomaiwa_ has joined #openstack-swift15:28
*** tdasilva has left #openstack-swift15:34
*** kenhui has quit IRC15:35
*** kenhui has joined #openstack-swift15:35
*** haomaiwa_ has quit IRC15:48
openstackgerritOpenStack Proposal Bot proposed a change to openstack/python-swiftclient: Updated from global requirements  https://review.openstack.org/8925015:52
openstackgerritOpenStack Proposal Bot proposed a change to openstack/swift: Updated from global requirements  https://review.openstack.org/8873615:52
*** Trixboxer has joined #openstack-swift15:56
*** tdasilva has joined #openstack-swift15:58
*** kenhui1 has joined #openstack-swift16:00
*** kenhui has quit IRC16:04
*** mwstorer has joined #openstack-swift16:05
notmyname /back16:07
notmynamewhy did the netifaces req get bumped in global reqs?16:09
Alex_Gaynorpy3 support I think16:10
*** praveenkumar has quit IRC16:14
*** nacim has quit IRC16:15
*** mwstorer has quit IRC16:17
*** sneezewort has joined #openstack-swift16:21
sneezewortThis might be a dumb question , but is there an exclude option for Swift?16:23
sneezewortFo example:   swift upload mystuffcontainer /allstuffdir --exclude /allstuffdir/privatestuff16:23
notmynamesneezewort: how do you mean?16:23
notmynameah16:23
notmynamesneezewort: no, I don't think so. but it sounds like a nice feature :-)16:25
sneezewortOK thanks.16:27
*** kevinc___ has quit IRC16:27
*** kevinc___ has joined #openstack-swift16:32
*** zaitcev has joined #openstack-swift16:52
*** ChanServ sets mode: +v zaitcev16:52
notmynamepython-swiftclient 2.1.0 tag pushed to gerrit for release16:54
*** Longgeek_ has joined #openstack-swift16:58
portantenice!17:09
notmynamehttps://pypi.python.org/pypi/python-swiftclient/17:09
Alex_Gaynorwahoo!17:10
creihtwoot17:10
*** shri has joined #openstack-swift17:10
notmynamefor the retweets: https://twitter.com/notmyname/status/46988819412564377617:11
*** Longgeek_ has quit IRC17:13
*** bboris has joined #openstack-swift17:14
bborishello17:14
*** shri has quit IRC17:14
dfgacoles: what should I do with the dlo after_fn pipeline reordering thing? i want it after auth- but we can't really reference all the auths can we? should I just you tempauth, keystoneauth or something?17:42
dfgs/just you/just use/17:44
*** kenhui1 has quit IRC17:57
*** tdasilva has quit IRC18:03
*** jamie_h has quit IRC18:10
*** jamie_h has joined #openstack-swift18:13
*** r-daneel has left #openstack-swift18:35
*** Trixboxer has quit IRC18:36
*** tdasilva has joined #openstack-swift18:37
*** kevinc___ has quit IRC18:38
*** NM has joined #openstack-swift18:39
*** jogo is now known as flashgordon18:52
*** john3213 has joined #openstack-swift18:54
*** john3213 has left #openstack-swift18:59
*** mkollaro has quit IRC19:01
notmynamecreiht: you may get your wish of swift-specs vs object-specs ;-)19:16
creihthaha19:20
creihtyeah I saw the email19:20
notmynamecreiht: I told them I was ok with it.19:25
notmynameas I said earlier, IMO: consistency >> {project}-specs > {program}-specs19:26
notmynameso, you know. whatever. ;-)19:26
*** shri has joined #openstack-swift19:27
creihtlol19:29
*** jamie_h has quit IRC19:30
zaitcevSee, this ranges[] thing is exactly why I avoid approving reviews19:42
*** kevinc_ has joined #openstack-swift19:43
zaitcevNow I wonder if I still could trade Sam for https://review.openstack.org/93164, because I'm very proud of that patchlet19:43
zaitcevIt's absurdly insignificant, but I feel like I fixed an actual operational issue for a change.19:43
wkellyzaitcev: as a sysadmin, I appreciate that one!19:56
*** tdasilva has left #openstack-swift20:02
-openstackstatus- NOTICE: Gerrit will be offline for about 20 minutes in order to rename some projects starting at 21:00 UTC.20:34
*** annegentle_ has quit IRC20:34
*** kenhui has joined #openstack-swift20:47
*** kevinc_ has quit IRC20:59
*** kenhui has quit IRC21:01
bboriswhy does swift fail to create anything?21:05
bborisswift upload container file, returns: HTTP/1.1 404 Not Found21:06
bborisi can only stat the account21:06
bborisand list it, but returns nothing because i haven't uploaded anything yet21:07
creihtbboris: do you have account_autocreate = true in your swift-proxy.conf?21:08
*** mattoliverau has quit IRC21:08
bborisyes21:08
creihtok, then next step is to look at the logs to see what is failing21:09
creihtbboris: are you using an saio install?21:09
bborisnot exactly21:10
bborisi have a bunch of lxcs21:11
bboris1 auth, 1 proxy and 10 storage virtual machines21:11
*** mattoliverau has joined #openstack-swift21:11
creihtok21:11
bborisalso, this is logged when trying to create: Could not autocreate account '/AUTH_f892dea27eda447cb5ef10f8b71f6024' (txn: txe565c0650dfd475fbe916-00537fb934)21:12
creihtok that's why your request is failing21:12
creihtnext step is search the logs on all of the storage nodes for txe565c0650dfd475fbe916-00537fb93421:12
bboriswhat is txn?21:12
bboristransaction number?21:12
creihtyeah21:12
bborisok, let me check21:13
creihtjust a unique identifier for each unique request that comes into the system21:13
creihtevery log line posts it, so helps track down issues21:13
bborisi should really get a centralized log for this... sigh21:13
creihthehe21:13
creihtdsh is also your friend in cases like this :)21:14
*** bsdkurt has quit IRC21:14
bborisdoh, permission denied on the drives21:15
bborischown -R swift:swift /srv/node21:16
*** annegentle has joined #openstack-swift21:18
*** kevinc_ has joined #openstack-swift21:20
bboriscreiht: dsh looks like a copy of the script i made for swift testing :)21:22
creihthehe21:22
bboriscreiht: so it works now, thanks. i have another problem though21:27
bborisis it normal to take ~450 ms to get a response from swift?21:28
creihtshouldn't be21:28
*** annegentle has quit IRC21:28
bborislist/stat/whatever takes around that much21:28
bboriswhere could the bottleneck be?21:28
creihtI would do the same thing we did before to find the request in the proxy logs, then search for the transaction in the storage lofs21:29
creihtlogs21:29
creihtusually for something like that there is a misconfiguration, or other failure that causes timeouts to occur21:29
creihtswift will try to work around failures as much as possible, but this also can add latency to requests21:30
-openstackstatus- NOTICE: Gerrit is offline in order to rename some projects. ETA: 22:00.21:36
*** ChanServ changes topic to "Gerrit is offline in order to rename some projects. ETA: 22:00."21:36
*** NM has quit IRC21:39
bborisobject-expirer says that it fails to connect to memcached21:44
bborisisn't memcached used only on the proxy?21:45
creihtmostly21:45
creihtthough I don't think the object-expirer would add latency to your requests21:45
creihtit should be a background process21:45
bborisi see something else21:47
bborisas i have 10 storage nodes21:47
bborisit first tries to get it from storage1021:47
bborisit gets 40421:47
bboristhen it tries from storage08, where it gets 20421:48
creihtmake sure replication is working, and isn't causing errors21:49
bborisbut, when i tried to stat the account again, it tried storage08 directly and got 20421:49
creihtwell isn't failing due to errors21:49
bborisand time was 450 ms too21:49
creihthrm21:49
creihtwhat are you using for auth?21:50
bboriskeystone21:51
creihtahh21:51
creihtwhat type of tokens are you using?21:52
bborisuuid21:52
creihtk21:52
creihtthe proxy servers aren't showing any memcache connection issues?21:52
bboriswas using pki, but it took the same time and it looked ugly while testing21:52
creihthehe21:53
bborislet me see21:53
creihtif it can't cache the token, then it will have to ask keystone to validate on every requests21:53
bborisno errors about memcached21:53
bborisnot mentioned in the syslog at all21:54
creihtk21:54
creihtbboris: sorry that I have to go21:55
creihtat this point you will have to try to isolate what is causing the extra latency21:55
creihtyou could try temporarily removing auth from the pipeline just to validate if that is causing extra latency21:55
creihtyou could also watch the keystone logs to see how often validations are coming in21:56
zaitcevobject-expirer usually runs on a proxy21:56
creihtsorry I can't help any further this evening21:56
zaitcevdeceptive name21:56
bboriscreiht: thanks for the help :) i'll play around a bit, have a good night21:57
bboriszaitcev: it's on the storage nodes in my case21:57
bboristhe token really is different on every swift stat -v22:02
*** openstackgerrit has quit IRC22:14
*** openstackgerrit has joined #openstack-swift22:14
*** openstackstatus has quit IRC22:18
*** openstack has joined #openstack-swift22:19
*** openstackstatus has joined #openstack-swift22:20
*** ChanServ sets mode: +v openstackstatus22:20
*** kevinc_ has quit IRC22:26
openstackgerritMonty Taylor proposed a change to openstack/swift-specs: Updated gitreview file for repo rename  https://review.openstack.org/9529122:28
*** ChanServ changes topic to ""Swift: It's not just another object storage" -- cschwede"22:29
openstackgerritMonty Taylor proposed a change to openstack/swift-specs: Updated gitreview file for repo rename  https://review.openstack.org/9529622:34
notmynamemordred: what's the difference there?22:35
mordrednotmyname: trailing .git from a review in ironic. it actually doesn't matter - and I messed up the commit amend22:37
mordredso I just abandoned22:37
notmynamemordred: ok. I approved one of them22:37
notmynamemordred: but I don't know if jenkins got it22:37
mordrednotmyname: honestly, both are functional22:37
notmynamemordred: and now it's on me to set up the specs repo from the skeleton repo?22:39
mordrednotmyname: I believe so, yes22:40
notmynamemordred: cool. /me adds it to the weekend's TODO list22:41
mordrednotmyname: you want me to make you a base commit?22:41
notmynamemordred: nah, I'll get it22:41
mordredkk22:41
*** sneezewort has quit IRC23:01
*** NM has joined #openstack-swift23:02
openstackgerritA change was merged to openstack/python-swiftclient: Fix wrong assertions in unit tests  https://review.openstack.org/9492023:42
*** NM has quit IRC23:48
*** ozialien has joined #openstack-swift23:56
*** NM has joined #openstack-swift23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!