Tuesday, 2015-02-24

*** doxavore has quit IRC00:08
mattoliverautorgomatic's ec get patch looks good, as others have mentioned there are some nitpics, dlo etc, ive added my own, but i've +2'ed it, so it'll land now and we can keep moving, just take a look at nitpics on the next iteration (dlo fix?) :)00:19
notmynamemattoliverau: thanks!00:19
mattoliveraunotmyname: the power, mwuhahaha00:20
*** abhirc has quit IRC00:25
mattoliveraunotmyname: any chance i can be added to https://launchpad.net/~swift-coresec so i can get back to some bug auditing? :)00:26
notmynamemattoliverau: thanks for the reminder. on it00:26
notmynamemattoliverau: what's your LP id?00:27
mattoliveraunps :) I have more EC to review so no hurry00:27
notmynameah there00:27
notmynamematt-000:27
mattoliverauoh yeah.. umm let me double check00:27
mattoliverauyup, thats it00:27
mattoliverauannoying it couldnt give me mattoliverau :(00:28
zaitcevreally?00:28
notmynamemattoliverau: done00:29
mattoliveraunotmyname: ta00:29
mattoliverauzaitcev: yeah.. gerrit gave it to me but not launchpad :( maybe i did something stupid at the time :P00:30
*** madhuri_ has quit IRC00:34
*** dmorita has joined #openstack-swift00:34
*** ho has joined #openstack-swift00:38
*** erlon has quit IRC00:41
*** zul has quit IRC00:51
*** zul has joined #openstack-swift00:52
*** cca has quit IRC01:01
hogood morning guys!01:02
*** devlaps has quit IRC01:11
mattoliverauho: morning01:11
homattoliverau: morning! I will finish to update my recon patch today :)01:18
*** david-lyle is now known as david-lyle_afk01:19
mattoliverauNice :)01:22
peluseinteresting potential xfs fsync optimization: http://oss.sgi.com/archives/xfs/2014-06/msg00214.html01:29
pelusemattoliverau, hey thanks for looking at the EC stuff!01:30
mattoliveraupeluse: no probs! Nice work on it, very impressive!01:33
*** bill_az has quit IRC01:36
pelusemattoliverau, thanks from me and everyone else who has been in the weeds :) Still a ways to go but this is a big week for us I think01:37
*** abhirc has joined #openstack-swift01:40
*** zhill has quit IRC01:46
*** jamielennox is now known as jamielennox|away01:46
*** abhirc has quit IRC01:48
*** abhirc has joined #openstack-swift01:49
*** gyee has quit IRC01:55
notmynamewhoa. bus drive on the way home missed a turn. passengers who were skipped got mad and chased bus. bus driver and passengers got into shouting match. whole bus cleared and had to wait for next one since the first bus driver had called the cops02:02
*** abhirc has quit IRC02:02
notmynamepeluse: cool xfs thing02:03
klrmnnotmyname: welcome to SF02:03
notmynameklrmn: heh. first time I've seen that sort of thing02:03
klrmnnotmyname: i'm experiencing some probe test failures on master….it is hypothetically possible that it's due to some local resource constraint, but if other people can run the probe tests too, that'd be great02:04
notmynameklrmn: I just did with master right before I left the office. no problems02:05
klrmnnotmyname: i guess it's time for me to reboot my laptop then02:05
notmynameklrmn: probe tests are really cranky about the storage policy (the first one must be default and must have 4 devices evenly sized)02:05
klrmnnotmyname: using clay's saio vagrant box =)02:06
notmynameklrmn: sounds like you need to talk to clayg then ;-)02:07
notmyname*punt*02:07
klrmnnotmyname: it's more likely that the VM is having trouble with memory02:07
notmynameah.02:08
*** jamielennox|away is now known as jamielennox02:12
openstackgerritHisashi Osanai proposed openstack/swift: Enable Object Replicator's failure count in recon  https://review.openstack.org/13834202:13
*** lcurtis has joined #openstack-swift02:17
*** IRTermite1 has joined #openstack-swift02:22
*** abhirc has joined #openstack-swift02:24
*** IRTermite1 has quit IRC02:27
mattoliverauLol wow (re:bus)02:29
*** ahonda has quit IRC02:29
*** abhirc has quit IRC02:34
*** IRTermite1 has joined #openstack-swift02:41
*** mahatic has quit IRC02:50
*** MasterPiece has quit IRC03:11
*** lcurtis has quit IRC03:12
*** panbalag has quit IRC03:14
*** lcurtis has joined #openstack-swift03:24
*** bkopilov has quit IRC03:39
openstackgerritMerged openstack/swift: First pass at EC GET path  https://review.openstack.org/15426303:49
*** lcurtis has quit IRC04:13
*** devlaps has joined #openstack-swift04:17
*** lcurtis has joined #openstack-swift04:24
*** silor has joined #openstack-swift04:28
*** ppai has joined #openstack-swift04:44
*** SkyRocknRoll has joined #openstack-swift04:48
*** lcurtis has quit IRC04:49
*** bkopilov has joined #openstack-swift05:01
*** zaitcev has quit IRC05:08
*** jrichli_ has quit IRC05:09
*** devlaps has quit IRC05:17
hogerrit has been slow down...05:24
*** jamielennox is now known as jamielennox|away05:33
*** jamielennox|away is now known as jamielennox05:34
*** kei_yama has quit IRC05:57
*** km_ has joined #openstack-swift06:03
*** km has quit IRC06:05
*** ho__ has joined #openstack-swift06:09
*** ho has quit IRC06:09
mattoliverauK, I think I might call it a day. night all (though I may be back as wife is going out tonight)06:14
ho__mattoliverau: good night!06:15
*** reed has quit IRC06:45
*** chlong has quit IRC07:34
*** IRTermite1 has quit IRC07:34
*** SkyRocknRoll has quit IRC07:48
*** nshaikh has joined #openstack-swift07:50
*** rledisez has joined #openstack-swift08:11
*** Muntaner has joined #openstack-swift08:17
Muntanergood morning to everyone!08:17
MuntanerI'm having some serious problems with Swift08:17
MuntanerI installed it on my private Openstack Cloud (All-in-one installation)08:18
MuntanerIt worked fine, but then we had to change the public IP of the server where Openstack is deployed08:18
Muntanerwe got all the components working again (glance, cinder, etc), just swift isn't working08:19
Muntanerseems like all of my swift process are running08:19
Muntanerbut swift stat hangs... and via the dashboard horizon I can see the "containers" tab, but when I click on it I have a never ending hang08:19
Muntanerwhat logs can I look to diagnose my situation?08:20
openstackgerritMerged openstack/python-swiftclient: Verify MD5 of uploaded objects.  https://review.openstack.org/12925408:22
Muntanerwell, the error that I get after some time is  Account HEAD failed: http://192.168.0.62:8080/v1/AUTH_914cd4d8565e4fb8beee37f46a50aa4f 503 Internal Server Error08:23
Muntanerother informations: swift does not respond to any command, f. e. "swift list"08:27
*** ho__ has quit IRC08:31
*** ho has joined #openstack-swift08:31
*** km_ has quit IRC08:31
*** cca has joined #openstack-swift08:40
donaghMuntaner: have you specified where to put logs in /etc/swift/proxy-server.log?08:48
donaghIf not, logs would be in /var/log/syslog08:49
donaghYou want to look for proxy-server: as first step08:49
Muntanerdonagh, I haven't specified that - I have some logs in /var/log/syslog08:50
donaghActually, can I take you back to first steps: I assume you are using keystone?08:50
Muntanerdonagh, yes08:51
donaghOk. /bin/swift can hand if it can't talk to keystone. Can you do a "nova endpoints"08:51
donagh?08:51
Muntanerdonagh, done, need output?08:52
donaghNot realy. But it works?08:52
Muntaneryep, worked08:52
donaghNext question, doe sendpoint of Swift look ok?08:53
Muntanerdonagh, actually I found some errors in /var/log/syslog08:53
Muntanerdonagh, yes, looks fine08:53
donaghWhat are the errors?08:53
donaghExample?08:53
*** jistr has joined #openstack-swift08:54
Muntanergonna paste them08:54
donaghok08:54
Muntanerdonagh, http://paste.ubuntu.com/10385612/08:55
donaghOk. A few problems here.08:57
Muntaneractually, 10.0.0.230 is my old ip08:57
donagh1/ the memcache error says that the proxy-server can't even talk to localhost ... or maybe memcached is not started?08:57
Muntaner192.168.0.62 is the new one, that I'm using actually08:57
cschwededencaval: notmyname: i still think the bp (https://blueprints.launchpad.net/swift/+spec/sorting-method-storage-policy) needs a DB update because of missing indexes - for example, if you want the container listing sorted by date. of course this increases the DB size… that said, please have a look at the comments on this abandoned patch: https://review.openstack.org/#/c/121278/08:58
Muntanerdonagh, seems like memcached is running08:58
donagh2/ The "ERROR with Account server" says that proxy-server can't talk to 10.0.0.230  -- againmaybe Swift has not been started on that node?08:58
Muntanerdonagh, 10.0.0.230 is the old IP of my server machine08:59
Muntanernaturally, I updated that in conf files08:59
donaghAh, I read your original as that you changed the endpoint for the system...not the IP addresses of the swift nodes.09:00
donaghSo you changed the IP of the swift nodes? Is it node or nodes?09:00
*** jordanP has joined #openstack-swift09:00
Muntanerdonagh, infortunately, we have an all-in-one installation: all the components are in a single server (quite rude, I know)09:00
donaghNo problem. However, the ring files containe the IP address of the original node. Actually, in my jon I never mess with ring files09:01
Muntanerdonagh, how can I update that?09:01
donagh...but there is an option in the ring builder to simply change the IP address09:01
Muntanerdonagh, totally missed that09:02
donaghIf you can't find the syntax for the command, come back here09:02
*** km has joined #openstack-swift09:03
donaghAlso, I suggest you put proxy-logging in the pipeline. It needs to go in two places. That way, the logs will contain more information09:03
*** dvorkbjel has quit IRC09:04
Muntanerdonagh, fighting with that command syntax - how can I actually change those IPs?09:08
donaghhold on09:08
Muntanerdonagh, solved it, seems like I have only the memcached problem now09:09
Muntanerdonagh, http://paste.ubuntu.com/10385755/09:09
donaghDo you have a [memcache] section in /etc/proxy-server.conf?09:10
donaghSorry [filter:cache] is correct name09:11
Muntanerdonagh, yes, with this lone line: use = egg:swift#memcache09:11
donaghYou could try adding:09:12
donaghmemcached_servers = <new-ip>:1121109:12
donaghI'm assuming memcached is actually running?09:13
Muntaneryes, service memcached tells me it's running09:13
Muntanerdonagh, same errors09:13
*** silor has quit IRC09:14
donaghhold on09:14
donaghIn /etc/memcached.conf is -l set t anything (IP it listens on)09:15
donaghBTW: I assume you restarted it after changing the nodes' IP09:15
Muntanerdonagh, yes: -l 192.168.0.6209:15
Muntanerdonagh, naturally09:15
Muntanerrestarted all swift services and memcached09:16
donaghSo is memcached_servers = 192.168.0.62:1121109:16
Muntaneryes donagh09:16
donaghin [filter:cache] section?09:16
Muntaneryes donagh, get same errors09:18
donaghTHe ECONNREFUSED says 127.0.0.1 -- or is that an older message?09:18
Muntanerdonagh, fresh error: Feb 24 10:18:03 openstack proxy-server STDOUT: ERROR:root:Error connecting to memcached: 127.0.0.1:11211#012Traceback (most recent call last):#012  File "/usr/lib/python2.7/dist-packages/swift/common/memcached.py", line 208, in _get_conns#012    fp, sock = self._client_cache[server].get()#012  File "/usr/lib/python2.7/dist-packages/swift/common/memcached.py", line 123, in get#012    fp, sock = Pool.get(self)#012  File "/usr/09:18
Muntanerlib/python2.7/dist-packages/eventlet/pools.py", line 119, in get#012    created = self.create()#012  File "/usr/lib/python2.7/dist-packages/swift/common/memcached.py", line 119, in create#012    sock.connect((host, int(port)))#012  File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 194, in connect#012    socket_checkerr(fd)#012  File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 46, in socket_checkerr#012    raise s09:18
Muntanerocket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: tx4d8fcf52164d4c2fa2970-0054ec41cb) (client_ip: 192.168.0.62)09:18
Muntaner(sorry for pasting there)09:19
donaghdid you restart proxy-server since adding memcached_servers= to /etc/proxy-server.conf?09:19
Muntanerdonagh, I did a swift-init all restart09:20
donaghOk. Go to /etc/memcached.conf and set -l to 127.0.0.1, also do same in /etc/swift/proxy-server.conf and restart both swift and memcached09:22
donaghBTW: I presume Swift is working...just maybe slower?09:22
donagh...the memcache error won't stop swift from working09:23
Muntanerdonagh, it's not working actually09:24
Muntanerswift stat returns immediately a Account HEAD failed: http://192.168.0.62:8080/v1/AUTH_914cd4d8565e4fb8beee37f46a50aa4f 400 Bad Request09:24
*** aix has quit IRC09:25
donaghDoes swift stat -v09:25
donaghprint storage URL and auth token?09:25
Muntanerdonagh, brb in 5 minutes - sorry09:26
donaghie. add -v09:26
donaghdito -- I'm going for coffee09:26
Muntanerdonagh, swift stat -v Account HEAD failed: http://192.168.0.62:8080/v1/AUTH_914cd4d8565e4fb8beee37f46a50aa4f 400 Bad Request09:31
*** openstackgerrit has quit IRC09:36
*** openstackgerrit has joined #openstack-swift09:36
*** acoles_away is now known as acoles09:36
acolesnotmyname: ok, noted :) nearly stopped reading when i got to george-cloo...09:39
acolesnotmyname: do you have a preference among the varieties?09:40
*** nellysmitt has joined #openstack-swift09:44
donaghMuntaner: can you run09:51
Muntanerdonagh, solved it, thanks :)09:51
Muntanerwas a problem with the rings09:51
donaghswift-get-nodes /etc/swift/account.ring.gz  /AUTH_914cd4d8565e4fb8beee37f46a50aa4f09:51
openstackgerritHisashi Osanai proposed openstack/swift: WIP: Enable Role-based access control using oslo.policy in Swift  https://review.openstack.org/14993009:51
*** aix has joined #openstack-swift09:52
donaghGood. The purpose of swift-get-nodes was to check that the IP addresses in ring were ok09:52
*** cca has quit IRC10:05
*** khivin has joined #openstack-swift10:08
*** joeljwright has joined #openstack-swift10:25
*** aix has quit IRC10:30
*** jamielennox is now known as jamielennox|away10:42
*** aix has joined #openstack-swift10:43
*** ho has quit IRC10:49
donaghnotmyname: So my "local" offlicense is https://www.google.co.uk/maps/dir/Galway+City,+Ireland/Ashby+Rd,+Woodville,+Swadlincote,+UK/@53.0885464,-7.6486123,7z/data=!3m1!4b1!4m13!4m12!1m5!1m1!1s0x485b96eca4f63c15:0x9f202f408ade89a3!2m2!1d-9.0566819!2d53.270564!1m5!1m1!1s0x4879febbb02caed9:0x81f425b347e82a59!2m2!1d-1.5066527!2d52.7692252?hl=en10:54
*** jamielennox|away is now known as jamielennox10:58
*** jamielennox is now known as jamielennox|away11:01
*** jamielennox|away is now known as jamielennox11:02
acolesdonagh: these guys will ship it to you https://www.thewhiskyexchange.com/search?q=casamigos11:05
mattoliverauNo drinking on the job guys.. Unless your working from home ;p11:05
*** jamielennox is now known as jamielennox|away11:08
*** Muntaner has quit IRC11:09
*** jamielennox|away is now known as jamielennox11:10
acolesmattoliverau: sshh ;)11:11
*** ppai has quit IRC11:11
mattoliverauacoles: oh this isn't private.. Sorry :p11:11
openstackgerritMerged openstack/swift: Fix efficient replication handoff delete  https://review.openstack.org/15729011:15
*** ppai has joined #openstack-swift11:23
*** jamielennox is now known as jamielennox|away11:38
*** jamielennox|away is now known as jamielennox11:41
*** mahatic has joined #openstack-swift11:41
*** chlong has joined #openstack-swift11:56
*** dmorita has quit IRC12:00
*** pconstantine has quit IRC12:13
*** km has quit IRC12:16
*** panbalag has joined #openstack-swift12:24
*** ppai has quit IRC12:29
*** ppai has joined #openstack-swift12:43
*** chlong has quit IRC12:44
*** bkopilov has quit IRC13:04
*** ppai has quit IRC13:16
*** jamielennox is now known as jamielennox|away13:18
*** jamielennox|away is now known as jamielennox13:20
*** ppai has joined #openstack-swift13:30
*** RackerShagz has joined #openstack-swift14:05
*** RackerSh_ has joined #openstack-swift14:06
*** RackerShagz has quit IRC14:09
*** navid__ has joined #openstack-swift14:17
openstackgerritDaniel Wakefield proposed openstack/python-swiftclient: Reduce memory usage for download/delete and add --no-shuffle option to st_download  https://review.openstack.org/13033914:21
openstackgerritDaniel Wakefield proposed openstack/python-swiftclient: Add changed flag to download.  https://review.openstack.org/15870114:21
*** zul has quit IRC14:23
*** zul has joined #openstack-swift14:24
*** nellysmitt has quit IRC14:28
*** nshaikh has left #openstack-swift14:29
*** bkopilov has joined #openstack-swift14:29
*** nellysmitt has joined #openstack-swift14:29
*** navid__ has left #openstack-swift14:29
*** nellysmitt has quit IRC14:29
*** bill_az has joined #openstack-swift14:29
*** nellysmitt has joined #openstack-swift14:29
*** mahatic has quit IRC14:33
*** dmsimard_away is now known as dmsimard14:38
*** abhirc has joined #openstack-swift14:38
*** abhirc has quit IRC14:39
*** ppai has quit IRC14:44
*** IRTermite1 has joined #openstack-swift14:50
*** bsdkurt has joined #openstack-swift14:50
*** bsdkurt has quit IRC14:51
*** kurt2 has joined #openstack-swift14:53
*** nellysmitt has quit IRC14:55
*** mahatic has joined #openstack-swift14:56
*** IRTermite1 has quit IRC14:56
*** Nadeem has joined #openstack-swift15:03
*** kurt2 has quit IRC15:05
*** kurt2 has joined #openstack-swift15:07
*** Manshoon has joined #openstack-swift15:12
*** kurt2 has quit IRC15:13
*** bsdkurt has joined #openstack-swift15:15
*** nellysmitt has joined #openstack-swift15:18
*** jrichli_ has joined #openstack-swift15:19
openstackgerritOpenStack Proposal Bot proposed openstack/python-swiftclient: Updated from global requirements  https://review.openstack.org/8925015:22
openstackgerritOpenStack Proposal Bot proposed openstack/swift: Updated from global requirements  https://review.openstack.org/8873615:22
*** devlaps has joined #openstack-swift15:24
*** devlaps has quit IRC15:24
*** nellysmitt has quit IRC15:32
*** nellysmitt has joined #openstack-swift15:32
acolesjrichli_: https://review.openstack.org/#/c/158735/ <- will enable you to send alternate etag and content-type values to container15:43
jrichli_acoles: thanks!15:44
acolesusing x-backend-update-override-etag15:44
acolesI based it on https://review.openstack.org/#/c/158401/2 , which it doesn't need to depend on but it will allow you to more easily pull both patches and rebase on them as and when you need to15:45
*** david-lyle_afk is now known as david-lyle15:47
acolesjrichli_: i *think* those two patches provide the backend support that the crypto middlewares will need15:48
acolesapart from the etag header trailing the PUT body15:49
jrichli_acoles: sounds good.  Does that mean that the CHANGELOG should be modified at this point to mention the changes?15:49
acolesjrichli_: no, not until any of this stuff lands ;)15:49
jrichli_oh, ok15:50
*** zul has quit IRC15:50
acolesjrichli_: steady on :D15:50
*** zul has joined #openstack-swift15:50
*** abhirc has joined #openstack-swift15:54
*** annegent_ has joined #openstack-swift15:58
*** abhirc_ has joined #openstack-swift15:58
*** abhirc has quit IRC15:59
*** vt102 has quit IRC16:05
*** reed has joined #openstack-swift16:09
*** lcurtis has joined #openstack-swift16:10
notmynamegood morning16:11
openstackgerritMerged openstack/swift: Add multiple reseller prefixes and composite tokens  https://review.openstack.org/13708616:12
*** jamielennox is now known as jamielennox|away16:15
*** jamielennox|away is now known as jamielennox16:16
lcurtishello all..is there a real world / practical limit to number of objects per container?16:17
lcurtisor rule of thumb for not to exceed?16:17
ctennisif you aren't using SSDs for your container/account dbs, then in the order of "millions"16:19
notmynamethe "limit" is one of how many puts per second you can sustain16:20
lcurtisokay...fair enough16:20
*** NM has joined #openstack-swift16:21
lcurtisare there general benchmarks for ssd's versus spinning disks16:21
lcurtistheoretical max vs requests per second?16:21
openstackgerritOpenStack Proposal Bot proposed openstack/swift: Updated from global requirements  https://review.openstack.org/8873616:22
donaghacoles: clayg: thanks for your work in reviewing the composite token stuff. I'm aware that it involved a lot of work setting up DevStack etc.16:23
acolesdonagh: its merged - nice work16:23
donaghwhew16:24
*** NM1 has joined #openstack-swift16:25
jrichli_good mornign16:25
*** NM has quit IRC16:25
*** dencaval has joined #openstack-swift16:25
ctennislcurtis: it changes all of the time and is very dependent on overall setup fo the cluster.  SSDs are orders of magnitude faster though,so you can have tens of millions of objects in a container before the container becomes the bottleneck.16:32
lcurtisctennis thank you16:32
ctennislcurtis: keep in mind I'm only talking about using the ssd to store the container information, not the object data itself. you can get away with a very small amount of SSDs in a cluster and still keep all of your object data on spinning media.16:39
lcurtisctennis - got it..thats along lines of what I am thinking16:39
ctennisa cheaper method may be to still use spinning disks for account/container info, but just keep them dedicated to just those functions vs. also keeping object data on them, which will keep as much IO available as possible.16:40
*** Nadeem has quit IRC16:41
lcurtishas swift published any numbers or tests?16:41
ctennisI don't think so16:42
lcurtiswe are currently thinking of ssds for just account / container boxes16:42
lcurtisnodes that is16:42
openstackgerritAlistair Coles proposed openstack/python-swiftclient: Add tests for uploads deleting DLO segments  https://review.openstack.org/15317716:57
peluselcurtis, there are lots of benchmakring articles, presentations, etc., available for swift just by googling.  not saying I endorse any of them or that any one of them has a golden set of numbers, however...16:59
peluselcurtis, this one is a little old but I think has decent general concepts/thoughts http://www.zmanda.com/blogs/?tag=swift-benchmark16:59
*** doxavore has joined #openstack-swift17:02
*** BaconIsKing has joined #openstack-swift17:02
lcurtispeluse thank you..sometimes there are some well known ones in the community that might be overlooked.17:03
pelusefor sure...17:06
*** annegent_ has quit IRC17:06
notmynamelcurtis: basically, "ssds are orders of magnitude faster" is the common wisdom of the community. anything else is very dependent on your hardware and usage pattern17:09
*** annegent_ has joined #openstack-swift17:10
lcurtisnotmyname -- got it17:13
lcurtisthanks as always for your help17:13
peluseesepcially Intel SSDs :)17:13
lcurtisyes!17:13
* peluse couldn't resist...17:13
notmynamewhew. rode to work on my bike for the first time in about 8 months (since my accident)17:15
notmynamemy legs aren't used to that! ;-)17:15
lcurtisare there performance benefits to using multiple accounts in same way as multiple containers?17:16
*** rledisez has quit IRC17:16
mahaticnotmyname, hello17:16
notmynamemahatic: good morning!17:16
mahaticnotmyname, good morning to you!17:16
openstackgerritAlistair Coles proposed openstack/swift: Fix failing tox -e py26 tests  https://review.openstack.org/15878417:17
mahaticnotmyname, I have a question: the --validate-servers all option, should I just include it and let it work accordingly (as it now takes default object, and if container/account it works accordingly)17:17
*** nellysmitt has quit IRC17:17
*** jistr has quit IRC17:18
*** nellysmitt has joined #openstack-swift17:18
*** erlon has joined #openstack-swift17:19
pelusenotmyname, glad you made it in safely :)17:19
notmynamemahatic: yes, that sounds right.17:19
notmynamepeluse: so is my wife!17:19
hugokuobonjour notmyname17:20
notmynamemahatic: with the all option, the operator gets every report about whatever hosts were chosen. eg tell me everything about every server. or tell me everything about this particular zone. etc17:21
mahaticnotmyname, yeah, I figured :D17:21
hugokuoA quick question for anyone who knows about the answer. If there's a blueprint of python-swiftclient added to LP, could anyone just take it and implement it directly ?17:23
notmynamehugokuo: sure. also, you could do it if there isn't a blueprint too :-)17:23
hugokuonotmyname: got it. We may try to implement https://blueprints.launchpad.net/python-swiftclient/+spec/swiftclient-download-pseudo-folder-to-specific-target17:24
notmynamehugokuo: great!17:24
mahaticit was easy, I looked into the code and some manpage on the internet ;)17:24
notmynamehugokuo: and by "we" you mean charz? ;-)17:24
hugokuonotmyname: oh yes17:25
notmynamehugokuo: lol17:25
*** bill_az has quit IRC17:25
charzLoL17:25
hugokuocool beans17:25
notmynameacoles: big news yesterday what with the top-level reorg. do you expect it to affect you at all?17:27
openstackgerritMahati proposed openstack/swift: Ring checker in swift-recon  https://review.openstack.org/15361717:27
*** bill_az has joined #openstack-swift17:30
openstackgerritDavid Goetz proposed openstack/swift: swift-ring-builder remove doesn't print message (tabbing error)  https://review.openstack.org/15878917:33
acolesnotmyname: no, i don't expect any disruption17:34
*** annegent_ has quit IRC17:37
*** omame has quit IRC17:40
*** omame has joined #openstack-swift17:43
notmynameFYI swiftstack announced a ceo change today. press release happened, but I think we'll have a blog post later today17:44
notmynamejoe is definitely still around and very involved.17:44
btorchso what happened to swift-ring-builder ? did something change and messed it up ?17:44
notmynamehttp://www.enterprisetech.com/2015/02/24/swiftstack-appoints-new-ceo/17:44
notmynamebtorch: what do you mean?17:44
omamenotmyname: you got a promotion?17:45
klrmnclayg: is there something that needs to be done to a vagrant saio after halting and re-uping it in order to get it to work again, or do i need to destroy and rebuild?17:45
notmynameomame: lol17:45
omamedon't forget to update /about17:46
notmynameomame: being done now17:46
btorchnotmyname: issues removing a zone/device where it has been set as drained .. nothing is removing the device(s)17:46
btorchrebalance doesn't work, obviously there is nothing to rebalanace since there are no parts to shuffle around17:47
btorchwrite_ring also doesn't make any changes , device(s) are still kept in builder file17:47
notmynamebtorch: to clarify so I can test locally...you have a zero-weight device and you can't remove it?17:48
btorchit seems to work on newly build rings17:48
btorchdo you have a 5y old ring laying around :) ?17:48
omameexisting ring -> set weight to 0 -> rebalance -> remove host -> rebalance -> got rejected17:49
btorchit only got removed from object ring but because there was another device that was actually removed and ring actually needed rebalancing17:50
btorchbtw it used to not too long ago17:52
btorchused to work17:52
notmynameomame: what's the "got rejected"? couldn't rebalance?17:55
btorchNo partitions could be reassigned.17:55
notmynameok17:56
btorchEither none need to be or none can be due to min_part_hours [1].17:56
btorchring hasn't been modified for several days so not a min_part_hours issue17:57
notmynamebtorch: what version of swift are you using?18:00
btorchhaha18:01
btorch:)18:01
*** annegent_ has joined #openstack-swift18:01
btorchnotmyname: ver 2.2.1-118:02
notmynamebtorch: `curl https://storage101.dfw1.clouddrive.com/info | python -m json.tool` says 2.1.018:02
notmyname:-)18:02
btorchyeah I forgot about damn recon :(18:03
notmynamerecon? or /info18:03
btorcheither way18:03
notmynamehugokuo: charz: on the community QA cluster, when we get that back online, how possible would it be to add the py26 unit tests?18:11
notmynametorgomatic: acoles: ^^ idea18:11
*** devlaps has joined #openstack-swift18:11
torgomaticnotmyname: sounds like a good idea to me18:12
openstackgerritSamuel Merritt proposed openstack/swift: Clean up a couple deprecation warnings  https://review.openstack.org/15880618:12
*** zaitcev has joined #openstack-swift18:12
*** ChanServ sets mode: +v zaitcev18:12
notmynametorgomatic: did you see the question btorch has? can't remove empty weight devices from the ring18:13
charznotmyname: That's not a problem, I can add it :)18:14
notmynamecharz: thanks18:14
torgomaticnotmyname: yeah, and it doesn't happen with newer rings I guess?18:14
notmynametorgomatic: btorch said it doesn't but it might, actually. I'm looking at master right now18:14
torgomaticI know it doesn't print the message, but dfg just fixed that18:15
acolesnotmyname: good idea. think i added you as reviewer on this one as an fyi https://review.openstack.org/#/c/158671/18:16
notmynametorgomatic: btorch: ah. it does remove. or more specifically, you have to rebalance after removing to actually remove it. but that works on current master18:16
btorchyeah we gonna try the new --force flag in a bit to see if it works18:17
notmynamebtorch: when's the last time you did a rebalance on that ring before now? ie what version of swift were you running the last time the builder file was written to disk?18:17
notmynamebtorch: ie I can build a ring like that and then try the removal to attemp a repro18:17
btorchlast year was last rebalance and prior was the version you got from dfw url18:18
notmynameok. so probably 2.0.018:18
btorchit was around 2014-07-1718:19
btorchlet me check what version it was back then but yeah likely 2.0.018:19
btorchnotmyname: yeah 2.0.018:20
notmynameok, now I have a 2.0.0 era ring18:21
*** vt102 has joined #openstack-swift18:21
*** annegent_ has quit IRC18:22
*** doxavore has quit IRC18:22
*** doxavore has joined #openstack-swift18:23
notmynamebtorch: ok, going through that I had to do a pretend_min_part_hours_passed but after that it was removed18:23
notmynamewith the 2.1.0 code18:24
doxavoredoes swift's proxy-server use 499 only in cases where the client disconnects, or are there other possible reasons?18:25
notmynamedoxavore: that should be the only reason18:25
acolestorgomatic: yes i will move https://review.openstack.org/158735 to master and tweak the header name18:26
torgomaticacoles: thanks!18:26
notmynamethis is what I did: https://gist.github.com/anonymous/9a7198db76d4612d23b818:28
*** jordanP has quit IRC18:29
zaitcevI am curious why we chose 499 when 408 looks like exact that error, only standard18:29
zaitcevNot that it matters really, but I'm curious.18:29
*** aix has quit IRC18:29
notmynamezaitcev: http://tools.ietf.org/html/rfc7231#section-6.5.7 says that's for the server receiving stuff from the client. the 499 is when the client goes away on a response and we need to log *something* in the log line18:31
*** gyee has joined #openstack-swift18:31
notmynamewe may could separate the 2. maybe we do 499 for both kinds of timeouts. but could log a 408 if the client doesn't send data fast enough. I'm not sure if that would break anything for log parsing18:32
notmynameacoles: thanks for the functests job patch. lgtm18:34
notmynameI mentioned it in -infra both so they'd notice and also in case there were other jobs that had a 'sudo' added that may need to be audited18:39
*** mahatic has quit IRC18:40
acolesnotmyname: i suspect the job was temporariy broken between the change to 'tox -e func' and the later adding of 'sudo' ;)18:48
acolestorgomatic: if i cherry pick that override patch onto master and leave the same change id, will gerrit maintain the same review but do the 'right thing' wrt moving it to master?18:51
acolesor should i generate a new change id?18:52
*** zul has quit IRC18:52
torgomaticacoles: I think so, but I've never trie dit18:52
torgomatic*tried it18:52
torgomaticso we're about to learn something about Gerrit either way :)18:52
*** zul has joined #openstack-swift18:52
notmynameacoles: I think it does the right thing. IIRC from doing backports. not sure though18:53
notmynamemaybe with backports I did a new change id18:53
*** zul has quit IRC18:53
*** zul has joined #openstack-swift18:53
acolestorgomatic: ok here goes18:54
openstackgerritAlistair Coles proposed openstack/swift: Enable override of container update headers  https://review.openstack.org/15882118:54
notmynameacoles: torgomatic: fun! (not really) https://review.openstack.org/#/q/I7d846ed54ff173d08c66c6d5b0ecf7dff27f5a87,n,z18:55
acolesnotmyname: torgomatic : oh :(18:56
acolesall good things come in two's ?!?18:56
notmynamelooks like gerrit has (change_id, branch) tuples for the master key18:56
acolesi'll abandon and try again18:56
notmynameacoles: I think you can leave the one on master18:56
acolesnotmyname: yes, true18:56
acolesnotmyname: torgomatic: ok i abandoned the one on feature/crypto19:00
acolesjrichli_: ^^ context for the abandon19:00
claygacoles: thanks for pretending we still support py2.6 with me!19:03
*** annegent_ has joined #openstack-swift19:03
*** bill_az has quit IRC19:04
acolesclayg: you could open another linkedin group for those who do ;)19:04
*** acoles is now known as acoles_away19:09
jrichli_acoles_away: thanks19:15
*** silor has joined #openstack-swift19:17
claygmattoliverau: I think confirmed/low should be enough to push it out of our bird-doggin' list; but you're right with something like that it's really hard to say when it's closed - https://bugs.launchpad.net/swift/+bug/136508419:20
openstackLaunchpad bug 1365084 in OpenStack Object Storage (swift) "Spelling mistakes corrected in comments." [Low,Confirmed]19:20
clayginstead of won't fix - they need "will try"19:20
claygredbo: what do you think about invalidate vs. deleting tombstones in https://review.openstack.org/#/c/134855/19:21
*** annegent_ has quit IRC19:23
*** annegent_ has joined #openstack-swift19:26
*** devlaps has quit IRC19:33
*** RackerSh_ has quit IRC19:35
notmynamepeluse: clayg: torgomatic: acoles_away: I added https://review.openstack.org/#/c/158821/ to the EC priority reviews section so it can be tracked19:47
torgomaticnotmyname: sounds good19:48
torgomaticI'm looking at that one now, but I'm possibly embarking on a yak shave getting probetests to pass on master so I have a baseline for probetests on that change19:48
notmynametorgomatic: probe tests passed on master for me yesterday19:49
torgomaticnotmyname: same here; not sure what's up with this machine19:49
notmynamea "validate that the config is what probetests expect" tool would be nice19:50
notmynamealso, a pony19:50
*** annegent_ has quit IRC19:54
zaitcevThe fix for Neutron that my gate job needs to proceed (158162) had hit several gate bugs, each of which...19:56
zaitcevThis is like some Kessler syndrome in miniature19:57
*** RackerShagz has joined #openstack-swift20:01
*** dencaval has quit IRC20:04
*** sandywalsh_ is now known as sandywalsh20:04
*** annegent_ has joined #openstack-swift20:04
*** zhill has joined #openstack-swift20:04
*** BaconIsKing has quit IRC20:05
*** RackerShagz has quit IRC20:05
*** RackerShagz has joined #openstack-swift20:07
klrmnclayg: is there something that needs to be done to a vagrant saio after halting and re-uping it in order to get it to work again, or do i need to destroy and rebuild?20:15
claygif you've changed your localconf - you need to vagrant provision to get the chef bits in the vm up to snuff - if you just need to make the vm be like it was at chef time you can rebuildswift20:16
claygrebuildswift is just a script in the vagrant users path20:16
claygif you need to just erase data on the vm and clear out swift but not undo any local conf edits - then resetswift will work20:16
claygif you just changed branches you need to reinstallswift20:16
claygif you want to be angel you could type up something like that in the README and push it to swiftstack/vagrant-swift-all-in-one20:17
klrmnclayg: i can be an angel on that. i wonder how i managed to get anything to work the past 2 weeks not knowing that20:24
*** nellysmitt has quit IRC20:26
*** lcurtis has quit IRC20:26
*** silor has quit IRC20:28
klrmnclayg: do any of the commands imply the other commands?20:32
claygyeah rebuildswift > resetswift > reinstallswift IIRC20:38
klrmnclayg: so resetswift does reinstallswift?20:38
claygi'm looking at the scripts in ./bin now and I'm not sure - it doesn't seem to call it directly20:39
klrmnclayg: it looks like swift proxy won't work on a VM thats been halted and restarted…it can't bind to 0.0.0.0:808020:40
claygklrmn: that's weird20:41
clayganyway, no it doesn't look like resetswift will reinstall the swift branch20:41
klrmnclayg: i've seen problems with interfaces not working right w/o provisioning  before20:41
clayg... interesting20:42
klrmnclayg: i've also gotten a bunch of errors about solo.rb and reload.json20:43
openstackgerritIan Cordasco proposed openstack/python-swiftclient: Release connection after consuming the content  https://review.openstack.org/14904320:43
klrmnclayg: and dna.json20:43
claygi've never seen any errors reference those file20:43
klrmnclayg: https://gist.github.com/klrmn/74412a7b1a28cc5f87cd20:44
claygklrmn: I don't think you should be running any of those scripts as sudo - probably something got the wrong permissions20:46
claygthat bind error looks like some of the servers aren't fully shutdown - you can swift-init all stop; pkill -f swift and make sure ps -ef | grep swift is empty20:46
klrmnclayg: the output when i did it w/o sudo is now in a comment on the gist20:47
claygok, yeah you probably just need to run vagrant provision from your host to get the chef bits filled back in on /tmp (they would have been removed on the reboot)20:48
*** marzif_ has joined #openstack-swift20:49
marzif_Hi all, I have a question about python-swiftclient: why with class Connection(object) from client.py is not possible to set a proxy?20:51
claygidk, i sorta remember httplib doesn't honor the env vars by default unless you do some magic - but I thought requests would?20:52
*** Manshoon_ has joined #openstack-swift20:53
*** BaconIsKing has joined #openstack-swift20:54
marzif_clayg, thanks,  yes requests does, but when I've tried it didn't work, probably I was using a swiftclient version, with requests deps that still didn't supported that?20:55
marzif_I don't know... so in theory if I use let's say the last version of python-swiftclient with env var set for proxy it should work?20:55
*** Manshoon has quit IRC20:56
marzif_yes requests honor HTTP_PROXY and HTTPS_PROXY (capital)...20:56
claygi don't believe I've tested it either way20:57
*** RackerShagz has quit IRC20:57
*** RackerSh_ has joined #openstack-swift20:58
marzif_ok. thanks a lot for your response :)20:58
claygif you have the latest version (or installed from source) and it doesn't work - you could open a bug - I think it'd be nice if it did - dunno if it'd be hard to fix whatever the issue is20:58
*** BaconIsKing has quit IRC20:59
marzif_agreed21:00
marzif_my doubt was because from  class HTTPConnection(object): you can explicitly set proxy=, while with Connection not.... btw tomorrow I'll try with env HTTP(S)_PROXY21:03
*** annegent_ has quit IRC21:04
claygoic, i thought you were using the env vars - maybe there's a way to make the kwargs - you could open a bug for that too even if the env vars work21:05
*** Manshoon has joined #openstack-swift21:05
*** Manshoon_ has quit IRC21:07
marzif_clayg,  https://bugs.launchpad.net/python-swiftclient/+bug/142528321:10
openstackLaunchpad bug 1425283 in python-swiftclient "proxy argument needed in client.Connection " [Undecided,New]21:10
*** RackerSh_ has quit IRC21:11
marzif_clayg,  from https://github.com/openstack/python-swiftclient/blob/master/swiftclient/client.py#L1218 http_connection is a redefined method inside Connection, but it doesn't allow to take proxy= as arg21:13
marzif_I'm not sure but probably adding proxy=  there and in __init__  of Connection may solve the issue, tomorrow I can test it and let you know if you want21:14
claygyeah just update the bug if you find a patch that works for you21:14
claygthanks!21:14
marzif_ok :)21:15
*** Manshoon has quit IRC21:19
*** Manshoon has joined #openstack-swift21:20
*** sandywalsh has quit IRC21:22
*** sandywalsh has joined #openstack-swift21:23
*** Manshoon has quit IRC21:24
*** marzif_ has quit IRC21:32
klrmnclayg: i decided the VM was hosed and rebuilt it. even tho my localrc had 3 policies in it, the VM only has the default policy21:34
klrmnclayg: ah, maybe because i designated one as an EC policy but am not on the EC branch...21:35
claygklrmn: i think it would have tried to setup the ec policy and swift's checks on swift.conf would have complained21:36
claygif your localrc is different from you get on your node you need a vagrant provision to stuff the chef bits into the vm21:36
*** RackerShagz has joined #openstack-swift21:36
*** RackerShagz has quit IRC21:38
*** NM1 has quit IRC21:42
openstackgerritMerged openstack/swift: swift-ring-builder remove doesn't print message (tabbing error)  https://review.openstack.org/15878921:44
mattoliverauMorning21:50
tdasilvamattoliverau: hello22:05
*** delattec has joined #openstack-swift22:08
*** gyee has quit IRC22:09
*** gyee has joined #openstack-swift22:17
*** gyee has quit IRC22:18
*** gyee has joined #openstack-swift22:26
*** erlon has quit IRC22:31
klrmnclayg: it helps if i remember to source localrc22:32
*** gyee has quit IRC22:34
*** openstackgerrit has quit IRC22:37
*** openstackgerrit has joined #openstack-swift22:37
claygoh yeah that22:43
claygw22:43
*** jrichli_ has quit IRC22:47
torgomaticlooks like f578a35100f5dcd0046c79e810441633d28f55ff broke some probe tests23:14
*** chlong has joined #openstack-swift23:19
*** kei_yama has joined #openstack-swift23:21
*** joeljwright has quit IRC23:23
*** gyee has joined #openstack-swift23:27
klrmnclayg: #$%^&* so, because i had to re-do the VM, i've lost my branch and all it's commits23:40
claygklrmn: wat?  the code should all be on the host?23:42
klrmn~/Swift/vagrant-swift-all-in-one$ git branch23:42
klrmn* master23:42
klrmnoh! duh23:42
klrmnclayg: you're right. i'm clearly kinda slow today23:43
*** ahonda has joined #openstack-swift23:45
*** chlong_ has joined #openstack-swift23:50

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!