Thursday, 2015-04-30

notmynameswift-bench --saio fails spectacularly00:01
*** km has joined #openstack-swift00:03
*** km__ has quit IRC00:03
*** km__ has joined #openstack-swift00:05
*** km has quit IRC00:07
notmynamefucntional test results:00:08
*** breitz has quit IRC00:08
notmynameRan 340 tests in 117.440s00:08
notmynameFAILED (SKIP=12, errors=4, failures=8)00:08
*** breitz has joined #openstack-swift00:08
notmynamelooks like some container listing stuff, some utf8 stuff, and some cross-policy stuff00:09
notmynameah, looks like my original swift-bench issue was because I had an EC policy as default00:10
claygnotmyname: the docs *said* not to use storage policies00:11
swifterdarrellwho reads docs?00:11
claygswifterdarrell: not swift devs :\00:11
swifterdarrellclayg: touché00:12
notmynameswift-init <server> <action> vs hummingbird <action> <server>00:14
claygswift-init can do *either*00:14
notmynameoh really? TIL00:14
notmynamewhy would I try something new if the old way works ;-)00:15
claygnotmyname: for sure00:15
claygi just kept typing it backwards and both ways - so i made the code read my mind00:15
notmynameso my initial tests on my virtualized SAIO (so TOTALLY exactly like production) shows that hummingbird is faster00:16
claygi got about 100% on PUT00:16
notmynameok, gotta run. wife calling, kids crazy, etc00:16
claygyeah so I got to trace some code through hummingbird/main.go - grep found the definition in common/utils - and code explained what I was doing wrong - this will totally work00:18
claygredbo: I wish you didn't dump the whole thing out at once, it's going to take a long while to digest 8K lines00:20
claygredbo: like the objectserver has a REPLICATE verb?00:22
*** gyee has quit IRC00:22
claygoh, that's just the hashes00:23
claygoh it has SYNC00:23
claygok, i'm quitting out for this afternoon - thanks for the code dump redbo!00:28
*** dmorita has joined #openstack-swift00:29
claygdfg_: hurricanerix: scotticus: great work!00:29
*** tsg has quit IRC00:30
*** shri has left #openstack-swift01:03
*** kota_ has joined #openstack-swift01:23
kota_morning, again.01:23
*** asettle is now known as asettle-gym01:38
mattoliveraukota_: morning (again)01:45
*** tsg has joined #openstack-swift01:50
kota_mattoliverau: :)01:53
*** tsg has quit IRC02:05
*** tsg has joined #openstack-swift02:25
*** jrichli has joined #openstack-swift02:26
*** bkopilov has quit IRC02:36
*** fifieldt has joined #openstack-swift02:36
*** ir2ivps8_ has quit IRC02:48
*** ir2ivps8 has joined #openstack-swift02:48
jrichlinotmyname: I created an etherpad for the encryption fishbowl session.  Should we use that one for the working session also?02:57
*** asettle-gym is now known as asettle02:58
*** erlon has quit IRC03:11
*** jrichli has quit IRC03:29
*** tsg has quit IRC03:34
*** gvernik has joined #openstack-swift03:36
*** jrichli has joined #openstack-swift03:41
*** ho has quit IRC03:42
jrichlinotmyname: nm.  I think I want a separate etherpad for them both now that I think about it.03:42
notmynamejrichli: glad to help ;-)03:44
*** thumpba has joined #openstack-swift03:46
*** jrichli has quit IRC03:53
*** tsg has joined #openstack-swift03:57
*** bkopilov has joined #openstack-swift04:25
*** zhill has joined #openstack-swift04:38
*** zhill has quit IRC04:38
*** gvernik has quit IRC04:44
openstackgerritKota Tsuyuzaki proposed openstack/swift: Remove confusable query string on post as copy
*** zhill has joined #openstack-swift04:54
*** zaitcev has quit IRC04:57
*** ppai has joined #openstack-swift05:19
*** openstackgerrit has quit IRC05:22
*** openstackgerrit has joined #openstack-swift05:22
*** esker has quit IRC05:28
*** esker has joined #openstack-swift05:33
*** zhill has quit IRC05:36
mattoliverausharding update: Don't want to jinx anything.. But sharing in the POC seems to be sharding. Now hopefully I've correctly plumbed the replicators to replicate the shard nodes table in the DB.. Think I need a break before the next round of debugging :p05:38
mattoliverauSoon comes testing large containers and tuning/fixing bottlenecks05:39
*** silor has joined #openstack-swift05:42
openstackgerritKota Tsuyuzaki proposed openstack/swift: Remove confusable query string on post as copy
*** joeljwright has left #openstack-swift06:43
*** zul has joined #openstack-swift06:53
*** tsg has quit IRC07:02
*** tsg_ has joined #openstack-swift07:05
*** zul has quit IRC07:12
*** zul has joined #openstack-swift07:12
*** thumpba has quit IRC07:13
*** thumpba has joined #openstack-swift07:14
*** thumpba has quit IRC07:18
*** geaaru has joined #openstack-swift07:20
*** jordanP has joined #openstack-swift07:22
*** tsg_ has quit IRC07:33
*** chlong has quit IRC07:42
openstackgerritKota Tsuyuzaki proposed openstack/swift: Remove confusable query string on post as copy
*** geaaru has quit IRC07:52
kota_hi, matthow, still here?07:57
kota_no response but that's okay I'll ask someone at aonther timing.07:59
*** jistr has joined #openstack-swift08:04
mattoliveraukota_ what's up?08:07
kota_mattoliverau: ah, nice08:11
kota_mattoiverau: jast for confirmation, now I'm looking at EC code and I'm wondering the behavior of reconstructor.08:11
kota_mattoiverau: current reconstructor seems to select an avaialble handoff nodes from primary nodes, does it correct?08:12
kota_mattoliverau: AFAIK, replicator will pick up the handoff nodes from get_more_nodes (i.e. different nodes from primaries)08:13
kota_mattoliverau: Do you know the reason for the EC behavior?08:14
kota_mattoliverau: on the case partner nodes unmounted the disk.08:15
mattoliverauYup, when replicating we want to make sure we have the right number of replicas out there for data retention purposes08:15
mattoliverauThe EC recomstructor can only rebuild using EC parts (to recreate missing bits)08:16
mattoliverauI'd need to look closely, but if a primary is missing on reconstruction (primary down, then it'll use a hand off node).. I hope (assume)08:17
mattoliverauNot in front of a computer at the moment but will confirm when I am :)08:18
kota_mattoliverau: hah08:19
kota_mattoliverau: thanks for going to look at :)08:19
kota_mattoliverau: I also test the behavior more, and I think we should use the handoff nodes in the reconstruction case too, right?08:20
kota_mattoliverau: for data retention and some reason like that.... (i.e. one disk have to keep one fragment archive)08:21
mattoliverauYeah, I'd hope so.. When a primary node can't be contacted.08:22
kota_mattoliverau: ok thanks, I'm happy to hear your opinion ;)08:23
*** krykowski has joined #openstack-swift08:30
*** early has quit IRC08:31
*** geaaru has joined #openstack-swift08:32
*** zul has quit IRC08:38
*** acoles_away is now known as acoles08:38
acoleskota_: hi08:42
kota_acoles: hi08:42
acolesthere are two kinds of reconstructor jobs - 'sync' and 'revert'...08:44
kota_acoles: like as update and update_deleted on replicator08:44
mattoliverauYay its acoles!08:44
acoleskota_: a sync job picks partner primary nodes and attempts to rebuild any fragment that is missing on the partner08:44
acolesmattoliverau: hi!08:45
acoleskota_: yes similar08:46
acolesa 'revert' job attempts to move fragments to their 'correct' primary node08:46
acoleswhere the fragment exists but is on the wrong node08:46
acolesa 'revert' will first try the correct primary for the frag index, then try another handoff that may be a better handoff08:47
kota_acoles: yes, _revert calls get_more_nodes for destination.08:48
acoleskota_: but at the moment the reverts to a handoff don't work - there is a KeyError raised in reconstructor because get_more_nodes don't have an 'index' key08:48
acoleskota_: there's a patch to fix that, one min...08:48
*** early has joined #openstack-swift08:49
kota_acoles: oh..I've seen the fix, perhaps.08:49
kota_acoles: nice08:50
kota_acoles: reading...08:51
acoleskota_: right now clayg and i are uncertain how to proceed on that - its possible that if you revert to another handoff you can end up with two copies of the same frag on two handoffs - the probe test in that patch shows that happening08:51
acoleskota_: so basically there is some more work required108:52
acoless/1/!/ !)08:52
kota_I see...08:53
kota_acoles: for the summary, we are now working to achieve that reconstructor places the frag to the handoff with revert08:55
kota_acoles: right?08:55
acoleskota_: yes...but...:) ...there's another 'open question' which is do we want to revert a fragment to another handoff if that handoff already has another fragment for same object ie do we want to deliberately have and on same handoff?08:56
acoleskota_: i *think* clayg has opinion that that is not good from a durability point of view08:57
acoleskota_: and i *think* i agree with clayg ;)08:57
acoleskota_: so my current thinking is to work towards being able to revert a fragment from one handoff to a better handoff IFF the better handoff does not already have another fragment of same object, and IFF it can be done without accidentally ending up with two copies of same fragment. phew!08:59
*** km__ has quit IRC08:59
acoleskota_: the second part (IFF it can be done without accidentally ending up with two copies of same fragment) needs some changes to ssync which i have in my head but not on gerrit :P08:59
mattoliverauCan we review and sent patches to acoles head ( the new gerrit ) :p09:01
kota_acoles: yeah, i *think* i agree with you we shouldn't allow the handoff nodes to have more than one fragment of the same object.09:01
acolesmattoliverau: i think that could prove even less reliable than the real gerrit :P09:02
mattoliverauFI on any handoff node (even more then one) is better then on no handoff nodes.. Though yes, better to keep FI on handoffs seperate09:04
acoleskota_: yeah, so i see two reasons to avoid that - (1) right now the proxy won't read both frags from one node (i think) (2) if one fragment has bit rot the auditor will quarantine them both :/09:04
mattoliverauFI == FA :p09:04
kota_acoles: right, maybe my collegue hit the behavior.09:05
acolesmattoliverau: oh, yes, agree. question is whether the reconstructor chooses to move an FA to cohabit with another FA of same object09:05
kota_acoles: I'm not sure but he reported GET object failed with in the PyECLib layer said "no enough fragments to decode"09:06
acolesmattoliverau: heh, 'cohabiting' FAs sounds real friendly09:07
kota_acoles: I didn't done to understand whole codes but it seems to allow duplicated fragments like as09:07
acoleskota_: interesting09:07
acolesmattoliverau: but the have a death pact :/09:07
kota_acoles: nstream_decode[0, 0, 1, 2 ,3] (the number means FI) on k=5 and m=2 or so on....09:08
kota_acoles: *maybe*09:08
acoleskota_: so i am interested if you/colleague have seen two FAs land on same node - I don't yet know a way for that to happen09:08
mattoliverauWell they do all share the same tombstone :p09:09
*** silor has quit IRC09:09
acolesmattoliverau: it says 'Rest In Pieces'09:09
acoleskota_: huh, interesting09:09
kota_acoles: not finished to look at yet but I'm thinking reconstructor building the fragment and pushing to other primaries09:10
acoleskota_: but that could be same fragment on different nodes, yes?09:10
kota_acoles: on no failure case, we don't have same fragment on different nodes i think.09:11
mattoliverauLol! Love it09:11
kota_acoles: so that point I'm still wondering and found the reconstrucor might select the destination nodes from primaries.09:11
kota_acoles: on the sync job. but I'm not sure :\09:12
acoleskota_: hmmm, i see09:12
kota_acoles: that's why I asked you and mattoliverau.09:12
kota_acoles: currently I'm testing whether the different nodes have a possibility to have the same fragment of the same object09:13
kota_acoles: (maybe we need "probe" test for that :/)09:14
acoleskota_: so tbh clayg is probably more up to speed on this, but maybe there is a case where frag 0 lands on a handoff and then a sync job also rebuilds frag 0 on its primary, so there are two copies of frag 0 until the reconstructor runs on the handoff and should clean up its copy.09:14
acoleskota_: probe test would be great!09:15
kota_acoles: oh, yes. correct, we could feed the same fragment to decode and goes to fail unfortunately.09:16
acoleskota_: right, if during that window another primary fails then the proxy may bet frag 0 from two nodes??09:17
kota_acoles: for that case, maybe, we have to validate the frag-index before starting to decode.09:17
kota_acoles: nice assumption, I'll also try to make the test and code improvement.09:19
kota_acoles: Thanks, my head is becoming to be clear.09:20
acoleskota_: if you can reproduce with a probe test then even post only that to gerrit and we can discuss how to fix in comments. i know that clayg has thought about this stuff perhaps more than me09:23
kota_acoles: ok :)09:24
kota_acoles, mattoliverau: FYI, (*unfortunately*) I have a week holidays since tommorow untill thursday next week because there are many continious national holidays in the term in Japan :(09:24
kota_maybe I'll be watching on IRC and gerrit during the week :P09:25
mattoliveraukota_: not unfortuantly, go have fun :) We leave you a bunch of work to do when you get back :P09:26
acoleskota_: sounds good - have a good holiday! we have Monday as holiday in UK but I like the sound of 'continuous national holidays' :D09:26
kota_mattoliverau, acoles: lol, thanks ;)09:27
*** zul has joined #openstack-swift09:38
*** dmorita has quit IRC09:44
kota_leaving from office for going back to home and dinner.10:06
*** kota_ has quit IRC10:07
openstackgerritMerged openstack/swift: Fix tempauth acl checks when simplejson has no speedups
*** openstackgerrit_ has joined #openstack-swift10:15
*** knl has joined #openstack-swift10:21
*** silor has joined #openstack-swift10:23
*** krykowski has quit IRC10:26
*** krykowski_ has joined #openstack-swift10:26
*** openstackgerrit_ has quit IRC10:31
*** zul has quit IRC10:35
*** krykowski_ has quit IRC10:37
*** krykowski_ has joined #openstack-swift10:38
*** mwheckmann has quit IRC10:40
*** xnox has joined #openstack-swift10:45
xnox"We respectfully ask that you voluntarily remove and not republish the libraries JErasure 2.0 and GF_Complete, as well as any other packages or releases that incorporate them or depend on them. We believe these packages and releases include: libjerasure-dev, libjerasure2, libgf-complete-dev, libgf-complete1, liberasurecode-dev, liberasurecode1, Pyeclib, CEPH (see ceph/src/erasure-code/jerasure) and10:45
xnox Swift
xnoxah old news10:48
*** krykowski_ has quit IRC11:02
*** krykowski has joined #openstack-swift11:02
*** jamielennox is now known as jamielennox|away11:20
*** aix has joined #openstack-swift11:21
*** krykowski has quit IRC11:34
*** knl has quit IRC11:48
*** zul has joined #openstack-swift11:50
*** dencaval has joined #openstack-swift11:51
*** kota_ has joined #openstack-swift12:04
*** ppai has quit IRC12:05
*** chuck__ has joined #openstack-swift12:08
kota_hmm...sync job will reconstruct the FA with the target fragment index. It makes me sense the reason the reconstructor sets only part_nodes to dest_nodes for now.12:09
*** zul has quit IRC12:10
*** krykowski has joined #openstack-swift12:27
*** bkopilov has quit IRC12:31
*** panbalag has joined #openstack-swift12:35
*** knl has joined #openstack-swift12:35
*** jistr has quit IRC12:39
*** ahale has joined #openstack-swift12:44
kota_FWIW, I'm a little worried the revert job for tombstones would fan a new .data came after building jobs out to primaries.12:46
kota_not sure and need to look at more deeply :/12:47
*** chuck__ has quit IRC12:53
kota_hmm...I need more fresh head to think clealy so stop here today.12:54
*** kota_ has quit IRC12:54
*** jistr has joined #openstack-swift12:54
*** jistr is now known as jistr|biab12:55
*** geaaru has quit IRC12:57
*** thumpba has joined #openstack-swift12:58
*** thumpba_ has joined #openstack-swift13:00
*** thumpba has quit IRC13:04
*** fifieldt has quit IRC13:04
eikkeare Swift 2.2.0 functional tests supposed to pass on a Python 2.6 system?13:13
*** esker has quit IRC13:18
*** esker has joined #openstack-swift13:18
*** esker has quit IRC13:19
eikkenotmyname: ^^13:20
*** cdelatte has joined #openstack-swift13:33
*** zul has joined #openstack-swift13:35
*** ppai has joined #openstack-swift13:36
*** zul has quit IRC13:40
*** krykowski has quit IRC13:44
*** jrichli has joined #openstack-swift13:49
*** jistr|biab is now known as jistr13:53
*** proteusguy has joined #openstack-swift13:55
*** chlong has joined #openstack-swift13:56
*** erlon has joined #openstack-swift13:56
*** mwheckmann has joined #openstack-swift13:58
*** wbhuber has joined #openstack-swift13:59
jrichlieikke: how many failures are you getting?  Have you tried a resetswift and then run again?  swift still supports 2.613:59
*** zul has joined #openstack-swift14:00
*** esker has joined #openstack-swift14:02
jordanPjrichli, we see more than 60 failures, related to Unicode Support in the swift client14:11
*** ppai has quit IRC14:13
acoleseikke: jordanP : i just ran functests ok in a py26 venv on my SAIO14:14
*** ppai has joined #openstack-swift14:14
*** vinsh has quit IRC14:23
*** NM has joined #openstack-swift14:24
*** zul has quit IRC14:26
*** JelleB is now known as a1|away14:30
*** vinsh has joined #openstack-swift14:41
*** wbhuber__ has joined #openstack-swift14:41
*** lpabon has joined #openstack-swift14:42
*** wbhuber has quit IRC14:43
*** zul has joined #openstack-swift14:49
*** chlong has quit IRC14:53
*** ppai has quit IRC14:59
*** aerwin has joined #openstack-swift15:00
pelusexnox, Swift core project does not include EC libraries.  They are external to the project and users are free to choose from multiple options like jerasure or ISA-L from Intel15:01
pelusexnox, but thanks and yes we are all well aware of the info you passed on :)15:01
*** annegentle has joined #openstack-swift15:02
xnoxpeluse: right. i'll go back to ignoring trolls.15:03
pelusethat's always the best option for sure!15:03
notmynamepeluse: thanks15:17
notmynameyeah, swift does not depend on jerasure15:17
notmynameeikke: and jrichli told you correctly. current swift should still work under py2615:17
lpabonnotmyname: hi... what's the story with hummingbird?  Should I spend some time reviewing the Go code, or should I wait for it to stabilize?15:20
notmynamelpabon: take a look, see what you think. vancouver will be a great place to have a good discussion15:21
lpabonnotmyname: will do15:21
notmynameright now it's an idea in a feature branch, and between now and the tokyo summit I'd love to explore the idea of "compiled language object server"15:21
lpabonlpabon: Sweeet.. plus.. Go is really awesome15:22
portantelpabon: are you talking to your self about Go again? ;)15:25
lpabonportante: you know it.. I want to shout it from the mountain tops15:25
*** jordanP has quit IRC15:25
dmsimardpeluse: So, I couldn't sleep until I was satisfied last night. I got up to 8 Gbps15:28
dmsimardCould probably do more but that's where I went to sleep15:28
pelusedmsimard, great, what did ya do?15:28
pelusejust lots of coffee?15:29
dmsimardSo long story short, there were two things15:29
dmsimard1) I wasn't pushing ssbench enough so I boosted the scenarios in amount of files, their size, the amount of operations but especially the user_count and the worker concurrency15:29
dmsimardOnce I had done that, haproxy was choking under the load so I looked into how I could solve that15:30
dmsimardTurns out a single haproxy process wasn't enough - I put tentatively nbproc at 4 and the traffic it was able to pull was greatly increased15:31
pelusehmm, interesting.  don't know what ours was setup for but will check15:31
dmsimardWith that done, I was getting a lot of CPU %sys and Soft IRQs so it looks like the next bottleneck is the network and SSL TPS processing15:31
dmsimardWe've had to do some network card config tweaking before due to the high amount of pps, by default the processing will only be done on one core but you can "spray" the packets across multiple cores for improved throughput15:33
pelusehow are your card configured - num links, bonded, etc?15:34
dmsimardLoad balancers are 2x 10 Gbps in LACP, hash layer 2+3 - that same bond handles both WAN and LAN connectivity (storage nodes are behind in a LAN)15:35
peluseOK, IM'ing with our benchmark guy and he said they didn't have the cards balanced properly accross cpus and are going to try again today15:37
peluseBTW, same config here on the NICs in the LB15:38
peluseglad to hear you made such great progress!15:39
*** annegentle has quit IRC15:53
*** EmilienM is now known as EmilienM|afk15:55
*** mahatic has joined #openstack-swift16:01
*** lpabon has quit IRC16:01
*** delattec has joined #openstack-swift16:06
*** delatte has joined #openstack-swift16:06
*** gyee has joined #openstack-swift16:07
*** cdelatte has quit IRC16:09
*** zaitcev has joined #openstack-swift16:13
*** ChanServ sets mode: +v zaitcev16:13
openstackgerritJohn Dickinson proposed openstack/swift: Merge branch 'stable/kilo' into master
notmyname^ that one will make the tags work out for versioning16:21
*** mahatic has quit IRC16:33
*** gyee has quit IRC16:37
*** guitarzan has joined #openstack-swift16:39
*** knl has quit IRC16:40
*** zul has quit IRC16:40
notmynamemailing list post:
notmynamegood mornign hurricanerix16:41
*** mahatic has joined #openstack-swift16:42
eikkenotmyname: thats what we thought17:05
eikkenotmyname: seems related to #1190190, on centos617:05
eikkenotmyname: will investigate next week (sorry for multi-message, on a train, terrible latencies)17:06
pelusenotmyname, FYI I just added a py3.0 perf discussion topic for the Fri session - what we talked about before about working with our compiler/interpreter optimization team on improvements in 3.017:08
* eikke also *very* interested in the whole language/platform thing, but not exactly a Go fan17:12
*** mahatic has quit IRC17:14
pelusego go gadget... something17:15
*** cutforth has quit IRC17:21
*** zhill has joined #openstack-swift17:22
*** tsg_ has joined #openstack-swift17:29
openstackgerritAlistair Coles proposed openstack/python-swiftclient: Fix --skip-identical to allow identical xLO to replace non-xLO
*** jrichli_ has joined #openstack-swift17:37
*** thumpba has joined #openstack-swift17:37
*** jrichli_ has quit IRC17:40
*** BAKfr has quit IRC17:40
*** jrichli_ has joined #openstack-swift17:40
*** jrichli_ has quit IRC17:41
*** haomaiw__ has joined #openstack-swift17:41
*** anticw_ has joined #openstack-swift17:42
*** guitarza1 has joined #openstack-swift17:42
*** jeblair_ has joined #openstack-swift17:42
openstackgerritMerged openstack/swift: Merge branch 'stable/kilo' into master
*** dmsimard_ has joined #openstack-swift17:45
*** aerwin has quit IRC17:46
*** jrichli has quit IRC17:47
*** thumpba_ has quit IRC17:47
*** haomaiwang has quit IRC17:47
*** anticw has quit IRC17:47
*** dmsimard has quit IRC17:47
*** omame has quit IRC17:47
*** jistr has quit IRC17:47
*** jeblair has quit IRC17:47
*** a1|away has quit IRC17:47
*** guitarzan has quit IRC17:47
*** delattec has quit IRC17:47
*** morganfainberg has quit IRC17:47
*** alpha_ori has quit IRC17:47
*** jd__ has quit IRC17:47
*** redbo has quit IRC17:47
*** tanee has quit IRC17:47
*** dmsimard_ is now known as dmsimard17:47
*** delattec has joined #openstack-swift17:49
*** morganfainberg has joined #openstack-swift17:49
*** alpha_ori has joined #openstack-swift17:49
*** jd__ has joined #openstack-swift17:49
*** redbo has joined #openstack-swift17:49
*** tanee has joined #openstack-swift17:49
*** sets mode: +v redbo17:49
*** bkopilov has joined #openstack-swift17:54
*** a1|away has joined #openstack-swift17:55
*** BAKfr has joined #openstack-swift18:02
*** acoles is now known as acoles_away18:05
*** annegentle has joined #openstack-swift18:05
*** aix has quit IRC18:07
*** EmilienM|afk is now known as EmilienM18:11
*** jrichli has joined #openstack-swift18:16
jlkHey all, I'm back with more tempurl problems18:22
*** bkopilov has quit IRC18:23
jlkI can't seem to make tempurl auth work18:23
jlkand I can't convince swift-proxy to tell me why it's 401ing the url18:23
jlkThis is the log entry in swift-proxy I'm getting
*** bkopilov has joined #openstack-swift18:26
jlkI generated the Meta Temp-Url-Key, and used swift tempurl to create a GET url18:27
jlkswift tempurl GET 200 /v1/AUTH_tempest4/test1/upgrade.retry password118:27
jlk"tempest4" is the name of the tenant my user belongs in18:27
jlkCan somebody help me debug what's going on here?18:28
notmynamejlk: do a HEAD to your account (AUTH_tempest4) to make sure that the tempurl key is actually set18:32
jlkwhat would that URL look like?18:32
jlkor would a simple swift stat suffice?18:32
jlkswift stat shows:18:32
jlk              Meta Temp-Url-Key: password118:32
jlkfor the AUTH_ part, should I be using tenant name, tenant uuid, user uuid?18:34
jlkI see mixed examples on the 'net18:34
notmynameit's not something like not escaping the & on the command line, right?18:34
jlkwell, I used swift tempurl to make it, which appears to properly escape all the parts of the url18:35
*** bkopilov has quit IRC18:35
jlkit spat out:18:35
*** esmute has quit IRC18:35
notmynamejlk: that depends on what your auth system says. the only hard rule, is that you use the storage url that the auth system gives you18:35
*** bkopilov has joined #openstack-swift18:35
*** esmute has joined #openstack-swift18:35
jlker, where would my auth system say that? I'm using keystone for auth with swift, although in the swift-proxy I have tempurl listed before authtoken or keystoneauth18:36
jlkand the tempurl filter just lists use = egg:swift#tempurl18:37
jlkHere's a more complete output
torgomaticjlk: did you set allow_overrides = true in your keystoneauth config? or delay_auth_decision or whatever it is?18:41
torgomatichang on18:41
jlkyeah, delay_auth_decision is set18:41
torgomaticokay, that's good18:41
jlkApr 30 18:38:34 localhost.localdomain proxy-server: Invalid user token - deferri18:41
jlkng reject downstream18:41
jlkApr 30 18:38:34 localhost.localdomain proxy-server: Authorizing from an overridi18:42
jlkng middleware (i.e: tempurl) (txn: tx0844182784cc4f6f874c5-00554276aa)18:42
torgomaticso the tempurl middleware won't ever return a 401 like that; the body will say "Temp URL invalid" or something similar18:42
torgomaticjlk: okay, so that looks good18:42
jlktorgomatic: if I do a straight curl instead of a HEAD, I get 401 Unauthorized: Temp URL invalid18:43
*** bkopilov has quit IRC18:43
torgomaticjlk: the gist you sent does not include the body of the response; could you post a new one with the entire failing curl?18:43
*** guitarza1 is now known as guitarzan18:45
jlkIs this what you meant?
jlkre-ran with -v
zaitcevy u not -v18:47
claygwhat is going on!18:47
torgomaticjlk: okay, and how about the output of "swift stat"? that looks to me like your temp url key isn't set properly18:48
openstackgerritMerged openstack/python-swiftclient: Compare each chunk of large objects when uploading
jlkodd that stat is showing 0 objects even though I've uploaded an object into  my container18:49
torgomaticjlk: eh, it's eventually consistent18:49
torgomaticjlk: so that says your account is 7bb348216c144eb69cae5d7777cb604f18:51
torgomaticyour tempurl is for AUTH_tempest418:52
jlkyeah, one is the name, the other is the UUID18:52
torgomaticuse the uuid18:52
*** annegentle has quit IRC18:52
torgomatic /v1/<uuid>/container/object18:52
jlknot even AUTH_<uuid> ?18:53
torgomaticdoesn't look like it18:53
torgomaticuse the thing that comes after "Account: " in the output of `swift stat`, because that's the account's name on disk18:53
jlkcurl ""18:53
torgomaticjlk: did you regenerate the tempurl with the new name?18:54
jlk(after generating a new one)18:54
jlkoh wait18:54
jlkgeez, now it works18:54
jlkso the docs for this... really not good18:55
*** bkopilov has joined #openstack-swift18:56
jlkand tempest passes now, because my catalog was bad.19:01
*** cdelatte has joined #openstack-swift19:01
*** delattec has quit IRC19:01
*** delatte has quit IRC19:02
*** delattec has joined #openstack-swift19:02
*** bkopilov has quit IRC19:03
*** jd__ has quit IRC19:04
*** jd__ has joined #openstack-swift19:06
jlktorgomatic: notmyname: thank you for the help19:09
torgomaticjlk: you're quite welcome19:09
*** thumpba has quit IRC19:13
*** bkopilov has joined #openstack-swift19:15
*** bkopilov has quit IRC19:16
openstackgerritTim Burke proposed openstack/python-swiftclient: Actually function under python 3
*** bkopilov has joined #openstack-swift19:20
*** bkopilov has quit IRC19:29
zaitcevjlk: I filed a bug to remind ourselves about docs
openstackLaunchpad bug 1450606 in python-swiftclient "the tempurl subcommand is missing from the man page" [Undecided,New]19:32
jlkoh thanks!19:32
*** omame has joined #openstack-swift19:32
clayghas anyone tried to get pyeclib installed on a mac?19:32
claygit's starting to bum me out that I can't import swift in a repl from a virtualenv on my mac19:33
*** a1|away has quit IRC19:33
*** wolsen has quit IRC19:33
*** d0ugal has quit IRC19:33
*** wolsen_ has joined #openstack-swift19:33
zaitcevGod, no. Linux was trouble enough.19:33
*** mwheckmann has quit IRC19:33
claygmwheckmann: no way man - stick with it!19:33
claygzaitcev: :P19:34
*** a1|away has joined #openstack-swift19:34
*** d0ugal has joined #openstack-swift19:34
*** bkopilov has joined #openstack-swift19:35
*** esker has quit IRC19:36
jlkwrong win19:37
*** bkopilov has quit IRC19:41
jrichlilol.  I just know I am gonna post one of my passwords one of these days.19:42
*** bkopilov has joined #openstack-swift19:42
clayggot a compile error on my mac - looks like -Werror might be just being mean to me?19:48
claygtsg_: I don't recall seeing anything like this on my linux box - but I *did* just pull master?19:48
*** bkopilov has quit IRC19:49
claygidk, it sure does look like we're testing if an unsigned int is less than 0 - that does *seem* silly?19:51
claygis a 900+ line Makefile big?  feels kinda epic19:53
peluseclayg, our internal IM shows tsg offline for the last 12 hrs, he might be travelling19:58
*** jd__ has quit IRC19:58
claygpeluse: I try to pretend that i can talk to people on irc even if they're not here :\19:59
clayghow is that after make I don't have any .so's in my liberasurecode folder19:59
claygdoes freebsd/mac not call shared object's .so?20:00
*** tsg_ has quit IRC20:05
*** tsg_ has joined #openstack-swift20:06
*** vinsh_ has joined #openstack-swift20:06
tsg_clayg: noticed your comment above on libec error20:07
tsg_clayg: let me check the latest Kota commits20:07
claygtsg_: dont' worry about it right just yet - i fixed that20:07
claygso I have a make that finishes on my mac - but I don't see the .so's?20:07
tsg_clayg: you should have .dylibs?20:08
* clayg was never good a building c projects :'(20:08
claygtsg_: YES!20:08
claygyou rock20:08
claygah ./src/.libs20:08
claygdidn't notice the dot-dir20:08
tsg_that's libtool!20:08
claygtsg_: so I'm trying to avoid a `sudo make install` and would rather just tell my swift venv how to find the liberasure libs it needs20:09
tsg_clayg: you can do "make install DESTDIR=<dir>"?20:09
claygnot that my mac is so "pristine" or anything - but I couldn't even really tease out from the Makefile what 'install' would *do* - it wasn't obvious to me that it would "do the right thing" on a mac20:09
tsg_and add that dir to ldconfig?20:09
*** vinsh has quit IRC20:10
tsg_(or the equivalent on mac .. )20:10
claygmac says ldconfig not found - do i really need to brew install that?  I thought if I added something to some env somewhere the pyeclib build could find it?20:10
tsg_clayg: DYLD_LIBRARY_PATH20:11
claygtsg_: ok that's good - turns out that pyeclib install is trying to do the download and compile trick20:12
* notmyname is convinced that tsg_ has an encyclopedic knowledge of every platform's build tooling and environment20:12
claygand it's blowing up on the same error with the unsigned int thing - so maybe I should have just let you look at that :P20:12
claygtsg_: but since i have managed to get liberasure build in another dir - it seems like i should be able to convince the pyeclib install to just use that20:13
claygi set the DYLD_LIBRARY_PATH but either i did it wrong or it didn't buy it20:13
* tsg_ checking "man dyld"20:14
*** erlon is now known as erlon_awaY20:19
tsg_clayg: btw, the Makefile is generated w/ automake so is full of GNU'isms .. thus LONG :)20:22
tsg_clayg: and I checked DYLD_LIBRARY_PATH should have worked :|20:22
tsg_clayg: let me look at the -Werror thing first20:23
claygtsg_: awesome - super helpful thanks!20:23
*** dencaval has quit IRC20:27
tsg_clayg: are you running Yosemite?  what gcc version?20:28
claygApple LLVM version 6.1.0 (clang-602.0.49) (based on LLVM 3.6.0svn)20:29
clayg^ so you know - total garbage!20:29
tsg_clayg: ah clang .. :) ok gcc didn't complain20:29
claygheh :P20:29
claygyeah i mean fixing my environment so i can compile without errors during the pyeclib install seems like a reasonable work-around?20:29
tsg_clayg: nah .. it is a legitimate warning - let me push a fix20:30
tsg_clayg: tells me I need to try it building manually with clang (kept thinking Eric Lambert had a Jenkins job testing with clang .. need to check with him)20:31
ekarlsonotmyname: are you at rackspace atm or smth since you mentioned cloudfiles team ? :P20:31
notmynameekarlso: no, I've worked at swiftstack for 3 years now20:35
ekarlsonotmyname: ok :D20:36
*** jd__ has joined #openstack-swift20:41
tsg_clayg: tried with clang, also added a "-Wall" - looks like the uint comparison was the only leftover warning20:41
claygtsg_: right!  so it's easy20:42
tsg_pushed a fix (along with the -Wall addition)20:43
tsg_clayg: are you still seeing the DYLD_LIBRARY_PATH issue?20:44
clayghrmm... so I tried with "gcc (Homebrew gcc 4.9.2_1) 4.9.2" and got a different "no such instruction: `vmovdqa (%rdi,%rax), %xmm0'" error20:44
claygtsg_: I was kicking around trying to let pyeclib build libearusre on install with a different compiler20:45
tsg_clayg: stale binaries perhaps?20:45
claygwho's stale now?20:45
tsg_clayg: :)20:45
tsg_clayg: assuming you are building in the same tree as the clang build, does a "make mrproper && ./configure && make" help?20:46
tsg_make distclean20:46
* tsg_ has been building linux kernels too often lately! (that's where the mrproper came from)20:47
claygwell yeah so the make works with clang after the fix20:47
claygthe undefined symbol thing was trying to build with homebrew gcc20:47
claygso - like whole different direction20:47
tsg_clayg: unless the gcc build was in a completely separate tree, I was thinking there were perhaps some leftover binaries from the clang build that gcc tried to use ..20:48
claygoh oh oh - yeah it was in a totally seperate tree - because it was a tempdir that pip pulled down20:48
tsg_clayg: ok .. so might be something to do with the gcc on Mac (I have little experience with that) - but I know that Kevin has been using gcc on Mac20:49
tsg_let me ask him what rev20:49
claygsure - i'd honestly be happier building it seperatly and making pip pyeclib install able to find the liberasure i already built20:50
claygbut yeah there the dyld thing didn't seem to be doing jack20:50
claygmaybe I could install pyeclib by hand and let the swift install just find that20:50
tsg_clayg: pip install may not be able to find liberasurecode if the DYLD_LIBRARY_PATH isn't global20:51
tsg_something that might work - "DYLD_LIBRARY_PATH=<path> pip install pyeclib"20:52
claygyeah that's *sorta* what I did20:52
claygnah it's still trying to download and install liberasurecode20:53
tsg_clayg: let me find the Mac guy20:53
*** annegentle has joined #openstack-swift21:00
claygtsg_: making progress installing pyeclib from source -> ld: library not found for -lerasurecode21:00
claygthis is where I would *think* the DYLD_LIBRARY_PATH think would help21:00
tsg_clayg: Just tried these steps on a Mac and seemed to work21:02
tsg_cd liberasurecode.git; ./configure21:02
tsg_; make;21:02
tsg_export DYLD_LIBRARY_PATH="src/.libs"21:02
tsg_; sudo pip install pyeclib21:02
tsg_clayg: sorry about the extra ';'s .. didn't know they'd turn into newlines :)21:03
claygtsg_: wait... is the remote origin for pyeclib - i pulled master and got 1.0.5?21:04
tsg_you should have gotten 1.0.7m21:04
claygwell it doesn't matter21:05
claygit worked :)21:05
clayger... what you told me to do worked21:05
tsg_cool! :)21:05
claygthe only think I had to add was C_INCLUDE_PATH=21:06
claygzaitcev: see it wasn't so bad - like *barely* three pages of scrollback21:06
zaitcevclayg: meanwhile I decompiled my CRUSH map and set a parameter there "step chooseleaf firstn 0 type osd", and now my "CAIO" works21:08
zaitcevalso took a bit of hand-holding on IRC21:09
claygyay IRC!21:09
openstackgerritJanie Richling proposed openstack/swift: WIP - working on the encryption feature.
zaitcevoh, speaking of those include paths21:11
tsg_clayg: what's with C_INCLUDE_PATH?21:12
claygtsg_: i got an error couldn't find the erasurecode.h or some such21:13
zaitcevclayg, tsg_: I eventually built liberasurecode like so:
zaitcevA smart guy suggested me to trick the code by placing all the right flags into CPPFLAGS21:14
claygzaitcev: that doesn't seem so unreasonable - why not push back?21:14
zaitcevBut I chose to mod the source instead. It's open source, right.21:14
zaitcevOur build environment passes CFLAGS down, which includes mandatory things in Fedora, like -fsomething-something-stack-protector21:15
tsg_zaitcev: ah ok .. I was about to ask21:15
zaitcevBut without that patch, liberasurecode ignores CFLAGS21:16
tsg_zaitcev: you mean overrides it?21:16
claygtsg_: no he just *adds* to it21:16
zaitcevtsg_: that's a good verb too21:16
zaitcevlike so - bug 1208695 in Package Review "Review Request: liberasurecode - Erasure Code API library written in C with pluggable backends" [Medium,Assigned] - Assigned to ppisar21:17
tsg_zaitcev: ok .. I will double check the top-level automake/autoconf setup21:17
*** jrichli has quit IRC21:18
tsg_zaitcev: thank you - as you mentioned in comment#5 there, please do submit the patch upstream21:19
tsg_zaitcev: also the m4 cleanup is almost done (Kevin and I are working on finalizing that)21:19
zaitcevtsg_: thanks, guys21:19
*** vinsh has joined #openstack-swift21:20
tsg_zaitcev: we are debating whether to support multiple cross-build targets (for SSE, AVX, NEON etc), or do the cpu capabilities detection at runtime21:21
tsg_zaitcev: in the former, we'd end up creating multiple liberasurecode subpackages21:21
zaitcevtsg_: Kevin e-mails me that it's inconvenient to make everything runtime. I trust you guys to figure it out... As long as I can build something on x86_64 that works on both Intel and AMD.21:22
*** vinsh_ has quit IRC21:23
tsg_zaitcev: :) OK .. we'll figure out something that works cross platform21:23
*** mragupat has joined #openstack-swift21:25
*** esker has joined #openstack-swift21:33
*** annegentle has quit IRC21:42
*** annegentle has joined #openstack-swift21:42
*** bkopilov has joined #openstack-swift21:48
*** annegentle has quit IRC21:48
pelusezaitcev, wait.. works on what?  :)22:07
*** annegentle has joined #openstack-swift22:08
*** gyee has joined #openstack-swift22:13
*** jamielennox|away is now known as jamielennox22:16
*** erlon_awaY has quit IRC22:21
*** mragupat has quit IRC22:22
zaitcevpeluse: That's nothing. You should see the OpenStack Manila RPMs that I have built today.22:35
zaitcevIt made me love Swift's modest requirements even more22:35
zaitcevHow would you like to bug maintainers to build python-oslo-concurrency >= 1.8.0, python-oslo-config >= 1.7.0, python-oslo-db >= 1.7.1, python-oslo-i18n >= 1.5.0, python-oslo-messaging >= 1.3.0-0.1.a9, python-oslo-serialization >= 1.4.0, python-oslo-utils >= 1.4.022:36
*** chlong has joined #openstack-swift22:36
zaitcevNone which I have access rights to build myself22:36
zaitcevThey also require the following transients: python-netifaces >= 0.10.4, python-alembic >= 0.7.2, python-migrate >= 0.9.5, python-stevedore >= 1.3.022:36
zaitcevI probably meant "transitives" there, if that is a word.22:37
*** esker has quit IRC22:42
*** annegentle has quit IRC22:43
*** NM has quit IRC22:48
InAnimaTehey all. im finding myself on a trusty box but needing to install grizzly (1.8.0)22:54
InAnimaTeadding the ppa obv yells at me22:54
InAnimaTebesides tracking down the dpkg's manually, any ideas?22:54
notmynameInAnimaTe: don't install from the PPAs. otherwise it should work22:54
notmynameworst case, you could install from source22:54
notmynameie if you can't find packages22:55
InAnimaTeyeah im looking for them right now22:55
*** wbhuber__ has quit IRC22:56
InAnimaTe^that should do :)22:57
*** annegentle has joined #openstack-swift22:58
InAnimaTealthough those are i38622:58
notmynameI'd suspect you'd be able to get packages from today's kilo release (2.3.0) from canonical23:01
notmynameoh, you need to use 1.8.023:02
InAnimaTethose package links are *_all.deb23:02
notmynamejust to state the thing that of course is always said by the maintainers of a project: you should upgrade :-)23:02
InAnimaTeehh, its not an option23:02
InAnimaTetrust me, i pushed for it23:02
InAnimaTemgmt ftl23:02
notmynameheh. like I said, it's just my duty to say that23:02
InAnimaTelol thx23:02
*** annegentle has quit IRC23:03
notmynametell them "some guy on the internet said I should upgrade". management loves that23:03
InAnimaTewere these packages only built i38623:04
InAnimaTei cant seem to find them for 64bit23:04
*** annegentle has joined #openstack-swift23:08
InAnimaTeArchitecture: all23:11
InAnimaTeso its just blanket supports everything23:12
zaitcevWell it's Python23:22
*** vinsh has quit IRC23:34
*** annegentle has quit IRC23:36
*** ho has joined #openstack-swift23:42
hogood morning23:58

Generated by 2.14.0 by Marius Gedminas - find it at!