Wednesday, 2014-02-19

*** mmcardle has joined #openstack-swift00:16
*** mmcardle1 has joined #openstack-swift00:18
*** mmcardle has quit IRC00:18
*** matsuhashi has joined #openstack-swift00:22
*** mmcardle1 has quit IRC00:23
*** mmcardle has joined #openstack-swift00:26
*** mmcardle has quit IRC00:31
*** matsuhashi has quit IRC00:32
*** matsuhas_ has joined #openstack-swift00:34
openstackgerritAlex Pecoraro proposed a change to openstack/swift: Allow hostname for nodes in Ring
*** bsdkurt has quit IRC01:05
*** russellb has quit IRC01:05
*** bsdkurt has joined #openstack-swift01:06
*** jergerber has joined #openstack-swift01:21
*** zaitcev has quit IRC01:21
*** mmcardle has joined #openstack-swift01:28
*** mmcardle has quit IRC01:32
*** sfineberg has joined #openstack-swift01:34
*** nosnos has joined #openstack-swift01:36
*** shri1 has quit IRC01:42
*** mmcardle has joined #openstack-swift02:29
*** byeager has joined #openstack-swift02:29
*** mmcardle has quit IRC02:33
*** Cotes has quit IRC02:35
*** zackf has quit IRC02:37
*** jergerber has quit IRC02:42
creihtmjseger: I still plan on doing more testing, that's just what I have done so far02:42
*** peluse has quit IRC03:11
*** peluse has joined #openstack-swift03:12
*** yuan has quit IRC03:13
*** byeager has quit IRC03:20
openstackgerritLuis de Bethencourt proposed a change to openstack/python-swiftclient: make the help strings constant
*** matsuhas_ has quit IRC03:29
*** mmcardle has joined #openstack-swift03:30
*** mmcardle has quit IRC03:34
*** byeager has joined #openstack-swift03:55
*** pandemicsyn has quit IRC04:00
*** gyee has quit IRC04:00
*** pandemicsyn has joined #openstack-swift04:07
*** ppai has joined #openstack-swift04:22
*** mmcardle has joined #openstack-swift04:32
*** matsuhashi has joined #openstack-swift04:35
*** mmcardle has quit IRC04:36
*** jokke_ has quit IRC04:43
*** jokke_ has joined #openstack-swift04:43
*** byeager has quit IRC04:49
*** yuan has joined #openstack-swift05:01
*** chandan_kumar has joined #openstack-swift05:26
*** nshaikh has joined #openstack-swift05:27
*** nshaikh has left #openstack-swift05:28
*** mmcardle has joined #openstack-swift05:34
*** mmcardle has quit IRC05:38
*** haomaiwang has quit IRC06:04
*** haomaiwang has joined #openstack-swift06:04
*** nshaikh has joined #openstack-swift06:22
*** mmcardle has joined #openstack-swift06:34
*** matsuhashi has quit IRC06:35
*** matsuhashi has joined #openstack-swift06:37
*** mmcardle has quit IRC06:39
*** Dharmit has joined #openstack-swift06:39
*** Dharmit has quit IRC06:45
*** Dharmit has joined #openstack-swift06:50
*** haomaiw__ has joined #openstack-swift07:12
*** haomaiwang has quit IRC07:13
*** basha has joined #openstack-swift07:16
*** chandan_kumar has quit IRC07:18
*** cschwede has left #openstack-swift07:20
*** basha has quit IRC07:20
*** nshaikh has quit IRC07:21
*** chandan_kumar has joined #openstack-swift07:25
*** Dharmit has quit IRC07:28
*** saju_m has joined #openstack-swift07:30
*** mmcardle has joined #openstack-swift07:36
*** mmcardle has quit IRC07:40
*** mmcardle has joined #openstack-swift07:50
*** basha has joined #openstack-swift08:08
*** cschwede has joined #openstack-swift08:13
*** saju_m has quit IRC08:21
*** Dharmit has joined #openstack-swift08:30
*** openstack has joined #openstack-swift08:42
*** zackmdavis has joined #openstack-swift08:43
*** alpha_ori has joined #openstack-swift08:43
*** chandan_kumar has joined #openstack-swift08:44
*** mlipchuk has joined #openstack-swift08:44
*** acorwin has joined #openstack-swift08:44
*** joeljwright has joined #openstack-swift08:44
*** amandap has joined #openstack-swift08:45
*** creiht has joined #openstack-swift08:45
*** ChanServ sets mode: +v creiht08:45
*** foexle has joined #openstack-swift08:46
*** otherjon has joined #openstack-swift08:46
*** swifterdarrell has joined #openstack-swift08:46
*** ChanServ sets mode: +v swifterdarrell08:46
*** anderstj has joined #openstack-swift08:47
*** mlanner has joined #openstack-swift08:48
*** hugokuo has joined #openstack-swift08:48
*** koolhead17 has joined #openstack-swift08:50
*** basha has joined #openstack-swift08:50
*** nosnos_ has joined #openstack-swift08:59
*** nosnos has quit IRC08:59
*** tane-away is now known as tane09:00
*** tane is now known as tanee-away09:00
*** tanee-away is now known as tane09:03
*** tane is now known as tanee09:03
*** chandan_kumar has quit IRC09:23
*** fbo_away is now known as fbo09:37
*** chandan_kumar has joined #openstack-swift09:41
*** chandankumar_ has joined #openstack-swift09:42
*** Midnightmyth has joined #openstack-swift09:45
*** chandan_kumar has quit IRC09:46
*** matsuhashi has quit IRC09:46
*** nosnos_ has quit IRC09:46
*** nosnos has joined #openstack-swift09:47
*** matsuhashi has joined #openstack-swift09:47
*** saju_m has joined #openstack-swift09:58
*** mlipchuk has quit IRC10:04
*** jsuchome has joined #openstack-swift10:04
*** mlipchuk has joined #openstack-swift10:21
*** chandankumar_ has quit IRC10:32
*** mlipchuk1 has joined #openstack-swift10:43
*** mlipchuk has quit IRC10:47
*** chandankumar_ has joined #openstack-swift10:47
*** mkollaro has joined #openstack-swift10:49
*** Trixboxer has joined #openstack-swift10:56
*** basha has quit IRC11:11
*** chandankumar_ has quit IRC11:14
*** chandan_kumar has joined #openstack-swift11:15
*** matsuhashi has quit IRC11:28
*** matsuhashi has joined #openstack-swift11:31
*** ppai has quit IRC11:34
*** ctennis has quit IRC11:35
*** ctennis has joined #openstack-swift11:35
*** ppai has joined #openstack-swift11:51
*** Anju has quit IRC11:57
*** saurabh_ has quit IRC11:58
*** matsuhashi has quit IRC12:20
*** matsuhashi has joined #openstack-swift12:25
*** ppai has quit IRC12:40
mjsegercreiht: cschwede: good morning12:44
cschwedemjseger: good morning!12:44
mjsegerI've been thinking a little more about why we're seeing differences and wonder if it might be configuration/version related?12:44
cschwedeversion related? you mean swift itself?12:48
tristanCmjseger: I gave a try and found similar results between 1.9 and 2.0. What is the worst case scenario in your environment ?12:52
mjsegerswift and others things.  for example we have separate physical proxy and object servers12:53
mjsegerand we run container services on the object servers12:53
tristanCmjseger: I also experiments with requests.Session, could you try to apply this patch: ?12:55
mjsegerunfortunately I don't really have any control over the software itself so I'm going to have to ask someone else to deploy that and get back to you.  sorry...12:56
tristanCmjseger: it's for swiftclient12:56
*** nosnos has quit IRC12:56
mjsegerhmm, when I try to access that page I'm getting an application error/page not found.  I am logged in.12:58
cschwedetristanC: is that patch marked as draft? can't access it...12:58
tristanCoups sorry, I put it in review mode, it should be ok now12:59
*** koolhead17 has quit IRC13:01
*** koolhead17 has joined #openstack-swift13:01
*** mlipchuk1 has quit IRC13:03
*** matsuhashi has quit IRC13:08
*** matsuhashi has joined #openstack-swift13:08
*** matsuhashi has quit IRC13:08
*** Anju has joined #openstack-swift13:10
*** saurabh_ has joined #openstack-swift13:10
cschwedetristanC: -> ~10% faster for puts13:15
tristanCThat's good to hear... I'm having a hard time to get consistent results between two batch of tests on the same client version13:22
cschwedetristanC: yepp, me too, but then i increased the number of requests to 1000 and now the results are quite consistent13:25
*** Anju has quit IRC13:26
*** saurabh_ has quit IRC13:27
*** pconstantine has quit IRC13:33
*** mmcardle has quit IRC13:36
*** mmcardle has joined #openstack-swift13:36
*** mmcardle has quit IRC13:41
mjsegertristanC: I finally got around to that patch and it does seem to make a BIG difference.  I'm going to run some longer tests to make sure...13:41
tristanCmjseger: cool! thanks you!13:43
mjsegerbut wait, there's no joy...  I ran a 30 second test and it's slow again.  seems like those few at a higher rate weren't representative13:44
mjsegerthe real problem with this sort of thing is performance does vary a lot from run to run.  now I'm doing some quick 100 object puts and they are performign well so maybe it was the 30 second test that wasn't representative13:45
tristanCmjseger: well even on a local vm (devstack +, results vary a lot, sometimes better, sometimes worst13:47
mjsegercontinuing to test and things are looking brighter ;)13:47
*** Anju has joined #openstack-swift13:47
*** saurabh_ has joined #openstack-swift13:48
mjsegertristanC: so in our configuration, with 1.9 I tend to see about 20 IOPS for 1k puts and almost 40 for 2K.  using your patch for 2.0 I'm seeing numbers for both in the mid 30s so I think I'm happy13:49
mjsegerright now I'm runniing my 30 second test 5 times (another switch lets me do that ;)) and I'll post the results when it completes13:50
mjsegerbtw I'm not bothering with unique container names, not because I don't want to use them but because I forgot to add the extra switch13:50
tristanCmjseger: my initial guess is that if you reuse the http_conn object it might improve performance (as session force connections pooling)13:51
mjsegernot sure I understand.  I get a connection and then do all my puts on that connection.  is that what you mean or something else?13:52
tristanC(compared to a CLI usage where you do a 'put' per swift instance)13:53
mjsegerI assume you're referring to things like curl, right or sticking the swift command in a loop or something like that?13:54
mjsegera real no-no13:54
mjsegercheck this out: I'm a happy camper!  this is with patch 7444413:55
mjsegertristanC: have you found getput useful?13:57
tristanCmjseger: indeed it is useful to have a test to refer to for benchmarks14:01
tristanCmjseger: there is a meeting tonight for swift and I guess we are going to give swiftclient performance a topic14:02
mjsegerif you get a chance, you shoudl have a look at the readme as it explains [i hope] how to set up suites of tests for running on parallel client14:02
*** mmcardle has joined #openstack-swift14:04
mjsegertristanC: what are the meeting coordinates?  is it a conference bridge?14:04
tristanCmjseger: it's schedule at 19:00 UTC every wednesday, see
*** pconstantine has joined #openstack-swift14:05
mjsegertristanC: ahh, so this is entirely on IRC then.  I think I can make it14:07
*** saju_m has quit IRC14:11
*** Midnightmyth has quit IRC14:23
*** dmsimard has joined #openstack-swift14:31
*** russellb has joined #openstack-swift14:47
*** mlipchuk has joined #openstack-swift14:56
*** lpabon has joined #openstack-swift15:00
*** zaitcev has joined #openstack-swift15:01
*** ChanServ sets mode: +v zaitcev15:01
*** Midnightmyth has joined #openstack-swift15:02
*** mkollaro has quit IRC15:02
*** mkollaro1 has joined #openstack-swift15:02
*** tongli has joined #openstack-swift15:05
*** openstackgerrit has joined #openstack-swift15:09
*** jergerber has joined #openstack-swift15:16
*** haomaiw__ has quit IRC15:17
*** zackf has joined #openstack-swift15:31
*** mkollaro has joined #openstack-swift15:32
*** haomaiwa_ has joined #openstack-swift15:32
*** otoolee has joined #openstack-swift15:33
*** mkollaro1 has quit IRC15:34
*** haomaiw__ has joined #openstack-swift15:37
*** haomaiwa_ has quit IRC15:39
openstackgerritLuis de Bethencourt proposed a change to openstack/python-swiftclient: make the help strings constant
*** judd7 has joined #openstack-swift15:42
*** zackf has quit IRC15:44
*** fbo is now known as fbo_away15:50
luisbgmorning :)15:54
notmynamegood morning everyone15:56
*** joeljwright has quit IRC15:58
portantenotmyname: morning15:59
luisbgnotmyname, morning :)15:59
*** mlipchuk has quit IRC16:00
*** otoolee1 has joined #openstack-swift16:02
notmynamewhat's new in the land of swift?16:03
portantenotmyname: did some work on an optimization for the auditor to reduce disk-head seeks16:03
*** otoolee1 has left #openstack-swift16:04
notmynameah, cool16:04
portantebut have not had time to setup a good test environment to verify it actually works16:04
notmynameis it simple enough for creiht to like? ;-) (sorry creiht, couldn't resist)16:04
portantethis is a play that comes from experience with a customer having a similar problem16:05
portantecreiht: I am not going to propose it until I can show it works in my little test environment. :)16:05
notmynamethat's always a good place to pull requirements from :-)16:05
notmynameportante: so I'm going to be in your area next week. how many toes am I going to lose to frostbite?16:06
*** lpabon has quit IRC16:06
*** lpabon has joined #openstack-swift16:06
portantenotmyname: it supposed to get to the 50s over the next copule of days, so bring more of that with you!16:06
creihtthat's the type of auditor optimizing that I *want* to see :)16:06
portantecreiht: did any more work on the bulk stat stuff proceed16:06
creihtbulk stat stuff?16:07
notmynamedfg's patch?16:07
portanteusing libxfs or something, bulk stat ioctl16:07
creihtI have no idea16:07
portantemaybe redbo worked on it at one point?16:07
luisbgnotmyname, I am currently writing the media player simple web to play with the API, and sent a patch to fix some style/consistency issues16:07
creihtI know he has played around with it, but don't think he every got to anything official16:08
luisbgnotmyname, since you asked "what is new?"16:08
notmynameluisbg: thanks. how's the media player project going?16:08
luisbgnotmyname, learning how to do all I need with the API in efficient ways, by reading a lot of python-swiftclient code16:08
notmynamecreiht: FYI I want to cover project IPA at the meeting today. if we as a group are pretty good with it, then I'll start talking about it with -infra. I'm imagining that there will be at least one more patch set to include comments/docstrings with stull like wkelly mentioned in his review (ie just like we have in swob and memcache)16:09
luisbgnotmyname, but going well, got a bunch of albums in the storage, and I can retrieve them to play them with gstreamer16:09
notmynameluisbg: cool16:09
redbothe who with the what?  I have a python interface to bulkstat that we use sometimes.16:09
creihtnotmyname: k16:10
luisbgnotmyname, and now thinking about what features could make the experience better16:10
luisbgnotmyname, reading python-swiftclient code made me think though I should release this media player as a sample swift app, for well, having at least one sample app people can play with16:10
luisbgproof of concept, demo, etc16:11
notmynameI'd love to see the DiskFile stuff extended to include a full abstraction of the disk (ie replication and auditing too). that way we could have an xfs one that's more efficient and also ones for otherFS16:11
notmynameluisbg: that would be great16:11
*** tanee is now known as tane-away16:13
luisbgnotmyname, :) doing it with pygtk now, since I had a simple python music player with that I co-wrote for a proof of concept a year ago16:14
luisbgnotmyname, but ideally I should make it a html5 app, right?16:14
portantenotmyname: let me know when you are in town16:17
redboIn theory you could "pip install xfs", then "xfs.bulkstat(mountpoint)".  I need to figure out how to make the .c file get packaged in the .tar.gz so cython isn't required to build it.16:18
notmynameportante: going to be a whirlwind trip. fly into Hartford on Tuesday, NYC on Wed, and Philly on Friday16:18
portanteredbo, we can manually define the data structures in python16:18
portantenotmyname: you folks from large states, think that is considered "in your area"? :)16:19
redboand hope they never change16:20
portanteredbo, and then just make the ioctl calls directly maybe?16:20
portantewell, there is that ...16:20
notmynameportante: heh. pretty much. it's all "somewhere up north, about 30 minutes for 6 different states" to me ;-)16:20
redboI'm sure you could do it in pure python.  But it'd be pretty ugly and you'd have to hope constants and data structures don't ever change.  I'm fine with having my little binary module.16:22
*** joeljwright has joined #openstack-swift16:28
*** chuck__ has joined #openstack-swift16:29
portanteredbo: have you made that bulk stat work available somewhere?16:29
portantegreat, thanks16:30
*** joeljwright has quit IRC16:39
*** gyee has joined #openstack-swift16:43
*** kris_h has joined #openstack-swift16:47
*** byeager has joined #openstack-swift16:48
*** byeager has quit IRC16:55
*** nacim has quit IRC17:05
*** jsuchome has quit IRC17:05
*** Dharmit has quit IRC17:07
*** zul has quit IRC17:08
*** chuck__ has quit IRC17:08
notmynameswift team meeting is in about 2 hours in #openstack-meeting. I just updated the agenda
*** zul has joined #openstack-swift17:09
*** pberis_ has joined #openstack-swift17:12
*** pberis_ has quit IRC17:14
*** mjseger has quit IRC17:15
*** asselin_ has joined #openstack-swift17:15
*** asselin_ has left #openstack-swift17:17
*** chandankumar_ has joined #openstack-swift17:24
*** chandankumar_ has quit IRC17:27
*** ccarrizo has joined #openstack-swift17:31
*** gyee has quit IRC17:33
*** joeljwright has joined #openstack-swift17:35
*** shri has joined #openstack-swift17:36
*** joeljwright has quit IRC17:40
*** mvenesio has joined #openstack-swift17:41
*** basha has joined #openstack-swift17:48
notmynameswift uncategorized errors in jenkins jobs:
notmynameclarkb: ^^ there are 3 there that look really weird to me, and most of the rest are "keystone didn't start"18:01
*** tong_ has joined #openstack-swift18:05
*** tongli has quit IRC18:07
zaitcevnotmyname: What's the situation with swiftclient in the master nowadays? I remember I missed some kind of error when I reviewed the port to python-request, and I'm not talking about foo(..., files=content) that ate memory. We had some kind of worse problem that you apparently successfuly handled, but I completely blank on what it was.18:08
notmynamezaitcev: 2 things: the bin script didn't properly plumb through --insecure for v2, so it was broken against self-signed certs via the CLI ( worked). the other was that auth v1 just plain didn't work (client or CLI) with the --insecure setting. it wasn't plumbed through at all, so you couldn't do anything there18:10
zaitcevnotmyname: Argh, sorry. One visit to "git log" and I have my answers... I was Tristan's fix, not yours.18:11
zaitcevnotmyname: V.sorry for bothering you, thanks for clarification.18:11
notmynameyes, "argh" was close to how I reacted ;-)18:11
notmynamezaitcev: no bother. that's why I'm here :-)18:11
zaitcevyeah and it's like people who refuse to google before asking. kinda paniced here18:11
*** chandan_kumar has quit IRC18:12
*** basha has quit IRC18:13
notmynamechmouel: when can we get tom on openstack reactions?
pberisQuick Q regarding object deletion:  At what point is the object's entry in the containerdb deleted?18:13
notmynamepberis: when the container server gets the delete (pedantic, but let me explain)18:14
notmynamepberis: the object server will send a request to the container server after it deletes the object locally. but if that request fails, it will be queued for resending later. but also note that other replicas may have succeeded and replication may have propegated18:15
zaitcevnotmyname: actually err, my question was different... *facepalm* I meant to ask how confident we are in current master that uses python-requests, now that 3 issues are fixed (Config-Length for Glance, auth v1 plumbing, files= to data=)?18:16
notmynamezaitcev: I'm confident in the current state of python-swiftclient 2.0.2 (wrt basic functionality support and the new feature of cert checking)18:18
pberisnotmyname: So, the entry in the container is flagged as deleted until the first copy is physically deleted at which time the object server notifies the container ... regardless of presence of other copies? ... is this method applicable to containers removed from accounts as well?18:19
*** mmcardle has quit IRC18:20
notmynamepberis: no. the object is deleted. then the listing is updated (ie removed). if the listing update fails, it is retried again later. replication is also running in the background, so if one container replica gets the update, it may propagate before the retry happens18:22
*** gvernik_ has joined #openstack-swift18:22
*** gvernik_ has quit IRC18:23
notmynamepberis: I'm including that last detail simply to complete the picture. there are 3 ways a container listing gets updated: (1) synchronously with the request (2) asynchronously later (3) via replication, if another replica had methods 1 or 2 succeed18:23
*** gvernik_ has joined #openstack-swift18:24
pberisnotmyname: OK ... got it.  I gather that a similar process is used for removing the container listing from the accountdb ... bottom line, the lists contained in the containerdb and accountdb should not contain deleted instances, at least not for any extended period.  Correct?18:25
pberisnotmyname: IE entries are not simply flagged as deleted and left in the db ...18:29
notmynamecorrect (with handwavey about "any extended period")18:29
notmynameIIRC. (I should go look at that code)18:29
pberisnotmyname: Guess it depends on what your definition of 'extended' is ..18:29
pberisnotmyname: Thanks!18:32
*** krtaylor has quit IRC18:34
*** krtaylor has joined #openstack-swift18:36
*** joeljwright has joined #openstack-swift18:36
*** donagh_ has joined #openstack-swift18:37
openstackgerritAlistair Coles proposed a change to openstack/swift: Fix invalid account acl generating 500 response.
*** donagh has quit IRC18:39
*** donagh_ has quit IRC18:40
*** donagh has joined #openstack-swift18:40
*** joeljwright has quit IRC18:40
*** dtalton has joined #openstack-swift18:42
*** csd has joined #openstack-swift18:45
zaitcevdo we have a meeting in #openstack-meeting in a few minutes?18:51
notmynamezaitcev: yes18:52
notmynamepberis: looking through the code, I'm reminded of one other thing there. the container listing entry isn't immediately deleted. a "tombstone" row is inserted. I'm looking for when that is removed18:53
*** joeljwright has joined #openstack-swift18:55
notmynamepberis: the tombstone row is removed by replication after reclaim_age seconds have passed (defaults to 7 days)18:57
notmynamepberis: FWIW, this is exactly the same pattern as is used by objects themselves ("delete" by writing a tombstone, then remove the tombstone after a set time, default one week)18:58
notmynameok, meeting time19:00
notmynameportante: around for the meeting?19:04
openstackgerritAlex Pecoraro proposed a change to openstack/swift: Allow hostname for nodes in Ring
cschwedetorgomatic: just thinking if it might be useful to include it in the container updater (to avoid walking all containers twice). well, i'll read your gist first and then start thinking :)19:14
torgomaticcschwede: imagine that you find 20 GB worth of misplaced objects; that'll put a nice big pause in your container update run :)19:14
torgomaticI'm not opposed to the idea though19:15
creihtwhat is the likely hood of this happening?19:15
cschwedetorgomatic: ok, got it, makes a lot of sense to separete in that case :)19:15
torgomaticcreiht: low19:15
torgomaticso maybe it should be all one daemon?19:15
* torgomatic hasn't given this question enough thought19:16
creihtjust seems like a lot of work and stuff going on to catch something that isn't very likely to happen19:16
creihtor fix19:16
creihtneed to think about it a bit19:16
creihttorgomatic: would it be better to have something that would report when this happen, then have a tool that can migrate objects from one policy to another?19:21
torgomaticcreiht: I'm partial to automatically fixing it; if you've got a good ops team then moving stuff manually is fine, but not everyone has a good ops team19:22
creihttorgomatic: i would be for that if it were something that is likely to happen19:23
omameI take it as a compliment :)19:23
torgomaticcreiht: I think it's more likely to happen in multi-region clusters19:25
torgomaticthose slow flaky WAN links can give you little mini-splits all day long19:26
*** judd7 has quit IRC19:26
gholtIsn't .99999 one in 100,000 not a million, literally? On a serious note though, what is done about container servers arguing about what the storage policy is?19:27
creihttorgomatic: so what's bugging me a bit, is what is the use case or likely hood that a user (in a short period of time) is going to create the same container more than once on the separate network partitions with different storage policies?19:28
torgomaticgholt: bah, not enough nines ;)19:28
torgomaticthere's 1000 milliseconds per second and 1000 microseconds per millisecond, so it should work out to a million if I can do math :)19:29
portantecreiht: I am not sure it is the case where somebody tries to make that happen, it seems to be more about the case where a mistake is made and then one can't recover19:29
gholtI can't really do math. I ask Google for help all the time.19:29
*** judd7 has joined #openstack-swift19:29
creihtportante: well that's what I'm trying to figure out is exactly what would have to happen to trigger this19:29
creihtI'm not saying that a user purposely is trying to cause an issue19:30
torgomaticgholt: okay, I added more nines (if only it were always so easy...)19:30
gholtI'm trying to figure out how you have two different answers as to what the storage policy is for a container, and how that is resolved, and why that can't also start resolving misplaced objects.19:30
torgomaticgholt: hang on, lemme find the transcript... it involves a network partition19:30
torgomaticstep 1: partition your cluster by hosing the network19:31
torgomaticstep 2: on side A, create a container in policy 1 and upload an object19:31
torgomaticstep 3: on side B, create the same container with policy 2 and upload an object19:31
torgomaticstep 4: unhose the network19:31
gholtYeah, so don't the container servers replicate and figure out the "right" answer for the policy for the container? How?19:32
torgomaticgholt: there's a proposed patch for that, but I think it's gone stale... it's basically like they do the metadata, but the older answer wins (not the newer)19:33
torgomaticthe idea is that the older copy probably has more objects in it, so it's less crap to move doing it that way19:33
gholtI guess what I'm driving at is when that resolution occurs, you magically know you need to fix objects; no need to scan.19:34
torgomaticI'm sort of concerned about what happens if you crash midway through resolving stuff though; either you end up repeating work, or objects get lost19:36
torgomaticputting it in the container updater would at least save Swift from a new filesystem walker, and in the no-resolving-needed case it's just a read of container_stat (like it's already doing)19:37
openstackgerritpaul luse proposed a change to openstack/swift: Add Storage Policy Support to Account HEAD
pelusetorgomatic:  ^ done :)19:37
torgomaticpeluse: cool; will look soon19:37
gholtYeah, kinda still reading the gist.19:41
gholtI'm not sure anything of this would be what the user wants. :)19:41
torgomaticgholt: the user just wants their objects back, so we can't lose them19:42
torgomaticthat's the ultimate goal here: don't lose objects19:42
gholtYeah, but one user wanted one thing for objects and the other wanted another thing.19:42
creihtwell if the storage policy was to /dev/null storage, then they are out of luck ;019:43
gholtIf there was a way to just leave them separate and have the user resolve the issue, that'd be pretty nice.19:43
gholtI'm talking out of ignorance here, but one ends up the default and the other can be selected by header or something.19:43
torgomaticas soon as you ask the user if they want policy 1 or 2, they'll ask for policy 7, and they'll want it in a periwinkle blue ;)19:45
torgomaticI don't think we can rely on users to fix anything; mapping a container back to a human isn't always easy, and getting that human to do something is hard19:46
* gholt wishes we reserved space to embed more things into the url path.19:48
gholtBut I'm evil that way, push things off to the data owner/creator.19:49
torgomaticgholt: it's awesome when it works :)19:49
torgomaticreq.path_info = base64.encode(json.dumps({'account': 'bob', 'container': ...19:49
gholtI didn't catch what you guys do with conflicting obj names19:50
torgomaticgholt: newest timestamp wins, same as before19:50
gholtCool, nothing special there, just special for the policy, makes sense. I don't like the extras that entails, but I got nothing better.19:56
torgomaticI'm not especially enthused about all this either, but this policy stuff opens up a new consistency-failure mode19:58
*** gvernik_ has quit IRC19:58
torgomaticIf you have a way that I could avoid writing all this code, I'm all ears :)19:59
openstackgerritpaul luse proposed a change to openstack/swift: Add Storage Policy Support to Account HEAD
pelusetorgomatic:  for real this time :)20:03
torgomaticpeluse: :)20:03
notmynameFYI if you didn't see links for the reconciler gist or the swiftclient post-mortem, they're on the minutes
zaitcevwhat if NTP goes mad on you and a node creates a bunch of junk with very fresh timestamps20:07
zaitcevam I overthinking20:07
*** Trixboxer has quit IRC20:08
* notmyname steps away for a bit20:08
zaitcevI just had something untarred and it cropped a bunch of "timestamp in the fuuuuuture!!" (time zone was incorrect somewhere)20:08
zaitcevI understand we do not support resistance against Byzantine failures, but I was just wondering if anyone at RAX ever had comething like this screwed up by mistake.20:10
gholtOnly a little, since we accept x-timestamp headers users can override anything20:21
gholtSo that can cause some interesting scenarios20:21
*** foexle has quit IRC20:23
creihtzaitcev: so I saw your comment in the code, and I'm tending to believe that ideally the guru stuff should be unobtrusive enough that it is always running20:45
creihtand not require a config var20:45
zaitcevright... otherwise Murphy says you won't have it when you really need it20:46
zaitcevI was on the fence about it but I'll drop it now.20:47
*** joeljwright has quit IRC21:00
*** dtalton has left #openstack-swift21:03
*** csd has quit IRC21:06
*** mvenesio has quit IRC21:07
*** joeljwright has joined #openstack-swift21:09
*** csd has joined #openstack-swift21:18
*** cschwede_ has joined #openstack-swift21:20
*** cschwede_ has quit IRC21:21
*** keving has joined #openstack-swift21:29
*** fbo_away is now known as fbo21:34
creihtzaitcev: so I'm trying to enable the guru stuff on my saio21:36
creihtin one of my object server confs, I added21:36
creihtgurumed = yes21:37
creihtgurumed_dir = /var/lib/swift/dump21:37
creihtunder DEFAULT21:37
creihtthen I do21:37
creihtkill -s SIGUSR1 PID21:38
openstackgerritVyacheslav Rafalskiy proposed a change to openstack/swift: Add support for reading from archives
creihtbut I don't get a dump21:38
creihtI see Removing dead child and Started child in the logs21:38
creihtam I missing something?21:38
*** tong_ has quit IRC21:40
zaitcevwell... I only added hooks to small set of problematic daemons. Those that don't have the hook will die on SIRUSR1.21:42
zaitcevcreiht: So, the only thing that works is  swift-init object-replicator dump21:43
creihtahh ok21:43
zaitcevI'm sorry, I'll add hookage across the board.21:43
creihtI think that will be useful21:44
creihtif all of this works21:44
zaitcevyeah... I thought maybe find a common place instead of having 21 identical hook21:44
openstackgerritFabien Boucher proposed a change to openstack/python-swiftclient: Allow get_account and get_container to return an iterator
luisbgpython-swiftclient in Jenkins has the python3.3 builds fail with: FileNotFoundError: [Errno 2] No such file or directory: './subunit_log.txt'21:51
luisbgwhy is that?21:51
luisbgthey are marked as (non-vote) so builds still pass, but wondering why those tests are broken21:51
torgomaticluisbg: probably because swiftclient isn't py3 compatible21:52
torgomaticmy guess is it doesn't even compile, so there's no chance for the testrunner to do anything, including write a log21:52
*** foexle has joined #openstack-swift21:53
luisbgtorgomatic, ooooh, that makes sense, didn't knew21:53
luisbgtorgomatic, and I guess Jenkins needs something, even if it setup to ignore it21:53
torgomaticfolks are now working on making it py3 compatible, but it's not done yet AFAIK21:53
*** cschwede_ has joined #openstack-swift21:57
notmynametorgomatic: FYI on we don't currently use six at all, so this adds a test dependency (that's not referenced in test-requirements)22:02
torgomaticnotmyname: ...then how do the tests pass?22:02
torgomaticwhat are they importing?22:03
notmynameyou already have it installed?22:03
notmynameis six in the stdlib?22:03
torgomaticmaybe...? what about the Jenkins stuff?22:03
notmynameya, because jenkins uses the global requirements (instead of what the project says)22:03
torgomaticall the check jobs passed, so they must have done something reasonable when processing the "import six" directive22:03
notmynamekinda like it always uses the tip of master for client libs, even if the requirements have a version cap (ie the problems they had with swiftclient in grizzly)22:04
torgomaticsure, what could POSSIBLY GO WRONG22:04
gholtThere is no tip, this is git. ;)22:04
notmynamefrom Guido: " That's the problem with Twitter. You must lie to fit in 140 chars."22:13
luisbgnotmyname, van Rosum?22:13
notmynamethe one and only22:13
luisbgnotmyname, :)22:13
luisbgnotmyname, there is also this:
luisbgGuido van Robot, haha22:14
*** foexle has quit IRC22:33
*** cschwede_ has quit IRC22:44
*** foexle has joined #openstack-swift22:57
*** byeager has joined #openstack-swift23:02
*** fbo is now known as fbo_away23:02
*** foexle has quit IRC23:05
*** joeljwright has quit IRC23:06
*** foexle has joined #openstack-swift23:06
*** foexle has quit IRC23:12
*** foexle has joined #openstack-swift23:12
*** foexle_ has joined #openstack-swift23:16
openstackgerritA change was merged to openstack/python-swiftclient: Remove useless statement
*** Midnightmyth has quit IRC23:18
*** asselin_ has joined #openstack-swift23:20
*** asselin_ has left #openstack-swift23:21
*** dmsimard has quit IRC23:23
*** foexle_ has quit IRC23:24
*** foexle has quit IRC23:25
*** csd has quit IRC23:29
*** openstack has joined #openstack-swift23:34
*** kris_h has quit IRC23:41
*** jergerber has quit IRC23:46
*** mkollaro has quit IRC23:47

Generated by 2.14.0 by Marius Gedminas - find it at!