Friday, 2016-09-02

*** arch-nemesis has quit IRC00:03
*** diogogmt has joined #openstack-swift00:43
*** Jeffrey4l has joined #openstack-swift00:46
*** m_kazuhiro has joined #openstack-swift00:51
kota_good morning01:09
timburkegood morning kota_01:12
mattoliveraukota_: morning01:15
kota_timburke, mattoliverau: hello :-D01:15
kota_timburke: thanks for reviewing for some my patches, will take a look to respond the comments01:16
* kota_ is wondering if patchbot is back?01:17
kota_patch 21916501:17
patchbothttps://review.openstack.org/#/c/219165/ - swift - EC Fragment Duplication - Foundational Global EC C...01:17
kota_hello patchbot. thanks for working01:18
kota_ok, patchbot is now working01:18
timburkehuh. so http://paste.openstack.org/show/565924/ seems like a totally reasonable change, right? but it makes .unittests spew a whole lot of "Exception AttributeError: AttributeError("'NoneType' object has no attribute 'replace'",) in <generator object _get_response_parts_iter at 0x7f5a79c7c500> ignored" warnings :(01:33
timburkeeh, i'll look into it more tomorrow01:33
*** gyee has quit IRC01:47
kota_timburke: it looks like swift3 + Swift master thing?01:49
*** nadeem has joined #openstack-swift01:49
openstackgerritKazuhiro MIYAHARA proposed openstack/swift: Fix api reference of object GET request with Range parameter  https://review.openstack.org/36410401:55
openstackgerritKazuhiro MIYAHARA proposed openstack/swift: Fix api reference of object GET request with Range parameter  https://review.openstack.org/36410401:59
*** chlong has joined #openstack-swift02:05
*** nadeem has quit IRC02:14
*** tqtran has quit IRC02:15
*** dmorita has quit IRC02:26
*** nadeem has joined #openstack-swift02:29
*** asettle has joined #openstack-swift03:05
*** dmorita has joined #openstack-swift03:06
*** dmorita_ has joined #openstack-swift03:08
*** dmorita has quit IRC03:08
*** asettle has quit IRC03:10
*** dmorita_ has quit IRC03:13
*** m_kazuhi_ has joined #openstack-swift03:22
*** m_kazuhiro has quit IRC03:23
*** jith_ has quit IRC03:34
*** diogogmt has quit IRC03:41
*** links has joined #openstack-swift03:46
*** jith_ has joined #openstack-swift03:47
*** chlong has quit IRC04:01
mahaticzaitcev: hello, are you here?04:03
*** jith_ has quit IRC04:04
zaitcevmahatic: Yes04:04
zaitcevDid you think about checking the relcaim time before trashing the hash?04:05
mahaticyes, was fetching a link04:06
mahaticzaitcev: https://github.com/openstack/swift/blob/master/swift/obj/diskfile.py#L85304:06
mahaticget_hashes calls this, which checks for reclaimable age04:06
mahaticand only then marks it obsolete04:07
zaitcevLet me see, I may be missing something.04:07
mahaticsure np04:08
*** dmorita has joined #openstack-swift04:10
zaitcevSo, how is it invoked? I don't see a path to it from get_hashes, which is what is executed on the context of the auditor.04:11
mahaticzaitcev: https://github.com/openstack/swift/blob/master/swift/obj/diskfile.py#L103804:12
mahaticwhich invokes https://github.com/openstack/swift/blob/master/swift/obj/diskfile.py#L251204:12
mahaticand in turn calls https://github.com/openstack/swift/blob/master/swift/obj/diskfile.py#L91504:13
*** dmorita has quit IRC04:14
*** chlong has joined #openstack-swift04:18
zaitcevThis looks worse than I thought.04:23
zaitcevSo we get PathNotDir and the key is dropped from the hash, and then we call ourselves.04:23
zaitcevBut do you know why is it that hash is missing for the deleted objects?04:24
zaitceve.g. why does this fire https://github.com/openstack/swift/blob/master/swift/obj/diskfile.py#L103504:24
zaitcevoh, wait04:26
zaitcev(suffix, None), right04:26
mahaticright04:28
zaitcevmahatic: I think I see now how this is supposed to work, but I'm not comfortable with the performance impact. We need someone like clayg on this. He has guinea pigs.04:29
*** psachin has joined #openstack-swift04:32
mahaticzaitcev: I agree. Well, there was a performance impact issue with a previous way of implementation too (remove .ts or invalidate hashes) but kota_ and I thought this was better04:33
mahaticclayg: your insights please04:34
*** sanchitmalhotra1 has joined #openstack-swift04:35
*** csmart has quit IRC04:35
zaitcevmahatic: What jumped on me is, DiskFileDeleted already has the metadata, which one can use to make the determination if reclaim time is reached. But in this patch we discard this knowledge unconditionally and, on _every auditor pass, for every tombstone_ dive right into _get_hashes and supply the 1-element revalidate list.04:35
zaitcevsorry, recalculate= list04:36
*** sanchitmalhotra has quit IRC04:36
*** sanchitmalhotra1 is now known as sanchitmalhotra04:36
*** ppai has joined #openstack-swift04:41
mahaticzaitcev: hmm yeah we could call get_hashes conditionally i.e. depending on reclaim_age. But I need to think if there was a reason this was missed in the previous patch :)04:41
mahaticother than oversight04:42
zaitcevSo, the question is: how loaded are auditors, and is it common that they can't keep up? And in addition, how common are tombstones, is it enough to make a marginal auditor to boggle down and not keep up anymore. It's possiby I'm concerned over nothing, just my overactive imagination.04:42
*** nadeem has quit IRC04:54
*** nadeem has joined #openstack-swift04:55
mahaticI'm not too inclined on the "auditor not keeping up" part. Would be helpful if someone with real prod environments could assert that04:56
zaitcevyeah05:00
*** ChubYann has quit IRC05:00
zaitcevOf course we could just approve this patch, and when people get auditors bogged down, add one little if to it.05:01
zaitcevNot sure how responsible this course of action is...05:01
mahatic:) adding a little if now, doesn't seem a problem05:02
*** hosanai has quit IRC05:04
-openstackstatus- NOTICE: The logs volume is full causing jobs to fail with POST_FAILURE. This is being worked on, please do not recheck until notified.05:11
*** ChanServ changes topic to "The logs volume is full causing jobs to fail with POST_FAILURE. This is being worked on, please do not recheck until notified."05:11
*** philipw has quit IRC05:12
*** chlong has quit IRC05:27
*** ChanServ changes topic to "Topic: Let's talk. We're nice. | Ideas: https://wiki.openstack.org/wiki/Swift/ideas | Logs: http://eavesdrop.openstack.org/irclogs/%23openstack-swift/ | Meetings: https://wiki.openstack.org/wiki/Meetings/Swift"05:37
-openstackstatus- NOTICE: Space has been freed up on the log server. If you have POST_FAILURE results it is now safe to issue a 'recheck'05:37
*** philipw has joined #openstack-swift05:42
*** chlong has joined #openstack-swift05:47
*** csmart_ has joined #openstack-swift06:00
*** csmart_ is now known as csmart06:02
*** ttrumm has joined #openstack-swift06:02
openstackgerritMahati Chamarthy proposed openstack/swift: Delete old tombstones  https://review.openstack.org/34686506:18
openstackgerritChristian Schwede proposed openstack/swift: Deprecate swift-temp-url  https://review.openstack.org/34817306:20
*** _JZ_ has quit IRC06:23
*** nadeem has quit IRC06:23
*** sanchitmalhotra has quit IRC06:31
*** sanchitmalhotra has joined #openstack-swift06:32
*** hosanai has joined #openstack-swift06:36
*** ChanServ sets mode: +v hosanai06:36
*** SkyRocknRoll has joined #openstack-swift06:39
* mattoliverau is calling it a day.. its friday and I started early. Have a great weekend all. 06:40
*** rcernin has joined #openstack-swift06:40
claygredbo: glange: can you guys teach nadeem and dfg how use a bouncer or irssi or something?06:40
claygnadeem: dfg: can i get some help reviewing patch 346865 to free me up to look at patch 362200 and patch 36407506:42
openstackgerritMerged openstack/swift: Remove meaningless lines in reconciler unit test  https://review.openstack.org/36426906:42
patchbothttps://review.openstack.org/#/c/346865/ - swift - Delete old tombstones06:42
patchbothttps://review.openstack.org/#/c/362200/ - swift - don't wait on remote container updates with write ...06:42
patchbothttps://review.openstack.org/#/c/364075/ - swift (feature/hummingbird) - go: move REPCONN and REPLICATE call from object se...06:42
*** geaaru has joined #openstack-swift06:44
*** hosanai has quit IRC06:44
claygwhy does no one believe me that it's totally reasonable to just delete old tombstone hashdirs and not invalidate the suffix in the process?06:44
clayglike is it just cause right now the code that cleans tombstones older than reclaim age is mixed in with the code that invalidates the suffix?  or like are we worried we're going to break replication?06:44
mahaticclayg: "like is it just cause right now the code that cleans tombstones older than reclaim age is mixed in with the code that invalidates the suffix?"06:45
mahaticthat's the one06:45
claygnot that there would be anything to "replicate" except possibly some other jerk off that had managed to reap his reclaimable tombstone?06:45
claygacphacphacpcahcpah - but but but06:46
clayg;)06:46
openstackgerritMerged openstack/swift: Remove unnecessary variable  https://review.openstack.org/35640206:46
ahale_"are we worried we're going to break replication?"06:47
ahale_hehehe06:47
claygwhy is all this code merging?  did mattoliverau go a review bender?06:47
claygahale_: hey!  repconn still uses hash suffixes!  it's not that *that* funny ;)06:47
clayg(it's kinda funny)06:47
ahale_oh hah yeah no i meant the thought that replicatioin might not already be broken !06:47
* clayg snickers06:48
*** pcaruana has joined #openstack-swift06:48
claygahale_: you watch man mahatic's gunna get all those old tombstones cleaned up - you're gunna have so many free inodes - your filesystems are gunna purr like kittens06:48
*** chlong has quit IRC06:49
ahale_:D thats gonna be awesome06:49
mahaticclayg: one more +2 and that's what gonna happen ;)06:49
mattoliverauclayg: lol, I got tired of looking at container replication and sharding so decided to do something else :)06:49
claygahale_: you ever counted how many reclaimable tombstones you have laying around?  Maybe most of your suffixes are dense enough these days it's not a big issue for you (stead traffic turns them over frequently enough)06:49
ahale_i spent a lot of yesterday looking at "really weird partitions" , what i found so far im just gonna blame on XFS under stress I think06:49
clayg... but IME it's an annoying problem06:49
ahale_nope - never looked at that06:50
ahale_but - that bug you posted yesterday with asyncs older than reclaim ... thats an annoying problem we have seen06:50
claygoh yeah totally :'(06:50
claygcharz and hugokuo have some ideas - if you've given it any serious thought you might write down something as well06:51
ahale_i really really hope this is xfs - but i was looking at parts like this - (finding out why out of place ones werent being removed after replication)06:51
ahale_srv/node/c2u13/objects/131123/e02/20033514532f470b528fc344dc943e02/1435610792.18848.data/1407180761.27739.ts06:51
ahale_srv/node/c2u22/objects/756094/dee/b897e90553c9eaf8eaf9a9d5841d8dee/05283a1835164a67bd1400a7ed65418b/1435414158.76132.data06:51
claygcharz: what's the plan again?  do something with 507 to async responses that was for sure - was there something else?  with... once a async PUT get's old enough it might reclaim?06:52
claygahale_: those paths are *epic* - new to me - how does the auditor not quarantine that shit?!06:52
charzclayg: maybe updater needs to check handoff place if there is a container DB, it should update it.06:52
claygoh... me crosses fingers its a hummingbird only bug and dreams of the day we're all pulling in the same direction again06:53
claygcharz: yeah that's he business +206:53
claygwe have so many loops where were iter nodes + handoffs and break after 3 non 507 responses - the updater should totally do that06:54
ahale_i think its a really full disk , xfs breaks down, rather than hbird repli - not totes sure tho06:54
charzAnd maybe updater should show some alert when the async_pending timestamp is too old, but idk how to get container reclaim age.06:54
claygcharz: yes I think definately updater needs to be skeptical putting something in a container that's been failing for a reclaim age - esp if it's alreay got it into the other containers06:55
charzclayg: And I'm investigating why the same object's async_pending will distribute to different nodes.06:56
*** tesseract- has joined #openstack-swift07:02
claygzaitcev: thank you so much for catching the if branch on the reclaim tombstone hash invalidation ;)07:05
kota_back for a short time07:07
charzclayg: or maybe log the time the device was unmounted in somewhere? recon cache? and container replicator can check the time to extended the real reclaim age when the drive are mounted again. not sure is it a good idea.07:07
kota_looking at the log, clayg, zeitcev, mahatic got the answer for reclaim ts?07:07
openstackgerritMerged openstack/swift: Fix api reference of object GET request with Range parameter  https://review.openstack.org/36410407:08
charzclayg: Maybe force updater to check handoff place should able to solve this issue, because you won't see the same async_pending job in next run.07:11
mahatickota_: please take a look at the updated patch07:11
mahaticI'm off to lunch and then a meeting, be back in an hour07:11
kota_mahatic: ok, thanks07:14
claygcharz: I think that's the key - we should make async pending never get to be as old as half a reclaim age without massive warning bells07:15
claygcharz: putting the async into the container layer (on a handoff node when a device is 507'd) is a good way to avoid stupid fire drills07:15
claygkota_: i've been reviewing everyone's knows - I think everyone is doing great - it's a tricky problem07:16
kota_clayg: yeah, it's absolutely tricky. during the reviewing, tbh, I've been feeling I'd like to refactor around there because maybe we could make DiskFile/AuditLocation more familer with current mahatic's solution07:20
kota_but I've hold that on just in myself until mahatic's work completed.07:21
claygacoles_: what is the axiom for orphaned .meta files laying on a .ts?07:21
*** jmccarthy has quit IRC07:21
clayglike once the .ts gets to be reclaimable we have to assume some poor sucker just had a stale .data and accidently accepted the POST?07:22
*** jmccarthy has joined #openstack-swift07:22
claygkota_: FYI some internal intrest building on EC fragment duplication07:23
zaitcevI thought that function deletes all of them... it even tries to remove the directory07:23
zaitcevclean_object_files or what is it07:24
claygit's still "future" for us - but - well luckily for us we know someone that's been living in the future for years!07:24
claygzaitcev: nom nom nom07:24
claygis acoles_ still on vacation?07:24
kota_clayg: i think so07:25
claygi was eating some british candy today - it was called "a crunchie" - how adorable is that!?  "a crunchie" - it's even more fun with an accent07:25
kota_clayg: what's quesiton on EC frag duplication?07:26
claygzaitcev: so the bottom line is that doing a hash recalculation for the entire part - and *forcing* for it for the entire suffix - just because we want to reap an old tombstone is not great07:26
zaitcevclayg: Yes. That was my last objection. But the latest version seems to work, at the cost of an extra time comparison check....07:28
*** urth has quit IRC07:28
zaitcevI don't mean the CPU cost of course, just that the same comparison is made in two unrelated places07:28
claygwell... yeah but I mean... the first time this run when you discover a bunch of these - you're going to turning over file system cache like mad?07:29
zaitcevwell that's true07:30
zaitceveven so, all those listdirs and the like, they aren't entirely free even if they hit cache07:30
zaitcevI think07:31
claygand moving forward - there *has* to be some oppertunity where you'd have two reclaimable suffixes in the same part!  probably sometimes even in the same suffix!07:31
clayg... but we're going to do all that file system work - recalc the whole partiiont (!?) for each one07:31
kota_clayg: yes!!!07:31
claygi mean REPLICATE verbs be flying - if we just invalidate it ... I think that'd be better than doing the recalc optomistically07:32
kota_clayg: I was thinking we could cache the suffixes in the part and just run get_hashes only once07:32
*** urth has joined #openstack-swift07:32
claygkota_: see kota_ gets it07:32
kota_not for each time when we find the reclaimable ts07:32
kota_but i think it requires some cleanups for loop i thought07:33
kota_think twice :/07:33
claygkota_: even if it only helps on the first run when we release the code I think it's worth it to write it - but honestly - well i don't really have a good way to say the "rate" at which reclaimable tombstones "build up"07:33
claygso I don't know how likely I'm going to see multiple reclaimable tombstones in the same part or maybe even same suffix in the period of a auditor zbf scan07:33
clayghell - my zbf scans would probably be a shit ton quicker if I didn't have all these stable tombstones!07:34
claygi swear to god I think we should just unlink the file and see if it doesn't get way better :D07:34
kota_oh, true,07:34
claygwell - i mean i guess I want it to invalidate the hash sort of... eventually... really i just want the hashdir gone :\07:35
claygi'd be ok with unlink the ts and invalidate the hash - pretty sure07:35
claygaside from the scary upfront sweep cleaning up the hole - i think it'd be pretty good going forward - maybe ideal07:36
*** dmorita has joined #openstack-swift07:36
claygas long as the remote replicator doesn't ship stale tombstones :\07:37
claygthat was what the 1.1 was for yeah?07:37
claygmahatic: ok you got me - i'm checking it out :P07:38
*** dmorita has quit IRC07:40
kota_clayg: fyi, you still have interest on ec duplicated fragment, hopefully we works on Optimistic Get toghether to land first07:46
claygi keep forgetting you can't make a tombstone older than reclaim age with a DELETE request07:46
claygkota_: oh right!  yes that would be wonderful to land!07:46
kota_clayg: i'm now reviewing that locally, my plan, land it in 2 weeks and then my EC frag duplication land in the next 2 weeks and then Newton :P07:47
kota_hopefully07:48
claygyeah let's do it!07:48
kota_if you could help me to do that07:48
claygi'd love to see that happen!07:48
* kota_ hug clayg07:49
*** SkyRocknRoll has quit IRC07:50
openstackgerritChristian Schwede proposed openstack/swift: Fix swiftdir option and usage of storage policy aliases  https://review.openstack.org/34469307:54
*** ttrumm has quit IRC07:54
*** ttrumm has joined #openstack-swift07:55
*** ttrumm has quit IRC08:04
*** ttrumm has joined #openstack-swift08:04
*** asettle has joined #openstack-swift08:07
clayg^ it's the most annoying thing I never noticed - I had it that I can't do like swift-get-nodes unless i'm in the cluster08:11
mahaticI'm back08:13
* mahatic is reading the log08:13
mahaticclayg: cool, did you check it out yet?08:13
claygyeah i'm kicking the cobwebs off my tool for poking at the hash suffix api -> https://gist.github.com/clayg/035dc3b722b7f89cce66520dde285c9a08:16
claygi have a reclaimable tombstone hiding behind a valid hash08:16
claygconsidering options on how i'm going to poke it08:17
mahaticnice08:21
clayg... one more option08:22
claygok, so if one node has a clean hash for the suffix (like the tombstone was reclaimed and the part we rehashed including a recalculate for that suffix) ...08:24
clayg... when another node who has a hashed suffix shadowing a reclaimable tombstone goes to *replicate* to clean node ...08:24
claygboth end up clean - which is tons better than it successfully pushing over it's tombstone08:25
claygbut I'm not so sure that it didn't acctually just do that - like I sorta believe it might have rsync'd the zero-byte tombstone - then everyone recalculates the synced suffixes - which finally triggers the clean_ondisk_file for the hashdirs and drops the now empty suffix08:26
mahaticthe current patch change is for auditor08:28
mahaticclayg: did you try the replicate part after reclaim_age08:28
claygaphcaphcv? but it's gunna effect replication08:28
claygthis is where i'm at currently -> https://gist.github.com/clayg/e16199868b3fc75e27c87b6d828a6c3808:29
mahaticokay yeah you said that above08:29
claygthe tombstone is older than reclaim age - it was on 1 & 3 - but I audited 1 and then replicated 308:30
claygi'm sorta surpised a) 3 cleaned it self up just cause 1 was clean and b) that 3 replicating with 4 didn't clean up 408:30
claygyeah... it's stable like this?  i'm guessing 4 would clean himself up now - but 1 or 3 pushing to him don't make him recalculate08:31
mahatichmm 4 would cleanup only when you do an audit i think08:32
openstackgerritChristian Schwede proposed openstack/python-swiftclient: Make options arguments truely optional  https://review.openstack.org/35706308:34
claygmahatic: nope - so it seems that as soon as one node clears the suffix everyone else will follow suit when they talk to him!08:34
*** jmccarthy has quit IRC08:34
claygit's weird tho - because him talking to *them* doesn't appear to do anyhting :D08:34
claygnow for code08:34
clayglove it!  https://github.com/openstack/swift/blob/7ff1055ff3cf3b532ebdc3aed65db31a23b8e801/swift/obj/replicator.py#L45408:36
claygoh nice, then at 466 after it recalculates the suffixes that are different ('mystalehash' != None) it goes again to filter out only the suffixes that are *still* different (after recalc None == None) before calling into rsync08:38
claygoh oh oh - and the reason me pushing to you can't clean you is because I only look at suffixes in my local_hash that are differnt from yours08:40
mahaticyes line 454 thing08:40
claygso when I talk to you having been audited and you respond with 'yourstalehash' it's just not relevent to me because I don't even have that hash - hrmm..08:40
claygthat would be different behavior if there were multiple objects in the same suffix08:41
claygi think I have a script for that08:41
clayggood 'ol find_names_with_same_part_suffix.py -> https://gist.github.com/clayg/590caf4c5b650af950add2aab3943e3608:51
clayghttps://gist.github.com/clayg/e89dce252fcec82cd6a6853910fb3e57, oh it's gunna be *exciting* -> (86400 * 7) - (time.time() - 1472201951.39000) = 7.92830514907836908:59
*** dmorita has joined #openstack-swift09:01
*** ouchkernel has quit IRC09:02
mahaticclayg: how do you put two objects in the same suffix? how are you testing that part I mean09:03
claygok, i updated the gist - which direction is going to be more interesting - the guy that did the reclaim pushing to the guys with the stale suffix?  probably the guy with the stale suffix pushing to the recalimed guy09:05
claygmahatic: so that'd be https://gist.github.com/clayg/590caf4c5b650af950add2aab3943e36 - i just keep cookiing up names until i find two objects (same account container) that just so happen to hash to the same part and suffix!09:06
*** dmorita has quit IRC09:06
clayghrmm... again he just cleaned him self up - the other fool with the stale hashes don't care (?)09:06
clayghrmm... no i gotta do it again09:07
claygthis is also my new favorite "create an old tombstone" -> ''' curl -XDELETE http://saio:8090/v1/AUTH_test/test/obj481 -H "x-timestamp: $(python -c 'import time; print time.time() - 7 * 86400 + 300')" '''09:09
*** ouchkernel has joined #openstack-swift09:09
claygthe trick is of course the noauth server on 8090 with shunt x-timestamp turned off09:09
mahaticclayg: that's a nice script09:13
*** hseipp has joined #openstack-swift09:13
*** ppai has quit IRC09:13
*** dmorita has joined #openstack-swift09:14
claygoic, so it's just non deterministic because of ordering09:15
claygif a stale guy talks to an audited guy he cleans himself up09:15
claygif he talks to another stale guy first then he won't clean him up - if he talks to the audited guy first - now he's a clean guy!  and clean guys always clean up stale guys09:16
openstackgerritChristian Schwede proposed openstack/swift: Fix swiftdir option and usage of storage policy aliases  https://review.openstack.org/34469309:16
claygso the multiple objects in the same suffix is significantly more interesting09:16
claygBUT - i like that we're not shipping around old tombstones!09:16
claygi think it's all just a matter on how much rehashing we do09:17
claygk... this is good09:17
* clayg ponders09:17
*** dmorita has quit IRC09:19
openstackgerritOpenStack Proposal Bot proposed openstack/swift: Updated from global requirements  https://review.openstack.org/8873609:34
*** admin6_ has joined #openstack-swift09:35
*** m_kazuhi_ has quit IRC09:37
claygkota_: i like what you were saying about maybe having get_diskfile_from_audit_location (or really from_hash_dir) call cleanup_ondisk_files like get_diskfile_from_hash does09:38
kota_clayg: yeah09:40
kota_i think just ordering diskfile to cleanup is a way to a winner rather than unlink by hand09:42
claygi'm not sure it saves everything tho - you need to know if cleanup triggered a reclaim09:42
kota_cleanup_ondisk files does, that is cleanup_list_dir for old name AFAIK.09:43
kota_the point we need to care is the deleting hash dir for my opinion09:44
kota_that needs a lock to prevent racing, right?09:44
kota_i mean, recaim ts, deleting hash dir while a client is putting a new object09:45
kota_maybe09:45
claygkota_: and it's *kind* of inefficient of ssync to do that - call cleanup before opening the disk file - you just did the listdir and parsed all the ondisk info - then you open the disk file and it does the listdir and parses all the ondisk info :'(09:46
kota_I didn't look at the new patch set from mahatic sorry, i might be saying something wrong.09:46
kota_ah gah, exactly!09:46
kota_don't listdir and parse may times!09:47
claygi think you just pass on the ENOTEMPTY when you rmdir - on the put side - i think the renamer should mostly cover you?09:47
kota_maybe? wait a sec, looking the code09:48
kota_oh yeah, right now just pass OSError make sense.09:51
kota_yeah, we don't need a lock for it, I was wrong on that.09:55
mahaticdunno how having get_diskfile_from_audit_location call cleanup_ondisk_files will be better as opposed to calling cleanup_ondisk_files (via get_hashes) only on reclaimable ts09:57
mahaticmaybe i'm missing some other problem09:57
claygi'm having a tough time pulling out _cleanup_ondisk_results so that you can call it with an existing df's _ondisk_info cache09:58
claygmahatic: I think mostly it's just symmetry - there's a lot of interfaces in diskfile - it'd be better if there were less and they were more similar and more composable10:00
claygbut the *problem* as it were isn't so much calling cleanup_ondisk_files when you build it from audit location or when you open the file - it's the calling via get_hashes10:01
claygthat get_hashes method the way it's being called now will rehash every invalid/unhashes suffix in the whole damn part PLUS the one you're explicitly forcing it to rehash/recal no matter it's invalid or not10:02
claygthat's a lot of work - so the question is - is it all needed - and the answer is pretty clearly no - we could do a lot less file system thrashing and still ahve all the behaviors we want10:03
claygi'm mostly leaning toward just invalidating the hash and calling it a day10:06
mahaticthat makes sense. maybe could just call _hash_suffix_dir instead of get_hashes10:07
claygif I saw a clean way to call a supported interface in diskfile that handles the cleanup of a hashdir such that I could gently try: os.rmdir(); except ENOENT: pass my way up from hash to suffix and leave replication out of it i'd probably been keen on that too10:08
mahaticclayg: yes it's super late for you. thanks much for spending time on this!10:08
claygmahatic: looking!10:08
kota_clayg: +1 that, we need that clean way to do that10:09
kota_and agree with mahatic you should go your bed10:09
openstackgerritDavanum Srinivas (dims) proposed openstack/swift: [WIP] Testing latest u-c  https://review.openstack.org/31844110:10
openstackgerritDavanum Srinivas (dims) proposed openstack/swift: [WIP] Testing latest u-c  https://review.openstack.org/31844110:10
claygnope, too much - we shouldn't be evaluating every hashdir int he suffix everytime we encounter one of these - there are going to be cases of multiple recliamable tombstones in the same suffix hit in the same audit pass (at leasrt initially)10:10
claygkota_: we *want* a clean way to do that - short of that I'm getting more and more happy with just invalidate the hash and let replication sort it out10:11
claygyeah i should go to bed - i was *going* to work on devstack tonite!10:11
kota_my bad english, that is my first comment on the patch I said " One is a suggestion to leave the role to delete the reclaimable ts to replicator."10:12
kota_:/10:12
kota_clayg:^^10:13
claygyeah auditor just notices it and flags it for the replicator10:14
claygi thought you ment making it for the replicator to notice it - which I didn't like - because I don't want the replicator going into a bunch of hashed suffixes - THAT'S WHY WE HASHED THEM10:15
kota_clayg: that's simple sorry for stopping you to go bed.10:15
clayglol10:15
kota_yes10:15
kota_ah, you mean auditer may notice via another (new) way to do that?10:16
kota_no, no more question10:16
kota_will ask tommorow10:16
* kota_ is going to be back home10:17
mahaticclayg: good night! kota_ have a good evening. Thanks to both of you.10:18
claygidk, just calling invalidate seems to take two passes to reap the hashdir (!?)10:19
openstackgerritGábor Antal proposed openstack/swift: Use more specific asserts in test/unit/obj tests  https://review.openstack.org/34283010:20
mahaticcalling cleanup_ondisk_files giving the hash path of the ts file might achieve what we want10:24
mahaticok, but the issue is being able to call it with an existing df I guess :) looking at code10:27
claygmahatic: yeah i gotta give up for tonite - you have landed yourself in a right proper hozer this one - don't give up!10:33
claygg'ight10:33
mahatic:) g'night10:34
*** zul has quit IRC10:58
*** ouchkernel has quit IRC11:07
*** cdelatte has quit IRC11:07
*** darrenc has quit IRC11:07
*** zhengyin has quit IRC11:07
*** ejat has quit IRC11:07
*** cdelatte has joined #openstack-swift11:09
*** darrenc has joined #openstack-swift11:09
*** zhengyin has joined #openstack-swift11:09
*** ejat has joined #openstack-swift11:09
*** ouchkernel has joined #openstack-swift11:10
*** cdelatte has quit IRC11:15
*** cdelatte has joined #openstack-swift11:30
*** zul has joined #openstack-swift11:30
mahaticclayg: kota_ https://gist.github.com/MahatiC/f64ae7c08f5897ebe43415677486ed3c11:38
mahaticdid we consider that?11:38
*** dmorita has joined #openstack-swift11:51
*** takashi has joined #openstack-swift11:54
*** dmorita has quit IRC11:55
*** kei_yama has quit IRC11:56
*** ekarlso_ has joined #openstack-swift12:05
mahatic^ sorry that gist is not complete yet, but using cleanup_ondisk_files that way was what I meant to convey12:06
*** manous has joined #openstack-swift12:09
*** ekarlso_ has quit IRC12:11
*** asettle has quit IRC12:13
*** zul_ has joined #openstack-swift12:15
*** zul_ has quit IRC12:16
*** zul_ has joined #openstack-swift12:17
*** zul has quit IRC12:17
*** manous has quit IRC12:32
*** Trixboxer has quit IRC12:38
openstackgerritMathias Bjoerkqvist proposed openstack/swift: WIP: Storing encryption root secret in Barbican  https://review.openstack.org/36487812:39
*** zul_ has quit IRC12:42
*** zul has joined #openstack-swift12:43
*** zul has quit IRC12:45
*** zul has joined #openstack-swift12:45
*** manous has joined #openstack-swift12:45
*** Trixboxer has joined #openstack-swift12:50
openstackgerritAndreas Jaeger proposed openstack/swift: Move bandit to pep8  https://review.openstack.org/36489213:00
openstackgerritChristian Schwede proposed openstack/swift: Fix swiftdir option and usage of storage policy aliases  https://review.openstack.org/34469313:03
*** dmorita has joined #openstack-swift13:03
*** manous has quit IRC13:03
*** psachin has quit IRC13:04
*** dmorita has quit IRC13:08
*** tongli has joined #openstack-swift13:09
*** klamath has joined #openstack-swift13:09
*** ttrumm has quit IRC13:10
*** klamath has quit IRC13:11
*** ttrumm has joined #openstack-swift13:11
*** klamath has joined #openstack-swift13:12
*** ttrumm has quit IRC13:12
*** asettle has joined #openstack-swift13:13
*** manous has joined #openstack-swift13:16
*** asettle has quit IRC13:19
*** madorn has quit IRC13:27
*** kencjohnston has quit IRC13:27
*** StraubTW has joined #openstack-swift13:27
*** StraubTW has quit IRC13:28
*** StraubTW has joined #openstack-swift13:28
*** kencjohnston has joined #openstack-swift13:32
*** madorn has joined #openstack-swift13:34
*** links has quit IRC13:41
*** tesseract- has quit IRC14:03
*** _JZ_ has joined #openstack-swift14:06
*** sams-gleb has joined #openstack-swift14:10
*** mingyu has quit IRC14:11
*** tesseract- has joined #openstack-swift14:13
*** dmorita has joined #openstack-swift14:15
*** dmorita has quit IRC14:20
*** takashi has quit IRC14:21
*** asettle has joined #openstack-swift14:25
*** takashi has joined #openstack-swift14:27
*** diogogmt has joined #openstack-swift14:28
*** schizo_ has quit IRC14:36
*** schizo has joined #openstack-swift14:38
*** manous has quit IRC14:44
*** links has joined #openstack-swift14:47
*** thebloggu has joined #openstack-swift14:55
*** manous has joined #openstack-swift14:56
*** rcernin has quit IRC15:01
*** links has quit IRC15:03
*** sams-gleb has quit IRC15:04
*** jordanP has joined #openstack-swift15:07
*** pcaruana has quit IRC15:15
*** arch-nemesis has joined #openstack-swift15:19
*** mingyu has joined #openstack-swift15:22
-openstackstatus- NOTICE: The Gerrit service on review.openstack.org is restarting quickly to relieve resource pressure and restore normal performance15:22
*** sams-gleb has joined #openstack-swift15:23
*** arch-nemesis has quit IRC15:25
notmynamegood morning15:26
*** chsc has joined #openstack-swift15:33
*** chsc has joined #openstack-swift15:33
*** silor has joined #openstack-swift15:37
*** klamath has quit IRC15:37
*** arch-nemesis has joined #openstack-swift15:39
*** tesseract- has quit IRC15:42
*** klamath has joined #openstack-swift15:51
*** nadeem has joined #openstack-swift15:52
*** sams-gleb has quit IRC15:53
*** klamath has quit IRC15:53
openstackgerritChristian Schwede proposed openstack/swift: Fix swiftdir option and usage of storage policy aliases  https://review.openstack.org/34469315:58
*** manous has quit IRC16:02
*** tongli has quit IRC16:02
*** baojg has joined #openstack-swift16:05
*** mingyu has quit IRC16:11
*** baojg has quit IRC16:17
*** jordanP has quit IRC16:19
*** asettle has quit IRC16:21
notmynamegerrit's going down in about 2 hours16:21
notmyname4 hour scheduled downtime16:22
*** zul has quit IRC16:23
*** takashi has quit IRC16:26
*** hseipp has quit IRC16:26
*** admin6_ has quit IRC16:27
*** tongli has joined #openstack-swift16:28
*** tongli has quit IRC16:32
*** zul has joined #openstack-swift16:35
*** dmorita has joined #openstack-swift16:40
*** tqtran has joined #openstack-swift16:43
*** takashi has joined #openstack-swift16:46
*** tongli has joined #openstack-swift16:49
*** ChubYann has joined #openstack-swift16:52
*** sams-gleb has joined #openstack-swift16:54
*** tongli has quit IRC16:54
openstackgerritTravis McPeak proposed openstack/swift: Updating Bandit config file  https://review.openstack.org/36506516:55
*** Suyi_ has joined #openstack-swift16:55
*** tmcpeak has joined #openstack-swift16:55
notmynamezaitcev: haven't you looked at the bandit stuff before? ^16:55
zaitcevnotmyname: The name sounds familiar but I did not look at it. Maybe I promised to review it an underdelivered as usual....16:56
tmcpeako/16:56
tmcpeakI proposed the patch16:57
zaitcevOh god, not YAML, please anything16:57
tmcpeaklol16:57
tmcpeakJSON fans?16:57
notmynamezaitcev: nah, I probably am just misremembering16:57
notmynametmcpeak: did you know there are 63 ways to format a multi-line string in yaml and you need a turing-complete parser to evaluate it? I discovered there facts abotu yaml recently. fun!16:58
tmcpeak:\16:58
tmcpeakthat's insane16:58
notmynameI agree ;-)16:59
*** nadeem has quit IRC17:00
*** sams-gleb has quit IRC17:01
notmynametmcpeak: but to remove any confusion, we've got yaml in the repo now, and if that's what bandit uses, then that's what bandit uses. personally, I don't know much about it, and I saw that there's some work to make it a voting job17:03
tmcpeakyeah, I proposed this patch to fix the job17:03
tmcpeakBandit changed the config file format, this updates it so the Bandit job works.  Once it merges you can pick up the voting stuff again17:03
tmcpeakwell not you, but whoever17:03
*** baojg_ has joined #openstack-swift17:06
*** takashi has quit IRC17:06
*** nadeem has joined #openstack-swift17:06
*** nadeem has quit IRC17:07
*** nadeem has joined #openstack-swift17:08
*** mingyu has joined #openstack-swift17:11
*** ttrumm has joined #openstack-swift17:15
*** takashi has joined #openstack-swift17:16
*** mingyu has quit IRC17:16
*** ttrumm has quit IRC17:22
*** klamath has joined #openstack-swift17:23
*** tongli has joined #openstack-swift17:25
*** tongli has quit IRC17:29
-openstackstatus- NOTICE: The infrastructure team is taking Gerrit offline for maintenance, beginning shortly after 18:00 UTC for a potentially 4 hour maintenance window.17:39
*** tongli has joined #openstack-swift17:49
*** nadeem has quit IRC17:50
*** thebloggu has quit IRC17:52
*** tongli has quit IRC17:53
*** geaaru has quit IRC17:56
*** silor has quit IRC17:59
*** silor has joined #openstack-swift18:00
*** baojg_ has quit IRC18:05
notmynameFYI https://review.openstack.org/#/c/355253/18:05
patchbotpatch 355253 - openstack-dev/devstack - Add 'hitch' support for 'tls-proxy' option.18:05
-openstackstatus- NOTICE: Gerrit is now going offline for maintenance, reserving a maintenance window through 22:00 UTC.18:06
*** ChanServ changes topic to "Gerrit is now going offline for maintenance, reserving a maintenance window through 22:00 UTC."18:06
*** sams-gleb has joined #openstack-swift18:12
*** mingyu has joined #openstack-swift18:14
*** silor has quit IRC18:16
*** mingyu has quit IRC18:19
*** sams-gleb has quit IRC18:23
claygmahatic: for me https://gist.github.com/MahatiC/f64ae7c08f5897ebe43415677486ed3c is very close!18:24
claygmahatic: the only downside is that it makes an additional listdir call18:24
claygnotmyname: what's this hitch tls-proxy about?18:27
notmynameclayg: I'm not completely sure TBH18:27
notmynameclayg: but it caught my eye on the mailing list because I though it might have something to do with swift proxies in devstack. not sure. warrants a look18:28
claygBUT I CAN'T LOOK AT IT!?18:28
notmynameclayg: turns out I have a local clone of devstack.18:28
*** caiobrentano_ has joined #openstack-swift18:29
claygtmcpeak: i thought bandit flagged a bunch of stuff in swift?  cause idk, we use hashlib or something?18:29
claygnowhey1?18:29
notmynamehuh. was only 3542 commits behind master ;-)18:29
tmcpeakclayg: that's probably been disabled in the profile18:29
tmcpeakwhatever you had before I just copied over18:29
notmynamehmm...looks like tls-proxy is a service that can be used in devstack to give a TLS endpoint. so that's nice18:30
notmynameclayg: ok, prob not a thing to worry about per se, but the change from stud to hitch is good to know (that stud isn't a thing anymore in ubuntu)18:31
*** zaitcev has quit IRC18:32
*** zaitcev has joined #openstack-swift18:32
*** ChanServ sets mode: +v zaitcev18:32
*** ogelbukh has quit IRC18:33
*** ogelbukh has joined #openstack-swift18:34
*** tongli has joined #openstack-swift18:38
claygoh snap - gerrits back maybe?18:40
notmynamethe window's 4 hours long (staring 40 minutes ago)18:41
*** tongli has quit IRC18:42
*** zhengyin has quit IRC18:45
*** zhengyin has joined #openstack-swift18:46
*** caiobrentano_ has quit IRC18:49
openstackgerritDavid Goetz proposed openstack/swift: go: add part / replica count funcs into ring  https://review.openstack.org/36509318:52
*** silor has joined #openstack-swift19:00
*** silor1 has joined #openstack-swift19:04
*** silor has quit IRC19:06
*** silor1 is now known as silor19:06
*** Jeffrey4l has quit IRC19:07
*** tongli has joined #openstack-swift19:09
*** tongli has quit IRC19:09
*** tongli has joined #openstack-swift19:09
*** tongli has quit IRC19:09
*** tongli has joined #openstack-swift19:09
*** ChanServ changes topic to "Topic: Let's talk. We're nice. | Ideas: https://wiki.openstack.org/wiki/Swift/ideas | Logs: http://eavesdrop.openstack.org/irclogs/%23openstack-swift/ | Meetings: https://wiki.openstack.org/wiki/Meetings/Swift"19:15
-openstackstatus- NOTICE: Gerrit is completing an online re-index, you may encounter slowness until it is complete19:15
*** klamath has quit IRC19:16
*** silor has quit IRC19:21
*** delattec has joined #openstack-swift19:22
*** cdelatte has quit IRC19:25
*** delattec has quit IRC19:30
*** links has joined #openstack-swift19:43
*** tmcpeak has quit IRC19:54
*** klamath has joined #openstack-swift19:55
*** links has quit IRC19:55
*** klamath has quit IRC19:57
*** mingyu has joined #openstack-swift20:03
*** openstackgerrit has quit IRC20:04
*** openstackgerrit has joined #openstack-swift20:04
*** tqtran has quit IRC20:07
*** mingyu has quit IRC20:09
openstackgerritMerged openstack/swift: Deprecate swift-temp-url  https://review.openstack.org/34817320:21
claygwhooo!   eat it swift-temp-url!20:30
claygso now lp bug #1607521 is a bigger deal :D20:31
openstackLaunchpad bug 1607521 in python-swiftclient "tempurl uses wrong HMAC with full path" [Medium,New] https://launchpad.net/bugs/160752120:31
*** tongli has quit IRC20:33
*** tmcpeak has joined #openstack-swift20:33
*** _JZ_ has quit IRC20:35
timburkeclayg: nah, cschwede's got it :-)20:39
openstackgerritMerged openstack/swift: go: move REPCONN and REPLICATE call from object server to replicator  https://review.openstack.org/36407520:43
*** nadeem has joined #openstack-swift20:44
*** tqtran has joined #openstack-swift20:54
*** chsc has quit IRC21:11
*** dmorita has quit IRC21:37
*** dmorita has joined #openstack-swift21:39
*** tmcpeak has left #openstack-swift21:43
*** StraubTW has quit IRC21:46
*** arch-nemesis has quit IRC21:47
*** dmorita has quit IRC21:50
*** dmorita has joined #openstack-swift21:51
*** cebreidian has quit IRC21:58
*** dmorita has quit IRC22:03
*** cebreidian has joined #openstack-swift22:06
*** dmorita has joined #openstack-swift22:07
openstackgerritMathias Bjoerkqvist proposed openstack/swift: WIP: Storing encryption root secret in Barbican  https://review.openstack.org/36487822:15
*** david-lyle has quit IRC22:15
*** david-lyle has joined #openstack-swift22:17
*** sanchitmalhotra has quit IRC22:26
*** sanchitmalhotra has joined #openstack-swift22:26
*** dmorita has quit IRC22:33
*** dmorita has joined #openstack-swift22:34
*** nadeem has quit IRC22:38
*** dmorita has quit IRC22:38
*** dmorita has joined #openstack-swift22:50
*** mingyu has joined #openstack-swift22:53
*** mingyu has quit IRC22:58
*** jeblair is now known as corvus23:01
*** vint_bra has quit IRC23:04
*** asettle has joined #openstack-swift23:05
*** Suyi_ has quit IRC23:09
*** asettle has quit IRC23:10
*** openstackgerrit has quit IRC23:18
*** openstackgerrit has joined #openstack-swift23:19
*** tqtran has quit IRC23:41
*** asettle has joined #openstack-swift23:46
*** dmorita has quit IRC23:57
timburketimur: question for next week: what's the expected behavior around https://review.openstack.org/#/c/363316/4/swift/proxy/controllers/base.py@1071 if end (or self.backend_headers['Range'], for that matter) is None?23:59
patchbotpatch 363316 - swift - Silence "Client disconnected" warnings on reads. (MERGED)23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!