21:00:46 <timburke> #startmeeting swift
21:00:47 <openstack> Meeting started Wed Jun 19 21:00:46 2019 UTC and is due to finish in 60 minutes.  The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:48 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:51 <openstack> The meeting name has been set to 'swift'
21:01:00 <timburke> who's here for the swift meeting?
21:01:03 <mattoliverau> o/
21:01:06 <alecuyer> o/
21:02:15 <clayg> o/
21:02:31 <timburke> sounded like kota_ wouldn't be able to join, and rledisez is still moving
21:02:55 <timburke> let's get started!
21:03:10 <timburke> agenda's pretty short
21:03:13 <timburke> #link https://wiki.openstack.org/wiki/Meetings/Swift
21:03:27 <timburke> so, let's do some updates
21:03:30 <timburke> #topic py3
21:03:43 <timburke> all the unit tests have been approved!
21:04:22 <timburke> just waiting on locale and tempurl to make their way through the gate
21:04:43 <timburke> last blocker for a release is being able to run func tests under py2 against services running py3
21:06:13 <timburke> that requires a couple small fixups
21:06:14 <timburke> https://review.opendev.org/#/c/665487/ to make func tests not break prefixes part-way through a codepoint
21:06:36 <timburke> https://review.opendev.org/#/c/665746/ to get some things in s3api to be native strings
21:06:41 <mattoliverau> wow, cool nice work. Sorry I'm been away most the last week (since before the last meeting) and still playing catchup here and at work.
21:07:36 <mattoliverau> Is the priority reviews page up to date?
21:07:38 <timburke> https://review.opendev.org/#/c/665773/ to make ECAppIter happy, but i might be willing to known-issues it for now
21:08:00 <timburke> mattoliverau, mostly. i'll make sure it is shortly after the meeting
21:08:03 <mattoliverau> oh looks like it is. nice
21:08:05 <mattoliverau> cool
21:08:15 <timburke> a bunch of stuff landed just this morning :-)
21:09:03 <timburke> https://review.opendev.org/#/c/662546/ is more meaty, required to work around https://bugs.python.org/issue37093 so we can send and receive non-ascii header names
21:10:03 <timburke> i think once we get those in, i'll tag 2.22
21:10:31 <mattoliverau> Awesome
21:10:43 <clayg> TAG IT
21:11:02 <timburke> next steps are things like getting a py3 dsvm job (https://review.opendev.org/#/c/653548/) and porting func tests (there's a list on https://wiki.openstack.org/wiki/Swift/PriorityReviews)
21:11:09 <timburke> clayg, REVIEW IT
21:11:10 <timburke> :P
21:11:22 <clayg> seems reasonable
21:11:47 <mattoliverau> lol
21:12:14 <timburke> after those, we try to suss out what kind of a test plan we'd like to see to declare full support
21:12:21 <timburke> well, that and probe tests
21:12:44 <timburke> anybody have questions on py3 or what needs doing there?
21:13:47 <timburke> #topic lots of small files
21:13:55 <timburke> alecuyer, how's it going?
21:14:29 <alecuyer> I have written more tests for vfile.py, the classes this time. That was slow and I restarted once, but I hope to have something "reviewable" soon
21:14:42 <timburke> yay!
21:14:55 <alecuyer> I pushed it here https://review.opendev.org/#/c/666378/ but I don't think it's worth looking at yet :)
21:15:58 <timburke> what can we do to help? more reviews, more patches, more manual testing?
21:16:57 <alecuyer> Well if someone wants to take a quick look , just to check if it's very wrong, but otherwise I think I just have to write these tests :)
21:17:03 <timburke> i really need to try out https://github.com/swiftstack/vagrant-swift-all-in-one/pull/80 sometime ...
21:17:14 <timburke> sounds good
21:17:50 <timburke> #topic open discussion
21:18:00 <timburke> anything else to bring up?
21:18:35 <timburke> i know clayg's been looking at symlinks and versioning and s3api lately
21:18:59 <clayg> it's gettin there
21:19:21 <clayg> the hardlink to symlink thing is getting so annoying
21:20:35 <timburke> should we give a little background on it to help mattoliverau and alecuyer catch up?
21:21:06 <mattoliverau> yes please :)
21:21:27 <alecuyer> hardlink is "symlink with md5 of target" or is that very wrong
21:21:35 <clayg> that's it
21:21:51 <clayg> the neat thing this does is allow us to keep some information uptodate in the container listing
21:22:13 <clayg> so normally when you put a symlink your container listings just show these zero byte objects with the symlink_target key
21:22:43 <mattoliverau> md5 of target, sounds like a bitch to keep up to date in the symlink
21:22:50 <clayg> but when you KNOW this symlink doesn't work unless the target doesn't change you can keep the hash (and the content-length) in the listing and you know it's valid (at least for the lifetime of the hardlink)
21:23:08 <clayg> yeah so the use-case was actually versioned writes
21:23:53 <clayg> so versioned writes today does a lot of moving data around to keep the "active" objects actually "in" the container... but with symlinks we can solve for data protection requirements in place
21:24:26 <clayg> and with using hardlinks we can even make listings of the version'd container looking reasonable (as far as etags & bytes)
21:24:29 <clayg> so that's all great
21:25:02 <timburke> it also resolves some race conditions when you have concurrent overwrites to the same (biggish) object
21:25:26 <clayg> somewhere along the way the idea of "use hardlinks to point to the versions" got sideways when the the "version" was actually a symlink... just cause when we fleshed out hardlinks we didn't really define the behavior for hardlink to symlink...
21:26:02 <clayg> right as "a bunch of data movement" wasn't bad enough - there's also the fact that it takes awhile and leaves wide race windows
21:26:20 <mattoliverau> lol, wow.
21:26:23 <alecuyer> that's a case we need to support then ?
21:26:24 <clayg> symlink based versions is awesome, and with hardlinks there's very little downside
21:27:29 <clayg> the hardlink to symlink use-case?  currently I have versioned writes just doing symlinks when the target in the version'd container is itself a symlink... and that may be the way to go if we can't come up with a definition for the behavior of hardlink to a symlink that we WANT
21:27:48 <timburke> alecuyer, i like your thought process :-)
21:28:03 <timburke> i think part of it was that we have existing functional tests with versioning and symlinks, and we want to make sure they still pass
21:28:15 <alecuyer> but now I think I see the case for it with clay's explanation ;)
21:28:44 <clayg> the usecase for *hardlinks* or specifically for hardlinks *to symlinks*
21:29:15 <alecuyer> ah, hardlinks I had no doubts, hardlinks to symlinks I meant
21:29:41 <clayg> I think the use-case for hadnlinks is pretty solid - the etag validation is nice, but the benifit is mostly the container listings - by making more concret the relationship with the link and the target it because reasonable to maintain state about what that hardlink points to...
21:29:43 <clayg> so that's great
21:29:50 <clayg> yeah hardlinks to symlinks 🤮
21:30:25 <mattoliverau> oh yeah, ok so for versioned writes hardlinks sound great.. once the edge cases are thought through. Tho a random symlink with a pointer to an object elsewhere, with an md5 of the target seems hard to keep up to date. Unless you know the target never changes.
21:30:27 <clayg> it just ends up akward in versioned writes having to track if it's versioning an object (write a hardlink) or a symlink (write a symlink)
21:32:35 <mattoliverau> but hey maybe people actually want that. You'll then know if somethings changed. maybe there are usecases you'd actaully want to use that. Like a tamper seal
21:33:33 <clayg> I think there's definately use-cases for symlinks and hardlinks
21:33:48 <mattoliverau> But I haven't looked at the code, so only making assumptions on the discussion here. I should go take a look :)
21:33:56 <clayg> versioned writes definately prefers hardlinks because it makes the redirection more transparent
21:34:19 <clayg> everything I have posted is rusty - i'll try to push up some current revisions soon
21:34:19 <alecuyer> I'm lost now, I thought you woulnd't update the md5 (if the target changes the link is broken ?) but I should try it and read the code :)
21:34:28 <alecuyer> it's that one right ? https://review.opendev.org/#/c/633094/
21:34:33 <clayg> no, i need to push
21:34:39 <clayg> idk why tim brought it up
21:34:42 <clayg> :P
21:34:54 <clayg> by next week i'll have some patches
21:35:08 <mattoliverau> because it's interesting? ;)
21:35:09 <timburke> mainly 'cause you mentioned it in channel this morning ;-)
21:35:10 <clayg> the end game is s3api compatible object versioning support
21:35:25 <timburke> and i know that we want to be reviewing it soonish
21:35:58 <timburke> sooner we can get other people thinking about it, too, the better
21:36:49 <timburke> anything else we want to discuss?
21:38:16 <timburke> all right, then we oughta let mattoliverau get breakfast and alecuyer get to bed :D
21:38:28 <mattoliverau> thanks :)
21:38:30 <timburke> thank you all for coming, and thank you for working on swift!
21:38:35 <timburke> #endmeeting