21:00:02 <timburke> #startmeeting swift
21:00:03 <openstack> Meeting started Wed Jul 24 21:00:02 2019 UTC and is due to finish in 60 minutes.  The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:04 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:06 <openstack> The meeting name has been set to 'swift'
21:00:10 <timburke> who's here for the swift meeting?
21:00:14 <kota_> o/
21:00:45 <mattoliverau> o/
21:01:15 <rledisez> o/
21:02:29 <timburke> agenda's at https://wiki.openstack.org/wiki/Meetings/Swift
21:02:50 <timburke> sorry, i haven't updated it -- but i think we're mostly following up on things anyway
21:03:09 <timburke> #topic shanghai
21:03:37 <timburke> reminder that that's a thing. you should've gotten discount codes recently
21:03:47 <clayg> o/
21:03:50 <rledisez> got it
21:04:14 <kota_> oh really?
21:04:19 <timburke> i don't need the topics fully fleshed out or anything, but it'll be good to get the foundation a rough headcount to expect
21:04:31 <clayg> is anyone besides tim going *for sure* at this point?
21:04:35 <timburke> kota_, maybe it'll arrive in the next day or two
21:04:53 <kota_> timburke: ok
21:05:19 <timburke> i know they often have to stagger these things so they don't immediately get blackholed as spam
21:05:24 <kota_> ah... i have to check older e-mail box too.
21:05:28 <clayg> mine was titled "Invitation & Discount Registration to Open Infrastructure Summit Shanghai" and it came from summitreg@openstack.org - but it looks like they used e2ma.net as the mass mailing service
21:05:30 <mattoliverau> Yeah, I got it during the night, Ie I saw it in my email this morning
21:05:33 <rledisez> clayg: alecuyer and I are still expecting to go, no formal confirmation yet
21:06:12 <kota_> I'll check that today. thanks good info.
21:06:37 <timburke> i still need to write an email to the mailing list to extend an invitation to not-you-guys and try to get a feel for the onboarding vs design/discussion split
21:07:28 <timburke> #topic release
21:07:49 <timburke> we've got a 2.22.0 release out! it's got some py3 support!
21:08:36 <clayg> 🥳
21:08:58 <timburke> #link http://lists.openstack.org/pipermail/release-announce/2019-July/007429.html
21:08:59 <zaitcev> Some?
21:09:52 <timburke> zaitcev, probe tests don't pass. i found a few (mostly small?) issues when writing https://review.opendev.org/#/c/671333/
21:09:53 <patchbot> patch 671333 - swift - py3: (mostly) port probe tests - 2 patch sets
21:10:12 <timburke> but we can get to that in a sec ;-)
21:11:15 <timburke> there's a bunch of *other* great stuff, too, though! which is kind of amazing
21:11:27 <timburke> configurable, anonymizable logs!
21:11:37 <timburke> docker image!
21:12:11 <timburke> continuous improvements to a variety of subsystems!
21:12:24 <timburke> thank you all so much for helping to make this the best Swift yet :-D
21:12:40 <mattoliverau> \o/
21:13:21 <timburke> shortly after that went in, i approved a few patches that i wanted to wait until after the release
21:13:34 <clayg> best swift evar!
21:14:02 <timburke> p 665758, p 657034, p 669511 in particular
21:14:03 <patchbot> https://review.opendev.org/#/c/665758/ - swift - Bump up our minimum eventlet version (MERGED) - 4 patch sets
21:14:05 <patchbot> https://review.opendev.org/#/c/657034/ - swift - Make py36 job voting (MERGED) - 1 patch set
21:14:06 <patchbot> https://review.opendev.org/#/c/669511/ - swift - Add Python 3 Train unit tests (MERGED) - 1 patch set
21:15:06 <timburke> on to other updates!
21:15:14 <timburke> #topic py3
21:15:34 <tdasilva> timburke: you also proposed to devstack to start running swift under py3!!!
21:16:21 <timburke> yup! p 670353 has one +2, waiting for final approval
21:16:22 <patchbot> https://review.opendev.org/#/c/670353/ - devstack - Remove Swift from default DISABLED_PYTHON3_PACKAGES - 1 patch set
21:17:06 <timburke> which will unblock p 671554 -- bringing back tempest tests
21:17:07 <patchbot> https://review.opendev.org/#/c/671554/ - swift - Add integrated gate tempest and grenade job - 3 patch sets
21:17:44 <timburke> (py3 support wasn't really the blocker there, i don't think -- but it was certainly a nice-to-have)
21:18:03 <clayg> @timburke that does change how the func tests get started?
21:18:27 <clayg> i only ask cause... we still run them under py2, right?
21:18:46 <timburke> clayg, no -- it just runs another set of independently-developed func tests
21:19:29 <timburke> there are a bunch of places where we still need to port the func tests themselves to be able to run in-process py3 func tests
21:20:55 <timburke> but i realized that, since we now run *all func tests* (under py2) against services running py3, the next-most-valuable testing is (or, seems to me to be) probe!
21:21:06 <timburke> leading to p 671554
21:21:07 <patchbot> https://review.opendev.org/#/c/671554/ - swift - Add integrated gate tempest and grenade job - 3 patch sets
21:21:17 <timburke> er..
21:21:18 <timburke> #undo
21:21:19 <openstack> Removing item from minutes: #link https://review.opendev.org/#/c/671554/
21:21:33 <timburke> bah. whatever.
21:21:43 <timburke> wrong bot
21:21:45 <timburke> p 671333
21:21:45 <patchbot> https://review.opendev.org/#/c/671333/ - swift - py3: (mostly) port probe tests - 2 patch sets
21:22:31 <timburke> that kicked out p 671167 p 670932 p 670933 and p 671168
21:22:32 <patchbot> https://review.opendev.org/#/c/671167/ - swift - py3: fix up listings on sharded containers - 1 patch set
21:22:33 <patchbot> https://review.opendev.org/#/c/670932/ - swift - py3: fix up swift-orphans - 1 patch set
21:22:35 <patchbot> https://review.opendev.org/#/c/670933/ - swift - py3: fix object-replicator rsync output parsing - 2 patch sets
21:22:37 <patchbot> https://review.opendev.org/#/c/671168/ - swift - Fix up errno checking (MERGED) - 1 patch set
21:22:51 <timburke> and that last one's even merged already! thanks tdasilva :-)
21:24:01 <tdasilva> o/
21:24:27 <timburke> that's about what i've got for py3. of course, reviewing any of those or the already-written py3 func test changes would be tremendously helpful
21:25:04 <timburke> as would writing new patches to port other func tests -- in particular, the s3api tests seemed hairy (FWIW)
21:25:22 <kota_> :/
21:26:38 <zaitcev> So
21:26:42 <timburke> i seem to recall running into some issue with boto and eventlet interactions leading to a recursion problem in down in stdlib's ssl... even though i wasn't using https :-(
21:26:51 <zaitcev> I still have the problem with duplicate keys even on all-py3 cluster.
21:27:17 <zaitcev> metatest.txt: "X-Account-Meta-Uni\u00e0\u00b8\u0092": ["1", "1563926869.20886"],
21:27:17 <zaitcev> metatest.txt: "X-Account-Meta-Uni\u0e12": ["1", "1563407142.42696"],
21:27:48 <kota_> curious
21:28:03 <zaitcev> Unfortunately, it was a dirty experiment. I realized that I left a handoff db intact when I purged the meta with  UPDATE account_stat SET metadata = '';
21:28:53 <timburke> oh, interesting... so you tried clearing everything out and running functests again, and it showed up again?
21:29:05 <zaitcev> re-trying now
21:29:14 <timburke> i still want to figure out what's going on there -- zaitcev, can we work on that after the meeting?
21:30:34 <zaitcev> see, the timestamps are different now. So native is old, and the latin-1 is new, re-introduced by functests. I think it's a bad idea to store latin-1, actually... It's incompatible with py2 for sure.
21:31:17 <timburke> i agree -- the wsgi-fied string is no good
21:32:39 <timburke> let's keep moving for now
21:32:41 <timburke> #topic lots of small files
21:32:41 <zaitcev> I didn't have this problem with objects thus far.
21:32:43 <zaitcev> ok
21:32:57 <timburke> rledisez, kota_ anything new to report?
21:33:14 <kota_> sorry nothing new for me
21:33:36 <timburke> that's fine! i just want to check in :-)
21:33:49 <rledisez> timburke: not as I know. alecuyer is now off for 3 weeks
21:34:01 <timburke> 👍
21:34:13 <timburke> #topic auto sharding
21:34:26 <timburke> mattoliverau, i saw you rebased
21:35:03 <mattoliverau> yeah
21:35:21 <mattoliverau> I rebased the patch chain off the new ring info patch
21:35:40 <mattoliverau> I've also been working on the sharder a bit more.
21:36:03 <clayg> i think i missed the ring info patch - is it amazing!?
21:36:26 <timburke> p 670673
21:36:27 <patchbot> https://review.opendev.org/#/c/670673/ - swift - ring: Track more properties of the ring - 3 patch sets
21:36:59 <timburke> mattoliverau, need anything from us? or just reviews? ;-)
21:37:06 <mattoliverau> Trying out the idea of when the scanner has scanned and checked that they are still the scanner, instead of just writing the shard ranges to their DB. It tries to UPDATE the shard ranges to the other primaries, if any succeed, it's safe to write locally.
21:37:28 <mattoliverau> that way we always have at least 2 shard ranges out there.
21:37:48 <mattoliverau> not 100% convinced, but thought about it yest to giving it a whirl.
21:38:02 <timburke> interesting...
21:38:24 <timburke> i'll try to take a look. not sure i can get to it this week though
21:38:52 <timburke> #topic symlinks and versioning
21:38:55 <mattoliverau> I wanted to write down then replicate and then roll back (potentually if all failed) but you don't know if replication has happened in between.
21:39:02 <timburke> #undo
21:39:03 <mattoliverau> nps
21:39:04 <openstack> Removing item from minutes: #topic symlinks and versioning
21:39:30 <mattoliverau> like I said, just thinking out loud and in code.. not 100% convinced :P
21:39:49 <mattoliverau> I'll push up what I have today (I was meant to last night but forgot)
21:39:59 <mattoliverau> now you can move on :P
21:40:08 <clayg> p 633094 was done, but tdasilva was asking if we could figure out how to use x-etag-is-at headers on put to make hardlinks to slo's use the slo quoted etag instead of the manifest etag...
21:40:08 <patchbot> https://review.opendev.org/#/c/633094/ - swift - Allow "harder" symlinks - 16 patch sets
21:40:16 <timburke> heh, yeah -- i'm still torn, too. i'm not sure that non-leader should be the first one responsible for owning shard ranges...
21:40:18 <timburke> #topic symlinks and versioning
21:41:05 <clayg> i started to rework the versions one on the assumption that hardlink target etag should be the manifest etag and while I mostly got that "working" there's still some knowledge about slo bleeding into versionsing 😞
21:41:28 <clayg> meanwhile I need to freshen up my s3-versions patch ontop of all this
21:41:39 <clayg> so ... it's all pretty shaky
21:42:09 <clayg> but also unfortunately pretty serialized - so I'll keep chugging and let ya'll good folks know when I think it's done, or closer, or I need a review or something
21:42:17 <timburke> it's hard making sure that 3 dramatically different features all work together the "right" way
21:42:25 <kota_> thx for working s3 versioning!
21:42:43 <timburke> thanks for working to make it all reasonable for the rest of us!
21:42:45 <tdasilva> clayg: any way we can help atm?
21:42:52 <clayg> yeah last week the subject of versinsing and object expiration came up?
21:43:19 <clayg> basically we don't know how it really interacts and how symlinks will change that - or how it *should* work
21:43:55 <clayg> we thought maybe we should look at s3 bucket policies for versioned buckets and their expiration features to get an idea of what we would aim to support as s minimum subset
21:44:05 <clayg> but I haven't done that - so someone could think about that
21:44:24 <clayg> ^ tdasilva
21:44:32 <clayg> but other than that, nothing I can think of
21:45:26 <tdasilva> fwiw, aws states that for s3 buckets can't have versioning and expiration policy together
21:45:47 <tdasilva> gotta pick one or the other
21:45:55 <timburke> interesting
21:46:08 <timburke> but they have a way to expire old versions, no?
21:46:16 <clayg> tdasilva: rly?  do you have that reference handy - it's possible I don't understand which features those words map too
21:47:11 <tdasilva> "You can use the Object Expiration feature on buckets that are stored using Standard or Reduced Redundancy Storage. You cannot, however, use it in conjunction with S3 Versioning (this is, as they say, for your own protection). You will have to delete all expiration rules for the bucket before enabling versioning on that bucket."
21:47:19 <tdasilva> got it from here: https://aws.amazon.com/blogs/aws/amazon-s3-object-expiration/
21:47:24 <clayg> thanks!
21:47:40 <tdasilva> but just noticed it's from 2011, maybe it has changed???
21:47:51 <tdasilva> i'll try it out tomorow
21:48:09 <timburke> tdasilva, would you be able to get a summary of the state of policies and versioning (and any other relevant features) for next week? or just to drop in channel in the next day or two?
21:48:25 <tdasilva> timburke: yep!
21:48:34 <timburke> cool. i've got one more topic i'd like to bring up...
21:48:53 <timburke> #topic 404s from handoffs
21:49:12 <timburke> so i still need to write up a good bug to go with my patch, but...
21:49:15 <timburke> #link https://review.opendev.org/#/c/672186/
21:49:16 <patchbot> patch 672186 - swift - Ignore 404s from handoffs for objects when calcula... - 4 patch sets
21:50:10 <timburke> i had some complaints about an EC object not turning up
21:50:26 <timburke> and it looked like we found some frags but not enough to reconstruct
21:50:46 <timburke> so the proxy dug into handoffs and found a quorum's worth of 404s
21:51:14 <timburke> this is somewhat related to https://bugs.launchpad.net/swift/+bug/1833612 but at the object level
21:51:15 <openstack> Launchpad bug 1833612 in OpenStack Object Storage (swift) "Overloaded container can get erroneously cached as 404" [Undecided,Fix released]
21:53:28 <timburke> but it looks like we have some pretty solid tests that explore the behavior and decided on 404 over 503
21:55:36 <mattoliverau> oh, so where there enough frags out there to reconstruct?
21:55:48 <mattoliverau> and 404s just got quorum first?
21:56:42 <timburke> if the servers weren't overloaded, yes
21:57:18 <mattoliverau> just making sure :P
21:58:06 <timburke> see https://github.com/openstack/swift/blob/2.22.0/test/unit/proxy/controllers/test_obj.py#L2118-L2150 for one of the tests i had to change (iirc)
21:58:31 <timburke> we find frags, they're marked durable, but we go with a 404 ATM
21:59:12 <timburke> i feel like i'm dramatically changing the way eventual consistency currently works for swift
21:59:26 <timburke> please take a look at the patch if you get a chance
21:59:34 <timburke> i'm afraid i let us run a bit long
21:59:40 <notmyname> timburke: link the etherpad too
21:59:54 <notmyname> #link https://etherpad.openstack.org/p/swift-fail-responses
21:59:56 <kota_> notmyname: !
22:00:00 <notmyname> kota_: !!
22:00:08 <timburke> :-) thanks notmyname
22:00:20 <timburke> but now we really *are* at time
22:00:21 <timburke> #endmeeting