21:00:02 #startmeeting swift 21:00:03 Meeting started Wed Jul 24 21:00:02 2019 UTC and is due to finish in 60 minutes. The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:00:04 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:00:06 The meeting name has been set to 'swift' 21:00:10 who's here for the swift meeting? 21:00:14 o/ 21:00:45 o/ 21:01:15 o/ 21:02:29 agenda's at https://wiki.openstack.org/wiki/Meetings/Swift 21:02:50 sorry, i haven't updated it -- but i think we're mostly following up on things anyway 21:03:09 #topic shanghai 21:03:37 reminder that that's a thing. you should've gotten discount codes recently 21:03:47 o/ 21:03:50 got it 21:04:14 oh really? 21:04:19 i don't need the topics fully fleshed out or anything, but it'll be good to get the foundation a rough headcount to expect 21:04:31 is anyone besides tim going *for sure* at this point? 21:04:35 kota_, maybe it'll arrive in the next day or two 21:04:53 timburke: ok 21:05:19 i know they often have to stagger these things so they don't immediately get blackholed as spam 21:05:24 ah... i have to check older e-mail box too. 21:05:28 mine was titled "Invitation & Discount Registration to Open Infrastructure Summit Shanghai" and it came from summitreg@openstack.org - but it looks like they used e2ma.net as the mass mailing service 21:05:30 Yeah, I got it during the night, Ie I saw it in my email this morning 21:05:33 clayg: alecuyer and I are still expecting to go, no formal confirmation yet 21:06:12 I'll check that today. thanks good info. 21:06:37 i still need to write an email to the mailing list to extend an invitation to not-you-guys and try to get a feel for the onboarding vs design/discussion split 21:07:28 #topic release 21:07:49 we've got a 2.22.0 release out! it's got some py3 support! 21:08:36 🥳 21:08:58 #link http://lists.openstack.org/pipermail/release-announce/2019-July/007429.html 21:08:59 Some? 21:09:52 zaitcev, probe tests don't pass. i found a few (mostly small?) issues when writing https://review.opendev.org/#/c/671333/ 21:09:53 patch 671333 - swift - py3: (mostly) port probe tests - 2 patch sets 21:10:12 but we can get to that in a sec ;-) 21:11:15 there's a bunch of *other* great stuff, too, though! which is kind of amazing 21:11:27 configurable, anonymizable logs! 21:11:37 docker image! 21:12:11 continuous improvements to a variety of subsystems! 21:12:24 thank you all so much for helping to make this the best Swift yet :-D 21:12:40 \o/ 21:13:21 shortly after that went in, i approved a few patches that i wanted to wait until after the release 21:13:34 best swift evar! 21:14:02 p 665758, p 657034, p 669511 in particular 21:14:03 https://review.opendev.org/#/c/665758/ - swift - Bump up our minimum eventlet version (MERGED) - 4 patch sets 21:14:05 https://review.opendev.org/#/c/657034/ - swift - Make py36 job voting (MERGED) - 1 patch set 21:14:06 https://review.opendev.org/#/c/669511/ - swift - Add Python 3 Train unit tests (MERGED) - 1 patch set 21:15:06 on to other updates! 21:15:14 #topic py3 21:15:34 timburke: you also proposed to devstack to start running swift under py3!!! 21:16:21 yup! p 670353 has one +2, waiting for final approval 21:16:22 https://review.opendev.org/#/c/670353/ - devstack - Remove Swift from default DISABLED_PYTHON3_PACKAGES - 1 patch set 21:17:06 which will unblock p 671554 -- bringing back tempest tests 21:17:07 https://review.opendev.org/#/c/671554/ - swift - Add integrated gate tempest and grenade job - 3 patch sets 21:17:44 (py3 support wasn't really the blocker there, i don't think -- but it was certainly a nice-to-have) 21:18:03 @timburke that does change how the func tests get started? 21:18:27 i only ask cause... we still run them under py2, right? 21:18:46 clayg, no -- it just runs another set of independently-developed func tests 21:19:29 there are a bunch of places where we still need to port the func tests themselves to be able to run in-process py3 func tests 21:20:55 but i realized that, since we now run *all func tests* (under py2) against services running py3, the next-most-valuable testing is (or, seems to me to be) probe! 21:21:06 leading to p 671554 21:21:07 https://review.opendev.org/#/c/671554/ - swift - Add integrated gate tempest and grenade job - 3 patch sets 21:21:17 er.. 21:21:18 #undo 21:21:19 Removing item from minutes: #link https://review.opendev.org/#/c/671554/ 21:21:33 bah. whatever. 21:21:43 wrong bot 21:21:45 p 671333 21:21:45 https://review.opendev.org/#/c/671333/ - swift - py3: (mostly) port probe tests - 2 patch sets 21:22:31 that kicked out p 671167 p 670932 p 670933 and p 671168 21:22:32 https://review.opendev.org/#/c/671167/ - swift - py3: fix up listings on sharded containers - 1 patch set 21:22:33 https://review.opendev.org/#/c/670932/ - swift - py3: fix up swift-orphans - 1 patch set 21:22:35 https://review.opendev.org/#/c/670933/ - swift - py3: fix object-replicator rsync output parsing - 2 patch sets 21:22:37 https://review.opendev.org/#/c/671168/ - swift - Fix up errno checking (MERGED) - 1 patch set 21:22:51 and that last one's even merged already! thanks tdasilva :-) 21:24:01 o/ 21:24:27 that's about what i've got for py3. of course, reviewing any of those or the already-written py3 func test changes would be tremendously helpful 21:25:04 as would writing new patches to port other func tests -- in particular, the s3api tests seemed hairy (FWIW) 21:25:22 :/ 21:26:38 So 21:26:42 i seem to recall running into some issue with boto and eventlet interactions leading to a recursion problem in down in stdlib's ssl... even though i wasn't using https :-( 21:26:51 I still have the problem with duplicate keys even on all-py3 cluster. 21:27:17 metatest.txt: "X-Account-Meta-Uni\u00e0\u00b8\u0092": ["1", "1563926869.20886"], 21:27:17 metatest.txt: "X-Account-Meta-Uni\u0e12": ["1", "1563407142.42696"], 21:27:48 curious 21:28:03 Unfortunately, it was a dirty experiment. I realized that I left a handoff db intact when I purged the meta with UPDATE account_stat SET metadata = ''; 21:28:53 oh, interesting... so you tried clearing everything out and running functests again, and it showed up again? 21:29:05 re-trying now 21:29:14 i still want to figure out what's going on there -- zaitcev, can we work on that after the meeting? 21:30:34 see, the timestamps are different now. So native is old, and the latin-1 is new, re-introduced by functests. I think it's a bad idea to store latin-1, actually... It's incompatible with py2 for sure. 21:31:17 i agree -- the wsgi-fied string is no good 21:32:39 let's keep moving for now 21:32:41 #topic lots of small files 21:32:41 I didn't have this problem with objects thus far. 21:32:43 ok 21:32:57 rledisez, kota_ anything new to report? 21:33:14 sorry nothing new for me 21:33:36 that's fine! i just want to check in :-) 21:33:49 timburke: not as I know. alecuyer is now off for 3 weeks 21:34:01 👍 21:34:13 #topic auto sharding 21:34:26 mattoliverau, i saw you rebased 21:35:03 yeah 21:35:21 I rebased the patch chain off the new ring info patch 21:35:40 I've also been working on the sharder a bit more. 21:36:03 i think i missed the ring info patch - is it amazing!? 21:36:26 p 670673 21:36:27 https://review.opendev.org/#/c/670673/ - swift - ring: Track more properties of the ring - 3 patch sets 21:36:59 mattoliverau, need anything from us? or just reviews? ;-) 21:37:06 Trying out the idea of when the scanner has scanned and checked that they are still the scanner, instead of just writing the shard ranges to their DB. It tries to UPDATE the shard ranges to the other primaries, if any succeed, it's safe to write locally. 21:37:28 that way we always have at least 2 shard ranges out there. 21:37:48 not 100% convinced, but thought about it yest to giving it a whirl. 21:38:02 interesting... 21:38:24 i'll try to take a look. not sure i can get to it this week though 21:38:52 #topic symlinks and versioning 21:38:55 I wanted to write down then replicate and then roll back (potentually if all failed) but you don't know if replication has happened in between. 21:39:02 #undo 21:39:03 nps 21:39:04 Removing item from minutes: #topic symlinks and versioning 21:39:30 like I said, just thinking out loud and in code.. not 100% convinced :P 21:39:49 I'll push up what I have today (I was meant to last night but forgot) 21:39:59 now you can move on :P 21:40:08 p 633094 was done, but tdasilva was asking if we could figure out how to use x-etag-is-at headers on put to make hardlinks to slo's use the slo quoted etag instead of the manifest etag... 21:40:08 https://review.opendev.org/#/c/633094/ - swift - Allow "harder" symlinks - 16 patch sets 21:40:16 heh, yeah -- i'm still torn, too. i'm not sure that non-leader should be the first one responsible for owning shard ranges... 21:40:18 #topic symlinks and versioning 21:41:05 i started to rework the versions one on the assumption that hardlink target etag should be the manifest etag and while I mostly got that "working" there's still some knowledge about slo bleeding into versionsing 😞 21:41:28 meanwhile I need to freshen up my s3-versions patch ontop of all this 21:41:39 so ... it's all pretty shaky 21:42:09 but also unfortunately pretty serialized - so I'll keep chugging and let ya'll good folks know when I think it's done, or closer, or I need a review or something 21:42:17 it's hard making sure that 3 dramatically different features all work together the "right" way 21:42:25 thx for working s3 versioning! 21:42:43 thanks for working to make it all reasonable for the rest of us! 21:42:45 clayg: any way we can help atm? 21:42:52 yeah last week the subject of versinsing and object expiration came up? 21:43:19 basically we don't know how it really interacts and how symlinks will change that - or how it *should* work 21:43:55 we thought maybe we should look at s3 bucket policies for versioned buckets and their expiration features to get an idea of what we would aim to support as s minimum subset 21:44:05 but I haven't done that - so someone could think about that 21:44:24 ^ tdasilva 21:44:32 but other than that, nothing I can think of 21:45:26 fwiw, aws states that for s3 buckets can't have versioning and expiration policy together 21:45:47 gotta pick one or the other 21:45:55 interesting 21:46:08 but they have a way to expire old versions, no? 21:46:16 tdasilva: rly? do you have that reference handy - it's possible I don't understand which features those words map too 21:47:11 "You can use the Object Expiration feature on buckets that are stored using Standard or Reduced Redundancy Storage. You cannot, however, use it in conjunction with S3 Versioning (this is, as they say, for your own protection). You will have to delete all expiration rules for the bucket before enabling versioning on that bucket." 21:47:19 got it from here: https://aws.amazon.com/blogs/aws/amazon-s3-object-expiration/ 21:47:24 thanks! 21:47:40 but just noticed it's from 2011, maybe it has changed??? 21:47:51 i'll try it out tomorow 21:48:09 tdasilva, would you be able to get a summary of the state of policies and versioning (and any other relevant features) for next week? or just to drop in channel in the next day or two? 21:48:25 timburke: yep! 21:48:34 cool. i've got one more topic i'd like to bring up... 21:48:53 #topic 404s from handoffs 21:49:12 so i still need to write up a good bug to go with my patch, but... 21:49:15 #link https://review.opendev.org/#/c/672186/ 21:49:16 patch 672186 - swift - Ignore 404s from handoffs for objects when calcula... - 4 patch sets 21:50:10 i had some complaints about an EC object not turning up 21:50:26 and it looked like we found some frags but not enough to reconstruct 21:50:46 so the proxy dug into handoffs and found a quorum's worth of 404s 21:51:14 this is somewhat related to https://bugs.launchpad.net/swift/+bug/1833612 but at the object level 21:51:15 Launchpad bug 1833612 in OpenStack Object Storage (swift) "Overloaded container can get erroneously cached as 404" [Undecided,Fix released] 21:53:28 but it looks like we have some pretty solid tests that explore the behavior and decided on 404 over 503 21:55:36 oh, so where there enough frags out there to reconstruct? 21:55:48 and 404s just got quorum first? 21:56:42 if the servers weren't overloaded, yes 21:57:18 just making sure :P 21:58:06 see https://github.com/openstack/swift/blob/2.22.0/test/unit/proxy/controllers/test_obj.py#L2118-L2150 for one of the tests i had to change (iirc) 21:58:31 we find frags, they're marked durable, but we go with a 404 ATM 21:59:12 i feel like i'm dramatically changing the way eventual consistency currently works for swift 21:59:26 please take a look at the patch if you get a chance 21:59:34 i'm afraid i let us run a bit long 21:59:40 timburke: link the etherpad too 21:59:54 #link https://etherpad.openstack.org/p/swift-fail-responses 21:59:56 notmyname: ! 22:00:00 kota_: !! 22:00:08 :-) thanks notmyname 22:00:20 but now we really *are* at time 22:00:21 #endmeeting