21:00:06 <timburke> #startmeeting swift
21:00:07 <openstack> Meeting started Wed Jan  8 21:00:06 2020 UTC and is due to finish in 60 minutes.  The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:08 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:10 <openstack> The meeting name has been set to 'swift'
21:00:15 <timburke> who's here for the swift meeting?
21:00:31 <kota_> o/
21:00:32 <seongsoocho> o/
21:00:41 <rledisez> hi o/
21:01:27 <timburke> i'm guessing mattoliverau's around, too ;-)
21:01:34 <timburke> agenda's at https://wiki.openstack.org/wiki/Meetings/Swift
21:01:44 <mattoliverau> o/
21:02:01 <timburke> first up, i just had a question i wanted to pose to people
21:02:18 <tdasilva> o/
21:02:24 <timburke> #topic probe tests and non-standard configs
21:02:56 <timburke> actually, i take that back
21:03:10 <timburke> *first up*, welcome back everyone! it's been a bit :-)
21:03:26 <timburke> hope everyone's end-of-year breaks were good
21:03:47 <zaitcev> I do not understand why *PROBE* tests need to be "tolerant". They already assume the structure of the cluster, and so the cluster must be artificial. This is in a stark difference with *functional* tests, which can and do operate on production clusters.
21:03:54 <mattoliverau> Yeah, happy new year everyone!
21:04:38 <timburke> zaitcev, fair enough. my main thinking was that i'd like to get my dev environment closer to the real thing
21:05:34 <timburke> so, back to my original question: how much divergence from "standard" SAIO configs can we reasonably expect for probe tests?
21:06:22 <timburke> this came up in the context of https://review.opendev.org/700818
21:06:22 <zaitcev> I would imagine not much. Although of course an LB appears to be immaterial, remember that they percolate because of that url_base thing.
21:06:23 <patchbot> patch 700818 - swift - Deprecate per-service auto_create_account_prefix (MERGED) - 6 patch sets
21:06:48 <timburke> where i wanted to try out the new config option and see some existing tests pass
21:08:11 <timburke> that spun out https://review.opendev.org/701075 (because the reconciler *really* wants auto_create_account_prefix to be "."
21:08:11 <patchbot> patch 701075 - swift - probe-tests: Get reconciler test passing - 2 patch sets
21:08:12 <clayg> to the extent it's *reasonable* to make probe tests more tolerant of various configs we want to support I think it's a nobel endeavor
21:08:21 <zaitcev> OK, look. I'm okay with anything like this, as long as the source of the tests does not get too complicated. If you post something to address the non-standard prefix and it's just a bunch of parameters passed in configs, I would be happy to review.
21:08:43 <clayg> zaitcev: 👍
21:08:54 <timburke> and https://review.opendev.org/701076 where we were assuming that *no constraints at all* were set, at least for py3
21:08:55 <patchbot> patch 701076 - swift - probe-tests: Avoid a DuplicateSectionError on py3 - 1 patch set
21:09:51 <clayg> yeah that one surprised me a little bit - probably worth loosing up that a bit
21:10:46 <timburke> but then i got thinking about it more broadly, in particular while trying to get some SSL support for my dev environment... and i'm finding that it's maybe gonna be painful to make that somewhere that i can still run probe tests :-(
21:11:51 <clayg> "painful to make that somewhere" ?
21:12:34 <zaitcev> I read it as "make that _work_ somewhere etc."
21:12:45 <timburke> it seems like it will be painful to make my SSL-enabled dev environment be a place that i can still run probe tests
21:12:46 <zaitcev> I accidentally the whole Internet
21:13:17 <timburke> anyway, as much as anything, i just wanted to seed the idea a bit, get you guys thinking about it. i feel like we'll probably argue about it over beers in vancouver ;-)
21:13:54 <timburke> #topic bulk upload regression
21:14:00 <timburke> #link https://launchpad.net/bugs/1857546
21:14:00 <openstack> Launchpad bug 1857546 in OpenStack Object Storage (swift) "Auto-extract looses X-Delete-At and X-Delete-After headers" [High,Confirmed]
21:14:01 <clayg> YEAH!!!  Only... I think the argument is more about "value & priority" vs "should probtests be well designed and flexible"
21:14:31 <mattoliverau> Sorry I'm a bit slow, but could we start using conf.d in saio and then probe tests can drop .conf and remove then as they restart the cluster?
21:14:32 <clayg> oh yeah... that bulk regression 🤮
21:15:03 <rledisez> that one comes from one of our customer (he was faster than us to report the bug :D)
21:15:26 <timburke> this came in over the break, wanted to make people aware of it. sorry, that one was my bad
21:15:34 <clayg> I really *liked* the refactor tho, it's something about copied requests and what belongs in the environ?  Is there a compromise that's not just "oops, tried to make it better... didn't on one axis - REVERT"
21:15:40 <timburke> #link https://github.com/openstack/swift/commit/6f00d42
21:15:45 <timburke> cause ^^^
21:15:50 <rledisez> i'll get sure to review the PR. I guess reverting is just to probe the test, right?
21:16:05 <rledisez> *to prove
21:16:14 <timburke> #link https://review.opendev.org/#/c/700652/
21:16:14 <patchbot> patch 700652 - swift - Revert "bulk: Use make_subrequest to make subreque... - 1 patch set
21:16:17 <timburke> fix ^^^
21:16:50 <rledisez> should some headers be copied (which one then?) or just copy them all?
21:17:25 <timburke> clayg, so it's definitely *not* a complete revert -- the bug i was trying to close is (should be?) still closed
21:17:37 <clayg> ORLY!?
21:18:10 <timburke> maybe i should have separated it into two commits... oh well
21:18:39 <timburke> rledisez, yeah, it seemed like the way we decide which headers to store (config option in the object server) kinda means that we *have to* copy all headers..
21:19:04 <timburke> at which point the sanitization we get from make_subrequest does more harm than good
21:19:18 <clayg> timburke: so some places you added new_env['swift.proxy_access_log_made'] = False to keep the original bug closed 🤔
21:19:30 <timburke> yup
21:19:50 <timburke> i still think logging subrequests must be a good idea
21:20:12 <clayg> yes 👍
21:20:33 <rledisez> totally
21:20:56 <clayg> ok, well I guess we can probably merge the "revert" and just have two fixed bugs... I thought it was either/or somehow
21:20:56 <timburke> anyway, i think we can have more discussions about it on the review -- just wanted to raise awareness and mention that i intend to backport it once we've got the fix on master
21:21:13 <rledisez> That behavior was actually unspecified. I'm thinking it could be a specific option instead of just relying on "some headers" to pass, and some not to
21:21:31 <rledisez> like X-Bulk-Delete-After ?
21:22:21 <timburke> *shrug* i'm not necessarily opposed
21:22:32 <timburke> but first i want to fix your customer :-)
21:22:40 <timburke> #topic SHA-1
21:23:04 <clayg> rledisez: bulk has a body we can add new options there
21:23:38 <timburke> so some researchers have demonstrated an improved chosen-prefix attack on SHA-1!
21:23:42 <timburke> #link https://sha-mbles.github.io/
21:24:28 <timburke> my memory is that we currently only use SHA-1 for tempurls and formpost
21:24:42 <timburke> oh, and admin access to /info probably...
21:25:21 <timburke> fortunately, tempurl now has a config option to specify the allowed digests
21:25:23 <timburke> #link https://github.com/openstack/swift/commit/5a4d3bdfc
21:25:46 <timburke> formpost does not yet
21:25:53 <timburke> #link https://bugs.launchpad.net/swift/+bug/1794601
21:25:53 <openstack> Launchpad bug 1794601 in OpenStack Object Storage (swift) "Formpost middleware should support stronger hash functions" [Undecided,New]
21:26:01 <zaitcev> so, what now? SHA-256?
21:26:17 <timburke> we might want to get on that bug sooner rather than later
21:26:37 <timburke> zaitcev, that was my thinking. SHA-256 or SHA-512
21:26:58 <clayg> bits for days!
21:27:22 <clayg> By renting a GPU cluster online, the entire chosen-prefix collision attack on SHA-1 costed us about 75k USD
21:27:44 <timburke> and we should think about starting the deprecation/removal process for SHA-1 signatures. we've kinda known about the need for that for a while
21:27:44 <clayg> I would easily remove formpost from swift for that kind of money 💰
21:28:07 <timburke> clayg, i think we use it for uploading tarballs ;-)
21:28:13 <clayg> BOO
21:28:17 <timburke> :P
21:28:21 <clayg> i guess we'll fix it then
21:28:38 <timburke> it shouldn't be too hard to do
21:28:46 <rledisez> I'm not sure how it impact tempurl. generating collision does not mean you know what you're looking for (aka. you don't know the secret, or you don't need colisions). so yes, we must deprecate SHA1, just because, but there is no emergency i think
21:29:10 <clayg> @timburke by "deprecate" you mean like "log warning" => "refused to start" if you've configured these things to use signatures we're not happy with?
21:29:30 <clayg> do you ever need to support more than one kind of signature at a time?  Like a migration path sort of plan?
21:29:41 <timburke> rledisez, 👍 good to get a public cloud operator's perspective
21:30:17 <timburke> clayg, log warning => start ignoring sha-1 if you've got it listed, was my thinking
21:30:28 <clayg> deprecating stuff is always a good idea, we should deprecate auto_create_account_prefix
21:30:31 <timburke> clayg, tempurl *today* should support multiple signatures
21:30:44 <clayg> sweet so prior art even
21:31:06 <clayg> hrmm... I wonder how we make sure it get's done "eventually"
21:31:18 <timburke> of course, this also means swiftclient should learn how to do the new stuff ;-)
21:31:25 <rledisez> hum, if you ignore it, you just break all already generated url, right? you can't do that, it's up to the operator to decide to remove it
21:31:35 <clayg> maybe someone will come click that little follow button or make it 🔥 or something
21:32:45 <clayg> timburke: thanks for bringing this one up!
21:33:20 <timburke> rledisez, i was hoping to get away with having a fairly long window, maybe even have a step of logging warnings *per-request* when handling sha-1 sigs... actual ignoring could be a ways off
21:33:38 <timburke> that about does it for big topics i wanted to bring up
21:33:45 <timburke> on to updates!
21:33:54 <timburke> #topic versioning
21:34:09 <clayg> 😎
21:34:12 <timburke> clayg, tdasilva i feel like there's been a lot of progress lately :-D
21:34:37 <clayg> we are locking shit DOWN!
21:35:01 <clayg> I need to re-look at this one https://review.opendev.org/#/c/698139/
21:35:02 <patchbot> patch 698139 - swift - Skip container-sync when versioning enabled - 9 patch sets
21:35:28 <clayg> which is the only dangling "probably should squash" (or at least fast follow merge) before it's on master
21:36:06 <clayg> rledisez: you may have to rub some braincells together about container-sync and versioning - for now they're not going to work together, but we won't *break* either - so I guess we were hoping that would be "enough"
21:36:20 <clayg> this is still the meat -> https://review.opendev.org/#/c/682382/
21:36:21 <patchbot> patch 682382 - swift - New Object Versioning mode - 70 patch sets
21:36:31 <clayg> and the money -> https://review.opendev.org/#/c/673682/
21:36:32 <patchbot> patch 673682 - swift - s3api: Implement object versioning API - 44 patch sets
21:37:05 <timburke> how quickly should we try to get the client support from https://review.opendev.org/691877 merged after the swift patch lands?
21:37:06 <patchbot> patch 691877 - python-swiftclient - object versioning features - 8 patch sets
21:37:14 <timburke> how are you feeling about the client support?
21:37:20 <clayg> of course if you have swiftclient with versions support (p 691877) you don't need to mess with that aws s3api list-object-versions non-sense
21:37:29 <rledisez> clayg: versioning at source or at destination will not work with container-sync?
21:37:48 <clayg> timburke: thanks for reminding me!  I should work on `swift delete --all` 👍
21:38:23 <clayg> rledisez: at *source* was the primary concern, I'm not sure I've tested at dest
21:38:24 <timburke> rledisez, source, i believe. clients won't be able to set both x-versions-enabled and x-container-sync-to or something like that
21:38:36 <clayg> might be worth saying something to that effect ont he container-sync + versionsing patch
21:38:49 <tdasilva> rledisez: I believe dest too, in that we won't allow enabling versioning if sync header is there
21:39:27 <timburke> do we know how well a VW-enabled container serves as a sync destination?
21:39:29 <clayg> rledisez: you might have to refresh us how container-sync works
21:39:54 <clayg> timburke: could be a interesting data point - I don't think it's something we test - althougth we do have some probetests for container-sync
21:41:02 <rledisez> clayg: a process scan the sqlite containers, iterate through the "new rows" (based on some sync point), and PUT/DELETE the objects to the remote endpoint. currently I think it works with versionning at destination. I don't see why it wouldn't work actually, but I didn't tested
21:42:49 <timburke> clayg, so how soon do we expect these to land on master? and what can/should people be doing to help out?
21:43:31 <clayg> I'd like to make a decision about "can we land new versioning on master w/o container-sync" because... we don't really have any plans to make that situation any better
21:44:09 <clayg> if the answer is "yes" but it needs to fail/error in expected and obvious ways... then I'm sure we'll have that tied up sometime next week
21:44:42 <clayg> we were doing some release stuff, but now I'm available to finish up whatever we need to do to land upstream
21:44:50 <clayg> has anyone tried out the API?!
21:45:13 <timburke> (besides me, clayg, and tdasilva ;-)
21:45:30 <clayg> nothing of any significance has changed in AGES - it's just been spit and polish
21:45:57 <clayg> we're certainly past the point of making significant redesign sort of conversations - so if you haven't looked at it yet, it's probably more just FYI at this point
21:46:00 <tdasilva> viks was asking about testing the s3 api last night, which is great
21:46:03 <clayg> we'll merge it when it's done
21:46:20 <clayg> 🤗 viks___
21:46:22 <timburke> 👍
21:46:38 <timburke> #topic slo ranged reads
21:46:49 <timburke> just a quick update on the patch
21:46:51 <timburke> #link https://review.opendev.org/697739/
21:46:52 <patchbot> patch 697739 - swift - Have slo tell the object-server that it wants whol... - 6 patch sets
21:47:27 <timburke> i tried out having symlink and dlo use the same header as slo, worked like a charm!
21:47:45 <timburke> i think that cleans up the status codes getting logged at the object layer nicely, too
21:48:04 <timburke> it'll be interesting getting opinions from other people on it
21:48:22 <timburke> #topic quoted etags
21:48:30 <timburke> #link https://review.opendev.org/700056
21:48:30 <patchbot> patch 700056 - swift - Middleware that allows a user to have quoted Etags - 5 patch sets
21:48:30 <clayg> timburke: ¡excelente!
21:48:36 <timburke> patch has tests and docs now!
21:48:57 <timburke> thanks for proposing the new middleware, rledisez! i'm liking that direction more and more
21:49:55 <clayg> nice!
21:50:05 <timburke> i think the only reservations i have with it now come down to naming (both for the config option and the cotnainer-metadata header)
21:50:09 <tdasilva> my hesitation with that patch is the need for a brand new middleware just for quoting etags. It feels like it should be an option on/off
21:50:34 <clayg> @rledisez what do you think about a global cluster wide option as well? sounds like legacy use-cases could still turn it off per account/container with the explicit metadata
21:50:56 <timburke> clayg, i already added that :-) defaults to off
21:51:06 <clayg> timburke: well then let's merge that shit!
21:51:44 <rledisez> tdasilva: i don't like the middleware neither, but a global option does not give the flexibility if per account/container. without that flexibility, we will breaks so many apps :(
21:52:11 <timburke> tdasilva, idk -- i kinda like the separation of concerns. *especially* given the granularity rledisez built in
21:53:11 <timburke> now, to take another look at torgomatic's https://review.opendev.org/#/c/504472/ ...
21:53:11 <patchbot> patch 504472 - swift - Shorten typical proxy pipeline. - 4 patch sets
21:53:37 <clayg> i don't mind having a small single purpose middleware - compared to what we do in bulk
21:54:30 <clayg> oh and even for clusters where "oh I definately want that on everywhere" - they may someday find one account/container that wants to use some old app that blows up with quoted etags - so I love having the explicit metadata options per collection
21:54:42 <rledisez> what I'm mostly concerned about middleware (except that paste does not really seem maintained) is the side effect of a bad ordered pipeline
21:55:09 <clayg> yeah, with the defaults should we think about making it auto-inserted?
21:55:49 <rledisez> timburke: about the naming of the header, I trust your english better than mine ;)
21:55:53 <timburke> rledisez, for *sure*. i think i've come across like 4 or 5 different bug reports with super-strange (and not necessarily overlapping!) errors that came down to "pipeline was bad"
21:55:54 <clayg> actually - that doesn't really help us... even if you auto-insert it you have to configure it
21:56:09 <rledisez> i'll just notify the only user of this feature to prepare before we upgrade
21:56:25 <clayg> pipelines suck..
21:56:29 <timburke> clayg, well, auto-inserted allows users to do self-service
21:56:50 <clayg> timburke: oh that's nice!
21:56:55 <timburke> rledisez, i'd also be ok with looking for an old header name for you. that's not a big deal
21:57:05 <timburke> *shrug*
21:57:12 <timburke> anyway, that's all i've got
21:57:16 <clayg> timburke: althought auto-insert migth not be really what I want... it be more like "auto-ORDERED" pipelines or osmething
21:57:18 <timburke> #topic open discussion
21:57:36 <timburke> clayg, auto-ordered pipelines would be *amazing*
21:57:47 <timburke> but maybe also crazy hard, i think
21:57:55 <rledisez> timburke: let's not polute the code, i'll maintain a small patch internaly until i can drop it
21:58:23 <timburke> and it'd require that we actually specify middleware dependencies
21:59:09 <timburke> maybe we could put some attributes on the filter_factories? start there?
21:59:10 <timburke> idk
21:59:21 <timburke> it'd be cool, though!
21:59:49 <mattoliverau> I had some code that did that.. just to play years ago.
22:00:00 <rledisez> timburke: for sure. even dropping the notion of pipeline by default and just have flags "enable_bulk", "enable_etag_quoter", …. keep the pipeline for advanced users
22:00:07 <mattoliverau> it might still be up in gerrit somewhere. Probably need alot of work
22:00:16 <rledisez> I think it was kinda like torgomatic's patch
22:00:24 <clayg> i gotta bounce
22:00:27 <timburke> all right, looks like we're out of time
22:00:37 <timburke> thank you all for coming, and thank you for working on swift!
22:00:38 <clayg> rledisez: can you drop a comment/review on it - i didn't really underestand it I don't think
22:00:54 <timburke> #endmeeting