21:01:22 <timburke> #startmeeting swift
21:01:23 <openstack> Meeting started Wed Dec  2 21:01:22 2020 UTC and is due to finish in 60 minutes.  The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:01:24 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:01:27 <openstack> The meeting name has been set to 'swift'
21:01:31 <timburke> who's here for the swift meeting?
21:01:38 <zaitcev> o/
21:01:41 <seongsoocho> o/ hi!
21:01:50 <kota_> o/
21:01:52 <alecuyer> o/
21:02:39 <mattoliverau> o/
21:02:48 <rledisez> o/
21:03:26 <clayg> o/
21:03:30 <timburke> agenda's at https://wiki.openstack.org/wiki/Meetings/Swift
21:03:54 <timburke> sorry, i'm still catching up from the week off, so i haven't updated it
21:04:08 <timburke> #topic audit watchers
21:04:21 <timburke> mattoliverau, i saw you did some reviews! thanks!
21:05:04 <zaitcev> yeah, well...
21:05:17 <zaitcev> I'm looking at Matt's comments.
21:05:44 <zaitcev> Honestly I'm tempted to ask for a follow-up patch for the biggest of them, where he wants to reorganize the tests.
21:06:06 <mattoliverau> Sorry, but I think it's great and am almost ready to +2
21:07:17 <zaitcev> I wish we could just sic acoles on it. That man is a genius at refactoring our tests that are sometimes convoluted.
21:07:27 <timburke> out of curiosity, what sort of test watchers did you make, mattoliverau?
21:07:39 <mattoliverau> Yeah, if we want that in a follow up, seeing as it's just test reorg.
21:08:21 <timburke> i've been meaning to try some SLO-validator-type watchers...
21:08:43 <mattoliverau> Nothing amazing, just wrote one that logged things so I could get an idea of how it all worked, passed in some params via the calendar config.
21:08:53 <mattoliverau> Etc
21:09:16 <mattoliverau> Based it off the commit message code skel, which is why I found it wrong
21:09:32 <timburke> cool. i'll still try to get a fresh review in soon, too. sorry it's taking me so long, zaitcev
21:10:15 <mattoliverau> Should do something more interesting. But didn't want to take too long.
21:10:35 <zaitcev> I'm sure you will do the right thing. I kind of doubt myself because I want it "over", so my judgement may be off. But of course if I promise the follow-up, I'll do it.
21:12:48 <zaitcev> timburke: Anyway, I think we can move on.
21:13:15 <timburke> i agree that test re-org can and should be a follow-up, and i don't even know that having it proposed should be a blocker on landing what we've got. sounds like we all mostly know what we need to do next, yeah?
21:13:35 <timburke> (sorry for the delay, plumber finally came)
21:13:49 <zaitcev> the reality of life
21:14:11 <zaitcev> My biggest fear is that you'll find a cool application of watchers that requires them to be stateful
21:14:11 <timburke> #topic s3api, +segments, and ACLs
21:14:34 <clayg> i thought we merged something that address that
21:14:57 <timburke> clayg, i saw you +2'ed https://review.opendev.org/c/openstack/swift/+/763106/, any reason not to +A?
21:15:08 <timburke> (beyond general gate breakage, i suppose...)
21:15:32 <zaitcev> You're sure you're not thinking about PUT+POST where we proposed changes just to keep the state? Didn't merge though, it's sitting hibernating in that revew.
21:16:09 <clayg> nope, looks like a win to me
21:17:16 <timburke> zaitcev, i'm not opposed to proposing things that aren't necessarily going to land (hi, https://review.opendev.org/c/openstack/swift/+/648263 !). i do feel bad that we haven't done more with PUT+POST though, sorry :-(
21:18:14 <timburke> i just mean to say that we can take on the test re-org later, and having the follow-up written should not be a blocker
21:18:28 <timburke> #topic gate breakage
21:18:51 <timburke> *hopefully* this will all be resolved shortly, but i figure i ought to keep everyone informed
21:19:08 <zaitcev> I see Clay landed 763106 just now
21:19:39 <clayg> yeah that one looked straightforward to me - we're trying to get it running in prod currently
21:19:47 <clayg> but packaging software sucks
21:20:00 <clayg> but packaging^W software sucks
21:21:13 <timburke> recently, some of the integrated jobs broke because of some changes to the pip resolver -- details aren't so important (i think), but they should be fixed by https://review.opendev.org/c/openstack/devstack/+/764811
21:21:42 <clayg> yay!
21:23:08 <timburke> but we've also got *another* constantly-failing job with lower-constraints, where it'd try to install a py3-only version of keystoneclient on py2
21:23:34 <timburke> that one seems to be fixed by https://review.opendev.org/c/openstack/swift/+/764856, though it needs a recheck
21:24:00 <timburke> *hopefully* then we can finally start merging real patches again
21:25:00 <timburke> like i said, ideally you all won't need to think about it too much
21:25:10 <timburke> one last impromptu topic
21:25:25 <timburke> #topic part power increase
21:26:21 <timburke> clayg, acoles and i have been realizing we need to actually use this great feature cschwede wrote!
21:26:50 <timburke> and i thought i'd see if anyone else has used it much, and if so what tips they might have about it
21:26:51 <zaitcev> So, what's the issue? It worked in my tests.
21:27:01 <zaitcev> no, not "much", sorry
21:27:07 <mattoliverau> it is pretty great, but I'm only used in the dev.
21:27:26 <rledisez> we did it in production, multiple times. it worked. we have a doc with useful things to do before/after (to check everything went fine, etc…). Can paste that somewhere
21:27:42 <zaitcev> In our case it gets excercised because Director makes rings with just 9 bits
21:27:58 <mattoliverau> I always thought we'd use it and think of ways to make it more automatic rather then stopping replication engine.. also expend it to work with account and containers.
21:28:19 <mattoliverau> Because they sometimes need an increase too.
21:28:22 <timburke> cool, thanks rledisez! and thanks for https://review.opendev.org/c/openstack/swift/+/695344 :-)
21:29:04 <timburke> yeah, it's the "stopping replication engine" that has me the most worried. hard to do when you're *also* constantly expanding
21:29:43 <rledisez> From experience, with I/O limitation to not impact performance too much, it take about 24h/48h to do a relink
21:30:07 <clayg> rledisez: good info!  how did you ratelimit the relinker?
21:30:37 <timburke> nice! yeah, we were thinking about adding a ratelimiting option...
21:30:50 <timburke> i'm also debating about trying to enable moving 2 or 3 steps at a time -- we weren't paying enough attention and have some catching up to do :-(
21:30:58 <rledisez> we start one relinker per device, each is put in a cgroup with io limitation. we have a script for that. I can share it too, there is no secret sauce in it :)
21:32:01 <timburke> oh yeah, and i guess i did https://review.opendev.org/c/openstack/swift/+/733296/ a while back... might want to merge that, too...
21:32:34 <timburke> all right, that's all i've got
21:32:39 <timburke> #topic open discussion
21:33:01 <timburke> anything else we ought to discuss this week?
21:34:40 <timburke> heads up on another py3 bug: https://launchpad.net/bugs/1906289
21:34:42 <openstack> Launchpad bug 1906289 in OpenStack Object Storage (swift) "Uploading a large object (SLO) in foreign language characters using S3 browser results in 400 BadRequest - Error in completing multipart upload" [High,Confirmed]
21:34:57 <timburke> have a fix already at https://review.opendev.org/c/openstack/swift/+/765204/
21:35:14 <timburke> i'll plan on backporting as far back as train
21:37:12 <mattoliverau> ahh more wsgi str fun :P
21:37:19 <zaitcev> I'm still stuck with "secure" and "consistent" RBAC, basically tinkering with authorize().
21:37:25 <acoles> hi sorry I'm late
21:37:31 <zaitcev> That code is scary, too much baggage.
21:38:52 <timburke> yeah, auth is thorny and we've (too) often neglected it
21:39:55 <timburke> reminds me that i ought to revisit https://review.opendev.org/#/c/620189/ and those RBAC tests we've got...
21:40:45 <zaitcev> Yeah, the curse of RBAC and how it made Ho-san quit
21:40:50 <zaitcev> Concerning!
21:42:26 <timburke> all right, i think that's about it then
21:42:37 <timburke> thank you all for coming, and thank you for working on swift!
21:42:41 <timburke> #endmeeting