21:01:22 <timburke> #startmeeting swift 21:01:23 <openstack> Meeting started Wed Dec 2 21:01:22 2020 UTC and is due to finish in 60 minutes. The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:01:24 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:01:27 <openstack> The meeting name has been set to 'swift' 21:01:31 <timburke> who's here for the swift meeting? 21:01:38 <zaitcev> o/ 21:01:41 <seongsoocho> o/ hi! 21:01:50 <kota_> o/ 21:01:52 <alecuyer> o/ 21:02:39 <mattoliverau> o/ 21:02:48 <rledisez> o/ 21:03:26 <clayg> o/ 21:03:30 <timburke> agenda's at https://wiki.openstack.org/wiki/Meetings/Swift 21:03:54 <timburke> sorry, i'm still catching up from the week off, so i haven't updated it 21:04:08 <timburke> #topic audit watchers 21:04:21 <timburke> mattoliverau, i saw you did some reviews! thanks! 21:05:04 <zaitcev> yeah, well... 21:05:17 <zaitcev> I'm looking at Matt's comments. 21:05:44 <zaitcev> Honestly I'm tempted to ask for a follow-up patch for the biggest of them, where he wants to reorganize the tests. 21:06:06 <mattoliverau> Sorry, but I think it's great and am almost ready to +2 21:07:17 <zaitcev> I wish we could just sic acoles on it. That man is a genius at refactoring our tests that are sometimes convoluted. 21:07:27 <timburke> out of curiosity, what sort of test watchers did you make, mattoliverau? 21:07:39 <mattoliverau> Yeah, if we want that in a follow up, seeing as it's just test reorg. 21:08:21 <timburke> i've been meaning to try some SLO-validator-type watchers... 21:08:43 <mattoliverau> Nothing amazing, just wrote one that logged things so I could get an idea of how it all worked, passed in some params via the calendar config. 21:08:53 <mattoliverau> Etc 21:09:16 <mattoliverau> Based it off the commit message code skel, which is why I found it wrong 21:09:32 <timburke> cool. i'll still try to get a fresh review in soon, too. sorry it's taking me so long, zaitcev 21:10:15 <mattoliverau> Should do something more interesting. But didn't want to take too long. 21:10:35 <zaitcev> I'm sure you will do the right thing. I kind of doubt myself because I want it "over", so my judgement may be off. But of course if I promise the follow-up, I'll do it. 21:12:48 <zaitcev> timburke: Anyway, I think we can move on. 21:13:15 <timburke> i agree that test re-org can and should be a follow-up, and i don't even know that having it proposed should be a blocker on landing what we've got. sounds like we all mostly know what we need to do next, yeah? 21:13:35 <timburke> (sorry for the delay, plumber finally came) 21:13:49 <zaitcev> the reality of life 21:14:11 <zaitcev> My biggest fear is that you'll find a cool application of watchers that requires them to be stateful 21:14:11 <timburke> #topic s3api, +segments, and ACLs 21:14:34 <clayg> i thought we merged something that address that 21:14:57 <timburke> clayg, i saw you +2'ed https://review.opendev.org/c/openstack/swift/+/763106/, any reason not to +A? 21:15:08 <timburke> (beyond general gate breakage, i suppose...) 21:15:32 <zaitcev> You're sure you're not thinking about PUT+POST where we proposed changes just to keep the state? Didn't merge though, it's sitting hibernating in that revew. 21:16:09 <clayg> nope, looks like a win to me 21:17:16 <timburke> zaitcev, i'm not opposed to proposing things that aren't necessarily going to land (hi, https://review.opendev.org/c/openstack/swift/+/648263 !). i do feel bad that we haven't done more with PUT+POST though, sorry :-( 21:18:14 <timburke> i just mean to say that we can take on the test re-org later, and having the follow-up written should not be a blocker 21:18:28 <timburke> #topic gate breakage 21:18:51 <timburke> *hopefully* this will all be resolved shortly, but i figure i ought to keep everyone informed 21:19:08 <zaitcev> I see Clay landed 763106 just now 21:19:39 <clayg> yeah that one looked straightforward to me - we're trying to get it running in prod currently 21:19:47 <clayg> but packaging software sucks 21:20:00 <clayg> but packaging^W software sucks 21:21:13 <timburke> recently, some of the integrated jobs broke because of some changes to the pip resolver -- details aren't so important (i think), but they should be fixed by https://review.opendev.org/c/openstack/devstack/+/764811 21:21:42 <clayg> yay! 21:23:08 <timburke> but we've also got *another* constantly-failing job with lower-constraints, where it'd try to install a py3-only version of keystoneclient on py2 21:23:34 <timburke> that one seems to be fixed by https://review.opendev.org/c/openstack/swift/+/764856, though it needs a recheck 21:24:00 <timburke> *hopefully* then we can finally start merging real patches again 21:25:00 <timburke> like i said, ideally you all won't need to think about it too much 21:25:10 <timburke> one last impromptu topic 21:25:25 <timburke> #topic part power increase 21:26:21 <timburke> clayg, acoles and i have been realizing we need to actually use this great feature cschwede wrote! 21:26:50 <timburke> and i thought i'd see if anyone else has used it much, and if so what tips they might have about it 21:26:51 <zaitcev> So, what's the issue? It worked in my tests. 21:27:01 <zaitcev> no, not "much", sorry 21:27:07 <mattoliverau> it is pretty great, but I'm only used in the dev. 21:27:26 <rledisez> we did it in production, multiple times. it worked. we have a doc with useful things to do before/after (to check everything went fine, etc…). Can paste that somewhere 21:27:42 <zaitcev> In our case it gets excercised because Director makes rings with just 9 bits 21:27:58 <mattoliverau> I always thought we'd use it and think of ways to make it more automatic rather then stopping replication engine.. also expend it to work with account and containers. 21:28:19 <mattoliverau> Because they sometimes need an increase too. 21:28:22 <timburke> cool, thanks rledisez! and thanks for https://review.opendev.org/c/openstack/swift/+/695344 :-) 21:29:04 <timburke> yeah, it's the "stopping replication engine" that has me the most worried. hard to do when you're *also* constantly expanding 21:29:43 <rledisez> From experience, with I/O limitation to not impact performance too much, it take about 24h/48h to do a relink 21:30:07 <clayg> rledisez: good info! how did you ratelimit the relinker? 21:30:37 <timburke> nice! yeah, we were thinking about adding a ratelimiting option... 21:30:50 <timburke> i'm also debating about trying to enable moving 2 or 3 steps at a time -- we weren't paying enough attention and have some catching up to do :-( 21:30:58 <rledisez> we start one relinker per device, each is put in a cgroup with io limitation. we have a script for that. I can share it too, there is no secret sauce in it :) 21:32:01 <timburke> oh yeah, and i guess i did https://review.opendev.org/c/openstack/swift/+/733296/ a while back... might want to merge that, too... 21:32:34 <timburke> all right, that's all i've got 21:32:39 <timburke> #topic open discussion 21:33:01 <timburke> anything else we ought to discuss this week? 21:34:40 <timburke> heads up on another py3 bug: https://launchpad.net/bugs/1906289 21:34:42 <openstack> Launchpad bug 1906289 in OpenStack Object Storage (swift) "Uploading a large object (SLO) in foreign language characters using S3 browser results in 400 BadRequest - Error in completing multipart upload" [High,Confirmed] 21:34:57 <timburke> have a fix already at https://review.opendev.org/c/openstack/swift/+/765204/ 21:35:14 <timburke> i'll plan on backporting as far back as train 21:37:12 <mattoliverau> ahh more wsgi str fun :P 21:37:19 <zaitcev> I'm still stuck with "secure" and "consistent" RBAC, basically tinkering with authorize(). 21:37:25 <acoles> hi sorry I'm late 21:37:31 <zaitcev> That code is scary, too much baggage. 21:38:52 <timburke> yeah, auth is thorny and we've (too) often neglected it 21:39:55 <timburke> reminds me that i ought to revisit https://review.opendev.org/#/c/620189/ and those RBAC tests we've got... 21:40:45 <zaitcev> Yeah, the curse of RBAC and how it made Ho-san quit 21:40:50 <zaitcev> Concerning! 21:42:26 <timburke> all right, i think that's about it then 21:42:37 <timburke> thank you all for coming, and thank you for working on swift! 21:42:41 <timburke> #endmeeting