21:06:34 <timburke> #startmeeting swift
21:06:34 <opendevmeet> Meeting started Wed Feb  8 21:06:34 2023 UTC and is due to finish in 60 minutes.  The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:06:34 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:06:34 <opendevmeet> The meeting name has been set to 'swift'
21:06:44 <timburke> who's here for the swift meeting?
21:06:51 <kota> o/
21:06:51 <mattoliver> o/
21:07:04 <indianwhocodes> o/
21:08:16 <timburke> i've been a bit distracted by hardware issues, so i haven't update the agenda
21:08:48 <mattoliver> nps, hows your laptop going?
21:08:56 <timburke> but it couldn't hurt to close the loop on the s3api CVE
21:10:19 <timburke> mattoliver, still dead-ish, but at least now i've got my desktop booting off the old hard drive -- feels much more like my work computer than before (when i was trying to do everything off my personal OS installs)
21:10:23 <timburke> #topic s3api CVE
21:11:06 <opendevreview> ASHWIN A NAIR proposed openstack/swift master: proxy-server exception logging shows replication_ip/port  https://review.opendev.org/c/openstack/swift/+/860866
21:11:07 <timburke> just to close this out -- a tag is now up (2.28.1) for stable/xena
21:11:31 <mattoliver> cool
21:11:55 <timburke> since that's the last non-extended-maintenance branch, no more tags are expected
21:12:26 <timburke> all the backports have landed (iirc that was already true last week, but never hurts to reiterate)
21:12:47 <mattoliver> So now no excuse, update your clusters peoples :)
21:12:59 <kota> great
21:13:08 <indianwhocodes> done already :wink
21:13:30 <mattoliver> Thanks for getting that all backported timburke
21:13:38 <acoles> +1
21:13:51 <timburke> ...and that's all there is to it! CVE resolved!
21:14:00 <mattoliver> \o/
21:14:10 <timburke> #topic March vPTG
21:14:33 <timburke> again, the etherpad to collect topics is up
21:14:35 <timburke> #link https://etherpad.opendev.org/p/swift-ptg-bobcat
21:14:48 <timburke> and acoles has already added some to it :-)
21:15:17 <mattoliver> nice! I'm still to dump whatever I can think of there... I'll try and get to that before next meeting
21:15:42 <indianwhocodes> same here, wanna have something more precisefirst
21:15:57 <timburke> i still need to add ring v2, and maybe a link to an ideas-style doc
21:16:20 <timburke> never hurts to add a placeholder, though, and fill in more info as you get there :D
21:17:01 <mattoliver> If ring v2 isn't there when I get to adding some topics I'll add it, but you can always update it
21:17:21 <timburke> 👍
21:17:27 <indianwhocodes> imo, the last etherpad link is useful too, ref: https://etherpad.opendev.org/p/swift-ptg-antelope
21:17:31 <mattoliver> (as I go through open patches from days old)
21:17:52 <mattoliver> what, you mean those are stil things :P
21:18:05 <timburke> i still haven't gotten to the time-slots poll, sorry -- seems likely that we'll settle on similar slots to the last few vPTGs
21:18:47 <timburke> oh yeah, if anybody's looking for even *older* etherpads, there's also https://wiki.openstack.org/wiki/Swift/Etherpads
21:19:44 <mattoliver> I've already added our news one to that
21:19:56 <mattoliver> *new ones
21:20:04 <timburke> i noticed someone had done it -- thanks mattoliver!
21:20:05 <indianwhocodes> Thanks Tim, only found the ops doc now, seems very relevant!
21:20:35 <timburke> that's all i've got right now -- who's got topics they'd like to bring up?
21:20:46 <timburke> #topic open discussion
21:21:09 <mattoliver> yeah, ops feedback is always really useful for thinking about pain points and future work
21:21:11 <mattoliver> I've been working on some more stuck shard ranges
21:21:25 <mattoliver> It's a weird edgecase. but something like
21:21:31 <mattoliver> #link https://review.opendev.org/c/openstack/swift/+/872852
21:22:30 <mattoliver> should stop them from getting into failing a sharding_enabled check and get to the audit code to pull in new shard ranges from root.
21:23:21 <indianwhocodes> been working on how i could increase the delay in between when the object is marked for expiry and when the expirer actually issues the delete
21:23:44 <indianwhocodes> by been working (I mean just started)
21:23:52 <mattoliver> The other fix was to add a db_state check in sharding_Enabled, but that would mean an extra directory listing check on every broker the sharder looks at (all of them). So this approach is a noop (well no extra io checks)
21:24:07 <mattoliver> indianwhocodes: lol, and cool
21:24:59 <indianwhocodes> Also if there is a way or an API we can introduce to recover expired data if the data they put under was incorrect
21:25:13 <mattoliver> why would we increase the delay between expiry deletes? is there a bug?
21:25:24 <mattoliver> oh I think you just answered that question
21:25:50 <mattoliver> can't you just put new or remove the expiry metadata, so a post?
21:26:06 <indianwhocodes> ya something like that
21:26:13 <mattoliver> or do you mean post delete
21:26:17 <timburke> can have things like a client that wants to extend the expiry, but there's a bunch of failures and they exhaust their retries
21:26:42 <indianwhocodes> "x-delete-at:" -XPOST
21:27:10 <indianwhocodes> but does it really work ? because i tired doing it and a head on the object still worked!
21:27:17 <indianwhocodes> needs further investigation
21:27:22 <timburke> pretty sure indianwhocodes is looking to introduce a recovery option in between the x-delete-at lapsing and the expirer actually laying down a tombstone
21:27:32 <indianwhocodes> yup
21:28:09 <indianwhocodes> many possible scenarios need to explain it better too, lol
21:28:23 <mattoliver> ok, so a POST after the x-delete-at date, the obj server would consider it already deleted
21:28:36 <mattoliver> even if it isn't, i guess
21:28:44 <indianwhocodes> ya!
21:28:48 <mattoliver> and it's a recover from that POV.
21:28:51 <mattoliver> I get it!
21:30:18 <mattoliver> well, you'd need to know it was in that state, maybe some kind of force. Or allow posts with x-delete-at's through the obj server barrier, the obj server is already looking at the x-delete-at date set on the object.. .
21:30:21 <timburke> so down in the diskfile, we check for past-expiry -- and the object-server will 404 any POST to avoid exactly this scenario
21:30:25 <mattoliver> anyway we can take this offline
21:30:25 <timburke> #link https://github.com/openstack/swift/blob/2.31.0/swift/obj/diskfile.py#L2736-L2750
21:30:52 <mattoliver> yeah
21:31:32 <timburke> but we could plumb in some (likely reseller-admin-gated) header to twiddle _open_expired
21:32:11 <mattoliver> yeah ok, I look forward to reviewing some options. Ping me if we want to chat more
21:32:25 <timburke> mattoliver, the sharding patch looks pretty clean -- did we already audit for other sharding-related sysmeta we should also add?
21:33:03 <mattoliver> Good thinking, let me double check, just in case before we land anything
21:35:05 <timburke> cleave context meta, maybe?
21:35:25 <mattoliver> Oh only other thing of note, I've been looking into otel metrics again, so going to have aplay with the api some in some downstream tests here at work, and I hope to turn that learning into how we'd go about things in Swift... just a heads up, and probably a topic for the ptg
21:35:50 <indianwhocodes> exciting!
21:35:57 <timburke> nice!
21:36:41 <timburke> as a heads-up, it's openstack election season again -- i'll try to get my self-nomination up by friday
21:37:11 <kota> nice, and thanks for continuing it
21:37:18 <mattoliver> Yeah timburke for PTL!
21:38:54 <timburke> all right, i'll let y'all go then
21:39:04 <timburke> thanks for coming, and thank you for working on swift!
21:39:10 <timburke> #endmeeting