21:04:31 <timburke> #startmeeting swift
21:04:31 <opendevmeet> Meeting started Wed Mar  8 21:04:31 2023 UTC and is due to finish in 60 minutes.  The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:04:31 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:04:31 <opendevmeet> The meeting name has been set to 'swift'
21:04:39 <timburke> who's here for the swift team meeting?
21:04:55 <mattoliver> o/
21:06:40 <acoles> o/ sorry I'm late
21:07:06 <timburke> quickly going over old business...
21:07:15 <timburke> #topic recovering expired objects
21:08:12 <timburke> looks like indianwhocodes has been getting some reviews from acoles -- i think we're planning on trying this out with our prod cluster soon
21:09:12 <timburke> #topic ssync, timestamp offsets, and .metas
21:09:25 <mattoliver> oh my
21:10:19 <timburke> we recently rolled that out to prod; has it resolved the problem we were seeing?
21:11:11 <mattoliver> thats a good question, Al any idea? I haven't heard anything, which could be a good thing :)
21:11:57 <acoles> AFAICT the problem got resolved
21:12:43 <acoles> I'm hunting down firm evidence
21:15:35 <timburke> well, we'll keep looking -- once we know either way, let's leave a comment on the review -- assuming it resolved it, i think we're good to merge
21:16:08 <timburke> #topic http keepalive timeout
21:16:34 <timburke> i still need to respin the patch to leverage eventlet's new option
21:16:37 <timburke> #link https://review.opendev.org/c/openstack/swift/+/873744
21:17:08 <timburke> #topic per policy quotas
21:17:39 <timburke> the pre-reqs have merged, and mattoliver's taken a look at the main patch
21:17:41 <timburke> #link https://review.opendev.org/c/openstack/swift/+/861282
21:18:19 <mattoliver> yup, it looks good. Will have a play with it in my VSAIO today then I should be able to +2 it I think
21:18:34 <timburke> he's also pushed up a follow-up to get the account-wide limit out of user meta and into sysmeta
21:18:37 <timburke> #link https://review.opendev.org/c/openstack/swift/+/876821
21:18:43 <timburke> thanks, mattoliver!
21:19:28 <mattoliver> nps, or at least make it so we use the same metadata namespace for all account quotas, so it's not a mismash of user meta and sysmeta but not really user meta.
21:21:10 <acoles> +1
21:21:14 <timburke> that's it for items from last week
21:21:38 <timburke> new business
21:21:53 <timburke> #topic sysmeta updates and container Last-Modified
21:23:43 <timburke> acoles had some users notice that we were updating a container's last modified due to internal POSTs to sysmeta; this was breaking some workflow that seemed to rely on clients doing container-level PUTs/POSTs as part of uploads so they could cheaply watch for changes
21:24:40 <timburke> we're thinking about how to avoid that, and also how to better expose and define some container-level timestamps for clients
21:24:43 <timburke> #link https://review.opendev.org/c/openstack/swift/+/875982
21:25:14 <acoles> yeah. So there's a few places where we use sysmeta to manage internal state, but so far in upstream swift I think we only ever modify it directly on the DB
21:26:30 <acoles> In our use case we POST sysmeta to the container server, and the server updates the put_timestamp as a side-effect. Given that this sysmeta is not user-visible, it's not necessary or appropriate, and the user would prefer the last-modified time to not be changed
21:27:57 <mattoliver> yeah interesting. Because on one hand the db has been changed.. on the other the user can't see the change.
21:28:21 <timburke> fwiw, it seems to me that there are at least three timestamps clients would likely be interested in: when a container was created, when a client last changed some container metadata (regardless of whether it was stored in user or sysmeta), and when the last write into the container occurred (regardless of whether it was an object PUT, POST, or DELETE)
21:28:51 <acoles> PUT with shard ranges already bypasses the put_timestamp update
21:29:58 <timburke> our user was put out because we changed (2) despite no client action, but it sounds like what they really want is (3)
21:29:58 <acoles> and the sharder write sysmeta (like sharding contexts) but without modifying put_timestamp
21:30:47 <mattoliver> ahh prior art.. that makes me feel better (or bad for not updating the ts :P )
21:30:56 <mattoliver> kk
21:32:02 <acoles> mattoliver: don't feel bad, I'm not sure there is a reason to update it for internal state changes (and actually, good reasons to not update it)
21:32:58 <timburke> don't think there's too much to bring up here for now, but interested parties should follow the review
21:33:15 <mattoliver> cool. Well I'll take a look at the patch
21:33:31 <timburke> #topic wsgi manager-process cpu usage
21:34:07 <mattoliver> Ahh I have this patch open in a tab right now. This is pretty cool.
21:34:41 <timburke> i brought up a vsaio recently and was noticing that virtualbox seemed fairly busy despite not actually doing much
21:35:25 <mattoliver> #link https://review.opendev.org/c/openstack/swift/+/876662
21:35:34 <timburke> digging into it a bit, i saw that each of the 14 wsgi-manager processes were consuming 1-2% cpu
21:35:54 <timburke> which seemed to be because of a busy-loop in eventlet.green.os.wait
21:37:05 <timburke> i'm pretty sure the only reason we're touching eventlet there is to be able to do that wait with a timeout -- there are no other greenthreads running in that process
21:37:48 <timburke> so i took a stab at removing that use of eventlet by using signaled alarms
21:38:38 <timburke> not sure how much it affects a "real" system, but if you're ever cpu-bound, i'd imagine the proposed patch ought to help a bit
21:40:14 <timburke> all right, i think that's all i've got
21:40:16 <mattoliver> it's really interesting stuff. How'd it effect your VSAIO?
21:40:18 <timburke> #topic open discussion
21:41:25 <timburke> oh, it was great! from my host, virtualbox went from like 30% cpu to 3-5%
21:41:50 <timburke> anything else we should bring up this week?
21:42:52 <mattoliver> I'm playing with object-updater + memcache updates, but still needs more work, we can talk about it next week or at the PTG :)
21:43:57 <mattoliver> I've broken it up in to some refactors first, chain starts:
21:44:15 <mattoliver> or rather ends: https://review.opendev.org/c/openstack/swift/+/87472
21:44:23 <mattoliver> #link https://review.opendev.org/c/openstack/swift/+/87472
21:45:29 <mattoliver> just in case anyone is insterested. That's all I can think of for now
21:46:40 <timburke> all right, i think i'll call it then
21:46:50 <timburke> thank you all for coming, and thank you for working on swift!
21:46:54 <timburke> #endmeeting