21:09:50 <timburke> #startmeeting swift
21:09:50 <opendevmeet> Meeting started Wed Aug 14 21:09:50 2024 UTC and is due to finish in 60 minutes.  The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:09:50 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:09:50 <opendevmeet> The meeting name has been set to 'swift'
21:09:58 <timburke> who's here for the swift meeting?
21:10:06 <mattoliver> o/
21:10:09 <fulecorafa> o/
21:10:39 <acoles> \o
21:10:55 <acoles> I'm just here til 30 after the hour
21:11:32 <timburke> sorry for the delay -- between kids being home for this last week before going back to school and me getting nerd-sniped by some performance investigation, i'm not well prepared today ;-)
21:11:59 <mattoliver> lol, understandable!
21:12:00 <timburke> acoles, since you've got to cut out early anyway, is there anything you'd like to bring up?
21:13:19 <acoles> timburke: the patch to deprecate log_statsd_* options...
21:13:51 <acoles> https://review.opendev.org/c/openstack/swift/+/922518/
21:13:52 <patch-bot> patch 922518 - swift - statsd: deprecate log_ prefix for options - 11 patch sets
21:14:38 <acoles> clayg: made some good observations about the decision to blow up if both the old and new options were found in conf:
21:14:51 <acoles> 1. that seems too harsh if the old and new options have the same value!
21:15:15 <timburke> fair
21:15:40 <acoles> 2. that prevents ops adding config options *in anticipation* of upgrade / problematic if a node isn;t upgraded
21:16:20 <timburke> but then why move off the old options at all? suffer the warning until you can move with confidence!
21:18:08 <acoles> 3. with conf.d style it's easy to miss that there's a legacy option still in there (this would be mitigated by tolerating same value)
21:18:51 <acoles> we noted that if we turned it down to just a warning, then worst case some stats stop appearing and ops check logs and see warning and go fix conf
21:19:13 <acoles> anyways, I wanted to solicit more opinion...
21:20:00 <timburke> eh, it's certainly not a hill i'm looking to die on
21:20:01 <acoles> I certainly think that tolerating old-and-new-but-same-value without blowing up seems reasonable
21:20:34 <acoles> hehe, I'm not sure anyone wants to dis on this particular hill ;-)
21:20:56 <timburke> tolerating both, even with different values seems perfectly in line with other times that we've tried to push for renaming options
21:21:51 <acoles> we'd obvs make the warning pretty clear - "you gave me x and y and I'm ignoring y"
21:22:03 <mattoliver> yeah supporting new but falling back to old seems the normal migration path. With warnings to me makes sense. But maybe I'm missing something obvious (it is early) :P
21:23:10 <acoles> the context is this comment https://review.opendev.org/c/openstack/swift/+/922518/comment/56eb874d_9707e7c5/
21:23:11 <patch-bot> patch 922518 - swift - statsd: deprecate log_ prefix for options - 11 patch sets
21:24:00 <acoles> we're concerned that an op might update to new option in DEFAULT and not realise it overrides what was previously a different value in proxy-logging
21:25:24 <acoles> which led us to "blow up if old and new are present" ... but now reflecting on whether that is too brittle
21:25:37 <timburke> i think my perspective when thinking about going for an error was if you've got log_statsd_host=foo, log_statsd_port=12345, statsd_host=bar -- that having some but not all options coming from the correct place seems likely a misconfiguration
21:26:51 <acoles> yup, but does misconfiguration result in warning or ValueError ???
21:27:15 <mattoliver> oh I see. But if they know they're moving to the new (and done so in DEFAULT) isn't it on them to finish the job.. I think warning of settings is enough, to alert them.
21:27:33 <mattoliver> But I guess it depends on where we think metrics sit on importance in swift
21:27:50 <mattoliver> if it's critial then it should ValueError
21:28:21 <mattoliver> but it seems kinda optional (which maybe wrong) so warning is enough
21:29:08 <timburke> maybe my real concern is that i want a statsd_endpoint config option that looks like host:port -- which is definitely out of scope :P
21:29:49 <mattoliver> Downstream our metrics are super important, so I'd assume ValueError would be more useful, because we may not check/see warning for some time.
21:30:45 <mattoliver> so yeah, I think I finally clicked onto the delema :P
21:31:31 <timburke> eh, the warning's probably fine
21:31:37 <timburke> i've probably been overthinking it
21:31:40 <acoles> worth noting that downstream, with ValueError approach, we might have to do *work* (i.e. rip stuff out of controller) before we could use new metrics via ansible conf.d, so we might be stuck with legacy for a while
21:31:47 <mattoliver> gut still feels like a warning, as thats more historic swift
21:33:03 <acoles> tell you what - I'll spin a warning only version and we can go back and forth more on gerrit - I've already used half the meeting time :)
21:33:10 <mattoliver> maybe we need a config check tool.. maybe swift-config can do the audit
21:33:16 <timburke> no worries
21:33:23 <mattoliver> +1
21:34:33 <timburke> ok, next up
21:34:42 <mattoliver> get swift-config to print dep warnings and tell users they should always run new config through swift-config :)
21:35:14 <timburke> oh, i think i like that... we should probably use that tool more
21:35:40 <timburke> i realized that we're getting toward the end of the dalmatian cycle
21:35:47 <timburke> #link https://releases.openstack.org/dalmatian/schedule.html
21:36:11 <timburke> i need to be getting a client release together soonish
21:36:47 <mattoliver> kk, let's fire up priority reviews page :)
21:36:47 <timburke> i *did* finally get the liberasurecode release out that i'd meant to do months ago at least :-)
21:37:00 <mattoliver> \o/
21:37:30 <timburke> and within the next couple weeks or so we should get a swift release out
21:38:39 <mattoliver> Then I want to make sure the account-qoutas follow up lands before then.
21:38:46 <timburke> +1
21:38:57 <timburke> i should take another look at that, too
21:39:07 <mattoliver> me 3
21:39:58 <timburke> we probably ought to make a decision on https://review.opendev.org/c/openstack/swift/+/924795 by then, too -- either remove now, or add the warnings before release
21:39:58 <patch-bot> patch 924795 - swift - Remove legacy bin/ scripts - 3 patch sets
21:40:16 <mattoliver> yes. I added my vote ;)
21:40:39 <timburke> i saw! thanks
21:40:58 <mattoliver> maybe we give it a timelimit. and count the +/- 1's
21:41:04 <timburke> i'm sad we still haven't been able to feel comfortable merging https://review.opendev.org/c/openstack/swift/+/853590
21:41:05 <patch-bot> patch 853590 - swift - Drop py2 support - 15 patch sets
21:41:06 <mattoliver> say by next meeting.
21:42:13 <mattoliver> ohh, maybe we should.. or is that rocking the boat too much and to quickly :P
21:42:38 <mattoliver> or do we merge that just after this release.. and that's the line?
21:43:07 <timburke> my main concern is similar to acoles's on the config warning/error question -- would it just be setting us up to have to do work downstream to work around it...
21:43:46 <mattoliver> well yeah, but that's downstream, and maybe the kick in the pants we need :P
21:45:10 <timburke> i suppose we could always merge it, then immediately propose a revert, which we can carry as needed :P
21:45:23 <mattoliver> lol
21:46:40 <mattoliver> well we have a few weeks at the most, let's probe downstream and see where the current blockers are.. but it would be nice to not worry about py2 anymore. I don't think anyone upstream in the community other then us has some dep issues on py2  code in an older codebase
21:47:01 <timburke> no, i don't believe so either
21:48:49 <timburke> next ptg, we should have a straw poll on oldest python version that should still be supported by master -- because i think we could probably drop py36(and maybe even py37/py38?) as well
21:48:50 <mattoliver> k, lets put a pin in it for this meeting an re-discuss next when we might have some more data.
21:49:06 <mattoliver> oh good idea
21:49:17 <timburke> all right, i think those are the main things i've got
21:49:25 <timburke> anything else we should bring up this week?
21:49:37 <fulecorafa> If I may
21:49:38 <zaitcev> Enumerate the distros you're still willing to support, find what Python they ship, and there's your answer.
21:50:18 <timburke> fulecorafa, go ahead!
21:50:37 <fulecorafa> We're having some problems with some users either deleting enourmous files or deleting a large quantity of them. Essentially any delete objects that takes some time to resolve the HTTP request
21:51:17 <fulecorafa> From what I've tested, it seems like it is a simple problem of connection timeout because the operation takes a long time
21:51:45 <fulecorafa> However, I think this should open the possibility of making deletions async
21:52:21 <mattoliver> yeah interesting. async deletion. I guess the question is what status code do you get.
21:52:45 <mattoliver> did build delete ever get the keep-alive heartbeat love.
21:52:52 <mattoliver> *bulk delete
21:53:04 <timburke> 204 Accepted -- good for so many things :D
21:53:33 <fulecorafa> For the actual implementation, I think I would go acoles direction to make deletion markers. Although I remember there is something similar already there, even though I didn't find it around in the repo lately...
21:53:51 <mattoliver> well maybe passing in a query arg to indicate an async delete might be ok
21:54:00 <zaitcev> As a workaround, could you do a bulk delete with just 1 entry?
21:54:29 <fulecorafa> That is an idea zaitcev and mattoliver, didn't try that yet
21:54:31 <mattoliver> yeah possibly
21:54:31 <timburke> mattoliver, i'd forgotten that heartbeating was opt-in -- but yeah, pretty sure bulk delete and slo both have it now
21:54:40 <fulecorafa> Will check the possibility
21:54:50 <timburke> fulecorafa, are your users using swift or s3 api? or both?
21:54:59 <fulecorafa> s3api mostly
21:55:04 <zaitcev> oh
21:55:16 <fulecorafa> Yep
21:55:17 <zaitcev> I'm not sure if ?bulk-delete is accessible through S3.
21:55:47 <zaitcev> Sorry I just wanted to get your users going while we're tinkering.
21:55:53 <mattoliver> otherwise dont mind the idea of async just need to think about how. drop a tombstone and wait for something to clean it up, drop it into expirer queue
21:55:53 <fulecorafa> There is a multi-delete controller, but we're having problem with that too
21:56:28 <timburke> oh, no! i was wrong about bulk -- and was in fact thinking about slo PUTs! https://github.com/openstack/swift/blob/master/swift/common/middleware/slo.py#L95-L100
21:56:29 <fulecorafa> One idea I was having, such that we could give backwards support for today's s3api
21:57:04 <mattoliver> timburke: yeah I was wondering if that was something I was going to add to the summer school students and why it was in my head :P
21:57:36 <fulecorafa> You can send a combination of config and query param requesting for async, creating controllers for async where we would want it. The configuration defaults the behaviour, while the query param overwrites it
21:57:43 <mattoliver> i beleive there is a a feature request for it
21:59:02 <mattoliver> fulecorafa: sounds like we're on board, maybe we need to write up something (bug/feature request or wait until next meeting to discuss further) so we can continue the disucssion async
21:59:07 <mattoliver> see what I did there :P
21:59:34 <mattoliver> but think it's a great idea, and useful feature as objects can get pretty big :)
22:00:13 <fulecorafa> Thanks mattoliver. Wanted to be sure this was not available today. Since it is a nice touch, I will open a feature request for that soon than
22:00:26 <mattoliver> ta
22:00:36 <mattoliver> I think we're at time
22:01:15 <mattoliver> I did want to mention that I might have a patch to solving the early active issue we see in getting auto shrinking happening in sharding: https://review.opendev.org/c/openstack/swift/+/926036
22:01:16 <patch-bot> patch 926036 - swift - ShardRange: track last state change timestamp - 3 patch sets
22:01:37 <timburke> fulecorafa, you said it happens when deleting enormous files -- is allow_async_delete configured for slo? it should default to on which is what you'd want https://github.com/openstack/swift/blob/master/etc/proxy-server.conf-sample#L1114
22:02:18 <timburke> s3api should be trying to use that functionality: https://github.com/openstack/swift/blob/2.33.0/swift/common/middleware/s3api/s3request.py#L1518-L1535
22:03:20 <mattoliver> oh good call, I guess they probably are SLOs
22:03:53 <fulecorafa> Thx timburke, I didn't check that, didn't remember that or we're in an old version that it didn't appear to me.
22:04:39 <timburke> oh, and bulk *always* wants to do that, but the option is called yield_frequency https://github.com/openstack/swift/blob/master/etc/proxy-server.conf-sample#L1054-L1057
22:05:15 <timburke> but i don't think s3api's bulk-delete-equivalent uses that
22:06:05 <fulecorafa> It doesn't ;-;
22:07:20 <timburke> the complete-multipart-upload code might be a useful starting point for similar functionality: https://github.com/openstack/swift/blob/2.33.0/swift/common/middleware/s3api/controllers/multi_upload.py#L788-L818
22:07:56 * mattoliver needs to go wrangle kids and get them ready for school. So I gotta drop.
22:08:12 <timburke> all right, mattoliver's right, we're past time now -- i should wrap up
22:08:22 <timburke> thank you all for coming, and thank you for working on swift!
22:08:26 <timburke> #endmeeting