21:01:58 <timburke> #startmeeting swift
21:01:58 <opendevmeet> Meeting started Wed Jun  7 21:01:58 2023 UTC and is due to finish in 60 minutes.  The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:01:58 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:01:58 <opendevmeet> The meeting name has been set to 'swift'
21:02:06 <timburke> who's here for the swift meeting?
21:02:34 <seongsoocho> o/
21:02:45 <kota> good morning
21:04:29 <timburke> i didn't get around to updating the agenda, but i've only really got one new thing to bring up
21:04:34 <timburke> first, tho...
21:04:51 <timburke> #topic ssync metadata corruption on py3
21:04:56 <timburke> #link https://review.opendev.org/c/openstack/swift/+/884240
21:05:35 <timburke> i need to get some more reviews on this, and probably more tests around some of the other places i needed to touch
21:06:10 <timburke> there's a follow-up probe test change
21:06:14 <timburke> #link https://review.opendev.org/c/openstack/swift/+/884954
21:06:32 <opendevreview> Merged openstack/liberasurecode master: Replace Flat XOR link with an archived version  https://review.opendev.org/c/openstack/liberasurecode/+/869266
21:07:32 <timburke> i'm a little torn about it, though, since i pulled out a check that we can retrieve the reconstructed object via the proxy
21:08:04 <timburke> ...because requests/urllib3/stdlib can't parse utf8 in header names
21:08:22 <zaitcev> I don't see any lost tests in 884240.
21:09:14 <zaitcev> https://review.opendev.org/c/openstack/swift/+/884240/4/test/unit/obj/test_ssync_sender.py
21:09:43 <timburke> no -- the change is in the probe test follow-up; it no longer checks that we get expected meta back through the proxy, but instead relies on direct_client responses
21:09:51 <timburke> https://review.opendev.org/c/openstack/swift/+/884954/3/test/probe/test_reconstructor_revert.py#b97
21:10:18 <mattoliver> o/ (sorry im late)
21:12:11 <timburke> so if you've got qualms or ideas for that, let me know on the review
21:13:05 <timburke> #topic missing account/container info request logging & metrics
21:13:10 <timburke> #link https://review.opendev.org/c/openstack/swift/+/884931
21:13:56 <timburke> acoles and clayg have taken a look, had some feedback
21:15:20 <timburke> hopefully we can get a fix in soonish; it makes me nervous that i caused us to lose some telemetry like that
21:16:39 <timburke> i don't know that there's much more to say about it, though, other than to beg reviews ;-)
21:16:57 <timburke> so my new topic...
21:17:09 <timburke> #topic labeled metrics
21:17:45 <timburke> i think i mentioned this as a thing i was playing with a while ago, but hadn't gotten a patch up y'all could look at
21:18:01 <timburke> but now i have :-) https://review.opendev.org/c/openstack/swift/+/885321
21:18:38 <mattoliver> Yay, it's a big one, but that's expected for what it does.
21:19:27 <mattoliver> I have it on my list to review.. just been distracted with some downstream work.
21:19:44 <mattoliver> Is there anything special we need to do to use it in a saio?
21:19:52 <timburke> it's still very much a work in progress -- there's a bunch of places where i raise errors that we won't want when it lands, gotta think through the upgrade path for 3rd party middlewares more, got a heap of unit tests to fix/adjust
21:20:13 <mattoliver> Kk
21:21:04 <timburke> i'm going to work on getting a doc change up to talk about adding a lightweight metrics stack to a SAIO -- mostly centered around https://github.com/prometheus/statsd_exporter and prometheus
21:23:42 <timburke> longer term, we might want to be able to switch between statsd extensions (which that patch offers) and gRPC/OTLP (via some of the opentelemetry libraries mattoliver is looking at for tracing)
21:24:31 <mattoliver> Cool, yeah that'll make it easier to review and take for a spin 😀 (statsdexporter and doc)
21:25:13 <timburke> but my hope is that the metrics api calls we make in swift code won't need to change to do that, assuming the statsd extensions land first
21:25:34 <timburke> i don't really want to touch all the world again like that :-(
21:26:23 <mattoliver> What ever library we end up using is something we can worry about later, getting multilabel metrics integrated into swift if important.
21:27:49 <timburke> the big thing i want out of it is to give operators the ability to opt in to much higher granularity metrics -- not just see how many req/s the cluster as a whole is doing, but how many are for each account, or even container
21:28:09 <mattoliver> +100
21:28:55 <timburke> and apply that throughout -- get account/container info on what the updaters or the sharders are processing, etc
21:30:00 <timburke> if you want to start looking at it, i'd look at the statsd client changes in https://review.opendev.org/c/openstack/swift/+/885321/2/swift/common/utils/__init__.py
21:30:33 <mattoliver> Kk
21:30:39 <timburke> and maybe proxy logging as an example of how we'd update callers: https://review.opendev.org/c/openstack/swift/+/885321/2/swift/common/middleware/proxy_logging.py
21:31:07 <timburke> that's all i've got
21:31:13 <timburke> #topic open discussion
21:31:21 <timburke> anything else we want to bring up this week?
21:33:28 <mattoliver> Nope, like I said I've been a little distractedthis week, I am attempting to push tracing a bunch more, so look forward to me talking and asking for reviews on that soon 😜
21:34:41 <mattoliver> Like multi label statsd metrics, it's a big and over arching change.. so fin times ahead, but the result is pretty traces 😉
21:34:53 <timburke> sounds good to me :-)
21:35:17 <timburke> all right -- i think i'll wrap the meeting early
21:35:30 <timburke> thank you all for coming, and thank you for working on swift!
21:35:34 <timburke> #endmeeting