21:01:34 <timburke> #startmeeting swift
21:01:34 <opendevmeet> Meeting started Wed May 24 21:01:34 2023 UTC and is due to finish in 60 minutes.  The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:01:34 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:01:34 <opendevmeet> The meeting name has been set to 'swift'
21:01:41 <timburke> who's here for the swift meeting?
21:03:21 <mattoliver> o/
21:03:41 <kota> o/
21:05:11 <zaitcev> o/ ...
21:05:29 <timburke> sorry, distracted for a sec
21:05:35 <timburke> as usual, the agenda's over at
21:05:46 <timburke> #link https://wiki.openstack.org/wiki/Meetings/Swift
21:06:18 <timburke> first up
21:06:28 <timburke> #topic non-cooperative EC PUTs
21:06:39 <timburke> #link https://review.opendev.org/c/openstack/swift/+/883367
21:06:56 <timburke> has merged! thanks acoles for running down the issue
21:07:18 <mattoliver> \o/
21:07:45 <timburke> in retrospect, it's kind of surprising it took us this long to bump into it
21:07:59 <timburke> next
21:08:03 <timburke> #topic netifaces
21:08:10 <timburke> #link https://review.opendev.org/c/openstack/swift/+/873222
21:08:30 <timburke> has also merged! thanks to everybody for reviewing it
21:08:43 <mattoliver> nice
21:08:56 <timburke> (honestly, it got more eyes on it than i was expecting)
21:10:18 <timburke> i'll rebase the follow-up to fully remove netifaces soon, but we'll still give ops some time to see the warning and file a bug before actually looking to merge
21:10:35 <timburke> #link https://review.opendev.org/c/openstack/swift/+/883035
21:10:54 <mattoliver> yeah good plan
21:11:09 <timburke> #topic ssync metadata corruption on py3
21:11:39 <timburke> edausq wrote up
21:11:41 <timburke> #link https://bugs.launchpad.net/swift/+bug/2020667
21:12:10 <timburke> and there's even a fix already proposed
21:12:14 <timburke> #link https://review.opendev.org/c/openstack/swift/+/884240
21:13:12 <mattoliver> oh cool, love it when ovh find something! they always also propose a solution!
21:13:40 <timburke> i'll try to take a look at fixing up unit tests for next week, maybe enhance some probe tests to exercise it
21:15:14 <timburke> unfortunately, i'm not sure there's much we can do to address corrupted data...
21:17:23 <timburke> *maybe* you could compare meta across different replicas? as long as one of them still has the original meta, you might be able to tell what it was supposed to be...
21:19:33 <timburke> i think it might also be the sort of thing where it continues to get worse over time -- if there have been a bunch of rebalances, the bug could trigger multiple times, increasing the size of the metadata each time...
21:20:16 <timburke> all the more reason i want to look at reproing in a probe test
21:20:48 <timburke> all right, that's all i've got for this week
21:20:52 <timburke> #topic open discussion
21:21:01 <mattoliver> yeah, probe test would be a good start in anycase
21:21:18 <timburke> anything else we should bring up?
21:21:59 <mattoliver> We have stats being emitted when sharding happens so we can track timings (thanks JianJian)
21:23:00 <mattoliver> but there was a bug where timing_since couldn't read Timestamp objects. So fixed it (both ways). The more accepted approach is type casting the epoch to a float in: https://review.opendev.org/c/openstack/swift/+/883788
21:23:32 <mattoliver> But to test it, I went and created a FakeStatsdClient and added it to the debug_logger. So there is quite a bit of test churn
21:23:50 <mattoliver> Thanks zaitcev for reviewing!
21:23:58 <zaitcev> It was nothing.
21:24:45 <mattoliver> so a bunch of churn but I think it makes testing better, we know actually run the statsd methods and just don't ever open the socket and send the data.
21:26:31 <mattoliver> Only other thing I had (or can think of, of to the top of my head this morning) was I still have the py2 no-op tracing patch at the end of my tracing chain.
21:27:07 <mattoliver> Real question is, when (as an upstream project) are we going to rip out py2.
21:27:51 <mattoliver> ie, maybe it's just too crazy so we can just wait, or is mocking out otel classes something we want to carry until we do?
21:28:20 <mattoliver> Not expecting an answer t o that right now, just something to think about at least :)
21:28:48 <timburke> the moment it's not super painful for me (downstream) to do it 😁
21:28:54 <zaitcev> sorry ttyl
21:29:23 <timburke> speaking of, though -- i'm happy to report that we've moved all of our swift services to py3 in our largest swift cluster!
21:29:44 <kota> excellent
21:29:47 <mattoliver> \o/
21:30:02 <mattoliver> also why it made me think of it. The reason for the latest patch that is.
21:30:08 <mattoliver> maybe I don't need it anymore ;)
21:30:13 <timburke> now we just need to worry about the management plane...
21:31:23 <timburke> i should rebase https://review.opendev.org/c/openstack/swift/+/853590
21:32:43 <mattoliver> yeah, well if we can just stop worrying about py2 and even if that means slowly moving out the py2 isms over time. I can just ignore py2 failing to run tracing and let it fail to find otel
21:34:37 <mattoliver> anyway food for thought
21:35:32 <timburke> oh, whoa -- https://review.opendev.org/c/openstack/swift/+/883050 took a different approach than i was thinking to deal with the problem, mattoliver -- i would've just done the lazy imports, then error hard at runtime if you tried to configure tracing when the package isn't available
21:36:28 <mattoliver> there is just too much tracing-isms and context managers in the code.
21:36:39 <mattoliver> but yeah, maybe I over engineered :P
21:36:40 <timburke> ah, fair enough
21:38:39 <timburke> all right, i think i'll call it
21:38:49 <timburke> thank you all for coming, and thank you for working on swift!
21:38:52 <timburke> #endmeeting