21:01:30 <timburke> #startmeeting swift
21:01:30 <opendevmeet> Meeting started Wed Aug 25 21:01:30 2021 UTC and is due to finish in 60 minutes.  The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:01:30 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:01:30 <opendevmeet> The meeting name has been set to 'swift'
21:01:38 <timburke> who's here for the swift meeting?
21:01:52 <mattoliver> o/
21:01:57 <seongsoocho> o/
21:02:19 <kota> o/
21:03:26 <timburke> first up
21:03:30 <timburke> #topic elections
21:03:56 <timburke> in case you missed it, the PTL and TC candidacy period ended
21:04:03 <timburke> #link http://lists.openstack.org/pipermail/openstack-discuss/2021-August/024401.html
21:04:54 <timburke> for the TC, there were four candidates for four slots. for the projects, only six had no candidate, which seems like an improvement over recent cycles
21:05:34 <timburke> just an FYI -- shouldn't really impact us
21:06:05 <timburke> #topic next meeting to be September 8
21:06:48 <timburke> i'm going to be out of town next week, and i'm guessing nobody wants to chair the meeting ;-)
21:07:21 <mattoliver> happy to if there is anything important, but think its probably ok to skip.. does mean a sleep in ;)
21:08:01 <timburke> #topic ring v2 format
21:08:44 <timburke> so we've got a few different initiatives that want to get more or different data into rings
21:09:02 <timburke> #link https://review.opendev.org/c/openstack/swift/+/761794
21:09:24 <timburke> ^^^ wants to allow wider device IDs so we can have more than 64k devices in a single ring
21:09:35 <timburke> #link https://review.opendev.org/c/openstack/swift/+/790550
21:09:46 <mattoliver> nice problem to have :) (re: more devices)
21:10:19 <timburke> ^^^ wants to add an extra replica, essentially, to track the set of previous primaries during a rebalance
21:11:21 <timburke> ...but that has some complications with fractional replicas, which led to https://review.opendev.org/c/openstack/swift/+/803665 getting split out from it
21:13:20 <timburke> between the two, we've started to feel out what we'd want from a new ring format, and there are a few different ideas for how we could serialize the data in a way that would lend itself toward further extensibility
21:13:43 <mattoliver> i probably need to update that commit message.
21:13:43 <timburke> a bunch of which mattoliver helpfully gathered into an etherpad:
21:13:47 <timburke> #link https://etherpad.opendev.org/p/swift-ring-v2-serialization
21:15:59 <timburke> so, if anyone else wants to think about how best to serialize (or more concretely, about what *else* we may want to serialize), join the discussion! i'd start with either that etherpad or the last patch i linked
21:17:00 <mattoliver> note: current patchset of 803665 is implementing option 3 for part2dev.. not that it's the decision, but wanted to see how it shook out.
21:19:15 <timburke> next up
21:19:28 <timburke> #topic misplaced objects in sharded containers
21:20:35 <timburke> i haven't been following this work that closely, but i figured i ought to see how it's going after seeing "DNM: shard policy migration: it's still not great" go by ;-)
21:21:11 <mattoliver> yeah, to be honest I need to loop back to this and was my plan for today.
21:21:46 <mattoliver> acoles: found an edge case where the patch up for review had an issue
21:22:44 <mattoliver> and it has to do with the root -> shard migration code not quite being the same as how the  reconcoler makes decision.
21:23:05 <timburke> 👍 no worries, we can check back in later. interesting... do you feel like you guys are getting blocked?
21:23:20 <mattoliver> So I think its more about getting those more aligned. But I'll now more once I read the DNM patch.
21:23:36 <mattoliver> *know
21:23:58 <timburke> cool
21:24:06 <timburke> #topic open discussion
21:24:16 <timburke> anything else we ought to bring up this week?
21:25:18 <mattoliver> I've been playing with swift tagging and tracing requests, and it's starting to look pretty nice. It tracks and times every middleware and even follows down into the storage nodes.
21:25:57 <mattoliver> so someone can add a trace_sig to the query string of a request and it gets traced.
21:26:16 <mattoliver> And is using OpenTracing so a nice open standard.
21:26:51 <mattoliver> I think I already have it as a topic at the PTG, so can show it off with pics there :)
21:26:58 <seongsoocho> wow.. Is it different from transaction id?
21:28:30 <mattoliver> yeah, its to get better detail into whats happening by being able to literally trace a request though the cluster and get more information then what you get with just logs.
21:28:52 <mattoliver> so extends and works with txid :)
21:29:26 <seongsoocho> cool !
21:32:16 <mattoliver> here is an example: https://drive.google.com/file/d/15eDwn3RF7DJhlNtgKB-dKdiQ-9oNF8TK/view?usp=sharing
21:32:48 <mattoliver> its just a screen shot of a trace from jaeger trace. It's tracing an EC PUT through my VSAIO.
21:34:35 <mattoliver> There are also  logs (kv pairs) attached to different spans (the lines) that I'm pulling out. Like other timings and things like error limited nodes from the proxy.. but you cant see from that image, thats just giving you the overview.
21:35:37 <timburke> very cool. the long delay between getting into the proxy app and getting to the object servers is definitely worth investigating
21:35:52 <mattoliver> +1
21:36:24 <timburke> i wonder how much of that it is backend connection setup
21:36:59 <mattoliver> just needs more instramenting to see what's happening. Starting to visualize might find targets for optimisations or even regressions
21:37:49 <timburke> all right, i'm going to call it
21:38:00 <timburke> thank you all for coming, and thank you for working on swift!
21:38:48 <mattoliver> thanks timburke enjoy your trip!
21:38:55 <timburke> #endmeeting