21:01:33 <timburke> #startmeeting swift
21:01:33 <opendevmeet> Meeting started Wed Nov 16 21:01:33 2022 UTC and is due to finish in 60 minutes.  The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:01:33 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:01:33 <opendevmeet> The meeting name has been set to 'swift'
21:01:39 <timburke> who's here for the swift meeting?
21:01:44 <acoles> \o
21:01:46 <seongsoocho> o/
21:01:50 <mattoliver> o/
21:02:35 <timburke> sorry, haven't updated the agenda, so i'll be kinda making this up as i go ;-)
21:02:43 <mattoliver> lol
21:02:48 <kota> o/
21:02:51 <timburke> #topic failing func tests
21:03:09 <timburke> as i mentioned last week, a recent cpython change broke some of our func tests
21:03:36 <timburke> (specifically, ones that wanted to include a leading '//' in request paths)
21:05:08 <timburke> the rest of openstack is looking to move base CI images from focal to jammy -- as things stand, that will mean that at least our dsvm job will fail
21:05:42 <timburke> i've got a fix, but it's a bit of an ugly hack
21:05:44 <timburke> #link https://review.opendev.org/c/openstack/swift/+/863441
21:06:16 <timburke> other options include either marking the job non-voting
21:06:31 <timburke> or pinning it specifically to continue using focal
21:07:59 <timburke> from talking with gmann, sounds like the transition to jammy will be friday
21:08:08 <mattoliver> oh that's interesting. but annoying that we may need to support that behaviour. And I assume it does it before it hits our domain_remap middleware (ie up in wsgi).
21:08:55 <timburke> yes -- before anything's even started building a WSGI environ, in fact. it's pretty low level
21:09:06 <timburke> #link https://github.com/python/cpython/issues/99220
21:09:12 <timburke> for a little more context
21:11:00 <timburke> so, can anyone help review the fix, make sure i'm not doing anything too egregious? and if we can't get it merged by then, are there strong opinions about how to ensure we don't start next week with a broken gate? my gut says to pin to focal for now
21:11:40 <mattoliver> yup, I'll have a play with it
21:12:28 <timburke> thanks mattoliver
21:13:23 <timburke> next up
21:13:43 <timburke> #topic large objects and swiftclient
21:14:59 <opendevreview> ASHWIN A NAIR proposed openstack/swift master: s3api errors for unsupported headers x-delete-at, x-delete-after  https://review.opendev.org/c/openstack/swift/+/864788
21:15:28 <timburke> i recently encountered some users uploading dlos then getting bit by listings showing them as 0-byte objects. they didn't really care about the dlo/slo distinction, they just wanted some >5GB objects written
21:16:37 <timburke> and it occurred to me: we've had SLO for nearly ten years now. and while it isn't strictly required when deploying a swift cluster, my understanding is that it's generally available
21:16:54 <timburke> so, should swiftclient switch to writing SLOs by default?
21:17:00 <timburke> #link https://review.opendev.org/c/openstack/python-swiftclient/+/864444
21:17:23 <mattoliver> +1, I think in pretty much every case an SLO is better then a DLO.
21:18:21 <timburke> there's still an escape hatch -- it adds a new `--use-dlo` flag. and the existing `--use-slo` option is still parsed, so we won't break existing workflows
21:19:14 <seongsoocho> +1 to that idea :)
21:19:35 <acoles> remind me, does swift info advertise if slo is supported?
21:20:04 <timburke> i considered doing a /info call if neither flag was specified, then using slo if was available -- but that seemed overly complicated and would require an extra network round-trip
21:20:11 <acoles> just wondering if the client could check before defulting to dlo
21:20:28 <timburke> it could, yes
21:20:43 <acoles> timburke: yeah, that :)
21:21:56 <timburke> if i was focused on nothing but the client, i'd really push for getting some persistent cache into the client -- specifically for /info responses and tokens/storage-urls
21:22:20 <acoles> what happens if client tries slo with a cluster that does not support it?
21:22:53 <timburke> excellent question -- haven't tried it. sounds like a bug i remember seeing, though...
21:23:19 <timburke> related question: should we start making slo a required middleware server-side?
21:23:34 <acoles> just wondering if there is a fallback route to retry with dlow
21:24:20 <acoles> err, if client defaults to slo then it certainly seems reasonable that clusters would provide it by default
21:26:48 <timburke> here's the bug i was thinking about! looks like clayg did some investigation, and putting an SLO to a cluster that doesn't have it enabled just writes the manifest as an object :-(
21:26:48 <timburke> #link https://bugs.launchpad.net/swift/+bug/1680083
21:27:15 <timburke> so i suppose maybe i ought to go ahead and do that /info call...
21:28:07 <timburke> good news is that modern swiftclients don't send a X-Static-Large-Object header anymore https://review.opendev.org/c/openstack/python-swiftclient/+/455470/
21:29:08 <timburke> also makes me think that gatekeeper should probably strip out the header...
21:30:18 <timburke> ok, i think i got the feedback i needed -- seems like we're generally agreed that the default should change, but it's worth the /info call to ensure that slo is actually available. i'll try to respin soon
21:30:35 <mattoliver> kk
21:30:39 <timburke> #topic ring v2
21:30:59 <timburke> thanks for looking at the initial patch again, mattoliver!
21:31:15 <mattoliver> nps
21:31:41 <timburke> and clayg's been offering some feedback recently internally
21:32:00 <seongsoocho> 👏
21:32:06 <mattoliver> if people haven't seen I've put together a doc patch for it
21:32:10 <mattoliver> #link https://review.opendev.org/c/openstack/swift/+/864494
21:32:23 <acoles> timburke: one last thought - the pre-check info could be disabled by an option??
21:32:48 <mattoliver> would --use-slo or --use-dlo not use the pre-check option
21:32:55 <mattoliver> ie, if you say so
21:33:14 <timburke> acoles, yes, that was my thinking -- if you explicitly say --use-slo or --use-dlo, skip the /info and assume the user knows what they're doing
21:33:26 <acoles> yup :thu
21:33:52 <timburke> based clayg's feedback, i might try putting out yet another take on how to structure v2 rings -- this time based on zipfile
21:34:07 <acoles> sorry I am struggling with an unfamilar keyboard ... 👍
21:34:25 <acoles> mattoliver: docs! super!
21:34:54 <timburke> and making the "sections" in the current patch into separate files within the zip
21:36:35 <mattoliver> I guess the question is, can we seek and pull out files nicely or would we have the load the whole thing up
21:36:55 <mattoliver> thinking when a ring includes the builder data too.
21:37:14 <timburke> i think my main concerns would be around on-disk file size and speed of deserialization, but they'll be hard to judge until there's an implementation
21:38:11 <timburke> yeah, zip includes a "central directory" at the end, which has the whole file structure and pointers to offsets for each file
21:38:12 <mattoliver> well that's the other thing. We've had a few PTGs getting the current design together and a patch that works.
21:38:25 <mattoliver> oh cool
21:39:38 <timburke> it'll involve some other changes to look for *either* a ring.gz or a ring.zip -- which probably *also* means more changes for our internal ring-management tooling...
21:40:16 <mattoliver> yup, it would give zeitcev what we are asking
21:40:40 <mattoliver> ie, different names to you can see at an lls level
21:40:45 <mattoliver> *ls
21:41:11 <timburke> idk -- i thought zaitcev wanted a separate name specifically when it's also got builder info in it
21:41:35 <zaitcev> (hold on, I wasn't looking for a moment, sorry)
21:41:43 <timburke> no worries :-)
21:42:45 <zaitcev> Okay, yes. I did want a separate name for the unified builder-ring thing.
21:43:09 <zaitcev> I'm looking at the double-slash thing...
21:43:17 <zaitcev> meanwhile, I mean
21:43:18 <timburke> thanks!
21:43:58 <timburke> i think this would also be fairly amenable to that -- having something like object.builder.zip
21:45:56 <timburke> could maybe even have the search order for services be something like object.ring.zip else object.ring.gz else object.builder.zip
21:46:03 <timburke> then write_ring could still be useful to have a snapshot while you're manipulating the builder
21:46:14 <timburke> /shrug
21:47:50 <timburke> anyway, i think that's all i've got
21:47:55 <timburke> #topic open discussion
21:48:03 <timburke> anything else we should bring up this week?
21:48:36 <mattoliver> I've managed to get an in memory tracer working for unit tests.
21:48:44 <timburke> \o/
21:48:53 <mattoliver> So started some basic tracing unit tests
21:49:49 <mattoliver> not sure about cross service (ie in probe tests) but that's the next bit of testing
21:53:30 <mattoliver> thats all I had for now
21:53:46 <timburke> all right
21:53:57 <timburke> thank you all for coming, and thank you for working on swift!
21:54:00 <timburke> #endmeeting