21:00:01 <timburke> #startmeeting swift
21:00:01 <opendevmeet> Meeting started Wed Jun 26 21:00:01 2024 UTC and is due to finish in 60 minutes.  The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:01 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:01 <opendevmeet> The meeting name has been set to 'swift'
21:00:08 <timburke> who's here for the swift meeting?
21:00:25 <fulecorafa> I am o/
21:00:50 <kota> hi
21:01:43 <acoles> o/
21:02:17 <timburke> sorry, i feel like it's been a while since i held the regular meeting
21:02:31 <timburke> but i did actually update the agenda this time!
21:02:37 <timburke> #link https://wiki.openstack.org/wiki/Meetings/Swift
21:02:50 <jianjian> o/
21:03:04 <timburke> first up, a couple recently-reported bugs
21:03:15 <timburke> #topic account-reaper and sharded containers
21:03:28 <timburke> it looks like they don't work!
21:03:36 <timburke> #link https://bugs.launchpad.net/swift/+bug/2070397
21:03:37 <patch-bot> Bug #2070397 - Account reaper do not reap sharded containers (New)
21:04:39 <timburke> since the reaper is relying on direct_client instead of internal_client, it gets no objects to delete, tries to delete the container, and gets back a 409 -- next cycle, same deal
21:05:58 <timburke> this was also brought up on the mailing list, as zaitcev helpfully pointed out to me yesterday
21:06:00 <timburke> #link https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/thread/KOJS5G5W24PARPLNQS2D5A6M5F3JRM7V/
21:06:49 <zaitcev> That's because I went through exactly this thing with the dark data watcher.
21:07:06 <zaitcev> So a simple thing is to copy the code from the watcher and be done.
21:07:29 <zaitcev> Maybe some useful refactoring could be done... Or switch both of them to another API... I dunno.
21:08:15 <zaitcev> It took so long to engage Alistair's attention on this, that I would prefer just clone the code and forget about it.
21:08:30 <timburke> yeah, i was wondering if it might make sense to switch to using internal_client... i'm trying to remember why we didn't go that route for the watcher
21:09:12 <timburke> i think there'd be some complication around avoiding the 410, but surely we could get around that by adding some x-backend-... header flag
21:09:15 <zaitcev> Because watcher runs in an object auditor, and internal_client is a part of the proxy. It needs a pipeline set up, for one. IIRC.
21:10:07 <timburke> hmm. true enough -- and having a pipeline leaves more room for misconfiguration
21:10:53 <timburke> zaitcev, had you taken a look at the linked patch yet? i haven't, i'll admit
21:11:21 <acoles> so is the root cause that the reaper uses direct client to list objects?
21:11:21 <zaitcev> Not really, sorry. It looked faithfully cargo-culted from the watcher.
21:12:08 <timburke> acoles, yeah, afaik
21:12:53 <acoles> I'm wondering, even if all the objects are deleted from all the shards, will the container delete succeed, until all the shards have been shrunk away??
21:13:36 <zaitcev> uh-oh
21:13:39 <acoles> isn't there this weird property that a container with shards isn't "empty"? although IIRC we haven't several definitions of "empty"
21:14:00 <timburke> hrm... i thought we had some special handling when all shards report zero objects, but i don't remember for certain...
21:14:24 <timburke> a good first step might be for one of us to write a failing probe test, then try the linked patch, and iterate from there
21:14:52 <acoles> I just remember that we have recurring warnings in prod for containers that cannot be reclaimed because they have shards...but maybe there is a workaround for reaper 🤔
21:15:09 <acoles> yeah a probe test is definitely a good idea
21:15:34 <timburke> that does sound familiar...
21:15:38 <timburke> can anyone volunteer to get a probe test together?
21:16:47 <zaitcev> The v-word scares me. I'm up to my chest in Cinder nowadays.
21:17:10 <timburke> hehe -- all right, i'll put it on my to-do list and hopefully i can get to it this week
21:17:18 <acoles> zaitcev: that still leaves your brain available ;-)
21:17:28 <timburke> next up
21:17:35 <timburke> #topic busted docker image
21:17:41 <timburke> #link https://bugs.launchpad.net/swift/+bug/2070029
21:17:42 <patch-bot> Bug #2070029 - swift-docker: saio docker alpine uncorrect (New)
21:18:20 <timburke> sounds like our published docker image isn't actually working correctly
21:19:27 <timburke> i guess my first question, though, is how much we really want to commit to maintaining it? it seems like a nice-to-have, but we surely ought to have more validation of the image if we're properly maintaining it
21:20:54 <acoles> does any core  dev use it? AFAIK vSAIO is the recommended dev environment
21:21:49 <zaitcev> I don't but it's because we ship our own images in RHOSP. I don't know how to build them, honestly.
21:21:49 <jianjian> I just got to know there is a "saio docker" env
21:21:57 <fulecorafa> I personally use it when developping here. I noticed the docker image was failing, but I though it was some mistake in my config here
21:22:07 <timburke> i've got a little single-disk cluster i use it for, but i don't even remember the last time i updated it
21:23:32 <fulecorafa> I did try to rebuild it using a lxc environment by hand and it still didn't work
21:24:14 <timburke> thanks for the data point fulecorafa -- fwiw, it seems like swift must not have gotten installed correctly, judging by the bug report
21:25:13 <timburke> i'll try to dig into it some more -- looks like i was the last one to touch it, in p 853362
21:25:13 <patch-bot> https://review.opendev.org/c/openstack/swift/+/853362 - swift - Fix docker image building (MERGED) - 1 patch set
21:25:15 <acoles> fulecorafa: you may want to try https://github.com/NVIDIA/vagrant-swift-all-in-one which many of use daily for dev
21:25:48 <timburke> next up
21:25:53 <acoles> FWIW I did have a go at adding a docker target for vSAIO, almost got it working
21:26:07 <timburke> #topic log_ prefix for statsd client configs
21:26:15 <timburke> #link https://review.opendev.org/c/openstack/swift/+/922518
21:26:15 <patch-bot> patch 922518 - swift - statsd: deprecate log_ prefix for options - 3 patch sets
21:26:59 <timburke> this was split out of https://review.opendev.org/c/openstack/swift/+/919444 and the general desire to separate logging and stats
21:26:59 <patch-bot> patch 919444 - swift - Add get_statsd_client function (MERGED) - 13 patch sets
21:28:23 <timburke> but when i got into proxy-logging, i was reminded of how we already have (at least?) two ways of spelling these configs, and i wanted to make sure that we had some consensus to make a recommendation for how the configs *should* be specified going forward
21:29:59 <timburke> we already have things like log_statsd_host and access_log_statsd_host, and while i like the idea of moving toward just statsd_host or access_statsd_host, i don't really want to add support for *both*
21:32:55 <timburke> i want to say that the access_ prefix got added because of confusion over how to do config overrides with PasteDeploy
21:34:34 <timburke> does anybody have input on what should be our preferred option name? or maybe this is all a giant yak-shave that doesn't really have any hope of getting off the ground until we can get rid of PasteDeploy and make config overrides sensible?
21:35:23 <acoles> can we think of the differentiating prefix being 'access_log_' rather than just 'access_' and then support 'access_log_statsd_host', 'statsd_host', 'log_statsd_host' in that order of preference?
21:35:52 <zaitcev> XCKD_14_standards.jpg
21:35:53 <acoles> meaning, if you want proxy logging to have different statsd host you should configure access_log_statsd_host
21:38:47 <timburke> that could work... it feels a little funny to me, though, like it's "most-specific > most-generic > middle" when i'd expect middle to be in the middle... *shrug*
21:40:07 <timburke> well, we don't have to sort it out right now. please leave thoughts/comments/ideas on the review!
21:40:34 <timburke> speaking of statsd/logging separation...
21:40:42 <timburke> #topic LoggerStatsdFacade
21:40:47 <timburke> #link https://review.opendev.org/c/openstack/swift/+/915483
21:40:47 <patch-bot> patch 915483 - swift - class LoggerStatsdFacade - 25 patch sets
21:41:52 <acoles> I though of it as "most-specific > generic > deprecated-generic" but, yeah, it's not ideal
21:42:11 <timburke> nothing too much to report, just wanted to highlight that work continues and there might be some light at the end of that particular tunnel
21:42:49 <timburke> i think it's getting close
21:44:23 <timburke> the last two topics, you can catch up on from the agenda -- i don't think there's anything too much to do for them, but they've been hogging a decent bit of my time the last week or two
21:45:00 <timburke> the entry-points patch chain starting at https://review.opendev.org/c/openstack/swift/+/918365 could use some review if anyone has bandwidth
21:45:00 <patch-bot> patch 918365 - swift - Use entry_points for server executables - 4 patch sets
21:45:28 <timburke> but especially since we've got a new face here, i wanted to leave time for
21:45:33 <timburke> #topic open discussion
21:45:42 <timburke> anything else we should bring up this week?
21:45:53 <fulecorafa> I would like to bring a topic out, if I may
21:46:01 <timburke> by all means!
21:47:38 <fulecorafa> I've been studying how to implement multi-storage-policy containers in swift. A demand for work. I was hopping you guys could share your ideas and/or views on the solution for this
21:48:23 <fulecorafa> multi-storage-policy as in I can have 2 objects in the same bucket but one would be in normal policy and another in a "glacier" one
21:49:43 <acoles> symlinks would enable this
21:49:47 <timburke> fulecorafa, there's definitely been various interest in it over the years -- how are you thinking about objects moving between policies? would you want it to happen automatically after some period of time, or based on user action?
21:50:43 <acoles> ^^^ I was about to continue that the interesting/challenging part is 'automating' policy placement and migration between policies
21:51:02 <fulecorafa> From what I've gathered, I think the solution for this would be to use the same basic logic as SLOs: to have a normal bucket and another marked bucket (i.e. bucket+glacier). When uploading a file, given the policy, it would store the actual object in cold
21:51:18 <fulecorafa> And keep a symlink/manifest of sorts in the original bucket
21:52:19 <timburke> makes sense enough -- similar to x-amz-storage-class for s3
21:52:37 <acoles> one thing we have learnt from SLOs and S3 mpus is that it's not a great idea to have 'extra' 'shadow' +segments buckets visible in the users namespace.
21:53:13 <acoles> but we have the capability to maintain 'hidden' buckets (which is the direction native-MPU is going on feature/mpu)
21:53:32 <timburke> yeah, we'd almost certainly want to use the null-namespace stuff introduced with the most-recent iteration on versioning
21:53:37 <fulecorafa> As for objects moving between policies, I know AWS's API has something more on moving objects, but I haven't quite gotten into it. In my specific use-case, we could just use a client to re-put the object? But I'll have to think more on that
21:54:17 <fulecorafa> acoles, I remenber you commenting this on the orphan patch I've submitted last month.
21:54:30 <acoles> right
21:54:33 <timburke> fulecorafa, yeah -- i'd expect to be able to do a server-side copy with a new destination policy
21:54:53 <fulecorafa> I recon that if we evolve on feature/mpu, we could keep this other marked buckets hidden too right?
21:55:21 <timburke> yup, i think that's acoles's plan too
21:55:26 <acoles> I think there might be a good deal of commonality with native MPU
21:55:57 <opendevreview> Yan Xiao proposed openstack/swift master: stats: API for native labeled metrics  https://review.opendev.org/c/openstack/swift/+/909882
21:56:30 <fulecorafa> Should be good to go foward with this plan!
21:56:51 <acoles> possibly even a degenerate use case "Single Part Upload" :) that uses the manifest/symlink/hidden container logic
21:57:44 <timburke> i'd only note that things may start to get pretty confusing when you've got, say, a versioned MPU uploaded to one of these mixed-policy containers...
21:57:53 <fulecorafa> Just one last question: For my specific use-case, we're just working with the S3 api in an older version. Even if we do this, the bucket should not be visible to s3 clients, correct?
21:57:56 <jianjian> fulecorafa, will it be enough that application move objects between a normal container and a cold policy container?
21:58:25 <acoles> there's probably similar cleanup concerns - what happens to the target object in "cold" when the user object is overwritten? it needs to be cleaned up like orphan segments do
21:59:27 <acoles> timburke: yeah, whilst I see commonality I'm also wary of overloading the goals of native-MPUs. One step at a time..
21:59:40 <fulecorafa> jianjian, I don't think so, but it would be a first step. Taking AWS S3 as a counterpart, there should be other different behaviours we would need to take into account. As of now, I think to have a simple 2 policy solution would be a step in the right direction
21:59:45 <acoles> MPUs + versioning is complex enough for my small brain!
22:00:29 <acoles> fulecorafa: just curious, if you are able to share the info, do you use object versioning?
22:01:14 <fulecorafa> acoles: As soon as I started coding I noted that :). I very much liked the way you did the Orphan collector on feature/mpu. I think it would make it very simple to then collect the cold parts
22:02:20 <acoles> we still have a way to go to on finishing the orphan auditor but we're excited about that approach
22:02:24 <fulecorafa> acoles: yeah, so we do allow for versioning but just for simple, normal policy files. In the future it may be needed to get support for versioning in other types, but that's not allowed yet
22:02:53 <acoles> and do your clients use s3 API mostly?
22:03:20 <fulecorafa> mostly, yes. We're trying to make it easy to migrate
22:04:41 <fulecorafa> timburke: Yeah, I think it can become complicated quite fast. but I'm not sure there would be another way. Maybe tinkering with the underlying hashing system? (I mean, the one that actually sets where the file can be found on disk
22:07:32 <timburke> do you expect to need separate disk pools for each policy, or would there likely be a high level of overlap? another similar-but-different idea we've had was to have something like an EC policy, but sufficiently small uploads would be stored replicated on the first three assignments
22:08:00 <acoles> sorry I need to drop. fulecorafa: thanks for coming along and sharing your ideas, hope we can make progress together. Ask for help here and we'll do our best :)
22:08:31 <timburke> so it'd all use the same policy index and the same disks, but have different storage overheads
22:08:36 <fulecorafa> acoles: thank you very much for your help receptivity, alongside the comunity's! have a good one
22:09:03 <timburke> oh yeah, i suppose we're a decent bit over time :-)
22:09:17 <fulecorafa> timburke: I think one of the main uses would be to have different disk pools
22:09:22 <timburke> ack
22:10:14 <timburke> all right, i should wrap this up -- but i'll add mixed-policy containers to the agenda for next week, fulecorafa
22:10:27 <timburke> thank you all for coming, and thank you for working on swift!
22:10:34 <timburke> #endmeeting