| *** tetsuro has joined #openstack-meeting | 00:03 | |
| *** martial has quit IRC | 00:06 | |
| *** tetsuro has quit IRC | 00:07 | |
| *** tetsuro has joined #openstack-meeting | 00:10 | |
| *** rbudden has quit IRC | 00:23 | |
| *** yasufum has joined #openstack-meeting | 00:33 | |
| *** tetsuro has quit IRC | 00:40 | |
| *** gyee has quit IRC | 01:08 | |
| *** rfolco|ruck has quit IRC | 01:14 | |
| *** yamamoto has quit IRC | 01:18 | |
| *** cgoncalves has quit IRC | 01:23 | |
| *** cgoncalves has joined #openstack-meeting | 01:23 | |
| *** Roamer` has quit IRC | 01:24 | |
| *** yamamoto has joined #openstack-meeting | 01:28 | |
| *** rbudden has joined #openstack-meeting | 01:34 | |
| *** manpreet has quit IRC | 01:56 | |
| *** Liang__ has joined #openstack-meeting | 02:11 | |
| *** whoami-rajat__ has joined #openstack-meeting | 02:13 | |
| *** arne_wiebalck has quit IRC | 02:14 | |
| *** clayg has quit IRC | 02:14 | |
| *** hillpd has quit IRC | 02:14 | |
| *** vkmc has quit IRC | 02:14 | |
| *** patrickeast has quit IRC | 02:14 | |
| *** phrobb has quit IRC | 02:14 | |
| *** Liang has quit IRC | 02:14 | |
| *** ttx has quit IRC | 02:14 | |
| *** mattoliverau has quit IRC | 02:14 | |
| *** freefood has quit IRC | 02:14 | |
| *** hillpd has joined #openstack-meeting | 02:15 | |
| *** vkmc has joined #openstack-meeting | 02:15 | |
| *** phrobb has joined #openstack-meeting | 02:15 | |
| *** clayg has joined #openstack-meeting | 02:15 | |
| *** arne_wiebalck has joined #openstack-meeting | 02:15 | |
| *** patrickeast has joined #openstack-meeting | 02:20 | |
| *** mattoliverau has joined #openstack-meeting | 02:20 | |
| *** freefood has joined #openstack-meeting | 02:20 | |
| *** b1airo has quit IRC | 02:21 | |
| *** manpreet has joined #openstack-meeting | 02:27 | |
| *** viks____ has joined #openstack-meeting | 02:28 | |
| *** armax has quit IRC | 02:33 | |
| *** armax has joined #openstack-meeting | 02:34 | |
| *** armax has quit IRC | 02:34 | |
| *** rcernin has quit IRC | 02:55 | |
| *** rcernin_ has joined #openstack-meeting | 02:57 | |
| *** yonglihe has quit IRC | 03:10 | |
| *** psachin has joined #openstack-meeting | 03:31 | |
| *** psachin has quit IRC | 03:32 | |
| *** psachin has joined #openstack-meeting | 03:33 | |
| *** Liang__ has quit IRC | 04:01 | |
| *** rh-jelabarre has quit IRC | 04:10 | |
| *** Lucas_Gray has joined #openstack-meeting | 04:31 | |
| *** evrardjp has quit IRC | 04:33 | |
| *** evrardjp has joined #openstack-meeting | 04:33 | |
| *** rbudden has quit IRC | 04:57 | |
| *** hongbin has quit IRC | 05:07 | |
| *** moguimar has joined #openstack-meeting | 05:15 | |
| *** yamamoto has quit IRC | 05:22 | |
| *** moguimar has quit IRC | 05:22 | |
| *** moguimar has joined #openstack-meeting | 05:22 | |
| *** yamamoto has joined #openstack-meeting | 05:26 | |
| *** Lucas_Gray has quit IRC | 05:32 | |
| *** Lucas_Gray has joined #openstack-meeting | 05:38 | |
| *** lpetrut has joined #openstack-meeting | 06:24 | |
| *** dklyle has quit IRC | 06:38 | |
| *** ralonsoh has joined #openstack-meeting | 06:38 | |
| *** bbowen has quit IRC | 07:04 | |
| *** bbowen has joined #openstack-meeting | 07:05 | |
| *** rcernin_ has quit IRC | 07:06 | |
| *** slaweq has joined #openstack-meeting | 07:09 | |
| *** jiaopengju1 has joined #openstack-meeting | 07:13 | |
| *** jiaopengju2 has quit IRC | 07:15 | |
| *** jiaopengju1 has quit IRC | 07:15 | |
| *** jiaopengju1 has joined #openstack-meeting | 07:15 | |
| *** ttx has joined #openstack-meeting | 07:17 | |
| *** rcernin_ has joined #openstack-meeting | 07:18 | |
| *** rcernin_ has quit IRC | 07:20 | |
| *** rcernin has joined #openstack-meeting | 07:20 | |
| *** tosky has joined #openstack-meeting | 07:30 | |
| *** Lucas_Gray has quit IRC | 07:32 | |
| *** ociuhandu has joined #openstack-meeting | 07:50 | |
| *** ociuhandu has quit IRC | 07:54 | |
| *** jiaopengju1 has quit IRC | 07:55 | |
| *** jiaopengju1 has joined #openstack-meeting | 07:56 | |
| *** ociuhandu has joined #openstack-meeting | 08:09 | |
| *** whoami-rajat__ has quit IRC | 08:22 | |
| *** rcernin has quit IRC | 08:48 | |
| *** yamamoto has quit IRC | 08:52 | |
| *** yamamoto has joined #openstack-meeting | 08:52 | |
| *** yamamoto has quit IRC | 08:53 | |
| *** moguimar has quit IRC | 09:13 | |
| *** moguimar has joined #openstack-meeting | 09:24 | |
| *** yasufum has quit IRC | 09:32 | |
| *** yamamoto has joined #openstack-meeting | 09:33 | |
| *** moguimar has quit IRC | 09:33 | |
| *** moguimar has joined #openstack-meeting | 09:33 | |
| *** yasufum has joined #openstack-meeting | 09:41 | |
| *** yasufum has quit IRC | 09:52 | |
| *** Lucas_Gray has joined #openstack-meeting | 10:31 | |
| *** haleyb has quit IRC | 10:47 | |
| *** yamamoto has quit IRC | 10:47 | |
| *** yamamoto has joined #openstack-meeting | 10:52 | |
| *** slaweq_ has joined #openstack-meeting | 10:52 | |
| *** dansmith has quit IRC | 10:53 | |
| *** slaweq has quit IRC | 10:55 | |
| *** dansmith has joined #openstack-meeting | 10:55 | |
| *** yamamoto has quit IRC | 10:59 | |
| *** yamamoto has joined #openstack-meeting | 11:02 | |
| *** Lucas_Gray has quit IRC | 11:03 | |
| *** e0ne has joined #openstack-meeting | 11:09 | |
| *** yamamoto has quit IRC | 11:11 | |
| *** whoami-rajat__ has joined #openstack-meeting | 11:19 | |
| *** haleyb has joined #openstack-meeting | 11:34 | |
| *** yamamoto has joined #openstack-meeting | 11:44 | |
| *** yasufum has joined #openstack-meeting | 11:50 | |
| *** rh-jelabarre has joined #openstack-meeting | 11:51 | |
| *** rcernin has joined #openstack-meeting | 11:51 | |
| *** rfolco has joined #openstack-meeting | 11:53 | |
| *** rfolco is now known as rfolco|ruck|bbl | 11:53 | |
| *** yamamoto has quit IRC | 11:55 | |
| *** raildo has joined #openstack-meeting | 11:55 | |
| *** rcernin has quit IRC | 12:16 | |
| *** moguimar has quit IRC | 12:20 | |
| *** yamamoto has joined #openstack-meeting | 12:27 | |
| *** yamamoto has quit IRC | 12:32 | |
| *** lbragstad has quit IRC | 12:42 | |
| *** yasufum has quit IRC | 12:45 | |
| *** ricolin_ has joined #openstack-meeting | 12:48 | |
| *** yasufum has joined #openstack-meeting | 12:50 | |
| *** rbudden has joined #openstack-meeting | 12:59 | |
| *** moguimar has joined #openstack-meeting | 13:10 | |
| *** ricolin_ has quit IRC | 13:11 | |
| *** yasufum has quit IRC | 13:14 | |
| *** lbragstad has joined #openstack-meeting | 13:16 | |
| *** Lucas_Gray has joined #openstack-meeting | 13:20 | |
| *** soniya29 is now known as soniya29|ruck | 13:24 | |
| *** priteau has joined #openstack-meeting | 13:24 | |
| *** yasufum has joined #openstack-meeting | 13:26 | |
| *** Lucas_Gray has quit IRC | 13:30 | |
| *** ricolin_ has joined #openstack-meeting | 13:31 | |
| *** slaweq_ has quit IRC | 13:33 | |
| *** yasufum has quit IRC | 13:41 | |
| *** slaweq has joined #openstack-meeting | 13:46 | |
| *** yasufum has joined #openstack-meeting | 13:54 | |
| *** thgcorrea has joined #openstack-meeting | 14:00 | |
| *** haleyb has quit IRC | 14:01 | |
| *** e0ne_ has joined #openstack-meeting | 14:02 | |
| *** e0ne has quit IRC | 14:02 | |
| *** haleyb has joined #openstack-meeting | 14:05 | |
| *** ricolin_ has quit IRC | 14:14 | |
| *** rfolco|ruck|bbl is now known as rfolco | 14:15 | |
| *** whoami-rajat has joined #openstack-meeting | 14:18 | |
| *** TrevorV has joined #openstack-meeting | 14:23 | |
| *** armax has joined #openstack-meeting | 14:27 | |
| *** yamamoto has joined #openstack-meeting | 14:29 | |
| *** yamamoto has quit IRC | 14:34 | |
| *** mlavalle has joined #openstack-meeting | 14:48 | |
| *** yasufum has quit IRC | 14:52 | |
| *** gyee has joined #openstack-meeting | 14:59 | |
| *** lpetrut has quit IRC | 15:04 | |
| *** eharney has quit IRC | 15:06 | |
| *** dklyle has joined #openstack-meeting | 15:11 | |
| *** lpetrut has joined #openstack-meeting | 15:14 | |
| *** david-lyle has joined #openstack-meeting | 15:16 | |
| *** david-lyle has quit IRC | 15:17 | |
| *** lpetrut has quit IRC | 15:27 | |
| *** belmoreira has joined #openstack-meeting | 15:33 | |
| *** moguimar has quit IRC | 15:34 | |
| *** tosky has quit IRC | 16:19 | |
| *** yamamoto has joined #openstack-meeting | 16:30 | |
| *** psachin has quit IRC | 16:34 | |
| *** yamamoto has quit IRC | 16:34 | |
| *** ociuhandu has quit IRC | 16:57 | |
| *** yasufum has joined #openstack-meeting | 17:00 | |
| *** yasufum has quit IRC | 17:08 | |
| *** dmacpher has joined #openstack-meeting | 17:20 | |
| *** bbowen has quit IRC | 17:33 | |
| *** e0ne_ has quit IRC | 17:50 | |
| *** eharney has joined #openstack-meeting | 17:58 | |
| *** ralonsoh has quit IRC | 18:11 | |
| *** andrebeltrami has joined #openstack-meeting | 18:16 | |
| *** bbowen has joined #openstack-meeting | 18:47 | |
| *** e0ne has joined #openstack-meeting | 18:48 | |
| *** e0ne_ has joined #openstack-meeting | 18:52 | |
| *** e0ne has quit IRC | 18:53 | |
| *** mattia has quit IRC | 18:56 | |
| *** priteau has quit IRC | 18:56 | |
| *** e0ne_ has quit IRC | 18:58 | |
| *** ociuhandu has joined #openstack-meeting | 19:02 | |
| *** e0ne has joined #openstack-meeting | 19:04 | |
| *** belmoreira has quit IRC | 19:05 | |
| *** ociuhandu has quit IRC | 19:06 | |
| *** e0ne has quit IRC | 19:16 | |
| *** tosky has joined #openstack-meeting | 19:32 | |
| *** jokke has quit IRC | 19:33 | |
| *** whoami-rajat has quit IRC | 19:35 | |
| *** patchbot has joined #openstack-meeting | 20:54 | |
| *** raildo has quit IRC | 20:56 | |
| timburke | #startmeeting swift | 21:00 |
|---|---|---|
| openstack | Meeting started Wed Sep 30 21:00:18 2020 UTC and is due to finish in 60 minutes. The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot. | 21:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 21:00 |
| *** openstack changes topic to " (Meeting topic: swift)" | 21:00 | |
| openstack | The meeting name has been set to 'swift' | 21:00 |
| timburke | who's here for the swift meeting? | 21:00 |
| seongsoocho | o/ | 21:00 |
| mattoliverau | o/ | 21:00 |
| alecuyer | o/ | 21:01 |
| rledisez | o/ | 21:01 |
| kota_ | hi | 21:01 |
| timburke | not a whole lot on the agenda | 21:02 |
| timburke | #link https://wiki.openstack.org/wiki/Meetings/Swift | 21:02 |
| timburke | #topic ptg | 21:02 |
| *** openstack changes topic to "ptg (Meeting topic: swift)" | 21:02 | |
| timburke | we're less than a month away! | 21:02 |
| timburke | planning etherpad is at | 21:03 |
| timburke | #link https://etherpad.opendev.org/p/swift-ptg-wallaby | 21:03 |
| timburke | i'll plan on an ops-feedback-style session for one of our slots, and make another etherpad just for that | 21:03 |
| timburke | moving on | 21:04 |
| clayg | o/ | 21:04 |
| timburke | #topic replication lock | 21:05 |
| *** openstack changes topic to "replication lock (Meeting topic: swift)" | 21:05 | |
| timburke | #link https://review.opendev.org/#/c/754242/ | 21:05 |
| patchbot | patch 754242 - swift - Fix a race condition in case of cross-replication - 4 patch sets | 21:05 |
| timburke | clayg, i think you added this, right? | 21:05 |
| clayg | i forget why I wanted to talk about this - rledisez has been pushing up new patchsets daily and it just keeps getting better! | 21:05 |
| clayg | rledisez: do you want some help with a ssync_reciever test? I'm not sure I can a probe test 🤔 | 21:05 |
| rledisez | I still have one (major) point to discuss. this patchset only fix the issue for SSYNC. But rsync is(must be) impacted too | 21:06 |
| rledisez | and yeah, I don't know how to write a probetest for this | 21:07 |
| rledisez | I don't know how to fix rsync actually | 21:08 |
| clayg | meh rsync | 21:08 |
| timburke | part of me wants the excuse to change our "saio" to a "saif" or so... | 21:09 |
| clayg | swift-all-in-fhawhatnow? | 21:09 |
| timburke | four! so like, `vagrant up` and you get four vms, each independently configurable | 21:10 |
| clayg | rledisez: maybe this is just another reason to get of rsync and start working more on tsync - do you think we could "catch" rsync doing this? how did you find it with ssync? | 21:10 |
| timburke | or i could learn the dockers 🤔 | 21:10 |
| rledisez | clayg: we wrote a tool do scan and discover in near real time disappearing fragments, and we started a rebalance. And we saw it happening | 21:11 |
| clayg | I think we can say "closes-bug" with a caveat it still exists in rsync and has forever I think? not being in the datapath takes so many good options off the table. | 21:11 |
| rledisez | we have a cluster were it is (too) easily reproducible :( | 21:11 |
| clayg | 😢 | 21:12 |
| timburke | that's a damn cool tool | 21:12 |
| mattoliverau | could we add the current ring version in the replicate verb call, and if I receive one newer I look for a new ring? | 21:12 |
| *** _erlon_ has joined #openstack-meeting | 21:12 | |
| clayg | so it's kinda like a dispersion report sort of thing? like it checks on individual frags? | 21:13 |
| mattoliverau | that plus a lock, cause thre would still be overlap when things can happen | 21:13 |
| clayg | mattoliverau: so the tricky thing with update_deleted is it doesn't check with the remote node via REPLICATE (hash recalc) before pushing it's local data over rsync | 21:13 |
| rledisez | alecuyer: is the one writing it. ir scan disk and push in a single DB all files. then a tool scan the DB and list objects that don't have the required number of frags or replicas. | 21:13 |
| clayg | it just says "this isn't MY part; rsync away - 1, 2, 3 - everyone good? BELETED" | 21:14 |
| rledisez | mattoliverau: the object-server does not have the ring loaded so it can't know the ring version. | 21:14 |
| rledisez | + what clayg just said | 21:15 |
| mattoliverau | oh yes, the whole no ring in the storage daemon thing. | 21:15 |
| timburke | (not having the ring loaded isn't the end of the world; espescially if we only care about some of the json metadata at the front and not the full set of assignments -- it's something solvable) | 21:16 |
| clayg | I mean, there's some version of "load the ring" that could just pull the metadata like the ring version, I think it'd be useful to pursue but the lock is better | 21:16 |
| clayg | it's kinda weird that the replicator is so quick to pick up and notice a new part but so slow to realize it has new rings... | 21:17 |
| rledisez | to be clear, it happens because our ring deployment takes about 30 minutes. if the ring deployment was synchronised, we may have never hit that bug | 21:18 |
| clayg | everything is probably worse with EC than replicated - just in that deleting a copy doesn't trigger a rebuild | 21:18 |
| timburke | i think the trouble is that it may *not* have the new ring yet -- yeah, what rledisez said | 21:18 |
| clayg | so someone pushed it parts, and it doesn't believe they belong there... 🤔 lots more options with ssync than rsync | 21:19 |
| clayg | anyway, I think rsync can suck a lemon and fixing ssync is going to be great! | 21:20 |
| timburke | and much mor costly to be wrong with ssync than rsync. i agree; we ought to fix ssync and footnote rsync | 21:20 |
| clayg | I'm super open to saying we want to deprecate rsync replication too, like you can't complain about bugs in deprecated deployment options right? | 21:20 |
| clayg | timburke gets it | 21:20 |
| rledisez | tbh, i'm not confortable to keep rsync in the code if the bug is not fixed. it's pretty bad, it bite us pretty hard… so i can't imagine that kind of bug being unfixed if we know it's there | 21:22 |
| timburke | it's not that we *shouldn't* fix it, it's that we should land what fixes we have in hand, and shore up the rest when we can | 21:23 |
| clayg | well we can't pull it out with deprecating - but it could be a strong warning - also I might be biased cause I'm not sure how to fix it... so maybe we can just keep thinking on that harder after we fix ssync (no reason to wait on fixing that right?) | 21:24 |
| clayg | rledisez: it's not a new bug - there may be something about rsync with replicated data that makes it less of a problem for some reason we don't entirely understand yet 🤔 | 21:25 |
| rledisez | clayg: agree with that. we can fix ssync quickly and think of rsync at the same time | 21:25 |
| timburke | so, wanting to work toward that rsync fix: what about a pre-rsync REPLICATE that grabs the partition lock with some (configurable) lease time and a post-rsync REPLICATE to release it again? | 21:26 |
| clayg | timburke: 🤯 | 21:26 |
| rledisez | it's gonna be tricky to keep the lock without an active connection, but it's something we can try | 21:27 |
| clayg | i'd hate to add additional suffix hashing that not's necessary - so it'd probably look more like "fool, i'm reverthing this to you - don't delete it for X hours" kind of new pre-flight message | 21:27 |
| rledisez | i was thinking an SSYNC connection that just "wrap" the rsync run | 21:27 |
| rledisez | and keeping it active by sending "noop" action | 21:28 |
| clayg | rledisez: that sounds not terrible! | 21:28 |
| timburke | either of those could work, too :-) | 21:29 |
| clayg | look us acting like we know how to build software - thanks for finding the bug @rledisez !!! | 21:29 |
| mattoliverau | +1 | 21:29 |
| kota_ | +1 | 21:30 |
| rledisez | i'll finish the ssync patch and i'll do some test on rsync. any idea how to test that in an automated way? | 21:31 |
| timburke | i'll start trying to get a multi-vm dev setup going with an explicit goal of repro-ing this and let you know how it goes | 21:32 |
| mattoliverau | I guess we need to have a different rings on different "nodes" so maybe a ring override folder for siao | 21:32 |
| clayg | rledisez: i wouldn't block anything on probe test - there's a lot of new requirements (timing, ring changes) | 21:32 |
| rledisez | ok | 21:33 |
| clayg | rledisez: there's a unittest that already has an ssync_reciever object server and reconstructor daemon - I'd start there | 21:33 |
| timburke | anything else to bring up on this? | 21:34 |
| rledisez | not on my side | 21:34 |
| timburke | #topic async SLO segment deletion | 21:34 |
| *** openstack changes topic to "async SLO segment deletion (Meeting topic: swift)" | 21:35 | |
| timburke | #link https://review.opendev.org/#/c/733026/ | 21:35 |
| patchbot | patch 733026 - swift - Add a new URL parameter to allow for async cleanup... - 11 patch sets | 21:35 |
| timburke | so clay pointed out that we should bring this up with a broader audience before merging as-s | 21:36 |
| clayg | well, but also you already added the operator config value | 21:36 |
| clayg | what do folks think about having users dump a lot of "to be expired *rfn*" objects into your .expiring_objects quuee | 21:37 |
| clayg | does anyone monitor that shit? | 21:37 |
| timburke | still, seems worth asking whether i got the default for the config option right ;-) | 21:37 |
| rledisez | i'd like to monitore it, but I never took time to wrote that patch in the expirer that would send a metric of the queue size | 21:38 |
| mattoliverau | that's what the .expiring_objects queue is for. async delete. it's a good idea. | 21:38 |
| rledisez | we sometime get late on deletion because of a burst, so it's useful to know something is going wrong | 21:38 |
| *** slaweq has quit IRC | 21:39 | |
| mattoliverau | maybe we need to add some tooling around viewing and maintaining the queue would be useful though | 21:39 |
| clayg | rledisez: something like the "lag" metric for the container-updater? I never quite understood how that metric works in a time series visualization... | 21:39 |
| clayg | ... although I *do* have some better tools for development with metrics - maybe I just need to look at it again! | 21:39 |
| clayg | mattoliverau: all I have is https://gist.github.com/clayg/7f66eab2a61c77869e1e84ac4ed6f1df (run it on any node every N minutes and plot the numbers over time) | 21:40 |
| mattoliverau | I always though expired queue deletion was a soft shedule. it shouldn't be deleted until at least that time. | 21:41 |
| clayg | stale entires should be zero-ish, but any time someone does an SLO delete it would spike and depending on how aggressive your object-epxirers are configured... | 21:41 |
| timburke | speaking of lag metrics, there's https://review.opendev.org/#/c/735271/ ... | 21:41 |
| patchbot | patch 735271 - swift - metrics: Add lag metric to expirer - 1 patch set | 21:41 |
| clayg | timburke: yeah! i'll take a look at that one again, maybe that's just what we need (or a version of that which is slo-async-delete aware) | 21:42 |
| mattoliverau | clayg: cool that's something, tracking the size is great :) | 21:42 |
| *** slaweq has joined #openstack-meeting | 21:42 | |
| clayg | i'm not even sure I could easily get all those lag metrics to even pop up in a dev env | 21:44 |
| *** mlavalle has quit IRC | 21:44 | |
| timburke | clayg, good point -- it should probably emit different metrics for expired vs. async deleted.... | 21:44 |
| clayg | but just poking at the queue listings doesn't scale horizontally, one of the container walkers could dump stats as it goes and then you just ... divide by replica count? 🤮 | 21:45 |
| clayg | ok - so "monitor expiring queue" sounds like something everyone is keen on, but no objections to extending SLO delete in this fashion? | 21:47 |
| timburke | does anyone feel uncomfortable having the async-delete behavior as the default? | 21:48 |
| clayg | having the objects in the segment container listing until their cleared seems reasonable - and it doesn't hurt anything if the client ends up deleting them on their own | 21:48 |
| mattoliverau | no I think it's a great idea because it means we use an existing mechanism that is ment to do it. I'll review the patch today | 21:48 |
| clayg | once the async delete successfully queues everything - the manifest is gone | 21:49 |
| mattoliverau | timburke: I'm warming to the idea of it being the default | 21:49 |
| mattoliverau | Will think about it while I review the patch. | 21:50 |
| timburke | cool, sounds like we're good to go then, thanks. between mattoliverau, clayg, and zaitcev, i'm sure this is gonna turn out great :-) | 21:50 |
| timburke | #topic open discussion | 21:50 |
| *** openstack changes topic to "open discussion (Meeting topic: swift)" | 21:51 | |
| timburke | last few minutes: anything else to bring up? | 21:51 |
| timburke | i updated https://review.opendev.org/#/c/738959/ to have LIBERASURECODE_WRITE_LEGACY_CRC unset, set to "", and set to "0" all mean the same thing. i realized the commit message needs some clean up, but otherwise i think it's good to go | 21:53 |
| patchbot | patch 738959 - liberasurecode - Be willing to write fragments with legacy crc - 3 patch sets | 21:53 |
| clayg | 🥳 | 21:53 |
| kota_ | ok, thx. I'll circle back to the patch around Friday. | 21:53 |
| timburke | thanks kota_! | 21:53 |
| timburke | the swift side at https://review.opendev.org/#/c/739164/ should maybe be updated to set LIBERASURECODE_WRITE_LEGACY_CRC=1 instead of true, but that's pretty minor | 21:54 |
| patchbot | patch 739164 - swift - ec: Add an option to write fragments with legacy crc - 2 patch sets | 21:54 |
| clayg | I was playing with p 749400 trying to preserve existing behavior - but it got wonky when I put p 749401 on top of it 😞 | 21:55 |
| patchbot | https://review.opendev.org/#/c/749400/ - swift - proxy: Put storage policy index in object responses - 3 patch sets | 21:55 |
| patchbot | https://review.opendev.org/#/c/749401/ - swift - s3api: Ensure backend headers make it through s3api - 3 patch sets | 21:55 |
| clayg | existing behavior for symlinks relies on the client request headers getting a x-backend-storage-policy index - and the s3api bug is that proxy-logging doesn't see those swift_req headers | 21:57 |
| timburke | yeah, i'm getting less and less certain that we want p 749400 :-/ | 21:57 |
| patchbot | https://review.opendev.org/#/c/749400/ - swift - proxy: Put storage policy index in object responses - 3 patch sets | 21:57 |
| clayg | timburke: I tried changing it to x-backend-proxy-policy-index; but that was mostly because all the resp.headers.pop were making me nervous (i thought we'd leave 'em just in case) | 21:57 |
| timburke | it *seems like* the wort of thing that's reasonable... until you start actually seeing what it does to logs :-( | 21:58 |
| *** rbudden has quit IRC | 21:58 | |
| clayg | but it doesn't really matter - because if we're using the response headers it's whatever request was last - so swift api would prefer the client.req header and s3api wouldn't have that - so it'd use the resp header it just sucked | 21:58 |
| clayg | ah yeah - so for *me* the bug is just s3api metrics not having policy index since we started actually logging the s3api requests (instead of just their underlying swift request) | 21:59 |
| *** whoami-rajat__ has quit IRC | 21:59 | |
| clayg | putting the policy index in the response headers may not be the way to go - i'll try to refocus on just the s3api bug | 22:00 |
| timburke | i think we end up needing to make sure that s3api copies some (request!) headers back into the provided env from a subrequest env -- yuk | 22:00 |
| timburke | oh! one last thing: the election nomination period is over -- we should all read over TC candidacy messages so we're ready for the election | 22:00 |
| timburke | #link http://lists.openstack.org/pipermail/openstack-discuss/2020-September/017668.html | 22:00 |
| timburke | and #link https://governance.openstack.org/election/#wallaby-tc-candidates | 22:01 |
| clayg | and i'm also struggling with making SLO responses that fail mid-transfer show up as 503 in logs p 752770 but that is ALSO a request/resp/environ/context mess 😡 | 22:01 |
| patchbot | https://review.opendev.org/#/c/752770/ - swift - Log error processing manifest as ServerError - 3 patch sets | 22:01 |
| timburke | all right, we're about out of time | 22:01 |
| timburke | but we can argue about the wisdom of WSGI back in -swift ;-) | 22:02 |
| timburke | thank you all for coming, and thank you for working on swift! | 22:02 |
| timburke | #endmeeting | 22:02 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 22:02 | |
| openstack | Meeting ended Wed Sep 30 22:02:42 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 22:02 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/swift/2020/swift.2020-09-30-21.00.html | 22:02 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/swift/2020/swift.2020-09-30-21.00.txt | 22:02 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/swift/2020/swift.2020-09-30-21.00.log.html | 22:02 |
| *** andrebeltrami has quit IRC | 22:06 | |
| *** yamamoto has joined #openstack-meeting | 22:08 | |
| *** mlavalle has joined #openstack-meeting | 22:08 | |
| *** rcernin has joined #openstack-meeting | 22:12 | |
| *** slaweq has quit IRC | 22:17 | |
| *** slaweq has joined #openstack-meeting | 22:18 | |
| *** rfolco has quit IRC | 22:18 | |
| *** slaweq has quit IRC | 22:24 | |
| *** tosky has quit IRC | 22:54 | |
| *** mlavalle has quit IRC | 23:14 | |
| *** TrevorV has quit IRC | 23:20 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!