| opendevreview | Merged openstack/pyeclib master: Clean up install warning https://review.opendev.org/c/openstack/pyeclib/+/815407 | 01:25 |
|---|---|---|
| opendevreview | Merged openstack/pyeclib master: Clean up compile warning https://review.opendev.org/c/openstack/pyeclib/+/815408 | 01:27 |
| opendevreview | Tim Burke proposed openstack/swift master: Ensure close socket for memcached if got timeout https://review.opendev.org/c/openstack/swift/+/338819 | 01:29 |
| opendevreview | Dr. Jens Harbott proposed openstack/swift master: Clean up devstack job definitions https://review.opendev.org/c/openstack/swift/+/817345 | 05:40 |
| opendevreview | Merged openstack/swift master: Clean up devstack job definitions https://review.opendev.org/c/openstack/swift/+/817345 | 19:42 |
| acoles | timburke: apologies, I won't make today's meeting | 20:02 |
| timburke | no worries. i should refresh the agenda | 20:03 |
| timburke | #startmeeting swift | 21:00 |
| opendevmeet | Meeting started Wed Nov 10 21:00:03 2021 UTC and is due to finish in 60 minutes. The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot. | 21:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 21:00 |
| opendevmeet | The meeting name has been set to 'swift' | 21:00 |
| timburke | who's here for the swift meeting? | 21:00 |
| kota | hi | 21:00 |
| mattoliver | o/ | 21:02 |
| timburke | i know acoles won't make it; we'll see if we pick up clayg or zaitcev as we go ;-) | 21:02 |
| timburke | as usual, the agenda's at https://wiki.openstack.org/wiki/Meetings/Swift | 21:02 |
| timburke | first up | 21:02 |
| timburke | #topic next meeting | 21:02 |
| timburke | i had a death in the family, and i'm unlikely to be able to be able to chair next week | 21:03 |
| zaitcev | Oh, right. The UTC. | 21:03 |
| zaitcev | Sorry for your loss. | 21:03 |
| kota | sorry to hear that | 21:04 |
| timburke | thanks all | 21:04 |
| timburke | given the US holiday the week after, i propose we set the next meeting for the week after | 21:04 |
| mattoliver | timburke: sorry to hear that. | 21:04 |
| mattoliver | KK sounds good, if something comes up in the meantime, I'm happy to chair if need be. | 21:05 |
| mattoliver | but a few weeks off is also ok :) | 21:05 |
| timburke | 👍 | 21:05 |
| timburke | there's no reason i *have* to be there ;-) | 21:05 |
| timburke | next up | 21:05 |
| timburke | #topic https://bugs.launchpad.net/swift/+bug/1685798 | 21:06 |
| timburke | i think clayg wanted to bring this up | 21:06 |
| timburke | it's an old but to do with logging tempurl signatures | 21:06 |
| timburke | even has a CVE attached | 21:06 |
| clayg | so it's legit? | 21:07 |
| clayg | I was thinking we could just change the status to resolved or whatever and no one would know | 21:07 |
| timburke | i think cschwede even has a patch attached? | 21:07 |
| clayg | well now that it's public we can just put that in gerrit then? | 21:07 |
| clayg | git it rebased and merged and THEN mark it as resolved or whatever? | 21:07 |
| timburke | clayg, idk -- i'd view it as more of a security hardening opportunity *shrug* | 21:08 |
| clayg | that seems totally legit 😎 | 21:08 |
| mattoliver | sounds like a plan (not having looked at the patch) | 21:08 |
| timburke | iirc the one complication was that there's presumably a similar issue with s3api presigned requests | 21:08 |
| timburke | who wants to take ownership on getting the patch rebased and merged? | 21:09 |
| mattoliver | I'll do it. | 21:09 |
| mattoliver | unless clayg has a burning desire | 21:09 |
| timburke | thanks mattoliver | 21:10 |
| opendevreview | Clay Gerrard proposed openstack/swift master: Fix CVE-2017-8761 https://review.opendev.org/c/openstack/swift/+/817476 | 21:10 |
| timburke | or mattoliver will review it ;-) | 21:10 |
| mattoliver | oh even easier :P | 21:10 |
| mattoliver | lol yup | 21:10 |
| clayg | this was great - can we leave it on the agenda and follow up in two weeks as to why the bug is not closed or close it | 21:11 |
| timburke | 👍 | 21:11 |
| timburke | #topic storage policies and sharding | 21:11 |
| timburke | i just wanted to check in on the current thinking for this issue -- i feel like things have gone in a few different directions | 21:12 |
| mattoliver | Good question, and I haven't really been following along, but also wanted to get up to speed. | 21:13 |
| timburke | i guess it's on clayg then, with acoles not here ;-) | 21:13 |
| zaitcev | What's the link between the policies and sharding? | 21:13 |
| timburke | so *that much* i did get loaded into my head | 21:14 |
| timburke | the issue is that a sharded container could get deleted then recreated, and the shards won't pick up the right storage policy | 21:14 |
| mattoliver | acoles: and clayg have been playing with reworking the reconciler cmp_policy_info | 21:15 |
| mattoliver | based around the discussions we had at PTG | 21:15 |
| timburke | it can lead to some dark data iirc, where pre-sharding the reconciler would eventually move the data to the right policy and it'd show up in listings again | 21:15 |
| mattoliver | But the getting policy index from root -> shards is still a thing that needs to happen | 21:15 |
| mattoliver | So we can either fix up reconciler cmp function and then fix shards | 21:16 |
| mattoliver | or fix shards and work on the cmp function as seperate things. | 21:17 |
| mattoliver | I guess if we make decisions on shards as we do a root using cmp_policy_info then when it itself is fixed we'd still get getting root -> shards fixed/improved. | 21:18 |
| mattoliver | I'll take a look at the current state of the root -> shard policy_index patches today to see what state their in, report back here in channel and maybe we can make a decision. | 21:19 |
| timburke | sounds good, thanks mattoliver. worst case, we can check in again next meeting | 21:20 |
| mattoliver | yup | 21:20 |
| timburke | #topic memcache issues | 21:20 |
| timburke | i hit an issue in my home cluster that was a little funny to run down, but i kicked up a patch out of it | 21:21 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/817307 | 21:21 |
| timburke | basically, the memcache pool could be depleted (and never replenished) if a MemcachePoolTimeout gets raised while we're trying to create a new connection | 21:22 |
| zaitcev | Oh I see | 21:22 |
| mattoliver | oh interesting | 21:22 |
| mattoliver | your home cluster is a good source of bugs, nice :) | 21:23 |
| timburke | while i was trying to functionally test the fix, i noticed that i wouldn't actually drop to zero open connections to memcache | 21:23 |
| timburke | which i suspect was due to *another* issue that kota already had a patch for :-) | 21:24 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/338819 | 21:24 |
| kota | Oh, i think it's much old one | 21:24 |
| mattoliver | nice one kota | 21:25 |
| timburke | only 5 years or so ;-) | 21:25 |
| mattoliver | 5 years ago | 21:25 |
| timburke | sorry i hadn't reviewed it :-) | 21:25 |
| timburke | i changed a few things, so it might be good to have someone else take a look too, but LGTM | 21:26 |
| timburke | i mostly just wanted to raise awareness of the issue -- fwiw, it'd manifest (for me) as the proxy returning 401 for all requests, since it couldn't contact memcache at all | 21:27 |
| timburke | if you've got keystone or some other system where your tokens are stored in more than just memcache, it would likely manifest as dramatically increased load on your auth system | 21:28 |
| timburke | #topic request tracing | 21:28 |
| timburke | mattoliver, i think i saw some more movement on these patches, is that right? | 21:29 |
| mattoliver | Yup, I've cleaned it up a bunch. So it's much smaller then it was. | 21:29 |
| mattoliver | And I've now implemented a new span per proxy request to storage nodes (I think I got them all), and if there is a timeout we log the exception to that span. | 21:30 |
| mattoliver | in essence, we can see when it fails to communitcate to a node, and which node that was | 21:31 |
| mattoliver | Previously it'll only track through responses, so failing to talk to nodes wouldn't show up. | 21:31 |
| timburke | very cool! is there anything we need to discuss for it, or does it mainly just need review at this point? | 21:31 |
| mattoliver | yeah, if you want to review it, great, but it's still in a bit of flux. | 21:32 |
| mattoliver | trying to get it working best it can because I need to write a talk on it for Linux.conf.au | 21:32 |
| timburke | sounds good | 21:33 |
| timburke | all right, that's all i've got | 21:34 |
| timburke | #topic open discussion | 21:34 |
| timburke | anything else we ought to bring up today? | 21:34 |
| mattoliver | Here's a link to my random presentation of quickly wrote up because my tweet I sent about the PTG was asked by an Linux Australia concil member to be submitted as a talk: https://lca2022.linux.org.au/schedule/presentation/32/ | 21:34 |
| mattoliver | and it was approved, so it's motivating me to get tracing working well :P | 21:34 |
| timburke | \o/ | 21:35 |
| timburke | all right, let's call it | 21:38 |
| timburke | thank you all for coming, and thank you for working on swift! | 21:38 |
| timburke | #endmeeting | 21:38 |
| opendevmeet | Meeting ended Wed Nov 10 21:38:32 2021 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 21:38 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/swift/2021/swift.2021-11-10-21.00.html | 21:38 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/swift/2021/swift.2021-11-10-21.00.txt | 21:38 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/swift/2021/swift.2021-11-10-21.00.log.html | 21:38 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!