21:00:08 <timburke> #startmeeting swift
21:00:08 <opendevmeet> Meeting started Wed Nov  9 21:00:08 2022 UTC and is due to finish in 60 minutes.  The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:08 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:08 <opendevmeet> The meeting name has been set to 'swift'
21:00:18 <timburke> who's here for the swift team meeting?
21:00:38 <mattoliver> o/
21:00:56 <mattoliver> (looks like I just made it in time)
21:01:51 <timburke> sorry to have skipped out on so many meetings -- but i did at least update the agenda today!
21:01:59 <timburke> #link https://wiki.openstack.org/wiki/Meetings/Swift
21:02:19 <timburke> i only have a couple topics, so let's get started
21:02:31 <timburke> #topic broken func tests
21:03:13 <timburke> recently there was a fix for a CPython CVE
21:03:16 <timburke> #link https://github.com/python/cpython/issues/87389
21:03:39 <kota> sorry, late a bit
21:03:42 <timburke> that's been working its way through backports
21:03:59 <timburke> #link https://python-security.readthedocs.io/vuln/http-server-redirection.html
21:04:42 <mattoliver> ahh that explains the // patch I saw pop up
21:05:54 <timburke> the fix made it so urls like http://swift.example.com//v1/AUTH_test (which used to 400) would get translated to http://swift.example.com/v1/AUTH_test before they get to us
21:06:57 <timburke> which doesn't seem so bad -- but when you have something like domain_remap, it can change the object name for a url like http://container.account.swift.example.com//object
21:07:25 <timburke> i've written up a bug for them
21:07:27 <timburke> #link https://github.com/python/cpython/issues/99220
21:08:20 <timburke> and a patch for us
21:08:23 <timburke> #link https://review.opendev.org/c/openstack/swift/+/863441
21:09:23 <timburke> we first saw the issue with some non-voting centos9/fips func tests, but since they were non-voting the investigation wasn't really prioritized
21:10:22 <timburke> recently, though, it started affecting our rolling upgrade jobs -- and since there's no fixing old proxy code, i had to just make them non-voting for now
21:10:31 <timburke> #link https://review.opendev.org/c/openstack/swift/+/863929
21:11:31 <timburke> if anyone has time to review the fix, that'd be much appreciated! once we have a fix landed, i'll propose backports, too
21:12:15 <timburke> any questions or comments?
21:12:38 <mattoliver> I'll put it on my list. Although I've been really distracted these last few weeks. But will try to get to them in the next day or 2.
21:12:57 <timburke> thanks
21:14:01 <timburke> not that i really *want* to maintain a new library, but eventually i feel like we'll kind of have to just write our own request parser :-(
21:14:15 <timburke> speaking of cpython...
21:14:22 <timburke> #topic testing under py310
21:14:41 <timburke> we're getting closer and closer to testing on py310!
21:15:19 <timburke> the pytest is nearly passing
21:15:22 <timburke> #link https://review.opendev.org/c/openstack/swift/+/851099
21:15:42 <timburke> "the pytest patch" i mean
21:16:11 <mattoliver> lol, yeah, it's been harder then expected to just work.
21:16:27 <timburke> the only remaining issue is the requirements-check
21:16:31 <mattoliver> mostly due to requirement versions etc
21:16:48 <timburke> so i proposed a patch to add pytest-cov to requirements
21:16:52 <timburke> #link https://review.opendev.org/c/openstack/requirements/+/863581/
21:18:05 <timburke> and fortunately, once those land, it looks like py310 unit tests already pass
21:18:07 <timburke> #link https://review.opendev.org/c/openstack/swift/+/850947
21:18:26 <mattoliver> oh nice!
21:19:39 <timburke> so we're getting there! next, i might try to switch our func (or even probe) tests to jammy
21:20:18 <mattoliver> oh good idea.
21:20:25 <timburke> that's all i've got
21:20:31 <timburke> #topic open discussion
21:20:40 <timburke> what else should we discuss this week?
21:21:35 <mattoliver> I've been looking at Al's shard ranges from root final patch. Looking good. Think I'm almost happy it wont break things. But will solve some brittleness
21:22:22 <mattoliver> I also talked to clay briefly about ring v2. And we wants me to revisit the builder into ring stuff.
21:23:29 <timburke> nice -- am i right to think that acoles's patch you're talking about is https://review.opendev.org/c/openstack/swift/+/852905 ?
21:23:36 <mattoliver> I said probably needs some disucssions esp with timburke regarding my serialzation classes stuff. So I think I have him onboard with landing the first few patches.
21:23:50 <mattoliver> yup that's it
21:24:21 <mattoliver> then looping back to get ringbulder classes after
21:25:05 <timburke> i definitely like the idea of having some movement on that chain :-)
21:25:31 <mattoliver> the default will be v1 rings, so if need be we can wait a bit if we want to include more bits into it (before reworking our downstream controller).
21:25:46 <mattoliver> yeah +1
21:26:16 <mattoliver> So I plan, if I'm not pulled into any more fires, to prioritize re-reviewing those first 2 and getting them landed
21:26:36 <timburke> 👍
21:27:25 <mattoliver> I also reworked the Otel tracing chain to support an Otel Collector.
21:27:32 <mattoliver> so not just jaeger
21:27:44 <mattoliver> As we have movement on wanting to test that at work.
21:27:50 <timburke> nice
21:28:21 <mattoliver> #link https://review.opendev.org/c/openstack/swift/+/857559
21:28:43 <timburke> oh! i also have a patch to improve wsgi server start-up time when you configure a bunch of workers
21:28:48 <timburke> #link https://review.opendev.org/c/openstack/swift/+/862843
21:28:49 <mattoliver> And I sometimes do wonder if I should convince someone in testing out the sharded pending files patch.
21:28:59 <mattoliver> oh nice one!
21:29:38 <mattoliver> great, I have notices a "reload" can take a while. I assume this would speed that up too?
21:29:57 <mattoliver> though that might be caused by finishing old requests.
21:30:12 <timburke> yep -- i saw a reload that took ~70s come down to ~15s
21:30:22 <mattoliver> \o/
21:30:26 <mattoliver> that's awesome!
21:30:32 <timburke> reload should just wait for the new workers to come up; old workers can continue running
21:31:32 <mattoliver> The reload stuff is awesome btw!
21:31:56 <mattoliver> we used it a bunch in a new mcrouter rollout. And it's freakin cool!
21:32:37 <timburke> nice!
21:33:13 <timburke> oh! speaking of old workers -- i added a configurable timeout following a reload to start stopping workers more forcefully
21:33:15 <timburke> #link https://review.opendev.org/c/openstack/swift/+/789035
21:33:53 <timburke> that patch started out just being about quieting some log messages, but i really like having that extra state on hand
21:34:22 <mattoliver> nice
21:34:50 <mattoliver> I mean maybe they're dealing with a long running request.. but nice to have a hammer
21:35:51 <timburke> yeah... 2hr might be a little short for the default, i suppose...
21:37:29 <mattoliver> who knows, that's one of those hard things to pick
21:37:34 <mattoliver> how long is a peice of string.
21:37:36 <timburke> as much as anything, i just want to give ops a way that they can be *sure* no old workers are running with stale configs
21:37:55 <mattoliver> 2hrs seems like a ok starting point. and we can extend from there
21:38:10 <mattoliver> +1
21:39:26 <mattoliver> That's all I have for the moment
21:39:26 <timburke> all right, i think i'll call it
21:39:37 <timburke> thank you for coming, and thank you for working on swift!
21:39:40 <timburke> #endmeeting