21:01:28 <timburke> #startmeeting swift
21:01:28 <opendevmeet> Meeting started Wed Jan 18 21:01:28 2023 UTC and is due to finish in 60 minutes.  The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:01:28 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:01:28 <opendevmeet> The meeting name has been set to 'swift'
21:01:36 <timburke> who's here for the swift meeting?
21:02:03 <mattoliver> o/
21:02:49 <zaitcev> o7
21:02:58 <kota> o/
21:03:05 <indianwhocodes> o/
21:03:08 <timburke> as usual, the agenda's at
21:03:10 <timburke> #link https://wiki.openstack.org/wiki/Meetings/Swift
21:03:27 <timburke> first up
21:03:57 <timburke> #topic CVE-2022-47950
21:04:28 <timburke> there's a critical CVE for swift that was made public this week
21:05:53 <timburke> security researchers at OVH discovered a way for authenticated clients to read arbitrary files from proxy-servers
21:06:08 <timburke> #link https://bugs.launchpad.net/swift/+bug/1998625
21:07:35 <timburke> this included things like tempauth credentials, keymaster root secrets, swift.conf swift_hash_path_prefix/suffix values, and auth_token service user credentials
21:08:22 <timburke> the good news is, this only affects clusters with S3 access enabled
21:09:35 <timburke> the bad news is, it's not limited to s3api; swift3 is also affected (for clusters that aren't on rocky yet)
21:10:04 <timburke> the fix is already merged to master
21:10:12 <timburke> #link https://review.opendev.org/c/openstack/swift/+/870823
21:10:45 <mattoliver> do we still have a swift3 repo? I haven't looked at that for years
21:10:58 <timburke> and backports are up for wallaby through zed -- though i'm realizing that the functional test may need a small update
21:11:37 <timburke> https://opendev.org/x/swift3/ is still a thing, though inactive
21:12:28 <mattoliver> kk
21:12:43 <timburke> maybe also worth noting: the github mirroring went away a while ago: https://github.com/openstack-archive/swift3/
21:13:42 <timburke> i don't think any patch is expected for swift3 -- i just want to be clear about the scope of affected clusters
21:14:15 <mattoliver> ok, and the readme does say it's frozen and all new patches goto the s3api middleware, so maybe its ok.
21:14:22 <mattoliver> but a good reason for anyone to finally upgrade from queens or before
21:15:19 <mattoliver> zaitcev: did you guys have anyone still on pre-rocky? or is that something to ask cschwede ?
21:15:26 <timburke> for sure! thankfully, even if operators can't upgrade directly to a more-recent swift, the code change is literally one line
21:15:29 <timburke> https://review.opendev.org/c/openstack/swift/+/870823/1/swift/common/middleware/s3api/etree.py
21:16:54 <zaitcev> mattoliver: Yes. We still have customers on Queens.
21:17:09 <timburke> unfortunately, the best mitigation i see for anyone that can't do *any* sort of code change is to disable S3 access
21:17:36 <mattoliver> the backport is going to what, as low as wallaby you said, so its probably more of a case to anything before there.
21:17:50 <mattoliver> Like you say tho isn't a 1 liner
21:18:29 <mattoliver> if redhat still package swift3 for queens maybe they can just at a patch to it.
21:18:41 <timburke> if i can get stable gates happy again, i'm happy to do more backports -- more on that later, tho
21:18:54 <mattoliver> kk
21:20:40 <timburke> any other questions or comments on the CVE? i want to give everyone a chance to digest it a bit
21:23:17 <timburke> all right, we'll move on -- if anyone needs more info about it, feel free to reach out to me, either in channel or privately
21:23:34 <timburke> and i'd like to give special thanks to OVH for discovering the issue, mattoliver acoles and clayg for reviewing the patch, and fungi for getting the patches into gerrit!
21:23:52 <timburke> #topic vPTG
21:24:08 <fungi> my pleasure!
21:24:22 <timburke> just a reminder that there's a vPTG scheduled for the end of march
21:24:55 <timburke> mattoliver and i still need to get doodle poll and etherpad up
21:25:19 <timburke> but everyone should go ahead and register if they haven't already!
21:25:21 <timburke> #link https://www.eventbrite.com/e/project-teams-gathering-march-2023-tickets-483971570997
21:25:43 <mattoliver> oh yeah, I'll get on that etherpad, so we can start gathering ideas
21:25:54 <timburke> thanks mattoliver
21:26:10 <timburke> #topic gate health
21:26:22 <timburke> another week, another gate breakage
21:26:34 <mattoliver> sigh
21:27:02 <timburke> dnspython 2.3.0 was recently released, which caused eventlet's monkey-patching of it to break
21:27:32 <mattoliver> oh great :(
21:27:35 <timburke> this would happen as soon as you went to import eventlet
21:27:42 <timburke> #link https://github.com/eventlet/eventlet/issues/781
21:28:14 <timburke> good news is that a fix has merged to eventlet, and temoto tagged a new 0.33.3 release that includes it
21:28:27 <mattoliver> So we need to pin the package until there is an upstream fix?
21:28:34 <mattoliver> oh wow, I like a quick turn around!
21:29:08 <timburke> ...yes and no. so, we *should* have been safe from this because of the general openstack upper-constraints policy
21:29:38 <timburke> https://github.com/openstack/requirements/blob/master/upper-constraints.txt still lists dnspython===2.2.1
21:30:00 <timburke> and in fact, *most* of our jobs were still fine
21:30:31 <timburke> buuuut... our docs build previously only specified the extra doc-building requirements
21:30:31 <mattoliver> hey, it looks like you fixed it in upstream eventlet timburke, nice work!
21:30:48 <timburke> :D
21:31:25 <timburke> i also went ahead and fixed our docs job to properly use constraints
21:31:27 <timburke> #link https://review.opendev.org/c/openstack/swift/+/870853
21:31:39 <fungi> if it makes anyone feel any better (it probably doesn't), you're not alone. mistral spotted the exact same problem today, so i was able to point amorin at timburke's fix. worked like a charm there as well
21:32:14 <timburke> 👍 as long as the pain saves someone else some trouble, right?
21:33:16 <mattoliver> nice
21:33:18 <timburke> really, i don't think i would've minded *nearly* as much if we weren't also trying to get the CVE fix merged
21:34:52 <timburke> the doc-building fix also got backported -- wallaby through zed all have it now
21:34:52 <mattoliver> lol, true
21:35:32 <timburke> victoria started running into other issues, though
21:35:34 <timburke> #link https://review.opendev.org/c/openstack/swift/+/870784
21:36:53 <timburke> this is unfortunate and annoying -- not two weeks ago that gate was working and we merged the fix for the cpython '//' bug
21:36:56 <timburke> #link https://review.opendev.org/c/openstack/swift/+/868143
21:38:48 <timburke> i'll work on trying to get it functional again, but wanted to float a couple ideas
21:39:25 <timburke> 1. declare more stable branches end-of-life
21:40:02 <mattoliver> failures seem to be caused by the smmap package verson mismatch, maybe needs a specific pin?
21:40:10 <timburke> 2. remove (or mark non-voting) some/many jobs from stable branches
21:41:36 <mattoliver> yeah true, is there an openstack policy on how many stable branches to maintain?
21:41:45 <timburke> mattoliver, yeah -- i think it's probably related to py2 support. thinking longer term, though, i'm willing to bet we'll see similar issues cropping up for py36 (which is no longer maintained by upstream cpython)
21:42:56 <mattoliver> yeah, definitely looks py2, as obvioulsy the new version of smmap looks only py3+
21:42:59 <mattoliver> so yeah, we're just going to get more and more of these
21:43:30 <timburke> there's https://docs.openstack.org/project-team-guide/stable-branches.html -- "OpenStack development typically has 3 branches active at any point of time, master (the current development release), stable (the most recent release) and oldstable (previous release)."
21:43:45 <mattoliver> we either need to lock all pre py3 branches down, pinning everything. Or maybe mark them as non-voting knowing that py2 support has bacially gone
21:44:12 <timburke> with the introduction of "extended maintenance" i'm not sure that quote is entirely true
21:44:24 <mattoliver> Well based on n-2, we could remove a bunch of stable
21:44:54 <mattoliver> although I know redhat do support for longer. So do they need more stable branches upstream?
21:45:59 <mattoliver> or do we just need to support n-2 + some point in time LTS, but the latter can be non-voting for a bunch of tests?
21:47:38 <timburke> my impression is that we have a lot of discretion about both how many stable branches to support and what level of testing and support "extended maintenance" should mean
21:48:19 <mattoliver> extended maintence does say reduced CI commitment which makes me think, non-voting or even removed gate checks
21:49:19 <timburke> fwiw, i've left so many stable branches open so far for exactly the sort of situation in which we find ourselves: a pretty big security issue for which it'd be nice to offer downstream packagers an official release for old versions
21:49:45 <mattoliver> yeah
21:49:50 <mattoliver> which is good
21:50:34 <timburke> but if it turns into me fighting with CI off and on for a month or two to be able to land *anything*, i'm not sure it's worth it
21:51:55 <mattoliver> go n-2 of them should be supported (cI etc) the rest we should be able to reduce  CI committment. maybe reduce it to pep8 and unit tests?
21:52:16 <mattoliver> I guess functional and probe too, but we seem to be having package issues breaking those
21:53:08 <mattoliver> OR, do we just force in this security bug to the old old stables and then EOF everything below wallaby and be done with it for now?
21:53:20 <mattoliver> *security patch
21:53:36 <timburke> not on victoria, at least. could keep in-process func tests but drop dsvm, for example
21:53:53 <mattoliver> true
21:54:37 <timburke> idk -- i'll poke at it for another day or two, see what i can come up with. seems like a good topic for the PTG tho :-)
21:54:43 <timburke> #topic open discussion
21:54:51 <timburke> anything else we should bring up this week?
21:55:07 <mattoliver> Well I can't speak for anyone, but for now, I think reduce CI to make them work and get the current security fix in, then we can discuss ditching them at the PTG
21:55:38 <mattoliver> or at least having our own definition of reduced CI committment that doesn't take all your time
21:55:47 <mattoliver> also sorry about it always falling to you timburke
21:56:43 <timburke> eh, no worries -- there are worse things. just feels like i'm spinning my wheels sometimes, tho
21:58:13 <mattoliver> OK open discussion, I'm playing with a sharding statemachine change to just freakin update the state_timestamp everytime a state changes (why we don't already is historic), but I feel it'll make life easier. So having a play with how much churn it causes
21:58:15 <mattoliver> alot it seems in tests.
21:58:23 <mattoliver> haven't pushed it up yet, and it's currently only an experiment, but if it works I think it'll make the early cleave and active stuff much more trivial to deal with.
21:59:18 <timburke> nice! looking forward to it
21:59:21 <mattoliver> trying to tread carefully because the state_timestamp is used to cycle round the statemachine, and statemachine changes can be fickle, so hoping I haven't missed something past al or I knew that currement me doesn't.
21:59:47 <timburke> i do wonder why it isn't like that already...
22:00:29 <mattoliver> yeah, I agree, and it's because state_timestamp was added later to allow us to do back to active in roots, ie restart the statemachine
22:00:59 <timburke> all right, we're at time so i'll call it
22:01:07 <mattoliver> kk
22:01:12 <timburke> thank you all for coming, and thank you for working on swift!
22:01:15 <timburke> #endmeeting