Wednesday, 2023-05-31

opendevreviewMerged openstack/swift master: memcached: handle failed incr+add+incr better
opendevreviewRomain de Joux proposed openstack/swift master: Encode header in latin-1 with wsgi_to_bytes
opendevreviewMerged openstack/swift master: memcached: add method and time_spent to exception logs
opendevreviewTim Burke proposed openstack/swift master: proxy: Bring back logging for get_*_info requests
opendevreviewTim Burke proposed openstack/swift master: proxy: Bring back logging/metrics for get_*_info requests
timburke#startmeeting swift21:00
opendevmeetMeeting started Wed May 31 21:00:33 2023 UTC and is due to finish in 60 minutes.  The chair is timburke. Information about MeetBot at
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.21:00
opendevmeetThe meeting name has been set to 'swift'21:00
timburkewho's here for the swift meeting?21:00
timburkeas usual, the agenda's at
timburkei only had a couple items i wanted to bring up21:02
zaitcevI don't know why, but I'm here. I guess I'll just try and ask acoles to look at
timburkei keep meaning to circle back on that one, too :-(21:02
timburke#topic py3 ssync metadata bug21:03
timburkeand the fix at21:04
acoleszaitcev: darn, sorry!21:04
timburkei haven't left my review yet, but i should later today. i've got a couple small changes to tests i'd like to see, and i'm part way into updating a probe test -- but it's a little tricky since swiftclient can't parse the non-ascii meta :-(21:05
timburkecore change looks correct, though21:06
timburkei'll probably push over with my unit test changes but leave the probe test as a follow-up21:07
timburke#topic logging/metrics for account/container info requests21:07
timburkei realized this morning that we no longer emit subrequest logging/metrics for get_account/container_info requests21:08
timburkeit was an unintended consequence of
timburkeby going directly to the proxy-app, i bypassed logging :-(21:09
timburkei've got a fix up -- it's still a little ugly, though21:09
mattolivero/ (sorry im late)21:11
acolesI'm reminded of some patch mattoliver was working on where we added a pipeline property that held a list of the apps...was it the internal client iter_shard_ranges patch?21:11
timburkei'm torn about whether to stick with what i've got there (take the second-to-last item in the pipeline, assuming it's a ProxyLoggingMiddleware, otherwise go to the proxy app), or go searching for the right-most logging filter21:12
timburkeacoles, i think21:12
mattoliverYeah we're was that. But yeah it stored them as a list, so it would be easy to search21:13
acolesyeah, thats' where we added _pipeline21:13
timburkeit's definitely handy for this fix :-)21:13
acolestimburke: are you concerned about the overhead of searching every time?21:14
timburkeno -- my concern is more that there could be some middleware with unexpected consequences in between the last logging mware and proxy-server21:15
timburkesomething that could do some sysmeta manipulations, say. also, a little worried about the possibility of recursion errors21:16
acolesoh, now I'm confused...can you elaborate? what advantage does _pipeline[-2] have?21:16
zaitcevI gave up wrapping my mind around that too.21:17
acolesI guess [-2] means logging is the last middleware, then there's the app21:19
timburkeif it's proxy logging, great. we wrote that and can trust it. otherwise, the patch as-written goes to the proxy app; we lose logging/metrics again, but at least we know all sysmeta, acls, etc in the response is intact21:19
timburkei don't think it's *likely* that anyone would have configured their pipelines with a problematic middleware in between last logging and proxy-server, but it made me a little nervous21:20
acolesbut documenting 'if you put *any* middleware after the final logger then there's some metric that may not be emitted' seems odd vs documenting 'if you put middleware after logging that does weird stuff then blah blah'21:20
acoleswhat would have happened, with a weird middleware, before you added the cut-through21:22
timburkeyou'd get weird behaviors. all the more so if it's non-deterministic at which point in the pipeline the get-info call happens21:23
timburkegetting rid of the possibility of weird behaviors was definitely on my mind when i did the cut-through -- i'm not thrilled about re-introducing them :-/ but maybe it's the most defensible path21:24
acolesI think I need to study the patch and the weirdness potential when I'm more awake21:25
acolesthanks for alerting us to it!21:25
timburkeseems fair. fwiw, one of the weird cases i was thinking of was something like the 1space shunt, where you're trying to merge namespaces from different accounts/clusters21:26
timburkeanyway, that's all i've got21:26
timburke#topic open discussion21:26
timburkewhat else should we bring up this week?21:26
mattoliverI've been out this week, so I've got nothing I can think of atm21:27
zaitcevMe neither.21:27
timburkeall right. i think i'll wrap it early, then21:28
acoleswe shipped this change this week
acolesand it does seem to have eliminated some curious timeouts21:28
timburkeyes! it seemed so strange that we were bumping memcached timeouts at all, much less so long after the configured timeout21:30
acolesI guess one takeaway is that the network could move bytes faster than the proxy could EC encode them, so network never would block and the EC put greenthread never would yield to another greenthread :O21:30
timburkeall right21:32
timburkethank you all for coming, and thank you for working on swift!21:32
opendevmeetMeeting ended Wed May 31 21:32:14 2023 UTC.  Information about MeetBot at . (v 0.1.4)21:32
opendevmeetMinutes (text):
opendevreviewTim Burke proposed openstack/swift master: Encode header in latin-1 with wsgi_to_bytes
opendevreviewTim Burke proposed openstack/swift master: Add non-ascii meta to ssync probe test

Generated by 2.17.3 by Marius Gedminas - find it at!