Wednesday, 2020-09-02

*** martial has quit IRC00:08
*** baojg has quit IRC00:09
*** baojg has joined #openstack-meeting00:10
*** mlavalle has quit IRC00:27
*** yasufum has joined #openstack-meeting00:30
*** mlavalle has joined #openstack-meeting00:31
*** TrevorV has joined #openstack-meeting00:33
*** yasufum has quit IRC00:42
*** ayoung has quit IRC00:43
*** ykatabam has quit IRC00:43
*** ayoung has joined #openstack-meeting00:46
*** rcernin_ has joined #openstack-meeting00:46
*** rcernin has quit IRC00:47
*** ykatabam has joined #openstack-meeting00:48
*** masahito has joined #openstack-meeting01:04
*** ijw__ has quit IRC01:20
*** ijw has joined #openstack-meeting01:20
*** njohnston has quit IRC01:21
*** ayoung has quit IRC01:29
*** ayoung has joined #openstack-meeting01:30
*** ayoung has quit IRC01:34
*** ykatabam has quit IRC01:35
*** ykatabam has joined #openstack-meeting01:36
*** manuvakery has joined #openstack-meeting02:19
*** baojg has quit IRC02:32
*** baojg has joined #openstack-meeting02:33
*** ykatabam has quit IRC03:10
*** rcernin_ has quit IRC03:19
*** ykatabam has joined #openstack-meeting03:33
*** rcernin_ has joined #openstack-meeting03:35
*** psachin has joined #openstack-meeting03:35
*** masahito has quit IRC03:40
*** TrevorV has quit IRC03:43
*** whoami-rajat__ has joined #openstack-meeting03:55
*** whoami-rajat has joined #openstack-meeting03:56
*** zaneb has quit IRC04:12
*** zaneb has joined #openstack-meeting04:12
*** moguimar has joined #openstack-meeting04:16
*** mahatic has joined #openstack-meeting04:27
*** evrardjp has quit IRC04:33
*** evrardjp has joined #openstack-meeting04:33
*** apetrich has joined #openstack-meeting04:36
*** masahito has joined #openstack-meeting04:38
*** manuvakery has quit IRC04:39
*** rh-jelabarre has quit IRC04:41
*** apetrich has quit IRC04:41
*** vishalmanchanda has joined #openstack-meeting04:55
*** viks____ has joined #openstack-meeting04:59
*** moguimar has quit IRC05:03
*** gyee has quit IRC05:07
*** baojg has quit IRC05:12
*** baojg has joined #openstack-meeting05:13
*** yasufum has joined #openstack-meeting05:14
*** armax has quit IRC05:24
*** kevinz has joined #openstack-meeting05:34
*** slaweq has joined #openstack-meeting05:53
*** manpreet has joined #openstack-meeting05:53
*** yasufum has quit IRC05:54
*** lamt has quit IRC06:00
*** viks____ has quit IRC06:01
*** luyao has quit IRC06:01
*** arne_wiebalck has quit IRC06:01
*** arne_wiebalck has joined #openstack-meeting06:02
*** zeestrat has quit IRC06:02
*** luyao has joined #openstack-meeting06:02
*** zeestrat has joined #openstack-meeting06:02
*** baojg has quit IRC06:02
*** evgenyl has quit IRC06:03
*** evgenyl_ has joined #openstack-meeting06:03
*** baojg has joined #openstack-meeting06:03
*** viks____ has joined #openstack-meeting06:04
*** slaweq has quit IRC06:18
*** slaweq has joined #openstack-meeting06:24
*** slaweq has quit IRC06:29
*** ralonsoh has joined #openstack-meeting06:39
*** apetrich has joined #openstack-meeting06:41
*** diablo_rojo has quit IRC06:47
*** rcernin_ has quit IRC06:51
*** slaweq has joined #openstack-meeting06:55
*** rcernin_ has joined #openstack-meeting06:55
*** manuvakery has joined #openstack-meeting06:57
*** ralonsoh has quit IRC07:04
*** ralonsoh has joined #openstack-meeting07:09
*** e0ne has joined #openstack-meeting07:26
*** ZhuXiaoYu has joined #openstack-meeting07:27
*** ZhuXiaoYu has quit IRC07:28
*** rcernin_ has quit IRC07:35
*** tosky has joined #openstack-meeting07:37
*** ralonsoh has quit IRC07:43
*** ykatabam has quit IRC07:44
*** baojg has quit IRC07:53
*** baojg has joined #openstack-meeting07:53
*** baojg has quit IRC07:59
*** moguimar has joined #openstack-meeting08:30
*** rcernin_ has joined #openstack-meeting08:40
*** rcernin_ has quit IRC08:46
*** baojg has joined #openstack-meeting09:28
*** baojg has quit IRC09:54
*** ykatabam has joined #openstack-meeting10:18
*** ijw has quit IRC10:19
*** ijw has joined #openstack-meeting10:20
*** ijw has quit IRC10:25
*** ykatabam has quit IRC10:38
*** ijw has joined #openstack-meeting10:39
*** ijw has quit IRC10:44
*** lpetrut has joined #openstack-meeting10:47
*** armax has joined #openstack-meeting10:55
*** masahito has quit IRC11:00
*** sridharg has joined #openstack-meeting11:03
*** ijw has joined #openstack-meeting11:11
*** ijw has quit IRC11:15
*** armstrong has joined #openstack-meeting11:36
*** ijw has joined #openstack-meeting11:42
*** Lucas_Gray has joined #openstack-meeting11:47
*** ijw has quit IRC11:47
*** rh-jelabarre has joined #openstack-meeting11:53
*** raildo has joined #openstack-meeting11:56
*** Lucas_Gray has quit IRC11:57
*** moguimar has quit IRC12:00
*** Lucas_Gray has joined #openstack-meeting12:05
*** moguimar has joined #openstack-meeting12:09
*** sridharg has quit IRC12:12
*** ijw has joined #openstack-meeting12:14
*** ijw has quit IRC12:19
*** Lucas_Gray has quit IRC12:30
*** moguimar has quit IRC12:32
*** moguimar has joined #openstack-meeting12:35
*** lbragstad has quit IRC12:35
*** lbragstad has joined #openstack-meeting12:38
*** Lucas_Gray has joined #openstack-meeting12:39
*** ijw has joined #openstack-meeting12:46
*** moguimar has quit IRC12:59
*** moguimar has joined #openstack-meeting13:10
*** johanssone has quit IRC13:11
*** _mlavalle_1 has joined #openstack-meeting13:44
*** armstrong has quit IRC13:45
*** TrevorV has joined #openstack-meeting13:46
*** mlavalle has quit IRC13:46
*** Liang__ has joined #openstack-meeting13:58
*** Liang__ is now known as LiangFang13:59
*** ijw has quit IRC14:00
*** Lucas_Gray has quit IRC14:05
*** sluna has joined #openstack-meeting14:09
*** LiangFang has quit IRC14:12
*** _mlavalle_2 has joined #openstack-meeting14:14
*** Liang__ has joined #openstack-meeting14:14
*** ijw has joined #openstack-meeting14:17
*** _mlavalle_1 has quit IRC14:17
*** ijw has quit IRC14:21
*** Lucas_Gray has joined #openstack-meeting14:32
*** johanssone has joined #openstack-meeting14:36
*** ijw has joined #openstack-meeting14:48
*** _mlavalle3 has joined #openstack-meeting14:50
*** _mlavalle_2 has quit IRC14:53
*** ijw has quit IRC14:53
*** _mlavalle3 has quit IRC14:53
*** diablo_rojo has joined #openstack-meeting14:58
*** belmoreira has joined #openstack-meeting14:59
*** ijw has joined #openstack-meeting15:19
*** yasufum has joined #openstack-meeting15:21
*** mlavalle has joined #openstack-meeting15:22
*** ijw has quit IRC15:24
*** yasufum_ has joined #openstack-meeting15:25
*** yasufum has quit IRC15:26
*** yasufum_ is now known as yasufum15:26
*** Liang__ has quit IRC15:29
*** tosky has quit IRC15:31
*** armstrong has joined #openstack-meeting15:32
*** ijw has joined #openstack-meeting15:52
*** lpetrut has quit IRC15:53
*** belmoreira has quit IRC15:53
*** ijw has quit IRC15:57
*** ijw has joined #openstack-meeting16:00
*** ijw has quit IRC16:01
*** ijw has joined #openstack-meeting16:01
*** Lucas_Gray has quit IRC16:50
*** gibi has quit IRC16:57
*** psachin has quit IRC17:09
*** yasufum has quit IRC17:17
*** njohnston has joined #openstack-meeting17:31
*** ayoung has joined #openstack-meeting17:37
*** armstrong has quit IRC17:42
*** manuvakery has quit IRC18:05
*** yasufum has joined #openstack-meeting18:15
*** e0ne has quit IRC18:29
*** vishalmanchanda has quit IRC18:43
*** yasufum has quit IRC18:44
*** manpreet has quit IRC19:18
*** toabctl has quit IRC19:22
*** toabctl has joined #openstack-meeting19:24
*** jmasud has quit IRC19:26
*** e0ne has joined #openstack-meeting19:51
*** apetrich has quit IRC20:07
*** gyee has joined #openstack-meeting20:17
*** ijw has quit IRC20:34
*** ijw has joined #openstack-meeting20:35
*** Lucas_Gray has joined #openstack-meeting20:42
*** e0ne has quit IRC20:44
*** dustinc has joined #openstack-meeting20:58
timburke#startmeeting swift21:00
openstackMeeting started Wed Sep  2 21:00:33 2020 UTC and is due to finish in 60 minutes.  The chair is timburke. Information about MeetBot at
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.21:00
*** openstack changes topic to " (Meeting topic: swift)"21:00
openstackThe meeting name has been set to 'swift'21:00
timburkewho's here for the swift meeting?21:00
timburkei guess everyone else is lurking or late ;-)21:03
timburkeagenda's at
timburkefirst up, PTG planning!21:03
timburke#topic PTG21:03
*** openstack changes topic to "PTG (Meeting topic: swift)"21:03
*** jmasud has joined #openstack-meeting21:04
timburkei see clayg's been busy adding topics to the etherpad :D21:04
mattoliverautimburke: if they too late then we can decide everything ;)21:04
claygyup, gunna be a great PTG21:04
timburkeif you haven't already, please respond to the doodle i made to figure out meeting times that week21:05
claygmattoliverau & timburke BDsFL!21:05
timburkei just need to respond by next week, iirc21:06
mattoliverauclayg: thanks for adding the topic to etherpad21:06
timburkeany questions on ptg logistics?21:06
*** rcernin_ has joined #openstack-meeting21:06
claygI had no idea there was a doodle. I will put it in my list.21:07
timburkeall right, let's look at some patches then!21:07
timburke#topic waterfall ec21:08
*** openstack changes topic to "waterfall ec (Meeting topic: swift)"21:08
timburkethis is starting to become table stakes for a lot of the work clayg and i have ben doing lately21:08
timburkewe packaged and put it in prod recently21:09
mattoliverauHow's it going in prod?21:09
timburkethat said, it's a fairly involved change -- having an extra set of eyes take a look would likely be beneficial21:10
timburkemattoliverau, really well! dramatic improvement to ttfb metrics21:10
claygmattoliverau: GREAT!21:10
claygis there a paste bin but for graphs?21:10
mattoliverauIve been meaning to look, but been stuck with an upcoming product milestone at work. But will definitely try.21:11
*** rcernin_ has quit IRC21:12
claygoh acctually it's not that exciting cause we moved metric names around :\21:12
claygbut our ttfb 99.9% used to be like 5s - then it was ~500ms and with the extra ec request it's more like 30ms 💪21:13
claygthere's a ton of flexibility in there - no issues with regressions or 3R or anything like that - worked basically exactly how we wanted it to 🎉21:14
*** ijw has quit IRC21:14
*** ijw has joined #openstack-meeting21:14
timburkeno worries -- i get it -- do you think you'd be able to get a look in this week? i'm tempted to say we just merge it and address further reviews as new patches21:14
claygi'm pretty sure anyone running EC wants this change - as long as everyone understands "the gist" and can find the option name...21:15
claygpretty sure it ended up as "just better by default"21:15
*** jmasud has quit IRC21:15
timburkehelps that clayg did all the plumbing you'd need to make sure you *can* configure concurrent gets just for replicated if you wanted21:15
claygthe per-policy concurrency options are something to be aware of - but the defaults all fall through - there's no "upgrade impact" or anything21:15
timburkeUpgradeImpact: your EC GETs just got better ;-)21:16
mattoliverauThe fact that it's tested and run in prod helps. I'll make sure I give it some kind of review this week21:16
claygit's difficult to continue carrying it as patch just because we're doing other things to improve other stuff unrelated to waterfall ec - but the change moved a lot of code around21:16
claygbasically for us it's the new master - we're not merging anything else ahead of it - it's blocking a bunch of unrelated stuff (like Tim's trans-id fix)21:17
mattoliverauNow that v0.1 shard audit is out of my head I have more room :p21:17
claygok, one week doesn't make a big difference to me if you want to look at it before it merged I guess...21:18
claygit's just a headache to keep managing it off master - but i'm already going through that again at least one more time21:19
mattoliverauI'll have at least second high level look today and tomorrow.21:19
timburkeall right then21:20
timburke#topic worker management21:20
*** openstack changes topic to "worker management (Meeting topic: swift)"21:20
timburkeso after looking at how many connections per worker, i started playing with signaling workers to exit gracefully21:21
timburke...and wound up bringing back some of the interfaces i removed in the first patch (in particular, do_bind_ports)21:22
timburkei'm wondering if it'd be better/easier to review if i keep them as separate patches or squash them all together into a giant "manage workers better patch"21:23
timburkefor reference, improves connection distribution21:24
patchbotpatch 745603 - swift - Bind a new socket per-worker - 4 patch sets21:24
timburke lets workers drain connections in response to a HUP or USR121:24
patchbotpatch 747332 - swift - wsgi: Allow workers to gracefully exit - 3 patch sets21:24
timburkeand has the workers just stop listening (rather than close their socket) to make the cycling less disruptive (since any connected-but-not-yet-accepted clients should still get a response)21:25
patchbotpatch 748721 - swift - wsgi: stop closing listen sockets when workers die - 1 patch set21:25
claygtimburke: fwiw the "bind a new socket per-worker" patch is going to be much more important for us to get out ASAP than the HUP/USR1 handling for workers (the rss killer will have to learn to "try to kill gracefully, but still timeout and murder with recklessabandon" (i.e. we can't "just" change TERM to HUP)21:25
timburkei've also got another patch locally to make it so workers can *either* stop accepting *or* shutdown the listen socket, and then the parent opens fresh sockets as needed21:26
mattoliverauIf one is more important to get out then the other, then split em I say21:26
claygbut if you have some vision for how it all goes back together and want to refactor the bind a new socket per worker patch that's maybe ok?  I know you'd already tested the code as is - so I'd like to package that change w/o modification and we can keep making it better in the interum21:26
clayg... including squash everything to one if that's the way it makes the most sense21:27
mattoliverauThen we can do our best at getting the one you need in21:27
claygyeah I think the worker spread is going to be a big win for everyone (I also just happen to like some of the refactors that tim did) - the graceful worker is a nice-to-have21:28
claygtimburke: make respin everything squashed into a separate patch just for everyone to look at?21:28
timburkei mean, we've already got +2 on the first two...21:28
timburkeclayg, yeah, i can do that21:28
claygmaybe when it's in gerrit in green and red it'll be easier to see if/how it should be split21:28
claygyeah I'd be fine merging the "bind a new socket per-worker" - the only reason I think I held off was because you made some reference to wanting to squash or refactor something?21:29
claygbut it sounds like mattoliverau and I are both pushing back on that... should we just merge bind a new socket per-worker and let you work from there?21:29
claygi have most of that machinery in my head - if we want to keep noodling there I think I can keep up - i am interested in making worker killing better21:30
timburkefwiw, i ran into some weird deadlock running probe tests on my un-pushed work, where the client's sent the rest of its body and is waiting on a response, but the server is waiting on the client to send all the bytes...21:30
claygi miss rledisez :'(21:30
claygzohno!?  i blame eventlet21:30
timburkeidk -- i might just need to learn more about sockets ;-)21:31
timburketho come to think of it -- i think it may have been waiting down in eventlet's discard()...21:31
claygSEE!  eventlet 😡21:31
claygit's like "learn sockets" ok, now forget everyhting you knew about sockets 🤣21:32
claygexcept for the part you glanced over about non-blocking - THAT part you need to know really well 😁21:32
timburkeidk. i'll keep noodling with it -- but i'm starting to think we probably should go ahead and merge what's got +2s already21:33
timburke#topic storage policies, logging, and s3api21:34
*** openstack changes topic to "storage policies, logging, and s3api (Meeting topic: swift)"21:34
timburkeif anyone's looking for a quicker review, here's something a little smaller and more targetted ;-)21:34
timburkewe noticed that the normal {policy_index} field from doesn't get populated properly when using s3api21:35
timburkethe main problem was that proxy-logging relies on the x-backend-storage-policy-index header getting populated in the wsgi env, but s3api pretty frequently just makes up new environs21:38
timburkethe kind of funny thing is that proxy-logging looks at the response headers, too, but we don't actually populate them (on master; see
openstackLaunchpad bug 1634382 in OpenStack Object Storage (swift) "return x-backend-storage-policy-index header from object server" [Wishlist,Confirmed] - Assigned to Christian Hugo (christianhugo)21:40
timburke tried to address that in the object-server, but we found there was a problem during rolling upgrades21:41
patchbotpatch 693893 - swift - Put storage policy index in object-server responses - 1 patch set21:41
claygtimburke: I think maybe some of the container requests do populate that field - but yeah, not object21:41
timburkeso has the proxy add it locally shortly after determining the best response from backends21:42
patchbotpatch 749400 - swift - proxy: Put storage policy index in object responses - 3 patch sets21:42
timburke then just makes sure s3api passes it through (since it wouldn't previously)21:43
patchbotpatch 749401 - swift - s3api: Ensure backend headers make it through s3api - 3 patch sets21:43
timburkethat's all i had planned to talk about21:43
timburke#topic open discussion21:43
*** openstack changes topic to "open discussion (Meeting topic: swift)"21:43
timburkeanything else we should talk about today?21:44
claygtimburke: what's the deal with the s3api quota error patch?21:44
timburkeoh, yeah!21:44
claygdoes aws api have an over quota response?21:45
claygdoes it like suck or something?21:45
timburkenope! no concept of object quotas (afaik)21:45
timburkeit's like they prefer to just charge you more until you change your behavior or something ;-)21:46
timburkeso the first patchset was very targetted and tactical:
patchbotpatch 749382 - swift - s3api: Make quota-exceeded errors more obvious - 2 patch sets21:46
mattoliverauJust a random thought I had while cooking. Seems quic protocol has been decided to be http/3 I wonder if it'll speed up internal swift communication (if we trued to implement it there). Quic is http over UDP but still has some acks. Just made to go faster. Could be an interesting poc (if anyone ever has any time) :p21:46
timburkesecond patch took on a decent bit of refactoring; i'd be interested in kota_'s opinion21:48
timburkemattoliverau, for sure interesting -- i'd love to have the bandwidth to look at reworking things from the protocol up21:48
kota_timburke: will look. the p 749382 ?21:49
patchbot - swift - s3api: Make quota-exceeded errors more obvious - 2 patch sets21:49
timburkeyup -- thanks!21:49
*** armstrong has joined #openstack-meeting21:50
kota_oh, it's just a couple of lines changed...21:50
timburkekota_, is that the first patchset, maybe?21:51
kota_ah, not. it was the first.21:51
kota_timburke: correct. thx.21:51
claygtimburke: did you get what you needed on libec21:54
claygi know you said we're super down to the wire on needing to cut releases for upstream21:54
timburkei'm still a little torn on how to handle the env var in
patchbotpatch 738959 - liberasurecode - Be willing to write fragments with legacy crc - 2 patch sets21:55
timburkei don't really want to log anything on unexpected values -- that seems like it'd be *incredibly* noisy on a busy proxy :-/21:57
timburkei think there's still time for us to revisit. we've had this bug a couple years already; what's another cycle?21:59
timburkei *should* get client patches in order though :-(21:59
timburkei'll make sure the priority reviews page is updated for next week21:59
timburkeall right, we're at time22:01
timburkethank you all for coming, and thank you for working on swift!22:01
*** openstack changes topic to "OpenStack Meetings ||"22:01
openstackMeeting ended Wed Sep  2 22:01:23 2020 UTC.  Information about MeetBot at . (v 0.1.4)22:01
openstackMinutes (text):
*** patchbot has left #openstack-meeting22:03
*** rcernin_ has joined #openstack-meeting22:16
*** TrevorV has quit IRC22:18
*** raildo_ has joined #openstack-meeting22:30
*** raildo has quit IRC22:32
*** jmasud has joined #openstack-meeting22:41
*** jmasud has quit IRC22:59
*** ykatabam has joined #openstack-meeting23:00
*** rcernin_ has quit IRC23:04
*** rcernin has joined #openstack-meeting23:04
*** mlavalle has quit IRC23:07
*** Lucas_Gray has quit IRC23:07
*** dustinc has quit IRC23:08
*** jmasud has joined #openstack-meeting23:22
*** lifeless has quit IRC23:29
*** irclogbot_1 has quit IRC23:29
*** irclogbot_2 has joined #openstack-meeting23:33
*** whoami-rajat has quit IRC23:35
*** kopecmartin has quit IRC23:40
*** kopecmartin has joined #openstack-meeting23:40

Generated by 2.17.2 by Marius Gedminas - find it at!