Wednesday, 2020-08-12

*** armax has joined #openstack-meeting00:01
*** andrebeltrami has quit IRC00:09
*** armax has quit IRC00:18
*** yamamoto has joined #openstack-meeting00:29
*** yamamoto has quit IRC01:03
*** ayoung has quit IRC01:08
*** ayoung has joined #openstack-meeting01:11
*** Liang__ has joined #openstack-meeting01:17
*** yamamoto has joined #openstack-meeting01:33
*** gyee has quit IRC01:49
*** Liang__ has quit IRC01:54
*** ayoung has quit IRC02:32
*** ayoung has joined #openstack-meeting02:34
*** psahoo has joined #openstack-meeting03:07
*** rcernin has quit IRC03:10
*** rcernin has joined #openstack-meeting03:18
*** ayoung has quit IRC03:19
*** manpreet has joined #openstack-meeting03:22
*** ayoung has joined #openstack-meeting03:22
*** psachin has joined #openstack-meeting03:40
*** ayoung has quit IRC03:50
*** ayoung has joined #openstack-meeting03:53
*** ociuhandu has joined #openstack-meeting04:07
*** ociuhandu has quit IRC04:12
*** psahoo has quit IRC04:20
*** markvoelker has joined #openstack-meeting04:25
*** dklyle has quit IRC04:29
*** dklyle has joined #openstack-meeting04:29
*** markvoelker has quit IRC04:30
*** evrardjp has quit IRC04:33
*** psahoo has joined #openstack-meeting04:33
*** evrardjp has joined #openstack-meeting04:33
*** markvoelker has joined #openstack-meeting05:09
*** markvoelker has quit IRC05:21
*** manpreet has quit IRC05:31
*** sridharg has joined #openstack-meeting05:51
*** kaisers has quit IRC05:59
*** dmacpher_ has joined #openstack-meeting06:04
*** johanssone has quit IRC06:04
*** tobberydberg has quit IRC06:05
*** ricolin has quit IRC06:08
*** dmacpher has quit IRC06:08
*** johanssone has joined #openstack-meeting06:11
*** tobberydberg has joined #openstack-meeting06:11
*** kaisers has joined #openstack-meeting06:12
*** mahatic has quit IRC06:14
*** slaweq has joined #openstack-meeting06:28
*** dklyle has quit IRC06:46
*** ykatabam has quit IRC07:00
*** moguimar has joined #openstack-meeting07:03
*** maciejjozefczyk has joined #openstack-meeting07:04
*** rcernin has quit IRC07:07
*** ssbarnea has joined #openstack-meeting07:17
*** martial has quit IRC07:28
*** zbr has quit IRC07:29
*** ssbarnea has quit IRC07:29
*** zbr9 has joined #openstack-meeting07:29
*** tosky has joined #openstack-meeting07:41
*** e0ne has joined #openstack-meeting07:46
*** Lucas_Gray has joined #openstack-meeting07:51
*** markvoelker has joined #openstack-meeting07:55
*** markvoelker has quit IRC08:00
*** masahito has joined #openstack-meeting08:00
*** masahito has quit IRC08:06
*** masahito has joined #openstack-meeting08:06
*** zbr9 has quit IRC08:12
*** zbr has joined #openstack-meeting08:13
*** evrardjp has quit IRC08:14
*** evrardjp has joined #openstack-meeting08:21
*** evrardjp has quit IRC08:26
*** evrardjp has joined #openstack-meeting08:27
*** Lucas_Gray has quit IRC08:38
*** Lucas_Gray has joined #openstack-meeting08:42
*** masahito has quit IRC08:43
*** ociuhandu has joined #openstack-meeting08:51
*** ociuhandu has quit IRC08:56
*** ykatabam has joined #openstack-meeting09:06
*** ociuhandu has joined #openstack-meeting09:28
*** ykatabam has quit IRC09:41
*** psahoo has quit IRC09:47
*** psahoo has joined #openstack-meeting10:03
*** manpreet has joined #openstack-meeting10:13
*** Lucas_Gray has quit IRC10:16
*** ykatabam has joined #openstack-meeting10:18
*** Lucas_Gray has joined #openstack-meeting10:21
*** ykatabam has quit IRC10:25
*** evrardjp has quit IRC10:29
*** Lucas_Gray has quit IRC10:34
*** evrardjp has joined #openstack-meeting10:35
*** ociuhandu has quit IRC10:38
*** ociuhandu has joined #openstack-meeting10:39
*** yamamoto has quit IRC10:41
*** Lucas_Gray has joined #openstack-meeting10:45
*** lpetrut has joined #openstack-meeting10:45
*** evrardjp has quit IRC10:48
*** yamamoto has joined #openstack-meeting10:50
*** vishalmanchanda has joined #openstack-meeting10:57
*** ociuhandu_ has joined #openstack-meeting10:59
*** ociuhandu has quit IRC11:02
*** ociuhandu_ has quit IRC11:03
*** ykatabam has joined #openstack-meeting11:09
*** ykatabam has quit IRC11:09
*** ociuhandu has joined #openstack-meeting11:18
*** JJH42 has joined #openstack-meeting11:22
*** JJH49 has joined #openstack-meeting11:23
*** ociuhandu has quit IRC11:23
*** JJH42 has quit IRC11:26
*** JJH49 has quit IRC11:26
*** jheyd has joined #openstack-meeting11:31
jheydSorry, too late to the party ;-)11:32
*** raildo has joined #openstack-meeting11:37
*** yamamoto has quit IRC11:41
*** yamamoto has joined #openstack-meeting11:41
*** carloss has joined #openstack-meeting11:42
*** yamamoto has quit IRC11:42
*** yamamoto has joined #openstack-meeting11:57
*** Lucas_Gray has quit IRC12:17
*** yamamoto has quit IRC12:28
*** masahito has joined #openstack-meeting12:35
*** ianychoi has quit IRC12:40
*** e0ne_ has joined #openstack-meeting12:45
*** masahito has quit IRC12:48
*** e0ne has quit IRC12:49
*** psahoo has quit IRC12:50
*** Lucas_Gray has joined #openstack-meeting12:51
*** masahito has joined #openstack-meeting12:52
*** ociuhandu has joined #openstack-meeting12:56
*** masahito has quit IRC12:57
*** dulek has quit IRC13:01
*** psahoo has joined #openstack-meeting13:02
*** yamamoto has joined #openstack-meeting13:07
*** ZhuXiaoYu has joined #openstack-meeting13:17
*** yamamoto has quit IRC13:18
*** ricolin has joined #openstack-meeting13:22
*** TrevorV has joined #openstack-meeting13:23
*** hongbin has joined #openstack-meeting13:40
*** ianychoi has joined #openstack-meeting13:44
*** jheyd has quit IRC13:49
*** liuyulong has joined #openstack-meeting13:55
*** eharney has quit IRC13:59
liuyulong#startmeeting neutron_l314:00
openstackMeeting started Wed Aug 12 14:00:19 2020 UTC and is due to finish in 60 minutes.  The chair is liuyulong. Information about MeetBot at
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.14:00
*** openstack changes topic to " (Meeting topic: neutron_l3)"14:00
openstackThe meeting name has been set to 'neutron_l3'14:00
haleybhi, and just an fyi that i'm in a conflicting meeting but will notice a ping with my nick14:01
liuyulongToday we will be a short meeting.14:01
liuyulongI have no announcements and only one L3 bug.14:01
*** eharney has joined #openstack-meeting14:01
liuyulongOK, let's directly starts the Bugs topic.14:03
liuyulong#topic Bugs14:03
*** openstack changes topic to "Bugs (Meeting topic: neutron_l3)"14:03
liuyulongFirst one14:04
openstackLaunchpad bug 1889388 in neutron "[OVN] access between Floatings ip and instance with Direct External IP" [Undecided,New]14:04
liuyulongIt is taged as OVN backend, but I have never try to run instance in such case for OVS backend.14:05
liuyulongIt's a corner case, I will check it in our OVS deployment.14:06
slaweqliuyulong: hi14:07
slaweqI have some urgent d/s issue so I will just be lurking here14:07
haleyband i'm assuming there's a SG rule for access?  there was no answer to rodolfo's question either14:07
liuyulongslaweq, hi14:07
*** yamamoto has joined #openstack-meeting14:07
liuyulongslaweq, sure, go ahead, : )14:08
liuyulonghaleyb, IMO whether in the same host should not be the problem, the instance can be created in different hosts.14:09
liuyulongFor OVN, I guess the data pipeline may lack of some metadata/tag/header for packets between this two VMs.14:10
haleybwell, sometimes there are specific issues when both are on the same host, hairpin issue for example14:10
*** Liang__ has joined #openstack-meeting14:11
liuyulongFor OVS, this may do not work either since there are also some block flows in the physical bridges.14:11
haleybright, someone would have to reproduce it14:12
liuyulongBesides, amotoki asked one question about the tag "l3-dvr-backlog".14:13
liuyulong"l3-dvr-backlog tag was originally introduced to identify DVR feature14:14
liuyulonggaps. What should we use for OVN L3 feature gaps?"14:14
*** yamamoto has quit IRC14:14
*** hongbin has quit IRC14:14
haleybi think we have just an 'ovn' tag14:14
haleybwe could add a new one for the gaps14:15
liuyulongI don't know the tag history, but IMO, a one shot tag may same filter works.14:15
haleybi know we added the ovn one when we merged the code14:16
liuyulongOK, thank you for the information.14:17
openstackLaunchpad bug 1890400 in neutron "Default gateway in HA router namespace not set if using Keepalived 1.x" [High,In progress] - Assigned to Slawek Kaplonski (slaweq)14:17
liuyulongslaweq has uploded the fix:
liuyulongA simple config option, it's fine to me.14:18
liuyulongUsers should be informed by the release note.14:19
*** Liang__ has quit IRC14:19
haleybyes, it's not perfect, but we have no way of specifying a keepalived version afaik14:20
liuyulongyes, that's the main point.14:20
liuyulongAlright, no more L3 bugs from me now.14:21
liuyulongLet's continue to discuss something in next section.14:21
liuyulong#topic On demand agenda14:22
*** openstack changes topic to "On demand agenda (Meeting topic: neutron_l3)"14:22
liuyulongI noticed that slaweq uploaded a new patch set for this one.14:22
*** armax has joined #openstack-meeting14:24
liuyulongBut no update for the comments of patch set 42, if this will be carried on again, we should fix those.14:24
liuyulongNext should be this:14:25
*** Liang__ has joined #openstack-meeting14:25
liuyulongFloating IP's for routed networks14:25
slaweqliuyulong: yes, for now I only repaced it and resolved conflicts14:26
slaweqnothing else14:26
liuyulongslaweq, cool14:27
liuyulongFor this routed network stuffs, it is perrty good now, we can move forward IMO.14:27
*** e0ne has joined #openstack-meeting14:28
*** e0ne_ has quit IRC14:29
liuyulongOK, no more things from me now.14:29
liuyulongI will hold this section a few minutes open, feel free to start your topic.14:30
haleybnothing else from me14:32
slaweqnothing from me neighter14:36
*** liuyulong has quit IRC14:37
*** hongbin has joined #openstack-meeting14:39
*** mlavalle has joined #openstack-meeting14:49
*** liuyulong has joined #openstack-meeting14:49
liuyulongSorry lost the connection14:49
*** openstack changes topic to "OpenStack Meetings ||"14:50
openstackMeeting ended Wed Aug 12 14:50:11 2020 UTC.  Information about MeetBot at . (v 0.1.4)14:50
openstackMinutes (text):
*** e0ne_ has joined #openstack-meeting14:56
*** liuyulong has quit IRC14:57
*** e0ne has quit IRC14:59
*** priteau has joined #openstack-meeting14:59
*** dklyle has joined #openstack-meeting15:05
*** Liang__ has quit IRC15:07
*** ykatabam has joined #openstack-meeting15:33
*** ykatabam has quit IRC15:38
*** lpetrut has quit IRC15:58
*** psahoo has quit IRC16:12
*** yamamoto has joined #openstack-meeting16:13
*** ayoung has quit IRC16:22
*** yamamoto has quit IRC16:23
*** ayoung has joined #openstack-meeting16:24
*** psahoo has joined #openstack-meeting16:25
*** maciejjozefczyk has quit IRC16:28
*** ayoung has quit IRC16:31
*** ayoung has joined #openstack-meeting16:32
*** Lucas_Gray has quit IRC16:36
*** ayoung has quit IRC16:38
*** psahoo has quit IRC16:39
*** psachin has quit IRC16:39
*** ayoung has joined #openstack-meeting16:40
*** hongbin has quit IRC16:43
*** hongbin has joined #openstack-meeting17:00
*** sridharg has quit IRC17:00
*** ayoung has quit IRC17:07
*** ayoung has joined #openstack-meeting17:08
*** ayoung has quit IRC17:22
*** ayoung has joined #openstack-meeting17:23
*** priteau has quit IRC17:27
*** ayoung has quit IRC17:31
*** ayoung has joined #openstack-meeting17:33
*** ZhuXiaoYu has quit IRC17:36
*** e0ne_ has quit IRC17:41
*** e0ne has joined #openstack-meeting17:42
*** e0ne has quit IRC17:44
*** e0ne has joined #openstack-meeting17:49
*** maciejjozefczyk has joined #openstack-meeting17:52
*** gyee has joined #openstack-meeting17:56
*** hyunsikyang__ has joined #openstack-meeting18:11
*** knikolla_ has joined #openstack-meeting18:13
*** vkmc_ has joined #openstack-meeting18:13
*** hyunsikyang has quit IRC18:20
*** darvon has quit IRC18:20
*** gouthamr has quit IRC18:20
*** knikolla has quit IRC18:20
*** vkmc has quit IRC18:20
*** knikolla_ is now known as knikolla18:20
*** vkmc_ is now known as vkmc18:20
*** gouthamr has joined #openstack-meeting18:22
*** manpreet has quit IRC18:23
*** e0ne has quit IRC18:23
*** yamamoto has joined #openstack-meeting18:23
*** ricolin has quit IRC18:23
*** e0ne has joined #openstack-meeting18:29
*** yamamoto has quit IRC18:32
*** Lucas_Gray has joined #openstack-meeting18:50
*** jmasud_ has joined #openstack-meeting18:59
*** jmasud_ has quit IRC19:00
*** Lucas_Gray has quit IRC19:00
*** hongbin has quit IRC19:26
*** maciejjozefczyk has quit IRC19:39
*** hongbin has joined #openstack-meeting19:44
*** e0ne has quit IRC19:49
*** mattia has quit IRC19:59
*** TrevorV has quit IRC20:24
*** yamamoto has joined #openstack-meeting20:32
*** yamamoto has quit IRC20:39
*** e0ne has joined #openstack-meeting20:40
*** Lucas_Gray has joined #openstack-meeting20:52
*** rh-jelabarre has quit IRC20:52
*** rh-jelabarre has joined #openstack-meeting20:52
*** ayoung has quit IRC20:57
timburke#startmeeting swift21:00
openstackMeeting started Wed Aug 12 21:00:40 2020 UTC and is due to finish in 60 minutes.  The chair is timburke. Information about MeetBot at
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.21:00
*** openstack changes topic to " (Meeting topic: swift)"21:00
openstackThe meeting name has been set to 'swift'21:00
timburkewho's here for the swift meeting?21:00
*** patchbot has joined #openstack-meeting21:00
*** ayoung has joined #openstack-meeting21:02
*** dklyle has quit IRC21:02
timburkeagenda's at
timburke#topic summit and ptg21:03
*** openstack changes topic to "summit and ptg (Meeting topic: swift)"21:03
timburkelooks like the dates are set for the ptg21:03
timburkeit will immediately follow the summit21:03
timburkeso, summit will be oct 19-23, then ptg oct 26-3021:04
claygit's all virtual like last time?  that seemed to work ok...21:04
timburkeregistration's already open, and both virtual events are free21:04
kota_good info. thx.21:05
timburkei'll keep an eye out for what we need to do to sign up for space and such, and try to be more on it than last time21:05
claygtimburke: or just let mattoliverau do it again - things worked great!?21:05
timburkethough mattoliverau did a great job getting everything organized!21:05
timburkethat's all i had by way of announcements; any questions or comments?21:06
timburkeall right21:07
claygtimburke: great update!  especially the part about mattoliverau being awesome 👍21:07
timburke#topic s3api subrequest logging21:08
*** openstack changes topic to "s3api subrequest logging (Meeting topic: swift)"21:08
timburkeclayg, i think you added this, yeah?21:08
claygoh no 😬21:08
claygdid I add a patch?21:08
timburketwo of them!21:08
timburkeand now one's landed :-)21:08
claygwhy is zuul so mad at p 735221 😡21:08
patchbot - swift - s3api: Use swift.backend_path to proxy-log s3api r... - 2 patch sets21:08
timburkeer, approved -- i'll keep poking at the gate21:09
claygso if we CAN get that one landed I'm pretty much fine with p 735220 - in fact I'd like to do some follow-on to update the example configs21:09
patchbot - swift - proxy-logging: Be able to configure log_route - 3 patch sets21:09
claygrledisez: do you run with two proxy-logging middlewares in your pipeline?  we got SUPER confused having statsd metrics for client facing responses mixed in with subrequests21:10
claygwe're splitting everything up right now - internal_client.conf is going to get it's own metrics namespace too - i'll let you know how it works out; but we expect it to be pretty great21:11
rledisezclayg: yes, we use the 2 proxy-logging instance21:11
rledisezclayg: I though statsd metrics were emited by the left instance of proxy-logging, which is not in the path of subrequests21:12
rledisezI guess I assumed wrong :)21:13
claygtimburke: ok, I think i got what I need - the first one will merge, no one is going have much input on "fixing" prefix if you want it...21:13
claygso if there's any dicussion to have around "best practices" it should happen on the follow-on patch to update example configs21:13
timburkethey'll *both* emit statsd metrics21:13
claygrledisez: so we fixed something that made s3 requests parse a, c, o and NOW they emit metrics21:13
claygi mean they always both emitted... it's a disastor21:14
timburkebut you're probably already getting doubled-up metrics from things like SLO21:14
clayg^ 👍21:14
clayglike transfer stats... the extra response codes and GET timings don't quite look so weird...21:15
claygwell - depending on what you think they mean21:15
*** e0ne has quit IRC21:15
claygcause it wasn't too hard for us to get confused - so we're gunna break it up!21:15
timburkeall right, sounds like we know what's happening with the two linked patches, and we'll expect more discussion to happen on a follow-up (and possibly at the ptg)21:17
timburke#topic py3 encryption bug and upgrades21:18
*** openstack changes topic to "py3 encryption bug and upgrades (Meeting topic: swift)"21:18
patchbotpatch 742033 - swift - py3: Work with proper native string paths in crypt... - 4 patch sets21:18
timburkeso that patch now has the config option from p 742756 squashed-in21:18
patchbot - swift - crypto: Add config option to support rolling upgrades (ABANDONED) - 3 patch sets21:18
*** Lucas_Gray has quit IRC21:18
timburkei think the UpgradeImpact has reasonably useful/actionable information21:19
timburkei've still got some concerns about an effective checkpoint release for switching from py2 to py3, and what that'll mean once the fix gets backported to train and ussuri21:20
timburkebut maybe those fears are misplaced?21:21
claygtimburke: this one is definitely on my list, i'm onboard with the direction - if anyone wants to merge it for me it'd be a big help!21:21
timburkei was just about to ask who could review, so i could get the backport ball rolling :-)21:21
claygtimburke: oh right; you left a comment about that?21:22
claygnm, I don't see it - so I don't think I understand your concern21:23
claygdo you want to add it to the review; or what to see if I notice the same thing when reviewing?21:23
*** rh-jelabarre has quit IRC21:23
timburkejust in general, i'm worried about the added complexity for operators in needing to upgrade swift while still on py2, then upgrade to py3. if someone's following ubuntu's repos for example, i'm not sure they're gonna have a good time21:24
claygagain; if anyone else besides timburke has this loaded in their head (seongsoocho?) i'd love some help shaking out anything else needed21:24
timburkewe might want to divvy up upgrade testing, too, to make sure we're covering all the cases we care about21:25
claygtimburke: so the issue is for py3 rollings to go smoothly py3 by default writes v2 broken?  so py2 reads will fail - unless everyone is on v3 before upgrading.21:25
claygtimburke: i only really have bandwidth enough to functionally test for py2 upgrade case - if someone wants to do more than that BEFORE we +A I'd be happy to hold off21:27
timburkeyup. and there's no way (currently; i suppose we could add yet another option...) to configure new py3 to write something that old py2 knows how to read21:27
claygdo we have a community deployment that's waiting on this fix?  or everyone is happily in the python version island with v2 and working fine?21:27
timburkeormandj first reported the bug; i think he's running on a patched ussuri to pick up the fix21:28
claygtimburke: ok so realistically you can't upgrade from a current version of swift running py2 to a newer version of swift running py3 even with this patch... so... what are we doing?  🤣21:29
timburkechasing my tail, i think21:29
claygfor US writing v3 is a blocker to migration onto py3 - so we want this patch, the rolling-upgrade behavior is seemless and all we have to do is flip to v3 asap so when we finally upgrade to py3 it's a non issue21:30
claygand people already on py3 are happy21:31
claygthey also have a similar; easy upgrade; switch config afterwards story21:31
claygok, so I think we'll just have to say "When switching from Python 2 to Python 3, first upgrade Swift while on Python 2, then upgrade to Python 3." is reasonable and also the best we can do.21:32
timburkecool -- so i probably *was* overly concerned and we just need to review it, land it, and backport it21:32
timburkei think i've got what i need then. thanks for offering to review, clayg! again, if anyone else has a chance to review/test it, that'd be much appreciated21:33
timburke#topic shrinking and overlapping shard ranges21:33
*** openstack changes topic to "shrinking and overlapping shard ranges (Meeting topic: swift)"21:33
claygthanks to the awesome mattoliverau for spotting bugs (i haven't written the unittest yet; pls help)21:34
timburkeso when i originally added this, i wanted to make sure we had some consensus about which path forward we ought to take21:34
timburkebut i think it's settled pretty well onto clayg's p 74172121:34
patchbot - swift - add swift-manage-shard-ranges shink command - 4 patch sets21:34
timburkeand i'm going to continue working with getting a probe test to exercise it in p 74425621:35
patchbot - swift - sharding: probe test to exercise manual shrinking - 1 patch set21:35
timburke(mostly extracted from p 738149)21:35
patchbot - swift - Have shrinking and sharded shards save all ranges ... - 5 patch sets21:35
mattoliverauthanks guys for looking into it, and breaking things, be it shrinking in general or related to the manual patch, it's good to start excersizing the shrinking code.21:36
clayg👍 gunna be so great21:37
timburkei'm not actually sure we need much more discussion, just more review bandwidth :-D21:37
*** dklyle has joined #openstack-meeting21:37
claygmattoliverau: did revewaz; mattoliverau is awesome; be like mattoliverau21:38
*** slaweq has quit IRC21:38
timburkeso, just one more last-minute topic21:38
timburke#topic libec upgrades21:38
*** openstack changes topic to "libec upgrades (Meeting topic: swift)"21:38
claygalso mattoliverau - do more reviews 😁21:38
mattoliverauI might try and progress the shard range audit stuff to fix gaps, split brains and overlaps so we can more easily recover from these kinda issues. (even if it starts of manual).21:38
mattoliveraulol, I'll find more reviewing time :)21:38
timburkezaitcev's taken a look already, but i think it's probably worth a second set of eyes21:38
timburkep 738959 and p 73916421:39
patchbot - liberasurecode - Be willing to write fragments with legacy crc - 2 patch sets21:39
patchbot - swift - ec: Add an option to write fragments with legacy crc - 1 patch set21:39
claygmattoliverau: I love that - if we can get where we automatically and confidently fix overlapping shard ranges it's gunna be WAY easier to start electing leaders to automatically shard 😍21:39
timburkeheck, we might even get to the point where we say, yeah, replica 0's probably good enough :-)21:40
claygi'm still scared of this patch; kota_ has always stronger at libec that I've been - for me ideally this can merge without me thinking about it anymore21:40
claygit's supposed to be a similar "upgrade nothing changes; then flip the switch" change?21:40
kota_i may take time to review on ec patch.21:41
claygonly caveat is you also have to repackage a new libec/pyeclib21:41
kota_is that ok if I will do that in this week?21:41
timburkekota_, that'd be great, thanks!21:41
claygkota_: I don't have any timeline21:41
kota_ok. will try it.21:42
timburkeclayg, no pyeclib changes necessary, given the current state of the patch21:42
claygagree; if we have update/progress to show by next meeting that's a huge win21:42
claygtimburke: i'm not sure WE can release a new libec w/o a new pyeclib - but that could easily just be our fault 🤷21:42
claygand I could be wrong - i haven't looked at the dependency chain21:42
claygthey may already be independent/orthogonal21:43
timburkethere is a bit of a detection problem for upgrades: you *may* need to tell new swift to write legacy fragments... or that may be the exact wrong thing to do :-(21:43
claygthe default isn't rolling-upgrade compatible?21:43
timburke... it is so long as you aren't touching libec...21:44
* clayg tries again to not think about this anymore21:44
timburkethe trouble is, we don't know what the upgrade-compatible behavior will be. there's a chance (depending on library load order) that you were already writing zlib crcs21:45
claygoh RIGHT!  🤮21:45
timburkefwiw, i wrote up p 744078 to try to help operators figure out what the right thing to do is (and to try out the audit-watchers interface)21:46
claygtimburke: you go LOOKING for this bugs 😡 you have a nose for trouble21:46
patchbot - swift - watchers: Add EC stat gatherer - 1 patch set21:46
claygexcept we don't have audit watchers 🤷‍♂️21:47
timburkewell, it's all based on the tool i added to, so you could still just use that :P21:48
openstackLaunchpad bug 1886088 in OpenStack Object Storage (swift) "Mixed versions of liberasurecode leads to quarantined fragments" [Undecided,In progress]21:48
*** ykatabam has joined #openstack-meeting21:48
clayg👍 and it sounds like "figuring out which version you're on" needs to happen before you upgrade so you can set the default for the new swift version21:49
timburkeall right, those are the major things i wanted to cover21:49
timburke#topic open discussion21:49
*** openstack changes topic to "open discussion (Meeting topic: swift)"21:49
timburkewhat else should we discuss?21:49
*** yamamoto has joined #openstack-meeting21:49
claygFYI: we're going to push waterfall-ec into the pipeline so we can do more testing in staging and production21:52
claygthanks for everyone's help with the design; i'm happy with where it ended up21:52
claygtimburke: did a bunch of cleanup I was able to squash in and mattoliverau found a few more splinters21:52
timburkeit's gonna be so great21:52
claygbut ultimately I think the results look good and the code is going to be fine21:52
mattoliverauthanks for all your hard work on it clayg!21:53
claygwe'll probably start to push to get some of it landed and I think that's also going to be fine - I hope folks try it out after their next upgrade!21:53
claygtimburke: do you have the socket/accept/worker patch handy?21:53
timburkenot sure if anyone else has seen this, but we noticed some lumpy request distribution between proxy-server workers -- resolution seems to be to bind a listen socket per-worker and let the kernel do a better job distributing work21:54
timburkei was just about to mention it!21:54
patchbotpatch 745603 - swift - Bind a new socket per-worker - 3 patch sets21:54
rlediseztimburke: yes, we observed that. it especially important with EC as the process gets CPU bound21:54
clayg^^^ so much this21:54
*** yamamoto has quit IRC21:54
*** vishalmanchanda has quit IRC21:55
timburkei still haven't gotten around to testing it not-in-my-dev-VM, though -- debating about applying it on just one machine in prod to see that the distribution gets fixed21:56
*** raildo has quit IRC21:57
timburkeall right, we're about out of time21:58
timburkethank you all for coming, and thank you for working on swift!21:58
*** openstack changes topic to "OpenStack Meetings ||"21:59
openstackMeeting ended Wed Aug 12 21:59:02 2020 UTC.  Information about MeetBot at . (v 0.1.4)21:59
openstackMinutes (text):
*** kopecmartin is now known as kopecmartin|pto22:16
*** markvoelker has joined #openstack-meeting22:30
*** markvoelker has quit IRC22:35
*** rcernin has joined #openstack-meeting22:43
*** haleyb has quit IRC22:55
*** haleyb has joined #openstack-meeting22:55
*** rfolco has quit IRC23:04
*** moguimar has quit IRC23:05
*** carloss has quit IRC23:11
*** tosky has quit IRC23:13
*** ayoung has quit IRC23:17
*** ayoung has joined #openstack-meeting23:18
*** apetrich has quit IRC23:55

Generated by 2.17.2 by Marius Gedminas - find it at!