Wednesday, 2020-01-08

*** d34dh0r53 has quit IRC00:16
*** d34dh0r53 has joined #openstack-meeting00:25
*** d34dh0r53 has quit IRC00:25
*** d34dh0r53 has joined #openstack-meeting00:28
*** d34dh0r53 has quit IRC00:30
*** martial has quit IRC00:31
*** maohongbo has quit IRC00:34
*** d34dh0r53 has joined #openstack-meeting00:35
*** d34dh0r53 has quit IRC00:36
*** d34dh0r53 has joined #openstack-meeting00:39
*** d34dh0r53 has quit IRC00:39
*** d34dh0r53 has joined #openstack-meeting00:43
*** d34dh0r53 has quit IRC00:44
*** d34dh0r53 has joined #openstack-meeting00:52
*** d34dh0r53 has quit IRC00:53
*** ricolin_ has joined #openstack-meeting00:57
*** ricolin_ has quit IRC00:58
*** d34dh0r53 has joined #openstack-meeting00:59
*** d34dh0r53 has quit IRC00:59
*** d34dh0r53 has joined #openstack-meeting01:01
*** d34dh0r53 has quit IRC01:04
*** d34dh0r53 has joined #openstack-meeting01:14
*** d34dh0r53 has quit IRC01:14
*** liuyulong has quit IRC01:23
*** d34dh0r53 has joined #openstack-meeting01:28
*** rfolcOUT has quit IRC01:28
*** d34dh0r53 has quit IRC01:28
*** d34dh0r53 has joined #openstack-meeting01:33
*** d34dh0r53 has quit IRC01:34
*** d34dh0r53 has joined #openstack-meeting01:37
*** d34dh0r53 has quit IRC01:38
*** d34dh0r53 has joined #openstack-meeting01:41
*** d34dh0r53 has quit IRC01:42
*** dtrainor has quit IRC01:46
*** maohongbo has joined #openstack-meeting01:46
*** d34dh0r53 has joined #openstack-meeting01:52
*** d34dh0r53 has quit IRC01:52
*** d34dh0r53 has joined #openstack-meeting01:56
*** d34dh0r53 has quit IRC01:57
*** d34dh0r53 has joined #openstack-meeting02:02
*** d34dh0r53 has quit IRC02:03
*** d34dh0r53 has joined #openstack-meeting02:05
*** d34dh0r53 has quit IRC02:05
*** d34dh0r53 has joined #openstack-meeting02:09
*** d34dh0r53 has quit IRC02:10
*** d34dh0r53 has joined #openstack-meeting02:19
*** d34dh0r53 has quit IRC02:19
*** Lucas_Gray has quit IRC02:21
*** d34dh0r53 has joined #openstack-meeting02:21
*** d34dh0r53 has quit IRC02:22
*** d34dh0r53 has joined #openstack-meeting02:38
*** d34dh0r53 has quit IRC02:39
*** mmethot has quit IRC02:43
*** d34dh0r53 has joined #openstack-meeting02:45
*** d34dh0r53 has quit IRC02:45
*** maohongbo has quit IRC02:45
*** maohongbo has joined #openstack-meeting02:47
*** rfolcOUT has joined #openstack-meeting02:48
*** maohongbo1 has joined #openstack-meeting03:04
*** maohongbo has quit IRC03:05
*** maohongbo1 is now known as maohongbo03:05
*** d34dh0r53 has joined #openstack-meeting03:09
*** d34dh0r53 has quit IRC03:09
*** ykatabam has quit IRC03:14
*** d34dh0r53 has joined #openstack-meeting03:19
*** d34dh0r53 has quit IRC03:19
*** maohongbo has quit IRC03:22
*** maohongbo has joined #openstack-meeting03:23
*** d34dh0r53 has joined #openstack-meeting03:28
*** d34dh0r53 has quit IRC03:28
*** ykatabam has joined #openstack-meeting03:31
*** psachin has joined #openstack-meeting03:33
*** mmethot has joined #openstack-meeting03:34
*** d34dh0r53 has joined #openstack-meeting03:41
*** d34dh0r53 has quit IRC03:41
*** d34dh0r53 has joined #openstack-meeting03:48
*** maohongbo1 has joined #openstack-meeting04:03
*** maohongbo has quit IRC04:04
*** maohongbo1 is now known as maohongbo04:04
*** d34dh0r53 has quit IRC04:40
*** d34dh0r53 has joined #openstack-meeting04:43
*** d34dh0r53 has quit IRC04:44
*** d34dh0r53 has joined #openstack-meeting04:45
*** d34dh0r53 has quit IRC04:45
*** d34dh0r53 has joined #openstack-meeting04:47
*** d34dh0r53 has quit IRC04:48
*** whoami-rajat__ has joined #openstack-meeting04:52
*** d34dh0r53 has joined #openstack-meeting04:53
*** d34dh0r53 has quit IRC04:53
*** d34dh0r53 has joined #openstack-meeting04:54
*** d34dh0r53 has quit IRC04:56
*** gyee has quit IRC04:57
*** d34dh0r53 has joined #openstack-meeting05:01
*** links has joined #openstack-meeting05:15
*** tetsuro has joined #openstack-meeting05:41
*** yaawang has quit IRC06:12
*** yaawang has joined #openstack-meeting06:12
*** hyunsikyang__ has joined #openstack-meeting06:22
*** hyunsikyang has quit IRC06:26
*** jawad_axd has joined #openstack-meeting06:49
*** jawad_ax_ has joined #openstack-meeting06:52
*** jawad_axd has quit IRC06:54
*** pcaruana has joined #openstack-meeting07:38
*** simon-AS559 has joined #openstack-meeting07:38
*** ociuhandu has joined #openstack-meeting07:46
*** ociuhandu has quit IRC07:58
*** e0ne has joined #openstack-meeting07:59
*** tesseract has joined #openstack-meeting08:06
*** e0ne has quit IRC08:08
*** e0ne has joined #openstack-meeting08:09
*** mahatic has quit IRC08:09
*** e0ne has quit IRC08:23
*** simon-AS5591 has joined #openstack-meeting08:25
*** simon-AS559 has quit IRC08:28
*** apetrich has joined #openstack-meeting08:31
*** e0ne has joined #openstack-meeting08:33
*** rsimai has joined #openstack-meeting08:40
*** maohongbo has quit IRC08:42
*** dmacpher has joined #openstack-meeting08:43
*** maohongbo has joined #openstack-meeting08:43
*** e0ne has quit IRC08:46
*** brinzhang_ has quit IRC08:46
*** dmacpher_ has joined #openstack-meeting08:47
*** dmacpher has quit IRC08:51
*** ralonsoh has joined #openstack-meeting08:52
*** jraju__ has joined #openstack-meeting09:03
*** links has quit IRC09:04
*** diablo_rojo has joined #openstack-meeting09:07
*** simon-AS559 has joined #openstack-meeting09:08
*** simon-AS5591 has quit IRC09:11
*** brinzhang has joined #openstack-meeting09:12
*** brinzhang_ has joined #openstack-meeting09:14
*** brinzhang_ has quit IRC09:16
*** brinzhang_ has joined #openstack-meeting09:16
*** brinzhang has quit IRC09:18
*** rpittau|afk is now known as rpittau09:25
*** brinzhang has joined #openstack-meeting09:31
*** brinzhang_ has quit IRC09:34
*** brinzhang has quit IRC09:36
*** maohongbo1 has joined #openstack-meeting09:38
*** maohongbo has quit IRC09:40
*** maohongbo1 is now known as maohongbo09:40
*** e0ne has joined #openstack-meeting09:50
*** simon-AS559 has quit IRC10:01
*** ociuhandu has joined #openstack-meeting10:02
*** simon-AS559 has joined #openstack-meeting10:05
*** ociuhandu has quit IRC10:19
*** links has joined #openstack-meeting10:22
*** jraju__ has quit IRC10:23
*** geguileo has joined #openstack-meeting10:28
*** e0ne has quit IRC10:29
*** ykatabam has quit IRC10:44
*** dmacpher__ has joined #openstack-meeting10:46
*** dmacpher_ has quit IRC10:48
*** ykatabam has joined #openstack-meeting10:49
*** e0ne has joined #openstack-meeting10:57
*** ociuhandu has joined #openstack-meeting10:59
*** ociuhandu has quit IRC11:18
*** ociuhandu has joined #openstack-meeting11:29
*** ociuhandu has quit IRC11:34
*** vishakha has quit IRC11:35
*** diablo_rojo has quit IRC11:38
*** simon-AS559 has quit IRC11:45
*** simon-AS559 has joined #openstack-meeting11:46
*** rfolcOUT is now known as rfolco12:07
*** dviroel has joined #openstack-meeting12:07
*** jawad_ax_ has quit IRC12:38
*** whoami-rajat__ has quit IRC12:45
*** liuyulong has joined #openstack-meeting12:52
*** enriquetaso has joined #openstack-meeting12:54
*** jawad_axd has joined #openstack-meeting12:55
*** liuyulong has joined #openstack-meeting12:55
*** liuyulong has quit IRC12:56
*** anastzhyr has joined #openstack-meeting12:56
*** liuyulong has joined #openstack-meeting12:58
*** jawad_ax_ has joined #openstack-meeting12:59
*** liuyulong has quit IRC12:59
*** jawad_axd has quit IRC12:59
*** jawad_ax_ has quit IRC13:03
*** jawad_axd has joined #openstack-meeting13:04
*** liuyulong has joined #openstack-meeting13:08
*** liuyulong_ has joined #openstack-meeting13:08
*** jawad_axd has quit IRC13:09
*** ociuhandu has joined #openstack-meeting13:09
*** jawad_axd has joined #openstack-meeting13:09
*** jawad_ax_ has joined #openstack-meeting13:12
*** jawad_axd has quit IRC13:14
*** jawad_axd has joined #openstack-meeting13:16
*** jawad_ax_ has quit IRC13:16
*** ociuhandu has quit IRC13:17
*** jawad_axd has quit IRC13:21
*** whoami-rajat__ has joined #openstack-meeting13:27
*** ociuhandu has joined #openstack-meeting13:38
*** ociuhandu has quit IRC13:42
*** jawad_axd has joined #openstack-meeting13:45
*** dmacpher__ has quit IRC13:45
*** dmacpher__ has joined #openstack-meeting13:46
*** jawad_axd has quit IRC13:49
*** Liang__ has joined #openstack-meeting13:53
*** jawad_axd has joined #openstack-meeting13:56
*** Liang__ is now known as LiangFang13:58
*** jawad_axd has quit IRC14:00
liuyulong#startmeeting neutron_l314:01
openstackMeeting started Wed Jan  8 14:01:12 2020 UTC and is due to finish in 60 minutes.  The chair is liuyulong. Information about MeetBot at
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.14:01
*** openstack changes topic to " (Meeting topic: neutron_l3)"14:01
openstackThe meeting name has been set to 'neutron_l3'14:01
liuyulong#chair liuyulong_14:01
openstackCurrent chairs: liuyulong liuyulong_14:01
liuyulongHappy new year everyone!14:02
*** thgcorrea has joined #openstack-meeting14:02
*** andrebeltrami has joined #openstack-meeting14:02
*** sfernand has joined #openstack-meeting14:02
liuyulong#topic Announcements14:03
*** openstack changes topic to "Announcements (Meeting topic: neutron_l3)"14:03
liuyulongExpected: 2020-02-1214:04
liuyulongThere will be about 10 days holidays for Chinese New Year this month.14:05
liuyulongSomeone may not online, so time is running out...14:06
openstackLaunchpad bug 1858419 in neutron "Docs needed for tunables at large scale" [Undecided,Confirmed]14:08
liuyulongSlawek asked me something in mail about this large scale cloud.14:08
liuyulongAllow me to say something here14:09
liuyulongThis could be a really long story.14:09
liuyulongConfig option tunning may have a lot choices.14:10
liuyulongBut neutron itself still have some architecture defect, which may not be resolved by configuration.14:10
slaweqsorry for being late14:10
liuyulongAs you may see in the comment #1, we did some local works for neutron itself.14:11
slaweqliuyulong: I know that we can't solve everything by config options14:11
liuyulong(Some of them was talked during Shanghai PTG.)14:11
slaweqbut it's rather more about identyfing options which are crucial for large scale and to add some note for some options that e.g. "setting this to high/load value may have impact on large scale because it will make huge load on rabbitmq" (it's just an example for non existing option now :))14:12
liuyulongYes, we can start in such way.14:14
liuyulongAnyway, I will share some config tunning running in our cloud deployment.14:14
slaweqliuyulong: thx a lot14:15
liuyulongOK, let's move on.14:15
liuyulong#topic Bugs14:15
*** openstack changes topic to "Bugs (Meeting topic: neutron_l3)"14:15
liuyulongAnd this I guess:14:16
liuyulongMay be also this:14:17
liuyulongOK, first one:14:17
openstackLaunchpad bug 1858086 in neutron "qrouter's local link route cannot be restored " [Medium,Confirmed]14:17
liuyulongThis should be an API leak for the user input check.14:18
liuyulongWe should not allow user to add some route destination CIDR which overlaps the subnet.14:19
liuyulongThere are too many potential risks for DVR related traffic.14:19
haleybyes, i thought i was reading that wrong but how can you add a route to a local subnet via a non-local IP ?14:19
liuyulongIt is router route-add action?14:21
*** jawad_axd has joined #openstack-meeting14:21
liuyulongNot the subnet static route, right?14:22
slaweqit's "extra-route" but I'm not sure what action is called on server side for it14:24
slaweqon client's side You do "neutron router-update --extra-route"14:24
*** jawad_axd has quit IRC14:26
liuyulongYes, "openstack  router set --route destination=<subnet>,gateway=<ip-address>]"14:26
liuyulongSuch overlap should not be allowed.14:28
liuyulongThis is obvious, when you add an IP address to your host, the system will add a default on-link route for it.14:30
liuyulongThat means "this subnet is directly accessible.", change it does not make any sense in most scenario.14:32
liuyulongBut by the way, the bug reporter said neutron does not recover that route automatically.14:33
haleybi would tend to agree, actually surprised it didn't throw an exception when adding it14:33
liuyulongThis can be another view of the bug, since neutron does not handle such on-link route in the qrouter namespace when it is directly accessible.14:33
liuyulongSo, I think it's OK to terminate it at the very beginning of API.14:35
slaweqsounds good for me14:35
liuyulongOK, next one.14:35
openstackLaunchpad bug 1857422 in neutron "neutron-keepalived-state-change and keeplived cannot be cleanup for those routers which is deleted during l3-agent died" [Undecided,New]14:36
*** links has quit IRC14:37
*** ociuhandu has joined #openstack-meeting14:38
liuyulongFirstly, because the L3-agent is dead, so the "delete RPC" will not be processed, this could be a reason why the processed remained.14:38
haleybif i'm remembering correctly, the l3-agent should clean-up the namespace(s) at the end of it's sync, but is it just not cleaning keepalived stuff because it didn't know that the associated router was ha ?14:40
liuyulongBut we did encounter similar phenomena in our own deployment when L3 agent is alive. The "neutron-keepalived-state-change" and "radvd" processes sometimes remain when routers were deleted.14:40
haleybis this the same thing?14:42
liuyulonghaleyb, I'm not sure, maybe the user's L3-agent is just dead too long time to re-process the delete RPC.14:42
*** eharney has quit IRC14:42
*** ociuhandu has quit IRC14:43
liuyulonghaleyb, no, just some similar phenomena.14:43
haleybright, if for example it didn't get the RPC, that's when the resources get orphaned?14:43
*** ociuhandu has joined #openstack-meeting14:44
liuyulongYes, according to the "reproduction steps" in the bug description.14:44
haleybi guess it seems like a valid bug14:46
liuyulongIf we need to cover this situation, the L3-agent may need a persistent cache to distinguish which router was delete during the down time. And then starts the delete procedure for the stale routers.14:47
*** ociuhandu has quit IRC14:47
*** enriquetaso has quit IRC14:49
*** whoami-rajat__ has quit IRC14:49
liuyulongAnd I still have questions, the router namespace, meta-proxy and radvd process will remain too? Or just neutron-keepalived-state-change and keeplived ?14:50
haleybat the end of sync, the l3-agent should have cleaned the router namespace14:50
haleybinitial sync at startup that is14:51
liuyulongAnd +1 to Miguel's comment, if this is not seen in the production environment, then it is contrived. : )
openstackLaunchpad bug 1857422 in neutron "neutron-keepalived-state-change and keeplived cannot be cleanup for those routers which is deleted during l3-agent died" [Undecided,New]14:53
liuyulongLast one:14:54
openstackLaunchpad bug 1856839 in neutron "[L3] router processing time increase if there are large set ports" [Medium,In progress] - Assigned to LIU Yulong (dragon889)14:54
liuyulongCode is here:
*** whoami-rajat has joined #openstack-meeting14:55
liuyulongIt is an optimization for large scale cloud. : )14:55
slaweqI would also like to ask You for review if You will have some time14:57
liuyulongWe are running out of time, maybe you can leave the comment in the gerrit.14:58
liuyulongAlright, let's end here.14:59
*** openstack changes topic to "OpenStack Meetings ||"14:59
openstackMeeting ended Wed Jan  8 14:59:26 2020 UTC.  Information about MeetBot at . (v 0.1.4)14:59
openstackMinutes (text):
*** ayoung has joined #openstack-meeting15:00
*** LiangFang has quit IRC15:00
*** rcernin has quit IRC15:01
*** ykatabam has quit IRC15:04
*** priteau has joined #openstack-meeting15:06
*** liuyulong has quit IRC15:18
*** liuyulong_ has quit IRC15:18
*** ociuhandu has joined #openstack-meeting15:18
*** ociuhandu has quit IRC15:23
*** whoami-rajat has quit IRC15:24
*** artom has quit IRC15:26
*** eharney has joined #openstack-meeting15:28
*** ociuhandu has joined #openstack-meeting15:42
*** ociuhandu has quit IRC15:50
*** ociuhandu has joined #openstack-meeting15:51
*** ociuhandu has quit IRC15:58
*** artom has joined #openstack-meeting16:01
*** whoami-rajat has joined #openstack-meeting16:03
*** whoami-rajat has quit IRC16:09
*** dmacpher__ has quit IRC16:11
*** dmacpher__ has joined #openstack-meeting16:11
*** mattw4 has joined #openstack-meeting16:17
*** enriquetaso has joined #openstack-meeting16:28
*** gyee has joined #openstack-meeting16:45
*** thgcorrea has quit IRC17:02
*** ociuhandu has joined #openstack-meeting17:03
*** rsimai is now known as rsimai_away17:05
*** artom has quit IRC17:06
*** ociuhandu has quit IRC17:08
*** psachin has quit IRC17:12
*** simon-AS559 has left #openstack-meeting17:12
*** ociuhandu has joined #openstack-meeting17:26
*** ociuhandu has quit IRC17:27
*** ociuhandu has joined #openstack-meeting17:27
*** tbarron_ is now known as tbarron17:30
*** ociuhandu has quit IRC17:31
*** ociuhandu has joined #openstack-meeting17:34
*** jawad_axd has joined #openstack-meeting17:40
*** ociuhandu has quit IRC17:41
*** jawad_axd has quit IRC17:44
*** ociuhandu has joined #openstack-meeting17:44
*** ociuhandu has quit IRC17:49
*** rpittau is now known as rpittau|afk17:51
*** artom has joined #openstack-meeting17:58
*** jawad_axd has joined #openstack-meeting18:00
*** priteau has quit IRC18:01
*** andrebeltrami has quit IRC18:02
*** jawad_axd has quit IRC18:05
*** e0ne has quit IRC18:07
*** ociuhandu has joined #openstack-meeting18:13
*** tesseract has quit IRC18:15
*** ociuhandu has quit IRC18:17
*** jiaopengju has quit IRC18:25
*** jiaopengju has joined #openstack-meeting18:25
*** anastzhyr has quit IRC18:25
*** pcaruana has quit IRC18:27
*** dmacpher_ has joined #openstack-meeting18:50
*** dmacpher__ has quit IRC18:53
*** enriquetaso has quit IRC18:56
*** jawad_axd has joined #openstack-meeting19:02
*** jawad_axd has quit IRC19:07
*** bnemec has quit IRC19:47
*** bnemec has joined #openstack-meeting19:52
*** eharney has quit IRC20:01
*** e0ne has joined #openstack-meeting20:14
*** eharney has joined #openstack-meeting20:15
*** e0ne_ has joined #openstack-meeting20:28
*** e0ne has quit IRC20:28
*** eharney has quit IRC20:38
*** patchbot has joined #openstack-meeting20:58
timburke#startmeeting swift21:00
openstackMeeting started Wed Jan  8 21:00:06 2020 UTC and is due to finish in 60 minutes.  The chair is timburke. Information about MeetBot at
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.21:00
*** openstack changes topic to " (Meeting topic: swift)"21:00
openstackThe meeting name has been set to 'swift'21:00
timburkewho's here for the swift meeting?21:00
rledisezhi o/21:00
*** mattoliverau has joined #openstack-meeting21:01
*** zaitcev has joined #openstack-meeting21:01
timburkei'm guessing mattoliverau's around, too ;-)21:01
timburkeagenda's at
timburkefirst up, i just had a question i wanted to pose to people21:02
timburke#topic probe tests and non-standard configs21:02
*** openstack changes topic to "probe tests and non-standard configs (Meeting topic: swift)"21:02
timburkeactually, i take that back21:02
timburke*first up*, welcome back everyone! it's been a bit :-)21:03
timburkehope everyone's end-of-year breaks were good21:03
zaitcevI do not understand why *PROBE* tests need to be "tolerant". They already assume the structure of the cluster, and so the cluster must be artificial. This is in a stark difference with *functional* tests, which can and do operate on production clusters.21:03
mattoliverauYeah, happy new year everyone!21:03
timburkezaitcev, fair enough. my main thinking was that i'd like to get my dev environment closer to the real thing21:04
timburkeso, back to my original question: how much divergence from "standard" SAIO configs can we reasonably expect for probe tests?21:05
timburkethis came up in the context of
zaitcevI would imagine not much. Although of course an LB appears to be immaterial, remember that they percolate because of that url_base thing.21:06
patchbotpatch 700818 - swift - Deprecate per-service auto_create_account_prefix (MERGED) - 6 patch sets21:06
timburkewhere i wanted to try out the new config option and see some existing tests pass21:06
timburkethat spun out (because the reconciler *really* wants auto_create_account_prefix to be "."21:08
patchbotpatch 701075 - swift - probe-tests: Get reconciler test passing - 2 patch sets21:08
claygto the extent it's *reasonable* to make probe tests more tolerant of various configs we want to support I think it's a nobel endeavor21:08
zaitcevOK, look. I'm okay with anything like this, as long as the source of the tests does not get too complicated. If you post something to address the non-standard prefix and it's just a bunch of parameters passed in configs, I would be happy to review.21:08
claygzaitcev: ๐Ÿ‘21:08
timburkeand where we were assuming that *no constraints at all* were set, at least for py321:08
patchbotpatch 701076 - swift - probe-tests: Avoid a DuplicateSectionError on py3 - 1 patch set21:08
claygyeah that one surprised me a little bit - probably worth loosing up that a bit21:09
timburkebut then i got thinking about it more broadly, in particular while trying to get some SSL support for my dev environment... and i'm finding that it's maybe gonna be painful to make that somewhere that i can still run probe tests :-(21:10
*** ociuhandu has joined #openstack-meeting21:10
clayg"painful to make that somewhere" ?21:11
zaitcevI read it as "make that _work_ somewhere etc."21:12
timburkeit seems like it will be painful to make my SSL-enabled dev environment be a place that i can still run probe tests21:12
zaitcevI accidentally the whole Internet21:12
timburkeanyway, as much as anything, i just wanted to seed the idea a bit, get you guys thinking about it. i feel like we'll probably argue about it over beers in vancouver ;-)21:13
timburke#topic bulk upload regression21:13
*** openstack changes topic to "bulk upload regression (Meeting topic: swift)"21:13
openstackLaunchpad bug 1857546 in OpenStack Object Storage (swift) "Auto-extract looses X-Delete-At and X-Delete-After headers" [High,Confirmed]21:14
claygYEAH!!!  Only... I think the argument is more about "value & priority" vs "should probtests be well designed and flexible"21:14
mattoliverauSorry I'm a bit slow, but could we start using conf.d in saio and then probe tests can drop .conf and remove then as they restart the cluster?21:14
claygoh yeah... that bulk regression ๐Ÿคฎ21:14
rledisezthat one comes from one of our customer (he was faster than us to report the bug :D)21:15
timburkethis came in over the break, wanted to make people aware of it. sorry, that one was my bad21:15
claygI really *liked* the refactor tho, it's something about copied requests and what belongs in the environ?  Is there a compromise that's not just "oops, tried to make it better... didn't on one axis - REVERT"21:15
timburkecause ^^^21:15
rledisezi'll get sure to review the PR. I guess reverting is just to probe the test, right?21:15
rledisez*to prove21:16
patchbotpatch 700652 - swift - Revert "bulk: Use make_subrequest to make subreque... - 1 patch set21:16
timburkefix ^^^21:16
rledisezshould some headers be copied (which one then?) or just copy them all?21:16
timburkeclayg, so it's definitely *not* a complete revert -- the bug i was trying to close is (should be?) still closed21:17
timburkemaybe i should have separated it into two commits... oh well21:18
timburkerledisez, yeah, it seemed like the way we decide which headers to store (config option in the object server) kinda means that we *have to* copy all headers..21:18
timburkeat which point the sanitization we get from make_subrequest does more harm than good21:19
claygtimburke: so some places you added new_env['swift.proxy_access_log_made'] = False to keep the original bug closed ๐Ÿค”21:19
timburkei still think logging subrequests must be a good idea21:19
claygyes ๐Ÿ‘21:20
claygok, well I guess we can probably merge the "revert" and just have two fixed bugs... I thought it was either/or somehow21:20
timburkeanyway, i think we can have more discussions about it on the review -- just wanted to raise awareness and mention that i intend to backport it once we've got the fix on master21:20
rledisezThat behavior was actually unspecified. I'm thinking it could be a specific option instead of just relying on "some headers" to pass, and some not to21:21
rledisezlike X-Bulk-Delete-After ?21:21
*** sfernand has quit IRC21:22
timburke*shrug* i'm not necessarily opposed21:22
timburkebut first i want to fix your customer :-)21:22
timburke#topic SHA-121:22
*** openstack changes topic to "SHA-1 (Meeting topic: swift)"21:22
claygrledisez: bulk has a body we can add new options there21:23
timburkeso some researchers have demonstrated an improved chosen-prefix attack on SHA-1!21:23
timburkemy memory is that we currently only use SHA-1 for tempurls and formpost21:24
timburkeoh, and admin access to /info probably...21:24
timburkefortunately, tempurl now has a config option to specify the allowed digests21:25
timburkeformpost does not yet21:25
openstackLaunchpad bug 1794601 in OpenStack Object Storage (swift) "Formpost middleware should support stronger hash functions" [Undecided,New]21:25
zaitcevso, what now? SHA-256?21:26
timburkewe might want to get on that bug sooner rather than later21:26
timburkezaitcev, that was my thinking. SHA-256 or SHA-51221:26
claygbits for days!21:26
claygBy renting a GPU cluster online, the entire chosen-prefix collision attack on SHA-1 costed us about 75k USD21:27
timburkeand we should think about starting the deprecation/removal process for SHA-1 signatures. we've kinda known about the need for that for a while21:27
claygI would easily remove formpost from swift for that kind of money ๐Ÿ’ฐ21:27
timburkeclayg, i think we use it for uploading tarballs ;-)21:28
claygi guess we'll fix it then21:28
timburkeit shouldn't be too hard to do21:28
rledisezI'm not sure how it impact tempurl. generating collision does not mean you know what you're looking for (aka. you don't know the secret, or you don't need colisions). so yes, we must deprecate SHA1, just because, but there is no emergency i think21:28
clayg@timburke by "deprecate" you mean like "log warning" => "refused to start" if you've configured these things to use signatures we're not happy with?21:29
*** rfolco has quit IRC21:29
claygdo you ever need to support more than one kind of signature at a time?  Like a migration path sort of plan?21:29
timburkerledisez, ๐Ÿ‘ good to get a public cloud operator's perspective21:29
timburkeclayg, log warning => start ignoring sha-1 if you've got it listed, was my thinking21:30
claygdeprecating stuff is always a good idea, we should deprecate auto_create_account_prefix21:30
timburkeclayg, tempurl *today* should support multiple signatures21:30
*** ralonsoh has quit IRC21:30
claygsweet so prior art even21:30
clayghrmm... I wonder how we make sure it get's done "eventually"21:31
timburkeof course, this also means swiftclient should learn how to do the new stuff ;-)21:31
rledisezhum, if you ignore it, you just break all already generated url, right? you can't do that, it's up to the operator to decide to remove it21:31
claygmaybe someone will come click that little follow button or make it ๐Ÿ”ฅ or something21:31
*** armax has joined #openstack-meeting21:32
claygtimburke: thanks for bringing this one up!21:32
timburkerledisez, i was hoping to get away with having a fairly long window, maybe even have a step of logging warnings *per-request* when handling sha-1 sigs... actual ignoring could be a ways off21:33
timburkethat about does it for big topics i wanted to bring up21:33
timburkeon to updates!21:33
timburke#topic versioning21:33
*** openstack changes topic to "versioning (Meeting topic: swift)"21:33
timburkeclayg, tdasilva i feel like there's been a lot of progress lately :-D21:34
claygwe are locking shit DOWN!21:34
claygI need to re-look at this one
patchbotpatch 698139 - swift - Skip container-sync when versioning enabled - 9 patch sets21:35
claygwhich is the only dangling "probably should squash" (or at least fast follow merge) before it's on master21:35
claygrledisez: you may have to rub some braincells together about container-sync and versioning - for now they're not going to work together, but we won't *break* either - so I guess we were hoping that would be "enough"21:36
claygthis is still the meat ->
patchbotpatch 682382 - swift - New Object Versioning mode - 70 patch sets21:36
claygand the money ->
patchbotpatch 673682 - swift - s3api: Implement object versioning API - 44 patch sets21:36
timburkehow quickly should we try to get the client support from merged after the swift patch lands?21:37
patchbotpatch 691877 - python-swiftclient - object versioning features - 8 patch sets21:37
timburkehow are you feeling about the client support?21:37
claygof course if you have swiftclient with versions support (p 691877) you don't need to mess with that aws s3api list-object-versions non-sense21:37
rledisezclayg: versioning at source or at destination will not work with container-sync?21:37
claygtimburke: thanks for reminding me!  I should work on `swift delete --all` ๐Ÿ‘21:37
claygrledisez: at *source* was the primary concern, I'm not sure I've tested at dest21:38
timburkerledisez, source, i believe. clients won't be able to set both x-versions-enabled and x-container-sync-to or something like that21:38
claygmight be worth saying something to that effect ont he container-sync + versionsing patch21:38
tdasilvarledisez: I believe dest too, in that we won't allow enabling versioning if sync header is there21:38
*** ociuhandu has quit IRC21:39
timburkedo we know how well a VW-enabled container serves as a sync destination?21:39
claygrledisez: you might have to refresh us how container-sync works21:39
claygtimburke: could be a interesting data point - I don't think it's something we test - althougth we do have some probetests for container-sync21:39
rledisezclayg: a process scan the sqlite containers, iterate through the "new rows" (based on some sync point), and PUT/DELETE the objects to the remote endpoint. currently I think it works with versionning at destination. I don't see why it wouldn't work actually, but I didn't tested21:41
timburkeclayg, so how soon do we expect these to land on master? and what can/should people be doing to help out?21:42
claygI'd like to make a decision about "can we land new versioning on master w/o container-sync" because... we don't really have any plans to make that situation any better21:43
claygif the answer is "yes" but it needs to fail/error in expected and obvious ways... then I'm sure we'll have that tied up sometime next week21:44
claygwe were doing some release stuff, but now I'm available to finish up whatever we need to do to land upstream21:44
clayghas anyone tried out the API?!21:44
timburke(besides me, clayg, and tdasilva ;-)21:45
claygnothing of any significance has changed in AGES - it's just been spit and polish21:45
claygwe're certainly past the point of making significant redesign sort of conversations - so if you haven't looked at it yet, it's probably more just FYI at this point21:45
tdasilvaviks was asking about testing the s3 api last night, which is great21:46
claygwe'll merge it when it's done21:46
clayg๐Ÿค— viks___21:46
timburke#topic slo ranged reads21:46
*** openstack changes topic to "slo ranged reads (Meeting topic: swift)"21:46
timburkejust a quick update on the patch21:46
patchbotpatch 697739 - swift - Have slo tell the object-server that it wants whol... - 6 patch sets21:46
timburkei tried out having symlink and dlo use the same header as slo, worked like a charm!21:47
timburkei think that cleans up the status codes getting logged at the object layer nicely, too21:47
timburkeit'll be interesting getting opinions from other people on it21:48
timburke#topic quoted etags21:48
*** openstack changes topic to "quoted etags (Meeting topic: swift)"21:48
patchbotpatch 700056 - swift - Middleware that allows a user to have quoted Etags - 5 patch sets21:48
claygtimburke: ยกexcelente!21:48
timburkepatch has tests and docs now!21:48
timburkethanks for proposing the new middleware, rledisez! i'm liking that direction more and more21:48
timburkei think the only reservations i have with it now come down to naming (both for the config option and the cotnainer-metadata header)21:50
tdasilvamy hesitation with that patch is the need for a brand new middleware just for quoting etags. It feels like it should be an option on/off21:50
clayg@rledisez what do you think about a global cluster wide option as well? sounds like legacy use-cases could still turn it off per account/container with the explicit metadata21:50
timburkeclayg, i already added that :-) defaults to off21:50
claygtimburke: well then let's merge that shit!21:51
rlediseztdasilva: i don't like the middleware neither, but a global option does not give the flexibility if per account/container. without that flexibility, we will breaks so many apps :(21:51
timburketdasilva, idk -- i kinda like the separation of concerns. *especially* given the granularity rledisez built in21:52
timburkenow, to take another look at torgomatic's ...21:53
patchbotpatch 504472 - swift - Shorten typical proxy pipeline. - 4 patch sets21:53
claygi don't mind having a small single purpose middleware - compared to what we do in bulk21:53
claygoh and even for clusters where "oh I definately want that on everywhere" - they may someday find one account/container that wants to use some old app that blows up with quoted etags - so I love having the explicit metadata options per collection21:54
rledisezwhat I'm mostly concerned about middleware (except that paste does not really seem maintained) is the side effect of a bad ordered pipeline21:54
claygyeah, with the defaults should we think about making it auto-inserted?21:55
rlediseztimburke: about the naming of the header, I trust your english better than mine ;)21:55
timburkerledisez, for *sure*. i think i've come across like 4 or 5 different bug reports with super-strange (and not necessarily overlapping!) errors that came down to "pipeline was bad"21:55
claygactually - that doesn't really help us... even if you auto-insert it you have to configure it21:55
rledisezi'll just notify the only user of this feature to prepare before we upgrade21:56
claygpipelines suck..21:56
timburkeclayg, well, auto-inserted allows users to do self-service21:56
claygtimburke: oh that's nice!21:56
timburkerledisez, i'd also be ok with looking for an old header name for you. that's not a big deal21:56
timburkeanyway, that's all i've got21:57
claygtimburke: althought auto-insert migth not be really what I want... it be more like "auto-ORDERED" pipelines or osmething21:57
timburke#topic open discussion21:57
*** openstack changes topic to "open discussion (Meeting topic: swift)"21:57
timburkeclayg, auto-ordered pipelines would be *amazing*21:57
timburkebut maybe also crazy hard, i think21:57
rlediseztimburke: let's not polute the code, i'll maintain a small patch internaly until i can drop it21:57
timburkeand it'd require that we actually specify middleware dependencies21:58
timburkemaybe we could put some attributes on the filter_factories? start there?21:59
timburkeit'd be cool, though!21:59
mattoliverauI had some code that did that.. just to play years ago.21:59
rlediseztimburke: for sure. even dropping the notion of pipeline by default and just have flags "enable_bulk", "enable_etag_quoter", โ€ฆ. keep the pipeline for advanced users22:00
mattoliverauit might still be up in gerrit somewhere. Probably need alot of work22:00
rledisezI think it was kinda like torgomatic's patch22:00
claygi gotta bounce22:00
timburkeall right, looks like we're out of time22:00
timburkethank you all for coming, and thank you for working on swift!22:00
claygrledisez: can you drop a comment/review on it - i didn't really underestand it I don't think22:00
*** openstack changes topic to "OpenStack Meetings ||"22:00
openstackMeeting ended Wed Jan  8 22:00:54 2020 UTC.  Information about MeetBot at . (v 0.1.4)22:00
openstackMinutes (text):
*** zaitcev has left #openstack-meeting22:01
*** jawad_axd has joined #openstack-meeting22:06
*** eharney has joined #openstack-meeting22:08
*** patchbot has left #openstack-meeting22:08
*** jawad_axd has quit IRC22:10
*** slaweq has quit IRC22:15
*** e0ne_ has quit IRC22:31
*** armax has quit IRC22:33
*** ykatabam has joined #openstack-meeting22:47
*** jawad_axd has joined #openstack-meeting23:08
*** slaweq has joined #openstack-meeting23:11
*** jawad_axd has quit IRC23:12
*** dviroel has quit IRC23:12
*** rcernin has joined #openstack-meeting23:14
*** slaweq has quit IRC23:16
*** armax has joined #openstack-meeting23:19
*** eharney has quit IRC23:19
*** jawad_axd has joined #openstack-meeting23:29
*** jawad_axd has quit IRC23:33
*** tetsuro_ has joined #openstack-meeting23:50
*** tetsuro has quit IRC23:52

Generated by 2.15.3 by Marius Gedminas - find it at!