Wednesday, 2020-03-18

*** b1airo has quit IRC00:10
*** jamesmcarthur has joined #openstack-meeting00:15
*** maohongbo has quit IRC00:18
*** maohongbo has joined #openstack-meeting00:18
*** dmacpher_ has joined #openstack-meeting00:24
*** dmacpher has quit IRC00:26
*** tetsuro has joined #openstack-meeting00:28
*** gyee has quit IRC00:38
*** ociuhandu has joined #openstack-meeting00:42
*** ijw has joined #openstack-meeting00:46
*** larainema has quit IRC00:48
*** ociuhandu has quit IRC00:49
*** ijw has quit IRC00:51
*** jmasud has quit IRC00:53
*** jmasud has joined #openstack-meeting00:56
*** jamesmcarthur has quit IRC00:57
*** jamesmcarthur has joined #openstack-meeting00:57
*** nicolasbock has quit IRC01:01
*** ysandeep|afk is now known as ysandeep01:04
*** boldo has quit IRC01:08
*** ijw has joined #openstack-meeting01:18
*** brinzhang__ has joined #openstack-meeting01:20
*** ijw has quit IRC01:23
*** brinzhang_ has quit IRC01:23
*** ijw has joined #openstack-meeting01:49
*** mmethot has quit IRC01:53
*** mmethot has joined #openstack-meeting01:53
*** ijw has quit IRC01:54
*** yaawang has quit IRC01:56
*** yaawang has joined #openstack-meeting01:57
*** jamesmcarthur has quit IRC02:02
*** armax has quit IRC02:02
*** jamesmcarthur has joined #openstack-meeting02:02
*** diablo_rojo has quit IRC02:03
*** masahito has joined #openstack-meeting02:10
*** masahito has quit IRC02:11
*** masahito has joined #openstack-meeting02:12
*** maohongbo has quit IRC02:13
*** maohongbo has joined #openstack-meeting02:14
*** yamamoto has joined #openstack-meeting02:16
*** ijw has joined #openstack-meeting02:20
*** ijw has quit IRC02:25
*** ijw has joined #openstack-meeting02:52
*** tetsuro_ has joined #openstack-meeting02:53
*** jamesmcarthur has quit IRC02:54
*** tetsuro has quit IRC02:56
*** ijw has quit IRC02:56
*** jamesmcarthur has joined #openstack-meeting02:57
*** ricolin_ has joined #openstack-meeting03:00
*** mmethot_ has joined #openstack-meeting03:05
*** mmethot has quit IRC03:07
*** yaawang has quit IRC03:08
*** yaawang has joined #openstack-meeting03:09
*** apetrich has quit IRC03:09
*** rh-jelabarre has quit IRC03:10
*** dtroyer has joined #openstack-meeting03:14
*** jamesmcarthur has quit IRC03:16
*** ijw has joined #openstack-meeting03:23
*** masahito has quit IRC03:24
*** masahito has joined #openstack-meeting03:24
*** ijw has quit IRC03:28
*** jamesmcarthur has joined #openstack-meeting03:31
*** ijw has joined #openstack-meeting03:54
*** ijw has quit IRC03:59
*** armax has joined #openstack-meeting04:11
*** yaawang has quit IRC04:24
*** yaawang has joined #openstack-meeting04:25
*** ijw has joined #openstack-meeting04:26
*** tetsuro has joined #openstack-meeting04:29
*** ijw has quit IRC04:31
*** tetsuro_ has quit IRC04:32
*** jmasud has quit IRC04:53
*** jmasud has joined #openstack-meeting04:55
*** masahito has quit IRC04:56
*** ijw has joined #openstack-meeting04:58
*** ijw has quit IRC05:02
*** dklyle has quit IRC05:04
*** ricolin_ has quit IRC05:05
*** ricolin_ has joined #openstack-meeting05:05
*** ricolin_ has quit IRC05:05
*** jamesmcarthur has quit IRC05:11
*** tetsuro_ has joined #openstack-meeting05:32
*** tetsuro has quit IRC05:34
*** dtroyer has left #openstack-meeting05:39
*** links has joined #openstack-meeting05:40
*** ijw has joined #openstack-meeting06:00
*** ijw has quit IRC06:04
*** ijw has joined #openstack-meeting06:31
*** ijw has quit IRC06:35
*** tetsuro has joined #openstack-meeting06:40
*** tetsuro_ has quit IRC06:44
*** ociuhandu has joined #openstack-meeting06:48
*** ociuhandu has quit IRC06:52
*** ijw has joined #openstack-meeting07:02
*** tetsuro_ has joined #openstack-meeting07:04
*** tetsuro has quit IRC07:07
*** ijw has quit IRC07:07
*** ralonsoh has joined #openstack-meeting07:31
*** ijw has joined #openstack-meeting07:34
*** ijw has quit IRC07:38
*** dmsimard|off7 has joined #openstack-meeting07:48
*** dmsimard|off has quit IRC07:48
*** dmsimard|off7 is now known as dmsimard|off07:48
*** tesseract has joined #openstack-meeting07:50
*** maciejjozefczyk has joined #openstack-meeting07:59
*** e0ne has joined #openstack-meeting07:59
*** slaweq has joined #openstack-meeting08:03
*** maciejjozefczyk_ has joined #openstack-meeting08:04
*** maciejjozefczyk has quit IRC08:05
*** ijw has joined #openstack-meeting08:05
*** ijw has quit IRC08:10
*** rcernin has quit IRC08:21
*** maohongbo has quit IRC08:24
*** maohongbo has joined #openstack-meeting08:24
*** slaweq has quit IRC08:28
*** slaweq has joined #openstack-meeting08:29
*** Lucas_Gray has joined #openstack-meeting08:30
*** tetsuro has joined #openstack-meeting08:35
*** ijw has joined #openstack-meeting08:37
*** tetsuro_ has quit IRC08:38
*** maohongbo has quit IRC08:39
*** apetrich has joined #openstack-meeting08:40
*** maohongbo has joined #openstack-meeting08:40
*** ijw has quit IRC08:42
*** ociuhandu has joined #openstack-meeting08:45
*** rpittau|afk is now known as rpittau08:45
*** jmasud has quit IRC08:55
*** jmasud has joined #openstack-meeting08:56
*** strigazi has joined #openstack-meeting08:57
*** Lucas_Gray has quit IRC08:58
*** Lucas_Gray has joined #openstack-meeting09:00
*** tetsuro has quit IRC09:00
*** maciejjozefczyk_ is now known as maciejjozefczyk09:01
*** ijw has joined #openstack-meeting09:09
*** maohongbo1 has joined #openstack-meeting09:12
*** ociuhandu has quit IRC09:12
*** ociuhandu has joined #openstack-meeting09:12
*** maohongbo has quit IRC09:12
*** maohongbo1 is now known as maohongbo09:12
*** ijw has quit IRC09:14
*** yamamoto has quit IRC09:28
*** ijw has joined #openstack-meeting09:40
*** ijw has quit IRC09:45
*** mmethot has joined #openstack-meeting09:52
*** mmethot_ has quit IRC09:54
*** Lucas_Gray has quit IRC09:59
*** Lucas_Gray has joined #openstack-meeting10:00
*** corvus_ has joined #openstack-meeting10:01
*** corvus has quit IRC10:04
*** ociuhandu has quit IRC10:09
*** ociuhandu has joined #openstack-meeting10:10
*** ijw has joined #openstack-meeting10:11
*** yamamoto has joined #openstack-meeting10:15
*** ociuhandu has quit IRC10:15
*** ijw has quit IRC10:16
*** ykatabam has quit IRC10:17
*** ociuhandu has joined #openstack-meeting10:21
*** ociuhandu has quit IRC10:27
*** ijw has joined #openstack-meeting10:42
*** ijw has quit IRC10:47
*** jawad_axd has joined #openstack-meeting10:50
*** nicolasbock has joined #openstack-meeting11:03
*** yamamoto has quit IRC11:07
*** yamamoto has joined #openstack-meeting11:10
*** ijw has joined #openstack-meeting11:14
*** tesseract-RH has joined #openstack-meeting11:16
*** tesseract has quit IRC11:18
*** ociuhandu has joined #openstack-meeting11:19
*** Wryhder has joined #openstack-meeting11:22
*** Lucas_Gray has quit IRC11:23
*** Wryhder is now known as Lucas_Gray11:23
*** ykatabam has joined #openstack-meeting11:23
*** Lucas_Gray has quit IRC11:32
*** Lucas_Gray has joined #openstack-meeting11:34
*** Lucas_Gray has quit IRC11:35
*** ociuhandu has quit IRC11:35
*** Lucas_Gray has joined #openstack-meeting11:39
*** Lucas_Gray has quit IRC11:47
*** Lucas_Gray has joined #openstack-meeting11:49
*** rh-jelabarre has joined #openstack-meeting11:53
*** raildo has joined #openstack-meeting11:56
*** rpittau is now known as rpittau|bbl12:01
*** ralonsoh has quit IRC12:05
*** hyunsikyang__ has quit IRC12:09
*** ralonsoh has joined #openstack-meeting12:09
*** yamamoto has quit IRC12:14
*** yamamoto has joined #openstack-meeting12:16
*** ijw has quit IRC12:17
*** ykatabam has quit IRC12:30
*** yamamoto has quit IRC12:35
*** ijw has joined #openstack-meeting12:46
*** ijw has quit IRC12:50
*** jmasud has quit IRC12:55
*** jmasud has joined #openstack-meeting12:56
*** rpittau|bbl is now known as rpittau13:01
*** tesseract has joined #openstack-meeting13:05
*** tesseract-RH has quit IRC13:08
*** Lucas_Gray has quit IRC13:08
*** yamamoto has joined #openstack-meeting13:12
*** Lucas_Gray has joined #openstack-meeting13:12
*** yamamoto has quit IRC13:14
*** Lucas_Gray has quit IRC13:16
*** ijw has joined #openstack-meeting13:17
*** Lucas_Gray has joined #openstack-meeting13:18
*** vishalmanchanda has joined #openstack-meeting13:19
*** ijw has quit IRC13:21
*** brinzhang__ has quit IRC13:31
*** brinzhang__ has joined #openstack-meeting13:31
*** number80 has quit IRC13:33
*** ociuhandu has joined #openstack-meeting13:34
*** TrevorV has joined #openstack-meeting13:36
*** ociuhandu has quit IRC13:38
*** jraju__ has joined #openstack-meeting13:43
*** armax has quit IRC13:43
*** links has quit IRC13:44
*** liuyulong has joined #openstack-meeting13:46
*** enriquetaso has joined #openstack-meeting13:48
*** ijw has joined #openstack-meeting13:48
*** ijw has quit IRC13:53
*** ykatabam has joined #openstack-meeting13:56
*** Liang__ has joined #openstack-meeting13:57
*** Liang__ is now known as LiangFang13:57
liuyulong#startmeeting neutron_l314:00
openstackMeeting started Wed Mar 18 14:00:37 2020 UTC and is due to finish in 60 minutes.  The chair is liuyulong. Information about MeetBot at
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.14:00
*** openstack changes topic to " (Meeting topic: neutron_l3)"14:00
openstackThe meeting name has been set to 'neutron_l3'14:00
*** sfernand has joined #openstack-meeting14:00
liuyulong#topic Announcements14:00
*** openstack changes topic to "Announcements (Meeting topic: neutron_l3)"14:00
liuyulonglet's recall wha14:01
openstackRemoving item from minutes: #link
liuyulonglet's recall the team's Announcements.14:01
liuyulongCongrats to Lajos Katona.14:02
liuyulongWelcome to the core team.14:03
liuyulongwe now have 11 members.14:04
*** Lucas_Gray has quit IRC14:04
liuyulongOh, and ovn cores, it should be 15.14:05
slaweqand most of them are active, so I think our team is in good shape now14:05
slaweqI also have some another potential candidates in mind, but not for now (yet) :)14:05
liuyulongYes, the team are active, and we also have 4 meetings per week.14:07
liuyulongOK, next topic should be the virtual PTG.14:07
liuyulongMy concern is jet lag, it the virtual PTG will be held a full day, someone may not be available to attend some topics.14:08
liuyulongI have applied for my travel support and no reply at present.14:09
slaweqcurrently lets just focus on planning topics to discuss14:10
slaweqand we will see how it will be :)14:10
*** dklyle has joined #openstack-meeting14:11
liuyulongI hope that people all over the world can safely defeat the virus.14:11
liuyulongOK, let's move on.14:12
liuyulong#topic Bugs14:12
*** openstack changes topic to "Bugs (Meeting topic: neutron_l3)"14:12
*** brinzhang__ has quit IRC14:14
liuyulongJust one related to L3 OVN:14:15
openstackLaunchpad bug 1867122 in neutron "Unnecessary network flapping while update floatingip without port or fixed ip changed" [Low,In progress] - Assigned to Taoyunxiang (taoyunxiang)14:15
liuyulongIt has a fix:
ralonsoh(maybe we should add [OVN] in the title)14:16
liuyulongIt has a tag [OVN]14:16
slaweqralonsoh: we have "ovn" tag which IMO should be enough14:16
liuyulongThe patch is related to OVN north DB14:19
*** ijw has joined #openstack-meeting14:19
liuyulongIt's a bit out of our scope again, so let's continue the review in gerrit.14:22
liuyulongNo more L3 bugs from the bug deputy report. So let's have a quick scan of the LP bug list.14:22
*** ijw has quit IRC14:24
liuyulongLooks like it was a steady and quiet week, one L3 bug from me today.14:25
liuyulongYou guys have any updates?14:25
haleybnone from me14:25
liuyulongOK, let's move on.14:26
liuyulong#topic OVN_L314:26
*** openstack changes topic to "OVN_L3 (Meeting topic: neutron_l3)"14:26
liuyulonglucasagomes, maciejjozefczyk any updates for L3 of OVN?14:27
liuyulongAlright, they are not here.14:28
maciejjozefczykliuyulong I would like to ask for review this patch
ralonsohrelated to L3, the FIP QoS support is still under investigation14:28
maciejjozefczykThats the patch we talked a bit earler, about rescheduling14:28
maciejjozefczykyes, for now I send e-mail to OVS ml2 if its possible to do QoS on FIP, for not any answer14:29
maciejjozefczykI *think* without any significant changes in Core OVN its not possible, but im not an expert at all in core ovn14:29
maciejjozefczykI'm gonna ping our core-ovn team in order to have any insights about it14:30
liuyulongYes, I remember that patch, it looks good to me.14:32
maciejjozefczykthats all what I have about OVN&L314:32
maciejjozefczykthanks liuyulong ;)14:32
liuyulongIt can be tested in a all-in-one devstack deployment?14:32
ralonsohyes, it's possible14:34
ralonsohI don't remember now, but there are some local.conf examples for this14:34
ralonsoh(I'll send you the links)14:34
liuyulongI'd like to run that code locally, so I want to know if one node is enough to test it : )14:34
maciejjozefczykliuyulong, yes, but I think that needs multinode deployment, at least to have more than one chassis14:34
maciejjozefczykI worked on it having env with 3 chassis14:34
ralonsohmaciejjozefczyk, did you use devstack?14:35
maciejjozefczykso 1 node all in one and 2 nodes with only nova-compute and ovn-controllers14:35
maciejjozefczykralonsoh, yes, lemme find a likn14:35
maciejjozefczykok, so first node:
maciejjozefczykand computes:
maciejjozefczykthat should work, at least worked a few days back :D14:36
liuyulongCool, this could be a good advice for reviewers when they want to run the code in a running deployment.14:37
maciejjozefczykliuyulong, ok, I added a comment there how to test it.14:38
liuyulongI have a 5 node devstack deployment for neutron agents(none-OVN), one controller, 2 compute nodes and 2 network nodes. : )14:38
liuyulongAbout the OVN FIP QoS, there is a new implementation  uploaded14:40
liuyulong recently.14:40
ralonsohin Neutron?14:41
ralonsohwell, not for FIP precisely but a refactor of the QoS extension in the OVN client14:42
liuyulongThis one ^14:42
ralonsohahhhhh ok14:42
ralonsohgood to know this14:42
*** ysandeep is now known as ysandeep|afk14:42
liuyulongBut seems the author just want to run the CI. : (14:42
maciejjozefczykWith meter actions it changes a lot, I think that not trivial to support FIP on QoS when ovs meter is used.14:44
*** jamesmcarthur has joined #openstack-meeting14:44
maciejjozefczykAnd for now we just switched QoS to use meters, because normal 'tc' didn't work while the traffic went throught geneve tunnels,14:44
maciejjozefczykI don't remember the specifics, but ovs meters is the way to go now.14:44
liuyulongThe code seems copied from this:
ralonsohso the base strategy is to setup the qos on the GW port14:46
maciejjozefczykralonsoh, so thats bad right? it will limit N/S traffic for all ports connected to that router.14:47
ralonsohthis is not like TC, where you can specify the src IP or MAC14:47
slaweqif it's qos for fip then it has to be only N/S traffic, right?14:48
ralonsohto filter the class shaping14:48
slaweqahh, ok14:48
ralonsohyes, but you should be able to apply a QoS per FIP14:48
maciejjozefczykbut, anyways, thats also needed. I mean from what I remember there is possibility to create QoS on gateway port, right?14:48
ralonsohhere you can only define one QoS OVN rule per direction and port14:48
maciejjozefczykbut ralonsoh changes could also support it out of the box14:48
slaweqso there is no way to say "limit only traffic with src/dest == a.b.c.d", correct?14:48
slaweqok, thx14:48
slaweqgot it now14:49
maciejjozefczykslaweq, no, we can specify only OVN 'inport', which means OVN Logical_Switch_Port14:49
maciejjozefczykOVN FIP is only a entry about DNAT/SNAT action14:49
liuyulong may be here has a clue about it.14:50
*** ijw has joined #openstack-meeting14:51
maciejjozefczykI send an email about it to ovs-discuss ml.14:51
ralonsohliuyulong, yes, but this is like applying a QoS to a specific port14:51
ralonsohthis is not FIP QOS14:52
maciejjozefczykliuyulong, hmm, maybe construction like: "'%s == "%s" && ip4.%s == %s && is_chassis_resident("%s")" will solve it, I don't know14:52
maciejjozefczykLine 26 from proposition14:52
ralonsohis this possible???14:52
liuyulongmaciejjozefczyk, yes, that match has an IP address.14:53
ralonsohI didn't see that in the NB or SB info14:53
maciejjozefczykmaybe the match action is more sophisticated, I mean maybe its able to match more than only inport14:53
maciejjozefczykyes, it is worth exploring14:53
maciejjozefczykwith ralonsoh refactor that would be pretty easy to add14:53
ralonsohfor sure14:54
ralonsohI need to investigate this ASAP14:54
liuyulongAbout the QoS, I have an addition, it's the Gateway IP.14:54
liuyulongQoS on gateway port may limit all traffic like floating IP and SNAT (VMs to outside world without fip).14:54
liuyulongGateway IP should be SNAT only.14:55
*** ijw has quit IRC14:55
*** macz_ has joined #openstack-meeting14:55
*** corvus_ is now known as jeblair14:56
*** jeblair is now known as corvus14:56
maciejjozefczykliuyulong, yes14:56
liuyulongOK, last topic.14:57
liuyulong#topic On demand agenda14:57
*** openstack changes topic to "On demand agenda (Meeting topic: neutron_l3)"14:57
openstackLaunchpad bug 1867119 in neutron "[security] Add allowed-address-pair to one port will open all others' protocol under same security group" [Critical,In progress] - Assigned to LIU Yulong (dragon889)14:57
liuyulongI just updated the patch, reviews are welcomed.14:57
liuyulongIt's not related L3 IMO. : )14:58
maciejjozefczykadded to my list14:59
slaweqliuyulong: I will test it14:59
*** priteau has joined #openstack-meeting15:00
liuyulongThanks : )15:00
liuyulongOK, so let's end here.15:00
*** openstack changes topic to "OpenStack Meetings ||"15:00
openstackMeeting ended Wed Mar 18 15:00:27 2020 UTC.  Information about MeetBot at . (v 0.1.4)15:00
openstackMinutes (text):
*** ijw has joined #openstack-meeting15:22
*** ijw has quit IRC15:26
*** vishalmanchanda has quit IRC15:31
*** enriquetaso has quit IRC15:35
*** LiangFang has quit IRC15:41
*** jawad_axd has quit IRC15:45
*** ijw has joined #openstack-meeting15:53
*** yamamoto has joined #openstack-meeting15:58
*** ijw has quit IRC16:00
*** ijw has joined #openstack-meeting16:10
*** martial has quit IRC16:14
*** slaweq has quit IRC16:28
*** jmasud has quit IRC16:34
*** jmasud has joined #openstack-meeting16:34
*** baojg has joined #openstack-meeting16:37
*** johanssone has quit IRC16:51
*** jmasud has quit IRC16:54
*** jraju__ has quit IRC16:54
*** johanssone has joined #openstack-meeting16:56
*** jmasud has joined #openstack-meeting16:56
*** johanssone has quit IRC16:57
*** tesseract has quit IRC16:59
*** johanssone has joined #openstack-meeting17:03
*** johanssone has quit IRC17:03
*** johanssone has joined #openstack-meeting17:05
*** diablo_rojo has joined #openstack-meeting17:05
*** johanssone has quit IRC17:05
*** johanssone has joined #openstack-meeting17:06
*** johanssone has quit IRC17:08
*** johanssone has joined #openstack-meeting17:14
*** johanssone has quit IRC17:15
*** johanssone has joined #openstack-meeting17:20
*** johanssone has quit IRC17:20
*** johanssone has joined #openstack-meeting17:23
*** rmcall has joined #openstack-meeting17:26
*** gyee has joined #openstack-meeting17:27
*** johanssone has quit IRC17:27
*** priteau has quit IRC17:42
*** yamamoto has quit IRC17:44
*** jawad_axd has joined #openstack-meeting18:13
*** diablo_rojo has quit IRC18:43
*** diablo_rojo has joined #openstack-meeting18:45
*** e0ne has quit IRC18:46
*** nicolasbock has quit IRC18:47
*** nicolasbock has joined #openstack-meeting18:54
*** jawad_axd has quit IRC18:54
*** maciejjozefczyk has quit IRC18:56
*** enriquetaso has joined #openstack-meeting19:31
*** enriquetaso has quit IRC19:37
*** ociuhandu has joined #openstack-meeting19:39
*** rpittau is now known as rpittau|afk19:40
*** diablo_rojo has quit IRC19:52
*** ociuhandu has quit IRC19:57
*** gmann is now known as gmann_afk19:57
*** ociuhandu has joined #openstack-meeting19:58
*** diablo_rojo has joined #openstack-meeting20:00
*** ralonsoh has quit IRC20:01
*** ociuhandu has quit IRC20:02
*** ociuhandu has joined #openstack-meeting20:02
*** jamesmcarthur has quit IRC20:09
*** jamesmcarthur has joined #openstack-meeting20:28
*** gmann_afk is now known as gmann20:33
*** ociuhandu has quit IRC20:37
*** ociuhandu has joined #openstack-meeting20:38
*** rmcall has quit IRC20:38
*** dklyle has quit IRC20:39
*** ociuhandu has quit IRC20:39
*** ociuhandu has joined #openstack-meeting20:40
*** nicolasbock has quit IRC20:40
*** dklyle has joined #openstack-meeting20:40
*** ykatabam has quit IRC20:49
*** ykatabam has joined #openstack-meeting20:50
*** jamesmcarthur has quit IRC20:52
*** jamesmcarthur has joined #openstack-meeting20:53
*** jmasud has quit IRC20:55
*** jmasud has joined #openstack-meeting20:56
*** ociuhandu has quit IRC20:56
*** ociuhandu has joined #openstack-meeting20:57
*** jamesmcarthur has quit IRC20:58
*** jamesmcarthur has joined #openstack-meeting20:58
*** alecuyer has joined #openstack-meeting20:58
timburke#startmeeting swift21:00
openstackMeeting started Wed Mar 18 21:00:17 2020 UTC and is due to finish in 60 minutes.  The chair is timburke. Information about MeetBot at
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.21:00
*** openstack changes topic to " (Meeting topic: swift)"21:00
openstackThe meeting name has been set to 'swift'21:00
timburkewho's here for the swift meeting?21:00
*** jamesmcarthur has quit IRC21:00
claygmore like *party* time21:02
timburkeagenda's at
*** patchbot has joined #openstack-meeting21:02
timburke#topic covid-19 / Vancouver21:02
*** openstack changes topic to "covid-19 / Vancouver (Meeting topic: swift)"21:02
timburkeso i'd meant to mention this thread last week but forgot (things have been a little hectic with my recent job transition)21:02
*** ociuhandu has quit IRC21:02
timburkebut it looks like i've got a new message to reference now, anyway!21:02
timburkelooks like the PTG is going virtual!21:02
*** jamesmcarthur has joined #openstack-meeting21:03
timburkei know just this week the bay area (where i live) has recommended people shelter-in-place for the next three weeks21:03
timburkeso i guess this isn't entirely surprising21:03
timburke(also, apologies in advance -- i'm probably going to be less available than usual if i've got two small kids at home full time)21:04
alecuyeryup.. I'm trying that now (and with just one :) )21:04
rledisezhi /21:04
clayg*virtual* PTG 🤔21:04
claygI hadn't heard that - thanks @timburke21:04
kota_no worry, a lot of people have same situation. I also take care of my kids in my home.21:05
alecuyerclayg: sounds as good as a "virtual" beer no ? but I guess it's for the best21:05
kota_+1 for the virtual drinking.21:06
timburkei'm sure there will be more organizing and planning going on over the next few months21:06
timburkeand i'm hopeful about us finding a way to have some dedicated time together to think hard about swift :-)21:07
timburkestay safe everyone!21:07
claygperhaps even in person post apocalypse!21:07
*** jamesmcarthur has quit IRC21:08
timburkewe'll all meet up at matt's ~~beach house~~ bunker!21:08
timburke#topic jerasure support in liberasurecode21:09
*** openstack changes topic to "jerasure support in liberasurecode (Meeting topic: swift)"21:09
timburkeso i noticed recently that the liberasurecode gate is currently broken21:09
timburkeall jobs fail with something like `fatal: repository '' not found`21:09
timburkeat first, i was inclined to just replace the repo with a working mirror (such as ceph's fork on github, done in
patchbotpatch 712842 - liberasurecode - Use ceph's GitHub mirrors for gf-complete/jerasure - 1 patch set21:09
timburkebut investigating further, i found21:09
timburkethe notice toward the top indicates that jerasure is no longer supported and the source has been taken down21:09
timburke(all-in-all, it sounds like part of a patent-suit settlement)21:10
*** mattw4 has joined #openstack-meeting21:10
timburkeso i guess my main question is: do we go chasing forks/mirrors (which may share a similar fate), or stop supporting jerasure? or maybe just stop *testing* jerasure? but then it'll be difficult to tell when/whether we've broken support21:10
timburkei suppose that last one is the closest to our current support model for shss and libphazr... but i don't know that we'd even get reports of breakage, much less any assistance in resolving issues :-/21:10
rledisezstopping test does not seem good. I would vote in favor of deprecating it, but still supporting it through mirror for some time (1 year?)21:12
timburkedoes anyone have clusters running with jerasure? i know swiftstack would always go with isa-l...21:12
rledisezwhat are the other options instead if ISA-L to support the same EC schema?21:12
claygrledisez: that's pretty reasonable!21:12
rledisezwe run jerasure but i(ve been considering to move to isa-l recently21:12
kota_AFAIK, isa-l or shss for NTT groups21:13
claygrledisez: oh ouch - i remember when we looked at jerasure the lawsuit stuff turned us off 😬21:13
rledisezbut I'm thinking what about people running swift on non-x86 processor (does somebody do that?). can they run isa-l in replacement of jerasure?21:13
claygrledisez: I don't think isa-l is going to be "compatible" so much as it'd just be a different ec policy with a different scheme - you'd want to "support" jerasure forever (or re-encode all your data!)21:14
kota_I'd make sure if the liberasurecode_rs_vand is not effected by the GF-complete problem?21:14
rledisezclayg: i did basci test and it was working, but it's on my todo to run extensive testing on that21:14
claygrledisez: oh WOW - it'd be *amazing* if I was wrong about that21:15
rledisezclayg: just a basic test running pyeclib manualy, still need a lot of confirmation21:15
claygkota_: was the GF-complete the thing where decode would return bad data if you gave it specific combinations of frags?21:15
*** slaweq has joined #openstack-meeting21:15
timburkekota_, i know libec's built-in algo doesn't link against gf-complete -- though whether it would run into patent trouble is a separate issue...21:15
claygtimburke: ok, well 1) awesome find, i'm sure no one else was paying attention to gate tests for pyeclib and 2) does rledisez 's suggestion of "support" through ceph mirror with big WARNING WILL REMOVE somewhere in the changelog ASAP?21:16
timburkeclayg, no, the bad data thing was an isa-l bug21:16
kota_clayg: I don't think so. that problem was in isa-l rs_vand.21:16
claygtimburke: then hopefully rledisez can drive putting together a "how to not with the jerasure" guide that we can publish when we pull the plu21:16
timburkethat all sounds like a great plan :-)21:17
claygrledisez: and god speed on getting of jerasure 👍21:17
timburke(this, and the quadiron patches, reminds me how i rather wish we had some alternate plugin model that more-explicitly pushed the glue-code responsibility down to each driver...)21:17
rledisezwell, I hope my plan is gonna work then :D21:17
*** jamesmcarthur has joined #openstack-meeting21:18
claygrledisez: well you can be like "look upstream is removing support for jerasure" :P21:18
claygtimburke: yes plugins are so hard to do right 😞21:18
timburkeespecially in a language you're not super-familiar with21:18
*** ociuhandu has joined #openstack-meeting21:18
timburkeall right, i think i've got what i need out of that -- on to updates!21:19
timburke#topic waterfall EC21:19
*** openstack changes topic to "waterfall EC (Meeting topic: swift)"21:19
timburkeclayg, how's it looking?21:20
claygso I think my last update was two weeks ago - at that time I was like "yeah we can't just extend replicated concurrent gets; because the control is in the wrong place"21:20
claygso then I thought I'd just decouple EC get from database & replicated GETs then I'd be able to "make it so much simpler!!!"21:20
claygyeah that didn't work21:21
claygthe first thing I wanted to "rip out" was the "resuming stream feature"21:21
claygbasically I never liked it and don't have a clear picture of how often a chunkreadtimeout actually turns into a resume'd get - and even less so how often that WORKS - even for replicated!21:21
claygthen I started looking at how it fails in the EC case and was all like
claygSo i decided EC GET's don't NEED resuming behavior and THEN I can "make it so much simpler!!!21:23
claygbut yeah that didn't work21:23
claygI removed a couple hundred lines of resume code - but there's still like 400 lines of "multi-byte range" response handling code that is ALSO burried in the GETorHEADHandlerBase/ResumingGetter mess21:24
claygand I'm not sure I can convince myself EC GET's don't NEED multi-byte-range responses21:24
claygI mean... they probably don't - I think Sam just added it because he wanted too and no one stopped him... but I could be wrong, maybe someone wants it21:25
claygand since I don't really have a good reason to pull it off of replicated objects it seems like we're probably stuck with it on EC21:25
clayg^ that's actually up for debate I guess?21:25
claygtdasilva: seemed to think "well maybe we CAN drop it!?"21:25
*** ykatabam has quit IRC21:26
rledisezif it was broken I would say drop it, but I think it's working, and I can tell for sure that somebody somewhere in the world is using it. so changing the API, mmm…21:27
*** ociuhandu has quit IRC21:27
clayganyways - aside from maybe a little forward progress on the core EC GET request handling code and related tests I'm kinda back to square one 😞21:27
claygyup, that's my gut as well21:27
*** ykatabam has joined #openstack-meeting21:27
timburkei'm still wondering whether it might make things easier to reason about if we at least pulled the multi-range support out to middleware -- though i think SLO uses it, so ordering may be a little annoying...21:27
kota_IIRC, the multi range supports for EC is needed because a segment may belong to 2 fragments21:28
kota_due to the user range GET request.21:28
claygkota_: there IS some byte range translation for client requests - and you need that even for SINGLE range requests - but the ability for bytes=0-4,8-12 to turn into a MIME document isn't really dependent on the storage policy21:29
claygin FACT - we could *BUILD* multi-byte-range responses (the MIME responses) in middleware using ONLY single byte-range requests to the proxy21:29
claygstart a MIME response, fetch bytes 0-4 and send those, then fetch 8-12 and send those21:30
kota_ah, it should follow the storage policy. I don't think the translation is needed for the repliated one.21:30
claygthat actually seems like a MUCH better way to do multi-byte-range responses than what we have now (that threads mime handling all through the proxy and storage layer)21:30
claygright for multi-byte-range request to replicated data we just return the object server's MIME response (which is... idk, gross to me for some reason)21:31
timburkethere's going to be some corner cases we'd have to consider if we moved it to middleware -- a 416 on the first range may or may not mean we should 416 the whole request, for example21:31
clayglike I don't WANT my object servers to know how to make MIME responses - I think Sam just got a little crazy with multipart messages once he did that thing for EC PUT 🤷21:31
timburkeand *definitely* need to make sure we plumb in an If-Match header on subsequent requests21:31
*** ykatabam has quit IRC21:32
claygtimburke: yeah... if we decided to stop and say "ok, you can't have better backend EC request handling until you pull multi-part-byte requests to middleware" it'd be LONG haul21:32
timburkefwiw, AWS only supports a single range per request21:33
claygso realistically I guess I'll probably take another stab at pulling apart GETorHEADHandler somehow21:34
*** ykatabam has joined #openstack-meeting21:34
claygleave the resuming and multi-byte-range handling in place and extract the connection logic so it's either like dependency injection, or just subclasses21:35
claygmaybe ResummingGetter becomes BaseMultiRangeResumingGetter and GETorHEADHandler becomes ReplicatedGETorHEADHandler and some of ECObjectController._get_or_head_response goes into a new ECGETorHEADHandler that does all the Response Bucket stuff21:37
claygso, I guess that's the plan21:37
tdasilvajust to add a bit more about my idea of just dropping it. my reasoning was: 1. s3 doesn't support it (hence my assumption very few (if any) people care about it. 2. we can have build it in middleware. So my idea was "drop it" and if someone complains, add it to middleware21:37
clayg3rd times the charm!21:37
claygtdasilva: I didn't mean to throw you under the bus - FWIW I totally understood that line of reasoning and find it compelling21:38
tdasilvaif no one complains, less code for us to support.21:38
tdasilvaclayg: I gotcha, just wanted to provide some thoughts behind it, cause I honestly don't think it's a bad idea. but that's just my opinion...21:39
tdasilvawe could have the middleware ready21:39
claygalso having investigated how much work it'll be to make "waterfall-ec" mergable - it's entirely possible priorities may shift and this will be a slow burn rather than hard push21:39
claygrledisez: straw man - if we had a change that made EC demonstrably better, plus simpler code - but dropped multi-byte-range responses BUT in followup patch we reimplemented multi-byte-range as middleware ware ...21:40
*** jamesmcarthur has quit IRC21:40
claygcould we merge the first one w/o merging the second one until we need it? 😁21:41
*** ykatabam has quit IRC21:41
*** ociuhandu has joined #openstack-meeting21:42
rledisezclayg: well, that's a tough position for me. like I have to wait for a customer to complain, then we merge it. in the mean time, my customer says he will move to OTHER-CLOUD-PROVIDER because it didn't break his workflow21:42
rledisezmaybe I should add a timeseries to monitore if somebody use it21:42
claygrledisez: i guess it depends how much you want it out21:43
claygrledisez: and it sounds like you're probably justifyable saying "it's not causing ME any pain; please don't make pain for me" and that seems reasonable21:43
tdasilvaI this it's reasonable to think that over time we add cruft to the code base that over time is no longer used/needed. It's really hard (almost impossible) to find it, but I think we should make attempts as it would simplify the code, making it better21:43
clayglet me take one more stab at this with less code churn21:43
tdasilvas/I this/I think21:44
claygif I fail again I may come back and beg you to do some more qualification on multi-range responses21:44
timburketdasilva or i could start poking at multi-range-as-middleware if we get serious about going that route, anyway21:44
rledisezI guess we have some time to decide on this (if we need the middleware). I'll try to find out if somebody use multi-byte range on my clusters21:44
timburkesounds good. we oughta keep moving21:44
timburke#topic lots of small files21:45
*** openstack changes topic to "lots of small files (Meeting topic: swift)"21:45
timburkerledisez, i saw a merge from master!21:45
rledisezyep, I'll let alecuyer explain where he is now on losf21:45
alecuyerI've posted a list of the main changes planned so far, here21:45
alecuyerIf you have questions, go ahead, or I can put it on an etherpad if that's better21:45
alecuyerOtherwise, I haven't posted code yet, for lack of time these past few days, but also because of going back and forth and changing my mind about some things21:46
*** ociuhandu has quit IRC21:46
timburkeso does hashes.pkl get written in the volume, or somewhere else?21:46
alecuyerit's written in the same place as it is in the regular diskfile, currently21:47
*** zaitcev has joined #openstack-meeting21:47
alecuyerobject-X/partition - but , that could change to be below the "losf" directory21:47
timburkecool - i couldn't remember where we wrote it currently ;-)21:48
timburkei look forward to seeing the next few patches!21:48
rledisezright now the development is happening in our internal branch. how do you see the reconciliation with feature/losf?21:48
rledisezalecuyer: ^21:49
alecuyerwell I think I still need to do some testing, and once I get something that I think works, I'll try to split it in proper patches21:50
rledisezgreat. i guess we will try that future dev happen directly on feature branch :)21:51
timburke#topic CORS21:51
*** openstack changes topic to "CORS (Meeting topic: swift)"21:51
timburkep 712585 adds a cors gate job, and it even passes!21:51
patchbot - swift - Add gate job for CORS func tests - 11 patch sets21:51
timburkenext up i'll work on stacking the s3api changes on top of that, and getting the s3 tests in p 710354 distributed across the s3api patches so you can see what gets enabled by each patch21:52
patchbot - swift - Add CORS func tests for s3api - 3 patch sets21:52
timburkehas anyone tried running the new tests in p 533028? or even looked at them? i want to figure out whether this is even a palatable way to have func tests with an actual browser, or if i need to sort out something different21:52
patchbot - swift - Add some functional CORS tests - 8 patch sets21:52
timburkei saw that clayg has opinions :-)21:52
claygso on p 533028 - should all of the tests PASS?21:52
patchbot - swift - Add some functional CORS tests - 8 patch sets21:52
timburke(with the two patches that it's stacked on top of)21:53
timburkewell, pass or skip, anyway21:54
timburkeactually, maybe it's better to follow-up in -swift -- i wanted to leave time for21:55
timburke#topic open discussion21:55
*** openstack changes topic to "open discussion (Meeting topic: swift)"21:55
timburkeanything else for us to bring up?21:55
alecuyerI'm curious to know the proportion of HEAD requests you all get on your clusters. Do share if you can!21:56
alecuyer(I think I asked that once already actually ;) )21:56
rledisezso for us, 54% HEAD, 23% GET21:56
rledisezwe've been trying to evaluate the cost of HEAD (cost in I/O)21:57
rledisezit's not that easy21:57
claygI don't have that metric in aggregate offhand - I'll drop a note to try and sample some clusters we can look at21:58
timburkealecuyer, rledisez do you also have stats on user agents? i know python-swiftclient tends to be noisy with the HEADs...21:59
rledisezjust a note on the drop-md5 work, i uploaded a "working" patch (some tests still need to be fixed). if you're interrested you can look at it. on replication policy it increase the download speed like x3. let me find the link21:59
alecuyerrledisez: if you don't, I will look it at (user -agent) I don't have it now21:59
rlediseztimburke: I can check that21:59
rledisezor alecuyer will :)22:00
*** slaweq has quit IRC22:00
patchbotpatch 713059 - swift - WIP: Make the hashing algorithm configurable - 2 patch sets22:00
*** ykatabam has joined #openstack-meeting22:00
timburkei've got a snippet of logs from one of our clusters that's got like 300:9:1 for GET:HEAD:PUT, but it's a pretty short timespan iirc22:00
zaitcevholy cow, where do all these HEAD come from?22:00
alecuyerzaitcev:  my thoughts exactly22:00
seongsoocho80/15/5 for GET/HEAD/PUT22:01
*** david-lyle has joined #openstack-meeting22:01
alecuyerseongsoocho:  thanks22:01
*** ociuhandu has joined #openstack-meeting22:01
timburkeall right, we're at time22:02
timburkethank you all for coming, and thank you for working on swift!22:02
*** openstack changes topic to "OpenStack Meetings ||"22:02
openstackMeeting ended Wed Mar 18 22:02:07 2020 UTC.  Information about MeetBot at . (v 0.1.4)22:02
openstackMinutes (text):
*** patchbot has left #openstack-meeting22:02
*** zaitcev has left #openstack-meeting22:03
*** dklyle has quit IRC22:03
*** ykatabam has quit IRC22:05
*** ociuhandu has quit IRC22:06
*** david-lyle has quit IRC22:08
*** jamesmcarthur has joined #openstack-meeting22:20
*** nicolasbock has joined #openstack-meeting22:22
*** nicolasbock has quit IRC22:27
*** nicolasbock has joined #openstack-meeting22:28
*** jamesmcarthur has quit IRC22:37
*** ykatabam has joined #openstack-meeting22:38
*** ijw has quit IRC22:39
*** jamesmcarthur has joined #openstack-meeting22:39
*** ijw has joined #openstack-meeting22:40
*** TrevorV has quit IRC22:41
*** ykatabam has quit IRC22:42
*** ykatabam has joined #openstack-meeting22:43
*** rcernin has joined #openstack-meeting22:44
*** macz_ has quit IRC22:56
*** jamesmcarthur has quit IRC23:16
*** jamesmcarthur has joined #openstack-meeting23:20
*** jamesmcarthur has quit IRC23:21
*** jamesmcarthur has joined #openstack-meeting23:23
*** jamesmcarthur has quit IRC23:23
*** diablo_rojo has quit IRC23:29
*** diablo_rojo has joined #openstack-meeting23:31
*** jmasud has quit IRC23:31
*** jmasud has joined #openstack-meeting23:39
*** jamesmcarthur has joined #openstack-meeting23:53
*** verdurin has quit IRC23:54
*** verdurin has joined #openstack-meeting23:55

Generated by 2.15.3 by Marius Gedminas - find it at!