Wednesday, 2020-07-29

*** rcernin has quit IRC00:05
*** yasufum has joined #openstack-meeting00:26
*** rcernin has joined #openstack-meeting00:31
*** yasufum has quit IRC00:31
*** yasufum has joined #openstack-meeting00:38
*** zeestrat has quit IRC00:57
*** hillpd has quit IRC00:57
*** zeestrat has joined #openstack-meeting01:01
*** hillpd has joined #openstack-meeting01:01
*** gyee has quit IRC01:02
*** rfolco has quit IRC01:08
*** rfolco has joined #openstack-meeting01:15
*** ociuhandu has joined #openstack-meeting01:17
*** rfolco has quit IRC01:19
*** ociuhandu has quit IRC01:21
*** armax has quit IRC01:27
*** ociuhandu has joined #openstack-meeting01:43
*** ociuhandu has quit IRC01:48
*** rfolco has joined #openstack-meeting02:08
*** hyunsikyang has quit IRC02:12
*** ricolin has quit IRC02:15
*** yaawang has quit IRC02:21
*** ricolin has joined #openstack-meeting02:21
*** yaawang has joined #openstack-meeting02:22
*** moguimar_ has quit IRC02:26
*** rfolco has quit IRC02:28
*** yasufum has quit IRC02:45
*** ociuhandu has joined #openstack-meeting02:53
*** yasufum has joined #openstack-meeting02:56
*** ociuhandu has quit IRC02:57
*** hyunsikyang has joined #openstack-meeting02:58
*** diurnalist has joined #openstack-meeting02:59
*** diurnalist has quit IRC03:20
*** armax has joined #openstack-meeting03:28
*** yasufum has quit IRC03:28
*** psachin has joined #openstack-meeting03:37
*** armax has quit IRC03:43
*** psahoo has joined #openstack-meeting03:52
*** yasufum has joined #openstack-meeting03:59
*** rh-jelabarre has quit IRC04:32
*** rh-jlabarre has joined #openstack-meeting04:32
*** evrardjp has quit IRC04:33
*** rh-jlabarre has quit IRC04:33
*** evrardjp has joined #openstack-meeting04:33
*** rh-jlabarre has joined #openstack-meeting04:33
*** yasufum has quit IRC04:44
*** Lucas_Gray has quit IRC04:44
*** diurnalist has joined #openstack-meeting05:01
*** diurnalist has quit IRC05:06
*** yasufum has joined #openstack-meeting05:09
*** diurnalist has joined #openstack-meeting05:16
*** diurnalist has quit IRC05:22
*** tetsuro has joined #openstack-meeting05:25
*** tetsuro has quit IRC05:25
*** tetsuro has joined #openstack-meeting05:26
*** tetsuro has quit IRC05:27
*** yasufum has quit IRC05:41
*** slaweq has joined #openstack-meeting05:52
*** yaawang has quit IRC05:52
*** yaawang has joined #openstack-meeting05:53
*** ociuhandu has joined #openstack-meeting05:59
*** slaweq has quit IRC06:02
*** yasufum has joined #openstack-meeting06:04
*** masahito has joined #openstack-meeting06:05
*** diurnalist has joined #openstack-meeting06:08
*** diurnalist has quit IRC06:12
*** brtknr has quit IRC06:17
*** moguimar_ has joined #openstack-meeting06:30
*** vishalmanchanda has joined #openstack-meeting06:39
*** yaawang has quit IRC06:42
*** dklyle has quit IRC06:42
*** yaawang has joined #openstack-meeting06:42
*** slaweq has joined #openstack-meeting06:57
*** diurnalist has joined #openstack-meeting06:59
*** diurnalist has quit IRC07:04
*** apetrich has joined #openstack-meeting07:27
*** yonglihe has joined #openstack-meeting07:31
*** yasufum has quit IRC07:35
*** yasufum has joined #openstack-meeting07:39
*** tosky has joined #openstack-meeting07:39
*** ralonsoh has joined #openstack-meeting07:40
*** diurnalist has joined #openstack-meeting07:50
*** diurnalist has quit IRC07:55
*** evrardjp has quit IRC08:06
*** evrardjp has joined #openstack-meeting08:08
*** maciejjozefczyk has joined #openstack-meeting08:14
*** e0ne has joined #openstack-meeting08:24
*** diurnalist has joined #openstack-meeting08:41
*** brtknr has joined #openstack-meeting08:46
*** diurnalist has quit IRC08:46
*** ociuhandu has quit IRC08:56
*** masahito has quit IRC08:58
*** yasufum has quit IRC09:06
*** diurnalist has joined #openstack-meeting09:07
*** diurnalist has quit IRC09:12
*** rcernin has quit IRC09:12
*** Lucas_Gray has joined #openstack-meeting09:23
*** Lucas_Gray has quit IRC09:27
*** Lucas_Gray has joined #openstack-meeting09:30
*** moguimar_ is now known as moguimar09:32
*** ociuhandu has joined #openstack-meeting09:33
*** ociuhandu has quit IRC09:37
*** ociuhandu has joined #openstack-meeting09:42
*** diurnalist has joined #openstack-meeting09:58
*** diurnalist has quit IRC10:03
*** yaawang has quit IRC10:06
*** yaawang has joined #openstack-meeting10:07
*** psahoo has quit IRC10:44
*** diurnalist has joined #openstack-meeting10:49
*** ricolin has quit IRC10:52
*** ykatabam has quit IRC10:52
*** oneswig has joined #openstack-meeting10:53
*** diurnalist has quit IRC10:54
*** psahoo has joined #openstack-meeting10:56
*** bbezak has joined #openstack-meeting11:00
oneswig#startmeeting scientific-sig11:00
openstackMeeting started Wed Jul 29 11:00:53 2020 UTC and is due to finish in 60 minutes.  The chair is oneswig. Information about MeetBot at http://wiki.debian.org/MeetBot.11:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.11:00
*** openstack changes topic to " (Meeting topic: scientific-sig)"11:00
openstackThe meeting name has been set to 'scientific_sig'11:00
oneswigHi11:01
*** b1airo has joined #openstack-meeting11:03
oneswigI have nothing planned this week, alas11:03
oneswigbeen overfaced with other stuff11:03
b1airome either, but hi anyway :-)11:03
oneswighey b1airo, how's things?11:03
oneswig#chair b1airo11:03
openstackCurrent chairs: b1airo oneswig11:03
b1airo(i had a late game of water-bug polo so thought i'd drop in before bed)11:04
oneswigwater bug?11:04
b1airowas just watching a short talk on funcX11:04
b1airoha, yeah let me see if i can find a picture to explain...11:05
b1airo#link stuff.co.nz/the-press/sport/104951385/bug-polo-the-unknown-but-longstanding-kiwi-sport11:05
oneswigcool, but why the bug?11:10
b1airothe bugs were invented for whitewater sport use, kinda an alternative to whitewater kayaking. bug-polo is just an excuse to do something else with them11:11
oneswigAre you thinking of a talk for the virtual summit?11:12
*** ykatabam has joined #openstack-meeting11:13
b1airoi hadn't considered one so far, though thinking about it, we might have a new system design fixed enough by that point that we could talk about it. it wouldn't be anything to crazy though, so probably not worth a full talk anyway11:14
b1airoyou? must be a few new interesting tidbits from various engagements?11:15
oneswigI'd like to but no ideas as yet11:16
b1airoread the baremetal whitepaper last night - nice work B-)11:16
oneswigha, thanks - I only contributed to our case study, but they did a great job on the rest of the white paper I think11:18
oneswigb1airo: user stories always seem to go well I think, don't undersell it.11:19
oneswigShall we wrap up the meeting?  Don't want to keep you up for no other reason11:20
b1airotrue true, yeah probably time to hit the hay11:20
b1airoi was curious about how the cambridge crew and machines are going, but we could pick that up later11:20
oneswigSure, there's definitely a good story to tell there.11:21
oneswigOK b1airo, I'll catch you on Slack in due course.  Perhaps we can get someone to talk about the Cambridge user story in a meeting soon.11:21
oneswig#endmeeting11:21
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"11:21
openstackMeeting ended Wed Jul 29 11:21:51 2020 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)11:21
oneswigcheerio11:21
openstackMinutes:        http://eavesdrop.openstack.org/meetings/scientific_sig/2020/scientific_sig.2020-07-29-11.00.html11:21
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/scientific_sig/2020/scientific_sig.2020-07-29-11.00.txt11:21
openstackLog:            http://eavesdrop.openstack.org/meetings/scientific_sig/2020/scientific_sig.2020-07-29-11.00.log.html11:21
b1airocya11:21
*** psahoo has quit IRC11:37
*** diurnalist has joined #openstack-meeting11:40
*** markvoelker has joined #openstack-meeting11:40
*** raildo has joined #openstack-meeting11:44
*** diurnalist has quit IRC11:45
*** markvoelker has quit IRC11:47
*** rfolco has joined #openstack-meeting11:51
*** psahoo has joined #openstack-meeting11:52
*** moguimar has quit IRC12:20
*** moguimar has joined #openstack-meeting12:20
*** moguimar has quit IRC12:20
*** moguimar has joined #openstack-meeting12:21
*** diurnalist has joined #openstack-meeting12:31
*** diurnalist has quit IRC12:36
*** bbowen has quit IRC12:38
*** bbowen has joined #openstack-meeting12:39
*** lpetrut has joined #openstack-meeting12:44
*** moguimar has quit IRC12:45
*** moguimar has joined #openstack-meeting12:45
*** psahoo has quit IRC12:45
*** markvoelker has joined #openstack-meeting12:48
*** psahoo has joined #openstack-meeting12:48
*** diurnalist has joined #openstack-meeting13:01
*** diurnalist has quit IRC13:06
*** moguimar has quit IRC13:21
*** moguimar has joined #openstack-meeting13:23
*** sluna has quit IRC13:27
*** sluna has joined #openstack-meeting13:28
*** b1airo has quit IRC13:31
*** liuyulong has joined #openstack-meeting13:35
*** moguimar has quit IRC13:39
*** moguimar has joined #openstack-meeting13:41
*** TrevorV has joined #openstack-meeting13:50
liuyulong#startmeeting neutron_l314:00
openstackMeeting started Wed Jul 29 14:00:27 2020 UTC and is due to finish in 60 minutes.  The chair is liuyulong. Information about MeetBot at http://wiki.debian.org/MeetBot.14:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.14:00
*** openstack changes topic to " (Meeting topic: neutron_l3)"14:00
openstackThe meeting name has been set to 'neutron_l3'14:00
ralonsohhi14:01
liuyulongHi14:01
liuyulongOK, let's start.14:01
liuyulong#topic Announcements14:01
*** openstack changes topic to "Announcements (Meeting topic: neutron_l3)"14:01
liuyulong#link http://eavesdrop.openstack.org/#Neutron_L3_Sub-team_Meeting14:01
liuyulong#link https://review.opendev.org/#/c/741876/2/meetings/neutron-l3-sub-team-meeting.yaml14:01
patchbotpatch 741876 - opendev/irc-meetings - Change Neutron L3 Sub-team Meeting frequency (MERGED) - 2 patch sets14:01
liuyulongI changed the L3 meetign frequency last week.14:02
slaweqhi14:03
liuyulongWe will have it in every two weeks officially.14:03
liuyulongslaweq, hi14:03
* slaweq is updating calendar right now14:03
*** ZhuXiaoYu has joined #openstack-meeting14:05
* slaweq has calendar updated already :)14:05
liuyulongThe irc channel and the time slot are as usual, so it looks good for me now. : )14:05
liuyulongI should send a new email to the mail list. Will do that later.14:06
liuyulongOK, no more things from me now.14:06
liuyulongNext topic14:06
liuyulong#topic Bugs14:07
*** openstack changes topic to "Bugs (Meeting topic: neutron_l3)"14:07
liuyulong#link http://lists.openstack.org/pipermail/openstack-discuss/2020-July/016112.html14:07
liuyulong#link http://lists.openstack.org/pipermail/openstack-discuss/2020-July/016002.html14:07
liuyulongWe have 2 bug lists since we have bi-weekly meeting.14:08
liuyulongFirstly, two bugs related to IPv614:08
*** yaawang has quit IRC14:09
liuyulong#link https://bugs.launchpad.net/neutron/+bug/188825614:09
openstackLaunchpad bug 1888256 in neutron "Neutron start radvd and mess up the routing table when: ipv6_ra_mode=not set ipv6-address-mode=slaac" [Undecided,New]14:09
liuyulongThe use case is a bit strange to me, they start a shared network with public IPv4/v6 address.14:10
liuyulongThen directly create instance in it.14:10
*** yaawang has joined #openstack-meeting14:10
liuyulongIn order to achive the metadata HA, they choose HA router related data path.14:11
liuyulongSo IMO, this case should be changed like this:14:12
liuyulong1. stop using the neutron router, and move the gateway to the physical world (router).14:13
liuyulong2. metadata data-path can be achived by dhcp namespace, DHCP instance (namespace and related process) has the auto-reschedule mechanism for the dead agent.14:14
ralonsohis it possible to reschedule the metadata agent to use the DHCP instead of the router?14:15
slaweqralonsoh: I think You need to enable isolated metadata on dhcp agent's side14:16
slaweqralonsoh: but I'm not sure now exactly14:17
liuyulong3. let the physical router send the NA out14:17
liuyulongsorry, a bit bad network traffic...14:17
ralonsohthat's the point, I don't know if you can do this once the agent is running in the router14:18
liuyulongralonsoh, for the existing VMs, they may need to change the 169.254.169.254 route rule in the VM.14:18
ralonsohthat's done during the VM boot process14:18
slaweqbut my question is: what is exact purpose of "ipv6_ra_mode=None"?14:20
slaweqshould radvd be run in such case?14:20
ralonsohyes14:20
ralonsohno no, ipv6-address-mode=slaac and ipv6_ra_mode=not set14:21
ralonsohThe instance receives an IPv6 address from the external router (not managed by OpenStack Networking) using SLAAC.14:21
ralonsohhttps://access.redhat.com/documentation/en-us/red_hat_enterprise_linux_openstack_platform/7/html/networking_guide/sec-ipv614:21
slaweqralonsoh: so why we run radvd in such case?14:22
ralonsohno idea...14:22
slaweqif we wouldn't run it, there wouldn't be problem IMO14:22
slaweqas there wouldn't be any process on our side which would inject this bad default route14:22
liuyulongyes, no Neutron router should be added to this case14:22
ralonsohaccording to this document (similar to the OpenStack documentation)14:23
ralonsohwhen ipv6_ra_mode=not set, we don't use radvd14:23
slaweqralonsoh: so that seems for me like RC of this issue14:24
liuyulong#link https://bugs.launchpad.net/neutron/+bug/1888256/comments/714:24
openstackLaunchpad bug 1888256 in neutron "Neutron start radvd and mess up the routing table when: ipv6_ra_mode=not set ipv6-address-mode=slaac" [Undecided,New]14:24
liuyulongaccording to this comment, seems that "AdvSendAdvert on;" is always available.14:24
slaweqbut IMO we shouldn't have this section:14:25
liuyulongIt is hard code.14:25
slaweqinterface qr-a6d7ceab-8014:25
slaweq{14:25
liuyulong#link https://github.com/openstack/neutron/blob/8c80267bb6699c86e10aade13c54b715e1eae1bf/neutron/agent/linux/ra.py#L4114:25
slaweq   AdvSendAdvert on;14:25
slaweq   MinRtrAdvInterval 30;14:25
slaweq   MaxRtrAdvInterval 100;14:25
slaweq   AdvLinkMTU 1500;14:25
slaweq};14:25
slaweqin the radvd.conf at all if subnet is configured like that one in LP14:25
liuyulong#link https://github.com/openstack/neutron/blob/master/neutron/agent/linux/ra.py#L4114:26
liuyulong^ change to master14:26
slaweqso radvd shouldn't even listen on this interface14:26
*** rubasov has quit IRC14:28
liuyulongI don't know, maybe because it is the gateway.14:28
*** rubasov has joined #openstack-meeting14:28
liuyulongThis should be changed to as the action 1. "move the gateway to physical router."14:29
*** dklyle has joined #openstack-meeting14:29
liuyulongFrom my personal experiences, for such VLAN shared network, we topically did such work, the gateway will not be set on Neutron router. And there is no Neutron router.14:30
slaweqliuyulong: if You want I can take a look into that issue locally and will write a comment and/or propose patch14:31
liuyulongslaweq, great, go ahead, it's yours. : )14:32
slaweqthx14:32
liuyulong#link https://bugs.launchpad.net/neutron/+bug/188846414:32
openstackLaunchpad bug 1888464 in neutron "IPv6 PD with DVR does not assign correct snat sg address" [Undecided,New]14:32
liuyulongThis is the another IPv6 related bug.14:33
liuyulongAllow me to clarify the fact first, the sg-dev should have an IPv6 address from users internal subnet.14:34
liuyulongWhile the qg-dev should have an IPv6 address from external network.14:35
liuyulongSo from the bug description, we can only see there is an IPv6 subnet from the external network.14:35
ralonsohso sg-230491ca-5b should have an internal ipv6 address14:36
ralonsohis that correct?14:36
* slaweq is on another meeting but will be lurking here too14:36
liuyulongSo maybe this is not a bug, just because the user does not create the IPv6 subnet for there internal network.14:36
liuyulongIf the internal network has 2 subnet (v4 and v6), there should have two sg-devs.14:37
liuyulongSo, let me mark it as incomplete for now.14:39
*** ricolin has joined #openstack-meeting14:40
liuyulongralonsoh, IMO, the user was misunderstanding the network topology.14:40
*** mlavalle has joined #openstack-meeting14:40
ralonsohI still need to review this part of the code to be 100% sure14:40
liuyulongThe subnet uuid, router uuid, fip-network uuid does not match at all.14:40
*** Lucas_Gray has quit IRC14:41
ralonsohyou are right14:41
*** rcernin has joined #openstack-meeting14:41
liuyulongOK, no more bugs from me now.14:42
liuyulongAny updates?14:42
ralonsohnope14:42
liuyulongOK, let's move on.14:43
liuyulong#topic On demand agenda14:43
*** openstack changes topic to "On demand agenda (Meeting topic: neutron_l3)"14:43
liuyulong#link https://review.opendev.org/#/c/658511/14:43
patchbotpatch 658511 - neutron-specs - L3 agent self-service metering - 10 patch sets14:43
*** lpetrut has quit IRC14:43
liuyulongIMO, this was approved before in drivers meeting.14:44
liuyulongSo, need more attention. : )14:44
ralonsohok14:44
liuyulong#link https://review.opendev.org/#/c/675654/14:44
patchbotpatch 675654 - neutron - L3 agent metering extension - 7 patch sets14:44
liuyulongThis is the patch for it.14:44
liuyulongYes, it is still in-progress14:45
liuyulongI will continuesly work on this.14:45
slaweqliuyulong: I will try to review it this week14:45
liuyulongNext one should be this14:46
liuyulong#link https://review.opendev.org/#/c/728628/14:46
patchbotpatch 728628 - neutron-specs - L3 router support ndp proxy - 17 patch sets14:46
liuyulongAfter mine request, I see the author uploaded the POC to gerrit.14:46
liuyulong#link https://review.opendev.org/#/c/743142/14:47
patchbotpatch 743142 - neutron - [WIP][PoC][Server Side] L3 router support ndp proxy - 2 patch sets14:47
liuyulongIt is all in one, IMO14:47
liuyulongThe patch is including the agent side change.14:47
liuyulongThis is good for us to understand the real proposal.14:48
liuyulongAnd the details about the DVR, HA can be found in that as well.14:48
liuyulong#link https://review.opendev.org/#/c/729532/14:49
patchbotpatch 729532 - neutron-specs - L3 router support ecmp - 28 patch sets14:49
liuyulongSame work for this one, it also has a POC.14:49
liuyulong#link https://review.opendev.org/#/c/743661/14:49
patchbotpatch 743661 - neutron - L3 router support ECMP - 1 patch set14:49
ralonsohbut that wasn't approved14:49
*** Lucas_Gray has joined #openstack-meeting14:49
liuyulongNo?14:50
liuyulongI missed that.14:50
ralonsohhttps://bugs.launchpad.net/neutron/+bug/1880532 is not approved yet14:50
openstackLaunchpad bug 1880532 in neutron "[RFE]L3 Router should support ECMP" [Wishlist,New] - Assigned to XiaoYu Zhu (honglan0914)14:50
*** lbragstad_ has joined #openstack-meeting14:51
liuyulongBy the way, thing POC has no control plane related code, just some route related implementation.14:51
*** irclogbot_1 has quit IRC14:52
*** rcernin_ has joined #openstack-meeting14:52
*** dklyle has quit IRC14:53
liuyulongSo the reuse of the exsiting route API will be the control method.14:53
liuyulongLooks good to me, it will be small and clear.14:54
liuyulongLast one14:55
liuyulong#link https://review.opendev.org/#/q/topic:ovn/port_forwarding14:55
liuyulongOVN port forwarding is good in progress.14:55
ralonsohyes, Flavio is working on this full time14:56
*** rcernin_ has quit IRC14:56
liuyulongIf anyone is interested, pleae go ahead and do code review.14:57
liuyulongLet's narrow down the gaps. : )14:57
liuyulongOK, no more things from me now.14:57
*** rcernin has quit IRC14:58
ZhuXiaoYuhttps://review.opendev.org/#/c/743661/14:58
ZhuXiaoYuHere is the patch14:58
*** lbragstad has quit IRC14:58
patchbotpatch 743661 - neutron - L3 router support ECMP - 1 patch set14:58
liuyulongZhuXiaoYu, hi, I've mentioned that. : )14:59
*** irclogbot_0 has joined #openstack-meeting14:59
ZhuXiaoYu:)14:59
liuyulongZhuXiaoYu, by the way, it will be better if you add some test cases.14:59
ZhuXiaoYuOK, I will.15:00
liuyulongAlright, time is up.15:00
liuyulongBye15:00
ralonsohbye15:00
liuyulong#endmeeting15:00
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"15:00
openstackMeeting ended Wed Jul 29 15:00:45 2020 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)15:00
openstackMinutes:        http://eavesdrop.openstack.org/meetings/neutron_l3/2020/neutron_l3.2020-07-29-14.00.html15:00
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/neutron_l3/2020/neutron_l3.2020-07-29-14.00.txt15:00
openstackLog:            http://eavesdrop.openstack.org/meetings/neutron_l3/2020/neutron_l3.2020-07-29-14.00.log.html15:00
*** diurnalist has joined #openstack-meeting15:01
*** liuyulong has quit IRC15:01
*** dklyle has joined #openstack-meeting15:01
*** lajoskatona has joined #openstack-meeting15:10
*** lajoskatona has left #openstack-meeting15:11
*** toabctl has quit IRC15:32
*** toabctl has joined #openstack-meeting15:33
*** psahoo has quit IRC15:36
*** armax has joined #openstack-meeting15:37
*** ykatabam has quit IRC16:02
*** markvoelker has quit IRC16:13
*** ociuhandu_ has joined #openstack-meeting16:21
*** ociuhandu has quit IRC16:23
*** ociuhandu_ has quit IRC16:27
*** gyee has joined #openstack-meeting16:28
*** ociuhandu has joined #openstack-meeting16:35
*** armstrong has joined #openstack-meeting16:38
*** ociuhandu has quit IRC16:41
*** mlavalle has quit IRC16:46
*** mlavalle has joined #openstack-meeting16:48
*** psachin has quit IRC16:53
*** Lucas_Gray has quit IRC16:54
*** ricolin has quit IRC17:01
*** armax has quit IRC17:19
*** armax has joined #openstack-meeting17:19
*** dmacpher_ has joined #openstack-meeting17:21
*** dmacpher has quit IRC17:25
*** maciejjozefczyk has quit IRC18:24
*** vishalmanchanda has quit IRC18:29
*** armstrong has quit IRC18:48
*** ralonsoh has quit IRC18:53
*** lbragstad_ is now known as lbragstad19:01
*** ZhuXiaoYu has quit IRC19:06
*** bbowen has quit IRC19:46
*** moguimar_ has joined #openstack-meeting19:55
*** moguimar has quit IRC19:55
*** moguimar_ has quit IRC20:03
*** moguimar_ has joined #openstack-meeting20:04
*** yaawang has quit IRC20:07
*** yaawang has joined #openstack-meeting20:07
*** ykatabam has joined #openstack-meeting20:17
*** moguimar_ has quit IRC20:34
*** ociuhandu has joined #openstack-meeting20:39
*** ociuhandu has quit IRC20:43
*** zaitcev has joined #openstack-meeting20:57
timburke#startmeeting swift21:03
openstackMeeting started Wed Jul 29 21:03:18 2020 UTC and is due to finish in 60 minutes.  The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot.21:03
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.21:03
*** openstack changes topic to " (Meeting topic: swift)"21:03
openstackThe meeting name has been set to 'swift'21:03
timburkewho's here for the swift meeting?21:03
mattoliverauo/21:03
seongsoochoo/21:03
kota_o/21:04
rledisezhi o/21:04
timburkeclayg, zaitcev, tdasilva?21:05
claygi'm here!21:05
claygthanks21:05
timburkeagenda's at https://wiki.openstack.org/wiki/Meetings/Swift21:06
timburkefirst, a couple announcements21:06
timburke#topic ptg21:06
*** openstack changes topic to "ptg (Meeting topic: swift)"21:06
zaitcevIt's a storm here but I'm here thus far.21:06
timburkethe next ptg will be all-online, like the last one21:06
timburkeand there's a poll up for *when* exactly it should be21:06
timburke#link http://lists.openstack.org/pipermail/openstack-discuss/2020-July/016098.html21:07
claygoh, thanks21:07
*** slaweq has quit IRC21:07
zaitcevMaybe we ought to run our own hackathon in person. Of course, masks, hand sanitizer all around.21:07
claygsmall pods21:08
timburkezaitcev, that is *so* tempting for me21:08
mattoliverauTho that only works for peeps in the same country21:08
timburkeand even then, it'll be harder than usual to get employer buy-in21:09
timburkeso i think the plan for now should be: do a virtual ptg again21:10
timburkethough i fully acknowledge that there's something lost in doing it that way21:10
timburke#topic London OpenInfra virtual meetup21:11
*** openstack changes topic to "London OpenInfra virtual meetup (Meeting topic: swift)"21:11
timburkethere's a meetup thing going on tomorrow! seems like it might be worth checking out if anyone has time21:11
timburke#link http://lists.openstack.org/pipermail/openstack-discuss/2020-July/016109.html21:11
*** oneswig has quit IRC21:12
timburkethat's all i've got for announcements... on to more swifty things21:13
timburke#topic py3 crypto bug21:13
*** openstack changes topic to "py3 crypto bug (Meeting topic: swift)"21:13
timburke#link https://launchpad.net/bugs/188803721:13
openstackLaunchpad bug 1888037 in OpenStack Object Storage (swift) "Encryption writes different paths for key_id on py2 and py3" [High,In progress]21:13
timburkecurrently, there's an availability issue when upgrading from swift-on-py2 to swift-on-py321:14
zaitcev\u00f0\u009f\u008c\u00b4 is a WSGI string, isn't it21:14
timburkespecifically, any data that was encrypted for a path that included any non-ascii characters will cause 500s21:14
timburkezaitcev, yup :-(21:15
timburkegood news is, we've got a fix: https://review.opendev.org/#/c/742033/21:15
patchbotpatch 742033 - swift - py3: Work with proper native string paths in crypt... - 3 patch sets21:15
timburkebad news is, it currently causes a rolling upgrade issue (because it bumps the version number stored in crypto metadata, so anything written by an upgraded proxy won't be able to be read by an old proxy)21:16
*** TrevorV has quit IRC21:16
claygjoin the debate!  https://etherpad.opendev.org/p/crypto-meta-version-321:17
timburkewe've done this once before... for https://launchpad.net/bugs/1813725 about a year and a half ago21:17
openstackLaunchpad bug 1813725 in OpenStack Object Storage (swift) "Object encryption key truncated sometimes when used with Barbican" [Medium,Confirmed]21:17
timburkebut that doesn't mean it was a good thing21:17
mattoliverauwow, it's like py2 and py3 strings bite us again. All when you think it's safe :P21:18
claygyeah the py2 works with py2 and py3 works with py3 thing in v2 is annoying21:19
timburkei also did a patch to add an option to continue doing things the old way(s): https://review.opendev.org/#/c/742756/21:19
patchbotpatch 742756 - swift - crypto: Add config option to support rolling upgrades - 3 patch sets21:19
rledisezis it not an other argument for checkpoint release?21:20
claygbut the problem of a py2 upgrading to py3 and expecting it in the py3-v2 format is still bad21:20
timburkebut i'm realizing that even that won't work well for a rolling py2->py3 upgrade21:20
claygrledisez: I don't think we have ever done anything we'd call a checkpoint release21:20
rledisezclayg: my point is maybe we should. it would help us a lot when managing legacy code & co. we also know at some point we can drop old code21:21
rledisez(when it's not about data written on disk, of course)21:21
claygi agree, having a checkpoint release process that's robust and available would be super userful21:21
claygI think we've mostly juts done like a 4 year deprecation cycle or.. like "never" also works21:22
timburkeheh -- reminds me of https://review.opendev.org/#/c/736787/21:23
patchbotpatch 736787 - swift - Rip out pickle support in our memcached client - 1 patch set21:23
zaitcevThat's different. It only takes 1 reboot to make sure that old format is no more. But drives can be around for 10 years.21:24
timburkei meant more the 'or.. like "never" also works' ;-)21:25
rledisezzaitcev: in this situation, we could "force" the operator to upgrade to a version that support both v2 and v3 format, before letting him move to a version that use v3 by default21:25
claygtimburke: 🤣21:26
*** raildo has quit IRC21:26
claygwe probably *could* drop "reading the pickle format from memcache" support - see!  eventually it'll seem obvious that we don't have to write v2!21:27
timburkeso, back tot he topic at hand: how can we make sure a rolling upgrade is still successful? i guess, more config options that we hope operators never actually need to use?21:28
timburkewhat should the defaults be?21:28
claygrledisez: you betcha!  a legit checkpoint!  there's just the code needed to support the mechanics and the process needed to make sure all the clusters we care about upgrade like we want21:28
rlediseztimburke: default should be v2. The operator knows when he upgraded all the cluster and so decides to switch to v3 when he's ready. but we also have to assume that some won't do it and will get bitten one day when v3 become the default21:29
claygI'm 100% sure continuing to write in the current format is the correct thing todo for current clusters - py2 clusters could even skip the latin-1 shiz; py3 tho won't know if it has py2/v2 or py3/v2 so it'll have to do the extra work21:29
claygrledisez: as long as they have code that can READ v3 we can start writing it21:30
claygrledisez: so... like 2 years or whatever... it'll be fine21:30
timburkerledisez, *which* v2? the v2 that py2 would've written down (which is essentially identical to v3), or the v2 that py3 would've written down?21:30
claygthe real suck is if we have a v4 by then - then we have to think about upgrading from something can't read v3 😞21:30
claygtimburke: v2 will have to continue to be different on py2 vs py3 until after they've upgraded AFICT21:31
timburkeand actually -- i kinda feel like it's worth thinking about whether the default *should be* v121:31
claygtimburke: I think v1 had a BUG tho - loss of information?  and we've made it past the upgrade already?  why do we want to go back?21:32
timburkeclayg, so to do a py2->py3 transition, you need to go old-swift-on-py2 -> new-swift-on-py2 -> new-swift-on-py3?21:32
claygi think old-swift-on-py2 to new-swift-on-py3 should be fine as long as they both know how to read the right formats (e.g. current swift py3 can't read old-swift-py2 format)21:33
*** bbowen has joined #openstack-meeting21:33
claygbut once we've cut new code such that new-swift-py3 can read old-swift-py2 format (even if that old-swift-py2 is still writing v2); we should be fine?21:34
timburkeclayg, you and i have, certainly. how many clusters are still out there from rocky or earlier?21:34
claygok, so there's clusters still writing v1 that haven't upgraded to a swift that's now writing v2 so their yet-to-be-done rolling-upgrade WILL cry when old proxies read v2/v3 for the first time21:35
timburkeclayg, the way the patches are currently, new-swift-on-py3 won't be able to write a v2 that old-swift-on-py2 could read21:35
claygso this is sort of the 'v4 requires code that knows how to read v3 to upgrade" problem... but earlier21:36
claygtimburke: but a new-swift-on-py3 *COULD* write a *v3* that new-swift-on-py2 could read so maybe there's a "min swift version prior to upgrade to py3"21:37
claygunless you already have... in which case "thanks for helping us find all these bugs!"21:37
timburkei've got this nagging feeling like that version's going to be ever-increasing until we drop support for py2...21:38
claygtimburke: 🤗21:38
claygseongsoocho: rledisez: anyone else want to try and jump in?  Any questions about the bug report, the wip patch, or the etherpad21:39
rlediseznah, i agree with last comment from timburke (so maybe there's a "min swift version prior to upgrade to py3")21:40
timburkeshould i squash the two patches together, so there's a nice spot to write a fairly complete UpgradeImpact?21:41
claygwe could try a "checkpoint process" that's mostly convention w/o code to enforce it21:41
claygbut i'm not going to exercise it; cause as soon as we upgrade we'll turn on v3 so if the default changes later it won't effect us.21:42
seongsoochoI'm still try to reproduce this bug in my dev cluster .  :-(21:42
claygyou have to write py2 crypto - then read it py3 to see the bug really21:43
timburkeseongsoocho, so you're in a fairly unique position (to my knowledge): your cluster's been py3 from the beginning, right?21:43
claygpy2/v2 is unicode from utf8 - py3/v2 is unicode from latin-121:44
seongsoochotimburke:   I have 3 cluster and the 3rd cluster is py3 from the beginning.21:45
timburkeah, got it. thanks; good to know21:45
timburkedo you run with encryption enabled?21:45
claygoh neat!  so you'll definitely want this fixed so you can upgrade the other clusters to py3!!!  💪21:45
claygoh... good question... why did I think seongsoocho's cluster was encrypted 🤔21:46
seongsoochoyes .. actually my cluster doesn't use encrypted. but now i'm build a new cluster (support object encryption) with py3.21:47
seongsoochoRather than encrypting object on the server, I'm telling our customer to encrypt it on the client.21:49
claygseongsoocho: excellent recommendation21:49
timburkealways a good stance to take :-)21:49
seongsoochoBut, new cluster support server-side object encryption..21:49
timburkeok, i'm running out of steam on this. i'm still not sure what i should be coding21:52
claygv2 default so we can upgrade w/o having to push a config first!21:52
claygafter we upgrade we'll turn on v3 and we don't have to think about this dumb problem for a little while 🤣21:52
timburkeand release note that you need to upgrade to swift 2.26.0 (or whatever) before switching from py2 to py3?21:53
timburkei can do that. nobody else worried about the v1 upgrade issue?21:53
timburkei guess not ;-)21:55
timburkeok, last few minutes21:55
timburke#topic open discussion21:55
*** openstack changes topic to "open discussion (Meeting topic: swift)"21:55
claygupgrade *and* be writing v3 I think - unless new-py2 will be able to read the old-py3-default 🤔21:55
timburkeanything else to bring up?21:55
claygI wanted to tell people I'm trying to simply the config for concurrent gets with ec!21:55
clayghttps://review.opendev.org/#/c/737096/21:55
patchbotpatch 737096 - swift - Add concurrent_ec_extra_requests - 4 patch sets21:55
seongsoochohttps://bugs.launchpad.net/swift/+bug/1889386   <- Does anyone know about this bug(??)??21:56
openstackLaunchpad bug 1889386 in OpenStack Object Storage (swift) "[ s3 api ] The CreationDate of listing bucket always return '2009-02-03T16:45:09.000Z'" [Undecided,New]21:56
zaitcevto simply? Was it "to simplify" perchance?21:56
claygnow that it doesn't have all that "per replica" crap I'm thinking about squishing the concurrent_ec_extra_requests option into p 71134221:56
patchbothttps://review.opendev.org/#/c/711342/ - swift - Add concurrent_gets to EC GET requests - 14 patch sets21:56
timburkeclayg, yeah, new-py2 should be fine to deal with old-py321:56
claygtimburke: well, people should still write v3 😁21:56
timburkeoh, for sure21:57
claygzaitcev: simplify!  yes, thank you!21:57
timburkeseongsoocho, yeah... that's a sad confusing mess, isn't it...21:58
claygdo the account db's tables not have any kind of date?  modified or something?21:58
seongsoochotimburke:  yes... I have no idea to fix this problem.21:58
timburkei think we could address it -- add a new key to the json responses the account server sends back, have s3api look for that21:58
claygseongsoocho: i'd be onboard with a schema update that includes pushing new info up to the listings from the container-updater21:59
clayganything that makes us more like s3 is helpful!  (it's a big patch tho, I don't know how to cheat)21:59
timburkeno one's felt enough pressure yet to get it fixed. but i'd be happy to review any patches for it!21:59
clayg❤️21:59
claygtimburke: is it a datamodel change too - or just api?22:00
timburkei *think* mostly just api. pretty sure we've already got the create date in the db table, though i ought to double-check22:00
claygzaitcev:  I think the concurrent_ec_extra_requests option will allow for most of the interesting things we might have tried with per-replica-timeouts and will be much easier to configure22:01
claygoh... shoot - that's time!22:01
timburkei put it up there with ?partNumber=<N> support -- it shouldn't actually be too bad... mostly just a matter of prioritization22:01
timburkeso it is22:01
timburkethank you all for coming, and thank you for working on swift!22:01
timburke#endmeeting22:01
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"22:01
openstackMeeting ended Wed Jul 29 22:01:49 2020 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)22:01
openstackMinutes:        http://eavesdrop.openstack.org/meetings/swift/2020/swift.2020-07-29-21.03.html22:01
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/swift/2020/swift.2020-07-29-21.03.txt22:01
openstackLog:            http://eavesdrop.openstack.org/meetings/swift/2020/swift.2020-07-29-21.03.log.html22:01
*** zaitcev has left #openstack-meeting22:01
*** e0ne has quit IRC22:16
*** e0ne has joined #openstack-meeting22:19
*** e0ne has quit IRC22:20
*** e0ne has joined #openstack-meeting22:31
*** rcernin_ has joined #openstack-meeting22:36
*** e0ne_ has joined #openstack-meeting22:37
*** e0ne has quit IRC22:37
*** e0ne_ has quit IRC22:37
*** rcernin_ has quit IRC22:48
*** rcernin has joined #openstack-meeting22:48
*** ociuhandu has joined #openstack-meeting23:01
*** ociuhandu has quit IRC23:06
*** tosky has quit IRC23:21
*** diurnalist has quit IRC23:59

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!