Wednesday, 2019-11-13

*** brinzhang has quit IRC00:01
*** martial has quit IRC00:09
*** ociuhandu has joined #openstack-meeting00:13
*** ociuhandu has quit IRC00:18
*** yamamoto has joined #openstack-meeting00:32
*** macz has quit IRC00:33
*** yamamoto has quit IRC00:41
*** yamamoto has joined #openstack-meeting00:46
*** rbudden has quit IRC00:48
*** brinzhang has joined #openstack-meeting00:53
*** brinzhang_ has quit IRC00:57
*** armax has quit IRC00:58
*** Liang__ has joined #openstack-meeting01:01
*** brinzhang_ has joined #openstack-meeting01:02
*** brinzhang has quit IRC01:05
*** brinzhang has joined #openstack-meeting01:08
*** brinzhang has quit IRC01:10
*** brinzhang has joined #openstack-meeting01:10
*** ociuhandu has joined #openstack-meeting01:16
*** ociuhandu has quit IRC01:24
*** yamamoto has quit IRC01:30
*** ociuhandu has joined #openstack-meeting01:36
*** jamesmcarthur has joined #openstack-meeting01:39
*** yamamoto has joined #openstack-meeting01:39
*** gyee has quit IRC01:41
*** ociuhandu has quit IRC01:46
*** yamamoto has quit IRC01:48
*** macz has joined #openstack-meeting02:01
*** macz has quit IRC02:06
*** armax has joined #openstack-meeting02:08
*** jamesmcarthur has quit IRC02:13
*** armax has quit IRC02:17
*** armax has joined #openstack-meeting02:18
*** armax has quit IRC02:23
*** brault has quit IRC02:32
*** brault has joined #openstack-meeting02:32
*** yamamoto has joined #openstack-meeting02:34
*** mhen has quit IRC02:38
*** mhen has joined #openstack-meeting02:38
*** jamesmcarthur has joined #openstack-meeting02:40
*** jamesmcarthur has quit IRC02:43
*** yamamoto has quit IRC02:46
*** jamesmcarthur has joined #openstack-meeting02:54
*** brinzhang has quit IRC02:59
*** brinzhang_ has quit IRC03:00
*** ociuhandu has joined #openstack-meeting03:17
*** ociuhandu has quit IRC03:22
*** liuyulong has quit IRC03:46
*** zhubx has quit IRC03:59
*** boxiang has joined #openstack-meeting03:59
*** jamesmcarthur has quit IRC04:11
*** Lucas_Gray has joined #openstack-meeting04:11
*** yaawang has quit IRC04:15
*** yaawang has joined #openstack-meeting04:16
*** yamamoto has joined #openstack-meeting04:24
*** ociuhandu has joined #openstack-meeting04:43
*** ociuhandu has quit IRC04:48
*** e0ne has joined #openstack-meeting04:49
*** e0ne_ has joined #openstack-meeting05:02
*** e0ne has quit IRC05:03
*** Lucas_Gray has quit IRC05:28
*** yamamoto has quit IRC05:30
*** ociuhandu has joined #openstack-meeting05:31
*** ociuhandu has quit IRC05:35
*** yamamoto has joined #openstack-meeting05:42
*** e0ne_ has quit IRC06:00
*** Luzi has joined #openstack-meeting06:40
*** slaweq has quit IRC06:45
*** sridharg has joined #openstack-meeting06:46
*** yamamoto has quit IRC06:47
*** yamamoto has joined #openstack-meeting06:47
*** yamamoto_ has joined #openstack-meeting06:50
*** yamamoto has quit IRC06:52
*** dklyle has quit IRC07:19
*** dklyle has joined #openstack-meeting07:19
*** igordc has quit IRC07:25
*** ociuhandu has joined #openstack-meeting07:35
*** ociuhandu has quit IRC07:41
*** yamamoto_ has quit IRC07:52
*** yamamoto has joined #openstack-meeting07:55
*** mmethot has quit IRC07:56
*** trident has quit IRC07:57
*** tesseract has joined #openstack-meeting08:00
*** e0ne has joined #openstack-meeting08:03
*** e0ne has quit IRC08:05
*** trident has joined #openstack-meeting08:06
*** rpittau|afk is now known as rpittau08:17
*** yaawang has quit IRC08:24
*** yaawang has joined #openstack-meeting08:24
*** ralonsoh has joined #openstack-meeting08:29
*** links has joined #openstack-meeting08:40
*** Lucas_Gray has joined #openstack-meeting08:50
*** slaweq has joined #openstack-meeting08:52
*** armax has joined #openstack-meeting09:02
*** ociuhandu has joined #openstack-meeting09:03
*** armax has quit IRC09:06
*** ociuhandu has quit IRC09:09
*** ociuhandu has joined #openstack-meeting09:10
*** Lucas_Gray has quit IRC09:13
*** yamamoto has quit IRC09:18
*** yamamoto has joined #openstack-meeting09:19
*** yamamoto has quit IRC09:21
*** ociuhandu has quit IRC09:25
*** ociuhandu has joined #openstack-meeting09:26
*** Lucas_Gray has joined #openstack-meeting09:26
*** ociuhandu has quit IRC09:29
*** ociuhandu has joined #openstack-meeting09:30
*** tssurya has joined #openstack-meeting09:30
*** ociuhandu has quit IRC09:31
*** ociuhandu has joined #openstack-meeting09:31
*** ykatabam has quit IRC09:33
*** ociuhandu has quit IRC09:37
*** ociuhandu has joined #openstack-meeting09:37
*** e0ne has joined #openstack-meeting09:43
*** Liang__ has quit IRC09:52
*** priteau has joined #openstack-meeting10:05
*** gibi is now known as gibi_off10:11
*** ociuhandu has quit IRC10:13
*** ociuhandu has joined #openstack-meeting10:14
*** rsimai_away is now known as rsimai10:14
*** ociuhandu has quit IRC10:19
*** zbr has quit IRC10:21
*** ianychoi has quit IRC10:24
*** brinzhang has joined #openstack-meeting10:24
*** whoami-rajat__ has joined #openstack-meeting10:27
*** brinzhang_ has joined #openstack-meeting10:29
*** brinzhang has quit IRC10:31
*** brinzhang has joined #openstack-meeting10:42
*** brinzhang_ has quit IRC10:45
*** yamamoto has joined #openstack-meeting10:46
*** zbr has joined #openstack-meeting10:47
*** yamamoto has quit IRC10:50
*** ralonsoh has quit IRC11:07
*** jamesmcarthur has joined #openstack-meeting11:09
*** ralonsoh has joined #openstack-meeting11:09
*** jamesmcarthur has quit IRC11:10
*** ociuhandu has joined #openstack-meeting11:14
*** ociuhandu has quit IRC11:18
*** yamamoto has joined #openstack-meeting11:22
*** brinzhang_ has joined #openstack-meeting11:29
*** yamamoto has quit IRC11:29
*** brinzhang has quit IRC11:32
*** ociuhandu has joined #openstack-meeting11:36
*** rfolco has joined #openstack-meeting11:56
*** yamamoto has joined #openstack-meeting11:59
*** kaisers has quit IRC12:02
*** kaisers has joined #openstack-meeting12:05
*** raildo has joined #openstack-meeting12:06
*** electrofelix has joined #openstack-meeting12:12
*** priteau has quit IRC12:19
*** priteau has joined #openstack-meeting12:21
*** brinzhang has joined #openstack-meeting12:25
*** yamamoto has quit IRC12:26
*** Lucas_Gray has quit IRC12:28
*** brinzhang_ has quit IRC12:28
*** ociuhandu has quit IRC12:35
*** ociuhandu has joined #openstack-meeting12:35
*** ociuhandu has quit IRC12:36
*** ociuhandu has joined #openstack-meeting12:37
*** yamamoto has joined #openstack-meeting12:38
*** yamamoto has quit IRC12:38
*** yamamoto has joined #openstack-meeting12:38
*** yamamoto has quit IRC12:44
*** yamamoto has joined #openstack-meeting12:44
*** brinzhang_ has joined #openstack-meeting12:53
*** rcernin has quit IRC12:54
*** brinzhang has quit IRC12:55
*** liuyulong has joined #openstack-meeting12:57
*** ianychoi has joined #openstack-meeting13:00
*** yamamoto has quit IRC13:05
*** ociuhandu has quit IRC13:08
*** ociuhandu has joined #openstack-meeting13:09
*** brinzhang has joined #openstack-meeting13:09
*** brinzhang has quit IRC13:10
*** brinzhang_ has quit IRC13:11
*** ociuhandu has quit IRC13:14
*** jamesmcarthur has joined #openstack-meeting13:16
*** jamesmcarthur has quit IRC13:32
*** jamesmcarthur has joined #openstack-meeting13:33
*** yamamoto has joined #openstack-meeting13:34
*** jamesmcarthur has quit IRC13:35
*** mriedem has joined #openstack-meeting13:53
*** jamesmcarthur has joined #openstack-meeting13:53
*** davee_ has joined #openstack-meeting13:55
*** eharney has joined #openstack-meeting13:56
*** mmethot has joined #openstack-meeting13:57
*** liuyulong_ has joined #openstack-meeting13:57
liuyulong#startmeeting neutron_l314:02
openstackMeeting started Wed Nov 13 14:02:24 2019 UTC and is due to finish in 60 minutes.  The chair is liuyulong. Information about MeetBot at
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.14:02
*** openstack changes topic to " (Meeting topic: neutron_l3)"14:02
openstackThe meeting name has been set to 'neutron_l3'14:02
liuyulong#chair slaweq14:02
openstackCurrent chairs: liuyulong slaweq14:02
liuyulong#chair haleyb14:02
openstackCurrent chairs: haleyb liuyulong slaweq14:02
liuyulong#chair liuyulong_14:02
openstackCurrent chairs: haleyb liuyulong liuyulong_ slaweq14:02
liuyulongGood to see you guys online here.14:03
slaweqyou too :)14:04
liuyulongWe have a memorable summit in Shanghai last week.14:04
liuyulongAnd thanks to slaweq for the summary:
njohnstonIt was great seeing you liuyulong_, wish I could have been there!14:05
liuyulongnjohnston, we had online conference with you, : )14:06
liuyulong#topic Announcements14:06
*** openstack changes topic to "Announcements (Meeting topic: neutron_l3)"14:06
liuyulongI have no announcement, since we have no team meeting after the summit.14:07
liuyulongSo, maybe the blueprint should be mentioned here:14:08
*** yamamoto has quit IRC14:09
liuyulongThey did not move to U milestone。14:09
liuyulongNo bugs and blueprints here.14:10
*** yamamoto has joined #openstack-meeting14:10
*** yamamoto has quit IRC14:10
*** yamamoto has joined #openstack-meeting14:11
liuyulongAny updates?14:11
liuyulongUssuri 2020-05-13 estimated (schedule)14:13
*** dviroel has joined #openstack-meeting14:13
liuyulongOK, let's move on14:13
liuyulong#topic Bugs14:13
*** openstack changes topic to "Bugs (Meeting topic: neutron_l3)"14:13
liuyulongRyan was our bug deputy last week, thanks.14:14
liuyulongFirst one:14:14
openstackLaunchpad bug 1851659 in neutron "removing a network from a DHCP agent removes L3 rules even if it shouldn't" [High,New]14:14
*** yamamoto has quit IRC14:15
liuyulongThis looks a bit confusing to me, the 'rules' is policy route rule, dvr related flow?14:16
liuyulongDue to the service port, for this bug it is DHCP port (instance)14:17
liuyulongthe dvr local router will be hosted to this scheduled host, if the compute node is running with DHCP agent, the compute port and DHCP port can both make the local router installation happen.14:19
liuyulongSo, if remove DHCP related port can cause the problem, that may point out that two compute port are hosted under same DVR router in one same host,14:20
liuyulongRemove one VM will cause another VM traffic down? I don't think so.14:21
liuyulongSo let's test and investigate in details.14:21
openstackLaunchpad bug 1851194 in neutron "FWaaSv2 configures iptables with invalid port name" [Medium,New]14:22
liuyulongYes, this is known issue, same to
openstackLaunchpad bug 1798577 in neutron "[FWaas-DVR]wrong port name in iptables rules" [Medium,In progress] - Assigned to Wang Weijia (wangweij)14:23
liuyulongThis is the fix:
liuyulongSurely it's worthy to fix, so let's start the review.14:25
liuyulongNext one:14:25
openstackLaunchpad bug 1851609 in neutron "Add an option for graceful l3 agent shutdown" [Medium,In progress] - Assigned to Oleg Bondarev (obondarev)14:25
slaweqthis one I would like You (L3 team) to triage first and than we can discuss that on drivers meeting14:26
liuyulongActually I'm fine with a new config option for varied deployments.14:26
liuyulongOf course with default value False to not break the existing cloud.14:27
liuyulongBut according to the bug description, it the pod goes down, then everything is gone. (Maybe not, I'm not familiar with kubernetes...)14:30
*** links has quit IRC14:30
liuyulongOr the real story is the the l3-agent is running in container but the child processes are all in the host. Then everything remains after pod down.14:31
liuyulongIn this point of view, the proposal makes sense to me.14:33
liuyulongBut for sure, we need the bug reporter to give us more details.14:34
liuyulongOleg Bondarev is not here, I will add some concern to the bug.14:35
liuyulongslaweq, so let's continue the discussion in LP first.14:37
slaweqliuyulong: thx a lot14:38
slaweqsounds like a good plan :)14:38
liuyulongLooks like I'm now in a single person meeting, because I receive your message delay a lot...14:38
slaweqI have also some urgent d/s stuff to do14:39
slaweqso I'm doing two things in same time14:39
liuyulongNo, I mean my network connection to freenode...14:39
slaweqahh, ok14:39
liuyulongOK, let's scan the bug list.14:40
openstackLaunchpad bug 1850864 in neutron "DHCP agent takes very long time to report when port is provisioned" [Medium,In progress] - Assigned to Slawek Kaplonski (slaweq)14:41
liuyulongLooks like another bad background of DHCP agent...14:41
liuyulongI will raise the distributed DHCP based on openflow for ovs-agent sometime... I hope I have enough time to...14:42
slaweqI sent patch for that already14:43
slaweqplease review it if You have some time14:43
*** ociuhandu has joined #openstack-meeting14:44
liuyulongSure, my colleague may want to reproduce and test it, I will add the updates.14:45
openstackLaunchpad bug 1850818 in neutron "[RFE][floatingip port_forwarding] Add description field" [Undecided,In progress] - Assigned to Pedro Henrique Pereira Martins (pedrohpmartins)14:45
liuyulongIMO, this makes sense to me14:45
liuyulongAnd I've reviewed some of the patch sets.14:45
liuyulongAnd I may also want the author to add the name for the port forwarding.14:46
slaweqyes, that makes sense for me too14:46
liuyulongSometimes, the name can be used for sorting, searching and classifying14:46
slaweqbut as this is api change, we need to discuss that on drivers meeting also14:47
liuyulongdescription maybe to long for us.14:47
liuyulongslaweq, sure, please add to the drivers list. : )14:47
liuyulongOK, last one14:48
openstackLaunchpad bug 1850779 in neutron "[L3] snat-ns will be initialized twice for DVR+HA routers during agent restart" [Medium,In progress] - Assigned to LIU Yulong (dragon889)14:48
liuyulongWe met this issue recently during a l3-agent upgrading14:48
liuyulongThere are too many routers in one agent, the restart time is really long.14:48
liuyulongSo we investigate the time spreading, this is one point.14:49
liuyulongMany external commands will be executed in the namespace create function.14:50
*** yamamoto has joined #openstack-meeting14:51
liuyulongIf you run the agent with rootwrap, it spawns child processes, this is a bit expensive if there are tons of resources to process.14:52
*** Luzi has quit IRC14:52
*** tesseract has quit IRC14:52
liuyulongThe fix is here:
liuyulongOK, we are running out of time.14:53
liuyulongNext topic14:53
liuyulong#topic On demand agenda14:53
*** openstack changes topic to "On demand agenda (Meeting topic: neutron_l3)"14:54
liuyulongI have updated some of spec for the topic raised during the PTG Shanghai.14:54
*** boxiang has quit IRC14:54
*** tssurya has quit IRC14:54
*** boxiang has joined #openstack-meeting14:55
*** tesseract has joined #openstack-meeting14:55
liuyulongSo comments are always welcomed.14:55
*** yamamoto has quit IRC14:55
liuyulong_Alright, let's end here.14:57
liuyulong_Thanks for attending...14:57
*** openstack changes topic to "OpenStack Meetings ||"14:57
openstackMeeting ended Wed Nov 13 14:57:30 2019 UTC.  Information about MeetBot at . (v 0.1.4)14:57
openstackMinutes (text):
*** liuyulong_ has quit IRC14:57
slaweqthx liuyulong for driving this meeting :)14:58
*** lpetrut has joined #openstack-meeting14:58
*** ociuhandu has quit IRC15:04
*** whoami-rajat__ has quit IRC15:04
*** armax has joined #openstack-meeting15:05
*** ociuhandu has joined #openstack-meeting15:06
*** armax has quit IRC15:10
*** ociuhandu has quit IRC15:11
*** liuyulong has quit IRC15:11
*** jamesmcarthur has quit IRC15:31
*** armax has joined #openstack-meeting15:36
*** whoami-rajat has joined #openstack-meeting15:45
*** tosky has joined #openstack-meeting15:46
*** rosmaita has joined #openstack-meeting15:46
*** m5z has joined #openstack-meeting15:56
rosmaitajungleboyj whoami-rajat rajinir lseki carloss pots woojay erlon geguileo eharney rosmaita enriquetaso e0ne smcginnis davidsha walshh_ xyang hemna _hemna tosky sfernand15:59
rosmaitacinder meeting reminder ^^15:59
rosmaita#startmeeting cinder16:00
openstackMeeting started Wed Nov 13 16:00:13 2019 UTC and is due to finish in 60 minutes.  The chair is rosmaita. Information about MeetBot at
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:00
*** openstack changes topic to " (Meeting topic: cinder)"16:00
openstackThe meeting name has been set to 'cinder'16:00
geguileohi! o/16:00
rosmaitalooking like a good turnout16:00
rosmaitavery light agenda today16:00
*** woojay has joined #openstack-meeting16:01
rosmaitaok, first point of business: if you like the courtesy meeting ping, please add your name to the list at the top of the etherpad16:02
*** davidsha has joined #openstack-meeting16:02
rosmaitai will wait a minute so you can do it now16:02
rosmaitaok, next business is some polls16:03
rosmaitaquick question -- i used google forms -- will people in China be able to access them?16:04
rosmaitai didn't think of the accessibility issue until i sent out the info to the ML16:04
geguileorosmaita: most likely no16:04
lsekiprobably not16:05
rosmaitathanks lseki16:05
rosmaitaok, i will have to put together doodle or some other service that works16:05
rosmaitai'll combine the results from the google forms, so don't wait to fill out the polls16:06
rosmaitawhich are:16:06
rosmaitameeting time change:
rosmaitaif we do change, it will happen the first meeting in December16:07
rosmaitaso next week, will be same time as today16:07
lsekimanila folks are using this service for the virtual ptg poll: https://xoyondo.com16:07
rosmaitalseki: thanks, i will take a look at that16:07
*** lpetrut has quit IRC16:08
rosmaitaour virtual ptg poll is
smcginnisKeep in mind that that Thursday is a US holiday.16:08
rosmaitathat one closes next Tuesday so we'll know the results by Wednesday's meeting16:08
rosmaitasmcginnis: i did not keep that in mind!16:08
rosmaitaall USA people can vote "cannot attend" though16:09
smcginnisYep. Maybe it works well for many.16:09
rosmaitafor the virtual ptg, I'd like to do wednesday and one other day, most likely tuesday16:09
rosmaitabut you have choices on the poll16:09
rosmaitaok, so to be clear: if those polls are accessible to you, go ahead and use them before you forget to vote16:10
rosmaitai will send out a follow-up with another link and combine the results16:10
rosmaitaas far as what to do at the virtual ptg -- some follow up on PTG topics and new topics by people who weren't able to attend16:11
*** rsimai is now known as rsimai_away16:12
rosmaitawe'll probably use the same etherpad16:12
rosmaita#action rosmaita download a copy of current PTG etherpad16:12
rosmaitaok, final announcement16:12
rosmaitaI revised the cinder-core list16:13
rosmaitabut no surprises, just removed some inactive people16:13
rosmaitathat leaves us with 8 active cores, same as for Train16:13
rosmaitai think that's all from me16:14
rosmaitathere's nothing else on the agenda, so ...16:15
rosmaita#topic open discussion16:15
*** openstack changes topic to "open discussion (Meeting topic: cinder)"16:15
eharneyi'll point out that has me wondering what the deal is with the gate16:16
eharneyi've been too lazy to really figure it out yet, but, slightly concerning16:16
rosmaitais that the 2 char change patch?16:16
rosmaitathe gate has definitely seemed more unstable that usual16:16
rosmaitai think a lot of problems during the setup phase of tempest/grenade16:17
smcginnisI know there were some requirements conflicts that have been resolved.16:17
*** gyee has joined #openstack-meeting16:18
eharneyi'll dig into it further once i have a chance16:18
smcginnisSo far today things have seemed a little more stable, so maybe one more recheck will do the trick.16:18
rosmaitai'll try to keep an eye on things today16:18
toskyit's a test failing; grenade itself (the upgrade) terminates successfully16:19
rosmaitatosky: ok16:19
eharneyi saw a handful of cases where nova failed to schedule an instance, unclear why16:19
toskyunfortunately it's the less descriptive openstack error ever, "'No valid host was found. There are not enough hosts available.'"16:19
toskywhich can mean anything (resource issues?)16:20
toskyleast* descriptive16:20
rosmaitathat does sound like a resource issue16:20
*** ociuhandu has joined #openstack-meeting16:20
rosmaitaanything else?16:22
davee_nothing here16:23
rosmaitai'm planning to get a preliminary summary of what happened at the PTG together today or tomorrow, so we can use that as the basis for what to discuss at the virtual PTG16:23
rosmaitai will send out an announcement to the ML, so watch for that16:24
rosmaitaplease fill out the polls (if you can access them) for meeting time and Virtual PTG dates16:24
whoami-rajatThanks Brian for the clean polls16:25
rosmaitai'll get china-accessible polls out right after the meeting16:25
rosmaitawhoami-rajat: ty, i tried to make them clear16:25
rosmaitaok, i guess that's it for today -- i'm off to, i guess16:26
smcginnisThanks rosmaita!16:26
rosmaitai guess i don't need to remind people to do reviews :)16:27
rosmaitawith your 1/2 hour free time!16:27
rosmaitathanks everyone, see you next week16:27
*** openstack changes topic to "OpenStack Meetings ||"16:27
openstackMeeting ended Wed Nov 13 16:27:41 2019 UTC.  Information about MeetBot at . (v 0.1.4)16:27
openstackMinutes (text):
*** woojay has left #openstack-meeting16:27
*** m5z has left #openstack-meeting16:28
*** tosky has left #openstack-meeting16:28
*** yamamoto has joined #openstack-meeting16:28
*** ociuhandu has quit IRC16:30
*** ociuhandu has joined #openstack-meeting16:31
*** yamamoto has quit IRC16:32
*** ociuhandu has quit IRC16:44
*** ociuhandu has joined #openstack-meeting17:01
*** igordc has joined #openstack-meeting17:02
*** rpittau is now known as rpittau|afk17:08
*** apetrich has quit IRC17:24
*** igordc has quit IRC17:32
*** igordc has joined #openstack-meeting17:35
*** diablo_rojo__ has joined #openstack-meeting17:42
*** davidsha has quit IRC17:44
*** dtrainor has quit IRC17:58
*** yamamoto has joined #openstack-meeting18:17
*** ociuhandu has quit IRC18:17
*** ociuhandu has joined #openstack-meeting18:18
*** sridharg has quit IRC18:20
*** yamamoto has quit IRC18:21
*** ociuhandu has quit IRC18:23
*** slaweq has quit IRC18:25
*** priteau has quit IRC18:32
*** tesseract has quit IRC18:35
*** dtrainor has joined #openstack-meeting18:39
*** raildo has quit IRC18:55
*** jamesmcarthur has joined #openstack-meeting19:00
*** electrofelix has quit IRC19:06
*** jamesmcarthur has quit IRC19:10
*** e0ne has quit IRC19:10
*** ralonsoh has quit IRC19:25
*** dtrainor has quit IRC19:32
*** yamamoto has joined #openstack-meeting19:36
*** rosmaita has left #openstack-meeting19:38
*** yamamoto has quit IRC19:41
*** ociuhandu has joined #openstack-meeting19:50
*** jamesmcarthur has joined #openstack-meeting19:50
*** jamesmcarthur has quit IRC19:51
*** jamesmcarthur has joined #openstack-meeting19:52
*** jamesmcarthur has quit IRC19:53
*** ociuhandu has quit IRC19:56
*** ociuhandu has joined #openstack-meeting19:59
*** slaweq has joined #openstack-meeting20:01
*** diablo_rojo__ has quit IRC20:05
*** ociuhandu has quit IRC20:06
*** yamamoto has joined #openstack-meeting20:11
*** e0ne has joined #openstack-meeting20:14
*** e0ne has quit IRC20:15
*** yamamoto has quit IRC20:15
*** ociuhandu has joined #openstack-meeting20:16
*** dklyle has quit IRC20:20
*** dklyle has joined #openstack-meeting20:23
*** ociuhandu has quit IRC20:25
*** jessegler has joined #openstack-meeting20:27
*** whoami-rajat has quit IRC20:34
*** e0ne has joined #openstack-meeting20:42
*** rbudden has joined #openstack-meeting20:47
*** dtrainor has joined #openstack-meeting20:47
*** jessegler has quit IRC20:50
*** e0ne has quit IRC20:59
*** ociuhandu has joined #openstack-meeting21:00
*** alecuyer has joined #openstack-meeting21:00
*** zaitcev has joined #openstack-meeting21:02
timburke#startmeeting swift21:04
openstackMeeting started Wed Nov 13 21:04:16 2019 UTC and is due to finish in 60 minutes.  The chair is timburke. Information about MeetBot at
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.21:04
*** openstack changes topic to " (Meeting topic: swift)"21:04
openstackThe meeting name has been set to 'swift'21:04
timburkewho's here for the swift meeting?21:04
timburkesorry to run a bit late; was upgrading the OS on my laptop21:05
claygalecuyer: rledisez: I feel like I *just* saw you guys!?21:05
alecuyerchinese flashback!21:05
timburkeon that note...21:05
timburke#topic PTG recap21:05
*** openstack changes topic to "PTG recap (Meeting topic: swift)"21:05
rledisezclayg: did you travel in time when flighing home? it may be the explanation ;)21:06
*** ociuhandu has quit IRC21:06
timburkeas expected, it was wonderful seeing everyone who made it :-)21:06
claygrledisez: so funny story, I started my travel day by going to the wrong airport - but... i made it home21:06
timburkeeveryone else, hopefully i'll get to see you in vancouver21:06
timburkeclayg, oh jeeze! glad you still made it home!21:07
claygi hear vancouver has a bunch of chinese millionares21:07
timburkewe had some really good discussions, hopefully we more or less kept the etherpad updated21:08
* rledisez wish he was a canadian millionares in china21:08
timburkewe met a new Korean operator -- if you see seongsoocho in -swift, you ought to say hi :D21:09
timburkehad a good ops feedback session21:09
mattoliveraunice :)21:09
claygit's 6am in seoul21:09
claygi'm sure kota_ and mattoliverau wouldn't complain if we wanted to make the meeting a little later 🤷‍♂️21:09
mattoliverau8am here now, so isn't too bad. But its 6 in Tokyo isn't it kota_?21:10
mattoliveraudaylight savings for the win here21:10
tdasilvawhen is ptg in vancouver?21:11
timburkei want to say june? let me find the email...21:11
kota_mattoliverau: yup21:11
kota_same with Tokyo timezone21:11
rledisezyep, 8 to 11 june I think21:11
mattoliverauSeeing as I'm not in the cloud team anymore at Suse I probably wont have any travel funding to go. I guess I could attempt to get a talk in, maybe if it gets accepted they'll send me. I'd have to talk to my new manager.21:12
rledisezI'm sure it ends the june, 11th21:12
timburke says Jun 8-1121:12
timburkemattoliverau, there's also the travel support program:
timburkei should make sure seongsoocho knows about it, too21:13
mattoliverauthat's true, I have used it before :)21:13
timburkebig takeaways i got out of the week (and feel free to chime in with corrections or additional info):21:13
claygtdasilva: aren't you going to New Zealand at some point?  Can you pick up matt on your way back to the states?21:14
tdasilvaheh, sounds like a good idea21:14
timburkeon LOSF: the main cluster that rledisez and alecuyer needed this for is getting phased out, so its future is up in the air a bit21:14
tdasilvaor maybe you all come down to join us21:14
rlediseztimburke: a bit of precision, it will happens in about 12 to 18 months, so we still get a bit of time on it ;)21:15
mattoliverau^ that :)21:15
timburkewe know there are a bunch more tests that we'd like to see, but we're also not sure we want to take on the maintenance burden when we might be able to get a lot of benefit out of things like xfs's realtime device support21:15
claygrledisez: well, but even given the runway on the phase out - aren't you also being tasked with planning for the NEW cluster (i.e. benchmarking alternatives to existing LOF index & slab storage)21:16
rledisezright now, we are still investigating all possibilities:21:17
timburkewe know drives are only going to get bigger over time, though -- and we suspect that lots-of-small-files as a problem is going to start to look like lots-of-files21:17
rledisezxfs realtime => I asked the status on XFS ML, to see if it maintained/tested/…21:17
rledisezzfs => looks a nice possibility21:17
*** priteau has joined #openstack-meeting21:17
rledisezLOSF/LOF => still in course21:17
rledisezopen-cas => does not seem stable, but is maintained so we may talk to them21:17
timburkerledisez, alecuyer was saying that zfs doesn't have good recovery tooling, though, yeah?21:18
*** priteau has quit IRC21:18
rledisezI think it was about what happen in case of an I/O error. the only option might be to reboot the server21:18
rledisezso we need to check that, right21:18
alecuyerI started to work on eBPF scripts to monitor block device IO and link it to inode/xattr access, vs file data. Good going on SAIO, but not working on our prod (need diff kernel options). Once I have something good i'll share so everyone can check if they'd benefit from xattr stored or cached on a fast device (allowing for the use of larger HDDs for data)21:18
tdasilvaanyone ever look how hard it would be to use something like bluestore?21:19
rlediseztdasilva: does it allow to place the rocksdb on a different device ? (SSD/NVme)21:19
*** priteau has joined #openstack-meeting21:19
tdasilvarledisez: why?21:20
mattoliveraualecuyer: yeah21:20
rledisezthe goal is place filesystem metadata (inode/xattr) on a faster device (or the LOSF index)21:20
mattoliverauyou can decide if you want the rocks db and wal on a seperate device.21:21
rledisezmattoliverau: so yes, that's something that could be investigated yes, but it's pretty much the same solution than LOSF, so I wouls stick to LOSF for now as it is designed for swift especially21:21
timburkeon versioning: people seemed enthusiastic. null namespace didn't seem to scare anyone off, and swift growing another versioning scheme seems like a good idea given how poorly the current one maps to s3 versioning. iirc, everyone wants s3 versioning21:22
timburkeso, clayg, tdasilva, and i will be working on that a lot, hopefully getting it ready to merge to master within the next few weeks21:23
timburkeon "atomic large objects": we recognize the utility, but still aren't sure about how to implement it. had a couple discussions but no clear resolution -- will probably come up again the next time we meet in person21:24
*** priteau has quit IRC21:25
claygyes bluestore was very much aimed at putting the index/metadata db on a seperate device from the blob slab21:26
claygyou can also put it on the same device, but it's not quite as awesome as filestore21:26
timburkeon the object updater, i had an idea about grouping async pendings by container and sending UPDATE requests to do them all at once21:26
timburkewill almost certainly need some benchmarking before we know whether its actually a *good* idea21:26
mattoliverauoh cool21:26
mattoliverauwith the new UPDATE that makes sense.21:27
claygeitherway it's VERY ceph sepcific - I couldn't find any documentation for a ABI or something that allows general access to volume21:27
timburkeon recon dumps: it'd be nice to get more/better keys, but where we *really* want to go involves tracking what the oldest unprocessed work item is21:27
mattoliverauclayg: yeah I've struggled to find decent info on it too21:27
timburkelike, replicator should be able to track when partition have successfully synced with all primaries, expose the one that needs to sync most, and prioritize that work21:29
timburkein the mean time, getting missing keys into recon could be a good short-term win21:29
timburkeon tiering... we didn't actually talk much about it. sorry mattoliverau. fwiw, i know that we have customers wanting that sort of behavior, though, and i think that the null namespace could be very useful for the implementation21:31
mattoliverauyeah, I thought so too. if null namespace is the future, we shouhld hold off so we use it.21:31
timburkein particular, it'd be useful in combination with versioning -- so you could tier off the non-active versions to somewhere cheaper21:32
mattoliverauspeaking of recon, I wrote this ages ago.. need to see if it still is correct and everything, it's a little old:
timburkei think it'd be great to keep in mind as we think about implementing some of s3's bucket policies (in particular, deleting non-active versions older than X)21:33
zaitcevDude. I once tried to write a Ceph client in Python. The current one at the time spawned a thread for each RADOS request, which called into C++. But it was impossible with the lack of docs, and the code was fairly impregnable. You may be able to reverse-engineer Bluestore API, but it's not going to be easy, I can guarantee that much.21:33
timburkemattoliverau, oh, nice! yeah, that does seem useful21:33
claygzaitcev: i bet alecuyer could do it :nerd_snipe:21:35
timburkeon performance: looks good, clayg brought some nice history to the discussion that made us all feel a lot better about getting rid of the queues21:35
alecuyerclayg: all that talk certainly got me curious ;)21:35
timburkeworking its way through the gate now21:35
claygEAT IT GATE21:35
claygtimburke: rledisez: i was unclear on if dropping the q significantly helped throughput - or it just mostly reduced cpu?21:36
rledisezclayg: both actually21:36
rledisezI'll submit soon a patch to get rid of the with Chunk*Timeout21:36
timburkeclayg, says just no-queue brought like 45% better throughput21:36
mattoliverauheres my dodgy benchmark results from a SAIO:
rledisezafter that, I found some other place that could provide some perf improvment (especially a place in the proxy that does a cache that is reseted at every request :))21:37
mattoliverauwhich was just a look with ssbench and getput21:37
mattoliveraurledisez: oh that sounds like a useful cache :P21:37
rledisezand after that, I want to get rid if MD5 as a checksum algorithm (not as placement algorithm)21:38
claygok then!21:39
timburkerledisez, i wonder if it'd actually be easier to get rid of it as a placement algo... or at least, make the choice of algo a property of the ring21:39
rlediseztimburke: yeah, but I'm not sure yet the gain is worth it21:39
rledisezwhile as checksum, he, the double md5 calculation in EC is really killing perf21:40
rledisezbut it's gonna be hard to drop MD5 because I think it's part of API (etag header)21:40
timburkeyup, that was my thought, too :-(21:40
timburkestill, if EC only had to do one MD5 and one (HW-optimized, yeah?) SHA-256 or SHA-512... might be a solid win21:41
timburkesimilar with encryption21:41
rledisezyeah, that was my though too21:41
timburkeon swiftclient test directory layout: yeah, just make it consistent with swift. merged.21:42
rledisezi'm thinking of adler32 maybe, which is designed especially for that (and 4 times faster than md5 on my bench server)21:42
*** rfolco has quit IRC21:43
timburkei *think* that about covers the PTG... am i forgetting anything?21:44
claygi hope not!  I gotta go async21:44
timburkethere were some interesting talks from cmurphy about keystone scoping and access rules -- seems like they might fit well with swift21:45
timburkeand i know people are interested in having keystone application credential support in swiftclient21:46
mattoliverauoh really, I might have to go look those up. cmurphy is awesome21:46
timburkeall right, i think that's all i've got21:47
* cmurphy blushes21:47
timburke#topic open discussion21:47
*** openstack changes topic to "open discussion (Meeting topic: swift)"21:47
alecuyerI'll just leave the link here, anyone has ideas about using larger drives, please add to it:
timburkeall right. thank you all for coming, and thank you for working on swift!21:50
*** openstack changes topic to "OpenStack Meetings ||"21:50
openstackMeeting ended Wed Nov 13 21:50:37 2019 UTC.  Information about MeetBot at . (v 0.1.4)21:50
openstackMinutes (text):
*** dviroel has quit IRC21:52
*** eharney has quit IRC21:54
*** zaitcev has left #openstack-meeting21:54
*** yamamoto has joined #openstack-meeting22:05
*** slaweq has quit IRC22:06
*** yamamoto has quit IRC22:10
*** yamamoto has joined #openstack-meeting22:15
*** slaweq has joined #openstack-meeting22:23
*** slaweq has quit IRC22:28
*** rcernin has joined #openstack-meeting22:32
*** ykatabam has joined #openstack-meeting23:16
*** yamamoto has quit IRC23:20
*** igordc has quit IRC23:26
*** ociuhandu has joined #openstack-meeting23:31
*** diablo_rojo__ has joined #openstack-meeting23:33
*** ociuhandu has quit IRC23:35
*** brault has quit IRC23:59

Generated by 2.15.3 by Marius Gedminas - find it at!