Wednesday, 2020-03-25

*** ijw has joined #openstack-meeting00:01
*** jamesmcarthur has joined #openstack-meeting00:01
*** jamesmcarthur has quit IRC00:04
*** jamesmcarthur has joined #openstack-meeting00:04
*** ijw_ has quit IRC00:04
*** tetsuro has joined #openstack-meeting00:08
*** ociuhandu has quit IRC00:13
*** ociuhandu has joined #openstack-meeting00:14
*** ociuhandu has quit IRC00:18
*** tetsuro has quit IRC00:19
*** tetsuro has joined #openstack-meeting00:21
*** bbowen has joined #openstack-meeting00:27
*** jamesmcarthur has quit IRC00:29
*** jamesmcarthur has joined #openstack-meeting00:32
*** larainema has joined #openstack-meeting00:56
*** jamesmcarthur has quit IRC01:02
*** jamesmcarthur has joined #openstack-meeting01:06
*** Liang__ has joined #openstack-meeting01:13
*** macz_ has joined #openstack-meeting01:24
*** liuyulong has quit IRC01:28
*** macz_ has quit IRC01:28
*** masahito has joined #openstack-meeting01:55
*** jamesmcarthur has quit IRC02:01
*** ykatabam has quit IRC02:10
*** macz_ has joined #openstack-meeting02:17
*** ykatabam has joined #openstack-meeting02:20
*** gyee has quit IRC02:21
*** macz_ has quit IRC02:21
*** ianw has quit IRC02:30
*** ianw has joined #openstack-meeting02:36
*** ianw has quit IRC02:37
*** jamesmcarthur has joined #openstack-meeting02:39
*** ianw has joined #openstack-meeting02:40
*** jamesmcarthur has quit IRC02:46
*** jamesmcarthur has joined #openstack-meeting03:02
*** jamesmcarthur has quit IRC03:06
*** jamesmcarthur has joined #openstack-meeting03:08
*** apetrich has quit IRC03:09
*** ysandeep|away is now known as ysandeep03:12
*** jamesmcarthur has quit IRC03:14
*** psachin has joined #openstack-meeting03:32
*** ociuhandu has joined #openstack-meeting03:34
*** ociuhandu has quit IRC03:38
*** jamesmcarthur has joined #openstack-meeting03:38
*** jamesmcarthur has quit IRC03:47
*** dmsimard|off has joined #openstack-meeting04:18
*** mahatic has joined #openstack-meeting04:36
*** jamesmcarthur has joined #openstack-meeting04:41
*** jamesmcarthur has quit IRC04:54
*** vishalmanchanda has joined #openstack-meeting05:03
*** jamesmcarthur has joined #openstack-meeting05:05
*** jamesmcarthur has quit IRC05:07
*** jamesmcarthur has joined #openstack-meeting05:11
*** masahito has quit IRC05:15
*** ysandeep is now known as ysandeep|05:28
*** links has joined #openstack-meeting05:29
*** ysandeep has joined #openstack-meeting05:30
*** ysandeep is now known as san05:30
*** ysandeep| is now known as ysandeep05:31
*** san is now known as Guest6833305:31
*** Guest68333 has left #openstack-meeting05:31
*** jamesmcarthur has quit IRC05:33
*** ociuhandu has joined #openstack-meeting05:36
*** ociuhandu has quit IRC05:41
*** dklyle has quit IRC05:50
*** masahito has joined #openstack-meeting05:51
*** macz_ has joined #openstack-meeting05:53
*** masahito has quit IRC05:56
*** macz_ has quit IRC05:58
*** jmasud has quit IRC06:06
*** jmasud has joined #openstack-meeting06:07
*** masahito has joined #openstack-meeting06:18
*** tetsuro has quit IRC07:13
*** JangwonLee has quit IRC07:13
*** JangwonLee has joined #openstack-meeting07:14
*** vesper11 has quit IRC07:16
*** vesper has joined #openstack-meeting07:16
*** apetrich has joined #openstack-meeting07:27
*** njohnston has quit IRC07:45
*** belmoreira has joined #openstack-meeting07:46
*** njohnston has joined #openstack-meeting07:48
*** maciejjozefczyk has joined #openstack-meeting07:52
*** e0ne has joined #openstack-meeting07:56
*** slaweq has joined #openstack-meeting07:59
*** ralonsoh has joined #openstack-meeting08:01
*** ykatabam has quit IRC08:09
*** ykatabam has joined #openstack-meeting08:18
*** stephenfin has quit IRC08:19
*** tetsuro has joined #openstack-meeting08:26
*** stephenfin has joined #openstack-meeting08:30
*** rpittau|afk is now known as rpittau08:31
*** slaweq has quit IRC08:41
*** raymonddeng has joined #openstack-meeting08:48
raymonddenghi is the automation meeting going to start here in 10 or am i horribly mistaken08:49
raymonddeng** large-scale meeting08:52
*** mdelavergne has joined #openstack-meeting08:53
*** jmasud has quit IRC08:54
*** jmasud has joined #openstack-meeting08:56
*** witek has joined #openstack-meeting09:00
ttxhi!09:00
ttxraymonddeng: you're not mistaken09:00
ttx#startmeeting large_scale_sig09:00
openstackMeeting started Wed Mar 25 09:00:37 2020 UTC and is due to finish in 60 minutes.  The chair is ttx. Information about MeetBot at http://wiki.debian.org/MeetBot.09:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.09:00
mdelavergneHi!09:00
*** openstack changes topic to " (Meeting topic: large_scale_sig)"09:00
openstackThe meeting name has been set to 'large_scale_sig'09:00
ttx#topic Rollcall09:00
*** openstack changes topic to "Rollcall (Meeting topic: large_scale_sig)"09:00
ttxHi! Who is here for the Large Scale SIG meeting ?09:00
ttxmdelavergne: hi! How is it going for you ?09:00
raymonddenggreat! haha09:01
raymonddengi am present for the large scale sig meeting09:01
ttxraymonddeng: welcome!09:01
*** dh3 has joined #openstack-meeting09:01
mdelavergneEverything's fine, hope it is the same for everyone here :)09:01
ttxpinging amorin belmoreira masahito jiaopengju209:01
masahitohi09:02
ttxoneswig is not in channel09:02
etphi09:02
belmoreirahi, thank for ping ttx09:02
ttxmasahito: how is the situation in Japan, masahito ?09:02
masahitoNot good :-<09:03
masahitoI'm working from home everyday....09:03
ttxok, same here09:03
masahitoBut not a lock down, so I can go out side09:04
ttxah, here we are under strict lockdown09:04
ttx(France)09:04
*** ociuhandu has joined #openstack-meeting09:04
raymonddengwhere are you guys? i'm in los angeles, ca09:04
raymonddengmost people work remotely right now here09:04
ttxraymonddeng: mostly Europe (belmoreira, oneswig, amorin, me) and Asia (masahito, jiaopengju2...)09:05
mdelavergneFrance also09:05
ttxok, let's get started09:05
*** tetsuro has quit IRC09:05
ttxOur agenda for today is at:09:05
ttx#link https://etherpad.openstack.org/p/large-scale-sig-meeting09:05
ttxI'll reorder the topics to give amorin and oneswig some time to join09:06
ttx#topic Large scale operations now-virtual OpenDev track09:06
*** openstack changes topic to "Large scale operations now-virtual OpenDev track (Meeting topic: large_scale_sig)"09:06
ttx#link https://etherpad.openstack.org/p/LargeScaleOps_OpenDev09:06
ttxWe had a TODO for everyone to check/comment that etherpad...09:06
ttxbelmoreira: I suspect there will be changes to it as you all work out how to run it virtually ?09:06
belmoreiraI don't have more information... I think the work done will be re-used09:08
ttxyeah, the general idea is to start the discussion on those topics through online sessions09:08
ttxbut some tracks are more likely to be successful at that than others09:08
*** ociuhandu has quit IRC09:08
ttxso in the coming weeks there will be a set of discussions to see what should be done09:09
ttxI'd say reviewing the etherpad is still useful, in that context09:09
ttxbelmoreira: anything else on that topic ?09:09
belmoreirano, when I have more info I will let you know09:10
ttx#topic Progress on "Scaling within one cluster" goal09:10
*** openstack changes topic to "Progress on "Scaling within one cluster" goal (Meeting topic: large_scale_sig)"09:10
ttx#link https://etherpad.openstack.org/p/large-scale-sig-cluster-scaling09:10
ttxWe had one TODO for everyone to review new patchset of oslo.metrics spec09:10
ttx#link https://review.opendev.org/#/c/704733/09:10
ttxmasahito: do you plan to add the metrics information schema in a new patchset ?09:11
ttxit seems to be the only item missing at this point before approval09:12
masahitoI want, but I'm not sure when I can finish it.09:13
ttxmasahito: would you rather wait, or push for approval at the current version ?09:13
masahitoIs the actual information required to be approved?09:13
ttxmasahito: probably not, we could push for approval under the current version09:13
ttxI think bnemec was interested in getting the code out there as soon as possible09:14
ttxso I'd rather focus on that part than on documenting the spec09:14
masahitoAh, okay. then please push the patch set to be approved to make things forwards.09:14
ttxmasahito: when do you think the code could be pushed ?09:15
ttx(I can also push for repository creation if it's ready, but I suspect you ahve to check a lot of things before doing that)09:15
ttx#action ttx to push for oslo.metric spec approval09:16
masahitomaybe until end of April? I'm a bit busy in other task right now.09:16
ttxmasahito: I can imagine. No worry, just wondering about the general timeframe. So the spec approval is not super-urgent09:17
ttxbut would be good to have in the coming weeks09:17
ttx#info masahito thinks the oslo.metric code could be pushed by end of April at the earliest09:18
*** slaweq has joined #openstack-meeting09:18
ttxOK, otherwiose we had no recent story posted on https://etherpad.openstack.org/p/scaling-stories09:18
ttxoneswig had a TODO to contribute a scaling story on bare metal cluster scaling, but that was not done yet09:18
ttxSo I'll push it back to next meeting09:18
ttx#action oneswig to contribute a scaling story on bare metal cluster scaling09:19
ttxAnything else on the "Scaling within one cluster" goal ?09:19
ttxok, next topic then09:19
ttx#topic Progress on "Documenting large scale operations" goal09:20
*** openstack changes topic to "Progress on "Documenting large scale operations" goal (Meeting topic: large_scale_sig)"09:20
ttx#link https://etherpad.openstack.org/p/large-scale-sig-documentation09:20
ttxWe had two TODOs for amorin from last meeting09:20
ttxI haven't seen anything posted, but I may have missed it09:20
ttxIn doubt Ill push back those TODOs to next meeting09:20
ttx#action amorin to create a wiki page for large scale documentation09:21
ttx#action amorin to propose patch against Nova doc09:21
ttxwe discussed privsep in recent meetings... note that continuing the transition to privsep has been proposed as a Victoria (next cycle) goal:09:21
ttx#link http://lists.openstack.org/pipermail/openstack-discuss/2020-March/013409.html09:21
ttxFeel free to chime in on that thread to encourage this, with your operator hat on !09:22
ttx(it's not clear for everyone that this goal would have operational benefits)09:22
ttxI'll try to help with that community goal09:23
ttxAnything else on "Documenting large scale operations" ?09:23
ttx#topic Next meeting09:23
*** openstack changes topic to "Next meeting (Meeting topic: large_scale_sig)"09:23
ttxWe can do next meeting in two weeks, April 8.09:24
ttxSince Europe will switch to DST next week, we could easily move the meeting to 8 UTC (one hour earlier for Japan and China) if that helps09:24
ttxraymonddeng, masahito: would it be better or worse for you?09:24
raymonddengit's about the same for me09:25
masahitoIt works for me.09:26
ttxbelmoreira: any preference?09:26
ttx(8utc would keep it at 10am for us)09:26
ttxok, let's try it09:27
ttx#info next meeting: April 8, 8:00UTC09:27
ttxI'll post the meeting change to update the meeting calendar and all.09:28
ttx#topic Open discussion09:28
*** openstack changes topic to "Open discussion (Meeting topic: large_scale_sig)"09:28
ttxraymonddeng: was there anything specific you wanted to discuss? Any particular reason for joining the SIG?09:28
ttx(like, are you running a large scale deployment yourself?)09:28
ttxFWIW, the SIG has tried to avoid boiling the ocean, and that's why we tried to set reasonable, focused goals to start09:29
ttxbut the SIG obviously goes where its active members push it, so if there is anything you'd like to lead, it's pretty open :)09:30
raymonddengno i didn't have anything specific to discuss, i found this sig a few hours ago and was fortunate that the meeting was about to start09:31
ttxah! perfect timing indeed09:31
raymonddengi'm a university student interested in 'large scale infrastructure', doing some devops stuff at Code for America and another company and wanted to see what this sig was about see what was09:31
raymonddengoops, **and wanted to see what this sig was about09:31
raymonddengyeah very fortunate timing09:32
*** raymonddeng has quit IRC09:32
ttxThe general goal is to try to regroup people with common problems, so that they can work on common solutions09:32
*** ykatabam has quit IRC09:32
ttxah, we lost raymond09:32
mdelavergnethis timing was not so fortunate :p09:33
ttxlol09:33
*** raymonddeng has joined #openstack-meeting09:33
ttxOh well, that probably means the meeting is over, unless anyone has anything more to discuss?09:33
ttxraymonddeng: Welcome back! The general goal of SIGs is to try to regroup people with common problems, so that they can work on common solutions09:33
ttxSo here we are trying to regroup operators of large scale infrastructure (based on openstack) so that they share problems and work on solutions09:34
ttxtraditionally everyone work on solutions on their side, resulting in lots of duplication of effort09:34
*** dmacpher has joined #openstack-meeting09:35
ttxand the software does not get any better in handling the problems for the next person09:35
ttxThis SIG is trying to solve that, pool resources to work on the same objectives, and ultimately improve the software09:36
ttxIt's not easy, as we all have other day jobs, but it's important :)09:36
ttxAnd on those good words, I think we can close today's meeting, unless anyone else has anything to add09:37
raymonddengdefinitely very important, really impressive that you guys handle day jobs and os contributions09:37
raymonddenghope i can help out in some capacity!09:37
ttxraymonddeng: I hope so! Thanks for joining09:38
ttxThanks everyone for attending the meeting! Stay safe!09:38
ttx#endmeeting09:38
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"09:38
openstackMeeting ended Wed Mar 25 09:38:18 2020 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)09:38
openstackMinutes:        http://eavesdrop.openstack.org/meetings/large_scale_sig/2020/large_scale_sig.2020-03-25-09.00.html09:38
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/large_scale_sig/2020/large_scale_sig.2020-03-25-09.00.txt09:38
openstackLog:            http://eavesdrop.openstack.org/meetings/large_scale_sig/2020/large_scale_sig.2020-03-25-09.00.log.html09:38
*** raymonddeng has quit IRC09:38
masahitobye09:39
mdelavergneThanks everyone, bye09:39
etpthanks, bye09:39
belmoreirabyebye09:41
*** mdelavergne has quit IRC09:43
*** dh3 has quit IRC09:44
*** dmacpher has quit IRC09:47
*** dmacpher has joined #openstack-meeting09:49
*** Liang__ has quit IRC09:57
*** e0ne has quit IRC10:03
*** e0ne has joined #openstack-meeting10:04
*** ociuhandu has joined #openstack-meeting10:14
*** rcernin has quit IRC10:17
*** maciejjozefczyk_ has joined #openstack-meeting10:27
*** trident has quit IRC10:29
*** maciejjozefczyk has quit IRC10:29
*** trident has joined #openstack-meeting10:31
*** masahito has quit IRC10:32
*** trident has quit IRC10:33
*** ykatabam has joined #openstack-meeting10:37
*** trident has joined #openstack-meeting10:37
*** ykatabam has quit IRC10:37
*** ociuhandu has quit IRC10:52
*** janders has joined #openstack-meeting10:57
*** oneswig has joined #openstack-meeting10:59
oneswig#startmeeting scientific-sig11:00
openstackMeeting started Wed Mar 25 11:00:16 2020 UTC and is due to finish in 60 minutes.  The chair is oneswig. Information about MeetBot at http://wiki.debian.org/MeetBot.11:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.11:00
*** openstack changes topic to " (Meeting topic: scientific-sig)"11:00
openstackThe meeting name has been set to 'scientific_sig'11:00
jandersg'day! :)11:01
jandershope everyone is keeping safe11:01
*** ociuhandu has joined #openstack-meeting11:01
verdurinMorning.11:01
oneswighi all11:01
jandershere packets are still allowed to leave NIC ports, we'll see for how long11:01
belmoreirahello11:02
oneswigall healthy here11:02
jandersthat's great to hear :)11:03
oneswigThere isn't an agenda today - apologies - I've been helping with a training workshop this week11:03
jandersno worries11:03
oneswigI've heard a few things related to coronavirus and providing compute resources to help research11:06
oneswigAnyone else active in this?11:06
verdurinYes, we're active there - lots of frantic requests and cluster re-arrangements.11:07
janderswe're under change moratorium to make sure systems are up and running for COVID research11:07
janderspersonally I'm not involved in anything beyond this at this point in time :(11:07
oneswigI registered for folding@home, but in 2 days (so far) I've had just 1 work unit... apparently the response has been so huge their queues have drained.11:08
oneswigWhat form is the compute workload taking?11:08
oneswigjanders: I assume the fires are all done with, or is that still ongoing in the background?11:11
jandersluckily, yes11:11
janderson to the next crisis11:11
verdurinWe have some people preparing to work on patient data, others running simulations.11:12
verdurinSome working on drug targets/therapies.11:12
oneswigverdurin: from the news I've seen Oxford University does appear to be very active.11:12
verdurinYes. I wasn't deemed glamorous enough to appear on the News, though.11:13
oneswigAh11:13
oneswigElsewhere I've not heard how it's affecting IT supply chains but inevitably there must be consequences.11:14
verdurinIt has. We had to pay over the odds for some parts to ensure timely delivery, by going with the only vendor who had stock already.11:15
oneswigjanders: is Canberra in lockdown?  We started in earnest this week.11:16
janderspartially, yes11:16
jandersthough comms aren't particularly clear11:16
janderswe've been working from home for weeks now so not much change11:17
jandersbut the authorities are gradually cracking down on pretty much anything11:17
oneswigDid you see that Ceph Octopus has been released?11:20
jandersnot yet... are there any new features or improvements relevant to HPC?11:21
jandersRDMA support, improvements to CephFS?11:21
oneswigFrom what I've heard the Ceph-Ansible support is doubtful.  A pity considering it tends to work well nowadays11:21
jandersthats unfortunate11:22
oneswigI think there may be less support for RDMA, but an opportunity to do it better second time round.11:22
oneswigI'm not sure this represents the official plan of record but it's along the right lines: https://docs.ceph.com/docs/master/dev/seastore/11:24
verdurinThe big thing I noticed is this new 'cephadm' tool.11:25
oneswigverdurin: is that part of the orchestrator?11:27
*** raildo has joined #openstack-meeting11:31
verdurinIt's a new installation tool that is yes somehow integrated with the orchestrator.11:34
verdurinI just scanned the release notes and flipped through the new docs, though.11:39
janderson another note, I had a chance to finally get back to VPI work on CX6 and I did get it to work with MOFED5 and latest FW11:39
janderswill probably use it more over the coming weeks11:39
jandersone interesting challenge I ran into is - how to tackle dual-PCIe-slot setup for 100/200GE11:40
verdurinjanders: was this with newer firmware than before or did you need to do something different?11:40
jandersverdurin: newer firmware11:40
jandersI had issues with VPI in the past and ended up using onboard 10GE for ethernet comms11:40
jandersnow I'm trying to move back to the Mellanox cards11:40
janderswhere it gets interesting is - for the dual-slot, dual-port card, how to wire this up to ovs?11:41
jandersit comes out as two ports11:41
janderseven though physically it's one11:41
janderswith IB it's easy as multipathing is more natural11:41
*** rpittau is now known as rpittau|bbl11:42
verdurinAh, you have the dual-slot card. We haven't used any of those yet.11:42
janderswith ethernet I'm not sure - do you guys have any experience with that?11:42
jandersyes11:42
jandersthey work beautifully for storage, even if they are a little convoluted11:42
*** oneswig_ has joined #openstack-meeting11:43
oneswig_sorry, dropped off11:43
*** eharney has quit IRC11:44
jandersI will likely chat to Mellanox about this over coming days, happy to relay back what I learned if you're interested11:45
jandersit seems that for non-performance-sensitive workloads, just using one of the two interfaces will suffice11:45
*** oneswig has quit IRC11:45
janderswhat im wondering about though is if we're using leftover PCIe bandwidth for ethernet traffic, maybe it's better for IB if ethernet load balances across PCIe slots as well11:46
jandersIB likes symmetry11:46
jandersif each card gets say 12.5GB/s but we try to snap 5GB/s on one port for eth, I am not too sure if we still get 20GB/s on IB using what's left11:47
jandersso this may be motivation for figuring this out11:47
jandersto prevent IB slowdowns due to intermittent ethernet bandwidth peaks11:47
verdurinHmm. Sounds potentially troublesome, as you say.11:48
jandersyeah11:48
verdurinWould be interested to hear.11:48
jandersdual-slot CX6s are fast but tricky to drive at times11:48
jandersluckily GPFS uses them very efficiently11:48
oneswig_Surprising that VPI has been an issue, you'd think a lot of people would want that.11:48
jandersI think it got fixed initially around October, but that had the limitation of having to use a specific port for IB, wouldn't work other way round11:49
*** eharney has joined #openstack-meeting11:49
jandersI think now it is fully functional11:49
jandersnew MOFED packaging convention is interesting though11:50
jandersthe archive has multiple repos and accidentally using more than one leads to dependency hell11:50
jandersI think it's been like that since around 4.7 but I only really looked into this today on 5.011:50
oneswig_oh no...11:50
jandersmakes sense when you really look through but takes a while to get used to11:50
oneswig_We've been using this - https://github.com/stackhpc/stackhpc-image-elements/tree/master/elements/mlnx-ofed - but sometimes the kernel weak-linkage doesn't work correctly, perhaps it's related to the synthetic DIB build environment.11:52
jandersyeah I had some issues with weak linkage in the past11:52
jandersbest rebuild against a specific kernel revision11:53
*** ykatabam has joined #openstack-meeting11:54
*** rh-jelabarre has joined #openstack-meeting11:56
oneswig_That can be tricky in diskimage-builder, but I expect there's a way through.11:57
oneswig_Nearly at time11:58
jandersstay safe everyone :)11:58
oneswig_same to you!11:58
jandershopefully there won't be curfew on internet comms next week :)11:58
oneswig_overload, more like.11:58
jandersI don't think COVID transmits through RDMA though11:59
jandersso that may be the safest comms channel :)11:59
oneswig_cheers all12:02
oneswig_#endmeeting12:02
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"12:02
openstackMeeting ended Wed Mar 25 12:02:10 2020 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)12:02
openstackMinutes:        http://eavesdrop.openstack.org/meetings/scientific_sig/2020/scientific_sig.2020-03-25-11.00.html12:02
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/scientific_sig/2020/scientific_sig.2020-03-25-11.00.txt12:02
openstackLog:            http://eavesdrop.openstack.org/meetings/scientific_sig/2020/scientific_sig.2020-03-25-11.00.log.html12:02
*** janders has quit IRC12:08
*** links has quit IRC12:27
*** links has joined #openstack-meeting12:28
*** rfolco has joined #openstack-meeting12:30
*** jmasud has quit IRC12:53
*** jmasud has joined #openstack-meeting12:56
*** rpittau|bbl is now known as rpittau13:00
*** enriquetaso has joined #openstack-meeting13:08
*** andrebeltrami has joined #openstack-meeting13:16
*** ociuhandu has quit IRC13:19
*** ociuhandu has joined #openstack-meeting13:20
*** ociuhandu has quit IRC13:25
*** ysandeep is now known as ysandeep|away13:31
*** oneswig_ has quit IRC13:35
*** Adri2000 has quit IRC13:39
*** TrevorV has joined #openstack-meeting13:49
*** thgcorrea has joined #openstack-meeting13:53
*** Liang__ has joined #openstack-meeting13:55
*** sfernand has joined #openstack-meeting13:56
*** Liang__ is now known as LiangFang13:56
*** liuyulong has joined #openstack-meeting13:59
*** throne82 has joined #openstack-meeting14:00
liuyulong#startmeeting neutron_l314:03
openstackMeeting started Wed Mar 25 14:03:12 2020 UTC and is due to finish in 60 minutes.  The chair is liuyulong. Information about MeetBot at http://wiki.debian.org/MeetBot.14:03
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.14:03
*** openstack changes topic to " (Meeting topic: neutron_l3)"14:03
openstackThe meeting name has been set to 'neutron_l3'14:03
haleybhi14:03
liuyulonghi14:03
ralonsohhi14:04
liuyulongOK, let's start.14:05
liuyulong#topic Announcements14:05
*** openstack changes topic to "Announcements (Meeting topic: neutron_l3)"14:05
slaweqhi14:05
liuyulong#link http://eavesdrop.openstack.org/meetings/networking/2020/networking.2020-03-24-14.00.log.html#l-1114:05
liuyulongRecall the team Announcements14:05
liuyulongSince we are an online community at most time, a virtual PTG should work for our community members.14:07
liuyulong#link https://www.openstack.org/events/opendev-ptg-2020/14:07
liuyulongSo the Vancouver PTG is officially cancelled.14:08
liuyulongI've been working from home about 2 months.14:10
liuyulongAnd more days may still be needed.14:11
liuyulongSo, take care of yourself guys.14:11
slaweqliuyulong: thx, I'm working from home for last 2 years already ;)14:11
liuyulongslaweq, cool, good to know that.14:11
liuyulongslaweq, you are really safe now and then.14:12
liuyulongOK, according to the bug deputy email14:12
liuyulong#link http://lists.openstack.org/pipermail/openstack-discuss/2020-March/013510.html14:12
liuyulongIMO, no L3 bugs last week.14:12
liuyulongSo, anyone has bug to discuss here? Or let's just go next topic.14:13
liuyulongOK14:14
liuyulong#topic OVN_L314:14
*** openstack changes topic to "OVN_L3 (Meeting topic: neutron_l3)"14:14
liuyulong#link https://bugs.launchpad.net/neutron/+bug/186786914:14
openstackLaunchpad bug 1867869 in neutron "[OVN] Remove dependency on fip_object" [Low,In progress] - Assigned to zhanghao (zhanghao2)14:14
ralonsohthis is still ongoing14:15
liuyulongOne bug from OVN14:15
ralonsohwe need to know if this will be needed in QoS FIP14:15
ralonsohso, IMO, I'll wait to remove this dependency14:16
liuyulongralonsoh, sure, thank you for the information. Mostly are TODOs from lucasagomes, IMO, he may also have a deep review.14:17
liuyulongHe is already in the list.14:18
*** maciejjozefczyk_ is now known as maciejjozefczyk14:19
liuyulongNext should be the patch:14:19
liuyulong#link https://review.opendev.org/#/c/705660/14:19
liuyulongI run it locally last week, it works. But it now needs a rebase.14:19
ralonsohmaciejjozefczyk, ^14:20
ralonsohI'll ping him in gerrit or IRC14:21
maciejjozefczykralonsoh, thanks for testing!14:22
liuyulongSure, I will run the rebased code again if I have enough time. : )14:22
maciejjozefczykI'll rebase it14:22
liuyulong#link https://review.opendev.org/#/c/710881/2/doc/source/ovn/gaps.rst14:23
liuyulongThis is from the gaps from between ML2/OVS and OVN, most of them looks like are related to L3.14:23
maciejjozefczykOk, so for 'QoS for Layer 3 IPs': This week I tested OVN QoS on FIP - there is a possibility. Its only a matter of proper match in QoS row.14:24
maciejjozefczyk#link: https://mail.openvswitch.org/pipermail/ovs-discuss/2020-March/049857.html14:25
liuyulongI want to say that "QoS for Layer 3 IPs" has the higher priority from the perspective of cloud provider.14:25
liuyulongDatacenter could not ensure that every external IPs to reach the max bandwidth at the same time.14:26
*** psachin has quit IRC14:27
*** Adri2000 has joined #openstack-meeting14:29
liuyulongmaciejjozefczyk, cool, so just logical flows should be added from L3_ovn plugin to north DB, and then ovn-controller, gateway chassis should do the meters installation.14:31
maciejjozefczykliuyulong, yes. For now we're working on refactor of adding of QoS rows in NBDB table. From this table the Logical Flows are generated.14:33
maciejjozefczykAfter a refactor that would be possible to add specific QoS row that will match a FIP14:33
maciejjozefczykfrom Core OVN perspective its possible and I tested it manually, N/S traffic is shaped, while E/W not :)14:33
liuyulongSo the OVN DB schema should be refactored first at core OVN?14:36
*** ociuhandu has joined #openstack-meeting14:36
liuyulongOK, I found the answer. https://mail.openvswitch.org/pipermail/ovs-discuss/2020-March/049863.html14:37
maciejjozefczykliuyulong, There is no need to change anything in Core OVN - in relation to QoS on FIP.14:38
liuyulong"inport == \"96242e8f-602f-4077-984a-5659e84a8c4f\" && ip4.src == 172.24.5.126 && is_chassis_resident(\"cr-lrp-96242e8f-602f-4077-984a-5659e84a8c4f\")"14:40
liuyulongI'm not quite sure about this, so in gateway chassis, the packet has already be NATed (source IP) to floating IP.14:41
maciejjozefczykliuyulong, yes, Neutron, while creating the QoS row should define match similar to this oen14:41
liuyulongBut, what about the ingress from outside world, the destination IP is floating IP.14:41
ralonsohthat will be another rule14:42
ralonsohdepending on the rule direction14:42
ralonsohthe OVN qos rules will match the Neutron QoS ones14:42
ralonsohand will add the needed "match" field in the NB14:43
ralonsoh(matching src/dst IP or whatever is needed)14:43
maciejjozefczykliuyulong, yes, the match field in the rule will look differently14:43
liuyulongSo, this line is only for egress (from the perspective of VM) to outside world.14:44
ralonsohyes14:44
liuyulongralonsoh, maciejjozefczyk, thank you guys, : )14:44
liuyulongOK, no more OVN related things from me then.14:45
liuyulongAny updates?14:45
liuyulongOK, let's move on.14:46
liuyulong#topic On demand agenda14:46
*** openstack changes topic to "On demand agenda (Meeting topic: neutron_l3)"14:46
liuyulongSomething related to IPv6 again.14:46
*** lajoskatona has joined #openstack-meeting14:47
liuyulongI rebased the patch https://review.opendev.org/#/c/662111/ from Swaminathan, hope we could continue the work.14:47
*** jraju__ has joined #openstack-meeting14:49
haleybliuyulong: yes, that would be good to move forward. We should think about OVN as well14:49
*** links has quit IRC14:49
liuyulonghaleyb, indeed, we should not introduce another gap. : )14:51
*** macz_ has joined #openstack-meeting14:52
*** dklyle has joined #openstack-meeting14:52
haleybi know, but knowing it exists and adding it to the list is good, that way it can be worked on.14:53
liuyulong+1 make sense14:55
liuyulongI will test Swaminathan's patch more deeply locally, and comment the feedback.14:56
*** ociuhandu has quit IRC14:56
liuyulongIMO, it is pretty close now.14:57
liuyulongAlright, no more topics from me now.14:57
*** brinzhang has joined #openstack-meeting14:59
liuyulongTime is up.15:00
liuyulongLets end here.15:00
liuyulong#endmeeting15:00
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"15:00
openstackMeeting ended Wed Mar 25 15:00:33 2020 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)15:00
openstackMinutes:        http://eavesdrop.openstack.org/meetings/neutron_l3/2020/neutron_l3.2020-03-25-14.03.html15:00
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/neutron_l3/2020/neutron_l3.2020-03-25-14.03.txt15:00
openstackLog:            http://eavesdrop.openstack.org/meetings/neutron_l3/2020/neutron_l3.2020-03-25-14.03.log.html15:00
liuyulongThank you guys for attending.15:00
liuyulongBye15:00
ralonsohbye15:00
slaweqbye15:00
*** ociuhandu has joined #openstack-meeting15:09
*** eharney has quit IRC15:18
*** macz_ has quit IRC15:20
*** macz_ has joined #openstack-meeting15:21
*** eharney has joined #openstack-meeting15:21
*** LiangFang has quit IRC15:32
*** jamesmcarthur has joined #openstack-meeting15:37
*** dmacpher has quit IRC15:38
*** dmacpher has joined #openstack-meeting15:39
*** gyee has joined #openstack-meeting15:40
*** jraju__ has quit IRC15:47
*** liuyulong has quit IRC16:42
*** throne82 has quit IRC16:46
*** ociuhandu has quit IRC16:47
*** witek has quit IRC16:47
*** ociuhandu has joined #openstack-meeting16:48
*** ociuhandu has quit IRC16:53
*** jmasud has quit IRC16:53
*** jmasud has joined #openstack-meeting16:56
*** rfolco is now known as rfolco|bbl16:58
*** belmoreira has quit IRC17:01
*** JangwonLee_ has joined #openstack-meeting17:05
*** JangwonLee has quit IRC17:09
*** jamesmcarthur has quit IRC17:12
*** thgcorrea has quit IRC17:13
*** jamesmcarthur has joined #openstack-meeting17:13
*** rpittau is now known as rpittau|afk17:13
*** jamesmcarthur has quit IRC17:24
*** jamesmcarthur has joined #openstack-meeting17:24
*** e0ne has quit IRC17:36
*** e0ne has joined #openstack-meeting17:37
*** e0ne has quit IRC17:37
*** e0ne has joined #openstack-meeting17:38
*** ociuhandu has joined #openstack-meeting17:39
*** e0ne has quit IRC17:39
*** dmacpher has quit IRC17:44
*** ociuhandu has quit IRC17:45
*** dmacpher has joined #openstack-meeting17:46
*** dmacpher has quit IRC17:46
*** dmacpher has joined #openstack-meeting17:46
*** jamesmcarthur has quit IRC18:01
*** jamesmcarthur has joined #openstack-meeting18:03
*** jamesmcarthur has quit IRC18:08
*** jamesmcarthur has joined #openstack-meeting18:15
*** number80 has joined #openstack-meeting18:15
*** jamesmcarthur has quit IRC18:22
*** jamesmcarthur has joined #openstack-meeting18:23
*** jamesmcarthur has quit IRC18:25
*** jamesmcarthur has joined #openstack-meeting18:25
*** irclogbot_1 has quit IRC18:37
*** rfolco|bbl is now known as rfolco18:39
*** rh-jelabarre has quit IRC18:52
*** rh-jelabarre has joined #openstack-meeting18:52
*** armax has joined #openstack-meeting18:54
*** jamesmcarthur has quit IRC18:58
*** maciejjozefczyk has quit IRC19:01
*** ociuhandu has joined #openstack-meeting19:01
*** irclogbot_2 has joined #openstack-meeting19:02
*** jamesmcarthur has joined #openstack-meeting19:06
*** ociuhandu has quit IRC19:13
*** macz_ has quit IRC19:17
*** larainema has quit IRC19:20
*** ralonsoh has quit IRC19:26
*** Trevor_V has joined #openstack-meeting19:28
*** TrevorV has quit IRC19:32
*** irclogbot_2 has quit IRC19:37
*** irclogbot_2 has joined #openstack-meeting19:40
*** irclogbot_2 has quit IRC19:42
*** irclogbot_3 has joined #openstack-meeting19:45
*** macz_ has joined #openstack-meeting19:50
*** lajoskatona has quit IRC19:50
*** irclogbot_3 has quit IRC20:00
*** ociuhandu has joined #openstack-meeting20:01
*** jamesmcarthur has quit IRC20:01
*** jamesmcarthur has joined #openstack-meeting20:03
*** irclogbot_1 has joined #openstack-meeting20:04
*** jamesmcarthur has quit IRC20:08
*** irclogbot_1 has quit IRC20:12
*** jamesmcarthur has joined #openstack-meeting20:15
*** irclogbot_0 has joined #openstack-meeting20:15
*** diablo_rojo has quit IRC20:16
*** irclogbot_0 has quit IRC20:16
*** irclogbot_0 has joined #openstack-meeting20:21
*** ociuhandu has quit IRC20:27
*** ociuhandu has joined #openstack-meeting20:28
*** ociuhandu has quit IRC20:33
*** ericyoung has quit IRC20:47
*** ericyoung has joined #openstack-meeting20:49
*** jmasud has quit IRC20:54
*** alecuyer has joined #openstack-meeting20:55
*** jmasud has joined #openstack-meeting20:55
*** jgriffith has quit IRC20:57
*** rfolco has quit IRC20:58
*** raildo_ has joined #openstack-meeting20:58
*** jgriffith has joined #openstack-meeting20:58
*** diablo_rojo has joined #openstack-meeting21:00
*** raildo has quit IRC21:00
*** mattoliverau has joined #openstack-meeting21:00
*** zaitcev has joined #openstack-meeting21:00
*** patchbot has joined #openstack-meeting21:00
timburke#startmeeting swift21:00
openstackMeeting started Wed Mar 25 21:00:59 2020 UTC and is due to finish in 60 minutes.  The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot.21:01
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.21:01
*** openstack changes topic to " (Meeting topic: swift)"21:01
openstackThe meeting name has been set to 'swift'21:01
timburkewho's here for the swift meeting?21:01
mattoliverauo/21:01
kota_hi21:01
alecuyero/21:01
rledisezo/21:01
tdasilvahi21:01
seongsoochoo/21:01
timburkeagenda's at https://wiki.openstack.org/wiki/Meetings/Swift21:02
timburke#topic surveys21:02
*** openstack changes topic to "surveys (Meeting topic: swift)"21:02
timburkethe 2020 user survey is out!21:02
timburke#link https://www.openstack.org/user-survey/survey-2020/21:03
timburkehelp inform the foundation/tc/projects about what's getting used and how, and where things could be improved :-)21:03
timburkethere's *also* some kind of special survey going on...21:04
timburke#link http://lists.openstack.org/pipermail/openstack-discuss/2020-March/013577.html21:04
timburkewhich references...21:04
timburke#link https://openstackfoundation.formstack.com/forms/10_years_of_openstack21:04
timburkei don't think there's much more to say about them. they exist, i plan on filling them out :-)21:05
claygo/21:05
timburke#topic virtual PTG21:06
*** openstack changes topic to "virtual PTG (Meeting topic: swift)"21:06
timburkei created an etherpad to start collecting topic21:06
timburke#link https://etherpad.openstack.org/p/swift-ptg-victoria21:06
timburkei think it's going to be interesting making this be a good, useful time for us, but i know we're up for it21:07
mattoliverauit sux that the PTG was cancelled, but actually works out well for me :)21:07
timburkeme too :-) i wasn't going to be able to go otherwise21:08
kota_:-)21:08
mattoliverau\o/21:08
*** jmasud has quit IRC21:10
timburkeas it gets closer, i'll send out some surveys about what time slots people will be around for -- i wonder if it might make sense to stagger the times over the course of the week, with the expectation that not everyone will attend every session21:10
alecuyersounds good21:10
timburkeotoh, clayg and i both seem to post reviews at all hours, so maybe we just optimize for asia/europe and make it work :P21:11
timburke(hopefully rledisez and tdasilva don't kill me ;-)21:11
timburkethings to consider, anyway21:12
mattoliveraulol21:12
rledisezi'm pretty sure I can make it, I'm kinda an early bird usually ;)21:12
timburkethat's all i had by way of announcements -- any comments or questions on them?21:12
timburkeon to updates!21:14
zaitcevKernel used to run over e-mail. It took me a while to get used to how meeting-happy OpenStack was.21:14
timburke#topic waterfall EC21:14
*** openstack changes topic to "waterfall EC (Meeting topic: swift)"21:14
*** jgriffith has quit IRC21:14
timburkeclayg, i heard you were working on a third attempt21:15
*** enriquetaso has quit IRC21:17
timburkemaybe it's still not to a point worth bringing up ;-)21:18
timburkei remember hearing admiration for torgomatic, and having everything feel almost-but-not-quite related enough to have it all make sense21:19
timburke#topic lots of small files21:19
*** openstack changes topic to "lots of small files (Meeting topic: swift)"21:19
timburkealecuyer, rledisez, how's it going?21:19
alecuyerI posted some patches we had there we had not pushed, thanks Tim for looking at these21:20
timburke👍21:20
alecuyerOtherwise,  I have been running POCs about volume selection when writing, and storing object metadata in the DB. That's about it21:20
timburkesounds good21:20
*** jamesmcarthur has quit IRC21:21
timburkei'm still a little unsure about what to do for p 71391421:21
patchbothttps://review.opendev.org/#/c/713914/ - swift (feature/losf) - Add a sleep() between index-server health checks - 1 patch set21:21
*** jamesmcarthur has joined #openstack-meeting21:21
*** jgriffith has joined #openstack-meeting21:21
alecuyerI think I will remove the hardcoded sleep(10) from the other patch that prompted this one, so we can go ahead and think about how to do handle the health checks properly on slow machines21:22
timburkeit sounds like you have a similar sleep added in the replicator & reconstructor -- but rledisez and zaitcev seemed hesitant to turn it into a config option... should we just bump up the timeout? or ... something else?21:22
rledisezI think for replicator/reconstructor we can just do a getmtime() on ring and optimize it. though, 0.1 second is like really small, maybe bump it a little (like 1s, 2 s?)21:23
zaitcevI was hoping that 0.01 would be sufficient for everyone.21:23
rledisezfor LOSF, I don't know it we can "optimize" the current check or if bumping the sleep is the only solution21:23
alecuyerI can cache that result a few seconds in the health check - but we also have the issue with "regular" daemons, on atom CPUs21:24
*** jamesmcarthur has quit IRC21:26
rledisezI'm thinking, currently, there is only non-essential daemon that use this class (replicator, reconstructor, …), right? it can't hurt to be a little slower to spawn new process. it can even save of a situation where process is just diying and is respawned continuously21:27
rlediseznon-essential == not in the data-path21:28
timburkeseems reasonable21:28
timburkethanks21:28
timburke#topic cors21:29
*** openstack changes topic to "cors (Meeting topic: swift)"21:29
timburkeso i've tried to get a minimal chain for the s3api change that i'm actually looking for21:29
timburkep 533028, p 710330, p 71035521:30
patchbothttps://review.opendev.org/#/c/533028/ - swift - Add some functional CORS tests - 11 patch sets21:30
patchbothttps://review.opendev.org/#/c/710330/ - swift - s3api: Pass through CORS headers - 7 patch sets21:30
patchbothttps://review.opendev.org/#/c/710355/ - swift - s3api: Allow CORS preflight requests - 10 patch sets21:30
timburkei know clayg has looked at the new cors tests long enough to kick out p 713754 and p 714714, thanks!21:31
patchbothttps://review.opendev.org/#/c/713754/ - swift - Move cors out of functests (ABANDONED) - 2 patch sets21:31
patchbothttps://review.opendev.org/#/c/714714/ - swift - Add &test= query param to CORS test infra - 1 patch set21:31
timburkebut i also got a sense that he's hesitant to take on maintaining some new one-off javascript testing framework (which is fair)21:32
timburkehow does everyone else feel about that? i just don't know how i can get a feel for how cors works without writing some javascript...21:33
rledisezI assume there is no way to simulate a browser behavior without a real browser javascript?21:35
rledisezcould it be an external service that do that? I don't like it, but the day that service change…21:36
timburkei suppose i could... but i'd really worry about how well it'd actually match what real browsers *do* -- i've already spotted places where browsers don't interpret the spec the same way, so different headers get exposed depending on which browser you're using21:37
timburkesee p 71223821:37
patchbothttps://review.opendev.org/#/c/712238/ - swift - cors: Add Content-Length to default-exposed headers - 5 patch sets21:37
*** jamesmcarthur has joined #openstack-meeting21:41
*** jgriffith has quit IRC21:41
timburkesomething to think about, anyway. maybe i should look at writing some (python) func tests for those; between func & unit i'd feel pretty good about us being unlikely to regress, so maybe we could flip the order on the chain and let p 533028 be a separate, longer discussion...21:42
patchbothttps://review.opendev.org/#/c/533028/ - swift - Add some functional CORS tests - 11 patch sets21:42
zaitcevtimburke: didn't you try to use Selenium?21:42
*** jgriffith has joined #openstack-meeting21:43
timburkezaitcev, yep, that patch adds a new gate job that uses selenium to drive a headless browser to exercise some javascript21:43
timburkethe question is, do we really want to maintain javascript ;-)21:43
timburkei like the diff i got at https://review.opendev.org/#/c/710355/10/test/cors/test-s3-obj.js with the chain the way it is, though21:44
patchbotpatch 710355 - swift - s3api: Allow CORS preflight requests - 10 patch sets21:44
timburkeCorsBlocked -> S3ObjectOK sure feels like a win :D21:44
*** ociuhandu has joined #openstack-meeting21:45
timburkewell, that was the main question i still had21:46
timburke#topic open discussion21:46
*** openstack changes topic to "open discussion (Meeting topic: swift)"21:46
timburkeanything else to bring up this week?21:46
zaitcevWell, I'm still beating the drum for the dark data watcher.21:49
zaitcevRomain took a look, and I think I mostly answered his remarks.21:49
zaitcevOr completely.21:49
*** ociuhandu has quit IRC21:49
rledisezzaitcev: I'll go check that again21:50
timburkei'll try to take a look this week, too. there's a project i'm looking at where we somehow got some orphaned shards, so it seems not-so-crazy to me that there might be some dark data, too, by the time i've got *that* cleaned up21:51
timburkeall right, thank you all for coming and thank you for working on swift!21:52
timburke#endmeeting21:52
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"21:52
openstackMeeting ended Wed Mar 25 21:52:16 2020 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)21:52
openstackMinutes:        http://eavesdrop.openstack.org/meetings/swift/2020/swift.2020-03-25-21.00.html21:52
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/swift/2020/swift.2020-03-25-21.00.txt21:52
openstackLog:            http://eavesdrop.openstack.org/meetings/swift/2020/swift.2020-03-25-21.00.log.html21:52
*** patchbot has left #openstack-meeting21:52
*** zaitcev has left #openstack-meeting21:53
*** andrebeltrami has quit IRC22:06
*** rh-jelabarre has quit IRC22:16
*** jmasud has joined #openstack-meeting22:29
*** raildo_ has quit IRC22:34
*** slaweq has quit IRC22:34
*** vishalmanchanda has quit IRC22:35
*** slaweq has joined #openstack-meeting22:46
*** Trevor_V has quit IRC22:49
*** slaweq has quit IRC22:51
*** brinzhang has quit IRC22:51
*** rcernin has joined #openstack-meeting22:56
*** efried_gone has quit IRC23:18
*** macz_ has quit IRC23:24
*** sfernand has quit IRC23:46

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!