21:02:22 <oneswig> #startmeeting scientific-wg
21:02:22 <openstack> Meeting started Tue Feb  7 21:02:22 2017 UTC and is due to finish in 60 minutes.  The chair is oneswig. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:02:23 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:02:26 <openstack> The meeting name has been set to 'scientific_wg'
21:02:31 <trandles> hi everyone
21:02:35 <oneswig> Hi zioproto - up late!
21:02:38 <jonmills_nasa> hi
21:02:38 * ildikov is lurking :)
21:02:39 <martial> Hi Stig
21:02:40 <priteau> Hello
21:02:43 <oneswig> Hi all
21:02:47 <oneswig> #chair martial
21:02:48 <openstack> Current chairs: martial oneswig
21:02:48 <martial> Hi all
21:02:48 * cdent waves
21:02:51 <rbudden> hello everyone
21:03:00 <martial> (sorry dual meeting right now -- in person + IRC)
21:03:24 <oneswig> we'll try not to cause you any sudden outbursts in your real meeting Martial :-)
21:03:53 <oneswig> #link Agenda for today https://wiki.openstack.org/wiki/Scientific_working_group#IRC_Meeting_February_7th_2017
21:03:57 <oneswig> freshly minted
21:03:59 <martial> oneswig: thanks will be listening in both :) [working my multitasking skills :) ]
21:04:29 <oneswig> Perfect.  Anyone seen Blair yet?
21:04:57 <trandles> freaky
21:04:59 <oneswig> #chair b1airo
21:05:00 <openstack> Current chairs: b1airo martial oneswig
21:05:07 <oneswig> I rubbed the lamp - that's the truth
21:05:09 <b1airo> morning, bit tardy sorry
21:05:22 <b1airo> what have i just walked in on?
21:05:26 <oneswig> Hi b1airo
21:05:35 <oneswig> only the minutes can answer that!
21:06:15 <oneswig> Ready to talk HPL and hypervisors?
21:06:22 <b1airo> rubbed the lamp... did you win something, or literally find a lamp?
21:06:30 <b1airo> sure
21:06:33 <oneswig> ... aladdin
21:06:35 <oneswig> #topic HPL and hypervisors
21:07:01 <b1airo> ok now that i say yes i need to paste some graphs
21:08:15 <b1airo> this one is SMP linpack:
21:08:18 <b1airo> #link http://pasteboard.co/vBwRsur2Q.png
21:09:04 <oneswig> Taking a while to reach me here...
21:09:08 <zioproto> is slow
21:09:12 <zioproto> ok I have loaded it
21:09:13 <oneswig> aha, got it
21:09:55 <b1airo> this one is HPL with OpenMPI (still single host though): http://pasteboard.co/vByv44w1d.png
21:09:59 <b1airo> #link http://pasteboard.co/vByv44w1d.png
21:10:28 <b1airo> acks to my colleague Phil Chan for generating these
21:10:58 <oneswig> These are hostnames along the x-axis?
21:12:19 <b1airo> the shortname yeah - they are not publicly accessible though so not too worried about security if that was your thinking?
21:12:42 <oneswig> Just wondering what they were - hostnames, parameter variations, ?
21:12:58 <b1airo> so you have guest name and hypervisor name along the x
21:13:23 <b1airo> so basically the story goes like this...
21:13:23 <oneswig> What are the five 120k outliers in the middle of the HPL chart due to?
21:13:31 <b1airo> indeed, what?
21:13:52 <oneswig> I came here tonight for answers!
21:14:33 <b1airo> we only put these nodes into production semi-recently as mellanox had been kind of using them as a RoCE MPI testbed for a while (there's a bunch of fixes and improvements in MOFED 4.0 thanks to this)
21:15:27 <b1airo> anyway, some of our users complained of jobs running much slower on these nodes than on the other 4 year old cluster, so we started digging
21:17:11 <b1airo> just thought i'd share what we have so far as it is a nice sample showing how tight BM-guest can be
21:18:07 <oneswig> VMs appear to have a performance edge for hpl - any theories on why?
21:18:12 <b1airo> the next thing is to figure out what is going on with those outliers - presumably something on the hypervisor is getting in the way at the highest memory utilisation levels, but not sure what yet as they are all the same
21:18:51 <b1airo> oneswig, at this stage i'm guessing it is just the difference in task scheduling between kernel versions
21:19:26 <b1airo> host is Ubuntu with 4.4, guest is CentOS7 with 3.10(?)
21:19:54 <oneswig> Is the NIC involved at all here, or is it memory-copy between VMs?
21:19:56 <b1airo> the guest is pinned etc so host task scheduling is somewhat taken out of the picture there
21:20:08 <oneswig> Or is it just one VM
21:20:27 <b1airo> yeah just single host runs, no IPC
21:21:39 <b1airo> i started looking at whether the guest memory was pinned like i thought it was, but seems the idea i had that pci passthrough forces all guest memory to be pinned was wrong
21:21:41 <oneswig> perf in the hypervisor any help?
21:22:29 <b1airo> at least the only place i've found to look for that is Mlocked memory in /proc/meminfo
21:22:47 <oneswig> You can apparently get perf traces of stacks from both hypervisor and guest, which is pretty cool (but I've never done it)
21:23:08 <b1airo> oneswig, yes that's one of the next things to look at now that we have a known set of misbehaving machines
21:23:40 <b1airo> i have never used it yet either, just started reading
21:24:02 <oneswig> Have you compared bios settings with racadm?  Might not explain why only the large-block cases are problematic
21:24:36 <b1airo> think i heard about an ansible role or two that might be useful for that ;-)
21:24:50 <oneswig> Funny you should say that!
21:25:14 <b1airo> have not compared them with a fine-tooth comb just yet, but the big "performance" items have been checked
21:25:47 <oneswig> You'll report back if you figure it out I hope!
21:26:01 <b1airo> in particular, all these hosts now have highest pstate forced via kernel command line
21:26:41 <b1airo> so they idle at a higher frequency than when they are actually working (AVX is a bit disconcerting like that)
21:27:16 <oneswig> Because AVX heats the chip up?
21:27:18 <b1airo> yes, hopefully that'll be next week! i feel like an amateur detective at the moment, i'm just bouncing from one performance issue to another
21:27:51 <oneswig> Time for the lowdown on ECMP?
21:28:19 <b1airo> oneswig, yes all the modern Intel CPUs have lower base frequencies for AVX instructions (you won't find that anywhere prominent on the product pages though!)
21:28:39 <b1airo> yes ECMP
21:28:39 <oneswig> #topic RoCE over layer-3 ECMP fabrics
21:29:00 <zioproto> what RoCE stands for ?
21:29:02 <oneswig> So this is layer-2 RoCE over VXLAN over ECMP, right?
21:29:09 <oneswig> RDMA-over-Converged-Ethernet
21:29:21 <zioproto> memory over ethernet ?
21:29:24 <oneswig> also known as Infinband-on-ethernet
21:29:27 <oneswig> corret
21:29:48 <oneswig> It's low latency networking
21:29:58 <b1airo> a couple of weeks ago we started our big datacentre migration, in the process we replaced most of the switching in our research compute infrastructure
21:30:48 <b1airo> we were previously mostly using Mellanox SX1036 40/56GbE in a layer-2 configuration
21:32:05 <oneswig> b1airo: what caused you to change?
21:33:05 <b1airo> sorry burnt toast emergency
21:33:07 <b1airo> back now
21:33:26 <b1airo> the size of the fabric we could build was limited to 2x spine switches
21:33:44 <oneswig> Due to MLAG?
21:33:46 <b1airo> using either MLAG or a multi-STP config
21:34:20 <b1airo> and the MLAG had a habit of being a bit fragile
21:34:39 <zioproto> so now you have a L3 leaf/spine architecture ?
21:34:44 <b1airo> so all our new gear is Mellanox Spectrum (100/50/25GbE)
21:34:46 <zioproto> but with more than 2 spines ?
21:34:56 <b1airo> with switches all running Cumulus
21:35:43 <b1airo> zioproto, actually not yet - but yes we could have more spines or insert another aggregation layer
21:36:11 <jonmills_nasa> We presently running with MLAG, all Cumulus, 2 spines and 10 leaf switches.  So far so good.
21:36:26 <b1airo> the switches speak BGP to each other and VXLAN encap/decap traffic to/from the edge ports
21:36:41 <b1airo> (in hardware)
21:36:41 <oneswig> We are old school - 2 core, 9 edge, MLNX-OS + NEO
21:36:54 <jonmills_nasa> Is this Cumulus 3.2.0?
21:37:03 <zioproto> b1airo, are you using puppet or ansible to configure the cumulus switches ?
21:37:03 <b1airo> jonmills_nasa, yep
21:37:10 <b1airo> zioproto, ansible
21:37:27 <b1airo> honestly i don't think puppet really belongs with switching
21:37:53 <jonmills_nasa> I sorta thought that hardware accelerated VTEP wasn't supported in Cumulus until 3.3 release or even 3.4....
21:38:13 <oneswig> b1airo: how are the VXLAN VNIs configured?  Is there a physical-aware ML2 driver behind that?
21:38:26 <b1airo> anyway, it was a huge effort to get it all built and configured - we actually started moving hosts before the network configs were completed
21:39:32 <b1airo> oneswig, no at this stage we haven't added any features just done the migration - so the automation has port profiles setup which configure the VNIs appropriately for the required host-facing VLANs
21:40:12 <oneswig> Ah OK, so the host's vlans are mapped to ranges of vnis in the vtep?
21:40:29 <b1airo> jonmills_nasa, this is using Cumulus LNV to coordinate the VNIs across switches, switching definitely happening in hardware
21:40:38 <b1airo> oneswig, yep
21:41:02 <b1airo> once we had all the gear migrated to the new DC we started properly testing the new network
21:41:15 <jonmills_nasa> b1airo, this is great stuff, this is my long-term goal -- just didn't think I could get there on first pass, so we started with all L2
21:41:49 <b1airo> jonmills_nasa, cool - glad to have some company! Cumulus is very nice to work with I have to say
21:41:58 <jonmills_nasa> I am loving it
21:42:11 <b1airo> we used Lustre LNET self test
21:42:26 <jonmills_nasa> I generated 100% of my L2 edge port config using a very small shell script with NCLU commands
21:42:39 <b1airo> we run Lustre over the o2ib driver as usual but on RoCEv1 instead of IB
21:42:48 <martial> (so far very cool)
21:42:51 <zioproto> b1airo, we also have Cumulus at SWITCH
21:43:15 <zioproto> b1airo, but we are using puppet and OSPF unnumbered for the fabric
21:43:49 <b1airo> we firstly had to ensure the network could handle RoCEv1 at all, which means needing flow-control everywhere (RoCEv1 does not like lossy networks)
21:44:32 <b1airo> the first hurdle there was that we couldn't just turn on global pause like we had previously on the MLNX-OS L2 fabric
21:44:35 <oneswig> I hadn't realised that SR-IOV can be lossy - flow control ends at the phy on Mellanox NICs
21:45:23 <b1airo> oneswig, actually we should go into that more in a moment as they haven't told us anything particular about that (but of course I am interested to hear it!)
21:46:24 <b1airo> the Cumulus folks were leery about enabling pause on the inter-switch links because they were worried BGP peer exchange messages would might get held up long enough that it would then impact the fabric topology
21:47:07 <b1airo> so we had to have priority flow-control configured inter-switch with switch-switch traffic at the highest CoS
21:48:04 <b1airo> then we start pushing large streams around and immediately noticed only a single link utilised up and down when crossing spines
21:49:22 <b1airo> turns out that the VXLAN encap happens before the packet is routed to the next hop, so the ECMP hash is done on the VXLAN'd packet, and for RoCEv1 at least that 5-tuple looks exactly the same for every packet on the same VNI
21:50:21 <zioproto> blair, that is a typical hashing problem
21:50:43 <zioproto> most hashing stuff expect UDP or TCP
21:51:04 <zioproto> when you has other protocols hashing algorithm is dumb and sends all to the same link
21:51:16 <b1airo> hence, one out of four 100G paths used and crappy performance - we actually had a simple cluster test we used to simulate what some user workloads to, we call it "crazy cat lady" - it basically just cats a bunch of large files into /dev/null in a parallel job
21:51:52 <priteau> love that name!
21:51:54 <oneswig> why is the destination IP of the VXLAN frame not different for flows to different machines?
21:52:05 <b1airo> zioproto, it is particularly bad with flow-control, as the link gets highly congested with pause frames as well
21:53:22 <b1airo> oneswig, this is for flows from any host in rack A to any host in rack B on the storage VLAN - that's a single VNI for each ToR
21:53:36 <oneswig> ahhh, got it, thanks
21:53:48 <oneswig> what's the fix?
21:54:13 <b1airo> anyway, the Cumulus support guys spent a couple of days in video conference with us working through testing and workaround
21:54:15 <oneswig> Can you hash on the decapsulated packet?
21:54:58 <martial> (how are we doing on time versus agenda?)
21:55:24 <oneswig> martial: not very well...
21:55:26 <b1airo> the fix was to run a little python script that they hacked up which altered some base port settings in the ASIC and introduced some randomness to the hash
21:56:03 <oneswig> b1airo: is this going to be your gift to Cumulus Linux?
21:56:35 <b1airo> their engineering wrote it, but i'll claim i provided the inspiration
21:56:56 <b1airo> i believe this is going into 3.2.1 permanently
21:56:57 <oneswig> Is there a long-term fix in the works, I assume?
21:57:03 <oneswig> OK
21:57:26 <oneswig> We should cover some other matters before the clock strikes...
21:57:30 <b1airo> looks like i monopolised somewhat today (sorry!)
21:57:43 <trandles> blame the burnt toast ;)
21:57:46 <oneswig> #topic Boston Cloud Declaration
21:57:56 <martial> May 11 & 12
21:58:01 <oneswig> Hooray - the date is moved to the same week as the summit :-)
21:58:17 <oneswig> #topic Repo for WG docs
21:58:27 <oneswig> #link http://git.openstack.org/cgit/openstack/scientific-wg/
21:58:33 <zioproto> guys I have to go
21:58:34 <priteau> oneswig: that's good news. I assume the wiki page will be updated? https://wiki.openstack.org/wiki/Boston-Cloud-Congress
21:58:34 <oneswig> we've got somewhere where we can write things now
21:58:48 <zioproto> bye :) sorry for leaving earlier
21:58:49 <zioproto> ciao
21:58:49 <oneswig> priteau: I think so - should have been already
21:58:53 <oneswig> thanks zioproto
21:59:05 <oneswig> #topic SC2017 workshops
21:59:20 <oneswig> I filed for two workshops at SC2017
21:59:27 <oneswig> one on infrastructure and one on platforms/apps
21:59:41 <oneswig> #link as per discussion at https://etherpad.openstack.org/p/SC17WorkshopWorksheet
21:59:55 <b1airo> oneswig, nice one thanks for getting that done. i saw the notification email come by
21:59:59 <oneswig> There's still a week (apparently) before the submissions ship, any more volunteers please?
22:00:13 <oneswig> Thanks b1airo, no problem
22:00:24 <oneswig> Some improvement on the wording would help!
22:00:32 <b1airo> oneswig, we haven't pinged the MLs about that have we?
22:00:41 <oneswig> If anyone else will be at SC and can contribute, please put your names in the etherpad
22:01:05 <rbudden> oneswig: i’ll read it over and can likely volunteer to help
22:01:08 <oneswig> #action oneswig to put the workshops onto the ML
22:01:09 <rbudden> i’m always at SC
22:01:13 <oneswig> thanks rbudden
22:01:17 <b1airo> rbudden, that'd be great!
22:01:22 <oneswig> we are out of time, alas
22:01:33 <jonmills_nasa> I'm honestly hoping to avoid SC17.  I'll be at Boston Summit tho
22:01:35 <rbudden> and as always we have the PSC booth for other SC content
22:01:36 <oneswig> Thanks all and thanks b1airo, I like my toast to be high roast
22:01:51 <b1airo> you should all prioritise the Sydney Summit!
22:02:02 <rbudden> b1airo: i’m trying to pull strings ;)
22:02:05 <oneswig> That's my plan b1airo!
22:02:13 <trandles> Sydney's a go for me
22:02:13 <oneswig> time to close, alas
22:02:16 <jonmills_nasa> Gov would never pay for me to go there
22:02:20 <rbudden> the wife has always wanted to visit as well!
22:02:32 <b1airo> awesome, i should start looking for venues!
22:02:35 <oneswig> jonmills_nasa: not since they've fingered you as @RogueNASA?
22:02:45 <b1airo> lol
22:02:47 <jonmills_nasa> <cough>
22:02:54 <oneswig> really gotta close now :-)
22:02:55 <oneswig> #endmeeting