21:01:50 <oneswig> #startmeeting scientific-wg
21:01:50 <openstack> Meeting started Tue Oct  4 21:01:50 2016 UTC and is due to finish in 60 minutes.  The chair is oneswig. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:01:52 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:01:54 <openstack> The meeting name has been set to 'scientific_wg'
21:01:57 <oneswig> #chair b1airo
21:01:57 <openstack> Current chairs: b1airo oneswig
21:02:02 <oneswig> Greetings!
21:02:06 <rbudden> hello!
21:02:10 <dfflanders> morning
21:02:12 <georgem1> hello
21:02:16 <cphoffma> Hey
21:02:21 <trandles> hey folks
21:02:23 <martial> hello all
21:02:25 <oneswig> Fantastic to see you all
21:02:34 <b1airo> g'day!
21:02:40 <oneswig> b1airo: you want to drive today?
21:02:51 <b1airo> sure
21:03:07 <oneswig> before we go too far, I forgot.
21:03:19 <oneswig> #link Agenda for today https://wiki.openstack.org/wiki/Scientific_working_group#IRC_Meeting_October_4th_2016
21:03:27 <b1airo> ahh was just grabbing that too
21:03:30 <b1airo> cheers
21:03:39 <oneswig> np
21:03:47 <oneswig> I am a back seat driver
21:03:50 <b1airo> #topic Virtualisation Tuning
21:04:18 <b1airo> i just thought i could briefly talk about some tuning we've been doing lately on our HPC cloud infra
21:04:32 <b1airo> and share a few numbers
21:05:06 <b1airo> we've been using intel optimised linpack as our micro-benchmark
21:05:54 <oneswig> A good stress of CPU and memory?  How dominant is network?
21:05:57 <b1airo> on typical 2-socket NUMA systems with ~250GB ram and 12 cores per socket
21:06:25 <b1airo> we're doing network separately still at the moment - this is just for CPU+mem
21:06:51 <b1airo> so it's an SMP version of linpack
21:07:13 <oneswig> ok good to know
21:07:35 <b1airo> our hosts are running Trusty with Xenial (4.4) kernel and Mitaka cloudarchive Qemu/KVM (2.5)
21:08:08 <oneswig> How sensitive are kernel and kvm versions do you think?
21:08:08 <b1airo> guest OS is CentOS 7.2 (which is 3.10 kernel from memory...)
21:08:12 <clarkb> out of curiousity why not xenial userland too?
21:08:29 <clarkb> I guess its cloudarchive for the bits that are interesting
21:08:46 <b1airo> clarkb, you mean the hypervisor distro?
21:08:49 <clarkb> yes
21:09:17 <b1airo> just because we operate this as a Cell in the NeCTAR Research Cloud and we are on Liberty for most things
21:09:20 <jonmills> Are there specific advantages to a 4.4 kernel on the hypervisor?  I'm running CentOS 7.2 hypervisors, but I could install a 4.4 kernel from elrepo
21:09:45 <b1airo> so once we upgrade to Mitaka in the next few months we'll be at the intersection point for the Trusty/Xenial change over
21:10:36 <b1airo> jonmills, i think we initially upped the kernel for some PCI passthrough issues
21:10:50 <jonmills> okay
21:11:10 <b1airo> then we ended up on 4.4 recently because it has longer support being from Xenial as opposed to the shorter Ubuntu releases
21:11:47 <b1airo> oneswig, as for the versions impacting performance, yes they do
21:12:19 <oneswig> b1airo: are there specific features/versions to look for?
21:12:23 <b1airo> i don't have any bisectional data for our setup but i have seen plenty of other reports of performance regressions in the late 3.xx kernels
21:12:43 <b1airo> e.g. there is an article somewhere on phoronix looking at this
21:13:27 <b1airo> we were also interested in the kernel's automatic numa balancing, which was introduced late in 3 i think
21:13:39 <b1airo> anyway, back to some numbers :-)
21:14:00 <b1airo> so on the baremetal hypervisor we see e.g.:
21:14:32 <martial> b1airo: are you working on a publication on those tests ?
21:14:59 <b1airo> hmm not sure if this is gonna paste well
21:15:09 <b1airo> i'll try, bare with...
21:15:10 <b1airo> Size   LDA    Align.  Average  Maximal
21:15:10 <b1airo> 160    176    4       6.0014   7.0252
21:15:10 <b1airo> 1600   1616   4       251.6930 258.5186
21:15:10 <b1airo> 5600   5616   4       554.8785 557.1577
21:15:10 <b1airo> 11200  11216  4       686.9769 688.3435
21:15:12 <b1airo> 22000  22000  4       761.3992 768.7692
21:15:14 <b1airo> 40000  40016  4       757.9887 758.6859
21:15:16 <b1airo> 80000  80016  4       757.7656 759.1828
21:15:18 <b1airo> 160000 160016 4       773.7305 773.9832
21:15:20 <b1airo> oh nice work hexchat
21:15:55 <b1airo> so the last size (160000) is about 90% mem usage
21:16:37 <b1airo> this is on E5-2680v3 CPUs
21:17:59 <b1airo> in a "naive" guest, i.e., no cpu or mem pinning and no topology exposure, so just a flat 24 CPUs/sockets, one NUMA node guest using 240GB ram
21:18:03 <georgem1> b1airo: can you share the config used and some setup instructions? it would be interesting to run on our systems as well
21:18:04 <b1airo> Size   LDA    Align.  Average  Maximal
21:18:04 <b1airo> 160    176    4       1.9916   3.3462
21:18:05 <b1airo> 1600   1616   4       209.0969 248.5604
21:18:05 <b1airo> 5600   5616   4       438.6423 441.8746
21:18:05 <b1airo> 11200  11216  4       545.3753 546.7031
21:18:05 <b1airo> 22000  22000  4       490.6886 494.0714
21:18:07 <b1airo> 40000  40016  4       487.2085 489.2111
21:18:09 <b1airo> 80000  80016  4       534.8873 544.3701
21:18:11 <b1airo> 160000 160016 4       606.1984 607.2593
21:18:26 <jmlowe> hi, sorry I'm late
21:18:34 <b1airo> georgem1, sure it's pretty easy, will get to that in a sec...
21:18:35 <oneswig> Interesting that you get maximum performance with such large blocks, or am I misreading the parameters?
21:18:42 <oneswig> hi jmlowe
21:19:19 <b1airo> oneswig, i'm no linpack guru, just going by the readme to set it up
21:19:38 <b1airo> but my colleagues have independently come up with the same sort of config
21:19:59 <b1airo> and here's the money shot - guest with cpu and mem pinning and topology exposed:
21:20:11 <trandles> blairo: is KSM enabled?
21:20:18 <b1airo> Size   LDA    Align.  Average  Maximal
21:20:18 <b1airo> 160    176    4       4.9478   7.2315
21:20:18 <b1airo> 1600   1616   4       247.3868 258.3221
21:20:18 <b1airo> 5600   5616   4       548.8194 551.7628
21:20:18 <b1airo> 11200  11216  4       667.9128 672.6079
21:20:19 <b1airo> 22000  22000  4       740.7616 752.1660
21:20:21 <b1airo> 40000  40016  4       756.2580 758.0187
21:20:23 <b1airo> 80000  80016  4       754.5520 754.7635
21:20:27 <b1airo> 160000 160016 4       767.8318 767.9426
21:20:27 <jmlowe> we should publish the tarball of the linpack we used to benchmark for acceptance
21:20:33 <b1airo> trandles, no KSM and THP disabled
21:21:05 <b1airo> so this is with regular 4K pages
21:21:09 <oneswig> Nice work b1airo
21:21:14 <anteaya> for reference if you need to post this kind of data into another channel please use a pastebin service, like paste.openstack.org
21:21:33 <anteaya> fine for your own meeting but others may not like it so much in other channels
21:21:48 <anteaya> and nice work b1airo
21:21:57 <jonmills> So is there a published cheatsheet to produce these nice results?
21:22:06 <b1airo> anteaya, sure i was expecting to have to do it to get it formatted nicely but looks ok here
21:22:13 <anteaya> it does so
21:22:46 <b1airo> jonmills, yes i think we can add this into the HPC whitepaper
21:23:04 <jonmills> cool
21:23:08 <oneswig> b1airo: sounds like a good idea to me, but how soon could you document?
21:23:10 <georgem1> cool indeed
21:23:16 <b1airo> but i'm also going to add something to the hypervisor tuning guide
21:23:32 <b1airo> georgem1, here's the config...
21:23:52 <b1airo> $ cat lininput_xeon64
21:23:52 <b1airo> Sample Intel(R) Optimized LINPACK Benchmark data file (lininput_xeon64)
21:23:52 <b1airo> Intel(R) Optimized LINPACK Benchmark data
21:23:52 <b1airo> 8                     # number of tests
21:23:52 <b1airo> 160 1600 5600 11200 22000 40000 80000 160000 # problem sizes
21:23:53 <b1airo> 176 1616 5616 11216 22000 40016 80016 160016 # leading dimensions
21:23:57 <b1airo> 5 5 5 5 5 5 3 3 # times to run a test
21:23:59 <b1airo> 4 4 4 4 4 4 4 4 # alignment values (in KBytes)
21:24:19 <b1airo> using intel mkl benchmarks (l_mklb_p_2017.0.010.tgz)
21:24:49 <georgem1> thanks
21:25:22 <georgem1> I'll get off the train in 10 min :(
21:25:23 <b1airo> specifically benchmarks_2017/linux/mkl/benchmarks/linpack/runme_xeon64
21:25:48 <b1airo> this seems to be a nice test because it is a portable black box
21:25:51 <martial> so CPU only, no Phi ?
21:25:53 <oneswig> b1airo: what config do you apply outside of the OpenStack domain?
21:26:05 <anteaya> georgem1: you can read the channel log here: http://eavesdrop.openstack.org/irclogs/%23openstack-meeting/%23openstack-meeting.2016-10-04.log.html
21:26:18 <b1airo> we don't have any Phis, got a bunch of K80s though
21:26:21 <georgem1> I know, but it's not the same
21:26:28 <anteaya> georgem1: indeed
21:26:56 <georgem1> everybody got the hotel booked for Barcelona?
21:27:02 <b1airo> in relation to the hypervisor tuning guide i think we need a set of micro-benchmarks that can be commonly used to verify tuning
21:27:04 <jmlowe> I'm in the W
21:27:22 <b1airo> georgem1, yes - i think i was very lucky to get the last room at the SB
21:27:49 <georgem1> nice, I'll be 1 km away from the conference site
21:28:00 <b1airo> so i'm interested in collecting suggestions for such benchmarks
21:28:04 <oneswig> A good city for walking I believe :-)
21:28:36 <oneswig> b1airo: did you figure out your issues with gpu direct virtualised?
21:28:40 <rbudden> georgem1: yep, i’m booked the hilton
21:28:50 <b1airo> i haven't done anything with the tuning guide this cycle but i intend to chat with interested folks in barcelona about what should happen - i'm of the opinion it could because a repo of scripts and benchmarks
21:29:10 <b1airo> any prose can go into the admin/ops guide
21:29:39 <b1airo> so that brings us to:
21:29:55 <b1airo> #topic Barcelona activities and WG space etc
21:30:49 <b1airo> we heard back from the summit folks that a bigger room became available
21:31:31 <b1airo> so looks tentatively like we have a 100 person capacity room on Tues and Weds at (i think...) 2:45pm
21:31:37 <b1airo> oneswig, that right?
21:31:54 <oneswig> wasn't it one or the other? let me check
21:32:01 <georgem1> I'm out but I'll check the meeting logs later
21:32:37 <martial> by georgem1
21:32:41 <georgem1> bye
21:32:42 <b1airo> thanks georgem1 !
21:32:48 <martial> (sigh can not type anymore)
21:33:02 <oneswig> Its 2x 40 minute slots - you're right
21:33:20 <oneswig> 2:15pm Tuesday and 2:15pm Wednesday
21:33:26 <oneswig> How does that look on the schedule?
21:34:00 <b1airo> i think it's ok - only issue is the clash with the long Ops session on Tues arvo
21:34:30 <oneswig> It's quite an unfortunate conflict
21:34:46 <b1airo> i know we have a few folks who will be torn between the two, but the Ops session goes most of the arvo so folks can always join us initially and then move on to that
21:35:00 <oneswig> Is the ops session close by?
21:35:12 <b1airo> good question...
21:35:38 <oneswig> One alternative - a smaller session (30 people) for committee meeting on Wednesday morning and a larger 100 session Wednesday afternoon for the BoF
21:35:44 <dfflanders> What is the divide between the two sesh agenda-wise?
21:35:46 <oneswig> Keeps Tuesday clear
21:36:14 <b1airo> it's still TBA by the looks of it
21:36:33 <oneswig> dfflanders: one's committee meeting, the other's BoF - procedural vs open house I'd guess
21:36:54 <dfflanders> :thumbs up:
21:37:25 <dfflanders> etherpad for listing these options and starting to plan?
21:37:51 <b1airo> you did create one already didn't you oneswig
21:37:57 <oneswig> 1 sec, I'll dig it out
21:39:35 <oneswig> not having much luck here...
21:39:53 <oneswig> I'll keep looking, it was agenda items for the committee meeting
21:40:46 <b1airo> should be in last week's log
21:40:49 <oneswig> #link Barcelona agenda items for committee meeting https://etherpad.openstack.org/p/scientific-wg-barcelona-agenda
21:40:56 <oneswig> thanks b1airo, exactly where it was
21:41:36 <b1airo> dfflanders, i figure this is what we'll flesh out properly on saturday
21:41:43 <oneswig> Not much there at present but it's a 40 minute meeting anyway
21:41:45 <b1airo> (b4 the summit)
21:42:11 <dfflanders> blair, +1
21:42:16 <oneswig> b1airo: sounds good to me
21:42:27 <b1airo> better scoot along here...
21:42:45 <b1airo> #Topic Scientific OpenStack BoF
21:43:29 <b1airo> we have had a few volunteers keen to do lightening talks in the BoF which is great, but I don't think we've actually started recording that to organise things yet?
21:43:51 <oneswig> When we've got the slot booked for the BoF, we should also get the lightning talks nailed - think it might need ot happen in that order
21:43:51 <b1airo> (at least i haven't :-)
21:44:15 <trandles> is this the Summit BoF or SC BoF?
21:44:22 <b1airo> fair enough, so should we discuss the time slots one last time then?
21:44:27 <oneswig> Perhaps whatever we do for the meeting, we could fix the bof for Wednesday afternoon in the 100-seater room
21:44:29 <b1airo> trandles, Summit
21:44:49 <b1airo> oneswig, yes of course
21:44:58 <b1airo> silly me
21:45:17 <oneswig> OK, if we do that I'll go to the list and do a "CFP"...
21:45:36 <b1airo> great1
21:45:46 <oneswig> #action oneswig (carried over from last week) fix the BoF timeslot and call for lightning talks
21:46:20 <dfflanders> If there is a particularly good lightning talk let's plan on getting it recorded somehow as BoF and WG rooms are not auto recorded... If nothing else I can grab a goPro and we can ad hoc it in the hall ;-)
21:46:37 <martial> :)
21:46:46 <b1airo> dfflanders, you could live tweet it :-P
21:47:06 <b1airo> #topic Subsidy for social (Thursday night)
21:47:17 <dfflanders> don't tease me blairo you know how I love a good live tweeting ;-)
21:47:36 <oneswig> OK we've got a small decision to take wrt the social evening
21:47:41 <oneswig> It's been really popular!
21:47:41 <b1airo> so far it's only oneswig's charms that have literally paid off
21:48:03 <oneswig> We have approaching 50 people down for it
21:48:07 <oneswig> And subsidy for 30
21:48:11 <b1airo> i have tried but my dell contact has gone mysteriously silent
21:48:38 <oneswig> Vendors eh.
21:49:00 <jmlowe> they do that sometimes even when you are trying to give them money
21:49:07 <b1airo> i'll give redhat one last push
21:49:12 <dfflanders> how much more money is needed per head?
21:49:29 <oneswig> We'd like to propose going for the bigger event and having people make a small contribution.
21:49:36 <oneswig> €40 per head
21:49:53 <b1airo> so we're currently short 800 ?
21:50:00 <oneswig> That's for food and drink (unspecified how much drink, cava + red wine + mineral water)
21:50:02 <oneswig> yes
21:50:22 <oneswig> might be just shy of that.
21:50:55 <b1airo> though likely to have more register if we were to promote it a little more, e.g. in the summit sessions
21:51:31 <oneswig> b1airo: true but eventually we'll hit a limit with the restaurant
21:52:04 <oneswig> Currently we're looking at roughly €16/head cost
21:52:13 <oneswig> How do people feel about that?
21:52:24 <jmlowe> I'm good with that
21:52:28 <martial> I am confused, was it 40 or 16 ?
21:52:37 <martial> I am okay with either, personally
21:52:43 <oneswig> 40 total, 16 after subsidy
21:53:08 <martial> might want to contact the people that already registered is all
21:53:28 <oneswig> martial: I'll mail all people with tickets and confirm once we are confirmed
21:53:37 <oneswig> should be any day I think
21:53:50 <rbudden> i’m good with that as well
21:54:05 <oneswig> excellent thanks all
21:54:21 <b1airo> i doubt anyone will mind, it's just a question of how we organise the logistics...?
21:55:00 <oneswig> Payment is going to be difficult, and receipts even more so.  I'll work it with the restaurant
21:55:21 <b1airo> oneswig, were you thinking of having people pay on the evening? maybe we can do it via eventbrite?
21:55:31 <jmlowe> I don't like to carry cash, so a card friendly solution would be appreciated
21:55:50 <rbudden> jmlowe: +1 I rarely carry cash if at all possible
21:56:08 <rbudden> eventbrite seems like a simple solution to that
21:56:16 <rbudden> as b1airo suggested
21:56:23 <oneswig> b1airo: eventbrite might solve it.  Good plan.  Might make a second event for which we'd all get tickets
21:56:50 <martial> oneswig: yes a new replacement event might well be the better solution for this
21:57:28 <b1airo> yeah i think that would simplify it, might add a small transaction fee of course, but again i don't think any of us mind as it's mostly work paying :-)
21:57:39 <oneswig> Grand, I like it.  I'll duplicate.  Eventbrite take a small cut but it's worth it.
21:58:16 <oneswig> #action oneswig to mail ticketholders and arrange payment via eventbrite
21:58:16 <dfflanders> you could add a second ticket to the same eventbrite... end 'sales' on the first ticket. save minting new URL? /thinkingOutLoud
21:58:41 <b1airo> sounds like you have been around this rodeo dfflanders
21:58:43 <oneswig> dfflanders: sounds like a good plan but we'll want all the people currently with tickets to get first dibs
21:59:23 <b1airo> ok well we have a plan there, let's action it next week oneswig ?
21:59:35 <oneswig> OK b1airo agreed
21:59:39 <dfflanders> timecheck
21:59:40 <b1airo> give us another few days to see whether we can shake any change loose from the vendors
21:59:48 <b1airo> yep, that's a wrap!
21:59:54 <oneswig> thanks b1airo, great session
21:59:59 <dfflanders> +1
22:00:05 <oneswig> look forward to your report :-)
22:00:06 <rbudden> +1
22:00:08 <dfflanders> l8er ya'll
22:00:15 <b1airo> thanks all!
22:00:15 <rbudden> catch ya’ll later!
22:00:23 <trandles> later
22:00:28 <b1airo> #endmeeting