21:01:50 <oneswig> #startmeeting scientific-wg 21:01:50 <openstack> Meeting started Tue Oct 4 21:01:50 2016 UTC and is due to finish in 60 minutes. The chair is oneswig. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:01:52 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:01:54 <openstack> The meeting name has been set to 'scientific_wg' 21:01:57 <oneswig> #chair b1airo 21:01:57 <openstack> Current chairs: b1airo oneswig 21:02:02 <oneswig> Greetings! 21:02:06 <rbudden> hello! 21:02:10 <dfflanders> morning 21:02:12 <georgem1> hello 21:02:16 <cphoffma> Hey 21:02:21 <trandles> hey folks 21:02:23 <martial> hello all 21:02:25 <oneswig> Fantastic to see you all 21:02:34 <b1airo> g'day! 21:02:40 <oneswig> b1airo: you want to drive today? 21:02:51 <b1airo> sure 21:03:07 <oneswig> before we go too far, I forgot. 21:03:19 <oneswig> #link Agenda for today https://wiki.openstack.org/wiki/Scientific_working_group#IRC_Meeting_October_4th_2016 21:03:27 <b1airo> ahh was just grabbing that too 21:03:30 <b1airo> cheers 21:03:39 <oneswig> np 21:03:47 <oneswig> I am a back seat driver 21:03:50 <b1airo> #topic Virtualisation Tuning 21:04:18 <b1airo> i just thought i could briefly talk about some tuning we've been doing lately on our HPC cloud infra 21:04:32 <b1airo> and share a few numbers 21:05:06 <b1airo> we've been using intel optimised linpack as our micro-benchmark 21:05:54 <oneswig> A good stress of CPU and memory? How dominant is network? 21:05:57 <b1airo> on typical 2-socket NUMA systems with ~250GB ram and 12 cores per socket 21:06:25 <b1airo> we're doing network separately still at the moment - this is just for CPU+mem 21:06:51 <b1airo> so it's an SMP version of linpack 21:07:13 <oneswig> ok good to know 21:07:35 <b1airo> our hosts are running Trusty with Xenial (4.4) kernel and Mitaka cloudarchive Qemu/KVM (2.5) 21:08:08 <oneswig> How sensitive are kernel and kvm versions do you think? 21:08:08 <b1airo> guest OS is CentOS 7.2 (which is 3.10 kernel from memory...) 21:08:12 <clarkb> out of curiousity why not xenial userland too? 21:08:29 <clarkb> I guess its cloudarchive for the bits that are interesting 21:08:46 <b1airo> clarkb, you mean the hypervisor distro? 21:08:49 <clarkb> yes 21:09:17 <b1airo> just because we operate this as a Cell in the NeCTAR Research Cloud and we are on Liberty for most things 21:09:20 <jonmills> Are there specific advantages to a 4.4 kernel on the hypervisor? I'm running CentOS 7.2 hypervisors, but I could install a 4.4 kernel from elrepo 21:09:45 <b1airo> so once we upgrade to Mitaka in the next few months we'll be at the intersection point for the Trusty/Xenial change over 21:10:36 <b1airo> jonmills, i think we initially upped the kernel for some PCI passthrough issues 21:10:50 <jonmills> okay 21:11:10 <b1airo> then we ended up on 4.4 recently because it has longer support being from Xenial as opposed to the shorter Ubuntu releases 21:11:47 <b1airo> oneswig, as for the versions impacting performance, yes they do 21:12:19 <oneswig> b1airo: are there specific features/versions to look for? 21:12:23 <b1airo> i don't have any bisectional data for our setup but i have seen plenty of other reports of performance regressions in the late 3.xx kernels 21:12:43 <b1airo> e.g. there is an article somewhere on phoronix looking at this 21:13:27 <b1airo> we were also interested in the kernel's automatic numa balancing, which was introduced late in 3 i think 21:13:39 <b1airo> anyway, back to some numbers :-) 21:14:00 <b1airo> so on the baremetal hypervisor we see e.g.: 21:14:32 <martial> b1airo: are you working on a publication on those tests ? 21:14:59 <b1airo> hmm not sure if this is gonna paste well 21:15:09 <b1airo> i'll try, bare with... 21:15:10 <b1airo> Size LDA Align. Average Maximal 21:15:10 <b1airo> 160 176 4 6.0014 7.0252 21:15:10 <b1airo> 1600 1616 4 251.6930 258.5186 21:15:10 <b1airo> 5600 5616 4 554.8785 557.1577 21:15:10 <b1airo> 11200 11216 4 686.9769 688.3435 21:15:12 <b1airo> 22000 22000 4 761.3992 768.7692 21:15:14 <b1airo> 40000 40016 4 757.9887 758.6859 21:15:16 <b1airo> 80000 80016 4 757.7656 759.1828 21:15:18 <b1airo> 160000 160016 4 773.7305 773.9832 21:15:20 <b1airo> oh nice work hexchat 21:15:55 <b1airo> so the last size (160000) is about 90% mem usage 21:16:37 <b1airo> this is on E5-2680v3 CPUs 21:17:59 <b1airo> in a "naive" guest, i.e., no cpu or mem pinning and no topology exposure, so just a flat 24 CPUs/sockets, one NUMA node guest using 240GB ram 21:18:03 <georgem1> b1airo: can you share the config used and some setup instructions? it would be interesting to run on our systems as well 21:18:04 <b1airo> Size LDA Align. Average Maximal 21:18:04 <b1airo> 160 176 4 1.9916 3.3462 21:18:05 <b1airo> 1600 1616 4 209.0969 248.5604 21:18:05 <b1airo> 5600 5616 4 438.6423 441.8746 21:18:05 <b1airo> 11200 11216 4 545.3753 546.7031 21:18:05 <b1airo> 22000 22000 4 490.6886 494.0714 21:18:07 <b1airo> 40000 40016 4 487.2085 489.2111 21:18:09 <b1airo> 80000 80016 4 534.8873 544.3701 21:18:11 <b1airo> 160000 160016 4 606.1984 607.2593 21:18:26 <jmlowe> hi, sorry I'm late 21:18:34 <b1airo> georgem1, sure it's pretty easy, will get to that in a sec... 21:18:35 <oneswig> Interesting that you get maximum performance with such large blocks, or am I misreading the parameters? 21:18:42 <oneswig> hi jmlowe 21:19:19 <b1airo> oneswig, i'm no linpack guru, just going by the readme to set it up 21:19:38 <b1airo> but my colleagues have independently come up with the same sort of config 21:19:59 <b1airo> and here's the money shot - guest with cpu and mem pinning and topology exposed: 21:20:11 <trandles> blairo: is KSM enabled? 21:20:18 <b1airo> Size LDA Align. Average Maximal 21:20:18 <b1airo> 160 176 4 4.9478 7.2315 21:20:18 <b1airo> 1600 1616 4 247.3868 258.3221 21:20:18 <b1airo> 5600 5616 4 548.8194 551.7628 21:20:18 <b1airo> 11200 11216 4 667.9128 672.6079 21:20:19 <b1airo> 22000 22000 4 740.7616 752.1660 21:20:21 <b1airo> 40000 40016 4 756.2580 758.0187 21:20:23 <b1airo> 80000 80016 4 754.5520 754.7635 21:20:27 <b1airo> 160000 160016 4 767.8318 767.9426 21:20:27 <jmlowe> we should publish the tarball of the linpack we used to benchmark for acceptance 21:20:33 <b1airo> trandles, no KSM and THP disabled 21:21:05 <b1airo> so this is with regular 4K pages 21:21:09 <oneswig> Nice work b1airo 21:21:14 <anteaya> for reference if you need to post this kind of data into another channel please use a pastebin service, like paste.openstack.org 21:21:33 <anteaya> fine for your own meeting but others may not like it so much in other channels 21:21:48 <anteaya> and nice work b1airo 21:21:57 <jonmills> So is there a published cheatsheet to produce these nice results? 21:22:06 <b1airo> anteaya, sure i was expecting to have to do it to get it formatted nicely but looks ok here 21:22:13 <anteaya> it does so 21:22:46 <b1airo> jonmills, yes i think we can add this into the HPC whitepaper 21:23:04 <jonmills> cool 21:23:08 <oneswig> b1airo: sounds like a good idea to me, but how soon could you document? 21:23:10 <georgem1> cool indeed 21:23:16 <b1airo> but i'm also going to add something to the hypervisor tuning guide 21:23:32 <b1airo> georgem1, here's the config... 21:23:52 <b1airo> $ cat lininput_xeon64 21:23:52 <b1airo> Sample Intel(R) Optimized LINPACK Benchmark data file (lininput_xeon64) 21:23:52 <b1airo> Intel(R) Optimized LINPACK Benchmark data 21:23:52 <b1airo> 8 # number of tests 21:23:52 <b1airo> 160 1600 5600 11200 22000 40000 80000 160000 # problem sizes 21:23:53 <b1airo> 176 1616 5616 11216 22000 40016 80016 160016 # leading dimensions 21:23:57 <b1airo> 5 5 5 5 5 5 3 3 # times to run a test 21:23:59 <b1airo> 4 4 4 4 4 4 4 4 # alignment values (in KBytes) 21:24:19 <b1airo> using intel mkl benchmarks (l_mklb_p_2017.0.010.tgz) 21:24:49 <georgem1> thanks 21:25:22 <georgem1> I'll get off the train in 10 min :( 21:25:23 <b1airo> specifically benchmarks_2017/linux/mkl/benchmarks/linpack/runme_xeon64 21:25:48 <b1airo> this seems to be a nice test because it is a portable black box 21:25:51 <martial> so CPU only, no Phi ? 21:25:53 <oneswig> b1airo: what config do you apply outside of the OpenStack domain? 21:26:05 <anteaya> georgem1: you can read the channel log here: http://eavesdrop.openstack.org/irclogs/%23openstack-meeting/%23openstack-meeting.2016-10-04.log.html 21:26:18 <b1airo> we don't have any Phis, got a bunch of K80s though 21:26:21 <georgem1> I know, but it's not the same 21:26:28 <anteaya> georgem1: indeed 21:26:56 <georgem1> everybody got the hotel booked for Barcelona? 21:27:02 <b1airo> in relation to the hypervisor tuning guide i think we need a set of micro-benchmarks that can be commonly used to verify tuning 21:27:04 <jmlowe> I'm in the W 21:27:22 <b1airo> georgem1, yes - i think i was very lucky to get the last room at the SB 21:27:49 <georgem1> nice, I'll be 1 km away from the conference site 21:28:00 <b1airo> so i'm interested in collecting suggestions for such benchmarks 21:28:04 <oneswig> A good city for walking I believe :-) 21:28:36 <oneswig> b1airo: did you figure out your issues with gpu direct virtualised? 21:28:40 <rbudden> georgem1: yep, i’m booked the hilton 21:28:50 <b1airo> i haven't done anything with the tuning guide this cycle but i intend to chat with interested folks in barcelona about what should happen - i'm of the opinion it could because a repo of scripts and benchmarks 21:29:10 <b1airo> any prose can go into the admin/ops guide 21:29:39 <b1airo> so that brings us to: 21:29:55 <b1airo> #topic Barcelona activities and WG space etc 21:30:49 <b1airo> we heard back from the summit folks that a bigger room became available 21:31:31 <b1airo> so looks tentatively like we have a 100 person capacity room on Tues and Weds at (i think...) 2:45pm 21:31:37 <b1airo> oneswig, that right? 21:31:54 <oneswig> wasn't it one or the other? let me check 21:32:01 <georgem1> I'm out but I'll check the meeting logs later 21:32:37 <martial> by georgem1 21:32:41 <georgem1> bye 21:32:42 <b1airo> thanks georgem1 ! 21:32:48 <martial> (sigh can not type anymore) 21:33:02 <oneswig> Its 2x 40 minute slots - you're right 21:33:20 <oneswig> 2:15pm Tuesday and 2:15pm Wednesday 21:33:26 <oneswig> How does that look on the schedule? 21:34:00 <b1airo> i think it's ok - only issue is the clash with the long Ops session on Tues arvo 21:34:30 <oneswig> It's quite an unfortunate conflict 21:34:46 <b1airo> i know we have a few folks who will be torn between the two, but the Ops session goes most of the arvo so folks can always join us initially and then move on to that 21:35:00 <oneswig> Is the ops session close by? 21:35:12 <b1airo> good question... 21:35:38 <oneswig> One alternative - a smaller session (30 people) for committee meeting on Wednesday morning and a larger 100 session Wednesday afternoon for the BoF 21:35:44 <dfflanders> What is the divide between the two sesh agenda-wise? 21:35:46 <oneswig> Keeps Tuesday clear 21:36:14 <b1airo> it's still TBA by the looks of it 21:36:33 <oneswig> dfflanders: one's committee meeting, the other's BoF - procedural vs open house I'd guess 21:36:54 <dfflanders> :thumbs up: 21:37:25 <dfflanders> etherpad for listing these options and starting to plan? 21:37:51 <b1airo> you did create one already didn't you oneswig 21:37:57 <oneswig> 1 sec, I'll dig it out 21:39:35 <oneswig> not having much luck here... 21:39:53 <oneswig> I'll keep looking, it was agenda items for the committee meeting 21:40:46 <b1airo> should be in last week's log 21:40:49 <oneswig> #link Barcelona agenda items for committee meeting https://etherpad.openstack.org/p/scientific-wg-barcelona-agenda 21:40:56 <oneswig> thanks b1airo, exactly where it was 21:41:36 <b1airo> dfflanders, i figure this is what we'll flesh out properly on saturday 21:41:43 <oneswig> Not much there at present but it's a 40 minute meeting anyway 21:41:45 <b1airo> (b4 the summit) 21:42:11 <dfflanders> blair, +1 21:42:16 <oneswig> b1airo: sounds good to me 21:42:27 <b1airo> better scoot along here... 21:42:45 <b1airo> #Topic Scientific OpenStack BoF 21:43:29 <b1airo> we have had a few volunteers keen to do lightening talks in the BoF which is great, but I don't think we've actually started recording that to organise things yet? 21:43:51 <oneswig> When we've got the slot booked for the BoF, we should also get the lightning talks nailed - think it might need ot happen in that order 21:43:51 <b1airo> (at least i haven't :-) 21:44:15 <trandles> is this the Summit BoF or SC BoF? 21:44:22 <b1airo> fair enough, so should we discuss the time slots one last time then? 21:44:27 <oneswig> Perhaps whatever we do for the meeting, we could fix the bof for Wednesday afternoon in the 100-seater room 21:44:29 <b1airo> trandles, Summit 21:44:49 <b1airo> oneswig, yes of course 21:44:58 <b1airo> silly me 21:45:17 <oneswig> OK, if we do that I'll go to the list and do a "CFP"... 21:45:36 <b1airo> great1 21:45:46 <oneswig> #action oneswig (carried over from last week) fix the BoF timeslot and call for lightning talks 21:46:20 <dfflanders> If there is a particularly good lightning talk let's plan on getting it recorded somehow as BoF and WG rooms are not auto recorded... If nothing else I can grab a goPro and we can ad hoc it in the hall ;-) 21:46:37 <martial> :) 21:46:46 <b1airo> dfflanders, you could live tweet it :-P 21:47:06 <b1airo> #topic Subsidy for social (Thursday night) 21:47:17 <dfflanders> don't tease me blairo you know how I love a good live tweeting ;-) 21:47:36 <oneswig> OK we've got a small decision to take wrt the social evening 21:47:41 <oneswig> It's been really popular! 21:47:41 <b1airo> so far it's only oneswig's charms that have literally paid off 21:48:03 <oneswig> We have approaching 50 people down for it 21:48:07 <oneswig> And subsidy for 30 21:48:11 <b1airo> i have tried but my dell contact has gone mysteriously silent 21:48:38 <oneswig> Vendors eh. 21:49:00 <jmlowe> they do that sometimes even when you are trying to give them money 21:49:07 <b1airo> i'll give redhat one last push 21:49:12 <dfflanders> how much more money is needed per head? 21:49:29 <oneswig> We'd like to propose going for the bigger event and having people make a small contribution. 21:49:36 <oneswig> €40 per head 21:49:53 <b1airo> so we're currently short 800 ? 21:50:00 <oneswig> That's for food and drink (unspecified how much drink, cava + red wine + mineral water) 21:50:02 <oneswig> yes 21:50:22 <oneswig> might be just shy of that. 21:50:55 <b1airo> though likely to have more register if we were to promote it a little more, e.g. in the summit sessions 21:51:31 <oneswig> b1airo: true but eventually we'll hit a limit with the restaurant 21:52:04 <oneswig> Currently we're looking at roughly €16/head cost 21:52:13 <oneswig> How do people feel about that? 21:52:24 <jmlowe> I'm good with that 21:52:28 <martial> I am confused, was it 40 or 16 ? 21:52:37 <martial> I am okay with either, personally 21:52:43 <oneswig> 40 total, 16 after subsidy 21:53:08 <martial> might want to contact the people that already registered is all 21:53:28 <oneswig> martial: I'll mail all people with tickets and confirm once we are confirmed 21:53:37 <oneswig> should be any day I think 21:53:50 <rbudden> i’m good with that as well 21:54:05 <oneswig> excellent thanks all 21:54:21 <b1airo> i doubt anyone will mind, it's just a question of how we organise the logistics...? 21:55:00 <oneswig> Payment is going to be difficult, and receipts even more so. I'll work it with the restaurant 21:55:21 <b1airo> oneswig, were you thinking of having people pay on the evening? maybe we can do it via eventbrite? 21:55:31 <jmlowe> I don't like to carry cash, so a card friendly solution would be appreciated 21:55:50 <rbudden> jmlowe: +1 I rarely carry cash if at all possible 21:56:08 <rbudden> eventbrite seems like a simple solution to that 21:56:16 <rbudden> as b1airo suggested 21:56:23 <oneswig> b1airo: eventbrite might solve it. Good plan. Might make a second event for which we'd all get tickets 21:56:50 <martial> oneswig: yes a new replacement event might well be the better solution for this 21:57:28 <b1airo> yeah i think that would simplify it, might add a small transaction fee of course, but again i don't think any of us mind as it's mostly work paying :-) 21:57:39 <oneswig> Grand, I like it. I'll duplicate. Eventbrite take a small cut but it's worth it. 21:58:16 <oneswig> #action oneswig to mail ticketholders and arrange payment via eventbrite 21:58:16 <dfflanders> you could add a second ticket to the same eventbrite... end 'sales' on the first ticket. save minting new URL? /thinkingOutLoud 21:58:41 <b1airo> sounds like you have been around this rodeo dfflanders 21:58:43 <oneswig> dfflanders: sounds like a good plan but we'll want all the people currently with tickets to get first dibs 21:59:23 <b1airo> ok well we have a plan there, let's action it next week oneswig ? 21:59:35 <oneswig> OK b1airo agreed 21:59:39 <dfflanders> timecheck 21:59:40 <b1airo> give us another few days to see whether we can shake any change loose from the vendors 21:59:48 <b1airo> yep, that's a wrap! 21:59:54 <oneswig> thanks b1airo, great session 21:59:59 <dfflanders> +1 22:00:05 <oneswig> look forward to your report :-) 22:00:06 <rbudden> +1 22:00:08 <dfflanders> l8er ya'll 22:00:15 <b1airo> thanks all! 22:00:15 <rbudden> catch ya’ll later! 22:00:23 <trandles> later 22:00:28 <b1airo> #endmeeting