09:00:40 #startmeeting scientific_wg 09:00:42 Meeting started Wed Jul 6 09:00:40 2016 UTC and is due to finish in 60 minutes. The chair is oneswig. Information about MeetBot at http://wiki.debian.org/MeetBot. 09:00:43 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 09:00:46 The meeting name has been set to 'scientific_wg' 09:01:24 #link agenda for today (such as it is) https://wiki.openstack.org/wiki/Scientific_working_group#IRC_Meeting_July_6th_2016 09:01:33 Hello ... 09:01:39 #topic roll call 09:01:48 Anyone remember the new time? 09:02:06 Yes, good morning. Peter here. 09:02:10 Good morning oneswig 09:02:33 Greetings! 09:03:20 I had a message from Blair, he's likely to be joining late 09:03:34 good morning, this is a much more civilised time for us UK people :) 09:04:04 Great, more the merrier 09:04:19 Hi james@sanger 09:05:46 OK, lets get going 09:06:35 I just messaged with bauzas and hopefully he'll be joining later to talk Blazar, let's defer discussion on that if we can 09:07:13 #topic Bare metal 09:07:35 Do we have anyone else with an interest here (apart from me)? 09:07:54 interested but no experience yet 09:08:03 same here 09:08:23 the previous minutes linked to a youtube vid 09:08:24 (other than using RH Director to deploy overclouds) 09:08:32 A lot of interest for Chameleon obviously 09:08:43 can you summarise that vid quickly? 09:08:51 Hello! 09:08:53 and experience as well 09:09:22 ptrlv: can you remind me of the link... was it the chameleon talk? 09:09:35 yes 09:09:36 dariov: hi, welcome 09:09:48 http://eavesdrop.openstack.org/meetings/scientific_wg/2016/scientific_wg.2016-06-28-21.01.html 09:10:14 Ah, right, Kate Keahey's talk from the summit 09:11:09 Some of the interesting follow-up from that talk I heard of was the issues with scheduling access (blazar) and issues with managing bare metal serial consoles. 09:11:51 priteau: do you know if there was any progress on serial consoles? I still don't have them managed through Ironic on our deployment 09:12:57 oneswig: I saw a little activity on the Gerrit topics in the past two weeks, but not much. On our side we just shipped new features for Chameleon and will now get back to evaluate serial console 09:13:44 I may have more information by the end of July, once we are able to test the patches proposed on Gerrit 09:13:56 Sorry I'm late - Central Line ate my homework 09:14:05 priteau: OK thanks I'll keep tracking that myself, would hope to have some more experience to report in about the same timeframe 09:14:15 verdurin: hi! 09:15:34 We've had some curious problems this week with bare metal and Dell kit that might be worth sharing 09:16:39 Using RH director (ie TripleO), the Ironic python agent ramdisk was unable to cope with the long start time of the raid 09:17:06 Also have some projects underway to configure Dell BIOS and RAID config ... via Ansible... 09:18:00 Also came across a really excellent tool for identifying discrepancies in Ironic introspection data using a tool called cardiff 09:18:03 oneswig: if you can share those, perhaps they can be generalized e.g. to Lenovo too? 09:18:33 We would love to learn more about your experience with BIOS config 09:18:42 verdurin: it's using the iDRAC interface, I suspect that's unique to Dell alas? 09:19:04 could it be done using the redfish interfaces? 09:19:11 oneswig: Are you using the PXE DRAC Driver? 09:19:18 oneswig: still useful given we likely procure Dell stuff 09:19:38 priteau: share and enjoy! Will keep you updated. Yes we (sometimes) use pxe_drac but it's not solid 09:20:17 Sadly redfish support is embryonic at best for many vendors 09:20:25 apdibbo: Is that the interface for OCP? Sounds good to me. I'm building the modules on Ironic's python-dracclient but they ought to be transferrable 09:20:46 What's the current link to cardiff? I found this with Google, but the GitHub link is a 404: https://www.redhat.com/archives/rdo-list/2015-April/msg00188.html 09:20:59 I think it's part of python-hardware 09:21:36 oneswigh: we are using the redfish interface to collect some info, it was initially from OCP but has been picked up pretty well by Dell and Supermicro 09:21:49 then it's hardware-cardiff 09:22:14 apdibbo: thanks I'll look out for that, is it also possible to modify settings through redfish on those bioses? 09:22:43 oneswig: I believe so 09:23:00 apdibbo: Interesting, will check 09:23:14 AOB for bare metal? 09:23:20 #link https://github.com/redhat-cip/hardware 09:24:17 OK lets roll on 09:24:24 #topic parallel filesystems 09:25:10 I don't have anything myself here. We intend to connect our deployment to a new (external) lustre store later today. The two have not met yet. 09:26:09 evening 09:26:10 we have not connected Openstack and Lustre yet either, concerns over security i.e. no multi-tenant in Lustre (same concerns as accessing NFS from instances) 09:26:16 Hi b1airo 09:26:19 #chair b1airo 09:26:20 Current chairs: b1airo oneswig 09:27:02 really need to get my irc bouncer setup again - missing the history 09:27:24 speaking of HPFS by the sound of it? 09:27:25 daveh: Right. Our policy is to put the filesystem on a provider network visible only to the project using it 09:27:38 oneswig, ours too 09:27:49 so... each filesystem is used by a single project? we have shared filesystems which makes life... more interesting 09:27:50 hi sorry for being late 09:27:51 b1airo: yes, just covered bare metal and cooling down with hpfs :-) 09:28:09 on a colliding meeting 09:28:14 daveh: agreed - not my area alas 09:28:21 aloga: hi, thanks for coming 09:28:25 FYI we are using GPFS 09:28:32 with some nasty and ugly workarounds 09:28:39 aloga: In OpenStack? Can you share the config? 09:28:58 has anyone got their hands dirty with AWS EFS announce recently? 09:29:01 oneswig: we're running sun grid engine worker nodes as OpenStak instances 09:29:11 this is in Indigo aloga ? 09:29:15 b1airo: nope 09:29:24 b1airo: this is internally at out CSIC infrastructure 09:29:28 ptrlv, planning to do so in the future 09:29:30 ptrlv: not on my radar unfortunately 09:29:46 experimentally we've tried a similar per-project provider network isolation approach with GPFS 09:29:47 #link https://aws.amazon.com/blogs/aws/amazon-elastic-file-system-shared-file-storage-for-amazon-ec2/ 09:29:53 b1airo: we are a scientific datacenter: WLCG, astrophysics, bioinformatics, engineering, etc. 09:30:03 ptrlv: hopefully by september I think 09:30:07 b1airo: we have two different infrastructures, an HPC one and a HTC one 09:30:19 the HTC one is running on top of OpenStack since long 09:30:32 with access to our GPFS cluster 09:30:56 aloga: Do the two infrastructures share a common gpfs filesystem? 09:31:20 oneswig: no 09:31:31 oneswig: the HPC one goes over infiniband 09:31:48 aloga: is there a need to share datasets between them, if so how? 09:32:22 oneswig: no, they are separated, as they have different entry policies 09:32:42 verdurin, you said experimentally - did it work out? 09:34:52 b1airo: perhaps he's gone on the central line again? :-) 09:35:13 (perhaps you missed that bit...) 09:35:16 oh, flakely connection? 09:35:31 "flakely"... what? 09:35:35 :-) 09:35:36 homework excuse! 09:36:14 so aloga, is your hpc using openstack too? 09:36:14 * bauzas waves 09:36:20 Shall we move on? 09:36:22 Hi bauzas 09:36:29 b1airo: nope 09:36:37 thanks for joining 09:36:55 #topic accounting and scheduling 09:37:01 b1airo: although there are plans on the sight :) 09:37:31 We have had some interesting discussions recently 09:37:45 and they have often included Blazar for resource reservation management 09:38:22 Blue sky stuff: Sometimes with how it might (in theory) be combined with preemtible instances (OPIE) to good effect 09:38:36 b1airo: it works in testing, we're reluctant to roll out more widely yet 09:38:46 FWIW, Blazar would need a certain amount of effort for catching-up with the latest Nova API 09:39:00 aloga, sounds good - perhaps an announcement when the summit comes to you (though i guess you are in madrid?) 09:39:47 b1airo: actually 400km north :-P 09:39:52 b1airo: Santander, over the sea 09:40:00 priteau: did you have any success with moving your patches upstream for blazar? 09:40:45 bauzas: how far out of date do you think it is? 09:40:45 oneswig: I haven't had the time to push any yet, but I am working on this now 09:41:43 oneswig: so, there are 2 backends 09:42:03 oneswig: one is basically using aggregates, and those haven't really changed 09:42:24 oneswig: that's the backend you use when you want to do a full host reservation 09:42:42 oneswig: the problem is with the other backend, used for instance reservation 09:43:13 oneswig: it was using an old compute v2.0 extension mechanism that has been deprecated 09:43:32 (in Chameleon we use the first backend) 09:43:36 priteau: is this problem familiar to you and is it part of your changes? 09:43:36 so, tbh, I see the functionality broken 09:44:18 oneswig: one way to solve the problem could be to deprecate that Blazar API resource, and only allow to reserve hosts 09:44:35 oneswig: We are not using the instance reservation feature at all, and haven't made any fixes to it 09:44:44 particularly if the Chameleon project isn't really using ity 09:45:05 bauzas: so for bare metal hosts we'd use the first backend only, and that's effectively partitioning off an entire nova compute node in the virtualised case? 09:45:22 bauzas: Do you know if OPNFV is using or intending to use instance reservation? 09:45:43 oneswig: I'd say reserving an host would make more sense from a baremetal PoV 09:46:01 even if the Ironic driver conceptualizes that differently 09:46:28 priteau: I haven't heard about OPNFV intending to use Blazar yet 09:46:43 i think the instance reservation use-case would be the most popular generally 09:47:04 i.e., for general purpose science clouds 09:47:51 that's a valid concern 09:47:56 there is often a need to provide some fraction of the resource as guaranteed capacity to projects doing time blocked things like training courses and so forth 09:48:09 I'm just expressing the point that I'm pretty sure it won't work with the latest Nova release :) 09:48:18 bauzas: there was some discussion on the constraint that Blazar effectively manages its own partitioned group of nova computes. I think I'm getting to understand the issue behind that now but is there a way around it so that we might be able to get higher utilisation? 09:48:50 oneswig: well, the problem is that you're in a cloud, right? 09:48:56 at the moment we achieve this in NeCTAR by manually manipulating host aggregates, but that really sucks because it puts an admin in the critical path and means resources are usually held idle for a long time 09:49:03 oneswig: so you can't really control which workloads will run in the future 09:49:37 bauzas, yes understood re. api compatibility, thanks for raising that 09:49:44 oneswig: that's why the host reservation backend used aggregates, to isolate a full group of hosts we could control 09:50:06 that said, the instance reservation backend was working totally differently 09:50:20 without really isolating groups of hosts 09:50:23 bauzas: right, but it reminds me of other circumstances where a block of resource gets scheduled on demand, somehow we get there... 09:50:59 the instance reservation backend was basically spawning an instance, and then shelving it 09:51:25 and then unshelving it on purpose (when the lease started) 09:52:05 oh wow, and that worked? 09:53:09 bauzas: there's been some discussion previously on the possibility of alternatively preempting running instances according to some policy to make way for a resource reservation. Any hope for that? 09:53:23 seems like there must be some overlap between the scheduler mechanism opie is proposing for preemptible instances and what would be needed for reservations 09:53:43 bauzas: but that would lock all the resources even when the instance is shelved, right? 09:53:46 b1airo: well, the problem with shelve offload is that nova frees up the resource usage 09:54:01 b1airo: that's not a problem for Nova, but that's a problem for Blazar 09:54:39 b1airo: that means that when you unshelve the instance, you would then reschedule the instance to another host 09:54:55 with no guarantee that you'll have enough capacity by this time 09:55:30 dariov: resources are locked only when you shelve an instance, not when you shelve offload it 09:56:08 dariov: you could say "easy, let's just shelve the instance without offloading it" 09:56:22 may as well just suspend it then 09:56:44 dariov: but then, the host would need to support the added capacity for the instance that is going to be used only in 1 month 09:57:22 bauzas: yep, that was my concern 09:57:25 dariov: suspend doesn't really differ from the active state, except that your VM is paused 09:57:59 i think we can agree shelving might have been a convenient hack, but it's not an adequate solution to this problem 09:58:16 that's why I'm particularly thinking that it's kinda necessary to allocate a certain amount of resources (hosts or whatever else) for Blazar 09:58:36 yes, but even if you reserve those resources for blazar, they are going to be infrautilized 09:58:45 because you need to control and guarantee that you'll be able to spawn the instances when the lease will start 09:58:57 as there will be periods when the hosts will be empty, right? 09:58:58 aloga: I agree 09:59:23 Unfortunately we are out of time 09:59:23 aloga: well, of course, that depends on your lease management system 09:59:37 bauzas, for us i think if we could backfill those hosts with preemptible workload we'd be happy for a while, but ultimately i think both reservations and preemption are things we'd like to see built into nova scheduler 09:59:45 bauzas: any thoughts on that? 09:59:46 bauzas: that’s where preemptible instances come to the rescue 09:59:51 b1airo: +1 10:00:34 we struggle to get our resources near 100% utilization 10:00:39 b1airo: well, that's an open point that can't be solved by the time we have :) 10:00:48 interesting food for thought, but doing anything with nova scheduler has seemed fraught for a few cycles at least 10:00:57 Sorry all, we must release the channel - perhaps shelve the discussion... 10:01:07 nice one oneswig 10:01:08 shelve offload or just shelve? :P 10:01:18 Thanks all! Until next time 10:01:23 #endmeeting