21:00:21 <oneswig> #startmeeting scientific-sig 21:00:22 <openstack> Meeting started Tue Jul 9 21:00:21 2019 UTC and is due to finish in 60 minutes. The chair is oneswig. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:00:24 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:00:26 <openstack> The meeting name has been set to 'scientific_sig' 21:00:44 <oneswig> hello world? 21:00:56 <oneswig> #link agenda for today https://wiki.openstack.org/wiki/Scientific_SIG#IRC_Meeting_July_9th_2019 21:00:59 <rbudden> hello 21:01:14 <oneswig> Hey Bob, how's the Cinder work going? 21:02:08 <rbudden> Going good 21:02:20 <oneswig> As this meeting is running, I'm paring back OSDs from a Ceph cluster, one by one... 21:02:22 <rbudden> Migrated everything off GPFS onto our new NetApp filers 21:02:23 <rbudden> :) 21:02:35 <oneswig> NetApp, fancy... 21:02:47 <rbudden> I’ll be very happy to get off the broken GPFS drivers for Cinder 21:03:03 <oneswig> How are they breaking for you? 21:03:09 <oneswig> Showing signs of age? 21:03:20 <rbudden> bugs that haven’t been patched for quite some time 21:03:30 <rbudden> cinder retype is has been broken since Liberty 21:03:39 <rbudden> so we couldn’t live migrate out backend storage 21:04:19 <oneswig> That somewhat negates the advantage of network filesystems... 21:04:44 <rbudden> the GPFS driver doesn’t properly update all the UUIDs so you get orphaned and broken UUID entries in you database (among other problems) 21:05:39 <oneswig> That sounds unfortunate. Is there nobody outside of IBM maintaining it (and nobody inside IBM either)? 21:05:50 <rbudden> not sure 21:05:57 <rbudden> i’ve been patching things myself as needed 21:06:09 <rbudden> Live migration was broken for awhile as well 21:06:41 <oneswig> So today's a good day overall... 21:06:48 <rbudden> oh yes 21:06:51 <rbudden> manually patched Nova for that until patched in libvirt 5.1 (which isn’t out for CentOS yet) 21:07:05 <rbudden> it’ll be nice to have things work smoothly :) 21:07:35 <rbudden> already seeing the benefits of dedup as we speak 21:07:45 <rbudden> is it just us today? 21:08:13 <oneswig> so far... trandles sent his apologies 21:08:31 <oneswig> martial also 21:08:35 <rbudden> gotcha 21:08:57 <rbudden> i have to jet about halfway to pick up my son 21:09:05 <oneswig> No problem 21:09:11 <oneswig> I've been working on an interesting problem with CephFS which might amuse you 21:09:56 <oneswig> I've got a Slurm deployment with compute partitions in Cambridge, Oxfordshire and Edinburgh with Manchester coming up. 21:10:17 <rbudden> nice 21:10:29 <rbudden> how are you liking CephFS? 21:10:39 <oneswig> I don't have a filer in any of those places so I made a scalable home dir using CephFS converged on the compute nodes 21:10:44 <oneswig> CephFS has potential. 21:11:10 <oneswig> It's deficient in some ways but does some remarkable things occasionally 21:11:24 <jmlowe> did you guys start playing with ceph on that surplus hardware? 21:11:37 <oneswig> For this system, I can make directories with local IO, eg /cambridge, /edinburgh etc. 21:11:53 <oneswig> a copy from one to the other sees the data jetting off up the country... 21:11:58 <oneswig> Hi jmlowe 21:12:01 <jmlowe> hi 21:12:10 <oneswig> surplus hardware? 21:12:27 <oneswig> Was that to rbudden? 21:12:45 <jmlowe> I think Goddard had some extra hardware rbudden was going to try some ceph experiments on 21:12:57 <oneswig> This project was more about making do with a shortage of hardware... 21:13:05 <rbudden> jmlowe: we haven’t yet, but it’s on the radar for possibly our TDS 21:13:38 <rbudden> jmlowe: today was Cinder backend from GPFS -> NetApp (mentioned earlier, before you joined) 21:13:53 <jmlowe> right, I figured 21:15:51 <oneswig> Are you involved in OSC's OpenOnDemand? I've heard some good things about it. 21:17:10 <rbudden> I know PSC helped test OOD while they were developing it 21:17:26 <rbudden> we have some potential interest at Goddard 21:17:35 <rbudden> for Jupyter notebooks if I recall 21:17:56 <rbudden> i don’t have much hands on with it personally 21:19:52 <oneswig> Be interesting to hear if there's a broader pulse in the project. 21:20:41 <oneswig> jmlowe: rbudden: I forget, is either of you planning to go to Shanghai? 21:21:12 <jmlowe> no way I can swing that 21:21:42 <rbudden> no 21:21:56 <oneswig> Ah, too bad. 21:22:26 <jmlowe> any word on the 2020 North American Summit? 21:22:28 <rbudden> eh, i wouldn’t want to deal with the paperwork 21:22:31 <rbudden> ;) 21:22:40 <rbudden> yeah, any news? i was curious if it was announced 21:23:11 <oneswig> The rate things are going, not sure if Brits will be allowed in the States by then :-) 21:23:30 <oneswig> I haven't heard anything, to answer your question 21:24:00 <rbudden> lol 21:24:10 <rbudden> China is just off limits completely… 21:24:23 <rbudden> other internation is possible, just more involved 21:24:34 <rbudden> hoping for somewhere nice for 2020! 21:25:07 <jmlowe> I can't use NSF funds to travel internationally so I have to use IU funds, comes with a different set of strings attached 21:26:12 <oneswig> The freedom of the private company is a double-edged sword :-) 21:26:19 <b1airo> hi gang 21:26:45 <b1airo> apologies for being tardy - school holiday head-f*&kery here :-/ 21:26:54 <b1airo> and i' 21:26:57 <oneswig> Morning b1airo, glad you made it 21:27:02 <oneswig> #chair b1airo 21:27:03 <openstack> Current chairs: b1airo oneswig 21:27:10 <b1airo> i'm about to join a face to face, so will follow along from sidelines 21:27:38 <oneswig> Actually we didn't have much to cover - rbudden was talking Cinder migration and I was talking CephFS 21:27:55 <oneswig> How's your Bright system? 21:27:56 <b1airo> how'd you go with that CephFS metadata conundrum? 21:28:03 <b1airo> urgh 21:28:17 <oneswig> Aha, CephFS MDS - had a breakthrough from Dan van der Ster today. 21:28:45 <oneswig> So, it turns out you can pin metadata service to specific directories and MDS instances 21:28:48 <b1airo> nice. i assume you were already sharding the dirs? 21:29:00 <oneswig> It's not documented anywhere apart from a blog post 21:29:18 <b1airo> yes, was going to ask whether you'd tried that last week, but was too slow before you left 21:29:51 <oneswig> directories have affinity to local OSDs in the same region. 21:30:10 <oneswig> metadata was really painful but Dan's given me some good ideas on that. 21:30:22 <b1airo> oh, so you've got multiple data pools under this too? 21:30:52 <oneswig> yup, it's across multiple sites 21:31:03 <oneswig> #link CephFS subtree pinning https://ceph.com/community/new-luminous-cephfs-subtree-pinning/ 21:31:19 <b1airo> yeah i'd say pinning is a good start, i think Dan found the auto-balancing is still a bit iffy 21:32:05 <oneswig> Are you using Ceph at nesi? 21:32:22 <b1airo> no not at the moment 21:32:26 <rbudden> sorry to interrupt, gotta jet to pick up my son. have a good day/evening everyone! 21:32:30 <b1airo> a couple of people keen on trying it though 21:32:34 <oneswig> same to you rbudden 21:33:17 <oneswig> b1airo: so what's the OpenStack takeover looking like now? 21:35:18 <oneswig> BTW did you see the news on the Zoom vulnerability? 21:35:43 <b1airo> a slow burn i'm afraid. we're taking some slow steps, but won't have much latitude to make major changes/inroads until we setup NeSI.3, which has been pushed out two years by the Gov 21:36:11 <b1airo> yes, only impacts people using Apple Mail client though i believe? 21:36:34 <oneswig> I believe so. Which is unfortunate for me... 21:37:03 <b1airo> i figured you'd be using an emacs plugin... ;-) 21:37:32 <oneswig> ha - emacs... great OS, shame about the editor 21:38:05 <b1airo> careful with that trash talk around here o_0 21:38:28 <oneswig> M-x undo! 21:38:50 <b1airo> dad joke! 21:41:46 <oneswig> anyway - we are pretty quiet on content today. I'm thinking of other OS-related things that have happened here this week. I've been down the rabbithole with some weird fluentd behaviour. Not very exciting 21:44:04 <oneswig> Any other business? 21:46:03 <oneswig> OK, let's close for the day - thanks guys 21:46:07 <oneswig> #endmeeting