21:00:21 <oneswig> #startmeeting scientific-sig
21:00:22 <openstack> Meeting started Tue Jul  9 21:00:21 2019 UTC and is due to finish in 60 minutes.  The chair is oneswig. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:24 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:26 <openstack> The meeting name has been set to 'scientific_sig'
21:00:44 <oneswig> hello world?
21:00:56 <oneswig> #link agenda for today https://wiki.openstack.org/wiki/Scientific_SIG#IRC_Meeting_July_9th_2019
21:00:59 <rbudden> hello
21:01:14 <oneswig> Hey Bob, how's the Cinder work going?
21:02:08 <rbudden> Going good
21:02:20 <oneswig> As this meeting is running, I'm paring back OSDs from a Ceph cluster, one by one...
21:02:22 <rbudden> Migrated everything off GPFS onto our new NetApp filers
21:02:23 <rbudden> :)
21:02:35 <oneswig> NetApp, fancy...
21:02:47 <rbudden> I’ll be very happy to get off the broken GPFS drivers for Cinder
21:03:03 <oneswig> How are they breaking for you?
21:03:09 <oneswig> Showing signs of age?
21:03:20 <rbudden> bugs that haven’t been patched for quite some time
21:03:30 <rbudden> cinder retype is has been broken since Liberty
21:03:39 <rbudden> so we couldn’t live migrate out backend storage
21:04:19 <oneswig> That somewhat negates the advantage of network filesystems...
21:04:44 <rbudden> the GPFS driver doesn’t properly update all the UUIDs so you get orphaned and broken UUID entries in you database (among other problems)
21:05:39 <oneswig> That sounds unfortunate.  Is there nobody outside of IBM maintaining it (and nobody inside IBM either)?
21:05:50 <rbudden> not sure
21:05:57 <rbudden> i’ve been patching things myself as needed
21:06:09 <rbudden> Live migration was broken for awhile as well
21:06:41 <oneswig> So today's a good day overall...
21:06:48 <rbudden> oh yes
21:06:51 <rbudden> manually patched Nova for that until patched in libvirt 5.1 (which isn’t out for CentOS yet)
21:07:05 <rbudden> it’ll be nice to have things work smoothly :)
21:07:35 <rbudden> already seeing the benefits of dedup as we speak
21:07:45 <rbudden> is it just us today?
21:08:13 <oneswig> so far... trandles sent his apologies
21:08:31 <oneswig> martial also
21:08:35 <rbudden> gotcha
21:08:57 <rbudden> i have to jet about halfway to pick up my son
21:09:05 <oneswig> No problem
21:09:11 <oneswig> I've been working on an interesting problem with CephFS which might amuse you
21:09:56 <oneswig> I've got a Slurm deployment with compute partitions in Cambridge, Oxfordshire and Edinburgh with Manchester coming up.
21:10:17 <rbudden> nice
21:10:29 <rbudden> how are you liking CephFS?
21:10:39 <oneswig> I don't have a filer in any of those places so I made a scalable home dir using CephFS converged on the compute nodes
21:10:44 <oneswig> CephFS has potential.
21:11:10 <oneswig> It's deficient in some ways but does some remarkable things occasionally
21:11:24 <jmlowe> did you guys start playing with ceph on that surplus hardware?
21:11:37 <oneswig> For this system, I can make directories with local IO, eg /cambridge, /edinburgh etc.
21:11:53 <oneswig> a copy from one to the other sees the data jetting off up the country...
21:11:58 <oneswig> Hi jmlowe
21:12:01 <jmlowe> hi
21:12:10 <oneswig> surplus hardware?
21:12:27 <oneswig> Was that to rbudden?
21:12:45 <jmlowe> I think Goddard had some extra hardware rbudden was going to try some ceph experiments on
21:12:57 <oneswig> This project was more about making do with a shortage of hardware...
21:13:05 <rbudden> jmlowe: we haven’t yet, but it’s on the radar for possibly our TDS
21:13:38 <rbudden> jmlowe: today was Cinder backend from GPFS -> NetApp (mentioned earlier, before you joined)
21:13:53 <jmlowe> right, I figured
21:15:51 <oneswig> Are you involved in OSC's OpenOnDemand?  I've heard some good things about it.
21:17:10 <rbudden> I know PSC helped test OOD while they were developing it
21:17:26 <rbudden> we have some potential interest at Goddard
21:17:35 <rbudden> for Jupyter notebooks if I recall
21:17:56 <rbudden> i don’t have much hands on with it personally
21:19:52 <oneswig> Be interesting to hear if there's a broader pulse in the project.
21:20:41 <oneswig> jmlowe: rbudden: I forget, is either of you planning to go to Shanghai?
21:21:12 <jmlowe> no way I can swing that
21:21:42 <rbudden> no
21:21:56 <oneswig> Ah, too bad.
21:22:26 <jmlowe> any  word on the 2020 North American Summit?
21:22:28 <rbudden> eh, i wouldn’t want to deal with the paperwork
21:22:31 <rbudden> ;)
21:22:40 <rbudden> yeah, any news? i was curious if it was announced
21:23:11 <oneswig> The rate things are going, not sure if Brits will be allowed in the States by then :-)
21:23:30 <oneswig> I haven't heard anything, to answer your question
21:24:00 <rbudden> lol
21:24:10 <rbudden> China is just off limits completely…
21:24:23 <rbudden> other internation is possible, just more involved
21:24:34 <rbudden> hoping for somewhere nice for 2020!
21:25:07 <jmlowe> I can't use NSF funds to travel internationally so I have to use IU funds, comes with a different set of strings attached
21:26:12 <oneswig> The freedom of the private company is a double-edged sword :-)
21:26:19 <b1airo> hi gang
21:26:45 <b1airo> apologies for being tardy - school holiday head-f*&kery here :-/
21:26:54 <b1airo> and i'
21:26:57 <oneswig> Morning b1airo, glad you made it
21:27:02 <oneswig> #chair b1airo
21:27:03 <openstack> Current chairs: b1airo oneswig
21:27:10 <b1airo> i'm about to join a face to face, so will follow along from sidelines
21:27:38 <oneswig> Actually we didn't have much to cover - rbudden was talking Cinder migration and I was talking CephFS
21:27:55 <oneswig> How's your Bright system?
21:27:56 <b1airo> how'd you go with that CephFS metadata conundrum?
21:28:03 <b1airo> urgh
21:28:17 <oneswig> Aha, CephFS MDS - had a breakthrough from Dan van der Ster today.
21:28:45 <oneswig> So, it turns out you can pin metadata service to specific directories and MDS instances
21:28:48 <b1airo> nice. i assume you were already sharding the dirs?
21:29:00 <oneswig> It's not documented anywhere apart from a blog post
21:29:18 <b1airo> yes, was going to ask whether you'd tried that last week, but was too slow before you left
21:29:51 <oneswig> directories have affinity to local OSDs in the same region.
21:30:10 <oneswig> metadata was really painful but Dan's given me some good ideas on that.
21:30:22 <b1airo> oh, so you've got multiple data pools under this too?
21:30:52 <oneswig> yup, it's across multiple sites
21:31:03 <oneswig> #link CephFS subtree pinning https://ceph.com/community/new-luminous-cephfs-subtree-pinning/
21:31:19 <b1airo> yeah i'd say pinning is a good start, i think Dan found the auto-balancing is still a bit iffy
21:32:05 <oneswig> Are you using Ceph at nesi?
21:32:22 <b1airo> no not at the moment
21:32:26 <rbudden> sorry to interrupt, gotta jet to pick up my son. have a good day/evening everyone!
21:32:30 <b1airo> a couple of people keen on trying it though
21:32:34 <oneswig> same to you rbudden
21:33:17 <oneswig> b1airo: so what's the OpenStack takeover looking like now?
21:35:18 <oneswig> BTW did you see the news on the Zoom vulnerability?
21:35:43 <b1airo> a slow burn i'm afraid. we're taking some slow steps, but won't have much latitude to make major changes/inroads until we setup NeSI.3, which has been pushed out two years by the Gov
21:36:11 <b1airo> yes, only impacts people using Apple Mail client though i believe?
21:36:34 <oneswig> I believe so.  Which is unfortunate for me...
21:37:03 <b1airo> i figured you'd be using an emacs plugin... ;-)
21:37:32 <oneswig> ha - emacs... great OS, shame about the editor
21:38:05 <b1airo> careful with that trash talk around here o_0
21:38:28 <oneswig> M-x undo!
21:38:50 <b1airo> dad joke!
21:41:46 <oneswig> anyway - we are pretty quiet on content today.  I'm thinking of other OS-related things that have happened here this week.  I've been down the rabbithole with some weird fluentd behaviour.  Not very exciting
21:44:04 <oneswig> Any other business?
21:46:03 <oneswig> OK, let's close for the day - thanks guys
21:46:07 <oneswig> #endmeeting