21:00:33 <oneswig> #startmeeting scientific-sig
21:00:34 <openstack> Meeting started Tue Dec  8 21:00:33 2020 UTC and is due to finish in 60 minutes.  The chair is oneswig. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:35 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:37 <openstack> The meeting name has been set to 'scientific_sig'
21:00:46 <oneswig> ... back in the room
21:00:53 <trandles> Hi oneswig
21:01:04 <oneswig> Hey trandles, long time no see
21:01:07 <oneswig> what's new?
21:01:12 <martial> Hi Stig
21:01:15 <trandles> mostly lurking today, also on a webex at the same time :(
21:01:17 <martial> Hi Tim
21:01:21 <oneswig> Hey martial
21:01:21 <trandles> Hi Martial
21:01:23 <oneswig> #chair martial
21:01:24 <openstack> Current chairs: martial oneswig
21:02:06 <oneswig> CentOS 8.3 turned up in the appstream and a few things broke - qemu was the first (something about gcrypt)
21:02:27 <oneswig> hopefully it'll settle down soon
21:05:04 <oneswig> The CentOS move to being a beta for RHEL - understandable but I've heard from a few sites that have concern about this change.
21:08:23 <oneswig> martial: does Datamachines base on CentOS?
21:09:42 <martial> really depends on the project
21:10:07 <martial> We have a few CentOS based installs, but a big set of our cluster is Ubuntu server
21:10:35 <martial> after that it is a matter of Ansible-ize the host for what we need
21:11:59 <oneswig> Nice to keep the portability between distros like that
21:12:58 <martial> well the next steps are installs of an orchestration mechanism for us
21:13:08 <trandles> we tend to do everything with RHEL or a RHEL derivative so I was really hoping to use CentOS for our VMs just to avoid licensing costs
21:13:40 <trandles> our RHOSP license price difference between unlimited RHEL clients vs. non was a BIG difference
21:14:31 <oneswig> does this change your thinking trandles?
21:14:57 <trandles> not a lot...we were leaving dealing with licenses up to our users
21:15:13 <trandles> it might change what we provide and support for images
21:15:28 <trandles> but not by too much...
21:16:32 <trandles> I have heard from 3 different friends, all working at different places, that people are already talking about rolling their own RH-based distro
21:16:38 <martial> reading the PR I am a little confused if "Stream" is going to be that different
21:16:42 <trandles> since AFAIK the RH source will still be on github
21:17:18 <trandles> we're licensed RHEL on all of our baremetal systems
21:17:18 <oneswig> I'll be interested to see how it affects things like RDO - might cause that to be more stream-like too
21:18:25 <oneswig> trandles: is openstack-ansible still dual centos and ubuntu?
21:19:14 <trandles> oneswig: I don't know, I don't do ubuntu. It still works with centos last I checked (~10 months ago)
21:20:48 <oneswig> ok, interesting.
21:21:09 <martial> not sure, I used Ubuntu for Openstack Kolla Ansible
21:24:14 <oneswig> So what else is new?
21:24:36 <trandles> not too much from me that would be interesting to this group
21:24:59 <martial> quiet month usually December :)
21:25:18 <trandles> I'm neck deep in end-of-year program and project reviews. i.e. nothing fun is happening :(
21:25:42 <oneswig> Given what Covid did to the summer, a lot of our team are sitting on big backlogs of PTO.  December's going to be very quiet over here.
21:27:48 <trandles> I can't wait for the second half of 2021 when about 50% of the western world tries to take holidays at the same time.
21:28:02 <martial> a little more chaos here but a lot quieter than in the past
21:28:13 <martial> (but then again: December :) )
21:28:55 <trandles> Our project budgets are getting blown up because no one is taking their PTO. We have lots of people on paper working more than 1 FTE for the year, me included. I've used a total of 22 hours of PTO in 2020.
21:29:33 <oneswig> For an American that's, like, half of it, right? :-)
21:29:55 <trandles> LOL if you're an American lucky enough to have a job that provides you with PTO
21:30:10 <oneswig> yee-ouch
21:30:25 <trandles> When I was growing up my dad got a total of 40 hours PTO and was only allowed to use it between November and March.
21:31:22 <oneswig> poor guy.  I expect you did what you could to make those 5 days extra special :-)
21:31:28 <trandles> I get 120 hrs vacation PTO/year, plus 10hrs/month sick time
21:32:04 <trandles> I have ~600 hours of full-paid sick time, just in case a disaster happens
21:33:30 <oneswig> On a different subject, we've been having some really interesting time with qemu and backing image performance.
21:34:28 <oneswig> We have a project with some beefy hypervisors, but in the initial configuration of 10 HDDs for VM image backing the performance was, as they say in France, comme le chien.
21:34:53 <oneswig> I mean, truly awful when observed from the VM.
21:35:17 <trandles> RAID or erasure?
21:36:10 <oneswig> It's RAID 10 but using LVM.  Tried a couple of configurations of that with striping aiming to improve utilisation across all drives.
21:36:30 <oneswig> I think the next step was to move to md for the device management.
21:36:54 <oneswig> Cherry on the cake was to format /var/lib/nova with xfs instead of ext4
21:37:15 <trandles> Has anyone tried ZFS?
21:37:44 <oneswig> My colleague Michal found an ancient bug on this which apparently still applies
21:37:56 <oneswig> #link https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/550409
21:37:57 <openstack> Launchpad bug 550409 in libvirt (Ubuntu) "Slow disk IO for all guests with all disc modes and types." [Low,Confirmed]
21:38:20 <oneswig> ZFS?  No, not here.  People who use it seem to like it.  Almost like a cult thing
21:39:38 <trandles> ZFS backs all of our large lustre filesystems. I used ZFS + kernel NFS (not ZFS NFS) for 3PB of local storage on a cluster. I'm thinking of repurposing those storage nodes for a prototype RHOSP deployment I'm doing.
21:40:37 <oneswig> ZFS and Lustre is a popular combination
21:41:05 <trandles> I like it because our storage gurus all know it very well. When I cock up something I can ask for help. ;)
21:41:10 <oneswig> Gives Lustre the resiliency it otherwise wouldn't have...
21:42:21 <trandles> The other meeting is done. I can focus.
21:42:38 <trandles> Of general interest may be that we just started an install today of OpenShift on OpenStack
21:43:23 <oneswig> Ooh, fancy.  How well do the two integrate?  Is it similar to Magnum?
21:43:39 <trandles> The OpenStack is CentOS 8 + RDO. One of our k8s wizards is handling the OpenShift so I'm looking over his virtual shoulder and learning.
21:44:01 <oneswig> :q
21:44:07 <oneswig> oops, wrong window :-)
21:44:30 <trandles> RHAT supports that deployment model so it's what we're using.
21:45:09 <trandles> One of our partners from another lab is pushing us to use portworx for storage. Anyone have experience? I'm pushing back since we have no portworx experience internally.
21:45:47 <trandles> martial: I thought maybe Data Machines might have looked at Portworx
21:46:10 <oneswig> Mind out with their performance figures, they have a benchmark mode that's relaxed about committing to disk.  You probably don't want to run your databases in that mode.
21:46:21 <martial> name rings a bell
21:46:37 <oneswig> Although I think in general it performs well anyway.  I don't have operational experience.
21:47:19 <martial> I remember seeing it in the past but no direct use here; Ceph or min.io mostly
21:59:03 <oneswig> Amusing to follow the thread Jon Mills posted on hn to the point where someone recommends Oracle Linux as an alternative to CentOS... hmmm...
21:59:13 <martial> ouch
21:59:23 <oneswig> Nearly at time, final comments?
21:59:23 <martial> reaching the end of the hour too
22:00:01 <oneswig> Ok y'all, time to wrap.
22:00:05 <oneswig> #endmeeting