21:00:01 <oneswig> #startmeeting scientific-sig
21:00:02 <openstack> Meeting started Tue Apr  2 21:00:01 2019 UTC and is due to finish in 60 minutes.  The chair is oneswig. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:03 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:06 <openstack> The meeting name has been set to 'scientific_sig'
21:00:14 <oneswig> Hello o/
21:00:22 <rbudden> hello
21:00:40 <oneswig> #link agenda for today https://wiki.openstack.org/wiki/Scientific_SIG#IRC_Meeting_April_2nd_2019
21:00:49 <oneswig> Hey Bob!
21:01:05 <oneswig> martial_: you there?
21:01:12 <rbudden> how’s it going?
21:01:32 <oneswig> very good thanks.  A little distracted - baking a cake currently
21:01:39 <oneswig> How is GSFC?
21:01:46 <martial_> Hello Stig
21:01:56 <oneswig> Hi martial_
21:01:59 <oneswig> #chair martial_
21:02:03 <openstack> Current chairs: martial_ oneswig
21:02:18 <janders> g'day!
21:02:27 <oneswig> morning janders
21:02:32 <rbudden> it’s going well, getting settled in a bit finally and hopefully lending Jonathan a hand
21:02:32 <oneswig> bright and early as always!
21:02:53 <oneswig> rbudden: cool, I'm sure he's got some good projects for you to work on.
21:03:16 <oneswig> #topic HPC Containers event at ISC
21:03:24 <rbudden> yeah, we’ve got a lot of things planned, so i’ll be fairly busy with openstack work
21:03:25 <oneswig> martial_: tell us all about it
21:03:41 <martial_> so I am on the program committee for HPCW
21:03:43 <oneswig> rbudden: nice work if you can get it :-)
21:04:09 <martial_> #link https://easychair.org/cfp/hpcw2019
21:04:21 <martial_> we invite people to submit nice articles of course :)
21:04:33 <oneswig> martial_: what is HPCW and who's involved?
21:04:37 <martial_> Christian Kniep (some of you have met him at SC19 last year)
21:04:48 <martial_> (formerly at Docker)
21:05:21 <martial_> `5th High Performance Containers Workshop - In conjunction with ISC HIGH PERFORMANCE 2019` (Frankfurt, Germany) (June)
21:05:35 <janders> I just emailed the link to my precision medicine gurus - they heavily use containers
21:05:54 <janders> would make a great preso - if they can make it (I probably won't attend the ISC)
21:05:57 <martial_> Abstracts due Apr 20
21:06:48 <martial_> The CFP is at the link I just provided, which also gives you additional details
21:06:57 <oneswig> martial_: is it part of ISC or kind of next to it?  Wasn't sure what "in conjunction" means
21:07:23 <martial_> but basically it was the result of conversation between Nvidia/Docker and universities to do a HPC container track
21:07:31 <martial_> well it is a workshop part of the main conference
21:07:37 <oneswig> cool, thanks
21:07:41 <janders> but different venue right?
21:07:42 <martial_> so "part of" ? :)
21:08:02 <martial_> janders: you know I do not know yet :)
21:08:20 <martial_> we are still heavily into the get ready to review papers phase
21:08:20 <janders> (as in walking distance, but not in frankfurter messe - it's Marriott)
21:08:31 <janders> I've attended something similar few years back, they did it the same way
21:08:35 <oneswig> Were you there last year?  What's it like?
21:09:50 <martial_> I was not at ISC last year, and was not part of HPCW then
21:10:43 <oneswig> Ah OK.  One of my colleagues (John) will be at ISC.  I'll pass on the details to him.
21:10:54 <oneswig> Good luck with the CFP martial_!
21:10:55 <martial_> so there is that, as always CFP is open so feel free to submit a paper
21:11:08 <janders> I'll find out who's going from CSIRO and take things from there
21:11:35 <janders> thank you for bringing this to my attention, I find this very relevant
21:11:35 <martial_> Also, Accepted ISC 2019 workshop papers will be published in the Springer’s Lecture Notes in Computer Science (LNCS) series.
21:11:51 <janders> value add!
21:12:32 <martial_> that is it for now, more as we get closer :)
21:12:50 <janders> on to OpenStack's gaps?
21:13:31 <martial_> #topic OpenStack's gaps: Help most needed for the Scientific SIG
21:13:58 <martial_> #link https://www.openstack.org/summit/denver-2019/summit-schedule/events/23612/help-most-needed-for-sigs-and-wgs
21:14:02 <janders> very timely
21:14:10 <martial_> welcome b1airo
21:14:11 <b1airo> made it, i think
21:14:18 <martial_> #chair b1airo
21:14:19 <openstack> Current chairs: b1airo martial_ oneswig
21:14:20 <oneswig> Hi b1airo
21:14:28 <b1airo> morning
21:14:40 <b1airo> slow flight sorry
21:14:44 <oneswig> indeed, top of the morning!
21:14:49 <b1airo> how are things?
21:15:13 <martial> we just jumped on the "OpenStack's gaps: Help most needed for the Scientific SIG" topic
21:15:20 <janders> my 5 cents - we really need to get the SDN story straight
21:15:38 <janders> there is a great deal of duplication of effort in the field
21:15:46 <janders> and square wheel reinvention is alive and well
21:16:05 <janders> each vendor seems to have own SDN platform
21:16:16 <janders> Juniper Mellanox Cisco Cumulus X Y Z
21:16:18 <oneswig> janders: tricky to standardise on somebody else's standard, I guess
21:16:31 <janders> true, but I think ML2 proves it can be done
21:16:55 <janders> most (not all) SDN-OpenStack solutions have 2 (or  more) sources of truth problem
21:17:08 <janders> and when these diverge, Bad Things (TM) happen
21:17:24 <janders> SDN is critical to multi tenant baremetal
21:17:39 <janders> and multi tenant baremetal is critical to OpenStack on HPC/scientific computing as far as I am concerned
21:18:03 <martial> so yes making a list of topics :)
21:18:05 <janders> I think we need to do something about this and the Summit and PTG should be a good opportunity to start that conversation
21:18:06 <oneswig> janders: certainly resonates here
21:18:22 <janders> what is to our advantage is - Telcos are OpenStack's cash cow
21:18:25 <janders> and guess what
21:18:28 <janders> it's exactly the same shit
21:18:34 <janders> except at higher stakes
21:18:42 <janders> so I think it's a solvable problem
21:19:00 <janders> let's gather the pain points, make a prioritised list, find out who to talk to
21:19:21 <janders> I am happy to use my high profile RH customer leverage to help make this happen
21:19:29 <oneswig> Rico Lin's running the event.  I don't know if there's an etherpad for gathering data
21:19:39 <martial> should we start an etherpad for this?
21:19:43 <janders> yes!
21:20:02 <janders> another pain point would be poor selection of storage choices for baremetal
21:20:19 <janders> for VM-centric world, no ability to separate boot from ephemeral is not ideal
21:20:43 <janders> (either of these nothing new)
21:21:08 <janders> but the great SDN drama is by far no 1) painpoint to me - addressing this will make or break larger-scale OpenStack based supercomputers
21:21:09 <oneswig> janders: absolutely, good point on the storage.
21:21:20 <martial> Fresh Etherpad
21:21:20 <b1airo> seems like a problem for edge deployments too, presumably they want cloud all the way down so they get same agility managing the underlying infrastructure as they do for their "customer" facing services
21:21:24 <martial> #link https://etherpad.openstack.org/p/scientific-sig-denver2019-gaps
21:21:38 <oneswig> Are you aware of the baremetal program that Chris Hoge has been getting together?
21:21:54 <oneswig> Nice, thanks martial
21:22:48 <janders> yes - and I am hoping to participate in this
21:23:01 * ijw wakes up since people are talking SDN
21:23:16 <janders> I think two years fast forward, Ironic will be the new Keystone
21:23:23 <janders> nothing will happen without it
21:24:03 <ijw> If you want help on SDN problems and solutions, come shout
21:24:29 <oneswig> Thanks ijw - any forum events lined up for that do you know?
21:24:32 <janders> ijw: what would be the best place for discussions on this topic in Denver?
21:24:47 <ijw> Honestly, the usual answer to that is 'the corridors'
21:25:01 <janders> also, should we have some discussions prior to Denver? What is the best audience and platform?
21:25:58 <oneswig> janders: it's not that far off, what do you have in mind?
21:26:05 <ijw> SDN is almost precisely not what Neutron does - it provides the API.  If we can establish your actual needs we can work from there
21:26:29 <ijw> Also, k8s and OpenStack take radically different approaches, so this isn't a one-size-fits-all discussion
21:27:21 <janders> my concern is the fact that many SDN platforms sort of hold their own copy of neutron state. In certain circumstances, the two copies diverge and that's no good.
21:27:26 <oneswig> be interesting to go into that in greater depth, in a corridor somewhere :-)
21:27:42 <janders> I haven't seen any mechanism that would force re-sync from neutron to SDN in the SDN solutions that I know of
21:28:02 <janders> and what completely kills it is the fact that SDN vendors go "it never breaks in my lab" and don't really pick this up
21:28:38 <martial> so far I see SDN in the Etherpad page, we ought to publicize it further
21:28:42 <oneswig> janders: it's a problem that events and changes can come from both directions - reconfigurations from above, physical network changes from below
21:28:52 <oneswig> brb...
21:28:57 <janders> also taking a consistent end to end backup is non trivial
21:29:15 <janders> all of this worries me in the context of operations on a say 2000 node HPC system powered by baremetal OpenStack
21:29:26 <janders> it can be done today, but it is much more stressful than it needs to be
21:29:35 <janders> and the very same issue will hit telcos
21:30:10 <janders> so yeah I would be very keen to have an in-depth discussion on this in Denver, including at the PTG
21:30:23 <janders> it is a common worry of most of the OpenStack power-users I know
21:30:46 <ijw> janders: ultimately, multiple copies of state have to exist if you want SDN or anything distributed to work; the 'source of truth' argument is more a question of who wins when they get out of sync
21:30:58 <martial> (brb)
21:31:13 <ijw> And, I think more importantly, that they even notice that they're out of sync.
21:31:33 <janders> ..and how to safely put them back in sync
21:31:37 <janders> I couldn't agree more!
21:32:01 <ijw> The model we built for networking-vpp has a model for this, which is basically that everything that isn't Neutron is (a) effectively a combination of cache and communications model and (b) loses every fight
21:33:02 <ijw> It's an SDN solution, but it doesn't have a central controller; its job is just to get the various forwarders in the system to follow along with Neutron's desired
21:33:04 <ijw> staet
21:35:31 <martial> (back)
21:35:37 <oneswig> ijw: does that system require physical network knowledge or is it built upon an IP fabric?
21:35:54 <janders> mine is all-InfiniBand
21:36:09 <b1airo> Denver summit aside: can anyone remember the URL for the lightning talk etherpad...?
21:36:25 <oneswig> I'll try digging...
21:36:41 <janders> so the baremetal nodes do have much more awareness of the fabric than just the IP layer
21:38:39 <janders> okay I finished putting my points into the etherpad
21:38:55 <janders> what other gaps do you guys see?
21:39:39 <oneswig> would be lovely to get behind more common components for monitoring, frankly.
21:41:29 <janders> agreed!
21:41:43 <janders> (and what are best practices of monitoring containerised services? :)
21:42:07 <janders> a nagios check doing "openstack-service status" would get one a long way historically
21:42:36 <janders> now it looks like we'll be checking if containers are running smoothly and checking HAProxy stats?
21:42:42 <oneswig> A lot of sites have Nagios in common
21:43:09 <janders> in the old days Red Hat would have nagios bits in PackStack and that would get 80% deployment work done in 15mins
21:43:23 <janders> it is what it is but it does the job and that sort of standardisation was nice
21:45:46 <oneswig> Kolla-Ansible is doing some good things to integrate Monasca or ELK + Prometheus
21:46:20 <oneswig> Some of the people in our team have been involved in both of those
21:46:59 <janders> is that work focusing on the infra monitoring, workload monitoring or both?
21:47:10 <oneswig> I'll add some thoughts to the Etherpad...
21:47:17 <janders> great!
21:47:54 <oneswig> Monasca is both, ELK+Prometheus is just for the control plane.
21:50:55 <janders> shall we move on to the lightning talks?
21:51:35 <martial> #topic Denver Summit: SIG meeting and BoF
21:51:57 <martial> b1airo: did you already share an etherpad for this one?
21:52:53 <martial> Sharing a new Etherpad I guess :)
21:52:58 <martial> #link https://etherpad.openstack.org/p/scientific-sig-denver2019-bof_lt
21:53:13 <martial> #link https://www.openstack.org/summit/denver-2019/summit-schedule/events/23741/scientific-sig-bof-and-lightning-talks
21:53:38 <martial> We are on Monday this summit
21:53:53 <martial> we will run both the usual BoF and the lighting talks
21:54:11 <b1airo> martial: i did already create one
21:54:30 <martial> if you are interested in presenting during the ligthing talk session
21:54:33 <b1airo> i'll dig through email/eavesdrop logs... want to get it included on the schedule page
21:54:54 <janders> great idea!
21:54:56 <martial> b1airo: I thought so, could not find it when I looked last week, I apologize
21:55:05 <oneswig> b1airo: +1 to that, would be good to broaden submission as much as we can.
21:55:59 <janders> I have a couple ideas based on current projects, I shall see if these will get to a presentable shape in the Denver timeframe
21:56:29 <martial> so add your entry to the etherpad that will be shared in the summit session and disregard this one posted above :)
21:56:40 <b1airo> here we are...
21:56:42 <b1airo> #link https://etherpad.openstack.org/p/scientific-sig-denver19-bof
21:56:58 <oneswig> good digging b1airo
21:57:23 <oneswig> Everyone OK for me to source a modest prize from the usual channels?
21:57:33 <janders> sure! :)
21:57:47 <janders> also - shall we start thinking about a SIG dinner venue?
21:58:20 <oneswig> That would be nice but I have zero local knowledge
21:58:33 <janders> I've got a friend who's from Colorado and will likely come to the SIG dinner
21:58:36 <janders> will ask him
21:58:42 <rbudden> indeed, would be good to get everyone together for a meal and some beers :)
21:58:51 <oneswig> barbecue?
21:58:53 <janders> great craft beer scene there I hear
21:59:15 <rbudden> yes, there is
21:59:18 <janders> shall we look for something at the intersection of bbq and craft beers? :)
21:59:34 <oneswig> such an unusual combination :-)
21:59:35 <b1airo> could get adventurous and head to Boulder
21:59:43 <martial> https://www.irccloud.com/pastebin/kCzoYA77/
21:59:44 <rbudden> good Ramen too if anyone else is a fan
21:59:48 <ijw> oneswig: that one is both a L2 (VLAN) and L3 (GPE) underlay, but it doesn't really alter the approach.  Happy to discuss this more at the summit or PTG, maybe over lunch or beer
21:59:49 <b1airo> bit more than craft beer available there...
21:59:56 <janders> my friend is from Boulder :)
22:00:47 <janders> ok we're running out of time
22:00:53 <janders> I will ask around and report back next week
22:00:56 <janders> thanks guys!
22:01:00 <b1airo> cheers!
22:01:10 <oneswig> ah, thanks everyone, until next time
22:01:13 <oneswig> #endmeeting