21:00:50 <b1airo> #startmeeting Scientific-SIG
21:00:51 <openstack> Meeting started Tue May 25 21:00:50 2021 UTC and is due to finish in 60 minutes.  The chair is b1airo. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:52 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:54 <openstack> The meeting name has been set to 'scientific_sig'
21:00:57 <martial> time for another exiting Scientific ... what Blair said :)
21:00:59 <b1airo> morning
21:01:26 <b1airo> i'm in a work call with Stig at the moment, so we'll be a bit tardy..
21:02:14 <martial> roger
21:02:25 <martial> anything new for our HPC friends?
21:02:36 <martial> I saw Belmiro during the last Open Infra Live
21:02:57 <martial> https://www.youtube.com/watch?v=yf5iFiCg_Tw
21:03:27 <martial> the topic was very relevant to our crowd: " Upgrades in Large Scale OpenStack Infrastructure"
21:07:09 <b1airo> uh oh, time for a Zoom update
21:07:17 <b1airo> #chair martial
21:07:18 <openstack> Current chairs: b1airo martial
21:08:12 <martial> not sure if that is a good sign :)
21:08:32 <b1airo> I haven't seen Belmiro's talk yet, but based on Tim's Twitter feed I have an idea of the content :-)
21:08:43 <martial> ;)
21:09:20 <martial> it was good, and the questions relevant
21:09:41 <martial> anybody else had a chance to view it and/or had comments?
21:10:07 <oneswig> g'day
21:10:54 <oneswig> b1airo: hey
21:11:02 <b1airo> o/
21:11:17 <b1airo> deja vu :-)
21:15:29 <martial> What is the status of our plan for ISC ?
21:15:44 <oneswig> Hi martial, didn't spot you there.
21:15:50 <oneswig> It's true, there's something going ahead.
21:16:16 <martial> #chair oneswig
21:16:17 <openstack> Current chairs: b1airo martial oneswig
21:16:55 <oneswig> I haven't checked what needs to be done but a ticket needs to be organised for one.
21:17:15 <martial> I got the email a while back but no action since so unclear
21:17:19 <oneswig> We had a really good panel of speakers involved too IIRC
21:17:41 <oneswig> It's the fatigue with online conferences.
21:18:00 <martial> same
21:19:52 <martial> welcome Tim
21:20:23 <oneswig> Been having fun with OVN recently
21:20:39 <oneswig> in a "actually not fun" sense
21:20:42 * trandles reads "OVN" and starts twitching
21:22:47 <oneswig> Hardware offload of security groups.  It's neat, but requires the latest Mellanox firmware.  Too recent if you bought your NICs from an OEM
21:24:20 <oneswig> There's a "seamless" trap to software if you can't offload the flow onto the NIC.  RDMA even works - at a small number of gb/s
21:25:46 <oneswig> OVS is like nft - one of those hard-to-learn things you only ever have to learn when something's broken.  It doesn't set you up to want to explore it's many treasures.
21:25:58 <oneswig> Or SELinux perhaps ;-)
21:27:09 <martial> "treasures" indeed :
21:30:36 <oneswig> Had another interesting issue today with Slurm automation - setup would mysteriously fail in the playbooks, but work if the steps were repeated manually.  Turns out the systemd service for slurmctld in OpenHPC does not wait for the service to complete startup.  If your config file takes 4s to read and process, other daemons would out-race it and barf on its absence.  Gah.
21:31:03 <oneswig> Easily fixed if you wait_for it to bind.
21:31:40 <trandles> To be fair, my OVS+OVN+HA (no DVR) deploy of RHOSP went very smoothly and is working well. But it is a lot to grok if you've never dug into this stuff before.
21:32:49 <trandles> I had never dug into it before. I was very happy it "just worked" because then I could start playing with things and watching the results to learn WTF is actually going on in there.
21:33:03 <b1airo> sounds like systemd doing its job then oneswig :-o
21:33:30 <oneswig> OVN does work and is simpler than the OVS Hybrid driver.  It's just the contortions that optimisation puts us through.
21:34:22 <trandles> A colleague once gave a summer intern the project to "serialize systemd services." /o\
21:34:25 <oneswig> trandles: if you're using it with Ironic you've probably had fun integrating VLANs, metadata and FIPs into OVN together.
21:34:53 <trandles> oneswig: not doing that fortunately
21:36:15 <oneswig> trandles: is the networking flat for your Ironic deploy?
21:36:20 <trandles> yes
21:36:45 <oneswig> certainly keeps things simple
21:37:47 <b1airo> Ironically simple...
21:39:09 <b1airo> martial: regarding ISC - yes, I received a note on that last week that I haven't processed yet. has been near the top of my todo list for last couple of weeks to contact the panel and try to get some conversation going and/or setup a pre-meeting ...
21:41:07 <oneswig> I think a pre-meeting is good, although likely hard to schedule.  Even a group email thread would help
21:41:32 <trandles> Ah, kkillsfirst is here. Welcome to the LANL summer intern who will be working on kexec in Ironic.
21:41:46 <oneswig> Wow!  Hi kkillsfirst, welcome
21:41:52 <oneswig> Great project trandles
21:41:59 <trandles> thx oneswig
21:42:28 <oneswig> Will there be a public document at the end of the internship?
21:42:52 <oneswig> How can it be summer already?  It hasn't stopped raining for weeks.
21:43:35 <kkillsfirst> Hello, Glad to be here helping.
21:43:50 <trandles> I'm really hoping by the end of the summer the functionality is merged and documented. That's the goal at least.
21:43:59 <oneswig> Good plan
21:44:28 <trandles> Julia and her team are chomping at the bit to get Kam started. Probably next week after the obligatory week of LANL hazing.
21:44:39 <trandles> I mean, training.
21:45:55 <oneswig> I'm sure.
21:46:50 <oneswig> How much deploy time do you think can be saved?
21:47:21 <trandles> depends a lot on how long it takes to go back through the BIOS
21:47:43 <trandles> I have some 2TB nodes that take nearly 20 minutes to start booting
21:47:45 <oneswig> of course.  A long and arduous journey
21:48:07 <oneswig> ouch
21:50:43 <trandles> It should be a lot of gain, even doing all the shutdown-style things you have to do before the kexec. Unmounting network filesystems, etc...you do that anyway. For the stateless HPC compute node case you don't really care to shut down many services gracefully. It'll be unmount, downlaod kernel+ramdisk, kexec-pivot_root and go.
21:51:46 <oneswig> Do you think it could work with trusted boot?  Perhaps a long shot.  Not even sure why I'm asking
21:52:49 <trandles> Haven't thought about that much...
21:53:15 <trandles> It would definitely work the way we do things today. We've tested it manually just to see what happens.
21:53:48 <oneswig> Way back in the day, I worked on a LinuxBIOS equivalent for AlphaServers, replacing the BIOS with an embedded Linux kernel.  I was always deeply skeptical about chainloading kernels.  I want to state for the record I was probably overly skeptical and wrong :-)  Only took me 20 years.
21:55:14 <julianp> :D Interesting. Ahead of your time.
21:55:15 <trandles> I'm going to have to run...another meeting in 5 minutes. Anyway, kkillsfirst is also on the openstack-scientific-sig slack.
21:55:31 <oneswig> Cool, see you around Kam!
21:55:36 <trandles> Hopefully more updates to come during the summer :)
21:56:33 <oneswig> In fact we are nearly out of time.
21:57:06 <oneswig> b1airo: can you mail folks for the ISC BoF? That would be a great help
21:57:35 <b1airo> yep absolutely - can do that todaay
21:58:12 <oneswig> Thanks Blair
21:59:20 <oneswig> I'm on vacation next week (woohoo), much though I enjoy yappin' on IRC it aint happening
21:59:37 <trandles> Enjoy vacation oneswig
21:59:52 <b1airo> i'm not jealous...
21:59:54 <oneswig> Thanks trandles :-)
22:00:04 <kkillsfirst> See you later everyone. I hope to get a tiny cluster working at home this week. :)
22:00:14 <oneswig> b1airo: summer's on the way
22:00:24 <julianp> Nice kkillsfirst. See y'all next time.
22:00:25 <oneswig> kkillsfirst: nice to meet you.  Good luck!
22:00:39 <oneswig> martial: time to close
22:00:47 <martial> yep
22:00:57 <martial> #endmeeting