11:00:08 <oneswig> #startmeeting scientific-sig
11:00:09 <openstack> Meeting started Wed Sep 12 11:00:08 2018 UTC and is due to finish in 60 minutes.  The chair is oneswig. Information about MeetBot at http://wiki.debian.org/MeetBot.
11:00:10 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
11:00:12 <openstack> The meeting name has been set to 'scientific_sig'
11:00:20 <oneswig> I will spell it correctly one day!
11:00:37 <oneswig> #link Agenda for today https://wiki.openstack.org/wiki/Scientific_SIG#IRC_Meeting_September_12th_2018
11:00:40 <janders> :)
11:00:44 <oneswig> What-ho
11:00:56 <oneswig> evening janders, how's stuff going?
11:01:13 <priteau> Good morning / afternoon!
11:01:23 <janders> good :) I'm just back from a short break, doing exciting things like.. rebuilding my laptop with FC28 :)
11:01:34 <oneswig> priteau: crikey, you're up early :-)
11:01:55 <oneswig> janders: is that with os-tree?
11:02:25 <priteau> I suppose jet lag will be over once it's time to fly back home
11:02:29 <verdurin> Afternoon.
11:02:46 <oneswig> Hi verdurin, greetings
11:02:55 <janders> oneswig: I will disappoint you - just a plain USB install :) I needed to start fresh. It seems to be working well though.
11:03:14 <janders> on more interesting things front, I think I solved a simple problem with SRIOV VMs and baremetals coexisting within one AZ
11:03:26 <oneswig> janders: I'd be interested to know the point of ostree, it only seems to annoy when I've tried to use it.
11:03:41 <janders> last time I played with SRIOV was Mitaka - and there scheduler filters are explicitly set
11:03:54 <oneswig> janders: is this your running-virt-in-baremetal setup?
11:04:07 <janders> with Queens, there's some intermediate variable which confused things a little
11:04:13 <janders> correct
11:05:05 <oneswig> janders: you're talking TripleO config or something directly in Nova ?
11:05:06 <janders> I have to try Mark's suggestion regarding adding vxlan support, too - that would give me baremetal, SRIOV or virtualised ethernet on one cloud depending on the requirements
11:05:56 <janders> it's not a tripleo scenario - my KVM nodes are baremetal instances like any other ironic-to-tenant instance
11:06:18 <janders> it just so happens they are created on the internalapi network which is again a standard neutron network
11:06:42 <janders> I will port all this to tripleo some time soon, but for now it's all custom to figure out the exact architecture
11:07:03 <janders> speaking of.. looks like a great discussion at the PTG - answered some of my questions
11:07:20 <oneswig> let's cover that...
11:07:24 <oneswig> #topic PTG wash-up
11:07:36 <oneswig> priteau: were you in the Scientific SIG session?
11:07:55 <priteau> I wasn't unfortunately, it was conflicting with Blazar
11:08:03 <oneswig> Ah, ok.
11:08:25 <oneswig> #link PTG session etherpad https://etherpad.openstack.org/p/scientific-sig-denverptg2018
11:08:45 <oneswig> I heard from Martial and other people it was a useful session
11:08:50 <priteau> I talked with Martial afterwards, apparently the focus was mostly on bare-metal (points 2, 3, and 4 in the Etherpad)
11:09:10 <janders> oneswig: I have one question regarding Mark's comment - maybe you will know the answer
11:09:16 <oneswig> priteau: it's all about who contributed to the agenda - don't ask, don't get :-)
11:09:27 <oneswig> janders: unlikely but I can try!
11:09:41 <janders> Mark mentioned that you guys already use a neutron router to uplink the provisioning_network to the API network
11:10:04 <janders> while I was doing research on this I was told that ironic-conductor might need to connect directly back to the IPA
11:10:24 <oneswig> aha.  Did he explicitly say *neutron* router?
11:10:50 <janders> oneswig: haha, you got me there! :)
11:11:01 <oneswig> We've done a mix.  There are some tricky pieces with multi-tenant networking that generally break assumptions
11:11:21 <janders> he did not and that might mean there's no NAT and that answers my question - thanks! :)
11:12:01 <oneswig> Even if the router was Neutron (can't recall how it was done on the last deploy), I think we'd set it up without NAT.
11:12:38 <oneswig> I might be able to check
11:13:24 <oneswig> Yes, we apparently have Neutron routers for inspection, cleaning and provisioning networks.
11:13:48 <janders> neutron routers with NAT disabled, right?
11:14:07 <oneswig> yes, it would be
11:14:33 <janders> does your configuration use something called address scope?
11:14:54 <oneswig> Not seeing a reference to that on the router
11:14:59 <janders> (googling around as we speak - I haven't used neutron routers without NAT before)
11:15:33 <janders> http://superuser.openstack.org/articles/disable-nat-ipv4-openstack/
11:17:20 <janders> thinking about it, maybe it would even work with a standard NAT-enabled router provided that the ironic-conductor node has the route set for the provisioning_network (and that route is the external IP of the neutron router)
11:17:26 <priteau> There's also this option for `openstack router set`: --disable-snat        Disable Source NAT on external gateway
11:18:19 <janders> priteau: thank you. Without testing, my guess is it might work without any of the complexity covered in the link I posted above
11:19:12 <janders> oneswig: I will touch on this in my reply to Mark
11:19:15 <janders> thanks guys!
11:19:16 <oneswig> We are hoping to extend routing Ironic traffic further to enable (eg) multi-tenant support for boot-from-volume
11:20:31 <oneswig> janders: do keep in touch with Mark on this I'm sure he'd be interested to hear how you are doing.
11:20:45 <janders> oneswig: will definitely do
11:22:02 <oneswig> Judging by the discussion on trusted boot, there's quite a way to go on this.  Plenty of comments along the lines of "Ironic could in theory..."
11:22:15 <janders> oneswig: I saw a mention of the boot-from-volume challenges in the PTG/SIG. Is it the initial step - pxeboot - where the problems start in a multi-tenant context?
11:22:48 <oneswig> Is that boot from Ceph RBD volume, specifically?
11:23:37 <janders> I was looking at 4.1.2 (I think Julia made this note) - it looks fairly generic (not limited to RBD)
11:23:38 <oneswig> There's working support for boot from iSCSI but I am not sure if it supports multi-tenant network isolation.
11:24:06 <oneswig> aha, exactly that janders
11:24:55 <oneswig> So I think Ironic's comfortable with PXE-booting on the provisioning network but expects instances to boot without pxe assistance on the tenant networks.
11:25:07 <janders> here's a quick and dirty idea - how about we write ipxe image over the provisioning network and when the baremetal instance gets moved to the tenant network post-provisioning, it's "intercepted" by another PXE server?
11:25:37 <janders> I've done something along those lines for a completely different application, but it did give me full multi-tenant pxe capability
11:25:54 <oneswig> There is support for separate kernel and ramdisk images for nodes that cannot configure boot-to-disk (eg, nodes without IPMI).
11:26:04 <oneswig> I wonder if that works with multi-tenant networking.
11:26:34 <oneswig> I believe that would require PXE on the tenant networks, which as you say is what is wanted
11:27:26 <priteau> oneswig: We had the same problem on Chameleon with ARM64 nodes which were booting so-called partition images, which boot kernel and ramdisk from the network
11:27:51 <oneswig> It doesn't seem so far off, that's for sure.  Mark was in the discussion and mentioned that there'd be a lot of data funneled through this poor neutron router if the nodes were booting through it.
11:28:01 <priteau> To use tenant networks they needed to be changed to boot from disk
11:28:29 <oneswig> priteau: sounds like the same issue.
11:29:12 <janders> pxebooting in a VLAN is always a lot of "fun"
11:29:23 <oneswig> I think my happy solution would be a means of configuring these routers in the network silicon, via NGS perhaps.
11:29:39 <janders> but.. better over VLAN than a VXLAN :)
11:29:44 <oneswig> If firewall rules could also be attached, even better
11:30:01 <verdurin> NGS?
11:30:08 <oneswig> janders: how are your experiences with VXLAN now?
11:30:23 <oneswig> verdurin: ah, networking-generic-switch - bare metal switch driver
11:30:47 <verdurin> ah, thanks oneswig
11:31:20 <janders> I haven't had a chance to try Mark's hints yet, so I'm in the same place as two weeks back except I had some brief exchanges with Mellanox and they might look at better supporting the vxlan / IB scenari
11:31:23 <janders> *scenario
11:31:52 <janders> I think it would be cool to have virtual ethernet + SRIOV IB + baremetal IB in one cluster
11:32:04 <janders> maximum flexibility
11:32:29 <verdurin> janders: that does sound appealing
11:32:34 <oneswig> janders: for a certain definition of "fun", it's maximum fun
11:32:52 <janders> and some spare HPC gear can do a great job running non-HPC applications. Nova-compute in baremetal instances means super easy scaling.
11:34:15 <oneswig> It's a pretty interesting setup.  Have you documented it / blogged it anywhere yet?
11:35:19 <janders> No, not yet. I was hoping to give a preso at the Berlin summit about the hypervisor-in-an-ironic-instance idea, but it doesn't look like it made it
11:35:37 <oneswig> Oh that's disappointing.
11:36:03 <oneswig> Scientific SIG lightning talk perhaps?  Sure you can cover it in 3 minutes...
11:36:18 <janders> I wonder if that's because it's niche or because it's too early for such ideas
11:36:32 <janders> Sure thing! Would be more than happy to do that
11:36:53 <verdurin> Yes, please do.
11:37:09 <janders> 90s talking and 90s demo of ansible spinning up an extra compute node in a baremetal instance
11:37:23 <janders> :)
11:37:35 <oneswig> I am sure plenty of people would be interested to see it.
11:37:44 <janders> or 60 + 60 + 60 for questions? :)
11:38:16 <oneswig> Mixing virt and ironic has historically been a real PITA and this solution is a creative one :-)
11:39:10 <oneswig> janders: if you have anything online about it we could put it on the agenda for an IRC session (but more people attend the summit sessions)
11:40:32 <oneswig> priteau: anything to report from elsewhere at the PTG?
11:41:08 <priteau> We had an interesting session with a lot of placement folks to discuss an approach for Blazar to use
11:41:47 <priteau> This brought up a requirement for placement to support reporting allocation candidates on providers which are *not* part of a specific aggregate
11:42:16 <priteau> This should give us Ironic compatibility in Blazar as well. I am going to spec and prototype in the near future.
11:43:09 <janders> oneswig: regarding compute-on-ironic - I don't have anything yet, but I could get something ready in say a month's time - which would be around a month ahead of the Summit.. There's a lot of planning work going on at CSIRO at the moment which is pulling me away from more technical work.
11:43:13 <oneswig> Sounds productive.  And good to bring Blazar integrations into consideration
11:43:31 <janders> priteau: sounds great!
11:43:36 <oneswig> janders: would be great if you do that, let me know
11:43:54 <oneswig> priteau: anything on premptibles and blazar yet?
11:45:11 <priteau> I briefly outlined an approach for providing preemptibles via Blazar, there wasn't much discussion though. Just need to do it I guess :-)
11:46:18 <oneswig> Anything else from the PTG to cover?
11:46:25 <janders> oneswig: yes
11:46:38 <janders> point 2 - BIOS management via Ironic
11:47:02 <janders> have you used these features much? With what results?
11:48:20 <oneswig> We have not.  We use ansible modules externally to talk to Dell BIOSes - https://galaxy.ansible.com/stackhpc/drac and https://galaxy.ansible.com/stackhpc/drac-facts
11:48:50 <oneswig> One of the team just recently looked into https://github.com/dell/Dell-EMC-Ansible-Modules-for-iDRAC - anyone use these?
11:49:02 <oneswig> But no I haven't used the Ironic interface yet.
11:49:22 <janders> ok! similar here - I just used Dell-provided tools (pretty average)
11:49:22 <verdurin> It would be a shame to stick with these vendor-specific interfaces rather than Redfish.
11:49:42 <janders> these ansible playbooks look great! I look forward to trying these out
11:49:59 <oneswig> verdurin: indeed.  Although I think Redfish only removes a vendor-specific transport.  Like SNMP, the useful data's still in a vendor-specific mib-like thingy
11:50:01 <verdurin> Though I found that Ironic still had some Redfish wrinkles.
11:50:05 <janders> what server models have you tested these with? I wonder if my ancient M620s would cooperate..
11:51:04 <oneswig> janders: I am doubtful... I think it works with Dell C6320 but not C6220.  We've also used it on R630, R740, etc.
11:51:07 <oneswig> that era of kit
11:51:32 <janders> right - I have some M630s (much less painful to work with) so there's hope
11:51:33 <oneswig> verdurin: what wrinkles did you find with redfish?
11:52:27 <verdurin> One the HPE gear I was testing on, while all the introspection would work, it would fail at setting up power management
11:53:00 <verdurin> It may have been an outdated Sushy library. There were some similar bugs reported.
11:53:07 <oneswig> as in, power on/off, rather than server power/performance profiles?
11:53:25 <verdurin> The former, yes.
11:53:29 <janders> another slightly related question: I'm thinking of trying to use the smarts in SSD/NVMe (trim/discard type of stuff) for node cleaning (instead of dding zeroes all over the drives, wasting time and flash writes). Do you have any experience with this?
11:53:58 <oneswig> Why, yes we do :-)
11:54:36 <oneswig> It mostly works and saves a ton of time on fat nodes!
11:55:06 <janders> do you trigger these through a custom node cleaning step, or is there some integration allowing ironic to do it more consciously?
11:55:20 <oneswig> janders: I'll ask my colleague who did the work to find a link.
11:56:26 <janders> oneswig: thank you! :) it's great to hear it's a real thing
11:56:30 <priteau> There is an option in Ironic, enable_ata_secure_erase. I wonder if Linux will take this request for SSDs as well?
11:57:05 <oneswig> priteau: I believe that's all that is needed.  It works for SSDs
11:57:16 <priteau> #link https://ata.wiki.kernel.org/index.php/ATA_Secure_Erase
11:57:48 <priteau> Apparently enable_ata_secure_erase in Ironic default to true
11:58:48 <oneswig> We are just about out of time.. anything more to add?
11:59:00 <oneswig> verdurin: are you planning to attend CIUK in December?
11:59:03 <priteau> But if it fails, shred_random_overwrite_iterations and shred_final_overwrite_with_zeros pick up with overwriting data
11:59:24 <priteau> janders: Check your config for these option, and the IPA logs during cleaning
11:59:39 <priteau> Maybe your ATA Secure Erase is failing for some reason
11:59:53 <verdurin> oneswig: if we're not in the middle of a tender, yes
11:59:54 <janders> ok, thanks guys! looks like all the smarts are already in Ironic
12:00:01 <oneswig> We did get some spurious errors with secure erase on some SSDs, I think a FW update was needed.
12:00:23 <janders> great chat, thank you all!
12:00:25 <verdurin> Bye all.
12:00:31 <oneswig> Thanks everyone
12:00:36 <oneswig> #endmeeting