11:00:08 #startmeeting scientific-sig 11:00:09 Meeting started Wed Sep 12 11:00:08 2018 UTC and is due to finish in 60 minutes. The chair is oneswig. Information about MeetBot at http://wiki.debian.org/MeetBot. 11:00:10 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 11:00:12 The meeting name has been set to 'scientific_sig' 11:00:20 I will spell it correctly one day! 11:00:37 #link Agenda for today https://wiki.openstack.org/wiki/Scientific_SIG#IRC_Meeting_September_12th_2018 11:00:40 :) 11:00:44 What-ho 11:00:56 evening janders, how's stuff going? 11:01:13 Good morning / afternoon! 11:01:23 good :) I'm just back from a short break, doing exciting things like.. rebuilding my laptop with FC28 :) 11:01:34 priteau: crikey, you're up early :-) 11:01:55 janders: is that with os-tree? 11:02:25 I suppose jet lag will be over once it's time to fly back home 11:02:29 Afternoon. 11:02:46 Hi verdurin, greetings 11:02:55 oneswig: I will disappoint you - just a plain USB install :) I needed to start fresh. It seems to be working well though. 11:03:14 on more interesting things front, I think I solved a simple problem with SRIOV VMs and baremetals coexisting within one AZ 11:03:26 janders: I'd be interested to know the point of ostree, it only seems to annoy when I've tried to use it. 11:03:41 last time I played with SRIOV was Mitaka - and there scheduler filters are explicitly set 11:03:54 janders: is this your running-virt-in-baremetal setup? 11:04:07 with Queens, there's some intermediate variable which confused things a little 11:04:13 correct 11:05:05 janders: you're talking TripleO config or something directly in Nova ? 11:05:06 I have to try Mark's suggestion regarding adding vxlan support, too - that would give me baremetal, SRIOV or virtualised ethernet on one cloud depending on the requirements 11:05:56 it's not a tripleo scenario - my KVM nodes are baremetal instances like any other ironic-to-tenant instance 11:06:18 it just so happens they are created on the internalapi network which is again a standard neutron network 11:06:42 I will port all this to tripleo some time soon, but for now it's all custom to figure out the exact architecture 11:07:03 speaking of.. looks like a great discussion at the PTG - answered some of my questions 11:07:20 let's cover that... 11:07:24 #topic PTG wash-up 11:07:36 priteau: were you in the Scientific SIG session? 11:07:55 I wasn't unfortunately, it was conflicting with Blazar 11:08:03 Ah, ok. 11:08:25 #link PTG session etherpad https://etherpad.openstack.org/p/scientific-sig-denverptg2018 11:08:45 I heard from Martial and other people it was a useful session 11:08:50 I talked with Martial afterwards, apparently the focus was mostly on bare-metal (points 2, 3, and 4 in the Etherpad) 11:09:10 oneswig: I have one question regarding Mark's comment - maybe you will know the answer 11:09:16 priteau: it's all about who contributed to the agenda - don't ask, don't get :-) 11:09:27 janders: unlikely but I can try! 11:09:41 Mark mentioned that you guys already use a neutron router to uplink the provisioning_network to the API network 11:10:04 while I was doing research on this I was told that ironic-conductor might need to connect directly back to the IPA 11:10:24 aha. Did he explicitly say *neutron* router? 11:10:50 oneswig: haha, you got me there! :) 11:11:01 We've done a mix. There are some tricky pieces with multi-tenant networking that generally break assumptions 11:11:21 he did not and that might mean there's no NAT and that answers my question - thanks! :) 11:12:01 Even if the router was Neutron (can't recall how it was done on the last deploy), I think we'd set it up without NAT. 11:12:38 I might be able to check 11:13:24 Yes, we apparently have Neutron routers for inspection, cleaning and provisioning networks. 11:13:48 neutron routers with NAT disabled, right? 11:14:07 yes, it would be 11:14:33 does your configuration use something called address scope? 11:14:54 Not seeing a reference to that on the router 11:14:59 (googling around as we speak - I haven't used neutron routers without NAT before) 11:15:33 http://superuser.openstack.org/articles/disable-nat-ipv4-openstack/ 11:17:20 thinking about it, maybe it would even work with a standard NAT-enabled router provided that the ironic-conductor node has the route set for the provisioning_network (and that route is the external IP of the neutron router) 11:17:26 There's also this option for `openstack router set`: --disable-snat Disable Source NAT on external gateway 11:18:19 priteau: thank you. Without testing, my guess is it might work without any of the complexity covered in the link I posted above 11:19:12 oneswig: I will touch on this in my reply to Mark 11:19:15 thanks guys! 11:19:16 We are hoping to extend routing Ironic traffic further to enable (eg) multi-tenant support for boot-from-volume 11:20:31 janders: do keep in touch with Mark on this I'm sure he'd be interested to hear how you are doing. 11:20:45 oneswig: will definitely do 11:22:02 Judging by the discussion on trusted boot, there's quite a way to go on this. Plenty of comments along the lines of "Ironic could in theory..." 11:22:15 oneswig: I saw a mention of the boot-from-volume challenges in the PTG/SIG. Is it the initial step - pxeboot - where the problems start in a multi-tenant context? 11:22:48 Is that boot from Ceph RBD volume, specifically? 11:23:37 I was looking at 4.1.2 (I think Julia made this note) - it looks fairly generic (not limited to RBD) 11:23:38 There's working support for boot from iSCSI but I am not sure if it supports multi-tenant network isolation. 11:24:06 aha, exactly that janders 11:24:55 So I think Ironic's comfortable with PXE-booting on the provisioning network but expects instances to boot without pxe assistance on the tenant networks. 11:25:07 here's a quick and dirty idea - how about we write ipxe image over the provisioning network and when the baremetal instance gets moved to the tenant network post-provisioning, it's "intercepted" by another PXE server? 11:25:37 I've done something along those lines for a completely different application, but it did give me full multi-tenant pxe capability 11:25:54 There is support for separate kernel and ramdisk images for nodes that cannot configure boot-to-disk (eg, nodes without IPMI). 11:26:04 I wonder if that works with multi-tenant networking. 11:26:34 I believe that would require PXE on the tenant networks, which as you say is what is wanted 11:27:26 oneswig: We had the same problem on Chameleon with ARM64 nodes which were booting so-called partition images, which boot kernel and ramdisk from the network 11:27:51 It doesn't seem so far off, that's for sure. Mark was in the discussion and mentioned that there'd be a lot of data funneled through this poor neutron router if the nodes were booting through it. 11:28:01 To use tenant networks they needed to be changed to boot from disk 11:28:29 priteau: sounds like the same issue. 11:29:12 pxebooting in a VLAN is always a lot of "fun" 11:29:23 I think my happy solution would be a means of configuring these routers in the network silicon, via NGS perhaps. 11:29:39 but.. better over VLAN than a VXLAN :) 11:29:44 If firewall rules could also be attached, even better 11:30:01 NGS? 11:30:08 janders: how are your experiences with VXLAN now? 11:30:23 verdurin: ah, networking-generic-switch - bare metal switch driver 11:30:47 ah, thanks oneswig 11:31:20 I haven't had a chance to try Mark's hints yet, so I'm in the same place as two weeks back except I had some brief exchanges with Mellanox and they might look at better supporting the vxlan / IB scenari 11:31:23 *scenario 11:31:52 I think it would be cool to have virtual ethernet + SRIOV IB + baremetal IB in one cluster 11:32:04 maximum flexibility 11:32:29 janders: that does sound appealing 11:32:34 janders: for a certain definition of "fun", it's maximum fun 11:32:52 and some spare HPC gear can do a great job running non-HPC applications. Nova-compute in baremetal instances means super easy scaling. 11:34:15 It's a pretty interesting setup. Have you documented it / blogged it anywhere yet? 11:35:19 No, not yet. I was hoping to give a preso at the Berlin summit about the hypervisor-in-an-ironic-instance idea, but it doesn't look like it made it 11:35:37 Oh that's disappointing. 11:36:03 Scientific SIG lightning talk perhaps? Sure you can cover it in 3 minutes... 11:36:18 I wonder if that's because it's niche or because it's too early for such ideas 11:36:32 Sure thing! Would be more than happy to do that 11:36:53 Yes, please do. 11:37:09 90s talking and 90s demo of ansible spinning up an extra compute node in a baremetal instance 11:37:23 :) 11:37:35 I am sure plenty of people would be interested to see it. 11:37:44 or 60 + 60 + 60 for questions? :) 11:38:16 Mixing virt and ironic has historically been a real PITA and this solution is a creative one :-) 11:39:10 janders: if you have anything online about it we could put it on the agenda for an IRC session (but more people attend the summit sessions) 11:40:32 priteau: anything to report from elsewhere at the PTG? 11:41:08 We had an interesting session with a lot of placement folks to discuss an approach for Blazar to use 11:41:47 This brought up a requirement for placement to support reporting allocation candidates on providers which are *not* part of a specific aggregate 11:42:16 This should give us Ironic compatibility in Blazar as well. I am going to spec and prototype in the near future. 11:43:09 oneswig: regarding compute-on-ironic - I don't have anything yet, but I could get something ready in say a month's time - which would be around a month ahead of the Summit.. There's a lot of planning work going on at CSIRO at the moment which is pulling me away from more technical work. 11:43:13 Sounds productive. And good to bring Blazar integrations into consideration 11:43:31 priteau: sounds great! 11:43:36 janders: would be great if you do that, let me know 11:43:54 priteau: anything on premptibles and blazar yet? 11:45:11 I briefly outlined an approach for providing preemptibles via Blazar, there wasn't much discussion though. Just need to do it I guess :-) 11:46:18 Anything else from the PTG to cover? 11:46:25 oneswig: yes 11:46:38 point 2 - BIOS management via Ironic 11:47:02 have you used these features much? With what results? 11:48:20 We have not. We use ansible modules externally to talk to Dell BIOSes - https://galaxy.ansible.com/stackhpc/drac and https://galaxy.ansible.com/stackhpc/drac-facts 11:48:50 One of the team just recently looked into https://github.com/dell/Dell-EMC-Ansible-Modules-for-iDRAC - anyone use these? 11:49:02 But no I haven't used the Ironic interface yet. 11:49:22 ok! similar here - I just used Dell-provided tools (pretty average) 11:49:22 It would be a shame to stick with these vendor-specific interfaces rather than Redfish. 11:49:42 these ansible playbooks look great! I look forward to trying these out 11:49:59 verdurin: indeed. Although I think Redfish only removes a vendor-specific transport. Like SNMP, the useful data's still in a vendor-specific mib-like thingy 11:50:01 Though I found that Ironic still had some Redfish wrinkles. 11:50:05 what server models have you tested these with? I wonder if my ancient M620s would cooperate.. 11:51:04 janders: I am doubtful... I think it works with Dell C6320 but not C6220. We've also used it on R630, R740, etc. 11:51:07 that era of kit 11:51:32 right - I have some M630s (much less painful to work with) so there's hope 11:51:33 verdurin: what wrinkles did you find with redfish? 11:52:27 One the HPE gear I was testing on, while all the introspection would work, it would fail at setting up power management 11:53:00 It may have been an outdated Sushy library. There were some similar bugs reported. 11:53:07 as in, power on/off, rather than server power/performance profiles? 11:53:25 The former, yes. 11:53:29 another slightly related question: I'm thinking of trying to use the smarts in SSD/NVMe (trim/discard type of stuff) for node cleaning (instead of dding zeroes all over the drives, wasting time and flash writes). Do you have any experience with this? 11:53:58 Why, yes we do :-) 11:54:36 It mostly works and saves a ton of time on fat nodes! 11:55:06 do you trigger these through a custom node cleaning step, or is there some integration allowing ironic to do it more consciously? 11:55:20 janders: I'll ask my colleague who did the work to find a link. 11:56:26 oneswig: thank you! :) it's great to hear it's a real thing 11:56:30 There is an option in Ironic, enable_ata_secure_erase. I wonder if Linux will take this request for SSDs as well? 11:57:05 priteau: I believe that's all that is needed. It works for SSDs 11:57:16 #link https://ata.wiki.kernel.org/index.php/ATA_Secure_Erase 11:57:48 Apparently enable_ata_secure_erase in Ironic default to true 11:58:48 We are just about out of time.. anything more to add? 11:59:00 verdurin: are you planning to attend CIUK in December? 11:59:03 But if it fails, shred_random_overwrite_iterations and shred_final_overwrite_with_zeros pick up with overwriting data 11:59:24 janders: Check your config for these option, and the IPA logs during cleaning 11:59:39 Maybe your ATA Secure Erase is failing for some reason 11:59:53 oneswig: if we're not in the middle of a tender, yes 11:59:54 ok, thanks guys! looks like all the smarts are already in Ironic 12:00:01 We did get some spurious errors with secure erase on some SSDs, I think a FW update was needed. 12:00:23 great chat, thank you all! 12:00:25 Bye all. 12:00:31 Thanks everyone 12:00:36 #endmeeting