| opendevreview | Merged openstack/ironic-python-agent stable/train: Remove legacy centos7 build jobs https://review.opendev.org/c/openstack/ironic-python-agent/+/827358 | 06:26 |
|---|---|---|
| arne_wiebalck | Good morning, Ironic! | 07:22 |
| arne_wiebalck | dtantsur: rpittau: remind me, do you still want patches for https://github.com/metal3-io/ironic-hardware-inventory-recorder-image ? (I am using this to update the inventory of active nodes and updated the Dockerfile some months ago, but with the EOL of c8 it needs another update). Happy to send it, just wondering if you declared this repo archived :) | 08:03 |
| rpittau | good morning ironic! o/ | 08:15 |
| rpittau | arne_wiebalck: hi! it's not archived but it's true that the image is not really used anymore. If you still use it and want to send patches, I can review them :) | 08:18 |
| arne_wiebalck | rpittau: ok, will do ... I think it would be great to keep it as there might be more users in the future who would like to run inspection on active nodes | 08:28 |
| arne_wiebalck | rpittau: and the container is a pretty straight-forward way to do it | 08:28 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent-builder master: Update documentation on supported CentOS version https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/827605 | 08:58 |
| arne_wiebalck | rpittau: https://github.com/metal3-io/ironic-hardware-inventory-recorder-image/pull/13 | 09:31 |
| rpittau | mmm I forgot we didn't have any CI there | 09:47 |
| rpittau | we should at least make sure the image builds | 09:47 |
| arne_wiebalck | it does for the only user :) | 09:48 |
| rpittau | :) | 09:48 |
| arne_wiebalck | (ofc I agree with you) | 09:48 |
| janders | good morning arne_wiebalck rpittau and Ironic o/ | 10:26 |
| arne_wiebalck | hey janders o/ | 10:27 |
| rpittau | hey janders :) | 10:30 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent-builder master: [WIP] build tinyipa on tinycore 13.x https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/827137 | 11:08 |
| iurygregory | good morning Ironic o/ | 11:23 |
| dtantsur | arne_wiebalck: I think it may be better for you (= CERN) to take ownership of it | 11:23 |
| dtantsur | we can review things, but it won't scale long-term | 11:23 |
| dtantsur | cc rpittau | 11:23 |
| rpittau | I agree, in the long term seems like the best solution | 11:24 |
| arne_wiebalck | dtantsur: you mean move it out of the metal3 project | 11:24 |
| arne_wiebalck | on github? | 11:24 |
| dtantsur | yeah | 11:25 |
| arne_wiebalck | dtantsur: rpittau: The repo/image may not make sense as part of metal3 anymore, but how about adding it to IPA or IPAB? If we think it has value and we should keep it, it should probably be part of the project to be found easily (rather than become a CERN thing). | 12:14 |
| arne_wiebalck | Otherwise I am also fine with making it a downstream only repo. | 12:15 |
| dtantsur | arne_wiebalck: not sure about adding it to existing repos, but we could potentially take the repo under our umbrella | 12:51 |
| dtantsur | well, yeah, IPA-builder may be the place | 12:52 |
| dtantsur | now that I think about it | 12:52 |
| arne_wiebalck | some worked though | 13:08 |
| arne_wiebalck | sorry, wrong window | 13:12 |
| arne_wiebalck | dtantsur: sooner or later we will have someone who wants to do manual inspection, would be good if had all in place then (and what we have at the moment seems to work ok) | 13:13 |
| * arne_wiebalck is launching the container on 200 nodes as we speak | 13:13 | |
| dtantsur | cool :) | 13:13 |
| dtantsur | yeah, let's start with IPA-builder? and let's try to make it as generic as possible | 13:14 |
| arne_wiebalck | sounds good | 13:16 |
| * arne_wiebalck has DoS-attacked the target inspector, needs to go slower :-D | 13:16 | |
| dtantsur | :D | 13:21 |
| dtantsur | well, it's called "load testing", I hope you've learned something useful out of it? :) | 13:21 |
| rpittau | dtantsur, arne_wiebalck, it's fine to me to add that to IPA-builder as far as we add a CI job (even non-voting) to verify the image actually builds fine | 13:38 |
| dtantsur | should be quite trivial | 13:38 |
| arne_wiebalck | dtantsur: yeah: don't do it! | 13:39 |
| arne_wiebalck | :-D | 13:39 |
| rpittau | tinycore13-based tinyipa is passing CI, going to add a release note https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/827137 | 13:41 |
| dtantsur | wow, neat | 13:41 |
| dtantsur | any memory increases needed? :) | 13:41 |
| rpittau | mmm need to double-check | 13:41 |
| rpittau | I mean, it was not necessary in the patch | 13:42 |
| rpittau | so probably not | 13:42 |
| dtantsur | rpittau: maybe create a testing IPA patch with depends-on? | 13:42 |
| rpittau | yeah | 13:42 |
| rpittau | on it | 13:43 |
| dtantsur | iurygregory: hi, should we start doing sprint 2 releases today? | 13:43 |
| dtantsur | or at least clean up release notes? | 13:43 |
| iurygregory | dtantsur, if CI is back (for master at least) I think we should =) | 13:44 |
| dtantsur | I think it has been in a good state since yesterday? | 13:44 |
| dtantsur | there are few bifrost patches that can be landed | 13:45 |
| *** akahat is now known as akahat|rover | 13:45 | |
| iurygregory | ack | 13:45 |
| iurygregory | let me take a look | 13:45 |
| dtantsur | ironic-inspector seems good to go | 13:45 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent master: [DNM] Test tinycore13-based tinyipa https://review.opendev.org/c/openstack/ironic-python-agent/+/827664 | 13:46 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic-python-agent master: Clean up release notes https://review.opendev.org/c/openstack/ironic-python-agent/+/827666 | 13:49 |
| dtantsur | iurygregory: IPA should be good after ^^^ | 13:49 |
| iurygregory | dtantsur, ack | 13:50 |
| iurygregory | I'm looking at https://review.opendev.org/c/openstack/bifrost/+/827009 | 13:50 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent-builder master: Build tinyipa on tinycore 13.x https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/827137 | 13:50 |
| iurygregory | I got a resource not found for the artifact in the upgrade job | 13:50 |
| dtantsur | iurygregory: hmm, maybe we don't generate bifrost.log for upgrade jobs | 13:51 |
| dtantsur | please leave a comment and ignore it for now. this is not release-bound. | 13:51 |
| iurygregory | yeah =) | 13:51 |
| iurygregory | the other jobs looks ok =) | 13:51 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent master: [DNM] Test tinycore13-based tinyipa https://review.opendev.org/c/openstack/ironic-python-agent/+/827664 | 13:53 |
| dtantsur | https://review.opendev.org/c/openstack/ironic/+/823347/ should be good to go for ironic | 13:56 |
| *** akahat|rover is now known as akahat | 13:56 | |
| dtantsur | I'd also quite appreciate https://review.opendev.org/c/openstack/ironic/+/826927 and https://review.opendev.org/c/openstack/ironic/+/825305, this is blocking netboot deprecation | 13:56 |
| dtantsur | iurygregory: ^^ | 13:57 |
| iurygregory | ack o/ | 13:57 |
| dtantsur | https://review.opendev.org/c/openstack/ironic/+/823913/ should be also good to go | 13:58 |
| dtantsur | https://review.opendev.org/c/openstack/ironic/+/826467/ and https://review.opendev.org/c/openstack/ironic/+/826470/ are quite trivial, but useful for combined ironic | 13:58 |
| dtantsur | https://review.opendev.org/c/openstack/ironic/+/826143/ fixes a pretty serious bug btw, attention appreciated | 13:59 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic bugfix/18.1: Remove redfish cache entry upon errors https://review.opendev.org/c/openstack/ironic/+/820589 | 14:00 |
| opendevreview | Merged openstack/ironic bugfix/19.0: Add additional ramdisk tests https://review.opendev.org/c/openstack/ironic/+/827237 | 14:05 |
| rpittau | just great, I can't reproduce the issue with debian ipa on bifrost locally.... it just works | 14:11 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic-tempest-plugin master: Apply the correct image reference when booting an instance https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/827340 | 14:11 |
| * rpittau yells at cloud | 14:11 | |
| dtantsur | rpittau: \o/ | 14:11 |
| dtantsur | and what's the issue? | 14:12 |
| rpittau | the issue is network interfaces not getting ips from dhcp | 14:12 |
| dtantsur | using dhcp-all-interfaces? | 14:12 |
| rpittau | https://review.opendev.org/c/openstack/bifrost/+/827293 <- this | 14:12 |
| rpittau | yeah | 14:12 |
| rpittau | in CI does not work, tried locally now and it works | 14:12 |
| opendevreview | Merged openstack/ironic stable/xena: Add additional ramdisk tests https://review.opendev.org/c/openstack/ironic/+/826507 | 14:15 |
| opendevreview | Merged openstack/ironic stable/wallaby: Add additional ramdisk tests https://review.opendev.org/c/openstack/ironic/+/826508 | 14:17 |
| rpittau | nvm... I used a prebuilt ipa | 14:18 |
| rpittau | /me should not multitask too much | 14:18 |
| TheJulia | I figured out a downside to being on the board and having my name in press releases.... | 14:23 |
| opendevreview | Merged openstack/bifrost master: CI: store bifrost.log as a Zuul artifact https://review.opendev.org/c/openstack/bifrost/+/827009 | 15:01 |
| *** sdanni is now known as Guest1646 | 15:01 | |
| *** sdanni_ is now known as sdanni | 15:01 | |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: CI: properly report failures in the upgrade job https://review.opendev.org/c/openstack/bifrost/+/827687 | 15:11 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: CI: properly publish artifacts for the upgrade job https://review.opendev.org/c/openstack/bifrost/+/827688 | 15:11 |
| dtantsur | core reviewers, could you please check the patches I pasted above? would be good to fit them into the release? | 15:19 |
| rpittau | I'm havin a look now | 15:20 |
| dtantsur | thanks! | 15:20 |
| dansmith | dtantsur: do you know if anyone from ironic is looking at the zuul config errors that infra asked to have cleaned up? | 15:23 |
| TheJulia | dansmith: I doub't we're consciously aware of such. | 15:24 |
| TheJulia | dansmith: Where can we learn of this? | 15:24 |
| dansmith | TheJulia: https://zuul.opendev.org/t/openstack/config-errors | 15:24 |
| dansmith | there was a mail, I missed it too | 15:24 |
| dansmith | the top two entries there are issues in ironic jobs that reference old/EOLed project trees | 15:25 |
| dansmith | infra wants them cleaned up | 15:25 |
| dansmith | oh there are some others further down | 15:26 |
| dansmith | I think if you search in the page for "openstack/ironic -" you will jump to them | 15:26 |
| dansmith | status tracking here: https://etherpad.opendev.org/p/zuul-config-error-openstack | 15:27 |
| dtantsur | dansmith: I think they touch ancient versions | 15:32 |
| dtantsur | which nobody is going to change now | 15:32 |
| dtantsur | I think I even responded to the ML, but maybe I forgot | 15:32 |
| dansmith | that's the problem as I understand it | 15:33 |
| dansmith | those ancient jobs are technically still run-able by zuul, but reference projects that aren't | 15:33 |
| opendevreview | Merged openstack/ironic master: Fix redfish RAID failed tasks https://review.opendev.org/c/openstack/ironic/+/823347 | 15:34 |
| opendevreview | mitya-eremeev-2 proposed openstack/ironic master: Delete resource provider after node deletion. https://review.opendev.org/c/openstack/ironic/+/827295 | 15:37 |
| hjensas | dtantsur: The baremetal_node_info ansible module. I'm looking at output: https://paste.opendev.org/show/812501/ | 15:37 |
| hjensas | dtantsur: does 'nics' make sense, or should we instead replace 'ports' which currently just have href links? | 15:38 |
| hjensas | dtantsur: also, would it make sense to clean out some data, like nested location and links in ports? And possibly states which is href links? | 15:38 |
| TheJulia | hjensas: nics was an early bifrost support thing | 15:39 |
| TheJulia | I guess it is basically still used | 15:39 |
| TheJulia | nics: address: $mac_address | 15:40 |
| hjensas | TheJulia: yes, it is used in baremetal_node, but this is new module i.e os_ironic_info. | 15:40 |
| hjensas | TheJulia: In my first iteration I extend the node with 'nics' property which is basically list_nics_for_machine(server.uuid) | 15:41 |
| TheJulia | ahh! | 15:41 |
| * TheJulia is being scaled by a cat | 15:42 | |
| * TheJulia is a cat substrate | 15:42 | |
| hjensas | TheJulia: I think maby just replacing ports would be better. | 15:42 |
| TheJulia | ++ | 15:42 |
| TheJulia | yeah, the links are semi-unfriendly | 15:42 |
| TheJulia | :( | 15:42 |
| TheJulia | since they can't directly curl the url | 15:43 |
| hjensas | ack, I wonder if I should remove other stuff that is 'links' as well. Like 'states', 'portgroups' and 'links' in each port entry. | 15:44 |
| TheJulia | portgroups might be useful too... | 15:45 |
| TheJulia | fwiw | 15:45 |
| TheJulia | but yeah, I feel like we nuked links elsewhere | 15:45 |
| dtantsur | hjensas: yeah, we need to nuke links | 15:55 |
| dtantsur | re "nics" vs "ports"... dunno. do we have prior art in other modules? | 15:55 |
| hjensas | ack, I'll add some cleanu. | 15:55 |
| dtantsur | "states" is largely useless, everything is available on the node | 15:56 |
| dtantsur | hjensas: btw, check https://opendev.org/openstack/bifrost/src/branch/master/playbooks/library/os_ironic_node_info.py for any useful tips | 15:57 |
| dtantsur | (doesn't seem to be much, but just in case) | 15:57 |
| * TheJulia wonders... where did the brain go after the last call | 16:00 | |
| opendevreview | Dmitry Tantsur proposed openstack/ironic-tempest-plugin master: Apply the correct image reference when booting an instance https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/827340 | 16:03 |
| dtantsur | I think this ^^ is really close, the last revision fixes the assumption that ephemeral partitions are created for whole-disk images. | 16:04 |
| TheJulia | oh joy | 16:08 |
| TheJulia | dtantsur: dansmith what list was this email on? | 16:09 |
| dtantsur | TheJulia: openstack-discuss | 16:09 |
| dansmith | http://lists.openstack.org/pipermail/openstack-discuss/2021-November/025797.html | 16:09 |
| TheJulia | oh! november | 16:09 |
| TheJulia | that is why I'm not seeing it | 16:09 |
| iurygregory | oh wow | 16:09 |
| dtantsur | I did respond apparently http://lists.openstack.org/pipermail/openstack-discuss/2021-November/025874.html | 16:10 |
| hjensas | dtantsur: ah, thanks I was not aware of the node info module in bifrost. | 16:11 |
| dtantsur | hjensas: your version is definitely more advanced | 16:11 |
| hjensas | dtantsur: It seems to use 'macs' ... however I feel replacing ports is more aligned to ironic nameing convention. | 16:12 |
| dtantsur | yeah, macs is pretty bad | 16:14 |
| dtantsur | ironic ports are more than just addresses nowadays | 16:14 |
| TheJulia | ++ | 16:15 |
| dtantsur | was different 7 years ago I assume :) | 16:15 |
| TheJulia | very different | 16:15 |
| TheJulia | long before pxe_enabled even | 16:15 |
| dtantsur | long before indeed | 16:15 |
| dtantsur | it's actually pretty fun to find things in bifrost that are so old :) brings back memories | 16:15 |
| TheJulia | so I have 45 minutes before my next call, I can try and do some quick cleanup, but I'll likely just force approve my own patches since they are dead branches basically | 16:16 |
| TheJulia | Mostly it brings back memory of bars where I was quite drunk when I was writing code | 16:16 |
| dtantsur | OF COURSE the CI is already broken, why not | 16:17 |
| TheJulia | are fixes tagged? I can do reviews instead | 16:17 |
| iurygregory | I only wrote code while I was drunk when I was in the university XD | 16:17 |
| dtantsur | no. I'm fixing error reporting in the bifrost upgrade job, and of course it's broken | 16:17 |
| TheJulia | ugh | 16:18 |
| TheJulia | iurygregory: I had long meetings at HP in Seatle and had nothing to do so I ended up at the hotel bar each night with my laptop | 16:18 |
| TheJulia | it was not ideal. | 16:18 |
| iurygregory | oh wow | 16:18 |
| dtantsur | bloody libvirt | 16:20 |
| JayF | TheJulia: saw that for a sec and was like "Julia's in Seattle?!" then realized it was past tense lol | 16:20 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: Revert "Install libvirt-python from source instead of a wheel" https://review.opendev.org/c/openstack/bifrost/+/827648 | 16:21 |
| dtantsur | maybe this will help ^^ or break even more | 16:22 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: CI: properly publish artifacts for the upgrade job https://review.opendev.org/c/openstack/bifrost/+/827688 | 16:23 |
| opendevreview | Julia Kreger proposed openstack/ironic stable/rocky: Cleanup stable/rocky legacy jobs https://review.opendev.org/c/openstack/ironic/+/827709 | 16:23 |
| opendevreview | Julia Kreger proposed openstack/ironic stable/queens: Remove legacy experimental jobs https://review.opendev.org/c/openstack/ironic/+/827713 | 16:28 |
| TheJulia | other ci systems barf on the config now, but *shrugs* | 16:28 |
| TheJulia | oh, they would have barfed anyway | 16:29 |
| TheJulia | unrelated | 16:29 |
| TheJulia | are we taking bets with zuul cookies on odds for anything actually working on those branches? | 16:31 |
| rpittau | see ya tomorrow! o/ | 16:39 |
| dtantsur | TheJulia: rocky and queens? oh I doubt | 16:43 |
| * TheJulia feels like the home office needs a nice warm blanket | 16:44 | |
| * dtantsur has a thick woollen blanket from Russia :) | 16:45 | |
| TheJulia | typically I don't need one, but we've got a wind storm bringing cold air down upon us | 16:47 |
| opendevreview | Merged openstack/ironic bugfix/18.1: Remove redfish cache entry upon errors https://review.opendev.org/c/openstack/ironic/+/820589 | 16:51 |
| dtantsur | is there an IPA change we could merge? the current DIB builds are broken because of zeroconf version. | 17:07 |
| opendevreview | Merged openstack/ironic master: Make account prefix of Swift confgurable https://review.opendev.org/c/openstack/ironic/+/823913 | 17:17 |
| dtantsur | TheJulia: could you merge https://review.opendev.org/c/openstack/ironic-python-agent/+/827666/ please? we need to trigger a build, the current DIB image is broken. | 17:18 |
| dtantsur | .. which teaches us a lesson: we need a gating DIB job on IPA, at least to ensure we don't publish broken images | 17:20 |
| dtantsur | ugh, we don't have a single DIB job on IPA, do we? | 17:28 |
| opendevreview | Merged openstack/ironic master: Fix resource_url in the remaining resources https://review.opendev.org/c/openstack/ironic/+/826143 | 17:30 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic stable/xena: Fix resource_url in the remaining resources https://review.opendev.org/c/openstack/ironic/+/827731 | 17:30 |
| *** sshnaidm is now known as sshnaidm|afk | 17:30 | |
| opendevreview | Dmitry Tantsur proposed openstack/ironic bugfix/19.0: Fix resource_url in the remaining resources https://review.opendev.org/c/openstack/ironic/+/827732 | 17:30 |
| arne_wiebalck | bye everyone, see you tomorrow o/ | 17:31 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic-python-agent master: Switch one of the voting jobs to DIB images https://review.opendev.org/c/openstack/ironic-python-agent/+/827729 | 17:36 |
| dtantsur | hmm, actually, they all use DIB. wtf. | 17:39 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic-python-agent master: CI: be explicit that the jobs are using the DIB ramdisk https://review.opendev.org/c/openstack/ironic-python-agent/+/827729 | 17:41 |
| TheJulia | released vs master maybe? | 17:53 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: Clean up jobs with legacy names https://review.opendev.org/c/openstack/ironic/+/827752 | 17:59 |
| dtantsur | or maybe I'm just tired | 17:59 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: Clean up jobs with legacy names https://review.opendev.org/c/openstack/ironic/+/827752 | 18:01 |
| dtantsur | oh lovely, whole disk netboot does not work under UEFI | 18:04 |
| opendevreview | Merged openstack/ironic-python-agent master: Clean up release notes https://review.opendev.org/c/openstack/ironic-python-agent/+/827666 | 18:05 |
| dtantsur | TheJulia: any thoughts on forcing local boot for whole disk images? | 18:08 |
| iurygregory | wow a lot of conversation while I was having lunch :D | 18:08 |
| dtantsur | hehe | 18:08 |
| iurygregory | I think our plan in IPA was "we will use DIB by default in all jobs, if we find problems in some jobs we wouldn't use DIB" =) | 18:09 |
| dtantsur | yeah, I missed the fact that DIB is still the default in ironic-base | 18:11 |
| iurygregory | gotcha =) | 18:12 |
| iurygregory | aren't we default to local boot for whole disk images? | 18:12 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: [DNM] Testing the CI https://review.opendev.org/c/openstack/ironic/+/827500 | 18:13 |
| dtantsur | iurygregory: nope. network boot is possible for whole disk images, although it boils down to just 'sanboot --no-describe'. | 18:13 |
| dtantsur | which, according to https://ipxe.org/err/2c2220 and our CI, does not work on UEFI | 18:13 |
| iurygregory | enr ... | 18:13 |
| iurygregory | D: | 18:13 |
| parasitid | hi | 18:19 |
| parasitid | i've pushed this repo, for what it's worth https://github.com/yanndegat/irobox | 18:20 |
| parasitid | it's heavily based on metal3 ironic image | 18:20 |
| dtantsur | parasitid: it's interesting. although, how much overlap does it have with kolla? | 18:21 |
| parasitid | i know there are other similar projects such as kyaobe or bifrost, still, i'm more comfortable with the docker compose stuff to pop up a stack | 18:21 |
| parasitid | dtantsur: don't now, didn't try kolla, i only have a bad experience with kolla ansible | 18:23 |
| parasitid | it's more ironic oriented | 18:23 |
| dtantsur | ah, I see | 18:23 |
| dtantsur | parasitid: it would be cool to hear your feedback on what can be improved or simplified in ironic | 18:23 |
| dtantsur | based on what you've experienced while building this project | 18:23 |
| * TheJulia needs to find fresh brains | 18:24 | |
| * TheJulia is exhausted and it is not even noon yet | 18:24 | |
| dtantsur | FREAAASH BRAINZZ | 18:24 |
| TheJulia | lol | 18:24 |
| * TheJulia needs to make more coffee or something | 18:24 | |
| TheJulia | or an early lunch | 18:25 |
| dtantsur | speaking of exhausted, I should probably go already | 18:25 |
| parasitid | the most struggling part (and not achieved yet) is having a full ironic+neutron+genericswitch with vlan working :) | 18:25 |
| parasitid | i miss some docs on this topic | 18:25 |
| dtantsur | yeah, that's the hardest part | 18:25 |
| TheJulia | parasitid: that is a great goal, I think i eventually abandoned the bifrost patch that would have done the neutron + networking_generic_switch integration :( | 18:26 |
| dtantsur | TheJulia: I'm dreaming of reviving it one day :) | 18:26 |
| TheJulia | dtantsur: ++ | 18:26 |
| dtantsur | but I'd like OVN to start supporting Ironic first, so that we don't have to bring back rabbitmq | 18:26 |
| iurygregory | coffee++ | 18:27 |
| parasitid | do you OVN, in the sense that it would pilot switch config through openflow ? instead of netmiko in the generic switch ? | 18:27 |
| dtantsur | parasitid: no, I mean the neutron's OVN-based ML2 driver instead of the half-deprecated OVS one | 18:28 |
| dtantsur | it won't help you much | 18:28 |
| TheJulia | parasitid: so I was doing ovs, but that was before ovn. | 18:28 |
| TheJulia | yeah, the openflow updates won't propogate to a real switch | 18:28 |
| TheJulia | aeva was super unhappy upon figuring that out a long time ago | 18:28 |
| dtantsur | I can imagine :) | 18:29 |
| dtantsur | I think networking-generic-switch is the way to go for physical switches for the time being | 18:29 |
| TheJulia | ++ | 18:30 |
| * TheJulia needs to go find wifey and talk to her for a minujte | 18:30 | |
| parasitid | TheJulia: is there a chance the bifrost/generic switch patch you were referring to is still dangling in some unknown git branch somewhere ? | 18:34 |
| TheJulia | branch no | 18:34 |
| TheJulia | but in gerrit yes | 18:34 |
| TheJulia | give me a few and I'll look | 18:35 |
| TheJulia | I started developing it for a keynote actually | 18:35 |
| dtantsur | parasitid, TheJulia, https://review.opendev.org/c/openstack/bifrost/+/452514 | 18:37 |
| dtantsur | and following after it | 18:37 |
| TheJulia | parasitid: https://review.opendev.org/c/openstack/bifrost/+/452514/ https://review.opendev.org/c/openstack/bifrost/+/498271 https://review.opendev.org/c/openstack/bifrost/+/452515 https://review.opendev.org/c/openstack/bifrost/+/498972 | 18:37 |
| TheJulia | the last one is LLDP related, so the information to manage switchports could be discovered via inspector | 18:37 |
| dtantsur | okay, going for real now, see you tomorrow | 18:40 |
| opendevreview | mitya-eremeev-2 proposed openstack/ironic master: Delete resource provider after node deletion. https://review.opendev.org/c/openstack/ironic/+/827295 | 18:52 |
| parasitid | TheJulia: dtantsur thanks a lot | 19:12 |
| aman | TheJulia, I'm an intern working on ESI, currently investigating issues around leasing Bare Metal hardware. I want to know the limits of rescue mode, more specifically if the drivers are tempered, let's say a lessee messes up BIOS or iDRAC settings | 19:14 |
| TheJulia | aman: tempered as in hardened? | 19:15 |
| aman | Tampered* lol my bad | 19:15 |
| JayF | I mean, Rescue mode at its core is just "boot a ramdisk on the instance designed to be logged into by the end user" | 19:16 |
| JayF | Nothing in stock-Ironic would be able to rescue a machine where BIOS/UEFI settings had been tampered with to a point where that was not possilbe. | 19:17 |
| JayF | Classic use cases for rescue mode include trying to recover an OS that was damaged, data recovery in some disk failure situations, password recovery, etc | 19:17 |
| TheJulia | aman: ahh, okay. So... rescue mode is not going to help if someone messes up the BMC horribly. Of course, that was never intended and intentionally (by default) don't grant acess api wise which could be used as such. If the lessee had credentials and in-band methods or connectivity (which is bad and should be disabled/prevented... for this is why ironic exists) in the first place. | 19:17 |
| JayF | I'll say in general this problem space is not largely explored by Ironic -- you kinda have to bring your own hardware security, and hardware security knowledge, then Ironic will happily help you automate the steps you want to take to secure hardware | 19:18 |
| TheJulia | aman: BIOS wise... we kindly ask the hardware to network boot as JayF indicated. If it no longer does so, then that will put a machine in Rescue Fail and at absolute worst if someone has broken the machine it is a roll someone out to the data center sort of intervention. | 19:18 |
| TheJulia | which is a last resort sort of thing, but at that point it sounds like the lessee has bricked the machine potentially | 19:19 |
| JayF | For instance, when I ran a large public Ironic cloud, "leasing" machines out (not Ironic-lease, but Nova instances, but a similar idea -- untrusted customer with root on the machine) -- I'd say 40% of the overall development effort was in validating hardware, hardening firmwares, and working with vendors to make that possible | 19:20 |
| TheJulia | and definitely not granting them access to lower level settings | 19:21 |
| JayF | Ironic has more knobs to automate that stuff now than it did then; but you still have to do the legwork to determine how to secure the hardware | 19:21 |
| JayF | And quite frankly: I'm not sure hardware exists off the shelf that can be secured to that level | 19:21 |
| JayF | We achieved probably 90% security coverage on firmwares by having custom firmwares made, working closely with vendors to lock them down, etc | 19:21 |
| TheJulia | HPE has some awesome lockdown knobs if memory serves | 19:22 |
| TheJulia | but this is one of the cases engineering there thinks about | 19:22 |
| JayF | TheJulia: leftover influence from you? :D | 19:22 |
| TheJulia | JayF: doubt it :) | 19:22 |
| aman | JayF TheJulia thanks that's all the info I needed. I was thinking the same, if boot sequence is changed and network boot is moved down, rescue shouldn't work. We just want to document the risks for now. | 19:25 |
| JayF | I mean, Ironic tells the BMC "network boot" | 19:25 |
| JayF | So it does attempt to manipulate the boot order | 19:25 |
| TheJulia | What JayF said | 19:26 |
| JayF | but unless you've successfully locked down the BIOS/UEFI from in-band modification, you cannot be guaranteed an attacker wouldn't subvert that | 19:26 |
| JayF | the weak link in the chain is "someone with root on the box who is untrusted", becuase most hardware isn't designed to protect itself from a root user | 19:26 |
| TheJulia | We don't edit a bios boot order. UEFI boot order we do change upon image deployment, unfortunately I don't think that *can* be really locked down from the OS... :( | 19:26 |
| JayF | TheJulia: not without the custom firmware and working with vendors as I referenced above | 19:28 |
| JayF | but even that was like... "pretty secure" but not great lol | 19:28 |
| TheJulia | Yeah, the conundrum is we've got things like shim which looses it's mind and crashes the machine if it has a duplicate or can't force insert itself as the boot order entry | 19:29 |
| JayF | The machines I worked on were BIOS boot only, which simplified things :) | 19:29 |
| TheJulia | and complicates some other things :) | 19:29 |
| aman | JayF, I get your point now, the weak link in the chain made it clear. I am going to c/p our conversation, and discuss it further with tzumainn | 19:32 |
| aman | https://etherpad.opendev.org/p/ironic_rescue_limits these were my findings yet | 19:33 |
| aman | JayF, TheJulia thanks a lot! | 19:36 |
| parasitid | TheJulia: hi again, i'm digging into your patches. one thing struggles me: do you remember why you didn't put the baremetal mech drivers alongside the openvswitch+ genericswitch ? i still don't get if these 2 drivers are complementary or not (https://review.opendev.org/c/openstack/bifrost/+/452514/23/playbooks/roles/bifrost-neutron-install/templates/ml2_conf.ini.j2#5 ) | 20:03 |
| TheJulia | parasitid: it didn't exist then | 20:07 |
| parasitid | TheJulia: oh i see, so now both should be enabled ? correct ? | 20:10 |
| TheJulia | parasitid: likely yes | 20:14 |
| timeu | parasitid: AFAIK this is related to neutron's hierarchical port binding in neutron. Openvswitch will take care of the network configuration on hypervisors/controllers and the genericswitch driver will configure the switch ports of the switch that the baremetal machines are connected to. | 20:38 |
| TheJulia | timeu: yes | 21:35 |
| TheJulia | networking-baremetal does do neutorn port maping updates internally | 21:35 |
| TheJulia | there is a case where it is required, and it also does the flat binding completion on vnic_baremetal | 21:36 |
| TheJulia | so it doesn't look like binding failed | 21:36 |
| TheJulia | stevebaker[m]: so I've updated that etherpad with a idea of a _test_request method andminor helper which might work. Thinking about it, if we want to do it right, we might need an intermediate class of steps. For example, know we do x as admin, y as user, but that z should fail for that user. | 21:37 |
| TheJulia | stevebaker[m]: I guess the conundrum... UUIDs for resources | 21:37 |
| stevebaker[m] | TheJulia: ok, I'll take a look on Tuesday. I'm taking sick leave today | 21:38 |
| TheJulia | stevebaker[m]: eek! :( | 21:38 |
| TheJulia | feel better! | 21:38 |
| stevebaker[m] | thanks, I'll feel better when the covid test results are back :) | 21:38 |
| TheJulia | oh noes :( | 21:38 |
| opendevreview | Bob Fournier proposed openstack/sushy-tools master: Accept integer types for BIOS settings https://review.opendev.org/c/openstack/sushy-tools/+/827769 | 22:02 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!