| TheJulia | Out of curiosity, the idea, or via something like SZTP? | 00:00 |
|---|---|---|
| cardoe | My wife slapped a treadmill under my standing desk... | 00:01 |
| TheJulia | She sounds very wise | 00:01 |
| cardoe | Well need to use SZTP | 00:01 |
| cardoe | For me it's gonna be a marathon and not a sprint cause there's a LOT of legacy install base. | 00:02 |
| TheJulia | semi-ouch in that has a very particular model really not geared for "oh, I have servers in a rack" | 00:02 |
| cardoe | okay well lemme walk that back not like a hard requirement | 00:03 |
| TheJulia | Then again, maybe after the fact once you deal with the qr code and all | 00:03 |
| TheJulia | oh, okay | 00:03 |
| TheJulia | yes, plan your process :) | 00:03 |
| TheJulia | one of the former maintainers once told me about a whole workflow he had worked out where they would wheel servers in and they would be consumed/inspected/checked/benched/deployed | 00:04 |
| cardoe | Ultimately I need to burn down a LOT of legacy code and a LOT of existing process and such. | 00:04 |
| TheJulia | oh, those are the best marathons :) | 00:04 |
| cardoe | But the effort is out in the open from day one, though a lot of convos and such aren't and I'm working to move it all open. | 00:04 |
| TheJulia | That is one of the hardest aspects of any transformation project | 00:05 |
| TheJulia | that and keeping people from going "no! that is my process! (because without it I wouldn't have a job)" | 00:06 |
| cardoe | You actually know a handful of the folks working with me. I'll let them out themselves. | 00:06 |
| TheJulia | rutro! | 00:06 |
| TheJulia | :) | 00:06 |
| TheJulia | it is all good | 00:06 |
| cardoe | I come from the servers world though so network is newer to me. | 00:06 |
| TheJulia | That is a big vendor centric space :( | 00:06 |
| cardoe | So right now driving down smaller "demos" if you will. Using Nautobot and aiming to do some integrations between it and Neutron via plugins. | 00:07 |
| cardoe | Anyway, happy to contribute and be involved to the conversations and code. | 00:10 |
| TheJulia | very cool | 00:10 |
| cardoe | Similar-ish to how the Yaook Cloud folks have the OpenStack in k8s operator and are using Netbox as their IPAM. | 00:11 |
| cardoe | Nah. The cool work is from you guys. Just standing on the shoulders of giants. | 00:13 |
| TheJulia | The whole thing with networking has kind of uncovered/highlighted some of the challenges before Neutron itself since it wants/really expects to be IPAM | 00:13 |
| cardoe | Neutron thinking it's an IPAM makes me think of the southern saying "bless your heart". As in Neutron saying "I'm a full IPAM!!!".... bless your heart.. | 00:15 |
| TheJulia | oh my | 00:15 |
| * TheJulia knows this saying very well | 00:15 | |
| * TheJulia feels the need for a mint julep | 00:16 | |
| cardoe | Maybe I'm wrong... but every other IPAM out there has quite a bit more in it. | 00:16 |
| TheJulia | Indeed! | 00:17 |
| cardoe | But first I'm trying to figure out everything the ironic conductor needs cause the OpenStack Helm project currently starts up the conductor container with the host IPC, network, and PID namespace. Runs as UID 0, mapped to host UID 0. And mounts /dev, /proc, /var, and /sys from the host filesystem. Something tells me that might be a touch overkill. | 00:39 |
| TheJulia | I could see some of that back in the iscsi deployment interface days | 00:42 |
| TheJulia | but after the wallaby release, that was... eliminated. | 00:43 |
| opendevreview | Jacob Anders proposed openstack/sushy-tools master: [WIP] Add support for BIOS update emulation https://review.opendev.org/c/openstack/sushy-tools/+/909500 | 04:56 |
| rpittau | good morning ironic! o/ | 07:06 |
| opendevreview | Riccardo Pittau proposed openstack/virtualpdu master: [DNM] Test timeout https://review.opendev.org/c/openstack/virtualpdu/+/922158 | 07:14 |
| rpittau | JayF, TheJulia, I've created a revert for the time being https://review.opendev.org/c/openstack/requirements/+/922186 | 07:20 |
| rpittau | We'll be more relaxed when working on virtualpdu I guess :) | 07:20 |
| opendevreview | Pierre Riteau proposed openstack/tenks master: Bump minimum Ansible version https://review.opendev.org/c/openstack/tenks/+/921620 | 07:22 |
| opendevreview | Pierre Riteau proposed openstack/tenks master: Bump minimum Ansible version https://review.opendev.org/c/openstack/tenks/+/921620 | 08:07 |
| mnasiadka | Good morning Ironic | 09:12 |
| mnasiadka | Is there a way to supply custom dhcp opts to Neutron e.g. ntp server or other things? | 09:12 |
| opendevreview | Przemyslaw Szczerbik proposed openstack/ironic master: Fix execution of node servicing steps exposed by IPA's HardwareManager https://review.opendev.org/c/openstack/ironic/+/922024 | 09:26 |
| sylvr | Hello ! I'd like to know more about LLDP for autodiscovery, is there other documentation/reference on that apart from this link : https://www.stackhpc.com/ironic-idrac-ztp.html ? Thanks ! :) | 10:47 |
| sylvr | oops, this is more about Kayobe than Ironic, should I ask in #openstack-kolla? | 10:48 |
| Sandzwerg[m] | Morning Ironic. I have A question regarding conductors: how can I delete one? We've deleted the compute service, and the running agent but I still see the conductors with "baremetal conductor list" but there is no "conductor delete" So we seem to be missing something to remove the conductor completely. | 11:02 |
| dtantsur | Sandzwerg[m]: 'systemctl stop ironic-conductor' :) | 12:21 |
| Sandzwerg[m] | The conductor itself is gone as well (and thus reported as dead) | 12:22 |
| dtantsur | Ah, I see. It's probably going to stay there | 12:23 |
| Sandzwerg[m] | So the only possibility to remove it, would be to remove it directly from the DB? | 12:24 |
| Sandzwerg[m] | They don't really hurt but we're now decommissioning more and more old hardware and thus conductors with them so over time it will clutter | 12:26 |
| dtantsur | I think so, I don't remember this part of the code very well | 12:29 |
| dtantsur | (maybe we need a periodic task to remove really old records) | 12:30 |
| TheJulia | good morning | 12:57 |
| TheJulia | We likely *do* need such a periodic | 13:48 |
| TheJulia | delete conductors >2 weeks old or something like that | 13:48 |
| JayF | That would be a pretty low hanging feature too, basically just ape the implementation for node history | 14:13 |
| samcat116 | Related to dead agents: it seems like when you delete an ironic node, the ironic-neutron-agent instance for that node isn't deleted and stays around as a dead agent. I see this on it: https://access.redhat.com/solutions/3913411, but I can't access that. Not sure if this is an ironic or a neutron thing. | 14:20 |
| JayF | samcat116: might be worth documenting that in an ironic+neutron bug | 14:30 |
| dtantsur | samcat116: the solution suggests $ openstack network agent delete UUID | 14:31 |
| JayF | I'm wondering... is there space for a feature for driver migration *inside Ironic* | 14:40 |
| JayF | e.g. if you wanna go from old vendor driver -> generic redfish,having ironic do it for you over time? | 14:41 |
| rpittau | Reverbverbverb: I had a first read of the ironic docs analysis, it's brutal, I like it :D | 14:43 |
| rpittau | I left a couple of simple comments to start with | 14:43 |
| TheJulia | samcat116: I suspect there is room for the agent to have some cleanup/reconcile logic as well | 14:48 |
| TheJulia | JayF: Maybe, maybe not, dunno. If there is a time to do it, the time to strike the steel is now-ish | 14:48 |
| JayF | that is basically what I am thinking, too | 14:49 |
| JayF | My downstream would likely use an ilo->redfish converter thing | 14:49 |
| JayF | I just can't wrap my mind around how to configure it | 14:49 |
| JayF | [ilo] convert_to_redfish = True => runs a periodic task that updates nodes in AVAILABLE / MANAGABLE state to redfish driver? | 14:49 |
| JayF | but then you'd have little option to configure that move | 14:50 |
| dtantsur | JayF: sounds like an optional (?) online_data_migration | 14:50 |
| JayF | or maybe even, just if you update node.driver with a new enough api microversion, we will automatically migrate creds to the new keys that driver uses | 14:50 |
| TheJulia | maybe not periodic tasks | 14:51 |
| JayF | I will ponder on this, I am going to point my brain at getting virtualbmc unscrewed (rollback or not it's going to be a problem, we can't ignore it forever unless we wanna just get rid of snmp ci) | 14:51 |
| JayF | I think I might rewrite it to embrace asyncio instead of trying to thread out all the separate snmp server instances | 14:52 |
| JayF | instead of getting asyncio+threads to play nice | 14:52 |
| TheJulia | *but* there might be room for a general "cleanup/housekeeping periodic" overall, we sort of already have one. | 14:52 |
| JayF | and have configurable things be added to it | 14:52 |
| JayF | like removing old conductors | 14:52 |
| JayF | converting from old drivers | 14:52 |
| JayF | etc? | 14:52 |
| TheJulia | I think some folks will appreciate that a lot since there are folks out there who still use snmp | 14:52 |
| TheJulia | we always get a couple people who come out of the woodwork when we talk about getting rid of it | 14:53 |
| JayF | Yeah, SNMP is a good driver, and I think this is the force-me-to-learn-asyncio issue I've been waiting on | 14:53 |
| TheJulia | ++ | 14:53 |
| rpittau | JayF: about virtualpdu, I think this https://review.opendev.org/c/openstack/virtualpdu/+/922158 is kind of in the right direction, the unit tests are not failing but it's timing out | 14:55 |
| rpittau | mmm ok there' still one failure | 14:57 |
| JayF | I agree you're close to making it work; but I'm not sure what we'd end up with would be well-structured | 14:58 |
| JayF | versus if we ripped out the threads and went full coroutine given it's already forcing us to use asyncio to use those lower level apis | 14:58 |
| JayF | AFAICT pysnmp only left the sync-style APIs for the high level stuff, I don't think we can operate at the level vbmc does without interacting with it | 14:58 |
| JayF | but I'm dealing with MASSIVE amounts of unknown here so it's possible I'm way, way off | 14:58 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: WIP migration guide from inspector https://review.opendev.org/c/openstack/ironic/+/922089 | 15:01 |
| dtantsur | since we're on the topic of docs, early feedback welcome ^^ | 15:01 |
| tlegentil | Hi Everyone ! | 15:02 |
| tlegentil | I'm willing to manage baremetal servers with ironic, and I would like to use it within AIO. I've followed the https://docs.openstack.org/ironic/latest/contributor/devstack-guide.html BUT I always end up with the error : "++lib/neutron_plugins/services/l3:create_neutron_initial_network:202 oscwrap --os-cloud devstack --os-region RegionOne network create private -f value -c id | 15:03 |
| tlegentil | Error while executing command: HttpException: 503, Unable to create the network. No tenant network is available for allocation. | 15:03 |
| tlegentil | ++functions-common:oscwrap:2461 return 1 | 15:03 |
| tlegentil | +lib/neutron_plugins/services/l3:create_neutron_initial_network:202 NET_ID= | 15:03 |
| tlegentil | " | 15:03 |
| tlegentil | Could you please give me some hint ? Thanks ! | 15:03 |
| JayF | What configuration from that page are you using? Are you sure you actually ran the bash script instead of copying it into the file (a common mistake) | 15:17 |
| JayF | I know cid has been using configs from that page without issue so we just have to figure out what's up with your environment or process to get ya working | 15:17 |
| TheJulia | failure to create a tenant network is typically some level of access or there is a base networking misconfiguration somewhere, sort of depends on the actual susituted configuration into any example as well. | 15:19 |
| tlegentil | if I create the local.conf file with the example provided " Ironic with Nova ", then I have the HOST_ID that is not found. So I added the HOST_ID in the local.conf file. | 15:21 |
| tlegentil | network config is like this : "stack@Openstack-AiO:~/devstack$ ip a | 15:21 |
| tlegentil | 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 | 15:21 |
| tlegentil | link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 | 15:21 |
| tlegentil | inet 127.0.0.1/8 scope host lo | 15:21 |
| tlegentil | valid_lft forever preferred_lft forever | 15:21 |
| tlegentil | inet6 ::1/128 scope host | 15:21 |
| tlegentil | valid_lft forever preferred_lft forever | 15:21 |
| tlegentil | 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000 | 15:21 |
| tlegentil | link/ether bc:24:11:c6:eb:a8 brd ff:ff:ff:ff:ff:ff | 15:21 |
| tlegentil | altname enp0s18 | 15:21 |
| tlegentil | inet 10.210.44.201/24 brd 10.210.44.255 scope global eth0 | 15:22 |
| tlegentil | valid_lft forever preferred_lft forever | 15:22 |
| tlegentil | inet6 fe80::be24:11ff:fec6:eba8/64 scope link | 15:22 |
| tlegentil | valid_lft forever preferred_lft forever | 15:22 |
| tlegentil | 3: ens19: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 | 15:22 |
| tlegentil | link/ether bc:24:11:91:c2:d3 brd ff:ff:ff:ff:ff:ff | 15:22 |
| tlegentil | altname enp0s19 | 15:22 |
| tlegentil | 4: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 | 15:22 |
| tlegentil | link/ether ae:2f:3b:a9:d8:5d brd ff:ff:ff:ff:ff:ff | 15:22 |
| tlegentil | 5: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 | 15:22 |
| tlegentil | link/ether d2:9b:13:8f:07:39 brd ff:ff:ff:ff:ff:ff | 15:22 |
| tlegentil | 6: br-ex: <BROADCAST,MULTICAST> mtu 1400 qdisc noop state DOWN group default qlen 1000 | 15:22 |
| tlegentil | link/ether 4e:71:3f:fe:a2:40 brd ff:ff:ff:ff:ff:ff | 15:22 |
| tlegentil | 7: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000 | 15:22 |
| tlegentil | link/ether 52:54:00:82:9b:87 brd ff:ff:ff:ff:ff:ff | 15:22 |
| tlegentil | inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0 | 15:22 |
| tlegentil | valid_lft forever preferred_lft forever | 15:22 |
| tlegentil | 8: brbm: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 | 15:22 |
| tlegentil | link/ether fa:f1:9e:66:1f:4f brd ff:ff:ff:ff:ff:ff | 15:22 |
| tlegentil | 9: br-tun: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 | 15:22 |
| tlegentil | " | 15:22 |
| tlegentil | oups sorry for the flood | 15:22 |
| cid | tlegentil: Good afternoon. | 15:27 |
| cid | You could use this for sharing big chunks like that ^^ : https://paste.opendev.org/ :)) | 15:29 |
| cid | I have a level of familiarity with setting up devstack at this point. But it's a bit unclear to me what the problem might be in your case. | 15:32 |
| cid | FIY: I'm a bit of a noob too | 15:32 |
| JayF | Yeah tlegentil a couple of things: first of all, if you've had a failed install, it can be really tough to get that machine working. I'd suggest starting from a fresh ubuntu install if possible. Then, if you still have issues, paste the devstack log somewhere and we can look at help | 15:33 |
| cid | Maybe, start by ./unstack[ing] and ./stack[ing] once again. And, probably just remove the current local.conf and get a fresh one from the documentation. | 15:33 |
| cardoe | So a bit of a follow up to the convo about redfish-https sending an ISO. I realize now that's really poking the BMC and telling the boot interfaces to download the ISO. Not mounting the ISO through the BMC like virtual media. So in this case this would work with my network constraints. That ISO is pre-built and the same for all servers or is it being dynamically generated? | 15:43 |
| cardoe | It wasn't clear to me from the docs. | 15:43 |
| TheJulia | cardoe: dynamically generated by default | 15:46 |
| tlegentil | Thanks @cid for the paste site and the unstack / stack that I will try now | 15:49 |
| cardoe | Thanks TheJulia | 15:49 |
| TheJulia | tlegentil: our CI uses a very similar configuration, I guess a question might be if your trying to use a specific branch | 15:54 |
| tlegentil | TheJulia : I'm on the latest, 'git clone etc' | 15:57 |
| rpittau | JayF, TheJulia, the pysnmp-lextudio patch revert has merged, snmp related things should be functional again | 16:02 |
| rpittau | good night! o/ | 16:06 |
| JayF | o/ | 16:09 |
| TheJulia | tlegentil: since we don't often run devstack directly, that does seem a little weird. Typically when we have such issues in CI, there is some sort of preceeding error. your error is originating from Neutron failing to create a network, I guess I'd look at the q-* log files to see what neutron is reporting as to why it couldn't create a network | 16:14 |
| tlegentil | ok after unstack / stack I have a different one : https://paste.opendev.org/show/b9UgnEVLAFunMWirpn49/ | 16:15 |
| tlegentil | Also my goal is only to manage baremetal provisionning | 16:15 |
| cid | tlegentil: Your best bet may be doing as JayF suggested. | 17:03 |
| cid | - start from a fresh Ubuntu install, with a fresh configuration | 17:03 |
| cid | - if you still have issues, paste the devstack log somewhere and we can look at help | 17:03 |
| TheJulia | tlegentil: bifrost might be an easier path if you don't need extra openstack parts and pieces | 17:06 |
| JayF | We do *not* reccomend or support the use of devstack in production. You 100% want a solution like bifrost rather than a devstack testenv. | 17:28 |
| Sandzwerg[m] | <samcat116> "Related to dead agents: it seems..." <- We don't have ironic-neutron-agents so I'm not sure if that is related. | 17:45 |
| TheJulia | Sandzwerg[m]: do you have networking-baremetal installed and configured in neutron? | 17:55 |
| Sandzwerg[m] | Nope | 17:55 |
| TheJulia | hmm | 17:55 |
| Sandzwerg[m] | We use arista/cisco as Top of the Rack switch and for Arista we wrote our own driver, not sure about cisco. But I assume our setup in this regard is not very common | 17:57 |
| opendevreview | cid proposed openstack/ironic master: [WIP but open to reviews] Allow disabling bios deployments https://review.opendev.org/c/openstack/ironic/+/922243 | 17:57 |
| opendevreview | cid proposed openstack/ironic master: [WIP but open to reviews] Allow disabling bios deployments https://review.opendev.org/c/openstack/ironic/+/922243 | 17:59 |
| TheJulia | Sandzwerg[m]: but do you have dead agents? | 17:59 |
| Sandzwerg[m] | Not that I'm aware of. I see conductor left overs in the ironic conductor list, nowhere else (so far) | 18:00 |
| TheJulia | okay, yeah, those are distinctly different pieces of data | 18:01 |
| TheJulia | and processes | 18:01 |
| Sandzwerg[m] | I filled a bug for this: https://bugs.launchpad.net/ironic/+bug/2069771 can add more details later or tomorrow, I have a private appointment now :) | 18:01 |
| *** awb_ is now known as awb | 18:12 | |
| opendevreview | cid proposed openstack/ironic master: Provision ARM (aarch64) fake-bare-metal-vms https://review.opendev.org/c/openstack/ironic/+/915441 | 18:57 |
| opendevreview | Julia Kreger proposed openstack/ironic-python-agent-builder master: Remove centos7 specific logic check https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/922248 | 19:18 |
| cid | o/ | 19:32 |
| opendevreview | Julia Kreger proposed openstack/ironic stable/2024.1: fix: Fix class typo for portgroup. Portgroup instead of PortGroup https://review.opendev.org/c/openstack/ironic/+/922250 | 19:37 |
| opendevreview | Julia Kreger proposed openstack/ironic stable/2023.2: fix: Fix class typo for portgroup. Portgroup instead of PortGroup https://review.opendev.org/c/openstack/ironic/+/922251 | 19:37 |
| opendevreview | Julia Kreger proposed openstack/ironic stable/2023.1: fix: Fix class typo for portgroup. Portgroup instead of PortGroup https://review.opendev.org/c/openstack/ironic/+/922252 | 19:38 |
| opendevreview | Verification of a change to openstack/ironic master failed: Update version change log with special treatment of .json removal https://review.opendev.org/c/openstack/ironic/+/921966 | 20:22 |
| * cid Oops... did a few rechecks after seeing a passing snmp job 😬 | 21:05 | |
| JayF | I think it should be fixed now, if it's not you might want to dig into what's happening. Just make sure that the new constraints file was used -- you should have, in the logs somewhere, a list of the exact packages installed in the venv with ironic and virtualpdu | 21:47 |
| tlegentil | Thanks @cid and @TheJulia for the bifrost recommendation, I will have a look. | 22:26 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Remove ibmc hardware type https://review.opendev.org/c/openstack/ironic/+/922259 | 23:10 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Remove ibmc hardware type https://review.opendev.org/c/openstack/ironic/+/922259 | 23:26 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Remove deprecated xclarity hardware type https://review.opendev.org/c/openstack/ironic/+/922260 | 23:26 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Remove ibmc hardware type https://review.opendev.org/c/openstack/ironic/+/922259 | 23:28 |
| opendevreview | Merged openstack/ironic master: Update version change log with special treatment of .json removal https://review.opendev.org/c/openstack/ironic/+/921966 | 23:29 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Remove deprecated xclarity hardware type https://review.opendev.org/c/openstack/ironic/+/922260 | 23:34 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Remove ibmc hardware type https://review.opendev.org/c/openstack/ironic/+/922259 | 23:34 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!