| JayF | cardoe: maybe we are thinking about the json schema stuff backwards. If the new schema ideas are all about making it easy to generate stuff, one of the things we can generate is an entire directory full of micro version schemas for each endpoint based on the code | 00:04 |
|---|---|---|
| JayF | I might point Claude at that with carte blanche to write something overnight 😂 | 00:05 |
| JayF | I do appreciate you bringing it up because we need to get out of the half migrated state anyway. Maybe I'll try to get some resources for that next cycle (by resources, I probably mean volunteering myself) | 00:09 |
| opendevreview | Merged openstack/bifrost master: ci: undo non-voting for centos10 upgrade https://review.opendev.org/c/openstack/bifrost/+/971258 | 00:13 |
| opendevreview | Merged openstack/bifrost master: Upload to and use artifact from OCI registry https://review.opendev.org/c/openstack/bifrost/+/968417 | 00:14 |
| cardoe | JayF: that’s where I was going. And then was looking to have a test to walk through all schema versions to make sure the code base supported it still. So when you change a model yes it’s not DRY but you just copy the schema to the next version and add the field there. | 00:27 |
| cardoe | Well the test part was just a future idea. | 00:29 |
| opendevreview | Allain Legacy proposed openstack/ironic master: Refactor switchport config parsing into SwitchPortConfig dataclass https://review.opendev.org/c/openstack/ironic/+/979077 | 00:36 |
| cardoe | alegacy_: while you’re in there you have network_type as a free form string but it’s an enum https://opendev.org/openstack/ironic/src/commit/9d43e45c82a1823117b05e1d834a0358fc62587d/ironic/drivers/modules/network/common.py#L39 there | 00:42 |
| cardoe | I caught some places that we didn’t cover all of them. Using the type let’s tooling ensure we catch it all | 00:45 |
| opendevreview | Allain Legacy proposed openstack/bifrost master: Add support for standalone ironic networking https://review.opendev.org/c/openstack/bifrost/+/962394 | 00:55 |
| alegacy_ | cardoe: thanks... I'll take a look at that in the morning! | 00:56 |
| rpittau | good morning ironic! happy friday ! o/ | 07:50 |
| rpittau | FYI there's a massive failure due to new version of tox -> https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/979118 | 09:21 |
| opendevreview | Verification of a change to openstack/ironic master failed: Increase networking-generic-switch version https://review.opendev.org/c/openstack/ironic/+/979079 | 09:33 |
| opendevreview | Verification of a change to openstack/ironic-python-agent-builder master failed: Move ironic-python-agent-podman element https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/979037 | 10:35 |
| opendevreview | Verification of a change to openstack/ironic master failed: Properly register [agent_container] config https://review.opendev.org/c/openstack/ironic/+/979034 | 10:49 |
| opendevreview | Verification of a change to openstack/ironic master failed: Abort now always aborts https://review.opendev.org/c/openstack/ironic/+/978953 | 10:58 |
| opendevreview | Verification of a change to openstack/bifrost stable/2025.1 failed: Remove unused sphinxcontrib-blockdiag https://review.opendev.org/c/openstack/bifrost/+/977085 | 11:09 |
| stephenfin | cardoe: fwiw, the reason for going with dicts was because I started with nova first and they already schemas for request validation (albeit, very inconsistently). I then later found out that tempest was also doing it for response validation (though again, inconsistently) | 12:04 |
| stephenfin | cardoe: Adapting the solution to map to something the project was already using made it ~a much easier lift~ viable. I'm well aware of pydantic (and marshmallow, and voluptuous, and attrs...) but doing response validation differently to request validation (or worse, rewriting all the existing request validation tooling to use it) was not going to fly | 12:05 |
| stephenfin | and as JayF rightly points out, now that nova, keystone and manila are using that model, it would be highly beneficial to have other projects also work this way, even if it's not the best technical approach | 12:08 |
| stephenfin | (if I had my way, I'd get rid of microversions and migrate all APIs over to FastAPI, but I think I'm already doing *more* enough to improve the state of typing in OpenStack :) https://review.opendev.org/q/owner:stephenfin@redhat.com+topic:typing) | 12:08 |
| alegacy_ | cardoe: looks like 2 of our commits crossed in the night... i had add similar constants here: https://opendev.org/openstack/ironic/src/commit/9d43e45c82a1823117b05e1d834a0358fc62587d/ironic/common/network.py#L23 | 12:44 |
| cardoe | stephenfin: yep not saying what you have is wrong. It’s aiming for consistency. Plus OpenStack would never go to FastAPI. | 13:06 |
| dtantsur | stephenfin: I'm sorry, I tried really hard to fight microversions many years ago when they got introduced :) | 13:08 |
| JayF | stephenfin: fwiw, I have an mlh fellow working on python ironic client type hints. I anticipate we'll try to finish that work across other ironic repos even if he doesn't | 13:31 |
| TheJulia | good morning | 14:05 |
| opendevreview | Verification of a change to openstack/ironic master failed: Abort now always aborts https://review.opendev.org/c/openstack/ironic/+/978953 | 14:36 |
| cardoe | OSError: No socket could be created -- (('0.0.0.0', 6385): [Errno 98] Address already in use) Traceback (most recent call last): File "/var/lib/openstack/lib/python3.12/site-packages/oslo_service/backend/_threading/service.py", line 82, in run self.service_instance.start() File "/var/lib/openstack/lib/python3.12/site-packages/ironic/common/wsgi_service.py", line 108, in start self.server.prepare() File | 14:40 |
| cardoe | "/var/lib/openstack/lib/python3.12/site-packages/cheroot/server.py", line 1799, in prepare raise socket.error(msg) OSError: No socket could be created -- (('0.0.0.0', 6385): [Errno 98] Address already in use) Traceback (most recent call last): File "/var/lib/openstack/lib/python3.12/site-packages/oslo_service/backend/_threading/service.py", line 82, in run self.service_instance.start() File | 14:40 |
| cardoe | "/var/lib/openstack/lib/python3.12/site-packages/ironic/common/wsgi_service.py", line 108, in start self.server.prepare() File "/var/lib/openstack/lib/python3.12/site-packages/cheroot/server.py", line 1799, in prepare raise socket.error(msg) | 14:40 |
| cardoe | oh man I didn't expect that to paste like that. I expected it to auto pastebin me. | 14:41 |
| rpittau | :D | 14:41 |
| TheJulia | UGH | 14:42 |
| * TheJulia needs more coffee | 14:42 | |
| cardoe | But that's the issue we've been seeing with using ironic-api as the start up script in the Dockerfile. | 14:46 |
| rpittau | which Dockerfile? :) | 14:47 |
| opendevreview | Verification of a change to openstack/ironic master failed: Properly register [agent_container] config https://review.opendev.org/c/openstack/ironic/+/979034 | 14:50 |
| opendevreview | Verification of a change to openstack/ironic master failed: Increase networking-generic-switch version https://review.opendev.org/c/openstack/ironic/+/979079 | 14:51 |
| rpittau | cardoe: you either have another process blocking the same port, or a previous process that didn't clean up properly its socket; if you can exec into the running or failing container, you can check what's taking that port | 14:52 |
| rpittau | cardoe: I would also check how you're starting the container, if your entrypoint is starting the service but you also have a sciprt or something executing ironic-api, then you have a conflict | 14:53 |
| cardoe | It's just cmd: ironic-api and args are --config-file /etc/ironic/ironic.conf | 14:53 |
| rpittau | are you running the container using --net=host ? you have other services bound to port 6385 ? | 14:55 |
| cardoe | This is in OpenStack Helm. But trying to make their container slim as heck. | 14:55 |
| cardoe | rpittau: I would hope that OpenStack Helm isn't net=host for the api | 14:57 |
| TheJulia | Sure has me wondering if it is based upon the error, the only other possibility is it is somehow trying to launch two socket binds at once without something managing the sockets, but htat seems bonkers and like it would have broken sooner | 14:58 |
| rpittau | cardoe: is ti possible to check the config file ? | 14:59 |
| cardoe | yeah. I'm looking at it. I forget what the key is on the pod to say if its hostNetwork | 15:01 |
| rpittau | in the spec? should be jsut hostNetwork | 15:03 |
| rpittau | I actually expect that to be true considering the networking traffic generated by ironic, unless it's using some advanced configuration | 15:06 |
| cardoe | it looks like only the ironic-conductor pod does hostNetwork | 15:24 |
| * TheJulia wonders if anyone actually uses apstra | 15:24 | |
| cardoe | https://github.com/rackerlabs/understack/pull/1759/changes is apparently what makes that not happen. | 15:25 |
| * TheJulia guesses it is more just less public posts giving claude grounding contet | 15:25 | |
| cardoe | api.api_workers=1 fwiw | 15:25 |
| TheJulia | That makes sense if your doing direct service launch and not using uwsgi | 15:26 |
| TheJulia | Since the option is for uwsgi where it handles the host socket | 15:27 |
| cardoe | Well we are using uwsgi but the stock OpenStack Helm is using ironic-api directly. | 15:27 |
| TheJulia | I bet oslo.service changes have broken uwsgi runtimes in some cases | 15:31 |
| TheJulia | I think we only run with one worker in CI, we should try to up that in a WIP change to see how massively that detonates | 15:32 |
| opendevreview | Julia Kreger proposed openstack/networking-generic-switch master: WIP: Crazy idea: NDFC support in NGS? https://review.opendev.org/c/openstack/networking-generic-switch/+/968484 | 16:00 |
| opendevreview | Julia Kreger proposed openstack/networking-generic-switch master: WIP: apstra support https://review.opendev.org/c/openstack/networking-generic-switch/+/979298 | 16:00 |
| cardoe | TheJulia: I'm saying if we kick it off with uwsgi we're good. If we kick it off with the entry point installed "ironic-api" with api_workers > 1 then it throws that error. | 16:00 |
| TheJulia | yeah, that is my take as well | 16:01 |
| cardoe | looks like something changed in tox on the Zuul runners so everything is busted. | 16:20 |
| TheJulia | tox breakage | 16:28 |
| * TheJulia doesn't have context of what needs to happen at the moment, busy dealing with the fallout of a customer going "no, i must have multicast" | 16:29 | |
| * TheJulia expects Leeloo Dallas to appears and say "multicast" | 16:29 | |
| opendevreview | Léonard Suslian proposed openstack/ironic master: Use ServiceRoot.Vendor for detect_vendor() instead of System.Manufacturer https://review.opendev.org/c/openstack/ironic/+/978439 | 16:57 |
| clif | JayF: Do you know of any constraints on ports that make up a portgroup? One thing that stands out is that if the physical_network of a portgroup changes, then that change cascades to its constituent ports. If I'm creating a new portgroup do all physical_networks of constituent ports need to match? That doesn't seem to be enforced by the API controller. | 17:12 |
| opendevreview | Merged openstack/ironic-python-agent-builder master: Move ironic-python-agent-podman element https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/979037 | 17:15 |
| JayF | My initial reaction is to say that they all should have to already be on the same physical Network. But I'm honestly not 100% sure and cardoe or TheJulia might have a more certain answer | 17:54 |
| JayF | I think your observation about making things match on create is a good identification of a bug, perhaps | 18:00 |
| JayF | But I'd file it and leave it for later | 18:01 |
| cardoe | clif: yeah I thought we set that as a "rule" when we were discussing the spec. | 18:02 |
| cardoe | And that's how the behavior was suppose to flow in the API. If you change the portgroup, it changes the ports. | 18:02 |
| clif | Yea it's in the spec: "All port group members must have the same physical network." https://opendev.org/openstack/ironic-specs/blame/branch/master/specs/approved/trait-based-port-scheduling.rst#L462 | 18:04 |
| clif | and yes it appears its a bug that it's not enforced at create time afaict | 18:04 |
| clif | I'll file a bug and work on a fix for it after I finish up implementating group_and_attach_port for TBN | 18:05 |
| opendevreview | Merged openstack/bifrost stable/2025.1: Remove unused sphinxcontrib-blockdiag https://review.opendev.org/c/openstack/bifrost/+/977085 | 18:13 |
| TheJulia | clif: everything has to be on the same physical network, generally same switch, but it doesn't *HAVE* to be with things like virtual port-channels. That gets funky though. We might need to do some NGS work there to better support the cases. | 18:30 |
| cardoe | TheJulia: so for us "physical_network" = virtual chassis | 18:33 |
| TheJulia | And that is a totally valid approach | 18:33 |
| cardoe | That way port-channels work. | 18:33 |
| TheJulia | Specifically, it is all about how you, the operator, want to model your physical and logical constraints together into $something | 18:33 |
| cardoe | Coming from the limited network hardware I work with, I would think NGS should let you mark two switches ask being virtual-switch=blah and then utilize that for the matching. | 18:35 |
| TheJulia | It *really* depends on the switch OS and the config | 18:37 |
| cardoe | Well it'd be up for the specific device backend in NGS to decide what to do with it. | 18:39 |
| TheJulia | I have a customer who has virtual port channels setup as static entities so i wrote precise destructions for how the config model works for them, dynamic I *fear* we might need to do a little more ML2 plugin work just to take and draw the line on config because some switches you just configure it and have to do vlan config on the base port, others it is on the virtual port channel interface, others you have to | 18:39 |
| TheJulia | individually configure | 18:39 |
| cardoe | But at least it'd be clean for the operator to tell NGS "hey these are providing port-channels" | 18:39 |
| TheJulia | exactly | 18:39 |
| TheJulia | The key, really, is get enough data into the binding calls on the ml2 driver, and let it sort it out | 18:41 |
| cardoe | Yep | 18:41 |
| cardoe | So what I'm saying is today ML2 gets the port which has binding_interfaces in it tied to ironic ports so it walks both of those independently. | 18:41 |
| cardoe | And what I'm saying is allow the operator to tell NGS these are paired switches so when it looks up one switch it can say oh the next port is for my paired switch | 18:43 |
| cardoe | And then the backend device code can have a do_individual_or_do_channel_interface() | 18:44 |
| cardoe | So it can be smart for that in NGS | 18:44 |
| cardoe | Cause right now what you're describing involves creating only 1 port in Ironic and setting the port_id to the virtual port channel interface name | 18:45 |
| cardoe | ask me how I know. :) | 18:45 |
| TheJulia | thats a single port bind, it should be sending both ports on a portgroup bind | 18:47 |
| TheJulia | if we're not, we regressed at some point | 18:48 |
| cardoe | Yeah but when you need it to touch the virtual port channel instead of the base ports | 18:48 |
| TheJulia | depends on the switch | 18:49 |
| cardoe | Yep | 18:49 |
| cardoe | I mean I've got some bats and we can go make our own Office Space ending scene if you're free. | 18:50 |
| TheJulia | Oh, that would be so very fun. | 18:50 |
| opendevreview | Julia Kreger proposed openstack/networking-generic-switch master: WIP: Multicast traffic options for cisco driver https://review.opendev.org/c/openstack/networking-generic-switch/+/979332 | 18:50 |
| TheJulia | so this makes me sad, but I have a customer which won't be doing ingress-replication so :( | 18:51 |
| TheJulia | cardoe: you likely have opinions, but that is sort of what claude hammered out on a first pass | 18:51 |
| TheJulia | I think mcast_groups specficially was the arista modeling, but I don't remember. I created stories for every switch vendor in my team's backlog | 18:51 |
| TheJulia | well, that we implemented vxlan for | 18:52 |
| TheJulia | cardoe: and for the record, the printer scene. Ending scene is totally different... :) | 18:54 |
| cardoe | oh yeah whoops | 18:54 |
| cardoe | I was looking at the changes.. I'm gonna smile and nod at the switch configs | 18:54 |
| TheJulia | lol | 18:54 |
| TheJulia | jrosser: looks like we discussed multicast back on 2/10. Its now on my todo list overall and if you have thoughts or opinions on tuning, please let me know. I believe you were arista based so likely Tuesday is when my brain will get there. | 19:04 |
| jrosser | TheJulia: yes, specifically we have a workload is mostly multicast, and we're using OISM on arista for that | 19:10 |
| jrosser | andrewbonney is doing the bulk of the work on that with ironic - which is currently a huge hack and just presented as a set of provider vlans configured up front | 19:12 |
| TheJulia | what is OISM in your context? | 19:13 |
| TheJulia | that seems... awful, but I can get how you ended up there | 19:13 |
| jrosser | Optimised Inter Subnet Multicast | 19:14 |
| TheJulia | Thanks | 19:14 |
| jrosser | its what routes multicast efficiently in the context of it being used in conjunction with vxlan | 19:15 |
| TheJulia | I guess, structurally, that makes sense (without looking into the fine details) | 19:17 |
| jrosser | OISM is rfc9625 | 19:17 |
| TheJulia | since it is higher level traffic handling, it looks like, it would really depend on what the underlying configuration needs to be to support that higher level efficiently. | 19:23 |
| TheJulia | jrosser: i guess if you can sync up with andrewbonney and float forward a little configuration sample, it might be possible to figure out a way to model or enable it to be tuned as so desired | 19:51 |
| opendevreview | Harald Jensås proposed openstack/networking-generic-switch master: AttributeError: delete_port_postcommit segment=None https://review.opendev.org/c/openstack/networking-generic-switch/+/979350 | 20:21 |
| opendevreview | Harald Jensås proposed openstack/networking-generic-switch master: Fix KeyError in vlan_has_vni for Cisco NX-OS https://review.opendev.org/c/openstack/networking-generic-switch/+/979351 | 20:21 |
| opendevreview | Harald Jensås proposed openstack/networking-generic-switch master: Enable L2VNI support for Cisco NX-OS switches https://review.opendev.org/c/openstack/networking-generic-switch/+/979352 | 20:21 |
| cardoe | TheJulia: https://review.opendev.org/c/openstack/nova/+/979354 thoughts/feels welcome | 20:51 |
| opendevreview | Merged openstack/ironic master: Abort now always aborts https://review.opendev.org/c/openstack/ironic/+/978953 | 21:59 |
| opendevreview | Merged openstack/ironic master: Increase networking-generic-switch version https://review.opendev.org/c/openstack/ironic/+/979079 | 21:59 |
| opendevreview | Merged openstack/networking-generic-switch master: Add troubleshooting and performance guide. https://review.opendev.org/c/openstack/networking-generic-switch/+/978799 | 23:13 |
| opendevreview | Merged openstack/ironic master: Properly register [agent_container] config https://review.opendev.org/c/openstack/ironic/+/979034 | 23:21 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!