Wednesday, 2025-02-12

* JayF afk for a while00:00
cardoeDang I go to Costco and there’s a flurry of activity.00:06
cardoeTheJulia: I'd love to hear your feelings on OVN.01:30
cardoeSo I'll be real honest I'm clueless about tempest. Something I need to learn about. But I'm looking at sticking someone on some test work and I'd be happy to have him work on upstream bits.01:31
opendevreviewDoug Goldstein proposed openstack/ironic master: move hooks execution into shareable module  https://review.opendev.org/c/openstack/ironic/+/94033201:39
opendevreviewDoug Goldstein proposed openstack/ironic master: allow multiple inspection interfaces to load hooks  https://review.opendev.org/c/openstack/ironic/+/94033301:39
opendevreviewDoug Goldstein proposed openstack/ironic master: allow running inspection hooks on redfish interface  https://review.opendev.org/c/openstack/ironic/+/93306601:39
TheJuliacardoe: I still need to get a membership….01:40
TheJuliaAnd as for OVN, I wrote doc ;)01:41
cardoeThey literally opened one about 6 minutes away.01:41
cardoeAre you referring to the ironic ovn doc?01:47
TheJuliacardoe: yeah03:09
opendevreviewDoug Goldstein proposed openstack/ironic master: move hooks execution into shareable module  https://review.opendev.org/c/openstack/ironic/+/94033205:20
opendevreviewDoug Goldstein proposed openstack/ironic master: allow multiple inspection interfaces to load hooks  https://review.opendev.org/c/openstack/ironic/+/94033305:20
opendevreviewDoug Goldstein proposed openstack/ironic master: allow running inspection hooks on redfish interface  https://review.opendev.org/c/openstack/ironic/+/93306605:20
cardoeSo maybe different stupid pivot networking wise... I wonder rather than an ML2 plugin if I can somehow just tell NGS to talk to a central thing.05:51
cardoeCause at the end of the day we're doing much of the same. The difference is we've got piles of config templates that are merged together with jinja and nornir data. Nornir uses NAPALM to provide that "all devices look the same" like NGS is doing and then Netmiko is ultimately used.05:53
cardoeOr maybe NGS could use NAPALM rather than trying to template out commands itself?05:53
cardoeEffectively I have to prepare a full config and my diff has to match my intent.05:54
rpittaugood morning ironic! o/08:42
rpittauJayF: I was looking at the grenade job yesterday and my professional technical analysis is that is a mess :D08:45
rpittauit's broken since September/October 202408:45
rpittaumy proposal is to disable it to make the ironic-lib patches pass, then re-enable it and focus on fixing it08:45
rpittauat least we'll have ironic-lib ready to be deprecated for epoxy, which is a great thing08:45
opendevreviewRiccardo Pittau proposed openstack/ironic-inspector master: Temp disable grenade job to let ironic-lib deprecation patches pass  https://review.opendev.org/c/openstack/ironic-inspector/+/94134109:09
opendevreviewAndrew Bonney proposed openstack/networking-generic-switch master: Add support for Neutron routed segments  https://review.opendev.org/c/openstack/networking-generic-switch/+/93921109:10
opendevreviewAndrew Bonney proposed openstack/ironic master: Filter physnets when ports are pre-allocated to a segment  https://review.opendev.org/c/openstack/ironic/+/94134309:14
opendevreviewHarald Jensås proposed openstack/ironic master: devstack - Allow setting MTU for provision net  https://review.opendev.org/c/openstack/ironic/+/94134609:28
opendevreviewHarald Jensås proposed openstack/ironic master: [WIP] OVN UEFI IPv6  https://review.opendev.org/c/openstack/ironic/+/94135210:13
opendevreviewHarald Jensås proposed openstack/ironic master: [WIP] OVN UEFI IPv6  https://review.opendev.org/c/openstack/ironic/+/94135210:39
opendevreviewAndrew Bonney proposed openstack/ironic master: Filter physnets when ports are pre-allocated to a segment  https://review.opendev.org/c/openstack/ironic/+/94134310:43
opendevreviewAndrew Bonney proposed openstack/networking-generic-switch master: Add support for Neutron routed segments  https://review.opendev.org/c/openstack/networking-generic-switch/+/93921110:45
rpittauJayF, TheJulia, dtantsur, can you please review https://review.opendev.org/c/openstack/ironic-python-agent/+/941089 when you get a chance? thanks!10:45
opendevreviewHarald Jensås proposed openstack/ironic master: [WIP] OVN UEFI IPv6  https://review.opendev.org/c/openstack/ironic/+/94135211:55
opendevreviewMerged openstack/ironic-python-agent bugfix/10.1: Configure ipa bugfix 10.1  https://review.opendev.org/c/openstack/ironic-python-agent/+/94108913:53
TheJuliagood morning14:05
TheJuliacardoe: perhaps, although something has to fully track and manage the consolidated config and be able to look it up and assert it back out if it is doing the whole config, and that also is increased friction to operators who want the old "I wnat to login and manage my switch" attitude14:07
opendevreviewJulia Kreger proposed openstack/ironic-python-agent master: follow-up: update release note for bootable container work  https://review.opendev.org/c/openstack/ironic-python-agent/+/94113914:39
JayFJust making a note here: we need to remove the explicit code spell job from IPA now that it's rolled into pre-commit14:52
JayFI'll do it when I'm actually at a PC instead of mobile if nobody else has by then14:52
TheJuliaRegarding https://review.opendev.org/c/openstack/ironic/+/941153 ... We've long explicitly said ipmitool is a requirement. Are we okay with making that optional?14:59
dtantsurTheJulia: given both ipmitool maintenance issues and the reality that the world is moving towards Redfish, I'm all for it15:00
dtantsurwe're not deprecating it anyway, are we?15:00
rpittauI'm 100% in favor of making ipmitool optional15:01
TheJuliano, just feels like maybe we should be doing something more15:01
TheJuliadunno15:01
TheJuliaI suspect our docs might say it is required still15:01
JayFTheJulia: if you'll put that feedback on the change, we can make sure the docs get updated15:02
JayFBy no means is that a suggestion. The driver should be removed, but more an observation that operators may not want our default installed config to require them to install ipmitool15:02
opendevreviewcid proposed openstack/ironic-python-agent master: Trivial:Remove codespell job (moved to pre-commit)  https://review.opendev.org/c/openstack/ironic-python-agent/+/94145615:03
TheJuliadone15:03
TheJuliacomment added15:06
shermanmbtw, I'd found this ngs issue/fix with l2-only ports, but wasn't sure if I was missing something, because it used to work? https://review.opendev.org/c/openstack/networking-generic-switch/+/94117215:23
TheJuliaoh wow15:34
cardoeTheJulia: yeah absolutely. You've got folks that are just like "I wanna just SSH into my switch and sling my commands!!!1111one" and then you've got the Net Automation / Infra as Code / GitOps / (pike your other words) that want to apply states.15:37
cardoeI'm in the later camp. I wanna define my ideal system state and have the tools work to achieve that state even while I sleep.15:37
TheJuliaYeah, anything to make things better15:38
TheJuliayour talking to someone who had a controller failover config which was not synced over15:38
TheJulia.... Needless to say I was not happy that morning15:38
TheJuliaIt didn't take too long to fix once I realized what was going on, but... yeah.15:39
TheJuliabecause someone slung commands and didn't "wri mem"15:39
TheJulia(oh, and even better because that controller failure was caused by a lightning strike about 45' from our grounding triangle next to our generator pads)15:40
TheJulia(the only thing impacted was the switch controller too... Everything else kept on humming.)15:42
TheJulia((and yes, the tree promptly died visibly over the following few days due to the heat because we initially thought the building had been struck and had to do a full roof inspection when the sun came up))15:42
cardoeI still remember my first software engineering job where we wrote the credit card processing software for some large point of sale places. I can remember starting and someone telling me "Trial by fire... we have to deploy a hot fix to every Papa Johns location" and at this point I had already been a Gentoo dev for most of college so I figured "okay. they'll have some sweet automation."15:43
TheJulialol15:44
cardoeNah... You gotta ATDT your way into every store location and upload that (what was that? zbinary or something?) to them.15:44
TheJuliaZ-Transfer?15:44
TheJuliaahh, the days of dial tone15:45
cardoeYeah it's been too many years.15:45
cardoeI think one of the other people on my team said something like "Hopefully he picks it up quick and we can get out of here before midnight." to another co-worker.15:45
cardoeokay back to the other thing that shermanm said that triggered me... l2-only ports.15:48
shermanmI feel that. I've got an undergrad class of 200 students learning "what is a router" right now,  and trying to use unrouted,   L2 only neutron networks so they can run their own routers inside instances15:52
shermanmmany headaches avoiding neutron's defaults15:52
cardoeCan you expand on the commit message or point me at what CONNECTIVITY_L2 does?15:53
shermanmsure, this is par of  how ML2 plugins advertise to neutron what kind of port they will handle.15:54
shermanmif CONNECTIVITY_L2 isn't set (==`'l2'`) , neutron will only use the the plugin for ports with an ip address15:54
shermanmso NGS just gets skipped entirely15:55
cardoeokay that's kinda what I was thinking but didn't know.15:55
cardoehttps://review.opendev.org/c/openstack/networking-generic-switch/+/939211 is Andrew's change...15:55
cardoeMy concern here is for L2_ADJACENCY. Where I've got segments but they're all L2.15:56
cardoeAnd maybe my approach is stupid.15:56
cardoeBut in my current dev environment I'm back to network_type=vxlan and I've got a subnet assigned to that network but then I've got segments that are network_type=vlan that have a physical_network=$switchname15:57
cardoeI know Andrew's change follows what Neutron is doing today but I feel like a port shouldn't figure out where it's connected to based on its subnet but instead the port needs the segment reference.15:58
shermanmfrom what I can tell, my change should? have been included in this one15:58
shermanmhttps://opendev.org/openstack/networking-generic-switch/commit/981923ac9926a6daeeb0a18a858875c9fb1e3fc715:58
shermanmbut it's possible that the assumptions that `api.MechanismDriver` make have changed since then to break it?15:58
shermanmI don't think it would affect any behavior other than "should NGS be skipped if port does not have ip address"15:58
cardoeI know my usage of VXLAN doesn't correlate to what NGS does today but I'm hoping to create a model for L2VNI and make that happen in Ironic and Neutron and NGS in the future.15:59
cardoeI also want to create a model that would be useful to others.15:59
jrosseris to worth getting some more info down somewhere about what andrewbonney and I are trying (currently failing - hence patches) to do with ironic and ngs?16:00
jrosserfrom lurking for a while it seems there are multiple different struggles currently all pulling in slight different directions16:01
shermanmcardoe: we've definitely got a use-case for that at least, right now our virtualized and baremetal clouds aren't networked to each other due to the vlan/vxlan mismatch16:01
* TheJulia approves of this discussion16:02
TheJuliajrosser: it would likely help the overall clarity to get andrewbonny to chime in16:06
cardoeshermanm: so my ultimate goal is to be able to plug OVN/OVS into my VXLAN fabric so that virtualized workloads can be on the same network as baremetal workloads.16:09
cardoeWhich is why I'm using network_type=vxlan16:09
andrewbonneyI may not have read back enough, but we have a use case for routed segments (as neutron currently provides for them) and found this was a relatively simple way to achieve the goal16:11
andrewbonneyFwiw we are interconnecting VMs and bare metal with this approach. Most VMs are currently using LXB based VXLAN, but have additional VLAN/routed segment ports attached to interoperate with bare metal16:14
andrewbonneyThere are some performance related reasons for doing it this way today as we can't offload the VXLAN encap as things stand16:15
opendevreviewHarald Jensås proposed openstack/ironic master: [WIP] OVN UEFI IPv6  https://review.opendev.org/c/openstack/ironic/+/94135216:23
cardoeWhat's LXB?16:23
cardoeandrewbonney: shermanm: would you two like to have a networking chat sometime? vsaienko as well?16:24
andrewbonneyLinux bridge (which we'll have to migrate to OVN from the Epoxy release)16:24
andrewbonneyHappy to16:24
vsaienkohow can I help folks?16:25
shermanmsure16:25
TheJuliaandrewbonney: I'm surprised y'all are still using lxb16:27
TheJuliabut, if it works!16:27
andrewbonneyAh, well it works and we have lots of other things to do :)16:27
TheJuliaas an fyi folks, centos ipa ramdisk is up to 501MB on the latest build16:27
TheJuliawe likely need to do more pruning16:28
TheJuliathat is the download, the initramfs is 511 mb16:29
TheJulia858 uncompressed16:30
cardoevsaienko: Just wondering if you'd be interested in having a chat about networking and ironic.16:34
cardoeWould you folks like me to send something to the ML to coordinate? (I'm trying to help Julia here because she previously scheduled these).16:34
vsaienkosure, I can participate16:35
cardoeWe can pick a date/time and a location (IRC or Zoom).16:35
jrossercardoe: we have 3 sort-of related patches https://review.opendev.org/c/openstack/neutron/+/941345 https://review.opendev.org/c/openstack/ironic/+/941343 https://review.opendev.org/c/openstack/networking-generic-switch/+/93921116:35
cardoeBut since you 3 are on right now is there a time / day of week that works better?16:35
cardoejrosser: yeah I saw you ping James. You can't have his attention cause he's currently ignoring me and he's mine first. :D16:36
cardoejrosser: but those patches are what made me bring up the convo piece here.16:36
cardoeI know vsaienko has some patches in flight for VLANs and trunks.16:36
cardoevsaienko: I actually wanted to talk to you about OSH as well (I can switch to Slack if that's better).16:36
vsaienkoyes slack is better16:37
andrewbonneyWe're UK/GMT time zone so generally anything in our working hours is fine16:38
cardoevsaienko: what's your time zone?16:40
cardoeshermanm: you're Chicago right?16:41
shermanmyep, central time16:41
vsaienkoGMT+216:41
rpittaugood night! o/16:54
opendevreviewHarald Jensås proposed openstack/ironic master: [WIP] OVN UEFI IPv6  https://review.opendev.org/c/openstack/ironic/+/94135217:37
dtantsurTheJulia: the dynamic of the initramfs size is actually pretty interesting: https://tarballs.opendev.org/openstack/ironic-python-agent/dib/files/17:43
dtantsur2023.2 added 70M, 2024.1 10M on top of that17:43
dtantsuron master we won back around 20M17:44
dtantsurDebian is at least 150M smaller per https://tarballs.opendev.org/openstack/ironic-python-agent-builder/dib/files/17:44
dtantsurI wonder if we should switch to Debian (especially since we use it anyway for ARM64 images)17:45
TheJuliayeah, we've also recently merged a patch or two to strip more out, or at least it is in review17:45
TheJuliawe're just getting super close to having CI issues due to the swelling17:45
dtantsurIt's definitely a concern17:47
vsaienkoplease review 1 line patch to enable neutron trunks extension for tinyipa-multinode https://review.opendev.org/c/openstack/ironic/+/94102317:55
JayFI'll note I have code locally for a gentoo-based IPA image that's smaller than existing but larger than tinyipa17:59
JayFI stopped trying to complete it upon realization that it wouldn't get tinyipa-small17:59
TheJulialittle will get tinyipa small ;)18:01
JayF*blink*18:03
JayFI don't get what you mean? or the joke/reference if there is one18:03
TheJuliaJust the reality that tinyipa strips a TON of stuff down18:04
TheJuliaon top of a minimalist distro to begin with18:04
JayFOH as in "not many things will get tinyipa small"18:05
TheJuliaexactly18:05
JayFI would say I need my coffee, but I don't think it helps as muich as it used to lol18:05
JayFhonestly I've thought the path to like, a real small ipa ramdisk18:05
JayFmight be looking into one of those "package up a python software into a single binary bundle" things18:05
JayFand then just having a ramdisk with only that + deps + run some kinda script to use ldd/elfinfo to only pull in the exact files needed to run those18:06
JayFjust like you used to have to do for old school chroots18:06
TheJulia.... that is remarkably similar to one of the many things done in tinyipa18:06
JayFthe real pain with our tinyipa builds is just getting them built every time18:07
TheJuliaquay, why do you hate me18:07
JayFI wonder if someone took like, a day to spike and add retries to every download in build_tinyipa.sh how many CI failures we'd dodge...18:07
JayFhmmmmmmmmmmmmmmmmmmm18:07
* JayF has already promised out his day or that would be a nerd-snipe success on himself18:07
JayFhmmm... my "remove provision from inspection" things are failing in a way which kinda implies there may have been a dependency?18:15
JayF> Details: Node 624b0518-e4b1-4978-a417-e077f3c5caf9 reached failure state inspect failed while waiting for provision_state=['inspect wait']. Error: Failed to inspect hardware. Reason: unable to start inspection: Unable to find kernel, ramdisk for building ISO, or explicit ISO for 624b0518-e4b1-4978-a417-e077f3c5caf918:15
JayFhttps://c84fb688e47a41eceab0-07197e0a54baf445efd8154b2af608b4.ssl.cf5.rackcdn.com/941302/1/check/ironic-standalone-redfish/15a1fea/testr_results.html18:16
* JayF will likely not have time to dig this himself until tomorrow18:16
JayFbut it's also passing on older branches, so I'm going to recheck :) 18:16
JayFand that job passed last time18:16
opendevreviewJulia Kreger proposed openstack/ironic master: oci: fix hang with 429 error code handling  https://review.opendev.org/c/openstack/ironic/+/94147418:48
opendevreviewHarald Jensås proposed openstack/ironic master: [WIP] OVN UEFI IPv6  https://review.opendev.org/c/openstack/ironic/+/94135219:31
opendevreviewJulia Kreger proposed openstack/ironic master: oci: fix auth config loading  https://review.opendev.org/c/openstack/ironic/+/94147919:48
TheJulia| last_error             | Failed to prepare to deploy. Exception: 503 Server Error: Service Unavailable for url: https://192.168.130.11/v2/  <-- le sigh20:20
cardoeSo stupid question time… the anaconda deploy is based on PXE. Ironic won’t make an ISO for virtual media for that will it?21:00
cardoeFollow on… ESXi has a ton of individual files on the ISO that get passed as options. Can I somehow make a dict of instance_info to include them all?21:01
JayFanything is possible, but you're getting well into "why not just make an esxi installer driver in the model of the anaconda driver"21:01
JayFI think any avoidance of going whole-hog here will cause you more pain in the longer term21:02
cardoeYeah that’s what I’m gonna do.21:04
JayFsomeone care to land this? https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/941302 21:04
JayFit's expected to unblock inspector grenade which unblocks ironic-lib-deprecation21:04
cardoeAll that data is read from21:05
cardoeInstance info21:05
JayFbeware of the length of instance_info vs the files you'll have to put in21:05
JayFjust something to keep in mind21:05
cardoeYeah if I can pass it without instance info that’s fine too. Somehow.21:06
cardoeI +2’d that change. It makes sense to me.21:06
JayFI find myself hrm'ing as to if +A on my own patch when it has 2x+2 is okay21:07
JayFI'm deciding yes, people can fuss at me if they want that to change :) 21:07
JayFWhen was the last time anyone used bifrost testenv cc: rpittau dtantsur  -- it looks broken? 21:12
JayFat least the docs are EXTREMELY out of date at a minimum https://docs.openstack.org/bifrost/latest/contributor/testenv.html21:13
JayFafaict it's not even installing ironic21:13
JayFTASK [bifrost-pip-install : Install requirements from {{ sourcedir }} using pip] <-- the {{ sourcedir }} there looks concerning21:14
JayFhmm, is there an undocumented step required of running bifrost-cli install after testenv now?21:16
JayFthat's what it looks like CI is doing21:16
cardoeHmm. Whoops.21:17
JayFyep, that seems to have done the trick.21:19
JayFsomeone who actually knows, please update that doc. otherwise I'll try to get around to it21:19
cardoeI’m also knee deep in Oslo.config and am lost. But I’m already writing a new doc.21:20
JayFI can probably help with oslo config21:20
JayFbut now I g2g take the cat to the vet21:20
cardoeOkay I would appreciate it.21:20
cardoeSub-question. Why are projects switching to uWSGI when it’s dead upstream? It’s burning me cause of its bad shutdown handling.21:24
TheJulia... That one hasn't made any sense to me beyond it is like a 5+ year effort21:43
opendevreviewcid proposed openstack/ironic-python-agent master: Ensure IPA is locked down in rescue mode  https://review.opendev.org/c/openstack/ironic-python-agent/+/94148921:47
opendevreviewcid proposed openstack/ironic-python-agent master: Ensure IPA is locked down in rescue mode  https://review.opendev.org/c/openstack/ironic-python-agent/+/94148922:06
opendevreviewMerged openstack/ironic master: oci: fix hang with 429 error code handling  https://review.opendev.org/c/openstack/ironic/+/94147422:10
JayFcardoe: in practice many projects are going gunicorn in a post eventlet world22:15
cardoeYeah that's what I need to do.22:16
opendevreviewHarald Jensås proposed openstack/ironic master: [WIP] OVN UEFI IPv6  https://review.opendev.org/c/openstack/ironic/+/94135222:24
opendevreviewMerged openstack/bifrost master: Remove workaround for installing Ussuri and older  https://review.opendev.org/c/openstack/bifrost/+/93811922:32
cardoeMaybe I need to poke vsaienko about gunicorn instead of uWSGI. There's already a few OpenStack Helm users on openstack-discuss having issues since OSH switched to uWSGI.22:34
cardoeso yeah JayF I put my questions in #openstack-oslo if ya get some time.22:37
cardoeTheJulia: so I don't wanna -1 Andrew Bonney's change for Neutron routed segments but its certainly wrong in what I believe is a valid use case for the neutron spec.22:39
cardoeI should mention, Andrew's change makes NGS behave like the rest of Ironic.22:40
opendevreviewMerged openstack/bifrost master: CI: Update previous release for upgrade jobs  https://review.opendev.org/c/openstack/bifrost/+/93811822:41
cardoeI spent some time with James Denton today going over it cause its the use case I'm tinkering with. But I'll write up something further. But effectively Neutron support L2 adjacency where you only have 1 subnet for the entire network even though the network has multiple segments. 22:41
cardoeI've got multiple VLAN segments since each baremetal box needs a VLAN on that switch to be assigned.22:41
cardoeThe existing code in NGS just gets all the segments and calls bind on the first segment. Andrew's change walks over each segment and uses the port's fixed IP to find the correct segment by looking at the subnet assigned to each segment.22:43
cardoeThat's great in an L3 case. But L2 it doesn't work. So the code falls back to just grabbing all the segments and binding the first one.22:44
cardoeIts what I've been trying to figure out and express to the neutron folks unsuccessfully and reviewing Andrew's change and the fact that he referenced the upstream behavior just made it pop.22:45
opendevreviewDoug Goldstein proposed openstack/ironic master: allow running inspection hooks on redfish interface  https://review.opendev.org/c/openstack/ironic/+/93306622:47
opendevreviewMerged openstack/ironic-tempest-plugin master: Inspector tests need not provision afterwards  https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/94130222:49
JayFcardoe: I don't fully understand that question in -oslo. Can you put context in an etherpad or bug? I'm also on mobile so may have missed something22:54
cardoeJayF: https://etherpad.opendev.org/p/oslo-config-project-prog23:19
opendevreviewHarald Jensås proposed openstack/ironic master: [WIP] OVN UEFI IPv6  https://review.opendev.org/c/openstack/ironic/+/94135223:48
JayFfwiw that bifrost issue is still active; I was unable to get a working bifrost even with the install + baremetal create baremetal-nodes.json done afterwards -- cleaning fails, and the vms don't work with virsh console so I can't really see why.23:54
JayFI will likely have to dig into this tomorrow so even just a pointer from someone more familiar would be useful23:54
JayFcardoe: bwahaha. Please file a bug against oslo.config with those details. Ideally, you could actually just log the value of prog there and validate your hypothesis too :D 23:58

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!