| janders | good morning Ironic o/ | 00:05 |
|---|---|---|
| TheJulia | o/ | 00:30 |
| opendevreview | Merged openstack/metalsmith stable/yoga: Add linters job https://review.opendev.org/c/openstack/metalsmith/+/849523 | 03:10 |
| opendevreview | Merged openstack/metalsmith stable/wallaby: Add linters job https://review.opendev.org/c/openstack/metalsmith/+/849583 | 03:19 |
| opendevreview | Merged openstack/metalsmith stable/xena: Add linters job https://review.opendev.org/c/openstack/metalsmith/+/849524 | 03:19 |
| opendevreview | Aija Jauntēva proposed openstack/ironic master: Fix markup typo in Redfish driver docs https://review.opendev.org/c/openstack/ironic/+/849617 | 06:56 |
| kaloyan | TheJulia: Yes, I was refering to ipxe configuration sym-links having names in /httpboot. Maybe I don't understand well enough how iPXE works but it was a bit misleading having a message saying: "Attempting to boot from MAC X", where X refers to a NIC that's not PXE-capable (there's no DHCP in that VLAN). | 07:09 |
| kaloyan | TheJulia: It turned out it was caused by the iSCSI SAN having specific requirements when using L3 networks. Placing its portals in the same VLAN and subnet resolves the problem. | 07:13 |
| kaloyan | TheJulia: It "resolves" the problem, because iPXE can now talk directly via the configure interface, regardless of the message output | 07:16 |
| kaloyan | TheJulia: My point is that having iPXE configuration in /httpboot for ironic ports that aren't marked as PXE-capable is confusing. I get the point that it's best to let iPXE choose, but wouldn't it be better to let it chose from ports that are truly PXE capable? | 07:18 |
| rpittau | good morning ironic! o/ | 07:24 |
| opendevreview | Merged openstack/ironic master: Update known issue for iDRAC Swift firmware update https://review.opendev.org/c/openstack/ironic/+/849495 | 08:23 |
| opendevreview | Aija Jauntēva proposed openstack/ironic bugfix/20.2: Update known issue for iDRAC Swift firmware update https://review.opendev.org/c/openstack/ironic/+/849633 | 08:27 |
| opendevreview | Aija Jauntēva proposed openstack/ironic stable/yoga: Update known issue for iDRAC Swift firmware update https://review.opendev.org/c/openstack/ironic/+/849634 | 08:28 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: CI: use openstacksdk from source if it's cloned https://review.opendev.org/c/openstack/bifrost/+/849563 | 08:36 |
| opendevreview | Mark Goddard proposed openstack/ironic stable/wallaby: Improve dependency resolution in driver-requirements.txt https://review.opendev.org/c/openstack/ironic/+/849491 | 08:41 |
| timeu | Hi Ironic, I am trying to use baremetal port groups for setting up LACP on a baremetal node. If understand the documentation correctly if I don't set a MAC address on the port group, it should use the one from the neutron port (VIF). However for some reason the cloud-init network-data contains a random MAC address and not the VIF one which cause the node to not acquire a | 08:49 |
| timeu | DHCP lease. I guess the workaround is to use configdrive or explicitly set a MAC address but I was wondering if I do something wrong | 08:49 |
| * dtantsur has no exposure to LACP. mgoddard ^^^ | 08:56 | |
| dtantsur | arne_wiebalck: polling for your interest. I may have a case for "active steps" downstream, specifically for firmware upgrades. Is it something that still interests you? | 08:56 |
| mgoddard | timeu: I would expect the metadata to match the neutron port's MAC. Although I set the port group's MAC to one of the member ports (the one that is 'force up' if PXE booting over the bond) | 08:57 |
| timeu | yeah that was also my expectation reading the docs but it does not seem to do it. maybe it's not supported in the Train version that I try it on. I will try to set the MAC address to the first interface and see if it sets it. One other question tough the baremetal should be able to access the metadata service if it can reach the OS controllers ? | 09:01 |
| timeu | for some reason we also can't get the LACP bond working with NetworkManager. It always starts the bond and after a while it removes the slave and starts all over again. It works if I uninstall NetworkManager and install network-scripts in the image | 09:04 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: Wait for cleaning to be over when enrolling https://review.opendev.org/c/openstack/bifrost/+/849652 | 09:15 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: CI: use openstacksdk from source if it's cloned https://review.opendev.org/c/openstack/bifrost/+/849563 | 09:15 |
| mgoddard | timeu: baremetal should be able to reach metadata with OVS, assuming there is a non-bonded network up | 09:52 |
| mgoddard | timeu: if you just have a bond, use configdrive | 09:52 |
| mgoddard | timeu: if you are using OVN, you can't get metadata currently | 09:53 |
| timeu | mgoddard thanks for the info. Whhat's the technical reason why it doesn't work with bonded network ? | 09:54 |
| mgoddard | timeu: chicken and egg - you need the metadata to be able to bring up the bond | 09:54 |
| timeu | ah yeah of course | 09:54 |
| timeu | makes sense | 09:54 |
| timeu | I think in our case it worn't work anyways because we are using Cisco SDN + their distributed metadata agents | 09:55 |
| mgoddard | timeu: I have seen the bond cycling issue before. IIRC I had the node on the wrong VLAN, so no connectivity | 09:55 |
| timeu | config drives works also setting the MAC address on the port group fixed the DHCP issue | 09:55 |
| timeu | the last remaining issue is why the bond does't stay up with NetwokManager | 09:55 |
| timeu | I tried it with setting NM_CONTROLLED=no on the intefaces. that fixes the issue as soon as I remove it NetworkManager starts to remove the slaves for some reason | 09:56 |
| mgoddard | see my last comment | 09:56 |
| mgoddard | I have it working with NM | 09:56 |
| timeu | hmm we don't have vlans configured only access | 09:57 |
| timeu | we also have other nodes that work fine with NM and LACP | 09:57 |
| timeu | not sure why these nodes have issues. That's with RHEL 8.4 and MLX 5 dual port nics | 09:57 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: Wait for cleaning to be over when enrolling https://review.opendev.org/c/openstack/bifrost/+/849652 | 09:58 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: CI: use openstacksdk from source if it's cloned https://review.opendev.org/c/openstack/bifrost/+/849563 | 09:58 |
| timeu | interestingly once the bond is up and running once without NetworkManager. Starting it again with NetworkManager seems to work and stable | 10:08 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic-lib master: json_rpc.client: log the URL and exceptions https://review.opendev.org/c/openstack/ironic-lib/+/849661 | 10:38 |
| opendevreview | waleed mousa proposed openstack/ironic-python-agent master: Update and configure Nvidia nics fw by ironic-python-agent https://review.opendev.org/c/openstack/ironic-python-agent/+/566544 | 10:45 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: CI: use openstacksdk from source if it's cloned https://review.opendev.org/c/openstack/bifrost/+/849563 | 10:53 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: [WIP] Wait for cleaning to be over when enrolling https://review.opendev.org/c/openstack/bifrost/+/849652 | 10:53 |
| opendevreview | Merged openstack/ironic-inspector master: Removed blacklisted to blocked https://review.opendev.org/c/openstack/ironic-inspector/+/845919 | 11:11 |
| dtantsur | folks, could I get a 2nd +2 on https://review.opendev.org/c/openstack/bifrost/+/849074 and https://review.opendev.org/c/openstack/bifrost/+/848741 please? They're quite important. | 11:37 |
| dtantsur | also any reviews on https://review.opendev.org/c/openstack/bifrost/+/849563/ please | 11:37 |
| iurygregory | dtantsur, will look after the meeting =) | 11:43 |
| dtantsur | thx! | 11:43 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: Wait for cleaning to be over when enrolling https://review.opendev.org/c/openstack/bifrost/+/849652 | 11:56 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent stable/xena: Gather details about bond interfaces if present https://review.opendev.org/c/openstack/ironic-python-agent/+/848868 | 12:30 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent bugfix/8.1: Gather details about bond interfaces if present https://review.opendev.org/c/openstack/ironic-python-agent/+/848869 | 12:35 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent stable/victoria: Gather details about bond interfaces if present https://review.opendev.org/c/openstack/ironic-python-agent/+/848991 | 12:43 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent stable/wallaby: Gather details about bond interfaces if present https://review.opendev.org/c/openstack/ironic-python-agent/+/848990 | 12:50 |
| iurygregory | dtantsur, done =) | 13:00 |
| opendevreview | Merged openstack/bifrost master: Stop passing ironic_url to ansible modules https://review.opendev.org/c/openstack/bifrost/+/848741 | 13:07 |
| TheJulia | kaloyan: so I believe pxe_enabled is more "where we put PXE instructions in for Neutron", but I believe we've always created links for all interfaces which links to a central configuration file for the machine. iPXE does get to choose, but if you see it attempting multiple interfaces, then it may be that it is not taking the order of what it boots from. I guess we'd need a video recording of it network booting to really | 13:07 |
| TheJulia | understand since it seems different from the ipxe behavior I'm used to | 13:07 |
| opendevreview | Julia Kreger proposed openstack/ironic master: DNM: Anaconda CI changes https://review.opendev.org/c/openstack/ironic/+/849587 | 13:10 |
| kaloyan | TheJulia: I'm using iPXE, maybe I didn't state that clearly | 13:13 |
| iurygregory | just me or gerrit is really SLOW today? | 13:17 |
| TheJulia | well, to load ipxe, unless you've replaced the firmware in the cards, your ultimately initiating PXE first for a small network boot program | 13:17 |
| TheJulia | and then it continues booting from there | 13:18 |
| opendevreview | Julia Kreger proposed openstack/metalsmith stable/victoria: Add linters job https://review.opendev.org/c/openstack/metalsmith/+/849674 | 13:26 |
| opendevreview | Merged openstack/ironic-python-agent stable/yoga: Gather details about bond interfaces if present https://review.opendev.org/c/openstack/ironic-python-agent/+/848742 | 13:33 |
| opendevreview | Merged openstack/ironic master: Fix markup typo in Redfish driver docs https://review.opendev.org/c/openstack/ironic/+/849617 | 13:33 |
| opendevreview | Julia Kreger proposed openstack/ironic stable/victoria: CI: disable some jobs which https://review.opendev.org/c/openstack/ironic/+/849677 | 13:35 |
| opendevreview | Verification of a change to openstack/bifrost master failed: Deprecate os_ironic_node_info https://review.opendev.org/c/openstack/bifrost/+/848765 | 13:38 |
| opendevreview | Verification of a change to openstack/bifrost master failed: Reduce the usage of no_log in keystone code https://review.opendev.org/c/openstack/bifrost/+/849114 | 13:38 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Deprecate syslinux https://review.opendev.org/c/openstack/ironic/+/842229 | 13:51 |
| TheJulia | iurygregory: dtantsur do we have a list of bugfix branches which can be closed out? | 14:03 |
| iurygregory | TheJulia, yes! | 14:04 |
| TheJulia | we have way too many ones with busted CI | 14:04 |
| iurygregory | for ironic the ones we care are 20.2 / 19.0 / 18.1 | 14:05 |
| iurygregory | the others I will ask to be removed | 14:05 |
| TheJulia | iurygregory: can you trigger abandons on patches on those branches which have not merged? | 14:05 |
| iurygregory | will do after the downstream meeting | 14:06 |
| TheJulia | k | 14:06 |
| opendevreview | Verification of a change to openstack/ironic master failed: Allocation candidates prefer matching name https://review.opendev.org/c/openstack/ironic/+/845856 | 14:08 |
| TheJulia | I've abandoned mine that are standing out to me | 14:09 |
| TheJulia | CI must hate me | 14:12 |
| TheJulia | looks stuff is now failing all over the place where stuff worked as early as a few hours ago | 14:12 |
| TheJulia | iurygregory: just so we keep in mind, https://review.opendev.org/c/openstack/ironic-python-agent/+/848969 is a major version on next release | 14:18 |
| opendevreview | Verification of a change to openstack/ironic stable/wallaby failed: Improve dependency resolution in driver-requirements.txt https://review.opendev.org/c/openstack/ironic/+/849491 | 14:20 |
| iurygregory | TheJulia, tks for the heads-up | 14:26 |
| TheJulia | i guess both get major versions on next release | 14:35 |
| kaloyan | TheJulia: the part with loading iPXE as a NBP works fine. The trouble arises from the fact that first script that iPXE loads is the boot.ipxe which loop through all NICs on the machine, trying to chainboot from them. This is the point where it failed | 14:42 |
| TheJulia | kaloyan: uefi or bios mode? | 14:44 |
| -opendevstatus- NOTICE: Due to an incident in our hosting provider, the tarballs.opendev.org site (and possibly other sites served from static.opendev.org) is offline while we attempt recovery | 14:53 | |
| opendevreview | Verification of a change to openstack/ironic master failed: Make anaconda non-image deploys sane https://review.opendev.org/c/openstack/ironic/+/834709 | 15:03 |
| dtantsur | o/ | 15:08 |
| kaloyan | TheJulia: UEFI | 15:25 |
| kaloyan | and BTW, https://tarballs.opendev.org/openstack/ironic-python-agent/dib/files/ipa-centos9-master.kernel returns a 403 as well as the directory, any idea why? | 15:26 |
| rpittau | long weekend here, bye everyone, see you on monday! o/ | 15:37 |
| opendevreview | Verification of a change to openstack/ironic-python-agent master failed: Drop support for instance netboot https://review.opendev.org/c/openstack/ironic-python-agent/+/848969 | 16:19 |
| iurygregory | maybe we should hold rechecks RETRY_LIMIT shows we couldn't reach some mirrors | 16:25 |
| JayF | Yeah, note the status posted above | 16:49 |
| JayF | if tarballs.opendev is down | 16:49 |
| JayF | nothing is gonna pass I think | 16:49 |
| JayF | IPA jobs will want to upload there in post (so we shouldn't merge even if we could) | 16:50 |
| JayF | and Ironic jobs will get a prebuilt ramdisk from there | 16:50 |
| JayF | kaloyan: I guess you missed the notice? Notice(opendevstatus): NOTICE: Due to an incident in our hosting provider, the tarballs.opendev.org site (and possibly other sites served from static.opendev.org) is offline while we attempt recovery | 16:51 |
| JayF | kaloyan: sorry, hopefully it gets back online soon | 16:51 |
| TheJulia | kaloyan: hmm, I'm guessing with uefi it is enumerating instead of just starting with the interface it has first | 17:09 |
| * TheJulia tries to remember the code path it takes and suspects that is the case because it gets all of that from the simple networking stack in uefi | 17:09 | |
| TheJulia | ugh, until tarballs is back, our CI is toasty | 17:10 |
| iurygregory | yeah | 18:07 |
| kaloyan | TheJulia: here's my boot.ipxe https://pastebin.com/JJex3ewv I didn't use a custom template | 19:54 |
| opendevreview | Julia Kreger proposed openstack/ironic-tempest-plugin master: Undo maintenace state on protected node tests https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/849752 | 20:10 |
| TheJulia | kaloyan: hmm, Lucas's note there makes me wonder... | 20:24 |
| stevebaker[m] | good morning! | 20:47 |
| TheJulia | good morning | 20:47 |
| * TheJulia raises an eyebrow at https://8c10dee59207be00b60c-b45f7fdceccc06baa60c38e38ca87fcf.ssl.cf1.rackcdn.com/840006/3/check/ironic-tox-unit-with-driver-libs/7d40c4d/testr_results.html | 20:50 | |
| JayF | > AssertionError: Expected 'sleep' to be called once. Called 8 times. | 20:53 |
| JayF | that is ... strange | 20:54 |
| TheJulia | yeah | 20:55 |
| TheJulia | I haven't looked at the code, but I could guess maybe a rare race condition in one of the tests | 20:55 |
| TheJulia | btw, I rechecked a bunch of stuff impacted by the great tarballs.o.o outage | 20:56 |
| opendevreview | Verification of a change to openstack/ironic master failed: Make anaconda non-image deploys sane https://review.opendev.org/c/openstack/ironic/+/834709 | 21:05 |
| opendevreview | Verification of a change to openstack/ironic stable/wallaby failed: Improve dependency resolution in driver-requirements.txt https://review.opendev.org/c/openstack/ironic/+/849491 | 21:06 |
| opendevreview | Verification of a change to openstack/ironic master failed: Allocation candidates prefer matching name https://review.opendev.org/c/openstack/ironic/+/845856 | 21:07 |
| kaloyan | TheJulia: I'm kinda new to IRC so I didn't get what Lukas's note was :/ | 21:24 |
| TheJulia | oh, https://pastebin.com/JJex3ewv which is what you shared | 21:26 |
| TheJulia | kaloyan: ^^ | 21:27 |
| TheJulia | I'd have to look at the UEFI code, but I believe we're just looking at a different pattern of behavior | 21:27 |
| TheJulia | with UEFI vs non-uefi | 21:27 |
| opendevreview | Merged openstack/ironic bugfix/18.1: CI: Removing ironic job queue https://review.opendev.org/c/openstack/ironic/+/843520 | 23:22 |
| janders | good morning Ironic o/ | 23:37 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!