| JayF | holy crap every job passed on my restore ovn-ipv6 pr https://review.opendev.org/c/openstack/ironic/+/951021/1#message-74a4c0eb56b214be3de5aa7a1211ad184c0474c6 | 00:54 |
|---|---|---|
| JayF | literally every job, including -nv | 00:54 |
| iurygregory | it's early to celebrate? | 00:56 |
| JayF | it's not too early to approve the pr, lol | 00:56 |
| iurygregory | +W you got it o/ | 00:56 |
| JayF | I suspect neutron's cleanup may have fixed stuff for us overall even if it broke us in the interim | 00:56 |
| JayF | or I am wishfully thinking | 00:56 |
| iurygregory | we can always blame networking | 00:57 |
| JayF | either way taking the victory for now :D | 00:57 |
| iurygregory | :D | 00:57 |
| JayF | honestly those failures bug me too | 00:57 |
| iurygregory | if stops working is their fault lol | 00:57 |
| JayF | because I *know* they happen in the real world | 00:57 |
| JayF | it'd be the most killer feature ever to actually have ironic retry on random network failures | 00:57 |
| JayF | but I don't think we /quite/ have all the knobs yet | 00:57 |
| iurygregory | ++ | 00:58 |
| opendevreview | cid proposed openstack/ironic master: [WIP] Eventlet: Migrate API & JSON-RPC to cheroot https://review.opendev.org/c/openstack/ironic/+/951054 | 01:34 |
| opendevreview | cid proposed openstack/ironic master: setup,docs: drop `ironic-api-wsgi` script https://review.opendev.org/c/openstack/ironic/+/951055 | 01:34 |
| opendevreview | cid proposed openstack/ironic master: [WIP] Eventlet: Migrate API & JSON-RPC to cheroot https://review.opendev.org/c/openstack/ironic/+/951054 | 01:38 |
| opendevreview | cid proposed openstack/ironic master: [WIP] Eventlet: Migrate API & JSON-RPC to cheroot https://review.opendev.org/c/openstack/ironic/+/951054 | 01:50 |
| opendevreview | Merged openstack/ironic master: [ci] Restore ovn-ipv6 job to voting https://review.opendev.org/c/openstack/ironic/+/951021 | 02:33 |
| *** jroll00 is now known as jroll0 | 06:33 | |
| cardoe | JayF: only did networking-baremetal. Not NGS. | 07:03 |
| rpittau | good morning ironic! o/ | 07:03 |
| abongale | Good morning ironic! | 09:20 |
| iurygregory | good morning ironic | 10:00 |
| opendevreview | Takashi Kajinami proposed openstack/ironic master: Drop explicit executor argument https://review.opendev.org/c/openstack/ironic/+/950957 | 10:07 |
| opendevreview | Syed Haseeb Ahmed proposed openstack/ironic master: This patch adds a condition [inspector]update_pxe_enabled to control whether to perform updates to pxe_enabled field of Port while performing inspection, defaults to False https://review.opendev.org/c/openstack/ironic/+/951114 | 10:56 |
| opendevreview | Syed Haseeb Ahmed proposed openstack/ironic master: This patch adds a condition [inspector]update_pxe_enabled to control whether to perform updates to pxe_enabled field of Port while performing inspection, defaults to True https://review.opendev.org/c/openstack/ironic/+/951114 | 11:19 |
| opendevreview | Merged openstack/ironic master: Replace eventlet usage in `pxe_filter` https://review.opendev.org/c/openstack/ironic/+/950904 | 11:30 |
| opendevreview | Merged openstack/ironic master: Replace `eventlet.spawn_n` in Inspector Interface https://review.opendev.org/c/openstack/ironic/+/950905 | 11:30 |
| opendevreview | cid proposed openstack/networking-baremetal master: Add conductor group sharding support https://review.opendev.org/c/openstack/networking-baremetal/+/948432 | 13:23 |
| * cid erm, that wasn't it. | 13:24 | |
| opendevreview | cid proposed openstack/networking-baremetal master: Add conductor group sharding support https://review.opendev.org/c/openstack/networking-baremetal/+/948432 | 13:25 |
| opendevreview | Julia Kreger proposed openstack/ironic-tempest-plugin master: Tempest science - remove the decorator?! https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/951139 | 13:39 |
| rm_work[m] | wait, did someone say that nova could ALSO do power events to ironic nodes? | 14:12 |
| rm_work[m] | I've got debug logging on in ironic now and had a node get shut off, but I DO NOT see any ipmi events for that node / at that time (i do see some ipmi calls, so I know it's logging them) | 14:13 |
| TheJulia | I'm not sure what your talking about, could you elaborate a little more, thanks! | 14:13 |
| TheJulia | shut off by Ironic, or shut off separately? | 14:14 |
| TheJulia | The way Ironic works is it polls the power state of the nodes, and asserts what it believes the state to be based upon the request state, so if a node is requested off by the API, then ironic will ultimately power the node down. | 14:14 |
| rm_work[m] | sorry, was talking before about this but obviously context stays with me better XD | 14:14 |
| rm_work[m] | we have nodes getting randomly shut off | 14:14 |
| TheJulia | ohhhhhh | 14:15 |
| TheJulia | okay | 14:15 |
| rm_work[m] | trying to track it down ... we have the options set to not try to force the power state to match DB | 14:15 |
| TheJulia | oh, hmm | 14:15 |
| TheJulia | so, disabling powers state enforcement can get things to become a little weird | 14:15 |
| rm_work[m] | and I see logs like this from nova-compute-ironic: INFO nova.compute.manager [None req-b58c5858-31df-49bc-bca7-64b3622da041 - - - - - -] [instance: bd13108c-74e9-4494-bb28-7e58685cf2ce] During _sync_instance_power_state the DB power_state (1) does not match the vm_power_state from the hypervisor (4). Updating power_state in the DB to match the hypervisor. | 14:15 |
| TheJulia | but it sounds like your trying to see if Ironic is powering the nodes off, or what is ultimately causing the nodes to be powered down | 14:16 |
| rm_work[m] | yeah, if we can figure this out and make it stop, we'd feel more comfortable turning that back on :) | 14:16 |
| TheJulia | Yeah, so nova also polls ironic for each node it is repsonsible for | 14:16 |
| TheJulia | depending on the setting, nova also tries to enforce the power state | 14:16 |
| rm_work[m] | but also seems like it's making DB match reality, not trying to force reality to match DB | 14:16 |
| rm_work[m] | "Updating power_state in the DB to match the hypervisor." | 14:17 |
| rm_work[m] | right? | 14:17 |
| TheJulia | yeah, because it sounds like its been disabled configuration wise | 14:17 |
| TheJulia | yeah | 14:17 |
| rm_work[m] | so ... if nova and ironic are set to not enforce power state... and ironic with debug logs on does not show any ipmi events around the time of the shutoff (or in fact anything with that node's IP or ID at all) ... | 14:18 |
| opendevreview | Julia Kreger proposed openstack/ironic master: DNM: CI Science - Expand the multinode job https://review.opendev.org/c/openstack/ironic/+/950206 | 14:18 |
| rm_work[m] | how did this machine shut off | 14:18 |
| rm_work[m] | ... DC gremlins? | 14:18 |
| TheJulia | is it a particular node? | 14:19 |
| TheJulia | or specific particular nodes? | 14:19 |
| rm_work[m] | it's random ones | 14:19 |
| TheJulia | hmmm | 14:19 |
| rm_work[m] | we get 2-3 per day | 14:19 |
| rm_work[m] | I don't see a specific pattern | 14:19 |
| rm_work[m] | but I will try to keep charting out all the data and see if anything jumps out | 14:19 |
| TheJulia | That does sound suspicious | 14:19 |
| rm_work[m] | I just figured it had to be coming from an ipmi call or something | 14:19 |
| TheJulia | Depending on the BMC, you may be able to get request logs | 14:19 |
| rm_work[m] | the firmware is kinda garbage, and some ipmi commands are like ... broken? and COULD cause the machine to power off | 14:20 |
| TheJulia | and that *might* show the IP the requests are coming from, or maybe it is just DC gremlins | 14:20 |
| TheJulia | so | 14:20 |
| rm_work[m] | hmmm, I can try logging into the BMC stuff and see what it'll show me | 14:20 |
| TheJulia | if memory serves, the IPMI flag for power on and get power status and power off are very close to each other | 14:20 |
| TheJulia | so if the BMC has some sort of sporatic off-by-one error, then it seems concievable | 14:21 |
| TheJulia | still.. *super* weird | 14:21 |
| rm_work[m] | yes, been trying to figure it out for actual months | 14:21 |
| TheJulia | I'd check to see if there are multiple sessions being observed by the BMC. Some will just de-duplicate IPMI client interactions for a session list | 14:21 |
| TheJulia | ugh | 14:22 |
| JayF | make sure to reset your bmc as "hard" as you can without losing IP info | 14:25 |
| JayF | that fixed flakey ipmi stuff for me in the past :| | 14:25 |
| rm_work[m] | hold up ... someone just sent me a syslog timestamp for the "shutdown"... does IPMI actually propagate a shutdown to the OS or is it just ... a hard power off? | 14:26 |
| JayF | it's configurable; if you have the correct ipmi modules loaded in the OS and Ironic is configured to (I do not remember the default), you can send a soft power off | 14:28 |
| JayF | and it's sent to the OS similar to an ACPI power button press | 14:29 |
| rm_work[m] | I assumed it didn't actually send anything to the OS, but I guess the power button on my desktop does send a shutdown -h when I press it once... so hmm | 14:33 |
| opendevreview | Syed Haseeb Ahmed proposed openstack/ironic master: This patch adds a condition [inspector]update_pxe_enabled to control whether to perform updates to pxe_enabled field of Port while performing inspection, defaults to True https://review.opendev.org/c/openstack/ironic/+/951114 | 14:33 |
| rm_work[m] | WAIT HOLD UP, I was dumb and assumed the IP they were giving me was the IPMI address and the ID was the ironic node... neither of which were actually the case for some reason. I totally see ipmi getting called, just not sure why | 14:38 |
| TheJulia | Ironic *by default* does send hard power off | 14:38 |
| rm_work[m] | this is "power soft" O_o | 14:38 |
| JayF | power soft is what it sounds like | 14:39 |
| TheJulia | yup, which sends a signal to the OS to shutdown | 14:39 |
| JayF | and ironic will check, if it's not off by $timeout; we'll send the power off | 14:39 |
| rm_work[m] | what would cause ironic-conductor to send a power soft | 14:39 |
| TheJulia | an explicit ask to do so | 14:39 |
| rm_work[m] | wut | 14:39 |
| TheJulia | but that would be logged | 14:39 |
| JayF | which can be in two forms: | 14:39 |
| JayF | 1) someone running `openstack server stop $instance` for a server hsoted by ironic | 14:39 |
| JayF | 2) somehow the DB says "node is powered off", but it's actually on, and power status loop rectifies it | 14:40 |
| JayF | (obviously direct API request is another way) | 14:40 |
| TheJulia | soft power off can also be a gremlin pushing a physical button. | 14:40 |
| TheJulia | or a button shorting out | 14:40 |
| JayF | well, if he has a log saying `power soft` | 14:40 |
| JayF | with ipmitool | 14:40 |
| JayF | it's ironic, right? | 14:40 |
| JayF | or some other ipmi-talkin' tool | 14:40 |
| TheJulia | yup | 14:40 |
| TheJulia | *but* if the host just registeres soft power off suddenly, for all we know it could be the button degrading or soemthing | 14:41 |
| JayF | ++ | 14:41 |
| TheJulia | since it is a momentary depress to initiate a soft power odwn | 14:41 |
| rm_work[m] | i mean this is ironic-conductor logs | 14:41 |
| JayF | what's just before it? | 14:41 |
| JayF | why did it call soft power | 14:41 |
| JayF | it should be in the log | 14:41 |
| TheJulia | so is the conductor sending the soft off? | 14:41 |
| * TheJulia wonders if there is a chaos monkey afoot | 14:42 | |
| rm_work[m] | trying to assemble a pastebin | 14:44 |
| TheJulia | ok | 14:44 |
| rm_work[m] | sorry, top to bottom newest to oldest messages: | 14:52 |
| rm_work[m] | https://paste.openstack.org/show/bM6fUPKIAHuipunVSoSe/ | 14:52 |
| rm_work[m] | not sure why some of these have ... totally different dates? odd | 14:52 |
| rm_work[m] | i wonder if there's some error in our log ingestion | 14:53 |
| rm_work[m] | oh, timezones, lol | 14:53 |
| JayF | > Stderr: 'Error: Unable to establish IPMI v2 / RMCP+ session\n': oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command. | 14:53 |
| JayF | This is the entire answer I think. You have unreliable IPMI connectivity to these nodes | 14:54 |
| rm_work[m] | see, this does not look to me like power state sync is disabled: ` | 14:54 |
| JayF | which means Ironic is not effecting power state changes and timing out | 14:54 |
| rpittau | bye everyone, see you in 2 weeks o/ | 14:54 |
| JayF | rpittau: have a good break o/ | 14:55 |
| JayF | rpittau: maybe not see you until 8/1 :D | 14:55 |
| JayF | rpittau: /me is gone for 6 weeks on June 18 | 14:55 |
| TheJulia | Yeah, basically when you see "Unexpected error" with a delay, it means the BMC is out to lunch and didn't respond. | 14:55 |
| TheJulia | That being said, you do have some soft power ops | 14:55 |
| rm_work[m] | so err... yes. there's a couple things going on. one of them is we likely have something misconfigured because the power state checkers get behind and everything is bad lol, part of why we turned off power state sync | 14:55 |
| rm_work[m] | but also the BMCs are flaky as #*($% | 14:55 |
| TheJulia | I'm suspecting the soft offs are someone or something | 14:56 |
| JayF | yeah this is still missing the side that's triggering the power calls | 14:57 |
| JayF | which might be in api logs | 14:57 |
| rm_work[m] | hmmm | 14:57 |
| JayF | soft power off is API-exposed too | 14:57 |
| JayF | rm_work[m]: you integrated with nova at all? | 14:57 |
| rm_work[m] | yes | 14:57 |
| JayF | rm_work[m]: I had an issue similar to this literally a decade ago in onmetal. Some mix of flakey bmcs, Ironic's power status loop ,and Nova's instance power state loop basically getting into a bad state | 14:58 |
| JayF | rm_work[m]: I'd check the instances tied to this node and see if they report powered on/off from nova, ironic, and ipmi directly | 14:58 |
| JayF | and see if they all agree | 14:58 |
| rm_work[m] | lol yeah ok so ironic api: | 14:58 |
| rm_work[m] | 2025-05-28 00:01:17.050 2025-05-28 04:01:17.050 7 INFO eventlet.wsgi.server [None req-fe1c1952-7597-447d-9706-1abff3864fa6 a1064db8099c4ac884c61d6cf2e5126c be7f1cfde7c5473281d556ddec92e005 - - e7173e738c174e4ebea3fdbdf0ee2d8f e7173e738c174e4ebea3fdbdf0ee2d8f] 10.0.1.170 "PUT /v1/nodes/dd43fc20-6794-4077-b789-3349c39ff48c/states/power HTTP/1.1" status: 202 len: 406 time: 0.1088283 | 14:58 |
| JayF | so someone explicitly called power | 14:59 |
| rm_work[m] | I assume this is something telling ironic to power it off | 14:59 |
| JayF | is 10.0.1.170 your nova compute? | 14:59 |
| JayF | (hint: it almost certainly is) | 14:59 |
| rm_work[m] | likely | 14:59 |
| JayF | yeah | 14:59 |
| JayF | so then backtrace to nova logs and see why | 14:59 |
| JayF | either someone called the stop api | 14:59 |
| JayF | or nova thinks the instance should be on, ironic reports it off, it tells ironic to turn it on, the power on fails because the BMC is bad, goto 1 | 14:59 |
| TheJulia | Yeah, I'd trace the request id | 15:00 |
| TheJulia | since that *should* be user submitted on the request | 15:00 |
| JayF | this can happen in a non-user-interactive case, sadly, because of nova instance power syncing periodics | 15:00 |
| JayF | so we can't rule out robot shenanigans :( | 15:00 |
| rm_work[m] | I really really doubt it's a person | 15:00 |
| rm_work[m] | but i mean... could be wrong | 15:00 |
| JayF | what's so hard about syncing a state between three different systems that all think they are the source of truth 😅 | 15:01 |
| TheJulia | Your going to have to trace through nova-compute and all that logging to verify it | 15:01 |
| rm_work[m] | getting zero hits on that reqid on the nova side, weird. wonder if i'm looking up the right thing | 15:01 |
| TheJulia | hmmm | 15:01 |
| JayF | you have one piece of assumption you can work out first: | 15:01 |
| JayF | 10.0.1.170 | 15:01 |
| JayF | what is it | 15:01 |
| JayF | answer that definitively before you dig for logs too hard | 15:01 |
| rm_work[m] | looking >_< | 15:05 |
| rm_work[m] | FWIW: https://paste.openstack.org/show/bF7gIfEg1C8g8QWtvE2O/ | 15:05 |
| rm_work[m] | wtf is that auth plugin error lol | 15:06 |
| TheJulia | so power state hint notifications is not configured in ironic | 15:07 |
| rm_work[m] | does that matter | 15:08 |
| TheJulia | It was done to help allieviate some of the out of sync state issues which can occur | 15:10 |
| rm_work[m] | JayF: so yes FWIW that IP is our nova-compute-ironic (err, and maybe other things | 15:10 |
| TheJulia | *but* in your state weith everything disabled | 15:10 |
| TheJulia | it doesn't really matter | 15:10 |
| rm_work[m] | ) | 15:10 |
| JayF | yeah so I suspect I've described the behavior you're seeing | 15:10 |
| TheJulia | It looks like a request is coming in then to soft power off the node | 15:10 |
| JayF | given that seems to be the feature that would mitigate it | 15:10 |
| JayF | nova DB says instance stopped -> nova requests soft power off to make DB+reality match -> ironic unable to respect request -> goto start | 15:11 |
| rm_work[m] | I mean ... is there a way I can tell nova to just... NOT | 15:11 |
| JayF | I'm just surprised you've not found any logs from nova around the insance power sync state | 15:11 |
| rm_work[m] | yeah IDK maybe I'm searching for the wrong thing | 15:11 |
| rm_work[m] | ok I just searched for "power" and filtered by date and I think I found it | 15:13 |
| JayF | rm_work[m]: https://docs.openstack.org/nova/2024.2/configuration/sample-config.html sync power state interval is the option on nova side. My knowledge of all this flow is approximately 1/4 as old as I am, so please do your own research and ensure it's the right thing for you | 15:13 |
| rm_work[m] | ok I guess we missed that one, because i am not seeing it in any config on this node, meaning it's defaulting to 600 | 15:16 |
| opendevreview | Jay Faulkner proposed openstack/networking-baremetal master: Remove explicit use of eventlet https://review.opendev.org/c/openstack/networking-baremetal/+/947985 | 15:16 |
| JayF | rm_work[m]: that does impact vms fwiw | 15:17 |
| rm_work[m] | hmmmm | 15:17 |
| rm_work[m] | so that hint thing that you said I don't have configured... was made to help alleviate this issue? | 15:17 |
| JayF | you've reached the border of my knowledge; I didn't know such hinting existed until this conversation | 15:18 |
| rm_work[m] | :D | 15:18 |
| JayF | and I don't have the time to dig on this subject now, sorry | 15:18 |
| rm_work[m] | oh, no worries | 15:19 |
| rm_work[m] | you've been extremely helpful | 15:19 |
| JayF | yeah the key is to remember the sync is three way | 15:19 |
| JayF | and that the **actual problem** is the BMC | 15:19 |
| JayF | if the BMC did what you paid your vendor for it to do, none of this happens | 15:19 |
| JayF | it's just a side effect of when one link in the chain refuses to do it's job | 15:19 |
| TheJulia | oh, the sync power state notification (hint) being sent back just needs to be configured on the ironic side. I don't remember the exact details of it, I do know its rarely used in reality | 15:20 |
| opendevreview | Jay Faulkner proposed openstack/networking-generic-switch master: Remove explicit use of eventlet https://review.opendev.org/c/openstack/networking-generic-switch/+/951026 | 15:20 |
| rm_work[m] | During _sync_instance_power_state the DB power_state (1) does not match the vm_power_state from the hypervisor (4). Updating power_state in the DB to match the hypervisor. | 15:20 |
| rm_work[m] | I wish I knew what vm_power_state 4 was | 15:20 |
| rm_work[m] | trying to dig through nova code lol | 15:20 |
| rm_work[m] | I bet it's like "UNKNOWN" | 15:20 |
| rm_work[m] | oh so wait... "Updating power_state in the DB to match the hypervisor." ... so it updates the power state from 1 (on?) to 4 in the db... which... eventually then the VM thinks it is not "on" and runs a shutoff? I am not understanding this | 15:21 |
| JayF | well, I think you discovered nova is syncing in the other direction | 15:22 |
| JayF | if it's updating DB to match, IDK what's going on | 15:22 |
| JayF | could you have a user calling stop on their machine via nova as a result of it being unknown? | 15:22 |
| JayF | I don't think you've still connected the dots between "What exactly is calling ironic's api" and "nova computer" | 15:22 |
| rm_work[m] | yeah like... this is just by itself in the log, literally nothing for a minute before or after: | 15:24 |
| rm_work[m] | 15:24 | |
| rm_work[m] | 2025-05-28 03:56:42.242 1184865 INFO nova.virt.ironic.driver [None req-758d368f-73dd-434e-a3f4-e9f405cacd7f - - - - - -] [instance: f786da6f-a8c0-4617-9cd5-ecd01a540b9c] Successfully hard powered off Ironic node dd43fc20-6794-4077-b789-3349c39ff48c | 15:24 |
| rm_work[m] | ok this is weird tho... when it says that... is not when it got turned off | 15:25 |
| JayF | I'll also note you've had weird timestamps in logs | 15:26 |
| JayF | so can you trust that/ | 15:26 |
| JayF | are you checking only nova-compute logs? | 15:26 |
| rm_work[m] | yeah sorry my logs are double-timestamped | 15:26 |
| JayF | I'm pretty sure these periodics run on the conductor or higher up | 15:26 |
| rm_work[m] | in two timezones >_< | 15:26 |
| JayF | compute is just the remote hands | 15:26 |
| JayF | that req id should be the key | 15:26 |
| rm_work[m] | yeah I couldn't find anything with the req-id, was odd | 15:26 |
| rm_work[m] | this is just odd tho: https://paste.openstack.org/show/bEMtvVrfkXDy0Re4uOIB/ | 15:28 |
| rm_work[m] | first is that DB update | 15:28 |
| rm_work[m] | then like... 3+ hours later it is like "successfully hard powered off!" | 15:28 |
| rm_work[m] | out of nowhere | 15:28 |
| rm_work[m] | except it doesn't actually power off yet? | 15:29 |
| rm_work[m] | then 5 minutes later it's like "huh, instance isn't stopped, calling the stop API" | 15:29 |
| rm_work[m] | and that actually does it | 15:29 |
| JayF | so the thing above that | 15:36 |
| JayF | you can't see | 15:36 |
| JayF | is it acts like someone called `nova stop []` on it | 15:36 |
| JayF | er, openstack server stop or whatever it is in the new world lol | 15:36 |
| JayF | old knowledge comes with somehow still-rememberd syntax for old tools | 15:36 |
| rm_work[m] | ok pretty confident saying the first time that request-id appears is in the ironic-api there... and then goes to the conductor | 15:37 |
| rm_work[m] | very odd, either i'm missing logs or nova just didn't log that request-id for some reason | 15:38 |
| JayF | request ids are well hooked up for user direct stuff | 15:38 |
| JayF | I've seen inconsistencies for service<>service stuff | 15:39 |
| TheJulia | yeah, there can be inconsistencies on service initiated stuff | 15:44 |
| TheJulia | for example if a periodic triggers a thing in nova, then I don't think that gets a request id | 15:44 |
| opendevreview | Merged openstack/ironic-ui master: Drop environments for nose https://review.opendev.org/c/openstack/ironic-ui/+/950940 | 16:08 |
| rm_work | That makes sense then if it was from that sync interval thing | 16:16 |
| opendevreview | Mithun Krishnan Umesan proposed openstack/networking-generic-switch master: Improve NGS documentation. Some minor nitpicks and changes https://review.opendev.org/c/openstack/networking-generic-switch/+/951022 | 16:17 |
| opendevreview | Mithun Krishnan Umesan proposed openstack/networking-generic-switch master: Improve NGS documentation. Some minor nitpicks and changes https://review.opendev.org/c/openstack/networking-generic-switch/+/951022 | 16:18 |
| opendevreview | Merged openstack/python-ironicclient stable/2025.1: Stop using deprecated format_* from osc_utils https://review.opendev.org/c/openstack/python-ironicclient/+/949704 | 16:35 |
| opendevreview | Merged openstack/python-ironicclient stable/2024.2: Stop using deprecated format_* from osc_utils https://review.opendev.org/c/openstack/python-ironicclient/+/949705 | 17:14 |
| opendevreview | Merged openstack/python-ironicclient stable/2024.1: Stop using deprecated format_* from osc_utils https://review.opendev.org/c/openstack/python-ironicclient/+/949706 | 17:14 |
| opendevreview | Merged openstack/ironic master: Only try and do deep network config validate if admin https://review.opendev.org/c/openstack/ironic/+/942921 | 17:33 |
| opendevreview | Merged openstack/ironic master: [CI] Fix libvirt network names in metal3 job logs collection https://review.opendev.org/c/openstack/ironic/+/950950 | 17:33 |
| opendevreview | Merged openstack/ironic master: [CI] metal3 job back to voting https://review.opendev.org/c/openstack/ironic/+/950333 | 17:33 |
| opendevreview | Merged openstack/ironic master: Remove warning filter for old oslo.db https://review.opendev.org/c/openstack/ironic/+/950987 | 17:33 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Consider missing MTU invalid metadata https://review.opendev.org/c/openstack/ironic/+/949385 | 17:37 |
| JayF | dtantsur: TheJulia: the author of https://opendev.org/openstack/ironic-python-agent/commit/7341511b98924ef62769e841bd558698af0efe64 is going to be joining us for eventlet/network fun | 18:30 |
| JayF | \o/ | 18:30 |
| JayF | blast from the far, far, far past lol | 18:30 |
| TheJulia | oh my | 18:32 |
| opendevreview | Verification of a change to openstack/ironic-python-agent stable/2025.1 failed: netutils: Use ethtool ioctl to get permanent mac address https://review.opendev.org/c/openstack/ironic-python-agent/+/950489 | 18:47 |
| opendevreview | Jay Faulkner proposed openstack/networking-baremetal master: Use new syntax for neutron enablement https://review.opendev.org/c/openstack/networking-baremetal/+/951194 | 18:51 |
| opendevreview | Julia Kreger proposed openstack/ironic master: DNM: CI Science - Expand the multinode job https://review.opendev.org/c/openstack/ironic/+/950206 | 19:18 |
| TheJulia | we're getting close, I might have to break the var groups into a group per node | 19:21 |
| JayF | did you look at my nbm fix? | 19:22 |
| TheJulia | Trying to cheat with our plugin | 19:22 |
| JayF | it's depressingly simple. | 19:22 |
| TheJulia | nbm? | 19:22 |
| JayF | networking-baremetal | 19:22 |
| JayF | they charge me by the keystroke over here /s | 19:22 |
| TheJulia | heh | 19:22 |
| TheJulia | we still have the old syntax in ironic's config, easy to fix | 19:23 |
| JayF | it using q-svc | 19:25 |
| JayF | the old old old syntax | 19:25 |
| JayF | instead of q-api or neturon-api | 19:25 |
| JayF | I believe was the actual break | 19:25 |
| JayF | trivial enough to fix and it'll do us good to get q-* outta our dang jobs | 19:25 |
| TheJulia | yeah | 19:44 |
| TheJulia | exactly | 19:44 |
| TheJulia | I suspect my changes won't pass CI on the latest change and I'll nee dto do a variable group for each, which *should* be okay. I just need to doublecheck syntax. | 20:40 |
| opendevreview | cid proposed openstack/ironic master: Add port/portgroup list conductor groups filter https://review.opendev.org/c/openstack/ironic/+/862292 | 21:23 |
| -opendevstatus- NOTICE: The Gerrit service on review.opendev.org is temporarily unreachable due to an ongoing issue in the hosting provider where it resides | 22:07 | |
| opendevreview | Ivan Anfimov proposed openstack/ironic-ui master: wip https://review.opendev.org/c/openstack/ironic-ui/+/951215 | 22:20 |
| opendevreview | Ivan Anfimov proposed openstack/ironic-ui master: wip https://review.opendev.org/c/openstack/ironic-ui/+/951215 | 22:23 |
| opendevreview | Julia Kreger proposed openstack/ironic master: DNM: CI Science - Expand the multinode job https://review.opendev.org/c/openstack/ironic/+/950206 | 22:56 |
| JayF | would be greatly appreciated if someone could land https://review.opendev.org/c/openstack/networking-baremetal/+/951194 to fix that gate | 23:19 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!