| opendevreview | Iury Gregory Melo Ferreira proposed openstack/ironic master: Add DB model for Firmware https://review.opendev.org/c/openstack/ironic/+/883031 | 01:52 |
|---|---|---|
| iurygregory | let's see if this helps to get more logs .-. | 01:52 |
| janders | good morning Ironic o/ | 01:53 |
| iurygregory | morning janders o/ | 01:53 |
| iurygregory | good morning Ironic | 11:06 |
| opendevreview | Vanou Ishii proposed openstack/ironic master: [iRMC] Fix IPMI incompatibility handling error https://review.opendev.org/c/openstack/ironic/+/883101 | 11:30 |
| opendevreview | Merged openstack/virtualbmc master: Bugs are now in Launchpad, update docs to reflect https://review.opendev.org/c/openstack/virtualbmc/+/883454 | 12:27 |
| iurygregory | ok this is weird, I think this would add the new configuration option in ironic.conf https://review.opendev.org/c/openstack/ironic/+/883031/9/devstack/lib/ironic#1504 for all jobs.. but it didn't work for grenade O.o | 12:34 |
| opendevreview | Mahnoor Asghar proposed openstack/python-ironicclient master: Add support for node inventory https://review.opendev.org/c/openstack/python-ironicclient/+/883833 | 12:36 |
| *** dmellado90 is now known as dmellado | 12:48 | |
| opendevreview | Mahnoor Asghar proposed openstack/python-ironicclient master: Add support for node inventory https://review.opendev.org/c/openstack/python-ironicclient/+/883833 | 13:36 |
| JayF | TheJulia: iurygregory: would one of you mind running the meeting? | 13:42 |
| TheJulia | just waking up | 13:44 |
| iurygregory | JayF, I can run | 13:50 |
| JayF | Thank you. I'll likely be away today | 13:55 |
| dtantsur | We have another instance of the Metal3 meetup. JFYI. | 13:56 |
| iurygregory | right | 14:02 |
| iurygregory | we will probably not have a lot of folks available I think .-. | 14:02 |
| dtantsur | and it starts NOW! | 14:02 |
| iurygregory | I'm skipping because I'm trying to figure out the timeout in grenade .-. | 14:03 |
| iurygregory | >.< | 14:03 |
| dtantsur | I'd say the meetup may be more important, especially that we're going to talk about more Ironic features | 14:03 |
| dtantsur | iurygregory: ^^ | 14:04 |
| TheJulia | iurygregory: is it locking up on the db upgrade? | 14:05 |
| TheJulia | or, still I should say from Friday | 14:05 |
| iurygregory | TheJulia, yeah >.< still | 14:05 |
| iurygregory | my attempt to add more verbosity didn't work in the grenade job https://review.opendev.org/c/openstack/ironic/+/883031/9/devstack/lib/ironic#1504 | 14:06 |
| iurygregory | dtantsur, ack I will try to join and just listen | 14:06 |
| * TheJulia wonders why we're suddenly seein gthis | 14:19 | |
| opendevreview | Iury Gregory Melo Ferreira proposed openstack/ironic master: Add DB model for Firmware https://review.opendev.org/c/openstack/ironic/+/883031 | 14:24 |
| TheJulia | hmmmmm | 14:27 |
| TheJulia | iurygregory: so it is not oslo.db or logging in oslo, it is alembic | 14:36 |
| iurygregory | oh I though that adding in oslo.db would help >.< | 14:37 |
| iurygregory | ok, seems like we would need to update alembic.ini with [logger_sqlalchemy] level = DEBUG | 14:40 |
| TheJulia | maybe | 14:43 |
| TheJulia | yeah, we just call alembic.command.upgrade( | 14:47 |
| TheJulia | so it is already set to log at info level | 14:48 |
| TheJulia | which matches all the places it logs | 14:48 |
| TheJulia | so it is just not generating any logging | 14:48 |
| iurygregory | magic | 14:49 |
| TheJulia | lets do this, lets get infra to hold the next ci node which fails with ironic-grenade | 14:53 |
| TheJulia | we can recheck both changes which do schema upgrdes since they seem... vulnerable to this issue, and then investigate trying to reproduce internally | 14:53 |
| TheJulia | otherwise, I don't think we're going to find a simple path forward | 14:54 |
| iurygregory | yeah | 14:54 |
| iurygregory | makes sense | 14:54 |
| iurygregory | funny thing is that I remember your patch with success on grenade | 14:57 |
| TheJulia | yeah. It was succeeding | 14:58 |
| TheJulia | and I haven't changed the migration :( | 14:58 |
| iurygregory | yup =( | 14:59 |
| iurygregory | #startmeeting ironic | 15:00 |
| iurygregory | #startmeeting ironic | 15:00 |
| opendevmeet | Meeting started Mon May 22 15:00:30 2023 UTC and is due to finish in 60 minutes. The chair is iurygregory. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| opendevmeet | The meeting name has been set to 'ironic' | 15:00 |
| iurygregory | Hello everyone o/ | 15:00 |
| dtantsur | o/ | 15:00 |
| * dtantsur is in two meetings | 15:00 | |
| iurygregory | This will likely be a short meeting XD | 15:00 |
| masghar | o/ | 15:00 |
| TheJulia | o/ | 15:01 |
| iurygregory | welcome to our weekly meeting \o/ | 15:01 |
| iurygregory | #topic announcements/reminder | 15:01 |
| iurygregory | #note Standing reminder to review patches tagged ironic-week-prio and to hashtag any patches ready for review with ironic-week-prio | 15:01 |
| hjensas | o/ | 15:01 |
| iurygregory | #link https://tinyurl.com/ironic-weekly-prio-dash | 15:01 |
| iurygregory | #note Please avoid running a `recheck` command bare, without any other comments. Obviously, best case is to troubleshoot and fix an issue in CI, but in case of some ephemeral failure, please just note that -- e.g. `recheck jobname failed`. This is tracked at an OpenStack level | 15:02 |
| iurygregory | #link https://etherpad.opendev.org/p/recheck-weekly-summary | 15:02 |
| iurygregory | #info 3 weeks for the OIS Vancouver | 15:03 |
| TheJulia | iurygregory: hold set | 15:03 |
| iurygregory | holing | 15:03 |
| iurygregory | was just copying the announcements | 15:03 |
| iurygregory | Does anyone have something we should add to announcements/reminder? | 15:04 |
| TheJulia | nothing on my end | 15:05 |
| dtantsur | As an outcome of Metal3 meetups, we'll build a wishlist for Ironic :) | 15:05 |
| dtantsur | (We'll implement things ourselves, just need agreement) | 15:05 |
| iurygregory | nice \o/ | 15:05 |
| TheJulia | cool cool | 15:06 |
| iurygregory | we don't have action items so skipping next topic =) | 15:06 |
| iurygregory | #topic Review Ironic CI status & update whiteboard if needed | 15:06 |
| iurygregory | grenade is misbehaving for patches with DB migrations hitting timeout =( | 15:07 |
| iurygregory | TheJulia had a nice idea, we will try to get infra to hold the next ci node which fails with ironic-grenade | 15:07 |
| iurygregory | we landed a few fixes last week to improve CI, so we should be in a good shape in other patches | 15:08 |
| iurygregory | does anyone would like to add something regarding CI? | 15:09 |
| TheJulia | We may need to backport some of the things | 15:10 |
| TheJulia | but overall, master branch seems to be a little happier | 15:10 |
| iurygregory | yeah | 15:10 |
| iurygregory | #topic Review ongoing 2023.2 Workstreams | 15:11 |
| iurygregory | #link https://etherpad.opendev.org/p/IronicWorkstreams2023.2 | 15:11 |
| iurygregory | I'm updating the Firmware Interface with the other patches I have open | 15:12 |
| TheJulia | same on service/hold | 15:13 |
| TheJulia | and dpus | 15:13 |
| iurygregory | ok moving on | 15:15 |
| iurygregory | #topic Open discussion | 15:15 |
| iurygregory | Does anyone have something to discuss? | 15:16 |
| TheJulia | I do not | 15:16 |
| iurygregory | ok, I think this is all for today | 15:17 |
| iurygregory | tks everyone! | 15:17 |
| iurygregory | #endmeeting | 15:17 |
| opendevmeet | Meeting ended Mon May 22 15:17:57 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:17 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/ironic/2023/ironic.2023-05-22-15.00.html | 15:17 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/ironic/2023/ironic.2023-05-22-15.00.txt | 15:17 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/ironic/2023/ironic.2023-05-22-15.00.log.html | 15:17 |
| TheJulia | iurygregory: I've rechecked my dpu chnage | 15:21 |
| TheJulia | I'm likely going to go work on my summit presentation for an hour or two | 15:22 |
| iurygregory | ack | 15:22 |
| TheJulia | so we can let the changes just do their thing | 15:22 |
| iurygregory | I will probably be in my lunch time in about 1h | 15:22 |
| iurygregory | yeah | 15:22 |
| fungi | i just saw someone try to patch a regression in sushy-tools through the github mirror, if someone's interested in helping usher it into gerrit: https://github.com/openstack/sushy-tools/pull/6 | 15:51 |
| dtantsur | oh | 16:00 |
| dtantsur | fungi: I assume we cannot disable PRs? | 16:01 |
| fungi | dtantsur: it's a long-standing missing feature of github, which seems to be quite intentional given how many people have asked for it over the past decade | 16:02 |
| fungi | we auto-close them with a bot that replies with a message linking to our contributor documentation, but sometimes that can seem cold and unfeeling so a personal touch can help avoid alienating potential new contributors | 16:03 |
| dtantsur | yep, I agree. this is very unfortunately (and very intentional, yes) | 16:09 |
| iurygregory | TheJulia, habemus node https://zuul.opendev.org/t/openstack/nodes | 19:04 |
| TheJulia | iurygregory: do you want to get started or do you want to dig together? I'm busy for the next 1.5 hours most likely | 19:06 |
| iurygregory | I will start looking | 19:07 |
| TheJulia | okay | 19:12 |
| TheJulia | I'll ping you when I'm done with my meeting which starts in 15 minutes | 19:13 |
| * TheJulia needs to make coffee | 19:13 | |
| iurygregory | I think it was a bad idea run "ironic-status upgrade check" in the node .-. | 19:48 |
| iurygregory | 7min and no output | 19:49 |
| TheJulia | so reproduced... | 19:53 |
| TheJulia | control-c ?! | 19:53 |
| TheJulia | or control-z and then kill the pid? | 19:53 |
| iurygregory | planning on do that | 19:54 |
| iurygregory | well, it got stuck in ironic-status upgrade check... | 19:54 |
| iurygregory | I was expecting it would succeed | 19:54 |
| iurygregory | so I would run the dbsync right after | 19:54 |
| TheJulia | heh | 19:55 |
| TheJulia | oh, upgrade check is frozen | 19:55 |
| TheJulia | *because* the db is locked | 19:55 |
| TheJulia | it can't select so it is waiting to select | 19:55 |
| TheJulia | so... what is the db is doing is the question | 19:55 |
| iurygregory | funny that the mysql error.log doesn't show anything | 19:57 |
| TheJulia | yup | 20:07 |
| TheJulia | we could turn on query logging, maybe | 20:07 |
| TheJulia | requires restarting mysql | 20:07 |
| TheJulia | have you ever used strace? | 20:07 |
| iurygregory | nope, but I can try to figure out | 20:08 |
| TheJulia | so | 20:08 |
| TheJulia | hmm | 20:08 |
| TheJulia | call is almost over, so I can jump on hopefully | 20:08 |
| TheJulia | I guess the first thing is to ask the database what it thinks is going on | 20:09 |
| iurygregory | https://paste.opendev.org/show/b1z9IwW8HdxVkbsxt8Yx/ | 20:10 |
| TheJulia | https://www.devart.com/dbforge/mysql/studio/show-running-queries-in-processlist.html | 20:11 |
| iurygregory | | 193 | root | localhost:39972 | nova_cell1 | Sleep | 16813 | | NULL | | 20:11 |
| iurygregory | | 205 | root | localhost:40930 | ironic | Query | 16774 | Waiting for table metadata lock | ALTER TABLE nodes ADD COLUMN firmware_interface VARCHAR(255) | 20:11 |
| iurygregory | waiting for table metadata lock?! | 20:11 |
| TheJulia | oooh | 20:11 |
| TheJulia | yeah | 20:11 |
| TheJulia | wow | 20:11 |
| iurygregory | I think I did something wrong in my patch | 20:13 |
| TheJulia | wait a second | 20:14 |
| TheJulia | https://stackoverflow.com/questions/13148630/how-do-i-find-which-transaction-is-causing-a-waiting-for-table-metadata-lock-s | 20:14 |
| iurygregory | things changed since we are using verions 8 | 20:22 |
| TheJulia | ugh | 20:23 |
| iurygregory | I've found the transaction | 20:23 |
| TheJulia | \o/ | 20:24 |
| TheJulia | what is it?!? | 20:24 |
| TheJulia | :) | 20:24 |
| iurygregory | ---TRANSACTION 421683208347256, not started mysql tables in use 1, locked 1 0 lock struct(s), heap size 1128, 0 row lock(s) | 20:24 |
| iurygregory | trying to figure out how to match the ID | 20:25 |
| TheJulia | well, not started | 20:25 |
| TheJulia | Any other transactions pending? | 20:25 |
| iurygregory | https://paste.opendev.org/show/bkqI5fTG6HRJarTLBour/ | 20:25 |
| iurygregory | nope .-. | 20:26 |
| iurygregory | all of them are not started .-. | 20:26 |
| TheJulia | and off my call | 20:26 |
| clarkb | I've also seen mariadb and mysql act differently in weird ways. | 20:27 |
| clarkb | (not sure if that is involvedhere) | 20:28 |
| iurygregory | maybe .-. | 20:28 |
| TheJulia | do we have any signs of the query counter stuff running? | 20:28 |
| TheJulia | just trying to think of *anything* which would read from the overall database schema data/informaiton/configuration | 20:29 |
| clarkb | for example mysql caught a transaction issue that mariadb did not when udpating to sqla 2.0. Also trying to move to mariadb produces semi random failures in the whole test case setup a random temporary db process | 20:29 |
| iurygregory | TheJulia, do you want me to add you key ? | 20:31 |
| TheJulia | yeah, and lets jump on a call | 20:31 |
| NobodyCam | Good afternoon Ironic Folks | 21:52 |
| NobodyCam | anyone happen to know if its possible to use ramdisk deployment thru nova (Ie. `server create`) is there an example showing how kernel and ramdisk are specified? | 21:54 |
| NobodyCam | works great thru ironic | 21:54 |
| TheJulia | zzzeek_: o/ are you aware of any issues with alembic going to add a column mysql locking on a transaction lock (yes, I realize the ask is a long shot) :) | 21:59 |
| TheJulia | NobodyCam: so, the idea has always. been it just uses the named kernel/ramdisk on a glance "image", ignoring the image contents as if it was a partition image uploaded into glance | 22:01 |
| TheJulia | which should just work and I'm fairly sure we've had someone do it in the past | 22:02 |
| TheJulia | NobodyCam: It might be the scientific folks are using ironic directly, but I bet slurm is their way and I suspect htat just uses nova | 22:16 |
| NobodyCam | hummm | 22:46 |
| NobodyCam | oh iso method, I have kernel and ramdisk | 22:49 |
| NobodyCam | oh ISO method was a question... even if it had no question mark | 22:50 |
| TheJulia | oh, with nova, unlikely | 22:53 |
| TheJulia | I mean, you could patch nova to teach it what to do with an iso, but it is something we've not done because nobody has asked | 22:54 |
| TheJulia | iurygregory: got a query log out of the db... nothing much more than we already captured | 22:55 |
| TheJulia | can reroduce it locally, thinking of taking the log and actually seeing if I can lock the db connection using it | 22:55 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!