| NobodyCam | TheJulia: Thank you, working now | 00:10 |
|---|---|---|
| TheJulia | iurygregory: so, replaying everything on the connection, can't reproduce | 00:12 |
| iurygregory | TheJulia, ack =( | 03:32 |
| TheJulia | I had a theory to explore tomorrow | 03:32 |
| iurygregory | ok | 03:35 |
| iurygregory | fingers crossed | 03:35 |
| opendevreview | Iury Gregory Melo Ferreira proposed openstack/ironic master: Add DB model for Firmware https://review.opendev.org/c/openstack/ironic/+/883031 | 03:52 |
| iurygregory | new attempt just to see | 03:52 |
| iurygregory | gnight | 03:52 |
| opendevreview | Graeme Moss proposed openstack/networking-generic-switch master: Fix fallback to plug_port_to_network on 802_3ad bonds https://review.opendev.org/c/openstack/networking-generic-switch/+/881959 | 07:51 |
| opendevreview | Graeme Moss proposed openstack/networking-generic-switch master: Fix fallback to plug_port_to_network on 802_3ad bonds https://review.opendev.org/c/openstack/networking-generic-switch/+/881959 | 09:29 |
| iurygregory | good morning Ironic | 11:42 |
| opendevreview | Jacob Anders proposed openstack/ironic master: [WIP] Follow up change to 880575 https://review.opendev.org/c/openstack/ironic/+/883938 | 13:14 |
| iurygregory | <eyes> ^ | 13:16 |
| TheJulia | iurygregory: still froze eh? | 13:19 |
| iurygregory | TheJulia, yeah, I've spent part of my morning now reading things | 13:20 |
| TheJulia | anything interesting? | 13:20 |
| iurygregory | found https://github.com/miguelgrinberg/Flask-Migrate/issues/41 | 13:20 |
| iurygregory | saw some people mentioning that a process is try to access the DB we are modifying and it could cause this problem | 13:21 |
| TheJulia | yeah | 13:22 |
| TheJulia | I could stop everything, restart just the db, and reproduce yesterday | 13:22 |
| iurygregory | https://prog.world/the-process-gets-stuck-in-info-alembic-migration-running-upgrade/ | 13:22 |
| TheJulia | I'm starting to think it is alembic | 13:22 |
| iurygregory | yeah, I remember that part .-. | 13:22 |
| iurygregory | now I'm reading https://dev.mysql.com/doc/relnotes/mysql/8.0/en/news-8-0-33.html to see if there is something | 13:23 |
| TheJulia | good plan | 13:23 |
| TheJulia | I started digging through our migraiton to alembic migration code to get an understanding of it yesterday | 13:24 |
| TheJulia | what I don't understand is it looks like alembic is trying some sort of random order | 13:24 |
| TheJulia | and then figures out what to do from there | 13:24 |
| iurygregory | oh jesus .-. | 13:44 |
| iurygregory | what blows my mind is "why only in grenade?" | 13:45 |
| TheJulia | every other job runs as a clean slate | 13:54 |
| iurygregory | yeah | 13:56 |
| BobZAnnapolis____ | Following docs to install Bifrost (ussuri,victoria,wallaby,yoga) w/testenv, crashing out on sushy-tools installation : AttributeError: module 'lib' has no attribute 'X509_V_FLAG_CB_ISSUER_CHECK' <- no Google search responses to this error code worked; https://docs.openstack.org/bifrost/any_vsn/install/index.html# <- all failing :-( tia | 13:56 |
| TheJulia | BobZAnnapolis____: any chance we can get more around that error, possibly a exception backtrace? | 13:58 |
| TheJulia | BobZAnnapolis____: also, same with master branch? what distribution are you using? | 14:11 |
| iurygregory | seems like a problem with pyopenssl | 14:15 |
| iurygregory | I never saw it in my deployments | 14:15 |
| TheJulia | pip freeze output would help too | 14:19 |
| * TheJulia does the whole t'pol intruiging look from enterprise and digs back into the migration code | 14:19 | |
| * JayF wonders if pyopenssl wants OpenSSL 3.0 and BobZAnnapolis____ has 1.x or vice-versa | 14:19 | |
| opendevreview | Alex Welsh proposed openstack/bifrost master: Replace Cirros references with generic descriptors https://review.opendev.org/c/openstack/bifrost/+/883943 | 14:25 |
| BobZAnnapolis____ | i deleted the vm, will rebuild/retry. The stack trace indicates the lib imported from _lib OpenSSL can't find those attributes;i've tried different versions of openSSL, pyOpenSSL (>22.#) & cryptography (>34.#) - i know it works (obviously), just trying to find the correct link / instructions / docs - tia | 14:31 |
| JayF | iurygregory: TheJulia: aw hell ... https://3df9d8624791e146e7ca-5b369dc32177e920b38f658289f2b193.ssl.cf2.rackcdn.com/836966/5/check/ironic-grenade-skip-level/69a6b89/controller/logs/grenade.sh_log.txt | 14:38 |
| JayF | iurygregory: TheJulia: tl;dr: on the grenade-skip-level job, it looks like we're hanging on *different* migrations from last cycle | 14:39 |
| JayF | iurygregory: TheJulia: implying to me right now with the current state of master, we're unable to run migrations :| | 14:39 |
| TheJulia | yup | 14:39 |
| iurygregory | oh jesus | 14:41 |
| * TheJulia resumes trying to get the upgrade command to dump the process list output | 14:41 | |
| iurygregory | maybe we forgot something from Node Inventory - Shard - New Migrations? | 14:44 |
| TheJulia | dunno | 14:45 |
| TheJulia | https://paste.openstack.org/show/b2UHxNEXW2ZvmBstjBTT/ | 14:47 |
| iurygregory | maybe we could stop everything :D and only start them after we do things (but I don't think it would help, since we tested without any process running... ) | 14:48 |
| TheJulia | what do you mean stop everything? | 14:49 |
| iurygregory | all serivces | 14:50 |
| iurygregory | :D | 14:50 |
| iurygregory | nova, neutron etc | 14:50 |
| TheJulia | login and take a look, basically that is what I did, I've not managed to get the wsgi apps from re-spawning | 14:50 |
| TheJulia | but non of them open the db | 14:50 |
| TheJulia | the ironic db that is | 14:50 |
| TheJulia | https://paste.openstack.org/show/bZJaNtZriBO6hJhlPX48/ | 14:51 |
| TheJulia | when the ironic-dbsync process has been control-c'ed, and I've run the db query directly | 14:51 |
| * TheJulia wonders about downgrading mysql | 14:54 | |
| JayF | I owe something downstream that's now late since I had to take yesterday off; I'm gonig to focus on that but if we're still here when I come back, I'll start setting up a testbed | 14:57 |
| TheJulia | so I got it to complete, but I had to kill all open database connections also opened by alembic looking at the database | 15:19 |
| iurygregory | wow | 15:36 |
| iurygregory | TheJulia, so downgrading mysql seems to work? | 15:37 |
| * iurygregory brb lunch | 15:37 | |
| TheJulia | well, got it to complete by hunting down one of the hanging alembic connections | 15:38 |
| TheJulia | well, I've gotten a query to hang directly against the db | 15:52 |
| TheJulia | somehow https://paste.openstack.org/show/bbz0PNZmXNwpJftylCPG/ requires a lock | 15:56 |
| iurygregory | wow | 15:57 |
| TheJulia | https://paste.openstack.org/show/b88nn8APJ9bUZFvLzxcS/ | 15:58 |
| TheJulia | it is only frozen because we're already frozen though | 15:58 |
| TheJulia | but the fact it *needs* a lock is... weird to me | 15:59 |
| iurygregory | interesting | 16:04 |
| iurygregory | from https://dev.mysql.com/doc/relnotes/mysql/8.0/en/news-8-0-3.html https://paste.opendev.org/show/bGjTJuIZPRcInACDqBrI/ | 16:09 |
| iurygregory | seems like there was a change in the default of innodb_autoinc_lock_mode | 16:09 |
| TheJulia | 8.0.3 vs 8.0.33 | 16:11 |
| JayF | holy hell that's a giant change in behavior | 16:14 |
| TheJulia | 2017 | 16:15 |
| iurygregory | for some reason I only looked at the first number after the 0 /facepalm | 16:15 |
| TheJulia | mysql doesn't permit downgrade | 16:33 |
| iurygregory | .-. | 16:38 |
| dtantsur | So, I assume I'm not the only person who wants to burn all the bloody computers today? | 16:38 |
| iurygregory | I want to do this since last week | 16:39 |
| TheJulia | dtantsur: I was thinking molten computers would be appropriate | 16:53 |
| dtantsur | *nod* | 16:57 |
| TheJulia | https://dev.mysql.com/doc/refman/8.0/en/metadata-locking.html#metadata-lock-release | 17:10 |
| TheJulia | well, the entire page, not just on how the lock release pattern works | 17:10 |
| JayF | What version of alembic we using? https://alembic.sqlalchemy.org/en/latest/changelog.html#change-3374635e9b84c495d2ede3c978cd57e6 | 17:12 |
| JayF | that sounds highly suspect | 17:12 |
| JayF | I was thinking, if mysql changed behavior there might be a matching alembic change/fix | 17:12 |
| TheJulia | oh, I think I see what is going on | 17:18 |
| TheJulia | bbingo | 17:21 |
| TheJulia | working on two patches now | 17:44 |
| iurygregory | \o/ | 18:08 |
| JayF | TheJulia: is it an ironic bug or something systemmic for openstack? | 18:12 |
| TheJulia | its our code | 18:12 |
| TheJulia | well, from an openstack pattern | 18:12 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Explicitly use a session for DB version check https://review.opendev.org/c/openstack/ironic/+/883967 | 18:22 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Remove model_query use from general dbapi calls https://review.opendev.org/c/openstack/ironic/+/883968 | 18:22 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Remove unused get_not_vesions from dbapi https://review.opendev.org/c/openstack/ironic/+/883969 | 18:22 |
| TheJulia | just the first one is the one that is needed | 18:22 |
| opendevreview | Julia Kreger proposed openstack/ironic master: DPU modeling - parent_node DB/Model/API https://review.opendev.org/c/openstack/ironic/+/880114 | 18:23 |
| JayF | TheJulia: so we had a query object outstanding until it was GC'd, more or less? | 18:29 |
| TheJulia | yup, exactly | 18:30 |
| TheJulia | and because the way the locking works if they didn't compeltely rollback a transaction, they blocked the alter table update | 18:30 |
| TheJulia | nothing forces it in the model_query pattern, the with _session_for_read() pattern *does* force it on the unwind of the session | 18:31 |
| TheJulia | iurygregory: okay if I let infra remove the held vm? | 18:45 |
| opendevreview | Julia Kreger proposed openstack/ironic-tempest-plugin master: CI: Advance stable jobs to 2023.1 https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/883600 | 19:09 |
| opendevreview | Julia Kreger proposed openstack/ironic-tempest-plugin master: CI: Advance stable jobs to 2023.1 https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/883600 | 19:11 |
| opendevreview | Harald Jensås proposed openstack/ironic-tempest-plugin master: rbac - Fix vif_attach expected return values https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/883032 | 19:27 |
| opendevreview | Verification of a change to openstack/ironic stable/2023.1 failed: [stable-only] [CI] BFV, RBAC jobs marked non-voting https://review.opendev.org/c/openstack/ironic/+/883697 | 19:39 |
| opendevreview | Pierre Riteau proposed openstack/virtualbmc master: Remove duplicate space in log message https://review.opendev.org/c/openstack/virtualbmc/+/883977 | 20:07 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Explicitly use a session for DB version check https://review.opendev.org/c/openstack/ironic/+/883967 | 20:24 |
| JayF | TheJulia: you missed your own comment on that patch (adding back the _) | 20:25 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Explicitly use a session for DB version check https://review.opendev.org/c/openstack/ironic/+/883967 | 20:26 |
| TheJulia | ack, going to the post office | 20:26 |
| JayF | TheJulia: did that need a backport, too, once it lands? | 20:26 |
| TheJulia | yeah, likely | 20:30 |
| TheJulia | we've seen some raceiness before py3.10, but 3.10 seems to be very.. racey | 20:31 |
| JayF | aight, you have a +2 from me | 20:36 |
| iurygregory | TheJulia, sure (sorry I was at the physiotherapy | 20:38 |
| iurygregory | ok, so we shouldn't be using model_query ? | 20:40 |
| JayF | so model_query, and similar, returns a result that's a sqla model | 20:42 |
| JayF | that until it's GC'd represents an open session to the db | 20:42 |
| iurygregory | I think need more coffee now | 20:43 |
| iurygregory | and I will review the patches | 20:43 |
| TheJulia | yeah, we should not be using model query | 20:47 |
| TheJulia | I think we need to rip it out entirely | 20:47 |
| TheJulia | but we can't go backporting that | 20:47 |
| TheJulia | use "with _session_for_read() as session:\n session.query(model).filter.whatever.magic | 20:49 |
| TheJulia | " | 20:49 |
| iurygregory | time refactor my patche and put in chain with the first one with the fix | 20:56 |
| TheJulia | I need to fix my second patch, it basically weeds the rest of model_query out | 20:57 |
| iurygregory | but with the first one we should get grenade working right? | 20:58 |
| TheJulia | Yeah | 20:58 |
| iurygregory | cool cool cool | 20:58 |
| opendevreview | Iury Gregory Melo Ferreira proposed openstack/ironic master: Add DB model for Firmware https://review.opendev.org/c/openstack/ironic/+/883031 | 21:17 |
| opendevreview | Iury Gregory Melo Ferreira proposed openstack/ironic master: Add ironic-grenade-skip-level Job https://review.opendev.org/c/openstack/ironic/+/836966 | 21:19 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Remove model_query use from general dbapi calls https://review.opendev.org/c/openstack/ironic/+/883968 | 21:38 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Remove unused get_not_versions from dbapi https://review.opendev.org/c/openstack/ironic/+/883969 | 21:38 |
| opendevreview | Iury Gregory Melo Ferreira proposed openstack/ironic master: Add DB API for Firmware https://review.opendev.org/c/openstack/ironic/+/883062 | 21:41 |
| opendevreview | Merged openstack/virtualbmc master: Remove duplicate space in log message https://review.opendev.org/c/openstack/virtualbmc/+/883977 | 21:42 |
| opendevreview | Iury Gregory Melo Ferreira proposed openstack/ironic master: FW Object structure https://review.opendev.org/c/openstack/ironic/+/883093 | 21:46 |
| TheJulia | and... looks like they reverted the revert to the revert on the cirros version | 22:06 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Explicitly pin CIRROS_VERSION https://review.opendev.org/c/openstack/ironic/+/883983 | 22:11 |
| opendevreview | Merged openstack/ironic stable/2023.1: [stable-only] [CI] BFV, RBAC jobs marked non-voting https://review.opendev.org/c/openstack/ironic/+/883697 | 22:41 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!