| gmaan | gibi: sean-k-mooney : this is fix for the live migration task leaking https://review.opendev.org/c/openstack/nova/+/1003386 | 02:02 |
|---|---|---|
| *** bauzas4 is now known as bauzas | 02:08 | |
| opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Support cold migration with virtiofs shares https://review.opendev.org/c/openstack/nova/+/989633 | 05:19 |
| opendevreview | Adam McArthur proposed openstack/nova master: Use safe allocation deletion on instance delete https://review.opendev.org/c/openstack/nova/+/1003208 | 06:12 |
| *** zseguin is now known as Guest16914 | 06:31 | |
| *** bauzas0 is now known as bauzas | 07:43 | |
| *** erlon8 is now known as erlon | 09:43 | |
| gibi | can I get an core eye on https://review.opendev.org/c/openstack/nova/+/1003245 it is a functional test stabilization | 10:02 |
| sean-k-mooney | so the intet is to set it to a largere value later but we are settign it to 10 for now so that if we do leak it does not cause the run to fail | 10:12 |
| sean-k-mooney | i see in https://review.opendev.org/c/openstack/nova/+/1003266/1/nova/tests/functional/regressions/test_bug_1888395.py your bumping it to 300 and hten claping it to 10 on the tests that have know issues | 10:15 |
| sean-k-mooney | any that are leaking a thread will timeout and you can clamp them until its fixed | 10:16 |
| sean-k-mooney | gibi: we can technially start deleting evnetlet in like a weak to eliminate those leaks :) | 10:17 |
| sean-k-mooney | gibi: by the way did you notice that the threading functional test now run in half the time of the eventlet ones | 10:19 |
| sean-k-mooney | oh nope | 10:20 |
| sean-k-mooney | thats the non funtionla treadign tests | 10:20 |
| gibi | sean-k-mooney: we cannot delete them until post 2027.1 | 10:21 |
| sean-k-mooney | we technially can | 10:22 |
| sean-k-mooney | but i know we are givign a very long tail in nova | 10:22 |
| gibi | sc-tmp-lighbits-rhoso-certification | 10:22 |
| gibi | https://governance.openstack.org/tc/goals/selected/remove-eventlet.html#completion-criteria | 10:22 |
| gibi | u(2027.1) Deprecate the Eventlet support. To have a SLURP target release supporting both the old Eventlet and any new concurrency mode, this release keeps the Eventlet support. | 10:22 |
| sean-k-mooney | those are the letes tg we can do those | 10:23 |
| sean-k-mooney | not the first time for each milestone | 10:23 |
| sean-k-mooney | several service just remove eventlet with no opertunity to have a singel slurp release that ran both | 10:23 |
| gibi | but I think what is written in the goal is there because we wanted to do it that way :) | 10:24 |
| sean-k-mooney | we did im just saying if it continue to be a maintiance isseu we can change that planb | 10:24 |
| sean-k-mooney | on the ohter hand | 10:24 |
| sean-k-mooney | if the eventlet test find leaks that we dont see in threaded | 10:25 |
| sean-k-mooney | we probably shoudl fix those anyway | 10:25 |
| gibi | yeah I haven't looked into the eventlet hangs in that patch | 10:25 |
| gibi | but I will | 10:25 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Stabilize _test_interface_attach_and_detach https://review.opendev.org/c/openstack/nova/+/1003452 | 10:47 |
| gibi | sean-k-mooney: bauzas: ^^ another functional stabilization patch | 10:47 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: [functional]Do not patch req-id generation https://review.opendev.org/c/openstack/nova/+/1003024 | 13:21 |
| gibi | cores this is another stabilization ^^ | 13:21 |
| stephenfin | sean-k-mooney: did you already bring up https://review.opendev.org/c/openstack/releases/+/1003257? | 14:16 |
| stephenfin | https://review.opendev.org/c/openstack/releases/+/1003311 is blocked by that | 14:17 |
| sean-k-mooney | sorry no Uggla bauzas ^ we need to release os-vif again | 14:32 |
| sean-k-mooney | elodilles: ^ fyi | 14:33 |
| Uggla | sean-k-mooney ok I have seen the patches | 14:33 |
| elodilles | sean-k-mooney: nova team needs to agree on FFE and an RFE needs to be requested from requirements team, otherwise LGTM | 14:41 |
| sean-k-mooney | the config bug is a reguress intoduced in this cycle os i think it would qualify for a FFE adn enve after RC1 but idealy we do it before we do it | 14:48 |
| gibi | I'm OK from nova perspective to accept this os_vif fix as FFE | 14:52 |
| gibi | https://review.opendev.org/c/openstack/nova/+/1003266/1/nova/tests/fixtures/conf.py#73 | 14:52 |
| gibi | this is being the fix | 14:52 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: [DNM]reproduce "database is locked" error https://review.opendev.org/c/openstack/nova/+/1003554 | 15:37 |
| gibi | sean-k-mooney: ^^ finally after 2days and many deep rabbit holes this reproduces the DB locking issue pretty regularly to me locally | 15:38 |
| gibi | there is something called checkpointing the WAL | 15:39 |
| gibi | that happens after the last DB connection is closed | 15:40 |
| gibi | happens outside of the db context manager | 15:40 |
| gibi | and sqlite takes a DB write lock for it | 15:40 |
| gibi | as we constantly open and close DB connections there is plenty of, very short, window when there is a last-connection-closed situation | 15:41 |
| gibi | and if we are unlucky the next connection is a write transaction that hits within the window of the WAL checkpointing and seeing the DB as locked | 15:41 |
| gmaan | gibi: https://review.opendev.org/c/openstack/nova/+/1003386 is green to fix the leaking task. can you rebase your manager_shutdown_timeout=300 change to this and we can see what all are fixed with this | 15:42 |
| gibi | gmaan: on my list still for today | 15:42 |
| gmaan | thanks | 15:42 |
| melwitt | so that per something lock change did not help huh? | 15:42 |
| gibi | melwitt: it helps | 15:44 |
| gibi | we need to serialize writes | 15:44 |
| gibi | but | 15:44 |
| melwitt | oh ok | 15:45 |
| gibi | there is this secondary case when sqlite takes a DB write lock | 15:45 |
| gibi | outside of a write transaction | 15:45 |
| gibi | the case being, the last connection to the DB is closed. | 15:45 |
| gibi | so the immedate fix is, do not allow such situation to happen :) | 15:45 |
| gibi | i.e. keep a connection open during the test | 15:46 |
| melwitt | ah, yeah. that's how the in-memory db worked so it didn't come up in the past | 15:46 |
| gibi | yes in memory DB used a StaticPool to have only a single connection to a DB all the time | 15:46 |
| melwitt | right | 15:47 |
| gibi | the file based uses a NullPool | 15:47 |
| gibi | meaning one connection per transaction | 15:47 |
| gibi | the StaticPool is not good for us in threading as that allows two parallel transaction to overlap on that single global connection | 15:48 |
| gibi | but NullPool + WAL triggers this edge case of checkpointing the WAL | 15:48 |
| gibi | my learning from the last two days: DB engines are complicated beasts | 15:49 |
| melwitt | yes they are :( in my limited experience | 15:49 |
| sean-k-mooney | gibi: nice find | 15:56 |
| sean-k-mooney | the checkpointing this vagly rings a bell but only in that i think i came across that at some poihnt | 15:57 |
| sean-k-mooney | but not that it happend on connection close/open | 15:57 |
| sean-k-mooney | keeping the conenction open during the test is proably fine if we can do that cleanly | 15:58 |
| gibi | I think this is part of the maintenance of the WAL itself | 15:58 |
| gibi | > cleanly | 15:58 |
| gibi | working on it :) | 15:58 |
| sean-k-mooney | i was thinking of some kind fo per test singelton that we setup and tear down | 15:59 |
| sean-k-mooney | the impornat thing is you have a repoducer that does not need an entire ci run now | 16:00 |
| gibi | yepp by running with timeout=0 in the DB url (which is busy_timeout for SQLite) I can trigger the issue locally in my machine pretty frequently | 16:03 |
| gibi | so I can test a fiz | 16:03 |
| gibi | fix | 16:03 |
| sean-k-mooney | if you push one ill let it run on my mac min for an hour or two in a loop while we wait on ci | 16:04 |
| gibi | yeah working on the cleaniness | 16:05 |
| gibi | of the fix | 16:05 |
| gibi | but I saw having a connection open removing the issue locally | 16:05 |
| gibi | closing the connection is a pain as the place where we need to open it is before the fixture setUp so I cannot register a function via addCleanup. Probably I need to create just an additional fixture for the connection and hope that opening the extra connection a bit after the DB creation is still OK to fix this | 16:09 |
| sean-k-mooney | you can just directlly add it to tearDown | 16:10 |
| sean-k-mooney | if needed | 16:11 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: [func]Avoid WAL checkpointing https://review.opendev.org/c/openstack/nova/+/1003586 | 17:46 |
| gibi | sean-k-mooney: ^^ | 17:46 |
| sean-k-mooney | cool, the DNM hung when i ran it so ill start running that | 17:47 |
| sean-k-mooney | do you need to handel oterh escptions form the opertunistic test fixture | 17:48 |
| sean-k-mooney | and or only do this when we are uing squlight | 17:49 |
| sean-k-mooney | *sqlite | 17:49 |
| sean-k-mooney | i guess no | 17:49 |
| sean-k-mooney | because you rusign it from _bind_database_fixture_to_file_wal_sqlite | 17:49 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: [func test]Catch hanging task at graceful shutdown https://review.opendev.org/c/openstack/nova/+/1003266 | 17:50 |
| gibi | yepp it is sqlite specific | 17:50 |
| gibi | gmaan: cherry picked https://review.opendev.org/c/openstack/nova/+/1003266 top of you fix and removed the TODOs, I will check back on the result tomorrow | 17:51 |
| gibi | aaand I drop for today o/ | 17:51 |
| sean-k-mooney | o/ | 17:51 |
| gmaan | gibi: thanks and o/ | 17:56 |
| sean-k-mooney | functional-py313-threading: OK (573.29=setup[8.02]+cmd[0.01,564.34,0.93] seconds) | 17:59 |
| sean-k-mooney | ok im going to run that with --until failure for an hour or so | 18:00 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: [func test]Catch hanging task at graceful shutdown https://review.opendev.org/c/openstack/nova/+/1003266 | 18:49 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!