| *** erlon4 is now known as erlon | 03:06 | |
| opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Allow rebuild for instances with virtiofs shares https://review.opendev.org/c/openstack/nova/+/999048 | 06:28 |
|---|---|---|
| ralonsoh | sean-k-mooney, please, ping me whenever you have some time (https://bugs.launchpad.net/nova/+bug/2161894/comments/5) | 07:39 |
| opendevreview | Joan Gilabert proposed openstack/nova master: Report OWNER_NOVA trait on resource providers https://review.opendev.org/c/openstack/nova/+/994299 | 07:40 |
| opendevreview | Joan Gilabert proposed openstack/nova master: Add libvirt driver MDEV ARQ support for Cyborg https://review.opendev.org/c/openstack/nova/+/995840 | 07:40 |
| opendevreview | Anton Iacobaeus proposed openstack/nova master: libvirt: Generalize memory encryption config https://review.opendev.org/c/openstack/nova/+/998466 | 09:25 |
| opendevreview | Anton Iacobaeus proposed openstack/nova master: libvirt: Add memory encryption config for TDX https://review.opendev.org/c/openstack/nova/+/998608 | 09:25 |
| opendevreview | Anton Iacobaeus proposed openstack/nova master: libvirt: Launch security Intel TDX https://review.opendev.org/c/openstack/nova/+/999249 | 09:25 |
| *** M00SE3 is now known as M00SE | 10:14 | |
| opendevreview | Kamil Sambor proposed openstack/nova master: Add regression test for BFV evacuate + Cinder timeout https://review.opendev.org/c/openstack/nova/+/999261 | 10:38 |
| *** sambork_ is now known as sambork | 12:16 | |
| *** tdemaret2 is now known as tdemaret | 12:18 | |
| gibi | bauzas: have you seen compile_mdev_samples step hanging in devstack? It seems to me it causing the nova-whitebox job to time out here https://zuul.opendev.org/t/openstack/build/db30a8569196402898d754be9ce85426/log/job-output.txt#33470 | 13:28 |
| gibi | (or I'm not sure how knows more about that step) | 13:28 |
| bauzas | hmmm, nope | 13:28 |
| gibi | so far this is a single data point. I will report a bug if I see it happening multiple times | 13:30 |
| bauzas | gibi: hmmm, when looking at the build logs, I see the samples created correctly | 13:31 |
| bauzas | I can read " | 13:33 |
| bauzas | 2026-07-30 09:14:57.345940 | controller | [3695 Async compile_mdev_samples:43255]: finished compile_mdev_samples with result 0 in 78 seconds | 13:33 |
| bauzas | " | 13:33 |
| gibi | are we looking at the same logs? | 13:35 |
| gibi | 2026-07-30 09:24:02.142618 | compute-host | [3806 Async compile_mdev_samples:29935]: Waiting for completion of compile_mdev_samples running on PID 29935 (2 other jobs running) | 13:35 |
| gibi | 334712026-07-30 09:24:02.182165 | compute-host | [3806 Async compile_mdev_samples:29935]: Signaling child to exit | 13:35 |
| gibi | 334722026-07-30 09:24:03.318037 | compute-host | + inc/async:async_inner:64 : rc=0 | 13:35 |
| gibi | 334732026-07-30 09:24:03.363702 | compute-host | [35342 Async configure_neutron_nova:35342]: finished successfully | 13:35 |
| gibi | 334742026-07-30 12:01:31.411526 | RUN END RESULT_TIMED_OUT: [untrusted : opendev.org/openstack/tempest/playbooks/devstack-tempest.yaml@master] | 13:35 |
| gibi | this shows at 09:24:02 that it starts compiling | 13:35 |
| bauzas | I'm looking at https://zuul.opendev.org/t/openstack/build/db30a8569196402898d754be9ce85426/log/job-output.txt#33470 | 13:35 |
| gibi | so this is not the same compilation that finishes at 09:14 | 13:35 |
| bauzas | ah I see | 13:36 |
| gibi | so on the controller it finished ot 09:14 but on the compute-host it just start at 09:24 | 13:36 |
| bauzas | it was queued | 13:37 |
| bauzas | but not started AFAICS | 13:37 |
| bauzas | with the controller, we see the thread awaken up here https://zuul.opendev.org/t/openstack/build/db30a8569196402898d754be9ce85426/log/job-output.txt#16760 | 13:39 |
| jgilaber | melwitt, sean-k-mooney hi I'm seeing some failures on the functional test from my patch adding the OWNER_NOVA trait that I don't quite understand | 15:04 |
| jgilaber | the tests worked fine in patchset 10, and after what I thought were mostly cosmetic change they are failing apparently randomly https://review.opendev.org/c/openstack/nova/+/994299/10..13 | 15:05 |
| jgilaber | e.g https://zuul.opendev.org/t/openstack/build/46982818a76243dfa64f66aaa34c2ef3 only one test fails but I see no error | 15:05 |
| jgilaber | and in the same buildset the py313 job failed with different tests https://zuul.opendev.org/t/openstack/build/46982818a76243dfa64f66aaa34c2ef3 | 15:06 |
| jgilaber | sorry wrong link, I wanted to paste https://zuul.opendev.org/t/openstack/build/1b76d14e3a7a48278e4d8b84027e546b | 15:07 |
| melwitt | jgilaber: I was just starting to look at that. so far I am suspecting something with database access (perhaps improper transaction context manager nesting or something of that nature). I began to look through the bottom patch and I'm not sure about the location of the minimum service version check and trait addition, my inclination is it shouldn't be in the request spec object there, | 15:07 |
| sean-k-mooney | jgilaber: you removed resting the cach | 15:08 |
| sean-k-mooney | in https://review.opendev.org/c/openstack/nova/+/994299/10..13/nova/tests/functional/integrated_helpers.py | 15:08 |
| melwitt | and there is a possibility that moving that part might change whatever is going wrong. BUT this is just my preliminary thoughts and I need to continue to look at it | 15:08 |
| sean-k-mooney | you were clearign the cash but with the decoreateo you need ot call clear on the funtion | 15:08 |
| jgilaber | right, but even then, that should only skip using the filter | 15:09 |
| jgilaber | the scheduler should not hang because of that, right? | 15:09 |
| sean-k-mooney | jgilaber: also what melwitt said but i woudl try clearing the cash like thi https://github.com/openstack/nova/blob/master/nova/test.py#L319 | 15:10 |
| sean-k-mooney | well the filter is adding som eaddtional db calls | 15:10 |
| sean-k-mooney | to get the min version | 15:10 |
| sean-k-mooney | actully that is a bad exampel that using the run_once decortor not cache | 15:11 |
| sean-k-mooney | https://docs.python.org/3/library/functools.html#:~:text=cache%5Fclear%28%29 | 15:12 |
| sean-k-mooney | but ya we normally clear thsoe | 15:12 |
| sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/test.py#L201 | 15:12 |
| jgilaber | ack, thanks I'll try that | 15:13 |
| sean-k-mooney | so lookign at the fialures more closely | 15:17 |
| jgilaber | melwitt, I'm trying to understand your comment, I'm still not too familiar with most of the nova code | 15:18 |
| sean-k-mooney | the failrue seam to be in in the clean up | 15:18 |
| jgilaber | so you think that I should not add the version check in the request spec object? | 15:18 |
| melwitt | jgilaber: I think it should be elsewhere yes but I will add detailed comments to the review on this once I go through it properly. I just wanted to give you a quick heads up, it is ok to ignore for now | 15:19 |
| sean-k-mooney | melwitt: perhaps in the resouce translation logic | 15:20 |
| melwitt | mainly I wanted to mention that there is a possibility it could be related to the random failures if there is something with the database session nesting, that's all | 15:20 |
| jgilaber | ack, thanks, I'll respin with the cache_clear for now to see if that helps | 15:20 |
| melwitt | sounds good | 15:20 |
| jgilaber | thanks for looking! | 15:21 |
| opendevreview | Joan Gilabert proposed openstack/nova master: Report OWNER_NOVA trait on resource providers https://review.opendev.org/c/openstack/nova/+/994299 | 15:22 |
| opendevreview | Joan Gilabert proposed openstack/nova master: Add libvirt driver MDEV ARQ support for Cyborg https://review.opendev.org/c/openstack/nova/+/995840 | 15:22 |
| sean-k-mooney | so what failing initally is a server show request | 15:22 |
| sean-k-mooney | GET /v2.1/servers/9cb3f749-f071-450f-aa74-122d940bd364" status: 500 | 15:23 |
| sean-k-mooney | but that is just us pooling for the delete | 15:24 |
| sean-k-mooney | 2026-07-30 08:06:21,544 INFO [nova.api.openstack.requestlog] 127.0.0.1 "DELETE /v2.1/servers/9cb3f749-f071-450f-aa74-122d940bd364" status: 204 len: 0 microversion: 2.104 time: 0.071912 | 15:24 |
| sean-k-mooney | to complete | 15:24 |
| melwitt | on one of the HTTP 500 errors I saw that it got stuck in a sql connection code and then timed out | 15:24 |
| melwitt | that's what made me think about database | 15:25 |
| sean-k-mooney | yes | 15:25 |
| sean-k-mooney | its coming form | 15:25 |
| sean-k-mooney | File "/home/zuul/src/opendev.org/openstack/nova/nova/api/openstack/compute/views/servers.py", line 469, in show | 15:25 |
| sean-k-mooney | server['server']['scheduler_hints'] = self._get_scheduler_hints( | 15:25 |
| sean-k-mooney | ^^^^^^^^^^^^^^^^^^^^^^^^^^ | 15:25 |
| sean-k-mooney | File "/home/zuul/src/opendev.org/openstack/nova/nova/api/openstack/compute/views/servers.py", line 256, in _get_scheduler_hints | 15:25 |
| sean-k-mooney | req_spec = objects.RequestSpec.get_by_instance_uuid( | 15:25 |
| sean-k-mooney | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | 15:25 |
| sean-k-mooney | we are pullign the schduelr hinits because of the newer microver which is gettign the requst spec | 15:25 |
| sean-k-mooney | and that call is getting stuck | 15:26 |
| sean-k-mooney | but it might be because of os-brick | 15:26 |
| sean-k-mooney | File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py311/lib/python3.11/site-packages/os_brick/utils.py", line 47, in _sleep | 15:26 |
| sean-k-mooney | _time_sleep(secs) | 15:26 |
| sean-k-mooney | sqlachme is calling _thread_yeild which is time.sleep(0) then os-birck is calling time_sleep(secs) | 15:27 |
| sean-k-mooney | and then we basily end up timign out, so not sure if osbick is the problem or not | 15:28 |
| sean-k-mooney | im looking at test_resize_revert | 15:28 |
| sean-k-mooney | in https://5bf0723027f4ff260a96-e8d1ca7bab9532ed61be7cb4aac9c3f9.ssl.cf5.rackcdn.com/openstack/1b76d14e3a7a48278e4d8b84027e546b/testr_results.html | 15:28 |
| sean-k-mooney | when i see a time out after som eslep calls i wonder if that sleep was a real thread sleep or not | 15:29 |
| sean-k-mooney | espically when we see warnding like | 15:29 |
| sean-k-mooney | 2026-07-30 08:08:57,991 WARNING [oslo.service.backend._common.loopingcall] Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 21.99 sec | 15:29 |
| sean-k-mooney | jgilaber: im going to pull your patch onto my macmin and let it run the funcitonal tests in a loop and see if it repoduces locally | 15:30 |
| jgilaber | thank sean-k-mooney, it's running in CI now with the cache clear | 15:31 |
| melwitt | the job fail I looked at was nova-tox-functional-py311 which afaik is eventlet | 15:31 |
| sean-k-mooney | ya i think it should be eventlet as well | 15:38 |
| sean-k-mooney | melwitt: there are som condering job wehn its calling placement however | 15:40 |
| sean-k-mooney | 2026-07-30 08:07:18,669 DEBUG [placement.requestlog] Starting request: 127.0.0.1 "GET /placement/allocations/4f467034-9ccf-4aee-a7a1-754283e011ce" | 15:40 |
| sean-k-mooney | 2026-07-30 08:08:12,129 WARNING [oslo.service.backend._common.loopingcall] Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 31.64 sec | 15:40 |
| sean-k-mooney | that get completed at | 15:40 |
| sean-k-mooney | 2026-07-30 08:08:24,892 INFO [placement.requestlog] 127.0.0.1 "GET /placement/allocations/4f467034-9ccf-4aee-a7a1-754283e011ce" status: 200 len: 317 microversion: 1.28 | 15:40 |
| sean-k-mooney | but while it was happeing there are waring about timeout and such | 15:41 |
| sean-k-mooney | i.e. https://paste.opendev.org/show/bVL566Wf6dqJEmComlVI/ | 15:41 |
| sean-k-mooney | like that cell itmout looks sus | 15:42 |
| sean-k-mooney | [nova.context] Timed out waiting for response from cell efc8e2e2-e5fb-446c-b0c4-03451f6aedc1. Left the cell worker thread to finish in the background. | 15:42 |
| sean-k-mooney | melwitt: im partly wondering if the placement fixture is blockign nvoa db accesse or something while its servicing a request? | 15:43 |
| sean-k-mooney | melwitt: it defintly looks liek somethign related tot he db for sure | 15:45 |
| melwitt | I'm thinking about the file-based sqlite database but I'm pretty sure that's only used when nova is not monkey patched | 15:45 |
| melwitt | basically wondering if one of the changes to cell database fixtures may be related | 15:46 |
| melwitt | yeah file-based is guarded by "if utils.concurrency_mode_threading():" at least. but I think some of the other fixture changes may have been across the board. don't really have reason to suspect it yet but just something to keep in mind | 15:48 |
| melwitt | (maybe some sort of lock race in the fixture) | 15:56 |
| sean-k-mooney | so orginally i wanted to use filebacked alwasy | 16:02 |
| sean-k-mooney | that what i tought we were doing for the unit tests and its what we are doning in cybrog and watcher | 16:03 |
| sean-k-mooney | but i know that there were some issue with placmeent to get that to work | 16:03 |
| sean-k-mooney | sambork and gibi woudl know more | 16:04 |
| sean-k-mooney | jgilaber: AttributeError: module 'nova.scheduler.request_filter' has no attribute 'cache_clear' | 16:36 |
| sean-k-mooney | im runing the previous version instead to see if it fails locally | 16:40 |
| jgilaber | oh I used the module instead of the function | 16:40 |
| jgilaber | I'll fix it and move it to the base TestCase | 16:40 |
| opendevreview | Joan Gilabert proposed openstack/nova master: Report OWNER_NOVA trait on resource providers https://review.opendev.org/c/openstack/nova/+/994299 | 16:44 |
| opendevreview | Joan Gilabert proposed openstack/nova master: Add libvirt driver MDEV ARQ support for Cyborg https://review.opendev.org/c/openstack/nova/+/995840 | 16:44 |
| gibi | sean-k-mooney: melwitt: so yeah the intention is to have the file backed SQL only for the threading version of the jobs | 16:47 |
| gibi | but in threading mode we do have a global lock today | 16:48 |
| sean-k-mooney | is that global lock hsared btween nova an placmeent? | 16:48 |
| sean-k-mooney | or one per service | 16:48 |
| gibi | and we have a patch to change that to a per DB lock https://review.opendev.org/c/openstack/nova/+/992862 | 16:48 |
| gibi | yeas it is a shared global lock I think | 16:48 |
| sean-k-mooney | ack | 16:49 |
| gibi | but only should be in place for the threading tests | 16:49 |
| sean-k-mooney | so that could expalin some of the warnings and slowness | 16:49 |
| gibi | threading version of the test jobs | 16:49 |
| sean-k-mooney | oh right | 16:49 |
| sean-k-mooney | this was in the eventlet based job | 16:49 |
| gibi | we can land https://review.opendev.org/c/openstack/nova/+/992862 and see if it helps | 16:49 |
| sean-k-mooney | i mena i can go review that now in either case | 16:50 |
| gibi | we could add some tracing to the lock handling and see if that shows that we somehow introduced the lock to the eventlet job as well | 16:50 |
| sean-k-mooney | runing jgilaber's patch locally its not repoduceing for me | 16:50 |
| sean-k-mooney | hum | 16:51 |
| sean-k-mooney | gibi: so this woudl be a good case to use the FiarLockGuarde i wrote | 16:52 |
| sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/utils.py#L1149 | 16:52 |
| gibi | I'm not following how | 16:59 |
| sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/992862/comment/6022582b_82867746/ | 17:00 |
| gibi | but fairlockguard is to handle when multiple locks is being taken in an order | 17:01 |
| gibi | here we only take one lock | 17:01 |
| sean-k-mooney | no | 17:01 |
| sean-k-mooney | that a usecase | 17:01 |
| sean-k-mooney | it was intened to be used anytime you need to aquire the same named lock in diffent context | 17:01 |
| sean-k-mooney | not just multipel locks | 17:01 |
| gibi | I feel like here we have a simpler use case than what the fairlockguard is created for. Sure fairlockguard is also capable of doing this but to do this we don't necessary need the complexity of the fairlockguard | 17:03 |
| sean-k-mooney | right but i wnated ot creat that so we would stop using one of version of it | 17:04 |
| sean-k-mooney | this is reimpelnted a new lock guard | 17:04 |
| sean-k-mooney | we dont have to use it i think it woudl be good to use it when we need soemthing like this in the futre | 17:05 |
| gibi | it was a long day so I might not capable of seeing the whole picture any more but fairlockguard could be used in any place where a single lock is taken, but we don't want to replace all the locks in our code with a fairlockguard as that would be overkill | 17:06 |
| gibi | fairlockguard shines where you have a list of lock to be taken | 17:07 |
| gibi | and you want to do that safely | 17:07 |
| gibi | if you have just a single lock to take at a time then it is overkill in my eyes | 17:07 |
| sean-k-mooney | the bit i dont like is the fact your creating collections.defaultdict(threading.RLock) | 17:08 |
| sean-k-mooney | and locking that collection access | 17:08 |
| sean-k-mooney | that changed it form a singel lock to a set of related locks | 17:08 |
| sean-k-mooney | even if any one invoction of the _locked_scope context manager is only useing one lock | 17:09 |
| gibi | we could monkey patch the lock object into transaction_context_manager._root for each DB then the DatabaseWriteLock does not need to look up the lock in a dict that maps the context manager to a lock | 17:12 |
| sean-k-mooney | we could im not aginst merging the patch as it is | 17:13 |
| gibi | but as we already hooking into sqla here it would be extra complexity to hook this deep into it again at DB creation | 17:13 |
| sean-k-mooney | but we spent a lot of effort addign nesting suprot and other infra in the fairlockgurard and testing to verify it for complex lockign cases like this | 17:13 |
| gibi | but exactly those things are not needed in this case | 17:15 |
| gibi | (I think) | 17:15 |
| gibi | it is not such a complex case | 17:15 |
| sean-k-mooney | you have muyltiple related locks, nesting / rentrancy | 17:16 |
| sean-k-mooney | hence the callout why you needed to RLOCKS | 17:16 |
| gibi | these locks are not related each is for its own DB | 17:16 |
| gibi | they are not guarding the same shared resouce it is one lock - one DB | 17:16 |
| sean-k-mooney | i know | 17:17 |
| sean-k-mooney | but you all out thing like "An RLock is used per database because a single thread may nest writer | 17:17 |
| sean-k-mooney | transactions against the same engine facade." | 17:17 |
| gibi | and that is exactly what an RLock is for :) | 17:18 |
| sean-k-mooney | and its also one of the thing build into the FairlockGurads constaitns | 17:18 |
| sean-k-mooney | ill upgarde to a +2 for now | 17:18 |
| sean-k-mooney | but if this proves unstabel i would prefrer ot adotp the FairLockGuard instead fo theying ot fix this implemetion | 17:19 |
| sean-k-mooney | gibi: does that work for you | 17:19 |
| gibi | if the instability is due to a problem that is solved by the fairlockguard then yes it is good for me | 17:20 |
| sean-k-mooney | patchset 8 was created 21 days ago | 17:22 |
| sean-k-mooney | so that before your shutdown speedup patch landed yes | 17:22 |
| sean-k-mooney | so we woudl execte a new run to also be faster | 17:23 |
| sean-k-mooney | or rather the thereaded and eventet version shoudl be closer now | 17:23 |
| jgilaber | sean-k-mooney, so after clearing the cache the functional tests are green again https://zuul.opendev.org/t/openstack/buildset/6a7d3edc4ae84ba7b33487e037f1acc5 | 17:41 |
| sean-k-mooney | jgilaber: with an without that change i was not able to cause it to fial locally | 17:41 |
| sean-k-mooney | i still have it runingin the backgorund | 17:41 |
| sean-k-mooney | but if ci is happy i guess that is progress | 17:42 |
| jgilaber | yeah, I don't like that I don't know what was wrong there, but good to advance | 17:43 |
| jgilaber | I guess CI harware is slower than our local machines and caused the problem to surface somehow | 17:44 |
| sean-k-mooney | yes that happens | 17:44 |
| sean-k-mooney | you can sort of emulatie that by messign with some of the test teimouts | 17:45 |
| sean-k-mooney | but then its hard to tell if tis a real failure or not | 17:46 |
| gmaan | gibi: you have time to talk about unify the executors get/destroy comment https://review.opendev.org/c/openstack/nova/+/998571/comment/23f68a57_21843d10/ | 17:47 |
| gmaan | if you are done for the day, I can add comment on gerrit and we can discuss there or tomorrow | 17:48 |
| opendevreview | Merged openstack/nova master: tests: Use per-database write locks instead of global lock https://review.opendev.org/c/openstack/nova/+/992862 | 18:01 |
| opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Fix deny_share host locality for NFS https://review.opendev.org/c/openstack/nova/+/998752 | 19:22 |
| opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Support cold migration with virtiofs shares https://review.opendev.org/c/openstack/nova/+/989633 | 19:22 |
| Zhan[m] | Hi friends, recently I found a somewhat tricky bug with live migration: https://bugs.launchpad.net/nova/+bug/2160728. Wondering if I can get some feedbacks on it and I can work on the fix too, thanks! | 19:54 |
| opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Allow rebuild for instances with virtiofs shares https://review.opendev.org/c/openstack/nova/+/999048 | 22:47 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!