| gokhan | Hello folks, We are experiencing an intermittent issue affecting some virtual machines running database workloads in our OpenStack environment. Occasionally, these instances temporarily lose access to their attached volumes, resulting in “disk I/O errors” at the guest OS level. The affected volumes are provided via shared storage using Fibre Channel with multipath enabled. We do not see any related errors in the Nova or Cinder logs—onl | 08:15 |
|---|---|---|
| gokhan | y the VM’s kernel logs report disk I/O errors. In most cases, the issue is resolved by performing a hard reboot of the instance; however, in some cases the filesystem becomes corrupted and requires manual repair (fsck) afterwards. We are looking for guidance on possible root causes and recommended troubleshooting steps for environments using FC-backed Cinder volumes with multipath. | 08:15 |
| ykarel | Hi is the kernel panic with test_rebuil_server a known thing? seeing this quite frequently, example https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_034/openstack/034dfb1649dc475c913e56927363ffd2/testr_results.html | 10:03 |
| opendevreview | Taketani Ryo proposed openstack/nova master: mem-enc: nova.virt.libvirt.host: introduce Host.supports_mem_encryption https://review.opendev.org/c/openstack/nova/+/967969 | 10:04 |
| opendevreview | Taketani Ryo proposed openstack/nova master: mem-enc: fix SEV-specific requirement checks for VMs https://review.opendev.org/c/openstack/nova/+/967970 | 10:04 |
| opendevreview | Taketani Ryo proposed openstack/nova master: mem-enc: remove explicit SEV dependency from nova.scheduler.utils https://review.opendev.org/c/openstack/nova/+/967971 | 10:04 |
| opendevreview | Taketani Ryo proposed openstack/nova master: mem-enc: Make RP creation independent of specific encryption models https://review.opendev.org/c/openstack/nova/+/967972 | 10:04 |
| opendevreview | Taketani Ryo proposed openstack/nova master: mem-enc: fix code that assumes SEV in configuring guests https://review.opendev.org/c/openstack/nova/+/967973 | 10:04 |
| opendevreview | Taketani Ryo proposed openstack/nova master: mem-enc: fix requirement checks for mem_encryption guests https://review.opendev.org/c/openstack/nova/+/967974 | 10:04 |
| jkulik | Just stumbled over https://review.opendev.org/c/openstack/nova/+/911374 again. It's a docs correction regarding the default quota driver switch by melwitt. Why didn't we merge that? | 10:10 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Make RBD Tpool usage conditional https://review.opendev.org/c/openstack/nova/+/956089 | 10:50 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Make guestfs' Tpool usage optional https://review.opendev.org/c/openstack/nova/+/965821 | 10:50 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Make image upload tpool usage conditional https://review.opendev.org/c/openstack/nova/+/966505 | 10:50 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Remove tpool_execute as it is unused https://review.opendev.org/c/openstack/nova/+/966552 | 10:50 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Fix ProviderTree copying with threading Lock https://review.opendev.org/c/openstack/nova/+/956091 | 10:50 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: [test]Further categorization of disabled unit tests https://review.opendev.org/c/openstack/nova/+/956092 | 10:50 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Do not fork compute workers in native threading mode https://review.opendev.org/c/openstack/nova/+/965466 | 10:50 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Compute manager to use thread pools selectively https://review.opendev.org/c/openstack/nova/+/966016 | 10:50 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Libvirt event handling without eventlet https://review.opendev.org/c/openstack/nova/+/965949 | 10:50 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Run nova-compute in native threading mode https://review.opendev.org/c/openstack/nova/+/965467 | 10:50 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: WIP:Move eventlet specific libvirt event handling https://review.opendev.org/c/openstack/nova/+/967178 | 10:50 |
| gibi | stephenfin: thanks for the reviews in the eventlet removal series. I fixed / answered them ^^ | 10:52 |
| gibi | stephenfin: wait a bit | 10:53 |
| gibi | I made a rebase mistake :/ | 10:53 |
| stephenfin | gibi: ack. Also, replied on https://review.opendev.org/c/openstack/nova/+/956089/comment/caec94ed_4fdb16bf/ | 10:54 |
| stephenfin | gibi: also, on https://review.opendev.org/c/openstack/nova/+/966505/comment/b6cfa860_58bb6342/, I'm still confused, sorry :) | 10:58 |
| stephenfin | My point was that previously we were calling 'tpool.execute(target)' and now we're calling 'tpool.Proxy(target)' (but only if running in eventlet mode). I'm asking if tpool.execute($thing) is the same as tpool.Proxy($thing) | 10:58 |
| stephenfin | ...and not what the rationale for only wrapping it it something in eventlet mode was. Sorry if I was unclear | 10:59 |
| gibi | what we do now is tpool.Proxy($thing)(args, kwargs) | 10:59 |
| gibi | so we wrap and call the wrapped | 10:59 |
| gibi | and this has the same effect as the execute in the past | 11:00 |
| gibi | because any call on the Proxy object will be done on the native thread of the tpool. Similarly how the tpool.execute executed the call directly on the tpool native thread | 11:00 |
| stephenfin | Sorry for being slow /o\ but we seem to be calling tpool.execute, not tpool.Proxy? | 11:01 |
| stephenfin | a: https://github.com/openstack/nova/blob/master/nova/image/glance.py#L590 | 11:01 |
| stephenfin | b: https://github.com/openstack/nova/blob/master/nova/utils.py#L634 | 11:01 |
| stephenfin | oh, wait, so tpool.Proxy had a __call__ dunder, and tpool.execute is an alias? | 11:02 |
| gibi | yes. In the past we passed a func to the tpool.execute. That run the func on native thread of the tpool | 11:02 |
| gibi | I'm not sure about the impl | 11:02 |
| gibi | but any call on the Proxy object will be run in the tpool | 11:02 |
| gibi | that is the contract | 11:02 |
| stephenfin | okay, I'm still not getting it but you seem to understand my confusion and I trust you've done the right thing. Sorry for the noise! | 11:03 |
| gibi | https://github.com/eventlet/eventlet/blob/8cb20296f3455a1836cdbcbf1d3545666ee7f867/eventlet/tpool.py#L207-L211 | 11:04 |
| gibi | indeed it has a __call__ | 11:04 |
| gibi | so yes on the impl question as well | 11:04 |
| stephenfin | yup, okay, that checks out. Thanks for confirming 🙏 | 11:04 |
| gibi | thanks for the review. I will refine the commit message based on this | 11:05 |
| gibi | (I just have to redo one unintended squash in my rebase) | 11:05 |
| *** tkajinam is now known as Guest31836 | 11:05 | |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Make RBD Tpool usage conditional https://review.opendev.org/c/openstack/nova/+/956089 | 11:13 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Make libvirt Tpool proxying conditional https://review.opendev.org/c/openstack/nova/+/956090 | 11:13 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Make guestfs' Tpool usage optional https://review.opendev.org/c/openstack/nova/+/965821 | 11:13 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Make image upload tpool usage conditional https://review.opendev.org/c/openstack/nova/+/966505 | 11:13 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Remove tpool_execute as it is unused https://review.opendev.org/c/openstack/nova/+/966552 | 11:13 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Fix ProviderTree copying with threading Lock https://review.opendev.org/c/openstack/nova/+/956091 | 11:13 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: [test]Further categorization of disabled unit tests https://review.opendev.org/c/openstack/nova/+/956092 | 11:13 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Do not fork compute workers in native threading mode https://review.opendev.org/c/openstack/nova/+/965466 | 11:13 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Compute manager to use thread pools selectively https://review.opendev.org/c/openstack/nova/+/966016 | 11:13 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Libvirt event handling without eventlet https://review.opendev.org/c/openstack/nova/+/965949 | 11:13 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Run nova-compute in native threading mode https://review.opendev.org/c/openstack/nova/+/965467 | 11:13 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: WIP:Move eventlet specific libvirt event handling https://review.opendev.org/c/openstack/nova/+/967178 | 11:13 |
| gibi | OK I think I managed to restore sanity in the seriese | 11:13 |
| jkulik | hm ... I don't seem to be able to run unit-tests (tox -e py) on nova master anymore unless I specify OS_NOVA_DISABLE_EVENTLET_PATCHING (with either true or false) | 11:20 |
| jkulik | https://paste.opendev.org/show/b8uHX1hkkuwNJnQhni4L/ is the error I get | 11:24 |
| opendevreview | Johannes Kulik proposed openstack/nova master: Enhance performance of get instance groups from DB https://review.opendev.org/c/openstack/nova/+/967977 | 11:29 |
| jkulik | sean-k-mooney: ^ the other commit for server group performance. I'd appreciate a review. | 11:30 |
| sean-k-mooney | jkulik: ah cool and for tox its "tox -e py3" | 11:37 |
| sean-k-mooney | tox -e unit might also work i dont recally if i did that in nova or not | 11:37 |
| jkulik | oh, -e py3 works. thanks. -e py worked until a couple of weeks ago | 11:39 |
| sean-k-mooney | https://github.com/openstack/nova/blob/master/tox.ini#L59 | 11:39 |
| sean-k-mooney | jkulik: -e py was never officlaly supported, that ahs not changed in our config in a long time but if a env starts with py | 11:40 |
| jkulik | yeah, took a look already. not sure why I thought it should be py. env variable is set when using one of these testenvs, yes | 11:40 |
| sean-k-mooney | tox will invent it nd run the command form the base target | 11:40 |
| sean-k-mooney | https://github.com/openstack/nova/blob/master/tox.ini#L54-L57 | 11:40 |
| sean-k-mooney | which happens to be our unit tests | 11:40 |
| sean-k-mooney | my guess is this behvior has changed in a new version of tox | 11:40 |
| sean-k-mooney | we coudl trivially add py | 11:41 |
| jkulik | I can easily adapt :D | 11:41 |
| sean-k-mooney | but ya its been py3 since we added python 3 support i tas py27 i thnk orginally | 11:41 |
| nelljerram | elodilles: it looks like https://review.opendev.org/c/openstack/grenade/+/963716 is ready to proceed now (i.e. for a +W), and hopefully that will then unblock https://review.opendev.org/c/openstack/nova/+/967569 | 11:59 |
| elodilles | nelljerram: yepp-yepp. i'll keep an eye on that patch and recheck the nova patch after that merged :) | 12:30 |
| nelljerram | elodilles: thanks | 12:31 |
| nicolairuckel | sean-k-mooney, I started to look into cold migration. You said that we need to use `copy_image` and I'm wondering where exactly I should do this to only do that for cold migration. My guess would be to add that to the `snapshot` function but I'm not sure about that. | 12:42 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: WIP:Move eventlet specific libvirt event handling https://review.opendev.org/c/openstack/nova/+/967178 | 14:24 |
| opendevreview | Merged openstack/nova master: Init virt driver before use https://review.opendev.org/c/openstack/nova/+/966446 | 14:27 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Libvirt event handling without eventlet https://review.opendev.org/c/openstack/nova/+/965949 | 14:38 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Run nova-compute in native threading mode https://review.opendev.org/c/openstack/nova/+/965467 | 14:38 |
| jkulik | hey, can we merge this docs typo fix? https://review.opendev.org/c/openstack/nova/+/922850 | 14:38 |
| jkulik | seems non-controversial to me that this is wrong in the docs. one line change :) | 14:39 |
| sean-k-mooney | hum | 14:39 |
| sean-k-mooney | well teh only contoveral part is we try not to use _ like that in parmater names | 14:39 |
| sean-k-mooney | let me check tha actual code | 14:39 |
| jkulik | sure. thanks | 14:40 |
| sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L2790-L2792 | 14:40 |
| sean-k-mooney | so it is uesing an underscore | 14:40 |
| jkulik | yes. | 14:41 |
| sean-k-mooney | i woudl personlly prefer to supprot both but we coudl do that in a follow up i guess | 14:41 |
| opendevreview | Merged openstack/nova-specs master: Repropose openapi spec for Gazpacho https://review.opendev.org/c/openstack/nova-specs/+/958869 | 14:42 |
| sean-k-mooney | the i guess we are inconsitent | 14:42 |
| sean-k-mooney | we use '--until-complete' but also --max_rows | 14:42 |
| sean-k-mooney | we shoudl proably normalise al of these to the gnu convention of only using hyphens in a sperate patch | 14:42 |
| jkulik | I agree. do we need a bug to remember this? | 14:42 |
| sean-k-mooney | it would not hurt | 14:43 |
| jkulik | I'll create one | 14:43 |
| opendevreview | Merged openstack/nova-specs master: Repropose remove-os-volumes_boot-api https://review.opendev.org/c/openstack/nova-specs/+/958865 | 14:45 |
| jkulik | https://bugs.launchpad.net/nova/+bug/2132130 hope the description fits the discussion | 14:48 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Make RBD Tpool usage conditional https://review.opendev.org/c/openstack/nova/+/956089 | 15:05 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Make libvirt Tpool proxying conditional https://review.opendev.org/c/openstack/nova/+/956090 | 15:05 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Make guestfs' Tpool usage optional https://review.opendev.org/c/openstack/nova/+/965821 | 15:05 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Make image upload tpool usage conditional https://review.opendev.org/c/openstack/nova/+/966505 | 15:05 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Remove tpool_execute as it is unused https://review.opendev.org/c/openstack/nova/+/966552 | 15:05 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Fix ProviderTree copying with threading Lock https://review.opendev.org/c/openstack/nova/+/956091 | 15:05 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: [test]Further categorization of disabled unit tests https://review.opendev.org/c/openstack/nova/+/956092 | 15:05 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Do not fork compute workers in native threading mode https://review.opendev.org/c/openstack/nova/+/965466 | 15:05 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Compute manager to use thread pools selectively https://review.opendev.org/c/openstack/nova/+/966016 | 15:05 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Libvirt event handling without eventlet https://review.opendev.org/c/openstack/nova/+/965949 | 15:05 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Run nova-compute in native threading mode https://review.opendev.org/c/openstack/nova/+/965467 | 15:05 |
| opendevreview | Merged openstack/nova master: docs: Corrects a typo on a placement audit command https://review.opendev.org/c/openstack/nova/+/922850 | 15:23 |
| opendevreview | Stephen Finucane proposed openstack/nova master: db: Move regex helpers to utils https://review.opendev.org/c/openstack/nova/+/958745 | 18:14 |
| opendevreview | Stephen Finucane proposed openstack/nova master: tests: Clean up flavors tests https://review.opendev.org/c/openstack/nova/+/958746 | 18:14 |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Simplify API version check for flavor description https://review.opendev.org/c/openstack/nova/+/958747 | 18:14 |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Add ability to filter flavors by name https://review.opendev.org/c/openstack/nova/+/958748 | 18:14 |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Remove dead fields from flavors response https://review.opendev.org/c/openstack/nova/+/958749 | 18:14 |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Restrict additional query string arguments https://review.opendev.org/c/openstack/nova/+/958750 | 18:14 |
| opendevreview | Stephen Finucane proposed openstack/nova master: tests: Fix typo https://review.opendev.org/c/openstack/nova/+/958855 | 18:14 |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Add runtime check for query additionalProperties https://review.opendev.org/c/openstack/nova/+/958856 | 18:14 |
| opendevreview | Stephen Finucane proposed openstack/nova master: tests: Add missing test coverage https://review.opendev.org/c/openstack/nova/+/958857 | 18:14 |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Add runtime check for general additionalProperties https://review.opendev.org/c/openstack/nova/+/958858 | 18:19 |
| opendevreview | Phillip Toohill proposed openstack/nova-specs master: WIP: Improve image caching behavior https://review.opendev.org/c/openstack/nova-specs/+/968065 | 19:46 |
| opendevreview | Phillip Toohill proposed openstack/nova-specs master: WIP: Improve image caching behavior https://review.opendev.org/c/openstack/nova-specs/+/968065 | 20:07 |
| JayF | If some folks could make time to review https://review.opendev.org/c/openstack/nova/+/967941 -- it represents a very straightforward fix to a nasty bug for users of the Ironic virt driver. Thanks o/ | 20:25 |
| nicolairuckel | Did anyone have this error before? https://paste.opendev.org/show/b1ZNcgc3WxTDchUGR0Yg/ | 22:18 |
| JayF | nicolairuckel: ensure your dependency versions, specifically oslo.service, match the version required in requirements.txt and upper-constraints.txt for the version you're using. | 22:19 |
| nicolairuckel | Let me check where to do this in DevStack | 22:20 |
| nicolairuckel | I was assuming that would work automatically. | 22:21 |
| nicolairuckel | thanks, that helped! | 22:25 |
| JayF | good luck; fwiw I don't think this channel is usually used for support; you might wanna email openstack-discuss@ next time instead :) | 22:28 |
| nicolairuckel | I thought it makes sense because I thought it was a Nova problem. | 22:29 |
| JayF | Well and I helped you with it :) Just letting you know that may not always be the case, async via mailing list is often more successful. YMMV :) | 22:31 |
| nicolairuckel | Got it! Thanks. :) | 22:39 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!