| *** mhen_ is now known as mhen | 02:30 | |
| opendevreview | Takashi Kajinami proposed openstack/cinder master: Replace deprecated FormatChecker.cls_checks https://review.opendev.org/c/openstack/cinder/+/935688 | 06:17 |
|---|---|---|
| gokhan | hello folks, We are experiencing an intermittent issue affecting some virtual machines running database workloads in our OpenStack environment. Occasionally, these instances temporarily lose access to their attached volumes, resulting in “disk I/O errors” at the guest OS level. The affected volumes are provided via shared storage using Fibre Channel with multipath enabled. We do not see any related errors in the Nova or Cinder logs—onl | 08:08 |
| gokhan | y the VM’s kernel logs report disk I/O errors. In most cases, the issue is resolved by performing a hard reboot of the instance; however, in some cases the filesystem becomes corrupted and requires manual repair (fsck) afterwards. We are looking for guidance on possible root causes and recommended troubleshooting steps for environments using FC-backed Cinder volumes with multipath. | 08:08 |
| opendevreview | Seunghun Lee proposed openstack/cinder master: rbd: add support flatten volume from image https://review.opendev.org/c/openstack/cinder/+/820375 | 09:08 |
| tkajinam | gokhan, did you also check OS logs in the host ? (for example do you see any multipathd error indicating path failures ?) | 09:21 |
| gokhan | tkajinam, I have checked the host OS logs (journalctl -k and journalctl -u multipathd) around the time of the disconnect events, but I didn't find any critical errors that explicitly indicated a Fibre Channel HBA reset or a persistent multipathd path failure at the Compute Node level. | 10:40 |
| gokhan | When the Database VM drops to read-only mode, the multipath -ll command executed on the Compute Node shows that the paths for that specific Cinder volume have completely disappeared from the output. | 10:40 |
| gokhan | I will try with hw_disk_bus: scsi ve hw_scsi_model: virtio-scsi . ı am using virtio now. | 10:42 |
| gokhan | disk in the vm xml is like: <disk type='block' device='disk'> | 10:43 |
| gokhan | <driver name='qemu' type='raw' cache='none' io='native' discard='unmap'/> | 10:43 |
| gokhan | <source dev='/dev/disk/by-id/dm-uuid-mpath-36eca1d1100d089f211b37d01000060f4' index='2'/> | 10:43 |
| gokhan | <backingStore/> | 10:43 |
| gokhan | <target dev='vdd' bus='virtio'/> | 10:43 |
| gokhan | <serial>05e41e13-9792-430f-94fb-a9d3a3e4ec16</serial> | 10:44 |
| gokhan | <alias name='ua-05e41e13-9792-430f-94fb-a9d3a3e4ec16'/> | 10:44 |
| gokhan | <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/> | 10:44 |
| tkajinam | gokhan, hm. it's wired that multipathd lost track of these paths but logs no path errors | 12:55 |
| tkajinam | gokhan, if hard-reboot fixes the problem I suspect that the problem exists in host side. I may dig into the other host logs around the time, while I also enable debug in nova-compute to see if any suspicious scsi/multipath device operation is executed | 12:57 |
| gokhan | tkajinam, yeah I also suspect that problem exists in hosts side. I will check nova compute with enabling debug mode. thanks :) | 13:03 |
| opendevreview | Alan Bishop proposed openstack/cinder master: WIP: Revert to any snapshot support for drivers https://review.opendev.org/c/openstack/cinder/+/948343 | 18:17 |
| opendevreview | Alan Bishop proposed openstack/cinder master: WIP: Revert to any snapshot support for drivers https://review.opendev.org/c/openstack/cinder/+/948343 | 20:49 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!