| opendevreview | Abhishek Kekane proposed openstack/glance master: Fix test_scrubber_restore_image intermittent failure https://review.opendev.org/c/openstack/glance/+/950197 | 06:24 |
|---|---|---|
| opendevreview | Abhishek Kekane proposed openstack/glance master: Remove leftover of scrubber from functional tests https://review.opendev.org/c/openstack/glance/+/950278 | 06:24 |
| opendevreview | Abhishek Kekane proposed openstack/glance master: Remove leftover of scrubber from functional tests https://review.opendev.org/c/openstack/glance/+/950278 | 14:32 |
| opendevreview | Abhishek Kekane proposed openstack/glance master: Fix test_scrubber_restore_image intermittent failure https://review.opendev.org/c/openstack/glance/+/950197 | 14:32 |
| opendevreview | Abhishek Kekane proposed openstack/glance master: Fix test_scrubber_restore_image intermittent failure https://review.opendev.org/c/openstack/glance/+/950197 | 14:43 |
| abhishekk | dansmith: o/ | 16:15 |
| dansmith | abhishekk: hey, I'm in the middle of something else but talk here and I'll respond when I can.. | 16:18 |
| abhishekk | sure!! | 16:19 |
| abhishekk | While deleting the image, RBD store raises InUseByStore from https://github.com/openstack/glance_store/blob/master/glance_store/_drivers/rbd.py#L487 but actually it deletes the data from the backend | 16:22 |
| abhishekk | What I am suggesting is instead of directly raising InUseByStore, should we check that image is present in RBD if yes then return the exception else, log something and let it proceed so that glance can mark the image as deleted? | 16:23 |
| abhishekk | Instead of raising conflict response | 16:23 |
| dansmith | it raises inuse but actually deletes anyway? | 16:24 |
| abhishekk | yes | 16:27 |
| abhishekk | that is the actual issue we reported | 16:27 |
| abhishekk | https://bugs.launchpad.net/glance/+bug/2045769/comments/1 | 16:28 |
| dansmith | okay that bug is from 2023 so I thought it wasn't even related to this | 16:30 |
| dansmith | but still, that seems to be implying that the behavior of it being deleted anyway is wrong | 16:31 |
| dansmith | however, this bug makes a good point that we *already* have a case where delete will fail with 409 because the image is being downloaded, right? | 16:32 |
| abhishekk | yes | 16:33 |
| abhishekk | the next download will say Image has no data | 16:33 |
| abhishekk | next dowmload of same image | 16:33 |
| dansmith | okay so hang on | 16:33 |
| dansmith | first off, that bug is bad because it doesn't delete the image (in glance) but deletes the data | 16:34 |
| dansmith | second, librbd is raising rbd.ImageBusy but actually deletes the data anyway right? | 16:34 |
| abhishekk | Yes, looks like that as per the finding by pranali | 16:35 |
| dansmith | so is your plan to delete, and if ImageBusy, check in rbd again to see if it actually got deleted and if not, avoid the raise of InUseByStore because it actually was deleted? | 16:35 |
| abhishekk | the logs suggests the same as well | 16:35 |
| abhishekk | yes, that is the safe way IMO to avoid any regression | 16:36 |
| dansmith | what was the pre-octopus behavior .. rbd raises ImageBusy and did _not_ delete the data? | 16:36 |
| abhishekk | image gets deleted successfully without InUseByStore exception and download interrupts as usual saying hash is not correct. | 16:37 |
| abhishekk | So it was deleting the data and glance download assumes that it has downloaded the data it has and during hash comparison it realizes that image is invalid as hash is not matching | 16:38 |
| dansmith | no, | 16:39 |
| dansmith | I mean before the 2023 bug that she reported about rbd deleting the image _and_ raising busy, what was the behavior? did it raise busy and not delete the data, or delete the data and not raise busy? | 16:39 |
| abhishekk | delete the data and not raise busy with ceph octopus version | 16:41 |
| dansmith | okay | 16:43 |
| abhishekk | I think when we found this issue we were using quincy/pacific version, at that time I suggested her to check with octopus version and we didn't encounterted this issue with octopus as it was deleting the data and not raising the Image busy | 16:43 |
| dansmith | my point is .. it seems like we need to know what the expected behavior is from ceph (i.e. from the ceph team) | 16:43 |
| dansmith | I suppose if you delete, and ignore busy if the image is gone after then that will not break regardless of if ceph changes behavior again, | 16:44 |
| dansmith | so that is *probably* okay, but it seems like we're really covering up some weirdness here | 16:44 |
| dansmith | rosmaita: ^ | 16:44 |
| abhishekk | right, | 16:44 |
| abhishekk | I don't have any update on whether she has discussed this with ceph team at that time or not | 16:45 |
| abhishekk | At that time, we have added wait logic in tempest to complete the hash calculation and moved ahead | 16:45 |
| abhishekk | this is the old glance weekly meeting discussion | 16:48 |
| abhishekk | https://meetings.opendev.org/meetings/glance/2023/glance.2023-12-14-14.00.log.html#l-25 | 16:48 |
| abhishekk | https://github.com/ceph/ceph/commit/cfae3f79bd0513e2753b0deb8c2624ab07cf2d1b | 16:49 |
| abhishekk | This is next discussion of the bug, https://meetings.opendev.org/meetings/glance/2023/glance.2023-12-07-14.00.log.html | 16:52 |
| opendevreview | Abhishek Kekane proposed openstack/glance master: Revert "Deprecate glance scrubber" https://review.opendev.org/c/openstack/glance/+/950340 | 17:15 |
| opendevreview | Cyril Roelandt proposed openstack/glance master: Remove the sqlite cache driver https://review.opendev.org/c/openstack/glance/+/930027 | 19:00 |
| opendevreview | Cyril Roelandt proposed openstack/glance master: Revert "Deprecate glance scrubber" https://review.opendev.org/c/openstack/glance/+/950340 | 19:22 |
| opendevreview | Merged openstack/glance master: Remove leftover of scrubber from functional tests https://review.opendev.org/c/openstack/glance/+/950278 | 19:53 |
| opendevreview | Merged openstack/glance master: Fix test_scrubber_restore_image intermittent failure https://review.opendev.org/c/openstack/glance/+/950197 | 20:03 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!