| opendevreview | Yian Zong proposed openstack/cinder master: Add Cinder active-active support for Dell PowerFlex driver https://review.opendev.org/c/openstack/cinder/+/899795 | 05:23 |
|---|---|---|
| opendevreview | Cuiye Liu proposed openstack/cinder master: Dell PowerMax : Add the function of Dell PowerMax live migration without a pool name https://review.opendev.org/c/openstack/cinder/+/898188 | 07:50 |
| *** geguileo is now known as Guest5939 | 10:24 | |
| *** elodilles_pto is now known as elodilles | 11:39 | |
| opendevreview | Yian Zong proposed openstack/cinder master: Add Cinder active-active support for Dell PowerFlex driver https://review.opendev.org/c/openstack/cinder/+/899795 | 12:38 |
| opendevreview | Bryan Neumann proposed openstack/cinder master: Dell EMC: PowerMax - Configurable SRDF snapshots https://review.opendev.org/c/openstack/cinder/+/899051 | 15:06 |
| noonedeadpunk | Hey folks. Quick question - should cinder-volumes have access to the DB in the cinder.conf? As I've spotted we do have that, but have concerns if that should be the case or cinder volume should communicate with DB only through scheduler/api alike to nova-computes? | 15:14 |
| noonedeadpunk | It looks like they do communicate with DB through RPC to me, but want to double-check to be sure :) | 15:17 |
| greatgatsby_ | Hello. I'm trying to use the cinder CLI version 9.4.0 against our yoga openstack environment. I keep getting "ERROR: Version 3.70 is not supported by the API. Minimum is 3.0 and maximum is 3.68". However, even if I pass --os-volume-api-version 3.68 I still get the same error. | 15:40 |
| greatgatsby_ | thanks for any suggestions you might have | 15:41 |
| greatgatsby_ | I've also tried the OS_VOLUME_API_VERSION env var, but no joy | 16:00 |
| *** gouthamr_ is now known as gouthamr | 16:09 | |
| rosmaita | greatgatsby_: i think you are getting that error because the API service you are contacting (not the client) only supports up to 3.68 | 16:21 |
| rosmaita | greatgatsby_: 3.68 is the max available in yoga: https://docs.openstack.org/api-ref/block-storage/api_microversion_history.html#maximum-in-yoga | 16:22 |
| rosmaita | greatgatsby_: and i misread your question ... you already know you only want 3.68 | 16:23 |
| noonedeadpunk | basically why I've started asking about cinder-volumes accessing DB, is that I'm trying to find out reasons for cinder catching deadlocks and not re-trying afterwards, ie: https://paste.openstack.org/show/boTkGGE7HZAMxXO8kywD/ | 16:26 |
| noonedeadpunk | I catch quite some of these things after upgrade to 2023.1 | 16:28 |
| greatgatsby_ | rosmaita: really appreciate the response. So am I misunderstanding what --os-volume-api-version does? Does that not make the client talk to the api at that specified version? | 16:28 |
| rosmaita | greatgatsby_: no, you are understanding, i misread your question ... the --os-volume-api-version is supposed to specify the mv you want the client to use | 16:29 |
| greatgatsby_ | maybe I'll try downgrading the client and using that option, perhaps it's something not working in a recent release | 16:30 |
| rosmaita | noonedeadpunk: cinder-volume server absolutely needs to talk to the db | 16:30 |
| noonedeadpunk | mhm, ok, I see. It just felt it talks through rpcc according to these stack traces... | 16:32 |
| greatgatsby_ | rosmaita: I'd still like to get the standalone cinder client working, but in case this is an XY problem, is there an OSC equivalent for `cinder quota-class-update`? | 16:33 |
| rosmaita | greatgatsby_: https://docs.openstack.org/python-openstackclient/2023.1/cli/decoder.html#cinder-cli | 16:37 |
| rosmaita | looks like there is, let me know what happens | 16:37 |
| greatgatsby_ | oh wow, thanks for that link! | 16:38 |
| noonedeadpunk | shouldn't be there some attempts to retry transaction in case of catching deadlocks? | 16:38 |
| noonedeadpunk | Because I feel a bit confused right now on where to dig to be frank... | 16:39 |
| noonedeadpunk | Like disabling deadlock detection in innodb doesn't sound very right, as then transactions may really start piling up | 16:39 |
| noonedeadpunk | and I had some hope that with coordination deadlocks shouldn't really happen.... | 16:40 |
| greatgatsby_ | rosmaita: seems this might be an XY problem anyway, appears the whole "class" thing is deprecated anyway, I'll have to figure out what we want to do here. Again, *really* appreciate the help. | 16:49 |
| rosmaita | greatgatsby_: if you have a few minutes, please file a bug against the cinderclient | 16:49 |
| rosmaita | https://launchpad.net/python-cinderclient | 16:50 |
| greatgatsby_ | rosmaita: yes, I'm downgrading the client until (hopefully) the original command with the api version specified works, and I'll include that in the ticket | 16:50 |
| rosmaita | thanks! | 16:50 |
| greatgatsby_ | with the mapped command, I'm not sure what we want to do yet | 16:50 |
| rosmaita | noonedeadpunk: what's in the cinder-volume log for that API call? you should be able to use the request-id to find the relevant stuff | 17:02 |
| rosmaita | noonedeadpunk: i have to go afk for about an hour, will check back later | 17:02 |
| noonedeadpunk | rosmaita: on cinder-volume it's pretty much same: https://paste.openstack.org/show/bJFQ91WBLW3Zbjshep6p/ | 17:05 |
| rosmaita | dang, i was hoping we'd get more context | 17:11 |
| noonedeadpunk | well | 17:13 |
| noonedeadpunk | I've found an interesting thing now. | 17:13 |
| noonedeadpunk | same request ID but different host: https://paste.openstack.org/show/bIkyBnC577JrtFaNl4iL/ | 17:13 |
| noonedeadpunk | So... they're really racing to execute the request | 17:13 |
| noonedeadpunk | and that basically what's causing the deadlock I assume | 17:14 |
| noonedeadpunk | it's active/active setup with ceph backend fwiw | 17:15 |
| noonedeadpunk | and using zookeeper as coordination driver | 17:15 |
| noonedeadpunk | it's really confusing... As from trace I see it goes through @synchronized.... | 17:20 |
| noonedeadpunk | And in Zookeeper I do have an empty object /cinder/locks/cinder-attachment_update-de2d8ab8-e743-4837-885f-eab22a5f8e31-compute13 | 17:24 |
| noonedeadpunk | fwiw, I have soooooooo much of them - feels like they're never being cleaned up | 17:25 |
| noonedeadpunk | and in zookeeper logs I don't see anything really fishy. actually, it seems normal to have that much data there, since locks are created inside each such object according to the log | 17:30 |
| noonedeadpunk | /cinder/locks/cinder-attachment_update-de2d8ab8-e743-4837-885f-eab22a5f8e31-compute13/0f4064ff7fb54ab680f2262585926c00__lock__0000000006+cinder-19ecf8fb-a517-4382-8ef9-8b513ec2ed37worldanyone | 17:31 |
| noonedeadpunk | I kinda wonder if I have active/active setup misconfigured, as that has happened with me before and I got confused with how to properly set it up in past.... | 17:47 |
| noonedeadpunk | huh, but accroding to trace from cinder-volume - it doesn't go through coordination driver, while I guess it should? | 17:50 |
| greatgatsby_ | rosmaita: https://bugs.launchpad.net/python-cinderclient/+bug/2042863 sorry if it's not the greatest bug report, I'm pretty under the weather right now | 17:51 |
| noonedeadpunk | and another issue I've just spotted is related to cinder-manage db purge.: https://paste.openstack.org/show/bE0lNjzEdZu0CzZz15Xx/ | 18:11 |
| noonedeadpunk | that looks a bit like commit is somehow needed to proceed... But it could be just smth obsolete having old schema.... | 18:11 |
| noonedeadpunk | I guess I'd need to check that volume_admin_metadata is having correct info in it and wasn't messed up manually.... | 18:12 |
| noonedeadpunk | So disregard that for now :) | 18:13 |
| noonedeadpunk | Race condition for cinder-volume with coordination is waaaay more interesting :p | 18:13 |
| *** thelounge5514 is now known as thelounge551 | 19:17 | |
| *** JayF is now known as Guest6061 | 19:55 | |
| *** JasonF is now known as JayF | 19:55 | |
| *** zigo_ is now known as zigo | 19:57 | |
| opendevreview | Bryan Neumann proposed openstack/cinder master: Dell EMC: PowerMax - Configurable SRDF snapshots https://review.opendev.org/c/openstack/cinder/+/899051 | 21:53 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!