| gmaan | cinder team, is there any recent change in cinder that it started failing with 401 instead of 403 for non-service user token calling detach_volume ? tempest test for that started failing https://bugs.launchpad.net/cinder/+bug/2153382 | 01:44 |
|---|---|---|
| gmaan | it is blocking tempest gate | 01:44 |
| gmaan | it seems it is blocking cinder gate too | 01:48 |
| gmaan | jbernard: ^^ | 01:48 |
| opendevreview | fuli proposed openstack/cinder-specs master: This specification proposes an enhancement to Cinder volume migration to allow migrating an available or in-use volume together with all of its snapshots between different backends, preserving the lineage of snapshots. https://review.opendev.org/c/openstack/cinder-specs/+/989322 | 03:33 |
| opendevreview | Nilesh Thathagar proposed openstack/cinder master: Dell PowerFlex: Fix release notes for storage username rotation. https://review.opendev.org/c/openstack/cinder/+/986600 | 04:47 |
| opendevreview | Vrinda Dhakad proposed openstack/cinder master: [SVf] Optimize the number of SSH calls for lsfcmap https://review.opendev.org/c/openstack/cinder/+/984309 | 07:02 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/cinder master: [backup] Implement backup container name templating https://review.opendev.org/c/openstack/cinder/+/962909 | 07:58 |
| opendevreview | minwoo seo proposed openstack/cinder master: Add cinder-manage image-cache warm/list subcommand https://review.opendev.org/c/openstack/cinder/+/986740 | 08:37 |
| opendevreview | Rico Lin proposed openstack/cinder master: Add tooz Kubernetes lock support https://review.opendev.org/c/openstack/cinder/+/956528 | 09:23 |
| opendevreview | Eric Harney proposed openstack/cinder master: RBD: Don't use closed ioctxs after errors occur https://review.opendev.org/c/openstack/cinder/+/984655 | 13:26 |
| opendevreview | Eric Harney proposed openstack/cinder master: Replace eventlet sleep calls in volume drivers https://review.opendev.org/c/openstack/cinder/+/950695 | 13:49 |
| agalica | eharney: is your middle initial P? | 13:59 |
| opendevreview | Erlon R. Cruz proposed openstack/cinder master: DNM: Testing CI Failures https://review.opendev.org/c/openstack/cinder/+/989386 | 13:59 |
| agalica | no reason I'm asking, just seems like it'd fit | 13:59 |
| erlon | Are we having the meeting today? | 14:07 |
| agalica | I don't see anything on the list indicating not. maybe Jon slept in :D | 14:10 |
| jbernard | heya everyone, sorry im late | 14:15 |
| jbernard | #startmeeting cinder | 14:15 |
| opendevmeet | Meeting started Wed May 20 14:15:13 2026 UTC and is due to finish in 60 minutes. The chair is jbernard. Information about MeetBot at http://wiki.debian.org/MeetBot. | 14:15 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 14:15 |
| opendevmeet | The meeting name has been set to 'cinder' | 14:15 |
| hvlcchao1 | o/ | 14:16 |
| jbernard | #link https://etherpad.opendev.org/p/cinder-hibiscus-meetings | 14:16 |
| VolodymyrBoiko[m] | o/ | 14:16 |
| jbernard | courtesy ping: jungleboyj rosmaita smcginnis tosky whoami-rajat m5z e0ne geguileo eharney jbernard hemna fabiooliveira yuval tobias-urdin adiare happystacker dosaboy hillpd msaravan sp-bmilanov Luzi sfernand simondodsley zaubea nileshthathagar flelain wizardbit agalica lutimura kaisers aloke_dev Anoop_Shukla erlon | 14:16 |
| jbernard | #topic roll call | 14:16 |
| agalica | o/ | 14:16 |
| jungleboyj | o/ | 14:16 |
| kaisers | o/ | 14:16 |
| Anoop_Shukla | O/ | 14:16 |
| fqi | o/ | 14:16 |
| jbernard | #topic annoucements | 14:19 |
| jbernard | I volunteered to check on the periodic weekly jobs | 14:19 |
| jbernard | #link https://zuul.openstack.org/builds?project=openstack%2Fpython-cinderclient&project=openstack%2Fpython-brick-cinderclient-ext&pipeline=periodic-weekly&limit=14 | 14:19 |
| jbernard | things appear to be looking good | 14:20 |
| jbernard | lotta green there | 14:20 |
| simondodsley | o/ | 14:20 |
| jbernard | I just started looking at https://bugs.launchpad.net/cinder/+bug/2153382, erlon and amit, thanks for jumping on that | 14:21 |
| auniyal | yeah its blocked - it seems keystonemiddleware | 14:22 |
| auniyal | but not 100% sure tbh | 14:22 |
| jbernard | is that a regression in keystone, or do we need to make a change to keep up? | 14:22 |
| auniyal | to me tempest and cinder seems ok - I did not find any related addtion in cinder code | 14:23 |
| erlon | hey guys | 14:24 |
| jbernard | have the keystone folks weighted in? blocked ci is… not good | 14:24 |
| erlon | yeap, trying to test a revert on that patch: https://review.opendev.org/c/openstack/cinder/+/989386 | 14:25 |
| jbernard | erlon: nice | 14:25 |
| erlon | All runs on that job since yesterday are failing: https://zuul.opendev.org/t/openstack/builds?job_name=tempest-slow-py3 | 14:25 |
| auniyal | https://zuul.openstack.org/builds?job_name=tempest-slow-py3&skip=0&limit=20 | 14:25 |
| erlon | I didn't find any change on cinder/tempest from yesterday, but even that Keystone middleware change predates the date the job start failling, but let's see how the revert affects the issue | 14:27 |
| simondodsley | if it is that patch then we should be fixing the failure, not the cause, as it is clearly documented that this parameter has been deprecated and should be True for security reasons | 14:27 |
| jbernard | ok, will watch | 14:28 |
| jbernard | simondodsley: that's what im wondering as well | 14:28 |
| erlon | My patches are just to confirm if that is an issue. If it pin points that has been the cause, it will be easier to address in the proper way | 14:29 |
| jbernard | yep, agree | 14:29 |
| auniyal | yeah revert might now be correct - but should be ok for now - we just need to allow it to retun 401 too, | 14:29 |
| auniyal | *might not | 14:29 |
| erlon | simondodsley: do you have the link for the documentation stating the deprecation? | 14:30 |
| simondodsley | it's in the actual patch - the parameter definition actually states this is deprecated | 14:30 |
| erlon | ah ok | 14:31 |
| jbernard | https://review.opendev.org/c/openstack/keystonemiddleware/+/970607/3/releasenotes/notes/service-token-roles-required-default-true-8a4f52c3d9b7e1f6.yaml | 14:31 |
| erlon | What does it make sense, is that that patch landed on April 10th, but the failure is only happening after yesterday. So, while it makes sense that that is the culprit, the timing is weird | 14:33 |
| erlon | *what doesnt make sense | 14:34 |
| auniyal | so cinder does not use project dir, it uses lib | 14:34 |
| agalica | maybe it's a race condition and the CI machine is under heavy load now | 14:35 |
| agalica | or something similar | 14:35 |
| auniyal | https://pypi.org/project/keystonemiddleware/ | 14:35 |
| jbernard | erlon: yeah, that is odd (the timing) | 14:36 |
| auniyal | latest release is May 13 | 14:36 |
| erlon | I didn't check for anything on Keystone though | 14:37 |
| erlon | Let me see | 14:37 |
| erlon | last merge is from 2026-04-23 | 14:38 |
| agalica | could it be related to this: https://review.opendev.org/c/openstack/neutron/+/989160? | 14:42 |
| auniyal | https://review.opendev.org/c/openstack/requirements/+/988412 | 14:44 |
| jbernard | auniyal: the gate installs keystone from constrains, not from git? so this would explain the timing issue? | 14:45 |
| erlon | auniyal that pinning could explain | 14:45 |
| auniyal | yep | 14:47 |
| auniyal | but I am not really certain I check nova, keystone cinder - no related code is added recently - will have to wait for CI on erlon patch | 14:48 |
| auniyal | *patch(s) | 14:48 |
| jbernard | ok, so erlon's patch will extra confirm, and this it looks like we have some work to do | 14:52 |
| jbernard | s/this/then | 14:52 |
| jbernard | and until then… ci is on fire. great ;) | 14:52 |
| erlon | yes, will that depends on overwrites the keystonedeppendency pinning? | 14:52 |
| jbernard | that's a good question | 14:53 |
| auniyal | so just check the same jo failed on https://review.opendev.org/c/openstack/cinder/+/989386 -- https://zuul.opendev.org/t/openstack/stream/d4ecbac1a73b4fa8b6b0043a95c16197?logfile=console.log | 14:54 |
| jbernard | we have about 10 minutes left, i dont have any more agenda, but feel free to jump in if you have something | 14:54 |
| erlon | but I think the track is getting hot in that direction. From the 2 failling tempest tests, there's a function that calls the cinder service with a 'non-service service token': https://github.com/openstack/tempest/blob/2803b57d6cbdf8281a0a495595c071f49e915042/tempest/scenario/test_server_volume_attachment.py#L42-L63 | 14:55 |
| fqi | Thank you for the comments in the PTG meeting. I have posted a spec patch for volume migration with snapshot. https://review.opendev.org/c/openstack/cinder-specs/+/989322 | 14:56 |
| fqi | Could you please take a look at it when you have a moment? | 14:56 |
| jayaanand | FYI cross pool cache upstream changes https://review.opendev.org/c/openstack/cinder/+/988315. Implementation is in line with PTG discussion. please review. Thank you | 14:58 |
| jbernard | #action review volume migration spec https://review.opendev.org/c/openstack/cinder-specs/+/989322 | 15:04 |
| jbernard | #action cross pool cache patch needs reviews https://review.opendev.org/c/openstack/cinder/+/988315 | 15:05 |
| Anoop_Shukla | @jbernard, @erlon, @whoami-rajat - we have a customer who is looking to early adopt the cross pool cache feature. Would be great if we can get some help with the upstream core changes in patch from @jayaanand. | 15:05 |
| jbernard | Anoop_Shukla: added to the list, we need to get our CI sorted first, and then circle back | 15:06 |
| agalica | the ever growing list of suffocating reviews | 15:07 |
| agalica | :D | 15:07 |
| Anoop_Shukla | Sure thank you. | 15:07 |
| jbernard | ok, last call | 15:09 |
| jbernard | thanks everyone | 15:11 |
| jbernard | #endmeeting | 15:11 |
| opendevmeet | Meeting ended Wed May 20 15:11:27 2026 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:11 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/cinder/2026/cinder.2026-05-20-14.15.html | 15:11 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/cinder/2026/cinder.2026-05-20-14.15.txt | 15:11 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/cinder/2026/cinder.2026-05-20-14.15.log.html | 15:11 |
| erlon | \o | 15:12 |
| opendevreview | Merged openstack/cinder master: Move scheduler-specific code to the scheduler https://review.opendev.org/c/openstack/cinder/+/859317 | 16:53 |
| gmaan | auniyal: hi, you mean default changed from service_token_roles_required to True causing cinder return 401 ? https://review.opendev.org/c/openstack/keystonemiddleware/+/970607 | 16:58 |
| gmaan | auniyal: your analysis is correct, that is root cause. I can see original change from gorka mentioning the same https://review.opendev.org/c/openstack/cinder/+/882835/2//COMMIT_MSG#52 | 17:02 |
| gmaan | till now, none of the jobs set the service_token_roles_required =True so cinder returned 403. let me fix the tempest tests | 17:02 |
| opendevreview | Eric Harney proposed openstack/cinder master: Remove mypy-wrap.sh https://review.opendev.org/c/openstack/cinder/+/985038 | 17:21 |
| opendevreview | Eric Harney proposed openstack/cinder master: mypy: Migrate to module-based configuration (1/2) https://review.opendev.org/c/openstack/cinder/+/985039 | 17:24 |
| opendevreview | Stephen Finucane proposed openstack/cinder master: typing: "Import" objects https://review.opendev.org/c/openstack/cinder/+/985090 | 17:25 |
| erlon | gmaan: I think those two Tempest tests (test_old_versions_reject, test_server_detach_rules) are sending non-service tokens by default. The test expects for [exceptions.Forbidden, exceptions.Conflict], but only for the case when the token are invalid. | 19:56 |
| erlon | https://github.com/openstack/tempest/blob/2803b57d6cbdf8281a0a495595c071f49e915042/tempest/scenario/test_server_volume_attachment.py#L148 | 19:56 |
| gmaan | erlon: yeah, I was fixing the test to handle the service_token_roles_required case https://review.opendev.org/c/openstack/tempest/+/989432 | 19:58 |
| gmaan | and later we found there is anotehr change proposed for that a year back https://review.opendev.org/c/openstack/tempest/+/938766 this is what we are going to merge for fix | 19:58 |
| erlon | gmaan, do you know if the token used in that test is a service token? | 20:20 |
| gmaan | it was not service token. that is what test is for that if non-serviec token can detach volume in cinder | 20:21 |
| erlon | ah got it, so we should always get an api failure. | 20:24 |
| gmaan | yeah, i replied in gerrit. i know it is confusing there in tests saying *valid token* but a *not valid service token* | 20:29 |
| gmaan | or it is confusing from kesytonemiddlware itself :) | 20:29 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!