| *** chuanm4 is now known as chuanm | 02:19 | |
| whoami-rajat | #startmeeting cinder | 14:00 |
|---|---|---|
| opendevmeet | Meeting started Wed Jun 28 14:00:04 2023 UTC and is due to finish in 60 minutes. The chair is whoami-rajat. Information about MeetBot at http://wiki.debian.org/MeetBot. | 14:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 14:00 |
| opendevmeet | The meeting name has been set to 'cinder' | 14:00 |
| whoami-rajat | #topic roll call | 14:00 |
| happystacker | Hello all | 14:00 |
| eharney | hi | 14:00 |
| rosmaita | o/ | 14:00 |
| senrique | hi | 14:00 |
| tosky | o/ | 14:00 |
| whoami-rajat | #link https://etherpad.opendev.org/p/cinder-bobcat-meetings | 14:01 |
| whoami-rajat | guess who forgot this was the last wednesday of the month? | 14:01 |
| Tony_Saad | 0/ | 14:01 |
| * whoami-rajat facepalms | 14:01 | |
| MatheusAndrade[m] | o/ | 14:01 |
| jungleboyj | o/ | 14:02 |
| thiagoalvoravel | o/ | 14:02 |
| felipe_rodrigues | o/ | 14:02 |
| luizsantos[m] | o/ | 14:03 |
| whoami-rajat | good turnout today | 14:04 |
| whoami-rajat | let's get started | 14:04 |
| whoami-rajat | #topic announcements | 14:04 |
| whoami-rajat | Milestone-2 (06 July) | 14:05 |
| whoami-rajat | we have the Volume and target Driver freeze coming up next week | 14:05 |
| whoami-rajat | #link https://etherpad.opendev.org/p/cinder-2023-2-bobcat-drivers | 14:05 |
| whoami-rajat | I have created this etherpad to track the open drivers | 14:05 |
| whoami-rajat | currently there are only 2, Yadro and Lustre | 14:05 |
| whoami-rajat | if there are any other drivers which i missed, please add it to the list | 14:06 |
| helenadantas[m] | o/ | 14:06 |
| whoami-rajat | others, please review the drivers | 14:06 |
| caiquemello[m] | o/ | 14:06 |
| whoami-rajat | next, SQLAlchemy 2.0 resolution | 14:07 |
| whoami-rajat | #link https://review.opendev.org/c/openstack/governance/+/887083 | 14:07 |
| whoami-rajat | JayF proposed a governance patch to provide details about migration to SQLAlchemy 2.0 | 14:07 |
| whoami-rajat | basically they are planning to bump the requirements to sqlalchemy2.0 early in 2024.1 cycle | 14:07 |
| whoami-rajat | to get an idea of which projects are compatible with it and which are not | 14:08 |
| whoami-rajat | it is also mentioned that projects need to move away from sqlalchemy-migrate and implement alembic | 14:08 |
| whoami-rajat | that work has already been done in Cinder, thanks to Stephen | 14:08 |
| whoami-rajat | hopefully the requirement bump to sqlalchemy 2.0 won't affect cinder but let's see | 14:08 |
| whoami-rajat | we will have sufficient amount of time to fix the gate | 14:09 |
| whoami-rajat | also the resolution isn't final yet so there might be changes but above points should be constant, as they have been for past few cycles | 14:09 |
| whoami-rajat | next, EM Discussion | 14:09 |
| whoami-rajat | there is no update on this | 14:09 |
| whoami-rajat | was just checking the TC meeting discussion and I guess Kristi or JayF is going to start a ML thread for continuing discussion from PTG | 14:10 |
| whoami-rajat | let's see if we get an update this week | 14:10 |
| whoami-rajat | not sure if we want to hold on our EOL patches till then | 14:10 |
| whoami-rajat | rosmaita, what are your thoughts? ^ | 14:10 |
| rosmaita | not sure, really | 14:11 |
| rosmaita | maybe we could do one more announcement to give packagers a heads-up | 14:12 |
| whoami-rajat | sounds good | 14:12 |
| rosmaita | i think we can separate out the overall EM discussion from what's best for cinder right now | 14:12 |
| whoami-rajat | do we want to use the old thread or create a new one? | 14:12 |
| rosmaita | i think create a new one | 14:12 |
| whoami-rajat | yeah true, not sure how long this discussion going to go | 14:12 |
| rosmaita | so we can get something like "FINAL CALL FOR COMMENTS: " in the subject line | 14:13 |
| whoami-rajat | sure, i will take an action item for this | 14:13 |
| whoami-rajat | and good idea to separate it out from the EM discussion | 14:13 |
| whoami-rajat | #action whoami-rajat to send a mail regarding reminder for cinder branches being EOL | 14:14 |
| rosmaita | sounds good | 14:14 |
| whoami-rajat | thanks! | 14:14 |
| whoami-rajat | that's all the announcements we had for today | 14:14 |
| whoami-rajat | anyone has anything else? | 14:15 |
| whoami-rajat | rosmaita, i did do a summary of your summary of PTG, not sure if you got time to go through it but apart from that, anything from PTG you would like to highlight? | 14:15 |
| rosmaita | i can't think of anything | 14:16 |
| rosmaita | i hope the poor turnout for the PTG was because of bad timing, not lack of interest | 14:16 |
| rosmaita | that is, we already had the 2023.2 PTG virtually a few months ago | 14:16 |
| whoami-rajat | yeah, that could be the same reason why we weren't able to gather much topics for cinder sessions as well | 14:17 |
| rosmaita | so maybe companies didn't think it was a priority to go to this one | 14:17 |
| whoami-rajat | agreed, doesn't seem the project is losing interest, it was just an event at odd time | 14:18 |
| whoami-rajat | ok, guess that's all for announcements | 14:19 |
| whoami-rajat | let's move to topics | 14:19 |
| whoami-rajat | #topic Bug: https://bugs.launchpad.net/cinder/+bug/2002535 | 14:19 |
| whoami-rajat | not sure who added it, there is no IRC nick | 14:19 |
| happystacker | my bad it's me | 14:19 |
| happystacker | so it seems that there is a conversion from qcow2 to raw during the creation from an image | 14:20 |
| whoami-rajat | oh ok, please go ahead | 14:20 |
| happystacker | and the metadata of the volume is not updated | 14:20 |
| happystacker | so it remains qcow2 | 14:20 |
| happystacker | while the block dev is raw | 14:21 |
| happystacker | as a result, when creating an instance the attachment shows qcow2 but it works fine | 14:21 |
| happystacker | until you want to resize it where an error is thrown | 14:21 |
| happystacker | saying that the format mismatch | 14:21 |
| happystacker | So I have proposed a fix: https://review.opendev.org/c/openstack/cinder/+/881549 | 14:22 |
| whoami-rajat | are you using the generic nfs driver or vendor nfs? | 14:22 |
| eharney | is that the right gerrit link? | 14:22 |
| happystacker | I'm using powerstore but it happens in the nfs generic space | 14:23 |
| happystacker | weird | 14:23 |
| happystacker | https://review.opendev.org/c/openstack/cinder/+/887081 | 14:23 |
| happystacker | that's the right one | 14:23 |
| senrique | I reproduced part of the problem of the launchpad bug.. | 14:24 |
| eharney | i think that patch has some problems but i'll review it more closely and comment there | 14:24 |
| happystacker | sure, I'm still learning | 14:24 |
| senrique | When creating a volume from a glance image.. cinder created a raw volume. This happens because the nfs driver first fetches the glance image glance and keeps using the same format. | 14:24 |
| eharney | (it probably autodetects the wrong format when using raw images) | 14:25 |
| senrique | fetch the image to raw* | 14:25 |
| whoami-rajat | happystacker, is this config option enabled in your deployment? nfs_qcow2_volumes | 14:25 |
| whoami-rajat | and are you using a master deployment or an older version? | 14:25 |
| happystacker | Yes, but it seems not to change anything | 14:25 |
| happystacker | master deployment | 14:26 |
| whoami-rajat | hmm, i think we have a problem but need to test it | 14:27 |
| happystacker | let me know if I can help more | 14:28 |
| whoami-rajat | we set the format during driver initialization to 'raw' https://github.com/openstack/cinder/blob/e673bcc368d3a24ec21713adbd83b4ab6cbcae18/cinder/volume/drivers/remotefs.py#L180 | 14:28 |
| whoami-rajat | but if we enable the qcow2 option, we switch the format to 'qcow2' https://github.com/openstack/cinder/blob/e673bcc368d3a24ec21713adbd83b4ab6cbcae18/cinder/volume/drivers/remotefs.py#L326 | 14:29 |
| whoami-rajat | but we will restart the service while doing the config change | 14:29 |
| whoami-rajat | so i guess it should be fine | 14:29 |
| whoami-rajat | anyway, as i said, need to be tested but it is possible that the format value is not assigned correctly in some cases | 14:30 |
| senrique | whoami-rajat, in L530 `image_utils.fetch_to_raw` | 14:30 |
| happystacker | problem happens here: https://github.com/openstack/cinder/blob/e673bcc368d3a24ec21713adbd83b4ab6cbcae18/cinder/volume/drivers/remotefs.py#L530 | 14:30 |
| happystacker | so maybe we just can add a validation step which prevents the conversion if the qcow2 option is enabled? | 14:31 |
| whoami-rajat | i think we always convert the qcow2 image to raw when we want to create a bootable volume from image | 14:32 |
| whoami-rajat | do we support writing a qcow2 image to a qcow2 volume? | 14:32 |
| happystacker | As for now, there is a check which verifies that both format are the same, if it's not the case, it'll update the metadata | 14:32 |
| happystacker | it's always converted yes | 14:32 |
| eharney | we could support qcow2->qcow2, not sure if it tries to currently | 14:32 |
| happystacker | so if the conversion happens, then the metadata should be updated as well right? | 14:33 |
| whoami-rajat | i remember disabling qcow2->qcow2 in glance but not sure what was the reason | 14:33 |
| eharney | it looks like the key is checking the format when copying the image and doing the right thing when the volume is supposed to be a qcow2 format | 14:33 |
| whoami-rajat | yep we don't support it in glance cinder store, let me see if i can find the reason https://github.com/openstack/glance_store/blob/5a81f77bd48e46eac6ab0636f0f52dbceec4e8d3/glance_store/_drivers/cinder/nfs.py#L72 | 14:35 |
| eharney | because the glance store is expecting to deal with cinder volumes that are raw, it doesn't support nfs snaps either | 14:35 |
| eharney | presumably because that restriction is much easier than adding support for all of that | 14:36 |
| happystacker | ok | 14:36 |
| senrique | I'm not sure if possible but maybe we replace fetch_to_raw with `fetch_to_volume_format` https://github.com/openstack/cinder/blob/e673bcc368d3a24ec21713adbd83b4ab6cbcae18/cinder/image/image_utils.py#L822 | 14:36 |
| happystacker | which means we need to keep the conversion | 14:36 |
| happystacker | but update the metadata to reflect the change? | 14:37 |
| happystacker | so that the attachment will be raw based but not qcow2 | 14:37 |
| whoami-rajat | https://github.com/openstack/glance_store/commit/85c7a06687291eba30510d63d3ee8b9e9cb33c5f | 14:37 |
| whoami-rajat | looks like there is some problem with extend volume in case of qcow2->qcow2, as i see written in the commit message | 14:38 |
| whoami-rajat | but don't remember anything else ... | 14:38 |
| eharney | also, we need to support writing images to qcow2 volumes to support nfs encrypted volumes | 14:40 |
| senrique | in that case.. i think updating the metadata looks good | 14:40 |
| senrique | yes, nfs encryption works with qcow2 | 14:40 |
| whoami-rajat | eharney, ack, i think your comments makes sense that we are lacking a lot of support around it so better to just block it | 14:40 |
| whoami-rajat | senrique, i don't exactly know the reason why we call fetch to raw here but we do that in all other reference drivers like lvm, ceph etc | 14:42 |
| whoami-rajat | but again they don't have to deal with qcow2 volumes | 14:43 |
| whoami-rajat | not really sure | 14:43 |
| whoami-rajat | let's continue this discussion in the next event, maybe good for midcycle -2 ? let's see | 14:46 |
| whoami-rajat | we have another topic to discussi | 14:46 |
| whoami-rajat | discuss | 14:46 |
| whoami-rajat | #topic test_rebuild_volume_backed_server failing 100% on ceph job | 14:47 |
| whoami-rajat | senrique, that's you | 14:47 |
| senrique | CI was failing because of that | 14:47 |
| senrique | #link https://bugs.launchpad.net/cinder/+bug/2025096 | 14:47 |
| senrique | but now the "test_rebuild_volume_backed_server" test is skipped for the Ceph job until the problem is fixed. | 14:47 |
| senrique | #link https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/887003 | 14:48 |
| senrique | that's all :) | 14:48 |
| whoami-rajat | senrique, i analyzed the job results yesterday and found the issue | 14:48 |
| whoami-rajat | let me see if it's still in my notes | 14:48 |
| senrique | i think the nfs job is failling with the same issue tho | 14:48 |
| whoami-rajat | currently we have a timeout of 20 seconds per GB on the nova side during reimage | 14:49 |
| whoami-rajat | which we optimistically set for a good deployment | 14:49 |
| whoami-rajat | but our gate jobs are pretty slow with IO and hence takes more time | 14:49 |
| whoami-rajat | the cinder logs tell the truth about the time taken for the operation | 14:50 |
| whoami-rajat | first log: 01:02:49.782746 | 14:50 |
| whoami-rajat | last log: 01:03:23.661182 | 14:50 |
| whoami-rajat | so it's around 34 seconds which is greater than the timeout | 14:50 |
| whoami-rajat | hence we fail on the nova side | 14:50 |
| whoami-rajat | Jun 26 01:03:11.721921 np0034441113 nova-compute[97944]: ERROR oslo_messaging.rpc.server eventlet.timeout.Timeout: 20 seconds | 14:50 |
| whoami-rajat | I didn't know there was a bug report, i will add my analysis to it though dansmith has already proposed patches to fix it | 14:51 |
| whoami-rajat | thanks for linking the bug senrique | 14:51 |
| senrique | thanks whoami-rajat, good report | 14:51 |
| rosmaita | that's a good catch whoami-rajat | 14:52 |
| eharney | i'm currently looking into cinder-tempest-plugin-cbak-ceph failures as well, unclear if those are related | 14:52 |
| senrique | ++ | 14:52 |
| whoami-rajat | sorry got busy updating the bug, will do it later | 14:57 |
| whoami-rajat | let's move to open discussion for 3 minutes | 14:57 |
| whoami-rajat | #topic open discussion | 14:57 |
| whoami-rajat | there is a big list of review requests today | 14:57 |
| whoami-rajat | please take a look when you get some time | 14:58 |
| happystacker | thank you! | 14:58 |
| drencrom | Hi all. I just would like to have some eyes on https://review.opendev.org/c/openstack/cinder/+/868485, its been a while. Thanks | 14:58 |
| rosmaita | drencrom: ack | 15:00 |
| whoami-rajat | we are out of time | 15:00 |
| whoami-rajat | thanks everyone for attending | 15:00 |
| whoami-rajat | #endmeeting | 15:00 |
| opendevmeet | Meeting ended Wed Jun 28 15:00:26 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:00 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/cinder/2023/cinder.2023-06-28-14.00.html | 15:00 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/cinder/2023/cinder.2023-06-28-14.00.txt | 15:00 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/cinder/2023/cinder.2023-06-28-14.00.log.html | 15:00 |
| *** chuanm5 is now known as chuanm | 22:30 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!