14:00:13 <rosmaita> #startmeeting cinder 14:00:13 <opendevmeet> Meeting started Wed Mar 23 14:00:13 2022 UTC and is due to finish in 60 minutes. The chair is rosmaita. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:00:13 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:00:13 <opendevmeet> The meeting name has been set to 'cinder' 14:00:19 <rosmaita> #topic roll call 14:00:22 <whoami-rajat> Hi 14:00:31 <simondodsley> Hi 14:00:48 <fabiooliveira> o/ 14:02:00 <eharney> hi 14:02:21 <rosmaita> hello everyone 14:02:23 <jungleboyj> o/ 14:02:30 <rosmaita> #link https://etherpad.opendev.org/p/cinder-yoga-meetings 14:02:36 <rosmaita> let's get started 14:02:42 <rosmaita> #topic announcements 14:02:45 <enriquetaso> hi 14:03:04 <rosmaita> European Summer Time starts 27 March, so this meeting may be 1 hour later for you next week 14:03:16 <rosmaita> and, next week is the final Yoga cycle meeting! 14:03:29 <andrebeltrami[m]> hello 14:03:31 <rosmaita> and, as the last meeting of the month, it will be in video 14:03:49 <rosmaita> usual connection info will be on the agenda etherpad 14:04:02 <rosmaita> the week after that is the Zed (virtual) PTG 14:04:16 <rosmaita> don't forget to register for the PTG if you haven't yet 14:04:23 <rosmaita> info is on the planning etherpad: 14:04:26 <rosmaita> #link https://etherpad.opendev.org/p/zed-ptg-cinder-planning 14:04:47 <rosmaita> there are a lot of topics proposed, so whoami-rajat will have his hands full doing the planning! 14:05:09 <whoami-rajat> isn't this meeting as per UTC timing? I never faced a time change during DST or not 14:05:39 <whoami-rajat> oh or is it just for US Europe region 14:05:41 <rosmaita> whoami-rajat: yes, we keep it in UTC, so it's only a relative change for some people 14:05:53 <whoami-rajat> ack got it 14:06:00 <rosmaita> yeah, i just wanted to give a heads up to Europe region folks 14:06:22 <rosmaita> tbh, i am not sure if European Summer Time is observed everywhere in Europe or not 14:06:23 <whoami-rajat> re: whoami-rajat will have his hands full doing the planning! -- but that shouldn't stop you from adding topics! 14:06:48 <whoami-rajat> ok 14:06:56 <enriquetaso> whoami-rajat++ 14:07:38 <rosmaita> I am thinking that if next week's agenda is light (like today's is), we can do a cycle retrospective at next week's meeting 14:07:49 <rosmaita> or not 14:08:19 <rosmaita> i will think about it, if anyone has an opinion on that, let me know 14:08:28 <whoami-rajat> sounds like a good idea 14:08:31 <rosmaita> ok, final announcement: 14:08:44 <rosmaita> Final Yoga Release Candidate must be cut *tomorrow* 14:08:47 <jungleboyj> Never a bad idea. 14:09:25 <rosmaita> ok, i will put retrospective on the agenda 14:09:47 <rosmaita> i will probably resurrect the retrospective surveys i posted a few cycles ago 14:09:54 <rosmaita> to prime the pump, as it were 14:10:25 <rosmaita> ok, so let's talk about the final yoga release candidate 14:10:33 <rosmaita> #topic final yoga release candidate 14:10:43 <rosmaita> #link https://etherpad.opendev.org/p/cinder-yoga-release-critical-changes 14:11:23 <rosmaita> ok, the first item is the prohibit managing to an encrypted volume type 14:11:34 <rosmaita> jungleboyj: could you renew your +2 on https://review.opendev.org/c/openstack/cinder/+/834478 ? 14:11:53 <rosmaita> i had a typo in the commit message that was corrected and wiped out your +2 14:11:54 <jungleboyj> Looking. 14:12:16 <rosmaita> and we need another stable core to look at https://review.opendev.org/c/openstack/cinder/+/834478 14:12:44 <rosmaita> the cgroups v1 -> v2 work is delayed 14:12:55 <rosmaita> so i will renew the "known issues" about that one 14:13:06 <jungleboyj> +2 from me renewed. 14:13:18 <rosmaita> thanks! 14:13:48 <rosmaita> at this point, i think the throttling for image downloads is unusable because it depends on cgroups v1, which probably is unsupported in most current operating system versions 14:13:54 <eharney> i'll look at 834478 14:14:03 <rosmaita> thanks eric 14:14:27 <rosmaita> so we will need to merge the fix early (like, in the next 2 weeks or so), and cut a new Yoga release 14:14:35 <rosmaita> after backporting it, of course 14:14:50 <rosmaita> and it will have to be backported to Xena where it was promised to be fixed 14:15:30 <rosmaita> next: nfs encryption fix ... will have to be backported when it's ready in master 14:16:24 <rosmaita> the ceph backup-restore issue: there was a request to make sure it was tested, but I'm not sure anyone had the bandwidth to do it 14:16:30 <rosmaita> #link https://review.opendev.org/c/openstack/cinder/+/750782/ 14:17:09 <rosmaita> has anyone had a chance to test it? 14:17:24 <enriquetaso> don't we have a tempest test to try this? 14:18:13 <rosmaita> enriquetaso: i am not sure ... i assume not, because otherwise it would have broken our ceph ci 14:18:28 <eharney> a tempest test that would back up a ceph volume and restore it to iscsi? seems unlikely since we don't usually have much that goes between different backends 14:19:01 <rosmaita> i think eharney is correct, we are passing because we probably test ceph-to-ceph, which doesn't show the problem 14:19:49 <enriquetaso> oh cool 14:19:50 <eharney> also needs testing around encrypted volumes, to be thorough 14:19:53 <rosmaita> i guess it's ok to hold off on this fix since we are already committing to an early Yoga release for cgroups 14:21:02 <rosmaita> ok, we will not include it tomorrow, but will prioritize working on it over the next 2 weeks 14:21:19 <rosmaita> final issue (not on the etherpad yet) 14:21:23 <rosmaita> is geguileo here? 14:21:40 <rosmaita> i see that he filed a big stack of bugs over NVMeOF 14:21:50 <rosmaita> in enriquetaso's bug report this week 14:22:12 <rosmaita> looks like that connector is broken for encryption (among other things) 14:22:32 <rosmaita> so I will need to write a "known issues" note about that 14:22:46 <rosmaita> i will follow up with geguileo 14:22:54 <tosky> we have tests that check the migration between different backends (if configured), but I doubt we have tests that configures the source and destination backend for a backup 14:23:17 <rosmaita> tosky: thanks, i was thinking of your matrix CI job 14:23:19 <tosky> we would need a multibackend job also (I think enriquetaso was working on such a job iirc) 14:23:45 <enriquetaso> I think I proposed something but I'm not sure if it's okay https://review.opendev.org/c/openstack/cinder/+/784815 14:24:18 <tosky> or that multibackend job, yes, it could run also an hypotetical test which performs cross-backend backup and restore 14:24:21 <rosmaita> ok, we have something to look at for this 14:24:35 <rosmaita> thanks enriquetaso and tosky 14:25:02 <rosmaita> i am thinking that we have a lot of combinations that could be tested 14:25:34 <rosmaita> we could run them on periodic jobs in zuul so that we don't consume too many resources on each patch 14:25:51 <rosmaita> or only run them on patches that touch the backup service 14:26:21 <rosmaita> or something ... anyway, my point is that it's worth putting this into tempest so that the testing is available 14:26:42 <enriquetaso> ++ 14:27:11 <whoami-rajat> I think neutron has modified their CI jobs quite a bit to run them on only the code affecting them, although i haven't checked or have a deeper insight on it 14:27:54 <rosmaita> that's good to know 14:28:09 <rosmaita> i guess we can deal with it when people complain that we are doing too much testing 14:28:11 <rosmaita> :) 14:28:54 <whoami-rajat> :) 14:29:17 <rosmaita> ok, so other than https://review.opendev.org/c/openstack/cinder/+/834478 , it looks like the only outstanding stuff for final RC is release notes 14:29:28 <rosmaita> so i will work on those today and ping people to review 14:29:50 <jungleboyj> ++ 14:29:54 <rosmaita> i guess i should also put up a release patch with a -1 so the release team knows we intend to do an RC-2 14:30:06 <rosmaita> i'll do that right after the meeting 14:30:27 <rosmaita> #action rosmaita put up a release patch with a -1 so the release team knows we intend to do an RC-2 14:30:38 <rosmaita> #action rosmaita "known issues" for yoga release 14:31:04 <rosmaita> ok, is there anything else brewing? anyone working on an absolutely critical bug? 14:33:11 <rosmaita> guess ont 14:33:17 <rosmaita> *not 14:33:39 <rosmaita> #topic open discussion 14:33:47 <enriquetaso> I have a bug that it's not critical but I'd like to mention it here 14:33:50 <rosmaita> ok 14:34:03 <enriquetaso> #link https://bugs.launchpad.net/cinder/+bug/1965847 14:34:10 <enriquetaso> During exporting and exporting a backup record the original backup gets deleted. 14:34:17 <rosmaita> i just left a comment on that one 14:34:30 <rosmaita> https://bugs.launchpad.net/cinder/+bug/1965847/comments/1 14:34:56 <rosmaita> that's definitely a bad bug 14:35:11 <enriquetaso> oh, cool 14:35:14 <enriquetaso> thanks rosmaita++ 14:35:20 <rosmaita> don't think it's release critical (though i am happy to consider other opinions) 14:35:27 <rosmaita> fix looks pretty straightforward 14:36:31 <rosmaita> i'm not sure we want to jam through a fix and backport it in the next 24 hours, though 14:36:49 <enriquetaso> i don't think so 14:36:58 <rosmaita> so the question is, should we mention this as a known issue? 14:37:22 <jungleboyj> Yuck. I think it would be a good idea to mention. 14:37:52 <eharney> yeah, seems worthwhile to document 14:38:15 <rosmaita> guess it won't hurt ... though in that situation, presumably if you make the call to import a second time, it should work because now the record is deleted 14:38:43 <rosmaita> ok, i will add it to knows issues 14:39:14 <rosmaita> in the meantime, it's worth looking at where this happens in the code from an engineering perspective 14:39:39 <rosmaita> nested try statements, where the outer one catches Exception ... it is a code smell 14:41:01 <whoami-rajat> I've a query regarding one multiattach test i proposed, I'm doing a dd on the block device from /dev/urandom but when trying to read it, it fails 14:41:10 <whoami-rajat> if anyone has time to take a look on it https://review.opendev.org/c/openstack/cinder-tempest-plugin/+/791921 14:41:56 <rosmaita> interesting 14:42:03 <eharney> fails when reading the data from the block device to verify it? 14:42:28 <whoami-rajat> eharney, yes 14:43:09 <whoami-rajat> UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9c in position 2: invalid start byte 14:43:11 <whoami-rajat> this is the error 14:44:00 <eharney> probably trying to convert bytes to str in a place it shouldn't 14:44:11 <rosmaita> maybe you want to explicitly read in binary mode? 14:44:38 <rosmaita> what eharney said .. make sure you are reading bytes and not strings 14:45:14 <rosmaita> that will definitely be a nice test to have 14:45:16 <whoami-rajat> yeah i think tempest is decoding the data somewhere it shouldn't, i will look for an option to skip it/read bytes 14:45:24 <eharney> i suspect it's the ssh transport 14:45:25 <whoami-rajat> thanks rosmaita eharney for the help 14:45:41 <eharney> maybe instead of shuffling the data blocks over ssh, just pipe the read into sha1sum instead and check that 14:46:00 <eharney> (i'd have to look deeper to see if that's really the issue)\ 14:47:06 <whoami-rajat> i guess you're right, the ssh module in tempest is doing the decode 14:47:06 <whoami-rajat> File "/opt/stack/tempest/tempest/lib/common/ssh.py", line 225, in exec_command 14:47:07 <whoami-rajat> out_data = out_data.decode(encoding) 14:47:44 <whoami-rajat> here's the test failure if you would like to see the whole traceback 14:47:44 <whoami-rajat> https://fe21a6496ae4e9bbcecb-5ff3dddcf26e50a416e8b904201353d9.ssl.cf1.rackcdn.com/791921/7/check/cinder-tempest-plugin-lvm-multiattach/35a57de/testr_results.html 14:48:26 <eharney> yeah it's probably only setup to interact w/ text output and not binary data, so using a checksum seems like a good fix 14:48:43 <whoami-rajat> ack, will try that, thanks 14:49:40 <rosmaita> that was productive! anything else for open discussion? 14:52:46 <rosmaita> easy docs patch for someone to review: https://review.opendev.org/c/openstack/cinder/+/834875 14:56:19 <jungleboyj> rosmaita: I will look at that now. 14:56:47 <rosmaita> thanks! 14:56:57 <rosmaita> ok, guess that's all for today 14:57:05 <rosmaita> thanks everyone! see you next week 14:57:18 <enriquetaso> thanks\o 14:57:36 <whoami-rajat> thanks! 14:57:38 <jungleboyj> Thank you! 14:58:22 <rosmaita> #endmeeting