14:00:00 <rosmaita> #startmeeting cinder 14:00:01 <openstack> Meeting started Wed Feb 24 14:00:00 2021 UTC and is due to finish in 60 minutes. The chair is rosmaita. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:00:02 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:00:04 <openstack> The meeting name has been set to 'cinder' 14:00:06 <rosmaita> #topic roll call 14:00:13 <whoami-rajat> Hi 14:00:16 <eharney> hi 14:00:23 <almir-okato> hi 14:01:13 <e0ne> hi 14:01:13 <enriquetaso> hi 14:01:21 <tosky> hi 14:01:35 <rosmaita> hello everyone 14:01:49 <zoharm> hi 14:01:53 <rosmaita> #link https://etherpad.openstack.org/p/cinder-wallaby-meetings 14:02:02 <rosmaita> ok, let's get started 14:02:07 <rosmaita> #topic announcements 14:02:21 <rosmaita> assert:supports-api-interoperability tag has been approved for cinder 14:02:31 <rosmaita> #link https://review.opendev.org/c/openstack/governance/+/773684 14:02:35 <rosmaita> hooray! 14:02:46 <rosmaita> upcoming deadlines 14:02:53 <rosmaita> next week: wallaby os-brick release 14:03:08 <rosmaita> week after that: wallaby cinderclient release plus wallaby feature freeze 14:03:23 <rosmaita> so the end of the cycle is coming up fast 14:03:59 <rosmaita> btw, with february being so short, i lost track that this is the last meeting of the month 14:04:28 <rosmaita> which would normally mean a video meeting, but we'll do it next month 14:04:42 * whoami-rajat just realised this is the last meeting of feb 14:04:46 <rosmaita> apologies to anyone really looking forward to a video meeting 14:05:21 <rosmaita> that's all the announcements from me, any else have something to mention? 14:06:03 <rosmaita> i should give a shout out to eharney for his suggestion about the festival of XS reviews we had last week 14:06:10 <rosmaita> i think a good time was had by all 14:06:23 <rosmaita> and we should do it again 14:06:30 <rosmaita> probably soon-ish 14:07:07 <rosmaita> in case i am being to subtle, i am looking for feedback about how often we should have such a festival 14:07:31 <eharney> often, until we have a much smaller review backlog :) 14:07:44 <rosmaita> exactly 14:08:06 <enriquetaso> agree 14:08:26 <rosmaita> maybe right after feature freeze, we can get a jump on bug fix reviews 14:08:46 <rosmaita> and speaking of bugs ... 14:08:57 <rosmaita> #topic Wallaby R-1 Bug Review 14:09:06 <rosmaita> enriquetaso: you have the floor 14:09:15 <enriquetaso> R-7* 14:09:21 <enriquetaso> Hello, we have 5 bugs reported since last week (2 Cinder, 1 Cinder clienta and 2 OsBrick) 14:09:29 <enriquetaso> #link https://etherpad.opendev.org/p/cinder-wallaby-r7-bug-review 14:09:41 <enriquetaso> bug_1: [os-brick] iSCSI+Multipath: Volume attachment hangs if session scanning fails 14:09:49 <enriquetaso> #link https://bugs.launchpad.net/os-brick/+bug/1915678 14:09:50 <openstack> Launchpad bug 1915678 in os-brick "iSCSI+Multipath: Volume attachment hungs if sessiong scanning fails" [High,New] 14:09:58 <enriquetaso> Currently we execute login to iscsi portals and device discovery in multiple threads concurrently.However if some commands like "multiple -m session", the thread can abort immediately without updating any counters like failed_logins or stopped_threads properly, because there are no try-except blocks to catch exceptions.However the main thread keeps waiting until these counters are updated, and this results in stuck volume 14:09:58 <enriquetaso> attachment process. 14:11:03 <enriquetaso> I think some people is already working on this but I couldn't find the gerrit links 14:11:22 <rosmaita> yeah, reminder to people working on bugs: 14:11:32 <rosmaita> the auto-update between gerrit and launchpad is broken 14:11:54 <enriquetaso> The next one is small thing but I think we'd like to fix this before wallaby cinderclient release 14:11:55 <rosmaita> so if you put up a patch to fix a bug, please open the bug in launchpad and add the review link yourself 14:12:07 <tosky> enriquetaso: https://review.opendev.org/c/openstack/os-brick/+/775545 ? 14:12:41 <enriquetaso> tosky++ 14:13:15 <enriquetaso> OK, I think we'd like to fix this before wallaby cinderclient release 14:13:19 <enriquetaso> bug_2: [cinder client] Fetching server version fails to support passing client certificates 14:13:25 <enriquetaso> #link https://bugs.launchpad.net/python-cinderclient/+bug/1915996 14:13:26 <openstack> Launchpad bug 1915996 in python-cinderclient "Fetching server version fails to support passing client certificates" [Medium,New] - Assigned to Sri Harsha mekala (harshayahoo) 14:13:32 <enriquetaso> >> from cinderclient import client as cinder_client 14:13:32 <enriquetaso> >> min_ver, max_ver = cinder_client.get_server_version(cin_url) 14:13:32 <enriquetaso> >> exception_from_error_queue 14:13:32 <enriquetaso> raise exception_type(errors) 14:13:32 <enriquetaso> OpenSSL.SSL.Error: [('SSL routines', 'ssl3_read_bytes', 'sslv3 alert handshake failure')] 14:14:22 <enriquetaso> cinder_client.get_server_version fails 14:15:03 <eharney> this bug assumes that we support strict mTLS from cinderclient which i don't know is a reasonable assumption 14:15:18 <eharney> worth looking into if someone is already working on it 14:15:57 <enriquetaso> It's Assigned to Sri Harsha mekala but again no link 14:16:07 <enriquetaso> I'll try to find it after the meeting 14:16:27 <rosmaita> or maybe tosky can do his gerrit search magic 14:16:33 <enriquetaso> ha 14:16:48 <enriquetaso> bug_3: [docs] Install and configure a storage node in cinder 14:16:55 <enriquetaso> #link https://bugs.launchpad.net/cinder/+bug/1916258 14:16:57 <openstack> Launchpad bug 1916258 in Cinder "[docs] Install and configure a storage node in cinder" [Low,New] 14:17:03 <enriquetaso> Looks like the documentation is outdated. 14:17:04 <enriquetaso> The 'python-keystone' should be replace to 'python3-keystone.However, 14:17:04 <enriquetaso> I can't find the python3-keystone[1] in RDO but it's on Ubuntu 20. 14:17:04 <enriquetaso> [1] https://trunk.rdoproject.org/centos8-master/component/tempest/3e/05/3e05a15d9c4c889aba8c4aad9e24ba8a8a71b7f3_a65d35cc/rpmbuild.log 14:18:13 <eharney> i guess we have an in-tree doc about RDO packages that is not up to date 14:18:57 <enriquetaso> yep 14:18:58 <enriquetaso> bug_4: [cinder] XtremIO does not support ports filtering 14:19:01 <eharney> i'm kind of surprised people are using cinder docs to hand-build an RDO deployment at this point 14:19:28 <enriquetaso> #link https://bugs.launchpad.net/cinder/+bug/1915800 14:19:29 <openstack> Launchpad bug 1915800 in Cinder "XtremIO does not support ports filtering" [Medium,In progress] - Assigned to Ivan Pchelintsev (pcheli) 14:19:33 <enriquetaso> Looks like this new implementation broke the support filtering https://review.opendev.org/c/openstack/cinder/+/775798 14:19:33 <enriquetaso> The bug is already assigned. 14:19:36 <enriquetaso> That's all I have for now :) 14:19:52 <rosmaita> ok, great ... thanks enriquetaso 14:20:40 <rosmaita> i think we'd like to get https://review.opendev.org/c/openstack/os-brick/+/775545 into the release next week if possible 14:20:50 <rosmaita> so please prioritize that review 14:20:54 <eharney> i already -1'd it 14:21:13 <rosmaita> that was fast 14:21:42 <rosmaita> ok, so when eric's revision has happened, please look for that patch 14:21:55 <rosmaita> ok, next topic is another brick fix we need to prioritize 14:22:10 <rosmaita> #topic os-brick nvmeof connector regression in 4.2.0 14:22:22 <rosmaita> let me get some links down for the record 14:22:34 <rosmaita> #link https://bugs.launchpad.net/os-brick/+bug/1916264 14:22:35 <openstack> Launchpad bug 1916264 in os-brick "NVMeOFConnector can't connect volume" [Medium,New] - Assigned to Zohar Mamedov (zoharm) 14:22:37 <rosmaita> ^^ the bug 14:22:53 <rosmaita> announcement to operators 14:22:56 <rosmaita> #link http://lists.openstack.org/pipermail/openstack-discuss/2021-February/020627.html 14:23:03 <rosmaita> announcement to us 14:23:11 <rosmaita> #link http://lists.openstack.org/pipermail/openstack-discuss/2021-February/020628.html 14:23:20 <rosmaita> and zoharm's current patch: 14:23:30 <rosmaita> #link https://review.opendev.org/c/openstack/os-brick/+/777086 14:23:46 <rosmaita> e0ne put up a revert patch, there's some discussion on it about how to fix this 14:24:03 <rosmaita> #link https://review.opendev.org/c/openstack/os-brick/+/776441 14:24:32 <rosmaita> the current patch introduces compatability code into the nvmeof connector to make it backward compatible 14:25:25 <e0ne> it's not an ideal solution, but we're in the end of cycle 14:25:45 <e0ne> splitting into two connectors will require nova change too 14:26:03 <e0ne> that's why we decided to make this fix for now 14:26:20 <rosmaita> i think it makes sense given the situation 14:26:36 <rosmaita> but cinder cores, please look at this patch! 14:26:36 <e0ne> I didn't review the patch yet 14:26:42 <rosmaita> https://review.opendev.org/c/openstack/os-brick/+/777086 14:26:54 <e0ne> but I tested it manually and we restored CI 14:27:09 <zoharm> it is definitely less than ideal, i think ultimately in a further release we could have all drivers use the same driver 14:27:25 <zoharm> same connector* - ie, unify the API 14:27:45 <rosmaita> so e0ne , is mellanox ci responding now? 14:27:55 <rosmaita> or do you still have to trigger it manually? 14:28:27 <e0ne> rosmaita: I'm not sure auto build is enabled now, need to check it 14:28:45 <rosmaita> ok 14:28:48 <e0ne> rosmaita: but we trigger it by adding 'retrigger-mlnx-spdk' comment 14:28:59 <rosmaita> so we can trigger it from zoharm's patch? 14:29:34 <zoharm> i saw it has been triggered there and had a successful run 14:29:39 <rosmaita> ok, cool 14:29:53 <rosmaita> this gives us a check for regressions 14:29:53 <zoharm> https://review.opendev.org/c/openstack/os-brick/+/777086/4#message-10c5858a614dacdc829c8b008056515ca9bd1ef7 14:30:11 <e0ne> rosmaita: it passed for patchset #2, newer patchsets updated commit message and unit-tests only 14:30:27 <rosmaita> also, anyone out there with a FC driver that uses the nvmeof brick connector ... please check your CI 14:30:43 <rosmaita> it may only be SPDK , but it's hard to tell 14:31:51 <rosmaita> zoharm: maybe you can add the retrigger comment whenever you upload a new patch set 14:32:33 <zoharm> sounds good 14:33:09 <rosmaita> ok, so cinder cores ... please prioritize reviewing https://review.opendev.org/c/openstack/os-brick/+/777086/ this week 14:33:24 <rosmaita> we need to get this fixed before next week's brick release 14:33:55 <hemna> I'll take a look 14:34:08 <rosmaita> hemna: ty 14:34:08 <rosmaita> #topic new option or just do it? 14:34:22 <rosmaita> #link https://review.opendev.org/c/openstack/cinder/+/766856 14:34:45 <rosmaita> that patch proposes to support fast-diff for RBD backups 14:34:53 <rosmaita> but it introduces a config option to turn it on 14:35:04 <rosmaita> and it also checks to make sure the backend supports it 14:35:36 <rosmaita> the question came up last week that maybe we should just check to see if fast-diff is supported by the backend, and if so, use it 14:35:48 <rosmaita> without requiring configuration 14:36:09 <rosmaita> i think that makes sense (which is why i am bringing it up!) 14:36:17 <e0ne> or try to use scheduler hints for that 14:36:19 <rosmaita> but i want to get feedback from the wider community 14:36:19 <eharney> i think we should do that 14:36:24 <hemna> is there a specific version that is required ? 14:36:28 <eharney> why scheduler hints..? 14:36:59 <eharney> fast-diff is supported in all versions of ceph that we support, it was added years ago 14:37:18 <e0ne> eharney: it that case is should be safe 14:37:36 <hemna> if that's the case, then I would vote for not needing a conf for it, unless there is a specific reason why the driver shouldn't use it if it's on? 14:37:38 <eharney> it was in infernalis if not before 14:37:41 <e0ne> eharney: how incremental backups will work if we enable fast-diff? 14:38:34 <eharney> the only thing that fast-diff changes is how changes are tracked between images/snaps on the ceph backend, backups would presumably work the same 14:41:14 <e0ne> sounds good to me 14:41:35 <rosmaita> ok, so it sounds like the consensus is to ask the proposer to revise the patch to remove the config option 14:41:52 <eharney> this patch does make me think that we need to check whether we need something similar for rbd volumes, or if we are already doing the ideal thing there 14:42:35 <rosmaita> also, looking at the docstring for the file, it could probably use a revision too 14:42:47 <rosmaita> suggests using at least Ceph Dumpling 14:43:07 <eharney> heh 14:43:30 <rosmaita> eharney: if you don't have time to look, maybe file a bug so we don't forget to check 14:43:44 <rosmaita> #topic Zadara cinder driver new features review 14:43:52 <rosmaita> rratnaka: that's you 14:44:00 <rosmaita> #link https://review.opendev.org/c/openstack/cinder/+/774463 14:44:01 <rratnaka> Hi everyone. 14:44:16 <rratnaka> Great pleasure meeting community folks 14:44:39 <rosmaita> we are happy to have you here! 14:44:47 <rratnaka> As rosmaita shared the review link, I am here to highlight my review requests 14:44:52 <rratnaka> *request 14:45:24 <rratnaka> I have addressed adding new features to Zadara cinder driver and targetted to push to Wallaby release 14:45:48 <rratnaka> I addressed first set of review comments 14:45:50 <rosmaita> when you say "update code layout" , do you mean moving some functions to common ? 14:45:58 <rratnaka> yeah 14:46:43 <rratnaka> Requesting to take a look for review and provide set of comments 14:47:02 <rosmaita> ok, so though it looks like a big change, it's mostly rearranging 14:47:28 <rosmaita> i like adding 500 lines of tests, that's always good to see 14:47:29 <rratnaka> Along with that added few features like multiattach, ipv6 14:47:40 <rratnaka> and other apiś like get_manageable_volumes/snapshots 14:47:59 <rratnaka> yeah. I have added the test cases too 14:48:52 <rosmaita> ok, great. can you add a blueprint to track this: https://blueprints.launchpad.net/cinder/wallaby 14:48:54 <rratnaka> Do we still have bandwidth to add the changes to Wallaby? 14:49:07 <rratnaka> sure rosmaita. I will add a blueprint 14:49:52 <rosmaita> ok, will help us make sure everyone is aware 14:50:27 <rosmaita> to answer the bandwidth question, let's see how the initial reviews go 14:50:49 <rosmaita> anything else? 14:51:08 <rratnaka> yeah. totally agree. 14:51:12 <rosmaita> thanks 14:51:21 <rosmaita> #topic Stable release update 14:51:28 <rratnaka> thatś it from my end. Awaiting the review comments. Thanks 14:51:34 <rosmaita> #link https://etherpad.opendev.org/p/stable-releases-review-tracker-22-02-2021 14:51:39 <rosmaita> whoami-rajat: that's you 14:51:42 <whoami-rajat> thanks rosmaita 14:52:15 <whoami-rajat> so this week there were not many reviews in stable branches 14:52:33 <whoami-rajat> victoria had 1 patch which rosmaita approved 14:52:52 <whoami-rajat> ussuri and train still has 3-4 patches unmerged 14:53:20 <whoami-rajat> I'm not sure if i should go ahead with victoria release after the last change merges or wait for ussuri and train to propose release simultaneously 14:53:25 <whoami-rajat> rosmaita: thoughts ? ^ 14:53:58 <rosmaita> i dont know that there's any advantage to a simultaneous release 14:54:24 <rosmaita> anyone have any observations? 14:55:02 <rosmaita> actually, forget i said that 14:55:11 <rosmaita> or maybe not 14:55:21 * whoami-rajat confused 14:56:04 <enriquetaso> :P 14:56:20 <rosmaita> the key thing is that we never want the situation where an earlier release (like train) contains a fix that isn't in the next release (ussuri) 14:56:31 <rosmaita> so people don't break on upgrade 14:57:13 <whoami-rajat> that shouldn't be happening because we propose backports in order 14:57:15 <rosmaita> so if you release victoria now, and then u & t in a week or so, that should be ik 14:57:18 <rosmaita> *ok 14:57:28 <whoami-rajat> oh, release wise, yes 14:57:41 <rosmaita> you got it 14:58:08 <whoami-rajat> ok, will propose the victoria release this week and wait for ussuri and train to get cleaned up 14:58:45 <rosmaita> ok, everyone needs to concentrate mostly on os-brick anyway, so other than u and t cleanup, we shouldn't have much backport action 14:58:57 <rosmaita> thanks, rajat 14:59:03 <rosmaita> #topic open discussion 14:59:08 <rosmaita> for 45 seconds 14:59:26 <whoami-rajat> also tosky had one question on the meeting pad, not sure if I answered it properly 15:00:10 <rosmaita> i think you answered correctly, we don't know that v backports are relevant to earlier branches until someone proposes them 15:00:15 <rosmaita> ok, make way for horizon! 15:00:18 <rosmaita> thanks everyone 15:00:20 <whoami-rajat> thanks! 15:00:22 <rosmaita> #endmeeting