14:00:00 <rosmaita> #startmeeting cinder
14:00:01 <openstack> Meeting started Wed Feb 24 14:00:00 2021 UTC and is due to finish in 60 minutes.  The chair is rosmaita. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:02 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:00:04 <openstack> The meeting name has been set to 'cinder'
14:00:06 <rosmaita> #topic roll call
14:00:13 <whoami-rajat> Hi
14:00:16 <eharney> hi
14:00:23 <almir-okato> hi
14:01:13 <e0ne> hi
14:01:13 <enriquetaso> hi
14:01:21 <tosky> hi
14:01:35 <rosmaita> hello everyone
14:01:49 <zoharm> hi
14:01:53 <rosmaita> #link https://etherpad.openstack.org/p/cinder-wallaby-meetings
14:02:02 <rosmaita> ok, let's get started
14:02:07 <rosmaita> #topic announcements
14:02:21 <rosmaita> assert:supports-api-interoperability tag has been approved for cinder
14:02:31 <rosmaita> #link https://review.opendev.org/c/openstack/governance/+/773684
14:02:35 <rosmaita> hooray!
14:02:46 <rosmaita> upcoming deadlines
14:02:53 <rosmaita> next week: wallaby os-brick release
14:03:08 <rosmaita> week after that: wallaby cinderclient release plus wallaby feature freeze
14:03:23 <rosmaita> so the end of the cycle is coming up fast
14:03:59 <rosmaita> btw, with february being so short, i lost track that this is the last meeting of the month
14:04:28 <rosmaita> which would normally mean a video meeting, but we'll do it next month
14:04:42 * whoami-rajat just realised this is the last meeting of feb
14:04:46 <rosmaita> apologies to anyone really looking forward to a video meeting
14:05:21 <rosmaita> that's all the announcements from me, any else have something to mention?
14:06:03 <rosmaita> i should give a shout out to eharney for his suggestion about the festival of XS reviews we had last week
14:06:10 <rosmaita> i think a good time was had by all
14:06:23 <rosmaita> and we should do it again
14:06:30 <rosmaita> probably soon-ish
14:07:07 <rosmaita> in case i am being to subtle, i am looking for feedback about how often we should have such a festival
14:07:31 <eharney> often, until we have a much smaller review backlog :)
14:07:44 <rosmaita> exactly
14:08:06 <enriquetaso> agree
14:08:26 <rosmaita> maybe right after feature freeze, we can get a jump on bug fix reviews
14:08:46 <rosmaita> and speaking of bugs ...
14:08:57 <rosmaita> #topic Wallaby R-1 Bug Review
14:09:06 <rosmaita> enriquetaso: you have the floor
14:09:15 <enriquetaso> R-7*
14:09:21 <enriquetaso> Hello, we have 5 bugs reported since last week (2 Cinder, 1 Cinder clienta and 2 OsBrick)
14:09:29 <enriquetaso> #link https://etherpad.opendev.org/p/cinder-wallaby-r7-bug-review
14:09:41 <enriquetaso> bug_1: [os-brick] iSCSI+Multipath: Volume attachment hangs if session scanning fails
14:09:49 <enriquetaso> #link https://bugs.launchpad.net/os-brick/+bug/1915678
14:09:50 <openstack> Launchpad bug 1915678 in os-brick "iSCSI+Multipath: Volume attachment hungs if sessiong scanning fails" [High,New]
14:09:58 <enriquetaso> Currently we execute login to iscsi portals and device discovery in multiple threads concurrently.However if some commands like "multiple -m session", the thread can abort immediately without updating any counters like failed_logins or stopped_threads properly, because there are no try-except blocks to catch  exceptions.However the main thread keeps waiting until these counters are updated, and this results in stuck volume
14:09:58 <enriquetaso> attachment process.
14:11:03 <enriquetaso> I think some people is already working on this but I couldn't find the gerrit links
14:11:22 <rosmaita> yeah, reminder to people working on bugs:
14:11:32 <rosmaita> the auto-update between gerrit and launchpad is broken
14:11:54 <enriquetaso> The next one is small thing but I think we'd like to fix this before  wallaby cinderclient release
14:11:55 <rosmaita> so if you put up a patch to fix a bug, please open the bug in launchpad and add the review link yourself
14:12:07 <tosky> enriquetaso: https://review.opendev.org/c/openstack/os-brick/+/775545 ?
14:12:41 <enriquetaso> tosky++
14:13:15 <enriquetaso> OK, I think we'd like to fix this before  wallaby cinderclient release
14:13:19 <enriquetaso> bug_2: [cinder client]  Fetching server version fails to support passing client certificates
14:13:25 <enriquetaso> #link https://bugs.launchpad.net/python-cinderclient/+bug/1915996
14:13:26 <openstack> Launchpad bug 1915996 in python-cinderclient "Fetching server version fails to support passing client certificates" [Medium,New] - Assigned to Sri Harsha mekala (harshayahoo)
14:13:32 <enriquetaso> >> from cinderclient import client as cinder_client
14:13:32 <enriquetaso> >> min_ver, max_ver = cinder_client.get_server_version(cin_url)
14:13:32 <enriquetaso> >> exception_from_error_queue
14:13:32 <enriquetaso> raise exception_type(errors)
14:13:32 <enriquetaso> OpenSSL.SSL.Error: [('SSL routines', 'ssl3_read_bytes', 'sslv3 alert handshake failure')]
14:14:22 <enriquetaso> cinder_client.get_server_version fails
14:15:03 <eharney> this bug assumes that we support strict mTLS from cinderclient which i don't know is a reasonable assumption
14:15:18 <eharney> worth looking into if someone is already working on it
14:15:57 <enriquetaso> It's Assigned to Sri Harsha mekala but again no link
14:16:07 <enriquetaso> I'll try to find it after the meeting
14:16:27 <rosmaita> or maybe tosky can do his gerrit search magic
14:16:33 <enriquetaso> ha
14:16:48 <enriquetaso> bug_3: [docs] Install and configure a storage node in cinder
14:16:55 <enriquetaso> #link https://bugs.launchpad.net/cinder/+bug/1916258
14:16:57 <openstack> Launchpad bug 1916258 in Cinder "[docs] Install and configure a storage node in cinder" [Low,New]
14:17:03 <enriquetaso> Looks like the documentation is outdated.
14:17:04 <enriquetaso> The 'python-keystone' should be replace to 'python3-keystone.However,
14:17:04 <enriquetaso> I can't find the python3-keystone[1] in RDO but it's on Ubuntu 20.
14:17:04 <enriquetaso> [1] https://trunk.rdoproject.org/centos8-master/component/tempest/3e/05/3e05a15d9c4c889aba8c4aad9e24ba8a8a71b7f3_a65d35cc/rpmbuild.log
14:18:13 <eharney> i guess we have an in-tree doc about RDO packages that is not up to date
14:18:57 <enriquetaso> yep
14:18:58 <enriquetaso> bug_4: [cinder] XtremIO does not support ports filtering
14:19:01 <eharney> i'm kind of surprised people are using cinder docs to hand-build an RDO deployment at this point
14:19:28 <enriquetaso> #link https://bugs.launchpad.net/cinder/+bug/1915800
14:19:29 <openstack> Launchpad bug 1915800 in Cinder "XtremIO does not support ports filtering" [Medium,In progress] - Assigned to Ivan Pchelintsev (pcheli)
14:19:33 <enriquetaso> Looks like this new implementation broke the support filtering https://review.opendev.org/c/openstack/cinder/+/775798
14:19:33 <enriquetaso> The bug is already assigned.
14:19:36 <enriquetaso> That's all I have for now :)
14:19:52 <rosmaita> ok, great ... thanks enriquetaso
14:20:40 <rosmaita> i think we'd like to get https://review.opendev.org/c/openstack/os-brick/+/775545 into the release next week if possible
14:20:50 <rosmaita> so please prioritize that review
14:20:54 <eharney> i already -1'd it
14:21:13 <rosmaita> that was fast
14:21:42 <rosmaita> ok, so when eric's revision has happened, please look for that patch
14:21:55 <rosmaita> ok, next topic is another brick fix we need to prioritize
14:22:10 <rosmaita> #topic os-brick nvmeof connector regression in 4.2.0
14:22:22 <rosmaita> let me get some links down for the record
14:22:34 <rosmaita> #link https://bugs.launchpad.net/os-brick/+bug/1916264
14:22:35 <openstack> Launchpad bug 1916264 in os-brick "NVMeOFConnector can't connect volume" [Medium,New] - Assigned to Zohar Mamedov (zoharm)
14:22:37 <rosmaita> ^^ the bug
14:22:53 <rosmaita> announcement to operators
14:22:56 <rosmaita> #link http://lists.openstack.org/pipermail/openstack-discuss/2021-February/020627.html
14:23:03 <rosmaita> announcement to us
14:23:11 <rosmaita> #link http://lists.openstack.org/pipermail/openstack-discuss/2021-February/020628.html
14:23:20 <rosmaita> and zoharm's current patch:
14:23:30 <rosmaita> #link https://review.opendev.org/c/openstack/os-brick/+/777086
14:23:46 <rosmaita> e0ne put up a revert patch, there's some discussion on it about how to fix this
14:24:03 <rosmaita> #link https://review.opendev.org/c/openstack/os-brick/+/776441
14:24:32 <rosmaita> the current patch introduces compatability code into the nvmeof connector to make it backward compatible
14:25:25 <e0ne> it's not an ideal solution, but we're in the end of cycle
14:25:45 <e0ne> splitting into two connectors will require nova change too
14:26:03 <e0ne> that's why we decided to make this fix for now
14:26:20 <rosmaita> i think it makes sense given the situation
14:26:36 <rosmaita> but cinder cores, please look at this patch!
14:26:36 <e0ne> I didn't review the patch yet
14:26:42 <rosmaita> https://review.opendev.org/c/openstack/os-brick/+/777086
14:26:54 <e0ne> but I tested it manually and we restored CI
14:27:09 <zoharm> it is definitely less than ideal, i think ultimately in a further release we could have all drivers use the same driver
14:27:25 <zoharm> same connector* - ie, unify the API
14:27:45 <rosmaita> so e0ne , is mellanox ci responding now?
14:27:55 <rosmaita> or do you still have to trigger it manually?
14:28:27 <e0ne> rosmaita: I'm not sure auto build is enabled now, need to check it
14:28:45 <rosmaita> ok
14:28:48 <e0ne> rosmaita: but we trigger it by adding 'retrigger-mlnx-spdk' comment
14:28:59 <rosmaita> so we can trigger it from zoharm's patch?
14:29:34 <zoharm> i saw it has been triggered there and had a successful run
14:29:39 <rosmaita> ok, cool
14:29:53 <rosmaita> this gives us a check for regressions
14:29:53 <zoharm> https://review.opendev.org/c/openstack/os-brick/+/777086/4#message-10c5858a614dacdc829c8b008056515ca9bd1ef7
14:30:11 <e0ne> rosmaita: it passed for patchset #2,  newer patchsets updated commit message and unit-tests only
14:30:27 <rosmaita> also, anyone out there with a FC driver that uses the nvmeof brick connector ... please check your CI
14:30:43 <rosmaita> it may only be SPDK , but it's hard to tell
14:31:51 <rosmaita> zoharm: maybe you can add the retrigger comment whenever you upload a new patch set
14:32:33 <zoharm> sounds good
14:33:09 <rosmaita> ok, so cinder cores ... please prioritize reviewing https://review.opendev.org/c/openstack/os-brick/+/777086/ this week
14:33:24 <rosmaita> we need to get this fixed before next week's brick release
14:33:55 <hemna> I'll take a look
14:34:08 <rosmaita> hemna: ty
14:34:08 <rosmaita> #topic new option or just do it?
14:34:22 <rosmaita> #link https://review.opendev.org/c/openstack/cinder/+/766856
14:34:45 <rosmaita> that patch proposes to support fast-diff for RBD backups
14:34:53 <rosmaita> but it introduces a config option to turn it on
14:35:04 <rosmaita> and it also checks to make sure the backend supports it
14:35:36 <rosmaita> the question came up last week that maybe we should just check to see if fast-diff is supported by the backend, and if so, use it
14:35:48 <rosmaita> without requiring configuration
14:36:09 <rosmaita> i think that makes sense (which is why i am bringing it up!)
14:36:17 <e0ne> or try to use scheduler hints for that
14:36:19 <rosmaita> but i want to get feedback from the wider community
14:36:19 <eharney> i think we should do that
14:36:24 <hemna> is there a specific version that is required ?
14:36:28 <eharney> why scheduler hints..?
14:36:59 <eharney> fast-diff is supported in all versions of ceph that we support, it was added years ago
14:37:18 <e0ne> eharney: it that case is should be safe
14:37:36 <hemna> if that's the case, then I would vote for not needing a conf for it, unless there is a specific reason why the driver shouldn't use it if it's on?
14:37:38 <eharney> it was in infernalis if not before
14:37:41 <e0ne> eharney: how incremental backups will work if we enable fast-diff?
14:38:34 <eharney> the only thing that fast-diff changes is how changes are tracked between images/snaps on the ceph backend, backups would presumably work the same
14:41:14 <e0ne> sounds good to me
14:41:35 <rosmaita> ok, so it sounds like the consensus is to ask the proposer to revise the patch to remove the config option
14:41:52 <eharney> this patch does make me think that we need to check whether we need something similar for rbd volumes, or if we are already doing the ideal thing there
14:42:35 <rosmaita> also, looking at the docstring for the file, it could probably use a revision too
14:42:47 <rosmaita> suggests using at least Ceph Dumpling
14:43:07 <eharney> heh
14:43:30 <rosmaita> eharney: if you don't have time to look, maybe file a bug so we don't forget to check
14:43:44 <rosmaita> #topic Zadara cinder driver new features review
14:43:52 <rosmaita> rratnaka: that's you
14:44:00 <rosmaita> #link https://review.opendev.org/c/openstack/cinder/+/774463
14:44:01 <rratnaka> Hi everyone.
14:44:16 <rratnaka> Great pleasure meeting community folks
14:44:39 <rosmaita> we are happy to have you here!
14:44:47 <rratnaka> As rosmaita shared the review link, I am here to highlight my review requests
14:44:52 <rratnaka> *request
14:45:24 <rratnaka> I have addressed adding new features to Zadara cinder driver and targetted to push to Wallaby release
14:45:48 <rratnaka> I addressed first set of review comments
14:45:50 <rosmaita> when you say "update code layout" , do you mean moving some functions to common ?
14:45:58 <rratnaka> yeah
14:46:43 <rratnaka> Requesting to take a look for review and provide set of comments
14:47:02 <rosmaita> ok, so though it looks like a big change, it's mostly rearranging
14:47:28 <rosmaita> i like adding 500 lines of tests, that's always good to see
14:47:29 <rratnaka> Along with that added few features like multiattach, ipv6
14:47:40 <rratnaka> and other apiś like get_manageable_volumes/snapshots
14:47:59 <rratnaka> yeah. I have added the test cases too
14:48:52 <rosmaita> ok, great.  can you add a blueprint to track this: https://blueprints.launchpad.net/cinder/wallaby
14:48:54 <rratnaka> Do we still have bandwidth to add the changes to Wallaby?
14:49:07 <rratnaka> sure rosmaita. I will add a blueprint
14:49:52 <rosmaita> ok, will help us make sure everyone is aware
14:50:27 <rosmaita> to answer the bandwidth question, let's see how the initial reviews go
14:50:49 <rosmaita> anything else?
14:51:08 <rratnaka> yeah. totally agree.
14:51:12 <rosmaita> thanks
14:51:21 <rosmaita> #topic Stable release update
14:51:28 <rratnaka> thatś it from my end. Awaiting the review comments. Thanks
14:51:34 <rosmaita> #link https://etherpad.opendev.org/p/stable-releases-review-tracker-22-02-2021
14:51:39 <rosmaita> whoami-rajat: that's you
14:51:42 <whoami-rajat> thanks rosmaita
14:52:15 <whoami-rajat> so this week there were not many reviews in stable branches
14:52:33 <whoami-rajat> victoria had 1 patch which rosmaita  approved
14:52:52 <whoami-rajat> ussuri and train still has 3-4 patches unmerged
14:53:20 <whoami-rajat> I'm not sure if i should go ahead with victoria release after the last change merges or wait for ussuri and train to propose release simultaneously
14:53:25 <whoami-rajat> rosmaita: thoughts ? ^
14:53:58 <rosmaita> i dont know that there's any advantage to a simultaneous release
14:54:24 <rosmaita> anyone have any observations?
14:55:02 <rosmaita> actually, forget i said that
14:55:11 <rosmaita> or maybe not
14:55:21 * whoami-rajat confused
14:56:04 <enriquetaso> :P
14:56:20 <rosmaita> the key thing is that we never want the situation where an earlier release (like train) contains a fix that isn't in the next release (ussuri)
14:56:31 <rosmaita> so people don't break on upgrade
14:57:13 <whoami-rajat> that shouldn't be happening because we propose backports in order
14:57:15 <rosmaita> so if you release victoria now, and then u & t in a week or so, that should be ik
14:57:18 <rosmaita> *ok
14:57:28 <whoami-rajat> oh, release wise, yes
14:57:41 <rosmaita> you got it
14:58:08 <whoami-rajat> ok, will propose the victoria release this week and wait for ussuri and train to get cleaned up
14:58:45 <rosmaita> ok, everyone needs to concentrate mostly on os-brick anyway, so other than u and t cleanup, we shouldn't have much backport action
14:58:57 <rosmaita> thanks, rajat
14:59:03 <rosmaita> #topic open discussion
14:59:08 <rosmaita> for 45 seconds
14:59:26 <whoami-rajat> also tosky  had one question on the meeting pad, not sure if I answered it properly
15:00:10 <rosmaita> i think you answered correctly, we don't know that v backports are relevant to earlier branches until someone proposes them
15:00:15 <rosmaita> ok, make way for horizon!
15:00:18 <rosmaita> thanks everyone
15:00:20 <whoami-rajat> thanks!
15:00:22 <rosmaita> #endmeeting