14:00:49 <rosmaita> #startmeeting cinder
14:00:50 <openstack> Meeting started Wed Mar 25 14:00:49 2020 UTC and is due to finish in 60 minutes.  The chair is rosmaita. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:51 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:00:53 <openstack> The meeting name has been set to 'cinder'
14:01:00 <rosmaita> #link https://etherpad.openstack.org/p/cinder-ussuri-meetings
14:01:01 <smcginnis> o/
14:01:10 <rosmaita> #topic roll call
14:01:12 <whoami-rajat> Hi
14:01:16 <m5z> hi
14:01:26 <tosky> o/
14:01:32 <rosmaita> pretty good turnout
14:01:41 <rosmaita> agenda looks short, but actually there is a lot to do
14:01:55 <rosmaita> #topic announcements
14:01:59 <rajinir> o/
14:02:03 <eharney> hi
14:02:04 <rosmaita> i hope everyone is coping adequately with the covid-19 pandemic
14:02:10 <rosmaita> i came across this interesting post, advice on handling isolation from astronauts
14:02:17 <rosmaita> #link https://www.space.com/astronaut-tips-for-handling-isolation-coronavirus.html
14:02:24 <rosmaita> i'd been thinking that the big stressor to cope with is not being able to get out much
14:02:31 <rosmaita> but there's also the fact that you are now stuck with the same people for 24 hours a day
14:02:38 <rosmaita> which can become stressful even if they are the nicest people you know
14:02:45 <whoami-rajat> :D
14:02:55 <rosmaita> #topic announcements - Victoria PTG
14:03:02 <rosmaita> it was announced shortly after last week's meething that the Victoria PTG will be virtual
14:03:09 <rosmaita> so keep the dates open: 8-11 June 2020
14:03:16 <rosmaita> exact format not yet known, sign up if you want to help plan:
14:03:23 <rosmaita> #link https://etherpad.openstack.org/p/Virtual_PTG_Planning
14:03:31 <rosmaita> and, of course, start thinking about topics for the cinder part:
14:03:37 <rosmaita> #link https://etherpad.openstack.org/p/cinder-victoria-ptg-planning
14:03:49 <rosmaita> on the plus side, we should get a pretty good turnout!
14:04:02 <rosmaita> #topic announcements - devstack-plugin-open-cas-core
14:04:10 <rosmaita> Liang Fang was added to the core team for the plugin this morning
14:04:16 <rosmaita> other core members are cinder-core and devstack-core
14:04:23 <rosmaita> congratulations, Liang!
14:04:25 <LiangFang> thanks, I see +2 and -2 now
14:04:30 <LiangFang> :)
14:04:33 <rosmaita> use them wisely!
14:04:41 <rosmaita> #topic announcements - upcoming deadlines
14:04:44 <whoami-rajat> Congrats LiangFang
14:04:53 <rosmaita> we are winding down the ussuri cycle, believe it or not
14:05:00 <rosmaita> so here are some dates to keep in mind
14:05:06 <rosmaita> next week (week of 30 March):
14:05:16 <rosmaita> * ussuri os-brick release
14:05:23 <rosmaita> week after that (week of 6 April):
14:05:25 <jungleboyj> Ah, here it is.
14:05:40 <rosmaita> * ussuri cinderclient release
14:05:40 <rosmaita> * ussuri milestone-3 release
14:05:40 <rosmaita> * FEATURE FREEZE
14:05:57 <rosmaita> jungleboyj: did you just find the room?
14:06:02 <rosmaita> two weeks after that (week of 20 April):
14:06:10 <rosmaita> * RC-1
14:06:17 <rosmaita> so, lots of stuff coming up fast
14:06:21 <jungleboyj> I thought I had joined it already.  Apparently not.  :-(
14:06:33 <rosmaita> #topic announcements - driver reviews
14:06:41 <rosmaita> so, i am aware of some grumbling about the non-speediness of driver reviews
14:06:47 <rosmaita> which is understandable
14:06:52 <enriquetaso> o/
14:06:54 <rosmaita> but i'd like to remind everyone about the OpenStack Credo:
14:07:00 <rosmaita> Never swim alone.
14:07:08 <rosmaita> (no, that's the Penguin Credo ... i think i have cabin fever)
14:07:14 <rosmaita> The OpenStack Credo: if you want to get reviews, you need to do reviews.
14:07:15 <walshh_> hi
14:07:22 <smcginnis> :)
14:07:23 <rosmaita> from the previous discussion you can see that we *must* get os-brick reviews done and merged before the next cinder meeting
14:07:30 <rosmaita> and python-cinderclient the week after that
14:07:36 <rosmaita> remember that you don't *have* to be a cinder-core to do reviews
14:07:42 <rosmaita> and you don't have to work for a vendor to review that vendor's patches
14:07:49 <rosmaita> and if you work for a vendor, you probably have a good idea of what's needed in driver patches
14:07:56 <rosmaita> is the release note clear? have any doc updates been included? do the unit tests make sense? are there unit tests?
14:08:03 <rosmaita> and you don't have to have an opinion on everything in the patch (like if there's some weird python usage you're not familiar with)
14:08:10 <rosmaita> just say what you did look at when leaving your vote on the patch
14:08:17 <rosmaita> this can speed up the reviews for everyone
14:08:19 <jungleboyj> ++
14:08:24 <whoami-rajat> rosmaita, ++
14:08:24 <rosmaita> same deal with os-brick or cinderclient patches, you can review those too
14:08:33 <rosmaita> here endeth the lesson
14:08:34 <enriquetaso> ++
14:08:41 <rosmaita> #topic stable branch releases
14:08:49 <rosmaita> #link https://etherpad.openstack.org/p/cinder-releases-tracking
14:08:55 <rosmaita> a few things to discuss here
14:09:11 <rosmaita> and i am doing them in reverse order from that etherpad (did not plan ahead, i guess)
14:09:18 <rosmaita> #topic stable branch releases - cinder-tempest-plugin
14:09:25 <rosmaita> not branched, so release would be from master
14:09:30 <rosmaita> there are 2 open reviews
14:09:37 <rosmaita> would be good to get some more tests in
14:09:43 <rosmaita> so this is a call to please review and we can release around M-3
14:09:51 <rosmaita> #topic stable branch releases - cinderclient
14:10:00 <rosmaita> nothing to do for the brick-cinderclient-ext
14:10:08 <rosmaita> there were no open reviews for python-cinderclient, so it's ready for release
14:10:24 <rosmaita> #topic stable branch releases - os-brick
14:10:32 <rosmaita> train is ready (i think ... pay attention to this discussion!)
14:10:40 <rosmaita> stein has one open review (but don't immediately rush off and vote on it yet)
14:10:49 <rosmaita> #link https://review.opendev.org/#/c/713082/
14:10:55 <rosmaita> it's an optimization (which could be fixing a legitimate bug if we're getting bad timeouts or something like that, so i don't think it should be disqualified simply because it's an optimization)
14:11:09 <rosmaita> we allowed the backport to train, but smcginnis left a comment on the train backport
14:11:21 <rosmaita> "I'm actually a little concerned about backporting this since it is such a divergence from how things have been working. Plus we haven't really had much runtime on it is ussuri, so there's a risk that there will be some issue found with doing it this way that we aren't aware of yet."
14:11:29 <rosmaita> "All that said, we can always revert this, and if it's already made it out in a stable release we can always blacklist that release and quickly get a new release out that goes back to the previous behavior. So I think there's a slight risk, but it could be a nice improvement to get out there."
14:11:43 <rosmaita> which is a good point
14:11:46 <rosmaita> so my questions for us here are (let me list all 4 questions first, then we can discuss)
14:11:50 <smcginnis> Yeah, just a little worried getting that backported all the way until we have confidence that it works right.
14:11:58 <rosmaita> (1) do we release 2.10.2 from stable/train containing the already merged fix?
14:11:58 <rosmaita> (2) do we hold off on backporting the fix to stein?
14:11:58 <rosmaita> (3) do we *deny* the backport to stein?
14:11:59 <rosmaita> (4) and if so, do we revert it from train?
14:12:13 <rosmaita> ok, here we go ...
14:12:13 <rosmaita> (1) do we release 2.10.2 from containing the fix?
14:12:19 <rosmaita> I think yes and keep an eye on it as smcginnis suggests
14:12:37 <rosmaita> (i have been typing like a maniac, i will now be quiet for a minute)
14:12:57 <eharney> so... the main reason i didn't vote on this patch yet is because i didn't get around to convincing myself it would have no ill effects
14:14:20 <jungleboyj> If we aren't sure ... Nothing is broken if we don't backport it.  Right?
14:14:29 <jungleboyj> It just is slower.
14:14:39 <rosmaita> does anyone here know that this issue is causing problems in someone's current production, or if someone knows that an operator has already done a local backport and is running with this change already?
14:15:01 <eharney> it was affecting one of our customers iirc
14:15:04 <rosmaita> jungleboyj: i'm not sure if it's "just" an optimization or whether its causing timeouts
14:15:14 <jungleboyj> Ok.
14:15:35 <eharney> some kind of scaling issue on large setups, i'd have to go find the details again
14:15:49 <rosmaita> ok, so we do have evidence that the issue is a bug
14:16:18 <rosmaita> yeah, if it only shows up at scale, it will be hard to see what this does
14:17:38 <smcginnis> On the surface, it looks like a more reasonable way to do things.
14:17:59 <smcginnis> Just we've been doing it the other way so long that it's an understood and well worn path.
14:18:02 <smcginnis> This isn't yet.
14:18:06 <rosmaita> exactly
14:18:53 <rosmaita> so, it's already in stable/train ... should we release and see what happens?
14:19:12 <eharney> i would think so
14:19:12 <rosmaita> i just wonder if we'll get any feedback
14:19:43 <rosmaita> ok, so (1): release from train including the fix
14:20:06 <rosmaita> so, (2) hold off on backporting to stein for now?
14:20:07 <jungleboyj> It looks like it should be safe .  We only will know if it causes issues by releasing it.  :-)
14:20:25 <rosmaita> jungleboyj: true enough
14:20:41 <jungleboyj> Chicken and egg.  :-)
14:20:50 <smcginnis> I think that's a good plan. I'm fine getting it in stein once we've had at least some real world run time to know it's not going to break something that isn't exactly configured like our gate images.
14:20:58 <eharney> i'll look over it and see if moving away from a process call to a read opens up a window for races etc that wasn't there before
14:21:10 <rosmaita> eharney: that would be great
14:21:48 <eharney> worth noting that this affects both regular nfs and netapp nfs, too
14:22:08 <rosmaita> ok, so it sounds like we don't deny it to stein yet, just don't include it in the release from stein this week
14:22:40 <jungleboyj> I am ok with that plan.
14:22:55 <rosmaita> eharney: good point, maybe someone from netapp can also take a look
14:23:03 <rosmaita> (hint, hint)
14:23:21 <rosmaita> ok, sounds like we are all set there
14:23:30 <rosmaita> #topic stable branch releases - cinderlib
14:23:38 <rosmaita> this one is easy, there are no unreleased changes or open reviews on stable/train
14:23:41 <rosmaita> :)
14:23:49 <rosmaita> #topic stable branch releases - cinder
14:23:57 <rosmaita> ok, last stable deliverable
14:24:05 <rosmaita> stable/stein looks ready to go ... maybe (i will explain in a minute)
14:24:14 <rosmaita> stable/train has one open review that i'd like to see merged, it fixes a segfault problem with the RBD driver
14:24:21 <rosmaita> #link https://review.opendev.org/#/c/714151/
14:24:27 <rosmaita> it has one +2 atm
14:24:36 <rosmaita> you'd think people would be screaming about a segfault problem, but it only happens during exception handling in a conditional branch controlled by a config setting
14:24:44 <rosmaita> that's probably why you haven't heard about it
14:24:52 <rosmaita> anyway, you can probably guess what i meant by stein was ready ... maybe
14:25:00 <rosmaita> i think we should backport the change to stein after it is approved for train
14:25:07 <rosmaita> and then release from stable/stein
14:25:13 <rosmaita> but i am open to other opinions
14:25:47 <smcginnis> That seems like a bug in the library that should get fixed.
14:25:58 <tosky> and what about this (new) one? Would it be worth to have it in train, or (if it's merged in master and backported) should wait for the next release?  https://review.opendev.org/#/c/711906/
14:26:05 <rosmaita> yes, eharney was saying that too
14:26:12 <eharney> i tend to agree, but the general situation for is that if you do things that python-rbd doesn't want you to do, it just breaks :/
14:26:20 <eharney> and i think that isn't going to change in the near term
14:26:24 <smcginnis> https://review.opendev.org/#/c/714151 fix actually seems opposite of the commit message description.
14:26:51 <smcginnis> It says it is closed too soon, but then it code change closes it sooner/
14:27:04 <rosmaita> well, it takes the close out of the finally
14:27:17 <smcginnis> Anyway, I don't need to understand it as long as it fixes things. :)
14:27:29 <jungleboyj> :-)
14:27:39 <rosmaita> it gets closed sooner when there's an exception
14:27:52 <rosmaita> (i spent a lot of time looking at this one)
14:27:52 <smcginnis> Yeah, but it says that's the problem.
14:28:37 <rosmaita> yeah, the problem is later, if you have to flatten the volume and get an exception there
14:28:41 <rosmaita> then we try to clean up
14:28:53 <rosmaita> but the source volume is already closed because of that "finally" at the top
14:29:14 <smcginnis> I don't see how this improves that situation, but that's OK. We can move along.
14:29:25 <rosmaita> ok
14:29:47 <rosmaita> smcginnis: part of the problem is that the logic in there is pretty non-optimal
14:30:16 <rosmaita> but the good news is that all this is handled on the rbd backend after the jewell release
14:30:30 <rosmaita> so we can refactor that entire thing down to like 4 lines
14:31:20 <rosmaita> ok, i lost track of where we are ...
14:31:47 <rosmaita> let's leave this at please look at https://review.opendev.org/#/c/714151
14:32:13 <rosmaita> if it is approved in the next few days, i'll propose a backport to stein
14:32:20 <jungleboyj> Should I remove my +W then?
14:32:33 <whoami-rajat> it's approved as i see
14:32:34 <rosmaita> only if you aren't sure about it!
14:32:49 <rosmaita> ok, when i looked before the meeting it was still open
14:33:18 <jungleboyj> I changed it to just a +2 so those who have concerns can vote.
14:33:32 <rosmaita> ok, sounds good
14:33:46 <rosmaita> let's leave it open for 24 hours
14:33:58 <rosmaita> so if you have concerns or general interest in the rbd driver please look
14:34:20 <rosmaita> we aren't really on a deadline for the stable branch releases anyway
14:34:21 <rosmaita> ok
14:34:30 <rosmaita> i ignored tosky's suggestion
14:34:39 <rosmaita> let's look at it
14:34:49 <rosmaita> #link https://review.opendev.org/#/c/711906/
14:35:35 <rosmaita> that hasn't even made it into master yet
14:36:08 <eharney> i'm a little concerned about this one, would like to dig into it more
14:36:17 <rosmaita> i think we can wait for next release
14:36:24 <smcginnis> Yep
14:36:27 <jungleboyj> ++
14:36:42 <rosmaita> ok, thanks ... and that's all for the stable branch releases!
14:36:45 <tosky> ack, thanks; just wanted to asses its importance
14:36:54 <eharney> we also need to more generally document some notes around all this volume renaming on migration i think -- it's bit us in a few different drivers
14:37:19 <rosmaita> yeah, i think i have already seen a patch for a different driver on this same issue
14:37:26 <rosmaita> #topic ussuri os-brick release
14:37:34 <rosmaita> we have to release ussuri os-brick next week
14:37:45 <rosmaita> this is what's been proposed:
14:37:46 <rosmaita> https://review.opendev.org/#/q/status:open+project:openstack/os-brick+branch:master
14:38:14 <rosmaita> a lot of WIP and -Ws in that list
14:38:31 <jungleboyj> Yeah.
14:39:17 <smcginnis> And unaddressed -1's.
14:39:25 <rosmaita> yes, that too
14:39:56 <rosmaita> so, if there is anything urgent, please say so now, or on the ML in the next day or so
14:39:58 <smcginnis> geguileo: You hace a couple interesting ones in there. Should we get those updated and try to get them in ussuri?
14:40:02 <smcginnis> *have
14:40:30 <geguileo> let me check...
14:41:43 <geguileo> smcginnis: I think https://review.opendev.org/#/c/695144/ is an interesing one
14:42:19 <geguileo> smcginnis: iirc the FQDN one is less important and probably harder to fix
14:42:57 <smcginnis> geguileo: Want to remove your -W?
14:43:00 <LiangFang> rosmaita: after os-brick be released next week, will the remaining patches be reviewed in the following weeks or monthes? thanks
14:43:09 <geguileo> smcginnis: I should test it first...
14:43:25 <geguileo> I'll test it this week
14:43:34 <smcginnis> geguileo: Thanks!
14:43:41 <rosmaita> LiangFang: what will happen is that we will create a stable/ussuri branch for os-brick, and work can continue in master
14:43:41 <geguileo> np
14:44:04 <LiangFang> rosmaita: thanks
14:44:05 <rosmaita> so normal review turnaround :)
14:44:45 <rosmaita> geguileo: great, thank you
14:44:54 <LiangFang> rosmaita: that would means the feature would go to V release, right?
14:45:05 <rosmaita> LiangFang: yes, that's correct
14:45:09 <LiangFang> OK
14:46:12 <rosmaita> moving along, then
14:46:29 <rosmaita> #topic Added tempest tests to zuul for devstack-plugin-open-cas
14:46:39 <rosmaita> #link https://review.opendev.org/#/c/713772/
14:46:54 <LiangFang> I have added the tempest test to zuul
14:47:12 <LiangFang> I take a look of the job log, open-cas installed correctly
14:47:40 <LiangFang> and I have removed debug info from code
14:47:46 <tosky> uhm, I guess you can drop the -py3 suffix from the job name at this point
14:47:58 <tosky> it's not like we are going to have a py2 job ever
14:47:59 <LiangFang> ok
14:48:09 <rosmaita> tosky: but not from the parent job?
14:48:22 <rosmaita> line 3 in .zuul.yaml
14:49:20 <tosky> that's tricky; as far as I remember, tempest-full should be py3 based nowadays; it may have been a problem if the new jobs were used in a branch < ussuri
14:50:15 <rosmaita> ok, so i guess we leave it in the parent job name and let qa team make a mass proposal at some point to correct it in every project
14:50:30 <rosmaita> but we can remove -py3 from our defined job
14:50:36 <tosky> I suspect that's not going to happen, so let's fix at least our job
14:50:38 <tosky> yep
14:51:02 <LiangFang> ok
14:51:07 <rosmaita> tosky: why don't you leave a comment on the review (if you haven't already)
14:51:18 <tosky> doing it
14:51:22 <rosmaita> great
14:51:42 <jungleboyj> I just added that comment.
14:51:47 <jungleboyj> Eric has some comments too.
14:52:14 <rosmaita> ok, so at this point, the tempest jobs run with open-cas installed but not actually used
14:52:31 <rosmaita> but that puts us in a good position as the other changes are made
14:52:52 <rosmaita> LiangFang: anything else?
14:53:07 <LiangFang> eharney: ERROR_ON_CLONE: false
14:53:45 <LiangFang> this is because devstack will fail to git clone .../open-cas-linux repo if don't add this line
14:54:18 <tosky> LiangFang: then something else should be changed
14:54:23 <tosky> you shouldn't need that
14:54:26 <eharney> that sounds odd
14:55:07 <LiangFang> tosky: yes, seems odd, I don't seen ceph plugin have this line
14:55:22 <LiangFang> but ceph plugin works
14:55:25 <tosky> LiangFang: I will recheck the last build without that change, but I believe we should remove it before merging
14:55:35 <rosmaita> ok, let's try to figure that out on the review
14:55:40 <tosky> yep
14:55:45 <rosmaita> #topic open discussion
14:56:09 <rosmaita> anyone ... now's your chance
14:57:02 <rosmaita> anyone who came in late, be sure to read my rant about reviewing during the "updates" part of the meeting
14:57:14 <rosmaita> :D
14:57:14 <jungleboyj> :-)
14:57:20 <tosky> just a zuul v3 reminder: even though this review leaves out one legacy job, it's still an improvement, and it would be nice to have it merged before branching ussuri: https://review.opendev.org/#/c/671945/
14:57:54 <eharney> yeah we should probably get that in
14:58:00 <rosmaita> agree
14:58:02 <hemna> +1
14:58:05 <rosmaita> tosky: thanks for the reminder
14:58:14 <rosmaita> #link https://review.opendev.org/#/c/671945/
14:58:31 <rosmaita> let's try to hit M-3 with that
14:58:49 <jungleboyj> hemna:  If you joined late, please take a look at os-brick patches that are outstanding.  :-)
14:58:58 <tosky> it's should also be backportable, just in case
14:59:00 <hemna> jungleboyj will do
14:59:06 <jungleboyj> hemna:  Thank you sir.
15:00:11 <rosmaita> hemna: especially https://review.opendev.org/#/c/706780/
15:00:25 <rosmaita> touches some code you've worked on
15:00:43 <eharney> i think that one still needs some tweaking
15:00:51 <hemna> cool, looks like it needs some work still
15:00:55 <rosmaita> yes
15:00:58 <rosmaita> and we are out of time
15:01:08 <rosmaita> ok, thanks everyone
15:01:09 <jungleboyj> :-)
15:01:16 <rosmaita> #endmeeting