14:01:13 <rosmaita> #startmeeting cinder
14:01:14 <openstack> Meeting started Wed Mar 10 14:01:13 2021 UTC and is due to finish in 60 minutes.  The chair is rosmaita. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:01:15 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:01:17 <openstack> The meeting name has been set to 'cinder'
14:01:30 <jungleboyj> o/
14:01:37 <almir-okato> o/
14:02:04 <whoami-rajat> Hi
14:02:19 <e0ne> hi
14:02:30 <eharney> hi
14:03:22 <tosky> hi
14:03:26 <TusharTgite> hi
14:03:45 <enriquetaso> hi
14:05:03 <rosmaita> hello everyone
14:05:06 <geguileo> hi!
14:05:25 <rosmaita> #link https://etherpad.openstack.org/p/cinder-wallaby-meetings
14:05:31 <rosmaita> #topic announcements
14:06:15 <rosmaita> the Xena PTG has been scheduled
14:06:25 <rosmaita> 19-23 April
14:06:40 <rosmaita> it's free to attend but you need to register
14:06:43 <rosmaita> #link http://lists.openstack.org/pipermail/openstack-discuss/2021-March/020778.html
14:06:46 <rosmaita> info ^^
14:06:59 <rosmaita> we need to start planning
14:07:03 <rosmaita> #link https://etherpad.opendev.org/p/xena-ptg-cinder-planning
14:07:24 <rosmaita> first thing is to decide what times we will meet
14:07:27 <rosmaita> and for how long
14:07:43 <rosmaita> i think we should stick with what we've done for the past virtual PTGs
14:07:49 <e0ne> +1
14:07:56 <enriquetaso> +1
14:07:57 <jungleboyj> rosmaita:  ++
14:08:00 <rosmaita> but, i am open to ideas to accommodate people who those times don't work for
14:08:40 <rosmaita> so if the stated times are bad for you, let me know either on the etherpad or in irc
14:08:54 <rosmaita> the other question is about how long to meet each day
14:09:03 <rosmaita> we've been doing 3 hour blocks, but could do 4
14:09:48 <rosmaita> i think 3 hours has been working ok
14:10:06 <e0ne> 3 hours sounds better to me. It's easier to dedicate a time
14:10:24 <jungleboyj> rosmaita:  ++
14:10:35 <geguileo> 3 hours sounds good
14:11:10 <whoami-rajat> 3 hour works for me
14:11:15 <rosmaita> ok, guess there's a consensus, we will stick with 3 hours
14:11:30 <rosmaita> that was easy!
14:11:35 <rosmaita> next item
14:11:45 <jungleboyj> :-)
14:11:45 <rosmaita> os-brick 4.3.0 has been released and the stable/wallaby branch has been cut
14:12:06 <rosmaita> you may have seen a complaint on the ML about the aggressive update to requirements
14:12:30 <rosmaita> not sure there's anything we can do about that now
14:12:38 <rosmaita> or if i want to, anyway
14:12:54 <rosmaita> key point for this meeting is that if you find a bug in os-brick 4.3.0
14:13:03 <rosmaita> bugfixes must be proposed to master and backported to stable/wallaby
14:13:08 <whoami-rajat> adding to it, we've similar bumps in all stable releases
14:13:43 <rosmaita> so os-brick master is now xena development
14:13:46 <rosmaita> welcome to the future
14:14:10 <jungleboyj> :-)
14:14:23 <rosmaita> ok, this week is cinderclient release and the feature freeze
14:14:25 <e0ne> :)
14:14:29 <rosmaita> also requirements freeze
14:14:52 <jungleboyj> Where does the time go?
14:14:57 <rosmaita> no kidding
14:15:04 <rosmaita> ok, as far as feature freeze goes
14:15:16 <rosmaita> we are working from the blueprints in launchpad for wallaby
14:15:46 <rosmaita> #link https://blueprints.launchpad.net/cinder/wallaby
14:16:24 <rosmaita> so, if you have a feature listed in there, it will be really helpful if you make sure links to your patches are in the "Whiteboard" section of you BP
14:16:42 <rosmaita> *your BP
14:17:55 <rosmaita> except for the cinderclient patches we are about to discuss, those are the review top priority for the next few days
14:18:04 <rosmaita> #topic cinderclient release
14:18:23 <rosmaita> i have made a handy etherpad
14:18:26 <rosmaita> #link https://etherpad.opendev.org/p/wallaby-cinderclient
14:18:56 <rosmaita> the brick extension just has one open easy maintenance patch
14:19:28 <rosmaita> cinderclient has a bunch of maintenance patches
14:19:34 <rosmaita> should be quick reviews
14:20:51 <rosmaita> important patch is the version bump
14:21:12 <rosmaita> i will reach out to cores personally once the release notes and requirements patches are ready
14:21:32 <rosmaita> looking at the open reviews
14:21:35 <rosmaita> #link https://review.opendev.org/q/project:openstack/python-cinderclient+status:open
14:21:56 <rosmaita> i don't see anything else other than maybe the client cert patch
14:22:04 <rosmaita> but i haven't looked at/thought about it
14:22:28 <rosmaita> any opinions?
14:22:37 <eharney> it seems kind of risky to slip in at the last minute to me
14:23:57 <rosmaita> that is my thought
14:24:15 <rosmaita> we will push that one to xena
14:24:18 <eharney> presumably work would be happening in other clients/services for mtls too if this is of interest, we should see what
14:24:29 <jungleboyj> rosmaita: ++ Good call.
14:24:35 <rosmaita> if it turns out to be important, we can do a very early xena release that will be compatible with wallaby
14:24:41 <jungleboyj> I started looking at it yesterday and had the same thought.
14:25:06 <rosmaita> ok, thanks
14:25:27 <rosmaita> you will notice the absence of a v2-support-removal patch
14:25:52 <rosmaita> i am in the middle of one, but i am not going to try to rush it in
14:26:16 <rosmaita> i don't think we'll be removing v2 from cinder this cycle either
14:26:26 <rosmaita> but, i think it should be top priority for xena
14:26:39 <rosmaita> both client and cinder
14:26:49 <rosmaita> we will organize it at the PTG
14:27:17 <rosmaita> last item
14:27:42 <rosmaita> looks like i need to put together an etherpad of the cinder feature reviews
14:27:47 <rosmaita> i will do that after the meeting
14:28:00 <rosmaita> asking people to go to launchpad and then find the reviews doesn't seem to be working
14:28:22 <rosmaita> i will send the link to the ML and #openstack-cinder when it's ready
14:28:39 <rosmaita> only items on that list will be eligible for feature freeze
14:28:55 <rosmaita> so watch for it and check to make sure your reviews are on it
14:29:12 <whoami-rajat> rosmaita: one ques, are driver features also included in the BPs?
14:29:23 <rosmaita> whoami-rajat: yes, they are supposed to be
14:29:39 <rosmaita> that's the "driver features declaration" deadline on the release schedule
14:29:40 <whoami-rajat> then i think zadara has one feature that they haven't created a bp for
14:29:42 <whoami-rajat> https://review.opendev.org/c/openstack/cinder/+/774463
14:30:15 <rosmaita> i will have to check, i thought there was a bp for that
14:30:32 <whoami-rajat> then i might've missed it, my bad
14:30:38 <rosmaita> or i did
14:31:13 <rosmaita> but we've discussed that patch at a previous meeting, so i think it should be included as a wallaby feature
14:31:30 <whoami-rajat> i don't find anything with zadara here https://blueprints.launchpad.net/cinder/wallaby
14:32:59 <rosmaita> ok, i will fix that
14:33:20 <whoami-rajat> thanks
14:33:31 <rosmaita> one note to people with driver patches
14:33:38 <rosmaita> some of the CIs are not responding automatically
14:34:04 <rosmaita> so please check (a) your CI, and (b) run it on your patch if you have to
14:34:25 <rosmaita> a lot of people prioritize reviews by whether the patch is passing its own CI
14:34:41 <rosmaita> so make sure the info is on the review for everyone to see
14:34:43 <rosmaita> also
14:35:02 <rosmaita> please follow the release note format so we don't waste cycles re-submitting
14:35:28 <rosmaita> #link https://docs.openstack.org/cinder/latest/contributor/releasenotes.html
14:35:29 <sfernand> ack
14:35:33 <rosmaita> thanks!
14:35:54 <rosmaita> ok, that's all from me
14:36:10 <rosmaita> #topic Cinder backup - upload chunk in parallel (ThreadPoolExecutor)?
14:36:20 <hemna> heh
14:36:22 <rosmaita> i think this was left over from last week, not sure who's item it is
14:36:38 <hemna> is there a review for it?
14:37:10 <rosmaita> not sure, but i have one for you hemna: https://review.opendev.org/c/openstack/rbd-iscsi-client/+/774748
14:37:25 <rosmaita> ok, we will postpone this topic
14:37:30 <hemna> ok cool thanks.
14:37:40 <eharney> https://review.opendev.org/c/openstack/cinder/+/779233  i think
14:37:52 <hemna> we have been seeing some perf issues with chunked backup driver and are looking into parallelism as well
14:38:09 <hemna> 2Tb volume backups take like 20+ hours :(
14:38:31 <jungleboyj> :-(
14:38:56 <eharney> does parallelism fix that?
14:39:13 <hemna> yah, I think we are cpu bound
14:39:16 <e0ne> the patch looks good but we need to be careful, it could affect several drivers
14:39:46 <rosmaita> ok, then i don't want to rush it in before FF
14:39:52 <eharney> if nothing else that patch needs an option added for # of threads IMO, also needs careful review
14:39:53 <rosmaita> let's push it to Xena
14:40:02 <hemna> nah, better be safe
14:40:11 <rosmaita> eharney: ++
14:40:28 <jungleboyj> ++
14:40:43 <rosmaita> iirc, swift will throttle writes to a single container, so it can be counterproductive if you're not careful
14:40:47 <rosmaita> ok, moving along
14:40:50 <rosmaita> #topic random issues with tgt, breakages in some jobs, proposal to switch the default  backend to lioadm
14:40:55 <rosmaita> tosky: that's you
14:41:21 * lyarwood is also here \o
14:41:21 <tosky> I'm merely the messenger; lyarwood could't attend, but you can see the report in the etherpad
14:41:24 <tosky> oh
14:41:26 * tosky hides
14:41:32 <rosmaita> ok, thanks
14:41:38 <lyarwood> hey sorry childcare means I wasn't sure if I'd make it
14:41:39 <rosmaita> everyone look at the etherpad
14:41:44 <rosmaita> oh, wait
14:41:48 <rosmaita> ok lyarwood you have the floor
14:41:52 <tosky> I was going to say "the proposal is to switch the default to lioadm in ubuntu too"
14:42:01 <lyarwood> thanks, so yeah lots of detail in the pad
14:42:20 <lyarwood> but tosky is correct that's my current proposal given the failures I've seen with encrypted volume tests
14:42:42 <lyarwood> I'm still unable to reproduce issues we've seen in the nova-live-migration job when detaching volumes that I think is related to this
14:43:48 <lyarwood> using lioadm on Bionic doesn't appear possible so we will need to pin the grenade jobs to tgtadm
14:44:09 <lyarwood> tosky: to reply to your point in the pad AFAICT the grenade job that failed was using Bionic
14:45:17 <eharney> i think lio should work on bionic
14:45:38 <eharney> at least, not sure why it wouldn't
14:45:49 <lyarwood> I assumed so but python3-rtslib-fb fails to install with `Failed to start rtslib-fb-targetctl.service: Unit rtslib-fb-targetctl.service is not loaded`
14:46:45 <eharney> huh, i'll leave figuring out that for later
14:47:02 <lyarwood> anyway appreciate there's lots of context to load here so I'm just looking for any reason not to do this after M3
14:47:39 <tosky> lyarwood: uhm, that's strange, as victoria is focal already (it was one of the goals), so the starting point for victoria should be focal
14:47:43 <tosky> lyarwood: I will ask gmann
14:48:20 <jungleboyj> tosky:  Yeah, I thought we had moved to focal.
14:48:22 <eharney> well, lio should in general be a better solution, but i am concerned that our lio-barbican gate job fails a lot, possibly due to a bug in the target code
14:48:24 <rosmaita> i am worried about doing this as we approach RC time
14:48:40 <hemna> rosmaita+1
14:48:43 <eharney> so if we start running it more places we may have to get a lot more engaged on figuring out what is going on there than we have been
14:48:56 <lyarwood> so the flip side is that we have a huge uptick in volume detach failures across the gate
14:48:58 <rosmaita> yes, if our lio-barbican job is a guide, lio is not going to increase stability
14:49:22 <lyarwood> kk I wasn't aware of the lioadm job being unstable
14:49:30 <lyarwood> but to finish my point
14:49:56 <lyarwood> if I can reproduce the detach issue and show it's related to the use of tgtadm then I think the move to lioadm would still be valid ahead of rc
14:50:34 <lyarwood> the encrypted volume failure I've reproduced thus far hasn't caused anywhere near as many failures
14:51:12 <tosky> and regardless of the decision, https://review.opendev.org/c/openstack/cinder-tempest-plugin/+/779697 makes sense anyway on its own
14:51:35 <lyarwood> yeah correct
14:52:01 <rosmaita> ok, well looking at the build history, mabye that job isn't as awful as experience indicates
14:52:07 <rosmaita> https://zuul.opendev.org/t/openstack/builds?job_name=cinder-tempest-plugin-lvm-lio-barbican&project=openstack%2Fcinder
14:52:40 <rosmaita> ok, let's revisit this next week
14:52:42 <rosmaita> thanks lyarwood
14:52:52 <jungleboyj> That still looks like quite a few failures.
14:52:54 <lyarwood> ack thanks rosmaita
14:53:00 <rosmaita> #topic bug report future
14:53:03 <rosmaita> enriquetaso: that's you
14:53:07 <enriquetaso> o/
14:53:15 <enriquetaso> In the last weekly meetings we were not able to cover all the topics proposed on the agenda. For this reason and considering the productivity of discussing the bugs of the week with the team, I would like to propose an additional mini meeting to this one. The new meeting would begin exactly after this one, it would last half an hour and it would be on the specific channel of cinder.
14:53:27 <rosmaita> enriquetaso: ++
14:53:32 <enriquetaso> I have not sent an email to the mailing list yet, but if it's okay with you, I'll do it this week.
14:53:39 <enriquetaso> \o/
14:53:46 <rosmaita> do we want to start today?
14:53:50 <enriquetaso> yes
14:53:56 <rosmaita> works for me
14:54:06 <rosmaita> any objections?
14:54:10 <jungleboyj> Makes sense.
14:54:10 <sfernand> nope
14:54:16 <jungleboyj> Since we often use this whole time.
14:54:25 <sfernand> sounds good
14:54:32 <rosmaita> great, let's move along then.  thanks, sofia
14:54:40 <rosmaita> #topic stable branch updates
14:54:47 <rosmaita> whoami-rajat: you have the floor
14:54:51 <whoami-rajat> Hi
14:55:02 <whoami-rajat> as mentioned in the etherpad, victoria is released \o/
14:55:11 <whoami-rajat> ussuri release patch is up
14:55:30 <tosky> \o/
14:55:31 <whoami-rajat> but we still have unmerged patches in train, specifically cinder (i guess 7 patches?) and 1 in os-brick
14:56:12 <rosmaita> ok, i think we will have to postpone train release until later next week
14:56:21 <whoami-rajat> i will update the tracker with all unmerged patches but for time being please review https://review.opendev.org/#/q/project:openstack/cinder+status:open+branch:stable/train
14:56:30 <rosmaita> need all reviewers focused on cinderclient today and features for the rest of the week
14:56:42 <whoami-rajat> rosmaita: yep, makes sense
14:56:53 <rosmaita> ok, thanks for organizing this
14:57:04 <enriquetaso> ++
14:57:04 <whoami-rajat> np
14:57:15 <rosmaita> #topic open discussion
14:57:25 <rosmaita> reminder: the cinder meeting is in UTC
14:57:44 <rosmaita> so for some people in north america, the local time will be different next week
14:57:59 <rosmaita> and europe goes to summer time at the end of the month
14:58:16 <rosmaita> so if you show up next week and no one is there, check the time
14:58:21 <eharney> https://review.opendev.org/c/openstack/cinder/+/777470   could use another +2, have had folks waiting on this fix for a bit
14:58:22 <rosmaita> :)
14:58:33 <jungleboyj> Oh, does DST happen this weekend?
14:58:48 <hemna> yes afaik
14:59:13 <hemna> does anyone understand the policies code?
14:59:25 <jungleboyj> rosmaita:  Thanks for letting me know.
14:59:27 <rosmaita> depends on what you mean by "understand"
14:59:29 <eharney> i think some people have been doing some work on policies lately
14:59:41 <hemna> heh
14:59:53 <hemna> so, I'm looking into adding https://review.opendev.org/c/openstack/cinder/+/779541
15:00:02 <hemna> which allows admins to see all the volume_admin_metadata
15:01:03 <rosmaita> just noticed we are out of time
15:01:12 <rosmaita> bug meeting in cinder channel now
15:01:13 <hemna> is there a way to add that policy but make it optionally work, so that the current default of not showing all admin_metadata can stay in place
15:01:18 <rosmaita> #endmeeting