14:01:13 <rosmaita> #startmeeting cinder 14:01:14 <openstack> Meeting started Wed Mar 10 14:01:13 2021 UTC and is due to finish in 60 minutes. The chair is rosmaita. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:01:15 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:01:17 <openstack> The meeting name has been set to 'cinder' 14:01:30 <jungleboyj> o/ 14:01:37 <almir-okato> o/ 14:02:04 <whoami-rajat> Hi 14:02:19 <e0ne> hi 14:02:30 <eharney> hi 14:03:22 <tosky> hi 14:03:26 <TusharTgite> hi 14:03:45 <enriquetaso> hi 14:05:03 <rosmaita> hello everyone 14:05:06 <geguileo> hi! 14:05:25 <rosmaita> #link https://etherpad.openstack.org/p/cinder-wallaby-meetings 14:05:31 <rosmaita> #topic announcements 14:06:15 <rosmaita> the Xena PTG has been scheduled 14:06:25 <rosmaita> 19-23 April 14:06:40 <rosmaita> it's free to attend but you need to register 14:06:43 <rosmaita> #link http://lists.openstack.org/pipermail/openstack-discuss/2021-March/020778.html 14:06:46 <rosmaita> info ^^ 14:06:59 <rosmaita> we need to start planning 14:07:03 <rosmaita> #link https://etherpad.opendev.org/p/xena-ptg-cinder-planning 14:07:24 <rosmaita> first thing is to decide what times we will meet 14:07:27 <rosmaita> and for how long 14:07:43 <rosmaita> i think we should stick with what we've done for the past virtual PTGs 14:07:49 <e0ne> +1 14:07:56 <enriquetaso> +1 14:07:57 <jungleboyj> rosmaita: ++ 14:08:00 <rosmaita> but, i am open to ideas to accommodate people who those times don't work for 14:08:40 <rosmaita> so if the stated times are bad for you, let me know either on the etherpad or in irc 14:08:54 <rosmaita> the other question is about how long to meet each day 14:09:03 <rosmaita> we've been doing 3 hour blocks, but could do 4 14:09:48 <rosmaita> i think 3 hours has been working ok 14:10:06 <e0ne> 3 hours sounds better to me. It's easier to dedicate a time 14:10:24 <jungleboyj> rosmaita: ++ 14:10:35 <geguileo> 3 hours sounds good 14:11:10 <whoami-rajat> 3 hour works for me 14:11:15 <rosmaita> ok, guess there's a consensus, we will stick with 3 hours 14:11:30 <rosmaita> that was easy! 14:11:35 <rosmaita> next item 14:11:45 <jungleboyj> :-) 14:11:45 <rosmaita> os-brick 4.3.0 has been released and the stable/wallaby branch has been cut 14:12:06 <rosmaita> you may have seen a complaint on the ML about the aggressive update to requirements 14:12:30 <rosmaita> not sure there's anything we can do about that now 14:12:38 <rosmaita> or if i want to, anyway 14:12:54 <rosmaita> key point for this meeting is that if you find a bug in os-brick 4.3.0 14:13:03 <rosmaita> bugfixes must be proposed to master and backported to stable/wallaby 14:13:08 <whoami-rajat> adding to it, we've similar bumps in all stable releases 14:13:43 <rosmaita> so os-brick master is now xena development 14:13:46 <rosmaita> welcome to the future 14:14:10 <jungleboyj> :-) 14:14:23 <rosmaita> ok, this week is cinderclient release and the feature freeze 14:14:25 <e0ne> :) 14:14:29 <rosmaita> also requirements freeze 14:14:52 <jungleboyj> Where does the time go? 14:14:57 <rosmaita> no kidding 14:15:04 <rosmaita> ok, as far as feature freeze goes 14:15:16 <rosmaita> we are working from the blueprints in launchpad for wallaby 14:15:46 <rosmaita> #link https://blueprints.launchpad.net/cinder/wallaby 14:16:24 <rosmaita> so, if you have a feature listed in there, it will be really helpful if you make sure links to your patches are in the "Whiteboard" section of you BP 14:16:42 <rosmaita> *your BP 14:17:55 <rosmaita> except for the cinderclient patches we are about to discuss, those are the review top priority for the next few days 14:18:04 <rosmaita> #topic cinderclient release 14:18:23 <rosmaita> i have made a handy etherpad 14:18:26 <rosmaita> #link https://etherpad.opendev.org/p/wallaby-cinderclient 14:18:56 <rosmaita> the brick extension just has one open easy maintenance patch 14:19:28 <rosmaita> cinderclient has a bunch of maintenance patches 14:19:34 <rosmaita> should be quick reviews 14:20:51 <rosmaita> important patch is the version bump 14:21:12 <rosmaita> i will reach out to cores personally once the release notes and requirements patches are ready 14:21:32 <rosmaita> looking at the open reviews 14:21:35 <rosmaita> #link https://review.opendev.org/q/project:openstack/python-cinderclient+status:open 14:21:56 <rosmaita> i don't see anything else other than maybe the client cert patch 14:22:04 <rosmaita> but i haven't looked at/thought about it 14:22:28 <rosmaita> any opinions? 14:22:37 <eharney> it seems kind of risky to slip in at the last minute to me 14:23:57 <rosmaita> that is my thought 14:24:15 <rosmaita> we will push that one to xena 14:24:18 <eharney> presumably work would be happening in other clients/services for mtls too if this is of interest, we should see what 14:24:29 <jungleboyj> rosmaita: ++ Good call. 14:24:35 <rosmaita> if it turns out to be important, we can do a very early xena release that will be compatible with wallaby 14:24:41 <jungleboyj> I started looking at it yesterday and had the same thought. 14:25:06 <rosmaita> ok, thanks 14:25:27 <rosmaita> you will notice the absence of a v2-support-removal patch 14:25:52 <rosmaita> i am in the middle of one, but i am not going to try to rush it in 14:26:16 <rosmaita> i don't think we'll be removing v2 from cinder this cycle either 14:26:26 <rosmaita> but, i think it should be top priority for xena 14:26:39 <rosmaita> both client and cinder 14:26:49 <rosmaita> we will organize it at the PTG 14:27:17 <rosmaita> last item 14:27:42 <rosmaita> looks like i need to put together an etherpad of the cinder feature reviews 14:27:47 <rosmaita> i will do that after the meeting 14:28:00 <rosmaita> asking people to go to launchpad and then find the reviews doesn't seem to be working 14:28:22 <rosmaita> i will send the link to the ML and #openstack-cinder when it's ready 14:28:39 <rosmaita> only items on that list will be eligible for feature freeze 14:28:55 <rosmaita> so watch for it and check to make sure your reviews are on it 14:29:12 <whoami-rajat> rosmaita: one ques, are driver features also included in the BPs? 14:29:23 <rosmaita> whoami-rajat: yes, they are supposed to be 14:29:39 <rosmaita> that's the "driver features declaration" deadline on the release schedule 14:29:40 <whoami-rajat> then i think zadara has one feature that they haven't created a bp for 14:29:42 <whoami-rajat> https://review.opendev.org/c/openstack/cinder/+/774463 14:30:15 <rosmaita> i will have to check, i thought there was a bp for that 14:30:32 <whoami-rajat> then i might've missed it, my bad 14:30:38 <rosmaita> or i did 14:31:13 <rosmaita> but we've discussed that patch at a previous meeting, so i think it should be included as a wallaby feature 14:31:30 <whoami-rajat> i don't find anything with zadara here https://blueprints.launchpad.net/cinder/wallaby 14:32:59 <rosmaita> ok, i will fix that 14:33:20 <whoami-rajat> thanks 14:33:31 <rosmaita> one note to people with driver patches 14:33:38 <rosmaita> some of the CIs are not responding automatically 14:34:04 <rosmaita> so please check (a) your CI, and (b) run it on your patch if you have to 14:34:25 <rosmaita> a lot of people prioritize reviews by whether the patch is passing its own CI 14:34:41 <rosmaita> so make sure the info is on the review for everyone to see 14:34:43 <rosmaita> also 14:35:02 <rosmaita> please follow the release note format so we don't waste cycles re-submitting 14:35:28 <rosmaita> #link https://docs.openstack.org/cinder/latest/contributor/releasenotes.html 14:35:29 <sfernand> ack 14:35:33 <rosmaita> thanks! 14:35:54 <rosmaita> ok, that's all from me 14:36:10 <rosmaita> #topic Cinder backup - upload chunk in parallel (ThreadPoolExecutor)? 14:36:20 <hemna> heh 14:36:22 <rosmaita> i think this was left over from last week, not sure who's item it is 14:36:38 <hemna> is there a review for it? 14:37:10 <rosmaita> not sure, but i have one for you hemna: https://review.opendev.org/c/openstack/rbd-iscsi-client/+/774748 14:37:25 <rosmaita> ok, we will postpone this topic 14:37:30 <hemna> ok cool thanks. 14:37:40 <eharney> https://review.opendev.org/c/openstack/cinder/+/779233 i think 14:37:52 <hemna> we have been seeing some perf issues with chunked backup driver and are looking into parallelism as well 14:38:09 <hemna> 2Tb volume backups take like 20+ hours :( 14:38:31 <jungleboyj> :-( 14:38:56 <eharney> does parallelism fix that? 14:39:13 <hemna> yah, I think we are cpu bound 14:39:16 <e0ne> the patch looks good but we need to be careful, it could affect several drivers 14:39:46 <rosmaita> ok, then i don't want to rush it in before FF 14:39:52 <eharney> if nothing else that patch needs an option added for # of threads IMO, also needs careful review 14:39:53 <rosmaita> let's push it to Xena 14:40:02 <hemna> nah, better be safe 14:40:11 <rosmaita> eharney: ++ 14:40:28 <jungleboyj> ++ 14:40:43 <rosmaita> iirc, swift will throttle writes to a single container, so it can be counterproductive if you're not careful 14:40:47 <rosmaita> ok, moving along 14:40:50 <rosmaita> #topic random issues with tgt, breakages in some jobs, proposal to switch the default backend to lioadm 14:40:55 <rosmaita> tosky: that's you 14:41:21 * lyarwood is also here \o 14:41:21 <tosky> I'm merely the messenger; lyarwood could't attend, but you can see the report in the etherpad 14:41:24 <tosky> oh 14:41:26 * tosky hides 14:41:32 <rosmaita> ok, thanks 14:41:38 <lyarwood> hey sorry childcare means I wasn't sure if I'd make it 14:41:39 <rosmaita> everyone look at the etherpad 14:41:44 <rosmaita> oh, wait 14:41:48 <rosmaita> ok lyarwood you have the floor 14:41:52 <tosky> I was going to say "the proposal is to switch the default to lioadm in ubuntu too" 14:42:01 <lyarwood> thanks, so yeah lots of detail in the pad 14:42:20 <lyarwood> but tosky is correct that's my current proposal given the failures I've seen with encrypted volume tests 14:42:42 <lyarwood> I'm still unable to reproduce issues we've seen in the nova-live-migration job when detaching volumes that I think is related to this 14:43:48 <lyarwood> using lioadm on Bionic doesn't appear possible so we will need to pin the grenade jobs to tgtadm 14:44:09 <lyarwood> tosky: to reply to your point in the pad AFAICT the grenade job that failed was using Bionic 14:45:17 <eharney> i think lio should work on bionic 14:45:38 <eharney> at least, not sure why it wouldn't 14:45:49 <lyarwood> I assumed so but python3-rtslib-fb fails to install with `Failed to start rtslib-fb-targetctl.service: Unit rtslib-fb-targetctl.service is not loaded` 14:46:45 <eharney> huh, i'll leave figuring out that for later 14:47:02 <lyarwood> anyway appreciate there's lots of context to load here so I'm just looking for any reason not to do this after M3 14:47:39 <tosky> lyarwood: uhm, that's strange, as victoria is focal already (it was one of the goals), so the starting point for victoria should be focal 14:47:43 <tosky> lyarwood: I will ask gmann 14:48:20 <jungleboyj> tosky: Yeah, I thought we had moved to focal. 14:48:22 <eharney> well, lio should in general be a better solution, but i am concerned that our lio-barbican gate job fails a lot, possibly due to a bug in the target code 14:48:24 <rosmaita> i am worried about doing this as we approach RC time 14:48:40 <hemna> rosmaita+1 14:48:43 <eharney> so if we start running it more places we may have to get a lot more engaged on figuring out what is going on there than we have been 14:48:56 <lyarwood> so the flip side is that we have a huge uptick in volume detach failures across the gate 14:48:58 <rosmaita> yes, if our lio-barbican job is a guide, lio is not going to increase stability 14:49:22 <lyarwood> kk I wasn't aware of the lioadm job being unstable 14:49:30 <lyarwood> but to finish my point 14:49:56 <lyarwood> if I can reproduce the detach issue and show it's related to the use of tgtadm then I think the move to lioadm would still be valid ahead of rc 14:50:34 <lyarwood> the encrypted volume failure I've reproduced thus far hasn't caused anywhere near as many failures 14:51:12 <tosky> and regardless of the decision, https://review.opendev.org/c/openstack/cinder-tempest-plugin/+/779697 makes sense anyway on its own 14:51:35 <lyarwood> yeah correct 14:52:01 <rosmaita> ok, well looking at the build history, mabye that job isn't as awful as experience indicates 14:52:07 <rosmaita> https://zuul.opendev.org/t/openstack/builds?job_name=cinder-tempest-plugin-lvm-lio-barbican&project=openstack%2Fcinder 14:52:40 <rosmaita> ok, let's revisit this next week 14:52:42 <rosmaita> thanks lyarwood 14:52:52 <jungleboyj> That still looks like quite a few failures. 14:52:54 <lyarwood> ack thanks rosmaita 14:53:00 <rosmaita> #topic bug report future 14:53:03 <rosmaita> enriquetaso: that's you 14:53:07 <enriquetaso> o/ 14:53:15 <enriquetaso> In the last weekly meetings we were not able to cover all the topics proposed on the agenda. For this reason and considering the productivity of discussing the bugs of the week with the team, I would like to propose an additional mini meeting to this one. The new meeting would begin exactly after this one, it would last half an hour and it would be on the specific channel of cinder. 14:53:27 <rosmaita> enriquetaso: ++ 14:53:32 <enriquetaso> I have not sent an email to the mailing list yet, but if it's okay with you, I'll do it this week. 14:53:39 <enriquetaso> \o/ 14:53:46 <rosmaita> do we want to start today? 14:53:50 <enriquetaso> yes 14:53:56 <rosmaita> works for me 14:54:06 <rosmaita> any objections? 14:54:10 <jungleboyj> Makes sense. 14:54:10 <sfernand> nope 14:54:16 <jungleboyj> Since we often use this whole time. 14:54:25 <sfernand> sounds good 14:54:32 <rosmaita> great, let's move along then. thanks, sofia 14:54:40 <rosmaita> #topic stable branch updates 14:54:47 <rosmaita> whoami-rajat: you have the floor 14:54:51 <whoami-rajat> Hi 14:55:02 <whoami-rajat> as mentioned in the etherpad, victoria is released \o/ 14:55:11 <whoami-rajat> ussuri release patch is up 14:55:30 <tosky> \o/ 14:55:31 <whoami-rajat> but we still have unmerged patches in train, specifically cinder (i guess 7 patches?) and 1 in os-brick 14:56:12 <rosmaita> ok, i think we will have to postpone train release until later next week 14:56:21 <whoami-rajat> i will update the tracker with all unmerged patches but for time being please review https://review.opendev.org/#/q/project:openstack/cinder+status:open+branch:stable/train 14:56:30 <rosmaita> need all reviewers focused on cinderclient today and features for the rest of the week 14:56:42 <whoami-rajat> rosmaita: yep, makes sense 14:56:53 <rosmaita> ok, thanks for organizing this 14:57:04 <enriquetaso> ++ 14:57:04 <whoami-rajat> np 14:57:15 <rosmaita> #topic open discussion 14:57:25 <rosmaita> reminder: the cinder meeting is in UTC 14:57:44 <rosmaita> so for some people in north america, the local time will be different next week 14:57:59 <rosmaita> and europe goes to summer time at the end of the month 14:58:16 <rosmaita> so if you show up next week and no one is there, check the time 14:58:21 <eharney> https://review.opendev.org/c/openstack/cinder/+/777470 could use another +2, have had folks waiting on this fix for a bit 14:58:22 <rosmaita> :) 14:58:33 <jungleboyj> Oh, does DST happen this weekend? 14:58:48 <hemna> yes afaik 14:59:13 <hemna> does anyone understand the policies code? 14:59:25 <jungleboyj> rosmaita: Thanks for letting me know. 14:59:27 <rosmaita> depends on what you mean by "understand" 14:59:29 <eharney> i think some people have been doing some work on policies lately 14:59:41 <hemna> heh 14:59:53 <hemna> so, I'm looking into adding https://review.opendev.org/c/openstack/cinder/+/779541 15:00:02 <hemna> which allows admins to see all the volume_admin_metadata 15:01:03 <rosmaita> just noticed we are out of time 15:01:12 <rosmaita> bug meeting in cinder channel now 15:01:13 <hemna> is there a way to add that policy but make it optionally work, so that the current default of not showing all admin_metadata can stay in place 15:01:18 <rosmaita> #endmeeting