14:00:20 <rosmaita> #startmeeting cinder
14:00:21 <openstack> Meeting started Wed Sep  2 14:00:20 2020 UTC and is due to finish in 60 minutes.  The chair is rosmaita. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:22 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:00:24 <openstack> The meeting name has been set to 'cinder'
14:00:31 <rosmaita> #topic roll call
14:00:33 <whoami-rajat> Hi
14:00:38 <e0ne> hi
14:00:39 <kaisers> hi
14:00:43 <enriquetaso> ji
14:00:44 <LiangFang> hi
14:00:44 <eharney> hi
14:00:45 <enriquetaso> hi*
14:00:47 <smcginnis> .o.
14:00:52 <hemna> mep
14:01:33 <rosmaita> good turnout
14:01:44 <rosmaita> #link https://etherpad.openstack.org/p/cinder-victoria-meetings
14:01:56 <rosmaita> #topic announcements
14:02:12 <rosmaita> reminder about Forum brainstorming
14:02:25 <rosmaita> i sent something to the ML to solicit ideas from operators and other cinder users
14:02:34 <rosmaita> #link http://lists.openstack.org/pipermail/openstack-discuss/2020-August/016659.html
14:02:47 <rosmaita> and here is where you can add ideas:
14:02:57 <rosmaita> #link https://etherpad.opendev.org/p/2020-Wallaby-cinder-brainstorming
14:03:21 <m5z> hi =]
14:03:40 <rosmaita> we will pick what topics to propose at next week's meeting
14:03:51 <rajinir> o/
14:03:56 <rosmaita> in the meantime, you need to be registered for the Summit to attend the Forum
14:04:16 <rosmaita> it's free, so go ahead and register
14:04:28 <rosmaita> #link https://openinfrasummit2020.eventbrite.com/
14:05:04 <rosmaita> also, we have the Wallaby PTG coming up
14:05:09 <enriquetaso> is the deadline for proposing topics on Wed 9 sept ?
14:05:35 <rosmaita> enriquetaso: i thought friday that week?
14:05:52 <enriquetaso> thanks!
14:06:18 <rosmaita> but if you have an idea, add it to the forum etherpad
14:06:39 <enriquetaso> ++
14:06:40 <rosmaita> and the wallaby PTG etherpad is
14:06:51 <rosmaita> #link https://etherpad.opendev.org/p/wallaby-ptg-cinder-planning
14:07:16 <rosmaita> #topic updates - releases
14:07:29 <rosmaita> ok, os-brick for victoria has to be released tomorrow
14:07:36 <rosmaita> we have one patch in the gate now
14:07:52 <rosmaita> #link https://review.opendev.org/730376
14:08:02 <rosmaita> previous release was 3.2.1
14:08:11 <rosmaita> this will be 4.0.0 because we removed a bunch of connectors for drivers that were removed from cinder
14:08:14 <e0ne> thanks everybody for the feedback to the patch above
14:08:50 <rosmaita> we have some stuff piled up for stable/ussuri and train in os-brick
14:09:04 <rosmaita> so i will put up patches for those releases soon
14:09:17 <rosmaita> next up: python-cinderclient release next week
14:09:26 <rosmaita> someone please look at this one! it's a 2 line change! cinderclient support for mv 3.61 (which has merged on cinder side)
14:09:32 <rosmaita> #link https://review.opendev.org/#/c/742994/
14:09:57 <rosmaita> the big thing to focus on is cinder patch for per-project default volume types:
14:10:04 <rosmaita> #link https://review.opendev.org/#/c/737707/
14:10:05 * e0ne is feeling like a release-blocker person
14:10:18 <rosmaita> because that has a cinderclient support patch:
14:10:25 <rosmaita> #link https://review.opendev.org/739223
14:10:39 <hemna> looks like that patch only bumps the microversion number ?
14:10:52 <rosmaita> hemna: yes
14:10:54 <e0ne> hemna: yes
14:11:25 <hemna> ok done
14:11:32 <rosmaita> whoami-rajat has revised https://review.opendev.org/#/c/742994/
14:11:46 <e0ne> smcginnis, hemna: thank you!
14:12:00 <rosmaita> so it could use some eyes, need to make sure it's acceptable so that we know the cinderclient side is correct
14:12:01 <whoami-rajat> rosmaita, that's not the patch ...
14:12:15 <rosmaita> oops
14:12:24 <whoami-rajat> #link https://review.opendev.org/#/c/739223/
14:12:25 <rosmaita> #link https://review.opendev.org/#/c/737707/
14:12:47 <rosmaita> we need the cinder-side patch merged first, i think
14:13:05 <rosmaita> so to be completely clear: these are really high priority patches
14:13:18 <rosmaita> #link https://review.opendev.org/#/c/737707/
14:13:21 <whoami-rajat> oh the API one
14:13:23 <whoami-rajat> ok
14:13:24 <rosmaita> #link https://review.opendev.org/739223
14:13:41 <rosmaita> ok, next priority are other cinder features for victoria
14:13:52 <rosmaita> use the blueprints:
14:13:52 <rosmaita> #link https://blueprints.launchpad.net/cinder/victoria
14:14:06 <rosmaita> but i will point out some really easy reviews (and they have been sitting quite a while ...)
14:14:13 <rosmaita> support zstd compression:
14:14:14 <rosmaita> #link https://review.opendev.org/726765
14:14:21 <rosmaita> extra-specs support for volume local cache:
14:14:22 <rosmaita> #link https://review.opendev.org/#/c/700799/
14:14:36 <rosmaita> stop sending notifications to deprecated nonstandard publisher_id
14:14:36 <rosmaita> #link https://review.opendev.org/747540
14:14:36 <rosmaita> hemna: would like you to take a look ^^
14:14:46 <hemna> ok
14:14:46 <rosmaita> remove deprecated rbd option for OSSN-0085
14:14:46 <rosmaita> #link https://review.opendev.org/#/c/747494/
14:14:46 <rosmaita> and there are the driver features
14:15:14 <hemna> ah yes cool
14:15:50 <rosmaita> ok, so to be completely clear: top priority is whoami-rajat's per-project-default-type stuff
14:15:58 <rosmaita> because we must release cinderclient next week
14:16:13 <rosmaita> so no possibility of FFE to help there
14:16:26 <rosmaita> and a reminder to people with driver features
14:16:49 <rosmaita> reviewing other people's patches will help you get yours reviewed faster
14:16:53 <rosmaita> (we do pay attention)
14:16:55 <hemna> lots of failures on the per project default type patch :(
14:17:28 <rosmaita> yeah, the gate has been really unstable
14:17:57 <rosmaita> any other announcements?
14:18:03 <rosmaita> (that's all from me)
14:18:42 <rosmaita> #topic RBD connector fix
14:18:56 <rosmaita> there was some discussion about this today in the cinder channel
14:19:21 <rosmaita> it may be possible to simplify the patch, but it requires more testing against various ceph versions
14:20:03 <rosmaita> #link https://review.opendev.org/#/c/730376/
14:20:20 <e0ne> I tested only with octopus and nautilus
14:21:22 <e0ne> the idea is to test against supported versions to check if [global] section works for every supported ceph release
14:21:50 <rosmaita> so to be clear,the patch is backward compatible right now, and since the issue has only been reported for octopus, we should be OK
14:22:02 <rosmaita> but it may be possible to remove the conditional logic
14:22:22 <rosmaita> so for ceph-inclined people, that's something to take a look at
14:22:30 <rosmaita> e0ne: anything else?
14:22:41 <e0ne> rosmaita: that's all for me
14:22:53 <e0ne> I raised this topic only because release deadline
14:23:06 <rosmaita> and quite appropriately, too
14:23:23 <rosmaita> #topic service creation race condition
14:23:29 <rosmaita> #link https://bugs.launchpad.net/cinder/+bug/1891330
14:23:30 <openstack> Launchpad bug 1891330 in Cinder "Duplicate Cinder services DB entries after upgrade" [Low,Triaged]
14:23:40 <rosmaita> i mostly just want to raise some awareness
14:23:46 <rosmaita> there is an abandoned patch (link in the bug)
14:23:53 <rosmaita> but the major objection was that cinder didn't support A/A yet
14:23:58 <rosmaita> which isn't the case any more
14:24:02 <e0ne> I'm all for restoring proposed patch to add unique constraints into the DB
14:24:26 <rosmaita> well, ordinarily, i would be too, but my experience this past week is making me a bit gun-shy
14:24:31 <rosmaita> as we will discuss in a minute
14:25:25 <rosmaita> but, it does look like an easy fix
14:25:58 <rosmaita> #topic ussuri upgrade issue
14:26:23 <rosmaita> this is the issue
14:26:28 <rosmaita> #link https://bugs.launchpad.net/cinder/+bug/1893107
14:26:29 <openstack> Launchpad bug 1893107 in Cinder "Upgrade to Ussuri fails if deleted volumes exist prior to Train" [High,In progress] - Assigned to Mohammed Naser (mnaser)
14:27:06 <rosmaita> well, i spent a bunch of time trying to figure out a way to fix this that was low-impact on operators
14:27:12 <rosmaita> and i thought i had a solution
14:27:18 <rosmaita> but, wound up going back to the original proposal
14:27:50 <rosmaita> the issue in a nutshell is that the online migrations to set __DEFAULT__ as the volume type for all untyped volumes/snapshots in Train
14:28:02 <rosmaita> did not take into account the soft-deleted volumes/snapshots
14:28:12 <rosmaita> so when you try to upgrade to ussuri
14:28:24 <rosmaita> and the db_sync puts a non-nullability constraint on the columns
14:28:28 <rosmaita> the db_sync fails
14:28:55 <rosmaita> this is complicated by the fact that someone discovered a way to delete the __DEFAULT__ volume type in train
14:29:06 <rosmaita> so, we don't know for sure that it actually exists
14:29:31 <rosmaita> but, the issue only applies to people who upgraded from Stein to Train without first purging the database
14:29:38 <rosmaita> (which may actually be a lot of people)
14:30:11 <rosmaita> anyway, the best thing i could come up with are these patches:
14:30:21 <rosmaita> #link https://review.opendev.org/#/c/748481/
14:30:28 <whoami-rajat> (but if they don't have issue purging it then they can purge anytime and upgrade successfully)
14:30:33 <rosmaita> #link https://review.opendev.org/#/c/748482/
14:31:00 <rosmaita> so, my first ask is for careful reviews of those patches
14:31:08 <rosmaita> the code change is simple, and there are tests
14:31:16 <e0ne> whoami-rajat: sometimes operators can't do purge because they use deleted volumes for billing
14:31:28 <rosmaita> but there is manual operator intervention required when you don't do the purge
14:31:45 <rosmaita> so PLEASE read the release notes carefully to see if i covered everything
14:31:45 <whoami-rajat> e0ne, yep, that's why i said if they don't have issue/don't need that info
14:32:13 <rosmaita> and if you are interested in all the crazy schemes i thought of that didn't work, you can read the sorry story here:
14:32:21 <hemna> fwiw, we do purging
14:32:22 <rosmaita> #link https://etherpad.opendev.org/p/ussuri-upgrade-issue
14:33:46 <rosmaita> yes, so this shows a weakness in grenade testing of T->U upgrades
14:33:59 <rosmaita> the problem only happens in S->T->U upgrades
14:34:13 <rosmaita> but i am not going to propose a 3 stage grenade job
14:35:09 <rosmaita> the reason i had this on the agenda was that i was going to propose using some of the "placeholder" db migrations
14:35:24 <rosmaita> but it turned out that wasn't going to really help
14:35:41 <rosmaita> so we are left with really long release notes explaining how to do it yourself
14:36:11 <rosmaita> that's all from me, unless anyone has questions
14:36:51 <whoami-rajat> excluding the part that they've renamed the __DEFAULT__ type, they just have to run migrations
14:38:37 <rosmaita> #topic open discussion
14:38:58 <kaisers> If i may, just a small note:
14:39:18 <kaisers> Our CI (Quobyte) is one of the failing 3rdparty CIs.
14:39:30 <kaisers> I've been trying to fix it in the last but that did not work out.
14:39:41 <kaisers> I'm currently setting up a completely new CI stack
14:39:49 <kaisers> So i hope i get this back online in the next days
14:39:58 <kaisers> <done>
14:40:06 <smcginnis> Thanks kaisers
14:40:11 <rosmaita> ok, thanks for the update
14:40:41 <whoami-rajat> i would like some feedback on the idea to backport https://review.opendev.org/#/c/741498/
14:40:41 <whoami-rajat> this doesn't change anything on a normal deployment just this allows the operators to delete the __DEFAULT__ type if they've
14:40:41 <whoami-rajat> set a valid default_volume_type in the conf
14:41:28 <rosmaita> on the plus side, that would make the behavior consistent from Train through Victoria
14:41:43 <rosmaita> and it doesn't really impact the upgrade issue we just talked about
14:41:48 <hemna> that patch makes setting the default_volume_type conf entry manditory
14:42:03 <hemna> which might bust older deployments if you are backporting this
14:42:11 <rosmaita> how?
14:42:12 <whoami-rajat> yep, and it defaults to __DEFAULT__
14:42:20 <hemna> if that setting isn't in cinder.conf
14:42:22 <hemna> what happens?
14:42:26 <hemna> since it's manditory
14:42:33 <hemna> cinder won't start?
14:42:36 <whoami-rajat> ^^
14:42:52 <rosmaita> well, if they don't set it, they get __DEFAULT__
14:43:00 <rosmaita> if they do set it, they get whatever they set
14:43:02 <hemna> so then it's not manditory
14:43:12 <whoami-rajat> hemna, https://review.opendev.org/#/c/741498/16/cinder/common/config.py
14:43:14 <eharney> i think mandatory here means it isn't set to empty/none ?
14:43:15 <rosmaita> well, depends on what you mean
14:43:15 * hemna has a confusion
14:43:25 <rosmaita> it's required but it also has a default value
14:43:34 <hemna> IMHO manditory means that you must set it in cinder.conf and have a valid value
14:43:49 <hemna> if it has a default value then it's not manditory to set it
14:44:11 <e0ne> hemna: +1
14:44:15 <hemna> it's confusing
14:45:18 <whoami-rajat> to preserve the current behavior, we have defaulted it to __DEFAULT__ so there
14:45:26 <whoami-rajat> 's no effort to set it manually
14:45:39 <whoami-rajat> but if you're going to set it, it should be a valid type
14:46:02 <whoami-rajat> also deleting the value set in default_volume_type is not allowed so we atleast have 1 type in deployment and don't allow untyped volumes
14:46:14 <rosmaita> of all the confusing things in cinder, this is like the least confusing thing i can think of! :)
14:47:11 <smcginnis> hemna: I had the same reaction. ;)
14:48:10 <rosmaita> i think what it comes down to is that we want to encourage operators to set that config value
14:48:42 <rosmaita> but, they don't have to
14:49:16 <rosmaita> you must have a default_volume_type correctly defined at all times, or you can't create volumes
14:49:33 <rosmaita> whether you let us define it as __DEFAULT__ or whether you do it yourself
14:49:52 <rosmaita> so that strikes me as "mandatory" in at least some aspect of the term
14:51:07 <whoami-rajat> all these changes are a part of a bug saying operators/users get *confused* with the name __DEFAULT__ when they've a different default_volume_type set
14:51:54 <hemna> I guess it's just the wording that makes it confusing is all.
14:52:41 <smcginnis> The confusion is all based on the fact that we didn't do that right and should have kept __DEFAULT__ a hidden internal thing if someone didn't provide a type.
14:52:49 <hemna> if it's manditory, then the deployer must set it.   but he doesn't because there is a default.   hence the confusing.
14:52:53 <rosmaita> it's not too late to revise the release note to be more clear
14:53:00 <rosmaita> https://review.opendev.org/#/c/741498/16/releasenotes/notes/allow-deleting-__DEFAULT__-type-d35dfb5d89760b9b.yaml
14:53:17 <rosmaita> so, anyone interested in this issue, please read through ^^
14:53:50 <rosmaita> we can get that clarified and then decide about the backports, squashing in the revised release note
14:56:17 <whoami-rajat> the reported bug mentions the train and ussuri releases so that's also a consideration to backport it
14:57:38 <rosmaita> coming up on 2 minutes left
14:58:59 <rosmaita> ok, looks like we're done for today!
14:58:59 <rosmaita> Please ... look at the release notes on the upgrade issue patches -- with Victoria release coming up, i think people will realize they need to upgrade to ussuri
14:59:09 <rosmaita> https://review.opendev.org/#/c/748481/
14:59:09 <rosmaita> https://review.opendev.org/#/c/748482/
14:59:34 <rosmaita> ok, thanks everyone!  let's clear out so horizon can start on time
14:59:39 <rosmaita> happy reviewing!
14:59:43 <rosmaita> #endmeeting