16:00:20 <jungleboyj> #startmeeting Cinder
16:00:21 <openstack> Meeting started Wed Oct  9 16:00:20 2019 UTC and is due to finish in 60 minutes.  The chair is jungleboyj. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:22 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:24 <openstack> The meeting name has been set to 'cinder'
16:00:44 <jungleboyj> Courtesy ping:  jungleboyj whoami-rajat rajinir lseki carloss pots woojay erlon geguileo eharney rosmaita enriquetaso e0ne smcginnis davidsha walshh_ xyang hemna _hemna tosky sfernand
16:00:45 <eharney> hey
16:00:51 <smcginnis> o/
16:00:53 <walshh_> hi
16:00:57 <xyang> hi
16:00:57 <whoami-rajat> Hi
16:00:58 <geguileo> hi! o/
16:00:59 <rosmaita> o/
16:01:06 <davidsha> o/
16:01:09 <e0ne> hi
16:01:15 <enriquetaso> o/
16:01:28 <dviroel> hi
16:01:34 <woojay> o/
16:01:35 <carloss> hi
16:02:20 <jungleboyj> @!
16:02:20 <_pewp_> jungleboyj ( *՞ਊ՞*)ﾉ
16:02:25 <jungleboyj> Look at all the people.
16:02:41 <tosky> o/
16:02:54 <rosmaita> big turnout
16:03:30 <jungleboyj> Figures given there is a light agenda.  :-)
16:03:59 <jungleboyj> Anyway, lets get started.
16:04:07 <jungleboyj> #topic announcements
16:04:23 <jungleboyj> Looks like the announcements are yours rosmaita ....
16:04:27 <rosmaita> ok
16:04:39 <rosmaita> first, RC-2 is available -- please test!
16:04:46 <jungleboyj> rosmaita: ++
16:05:02 <rosmaita> the "final RC" is whatever has been released on Friday (11 October)
16:05:27 <rosmaita> so if we have any critical bug fixes, we need to find, fix, and merge by Friday
16:05:48 <smcginnis> Which day?
16:05:52 <rosmaita> otherwise, if there's anything found after that, it's up to the release team (not us) whether it can be included in the release
16:06:01 <jungleboyj> smcginnis:  Oh no he didn't!
16:06:05 <jungleboyj> :-)
16:06:06 <smcginnis> :D :D
16:06:20 <jungleboyj> rosmaita:  Save yourself from the torture I got in the past.
16:06:37 * jungleboyj whispers just say Thursday
16:07:13 <rosmaita> oh, ok, Thursday (10 Oct)
16:07:18 <jungleboyj> :-)
16:07:23 <smcginnis> Good save. :P
16:07:31 <jungleboyj> smcginnis:  He is smarter than I am.
16:07:46 <rosmaita> anyway, i don't see any bugs filed or patches proposed for stable/train, and we are running out of time, so i think RC-2 will be our final RC
16:08:14 <jungleboyj> Good.  Nice that we could get it done in two RCs.  Thanks for the great leadership there rosmaita
16:08:24 <rosmaita> other announcement is the running reminder to please include topics on the PTG planning etherpad
16:08:34 <rosmaita> #link https://etherpad.openstack.org/p/cinder-ussuri-ptg-planning
16:08:55 <jungleboyj> rosmaita:  ++  If we are going all the way to Shanghai it should be productive.
16:08:58 <rosmaita> and i guess final announcement is Train coordinated release is one week from today!
16:09:26 <jungleboyj> Whooo whooo, chugga chugga ...
16:09:34 <rosmaita> that's all from me
16:09:54 <jungleboyj> rosmaita:  Awesome.  Thank you.
16:10:03 <rosmaita> :D
16:10:13 <jungleboyj> I only had one other topic for this week.
16:10:31 <jungleboyj> #topic Discuss Matt's proposal to remove legacy attach code
16:11:10 <jungleboyj> It looks like the last note in the thread is here:
16:11:14 <jungleboyj> #link http://lists.openstack.org/pipermail/openstack-discuss/2019-October/009959.html
16:11:40 <e0ne> can we really drop APIs with microversions and before cinder API v4?
16:11:57 <jungleboyj> Want to make sure that we get the people who understand this code involved.
16:12:39 <smcginnis> Yeah, I think we had realized we can't ever really drop code.
16:12:45 <jungleboyj> e0ne:  Good question.
16:12:55 <smcginnis> Even with a new microversion that excludes it, we still need the code there for the lower MVs.
16:13:03 <geguileo> we don't drop the API code, they just stop using it
16:13:06 <smcginnis> Nova can choose not to call the old code anymore of course.
16:13:07 <e0ne> smcginnis: +1
16:13:31 <geguileo> right now they may call both of them from the same host
16:13:45 <geguileo> depending on the instance and how/when the volumes where attached
16:13:53 <jungleboyj> But this isn't a question of removing it from Cinder, it is removing it from Nova.  Right?
16:14:02 <geguileo> yes
16:14:13 <smcginnis> We should figure out a good way to track things we do want to drop in case we ever do decide to do a v4 release and consolidate all mv changes into a new base version.
16:14:18 <geguileo> but to remove it from 'nova they need to migrate the BDM information to Cinder
16:14:34 <jungleboyj> smcginnis:  ++
16:17:23 <jungleboyj> geguileo:  So do we have a response to Matt's last note?
16:18:02 <geguileo> I don't have one yet (been sick in bed Mon and Tues)
16:18:20 <jungleboyj> geguileo:  Ugh.  :-(  Sorry to hear that.  Feel better.
16:19:06 <geguileo> I'm better now, thanks
16:19:12 <jungleboyj> geguileo:  If you are planning to respond though, I think we are good.  Just wanted to make sure that that wasn't forgotten.
16:20:14 <jungleboyj> So, I think that is all I had on that topic.
16:20:28 <jungleboyj> #topic Open Discussion
16:20:34 <geguileo> jungleboyj: I'm not entirely sure that it will work  :-(
16:20:47 <jungleboyj> :-(
16:20:58 <rosmaita> yeah, let's revisit this at next week's meeting
16:20:58 <geguileo> jungleboyj: in my opinion I would create a new microversion and accept the PUT method of the attachment to accept the connection information
16:21:25 <geguileo> rosmaita: sounds good
16:21:36 <jungleboyj> rosmaita: ++
16:21:42 <davee__> +1
16:21:51 <jungleboyj> Respond to Matt, see what he says and we can further discuss based on that.
16:22:11 <rosmaita> geguileo: that seems like a good idea, though, we make a call so that nova can pass us the "old" info and that way it won't get lost
16:22:43 <rosmaita> i mean, we don't make a call, we make a new call available for nova to use
16:22:58 <jungleboyj> Yeah.
16:23:06 <geguileo> rosmaita: we add functionality to the old attachments call
16:23:31 <rosmaita> ok
16:23:46 <jungleboyj> If we can't ever get rid of it though ... not the end of the world to help Nova get beyond it.
16:23:55 <jungleboyj> The Greater Good
16:25:05 <jungleboyj> Ok.  Any other topics for today?
16:25:25 <whoami-rajat> jungleboyj: just wanted to enquire if this is the right thing to do[1] since somehow the online migrations are already running on grenade jobs on gate (still looking into it)
16:25:25 <whoami-rajat> [1] https://review.opendev.org/#/c/686772/
16:25:31 <davee__> support it until a proper announcement with plenty of lead time for deprecation for users to update their environment appropriately
16:27:05 <jungleboyj> whoami-rajat:  Good question.
16:27:18 <jungleboyj> whoami-rajat:  I am not very familiar with Grenade.
16:27:24 <jungleboyj> eharney:  ^^^
16:27:33 <tosky> it's not much about grenade, I'd say
16:27:41 <smcginnis> I thought online migrations just ran at startup.
16:28:14 <tosky> grenade deploys using devstack, then it updates the code using the same configuration and some scripts which follows the suggested upgrade procedures
16:28:40 <tosky> the question is more whether the online migrations needs to be run automatically or not; the documentation seems to hint that they need to be executed manually
16:29:03 <tosky> or at least that's how my unexperienced eye reads them :) (looking for the link...)
16:29:41 <tosky> https://docs.openstack.org/cinder/latest/upgrade.html?highlight=online#online-data-migrations
16:30:47 <whoami-rajat> this man page [1] states 'This command should be run after upgrading the database schema.' which is exactly what i did in my patch
16:30:47 <whoami-rajat> [1] https://docs.openstack.org/cinder/latest/cli/cinder-manage.html
16:32:16 <jungleboyj> whoami-rajat:  But you think that it is already getting run even without your patch in place?
16:33:08 <smcginnis> Would be odd that we would need explicit handling for Cinder. Should be the same for all services.
16:33:12 <smcginnis> At least I would hope.
16:33:33 <whoami-rajat> smcginnis: if it runs after startup, is nova following a different approach to it[1]?
16:33:33 <whoami-rajat> https://opendev.org/openstack/grenade/src/branch/master/projects/60_nova/upgrade.sh#L133-L136
16:33:57 <smcginnis> whoami-rajat: Oh? So you're just doing the same thing nova is doing?
16:34:41 <tosky> that's the idea of whoami-rajat's patch, the question is: is it the right way? Is the documentation unclear?
16:34:50 <smcginnis> That looks like it should be fine then.
16:35:14 <jungleboyj> smcginnis:  ++
16:35:20 <davee__> smcginnis pulls pin on grenade and tosses to whoami-rajat ...
16:35:26 <smcginnis> :)
16:35:51 <whoami-rajat> jungleboyj:  yes, initially my patch failed due to a dependency on previous online migration, but when grenade updated master, it passed
16:35:53 <jungleboyj> He he
16:36:12 <jungleboyj> whoami-rajat:  Ok, than I think that is probably the direction we want to go.
16:36:24 <smcginnis> Unfortunately I think dulek and scottda were the ones that knew all of the DB migration stuff well.
16:36:59 <geguileo> well, we had a problem with online migration....
16:37:01 <jungleboyj> smcginnis:  Unfortunately.  I used to know it to some extent in the past but have been totally disconnected on the online stuff.
16:37:08 <geguileo> where we didn't follow the basics of it
16:37:16 <geguileo> and required services to be up and running
16:37:38 <smcginnis> Which is weird, because I would expect "online" migrations to require that anyway.
16:37:39 <geguileo> which is a no-no for online data migrations
16:37:59 <smcginnis> But yeah, I think it would be useful if all of the expectations and things were better documented somewhere.
16:38:01 * jungleboyj is so confused
16:38:10 <jungleboyj> So what does Online mean?
16:38:10 <smcginnis> Any maybe they are, but I don't know where.
16:38:11 <geguileo> smcginnis: I think I added it in a patch
16:38:33 <tosky> but then we may want to recheck how to make sure that granade (or any upgrade test) catches any issue linked to *not* executing the online migrations
16:39:27 <davee__> I t would be a reasonable assumption that any patch that uses a modified DB schema should also perform a DB Migration as part of that patch
16:40:07 <geguileo> https://opendev.org/openstack/cinder/src/branch/master/cinder/cmd/manage.py#L244
16:40:09 <tosky> does it mean that the online migration code is executed automatically?
16:40:14 * tosky confused
16:41:44 <geguileo> tosky: no, I think it is executed as part of any upgrade
16:41:44 <geguileo> the whole upgrade thing is quite complex
16:42:05 <smcginnis> "online-migrations" seems to be a bit of a misnomer.
16:42:16 <jungleboyj> So what makes them online vs any other DB upgrade?
16:42:20 <jungleboyj> smcginnis:  ++
16:42:30 <geguileo> but in sumary it's something like this
16:42:30 <geguileo> you make some changes in the DB that require either new info or info that you can get from other parts of the DB
16:42:31 <geguileo> then you add online migration code to your component
16:42:45 <geguileo> that runs every time you load the object and see it doesn't have the new data
16:42:58 <geguileo> so you load, check, if old, then change, and save
16:43:13 <geguileo> that way you are postponing the changes until the objects are used
16:43:15 <geguileo> and spread the load
16:43:34 <geguileo> then, for the next release you have an online migration code on the manager
16:43:49 <jungleboyj> Ok.
16:43:58 <geguileo> that does that serially, without requiring the services running (in case this is an FFU and not a rolling upgrade)
16:44:32 <geguileo> and on that same release you can remove the online migration from your service code so it doesn't get checked on the loads
16:44:54 <geguileo> because the cinder-manage online migration code will be run before the new services are started
16:45:12 <davee__> geguileo gets it  +2
16:45:20 <smcginnis> So these can be removed now on master, right? https://opendev.org/openstack/cinder/src/branch/master/cinder/cmd/manage.py#L253
16:45:21 <geguileo> maybe you cannot remove it on the next (N+1) but instead in N+2
16:45:31 <geguileo> I'm not entirely sure about that one
16:45:52 <geguileo> but when you run the cinder-manage online migrations, the amount of data to migrate should be considerably smaller
16:46:06 <geguileo> and therefore faster
16:46:10 <smcginnis> The code comment you added indicated N+1. I think we still state we only do single hop upgrades, but I suppose it would be a small performance impact to keep things around an extra release just in case.
16:46:39 <geguileo> smcginnis: you can remove the cinder-manage code in N+1
16:47:01 <geguileo> smcginnis: not sure about the service online migration code though, in case you are running rolling upgrades
16:47:07 <geguileo> I would have to think about it to be sure
16:47:23 <smcginnis> Ah, I think I get it.
16:47:24 <smcginnis> Thanks
16:47:30 <davee__> always a good idea to include a versioning field in the db schema aand base upgrade logic on that version number so you know what needs to be done to end up in a stable operational state
16:47:47 <smcginnis> That's basically how it works.
16:48:01 <jungleboyj> geguileo:  ++ Yeah, that is helpful.
16:48:05 <smcginnis> At least for the schema. Not the data that these appear to address.
16:49:30 <tosky> when everything is confirmed, could the documentation at https://docs.openstack.org/cinder/latest/upgrade.html?highlight=online#online-data-migrations be updated then to explain that it's not strictly needed, but it just speeds up everything?
16:49:42 <whoami-rajat> geguileo: so the online migration running is related to upgrade process and not cinder right? it might be a part of the volbak playbook or the upgrade process grenade follows ?
16:49:44 <tosky> (or whethever is the expected behavior)
16:50:09 <geguileo> whoami-rajat: the cinder-manage online migration is part of the upgrade
16:50:22 <geguileo> whoami-rajat: the online migrations should also be part of the cinder service code
16:52:14 <smcginnis> That's part of the confusion - there are two different "online migration" steps.
16:52:48 <jungleboyj> There is the part that runs online as part of the service and the part that runs later during upgrade?
16:53:00 <whoami-rajat> geguileo:  so it should be run manually during an upgrade or is automatically associated with the startup of services?
16:53:03 <geguileo> jungleboyj: yup, great summary :-)
16:53:20 <geguileo> whoami-rajat: it should be handled by the upgrade mechanism
16:53:24 <geguileo> whoami-rajat: whatever that is
16:53:41 <jungleboyj> geguileo:  Ah, so I am getting it.  So that is why people got in trouble is they had originally written it to assume the service was up but then when they moved it to upgrade and the service wasn't up, it blew up.
16:53:58 <geguileo> jungleboyj: yup
16:54:08 <geguileo> jungleboyj: it'll only be up if we are doing rolling upgrades
16:54:31 <geguileo> but if you do a forward upgrade from N to N+3, then you bring them down
16:54:48 <geguileo> and you run db-sync and online upgrade on each of the intermediary releases
16:54:49 <jungleboyj> geguileo:  Ok.  Good to know.
16:55:00 <jungleboyj> This makes more sense now.
16:55:11 <davidsha> Just want to get this in quickly before the meeting ends, I think this bug may have gone under the radar: https://bugs.launchpad.net/os-brick/+bug/1843431
16:55:11 <openstack> Launchpad bug 1843431 in os-brick "NVMeOF connector fails to disconnect a volume with the latest nvme-cli" [Undecided,New]
16:56:22 <smcginnis> Oh great. Any code ready for that yet?
16:56:33 <jungleboyj> davidsha:  Thanks for raising.  Look like we have someone fixing it.
16:57:20 <davidsha> This patch is it? https://review.opendev.org/#/c/683917/
16:57:46 <smcginnis> Ah, just didn't get linked to the bug report.
16:58:00 <jungleboyj> Weird.
16:58:11 <smcginnis> One other quick thing before we end - we had a few late driver submissions in train.
16:58:23 <smcginnis> I've put them under a better topic to help track them for ussuri.
16:58:29 <smcginnis> https://review.opendev.org/#/q/topic:ussuri-drivers+(status:open+OR+status:merged)
16:58:51 <smcginnis> Any other ones, please update the topic to ussuri-drivers to help us keep track of what's out there.
16:59:08 <jungleboyj> smcginnis:  Thanks!
16:59:32 <davidsha> Thanks!
17:00:11 <jungleboyj> Ok.
17:00:31 <jungleboyj> Our time is up.  Thank you everyone.
17:00:38 <jungleboyj> #endmeeting