17:00:26 <ildikov> #startmeeting cinder-nova-api-changes
17:00:26 <openstack> Meeting started Mon Nov 14 17:00:26 2016 UTC and is due to finish in 60 minutes.  The chair is ildikov. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:00:27 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:00:30 <openstack> The meeting name has been set to 'cinder_nova_api_changes'
17:00:51 <ildikov> scottda ildikov DuncanT ameade cFouts jaypipes takashin alaski e0ne jgriffith tbarron andrearosa hemna erlon gouthamr ebalduf patrickeast smcginnis diablo_rojo gsilvis  xyang1 raj_singh lyarwood
17:01:04 <jgriffith> o/
17:01:47 <ildikov> jgriffith: hi :)
17:02:14 <scottda> Hi
17:02:22 <jgriffith> howdy
17:02:51 <ildikov> Nova guys are busy today, so I agreed with them to continue the discussion on johnthetubaguy's spec: https://review.openstack.org/#/c/373203/
17:03:20 <smcginnis> o/
17:03:35 <scottda> I'm in transit ATM, in case I don't respond promptly
17:03:46 <ildikov> I thought we might have a chat still to see the roadmap of the Cinder API patch/activity https://review.openstack.org/#/c/387712/
17:04:18 <ildikov> scottda: copy that
17:04:26 <smcginnis> jgriffith: Need to go through your latest updates.
17:04:38 <smcginnis> jgriffith: Do we have a feel yet whether v1 or v2 are in favor?
17:04:55 <jgriffith> smcginnis: v3 :)
17:05:05 <smcginnis> lol
17:05:19 <jgriffith> smcginnis: I *think* the v2 updates are the direction that most closely satisfy the nova spec
17:05:22 <ildikov> smcginnis: we're getting there to have create/delete calls
17:05:54 <jgriffith> smcginnis: so it basically has 1 extra call for nova if they want it, otherwise it's pretty similar to the v1 approach
17:05:54 <ildikov> smcginnis: additionally we still have reserve in the patch, I guess that's for things like shelve
17:06:10 <smcginnis> OK, cool. My preference was v1, but like I think I mentioned in BCN, if we can't get Nova to agree with the change then it doesn't do much good.
17:06:20 <smcginnis> ildikov: Shelve looked like the tricky one.
17:06:26 <ildikov> smcginnis: and the latest chat on the review is whether we call 'update' as 'finalize'
17:06:36 * ildikov likes nit picking about names :)
17:06:36 <jgriffith> ildikov: smcginnis yeah; so the bigest thing is an "attachment_create" does a reserve on the volume (status = 'reserved') and creates an empty attachment
17:06:42 <jgriffith> ildikov: :)
17:06:46 <smcginnis> ildikov: Saw that. I agree, it's minor, but naming is important long term.
17:07:17 <smcginnis> finalize_reservation_or_do_update_to_existing()
17:07:25 <smcginnis> Rolls off the tongue.
17:07:31 <jgriffith> from my side it would be awesome to see if the general API's and workflow are the right direction for folks or not
17:07:33 <ildikov> jgriffith: is that 'reserve' something that we expose on the API?
17:07:43 <jgriffith> I'm hesitant to write much more code if this isn't what people want to see
17:08:06 <jgriffith> ildikov: nope; it's the "attachment_create" call
17:08:15 <ildikov> jgriffith: ok
17:08:39 <jgriffith> ildikov: the requirement for nova was the whole empty attachment thing; and they wanted to get rid of reserve (which makes sense to me) so that's what that call does now
17:08:55 <ildikov> jgriffith: would something like that be needed for shelve - as it kinda reserves the volume - or we're confident about having 'attachment_create' is enough?
17:08:56 <jgriffith> just sets status to reserved and returns an ID of an empty attachment object
17:09:38 <jgriffith> ildikov: yeah, for things like creating an attachment for a shelved Instance and for boot-from-volume that's the sort of thing you'd need this for
17:10:05 <jgriffith> ildikov: shelving in and of itself isn't problematic, attaching to a shelved Instance is the trick
17:10:29 <jgriffith> ildikov: even if it is sort of a strange use case
17:10:33 <ildikov> jgriffith: yeap, right, a bit of a nightmare-ish thing...
17:11:25 <ildikov> smcginnis: that's a bit of a long name :)
17:11:40 <smcginnis> ildikov: ;)
17:12:08 <ildikov> smcginnis: but at least contains everything, so I stop complaining
17:12:17 <smcginnis> Hah!
17:12:38 * ildikov uses Mondays to complain, sorry about that... :)
17:13:13 <ildikov> jgriffith: is your v2 patch containing v3-like changes up to date regarding the things we talked about above?
17:15:21 <smcginnis> jgriffith: I'll try to review this afternoon.
17:15:27 <smcginnis> Sorry, gotta run for a lunch appointment.
17:15:32 <ildikov> smcginnis: do you think it would worth having 2 minutes about this on the Cinder meeting this week to get more eyes on this from the team?
17:15:46 <smcginnis> ildikov: Yeah, certainly wouldn't hurt.
17:15:58 <smcginnis> ildikov: If I don't add it to the agenda, feel free to remind me.
17:16:12 <ildikov> smcginnis: ok cool, will do that tnx!
17:16:58 <ildikov> smcginnis: we need more reviews on this as it's a new API, which is hard to change later if people realize they don't like it
17:17:23 <jgriffith> ildikov: s/"hard to change"/"impossible to change"/
17:18:22 <ildikov> jgriffith: I try not to be this dramatic at the beginning of the week, but you certainly have a point
17:19:15 <ildikov> anything else for today?
17:19:44 <ildikov> #action smcginnis to add the API proposal topic to this week's Cinder meeting
17:20:53 <ildikov> jgriffith: I've just seen your reply on the review
17:21:09 <jgriffith> ildikov: yeah, hopefully those responses *help*
17:21:12 <ildikov> jgriffith: do you want to chat about the naming here or let's just leave it on the review?
17:21:17 <jgriffith> ildikov: I'm certainly fine with changing the naming around
17:21:29 <jgriffith> ildikov: totally up to folks here
17:21:44 <jgriffith> ildikov: I'm certainly not going to draw a line in the sand over naming
17:21:56 <jgriffith> ildikov: but be careful, or else the API will end up with a desert related name :)
17:22:06 <ildikov> jgriffith: as for me it's really only about the name, so I don't want to change the planned functionality on the API
17:23:11 <ildikov> jgriffith: personally I wouldn't mind, I have a good relation with deserts :)
17:23:34 <jgriffith> haha
17:23:36 <xyang1> jgriffith: hi, question for you about the comments here: https://review.openstack.org/#/c/387712/5/cinder/volume/manager.py@4391
17:23:53 <jgriffith> xyang1: yeah?
17:23:55 <ildikov> jgriffith: let's keep the discussion on the review then and give people the chance to come up with something sane
17:24:12 <xyang1> jgriffith: do you mean the use case described my Jay is not supported yet
17:24:18 <jgriffith> ildikov: works for me; it's easy to change to "update" if that's where we end up
17:24:41 <jgriffith> xyang1: the use case described by jay.xu appears to be multi-attach
17:25:01 <jgriffith> xyang1: so correct, it's not implemented here *yet* but the design allows it
17:25:04 <ildikov> jgriffith: +1
17:25:21 <xyang1> jgriffith: sorry, I am lost
17:25:36 <jgriffith> xyang1: ok... how can I help *find* you :)
17:25:42 <xyang1> :)
17:26:29 <xyang1> jgriffith: volA attached to VM1 and VM2 on the same host
17:26:42 <jgriffith> xyang1: so that's multi-attach
17:27:00 <jgriffith> xyang1: and we will have the ability via attachments table to detect and notify Nova of that
17:27:08 <jgriffith> xyang1: that's actually the *easy* case
17:27:19 <jgriffith> xyang1: the case that is commented in the review is different
17:27:53 <jgriffith> xyang1: Nova has expressed a case where VolA and VolB attached to InstanceX and InstanceY shares a single connection to HostZ
17:28:09 <jgriffith> xyang1: I don't know anything about what backends work like this or how they work :(
17:28:43 <jgriffith> xyang1: but regardless that would be pretty driver-specific, so I figured that has to live in the backend/driver
17:29:15 <jgriffith> xyang1: when we do multi-attach VolA attached to InstanceX, InstanceY on HostZ... that's easier to address
17:29:16 <xyang1> jgriffith: so our backend will create a different host lun number for every volume, so in this case, that may be 2 connections instead of 1
17:29:26 <jgriffith> xyang1: Cinder will have enough information to understand that
17:29:52 <jgriffith> xyang1: right, ALL backends that I'm aware of will have a different connection (lun ID)
17:30:21 <jgriffith> xyang1: but this is something that the Nova side has asked for specifically... it may be an NFS, or RBD thing
17:30:32 <jgriffith> xyang1: I fully admit I'm not very familiar with those backends
17:30:37 <jgriffith> and how they work
17:30:49 <xyang1> jgriffith: so seems we are confused with the shared connection definition
17:30:59 <jgriffith> xyang1: I'm not confused :)
17:31:08 <jgriffith> xyang1: I understand it perfectly :)
17:31:31 <xyang1> I mean Jay and I:)
17:31:43 <jgriffith> xyang1: I know... I'm just giving you grief :)
17:32:19 <xyang1> If I want to test this patch, which client patch is the right one
17:32:34 <jgriffith> xyang1: none of them currently I'm afraid
17:32:44 <xyang1> ok
17:32:46 <jgriffith> xyang1: so if I can get reviews on what's up there and the API I'll move forward
17:33:04 * johnthetubaguy sneaks in the back
17:33:12 <jgriffith> xyang1: but I spent a lot of time on the last couple of iterations and never got anywhere; so I'm admittedly being kinda lazy
17:33:29 <jgriffith> xyang1: not dumping a bunch of code until we're at least somewhat decided on this direction
17:33:33 <ildikov> jgriffith: I can look into the Nova patch in parallel when we get to this stage
17:33:41 <jgriffith> johnthetubaguy: is that you back there sneaking in late?
17:34:04 <jgriffith> johnthetubaguy: I'll remind you that this class begins promptly at 17:00 :)
17:34:05 <ildikov> johnthetubaguy: we're talking about shared connections, so if you have anything to add, this is the time :)
17:34:22 * jgriffith remembers those instructors back in school
17:34:48 <johnthetubaguy> heh
17:34:58 <ildikov> jgriffith: LOL :)
17:35:22 <johnthetubaguy> shared connections suck, but it seems they exist
17:35:39 <ildikov> jgriffith: you can help me reminding kinds to do their homeworks too as you know all the key expressions ;)
17:35:53 <jgriffith> haha, sadly I do
17:36:05 <johnthetubaguy> the nova spec goes into quite a lot of detail on how shared connections would work with the new proposed flow
17:36:09 <johnthetubaguy> with a few diagrams
17:36:12 <jgriffith> johnthetubaguy: yeah, they're annoying but as long as they're a thing we'll have to deal with them
17:36:23 <johnthetubaguy> jgriffith: yup
17:36:55 <ildikov> johnthetubaguy: BTW, do we have anything on the gate to test this use case?
17:38:15 <johnthetubaguy> ildikov: not on our side, AFAIK
17:38:56 <jgriffith> johnthetubaguy: if we can just figure out the "who" on those I can track down ways to get it tested
17:39:18 <johnthetubaguy> yeah, unsure if the os-brick folks have more details on the who
17:39:19 <jgriffith> johnthetubaguy: I thought Ceph was one of those cases (but perhaps I'm wrong) and NFS types
17:39:28 <johnthetubaguy> I am assuming its some fibrechannel thingy
17:39:36 <jgriffith> johnthetubaguy: I pinged hemna on that the other day and got nothing
17:39:47 <jgriffith> johnthetubaguy: even FC uses luns for each volume
17:39:59 <xyang1> jgriffith: +1
17:40:36 <johnthetubaguy> who knows when to drop the FC connection, or is that not really an option?
17:40:44 <xyang1> jgriffith: our FC backend creates a host lun id for every volume
17:41:28 <jgriffith> johnthetubaguy: doesn't really work that way; and if it does via zone-manager you drop it when you no longer have any devices; BUT the way that usually works is FC is *always* connected (connected in quotes)
17:41:39 <jgriffith> johnthetubaguy: you just may not have any devices mapped/connected
17:41:46 <jgriffith> johnthetubaguy: FC is just scsi
17:42:02 <johnthetubaguy> fair enough
17:42:02 <jgriffith> johnthetubaguy: scsi over FC... thus sometimes confusing
17:42:34 <johnthetubaguy> we should find out who does the shared thing
17:42:48 <jgriffith> johnthetubaguy: agreed.. maybe dev ML?
17:43:04 <johnthetubaguy> I guess
17:43:05 <johnthetubaguy> can I leave that with you cinder folks?
17:43:10 <jgriffith> johnthetubaguy: ildikov I can post a question up there if folks think it's helpful?
17:43:13 <jgriffith> johnthetubaguy: yep
17:43:19 <johnthetubaguy> thanks that would be good
17:43:21 <jgriffith> johnthetubaguy: we can/should figure that out
17:43:24 <johnthetubaguy> +1
17:43:25 <ildikov> jgriffith: that would be great, thanks!
17:43:41 <jgriffith> johnthetubaguy: ildikov who knows, maybe we get lucky and this goes away :)
17:43:51 * ildikov started to wonder whether shared connection is just an urban legend...
17:44:00 <jgriffith> although it's trivial to solve
17:44:04 <johnthetubaguy> ildikov: was thinking the same
17:44:19 <johnthetubaguy> so the latest version just drops a bit of locking logic, so thats not so bad
17:44:23 <jgriffith> There's no Urban Legends in OpenStack!!!!
17:44:24 <jgriffith> :)
17:44:36 <ildikov> LOL :)
17:44:54 <johnthetubaguy> countryside legends?
17:44:59 <jgriffith> Ok, I'll try and track down any cases where a connection is shared
17:45:24 <ildikov> it's good though if both of you feel it's an easy on to solve, I hope we will have only this kind if any
17:45:47 <jgriffith> ildikov: doesn't hurt to keep it in the specs/patches as is currently
17:46:04 <ildikov> jgriffith: I didn't mean to remove it
17:46:17 <jgriffith> and best of all requires the drivers who care about it to deal with it
17:46:42 <jgriffith> ildikov: no, I know what you meant; I'm just saying that it's not intrusive to leave the solution we have in place to deal with it
17:46:42 <johnthetubaguy> do we want to talk live-migrate, I think thats what we were doing at the end of the last hangout?
17:46:43 <ildikov> jgriffith: I just got surprised that after talking about it that much no one here knows a single example :)
17:46:43 <jgriffith> anyway
17:47:13 <jgriffith> johnthetubaguy: sure
17:48:05 <jgriffith> johnthetubaguy: my thought there was we just create a new attachment and connect like we do currently.  We just provide an option to indicate that it's part of a migration so even if we're not supporting multi-attach allow this to be called
17:48:07 <ildikov> jgriffith: I think we're on the same page :)
17:48:34 <johnthetubaguy> I tried to cover that all here:
17:48:35 <johnthetubaguy> http://docs-draft.openstack.org/03/373203/12/check/gate-nova-specs-docs-ubuntu-xenial/6e33afc//doc/build/html/specs/ocata/approved/cinder-new-attach-apis.html#live-migrate
17:48:42 <jgriffith> johnthetubaguy: so Nova would still create a new attachment, it would just have two attachments for the same volume for a brief period of time
17:48:56 <johnthetubaguy> yep, thats what I would like
17:49:04 <jgriffith> johnthetubaguy: your wish is my command
17:49:05 <jgriffith> :)
17:49:18 <johnthetubaguy> for me, its clear its related to a migration, because both attachments have the same instance_uuid
17:49:19 <jgriffith> ahh... yeah, doc works
17:49:33 <jgriffith> johnthetubaguy: ahh, good point
17:49:43 <jgriffith> johnthetubaguy: in that case I wouldn't even need to clutter the API with anything
17:49:45 <ildikov> johnthetubaguy: we might think about how to make it clear for everyone
17:49:48 <jgriffith> johnthetubaguy: that's even better
17:50:11 <ildikov> johnthetubaguy: it got clear to me too, when you reminded me, but for anyone who does not follow this activity it will still be tricky
17:50:30 <jgriffith> ildikov: johnthetubaguy I'll exploit the Instance UUID to detect it and add a field to the attachment record to idicate it's being migrated
17:50:37 <johnthetubaguy> multi_attach=True means allows multiple server_uuids
17:50:41 <jgriffith> ildikov: johnthetubaguy or actually just set the attach-status to "migrating"
17:51:10 <johnthetubaguy> honestly, this is a bit of our shared model we need people to understand, else we will hit nasty bugs
17:51:12 <jgriffith> johnthetubaguy: to clarify server_uuids == instance_uuids ?
17:51:19 <johnthetubaguy> yeah
17:51:27 <johnthetubaguy> Nova API says "server" not instance
17:51:30 <ildikov> jgriffith: johnthetubaguy: what if for some reason the second attach request happens by mistake?
17:51:38 <jgriffith> johnthetubaguy: right, just wanted to clarify
17:51:46 <johnthetubaguy> a "server" is on a "host", which is sadly a little unclear
17:51:55 <ildikov> johnthetubaguy: right, I will need longer transition time for that in my head, my bad, sorry :S
17:52:14 <jgriffith> johnthetubaguy: should I change the attach API's s/instance_xxx/server_xxxx/g ?
17:52:18 <johnthetubaguy> ildikov: we know which host it was, and that will probably need manual cleanup
17:52:30 <jgriffith> ildikov: second attach by accident?
17:52:39 <jgriffith> johnthetubaguy: +1
17:52:52 <jgriffith> or just "don't do that" :)
17:53:10 <ildikov> jgriffith: I personally didn't want to :)
17:53:15 <johnthetubaguy> jgriffith: it probably wants to be nova agnostic I guess, so maybe we need something that is easier to get
17:53:34 <johnthetubaguy> the evacuate case makes things interesting, FWIW
17:53:42 <johnthetubaguy> but thats the next discussion
17:53:48 <jgriffith> johnthetubaguy: well I'm already faking it anyway... if it's nova calling I use instance_uuid, if not I use the host
17:54:04 <ildikov> johnthetubaguy: will this renaming be cleaned up in the Nova code ever?
17:54:23 <johnthetubaguy> ildikov: seems unlikely, way too hard
17:54:33 <ildikov> johnthetubaguy: do you mean the questions about who puts the attachment into 'error' state, etc?
17:54:42 <johnthetubaguy> yeah
17:54:51 <ildikov> johnthetubaguy: that's what I thought just wanted confirmation from an insider
17:54:54 <johnthetubaguy> so os-brick disconnect fails
17:55:35 <johnthetubaguy> I think put attachment into the error state is useful
17:56:29 <ildikov> is that about evacuate or os-brick disconnect in general?
17:56:36 <johnthetubaguy> in general at first
17:57:26 <johnthetubaguy> I think I talk about it here: http://docs-draft.openstack.org/03/373203/12/check/gate-nova-specs-docs-ubuntu-xenial/6e33afc//doc/build/html/specs/ocata/approved/cinder-new-attach-apis.html#spawn-and-delete
17:57:45 <johnthetubaguy> basically, the volume is attached from the VM, but not disconnected from the host
17:57:57 <johnthetubaguy> so put the attachment into an ERROR state, so we know whats happening
17:58:18 <johnthetubaguy> that would allow someone else to attach to that volume, as its not attached on first VM any more, I guess
17:59:25 <ildikov> I guess except if the VM is on the same host as the previous one?
17:59:39 <johnthetubaguy> well, that would probably fail the connect
17:59:48 <ildikov> although I'm not an expert of what that means practically of disconnect fails
17:59:52 <johnthetubaguy> which is where we just delete the attachment
18:00:08 <johnthetubaguy> me neither, honestly
18:00:15 <johnthetubaguy> its just cases that have been brought up
18:00:25 <johnthetubaguy> I have attempted to go through them.
18:01:49 <ildikov> I think we need some os-brick insides to go through these cases in details
18:02:51 <ildikov> jgriffith: will that be Cinder in the future who sets the state of the attachment to 'error'?
18:03:53 <jgriffith> ildikov: the concept I thought we discussed with Matt was to allow them to just then delete/remove the attachment
18:04:29 <jgriffith> ildikov: wait...for the disconnect case?
18:04:48 <ildikov> jgriffith: for example for that
18:05:25 <jgriffith> ildikov: I guess we could add the ability to put the attachment into error state, but I'm unclear what that buys us
18:05:58 <jgriffith> ildikov: error, removed... what would the difference be?
18:06:09 <ildikov> jgriffith: I guess the question here is how clean up will happen in certain cases and what info is needed for that, etc.
18:06:26 <jgriffith> ildikov: in either case we would need to dissociate the volume with that attachment record anyway
18:06:44 <jgriffith> ildikov: that's kind of a brick problem isn't it?
18:06:46 <ildikov> jgriffith: if os-brick is enough to figure that out, then removing the attachment sounds fine to me
18:07:06 <jgriffith> ildikov: I mean from cinder's perspective, disassociate the volume from the connection so it's not tied up any longer
18:07:22 <ildikov> jgriffith: my thought was just that we kind of removing the "evidence" of something went wrong
18:07:43 <jgriffith> ildikov: you'll still have logs though
18:07:46 <ildikov> jgriffith: which might be bad
18:08:12 <jgriffith> and honestly this goes down the philosophy of a hyper HA device vs a cloud IMO
18:08:47 <jgriffith> I guess I'd have to think/talk through this case more
18:08:48 <ildikov> fair enough
18:09:06 <jgriffith> maybe it's more important/critical than I'm thinking
18:09:27 <ildikov> I think we would need someone who knows os-brick better and see how failover works
18:09:32 <jgriffith> but I don't see it as a really big deal or terrible thing.  We currently deal with it a little in Brick and a little in Cinder
18:11:23 <ildikov> I guess we can start with how it's handled today and see where that leads us
18:11:44 <jgriffith> ildikov: seems reasonable
18:11:46 <ildikov> if there's any chance not to overcomplicate things I'm all in for that
18:12:38 <ildikov> we're over time for today, but we can bring this up next week and see what others think
18:12:52 <ildikov> jgriffith: johnthetubaguy: does it sound ok? ^^
18:13:11 <jgriffith> ildikov: sounds good to me
18:13:57 <ildikov> coolio
18:14:43 <ildikov> then I let you go for today :)
18:14:49 <ildikov> have a nice day/evening!
18:15:11 <ildikov> talk to you next week the latest!
18:15:51 <ildikov> #endmeeting