16:01:38 <jgriffith> #startmeeting
16:01:39 <openstack> Meeting started Wed Jun  6 16:01:38 2012 UTC.  The chair is jgriffith. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:01:40 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:02:23 <jgriffith> Folks are typicall a bit late so we'll give them a few minutes
16:02:30 <jgriffith> renuka: how did the meetup go?
16:02:30 <renuka> sure
16:02:56 <renuka> jgriffith: it was good, i think... I may have had more beginners than we expected
16:03:06 <clayg> o/
16:03:47 <jgriffith> Now we're getting somewhere :)
16:04:01 <renuka> jgriffith: Here's my slides if anyone finds them useful http://www.slideshare.net/aptegetusername/openstack-cinder
16:04:32 <sleepsonthefloor> hello!
16:04:46 <jgriffith> renuka: Very cool, thanks!
16:04:52 <jgriffith> sleepsonthefloor: Howdy
16:05:06 <jgriffith> Ok, we've got a descent turn out today so let's get started
16:05:13 <jgriffith> #topic status
16:05:36 <jgriffith> The past week has been really good thanks to a lot of help from sleepsonthefloor
16:05:57 <jgriffith> Currently he's ben able to use the cinder service in devstack
16:06:06 <jgriffith> This includes creating a volume and attaching it
16:06:51 <jgriffith> There's a bunch of work in Draft status right now in both the client and in nova
16:07:07 <jgriffith> I think by the end of the week we should be ready to push these out
16:07:24 <jgriffith> sleepsonthefloor: Do you want to mention anything about the keystone work you're doing now?
16:07:49 <sleepsonthefloor> jgriffith: sure - we need to do a little tweaking to the keystone middleware to pass a service catalog...
16:08:08 <sleepsonthefloor> this will allow services like nova to know where to find services like cinder
16:08:37 <sleepsonthefloor> right now, we are hard-coded to find cinder at 127.0.0.1
16:09:02 <sleepsonthefloor> in the long run, glance and quantum should also use the catalog to find their respective services, I believe
16:09:12 <sleepsonthefloor> (right now they are defined in flags)
16:09:24 <sleepsonthefloor> should have that today
16:09:29 <jgriffith> Excellent!
16:09:38 <jgriffith> That might change my strategy a bit...
16:09:48 <clayg> I thought when nova-api was serving the openstack/volumes api - it already required an entry in the service catalog for python-novaclient
16:09:55 <clayg> ^ same for glance
16:09:56 <uvirtbot> clayg: Error: "same" is not a valid command.
16:10:23 <renuka> sleepsonthefloor: (pardon my ignorance) but why can't they continue to be flags? or are we combining some amount of cleanup with this?
16:11:00 <sleepsonthefloor> clayg: yes that is the service catalog is returned to the nova client yes, but that information is not readily available in nova
16:11:38 <clayg> sleepsonthefloor: ah yes, I see your point, nova flags define where glance is... same thing for now for cinder
16:12:04 <sleepsonthefloor> renuka: I mention that as an aside, that would not be a cinder-related change
16:12:45 <renuka> sleepsonthefloor: gotcha
16:12:56 <jgriffith> sleepsonthefloor: So we could still use a flag for "which" volume service to use and then the catalogue to figure out where/how to connect if it's cinder yes?
16:12:57 <sleepsonthefloor> clayg: when nova validates a token, keystone does not return the service catalog associated with the token - need some tweaking to make that happen
16:13:22 <sleepsonthefloor> jgriffith - yes
16:13:24 <jgriffith> sleepsonthefloor: Or more accuratley which volume_api to use
16:13:52 <sleepsonthefloor> jgriffith: I think we would stick with the flag for which volume.api
16:14:16 <clayg> are their any relevant nova patches needed with devstack c/7042 - or is everything mostly already in nova and cinder master
16:14:16 <jgriffith> sleepsonthefloor: Sounds good
16:14:29 <sleepsonthefloor> clayg: yes there are several
16:14:50 <clayg> i didn't see them in renuka's awesome slides (thanks renuka!)
16:14:56 <renuka> I remember there was some talk in the summit before last about how we should "expose" a volume to a host only when an attach request comes in, for better security... have we planned for that in any way?
16:15:08 <renuka> clayg: sure :)
16:15:21 <sleepsonthefloor> clayg: there are patches for devstack + cinder + cinder client + nova and soon to be keystone
16:15:32 <jgriffith> renuka: I think with having the service outside of nova that's going to happen by default
16:16:05 <clayg> I think cinder could choose do to the export in initialize_connection instead of create_volume, nova-compute would have all relevant info when making the initialze call
16:16:37 <jgriffith> clayg: exactly
16:17:11 <clayg> sleepsonthefloor: ok, well if you can get a list of working patches that I should be testing - I can help test
16:17:24 <sleepsonthefloor> clayg: great - I will do that one sec
16:17:25 <jgriffith> renuka: clayg Vishy wrote up the steps... #link http://etherpad.openstack.org/cinder-worksheet
16:17:35 <renuka> jgriffith: right, but if we need to use keystone for the auth, do we need to do anything special?
16:17:52 <clayg> jgriffith: ok, I'll need to go through that then - thanks
16:18:07 <jgriffith> renuka: You mean in terms of the connection info etc?
16:18:16 <renuka> jgriffith: yes
16:19:02 <jgriffith> renuka: The work sleepsonthefloor is doing now covers it I believe
16:19:10 <renuka> jgriffith: IIRC that was the proposal, to have some auth at that stage, so if anyone were to hack into a compute server, they cannot take over all the storage
16:20:27 <jgriffith> renuka: there's still a risk for attached/connected volumes
16:20:45 <jgriffith> I don't know how we fix that yet
16:21:40 <jgriffith> I don't recall the discussion at the last summit, or the proposal... maybe you could elaborate on it?
16:22:11 <sleepsonthefloor> clayg: http://etherpad.openstack.org/jDMBYz4VKp
16:22:12 <clayg> so it looks like the cinder-worksheet is sorta "pre" sleepsonthefloor's patches (share rpc, db, etc.)
16:22:37 <clayg> sleepsonthefloor: perfect
16:23:01 <jgriffith> clayg: Yeah, that worksheet is kinda "dead"
16:23:16 <jgriffith> clayg: but it has some of the initial thoughts/descriptions that might be useful
16:23:42 <jgriffith> clayg: going forward we wanted to use blueprints/bugs, the worksheet was really just initial stages
16:24:06 <clayg> I think i was kinda up to speed to that point (I may have even seen that before), I'll update sleepsonthefloor's new sheet if I have anything to note.
16:24:19 <jgriffith> clayg: sounds good
16:24:23 <renuka> jgriffith: This was the session: http://essexdesignsummit.sched.org/event/ce1dee155c9b9c62d395d001ff8e0ae4  ... I don't know much of it myself, I just remembered the proposal, so was curious
16:25:39 <jgriffith> renuka: Ok, thanks...  quick glance;
16:26:02 <letterj> Hi.   I would like to ask a couple of questions.
16:26:05 <jgriffith> I think that by having the attach go through the cinder service via token might address this
16:26:17 <clayg> renuka: I was in that meeting - I think the generally folks felt that it would be an improvement, but still had some holes.  I don't think the presenters ever submitted patches.
16:26:27 <jgriffith> renuka: I'll have to read it more closely later and see
16:26:45 <renuka> cool
16:26:53 <jgriffith> letterj: Sure, are they related to current topic?
16:27:14 <jgriffith> letterj: If not, maybe wait til we finish status discussion?
16:27:30 <letterj> ok,
16:28:39 <jgriffith> So we sort of medled status and outstanding items...
16:29:06 <jgriffith> Does anybody have anything to add/ask about current status or shall we got to specific outstanding items?
16:29:15 <jgriffith> s/got/go/
16:29:53 <jgriffith> Alright, let's talk about the things left for F2
16:29:59 <jgriffith> #topic Items for F2
16:30:22 <jgriffith> We already mentioned that patches and pieces that sleepsonthefloor is working on
16:30:43 <jgriffith> The other significant thing is some refactoring in the ec2 side of the house
16:31:23 <jgriffith> I'm working on this and should have it ready end of day today or tomorrow
16:31:45 <jgriffith> Mostly right now it's a matter of making the tests work
16:32:10 <jgriffith> ec2's creat_volume should just call into the cinder api and "work" without too much hassle
16:32:42 <jgriffith> We got most of the direct db calls factored out when we did the uuid migration so that's good
16:32:55 <clayg> jgriffith: I don't really see how eucatools are going to work with ec2 and ebs provided by seperate services - is this a requirement for F2?
16:33:29 <jgriffith> clayg: I wanted to have a functional replacement for nova-volume for f2
16:33:44 <jgriffith> clayg: depending on the interpreter that may or may not include eucatools
16:34:20 <jgriffith> clayg: But I don't see why they wouldn't just point to the cinder api instead of todays volume.api?
16:34:44 <jgriffith> the cinder API is a full abstraction so...
16:35:11 <jgriffith> what am I missing?
16:35:12 <clayg> jgriffith: I guess I don't see why either - I'm not eucatools expert, but I would have assumed if it was working - it was routing all requests (create & attach) through the nova-compute-api
16:35:29 <jgriffith> clayg: ahhh
16:35:51 <jgriffith> clayg: No, it goes to both nova-compute-api and nova-volume-api
16:35:57 <clayg> I don't think the client will be smart enough to make create to cinder then attach to nova, so nova (or whoever is providing ec2 compat) will have to re-route requests to the appropriate service)
16:36:26 <jgriffith> clayg: Yeah, but in most places it's already doing that today
16:36:41 <clayg> jgriffith: ok
16:36:52 <jgriffith> It's seperated fairly nicely between the api's
16:37:21 <jgriffith> clayg: could be that I just haven't stumbled across the section that blows up in my face yet
16:37:31 <jgriffith> we'll see
16:37:59 <jgriffith> if eucatools isn't ready by f2 I think that will be ok, but it's going to take away from what we can do for f3 etc
16:39:24 <jgriffith> Anything else on f2 action items?  Anybody see anything they want to work on?
16:40:17 <jgriffith> #topic outstanding reviews
16:40:22 <jgriffith> #link https://review.openstack.org/#/q/status:open+cinder,n,z
16:41:16 <jgriffith> Not too much here, I think the jekins issues should be fixed so I'll resubmit I5cd73a25
16:41:18 <clayg> sleepsonthefloor: get the big stuff first - https://review.openstack.org/#/c/8073/10/nova/volume/api.py,unified
16:41:23 <clayg> :)
16:41:46 <jgriffith> clayg: :)
16:42:24 <jgriffith> clayg: That and the other "big" one are still drafts
16:43:10 <sleepsonthefloor> clayg: :)
16:43:14 <jgriffith> I need: #link https://review.openstack.org/#/c/8076/
16:43:58 <clayg> jgriffith: that's all just pep8
16:44:00 <clayg> ?
16:44:21 <jgriffith> Yeah, so then when the drafts are ready we'll be set
16:45:10 <jgriffith> The other one that failed jenkins I think I can just go in and +2/a it again and jenkins will try it again
16:45:50 <jgriffith> clayg: I know it seems irrelevant, but there are a few things in draft or in personal branches that will be showing up "soon"
16:46:12 <jgriffith> clayg: Plus, doesn't do us much good if we can pass jenkins pep8 tests
16:46:24 <jgriffith> s/can/can't/
16:46:34 <clayg> no it's fine, I'm checking it out now
16:47:23 <jgriffith> #topic unassigned blueprints
16:47:55 <clayg> sleepsonthefloor: I'm I missing a patch that would show where nova acctually uses volumes/cinder.py?
16:48:27 <sleepsonthefloor> clayg - ah yes, I may have to push a few more devstack changes
16:48:50 <sleepsonthefloor> I'll update the devstack patch in a few mins
16:49:12 <jgriffith> #link https://blueprints.launchpad.net/cinder
16:49:18 <sleepsonthefloor> just need volume_api_class=nova.volume.cinder.API
16:49:47 <jgriffith> Ok, so we've got a number of things here, just wanted to do the weekly check to see if anybody wanted to sign up for any of these?
16:50:15 <jgriffith> Also need to add one for snapshots...
16:50:35 <jgriffith> https://bugs.launchpad.net/nova/+bug/1008866
16:50:37 <uvirtbot> Launchpad bug 1008866 in nova "Creating volume from snapshot on real/production/multicluster installation of OpenStack is broken" [Undecided,New]
16:51:26 <sleepsonthefloor> clayg - updated
16:51:40 <sleepsonthefloor> https://review.openstack.org/#/c/7042/
16:52:03 <jgriffith> It would be great if folks see some things here they might be interested in working on.
16:52:18 <jgriffith> All the pieces should be in place later this week to start hitting these
16:53:29 <jgriffith> Just let me know if somebody wants to grab any of these
16:53:39 <jgriffith> #topic open discussion
16:54:03 <jgriffith> letterj: You had some questions?
16:54:51 <letterj> yes,   I was looking at the api.   Should there be a force-detach
16:55:51 <letterj> also, what is a reserve volume?
16:56:25 <clayg> letterj: reserve volume is mark volume as attaching
16:56:47 <letterj> so it's just a status change
16:56:52 <jgriffith> letterj: detach doesn't need a force, it isn't dependent
16:57:55 <renuka> letterj: reserve volume is required for a race condition that can arise when multiple simultaneous attaches are called for the same volume
16:58:06 <letterj> If a detach gets stuck in an error state how is that handled?
16:58:43 <jgriffith> letterj: The detach actually just modifies the columns in the db
16:58:59 <jgriffith> Are you referring to the compute side possibly?
16:59:35 <renuka> jgriffith: he probably means the operation as a whole, when issued from command line or horizon
17:01:23 <renuka> do we have states in the attach/detach process? i.e. can we say if the attach/detach went through on the compute side? at that point, we can do something on the volumes side of the world
17:02:29 <letterj> I'm asking about what happens when the states get out of sync.    nova state vs cinder state
17:02:56 <letterj> and there is more clean up to do than just updating a db field.
17:03:03 <jgriffith> letterj: ahhh... so for example compute thinks it's attached and volumes/cinder thinks it's detached
17:03:35 <letterj> yes or the other way around
17:03:45 <jgriffith> letterj: right..
17:04:00 <renuka> we shouldn't change the state until we get a detach success from compute
17:04:00 <jgriffith> letterj: I think that definitely needs to be looked at
17:04:14 <renuka> wouldn't that be pretty straight forward?
17:04:36 <jgriffith> renuka: Yes, I believe so.  But as he mentions the other diretion is still possible as well
17:04:51 <jgriffith> By adding the force option we can recover from either situation
17:04:56 <renuka> we don't change the state until we get attach success from compute?
17:06:19 <letterj> I just asking because there is nothing to handle thing stuck in error-detaching or error-attaching situation in nova currnently except manually hacking
17:07:01 <jgriffith> letterj: I think I understand where you're coming from
17:07:03 <renuka> letterj: currently, since we don't use cinder, these states are not very cleanly separated
17:07:09 <jgriffith> similar to the stuck in "creating" problem
17:07:24 <sleepsonthefloor> yeah, was going to say, I think the issues letterj mention may not be cinder-specific
17:07:24 <letterj> yes sir
17:07:50 <jgriffith> letterj: It's an existing problem and yes it needs to be addressed
17:08:09 <jgriffith> letterj: If there's not a bug perhaps you could file one?
17:08:11 <renuka> letterj: also, it may not be as simple as force, for example, what if there was an error on the hypervisor side while detaching... not sure how cinder would be able to "force" it... or are you suggesting that we manipulate our db anyway
17:08:25 <letterj> I filed a bug quite a while ago  https://bugs.launchpad.net/nova/+bug/944383
17:08:28 <uvirtbot> Launchpad bug 944383 in nova "There is no way to recover/cleanup  a volume in an "attaching" state" [Medium,Confirmed]
17:08:37 <jgriffith> letterj: Yeah, sorry just found it
17:09:11 <jgriffith> so another thought is rather than "force" etc maybe some special recovery actions/functions
17:09:20 <clayg> renuka: I do think that cinder should be able to destroy the remote connection and update it's db as "available" even if the consumer (nova) can not respond to the users's reqeust
17:10:23 <jgriffith> clayg: But there's a problem here because we "don't know" what the actual state is
17:10:44 <renuka> clayg: what if detach has actually failed and the volume is still attached to the instance on the compute host?
17:11:54 <jgriffith> clayg: renuka so if we run with a force we catch the exception for the invalid state and just perform the steps anyway maybe
17:11:59 <clayg> renuka: I can cirtainly see that possibility, but it would be nice if you could still request cinder to break down the connection.
17:13:10 <clayg> renuka: I think the more likely case would nova says it's attached (host still has a connection) but the guest no longer sees the volume.
17:13:28 <renuka> hmm ok I agree with "needs some thought" because any inconsistency here could lead to corrupting a users data... e.g. attaching the volume to more than 1 instance where it is possible
17:13:47 <renuka> also billing, because cinder thinks it is detached, while the user still has access to it
17:15:15 <clayg> I would imagine most storage providers are going to bill on volumes that exist and take up space regardless of attached/in-use status
17:15:56 <clayg> I don't think cider should allow a "second attachment" even if nova *does* thinkg it's not in use.
17:16:29 <jgriffith> clayg: I agree on the second attachment, unless it's an attach to the same instance_uuid
17:16:30 <letterj> One other case I can think of is that if the guest goes away for what ever reason you might want to perform a detach locally in cinder without tying it to a nova transaction.
17:17:29 <edmc> Classic "split brain" problem… only solution is a common/shared arbitration mechanism (e.g. a "third vote")
17:17:42 <renuka> ok so we're talking of "cleanup" rather than force detach by the looks of it? maybe we need an admin api to purge any bad connections?
17:18:36 <jgriffith> renuka: That was my thought earlier, rather than force some api call that does sync/cleanup or whatever we decide to call it
17:18:40 <letterj> If things don't work correctly "cleanup" will usually be required
17:18:55 <clayg> renuka: I think the problem exists outside of cleanup, and has value to end-users - particularlly when the client is something besides nova.  In the case where cinder recieves a "foricibly terminate any remote connections for the volume" message - if nova supports it - we could warn them we're doing it.
17:18:56 <jgriffith> but takes the point from edmc and does a compare of sorts
17:19:24 <jgriffith> clayg: That could be "part" of the cleanup no?
17:19:59 <jgriffith> clayg: ie they run a terminate and it fails, now they need to run cleanup
17:20:40 <jgriffith> my point here is if you want to do it correctly/safely the implementation looks the same regardless of what you call it
17:20:43 <clayg> jgriffith: yes, if they can.  If not - we can still expose a force_detach to make the volume available - _regardless_
17:21:03 <jgriffith> if it's a force-detach or a cleanup
17:21:33 <letterj> But I also think nova is gong to have to have something like this as well as cinder
17:21:44 <renuka> clayg: why does this need to be user facing?
17:21:58 <jgriffith> Ok, so we're all in agreement that there's an issue here that needs to be addressed
17:22:41 <letterj> Cool.  Thanks for taking my question.
17:23:11 <clayg> renuka: Who else but the user can say if the volume should or should not be attached to an instance?
17:23:26 <renuka> I have a quick question... you can get back to me with the answer
17:24:23 <renuka> clayg: the user can say detach of course... in case of an error, we either invoke cleanup automatically or as some kind of purge daemon process... why should the user be involved with the force?
17:25:23 <renuka> So as for my question, next thursday, at the Openstack meetup, we were trying to get the attendees to do an end-to-end feature, something that is super simple, but touches most of the stack
17:25:49 <renuka> if anyone has any low-hanging-fruit kind of feature suggestions, please send me an email
17:25:52 <clayg> renuka: if the client (nova) never sent cinder the detach command - we don't know what the user wants.  So the cleanup all falls to nova.  Which I think is less than ideal.
17:26:14 <DuncanT> Wow, massive meeting! Sorry I'm (ridiculously) late
17:26:32 <sleepsonthefloor> renuka: see https://github.com/cloudbuilders/simple_horizon_plugin and https://github.com/cloudbuilders/simple_nova_extension
17:27:15 <renuka> sleepsonthefloor: thanks
17:27:36 <jgriffith> Ok, one last question to throw out
17:28:00 <jgriffith> clayg asked about why attach/detach etc is an extension and not core api
17:28:35 <clayg> apparently all I do is sit around and pontificate the nature of attach and detach all day
17:28:47 <jgriffith> My thought was extension because we may use cinder for "other things" and these could look differently depending
17:28:50 <jgriffith> clayg: :)
17:28:52 <renuka> hahaha
17:29:26 <DuncanT> attach/detach are the only parts (so far) that have to get their hands dirty dealing with nova
17:29:34 <jgriffith> does anybody have any strong opinions/thoughts around this? besides clayg :)
17:29:47 <jgriffith> DuncanT: connection info as well
17:30:43 <DuncanT> But connection info is an opaque action, still completely within cinder, yes? Attach is the first time the hypervisor in involved?
17:31:28 <renuka> i think logically, connection info is "more core" than attach, by DuncanT's reasoning
17:31:37 <letterj> I agree with clay on the attach/detach issue
17:31:39 <clayg> to clarify (hopefully), I had thought that almost any backend for cinder would want a notification on attach and detach (if for nothing else but book keeping in cinder) - so why not make it core
17:32:17 <clayg> intialize_connection/terminate_connection would work find in this context - whatever the core api wants to call them
17:33:59 <jgriffith> just FYI I currently have no real preference on this whatsoever, just wanted feedback from everybody else
17:34:46 <renuka> It would be nice to have a good reason why they should not be core, if we decide to make them extensions
17:35:26 <jgriffith> renuka: so they're already implemented as extensions thanks to sleepsonthefloor
17:35:39 <clayg> jgriffith: I'm ok with sleepsonthefloor's current patch and appreciate all of his hard work of course, but at somepoint it'd be nice to have a clean core api that make sense in multiple context.
17:36:10 <jgriffith> So part of this also just stems from how vish laid it out in the worksheet
17:36:33 <renuka> perhaps we need vish's opinion then
17:36:43 <vishy> hello?
17:36:50 <clayg> whoa
17:37:04 <renuka> oh cool.. i was going to suggest question for mailing list :)
17:37:47 <jgriffith> vishy: So the question is attach/detach and connection info being core api versus extension
17:38:28 <vishy> jgriffith: so initialize_connection / terminate_connection should be core imo
17:38:53 <vishy> attach/detach reserve/unreserve i'm not so sure about it.
17:39:14 <vishy> attach/detach should probably be generalized into just metadata
17:39:30 <clayg> boom
17:39:32 <vishy> reserve/unreserve possibly could be to if we can force atomic metadata updates somehow
17:41:56 <jgriffith> clayg: does this work for you?
17:42:34 <clayg> jgriffith: yeah, sounds like it's a longer term plan, but maybe before f-final we could have init/term in CORE?
17:43:33 <clayg> speaking of f-final - when does nova-compute-volume-extensions get removed?  when does nova-volume-api get deprecated?
17:44:58 <vishy> clayg: if we have feature parity by f-2 we replace it
17:45:03 <vishy> during f-3
17:45:49 <clayg> so there's no deprecation period?  upgrading from essex to folsom is migrating from nova-volumes to cinder?
17:48:25 <jgriffith> clayg: That's what I'm hoping for
17:48:55 <jgriffith> We're way over today
17:49:30 <jgriffith> everybody good for now, or should we hash some more of this out?
17:50:19 <jgriffith> Ok, thanks everyone!
17:50:22 <jgriffith> #endmeeting