16:00:01 <thingee> #startmeeting cinder
16:00:02 <openstack> Meeting started Wed Apr 15 16:00:01 2015 UTC and is due to finish in 60 minutes.  The chair is thingee. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:04 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:06 <openstack> The meeting name has been set to 'cinder'
16:00:07 <thingee> hi everyone!
16:00:08 <deepakcs> o/
16:00:09 <eharney> hi
16:00:10 <winston-d_> o/
16:00:10 <smcginnis> o/
16:00:10 <jungleboyj> o/
16:00:14 <rhe00> hi
16:00:16 <cebruns> Hi
16:00:16 <Swanson> hi
16:00:17 <scottda> hi
16:00:17 <geguileo> hi
16:00:18 <DuncanT> hi
16:00:22 <jgravel> hello
16:00:23 * bswartz thinks that thingee tries to time to meeting to start at EXACTLY 1600
16:00:41 <thingee> bswartz: I'm a little ocd yes
16:00:44 <rushiagr> o/
16:00:44 <thingee> :)
16:00:45 <jungleboyj> bswartz: Sounds like something thingee would do.
16:00:47 <jungleboyj> :-)
16:00:53 <thingee> annoucements!
16:01:00 <thingee> liberty summit proposals!
16:01:02 <thingee> http://lists.openstack.org/pipermail/openstack-dev/2015-February/057383.html
16:01:03 <xyang2> hi
16:01:17 <asselin> o/
16:01:18 <ameade> o/
16:01:20 <akerr> o/
16:01:26 <thingee> please submit them to the etherpad listed in the mailing list post.
16:01:27 <patrickeast> o/
16:01:34 <li_zhang> hi
16:01:47 <rushil> hi
16:01:54 <thingee> also we have a bit less that require fish bowl.
16:01:56 <e0ne> #link https://etherpad.openstack.org/p/cinder-liberty-proposed-sessions
16:02:16 <thingee> I will be reviewing them to make sure this is correct, but if you need user input, please consider using the fish bowl session type
16:02:25 <bswartz> thingee: that mail gives a date of April 27 which is a Monday
16:02:32 <thingee> otherwise I will be releasing these back in the summit pool for other projects to use.
16:02:48 <thingee> bswartz: yeah we'll do the follow wednesday doh!
16:02:51 <bswartz> is the real deadline april 29?
16:02:58 <thingee> sure
16:03:03 <thingee> I'll send a correcting email
16:03:03 <bswartz> k
16:03:19 <thingee> also potential rc bugs
16:03:21 <thingee> https://bugs.launchpad.net/cinder/+bugs?field.tag=kilo-rc-potential
16:03:21 <vilobhmm11> thingee : sorry just joined; we are talking about dealine for liberty specs you mean
16:03:42 <thingee> if you're looking for something to review, please look there. Also if you have something pressing, please tag it with 'kilo-rc-potential'
16:03:45 <bswartz> vilobhmm11: deadline for design summit proposals
16:03:53 <thingee> I review these so I'll remove something if it's not really blocking.
16:04:00 <thingee> but I will talk to you first
16:04:07 <thingee> vilobhmm11: yes
16:04:07 <vilobhmm11> bswartz : sure
16:04:21 <thingee> err yeah liberty summit talks
16:04:43 <thingee> I will be opening liberty specs https://review.openstack.org/#/c/173675/
16:04:59 <thingee> already audited and moved old specs around
16:05:23 <avishay> hi
16:05:34 <thingee> ok any other announcements before we begin?
16:05:37 <e0ne> thingee: may be merge https://review.openstack.org/#/c/149449/ first is a good idea
16:05:43 <vilobhmm11> thats cool
16:05:52 <thingee> e0ne: sure
16:06:03 <vilobhmm11> eone : +1
16:06:13 <thingee> I'll start pushing stuff later today
16:06:16 <jgriffith> sigh.... more paper work
16:06:22 <thingee> thanks everyone!
16:06:25 <vilobhmm11> nova already has that
16:06:44 <thingee> jgriffith: I've spoke to you directly about the use case section and you said it was a good idea.
16:06:51 <jgriffith> thingee: it is
16:07:03 <jgriffith> thingee: but the whole spec thing in general is out of control
16:07:11 <jgriffith> thingee: but no need to derail meeting on this
16:07:41 <jungleboyj> :-)
16:07:42 <thingee> and technically it's not *more*, it's break the problem description up
16:07:52 <thingee> breaking*
16:07:52 <jgriffith> thingee: ok boss
16:08:26 <thingee> the problem description mentions use cases, but people seem to miss it and then we end up giving a -1 anyways at the beginning because people don't list. I figured this would help avoid that.
16:08:27 <bswartz> I dislike specs repos, but if you're going to have and use one, at least do it right
16:09:10 <thingee> #topic active-active-[active...] cinder volume services
16:09:11 <thingee> DuncanT: hi
16:09:42 <DuncanT> Hi
16:10:07 <DuncanT> So I've been told my highest current priority is making active/active cinder-volume service work, bug free
16:10:09 <Arkady_Kanevsky> hello
16:10:28 <winston-d_> DuncanT: by mgmt?
16:10:38 <avishay> DuncanT: cinder-api too?
16:10:38 <DuncanT> I know of a bunch of issues with doing this, but I'm sure people know more, and I'd like a complete list of known or suspected issues
16:10:44 <DuncanT> winston-d_: Yeah
16:10:45 <bswartz> DuncanT: have you talked with vishy on that topic?
16:10:46 <thingee> I would say active/active and the rpc compat work for rolling upgrades in liberty.
16:10:52 <bswartz> I think he has experience making it work
16:10:54 <DuncanT> avishay: API already works?
16:11:07 <jbernard> i dont think so
16:11:10 <hemna> The API from nova -> cinder has to change to make active active work correctly
16:11:19 <hemna> we've already seen that with just trying to remove the volume manager locks
16:11:23 <hemna> nova craps out
16:11:23 <avishay> DuncanT: need atomic volume state updates (currently it gets the volume from DB, checks things, then updates - not atomic)
16:11:28 <e0ne> DuncanT: it dosn't work in Juno. i've got several raises few months ago
16:11:30 <winston-d_> FWIW, we have been using active-active-active since day 1
16:11:32 <DuncanT> hemna: That's on the list
16:11:37 <avishay> hemna: +1
16:11:49 <winston-d_> no major issue related to that
16:11:54 <DuncanT> hemna: If you have any specifics, please pass them on
16:12:11 <avishay> winston-d_: with all backends?
16:12:21 <hemna> DuncanT, sure, we can take it off line in cinder channel if you like.
16:12:28 <DuncanT> winston-d_: Kick off a tight loop of random creates, deletes, snaps, creates from snaps, it ends up in a mess
16:12:38 <geguileo> DuncanT: I've looked into the atomic state changes and it's not ready
16:12:53 <hemna> winston-d_, it will eventually fail if you don't remove the volume manager locks
16:12:57 <DuncanT> hemna: Sure, no point taking up meeting time, just trying to get anybody who's found issues to contact me
16:13:06 <vilobhmm11> geguileo : can you point us to those changes ?
16:13:17 <jbernard> DuncanT: email? or etherpad? what's preferred?
16:13:27 <winston-d_> DuncanT: in our use cases, we don't see much of issue with active-active
16:13:30 <geguileo> vilobhmm11: No, no changes yet, just had a look into ways to solve the issue
16:13:34 <DuncanT> geguileo: I might have  a crude first pass at that that might help, need to talk to an oslo.db person to check if it is feasible
16:13:36 <e0ne> DuncanT: i try to reproduce my past issues and will contact you
16:13:51 <smcginnis> Etherpad might be good so those interested can at least see what's going on.
16:13:52 <DuncanT> e0ne: Thanks, that's exactly what I was hoping people would say :-)
16:13:54 <winston-d_> avishay: most of deployment has only limited backend(s)
16:14:01 <jgriffith> winston-d_: you're not alone
16:14:03 <vilobhmm11> DuncanT : I would also send you details of issues i have been seeing
16:14:08 <geguileo> DuncanT: I have a working test code (out of Cinder) with multiple workers accessing same row in a Galera cluster
16:14:17 <DuncanT> vilobhmm11: Thanks
16:14:22 <jgriffith> winston-d_: I know of at least one other customer doing A/A since H
16:14:46 <DuncanT> We have A/A test machines, if you're gentle (i.e. normal workloads) it is fine
16:14:48 <e0ne> jgriffith: haproxy?
16:14:55 <DuncanT> If you push it hard, it breaks
16:15:07 <jgriffith> DuncanT: name something in OpenSTack that doesn't :)
16:15:14 <smcginnis> Hah
16:15:16 <jungleboyj> jgriffith: :-)
16:15:18 <e0ne> DuncanT: any thoughts about test automation for it?
16:15:19 <winston-d_> e0ne: relies on rabbit actually
16:15:38 <winston-d_> e0ne: no LB is required
16:15:48 <DuncanT> e0ne: Rally scenarios initially, can do something smarter once they pass
16:16:05 <e0ne> we still allows cinder-volume to change DB:(
16:16:21 <winston-d_> e0ne: why not?
16:16:23 <bswartz> winston-d_: haproxy is necessary if you want to actually use more than one cinder-api instance
16:16:52 <winston-d_> bswartz: ah, for cinder-api, we have LB. but c-vol, no
16:16:55 <e0ne> winston-d_: w/o distributed locks it will generate raise conditions
16:17:22 <bswartz> yes
16:17:29 <avishay> e0ne: as long as the changes are atomic (i.e., check and set), it should work, no?
16:17:31 <hemna> e0ne, that's what I was saying earlier about the local locks in the volume manager.  it will eventually break.
16:17:32 <e0ne> winston-d_: i mean if we'll deploy more than one cinder-volume
16:17:47 <e0ne> hemna: +1
16:17:53 <jbernard> avishay: yes, but that isn't what we have at the moment
16:17:53 <vilobhmm11> avishay: agree …need atomic transaction….DuncanT : may be a good idea to drop email to openstack-dev and get some more corner cases other people might have seen and also some feedback from oslo people…
16:17:55 <winston-d_> e0ne: i know, we have 3, in every region
16:17:58 <bswartz> if anyone has looked as the oslo concurrency code, it has support for distributed locking
16:18:12 <hemna> you can run 2 cinder volume services on 2 hosts as long as they aren't managing the same volumes.
16:18:13 <thingee> ok DuncanT, so are you going to be working with folks like e0ne harlowja_away and dulek ? ... or
16:18:15 <bswartz> not sure I'd trust it but it's there
16:18:28 <geguileo> bswartz: Do you mean the external synchronized option?
16:18:36 <bswartz> geguileo: I think that's the one
16:18:40 <e0ne> hemna: absolutely
16:18:50 <geguileo> bswartz: That uses a file for the lock
16:18:56 <DuncanT> thingee: Once I've got a list of all the problems, I'll be working with anybody and everybody I think can help fix them
16:19:01 <geguileo> bswartz: So you need to have the file shared among all instances
16:19:11 <DuncanT> thingee: We need a parallel attach on this to make any progress
16:19:17 <thingee> DuncanT: there are some patches already up that begin to work on this problem.
16:19:25 <vilobhmm11> thingee : the patch i propose for micro_states handles these atomic transations that avishay mentioned, jfyi…
16:19:27 <hemna> so if that's what you mean by active-active, then yah that will probably work.   but to me active-active means 2 different hosts running the same driver talking to the same backend, managing the same volumes.
16:19:31 <jgriffith> DuncanT: I'd love to help eliminate the stupid frikin locks everywhere :)
16:19:32 <geguileo> There are at least other 2 solutions I have tried that work without the need of using those locks
16:19:33 <thingee> DuncanT: ^
16:19:37 <DuncanT> thingee: Parts of this problem, yes
16:19:45 <hemna> jgriffith, +1
16:19:57 <hemna> jgriffith, this all feathers into the nova -> cinder api changes needed.
16:19:59 <DuncanT> thingee: One thing we don't have is a list of all of the known parts of the problem, I want to start there
16:20:05 <thingee> DuncanT: it would be good for you to test those out and sign off. maybe help those that are doing summit session coming up
16:20:11 <jgriffith> since nobody is going to work on fixing the problem of us NEVER cleaning them up maybe we can just quit using them
16:20:30 <DuncanT> thingee: Most of them don't detail what problem they are fixing
16:20:36 <hemna> jgriffith, well, we tried to do it in kilo.   but failed because of breaking nova in the process.
16:21:05 <thingee> DuncanT: so help those to detail what problems they fix. Since you can verify there is a problem, you can pull those patches in and try them out
16:21:09 <geguileo> DuncanT: How do you suggest we go about creating the list of all parts of the problem?
16:21:33 <thingee> DuncanT: I just want to make sure that we don't duplicate effort and use what people have developed last release if we agree with the approach
16:21:34 <jgriffith> hemna: I think we're talking about two different things :)
16:21:39 <DuncanT> geguileo: Initially, throw them all at me and I'll try to collate them and raise bugs
16:21:44 <e0ne> geguileo, DuncanT: omo, ehterpad will be good for it
16:21:53 <hemna> fixing the locks ?
16:21:54 <DuncanT> thingee: Absolutely. I just want to start with the list
16:21:54 <hemna> heh
16:21:56 <e0ne> s/omo/imo
16:22:20 <geguileo> e0ne: I agree, an Etherpad would be good, with brief explanation of each issue so we can see the whole picture
16:22:24 <DuncanT> etherpad is good, irc is good, email is good, I'll certainly be working via an etherpad
16:22:31 <e0ne> :)
16:22:41 <thingee> dulek, e0ne, vilobhmm11  any of the efforts you're doing, can you please work with DuncanT on which things on his compiled list are handled by which patch(es)?
16:22:48 <winston-d_> +1 for etherpad
16:22:57 <geguileo> +1 for etherpad
16:23:01 <vilobhmm11> thingee : sure will do...
16:23:02 <e0ne> thingee: sure, i'll do
16:23:03 <deepakcs> +1 for etherpad
16:23:30 <thingee> DuncanT: can you make a etherpad now so I can link it in the meeting notes?
16:23:49 <thingee> #action DuncanT will be compiling a list of know active/active issues
16:23:53 <vilobhmm11> DuncanT : please include me in any conversation regarding this problem; would be happy to help
16:24:14 <DuncanT> vilobhmm11: Will do
16:24:17 <thingee> #action dulek vilobhmm11 e0ne will match which patch(es) already posted will fix known issues from DuncanT's list
16:24:19 <DuncanT> thingee: Doing it now
16:24:29 <e0ne> does sheduler work in A-A?
16:24:33 <smcginnis> If/when we have something working, it would be good to also get some documentation together to make sure new code isn't introduced that breaks HA.
16:24:43 <winston-d_> e0ne: yes, in our case
16:24:49 <avishay> e0ne: i believe so
16:24:50 <hemna> smcginnis, +1, that's good for reviewers as well
16:24:52 <DuncanT> https://etherpad.openstack.org/p/cinder-active-active-vol-service-issues
16:24:57 <e0ne> how does it receive capabilities reports?
16:25:08 <DuncanT> e0ne: Yes, everything works except c-vol
16:25:16 <thingee> #link https://etherpad.openstack.org/p/cinder-active-active-vol-service-issues
16:25:20 <winston-d_> e0ne: vol-stats update is a fanout
16:25:20 <DuncanT> e0ne: the broadcast for that just works
16:25:32 <e0ne> DuncanT: glad to know it:)
16:25:50 <DuncanT> e0ne: The scheduling becomes slightly none-deterministic, but not in a bad way
16:26:18 <e0ne> winston-d_: it works in our case too, but i was asking about out-of-the-box
16:26:24 <thingee> DuncanT: anything else?
16:26:35 <DuncanT> That's it for now, thanks
16:26:38 <vilobhmm11> it would be nice to keep state management away from schedule
16:26:42 <thingee> #topic open discussion
16:26:42 <winston-d_> e0ne: yes, we use it out-of-the-box
16:26:45 <vilobhmm11> scheduler*
16:26:53 <e0ne> winston-d_: great!
16:27:01 <e0ne> i'll check how it works in our case
16:27:21 <e0ne> what do you think about https://github.com/openstack/openstack-specs/blob/master/specs/no-downward-sql-migration.rst?
16:27:21 <DuncanT> vilobhmm11: I don't think that is possible
16:27:35 <e0ne> downgrades is painful at least for sqlite
16:27:43 <vilobhmm11> DuncanT : agree
16:27:49 <hemna> winston-d_, but do you use c-vol services on different hosts to manage the same volumes/arrays?
16:28:01 <DuncanT> e0ne: Testing object code is hard without downgrades
16:28:14 <e0ne> DuncanT: good point
16:28:26 <rushiagr> is c-vol not A-A-able coz it maintains some state?
16:28:28 <winston-d_> hemna: yes, of course
16:28:40 <hemna> winston-d_, ok you are playing with fire then :)  It will fail.
16:28:46 <e0ne> :)
16:28:48 <DuncanT> rushiagr: Correct
16:29:00 <winston-d_> hemna: almost 2 years, things are fine
16:29:05 <vilobhmm11> rushiagr : right
16:29:29 <winston-d_> hemna: maybe it is just our users behave
16:29:30 <smcginnis> Just so it is also captured here - tentative schedule for Liberty release is:
16:29:33 <smcginnis> L1: June 25, L2: July 30, L3: Spet 3, Release Oct 15
16:29:39 <thingee> ok seems like we have no topics for open discussion. we can continue this chat in #openstack-cinder
16:29:50 <vilobhmm11> if we are done with no-sql downgrade discussion
16:29:53 <vilobhmm11> i have one
16:30:03 <vilobhmm11> what is our plan for heirarchical projects
16:30:06 <vilobhmm11> https://blueprints.launchpad.net/cinder/+spec/cinder-nested-quota-driver
16:30:24 <vilobhmm11> thingee, DuncanT, other cores : ^^
16:30:42 <rushiagr> vilobhmm11: is hierarchical multitenancy part supported by keystone already? I thought it's still some way to complete..
16:30:44 <li_zhang> hemna: we did also, 3 volume hosts each has 8 volume services managing one array
16:30:53 <vilobhmm11> yes its part of keystone in kilo
16:31:15 <vilobhmm11> as every other project is moving toward heriarchical support i guess we should also think in that direction
16:31:36 <thingee> vilobhmm11: well for one thing, we need to remove the deprecated code in DBQuoteDriver
16:31:40 <thingee> :)
16:31:51 <vilobhmm11> sure…
16:31:54 <DuncanT> thingee ++
16:31:57 <hemna> winston-d_, it's just a matter of time, or timing, literally.
16:32:03 <e0ne> thingee: +1
16:32:21 <vilobhmm11> do you think this a s apriority for liberty ?
16:32:31 <winston-d_> thingee: anyone working on that that you are aware of?
16:32:55 <thingee> winston-d_: nope
16:33:45 <vilobhmm11> winston-d_ : i have a spec out there for review  https://review.openstack.org/#/c/173141/ if anyone wants to have a look in there free time
16:33:51 <thingee> winston-d_: https://bugs.launchpad.net/cinder/+bug/1435807
16:33:52 <openstack> Launchpad bug 1435807 in Cinder "Volume type quotas trigger deprecation notification" [High,Triaged] - Assigned to Anton Arefiev (aarefiev)
16:34:12 <thingee> winston-d_: I take that back
16:34:33 <winston-d_> vilobhmm11: thx
16:35:01 <thingee> vilobhmm11: ok seems like you're already on top of it :)
16:35:13 <thingee> we will review after L specs opens
16:35:19 <DuncanT> vilobhmm11: For a first pass, I suggest skipping the updating of quotas by anybody except admin, just do nested quota support
16:35:27 <vilobhmm11> thingee : :) sure…please do
16:35:36 <thingee> anything else?
16:35:36 <DuncanT> vilobhmm11: Then add the editing by none-admin later
16:35:38 <thingee> anybody?
16:35:51 <vilobhmm11> DuncanT : sure…appreciate your feedback
16:36:33 <thingee> ok thanks everyone!
16:36:35 <thingee> #endmeeting