16:00:01 #startmeeting cinder 16:00:02 Meeting started Wed Apr 15 16:00:01 2015 UTC and is due to finish in 60 minutes. The chair is thingee. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:00:04 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:00:06 The meeting name has been set to 'cinder' 16:00:07 hi everyone! 16:00:08 o/ 16:00:09 hi 16:00:10 o/ 16:00:10 o/ 16:00:10 o/ 16:00:14 hi 16:00:16 Hi 16:00:16 hi 16:00:17 hi 16:00:17 hi 16:00:18 hi 16:00:22 hello 16:00:23 * bswartz thinks that thingee tries to time to meeting to start at EXACTLY 1600 16:00:41 bswartz: I'm a little ocd yes 16:00:44 o/ 16:00:44 :) 16:00:45 bswartz: Sounds like something thingee would do. 16:00:47 :-) 16:00:53 annoucements! 16:01:00 liberty summit proposals! 16:01:02 http://lists.openstack.org/pipermail/openstack-dev/2015-February/057383.html 16:01:03 hi 16:01:17 o/ 16:01:18 o/ 16:01:20 o/ 16:01:26 please submit them to the etherpad listed in the mailing list post. 16:01:27 o/ 16:01:34 hi 16:01:47 hi 16:01:54 also we have a bit less that require fish bowl. 16:01:56 #link https://etherpad.openstack.org/p/cinder-liberty-proposed-sessions 16:02:16 I will be reviewing them to make sure this is correct, but if you need user input, please consider using the fish bowl session type 16:02:25 thingee: that mail gives a date of April 27 which is a Monday 16:02:32 otherwise I will be releasing these back in the summit pool for other projects to use. 16:02:48 bswartz: yeah we'll do the follow wednesday doh! 16:02:51 is the real deadline april 29? 16:02:58 sure 16:03:03 I'll send a correcting email 16:03:03 k 16:03:19 also potential rc bugs 16:03:21 https://bugs.launchpad.net/cinder/+bugs?field.tag=kilo-rc-potential 16:03:21 thingee : sorry just joined; we are talking about dealine for liberty specs you mean 16:03:42 if you're looking for something to review, please look there. Also if you have something pressing, please tag it with 'kilo-rc-potential' 16:03:45 vilobhmm11: deadline for design summit proposals 16:03:53 I review these so I'll remove something if it's not really blocking. 16:04:00 but I will talk to you first 16:04:07 vilobhmm11: yes 16:04:07 bswartz : sure 16:04:21 err yeah liberty summit talks 16:04:43 I will be opening liberty specs https://review.openstack.org/#/c/173675/ 16:04:59 already audited and moved old specs around 16:05:23 hi 16:05:34 ok any other announcements before we begin? 16:05:37 thingee: may be merge https://review.openstack.org/#/c/149449/ first is a good idea 16:05:43 thats cool 16:05:52 e0ne: sure 16:06:03 eone : +1 16:06:13 I'll start pushing stuff later today 16:06:16 sigh.... more paper work 16:06:22 thanks everyone! 16:06:25 nova already has that 16:06:44 jgriffith: I've spoke to you directly about the use case section and you said it was a good idea. 16:06:51 thingee: it is 16:07:03 thingee: but the whole spec thing in general is out of control 16:07:11 thingee: but no need to derail meeting on this 16:07:41 :-) 16:07:42 and technically it's not *more*, it's break the problem description up 16:07:52 breaking* 16:07:52 thingee: ok boss 16:08:26 the problem description mentions use cases, but people seem to miss it and then we end up giving a -1 anyways at the beginning because people don't list. I figured this would help avoid that. 16:08:27 I dislike specs repos, but if you're going to have and use one, at least do it right 16:09:10 #topic active-active-[active...] cinder volume services 16:09:11 DuncanT: hi 16:09:42 Hi 16:10:07 So I've been told my highest current priority is making active/active cinder-volume service work, bug free 16:10:09 hello 16:10:28 DuncanT: by mgmt? 16:10:38 DuncanT: cinder-api too? 16:10:38 I know of a bunch of issues with doing this, but I'm sure people know more, and I'd like a complete list of known or suspected issues 16:10:44 winston-d_: Yeah 16:10:45 DuncanT: have you talked with vishy on that topic? 16:10:46 I would say active/active and the rpc compat work for rolling upgrades in liberty. 16:10:52 I think he has experience making it work 16:10:54 avishay: API already works? 16:11:07 i dont think so 16:11:10 The API from nova -> cinder has to change to make active active work correctly 16:11:19 we've already seen that with just trying to remove the volume manager locks 16:11:23 nova craps out 16:11:23 DuncanT: need atomic volume state updates (currently it gets the volume from DB, checks things, then updates - not atomic) 16:11:28 DuncanT: it dosn't work in Juno. i've got several raises few months ago 16:11:30 FWIW, we have been using active-active-active since day 1 16:11:32 hemna: That's on the list 16:11:37 hemna: +1 16:11:49 no major issue related to that 16:11:54 hemna: If you have any specifics, please pass them on 16:12:11 winston-d_: with all backends? 16:12:21 DuncanT, sure, we can take it off line in cinder channel if you like. 16:12:28 winston-d_: Kick off a tight loop of random creates, deletes, snaps, creates from snaps, it ends up in a mess 16:12:38 DuncanT: I've looked into the atomic state changes and it's not ready 16:12:53 winston-d_, it will eventually fail if you don't remove the volume manager locks 16:12:57 hemna: Sure, no point taking up meeting time, just trying to get anybody who's found issues to contact me 16:13:06 geguileo : can you point us to those changes ? 16:13:17 DuncanT: email? or etherpad? what's preferred? 16:13:27 DuncanT: in our use cases, we don't see much of issue with active-active 16:13:30 vilobhmm11: No, no changes yet, just had a look into ways to solve the issue 16:13:34 geguileo: I might have a crude first pass at that that might help, need to talk to an oslo.db person to check if it is feasible 16:13:36 DuncanT: i try to reproduce my past issues and will contact you 16:13:51 Etherpad might be good so those interested can at least see what's going on. 16:13:52 e0ne: Thanks, that's exactly what I was hoping people would say :-) 16:13:54 avishay: most of deployment has only limited backend(s) 16:14:01 winston-d_: you're not alone 16:14:03 DuncanT : I would also send you details of issues i have been seeing 16:14:08 DuncanT: I have a working test code (out of Cinder) with multiple workers accessing same row in a Galera cluster 16:14:17 vilobhmm11: Thanks 16:14:22 winston-d_: I know of at least one other customer doing A/A since H 16:14:46 We have A/A test machines, if you're gentle (i.e. normal workloads) it is fine 16:14:48 jgriffith: haproxy? 16:14:55 If you push it hard, it breaks 16:15:07 DuncanT: name something in OpenSTack that doesn't :) 16:15:14 Hah 16:15:16 jgriffith: :-) 16:15:18 DuncanT: any thoughts about test automation for it? 16:15:19 e0ne: relies on rabbit actually 16:15:38 e0ne: no LB is required 16:15:48 e0ne: Rally scenarios initially, can do something smarter once they pass 16:16:05 we still allows cinder-volume to change DB:( 16:16:21 e0ne: why not? 16:16:23 winston-d_: haproxy is necessary if you want to actually use more than one cinder-api instance 16:16:52 bswartz: ah, for cinder-api, we have LB. but c-vol, no 16:16:55 winston-d_: w/o distributed locks it will generate raise conditions 16:17:22 yes 16:17:29 e0ne: as long as the changes are atomic (i.e., check and set), it should work, no? 16:17:31 e0ne, that's what I was saying earlier about the local locks in the volume manager. it will eventually break. 16:17:32 winston-d_: i mean if we'll deploy more than one cinder-volume 16:17:47 hemna: +1 16:17:53 avishay: yes, but that isn't what we have at the moment 16:17:53 avishay: agree …need atomic transaction….DuncanT : may be a good idea to drop email to openstack-dev and get some more corner cases other people might have seen and also some feedback from oslo people… 16:17:55 e0ne: i know, we have 3, in every region 16:17:58 if anyone has looked as the oslo concurrency code, it has support for distributed locking 16:18:12 you can run 2 cinder volume services on 2 hosts as long as they aren't managing the same volumes. 16:18:13 ok DuncanT, so are you going to be working with folks like e0ne harlowja_away and dulek ? ... or 16:18:15 not sure I'd trust it but it's there 16:18:28 bswartz: Do you mean the external synchronized option? 16:18:36 geguileo: I think that's the one 16:18:40 hemna: absolutely 16:18:50 bswartz: That uses a file for the lock 16:18:56 thingee: Once I've got a list of all the problems, I'll be working with anybody and everybody I think can help fix them 16:19:01 bswartz: So you need to have the file shared among all instances 16:19:11 thingee: We need a parallel attach on this to make any progress 16:19:17 DuncanT: there are some patches already up that begin to work on this problem. 16:19:25 thingee : the patch i propose for micro_states handles these atomic transations that avishay mentioned, jfyi… 16:19:27 so if that's what you mean by active-active, then yah that will probably work. but to me active-active means 2 different hosts running the same driver talking to the same backend, managing the same volumes. 16:19:31 DuncanT: I'd love to help eliminate the stupid frikin locks everywhere :) 16:19:32 There are at least other 2 solutions I have tried that work without the need of using those locks 16:19:33 DuncanT: ^ 16:19:37 thingee: Parts of this problem, yes 16:19:45 jgriffith, +1 16:19:57 jgriffith, this all feathers into the nova -> cinder api changes needed. 16:19:59 thingee: One thing we don't have is a list of all of the known parts of the problem, I want to start there 16:20:05 DuncanT: it would be good for you to test those out and sign off. maybe help those that are doing summit session coming up 16:20:11 since nobody is going to work on fixing the problem of us NEVER cleaning them up maybe we can just quit using them 16:20:30 thingee: Most of them don't detail what problem they are fixing 16:20:36 jgriffith, well, we tried to do it in kilo. but failed because of breaking nova in the process. 16:21:05 DuncanT: so help those to detail what problems they fix. Since you can verify there is a problem, you can pull those patches in and try them out 16:21:09 DuncanT: How do you suggest we go about creating the list of all parts of the problem? 16:21:33 DuncanT: I just want to make sure that we don't duplicate effort and use what people have developed last release if we agree with the approach 16:21:34 hemna: I think we're talking about two different things :) 16:21:39 geguileo: Initially, throw them all at me and I'll try to collate them and raise bugs 16:21:44 geguileo, DuncanT: omo, ehterpad will be good for it 16:21:53 fixing the locks ? 16:21:54 thingee: Absolutely. I just want to start with the list 16:21:54 heh 16:21:56 s/omo/imo 16:22:20 e0ne: I agree, an Etherpad would be good, with brief explanation of each issue so we can see the whole picture 16:22:24 etherpad is good, irc is good, email is good, I'll certainly be working via an etherpad 16:22:31 :) 16:22:41 dulek, e0ne, vilobhmm11 any of the efforts you're doing, can you please work with DuncanT on which things on his compiled list are handled by which patch(es)? 16:22:48 +1 for etherpad 16:22:57 +1 for etherpad 16:23:01 thingee : sure will do... 16:23:02 thingee: sure, i'll do 16:23:03 +1 for etherpad 16:23:30 DuncanT: can you make a etherpad now so I can link it in the meeting notes? 16:23:49 #action DuncanT will be compiling a list of know active/active issues 16:23:53 DuncanT : please include me in any conversation regarding this problem; would be happy to help 16:24:14 vilobhmm11: Will do 16:24:17 #action dulek vilobhmm11 e0ne will match which patch(es) already posted will fix known issues from DuncanT's list 16:24:19 thingee: Doing it now 16:24:29 does sheduler work in A-A? 16:24:33 If/when we have something working, it would be good to also get some documentation together to make sure new code isn't introduced that breaks HA. 16:24:43 e0ne: yes, in our case 16:24:49 e0ne: i believe so 16:24:50 smcginnis, +1, that's good for reviewers as well 16:24:52 https://etherpad.openstack.org/p/cinder-active-active-vol-service-issues 16:24:57 how does it receive capabilities reports? 16:25:08 e0ne: Yes, everything works except c-vol 16:25:16 #link https://etherpad.openstack.org/p/cinder-active-active-vol-service-issues 16:25:20 e0ne: vol-stats update is a fanout 16:25:20 e0ne: the broadcast for that just works 16:25:32 DuncanT: glad to know it:) 16:25:50 e0ne: The scheduling becomes slightly none-deterministic, but not in a bad way 16:26:18 winston-d_: it works in our case too, but i was asking about out-of-the-box 16:26:24 DuncanT: anything else? 16:26:35 That's it for now, thanks 16:26:38 it would be nice to keep state management away from schedule 16:26:42 #topic open discussion 16:26:42 e0ne: yes, we use it out-of-the-box 16:26:45 scheduler* 16:26:53 winston-d_: great! 16:27:01 i'll check how it works in our case 16:27:21 what do you think about https://github.com/openstack/openstack-specs/blob/master/specs/no-downward-sql-migration.rst? 16:27:21 vilobhmm11: I don't think that is possible 16:27:35 downgrades is painful at least for sqlite 16:27:43 DuncanT : agree 16:27:49 winston-d_, but do you use c-vol services on different hosts to manage the same volumes/arrays? 16:28:01 e0ne: Testing object code is hard without downgrades 16:28:14 DuncanT: good point 16:28:26 is c-vol not A-A-able coz it maintains some state? 16:28:28 hemna: yes, of course 16:28:40 winston-d_, ok you are playing with fire then :) It will fail. 16:28:46 :) 16:28:48 rushiagr: Correct 16:29:00 hemna: almost 2 years, things are fine 16:29:05 rushiagr : right 16:29:29 hemna: maybe it is just our users behave 16:29:30 Just so it is also captured here - tentative schedule for Liberty release is: 16:29:33 L1: June 25, L2: July 30, L3: Spet 3, Release Oct 15 16:29:39 ok seems like we have no topics for open discussion. we can continue this chat in #openstack-cinder 16:29:50 if we are done with no-sql downgrade discussion 16:29:53 i have one 16:30:03 what is our plan for heirarchical projects 16:30:06 https://blueprints.launchpad.net/cinder/+spec/cinder-nested-quota-driver 16:30:24 thingee, DuncanT, other cores : ^^ 16:30:42 vilobhmm11: is hierarchical multitenancy part supported by keystone already? I thought it's still some way to complete.. 16:30:44 hemna: we did also, 3 volume hosts each has 8 volume services managing one array 16:30:53 yes its part of keystone in kilo 16:31:15 as every other project is moving toward heriarchical support i guess we should also think in that direction 16:31:36 vilobhmm11: well for one thing, we need to remove the deprecated code in DBQuoteDriver 16:31:40 :) 16:31:51 sure… 16:31:54 thingee ++ 16:31:57 winston-d_, it's just a matter of time, or timing, literally. 16:32:03 thingee: +1 16:32:21 do you think this a s apriority for liberty ? 16:32:31 thingee: anyone working on that that you are aware of? 16:32:55 winston-d_: nope 16:33:45 winston-d_ : i have a spec out there for review https://review.openstack.org/#/c/173141/ if anyone wants to have a look in there free time 16:33:51 winston-d_: https://bugs.launchpad.net/cinder/+bug/1435807 16:33:52 Launchpad bug 1435807 in Cinder "Volume type quotas trigger deprecation notification" [High,Triaged] - Assigned to Anton Arefiev (aarefiev) 16:34:12 winston-d_: I take that back 16:34:33 vilobhmm11: thx 16:35:01 vilobhmm11: ok seems like you're already on top of it :) 16:35:13 we will review after L specs opens 16:35:19 vilobhmm11: For a first pass, I suggest skipping the updating of quotas by anybody except admin, just do nested quota support 16:35:27 thingee : :) sure…please do 16:35:36 anything else? 16:35:36 vilobhmm11: Then add the editing by none-admin later 16:35:38 anybody? 16:35:51 DuncanT : sure…appreciate your feedback 16:36:33 ok thanks everyone! 16:36:35 #endmeeting