16:00:01 <smcginnis> #startmeeting Cinder
16:00:02 <openstack> Meeting started Wed Feb  3 16:00:01 2016 UTC and is due to finish in 60 minutes.  The chair is smcginnis. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:03 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:06 <openstack> The meeting name has been set to 'cinder'
16:00:06 <thingee> o/
16:00:08 <smcginnis> Courtesy ping:
16:00:09 <smcginnis> dulek duncant eharney geguileo winston-d e0ne jungleboyj jgriffith thingee smcginnis hemna xyang tbarron scottda erlon rhedlind vincent_hou kmartin patrickeast sheel dongwenjuan JaniceLee cFouts Thelo vivekd adrianofr
16:00:14 <e0ne> hi
16:00:15 <jungleboyj> Howdy!
16:00:15 <erlon> hey!
16:00:17 <patrickeast> hey
16:00:18 <eharney> hey
16:00:20 <kurtmartin> hello
16:00:20 <xyang1> hi
16:00:24 <adrianofr> hi
16:00:30 <tbarron> hi
16:00:30 <jgriffith_away> 0/
16:00:32 <smcginnis> Hey everyone!
16:00:34 <jgregor> Hello!
16:00:41 <DuncanT> Hi
16:00:46 <smcginnis> #topic Announcements
16:00:46 <ankit> hi
16:00:48 <yuriy_n17> Hi!
16:00:55 <scottda> Hi
16:01:01 <smcginnis> Realy quick announcements since we just had the midcycle. Then we can move one.
16:01:17 <smcginnis> Thanks everyone who participated last week. I think it went well.
16:01:30 <diablo_rojo> Hello :)
16:01:42 <jgriffith> smcginnis: +1
16:01:44 <smcginnis> And special thanks to hemna for the steaming and akerr for sitting in the corner all week to run local video. :)
16:01:56 <smcginnis> *streaming
16:02:07 <smcginnis> hemna: The amount of steaming was at a minimum. :P
16:02:16 <xyang1> :)
16:02:17 <erlon> smcginnis: did he held the camera in the head?
16:02:18 <flip214> hi
16:02:22 <smcginnis> #info Cinder - 502 open bugs, Python-cinderclient - 42 open bugs, OS-Brick - 13 open bugs
16:02:29 <scottda> And thanks to NetApp
16:02:33 <smcginnis> erlon: All over the place.
16:02:39 <smcginnis> scottda: Yes!
16:02:39 <erlon> lol
16:02:46 <erlon> nice
16:03:02 <smcginnis> Thanks NetApp for hosting and tim for coordinating!
16:03:07 <jungleboyj> smcginnis: Yes, it was good the steaming was at a minimum!
16:03:13 <smcginnis> Hehe
16:03:43 <smcginnis> OK, no major announcements yet. Please take time to do reviews and such. Let's move on.
16:03:44 <akerr> i'll take all the credit even though tim did all the planning
16:03:51 <smcginnis> akerr: ;)
16:04:01 <smcginnis> #topic Replication v2.1
16:04:18 <jgriffith> dum-dum-duuuuuu
16:04:22 <smcginnis> So if folks didn't get a chance to view the videos or read the notes...
16:04:35 <smcginnis> We decided to make a change in replication before it's too late.
16:04:54 <smcginnis> jgriffith: Want to say a few words about it? ;)
16:05:05 <jgriffith> smcginnis: sure
16:05:29 <jgriffith> smcginnis: so without a long background.... let's just say that we have a tendency to let scope creep kill us
16:05:49 <smcginnis> #link https://specs.openstack.org/openstack/cinder-specs/specs/mitaka/cheesecake.html New spec
16:05:53 <jgriffith> last week we took a look again at use cases, and let that be the definition
16:06:05 <jgriffith> So what we cam up with is a single DR use case
16:06:19 <jgriffith> Replication is set up on a backend.. no surprise there
16:06:27 <jgriffith> but fail-over is a backend-wide thing
16:06:32 <jgriffith> NOT volume based
16:06:50 <smcginnis> We really wanted to get back to a crawl, walk, run approach instead of over engineering right off the bat.
16:06:54 <jgriffith> So... if for example you have a backend and you have replication enabled and set up
16:07:12 <jgriffith> You create volumes on said backend, you can have type==replicated and just regular volumes
16:07:37 <jgriffith> if/when the DC catches fire and the admin issues a failover command, the backend driver is now pointed to the secondary device
16:07:55 <jgriffith> if volumes were replicated, cool.. you can stil access them.  if not, that's life
16:08:07 <jgriffith> This is more in line with actual real-worl use cases of replication
16:08:20 <jgriffith> make sense?
16:08:29 <smcginnis> It at least hits one of the primary use cases.
16:08:37 <smcginnis> I think we can expand over time to address more.
16:08:39 <Swanson> hello
16:08:47 <jungleboyj> jgriffith: Sounds good tome.
16:08:50 <smcginnis> But at least we can hit one use case well rather than several not well.
16:08:50 <patrickeast> jgriffith: got any code yet? :D
16:09:04 <jgriffith> patrickeast: yes
16:09:10 <jgriffith> patrickeast: I'll put a WIP up here shortly
16:09:14 * patrickeast cheers
16:09:21 <smcginnis> jgriffith: Thanks for your work on the spec and initial code.
16:09:25 <jgriffith> patrickeast: didn't get as far as I'd hoped to be by now... but at least it's a start
16:09:26 <fernnest_> jgriffith, is the idea that the replica is basically offline (no read-only) just receiving updates for the eventual fire.
16:09:26 <erlon> jgriffith: admin != tenant right?
16:09:37 <smcginnis> jgriffith: I think that's a huge help getting something out there, even just as a WIP for now.
16:09:49 <jgriffith> fernnest_: until a fail-over event yes... assuming I understand your question correctly
16:09:50 <patrickeast> smcginnis: +1
16:09:58 <jgriffith> smcginnis: ack
16:10:09 <smcginnis> fernnest_: Yes. Just a copy of the data off to another backend.
16:10:10 <fernnest_> jgriffith, I'm sure you do ;)
16:10:27 <diablo_rojo> erlon: Correct, admin != tenant
16:10:35 <smcginnis> fernnest_: Then it is basically read only from the control plane perspective other than attach/detach.
16:11:00 <smcginnis> IO plane is read write.
16:11:30 <jungleboyj> erlon: Right, that is one of the big things to remember in this approach is that it is all admin facing at this point.
16:11:30 <smcginnis> So watch for the code and please review and comment. Just let's please not nit pick this into Newton.
16:11:41 <jungleboyj> smcginnis: +1000
16:12:06 <smcginnis> This can get fixed up and evolve over time to what we need. But we need a good simple starting point to actually get there.
16:12:32 <erlon> jungleboyj: makes sense, it wouldnt make sense to allow the tenant have control over that IMO
16:12:44 <smcginnis> jgriffith: Thanks. Anything more to say about it? I know you have a conflict right now.
16:13:01 <jgriffith> Just questions if folks have them
16:13:08 <jgriffith> I'll be back in an hour or so
16:13:11 <jgriffith> hit me up on IRC
16:13:12 <jungleboyj> erlon: That is planned for future evolution.
16:13:17 <jgriffith> I'm pushing up some code right now
16:13:29 <smcginnis> Nice!
16:14:05 <smcginnis> OK, if no one has any immediate questions let's move on. Hit up jgriffith later if questions do come up.
16:14:11 <kurtmartin> jgriffith, thanks
16:14:20 <smcginnis> #topic Multiple Management IP Address Support Proposal
16:14:25 <jgriffith> crap.. rebase errors
16:14:32 <smcginnis> jgriffith: Doh! :)
16:14:35 <smcginnis> jungleboyj: Hey
16:14:41 <erlon> jungleboyj:  hmmm, I can' t imagine any situation where something like a disaster happen and the admin didn't have to interfere
16:14:42 <jungleboyj> smcginnis: Hey.
16:14:50 <jgriffith> DOHHH  the rpc version change merged into object
16:14:56 <jgriffith> k... gimmie just like 5 minutes
16:15:09 <smcginnis> jgriffith: No rush.
16:15:21 <smcginnis> jgriffith: Well, no huge rush. :)
16:15:33 <jungleboyj> So, we have had a request for the storwize_svc driver to mitigate a Single Point of Failure problem on the management IP.
16:15:35 <jgriffith> haha
16:15:41 <smcginnis> jungleboyj: That's that SDS controller, right?
16:15:44 <smcginnis> :P
16:15:53 <jungleboyj> smcginnis: :-p
16:16:32 <jungleboyj> Since it is based upon SANDriver we will probably have to make changes there to make this happen.
16:16:47 <smcginnis> jungleboyj: So the issue is you need to provide multiple management IPs?
16:17:11 <jungleboyj> Wanted to bring it up here to see if anyone else who is based off that class is concerned with us proposing this for Newton.
16:17:11 <patrickeast> why not just set up svc stuff behind like haproxy or something with a vip?
16:17:36 <smcginnis> Yeah, not clear why this is an issue, but I'm sure there's more to it.
16:17:36 <eharney> can you use a DNS entry which points to your multiple IPs and just configure cinder to talk to that?
16:17:45 <jungleboyj> Wonder if anyone else has need for it?
16:17:56 <eharney> i've never understood why we insist it's an "IP" in configuration rather than a generic host anyway
16:18:04 <e0ne> eharney: good question
16:18:07 <smcginnis> eharney: +1
16:18:28 <jungleboyj> So, the SVC allows multiple management IPs on the SVC box.
16:18:43 <DuncanT> eharney: DNS will round robin to dead IPs though, right?
16:18:48 <smcginnis> jungleboyj: If you want to use multiple, can't you just document to set san_ip to something like a comma separated list and have you're driver handle parsing that out?
16:18:51 <smcginnis> DuncanT: Yeah
16:18:55 <eharney> DuncanT: sure, but maybe that works with this client
16:18:57 <jungleboyj> Can you have one DNS name that points to multiple IPs?
16:19:06 <eharney> yes
16:19:07 <e0ne> jungleboyj: IMO, it would be good to get operators feedback for it
16:19:07 <DuncanT> jungleboyj: You certainly can
16:19:10 <e0ne> jungleboyj: yes
16:19:25 <smcginnis> jungleboyj: Yes, but DuncanT is right, it will respond with round robin through the addresses but no awareness of whether it is up or not.
16:19:27 <DuncanT> jungleboyj: requiring DNS for undercloud infra is a pain in the ass though
16:19:32 <patrickeast> is there a spec/code or something? what is the propsed change?
16:19:50 <eharney> i'm fine with supporting a list of addresses... just make them hosts/addresses and not "IPs"
16:19:55 <DuncanT> jungleboyj: comma separated list parsed in the driver seems totally reasonable to me
16:20:02 <DuncanT> eharney: +1
16:20:02 <jungleboyj> DuncanT: I agree that I would rather not tell them to go handle it with DNS.
16:20:16 <kurtmartin> jungleboyj, I need to check but I believe the LeftHand would support a list
16:20:22 <jungleboyj> DuncanT: smcginnis That seems like a reasonable and less invasive fix.
16:20:30 <DuncanT> DNS names being possible seems like a good thing to have
16:20:33 <smcginnis> And if it's a comma separated list (whether hostnames or IPs) that wouldn't require any other changes other than the consuming driver, right?
16:20:40 <e0ne> DuncanT: +1
16:20:46 <jungleboyj> kurtmartin: That would be good to know as far as a precedent is concerned.
16:21:33 <smcginnis> jungleboyj: OK, good for now?
16:22:01 <jungleboyj> smcginnis: Yep, that was why I wanted to bring it up here.  Figured you smart people would have already tackled it or have a good idea.
16:22:20 <jungleboyj> smcginnis: I am good.  Thank you DuncanT eharney smcginnis
16:22:27 <eharney> the glusterfs driver also has this concept of multiple addresses for the same backend, but it's configured differently.  this sounds good to me
16:22:46 <hemna> mornin
16:23:17 <smcginnis> OK, thanks.
16:23:27 <smcginnis> #topic Returning request ID to caller
16:23:34 <smcginnis> ankit: I changed the agenda item a little.
16:23:41 <smcginnis> ankit: Did you have something to discuss on this?
16:23:51 <ankit> yes I have seen that, thanks
16:23:59 <ankit> I request the cores to review return-request-id patches so that we can make it in Mitaka-3
16:24:09 <ankit> https://review.openstack.org/#/q/status:open+project:openstack/python-cinderclient+branch:master+topic:bp/return-request-id-to-caller
16:24:11 <smcginnis> ankit: The meeting isn't a place to just ask for reveiws, so this would be better to just ping in channel.
16:24:19 <smcginnis> ankit: Is there something to discuss?
16:24:51 <ankit> I am just looking for the feedback on these patches
16:25:06 <smcginnis> ankit: OK, then if there isn't anything else I'll move on.
16:25:14 <ankit> as it under review for long
16:25:22 <smcginnis> #topic Moving forward with nested quotas
16:25:24 <ankit> sure, thank you
16:25:27 * DuncanT is happy with them, I was hoping to see a few +1s though in case I missed something
16:25:30 <smcginnis> mc_nair: Hey, you're up.
16:25:32 <mc_nair> hey
16:25:36 <mc_nair> https://review.openstack.org/#/c/275734/ - First pass to get subset of nested quotas working, and disable the rest (i.e. -1 child limits)
16:25:52 <mc_nair> however, this has implications to existing deployments
16:26:05 <mc_nair> it requires Keystone fix https://review.openstack.org/#/c/270057/ if using Keystone v3 before doing anything with quotas (including create volume)
16:26:06 <smcginnis> #link https://review.openstack.org/#/c/275734/
16:26:16 <mc_nair> which is why it's failing so spectacularly on Tempest tests currently
16:26:28 <mc_nair> also it doesn't cleanup any messed up "allocated" values in DB for quotas of existing deployments which used NestedQuotas with -1 child limits
16:26:30 <DuncanT> mc_nair: Are there any tests for this? I think part of the problem last time was lack of (whitebox / functional) tests
16:26:36 <smcginnis> mc_nair: You should probably put a "Depends-On" line in the commit.
16:26:43 <mc_nair> smcginnis: will do
16:27:03 <mc_nair> DuncanT: I added more coverage, and will add more before remove WIP
16:27:24 <mc_nair> main question for now is - how do we want to deal with the fact that this would affect existing deployments?
16:27:43 <DuncanT> mc_nair: Can you say exactly what the impact is?
16:28:22 <smcginnis> mc_nair: So existing deployments are broken right now for nested quotas, correct?
16:28:28 <smcginnis> mc_nair: Does this make that any worse?
16:28:36 <mc_nair> yea, basically if you were to accept this patch you wouldn't be able to mange quotas or create volumes (because that affects quotas) if you're using Keystone v3 until you have that change to Keystone's policy.json
16:28:42 <mc_nair> and I don't see a way around that
16:29:20 <mc_nair> smcginnis: correct.  This also does not fix any problems in existing deployments if they're DB has messed up "allocated" values because -1 limits on child projects was not previously disallowed
16:29:52 <mc_nair> smcginnis: god I hope it doesn't make things worse ;)
16:29:57 <smcginnis> mc_nair: ;)
16:30:00 <mc_nair> should make things better moving forward
16:30:04 <smcginnis> mc_nair: There's a bug out there for this, right?
16:30:08 <mc_nair> just not a silver bullet by any means
16:30:09 <e0ne> mc_nair: can we implement DB migration for it?
16:31:13 <mc_nair> e0ne: it's *possible* to do something where we re-calculate the allocated values based on project hierarchies, but currently there's no way to get *entire* project hierarchy from Keystone (unless you're a cloud admin user)
16:31:41 <jungleboyj> Sounds like, at a minimum, we will need a really good release note to go with this.
16:31:56 <smcginnis> Yeah, release note will help somewhat.
16:32:03 <smcginnis> I would also like to see a bug files.
16:32:06 <smcginnis> *filed
16:32:14 <smcginnis> Then we can mark that as also affecting Liberty.
16:32:24 <smcginnis> And add notes to the bug explaining what to do.
16:32:33 <mc_nair> e0ne: I'm not articulating it well - but basically I think you *could* rectify this in some way using the project hierarchies, but we'd need someway to get the entire project hierarch (can follow up on this afterwards)
16:32:42 <mc_nair> smcginnis: will do - there's a few out there already I can link
16:32:45 <smcginnis> Then if it is resolved in Mitaka, at least the Liberty part can still indicate the problem existed there.
16:32:52 <smcginnis> mc_nair: OK, perfect.
16:33:00 <jungleboyj> smcginnis: Yeah, having a bug with the symptoms and steps to correct.  Similar info in the release note.
16:33:08 <mc_nair> but that's my next question - do we mark this as broken in any way for current Liberty?
16:33:18 <smcginnis> mc_nair: Yes, I think so.
16:33:24 <smcginnis> That's probably the best we can do.
16:33:41 <smcginnis> Any other ideas?
16:33:41 <e0ne> we have to add good info in release notes for it and do not breack existing deployments after upgrade
16:33:51 <DuncanT> mc_nair: Maybe a 'cinder-manage fix-quotas' that requires keystone admin creds?
16:34:28 <mc_nair> DuncanT: that's a possibility.  I'd like to get some auto-fix in place, but I think it will complicate the patch significantly also
16:34:54 <mc_nair> I'll work on that also though
16:35:17 <smcginnis> Existing deployments are already broken. So as long as it doesn't make it any worse...
16:35:21 <mc_nair> anyway, I can follow up on this in the Cinder channel, was just trying to get a path forward on this... two final quick questions
16:36:20 <mc_nair> 1. Even with the some cinder-manage fix-quotas command, once this patch is accepted the admin would need to get the new Keystone policy.json to get *even create volume* to work if they're using v3... is that acceptable?
16:36:34 <e0ne> smcginnis: agree, but we have to not break them more
16:36:54 <mc_nair> I don't know a way around it with the current approach, because otherwise we'd end up borking quotas again if we silently ignore the fact we can't grab project hierarchy
16:37:19 <e0ne> mc_nair: IMO, it's OK it this policy.json will be shipped with a new keystone
16:37:27 <smcginnis> mc_nair: Create volume with nested quotas? Or do you mean this impacts any volume creation?
16:37:34 <DuncanT> mc_nair: Would that be true even if you have no nested quotas?
16:37:43 <mc_nair> would be true if you're using Keystone v3
16:37:45 <scottda> I think quota-update in Liberty is broken without the policy.json fix : https://bugs.launchpad.net/cinder/+bug/1518513
16:37:46 <openstack> Launchpad bug 1518513 in Cinder "Quota update call fails in Liberty" [Medium,In progress] - Assigned to Dan Nguyen (daniel-a-nguyen)
16:38:05 <mc_nair> that's the big issue with the current design - it's not opt-in for Nested Quotas... once you get v3 you're using nested quotas
16:38:21 <mc_nair> and we can't tell if your quotas are nested if we cant grab the project hierarchy
16:38:23 <smcginnis> :/
16:38:32 <mc_nair> which is my next point
16:38:34 <mc_nair> Started prototyping in https://review.openstack.org/#/c/274825/
16:39:09 <DuncanT> Breaking cinder until you upgrade keystone is rather bad
16:39:14 <mc_nair> which I think would be nice to longer term move to - basically would be separate driver where you choose to use NestedQuotas
16:39:43 <mc_nair> DuncanT: yea :/ the other option is to add some config basically saying "I don't care about nested quotas" which would ignore any of the nested quotas stuff
16:40:06 <mc_nair> but you better not be creating hierarchical projects and ask for nested quotas later because it will already be messed up
16:40:26 <mc_nair> again, not well articulated, but hopefully somewhat gets the point across
16:40:58 <DuncanT> mc_nair: I guess we can all look at the patch and see if we can make suggestions.
16:41:25 <smcginnis> Yeah, I've got a bad feeling about this.
16:41:30 <DuncanT> mc_nair: I've suggested backing nested quotas out completely, but that breaks existing deployments too so isn't really any better
16:41:43 <mc_nair> smcginnis: that's what I aim foor
16:41:51 <smcginnis> :)
16:42:01 <mc_nair> DuncanT: yea, we're sort of in a bad place currently
16:42:31 <mc_nair> anyway, I think I've rambled enough, I can pick this up on the Cinder channel afterwards and if people could look at the approaches and ping me with questions / suggestions that'd be perfect
16:42:57 <smcginnis> #link https://review.openstack.org/#/c/274825/
16:43:06 <smcginnis> Please, the more eyes on this the better.
16:43:17 <smcginnis> This is sounding pretty scary right now.
16:43:39 <mc_nair> don't be afraid!
16:43:45 <mc_nair> j/k... you probably be
16:43:50 <mc_nair> *should be
16:43:54 <smcginnis> #topic Open Discussion
16:43:55 <smcginnis> :)
16:44:40 <smcginnis> Any other topics? Or we can keep talking about nested quotas.
16:44:49 <diablo_rojo> I have three cross project reviews that I want to draw attention to
16:45:02 <smcginnis> diablo_rojo: Oh, great. Yeah, go ahead.
16:45:09 <diablo_rojo> #link https://review.openstack.org/#/c/226157/ Backwards compatibility for libraries and clients
16:45:25 <diablo_rojo> #link  https://review.openstack.org/#/c/245629/ Common Policy Scenario more complex than admin or not admin
16:45:43 <diablo_rojo> #link https://review.openstack.org/#/c/257809/ Instances auto evacuation
16:45:58 <jungleboyj> https://review.openstack.org/#/c/245629/
16:46:10 <jungleboyj> Argh, copy paste fail.
16:46:26 <diablo_rojo> If I could get some eyes on those^^ we are planning to discuss them in our next wmeeting
16:47:29 <DuncanT> diablo_rojo: I've been trying to pretend the first one doesn't exist, since it's a mix of good ideas and bad. I'll actually write a review now though
16:47:40 <smcginnis> DuncanT: I agree.
16:47:45 <flip214> smcginnis: hemna: DuncanT: et.al: thanks for the help last week (DRBD & Nova). Much appreciated!
16:47:46 <smcginnis> Sounds great in theory.
16:47:46 <diablo_rojo> DuncanT:  That would be wonderful :)
16:48:19 <smcginnis> flip214: Yeah, don't know if we were much help, but at least it got brought up. Hopefully that helps move it forward.
16:48:46 <smcginnis> Anything else?
16:48:59 <DuncanT> Keeping back compatability is great. Stable branch clients are still really useful though - knowing which client features are supported by which release is otehrwise painful in the extreme
16:49:37 <smcginnis> DuncanT: I would think especially for packagers.
16:50:32 <smcginnis> OK, thanks everyone. I think that's all for today.
16:50:43 <smcginnis> #endmeeting