16:00:01 #startmeeting Cinder 16:00:02 Meeting started Wed Feb 3 16:00:01 2016 UTC and is due to finish in 60 minutes. The chair is smcginnis. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:00:03 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:00:06 The meeting name has been set to 'cinder' 16:00:06 o/ 16:00:08 Courtesy ping: 16:00:09 dulek duncant eharney geguileo winston-d e0ne jungleboyj jgriffith thingee smcginnis hemna xyang tbarron scottda erlon rhedlind vincent_hou kmartin patrickeast sheel dongwenjuan JaniceLee cFouts Thelo vivekd adrianofr 16:00:14 hi 16:00:15 Howdy! 16:00:15 hey! 16:00:17 hey 16:00:18 hey 16:00:20 hello 16:00:20 hi 16:00:24 hi 16:00:30 hi 16:00:30 0/ 16:00:32 Hey everyone! 16:00:34 Hello! 16:00:41 Hi 16:00:46 #topic Announcements 16:00:46 hi 16:00:48 Hi! 16:00:55 Hi 16:01:01 Realy quick announcements since we just had the midcycle. Then we can move one. 16:01:17 Thanks everyone who participated last week. I think it went well. 16:01:30 Hello :) 16:01:42 smcginnis: +1 16:01:44 And special thanks to hemna for the steaming and akerr for sitting in the corner all week to run local video. :) 16:01:56 *streaming 16:02:07 hemna: The amount of steaming was at a minimum. :P 16:02:16 :) 16:02:17 smcginnis: did he held the camera in the head? 16:02:18 hi 16:02:22 #info Cinder - 502 open bugs, Python-cinderclient - 42 open bugs, OS-Brick - 13 open bugs 16:02:29 And thanks to NetApp 16:02:33 erlon: All over the place. 16:02:39 scottda: Yes! 16:02:39 lol 16:02:46 nice 16:03:02 Thanks NetApp for hosting and tim for coordinating! 16:03:07 smcginnis: Yes, it was good the steaming was at a minimum! 16:03:13 Hehe 16:03:43 OK, no major announcements yet. Please take time to do reviews and such. Let's move on. 16:03:44 i'll take all the credit even though tim did all the planning 16:03:51 akerr: ;) 16:04:01 #topic Replication v2.1 16:04:18 dum-dum-duuuuuu 16:04:22 So if folks didn't get a chance to view the videos or read the notes... 16:04:35 We decided to make a change in replication before it's too late. 16:04:54 jgriffith: Want to say a few words about it? ;) 16:05:05 smcginnis: sure 16:05:29 smcginnis: so without a long background.... let's just say that we have a tendency to let scope creep kill us 16:05:49 #link https://specs.openstack.org/openstack/cinder-specs/specs/mitaka/cheesecake.html New spec 16:05:53 last week we took a look again at use cases, and let that be the definition 16:06:05 So what we cam up with is a single DR use case 16:06:19 Replication is set up on a backend.. no surprise there 16:06:27 but fail-over is a backend-wide thing 16:06:32 NOT volume based 16:06:50 We really wanted to get back to a crawl, walk, run approach instead of over engineering right off the bat. 16:06:54 So... if for example you have a backend and you have replication enabled and set up 16:07:12 You create volumes on said backend, you can have type==replicated and just regular volumes 16:07:37 if/when the DC catches fire and the admin issues a failover command, the backend driver is now pointed to the secondary device 16:07:55 if volumes were replicated, cool.. you can stil access them. if not, that's life 16:08:07 This is more in line with actual real-worl use cases of replication 16:08:20 make sense? 16:08:29 It at least hits one of the primary use cases. 16:08:37 I think we can expand over time to address more. 16:08:39 hello 16:08:47 jgriffith: Sounds good tome. 16:08:50 But at least we can hit one use case well rather than several not well. 16:08:50 jgriffith: got any code yet? :D 16:09:04 patrickeast: yes 16:09:10 patrickeast: I'll put a WIP up here shortly 16:09:14 * patrickeast cheers 16:09:21 jgriffith: Thanks for your work on the spec and initial code. 16:09:25 patrickeast: didn't get as far as I'd hoped to be by now... but at least it's a start 16:09:26 jgriffith, is the idea that the replica is basically offline (no read-only) just receiving updates for the eventual fire. 16:09:26 jgriffith: admin != tenant right? 16:09:37 jgriffith: I think that's a huge help getting something out there, even just as a WIP for now. 16:09:49 fernnest_: until a fail-over event yes... assuming I understand your question correctly 16:09:50 smcginnis: +1 16:09:58 smcginnis: ack 16:10:09 fernnest_: Yes. Just a copy of the data off to another backend. 16:10:10 jgriffith, I'm sure you do ;) 16:10:27 erlon: Correct, admin != tenant 16:10:35 fernnest_: Then it is basically read only from the control plane perspective other than attach/detach. 16:11:00 IO plane is read write. 16:11:30 erlon: Right, that is one of the big things to remember in this approach is that it is all admin facing at this point. 16:11:30 So watch for the code and please review and comment. Just let's please not nit pick this into Newton. 16:11:41 smcginnis: +1000 16:12:06 This can get fixed up and evolve over time to what we need. But we need a good simple starting point to actually get there. 16:12:32 jungleboyj: makes sense, it wouldnt make sense to allow the tenant have control over that IMO 16:12:44 jgriffith: Thanks. Anything more to say about it? I know you have a conflict right now. 16:13:01 Just questions if folks have them 16:13:08 I'll be back in an hour or so 16:13:11 hit me up on IRC 16:13:12 erlon: That is planned for future evolution. 16:13:17 I'm pushing up some code right now 16:13:29 Nice! 16:14:05 OK, if no one has any immediate questions let's move on. Hit up jgriffith later if questions do come up. 16:14:11 jgriffith, thanks 16:14:20 #topic Multiple Management IP Address Support Proposal 16:14:25 crap.. rebase errors 16:14:32 jgriffith: Doh! :) 16:14:35 jungleboyj: Hey 16:14:41 jungleboyj: hmmm, I can' t imagine any situation where something like a disaster happen and the admin didn't have to interfere 16:14:42 smcginnis: Hey. 16:14:50 DOHHH the rpc version change merged into object 16:14:56 k... gimmie just like 5 minutes 16:15:09 jgriffith: No rush. 16:15:21 jgriffith: Well, no huge rush. :) 16:15:33 So, we have had a request for the storwize_svc driver to mitigate a Single Point of Failure problem on the management IP. 16:15:35 haha 16:15:41 jungleboyj: That's that SDS controller, right? 16:15:44 :P 16:15:53 smcginnis: :-p 16:16:32 Since it is based upon SANDriver we will probably have to make changes there to make this happen. 16:16:47 jungleboyj: So the issue is you need to provide multiple management IPs? 16:17:11 Wanted to bring it up here to see if anyone else who is based off that class is concerned with us proposing this for Newton. 16:17:11 why not just set up svc stuff behind like haproxy or something with a vip? 16:17:36 Yeah, not clear why this is an issue, but I'm sure there's more to it. 16:17:36 can you use a DNS entry which points to your multiple IPs and just configure cinder to talk to that? 16:17:45 Wonder if anyone else has need for it? 16:17:56 i've never understood why we insist it's an "IP" in configuration rather than a generic host anyway 16:18:04 eharney: good question 16:18:07 eharney: +1 16:18:28 So, the SVC allows multiple management IPs on the SVC box. 16:18:43 eharney: DNS will round robin to dead IPs though, right? 16:18:48 jungleboyj: If you want to use multiple, can't you just document to set san_ip to something like a comma separated list and have you're driver handle parsing that out? 16:18:51 DuncanT: Yeah 16:18:55 DuncanT: sure, but maybe that works with this client 16:18:57 Can you have one DNS name that points to multiple IPs? 16:19:06 yes 16:19:07 jungleboyj: IMO, it would be good to get operators feedback for it 16:19:07 jungleboyj: You certainly can 16:19:10 jungleboyj: yes 16:19:25 jungleboyj: Yes, but DuncanT is right, it will respond with round robin through the addresses but no awareness of whether it is up or not. 16:19:27 jungleboyj: requiring DNS for undercloud infra is a pain in the ass though 16:19:32 is there a spec/code or something? what is the propsed change? 16:19:50 i'm fine with supporting a list of addresses... just make them hosts/addresses and not "IPs" 16:19:55 jungleboyj: comma separated list parsed in the driver seems totally reasonable to me 16:20:02 eharney: +1 16:20:02 DuncanT: I agree that I would rather not tell them to go handle it with DNS. 16:20:16 jungleboyj, I need to check but I believe the LeftHand would support a list 16:20:22 DuncanT: smcginnis That seems like a reasonable and less invasive fix. 16:20:30 DNS names being possible seems like a good thing to have 16:20:33 And if it's a comma separated list (whether hostnames or IPs) that wouldn't require any other changes other than the consuming driver, right? 16:20:40 DuncanT: +1 16:20:46 kurtmartin: That would be good to know as far as a precedent is concerned. 16:21:33 jungleboyj: OK, good for now? 16:22:01 smcginnis: Yep, that was why I wanted to bring it up here. Figured you smart people would have already tackled it or have a good idea. 16:22:20 smcginnis: I am good. Thank you DuncanT eharney smcginnis 16:22:27 the glusterfs driver also has this concept of multiple addresses for the same backend, but it's configured differently. this sounds good to me 16:22:46 mornin 16:23:17 OK, thanks. 16:23:27 #topic Returning request ID to caller 16:23:34 ankit: I changed the agenda item a little. 16:23:41 ankit: Did you have something to discuss on this? 16:23:51 yes I have seen that, thanks 16:23:59 I request the cores to review return-request-id patches so that we can make it in Mitaka-3 16:24:09 https://review.openstack.org/#/q/status:open+project:openstack/python-cinderclient+branch:master+topic:bp/return-request-id-to-caller 16:24:11 ankit: The meeting isn't a place to just ask for reveiws, so this would be better to just ping in channel. 16:24:19 ankit: Is there something to discuss? 16:24:51 I am just looking for the feedback on these patches 16:25:06 ankit: OK, then if there isn't anything else I'll move on. 16:25:14 as it under review for long 16:25:22 #topic Moving forward with nested quotas 16:25:24 sure, thank you 16:25:27 * DuncanT is happy with them, I was hoping to see a few +1s though in case I missed something 16:25:30 mc_nair: Hey, you're up. 16:25:32 hey 16:25:36 https://review.openstack.org/#/c/275734/ - First pass to get subset of nested quotas working, and disable the rest (i.e. -1 child limits) 16:25:52 however, this has implications to existing deployments 16:26:05 it requires Keystone fix https://review.openstack.org/#/c/270057/ if using Keystone v3 before doing anything with quotas (including create volume) 16:26:06 #link https://review.openstack.org/#/c/275734/ 16:26:16 which is why it's failing so spectacularly on Tempest tests currently 16:26:28 also it doesn't cleanup any messed up "allocated" values in DB for quotas of existing deployments which used NestedQuotas with -1 child limits 16:26:30 mc_nair: Are there any tests for this? I think part of the problem last time was lack of (whitebox / functional) tests 16:26:36 mc_nair: You should probably put a "Depends-On" line in the commit. 16:26:43 smcginnis: will do 16:27:03 DuncanT: I added more coverage, and will add more before remove WIP 16:27:24 main question for now is - how do we want to deal with the fact that this would affect existing deployments? 16:27:43 mc_nair: Can you say exactly what the impact is? 16:28:22 mc_nair: So existing deployments are broken right now for nested quotas, correct? 16:28:28 mc_nair: Does this make that any worse? 16:28:36 yea, basically if you were to accept this patch you wouldn't be able to mange quotas or create volumes (because that affects quotas) if you're using Keystone v3 until you have that change to Keystone's policy.json 16:28:42 and I don't see a way around that 16:29:20 smcginnis: correct. This also does not fix any problems in existing deployments if they're DB has messed up "allocated" values because -1 limits on child projects was not previously disallowed 16:29:52 smcginnis: god I hope it doesn't make things worse ;) 16:29:57 mc_nair: ;) 16:30:00 should make things better moving forward 16:30:04 mc_nair: There's a bug out there for this, right? 16:30:08 just not a silver bullet by any means 16:30:09 mc_nair: can we implement DB migration for it? 16:31:13 e0ne: it's *possible* to do something where we re-calculate the allocated values based on project hierarchies, but currently there's no way to get *entire* project hierarchy from Keystone (unless you're a cloud admin user) 16:31:41 Sounds like, at a minimum, we will need a really good release note to go with this. 16:31:56 Yeah, release note will help somewhat. 16:32:03 I would also like to see a bug files. 16:32:06 *filed 16:32:14 Then we can mark that as also affecting Liberty. 16:32:24 And add notes to the bug explaining what to do. 16:32:33 e0ne: I'm not articulating it well - but basically I think you *could* rectify this in some way using the project hierarchies, but we'd need someway to get the entire project hierarch (can follow up on this afterwards) 16:32:42 smcginnis: will do - there's a few out there already I can link 16:32:45 Then if it is resolved in Mitaka, at least the Liberty part can still indicate the problem existed there. 16:32:52 mc_nair: OK, perfect. 16:33:00 smcginnis: Yeah, having a bug with the symptoms and steps to correct. Similar info in the release note. 16:33:08 but that's my next question - do we mark this as broken in any way for current Liberty? 16:33:18 mc_nair: Yes, I think so. 16:33:24 That's probably the best we can do. 16:33:41 Any other ideas? 16:33:41 we have to add good info in release notes for it and do not breack existing deployments after upgrade 16:33:51 mc_nair: Maybe a 'cinder-manage fix-quotas' that requires keystone admin creds? 16:34:28 DuncanT: that's a possibility. I'd like to get some auto-fix in place, but I think it will complicate the patch significantly also 16:34:54 I'll work on that also though 16:35:17 Existing deployments are already broken. So as long as it doesn't make it any worse... 16:35:21 anyway, I can follow up on this in the Cinder channel, was just trying to get a path forward on this... two final quick questions 16:36:20 1. Even with the some cinder-manage fix-quotas command, once this patch is accepted the admin would need to get the new Keystone policy.json to get *even create volume* to work if they're using v3... is that acceptable? 16:36:34 smcginnis: agree, but we have to not break them more 16:36:54 I don't know a way around it with the current approach, because otherwise we'd end up borking quotas again if we silently ignore the fact we can't grab project hierarchy 16:37:19 mc_nair: IMO, it's OK it this policy.json will be shipped with a new keystone 16:37:27 mc_nair: Create volume with nested quotas? Or do you mean this impacts any volume creation? 16:37:34 mc_nair: Would that be true even if you have no nested quotas? 16:37:43 would be true if you're using Keystone v3 16:37:45 I think quota-update in Liberty is broken without the policy.json fix : https://bugs.launchpad.net/cinder/+bug/1518513 16:37:46 Launchpad bug 1518513 in Cinder "Quota update call fails in Liberty" [Medium,In progress] - Assigned to Dan Nguyen (daniel-a-nguyen) 16:38:05 that's the big issue with the current design - it's not opt-in for Nested Quotas... once you get v3 you're using nested quotas 16:38:21 and we can't tell if your quotas are nested if we cant grab the project hierarchy 16:38:23 :/ 16:38:32 which is my next point 16:38:34 Started prototyping in https://review.openstack.org/#/c/274825/ 16:39:09 Breaking cinder until you upgrade keystone is rather bad 16:39:14 which I think would be nice to longer term move to - basically would be separate driver where you choose to use NestedQuotas 16:39:43 DuncanT: yea :/ the other option is to add some config basically saying "I don't care about nested quotas" which would ignore any of the nested quotas stuff 16:40:06 but you better not be creating hierarchical projects and ask for nested quotas later because it will already be messed up 16:40:26 again, not well articulated, but hopefully somewhat gets the point across 16:40:58 mc_nair: I guess we can all look at the patch and see if we can make suggestions. 16:41:25 Yeah, I've got a bad feeling about this. 16:41:30 mc_nair: I've suggested backing nested quotas out completely, but that breaks existing deployments too so isn't really any better 16:41:43 smcginnis: that's what I aim foor 16:41:51 :) 16:42:01 DuncanT: yea, we're sort of in a bad place currently 16:42:31 anyway, I think I've rambled enough, I can pick this up on the Cinder channel afterwards and if people could look at the approaches and ping me with questions / suggestions that'd be perfect 16:42:57 #link https://review.openstack.org/#/c/274825/ 16:43:06 Please, the more eyes on this the better. 16:43:17 This is sounding pretty scary right now. 16:43:39 don't be afraid! 16:43:45 j/k... you probably be 16:43:50 *should be 16:43:54 #topic Open Discussion 16:43:55 :) 16:44:40 Any other topics? Or we can keep talking about nested quotas. 16:44:49 I have three cross project reviews that I want to draw attention to 16:45:02 diablo_rojo: Oh, great. Yeah, go ahead. 16:45:09 #link https://review.openstack.org/#/c/226157/ Backwards compatibility for libraries and clients 16:45:25 #link https://review.openstack.org/#/c/245629/ Common Policy Scenario more complex than admin or not admin 16:45:43 #link https://review.openstack.org/#/c/257809/ Instances auto evacuation 16:45:58 https://review.openstack.org/#/c/245629/ 16:46:10 Argh, copy paste fail. 16:46:26 If I could get some eyes on those^^ we are planning to discuss them in our next wmeeting 16:47:29 diablo_rojo: I've been trying to pretend the first one doesn't exist, since it's a mix of good ideas and bad. I'll actually write a review now though 16:47:40 DuncanT: I agree. 16:47:45 smcginnis: hemna: DuncanT: et.al: thanks for the help last week (DRBD & Nova). Much appreciated! 16:47:46 Sounds great in theory. 16:47:46 DuncanT: That would be wonderful :) 16:48:19 flip214: Yeah, don't know if we were much help, but at least it got brought up. Hopefully that helps move it forward. 16:48:46 Anything else? 16:48:59 Keeping back compatability is great. Stable branch clients are still really useful though - knowing which client features are supported by which release is otehrwise painful in the extreme 16:49:37 DuncanT: I would think especially for packagers. 16:50:32 OK, thanks everyone. I think that's all for today. 16:50:43 #endmeeting